SYSTEMS AND METHODS FOR OBJECT AND EVENT DETECTION AND FEATURE-BASED RATE-DISTORTION OPTIMIZATION FOR VIDEO CODING

Systems and methods for event and object detection and annotation in the video streams may include extracting a plurality of features in a picture in a video frame, grouping at least a portion of the plurality of features into at least one object, determining a region for the at least one object, assigning object identifiers to the at least one object and encoding the object identifiers into the bitstream. Feature-based rate distortion optimization may be employed for video coding including extracting a set of features from a picture in the video, generating a relevance map for the extracted features, determining a relevance score for portions of the picture using the relevance map, and encoding the portion of the picture with a bit rate determined at least in part by the relevance score.

FIELD OF THE INVENTION

The present invention generally relates to the field of video encoding and decoding. In particular, the present invention is directed to systems and methods for object and event detection and feature-based rate-distortion optimization for video coding.

BACKGROUND

A video codec can include an electronic circuit or software that compresses or decompresses digital video. It can convert uncompressed video to a compressed format or vice versa. In the context of video compression, a device that compresses video (and/or performs some function thereof) can typically be called an encoder, and a device that decompresses video (and/or performs some function thereof) can be called a decoder.

A format of the compressed data can conform to a standard video compression specification. The compression can be lossy in that the compressed video lacks some information present in the original video. A consequence of this can include that decompressed video can have lower quality than the original uncompressed video because there is insufficient information to accurately reconstruct the original video.

There can be complex relationships between the video quality, the amount of data used to represent the video (e.g., determined by the bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data losses and errors, case of editing, random access, end-to-end delay (e.g., latency), and the like.

Motion compensation can include an approach to predict a video frame or a portion thereof given a reference frame, such as previous and/or future frames, by accounting for motion of the camera and/or objects in the video. It can be employed in the encoding and decoding of video data for video compression, for example in the encoding and decoding using the Motion Picture Experts Group (MPEG)'s advanced video coding (AVC) standard (also referred to as H.264). Motion compensation can describe a picture in terms of the transformation of a reference picture to the current picture. The reference picture can be previous in time when compared to the current picture, from the future when compared to the current picture. When images can be accurately synthesized from previously transmitted and/or stored images, compression efficiency can be improved.

Recent trends in robotics, surveillance, monitoring, Internet of Things, etc. introduced use cases in which significant portion of all the images and videos that are recorded in the field is consumed by machines only, without ever reaching human eyes. Those machines process images and videos with the goal of completing tasks such as object detection, object tracking, segmentation, event detection etc. Recognizing that this trend is prevalent and will only accelerate in the future, international standardization bodies established efforts to standardize image and video coding that is primarily optimized for machine consumption. For example, standards like JPEG AI and Video Coding for Machines are initiated in addition to already established standards such as Compact Descriptors for Visual Search, and Compact Descriptors for Video Analytics.

Rate distortion optimization can be used to improve video encoding. As the name suggests, this process refers to the optimization of the amount of distortion (loss of video quality) against the amount of data required to encode the video, i.e., the rate. The present disclosure relates, in part, to systems and methods of rate distortion optimization applied in the context of video coding for machine consumption and hybrid video systems.

SUMMARY OF THE DISCLOSURE

A method of encoding video is provided that includes extracting a plurality of features in a picture in a video frame, grouping at least a portion of the plurality of features into at least one object, determining a region for the at least one object, assigning object identifiers to the at least one object, and encoding the object identifiers into the bitstream. In some embodiments, a feature model is used to extract the plurality of features. The object identifiers may include a region identifier and a label for each object.

The region is preferably represented by a geometric representation. In certain embodiments, the geometric representation is a bounding box or a contour. When the geometric representation is a bounding box, the bounding box may be a rectangle identified by the coordinates of a specific corner and the width and height of the bounding box. Alternatively, the bounding box may be a rectangle identified by the coordinates two diagonally opposing corners. If the geometric representation is a contour, the contour may be represented by a consecutive set of corners. For example, a first corner and consecutive corners clockwise or counterclockwise defining the entire contour. In each case, the bounding box or contour may be defined at a coding unit level and the corners represent a corner of a coding unit.

In some embodiments, an object may be further evaluated over a sequence of frames to determine an event. An event identifier cam be associated with an object and the event identifier encoded into the bitstream.

The object identifiers and event identifier can be inserted into the encoded bitstream. This information may be provided as supplemental enhancement information. Alternatively or in addition, the bitstream may include a slice header and the sliced header may signal the presence of an object in a given slice.

The video coding method for identifying objects and events may further include features for rate distortion optimization, including generating a relevance map for the extracted features, determining a relevance score for portions of the picture using the relevance map, and encoding the portion of the picture with a bit rate determined at least in part by the relevance score.

A method for encoding video with rate distortion optimization includes extracting a set of features from a picture in the video, generating a relevance map for the extracted features; determining a relevance score for portions of the picture using the relevance map, and encoding the portion of the picture with a bit rate determined at least in part by the relevance score. IN some embodiments, the picture is represented by a plurality of coding units and the relevance map is determined at the coding unit level with each coding unit having a coding unit relevance score. The encoding operation preferably includes allocating a bit rate for each coding unit. In some cases, the relevance score may include a relative relevance score for each coding unit.

In some embodiments, the encoding operations includes at least one of intra prediction, motion estimation, and transform quantization. In this case, the relative relevance score may be used in an explicit rate distortion optimization mode to alter the encoding during at least one of the intra prediction, motion estimation, and transform quantization processes. Alternatively, the relative relevance score can also be used in a rate distortion function to determine an adjusted bitrate for each coding unit.

The video encoding method with rate distortion optimization can also use the extracted features for object and event identification. In some embodiments, this may include grouping at least a portion of the extracted features into at least one object, determining a region for the at least one object, assigning object identifiers to the at least one object, and encoding the object identifiers into the bitstream.

An encoded video bitstream is also provided. The encoded bitstream includes encoded video content data which has at least one object identified by an encoder extracting a plurality of features of a picture in the video content. The bitstream includes at least one object identifier and associated object annotation and at least one event identifier and associated event annotation. The bitstream may include a supplemental enhancement information (SEI) message, wherein information related to the at least one object and at least one event is signaled in the SEI message. Alternatively or additionally, the bitstream may include a slice header, wherein information related to the at least one object and at least one event in a video slice is signaled in the slice header.

DETAILED DESCRIPTION

In many applications, such as surveillance systems with multiple cameras, intelligent transportation, smart city applications, and/or intelligent industry applications, traditional video coding may require compression of large number of videos from cameras and transmission through a network to machines and for human consumption. Subsequently, at a machine site, algorithms for feature extraction may applied typically using convolutional neural networks or deep learning techniques including object detection, event action recognition, pose estimation and others.FIG.1shows an exemplary embodiment of a standard VVC coder applied for machines. Conventional approaches unfortunately require a massive video transmission from multiple cameras, which may take significant time for efficient and fast real-time analysis and decision-making. In embodiments, a video coding for machines (“VCM”) approach may resolve this problem by both encoding video and extracting some features at a transmitter site and then transmitting a resultant encoded bit stream to a VCM decoder. As used herein, the term VCM is not limited to a specific proposed protocol but more generally includes all systems for coding and decoding video for machine consumption. At a decoder site video may be decoded for human vision and features may be decoded for machines. Systems which provide video for both human vision and for machine consumption are sometimes referred to as hybrid systems. The systems and methods disclosed herein are intended to apply to machine-based systems as well as hybrid systems.

A system and a method for rate-distortion optimization (RDO) for video coding based on the extracted features from the input video is disclosed. This method is suitable for any system that receives as input the video signal and can conduct both the feature extraction and video coding. Feature extraction can be classified as any computer vision task, such as edge detection, line detection, object detection, or more recent techniques such as convolutional neural networks where the output of the feature extraction can be spatially mapped back onto the pixel space of the input video. Video coding can include any standard video encoder that employs rate-distortion optimization, and/or encoding techniques such as partitioning, motion estimation and transform/quantization, such as Versatile Video Coding (VVC), or High Efficiency Video Coding (HEVC).

Embodiments of a system that supports the present methods is a Video Coding for Machines system depicted inFIGS.1-2below.

FIG.1is a high-level block diagram of a system for encoding and decoding video in a hybrid system which includes consumption of the video content by both human viewers and machine consumption. A source video is received by a video encoder105which provides a compressed bitstream for transmission over a channel to video decoder110. The video encoder may encode the video for human consumption as well as encoding the video for machine consumption. The video decoder110provides complimentary processing on the compressed bitstream to extract the video for human vision115as well as task analysis and feature extraction120for machine consumption.

Referring now toFIG.2, an exemplary embodiment of encoder for video coding for machines (VCM) is illustrated. VCM encoder200may be implemented using any circuitry including without limitation digital and/or analog circuitry; VCM encoder200may be configured using hardware configuration, software configuration, firmware configuration, and/or any combination thereof. VCM encoder200may be implemented as a computing device and/or as a component of a computing device, which may include without limitation any computing device as described below. In an embodiment, VCM encoder200may be configured to receive an input video204and generate an output bitstream208. Reception of an input video204may be accomplished in any manner described below. A bitstream may include, without limitation, any bitstream as described below.

VCM encoder200may include, without limitation, a pre-processor212, a video encoder216, a feature extractor220, an optimizer224, a feature encoder228, and/or a multiplexor232. Pre-processor212may receive input video204stream and parse out video, audio and metadata sub-streams of the stream. Pre-processor212may include and/or communicate with decoder as described in further detail below; in other words, Pre-processor212may have an ability to decode input streams. This may allow, in a non-limiting example, decoding of an input video204, which may facilitate downstream pixel-domain analysis.

Further referring toFIG.2, VCM encoder200may operate in a hybrid mode and/or in a video mode. When in the hybrid mode, VCM encoder200may be configured to encode a visual signal that is intended for human consumers, to encode a feature signal that is intended for machine consumers; machine consumers may include, without limitation, any devices and/or components, including without limitation computing devices as described in further detail below. Input signal may be passed, for instance when in hybrid mode, through pre-processor212.

Still referring toFIG.2, video encoder216may include without limitation any video encoder216as described in further detail below. When VCM encoder200is in hybrid mode, VCM encoder200may send unmodified input video204to video encoder216and a copy of the same input video204, and/or input video204that has been modified in some way, to feature extractor220. Modifications to input video204may include any scaling, transforming, or other modification that may occur to persons skilled in the art upon reviewing the entirety of this disclosure. For instance, and without limitation, input video204may be resized to a smaller resolution, a certain number of pictures in a sequence of pictures in input video204may be discarded, reducing framerate of the input video204, color information may be modified, for example and without limitation by converting an RGB video might be converted to a grayscale video, or the like.

Still referring toFIG.2, video encoder216and feature extractor220are connected and might exchange useful information in both directions. For example, and without limitation, video encoder216may transfer motion estimation information to feature extractor220, and vice-versa. Video encoder216may provide Quantization mapping and/or data descriptive thereof based on regions of interest (ROI), which video encoder216and/or feature extractor220may identify, to feature extractor220, or vice-versa. Video encoder216may provide to feature extractor220data describing one or more partitioning decisions based on features present and/or identified in input video204, input signal, and/or any frame and/or subframe thereof; feature extractor220may provide to video encoder216data describing one or more partitioning decisions based on features present and/or identified in input video204, input signal, and/or any frame and/or subframe thereof. Video encoder216feature extractor220may share and/or transmit to one another temporal information for optimal group of pictures (GOP) decisions. Each of these techniques and/or processes may be performed, without limitation, as described in further detail below.

With continued reference toFIG.2, feature extractor220may operate in an offline mode or in an online mode. Feature extractor220may identify and/or otherwise act on and/or manipulate features. A “feature,” as used in this disclosure, is a specific structural and/or content attribute of data. Examples of features may include SIFT, audio features, color hist, motion hist, speech level, loudness level, or the like. Features may be time stamped. Each feature may be associated with a single frame of a group of frames. Features may include high level content features such as timestamps, labels for persons and objects in the video, coordinates for objects and/or regions-of-interest, frame masks for region-based quantization, and/or any other feature that may occur to persons skilled in the art upon reviewing the entirety of this disclosure. As a further non-limiting example, features may include features that describe spatial and/or temporal characteristics of a frame or group of frames. Examples of features that describe spatial and/or temporal characteristics may include motion, texture, color, brightness, edge count, blur, blockiness, or the like. When in offline mode, all machine models as described in further detail below may be stored at encoder and/or in memory of and/or accessible to encoder. Examples of such models may include, without limitation, whole or partial convolutional neural networks, keypoint extractors, edge detectors, salience map constructors, or the like. When in online mode one or more models may be communicated to feature extractor220by a remote machine in real time or at some point before extraction.

Still referring toFIG.2, feature encoder228is configured for encoding a feature signal, for instance and without limitation as generated by feature extractor220. In an embodiment, after extracting the features feature extractor220may pass extracted features to feature encoder228. Feature encoder228may use entropy coding and/or similar techniques, for instance and without limitation as described below, to produce a feature stream, which may be passed to multiplexor232. Video encoder216and/or feature encoder228may be connected via optimizer224; optimizer224may exchange useful information between those video encoder216and feature encoder228. For example, and without limitation, information related to codeword construction and/or length for entropy coding may be exchanged and reused, via optimizer224, for optimal compression.

In an embodiment, and continuing to refer toFIG.2, video encoder216may produce a video stream; video stream may be passed to multiplexor232. Multiplexor232may multiplex video stream with a feature stream generated by feature encoder228; alternatively or additionally, video and feature bitstreams may be transmitted over distinct channels, distinct networks, to distinct devices, and/or at distinct times or time intervals (time multiplexing). Each of video stream and feature stream may be implemented in any manner suitable for implementation of any bitstream as described in this disclosure. In an embodiment, multiplexed video stream and feature stream may produce a hybrid bitstream, which may be is transmitted as described in further detail below.

Still referring toFIG.2, where VCM encoder200is in video mode, VCM encoder200may use video encoder216for both video and feature encoding. Feature extractor220may transmit features to video encoder216; the video encoder216may encode features into a video stream that may be decoded by a corresponding video decoder244. It should be noted that VCM encoder200may use a single video encoder216for both video encoding and feature encoding, in which case it may use different set of parameters for video and features; alternatively, VCM encoder200may two separate video encoder216s, which may operate in parallel.

Still referring toFIG.2, system200may include and/or communicate with, a VCM decoder236. VCM decoder236and/or elements thereof may be implemented using any circuitry and/or type of configuration suitable for configuration of VCM encoder200as described above. VCM decoder236may include, without limitation, a demultiplexor240. Demultiplexor240may operate to demultiplex bitstreams if multiplexed as described above; for instance and without limitation, demultiplexor240may separate a multiplexed bitstream containing one or more video bitstreams and one or more feature bitstreams into separate video and feature bitstreams.

Continuing to refer toFIG.2, VCM decoder236may include a video decoder244. Video decoder244may be implemented, without limitation in any manner suitable for a decoder as described in further detail below. In an embodiment, and without limitation, video decoder244may generate an output video, which may be viewed by a human or other creature and/or device having visual sensory abilities.

Still referring toFIG.2, VCM decoder236may include a feature decoder248. In an embodiment, and without limitation, feature decoder248may be configured to provide one or more decoded data to a machine. Machine may include, without limitation, any computing device as described below, including without limitation any microcontroller, processor, embedded system, system on a chip, network node, or the like. Machine may operate, store, train, receive input from, produce output for, and/or otherwise interact with a machine model as described in further detail below. Machine may be included in an Internet of Things (IoT), defined as a network of objects having processing and communication components, some of which may not be conventional computing devices such as desktop computers, laptop computers, and/or mobile devices. Objects in IoT may include, without limitation, any devices with an embedded microprocessor and/or microcontroller and one or more components for interfacing with a local area network (LAN) and/or wide-area network (WAN); one or more components may include, without limitation, a wireless transceiver, for instance communicating in the 2.4-2.485 GHZ range, like BLUETOOTH transceivers following protocols as promulgated by Bluetooth SIG, Inc. of Kirkland, Wash, and/or network communication components operating according to the MODBUS protocol promulgated by Schneider Electric SE of Rueil-Malmaison, France and/or the ZIGBEE specification of the IEEE 802.15.4 standard promulgated by the Institute of Electronic and Electrical Engineers (IEEE). Persons skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various alternative or additional communication protocols and devices supporting such protocols that may be employed consistently with this disclosure, each of which is contemplated as within the scope of this disclosure.

FIG.3is a block diagram further illustrating an encoder with feature and objection detection and rate distortion optimization. Input video is passed to the feature extractor which computes and extracts relevant features and sends pertinent information about those features (size, position, relevance score, labels, etc.) to the encoder, which uses this information to adjust the rate-distortion optimization such that the areas with more relevant features are encoded with higher quality (higher bitrate). The distribution of the available bandwidth is modulated based on the relevance mapping, instead of the default, content-agnostic distribution used by the encoder. Relevant components are depicted inFIG.3, including rate distortion optimization315, intra prediction320, transform/quantization325, motion estimation330, and entropy encoding335.

There are four basic stages of operation: feature extraction305, feature relevance mapping310, an application of the relevance map in the rate-distortion optimization process and encoding at a rate related to feature relevance by the video encoder300. This process is further illustrated in the flow diagram ofFIG.7. At a high level, an encoding method with rate distortion optimization in accordance with the present disclosure extracts features from the input video (step705), generates a relevance map of those features, preferably at the coding unit level (step710), determines a CU relevance score (step715) and encodes each CU at a rate that is determined, at least in part, by the relevance score of the CU. These steps are further described below.

FIG.3depicts the components of the proposed system and connections between them. Feature extractor305produces relevance map that is used by the encoder in two possible modes-explicit mode (dashed lines), or implicit mode (solid line). Following are detailed explanation of each of the components and modes.

Feature Extraction

As stated earlier, the feature extractor305conducts a process by which relevant features are extracted from the input video. Feature extractor305can implement simpler image processing and computer vision techniques such as edge, line, object detection, or more complex techniques such as Convolutional Neural Networks (CNNs) which can detect and identify objects and actions.

Any feature extraction process that can be mapped to the pixel positions of the input image can be used to generate relevance maps310. In the examples where edges, lines, and objects are detected, the corresponding pixels that represent/contain the edges, lines or objects are assigned appropriate high-relevance values, while the rest of the pixels in the picture are assigned low-relevance values. Each pixel of the input picture is assigned a relevance value in the relevance map.

In the examples where CNNs are used, the outputs of the arbitrary convolutional layer, also known as the feature maps, are mapped back onto the input pixels with appropriate pixel values.

Embodiments described herein may perform and/or be configured to perform object and event detection and annotation using the VCM encoder. An input picture is passed to both the feature extractor and video encoder. Video encoder is connected to the feature extractor305and can receive additional information about the input picture. Once the picture is processed by the feature extractor305, the relevant information about the detected objects and events is sent to the video encoder300.

Feature extractor305uses feature models such as convolutional neural networks, keypoint extractors, edge detectors, salience map constructors, etc. to extract relevant information about the objects and events in the input pictures.

The output of the feature extractor305is a set of [region, label] pairs for each picture. Regions can be represented as bounding boxes (FIG.4B), contours (FIG.4C), or other geometric representations. Labels are represented as words, strings, or other unique identifiers. Example of the input picture and feature detections is presented inFIGS.4A and4B, respectively.

The bounding boxes405,410can be represented using top left corner coordinates and width and height: (x, y, w, h). The contour can be represented using the clockwise consecutive set of corners, for example: (x1, y1, x2, y2, x3, y3, x4, y4, x5, y5), the edges between corners can be implicitly drawn using the neighboring corners' coordinates. The labels for each detection can be represented as strings of characters representing relevant words, such as “car”, “person”, etc.

Detections for the input picture are sent to the video encoder in the form of the set of triplets, for example: [(x1, y1, w1, h1), “car”, id1, (x2, y2, w2, h2), “person”, id2]—the third parameter is used for event id's and is equal to 0 if the detection is for object, not an event. The video encoder can copy or convert this information to the appropriate format of the annotations that are added to the video stream as a metadata or explicitly signaled.

To extend the concept of detections from objects to events, the feature extractor305can process multiple consecutive pictures and combine individual picture detections into a higher abstraction. One example of this process is when feature extractor305detects a car that occupies same spatial region in n consecutive pictures, and a person that occupies spatial region that is becoming closer to the car region in subsequent pictures. In this case, the whole sequence of detections can be abstracted into the event labeled “person entering car”. The event is signaled using the event id, which is present in multiple consecutive pictures and is interpreted as such by the video encoder, the list of events is sent to the video encoder as a set of pairs [id, “event”]. Example for an object detection: [(x1, y1, w1, h1), “car”, 0, (x2, y2, w2, h2), “person”, 0]. Example for an event detection: [(x1, y1, w1, h1), “car”, 1, (x2, y2, w2, h2), “person”, 1], [1, “person entering car”].

The video encoder can use the detections set information to map the detected regions to the coding units. Examples of the mapping are given inFIG.5. The coding unit can be represented as a macroblock, tree coding unit or a coding unit, depending on the video coding standard used. Any coding unit that contains whole region or any part of it is considered as an annotated coding unit (ACU). Encoder can use the ACU information to adjust parameters of the encoding process, such as quantization, partitioning, prediction type, etc. In some cases, the ACUs contain information that is considered to have higher priority than the rest of the picture and is encoded accordingly, usually using more bandwidth, which corresponds to lower quantization level for example. In the case of event detections, encoder can, for example, use finer resolution of the motion estimation and fractional motion vector precision to preserve more details.

Some embodiments disclosed herein may perform and/or be configured to perform object and event annotation signaling to the video decoder using metadata. As already described the information about detections is passed from the feature extractor to the video decoder in the form of sets of pairs or triplets. This information can be passed as-is (copied) or converted to different representation which is then inserted into video bitstream as a metadata, for example using the SEI (Supplemental Enhancement Information). One example of such SEI message syntax is presented in the following table:

SEI message contains elements that are defined within the initial payloadSize bytes, with additional payload with unspecified size that is reserved for future use and extensions.

Any decoder that implements SEI message parsing can extract the SEI message from the bitstream and process information about the objects and events that are detected in the video sequence. Parsed information can be used by the encoder to produce textual report about the objects and events in the video, or it can be used to render geometric shapes on top of the video together with textual information, such as labels to assist human viewer in identifying objects and events.

Embodiments described herein may be performed and/or be configured to perform object and event annotation explicit signaling to the video decoder. Information about the objects and events that is received from the feature extractor can also be converted into coding unit syntax elements that are present either at the slice level or at the level of the coding-tree unit (CTU).

In one implementation slice header (SH) is used to signal the presence of the object or event in the given slice. If the slice contains object, or event, or part of the object or the event, the proposed syntax elements signal to the decoder presence of the object or event. The slice header contains the list of the coding units that belong to the annotated object or the event, in the sequential raster-scan order. Example of the SH element is given in the following table:

Upon receiving slice header, decoder parses the information and marks all the CTUs that contain parts of the objects and events. In this implementation the region containing annotated objects and events is always represented as a group of contiguous CTUs.

An example of the feature extraction that detects objects and outputs object contours at the coding unit level is depicted inFIGS.5A-5C.

Relevance Map Generation:

Each pixel that belongs to the edge, line, object, or any other area that contains relevant features is assigned a value. Each pixel that does not belong to the relevant area is assigned a zero value, or some other low value. In the following examples, we will assume that the value range is between 0 and 1, and the real value number is assigned to each pixel. The proposed method supports other number ranges without limitations.

The values that are assigned are application-dependent and can be decided upon either in advance or normalized using the information obtained from the feature extraction process. For example, if only horizontal lines are detected, all pixel values that belong to the lines are assigned value 1.0, while all other pixels are assigned value 0.0. If extraction process detected lines of many orientations, the horizontal and vertical lines might be assigned higher values than the lines at the non-cardinal orientations, for example all cardinally oriented lines can be assigned value 1.0, all ordinally oriented lines can be assigned value 0.75, and all other lines 0.5. In the case of the object detection, if only one object is detected it can be assigned value 1.0, but if several objects are detected in the same picture, each can be assigned different value based on the size of the object or pre-determined importance of the given class of the objects. For example, the largest object can be assigned 1.0, and each subsequent object in the order of size can be assigned lower value. On the other hand, faces that are detected, regardless of the size can be assigned higher values than cars, etc.

InFIG.6A, we are depicting a simple example of an 8×8 pixel block that contains features with contour, and the resulting 8×8 relevance map illustrated inFIG.6B. The full relevance map has the same dimensions as the input picture and is used by the encoder for mapping of the pixel relevance to the rate distortion optimization (“RDO”) decisions.

Typically, video encoders do not make decisions on the single pixel level, but rather on the level of the so-called coding units (CUs). These are usually rectangular blocks of dimensions such as 64×64 pixels, 32×pixels, 16×16 pixels, etc. Since the RDO decisions are made on the level of single or group of CUs, the relevance map values of the pixels are averaged to obtain the CU relevance score.

As can be seen inFIGS.6A and6B, the CUs that contain features or parts of features will be designated as the more relevant as indicated by the value 1 inFIG.6B, compared to all other CUs in the given picture.

Consequently, the video encoder will try to encode each CU in the given picture considering the relevance score, on top of all the other considerations that are present in the RDO algorithm by default. In most of the cases, the CUs with a lower relevance score will be encoded using lower bitrate and vice versa.

For each pixel p in the n×n CU, and each relevance value v(p), the relevance score (“RS”) of the CU is calculated as follows:

This value is then compared to all the other RS(CU) values in the given picture and the relative relevance score (“RRS”) is computed:

where K is the total number of CU units in the given picture.

The RRS(CU) is calculated for each unit that is under consideration by the encoder at the time of encoding. In other words, encoder might be estimating RD cost for one 64×64 unit and calculating its RRS(CU) value, and then estimating cost for the four 32×32 sub-units and calculating their RRS(CU) value.

The RRS(CU) is used by the RDO to adjust the bitrate allocation for each coding unit. There are two modes that can be used by the encoder to apply RRS to the encoding parameters: (1) Explicit mode: in this mode RRS(CU) is used to modulate decisions in the particular stage of the encoding-intra prediction, motion estimation, or transform/quantization; (2) Implicit mode: In this mode RRS(CU) is used directly in the rate-distortion function.

The following are descriptions of each mode.

Explicit Mode

In the explicit mode the encoder uses RRS(CU) to modulate decisions in the following processes: (1) Intra prediction320—in particular, the partitioning process is adjusted based on the RRS(CU). Partitioning process is done in stages—each stage is performed at a higher depth of partitioning. Higher depth is producing smaller CUs, and hence allowing for finer details to be preserved. If the RRS(CU) is low, only the lower partitioning depth is estimated, and if the RRS(CU) is high, only the higher depth is estimated. In this way, the computational resources are saved, and the bitrate and quality are distributed according to the relevance. (2) Motion estimation330—in particular, the motion estimation precision and search ranges are adjusted based on the RRS(CU). Low scores are turning off the fractional motion vector precision and reducing the motion vector search range. High scores are doing the opposite. Again, the effects of the adjustment are similar to the intra case. (3) Transform/quantization325—in particular, the transform type and the quantization level are adjusted based on the RRS(CU). The transform that is used for lower score units is the simpler transform (for example, Hadamard transform instead of the Discrete Cosine Transform), while the higher score units still use full complexity transform. Quantization level is adjusted based on the RRS(CU) by directly applying the coefficient inversely proportional to the score to the quantization level (quantization parameter). Also, the highest RRS(CU) scores might use transform skip mode and encode as lossless areas of the picture that contain features of the highest relevance. Besides the transform skip mode, this can be achieved using the tools such as the ones available in the VVC standard for transform, scaling and quantization: disabling Sub-Block Transform (SBT), disabling Intra Sub-Partitions (ISP), disabling Multiple Transform Selection (MTS), disabling Low-Frequency Non-Separable Transform (LFNST), disabling Joint Coding of Chroma Residuals (JCCR), disabling Dependent Quantization (DQ), as well as the VVC tools for In-loop Filtering: disabling Deblocking Filter (DF), disabling Sample Adaptive Offset (SAO), disabling Adaptive Loop Filter (ALF), and disabling Lima Mapping with Chroma Scaling (LMCS).

Implicit Mode

In the implicit mode, the RRS(CU) is used directly in the rate-distortion function. Since this function determines the cost of the encoding decisions, this adjustment is implicitly affecting all other aspects of the encoding (partitioning, motion estimation, transform/quantization, etc.)

The standard RD function is of the following form: J=D+λR, where J is the cost function, D is the distortion measure and R is the bitrate, with the Lagrange multiplier λ that is used for the unconstrained optimization. The objective of the encoder is to find the encoding parameter set that minimizes the cost function J (find min(J)). The adjusted RD function is then J=D+λRR, where λRis calculated as

Here we are assuming that the RSS(CU) is normalized to the range (0.0, 1.0). In the formula, c is the adjustment coefficient from the range (0.0, 1.0), and d is the shift coefficient. For example, if c=0.2, d=0.5, formula becomes:

The value of λRdecreases with the higher RSS(CU), resulting in a higher bitrate used for those coding units, and opposite for the coding units with the lower relevance. The right coefficients can be calculated based on the application and use case. They can also be trained using neural networks to achieve desirable rate-distortion cost for a given set of features.

Referring now toFIG.8, an exemplary embodiment of a machine-learning module800that may perform one or more machine-learning processes as described in this disclosure is illustrated. Machine-learning module may perform determinations, classification, and/or analysis steps, methods, processes, or the like as described in this disclosure using machine learning processes. A “machine learning process,” as used in this disclosure, is a process that automatedly uses training data804to generate an algorithm that will be performed by a computing device/module to produce outputs808given data provided as inputs812; this is in contrast to a non-machine learning software program where the commands to be executed are determined in advance by a user and written in a programming language.

Referring now toFIG.9, an exemplary embodiment of neural network900is illustrated. A neural network900also known as an artificial neural network, is a network of “nodes,” or data structures having one or more inputs, one or more outputs, and a function determining outputs based on inputs. Such nodes may be organized in a network, such as without limitation a convolutional neural network, including an input layer of nodes904, one or more intermediate layers908, and an output layer of nodes912. Connections between nodes may be created via the process of “training” the network, in which elements from a training dataset are applied to the input nodes, a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is then used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce the desired values at the output nodes. This process is sometimes referred to as deep learning. Connections may run solely from input nodes toward output nodes in a “feed-forward” network, or may feed outputs of one layer back to inputs of the same or a different layer in a “recurrent network.”

Still referring toFIG.10, a “convolutional neural network,” as used in this disclosure, is a neural network in which at least one hidden layer is a convolutional layer that convolves inputs to that layer with a subset of inputs known as a “kernel,” along with one or more additional layers such as pooling layers, fully connected layers, and the like. CNN may include, without limitation, a deep neural network (DNN) extension, where a DNN is defined as a neural network with two or more hidden layers.

FIG.11is a system block diagram illustrating an example decoder1100. Decoder1100may include an entropy decoder processor1104, an inverse quantization and inverse transformation processor1108, a deblocking filter1112, a frame buffer1116, a motion compensation processor1120and/or an intra prediction processor1124.

In operation, and still referring toFIG.11, bit stream1128may be received by decoder1100and input to entropy decoder processor1104, which may entropy decode portions of bit stream into quantized coefficients. Quantized coefficients may be provided to inverse quantization and inverse transformation processor1108, which may perform inverse quantization and inverse transformation to create a residual signal, which may be added to an output of motion compensation processor1120or intra prediction processor1124according to a processing mode. An output of the motion compensation processor1120and intra prediction processor1124may include a block prediction based on a previously decoded block. A sum of prediction and residual may be processed by deblocking filter1112and stored in a frame buffer1116.

FIG.12is a system block diagram illustrating an example video encoder1200. Example video encoder1200may receive an input video1204, which may be initially segmented or dividing according to a processing scheme, such as a tree-structured macro block partitioning scheme (e.g., quad-tree plus binary tree). An example of a tree-structured macro block partitioning scheme may include partitioning a picture frame into large block elements called coding tree units (CTU). In some implementations, each CTU may be further partitioned one or more times into a number of sub-blocks called coding units (CU). A final result of this portioning may include a group of sub-blocks that may be called predictive units (PU). Transform units (TU) may also be utilized.

Still referring toFIG.12, example video encoder1200may include an intra prediction processor1208, a motion estimation/compensation processor1212, which may also be referred to as an inter prediction processor, capable of constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list, a transform/quantization processor1216, an inverse quantization/inverse transform processor1220, an in-loop filter1224, a decoded picture buffer1228, and/or an entropy coding processor1232. Bit stream parameters may be input to the entropy coding processor1232for inclusion in the output bit stream1236.

In operation, and with continued reference toFIG.12, for each block of a frame of input video, whether to process block via intra picture prediction or using motion estimation/compensation may be determined. Block may be provided to intra prediction processor1208or motion estimation/compensation processor1212. If block is to be processed via intra prediction, intra prediction processor1208may perform processing to output a predictor. If block is to be processed via motion estimation/compensation, motion estimation/compensation processor1212may perform processing including constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list, if applicable.

Further referring toFIG.12, a residual may be formed by subtracting a predictor from input video. Residual may be received by transform/quantization processor1216, which may perform transformation processing (e.g., discrete cosine transform (DCT)) to produce coefficients, which may be quantized. Quantized coefficients and any associated signaling information may be provided to entropy coding processor1232for entropy encoding and inclusion in output bit stream1236. Entropy encoding processor1232may support encoding of signaling information related to encoding a current block. In addition, quantized coefficients may be provided to inverse quantization/inverse transformation processor1220, which may reproduce pixels, which may be combined with a predictor and processed by in loop filter1224, an output of which may be stored in decoded picture buffer1228for use by motion estimation/compensation processor1212that is capable of constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list.

With continued reference toFIG.12, although a few variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, current blocks may include any symmetric blocks (8×8, 16×16, 32×32, 64×64, 128×128, and the like) as well as any asymmetric block (8×4, 16×8, and the like).

In some implementations, and still referring toFIG.12, a quadtree plus binary decision tree (QTBT) may be implemented. In QTBT, at a Coding Tree Unit level, partition parameters of QTBT may be dynamically derived to adapt to local characteristics without transmitting any overhead. Subsequently, at a Coding Unit level, a joint-classifier decision tree structure may eliminate unnecessary iterations and control the risk of false prediction. In some implementations, LTR frame block update mode may be available as an additional option available at every leaf node of QTBT.

In some implementations, and still referring toFIG.12, additional syntax elements may be signaled at different hierarchy levels of bitstream. For example, a flag may be enabled for an entire sequence by including an enable flag coded in a Sequence Parameter Set (SPS). Further, a CTU flag may be coded at a coding tree unit (CTU) level.

Some embodiments may include non-transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein.

FIG.13shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system1300within which a set of instructions for causing a control system to perform any one or more of the aspects and/or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and/or methodologies of the present disclosure. Computer system1300includes a processor1304and a memory1308that communicate with each other, and with other components, via a bus1312. Bus1312may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

Computer system1300may also include a storage device1324. Examples of a storage device (e.g., storage device1324) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device1324may be connected to bus1312by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device1324(or one or more components thereof) may be removably interfaced with computer system1300(e.g., via an external port connector (not shown)). Particularly, storage device1324and an associated machine-readable medium1328may provide nonvolatile and/or volatile storage of machine-readable instructions, data structures, program modules, and/or other data for computer system1300. In one example, software1320may reside, completely or partially, within machine-readable medium1328. In another example, software1320may reside, completely or partially, within processor1304.

Computer system1300may also include an input device1332. In one example, a user of computer system1300may enter commands and/or other information into computer system1300via input device1332. Examples of an input device1332include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device1332may be interfaced to bus1312via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus1312, and any combinations thereof. Input device1332may include a touch screen interface that may be a part of or separate from display1336, discussed further below. Input device1332may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

Computer system1300may further include a video display adapter1352for communicating a displayable image to a display device, such as display device1336. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter1352and display device1336may be utilized in combination with processor1304to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system1300may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus1312via a peripheral interface1356. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.