Patent Description:
Video coding (video encoding and decoding) is used in a wide range of digital video applications, for example broadcast digital TV, video transmission over internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders of security applications.

Since the development of the block-based hybrid video coding approach in the H. <NUM> standard in <NUM>, new video coding techniques and tools were developed and formed the basis for new video coding standards. One of the goals of most of the video coding standards was to achieve a bitrate reduction compared to its predecessor without sacrificing picture quality. Further video coding standards comprise MPEG-<NUM> video, MPEG-<NUM> video, ITU-T H. <NUM>/MPEG-<NUM>, ITU-T H. <NUM>, ITU-T H. <NUM>, High Efficiency Video Coding (HEVC), and extensions, e.g. scalability and/or three-dimensional (3D) extensions, of these standards. In standard video encoders and decoders compatible with H. <NUM>/AVC or HEVC (H. <NUM>) or next generations, the coding parameters, for example, frame rate, picture size or quantization parameter QP; used for video image encoding are often preset and signaled to the encoder. However, in particular for video surveillance applications a key problem is how to determine these coding parameters for an optimized performance of the computer vision device. In surveillance problems, a key aspect is to detect whether interesting objects, such as persons, motor vehicles or non-motor vehicles and the like, even exist in the video image apart from the image background, whether these objects move and how fast. Dependent on these factors, the coding parameters may be adapted on the fly so as to achieve an optimal surveillance result.

<CIT> discloses the determination of coding parameters based on the output of a neural network, which takes motion information and samples of the original image as input.

Embodiments are defined by the features of the independent claims, and further advantageous implementations of the embodiments by the features of the dependent claims. Enabling disclosure for the protected invention is provided with the embodiments described in relation to <FIG>. The other figures, aspects, and embodiments are provided for illustrative purposes and do not represent embodiments of the invention unless when combined with all of the features respectively defined in the independent claims.

Specifically, the present invention is defined by an apparatus for encoding samples of a video image according to independent claim <NUM> and a method for encoding samples of a video image according to independent claim <NUM>.

According to an aspect of the present disclosure, an apparatus is provided for encoding samples of a video image comprising a processing circuitry configured to determine motion information for a sample in the video image; determine a coding parameter based on an output of a machine-learning-based model, to which the motion information and the sample are input; and encode the sample by applying the coding parameter.

The use of a machine-learning-based model may provide an advantage of improving the determination of coding parameters by utilizing machine-learning models, which classify video images based on motion information.

In one exemplary implementation of the present disclosure, said sample is one of a predetermined number of samples in a block and the motion information for the sample is determined based on a motion vector of the block.

According to an aspect of the present disclosure, the processing circuitry of the apparatus is further configured to divide the video image into equally sized coding tree units; hierarchically split one of the coding tree units into a plurality of coding units; determine motion vector for each of the plurality of the coding units; input to the machine-learning-based model the coding tree unit and the respective motion vectors determined for the plurality of the coding units of the coding tree unit.

The provision of the input to the machine-learning model, including the video image and motion vector, in data pieces of a coding tree may provide an advantage of ensuring compatibility with standard video codes.

According to an aspect of the present invention, processing circuitry is configured to input to the machine-learning-based model (i) the coding tree unit; (ii) a block of the size of the coding tree unit including for each sample of the coding tree unit a horizontal component of the motion vector of the coding unit to which said sample belongs; and (iii) a block of the size of the coding tree unit including for each sample of the coding tree unit a vertical component of the motion vector of the coding unit to which said sample belongs.

In one exemplary embodiment of the present disclosure, the machine-learning-based model includes a first machine-learning-based sub-model to which the coding tree unit is input; a second machine-learning-based sub-model to which the motion vectors are input; and a third machine-learning-based sub-model to which an output of the first machine-learning-based sub-model and an output of the second machine-learning-based sub-model is input.

This may provide an advantage of performing pre-processing of the input data by the intermediate first and second sub-models in a separate manner. In particular, the pre-processing by the intermediate sub-models may include a separate classification of the input data, including the motion vector components and the coding tree unit, with each sub-model being optimized for its particular type of input. As a result, the coding parameter adaptation may be more accurate.

According to an aspect of the present disclosure, a method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image; determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input; and encoding the video frame by applying the coding parameter.

According to an aspect of the present disclosure, the method comprises further the steps of dividing the video image into equally sized coding tree units; hierarchically splitting one of the coding tree units into a plurality of coding units; determining motion vector for each of the plurality of the coding units; inputting to the machine-learning-based model the coding tree unit and the respective motion vectors determined for the plurality of the coding units of the coding tree unit.

According to an aspect of the present disclosure, the method comprises further the steps of inputting to the machine-learning-based model (i) the coding tree unit; (ii) a block of the size of the coding tree unit including for each sample of the coding tree unit a horizontal component of the motion vector of the coding unit to which said sample belongs; and (iii) a block of the size of the coding tree unit including for each sample of the coding tree unit a vertical component of the motion vector of the coding unit to which said sample belongs.

In one exemplary implementation of the present disclosure, the method steps related to the machine-learning-based model include the steps of inputting the coding tree unit to a first machine-learning-based sub-model; inputting the motion vectors to a second machine-learning-based sub-model; and inputting to a third machine-learning-based sub-model an output of the first machine-learning-based sub-model and an output of the second machine-learning-based sub-model.

In one exemplary implementation of the present disclosures, the method includes the step of classifying by the machine-learning-based model the input sample as belonging to a region of interest, ROI, or to a region of non-interest, RONI.

According to an aspect of the present disclosure, a computer-readable non-transitory medium is provided for storing a program, including instructions which when executed on a processor cause the processor to perform the method steps.

In the following, embodiments are described in more detail with reference to the attached figures and drawings, in which:.

In the following, identical reference signs refer to identical or at least functionally equivalent features.

In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments or specific aspects in which the described embodiments may be used. It is understood that embodiments may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

Video coding typically refers to the processing of a sequence of pictures, which form the video or video sequence. Instead of the term picture the terms frame or image may be used as synonyms in the field of video coding. Video coding comprises two parts, video encoding and video decoding. Video encoding is performed at the source side, typically comprising processing (e.g. by compression) the original video pictures to reduce the amount of data required for representing the video pictures (for more efficient storage and/or transmission). Video decoding is performed at the destination side and typically comprises the inverse processing compared to the encoder to reconstruct the video pictures. Embodiments referring to "coding" of video pictures (or pictures in general, as will be explained later) shall be understood to relate to both, "encoding" and "decoding" of video pictures. The combination of the encoding part and the decoding part is also referred to as CODEC (COding and DECoding).

In case of lossless video coding, the original video pictures can be reconstructed, i.e. the reconstructed video pictures have the same quality as the original video pictures (assuming no transmission loss or other data loss during storage or transmission). In case of lossy video coding, further compression, e.g. by quantization, is performed, to reduce the amount of data representing the video pictures, which cannot be completely reconstructed at the decoder, i.e. the quality of the reconstructed video pictures is lower or worse compared to the quality of the original video pictures.

Several video coding standards since H. <NUM> belong to the group of "lossy hybrid video codecs" (i.e. combine spatial and temporal prediction in the sample domain and 2D transform coding for applying quantization in the transform domain). Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks and the coding is typically performed on a block level. In other words, at the encoder the video is typically processed, i.e. encoded, on a block (video block) level, e.g. by using spatial (intra picture) prediction and temporal (inter picture) prediction to generate a prediction block, subtracting the prediction block from the current block (block currently processed/to be processed) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression), whereas at the decoder the inverse processing compared to the encoder is applied to the encoded or compressed block to reconstruct the current block for representation. Furthermore, the encoder duplicates the decoder processing loop such that both will generate identical predictions (e.g. intra- and inter predictions) and/or re-constructions for processing, i.e. coding, the subsequent blocks.

As video picture processing (also referred to as moving picture processing) and still picture processing (the term processing comprising coding), share many concepts and technologies or tools, in the following the term "picture" is used to refer to a video picture of a video sequence (as explained above) and/or to a still picture to avoid unnecessary repetitions and distinctions between video pictures and still pictures, where not necessary. In case the description refers to still pictures (or still images) only, the term "still picture" shall be used.

In the following, embodiments of an encoder <NUM>, a decoder <NUM> and a coding system <NUM> are described based on <FIG> (before describing embodiments in more detail based on <FIG>).

<FIG> is a schematic block diagram illustrating an embodiment of a coding system <NUM>, e.g. a picture coding system <NUM>, wherein the coding system <NUM> comprises a source device <NUM> configured to provide encoded data <NUM>, e.g. an encoded picture <NUM>, e.g. to a destination device <NUM> for decoding the encoded data <NUM>.

The source device <NUM> comprises an encoder <NUM> or encoding unit <NUM>, and may additionally, i.e. optionally, comprise a picture source <NUM>, a pre-processing unit <NUM>, e.g. a picture pre-processing unit <NUM>, and a communication interface or communication unit <NUM>.

The picture source <NUM> may comprise or be any kind of picture capturing device, for example for capturing a real-world picture, and/or any kind of a picture generating device, for example a computer-graphics processor for generating a computer animated picture, or any kind of device for obtaining and/or providing a real-world picture, a computer animated picture (e.g. a screen content, a virtual reality (VR) picture) and/or any combination thereof (e.g. an augmented reality (AR) picture). In the following, all these kinds of pictures and any other kind of picture will be referred to as "picture", unless specifically described otherwise, while the previous explanations with regard to the term "picture" covering "video pictures" and "still pictures" still hold true, unless explicitly specified differently.

A (digital) picture is or can be regarded as a two-dimensional array or matrix of samples with intensity values. A sample in the array may also be referred to as pixel (short form of picture element) or a pel. The number of samples in horizontal and vertical direction (or axis) of the array or picture define the size and/or resolution of the picture. For representation of color, typically three color components are employed, i.e. the picture may be represented or include three sample arrays. In RBG format or color space a picture comprises a corresponding red, green and blue sample array. However, in video coding each pixel is typically represented in a luminance/chrominance format or color space, e.g. YCbCr, which comprises a luminance component indicated by Y (sometimes also L is used instead) and two chrominance components indicated by Cb and Cr. The luminance (or short luma) component Y represents the brightness or grey level intensity (e.g. like in a grey-scale picture), while the two chrominance (or short chroma) components Cb and Cr represent the chromaticity or color information components. Accordingly, a picture in YCbCr format comprises a luminance sample array of luminance sample values (Y), and two chrominance sample arrays of chrominance values (Cb and Cr). Pictures in RGB format may be converted or transformed into YCbCr format and vice versa, the process is also known as color transformation or conversion. If a picture is monochrome, the picture may comprise only a luminance sample array.

The picture source <NUM> may be, for example a camera for capturing a picture, a memory, e.g. a picture memory, comprising or storing a previously captured or generated picture, and/or any kind of interface (internal or external) to obtain or receive a picture. The camera may be, for example, a local or integrated camera integrated in the source device, the memory may be a local or integrated memory, e.g. integrated in the source device. The interface may be, for example, an external interface to receive a picture from an external video source, for example an external picture capturing device like a camera, an external memory, or an external picture generating device, for example an external computer-graphics processor, computer or server. The interface can be any kind of interface, e.g. a wired or wireless interface, an optical interface, according to any proprietary or standardized interface protocol. The interface for obtaining the picture data <NUM> may be the same interface as or a part of the communication interface <NUM>.

In distinction to the pre-processing unit <NUM> and the processing performed by the pre-processing unit <NUM>, the picture or picture data <NUM> may also be referred to as raw picture or raw picture data <NUM>.

Pre-processing unit <NUM> is configured to receive the (raw) picture data <NUM> and to perform pre-processing on the picture data <NUM> to obtain a pre-processed picture <NUM> or pre-processed picture data <NUM>. Pre-processing performed by the pre-processing unit <NUM> may, e.g., comprise trimming, color format conversion (e.g. from RGB to YCbCr), color correction, or de-noising.

The encoder <NUM> is configured to receive the pre-processed picture data <NUM> and provide encoded picture data <NUM> (further details will be described, e.g., based on <FIG>).

Communication interface <NUM> of the source device <NUM> may be configured to receive the encoded picture data <NUM> and to directly transmit it to another device, e.g. the destination device <NUM> or any other device, for storage or direct reconstruction, or to process the encoded picture data <NUM> for respectively before storing the encoded data <NUM> and/or transmitting the encoded data <NUM> to another device, e.g. the destination device <NUM> or any other device for decoding or storing.

The destination device <NUM> comprises a decoder <NUM> or decoding unit <NUM>, and may additionally, i.e. optionally, comprise a communication interface or communication unit <NUM>, a post-processing unit <NUM> and a display device <NUM>.

The communication interface <NUM> of the destination device <NUM> is configured receive the encoded picture data <NUM> or the encoded data <NUM>, e.g. directly from the source device <NUM> or from any other source, e.g. a memory, e.g. an encoded picture data memory.

The communication interface <NUM> and the communication interface <NUM> may be configured to transmit respectively receive the encoded picture data <NUM> or encoded data <NUM> via a direct communication link between the source device <NUM> and the destination device <NUM>, e.g. a direct wired or wireless connection, or via any kind of network, e.g. a wired or wireless network or any combination thereof, or any kind of private and public network, or any kind of combination thereof.

The communication interface <NUM> may be, e.g., configured to package the encoded picture data <NUM> into an appropriate format, e.g. packets, for transmission over a communication link or communication network, and may further comprise data loss protection and data loss recovery.

The communication interface <NUM>, forming the counterpart of the communication interface <NUM>, may be, e.g., configured to de-package the encoded data <NUM> to obtain the encoded picture data <NUM> and may further be configured to perform data loss protection and data loss recovery, e.g. comprising error concealment.

Both, communication interface <NUM> and communication interface <NUM> may be configured as unidirectional communication interfaces as indicated by the arrow for the encoded picture data <NUM> in <FIG> pointing from the source device <NUM> to the destination device <NUM>, or bi-directional communication interfaces, and may be configured, e.g. to send and receive messages, e.g. to set up a connection, to acknowledge and/or re-send lost or delayed data including picture data, and exchange any other information related to the communication link and/or data transmission, e.g. encoded picture data transmission.

The decoder <NUM> is configured to receive the encoded picture data <NUM> and provide decoded picture data <NUM> or a decoded picture <NUM> (further details will be described, e.g., based on <FIG>).

The post-processor <NUM> of destination device <NUM> is configured to post-process the decoded picture data <NUM>, e.g. the decoded picture <NUM>, to obtain post-processed picture data <NUM>, e.g. a post-processed picture <NUM>. The post-processing performed by the post-processing unit <NUM> may comprise, e.g. color format conversion (e.g. from YCbCr to RGB), color correction, trimming, or resampling, or any other processing, e.g. for preparing the decoded picture data <NUM> for display, e.g. by display device <NUM>.

The display device <NUM> of the destination device <NUM> is configured to receive the post-processed picture data <NUM> for displaying the picture, e.g. to a user or viewer. The display device <NUM> may be or comprise any kind of display for representing the reconstructed picture, e.g. an integrated or external display or monitor. The displays may, e.g. comprise cathode ray tubes (CRT), liquid crystal displays (LCD), plasma displays, organic light emitting diodes (OLED) displays or any kind of other display, such as beamer, hologram (3D), or the like.

Although <FIG> depicts the source device <NUM> and the destination device <NUM> as separate devices, embodiments of devices may also comprise both or both functionalities, the source device <NUM> or corresponding functionality and the destination device <NUM> or corresponding functionality. In such embodiments the source device <NUM> or corresponding functionality and the destination device <NUM> or corresponding functionality may be implemented using the same hardware and/or software or by separate hardware and/or software or any combination thereof.

Therefore, the source device <NUM> and the destination device <NUM> as shown in <FIG> are just example embodiments and embodiments are not limited to those shown in <FIG>.

Source device <NUM> and destination device <NUM> may comprise any of a wide range of devices, including any kind of handheld or stationary devices, e.g. notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver device, or the like and may use no or any kind of operating system.

<FIG> shows a schematic/conceptual block diagram of an embodiment of an encoder <NUM>, e.g. a picture encoder <NUM>, which comprises an input <NUM>, a residual calculation unit <NUM>, a transformation unit <NUM>, a quantization unit <NUM>, an inverse quantization unit <NUM>, and inverse transformation unit <NUM>, a reconstruction unit <NUM>, a buffer <NUM>, a loop filter <NUM>, a decoded picture buffer (DPB) <NUM>, a prediction unit <NUM>, an inter estimation unit <NUM>, an inter prediction unit <NUM>, an intra-estimation unit <NUM>, an intra-prediction unit <NUM>, a mode selection unit <NUM>, an entropy encoding unit <NUM>, and an output <NUM>. A video encoder <NUM> as shown in <FIG> may also be referred to as hybrid video encoder or a video encoder according to a hybrid video codec.

For example, the residual calculation unit <NUM>, the transformation unit <NUM>, the quantization unit <NUM>, and the entropy encoding unit <NUM> form a forward signal path of the encoder <NUM>, whereas, for example, the inverse quantization unit <NUM>, the inverse transformation unit <NUM>, the reconstruction unit <NUM>, the buffer <NUM>, the loop filter <NUM>, the decoded picture buffer (DPB) <NUM>, the inter prediction unit <NUM>, and the intra-prediction unit <NUM> form a backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to the signal path of the decoder (see decoder <NUM> in <FIG>).

The encoder is configured to receive, e.g. by input <NUM>, a picture <NUM> or a picture block <NUM> of the picture <NUM>, e.g. picture of a sequence of pictures forming a video or video sequence. The picture block <NUM> may also be referred to as current picture block or picture block to be coded, and the picture <NUM> as current picture or picture to be coded (in particular in video coding to distinguish the current picture from other pictures, e.g. previously encoded and/or decoded pictures of the same video sequence, i.e. the video sequence which also comprises the current picture).

Embodiments of the encoder <NUM> may comprise a partitioning unit (not depicted in <FIG>), e.g. which may also be referred to as picture partitioning unit, configured to partition the picture <NUM> into a plurality of blocks, e.g. blocks like block <NUM>, typically into a plurality of non-overlapping blocks. The partitioning unit may be configured to use the same block size for all pictures of a video sequence and the corresponding grid defining the block size, or to change the block size between pictures or subsets or groups of pictures, and partition each picture into the corresponding blocks.

For example, in HEVC, the picture is subdivided into so-called coding tree units, CTUs. CTUs have the same size, which may be determined by some signaling parameters or fixed by standard. CTUs typically do not overlap. Each CTU may then be further split into coding and/or transform units, i.e. blocks for which prediction and/or transformation is performed, respectively. The splitting is performed recursively by providing possibility to divide each block further in four further blocks (quad-tree partitioning) or two further blocks (binary-tree partitioning) or any other kind of partitioning. The decision on whether a CTU is split and into which depth for each of the CTU's blocks may be taken by rate-distortion optimization. However, in general, the splitting may also be defined based on other criteria, such as presence of edges in the CTU, or the like. Accordingly, in the present disclosure, when referring to "block" or "unit" as a part of the image, CTU or any units obtained by CTU splitting - square or rectangular without being square - are meant. In general, embodiments are possible, in which the block may overlap.

Like the picture <NUM>, the block <NUM> again is or can be regarded as a two-dimensional array or matrix of samples with intensity values (sample values), although of smaller dimension than the picture <NUM>.

In other words, the block <NUM> may comprise, e.g., one sample array (e.g. a luma array in case of a monochrome picture <NUM>) or three sample arrays (e.g. a luma and two chroma arrays in case of a color picture <NUM>) or any other number and/or kind of arrays depending on the color format applied. The number of samples in horizontal and vertical direction (or axis) of the block <NUM> define the size of block <NUM>.

Encoder <NUM> as shown in <FIG> is configured encode the picture <NUM> block by block, e.g. the encoding and prediction is performed per block <NUM>.

The residual calculation unit <NUM> is configured to calculate a residual block <NUM> based on the picture block <NUM> and a prediction block <NUM> (further details about the prediction block <NUM> are provided later), e.g. by subtracting sample values of the prediction block <NUM> from sample values of the picture block <NUM>, sample by sample (pixel by pixel) to obtain the residual block <NUM> in the sample domain. Residual values (forming residual signal) correspond to prediction error (prediction error signal).

The transformation unit <NUM> is configured to apply a transformation, e.g. a spatial frequency transform or a linear spatial (frequency) transform, e.g. a discrete cosine transform (DCT) or discrete sine transform (DST), on the sample values of the residual block <NUM> to obtain transformed coefficients <NUM> in a transform domain. The transformed coefficients <NUM> may also be referred to as transformed residual coefficients and represent the residual block <NUM> in the transform domain.

The transformation unit <NUM> may be configured to apply integer approximations of DCT/DST, such as the core transforms specified for HEVC/H. Compared to an orthonormal DCT transform, such integer approximations are typically scaled by a certain factor. In order to preserve the norm of the residual block which is processed by forward and inverse transforms, additional scaling factors are applied as part of the transform process. The scaling factors are typically chosen based on certain constraints like scaling factors being a power of two for shift operation, bit depth of the transformed coefficients, tradeoff between accuracy and implementation costs, etc. Specific scaling factors are, for example, specified for the inverse transform, e.g. by inverse transformation unit <NUM>, at a decoder <NUM> (and the corresponding inverse transform, e.g. by inverse transformation unit <NUM> at an encoder <NUM>) and corresponding scaling factors for the forward transform, e.g. by transformation unit <NUM>, at an encoder <NUM> may be specified accordingly.

The quantization unit <NUM> is configured to quantize the transformed coefficients <NUM> to obtain quantized coefficients <NUM>, e.g. by applying scalar quantization or vector quantization. The quantized coefficients <NUM> may also be referred to as quantized residual coefficients <NUM>. For example for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, whereas larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a quantization parameter (QP). The quantization parameter may for example be an index to a predefined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes) and large quantization parameters may correspond to coarse quantization (large quantization step sizes) or vice versa. The quantization may include division by a quantization step size and corresponding or inverse dequantization, e.g. by inverse quantization <NUM>, may include multiplication by the quantization step size. Embodiments according to HEVC, may be configured to use a quantization parameter to determine the quantization step size. Generally, the quantization step size may be calculated based on a quantization parameter using a fixed point approximation of an equation including division. Additional scaling factors may be introduced for quantization and dequantization to restore the norm of the residual block, which might get modified because of the scaling used in the fixed point approximation of the equation for quantization step size and quantization parameter. In one example implementation, the scaling of the inverse transform and dequantization might be combined. Alternatively, customized quantization tables may be used and signaled from an encoder to a decoder, e.g. in a bitstream. The quantization is a lossy operation, wherein the loss increases with increasing quantization step sizes.

Embodiments of the encoder <NUM> (or respectively of the quantization unit <NUM>?) may be configured to output the quantization scheme and quantization step size, e.g. by means of the corresponding quantization parameter, so that a decoder <NUM> may receive and apply the corresponding inverse quantization. Embodiments of the encoder <NUM> (or quantization unit <NUM>) may be configured to output the quantization scheme and quantization step size, e.g. directly or entropy encoded via the entropy encoding unit <NUM> or any other entropy coding unit.

The inverse quantization unit <NUM> is configured to apply the inverse quantization of the quantization unit <NUM> on the quantized coefficients to obtain dequantized coefficients <NUM>, e.g. by applying the inverse of the quantization scheme applied by the quantization unit <NUM> based on or using the same quantization step size as the quantization unit <NUM>. The dequantized coefficients <NUM> may also be referred to as dequantized residual coefficients <NUM> and correspond - although typically not identical to the transformed coefficients due to the loss by quantization - to the transformed coefficients <NUM>.

The inverse transformation unit <NUM> is configured to apply the inverse transformation of the transformation applied by the transformation unit <NUM>, e.g. an inverse discrete cosine transform (DCT) or inverse discrete sine transform (DST), to obtain an inverse transformed block <NUM> in the sample domain. The inverse transformed block <NUM> may also be referred to as inverse transformed dequantized block <NUM> or inverse transformed residual block <NUM>.

The reconstruction unit <NUM> is configured to combine the inverse transformed block <NUM> and the prediction block <NUM> to obtain a reconstructed block <NUM> in the sample domain, e.g. by sample wise adding the sample values of the decoded residual block <NUM> and the sample values of the prediction block <NUM>.

The buffer unit <NUM> (or short "buffer" <NUM>), e.g. a line buffer <NUM>, is configured to buffer or store the reconstructed block and the respective sample values, for example for intra estimation and/or intra prediction. In further embodiments, the encoder may be configured to use unfiltered reconstructed blocks and/or the respective sample values stored in buffer unit <NUM> for any kind of estimation and/or prediction.

Embodiments of the encoder <NUM> may be configured such that, e.g. the buffer unit <NUM> is not only used for storing the reconstructed blocks <NUM> for intra estimation <NUM> and/or intra prediction <NUM> but also for the loop filter unit <NUM> (not shown in <FIG>), and/or such that, e.g. the buffer unit <NUM> and the decoded picture buffer unit <NUM> form one buffer. Further embodiments may be configured to use filtered blocks <NUM> and/or blocks or samples from the decoded picture buffer <NUM> (both not shown in <FIG>) as input or basis for intra estimation <NUM> and/or intra prediction <NUM>.

The loop filter unit <NUM> (or short "loop filter" <NUM>), is configured to filter the reconstructed block <NUM> to obtain a filtered block <NUM>, e.g. by applying a de-blocking filter, sample-adaptive offset (SAO) filter or other filters, e.g. sharpening or smoothing filters or collaborative filters. The filtered block <NUM> may also be referred to as filtered reconstructed block <NUM>.

Embodiments of the loop filter unit <NUM> may comprise (not shown in <FIG>) a filter analysis unit and the actual filter unit, wherein the filter analysis unit is configured to determine loop filter parameters for the actual filter. The filter analysis unit may be configured to apply fixed pre-determined filter parameters to the actual loop filter, adaptively select filter parameters from a set of predetermined filter parameters or adaptively calculate filter parameters for the actual loop filter as will be discussed in detail below.

Embodiments of the loop filter unit <NUM> may comprise (not shown in <FIG>) one or a plurality of filters, e.g. one or more of different kinds or types of filters, e.g. connected in series or in parallel or in any combination thereof, wherein each of the filters may comprise individually or jointly with other filters of the plurality of filters a filter analysis unit to determine the respective loop filter parameters, e.g. as described in the previous paragraph.

Embodiments of the encoder <NUM> (respectively loop filter unit <NUM>) may be configured to output the loop filter parameters, e.g. directly or entropy encoded via the entropy encoding unit <NUM> or any other entropy coding unit, so that, e.g., a decoder <NUM> may receive and apply the same loop filter parameters for decoding.

The decoded picture buffer (DPB) <NUM> is configured to receive and store the filtered block <NUM>. The decoded picture buffer <NUM> may be further configured to store other previously filtered blocks, e.g. previously reconstructed and filtered blocks <NUM>, of the same current picture or of different pictures, e.g. previously reconstructed pictures, and may provide complete previously reconstructed, i.e. decoded, pictures (and corresponding reference blocks and samples) and/or a partially reconstructed current picture (and corresponding reference blocks and samples), for example for inter estimation and/or inter prediction.

Further embodiments may also be configured to use the previously filtered blocks and corresponding filtered sample values of the decoded picture buffer <NUM> for any kind of estimation or prediction, e.g. intra and inter estimation and prediction.

The prediction unit <NUM>, also referred to as block prediction unit <NUM>, is configured to receive or obtain the picture block <NUM> (current picture block <NUM> of the current picture <NUM>) and decoded or at least reconstructed picture data, e.g. reference samples of the same (current) picture from buffer <NUM> and/or decoded picture data <NUM> from one or a plurality of previously decoded pictures from decoded picture buffer <NUM>, and to process such data for prediction, i.e. to provide a prediction block <NUM>, which may be an inter-predicted block <NUM> or an intra-predicted block <NUM>.

Mode selection unit <NUM> may be configured to select a prediction mode (e.g. an intra or inter prediction mode) and/or a corresponding prediction block <NUM> or <NUM> to be used as prediction block <NUM> for the calculation of the residual block <NUM> and for the reconstruction of the reconstructed block <NUM>.

Embodiments of the mode selection unit <NUM> may be configured to select the prediction mode (e.g. from those supported by prediction unit <NUM>), which provides the best match or in other words the minimum residual (minimum residual means better compression for transmission or storage), or a minimum signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which considers or balances both. The mode selection unit <NUM> may be configured to determine the prediction mode based on rate distortion optimization (RDO), i.e. select the prediction mode which provides a minimum rate distortion optimization or which associated rate distortion at least a fulfills a prediction mode selection criterion.

In the following the prediction processing (e.g. prediction unit <NUM> and mode selection (e.g. by mode selection unit <NUM>) performed by an example encoder <NUM> will be explained in more detail.

As described above, encoder <NUM> is configured to determine or select the best or an optimum prediction mode from a set of (pre-determined) prediction modes. The set of prediction modes may comprise, e.g., intra-prediction modes and/or inter-prediction modes.

The set of intra-prediction modes may comprise <NUM> different intra-prediction modes, e.g. non-directional modes like DC (or mean) mode and planar mode, or directional modes, e.g. as defined in H. <NUM>, or may comprise <NUM> different intra-prediction modes, e.g. non-directional modes like DC (or mean) mode and planar mode, or directional modes, e.g. as defined in H.

The set of (or possible) inter-prediction modes depend on the available reference pictures (i.e. previous at least partially decoded pictures, e.g. stored in DBP <NUM>) and other inter-prediction parameters, e.g. whether the whole reference picture or only a part, e.g. a search window area around the area of the current block, of the reference picture is used for searching for a best matching reference block, and/or e.g. whether pixel interpolation is applied, e.g. half/semi-pel and/or quarter-pel interpolation, or not.

Additional to the above prediction modes, skip mode and/or direct mode may be applied.

The prediction unit <NUM> may be further configured to partition the block <NUM> into smaller block partitions or sub-blocks, e.g. iteratively using quad-tree-partitioning (QT), binary partitioning (BT) or triple-tree-partitioning (TT) or any combination thereof, and to perform, e.g. the prediction for each of the block partitions or sub-blocks, wherein the mode selection comprises the selection of the tree-structure of the partitioned block <NUM> and the prediction modes applied to each of the block partitions or sub-blocks.

The inter estimation unit <NUM>, also referred to as inter picture estimation unit <NUM>, is configured to receive or obtain the picture block <NUM> (current picture block <NUM> of the current picture <NUM>) and a decoded picture <NUM>, or at least one or a plurality of previously reconstructed blocks, e.g. reconstructed blocks of one or a plurality of other/different previously decoded pictures <NUM>, for inter estimation (or "inter picture estimation"). a video sequence may comprise the current picture and the previously decoded pictures <NUM>, or in other words, the current picture and the previously decoded pictures <NUM> may be part of or form a sequence of pictures forming a video sequence.

The encoder <NUM> may, e.g., be configured to select a reference block from a plurality of reference blocks of the same or different pictures of the plurality of other pictures and provide a reference picture (or reference picture index) and/or an offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as inter estimation parameters <NUM> to the inter prediction unit <NUM>. This offset is also called motion vector (MV). The inter estimation is also referred to as motion estimation (ME) and the inter prediction also motion prediction (MP).

The inter prediction unit <NUM> is configured to obtain, e.g. receive, an inter prediction parameter <NUM> and to perform inter prediction based on or using the inter prediction parameter <NUM> to obtain an inter prediction block <NUM>.

Although <FIG> shows two distinct units (or steps) for the inter-coding, namely inter estimation <NUM> and inter prediction <NUM>, both functionalities may be performed as one (inter estimation requires/comprises calculating an/the inter prediction block, i.e. the or a "kind of" inter prediction <NUM>), e.g. by testing all possible or a predetermined subset of possible inter-prediction modes iteratively while storing the currently best inter prediction mode and respective inter prediction block, and using the currently best inter prediction mode and respective inter prediction block as the (final) inter prediction parameter <NUM> and inter prediction block <NUM> without performing another time the inter prediction <NUM>.

The intra estimation unit <NUM> is configured to obtain, e.g. receive, the picture block <NUM> (current picture block) and one or a plurality of previously reconstructed blocks, e.g. reconstructed neighbor blocks, of the same picture for intra estimation. The encoder <NUM> may, e.g., be configured to select an intra prediction mode from a plurality of intra prediction modes and provide it as intra estimation parameter <NUM> to the intra prediction unit <NUM>.

Embodiments of the encoder <NUM> may be configured to select the intra-prediction mode based on an optimization criterion, e.g. minimum residual (e.g. the intra-prediction mode providing the prediction block <NUM> most similar to the current picture block <NUM>) or minimum rate distortion.

The intra prediction unit <NUM> is configured to determine based on the intra prediction parameter <NUM>, e.g. the selected intra prediction mode <NUM>, the intra prediction block <NUM>.

Although <FIG> shows two distinct units (or steps) for the intra-coding, namely intra estimation <NUM> and intra prediction <NUM>, both functionalities may be performed as one (intra estimation comprises calculating the intra prediction block, i.e. the or a "kind of" intra prediction <NUM>) , e.g. by testing all possible or a predetermined subset of possible intra-prediction modes iteratively while storing the currently best intra prediction mode and respective intra prediction block, and using the currently best intra prediction mode and respective intra prediction block as the (final) intra prediction parameter <NUM> and intra prediction block <NUM> without performing another time the intra prediction <NUM>.

The entropy encoding unit <NUM> is configured to apply an entropy encoding algorithm or scheme (e.g. a variable length coding (VLC) scheme, an context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a context adaptive binary arithmetic coding (CABAC)) on the quantized residual coefficients <NUM>, inter prediction parameters <NUM>, intra prediction parameter <NUM>, and/or loop filter parameters, individually or jointly (or not at all) to obtain encoded picture data <NUM> which can be output by the output <NUM>, e.g. in the form of an encoded bitstream <NUM>.

<FIG> shows an exemplary video decoder <NUM> configured to receive encoded picture data (e.g. encoded bitstream) <NUM>, e.g. encoded by encoder <NUM>, to obtain a decoded picture <NUM>.

The decoder <NUM> comprises an input <NUM>, an entropy decoding unit <NUM>, an inverse quantization unit <NUM>, an inverse transformation unit <NUM>, a reconstruction unit <NUM>, a buffer <NUM>, a loop filter <NUM>, a decoded picture buffer <NUM>, a prediction unit <NUM>, an inter prediction unit <NUM>, an intra prediction unit <NUM>, a mode selection unit <NUM> and an output <NUM>.

The entropy decoding unit <NUM> is configured to perform entropy decoding to the encoded picture data <NUM> to obtain, e.g., quantized coefficients <NUM> and/or decoded coding parameters (not shown in <FIG>), e.g. (decoded) any or all of inter prediction parameters <NUM>, intra prediction parameter <NUM>, and/or loop filter parameters.

In embodiments of the decoder <NUM>, the inverse quantization unit <NUM>, the inverse transformation unit <NUM>, the reconstruction unit <NUM>, the buffer <NUM>, the loop filter <NUM>, the decoded picture buffer <NUM>, the prediction unit <NUM> and the mode selection unit <NUM> are configured to perform the inverse processing of the encoder <NUM> (and the respective functional units) to decode the encoded picture data <NUM>.

In particular, the inverse quantization unit <NUM> may be identical in function to the inverse quantization unit <NUM>, the inverse transformation unit <NUM> may be identical in function to the inverse transformation unit <NUM>, the reconstruction unit <NUM> may be identical in function reconstruction unit <NUM>, the buffer <NUM> may be identical in function to the buffer <NUM>, the loop filter <NUM> may be identical in function to the loop filter <NUM> (with regard to the actual loop filter as the loop filter <NUM> typically does not comprise a filter analysis unit to determine the filter parameters based on the original image <NUM> or block <NUM> but receives (explicitly or implicitly) or obtains the filter parameters used for encoding, e.g. from entropy decoding unit <NUM>), and the decoded picture buffer <NUM> may be identical in function to the decoded picture buffer <NUM>.

The prediction unit <NUM> may comprise an inter prediction unit <NUM> and an inter prediction unit <NUM>, wherein the inter prediction unit <NUM> may be identical in function to the inter prediction unit <NUM>, and the inter prediction unit <NUM> may be identical in function to the intra prediction unit <NUM>. The prediction unit <NUM> and the mode selection unit <NUM> are typically configured to perform the block prediction and/or obtain the predicted block <NUM> from the encoded data <NUM> only (without any further information about the original image <NUM>) and to receive or obtain (explicitly or implicitly) the prediction parameters <NUM> or <NUM> and/or the information about the selected prediction mode, e.g. from the entropy decoding unit <NUM>.

The decoder <NUM> is configured to output the decoded picture <NUM>, e.g. via output <NUM>, for presentation or viewing to a user.

<FIG> is a schematic diagram of a video coding device <NUM> according to an embodiment of the disclosure. The video coding device <NUM> is suitable for implementing the disclosed embodiments as described herein for example as an encoder or decoder. The video coding device <NUM> comprises ingress ports <NUM> and receiver units (Rx) <NUM> for receiving data; a processor, logic unit, or central processing unit (CPU) <NUM> to process the data; transmitter units (Tx) <NUM> and egress ports <NUM> for transmitting the data; and a memory <NUM> for storing the data. The video coding device <NUM> may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports <NUM>, the receiver units <NUM>, the transmitter units <NUM>, and the egress ports <NUM> for egress or ingress of optical or electrical signals. The video coding device <NUM> may also include wireless transmitters and/or receivers in some examples.

The processor <NUM> is implemented by hardware and software. The processor <NUM> may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor <NUM> is in communication with the ingress ports <NUM>, receiver units <NUM>, transmitter units <NUM>, egress ports <NUM>, and memory <NUM>. The processor <NUM> comprises a coding module <NUM>. The coding module <NUM> implements the disclosed embodiments described above. For instance, the coding module <NUM> implements, processes, prepares, or provides the various coding operations. The inclusion of the coding module <NUM> therefore provides a substantial improvement to the functionality of the video coding device <NUM> and effects a transformation of the video coding device <NUM> to a different state. Alternatively, the coding module <NUM> is implemented as instructions stored in the memory <NUM> and executed by the processor <NUM>.

The memory <NUM> comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory <NUM> may be volatile and/or non-volatile and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and/or static random-access memory (SRAM).

Note that this specification provides explanations for pictures (frames), but fields substitute as pictures in the case of an interlace picture signal.

Although embodiments have been primarily described based on video coding, it should be noted that embodiments of the encoder <NUM> and decoder <NUM> (and correspondingly the system <NUM>) may also be configured for still picture processing or coding, i.e. the processing or coding of an individual picture independent of any preceding or consecutive picture as in video coding. In general only inter-estimation <NUM>, inter-prediction <NUM>, <NUM> are not available in case the picture processing coding is limited to a single picture <NUM>. Most if not all other functionalities (also referred to as tools or technologies) of the video encoder <NUM> and video decoder <NUM> may equally be used for still pictures, e.g. partitioning, transformation (scaling) <NUM>, quantization <NUM>, inverse quantization <NUM>, inverse transformation <NUM>, intra-estimation <NUM>, intra-prediction <NUM>, <NUM> and/or loop filtering <NUM>, <NUM>, and entropy coding <NUM> and entropy decoding <NUM>.

The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step).

The terminology of "units" is merely used for illustrative purposes of the functionality of embodiments of the encoder/decoder and are not intended to limiting the disclosure.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely exemplary.

In addition, the functional units in the embodiments may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Embodiments may further comprise an apparatus, e.g. encoder and/or decoder, which comprises a processing circuitry configured to perform any of the methods and/or processes described herein.

Embodiments of the encoder <NUM> and/or decoder <NUM> may be implemented as hardware, firmware, software or any combination thereof. For example, the functionality of the encoder/encoding or decoder/decoding may be performed by a processing circuitry with or without firmware or software, e.g. a processor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or the like.

The functionality of the encoder <NUM> (and corresponding encoding method <NUM>) and/or decoder <NUM> (and corresponding decoding method <NUM>) may be implemented by program instructions stored on a computer readable medium. The program instructions, when executed, cause a processing circuitry, computer, processor or the like, to perform the steps of the encoding and/or decoding methods. The computer readable medium can be any medium, including non-transitory storage media, on which the program is stored such as a Blu ray disc, DVD, CD, USB (flash) drive, hard disc, server storage available via a network, etc..

An embodiment comprises or is a computer program comprising program code for performing any of the methods described herein, when executed on a computer.

An embodiment comprises or is a non-transitory computer readable medium comprising a program code that, when executed by a processor, causes a computer system to perform any of the methods described herein.

Video encoding may be employed in cameras or systems with an input from one or more cameras or may run on a computer to encode / compress or transcode videos. One of the challenging applications of camera systems is surveillance and/or driver assistance.

One major task of a surveillance system is detection and tracking (and often also identification) of objects, including persons, motor vehicle or non-motor vehicles or the like, in a video image, respectively, in its frames apart from the usually unimportant image background. In general, before a more detailed object identification is performed, the video, respectively, its frames are segmented first into a foreground image and background image, wherein the foreground image contains in most cases image details to be processed further. Commonly, the segmentation is performed through classification of a pixel or a group of pixels of the video image, respectively, of a frame whether the pixel or the pixel group belongs to the foreground or background based on certain classification criteria. These criteria are specified beforehand. The following <FIG> and <FIG> illustrate the problem of the erroneous classification of objects to belong to the uninteresting background or interesting foreground, when only a limited set of information is used. This results ultimately in an erroneous segmentation of the entire video image.

<FIG> shows three pedestrian persons <NUM> to <NUM> along a street <NUM> at different distances, resulting in different sizes. In view of the quite homogeneous texture of this exemplary image, a far distant pedestrian in particular may be difficult to be detected using image texture only. However, if motion information of the image, for example, based on motion vectors or optical flows, is additionally used, then even small objects may be detected even if the video image has a homogeneous texture. In other words, the motion information is a strong feature by which a region in the video image may be identified as relevant, corresponding to an interesting region. Those regions may be also referred to as regions of interest ROI, in which the motion of an object or multiple objects is equal to or larger than a predefined value, for example. The term motion refers, for example, to a velocity of an object within the video image, indicating the amount by which the object changes its location within the video image. Such velocity information for one or multiple objects may be obtained, for example, from the motion vectors and/or optical flows of the video image. In case, the video image is provided in terms of pixel units, such a velocity may be specified in units of pixel per second [pps]. By predefining one or more thresholds for the velocity or related quantities, the strength of a motion of an object may be specified and corresponding apparatus and/or method tasks may be triggered. For example, the presence of an object itself within a video image or its frames may be decided upon whether the object motion exceeds a predefined threshold or not. In turn, regions in the video image, which are not of interest or of less interest, may be referred to as regions of non-interest RONI, in which the motion of an object is not or less pronounced e.g. when the motion of the object is less than the predefined threshold.

<FIG> shows a similar scenario, where the texture of the image is now more complex, as it includes not only a pedestrian person <NUM> on the street <NUM>, but also an LED advertisement display <NUM> with moving content and a group of trees <NUM>, with the trees swinging slightly. In this scenario, the trees as a part of the image background were detected as those objects being part of the ROI, if motion information were used only. However, if the motion information is used in conjunction with texture (here the texture of trees), then the trees were (correctly) detected as being part of the RONI, since they belong to the, in general uninteresting, background.

<FIG> illustrates the image segmentation into background and foreground according to the approach of <NPL>. In their pixel-wise approach, a background image <NUM> is generated and maintained based on previous images. These images are then input to the CNN <NUM> for network training, using only texture to classify each pixel of a current image <NUM> whether or not the pixel belongs to the background <NUM> (i.e. RONI pixel) or to a (foreground) object (i.e. pixel belonging to a region of interest ROI). This approach has the disadvantage in case it is executed for classifying each pixel of the input image and thus suffers from a high computational complexity. Also, a background image <NUM> has to be generated and maintained explicitly. Moreover, no motion information, i.e. no motion statistics of possible interesting objects of differing size, is used, that may result in erroneous segmentation of a frame in ROI and RONI.

As disclosed in the following exemplary embodiments of the invention, the present disclosure exploits a machine-learning-based model, which is input with motion information and frames of a video image, for adapting one or multiple coding parameters used for encoding the video image.

According to an embodiment, an apparatus is provided wherein a video image, including a plurality of video frames, is encoded using coding parameters, so as to provide a video bitstream as the output. The encoding may be one of the known compression standards, such as AVC, HEVC, JPEG2000, or the like. In the following, the terms "video", "video image", "video frame", or "frame" are used interchangeably.

<FIG> shows an exemplary embodiment, where a video image is encoded in module <NUM> to generate a video bitstream based on coding parameters. In the embodiment of <FIG>, the coding parameters are also adapted within the module <NUM>. The parameters are adapted based on the output of a machine-learning model <NUM> and of the motion information determined by module <NUM>. The output of the machine-learning model <NUM> is determined based on the video image and the determined motion information, both of which are provided as input to the machine-learning model <NUM> during a training phase and/or during operation, depending on the type of the model. The video image may contain texture information, which is used by the machine-learning model <NUM> in conjunction with the motion information to perform foreground-background segmentation of the video image. The segmentation result of the machine-learning model <NUM> is output and provided as input to the module <NUM> to perform the coding parameter adaptation combined with the motion information. As further explained in the forthcoming embodiments, the image segmentation by machine-learning model is based on classification of the sample or the group of samples belonging to a region of interest ROI or a region of non-interest RONI.

The term texture information corresponds to texture of a sample or group of samples of a frame of the video image, which provides information about the spatial variation and/or arrangement of color (chroma) and/or intensity (luma) within the video image or a sample/pixel region of the video image. In other words, the term "texture" is used herein to denote the samples of a video image in the spatial domain.

The frames of the video image are used to determine the motion information <NUM> for a sample of the video image. The sample may correspond to one sample or multiple samples of the video image, i.e. a sample of one or more luma and/or color (i.e. chroma) components. The sample may correspond to a pixel of a video frame. The motion information and the video image sample(s) to which the motion information relates are provided as an input to a machine-learning model <NUM>, which may be subject to training or alternatively may be already trained.

The machine-learning-based model <NUM> may be any of a neural network NN, a deep neural network DNN, a convolutional neural network CNN, a fully connected neural network FCNN, a support vector machine SVM, or any other learning-based model.

The motion information <NUM> and the output of the machine-learning model <NUM> are used as the input to the encoder <NUM>. The machine-learning model <NUM> determines the input used for setting the coding parameters. In this illustrative embodiment, the encoder <NUM> determines the coding parameters by adapting them in accordance with its input, including the motion information and the output of the machine-learning-based model.

However, the embodiments are not limited to such an approach and, in general, the machine-learning model may be trained to directly deliver the coding parameters, which are then input to the encoder <NUM>. The coding parameters may be determined for one sample or a group of samples of the video image. The coding parameters of different single samples or different groups of samples may thus be different.

The sample is encoded by applying the determined coding parameters. The encoding of a sample or the samples is performed by an encoder <NUM> and provides the encoded video image as a bitstream video. The encoding of the sample may include encoding of one sample or a group of samples. The encoder may encode one sample or a group of samples differently in accordance with the different coding parameters. The groups of samples may be for instance blocks. In present codecs, some coding parameters may be varied on a block basis, others per one or more pictures (video frames). Alternatively, the groups of samples may be determined by segmentation and/or block detection without block raster.

In the embodiment of <FIG>, the processing circuitry of the apparatus is configured to perform both the adaptation of the coding parameters and the encoding. However, the coding parameter adaptation and the image encoding may be instead performed by different circuitries, arranged on the same or different chips. Alternatively, the coding parameter adaptation may be performed by the circuitry of the machine-learning model <NUM>. In other words, any of the tasks as discussed henceforth may be performed in a combined manner by a single circuitry or different circuitries. Parts of the tasks may also be combined in any suitable combination depending on the application.

In standard video encoders, the coding parameters, such as frame rate, spatial resolution, or compression strength are input to the encoder by a user, for example, by the application which uses the encoder and/or by a person employing the encoder. Some encoder implementations may, on the other hand, determine other coding parameters or even adapt the input coding parameters, based on rate-distortion optimization or rate-distortion-complexity optimization, or the like.

In the present disclosure, the coding parameters input to the encoder are (pre)determined automatically, based on the content characteristics, namely the texture (i.e. sample values) and the motion. This functionality is not necessarily a part of the encoder circuitry and may be implemented in a separate integrated circuit or other piece of software and/or hardware. However, the present disclosure is not limited to such implementation and the coding parameter determination may be implemented within the same hardware piece (IC, processor or the like) as the encoder.

In turn, the encoder (circuitry) may generate motion vectors which may be used as motion information to be input to the machine-based learning model. However, it is noted that motion information of the present disclosure is not limited to motion vectors determined by the encoder. Alternatively, the motion vectors as input for the coding parameter determination may be determined separately from the encoder. Moreover, the motion information is not limited to motion vectors, but rather may be provided by optical flows. Other quantifying parameters may be used as well, such as the speed and/or direction of motion. Alternatively or in addition, motion information may be determined based on, i.e. using the motion vectors and/or optical flow, as detailed in the following embodiments.

According to an embodiment shown in <FIG>, the processing circuitry of the apparatus <NUM> (marked by the dashed box) is configured to perform the tasks of the machine-learning based model <NUM>, the analysis of motion by a motion analyzer <NUM> based on the motion information, and the adaption of the coding parameters including frame rate adaptation <NUM>, picture size adaptation <NUM>, and generation of a QP map by a QP map generator <NUM> for QP adaptation.

The coding parameters picture size and quantization parameter QP refer to parameters of the spatial resolution. The frame rate refers to temporal resolution. The adaptation of picture size and frame rate is also denoted as ST adaptation. The coding parameters to be adapted for encoding the vie image are not limited to frame rate, picture size or quantization parameter. Other parameters used for encoding may include block splitting (how the video frame is segmented into blocks of possibly different size and/or shape), type of prediction or prediction mode, or the like.

Herein, the encoder, respectively, the circuitry is separate from the coding parameter adaptation, i.e. modules <NUM>, <NUM>, and <NUM>. The encoder provides motion information, for example, in terms of motion vectors of the video image and performs encoding.

The motion information may not be limited to motion vectors. Alternatively, the motion information may be based on optical flow of the video image, which may be obtained by well-known approaches, including Lucas-Kanade, Horn-Schunck, Buxton-Buxton, Black-Jepson, or the like.

Separating the circuitry of the encoder, i.e. the encoder itself, from the other circuitries allows using a video encoder (e.g. H. <NUM>/AVC or HEVC (H. <NUM>),as described above with reference to <FIG> and <FIG>, in which case the encoder needs an interfacing with the external functional blocks of the other modules. These modules then perform the coding parameter adaptation based on the motion information and the output of the machine-learning model, and provide the adapted coding parameters to the encoder by signaling for example. From the above description it is clear that each of the functional blocks of the apparatus may have its own circuitry in general, implemented as hardware and/or software.

The embodiments utilize a machine-learning-based model for the adaptation of the coding parameters, with motion information and video image information being provided as input to the learning model. Such a learning model, including NN, DNN, CNN, FCNN; SVM or the like, requires their input to be in a suitable format for further processing.

In order to assure compatibility with standard video encoders, for example H. <NUM>/AVC or HEVC (H. <NUM>), the input data including motion information and video image have to be transformed format for being suitable as input to the machine-learning model.

In HEVC, the picture is subdivided into so-called coding tree units, CTUs. CTUs have the same size, which may be determined by some signaling parameters or fixed by standard. A CTU may be further split into coding units CU and/or transform units. In other words, HEVC established CTU as a root for the tree-based hierarchical splitting, such as quad-tree splitting of CTUs and CUs to four square CUs until a maximum depth, corresponding to minimum size a CU is allowed to take.

How the motion information may be concatenated with the current CTU framework, is further detailed in the following embodiments shown in <FIG> by example of a convolutional neural model CNN.

The claimed embodiment of <FIG> shows how this is accomplished by a circuitry, which is configured to divide the video image into equally-sized coding-tree units CTUs <NUM>. The CTU is further hierarchically split into a plurality of coding units CUs. As mentioned, in H. <NUM>/AVC or HEVC (H. <NUM>), these CTUs/CUs are the basic block units, which are referred to as blocks. By default, the block size is <NUM> x <NUM> (in CTU unit) for HD videos, while <NUM> x <NUM> is also suitable. For <NUM>-videos, the recommended block size is <NUM> x <NUM>, while <NUM> x <NUM> is also used.

Next, motion information is determined for a sample of a block based on the motion vectors MVs of the block. In general, a block may include one sample or multiple samples. The term block refers to rectangular coding units, which may be but do not have to be square. The CTUs and the motion vectors are input to the machine-learning model in a suitable format.

The motion vectors MVs are provided by the encoder as motion information. The encoder may provide a motion vector for each sample, corresponding to a sample/pixel-wise motion vector. Alternatively, the motion vector may be provided for each block, i.e. for each of the plurality of CUs or the CTU. This corresponds block-wise motion vectors.

A motion vector has a horizontal x and vertical y component, i.e. MV = MV(x,y). The coordinates typically denote a vector with a starting point in a position of a block co-located in a reference picture with a current block in the current frame (picture), for which the motion vector is determined. The end position is the shift of a most similar block to the current block (among a limited number of candidates in a search space), wherein the similarity is determined by a cost function, such as a sum of absolute differences.

In the claimed embodiment of <FIG>, the motion vector matrix has the same size as the CTU. In other words, there is a motion vector associated with each sample/pixel of the CTU.

The 2D motion vector matrix MV(x,y) <NUM> is then split into its x and y (i.e. horizontal and vertical) components, MVx <NUM> and MVy <NUM>, both of which have also the same size as the CTU. As a result, the 2D MV components MVx and MVy can then be concatenated with the CTU matrix (array) <NUM> into one common 3D matrix. This concatenated 3D data <NUM>, including the three equal-sized 2D data matrices CTU <NUM>, MVx <NUM>, and MVy <NUM>, is then in a suitable format as input for the convolutional neural network CNN <NUM>.

The second exemplary embodiment of <FIG> shows how to prepare the input of the machine-learning model, when the MVs are not pixel-wise, but block-wise. This means that the MVs, as provided by the encoder for example, are not determined for each sample (i.e. pixel) of the CTU, but rather for each of the (sub-)blocks (i.e. the coding units) of the CTU.

Since the number of the coding units of a CTU is less than the number of pixels of a CTU, the size of a block-wise MV(x,y) <NUM> is typically smaller than the size of the CTU <NUM>. As a result, the components MVx <NUM> and MVy <NUM> of the block-wise motion vector <NUM> are also smaller than the CTU <NUM>. This is illustrated in <FIG>. Because of the different sizes of the CTU and the MV, respectively, the components MVx <NUM> and MVy <NUM> need to be first resized to the same size as the CTU. These data, including the CTU <NUM> and the resized components MVx <NUM> and MVy <NUM>, are then concatenated to the 3D data <NUM>. The data <NUM> is now in a suitable format and input to the CNN <NUM>.

The third exemplary embodiment of <FIG> considers a similar case as in the second exemplary embodiment of <FIG>, namely with the motion vector MV <NUM>, respectively, its components MVx <NUM> and MVy <NUM> being smaller than the CTU <NUM>. The difference to the second embodiment is that now a first machine-learning model <NUM> and a second machine-learning model <NUM> are included as intermediate machine-learning models.

The CTU is input to the first subnet <NUM>. The concatenated data <NUM> of the motion vector components MVx <NUM> and MVy <NUM> are input to the second subnet <NUM>. The second subnet <NUM> is configured to process the input data <NUM> such that the size of the output <NUM> of subnet <NUM> is the same as the output <NUM> of subnet <NUM>. As a result, both sets of output data <NUM> and <NUM> may be concatenated to the multi-dimensional data <NUM> into a suitable format for the CNN <NUM>. The data <NUM> is then input to the CNN <NUM>.

In general, all three networks <NUM>, <NUM>, and <NUM> may be different. In the exemplary embodiment shown in <FIG>, the subnets <NUM> and <NUM> may perform pre-processing separately, before the pre-processed data <NUM> and <NUM> are fused (i.e. merged or concatenated) to the data <NUM> and input to subnet <NUM>. In this example, the intermediate subnets <NUM> and <NUM> just provide intermediate results as output, without having a specific physical meaning. For example, the pre-processing may include filtering or the like of the input data.

Alternatively, intermediate subnets in particular may be configured such as to provide a physical output as result of pre-processed data. Since all subnets may be different, each of the subnets may be selected to perform a specific classification, for which the respective subnet is specifically designed for. For example, subnet <NUM> which is pre-processing the motion vector may be a network that is optimized for classifying motion information. As a result, subnet <NUM> may be, for example, very fast and accurate in classifying temporal changes in video images due to object motion. On the other hand, subnet <NUM> which is pre-processing the CTU may be a network that is optimized for classifying (here CTU-based) images using texture (or similar image features). As a result, subnet <NUM> may be, for example, very fast and accurate in classifying a CTU based on image features, which may be still image feature and/or dynamic image features. In turn, subnet <NUM> may be a network that may be less specific than the subnets <NUM> and <NUM>, as it uses the specific output data of subnets <NUM> and <NUM> as input to perform classification of the concatenated data. In other words, subnet <NUM> may be a network which is optimized to perform classification of mixed-type input data, comprising motion information and texture, for example. The embodiment of 10C is not limited to the one shown, and may include other network configurations, including different types of networks, different number of networks, and how they are connected with each other.

The above three embodiments of <FIG> account for the current video standard, where CTUs (including their coding units) are used as the basic units of the video image being encoded. However, the embodiment is not restricted to CTU/CU-based data structures, but rather is applicable to any other data structure suitable for next-generation coding-encoding technology. For example, the standard CTU/CU of H. <NUM>/AVC or HEVC (H. <NUM>), which relies on frame segmentation according a hierarchical tree partitioning, may in part or entirely be replaced by other image segmentation techniques based on clustering and/or using a machine-learning based model, wherein the motion information may not necessarily be based on motion vectors and/or optical flows of one sample and/or a group of samples of a block.

With the data <NUM>, <NUM>, and/or <NUM>, formatted in the manner as described above with reference to <FIG>, the machine-learning-based model <NUM> or <NUM> uses this data as input and performs classification of the input sample as to belonging to a region of interest ROI or a region of non-interest RONI. This ROI-RONI classification of a sample of a block results in an image segmentation of the video image, respectively, of its frames.

To perform the classification for an unknown sample and/or unknown group of samples of an unknown video, the model has been subjected to learning using known input data, obtained from known video images. Known video images means, for example, that image information of the video are known, including color intensity of a sample, texture, motion information, presence or absence of an object or multiple objects within the video, object type or the like are known and used for the training/learning of the model. Whether the sample or the samples of the block is classified as belonging to the ROI or RONI depends in particular on the motion information and texture of a sample of a block or of a block of samples.

According to an embodiment, the processing circuitry is configured to obtain as the output of the machine-learning-based model a map, which specifies the ROI and RONI of the video image. The map includes for each sample or block of samples an indication whether the said sample or block of samples belongs to the ROI or RONI.

The indication may be performed in various ways, wherein the map is a bitmap, for example. In this case, the map (also referred to as importance map) indicates on a one-bit level, whether the sample or the block of samples is a ROI or RONI by setting the <NUM>-bit for ROI and <NUM>-bit to RONI. In other words, the map may include for each sample of the video image an indication (such as a single bit), which indicates whether the respective sample belongs to ROI or to RONI. Alternatively, the indication may include more bits and indicate to which cluster (object) the respective sample belongs.

In case of a CTU/CU-based data structure, the machine-learning model may classify the entire CTUs or CUs as a ROI or RONI rather than each sample separately. In other words, the map may include an indication not for each sample of a video frame, but rather for each block (CU) of the video frame. The indication may be one or more bits. A one bit indication may take values "<NUM>" or "<NUM>", respectively for RONI or ROI. This ROI-RONI indication, based on the one-bit map, is just one example and not limited thereto. Other Boolean-type indicators may be used, such as "True/False", "Yes/No", and the like.

Therefore, the simultaneous use of texture and motion information improves the ROI-RONI classification of the image samples (pixels). As a result, the foreground-background image segmentation becomes more accurate and reliable. The segmentation may be performed for a frame (image) of the video or only for some frames.

According to an embodiment, the machine-learning-based model is configured to detect an object within the video image. The model detects an object within the video image using both texture of a sample or a group of samples and motion information (e.g. motion vectors). The output of the machine-learning model includes the detection result whether or not an object or multiple objects is/are in the video image. This output is also used by the coding parameter adaptation modules to adapt a coding parameter in accordance with the presence of an object.

The object detection provides the advantage of providing a more detailed information about the video image than just the ROI-RONI classification for coding-parameter adaptation. This, however, may result in a higher system complexity as opposed to a system that uses ROI-RONI classification only.

The combined use of texture and motion information enables a more reliable and accurate detection in particular of small objects (e.g. at far distances), which may move slowly.

The described embodiments so far describe the output of the machine-learning-based model, including the ROI-RONI map as a bit-wise importance map and the object detection result. As shown in <FIG>, the adaptation of the coding parameters, including frame rate, picture size and/or quantization parameter QP, depends not only on the output of the learning model, but also on the output of the motion analyzer <NUM> performing analysis of the motion described next.

In the embodiment corresponding to <FIG>, the calculated motion determines only the adaptation of the frame rate. Alternatively, the motion may determine any of the other coding parameters related to spatial resolution and/or temporal resolution.

According to an embodiment, the processing circuitry is further configured to calculate motion for a sample based on a metric of the motion vector of a block in which the sample is located.

The metric of the motion vector may be the p-norm, with p being a positive integer equal to or larger than one. Other types of metric fulfilling the well-known metric conditions may be used as well. For example, motion may be calculated as a length of the vector, i.e. as a square root of the sum of the vector component squares (square norm, i.e. p=<NUM>) or merely a size of one or both motion vector components, or the like. It is noted that the motion vector components may be counted in units of quarter pixel (quarter pel), a displacement of a quarter pel, half pel, and one pel, corresponding to a size of one, two, and <NUM>, respectively.

The term motion refers to a temporal change of the sample or a group of samples of a block, including changes of luma and/or color intensity within a video image and/or between different (e.g. adjacent) frames of a video. Determining such as change may not be restricted to motion vectors as motion information. Optical flow may be used alternatively or the like.

As shown in <FIG>, the motion is determined by the motion analyzer <NUM>. The analyzer is input with motion vectors as exemplary motion information, and provided by the video encoder <NUM> for example.

Various quantities may be calculated by the motion analyzer as motion, based on the motion vectors. For example, the motion may be the maximum motion of a video frame, with the maximum motion being measured in terms of the maximum motion velocity.

In case of the motion being the maximum motion velocity, the motion analyzer is configured to perform the following steps (assuming p = <NUM> in the p-norm):.

Other types of motion may be calculated in addition or alternatively. For example, the pixel-wise or block-wise motion vectors may be used to calculate statistical information related to motion statistics, including mean (i.e. mean motion or median motion) and/or standard deviation and/or skewness of the (motion) velocity, for example. Depending on any of such value, any of the coding parameters related to temporal or spatial resolution may be adapted accordingly, in addition to the output of the learning model.

In the above step-flow-chart, the frame rate adaptation <NUM> is part of the motion analyzer <NUM>, since the motion analyzer also performs the adaptation and the signaling of the adapted frame rate to the video encoder <NUM>.

However, as shown in the exemplary embodiment of <FIG>, the motion analyzer <NUM> and the frame rate adaptation <NUM> may perform the motion calculation and the parameter adaptation separately by different processing circuitries. This means that the signaling of the adapted frame rate to the encoder <NUM> is performed by the frame rate adaption <NUM>.

The described embodiments, specify how information is determined, which is used as input data for performing now the adaptation of the coding parameters. To recall, this entire information is obtained by use of motion information through e.g. motion vectors or video image, in conjunction with image texture and object detection.

With reference to <FIG>, it is described next how the outputs of the motion analyzer <NUM> and the machine-learning model <NUM> are used to adapt the respective coding parameters by the exemplary modules frame rate adaptation <NUM>, picture size adaptation <NUM>, and a QP map generator <NUM>. Their functionality are described with reference to <FIG>.

According to an embodiment, the processing circuitry is configured to determine the coding parameter by adapting the frame rate of the video images according to motion.

According to an embodiment, the frame rate is adapted in accordance with the amount of the motion detected in the video image. In addition or alternatively, the frame rate is adapted by setting the frame rate to a higher value when an object was detected in the video frame than in case no object was detected.

For example, with the maximum motion velocity vmax as the motion, the frame rate is increased when vmax is equal to or larger than a preset value T. In turn, when the object moves slowly, corresponding to a vmax smaller than the preset threshold T, the frame rate may be retained or lowered.

In other words, the frame rate is adapted based on the amount of motion contained of the video frame (i.e. the calculated output of the motion analyzer <NUM>) and/or whether or not an object is detected in the video frame (i.e. the output of the machine-learning model <NUM>).

For example, the frame rate may be increased from <NUM> fps (fps: frame per second) to <NUM> fps, when an object is detected and the motion is larger than a certain (empirical) predefined value. The frame rate of <NUM> fps is typical for normal and fast motion video clips. In turn, when an object is not detected at all corresponding to that the machine-learning based model has not classified a sample or block of samples as ROI, then no motion information is considered and the frame rate is lowered to <NUM> fps, for example.

Next, the adaptation of the coding parameters related to spatial resolution is discussed.

According to an embodiment, the processing circuitry is configured to determine the coding parameter by adapting the spatial resolution based on whether or not an object is detected in the video image. The parameter related to spatial resolution is picture size, for example.

Moreover, according to an embodiment, the coding parameter related to spatial resolution, being a number of samples per video image, is set to a higher value of the spatial resolution when an object is detected in the video image than when no object is detected.

As shown in <FIG>, the module picture size adaption <NUM> uses the output of the machine-learning model <NUM> (i.e. the result of the object detection) as input and adapts the picture size (i.e. use of the original resolution or the alternative resolution(s)) based on the object detection result. If no object has been detected in a sliding window of a time up to now, the resolution will be lowered from the default resolution, corresponding to a smaller picture size. If any object is detected in the current frame, the original (i.e. higher) resolution is taken. The adapted coding parameter picture size is signaled by the module <NUM> to the video encoder <NUM>.

In full HD, the default resolution is <NUM> x <NUM> (corresponding to picture size). An alternative of a lower resolution is <NUM> x <NUM>. Thus, the picture size is a coding parameter related to spatial resolution. The specified pixel numbers specifying picture sizes are not limited thereto. These may alter with advancing video image and display technology toward even higher spatial resolution, resulting in a larger picture size. For example, the default resolution in <NUM> technology will be <NUM> x <NUM>, with <NUM> x <NUM> or <NUM> x <NUM> being the lower alternative resolutions.

According to an embodiment, the coding parameter related to spatial resolution is a quantization parameter QP.

The quantization parameter refers to the inverse number of bits used for encoding. Thus, a small QP value corresponds to a large number of bits, respectively, a large number of smaller increments. As a result, finer spatial variations within the video image may be encoded, improving image quality.

According to an embodiment, the quantization parameter for a sample is set lower when the sample belongs to the ROI than in case the sample belongs to RONI.

As shown in <FIG>, the adaption (i.e. the setting) of the quantization parameter is performed by the QP map generator <NUM>, which uses the importance map (i.e. the ROI-RONI map) as input. This map is provided as output by the machine-learning model <NUM>. The QP map generator uses this importance map to generate a corresponding map of the quantization parameters. As described before, the ROI-RONI map may be a bitmap, wherein it is indicated bit-wise whether a sample or a block of samples belongs to ROI or RONI.

In case of standard HEVC codecs, this indication is done on a CTU/CU basis for compatibility reasons. When a bitmap is used, its entries contain for each CTU of a video frame either a value of "<NUM>" or "<NUM>" if it is an ROI or RONI. The QP map generator <NUM> uses this bitmap and adapts the coding parameter QP for each CTU/CU according to whether the respective entry in the bitmap is "<NUM>" or "<NUM>". When the CTU (or the coding block) is ROI, as indicated by the bit value "<NUM>" in the importance map, the module <NUM> sets the QP value to a lower value corresponding to a higher bit resolution. In other words, spatial gradations in the image content (in terms of luma and/or chroma) can be resolved by a larger number of (differing) values.

In turn, when the CTU/CU is RONI, as indicated by the bit value "<NUM>", the QP value is increased. The latter case would apply, for example, when the image content of the CTU/CU were a part of a sky, which has a weak, i.e. nearly homogeneous texture and/or no motion. Thus, in this exemplary case the content does not display strong spatial variations. A low(er) number of bits corresponding to a large QP value may be sufficient for encoding this part of the frame.

Based on the importance (bit)map, the QP map generator <NUM> generates a corresponding QP map, which contains the adapted QP value for a CTU/CU of the frame. This QP map is signaled by the module <NUM> to the video encoder <NUM>.

The object detection leads further to some bounding boxes for the ROI. For example, if a coding block CTU/CU is completely or partly within the ROI, a smaller QP value is assigned to this block. This leads to a higher image quality. In turn, when a coding block is completely within the RONI, a higher QP value is assigned to this block, resulting in a lower quality of the image.

For HEVC, the difference between the smaller QP and the higher QP may be <NUM>. However, this may be changed depending on the desired image quality of the ROI.

According to an embodiment, method is provided for encoding samples of a video image comprising the steps of determining motion information for samples of a block in the video image; determining a coding parameter based on an output of a machine-learning-based model, to which the motion information and the samples of the block are input; and encoding the video frame by applying the coding parameter.

According to an embodiment, a computer-readable non-transitory medium stores a program, including instructions which when executed on a processor cause the processor to perform the steps of the method.

<FIG> shows an exemplary embodiment, wherein the machine-learning engine, the encoder, the motion analyzer, and the parameter adaption described before, are part of a complete machine-vision-oriented video (en)coding system.

At the terminal side, there is a pipeline of an image signal processor ISP <NUM>, object detection by a CNN <NUM>, and video encoder <NUM>. The raw data from a camera sensor, including a lens <NUM>, are processed by the ISP, which results in video frames, for example, in a RGB format. Other alternative formats are possible such as YUV, YCbCr, or the like. Next, a CNN as one possible machine-learning model is used to apply the object detection algorithm to each of the RGB images. Finally, the detection result is used in the spatial-temporal adaptation <NUM> and QP map generator <NUM> for video coding. The coded video is transmitted to the cloud side <NUM> (e.g. a could server) through a network <NUM>, that may be a wired/wireless and/or mobile/immobile network.

The embodiments enable to perform a more accurate and reliable image segmentation of video images, using a machine-learning-based model, which is fed with motion information (e.g. motion vectors) of a frame and the video image, including image texture. The classification of a sample or a group of samples of a block (e.g. CTU/CU) of the video image as belonging to ROI or RONI, and the object detection using texture allows adaptation of spatial and temporal coding parameters in accordance with the result of the object detection, ROI-RONI segmentation and motion information.

As a result, the embodiments enable to adapt the coding complexity for a region within the video image and/or one of its frames by specifically tuning these parameters to image changes (i.e. time resolution such as frame rate) and to the importance and size of one or multiple detected moving objects (i.e. spatial resolution such as picture size or quantization parameter).

This improves the performance of automated computer-vision algorithms and apparatuses for machine vision applications, including image encoding/coding with optimal coding parameters.

Thus, the embodiments provide the advantageous effects of a more accurate background extraction and object detection, as relevant for video surveillance, computer-vision driven video (en)coding, or autonomous driving, where both texture, motion, and object detection are important.

The approach disclosed by the embodiments may be used and implemented on chips, in surveillance cameras, or other consumer devices with cameras having object-detection functionality.

Claim 1:
An apparatus (<NUM>,<NUM>) for encoding samples of a video image comprising:
a processing circuitry configured to:
determine motion information (<NUM>,<NUM>) for a sample in the video image;
determine a coding parameter (<NUM>,<NUM>,<NUM>,<NUM>) based on an output of a machine-learning-based model (<NUM>,<NUM>), to which the motion information and the sample are input; and
encode (<NUM>,<NUM>) the sample by applying the coding parameter,
wherein the processing circuitry is further configured to:
divide the video image into equally sized coding tree units (<NUM>);
hierarchically split one of the coding tree units into a plurality of coding units;
determine motion vector (<NUM>) for each of the plurality of the coding units;
input to the machine-learning-based model the coding tree unit (<NUM>) and the respective motion vectors (<NUM>) determined for the plurality of the coding units of the coding tree unit, and
wherein the processing circuitry is configured to input to the machine-learning-based model:
i) the coding tree unit;
characterized in that the processing circuitry is further configured to input to the machine-learning-based model:
ii) a block (<NUM>) of the size of the coding tree unit including for each sample of the coding tree unit a horizontal component of the motion vector (<NUM>) of the coding unit to which said sample belongs;
iii) a block (<NUM>) of the size of the coding tree unit including for each sample of the coding tree unit a vertical component of the motion vector (<NUM>) of the coding unit to which said sample belongs.