Feature-domain residual for video coding for machines

An apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extract features from the decoded data; decode encoded residual features to generate decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

TECHNICAL FIELD

The examples and non-limiting embodiments relate generally to multimedia transport and neural networks and, more particularly, to feature-domain residual for video coding for machines.

BACKGROUND

It is known to provide standardized formats for exchange of neural networks.

SUMMARY

In accordance with an aspect, an apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: encode original data with a first codec to generate encoded data with a bitrate lower than that of the original data, and decoded data; encode the original data with at least one second learned codec to generate encoded residual features and decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data generated with the first codec.

In accordance with an aspect, an apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extract features from the decoded data; decode encoded residual features to generate decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

In accordance with an aspect, a method includes decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extracting features from the decoded data; decoding encoded residual features to generate decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

When more than one drawing reference numeral is used herein with “/”, the “/” may be interpreted as either “or”, “and”, or “both”.

The following describes in detail a suitable apparatus and possible mechanisms for a video/image encoding process according to embodiments. In this regard reference is first made toFIG. 1andFIG. 2, whereFIG. 1shows an example block diagram of an apparatus50. The apparatus may be an Internet of Things (IoT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec.FIG. 2shows a layout of an apparatus according to an example embodiment. The elements ofFIG. 1andFIG. 2are explained next.

The electronic device50may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device. However, it would be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks.

The apparatus50may comprise a housing30for incorporating and protecting the device. The apparatus50further may comprise a display32in the form of a liquid crystal display. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display an image or video. The apparatus50may further comprise a keypad34. In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

The apparatus may comprise a microphone36or any suitable audio input which may be a digital or analog signal input. The apparatus50may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece38, speaker, or an analog audio or digital audio output connection. The apparatus50may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera capable of recording or capturing images and/or video. The apparatus50may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus50may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.

The apparatus50may comprise a controller56, processor or processor circuitry for controlling the apparatus50. The controller56may be connected to memory58which in embodiments of the examples described herein may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller56. The controller56may further be connected to codec circuitry54suitable for carrying out coding and/or decoding of audio and/or video data or assisting in coding and/or decoding carried out by the controller.

The apparatus50may further comprise a card reader48and a smart card46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

The apparatus50may comprise radio interface circuitry52connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus50may further comprise an antenna44connected to the radio interface circuitry52for transmitting radio frequency signals generated at the radio interface circuitry52to other apparatus(es) and/or for receiving radio frequency signals from other apparatus(es).

The apparatus50may comprise a camera capable of recording or detecting individual frames which are then passed to the codec54or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and/or storage. The apparatus50may also receive either wirelessly or by a wired connection the image for coding/decoding. The structural elements of apparatus50described above represent examples of means for performing a corresponding function.

With respect toFIG. 3, an example of a system within which embodiments of the examples described herein can be utilized is shown. The system10comprises multiple communication devices which can communicate through one or more networks. The system10may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA, LTE, 4G, 5G network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.

The system10may include both wired and wireless communication devices and/or apparatus50suitable for implementing embodiments of the examples described herein.

For example, the system shown inFIG. 3shows a mobile telephone network11and a representation of the internet28. Connectivity to the internet28may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

The example communication devices shown in the system10may include, but are not limited to, an electronic device or apparatus50, a combination of a personal digital assistant (PDA) and a mobile telephone14, a PDA16, an integrated messaging device (IMD)18, a desktop computer20, a notebook computer22. The apparatus50may be stationary or mobile when carried by an individual who is moving. The apparatus50may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.

The embodiments may also be implemented in a set-top box; i.e. a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.

Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection25to a base station24. The base station may be connected to a network server26that allows communication between the mobile telephone network11and the internet28. The system may include additional communication devices and communication devices of various types.

The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology. A communications device involved in implementing various embodiments of the examples described herein may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers.

The embodiments may also be implemented in so-called IoT devices. The Internet of Things (IoT) may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. The convergence of various technologies has and may enable many fields of embedded systems, such as wireless sensor networks, control systems, home/building automation, etc. to be included in the Internet of Things (IoT). In order to utilize the Internet IoT devices are provided with an IP address as a unique identifier. IoT devices may be provided with a radio transmitter, such as a WLAN or Bluetooth transmitter or a RFID tag. Alternatively, IoT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC).

An MPEG-2 transport stream (TS), specified in ISO/IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media as well as program metadata or other metadata, in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (a.k.a. packetized elementary stream) within the TS. Hence, a logical channel within an MPEG-2 TS may be considered to correspond to a specific PID value.

Available media file format standards include ISO base media file format (ISO/IEC 14496-12, which may be abbreviated ISOBMFF) and file format for NAL unit structured video (ISO/IEC 14496-15), which derives from the ISOBMFF.

A video codec consists of an encoder that transforms the input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form. A video encoder and/or a video decoder may also be separate from each other, i.e. need not form a codec. Typically the encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

Typical hybrid video encoders, for example many encoder implementations of ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

In temporal prediction, the sources of prediction are previously decoded pictures (a.k.a. reference pictures). In intra block copy (IBC; a.k.a. intra-block-copy prediction and current picture referencing), prediction is applied similarly to temporal prediction but the reference picture is the current picture and only previously decoded samples can be referred in the prediction process. Inter-layer or inter-view prediction may be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal prediction only, while in other cases inter prediction may refer collectively to temporal prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process as temporal prediction. Inter prediction or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

FIG. 4shows a block diagram of a general structure of a video encoder.FIG. 4presents an encoder for two layers, but it would be appreciated that presented encoder could be similarly extended to encode more than two layers.FIG. 4illustrates a video encoder comprising a first encoder section500for a base layer and a second encoder section502for an enhancement layer. Each of the first encoder section500and the second encoder section502may comprise similar elements for encoding incoming pictures. The encoder sections500,502may comprise a pixel predictor302,402, prediction error encoder303,403and prediction error decoder304,404.FIG. 4also shows an embodiment of the pixel predictor302,402as comprising an inter-predictor306,406(Pinter), an intra-predictor308,408(Pintra), a mode selector310,410, a filter316,416(F), and a reference frame memory318,418(RFM). The pixel predictor302of the first encoder section500receives 300 base layer images (I0,n) of a video stream to be encoded at both the inter-predictor306(which determines the difference between the image and a motion compensated reference frame318) and the intra-predictor308(which determines a prediction for an image block based only on the already processed parts of the current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector310. The intra-predictor308may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector310. The mode selector310also receives a copy of the base layer picture300.

Correspondingly, the pixel predictor402of the second encoder section502receives 400 enhancement layer images (I1,n) of a video stream to be encoded at both the inter-predictor406(which determines the difference between the image and a motion compensated reference frame418) and the intra-predictor408(which determines a prediction for an image block based only on the already processed parts of the current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector410. The intra-predictor408may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector410. The mode selector410also receives a copy of the enhancement layer picture400.

Depending on which encoding mode is selected to encode the current block, the output of the inter-predictor306,406or the output of one of the optional intra-predictor modes or the output of a surface encoder within the mode selector is passed to the output of the mode selector310,410. The output of the mode selector is passed to a first summing device321,421. The first summing device may subtract the output of the pixel predictor302,402from the base layer picture300/enhancement layer picture400to produce a first prediction error signal320,420(Do) which is input to the prediction error encoder303,403.

The pixel predictor302,402further receives from a preliminary reconstructor339,439the combination of the prediction representation of the image block312,412(P′n) and the output338,438(D′n) of the prediction error decoder304,404. The preliminary reconstructed image314,414(I′n) may be passed to the intra-predictor308,408and to the filter316,416. The filter316,416receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image340,440(R′n) which may be saved in a reference frame memory318,418. The reference frame memory318may be connected to the inter-predictor306to be used as the reference image against which a future base layer picture300is compared in inter-prediction operations. Subject to the base layer being selected and indicated to be the source for inter-layer sample prediction and/or inter-layer motion information prediction of the enhancement layer according to some embodiments, the reference frame memory318may also be connected to the inter-predictor406to be used as the reference image against which a future enhancement layer picture400is compared in inter-prediction operations. Moreover, the reference frame memory418may be connected to the inter-predictor406to be used as the reference image against which a future enhancement layer picture400is compared in inter-prediction operations.

Filtering parameters from the filter316of the first encoder section500may be provided to the second encoder section502subject to the base layer being selected and indicated to be the source for predicting the filtering parameters of the enhancement layer according to some embodiments.

The prediction error encoder303,403comprises a transform unit342,442(T) and a quantizer344,444(Q). The transform unit342,442transforms the first prediction error signal320,420to a transform domain. The transform is, for example, the DCT transform. The quantizer344,444quantizes the transform domain signal, e.g. the DCT coefficients, to form quantized coefficients.

The prediction error decoder304,404receives the output from the prediction error encoder303,403and performs the opposite processes of the prediction error encoder303,403to produce a decoded prediction error signal338,438which, when combined with the prediction representation of the image block312,412at the second summing device339,439, produces the preliminary reconstructed image314,414. The prediction error decoder304,404may be considered to comprise a dequantizer346,446(Q−1), which dequantizes the quantized coefficient values, e.g. DCT coefficients, to reconstruct the transform signal and an inverse transformation unit348,448(T−1), which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit348,448contains reconstructed block(s). The prediction error decoder may also comprise a block filter which may filter the reconstructed block(s) according to further decoded information and filter parameters.

The entropy encoder330,430(E) receives the output of the prediction error encoder303,403and may perform a suitable entropy encoding/variable length encoding on the signal to provide error detection and correction capability. The outputs of the entropy encoders330,430may be inserted into a bitstream e.g. by a multiplexer508(M).

Fundamentals of neural networks. A neural network (NN) is a computation graph consisting of several layers of computation. Each layer consists of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

Two widely used architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop: each layer takes input from one or more of the layers before and provides its output as the input for one or more of the subsequent layers. Also, units inside a certain layer take input from units in one or more of preceding layers, and provide output to one or more of the following layers.

Initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc. In recurrent neural nets, there is a feedback loop, so that the network becomes stateful, i.e., it is able to memorize information or a state.

Neural networks are being utilized in an ever-increasing number of applications for many different types of device, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

An important property of neural nets (and other machine learning tools) is that they are able to learn properties from input data, either in supervised way or in unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output's error, also referred to as the loss. Examples of losses are mean squared error, cross-entropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network's output, i.e., to gradually decrease the loss.

In this description of the provided examples, the terms “model”, “neural network”, “neural net” and “network” are used interchangeably, and also the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

Training a neural network is an optimization process, but the final goal is different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following items.

First, errors on the training set and on the validation set are monitored during the training process to understand if the network is learning at all—in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.

Second, errors on the training set and on the validation set are monitored during the training process to understand if the network is learning to generalize—in this case, also the validation set error needs to decrease and to be not too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized the training set's properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

Lately, neural networks have been used for compressing and de-compressing data such as images. The most widely used architecture for such task is the auto-encoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder (herein referred to simply as encoder and decoder, even though the examples described herein refer to algorithms which are learned from data instead of being tuned by hand). The encoder takes as input an image and produces a code which requires less bits than the input image. This code may have been obtained by additional steps after the encoder, such as a quantization process and lossless encoding. The decoder takes in this code and reconstructs the image which was input to the encoder. There may be additional steps being performed before the decoder neural network, such as lossless decoding and dequantization.

Such an encoder and decoder are usually trained to minimize a combination of bitrate and distortion, where the distortion is usually Mean Squared Error (MSE), PSNR, SSIM, or similar metrics. These distortion metrics are meant to be inversely proportional to the human visual perception quality. Examples of training loss used to minimize or reduce the bitrate are: an L1norm computed on the output of the encoder network or on the output of the quantization process; L1(x)/L2(x) where L1( ) and L2( ) are the L1norm and L2norm, respectively, and x is the output of the encoder network or the output of the quantization process; or an estimate or approximation of entropy of the output of the encoder network or the output of the quantization process.

The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in the spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and/or storing it as a prediction reference for the forthcoming frames in the video sequence.

In typical video codecs the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and/or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of the previously coded/decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and/or or co-located blocks in the temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding/decoding mechanism, often called merging/merge mode, where all the motion field information, which includes a motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification/correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and/or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidates filled with motion field information of available adjacent/co-located blocks.

In typical video codecs the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and the transform can in many cases help reduce this correlation and provide more efficient coding.

Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor λ to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area, where the cost function is of the form C=D+λR, where C is the Lagrangian cost to be minimized, D is the image distortion (e.g. Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

Background information on Video Coding for Machines (VCM). Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, i.e. consuming/watching the decoded image. Recently, with the advent of machine learning, especially deep learning, there is a rising number of machines (i.e., autonomous agents) that analyze data independently from humans and that may even take decisions based on the analysis results without human intervention. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. Example use cases and applications are self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart TV and smart advertisement, person re-identification, smart traffic monitoring, drones, etc. This fact may raise the issue that, when decoded data is consumed by machines, the aim should potentially be for a different quality metric—other than human perceptual quality—when considering media compression in inter-machine communications. Also, dedicated algorithms for compressing and decompressing data for machine consumption are likely to be different than those for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for machine consumption is referred to here as Video Coding for Machines.

It is likely that the receiver-side device has multiple “machines” or neural networks (NNs). These multiple machines may be used in a certain combination which is for example determined by an orchestrator sub-system. The multiple machines may be used for example in succession, based on the output of the previously used machine, and/or in parallel. For example, a video which was compressed and then decompressed may be analyzed by one machine (NN) for detecting pedestrians, by another machine (another NN) for detecting cars, and by another machine (another NN) for estimating the depth of all the pixels in the frames.

With respect to the examples described herein, machine and neural network are referred to interchangeably, and mean any process or algorithm (learned or not from data) which analyzes or processes data for a certain task. Throughout this description, other assumptions made regarding the machines considered by the examples described herein may be specified in further details.

Also, as used herein, the term “receiver-side” or “decoder-side” refers to the physical or abstract entity or device which contains one or more machines, and runs these one or more machines on some encoded and eventually decoded video representation which is encoded by another physical or abstract entity or device, the “encoder-side device”. In some cases, the two devices (encoder-side device and decoder-side device) may be parts of a single device or abstract entity.

The encoded video data may be stored into a memory device, for example as a file. The stored file may later be provided to another device. Alternatively, the encoded video data may be streamed from one device to another.

One of the possible approaches to perform video encoding and decoding for machines is to use a conventional codec, such as the Versatile Video Coding standard (also known as VVC and H.266), to encode the input image or video into a low bitrate code, and to use neural networks to encode and decode additional data which aids the task neural networks. In general, the conventional codec may even be a learned and fixed model.

FIG. 5is a block diagram5001depicting use of neural networks to encode and decode residual video for use by at least one task NN.FIG. 6is a block diagram5002illustrating the case where different neural auto-encoders are used to encode and decode the residual video for different task-NNs.

The low bitrate code (bitstream) may be achieved for example by first down-sampling504the input data501to a lower resolution, before the encoding process, and then up-sampling (516and546) the output of the decoder (510and514). An additional or alternative way to achieve a low bitrate is to use a high quantization parameter QP.

As the low bitrate code may not be sufficient for achieving high performance of task-NNs552/586, a residual524is computed520based on the original data501(ground-truth data) and the output518of the conventional decoder510, eventually after, being up-sampled516. For example, the residual524may be computed520by subtracting the original data501from the output518of the conventional decoder510. The residual524is then encoded and decoded526/566by one or more neural networks—typically one neural network for each task. Each of these NNs may be an auto-encoder, formed by an encoder neural network528/568and a decoder neural network540/580. Between the neural encoder528/568and decoder540/580there may be lossy and/or lossless compression steps, such as quantization530/570and entropy coding. The entropy coding may include entropy encoding534/574and entropy decoding538/578. As shown collectively byFIG. 5andFIG. 6, the entropy encoding534/574generates encoded residual video536/576which is then decoded by entropy decoding538/578. Quantization530/570may be for example one of the following: uniform scalar quantization, non-uniform scalar quantization, codebook-based quantization. Entropy coding may be for example arithmetic coding. Dequantization532/572includes dequantizing the output of entropy decoding538/578.

At decoder side, the decoded residual signal542is then combined with the data544decoded by the conventional decoder514. Combination may be for example a sum548/582. The output550/584of the combination may be processed by a post-processor, such as another neural network552/586.

The neural networks in this approach may be trained to minimize a certain cost function. This function may consist of one or more distortion functions and of one or more compression loss functions.

FIG. 5illustrates one possible block diagram5001of this approach, where a single neural auto-encoder526is used for encoding528and decoding540the feature residual524for one or more task-NNs552.

FIG. 5illustrates an approach where a conventional low-bitrate bitstream is achieved by downsampling504the input video501before encoding506. It needs to be understood thatFIG. 5similarly applies to other additional and alternative approaches for achieving a low-bitrate bitstream as described above.

FIG. 5(andFIG. 6) presents a video encoder block509and a video decoder block510at the encoder side as separate blocks. Many times a video encoder block509reconstructs the decoded video as a part of the encoding process506and thus a separate video decoding block510might not be needed. As shown, the video encoder block509generates an encoded signal512(e.g., video) as input to the video decoder510/514.

FIG. 6is a block diagram5002illustrating the case where different neural auto-encoders (namely auto-encoder526and auto-encoder566) are used to encode and decode the video residual524for different task-NNs (namely Task-NN1552and Task-NN2586).

However, this approach does not optimize the rate-distortion for the case where the task-NNs552/586at the decoder-side take features as inputs (instead of video), because it is designed to enhance decoded video550/584instead of decoded features.

InFIG. 5andFIG. 6, video decoder510, up-sampling516, and decoded video518are shown as being different entities, respectively, from video decoder514, up-sampling546, and decoded video544. In some examples, the entities may be the same (e.g., the same data or copy of the same data, or the same module or copy of the same module).

Research in the general domain of video coding and neural networks include international application number PCT/FI2019/050674 entitled “Compression for Machines”, EP application no. 19198496.2-1280 (published on Apr. 8, 2020 as 3633990) entitled “Rate-distortion Optimized Video/Image Coding Tuned for Machines”, U.S. provisional application No. 62/909,475 entitled “Joint Video Encoding and Neural Network Update for Machine-Targeted Content”, and FI national application 20205026 entitled “Supporting Dynamic Switching Between Tasks for Machine Targeted Video Coding”.

Described herein is an encoder and a decoder targeting video coding for machines, which is based on using a conventional codec and a learned codec, where the learned codec is optimized for aiding the reconstruction of features.FIG. 7is an example block diagram7001for implementing an encoder and a decoder targeting video coding for machines, based on the examples described herein. An encoder side701and a decoder side703according to an embodiment are illustrated inFIG. 7and described herein.

Encoder side701. A conventional video encoder709is used to encode video data701at a relatively low bitrate.FIG. 7presents a video encoder block709and a video decoder block710at the encoder side701as separate blocks. It needs to be understood that many times a video encoder block709reconstructs the decoded video718as a part of the encoding process and thus a separate video decoding block710might not be needed.

The low bitrate bitstream may be achieved for example by first down-sampling the input data to a lower resolution, before the encoding process, and then up-sampling the output of the video decoder (also at the decoder side703). The downsampling may be considered to be a part of the video encoder709or a pre-processing block for the video encoder709, and the upsampling may be considered to be a part of the video decoder710or a post-processing block for the video decoder710. An additional or alternative way to achieve a low bitrate is to use a high quantization parameter QP. An additional or alternative way to achieve a low bitrate is to tune the lambda parameter in the rate-distortion optimized mode selection to favor low bitrate over high quality.

Features713are extracted by a NN711applied on the original video data701. Features717are extracted by a NN715applied on the data718decoded by the conventional decoder710. A residual of features724is computed in the “compute difference” block720. Then, this feature residual724is encoded by a neural network728, such as an encoder part of an auto-encoder and additional compression steps. The “compute difference” block720computes the difference between the features717extracted715from the decoded video718and the features713extracted711from the original video701. The feature extraction711/715may be performed by a neural network (e.g., FX-NN1711and FX-NN2715).

Decoder side703. A conventional video bitstream712is decoded with a conventional video decoder714. The decoded video744may be upsampled as discussed in the encoder side701description.

Features are extracted by a NN745applied on the data744decoded by the conventional decoder714. The encoded residual features736are decoded by a neural network740(such as the decoder part of an auto-encoder and additional decompression steps).

The decoded residual features742are used to enhance the features747extracted745from the video744decoded by the conventional decoder714. In the “Compute sum” block748the decoded residual features742are combined with the features747extracted745from the data744decoded by the conventional decoder714to derive enhanced decoded features750.

A task NN752may be used to process or analyze enhanced decoded features750. The task NN752may be regarded as a part of the decoder side703or may be another entity, separate from the decoder side703. Further details of both the encoder side701and decoder side703are provided herein.

The feature-extraction neural network (FX-NN) at the encoder side701(namely FX-NN2715) and decoder side703(namely FX-NN2745) may be the same or may be different. Similarly, FX-NN1711and FX-NN2715may be the same or may be different. Similarly, FX-NN1711and FX-NN2745may be the same or may be different. In a typical embodiment, FX-NN1711, FX-NN2715and FX-NN2745may be the same. In one embodiment, one or more of the task-NNs752accept features as input. In one alternative embodiment, one or more of the task-NNs752accept visual data as input (such as images, video, etc.). In one alternative embodiment (shown byFIG. 9), for the case where the task-NNs752accept visual data as input, the enhanced data is generated by an additional neural network which combines the video744decoded by the conventional decoder714and the enhanced decoded features750.

In one alternative embodiment (shown byFIG. 10), for the case where the task-NNs752accept as input visual data, the enhanced data is generated by an additional neural network which combines the video744decoded by the conventional decoder714and decoded residual features742.

In one alternative embodiment, the encoder728and decoder740of residual features (724and736) are not neural networks, but may be any encoder and decoder, for example an image or video encoder and decoder. In this case, residual features encoder728may include a conversion from residual features724to feature map images in order to make them more suitable to be encoded by a conventional encoder such as H.266 based encoder. Residual features decoder740may include a conversion from decoded feature map images to decoded residual features742.

In the examples provided and described herein, the goal is to obtain a codec which targets the compression and decompression of data which is consumed by machines. In some embodiments it is possible that the decompressed data may also be consumed by humans, either at the same time or at different times with respect to when the machines consume the decompressed data. However, the examples described herein focus on describing the compression and decompression of data for machines.

In the case where some components of the proposed encoder and decoder are optimized at the development stage with respect to the task-NNs performance (such as when there are neural networks in the encoder and/or in the decoder), it is assumed that at least some of the task-NNs (machines) are models, such as neural networks, for which it is possible to compute a distortion that can be used to optimize some of the components of the encoder and/or decoder. In case there are neural networks in the encoder and/or decoder, the distortion may be a training signal for training neural networks in the encoder and/or decoder. The training signal may comprise the gradients of the output of one or more task-NN with respect to their input. For example, if the task-NNs are parametric models, gradients of their output may be computed first with respect to their internal parameters and then with respect to their input, by using the chain rule for differentiation in mathematics. In the case of neural networks, backpropagation may be used to obtain the gradients of the output of a NN with respect to its input.

The task-NNs that may be available during the development stage are representative of the task-NNs which may be used at inference time, i.e., when the codec may be deployed and used for compressing and decompressing data.

The task-NNs available during the development stage may have been previously trained. The data in the domain suitable to be input to the task-NNs available during the development stage may be available during the development stage. In some cases, this data may not be annotated, i.e., may not contain ground-truth labels.

The examples provided and described herein are not restricted to any specific type of data. However, for the sake of simplicity video data is considered for illustration purposes. Although, other example types of data that are relevant to the examples described herein include images, audio, speech, and text.

Main Embodiments. Described herein is an encoder and a decoder which encode the input video data into a base layer in the video domain and an enhancement layer in the feature domain. The terms base layer and enhancement layer are conceptual. They may but need not correspond to scalability layers of a multi-layer video codec. Options for arranging the base layer and enhancement layer signal are described subsequently.

FIG. 8is another example block diagram7002for implementing an encoder and a decoder targeting video coding for machines, based on the examples described herein.

The base layer may be obtained by using a conventional encoder706such as one which is compatible with the H.266 standard. The base layer may be encoded to low bitrates, for example by using a high quantization parameter (QP) or by down-sampling704the input video701before the encoding process. If down-sampling704is applied, the video712decoded by the conventional decoder710may need to be up-sampled716to its original resolution.

In an alternative embodiment, the base layer may be obtained by using a non-conventional encoder, such as a learned model. This model may be a neural network encoder. The decoding of the base layer may also be performed by a learned model such as a neural network decoder.

At the decoder side, the video712decoded (and eventually up-sampled746) by the conventional decoder714is input744to a feature-extraction neural network (FX-NN) (such as FX-NN2745), which extracts features. These may be referred to as base layer reconstructed features, or base features747for short. However, these base features747may not be sufficient for achieving a satisfactory performance of task-NNs752.

In order to enhance the base features747, residual features724are encoded in the following way. First, a feature residual signal724is computed720from the base features717and the original features713. The original features713are features extracted (such as by FX-NN1711) from the original (ground-truth) video701. For example, a subtraction may be computed720between the two tensors. The computed feature residual724is then encoded 728 and decoded740by one or more auto-encoders based codecs726.

In one embodiment, there may be a single auto-encoder based codec (such as codec726) for all task-NNs. In another embodiment, different auto-encoder based codecs for different task-NNs may be used. In another embodiment, multiple auto-encoder based codecs are used, where some of these codecs may be used for more than one task-NN (similar toFIG. 6).

An auto-encoder based codec726may comprise a neural network functioning as encoder728(for example, reducing the entropy of its input), a quantization step730, an entropy encoding step734, an entropy decoding step738, a dequantization step732, and a neural network functioning as decoder740. As shown by the example ofFIG. 8, the entropy decoding738decodes the encoded residual features736generated by the entropy encoding734.

Other architectures may be used for encoding and decoding the feature residual724. In one alternative embodiment, the feature residual724for one or more task-NNs752is encoded and decoded726by other types of algorithms than neural networks, such as other learned models or non-learned algorithms. Residual features encoder728may include a conversion from residual features to feature map images. Residual features decoder740may include a conversion from decoded feature map images to decoded residual features.

The decoded feature residual742is combined with the decoded base features747, for example by summation748. The resulting features may be referred to as enhanced features750. The enhanced features750are input to one or more task-NNs752. Here, these task-NNs752were trained to accept features750as input data, as opposed to other task-NNs which were trained to accept video or images as input data.FIG. 8as described illustrates one of the main ideas.

In practice, video encoders may reconstruct the decoded video as a “by-product” of the encoding, i.e. no separate video decoder block (such as video decoder710) in the encoder side is needed.

In one embodiment, the FX-NN1711(which extracts features from the original video701) and FX-NN2, such as FX-NN2715and FX-NN2745(which extracts features from the video decoded—including decoded video718and/or decoded video744—by the base layer codec), are different neural networks, in either the weights or the architecture, or both. However, in case the feature residual724is computed as a simple subtraction, the two FX-NNs (including FX-NN1711and FX-NN2715and/or FX-NN2745) may need to output tensors of same shape. Alternatively, post-processing is applied to make sure that the tensors' dimensions match.

In another embodiment, FX-NN1711and FX-NN2(such as FX-NN2715and/or FX-NN2745) are the same in either the weights or the architecture, or both.

Furthermore, each of the feature extraction NNs (including FX-NN1711, FX-NN2715, and/or FX-NN2745) may belong to one of the following categories:Pre-trained and “frozen” (i.e., not anymore modified after pre-training) when training other NNs in the encoder and/or decoder.Pre-trained and fine-tuned when training other NNs in the encoder and/or decoder.Trained from scratch (e.g., from random initialization of the weights) together with other NNs in the encoder and/or decoder.
The FX-NNs (including FX-NN1711, FX-NN2715, and/or FX-NN2745) need not be neural networks, but can be other types of feature extractors.

The training of the neural networks in the feature residual encoder728and decoder740may be performed by first computing a task loss for one or more task-NNs752, and a compression loss on the output of the encoder. These losses may be computed by using a sufficiently big dataset, which is representative of the data that may be used at the deployment stage. For supervised tasks, the loss computation may need the availability of ground-truth labels. The dataset may contain such ground-truth labels for one or more task-NNs752. For those supervised tasks for which there's no availability of labels, it may be possible to obtain non-ground-truth labels (sometimes referred to as soft labels) by running the task-NNs752on the original data (i.e., the data which is input to the encoder), using the obtained output of the task-NNs752as the labels for computing the loss of the task-NNs752when the input data is the enhanced decoded data750, and using this loss for training NNs in the encoder728and/or decoder740. For a task-NN performing a classification task, an example of loss of the task-NN is a cross-entropy loss. Training of the encoder728and/or decoder740may be performed by differentiating the task losses (i.e., losses of the task-NNs) and the compression losses with respect to the parameters of the encoder728and/or decoder740, thus obtaining gradients of the loss with respect to those parameters, and then updating the parameters of the encoder and/or decoder based on those gradients, by using a neural network optimization routine such as Stochastic Gradient Descent or Adam. In an additional or alternative embodiment, alternatively or in addition to the task losses, a feature-reconstruction error may be used as a training loss, such as the Mean Squared Error between the enhanced decoded features750and the original features713.

The examples described herein are not limited to any particular method for “Compute difference”720or “Compute sum”748. For example, “Compute difference”720may derive a component-wise scalar difference of the two input tensors, and respectively “Compute sum”748may derive a component-wise scalar sum of the two input tensors. Another example for “Compute sum”748is a neural network which may be trained together with the encoder and decoder neural networks (respectively728and740) for the feature-residual. Another example for “Compute difference”720is a neural network which may be trained together with the encoder and decoder neural networks (respectively728and740) for the feature-residual.

Alternative embodiment: enhancing the decoded video using enhanced features.FIG. 9is an example block diagram7003for implementing an embodiment that includes enhancing790the decoded video744using enhanced features750. In this alternative embodiment, the task-NNs752are assumed to accept image or video as input. The video744decoded by the base layer decoder714is enhanced790based on the enhanced features750. The enhanced features750are obtained in a similar way as described in the main embodiment of the examples described herein.

The enhancement790of the decoded video744may be performed for example by using an additional neural network, trained together with the other NNs in the encoder and decoder.FIG. 9is an illustration of this embodiment.

Alternative embodiment: enhancing the decoded video using decoded residual features.FIG. 10is an example block diagram7004for implementing an embodiment that includes enhancing791the decoded video744using decoded residual features742. In this alternative embodiment, the task-NNs752are assumed to accept image or video as input. The video744decoded by the base layer decoder714is enhanced790based on the decoded residual features742. The decoded residual features742are obtained in a similar way as described in the main embodiment of the provided examples.

The enhancement791of the decoded video744may be performed for example by using an additional neural network, trained together with the other NNs in the encoder and decoder.FIG. 10provides an illustration of this embodiment.

InFIG. 7,FIG. 8,FIG. 9, andFIG. 10, video decoder710, and decoded video718are shown as being different entities, respectively, from video decoder714and decoded video744. In some examples, the entities may be the same (e.g., the same data or copy of the same data, or the same module or copy of the same module). Furthermore, inFIG. 7,FIG. 8, andFIG. 9, feature extraction (FX-NN2)715and features717from decoded video718are shown as being different entities, respectively, from feature extraction (FX-NN2)745and features747from decoded video744. In some examples, the entities may be the same (e.g., the same data or copy of the same data, or the same module or copy of the same module). Furthermore, inFIG. 8,FIG. 9, andFIG. 10, up-sampling716is shown as being a different entity from up-sampling746. In some examples, up-sampling716and up-sampling746is the same entity.

Additional embodiment: enhance only a subset of features. In one alternative embodiment, only a subset of features extracted by FX-NN2715and/or FX-NN2745and/or FX-NN1711are selected for residual computation. Selection of such a subset may be done via selection of the most important features. One method for determining the most important features may consist of determining which features have higher average absolute value (L1norm) in the original features713and/or in the base features717, at the encoder side. For example, only two out of 128 feature maps may be determined to be important, and therefore the feature residual may be computed and encoded and decoded only for those two feature maps. Another method for determining the most important feature maps may consist of computing a first set of residual features724and then selecting only the residual features with average absolute value (L1norm). Other suitable methods may be used for determining the most important features. The encoder may signal to the decoder, either in-band or out-of-band with respect to the bitstream of the feature-residual, identifiers of the selected features for which feature-residual is encoded. Examples of such identifiers may be indexes of the features, or other unique identifiers. At decoder side, these identifiers are used to determine which decoded base feature747need to be enhanced.

Additional embodiments: Quantization aspects. In an additional embodiment, a quantization operation is applied to the residual features724after difference computation720.

In another embodiment, features713extracted711from the original video701and features717extracted715from decoded video718are initially quantized before the difference computation720. The level of quantization may be determined by the accuracy and performance of the Task-NN752and/or available bandwidth. Quantization may be linear, piece-wise linear or even a learned function based on the Task-NN752accuracy.

In another embodiment, multiple task752accuracies may be combined and multiple different quantization schemes may be applied to different feature maps for residual calculation.

Additional embodiment: temporal features prediction. In another embodiment, features from frame at time t are extracted and then a difference between these features and features from previous frame at time t-1 is computed. This is done for both the original video701and the decoded video718/744, thus obtaining two feature-differences. Then, a difference between these two feature-differences is computed. This difference of feature-differences is then encoded, instead of the difference of features. Alternatively, the coding of residual726may perform temporal prediction of the residual features.

Options for arranging the encoded video and encoded residual features. Embodiments similarly apply to an encoder side701that encodes a signal according to an option described below, and to a decoder side703that decodes a signal according to an option described below.

In an embodiment, the encoded video712complies with a conventional video bitstream format, such as H.266, and the encoded residual features724are present in the same video bitstream in a manner that a conventional video decoder710/714omits the encoded residual features736. Such manners may include but are not limited to one or more of the following:The encoded residual features736are present in Supplemental Enhancement Information (SEI) messages in the video bitstream.The encoded residual features736are present in Network Abstraction Layer (NAL) units that are omitted by a conventional video decoder710/714. For example, NAL units that have been left “unspecified” in a coding standard may be used or NAL units that were previously reserved for future extensions (when a conventional video bitstream format was specified) may be used.

In an embodiment, a multi-layer video bitstream format (such as h.266 or scalable HEVC) is used, where the format provides capability of separating the data into multiple scalability layers. The encoded video712resides in a first scalability layer that is independent of any other layers. The encoded residual features736are present in a second scalability layer that depends on the first layer. The bitstream may indicate the type of the second scalability layer and/or which residual features decoder740and/or feature extraction may be used for decoding the second scalability layer.

In an embodiment, a video bitstream comprising the encoded video712is present in a first track of a container file and encoded residual features736are present in a second track of the container file. The container file provides means for aligning or synchronizing samples of the first track and the second track so that the decoder side703is able to combine the base features747and the decoded residual features742that are time-aligned.

FIG. 11is an example apparatus1100, which may be implemented in hardware, configured to implement feature-domain residual for video coding, based on the examples described herein. The apparatus1100comprises a processor1102, at least one non-transitory memory1104including computer program code1105, wherein the at least one memory1104and the computer program code1105are configured to, with the at least one processor1102, cause the apparatus to implement feature-domain residual for video coding1106, based on the examples described herein. The apparatus1100optionally includes a display1108that may be used to display content during task/machine/NN processing or rendering. The apparatus1100optionally includes one or more network (NW) interfaces (I/F(s))1110. The NW I/F(s)1110may be wired and/or wireless and communicate over the Internet/other network(s) via any communication technique. The NW I/F(s)1110may comprise one or more transmitters and one or more receivers. The N/W I/F(s)1110may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder/decoder circuitry(ies) and one or more antennas. In some examples, the processor1102is configured to implement item1106without use of memory1104.

The memory1104may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The memory1104may comprise a database for storing data. Interface1112enables data communication between the various items of apparatus1100, as shown inFIG. 11. Interface1112may be one or more buses, or interface1112may be one or more software interfaces configured to pass data between the items of apparatus1100. For example, the interface1112may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The apparatus1100need not comprise each of the features mentioned, or may comprise other features as well. The apparatus1100need not comprise each of the features mentioned, or may comprise other features as well. The apparatus1100may be an embodiment of apparatuses shown inFIG. 1,FIG. 2,FIG. 3,FIG. 4,FIG. 5,FIG. 6,FIG. 7,FIG. 8,FIG. 9, orFIG. 10.

FIG. 12is an example method1200to implement feature-domain residual for video coding for machines, based on the examples described herein. At1202, the method includes encoding original data to generate encoded data with a bitrate lower than that of the original data, and decoded data. At1204, the method includes encoding the original data, using in part a learning method, to generate encoded residual features and decoded residual features. At1206, the method includes generating enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data. Method1200may be implemented with an encoder.

FIG. 13is another example method1300to implement feature-domain residual for video coding for machines, based on the examples described herein. At1302, the method includes decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extracting features from the decoded data. At1304, the method includes decoding encoded residual features to generate decoded residual features. At1306, the method includes generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data. Method1300may be implemented with a decoder.

An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: encode original data with a first codec to generate encoded data with a bitrate lower than that of the original data, and decoded data; encode the original data with at least one second learned codec to generate encoded residual features and decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data generated with the first codec.

The apparatus may further include wherein at least one machine processes or analyzes the decoded data using the enhanced decoded features.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: generate enhanced decoded video resulting from combining the decoded data with the enhanced decoded features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: rather than generate the enhanced decoded features, generate enhanced decoded video resulting from combining the decoded data with the decoded residual features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

The apparatus may further include wherein the residual features are encoded using at least one neural network, and the residual features are decoded using at least one neural network.

The apparatus may further include wherein the features extracted from the decoded data generated with the first codec are extracted using a neural network.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: extract features from the original data; extract features from the decoded data; and generate the residual features, prior to being encoded, as a result of computing a difference between the features extracted from the decoded data and the features extracted from the original data.

The apparatus may further include wherein the extracting of the features from the original data is performed using a neural network; and wherein the extracting of the features from the decoded data is performed using a neural network.

The apparatus may further include wherein the enhanced decoded video is generated using a neural network.

The apparatus may further include wherein the enhanced decoded video is generated using a neural network.

The apparatus may further include wherein the residual features are encoded using an image of a video encoder, the encoding of the residual features comprising converting the residual features to feature map images; and wherein the residual features are decoded using an image of a video decoder, the decoding of the residual features comprising converting decoded feature map images to the decoded residual features.

The apparatus may further include wherein the original data is video.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: downsample the original data prior to encoding the original data with the first codec; and upsample the decoded data generated with the first codec.

The apparatus may further include wherein the encoding of the original data with the at least one second learned codec comprises: quantization and entropy encoding to generate the encoded residual features; and entropy decoding and dequantization to generate the decoded residual features.

The apparatus may further include wherein the enhanced decoded features are generated as a result of computing a sum of the decoded residual features and the features extracted from the decoded data generated with the first codec.

The apparatus may further include wherein the sum operation is replaced with an operation performed with a neural network.

The apparatus may further include wherein the difference operation is replaced with an operation performed with a neural network.

The apparatus may further include wherein at least two neural networks used in a process to generate the enhanced decoded features are trained together.

An example apparatus includes means for encoding original data with a first codec to generate encoded data with a bitrate lower than that of the original data, and decoded data; means for encoding the original data with at least one second learned codec to generate encoded residual features and decoded residual features; and means for generating enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data generated with the first codec.

The apparatus may further include wherein at least one machine processes or analyzes the decoded data using the enhanced decoded features.

The apparatus may further include: means for generating enhanced decoded video resulting from combining the decoded data with the enhanced decoded features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

The apparatus may further include wherein the enhanced decoded video is generated using a neural network.

The apparatus may further include means for, rather than generating the enhanced decoded features, generating enhanced decoded video resulting from combining the decoded data with the decoded residual features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

The apparatus may further include wherein the enhanced decoded video is generated using a neural network.

The apparatus may further include wherein the residual features are encoded using at least one neural network, and the residual features are decoded using at least one neural network.

The apparatus may further include wherein the features extracted from the decoded data generated with the first codec are extracted using a neural network.

The apparatus may further include means for extracting features from the original data; means for extracting features from the decoded data; and means for generating the residual features, prior to being encoded, as a result of computing a difference between the features extracted from the decoded data and the features extracted from the original data.

The apparatus may further include wherein the extracting of the features from the original data is performed using a neural network; and wherein the extracting of the features from the decoded data is performed using a neural network.

The apparatus may further include wherein the difference operation is replaced with an operation performed with a neural network.

The apparatus may further include wherein the residual features are encoded using an image of a video encoder, the encoding of the residual features comprising converting the residual features to feature map images; and wherein the residual features are decoded using an image of a video decoder, the decoding of the residual features comprising converting decoded feature map images to the decoded residual features.

The apparatus may further include wherein the original data is video.

The apparatus may further include means for downsampling the original data prior to encoding the original data with the first codec; and means for upsampling the decoded data generated with the first codec.

The apparatus may further include wherein the encoding of the original data with the at least one second learned codec comprises: quantization and entropy encoding to generate the encoded residual features; and entropy decoding and dequantization to generate the decoded residual features.

The apparatus may further include wherein the enhanced decoded features are generated as a result of computing a sum of the decoded residual features and the features extracted from the decoded data generated with the first codec.

The apparatus may further include wherein the sum operation is replaced with an operation performed with a neural network.

The apparatus may further include wherein at least two neural networks used in a process to generate the enhanced decoded features are trained together.

An example method includes encoding original data to generate encoded data with a bitrate lower than that of the original data, and decoded data; encoding the original data, using in part a learning method, to generate encoded residual features and decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data.

An example non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations may be provided, the operations comprising: encoding original data to generate encoded data with a bitrate lower than that of the original data, and decoded data; encoding the original data, using in part a learning method, to generate encoded residual features and decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data.

An example apparatus may include circuitry configured to encode original data with a first codec to generate encoded data with a bitrate lower than that of the original data, and decoded data; circuitry configured to encode the original data with at least one second learned codec to generate encoded residual features and decoded residual features; and circuitry configured to generate enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data generated with the first codec.

An example apparatus includes means for decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and means for extracting features from the decoded data; means for decoding encoded residual features to generate decoded residual features; and means for generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

The apparatus may further include means for processing or analyzing the enhanced decoded features using at least one task neural network.

The apparatus may further include means for generating enhanced decoded video as a result of combining the decoded data with the enhanced decoded features; wherein the combining of the decoded data with the enhanced decoded features to generate the enhanced decoded video is performed using a neural network; and means for processing or analyzing the enhanced decoded video using at least one task neural network.

The apparatus may further include means for generating enhanced decoded video as a result of combining the decoded data with the decoded residual features; wherein the combining of the decoded data with the decoded residual features to generate the enhanced decoded video is performed using a neural network; and means for processing or analyzing the enhanced decoded video using at least one task neural network.

The apparatus may further include wherein the features are extracted from the decoded data using a neural network; and the encoded residual features are decoded using a neural network.

The apparatus may further include wherein the combining of the decoded residual features with the features extracted from the decoded data to generate the enhanced decoded features is a summation of the decoded residual features and the features extracted from the decoded data.

The apparatus may further include wherein the encoded residual features are a difference between features extracted from the original data, and features extracted from preliminary decoded data or the features extracted from the decoded data.

The apparatus may further include wherein the decoded residual features are decoded using entropy decoding and dequantization.

The apparatus may further include wherein the decoded residual features are decoded using an image of a video decoder, the decoding of the residual features comprising converting decoded feature map images to the decoded residual features.

The apparatus may further include wherein the original data is video data.

An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extract features from the decoded data; decode encoded residual features to generate decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: process or analyze the enhanced decoded features using at least one task neural network.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: generate enhanced decoded video as a result of combining the decoded data with the enhanced decoded features; wherein the combining of the decoded data with the enhanced decoded features to generate the enhanced decoded video is performed using a neural network; and process or analyze the enhanced decoded video using at least one task neural network.

The apparatus may further include wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: generate enhanced decoded video as a result of combining the decoded data with the decoded residual features; wherein the combining of the decoded data with the decoded residual features to generate the enhanced decoded video is performed using a neural network; and process or analyze the enhanced decoded video using at least one task neural network.

The apparatus may further include wherein the features are extracted from the decoded data using a neural network; and the encoded residual features are decoded using a neural network.

The apparatus may further include wherein the combining of the decoded residual features with the features extracted from the decoded data to generate the enhanced decoded features is a summation of the decoded residual features and the features extracted from the decoded data.

The apparatus may further include wherein the encoded residual features are a difference between features extracted from the original data, and features extracted from preliminary decoded data or the features extracted from the decoded data.

The apparatus may further include wherein the decoded residual features are decoded using entropy decoding and dequantization.

The apparatus may further include wherein the decoded residual features are decoded using an image of a video decoder, the decoding of the residual features comprising converting decoded feature map images to the decoded residual features.

The apparatus may further include wherein the original data is video data.

An example method includes decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extracting features from the decoded data; decoding encoded residual features to generate decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

An example non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations is provided, the operations comprising: decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extracting features from the decoded data; decoding encoded residual features to generate decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

The following acronyms and abbreviations that may be found in the specification and/or the drawing figures are defined as follows:3GPP 3rd Generation Partnership Project4G fourth generation of broadband cellular network technology5G fifth generation cellular network technology802.x family of IEEE standards dealing with local area networks and metropolitan area networksa.k.a. also known asCDMA code-division multiple accessDCT discrete cosine transformDSP digital signal processorFDMA frequency division multiple accessFX-NN feature extraction neural networkGSM Global System for Mobile communicationsH.222.0 MPEG-2 Systems, standard for the generic coding of moving pictures and associated audio informationH.26x family of video coding standards in the domain of the ITU-THEVC high efficiency video coding, also known as H.265 and MPEG-H Part2IBC intra block copyIEC International Electrotechnical CommissionIEEE Institute of Electrical and Electronics EngineersI/F interfaceIMD integrated messaging deviceIMS instant messaging serviceIoT internet of thingsIP internet protocolISO International Organization for StandardizationISOBMFF ISO base media file formatITU International Telecommunication UnionITU-T ITU Telecommunication Standardization SectorLTE long-term evolutionMMS multimedia messaging serviceMPEG moving picture experts groupMPEG-2 H.222/H.262 as defined by the ITUMPEG-H MPEG for Heterogeneous EnvironmentsMSE mean squared errorNAL network abstraction layerNN neural networkN/W or NW networkPC personal computerPDA personal digital assistantPID packet identifierPLC power line communicationPSNR peak signal-to-noise ratioQP quantization parameterRFID radio frequency identificationRFM reference frame memorySEI supplemental enhancement informationSMS short messaging serviceSSIM structural similaritytask-NN task neural networkTCP-IP transmission control protocol-internet protocolTDMA time divisional multiple accessTS transport streamTV televisionUICC Universal Integrated Circuit CardUMTS Universal Mobile Telecommunications SystemUSB Universal Serial BusVCM video coding for machinesVVC versatile video codingWLAN wireless local area network