Patent Publication Number: US-2023154054-A1

Title: Decoder-side fine-tuning of neural networks for video coding for machines

Description:
SUPPORT STATEMENT 
     The project leading to this application has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 783162. The JU receives support from the European Union&#39;s Horizon 2020 research and innovation programme and Netherlands, Czech Republic, Finland, Spain, Italy. 
     The project leading to this application has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 876019. The JU receives support from the European Union&#39;s Horizon 2020 research and innovation programme and Germany, Netherlands, Austria, Romania, France, Sweden, Cyprus, Greece, Lithuania, Portugal, Italy, Finland, Turkey. 
     TECHNICAL FIELD 
     The examples and non-limiting embodiments relate generally to multimedia transport and neural networks, and more particularly, to method, apparatus, and computer program product for decoder-side fine-tuning of neural networks. 
     BACKGROUND 
     It is known to provide standardized formats for exchange of neural networks. 
     SUMMARY 
     An example apparatus includes at least one processor; and at least one non-transitory memory comprising computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to iteratively perform following until a stopping criterion is met: provide a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN); compute an output of the DSNN as a processed FDC; compute a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC; compute an update to the DSNN; and apply the computed update to the DSNN. 
     The example apparatus may further include, wherein a sample of FDC is associated with one or more samples of AGT. 
     The example apparatus may further include, wherein a sample of AGT is associated with one or more samples of FDC. 
     The example apparatus may further include, wherein to compute the update to the DSSN, the apparatus is further caused to use a backpropagation algorithm to compute gradients of the loss with respect to one or more parameters of the DSNN; apply an optimizer routine to compute the update. 
     The example apparatus may further include, wherein the optimizer routine comprises a stochastic gradient descent. 
     The example apparatus may further include, wherein the apparatus is further caused to encode one or more parts of a video in higher quality (HQ) content and remaining parts of the video in lower quality (LQ) content; and use the HQ content to obtain the AGT. 
     The example apparatus may further include, wherein the apparatus is further caused to obtain the FDC at a decoder side by having a decoder-side device lower a quality of the HQ content. 
     The example apparatus may further include, wherein to lower the quality of the HQ content, the apparatus is further caused to perform at least one of the following: encode the HQ content by using an encoding configuration that causes the decoded content to be of lower quality with respect to the HQ content and decode the encoded HQ content; downsample the HQ content to a lower resolution; or input the HQ content to a NN, and treat the output of the NN as the lower quality content, wherein the NN is trained to output a content that has similar quality as the LQ content. 
     The example apparatus may further include, wherein the apparatus is further caused to obtain the FDC at a decoder side by having an encoder encode the FDC as a low quality version of the HQ content, or as LQ content which is related or similar to the HQ content. 
     The example apparatus may further include, wherein the apparatus is further caused to encode the LQ and HQ contents; and compute AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content. 
     The example apparatus may further include, wherein the apparatus is further caused to encode the LQ and HQ version of an input content; and compute AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content. 
     The example apparatus may further include, wherein apparatus is further caused to extract a high resolution patch from the HQ content; and extract a low resolution patch from the LQ content. 
     The example apparatus may further include, wherein the apparatus is further caused to extract the high and low resolution patches by using a patch selection algorithm comprising an object detection and tracking algorithm, a correlation-based patch-match algorithm, or an instance segmentation algorithm. 
     The example apparatus may further include, wherein the apparatus causes an encoder to encode the AGT and to provide the AGT to the decoder via a bitstream. 
     The example apparatus may further include, wherein the bitstream comprises a bitstream representing an encoded content or an out-of-band bitstream. 
     The example apparatus may further include, wherein the apparatus further causes an encoder side to run a loss proxy neural network for one or more input frames and encode an output of a loss proxy neural network, representing the AGT, as metadata. 
     The example apparatus may further include, wherein the apparatus further causes the encoder to generate the FDC and include the FDC in the bitstream to finetune the DSNN. 
     The example apparatus may further include, wherein the apparatus is caused to store the finetuned DSNN in a buffer; cause the encoder to signal when the stored DSNN is to be used to process subsequent regions or frames or needs to be updated. 
     The example apparatus may further include, wherein the apparatus further causes the encoder to signal an ID or index of an optimal DSNN that is available in the buffer. 
     The example apparatus may further include, wherein the AGT represents an approximation of a desired or correct output of a loss proxy neural network (LPNN) when an input comprises an uncompressed or a high quality data. 
     The example apparatus may further include, wherein the LPNN comprises a neural network performing at least one of an analysis, a processing task (task-NN), or may be a pretrained feature extractor NN. 
     An example method includes providing a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN); computing an output of the DSNN as a processed FDC; computing a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC; computing an update to the DSNN; and applying the computed update to the DSNN. 
     The example method may further include, wherein a sample of FDC is associated with one or more samples of AGT. 
     The example method may further include, wherein a sample of AGT is associated with one or more samples of FDC. 
     The example method may further include, wherein computing the update to the DSSN comprises using a backpropagation algorithm to compute gradients of the loss with respect to one or more parameters of the DSNN; applying an optimizer routine to compute the update. 
     The example method may further include, wherein the optimizer routine includes a stochastic gradient descent. 
     The example method may further include encoding one or more parts of a video in higher quality (HQ) content and remaining parts of the video in lower quality (LQ) content; and using the HQ content to obtain the AGT. 
     The example method may further include obtaining the FDC at a decoder side by having a decoder-side device lowering a quality of the HQ content. 
     The example method may further include, wherein lowering the quality of the HQ content incudes encoding the HQ content by using an encoding configuration that causes the decoded content to be of lower quality with respect to the HQ content and decode the encoded HQ content; downsampling the HQ content to a lower resolution; or inputting the HQ content to a NN, and treat the output of the NN as the lower quality content, wherein the NN is trained to output a content that has similar quality as the LQ content. 
     The example method may further include obtaining the FDC at a decoder side by having an encoder encode the FDC as a low quality version of the HQ content, or as LQ content which is related or similar to the HQ content. 
     The example method may further include encoding the LQ and HQ contents; and computing AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content. 
     The example method may further include encoding the LQ and HQ version of an input content; and computing AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content. 
     The example method may further include extracting a high resolution patch from the HQ content; and extracting a low resolution patch from the LQ content. 
     The example method may further include extracting the high and low resolution patches by using a patch selection algorithm comprising an object detection and tracking algorithm, a correlation-based patch-match algorithm, or an instance segmentation algorithm. 
     The example method may further include causing an encoder to encode the AGT and to provide the AGT to the decoder via a bitstream. 
     The example method may further include, wherein the bitstream comprises a bitstream representing an encoded content or an out-of-band bitstream. 
     The example method may further include causing an encoder side to run a loss proxy neural network for one or more input frames and encode an output of a loss proxy neural network, representing the AGT, as metadata. 
     The example method may further include causing the encoder to generate the FDC and include the FDC in the bitstream to finetune the DSNN. 
     The example method may further include storing the finetuned DSNN in a buffer; causing the encoder to signal when the stored DSNN is to be used to process subsequent regions or frames or needs to be updated. 
     The example method may further include causing the encoder to signal an ID or index of an optimal DSNN that is available in the buffer. 
     The example method may further include, wherein the AGT represents an approximation of a desired or correct output of a loss proxy neural network (LPNN) when an input comprises an uncompressed or a high quality data. 
     The example method may further include, wherein the LPNN comprises a neural network performing at least one of an analysis, a processing task (task-NN), or may be a pretrained feature extractor NN. 
     An example computer readable medium includes program instructions for causing an apparatus to perform at least the following: provide a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN); compute an output of the DSNN as a processed FDC; compute a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC; compute an update to the DSNN; and apply the computed update to the DSNN. 
     The example computer readable medium may further include, wherein the apparatus is further caused to perform the methods as described in any of the previous paragraphs. 
     The example computer readable medium may further include, wherein the computer readable medium comprises a non-transitory computer readable medium. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein: 
         FIG.  1    shows schematically an electronic device employing embodiments of the examples described herein. 
         FIG.  2    shows schematically a user equipment suitable for employing embodiments of the examples described herein. 
         FIG.  3    further shows schematically electronic devices employing embodiments of the examples described herein connected using wireless and wired network connections. 
         FIG.  4    shows schematically a block diagram of an encoder on a general level. 
         FIG.  5    is a block diagram showing the interface between an encoder and a decoder in accordance with the examples described herein. 
         FIG.  6    illustrates a system configured to support streaming of media data from a source to a client device; 
         FIG.  7    is a block diagram of an apparatus that may be specifically configured in accordance with an example embodiment. 
         FIG.  8    illustrates examples of functioning of neural networks (NNs) as components of a traditional codec&#39;s pipeline, in accordance with an example embodiment. 
         FIG.  9    illustrates an example of modified video coding pipeline based on neural networks, in accordance with an example embodiment. 
         FIG.  10    is an example neural network-based end-to-end learned video coding system, in accordance with an example embodiment. 
         FIG.  11    illustrates a pipeline of video coding for machines (VCM), in accordance with an embodiment. 
         FIG.  12    illustrates an example of an end-to-end learned approach for the use case of video coding for machines, in accordance with an embodiment. 
         FIG.  13    illustrates an example of how the end-to-end learned system may be trained for the use case of video coding for machines, in accordance with an embodiment. 
         FIG.  14    illustrates an example codec architecture, in accordance with an embodiment. 
         FIG.  15    illustrates an example baseline system, in accordance with an embodiment. 
         FIG.  16    illustrates an example baseline system, in accordance with another embodiment. 
         FIG.  17    illustrates an example of implementation, in accordance with an embodiment. 
         FIG.  18    illustrates example implementation, in accordance with another embodiment. 
         FIG.  19    illustrates example implementation, in accordance with another embodiment. 
         FIG.  20    illustrates an example implementation in which the encoder encodes the approximated ground truth (AGT) and provides the AGT to the decoder, in accordance with an embodiment. 
         FIG.  21    is an example apparatus, which may be implemented in hardware, and is configured to implement mechanisms for training or finetuning a decoder side neural network, based on the examples described herein. 
         FIG.  22    illustrates an example method for training or finetuning decoder side neural network, in accordance with an embodiment. 
         FIG.  23    is a block diagram of one possible and non-limiting system in which the example embodiments may be practiced. 
     
    
    
     DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
     The following acronyms and abbreviations that may be found in the specification and/or the drawing figures are defined as follows:
     3GP 3GPP file format   3GPP 3rd Generation Partnership Project   3GPP TS 3GPP technical specification   4CC four character code   4G fourth generation of broadband cellular network   technology   5G fifth generation cellular network technology   5GC 5G core network   ACC accuracy   AGT approximated ground truth data   AI artificial intelligence   AIoT AI-enabled IoT   ALF adaptive loop filtering   a.k.a. also known as   AMF access and mobility management function   APS adaptation parameter set   AVC advanced video coding   bpp bits-per-pixel   CABAC context-adaptive binary arithmetic coding   CDMA code-division multiple access   CE core experiment   ctu coding tree unit   CU central unit   DASH dynamic adaptive streaming over HTTP   DCT discrete cosine transform   DSP digital signal processor   DSNN decoder-side NN   DU distributed unit   eNB (or eNodeB) evolved Node B (for example, an LTE base station)   EN-DC E-UTRA-NR dual connectivity   en-gNB or En-gNB node providing NR user plane and control plane protocol terminations towards the UE, and acting as secondary node in EN-DC   E-UTRA evolved universal terrestrial radio access, for example, the LTE radio access technology   FDMA frequency division multiple access   f(n) fixed-pattern bit string using n bits written (from left to right) with the left bit first.   F 1  or F 1 -C interface between CU and DU control interface   FDC finetuning-driving content   gNB (or gNodeB) base station for 5G/NR, for example, a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GC   GSM Global System for Mobile communications   H.222.0 MPEG-2 Systems is formally known as ISO/IEC 13818-1 and as ITU-T Rec. H.222.0   H.26x family of video coding standards in the domain of the ITU-T   HLS high level syntax   HQ high-quality   IBC intra block copy   ID identifier   IEC International Electrotechnical Commission   IEEE Institute of Electrical and Electronics Engineers   I/F interface   IMD integrated messaging device   IMS instant messaging service   IoT internet of things   IP internet protocol   IRAP intra random access point   ISO International Organization for Standardization   ISOBMFF ISO base media file format   ITU International Telecommunication Union   ITU-T ITU Telecommunication Standardization Sector   JPEG joint photographic experts group   LMCS luma mapping with chroma scaling   LPNN loss proxy NN   LQ low-quality   LTE long-term evolution   LZMA Lempel-Ziv-Markov chain compression   LZMA 2  simple container format that can include both uncompressed data and LZMA data   LZO Lempel-Ziv-Oberhumer compression   LZW Lempel-Ziv-Welch compression   MAC medium access control   mdat MediaDataBox   MME mobility management entity   MMS multimedia messaging service   moov MovieBox   MP4 file format for MPEG-4 Part  14  files   MPEG moving picture experts group   MPEG-2 H.222/H.262 as defined by the ITU   MPEG-4 audio and video coding standard for ISO/IEC 14496   MSB most significant bit   NAL network abstraction layer   NDU NN compressed data unit   ng or NG new generation   ng-eNB or NG-eNB new generation eNB   NN neural network   NNEF neural network exchange format   NNR neural network representation   NR new radio (5G radio)   N/W or NW network   ONNX Open Neural Network eXchange   PB protocol buffers   PC personal computer   PDA personal digital assistant   PDCP packet data convergence protocol   PHY physical layer   PID packet identifier   PLC power line communication   PNG portable network graphics   PSNR peak signal-to-noise ratio   RAM random access memory   RAN radio access network   RBSP raw byte sequence payload   RD loss rate distortion loss   RFC request for comments   RFID radio frequency identification   RLC radio link control   RRC radio resource control   RRH remote radio head   RU radio unit   Rx receiver   SDAP service data adaptation protocol   SGD Stochastic Gradient Descent   SGW serving gateway   SMF session management function   SMS short messaging service   SPS sequence parameter set   st(v) null-terminated string encoded as UTF-8 characters as specified in ISO/IEC 10646   SVC scalable video coding   S 1  interface between eNodeBs and the EPC   TCP-IP transmission control protocol-internet protocol   TDMA time divisional multiple access   trak TrackBox   TS transport stream   TUC technology under consideration   TV television   Tx transmitter   UE user equipment   ue(v) unsigned integer Exp-Golomb-coded syntax element with the left bit first   UICC Universal Integrated Circuit Card   UMTS Universal Mobile Telecommunications System   u(n) unsigned integer using n bits   UPF user plane function   URI uniform resource identifier   URL uniform resource locator   UTF-8 8-bit Unicode Transformation Format   VPS video parameter set   WLAN wireless local area network   X2 interconnecting interface between two eNodeBs in LTE network   Xn interface between two NG-RAN nodes   

     Some embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the invention are shown. Indeed, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms ‘data,’‘content,’ ‘information,’ and similar terms may be used interchangeably to refer to data capable of being transmitted, received and/or stored in accordance with embodiments of the present invention. Thus, use of any such terms should not be taken to limit the spirit and scope of embodiments of the present invention. 
     Additionally, as used herein, the term ‘circuitry’ refers to (a) hardware-only circuit implementations (e.g., implementations in analog circuitry and/or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and/or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even when the software or firmware is not physically present. This definition of ‘circuitry’ applies to all uses of this term herein, including in any claims. As a further example, as used herein, the term ‘circuitry’ also includes an implementation comprising one or more processors and/or portion(s) thereof and accompanying software and/or firmware. As another example, the term ‘circuitry’ as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and/or other computing device. 
     As defined herein, a ‘computer-readable storage medium,’ which refers to a non-transitory physical storage medium (e.g., volatile or non-volatile memory device), can be differentiated from a ‘computer-readable transmission medium,’ which refers to an electromagnetic signal. 
     A method, apparatus and computer program product are provided in accordance with example embodiments for implementing mechanisms for training or finetuning a decoder side neural network. 
     In an example, the following describes in detail suitable apparatus and possible mechanisms for training or finetuning a decoder side neural network. In this regard reference is first made to  FIG.  1    and  FIG.  2   , where  FIG.  1    shows an example block diagram of an apparatus  50 . The apparatus may be an Internet of Things (IoT) apparatus configured to perform various functions, for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec.  FIG.  2    shows a layout of an apparatus according to an example embodiment. The elements of  FIG.  1    and  FIG.  2    will be explained next. 
     The apparatus  50  may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or a lower power device. However, it would be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks. 
     The apparatus  50  may comprise a housing  30  for incorporating and protecting the device. The apparatus  50  may further comprise a display  32 , for example, in the form of a liquid crystal display, light emitting diode (LED) display, organic LED display, and the like. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display media or multimedia content, for example, an image or video. The apparatus  50  may further comprise a keypad  34 . In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display. 
     The apparatus may comprise a microphone  36  or any suitable audio input which may be a digital or analogue signal input. The apparatus  50  may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece  38 , speaker, or an analogue audio or digital audio output connection. The apparatus  50  may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera  42  capable of recording or capturing images and/or video. The apparatus  50  may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus  50  may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection. 
     The apparatus  50  may comprise a controller  56 , a processor or a processor circuitry for controlling the apparatus  50 . The controller  56  may be connected to a memory  58  which in embodiments of the examples described herein may store both data in the form of an image, audio data, video data, and/or may also store instructions for implementation on the controller  56 . The controller  56  may further be connected to codec circuitry  54  suitable for carrying out coding and/or decoding of audio, image, and/or video data or assisting in coding and/or decoding carried out by the controller. 
     The apparatus  50  may further comprise a card reader  48  and a smart card  46 , for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network. 
     The apparatus  50  may comprise radio interface circuitry  52  connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus  50  may further comprise an antenna  44  connected to the radio interface circuitry  52  for transmitting radio frequency signals generated at the radio interface circuitry  52  to other apparatus(es) and/or for receiving radio frequency signals from other apparatus(es). 
     The apparatus  50  may comprise a camera  42  capable of recording or detecting individual frames which are then passed to the codec  54  or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and/or storage. The apparatus  50  may also receive either wirelessly or by a wired connection the image for coding/decoding. The structural elements of apparatus  50  described above represent examples of means for performing a corresponding function. 
     With respect to  FIG.  3   , an example of a system within which embodiments of the examples described herein can be utilized is shown. The system  10  comprises multiple communication devices which can communicate through one or more networks. The system  10  may comprise any combination of wired or wireless networks including, but not limited to, a wireless cellular telephone network (such as a GSM, UMTS, CDMA, LTE, 4G, 5G network, and the like), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth® personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet. 
     The system  10  may include both wired and wireless communication devices and/or apparatus  50  suitable for implementing embodiments of the examples described herein. 
     For example, the system shown in  FIG.  3    shows a mobile telephone network  11  and a representation of the Internet  28 . Connectivity to the Internet  28  may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways. 
     The example communication devices shown in the system  10  may include, but are not limited to, an electronic device or apparatus  50 , a combination of a personal digital assistant (PDA) and a mobile telephone  14 , a PDA  16 , an integrated messaging device (IMD)  18 , a desktop computer  20 , a notebook computer  22 . The apparatus  50  may be stationary or mobile when carried by an individual who is moving. The apparatus  50  may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport. 
     The embodiments may also be implemented in a set-top box; for example, a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding. 
     Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection  25  to a base station  24 . The base station  24  may be connected to a network server  26  that allows communication between the mobile telephone network  11  and the Internet  28 . The system may include additional communication devices and communication devices of various types. 
     The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology. A communications device involved in implementing various embodiments of the examples described herein may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection. 
     In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers. 
     The embodiments may also be implemented in internet of things (IoT) devices. The IoT may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. The convergence of various technologies has and may enable many fields of embedded systems, such as wireless sensor networks, control systems, home/building automation, and the like, to be included the IoT. In order to utilize Internet IoT devices are provided with an IP address as a unique identifier. IoT devices may be provided with a radio transmitter, such as WLAN or Bluetooth transmitter, or a RFID tag. Alternatively, IoT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC). 
     The devices/system described in  FIGS.  1  to  3    enable coding, encoding, and/or transporting, for example, a neural network representation and a media stream. 
     An MPEG-2 transport stream (TS), specified in ISO/IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media as well as program metadata or other metadata, in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (a.k.a. packetized elementary stream) within the TS. Hence, a logical channel within an MPEG-2 TS may be considered to correspond to a specific PID value. 
     Available media file format standards include ISO base media file format (ISO/IEC 14496-12, which may be abbreviated ISOBMFF) and file format for NAL unit structured video (ISO/IEC 14496-15), which derives from the ISOBMFF. 
     Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage/transmission and a decoder that can decompress the compressed video representation back into a viewable form, or into a form that is suitable as an input to one or more algorithms for analysis or processing. A video encoder and/or a video decoder may also be separate from each other, for example, need not form a codec. Typically, encoder discards some information in the original video sequence in order to represent the video in a more compact form (e.g., at lower bitrate). 
     Typical hybrid video encoders, for example, many encoder implementations of ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or ‘block’) are predicted, for example, by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, for example, the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (for example, Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate). 
     In temporal prediction, the sources of prediction are previously decoded pictures (a.k.a. reference pictures). In intra block copy (IBC; a.k.a. intra-block-copy prediction and current picture referencing), prediction is applied similarly to temporal prediction, but the reference picture is the current picture and only previously decoded samples can be referred in the prediction process. Inter-layer or inter-view prediction may be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal prediction only, while in other cases inter prediction may refer collectively to temporal prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction. 
     Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, for example, either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra-coding, where no inter prediction is applied. 
     One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently when they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction. 
       FIG.  4    shows a block diagram of a general structure of a video encoder.  FIG.  4    presents an encoder for two layers, but it would be appreciated that presented encoder could be similarly extended to encode more than two layers.  FIG.  4    illustrates a video encoder comprising a first encoder section  500  for a base layer and a second encoder section  502  for an enhancement layer. Each of the first encoder section  500  and the second encoder section  502  may comprise similar elements for encoding incoming pictures. The encoder sections  500 ,  502  may comprise a pixel predictor  302 ,  402 , prediction error encoder  303 ,  403  and prediction error decoder  304 ,  404 .  FIG.  4    also shows an embodiment of the pixel predictor  302 ,  402  as comprising an inter-predictor  306 ,  406 , an intra-predictor  308 ,  408 , a mode selector  310 ,  410 , a filter  316 ,  416 , and a reference frame memory  318 ,  418 . The pixel predictor  302  of the first encoder section  500  receives base layer picture(s)/image(s)  300  of a video stream to be encoded at both the inter-predictor  306  (which determines the difference between the image and a motion compensated reference frame) and the intra-predictor  308  (which determines a prediction for an image block based only on the already processed parts of current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector  310 . The intra-predictor  308  may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector  310 . The mode selector  310  also receives a copy of the base layer picture(s)  300 . Correspondingly, the pixel predictor  402  of the second encoder section  502  receives enhancement layer picture(s)/images(s) of a video stream to be encoded at both the inter-predictor  406  (which determines the difference between the image and a motion compensated reference frame) and the intra-predictor  408  (which determines a prediction for an image block based only on the already processed parts of current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector  410 . The intra-predictor  408  may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector  410 . The mode selector  410  also receives a copy of the enhancement layer pictures  400 . 
     Depending on which encoding mode is selected to encode the current block, the output of the inter-predictor  306 ,  406  or the output of one of the optional intra-predictor modes or the output of a surface encoder within the mode selector is passed to the output of the mode selector  310 ,  410 . The output of the mode selector is passed to a first summing device  321 ,  421 . The first summing device may subtract the output of the pixel predictor  302 ,  402  from the base layer picture(s)  300 /enhancement layer picture(s)  400  to produce a first prediction error signal  320 ,  420  which is input to the prediction error encoder  303 ,  403 . 
     The pixel predictor  302 ,  402  further receives from a preliminary reconstructor  339 ,  439  the combination of the prediction representation of the image block  312 ,  412  and the output  338 ,  438  of the prediction error decoder  304 ,  404 . The preliminary reconstructed image  314 ,  414  may be passed to the intra-predictor  308 ,  408  and to the filter  316 ,  416 . The filter  316 ,  416  receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image  340 ,  440  which may be saved in the reference frame memory  318 ,  418 . The reference frame memory  318  may be connected to the inter-predictor  306  to be used as the reference image against which a future base layer picture  300  is compared in inter-prediction operations. Subject to the base layer being selected and indicated to be source for inter-layer sample prediction and/or inter-layer motion information prediction of the enhancement layer according to some embodiments, the reference frame memory  318  may also be connected to the inter-predictor  406  to be used as the reference image against which a future enhancement layer picture(s)  400  is compared in inter-prediction operations. Moreover, the reference frame memory  418  may be connected to the inter-predictor  406  to be used as the reference image against which the future enhancement layer picture(s)  400  is compared in inter-prediction operations. 
     Filtering parameters from the filter  316  of the first encoder section  500  may be provided to the second encoder section  502  subject to the base layer being selected and indicated to be source for predicting the filtering parameters of the enhancement layer according to some embodiments. 
     The prediction error encoder  303 ,  403  comprises a transform unit  342 ,  442  and a quantizer  344 ,  444 . The transform unit  342 ,  442  transforms the first prediction error signal  320 ,  420  to a transform domain. The transform is, for example, the DCT transform. The quantizer  344 ,  444  quantizes the transform domain signal, for example, the DCT coefficients, to form quantized coefficients. 
     The prediction error decoder  304 ,  404  receives the output from the prediction error encoder  303 ,  403  and performs the opposite processes of the prediction error encoder  303 ,  403  to produce a decoded prediction error signal  338 ,  438  which, when combined with the prediction representation of the image block  312 ,  412  at the second summing device  339 ,  439 , produces the preliminary reconstructed image  314 ,  414 . The prediction error decoder may be considered to comprise a dequantizer  346 ,  446 , which dequantizes the quantized coefficient values, for example, DCT coefficients, to reconstruct the transform signal and an inverse transformation unit  348 ,  448 , which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit  348 ,  448  contains reconstructed block(s). The prediction error decoder may also comprise a block filter which may filter the reconstructed block(s) according to further decoded information and filter parameters. 
     The entropy encoder  330 ,  430  receives the output of the prediction error encoder  303 ,  403  and may perform a suitable entropy encoding/variable length encoding on the signal to provide a compressed signal. The outputs of the entropy encoders  330 ,  430  may be inserted into a bitstream, for example, by a multiplexer  508 . 
       FIG.  5    is a block diagram showing the interface between an encoder  501  implementing neural network based encoding  503 , and a decoder  504  implementing neural network based decoding  505  in accordance with the examples described herein. The encoder  501  may embody a device, software method or hardware circuit. The encoder  501  has the goal of compressing input data  511  (for example, an input video) to compressed data  512  (for example, a bitstream) such that the bitrate measuring the size of compressed data  512  is minimized, and the accuracy of an analysis or processing algorithm is maximized. To this end, the encoder  501  uses an encoder or compression algorithm, for example to perform neural network based encoding  503 , e.g., encoding the input data by using one or more neural networks. 
     The general analysis or processing algorithm may be part of the decoder  504 . The decoder  504  uses a decoder or decompression algorithm, for example to perform the neural network based decoding  505  (e.g., decoding by using one or more neural networks) to decode the compressed data  512  (for example, compressed video) which was encoded by the encoder  501 . The decoder  504  produces decompressed data  513  (for example, reconstructed data). 
     The encoder  501  and decoder  504  may be entities implementing an abstraction, may be separate entities or the same entities, or may be part of the same physical device. 
     An out-of-band transmission, signaling, or storage may refer to the capability of transmitting, signaling, or storing information in a manner that associates the information with a video bitstream. The out-of-band transmission may use a more reliable transmission mechanism compared to the protocols used for carrying coded video data, such as slices. The out-of-band transmission, signaling or storage can additionally or alternatively be used e.g. for ease of access or session negotiation. For example, a sample entry of a track in a file conforming to the ISO Base Media File Format may comprise parameter sets, while the coded data in the bitstream is stored elsewhere in the file or in another file. Another example of out-of-band transmission, signaling, or storage comprises including information, such as NN and/or NN updates in a file format track that is separate from track(s) containing coded video data. 
     The phrase along the bitstream (e.g. indicating along the bitstream) or along a coded unit of a bitstream (e.g. indicating along a coded tile) may be used in claims and described embodiments to refer to transmission, signaling, or storage in a manner that the ‘out-of-band’ data is associated with, but not included within, the bitstream or the coded unit, respectively. The phrase decoding along the bitstream or along a coded unit of a bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or the coded unit, respectively. For example, the phrase along the bitstream may be used when the bitstream is contained in a container file, such as a file conforming to the ISO Base Media File Format, and certain file metadata is stored in the file in a manner that associates the metadata to the bitstream, such as boxes in the sample entry for a track containing the bitstream, a sample group for the track containing the bitstream, or a timed metadata track associated with the track containing the bitstream. In another example, the phrase along the bitstream may be used when the bitstream is made available as a stream over a communication protocol and a media description, such as a streaming manifest, is provided to describe the stream. 
     An elementary unit for the output of a video encoder and the input of a video decoder, respectively, may be a network abstraction layer (NAL) unit. For transport over packet-oriented networks or storage into structured files, NAL units may be encapsulated into packets or similar structures. A bytestream format encapsulating NAL units may be used for transmission or storage environments that do not provide framing structures. The bytestream format may separate NAL units from each other by attaching a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, encoders may run a byte-oriented start code emulation prevention algorithm, which may add an emulation prevention byte to the NAL unit payload when a start code would have occurred otherwise. In order to enable straightforward gateway operation between packet and stream-oriented systems, start code emulation prevention may be performed regardless of whether the bytestream format is in use or not. A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of a raw byte sequence payload interspersed as necessary with emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0. 
     In some coding standards, NAL units consist of a header and payload. The NAL unit header indicates the type of the NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g. called nuh_layer_id in H.265/HEVC and H.266/VVC), which could be used e.g. for indicating spatial or quality layers, views of a multiview video, or auxiliary layers (such as depth maps or alpha planes). In some coding standards, the NAL unit header includes a temporal sublayer identifier, which may be used for indicating temporal subsets of the bitstream, such as a 30-frames-per-second subset of a 60-frames-per-second bitstream. 
     NAL units may be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coded slice NAL units. 
     A non-VCL NAL unit may be, for example, one of the following types: a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded pictures, whereas many of the other non-VCL NAL units are not necessary for the reconstruction of decoded sample values. 
     Some coding formats specify parameter sets that may carry parameter values needed for the decoding or reconstruction of decoded pictures. A parameter may be defined as a syntax element of a parameter set. A parameter set may be defined as a syntax structure that contains parameters and that can be referred to from or activated by another syntax structure, for example, using an identifier. 
     Some types of parameter sets are briefly described in the following, but it needs to be understood, that other types of parameter sets may exist and that embodiments may be applied, but are not limited to, the described types of parameter sets. 
     Parameters that remain unchanged through a coded video sequence may be included in a sequence parameter set (SPS). Alternatively, the SPS may be limited to apply to a layer that references the SPS, e.g. the SPS may remain valid for a coded layer video sequence. In addition to the parameters that may be needed by the decoding process, the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. 
     A picture parameter set contains such parameters that are likely to be unchanged in several coded pictures. A picture parameter set may include parameters that can be referred to by the VCL NAL units of one or more coded pictures. 
     A video parameter set (VPS) may be defined as a syntax structure containing syntax elements that apply to zero or more entire coded video sequences and may contain parameters applying to multiple layers. The VPS may provide information about the dependency relationships of the layers in a bitstream, as well as many other information that are applicable to all slices across all layers in the entire coded video sequence. 
     A video parameter set RBSP may include parameters that can be referred to by one or more sequence parameter set RBSPs. 
     The relationship and hierarchy between the VPS, the SPS, and the PPS may be described as follows. A VPS resides one level above an SPS in the parameter set hierarchy and in the context of scalability. The VPS may include parameters that are common for all slices across all layers in the entire coded video sequence. The SPS includes the parameters that are common for all slices in a particular layer in the entire coded video sequence, and may be shared by multiple layers. The PPS includes the parameters that are common for all slices in a particular picture and are likely to be shared by all slices in multiple pictures. 
     The APS may be specified in some coding formats, such as H.266/VVC. The APS may be applied to one or more image segments, such as slices. In H.266/VVC, the APS may be defined as a syntax structure containing syntax elements that apply to zero or more slices as determined by zero or more syntax elements found in slice headers or in a picture header. The APS may comprise a type (aps_params_type in H.266/VVC) and an identifier (aps_adaptation_parameter_set_id in H.266/VVC). The combination of an APS type and an APS identifier may be used to identify a particular APS. H.266/VVC comprises three APS types: an adaptive loop filtering (ALF), a luma mapping with chroma scaling (LMCS), and a scaling list APS types. The ALF APS(s) are referenced from a slice header (thus, the referenced ALF APSs can change slice by slice), and the LMCS and scaling list APS(s) are referenced from a picture header (thus, the referenced LMCS and scaling list APSs can change picture by picture). In H.266/VVC, the APS RBSP has the following syntax: 
     
       
         
           
               
               
             
               
                   
                   
               
               
                   
                 Descriptor 
               
               
                   
                   
               
             
            
               
                   
               
            
           
           
               
               
               
            
               
                   
                 adaptation_parameter_set_rbsp( ) { 
                   
               
               
                   
                  aps_params_type 
                 u(3) 
               
               
                   
                  aps_adaptation_parameter_set_id 
                 u(5) 
               
               
                   
                  aps_chroma_present_flag 
                 u(1) 
               
               
                   
                  if( aps_params_type = = ALF_APS ) 
               
               
                   
                   alf_data( ) 
               
               
                   
                  else if( aps_params_type = = LMCS_APS ) 
               
               
                   
                   lmcs_data( ) 
               
               
                   
                  else if( aps_params_type = = SCALING_APS ) 
               
               
                   
                   scaling_list_data( ) 
               
               
                   
                  aps_extension_flag 
                 u(1) 
               
               
                   
                  if( aps_extension_flag ) 
               
               
                   
                   while( more_rbsp_data( ) ) 
               
               
                   
                    aps_extension_data_flag 
                 u(1) 
               
               
                   
                  rbsp_trailing_bits( ) 
               
               
                   
                 } 
               
               
                   
                   
               
            
           
         
       
     
     Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units. A prefix SEI NAL unit can start a picture unit or alike; and a suffix SEI NAL unit can end a picture unit or alike. Hereafter, an SEI NAL unit may equivalently refer to a prefix SEI NAL unit or a suffix SEI NAL unit. An SEI NAL unit includes one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. 
     Several SEI messages are specified in H.264/AVC, H.265/HEVC, H.266/VVC, and H.274/VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for specific use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified. 
     The method and apparatus of an example embodiment may be utilized in a wide variety of systems, including systems that rely upon the compression and decompression of media data and possibly also the associated metadata. In one embodiment, however, the method and apparatus are configured to train or finetune a decoder side neural network. In this regard,  FIG.  6    depicts an example of such a system  600  that includes a source  602  of media data and associated metadata. The source may be, in one embodiment, a server. However, the source may be embodied in other manners when so desired. The source is configured to stream the media data and associated metadata to a client device  604 . The client device may be embodied by a media player, a multimedia system, a video system, a smart phone, a mobile telephone or other user equipment, a personal computer, a tablet computer or any other computing device configured to receive and decompress the media data and process associated metadata. In the illustrated embodiment, media data and metadata are streamed via a network  606 , such as any of a wide variety of types of wireless networks and/or wireline networks. The client device is configured to receive structured information containing media, metadata and any other relevant representation of information containing the media and the metadata and to decompress the media data and process the associated metadata (e.g. for proper playback timing of decompressed media data). 
     An apparatus  700  is provided in accordance with an example embodiment as shown in  FIG.  7   . In one embodiment, the apparatus of  FIG.  7    may be embodied by the source  602 , such as a file writer which, in turn, may be embodied by a server, that is configured to stream a compressed representation of the media data and associated metadata. In an alternative embodiment, the apparatus may be embodied by the client device  604 , such as a file reader which may be embodied, for example, by any of the various computing devices described above. In either of these embodiments and as shown in  FIG.  7   , the apparatus of an example embodiment includes, is associated with or is in communication with a processing circuitry  702 , one or more memory devices  704 , a communication interface  706  and optionally a user interface. 
     The processing circuitry  702  may be in communication with the memory device  704  via a bus for passing information among components of the apparatus  700 . The memory device may be non-transitory and may include, for example, one or more volatile and/or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer readable storage medium) comprising gates configured to store data (e.g., bits) that may be retrievable by a machine (e.g., a computing device like the processing circuitry). The memory device may be configured to store information, data, content, applications, instructions, or the like for enabling the apparatus to carry out various functions in accordance with an example embodiment of the present disclosure. For example, the memory device could be configured to buffer input data for processing by the processing circuitry. Additionally or alternatively, the memory device could be configured to store instructions for execution by the processing circuitry. 
     The apparatus  700  may, in some embodiments, be embodied in various computing devices as described above. However, in some embodiments, the apparatus may be embodied as a chip or chip set. In other words, the apparatus may comprise one or more physical packages (e.g., chips) including materials, components and/or wires on a structural assembly (e.g., a baseboard). The structural assembly may provide physical strength, conservation of size, and/or limitation of electrical interaction for component circuitry included thereon. The apparatus may therefore, in some cases, be configured to implement an embodiment of the present disclosure on a single chip or as a single ‘system on a chip.’ As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein. 
     The processing circuitry  702  may be embodied in a number of different ways. For example, the processing circuitry may be embodied as one or more of various hardware processing means such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an accompanying DSP, or various other circuitry including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like. As such, in some embodiments, the processing circuitry may include one or more processing cores configured to perform independently. A multi-core processing circuitry may enable multiprocessing within a single physical package. Additionally or alternatively, the processing circuitry may include one or more processors configured in tandem via the bus to enable independent execution of instructions, pipelining and/or multithreading. 
     In an example embodiment, the processing circuitry  702  may be configured to execute instructions stored in the memory device  704  or otherwise accessible to the processing circuitry. Alternatively or additionally, the processing circuitry may be configured to execute hard coded functionality. As such, whether configured by hardware or software methods, or by a combination thereof, the processing circuitry may represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to an embodiment of the present disclosure while configured accordingly. Thus, for example, when the processing circuitry is embodied as an ASIC, FPGA or the like, the processing circuitry may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processing circuitry is embodied as an executor of instructions, the instructions may specifically configure the processing circuitry to perform the algorithms and/or operations described herein when the instructions are executed. However, in some cases, the processing circuitry may be a processor of a specific device (e.g., an image or video processing system) configured to employ an embodiment of the present invention by further configuration of the processing circuitry by instructions for performing the algorithms and/or operations described herein. The processing circuitry may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processing circuitry. 
     The communication interface  706  may be any means such as a device or circuitry embodied in either hardware or a combination of hardware and software that is configured to receive and/or transmit data, including video bitstreams. In this regard, the communication interface may include, for example, an antenna (or multiple antennas) and supporting hardware and/or software for enabling communications with a wireless communication network. Additionally or alternatively, the communication interface may include the circuitry for interacting with the antenna(s) to cause transmission of signals via the antenna(s) or to handle receipt of signals received via the antenna(s). In some environments, the communication interface may alternatively or also support wired communication. As such, for example, the communication interface may include a communication modem and/or other hardware/software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB) or other mechanisms. 
     In some embodiments, the apparatus  700  may optionally include a user interface that may, in turn, be in communication with the processing circuitry  702  to provide output to a user, such as by outputting an encoded video bitstream and, in some embodiments, to receive an indication of a user input. As such, the user interface may include a display and, in some embodiments, may also include a keyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, a microphone, a speaker, or other input/output mechanisms. Alternatively or additionally, the processing circuitry may comprise user interface circuitry configured to control at least some functions of one or more user interface elements such as a display and, in some embodiments, a speaker, ringer, microphone and/or the like. The processing circuitry and/or user interface circuitry comprising the processing circuitry may be configured to control one or more functions of one or more user interface elements through computer program instructions (e.g., software and/or firmware) stored on a memory accessible to the processing circuitry (e.g., memory device, and/or the like). 
     Fundamentals of Neural Networks 
     A neural network (NN) is a computation graph consisting of several layers of computation. Each layer consists of one or more units, where each unit performs a computation. A unit is connected to one or more other units, and a connection may be associated with a weight. The weight may be used for scaling the signal passing through an associated connection. Weights are learnable parameters, for example, values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers. 
     Couple of examples of architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop, each layer takes input from one or more of the previous layers, and provides its output as the input for one or more of the subsequent layers. Also, units inside a certain layer take input from units in one or more of preceding layers and provide output to one or more of following layers. 
     Initial layers, those close to the input data, extract semantically low-level features, for example, edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers, there may be one or more layers performing a certain task, for example, classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, and the like. In recurrent neural networks, there is a feedback loop, so that the neural network becomes stateful, for example, it is able to memorize information or a state. 
     Neural networks are being utilized in an ever-increasing number of applications for many different types of devices, for example, mobile phones, chat bots, IoT devices, smart cars, voice assistants, and the like. Some of these applications include, but are not limited to, image and video analysis and processing, social media data analysis, device usage data analysis, and the like. 
     One of the properties of neural networks, and other machine learning tools, is that they are able to learn properties from input data, either in a supervised way or in an unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal. 
     In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output error, also referred to as the loss. Examples of losses are mean squared error, cross-entropy, and the like. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural network to make a gradual improvement in the network&#39;s output, for example, gradually decrease the loss. 
     Training a neural network is an optimization process, but the final goal is different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, for example, data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, for example, to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data, which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following:
         when the network is learning at all—in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.   when the network is learning to generalize—in this case, also the validation set error needs to decrease and be not too much higher than the training set error. For example, the validation set error should be less than 20% higher than the training set error. When the training set error is low, for example, 10% of its value at the beginning of training, or with respect to a threshold that may have been determined based on an evaluation metric, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized properties of the training set and performs well only on that set, but performs poorly on a set not used for training or tuning its parameters.       

     Lately, neural networks have been used for compressing and de-compressing data such as images. The most widely used architecture for such task is the auto-encoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. In various embodiments, these neural encoder and neural decoder would be referred to as encoder and decoder, even though these refer to algorithms which are learned from data instead of being tuned manually. The encoder takes an image as an input and produces a code, to represent the input image, which requires less bits than the input image. This code may have been obtained by a binarization or quantization process after the encoder. The decoder takes in this code and reconstructs the image which was input to the encoder. 
     Such encoder and decoder are usually trained to minimize a combination of bitrate and distortion, where the distortion may be based on one or more of the following metrics: mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), or the like. These distortion metrics are meant to be correlated to the human visual perception quality, so that minimizing or maximizing one or more of these distortion metrics results into improving the visual quality of the decoded image as perceived by humans. 
     In various embodiments, terms ‘model’, ‘neural network’, ‘neural net’ and ‘network’ may be used interchangeably, and also the weights of neural networks may be sometimes referred to as learnable parameters or as parameters. 
     Fundamentals of Video/Image Coding 
     Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage/transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form, for example, at lower bitrate. 
     Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly, pixel values in a certain picture area (or ‘block’) are predicted. In an example, the pixel values may be predicted by using motion compensation algorithm. This prediction technique includes finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded. 
     In other example, the pixel values may be predicted by using spatial prediction techniques. This prediction technique uses the pixel values around the block to be coded in a specified manner. Secondly, the prediction error, for example, the difference between the predicted block of pixels and the original block of pixels is coded. This is typically done by transforming the difference in pixel values using a specified transform, for example, discrete cosine transform (DCT) or a variant of it; quantizing the coefficients; and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation, for example, picture quality and size of the resulting coded video representation, for example, file size or transmission bitrate. 
     Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. 
     Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, for example, either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra-coding, where no inter prediction is applied. 
     One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently when they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction. 
     The decoder reconstructs the output video by applying prediction techniques similar to the encoder to form a predicted representation of the pixel blocks. For example, using the motion or spatial information created by the encoder and stored in the compressed representation and prediction error decoding, which is inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain. After applying prediction and prediction error decoding techniques the decoder sums up the prediction and prediction error signals, for example, pixel values to form the output video frame. The decoder and encoder can also apply additional filtering techniques to improve the quality of the output video before passing it for display and/or storing it as prediction reference for the forthcoming frames in the video sequence. 
     In typical video codecs the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded in the encoder side or decoded in the decoder side and the prediction source block in one of the previously coded or decoded pictures. 
     In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs, the predicted motion vectors are created in a predefined way, for example, calculating the median of the encoded or decoded motion vectors of the adjacent blocks. 
     Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and/or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded/decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and/or or co-located blocks in temporal reference picture. 
     Moreover, typical high efficiency video codecs employ an additional motion information coding/decoding mechanism, often called merging/merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification/correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and/or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent/co-located blocks. 
     In typical video codecs, the prediction residual after motion compensation is first transformed with a transform kernel, for example, DCT and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding. 
     Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, for example, the desired macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor λ to tie together the exact or estimated image distortion due to lossy coding methods and the exact or estimated amount of information that is required to represent the pixel values in an image area: 
         C=D+λR   equation 1
 
     In equation 1, C is the Lagrangian cost to be minimized, D is the image distortion, for example, mean squared error with the mode and motion vectors considered, and R is the number of bits needed to represent the required data to reconstruct the image block in the decoder including the amount of data to represent the candidate motion vectors. 
     Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. 
     Several SEI messages are specified in H.264/AVC, H.265/HEVC, H.266/VVC, and H.274/VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified. 
     A design principle has been followed for SEI message specifications: the SEI messages are generally not extended in future amendments or versions of the standard. 
     Filters in Video Codecs 
     Conventional image and video codecs use a set of filters to enhance the visual quality of the predicted and error-compensated visual content and can be applied either in-loop or out-of-loop, or both. In the case of in-loop filters, the filter applied on one block in the currently-encoded or currently decoded frame will affect the encoding or decoding of another block in the same frame and/or in another frame which is predicted from the current frame. An in-loop filter can affect the bitrate and/or the visual quality. An enhanced block may cause a smaller residual, e.g., a smaller difference between original block and filtered block, thus using less bits in the bitstream output by the encoder. An out-of-loop filter may be applied on a frame or part of a frame after it has been reconstructed, the filtered visual content may not be a source for prediction, and thus it may only impact the visual quality of the frames that are output by the decoder. 
     Information on Neural Network Based Image/Video Coding 
     Recently, neural networks (NNs) have been used in the context of image and video compression, by following mainly two approaches. 
     In one approach, NNs are used to replace or as an addition to one or more of the components of a traditional codec such as VVC/H.266. Here, by ‘traditional’, it is meant, those codecs whose components and their parameters are typically not learned from data by means of a training process, for example those codecs whose components are not neural networks. Some examples of uses of neural networks within a traditional codec include but are not limited to:
         Additional in-loop filter, for example by having the NN as an additional in-loop filter with respect to the traditional loop filters.   Single in-loop filter, for example by having the NN replacing all traditional in-loop filters.   Intra-frame prediction, for example as an additional intra-frame prediction mode, or replacing the traditional intra-frame prediction.   Inter-frame prediction, for example as an additional inter-frame prediction mode, or replacing the traditional inter-frame prediction.   Transform and/or inverse transform, for example as an additional transform and/or inverse transform, or replacing the traditional transform and/or inverse transform.   Probability model for the arithmetic codec, for example as an additional probability model, or replacing the traditional probability model.       

       FIG.  8    illustrates examples of functioning of NNs as components of a pipeline of traditional codec pipeline, in accordance with an embodiment. In particular,  FIG.  8    illustrates an encoder, which also includes a decoding loop.  FIG.  8    is shown to include components described below:
         A luma intra pred block or circuit  801 . This block or circuit performs intra prediction in the luma domain, for example, by using already reconstructed data from the same frame. The operation of the luma intra pred block or circuit  801  may be performed by a deep neural network such as a convolutional auto-encoder.   A chroma intra pred block or circuit  802 . This block or circuit performs intra prediction in the chroma domain, for example, by using already reconstructed data from the same frame. The chroma intra pred block or circuit  802  may perform cross-component prediction, for example, predicting chroma from luma. The operation of the chroma intra pred block or circuit  802  may be performed by a deep neural network such as a convolutional auto-encoder.   An intra pred block or circuit  803  and an inter-pred block or circuit  804 . These blocks or circuit perform intra prediction and inter-prediction, respectively. The intra pred block or circuit  803  and the inter-pred block or circuit  804  may perform the prediction on all components, for example, luma and chroma. The operations of the intra pred block or circuit  803  and the inter-pred block or circuit  804  may be performed by two or more deep neural networks such as convolutional auto-encoders.   A probability estimation block or circuit  805  for entropy coding. This block or circuit performs prediction of probability for the next symbol to encode or decode, which is then provided to the entropy coding module  812 , such as an arithmetic coding module, to encode or decode the next symbol. The operation of the probability estimation block or circuit  805  may be performed by a neural network.   A transform and quantization (T/Q) block or circuit  806 . These are actually two blocks or circuits. The transform and quantization block or circuit  806  may perform a transform of input data to a different domain, for example, the FFT transform would transform the data to frequency domain. The transform and quantization block or circuit  806  may quantize its input values to a smaller set of possible values. In the decoding loop, there may be inverse quantization block or circuit and inverse transform block or circuit Q − 1/T −1 813. One or both of the transform block or circuit and quantization block or circuit may be replaced by one or two or more neural networks. One or both of the inverse transform block or circuit and inverse quantization block or circuit  813  may be replaced by one or two or more neural networks.   An in-loop filter block or circuit  807 . Operations of the in-loop filter block or circuit  807  is performed in the decoding loop, and it performs filtering on the output of the inverse transform block or circuit, or anyway on the reconstructed data, in order to enhance the reconstructed data with respect to one or more predetermined quality metrics. This filter may affect both the quality of the decoded data and the bitrate of the bitstream output by the encoder. The operation of the in-loop filter block or circuit  807  may be performed by a neural network, such as a convolutional auto-encoder. In examples, the operation of the in-loop filter may be performed by multiple steps or filters, where the one or more steps may be performed by neural networks.   A post-processing filter block or circuit  808 . The post-processing filter block or circuit  808  may be performed only at decoder side, as it may not affect the encoding process. The post-processing filter block or circuit  808  filters the reconstructed data output by the in-loop filter block or circuit  807 , in order to enhance the reconstructed data. The post-processing filter block or circuit  808  may be replaced by a neural network, such as a convolutional auto-encoder.   A resolution adaptation block or circuit  809 : this block or circuit may downsample the input video frames, prior to encoding. Then, in the decoding loop, the reconstructed data may be upsampled, by the upsampling block or circuit  810 , to the original resolution. The operation of the resolution adaptation block or circuit  809  block or circuit may be performed by a neural network such as a convolutional auto-encoder.   An encoder control block or circuit  811 . This block or circuit performs optimization of encoder&#39;s parameters, such as what transform to use, what quantization parameters (QP) to use, what intra-prediction mode (out of N intra-prediction modes) to use, and the like. The operation of the encoder control block or circuit  811  may be performed by a neural network, such as a classifier convolutional network, or such as a regression convolutional network.   An ME/MC block or circuit  814  performs motion estimation and/or motion compensation, which are two key operations to be performed when performing inter-frame prediction. ME/MC stands for motion estimation/motion compensation       

     In another approach, commonly referred to as ‘end-to-end learned compression’, NNs are used as the main components of the image/video codecs. Some examples of the second approach include, but are not limited to following: 
     Option 1: re-use the video coding pipeline but replace most or all the components with NNs. Referring to  FIG.  9   , it illustrates an example of modified video coding pipeline based on neural networks, in accordance with an embodiment. An example of neural network may include, but is not limited, a compressed representation of a neural network.  FIG.  9    is shown to include following components:
         A neural transform block or circuit  902 : this block or circuit transforms the output of a summation/subtraction operation  903  to a new representation of that data, which may have lower entropy and thus be more compressible.   A quantization block or circuit  904 : this block or circuit quantizes an input data  901  to a smaller set of possible values.   An inverse transform and inverse quantization blocks or circuits  906 . These blocks or circuits perform the inverse or approximately inverse operation of the transform and the quantization, respectively.   An encoder parameter control block or circuit  908 . This block or circuit may control and optimize some or all the parameters of the encoding process, such as parameters of one or more of the encoding blocks or circuits.   An entropy coding block or circuit  910 . This block or circuit may perform lossless coding, for example based on entropy. One popular entropy coding technique is arithmetic coding.   A neural intra-codec block or circuit  912 . This block or circuit may be an image compression and decompression block or circuit, which may be used to encode and decode an intra frame. An encoder  914  may be an encoder block or circuit, such as the neural encoder part of an auto-encoder neural network. A decoder  916  may be a decoder block or circuit, such as the neural decoder part of an auto-encoder neural network. An intra-coding block or circuit  918  may be a block or circuit performing some intermediate steps between encoder and decoder, such as quantization, entropy encoding, entropy decoding, and/or inverse quantization.   A deep loop filter block or circuit  920 . This block or circuit performs filtering of reconstructed data, in order to enhance it.   A decode picture buffer block or circuit  922 . This block or circuit is a memory buffer, keeping the decoded frame, for example, reconstructed frames  924  and enhanced reference frames  926  to be used for inter prediction.   An inter-prediction block or circuit  928 . This block or circuit performs inter-frame prediction, for example, predicts from frames, for example, frames  932 , which are temporally nearby. An ME/MC  930  performs motion estimation and/or motion compensation, which are two key operations to be performed when performing inter-frame prediction. ME/MC stands for motion estimation/motion compensation.       

     In order to train the neural networks of this system, a training objective function, referred to as ‘training loss’, is typically utilized, which usually comprises one or more terms, or loss terms, or simply losses. Although here the Option 2 and  FIG.  10    considered as example for describing the training objective function, a similar training objective function may also be used for training the neural networks for the systems in  FIG.  6    and  FIG.  7   . In one example, the training loss comprises a reconstruction loss term and a rate loss term. The reconstruction loss encourages the system to decode data that is similar to the input data, according to some similarity metric. Following are some examples of reconstruction losses are:
         a loss derived from mean squared error (MSE);   a loss derived from multi-scale structural similarity (MS-SSIM), such as 1 minus MS-SSIM, or 1−MS-SSIM;   losses derived from the use of a pretrained neural network. For example, error(f 1 , f 2 ), where f 1  and f 2  are the features extracted by a pretrained neural network for the input (uncompressed) data and the decoded (reconstructed) data, respectively, and error( ) is an error or distance function, such as L 1  norm or L 2  norm; and   losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec. For example, adversarial loss can be used, which is the loss provided by a discriminator neural network that is trained adversarially with respect to the codec, following the settings proposed in the context of generative adversarial networks (GANs) and their variants.       

     The rate loss encourages the system to compress the output of the encoding stage, such as the output of the arithmetic encoder. ‘Compressing’ for example, means reducing the number of bits output by the encoding stage. 
     When an entropy-based lossless encoder is used, such as the arithmetic encoder, the rate loss typically encourages the output of the Encoder NN to have low entropy. The rate loss may be computed on the output of the Encoder NN, or on the output of the quantization operation, or on the output of the probability model. Following are some example of rate losses:
         A differentiable estimate of the entropy;   A sparsification loss, for example, a loss that encourages the output of the Encoder NN or the output of the quantization to have many zeros. Examples are L 0  norm, L 1  norm, L 1  norm divided by L 2  norm; and   A cross-entropy loss applied to the output of a probability model, where the probability model may be a NN used to estimate the probability of the next symbol to be encoded by the arithmetic encoder.       

     For training one or more neural networks that are part of a codec, for example, one or more neural networks in  FIG.  8    and/or  FIG.  9   , one or more of reconstruction losses and one or more of rate losses may be used. the loss terms may then be combined, for example, as a weighted sum to obtain the training objective function. Typically, the different loss terms are weighted using different weights, and these weights determine how the final system performs in terms of rate-distortion loss. For example, when more weight is given to one or more of the reconstruction losses with respect to the rate losses, the system may learn to compress less but to reconstruct with higher accuracy as measured by a metric that correlates with the reconstruction losses. These weights are usually considered to be hyper-parameters of the training session and may be set manually by the operator designing the training session, or automatically, for example, by grid search or by using additional neural networks. 
     For the sake of explanation, video is considered as data type in various embodiments. However, it would be understood that the embodiments are also applicable to other media items, for example, images and audio data. 
     It is to be understood that even in end-to-end learned approaches, there may be components which are not learned from data, such as an arithmetic codec. 
     Option 2 is illustrated in  FIG.  10   , and it consists of a different type of codec architecture. Referring to  FIG.  10   , it illustrates an example neural network-based end-to-end learned video coding system, in accordance with an example embodiment. As shown  FIG.  10   , a neural network-based end-to-end learned video coding system  1000  includes an encoder  1001 , a quantizer  1002 , a probability model  1003 , an entropy codec  1004 , for example, an arithmetic encoder  1005  and an arithmetic decoder  1006 , a dequantizer  1007 , and a decoder  1008 . The encoder  1001  and the decoder  1008  are typically two neural networks, or mainly comprise neural network components. The probability model  1003  may also comprise mainly neural network components. The quantizer  1002 , the dequantizer  1007 , and the entropy codec  1004  are typically not based on neural network components, but they may also potentially comprise neural network components. In some embodiments, the encoder, quantizer, probability model, entropy codec, arithmetic encoder, arithmetic decoder, dequantizer, and decoder, may also be referred to as an encoder component, quantizer component, probability model component, entropy codec component, arithmetic encoder component, arithmetic decoder component, dequantizer component, and decoder component respectively. 
     On the encoding side, the encoder  1001  takes a video/image as an input  1009  and converts the video/image in original signal space into a latent representation that may comprise a more compressible representation of the input. The latent representation may be normally a 3-dimensional tensor for image compression, where 2 dimensions represent spatial information, and the third dimension contains information at that specific location. 
     Consider an example, in which the input data is an image, when the input image is a 128×128×3 RGB image (with horizontal size of 128 pixels, vertical size of 128 pixels, and  3  channels for the Red, Green, Blue color components), and when the encoder downsamples the input tensor by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or ‘shape’) 64×64×32 (e.g., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). Please note that the order of the different dimensions may differ depending on the convention which is used. In some embodiments, for the input image, the channel dimension may be the first dimension, so for the above example, the shape of the input tensor may be represented as 3×128×128, instead of 128×128×3. 
     In the case of an input video (instead of just an input image), another dimension in the input tensor may be used to represent temporal information. 
     The quantizer  1002  quantizes the latent representation into discrete values given a predefined set of quantization levels. The probability model  1003  and the arithmetic encoder  1005  work together to perform lossless compression for the quantized latent representation and generate bitstreams to be sent to the decoder side. Given a symbol to be encoded to the bitstream, the probability model  1003  estimates the probability distribution of possible values for that symbol based on a context that is constructed from available information at the current encoding/decoding state, such as the data that has already encoded/decoded. The arithmetic encoder  1005  encodes the input symbols to bitstream using the estimated probability distributions. 
     On the decoding side, opposite operations are performed. The arithmetic decoder  1006  and the probability model  1003  first decode symbols from the bitstream to recover the quantized latent representation. Then, the dequantizer  1007  reconstructs the latent representation in continuous values and pass it to the decoder  1008  to recover the input video/image. The recovered input video/image is provided as an output  1010 . Note that the probability model  1003 , in this system  1000 , is shared between the arithmetic encoder  1005  and the arithmetic decoder  1006 . In practice, this means that a copy of the probability model  1003  is used at the arithmetic encoder  1005  side, and another exact copy is used at the arithmetic decoder  1006  side. 
     In this system  1000 , the encoder  1001 , the probability model  1003 , and the decoder  1008  are normally based on deep neural networks. The system  1000  is trained in an end-to-end manner by minimizing the following rate-distortion loss function, which may be referred to simply as training loss, or loss: 
         L=D+λR   equation 2
 
     In equation 2, D is the distortion loss term, R is the rate loss term, and X is the weight that controls the balance between the two losses. 
     The distortion loss term may be referred to also as reconstruction loss. It encourages the system to decode data that is similar to the input data, according to some similarity metric. Some examples of reconstruction losses are:
         a loss derived from mean squared error (MSE).   a loss derived from multi-scale structural similarity (MS-SSIM), such as 1 minus MS-SSIM, or 1−MS-SSIM.   losses derived from the use of a pretrained neural network. For example, error(f 1 , f 2 ), where f 1  and f 2  are the features extracted by a pretrained neural network for the input (uncompressed) data and the decoded (reconstructed) data, respectively, and erroro is an error or distance function, such as L 1  norm or L 2  norm.   losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec. For example, adversarial loss can be used, which is the loss provided by a discriminator neural network that is trained adversarially with respect to the codec, following the settings proposed in the context of generative adversarial networks (GANs) and their variants.       

     Multiple distortion losses may be used and integrated into D. 
     Minimizing the rate loss encourages the system to compress the quantized latent representation so that the quantized latent representation can be represented by a smaller number of bits. The rate loss may be computed on the output of the encoder NN, or on the output of the quantization operation, or on the output of the probability model. In one example embodiment, the rate loss may comprise multiple rate losses. Following are some examples of rate losses:
         a differentiable estimate of the entropy of the quantized latent representation, which indicates the number of bits necessary to represent the encoded symbols, for example, bits-per-pixel (bpp).   a sparsification loss, for example, a loss that encourages the output of the Encoder NN or the output of the quantization to have many zeros. Examples are L 0  norm, L 1  norm, L 1  norm divided by L 2  norm.   a cross-entropy loss applied to the output of a probability model, where the probability model may be a NN used to estimate the probability of the next symbol to be encoded by the arithmetic encoder  1005 .       

     A similar training loss may be used for training the systems illustrated in  FIG.  8    and  FIG.  9   . 
     For training one or more neural networks that are part of a codec, for example, one or more neural networks in  FIG.  8    and/or  FIG.  9   , one or more of reconstruction losses and one or more of rate losses may be used. The loss terms may then be combined for example as a weighted sum to obtain the training objective function. Typically, the different loss terms are weighted using different weights, and these weights determine how the final system performs in terms of rate-distortion loss. For example, when more weight is given to one or more of the reconstruction losses with respect to the rate losses, the system may learn to compress less but to reconstruct with higher accuracy as measured by a metric that correlates with the reconstruction losses. These weights are usually considered to be hyper-parameters of the training session and may be set manually by the operator designing the training session, or automatically for example by grid search or by using additional neural networks. 
     In one example embodiment, the rate loss and the reconstruction loss may be minimized jointly at each iteration. In another example embodiment, the rate loss and the reconstruction loss may be minimized alternately, e.g., in one iteration the rate loss is minimized and in the next iteration the reconstruction loss is minimized, and so on. In yet another example embodiment, the rate loss and the reconstruction loss may be minimized sequentially, e.g., first one of the two losses is minimized for a certain number of iterations, and then the other loss is minimized for another number of iterations. These different ways of minimizing rate loss and reconstruction loss may also be combined. 
     It is to be understood that even in end-to-end learned approaches, there may be components which are not learned from data, such as an arithmetic codec. 
     For lossless video/image compression, the system  1000  contains the probability model  1003 , the arithmetic encoder  1005  and the arithmetic decoder  1006 . The system loss function contains the rate loss, since the distortion loss is always zero, in other words, no loss of information. 
     Video Coding for Machines (VCM) 
     Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, e.g. consuming or watching the decoded images or videos. Recently, with the advent of machine learning, especially deep learning, there is a rising number of machines (e.g., autonomous agents) that analyze or process data independently from humans and may even take decisions based on the analysis results without human intervention. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, and the like. Example use cases and applications are self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart TV and smart advertisement, person re-identification, smart traffic monitoring, drones, and the like. Accordingly, when decoded data is consumed by machines, a quality metric for the decoded data may be defined, which may be different from a quality metric for human perceptual quality. Also, dedicated algorithms for compressing and decompressing data for machine consumption may be different than those for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for machine consumption is referred to here as Video Coding for Machines. 
     The decoder-side device may have multiple ‘machines’ or neural networks (NNs) for analyzing or processing decoded data. These multiple machines may be used in a certain combination which is for example determined by an orchestrator sub-system. The multiple machines may be used for example in temporal succession, based on the output of the previously used machine, and/or in parallel. For example, a video which was compressed and then decompressed may be analyzed by one machine (NN) for detecting pedestrians, by another machine (another NN) for detecting cars, and by another machine (another NN) for estimating the depth of objects in the frames. 
     An ‘encoder-side device’ may encode input data, such as a video, into a bitstream which represents compressed data. The bitstream is provided to a ‘decoder-side device’. The term ‘receiver-side’ or ‘decoder-side’ refers to a physical or abstract entity or device which performs decoding of compressed data, and the decoded data may be input to one or more machines, circuits or algorithms. The one or more machines may not be part of the decoder. The one or more machines may be run by the same device running the decoder or by another device which receives the decoded data from the device running the decoder. Different machines may be run by different devices. 
     The encoded video data may be stored into a memory device, for example as a file. The stored file may later be provided to another device. 
     Alternatively, the encoded video data may be streamed from one device to another. 
       FIG.  11    illustrates a pipeline of video coding for machines (VCM), in accordance with an embodiment. A VCM encoder  1102  encodes the input video into a bitstream  1104 . A bitrate  1106  may be computed  1108  from the bitstream  1104  in order to evaluate the size of the bitstream  1104 . A VCM decoder  1110  decodes the bitstream  1104  output by the VCM encoder  1102 . An output of the VCM decoder  1110  may be referred, for example, as decoded data for machines  1112 . This data may be considered as the decoded or reconstructed video. However, in some implementations of the pipeline of VCM, the decoded data for machines  1112  may not have same or similar characteristics as the original video which was input to the VCM encoder  1102 . For example, this data may not be easily understandable by a human, when the human watches the decoded video from a suitable output device such as a display. The output of the VCM decoder  1110  is then input to one or more task neural network (task-NN). For the sake of illustration,  FIG.  11    is shown to include three example task-NNs, a task-NN  1114  for object detection, a task-NN  1116  for image segmentation, a task-NN  1118  for object tracking, and a non-specified one, a task-NN  1120  for performing task X. The goal of VCM is to obtain a low bitrate while guaranteeing that the task-NNs still perform well in terms of the evaluation metric associated to each task. 
     One of the possible approaches to realize video coding for machines is an end-to-end learned approach.  FIG.  12    illustrates an example of an end-to-end learned approach, in accordance with an embodiment. In this approach, a VCM encoder  1202  and a VCM decoder  1204  mainly consist of neural networks. The video is input to a neural network encoder  1206 . The output of the neural network encoder  1206  is input to a lossless encoder  1208 , such as an arithmetic encoder, which outputs a bitstream  1210 . The lossless codec may take an additional input from a probability model  1212 , both in the lossless encoder  1208  and in a lossless decoder  1214 , which predicts the probability of the next symbol to be encoded and decoded. The probability model  1212  may also be learned, for example it may be a neural network. At a decoder-side, the bitstream  1210  is input to the lossless decoder  1214 , such as an arithmetic decoder, whose output is input to a neural network decoder  1216 . The output of the neural network decoder  1216  is a decoded data for machines  1218 , that may be input to one or more task-NNs, a task-NN  1220  for object detection, a task-NN  1222  for object segmentation, a task-NN  1224  for object tracking, and a non-specified one, a task-NN  1226  for performing task X. 
       FIG.  13    illustrates an example of how the end-to-end learned system may be trained, in accordance with an embodiment. For the sake of simplicity, only one task-NN is illustrated. However, it may be understood that multiple task-NNs may be similarly used in the training process. A rate loss  1302  may be computed  1304  from the output of a probability model  1306 . The rate loss  1302  provides an approximation of the bitrate required to encode the input video data, for example, by a neural network encoder  1308 . A task loss  1310  may be computed  1312  from a task output  1314  of a task-NN  1316 . 
     The rate loss  1302  and the task loss  1310  may then be used to train  1318  the neural networks used in the system, such as the neural network encoder  1308 , probability model, a neural network decoder  1320 . Training may be performed by first computing gradients of each loss with respect to the trainable parameters of the neural networks that are contributing or affecting the computation of that loss. The gradients are then used by an optimization method, such as Adam, for updating the trainable parameters of the neural networks. It is to be understood that, in alternative or in addition to one or more task losses and/or one or more rate losses, the training process may use additional losses which may not be directly related to one or more specific tasks, such as losses derived from pixel-wise distortion metrics (for example, MSE, MS-SSIM). 
     The machine tasks may be performed at decoder side (instead of at encoder side) for multiple reasons, for example, the encoder-side device may not have the capabilities (e.g. computational, power, or memory) for running the neural networks that perform these tasks, or some aspects or the performance of the task neural networks may have changed or improved by the time that the decoder-side device needs the tasks results (e.g., different or additional semantic classes, better neural network architecture). Also, there may be a need for customization, where different clients may run different neural networks for performing these machine learning tasks. 
     Alternatively to an end-to-end trained codec, a video codec for machines may be realized by using a traditional codec such as H.266/VVC. 
     Alternatively, as described already above for the case of video coding for humans, another possible design may comprise using a traditional codec such as H.266/VVC, which includes one or more neural networks. In one example implementation, the one or more neural networks may replace one or more of the components of the traditional codec, for example:
         One or more in-loop filters;   One or more intra-prediction modes;   One or more inter-prediction modes;   One or more transforms;   One or more inverse transforms;   One or more probability models, for lossless coding; or   One or more post-processing filters.       

     In another example implementation, the one or more neural networks may function as an additional component, for example:
         One or more additional in-loop filters;   One or more additional intra-prediction modes;   One or more additional inter-prediction modes;   One or more additional transforms;   One or more additional inverse transforms;   One or more additional probability models, for lossless coding; or   One or more additional post-processing filters.       

     Alternatively, another possible implementation may include using any codec architecture (such as a traditional codec, or a traditional codec which includes one or more neural networks, or an end-to-end learned codec), and having a post-processing neural network which adapts the output of the decoder so that the output can be analyzed more effectively by one or more machines or task neural networks. For example, the encoder and decoder may be conformant to the H.266/VVC standard, a post-processing neural network takes the output of the decoder, and the output of the post-processing neural network is then input to an object detection neural network. In this example, the object detection neural network is the machine or task neural network. 
       FIG.  14    illustrates an example codec architecture  1400 , in accordance with an embodiment. The codec architecture  1400  includes an encoder  1402 , a decoder  1404 , a post-processing filter  1406 , a set of task-NNs  1408 . The encoder  1402  and the decoder  1404  may represent a traditional image or video codec, such as a codec conformant with the VVC/H.266 standard, or may represent an end-to-end (E2E) learned image or video codec. The post-processing filter  1406  may be a neural network based filter. The set of task-NNs  1408  may be neural networks that perform tasks such as object detection, object segmentation, object tracking, and the like. 
     A decoder-side NN (or DSNN) may be defined to be one or more of the neural networks present at decoder side, for example:
         A decoder NN of an end-to-end (E2E) learned codec;   A NN that is used as part of the decoder (such as an in-loop filter), for either an E2E learned codec or a traditional codec such as VVC/H.266; or   A NN that is used as post-processing filter, for either an E2E learned codec or a traditional codec such as VVC/H.266.       

     A DSNN is usually trained during a development stage and subsequently deployed to the decoder-side devices (or, for the case of post-processing filter, to devices where post-processing is performed). 
     In order to further train (e.g. finetune) a NN based on the test input data (e.g., based on the image or video that needs to be encoded and decoded), ground-truth data or an approximation of the ground-truth data is required for computing the training loss. However, in the context of typical data compression use cases, ground-truth data is usually the original (uncompressed) data that is available at encoder-side and that is input to the encoder. Furthermore, in the more specific use case of VCM, a more useful ground-truth is the desired output of the one or more task neural networks (task-NNs) applied on the decoded (and eventually post-processed) data. 
     Various embodiments propose to perform a finetuning process at decoder-side, where a DSNN (e.g., one or more decoder-side NNs) is trained or finetuned. 
     In order to perform such finetuning, in an embodiment, approximated ground truth data, or simply (AGT) is used. The AGT may be made available to the decoder in different possible ways. Furthermore, the data to which the AGT corresponds may be referred to as finetuning-driving content (FDC). The FDC is the content that is the input to the DSNN during finetuning. The output of the DSNN when the input is FDC is then referred to as processed FDC. 
     A loss is computed based on the AGT and the output of one or more NNs (for example, the output of one or more task-NNs), where the input to the one or more task-NNs is the processed FDC or is derived from the processed FDC. 
     In one embodiment, some parts of the video (some frames and/or some regions) are encoded in higher quality (HQ content) than the rest of the video that is encoded in lower quality (LQ content). The HQ content may be used to obtain the AGT. In an implementation, according to an example embodiment, the FDC may be obtained at decoder-side based on applying one or more components at the decoder-side device that lower the quality of the HQ content so that the obtained lower quality version of the HQ content has similar quality as the LQ content, for example in terms of PSNR or other quality metric. In one example, the HQ content may comprise intra-coded frames, whereas the LQ content may comprise inter-coded frames. In another example, the HQ content may comprise inter-coded frames from temporal layer  0 , whereas the LQ content may comprise inter-coded frames from temporal layer  1 . One example implementation for obtaining the LQ content from the HQ content at decoder side may include encoding the decoded HQ content (e.g., an intra frame) by using a configuration of the encoder that results into a representation requiring a lower bitrate than the bitrate required by the encoded HQ content. The configuration may specify the use of a higher quantization parameter (QP). In another implementation, according to another embodiment, the FDC is obtained at decoder-side by having the encoder encode the FDC. 
     Yet another embodiment proposes to have the encoder encode the AGT and transfer the AGT to the decoder. 
     Preliminary Information and Assumptions 
     Various embodiments consider the case of compressing and decompressing data which is mainly consumed by machines. The decompressed data may also be consumed by humans, either at the same time or at different times with respect to when the machines consume the decompressed data. The codec may comprise multiple parts, where some parts are used for compressing or decompressing data for machine consumption, and some other parts are used for compressing or decompressing data for human consumption. 
     Various embodiments assume that an encoder-side device performs a compression or encoding operation by using an encoder. A decoder-side device performs decompression or decoding operation by using a decoder. The encoder-side device may also use some decoding operations, for example in a coding loop. The encoder-side device and the decoder-side device may be the same physical device, or different physical devices. 
     Various embodiments are not restricted to any specific type of data. However, for the sake of simplicity we will consider video data. By “video” we may mean one or more video frames, unless specified otherwise. Other example types of data are images, audio, speech, or text. 
     Machines may also be referred to as task neural networks, or task-NNs. An example of task-NN is an object detection neural network, performing object detection task. Another example is a semantic segmentation neural network, performing semantic segmentation. The input to a task-NN may be one or more video frames. The output of a task-NN may be referred to as a task result, or task output. An example of task result, for the case of an object detection task-NN, is a set of coordinates of one of more bounding boxes, representing the location and spatial extent of detected objects. Also, an object detection task-NN may output other data, such as the category or class of the detected objects, and/or a confidence value indicating an estimate of the probability that the bounding box and/or its class for a detected object is correct, and/or a set of values for each detected object representing estimates of probabilities that the detected object belongs to certain classes. An example of task result, for the case of a semantic segmentation task-NN, is a tensor of shape (K, H, W), where K may be the total number of semantic classes considered by the task-NN, H and W may be the height and width of the input video frame that was input to the task-NN. Each of the K matrices of size H×W may represent the segmentation of the corresponding class, for example, it may indicate whether each pixel of the input video frame belongs to the corresponding class or not. In case the number of video frames that are input to the task-NN is T, the output of the task-NN may be a tensor of shape (T, K, H, W). 
     In some embodiments, it is assumed that at least some of the task-NNs (e.g. machines) are models, such as neural networks, for which it is possible to compute gradients of their output with respect to their input. For example, when a task-NN is a differentiable parametric model (e.g., a neural network), gradients of the task-NN output with respect to the task-NN input may be computed by using the chain rule for differentiation in mathematics, such as by first computing gradients of the task-NN output with respect to the parameters of the last layer of the task-NN, then computing gradients with respect to the input of the last layer, then computing gradients with respect to the parameters of the second last layer of the task-NN, and so on. In the case of neural networks, backpropagation may be used to obtain the gradients of the output of a NN with respect to its input. 
     Some embodiments consider a baseline system which comprises at least one encoder, at least one decoder, at least one task-NN. The proposed embodiments build on top of this baseline system. In this regard,  FIG.  15    illustrates an example baseline system  1500 , in accordance with an embodiment. The baseline system  1500  includes an encoder  1502 , which may be any video encoder, such as a traditional encoder which is conformant with the H.266/VVC standard, or an encoder which combines a traditional encoder with one or more neural networks, or an end-to-end learned encoder (e.g., comprising mainly neural networks). 
     The baseline system also includes a decoder  1504 , which may be any video decoder, such a traditional decoder which is conformant with the H.266/VVC standard, or a decoder which combines a traditional decoder with one or more neural networks, or an end-to-end learned decoder (e.g., comprising mainly neural networks). 
     The baseline system further includes task-NN  1506 , which may be any task neural network performing an analysis task or a processing task. 
     In the baseline system  1500 , the input video is encoded into a bitstream  1503  by the encoder  1502 . The decoder  1504  decodes the bitstream  1503  into a decoded video  1505 . The decoded video is given as input to the task-NN  1506 . The task-NN  1506  outputs some analysis or processing results. The output of a task-NN may be interchangeably referred to as ‘task-NN&#39;s output’ (e.g. task-NN&#39;s output  1507 ), ‘output’ or as ‘result’. 
     Although at least some of the proposed embodiments may be applied to any DSNN, such as the decoder NN of an end-to-end learned video codec, or an in-loop filter, for the sake of simplicity various embodiments consider an example of a system that comprises a post-processing neural network representing the DSNN. The post-processing neural network may be part of the decoder, or may be external with respect to the decoder. The post-processing neural network may post-process the decoded video. The post-processed decoded video, or data derived from it, may then be input to the task-NNs. 
       FIG.  16    illustrates an example baseline system  1600 , in accordance with an embodiment. In addition to the baseline system of  1500 , the baseline system  1600  also includes a post-processing neural network (PP-NN) that is external with respect to the decoder. The PP-NN  1602  refers to the post-processing neural network, and a PP decoded video  1603  refers to the post-processed decoded video. In this example the DSNN includes a PP-NN. 
     The DSNN is assumed to have been initialized, for example, by pretraining its parameters, or by initializing its parameters based on one of the parameter initialization techniques known in the literature (e.g., a random initialization). The initialization may either be performed before the DSNN is deployed into the decoder-side device, or after. Furthermore, the DSNN may have been finetuned after the initialization. 
     Example Embodiment: Decoder-Side Finetuning for Improving Machine Task performance 
     This embodiment describes performing a finetuning process at decoder-side, where a DSNN (e.g., one or more decoder-side NNs) is trained or finetuned. Here, finetuning refers to a continuation of a previously-performed training, or to a continuation of a previously-performed finetuning, or to training an initialized DSNN. For simplicity, in the rest of the text, it is assumed that the DSNN is already trained, thus mainly the term finetuning is used. However, the embodiments are also applicable for the case where the DSNN was initialized without involving a training process (such as by random initialization), and thus the embodiments, in practice, will involve the process of training instead of finetuning. 
     In various embodiments, a ground truth data, or simply ground-truth, refers to data that correctly describes a content data (e.g., images or videos) with respect to a certain characteristic or task. For example, for an object detection task (for which an object detection task-NN may be available), ground-truth data is the correct (e.g., true) detections of objects, which may be represented in one of the several available formats, such as by specifying a bounding box and a class label for each supported object in the image or video, where a supported object refers to whether the class of an object is one of the recognized classes by the object detection task-NN. 
     In order to perform finetuning at decoder side, use of approximated ground truth data, or simply (AGT) is proposed. The AGT may be made available to the decoder in different possible ways as described below. 
     Furthermore, data to which the AGT corresponds (e.g., is associated with) is referred to as finetuning-driving content (FDC). A sample of FDC may be associated with one or more samples of AGT. Also, a sample of AGT may be associated with one or more samples of FDC. In one example implementation, the FDC is the content that is input to the DSNN during finetuning. In another example implementation, the content which is input to the DSNN may be derived from the FDC. When the input to the DSNN is either FDC or content derived from FDC, the output of the DSNN is referred to as processed FDC. 
     A loss proxy NN (LPNN) may be defined as a NN based on which a loss may be computed. In an example, a loss may be computed based on the output of a LPNN. The LPNN may be a task-NN (such as object detection NN, semantic segmentation NN, and the like), or a feature extractor (such as a visual geometry group (VGG) neural network architecture pretrained on ImageNet dataset, for which only a subset of layers may be used to extract features from the input data), where the feature extractor may be pretrained. 
     A loss is computed between the AGT and the output of one or more LPNNs (for example, the output of one or more task-NNs) that is computed based on the processed FDC. For example, cross-entropy loss may be computed between the AGT for an image classification task (the approximated correct classification label, for example represented as a one-hot vector) and the output of an image classification task-NN for which the input is an image that is output by a post-processing NN. 
     The finetuning may comprise performing one or more training iterations, where each iteration may comprise one or more of the following steps:
         Inputting the FDC or content derived from FDC to the DSNN;   Computing the output of the DSNN, i.e., the processed FDC;   Computing a loss based on the output of the DSNN and the AGT associated to the FDC (as described above);   Computing an update to the DSNN. For example by using backpropagation to compute gradients of the loss with respect to one or more parameters of the DSNN, and then applying one optimizer routine such as stochastic gradient descent (SGD) to compute an update of the one or more parameters of the DSNN; or   Applying the computed update to the DSNN.       

     The finetuning process may continue until a stopping criterion is met, such as a predefined number of iterations was performed, or a target loss was achieved, and the like. 
     Another Example Embodiment: Obtain AGT from High Quality and Low Quality Contents 
     This embodiment, proposes to encode one or more parts of the video (one or more frames and/or one or more regions) in higher quality (HQ content) than the rest of the video that is encoded in lower quality (LQ content). The HQ content is used to obtain the AGT. In an implementation, according to an example embodiment, the FDC is obtained at decoder-side by having the decoder-side device lower the quality of the HQ content, so that the obtained FDC has similar quality as the LQ content, as measured for example by PSNR. In a another implementation, according to another example embodiment, the FDC is obtained at decoder-side by having the encoder encode the FDC, which may be a lower quality version of the HQ content or may be other lower quality content. In this another implementation, the encoder may encode both the LQ and HQ version of the same original (uncompressed) content, or may encode LQ content and HQ content which are related or similar to each other based on one or more similarity criterion. For example, two similar or related LQ and HQ contents may be regions comprising objects of the same class or category. The AGT may then be computed as the output of the LPNN when the input is HQ content, or a version derived from the HQ content. 
     Regarding the implementation option, where the FDC is obtained at decoder-side by having the decoder-side device lower the quality of the HQ content, the lowering of the quality may include one or more of the following methods, that are performed at decoder-side:
         Encoding the HQ content by using an encoding configuration that causes the decoded content to be lower quality with respect to the HQ content, for example by using a higher quantization parameter (QP) than what was used to encode the HQ content. Then, decoding the content;   Downsampling the HQ content to a lower resolution. (Optionally, an upsampling process may be applied to bring back the content to the original resolution); or   Inputting the HQ content to a NN, and treating the output of the NN as the lower quality content. The NN may be trained to output content that has similar quality as the LQ content.       

       FIG.  17    illustrates an example of implementation, in accordance with an embodiment. This embodiment considers the implementation in which the FDC is obtained at decoder-side by having the decoder-side device lower the quality of the HQ content. The proposed process may be split into two phases: phase  1   1702  comprises finetuning of the DSNN, phase  2   1704  includes inference of the finetuned DSNN. A frame  1706  and a frame  1708 , which may be two frames of a video sequence, are input to an encoder  1710 . The encoder  1710  encodes the frame  1706  in high quality and encodes the frame  1708  in low quality. The output of the encoder  1710  is a bitstream representing the encoded versions of the frame  1706  and the frame  1708 . The bitstream is input to a decoder  1712 , which outputs decoded frames. A ‘decoded frame  1 , HQ’  1714  in the figure refers to the decoded version of the frame  1706  and it represents the HQ content. A ‘decoded frame  2 , LQ’  1716  in the figure refers to the decoded version of the frame  1708  and it represents the LQ content. The HQ content is input to a task-NN  1718 , and the output of the task-NN  1720  represents the AGT. The HQ content is also input to a process that downgrades  1722  its quality, for example by re-encoding, and/or by downsampling, and the like. The output of the downgrading process is a ‘decoded frame  1 , LQ’  1724 , which represents the lower quality version of the HQ content, thus it represents the FDC. The FDC is input to a post-processing NN filter  1726 , representing a DSNN. An output of the post-processing NN filter  1726  is input to a task-NN  1728 . A loss is computed  1732  based on the AGT and the output  1730  of the task-NN  1728  when the input is the output of the post-processing NN filter  1726 . A loss computed by the loss computation process  1732  is used to finetune  1734  the post-processing NN filter  1726 . The result of the finetuning  1734  may be the finetuned post-processing filter  1736 . The ‘decoded frame  2 , LQ’  1716  is input to the finetuned post-processing filter  1736 . An output of the finetuned post-processing filter  1736  is input to a task-NN  1738 . The task-NN  1738  provides output  1740  as, for example, object classifications. The task-NN  1718 , the task-NN  1728 , and the task-NN  1738  may be the same task-NN, or instances of the same task-NN, or different architectures of task-NNs that perform substantially the same task, for example object classification. 
       FIG.  18    illustrates another example implementation, in accordance with an embodiment. This embodiment considers the implementation in which the FDC is obtained at decoder-side by having the decoder-side device lower the quality of the HQ content. At encoder side, an input frame  1802  is input to a segmentation neural network  1804  which outputs a frame  1806  and annotations about segmented regions. For example, the annotations may comprise information about the location of the regions and their class or category (e.g., a region  1 , HQ  1808  and a region  2 , LQ  1809 ). The frame  1806  and the associated annotations are input to an encoder  1810 , which encodes some segmented regions in higher quality (HQ regions) and other segmented regions in lower quality (LQ regions). In particular, at least some of the LQ regions may be required to belong to the same class or category as at least some of the HQ regions. The output of the encoder is a bitstream representing the encoded frame and information about the locations, classes and quality levels of at least some of the segmented regions. The bitstream is decoded by a decoder  1811  into a frame comprising some HQ regions (e.g., a ‘decoded region  1 , HQ’  1812 ), and some LQ regions (a decoded region  2 , LQ’  1814 ). The HQ regions are input to a process  1815  that downgrades the quality of these regions, thus obtaining lower quality versions of the HQ regions (e.g., a decoded region  1 , LQ  1816 ). The lower quality versions of the HQ regions, representing the FDC  1816 , are input to a post-processing NN filter  1818 , representing a DSNN. Both the decoded HQ region  1812  and the output of the post-processing NN filter  1818  are input to a task-NN  1828 . An output  1822  of a task-NN  1820  when the input is the HQ regions represents the AGT. A loss is computed  1824  based on the AGT  1822  and the output  1826  of the task-NN  1828  when the input is the processed FDC (e.g., the output of the post-processing NN filter  1818 ). The loss computed by the loss computation process  1824  is used for finetuning  1830  the DSNN. The result of the finetuning  1830  may be the finetuned DSNN  1832 . The finetuned DSNN  1832  may be used for processing the LQ regions (e.g., the decoded region  2 , LQ  1814 ). The processed LQ regions may then be input to a task-NN  1834 . The task-NN  1934  provides output  1936  as, for example, object classifications. The task-NN  1820 , the task-NN  1828  and the task-NN  1834  may be the same task-NN, or instances of the same task-NN, or different architectures of task-NNs that perform substantially the same task, for example object classification. 
       FIG.  19    illustrates example implementation, in accordance with another embodiment. This embodiment considers the implementation in which the FDC is obtained at decoder-side by having the encoder encode the FDC, which is low quality content that is related or similar to the HQ content based on some similarity criterion. An input frame  1902  is input to a segmentation neural network  1904  which outputs a frame  1905  and annotations about the segmented regions. In particular, the annotations may comprise information about the location of the regions and their class or category (e.g., a region  1 , HQ  1906 , a region  2 , LQ  1908 , and region  2 , LQ  1909 ). The frame and the associated annotations are input to an encoder  1910 , which encodes some segmented regions in higher quality (HQ regions) and other segmented regions in lower quality (LQ regions). In particular, at least some of the LQ regions may be required to belong to the same class or category as at least some of the HQ regions. The output of the encoder is a bitstream representing the encoded frame and information about the segmented regions and quality level of the segmented regions. The bitstream is decoded by the decoder  1911  into a frame including some HQ regions (e.g., a ‘decoded region  1 , HQ’  1912 ), some LQ regions (e.g., a ‘decoded region  2 , LQ (FDC)’  1914  and a ‘decoded region  3 , LQ’  1916 ). The LQ region  1914  represent the FDC. The FDC is input to a post-processing NN filter  1918 , representing the DSNN. The output of the post-processing NN filter  1918  represents the processed FDC. The HQ regions are input to a task-NN  1922 . The processed FDC is input to a task-NN  1928 . The output  1920  of the task-NN  1922  when the input is the HQ regions (e.g. the decoded region  1 , HQ  1912 ) represents the AGT. A loss is computed  1924  based on the AGT  1920  and an output  1926  of the task-NN  1928  when the input is the output of the post-processing NN filter  1918 . The loss computed by the loss computation process  1924  is used for finetuning  1930  the DSNN  1918 . The result of the finetuning  1930  is may be the finetuned DSNN  1932 . The finetuned DSNN  1932  may be used for processing at least some of the LQ regions (e.g., the decoded region  3 , LQ  1916 ). The processed LQ regions may then be input to the task-NN  1934 . The task-NN  1934  provides the output  1936  as, for example, object classifications. The task-NN  1922 , the task-NN  1928  and the task-NN  1934  may be the same task-NN, or instances of the same task-NN, or different architectures of task-NNs that perform substantially the same task, for example object classification. 
     Furthermore, referring to the architecture of a ‘feature-domain residual video coding’ as disclosed in U.S. application Ser. No. 17/218,967, which is incorporated herein by this reference, this embodiment can be applied as follows. In the feature-domain residual video coding, there may be following two branches:
         Human branch: VVC codec. Used to encode frames in low quality. VVC codec is an example, and any other codec that is capable of encoding at low bitrate or low quality may be used instead.   Machine enhancement branch: E2E learned codec. Used to encode a feature-enhancement (residual), to improve task-NN results. E2E learned codec is an example, and any other codec that is capable of encoding a feature-enhancement may be used instead.       

     As the LQ content is already available at decoder side from the human branch, there is no need to obtain it explicitly (either by sending it from encoder to decoder, or by lowering the quality of the HQ content). In fact, the LQ content is the features extracted from the reconstructed image at the human branch. 
     The HQ content is the enhanced features. The pair of LQ content (from human branch) and HQ content (from human+machine branches) is used to compute the loss and finetune the DSNN. The DSNN may be a post-filter for filtering the enhanced features, or the decoder NN of the machine branch. 
     The feature residual codec may be regarded as a codec for multi-layer scalability where a first layer may be provided a low-quality representation of the video and a second layer may provide an enhancement over the first layer. In addition to the feature residual codec, embodiments apply similarly to any other codec with multi-layer scalability. In an embodiment utilizing multi-layer scalability, LQ decoded image(s) are obtained by decoding the first layer and respective HQ decoded image(s) are obtained by decoding the second layer. The decoding of the second layer may depend on the first layer, e.g. in a manner of utilizing inter-layer prediction. 
     As another example, use of spatio-temporally consistent patches is proposed. For example, a content-aware patch selection is employed, from the 1st frame a high-resolution patch is extracted and from the 2nd frame a low-quality patch of the same content is employed. The patch selection code can be done with an algorithm. Examples of such algorithms could be object detection and tracking, correlation-based patch-match, instance segmentation, and of the like. The algorithm may also use the motion information for the frames to extract the spatio-temporally consistent patches. 
     In an embodiment, when a traditional video codec such as HEVC or VVC is used, for a certain frame in the compressed video at certain temporal ID, the lower temporal ID reference frames may be referred to as HQ content and/or higher temporal layer ID frames may be referred to as LQ content for the training and/or finetuning stages. Alternatively, patches from the frames in different temporal IDs may be used as the HQ content and LQ content. The location of the patches may be signaled form the encoder to the decoder, or determined using the aforementioned spatial-temporally consistent patch extraction algorithm. 
     Yet Another Embodiment 
       FIG.  20    illustrates an example implementation in which the encoder encodes the AGT and provides the AGT to the decoder, in accordance with an embodiment. This embodiment proposes that the encoder encodes the AGT, and provides the AGT to the decoder in the form of bitstream, for example, as part of the bitstream representing the encoded content, or as an out-of-band bitstream. 
     The encoder side runs a LPNN ‘Task-NN (object classification)’  2002  for one or more input frames, which may be uncompressed frames, referred to as ‘frame  1 , original’  2004 . An output  2006  of the LPNN  2002  represents the AGT. The AGT, the frame from which AGT was determined, information about the association between the AGT and the frame from which AGT was determined, and other frames (e.g., a ‘Frame  2 , original’  2008 ) are encoded by the encoder  2010 . The encoded AGT may be encoded as metadata, for example, as the annotated regions SEI message. The bitstream output by the encoder may be decoded by a decoder  2012 . The output of the decoder may include a ‘decoded frame  1 , LQ’ (or FDC)  2014  and a ‘decoded frame  2 , LQ’ (or LQ content)  2016 , where the ‘decoded frame  1 , LQ’  2014  is the frame based on which the AGT was determined at encoder side. The FDC is input to a post-processing NN  2018  representing a DSNN. The output  2024  of the DSNN is an input to a task-NN  2020 . A loss is computed  2022  based on the output  2024  of the task-NN and the AGT  2006 . The loss computed by the loss computation process  2022  is used to finetune  2026  the DSNN  2018 . The result of the finetuning  2026  may be a finetuned DSNN  2028 . The finetuned DSNN  2028  is used to process the LQ content  2016 . The processed LQ content may be input to a task-NN  2030 . The task-NN  2030  provides output  2032 , for example, object classifications. When LPNN  2002  at the encoder side is a task-NN, it need not be the same as the task-NN  2020  or the task-NN  2030  in the decoder side (as long as they are for the same purpose). For example, these task-NNs may perform the task of object detection but have different architectures or different number of layers or parameters. For example, the task-NN at encoder side may be an object detection task-NN with less layers than the object detection task-NN at decoder-side. 
     In another embodiment, both the FDC and AGT may be included in the bitstream for the DSNN finetuning. The FDC may be data generated by the encoder, which may not be directly related to the data to be coded. The generated FDC and the corresponding AGT may be more efficient in finetuning the DSNN with regard to the data to processed. 
     Additional/Alternative Embodiments 
     When the LPNN is a feature extraction NN, such as a pretrained VGG or a pretrained ResNet backbone, features may be extracted from the HQ content and from the LQ content. A distortion or distance between the two sets of features is computed. The computed distortion or distance is used as the loss for finetuning the DSNN. In an additional embodiment, the encoder may encode a weight-update or updated weights. At decoder side, the encoded weight-update or updated weights may first be decoded, then the decoded weight-update or decoded updated weights may be used to update the LPNN. For example, the weight-update may comprise updates for the bias terms of the proxy NN. 
     Training hyper-parameters for finetuning, for example, learning rate, number of iterations, parameters to be updated, can be signaled from the encoder to the encoder. In this case, the encoder would perform the finetuning (e.g., by using either the task NN, or a related task NN, or a proxy NN), and find the optimal hyper-parameters based on the uncompressed data. 
     After the DSNN is finetuned, the new model can be stored in a buffer. Later, for subsequent regions or frames to be decoded and eventual postprocessed, the encoder can signal whether the model needs to be updated or whether one of the previously updated models can be re-used. In the latter case, the encoder would also signal the ID or index of the optimal post-processing filter that is available in the buffer. 
     When a DSNN contains discrete values, for example, integer weights, in the parameters to be finetuned, the discrete parameters may be replaced by continuous parameters followed by quantization operations. The continuous parameters may be initialized to the integer values before the finetuning. To efficiently finetune the discrete parameters, an example implementation for performing finetuning is described below:
         Start optimizing the loss function with regard to the continuous parameters using a large learning rate.   Record the best performing discrete parameters values during the optimization.   When the loss stops decreasing for a certain number of iterations, rewind the continuous parameters to the best performing discrete values and continue the optimization with a lower learning rate.   Repeat this procedure until a predefined minimal learning rate is reached.       

     This optimization procedure may be used to finetune other components or data in the codec. For example, the procedure may be used by the encoder of an end-to-end learned codec to finetune the latent representation output by the encoder NN. In this case, the loss function is the RD-loss function of the codec, and the discrete parameters are the discrete latent representation. 
       FIG.  21    is an example apparatus  2100 , which may be implemented in hardware, configured to implement mechanisms for training or finetuning a decoder side neural network, based on the examples described herein. The apparatus  2100  comprises at least one processor  2102 , at least one non-transitory memory  2104  including computer program code  2105 , wherein the at least one memory  2104  and the computer program code  2105  are configured to, with the at least one processor  2102 , cause the apparatus  2100  to implement mechanisms for training or finetuning decoder side neural network  2106 , based on the examples described herein. In an embodiment, the at least one neural network or the portion of the at least one neural network may be used at a decoder-side for decoding or reconstructing one or more media items. 
     The apparatus  2100  optionally includes a display  2108  that may be used to display content during rendering. The apparatus  2100  optionally includes one or more network (NW) interfaces (I/F(s))  2110 . The NW I/F(s)  2110  may be wired and/or wireless and communicate over the Internet/other network(s) via any communication technique. The NW I/F(s)  2110  may comprise one or more transmitters and one or more receivers. The N/W I/F(s)  2110  may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder/decoder circuitry(ies) and one or more antennas. 
     The apparatus  2100  may be a remote, virtual or cloud apparatus. The apparatus  2100  may be either a coder or a decoder, or both a coder and a decoder. The at least one memory  2104  may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The at least one memory  2104  may comprise a database for storing data. The apparatus  2100  need not comprise each of the features mentioned, or may comprise other features as well. The apparatus  2100  may correspond to or be another embodiment of the apparatus  50  shown in  FIG.  1    and  FIG.  2   , or any of the apparatuses shown in  FIG.  3   . The apparatus  2100  may correspond to or be another embodiment of the apparatuses shown in  FIG.  23   , including UE  110 , RAN node  170 , or network element(s)  190 . 
       FIG.  22    illustrates an example method  2200  for training or finetuning a decoder side neural network, in accordance with an embodiment. As shown in block  2106  of  FIG.  22   , the apparatus  2100  includes means, such as the processing circuitry  2102  or the like, for implementing mechanisms for training or finetuning a decoder side neural. At  2202 , the method  2200  includes providing a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN). At  2204 , the method  2200  includes computing an output of the DSNN as a processed FDC. At  2206 , the method  2200  includes computing a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC. At  2208 , the method  2200  includes computing an update to the DSNN. At  2210 , the method  2200  includes applying the computed update to the DSNN 
     Referring to  FIG.  23   , this figure shows a block diagram of one possible and non-limiting example in which the examples may be practiced. A user equipment (UE)  110 , radio access network (RAN) node  170 , and network element(s)  190  are illustrated. In the example of  FIG.  1   , the user equipment (UE)  110  is in wireless communication with a wireless network  100 . A UE is a wireless device that can access the wireless network  100 . The UE  110  includes one or more processors  120 , one or more memories  125 , and one or more transceivers  130  interconnected through one or more buses  127 . Each of the one or more transceivers  130  includes a receiver, Rx,  132  and a transmitter, Tx,  133 . The one or more buses  127  may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers  130  are connected to one or more antennas  128 . The one or more memories  125  include computer program code  123 . The UE  110  includes a module  140 , comprising one of or both parts  140 - 1  and/or  140 - 2 , which may be implemented in a number of ways. The module  140  may be implemented in hardware as module  140 - 1 , such as being implemented as part of the one or more processors  120 . The module  140 - 1  may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module  140  may be implemented as module  140 - 2 , which is implemented as computer program code  123  and is executed by the one or more processors  120 . For instance, the one or more memories  125  and the computer program code  123  may be configured to, with the one or more processors  120 , cause the user equipment  110  to perform one or more of the operations as described herein. The UE  110  communicates with RAN node  170  via a wireless link  111 . 
     The RAN node  170  in this example is a base station that provides access by wireless devices such as the UE  110  to the wireless network  100 . The RAN node  170  may be, for example, a base station for 5G, also called New Radio (NR). In 5G, the RAN node  170  may be a NG-RAN node, which is defined as either a gNB or an ng-eNB. A gNB is a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to a 5GC (such as, for example, the network element(s)  190 ). The ng-eNB is a node providing E-UTRA user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GC. The NG-RAN node may include multiple gNBs, which may also include a central unit (CU) (gNB-CU)  196  and distributed unit(s) (DUs) (gNB-DUs), of which DU  195  is shown. Note that the DU may include or be coupled to and control a radio unit (RU). The gNB-CU is a logical node hosting radio resource control (RRC), SDAP and PDCP protocols of the gNB or RRC and PDCP protocols of the en-gNB that controls the operation of one or more gNB-DUs. The gNB-CU terminates the F 1  interface connected with the gNB-DU. The F 1  interface is illustrated as reference  198 , although reference  198  also illustrates a link between remote elements of the RAN node  170  and centralized elements of the RAN node  170 , such as between the gNB-CU  196  and the gNB-DU  195 . The gNB-DU is a logical node hosting RLC, MAC and PHY layers of the gNB or en-gNB, and its operation is partly controlled by gNB-CU. One gNB-CU supports one or multiple cells. One cell is supported by only one gNB-DU. The gNB-DU terminates the F 1  interface  198  connected with the gNB-CU. Note that the DU  195  is considered to include the transceiver  160 , for example, as part of a RU, but some examples of this may have the transceiver  160  as part of a separate RU, for example, under control of and connected to the DU  195 . The RAN node  170  may also be an eNB (evolved NodeB) base station, for LTE (long term evolution), or any other suitable base station or node. 
     The RAN node  170  includes one or more processors  152 , one or more memories  155 , one or more network interfaces (N/W I/F(s))  161 , and one or more transceivers  160  interconnected through one or more buses  157 . Each of the one or more transceivers  160  includes a receiver, Rx,  162  and a transmitter, Tx,  163 . The one or more transceivers  160  are connected to one or more antennas  158 . The one or more memories  155  include computer program code  153 . The CU  196  may include the processor(s)  152 , memories  155 , and network interfaces  161 . Note that the DU  195  may also contain its own memory/memories and processor(s), and/or other hardware, but these are not shown. 
     The RAN node  170  includes a module  150 , comprising one of or both parts  150 - 1  and/or  150 - 2 , which may be implemented in a number of ways. The module  150  may be implemented in hardware as module  150 - 1 , such as being implemented as part of the one or more processors  152 . The module  150 - 1  may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module  150  may be implemented as module  150 - 2 , which is implemented as computer program code  153  and is executed by the one or more processors  152 . For instance, the one or more memories  155  and the computer program code  153  are configured to, with the one or more processors  152 , cause the RAN node  170  to perform one or more of the operations as described herein. Note that the functionality of the module  150  may be distributed, such as being distributed between the DU  195  and the CU  196 , or be implemented solely in the DU  195 . 
     The one or more network interfaces  161  communicate over a network such as via the links  176  and  131 . Two or more gNBs  170  may communicate using, for example, link  176 . The link  176  may be wired or wireless or both and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interface for other standards. 
     The one or more buses  157  may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers  160  may be implemented as a remote radio head (RRH) 195 for LTE or a distributed unit (DU)  195  for gNB implementation for 5G, with the other elements of the RAN node  170  possibly being physically in a different location from the RRH/DU, and the one or more buses  157  could be implemented in part as, for example, fiber optic cable or other suitable network connection to connect the other elements (for example, a central unit (CU), gNB-CU) of the RAN node  170  to the RRH/DU  195 . Reference  198  also indicates those suitable network link(s). 
     It is noted that description herein indicates that ‘cells’ perform functions, but it should be clear that equipment which forms the cell may perform the functions. The cell makes up part of a base station. That is, there can be multiple cells per base station. For example, there could be three cells for a single carrier frequency and associated bandwidth, each cell covering one-third of a 360 degree area so that the single base station&#39;s coverage area covers an approximate oval or circle. Furthermore, each cell can correspond to a single carrier and a base station may use multiple carriers. So when there are three 120 degree cells per carrier and two carriers, then the base station has a total of 6 cells. 
     The wireless network  100  may include a network element or elements  190  that may include core network functionality, and which provides connectivity via a link or links  181  with a further network, such as a telephone network and/or a data communications network (for example, the Internet). Such core network functionality for 5G may include access and mobility management function(s) (AMF(S)) and/or user plane functions (UPF(s)) and/or session management function(s) (SMF(s)). Such core network functionality for LTE may include MME (Mobility Management Entity)/SGW (Serving Gateway) functionality. These are merely example functions that may be supported by the network element(s)  190 , and note that both 5G and LTE functions might be supported. The RAN node  170  is coupled via a link  131  to the network element  190 . The link  131  may be implemented as, for example, an NG interface for 5G, or an S 1  interface for LTE, or other suitable interface for other standards. The network element  190  includes one or more processors  175 , one or more memories  171 , and one or more network interfaces (N/W I/F(s))  180 , interconnected through one or more buses  185 . The one or more memories  171  include computer program code  173 . The one or more memories  171  and the computer program code  173  are configured to, with the one or more processors  175 , cause the network element  190  to perform one or more operations. 
     The wireless network  100  may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors  152  or  175  and memories  155  and  171 , and also such virtualized entities create technical effects. 
     The computer readable memories  125 ,  155 , and  171  may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories  125 ,  155 , and  171  may be means for performing storage functions. The processors  120 ,  152 , and  175  may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors  120 ,  152 , and  175  may be means for performing functions, such as controlling the UE  110 , RAN node  170 , network element(s)  190 , and other functions as described herein. 
     In general, the various embodiments of the user equipment  110  can include, but are not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions. 
     One or more of modules  140 - 1 ,  140 - 2 ,  150 - 1 , and  150 - 2  may be configured to implement mechanisms for training or finetuning a decoder side neural network. Computer program code  173  may also be configured to implement mechanisms training or finetuning a decoder side neural network. 
     As described above,  FIG.  22    include a flowcharts of an apparatus (e.g.  50 ,  100 ,  602 ,  604 ,  700 ,  2100 , or  2300 ), method, and computer program product according to certain example embodiments. It will be understood that each block of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by various means, such as hardware, firmware, processor, circuitry, and/or other devices associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by a memory (e.g.  58 ,  125 ,  704 ,  21 , or  2304 ) of an apparatus employing an embodiment of the present invention and executed by processing circuitry (e.g.  56 ,  120 ,  702 ,  21 , or  2302 ) of the apparatus. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (e.g., hardware) to produce a machine, such that the resulting computer or other programmable apparatus implements the functions specified in the flowchart blocks. These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture, the execution of which implements the function specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart blocks. 
     A computer program product is therefore defined in those instances in which the computer program instructions, such as computer-readable program code portions, are stored by at least one non-transitory computer-readable storage medium with the computer program instructions, such as the computer-readable program code portions, being configured, upon execution, to perform the functions described above, such as in conjunction with the flowchart(s) of  FIG.  22   . In other embodiments, the computer program instructions, such as the computer-readable program code portions, need not be stored or otherwise embodied by a non-transitory computer-readable storage medium, but may, instead, be embodied by a transitory medium with the computer program instructions, such as the computer-readable program code portions, still being configured, upon execution, to perform the functions described above. 
     Accordingly, blocks of the flowcharts support combinations of means for performing the specified functions and combinations of operations for performing the specified functions for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions. 
     In some embodiments, certain ones of the operations above may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the operations above may be performed in any order and in any combination. 
     In the above, some example embodiments have been described with reference to an SEI message or an SEI NAL unit. It needs to be understood, however, that embodiments can be similarly realized with any similar structures or data units. Where example embodiments have been described with SEI messages contained in a structure, any independently parsable structures could likewise be used in embodiments. Specific SEI NAL unit and a SEI message syntax structures have been presented in example embodiments, but it needs to be understood that embodiments generally apply to any syntax structures with a similar intent as SEI NAL units and/or SEI messages. 
     In the above, some embodiments have been described in relation to a particular type of a parameter set (namely adaptation parameter set). It needs to be understood, however, that embodiments could be realized with any type of parameter set or other syntax structure in the bitstream. 
     In the above, some example embodiments have been described with the help of syntax of the bitstream. It needs to be understood, however, that the corresponding structure and/or computer program may reside at the encoder for generating the bitstream and/or at the decoder for decoding the bitstream. 
     In the above, where example embodiments have been described with reference to an encoder, it needs to be understood that the resulting bitstream and the decoder have corresponding elements in them. Likewise, where example embodiments have been described with reference to a decoder, it needs to be understood that the encoder has structure and/or computer program for generating the bitstream to be decoded by the decoder. 
     Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation. 
     It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims. 
     References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single/multi-processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, and the like. 
     As used herein, the term ‘circuitry’ may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and/or digital circuitry, and (b) combinations of circuits and software (and/or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s)/software including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even when the software or firmware is not physically present. This description of ‘circuitry’ applies to uses of this term in this application. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and/or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device.