Patent Publication Number: US-2022230362-A1

Title: Method and apparatus for adaptive neural image compression with rate control by meta-learning

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is based on and claims priority to U.S. Provisional Patent Application No. 63/139,156, filed on Jan. 19, 2021, the disclosure of which is incorporated by reference herein in its entirety. 
    
    
     BACKGROUND 
     ISO/IEC MPEG (JTC 1/SC 29/WG 11) has been actively searching for potential needs for standardization of future video coding technology. ISO/IEC JPEG has established the JPEG-AI group focusing on AI-based end-to-end Neural Image Compression (NIC) using Neural Networks (NN). The success of recent approaches has brought more and more industrial interests in advanced neural image and video compression methodologies. 
     Although prior arts have shown promising performance, flexible bitrate control remains a challenging issue for previous NIC methods. Conventionally, it may require training multiple model instances targeting each desired trade-off between a rate and a distortion (a quality of compressed images) individually. All these multiple model instances may be stored and deployed on a decoder side to reconstruct images from different bitrates. Also, these model instances cannot give arbitrary smooth bitrate control, because it is difficult to train and store an infinite number of model instances for every possible target bitrate. Prior arts have studied multi-rate NIC in which one model instance is trained to achieve compression of multiple pre-defined bitrates. However, arbitrary smooth bitrate control remains an unexplored open issue. 
     SUMMARY 
     According to embodiments, a method of adaptive neural image compression with rate control by meta-learning is performed by at least one processor and includes receiving an input image and a hyperparameter, and encoding the received input image, based on the received hyperparameter, using an encoding neural network, to generate a compressed representation. The encoding includes performing a first shared encoding on the received input image, using a first shared encoding layer having first shared encoding parameters, performing a first adaptive encoding on the received input image, using a first adaptive encoding layer having first adaptive encoding parameters, combining the first shared encoded input image and the first adaptive encoded input image, to generate a first combined output, and performing a second shared encoding on the first combined output, using a second shared encoding layer having second shared encoding parameters. 
     According to embodiments, an apparatus for adaptive neural image compression with rate control by meta-learning includes at least one memory configured to store program code, and at least one processor configured to read the program code and operate as instructed by the program code. The program code includes first receiving code configured to cause the at least one processor to receiving an input image and a hyperparameter, and encoding code configured to cause the at least one processor to encode the received input image, based on the received hyperparameter, using an encoding neural network, to generate a compressed representation. The encoding code is further configured to cause the at least one processor to perform a first shared encoding on the received input image, using a first shared encoding layer having first shared encoding parameters, perform a first adaptive encoding on the received input image, using a first adaptive encoding layer having first adaptive encoding parameters, combine the first shared encoded input image and the first adaptive encoded input image, to generate a first combined output, and perform a second shared encoding on the first combined output, using a second shared encoding layer having second shared encoding parameters. 
     According to embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor for adaptive neural image compression with rate control by meta-learning, cause the at least one processor to receive an input image and a hyperparameter, and encode the received input image, based on the received hyperparameter, using an encoding neural network, to generate a compressed representation. The instructions, when executed by the at least one processor, further cause the at least one processor to perform a first shared encoding on the received input image, using a first shared encoding layer having first shared encoding parameters, perform a first adaptive encoding on the received input image, using a first adaptive encoding layer having first adaptive encoding parameters, combine the first shared encoded input image and the first adaptive encoded input image, to generate a first combined output, and perform a second shared encoding on the first combined output, using a second shared encoding layer having second shared encoding parameters. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a diagram of an environment in which methods, apparatuses and systems described herein may be implemented, according to embodiments. 
         FIG. 2  is a block diagram of example components of one or more devices of  FIG. 1 . 
         FIGS. 3A and 3B  are block diagrams of meta-NIC architectures for adaptive neural image compression with rate control by meta-learning, according to embodiments. 
         FIG. 4A  is a block diagram of an apparatus for adaptive neural image compression with rate control by meta-learning, during a test stage, according to embodiments. 
         FIG. 4B  is a block diagram of a meta-NIC encoder of the apparatus shown in  FIG. 4A . 
         FIG. 4C  is a block diagram of a meta-NIC decoder of the apparatus shown in  FIG. 4A . 
         FIG. 5  is a block diagram of a training apparatus for adaptive neural image compression with rate control by meta-learning, during a training stage, according to embodiments. 
         FIG. 6  is a flowchart of a method of adaptive neural image compression with rate control by meta-learning, according to embodiments. 
         FIG. 7  is a block diagram of an apparatus for adaptive neural image compression with rate control by meta-learning, according to embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     The disclosure describes methods and apparatuses for a Meta Neural Image Compression (meta-NIC) framework for smooth bitrate control in NIC. A meta learning mechanism is used to adaptively compute rate-adaptive weight parameters of an underlying NIC model based on a current input image and a target bitrate, so that a single meta-NIC model instance can achieve image compression with arbitrary smooth different bitrates. 
       FIG. 1  is a diagram of an environment  100  in which methods, apparatuses and systems described herein may be implemented, according to embodiments. 
     As shown in  FIG. 1 , the environment  100  may include a user device  110 , a platform  120 , and a network  130 . Devices of the environment  100  may interconnect via wired connections, wireless connections, or a combination of wired and wireless connections. 
     The user device  110  includes one or more devices capable of receiving, generating, storing, processing, and/or providing information associated with platform  120 . For example, the user device  110  may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a radiotelephone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or a similar device. In some implementations, the user device  110  may receive information from and/or transmit information to the platform  120 . 
     The platform  120  includes one or more devices as described elsewhere herein. In some implementations, the platform  120  may include a cloud server or a group of cloud servers. In some implementations, the platform  120  may be designed to be modular such that software components may be swapped in or out. As such, the platform  120  may be easily and/or quickly reconfigured for different uses. 
     In some implementations, as shown, the platform  120  may be hosted in a cloud computing environment  122 . Notably, while implementations described herein describe the platform  120  as being hosted in the cloud computing environment  122 , in some implementations, the platform  120  may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based. 
     The cloud computing environment  122  includes an environment that hosts the platform  120 . The cloud computing environment  122  may provide computation, software, data access, storage, etc. services that do not require end-user (e.g., the user device  110 ) knowledge of a physical location and configuration of system(s) and/or device(s) that hosts the platform  120 . As shown, the cloud computing environment  122  may include a group of computing resources  124  (referred to collectively as “computing resources  124 ” and individually as “computing resource  124 ”). 
     The computing resource  124  includes one or more personal computers, workstation computers, server devices, or other types of computation and/or communication devices. In some implementations, the computing resource  124  may host the platform  120 . The cloud resources may include compute instances executing in the computing resource  124 , storage devices provided in the computing resource  124 , data transfer devices provided by the computing resource  124 , etc. In some implementations, the computing resource  124  may communicate with other computing resources  124  via wired connections, wireless connections, or a combination of wired and wireless connections. 
     As further shown in  FIG. 1 , the computing resource  124  includes a group of cloud resources, such as one or more applications (“APPs”)  124 - 1 , one or more virtual machines (“VMs”)  124 - 2 , virtualized storage (“VSs”)  124 - 3 , one or more hypervisors (“HYPs”)  124 - 4 , or the like. 
     The application  124 - 1  includes one or more software applications that may be provided to or accessed by the user device  110  and/or the platform  120 . The application  124 - 1  may eliminate a need to install and execute the software applications on the user device  110 . For example, the application  124 - 1  may include software associated with the platform  120  and/or any other software capable of being provided via the cloud computing environment  122 . In some implementations, one application  124 - 1  may send/receive information to/from one or more other applications  124 - 1 , via the virtual machine  124 - 2 . 
     The virtual machine  124 - 2  includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. The virtual machine  124 - 2  may be either a system virtual machine or a process virtual machine, depending upon use and degree of correspondence to any real machine by the virtual machine  124 - 2 . A system virtual machine may provide a complete system platform that supports execution of a complete operating system (“OS”). A process virtual machine may execute a single program, and may support a single process. In some implementations, the virtual machine  124 - 2  may execute on behalf of a user (e.g., the user device  110 ), and may manage infrastructure of the cloud computing environment  122 , such as data management, synchronization, or long-duration data transfers. 
     The virtualized storage  124 - 3  includes one or more storage systems and/or one or more devices that use virtualization techniques within the storage systems or devices of the computing resource  124 . In some implementations, within the context of a storage system, types of virtualizations may include block virtualization and file virtualization. Block virtualization may refer to abstraction (or separation) of logical storage from physical storage so that the storage system may be accessed without regard to physical storage or heterogeneous structure. The separation may permit administrators of the storage system flexibility in how the administrators manage storage for end users. File virtualization may eliminate dependencies between data accessed at a file level and a location where files are physically stored. This may enable optimization of storage use, server consolidation, and/or performance of non-disruptive file migrations. 
     The hypervisor  124 - 4  may provide hardware virtualization techniques that allow multiple operating systems (e.g., “guest operating systems”) to execute concurrently on a host computer, such as the computing resource  124 . The hypervisor  124 - 4  may present a virtual operating platform to the guest operating systems, and may manage the execution of the guest operating systems. Multiple instances of a variety of operating systems may share virtualized hardware resources. 
     The network  130  includes one or more wired and/or wireless networks. For example, the network  130  may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, or the like, and/or a combination of these or other types of networks. 
     The number and arrangement of devices and networks shown in  FIG. 1  are provided as an example. In practice, there may be additional devices and/or networks, fewer devices and/or networks, different devices and/or networks, or differently arranged devices and/or networks than those shown in  FIG. 1 . Furthermore, two or more devices shown in  FIG. 1  may be implemented within a single device, or a single device shown in  FIG. 1  may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) of the environment  100  may perform one or more functions described as being performed by another set of devices of the environment  100 . 
       FIG. 2  is a block diagram of example components of one or more devices of  FIG. 1 . 
     A device  200  may correspond to the user device  110  and/or the platform  120 . As shown in  FIG. 2 , the device  200  may include a bus  210 , a processor  220 , a memory  230 , a storage component  240 , an input component  250 , an output component  260 , and a communication interface  270 . 
     The bus  210  includes a component that permits communication among the components of the device  200 . The processor  220  is implemented in hardware, firmware, or a combination of hardware and software. The processor  220  is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor  220  includes one or more processors capable of being programmed to perform a function. The memory  230  includes a random access memory (RAM), a read only memory (ROM), and/or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and/or an optical memory) that stores information and/or instructions for use by the processor  220 . 
     The storage component  240  stores information and/or software related to the operation and use of the device  200 . For example, the storage component  240  may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and/or a solid state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and/or another type of non-transitory computer-readable medium, along with a corresponding drive. 
     The input component  250  includes a component that permits the device  200  to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and/or a microphone). Additionally, or alternatively, the input component  250  may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and/or an actuator). The output component  260  includes a component that provides output information from the device  200  (e.g., a display, a speaker, and/or one or more light-emitting diodes (LEDs)). 
     The communication interface  270  includes a transceiver-like component (e.g., a transceiver and/or a separate receiver and transmitter) that enables the device  200  to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface  270  may permit the device  200  to receive information from another device and/or provide information to another device. For example, the communication interface  270  may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like. 
     The device  200  may perform one or more processes described herein. The device  200  may perform these processes in response to the processor  220  executing software instructions stored by a non-transitory computer-readable medium, such as the memory  230  and/or the storage component  240 . A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices. 
     Software instructions may be read into the memory  230  and/or the storage component  240  from another computer-readable medium or from another device via the communication interface  270 . When executed, software instructions stored in the memory  230  and/or the storage component  240  may cause the processor  220  to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software. 
     The number and arrangement of components shown in  FIG. 2  are provided as an example. In practice, the device  200  may include additional components, fewer components, different components, or differently arranged components than those shown in  FIG. 2 . Additionally, or alternatively, a set of components (e.g., one or more components) of the device  200  may perform one or more functions described as being performed by another set of components of the device  200 . 
     Methods and apparatuses for adaptive neural image compression with rate control by meta learning will now be described in detail. 
     This disclosure proposes a meta-NIC framework that supports arbitrary smooth bitrate control. A meta-learning mechanism is used to adaptively compute rate-adaptive weight parameters of an underlying NIC model based on a current input image and a target bitrate, so that a single meta-NIC model instance can achieve image compression with arbitrary smooth different bitrates. 
     Given an input image x of size (h,w,c), where h, w, c are a height, a width, and a number of channels, respectively, a target of a test stage of an NIC workflow can be described as follows. A compressed representation  y  that is compact for storage and transmission is computed. Then, based on the compressed representation  y , an output image  x  is reconstructed, and the reconstructed output image  x  may be similar to the original input image x. A distortion loss D(x,  x ) is used to measure a reconstruction error, such as a peak signal-to-noise ratio (PSNR) or a structural similarity index measure (SSIM). A rate loss R( y ) is computed to measure a bit consumption of the compressed representation  y . A trade-off hyperparameter λ is used to form a joint Rate-Distortion (R-D) loss: 
         L ( x, x , y   )=λ D ( x, x   )+ R (   y   )  (1).
 
     Training with a large hyperparameter λ results in compression models with smaller distortion but more bit consumption, and vice versa. Traditionally, for each pre-defined hyperparameter λ, an NIC model instance will be trained, which will not work well for other values of the hyperparameter λ. Therefore, to achieve multiple bitrates of a compressed stream, traditional methods may require training and storing multiple model instances. Also, because it is difficult to train a model for every possible value of the hyperparameter λ in practice, traditional methods cannot achieve arbitrary smooth bitrate control. 
       FIGS. 3A and 3B  are block diagrams of meta-NIC architectures  300 A and  300 B for adaptive neural image compression with rate control by meta-learning, according to embodiments. 
     As shown in  FIG. 3A , the meta-NIC architecture  300 A includes a shared encoding NN  305 , an adaptive encoding NN  310 , a shared decoding NN  315  and an adaptive decoding NN  320 . 
     As shown in  FIG. 3B , the meta-NIC architecture  300 B includes shared encoding layers  325  and  330 , adaptive encoding layers  335  and  340 , shared decoding layers  345  and  350  and adaptive decoding layers  355  and  360 . 
     In this disclosure, model parameters of an underlying NIC encoder and an underlying NIC decoder are separated into 4 parts θ s   e , θ a   e , θ s   d , θ a   d , denoting Shared Encoding Parameters (SEP), Adaptive Encoding Parameters (AEP), Shared Decoding Parameters (SDP), and Adaptive Decoding Parameters (ADP), respectively.  FIGS. 3A and 3B  show two embodiments of an NIC network architecture. 
     In  FIG. 3A , SEP, SDP, AEP and ADP are separated individual NN modules, and these individual modules are connected to each other sequentially for network forward computation. Here,  FIG. 3A  shows a sequential order of connecting these individual NN modules. Other orders can be used here. 
     In  FIG. 3B , a parameter split is within NN layers. Let θ s   e (i), θ a   e (i), θ s   d (j), θ a   d (j) denote SEP, AEP, SDP, and ADP for an i-th layer of an NIC encoder and a j-th layer of an NIC decoder, respectively. The network will compute inference outputs based on corresponding inputs for the SEP and AEP (or the SDP and ADP), respectively, and these outputs will be combined (e.g., by addition, concatenation, multiplication, etc.) and then sent to a next layer. 
     The embodiment of  FIG. 3A  can be seen as a case of  FIG. 1B , in which layers in the shared encoding NN  305  θ a   e (i) are empty, layers in the adaptive encoding NN  310  θ s   e (i) are empty, layers in the shared decoding NN  315  θ s   d (j) are empty, and layers in the adaptive decoding NN  320  θ a   d (j) are empty. Therefore, in other embodiments, the network structures of  FIGS. 3A and 3B  can be combined, in which an NIC architecture includes both purely shared encoding/decoding layers and/or purely adaptive encoding/decoding layers, and mixed layers with partial shared encoding/decoding parameters and partial adaptive encoding/decoding parameters. 
       FIG. 4A  is a block diagram of an apparatus  400  for adaptive neural image compression with rate control by meta-learning, during a test stage, according to embodiments. 
     As shown in  FIG. 4A , the apparatus  400  includes a meta-NIC encoder  410  and a meta-NIC decoder  420 . 
       FIG. 4A  shows an overall workflow of the test stage of a meta-NIC framework. Let θ s   e (i) and θ a   e (i) denote SEP and AEP for an i-th layer of the meta-NIC encoder  410 , respectively. This is an example notation, because for a layer that is completely shared, θ a   e (i) is empty. For a layer that is completely adaptive, θ s   e (i) is empty. In other words, this notation can be used for both embodiments of  FIGS. 3A and 3B . 
     Given an input image x, and given a target trade-off hyperparameter λ, the meta-NIC encoder  410  computes a compressed representation  y . In the meta-NIC encoder  410 , the input image x is passed through a meta-NIC encoding NN. Let f(i) and f(i+1) denote an input tensor and an output tensor of an i-th layer. 
       FIG. 4B  is a block diagram of the meta-NIC encoder  410  of the apparatus  400  shown in  FIG. 4A . 
     As shown in  FIG. 4B , the meta-NIC encoder  410  includes an SEP inference portion  412 , an AEP prediction portion  414  and an AEP inference portion  416 . 
       FIG. 4B  gives an example embodiment of an inference workflow of the meta-NIC encoder  410  for an i-th layer. Based on a current input f(i) and SEP θ s   e (i), the SEP inference portion  412  computes a shared feature g(i) based on a shared inference function G i (f(i), θ s   e (i)) that is modeled by a forward computation using the SEP in the i-th layer. Based on the current input f(i), the shared feature g(i), AEP θ a   e (i) and a hyperparameter λ, the AEP prediction portion  414  computes estimated AEP {circumflex over (θ)} a   e (i) for the i-th layer. The AEP prediction portion  414  may be an NN, e.g., including convolution and fully connected layers, which predict the updated estimated AEP {circumflex over (θ)} a   e (i) based on the original AEP θ a   e (i), the current input f(i), and the target hyperparameter λ. In some embodiments, the current input f(i) is used as an input to the AEP prediction portion  414 . In some other embodiments, the shared feature g(i) is used instead of the current input f(i). In other embodiments, an SEP loss can be computed based on the shared feature g(i), and a gradient of the loss is used as input to the AEP prediction portion  414 . Based on the estimated AEP {circumflex over (θ)} a   e (i) and the shared feature g(i), the AEP inference portion  416  computes an output tensor f(i+1) based on an AEP inference function A i (g(i), {circumflex over (θ)} a   e (i)) that is modeled by the forward computation using the estimated AEP in the i-th layer. 
     Note that the workflow described in  FIG. 4B  is an example notation. For a layer that is completely shared with the AEP θ a   e (i) being empty, AEP-related modules and f(i+1)=g(i) may be omitted. For a layer that is completely adaptive with the SEP θ s   e (i) being empty, SEP-related modules and g(i)=f(i) may be omitted. 
     Assume there are a total of N layers for the meta-NIC encoder  410 , an output of a last layer is a compressed representation  y , which is sent to the meta-NIC decoder  420  (e.g., after being further compressed into a compact bitstream by quantization and entropy encoding). 
     Referring again to  FIG. 4A , on a decoder side, let θ s   d (j) and θ a   d (j) denote SDP and ADP for a j-th layer of the meta-NIC decoder  420 , respectively. Similar to the meta-NIC encoder  410 , this is an example notation, because for a layer that is completely shared, θ a   d (j) is empty, and for a layer that is completely adaptive, θ s   d (j) is empty. 
     On a decoder side, a recovered compressed representation  y ′ is obtained from a bitstream sent from the meta-NIC encoder  410  by entropy decoding and dequantization. Based on the recovered compressed representation  y ′ and the target trade-off hyperparameter λ, the meta-NIC decoder  420  computes a reconstructed output image  x . In the meta-NIC decoder  420 , the recovered compressed representation  y ′ is passed through the meta-NIC decoding NN. Let f(j) and f(j+1) denote an input tensor and an output tensor of a j-th layer. 
       FIG. 4C  is a block diagram of the meta-NIC decoder  420  of the apparatus  400  shown in  FIG. 4A . 
     As shown in  FIG. 4C , the meta-NIC decoder  420  includes an SDP inference portion  422 , an ADP prediction portion  424  and an ADP inference portion  426 . 
       FIG. 4C  gives an example embodiment of an inference workflow of the meta-NIC decoder  420  for a j-th layer. Based on a current input f(j) and SDP θ s   d (j), the SDP inference portion  422  computes a shared feature g(j) based on a shared inference function G j (f(j), θ s   d (j)) that is modeled by a network forward computation using the SDP of the j-th layer. Based on the current input f(j), the shared feature g(j), ADP θ a   d (j) and a hyperparameter λ, the ADP prediction portion  424  computes estimated ADP {circumflex over (θ)} a   d (j) for the j-th layer. The ADP prediction portion  424  may be an NN, e.g., with convolution and fully connected layers, which predict the updated estimated {circumflex over (θ)} a   d (j) based on the original ADP θ d   d (i), the current input f(j), and the target hyperparameter λ. In some embodiments, the current input f(j) is used as input to the ADP prediction portion  424 . In some other embodiments, the shared feature g(j) is used instead of the current input f(j). In other embodiments, an SDP loss is computed based on the shared feature g(j), and a gradient of the loss is used as input to the ADP prediction portion  424 . Based on the estimated ADP {circumflex over (θ)} a   d (j) and the shared feature g(j), the ADP inference portion  426  computes an output tensor f(j+1) based on an ADP inference function A j (g(j), {circumflex over (θ)} a   d (j)) that is modeled by the network forward computation using estimated ADP in the j-th layer. 
     Note that the workflow described in  FIG. 4C  is an example notation. For a layer that is completely shared with ADP θ a   d (j) being empty, ADP-related modules and f(j+1)=g(j) may be omitted. For a layer that is completely adaptive with SDP θ s   d (j) being empty, SDP-related modules and g(j)=f(j) may be omitted. 
     Assume there are a total of M layers for the meta-NIC decoder  420 , an output of a last layer is a reconstructed image output  x . 
     Note that the meta-NIC framework allows an arbitrary smooth trade-off hyperparameter λ, and a processing workflow will compute a compressed representation and a reconstructed output image to fit the target hyperparameter λ. 
     In some embodiments, the trade-off hyperparameter λ is the same for encoder and decoder. In some other embodiments, the hyperparameter λ can be different for the meta-NIC encoder  410  and the meta-NIC decoder  420 . In such a case, the meta-NIC decoder  420  tries to adapt the compressed representation to a different target quality from an original encoding target quality. 
     In a case, when the AEP prediction portion  414  and the ADP prediction portion  424  only perform prediction over a pre-defined set of trade-off hyperparameters with/without considering the input f(i) or f(j), a meta-NIC model reduces to a multi-rate NIC model that uses one model instance to accommodate compression effects of multiple pre-defined bitrates. 
       FIG. 5  is a block diagram of a training apparatus  500  for adaptive neural image compression with rate control by meta-learning, during a training stage, according to embodiments. 
     As shown in  FIG. 5 , the training apparatus  500  includes a task sampler  510 , an inner-loop loss generator  520 , an inner-loop update portion  530 , a meta loss generator  540 , a meta update portion  550  and a weight update portion  560 . 
     A training process aims at learning SEP θ s   e (i) and AEP θ a   e (i), i=1, . . . , N for the meta-NIC encoder  410  of  FIG. 4A  and SDP θ s   d (j) and ADP θ a   d (j), j=1, . . . , M for the meta-NIC decoder  420  of  FIG. 4A , as well as an AEP prediction NN (model parameters denoted as Φ e ) and an ADP prediction NN (model parameters denoted as Φ d ). 
     In embodiments, a Model-Agnostic Meta-Learning (MAML) mechanism is used for a training purpose.  FIG. 5  gives an example workflow of a meta-training framework. Other meta-training algorithms can be used here. 
     For training, there is a set of training data D tr (λ i ), i=1, . . . , K, where each D tr (λ i ) corresponds to a training trade-off hyperparameter λ i , and there are K training trade-off hyperparameters (thus K training data sets) in total. In addition, there is a set of validation data D val (λ j ), j=1, . . . , P, where each D val (λ j ) corresponds to a validation trade-off hyperparameter λ j , and there are P validation trade-off hyperparameters in total. The validation trade-off hyperparameters include different hyperparameters from the training set. The validation trade-off hyperparameters can also include those hyperparameters from the training set. 
     An overall training goal is to learn a meta-NIC model so that it can be broadly applied to all (including training and future unseen) trade-off hyperparameters, under an assumption that an NIC task with a target trade-off hyperparameter is drawn from a task distribution P(λ). To achieve this, a loss for learning the meta-NIC model is minimized across all training data sets across all training trade-off hyperparameters. 
     Let Θ s ={θ s   e , θ s   d } include all shared parameters in SEP and SDP, and let Θ a ={θ a   e , θ a   d } include all adaptive parameters in AEP and ADP. The MAML training process may have an outer loop and an inner loop for gradient-based parameter updates. For each outer loop iteration, the task sampler  510  first samples a set of K′ training trade-off hyperparameters (K′≤K). Then for each sampled training trade-off hyperparameter λ i , the task sampler  510  samples a set of training data {tilde over (D)} tr (λ i ) from the set of training data D tr (λ i ). Also, the task sampler  510  samples a set of P′ (P′≤P) validation trade-off hyperparameters, and for each sampled validation hyperparameter λ j , samples a set of validation data {tilde over (D)} val (λ j ) from the set of validation data D val (λ j ). Then for each sampled datum xϵ{tilde over (D)} tr (λ i ), a meta-NIC forward computation is conducted based on current parameters Θ s , Θ a , Φ e  and Φ d , and the inner-loop loss generator  520  then computes an accumulated inner-loop loss L {tilde over (D)}     tr     (λ     i     ) (Θ s , Θ a , Φ e , Φ d , λ i ): 
         L   {tilde over (D)}     tr     (λ     i     ) (Θ s ,θ a ,Φ e ,Φ d ,λ i )=Σ xϵ{tilde over (D)}     tr     (λ     i     )   L ( x,Θ   s ,Θ a ,Φ e ,Φ d ,λ i )  (2).
 
     The loss function L(x, Θ s , Θ a , Φ e , Φ d , λ i ) may include an R-D loss of Equation (1) and another regularization loss (e.g., an auxiliary loss of distinguishing an intermediate network output targeting different trade-offs). Then, based on the inner-loop loss L {tilde over (D)}     tr     (λ     i     ) (Θ s , θ a , Φ e , Φ d , λ i ), given step sizes α si  and α ai  as hyperparameters for λ i , the inner-loop update portion  530  computes an updated task-specific parameter update: 
       {circumflex over (Θ)} a =Θ a −Σ i=1   K′ α ai ∇ θ     a     L   {tilde over (D)}     tr     (λ     i     ) (Θ s ,Θ a ,Φ e ,Φ d ,λ i ),  (3); and
 
       {circumflex over (Θ)} s =Θ s −Σ i=1   K′ α si ∇ Θ     s     L   {tilde over (D)}     tr     (λ     i     ) (Θ s ,Θ a ,Φ e ,Φ d ,λ i )  (4).
 
     Gradient ∇ Θ     a   L {tilde over (D)}     tr     (λ     i     ) (Θ s , Θ a , Φ e , Φ d , λ i ) and gradient ∇ Θ     s   L {tilde over (D)}     tr     (λ     i     ) (Θ s , Θ a , Φ e , Φ d , λ i ) of the accumulated inner-loop loss L {tilde over (D)}     tr     (λ     i     ) (Θ s , Θ a , Φ e , Φ d , λ i ) are used to compute an updated version of adaptive parameters {circumflex over (Θ)} a  and {circumflex over (Θ)} s , respectively. 
     Then, a meta loss generator  540  computes an outer meta objective or loss over all sampled validation hyperparameters: 
         L (Θ s ,Θ a ,Φ e ,Φ d )=Σ j=1   P′   L   {tilde over (D)}     val     (λ     i     ) ({circumflex over (Θ)} s ,{circumflex over (Θ)} a ,Φ e ,Φ d ,λ j ),  (5); and
 
         L   {tilde over (D)}     val     (λ     i     ) ({circumflex over (Θ)} s ,{circumflex over (Θ)} a ,Φ e ,Φ d ,λ j )=Σ xϵ{tilde over (D)}     val     (λ     j     )   L ( x,{circumflex over (Θ)}   s ,{circumflex over (Θ)} a ,Φ e ,Φ d ,λ j )  (6),
 
     where L(x, {circumflex over (Θ)} s , {circumflex over (Θ)} a , Φ s , Φ a , λ j ) is the loss computed for an input x based on the meta-NIC forward computation using parameters {circumflex over (Θ)} s , {circumflex over (Θ)} a , Φ s , Φ a . Given step size β aj  and β sj  as hyperparameters for λ j , the meta update portion  550  updates the model parameters as: 
       Θ a =Θ a −Σ j=1   P′ β aj   L   {tilde over (D)}     val     (λ     i     ) ({circumflex over (Θ)} a ,{circumflex over (Θ)} a ,Φ e ,Φ d ,λ j )  (7); and
 
       Θ s =Θ s −Σ j=1   P′ β sj ∇ Θ     s     L   {tilde over (D)}     val     (λ     i     ) ({circumflex over (Θ)} s ,{circumflex over (Θ)} a ,Φ e ,Φ d ,λ j )  (8).
 
     In some embodiments, Θ s  is not updated in an inner loop, i.e., α si =0, {circumflex over (Θ)} s =Θ s . This may help to stabilize the training process. 
     As for parameters Φ e , Φ d  of the AEP Prediction NN and ADP prediction NN, the weight update portion  560  updates them in a regular training manner. That is, according to the training and validation data D tr (λ i ), i=1, . . . , K, D val (λ j ), j=1, . . . , P, based on the current parameters Θ s , Θ a , Φ e , Φ d  a loss L(x, Θ s , Θ a , Φ e , Φ d , λ i ) for all samples xϵD tr (λ i ) and a loss L(x, Θ s , Θ a , Φ e , Φ d , λ j ) for all samples xϵD val (λ j ) are computed. Gradients of all these losses can be accumulated (e.g., added up) to perform parameter updates over Φ e , Φ d  through regular back-propagation. 
     Note that this disclosure does not put any restrictions on an optimization algorithm or loss functions for updating these model parameters. 
     For a case in which the AEP prediction portion  414  of  FIG. 4B  and the ADP prediction portion  424  of a meta-NIC model only perform prediction over a pre-defined set of training trade-off hyperparameters, validation trade-off hyperparameters will be the same with the training ones. The same MAML training procedure can be used to train this reduced meta-NIC model (i.e., a multi-rate NIC model that uses one model instance to accommodate compression effects of multiple pre-defined bitrates). 
     The embodiments described herein allow an ability of using only one meta-NIC model instance to achieve image compression with arbitrary smooth bitrate control using meta learning. The methods and apparatuses can be used for both multi-rate compression with a single model and smooth bitrate control. The embodiments provide a flexible framework that accommodates various underlying NIC models and meta-learning methods. 
       FIG. 6  is a flowchart of a method of adaptive neural image compression with rate control by meta-learning, according to embodiments. 
     In some implementations, one or more process blocks of  FIG. 6  may be performed by the platform  120 . In some implementations, one or more process blocks of  FIG. 6  may be performed by another device or a group of devices separate from or including the platform  120 , such as the user device  110 . 
     As shown in  FIG. 6 , in operation  610 , the method  600  includes receiving an input image and a hyperparameter. 
     In operation  620 , the method  600  includes encoding the received input image, based on the received hyperparameter, using an encoding neural network, to generate a compressed representation. 
     The encoding includes performing a first shared encoding on the received input image, using a first shared encoding layer having first shared encoding parameters, performing a first adaptive encoding on the received input image, using a first adaptive encoding layer having first adaptive encoding parameters, combining the first shared encoded input image and the first adaptive encoded input image, to generate a first combined output, and performing a second shared encoding on the first combined output, using a second shared encoding layer having second shared encoding parameters. The encoding may further include performing a second adaptive encoding on the first combined output, using a second adaptive encoding layer having second adaptive encoding parameters. 
     In operation  630 , the method  600  includes receiving a recovered compressed representation and the hyperparameter. 
     In operation  640 , the method  600  includes decoding the received recovered compressed representation, based on the received hyperparameter, using a decoding neural network, to reconstruct an output image. 
     The decoding includes performing a first shared decoding on the received recovered compressed representation, using a first shared decoding layer having first shared decoding parameters, performing a first adaptive decoding on the received recovered compressed representation, using a first adaptive decoding layer having first adaptive decoding parameters, combining the first shared decoded recovered compressed representation and the first adaptive decoded recovered compressed representation, to generate a second combined output, performing a second shared decoding on the second combined output, using a second shared decoding layer having second shared decoding parameters, and performing a second adaptive decoding on the second combined output, using a second adaptive decoding layer having second adaptive decoding parameters. 
     The encoding may include generating a shared feature, based the received input image and the first shared encoding parameters, generating estimated adaptive encoding parameters, based on one or more of the received input image, the first adaptive encoding parameters, the generated shared feature, and the received hyperparameter, using a prediction neural network, and generating the compressed representation, based on the estimated adaptive encoding parameters and the received hyperparameter. 
     The prediction neural network may be trained by generating a first loss for training data corresponding to the received hyperparameter, and a second loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, the first adaptive decoding parameters, and prediction parameters of the prediction neural network, and updating the prediction parameters, based on gradients of the generated first loss and the generated second loss. 
     The decoding may include generating a shared feature, based the received input image and the first shared decoding parameters, generating estimated adaptive decoding parameters, based on one or more of the received input image, the first adaptive decoding parameters, the generated shared feature, and the received hyperparameter, using a prediction neural network, and reconstructing the output image, based on the estimated adaptive decoding parameters and the received hyperparameter. 
     The prediction neural network may be trained by generating a first loss for training data corresponding to the received hyperparameter, and a second loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, the first adaptive decoding parameters, and prediction parameters of the prediction neural network, and updating the prediction parameters, based on gradients of the generated first loss and the generated second loss. 
     The encoding neural network and the decoding neural network may be trained by generating an inner-loop loss for training data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, and the first adaptive decoding parameters, first updating the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters and the first adaptive decoding parameters, based on gradients of the generated inner-loop loss, generating a meta loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters, and second updating the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters, based on gradients of the generated meta loss. 
     Although  FIG. 6  shows example blocks of the method  600 , in some implementations, the method  600  may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in  FIG. 6 . Additionally, or alternatively, two or more of the blocks of the method  600  may be performed in parallel. 
       FIG. 7  is a block diagram of an apparatus  700  for task-adaptive pre-processing for neural image compression, according to embodiments. 
     As shown in  FIG. 7 , the apparatus  700  includes first receiving code  710 , encoding code  720 , second receiving code  730  and decoding code  740 . 
     The first receiving code  710  is configured to cause at least one processor to receiving an input image and a hyperparameter. 
     The encoding code  720  is configured to cause the at least one processor to encode the received input image, based on the received hyperparameter, using an encoding neural network, to generate a compressed representation. 
     The encoding code  720  is further configured to cause the at least one processor to perform a first shared encoding on the received input image, using a first shared encoding layer having first shared encoding parameters, perform a first adaptive encoding on the received input image, using a first adaptive encoding layer having first adaptive encoding parameters, combine the first shared encoded input image and the first adaptive encoded input image, to generate a first combined output, and perform a second shared encoding on the first combined output, using a second shared encoding layer having second shared encoding parameters. The encoding code  720  may be further configured to cause the at least one processor to perform a second adaptive encoding on the first combined output, using a second adaptive encoding layer having second adaptive encoding parameters. 
     The second receiving code  730  is configured to cause the at least one processor to receive a recovered compressed representation and the hyperparameter. 
     The decoding code  740  is configured to cause the at least one processor to decode the received recovered compressed representation, based on the received hyperparameter, using a decoding neural network, to reconstruct an output image. 
     The decoding code  740  is further configured to cause the at least one processor to perform a first shared decoding on the received recovered compressed representation, using a first shared decoding layer having first shared decoding parameters, perform a first adaptive decoding on the received recovered compressed representation, using a first adaptive decoding layer having first adaptive decoding parameters, combine the first shared decoded recovered compressed representation and the first adaptive decoded recovered compressed representation, to generate a second combined output, perform a second shared decoding on the second combined output, using a second shared decoding layer having second shared decoding parameters, and perform a second adaptive decoding on the second combined output, using a second adaptive decoding layer having second adaptive decoding parameters. 
     The encoding code  720  may be further configured to cause the at least one processor to generate a shared feature, based the received input image and the first shared encoding parameters, generate estimated adaptive encoding parameters, based on one or more of the received input image, the first adaptive encoding parameters, the generated shared feature, and the received hyperparameter, using a prediction neural network, and generate the compressed representation, based on the estimated adaptive encoding parameters and the received hyperparameter. 
     The prediction neural network may be trained by generating a first loss for training data corresponding to the received hyperparameter, and a second loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, the first adaptive decoding parameters, and prediction parameters of the prediction neural network, and updating the prediction parameters, based on gradients of the generated first loss and the generated second loss. 
     The decoding code  740  may be further configured to cause the at least one processor to generate a shared feature, based the received input image and the first shared decoding parameters, generate estimated adaptive decoding parameters, based on one or more of the received input image, the first adaptive decoding parameters, the generated shared feature, and the received hyperparameter, using a prediction neural network, and reconstruct the output image, based on the estimated adaptive decoding parameters and the received hyperparameter. 
     The prediction neural network may be trained by generating a first loss for training data corresponding to the received hyperparameter, and a second loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, the first adaptive decoding parameters, and prediction parameters of the prediction neural network, and updating the prediction parameters, based on gradients of the generated first loss and the generated second loss. 
     The encoding neural network and the decoding neural network may be trained by generating an inner-loop loss for training data corresponding to the received hyperparameter, based on the received hyperparameter, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, and the first adaptive decoding parameters, first updating the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters and the first adaptive decoding parameters, based on gradients of the generated inner-loop loss, generating a meta loss for validation data corresponding to the received hyperparameter, based on the received hyperparameter, the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters, and second updating the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters, based on gradients of the generated meta loss. 
     The proposed methods may be used separately or combined in any order. Further, each of the methods (or embodiments), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program that is stored in a non-transitory computer-readable medium. 
     The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. 
     As used herein, the term component is intended to be broadly construed as hardware, firmware, or a combination of hardware and software. 
     It will be apparent that systems and/or methods, described herein, may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods were described herein without reference to specific software code—it being understood that software and hardware may be designed to implement the systems and/or methods based on the description herein. 
     Even though combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set. 
     No element, act, or instruction used herein may be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.), and may be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.