SUBSTITUTIONAL QUALITY FACTOR LEARNING IN THE LATENT SPACE FOR NEURAL IMAGE COMPRESSION

Neural image compression using substitutional quality factor learning in a latent space, including receiving a compressed bitstream and a target quality factor indicating a target compression quality, calculating a decoded latent representation of the compressed bitstream, and calculating a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, computing a shared feature based on a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, computing estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and computing an output tensor based on the estimated ADP in the one or more layers of the convolutional neural network and the shared feature.

FIELD

Apparatuses and methods consistent with example embodiments of the present disclosure relate to the substitutional quality factor learning in the latent space for neural image compression.

BACKGROUND

ISO/IEC MPEG (JTC 1/SC 29/WG 11) has been actively searching for potential needs for standardization of future video coding technology, including advanced neural image and video compression methodologies. ISO/IEC JPEG has established the JPEG-AI group focusing on AI-based end-to-end Neural Image Compression (NIC) using Neural Networks (NN).

Although previous approaches have shown promising performance, flexible bitrate control remains a challenging issue for previous NIC methods. Conventionally, it may require training multiple model instances targeting each desired trade-off between a rate and a distortion (a quality of compressed images) individually. All these multiple model instances may be stored and deployed on a decoder side to reconstruct images from different bitrates. Also, these model instances cannot give arbitrary smooth bitrate control, because it is difficult to train and store an infinite number of model instances for every possible target bitrate. Previous approaches have studied multi-rate NIC in which one model instance is trained to achieve compression of multiple pre-defined bitrates. However, arbitrary smooth bitrate control remains an unexplored open issue.

SUMMARY

According to some embodiments, a method may be provided for neural image compression using substitutional quality factor learning in a latent space, the method being performed by at least one processor, the method including receiving a compressed bitstream and a target quality factor indicating a target compression quality; calculating a decoded latent representation of the compressed bitstream; and calculating a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, wherein calculating the reconstructed image includes computing a shared feature based a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, computing estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and computing an output tensor based on the estimated ADP in the one or more layers of the convolutional neural network and the shared feature.

According to exemplary embodiments, an apparatus may be provided that includes at least one memory configured to store computer program code; and at least one processor configured to access said at least one memory and operate as instructed by said computer program code, said computer program code including receiving code configured to cause at least one processor to receive a compressed bitstream and a target quality factor indicating a target compression quality; first calculating code configured to cause the at least one processor to calculate a decoded latent representation of the compressed bitstream; and second calculating code configured to cause the at least one processor to calculate a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, wherein the second calculating code is further configured to include first computing code configured to cause the at least one processor to compute a shared feature based on a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, second computing code configured to cause the at least one processor to compute estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and third computing code configured to cause the at least one processor to compute an output tensor based on the estimated ADP in the one or more layers and the shared feature.

According to some embodiments, a non-transitory computer-readable recording medium may be provided having instructions stored thereon, which when executed by at least one processor in a decoder cause the processor to perform a method for neural image compression using substitutional quality factor learning in a latent space, the method including receiving a compressed bitstream and a target quality factor indicating a target compression quality; calculating a decoded latent representation of the compressed bitstream; and calculating a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, wherein calculating the reconstructed image includes computing a shared feature based on a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, computing estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and computing an output tensor based on the estimated ADP in the one or more layers and the shared feature.

DETAILED DESCRIPTION

The disclosure describes methods and apparatuses for a Meta Neural Image Compression (meta-NIC) framework by finding substitutional Quality Factors (QF) in a decoded latent space. A meta learning mechanism may be used to adaptively compute the substitutional quality control parameter for each image on the encoder based on the decoded latent feature of the input image and the target compression quality. The substitutional quality control parameters may be used to improve the computed quality-adaptive weight parameters towards better recovery of the target image when the decoder is reconstructing the image.

FIG.1is a diagram of an environment100in which methods, apparatuses and systems described herein may be implemented, according to embodiments.

As shown inFIG.1, the environment100may include a user device110, a platform120, and a network130. Devices of the environment100may interconnect via wired connections, wireless connections, or a combination of wired and wireless connections.

The user device110includes one or more devices capable of receiving, generating, storing, processing, and/or providing information associated with platform120. For example, the user device110may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a radiotelephone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or a similar device. In some implementations, the user device110may receive information from and/or transmit information to the platform120.

The platform120includes one or more devices as described elsewhere herein. In some implementations, the platform120may include a cloud server or a group of cloud servers. In some implementations, the platform120may be designed to be modular such that software components may be swapped in or out. As such, the platform120may be easily and/or quickly reconfigured for different uses.

In some implementations, as shown, the platform120may be hosted in a cloud computing environment122. Notably, while implementations described herein describe the platform120as being hosted in the cloud computing environment122, in some implementations, the platform120may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

The cloud computing environment122includes an environment that hosts the platform120. The cloud computing environment122may provide computation, software, data access, storage, etc. services that do not require end-user (e.g., the user device110) knowledge of a physical location and configuration of system(s) and/or device(s) that hosts the platform120. As shown, the cloud computing environment122may include a group of computing resources124(referred to collectively as “computing resources124” and individually as “computing resource124”).

The computing resource124includes one or more personal computers, workstation computers, server devices, or other types of computation and/or communication devices. In some implementations, the computing resource124may host the platform120. The cloud resources may include compute instances executing in the computing resource124, storage devices provided in the computing resource124, data transfer devices provided by the computing resource124, etc. In some implementations, the computing resource124may communicate with other computing resources124via wired connections, wireless connections, or a combination of wired and wireless connections.

As further shown inFIG.1, the computing resource124includes a group of cloud resources, such as one or more applications (“APPs”)124-1, one or more virtual machines (“VMs”)124-2, virtualized storage (“VSs”)124-3, one or more hypervisors (“HYPs”)124-4, or the like.

The application124-1includes one or more software applications that may be provided to or accessed by the user device110and/or the platform120. The application124-1may eliminate a need to install and execute the software applications on the user device110. For example, the application124-1may include software associated with the platform120and/or any other software capable of being provided via the cloud computing environment122. In some implementations, one application124-1may send/receive information to/from one or more other applications124-1, via the virtual machine124-2.

The virtual machine124-2includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. The virtual machine124-2may be either a system virtual machine or a process virtual machine, depending upon use and degree of correspondence to any real machine by the virtual machine124-2. A system virtual machine may provide a complete system platform that supports execution of a complete operating system (“OS”). A process virtual machine may execute a single program, and may support a single process. In some implementations, the virtual machine124-2may execute on behalf of a user (e.g., the user device110), and may manage infrastructure of the cloud computing environment122, such as data management, synchronization, or long-duration data transfers.

The hypervisor124-4may provide hardware virtualization techniques that allow multiple operating systems (e.g., “guest operating systems”) to execute concurrently on a host computer, such as the computing resource124. The hypervisor124-4may present a virtual operating platform to the guest operating systems, and may manage the execution of the guest operating systems. Multiple instances of a variety of operating systems may share virtualized hardware resources.

FIG.2is a block diagram of example components of one or more devices ofFIG.1.

A device200may correspond to the user device110and/or the platform120. As shown inFIG.2, the device200may include a bus210, a processor220, a memory230, a storage component240, an input component250, an output component260, and a communication interface270.

The bus210includes a component that permits communication among the components of the device200. The processor220may be implemented in hardware, firmware, or a combination of hardware and software. The processor220is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor220includes one or more processors capable of being programmed to perform an operation. The memory230includes a random access memory (RAM), a read only memory (ROM), and/or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and/or an optical memory) that stores information and/or instructions for use by the processor220.

The input component250includes a component that permits the device200to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and/or a microphone). Additionally, or alternatively, the input component250may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and/or an actuator). The output component260includes a component that provides output information from the device200(e.g., a display, a speaker, and/or one or more light-emitting diodes (LEDs)).

The communication interface270includes a transceiver-like component (e.g., a transceiver and/or a separate receiver and transmitter) that enables the device200to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface270may permit the device200to receive information from another device and/or provide information to another device. For example, the communication interface270may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

The device200may perform one or more processes described herein. The device200may perform these processes in response to the processor220executing software instructions stored by a non-transitory computer-readable medium, such as the memory230and/or the storage component240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

Software instructions may be read into the memory230and/or the storage component240from another computer-readable medium or from another device via the communication interface270. When executed, software instructions stored in the memory230and/or the storage component240may cause the processor220to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

The number and arrangement of components shown inFIG.2are provided as an example. In practice, the device200may include additional components, fewer components, different components, or differently arranged components than those shown inFIG.2. Additionally, or alternatively, a set of components (e.g., one or more components) of the device200may perform one or more operations described as being performed by another set of components of the device200.

The disclosure proposes a meta-NIC framework that supports substitutional QF in a decoded latent space. A meta learning mechanism may be used to adaptively compute the substitutional quality control parameter for each image on the encoder based on the decoded latent feature of the input image and the target compression quality. The substitutional quality control parameters may be used to improve the computed quality-adaptive weight parameters towards better recovery of the target image when the decoder is reconstructing the image.

Given an input image x of size (h,w,c), where h, w, c are a height, a width, and a number of channels, respectively, a target of a test stage of an NIC workflow may be described as follows. The input image x may be a regular image frame (t=1), a 4-dimensional video sequence comprising more than one image frame (t>1), and so on. Each image frame may be a color image (c=3), a gray-scale image (c=1), an rgb+depth image (c=4), etc. A compressed representationythat may be compact for storage and transmission may be computed. Then, based on the compressed representationy, an output imagexmay be reconstructed, and the reconstructed output imagexmay be similar to the original input image x. A distortion loss D(x,x) may be used to measure a reconstruction error, such as a peak signal-to-noise ratio (PSNR) or a structural similarity index measure (SSIM). A rate loss R(y) may be computed to measure a bit consumption of the compressed representationy. A trade-off hyperparameter λ may be used to form a joint Rate-Distortion (R-D) loss:

Training with a large hyperparameter λ may result in compression models with smaller distortion but more bit consumption, and vice versa. Traditionally, for each pre-defined hyperparameter λ, an NIC model instance will be trained, which will not work well for other values of the hyperparameter λ. Therefore, to achieve multiple bitrates of a compressed stream, traditional methods may require training and storing multiple model instances. Further, because it may be difficult to train a model for every possible value of the hyperparameter λ in practice, traditional methods cannot achieve arbitrary smooth quality control such as arbitrary smooth bitrate control. Additionally, a model instance needs to be trained to optimize the loss measured by each type of metric, (e.g., for each distortion metric, i.e., PSNR, SSIM, a weighted combination of both, or other metrics) and traditional methods cannot achieve smooth quality metric control.

FIGS.3A and3Bare block diagrams of meta-NIC architectures300A and300B for adaptive neural image compression by meta-learning, according to embodiments.

As shown inFIG.3A, the meta-NIC architecture300A includes a shared decoding NN305and an adaptive decoding NN310.

As shown inFIG.3B, the meta-NIC architecture300B includes shared decoding layers325and330and adaptive decoding layers335and340.

In this disclosure, model parameters of an underlying NIC encoder and an underlying NIC decoder are separated into two parts θsdand θaddenoting Shared Decoding Parameters (SDP) and Adaptive Decoding Parameters (ADP) respectively.FIGS.3A and3Bshow two embodiments of an NIC network architecture.

InFIG.3A, SDP and ADP are separated individual NN modules, and these individual modules are connected to each other sequentially for network forward computation. Here,FIG.3Ashows a sequential order of connecting these individual NN modules. Other orders may be used as well.

InFIG.3B, a parameter may be split within NN layers. Let θsd(j) and θad(j) denote SDP and ADP for a j-th layer of an NIC decoder, respectively. The network will compute inference outputs based on corresponding inputs for SDP and ADP, respectively, and these outputs will be combined (e.g., by addition, concatenation, multiplication, etc.) and then sent to a next layer.

The embodiment ofFIG.3Amay be seen as a case ofFIG.3B, in which the layers in the shared decoding NN315θsd(j) and in the adaptive decoding NN320θad(j) are empty. Therefore, in other embodiments, the network structures ofFIGS.3A and3Bmay be combined, in which an NIC architecture includes both purely shared encoding/decoding layers and/or purely adaptive encoding/decoding layers, and mixed layers with partial shared encoding/decoding parameters and partial adaptive encoding/decoding parameters.

In some embodiments, the NN structure of the encoder does not have any restrictions. For example, for each image x, a compressed representationymay be generated by the NN-based encoder. The compressed representationymay then quantized, entropy encoded to generate the bitstreamy′, and then entropy decoded, and dequantized to generate a decoded latent representation ŷ. In some embodiments, an individual encoder model instance may be used for each desired compression quality. In other embodiments, a meta-NIC encoder similar to the meta-NIC decoder may be used with shared and adaptive encoding parameters.

FIGS.4A and4Bare block diagrams of an apparatus400for adaptive neural image compression by meta-learning, during a test stage, according to embodiments. Additionally,FIG.4Cis a block diagram of the inference workflow of the meta-NIC decoder.

As shown inFIG.4A, the apparatus400includes a decoder410and a meta-NIC decoder420.

As shown inFIG.4B, the meta-NIC architecture400B includes a decoder410, a Substitutional Perturbation Generation module420, and a meta-NIC decoder430.

InFIG.4C, the meta-NIC architecture400B includes a SDP Inference module422, a ADP Prediction module424, and a ADP Inference module426.

FIG.4Ashows an overall workflow of the decoder in the test stage of a meta-NIC framework. Let θsd(j) and θad(j) denote SDP and ADP for an j-th layer of the meta-NIC decoder420, respectively. This is an example notation, because for a layer that is completely shared, θad(j) is empty. For a layer that is completely adaptive, θsd(j) is empty. In other words, this notation may be used for both embodiments ofFIGS.3A and3B.

InFIGS.4A and4B, the compressed bitstreamy′ is received, which may be passed through a Decoding module410(typically comprising of entropy decoding and dequantization operations) to compute the decoded latent representation ŷ. At the same time, a target QF Λ may be received, which is sent from the encoder indicating the target compression quality of the reconstructed image. More details of the QF Λ will be described later in the encoding process. Additionally, inFIG.4B, a meta-NIC Decoding module430computes the reconstructed imagexbased on the latent representation ŷ and the target QF Λ. In the meta-NIC Decoding module430, ŷ may be passed through the meta-NIC Decoding NN. Let f(j) and f(j+1) denote the input and output tensor of the j-th layer.

FIG.4Cgives an embodiment of the inference workflow of the meta-NIC decoder for the j-th layer. Based on f(j) and θsd(j), an SDP Inference module422computes a shared feature g(j) based on a shared inference operation Gj(f(j), θsd(j)) (e.g., the operation may be modeled by the network forward computation using the SDP of the j-th layer). Based on f(j), g(j), θad(j) and Λ, an ADP Prediction module424computes an estimated ADP {circumflex over (θ)}ad(j) for the j-th layer. The ADP Prediction module424may typically be an NN (e.g., with convolution and fully connected layers), which predicts an updated {circumflex over (θ)}ad(j) based on the original ADP θdd(i), the current input, and the target quality indicator Λ. In the embodiment ofFIG.4C, f(j) may be used as input to the ADP Prediction module424. In some other embodiments, g(j) may be used instead of f(j). In other embodiments, an SDP loss may be computed based on g (j) and the gradient of the loss may be used as input to the ADP Prediction. Based on the estimated ADP {circumflex over (θ)}ad(j) and the shared feature g(j), an ADP Inference module426computes the output tensor f(j+1) based on an ADP inference426operation Aj(g(j), {circumflex over (θ)}ad(j)) (e.g., the operation may be modeled by the network forward computation using estimated ADP in the j-th layer).

The workflow described inFIG.4Cis a general notation. For a layer that is completely shared with θad(j) being empty, the ADP-related modules and f(j+1)=g(j) are omitted. For a layer that is completely adaptive with θsd(j) being empty, the SDP-related modules and g(j)=f(j) are omitted.

Assume there are a total of M layers for the meta-NIC decoder, the output of the last layer may result in the reconstructed imagex.

Additionally, inFIG.4B, the decoded latent ŷ may be passed through a Substitutional Perturbation Generation module420, which computes a substitutional latent ŷ′ based on latent ŷ and the QF Λ. This substitutional latent ŷ′ may then be passed into the meta-NIC Decoding module430instead of the original ŷ to compute the reconstructed imagex.

FIGS.5A and5Bare block diagrams of meta-NIC architectures500A and500B for the encoder workflow in the test stage by meta-learning, according to embodiments.

As shown inFIG.5A, the meta-NIC architecture500A includes an NN Encoding module505, an Encoding module,510, a Decoding module515, a meta-NIC Decoding module520, a Compute Distortion Loss module525, and a Back-Propagation module530.

As shown inFIG.5B, the meta-NIC architecture500B includes an NN Encoding module535, an Encoding module,540, a Decoding module545, a Substitutional Perturbation Generation module550, a meta-NIC Decoding module550, a Compute Distortion Loss module555, and a Back-Propagation module565.

InFIG.5A, given an input image x, and given an original target QFΛ, the NIC encoder505may generate the encoded latentyand the encoded bitstream. Furthermore, the encoder510may compute the decoded latent ŷ. The original target QFΛindicates the target compression quality, including the target quality metric, the target bitrate, etc. For example, assuming there are a number of q quality metrics D1(x,x), . . . , Dq(x,x) in total (e.g., PSNR, SSIM, etc.), the overall quality metric may be generally represented as a weighted combination of them as follows:

where weights wi≥0. The original target QFΛmay be a single vector comprising of all the weights wiand the target trade-off hyperparameter λ:Λ=[w1, . . . , wq, λ].

Next, the decoded latent ŷ, from the Decoding module510, may be passed into the meta-NIC Decoding module520, which operates the same as the meta-NIC Decoding module in the decoder described above. The meta-NIC Decoding520module computes the reconstructed imagexbased on ŷ and the target QF Λ. The initial Λ may be simply set to be the same as the original target QFΛ. Then the reconstruction loss (e.g., MSE or MSSSIM) between the original input x and the reconstructedxmay be computed in the Compute Distortion Loss module525. Then the gradient of the loss may be computed and back-propagated by the Back-Propagation module530to update the target QF Λ. Based on the updated target QF Λ and the decoded latent ŷ, the meta-NIC Decoding module520may compute the updated reconstructed imagex. The system may go through several such iterations, and finally obtain the updated target QF Λ, which may be sent together with the compressed bitstreamy′ to the decoder side.

In some embodiments, the updated target QF Λ may further go through encoding processes like quantization and entropy encoding to further reduce the transmission overhead.

InFIG.5B, the decoded latent ŷ, from the Decoding module545, may be passed through the Substitutional Perturbation Generation module550that is the same as the decoder side, which computes a substitutional latent ŷ′ based on latent ŷ and the QF Λ. This substitutional latent ŷ′ may then be passed into the meta-NIC Decoding module555instead of the original ŷ to compute the reconstructed imagex.

The proposed meta-NIC framework allows arbitrary smooth QF Λ on the decoder side for reconstruction. In other words, the processing workflow described above will compute the compressed representation and the reconstructed image to fit the arbitrary smooth target QF Λ.

According to some embodiments, a training process may be implemented and aims at learning the meta-NIC encoder, the SDP θsd(j) and ADP θad(j),j=1, . . . , M for the meta-NIC decoder, the ADP Prediction NN (model parameters denoted as Φd), as well as the parameters for the Substitutional Perturbation Generation module (e.g., seeFIGS.4B and5B).

The meta-NIC encoder and meta-NIC decoder may be trained in an end-to-end fashion through a Model-Agnostic Meta-Learning (MAML) mechanism. Once the underlying meta-NIC encoder and decoder are trained, the Substitutional Perturbation Generation module may be trained by fixing the meta-NIC encoder and decoder parameters while minimizing a fitting loss to compute a substitutional latent representation ŷ′ from the decoded latent ŷ, where the substitutional latent representation ŷ′ are better than the original decoded latent ŷ measured by some metrics (e.g., may generate reconstructed image better than the original decoded latent ŷ the meta-NIC Decoding module with less distortion or better perceptive quality).

FIG.6is a flowchart of an embodiments of a process600for neural image compression using substitutional quality factor learning in a latent space.

As shown inFIG.6, at operation610of process600, a compressed bitstream and a target compression quality are received. The process proceeds to operation620, where a decoded latent representation of the compressed bitstream is calculated, as shown inFIGS.4A and4B. That is, the decoded latent representation may be used to calculate the reconstructed image.

The process proceeds to operation630, where a shared feature may be computed based on the SDP. The process proceeds to operation640, where the ADP is computed for the one or more layers of the convolution neural network, as shown inFIG.4C. As such, an output tensor may be computed based the estimated ADP and the shared feature.