Patent ID: 12217467

DETAILED DESCRIPTION

Subject matter will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments. Subject matter may, however, be embodied in a variety of different forms and, therefore, covered or claimed subject matter is intended to be construed as not being limited to any exemplary embodiments set forth herein; exemplary embodiments are provided merely to be illustrative. Likewise, a reasonably broad scope for claimed or covered subject matter is intended. Among other things, for example, the subject matter may be embodied as methods, devices, components, or systems. The following detailed description is, therefore, not intended to be taken in a limiting sense.

The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. Likewise, the term “embodiments of the present invention” does not require that all embodiments of the invention include the discussed feature, advantage, or mode of operation.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising,”, “includes” and/or “including”, when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The following detailed description includes the best currently contemplated mode or modes of carrying out exemplary embodiments of the invention. The description is not to be taken in a limiting sense but is made merely for the purpose of illustrating the general principles of the invention, since the scope of the invention will be best defined by the allowed claims of any resulting patent.

The described features, structures, or characteristics of the invention may be combined in any suitable manner in accordance with the aspects and one or more embodiments of the invention. In the following description, numerous specific details are recited to provide an understanding of various embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the aspects of the invention.

Disclosed is a system and method that can overcome the drawbacks and challenges with the known image compression codecs by providing a codec that reduces the dimensionality of the input images while retaining spatial information. Disclosed is a Deep Learning codec that can apply a quantization operation during the training process. Referring toFIG.1which is a block diagram showing an exemplary embodiment of the disclosed system100. The system100can include processor110and a memory120connected through a system bus (not shown). The processor110can be any logic circuitry that responds to, and processes instructions fetched from the memory120. The processor can read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform one or more of the methodologies discussed herein. The memory120may include one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the processor110. The memory120can include modules which upon execution by the processor can perform one or more of the methodologies discussed herein. The modules can be a set of instructions including software, a program, an application, or other executable code for causing the processor to perform one or more of the methodologies discussed herein. The disclosed memory120can include a Quantized autoencoder network130which upon execution by the processor can provide for compression and decompression of image data with high fidelity. The Quantized autoencoder network130, also referred to herein as the codec, can include three essential modules: the encoder convolutional neural network module140(the encoder network), the intermediate convolutional neural network module150(the bottleneck network), and the decoder convolutional neural network module160(the decoder network).

Referring toFIG.2which is a block diagram illustrating the methodology disclosed herein. The input image210can be compressed to the output image220by the execution of the Quantized autoencoder network130. The Encoder convolutional neural network module140can reduce the dimensions of the input image210through learned filters. An implementation of the Encoder convolutional neural network module140is shown inFIG.3.FIG.3is a flowchart showing the input image X that can be compressed multiple times by passing through the compression blocks. The Encoder convolutional neural network module140upon execution by the processor can fed the input image data into the compression block310of the encoder140where a series of filters can be applied to the original image. In one implementation, the compression block310can include filters such as Strides Conv2D320, Conv2D330, Batch Normalization340, and ELU Activation350.

The most common type of convolution that can be used is the 2D convolution layer, abbreviated as conv2D. A filter or a kernel in a conv2D layer has a height and a width. These kernels are generally smaller than the input image, so should be moved across the whole image. Conv2D is known in the art and Strided define an overlap between applying operations, strided conv2d can specify if what is the distance between consecutive applications of convolutional filters. Batch normalization is a popular and effective technique that consistently accelerates the convergence of deep networks. The ELU filter or Exponential Linear Unit is a function that tends to converge cost to zero faster and produce more accurate results. Different from other activation functions, ELU has an extra alpha constant which should be a positive number. One novel aspect of using filters in the encoder module of this invention is that the filter configuration is flexible.FIG.3shows three consecutive blocks310, however, the detailed structure of the encoder module may not be fixed but can grow throughout the training process. The compression ratio can increase fourfold for each growth stage. The growth of the encoder model140may also depend on the performance during training. If the model is capable of handling the current compression rate with sufficient accuracy and fidelity, then the model can be further strained by adding another compression block.

Again, referring toFIG.2, the bottleneck module150can receive the encoder's output i.e., the unquantized compression representation of the input image data. The bottleneck module150upon execution by the processor can apply quantization with a custom gradient function to allow for gradient flow. Referring toFIG.4which illustrates an implementation of the bottleneck module150, a novel aspect of this invention is that the bottleneck network module utilizes fake quantization to allow quantization to occur during the training process. Fake quantization is a feature known in the art and commercially available. TensorFlow which is a free and open-source software library for machine learning and artificial intelligence utilizes fake quantization module. The quantization using a fake quantization module allows for gradient flow through the quantization process. The criteria for when to add another compression block are part of the novelty of this invention. The specific compression blocks to add are yet another novel aspect of this invention. This reduced image can be further reduced via one or more traditional codecs such as OCT/JPEG 2000, or losslessly via the image file format PNG (Portable Network Graphics). The output of the bottleneck network is a reduced image representation of the input image data.FIG.4shows the reduced image (float) Y410which is the output of the encoder network and received by the bottleneck network150, a Conv2D filter420, a fake quantization module430, a secondary encoder440, a secondary decoder450, and an output of the bottleneck network which is a quantized reduced image (8-bits) Y460.

Again, referring toFIG.2which shows the decoder module160can receive the compressed quantized representation from the bottleneck network150. The decoder network160can reconstruct the quantized compressed representation of the input image data. As shown inFIG.5, which shows an implementation of the decoder network160, the decoder network160can initially upscale the code image N times, where N is the current number of compression blocks. The upscaled image can then be fed through a residual network with a number of residual blocks. In one example, ten forward blocks can be in the residual network. The upscaled image forks at the start and is summed to the output of the last residual output, allowing for gradient flow to the encoder and bottleneck networks. This network effectively accounts for the JPEG (joint photographic experts group) compression losses and increases the quality of the output image.FIG.5shows the reduced image (8-bits) Y obtained from the bottleneck network, upSampling2D function520, upSampling2D function520, and upSampling2D function540all connected in series as shown inFIG.5, upscaled image550, forward block560, forward block570, and the reconstructed image580. The forward block is also illustrated inFIG.5including the Conv2D filter, Batch normalization filter, and the ELU function.

Greedy Training: The compression ratio of the model in this invention grows throughout the training process. By continuing to grow the compression ratio through the training process, no matter the dataset that is used, the model can achieve a superior compression ratio with minimal losses in the image quality and fidelity.

Composite Loss function: In the training phase, the following are the loss or objective functions to minimize:

Lr⁢e⁢c⁢o⁢n⁢s⁢t⁢r⁢u⁢c⁢t⁢i⁢o⁢n=1N⁢∑⁢(f⁡(x)-x)2⁢⁢Lc⁢o⁢m⁢p⁢r⁢e⁢s⁢s⁢i⁢o⁢n=1N⁢∑⁢(Y^-Q)2⁢⁢Where⁢⁢the⁢⁢total⁢⁢loss⁢⁢of⁢⁢our⁢⁢Network⁢⁢is⁢:⁢⁢Lt⁢o⁢t⁢a⁢l=Lr⁢e⁢c⁢o⁢n⁢s⁢t⁢r⁢u⁢c⁢t⁢i⁢o⁢n+Lc⁢o⁢m⁢p⁢r⁢e⁢s⁢s⁢i⁢o⁢n

Where N in both cases is the total number of data points in the squared term, this loss function is known as the mean squared error, f(x) is the function representing the whole model, wherein x is the input image. The output of this function is the final reconstructed image. Y is the output of the bottleneck network, and Q is the input of the JPEG compression layer.

While the reconstruction loss is widely known in prior art, the disclosed codec includes the compression loss. By including compression loss, the encoder network is able to change the input image to better fit the JPEG compression algorithm and thus reduce the losses caused by the compression.

Advantages and benefits of the invention include speed and performance. The performance of the model which is the subject of this invention was tested on the same dataset as that of Feng Jiang, et al.,An End-to-End Compression Framework Based on Convolutional Neural Networks, IEEE transactions on circuits and systems for video technology, Aug. 2, 2017. This particular dataset is used as a benchmark for a majority of works in this field. Thus, the performance of the model which is the subject of this invention can be compared fairly with previous state-of-the-art solutions. The results of such a comparison are shown inFIG.6. The model which is the subject of this invention is abbreviated as NQAE (Novel Quantized Auto Encoder). The known model is referred to as Jiang's. The performance of the NQAE model was significantly better than the previous state of the art solution in comparison as shown inFIG.6. The comparison used the PSNR (Peak Signal to Noise Ratio) image quality metric. The performance of the NQAE model was substantially superior as measured by the structural similarity index metric, as shown inFIG.7.

The encoder's compression ratio grows dynamically throughout the training process. The benefit of this aspect of the invention is that no matter what dataset is input, the model can achieve a superior compression ratio with meager losses in the image quality and fidelity as compared to all known existing solutions.

While the foregoing written description of the invention enables one of ordinary skill to make and use what is considered presently to be the best mode thereof, those of ordinary skill will understand and appreciate the existence of variations, combinations, and equivalents of the specific embodiment, method, and examples herein. The invention should therefore not be limited by the above-described embodiment, method, and examples, but by all embodiments and methods within the scope and spirit of the invention as claimed.