Office Action 1: Non-Final Rejection (2024-08-09)
Rejection types: 103 - Obviousness

CLAIMS:
WHAT IS CLAIMED IS:

1\. A method for image processing, comprising:
decoding, from a coded bitstream that carries a compressed image, an index that points a value in a set of values of a parameter, a value change of the parameter adjusting a compression rate of the compressed image, the compressed image being generated by a neural network based encoder based on the parameter;
inputting the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF), the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and
generating, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter.

2\. The method of claim 1, wherein the generating the computer vision task result comprises:
converting, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor;
inputting the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and
generating, by the second neural network, the computer vision task result according to the compressed image and the tensor.

3\. The method of claim 2, wherein the first neural network comprises one or more convolution layers.

4\. The method of claim 2, wherein the first neural network comprises a convolution layer with an activation function.

5\. The method of claim 2, wherein the second neural network is configured to generate the computer vision task result without generating a reconstructed image from the compressed image.

6\. The method of claim 2, wherein the second neural network is configured to generate a reconstructed image from the compressed image, and generate the computer vision task result from the reconstructed image.

7\. The method of claim 1, wherein the neural network based encoder is based on an encoder model in a neural image compression (NIC) framework, and the multi-rate compression domain computer vision task decoder is based on a decoder model in the NIC framework, the NIC framework is trained end-to-end.

8\. The method of claim 1, wherein a decoder model of the multi-rate compression domain computer vision task decoder is trained separate from an encoder model of the neural network based encoder.

9\. The method of claim 1, wherein the parameter is a hyperparameter for weighting a distortion in a calculation of a rate distortion loss.

10\. The method of claim 1, wherein the computer vision task comprises at least one of image classification, image denoising, object detection and super resolution.

11\. An apparatus for image processing, comprising processing circuitry configured to:
decode, from a coded bitstream that carries a compressed image, an index that points a value in a set of values of a parameter, a value change of the parameter adjusting a compression rate of the compressed image, the compressed image being generated by a neural network based encoder based on the parameter;
input the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF), the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and
generate, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter.

12\. The apparatus of claim 11, wherein the processing circuitry is configured to:
convert, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor;
input the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and
generate, by the second neural network, the computer vision task result according to the compressed image and the tensor.

13\. The apparatus of claim 12, wherein the first neural network comprises one or more convolution layers.

14\. The apparatus of claim 12, wherein the first neural network comprises a convolution layer with an activation function.

15\. The apparatus of claim 12, wherein the second neural network is configured to generate the computer vision task result without generating a reconstructed image from the compressed image.

16\. The apparatus of claim 12, wherein the second neural network is configured to generate a reconstructed image from the compressed image, and generate the computer vision task result from the reconstructed image.
17\. The apparatus of claim 11, wherein the neural network based encoder is based on an encoder model in a neural image compression (NIC) framework, and the multi- rate compression domain computer vision task decoder is based on a decoder model in the NIC framework, the NIC framework is trained end-to-end.

18\. The apparatus of claim 11, wherein a decoder model of the multi-rate compression domain computer vision task decoder is trained separate from an encoder model of the neural network based encoder.

19\. The apparatus of claim 11, wherein the parameter is a hyperparameter for weighting a distortion in a calculation of a rate distortion loss.

20\. The apparatus of claim 11, wherein the computer vision task comprises at least one of image classification, image denoising, object detection and super resolution. 37 38 39 40

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements submitted on 9/15/2023 and 3/22/2023 are in compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statement(s) are being considered by the examiner.
Claim Objections
Claims 1 and 11 are objected to because of the following informalities:
Regarding claim 1, the claim recites in lines 2-3: “an index that points a value in a set…”, however this should seemingly be --an index that points to a value in a set…-- for grammatical correctness.
Claim 11, claiming an apparatus form equivalent of claim 1, is objected to for analogous reasons.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 10-11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20200250003 A1) (hereinafter Yang) in view of Anderson et al. (US 20220224914 A1) (hereinafter Anderson).
Regarding claim 1, Yang discloses:
A method for image processing, comprising:
decoding, from a coded bitstream that carries a compressed image, [See Yang, ¶ 0281, 0293 discloses compressed visual data provided as input to CNN may first be partially decoded to separate and extract different syntax elements (e.g., motion vectors, macroblock (MB) coding modes, quantized prediction residuals), thus producing a subset of partial compression data.] discloses an index that points a value in a set of values of a parameter, [See Yang, ¶ 0259-0261 discloses an image format schema using an array-based database manager to specify parameters about the array that can be used to arrange image data.]  a value change of the parameter adjusting a compression rate of the compressed image, [See Yang, ¶ 0243 discloses that images can be compressed either in a lossless or lossy manner. Lossless image compression preserves all the information in the image, while lossy compression takes advantage of visual perception and statistical properties to achieve better compression rates, but results in some data being lost.] the compressed image being generated by a neural network based encoder based on the parameter; [See Yang, ¶ 0272 discloses that neural networks can be designed to operate on compressed visual data directly, thus eliminating the need to decompress visual data before it can be processed.  Further, that visual metadata could be compressed by pre-training a convolutional neural network (CNN) to classify visual metadata, replacing long strings of visual metadata with shorter symbols (e.g., pre-defined human codes), performing multi-scale de-duplication on the visual metadata, and finally compressing the resulting visual metadata using a compression algorithm.]
Yang does not appear to explicitly disclose:
inputting the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF),
the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and
generating, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter.
However, Anderson discloses:
inputting the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF), [See Anderson, ¶ 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map. As an example, the second parameter map may be represented as a one-hot encoded vector including four elements, each representing whether a target frame is encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame.] the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses  a frame decoder 414 may be configured as a neural network model.  Furthermore, that at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task.] according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and [See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map.]
generating, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter. [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task; See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang to add the teachings of Anderson in order to improve coding efficiency by permitting multiple rates – allowing encoding and decoding of different B-frame types with a single B-frame compression model.  This makes deployment easier since multiple models with different sets of parameters do not have to be constructed for each B-frame type.
Regarding claim 10, Yang in view of Anderson discloses all the limitations of claim 1.
Yang discloses:
wherein the computer vision task comprises at least one of image classification, image denoising, object detection and super resolution. [See Yang, ¶ 0226 discloses image classification is a common visual data operation that uses a neural network to identify the contents of an image. For example, in machine learning, a convolutional neural network (CNN) is a type of feed-forward artificial neural network where the input is generally assumed to be an image. CNNs are commonly used for image classification, where the goal is to determine the contents of an image with some level of confidence. For example, a CNN is first trained for a specific classification task using a set of images whose object classes or features have been labeled, and the CNN can then be used to determine the probability of whether other images contain the respective object classes.]
Regarding claim 11, this claim recites analogous limitations to claim 1 in the form of “an apparatus” rather than “a method”, and is therefore rejected on the same premise.  Please see examiner’s earlier rejection of claim 1 for corresponding motivation statement.
Further, claim 11 recites the following limitations which are not explicitly found from claim 1, but are addressed as follows:
Yang discloses:
An apparatus for image processing, comprising processing circuitry configured to: [See Yang, ¶ 0114, 0148 discloses a processing system includes any system that has a processor, such as, for example; a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.]
Regarding claim 20, this claim recites analogous limitations to claim 10, and is therefore rejected on the same premise.
Claim(s) 2-8 and 12-18 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Anderson in view of Wang et al. (US 20200280730 A1) (hereinafter Wang).
Regarding claim 2, Yang in view of Anderson discloses all the limitations of claim 1.
Yang in view of Anderson does not appear to explicitly disclose:
wherein the generating the computer vision task result comprises:
converting, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor;
inputting the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and
generating, by the second neural network, the computer vision task result according to the compressed image and the tensor.
However, Wang discloses:
wherein the generating the computer vision task result comprises:
converting, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor; [See Wang, ¶ 0406, 0407, 0412 discloses a convolutional neural network forming tensors from low and high resolution image parameters.]
inputting the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and [See Wang, ¶ 0072 discloses the method is performed at a first network node. Furthermore, optionally the hierarchical algorithm may be transmitted to a second network node.]
generating, by the second neural network, the computer vision task result according to the compressed image and the tensor. [See Wang, ¶ 0406 discloses The task of the example based model, in some embodiments a single image super resolution network, is to estimate a high resolution image given a low resolution image that is downscaled or downsampled from a corresponding high resolution image. Further, that estimation can be performed, for example by training a classifier or some visual comparison, and optionally this could be parametised for the super resolution process in some embodiments. In general, both the low and high resolution image have C colour channels, thus can be represented as real-valued tensors of size H×W×C and rH×rW×C respectively. In some embodiments, rather than recovering an high resolution image from an upscaled and interpolated version of the low resolution image, instead the low resolution image is used by a three layer convolutional network to avoid upscaling before the low-resolution image is fed into the network. In the network, a sub-pixel convolution layer is applied to upscale the low resolution feature maps to produce a high resolution image using super resolution.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson to add the teachings of Wang in order to introduce non-linearity to a model and allow the neural network to learn and represent complex patterns.
Regarding claim 3, Yang in view of Anderson in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the first neural network comprises one or more convolution layers. [See Anderson, ¶ 0098 discloses some components of the compression model 500 may be configured as a convolutional neural network model (CNN) that include a convolutional filter having weights repeatedly applied to a moving window of pixels of an input tensor. In one instance, the convolutional layer may include a set of convolutional filters each corresponding to a respective bitrate level, and different regions of the input tensor may be applied with different convolutional filters depending on the value of the encoding parameter for the region in the parameter map.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 4, Yang in view of Anderson in view of Wang discloses all the limitations of claim 2.
Although Anderson discloses the use of a compression model configured as a convolutional neural network in combination with a parameter map for specifying multiple rate encoding schemes, Yang in view of Anderson does not appear to explicitly disclose:
wherein the first neural network comprises a convolution layer with an activation function.
However, Wang discloses:
wherein the first neural network comprises a convolution layer with an activation function. [See Wang, ¶ 0199, 0412, 0417 discloses that as a part of a convolutional neural network-based encoding, an activation function can optionally be employed.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 5, Yang in view of Anderson in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the second neural network is configured to generate the computer vision task result without generating a reconstructed image from the compressed image. [See Anderson, ¶ 0090-0091 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task. The state information for a neural network component may be uninterpretable in the sense that it can be processed by the neural network component to store and retrieve any type of information necessary for performing the task, rather than having a specific, interpretable meaning on the information.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 6, Yang in view of Anderson in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the second neural network is configured to generate a reconstructed image from the compressed image, and generate the computer vision task result from the reconstructed image. [See Anderson, ¶ 0116 discloses a B-frame may be encoded and decoded in a hierarchical manner. For example, a B-frame may be encoded and decoded as a B1-frame, a B2-frame, or a B3-frame. A B1-frame is encoded and decoded using previously reconstructed frames that were reconstructed as an I-frame and/or a P-frame, a B2-frame is encoded and decoded using previously reconstructed frames that include one frame reconstructed as an I-frame or P-frame and another frame reconstructed as a B1-frame, and a B3-frame is encoded and decoded using previously reconstructed frames that were reconstructed as a B1-frame and a B2-frame.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 7, Yang in view of Anderson discloses all the limitations of claim 1.
Wang discloses:
wherein the neural network based encoder is based on an encoder model in a neural image compression (NIC) framework, and the multi-rate compression domain computer vision task decoder is based on a decoder model in the NIC framework, the NIC framework is trained end-to-end. [See Wang, ¶ 0594-0595 discloses a neural network training process as a full end-to-end system encapsulating how visual data being provided to the encoder 2830 will be decoded by the decoder 2850.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 8, Yang in view of Anderson discloses all the limitations of claim 1.
Wang discloses:
wherein a decoder model of the multi-rate compression domain computer vision task decoder is trained separate from an encoder model of the neural network based encoder. [See Wang, ¶ 0594-0595 discloses a neural network training process as a full end-to-end system encapsulating how visual data being provided to the encoder 2830 will be decoded by the decoder 2850.  (Examiner’s note: it is within the level of ordinary skill to understand that “end-to-end” routinely and conventionally corresponds with training being separated, which is further evidenced by WO2021/220008A1, which notes on pg. 83, section 1.2.2 “Training of the segmentation module, if it is parameterized as a neural network, may be performed separately or during the training of the compression network itself – referred to as end-to-end.)]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 12, this claim recites analogous limitations to claim 2, and is therefore rejected on the same premise.
Regarding claim 13, this claim recites analogous limitations to claim 3, and is therefore rejected on the same premise.
Regarding claim 14, this claim recites analogous limitations to claim 4, and is therefore rejected on the same premise.
Regarding claim 15, this claim recites analogous limitations to claim 5, and is therefore rejected on the same premise.
Regarding claim 16, this claim recites analogous limitations to claim 6, and is therefore rejected on the same premise.
Regarding claim 17, this claim recites analogous limitations to claim 7, and is therefore rejected on the same premise.
Regarding claim 18, this claim recites analogous limitations to claim 8, and is therefore rejected on the same premise.
Claim(s) 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Anderson in view of Li et al. (US 20200280717 A1) (hereinafter Li).
Regarding claim 9, Yang in view of Anderson discloses all the limitations of claim 1.
Yang in view of Anderson does not appear to explicitly disclose:
wherein the parameter is a hyperparameter for weighting a distortion in a calculation of a rate distortion loss.
However, Li discloses:
wherein the parameter is a hyperparameter for weighting a distortion in a calculation of a rate distortion loss. [See Li, ¶ 0115, 0181 discloses in addition to adjusting the number of features on each convolution layer, the architecture of FIGS. 9 and 12, and their variations described herein can be designed so that a global feature adjustment rate r may be applied across all layers for reducing/expanding the number of features. This feature adjustment rate can be a configurable hyper-parameter. After applying the feature adjustment rate, the number of features can be reduced/expended from n to n*r.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson to add the teachings of Li in order to identify quantization parameters that can be used to identify the tradeoff between rate and distortion.]
Regarding claim 19, this claim recites analogous limitations to claim 9, and is therefore rejected on the same premise.
Conclusion


NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements submitted on 9/15/2023 and 3/22/2023 are in compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statement(s) are being considered by the examiner.
Response to Arguments
Applicant's arguments filed 11/05/2024 have been fully considered but they are drawn towards newly amended claim language.
Regarding Rejections under 35 U.S.C. § 103,
Applicant contends that the cited prior art fails to disclose newly amended limitations of independent claims 1, 11, and 21 including at least “the parameter weighting a distortion and a rate in a calculation of a rate-distortion (R-D) loss”.
See the rejection below for how the cited art in light of new/existing references reads on the newly amended language as well as the examiner’s interpretation of the cited art in view of the presented claim set.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 10-11, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 20200250003 A1) (hereinafter Yang) in view of Anderson et al. (US 20220224914 A1) (hereinafter Anderson) in view of Besenbruch et al. (US 20230154055 A1; priority to PCT/GB2021/051041, filed 4/29/2021) (hereinafter Besenbruch).
Regarding claim 1, Yang discloses:
A method for image processing, comprising:
decoding, from a coded bitstream that carries a compressed image, [See Yang, ¶ 0281, 0293 discloses compressed visual data provided as input to CNN may first be partially decoded to separate and extract different syntax elements (e.g., motion vectors, macroblock (MB) coding modes, quantized prediction residuals), thus producing a subset of partial compression data.] discloses an index that points a value in a set of values of a parameter, [See Yang, ¶ 0259-0261 discloses an image format schema using an array-based database manager to specify parameters about the array that can be used to arrange image data.]  a value change of the parameter adjusting a compression rate of the compressed image, [See Yang, ¶ 0243 discloses that images can be compressed either in a lossless or lossy manner. Lossless image compression preserves all the information in the image, while lossy compression takes advantage of visual perception and statistical properties to achieve better compression rates, but results in some data being lost.] the compressed image being generated by a neural network based encoder based on the parameter; [See Yang, ¶ 0272 discloses that neural networks can be designed to operate on compressed visual data directly, thus eliminating the need to decompress visual data before it can be processed.  Further, that visual metadata could be compressed by pre-training a convolutional neural network (CNN) to classify visual metadata, replacing long strings of visual metadata with shorter symbols (e.g., pre-defined human codes), performing multi-scale de-duplication on the visual metadata, and finally compressing the resulting visual metadata using a compression algorithm.]
Yang does not appear to explicitly disclose:
inputting the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF),
the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and
generating, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter,
However, Anderson discloses:
inputting the value of the parameter to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF), [See Anderson, ¶ 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map. As an example, the second parameter map may be represented as a one-hot encoded vector including four elements, each representing whether a target frame is encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame.] the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses  a frame decoder 414 may be configured as a neural network model.  Furthermore, that at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task.] according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and [See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map.]
generating, by the multi-rate compression domain computer vision task decoder, a computer vision task result according to the compressed image in the coded bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter, [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task; See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang to add the teachings of Anderson in order to improve coding efficiency by permitting multiple rates – allowing encoding and decoding of different B-frame types with a single B-frame compression model.  This makes deployment easier since multiple models with different sets of parameters do not have to be constructed for each B-frame type.
Yang in view of Anderson does not appear to explicitly disclose:
the parameter weighting a distortion and a rate in a calculation of a rate-distortion (R-D) loss;
However, Besenbruch discloses:
the parameter weighting a distortion and a rate in a calculation of a rate-distortion (R-D) loss; [See Besenbruch, ¶ 0520 discloses a loss used for neural network training, being L = D + λ*R, where D is a distortion function, λ is a weighting factor, and R is the rate loss.  R is related to entropy, and both D and R are differentiable functions.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson to add the teachings of Besenbruch in order to permit calculation of loss resembling rate-distortion trade off in neural network training.
Regarding claim 10, Yang in view of Anderson in view of Besenbruch discloses all the limitations of claim 1.
Yang discloses:
wherein the computer vision task comprises at least one of image classification, image denoising, object detection and super resolution. [See Yang, ¶ 0226 discloses image classification is a common visual data operation that uses a neural network to identify the contents of an image. For example, in machine learning, a convolutional neural network (CNN) is a type of feed-forward artificial neural network where the input is generally assumed to be an image. CNNs are commonly used for image classification, where the goal is to determine the contents of an image with some level of confidence. For example, a CNN is first trained for a specific classification task using a set of images whose object classes or features have been labeled, and the CNN can then be used to determine the probability of whether other images contain the respective object classes.]
Regarding claim 11, this claim recites analogous limitations to claim 1 in the form of “an apparatus” rather than “a method”, and is therefore rejected on the same premise.  Please see examiner’s earlier rejection of claim 1 for corresponding motivation statement.
Further, claim 11 recites the following limitations which are not explicitly found from claim 1, but are addressed as follows:
Yang discloses:
An apparatus for image processing, comprising processing circuitry configured to: [See Yang, ¶ 0114, 0148 discloses a processing system includes any system that has a processor, such as, for example; a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.]
encode an index that points to the value in the set of values of the parameter. [See Yang, ¶ 0259-0261 discloses an image format schema using an array-based database manager to specify parameters about the array that can be used to arrange image data.]
Anderson discloses:
input a value of a parameter to a multi-rate image compression encoder, the multi-rate image compression encoder comprising one or more neural networks for encoding images according to a set of values of the parameter that are used for generating compressed images at multiple different compression rates, [See Anderson, ¶ 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map. As an example, the second parameter map may be represented as a one-hot encoded vector including four elements, each representing whether a target frame is encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame; See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses  a frame decoder 414 may be configured as a neural network model.  Furthermore, that at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task.]
encode, by the multi-rate image compression encoder an input image into a compressed image according to a corresponding compression rate from the multiple different compression rates based on the value of the parameter; and [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task; See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang to add the teachings of Anderson in order to improve coding efficiency by permitting multiple rates – allowing encoding and decoding of different B-frame types with a single B-frame compression model.  This makes deployment easier since multiple models with different sets of parameters do not have to be constructed for each B-frame type.
Besenbruch discloses:
the parameter weighting a distortion and a rate in a calculation of a rate-distortion (R-D) loss; [See Besenbruch, ¶ 0520 discloses a loss used for neural network training, being L = D + λ*R, where D is a distortion function, λ is a weighting factor, and R is the rate loss.  R is related to entropy, and both D and R are differentiable functions.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson to add the teachings of Besenbruch in order to permit calculation of loss resembling rate-distortion trade off in neural network training.
Regarding claim 21, Yang discloses:
A method of processing visual media data, the method comprising:
processing a bitstream of the visual media data according to a format rule, wherein the bitstream includes a syntax element indicating an index that points to a value in a set of values of a parameter, [See Yang, ¶ 0259-0261 discloses an image format schema using an array-based database manager to specify parameters about the array that can be used to arrange image data.]  a value change of the parameter adjusting a compression rate of a compressed image in the bitstream, the compressed image being generated based on the parameter, [See Yang, ¶ 0243 discloses that images can be compressed either in a lossless or lossy manner. Lossless image compression preserves all the information in the image, while lossy compression takes advantage of visual perception and statistical properties to achieve better compression rates, but results in some data being lost.]
the format rule specifies that:
the value of the parameter is input to a multi-rate compression domain computer vision task decoder in a compression domain computer vision task framework (CDCVTF), [See Anderson, ¶ 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map. As an example, the second parameter map may be represented as a one-hot encoded vector including four elements, each representing whether a target frame is encoded and decoded as a P-frame, a B1-frame, a B2-frame, or a B3-frame.] the multi-rate compression domain computer vision task decoder comprising one or more neural networks for performing a computer vision task from compressed images [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses  a frame decoder 414 may be configured as a neural network model.  Furthermore, that at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task.]  according to corresponding values of the parameter that are used for generating the compressed images at multiple different compression rates; and [See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.  The parameter map may at least in part incorporate the frame type of the target frame, and in one embodiment, may include at least a first parameter map indicating desired bitrate levels and a second parameter map indicating the frame type of the target frame concatenated to the first parameter map.]
a computer vision task result is generated according to the compressed image in the bitstream which is compressed at a corresponding compression rate from the multiple different compression rates based on the value of the parameter. [See Anderson, ¶ 0047, 0051, 0058, 0090, 0098 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task; See Anderson, ¶ 0005, 0025, 0116 discloses that a parameter map encodes whether a target frame is processed as a P-frame, a B1-frame, a B2-frame, or a B3-frame using the compression model 600. The ability to have multiple rates allows encoding and decoding of different B-frame types (e.g., B1, B2, B3) with a single B-frame compression model that takes as input the rate and frame type in the parameter map.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang to add the teachings of Anderson in order to improve coding efficiency by permitting multiple rates – allowing encoding and decoding of different B-frame types with a single B-frame compression model.  This makes deployment easier since multiple models with different sets of parameters do not have to be constructed for each B-frame type.
the parameter weighting a distortion and a rate in a calculation of a rate-distortion (R-D) loss; and  [See Besenbruch, ¶ 0520 discloses a loss used for neural network training, being L = D + λ*R, where D is a distortion function, λ is a weighting factor, and R is the rate loss.  R is related to entropy, and both D and R are differentiable functions.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson to add the teachings of Besenbruch in order to permit calculation of loss resembling rate-distortion trade off in neural network training.
Claim(s) 2-8, 12-14, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Anderson in view of Besenbruch in view of Wang et al. (US 20200280730 A1) (hereinafter Wang).
Regarding claim 2, Yang in view of Anderson in view of Besenbruch discloses all the limitations of claim 1.
Yang in view of Anderson in view of Besenbruch does not appear to explicitly disclose:
wherein the generating the computer vision task result comprises:
converting, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor;
inputting the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and
generating, by the second neural network, the computer vision task result according to the compressed image and the tensor.
However, Wang discloses:
wherein the generating the computer vision task result comprises:
converting, by a first neural network in the multi-rate compression domain computer vision task decoder, the value of the parameter to a tensor; [See Wang, ¶ 0406, 0407, 0412 discloses a convolutional neural network forming tensors from low and high resolution image parameters.]
inputting the tensor to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and [See Wang, ¶ 0072 discloses the method is performed at a first network node. Furthermore, optionally the hierarchical algorithm may be transmitted to a second network node.]
generating, by the second neural network, the computer vision task result according to the compressed image and the tensor. [See Wang, ¶ 0406 discloses The task of the example based model, in some embodiments a single image super resolution network, is to estimate a high resolution image given a low resolution image that is downscaled or downsampled from a corresponding high resolution image. Further, that estimation can be performed, for example by training a classifier or some visual comparison, and optionally this could be parametised for the super resolution process in some embodiments. In general, both the low and high resolution image have C colour channels, thus can be represented as real-valued tensors of size H×W×C and rH×rW×C respectively. In some embodiments, rather than recovering an high resolution image from an upscaled and interpolated version of the low resolution image, instead the low resolution image is used by a three layer convolutional network to avoid upscaling before the low-resolution image is fed into the network. In the network, a sub-pixel convolution layer is applied to upscale the low resolution feature maps to produce a high resolution image using super resolution.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson in view of Besenbruch to add the teachings of Wang in order to introduce non-linearity to a model and allow the neural network to learn and represent complex patterns.
Regarding claim 3, Yang in view of Anderson in view of Besenbruch in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the first neural network comprises one or more convolution layers. [See Anderson, ¶ 0098 discloses some components of the compression model 500 may be configured as a convolutional neural network model (CNN) that include a convolutional filter having weights repeatedly applied to a moving window of pixels of an input tensor. In one instance, the convolutional layer may include a set of convolutional filters each corresponding to a respective bitrate level, and different regions of the input tensor may be applied with different convolutional filters depending on the value of the encoding parameter for the region in the parameter map.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 4, Yang in view of Anderson in view of Besenbruch in view of Wang discloses all the limitations of claim 2.
Although Anderson discloses the use of a compression model configured as a convolutional neural network in combination with a parameter map for specifying multiple rate encoding schemes, Yang in view of Anderson does not appear to explicitly disclose:
wherein the first neural network comprises a convolution layer with an activation function.
However, Wang discloses:
wherein the first neural network comprises a convolution layer with an activation function. [See Wang, ¶ 0199, 0412, 0417 discloses that as a part of a convolutional neural network-based encoding, an activation function can optionally be employed.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 5, Yang in view of Anderson in view of Besenbruch in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the second neural network is configured to generate the computer vision task result without generating a reconstructed image from the compressed image. [See Anderson, ¶ 0090-0091 discloses at least a part of the information processed by a compression model (e.g., any inputs or outputs of the compression model 200 of FIG. 2, compression model 500 of FIG. 5, or compression model 600 of FIG. 6) is represented as state information that captures and propagates information used by a neural network component to perform an assigned task. The state information for a neural network component may be uninterpretable in the sense that it can be processed by the neural network component to store and retrieve any type of information necessary for performing the task, rather than having a specific, interpretable meaning on the information.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 6, Yang in view of Anderson in view of Besenbruch in view of Wang discloses all the limitations of claim 2.
Anderson discloses:
wherein the second neural network is configured to generate a reconstructed image from the compressed image, and generate the computer vision task result from the reconstructed image. [See Anderson, ¶ 0116 discloses a B-frame may be encoded and decoded in a hierarchical manner. For example, a B-frame may be encoded and decoded as a B1-frame, a B2-frame, or a B3-frame. A B1-frame is encoded and decoded using previously reconstructed frames that were reconstructed as an I-frame and/or a P-frame, a B2-frame is encoded and decoded using previously reconstructed frames that include one frame reconstructed as an I-frame or P-frame and another frame reconstructed as a B1-frame, and a B3-frame is encoded and decoded using previously reconstructed frames that were reconstructed as a B1-frame and a B2-frame.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 1.
Regarding claim 7, Yang in view of Anderson in view of Besenbruch discloses all the limitations of claim 1.
Wang discloses:
wherein the neural network based encoder is based on an encoder model in a neural image compression (NIC) framework, and the multi-rate compression domain computer vision task decoder is based on a decoder model in the NIC framework, the NIC framework is trained end-to-end. [See Wang, ¶ 0594-0595 discloses a neural network training process as a full end-to-end system encapsulating how visual data being provided to the encoder 2830 will be decoded by the decoder 2850.]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 8, Yang in view of Anderson in view of Besenbruch discloses all the limitations of claim 1.
Wang discloses:
wherein a decoder model of the multi-rate compression domain computer vision task decoder is trained separately from an encoder model of the neural network based encoder. [See Wang, ¶ 0594-0595 discloses a neural network training process as a full end-to-end system encapsulating how visual data being provided to the encoder 2830 will be decoded by the decoder 2850.  (Examiner’s note: it is within the level of ordinary skill to understand that “end-to-end” routinely and conventionally corresponds with training being separated, which is further evidenced by WO2021/220008A1, which notes on pg. 83, section 1.2.2 “Training of the segmentation module, if it is parameterized as a neural network, may be performed separately or during the training of the compression network itself – referred to as end-to-end.)]
The reasons to combine the cited prior art are applicable to those presented for previously rejected claim 2.
Regarding claim 12, this claim recites analogous limitations to claim 2, and is therefore rejected on the same premise.
Regarding claim 13, this claim recites analogous limitations to claim 3, and is therefore rejected on the same premise.
Regarding claim 14, this claim recites analogous limitations to claim 4, and is therefore rejected on the same premise.
Regarding claim 17, this claim recites analogous limitations to claim 7, and is therefore rejected on the same premise.
Regarding claim 18, this claim recites analogous limitations to claim 8, and is therefore rejected on the same premise.
Claim(s) 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Anderson in view of Besenbruch in view of Li et al. (US 20200280717 A1) (hereinafter Li).
Regarding claim 9, Yang in view of Anderson in view of Besenbruch discloses all the limitations of claim 1.
Yang in view of Anderson does not appear to explicitly disclose:
wherein the parameter is a hyperparameter.
However, Li discloses:
wherein the parameter is a hyperparameter. [See Li, ¶ 0115, 0181 discloses in addition to adjusting the number of features on each convolution layer, the architecture of FIGS. 9 and 12, and their variations described herein can be designed so that a global feature adjustment rate r may be applied across all layers for reducing/expanding the number of features. This feature adjustment rate can be a configurable hyper-parameter. After applying the feature adjustment rate, the number of features can be reduced/expended from n to n*r.]
It would have been obvious to the person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention disclosed by Yang in view of Anderson in view of Besenbruch to add the teachings of Li in order to identify quantization parameters that can be used to identify the tradeoff between rate and distortion.]
Regarding claim 19, this claim recites analogous limitations to claim 9, and is therefore rejected on the same premise.
Conclusion
THIS ACTION IS MADE FINAL.  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.

CITED REFERENCE SUMMARIES: Ref US20220224914A1: Claims:
A method of encoding a video including a set of frames, the method comprising: accessing a machine-learned compression model coupled to receive a frame and a parameter map for the frame, and output compressed code at a codelayer and a reconstructed version of the frame, wherein at least a portion of the compression model is configured as a neural network and the compression model is trained by: obtaining a video including a set of training frames, each training frame associated with a parameter map that indicates one or more values of an encoding parameter for the training frame, and changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the set of training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer; iteratively performing, for each current frame in the set of frames: obtaining a parameter map for the current frame, and applying at least a portion of the machine-learned compression model to the current frame and the parameter map for the current frame to generate compressed code for the current frame; and transmitting the compressed codes generated for the set of frames to a receiver system.
accessing a machine-learned compression model coupled to receive a frame and a parameter map for the frame, and output compressed code at a codelayer and a reconstructed version of the frame, wherein at least a portion of the compression model is configured as a neural network and the compression model is trained by: obtaining a video including a set of training frames, each training frame associated with a parameter map that indicates one or more values of an encoding parameter for the training frame, and changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the set of training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer;
obtaining a video including a set of training frames, each training frame associated with a parameter map that indicates one or more values of an encoding parameter for the training frame, and
changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the set of training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer;
iteratively performing, for each current frame in the set of frames: obtaining a parameter map for the current frame, and applying at least a portion of the machine-learned compression model to the current frame and the parameter map for the current frame to generate compressed code for the current frame; and
obtaining a parameter map for the current frame, and
applying at least a portion of the machine-learned compression model to the current frame and the parameter map for the current frame to generate compressed code for the current frame; and
transmitting the compressed codes generated for the set of frames to a receiver system.
The method of claim 1 , wherein the one or more values of the encoding parameter represent one or more bitrate levels, and wherein the codelength loss increases when a bitrate level for the training frame increases.
The method of claim 2 , further comprising: performing an interpolation operation to generate the parameter map for the current frame that represents an interpolated value for the encoding parameter between the one or more values of the encoding parameter.
performing an interpolation operation to generate the parameter map for the current frame that represents an interpolated value for the encoding parameter between the one or more values of the encoding parameter.
The method of claim 1 , wherein the compression model is further configured to include the parameter map in the compressed code for a decoder at the receiver system configured to obtain the parameter map from the compressed code.
The method of claim 1 , wherein the portion of the machine-learned compression model includes one or more subsets of weights, each subset of weights associated with a respective value for the encoding parameter, and wherein applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises selecting a respective subset of weights associated with a value for the encoding parameter for the current frame, and applying the respective subset of weights to the current frame.
The method of claim 1 , wherein the machine-learned compression model includes a flow autoencoder and a residue autoencoder, and applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises: obtaining zero or more previously reconstructed frames for the current frame, applying the flow autoencoder to the current frame, the zero or more previously reconstructed frames, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame, generating a compensated frame by applying a flow operator to the one or more previously reconstructed frames and the reconstructed flow, generating a residue frame by computing a difference between the current frame and the compensated frame, applying the residue autoencoder to the compensated frame and the residue frame to generate residue code as another part of the compressed code and a reconstructed residue frame, and generating a reconstructed version of the current frame by combining the reconstructed residue frame and the compensated frame.
obtaining zero or more previously reconstructed frames for the current frame,
applying the flow autoencoder to the current frame, the zero or more previously reconstructed frames, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame,
generating a compensated frame by applying a flow operator to the one or more previously reconstructed frames and the reconstructed flow,
generating a residue frame by computing a difference between the current frame and the compensated frame,
applying the residue autoencoder to the compensated frame and the residue frame to generate residue code as another part of the compressed code and a reconstructed residue frame, and
generating a reconstructed version of the current frame by combining the reconstructed residue frame and the compensated frame.
The method of claim 1 , wherein the machine-learned compression model includes a flow predictor portion, a flow autoencoder, and a residue autoencoder, and applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises: obtaining one or more previously reconstructed frames for the current frame, applying the flow predictor portion to the one or more previously reconstructed frames or previous flow for the one or more previously reconstructed frames to generate a flow prediction for the current frame, generating a predicted frame by applying a flow operator to the one or more previously reconstructed frames and the flow prediction, applying the flow autoencoder to the current frame and the predicted frame, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame, generating a compensated frame by applying a flow operator to the one or more previously reconstructed frames and the reconstructed flow, generating a residue frame by computing a difference between the current frame and the compensated frame, applying the residue autoencoder to the compensated frame and the residue frame to generate residue code as another part of the compressed code and a reconstructed residue frame, and generating a reconstructed version of the current frame by combining the reconstructed residue frame and the compensated frame.
obtaining one or more previously reconstructed frames for the current frame,
applying the flow predictor portion to the one or more previously reconstructed frames or previous flow for the one or more previously reconstructed frames to generate a flow prediction for the current frame,
generating a predicted frame by applying a flow operator to the one or more previously reconstructed frames and the flow prediction,
applying the flow autoencoder to the current frame and the predicted frame, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame,
The method of claim 1 , wherein the one or more values of the encoding parameter represent one or more bitrate levels, wherein the parameter map for the training frame includes a plurality of encoded vectors for a plurality of regions in the training frame, each encoded vector representing a bitrate level for a respective region in the training frame, and wherein the reconstruction loss for a region in the training frame increases when the respective bitrate level for the region increases.
The method of claim 2 , wherein the machine-learned compression model is further coupled to receive one or more previously reconstructed frames for the frame to generate the compressed code and the reconstructed version of the frame, and wherein the one or more values of the encoding parameter for the training frame further represents: (i) whether the training frame will be processed as a P-frame with a previously reconstructed frame that is placed temporally before the training frame in the video, (ii) whether the training frame will be processed as a B 1 -frame with two or more previously reconstructed frames that were processed as a P-frame or an I-frame, (iii) whether the training frame will be processed as a B 2 -frame with a previously reconstructed frame that was processed as a P-frame or an I-frame and another previously reconstructed frame that was processed as a B 1 -frame, or (iv) whether the training frame will be processed as a B 3 -frame with a previously reconstructed frame that was processed as a B 1 -frame and another previously reconstructed frame that was processed as a B 2-frame.
(i) whether the training frame will be processed as a P-frame with a previously reconstructed frame that is placed temporally before the training frame in the video,
(ii) whether the training frame will be processed as a B 1 -frame with two or more previously reconstructed frames that were processed as a P-frame or an I-frame,
(iii) whether the training frame will be processed as a B 2 -frame with a previously reconstructed frame that was processed as a P-frame or an I-frame and another previously reconstructed frame that was processed as a B 1 -frame, or
(iv) whether the training frame will be processed as a B 3 -frame with a previously reconstructed frame that was processed as a B 1 -frame and another previously reconstructed frame that was processed as a B 2-frame.
The method of claim 1 , wherein the parameter map is represented as a one-hot encoded vector including one or more elements each corresponding to a respective value in the one or more values of the encoding parameter.
A non-transitory computer-readable medium containing instructions for execution on a processor, the instructions comprising: accessing a machine-learned compression model coupled to receive a frame and a parameter map for the frame, and output compressed code at a codelayer and a reconstructed version of the frame, wherein at least a portion of the compression model is configured as a neural network and the compression model is trained by: obtaining a video including a set of training frames, each training frame associated with a parameter map that indicates one or more values of an encoding parameter for the training frame, and changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer; iteratively performing, for each current frame in the set of frames: obtaining a parameter map for the current frame, and applying at least a portion of the machine-learned compression model to the current frame and the parameter map for the current frame to generate compressed code for the current frame; and transmitting the compressed codes generated for the set of frames to a receiver system.
accessing a machine-learned compression model coupled to receive a frame and a parameter map for the frame, and output compressed code at a codelayer and a reconstructed version of the frame, wherein at least a portion of the compression model is configured as a neural network and the compression model is trained by: obtaining a video including a set of training frames, each training frame associated with a parameter map that indicates one or more values of an encoding parameter for the training frame, and changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer;
changing weights of the compression model to reduce a loss function, the loss function including a reconstruction loss that indicates a difference between the set of training frames and reconstructed versions of the training frames, and a codelength loss that indicates codelength of a compressed code for the training frames, wherein the loss function changes depending on a value of the encoding parameter for the training frames, and wherein the compressed codes for at least two training frames with different values of the encoding parameter are generated at the same codelayer;
The non-transitory computer-readable medium of claim 11 , wherein the one or more values of the encoding parameter represent one or more bitrate levels, and wherein the codelength loss increases when a bitrate level for the training frame increases.
The non-transitory computer-readable medium of claim 11 , wherein the instructions further comprise: performing an interpolation operation to generate the parameter map for the current frame that represents an interpolated value for the encoding parameter between the one or more values of the encoding parameter.
The non-transitory computer-readable medium of claim 11 , wherein the compression model is further configured to include the parameter map in the compressed code for a decoder at the receiver system configured to obtain the parameter map from the compressed code.
The non-transitory computer-readable medium of claim 11 , wherein the portion of the machine-learned compression model includes one or more subsets of weights, each subset of weights associated with a respective value for the encoding parameter, and wherein applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises selecting a respective subset of weights associated with a value for the encoding parameter for the current frame, and applying the respective subset of weights to the current frame.
The non-transitory computer-readable medium of claim 11 , wherein the machine-learned compression model includes a flow autoencoder and a residue autoencoder, and applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises: obtaining zero or more previously reconstructed frames for the current frame, applying the flow autoencoder to the current frame, the zero or more previously reconstructed frames, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame, generating a compensated frame by applying a flow operator to the one or more previously reconstructed frames and the reconstructed flow, generating a residue frame by computing a difference between the current frame and the compensated frame, applying the residue autoencoder to the compensated frame and the residue frame to generate residue code as another part of the compressed code and a reconstructed residue frame, and generating a reconstructed version of the current frame by combining the reconstructed residue frame and the compensated frame.
The non-transitory computer-readable medium of claim 11 , wherein the machine-learned compression model includes a flow predictor portion, a flow autoencoder, and a residue autoencoder, and applying the portion of the machine-learned compression model to the current frame and the parameter map for the current frame further comprises: obtaining one or more previously reconstructed frames for the current frame, applying the flow predictor portion to the one or more previously reconstructed frames or previous flow for the one or more previously reconstructed frames to generate a flow prediction for the current frame, generating a predicted frame by applying a flow operator to the one or more previously reconstructed frames and the flow prediction, applying the flow autoencoder to the current frame and the predicted frame, and the parameter map for the current frame to generate flow code as a part of the compressed code and a reconstructed flow for the current frame, generating a compensated frame by applying a flow operator to the one or more previously reconstructed frames and the reconstructed flow, generating a residue frame by computing a difference between the current frame and the compensated frame, applying the residue autoencoder to the compensated frame and the residue frame to generate residue code as another part of the compressed code and a reconstructed residue frame, and generating a reconstructed version of the current frame by combining the reconstructed residue frame and the compensated frame.
The non-transitory computer-readable medium of claim 11 , wherein the one or more values of the encoding parameter represent one or more bitrate levels, wherein the pa
[Truncated to input limit]