Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper uses a residual diffusion-based model to augment a learned image compression model. The combined model has two main advantages. First, the diffusion post-processing improves perceptual quality (at the expense of increased distortion, i.e. FID improves while PSNR falls). Second, the approach leads to a single model that can target different different distortion-perception trade-offs at inference time.

The model is also multirate meaning that a single model can target different bit rates. This capability alone is not novel, but the end result here is a single compression model that can be adjusted at runtime to target different points in the 3D rate-distortion-perception space. As far as I know (and as the authors claim) this is the first model to have this flexibility / capability.

The final contribution of the paper is showing that the model can skip a large percent of the diffusion steps with only a minor reduction in perceptual quality (as measured by FID). It's not entirely clear *why* this works, but the empirical demonstration is convincing, and the authors argue that it implies that the noise schedule is suboptimal.




strength_and_weaknesses: The primary strength of the paper is a the creation of a single model that can flexibly target different points in rate-distortion-perception space. The use of a residual diffusion model allows the model to target perceptual quality. Diffusion models can be slow (especially in pixel-space) but the authors mitigate this downside to some extent by showing that a large number of diffusion steps can be skipped with minimal degradation to perceptual quality (specifically, they show that up to 80 steps can be skipped when evaluating up to 100 diffusion steps).

There are two primary weaknesses of the paper. First, as mentioned above, the decoder needs to run a diffusion model (up to 20 steps), which means it's quite a bit slower than a typical neural image compression model (and these are already an order of magnitude slower than standard codecs). Second, even after 100 diffusion steps, the perceptual quality does not match/exceed the performance of HiFiC, a GAN-based image compression model that targets perceptual quality.

To some extent, the same argument can be made about distortion, i.e. DIRAC-1, which minimizes distortion (max PSNR) is not SOTA. However, since the approach uses two separate models, it seems fair to say that any codec could be plugged in to the first stage (which generates a low-distortion image) and thus non-SOTA PSNR is not a deficiency of the approach per se.

Finally, a user study is always appreciated when perceptual quality is a goal. So that would strengthen the paper, but I don't think publication should require it so long as reasonable perceptual metrics (like FID and LPIPS which are reported) are used.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper was clear and of high quality.  It took a while to understand the graph in Figure 1, but it is showing a lot of information and it eventually made sense.

I don't see any major hurdles to reproducibility.

Novelty is mild. On the one hand, I haven't seen a paper address the full rate-distortion-perception space with a single model before, I think that's quite interesting. On the other hand, using diffusion for image compression was bound to happen (and the authors note concurrent research). It may be that diffusion marks a significant step forward for the neural image compression subfield, but that's not clear yet. In other words, a comparison with a residual GAN-based model (with a multirate base codec) would also allow targeting different points in the rate-distortion-perception space and would probably have a lower runtime. Would it yield better perceptual results? I don't know, though the results in this paper show that HiFiC (which is GAN-based by not multirate or flexible in terms of distortion vs. perception) does give better perceptual quality.
summary_of_the_review: I have a "marginal accept" score because a flexible model in terms of rate, distortion, and perceptual is very interesting, and I haven't seen that before. Using diffusion for image compression makes sense but isn't particularly novel on its own. As mentioned above, further comparison to GAN-based methods and via a user study would strengthen the paper as would outperforming HiFiC, the generative image compression baseline.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors applied the diffusion model to the decoded image of the deep image compression model (Minnen 2018) learned with MSE, aiming to improve perceptual quality. Experimental results show that post-processing with the diffusion model improve LPIPS and FID than the original model (Minnen 2018).
The authors also show that the degree of post-processing can be adjusted at the test phase by early stopping the diffusion sampling.
strength_and_weaknesses: Strength
- It may be new that the diffusion process was utilized as a quality enhancement post-processing network for the compressed image, allowing the degree of post-processing can be adjusted at the test phase by early stopping the diffusion sampling.

Weakness
- The compression part used the conventional Minnen 2018, and the architecture of the diffusion process used Dhariwal & Nichol 2021 and Preechakul 2022, it seems no particular innovations. Although the authors combined these with the loss function but they did not seem to success to learning the effective entire end-to-end process, so the research contents is considered to be more like an image enhancement task with the diffusion model. (In this sense, it may be interesting to consider and verify to what extent the diffusion model is effective in reducing compression artifacts for other codecs, including traditional ones.)
- Perceptual quality was only evaluated by LPIPS and FID, and no subjective image quality evaluation was performed.
- Since this paper is considered to be close to the task of image enhancement, i think it should be quantitatively compared with other approaches of enhancement networks that aim to improve the perceptual quality of compressed images (e.g. Y. Kim et al., Towards the Perceptual Quality Enhancement of Low Bit-rate Compressed Images, CVPRW 2020).
clarity,_quality,_novelty_and_reproducibility: - For the reasons stated in the weakness above, I think the author's contributions were not clear.
- Also, it was difficult to follow where the proposed methods are located, as the sections on conventional research and proposed methods are not clearly separated.
- The motivation of tune the degree of post-processing of image quality enhancement was insufficiently explained and the value is seem to be limited.
- The background theory of diffusion model is well explained in the appendix. I found the text relatively easy to read.
- The authors did not provide source code, but they experimented with the source code of Dhariwal & Nichol (2021), and Table 1 describes the parameters. If the authors did not modify the base source code, it should be reproducible.
summary_of_the_review: Although the direction of using the diffusion model to improve the visual quality of compressed images is interesting, but i think the contents of this paper need to be reconsidered because of the difficulty in seeing the important contributions of the proposed method, the lack of sufficient evaluation, and the insufficient comparison with related studies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a post-processing method for neural image compression using a diffusion probabilistic model to improve the perceptual quality of the reconstruction results . With a variable-rate model that has been fully trained, no more bits need to be coded. The fidelity and perceptual quality are determined by adding residuals to different degrees with controllable sampling steps.
strength_and_weaknesses: Strengths
1. This work enables users to choose if an image is decoded with high fidelity or high perceptual quality by changing the sampling steps of the diffusion-based residual augmentation module.
2. This work shows that in the compression setting, the number of sampling steps can be reduced dramatically, although typical diffusion probabilistic models are notoriously expensive to sample from.

Weaknesses
1. This work belongs to the post-processing category, and the correlation with compression itself is not convincing and attractive to some extent. That is to say, the diffusion method, which has been illustrated as effective ways in many other low-level tasks (e.g., restoration, super-resolution) can be easily migrated in this compression task. As a comparison, the work HiFiC is an end-to-end trained method using generative loss to improve perceptual quality. The novelty of this work, especially for the specific compression task is quite limited. 
2. The curve diagrams that are appeared in the paper are not very intuitive for understanding, especially Fig.5. The author should modify them.
clarity,_quality,_novelty_and_reproducibility: The clarity of this work is clear, and it offers a diffusion-based solution for post-processing. It should be easy for others to reproduce.
summary_of_the_review: The paper proposes a good choice for compressed image enhancement. Further studies should be conducted to improve the novelty.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The method adds a diffusion model to improve images that are compressed using auto-encoding system and is able to traverse rate-distortion-perception tradeoff during test time.
The drawbacks/potential improvements of the paper are the following (from reviewer responses). 
- The method is closer to image enhancement methods and should be better compared to those. In this sense it is also not end to end compression method and perhaps could be made so.
- Better explanation of the benefit of adjusting image quality by number of diffusion steps.
- Overall the method is not very novel
Other potential improvements
- Subjective evaluation
- Comparison to using GAN instead of diffusion model in the same setting.
justification_for_why_not_higher_score: - The method is closer to image enhancement methods and should be better compared to those. In this sense it is also not end to end compression method and perhaps could be made so.
- Better explanation of the benefit of adjusting image quality by number of diffusion steps.
- Overall the method is not very novel
Other potential improvements
- Subjective evaluation
- Comparison to using GAN instead of diffusion model in the same setting.
Decision: Reject