Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a novel model architecture for lossy image compression. Specifically, they propose combining a VAE encoder architecture with a conditional diffusion decoder to compress image data. The authors derive an Evidence Lower Bound (ELBO) for their model, introducing a standard weighting coefficient for allowing different rate-distortion tradeoffs. They train their proposed model on the Vimeo-90k image dataset, and test it on five different image datasets, using a variety of sixteen different assessment metrics, showing promising results when compared with state-of-the art approaches.
strength_and_weaknesses: ## Strengths

**Novel image compression architecture:** The authors propose combining a VAE encoder together with a conditional diffusion model for performing image compression. This is a novel architecture which allows diffusion models to be used in conjunction with entropy coding. Specifically, the latent variable of the VAE can be coded using entropy coding, and the diffusion model can be conditioned on this latent variable to decode the image. This is an interesting idea since it enables high-performance diffusion models to be used with entropy coding, which is both fast and efficient (in terms of the code lengths it produces), rather than relying on relative entropy coding, which can be either prohibitively slow, restrictive in terms of its assumptions, or inefficient in terms of code length.

**Competitive performance and extensive evaluation:** The authors evaluate their architecture on several different test datasets, using a variety of evaluation metrics. On some of these metrics, their architecture outperforms the most competitive models which they evaluate, while on others it performs in par or markedly worse than many of the other models. While the proposed method does not outperform all competitive models across all metrics, the extensive evaluation which the authors conduct is very welcome. While they could have picked the subset of these metrics which is favourable towards their model, they have included a variety of metrics making the comparison more extensive and transparent.

## Weaknesses

**Significance of the contribution:** While the authors' architecture does exhibit performance competitive to that of several strong methods from the literature, it also performs markedly worse across several of the metrics used. As this is the sole contribution of the paper, it makes one wonder whether this is yet another compression architectures to the long list of existing methods. That being said, their architecture does enable entropy coding with a diffusion model as the decoder, and with further improvements this architecture could yield more promising results. However, I think that this work would benefit by addressing the following points:

1. The authors are advised to modify their assessment of how their model stacks up against alternatives. For example, in the conclusion they say "Our approach yields competitive rate-distortion(perception) performance against all baseline models", a claim which is not true since their model actually performs worse than state-of-the-art methods across several of the evaluation metrics. A more measured and nuanced position here (as well as throughout the paper) is necessary.
2. The authors could outline their ideas on further improving their architecture, to improve its compression performance. If, for example, the authors do not foresee any improvements on the performance that could be extracted from their architecture, then their contribution is of limited interest since it consists solely of this architecture and, presently, it is not entirely obvious that this architecture is clearly preferable than the alternatives they evaluate.
3. A more detailed discussion on the various evaluation metrics and their relationships to specific applications, would be welcome. For example, the authors could highlight the types of applications in which the evaluation metrics correlate with overall performance, and the kinds of settings in which their architecture might be more or less favourable.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** While the authors present an interesting and useful architecture, I think the paper could be clearer in terms of its presentation. The authors are doing their work a disservice by not presenting it more clearly, and I find that this detracts from the overall quality of the paper. Some of these include:

1. It is currently unclear whether a VAE decoder is used in the architecture (e.g. during training) or not. Judging from the loss function in eq. 7, a VAE decoder is not used. If so, the authors should clarify this because in Section 3.1, under "neural image compression", they make explicit reference to the VAE decoder.

2. Where exactly does the parameterisation in eq. 2 (parameterising a network which predicts the noise used to generate an image) fit within eq. 7? Presumably, the loss in eq. 2 is used to re-write the first term in the inner expectation in eq. 7 (which the authors refer to as $\log p_{\text{lower}}p(\mathbf{x}_0 | \mathbf{z})$) following Ho et al. 2020.

3. In Section 2, under "Diffusion models", the authors refer to relative entropy coding in a slightly misleading way. They state that "'relative entropy coding' (Flamich et al., 2020) is substantially slower than transform coding (Yang et al., 2022b; Balle et al., 2020)." However, this is not exactly the case because there exist several relative entropy coding algorithms such as those in Agustsson and Theis, 2020, Flamich et al. 2022 or Theis and Yosri, 2022 which render relative entropy coding computationally cheap. However, there does not (currently) seem to be an available method which is computationally fast without introducing additional coding costs or further restrictive assumptions.

**Quality:** Overall, the quality of the paper is good in terms of the technical contribution of the novel architecture introduced and the extensiveness of its empirical evaluation, however the quality of the writing could be improved. This includes improving the clarity of the paper, as well as addressing some of the grammar and syntax errors in the paper, which detract from its writing quality.

**Novelty:** While diffusion models have been used for the purposes of compression before, the architecture introduced by the authors appears novel. It is an interesting and useful architecture in the sense that it enables entropy coding to be used together with diffusion models and may therefore be of interest to the community.

**Reproducibility:** The authors provide several experimental details which appear sufficient to reproduce their results, although I have not examined this in detail. They provide two figures (Figures 1 and 5) explaining their architecture, and an outline of the algorithm (Algorithm 1) used for training and compression.


**References**

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.

Eirikur Agustsson and Lucas Theis, Universally Quantized Neural Compression, 2020.

Gergely Flamich, Stratis Markou and Jose Miguel Hernandez Lobato, Fast Relative Entropy Coding with A* Coding, 2022.

Lucas Theis and Noureldin Yosri, Algorithms for the Communication of Samples, 2022.
summary_of_the_review: In summary, I find the architecture proposed in this work an interesting and useful one. This architecture enables entropy coding to be used in conjunction with diffusion decoders, which is a good contribution to the literature, even though it does not exhibit the best performance across all the metrics used by the authors. The extensive evaluation done in the paper is also welcome.

However, I also found that the architecture was not particularly well explained, and the paper would benefit considerably from an effort to clarify how it operates, especially since this is the main contribution of this work. Currently, I would mark the paper as marginally below the acceptance threshold, giving it a score of 5, but encourage the authors to make an effort to improve their exposition, in which case I would be happy to consider improving this score. I also expect that the authors will adjust their claims regarding competitive performance across all evaluation metrics, which does not appear to be exactly the case.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper examines the use of conditional diffusion models (at the decoder side) for end-to-end optimized lossy image compression.
strength_and_weaknesses: Strengths:

1. The use of conditional diffusion models in the context of neural image compression is somewhat novel.

2. The experimental results, especially using more than 15 image quality metrics, are comprehensive.

Weaknesses:

1. The proposed method is not clearly described. 

a) Why the authors choose to start the derivation from the negative data log-likelihood (the inequality above Eq. (7)), which is more relevant to lossless image compression? 

b) When and how do the index $n$ and $\mathbf{x}_n$ come into play in Eq. (9), provided that there are no such terms on the LHS? 

c) Why is $\log p(\mathbf{z})$ instead of $\log P(\mathbf{\hat{z}})$ in the rate-distortion objective? Are we supposed to compute the discrete entropy as a proxy of the expected code length?

d) How to learn the quantization centers and how to model $\log P(\mathbf{\hat{z}})$ are not mentioned.

e) How to set the trade-off parameter $\lambda$ is not given. At lower bits, do we also need to reduce the dimensionality of the latent representation to encourage bit savings?

2. The rate-distortion performance is not outstanding compared to existing methods, e.g., HiFiC and MS-GMM, under standard and widely used metrics - PSNR and SSIM. Of particular interest, the proposed method jointly optimized for LPIPS (with $\rho=0.9$) under-performs HiFiC under the same metric - LPIPS. The authors may want to give some explanations.

3. The computational complexity in terms of the number of model parameters, and the encoding and decoding time should be provided to gauge whether the improved rate-distortion performance is worthwhile.
clarity,_quality,_novelty_and_reproducibility: Clarity: The introduction and related work is very well written. However, this is not the case for the algorithm description. The reviewer finds it difficult to comprehend.

Quality and Novelty: Somewhat limited due to the limited rate-distortion performance and the ungiven computational complexity analysis.

Reproducibility: Based on the current description, the reviewer shall say it is less feasible to reproduce the results in the paper.
summary_of_the_review: With a much clearer algorithm description, improved rate-distortion performance, and computational complexity analysis, this paper has a chance to get in.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a lossy image compression algorithm. The framework is similar to previous work of transform coding built on VAEs, and the proposed method replaced the the decoder with a conditional diffusion model. The paper provided comprehensive evaluations on various datasets and a number of metrics, and demonstrated that the proposed approach is compatible, and sometimes better than the state of the art.
strength_and_weaknesses: Strength

* Combining diffusion models and image compression is both interesting and expected. While previous works have explored this basic idea to a certain extent (as reviewed in this paper), this paper explored it most thoroughly to my knowledge. The comprehensive experiments provided valuable comparison between algorithms.

* The proposed perceptual loss is well placed in the diffusion model, which might find its use beyond the proposed approach. It is formally simple, and demonstrated to be able to substantially improve perceptual quality.

Weakness

* The paper rightly claimed their "competitive" performance against baseline models. However, the at best marginal improvement is hard to justify the significantly increased computational cost (e.g., 500 diffusion steps in decoding).

* The perceptual distortion loss could be better explored. E.g., It would be interesting to see how the rate-distortion trade-off is affected by different weights of the perceptual loss, but the paper only presented results from $\rho = 0$ and $\rho = 0.9$.
clarity,_quality,_novelty_and_reproducibility: The paper is overall of decent quality, and is very well written. The proposed approach is well motivated and positioned in existing literature.

The novelty is relatively limited, as the method is a relatively straightforward extension of an existing neural image compression algorithm with a conditional diffusion model.
summary_of_the_review: The paper is well written, and provides valuable empirical comparison between the proposed and existing approaches on neural image compression. However, the novelty of this work is limited, and its improvement over baselines is marginal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a lossy image compression method using the condition diffusion model. This is an end-to-end framework based on a condition diffusion model. This is a new attempt to use the diffusion model for lossy image compression. Extensive experiments are conducted to show the advantages and robustness of the proposed method. 
strength_and_weaknesses: Strength

+New attempt
This work tries to use the diffusion model for lossy image compression. As claimed, this is the first of this kind.

+Extensive experiment
To show the effectiveness of the proposed method, 5 datasets, and 16 IQA metrics are used. The proposed method achieves promising performance. 

Weakness

-Unclear motivation
This is not clear why the use of the diffusion model is necessary. What is the advantage of the diffusion model for this task? To my best knowledge, the diffusion model is hard to preserve fidelity while fidelity is crucial for image compression. Besides, the diffusion model is slow, which also challenges the requirement of image compression.

-Limited novelty
It is intuitive to compress the image into the latent space. The diffusion model is a choice.  However, the novelty of how to use the diffusion model is limited in the work. All these ideas are intuitive or they have been used in previous related works. 

-Insufficient analysis and comparison
1) The efficiency of different methods should be discussed. 
2) Under different IQA metrics, the proposed with different parameter settings obtains better performance. How to choose the parameter when using the proposed method in practical applications？
3) More methods published after the 2020 year should be included for comparison.

-Other issues
1) It seems that the proposed method can only process the image with a size of 64X. 
2) The ground truth image shown in the paper is not high-quality. It would be good if high-quality images can be used to show the advantages of the proposed method. For current visual comparison, it is hard to distinguish the performance of different methods as the ground truth images are still low-quality. 
3) The paper is not well-written. Some content is hard to understand. 

clarity,_quality,_novelty_and_reproducibility: The quality of this paper is low. The motivation is not clear. Besides, the writing of the paper needs improvement. The paper is hard to understand. 
summary_of_the_review: The paper has some significant novelty flaws and the technique and experiment are not convincing enough, which are the most important factors in my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new image compression model based on the paradigm of neural image compression. The main novelty is to introduce a diffusion based decoder, instead of the traditional Gaussian or Laplacian decoders. Some experiments are conducted to verify the effectiveness.
strength_and_weaknesses: Strengths:
1. The idea is very intuitive and impressive. To enhance the capability of the decoder in image compression, it replace the Gaussian and Laplacian decoder with a powerful diffusion model.
2. The paper adopt as many as possible metrics (16) to evaluate the performance, which comprehensively reflects the advantages and disadvantages of the proposed method. 

Weaknesses:
1. In this paper, $z$ is called semantic latent variable while $x_{1:N}$  texture latent variable. Why does $z$ and $x_{1:N}$ control the semantic and texture information respectively? If you could provide some visual proofs, it would be better.
2. The hyper-parameter $\rho$ makes a trade-off between the diffusion lower bound and the perceptual loss. More quantitative and qualitative comparisons under different $\rho$ values should be added to analyse its influence.
3. More visual results under different bitrates should be also provided.
4. The experimental comparisons are not sufficient enough, some recent works are not compared. E.g,
    Neural data-dependent transform for learned image compression, CVPR 2022.
    Enhanced invertible encoding for learned image compression, MM, 2021.
    Transformer-based transform coding, ICLR, 2021.
  
clarity,_quality,_novelty_and_reproducibility: The paper is well and clearly written.  The novelty is a little limited, since it only simply introduces the diffusion model as encoder.
summary_of_the_review: In summary, the motivation of this work is clear, but it lacks some important analysis on the proposed method. Additionally, the experiments are insufficient.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Summary:

In this work, the authors propose a novel model architecture for lossy image compression. Specifically, they propose combining a VAE encoder architecture with a conditional diffusion decoder to compress image data. The authors derive an Evidence Lower Bound (ELBO) for their model, introducing a standard weighting coefficient for allowing different rate-distortion tradeoffs. They train their proposed model on the Vimeo-90k image dataset, and test it on five different image datasets, using a variety of sixteen different assessment metrics, showing promising results when compared with state-of-the art approaches.

Strengths:

- Novel image compression architecture
- Competitive performance and extensive evaluation
- The use of conditional diffusion models in the context of neural image compression is somewhat novel.
- The experimental results, especially using more than 15 image quality metrics, are comprehensive.
- Combining diffusion models and image compression is both interesting and expected. 
- The proposed perceptual loss is well placed in the diffusion model,
- New attempt This work tries to use the diffusion model for lossy image compression. As claimed, this is the first of this kind.
- Extensive experiment To show the effectiveness of the proposed method, 5 datasets, and 16 IQA metrics are used. The proposed method achieves promising performance.
- The idea is very intuitive and impressive. T
- The paper adopt as many as possible metrics (16) to evaluate the performance.

Weaknesses:

- While the authors' architecture does exhibit performance competitive to that of several strong methods from the literature, it also performs markedly worse across several of the metrics used.
- the paper could be clearer in terms of its presentation.
- The proposed method is not clearly described.
- The rate-distortion performance is not outstanding compared to existing methods,
- The computational complexity in terms of the number of model parameters, and the encoding and decoding time should be provided to gauge whether the improved rate-distortion performance is worthwhile.
- The paper rightly claimed their "competitive" performance against baseline models. However, the at best marginal improvement is hard to justify the significantly increased computational cost 
- The perceptual distortion loss could be better explored. 
- Unclear motivation 
- Insufficient analysis and comparison
- More visual results under different bitrates should be also provided.
- The experimental comparisons are not sufficient enough

Recommendation:

A majority of reviewers vote for rejection, so I therefore recommend rejection. I encourage the authors to look at the feedback provided and use this to improve the paper and resubmit to another venue.
justification_for_why_not_higher_score: A majority of reviewers vote for rejection.
Decision: Reject