Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for quantizing floating-point representation to fixed-point precision in a learned image compression framework. The main advantage of the proposed method is, that it can be added to pretrained off-the-shelf compression models, making it useful in practice. The paper adapts an existing post-training quantization method to image compression task by considering the rate-distortion during optimization instead of only mean squared error. Results are shown by adding the approach to 3 existing methods.
strength_and_weaknesses: *Strengths*

- The loss due to the quantization is with the proposed method smaller than with other post-processing methods. 
- The method is simple and works better than other methods shown.
- The motivation, to use the loss function during optimization, which is also used in evaluation, makes sense.

*Weaknesses*
- The main technical contribution of the work is considering the compression rate in addition to the MSE loss. However, the comparison to the baseline using only MSE is only shown in the supplement. In my opinion, this comparison and discussion should be part of the main paper.
- Some details related to the experimental setup are missing (e.g., algorithm runtime, needed to get the quantized model and the optimized scaling factors, used hardware, calibration set selection).
- The quantization parameters are obtained by training on a small calibration dataset. From the paper it is unclear how this set has been selected and how sensitive the results are to the selection, e.g., by measuring the variance of performance over different image sets. I am wondering whether the calibration set is dataset specific in the experiments, and if that is the case, how the performance would be with an unsuitable calibration set. This clarification is important to fulfill the claim, that is truly a plug-and-play method without model retraining.
- For completeness it would be interesting to compare also to a method which retrains the compression model, as the paper claims to be more efficient in terms of training time and practicability, e.g., by reporting the time needed.
- The algorithm needs to be run for every desired bit rate.

clarity,_quality,_novelty_and_reproducibility: The main motivation is easy to follow, however the paper lacks in detailed description of the experiments and complete analysis (see weaknesses). The originality of the technical contribution (mainly modifying the loss function to include the compression rate to target image compression) is rather limited, therefore insightful and complete analysis would be important for the paper.
summary_of_the_review: The paper presents a simple, but apparently effective method, however the main weaknesses are in the experimental section and performance analysis. Without additional insights and description of the effect of the calibration dataset it is difficult to judge how useful the method is in practice.

**Post-rebuttal**  
The additional experiments including re-training methods are appreciated. However, for me the paper remains a borderline submission. The method works in practice but the analytical analysis/insights are limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper applies PTQ in learned image compression. Two methods are exploited. One is AdaRound (ICML'20) for weight, and the other is a learnable dynamic range for both weight and activation.
My main concern is about the novelty of this paper. AdaRound was presented in ICML'20, and learnable dynamic (clipping) range has also been studied in quite a few of literatures such as PACT.

PACT: Parameterized Clipping Activation for Quantized Neural Networks
strength_and_weaknesses: Strength

The paper is one of very few works which studied the network quantization for learned codec.

Weaknesses

1. The novelty is really margin. AdaRound and dynamic range determination have been proposed in previous works. I wonder what is new when you utilize AdaRound in the specific task (i.e. learned codec)?
2. The writing can be improved. In Section 4.1, the equations are quite similar with the ones in AdaRound (Section 2. Motivation). Section 4.2 looks like some common equations for rate-distortion loss. Section 4.3 just introduces the N (dynamic range scaling) for weight/bias/activation.
3. The experiments are not enough. There is no ablation studies for the final results. Now that you have three methods in detail, AdaRound for weight, dynamic range for weight and dynamic range for activation, which is the individual contribution to the final result?
4. There is no running time evaluation.
5. This paper applied the network quantization into a transformer-based learned codec. There is a related work which also quantized the transformer-based learned codec (MobileCodec), it is better to cite the paper and make a comparison.

    MMSys '22: MobileCodec: neural inter-frame video compression on mobile devices




clarity,_quality,_novelty_and_reproducibility: Quality: Due to the lack of novelty and experimental results, I am afraid the quality is not good.

Clarity: The paper is easy to follow in general.

Novelty: The novelty is really margin.

Reproducibility: The paper is reproducible since Adaround has been already imported in some model compression libraries such as AIMET.
summary_of_the_review: As described in the above weaknesses, the proposal looks like just a combination of some previous methods and some mathematical derivations are also quite close to the previous paper. Besides, the experimental results should be improved, there is no ablation study and the comparison with previous works can also be more comprehensive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for post-training quantization (PTQ) for learned image compression models. Quantization offers three main benefits: (1) fixed-point math is deterministic, which ensures reliable decoding (this is not guaranteed when different hardware is used to decode compressed representations using floating-point operations), (2) it reduces space complexity if int8 weights are used instead of float32, and (3) it reduces time complexity since typical parallel hardware can perform int8 operations more quickly than float32 or float16. Furthermore, PTQ occurs after training, as the name implies, and does NOT require additional fine-tuning to achieve good rate-distortion performance relative to the original float32 model. This simplifies application and reduces total training time.

The authors show, both theoretically and empirically, that selecting quantization parameters that minimize mean-squared error (mse) to the float parameters does NOT maximize rate-distortion performance. To address this, they propose rate-distortion optimization (RDO) for the quantization parameters and describe an algorithm for doing this in detail.

The empirical evaluation shows a relatively small drop in RD performance using RDO-PTQ. In particular, the drop is smaller (i.e. the resulting RD performance is better) than two previous methods (range-adaptive quantization (RAQ) and FQ-ViT). The authors run the evaluation using two image datasets and three different image compression models.


strength_and_weaknesses: Strengths:
1. The paper addresses an important, practical challenge for learned image compression: decode reliability and reducing space/time complexity.
2. The method is general (can be applied to any model) and appears to outperform other PTQ approaches.
3. The evaluation uses multiple datasets and multiple models, which minimizes concerns that it may not be widely applicable in practice. Also, one of the models is quite recent (Lu 2022).

Weaknesses:
1. Although post-training quantization is convenient, fine-tuning is not a major barrier. So additional evaluation showing the benefit (or lack thereof) of fine-tuning would strengthen the paper. In practical scenarios, additional training time is negligible compared to the lifetime of the codec and potential RD benefits.
2. The central insight is not very surprising or novel. Although RDO-PTQ outperforms the baseline methods, the basic expectation in compression is that RDO should always be performed and will always be best when it is possible/practical to apply.
clarity,_quality,_novelty_and_reproducibility: Overall, the clarity and quality of the paper is very high. I appreciated the theoretical explanation of the suboptimality of minimizing mse for quantization. The notation and various terms in Section 4.2 and 4.3 (where the core algorithm is explained) could be somewhat clearer, but it was understandable with a careful reading.

While I don't know of another paper that explicitly does RDO-PTQ, novelty still seems fairly low as described above.

I have no concerns about reproducibility, though it's always best to provide open source code.
summary_of_the_review: This paper addresses a practical problem for learned image compression, and the proposed algorithm outperforms the two baseline methods included in the empirical evaluation. The method, RDO-PTQ, is well-motivated and has many nice properties: it is generally applicable, relatively simple, does not require additional fine-tuning, and leads to only a relatively small drop in rate-distortion performance.

As discussed earlier, understanding the potential benefit of fine-tuning would strengthen the paper. I also think the outcome is more or less expected (RDO should always outperform a proxy loss for compression models unless it can be shown that the proxy is identical), and the applied nature of the result is interesting to learned image compression practitioners but is not a great fit for ICLR in general.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a framework for applying off-the-shelf post-training quantization (PTQ) methods for learned image compression (LIC) task. The authors show that the optimum quantization scaling factor may not be discovered by only minimizing the MSE loss for LIC task and provide theoretical insights on this. To alleviate this, they propose to use a rate-distortion (RD) loss as their objective function where both the distortion and bit-rate are coupled by a Lagrange multiplier. Their experiments on 2 benchmark datasets show that INT8 quantized LIC (using their PTQ method) can achieve similar RD performance compared to the original 32-bit counterparts while outperforming the other quantization baselines. 
strength_and_weaknesses: Pros:
The idea of utilizing the RD term as an objective function to find the optimum scaling factor in the context of LIC task seems novel and the main contribution of the paper. 

Cons:
I have several concerns about the papers as outlined below:

Novelty and Prior-works: The authors claim that applying PTQ for LIC task is one of their main contributions. However, there have been several prior works that have already applied PTQ on LIC tasks. See [1-4]. The authors refer [1], and [2] in the introduction section as a quantization aware training (QAT) method, which I believe, is not a correct statement. Rather these methods are PTQ followed by fine-tuning (which is very common in computer vision tasks, specially for bit-width lower than 8), similar to [4]. Other than that, the work in [3] have directly applied existing PTQ method on LIC task without requiring any fine-tuning, similar to this work. Although the authors refer [3] in the introduction section as a part of their motivation (cross-platform inconsistency issue of the floating-point LIC methods), they do not discuss how the proposed work is different from theirs. I believe the main claim of this work should be 'PTQ on LIC without requiring any fine-tuning' (if we do not consider [3], of course).

Significance: The authors claim that the quantization error and compression metric do not have any monotonic relation (which is the main motivation behind their proposed loss). However, from Fig. 2, this claim seems to be valid only at the edge of minimum R-D loss(i.e., $\Delta$J ~= 0) while they follow a monotonic relationship everywhere else. There was no theoretical justification or any insightful discussion behind such observation. This also questions the significance of using the R-D loss as the objective function rather than simple MSE loss as the authors in [3] showed pretty impressive results on the same benchmark (Fig 5(a) in [3]) with MSE loss only.

Experiments: Comparison with other PTQ methods are missing. While [3] and [4] might be considered as pretty recent works, at least comparison with [1] and [2] is desirable. Also, in the ablation study, there could be an additional experiment (a graph similar to Fig 4) that clearly shows the performance gain of R-D loss over MSE loss where everything else (except the objective function) is same for fair comparison. 

1. Sun, Heming, et al. "End-to-end learned image compression with fixed point weight quantization." 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020.
2. Sun, Heming, Lu Yu, and Jiro Katto. "Learned Image Compression with Fixed-point Arithmetic." 2021 Picture Coding Symposium (PCS). IEEE, 2021.
3. He, Dailan, et al. "Post-Training Quantization for Cross-Platform Learned Image Compression." arXiv preprint arXiv:2202.07513 (2022).
4. Sun, Heming, Lu Yu, and Jiro Katto. "Q-LIC: Quantizing Learned Image Compression with Channel Splitting." arXiv preprint arXiv:2205.14510 (2022). 

clarity,_quality,_novelty_and_reproducibility: Overall the paper is well written and easy to follow. 
Minor suggestion:
1) Figure 4 at its current state is very difficult to visualize since there are many lines overlapped with each other in a very small region. I would advice the authors to try out alternate ways (maybe changing the axis scale or bar plot instead of line plot) to make the plots more clear.
2) In the 3rd paragraph of the introduction, the line "QAT method requires model re-training" seems to be wrong, and might require correction. I think the authors wanted to refer 'PTQ with fine-tune' methods instead of 'QAT'.
3) In section 4.1, "Assuming a trained compression task model with R-D metric $J(x,w)$" in place of "Assuming a trained compression task model $J(x,w)$". 
summary_of_the_review: Overall, due to the limited novelty, lack of proper discussion about the related works, lack of proper experiments and justification on why R-D loss is a better objective function compared to the MSE loss, I can not recommend for acceptance. However, I am willing to change my opinion if the authors can provide solid answers to my above concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the use of Post-Training Quantization (PTQ) to directly process pretrained, off-the-shelf Learned Image Compression  (LIC) models.  The proposed study proves theoretically that minimizing the mean squared error (MSE) in PTQ is suboptimal for compression task, and proposes a novel Rate-Distortion (R-D) Optimized PTQ (RDO-PTQ) to best retain the compression performance. 
strength_and_weaknesses: Strength :
The paper is clear and well organized. The idea of the proposed contribution seems interesting. Indeed, the proposed method needs to compress few images to optimize the transformation of weight, bias, and activation of underlying LIC model from its native 32-bit floating-point (FP32) format to 8-bit fixed-point (INT8) precision for fixed-point inference onwards. In addition, the experimental results have shown the effectiveness of the proposed method.

Weaknesses :
Other state-of-the-art methods could be considered in the comparison.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well organized. The idea of the proposed contribution seems interesting. Indeed, the proposed method needs to compress few images to optimize the transformation of weight, bias, and activation of underlying LIC model from its native 32-bit floating-point (FP32) format to 8-bit fixed-point (INT8) precision for fixed-point inference onwards. An algorithm is provided to illustrate the different steps constituting the proposed method.


summary_of_the_review: The paper is clear and well organized. The proposed method is well described and the experimental results have shown their effectiveness. The proposed contribution is interesting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper suggests the use of post-training quantization to quantize neural image compression models, where the model parameters are compressed from Float32 format to Int8 precision. The weight, bias and activation parameters are processed with post-training optimization. Some theoretical justifications are provided to prove that the quantization error is not monotonically related to the rate-distortion performance. The experimental results show the proposed model quantization method can convert the network precision with only 3.70% BD-rate increases. 
strength_and_weaknesses: Strength:
1.	The proposed post-training quantization (PTQ) method can transform the model from Float32 format to Int8 precision without large-scale model retraining.
2.	Some theoretical discussions are provided to illustrate the motivation of the proposed method.
3.	The experimental results demonstrate the effectiveness of the PTQ method in neural compression models, i.e., tolerable performance drop but much more light-weighted neural network.
Weakness:
1.	The only difference between using PTQ in LIC models and conventional neural networks is the format of task loss. This paper simply applies the PTQ method for the task of LIC, thus makes somewhat trivial technical contributions. 
2.	The theoretical derivations and the relevant conclusions in this paper have been given in the paper of AdaRound [1].
3.	The setting of this paper is to transform the neural network from FP32 to INT8. But actually this setting is defined by the author without any explanations. Some other choices can be adopted, e.g., INT4 or unsigned INT16.
4.	There are some vague claims without evidence, such as, ‘Such inappropriate processing of bias may lead to catastrophic results. For example, having the bias in INT32 precision may cause data overflow of the INT32 accumulator; while setting bias as zero would degrade the model performance significantly.’ 
References:
[1] Up or down? adaptive rounding for post-training quantization. Nagel et al., ICML 2020.


clarity,_quality,_novelty_and_reproducibility: Clear presentation of the paper. But not novel enough. Reproducibility is good since the proposed method is not difficult.
summary_of_the_review: The PTQ method has been applied on other tasks for many times. This paper simply applies the PTQ method into learned image compression models. The theoretical analyses in this paper also follow previous papers without new conclusions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies Post-Training Quantization (PTQ) for Learned Image Compression (LIC) models. The paper shows that minimizing MSE in PTQ is suboptimal, and proposes a novel Rate-Distortion Optimized (RDO) PTQ. The paper lacks some ablations and experimental details to be more useful in future PTQ research work. In its current form, the contribution is rather incremental and would probably better fit a quantization or compression workshop than ICLR.
justification_for_why_not_higher_score: Too many reviewers whose reviews pointed flaws in the writing or in the experiments did not get swayed by the rebuttal.
Decision: Reject