SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.2.2 Results of Near-Idempotent Codec
For near-idempotent codes, we compare against prior work [Kim et al., 2020], as well as [Cai et al., 2022] which utilizes a partially invertible structure. To demonstrate the advantages over non-idempotent codes, we also compare with non-idempotent learned codes [Balle et al., 2018, Cheng et al., 2020, He et al., 2022] as well as traditional codes BPG and VTM.
Following the benchmark protocol in [Kim et al., 2020], we report the PSNR drop of the above codecs upto 50 re-compression. RD trade-off points whose first-time bpp is closest to but not greater than 0.8 bpp are chosen for each codec. The results is shown in Fig. 3(b) and listed in detail in Tab. 2. From the result we see that, in the near-idempotent setting, the PSNR drop of proposed framework is 0.87dB, whereas [Kim et al., 2020] and [Cai et al., 2022] has more than 2dB PSNR drop. Additionally, the PSNR drop of the proposed framework almost converges within 10 re-compression, while other near-idempotent frameworks still experience evident PSNR drop after 30 or even 50 re-compression. Codecs that do not consider idempotence suffers a much more severe drop of PSNR during re-compression.
We also provide the RD performance of the proposed idempotent and near-idempotent frameworks, as is shown in Fig.3(a). It is clear that near-idempotent framework has much better first-time compression RD performance than idempotent framework. In terms of re-compression RD performance, however, near-idempotent can only reach similar RD performance with much higher computation cost (8.40 GFLOPs v.s. 48.78 GFLOPs in Tab.1 and Tab.2).
These results clearly demonstrate that, even if we break the right-invertibility of the first layer in order to get higher first-time RD performance, the performance drop during re-compression is still acceptable and highly controllable, as opposed to prior works [Kim et al., 2020, Cai et al., 2022].
Table 2: PSNR drop during 50 re-compression of different non-idempotent and near idempotent codec. FLOPs and encode-decode time tested under the same condition as Tab. 1
PSNR Drop (dB) ↓	GFLOPs ↓	time (ms) ↓
round = 5	10	25	50
Non-Idempotent Codec
BPG	1.16	1.93	2.10	2.19	-	-
VTM	1.19	2.09	4.50	7.18	-	-
[Balle et al., 2018]	2.18	3.17	5.65	8.46	6.23	80
[Cheng et al., 2020]	2.44	4.76	8.59	12.40	51.99	>1000
Near-Idempotent Codec
[Kim et al., 2020]	0.18	0.61	3.18	8.26	6.23	80
[Cai et al., 2022]	1.36	2.01	2.75	-	131.46	240
Proposed Near-Idempotent	0.74	0.83	0.87	0.87	48.78	115
(a) RD performance of different architectures on Kodak
(b) Influence of coupling enhancement (better viewed zoomed-in)
Figure 4: Ablation studies for different architectures. (a) Re-compression RD performance of different idempotent architectures on Kodak dataset (except for w . GDN, for which we report first-time compression as it is non-idempotent). (b) qualitative results for w. (right) or w.o. (left) coupling enhancement.
Inverse v.s. Right-Inverse The importance of right-inverse for idempotent lossy compression can already be seen from the difference between the proposed framework and [Helminger et al., 2021] in Fig. 3(a) and Tab. 1. To further demonstrate this importance, we test what happens if the proposed framework is changed from right-invertible to invertible. Specifically, we increase the output dimension of the proposed blocked convolution (Sec. 3.1) to its input dimension, so that it becomes fully invertible. We also test what happens if the proposed blocked convolution is replaced with invertible  $1 \times 1$  convolution [Kingma and Dhariwal, 2018]. For both cases, the encoding transform is fully invertible.
Fig. 4 (a) shows that both invertible blocked convolution (inv.) and invertible  $1 \times 1$  convolution (w. conv1x1) suffer dramatic RD performance degradation compared with the proposed right-invertible framework (proposed). This result further demonstrated that it is important for idempotent LIC to have right-inverse rather than inverse.
Impact of Null-Space Enhancement Null-space enhancement is introduced in our framework to enable an adaptive right-inverse for linear layers rather than a fixed Moore-Penrose pseudo-inverse. As is shown in Fig. 4, for the two lower-bpp points, the framework with null-space enhancement
(proposed) has a 1 dB advantage over the framework without null-space enhancement (w.o.NE), whereas for the two higher-bpp points this advantage shrinks to be smaller than 0.5 dB. The reason is that the two lower-bpp points have a smaller code channels (192) than the two higher-bpp points (320). A smaller code channel means a larger null space, and thus null-space enhancement can make larger improvement.
Impact of Coupling Enhancement Coupling enhancement improves the receptive field of convolution. The proposed blocked convolution is parallel friendly, but also restricts the receptive field. Qualitatively, this constraint results in obvious deterioration in the flat area, such as the wall, the sky and the face, as is shown in the left column of Fig. 4(b). By introducing coupling enhancement after blocked convolution, the restriction is lifted and the block artifact on the reconstructed images are removed, as is shown in the right column of Fig. 4(b). Quantitatively, the RD performance is also improved by coupling enhancement, which is shown by comparing proposed and w.o.c-en in Fig. 4(a).
Impact of Right-Invertible Quantization As is pointed out in Sec. 3.3, the mean-shifted quantization is not guaranteed to be idempotent, and we propose two alternatives to circumvent this issue. In the proposed framework, we use the second-type circumvention. Here we compare these two choices. As is shown in Fig. 4(a), first-type circumvention (w.int mean) performs slightly worse than second-type circumvention (proposed). This is because first-type circumvention forces the mean to be integer, whereas second-type does not has this constraint. However, the second-type circumvention requires calculating the quantized CDF on the fly during inference.
Impact of Right-Invertible GDN To make the inverse of GDN layer actually computable in the proposed framework, we combine the coupling structure [Papamakarios et al., 2021] with GDN, as is described in Sec. 3.2. Here we demonstrate that such workaround does not affect the RD performance. Specifically, we change the c-GDN and c-GDN $^{-1}$  layer in our framework (Fig. 2(b)) back to GDN and iGDN layer [Ballé et al., 2015]. Note that this change makes the framework non-idempotent, and is only used to test the RD performance. From the results in Fig. 4(a) we see that, Whether to use coupling GDN (proposed) or original GDN (w . GDN) has negligible influence on RD performance. Thus coupling GDN is an efficient and effective replacement for GDN in idempotent compression framework.
The idempotence has been a crucial consideration for lossy image codec. For image codec without prediction encoding like JPEG [Wallace, 1991] and JPEG2000 [Taubman et al., 2002], idempotence is naturally assured as invertible encoding transforms like DCT or DWT is adopted. As long as the quantization is idempotent (which is true for scalar quantization), the whole codec becomes idempotent [Joshi et al., 2000, Richter et al., 2017].
The idempotence of learned image compression is firstly studied by [Kim et al., 2020], which proposes a near-idempotence solution that alleviates re-compression loss but does not eliminate it. [Cai et al., 2022] further improves over [Kim et al., 2020] while it is not able to achieve strict idempotence. [Helminger et al., 2021] is the first LIC that achieves idempotence, using fully invertible normalizing flow as encoding transform. However, it's RD performance dramatically falls behind modern LIC.
Our work is also related to SurVAE flow [Nielsen et al., 2020]. On the one hand, the idempotence of LIC requires the encoding transform to be surjective, which is similar to the goal of SurVAE flow. The difference is, we consider deterministic right inverse, whereas SurVAE flow considers stochastic right inverse. On the other hand, the techniques proposed in this paper, such as blocked convolution and null-space enhancement, can be used to improve SurVAE flow.
6 Discussion & Conclusion
To conclude, we first identify that invertibility is sufficient but not necessary for idempotent codec, and it can be instead relaxed to right-invertibility. Based on this identification, we investigate the practical implementation of right-inverse with efficiency and expressiveness. Empirically, We show that the proposed method achieves state-of-the-art RD performance among idempotent codec's. Furthermore,
our codec can be easily relaxed into a near-idempotent codec, which also achieves state-of-the-art re-compression performance among near-idempotent codecs.
For future work, one possible direction is constructing right-invertible transform without function composition. Currently, the right-invertible transform is constructed using composition of surjections. Such construction strictly restricts the latent dimension to be non-increasing throughout the transform. This restriction is conflict with the prevalent design logic of neural network and is detrimental to expressiveness. It would be interesting to see if this restriction could be removed and how much the performance could be improved.
Funded by Baidu Inc. through Apollo-AIR Joint Research Center.
Matrix multiplication background user's guide. URL https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html.
Moore-penrose inverse. URL https://en.wikipedia.org/wiki/Moore-Penrose_inverse.
J. Balle, V. Laparra, and E. P. Simoncelli. Density modeling of images using a generalized normalization transformation. arXiv preprint arXiv:1511.06281, 2015.
J. Balle, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
J. Balle, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations, 2018.
J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm. Overview of the versatile video coding (vvc) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, 31(10):3736-3764, 2021.
S. Cai, Z. Zhang, L. Chen, L. Yan, S. Zhong, and X. Zou. High-fidelity variable-rate image compression via invertible activation transformation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2021-2031, 2022.
Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7939-7948, 2020.
L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
R. Franzen. Kodak lossless true color image suite. source: http://r0k.us/graphics/kodak, 4(2), 1999.
D. He, Y. Zheng, B. Sun, Y. Wang, and H. Qin. Checkerboard context model for efficient learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14771-14780, 2021.
D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5718-5727, 2022.
L. Helminger, A. Djelouah, M. Gross, and C. Schroers. Lossy image compression with normalizing flows. In Neural Compression: From Information Theory to Applications-Workshop@ ICLR 2021, 2021.
R. L. Joshi, M. Rabbani, and M. A. Lepley. Comparison of multiple compression cycle performance for JPEG and JPEG 2000. In Applications of Digital Image Processing XXIII, volume 4115, pages 492-501. SPIE, 2000.
J.-H. Kim, S. Jang, J.-H. Choi, and J.-S. Lee. Instability of successive deep image compression. In Proceedings of the 28th ACM International Conference on Multimedia, pages 247-255, 2020.
D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible  $1 \times 1$  convolutions. Advances in neural information processing systems, 31, 2018.
A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128 (7):1956-1981, 2020.
M. Lezcano-Casado. Trivializations for gradient-based optimization on manifolds. In Advances in Neural Information Processing Systems, NeurIPS, pages 9154-9164, 2019.
S. Mac Lane. Categories for the working mathematician, volume 5. Springer Science & Business Media, 2013.
D. Minnen, J. Balle, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
D. Minnen, S. Singh, and J. Balle. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pages 3339-3343. IEEE, 2020.
D. Nielsen, P. Jaini, E. Hoogeboom, O. Winther, and M. Welling. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33: 12685-12696, 2020.
G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1): 2617-2680, 2021.
K. H. Randall. *Cilk: Efficient multithreaded computing*. PhD thesis, Massachusetts Institute of Technology, 1998.
T. Richter, J. Keinert, A. Descampe, G. Rouvroy, and A. Willeme. Multi-generation-robust coding with jpeg xs. In 2017 IEEE International Symposium on Multimedia (ISM), pages 6-13. IEEE, 2017.
J. Schwab, S. Antholzer, and M. Haltmeier. Deep null space learning for inverse problems: convergence analysis and rates. Inverse Problems, 35(2):025008, 2019.
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 22(12): 1649-1668, 2012.
D. S. Taubman, M. W. Marcellin, and M. Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 11(2):286-287, 2002.
G. K. Wallace. The JPEG still picture compression standard. Communications of the ACM, 34(4): 30-44, 1991.
Y. Wang, Y. Hu, J. Yu, and J. Zhang. Gan prior based null-space learning for consistent superresolution. arXiv preprint arXiv:2211.13524, 2022a.
Y. Wang, J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490, 2022b.
A Additional Explanation for the Methods
A.1 Complexity Analysis of Block Convolution
To discuss this problem, consider a 2-dimension convolution with kernel size  $K \times K$ , stride  $S \times S$ , input spatial size  $H \times W$ , input channel  $C_i$  and output channel  $C_o$ . Assuming the in-place parallel matrix multiplication[Randall, 1998] is used.
For serial method, as is described in Line 104-109, we first subtract the influence of the already-solved pixels, which takes  $O(C_i K^2)$  time. Then we solve for the rest pixels, which takes  $O(C_o)$  time. The point is, this subtract-then-solve procedure needs to be done  $O(HW / S^2)$  times, in serial, so the overall complexity is  $O(\frac{HW}{S^2} (C_i K^2 + C_o))$ .
For parallel method implemented with the proposed blocked convolution, we can solve for all pixels with  $O(C_{o})$  time complexity.
We can see that, compared with the parallel method, time complexity of serial method is forbiddingly higher and not viable for practical usages. In terms of re-compression performance, both serial method and parallel method are right-inverse, so idempotence can be achieved for both.
A.2 More on Null-space Enhancement
Parameterization of Surjective Linear Transform
For a linear transform with kernel  $K \in \mathbb{R}^{D \times d}$  to be surjective,  $K$  must have full column rank, i.e.,  $d$  must be less or equal to  $D$  and the rank of  $K$  is  $d$ . The vanilla parameterization of  $K$  as a matrix cannot ensure this property. Thus, we use singular value decomposition (SVD) parameterization to ensure that  $K$  is surjective. Specifically,  $K$  is decomposed as  $K = USV^T$ , where  $U$  is a  $D \times d$  orthonormal matrix,  $S$  is a  $d \times d$  diagonal matrix with non-zero diagonal elements, and  $V$  is a  $d \times d$  orthonormal matrix. With this decomposition,  $K$  is guaranteed to have full column rank, and is thus surjective. Accordingly, Eq. 10 is parameterized as
$$
X = Y V S ^ {- 1} U ^ {T} + f (Y) \left(I - U U ^ {T}\right) \tag {15}
$$
where  $S^{-1}$  is the inverse of non-zero diagonal matrix  $S$ . For orthonormal matrix  $U$  and  $V$ , we adopt the parameterization in [Lezcano-Casado, 2019]. The matrix  $S$  is diagonal and trivial to parameterize. To avoid arithmetic overflow, we restrict the diagonal elements of  $S$  to be within [0.1, 10].
Derivation of Null-space equality
$\forall F \in \mathbb{R}^{b \times D}, X = YK^{+}K + F(I - KK^{+})$  is a solution to  $Y = XK$ .
Proof. From the property of Moore-Penrose pseudo-inverse [Moo] we know that
$$
K K ^ {+} K = K. \tag {16}
$$
Additionally, for a orthonormal  $K$  (with linearly independent columns), we have
$$
K ^ {+} K = I. \tag {17}
$$
Then, right-multiply  $X(Y;K)$  by  $K$ , we get
$$
\begin{array}{l} X K = Y K ^ {+} K + F (I - K K ^ {+}) K \tag {18} \\ = Y + F K - F K = Y. \\ \end{array}
$$
B.1 More Experiment Setup
The detailed encoding and decoding transform is illustrated in Fig. 5. To extend the idempotent framework to near-idempotent framework, we change the first blocked convolution in the encoding
(a) Details of the proposed imdempotent framework
(b) Details of the proposed near-imdempotent framework
Figure 5: Detailed encoding and decoding transform for the proposed idempotent and nearidempotent framework. blocked refers to the proposed blocked convolution. The (input, output) channels are annotated in the brackets, and stride is annotated using down-arrow. NE refers to the proposed null-space enhancement, and  $f$  is learned parametric function. Here RBU/RBD are the residual block used for upsampling/downsampling (Fig. 6(b)), respectively. c-enhance refers to the proposed coupling enhancement using coupling structure (Fig. 6(a)). The kernel size of the convolution is annotated in the brackets. c-GDN refers to the proposed right-invertible normalization using coupling structure (Fig. 6(a)). Blanked rectangular refer to the right-inverse/inverse of the corresponding layer, and has no additional weights.
(a) Coupling structure used in c-enhence and c-GDN
(b) Residual block used in null-space enhancement
Figure 6: Submodules used in the proposed framework: (a) coupling structure used in c-enhance and c-GDN; (b) residual block downsampling (RBD) and residual block upsampling (RBU) used in the  $f(\cdot)$  of null-space enhancement.
(b)
Figure 7: Additional experiments on: (a) extendability of the proposed idempotent framework (b) functionality of different components of the proposed near-idempotent framework.
transform to non-surjective by increasing its output channel number from 10 to 128. Since this blocked convolution is no longer surjective, it is no longer right-invertible. However, its corresponding layer in the decoding transform can be make surjective and right-invertible. Thus, we make its corresponding layer in the decoding transform surjective, and use the null-space enhancement on the encoding side.
Coupling structure [Dinh et al., 2016] used in coupling enhancement (c-enhance) and coupling GDN (c-GDN) is illustrated in Fig. 6(a). For c-enhance, the scale  $s(\cdot)$  and translation  $t(\cdot)$  are convolution. For c-GDN, the scale  $s(\cdot)$  and translation  $t(\cdot)$  are GDN [Balle et al., 2015]. Following the usage guideline in [Dinh et al., 2016], we concatenate two coupling structure with the opposite way of splitting in one c-enhance/c-GDN.
We provide additional experiments on (a) extendability of the proposed idempotent framework (b) functionality of different components of the proposed near-idempotent framework. Specifically, to demonstrate the extendability of the proposed idempotent framework, we replace GDN with residual blocks as suggested by [He et al., 2022]  $(\mathbb{R}\mathbb{B}\times 1$  for one residual block, or  $\mathbb{R}\mathbb{B}\times 3$  for three residual blocks), and report the RD performance in Fig.7(a). It is clear that, replacing GDN with the more recent residual blocks would also improve our framework to a similar degree. Thus our proposed framework is compatible with the recent advances in LIC and has good extendability.
To analysis the functionality of different components of the proposed near-idempotent framework, we test what happens if the modification to each component is not applied, and report the PSNR drop during re-compression in Fig.7(b). Specifically, we test keeping the GDN layers unchanged (w. gdn) or keeping the convolution layers unchanged (w. conv). Keeping both the GDN layers and the convolution layers would reduce to baseline Balle2018. The results show that keeping more layers unchanged may slightly improve first-time compression performance, but is evidently harmful to re-compression performance. The proposed near-idempotent framework has the best re-compression performance among these settings.
We include the implement of other CODECs as follows. For CODEcs that have open-source implementations, we use that implementation. For CODEcs that do not have open-source implementations, we either use the data provided in the paper, or re-implement by ourselves if the detailed architecture is provided.
- Implementations from CompressAI [Bégaint et al., 2020]: Balle2017[Balle et al., 2017], Balle2018[Balle et al., 2018], Cheng2020[Cheng et al., 2020], JPEG2000[Taubman et al., 2002], BPG444[Sullivan et al., 2012], VTM444[Bross et al., 2021]
- Data from the original papers: Helminger2021[Helminger et al., 2021], Cai2022[Cai et al., 2022]
- Our re-implementation: Kim2020[Kim et al., 2020]. Specifically, we re-implement the FI loss proposed in this work on Balle2018 [Balle et al., 2018].
B.2 More Quantitative & Qualitative Results
See Fig. 8-11 for more quantitative results. See Fig. 12-15 for more qualitative results.
C More Discussion
C.1 Limitation
In this work, the surjective encoding transform is constructed using function composition of simple surjections. This construction strategy limits the latent dimension to be non-increasing throughout the encoding transform. This limitation contradicts the mainstream design logic of neural network, and is harmful to expressiveness.
A function composition of surjections is always a surjection, but a surjection needs not to be a function composition of surjections [Mac Lane, 2013]. Thus this restriction could be lifted by more advanced construction strategy of surjection.
C.2 Broader Impact
Improve the rate-distortion of re-compression has positive social impact. Re-compression constantly happens in the transmission and redistribution of image data. Reducing the bitrate can save the resources, energy and the carbon emission during these processes.
C.3 Reproducibility Statement
All theoretical results are proven in Appendix. A. For experimental results, all the datasets used are publicly available, and the implementation details are provided in Appendix. B. Furthermore, the source code for reproducing experimental results are provided in supplementary materials.
Figure 8: PSNR-BPP curve on Kodak. Idempotent codesc are marked as idemp, near-idempotent codec are marked as near id and non-idempotent codec are marked as non id. First-time compression performance is plotted in dotted line, and re-compression performance (upto 50 times) is plotted in solid line.