Idempotence and Perceptual Image Compression
Authors: Tongda Xu
Ziran Zhu
Dailan He
Yanghao Li
Lina Guo
Yuanyuan Wang
Zhe Wang
Hongwei Qin
Yan Wang
Jingjing Liu
Ya-Qin Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: perceptual image compression
neural image compression
Primary area: generative models
TL;DR: Indempotence constraint inversion of unconditional generative model achieve perceptual image compression.

Abstract: Idempotence is the stability of image codec to re-compression. At the first glance, it is unrelated to perceptual image compression. However, we find that theoretically: 1) Conditional generative model-based perceptual codec satisfies idempotence; 2) Unconditional generative model with idempotence constraint is equivalent to conditional generative codec. Based on this newfound equivalence, we propose a new paradigm of perceptual image codec by inverting unconditional generative model with idempotence constraints. Our codec is theoretically equivalent to conditional generative codec, and it does not require training new models. Instead, it only requires a pre-trained mean-square-error codec and unconditional generative model. Empirically, we show that our proposed approach outperforms state-of-the-art methods such as HiFiC and ILLM, in terms of Fréchet Inception Distance (FID). The source code is provided in https://github.com/tongdaxu/Idempotence-and-Perceptual-Image-Compression.

Published as a conference paper at ICLR 2024
IDEMPOTENCE AND PERCEPTUAL IMAGE COMPRES- SION
Tongda Xu :, Ziran Zhu Dailan He Yanghao Li Lina Guo Yuanyuan Wang* "Inttitute for AI Industry Research, Tsinghua University
2 DDpartment of Computer Science and Technology, Tsinghua University
Zhe Wang 22 Hongwei Qin', Yan Wang",, Jingjing Liu 1,6 & Ya-Qin Zhang 1.2.6+
Institute of Software, Chinese Academy of Sciences, 'SenseTime Research The Chinese University of Hong Kong, School of Vehicle and Mobility, Tsinghua University x. wowgdaanny..eaeeeoooeeooee wangyan ts innhuauua... edu.cn
ABSTRACT
Idempotence is the stability of image codec to re-compression. At the first glance, it is unrelated to perceptual image compression. However, find that theoretically: 1) Conditional generative model-based perceptual codec satisfies idempotence; 2) Unconditional generative model with idempotence constraint is equivalent to conditional generative codec. Based on this newfound equiva- lence, we propose new paradigm of perceptual image codec by inverting conditional generative model with idempotence constraints. Our codec is theo- retically equivalent to conditional generative codec, and it does not require train- ing new models. Instead, it only requires pre-trained rrean-squareeerov codec and unconditional generative model. Empirically, we show that our proposed approach outperforms state-of-the-a methods such HiFiC (Mentzer et al., 2020) and ILLM (Muckley et al. 2023), terms of Fréchet Inception Distance (FID). The source code is provided in https:::///. http:::///gitttt com/ wwt.:/oo///ooo////o/o/oo/// Idempotence- and Percedduerrpeulleeeec Image Image-
cara
Source (ImageNet)
HiFiC 0.17 0pp
0.11 bpp
Proposed 0.11 bpp
Figure 1: A visual comparison of our proposed approach with sate-of-the-ane perceptual image codee, such HiFiC (Mentzer al. 2020) and ILLM (Muckley et al. (023).
INTRODUCTION
Idempotence refers to the stability of image code to re-compression, which is crucial to practical image codec. For traditional codec standard (e.g.. JPEG (Wallace 1991), JPEG2000 (Taubman al.)
"o whom the correspondence should be addressed.

---

Published as a conference paper at ICLR 2024
2002). JPEG-XL (Alakuijala al. 2019); idempotence is already taken into consideration. For
neural image compression (NIC), by default idempotence is not considered. Thus, specific methods
such invertible network (Helminger et al. 2021 Cai et al. 2022 Li et al. 2023) and regularization loss (Kim et al. 2020) have been proposed to improve the idempotence of NIC methods.
In the meantime, there has been growing success in perceptual image compression. Many recent studies achieve perceptual near-lossless result with very low bitrate (Mentzer et all; (2020) Muckley al. 2023). The majority of perceptual image compression methods adopt conditional genera- tive model (Tscchnnenn et al,, 2018) Mentzer et al. 2020; He et al. 2022b Agustsson et al. 2022]
Hoogeboom et al. 2023) Muckley et al. (202)). More specifically. they train decoder that lears the posterior of a natural image conditioned on the bitstream. This conditional generative model-basee
approach later theoretically justified by Blau & Michaeli (2019); Yan et al. (2021), who show that such approach achieves perfect perceptual quality and is optimal in terms of rate-distortion- perception trade-off when the encoder deterministic.
At the first glance, idempotence and perceptual image compression are unrelated topics. Indeed, researchers in those two areas barely cite each other. However, we find that idempotence and per- ceptual image compression are, in fact, closely related. More specifically, we demonstrate that: 1) Perceptual image compression with conditional generative model brings idempotence; 2) Un- conditional generative model with idempotence constraints brings perceptual image compression. Inspired by the latter, we propose a new paradigm of perceptual image compression, by inverting an unconditional generative model with idempotence constraint. Compared with previous condi- tional generative codec, our approach requires only a pre-trained unconditional generative model and -quarer-rrr (MSE) codec. Furthermore, extensive experiments empirically show that our proposed approach achieves state-of-the-art perceptual quality.
PRELIMINARIES
2.1 IDEMPOTENCE OF IMAGE CODEC
Idempotence of image codec refers to the codec's stability to re-compression. More specifically, denote the original image as X, the encoder as f.), the code as Y f(X). the decoder as g(-) and the reconstruction as X = g(Y). We say that the codee idempotent if
f(X) Y. or g(f())) = X,
(1)
i.c., the codec is idempotent if the re-compression of reconstruction X produces the same result.
2.2 PERCEPTUAL IMAGE COMPRESSION
In this paper, use Blau & Michaeli (2018)'s definition of perceptual quality. More specifically, we say that the reconstructed image X has perfect perceptual quality if
(2) where Px is the source image distribution and px is the reconstruction image distribution. In this paper, we slightly abuse the word "perception" " for this divergence-based perception, and use the word "human perception" for human's 's perception instead.
Px = Px:
The majority of perceptual image codec achieves perceptual quality by conditional generative model (Tchannen et al.] 2018; Mentzer et al.) 2020 He et al. 2022b [gusssson al. 2022 Hooge- boom et al.| 2023). More specifically, they train a conditional generative model (such as conditional generative adversial network) to approximate the real image's posterior on the bitstream. And the reconstruction image X is obtained by sampling the posterior:
-YY) Px|y, where Y f(X).
(3)
Blau & Michaeli 2019 prove that this conditional generative codec achieves perfect perceptual quality Px px- Further, when f.) achieves the optimal MSE A", then the MSE of perceptual reconstruction is bounded by twice of the optimal MSE:
E|||X | 2A"
Moreover, Yan et al. 2022)) further justify conditional generative codec by proving that it is optimal in terms of tite-distortioon-prrepti (Blau & Michaeli 2019) among deterministic encoders.
2

---

Published as a conference paper at ICLR 2024
Idempolence
Natisfies
is equivalent 10
Perceplual codec w/
conditional generative model
unconditional generative model
Figure The relationship between idempotence and perceptual image compression.
3 IDEMPOTENCE AND PERCEPTUAL IMAGE COMPRESSION
In this section, connect idempotence and perceptual image compression. Specifically, we show that ideal conditional generative codec idempotent. On the other hand, sampling from uncon- ditional generative model with ideal idempotence constraint leads to conditional generative codec. Their relationship is shown Fig.
3.1 PERCEPTUAL IMAGE COMPRESSION BRINGS IDEMPOTENCE
We first show that ideal conditional generative codec satisfies idempotence, i.e, X Px|y = f(x) Y. Given specific value with non-trivial probability py y) * define the inverse image of y as f-1(y) [y] = {x) = y}. By definition, all the elements x E f [y] encode into Then if we can show X E f [y]. we can show that this perceptual codec is idempotent.
To show that X E [-]( let's consider a specific value with non-trivial probability px r) + O. As Y = s(X) a deterministic transform, when æ & [y] we have the likelihood Prix (Y y/X =) = 0. Therefore, for any x f- [y], we have (X 2, Y y)
Py = y/X xXpx(X x) 0. As pxlY Pxy for any & [y), have the posterior (X = x/Y = y) = O. And therefore, for any sample X ~ Yx we have Pr(x [-) 1, i.e., X E fwl almost surely. By definition of f f-]l this codec is idempotent. We summarize the above discussion as follows:
Theorem 1. (Perceptual quality brings idempotence) Denote X as source, f(.) as encoder, Y = f(X) as bitstream, g(.) as decoder and X = g(Y) as reconstruction. When encoder f(.) is deter- ministic, then conditional generative model-based image codec is also idempotent, i.e.,
== PXPY 1(8)=11
(5)
3.2 IDEMPOTENCE BRINGS PERCEPTUAL IMAGE COMPRESSION
In previous section, we have shown that perceptual quality brings idempotence. It is obvious that the simple converse is not true. A counter-example is JPEG2000 (Taubman et al. 2002), which is idempotent by design but optimized for MSE. However, we show that with unconditional generative model, idempotence does bring perceptual quality.
More specifically, we want to show that sampling from unconditional distribution X ~ px with idempotence constraint f = Y, is equivalent to sampling from the posterior pxjy- Again, we consider a non-trivial y with py (Y = y) + 0. Similar to previous section, as f(.) is deterministic, have Pr|x (Y X I) 1 if x E f (-()), [p)] and pr (Y y|X x) 0 i f-).
Then, we can compute the posterior distribution as
Pxiv(X x = y) x px (X = x)prix((Y = y|x =
{ px = x), I f f- x r- [v]-
(6)
0,
The above equation shows that when x $ {-)) the posterior likelihood Pxlx (X = x v) and no sample with value can be generated almost surely. And when x f [y], the posterior
likelihood Pxly (X = x|Y = y) is proportional to the source distribution px (X = x). Therefore,
sampling from the source Px with idempotence constraint f(X) = Y is equivalent to sampling from the posterior p We summarize the above discussion follows:
3

---

Published as a conference paper at ICLR 2024
Theorem 2. (Idempotence brings perceptual quality) Denote X as source, f() encoder, Y f(X) as bitstream, g(-) as decoder and X = =YY) as reconstruction. When encoder f(() is de- terministic, the unconditional generative model with idempotence constraint is equivalent to the conditional generative model-based image codec, i.e.
X ~ Px s.t. f(X) = Y PXIY
(7)
And more conveniently. the unconditional generative model with idempotence constraint also satis- fies the theoretical results from rate- wete--ittttoo-trreeeee (Blau & Michaeli 2019 Yan et al. (021])
Corollary 1. f() the encoder of codec with optimal MSE A', then the unconditional gener- ative model with idempotence constraint also satisfies
Px = px, EI| S
Furthermore, the codec induced by this approach is also optimal among deterministic encoders.
Besides, those results can be extended image restorations (See Appendix, A
PERCEPTUAL IMAGE COMPRESSION BY INVERSION
4.1
GENERAL IDEA
Theorem implies new paradigm to achieve perceptual image compression by sampling from pre-trained unconditional generative model with the idempotence constraint. More specifically, we can rewrite the left hand side of Eq. in Theorem 2 as
min [I s PX.
Then the problem has exactly the same form as broad family of works named 'generative model inversion' for image super-resolution and other restoration tasks (Menon et al. 2020; Daras et al.) Wang et al. 2022 Chung al. 2022a). From their perspective, f(.) is the down-sample operator, Y is the down-sampled image and [I/(X) YII is the "consistency" penalization that secures the super-resolved image X corresponds to the input down-samples image. The X px the 'realness" term, and ensures that X lies on the natural image manifold (Zhu et al., 2016). And solving Eq. iis the same as finding a sample that satisfies the consistency, which is the inverse problem of sampling. Therefore, they name their approach 'inversion of generative model'. For us, the f-) operator is the encoder, and Y is the bitstream. And long encoder is differentiable, we can adopt their inversion approach for image super-resolution to inverse the codec.
2021
4.2 ENCODE AND DECODE PROCEDURE
To better understand our proposed approach, we describe the detailed procedure of communicating an image from sender to receiver and implementation details. We start with the pre-trained model that is required for the sender and receiver:
The sender and receiver share a MSE optimized codec with encoder fo(.), decoder (o(.). Despite we can use any base codec, MSE-optimized codec leads to tightest MSE bound.
The receiver has unconditional generative model qx approximating the source px.
And the full procedure of encoding and decoding image with our codec is follows:
The sender samples an image from the source X px-
The sender encodes the image into bitstream Y fo(X), with the encoder of pre-trained MSE codec. And Y transmitted from sender to receiver.
Upon receiving Y, the receiver inverses a generative model with idempotence constraint:
min (x) Y11. s.t. & qx.
(10)
4

---

Published as a conference paper at ICLR 2024
For practical implementation, straint. This is because most generative model inversion methods (Menon et al. 2020; Daras
not directly the binary bitstream Y for idempotence
et al. 2021: Chung et al. 2022a) use the gradient V Vxllfo(X)) - -112 On the other hand, for most NIC approaches, the entropy coding with arithmetic coding (Rissanen & Langdon differentiable. We propose two alternative constraints to this problem:
1979) is not
y-domain constraint: We do not constrain the actual bitstream Y. Instead, we constrain the quantized symbols before arithmetic coding, Further, we use pass the gradient through quantization. And we call this constraint y-domain constraint.
through-estimaton to
x-domain constraint: On the other hand, constrain the difference between the MSE reconstructed image of source X and the MSE
also decode the re-compressed image and
reconstructed image of sample. More specifically, instead of solving Eq. T0 we solve:
min (loo(f(( 9o (2 s.t. X qx
(11)
Similarly, the quantization relaxed by STE allow gradient pass. And we call this constraint x-domain constraint.
The y-domain and x-domain constraint correspond to two idempotence definitions in Eq. | And when gu(-) is MSE optimal, those two constraints equivalent (See Theorem of Appendix. A) And beyond STE, other gradient estimators such as additive noise, salpi- noise and stochastic
gumbel annealing can also be adopted (Ballé et al. 2017 Xu et al. 2022 Yang al. 2020).
4.3 COMPARISON PREVIOUS WORK
Compared with previous works using conditional generative model (Mentzer et al., 2020 Muckley al., 2023), our proposed approach does not require specific conditional generative model for dif- ferent codec and bitrate. It only requires one unconditional generative model, and it can be directly applied on even pre-trained MSE codec. Furthermore, it is theoretically equivalent to conditional generative codec, which means that it conveniently shares the same theoretical properties in terms of titeeddistottion-eeeeptio trade-off (Blau & Michaeli 2019).
Compared with previous proof of concept using unconditional generative model (Ho et al. 2020 Theis et al. 2022), our proposed approach does not require bits-back coding (Townsend et al. 2018) sample communication (Li & El Gamal 2018), which might have rate complexity overhead. Further, it is implemented as a practical codec. Moreover, our approach uses exactly the same bitstream Y as MSE codec. This means that the receiver can choose to reconstruct MSE optimized image or perceptual image and even achieve perception--iistotionn trade-off (See Appendix, (
5
EXPERIMENTS
5.1
EXPERIMENT SETUP
Metrics We use Fréchet Inception Distance (FID) measure perceptual quality. This is because use Blau & Michaeli (2018)'s 's definition of perceptual quality, which the divergence between two image distributions. And FID is the most common choice for such purpose. We use MSE and Peak- Signal-Noise-Ratee (PSNR) to measure distortion, which are the default choice for image codec. We use bpp (bits per pixel) measure bitrate. To compare codec with different bitrate, we adopt Bjontegaard (BD) metrics (Bjontegaard] 2001): BD-FID and BD-PSNR. Those BD-metrics can be seen as the average FID and PSNR difference between codec over their bitrate range.
Datasets Following previous works in unconditional image generation (Krrras et al. 2019 Ho et al., 2020), we train our unconditional generative models on FFHQ (Karras al.) 2019) and ImageNet (Deng et al.) 2009) dataset. As (Chung et al.) 2022a). we split the first 1000 images of FFHQ as test set and the rest for training. And we use first 1000 images ImageNet validation split as test set and use the ImageNet training split for training. To test the generalization ability of our method on other datasets, we also use first 1000 images of COCO (Lin et al.) 2014) validation split and CLIC 2020 (Toderici al. 2020) test split as additional test set. As previous works (Karras et al. 2019 Ho et 20200) in unconditional generative model, we central crop image by their short edge and rescale them to 256 256.
5

---

Published as a conference paper at ICLR 2024
Previous State-of-the-art Perceptual Codec We compare approach against previous state-of-
the-art perceptual codec: HiFiC (Mentzer et al., 2020), Po-El Po-ELIC (He et al. 2022b), CDC Yang & Mandi 2023) and ILLM (Muckley et al. 2023). HiFiC is the first codec achieving perceptual near lossless compression with very low bitrate. Po-ELIC is the winner of CVPR CLIC 2022, a major competition for perceptual codec. CDC is very recent perceptual codec using conditional diffusion model. ILLM is the latest state-of-the-ar perceptual codec. We use the official model to test HiFiC, CDC and ILLM. We contact the authors of Po-ELIC obtain the test results on our datasets. For fairness of comparison, we also test re- re-trained version of HiFiC (Mentzer et al. 2020) and ILLM (Muckley et al., )023) on FFHQ and ImageNet dataset. So that their training dataset becomes the same ours. We acknowledge that there are other very competitive perceptual codec (Iwai et al.)
2021 Ma et al. 2021: Agustsson et al. 2022 Goose al., 2023) Hoogeboom et 2023), while we have not included them for comparison as they are either unpublished yet or do not provide pre-trained model for testing.
MSE Codec Baseline As we have discussed, approach requires MSE optimized codec base model. As we only requires the codec to be differentiable, most of works in NIC (Ballé et al. 2018 Minnen al. 2018 Minnen & Singh 2020 Cheng et al., 2020 He et al., 2022a Liu et al. 2023 can be used. Among those MSE codec, we choose two representative models: Hyper (Ballé et al, 2018) and ELIC (He et al., 2022a). Hyper is perhaps the most influential work in NIC. Its two level hyperprior structure inspires many later works. ELIC is the first NIC approach that outperforms state-of-the-at manual codec VTM (Bross 2021 with practical speed. For reference, we also choose two traditional codec baseline; BPG and VTM Bross et al. 2021).
Base
Inversion
Constraint
bpp
FID |
MSE J
MSE bound
ELIC
94.35
91.75
StyleGAN2
PULSE
X-domain
15.33
754.2
StyleGAN2
ILO
x-domain
26.15
689.2
0.07
Proposed (ELIC)
DDPM
MCG
x-domain
135.1
929.2
183.5
DDPM
DPS
y-domain
5.377
189.2
DDPM
DPS
x-domain
5.347
161.9
Table I: Ablation study with FFHQ and ELIC. Bold: best FID. Bold red: too large MSE.
Souree (FFHQ)
StyleGAN2 PULSE
StyleAN2 ILO
DDPM MCG
DDPM DPS
Figure Ablation study on unconditional generative model with FFHQ and ELIC.
5.2 ABLATION STUDY
Unconditional Generative Model As we have discussed, our approach requires an unconditional generative model for inversion. The most adopted models for inverse problem are StyleGAN family (Karras et al, 2019) and DDPM (Ho et al. 2020). Multiple approaches are proposed to invert StyleGAN and DDPM. And they are shown be effective for image super-resolution and other restoration tasks (Menon et al. 2020; Daras et al. 2021 Roich et al., 2022 Daras et al., 2022 Wang et al. 2022 Chung al. 2022b a). For StyleGAN inversion, we choose two mostly adopted approach: PULSE MMenon et al. 2020) and ILO (Daras et al. 2021). For DDPM inversion, choose MCG (Chung et al. (022b) and DPS (Chung et al. 2022a) as they are the most popular methods for non-linear inversion.
To find the most suitable unconditional generative model, we compare StyleGAN2 + PULSE. Style- GAN2 + ILO, DDPM + MCG and DDPM + DPS. The implementation details are presented in Appendix. B.I As Tab. and Fig. 3 show, compared with DDPM + DPS, other three methods either have too high FID have MSE significantly larger than MxMSE of ELIC. Further, they look

---

Published as a conference paper at ICLR 2024
visually less desirable and less similar to the source. Therefore, DDPM DPS inversion in
later experiments.
Domain of Idempotence Constraint As we have discussed, there two kinds of constraint for idempotence, namely y-domain and x-domain. theory, those two constraints are equivalent (Sce Theorem of Appendix AA.. While in practice, they can be different as the strict idempotence condi- tion So (X) = might not be achieved. We compare the y-domain and x-domain in Tab. Those two constraints achieve similar FID while x-domain has lower MSE. This might be due to that x- domain directly optimizes MSE on pixel-level. Therefore, we choose x-domain constraint in later experiments.
Source (ImageNet)
HiFiC 0.17 hpp
ILLM 0.11 hpp
Proposed 0.10 hpp
Sourse (CLIC)
HiFiC 0.16 hpp
ILLM 0.11 bpp
Proposed 0.10 bpp
Figure 4: visual comparison of our proposed approach with ofate-ofthh-aadt perceptual image codec, such as HiFiC (Mentzer et al. 2020) and ILI.M (Muckley et al. 2023).
20
13
Reconstruction std (N-16)
Three altemaatie reconstructions of proposed approach in addition to Fig.
Figure 5: Reconstruction diversity of proposed approach.
5.3
MAIN RESULTS
Perceptual Quality We compare the perceptual quality, in terms of FID, with state-of-the-art per- ceptual codec on multiple datasets. The results are shown in Tab. 2 and Fig. [T of Appendix. [B3] Tab. shows that our approach with ELIC (He et al. (2022a) achieves the lowest FID on all datasets. Furthermore, our approach with Hyper (Ballé et al. 2018) achieves second lowest FID on all datasets. We note that the base codec HiFiC and ILLM is Mean-Scale Hyper (Minnen et al.) 2018), which outperforms the Hyper. On the other hand, the base codec of Po-ELIC is ELIC, which is the same as ELIC. Besides, our approach outperforms CDC, which uses DDPM as generative model like Additionally, on FFHQ and ImageNet dataset, our approach also outperforms HiFiC and ILLM re-trained on those two dataset. Therefore, is clear that our approach outperforms pre- vious perceptual codec and achieves state-of-the-art FID metric, as we have excluded the difference

---

Published as a conference paper at ICLR 2024
of base codec, generative model and dataset. Furthermore, qualitative results in Fig. Fig. 4 and Fig. 12 I3 of Appendix B33 also show that our approach is visually more desirable. On the other hand. Fig. of Appendix B that the MSE of our approach is within twice of base codec, and Tab. shows that the PSNR our method is within 10 10lo 2 = 3.01 dB of base codec. This trade-off (Blau & Michaeli 2018). And as our approach shares bitstream with MSE codec, we can achieve perception-distoriion trade-off' (See Appendix, B.3 In addition, we also evaluate KID (Binkowski
means that proposed approach satisfies the MSE bound by eee-ddittortio--eeceetteee
et al.
2018 and LPIPS (Zhang et al. 2018 in Appendix, B.3
FFHQ
ImageNet
COCO
CLIC
Method
BD-FID 1 BD-PSNR BD-FID BD-PSNR BD-FID BD-PSNR BD-FID BD-PSNR
MSE Baselines
Hyper
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
ELIC
-9.740
1.736
-10.50
1.434
-8.070
1.535
-10.23
1.660
-4830
-0.8491
-8.830
-0.3562
-4.770
-0.3557
-4.460
-0.4860
VTM
-14.22
0.7495
-13.11
0.9018
-11.22
0,9724
-12.21
1.037
Conditional Generative Model-based
HiHC
-48.55
-2.036
-4.52
-1,418
-4.88
-1.276
-36.16
-1.621
HiFIC
-11885
-1.920
47.18
-1.121
Po-ELIC
-50.77
0.1599
-48.84
0.1202
-0.881
0.2040
-42.96
0.3305
CDC
-43.80
-8.014
-11.5
-6.416
-45.35
-6.512
-38.31
-7.043
ILLM
-50.58
-1.234
.48.22
-0.4802
-0.67
-0,5468
-4.955
-0.5956
ILLM
-52.32
-1.415
-47.99
-0.7513
Unconditional Generative Model-based
Proposed (Hyper) -54.14
-2.225
-52.12
-2.648 -1.492
-6677
-2.496 -1.370
-44.52
-2.920
Proposed (ELIC) -5.889
-0.9855
-55.18
-5.45
-46.52
-1.635
Table 2: Results on FFHQ, ImageNet, COCO and CLIC. re-trained on corresponding dataset. Bold: lowest FID. Underline: second lowest FID.
Diversity of Reconstruction Another feature of approach is the reconstruction diversity.
Though is theoretically beneficial to adopt stochastic decoder for conditional generative codec (Freirich et al. 2021), most of previous works (Mentzer al. 2020 He al. 2022b Muckley et al. (2023) adopt deterministic decoder and lost reconstruction diversity. On the other hand, our approach preserves this diversity. In Fig. 5 we show the pixel-level standard deviation a of our construction on the first image of Fig. . with sample size 16. Further, in Fig, 5 of Appendix [.3] we present alternative reconstructions, which differ a lot in detail but all have good visual quality.
Idempotence Consider we have MSE codec. The first time compression is (1) (x)()))) and re-compression (2) = go ( (X)))). We evaluate its idempotence by MSE between first time compression and re-compression || 12 According to our theoretical results, we can also use our approach improve idempotence of base codec. Specifically, after first time compression, we can decode perceptual reconstruction X by Eq. 10 and use this perceptual reconstruction for re-compression (2) go (x.())) By Theorem 1 should have ||X X ||2 i.e., this augmented codec is strictly idempotent. In practice, as shown Tab. 3 the augmented codec's re-compression MSE is smaller than the base codec. This indicates that our proposed approach can acts as an idempotence improvement module for MSE codec.
Re-compression metrics
Number of models
Method
Train
Test
MSE
PSNR (dB) t
HiFiC, ILLM MCG. DPS Proposed
K
K weeks
~0.1s
Hyper
6.321
40.42
50s
Hyper w/ Proposed
2.850
44.84
~60s
ELIC
11.80
37.60
ELIC w/ Proposed
7.367
40.93
Table 4: Complexity of different methods, K
refers to the number of bitrate supported, which 3 for HiFiC and for ILL.M.
Table 3: The idempotence comparison between base MSE codec and our approach.
8

---

Published as a conference paper at ICLR 2024
Complexity Compared with conditional generative codec (e.g., HiFiC (Mentzer et al. 2020) and ILLM (Muckley et al. 2.23)), our approach has lower training but higher testing complexity. Com- pared with inversion based image super-resolution (e.g. MCG (Chung et al. 2022b), DPS (Chung et al. 20 our approach's is complexity is similar. Tab, I that our approach greatly reduces training time. Specifically, for conditional generative codec, conditional generative model is quired for each rate. If the codec supports K rates, K generative models should be trained. For HiFiC and ILLM, each model takes approximately week. And the codec needs K weeks of train- ing. While for us, this extra training time is 0 as we can utilize pre-tra pre-trained unconditional generative model. While as the inversion-bas based image super-resolution, our method requires gradient ascent during testing, which increases the testing time from ~ 0.1s to 60s.
6 RELATED WORK
6.1 IDEMPOTENT IMAGE COMPRESSION
Idempotence is a consideration in practical image codec Joshi et al. 2000). In traditional codecs like JPEG (Wallace| 1991) and JPEG2000 (Taubman et al.] 2002), the encoding and decoding transforms can be easily made invertible, thus idempotence can be painlessly achieved in these codec. In NIC, however, requires much more efforts to ensure the idempotence (Kim et al. (020). This because neural network based transforms are widely adopted in NIC, and making these transforms invertible either hinders the RD performance (Helminger et al., (02I)) or complicates the coding process (Cai ct al. 2022). To the best of our knowledge, we are the first to build theoretical connection between idempotence and perceptual quality.
6.2 PERCEPTUAL IMAGE COMPRESSION
The majority of perceptual image codec adopt conditional generative models and successfully achieve near lossless perceptual quality with very low bitrate (Rippel & Bourdev 2017 Tschan- nen et al.] | 2018: Mentzer et al. 2020 Agustsson et al. 2022 Yang & Mandi] 2023] Muckley al,, 2023 Hoogeboom et al. (2023). Blau & Michaeli (2019) show that conditional generative codec achieves perfect perceptual quality with at most twice of optimal MSE. Yan et al. (2021) further prove that this approach is the optimal among deterministic encoders. The unconditional generative basel codec is explored by Ho et al. (2020); Theis et al. (0022). Though no actual codec implemented, they reveal the potential of unconditional generative model in image compression.
To the best of our knowledge, our approach is the first actual codec using unconditional generative
model. It does not require training new models and is equivalent conditional generative model- based perceptual codec.
7 DISCUSSION & CONCLUSION
A major limitation of our proposed approach is the testing time. This limitation is shared by all methods that use inversion of generative models (Menon et al. 2020: Daras et al.) 2021 Chung al. 2022a), and there are pioneering works trying accelerate it (Dinh al. (022)) Another limitation is that the resolution of the proposed approach is not as flexible We can use patches as workaround (Hoogeboom et al., 2023) but there can be consistency issue. This limitation is also
shared by all unconditional generative models, and there are also pioneering works trying to solve (Zhang al. 2022). As those two limitations are also important in broader generative modeling community, believe they will be solved soon the carly-stage methods grow mature.
To conclude, we reveal that idempodence and perceptual image compression are closely connected. We theoretically prove that conditional generative codec satisfies idempotence, and unconditional generative model with idempotence constrain is equivalent to conditional generative codec. Based on that, propose new paradigm of perceptual codec by inverting unconditional generative model with idempotence constraint. Our approach does not require training new models, and it outperforms previous state-of-the-are perceptual codec.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Improving the perceptual quality of NIC in low bitrate has positive social values, including reducing carbon emission by saving resources for image transmission and storage. However, there can also be negative impacts. For example, in FFHQ dataset, face with identity different from the original image can be reconstructed. And this mis-representation problem can bring issue in trustworthiness.
REPRODUCIBILITY STATEMENT
For theoretical results, the proof for all theorems are presented in Appendix A For experiment, all four datasets used publicly accessible. In Appendix [B.1] we provide additional implementation details including the testing scripts for baselines and how we tune provide source code for reproducing the experimental results as supplementary material.
Besides, we
ACKNOWLEDGMENTS
Funded by National Science and Technology Major Project (2022DD0115502) and Baidu Inc. through Apollo-AIR Joint Research Center.
REFERENCES
Eirikur Agustsson, David C. Minnen, George Toderici, and Fabian Mentzer. Multi-realism image compression with a conditional generator. ArXiv, abs/2212. 138224, 2022.
Jyrki Alakuijala, Ruud Van Asseldonk, Sami Boukortt, Martin Bruse, Iulia-Maria Comsa, Moritz Firsching, Thomas Fischbacher, Evgenii Kliuchnikov, Sebastian Gomez, Robert Obryk, et al. Jpeg xl next-generation image compression architecture and coding tools. In Applications of Digital Image Processing XLII, volume 11137, pp. 12-124. SPIE, 2019.
Johannes Ballé, Valero Laparra, and Ecro P Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Represen- tations. 2018.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: py- torch library and evaluation platform for end-to-end compression research. arXiv preprint arXh::2011.03029 2020.
Mikolaj Binkowski, Danica J. Sutherland, Michal Arbel, and Arthur Gretton. Demystifying mmd gans. ArXiv, abs/1801.01400 2018. URL
(o
Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. VCEG-M33, 2001. 2001.
Yochai Blau and Tomer Michaeli. The perception distortion perception-distortion tradeoff. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 0228-6237, 2018.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The vate-distortionnperreppii tradeoff. In International Conference Machine Learning, pp. 675-685. PMLR, 2019.
Benjamin Bross, Yu-Kui Wang, Yan Yc, Shan Liu, Jianle Chen, Gary Sullivan, and Ohm. Overview of the versatile video coding (vve) standard and its applications. IEEE Transac- fions Circuits and Systems for Video Technology, 31(10):3736-37 3764, 2021.
Shily Cai, Zhijun Zhang, Liqun Chen, Luxin Yan, Sheng Zhong, and Xu Zou. high-fidelity variable- rate image compression via invertible activation transformation. In Proceedings of the 30th ACM International Conference Multimedia, pp. 2021-2031, 2022.
Zhengxue Cheng, Heming Sun. Masaru Takeuchi. and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
10

---

Published as a conference paper at ICLR 2024
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint (tti:2200914688/ 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25696, 2022b.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros Dimakis. Intermediate layer optimization for inverse problems using deep generative models. arXiv preprint arXiv:2 2021.
Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. Score-guided intermediate layer optimization: Fast langevin mixing for inverse problem. arXiv preprint 204v 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11389-11399 1338, 2022.
Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the distortion-perceptinn tradeoff in wasserstein space. ArXiv, abs/21107005555 2021.
Noor Fathima Goose, Jens Petersen, Auke Wiggers, Tianlin Xu, and Guillaume Sauticre. Neural image compression with diffusion-based decoder. arXiv preprint arXiv:2301.05400 2023.
Dailan He, Ziming Yang. Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elie: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. arXiv preprint 2rXIv:2203.10886 2022a.
Dailan He, Ziming Yang, Hongjiu Yu, Tongda Xu, Jixiang Luo. Yuan Chen, Chenjian Gao, Xinjie Shi, Hongwei Qin, and Yan Wang. Po-elic: Perception-oriented efficient learned image coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1764-1769, 2022b.
Leonhard Helminger, Abdelaziz Djelouah, Markus Gross, and Christopher Schroers. Lossy im- age compression with normalizing flows. In Neural Compression: From Information Theory to snppiicttion-Worsshhon ICLR 2021, 2021.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Emiel Hoogeboom, Eirikur Agustsson, Fabian Mentzer, Luca Versari, George Toderici, and Lucas Theis. High-fidelity image compression with score-based generative models. arXiv preprint 1.Xi::3305.18231. 2023.
Shoma Iwai, Tomo Miyazaki, Yoshihiro Sugaya, and Shinichiro Omachi. Fidelity-controllable treme image compression with generative adversarial networks. 2020 25th International Confer- ence on Pattern Recognition (ICPR), pp. 8235-8242. 2020.
Shoma Twai, Tomo Miyazaki, Yoshihiro Sugaya, and Shinichiro Omachi. Fidelity treme image compression with generative adversarial networks. In 2020 25th International Con- ference Pattern Recognition (ICPR), pp.
IEEE, 2021.
Rajan L Joshi, Majid Rabbani, and Margaret A Lepley. Comparison of multiple compression cycle performance for jpeg and jpeg 2000. In Applications of Digital Image Processing XXIII, volume 41 4115, pp. 492-501. SPIE, 2000.
11

---

Published as a conference paper at ICLR 2024
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Hun-Hyuk Kim, Soobeom Jang, Jun- Ho Choi, and Jong- Lee. Instability of successive deep image compression. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 247-255, 2020.
Cheuk Ting Li and Abbas El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions on Information Theory, 64(1 1):6677-6778, 2018.
Yanghao Li, Tongda Xu, Yan Wang, Jingjing Liu, and Ya-Qin Zhang. Idempotent learned image compression with right-inverse. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer- cnn architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14388-14397 2023.
Yi Ma, Yongqi Zhai, Chunhui Yang. Jiayu Yang. Ruofan Wang, Jing Zhou, Kai Li, Ying Chen, and Ronggang Wang. Variable rate roi image compression optimized for visual quality. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1936-1940, 2021.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2434-2442, 2020. URL https://api semanticscholee ar. wwtp/oorgroooaeeeeee ID:
Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity gener- ative image compression. ArXiv, abs/2006.09965. 2020.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference Image Processing (ICIP), pp. 3339- 3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Matthew Muckley, Alaaeldin El-Nouby, Karen Ullrich, Hervé Jégou, and Jakob Verbeek. Im- proving statistical fidelity for neural image compression with implicit local likelihood models. 2023.
Guy Ohayon, Theo Joseph Adrai, Michael Elad, and Tomer Michaeli. Reasons for the superiority of stochastic estimators over deterministic ones: Robustness, consistency and perceptual quality. In International Conference on Machine Learning, pp. 26474-26494, PMLR, 2023.
Oren Rippel and Lubomir D. Bourdev. Real-time adaptive image compression. In International Conference on Machine Learning, 2017.
Jorma Rissanen and Glen G Langdon. Arithmetic coding. IBM of research and development, 1392)1149-162, 1979.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or Pivotal tuning for latent-based editing of real images. ACM Transactions on graphics (TOG), 42(1):1-13, 2022.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. ArXiv, abs/1806.000 806.00035, 2018.
12

---

Published as a conference paper at ICLR 2024
Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. ArXiv, abs/2012.047881,, 2020.
David S Taubman, Michael W Marcellin, and Majid Rabbani. Jpeg2000: Image compression fun- damentals, standards and practice. Journal of Electronic Imaging, 11(2):88--277 2002.
Lucas Theis, Tim Salimans, Matthew D. Hoffman, and Fabian Mentzer.
Lossy compres-
sion with gaussian diffusion. ArXiv. abs/2206.088889, 2022. URL h
resaniicscollar http:/ooooSSS110111555
George Toderici, Lucas Theis, Nick Johnston, Eirikur Agustsson, Fabian Mentzer, Johannes Ballé, Wenzhe Shi, and Radu Timofte. Clic 2020: Challenge on learned image compression, 2020, 2020.
James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent vari- ables using bits back coding. In International Conference Learning Representations, 2018.
Michael Tschannen, Eirikur Agustsson, and Mario Lucic. Deep generative models for distribution- preserving lossy compression. Advances in neural information processing systems, 31, 2018.
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34 (4):30-44, 1991.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.004900 2022.
Tongda Xu, Yan Wang, Dailan He, Chenjian Gao, Han Gao, Kunzan Liu, and Hongwei Qin. Multi- sample training for neural image compression. Advances in Neural Information Processing Sys- tems, 35:1502-1555 2022.
Tianfan Xue, Baian Chen, Jiajun Wu, D. Wei, and William T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, pp. 1-20, 2017. URL https:
7/api. www.eetiicccoiooooeooo.. www.arroorooooooooooo http://eeeaitcccoooooooooooooooo :tt.:aaa....r/Coooos::::11011110
Zeyu Yan, Fei Wen, Rendong Ying, Chao Ma, and Peilin Liu. On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework. ArXiv, abs/2106.02782,, 2021.
Zeyu Yan, Fei Wen, and Yei-Yunle Liu. Optimally controllable perceptual lossy compression. ArXiv, 1ass22066110822. 2022.
Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models. arXiv preprint NXXi:2209.006500 2023.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
Han Zhang, Ruili Feng, Zhantao Yang, Lianghua Huang, Yu Liu, Yifei Zhang Yujun Shen, Deli Zhao. Jingren Zhou, and Fan Cheng. Dimensionnliity-aayiin diffusion process. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14307-14316, 2022. URL
www.appisseaaaticcccccee itcoo arr org/ Cttp://opppupcceeeee htt.:/ooo.....oeeorrro.s.D.SS//. :
Richard Zhang, Phillip Isola, Alexci A. Efros, Eli Shechtman, and Oliver Wang. The unrea- perceptual metric. 2018 IEEE/CVF Conference
sonable effectiveness of deep features
on Computer Vision and Patter Recognition, pp. 586-595, 2018. URL https://api.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- 2016: 14th European Confer-
lation on the natural image manifold. In Computer ence, Amsterdam, The Netherlands. October 11-14, 2016, Proceedings, Part V 14, pp. 597-613. Springer, 2016.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF MAIN RESULTS
In previous sections we give intuition and proof sketch of the theoretical results. We provide formal proof in this appendix.
Notations We use the capital letter X to represent discrete random variable, lowercase letter x to represent specific realization of random variable, and calligraphic letter X' to represent the alphabet of a random variable. We use px to represent the probability law of random variable and px = x) to represent the probability of a specific realization of discrete random variable. We use Pr(-) to
represent the probability of specific event. We use - almost sure convergence.
to represent definition and R. to represent
Theorem 1. (Perceptual quality brings idempotence) Denote X source, f() encoder, Y = S(X) as bitstream, g(-) as decoder and X g(Y) as reconstruction. When encoder J() is deterministic, then conditional generative model-based image codec is also idempotent, i.e.
x=g(Y) PxlY = f(8) -
Proof. We start with specific value y E V where y is the alphabet of random variable Y. Without loss of generality. we assume py (Y = y) + O, i..., y lies within the support of py. We define the inverse image of set:
wl far x))()) =
(12)
where X' the alphabet of random variable X. According to the definition of idempotence, we need to show that X E [yy Note that as encoder f(.) deterministie, each x X only corresponds to one Again, we consider specific value x E X, (X = x) * O. We note that the likelihood of Y can be written as
{
1, f(x) =y
PY (Y y/X = x)
(13)
0, +y
Then, for all x & [y]] the joint distribution pxr = x, = y) =
= px x)yxxx =
u|X x) = O. And thus the posterior Pxlr = x, v) = O. In other words, for all samples X ~ (X y). the event Pr(A $ f (y)) = 0. And therefore, we conclude that
Pr f [v)) 1,
(14)
which indicates almost sure convergence f(X) a. Y.
Theorem 2. (Idempotence brings perceptual quality) Denote X as source, f(.) as encoder, Y = s(X) as bitstream, g(-) as decoder and X = g(Y) as reconstruction. When encoder J(.) deterministic, the unconditional generative model with idempotence constraint is equivalent to the conditional generative model-based image codec, i.e.,
X px. s.t. f(8)= Y PXlY.
Proof. Similar to the proof of Theorem we consider specific value of x E X' with px a) * 0. As
y= f(x)
(15)
is a deterministic transform, we have
1, f(x)=1)
Prix(Y = y/X = 2) -
(16)
0, f(x) 4y
Then by Bayesian rule, for each (Ir, v) X' V.
(x x/YY v) py (8 = y x)px(X = x)
(17)
x h(X 2,Y = y).
(18)
1 px (X =x) = =xx =)), f(x)=y
where 1(X x., Y Y)
(19)
0x xx (X X = = O.
f(x)
14

---

Published as a conference paper at ICLR 2024
We can treat 1(X = I, y) as un-normalized joint distribution. V(z, v) & x V, it has 0 mass where f(z) # y, and a mass proportional to px (X = z) where f(x) = y. As this holds true ((x,)) y). then we have
PxiY x (X,Y)
(20)
And sampling from this un-normalized distribution is equivalent to sampling from px with the constrain f(X) y. Therefore, sampling from posterior
X ~ (xx((Xl))
(21)
equivalent to the constrained sampling from marginal
X N~pxx px.
~x.x s.t. f(8)= =Y;
(22)
which completes the proof.
Corollary 1. If f()) is the encoder of a codec with optimal MSE A", then the unconditional gener- ative model with idempotence constraint also satisfies
Px = px, E|| X 24'
(23)
Furthermore, the codec induced by this approach is also optimal among deterministic encoders.
Proof. By Theorem 2, we have shown the equivalence of the unconditional generative model with idempotence constraint and conditional generative model. Then this corollary is simply applying Theorem of & Michaeli (2119) and Theorem 2, Theorem 3 of Yan et al. (202I) to conditional generative model.
In previous section, we state that for MSE optimal codec, the idempotence constraint f fXx Y is equivalent to g( (())) - gY)). Now we prove it in this appendix. To prove this, we only need to show that for MSE optimal codec, the decoder g()) is a invertible mapping. Blau & Michaeli (2019) already show that g(.) is deterministic mapping, i.c., Vy,. 3/2 E gYi(Y)) 7 g(Y2)= = 8/1 7 y2- Then, we only need to show that if the reconstruction is different, the bitstream is different, i.e., Vy,, U2 E V, y1 F y/2 = 9(Y)) & g(2)). Formally, we have:
Theorem 3. Denote X as source, f() as encoder, Y = f(X) as bitstream, g(.) as decoder and X g(Y) as reconstruction. When encoder f(.) is deterministic, for MSE optimal codec,
y1 + y2 g(v1) # 9(22).
(24)
Proof. We prove by contradiction. We assume that By * y2. g(v1) = g(22). Then we notice that we can always construct a new codec, with bitstream V1, y/2 merged into new one with higher probability py (Y = U1) + Py PY = B2). This means that this new codec has exactly the same reconstruction, while the bitrate is lower. This is in contradiction to the assumption that our codec is MSE optimal.
To provide a more intuitive illustration, we will include an example with discrete finite alphabet (our theoretical results applicable any countable alphabet or messu-lea source X and any deterministic measureable function f())):
Example 1. (I-dimension, discrete finite alphabet, optimal 1-bit codec): Consider discrete 1d ran- dom variable X, with alphabei X' = {0, 1, 2, 3, 5)5,, and Y with alphabet y = {0, We as- sume P(X a) 3. i.e., X follows uniform distribution. We consider a deterministic transform f(X): : X + y = round(X/3), Or to say, f(.) maps {0, 1.2] to {0), and {3, 4, 5) to {1}. This infact the MSE-oppimal 1-bit codec for source X (See Chapter 10 of [Elements of Information Theoryl). And the joint distribution P(X, Y), posterior P(X is just tabular in Tab. 5
We first examine Theorem. 7 which says that any sample from X. ~ P(X|Y y) satis-
fies f(X) y. Observing this table, this is indeed true. As for f(X) 4 D. the posterior
P(X= x
O.
We then examine Theorem. 2 which says that sampling from X ~ P(X) with f(X) y constraint
is the same as sampling from P = y). We first identify the set such that f(X) = y. When
y 0, this set is (0, 1, 1,. And when y = 1, this set is (3,4,5). And sampling from (()) with
15

---

Published as a conference paper at ICLR 2024
X
y
p(X
x.
y)
p(X
x/Y v)
((x)
1/6
1
0
1/6
2
0
1/6
3
0
4
0
0
0
5
0
0
0
0
2
0
1/6
4
1/6
5
1/6
Table 5: Tabular of probability in Example. 1.
constrain f(X) = y is equivalent to sampling from one of the two subset with probability x p(X). And this probability is exactly P(X =
Additionally, Note that we only limit f()) be deterministic and measureable. Thus, those theo- retical results can be extended other noise-free inversion-based image restoration, such as super- resolution. Despite inversion-based super-resolution has been studied for years empirically, their theoretical relationship with conditional model based super-resolution, and cestortion-perceplhe (Blau & Michaeli] 2018) trade-off is in general unknown. PULSE (Menon et al. (2020) are the pio- neer of this and they justify their approach by "natural image manifold" assumption. And later works inversion-based super-resolution follow theirs story. On the other hand, our Theorem, and Corollary. 3 can be extended into image super-resolution
Conditional generative super-resolution also satisfies idempotence, that is, the up-sampled image can down-sample into low-resolution image.
Inversion-basee super-resolution is theoretically equvalient to conditional generative super- resolution.
Inversion-basee super-resolution satisfies the theoretical results of distortion distortion-perception trade-off (Blau & Michaeli 2018), that is the MSE is at most double of best MSE.
We believe that our result provides non-trivial theoretical insights inversion- based super- resolution community. For example, Menon et al. (2020) claim that the advantage of inversion- based super-resolution over the conditional generative super-resolution is that inversion- based super- resolution "downscale correctly". However, with our theoretical result, we know that ideal condi- tional generative super-resolution also "downscale correctly". Currently they fail to achieve this due to implementing issue. Another example is that most inversion-based super-resolution (Menon eet al. 2020) (Daras et al. 202T) report no MSE comparison with sota MSE super-resolution, as is for sure that their MSE is worse and there seems to be relationship between their MSE and sota MSE. However, with our theoretical result, we know that their MSE should be smaller than 2x sota MSE. And they should examine whether their MSE falls below 2x sota MSE.
After the submission deadline, we become aware of an alternative Theorem. I and Theorem. which is presented in concurrent paper by Ohayon al. (2023).
B ADDITIONAL. EXPERIMENTS
B. ADDITIONAL EXPERIMENT SETUP
All the experiments are implemented Pytorch, and run a computer with AMD EPYC CPU and Nvidia A100 GPU.
For FID evaluation, we adopt the same code as official implementation of OASIS (Sushko et al. 2020 in http://hhhh...ooooooceaaaaee//// To ensure have enough num-
16

---

Published as a conference paper at ICLR 2024
ber of sample for FID, slice the images into 64 64 non-overlapping patches. All the BD metrics are computed over bpp 0.15 - 0.45, this is because HiFiC and CDC only have the bitrate over this range. And extrapolation beyond this range cause inaccurate estimation.
For the re-trained version of HiFiC (Mentzer et al. 2020). we adopt a widely used Pytorch implementation feey-ffidlitty-eeeeeee
https:/ htpp:///ithubb com/ Justin-Tar/ ion For the re-trained version of ILLM (Muckley et al., 2023), we adopt the official Pytorch implementation. We follow the original paper to train our models on FFHQ and ImageNet dataset.
We note that the official HiFiC (eetter al.) 2020) is implemented in TensorFlow in https: //github.com/teeee/ www./////otttoeeoecooeoee 1ow compress Pytorch implementation, and the results are Tab, terms of BD-FID, the Pytorch version outperforms Tensorflow version in FFHQ, ImageNet and CLIC dataset, while it is outperformed by version COCO. The overall conclusion impacted.
We test the official implementation and the
FFHQ
ImageNet
COCO
CLIC
Method
BD-FID y BD-PSNR BD-FID | BD-PSNR
BD-FID J BD-PSNR
BD-FID
BD-PSNR
HiFiC (Pytorch)
-48.55
-2.036
44.52
-1.418
44.88 -6.455
-1.276
16.116
-1.621
HiliCC (Tensorflow)
-46.44
-1.195
40.25
-0.828
-0.917
-35.90
-0.920
Table 6: Comparison Pytorch HiFiC and Tensorflow HiFiC.
the pre-trained model by CompressAl (Bégaint et al. 2020), which is trained on Vimeo (Xue et al. (017). For ELIC (He et al. 2022ap. we use the pre-trained model in https: //tthug..bbb..t//. cwm/ Cttp///ieeeeCCaaeeeeeaeeae
For Hyper (Ballé al., 2018),
ELiC-Re implemetation which is trained on ImageNet training split. To the best of our knowl- edge, their training dataset has no overlap with our test set.
For BGP, we use the latest version BPG 0.9.8 in https::/beelardd.oo we use the latest version VTM 22.0 in https: /ttpss:///qitt..o aregit..hi... hhi, fraunhofer de/jvet/ VVCSoftware, BttpssttwarreVMmeeeeeeeeeeeee wtt.:/eeeaaee//eeeeee
For VTM,
22.0 We convert the source to YUV444, encode with VTM and convert back to RGB to compute the metries. The detailed command line for BPG and VTM are as follows:
bpgenc -q (22,27 32,37,42,44 file -o filename.. bpg
bpgdec filename.. bpg -o file
./ EncoderApp -c encoder.intra. efg -i filename. yuv
-q (22 27 ......42.477 -o /dev/ null -b filename. bin
-Source -SSuurceHeight=256 --FrmmeRate-1 FramessToBeEncoteedll -- -nIputBittDepthhtt
-InpuchhromaFrmat-444 -Conformaneeindowwweoe
./ DecoderApp -b filename.. bin -o filename. yuv -d 8
For training of StyleGAN (Karras et al. 2019). we adopt the official implementation. The only difference is that our model does not have access to the test set of FFHQ. We directly use the pre- trained DDPM by (Chung et al. 2022a), which is trained without the test set.
For PULSE Meeon et al. 2020) inversion of StypleGAN, we follow the original paper to run spher- ical gradient descent with learning rate 0.4. run gradient ascent for 500 steps. For ILO (Daras) al. 2021) inversion of StyleGAN, we follow the original paper
spherical gradient descent on 4 different layers of StyleGAN with learning rate 0.4. We run gradient ascent for 200, 200, 100, 100 steps for each layer. For MCG (Chung al. 2022b) and DPS (Chung et al. 2022a). follow the original paper to run gradient descent for 1000 steps, with scale parameter ( increasing as bitrate
increases. We explain in detail about how to select Ç in next section.
17

---

Published as a conference paper at ICLR 2024
B.2 ADDITIONAL ABLATION STUDY
Why some inversion approaches fail In Tab. we show that some inversion approaches fail for image codec, even with our theoretical guarantee. This because our theory relies on the assumption that the generative model perfectly learns the natural image distribution, while this is hard to achieve practice. And we think this is where the gap lies. More specifically, GAN has better "precision" and worse "recall", while diffusion has a fair "precision" and fair "recall"" The "precision" and "recall" defined by [Sajaadi et al. (2018). Or to say, the distribution learned by GAN largely lies in the natural image distribution better, but many area of natural image distribution is not covered. The distribution learned by diffusion does not lies in natural image distribution as good as GAN, but it covers more area of natural image distribution. This phenomena is also observed and discussed by Dhariwal & Nichol (202T) and Ho Salimans (2022). The "precision" is more important for sampling. but "recall" is more important for inversion. That is why the GAN based approach fails. The MCG fails probably because it is not well suited for non-linear problems.
Selection The Ç parameter in DPS (Chung al. 2022a is like the learning rate parameter in PULSE (Menon et al. 2020) and ILO (Daras et al.) 2021). It controls the strength of idempotence constraint. When is too small, the perceptual reconstruction will deviate from the original image, and the MSE will go beyond the MSE upperbound. When ( is too large, artefacts will dominate the image, and again, the MSE will go beyond the MSE upperbound.
700
600
200
500
150
400
MSE
MSE between compress and ELIC
100
200
of proposed approach
50
100
MSE of MSE optimized codec
MSE upperbound
zeta
zeta
Figure 6: Ablation study MSE < with ImageNet dataset and ELIC. Left: MSE between source and reconstruction. Right: MSE between ELIC and re-compression.
As shown in Fig. 6 using C too small or too large will cause the reconstruction MSE higher than the theoretical upperbound. While using a proper Ç achieves a MSE satisfies the MSE upperbound. On the other hand, the MSE of ELIC and re-compressed reconstruction indeed goes lower goes up. This is because this term is exactly what ( is penalizing. Despite a large Ç strengthen the idempotence constraint, it can also push reconstruction off the learned natural image manifold. Another example is shown in Fig. g It is shown that using insufficiently large can sometimes lead weird reconstruction.
To further understand this phenomena, we visualize the perceptual reconstruction and the re- compression results of those perceptual reconstruction. As shown in Fig. 7 when < 0.15, 0.3, the re-compression image does not look like the ELIC reconstruction. And this indicates that the idempotence constraint is not strong enough. While when Ç 0.6, 1.2, 2.4, the re-compression image looks the same as ELIC reconstruction. However, when Ç = 1.2, 2.4, the perceptual recon- struction dominated by noise. And therefore its MSE still exceeds the MSE upperbound.
In practical implementation, we search from 0.3 and increase it when it is not large enough. For five bitrate of Hyper based model, we select < = {0.3, 0.6, 1.2, 1.6, 1.6] on FFHQ, [0.3, 0.6, 0.6, 1.2, 1.2] For five bitrate of ELIC based model, we select < = (0.3, 0.6, 1.2, 1.6} FFHQ f0.3, 0.6, 1.6) ImageNet, {0.6, 0.6, 0.6, 1.2, 1.2) COCO and (0.45, 0.6, 0.6, 1.2, 1.6} on CLIC. Sometimes the required for different images are different. Therefore, also introduce very simple adaptive mechanics. More specifically,
ImageNet, {0.6, 0.6, 1.2)
COCO and
(0.45, 0,94 0.2,
18

---

Published as a conference paper at ICLR 2024
Source (ImageNett
ELIC
Recompeession zeta=0.15
Recompesssion 2eta-0.3
Recompression zeto=0.6
Recompression zeia-1.2
Recompression zela=2,4
Percegtual 2ets
Pereeptual 2etam0.33
Perceptual zeta-0.6
Perceptual zetar11.2 1.2
Perceptual zeta-2.4
Figure Ablation study on adjusting Ç with ImageNet dataset and ELIC.
MSF Propsced
Figure An example of using improper Ç with ImageNet dataset and ELIC.
evaluate the MSE between re-coppreseed perceptual reconstruction and MSE reconstruction. If larger than 32, we multiple ( by 1.5 until K time. We set K = {1, 1, 2, 4,4 for methods and datasets. We note that it is possible to exhaustively search Ç for better perceptual quality. However, it will drastically slow down the testing.
B.3 ADDITIONAL RESULTS
Other Metrics Besides the FID, PSNR and MSE, we also test our codec using other metrics such as Kernel Inception Distance (KID) (Binkowski et al.) 2018) and LPIPS (Zhang et al. 2018). Similar FID and PSNR, report BD) metries:
The result of KID has same trend as FID, which means that our approach sota. While many other approaches outperform approach in LPIPS. This result is expected, as:
KID a divergence based metric. And our approach is optimized for divergence.
LPIPS is image-to-image distortion. By iistortion-perceptinn trade-off (Blau & Michaeli 2018), perception optimized approaches can not achieve SOTA LPIPS.
All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
19

---

Published as a conference paper at ICLR 2024
FFHQ
ImageNet
COCO
CLIC
Method
BD-HogKID BD-LPIPS BD-logKID BD-LPIPS BDD-lggKID BD-LPIPS BD-logKID BD-LPIPS
MSF. Haxelines
Hyper
0.000
ELIC
-0.232
-0.040
-0.348
-0.058
-0.236
-0.062
-0.406
-0.059
BPG
0.1506
-0.010
0.027
-0.010
0.126
-0.012
0.039
-0.008
-0.232
-0.031
-0.298
-0,216
-2.049
Conditional Generative Model-based
HiTiC
-3.132
-0,108
-2.274
-0.172
-2.049
-0.172
-1.925
-0.148
HiFIC*
-4266
-0,110
-2.880
-0.173
Po-ELIC
-3,504
-0.104
-2.877
-0.167
-2,671
-0.188
-2.609
-0.145
CDC
-2.072
-0.060
-1.968
-0.099
-1.788
-0.101
-2.122
-0.084
ILLM
-3,418
-0.109
-2.681
-0,181
-2.220
-0.180
-2.882
-0.155
ILLM
-4.256
-0.106
-2,673
-0.178
Unconditional Generative Madel-based
Proposed -5y117) -5.10
-0.086
-4.271
-0.058
-4.519 -5.360
-0.083
-3.787 -4.046
-0.056
Proposed (ELIC) -5.471
-0.099
-5.694
-0.106
-0.113
-0.079
Table 7: Results on FFHQ, ImageNet, COCO and CLIC. *: re-trained corresponding dataset.
Bold: lowest KID. Underline: second lowest KID.
FFHQ
ImageNet
COCO
CLIC
Hyper
0.0000
0.0000
0.0000
0.0000
ELIC
0.0098
0.0148
0.0146
0.0176
HiFiC
0.0030
0.0064
0.0071
0.0058
ILLM
0.0001
0.0041
0.0058
0.0042
Proposed (Hyper)
-0.0065
-0.0399
-0.0262
-0.0236
Proposed (ELIC)
-0.0029
-0.0203
-0.0135
-0.0132
Table 8: MS-SSIM on FFHQ, ImageNet, COCO and CLIC.
ILLM (Muckley et al. 2023) also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by autoencoder trained with LPIPS.
In terms of MS-SSIM, ELIC HiFiC > ILLM > Hyper > Proposed (ELIC) > Proposed (Hyper). We are reluctant to use MS-SSIM as perceptual metric, as this result obviously not aligned with visual quality, and should not be used when considering divergence based perceptual quality as (Bau Michaeli 2018), because:
MS-SSIM an image image distortion. By Blau Michaeli (2018), is odd with divergence based metrics such as FID, KID. Theoretically, there does not exist codec that achieve optimal FID and MS-SSIM at the same time.
In terms of MS-SSIM, ELIC, a mse optimzied codec, is the sota. This indicates that MS- SSIM correlates poorly with human perceptual quality.
Similar result is also reported by Muckley et al. (2023) Fig. 3, where the MS-SSIM of ILLM is not even as good as Hyper, which is a mse optimized codec. This indicates that RS-S correlates poorly with human perceptual quality.
In perceptual codec competition CVPR CLIC 2021, the human perceptual quality is tested as final result and MS- MS-SSIM, FID are evaluated. The 1st place of human perceptual test result among 23 teams "MIATL NG", also has lowest FID, which indicates FID correlates well with human perceptual quality. On the other hand, its MS-SSIM ranks 2 place among 23 teams, which indicates MS-SSIN correlates poorly with human perceptual qual- ity.
Rate Distortion Curve We do not have enough room in the main text to show rate-distortion curve. We present them in Fig. [1 of appendix instead.
Perception Distortion Trade-off couple of perception image codec have achieved perception-
distortion trade-off with the same bitstream (Twai al. 2020 Agustsson et al. 2022 Goose al.
20

---

Published as a conference paper at ICLR 2024
ImageNet
ImageNet
ImageNet
Hyper [Balle 2018
27
34
[Bellard 2014]
VTM [Brass 20211
[Mentzer
23
HiFCC [Mentzer 2020]
RID
PSNRT
30
2022]
CDC ('ang 2023]
100
28
[Muckley
illm [Muckley 2023]
26
Proposed (Hyper)
ELCC [He 2022]
Proposed (ELIC)
interpolated
0.6
1.0
0.4
0.6
12
bpp
bpp
bpp
Propused (EUI) on imagelle, Jelasel
103-0.1050
25
to3 001--0.2266
to3-0
tpp-0
tp2-1.100
Figure 9: Perception distortion trade-off by convex interpolation.
2023 As our proposed approach shares the same bitstream as the MSE codec, are already ca- pable of decoding the optimal perceptual quality and optimal MSE at the same time. To achieve the intermediate points of perception-diisortion trade-off, we can utilize the convex interpolation method proposed by Yan et al. (2022). More specifically, Yan et al. (2022) prove that when vergence is measured in Wasserstein-2 -2 (W2) distance, convex combination of perceptual and MSE reconstruction optimal in distortion distortion-perception trade-off sense. In our case, even if the divergence not W2, convex combination is able achieve reasonable result. More specifically, denote the MSE reconstruction as XA, and perceptual reconstruction as X,, The convex combination is
8 =ax, (1 a) 1.0).
(25)
We evaluate this approach on ImageNet dataset with ELIC as base codec, as this setting covers the widest range of perception and distortion. In Fig. 9 can that interpolated codec successfully cover a wide range of perception-distoriag perception codec has FID lower than ILLM (Muckley al. 2023) but MSE higher than ILLM. However, the interpolation with a = 0.8 still has FID lower than ILLM. But its MSE is already on par with ILL.M. And the interpolation with a 0.6 has a FID comparable ILLM, but its MSE is obviously lower. This indicates that the interpolated version of our codec remains competitive in perception-diistorion trade-off.
trade-off, More specifically, our optimal
Visual Comparison of Different Bitrate We visualize our reconstruction from low to high bitrate, along with ELIC as our bitrate the same. interesting to find that both our approach and ELIC converge to the source image bitrate grows high, but from different directions. For us, the visual quality does not differ much as bitrate goes high, while the reconstruction becomes more aligned with the source. For ELIC, the reconstruction grows from blurry sharp bitrate goes high.
Visual Comparison with Other Methods We do not have enough room in the main text to compare all approaches visually. In main text Fig. 4 only compare approach to HiFiC (Mentzer al. 2020) and ILLM (Muckley et al. 2023). Thus, we put the visual results of other methods in Fig. 12
15 here in Appendix.
21

---

Published as a conference paper at ICLR 2024
Source (IemggNNet)
Converge
ELIC
Convenge
Propesed
bpp 0.11
bpp 0.17
bpp 0.27
bpp 0.66
Figure 10: The visual result of ELIC and our proposed approach as bitrate goes high gradually.
Reconstruction Diversity In main text Fig. 5 used to compute standard deviation. Fig. 16 we present additional reconstructions show that our reconstruction has good diversity.
only present three alternative reconstructions that
22

---

Published as a conference paper at ICLR 2024
FFHQ
FFHQ
FFQ
27
34
RIJ
09NRT
-22
100
27
26
0.2
0.1
0.0
0.2
0.4
0.5
0.2
04
3.0
tpp
bpp
hpp
ImageNet
Imagelvet
ImageNet
35
100 FI3
PSNRT
22-
COCO
COCO
COCO
100
39
22
350
100 FDJ
200
21
150
0.6
0.6
2.0
100
bpp
Ips
CLIC
CLIC
CLIC
36
Hyper [Balic 2018)
EUC (He 2022|
250
30
BRG [Bellard 2011]
2021]
PSNKT
20
HRR [Mentper 2800]
HiFic [Mentzer 2020]
[He 2022]
CDC DYang 2023]
100
26
[Muc 2023)
[Muckley 2023]
Proposes (Hyper)
Proposed LeC)
0.6
0.8
02
0.5
1.C
0.2
0.4
0.6
1.0
Figure 11: The rate-distortion curve of different methods.
23

---

Published as a conference paper at ICLR 2024
Source (FTHQ)
Hyper 0.12 bpp
ELIC 0.07 bpp
CDC 0.25 hpp
HiFiC 0.15 bpp
ILLM 0.10 bpp
Po-ELIC 0.10 bpp
HiriC 0.14 hpp
ILLM 0.09 bpp
Proposed (Hype) 0.12 bpp
Proposed (ELIC) 0.07 bpp
Figure 12: A visual comparison of our proposed approach with other approaches.
24

---

Published as a conference paper at ICLR 2024
Souree (ImageNet)
Hyper 0.17 bpp
ELIC 0.10 hpp
CDC 0.25 hpp
IIFIC 0.17 bpp
ILLM 0.11 bpp
Po-EI IC 0.15 hpp
HiFiC 0.15 hpp
ILLM 0.11 hpp
Proposed (Hype) 0.17 bpp
Proposed (ELIC) 0.10 bpp
Figure 13: A visual comparison of our proposed approach with other approaches.
25

---

Published as a conference paper at ICLR 2024
Source (COCO)
Hyper 0.17 bpp
ELIC 0.11 bpp
CDC 0.25 bpp
HiFiC 0.17 hpp
I.L. hpp
Po-ELIC 0.15 bpp
Proposed (Hype) 0.17 bpp
Proposed (ELIC) 0.1 bpp
Figure 14: visual comparison of proposed approach with other approaches.
26

---

Published as a conference paper at ICLR 2024
Source (CI IC)
Hyper 0.16 hpp
ELIC O. 10 hpp
CDC 0.25 bpp
HiFiC 0.17 bpp
ILLM 0.11 bpp
Po-ELIC 0.14 hpp
Proposed (Iyype) 0.16 bpp
Proposed (ELIC) 0.10 bpp
Figure 15: visual comparison of our proposed approach with other approaches.
27

---

Published as a conference paper at ICLR 2024
Figure 16: Reconstruction diversity of proposed approach.
28


Reviews:
[{"review_id": "E1PWAjXi26", "reviewer": "Reviewer_SHZQ", "summary": "This paper studies the relation between the idempotence and perceptual image compression and find that the two are highly correlated. Specifically, the authors theoretically prove that: (1). conditional generative model-based perceptual codec satisfies idempotence; (2). Unconditional generative model with idempotence constraint is equivalent to conditional generative codec. \nThey also propose a new paragidm of perceptual image codec by inverting unconditional generative model with idempotence constraints, based on the above findings. The proposed method outperforms state-of-the-art methods in terms of perceptual quality metric, such as Frechet Inception Distance (FID).", "questions": "- In the experiments, a MSE optimized codec is used as the base model. Have the authors tried other codecs (perceptual optimized or mixed)?\n- Could the authors provide more results on other metrics, such as KID, LPIPS?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  This paper is the first to study the relation between the idempotence and perceptual image compression methods.\n-  The findings are very interesting and motivating, the high correlation between the idempotence and perceptual image compression may benefit both above compression areas.\n-  The theoretical analysis is rigorous and solid.\n-  The proposed new compression method achieves better perceptual quality than the competing methods.", "weaknesses": "- I admit that the findings of high correlation between  the idempotence and perceptual image compression are intersting and meaningful. However, the  subsequent idea of \"perceptual image compression by inversion\" does not bring new insights or knowledge. Apply generative model inversion on the low-level tasks such as super-resolution has been well studied for several years. The proposed method seems to just change the task from super-resolution to image compression. Applying a proven methodology to a similar task is, in my opinion, hardly meets the ICLR's bar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3PSROVXaYt", "reviewer": "Reviewer_CP1y", "summary": "This paper reveals the relationship between idempotence and perceptual image compression. There are two important theorems derived in this paper: (1) Perceptual quality (i.e., conditional generative model) brings idempotence. (2) Idempotence brings perceptual quality, with unconditional generative model. A specific idea in this paper is a new paradigm to achieve perceptual image compression by applying generative model inversion with pretrained models, which is conceptually simple but useful in practice. Experimental results also demonstrate the effectiveness of this idea  (Table 1 and Figure 3).", "questions": "See the abovementioned weaknesses.    \nIn addition, the approach proposed in this paper can utilize pre-trained unconditional generative model for compression at different bitrates. In Figure 9, it seems the advantage of the proposed method is more significant at 0.17bpp, compared with original ELIC. However, the result at 0.11 bpp seems to be weird, which is definitely what we do not want in practice. Is there is potential solution or idea to solve this issue?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The relationship between idempotence and perceptual image compression revealed in this paper is surprising and insightful. The conclusion would be unintuitive before I successfully follow all the derivations in this paper. In fact, the derivations in this paper are not difficult. Theorems 1&2 well summarize the main conclusions of this paper. If there is no technical errors regarding the derivations (at least I didn't find), the proposed idea in this paper, i.e., using generative model inversion to improve perceptual image compression (Section4.1), may form a new paradigm in this field, which is sound in theory and useful in practice. The pratical implementation can be summarized as: first use pretrained generative models to produce an image $\\hat{X}$ and encode it into $f_0(\\hat{X})$, then constrain the idempotence between $f_0(\\hat{X})$ and Y using gradient descent-based inversion. These steps are easy to implement with a pretrained codec and a pretrained unconditional generative model.\n\nExperimental results clearly demonstrate the effectiveness of this idea. Also, the presentation of this paper is great. Overall, I would be happy to accept this paper although I have two major concerns as written in the weaknesses part.", "weaknesses": "Despite the above strengths of this paper, there are two main issues especially after I ran the code provided as supplementary material.  \n(1) The time complexity of generative model inversion should be considered in practice (Table 4), which I personally believe there is space to improve it in the future. For example, now the authors are using DDPM with 1000 steps. There are some latest unconditional generative models largely reducing the number of generation steps.   \n(2) The issue regarding the image resolution should be paid with more attention. We can imagine that the gradient descent-based inversion would be hard to be implemented on high-resolution images. In the experimental section of this paper, it is stated like \"central crop image by their short edge and rescale them to 256 × 256\", which means almost all experiments are performed on this scale. On the one hand, some baseline models are trained for high-resolution generation or generative compression, and it may be unfair to directly compare with their pretrained models. On the other hand, it implies that the current method is hard to be applied to high-resolution images without cropping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "My8DxHNI0t", "reviewer": "Reviewer_VEis", "summary": "The paper proposes a theoretical justification for consideration of idempotence in the task of generative image compression. This includes a couple of theoretical results: the first is that a perceptual codec is idempotent, and the second is that an MSE-optimal codec paired with an unconditional generative model is optimal in terms of rate-distortion-perception theory. The paper includes a suite of empirical studies to justify the theory, showing that the inclusion of an idempotence constraint in the sampling process of a generative model (on top of a pretrianed MSE codec) gives better rate-distortion-perception performance than the previous methods considered when using a DDPM+DPS diffusion model sampler.", "questions": "1. The failures of many of the generative models to give good results suggests a divergence between theory in practice, i.e., although the theory suggests any unconditional generator will do, the generators that we have available to us and their various samplers exhibit widely different properties. Did the authors consider discussing this in the Discussion?\n2. Could you clarify why you used the OASIS FID implementation as opposed to the FID/256 method that is more standard in the compression community?\n3. The rate-distortion-perception tradeoff is portrayed in a confusing way in Figure 8 of the Appendix. Is the reason that you could not match bpps between methods as done in Figure 1 of (Muckley, 2023)?\n4. Would you consider to use language other than \"perception\" to describe distributional divergence? \"Perception\" is an overloaded term in the compression community and does not intuitively describe the phenomena of the paper. Although it is true that \"perception\" is the term of Blau/Michaeli to describe divergence phenomena, others in the community have adopted more specific teams such as \"realism\" or \"statistical fidelity\" that lead to less confusion with human perception.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper makes contributions on a number of areas to the field of generative image compression.\n\n1. It presents a theoretical justification for why generative codecs should be idempotent.\n2. It presents a theoretical justification for why an MSE-optimal codec paired with an unconditional model. This is an approach considered by other works in the field such as (Hoogeboom, 2023) and (Ghouse, 2023).\n3. Empirical results further justify the insights of theory, as the presented method is optimal among all generative codecs considered.\n4. Empirical results include ablations over the base MSE autoencoder, which allows fair comparisons to both the ILLM and HiFiC methods of previous work (which use older autoencoders).\n5. Qualitative results of the images are compelling.", "weaknesses": "1. The primary weakness is a lack of comparison to other diffusion-based methods. The only diffusion method considered is CDC, which may be relatively underpowered vs. DIRAC (Ghouse, 2023). CDC is almost certainly underpowered vs. the work of (Hoogeboom, 2023). Also, there are images released from the work of (Agustsson, 2022), which are not included in the paper.\n2. The x-domain constraint seems identical to that of (Hoogeboom, 2023).\n3. The paper uses a third-party implementation of HiFiC with a different training set that may give divergent results from the original paper. The authors of the HiFiC paper have made their models available at https://github.com/tensorflow/compression.\n4. Only two metrics are considered: FID and PSNR. Most other compression works include more metrics, such as LPIPS, to gauge model performance across a variety of axes.\n\nThe proposed method uses much more compute than its competitors, and it has limited image resolutions. However, I am not counting this is as a negative for my review, because this is clearly stated already in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TpDiZhoB6h", "reviewer": "Reviewer_zF7s", "summary": "This text introduces a new concept of idempotence within image codec stability, revealing its unexpected relationship with perceptual image compression. By leveraging this understanding, the proposed method utilizes idempotence constraints to invert unconditional generative models, presenting an equivalent and improved paradigm for perceptual image codecs.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper presents a new paradigm of perceptual image codec, which could bring new insight to the community. The approach doesn't necessitate new model training but rather utilizes pre-trained mean-square-error codecs and unconditional generative models.", "weaknesses": "However, I also have some concerns as follows:\n\n1) The relationship between idempotence and image compression is hard to understand in the current version.\n\n2) Authors should provide more evidence to support the points that Idempotence brings perceptual quality in Section 3.\n\n3) I can't see the superiority of the proposed method in the visual comparison with HiFiC and ILLM, especially in Fig 1.\n\n4) In BD-FID, the proposed method is better. But in BD-PSNR, others may be better. Could authors provide more evaluation metrics, such as MSSSIM and VMAF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)