Synonymous Variational Inference for Perceptual Image Compression
Authors: Zijian Liang
Kai Niu
Changshuo Wang
Jin Xu
Ping Zhang
Venue: ICML 2025 poster (2025)
Keywords: Synonymous
Variational Inference
Perceptual
Image Compression
Semantic Information Theory
Primary area: probabilistic_methods->variational_inference
TL;DR: This paper attempts to establish an unified theoretical framework for perceptual image compression in a synonymous relationship-based semantic information theory viewpoint.

Abstract: Recent contributions of semantic information theory reveal the set-element relationship between semantic and syntactic information, represented as synonymous relationships. In this paper, we propose a synonymous variational inference (SVI) method based on this synonymity viewpoint to re-analyze the perceptual image compression problem. It takes perceptual similarity as a typical synonymous criterion to build an ideal synonymous set (Synset), and approximate the posterior of its latent synonymous representation with a parametric density by minimizing a partial semantic KL divergence. This analysis theoretically proves that the optimization direction of perception image compression follows a triple tradeoff that can cover the existing rate-distortion-perception schemes. Additionally, we introduce synonymous image compression (SIC), a new image compression scheme that corresponds to the analytical process of SVI, and implement a progressive SIC codec to fully leverage the model's capabilities. Experimental results demonstrate comparable rate-distortion-perception performance using a single progressive SIC codec, thus verifying the effectiveness of our proposed analysis method.

Synonymous
Variational Inference for Perceptual Image Compression
Zijian Liang'
Kai Niu
Changshuo Wang' Jin Xu' Ping Zhang 3
Abstract
have pursued this goal with the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index (MS- SSIM) quality metrics through handcrafted algorithms in- cluding transform coding, quantization, and entropy coding. With the growing research in artificial intelligence, recent advancements in learned image compression (LIC) (Ballé et al., 2018; Minnen et al., 2018; Cheng et al., 2020; He et al., 2021: 2022a; Li et al.. 2024) combine the optimization principles of traditional image coding with the capabilities of deep learning. Its basic optimization idea can be the- oretically analyzed using a variational inference method similar to that in the variational auto-encoder (Kingma & Welling, 2013), leading to a loss function in the form of a rate-distortion tradeoff (Ballé et al., 2017; 2018). These methods have demonstrated significant rate-distortion per- formance compared with conventional methods.
Recent contributions of semantic information the- ory reveal the set-element relationship between semantic and syntactic information, represented as synonymous relationships. In this paper, we propose a synonymous variational inference (SVI) method based on this synonymity viewpoint to re- analyze the perceptual image compression prob- lem. It takes perceptual similarity as a typical syn- onymous criterion to build an ideal synonymous set (Synset), and approximate the posterior of its latent synonymous representation with a paramet- ric density by minimizing a partial semantic KL divergence. This analysis theoretically proves that the optimization direction of perception im- age compression follows triple tradeoff that can cover the existing schemes. Additionally, we introduce synonymous image compression (SIC), a new image compres- sion scheme that corresponds to the analytical process of SVI, and implement progressive SIC codec fully leverage the model's capabili- ties. Experimental results demonstrate compara- ble piiee-istortion-prreeee
As the inconsistency exists between "low distortion" and "high perceptual quality", Blau and Michaeli explored the tradeoff between distortion and perception (2018). They fur- ther incorporated the perceptual constraint into compression limit analysis and explored the rate-distortion distortion--erceptionn tradeoff (Blau & Michaeli, 2019). Further theoretical analy- sis (Theis & Agustsson, 2021; Theis & Wagner, 2021; Yan et al., 2021; Qian et al., 2022; Theis, 2024; Hamdi et al., 2025) and empirical results (Agustsson et al., 2019; Mentzer et al., 2020; He et al., 2022b; Theis et al., 2022; Agustsson et al., 2023; Muckley et al., 2023; Hoogeboom et al., 2023; Xu et al., 2023; Careil et al., 2024) demonstrate the effec- tiveness of this new optimization direction and suggest that perceptual image compression (PIC) with high perceptual quality at low bitrates can be achieved with generative com- pression (Santurkar et al., 2018) empowered by generative adversarial networks (GAN) (Goodfellow et al., 2014) and di diffusion models (Ho et al., 2020).
performance using single progressive SIC codec, thus verifying the effectiveness of our proposed analysis method.
1. Introduction
Image compression is typical topic for lossy source cod- ing, aiming to achieve the optimal tradeoff between recon- structed image quality and coding rate. Following the rate- distortion optimization instructed by Shannon's 's classic in- formation theory (1948; 1959), traditional image compres- sion like JPEG (Wallace, 1991) and BPG (Bellard, 2015)
The great success of the dite-distottion- -ptrtioop-eeeettionn trade-off has effectively shifted the focus from symbol-level accuracy in traditional image compression and tends more towards semantic information accuracy, which aligns with Shannon and Weaver's 's discussion on communication problem levels (Weaver, 1953). However, these works adopt diverse em- pirical optimization approaches for perceptual optimization, such as Kullback-Leiin Leibler (KL) divergence, discrininator- based adversarial loss like Wasserstein divergence (Blau & Michaeli, 2019), or mixed with "perceptual" measure like
Key Laboratory of Universal Wireless Communications, Min- istry of Education, Beijing University of Posts and Telecommuni- cations, Beijing, China *Png Cheng Laboratory, Shenzhen, China State Key Laboratory of Networking and Switching Technol- ogy, Beijing University of Posts and Telccommunication,, Beijing, China. Correspondence to: Kai Niu <niukai buthuiiu@uu.............
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Synonymous Variational Inference for Perceptual Image Compression
LPIPS (Zhang al., 2018; Mentzer al., 2020; Muckley et al., 2023) and DISTS (Ding al., 2020). The diversity and inconsistency of these methods motivate to establish a unified perspective for perceptual image compression with a well-established mathematical theoretical framework.
multiple sampling the detailed representations indepen- dently from this latent synset.
3. We implement progressive SIC codec to validate the theoretical analysis result, fully leveraging the model's s capabilities. Experimental results demonstrate compa- rable rate-distortion pereeettionnneo performance using a single neural progressive SIC image codec, thus veri- fying our method's effectiveness.
In this paper, we model the perceptual image compression problem mathematically based on semantic information theory viewpoint. Recent advancements in semantic infor- mation theory (Niu& & Zhang, 2024) highlight a set-element relationship between semantic information (i.e., the mean- ing) and syntactic information (i.e., data samples), where one meaning can be expressed in diverse syntactic forms. Building on this synonymity perspective, manipulating a set of samples with the same meaning (referred as to synony- mous set, abbreviated as "Synset") should be considered as the principle of semantic information processing. This viewpoint has the potential to surpass the theoretical limits of classical information theory while relaxing symbol-level accuracy (i.e., distortion) requirements. We emphasize that although the concept of synonymity originates from text data, is universal to various types of natural data. For ex- ample, in image data, perceptual similarity between different images can be seen as typical synonymous relationship.
2. Background
2.1. Rate-Distortion Theory and Variational Inference
As one of the fundamental theorems in Shannon's classical information theory, rate-distortion theory (Shannon, 1948; Shannon et al., 1959) aims address the lossy compression problem. It provides a theoretical lower bound of the com- pression rate Rt (D) with given distortion D, which can be characterized as rate- deestorttoionn function (Thomas & Joy, 2006)
R(D) min I (x: p(2(z)
(1)
s.t. Eg [d (x,, S D,
in which (x:x) represents mutual information between the source X and the reconstructed X, numerically equal to the average coding rate for compressing X with a given lossy codec; D) can be any reference distortion measure satisfying the condition that d (z. i) 0 if and only if x á. typified by the mean squared error (MSE).
On this basis, we re-analyze the optimization goal of percep- tual image compression and introduce novel variational inference method to analyze its optimization direction, aim- ing to guide the design of an image compression scheme. The contributions of our paper are as follows:
1. We propose Synonymous Variational Inference (SVI), a novel variational inference method to analyze the optimization direction of perceptual image compres- sion. By building an ideal synset based on typical criterion of perceptual similarity, it approximates the posterior of the corresponding latent synonymous rep- resentation with a parametric density by minimizing partial semantic KL divergence. This method theoreti- cally proves that the optimization direction of percep- tual image compression is an expected rate-distortion- perception tradeoff form that covers the existing rate- distortionnerrceptionn schemes. To the best knowledge of the authors, our method is the first work that can theoretically explain the fundamental reason for the divergence measure's existence in existing percep- tual image compression schemes.
To achieve this, learned image compression achieves op- timal rate-distortion performance through end-to-end op- timization training- While the ultimate optimization tar- get remains the rate-distortion tradeoff, aligning with the continuous changes in neural network model training, the optimization process is achieved through variational infer- ence for the generative model, specifically the variational auto-encoders (Kingma & Welling, 2013; Ballé et al., 2017; 2018). The core idea of variational inference is to build a parametric latent density ( (<) and minimize the KL diver- gence, a standard measure in classical information theory, to approximate the true posterior Pyj ze (V((zz), i.e.,
E Ex-p(a:) DKL [allptte] E E Egnu
log (ū)T)) log Pals (al - log Pū (7)
log
(2)
weighted distontion
rale
2. We establish Synonymous Image Compression (SIC), new image compression scheme that corresponds to the analytical process of SVI. By solely encoding the latent synonymous representation partially, SIC inter- prets this information as an equivalent quantized latent synset. It reconstructs multiple images satisfying the synonymous relationship with the original image by
const.
As the first term equals O under the assumption of a uniform density on the unit interval centered on y, and the last term is a constant, the optimization simplifies to the sum of a weighted distortion and a coding rate, thereby achieving the optimal rate-distortion tradeoff.
2

---

Synonymous Variational Inference for Perceptual Image Compression
2.2. The Rate-Distortion-Pereettti tradeoff'
portant foundation, semantic variable U corresponds to various possible synsets Ui {u, i E Mi where each sample u; is a possible value of the syntactic variable U and shares the same semantic meaning with all the pos- sible values (uy j E NN, indexed in Mr,- On this basis, the semantic entropy of U' is defined by
Since Blau and Michaeli demonstrated the apparent trade- off between perceptual quality and distortion measure that widely exists in various distortion measures (2018), they ex- tended the classic rate-distortion tradeoff to a triple tradeoff version (2019). Specifically, they define the perceptual qual- ity index dp (Px Pa) based on some divergence between distributions of the source and reconstructed images, and build new lower bound of compression rate (D, P) with considerations of the perception index, i.e.,
Σ p(u)) plui) (5) iENL
He (i)= (
Σ p (u: iEA,
in which the probability of the synset p (4,,) is defined as the sum of the probabilities of all the samples p (u:) within it. This directly leads to the inequality between the semantic entropy and the classical Shannon entropy, i.e., H. (i) S (U). being apparently valid, since the uncertainty of syntactic samples is no longer the focus.
R P) = min I (x;
(3)
s.t.
Eu a) (x, x:] S SD
dp Pp) p P.
Building on this triple tradeoff relationship, the percep- tual image compression methods (Agustsson et al., 2019; Mentzer et al., 2020; He et al., 2022b; Agustsson et al., 2023; Muckley et al., 2023) typically optimize the model using the following loss function form:
As the foundation of the synonymous variational inference proposed in this paper, new form of KL divergence needs to be introduced from (Niu & Zhang, 2024), referred to as partial semantic KL. divergence DRL., [ullps]. which is defined as
LRDP =>,- . T ((x Xx
+ da F.e pe (dr, at]
Σ a (u:) log g (v:) VEN
DEL. -
(6)
+ Ap dp (px,Pa)
p p(4.)
(4)
which represents a divergence between syntactic distribu- tion q and semantic distribution p.. 2 Clearly, these two distributions emphasize different levels of information, i.e., the syntactic level and the semantic level. However, examin- ing the distance between these distributions holds significant physical meaning in perceptual image compression, which will be detailed in Section 3.2.
2.3. Semantic Information Theory
As the optimization towards perceptual quality is more in- clined to the accuracy of conveying meaning (the semantic problem) instead of the symbol-level accuracy that classi- cal information theory focuses on (the technical problem) (Weaver, 1953), this paper considers analyzing the prob- lem of perceptual image compression based on semantic information theory.
3. Synonymous Variational Inference: A Semantic Information Viewpoint
Research on semantic information theory has been ongo- ing since the 1950s, with various viewpoints such as log- ical probability (Carnap & Bar-Hillel, 1952; Bar-Hillel & Carnap, 1953; Barwise & Perry, 1981; Floridi, 2004; Bao al., 2011) and fuzzy probability (De Luca & Termini, 1972; 1974; AI-Sharhan et al., 2001) employed discuss the essence and measures of semantic information, but no consensus has been reached over time. Furthermore, these viewpoints provide limited theoretical guidance for the prac- tical coding of natural information sources.
3.1. Overview
Consider an image codec, in which the encoder captures the semantic information of the image, while the decoder reconstructs an image with the same semantics as the origi- nal instead of directly restoring the original image's pixels. Obviously, in natural image data, there typically mul- tiple images that share the same semantic information as the original image. For example, all images that exhibit certain perceptual similarities to the original image can be considered to convey the same meaning "to some extent"
However, a recent contribution to semantic information the- ory (Niu& & Zhang, 2024) presents a potential turning point in the field, which suggests understanding the semantic in- formation from synonymity perspective. In this theory, semantic information is processed based on fundamen- tal principle: considering a set of syntactic samples with the same meaning (referred to as a synset). Corresponding semantic information measures are also provided. As an im-
'Refer to as Ū in Niu and Zhang's 's paper (2024). The ring hat symbol "*" is appplied distinguish it from the tilde hat symbol "** commonly used variational inference.
'The relationship between the partial semantic KL diver- gence and the standard KI. divergence satisfies DRL, (gl p.] S DEL [ql lr]. which be referred to partial semantic relative entropy in Niu and Zhang's 's paper (2024).
3

---

Synonymous Variational Inference for Perceptual Image Compression
X
X
X
X
X
X
Data Space
y
V
y
Space
ú
Paiie v X)
a (
Po (u.1 12)
q (311))
Pe (v. (2) (
Ps. x (3.12)
(
Latent Distributions
Towards Convergence
Original Image Sample
Quant Representation
"Noisy" Representation
y
"Noisy" Synonymoux Representation
X'
Ideal Synset
Reconstructed Synset
Latent Synset
Quantized Latent Synset
Figure 1. An illustration of the optimization directions of synonymous image compression. By continuously minimizing the partial semantic KI. divergence DEL.s [allp,,ıx] latent space, the reconstructed synset & gradually approaches the ideal synset X until complete overlap occurs. At that point, every sample è X is a "synonym" of the original image sample x.
while retaining distinct detailed information. Therefore, when placing these images in synset, there must be la- tent representation and the corresponding coding sequence capable of capturing the shared characteristics among all samples this set, which should be learned by the semantic encoder. Using this representation, the semantic decoder can sample any image from the synset of the original image, which satisfies certain perceptual similarity with the source image if the synonymous judge criterion is the perceptual similarity criterion.
terior by minimizing the partial semantic KL divergence between these two distributions, i.c..
min Ex DKL.. [ [pg...
Once this distribution divergence is minimized, a generative model Pælu.. (xl0.. 9.) (i.e., the semantic decoder, in which D, a sampling for details) can be finally optimized. On this basis, the reconstructed synset á produced by the semantic decoder can be considered ideal synset X, which ensures exhibiting certain perceptual similarities to the original image æ.
sample of the
When considering using a deep neural network to design the above codec, its continuous optimization problem can be analyzed using similar idea to the variational autoen- coder, based variational inference (Kingma & Welling, 2013). Figure illustrates the optimization direction of this problem and provides a schematic representation of the achievable effects within the data space. Ideally, the source image æ can be considered as a sample in the idcal synset X, shown as the dashed circles in Figure Correspondingly, there must be an ideal posterior Pg. x (V., (x) in the latent space to represent the latent synonymous representations y, that capture the shared characteristics of all the samples (z./<;; E X}. However, in practice, the ideal synset X' is unavailable, so we rely on the original image to construct a parametric latent density q (ÿ) a:) to approximate this pos-
3.2. Synonymous Variational Inference
Unlike the usual variational inference, the two distribu- tions of (7) work at different levels: The parametric density (ÿ) ar) works at the syntactic level, while the true posterior Pv. ()) operates at the semantic level, represented in the form of synsets. However, since ÿ can be decomposed into combination concatenation of synonymous repre- sentation D, and detailed representation V. it is possible effectively process the minimization of the partial semantic KL divergence. To distinguish from the existing variational inference methods, we give the following definition.
Definition 3.1. Synonymous Variational Inference (SVI) generic variational inference method that approximates the true posterior of the synonymous representations with parametric density at the syntactic level by minimizing the partial semantic KL divergence (7).
'In practice, people may have varying judgments about whether two images have the same meaning. due their varying judge criteria. Thus, samples in synset based on a given synonymous criterion must share some specific semantic information but do not necessarily have completely identical meanings.
By applying the proposed synonymous variational inference,

---

Synonymous Variational Inference for Perceptual Image Compression
Observed Data
V.
O,
Random Variables
Multi-sumples Required to Construct Synsets
(t.))
Y (
M
9
B.u
8,
Parametric Prior gb (Factorized Hyperprior)
"Noisy" Latent Synset y
Essential Dependence
x'= an.)
inference model (analysis transform)
generative model (synthethis transform)
Non-essential Dependence
X (a))
Ideal Synset
Figure 2. Left: Representation of the proposed encoder as synonymous variational inference model, and corresponding decoder generative Bayesian model. The latent representation y is a merge of the synonymous representation D, and the detailed representation V. achieved through some form of merging splicing. fully factorized (Ballé al., 2017) or a hyperprior like (Ballé et al., 2018; Minnen et al., 2018) entropy model can be employed in the "Parametric Prior" item. An autoregressive (Minnen et al., 2018) or parallel (Il al., 2021) context model can be utilized 0-. These two types methods can be used accurate probability estimations v, predictions for 9.- Right: Illustrations for the equivalent relationship of the "noisy" latent synset y and the ideal synset X.
the optimization direction of perceptual image compression can be determined. To facilitate the subsequent derivation, first state the following lemma (the detailed proof can be found in Appendix A.1):
Proof. As stated in Figure 2, the model of synonymous image compression can be considered as a generalized vari- ational auto-encoder. By using the proposed SVI, i.e., mini- mizing the partial semantic KI. divergence given in (7),
Lemma 3.2.
When the source considers the existence of an ideal synset X and the decoder places the reconstructed sample in a reconstructed synset X, the minimization of the expected negative log synonymous likelihood term
Em) DKL, [gllp,, 1x
log logg( log Px1T. (XlU.) log Py. (8.)
(10)
const.
min Eapp(()) Bign [- [-logPx16. (X()a)
The first term equals O under the assumption of a uniform density on the unit interval centered on y. and the last term is a constant for a determined æ and corresponding ideal synset X. For the third term, with a determined inference and generative model, the coding rate of the synonymous representation Exep((a) B.g.ng [ log Pū. (ū.)] is equal to (x:X). as stated in Appendix A.2.
min Aa Em-p(n) En E ú. [d (xx ā;)] (8)
+ +Xp Bing
E a.
DKL. [Pa |Pa.))
in which da and Xo are the tradeoff factors for the expected distortion (typically expected means-squeen means-squared error, i.e., E- MSE) term and the expected KL divergence (E-KLD) term, respectively.
By Lemma 3.2, the minimization of the second term is equivalent to minimizing weighted expected distortion E Eg [d (x, æ;)] plus a weighted E-KLD term E DKL [p |Pa]- These weights can be considered as Lagrange multipliers the rate term, which makes the optimization goal equivalent to minimizing
Based on this, we propose the following theorem:
Theorem 3.3. For image source together with its bounded expected distortion Bamplee)Beeed &.EE [d (xx 2;)] and expected KL diver- gence E Dxa. [p llpa;) the minimum achievable rate of perceptual image compression is
(xx)
X with quantized bounded expected distortion and E-KLD constraints to obtain the optimal p( ()). shown (9). This target corresponds to Synonymous Rate- Eistortion-Perceplii Tradeoff. which can be shown
I
(xX)
min 1 (x:Xx x())
s.l.
Exppla) E2
EX
(d 2,) S D,
(9)
Lx = Ay (Empp(a) [- logg (..)
E P DRI. (pen [p=.] < P.
+ >d E B E [d (x. 2:)
(11)
where I X: (x:X) = -H ( -H, (x(x) with semantic
+ Ap E DxL IP pa.l.
variable X corresponds to the reconstructed synset X.
thus we finish the proof of the theorem.
5

---

Synonymous Variational Inference for Perceptual Image Compression
Paramerric Prior
Parametric Prior
Shifting ng: 1-11, m.
Original Image Sumple
Original Image Sumple
9a
Q
S..
go
Q
Ideal Synset X
Ideal Synset
Lateat Synset
Latent Synsel
Sampling
Sampling
2
5 St
Reconstructed Synset &
Reconstmaced Synset x A
9.
M/
9.
(
(2) (
Shifting t- m,
(4)
Figure 3. Processing frameworks of SIC. (a): The general framework. (b): The progressive framework.
Due to space limitations, only a brief outline of the proof is provided above. Please refer to Appendix A.2 for detailed proof. Additionally, we state that the existing rate-distortion- perception tradeoff (4) a special case of (11) when there only one sample è in the reconstructed synset X. More comprehensive discussions are provided in Appendix A.3.
ture y into L synonymous levels, treating the first levels as the synonymous representation )²"), while the subsequent levels are considered the detailed representation 인ز! By varying | from 1 to L and optimizing through the loss func- tion of the corresponding level in the training process, the SIC model can be optimized for approaching different ideal synsets x After training, synonymous representations at each level can be encoded progressively and fed to the decoder, making SIC a progressive image codec that can produce images at diverse synonymous levels (correspond- ing to varying coding rates) using a single generator. In our upcoming experiments, we implement progressive SIC model to fully leverage the model's 's capabilities, where the L levels are equally slicing the channels C into L groups, shown as Figure in Appendix C.I.
4. Synonymous Image Compression
According to Figure 2, the main difference between our proposed processing scheme and the general LIC method is that the generator only needs part of the accurate latent fea- tures, while the other part can be obtained through random sampling instead of accurate coding. Thus, the generator can obtain all the information required to reconstruct the image. To this end, we name this new image compression scheme Synonymous Image Compression (SIC).
The designing of the training loss for the progressive SIC model should take several practical issues into account. Firstly, while the loss function requires traversing samples in the reconstructed synset X, this is not practical during the actual training process. Therefore, the real training loss function utilizes small number of reconstructed samples and computes the arithmetic mean as a practical approxima- tion. Secondly, the training should also take into account the synonymous rate-distortion ratedistortionn-erreptoo trade-offs at each level, meaning that the trade-off's between levels are also re- quired to be balanced. To this end, we design group of loss functions for the progressive SIC model that alternatively trains for the level = 1,2, step by step, i.c..
A general framework of SIC is given in Figure 3(a). The SIC codec requires an analysis transform Jo (z: Φy) and a synthesis transform 9x (U.. D:.j: 0y) to achieve the bidi- rectional nonlinear mapping between the data space and the latent space. Only the synonymous representation ú, is required be coded, and this representation is equivalent to quantized latent synset Y' contains all the samples with different detailed representations. With obtaining V.. the generative model can generate multiple images X' = = by sampling diverse detailed representations ($c.) in V.
A uniform noise with unit interval should be used for y, to achieve efficient continuous optimization, and an en- tropy model assisted by an arbitrary form of parametric prior should be employed to estimate the coding rate for y. whereas both these two are not essential for y, and 9.- Although this aligns with the analytical process of synony- mous variational inference, it results in the SIC model's capabilities being underutilized, as the details still have the potential to provide additional information.
( aLl +(1 a) x + c 1,2, L, (12)
in which + is represented by
(l)) = Em-p(a)
-X llog p0.0 00 (8!))
x
1 M
(1 MSE ((() + 1 LPIPS (x, a! a(())
)
(13)
The progressive framework of SIC is proposed to solve this problem, as shown Figure 3(b). It partition the latent fea-
where the MSE is utilized as the distortion item, and the L.PPS (Zhang et al., 2018) directly replace by the KI. diver-
6

---

Synonymous Variational Inference for Perceptual Image Compression
BPG
VTM
HIFI
MS MS-ILLM
MS-ILLM (No-GAN)
Progressive SIC (Ouns, M-1)
CI CLIC2200
DIV2K validation
Kodak
0.25
0.20
0.20
DISTS ())
DISTS
0.15
DISTS
0.15
0.2-
0.10
0.1
0.05
0.05
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Bits Per Pixel (BPP)
Bits Per Pixel (BPP)
Bits Per Pixel (BPP)
Figure 4. Comparisons of methods using DISTS different datasets. Each point the HiFiC and MS-ILLM performance curves from single model, while our entire performance curves are achieved by a single progressive SIC model.
gence term since accurately calculating the KL divergence for image datasets is challenging; a is set between the level and the L level (equivalent to the conventional rate- distortion-perceptioon tradeoff) to indirectly achieve multi- level tradeoffs; _ is additional constraints in training, which detailed elaborated in Appendix C.1. Moreover, M denotes the number of reconstructed samples in training. Subsequent experimental results will show that with M = the proposed method can achieve comparable rate-distortion- perception performance, while bigger M will perform ad- vantages in certain synonymous levels.
our models with the test set of CLIC2020 (Toderici et al., 2020), the validation set of DIV2K (Agustsson & Timofte, 2017), and the Kodak dataset 4 Refer to Appendix C. for the hyperparameter settings and implementation details.
Comparison schemes: The comparison schemes for tradi- tional image compression use BPG (Bellard, 2015) and the state-of-the-an VTM to serve as the benchmarks for dis- tortion measures. The PIC schemes for comparison include HiFiC (Mentzer et al., 2020) and MS-ILLM (Muckley et al., 2023), including the No-GAN fine-tuning version of MS- ILL.M. These two schemes use adversarial loss to optimize perceptual quality, serving as benchmarks for perceptual measures. As the pre-training model of MS-ILLM, the No-GAN scheme directly uses LPIPS, the same perception measure our choice, which is the focus of comparison.
5. Experimental Illustration
In this section, we examine the effectiveness of our proposed analytical theory through experimental results.
Evaluation Metrics: PSNR is utilized for distortion mea- suring, while LPIPS and DISTS (Ding et al., 2020) are for perception measuring. It should be noted that the DISTS measure, due to its resampling tolerance, aligns more closely with the human understanding of perceptual himilarity-typifet synonymous relationships than LPIPS, thus our focus in our following analysis.
5.1. Implementation Setup and Comparison Schemes
Model architecture: The analysis transform and the synthesis transform 9s are implemented using the Swin Transformer (Liu et al., 2021), while the coding rate is es- timated using a joint autoregressive and hierarchical prior architecture (Minnen et al., 2018) based deep convolu- tional neural networks, with structural adjustments made for our level partitioning mechanism. We set the number of latent representation channels to C = 512 and the number of the equally partitioned synonymous levels to L = 16, giving each synonymous level a channel dimension of 32. This allows single progressive SIC codec to support 16 coding rates and their corresponding image quality levels.
5.2. Performance Analysis
We first examine our progressive SIC model's capabilities in the adaptation of full-rate perception qualities. Figure 4 compares the performance of our method and the compar- ison schemes under DISTS, in which our model is trained with sampling y..j only once (i.e., M = 1), as this follows the common usage in existing generative model training. As shown in this figure, our scheme can achieve percep- tual quality adaptability across various rates using a single model, with the perceptual quality of the reconstructed im-
Model Training: We randomly select 100,000 images from the OpenImages V6 dataset (Kuznetsova et al., 2020) as the training data, resizing them to a uniform resolution of 256x256 using random crop and resized crop for the training process. We train our model for x 10 iterations with a batch size of 16, a learning rate of 1 x 10-4, and the AdamW
"Kodak PhotoCD dataset, URL http://rOkuu//rrahhes/oooo us/graphisskkodak/ VVVCSoftware VTM, URL https://vcgit.hht hhi hhi.fraunhofrr.dd/ jvet/VVCSoftware, ViM,git, Version 23.4.
optimizer with weight decay of 5 x 10-5. We evaluate
7

---

Synonymous Variational Inference for Perceptual Image Compression
Progressive SIC (Ours, M-1)
-
Progressive SIC (Ours, M-5)
CLIC2020
DIV2K validation
Kodak
0.09
0.10
0.14
0.12
0.08
0.08
DISTS
DISTS
DISTS
0.10
8.07
0.07-
0.08
0.06
0.06
0.05
0.06
0.100
0.15
0.20
0.25
0.30
0..00
0.15
0.20
0.25
0.30
0.35
0.40
0.04
0.1
0.2
0.3
0.4
0.5
Bits Per Pixel (BPP)
Bits Per Pixel (BPP)
Bits Per Pixel (BPP)
Figure 5. Comparisons of our progressive SIC schemes with different sampling numbers in reconstructed X' on different datasets.
age improving as the coding rate increases.
divergence term in the loss function. Based on related PIC work, the GAN-bases adversarial loss is suggested replacement for the KL divergence term to further improve model performance.
For the conceredd DISTS measure, our method surpasses the No-GAN MS-ILLM solution (also trained with LPIPS) in a large coding rate range. This performance is demon- strated under conditions where the PSNR quality contin- uously approaches and even exceeds the comparison No- GAN schemes, and the LPIPS quality remains very similar, thus verifying a comparable rate-distortion -prrricioottioo--erepiioonnn per- formance, shown Figure 11 in Appendix C.2. This sur- passing reflects the advantage of incorporating the concept of synset in our proposed SVI. However, this advantage is modest compared to HiFiC and MS-ILLM guided by the GAN's s adversarial loss, since our loss directly uses LPIPS to replace the KI. divergence. Therefore, utilizing discrim- inative mechanisms in SIC models with the synonymous viewpoint (i.e., replacing the E-KLD term) may further en- hance perceptual quality in future considerations.
To verify the impact of adversarial loss on performance, we build a discriminator based on HiFiC's structure and fine- tune the synthesis transform gs of our M = and M = models with non-saturating loss (Goodfellow et al., 2014) for 2 10 steps as supplementary experiments. Specif- ically, we use a single conditional discriminator (refer to Appendix D.1) to obtain the diseriminative loss for recon- structed images at all synonymous levels, with the syn- onymous representation أ of each synonymous level / introduced as separate condition. The non-saturating loss can be expressed as
[-log (D (a ()) )
E
(14)
G
In addition, Figure 5 compares the DISTS qualities for re- construction sampling numbers M = 1 and M = 5 un- der identical hyperparameter settings. With M = 5, the expected distortion and perception loss are estimated via arithmetic mean, aligning more closely with theoretical op- timization directions. Results indicate that expanding the reconstructed synset X by increasing Ve, samples during training offers slight performance advantages across various datasets in general. These advantages are especially evident in the low and medium rate range, while the intersections at relatively high rates are due to insufficient fine-tuning of the hyperparameter settings, which needs more precise exploration in future works.
C0 EE a
[- -log (1-
a
(15)
+ Empx((a))
llgg
D
x 9
and thus the perception term in Equation (13) is required to be adjusted from solely the LPIPS term the sum of the LPIPS term and the generative loss in Equation (14), ie..
LPIPS (x. a(() = LPIPS (x. (e" + c (16)
Figure 6 shows the fine-tuned performance using DISTS as the evaluation measure, while other measures, including PSNR, LPIPS, and FID (Heusel et al., 2017), and visualiza- tion results are given in Appendix D.2. These results show that:
For further additional analysis and visualization results, please refer to Appendix C.2.
6. Limitations
The perceptual quality (DISTS, FID) has improved. While LPIPS performance remains nearly unchanged, DISTS and FID show greater gains at higher bitrates, with DISTS gradually approaching that of MS-ILLMM
According to the above experimental results and analysis, the limitation of the implemented progressive SIC model mainly lies in using the LPIPS measure to replace the
8

---

Synonymous Variational Inference for Perceptual Image Compression
HIFC
MS-ILLM (with GAN)
MS- ASSI (No-GAN)
Progressive SIC (No-GIAN, M-1)
Progressive SIC GAN, M=1)
Progressive (No-GAN,
Progressive (With GAN, M-3)
CLIC2020
validation
Kodak
0.20
0.25
0.20
0.20
0.15
DISIS()
0.15
DISTS
DISTS
0.15
0.10
0.10-
0.10
0.05
0.05
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.5
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Bits Per Pixel (BPP)
Bits Per Piael (BPP)
Bils Por Pixel (BPP)
Figure 6. Comparisons of methods using DISTS different datasets (supplemented fine-tuned model performance). Each point the HiFiC and MS-ILLM performance curves is from a single model, while our entire performance curves are achieved by a single progressive SIC model.
(with GAN). This means that the fine-tuning loss with the adversarial loss is closer to the ideal optimization direction, which can make better perceptual qualities of the reconstructed images. Besides, while DISTS and FID values change little at low rates (bpp < 0.10), the visual qualities improve significantly.
7. Relevant Thoughts on Semantic Information Theory
As an extension to synonymity- semantic informa- tion theory, our work provide guidance for practical image coding designs. In Appendix B, we further discuss the rela- tionships with existing conclusions in semantic information theory, including the semantic entropy and the down seman- tic mutual information proposed in (Niu & Zhang, 2024).
The improvement of DISTS is relatively obvious, while the enhancement in FID remains limited. This is a noteworthy phenomenon since it is absent in the non-sampling schemes, as confirmed by the experi- mental results in the MS-ILLM paper (Muckley al., 2023). Unlike DISTS' resampling tolerance, FID fo- cuses on the consistency of the distribution between the original and reconstructed image groups. This suggests that our implementation is still insufficient in optimiz- ing the distribution of reconstructed images, especially in our detailed sampling mechanism.
8. Conclusions
In this paper, we consider the perceptual image compres- sion problem from the perspective of synonymity-band baased semantic information theory. Specifically, we propose a syn- onymous variational inference (SVI) method to re-analyze the optimization direction of perceptual image compression. Based on this analysis method, we theoretically prove that the optimization direction of perceptual image compres- sion triple tradeoff, i.e., synonymous rate-distortion- perception tradeoff, which is compatible with the existing eee-distortion-perrpppii tradeoff empirically presented by previous works. Additionally, we propose new perceptual image compression scheme, namely synonymous image compression, corresponding to the SVI analytical process, and implement a rough progressive SIC model to fully lever- age the model's capabilities. Experimental results demon- strate full-rate rate-distortion perception performance and notable advantages on DISTS, thereby verifying the effec- tiveness of our proposed analysis method.
The gap compared to HiFiC and MS-ILLM is still obvious, especially on FID. In addition to the reason of the detailed sampling mechanism, this issue may also be due to insufficient fine-tuning with multiple synonymous layers battling against each other, or in- efficient perceptual loss is chosen. Besides, to address the limited improvement at low bit rates (bpp < 0. 10), another solution is to find a better mechanism than equal channel slicing for synonymous level partition- ing.
The distortion (PSNR) has degraded, which can ver- ify the cestortion-perception tradeoff' as mentioned in (Blau& & Michaeli, 2018).
Software and Data
Based on the above analyses, the current SIC framework still has limitations but shows potential for further exploration and offers belief directions for future research on perceptual image compression.
We will upload code for reproducing our results to the repos- itory at https://github, com/ ZJLiiagg / SypoonyyoousImmmaeeCoppeeee
9

---

Synonymous Variational Inference for Perceptual Image Compression
Acknowledgements
Ballé, J., Chou, P. A., Minnen, D., Singh, S., Johnston, N., Agustsson, E., Hwang, S. J., and Toderici, G. Nonlinear transform coding. IEEE Jourmal of Selected Topics in Signal Processing, 15(2):339-3) 2020.
This work was supported by the National Natural Science Foundation of China (No. 62293481, No. 62471054).
Impact Statement
Bao, J., Basu, P., Dean, M., Partridge, C., Swami, A., Le- land, W., and Hendler, J. A. Towards a theory of semantic communication, In 2011 IEEE Network Science Work- shop. pp. 110-117. IEEE, 2011.
Our work makes a significant contribution semantic infor- mation processing. A key challenge in semantic information theory is the lack of a unified viewpoint, as many perspec- tives fail to effectively guide coding design. Guided by the synonymity of semantic information, this paper demon- strates the fundamental theoretical reason why the distribu- tion distance, used to measure perceptual quality in percep- tual image compression empirically. appears in the optimiza- tion objective. Moreover, our theoretical analysis provides consistent and universal guidance for designing image com- pression approaches, regardless whether compression is considered for perceptual quality. To summarize, our contri- bution links semantic information theory practical image coding problems, which paves the way for more unified viewpoint of semantic information theory.
Bar-Hillel, Y. and Carnap, R. Semantic information. The British Journal for the Philosophy of Science, 4(14):147- 157, 1953.
Journal of Philosophy, 78(1 78(11):668-69) 1981. Barwise, J. and Perry, J. Situations and attitudes. The
Bellard, F. BPG image format, 2015. URL https:// bell aalllad.ooom org/bpg
Blau, Y. and Michaeli, T. The perception distortion tradeoff. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Blau, Y. and Michaeli, T. Rethinking lossy compression:
References
The rate- dineooisttotiion--o perceetion tradeoff. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 675-685, PMLR, 2019.
Agustsson, E. and Timofte, R. Ntire 2017 challenge single image super-resoien resoutics: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, July 2017.
Careil, M., Muckley, M. J., Verbeek, J., and Lathuilière, S. Towards image compression with perfect realism at ultra- low bitrates. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., and Gool, L. V. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Carnap, R. and Bar-Hillel, Y. An outline of theory of semantic information. 1952.
Agustsson, E., Minnen, D., Toderici, G., and Mentzer, F. Multi-realism image compression with conditional gen- erator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22324-22333, June 2023.
Cheng. Z., Sun, H., Takeuchi, M., and Katto, J. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Al-Sharhan, S., Karray, F., Gueaieb, W., and Basir, O. Fuzzy entropy: brief survey. In 10th IEEE international confer- ence on fuzzy systems. No. 01CH37297), volume 3, IEEE, 2001.
De Luca, A. and Termini, S. definition of a nonprobabilis- tic entropy in the setting of fuzzy sets theory. Information and Control, 20(4):301-311 1972.
pp.
De Luca, A. and Termini, S. Entropy of L-fuzzy sets. Infor- mation and Control, 24(1):55-73, 1974.
Ballé, J., Laparra, V., and Simoncelli, E. P. End-to-end opti- mized image compression, In International Conference on Learning Reprexentations (ICLR), 2017.
Ding, K., Ma, K., Wang, S., and Simoncelli, E. Image quality assessment: Unifying structure and texture simi- larity. IEEE transactions pattern analysis and machine intelligence, 44(5):2567-2555 2020.
Ballé, J., Minnen, D., Singh, S., Hwang, S. J.. and Johnston, N. Variational image compression with a scale hyperprior. In International Conference on Learning Representations (ICLR), 2018
Floridi, L. Outline of a theory of strongly semantic informa- tion. Minds and machines, 14:197-221, 2004.
10

---

Synonymous Variational Inference for Perceptual Image Compression
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural In- formation Processing Systems (NIPS). Curran Associates, Inc., 2014.
Li, C., Luo, J., Dai, W., Li, C., Zou, J., and Xiong, H. Spatial-channel context-based entropy modeling for end- to-end optimized image compression. In 2020 IEEE International Conference on Visual Communications and Image Processing VCCI), pp. 222-225, 2020.
Hamdi, Y., Wagner, A. B., and Gunduz, D. The rate- eettoriio--prcepplee trade-off with algorithmic realism, 2025.
Li, H., Li. S., Dai, W., Li, C.. Zou, J.. and Xiong, H. Frequency-aware transformer for learned image pression. In The Twelfth International Conference Learning Representations' (ICLR), 2024.
He, D., Zheng, Y., Sun, B., Wang. Y., and Qin, H. Checker- board context model for efficient learned image compres- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14771-14780, June 2021.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022, October 2021.
He, D., Yang, Z.,, Peng, W., Ma, R., Qin, H., and Wang. Y. ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5718-5727, June 2022a.
Mentzer, F., Toderici, G. D., Tschannen, M., and Agusts- son, E. High-fidelity generative image compression. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.. and Lin, H. (eds.), Advances in Neural Information Pro- cessing Systems (NeurIPS), pp. 11913-11922 Curran Associates, Inc., 2020.
He, D., Yang, Z., Yu, H., Xu, T., Luo, J., Chen, Y., Gao, Shi, X., Qin, H., and Wang, Y. PO-ELIC: Perception- oriented efficient learned image coding. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1764-1769, June 2022b.
Minnen, D., Ballé, J., and Toderici, G. D. Joint autoregres- sive and hierarchical priors for learned image compres- sion. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems (NeurlPS). Cur- ran Associates, Inc., 2018.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by two time-scale update rule converge to local nash equilibrium. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems (NIPS). Curran Associates, Inc., 2017.
Muckley, M. J., El-Nouby, A.. Ullrich, K., Jegou, H., and Verbeek, J. Improving statistical fidelity for neural image compression with implicit local likelihood models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Inter- national Conference on Machine Learning (ICML), pp. 25426-25443 PMLR, 2023.
Ho, Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances Neural In- formation Processing Systems (NeurIPS), pp. 6840-6851. Curran Associates, Inc., 2020.
Niu, K. and Zhang, P. A mathematical theory of semantic communication. Journal Communications, 45(6): 7-59, 2024. URL https: / /ww. joconline. com. en wtt.:////aaaeeeeeniiiiiooieoa/ee 119599 issn. 1000 0800-436x.2221111
Hoogeboom, E., Agustsson, E., Mentzer, F., Versari, L., Toderici, G., and Theis, L. High-fidelity image compres- sion with score-based generative models. arXiv preprint 0rXiv:2305.18223: 2023.
Qian, J., Zhang, G., Chen, J., and Khisti, A. A rate- distortiion-percepiios theory for binary sources. In In- ternational Zurich Seminar on Informtion and Commu- nication (IZS 2022). Proceedings, pp. 34-38, ETH Zurich, 2022.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint 1312v: 14, 2013.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection at scale. International journal of computer vision,
Santurkar, S., Budden, D., and Shavit, N. Generative com- pression. In 2018 Picture Coding Symposium (PCS), pp. 258-262, 2018
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
2020.

---

Synonymous Variational Inference for Perceptual Image Compression
Shannon, C. E. al. Coding theorems for discrete source with fidelity criterion. IRE Nat. Conv. Rec, 4(142-163):
1, 1959.
Theis, L. Position: What makes image realistic? In Pro- ceedings of the st International Conference Machine Learning (ICML), 2024.
Theis, L. and Agustsson, E. On the advantages of stochastic encoders. In Neural Compression: From Information Theoor to Applications Workshop @ ICLR 2021, 2021.
Theis, L.. and Wagner, A. B. A coding theorem for the rate- distortion distortion-perreetion function. In Neural Compression: From Information Theory to Applications Workshop @ ICLR 2021, 2021.
Theis, L., Salimans, T., Hoffman, M. D., and Mentzer, F. Lossy compression with gaussian diffusion. arXiv preprint Xii::2206008889, 2022.
Thomas, M. and Joy, A. T. Elements of information theory.
Wiley-Interscience, 2006.
Toderici, G., Theis, L., Johnston, N., Agustsson, E., Mentzer, F., Ballé, J., Shi, W., and Timofte, R. CLIC 2020: Challenge on learned image compression, 2020, 2020.
Wallace, G. K. The JPEG still picture compression standard. Communications of the ACM, 34(4):30-44, 1991.
Weaver, W. Recent contributions to the mathematical theory of communication. ETC: a review of general semantics, pp. 261-281, 1953.
Xu, T., Zhang, Q., Li, Y., He, D., Wang. 7., Wang, Y., Qin, H., Wang, Y., Liu, J., and Zhang, Y.Q.. Conditional perceptual quality preserving image compression. Xrxiv preprint ar0iv:2308. 08154, 2023.
Xu, T. Zhu, Z., He, D., Li, Y., Guo, L., Wang, Y., Wang, Z., Qin, H., Wang, Y., Liu, J., and Zhang, Y.-Q. Idempo- tence and perceptual image compression. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Yan, Z., Wen, F., Ying, R., Ma, C., and Liu, On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 11682- 11692. PMLR, 2021.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E.,, and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
12

---

Synonymous Variational Inference for Perceptual Image Compression
A. The Proof of the Key Theoretical Results
In the main text, we present Lemma 3.2 and provide a brief proof of the optimization direction of SIC Theorem 3.3, based on this lemma. In this appendix section, we first provide detailed proofs of Lemma 3.2 and Theorem 3.3, and briefly discuss their relationships with the existing image compression theories.
A.1. The Proof of Lemma 3.2
Lemma 3.2. When the source considers the existence of an ideal synset X and the decoder places the reconstructed sample in reconstructed synset X, the minimization of the expected negative log synonymous likelihood term
min Bay |- Pxg (X(x.) < min Byng e. {>a Ex-p(a) (dx + Xp DxL PP [p..]
(17)
in which Xa and Ap are the tradeoff factors for the expected distortion term and the expected KL divergence term, respectively,
Proof. According to the generative model presented in Figure 2, any sample æ; in the ideal synset X' is expected to be capable of being generated using a synonymous representation U. and detailed sample De.i in which the detailed sample Dc., is sampled based on a specific prior Pg. 8.. (0,00; +b, 0p). It should be noted that De. # D. since the detailed representation G. is not required to be coded and transmitted to the decoder end.
Additionally, according to the semantic information theory based on synonymity (Niu Zhang, 2024), the probability of synset equal to the sum of the probability or the integral of the density of each sample within the set. Herein, we consider the integral form because image samples within ideal synset typically be transformed into one another through continuous changes.
Based on the above factors, we can expand the expression on the left side of (17) as follows:
Be-p(a) Bigu [- log Px
(XI0.: 0,)]
(n)
E Empp(1) B jm
log / Pu.
(Xl0, De.ji PV.., 1. (We.jlasi 0p) dūe.
Expp(a) Bngnn
{
log Eg..,l0, PO. so px px. (x 0
(18)
(c) Emppla) Egnu
/ J. Pen PailDs.
log Ey... ( M 2o
(I. L. De. 0g) dx
(d) Em- Byng
log E n EX u. [/
Umee Parlit (ar (b2)) doc }
where (a) is achieved by introducing De, into the conditional probability with a corresponding integral; (b) is according to the definition of mathematical expectation; (c) is according to the integral relationship between the probability of the ideal synset and its samples as stated before: (d) based on a determined generator O, which can definitely map the input D, and Je.j to the output žj, i.c., ã; = 9s (U. 3 ; 0g). It should be noted that in the equation (d), the reconstructed sample æ, is not required to be completely aligned with the source synonymous sample æ; but for the overall ideal synset X, which is why two subscripts, i.e., and j. are used here. By minimizing this term, we can use the synonym representation ú, to obtain the reconstructed synset X, bringing it closer to the ideal synset X, thus achieving the optimization objective shown in Figure 1.
Since the above results only involve the sample x; from the ideal synset centered by the original image x and the reconstructed sample ăt obtained by the SIC codec, the influence of the original image sample x has not directly involved. To this end, we consider to rewrite (18) as
Bxn {
V/ Ex
Parlay (x: dxx; }
log E a Ex
(a) E Expp(() Bino
I Paylon (#t (..)
Pa.. (x:;
-log EE E.
dx:
(19)
Jaxxx
pn; (mt)
(b) Exmpp( Begin
Pm (xx)
paz (z))
Pe: (xrs) dxy
}
-log E,zy E
Pæ, lae, (ātllnn))
Pa, (x,)
Pe (x)
Pay (xt,)
13

---

Synonymous Variational Inference for Perceptual Image Compression
Pue
(
in which (a) is by the Bayes' theorem, and (b) is achieved by introducing the reciprocal terms
and
Pæ,
Pæ
Furthermore, since any sample x; in the ideal synset (including the original image sample 2) shares the same synonymous representation V. by the ideal parametric SIC encoder with &g. the posterior term the (19) satisfy the following equations:
Pa, m. (ū)) Paylü. (ă,)ū..) Pa. (²,]2z).
(20)
Therefore, the result of (19) can be further derived as
Rxwp((x) Bar log E Un
Pa (x)
Pā, (xt))
Pæ: (xt))
Peelee (#rg
da:
Pa, (
Pa (æ))
Pay (
Pe (az)
Pā; (3t
Pm: (2:a)
E p Egng
loo Eg E
Pārylo (at ())
dx
Pæ. (क)
Pæ (x)
Pa. (a:
(b) E p())( B Bing
Pa; (
pa. (x:))
log 2
Pulās (xr/a²))
dx;
my E
Pe (x)
pa;
(xt))
p(x Enjng
log I. 2
Pælãs (ac)
pa, (Ø))
px:
(x:;)
day
(21)
Pu (x))
Pu; (ārz)
VaEE
(c) B Ba-p(a) B grag
Pa. (i,z)
1
Pai (xts)
- |2 }
- log 18 ee, Exx v.
Pula; (ze(ãe)
dx:
Pa (2))
X
BIEX PT; (o²)
(d)
Pe (x:
Pe: (2xa)
$ Empp(a) Eigng B E
log Pala (mrliry))
log
log |
dx:: - log |
pæ, (i))
D.E Pa, (i)))
(e) Bx B B 2, Ex v.
Pw (x)
Pes, [
Pata, (zz(ãt)) + log
log
dx: log |
pa; (āt))
X
J.EX
pā; +3))
where (a) replaces Payl le (i. (x;) with Paylæ l (ăt (zz) using (20); (b) is derived by the Bayes' theorem in reverse; (c) introduces | and its reciprocal, in which |X| denotes the size of the ideal synset X, and can be used to express an arithmetic mean along with the followed integral over X: (d) and (e) can be scaled based on the Jensen's 's inequalities, respectively, since - log (.) is a convex function.
Next, we examine the result of (21) separately.
1. The Derivation of the First Term. (xe=p(())BBre E [- log Pala, (a|ā²)] denotes an expected distortion which averaged on the source images and the corresponding samples of the reconstructed synsets. As a typical case, when the likelihood probability Pala, (x|æ,) follows a Gaussian distribution N' (æ/ o² (in which denotes the dimension of the original image x),, this term will be equivalent to
1
| = + d log (2TO²), (22)
Baa Bgng &
Ei 9. |- logPata, (x|a;)]
-Eanpla) Byng EX
2a²
in which the 2² is the variance term of the set Gaussian distribution, i.e., the power of the quantization noise. this case, the term be considered as weighted Expected Mean Squared Error (E-MSE) loss (instead of the Mean Squared Error (MSE) loss) plus constant.
1
is often replaced with a hyperparameter as the tradeoff factor to the MSE loss to the balance with the coding rate. However, in SIC, if E-MSE is used the distortion loss term, it is incomplete to define the physical meaning of the hyperparameter as 2, alone: The effect of scaling, due the inequality in (21)(d), should be also considered. Therefore, we summarize the analysis results
In typical LIC methods (Ballé et al., 2017: 2018), the multiplier 2,
Xd Em-p(a) R | -ā ~,
const,
(23)
in which Aa= = a. (x)) synset X, and can be implicitly incorporated into the value of the hyperparameter da thus it does not need to be
-0 The multiplier ar (1x²) is scaling factor influenced by the size of the reconstructed
1
14

---

Synonymous Variational Inference for Perceptual Image Compression
explicitly assigned. It should be noted that when the equality condition of Jensen inequality (21 (d)) is satisfied the reconstructed synset contains only one sample, the multiplier oua (1x+1 1, and the analysis result will be degraded
into X- Ex=p(() Eiy-o ||æ a 11 + const.
When other distortion measures (such as Expected MS-SSIM, abbreviated as E-MS- (MS-SSMM) are used instead of E-MSE, the situation is similar and will not be discussed further. Herein, we give general analysis result as
Xa Bw-p(a) By By-nnBae [d (æz, x; + const,
(24)
in which d (-) denotes any distortion measure between the original image and the reconstructed sample z.--
2. The Derivation of the Second Term and the Third Term. These two terms should be firstly considered together due to their linkage, which can be derived as
Pe (a))
Pu: (2::)
Ew)p(x) Berppa))Bugg Ba EXIU.
log
log
da:
Pă, (z:g)
X
Pā, (
Per (a:)
Pan. (x)i)
= (x)p(()) Eg By E V.
log
log
dx;
|2 X
p (i:))
pā; (ãc) (
1
Pw (2)
Pa. (x))) dat
ppxpp( Eignnn Fa E.
log
|2
pā, (x: Pu; (x:;)
(25)
1
Pee (x)
log
day
X
Pen: (x:)
Pr (x)
Reepple)
log
|
1.Ex
Pe: (I:))
DKL [Pell
dx; = f (xc, X),
|2|
Ø,EX
i.c., the arithmetic mean of the KL divergence between the original sample and the synonymous sample æ,, which non-negative function (ar, X)) of the original sample and the ideal synset X. If these factors at the source determined, the result of the function will be constant. For special case, when X' contains only one sample, i.e., the original image æ, this constant will be equal to 0 since the two distributions reduce to only one distribution Pa-
Despite the foregoing facts, if it is treated as a constant, the existence of the ideal synset on the source will lose its mcaning during the minimization process, and no samples with perceptual similarity to the source image will be available to provide a reference for the reconstructed samples. Therefore, we need to consider the second and third terms separately to determine the meaning of the constant value in the optimization process:
Pa (x)
For the second term Ex-pp(=)) Eg Es, EXIū
log
it can be further derived as Pæ, (x,)
Ex Eg Ei log
Pee (x)
Em-p(a)) log Pa (x)
pa, (it;) = Eg
E R E
Pā, (2t))
(26)
Egg E DKL [Pw (Pllpaa))
i... Expected KL Divergence (E-KLD) between the distribution of the original image Pa and the distribution of the reconstructed sample Pa, that averaged on the reconstructed synset X.
log Pa. (x: dx:; although it has similar X M.EX Pay (x)))
As for the third term, i.e., (e xx ú.
form to KL divergence, cannot be called KL divergence because uses the arithmetic mean instead of the mathematical expectation, thus lacking the non-negative properties of KI. divergence. It should be noted that the outer mathematic expectations (E the conditional probability Pa (ān az)) instead Pa, (ir)), thus the result cannot be regarded as KI. divergence from this perspective neither. In spite of this, we can intuitively find the conditions under which this term equals O, which can be expressed as
v. is actually calculated the expectation value according to
Pm, (x:))
Eewpp( Bing Ea, E
log Pā, (it))
dx:
= -0
Pe; (xt) = Pa, (Ø:)) Vi,j (27)
|28
15

---

Synonymous Variational Inference for Perceptual Image Compression
i.e., for arbitrary sample pair ay), the probabilities of both samples equal. To facilitate subsequent analysis, we label this term -gp
Based on the above analysis, we obtain the following equation relationship:
1
Biggg ex v. DKL [Pw [P:s] 6p
DKL (P dx; (xe, X).
(28)
|2 22:EX
From this formula, we can see that although o (ax X) is a constant when a and X are determined, we can ap- proximate it by minimizing the E-KLD term e DxL. [PallPay]- For special case, by forcing 6, with equal probability sampling for Z. the E-KLD term Ex,,, DKL [pællPa,] is equal to the arithmetic mean term [ Jmzex DKL Pa [P,,] dæ;.
Similar the first term, considering the effect of scaling in (3.2)dd), summarize the analysis results
Ap By ex eăo u. DKL [Px [pa,]
(29)
in which Ao = 1/21/11 (121)= op (1x)) is also scaling factor influenced by the size of the ideal synset X and the reconstrueted synset X, and x is indicated function of the size of the ideal synset X Similar to the first term, we discuss the following two special cases:
When the equality condition of Jensen inequality (21)(e) is satisfied, or the ideal synset considered with multiple
samples while the reconstructed synset contains only one, the multiplier 1 ||x (xx) and opp (1*)
and the analysis result will be degraded into Einn DRI. [p> [ps]
When the ideal synset contains only one sample, i.e., the original image æ, the multiplier 1|x1 (12() 0, which makes the analysis result equal to
3. The Derivation of the Fourth Term. With the determination of the ideal synset X' for the original image the term log X is a constant that cannot be optimized.
To summarize, by consolidating the above analysis results, we complete the proof of Lemma 3.2, that is,
min Be)) [- log log (X (x)
mmin Ad Bapp(a) E (d (x. 2:)] const] {>pp + min Bgng {Aa Eawnta)) (d (a. a:)] + Ap DKL (p 1lal
DKI. [p [pmllva,]
} const (30)
A.2. The Proof of Theorem 3.3
Theorem 3.3. For an image source x ~ p (x) together with its bounded expected distortion BB E (x, a;)] and expected KL divergence Eg, 9 Dxr. [pm [Pa;], the minimum uchievable rale of percepiual image compression is
R RxX) = min X x()
E Ex-p(on
E E
[d (x. x:)] S D, DEL [pollpa.] < P,
EX
(31)
E Ex
where I x (x: H, (x)
H, ($1x) with semantic variable X corresponds to the reconstructed synser X.
Proof. The key point to prove this problem is to consider an ideal scenario, in which there are multiple image samples æ; at the source with similar perceptual similarities to the original image æ. In this scenario, each sample can be assumed to be potentially generated by the ideal perceptual image decoder. To this end, it is necessary to assume the existence of an ideal
16

---

Synonymous Variational Inference for Perceptual Image Compression
synset X' the source which can encompass these samples (including the original image æ:). These samples must share the same synonymous representations which can be represented y, in the latent space, while the unique detailed features of each sample should be represented as y. in the latent space.
Based on this assumption, there should be an ideal image codec, in which the encoder ensures that any sample within the ideal synset X' obtains the same synonymous representation D, after encoding, while the decoder, upon receiving only the synonymous representation .. can generate different samples æ; within the ideal synset X' through sampling De.," The optimization process leading to this ideal encoder pair can be modeled as a variational auto-encoder model, which can be achieved by minimizing the partial semantic KL divergence based on the idea of the proposed synonymous variational inference (SVI), that is,
log (5)zz)
En--p(n) D [gl| Pg. xx] Ex- Byt mmy
Pū.1 1 (ÿ, xt)
9 (z)(z)
) By) amy Eawpla)) Byy any
log px.u, (X. (.))) /px (x)
(32)
q (ū xx)
log
PXIū, (x)
Pú. (Us) /px (2))
= Ex-p(a) Fig awy
[lg 9 (ū(x:) Px15. (x) - logps. (ū.) + log px (x)
Next, We examine the result of (32) term by term.
1. For the first term log q (z),;), since the noisy latent representation ÿ can be separated into two parts, i.e., a synonymous representation y, and detailed representation y. it can be expanded to
log q 9 x; Φ,) logq q (ÿ. De c Φ,)) = log q (Ü. bz; Φ,)) log q (ge x. U.i Φ,)
(33)
Since both ys and V. are determined based on a parametric inference model Ja (z;; o,) and uniform density on the unit interval centered on y, and yec, this term equals a constant O.
2. For the second term - log (X' (.))) with the outside expectations Eevp(a) Bo the minimization of this term can be equivalent to minimizing an weighted expected distortion (xxp)())) Fi B E [d (ax a:)] plus an weighted E-KLD term EE B e DKL. [p=] pa.]. by Lemma 3.2.
3. For the third term - loog Py. (U.). it is the coding rate of the synonymous representations. With the outside expectations, it is also equivalent the semantic entropy of semantic variable y corresponding to a latent synset V. This can be derived by
(uenp(a))Bllaae [- -log Pù. (U,) = Em-p(a)) awg
log
Pū, (Ü., D.) dÿ.
(34)
Ex-p((r) E aan
log
(ū) dÿ
yEE
(a) H
in which (a) is achieved based on the definition of semantic entropy in Niu and Zhang's paper (2024), with the help of the weak law of large numbers. Additionally, considering the determined codec, the semantic entropy H, ( is equivalent to single-side semantic mutual information, i.c..
H
H
x
bb)
H
x
H
X X
[c]
T(X; (x:x),
x;
(35)
where (a) is by giving a determined decoder to map the latent synset 5 to the reconstructed synset X: (b) is achieved by a determined encoder, which makes log centered on y.: (c) by the definition of this single-side semantic mutual information.
X / = log q (ū.lx) = with uniform density on the unit interval
17

---

Synonymous Variational Inference for Perceptual Image Compression
4. For the fourth term - Px (2)), with the determination of the ideal synset X' for the original image æ is constant that cannot be optimized.
To summarize, the minimization of (32) is equivalent to the following optimization directions
Cx Xa- Bxrpp()) By ElV. [d (z, a,) + >o Eg E80 DKL [P= [Pp.)
(36)
+ ppxp(()) B g [- log logPs, Pù. (ū.)]
At the convergence point, when the minimum value of the loss function is achieved, the model effectively minimizes the single-side semantic mutual information with the quantized form of X under the constraints of bounded quantized expected distortion and E- E-DD, as follows in (31). The weights A and Xp can be considered as Lagrange multipliers to the rate term. So we conclude this theorem.
From the proof of Lemma 3.2 and Theorem 3.3 above, we can see that the differences between the proposed SVI and conventional variational inference in guiding image compression tasks mainly include:
The Analysis Method for the Likelihood Term: Conventional variational inference treats the likelihood term -log (mlū) in usual LIC methods as weighted distortion (Ballé et al., 2017; Blau & Michaeli, 2018), primar- ily because the synonymous relationship emphasized in this paper is not incorporated in these works.
In our proposed SVI, since the synonymous relationship is considered, the likelihood term primarily focuses on the mapping relationship between the latent synset and the ideal synset, which is formed by - log 1xge, (XlT.) Lemma 3.2 states that the minimization of the expected synonymous likelihood term is equivalent to the minimization of a tradeoff function with weighted expected distortion and weighted E-KLD term.
Additionally, the analytical process of Lemma 3.2 emphasizes that the fundamental reason for the expected KL divergence term's 's existence is due to the consideration of the ideal synset X centered by the original image Once the ideal synset is unconsidered (equal to the ideal synset only contains the original image, i.e., X = {x> }), the expected KL divergence term will disappear in the analysis result, which means the degradation to solely the weighted distortion term. Therefore, we can give the following statement:
To the best knowledge of our authors, our method is the first work that can theoretically explain the fundamental reason for the divergence measure's existence in perceptual image compression, which stems from considering the ideal synset as the reconstruction reference.
The Considerations on Coding Rates: Conventional variational inference considers performing entropy coding for all latent representations, whose coding rate can be represented as - -logp p (ū).
In our proposed SVI, only a partial of the latent representation is required to be encoded, whose coding rate is expressed as - log p (ū.). We emphasize that the other partial, i.e.. the detailed representation, is not required to be coded: it can also be sampled from the latent synset y by the decoder, which is not necessary to keep consistency with it at the encoder end. This stems from the change in optimization direction from the original sample-orie oriented to the ideal synset-oriented, which allows the decoder to generate any samples that can exist in the ideal synset.
Since the detailed representation can be obtained by sampling at the receiving end and is allowed to contribute effective information for image reconstruction, encoding only the synonymous representation part theoretically improves coding efficiency compared to traditional methods. From the perspective of mutual information, this can be expressed as Xx; S 1
(x; X; x) in which the condition for the inequality to hold as equality is that the reconstruction the synonym is restricted to producing only one sample, meaning the decoder is not allowed sample Úe.j. nor is it permitted to use Ve,, as input to the generator.
A.3. Discussions on the Relationships with Existing Image Compression Theories
From Lemma 3.2, Theorem 3.3, and their respective proof processes, it is evident that the optimization objective derived in this paper through synonymous variational inference is compatible with the optimization objectives of existing image compression theories:
18

---

Synonymous Variational Inference for Perceptual Image Compression
Compatibility with Existing Rate-DistortionnPPrrretioo Tradeoff: According the triple tradeoff shown (31), when the reconstructed synset is not considered (equal to the reconstructed synset contains only sample, represented as X { the optimization objective will be degraded into the existing rale-distortioon-eerepioo tradeoff. This relationship can be expressed by
R (X) min I (x:X)
RDD,P)
min Xx p()
X (
s.t.
Ba- B El0
[d (z, a:)] S D. E E. DL [Par |a:] S P.
s.t. Ex-p(a) (dl (ax, .t) S D,
(37)
DKL PP [p S P.
in which the KL divergence DKL. PP= |al is typical measure of the divergence between distributions, i.e., dp (Pm Pa) in (3), as stated in (Blau & Michaeli, 2019). In view of this, the existing rate-distortion petttootioooperepttoon tradeoff (4) is special case of (1I) when there is only one sample é in the reconstructed synset X.
Compatibility with Traditional Rate-Distortion Tradeoff: Based on the analytical process of Lemma 3.2, when the ideal synset not considered (equal the ideal synset contains only the original image, represented as X {x}), F), the expected synonymous likelihood term will be degraded into the usual likelihood term, i.e.
Feee-p(e) Fgg- [- logPx1s. (x) x=
Fx)))(t)) [- log Par (x ()]
(38)
in which the minimization of the usual likelihood term Eg= [- log Paly l
(æ [ equivalent the minimiza-
tion of a weighted distortion xE Bgrug (x, ;)], makes the existence of the KL divergence term unnecessary.
Additionally, X = { will also implicitly makes the existence of Úe.j unnecessary, which results in X' = { Therefore, the relationship with the traditional rate-distortion tradeoff can be represented by
(x) = min (x: m()(a)))
R = min / ( x (x: p(0/2)
x=
s.b.
E (R [d (x, kt:] D,
(39)
(x=(=))
s.t. Em-p(()) [d (x, +)] S D.
E 22 DKL lez [Pa.] < P,
In view of this, we state that the traditional rate-distortion tradeoff is also special case of (11), where the condition is that the ideal synset X always contains only a single sample, i.e., the original æ.
Therefore, we demonstrate that the foundational theories underlying existing image compression methods can be viewed as special cases of our analysis viewpoint. In other words, our theoretical analysis provides consistent and universal guidance for designing image compression approaches, regardless of whether compression is considered for perceptual quality.
It should be noted that, although the E-KLD term in the above analysis represents the optimal distribution distance selection based on minimizing the partial semantic KL divergence, accurately calculating the E-KLD between two image sets may be unrealistic due to the complexity of the image source. Therefore, we can refer to existing empirical practices in perceptual image compression, such as using metrics that tend to be subjective, like LPIPS or DISTS, instead of the KL divergence calculation, or using adversarial losses in the GAN training process, such as Wasserstein loss, a substitute for KL divergence.
B. Relevant Thoughts on Semantic Information Theory
In this appendix section, we will briefly provide relevant thoughts on semantic information theory based on our analytical results. We emphasize that the semantic information theory referenced here specifically pertains to the synonymity-bass semantic information theory, primarily derived from Niu and Zhang's 's paper (Niu & Zhang, 2024).
B.I. Relationships with Existing Conclusions in Semantic Information Theory
Semantic Entropy: Given a semantic variable Ù, whose possible values are the synset U, = {ugli E N., = 1,2,.., N, and N₂, the set of ordinal numbers of all syntactic symbols u; E Ui the semantic entropy of the semantic
19

---

Synonymous Variational Inference for Perceptual Image Compression
variable is expressed as:
N Σ
H.
( Σ 1=1
log.)))
(Ui.)
Σ
((u) log
Σ p (u+)
(40)
1, iEN
NENi,
in which the probability a single semantic symbol corresponds to the synset 24 is the sum of the probability of each syntactic value u, E N,,, p (24.) EiEN, p (u+).
Additionally, the semantic source coding theorem in (Niu & Zhang, 2024) states that, for a semantic source Ù with its corresponding syntactic source U and its determined synonymous mapping between these two types of sources (which results in the determined synsets Ui = {us} /i E N., is = 1,2,.., N), the achievable coding rate for semantic lossless rate is 2 Hz (U) without the necessity to focus on symbol-Jevel accuracy.
In our work, we point out that the ultimate goal of synonymous variational inference is to construct an ideal synset X corresponding to the original image a by finding a synonymous mapping rule and encoding the shared latent features of the ideal synset y. as the coding sequences. In the ideal scenario where model training has converged, the reconstructed synset X at the SIC decoder can perfectly overlap with the ideal synset X. Under such conditions, the average coding rate of the synonymous representation U in the latent space can approach the semantic entropy of the semantic variable corresponding to the ideal synset, i.c.,
Ex-p(a) log (0.) (a) H. (i) (b) H1s
(c)
(d) 11. (x)
X
X
(41)
in which the established conditions of (a) (c) is the same as the conditions in (34) and (35), and (d) is achieved by the ideal SIC codec. At this point, the found synonymous mapping rule is determined by the bounded expected
distortion and the bounded E-KLD, i.e., Em- E 8. [x( a.)] < D, E DKL [Pe lpa..] < P.
Down Semantic Mutual Information: The down semantic mutual information is a measure defined
1s (U:V)==A 11s =1, (i) + 11 (V)-H((,V)) (U.v)
(42)
which is proved to be the minimum coding rate in semantic lossy source coding when the "semantic distortion" (denoted as dg (x, (x,X) in concept) satisfying de (x.*) < Des using an extended joint asymptotic equipartition property
(AEP) analysis called Semantically Joint AEP.
In our work, since the ideal synset is not explicitly constructed in practice, directly determining the overlap degree between the reconstructed synset and the ideal synset challenging. Consequently, the model obtained after training convergence may still function as a semantic lossy coding model. We propose that the coding rate of the SIC model should also serve as an upper bound for the lower semantic mutual information, expressed as:
bb)
(c)
H, (xxx)
Ex<p(a) |- log (0.)
H
H
X
H
x
+ H. X
H
X X)
(43)
(0)
H, {
+ H. X - H
X
1. X x
in which the established conditions of (a) ~ (c) is the same as the conditions in (34) and (35), (d) is achieved
by the equation H3 (xxx) = H. [ x H (x) and (e) can be proved by a scaling process according to
H ((x) > H and H, { < H
x, X which follows the fundamental properties of semantic variables
stated in (Niu Zhang, 2024).
20

---

Synonymous Variational Inference for Perceptual Image Compression
Analysis Transform
Stage
Stage
Stuge
Stage
== 512
Patch Pardtion
Swin mmnsformmet Block
Swin Transfomee Block
Swin Tmunformer Block
Swin Troosfomec Block
S/z
Channel Slicing L=
bedddinng
Patch Merging
Patch Merging
Patch Merging
B/
4XwX3
X1
X3
X52
X388
x.51
x.512
16
1,2,
Stage 4
Stage 3
Stuge 2
Stage 1
Quantizing
3C
Swin Transformet Block
Swin Trans foemer Black
Swrin Ttunsformee Black
Swin Trans former Block
Potch Div ision
Division
Patch Division
Patch Division
Cony 3x3
S Sumpling
Ax
M1-12
X1
X2
X2
X3
X
X256
X384
X512
X.1
16
Synthesis Transform 9.
Figure implementation details of the auto-encoder framework designed for progressive SIC model.
B.2, Extended Thinking: About Synonymous Idempotence Constraints
As discussed earlier, since the ideal synset is not explicitly constructed, directly determining the overlap between the reconstructed synset and the ideal synset is infeasible. However, since the ideal synset directly corresponds to the synonymous representation J, in the latent space, the distance between the two synsets in the data space can be evaluated by first re-encoding the samples from the reconstructed synset, then obtaining the new synonymous representation ÿ'.- and finally calculating the distance between ÿ, and ÿ This idea can be directly integrated into the training process. By incorporating constraints into the loss function, the SIC codec can explicitly guided to optimize toward maximizing the overlap between the reconstructed synset and the ideal synset during the optimization process, i.e.,
synnet
y
ys
(44)
in which the adopted distance measure is based on the MSE function in our model optimization.
Additionally, sufficient diversity among the samples in the reconstructed synset should be ensured, the following constraints can be incorporated into the loss function:
Edetail [ Ve.s
(45)
refers re-feeding the reconstructed image into the encoder, computing the difference between the new detailed representation V'c, and the original detail representation Ū..j- and incorporating into the optimization process of the SIC
codec.
When both of the above constraints are equal to 0, the idempotence property of the reconstructed image samples is effectively satisfied (in which idempotence "refers to the stability of image code to re-compression", as stated Xu's 's paper (2024)). Here, we term the above constraint as the synonymous idempotence constraint and incorporate it into the loss function for the neural SIC model implemented this paper. Please refer to Appendix C. for specific training details.
21

---

Synonymous Variational Inference for Perceptual Image Compression
ha
x
4
y
Context Model
y
EC
Bits
Bits
Factorized Entropy Model
Entropy
y
n,,
Parometers
y
h
ED
y
Sampling
Figure 8. The joint autoregressive and hierarchical prior architecture of the progressive SIC model.
C. Experimental Illustaatoon:: Implementations and Supplementary Results
In this section, we provide implementation details of the progressive SIC model and supplementary results for Section 5.
C.1. Implementation details
The auto-encoder architecture, including analysis transform the encoder and synthesis transform the decoder, is implemented based the Swin Transformer (Liu et al., 2021). The implementation details are shown in Figure 7. Both the analysis transform and the synthesis transform perform a 4-stage nonlinear processing, in which each layer includes dimension adjustment module (Linear Embedding, Patch Merging, and Patch Division) and a Swin Transformer module.
For the analysis transform, the image æ with the resolution of w x is first partitioned into patches with the resolution of Then, the patches are fed into the Linear Embedding module stage for dimension expansion, and the features further extracted using followed Swin Transformer Block. The following processing stages all use the Patch Merging block and several Swin Transformer blocks to extract deeper features. Finally, the analysis transform outputs feature map with dimension of X mi 512, in which the channel dimension C 512.
To support multiple synonymous level partitioning, we perform equal slicing along the channel dimension of the latent representation. We set the number of the synonymous levels I. 16, thus the latent representations are partitioned along the channel dimension into 16 groups. Each synonymous group contains sub-feature map with the size of A T6 x s x 32. When the I-th synonymous level is selected, the symonymous representations y, contains the first groups of the sub-feature map with the full size of x w 32/, thus makes the remaining levels
detailed representations ye-
For the synthesis transform, the quantized synonymous representations y, and sampled detailed representations ú are as input. Then, four upsampled Swin Transformer stages and final convolutional layer are applied to the input to integrate the global information of the image, in which each stage increases the input resolution through the corresponding number of Swin Transformer modules in the analysis transform and a Patch Division module. Finally, a convolutional layer outputs the reconstructed image U
For rate estimation, the progressive SIC model adopts a joint autoregressive and hierarchical prior architecture based on (Minnen al., 2018) shown in Figure 8. All these modules in this architecture implemented based convolutional neural networks represented by Figure 9, in which Q represents the quantization model using the round function.
In this architecture, the hyperprior hú and h2 are performed for both the distribution estimation for the synonymous representation 9, and the sampling for the detailed representation. The hyperprior performs the "forward adaptation" to estimate p. of and whh of based on the side information é, while the context model performs the "backward adaptation" " estimate f o based on the already-coded synonymous representations, in which the expressions "forward adaptation"
22

---

Synonymous Variational Inference for Perceptual Image Compression
384 24
21
Cony 5x5 e= 768 21
384
Conv 3x3 == 1024
384
Con: 5 e= 512
LeskyyRllUU
LeckyRoLU
LeskyReLU
LeskyRel.U
Cony 5
Hyperprior Analysis
Hyperprior Symthesis
Cony 6 060
Conv 1x1 a 967
Cone 1x1 a = 641
LeakyReL1 U
Leaky Rel.
Masked-Con 5x5 6: = 641
Context Model
Entropy Parameters
Figure 9. The implementation details of the modules in the joint autoregressive and hierarchical prior architecture.
and "backward adaptation" are stated in (Ballé et al., 2020). For the synonymous representations D, an Entropy Parameters modules are employed to integrate the input μ o and μe of to an output accurate estimation μt, a And for the detailed representation Ge a uniform sampling based on the following equation utilized:
De. Q (n. + + (-2.2))
(46)
in which the uniform distribution 41 (- is set empirically. We realize that this sampling method cannot fit the ideal conditional distribution in vector form: The SVI theoretical analysis provides an ideal detail sampling principle that follows a conditional distribution l0. in vector form (stated in the Equation (18)), which guides the prediction of the samples vector De. conditioned on synonymous representations Ú.- However, the current unideal sampling mechanism cannot ensure reasonable contextual structure in the details of the reconstructed images, which may affect the distribution consistency focused by measures like FID. We argue that sampling the detailed representation match the ideal conditional distribution in vector form is challenging, especially when multiple distinct samples (i.e., M > 1). Although we are still exploring effective solutions to this problem, adopting simple yet suboptimal sampling method is currently a necessary compromise. This will be a key breakthrough direction for our future research.
18.
To support the multiple synonymous level partitioning mechanism, we modify the masked convolutional layer of the spatial context autoregressive model in (Minnen et al., 2018) to a spatial-channel context autoregressive model like (Li al., 2020), in which the 3D mask for the masked Context Model is presented by Figure 10. The mechanism is to estimate the current feature's probability distribution by conditioning on encoded spatial and channel features. This spatial-channel context autoregressive module is implemented based a 5 x 5 convolutional layer, in which the spatial context autoregressive process within the single group of sub-latent feature map is achieved by the left matrix shown in Figure 10, and the channel context autoregressive process from the coded groups of sub-latent feature map to the current group is achieved by the full matrix shown as the right matrix in Figure 10.
Additionally, for the Entropy Parameters module, all the estimations of μt and or across diverse synonymous levels share the same module. To achieve this, each convolutional layer in the Entropy Parameters module assigns group parameter of 1, enabling the layer to process I independent groups in parallel. These groups perform the estimation processes separately for each sub-feature map, and their outputs are concatenated to form μ, and o.
Before model training, hyperparameter values for the loss function (12) and (13) must be specified before model training. Table presents these hyperparameters of our progressive SIC model, which are configured empirically.
Since multiple sub-feature maps are partitioned in the channel dimension to build different synonymous levels
23

---

Synonymous Variational Inference for Perceptual Image Compression
3D-Mask for Masked-Cony 5 5
1
1
1
1
1
1
1
0
0
0 0
0
0
0
C;
...
Cx-1 ...
Figure 10. The 3D mask for the masked convolutional layer in the context model.
Table 1. Hyperparameters Configurations for progressive SIC model training.
2
3
5
0.5
X
128
256
384
512
640
768
896
1024
(0)
239/8
238/8
237/8
236/8
35//8
234/8
233/8
2
x
45/8
242/8
29/8
2
33//8
20//8
227/8
224/8
10
11
12
13
14
15
16
0.5
X
1152
1280
1408
1536
1664
1792
1920
2048
231/8
230/8
2///
288/8
227/8
2
2
224/8
221/8
218/8
25//8
2 222/
29/8
2
2
24/8
1.2....,L. each synonymous level needs to learn different levels of information during model training, However, due to the limitations of computing resources during training, it is not possible to cover all synonymous levels in each forward process, and we can only use the loss functions of different synonymous levels alternately for training. This alternating training between layers can lead to some layers losing the ability to extract effective information early in training, causing the coding rate of the corresponding sub-feature map approach in the subsequent training process and in the final model. To avoid this, an effective trick is to introduce the following constraints into the loss function during the warming-up up phase, ensuring that each sub-feature map learns valid information:
C wwamming log Pù. (i)) + -st (- lolog Pū, (Uz)..... , -og log Püz (ūL)),
(47)
in which the former term is the minus coding rate estimation the current sub-fcature map corresponds to the synonymous level I; the latter term, i.e., std| (.) a standard deviation function, which calculates the standard deviation the coding rates of each sub-feature map; and are the corresponding tradeoff factors. This constraint increases the coding rate estimation of the sub-feature map and limits the standard deviation of all the sub-feature maps' coding rate estimation, allowing each sub-feature map to learn a certain amount of effective information during the warm-up stage without excessive learning. We empirically set a 4, = 64 in this constraint.
Combining the synonymous constraints discussed at the end of Appendix B.2, the constraints in the training loss function in the warming-up process can be summarized as
(1)
ssynett +
ddetail
cwarming
(48)
After the warming-un up phase, the constraints will be modified to
(()
reyuset
L
datail
(49)
24

---

Synonymous Variational Inference for Perceptual Image Compression
BPG
VIM
HIFIC
MS-ILLM
MS-ILLM (No-GAN)
Progressive SIC (Ours, M-1)
CLIC2020 Test
DIV2K validation
Kodak
40
35.0
35.0
PSNR (()
PSNRR (T)
32.5
35-
PSNR
30.0
30.0
27.5
27.5
25.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(BPP)
(BPP)
(BPP)
CLIC2020
DIV2K validation
Kodak
0.25
0.25
0.25
0.20
0.20
0.20
LPIPS
0.15
LPIPS
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Pixel (BPP)
(BPP)
(BPP)
Figure 11. Comparisons of methods using PSNR and LPIPS on different datasets. Each point on the HiFiC and MS-ILLM performance curves is from single model, while entire performance curves achieved by single progressive SIC model.
During training, we treat the first 12500 iterations as the warming-up up phase, and the subsequent iterations are for the formal training process.
In performance validation, for each input image at each synonymous level, we randomly select single sample â from the reconstructed synset X as the resulting image. Then we compare this result with the baseline methods by calculating the average coding rate, PSNR (distortion quality), and LPIPS and DISTS (perceptual quality) across the dataset.
C.2. Supplementary Results
Figure 11 provides the supplementary PSNR and LPIPS quality results for Figure 4, which are quality assessment measures corresponding to the distortion and perceptual terms in the training loss function. As shown in the figure, for the distortion evaluation measure PSNR, as the coding rate increases, PSNR progressively approaches the performance of No-GAN MS-ILLM and even that of BPG. For the perceptual quality evaluation measure LPIPS, our solution reaches near the LPIPS quality of the perceptual solution at full rate. Even for the Kodak dataset, at low rates, our method outperforms the No-GAN MS-ILLM scheme. this case, even without applying DISTS to the loss function for optimization, our method still demonstrates performance gains under most DISTS rates, shown as the results Figure 4, This indicates that our method aligns better with the resampling tolerance emphasized by DISTS.
Consequently, method achieves comparable nate-distortion-ereeee model, which demonstrates our advantages in variable-rate support.
performance using single progressive SIC
Besides, the visualization results are presented from Figure 12 to Figure 17, which can be divided in two groups:
1. The first group illustrates the process of improving the reconstructed image the synonymous level / increases, which includes Figure 12, Figure 13, and Figure 14, corresponding to the test image from CLIC2020 test, DIV2K validation, and the Kodak dataset, respectively.
We captured the effects of reconstructed images at specific synonymous levels to clearly demonstrate how switching synonymous levels impacts the quality of the reconstructed images: At low synonymous levels, the coding rate for the synonymous representation is relatively low. As a result, the reconstructed image is sampled from a larger synset, capturing only the global semantic content of the image, with limited pixel-level detail accuracy. As the synonymous level increases, the coding rate rises, which reduces the size of the reconstructed synset. This allows for more accurate
25

---

Synonymous Variational Inference for Perceptual Image Compression
0.0495 hpp
0.0774 hpp
0.1400 hpp
0.2495 hpp
Original
1=1
1=3
1=6
1= 10
t=
1=4
1=
1= 16
bpp
hpp
0.1990 hpp
bpp
Figure Visualization results of reconstructed images different synonymous levels using progressive SIC (M 1). Image from the CLIC2020 test set.
0.0532 bpp
0.0766 bpp
0.1243 bpp
0.2450 bpp
A
Original
1=3
1-2
1-1
1
1- 16
f Aoss
0.0664 bpp
0.0863 bpp
0.1927 bpp
0.3859 bpp
Figure Visualization results of reconstrueted images different synonymous levels using progressive SIC (M 1). Image from the DIV2K validation set.
0.0661 hpp
0.0936 bpp
0.1502 bpp
0.2868 bpp
63455
Original
20042
3
1=
1= 10
(=
0.0795 bpp
0.1055 hpp
0.2259 bpp
0.4466 byp
Figure 14, Visualization results of reconstructed images different synonymous levels using progressive SIC (M 1). Image from the Kodak validation set.
26

---

Synonymous Variational Inference for Perceptual Image Compression
Original
Rate 0.0976 hpp DISTS 0.0818
Rate 0.0961 hpp DISTS 0.0783
Figure 15. Visualization comparison of reconstructed images synonymous level 5 using progressive SIC with M and M 5. Image from the CLIC2020 test set.
Original
M =1
M =5
Rate 0.2158 bpp DISTS 0.0501
Rate 0.2156 bpp DISTS 0.0492
Figure 16. Visualization comparison of reconstructed images at synonymous level | = 6 using progressive SIC with M 1 and M = 5. Image from DIV2K validation set.
Original
M =1
M =
Rate 0.3600 bpp DISTS 0.1021
Rate 0.3281 bpp DISTS 0.0935
Figure 17. Visualization comparison of reconstructed images at synonymous level | 7 using progressive SIC with M 1 and M 5. Image from the Kodat dataset.
27

---

Synonymous Variational Inference for Perceptual Image Compression
Different Conditional Model
Shared Main Architecture
Different Synonymous Level
Conv12 I.Relu
NN 16
Conv64 2-4x4 Lrelu
Conv 12nv128 2-4x4 Lrelu
Conv256 2-4x4 Lrelu
Conv512 2-4x4 Lrelu
Convl-1x1 Sigmoid
y (
((x.g.)
D 9..)
Ø / /
Concat
Figure 18. Employed Discriminator Architecture in our finetuning attempts.
semantic information and progressively enhances the details the reconstructed image.
These visualization results demonstrate that the progressive SIC model we implemented can effectively leverage the switching of synonymous levels to adjust to different coding rates, enabling the accuracy of the reconstructed image to improve as the coding rate increases, while ensuring a smooth enhancement of perceptual quality.
2. The second group presents a visualized quality comparison of the reconstructed images corresponding to specific synonymous levels, with the number of detailed representations Øc. set to M and M 5 during training, includes Figure 15, Figure 16, and Figure 17, corresponding to the test image from CLIC2020 test, DIV2K validation, and the Kodak dataset, respectively. The perceptual quality is evaluated using the DISTS measure.
These visualization results demonstrate certain advantages of increasing the sampling number of ÿe.j perceptual qualities at certain synonymous levels, since more sampling results in effective learning of the shared characteristies within the reconstructed synset X.
We state that the results presented in this article some of our preliminary results. Currently, investigations on the perceptual loss (i.e., the divergence term) utilization, synonymous level partition mechanisms, sampling numbers, and hyperparameter settings on progressive SIC schemes are still insufficient. These factors may contribute to the issues observed in Figure 4 and Figure 5. Future works are needed to explore these aspects in more detail.
D. Limitations: Implementation Details and Supplementary Results
In this section, present the implementation details and results of the supplementary experiments in Section 6, focusing the primary concern of using GAN-based adversarial loss to replace the divergence term in the loss function of Equation (11).
D.1. Implementation Details
As described in Section 6, GAN-based adversarial loss replace the divergence term in the loss function and improve the performance of our implemented SIC model. Hence, the auto-encoder architecture remains consistent with those in Appendix C.I. This section mainly details the discriminator design.
The utilized conditional discriminator is a convolutional neural network with two input branches-the original image x (1)
main branch and the synonymous representation (() condition branch- and outputs the probability that the input image is judged as real. Besides, the discriminator model consists of two parts, i.e., group of conditional models and a main architecture. The conditional branch is first upsampled by 16 times, concatenated with the main branch in the channel dimension, and then fed into the main architecture to estimate the output probability. We fine-tune the synthesis transform gs (i.e., the generator) with a single discriminator model, using a corresponding conditional model for each synonymous level and sharing the main architecture across all synonymous levels.
reconstructed image 2
28

---

Synonymous Variational Inference for Perceptual Image Compression
D.2. Supplementary Results
Figure 19 shows the performance of the fine-tuned model (labeled "with GAN") using PSNR, LPIPS, DISTS, and FID, compared with the original model (labeled "no-GAN") and other comparison schemes across different datasets. It should be noted that calculating FID on the Kodak dataset is unavailable this dataset has only 24 images to yield the useful metrics, which is also stated in (Muckley et al., 2023). These results show that introducing adversarial loss improves perceived quality, indicating its optimization direction is closer to the optimal divergence than metrics like LPIPS.
However, unlike classical perceptual compression methods (Muckley et al., 2023), we observe that while DISTS more tolerent resampling- improves significantly, FID-which measures distribution distance-shon limited gains. This suggests our empirical detail sampling leads suboptimal reconstructed image distribution, likely because does not follow the ideal probability distribution of the detail representations in vector form, which verifies statements in Appendix C.I. Therefore, solving this problem is also one of the key research directions of our future work.
Figure 20 compares reconstructed images before and after fine-tuning with adversarial loss at synonymous level / = 1, using samples from different datasets. Although DISTS and FID show limited numerical improvement at this level, the perceptual improvement can be visibly enhanced, which confirms that adversarial loss improves generation quality across synonymous levels (i.e., across different rate ranges).
Although acknowledge performance gap between method and state-of-the-art approaches, also need to clarify once again the advantages of our approach. That is, our implementation scheme-unlike schemes that require separate models for each rate-offers much easier deployment by achieving acceptable perceptual qualities using one single model when we set different synonymous levels. We have identified its current limitations and proposed potential solutions; addressing these could enable SIC to approach the theoretically optimal scheme instructed by SVI and thus potentially surpass existing methods future works.
29

---

Synonymous Variational Inference for Perceptual Image Compression
HIFC
MS-ILLM (with GAN)
MS-ILLM (No-GAN)
Progressive SIC (No-GAN, M-1)
Progressive SIC (With GAN, M-1)
Progressive SIC (No-GAN, M-5)
Progressive SIC (With GAN, M-5)
CLIC2020 lest
DIV2K validation
Kodak
37.5
35
35.0
PSNR(ab) (1)
PSNRIAI (T)
32.5
30-
PSNR (dH)
30.0
27.5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
(BPP)
Pixel (BPP)
Pixel (BPP)
CLIC2020 test
DIV2K validation
Kodak
0.25
0.25
0.20
0.20
0.20
LPIPS
0.15
LPIPS
0.15
LPIPS (1)
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0.0
0.1
0.2
0.3
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.7
(BPP)
Pixel
(BPP)
CLIC2020 test
DIV2K validation
Kodak
0.20
DISTS (1)
0.15
0.15
DISTS
DISTS
0.10
0.05
0.05
0.1
0.2
0.3
0.5
0.6
0.0
0.1
0.2
0.4
0.5
0.6
0.0
0.3
0.4
0.5
0.7
Bits
(BPP)
Per Pixel (BPP)
Bits Per Pixel (BPP)
CLIC2020
DIV2K validation
2.
FID (1)
FDD
2.
0.1
0.6 Bits Per Pixel (BPP)
0.5
Bits Per Pixel (HPP)
Figure 19. Comparisons of methods using PSNR, LPIPS, DISTS, and on different datasets (Supplemented fine-tuned model performance). Each point the HiFIC and ILLM performance curves is from single model, while entire performance curves are achieved by a single progressive SIC model.
30

---

Synonymous Variational Inference for Perceptual Image Compression
Original
w/o Discriminator Finetuned
w/ Discriminator Finetuned
M =1 CIIC2020 H023
M 1
DIV2K H0835
Kodak H01
M 5
CLIC2020 #164
DIV2K
M =5
Kodak #15
Figure 20. Visualization results of reconstruction images: No-GAN vs. with GAN (Synonymous level = 1). Images from the CLIC2020 test set, the DIV2K validation set, the Kodak datasct.
31


Reviews:
[{"review_id": "1jJO9xWElT", "reviewer": "Reviewer_1wA9", "summary": "This paper proposes synonymous variational inference and introduces synonymous image compression. It is based on the observation that a given image to be encoded has a set of synonymous images that share the same semantic meaning. Instead of optimizing the variational distribution at the pixel level, we optimize it at the semantic level. Building on this insight, we propose a simple modification to the existing training scheme to achieve synonymous variational inference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1wA9,\n\nThank you for recognizing our work, especially our proposed analysis theory, i.e., **synonymous variational inference (SVI)**, as **a good contribution to the area of perceptual image compression**.\n\nWe think that your questions are valuable for improving our work. Below are our responses.\n\n**[Q1] Is the performance gain due to the SVI proposed in this paper, or the loss / network / other tricks?**\n\n[A1] We address your question from two perspectives: **theory and method**.\n\n- **Theory**: According to the proof and compatibility analysis in Appendix A of our manuscript, SVI can offer theoretical potential performance advantages for two reasons:\n  \n  - **Stochastic Effectiveness**. As Equation (15) (especially Lines 636~639) presents in our manuscript, each sampled random detail corresponds to a sample $\\tilde{\\boldsymbol{x}}_j$ in the reconstructed synset. When the ideal and reconstructed synsets fully overlap, that sample should match an image $\\boldsymbol{x}_j$ in the ideal synset, which is perceptually similar to the original one. Thus, the reconstruction quality can be supported by both coded synonymous representations and random details, rather than relying solely on the code sequence as in HiFiC and MS-ILLM.\n    \n  - **Encode only common features**. As Equation (31) (Lines 862~868) and (32) (Line 874) present, ideally, the coded rates of the synonymous representation reflects the rates of common features of the reconstructed synset $\\tilde{\\boldsymbol{\\mathcal{X}}}$, i.e, the minimized single-side semantic mutual information $I(\\boldsymbol{X};\\tilde{\\mathring{\\boldsymbol{X}}})$, rather than the minimized mutual information $I(\\boldsymbol{X};\\tilde{\\boldsymbol{X}})$ between the original image and specific reconstructed sample, in which the latter one is greater (line 925-926). This can be intuitively illustrated as the Venn Diagrams [Figure 1] in our [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C/), and means that our compression limits will be lower than the existing methods with the same distortion and perceptual quality constraints ideally.\n    \n- **Method**: The advantages of our proposed scheme including:\n  \n  - **Multi-rate adaptability**. This relies on **Stochastic Effectiveness**, since the adaptability across various rates is achieved by determining partial of the random details to the accurate representation of the source image, with the guidance of our derived loss function.\n    \n  - **Performance Improvements on DISTS**. This relies on both the two theoretical advantages, since the difference on the details is more acceptable to DISTS and human perception than other measures like LPIPS; besides, the compression efficiency can be further improved by encoding only synonymous representations.\n\nHowever, we admit that our proposed SIC method struggles to fully achieve SVI's theoretical advantage due to **the absence of better detail prediction and sampling method** as well as **better perceptual optimize direction like adversarial losses**. These are key directions for future research.\n\nTo verify its potential for subsequent optimization, we plot some additional experimental results to the [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C), in which we finetune our model with a CNN-based discriminator using non-saturating loss. Please check the rebuttal to **Reviewer 7ghf [A2]** for the relevant experimental configuration and results analysis.\n\n**[Q2] What if we do not use the stochastic code, but keep all other components, like loss / network to be the same? Will this reduce the performance?**\n\n[A2] Your suggestion is crucial since it touches on a core question we are concerned with: whether random details positively support both distortion and perceptual quality. Therefore, we added an ablation test result in our provided anonymous link [Figure 2~3], where we force the random details $\\hat{\\boldsymbol{y}}_{\\epsilon,j}$ to 0, preventing them from providing detail information. We obtained test results at each rate under $M=1$ and $M=5$ and compared them with the performance of random sampling. All other conditions remain unchanged.\n\nThe experimental results show that without random sampling, both models reduce the performance significantly in distortion and perceptual quality. This indicates that random details contribute effective information to the reconstructed image's distortion and perceptual quality, i.e., verifying the **Stochastic Effectiveness** mentioned above. Besides, it also suggests that perceptual image compression performance can be supported not only by coding sequences as existing RDP methods do. Based on this phenomenon, we conclude that when designed properly, the SVI-based method has the potential to surpass the performance of the existing RDP methods.\n\nAgain, we appreciate your affirmation of our work and the questions you feedback. We will update our manuscript accordingly.", "claims_and_evidence": "Yes. The claims are supported by experiments and proof.", "methods_and_evaluation_criteria": "Yes. The paper reports DISTS and BPP, and also report LPIPS and PSNR in appendix.", "theoretical_claims": "I checked the outline of the proof in the main text.", "experimental_designs_or_analyses": "The experiments are sound to me. Also, the author provides an ablation study on M.\n\nOne thing I am not sure of is that the proposed algorithm is simply modifying the latent code with a deterministic one and a stochastic one. What if we do not use the stochastic code, but keep all other components, like loss / network to be the same? Will this reduce the performance?\nOr to say, my question is: is the performance gain due to the SVI proposed in this paper, or the loss / network / other tricks?\n\nDisclaimer: I am familiar with the Rate-Distortion-Perception trade-off in general, but I only know some old baselines in this area, like HiFiC, and I am not familiar the recent development in this area. Therefore, I cannot evaluate the significance of the performance gain. Also, I my questions regarding the performance/experiments might be biased. If that is the case, feel free to correct me.", "relation_to_broader_scientific_literature": "The SVI proposed in this paper is new to me. I think this result is insightful yet simple enough to employ. I believe this is a good contribution to this area. \n\nHowever, I cannot confidently evaluate the significance of the performance of this approach.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper is well-structured and well-written. It is easy to follow and is a pleasure to read.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHDeIgGIyt", "reviewer": "Reviewer_uGiv", "summary": "This paper is about perceptual image compression, where previous works measure the perceptual quality by calculate certain divergence distance between the source distribution and the reconstructed distribution.   \n\nThis paper is inspired by a recent advancements in semantic information theory (Niu & Zhang, 2024), where manipulating a set\nof samples with the same meaning (referred as to a synonymous set, abbreviated as “Synset”) is considered as the principle of semantic information processing.  \n\nImportant terminology to understand the paper:\nsemantic information (i.e., the meaning) and syntactic information (i.e., data samples), where one meaning can be expressed in diverse syntactic forms.  \n\nThen this paper proposes the synonymous variational inference, which is based on a modified KL divergence. This partial semantic KL divergence is defined with regard to a syntactic distribution $q$ and a semantic distribution $p_s$. For VAE based learned image compression, q is defined as parametric latent density conditional on the source image $x$, which is viewed as a sample in the ideal and unknown synset X. By minimizing this KL divergence, the output of the semantic decoder can be considered as a sample of the ideal synset, thus producing the perceptual optimized reconstruction.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer uGiv,\n\nThank you for recognizing our contributions, especially the viewpoint that **our SVI theory will provide important and new insight for learned image compression community**.\n\nWe believe that your questions and suggestions are crucial for improving our work. Below are our responses.\n\n**[Q1] FID should be evaluated as previous perceptual image compression works.**\n\n[A1] Your suggestion is valid. Since FID is commonly used to measure distribution similarity between image groups, it is crucial for evaluating the performance of perceptual image compression. However, previous work (MS-ILLM) shows that using adversarial loss significantly improves FID quality. Since our method roughly uses LPIPS rather than adversarial loss to replace KL divergence for optimization, its FID performance is worse than HiFiC and MS-ILLM, which is why we did not include FID comparisons in the manuscript.\n\nSince several reviewers noted our method's performance weaknesses, especially compared to GAN-based perceptual image compression, we conducted additional experiments, adding a discriminator to our method and fine-tuning the model with a non-saturating loss in the loss function. The results including FID are available at the [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C). Please check the rebuttal to **reviewer 7ghf [A2]** for the relevant experimental setup and results analysis.\n\n**[Q2] The tradeoff between distortion and perception is not illustrated as previous rate-distortion-perception works. Besides, the implementation did not show the advantage of the method compared with previous perceptual image compression methods.**\n\n[A2] For your first sub-question:\n\n- From the perspective of our SVI theory analysis, the discussion of the tradeoff between distortion and perception is optional. If we view **Lemma 3.2** (Lines 245~258) inversely, the lemma shows that **different distortion-perception tradeoffs actually correspond to different criteria for determining ideal synonymous sets $\\boldsymbol{\\mathcal{X}}$**.\n  \n- From the perspective of the experimental results, indeed, our previous manuscript version did not discuss the distortion-perception tradeoff problem. In our provided anonymous link, we provide the performance curves with adversarial loss optimization. As perceptual metrics like DISTS and FID improve, the distortion of the reconstructed image decreases. This confirms that our method aligns with previous work on the distortion-perception tradeoff.\n\nFor your second sub-question:\n\n- Our theoretical analysis of SVI demonstrates that, under ideal conditions and with ideal model structure designs, the SIC method has the potential to outperform existing perceptual image compression methods. This is also mentioned in the rebuttal to **Reviewer 1wA9 [A1]**.\n  \n- As shown in [figure 5] of the provided anonymous link, introducing the discriminator for training effectively improves perceptual quality. We note that since the discriminator is balanced across different quality levels, the fine-tuning has not fully converged, which means that there is still potential for further improvement. Nevertheless, we still consider possible issues with the current design, including **the absence of better detail prediction and sampling method** and **the better choice of perceptual optimize direction**. These issues direct the key directions for our future work.\n\n**[Q3] Some previous works for semantic perceptual definition and optimization from different perspectives should be clarified.**\n\n[A3] Thanks for providing the relevant papers. They have given deeper insights into the area of perceptual image compression and our own works. Below are some insights on the correlation with these previous works.\n\n- The paper [R1] attemps to build a universal critic of **realism**. The authors build a universal realistic critic based on *Kolmogorov complexity*. This critic allows the measure to evaluate the realism without original image reference. In contrast, our analysis originates from semantic information theory and establishes a perceptual-oriented optimization direction with original image reference.\n  \n- The paper [R2] follows [R1] and proposes a new model for rate-distortion-perception tradeoff, which defines a log-likelihood ratio critic as the realistic critic without comparing to the original one, while our theory provides an expected log-likelihood ratio critic that requires that comparing.\n  \n- The paper [R3] is indeed related to our works, and your understanding is completely correct. They explored the optimization direction of conditional posteriors with common features but did not consider that these conditions can serve as synonymous representations for synonymous compression. Besides, they did not approach the optimization problem from the perspective of a perceptual-similar set.\n\nWe sincerely appreciate the questions you raised. These questions will help us improve our paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Dataset includes set CLIC2020, DIV2K and Kodak, which is sufficient. [Q1] While I think FID should be evaluated as previous perceptual image compression works.", "theoretical_claims": "I check the theorems, they are extended versions of previous derivation for rate-distortion-perception and they are correct to me.", "experimental_designs_or_analyses": "[Q2] The overall design is good. However, the tradeoff between distortion and perception is not illustrated as previous rate-distoration-perception works.  Another problem is the performance. The implementation did not show the advantage of this new synonymous variational formulation compared with previous perceptual image compression methods.", "relation_to_broader_scientific_literature": "This paper might be related to perceptual image quality assessment. Which is also discussed in What makes an image realistic? ICML 2024.", "essential_references_not_discussed": "[Q3] There exist some previous works for semantic perceptual definition and optimization from different perspectives. \n\n[R1] What makes an image realistic? ICML 2024.  This paper also provides an analysis of the defination of perceptual distance, which I think should be discussed as related works.\n\n[R2] The Rate-Distortion-Perception Trade-Off with Algorithmic Realism, this paper propose a perceptual metric for individual or batch images, which is quite related to this work.  \n\nAnother previous attempt to extend the RDP tradeoff for semantic compression is '[R3] Conditional perceptual quality preserving image compression', which porposes to measure the divergence conditional on semantic information and should also be discussed as related works.  \n\nIn my understanding, this paper formulates semantic oriented compression problems by introducing  Synset to previous variational inference framework, which is a meaningful extension. I think the nature of synset is quite similar to the conditional perception proposed in [R3], which referes to samples (synset) or sample distribution (conditional posterior in R3) with same semantics. The correlation with those three previous works should be clarified in the manuscript.", "other_strengths_and_weaknesses": "This paper is based on a very new advancements in semantic information theory (Niu & Zhang, 2024), I think this paper will provide important and new insight for learned image compression community.  \n\n\nIn equation 5, symbols like $U$ and $N_{i_s}$ are not explained. Though I can infer the meaning of those symbols, it's better to write it more clear for general readers.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please consider my concerns in previous sections [Q1][Q2][Q3]. To sum up, I like the formulation by semantic set and variational inference, which bring new insight to this field. However, the implementation did not show the advantage of this new formulation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "436AVIveeC", "reviewer": "Reviewer_VZKv", "summary": "This paper introduces a novel progressive training approach for image compression, which focuses on both improving image quality and maintaining semantic consistency during compression. By using synonymous latent representations, the model progressively decodes and recovers image details, ensuring high-quality reconstruction, especially at low bitrates. The loss function combines rate-distortion loss with a partial semantic KL-divergence loss to optimize semantic integrity. While comparable to direct LPIPS optimization in terms of perceptual quality, this method offers advantages in preserving semantic consistency and flexibility in quality recovery.\n\n## update after rebuttal\nThe authors have already resolved my issues. As a result, I raise the final score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VZKv,\n\nWe sincerely appreciate your recognition of our SVI analysis, especially your understanding of **semantic consistency** and the **crucial potential in multi-modal compression tasks of our SVI theory**.\n\nYour concerns are essential for refining our paper and guiding future research. Below are our responses.\n\n**[Q1]  The paper derives the Partial Semantic KL Divergence, emphasizing its role in maintaining semantic consistency. However, the final training loss (equation 12) is almost identical to existing Rate-Distortion-Perception (RDP) frameworks.**\n\n[A1] You may misunderstand the role of the Partial Semantic KL Divergence: It optimizes image samples in the reconstructed synset within the image space conditioned on **ensuring explicit semantic consistency in the latent space**. This will **finally result in implicit semantic consistency with the original image, aligning with the ideal synset**, as shown in Figure 1 of the manuscript.\n\nAs for the final loss function, which is almost identical to the existing RDP framework, there are two misunderstandings:\n\n- **The object of coding rates**: Only contains the **common features** of all samples in the reconstructed synset, which is **partial** latent features between the original image and any reconstructed image. This can be intuitively illustrated as the Venn Diagrams [Figure 1] in our [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C/). So the rate term is actually different from the existing RDP methods.\n  \n- **Almost identical means compatibility**: In Appendix A.3 (especially Lines 935~967), we show that SVI analysis is compatible with both the RDP and RD frameworks, as both are special cases of SVI under specific conditions. This leads to our training loss being almost identical to that of existing RDP schemes. However, this is not a weakness but **an advantage**—**our theory effectively explains the theoretical foundations of the existing RDP framework**. This is also mentioned by **Reviewer 7ghf** as our **strengths**.\n\n**[Q2] The experimental results raise concerns about whether the method provides any meaningful improvement.**\n\n[A2] We apologize that our implemented model design is relatively rough compared to the completeness of our theoretical analysis. The model is implemented to verify the effectiveness and potential of our SVI theory. To surpass existing methods, further exploring the model design is necessary, such as **better detail prediction and sampling module** to fit the true distribution, and **better choice on perceptual optimize direction**.\n\nTo verify the potential of further perceptual optimization, we implement a discriminator to finetune our model. The results are available at the [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C) [Figure 5]. Please check the rebuttal to **reviewer 7ghf [A2]** for the relevant experimental configuration and results analysis.\n\n**[Q3] Can the authors provide quantitative evidence that their method maintains better semantic consistency across bitrates?**\n\n[A3] Your suggestion is crucial. Verifying semantic consistency is essential as the ultimate goal of our solution optimization. To verify semantic consistency between samples in the reconstructed synset and original images across bitrates, we conducted the following supplementary tests on both $M=1$ and $M=5$ models:\n\n1. Set the synonymous level $l$ to obtain the reconstructed samples $\\tilde{\\boldsymbol{x}}_i^{(l)}$.\n  \n2. Sent back $\\tilde{\\boldsymbol{x}}_i^{(l)}$ to $g_a$ and get a new synonymous representation ${\\hat{\\boldsymbol{y}}'}_s^{(l)}$.\n  \n3. Compare ${\\hat{\\boldsymbol{y}}'}_s^{(l)}$ with the original $\\hat{\\boldsymbol{y}}_s^{(l)}$ from the original image $\\boldsymbol{x}$ and calculate the numerical difference ratio (Labeled as \"DiffRatio\").\n\nThe results on Kodak are plotted in the synonymous link **[Table 1]** and **[Table 2]**. These results show good semantic consistency of our models, as the DiffRatios across every bitrate are very low (means consistency higher than 98%). However, this also means the ultimate optimization goal of SVI is not finally achieved because of the rough design of our implemented model. One of the most intuitive and objective reasons is that errors arise from the **nonlinearity** and **irreversibility** of the adopted neural network structure. This suggests that in our future work, we can explore using reversible neural network structures to address these issues.\n\nBesides, another observation is the intersection of DiffRatios at different rates for $M=1$ and $M=5$, which mirrors the intersection trend of the performance curve in Figure 5 in our manuscript. This suggests that, in future research, we can try to address the issues in Figure 5 by resolving the problems in DiffRatio.\n\nIn summary, we sincerely thank you for your suggestions and will update the relevant content in the revised manuscript.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have carefully read the proofs for theory in the article, and they are all reasonable.", "experimental_designs_or_analyses": "Please refer to weakness in ‘Other Strengths And Weaknesses’.", "relation_to_broader_scientific_literature": "This paper builds on prior work in perceptual image compression, progressive coding, and semantic-aware latent representations. It extends perceptual compression techniques such as LPIPS-optimized methods by introducing synonymous latent representations to enhance semantic consistency across compression levels. Additionally, it relates to progressive coding approaches but differentiates itself by focusing on gradual semantic detail recovery rather than purely hierarchical bitstream refinement. However, its final training loss remains similar to existing rate-distortion-perception (RDP) methods, raising questions about its practical advantages over LPIPS-optimized compression.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strength：\n1. Unlike conventional perceptual image compression methods that primarily focus on pixel-wise reconstruction, this work leverages synonymous latent representations to ensure semantic consistency between the compressed and original images. This helps retain core image features while allowing for flexibility in fine details.\n2. By focusing on semantic information, the proposed approach is valuable for multi-modal compression tasks, such as image-text joint encoding, where maintaining cross-modal consistency is crucial.\n\nWeaknesses\n1. In Section 3.2, the paper derives the Partial Semantic KL Divergence, emphasizing its role in maintaining semantic consistency. However, the final training loss(equation 12) almost identical to existing Rate-Distortion-Perception (RDP) frameworks.\n2. The experimental results show that the proposed method performs almost identically to direct LPIPS-optimized methods in perceptual metrics (LPIPS, DISTS). This raises concerns about whether the method provides any meaningful improvement.\n3. The paper claims that synonymous latent representations improve semantic consistency, ensuring that images at different bitrates retain the same semantic information. However, there is no quantitative validation of this claim in the experiments. The experiments only report perceptual metrics (LPIPS, DISTS), which do not directly assess semantic consistency. Can the authors provide quantitative evidence that their method maintains better semantic consistency across bitrates?", "other_comments_or_suggestions": "Refer to weakness in ‘Other Strengths And Weaknesses’.", "questions_for_authors": "Refer to weakness in ‘Other Strengths And Weaknesses’.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNTDIbWKPj", "reviewer": "Reviewer_7ghf", "summary": "This paper presents a novel perspective to analyze the perceptual image compression problem, which is based on the notion of synonymy in semantic information theory, which suggests that images with perceptual similarity constitute a synonymous set. Based on this, the authors propose a synonymous variational inference method (SVI) and theoretically demonstrate that the optimization direction of perceptual image compression follows a three-pronged trade-off that encompasses bit rate, distortion, and perception, which is in line with existing related research. In addition, the paper designs a new image compression scheme, synonymous image compression (SIC), and verifies its effectiveness experimentally.\n\n## update after rebuttal\nThe author answered my questions, so I upped my score. I hope the author will make the code public in the future to promote the community.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7ghf,\n\nWe sincerely appreciate your high regard for our work, especially your recognition of our work including:\n\n- **New theoretical perspective on perceptual image compression**\n  \n- **Mathematically supported unified theory**, which can **extends and unifies the existing RD and RDP theories**\n  \n- **Holding important academic value**.\n\nYour main concern is the unsatisfactory experimental results, which many reviewers have also raised. Below are our responses to the relevant questions.\n\n**[Q1] How do the authors view the performance gap between the SIC methods and the advanced RDP methods in terms of DISTS?**\n\n[A1] We believe the insignificant performance of our method on DISTS is due to multiple factors:\n\n- **The choice of perceptual loss.** Your understanding of using LPIPS instead of KL divergence is accurate. With LPIPS replacing KL divergence as the perceptual loss, our scheme outperforms MS-ILLM (no-GAN) on DISTS mainly due to DISTS' resampling tolerance, which aligns with our method's random detail sampling. However, the advantage is limited. Previous work (MS-ILLM) shows that using adversarial loss significantly improves DISTS quality. Therefore, we believe that fine-tuning our model with a discriminator and adversarial loss can help bridge this gap to some extent.\n  \n- **Confronts between multiple synonymous levels**. To ensure each synonymous level in the progressive SIC model is effective, we use Equation (12) (line 319) for alternating training. This creates battles between different optimization directions within a single SIC model, inevitably impacting overall performance.\n  \n- **Suboptimal choice of sampling mechanism**. In the detail representation part, we empirically use uniform sampling, which may cause significant differences between the distribution of the reconstructed and real images.\n\nTherefore, future research should combine theoretical analysis and method design to address these issues.\n\n**[Q2] Future plans to integrate discriminative mechanisms or adversarial training to narrow this gap while still adhering to the semantic information theory?**\n\n[A2] In fact, this is also a concern of many reviewers. To this end, we tried to build a discriminator model based on HiFiC’s discriminator structure and fine-tuned our $M=1$ and $M=5$ models with non-saturating loss for $2×10^5$ steps. We plot the discriminator structure in [Figure 4] and the fine-tuned performance curves in [Figure 5] (PSNR, LPIPS, DISTS, and KID) of the [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C/). The supplementary results show that:\n\n- **The perceptual quality (DISTS, FID) has improved**, with more noticeable gains at higher bitrates, and DISTS gradually approaches the performance of MS-ILLM (with GAN). Besides, while DISTS and FID values change little at low rates, the visual quality improves significantly, as shown in [Figure 6] ([Table 3] corresponds to the bitrates in [Figure 6]).\n  \n- **The distortion has degraded,** which can verify the distortion-perception tradeoff as mentioned in (Blau & Michaeli, 2018).\n  \n- **The gap compared to HiFiC and MS-ILLM is still obvious,** especially on FID. This may be due to insufficient fine-tuning with multiple synonymous layers battling against each other. Besides, this also means that the issues stated in [A1] need to be solved in the future, including designing better perceptual optimization directions, broadening SVI theory to address multi-synonymous level confronts, and improving detail prediction and sampling methods to fit the true distribution.\n\n**[Q3] Concerns about the correlation between synset and \"JND\"?**\n\n[A3] This comment is crucial for completing our explanation of the synset. JND's ideas are somewhat similar to ours. However, while JND provides the minimum stimulus difference that causes a perceptible change, which may help us to determine a certain synset threshold to keep human-perceptual consistency, our concept of synset takes it a step further:\n\nIn certain cases, only ultra-low bitrates are sent to the decoder, where the reconstructed image can differ significantly from the original image—beyond the JND threshold—and still be tolerable to people in that context.\n\nA typical example is the use of diffusion models for ultra-low bitrate image compression, where the reconstructed images exhibit significant differences from the originals, yet maintain good perceptual quality that is acceptable to humans.\n\n**[Q4] \"Covering\" is mathematically supported, but not mean that all existing programs can be fully explained or guided by the theory.**\n\n[A4] You are correct, but this also means there is room for further development of our theory. We believe the SVI theory presented in this manuscript is just the beginning, and future research will involve theoretical explanations of various existing methods.\n\nIn summary, thanks for your valuable comments and suggestions. We will update our manuscript accordingly.", "claims_and_evidence": "The analysis proves theoretically that the optimization direction of perceptual image compression follows a triple tradeoff, i.e., a synonymous rate-distortion-perception tradeoff, and that this tradeoff can cover existing rate-distortion-perception schemes. Theorem 3.3 indeed gives a theoretical formulation of this triple tradeoff (Eq. 9), and Appendix A.2 provides a detailed proof procedure. Appendix A.3 also discusses compatibility with existing schemes. However, “covering” existing programs may mean that existing programs are special cases of the theory, which is mathematically supported. However, this does not necessarily mean that the design concepts of all existing programs can be fully explained or guided by the theory.", "methods_and_evaluation_criteria": "1) The SVI framework proposed in the paper provides a new theoretical foundation for perceptual image compression from the perspective of semantic information theory.\n2) The SIC compression scheme is a concrete realization based on this theory, and its design concept is consistent with the core idea of SVI.\n3) The design of progressive SIC takes into account the needs of practical applications and allows the generation of images with different perceptual qualities at different rates.\n4) The selection of commonly used benchmark datasets and an evaluation system that includes both traditional metrics (PSNR) and perceptual metrics (LPIPS, DISTS) enables a comprehensive evaluation of the performance of the proposed method and facilitates the comparison with existing techniques. The authors particularly emphasize the DISTS metrics, which are related to their proposed approach based on tautology.", "theoretical_claims": "1) Proof of Lemma 3.2: This lemma states the equivalence between minimizing the expected negative logarithmic tautological likelihood term and minimizing the weighted expected distortion term and the weighted expected KL scatter term. A detailed proof of the lemma is given in Appendix A.1.\n2) Proof of Theorem 3.3: The theorem presents a formula for the minimum achievable rate of perceptual image compression, i.e., the tautology rate-distortion-perceptual tradeoff. The detailed proof of the theorem is given in Appendix A.2.", "experimental_designs_or_analyses": "1) Use of LPIPS instead of KL scatter: although the authors explain the use of LPIPS instead of KL scatter due to computational challenges, LPIPS is, after all, a depth-feature based perceptual metric that is not mathematically equivalent to the KL scatter in the theoretical derivation. This may affect the extent to which the experimental results directly validate the theory.\n2) Simplicity of the synonym level division: a uniform division of the channel dimensions is used in the paper to define the different synonym levels. The effectiveness and optimality of this approach may require further exploration and experimental validation. More complex or data-driven based segmentation strategies may be able to better capture different levels of semantic information.\n3) Setting of hyperparameters: the paper mentions that the hyperparameters in the loss function are configured empirically. The selection of these hyperparameters is crucial to model performance, but the lack of systematic hyperparameter search and analysis may affect the robustness and reproducibility of the results.\n4) Impact of different numbers of samples (M): Fig. 5 illustrates a comparison of Progressive SIC performance using different numbers of samples M (M=1 and M=5) in the reconstructed synonym set X̂. The results show that different M values perform inconsistently at different code rates, sometimes M=1 is better and sometimes M=5 is better. The authors recognize that this may be related to the mechanism of synonym level division, number of samples, and insufficient setting of hyperparameters, which need to be further explored in future work.\n5) Interpretation of the visualization results: although Figs. 11-16 show the visual effects of the reconstructed images, a more in-depth analysis of these visualization results, such as the changing patterns of image details and perceived quality under different synonym levels, and the effects of different M values on the diversity and quality of the generated images may be more helpful in understanding the performance of SIC.", "relation_to_broader_scientific_literature": "1) Links between synonymous variational inference (SVI) and semantic information theory.\n2) The SVI connection to rate-distortion-perception (R-D-P) theory.\n3) The difference and connection between Synonymous Image Compression (SIC) and Learning Image Compression (LIC).", "essential_references_not_discussed": "Research on “Perceptual Equivalence” or “Just Noticeable Difference (JND)” in image compression: In this paper, perceptual similarity is considered as a criterion for tautological relationships. In the field of image processing and computer vision, there has been researched on when two images are perceptually “identical” or “indistinguishable”, such as image compression methods based on the JND model. These models attempt to identify perceptually unimportant information and remove it, thereby improving compression rates while maintaining perceptual quality. Although JND models do not directly correspond to the concept of “synonym set”, they are concerned with the sensitivity of the human perceptual system to image differences, which can complement the understanding of the theoretical basis of the “perceptual similarity” criterion in this paper.", "other_strengths_and_weaknesses": "Pros:\n1) The originality is highlighted by the innovative application of the concept of “tautology” in semantic information theory to the field of perceptual image compression. This not only provides a new theoretical perspective on the problem, but also makes the first attempt to explain, at a theoretical level, the fundamental reason for the existence of distributional distance metrics (e.g., KL dispersion or its substitutes) in perceptual image compression, which stems from the consideration of an ideal set of synonyms.\n2) The paper proposes a novel variational inference method, the Synonymy Variational Inference (SVI), and derives a synonymy rate-distortion-perception ternary trade-off based on this theory. This theoretically extends and unifies the existing rate-distortion and rate-distortion-perception theories by showing that the latter can be regarded as a special case of the former under specific conditions. This theoretical unification has important academic value.\n\nWeaknesses:\n1) The experimental results, although competitive, do not show a significant advantage in terms of perceptual quality over some state-of-the-art GAN-based methods such as HiFiC and MS-ILLM. The authors also acknowledge that this may be related to the direct use of LPIPS instead of KL dispersion in the loss function and propose to explore the discriminative mechanism in the future to further improve the perceptual quality. This suggests that there is still room for improvement in the practical performance of the currently proposed SIC model.", "other_comments_or_suggestions": "There were some problems with the format of the references, for example, some conference papers had “Proceedings” while others did not. In addition, there is no consistency in the case of the names of the conferences covered in the references.", "questions_for_authors": "The experimental results show that the advantage of the proposed method over some advanced GAN-based perceptual image compression methods (e.g., HiFiC and MS-ILLM) in terms of perceptual quality (DISTS metrics) is not significant. Considering that the SVI framework aims to provide a unified perspective on perceptual image compression, how do the authors view this performance gap? How are future plans to integrate discriminative mechanisms or adversarial training within the SVI framework to narrow this gap while still adhering to the semantic information theory based on tautology? A clear articulation of the author's understanding of this gap and promising future research directions would help to enhance the long-term potential and impact of this work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NANSrfKOSx", "reviewer": "Reviewer_KhzF", "summary": "This paper proposes a novel framework for perceptual image compression based on synonymous variational inference (SVI). Specifically, the paper introduces a method to analyze the optimization direction of perceptual image compression using semantic information theory. A new image compression scheme called Synonymous Image Compression (SIC) is devised to encode latent synonymous representations and reconstruct images through sampling. A progressive SIC codec is developed to leverage multiple synonymous levels for variable-rate compression. Besides, to the best knowledge of the authors, this method is the first work that can theoretically explain the fundamental reason for the divergence measure’s existence in existing perceptual image compression schemes. However, it lacks comprehensive comparisons with essential state-of-the-art methods and does not convincingly validate the effectiveness of the proposed models through ablation studies.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KhzF,\n\nThank you for recognizing our work, especially **the originality and novelty of our SVI theory analysis for perceptual image compression** and our **well-structured and clearly presented**.\n\nYour concern centers on our unsatisfactory experimental results, which is also a concerned by many reviewers. However, we want to claim that **our focus is on the SVI analysis theory, which offers a new theoretical framework for perceptual image compression**, while **the implemented SIC model serves only as a preliminary validation**. Compared to the completeness of our theory, our method is indeed relatively rough. This is due to several issues in the theoretical analysis that need further attention in method design, while these issues in turn offer significant potential for our future research.\n\nBelow are our responses to your concerns.\n\n**[Q1] The paper shows some improvements in DISTS and LPIPS metrics, but lacks comprehensive comparisons with state-of-the-art methods like HiFiC and MS-ILLM (with GANs).**\n\n[A1] As several reviewers have raised concerns about this issue, we conducted additional experiments, introduced a discriminator into the model structure, and fine-tuned our model using discriminative loss. We have included some of the results and comparison with HiFiC and MS-ILLM in [Figure 5] of the [[Anonymous Link]](https://anonymous.4open.science/r/supplementaryResults_SVI-F92C) and provided visualizations in [Figure 6]. For detailed configuration, reasoning, and discussion, please check the rebuttal to **Reviewer 7ghf [A2]**.\n\nAlthough our method still has some gaps in perceptual quality compared with HiFiC and MS-ILLM, it offers a clear advantage in deployment cost. Specifically, we can use a single model to achieve all bit rates, with a model size of 465MB. In contrast, an MS-ILLM model can only reconstruct the received information for one bitrate, with a single model size of 693MB. To support all bit rates (e.g., MS-ILLM with GAN, 6 points), you would need 6×693MB = 4158MB = 4.06GB, which is significantly larger than our model. Therefore, in this respect, our model has a clear advantage.\n\n**[Q2] It would be beneficial to include more detailed ablation studies.**\n\n[A2] We added an ablation study on the detail representation part in the provided anonymous link, comparing random sampling with forcing it to 0. Please refer to [Figure 2~3] of the above anonymous link.\n\nWe believe this ablation is important as it highlights the difference between our approach and existing methods (HiFiC and MS-ILLM). Specifically, it shows that random details provide effective information for image reconstruction, making distortion and perceptual quality not entirely dependent on the information provided by the coding sequence.\n\nAdditionally, although an ablation study on different components of SVI (i.e., partial semantic KL divergence, synonymous level partitioning) is essential, we prefer to discuss this issue based on our theoretical results:\n\n- If partial semantic KL divergence is not utilized, the existence of the ideal synset will not be considered (i.e., only consider the original image), which makes the loss function only a rate-distortion tradeoff without perceptual term. This is the core reason why we propose using this divergence for analysis: **It provides a sufficient theoretical foundation for using distribution distance as a perceptual loss.** This can be referred to Appendix A.3 (especially Lines 949~967) in our manuscript.\n  \n- If synonymous level partitioning is not utilized, the model should be a single-point model, which only accepts a specific coding rate, similar to how HiFiC and MS-ILLM operate. In this case, the performance of each point in our method has the potential to be improved, as there is no competition between different synonymous levels, as illustrated by the tradeoff loss in Equation (12). However, the disadvantage of this approach is that it increases the cost of deployment, as a separate model needs to be deployed for each bitrate.  \n\n**[Q3] It would be useful to include additional qualitative results.**\n\n[A3] In [figure 5] of our provided anonymous link, we added performance curves of our model (no-GAN and with GAN) for **FID** and compared the results with HiFiC and MS-ILLM (with GAN). Although the performances under FID are still unsatisfactory, we analyze the underlying causes and consider it an essential direction for future research. Please check the rebuttal to **Reviewer 7ghf [A2]** for this discussion.\n\n**[Q4] More results about the progression of image quality at different synonymous levels are required.**\n\n[A4] Thanks for your suggestion. We added a progression of images at different synonymous levels in [Figure 7~8] in the anonymous link, in which [Figure 7] shows gradual changes of the images by our $M=1$ model, and [Figure 8] by our $M=5$ model.\n\nIn summary, we sincerely thank you for your suggestions. These suggestions will help us improve our paper.", "claims_and_evidence": "Claim: Experimental results demonstrate comparable rate-distortion-perception performance using a single neural progressive SIC image codec, thus verifying our method’s effectiveness.\nEvidence: The paper shows some improvements in DISTS and LPIPS metrics, but lacks comprehensive comparisons with state-of-the-art methods like HiFiC and MS-ILLM (with GANs).\nEvaluation: This claim is not fully convincing due to the limited comparison. More extensive experiments with diverse datasets and state-of-the-art methods are needed.", "methods_and_evaluation_criteria": "No, the proposed methods and evaluation criteria do not fully align with the problem or application at hand. While the paper introduces a novel approach to perceptual image compression from the perspective of Synonymous Variational Inference (SVI) and theoretically explores the fundamental reason for the divergence measure’s existence in existing schemes, the experimental results fail to demonstrate sufficient competitiveness in practical applications. The evaluation criteria, though relevant to perceptual quality, do not comprehensively validate the effectiveness of the proposed method compared to state-of-the-art techniques. Therefore, the proposed methods and criteria are insufficient to address the practical needs of perceptual image compression.", "theoretical_claims": "Claim: The proposed Synonymous Variational Inference (SVI) method is the first to theoretically explain the fundamental reason for the divergence measure’s existence in existing perceptual image compression schemes.\nProof: The authors provide a theoretical analysis showing that the divergence measure arises naturally from the consideration of synonymous sets and semantic information theory. They introduce the concept of partial semantic KL divergence and demonstrate its relevance in the context of perceptual image compression.", "experimental_designs_or_analyses": "Yes, I have reviewed the experimental designs and analyses presented in the paper. The authors aim to validate the effectiveness of their proposed Synonymous Variational Inference (SVI) method and the corresponding Synonymous Image Compression (SIC) scheme.\n1.The choice of datasets and metrics is appropriate for evaluating the performance of the proposed method. These datasets are widely used in the field, ensuring that the results are comparable to other studies.\n2. While the paper provides comparisons with several methods, the results for some state-of-the-art perceptual image compression methods (e.g., HiFiC and MS-ILLM with GANs) are not comprehensive. This limits the strength of the claims regarding the superiority of the proposed method.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to several areas of the broader scientific literature, particularly in the fields of image compression, semantic information theory, and variational inference. It introduces novel ideas such as synonymous sets and partial semantic KL divergence, which offer new perspectives on handling perceptual similarity in image compression.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1.Originality and Novelty: The paper introduces a novel perspective on perceptual image compression by leveraging semantic information theory and synonymous variational inference (SVI). To the best knowledge of the authors, this method is the first work that can theoretically explain the fundamental reason for the divergence measure’s existence in existing perceptual image compression schemes.\n2.Clarity and Presentation:\nThe paper is well-structured and clearly presents the theoretical concepts, methods, and experimental results. The use of visualizations and detailed explanations helps in understanding the complex ideas and their practical implications.\n\nWeaknesses:\n1.Performance: The paper shows some improvements in DISTS and LPIPS metrics, but lacks comprehensive comparisons with state-of-the-art methods like HiFiC and MS-ILLM (with GANs).", "other_comments_or_suggestions": "1.The paper provides comprehensive experimental results, but it would be beneficial to include more detailed ablation studies. For example, analyzing the impact of different components of the proposed method (e.g., the effect of learnable weights, the choice of architecture) could provide deeper insights into the method's effectiveness.\n2.The description of the SIC scheme is thorough, but it could benefit from additional visualizations showing more results about the progression of image quality at different synonymous levels. This would help readers better understand the practical implications of the method.\n3.The paper provides several performance metrics (PSNR, LPIPS, DISTS), but it would be useful to include additional qualitative results.", "questions_for_authors": "1.Could the authors provide a more detailed comparison with state-of-the-art perceptual image compression methods, particularly those using GANs or other advanced techniques like HiFiC and MS-ILLM? How does the proposed SVI method differ from these methods in terms of performance and computational efficiency?\n2.Could the authors provide additional ablation studies to validate the theoretical claims made in the paper? For example, how does the performance change when different components of the SVI method (e.g., partial semantic KL divergence, synonymous level partitioning) are removed or modified?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a theoretical framework for perceptual image compression grounded in semantic information theory, introducing the concept of synonymous variational inference (SVI). It further presents an implementation called SIC, and evaluates its performance across standard datasets.
Several reviewers pointed out that while the empirical performance compared to existing methods like GAN are limited, theoretical formulation is clear and well-motivated and extends existing rate distortion perception frameworks in a principled manner. The authors provided additional experiments and clarifications in response to several concerns, including analysis of semantic consistency and comparisons under various metrics.
Although some practical aspects of the implementation might be improved, the theoretical contribution is clear and seem to be useful for the community. Based on these, I recommend acceptance.
Decision: Accept (poster)