Universal Rate-Distortion-Perception Representations for Lossy Compression
Authors: George Zhang
Jingjing Qian
Jun Chen
Ashish J Khisti
Venue: NeurIPS 2021 Poster (2021)
Keywords: Information Theory
Lossy Compression
Image Compression
Machine Learning

Abstract: In the context of lossy compression, Blau \& Michaeli (2019) adopt a mathematical notion of perceptual quality and define the information rate-distortion-perception function, generalizing the classical rate-distortion tradeoff. We consider the notion of universal representations in which one may fix an encoder and vary the decoder to achieve any point within a collection of distortion and perception constraints. We prove that the corresponding information-theoretic universal rate-distortion-perception function is operationally achievable in an approximate sense. Under MSE distortion, we show that the entire distortion-perception tradeoff of a Gaussian source can be achieved by a single encoder of the same rate asymptotically. We then characterize the achievable distortion-perception region for a fixed representation in the case of arbitrary distributions, and identify conditions under which the aforementioned results continue to hold approximately. This motivates the study of practical constructions that are approximately universal across the RDP tradeoff, thereby alleviating the need to design a new encoder for each objective. We provide experimental results on MNIST and SVHN suggesting that on image compression tasks, the operational tradeoffs achieved by machine learning models with a fixed encoder suffer only a small penalty when compared to their variable encoder counterparts.

Universal Rate-DistortionPeerrptio Representations for Lossy Compression
George Zhang
Jingjing Qian
Electrical and Computer Engineering University of Toronto
Electrical and Computer Engineering
McMaster University qi an 40@cmasterrrrrr...
g9 zhang@mail utwronto.coom ca
Jun Chen
Ashish Khisti Electrical and Computer Engineering University of Toronto akhisti@ece. utoronto... ca
Electrical and Computer Engincering
McMaster University chen peenjuunmcmmsttrr.ccm
Abstract
In the context of lossy compression, Blau & Michaeli [5] adopt mathematical notion perceptual quality and define the information rate-distortion- -etorttionneeecettion function, generalizing the classical rate-distortion tradeoff. We consider the notion of universal representations in which one may fix an encoder and vary the decoder achieve any point within collection of distortion and perception constraints. We prove that the corresponding information-theomm perception function operationally achievable in an approximate sense. Under MSE distortion, we show that the entire distortion-peree perrcpttoon tradeoff' of a Gaussian source can achieved by a single encoder the same rate asymptotically. We then characterize the achievable distortion-percant perception region for fixed representation in the case of arbitrary distributions, and identify conditions under which the aforementioned results continue to hold approximately. This motivates the study of practical constructions that are approximately universal across the RDP tradeoff, thereby alleviating the need to design new encoder for each objective. We provide experimental results on MNIST and SVHN suggesting that on image compression tasks, the operational tradeoffs achieved by machine learning models with fixed encoder suffer only a small penalty when compared to their variable encoder counterparts.
universal rate-distortion-
Introduction
Unlike in lossless compression, the decoder in a lossy compression system has flexibility in how to reconstruct the source. Conventionally, or SSIM/MS-SSIN [36, 37] is used as quality measure. Accordingly, lossy compression algorithms are analyzed through rate-distortion theory, wherein the objective to minimize the amount of distortion for a given rate. However, it has been observed that low distortion is not necessarily synonymous with high perceptual quality; indeed, deep learning based image compression has inspired works in which authors have noted that increased perceptual quality may come at the cost of increased distortion [1, 4]. This culminated in the work of Blau & Michaeli [5] who propose the rate-distortion riie-distortion-pereppioo perception thcoretical framework.
measure of distortion such mean squared error, PSNR
The main idea was to introduce a third perception axis which more closely mimics what humans would deem be visually pleasing. Unlike distortion, judgement of perceptual quality is taken to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

be inherently no-reference. The mathematical proxy for perceptual quality then comes in the form of a divergence between the source and the reconstruction distributions, motivated by the idea that perfect perceptual quality is achieved when they are identical. Leveraging generative adversarial networks [11] in the training procedure has made such a task possible for complex data-driven settings with efficacy even at very low rates [33]. Naturally, this induces a tradeoff between optimizing for perceptual quality and optimizing for distortion. But in designing a lossy compression system, one may wonder where exactly this tradeoff lies: is the objective tightly coupled with optimizing the representations generated by the encoder, or can most of this tradeoff be achieved by simply changing the decoding scheme?
Our contributions are as follows. We define the notion of universal representations which are generated by fixed encoding scheme for the purpose of operating at multiple perception-distortin tradeoff points attained by varying the decoder. We then prove a coding theorem establishing the relationship between this operational definition and an information universal rate-distortion- perception function. Under MSE distortion loss, we study this function for the special case of the Gaussian distribution and show that the penalty in fixing the representation map with fixed rate can be small in many interesting regimes. For general distributions, we characterize the achievable distortion- perception region with respect to an arbitrary representation and establish a certain approximate universality property.
We then turn to study how the operational tradeoffs achieved by machine learning models on image compression under a fixed encoder compared to varying encoders. Our results suggest that there is not much loss in reusing encoders trained for a specific point on the distortion-percenn across other points. The practical implication of this is to reduce the number of models to be trained within deep-learning enhanced compression systems. Building on [30, 31], one of the key steps in our techniques is the assumption of common randomness between the sender and receiver which will turn out to reduce the coding cost. Throughout this work, we focus the scenario where a rate is fixed in advance. We address the scenario when the rate is changed in the supplementary.
tradeoff
2 Related Works
Image quality measures include full-reference metrics (which require ground truth as reference), or no-reference metrics (which only use statistical features of inputs). Common full-reference metrics include MSE, SSIM/MS-SSMM [36. 371, PSNR or deep feature based distances [16, 41]. No-reference metrics include BRISQUE/NTQE/IIQE [23, 24, 35) and Fréchet Inception Distance [14]. Roughly speaking, one can consider the former set to be distortion measures and the latter set to be perception measures in the tite-dittttionn-g distortionn-errcetion framework. Since GANs capable of synthesizing highly realistic samples have emerged, using trained discriminators as a proxy for perceptual quality in deep learning based systems has also been explored [17]. This idea is principled as various GAN objectives can be interpreted estimating particular statistical distances [3, 25, 26).
Rate-distortion theory has long served as theoretical foundation for lossy compression [7]. Within machine learning, variations of rate-distortion theory have been introduced to address representation learning [2, 6, 34], wherein central task is to extract useful information from data on some sort of budget, and also in the related field of generative modelling [15]. On the other hand, distribution- preserving lossy compression problems have also been studied in classical information theory literature [27, 28, 40].
More recently, in an effort to reduce blurriness and other artifacts, machine learning research in lossy compression has attempted to incorporate GAN regularization into compressive autoencoders [1, 33], which were traditionally optimized only for distortion loss 32). This has led to highly successful data-driven models operating at very low rates, even for nigh-resolution images [22]. An earlier work of Blau & Michaeli [4] studied only the pnecection-distorrin tradcoff within deep learning enhanced image restoration using GANs. This idea was then incorporated with distribution- preserving lossy compression [33] to study the weedditttoionperreeppee tradcoff in full generality [5].
The work most similar to ours is [39], who observe that an optimal encoder for the "classic" rate- distortion function is also optimal for perfect perceptual compression at twice the distortion. Our work investigates the intermediate regime and also includes common randomness modelling
2

---

Rate (bits)
06
0.00
0.5
0.0
0:08
04
Ponepption Los (w2)
R 0.25
0.60
0.6
R 2.00
05
02
0.4
0.5
0.5
0.2
1.5
D
0
0
0.5
1.5
Distortion Lose (MSE)
(a)
(b)
Figure 1: (a) The information Deit---tiin--ereeeeeee function R(D, P) for standard Gaussian source X. (b) Distortion-percepee
cross-sections across multiple rates. The tension between perception and distortion is most visible at low rates. When both P and D) are active, Theorem 3 implies that the rate needed to achieve an entire cross-section along fixed rate the same as the rate to achieve any single point on the cross-section in the asymptotic setting.
assumption, which in principle allows us to achieve perfect perceptual quality at lower than twice the distortion. The concurrent work [10] also establishes the achievable distortion- peteepionpe region as in our Theorem 4 and provides geometric interpretation of the optimal interpolator in Wasserstein space.
3 Siite-Distortion-Peeepttt Representations
The backbone of rate-distortion theory characterizes an (operational) objective expressing what can be achieved by encoders and decoders with quantization bottleneck terms of an information function which is more convenient to analyze. Let X ~ Px be an information source to be compressed through quantization. The quality of the compressed source is measured by distortion function A X' x X' + R2 satisfying (xx,t) = 0 if and only if x â. We distinguish between the one-shot scenario in which we compress one symbol at time, and the asymptotic scenario which we encode i.i.d. samples from X jointly and analyze the behaviour needed to meet the distortion constraint D on average is denoted by R* (D) in the one-shot setting and by (D)))(D) in the asymptotic setting. These are studied through the information rate-distortion function
+ Co. The minimum rate
R(D) = inf I(X; P
s.t. B(A(XXX)) D,
(1)
where I(X; X) is the mutual information between a source and reconstruction X. The principal result of rate-distortion theory states that ((=)(()) = R(D) [7]. Furthermore, it is also possible to characterize R (D) using R(D)
will soon
In light of the discussion perceptual quality, the flexibility in distortion function is not necessarily a good method to capture how realistic the output may be perceived. To resolve this, Blau & Michaeli [5] introduce an additional constraint match the distributions of X and X in the form of non- negative divergence between probability measures d(-,-) satisfying d(p, g) 0 if and only if g. The one-shot pate-disttrtionnprrepee
function Rº (D, P) and asymptotic cec-disttortion-eeeeptie function R((e) [D, P) are defined in the same fashion as their rate-distortion counterparts, which we will later make precise.
Definition (iRDPF). The information rale-distootioonperropiii function for source X is defined as
R(D. = inf I(X;X) PRIX
s.t. BJAXX, X)] DD d(px,Px) SP.
3

---

The strong functional representation lemma [19, 31] establishes relationships between the operational and information functions:
R! PD, R(D,P) & R(D, P)
(3)
These results hold also for R(D) = R(D, o0). We make note that they were developed under more general set of constraints for which stochastic encoders and decoders with shared source of randomness were used. In practice, the sender and receiver agree on a random seed beforehand to emulate this behaviour.
3.1 Gaussian Case
We now present the closed form expression of R(D, P) for Gaussian source under MSE distortion and squared Wasserstein-2 perception losses (see also Figure 1(a) and Figure 1(b)). Recall that the squared Wasserstein-2 distance defined as
W2 = inf inf - 11),
(4)
where the infimum over all joint distributions of (X, X) with marginals px and px- Let and x E||| -x]
[
Theorem 1. For scalar Gaussian source X N ox), the information -ittoristottion- perception function under squared error distortion and squared Wasserstein-2 distance attained by some jointly Gaussian with X and is given by
03x /Pp
2 log
1ax
R(D.P)=
is V/P S ox v 10x -D] VVP> ox - D).
of max ( log
D 0}
When /P> ox
V DJ, the perception constraint is inactive and R(D, P) R(D). The choice of (-(-.)) perception loss turns out to not be essential; we show in the supplementary that R(D, P) can also be expressed under the KL-divergence.
3.2 Universal Representations
Whereas the RDP function is regarded as the minimal rate for which we can vary an encoder-decoder pair meet any distortion and perception constraints (D, P), the universal RDP (uRDP) function generalizes this to the case where we fix an encoder and allow only the decoder to adapt in order to meet multiple constraints (D, P) E O. For example, one case of interest is when θ is the set of all (D, P) pairs associated with given rate along the iRDP function; how much additional rate needed if this is be achieved by a fixed encoder, rather than varying across each objective? The hope is that the rate to use some fixed encoder across this set is not much larger than the rate to achieve any single point. As we will see for the Gaussian distribution, this is in fact the case in the asymptotic setting, and also approximately true in the one-shot setting. Below, we define the one-shot universal rate-distortion percciotonnneeiionnnnnn function and the information universal rate-distortion rale-distortionn-errcppooo perception function, then establish relationship between the two. In these definitions assume X is random variable and θ is an arbitrary non-empty set of (D, pairs.
Definition 2 (ouRDPF). O-universal encoder of rate is said exist if we can find random variable U, encoding function fo X Cy and decoding functions DUDD.P Cu - &, (D, P) e such that
E (X)) S ED(
) SP.
where Co is a uniquely decodable binary code specified by U. Åp, (f.D. and ((v((X) denotes the length binary codeword (X). The random variable U acts as a shared source of randomness. The infimum of such R is called the one-shot universal rate-distortion- perception function (ouRDPF) and denoted by R (0). When = ((D, P)), this specializes the one-shot petiistrtiio-perreptii function Rd (D,P).

---

Definition 3 (iuRDPF). Let Z be representation of X (i.e. generated by some random transform Pz|x). Let (2) be the set of transforms x x such that for each (D, P) O, there exists Pi for which
[|A EIA(X,XD,P)) < D and d(px, PlpxpPXx.ee P:
where X ++ Z ++ X are assumed to form a Markov chain. Define
R(O)
P2Ix IX:
inf
(5) We refer to this as the information universal rate-distortion- distortion-perception function (iuRDPF) and say that the random variable distributions Pxp,7 P z induce stochastic mappings transforming the representations to reconstructions XD. in order to meet specific (D, P) constraints.
a representation which is --universal with respect to X. The conditional
Note that we assume shared source of stochasticity within the ouRDPF as tool to prove the achievability of the iuRDPF, but within the definition of the iuRDPF itself. Moreover, source X, reconstruction DD, representation Z, and random seed are all allowed to be multivariate random variables.
Theorem 2. R(O)= & (O) S R(O) + log(R( (()) + +)+
In practice, the overhead log [[(0) + 1) + either makes the upper bound an overestimate of It (O) is negligible compared to R(O). This overhead vanishes completely in the asymptotic setting as we will show in the supplementary. We can therefore interpret R(@) as the rate required to meet an entire set é of constraints with the encoder fixed. Within the set θ, it is clear that sup(D.P)Es R(D, characterizes the rate required to meet the most demanding constraint. Now define
^(0)= R(O)
sup R(D, P);
(6)
(D,P)E0
which is the rate penalty incurred by meeting all constraints in θ with the encoder fixed. Let ((R) = P) R(D,P) S. R). It is ideal to have A(S((R) 0 for each R so that achieving the entire tradeoff with single encoder is essentially no more expensive than achieve any single point on the tradeoff, thereby alleviating the need to design host of encoders for different distortion- perception objectives with respect to the same rate.
One can also take the following alternative perspective. The proof of Theorem 2 shows that every representation Z can be generated from source X using an encoder of rate T(X; Z) + o([(X; Z)), and based on Z, the decoder can produce reconstruction PD. by leveraging random seed U to simulate conditional distribution Pxp. Therefore, the problem of designing an encoder boils down to identifying a suitable representation. Given a representation Z of X, we define the achievable distortion perriiioo-pprceptioinnoo region ((pzix) the set all (D, P) pairs for which there exists P Px,, r|z such that E[A(X, &D, < D and ((px.Pxp,p) S P. Intuitively, s(pz)x is the set of all possible distortion perrreption constraints that can be met based on representation Z. If [(pz)x) = (2()) for some representation Z with T(X; Z) = R, then has the maximal achievable distortion distortion--pereetioon region in the sense that M(z2'x) S M(pZ)x) for any Z' with [(X; 2') < R. In the supplementary material we establish mild regularity conditions for which the existence of such is equivalent to the aforementioned desired property A(I(R)) O. We shall show that that this ideal scenario actually arises in the Gaussian case and an approximate version can be found more broadly.
Theorem Let X ~ Nux xzx) be a scalar Gaussian source and assume MSE and ( losses. Let θ be any non-emply set of (D, P) pairs. Then
A(O) = 0.
Moreover, for any representation Z jointly Gaussian with such that
1(x;2)= = sup R(D,P) P), (D,P)EB
(8)
we have
+ C (((pz)) 2(I(X;
(9)
5

---

Next we consider a general source X ~ Pxx and characterize the achievable perreettoonn region for an arbitrary representation Z under MSE loss. We then provide some evidence indicating that every reconstruction D, achieving some point (D, P) on the distortion pprrrettooo tradeoff for given It likely has the property ((Px,.rIX) R ((.)...
Theorem 4 (Approximate universality for general sources), Assume MSE loss and any perception measure d(.,)). Let Z be any arbitrary representation of X. Then
((p2))) & (D. D R
x1 inf (x:d(pxp2)(()
WxPx c((((2))))
where X- | the reconstruction minimizing squared error distortion with X under the representation Z and cl(.) denotes set closure. In particular, the two extreme points (Día), p(al) = [EIJI XI'),, d(px.Px)) and (D(6), p()) (EIlIx XI + px)), 0) are contained in c(((())))))
To gain a better understanding, let Z be an opti- mal reconstruction XD. associated with some point P) on the distortion-perception tradcoff for a given R. i.e., I(X; xp = P) = R (as- suming that D and/or P cannot be decreased without violating P) R), E[||X =1' D, (PPx-PXn,,7 = P. We assume for simplicity that D. exists for every (D, P) on the tradeoff.
82(R)
an(pxo,.
a0(Pxe,...x
Perception
piol)
(Di.P)
Such (D, P) is on the boundary of cl(2(PXx,. xx)x))))
Theorem 4 indicates that cl( r(x)) contains
Ps)
two extreme points: the upper-left (D(a), p(a)) and the lower-right (D(b) p( Under the assump- tion that d(::-) is convex in its second argument, cl( ) is a convex region containing the aforementioned points.
(Ds,
(DOM, p()))
Distortion
Figure 2: Approximate universality for general source. Illustrated are boundaries of achievable distortion- regions for three representations: minimal distor- tion (D1, Pi) for R(D1, Pi) R(D1, midpoint (D2, P2), and perfect perceptual quality (Dz, Pa) where P3 O. We have M(px,, 1x) a S2(R), especially when R is small. The extreme points and (D(t), p(b)) for XD are shown. (D(a), p(a)) coincides with (D1, Pi).
Figure 2 illustrates (RR) and ((pxp. x) for sev- eral different choices of (D. When R 0, (PXx,11x ((R) for any such XD.P. So we have S' 2 (R) in the low-rate regime where the tension between distortion and percep- tion is most visible. More general quantitative re- sults are provided in the supplementary. Let ox
Ei| - [[x]][]] If &D.,, is chosen to be the opti- mal reconstruction in the conventional rate-distortion sense associated with point (D,, Pi), then the upper-left extreme points of S( .y (x) and ((R)
coincide (i.e., (D(o), p(a)) (Di, Pi)) and the lower-right extreme points of Q(p Do, P (x) and
1x)
((R) (i.e.. (D(), 0) and (Da,0) with R(D3, 0) R(D1, 20)) must be close to each other in the sense that
1 - 2 D())_ Da
D or o
D Disco
2 D3
(10)
which suggests that 02 X (x) is not much smaller than 9(R)). Moreover, in this case we have
D 22 - X11 s 2D1,
(11)
which implies that (2D,, 0) is dominated by extreme point (D(0), 0) and consequently must be Therefore, the optimal representation in the conventional rate-
contained in (( distortion sense can be leveraged to meet any perception constraint with no more than two-fold increase in distortion. As a corollary, recovers Theorem 2 in Blau & Michaeli (R(2D, 0) <

---

h1
Wilex.Pa.)
6
g1
R1
h2
Wilex.Pa.)
Q
92
h3
Wi(px.Px.)
6
93
An illustration of the experimental setup for the universal model. A single encoder is trained for an initial perception-distort
Figure 3:
tradcoff and has its weights frozen. Subsequently many other decoders {9 are optimized for different tradeoff points using the representations produced by f. The sender and receiver have access to shared source of randomness u for universal quantization [30, 42]. Q denotes the quantizer. Separate critic networks {h:} are trained along with each decoder to promote perceptual quality. In this figure, the top decoder places most weight distortion loss whereas the bottom decoder places most weight on perceptual loss. This has the effect of reducing the blurriness, but comes at the cost of a less faithful reconstruction of the original (in extreme cases even changing the identity the digit). The perception losses W (px, , Px,) are estimated using critics {h,} by replacing the expectations in Equation (14) with samples from the test set.
R(D, oo)). Hence, the numerical connection between R(2D, 0) and R(D, oo) is manifestation of the existence of approximately Q([((;; - representations Z. These analyses motivate the study of practical constructions for which we seek to achieve multiple (D, pairs with single encoder.
3.3 Successive Refinement
Up until now, have established the notion of distortion ceceutioon-eecepttoon universality for given rate. We can paint a more complete picture by extending this universality along the rate axis as well, known classically as successive refinement [9] when restricted to the rate-distortion function. Informally, given two sets of (D, P) pairs O and O2, we say that rate pair (R1, R2) is (operationally) aate-distortion-perreee
refinable if there exists a base encoder optimal for R(O1) which, when combined with a second refining encoder, also optimal for R(22). In other words, bits are transmitted in two stages and each stage achieves optimal peeeddiisorttoon-eereppiooo performance. This nice property is not true of general distributions but we show in supplementary section A.3 that it holds in the asypmtotic Gaussian case, thus generalizing Theorem Nonetheless, building on [18] we prove an approximate refinability property of general distributions and in section B.3 provide experimental results demonstrating approximate refinability on image compression using deep learning.
4 Experimental Results
The pntedistootioonperreppioo tradeoff was observed as a result of applying GAN regularization within deep-learning based image compression [5, 33]. Therein, entire end-to-end model is trained for each desired setting over rate, distortion, and perception. practice undesirable to develop an entire system from scratch for each objective and would like to reuse trained networks with
7

---

Inpot
Universal
Ein-Lo-end
10
0
q G
O 9 1
O O
0.008
2
q
9
0
9
3 9
0
1 5
0
3
0
-
4
k-
5
9
3
9
Parcopttion
9
5 O
0.00000
9 9
3 1
0
8
5 0
3 9
O 0 7 8 4
6
3 6
0
0
3
0.04
0.05
0.06
0.07
4
5
4
8
9
MSE Distortion
(b)
(a)
Inpot
Universal
End to-end
Au 0.0000
55
17
Perception
-50
2.5
0.0005
-45
2.0
-40
1.5
:35
6
8
8
Au - 0,00.0
2
0.006
0.007
0.008
0.009
Mst Distortion
(c)
(d)
Figure 4: (a) (c) Rate-distortion- patteddittrtioonpeeeeptooo tradeoff's along various rates. Points with black outline are losses reported for the end-to-end encoder-decoder pairs trained jointly for particular perception- distortion objective. Other points are the losses for universal models, which decoders are trained over frozen encoder optimized for small P (MNIST: X 0.015, SVHN: X 0.002). Universal model performance is very close to performance of end-to-end models across all tradcoffis {A₂ (b) (d) Outputs of selected models (MNIST: R 6, SVHN: R 60). As the emphasis perception loss X; increases, the outputs become sharper. The visual quality of both the end-to- end-to-end and universal models average comparable for each A, More experiment details are given in the supplementary.
frozen weights if possible. It is of interest to assess the distortion and perception penalties incurred by such model reusage, most naturally in the scenario of fixing a pre-trained encoder.
Concretely, we refer models where the encoder and decoder are trained jointly for an objective as end-to-end models, and models for which some encoder is fixed in advance as (approximately) universal models. The encoders used within the universal models are borrowed from the end-to-end models, and the choice of which to use will be discussed later in this section. Within the same dataset, universal models and end-to-end models using the same hyperparameter settings differ only in the trainability of the encoder.
4.1 Setup and Training
The architecture we use is a stochastic autoencoder with GAN regualarization, wherein a single model consists of an encoder f. decoder g. and critic h. Details about the networks can be found the supplementary; here, we summarize first the elements relevant to facilitating compression then the training procedure. Let x be an input image. The final layer of the encoder consists of a tanh activation to produce a symbol fax) E [-1, with the intent to divide this into I--lvell intervals of uniform length 2/(L 1) across d dimensions for some L. This gives upper bound

---

of d d'log(L) L) for the model rate, and it was found to be only slightly suboptimal by Agustsson et al. [1]; we found the estimate to be off by at most 6% on MNIST. To achieve high perceptual quality through generative modelling, stochasticity is necessary' [33]. In accordance with the shared
randomness assumption within the ouRDPF, our main experimental results use the universal/dithered quantization" 2 [12, 29, 30, 42] scheme where the sender and receiver both have access to sample u (--Ul-//LL1), + ( 1))². The sender computes
2 Quantize([(xt) + u)
(12)
and gives to the receiver. The receiver then reconstructs the image by feeding : to the decoder. The soft gradient estimator of [21] is used to backpropogate through the quantizer. Compared to alternate schemes where noise is added only at the decoder, this scheme has the advantage of reducing the quantization error by centering it around ff (z) and can be emulated on the agreement a random seed. Since this is not always possible in practice, the results for more restrictive quantization scheme where the sender and receiver do not have access to common randomness are included in Figure in the supplementary.
The rest of the design follows closely the design of Blau & Michaeli [5]. We first produce the end-to-end models, in which fig and are all trainable. We use MSE loss for the distortion metric and estimate the Wasserstein-1 perception metric. The loss function is given by
B|| X x1 + (Px.
(13)
where P is the reconstruction distribution induced by passing X through f. transmitting the representations via (12) then subtracting the noise and decoding through The particular tradeoff point achieved by the model is controlled by the weight A. Manotooooiic-Ruubisteee duality allows us to write the Wasserstein-1 distance as
Wi(px - max [X) E[)( hET
(14)
which expresses the objective as min-max problem and allows us to treat it using GANs. Here, F is the set of all bounded I-Lipschitz functions. In practice. this class is limited by the diseriminator architecture and the Lipschitz condition is approximated with gradient penalty [13] term. Optimiza- tion alternates between minimizing over f.g with h fixed and maximizing over with f.g fixed. In essence, is trained to produce reconstructions that are simultaneously low distortion and high perception, so it acts as both a decoder and generator. The reported perception loss is estimated using Equation (14) through test set samples. Figure 3 provides an overview of the entire scheme.
After the end-to-end models are trained, their encoders can be lent to construct universal models. The parameters of f are frozen we introduce new decoder 91 and critic h1 trained to minimize
L1 = E [ - 8111 +
where A₁ is another tradeoff parameter and Px, is the new reconstruction distribution. The weights of @1 are initialized from random while the weights of hı are initialized from h. This was done for stability and faster convergence but in practice, we found that initializing from random performed just as well given sufficient iterations. The rest of the training procedure follows that of the first stage. This second stage is repeated over many different parameters {}:} to generate a tradcoff curve. Further model and experimental details can be found in the supplementary material.
4.2 Results
curves multiple rates on MNIST and SVHN obtained by varying À from 0 to a selected upper bound for which training with the given hyperparameters remained stable. Note that the rate for each individual curve is fixed through using the same quantizer across all models. As the rate is increased by introducing better quantizers, optimizing for distortion loss has the side effect of reducing perception loss. The rates are thus chosen to be low as the tension between distortion and perception is most visible then. The points outlined in black are losses for end-to-end models and the other points correspond the universal models sharing an encoder trained
Figure 4 shows aile-distortion-perepppp
1 'Tiis prevents us from passing noiseless quantizated representations to the decoder. 2 'Tee use of the word "universal" here is unrelated to our notion of "universality".

---

from the end-to-end models. As can be seen, the universal models are able to achieve tradeoff which is very close to the end-to-end models (with outputs that are visually comparable) despite operating with fixed encoder.
For any fixed rate, decreasing the perception loss P induces outputs which are less blurry, at the cost of reconstruction which is less faithful to the original input. This is especially evident very low rates in which the compression system appears to act as generative model. However, our experiments indicate that an encoder trained for small P can also be used produce a low-distortion reconstruction by training a new decoder. Conversely, training a decoder to produce reconstructions with high perceptual quality on top of encoder trained only for distortion loss is also possible as the decoder is sufficiently expressive to act purely as generative model.
Discussion
Limitations. One limitation these experiments is that we can slightly reduce the distortion loss by using deterministic nearest neighbour quantization rather than universal quantization, but there would no longer be stochasticity to train the generative model. comparison of quantization schemes for the case of À 0 can be found in Table of the supplementary. may be beneficial employ more sophisticated quantization schemes and explore losses beyond MSE as well.
Potential Negative Societal Impacts. The goal of our work to advance perceptually-driin lossy compression, which confficts with optimizing for distortion. We presume that this will be harmless in most multimedia applications but where reconstructions are used for classification or anomaly detection this may cause problems. For example, low-rate face reconstruction deblurred by GAN may lead to false identity recognition.
6 Conclusion
The use of deep generative models in data compression has highlighted the tradeoff between opti- mizing for low distortion and high perceptual quality. Previous works have designed end-to-end systems in order to achieve points across this tradeoff.. Our results suggest that this may not be necessary, in that fixing good representation map and varying only the decoder is sufficient for image compression practice. We have also established a theoretical framework to study this scheme and characterized its limits, giving bounds for the case of specific distributions and loss functions. Future work includes evaluating the scheme on more diverse architectures, as well as employing the scheme to high-resolution images and videos.
10

---

References
[1] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 221 221-231, 2019.
[2] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168, 2018.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International Conference on Machine Learning, pages 214-223, 2017.
[4] Yochai Blau and Tomer Michaeli. The perception-distortoon tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6228-6237, 2018.
[5] Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortionnpereeee perreettoonn tradeoff. In International Conference on Machine Learning, pages
2019.
[6] Rob Brekelmans, Daniel Moyer, Aram Galstyan, and Greg Ver Steeg. Exact rate-distortion in autoencoders via echo noise. In Advances in Neural Information Processing Systems, pages 3889-39000 2019.
[7] Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 1999.
[8] DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 1203):550-455 1982.
[9] William HR Equitz and Thomas M Cover. Successive refinement of information. Transactions on Information Theory, 37(2) 1991.
IEEE
[10] Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the distortion-perception tradeoff in wasserstein space. arXiv preprint arXiv:2107.02555, 2021.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. volume 27, pages 2672-2680, 2014.
[12] Robert M Gray and Thomas G Stockham. Dithered quantizers. IEEE Transactions on Informa tion Theory, 39(3):805-88) 882, 1993.
[13] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767-5777, 2017.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
[15] Sicong Huang. Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compres- sion rates of deep generative models. In International Conference on Machine Learning, pages 4444 4454, 2020.
[16] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pages 694-711. Springer, 2016.
[17] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby. Hugo Larochelle, and Ole Winther. Autocncoding beyond pixels using a learned similarity metric. In International Conference on Machine Learning, pages 5558-156., 2016.
[18] Luis Lastras and Toby Berger, All sources are nearly successively refinable. IEEE Tronsactions on Information Theory, 47(3)9118 922666696.18111 2001.

---

[19] Cheuk Ting Li and Abbas EI Gamal. Strong functional representation lemma and applications coding theorems. IEEE Transactions on Information Theory, 64(11):6977-6978, 2018.
[20] Dong Liu, Haochen Zhang, and Zhiwei Xiong. On the classffiatioon-disorriooppprrppe tradeoff. In Advances in Neural Information Processing Systems, pages 1206-1215, 2019.
[21] Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Conditional probability models for deep image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4394-4402, 2018.
[22] Fabian Mentzer, George D Toderici, Michael Tschannen, and Eirikur Agustsson. High-fidelity generative image compression. In Advances in Neural Information Processing Systems, vol- ume 33, 2020.
[23] Anish Mittal, Anush K Moorthy, and Alan C Bovik. Blind/referenceless image spatial quality evaluator. In 2011 conference record of the forty fifth asilomar conference signals, systems and computers (ASILOMAR), pages 723-727, IEEE, 2011.
[24] Anish Mittal, Rajiv Soundararajan, and Alan Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):20--212, 2012.
[25] Youssef Mroueh, Tom Sercu, and Vaibhava Goel. Mcgan: Mean and covariance feature matching gan. In International Conference on Machine Learning, pages 2527-2535, 2017.
[26] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan:
training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, pages 271 271-279, 2016.
[27] Naci Saldi, Tamás Linder, and Serdar Yüksel. Randomized quantization and optimal design with a marginal constraint. In 2013 IEEE International Symposium on Information Theory, pages 2349 235392 IEEE, 2013.
(28) Naci Saldi, Tamás Linder, and Serdar Yüksel. Output constrained lossy source coding with limited common randomness. IEEE Transactions on Information Theory, 61(9):4984-4998 2015.
[29] Leonard Schuchman. Dither signals and their effect on quantization noise. IEEE Transactions on Communication Technology, 12(4): 162-116,, 1964.
[30] Lucas Theis and Eirikur Agustsson. On the advantages of stochastic encoders. arXiv preprint arXiv:2102.09270, 2021.
[31] Lucas Theis and Aaron B Wagner. A coding theorem for the rate-distortion uuteddisorrtionpeercettion function. arXiv preprini arXiv:2 104. 13662, 2021.
[32] George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2017.
[33] Michael Tschannen, Eirikur Agustsson, and Mario Lucie. Deep generative models for Destriuution-preseviing lossy compression. In Advances in Neural Information Processing Systems, pages 5929-5940, 2018.
[34] Michael Tschannen, Olivier Bachem, and Mario Lucie. Recent advances in autoencoder-based representation learning. arXiv preprint / 2018.
[35] N Venkatanath, D Praneeth, Maruthi Chandrasekhar Bh, Sumohana S Channappayya, and Swarup S Medasani. Blind image quality evaluation using perception based features. In 2015 Twenty First National Conference on Communications (NCC), pages 1-6. IEEE, 2015.
[36] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for im- age quality assessment. The Teritt-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398-1402 IEEE, 2003.
12

---

[37] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from enror visibility to structural similarity. IEEE transactions on image processing. 13(4): 600-612, 2004.
[38] Alan S. Willsky and Gregory W. Wornell 6.432 lecture notes. https://ww..rhb rle. mit edu sia/wp- ponteent/pploads221155///// //t//////////...//// pdf, 2005.
[39] Zeyu Yan, Fei Wen, Rendong Ying, Chao Ma, and Peilin Liu. On perceptual lossy compression: The cost of perceptual reconstruction and optimal training framework. In International Conference on Machine Learning, 2021.
[40] Ram Zamir and Kenneth Rose. Natural type selection in adaptive lossy compression. IEEE Transactions on information theory, 17 2001.
[41] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[42] Jacob Ziv. On universal quantization. IEEE Transactions on Information 3113)3333333444411 1985.
13


Reviews:
[{"review_id": "tMfnGzMCf4", "reviewer": "Reviewer_nYfy", "summary": "The paper suggests that previous end-to-end compression system do not need to train different encoders to achieve points across rate-distortion-perception curves but just fixes a good representation map and varying only the decoder is sufficient. The paper also establishes a theoretical framework to support the conclusion.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Good. The paper proposes a framework to achieve different distortion-perception tradeoff by training different decoder with a fixed encoder to and provide corresponding theoretical analysis, which ensures novelty and provides theoretical base for future study.\n\nQuality: Good. The submission is technically sound in my opinion and the advantages and limitations of this work are discussed carefully and honestly. The rate-distortion-perception tradeoff is well analyzed and serves as theoretical base for the practical framework.\n\nClarity: Good. The submission is written with sufficient clear definitions and formulas. And the organizations are well-designed.\n\nSignificance: Kind of doubtful. Considering different distortion-perception trade-off is not very common for compression, and usually this trade-off is fixed depended on the specific task. Therefore, the proposed framework does provides a solution to decreasing training resource facing this trade-off but I doubt that whether it is really a useful tool. However, the paper proves its mention well enough, both theoretically and practically, which is brilliant.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have discussed the limitations and negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "prXeLz9S9j", "reviewer": "Reviewer_1XEU", "summary": "The paper considers the rate-distoriton-perception trade-off setting, and studies what happens if we fix an encoder and train different decoders for different (D, P) targets. This is studied both theoretically and empirically. It is shown that for Gaussian sources, the rate penalty incurred by using a single encoder for multiple (D, P) target is zero. Empirically, it is shown that on MNIST and SVHN, jointly training decoders and encoders is only marginally better in terms of D-P than when using a frozen encoder.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, studies an interesting setting, and contains solid results. The math is at parts a bit dense and sometimes, intuition could have been provided before the formalism, but I was able to follow everything in the main text. I had more trouble following the proofs in the supplementary in detail, and was not able to verify them.\n\nThe only question I had was regarding L273: why are the critics initialized from the end-to-end model? Is this required?\n\nThe following are minor comments. Most are cosmetic, so feel free to ignore, as I also do like the clean notation that is used.\n- L32: Could use a \"\\emph\" on \"distributions\" for people new to this.\n- L50: compare -> compared\n- L54: It's a bit sad that Section A.3 on successive refinement is only mentioned very briefly in the main text. Possibly, a summary of the empirical results could be added to the main text, as it seems this setting may be interesting for practical full-resolution compression.\n- L119: uRDP was never introduced (although it is clear what it means)\n- L136: I find the overloading of $R^\\star$ depending on whether the argument is a single (D, P) pair or a set of pairs slightly confusing. Possibly, the latter could be $R^\\star_u$, at the price of more symbols littered around.\n- L138 (Def 3): Here, it would have been instructive to start by saying that Z will be the representation of X. \n- L138 (Def 3): I would have put the \"for each (D, P) $\\in \\Theta$\" before the equation, close to $p_{\\hat X_{D,P}}$, which uses D, P.\n- L154: The symbol $A$ is not very memorable. Possibly something like $R_\\mathcal{E}$ would be more suited, or otherwise highlighting the \"all\" on L155 (where the A presumably originates from?) would be memorable.\n- L167: $\\Omega$ is also overloaded, this time depending on whether a distribution or a rate is used.\n- Eq 12: L is already used for levels, maybe $\\mathcal{L}$ would be more common?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper contains an adequate discussion of social impacts and limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "f9tq3ib5d7H", "reviewer": "Reviewer_Wozp", "summary": "The rate-distortion-perception (RDP) function[5,23] is a generalization of the classical rate-distortion (RD) trade-off in lossy compression to also measure realism, which establishes a theoretical footing for generative image/video compression, a topic of active research[1,11,17,25].\n\nThis paper studies the RDP function under the constraint of a shared encoder for multiple D-P points, both from the theoretical side and experimentally. This is of significant practical interest, as detailed below.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "So far, it has been assumed that an encoder/decoder system will be need to be optimized for a particular trade-off between distortion and realism (perception). This is not ideal, as at encode time we may not know what we want to use the reconstructed images for. If they end up in the courtroom, we would not want the reconstructions to contain synthesized detail (despite having high realism) --- if they are displayed for entertainment purposes, we may not want them to be blurry (despite having lower distortion).\n\nThe topic of this paper is to study universal representations for the RDP trade-off, such that a single encoding could be decoded in multiple ways depending on the the desired D-P trade-off.\nThis is of significant practical interest, as this simplifies the problem at hand.\n\nThe paper makes the following contributions:\n* It builds a convincing theory of the RDP trade-off for universal representations. This includes:\n- Characterizing their trade-off operationally and informationally ( Def. 2&3 + Thm. 2.)\n- Showing universality is essentially \"free\" for gaussian sources + MSE distortion + Wasserstein distance (Thm. 3.)\n- Shows in Thm. 4 is an \"approximate universality\" for the general case of arbitrary distributions ( with MSE distortion).\n- Demonstrates experimentally on toy datasets (MNIST, SVHN) that the D-P trade-off for various bitrates can be navigated from a single representation without a significant cost in performance.\n\nA limitation of the work is that the experiments only considers low resolution images, but I consider that appropriate given the theoretical focus of the work.\n\nGiven the both the theoretical insights this paper provides, as well as their potential practical value,\nI recommend this paper for acceptance.\n\nFinal update: Having read the other reviews, I am confident in keeping my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts were adequately addressed by the authors.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "c2Gb8SouS9c", "reviewer": "Reviewer_ZPE7", "summary": "The authors present a theoretical contribution to the rate-distortion-perception framework for lossy compression recently proposed by Blau and Michaeli. The authors show that it is possible to create universal representations by fixing an encoder (and hence rate) and change decoders to achieve multiple distortion-perception tradeoffs, with minimal penalty with respect to variable encoders.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and rigorously proves the authors' claim. Exploring the new framework of rate-distortion-perception theory is an interesting topic and important to advance the results on lossy compression, in light of recent results on perceptual quality achieved by models like GANs. To the best of my knowledge, the presented result is novel and might have interesting implications in the design of new image compression schemes. The paper is a theoretical contribution but the authors also show some experimental results on the MNIST and SVNH datasets. While these datasets are too simple to extrapolate the results to more realistic image compression, they are adequate at showing the theoretical points discussed by the authors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact and limitations have been adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an interesting contribution of theoretical nature, that essentially shows that one can work with a single encoder but vary the decoder to achieve different rate-distortion-perception trades-off in (image) compression. The work is likely to stimulate interesting discussion at NeurIPS, and provides useful insights towards future research in lossy compression.
Decision: Accept (Poster)