On the choice of Perception Loss Function for Learned Video Compression
Authors: Sadaf Salehkalaibar
Truong Buu Phan
Jun Chen
Wei Yu
Ashish J Khisti
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Compression
Information Theory
Neural Compression
TL;DR: Our work establishes the rate-distortion-perception (RDP) tradeoff and a principle of universality for causal low-delay lossy video compression.

Abstract: We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss to target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD,  considers the joint distribution (JD) of all the video frames up to the current one, while the second metric, PLF-FMD,  considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have a significant effect on the reconstruction, especially at low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion  compared to PLF-FMD and further makes it more difficult to recover from errors  made in the earlier output frames. Although the choice of PLF decisively affects  reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. In particular, encoded representations generated by training a system to minimize the MSE (without requiring either PLF) can be  {\em near universal}  and can generate close to optimal reconstructions for either choice of PLF at the decoder.  We validate our results using (one-shot) information-theoretic analysis, detailed study of the rate-distortion-perception tradeoff of the Gauss-Markov source model as well as deep-learning based experiments on moving MNIST and KTH datasets.

On the Choice of Perception Loss Function for Learned Video Compression
Sadaf Salehkalaibar"
Buu Phan'
ECE Department
ECE Department
University of Toronto
University of Toronto
sadafs@ece.. utoronto.ca
truong phan@mail wtw.otto..com ca
Jun Chen
Wei Yu ECE Department University of Toronto weiyußece utoronto.com ca
Ashish Khisti
ECE Department
ECE Department University of Toronto
McMaster University
chenj Chenjun@nen jn@mmmasterr.r...m wetemmastterrom ca
akhisti@ece. utwronto.coo ca
Abstract
We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD, considers the joint distribution (JD) of all the video frames up the current one, while the second metric, PLF- FMD, considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have significant effect on the reconstruction, especially low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion compared PLF-FMD and further makes it more difficult to recover from errors made in the earlier output frames. Although the choice of PLF decisively affects reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. particular, encoded representations generated by training system to minimize the MSE (without requiring either PLF) can be near universal and can generate close to optimal reconstructions for either choice of PLF at the decoder. We validate our results using (one-shot) information-theoreti analysis, detailed study of the eeceddistortioo-perepppee tradeoff of the Gauss-Markon source model well deep-learning based experiments on moving MNIST and KTT datasets. Code will flow
be available at httpss//ggtub..om//rruuguuu////u
Introduction
There is an increasing demand for video compression algorithms that are able to generate visually pleasing videos low bitrates. Most the current video codecs use distortion measures such as PSNR (T MSE and MS-SSIM (355] to generate reconstructions which tend to be blurry at extremely low bitrates. recent years, there has been a growing interest (see e.g.. [6 in using deep generative models to make the reconstructions look more realistic. Such techniques introduce an additional perception loss function that measures a distance between distributions of the source and reconstruction, with perfect perception corresponding requiring that the two distributions be identical.
'Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

K
f1
91
Frane
31
22
83
21
p3
21
X1
23
4
4
4
Dosoder
2
2
2
K
K
9
4
25
f2
M
7
2
Eucoder
Decollr
7
2
9
4
X3
7
Easoda
Doodlrr 3
(b) Effects of different PL.s reconstructions for Moving
(a) Encoders and decoders for 7=3 frames.
MNIST and KTH datasets (best view the monitor).
X
x
X3
PLF-FMD &;
PLF-JD: X;
Avsrage Poal Per Frarze
X3
PMP
PLF Xg
Pontag: the krorely
Source
FMD
(c) Error permanence the UVG dataset. The PLF-JD reconstructions propagate the flaws in the color tone from previous I-frame reconstruction while decoder based PLF-FMD able to
flaws.
Figure 1: (a) Proposed System Model (b, c) Error permanence phenomenon under different PLFs. High fidelity but incorrect I-frame reconstruction propagates the to subsequent P-frames in O-PLF-JD reconstructions. MMSE and O-PLF-FMD reconstructions do not have this problem.
In compression systems, improving realism comes at the price increasing distortion. The work of Blau and Michaeli [11) establishes the theoretical also been validated 15). Furthermore universal encoded representations were proposed in (16) where the representation fixed at the encoder and the decoder is adapted to achieve performance near the optimal RDP tradeoff curve. The extension these works to video compression involves many challenges. First, the compression system must not only account for spatial redundancy as image compression, but also exploit the temporal redundancy across video frames, making the complex. Secondly, unlike the case of image compression, there may be
ddiitortion-eereepppii (RDP) tradeoff which has
system design clear choice of the perception loss function (PLF). Indeed, some prior works 7 consider PLF that preserves framewise marginal distribution (PLF-FMD) between the source and reconstruction, while other works consider joint distribution (PLF-JD) across multiple frames (9].
As illustrated in Fig. a we study causal, low-latency, sequential video compression when the output subjected to both mean squared-erre (MSE) distortion loss and either PLF-JD PLF-FMD metric for perception loss. Our contributions are as follows:
Differences reconstruction quality based on the choice of PLF: We demonstrate that the choice of PLF can decisively affect the reconstruction quality especially in the low bit-rate regime. We approximately characterize the operational RDP region on per-frame basis for first-order Markov source model and analyze the special case of Bauss-Markon sources in detail. We show that there a significant penalty in distortion when using PLF-JD the low-rate regime. the experimental side, we demonstrate that while PLFF-DD preserves better temporal consistency across video frames, suffers from the permanence of error phenomenon in which the mistakes in reconstructions propogate future frames. On the other hand, the PLF-FMD metric shows

---

more capability in correcting mistakes across frames (see Fig. I for visualizations involving three-frame videos). On the other hand, if the first frame transmitted at high bit-rate, we demonstrate that PLF-JD performs better than PLF-FMD.
Universality of minimumm mean square ervor (MMSE) reconstructions: We demonstrate that encoded representations generated from an encoder trained to minimize MSE reconstruction (without considering any PLF) suffice to produce close-to-optimal reconstructions for either choice of PLF. For general sources, we show that when using PLF-FMD, the MMSE reconstruction can be transformed to reconstruction satisfying perfect perceptual quality by increasing the distortion at most by a factor of two. While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. For the Gauss-Markov source model we demonstrate exact universality i.e., encoded representation for the MMSE reconstruction can be adapted achieve any other reconstruction in the RDP region. We also use deep learning based experiments to provide experimental evidence of these results.
We note that the above notion of universal encoded representations based MSE reconstruction can have significant advantages in practice. First, although the reconstructions associated with different choices of PLFs be visually very different, universal representations delegate the choice of PLF to the decoder rather than requiring the encoder to commit to a specific PLF Secondly, the universality of MMSE reconstructions is far more significant in the context of learned video compression. Given that perceptual reconstruction frequently generates novel details not present in the source frame, compressing motion flow vectors between the current frame and the prior percepiual reconstruction necessitates higher bit allocation compared to utilizing the MMSE reconstruction. Hence, a recommended approach is train end-to-end compression exclusively to minimize MSE and use our proposed scheme to achieve (near-optimal) tradeoff between the distortion and perception losses as desired by the user.
The study of RDP region for learned video compression is considerably more challenging than the study of RDP function for single frame in prior works [1 16)). This is because of the fact that the RDP region (for first-order Markov sources) involves a tradeoff between the compression rate assigned to each frame imposing Markov structure the reconstructions. For Gaussian sources, the proof of optimality of Gaussian reconstructions does not use the closed form of the RDP region due its complexity. As result, the study of RDP region for various operating regimes is quite more involved. Furthermore, for the proof of universality, one has to consider the achievability of the entire RDP region as opposed to just points on the boundary of RDP function in [16). Finally
the results on the fixed-encoder setup are more general than prior works ( [[I [6]) that required a characterization of the information RDP region, which we do not require.
Related Work
Perceptual Lossy Video Compression. Distribution preserving framework using Generative Ad- versarial Networks (GAN) has been widely adopted as surtogate metric for perceptual quality in image 17 and, recently, video compression (6)) Unlike image compression, where the choice of PLF Is straightforward. there is currently no agreed-upon objective for lossy video compression. For instance, DVC-P (6) employs PLF-FMD to improve the visual quality per frame, ignoring the temporal coherence. Similarly, Mentzer et al. (7) utilize the per- prr-frame metric with a conditional GAN model, and found no significant differences when using a GAN objective with multiple frames. Other works target temporal consistency by incorporating multiple frames in their GAN objective. This includes the work by Yang et al. [8]. where every two consecutive frames are included, and by Veerabadran et al. (9). where they employ the PLF-JD metric in a non-causal setting. Unlike previous works, we study the impact of two different perception objectives, i.c. PLF-JD and PLF-FMD, reconstructions in the causal setting, presenting theoretical properties that are verified by deep learning experiments. For the PLF-JD metric, we demonstrate the error permanence phenomenon, which, unlike the error propagation issue 18 20 cannot be resolved by increasing the code rate assigned to the P frames.
RDP Tradeoff and Principle of Universality. Targeting the distribution preserving framework in lossy image compression, several theoretical works have shown the presence RDP tradeoff [1] 23] where perfect perception comes at cost of increasing distortion by at most a factor of 2. Furthermore, an encoder that generates universal representations exists [5). which enables the decoder to freely choose the level of cettortion-perceptinn tradeoff it desires. Our work explores these avenues
3

---

in the context of causal video compression. As discussed previously we show that the MMSE representation can be used as a universal representation for both perception metrics which has several advantages in the context of video compression..
2 System Model
Let (X,,...., Xr) E X1 X X NT be T frames in a video (with each X; S Rd) distributed according to Px. The frames are available for encoding sequentially; X1 is available first, then X2 arrives, followed by X3 and so on. There is a shared randomness K E K which available at all encoders and decoders. The following (possibly stochastic) mappings define the encoding and decoding functions:
f X1 X ... x 23 x K M,
j= T.
9; M1 M2 x ... M, x KC xK
(2)
where M, E {0, 1)* denotes the set of (variable-lengthy lenghh) messages assigned by the jth encoder and
X2 C CR² is the j-th reconstruction alphabet (see Fig. lab. Let Px, Srl.............. be the conditional distribution of the reconstructed video given the original video which is basically determined by the mappings {f,};= and {g, 1- The above setting is one-shot setup as only a single source sample is compressed at a time. For each frame j. distortion metric is imposed on the output, which we assume throughout is the mean squared-error (MSE) function i.e. d(z,,. .ty) = || ,211? which commonly used in many applications. From a perceptual point of view, for given probability distribu- tions Px xx....,, and Px xx....;; on the original and reconstructed frame j. let Φ, (Px....y x Pxy...x be the perception function capturing the difference between them. Note that the function oy is defined based on the joint distribution of all first j frames. We call this metric as perception loss function bused on joint distribution (PIF--D). Note that when dby (Px...... Px,...// = O, we have:
Px....x; = P
j=
(3)
We refer to this case as zero- prooppecceptinn loss function based on joint distribution (O-PLF-JD). Altern- tively, the perception loss function based on framewise marginal distribution (PLF-FMD) is denoted by (Px,, Px₂) Px,) that O-PLF-FMD implies that Px, Px, for each j. In most of the paper, for simplicity of presenta- tion, we provide some of our results for T = 3 frames. In that case, use the shorthand notation X to denote the tuple (X1, X2, X3), e.g., M == (M1, M2, Ms), D == (D1, D2, Dz), f == (f1, f2, fs).
and is based on only the marginal distribution of the j-th frame. In particular, note
3 Distortion Analysis for a Fixed Encoder and Zero-perception Loss
In this section, we assume that the encoding functions f are fixed, but the decoding functions g can be optimized to generate different reconstructions. Equivalently, the distribution PM 1 = f(X, K)} is fixed, while by varying the reconstruction distribution P = g(M, K)}, one attains different reconstructions X, where 1 {-} denotes the indicator function. Furthermore defining Dp:EEpI|X, - X, [2], we denote D as the achievable distortion tuple associated with P
One natural choice of reconstructions is the minimum mean squared error (MMSE) reconstruc- tion function. At step j. the reconstruction, which we denote in this case by Xj. is obtained by taking the conditional expectation of X, given all information at the decoder up to time j i.e., Xy:=Iyp X, |M, ... functions minimize the reconstruction distortion i.e., if we define the set
.......,,,,, M,, K] for each j = 1, 2, 3. It is well known that the MMSE reconstruction
Pm (PMIX)) (D D2 2 EpI|X X2,1%
j 1,
then the distortion tuple D associated with any reconstruction Px Mą satisfies D E P (P (PMIXK).
The main result of this section is that assuming fixed encoder, the achievable distortions under O-PLF-FMD is at most twice of that under the MMSE distortion loss alone. The same conclusion also holds for O-PLF-JD for class of encoders operating at low rate. We first consider the case of O-PLF-FMD.

---

Definition 1 (0-PLF-EMD Distortion) For an encoder PaIXK, the set DDo denotes the set of all distortion tuples D for which there exists reconstruction P PIIKK satisfying Px, = =xx for each j E {1, 3).
Theorem The D (PMIXK) is characterized as follows:
P ( xMxx)= = DD: EplIXX; - -X,11²+ + w3 j 1=
(5)
where W2 (Px, Px,) denotes the Wasserstein-2 distance between the two distributions (26). Fur- thermore, we also have that:
Po 2 (D: : D3 2 2 - -x,17), j= 1,2,33]
(6)
i.e. minimum achievable distortion with O-PLF-FMD at most twice the MMSE distortion.
Proof: See Appendix A
We remark that the proof of Theorem operationally demonstrates that the MMSE reconstruction can be converted to another reconstruction satisfying O-PLF-FMD with at-most a factor of increase in distortion, generalizing the result in (16) for the single frame scenario (see also (2T))-
We next consider the case when zero perception loss is satisfied under the PLF-JD metric. Analogous to Po (PM)k) in Definition 1 one can define @p (PMIXk) to be the set of distortions associ- ated with reconstruction functions that satisfy (3) The analysis of 4 (PMIXK) is discussed in Appendix B as it is more involved. In general, the factor of two bound as in Theorem cannot be realized in this case as demonstrated by a counter-eat cuuntrr-examll in Appendix [ Nevertheless, for a special family of encoders we can obtain counterpart of Theorem [ In this family of encoders, the source X, at time j is nearly independent of the encoder outputs up to and including time j, i.e., we can
express:
phoisy X2) M, (1-1))Px + MQ MO..IlM,..M,KK.
noisy
j 1,
where μ sufficiently small constant and the distribution Quiiny (-) could be arbitrary conditional distribution with same marginal as Px,- We note that such encoders are studied in variety of problems in information theory (see e.g., (27]) that correspond to the low rate operating regime. The following result states that the factor-two bound holds approximately for such encoders.
Theorem 2
For the class of encoders given by 7 have
wojonn MiXK) 2 {D: D3 N 21 p [|
XX,131 O(k), j= 3,
(8)
Proof: See Appendix
We note that the low-rate operating regime practically important, as at higher rates MMSE based reconstructions can suffice and the use of PLF metrics may be less relevant.
4 DateeDDssootton-Perreptti Region
In this section, we assume that both the encoder PMIXK as well as the reconstruction P can be optimized and study the associated reeedisstortion-erreeee that for PL.F-D and PL.F-MD,, the PLFs are denoted by Φj (Px,..Xy, Px...., x,) and & (Px,, Px₂) respectively. In this case, the operational RDP region in the one-shot setting is defined follows.
(RDP) tradeoff. We remind the reader
Definition ( Operational RDP region) For a given Px, an RDP tuple (R, said to be achievable for the one-shot setting if there exist an encoder PMIXK and reconstruction P satisfying:
EiE(M)) SR3, -
0f Sx....,,,,
1,1,2,3, (9)
where ((M,) denotes the length of the message My. The closure of the set of all achievable tuples, denoted by CRDP, is the operational RDP region. Moreover, for a given (D, P), the operational DP rate region, denoted by R² (D, ), is the closure of the set of all tuples such that (R, D, CROP
The region CROP cannot be directly computed as it involves all possible one-shot encoders/decoders. But for first-order Markov source, it has tractable approximation in terms of mutual information.
5

---

4.1 RDP Region of First-Order Markov Sources
We first define the first-order Markov sources and then introduce an iRDP region which is computable.
Definition 3 We call X as a oirtt Markov source if the Markov chain X1-xX-xX0 holds.
Definition 4 (Information RDP region) For first-order Markov sources, the information RDP (iRDP) region, denoted by CRDP, is the set of all tuples (R, D, P) which satisfy the following
R1 > Xr,), R2 2
Rs 2 I(X3;
X,,
(10)
D; 2 B X,113,, Pj 2 oz (Pxi..., Px,..,,),
j j= 1,
(11)
for auxiliary random variables (Xr,1, Xn, 3)) and (81,82, X3) satisfying the following
X1 - (Xr,),, X2 = (x,,,11 X3 = X3.r
(12)
Xr,1 + X1 + (X2, X3), Xn. + (X2,Xr,11 + (X1,
(13)
Xr,3 + (Xs, xx + (X),
(14)
for some deterministic functions M (.) and n2(...). Moreover, for given (D, (D,P), the information DP (iDP) rate region, denoted by RDD P), is the closure of the set of all tuples R that (R, P) E CRDP.
The expression for the iRDP region involves a search over auxiliary random variables X, and X that satisfy ( (4) subject the constraints in (TO) (II.. For first-order Markov sources, the following theorem states that the operational DP rate region can be approximated by the iDP rate region.
Theorem For first-order Markov sources, a given (D, P) and R E P), we have + (R 1) + 5 E R (D. S PD.,
(15)
Proof: See Appendix D
From Theorem 3. it follows that R(D, P) with overhead log( ( [R+ R+1)++5 +5 is an inner bound to (D P). On the other hand, RDD, P) provides an outer bound to R (D, The two bounds match with each
other in high rates. It can be shown that the overhead also vanishes in the large-blo setting
where multiple symbols are encoded at time. In the remainder of this paper, we will approximate Rº (D, P) with R(D, P) and use the latter region for our analysis.
Remark (Encoded Representationss: The proof of the inner bound in Theorem 3 in Appendix D
provides an operational interpretation to the auxiliary random variables X,, = (Xr,1, Xr.2, X Xn3) defined in iRDP region in Definition 4 In particular, X,, is lossy version of the source sample X
generated by the encoder in step j. Ir is compressed and transmitted to the decoder at rate Rí in (TOp.
We refer to X,, as the encoded representation of the source X. The Markov chains (13) (14) indicate that without loss of optimality, an encoded representation Xr.j can be computed from the source X₃
and past reconstructions X X,,1...., Xr without using past source samples X1,...,X, X
Remark 2 (Deterministic Reconstructions); Note that the reconstruction functions generating X in Definition are deterministic functions of the encoded representations (c.f. (12)) In particular, the shared randomness K is not required in the reconstruction functions. However, as the proof of the inner bound of Theorem iillustrates, the shared randomness is required in the compression and construction of Xr.j- Moreover, by following the arguments in (28). one can set the reconstruction
function of the last frame to be identity. Thus, in Definition 4 we have set X, Xx,33 in 12 where
= 3. In the sequel, for T frumes we will set X = Xx,T-
Remark 3 The result in Theorem 3 also holds for the PLF-FMD. That is, one can repluce the PLF in and IT by (xx, Px)) and get a similar result (see Appendix D for the justification)
4.2 Gauss-Markov Source Model: RDP Region
In this section, we obtain practical insights through the analysis of the special case of first-order Gauss- sources. We assume that X, ~ o(),of).
X₂ P1 02 X1 + N1,
03 X2 + No.
X3 = P2
(16)
01
02

---

where N₂ is independent of X, with mean zero and variance (1- 13+3) for = 1,2. Note that the model extends naturally to the case of T time-steps. The perception metric assumed be the ==
Wasserstein-2 distance, i.e., 0j (2 (x.....; Px, x;,
Px (x...,,)) For the PLF-FMD, (P,,,, Px,).
the perception metric is given by Ey (Px,; (x,
The following result states that for Gaussian sources, jointly Gaussian reconstructions are optimal. Thus, for given tuple (D, P), the characterization of RDD, P) becomes computable.
Theorem For the Gauss-Markov source model, any tuple (R. D, P) E CROP can be attained by a jointly Gaussian distribution over X, and identity mappings for ny (-) in Definition 4
Proof: See Appendix E
Generally, the optimized distribution in the above theorem may not admit simple form. In the special case of T 2 frames, the optimal reconstructions are given in Appendix E To obtain practical insights, we consider various asymptotic operating regimes and provide detailed analysis [ for the case of T = 2 frames and with of = o². A summary of these results is provided in Table in the same Appendix. We briefly summarize some of these results next.
in Appendix
4.3 Gauss-Markoy Source Model: Extremal Rates
One of the key observations of this paper is that the choice of PLF has implication on the rate allocation across different frames. Specifically, first consider the case when both R are small ie.. 11=R0 R (for small enough e). We discuss how each PLF affects the reconstruction in the second step. In the first step. we note that reconstruction in both cases must be identical and of the form -C==v2c In 2X + Z1 where Z0,~N(0, (1- 2c In 2)o²) is independent of X1; the resulting distortion is given by D1 21=2(1- (o/l1-2e0 2 In 2)o2. However, the reconstructions in the second steps will be different for the two measures. For simplicity, we consider the extreme case when p=1 (i.c.. when I)= Here, the PLF-JD metric is required to preserve perfect correlation and thus has to set -Xp and results in D2=Dı, In other words, the decoder in the second step is unable to use any information transmitted in the second step as O-PLF-JD enforces the stringent constraint xf= In contrast, for the PLF-FMMD metric, can be shown that the reconstruction in the second step for p=1 reduces to X2 =VVv2 In 2X, + ZFMD and the associated distortion is given by D,=2(1-V 4e In 2)o², which is lower than PLF-JD. Extending this example to T steps (with p=1),, we note that PLF-JD will always be forced to output X,, while the reconstruction using PLF-FMD will successively improve. The following theorem formalizes this observation.
Theorem For sufficiently small e, let R3 = e and suppose that p; = p and 0j = a, for 1. T. The achievable distortions DYMD (for O-PL.FFMM) and DID.j (for O-PLFFID) are:
DYMp.j = 2(1 - ArMD. 2) D = 2(1 - -Ap.j V2 In 2)o²
(17)
where AFMD.
1+ p² 20--1
(202 and Ams = (1-1) 11j 2 2} v1 PC
Proof: See Appendix
In particular, specializing to p 1, AFMD, = 2 and AJD.j = This shows that the decrease in DPMD. is exponential at each step which implies the ability of decoder based on 0-PLF-FMD in correcting mistakes and not propagating them in future reconstructions. However, as discussed previously the decoder which uses O-PLF-JD is stuck at X, = X, when p = and results in no subsequent improvement in the distortion. We call this behaviour phenomenon is magnified in the case when R - 0 and R3 + oc, treated Table 2 (Appendix F as the PLF-JD severely constrains the decoder to copy the previous noisy reconstruction while the flexibility provided by PLF-FMD reduces the distortion in the second step.
permanence of error. This
The case when Rj + oc and R₂ treated in Table 2 in Appendix F corresponds to the case when X1 sent at sufficiently high rate (as is the case with some I-frames) while X2 is sent at a low rate. Naturally, we have X² X2 for both PLFs On the other hand, we once again see qualitatively different behaviour in the reconstruction of X2. For the case of O-PLF-FMD, we have X6 (1 &f + (g)) i..., the decoder essentially copies the previous frame with
7

---

little contribution from the second step. In contrast, for the case of O-PLF-JD, it can be shown that X ~ p1 X + X2 + ZID. where ZIn is independent Gaussian noise with variance close to p-p². We note that the PLF-JD metric prevents the decoder from simply "copying" the previous frame, but instead forces the decoder to generate more diverse representation consistent with the joint distribution between the two frames.
4.4 Universal Representations for Gauss-Markov Source Model
In this section, we show that the Gauss-Markon source model admits universal encoded represen- tations. Such representations can be transformed through appropriate reconstruction functions to achieve the entire DP rate region. This is the counterpart of the result for general sources in Theorem [ where it shown that the MMSE reconstructions can be transformed to some target reconstructions satisfying the 0-PLF-FMD with at most factor-2 increase in distortion. In contrast, we demonstrate that the Marks- model admits exact universality i.e., target reconstructions proposed this section achieve all points in the iDP rate region. Interestingly, the transformation is linear with possibly some additive noise. First, we formalize the notion of universal representations.
Definition (iDP-Tradeoff) For a given rate tuple R, the optimal iDP-tradeoff the closure of the set of all tuples (D, P) such that (R, P) CRDP and is denoted by (P)
Definition 6 (Universal Representation) A given encoded representation X, is called universal with respect to rate tuple R if satisfies the rate constraints (0) and the Markov chains in (13
and for each (D, E DP(R), there exists a reconstruction X generated from PxIx. achieving it.
For the Gauss-Markov source model, show that the MMSE reconstruction admits a universal representation. We consider the reconstruction X, that achieves minimum distortion in the DP (R) region. This point is explicitly characterized in Appendix H.] Furthermore, following Theorem 4 since the reconstruction functions ();()) are identity, the MMSE reconstruction is equivalent to MMSE
representation i.e., X, XRD The following theorem establishes that any point in DP(R) can be achieved from X,.
Theorem For the Gauss-Markov source model and a given rate tuple R with strictly positive components, let the MMSE representation be denoted as X = (X*R, 22 X Let (D,P) E
(P)R) and let X = =X1; X2,,,,X0 Xz) be the corresponding reconstruction achieving it. Then there exist K1. 9j. 02 11, dh2 and ths and noise variables (Z,, Zz, Zz) independent of (XRD r,2 XF2), which satisfy the following
X1 - X + Zz - - + 0,X Zz 1==11: + X 2 + Zz.
Proof: See Appendix H.2
The above theorem indicates that the MMSE representation can be linearly transformed to achieve any point in DP(R), In general the MMSE representation may have to be degraded through additional noise terms. In the proof of Theorem we identify conditions when such degradation is not needed.
H.2 Theorem holds for both PLFs. This suggests the idea that one can train an encoder to get MMSE representations which oblivious the choice of PLF. Then, the decoder can generate a reconstruction which satisfies either of PLFs by simply applying a linear transformation to the MMSE representation. Thus, the task of choosing the right PLF can be assigned to the decoder based on distortion and perception requirements. We conclude by noting that Appendix H.3 provides example where the coefficients in Theorem can be computed explicitly.
As discussed in Appendix
5 Experimental Results
We conduct experiments on the MovingMNIST dataset 29 (with 1 digit) using Wasserstein GAN 30), to verify the implications our theoretical claims to perceptual video compression. Additional results on the KTH dataset [31] are available in Appendix J.3 Our compression network is built on
the scale-space flow model [32] and conditional module [33] For a given rate and PLE we obtain
different cetottion-perception tradeoff points by optimizing the weighted sum between distortion
8

---

Gmurd
D-RIF.ID
CUPI F.P
MUSF
$
S
S
4
5
S
5
S
0
0
o
5
0
O
0
6
6
6
6
6
6
6
6
G
6
6
0,222
1e 04)
30
5
5
s
5
5
3
9
3
(uE
(u62
3
3
a
4
3
Goms
8
6
8
8
6014
CERLEME
X1
X;
Xa
X
X
X
X1
X2
X
X1
X;
X2
2nd
34
(a) Ground-truth GOP and their optimal reconstruc- tions with different PLFs for Rt = R3= 12 bits.
(b) Distortion per frame (X, x.) for different rates with = 1, 2,3.
Permanence of Error Phenomenon. In (a), we visually compare the reconstructions. Note that Xg is the same for both O-PLF-JD and O-PLF-PMD. In (b). we show the framewise distortion for different (Ra, Ra).
Figure 2:
and perception losses. Details about the architecture and training procedure are available in the Appendix [.1] The experimental setup is focused on validating our theory, rather than proposing state- of-the-art neural network architectures. Accordingly, we begin by (1) validating Theorems and andd 2 which characterize the factor-of-two bounds on the distortion of O-PLF reconstructions (2) empirically
demonstrating the error permanence phenomenon of the PLF-JD in Section 4.3 and (3) computing
the DP tradeoff function experimentally as well as confirming that the MMSE reconstruction provide
near universal representations, motivated by the results in Section 4.4
As our first experimental result in Ta- [lc[[] we validate the factor of two bounds in Theorems and We consider the com- pression of two frames X₁ and X2 at rates R1 and R₂ respectively. The compression of X1 is performed without any prior ref- erence and corresponds to the compression of the "I-frame", while the compression of X2 corresponds to the "P-frame", using X1 as the reference. We consider the cases when either Ri=oo or =1=12 bits, where the former corresponds to lossless compres- sion of i and the latter corresponds to the low rate regime (see Appendix [or a jus- tification). The average distortion for the first frame when R₁ 12 is 0.0124 for the MMSE reconstruction and 0.0235 for the O-PLF re- construction, thus satisfying the factor of two bound. In compression of X2, we systematically vary the value of the rate [4, 8, 12, oo). Following Table 1b for O-PL--JD reconstruction, only R2=4 bits (low rate) satisfies the factor of two bounds as expected. Intuitively, even as more bits acquired, the O-PLF-JD criteria actively restricts improving the reconstructions, resulting in persistently higher distortion. Even when R2 = co, the distortion remains non-zero as the decoder is forced to maintain temporal consistency with X1 In contrast, for FMD, the factor of 2 bound holds all rates, consistent with Theorem I
Table 1: Distortions optimal reconstructions different
regime (v means factor of 2 holds and Xmeans otherwise).
Distortion is scaled by 10-2
R₂
MMSE
O-PLF-FMD
0-PLF-JD
1.08 = 0.01
1.74 = 0.02 V
2.05 0.03 V
0.88 0.01
1.39 + 0.03
1.46 + 0.02 V
3.17
0.53 = 0.01
0.76 = 0.01 V
0.79 = 0.01 V
(a) Case 1: R oo bits
R₂
MMSE
(-PLF-FM
O-PLF-ID
4
1.23 + 0.01
2.21
0.04V
2.36 0.04 v
8
1.04 + 0.01
1.78 = 0.03 V
2.28 + 0.03 X
12
0.89 + 0.02
1.43 = 0.02 v
2.26 0.03 X
00
0.0
0.0
2.18 0.02 X
(b) Case 2: Ri= 12 bits(e).
In Fig. 2 we present our experimental results with group of pictures (GOP) of size 3 (i.c. one I-frame followed by two P-frames). In Fig- 2a we visualize sample reconstructions for MSE, 0- PLF-FMD and (-PLF-JD cases when operating in the low-rate regime with Ry=12 12 bits for 2=1,2,3.
Note that given incorrect digit reconstruction in X1, the decoder with O-PLF-JD consistently produces incorrect digits (or content) while the (-PLF-FMD gradually "corrects" it, which confirms the error permanence phenomenon discussed in the theoretical analysis in Section 4.3 and Table 2 Appendix F We also plot the framewise distortion in Fig. 2b show the difference in achievable distortion two perception metrics across different values for R2 and R3 as a function of the frame index. Consistent with Theorem [5] the achievable distortion decreases much faster for O-PLF-FMD than O-PLF-JD for all selection of rates. Finally, we show similar results in Figure for UVG dataset.
In Fig. 4 we plot the tradeoff curves between distortion and perception for the second reconstruction X₂ for both optimal (end-to-end) and universal representations for two cases: when R2 =00 and R 12 bits and for range of values for Rg. In general, the curves for both universal and optimal

---

Xj
X
X2
P.F-FMD: X3
PLF-JD: X;
PLF-PMD: &
PLPJD: Xs
Average
0.52
Figure 3: Error permanence phenomenon the UVG dataset. The PLF-JD reconstructions propagate the flaws the color tone from the previous I-frame reconstruction while PLF-FMD is able to fix these flaws. Compression rate for I-frame and P-frame are 0. 0.144bpp (low rate) and 4.632bpp (high rate) respectively.
representations are relatively close to each other at all rate regimes. The general shape of every curve is relatively similar with the exception of the PLF-JD metric in Fig. 4b where the curves for different rates seemingly converge since increasing the rate does not significantly improve the distortion in this case as noted previously. Finally, as the universal encoders are derived from MMSE solutions, these results imply that one can simply send the MMSE representation to the decoder and the user can flexibly change the DP tradeoff up their requirements. We further note that even when the end-to-end model targets an operating point different from the MMSE reconstruction, the latter is still required estimate the motion flow vectors best. The universal representation provides a natural way to reconstruct the MMSE reconstruction from the encoder output. In the plots of Fig, 4 we leverage on established universality results for I-frame compression in prior works [16) to construct the MMSE representation for motion compensation as we have GOP of size 2.
3:17 bi unveral
2 be crivenal
unionsa
lit
12 bii univeral
Bbx univasal
bi uniersal
bb escito-and
PrPPsso Phuke
PLF D. WilP Pr.ae
PIE-FMO- aS
0.035
01000
0005
6,09
0,05
0100
0.02
01000
0.00
Dieartio::
Ny- X:18
Desotion|]]X
8
Deverbon X- Xill
Datortion
8:B
(a) Case 1: =o Ra= (1. 3.117 bits
(b) Case 2: R, 12, Ra= (4, 12, oo) bits
Figure 4: RDP tradeoff curves for end-to-end end and universal models. We plot the tradeoff for the two regimes:
Ri =00 and = in (a) and (b) respectively. The universal and optimal curves are close each other.
Finally in Appendix J.5 we consider the ability of the decoder to generate diverse reconstructions when operating under either PLF-JD or PLF-FMD. We focus on the case when X1 is transmitted losslessly and when X₂ is compressed at low rates. Consistent with the theoretical analysis in
Section 4.2) and Table i Appendix(F] [F] the decoder optimized for PLF-JD is capable of producing diverse reconstructions by mimicking the actual motion between the frames. The PLF-FMD leads to reconstructions that are highly correlated and less desirable.
Conclusions
This work examines different perception loss functions for causal video coding, establishing its key theoretical properties such as the operational RDP region and universality principle. Our analysis highlights that while O-PLF-JD reconstruction preserves temporal correlation, it is susceptible to the emror permanence phenomenon. Moreover, our investigation of universality reveals that the encoder can transform the MMSE representation to other points on the DP tradeoffs. irrespective of the PLF. We suggest future research directions such as exploring region-based perceptual metrics (34] incorporating image-aware bits allocation, and leveraging conditional perception metric [18].
10

---

References
[1] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, "Scale-space flow for end-to-end optimized video compression," " in Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2020, pp. 8503-8512.
[2] R. Yang, Y. Yang. J. Marino, and S. Mandt, "Hierarchical autoregressive modeling for neural video compression." " 2020. [Online]. Available: |ntps://arxiv.oor///d///00000 10258.pdf
[3] O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, Lytle, and L. Bourdev, "Elf-ve: Efficient learned flexible-rate video coding," 2021. [Online]. Available: hutps://arkivorrgaabb/Z1111
[4] Li, B. Li, and Y. Lu, "Deep contextual video compression," in Advances in Neural Information Processing Systems, 2021, pp. 18 184-18 125.
[5] A. Golinski, R. Pourreza, Y. Yang. G. Sautiere, and T. S. Cohen, "Feedback recurrent autoen- coder for video compression," in Proceedings of the Asian Conference on Computer Vision, 2020.
[6] S. Zhang, M. Mrak, L. Herranz, M. G. Blanch, S. Wan, and F. Yang, "Dvc-p: Deep video compression with perceptual optimizations." in 2021 International Conference on Visual Com- munications and Image Processing (VCIP). IEEE, 2021, pp. 1-5.
[7] Mentzer, E. Agustsson, J. Ballé, D. Minnen, N. Johnston, and G. Toderici, "Neural video compression using gans for detail synthesis and propagation," in European Conference Computer Vision 2022.
[8] R. Yang, L. Van Gool, and R. Timofte, "Perceptual learned video compression with recurrent conditional gan," arXiv preprint arXiv:2109.03082, vol. 1, 2021.
[9] V. Veerabadran, R. Pourreza, A. Habibian, and T. Cohen, "Adversarial distortion for learned video compression," 2021. [Online]. Available: https::/arxxiv.or//pd///000 www.000...998..p..
[10] Y. Wang, P. Bilinski, F. Bremond, and A. Dantcheva, aG Disentangling appearance and mo- tion for video generation," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[11] Y. Blau and T. Michaeli, "Rethinking lossy compression:
The rate-distorrioon-prrreppoooo tradeoff," in International Conference on Machine Learning. PMLR, 2019, pp. 675-685.
[12] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool, "Generative adversarial networks for extreme learned image compression." " in Proceedings of the IEEE International Conference Computer Vision, 2019, pp. 221-231.
[13] J. Ballé, V. Laparra, and E. P. Simoncelli, "End-to-end optimized image compression," in 5th International Conference on Learning Representations, 2017.
[14] L. Theis, W. Shi, A. Cunningham, and F. Huszár, "Lossy image compression with compressive autoencoders," in Sth International Conference on Learning Representations, 2017.
[15] Mentzer, E. Agustsson, M. Tschannen, Timofte, and L. V. Gool, "Conditional probability models for deep image compression," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018
[16] G. Zhang, J. Qian, Chen, and A. Khisti, "Universal pate-distortion-pereppii representations for lossy compression," in Advances in Neural Information Processing Systems, 2021, pp. 11517-11
[17] C. Gao, T. Xu, D. He, Y. Wang, and H. Qin, "Flexible neural image compression via code editing," " Advances in Neural Information Processing Systems, vol. 35, pp. 12 184-12 196, 2022.
[18] Mentzer, G. Toderici, M. Tschannen, and Agustsson, fiigh-fidelity generative image compression." in Advances in Neural Information Processing Systems, 2020.
[19] E. Agustsson, M. Tschannen, Mentzer, R. Timofte, and L. V. Gool, "Generative adver- sarial networks for extreme leamne image compression." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 221-231.
[20] G. Lu, C. Cai, X. Zhang, L. Chen, Ouyang, D. Xu, and Z. Gao, "Content adaptive and error propagation aware deep video compression." in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer, 2020, pp- 456-472.
11

---

[21] Y. Blau and T. Michaeli, "The perception-distortion tradeoff," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. pp. 6228 6237.
[22] D. Freirich, T. Michaeli, and R. Meir, "A theory of the distortion- perceeton tradeoff in wasser- stein space," Advances in Neural Information Processing Systems, vol. 34, pp. 25 661-66116661 672, 2021.
[23] N. Saldi, T. Linder, and S. Yüksel, "Randomized quantization and optimal design with marginal constraint," 2013 IEEE International Symposium on Information Theory. IEEE, 2013, pp. 2349-22222 2353.
[24] Z. Yan, F. Wen, R. Ying, C. Ma, and P. Liu, "On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework," " in International Conference on Machine Learning. PMLR, 2021, pp. 11 682-81211 692.
[25] E. Agustsson, D. Minnen, G. Toderici, and Mentzer, "Multi-realism image compression with a conditional generator," arXiv preprint arXiv:2212.1 22x1:22212.13824. 2022.
[26] V. M. Panarctos and Y. Zemel, An invitation to statistics in Wasserstein space. Springer, 2020.
[27] A. Makur, Information contraction and decomposition. PhD Thesis, MIT, 2019.
[28] N. Ma and Ishwar, "On delayed sequential coding of correlated sources," IEEE Trans. on Info. Theory, vol. 57, no. 6, pp. 3763-3782, 2011.
[29] N. Srivastava, E. Mansimov, and R. Salakhudinov, "Unsupervised learning of video represen- tations using Istms," in International conference on machine learning. PMLR, 2015, pp. 843-852.
[30] Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of wasserstein gans," Advances in neural information processing systems, vol. 30, 2017.
[31] C. Schuldt, I. Laptev, and B. Caputo, "Recognizing human actions: local approach," in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.,
vol. 3. IEEE, 2004, pp. 32-36.
[32] E. Agustsson, D. Minnen, N. Johnston, J. Balle, Hwang, and G. Toderici, "Scale-space flow for end-to-end optimized video compression," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8503-8512.
[33] J. Li, B. Li, and Y. Lu, "Deep contextual video compression," Advances in Neural Information Processing Systems, vol. 34, pp. 18 1 18-188 125, 2021.
[34] E. Pergament, Tandon, O. Rippel, L. Bourdev, A. G. Anderson, B. Olshausen, Weissman,
S. Katti, and K. Tatwawadi, "Pim: Video coding using perceptual importance maps," arXiv preprint 12 10674, 2022.
[35] C. T. Li and A. El Gamal, "Strong functional representation lemma and applications to coding theorems," IEEE Trans. Info. Theory, vol. 64, no. 11, pp. 6967-6978. 2018.
[36] Stavrou, M. Skoglund, and Tanaka, "Sequential source coding for stochastic systems subject to finite rate constraints," IEEE Trans. on Automatic Control, vol. 67, no. 8, pp. 3822-38228888 3835, 2022.
[37] A. Khina, V. Kostina, A. Khisti, and B. Hassibi, "Tracking and control of gauss-markon processes packet-drop channels with acknowledgmentt." IEEE Trans. Cont. of Net. Systems, vol. 6, no. 2, pp. 549-560, 2019.
[38] A. El Gamal and Y. H. Kim, Network Information Theory. Cambridge University Press, 2011.
[39] E. Denton and R. Fergus, "Stochastic video generation with learned prior," in International conference on machine learning. PMLR, 2018, pp. 1174-1183.
[40] Y.H. Kwon and M.-G. Park, "Predicting future frames using retrospective cycle gan," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 181 1-120.
[41] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as perceptual metrie," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586-595.
12

---

[42] S. Hong, D. Yang, Y. Jang, T. Zhao, and H. Lee, "Diversity-sensitee conditional generative adversarial networks," in 7th International Conference on Learning Representations, ICLR
2019. International Conference on Learning Representations, ICLR, 2019.
[43] R. Yang and S. Mandt, "osssy image compression with conditional diffusion models," arXiv preprint (tti::2200.069550 2022.
13

---

A
Distortion Analysis for O-PLF-FMD
Recall the definition of Wasserstein-2 distance 26 follows. For given distributions Px, and Pxy"
let
(Px, Px,) :- inf Bi]lx; x,11),
(18)
where the infimum over all joint distributions of (X,, X)) with marginals Px₃ and Pxy-
Theorem
The set D (M)XK) characterized as follows:
Do (PM xK) = {D DD: 2 Epl1Xx X3,1 [ W3 Px,), j j= 1,2,33,
(19)
Furthermore, we also have that:
D (PMIXX) {D: DD 2 2EplIX,,- - - j j=1,2,3)
(20)
i.e., minimum achievable distortion with O-PLF-FMD most twice the MMSE distortion.
Proof: Define
D = (D: D3 B[ 3 Px,), j -= 1,233.
(21)
First, we show that D (Pa)xk) S D For any D E D (PM)XK), there exists &pa (X X Dg, x og) jointly distributed with (M,X, K) such that
X|| XD DΦ 11 $ Dj.
== 1=
(22)
Px
Pxpg'
(23)
Then, for example, the analysis for the second frame is as follows
D2 2 X2 - X8 bg [²]
(24)
[X -x2) (X Dg
(8pg- X2)1
(25)
= X2 x211 + + x [1
(26)
2 X2 x211 1W3(+ Pxog
(27)
= =I|I - xx217++ 5 WPx,, Px.),
(28)
where (26 holds because both X2 and X are functions of (M1, M2, K) and thus the MMSE
(X2 -x2) is uncorrelated with (Xpg X2): (28 follows because the O-PLF-FMD implies that
Px.g = Px, Following similar steps for other frames, we get & (PMIX)) S D
Next, we show that D S Pp (PMIXK). Assume that E DD Let Xi be an auxiliary random variable jointly distributed with (M1, K) such that it satisfies the following conditions
Px; Px..
(29)
and
Px arg inf Px,xę
Bp Xi11%
(30)
Px, Px, Pxp=Pxt
Moreover, let X3 be auxiliary random variable jointly distributed with (M1, M2, K) such that the following two conditions are satisfied
Px; = PX..
(31)
and
P P,x; arg
inf 8x,89'
TP X₂ 8.1131.
(32)
Px2 =Pxg
Px: Pxg
14

---

Similarly, we define X3. Now, notice that since D E D" we have:
D2 B x2 W Pxxx Px.)
(33)
It then directly follows that
1= X2](] E|lIX; X2|1 + B1|| X211
(34)
EllIX2 X211 + Wpx Px.)
(35)
ElIIX2 X211 + W2(Px. Px.)
(36)
s D2.
(37)
where
follows because X2 and X3 functions of (M1, M2, K) and thus the MMSE (X2
uncorrelated with (Xg- X2):
follows from 32
36 follows because Px; Px2-
Following similar steps for other frames, we get D E $ (xxjx)-
Now, notice that W Px.) S B|I X2 since the Wasserstein-2 distance takes the infimum over all possible joint distributions (X2, X2), but the expectation in [[||X2 X2 || is taken over the given x,,x,' Thus, we get
ElI X2113] W/ (Px. Px.) S 2|II| X2113).
(38)
This concludes the proof.
B Distortion Analysis for O-PLF-JD
Let Xt be defined as in 30). Moreover, let X; be an auxiliary random variable jointly distributed with (M1, M2, K such that the following conditions satisfied
P Pxy18==n PxaIXi=011
Vz1 X2.
(39)
and
Pxax;181=m = arg
inf Pxy8g187=8
Epll 831118 = x1),
Vol Xi.
(40)
Px xf==1 =fx2 x1 Fx 81-1 = =fxg 81-11
Then, the following result holds.
Theorem We have
Loin (PMIXK) {D: D2 E||| 1 Pxi).
D₂ 2 Flll X22 8211 + Σ Px (z) [P( Px.IXi=n.),
D3 2 E - x Σ Px,x, (2,, xxx1
x Px ).
21.22
(41)
Proof: Define
Do
Join (D: D1 > E|| - X W//Ps
D2 2 E 8211 + Σ (2) (Px "x PxIIX
21
D3 E|I x Σ
- Px.lx
21.22
(42)
15

---

Now, assume that D E D For the first frame, recall that Xi is an auxiliary random variable jointly distributed with (M1, K) such that it satisfies then follows that
30 From similar steps to 34p
36).
- x111 -
FII + Px.)
(43)
D1.
(44)
For the second frame, since D E Do joint we have:
D2 2 x2
x211 +
E Px, (2) (2))) Pxo(X
(45)
21
Recall that X5 is an auxiliary random variable jointly distributed with (M1, M2, K) such that 40p hold. It then directly follows that
39
TF| X
x = F|lI - X211 + F[1xX, X211
(46)
BillX2 - X221 + Σ Px; (z) [1)]B|IIX2 X211 = 21
(47)
E1llX² - 82119 + Σ
Pxp (z1) (P Pxg (x==z))
(48)
21
= EIIIX, - 8211 + Σ Px (x) W3 Xa1i;=11' PxiIX
(49)
where
46
follows because X2 and X3 are functions of (M1, M2, K) and thus the MMSE (X2
is uncorrelated with (Xg- X2),
follows from 440
follows because Px xx; x = Xx,X,.
Following similar steps for the third frame, we get D E 4 (PM)xx) This concludes the proof.
B.1 A Counterexample for Factor-Two Bound in Case of O-PLF-JD
Assume that we have only two frames, i.e., D3 -> xo. Let Mj be independent of X1 and M, = X2. Then, we have X; = 9 and X₂ = X2- Consider the achievable distortion region Theorem The distortion of the first step is given by the following
x X 11 W2 Px,) = 2
(50)
For the second frame, we have
E|I|X2 X211 Σ
(x P PxiIX
21
(x) (P2,)t:-m Pxg
(51)
Px (x) Px.Ix
(52)
where (5T) follows because X2 = X2 and 52 follows because X₂ independent of Xi (M₁ is independent of X1, then XT, which is a function of (M1 K), would be independent of X, and hence
independent of X2).
Now, notice that the MMSE distortion of the second step is zero since X₂ = X2. However, the achievable distortion of the second step for the reconstruction satisfying O-PLF JD is given in (22 which clearly does not satisfy the factor-two bound.
16

---

C Fixed Encoders Operating at Low rate regime
We consider the class of noisy encoders where the encoder distribution be written follows
P phoisy
X, MMMMM.MM M,K (1 Px) +
pQ x,
j = 1,33
(53)
MI,MM. K
where j is sufficintlly small constant and the distribution Quoisy (.) could be arbitrary conditional distribution with same marginal as Pxy'
Theorem For the class of encoders given by we have
M 2 (D DD:Da Dj 2 2I poaay
X x j j= .3.
(54)
Proof: We analyze the distortion for the second frame. A similar argument holds for other frames.
Denote the reconstruction of the second step by Xi and consider the expected distortion. From similar justification starting from (24) and leading to (26), we can write the distortion as follows
[1 X || E|IIX2 X21 1² [[[I X2
(55)
Now, we study the expected term El X2 X211 follows
F F[
X; = Σ Pxp 1111 - 12121 = m1)
(56)
In order analyze the above expression, we first approximate the MMSE reconstruction X₂ as follows
X₂ = puic M1 M2, K] (1- - pp + p.u [X2 M2, K}]
(57)
(58)
E[X²] + O(H),
(59)
where (58) follows from
53 Moreover, notice that 59 implies that
E X2 X21l²] X|||X2 - X2 + y I Equa ( M2, - (E[x))))
(60)
E|| X2 x 11 + O()).
(61)
Next, consider the expected term in (56) as follows
Σ
Pxt (21)
1 = -11 = Σ P - 1 x11 + O(4)
(62)
Σ (x)
X2 x1] +(u))
(63)
Σ (x) - X211 = x1 + +(u)
11
(64)
- - ElJ|X2 - 1 +0())
(65)
(66)
where
62
follows from (5
63
follows because the O-PLF-JD implies that PxşI8; Px.x, and [[[x2] is just
constant;
64
follows from (-PLF-JD where Px: = Pxi
(66)
follows from 61
Considering
55
and (66). get
|| - - O(z).
(67)
The proof for the third frame follows similar steps.
17

---

X
8
Representation
Resuastruction 1
X
X
X11
Nra N
Representation 2
Rermsstuction 2
Xs
Xy
Xx N X2
& X
Representation
Rexonstruction 3
Figure Encoded representations and reconstructions the iRDP region CRDP-
D Operational RDP Region
Recall the definition of iRDP region CROP for first-order Markov sources (Definition 4) as follows. the set of all tuples (R, D, P) satisfying
R1 2 [(1;; X,,1,
(68)
R2 2 [(2;; Xr.2 2
(69)
R3 2 [(x;; Xr,3 [x,,,, x1,2)
(70)
Dj > B[||X; -X,11
j = 1,2,3,
(71)
P3 2 Φ (Px1..X,, P Px &g.n,y)
== 1,2,3
(72)
for auxiliary random variables (X,,1 1 ,, Xr,3) and (81, X2, Xz) such that
X1 7/1 (Xr,1), 82 122 (Xr.1, X,,2), 83 Xr.a,
(73)
X, + X1 - Xz).
(74)
X, + (X2, X,,1) ) + Xz),
(75)
X,,3 + (X3; Xr,,, Xr,2) (X1, X2),
(76)
for some deterministic functions mı (.) and (2...))
Theorem For first-order Markov sources, a given (D, P) and R E R(D,P), P), we have R + log(R + E R
(77)
Moreover, the following holds:
(78)
Proof: Before stating the achievable scheme, we first discuss the strong functional representation lemma (35). states that for jointly distributed random variables X and Y, there exists a random variable U independent of X, and function á such that Y @(X, U). Here, U is not necessarily unique. The strong functional representation lemma states further that there exists a U which has information of Y in the sense that
H(YUU) S I(X;Y) + log( +)++
(79)
Notice that the strong functional representation lemma can be applied conditionally. Given PxYIW, we can represent Y function of (X, W, U) such that U is independent of (X, W) and
WW, < (X; log + 1)
(80)
Proof of 777 (Inner bound):
For a given (D,P) and R E R,D,P), let X,
[X,,1 Xr.2, X,,3) jointly distributed with 766) hold and the rate constraints in (68) 770)
X (2, Xa) where the Markov chains
18

---

X1
X
Mi E {0.1}"
(1)
compress
decompress
Recons.
Encoder 1
Decoder 1
2/2
X2
X
M2 E (0. 13
Xx.2
(2)
compress
decompress
Remons.
Encoder 2
Decoder 2
Figure Strong functional representation lemma for T 2 frames.
are satisfied such that there exist (X1, X2, Xz) for which distortion distortion-perception constraints
hold. Denote the joint distribution of (X, X,, X) by Pxx.x and notice that according the Markov
chains in (74) (76), it factorizes as the following
Pxx, Xx,x,X, Px.axx, PxaalXr.xx P
1(8 = (X,a)) 1(82 92 x,333 1(83 Xr,3).
(81)
For an illustration of encoded representations X, and reconstructions X in R(D, which induced by distribution xxx,x. see Fig.
Now, we show that + log(R + +)+++++ + R)D The achievable scheme is as follows. Fix the joint distribution Px. according to 81 which constructs the codebook, given by
Px. Px,, PX XX.
(82)
From the strong functional representation lemma
35 we know that
there exist random variable V1 independent of X1 and a deterministic function şi such that X,,1 = Q1 (X1, V/1 and
(X,,, & I(X1: + Xo, +1)
(83)
which means that the first encoder observes the source X1 and applies the function şi get Xr,1 whose distribution needs to be preserved according to (82) (see Fig, 6
according to the conditional strong functional representation lemma, there exist a random
variable V2 independent of (X,, and deterministic function 92 such that X,
92(X,,1. X2, V2) and
( V22 & [[x2; X + log(!(X2;
Xx,2 Xx, + +
(84)
At the second step, the representation X,, is available at the second encoder. So, upon observing the source X2, it applies the function 42 to get X,. whose conditional distribution
given X,, needs to be preserved according to (82) (see Fig. 6
according to the conditional strong functional representation lemma, there exist a random variable V3 independent of (X3, Xr,1, Xx;2) and deterministic function qa such that X,,3 ra(X,.1, Xa, Va) and H(X,,3/X,,1, X. V/)= < I(Xa: X X,2) +
X,.2 + 1) + 4.
(85)
Now, the encoding and decoding are as follows
With V, available all encoders and decoders, we can have class of prefix-free binary codes indexed by Vi with the expected codeword length not larger than I(X1; X,,1) log(T(X,;; X,,1 + + represent X,,1, losslessly (see Fig.
With V/2 available at the encoders and decoders, we can design a set of prefix-free binary codes indexed by (V2, X,,) with expected codeword length not larger than [[X2; 2/ (x X Xr.)) 5 to represent X, losslessly(sce Fig, 6)
19

---

Similarly, one can represent X4,3 losslessly with V33 available at the third encoder and decoder.
The decoders can use functions X1 (X,,)), &2 122 xx and X3 Xr,3 to get the reconstruction X.
This shows that R + log(R 1) 5 5 ().
Proof of 78 (Outer Bound);
For any (), R RD, P), shared randomness K, encoding functions fj: X1 x. x x
x
Mj and decoding functions Oj: Mi x x2 x x Mj x KC + R3 such that
R3 2 E[e(M)),,
j = 1,2,3,
(86)
and
Dj - P (Px P,....))))
j = 1,2,
(87)
== 2,
(88)
we lower bound the expected length of the messages. Define
X,,1 = (M1, K),
(89)
X, (M, M2, K),
(90)
and recall that according to the decoding functions, we have
X, Jj (M ................................ M,. M,,K),
j j= 1,2.3.
(91)
We can write
R1 2 B[((M))) 2 H(MIK)
(92)
= =I(Xi; MIlK)
(93)
= ((XI:M,,,, T(XI; Xr.1)-
(94)
(95)
Now, consider the following set of inequalities
R2 2 B((()2)) 2 H(M2)M1, K) =(X,, X2; M2 K) = I[X1, X2, X,,1).
(96)
(97)
(98)
Similarly, we have
RB M B[((Ms)) 2 H(M M2, K)
(99)
= =[(X, X2, Xs; Ma M1, M2, K) 2 I(X,, X2, X3; Xal X,,,, X,,2)-
(100)
(101)
Notice that the definitions in 90 imply the following Markov chains On the other hand, the decoding functions of the first and second steps imply that
Xr,1 + X1 + (X2, X3), X, + (X1, Xr.1) X3.
(102)
(103)
X = 91 (M1, K):
(104)
X, (M M2, K),
(105)
where together with definitions in (89 and 900. we can write
X gi (M,, K) = (Xr.1,, A² (M1, M2, K) = 72 XT,2),
(106)
(107)
such that (.) and ())...) are deterministic functions.
20

---

Now, consider the fact that the set of constraints in
87
88
(95)
98 (OOT)
with Markov chains in
(022 {T3) and deterministic functions (T6)
107
constitute an iRDP region, denoted by CROP.
which is the set of all tuples (R, D, P) such that
R1 2 T 7(X1;X,,11)
(108)
R2 2 I (X1,X2; xx (x,1,
(109)
R3 2 [2, X3i X3 (r x,,2),
(110)
Dj 2 B||| - 1,1
= 1,23,
(111)
P3 2 6 (P'x Px P,
j = 1,2,3,
(112)
for auxiliary random variables (X,, ,, x2) and (X1, X2, Xz) satisfying the following
X1 = m (Xr.)): X2 = 172 (Xr,1,
(113)
Xr,1 X1 (X2, Xz),
(114)
X + (X1,
+ X3-
(115)
for some deterministic functions 7/2 (.) and ()....).
Comparing the two regions CROP and CRDP, we identify the following differences. The Markov chain in (74) is more restricted comparing to (113). Moreover, the Markov chain (75) does not exist in
CRDP. The following lemma states that CRDP = CRDP. Now, for given (D, P). let R(D, P) denote
the set of rate tuples such (R, P) CRDP, then this lemma implies that R P) R(DD P)
which completes the proof of the outer bound. Moreover, notice that the above proof only deals with the statistics of the representations and reconstructions and does not depend on the choice of the PLF So, it holds for both PLF-FMD and PLF-JD. This concludes the proof.
We conclude this section by the following lemma.
Lemma 1 For first-order Markov sources, we have
CROP = CRDP-
(116)
Proof: This result for the scenario without perception constraint has been similarly observed in [36 Eq. (12)). The proof in this section is provided for completeness.
First, notice that the set of Markov chains in (74) (76) is more restricted than the ones in [14 115). hence CRDP CRDP. Now, it remains to prove that CRDP S CRDP. Consider the following facts
1. The distortion constraints in 1111 depend only on the joint distribution of (X,. X)): and thus on the joint distribution of (X,, ...,,.......
So, imposing the Markov chain
Xr.2 - (X2, Xx,)) + X, does not affect the expected distortion [E[][X2 - X21 since does not depend on the joint distribution of X1 with (X,,,, Xg). similar argument holds for other frames;
2. The perception constraints in (12) depend on the joint distributions Px,...X; and Px, (hence on Px.. Thus, imposing Xr.2 - (X2,X,,) - X1 does not af- fect di (2 P xx,), since it does not depend on the joint distribution of X, with (Xr,1. X2). A similar argument holds for other frames;
3. Moreover, the rate constraints (09 and 110 would be further lower bounded by
R₂ 2 [X1, X, 2 Xw.) Z I(X2; Xr.)
(117)
R3 2 T(X,, X2, N3; Xg/X,,, X,,2) 2 T(X3; Xz/X,,,, X.2))
(118)
Thus, the set of rate constraints is optimized by the set of Markov chains 774 (76).
4. The mutual information terms ((X1; T(X2; Xr and T(X3; Xr,, depend on distributions Px X "x,,,X,,,X5 and Xx distributions should be preserved by the set of Markov chains. The first two distributions are preserved by the choice of -77). Now, since we have first-order Markov sources (see Definition 3) preserving the joint distributions of Px., X2 and Px.,,X, 2 is sufficient to preserve the distribution Px,, Xx,,xX So, preserving the joint distribution of P X Xr.2
respectively. So, these
is sufficient to keep [(X3; X3 1x,1, X,,2) unchanged.
21

---

Considering the above four facts, without loss of optimality, one can impose the following Markov chains
X, + X1 + (X2, X3),
(119)
xr (X2, x,,1) + (X1: Xz),
(120)
X3 (X3, Xr,1, X2,2) + (2),
(121)
This concludes the proof for the PLF-JD. For the PLF-FMD, notice that the only difference is the second fact stated above. But, this also holds since the perception constraints depend only on Px, and Px, (hence Px,.-..X...)
E
Gauss-Markov Source Model
We first remark that the W distance can also be replaced by the Kl-diivegeene in most of the following analysis. The common properties between these two measures are convexity and the fact that they both depend on only second-order statisties when restricted Gaussian source model.
Theorem For the Gauss-Markov source model, any tuple (R. D, P) E CROP can be attained by jointly Gaussian distribution over (Xr,1, Xx, Xr,3) and identity mappings for 77(:) in Definition 4
Proof: First, notice that a proof for the setting without perception constraint is provided in [37]. The following proof is different from 37 in some steps and also involves the perception constraint.
For a given tuple (R, P) CROP. X X Xi = (X;,)) X; = (X X and X3 be random variables satisfying 173--75) Let P Pxş x and X be jointly Gaussian distributions such that the following conditions are satisfied.
cov(X X1) = (o
(122)
Cov(X X2 X2) cov( cov(Xi, 8 X2),
(123)
cov(X? X9, Xg. Xa) = cov(Xi, X2 X3,
(124)
In general, the Gaussian random variables which satisfy the constraints in in the following format
122
124 can be written
X1 vx + Zı,
(125)
X2 = w1 X2 + w2X, + Zz.
(126)
XC X1 + x28 XG + +33 + Z3,
(127)
for some real v, W1, W2, TI, 72, T3 where Xe N'O, xxx) X2 N(0,
xg)) Zi Z2 and Zg are
Gaussian random variables with zero mean and variances o o². oz. independent of XX. (&f X²) and (X Xx), respectively.
We explicitly derive the coefficients v, WT, w2, TI, 72 and T3 in the following. Multiplying both sides of (15)) by X and taking an expectation, we get
F x = DO x
(128)
According to 122 the above equation can be written as follows
E[X, )){
(129)
Multiplying both sides of (26)) by the vector [XC X2) and taking an expectation, we have
1[X2X9]
E
E
E[X2X111 = [w w2]
(130)
x
o
Considering the fact that F.. be written as follows
x pr Fx X and according (133
the above equation can
EE
X
A B [[1
E[X2X;]|
= [w wel
(131)
03
22

---

Similarly, multiplying both sides of 117 by the vector (X X Xsl, taking an expectation and considering 124 we get
B [[Xt3]
X[X E[X;-]
P1P2E[X,
[
Xgl
Xg1
= In T2 Tal
[X2
P1
Xil
P2 [x2]]
Ex E[X32]
(132)
Solving equations
129 p. [31
and 132
we get
(133)
x
1/ E| X1I
(134)
= :
af = of X[X
(135)
W1 up x Xi] - F
(136)
2²p200 8 - 02
0 2
EE 2x2x
W2 X
(137)
u² pi xf
X4 0200
XP
a - - 0 Xp
-
ww po X
(138)
For the third step. the coefficients and noise variance [12
are given as follows
[n 72 T3]
E| B[ X 11 [1p2E[XXX]]
E|X;X;1
[[ B[X;X1]
E 1|
P2 [xxx]
E[X39]
(139)
a3 - - - - 1
-
- 27 1313 X1] 2T 17233p21|X2XX
(140)
where (.)-1 denotes the inverse of matrix.
Now, we look at the rate constraints.
Rate Constraints:
Consider the rate constraint of the first step as follows
R1 X(x1;X;,)
(141)
= H(X,) (X)1X;1)
(142)
2 H(X)) (X)
(143)
H(X)) H(X, E(XL&11L
(144)
2 H(X,) H(X, E[X, (811)
(145)
-(X1)) H(X, E[X,199)
(146)
=I(Xi) μ(X, -[XL/88J189
(147)
x(x
(148)
where
143
follows because X1 is a function X;
follows because for a given covariance matrix in 112 the Gaussian distribution maximizes the differential entropy;
146
follows because the MMSE is uncorrelated from the data and since the random variables are Gaussian, the MMSE would be independent of the data.
147
23

---

Next, consider the rate constraint of the second step as the following
R2 2 XXx;
(149)
HXx2)X))) - (X2 X Xx,,, Xi:2)
(150)
2 ((x)XX))) - (X2/
(151)
2 H(X)(X))) - -XXX (20)
(152)
H(PIXI + +NI(XA)) HXX//XX X29
(153)
log 2 22 (X x9)
+
(154)
2
log (f22-2R12H(X)) + +22 -
(155)
2
where
151
follows because X and Xg are deterministic functions of X², and (X), X7,2
respectively;
152
follows because for a given covariance matrix in 123 the Gaussian distribution
maximizes the differential entropy;
(154
follows from entropy power incquality (EPI) 38 pp. 22];
155 follows from (142).
Similarly, consider the rate constraint of the third frame as the following.
R3 2 Xx X2
(156)
= -H(xs|X;,, X;2) I, X3)
(157)
2 1x, H(xs 82, Xz)
(158)
2 H(xx|XX,,, X2,2 - (Xxx 86. X8)
(159)
H(p2X2 + N2|X x7:2 H(Xal89 1,,,,,,,8 X9)
(160)
2 log P2
+ (2H( - HXx E X9)
(161)
p22 2n 22 x;a) + 222M(Na) H(x288f X9. X99)
log
(162)
log (0ip32 28, 22 + +232 2322-222220 202282222H + 22H((Na) HXXxXXXX XG, X9)
(163)
Next, we look at the distortion constraint.
Distortion Constraint: The choices in
122
124 imply that
Dj 2 -
- Ell - 199111
j j= 1,
(164)
Finally, we look at the perception constraint
Perception Constraint:
Define the following distribution
Po-v- == arg inf Pov: Po=Px, Py=Pxt
EpllU V.
(165)
Now, define Praya to be Gaussian joint distribution with the following covariance matrix
Cov(UG vG) = cov(U", V*).
(166)
Then, have the following set of inequalities:
24

---

Pi 2 (2
inf EpllU v
(167)
Pov: Px=Pxx Pv=Pxt
-IIlU' V*11
(168)
E 1
(169)
W2 Pvo)
(170)
inf EpllU -V
(171)
Pov: Pa=Puc
Pv=Pvo inf
EpllIU - V11
(172)
Pov:
Po=Pxi
Av=PxG
(Px,
(173)
where
(68
follows from the definition in (165);
(69 follows from 166 which implies that (U*, V*) and (UG, (UG,VO) have the same second- order statistics;
follows because Pya Px which is justified in the following. First, notice that both Pye and Pxc are Gaussian distributions. Denote the variance of VG by ot and recall that the variance of X is denoted by oxp. According to (166), of is equal to the variance of V Also, from (165), we know that Pv- = =xx;- hence the variances of 7
172
and Xj are the same. On the other side, according to (22). we know that the variance of Xi is equal to xxx Thus, we conclude that o² X ova. which yields Pye Pxe. A similar argument shows that Pye Pxi-
A similar argument holds for the perception constraint of the second and third steps for both PLFs.
Thus, we have proved the set of Gaussian auxiliary random variables (X (9, given in (25)
127 where the coefficients chosen according to distortion-perceptioo perception constraints, provides an outer bound to CRDP which is the set of all tuples (R, D, P) such that
R1 > X9),,
(174)
p 222-2R193 ((H() 22H(NN) H( X9).
R₂ 2
log
(175)
R3 2 2
log
(pip32 2R -2 + 032 2Ro22222HH(1)) + 22H(N))) (Xx
& X))
(176)
D, 2 FIIIX, - 13)1
j 1,2,3
(177)
P3 2
P Pxp...... Px....xa)
(178)
Now, need show that the above RDP region is also inner bound to CROP. This is simply verified by the following choice. In iRDP region of (68) (76), choose the following:
X
X,
1,2,3,
(179)
where (XC, Xx) satisfy 125
127) with coefficients chosen according to distortion constraints. The lower bounds on distortion and perception constraints (17) and (18) are immediately achieved by this choice. Now, will look at the rate constraints. The achievable rate constraint of the first step can be written as follows
Ri X(XI;
(180)
25

---

which immediately coincides with (74). The achievable rate of the second step can be written as follows
R2 2 1X
(181)
- H(X
(182)
H(PIXI +
X2)
(183)
1og((7220(21189) + (2H((N) - H(22X81 x2)
(184)
2
2 log (0t2 (22-22119 22H((1) 22HH(M))) (X X9),
1
(185)
2
where
84
follows because EPI holds with "equality" for jointly Gaussian distributions 38 pp.
221;
(185
follows from (175).
Thus, the bound in (185) coincides with (55). A similar argument holds for the achievable rate of the third frame.
Notice that the above proof (both converse and achievability) can be extended to T frames using the sequential analysis that was presented. Thus, without loss of optimality, one can restrict to the jointly Gaussian distributions and identity functions (7(.) and (2( in iRDP region CRDP.
For given rate R, the following corollary provides the optimization programs which lead to the characterization of the DP tradeoff (R) for the Gauss-Markov source model.
Corollary 1 For a given rate tuple R and T 2 frames, the optimal reconstructions of the DP- tradeoff (R) can be written as follows
X = vx = + 2n,
(186)
X = w X + w2 +
(187)
where Zi (resp Z2) is a Gaussian random variable independent of X1 (resp (X,,X2) (0, ) for = 1, 2, and v, w,w1,2 o7, 2 are the solutions of the following optimization program for the first step.
X2)) and XO ~
min of 6 -2vo,
(188a)
s.1.
io S 01 - -2- -2-2R:). )
(188b)
61) SP.
(188c)
and the following minimization problem for the second step and PLF-FMD,
min o, -
2ww1P10102 - 2wz
(189a)
3 - 2-R 2²0200 s.l.
< (03 woot 20102((11020(1 - 2 2-22s), (189h)
of
0z)= < Pg.
(189c)
or the following minimization problem for the second step and PLF--DD,
min
02 + 0² 2ww1p10102 2wzo?
(190a)
2n, 2² 02002
3 - 2n,
< (0z wîo 201 2 -20)), (190b)
s.l.
07
tr(E12 + E12 2(E1/2
1/2
E
1/211/22 s P2.
(190c)
where tr(.) denotes the trace of matrix and
of 010102
P10102)
Σ1 ==
(191)
of
of
w1 + vW2p10102
Σ :=
(192)
WiO?
vuppio102
83
26

---

Proof: We obtain the optimization programs for T = 2 frames as follows.
For given rate tuple R, the DP-tradeoff (R) is given by the set of all tuples (D, P) such that there exists X satisfying the following Markov chains
X X1 X2, X (Xf .2) X1:
(193)
(194)
and the following conditions,
R1 2 (()
(195)
R₂ 2 (+9)
(196)
and
Dj > - j j= 1,2,
(197)
P3 2 (Px PXG....G).
(198)
In general, the set of reconstructions that satisfy 193
194
can be written as follows
X vX1 Z1,
(199)
X X.
W2X2 + Zz.
(200)
Plugging the above into (195) and (96) yields the following rate expressions
07 R1,
log 01
S
(201)
V²O
03 (wi²)
22 j2
log
R2
(202)
2
02 w w 201W2rp10102
Re-arranging the terms in the above constraints yields the conditions in (188b) and 190b). Consider-
ing (97) with (199) 200 gives the following expressions for distortions
E|| X1 - x o7 [||| X2 - X + +
21E [ = of + o7 2vo?.
(203)
21 X2X X = of + 6 20i0p00 2w202
(204)
which are the objective functions in 188a and (90a). Now, we evaluate the perception constraint. Notice that the covariance matrices of (X1,X3 and (x)) are given by 1 and Σ defined in (191) and (192). respectively. The Wasserstein-2 distance between Gaussian distributions with
covariance matrices Σ1g and L12 is given in (190c) discussed in 26 pp. 18].
Similarly, the expressions in
189 for the decoder based on PLF-FMD can be obtained.
F Gauss-Markov Source Model: Extremal Rates
In this section, we derive the achievable reconstructions for some special cases. We assume that we have only two frames, i.e., D3, P3 + co. Moreover, let of = oî == o² for simplicity. In general, the reconstructions can be written as follows
x v + +1,
(205)
X wIXX W2X2 + Zz,
(206)
where X ~ NNO, 83) for j = 1,2. Recall the optimization program of the first step in [88 as follows
min 0² + 07 2uo m
(207a)
s.t.
120² s 01 2 212 ) (0-ô1) 3² < P1,
(207b)
(207c)
27

---

For given 0f. the objective function in (207a) is monotonically deacreasing function of v. hence one can restrict v to be nonnegative, without loss of optimality. So, the above optimization program can be written as
min 0² + o -
(208a)
0 5 σ1 / 2 2-281
s.t.
(208b)
(o 01)² - P1,
(208c)
Optimizing with respect the above program, we have
2-
(209)
where the optimization program reduces to
min 0² + 0 20ō - 2-28:
(210a)
s.t. - =
(210b)
Next, recall the optimization program of the second step for PLF-FMD in 189) as follows
min
σ² + o² 2vwip10 2uzo²
(21la)
Wh 03
v² piO²
s.t. 3 2-222
S (02 wjoo 22) 2 2-28,),
(211b)
07
6σ- 0²)² S P2
(211c)
Plugging 209 into the above program, we get
min O² + 5タ 201p1010v112 --200/1--221101 2-21 -2w20² #2
(212a)
2 (1 p72 211 (1 - 2- 22-2))) s (0) wGoo 20uuppi1oo 2-281
(1 2- 2
s.t.
(212b)
0z- < P2,
(2 (212c)
The optimization program for the second step of PLF-JD is similar to the above program 212 when (21cc) replaced by [190c). In this section, we study different rate regimes and obtain the solutions of the above optimization programs. In particular, we are interested in two perception thresholds P2 + oo and P2- = 0 where the former corresponds to the classical rate-distortion region and the latter is the case of O-PLF. For the O-PLF-FMD, we have ō 22 = a. For the O-PLF-JD, in addition to preserving the marginals, the correlation E E X aia² should be satisfied. For each of these cases, the optimization program in 212 simplified in the following.
Optimization Program of the Second Step for P + oo: In this case, there is no perception constraint in the setting and the optimization program in (212) reduces to the following
min 0² o²- 2211p1000 - 2 28, 2w0y0²
(213a)
12 2022
s.t.
3 (1 23220222000 211 (122-20))) < (0² wjot 1 1-2
2-2R )1 2 -22)).
(213b)
This case corresponds to the classical rate-distortion tradeoff' where it is shown that for a given rate,
the MMSE reconstructions are indeed optimal 28 37 The expressions for MMSE reconstructions
are given in Appendix H.I
Optimization Program of the Second Step for D-PLF-FMD In this case, we have à, = =ō² 5. So. the optimization program in (212) reduces to the following
min 20 - 201p10² V1 1-2-2216 2-22 2wzza Wi W2
(214a)
51 - 2²2 281 (1 2 2-2 )) s (1 w 2w1 V1-2 1-2-211 ((1 2 -10 ).
s.t.
(214b)
28

---

Here, W1 and W2 only need to satisfy the rate constraint given in (214b) which represents a larger search space than that of O-PLF-JD which will be discussed in the following.
Optimization Program of the Second Step for O-PLF-JD: In this case, addition to preserving marginals 5 62 o, we need to satisfy the constraint E[ X ~10². Thus, the opti- mization program of this case has an extra condition W1 + DW2p1 = f1 comparing to (214) and is given follows
min 20 - 22 /1 1--2 2-22 2wrro"
(215a)
Wi:2
s.t. w p¹² 2R (1 2 2-2 )) s (1 w 2w/ V1 2-2 (11))(11))) 2-282) )
2
VW2p1 P1.
(215b)
Comparing (215 with (214), we notice that the search space of the optimization program for O-PLE- JD is smaller than that of O-PLF-FMD. Thus, a larger distortion is expected for O-PLF-JD.
Before studying each case of extremal rates, we introduce another constraint the optimization program of all above three cases of perception metrics. We restrict nonnegative w/d//1 and get an upper bound on the programs (213), (214) and (215). So, in further discussion on these programs, the constraint W1 0 will be also considered.
Rj R₂ e for small
In the low-rate regime, notice that we can approximate the rate term as follows
2-
2c
2e In 2 + O(e)).
(216)
Plugging the above into 209 we have
2e 2e
(217)
Also, inserting (216 into the rate constraint of the second step (11c) yields the following
w 22 + & (0) अî 201 2 (22 ((+))
(218)
Re-arranging the terms in the above inequality yields the following
22 2 w3 - 1f2 In O(E2)) + wff + 2wnopprioo/2eh22+000
(219)
2eln+0((e
= w3o²
1n +0 +
(220)
So, in all of the optimization programs of the case Rí = R2 = e, the above constraint 220 will replace the rate constraint of the second step.
Now, we consider different cases based the perception measure.
a) Without a perception constraint: In this case, using 216). the optimization program of the first step in 210 simplifies to the following
D min of
o² + 20δ / 2e 1n2 2n2+
(221)
which gives us the following optimal solution
δ 2 V2e 20 + O(e).
(222)
Plugging the above solution into 217 and (211) get
2n 0(#))
(223)
"The inequalities the form f(e) O s(c) 0(e), where f(c).g(e) 2(e²), imply that f(c) < g(c). So, in such inequalities, we work with dominant terms (f(c), and ignore the small terms ( A similar argument holds if we have other orders of and the functions f(.).g()) approach zero slower than them.
29

---

and
D1 21 + 0(2)).
(224)
Now, we look at the optimization program of the second step (213) For a given wi and w2, the objective function is an increasing function of 03. optimizing over 03 yields the following
2 = =
+ +0(1) wfof + 201w2/1010 2e In2 O
(225)
2n
Thus, the optimization program (213) is further upper bounded by the following
min 62 12: warg 2143P120
o² + wgo²
+01)) + wfff 2(1 2etp://nno102222222222222 22
2wzo
2c
(226)
The optimal solution of the above minimization is given by the following
W1 = P1 + O(c), W2 2n
(227)
(228)
Thus, considering the dominant terms of (223) 227 and (228). we have
X (2e In X1 + Z1.
(229)
X X + (2e In 22 + Z2
(230)
and Z3 NN 200 2) for j Notice that
D1 (1 2e In 2)0²
(231)
D2 (1 (1 1i)
(232)
DJ-P..FFMD In this case, we have 0 = 02 = a. For the optimization program of the first step,
209 reduces to the following
== v 2e In + O(e),
(233)
and Dj is given in the following which is derived by 210
D = 2(1 v 2e 2 + O(e).
(234)
Now, we study the optimization program of the second step. The optimization program of (214) is further upper bounded by the following
min
20 2w1p10² 22 2wzo²
(235a)
Wiw2P:20
20
12
w/
+0(1)) + ww 2w1 2m 0(2))
(235b)
2e
Now, we further simplify the inequality this inequality implies that
(235b) in the following. Considering the fact that WhW 2 0.
w < 1,
(236)
w/ < 2c 2
(237)
So, using the above inequalities, the RHS of (235b) can be upper bounded as follows
+0(1))
w/ + 2w v 2 O(e))
2e
2
+0(1) + + (wf + WH)Pr 2c
w3
2
<
w/
+ +0(1)
+ +wf +0631))
(238)
30

---

Now, according to 238
the optimization program in 235 is further upper bounded by the following
min wh 1092
20 - 2WIP10 2
- -2w20²
(239a)
20
2c +0(1) + +
S.L.
(239b)
For given wi (resp w2), the objective function (239a) is monotonically decreasing function of w2 (i),
wi), so the optimal solution is attained on the boundary, i.e.,
ln + +01)) wf + O(63/2)
(240)
Thus, the program (239 further simplifies to the following
min 20² 2wi p 1n2 2n
2 0 - 20² v(1 - - (1==3f-0@22)))ehn2+000
Wi: wh 20
(241)
The optimal solution of the above program is given by
P V1+ + p
Oc)),
(242)
which together with (240) yields
2c 2
W2
+ O(c).
(243)
1 1 pi
Thus, considering dominant terms of 233
242
and 243
we get
X v 2c In
2X
+ Zr,
(244)
X
P1
Xc +1
2e In
X2 + Z2,
(245)
V1+ PT
V
where Z1 ~ NOO, (1 In 2) and
Zo NO0 (1
P
2
1 + 2p
2e In 2)0²).
(246)
1+ P
1 p
Notice that
Dj = 2(1 - V2e In 2)
(247)
D2 = 2(1 ((1+ In 2)0².
(248)
For the special case of p1 1, the expressions in 244 and 245 simplify follows
X = / In 1nX, + Zi,
(249)
Z1 + Zz. v2 Z1 + Z2 and notice that ZEDD NO
X = V2 +
(250)
V2
Define ZPMD
4c In 2)o?). Moreover, we have
D1 = 2(1 2e
(251)
D2 2
In
(252)
C) O-PLF-JD: In this case, the optimization program of the first step similar to the previous case. The optimization program of the second step is given in (215) where the condition W1 + pw2p1 P1 introduced. According to (233), // = ((ve) which suggests the following form for wi,
W1 P1 Se,
(253)
31

---

for some small de that goes to zero as с + O. The parameter óe will be determined later. Plugging wi = P1 be into (240). we find out that only the constant term w contributes dominant term for w2 which yields the following
W2 21 ( O(c).
-ff)+
(254)
Thus, we have
X = 22 1n2X + Z1,
(255)
X = (p) - + (1 pf) In 2X2 + Z2,
(256)
Now, applying the constraint 6[8 = p1o², we get
8, P1 pf( In 2).
(257)
However, notice that since 6, = (c), it does not contribute to dominant terms distortion. So, we can simply represent X and X as follows
X 22 In 2X, Z1,
(258)
Xe X1 v (1 22)2 In 1n2 + Zz.
(259)
where Z1 NO (1- 2c In 2) and Z:2 N( (1 o (1 (1- (1-17+20p p 1 2c In 2)o²).
The following distortions are also achievable
D1 = 2(1 - 22 2)
(260)
D2 = 2(1 - (p] + v 2 ln
(261)
For the special case of p 1, according to 259 and 261 we have Xe = =c and D2 = D1.
2) R1 + xo, R2 = for small e: In this case, since Ri + co, we have X X1, D1 = 0, and we only need to solve the optimization program of the second step. Also, we have the following approximation
1 2 2-2- = 1 2- = In O(e?).
2R
(262)
We consider three different cases based on the perception constraint.
a) Without a perception constraint: In this case, consider the optimization program 213 For a given w and W2, the objective function is an increasing function of ož, hence optimizing over o2, we get
3 (1 - př - O(e))
wio² + 2w1wzp1a2.
(263)
2e ln + (
The program in 213 is further upper bounded by the following
3 3² - + + w + 2u1 201p1022
min Wi .igg
-
2c 2
w1W2P120 20
(264)
The solution of the above optimization program is given by the following
W1 Pl (21 2), W2 2c In
(265)
(266)
Thus, we have
XC =X,,
(267)
LC (p1 P1 (2c (n) + (2 + Zz.
(268)
where Zn ~ N(0
(1 2 In 2). So, the reconstruction of the second frame closely resembles
the first frame. The distortions of the first and second frames are zero and (1 - (1 ) e In 2) respectively.
32

---

b) O-PLF-FMMD In this case, 01 = 02 = G. Thus, the optimization program in (214) is further upper bounded by the following
min 01
20 -2 2uzoo"
(269a)
w MI 20
S.t. For given wi (resp w2), the objective function (269a) is a monotonically decreasing function of wg (resp w1). So, the optimal solution is attained on the boundary, i.c. (269b) is satisfied with equality
3 (1- of + O()) S S (1- w/ www (2 O(e²))
(269b)
given follows
w p O(e)) = (1 - wf 2w ) (2c In O(e2).
(270)
It can be easily verified that the first-order terms of wn and we which optimize the program are 1 and 0, respectively. So, we write wi and we in the following form
W1 = + (2e In 2) + +((2),
(271)
W2 (2c In 2) + OE²),
(272)
for some real 61 and 82. Plugging the above (271 and (72 into (70 and considering the dominant terms, we get
( p 26, 2p102.
(273)
On the other side, we can write the objective function in (269) as follows
20 2w1p10² 2wzo²
220² - 2p/wi 0² 2w20² + 0
(274)
20 2010² - 2(p11d00 + 620 )20 In + 0
(275)
= 220² - 2p20² - -1-2pf820" pI - -p)) + (2 O(e).
(276)
Differentiating the above expression with respect to وU and letting it be zero, we have:
62 1
81 1 př
(277)
P1
2p7
Thus, have
X =X1,
(278)
X (1 (1 + 2m
2e In
X2 + Zz.
(279)
2p
P1
where Z2 ~ NO, (1 )2 In 2). Again, the reconstruction of the second frame is almost similar to
the first frame and the distortion is 2(1 - - - (1-p 2 In 2)o².
c) O-PLF-JD: First consider the case where #1 * 1. The optimization program is given in (15) where the constraint w1 + PPI4> = is introduced. Notice that w1 can be written in the following form
W1 = P1 + de,
(280)
for some 5- that goes to zero c - 0. The parameter 5, will be determined later. Plugging
w1 = P1 + 5e into (270) yields the following
W2 2e ln2 O(e),
(281)
which is derived only through the first-order term of w1 which is P1. Now, considering the fact that x[X X p1o², we obtain
8. = -1 ln2
(282)
Thus, have
xC X1,
(283)
Xc (p1 - P1 2c In 2) + 2 In 2X, + Z22, where Zz ~ N(O (1 2)) Here, the reconstruction first frame. The distortion of the second frame is 2(1 - o (1 - /2 In 2)
(284)
the second frame closely resembles the
If pi = 1, we simply have Xe = XC = X1 = X2 which can be derived from 283 X X2.
284 by letting
The analysis for the case of R₁ = e and R₂ + oo similar and omitted for brevity. The results of this section are summarized in T
33

---

Table 2: Achievable reconstructions for extremal rates and different PLFs (The first, second and third rows represent reconstructions corresponding the MMSE, O-PLF-FMD and O-PL.F-DD, respectively).
Ri Rg
Ri cc, R2 xP=X:
R2 R1
xf 1ff-(2 (2
x9 =f= 2)X1 x =x=X2
x2 =p XP (2r1n2))2
x9=(m1 1122 (2e1n2) 22
MMSE
2y,~N(0, 1n2)
20, (1 11)2002
21 20, 2 D1=(1 2) 1²=0
D1=(1 2c
D,=0
D2=(1 (1 (n)
D₂=(1- p4 -7-11 12)2c112)0
xf = 2X1 + 21
xf=x: (1tof:2eln) 1&P +
= 2X1 + 21
x9 21 xP + 11
2g
-f- 20f
2cle2 xx 22
X9 -X2
Vituf
m
0-PLI-PMD
(- 1= 12 (2-
2,,W(0, (1-.i00011-2000000 2)
(2,~((0
22~N(0, (1c2122))))
Zz,W/1, o7
1+26 2)
11
11e]
D1=2(1 22e 2)0²
D1=0
D1=2(1 - 22e
D2=2(1 (11 + 2))
Dz=2(1 P: (22p) )20 2)00
Dz=0
=9=12.1n2X, +2
8-=x,
= in + 22
xf==18f- v1 12
21V2el12
V2eln2X,+
= + +
0.PP-11
2,-N10 (1
ZzmN(0,, (1- pii0²)
21, 2c
In D1=2(1 2
D1=0
D1=20²
Dg=2(1 - (r) + - 22 D2=2(1 - r - - 2n
22=2(1
o
-p 2)
the coefficient wa (the cuefficient of X in & has some correction terms (()) which are igmored the presentation x since they da not contribute domineet terms distortion.
As justified in
253
G Comparison of PLFs in Low-Rate Regime
Theorem 6 For sufficiently small c, let R₁ and suppose that Pj p and o; for The achievable distortions DPMD. (for O-PLF-FMD), and DiD.i (for O-PLF-JD) are:
1, 1....,T. T.
DYND.j = 2(1 y 2n DID = 2(1 ADD / 2e In 2)0²
(285)
where AFMD. = p2
1202p-11 and AID == ((1-1) +1 2 2) vI
)
2p
Proof: We extend the proof in the previous section for the low-rate regime to T frames.
Distortion Analysis for O-PLF-FMD:
We follow similar steps to 233
248 for optimization problems of the third and fourth frames and then use induction derive expressions for T frames. For simplicity, assume that Pj p for all j.
Notice that in the following proof, (&G, X are as in (205 206 where v, wh and W2 are already
derived in 233
248
Now, consider the reconstruction of the third frame follows
X & T2XO + TaX3 + Zz.
(286)
for some T1, T2: Ts, where Xe o²) and Zy is Gaussian random variable independent of (XC, XC. Xz). The rate constraint of the third step is given by
R3 > 1(x; X X1X X9).
(287)
34

---

Evaluating the above constraint with the choice of random variables (X9, 22, xx) and re-arranging the terms, we get
T (1 112--222 (p'22-21- + p1 - - pp²)) S
(1 - 2 2 (1 - - -12 2T1T2W11 2T T2W2up 2r27z=1pp² 2T2Tzu2p 2T p3up²)o²
(288)
Similar to (240), considering the dominant terms of the above rate constraint and the fact that the solution of the optimization problem attained when the above inequality is satisfied with "equality". we get
(1 -1- - (1-72-2²00(7))
(289)
The distortion can be written follows
Ell|Xa - 20 2T 2nzw2pa² 2T up² -
(290)
So, the goal is to solve the following optimization problem for the third step
min
20 2T30² 2T 2Tgwiup' 2
(291)
s.t.
(1 (1- o - 7 + 0 2/2 0(2)) (
(292)
We restrict the search space to T1, T2, T3 2 and get an upper bound to the above optimization
program follows
min 72.7.
20² - 2T30² - pT2 - 2n' -2Tiup²0
(293)
s.t.
(1 - Tे + 0 2222222 +00)) = - (1 + O(e))
(294)
The above optimization problem is equivalent the following
(2e (2e ln2
+ ((2))( -1
r3 + g
min T1.7220
20
2
1+
2Tzwwppp 2Tzwiup' -
(295)
We proceed with solving the above optimization program. Taking the derivative of the objective function with respect to 7h and 1/2 yields the following:
122
P +(()),
(296)
1/
1/1
= 0(()
(297)
/1
n
Solving the above set of equations, we get
p²
+ +0(e,,
(298)
V1 + p + 2p
PV + p2
12
+ Oee).
(299)
V1+ p² 2p
Thus, considering the dominant terms, we get the following reconstruction for the third frame
0²
PV1+p2
V2
xC
VI+ +2 XG
&G
X3 + Zz.
(300)
VI+ p² 20
The above reconstruction yields the following distortion for the third frame
||| - 1 = 2(1 / 22 In In + 0² + 2p²
(301)
Finally, consider the reconstruction of the fourth frame as follows
X = A + Xex + &ax8 + +xXa + Zt,
(302)
35

---

where X2 ~ (N )²). The rate constraint of the fourth step implies that
(1 11-XF 1 X - 2 + ((e)) + O(e)) = 22 + O(e)).
(303)
The distortion be written follows
E||X - x = 20 - 22y0² 2231T30² - 2\zp T2way T2 - 223 u2
-27xp" TvOp 22ypp' wivo² 2 2 2 -21p'v
(304)
20 -2 (2E In 2m X9 - 13 - 13)0² - 22apTso²
-273p² T2wzo 2 - -2230 Tzwn -2
2Azp'urvo" - 2222"u200 - 2\1 Occ).
(305)
We take the derivative of the above expression with respect to 21, 22 and Xı and we get
A
p² +O()).
(306)
17
13 X2
A2
p² V1 1
0² + +((),
(307)
Aj
N/3 2/3
X
PV + + p² + +0(c)
(308)
X - X2 X3
Solving the above set of equations yields the following
\1
+0(c),
(309)
1 + p² + 2p 408
0²v1+ + P2
A2
+O(e),
(310)
V1 + p² + 22p 406
PV 1 + /² + 2p
A
Occc.
(311)
VI+ p² + 2p + 40
Thus, considering the dominant terms,
write
pa
1² @VI++ P2
X-
8G +
GG
X2
VI+ + P² + 2pt + 4.
VI+ p² + +2p² + 4p6
AV1 - 0² + 2p
/2e 1n2
8G +
X4 + 74.
(312)
I + p² + 2pt + 4p
V1+ p² + 2pt + 4p
The distortion term then becomes:
F 1 2(1 / 2e In 2(1 + p2 + 2pt + 4p0))
(313)
Now, we use induction to derive the terms for 7" frames. Define
FFMDJ =
1 +
2j -jー1-102(i-1,
j 2
(314)
1=1
(2p2)
V
1 + p2
(315)
2p² -
Thus, we have
1-1 ARMD.iD
V2e In
X
G +
X, + 7z3,
j j= 2, 2......,7 T,
(316)
AFMD.J
APMD.j
where Zy is Gaussian random variable independent of (X:r-
X 1-1: X,) and its variance is such
that 18(()))9 d². The distortion is given by the following expression
DFMD. E|||X, , 2(1 ARMD, 2e
j j= 2..... T.
(317)
36

---

For the special case where p = 1, then the distortion simplifies to the following
E|| x - ,
2(1 - 2' 22 2)o²,
j = 2
(318)
which shows an exponential decrease at each step. Distortion Analysis for O-PLF-JD:
In this case, the proof for T frames is similar to
254 261p. Thus, we have
X9 = = (11 l2 Zz. .,
(319)
where Zj is Gaussian random variable independent of (X 1; and its variance is such that ( (
2². It should be mentioned that preserving the correlation coefficients, e-g.. E X J-1 p. needs some correction terms of O(c) discussed in (257). However, as shown in (261) these correction terms do not contribute to dominant terms of distortion and hence, they can be ignored in the presentation of (319. Now, define
j-2
DDd = p 20-1) + v1 p²1
)
)
2_,...
(320)
1=0
and notice that
DID == E||X; X111
(321)
20² - 22]
(322)
20 2E|X, (AX (1
(323)
20² 2 (Nr 1x + v p 2x + + x,))][2eell 2a
(324)
= 2(11 - AID,, V 2e
(325)
For the special case of p = 1, we get AD. = 1 which remains a constant across different steps.
H
Universality Statement for Gauss-Markov Source Model
H.1
MMSE Representations for Given Rate
For given rate tuple R, the minimum distortions achievable by MMSE representations are derived in 28 377 and are given by
Dmmin o.2 022-211 2R2
(326)
Dmin
(pi 02
m + +o,, 12-22-2 12-11,
)2
(327)
σi
(p) 0 D + ) 22-22-2
Diain
(328)
02
where
N, = (1 - p
(329)
N₂ = (1 - p
(330)
The above distortions achieved by the following optimal reconstructions X, given in 28 Notice
that the MMSE representation is XRD - -,,, i.e., the functions n (.) and vı2 of iRDP region CROP (Definition 4) are identity functions (this statement follows from Theorem Now, we choose the
reconstruction X, in the following.
The reconstruction X, is chosen such that X,1 + X1 - (X2, Xz) holds a Markov chain and
X1 - + Z1,
(331)
where X., NN of - D and 21 ~ ~N Dimin) are independent random variables. Then,
the reconstruction X is chosen as follows. Let
W2 = P1 σ1
Z + N1.
(332)
37

---

which is the innovation from X,, to X2. Now, we find the random variables W2 and Zz such that
W2 W2 + Z2,
(333)
where W2- ~ (N pf Diin + o, -Dainn and Z2 ~ NN(0, D are independent from each other, and the Markov chain W2 (r,1 (X1, holds. Now, define
02 x,, + W2. x, = p1
(334)
01
Finally, we choose the reconstruction X as follows. Let
W3 = P2 03 Z2 + N2;
(335)
which is the innovation from X to X3. Now, we find random variables W3 and Z3 such that
W3 W3 + Z3.
(336)
where W3- ~ (O, P2 D + No - D and Zg ~ No0 Drsin) are independent from each other, and the Markov chain W3 + (Xs, x,22 + X2) holds. Now, define
X == p 02
03 22 + W3.
(337)
Thus, the optimal reconstruction X, is chosen and satisfies the rate constraint R.
H.2 Universality Statement
Theorem 7 For given rate tuple R with strictly positive components, let the MMSE representation
be denoted as X = =(XRD corresponding reconstruction achieving it. Then there exist Ki, O1, Oz, th, th2 and th3 and noise
y,1, r.2y X Let (D, P) E DP(R) and let X = =(1,, 1X,1X2,33 Xa) be the
variables (Z,, Zo Za) independent of (XRD, XRD
r,1 r,2 XF2). which satisfy the following
X1 xx= X + Z1,
02X
Zz = th 1 + XRO wh X 1. + Zz.
For given positive rate tuple R, let the MMSE representation X be in the set PRO (R). Also, let (D, P) DP(R) and X,, X be the corresponding representation and reconstruction achieving
Proof:
First, notice that according to the proof of Theorem for the Marss-Markov source model, one can set X X, in iRDP region of CROP, without loss of optimality. So, the following proof, the reconstruction X,. and representation X are used interchangeably, in some places.
We show the following statement. If
R1 2 1(X1;Xr.1).
(338)
R₂ 2 T(Xg; Xr,)
(339)
R3 2 7 7(x3; Xr,3/ X,,1, Xr,2),
(340)
then, there exist R1, 01. 82. 1h,, th and t/s and noise variables Z1, Zz. Zg independent of XRD 1.1 XRD x XRD), respectively, which satisfy the following
(XRD X.D), (X.D RD
x, K XRDD r. Zi.
(341)
8₂ 0, X + 02X + Zz:
(342)
X; 1 XRD
X2DD F2
1ha
x + Z3-
RRD r33
(343)
(33)
340 are satisfied with equality, then the noise random variables in 341 343) do not exist
and a linear combination is sufficient for converting (XRD.X RDD XRD) to (X, X2)Xx).
First, we prove the statement when all of inequalities in (388 the proof for = 2 frames. The extension to arbitrary number of frames is straightforward. To that end, we first prove the following two lemmas.
340 hold with "equality". We provide
38

---

Lemma Without loss of optimality, the reconstruction of the first step X, satisfies the following
nXi WW.
(344)
where
X1
(345)
and Wi is Gaussian random variable that its statistics do not depend the pair (Di,Pi)
Proof: According to Theorem 5 we know that (1) are jointly Gaussian. So, we can write X1
as follows
X1 X1 +Ti.
(346)
where Ti is Gaussian random variable independent of X1 with constant variance 2 Notice that (346) can written as follows
= = (X1 +Q),
(347)
af2-28;
where Q is a Gaussian random variable independent of X1 with zero-mean and variance T-2-2RT and
1 (1-2-2R11.
01 711
(348)
From 347 we get
21 (1-2-2R; ) Q).
(349)
Now, defining W, 21-2-2R, )Xi + +Q) yields the desired result.
Lemma 3 Without loss of optimality, the reconstructionss of the first and second steps (X1, X2) satisfy the following
x + 2xx22 = W2
(350)
where
\ X 10 E Xx [x2
(351)
2x X2 - [X
P1 E| X E X X 03 E3 X2
(352)
-E'(X
and W2 a Gaussian random variable that statistics do not depend on the pairs (D1, P1) and (D2, P2).
Proof: According Theorem 5 know that (X1, X2, X1, X2) are jointly Gaussian. So, we can write X₂ as follows
X₂ = X1 A22 X2 + T2,
(353)
where T2 is Gaussian random variable independent of X2) with a constant variance of 2-2 where
x² Xx1111
1 x,1x, = : log (pio 28, + 22M(N)))
(354)
Notice that 333) can be written as follows
i + X2X2 = (1_2-22a)(Xg + Q²)
(355)
where Q' is a Gaussian random variable independent of X₂ with zero-mean and variance o2 Xg & 2 28g
Defining W2 = (1 1--2 2- (X2 + Q') yields the desired result.
1-2 28g
39

---

Now, we proceed with the proof of the theorem. According to Lemma 2 there exist real " and 7 such that
71 = 7/x x- r,1
(356)
Define
K1
(357)
7/1
Then, according Lemma 3 there exist d1. 22. X/ and X/ such that
A1 + X2 = X1 X X2
(358)
The above equation can be written
X2 X1 AIKI
X/ X
X
(359)
A2
X2
= 8,,,1 + +02XP2 r,2
(360)
A similar justification holds for the third frame.
Next, we prove the statement when at least one of the rate constraints in (338 340 hold with strict inequality. In the following, we construct new reconstructions (X1, X.) based on (X1, X2) such that they satisfy the rate constraints (Ri, R2) with equality. Then, we will be able to apply the two lemmas we proved to show that (X1, X2) are linearly related to MMSE reconstructions (XRD, r.1 r,
Construction of X1:
Now, let
Ri = ((x;)x))
(361)
where R1 S R1. Also, recall that
R1 = ([1:xXP)
(362)
Now, let 8 such that 81 + XRD + X2 holds and
X = x + W1,
(363)
where W1 ~ NNO, v/) independent of X1 and of will be determined in the following. Notice that T(X1; X)) is a monotonically decreasing function of vz. So, one choose vî such that
T({;; X1) [(2;; X1) R1.
(364)
Now, according to Lemma 2 since X and X, have the same rates, there exists a coefficient x/ such that
x K X
(365)
K2 XRD W/
(366)
Now, define Zi = K W, and notice that
8 X + Zr.
(367)
Construction of X/:
Next, consider the second step. Define
AS := 1(X2;x21x11)
(368)
where R₂ < Rg. Also, recall that
R₂ 1(2;; X
(369)
Define X, : -ED X to be the MMSE reconstruction and consider that
R₂ = [[x;; X RD
(370)
(371)
40

---

where the last equality follows because both Markov chains X2 - 1X - X2 and X2 +
X₂ (XRD) X ,,2) hold where the latter one is satisfied for Gaussian random variables for which we can write X2 = Xr, 2x + W such that W is independent of ,
Now, we show that [[x [x R - [ (X ()))
This is justified in the following
I(X2; X2 = 1XXx:x RD + W1)
(372)
= H(x2|X7,1 W1) H 22 x + Wi)
(373)
2 (X2)/x, + W1, - (X2 XX X + W1)
(374)
(X2]/X W1) - (X22 X W1)
(375)
2 HX W1) - (x)
(376)
= -X - (
(377)
= (Xx x,1) (D) - ( 18,, xDD)
(378)
= =[(2; X2
(379)
where (77 follows because W, is independent of (X2, X and (378) follows from the Markov
chain X2 X2 + X,
Define
R² 1(x2;
(380)
and consider the fact that R² 2 R2. Now, we introduce X& such that X3 - (1) + X2 forms a Markov chain and
8 X2 + X1 + W2.
(381)
where W2 (O, vg) independent of (X2, X1) and vg will be determined in the following. Since I(2;; X { is monotonically decreasing function of vžz we can choose ug such that
I(X2; X5181) I(X2; X2(x11 fiz-
(382)
Then, according to Lemma 3 there exist X1. X². Å₁ and 22 such that
X X = 8 S282.
(383)
Plugging (333) (367) and 381 into the above expression and letting X₂ ax.R + 38 for some a, 3, we get
(X + (1+
- + X2 (X + X2) + X,W2 ú1 -22x2 (384)
Now define
0 = Xi + (1 + +a)x 'iT'
(385)
)
00 X/
(386)
22
2g = (X1 + Xb2
X
W +
W
Z
(387)
X2
Thus, have
X2 = 0, RD + 02X RD + Zz.
(388)
Notice that the above proof only uses the information about reconstructions of the operating points in DP-tradeoff and does not depend on the choice of PL. So, holds for both PLFFJD and PLF--MDD
This concludes the proof.
41

---

H.3 Gaussian Example
Assume that the sources are symmetric the sense that of = - : for
-
some 0 p I 1. Also, suppose that the perception thresholds are symmetric, i.., P1 P2 P3 P for some 0 < P & 1. We choose the rate tuple R such that the minimum distortions Dj D for j {1. 2, 3). According to Appendix H.1 such rates are given by
1
R1
log D
(389)
D²D + (1 p)-p)
R2 log
(390)
D
2 + (1 11ppp p1-
R3
log
(391)
D
The covariance matrix of the MMSE representations cov(X,,) r,1 X xF3) is given by D)E
where
1
p
2²
P
P
(392)
1
If introduce the O-PLF while keeping the rates those of MMSE reconstructions, it can be shown that the optimal distortions are all equal to D1 D2 D3 = 2 2v1 D. Denote the reconstructions by (X X B., BB,) and notice that the covariance matrix of the reconstructions is equal that of the sources and is given by Σ. Thus, the covariance matrix of (XRDD X XRD) is D) times the covariance matrix of (XD) D2 D22 X So, the reconstructions (XPP1 RRD X
and (XB, XD XB can be transformed to each other by the scaling factor -í This inspires the idea that reconstructions corresponding to different tuples (D, P) are linearly related to those of MMSE representations which is the essence of the following Theorem Moreover, both PL.s either based on FMD or JD perform similarly in this example since individually scaling the reconstruction of each frame finally ends up in matching the covariance matrix of all frames.
I Justification of low-rate regime for Moving MNSIT
In the MovingMNIST dataset, the digit in I-frame is generated uniformly across the x2 center region a 64x image, meaning that (02(32xx)))) 3 bits are required to localize the digits and any lower rate would result in much less correlated reconstructions, As such, one can consider R, =12 bits (2 extra bits for content and style) as low rate. For P-frames, the movement uniformly constrained within 10x10 region so any rate R& log, (10x10)=6.6 bits (excluding residual compensation) can be considered a low rate.
J Experiment Details
J.1 Training Setup and Overview
Our compression architecture is built on scale-space flow model (32), which allows end-to-end training without relying on pre-trained optical flow estimators. For better compression efficiency, we replace the residual compression module with conditioning one (33). In the following, we will interchangeably refer X, as the I-frame and subsequent ones as P-frames. The annotation for the encoder, decoder, and critic (discriminator) will be referred to as f.g. and h respectively and their specific functionality (c.g motion compression, joint perception critic) will be described within context through a subscript/superseript.
Distortion and Perception Measurement: We follow the setup in prior works [16 for distortion and perception measurement, Specifically, we use MSE loss - Wasserstein-1 distance as perception metric, which can estimated through the WGAN crities (following the Settoooooviih--biisteeee duality). For the marginal perception metric, optimize
distortion metric and
42

---

Fixed Eneader
Optimized Encoder Deedderrr
(X1, X1)
91
9i
X
9i
9i
X
9i
X,
X
X,,1
[ n.a.
X [--1]
[XP]]
X
[t--1]
[X
f1
fa
mm
f
Xi
(Xi-1, Xz)
(XT",
(X: (c) P-frame end-to-end
(X,", X²)
(a)I-frame
(b) P-frame universal
Figure 7: Compression diagram for (a) I-frame (b) P-frame with universal representation and (c) P-frame with optimized representation. For simplicity, we do not show the shared randomness K.
our critics hm to classify between original image X and synthetic ones X. This will then allow us to measure Wi (Px, Px) since:
sup Em (X)] E[ (X)]
(393)
where F is set of all bounded 1-Lipschitz functions. Similarly, the joint perception metric realized through Wi ( (Px, Xx., Px Xx....,) by training a critic hy that classifies between synthetic and authentic sequences:
W1 (Px Px Px....,)== sup (X .... X.) N
( X.)
(394)
In practice, the set of 1-Lipschitz functions is limited by the neural network architecture. Also, although our analysis employs the Wasserstein-2 distance perception metric, it is worth noting that the ideal reconstructions (0-PLF) for this metric and the one used in our study should be identical.
I-frame Compressor: We compress I-frames in a similar fashion as previous works [16 21). Our encoder f and decoder 91 Figure contain a series of convolution operations and we control the rate R1 by varying the dimension and quantization level in the bottleneck. The model utilizes common randomness through the dithered quantization operation. For a given rate R1, we vary the amount of DP tradeoff by controlling the hyper-parameter A, objective Lt:
marginal in the following minimization
L = E - 81111 + A. marginal W1 (PG Px.)
(395)
Following the results from Zhang et al. (16), fix the encoder after optimizing the encoder-decoder pair for MSE representations. We then fix the encoder and train another decoder to obtain the optimal reconstruction with perfect perception, i.e, Wi(Px. Px) O. We will leverage these universal representation results to compress P-frames (both end-to-end and universal).
P-frame Compressor:
We describe the loss functions before explaining architectures. Given previous reconstructions X x(=-1)==XX (x1, X2, ..., X,-1), one can adjust the distortion-joint perception tradeoff by controlling the hyper-paramen X in the following objective L.-
cjoint Bi||X, X.11 + joint Wi(Px P
(396)
Note that in order to achieve O-PLF-JD, previous reconstructions X 3-11 must also achieve O-PL-FJJD since it is impossible to reconstruct such X; if the previous X;=11 are not temporally consistent For the FMD metric, we use the loss function in (395).
In the universal model in Figure 7b, the motion encoder f." compresses and sends the quantized flow fields [Xm] between the MMSE reconstruction X,-1 and X,. Given [Xm] r,2l the flow decoder and warping module g,'h will transform X;-1 into X," (predicted frame). We use ff compress the
This follows from the inequality: (2 x,,,,, (Px, Px, +W3 Pxx)
43

---

residual information [X,,]] between X; and X," which will be decoded by g. We note that for MMSE representation, of only requires X," as a conditional input while an additional conditioning input [i--1 is required when perceptual optimization is involved. Together, f. g. J. and of optimized for MMSE reconstructions. 'o train for different DP tradeoffs,
fix fthh "9""" :f
and adapt the new decoder df (conditioning on X,"" (t)). We note that fixing D." for universal representation is essential since [X] is dependent on the outputs X;" of g."
In the end-to-end model in Figure 7c, as in the case of the universal model. The only difference is that the encoder f; also uses previous X, and the encoders will be jointly trained with the decoders.
MMSE representation estimate the motion vector,
J.2 Networks Architecture
In this section, we describe the network architecture for universal and end-to-end P-frame compressor models. F In the architecture layout, we denote BN2D and SN for the Batchnorm2D and Spectral Normalization layers. Convolutional and transposed convolutional layer are denoted as "conv" and "upconv" respectively, which is accompanied by number of filters, kernel size, stride, and padding.
Motion Encoder and Decoder. The universal and optimized end-to-end model shares the same architecture for the motion encoder and decoder. (ff" and 9,"h respectively). We follow the original implementations [32] and present the convolutional architecture in Table Different from the original implementation, however, we replace the last layer with dithered quantization layer (as in (16) in our implementation. The output dimension of the motion encoder is denoted as dm
Table 3: Motion Encoder f.r and Decoder g:"
(a) Encoder f
(b) Decoder 9,"h
Input-64x66n x4 x (2 channels)
Input-de dm
conv (64:5:2:0), BN2D, 1-ReLU
upconv (64:4:1:0) BN2D, I-ReLU
conv (64:5:2:0), BN2D, R-ReLU
upcony (64:5:2:0), BN2ID, I-ReLU upсony (64:5:2:0), BN2D, 1-ReLU
conv (64:5:2:0), BN2D, 1-ReLU
conv (64:5:2:0), BN2D, I-ReLU
upconv (64:5:2:0), BN2D, I-ReLU
conv :d::0::2:0),, BN2D Quantizer
upcony (3:5:2:0), BN2D
Residual Encoder and Decoder: The architecture of the conditional residual encoder is shown in Table 4b, where we stack multiple frames along their channel dimension as an input. As described previously, in the residual encoder, the universal model requires only X,, XY" while the end-to-end model will receive X,, X," and Xp- We denote the output dimension of this residual encoder as d,. In the decoding part, the decoder will first condition all the previous reconstructions X₃i 1] by projecting them into an embedding vector of size 192 (conditioning module in Table 4)). Then we concatenate this vector with the output of f." The concatenated vector will be fed into the decoder (Table 4 produce the reconstruction Xt.
FMD and JD Critics. For the video critics, our PLF-JD critic architecture is inspired by the work of Kwon and Park [40), where concatenate frames sequentially along their channel dimensions. For both PLF-FMD and PLF-JD critics, we add spectral normalization layers for better convergence. Their architecture is shown in Table[5]
Rate and output dimension The rate computed by logy (dene L), where the number of quantization levels and de Table 6 provides configurations of the rate, dmn d,, and L in the experiment.
Training Details: We use batch size of 64, RMSProp optimizer with learning rate of 5x 10 and train each model with 360 epochs, where the training set contains 60000 images. To accelerate
'erre, conditioning 33 instead of sending X; X." in the original work (2
$ For the I-frame compressor, we follow the DCGAN implementation by Denton et al 39 adding the dithered quantization layer in the encoder's 's last layer( httpt::t:///et thugg com htt///eeeettttw/eeeeeeeee blob/ master/ models/dcgan, py)
44

---

Table 4: Residual Encoder, Conditional Module, and Residual Decoder. (b)Conditional Module
(a)Encoder ft
Input
Input
conv (64:5:2:0), BN2D, I-ReLU conv (64:5:2:0), BN2D, 1-ReLU
conv (64:5:2:0), BN2D, I-ReI.U
conv (64:5:2:0), BN2D, I-ReLU
conv (64:5:2:0), BN2D, I-ReLU
(64:5:2:0), BN2D, I-ReLU
conv (64:5:2:0). BN2D, I-ReLU
conv (64:5:2:0), BN2D, I-ReLU
conv (d,;::1:0), BN2D
cony (192:4: BN2D
Quantizer
(e)Decdder
Input-(d,++ 192)
upconv (64:4: 1:0) uc4s 1. BN2D, I-ReLU
upconv (64:5:2:0), BN2D, I-ReLU
upconv (64:5:2:0), BN2D, I-ReLU
upcony (64:5:2:0), BN2D, I-ReLU
upcony (channels:5:2:0), BN2D
Table 5: PLF-FMD and PLF-JD critic for frame i.
(a) PLF-FMD Critic
(b) PLF-JD Critic
1uput-64x00 xchannels
Input 6xppt-66xx44xxxcchamaaa
SN, conv (64:4:2:1), I-ReLU
SN, conv (64:4:2:1), I-ReLU
SN, cony (128:4:2:1), R-RLLU
SN, conv (128:4:2:1), R-ReLU
SN. conv (256:4:2:1). I-ReLU
SN, conv (256:4:2:1), 1-ReLU
conv (51 (512:4:2:1), I-ReLU
conv (512:4:2:1)
Linear
Linear
training, we pre-train each model for 60 epochs with the MSE objective only. Under WGAN-GP framework [30]. use the gradient penalty of 10 and update the encoders/decoders for every iterations. The parameters À controlling the tradeoff are in Table Training takes 2 days per model on single NVIDIA P100 GPU. For the MovingMNIST factor of two bound and permanence of error experiments, we repeat the training 3 times.
Table Rate, embedding dimension dm,, d, and quantization level L.
(a) P-frame encoder, Ri
o0.
(b) P-frame encoder, Rı (12 bits).
R2
dm
dr
R2
dm
d,
bit
1
0
2
4 bit
2
2
2
2 bits
I
8 bits
2
3.17 bits
1
12 bits
2
J.3 Permanence of Error on KTH Datasets
The KTH dataset is widely-used benchmark dataset in computer vision research, consisting video sequences of human actions performed in various scenarios. We show more examples supporting our argument for the permanence error on this realistic dataset. We use bits for each frame. In general, the (-PL-FJJD decoder consistently outputs correlated but incorrect reconstructions due to the error induced by the first reconstructions, i.c., the P-frames will follow the wrong direction induced from the I-frame reconstruction. Besides the moving direction, (i.e. walking. jogging, and running) is also affected. On the other hand, while losing some temporal cohesion, MMSE and O-PLF FMD decoders manage to fix the movement error.
also notice that the type of actions
Finally, we computed LPIPS [4]] in Table g which is known as full-reference perceptual metric for images. We compare LPIPS for each reconstruction (MMSE, O-PLF-FM and O-PLF-JD) at each timeframe. This result aligns with our results on distortion loss, as the MMSE and O-PLF-FMD tend to correct the reconstruction (so the ground truth and reconstruction look more similar over time). On the other hand, due to error permanence, the (--L.F-JD reconstructions become different from their source sequence, causing the score to go up.
45

---

Table 7: Perception loss and their associated X
Perception Loss
x 10
0.0, 0,7.7,7..0 1.15, 12,, 1.25, L.3, 1.5, 1.5, 1.7 2.0, 5.0, 8.0, 10.0, 40.0,
Joint Distance (JD)
Frame Marginal Distance (FMD)
0.0, 0.4, 0.7, 1.D, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 7.0, 10.0, 40.0
GT
MASE
1
x
4
D-PUF FMD
8
4
x
C-F
4
8
GT
MNSSE
0-PFF RMD
0-PLF
MMSE GT
4
4
D-PUF RMD
4
4
0FLF
GT
MMSE
OPPF
RMD
O-7UF
Figure Additional Experimental Results for the Permanence of Error Phenomenon on KTH Dataset.
Table 8: LPIPS score on KTH dataset (lower is better).
MMSE
PLF-FMD
PLF-JD
1st frame
0.1036
0.0584
0.0584
2nd frame
0.0521
0.0313
0.0594
3rd frame
0.0413
0.0232
0.0613
J.4 RDP Tradeoff for frames
We extend our experimental results for the RDP-tradeoff and the principal of universality the case of GOP size 3. As mentioned in the main paper, while the universal model only requires MMSE representations, the optimal end-to-end model also requires the MMSE reconstructions from previous frames to provide best estimates for motion flow vectors. Practically, this is challenging for our employed architecture since only previous X1,x, are available. As a result, to compare the RDP tradeoff between universal and end-to-end model, we also provide the end-to-end model with the MMSE estimate from previous frames while noting that this is unfeasible in practice. Interestingly, show in Figure 9 the RDP tradeoff curves for the third frame X3 and its reconstruction X3.
46

---

unioenel 3.17 bis end-to-end
unieral
unionnal
univenal
uniconal
univenal
bit red-tooved
lit exd-to-end
bits end-to-and
bits end-to-end
sis end-to-end
Pl PIF-FMD: WilPs Pay
PLE 10: WilPaaoke Ps.
PLF EMD WalPr
PLF-JD. WiPPr.
Oisorton:]] 8
Distortion| [Xx
Disatto:||P: -
Distortion: Xj Xill
(a) Case 1: =oc, R2= Ro= 2, 3.17] bits
(b) Case 2: R₁ 12, R2= [4, 12] bits
Figure 9: RDP tradeoff curves for end-to end-to-end and universal models. We plot the tradeoff for the two regimes:
R1 =00 and =i=e in (a) and (b) respectively. The universal and optimal curves are close to each other.
observing that the universal and optimized curves are still relatively close to each other. When (Ri R2, (o), e, e), note that the distortion for X3 is larger than X2 since the allocated rate is not enough to correct the motion. Finally, for the case ( (R, R2, Ra) ((, 6, e), we note that the curves again converge as in the case of (R₁, (e) e) due the incorrect reconstruction in the I-frame.
J.5 Diversity and Correlation
When (R), (o)= e). our theoretical analysis predicted that the decoder optimized for JD is capable of producing diverse reconstructions. On the other hand, an optimized decoder for FMD will tend to produce reconstructions that are highly correlated with the previous reconstruction X 6 In our experiment, we also observe such behavior, summarized in Table and show several examples for R2 = bits in Figure I0 We observe that reconstructions from the joint metric deviate more randomly from X2 than the marginal reconstructions. The marginal reconstructions, the other hand, stay much closer to their original reconstruction X1
Xi
X;
(1-PLF
X. (0-PFF FMD
9
q
7
7
7
7
?
7
2
9
Figure 10: Diversity in reconstruction X₂ for O-PLF-JD and correlation with previous frames X; for O-PLF- JMD. We show X, in the first column. From the second column, the light-dark region represents X, and the color digit represents X2, X2. For each perception metric. we show two samples.
We measure the diversity in X2 reconstruction using E[Var(Xa, Xx|X1, X2)] and the correlation with X, by Ejsim(X,, where sim(u, is the cosine distance between u, Table 9a shows that as we increase the number of bits in Rg, the diversity decreases as the decoder can reconstruct the frame more precisely. In Table Dp, we see that the joint metric keeps the correlation relatively constant, showing that it actually preserves the temporal consistency. On the other hand, as the rate becomes larger, 1-PLF-FMD reconstruction tends be less correlated with the previous frame X1. Finally, note that our architecture innately utilizes common randomness to produce diverse reconstructions and does not suffer from mode-collapse behavior in general conditional GAN settings 442
Xi X1 in this regime.
47

---

Table 9: Diversity (a) between X2 and Correlation Measures (b) between X2 and X
(a) Diversity Mcasures 1.
(b) Correlation Measures.
R2
Joint
Marginal
R2
Joint
Marginal
1 bit
0.0096
0.0004
bit 0.5218
0.6202
2 bits
0.0082
0.0029
2 bits
0.5190
0.5969
3.17 bits
0.0042
0.0022
3.17 bits
0.5205
0.5508
x1
PLF PLF-FMD X22
PLF-JD:X;
X1
PLF FMD XX²
PLF JD:
X1
x1
X₂
PLF - FMD XX2
PLF PLF-JD:X;
X1
X1
X2
PLF FMD :X²
PLF PLF-JD:X:
X1
PLF FMD :X2
PLF -D:
Figure 11: Visualization of permanence. The PLF-JD reconstructions propagate the flaws in the color tonc from the previous I-frame reconstruction while the PLF-FMD able to such error. Compression rate for I-frame and P-frame are 0. 0.144bpp (low rate) 4.632bpp (high rate) respectively.
K Error Permanence on UVG Dataset
demonstrate the phenomenon error permanence a large-scale scenario using the UVG dataset. Our P-frame compressor is trained on the Vimeo-90k dataset. As illustrated in Figure [1] when reconstructing I-frames, an inaccurate color tone introduced, which persists when employing PLF-JD. However, PLF-FMD effectively rectifies this issue within P-frames. Numerical results are
48

---

Table 10:
Distortion (X2 - x2)² evaluated across 3900 UVG frames. The PLF-JD reconstructions exhibit notably greater distortion compared to MMSE and PLF-FMD, aligning with our theoretical findings.
MMSH
PLF-FMD
PLF-JD
Distortion (MSE)
0.0026
0.0032
0.0168
in Table. [0] We note that for the I-frame, we use the pretrained model in 43 that targets high
perceptual quality and is also trained on Vimeo-90k dataset.
L Limitations
This work studies the effects of different perception loss functions, namely the PLF-JD and PLF-FMD, on the performance of lossy causal video compression. Our theoretical analysis and experiment reveal the error permanence phenomenon and show the universality principle, suggesting that MMSE representation can be transformed into other points on the DP tradeoffs.
In practice, one might want to combine these two losses, for example, perfect framewise realism (O-PLF FMD) while retaining some degree of temporal cohesion (PLF-JD small), which is not considered in this work. Furthermore, analysis for other types of video compression schemes, such as with B-frame, and scaling the universality compression architecture to high-definition videos are also desired.
49


Reviews:
[{"review_id": "ih1Q4qSUwy", "reviewer": "Reviewer_pcBB", "summary": "This paper systematically studies the rated-distortion-perception tradeoff in neural video compression. Since videos are made of consecutive frames, this paper considers two different perceptual metrics, including the joint distribution of all video frames (PLF-JD), and per-frame perceptual loss (framewise marginal distribution, PLF-FDM). Some conclusions are proposed regarding the choice of PLF-JD or PLF-FDM. \n\nIn addition, motivated by the universal encoded representations in previous RDP papers from the field of image compression, the authors in this paper demonstrate the video-version universal encoded representations hold when the perceptual constraint is PLF-FDM. While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. The universal encoded representations in video compression have several advantages for video compression such as when it needs to extract motion information from an MSE-based reconstruction.  \n", "questions": "See the abovementioned “weaknesses”.", "limitations": "Limitations are discussed in the final section of Appendix. This paper is in the field of video compression which does not involve strong negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality and quality of this paper are pretty good. The PLF-JD and PLF-FDM are two different but both reasonable perceptual metrics used for optimization of video compression models. This is the first paper that studies the RDP issue in video compression and the results and conclusions are insightful. Some important conclusions regarding RDP include:\n\n(1) There is a significant penalty in distortion when using PLF-JD in the low-rate regime.\n\n(2) While PLF-JD preserves better temporal consistency across video frames, it suffers from the permanence of error phenomenon in which the mistakes in reconstructions propagate to future frames. \n\nThe universal encoded representations for video compression are studied as well, which are demonstrates to be always held when using PLF-FDM (similar to compressing images sequentially). \n\nBesides, the effectiveness of RDP in video compression models is well verified by theoretical analysis in Gaussian-Markov sources. \n", "weaknesses": "(1)\t(Perhaps not as a weakness) I am wondering whether these conclusions of RDP in video compression can be extended to distortion-perception tradeoff in other video generation / restoration tasks. The joint-frame loss PLF-JD and the per-frame loss PLF-FDM should hold similar conclusions for video generation.\n\n(2)\tThe state-of-the-art neural video compression framework contains a motion encoder/decoder and a residual encoder/decoder, which are effective for high-resolution videos. It seems this paper studies the other neural video compression framework that does not explicitly transmit motion information. How to apply the proposed theory to such motion-residual-separated video compression framework?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pcBB for considering our work “the first paper that studies the RDP issue in video compression” and praising its “originality and quality”. Please find your concerns addressed below.\n\n1) (Perhaps not as a weakness) I am wondering whether these conclusions of RDP in video compression can be extended to the distortion-perception tradeoffs in other video generation/restoration tasks. The joint-frame loss PLF-JD and the per-frame loss PLF-FDM should hold similar conclusions for video generation.\n\nReply: Thank you for pointing this out. Indeed, the tasks of restoration can be interpreted as a video compression problem. For the causal video restoration task, the incoming corrupted frame can be considered as a message $M_i$ that the decoder needs to decode (restore) and the encoder simply sends $M_i$ by corrupting the “clean” source image $X_i$. As such the factor-of-two bound should hold for the  PLF-FMD metric (following [1], which considers the image restoration setting), but not for the PLF-JD metric, by following the same arguments presented in our paper.\n\nIn the context of video generation, recent SOTA methods (StyleGAN-V, MoCoGAN) frequently adopt implicit generation approaches, such as GANs, that bypass distortion loss. In this context, PLF-JD appears as a generally preferable choice, given its emphasis on temporal consistency. Within the VAE framework, methods like He et al.'s \"Probabilistic Video Generation using Holistic Attribute Control\" (ECCV 2018), often incorporate a distortion term. This category of models frequently produces blurry videos. Consequently, we posit that incorporating a PLF-FMD term could enhance the generation of realistic frames while sacrificing temporal consistency. Conversely, utilizing PLF-JD might introduce training challenges (as elaborated in the \"Significance of Results\" section in the rebuttals). We believe this is an interesting question for future research.\n\nFor non-causal video generation/restoration, where a video is treated as a 3D image, PLF-JD should be a better metric overall since previous results for RDP image compression (factor of 2 bound, universality) should hold with the PLF-JD metric. This is because the proof from previous works does not make assumptions on the dimensionality of the input. As such, the implication for the non-causal case is more straightforward (similar to results for images). In fact, this is also the message that we want to send to the community, that theoretical results for rate-distortion-perception depend heavily on the coding scheme one is using.\n\n2) The state-of-the-art neural video compression framework contains a motion encoder/decoder and a residual encoder/decoder, which are effective for high-resolution videos. It seems this paper studies the other neural video compression framework that does not explicitly transmit motion information. How to apply the proposed theory to such motion-residual-separated video compression framework?\n\n\nReply: On the theoretical side, the current framework can be adapted to incorporate the motion-residual-separated architecture. At each step, the source includes both motion and residual contents. The message that is sent to the decoder can contain two parts where one of them compresses the motion content while the other one is the compressed version of the residual content. The operational RDP region is defined similarly to Definition 2. So, all the previous discussions hold with a simple redefinition of the random variables.\n\n On the experimental side, many of the current state-of-the-art architectures do indeed contain a motion encoder/decoder and residual encoder/decoder. This is also replicated in our experiment section, where the scale-space-flow module is responsible for computing the optical flow field and warping (see ref [32] of our paper), and the conditional module for efficient residual compression (see ref [4] of our paper). Our proposed theory is general and applicable to this architectural design, which has been illustrated in the experiment section. \n"}, {"review_id": "4VhUV08tgh", "reviewer": "Reviewer_QNA6", "summary": "This paper examines the choice of perceptual loss for causal, low-latency video compression model with distortion-perception optimization. By using information theoretic analysis and deep-learning based experiments, the authors demonstrate that the choice of perceptual loss can have a significant effect on the decoded reconstruction, especially at low-bit rates.", "questions": "As mentioned above, it is suggested to have some experiments on sequences with more complex motion and coded bits to examine the proposed findings in addition to those toy examples in the submission.", "limitations": "No.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates the proposed assertions through theoretical derivations and detailed experiments, which are convincing enough. In addition, these conclusions provide a good guideline basis for subjective optimization of learned video compression methods.", "weaknesses": "Since this paper is oriented to the discussion of learned video compression, the experimental setup may be too simple to have generalized conclusions. For example, the used sequences (e.g., MNIST and KTH) for evaluation are too homogeneous to present the characteristics of videos. It should be better to have some comparisons on more complex sequences such as the UVG dataset. It is critical to illustrate whether these findings would be useful in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer QNA6 for considering our work as “a good guideline basis for subjective optimization of learned video compression”. Please find your concern addressed below\n\n1) Since this paper is oriented to the discussion of learned video compression, the experimental setup may be too simple to have generalized conclusions. For example, the used sequences (e.g., MNIST and KTH) for evaluation are too homogeneous to present the characteristics of videos. It should be better to have some comparisons on more complex sequences such as the UVG dataset. It is critical to illustrate whether these findings would be useful in practical applications.\n\nReply: The current empirical setup for comparing neural codecs often entails training the networks on the Youtube/Kinetics/Vimeo90k dataset (which can encompass nearly a million frames) and subsequently testing on high-resolution videos like UVG and MCL-JCV. While we are diligently working towards generating results on UVG/MCL-JCV, it may not be feasible to achieve this within the constraints of the rebuttal process. The scale of these datasets and the complex nature of GAN training make it challenging to complete within a limited timeframe. \n\nWe would like to emphasize that our work's major contribution lies in being the first to explore the impact of perception loss functions on the RDP tradeoff for video compression, with a strong theoretical focus and we use our experimental results to confirm our theoretical findings. Prior works on the rate-distortion-perception tradeoff, such as Blau and Michaeli (ICML 2019) [11] and Zhang et al. (NeurIPS 2021) [16], also presented empirical results on simpler image datasets (MNIST and SVHN) to demonstrate their theoretical findings. \n\nWhile our experiments on two datasets MovingMNIST and KTH are homogeneous (lacking other effects such as zooming, camera moving, etc.). As it is illustrated in Fig. 1(b), Fig. 2(a) and Fig. 7 (in the Appendix), the error permanence phenomenon, associated with joint distribution-based discriminators in the low-rate regime, occurs in even these two homogeneous datasets. This observation strongly suggests that the same effect is likely to be more pronounced in complex non-homogeneous videos.\n\nFor other results, Table 1 validates the existence of a \"factor of 2\" distortion bound through the use of a frame-level discriminator, aligning seamlessly with Theorem 1. Additionally, Fig. 3 serves as a compelling testament to the universality of MMSE-based representations. Building upon their theoretical foundation, we believe these outcomes to extend their applicability to diverse datasets. \n\nFinally, we note that these findings also come with applications. The universality aspect of MMSE representation emerges as crucial during the training of perceptual video compression models. Given that perceptual reconstruction frequently generates novel details not present in the source frame (especially when using the FMD-JD loss), compressing motion flow vectors between the current frame and the prior perceptual reconstruction necessitates a higher bit allocation compared to utilizing the MMSE representation. Therefore, this suggests that the recommended approach is to train end-to-end compression exclusively using the rate-distortion loss. Subsequently, users have the flexibility to opt for their preferred decoder based on their perceptual preferences.\n\nWe hope this response alleviates your concerns about the need to implement our experiments on more complex datasets. We also kindly request you to consider increasing the score.\n"}, {"review_id": "WVrYmWQWum", "reviewer": "Reviewer_tXFk", "summary": "The paper consider theoretical considerations of generative video compression.\n\nWhile rate-distortion-perception theory (Blau & Michaeli, 2019) explains the trade-off between realism and distortion for generaive compression, the causal processing typically introduced for video models adds constraints that need to be considered.\n\nThe authors consider two settings: FMD (frame marginal distributions) and JD (joint distribution). In the FMD case, for full realism an observer cannot tell apart reconstructed frames and original frames when looking at one frame at a time, but there can be synthetic detail that is not temporally consistent. In the JD case, for full realism the joint distribution of reconstructed videos matches the originals.\n\nThe main results of the paper are:\n\nTheorem 1: in the FMD setting, we can achieve full realism (“perfect perception”) by at most doubling the distortion of an MSE optimized system. This generalizes a result from (Blau & Michaeli, 2019 [16]) for the single frame scenario.\n\nTheorem 2: In the JD setting, the authors obtain a similar result but for a more constrained setting. They need the source at time j to be nearly independent of the encoder outputs upto and including time j. Interestingly, they demonstrate an example in the appendix where the factor-of-two bound is not achieable. \n\nThe example can be summarized as follows: if you have some loss of information in reconstruction 1, you need reconstruction 2 to be consistent with it even if you have perfectly encoded the original frame. Hence you can’t achieve a zero reconstruction error in frame 2 despite having losslessly encoded it, violating the factor of two bound.\n\nTheorems 3-6: Here the analysis is done for the more general rate-distortion-perception setting, where the perceptual quality/realism is sacrificed to improve distortion. The authors consider both the setting of first-order Markov sources and do further analysis in the gaussian special case.\n", "questions": "-", "limitations": "Toy data / causal setting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Relatively readable paper\n- Valuable insights into the generative video compression field\n- The theoretical results are validated in experiments on small datasets (Moving MNIST and KTH).\n", "weaknesses": "- Intuitively, why would we only consider frame level marginal distributions? Just like pixel-level marginal distributions don’t make sense, this should lead to massive temporal artefacts/flickering in the low rate regime. While e.g. [7] used only a frame-level discriminator, they use warping for temporal consistency.\n- I found the proof of Theorem 1 to be quite similar to the proof of Thm. 2 in Appendix B in (Blau & Michaeli, 2019). It seems to me that proof also works as-is in the setting where you condition on previously encoded frames, as there are no assumptions on how the MSE optimized reconstruction is obtained (so whether it is using information from previous frames or not doesn’t seem to matter in the derivation). I would appreciate if the author could clarify differences here. \n- Experiments are limited to toy data.\n\n\nMinor (not affecting rating): *The supplement is very long* (30+ pages) and the main paper contains 24 references to it. It seems to me the authors have a lot more to say than fits into 9 pages. I'm not sure what to do with this but it feels slightly weird. I did not read everything in the supplement.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer tXFk for considering our work a “relatively readable paper” and “valuable insights into the generative video compression field”. Please find your concerns addressed below.\n\n1) intuition on the use of PLF-FMD in the low-rate regime\n\nReply: Please note that in all our experiments the neural architecture we implement is similar to reference [7]. Our architecture also includes warping to match the locations of the current source frame using the decoded flow fields for temporal consistency. We demonstrate that such an architecture in conjunction with a frame-level discriminator (PLF-FMD metric in our paper) appears to provide the most desirable reconstruction when the I-frame is compressed at low bit-rates. This is because the proposed loss function is flexible enough to introduce meaningful corrections in the reconstruction in later frames as more information is received. While this might introduce some flickering effect, the effects are not random since the (high-fidelity) reconstructions tend to sync up with the source, as shown in Figures 2 and 7. In contrast, a discriminator that preserves joint distribution across frames (PLF-JD metric in our paper) is not only considerably more challenging to implement (due to the need for a different conditional decoder at each step/timeframe), but also suffers from an “error permanence” phenomenon — errors made in earlier frames do not get corrected in later frames, even when the decoder has sufficient information for doing so. We suspect that similar considerations between the joint and framewise metrics were made in prior works such as references [6] and [7], that also implement a frame-level discriminator. In [7], the authors mention (Section 3.5, second paragraph) that their attempt to include multiple frames in their GAN (a version of our JD) “did not significantly alter reconstructions”. Our work is the first attempt to theoretically study the impact of PLF-FMD and PLF-JD metrics using the framework of rate-distortion-perception (RDP) tradeoff. Our results in Theorems 1, 2, 5, as well as Table 2 (in the Appendix) provide insightful comparisons between the two metrics, which we further validate using experimental results. We note that our work is theoretically grounded in the study of RDP tradeoffs, and as is the case of prior works of Blau and Michaeli [11] and Zhang et al [16],  we have also presented our experimental studies on two simpler datasets. We expect that larger scale experimental studies that incorporate more complex datasets will result in similar conclusions. For example, we expect errors made in the previous frames to continue to propagate when PLF-JD metric is used, regardless of which dataset is used and PLF-FMD to be more flexible in correcting such errors. We hope that you agree with the importance of the PLF-FMD metric and better appreciate its importance in the context of prior work. Finally, we note that there are other rate regimes where the choice of PLF-JD is preferred over PLF-FMD (see Table 2 of Appendix F and experimental results in Appendix J.5).\n\n2) On the comparison of Theorem 1 with Thm. 2 in Appendix B in (Blau & Michaeli, 2019)\n\nReply: While direct extension of the achievable scheme in Thm 2 of (Blau & Michaeli, 2019) [11] by conditioning on previously encoded frames suffices to achieve the “factor of 2” bound in our Theorem 1, we note that (in contrast to [11]) the main result in Theorem 1 involves an exact characterization (see Eq. 5) of the distortion region. The “factor of 2 bound”  is derived  (see Eq. 6)  as a consequence of this characterization. To establish our main result, in addition to the “achievability part”, we had to also establish a converse which is not present in [11]. Nevertheless we have not claimed significant novelty in this part of the work.  As noted in the paragraph immediately below Theorem 1, our result in Eq. (5) is a generalization of an analogous result for single frame setting in Zhang et al (see reference [16] in our paper).  The intuitive reason why Theorem 1 follows as a natural extension of previous works (on single frame)  is because it considers a fixed encoder setting without explicitly considering the rate constraints. Nevertheless this setting provides insightful comparisons between the PLF-FMD and PLF-JD metrics as discussed in Section 3. We note that when the entire rate-distortion-perception tradeoff is considered our theoretical analysis is considerably more challenging than the prior works on single-frame setting in references [11] and [16]. Please look at the global rebuttal, the part on “Novelty of Theoretical Analysis” for a brief review of our theoretical challenges in the work.\n\n\n3) Experiments are limited to toy data.\n\nReply: Please see the global rebuttal, the part on “scope of experimental results”.\n\n4) Causal setting\n\nReply: For video compression, there are two frameworks being considered in the current literature: causal and non-causal frameworks. For the non-causal framework, people often compress the whole video (or a block of multiple subsequent frames), as in the following reference\n Habibian, Amirhossein, et al. \"Video compression with rate-distortion autoencoders.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\nWe would like to emphasize that results from previous RDP tradeoffs for images should naturally hold for this setting (with PLF-JD metric), specifically the factor-of-two bound and universality property since we can consider the whole video as a whole 3D image in this sense. This setting, however, is not efficient in real-time applications due to the delay in block compression/decompression. Most recent neural video compression methods, on the other hand, often consider the causal setting.\n\nWe hope to have addressed all your concerns satisfactorily and kindly ask you to consider increasing the score."}, {"review_id": "kCL434SXDd", "reviewer": "Reviewer_1Qbf", "summary": "This paper studies the choice of perception loss for learned video compression and summarizes some valuable conclusions. The first one is the pros and cons of PLF-JD and PLF-FMD. The second one is the universality of MMSE reconstructions.", "questions": "In the experiments, I do not know how the proposed method precisely controls the bit rate to make the rates of different methods to be identical.", "limitations": "See the weakness part.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper conduct experiments and give some theoretical analysis of the perception loss functions for learned video compression. It is valuable when designing and training a video compression pipeline.", "weaknesses": "1.  The datasets used in the paper are too simple. It is questionable that the conclusions still work for the datasets that are closer to the real world. It is suggested to provide more experiments on the widely used datasets in learned video compression methods, such as UVG, MCL-JCV, and HEVC Class B datasets. Most papers conduct their experiments on these datasets. So we can see more performance comparisons and verify the proposed method is not a toy example.\n\n2.  Considering this paper focus on perception loss, why not try some metrics about perceptual quality? For example, the experiments can also provide widely used perceptual metrics such as  LPIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1Qbf for considering our work “valuable when designing and training a video compression pipeline”. Please find your concerns addressed below. \n\n1) The datasets used in the paper are too simple. It is questionable that the conclusions still work for the datasets that are closer to the real world. It is suggested to provide more experiments on the widely used datasets in learned video compression methods, such as UVG, MCL-JCV, and HEVC Class B datasets. Most papers conduct their experiments on these datasets. So we can see more performance comparisons and verify the proposed method is not a toy example.\n\nReply: Please see the global rebuttal, the part on “scope of experimental results”.\n\n\n2) Considering this paper focuses on perception loss, why not try some metrics about perceptual quality? For example, the experiments can also provide widely used perceptual metrics such as LPIPS.\n\n\nReply:  Our focus is on the RDP tradeoff and validating these results using DL experiments. We chose the perception loss functions as they are natural counterparts of the theoretical metrics used. We also provide quality assessment metrics for your consideration. In the KTH experiment, we computed LPIPS, which is known as a full-reference perceptual metric for images. We compare LPIPS for each reconstructions (MMSE, 0-PLF-FMD and 0-PLF-JD) for each timeframe.\n\nLPIPS (lower is better) on KTH dataset:\n\n 1st frame: MMSE - 0.1036, 0-PLF-FMD - 0.0584, 0-PLF-JD - 0.0584\n \n2nd frame: MMSE - 0.0521, 0-PLF-FMD - 0.0313, 0-PLF-JD - 0.0594 \n\n3rd frame: MMSE - 0.0413, 0-PLF-FMD - 0.0232, 0-PLF-JD - 0.0613 \n\nThis result aligns with our results on distortion loss, as the MMSE and 0-PLF-FMD tend to correct the reconstruction (so the ground truth and reconstruction look more similar over time). On the other hand, due to error permanence, the 0-PLF-JD reconstructions become different from their source sequence, causing the score to go up. Finally, we would like to thank you for this suggestion. We are going to add this set of experiments in the new version of our work.\n\n3) In the experiments, I do not know how the proposed method precisely controls the bit rate to make the rates of different methods to be identical.\n\nReply: We control the bit rate by controlling the dimension of the latent variables D (encoder’s output) and the number of quantization levels L. The bit rate can then be calculated by $D \\times log_2(L)$. While this approach is a bit sub-optimal, it makes controlling the bitrate much easier compared to the well-known approach by Balle et. al (see ref [13] of our paper). Also, it has been adopted in previous prior works for learned image compression, such as (ref [11], [16] and [12] of our paper). \n\nWe hope this answers your concern. We also ask you to kindly reconsider your evaluation."}]

Author rebuttal:
We thank the reviewers for acknowledging the significance of the work. Reviewer pcBB notes that our work is the first paper that studies the RDP issue in video compression and finds the results and conclusions to be insightful. Similarly reviewers QNA6, txFk and 1Qbf have noted the significance of our results and conclusions to learned video compression. Reviewers QNA6 and txFk have also appreciated the quality of the presentation. In our rebuttal, we have taken into account the feedback provided by the reviewers and included some additional experimental results involving LPIPS score as suggested by the reviewers (see pdf file below). We note that the conclusions are largely consistent with the original results in the paper. We provide some general comments that address the concerns raised by the reviewers.
 
**Scope of experimental results:** Some of the prior experimentally oriented papers in neural video compression involve extensive training (Youtube/Kinetics) and complex testing (UVG) to validate the use of new metrics or neural architectures. Our work's major contribution lies in being the first to explore the impact of perception loss functions (PLFs) on the rate-distortion-perception (RDP) tradeoff, with a strong theoretical focus. The purpose of experiments is to provide qualitative visualization of our theoretical results using simpler datasets. For example in Fig. 1(b), Fig. 2(a) and Fig. 7 (in the Appendix) we demonstrate the error permanence phenomenon in the low-rate regime associated with a discriminator network that considers the joint distribution across video frames. Since this observation is inherent to the nature of the PLF as verified by our theoretical result (Thm 5), we do not expect this property to be specific to our choice of datasets and can be generalized to other datasets. Similarly Table 1 demonstrates the “factor of 2” bound in distortion when a frame-level discriminator is used during training consistent with Theorem 1 in the paper.  Likewise Fig. 3 demonstrates the sufficiency of MMSE representations for achieving near optimal performance. Since all these results are grounded in our theoretical analysis we again expect other datasets to yield similar conclusions. Note that prior works that provided theoretical study of the RDP tradeoff, such as the works of Blau and Michaeli (ICML 2019) [11] and Zhang et al (NeurIPS 2021) [16] also demonstrated their empirical results on simpler datasets such as MNIST and SVHN. Based on these considerations we request the reviewers who have provided us lower scores simply because the experimental results did not include additional datasets reconsider their evaluation. We also note that while we are making every effort to produce results on additional datasets, it may not be practically feasible for us to finish these results during the rebuttal process as these datasets are extremely large and the GAN training may not complete in such a short timeframe. Finally, we added experimental results with LPIPS score (requested by reviewer 1Qbf) and they are consistent with our previous observations (see the pdf file). 

**Significance of Results:** We establish that encoded representations designed to minimize the MSE distortion loss can suffice to achieve near optimal distortion-perception tradeoff for both the joint and per-frame perception loss constraints. Our work is the first one to introduce such a principle of universality in the context of learned video compression previously studied in the context of image compression [16, 25]. In fact we believe that universality of MSE representations is far more significant in the context of learned video compression. Given that perceptual reconstruction frequently generates novel details not present in the source frame, compressing motion flow vectors between the current frame and the prior perceptual reconstruction necessitates a higher bit allocation compared to utilizing the MMSE representation. Hence, a recommended approach is to train end-to-end compression exclusively to minimize MSE and use the method in the proposed work to achieve a (near-optimal) tradeoff between the distortion and perception losses as desired by the user. We also provide insightful analysis using the framework of RDP to compare two commonly used PLFs. We demonstrate (using theoretical analysis and experimental results) that the PLF using a joint distribution constraint can be overly restrictive at low compression rates and can lead to an undesired error permanence phenomenon. In contrast, PLFs based on per frame metric can have enough flexibility to perform desirable corrections in the reconstruction frames as more information is available to the decoder and may be preferred at low bit rates. However, in some other rate regimes, the choice of PLF-JD is preferred over PLF-FMD (see Table 2 of Appendix F and experimental results of Appendix J.5).

**Novelty of Theoretical Analysis:** The study of RDP region for learned video compression is considerably more challenging than the study of RDP function for a single frame in prior works. As established in Thm 3, the RDP region (for first-order Markov sources) involves a tradeoff between the compression rate assigned to each frame. Even for Gaussian sources, the RDP region does not have a simple closed form which makes the proof of optimality of Gaussian reconstructions (Thm 4) more involved. As a result a significant effort was devoted towards obtaining insights in various operating regimes. We summarized the main results in Table 2 in Appendix F and in Thm 5. Furthermore the proof of universality in Thm 6 (for Gaussian sources) is considerably more challenging as one has to consider the achievability of the entire RDP region as opposed to just points on the boundary of RDP function in [16]. Finally the results on the fixed-encoder setup are more general than prior works that required a characterization of the information RDP region, which we do not require.

Meta-review: The paper received 1xAccept, 1xWeak Accept, 1xBorderline Accept and 1xBorderline Reject from four Reviewers.

The authors provided responses.

After carefully checking the paper, the reviews, and the authors' responses the ACs agree with the reviewers that the work has merits, is of interest for the community and it can be published.

The authors are asked to include (part of) the information/details provided in their responses to the reviewers' comments and further refine their paper for camera ready.
Decision: Accept (poster)