Title: On the choice of Perception Loss Function for Learned Video Compression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Compression

Information Theory

Neural Compression
Abstract: We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss to target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD,  considers the joint distribution (JD) of all the video frames up to the current one, while the second metric, PLF-FMD,  considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have a significant effect on the reconstruction, especially at low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion  compared to PLF-FMD and further makes it more difficult to recover from errors  made in the earlier output frames. Although the choice of PLF decisively affects  reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. In particular, encoded representations generated by training a system to minimize the MSE (without requiring either PLF) can be  {\em near universal}  and can generate close to optimal reconstructions for either choice of PLF at the decoder.  We validate our results using (one-shot) information-theoretic analysis, detailed study of the rate-distortion-perception tradeoff of the Gauss-Markov source model as well as deep-learning based experiments on moving MNIST and KTH datasets.

On the Choice of Perception Loss Function for Learned Video Compression

Abstract

We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss to target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD, considers the joint distribution (JD) of all the video frames up to the current one, while the second metric, PLFFMD, considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have a significant effect on the reconstruction, especially at low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion compared to PLF-FMD and further makes it more difficult to recover from errors made in the earlier output frames. Although the choice of PLF decisively affects reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. In particular, encoded representations generated by training a system to minimize the MSE (without requiring either PLF) can be near universal and can generate close to optimal reconstructions for either choice of PLF at the decoder. We validate our results using (one-shot) information-theoretic analysis, detailed study of the rate-distortion-perception tradeoff of the Gauss-Markov source model as well as deep-learning based experiments on moving MNIST and KTH datasets. Code will be available at https://github.com/truongbuu/URDP_flow.

1 Introduction

There is an increasing demand for video compression algorithms that are able to generate visually pleasing videos at low bitrates. Most of the current video codecs use distortion measures such as PSNR [1-4], MSE and MS-SSIM [3-5] to generate reconstructions which tend to be blurry at extremely low bitrates. In recent years, there has been a growing interest (see e.g., [6-10]) in using deep generative models to make the reconstructions look more realistic. Such techniques introduce an additional perception loss function that measures a distance between distributions of the source and reconstruction, with perfect perception corresponding requiring that the two distributions be identical.

(a)

Encoders and decoders for

T

=3

frames.

TX

TX

f1

M

91

文1

Frame#1

#2

#3

#1

#2

#3

#1

#2

#3

4

4

4

Encoder 1

Decoder 1

2

2

2

1

f2

TX

M2i

--

1

92

x2

MMSE

艺

2

9

G

9

h

h

Encoder 2

Decoder 2

9

7

2

K

K

a-

G

6

M3

a

b

f3

93

Encoder 3

Decoder 3

X1

文1

X2

PLF-FMD:X2

PLF-JD: X2

Sig

X3

PLF-FMD:X3

PLF-JD: X3

0.34

AveragePixel IntensityPerFrame

Source

FMD

0.24

2

X1

人

I

X21

X3

Frame

In compression systems, improving realism comes at the price of increasing distortion. The work of Blau and Michaeli [11] establishes the theoretical rate-distortion-perception (RDP) tradeoff which has also been validated in [12-15]. Furthermore universal encoded representations were proposed in [16] where the representation is fixed at the encoder and the decoder is adapted to achieve a performance near the optimal RDP tradeoff curve. The extension of these works to video compression involves many challenges. First, the compression system must not only account for spatial redundancy as in image compression, but also exploit the temporal redundancy across video frames, making the system design more complex. Secondly, unlike the case of image compression, there may be no clear choice of the perception loss function (PLF). Indeed, some prior works [7] consider PLF that preserves framewise marginal distribution (PLF-FMD) between the source and reconstruction, while other works consider joint distribution (PLF-JD) across multiple frames [9].

As illustrated in Fig. 1a, we study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss and either a PLF-JD or PLF-FMD metric for perception loss. Our contributions are as follows:

Differences in reconstruction quality based on the choice of PLF : We demonstrate that the choice of PLF can decisively affect the reconstruction quality especially in the low bit-rate regime. We approximately characterize the operational RDP region on a per-frame basis for a first-order Markov source model and analyze the special case of Gauss-Markov sources in detail. We show that there is a significant penalty in distortion when using PLF-JD in the low-rate regime. On the experimental side, we demonstrate that while PLF-JD preserves better temporal consistency across video frames, it suffers from the permanence of error phenomenon in which the mistakes in reconstructions propogate to future frames. On the other hand, the PLF-FMD metric shows

more capability in correcting mistakes across frames (see Fig. 1b for visualizations involving three-frame videos). On the other hand, if the first frame is transmitted at high bit-rate, we demonstrate that PLF-JD performs better than PLF-FMD.

Universality of minimum mean square error (MMSE) reconstructions : We demonstrate that encoded representations generated from an encoder trained to minimize MSE reconstruction (without considering any PLF) suffice to produce close-to-optimal reconstructions for either choice of PLF. For general sources, we show that when using PLF-FMD, the MMSE reconstruction can be transformed to a reconstruction satisfying perfect perceptual quality by increasing the distortion at most by a factor of two . While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. For the Gauss-Markov source model we demonstrate exact universality i.e., encoded representation for the MMSE reconstruction can be adapted to achieve any other reconstruction in the RDP region. We also use deep learning based experiments to provide experimental evidence of these results. We note that the above notion of universal encoded representations based on MSE reconstruction can have significant advantages in practice. First, although the reconstructions associated with different choices of PLFs can be visually very different, universal representations delegate the choice of PLF to the decoder rather than requiring the encoder to commit to a specific PLF. Secondly, the universality of MMSE reconstructions is far more significant in the context of learned video compression. Given that perceptual reconstruction frequently generates novel details not present in the source frame, compressing motion flow vectors between the current frame and the prior perceptual reconstruction necessitates a higher bit allocation compared to utilizing the MMSE reconstruction. Hence, a recommended approach is to train end-to-end compression exclusively to minimize MSE and use our proposed scheme to achieve a (near-optimal) tradeoff between the distortion and perception losses as desired by the user.

The study of RDP region for learned video compression is considerably more challenging than the study of RDP function for a single frame in prior works ( [11,16]). This is because of the fact that the RDP region (for first-order Markov sources) involves a tradeoff between the compression rate assigned to each frame imposing a Markov structure on the reconstructions. For Gaussian sources, the proof of optimality of Gaussian reconstructions does not use the closed form of the RDP region due to its complexity. As a result, the study of RDP region for various operating regimes is quite more involved. Furthermore, for the proof of universality, one has to consider the achievability of the entire RDP region as opposed to just points on the boundary of RDP function in [16]. Finally the results on the fixed-encoder setup are more general than prior works ( [11, 16]) that required a characterization of the information RDP region, which we do not require.

Related Work

Perceptual Lossy Video Compression. Distribution preserving framework using Generative Adversarial Networks (GAN) has been widely adopted as a surrogate metric for perceptual quality in image [12, 17-19] and, recently, video compression [6-9]. Unlike image compression, where the choice of PLF is straightforward, there is currently no agreed-upon objective for lossy video compression. For instance, DVC-P [6] employs PLF-FMD to improve the visual quality per frame, ignoring the temporal coherence. Similarly, Mentzer et al. [7] utilize the per-frame metric with a conditional GAN model, and found no significant differences when using a GAN objective with multiple frames. Other works target temporal consistency by incorporating multiple frames in their GAN objective. This includes the work by Yang et al. [8], where every two consecutive frames are included, and by Veerabadran et al. [9], where they employ the PLF-JD metric in a non-causal setting. Unlike previous works, we study the impact of two different perception objectives, i.e. PLF-JD and PLF-FMD, on reconstructions in the causal setting, presenting theoretical properties that are verified by deep learning experiments. For the PLF-JD metric, we demonstrate the error permanence phenomenon , which, unlike the error propagation issue [18, 20], cannot be resolved by increasing the code rate assigned to the P frames.

RDP Tradeoff and Principle of Universality. Targeting the distribution preserving framework in lossy image compression, several theoretical works have shown the presence of RDP tradeoff [11,21-23], where perfect perception comes at a cost of increasing distortion by at most a factor of 2. Furthermore, an encoder that generates universal representations exists [16,24,25], which enables the decoder to freely choose the level of distortion-perception tradeoff it desires. Our work explores these avenues

in the context of causal video compression. As discussed previously we show that the MMSE representation can be used as a universal representation for both perception metrics which has several advantages in the context of video compression..

2 System Model

Let ( X 1 , . . . , X T ) ∈ X 1 × . . . × X T be T frames in a video (with each X i ⊆ R d ) distributed according to P X 1 ...X T . The frames are available for encoding sequentially; X 1 is available first, then X 2 arrives, followed by X 3 and so on. There is a shared randomness K ∈ K which is available at all encoders and decoders. The following (possibly stochastic) mappings define the encoding and decoding functions:

where M j ∈ { 0 , 1 } ⋆ denotes the set of (variable-length) messages assigned by the j th encoder and ˆ X j ⊆ R d is the j -th reconstruction alphabet (see Fig. 1a). Let P ˆ X 1 ... ˆ X T | X 1 ...X T be the conditional distribution of the reconstructed video given the original video which is basically determined by the mappings { f j } T j =1 and { g j } T j =1 . The above setting is a one-shot setup as only a single source sample is compressed at a time. For each frame j , a distortion metric is imposed on the output, which we assume throughout is the mean squared-error (MSE) function i.e. d ( x j , ˆ x j ) = || x -ˆ x j || 2 , which is commonly used in many applications. From a perceptual point of view, for given probability distributions P X 1 ...X j and P ˆ X 1 ... ˆ X j on the original and reconstructed frame j , let ϕ j ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) be the perception function capturing the difference between them. Note that the function ϕ j is defined based on the joint distribution of all first j frames. We call this metric as perception loss function based on joint distribution (PLF-JD) . Note that when ϕ j ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) = 0 , we have:

We refer to this case as zero-perception loss function based on joint distribution ( 0 -PLF-JD) . Alternatively, the perception loss function based on framewise marginal distribution (PLF-FMD) is denoted by ξ j ( P X j , P ˆ X j ) and is based on only the marginal distribution of the j -th frame. In particular, note that 0 -PLF-FMD implies that P X j = P ˆ X j for each j . In most of the paper, for simplicity of presentation, we provide some of our results for T = 3 frames. In that case, we use the shorthand notation X to denote the tuple ( X 1 , X 2 , X 3 ) , e.g., M := ( M 1 , M 2 , M 3 ) , D := ( D 1 , D 2 , D 3 ) , f := ( f 1 , f 2 , f 3 ) .

3 Distortion Analysis for a Fixed Encoder and Zero-perception Loss

In this section, we assume that the encoding functions f are fixed, but the decoding functions g can be optimized to generate different reconstructions. Equivalently, the distribution P M | X K := ✶ { M = f ( X , K ) } is fixed, while by varying the reconstruction distribution P ˆ X | M K := ✶ { ˆ X = g ( M , K ) } , one attains different reconstructions ˆ X , where ✶ { . } denotes the indicator function. Furthermore defining D j := P [ ∥ X j -ˆ X j ∥ 2 ] , we denote D as the achievable distortion tuple associated with P ˆ X | M K .

Φ D min ( P M | X K ) = { D : D j ≥ ❊ P [ ∥ X j -˜ X j ∥ 2 ] , j = 1 , 2 , 3 } (4) then the distortion tuple D associated with any reconstruction P ˆ X | M K satisfies D ∈ Φ D min ( P M | X K ) .

❊ One natural choice of reconstructions is the minimum mean squared error (MMSE) reconstruction function. At step j , the reconstruction, which we denote in this case by ˜ X j , is obtained by taking the conditional expectation of X j given all information at the decoder up to time j i.e., ˜ X j := ❊ P [ X j | M 1 . . . M j , K ] for each j = 1 , 2 , 3 . It is well known that the MMSE reconstruction functions minimize the reconstruction distortion i.e., if we define the set

The main result of this section is that assuming fixed encoder, the achievable distortions under 0 -PLF-FMD is at most twice of that under the MMSE distortion loss alone. The same conclusion also holds for 0-PLF-JD for a class of encoders operating at low rate. We first consider the case of 0 -PLF-FMD.

Definition 1 ( 0 -PLF-FMD Distortion) For an encoder P M | X K , the set Φ D 0 ( P M | X K ) denotes the set of all distortion tuples D for which there exists a reconstruction P ˆ X | M K satisfying P X j = P ˆ X j for each j ∈ { 1 , 2 , 3 } .

Theorem 1 The set Φ D 0 ( P M | X K ) is characterized as follows:

Φ D 0 ( P M | X K ) = { D : D j ≥ ❊ P [ ∥ X j -˜ X j ∥ 2 ] + W 2 2 ( P ˜ X j , P X j ) , j = 1 , 2 , 3 } , (5) where W 2 2 ( P X j , P ˆ X j ) denotes the Wasserstein-2 distance between the two distributions [26]. Furthermore, we also have that:

Proof: See Appendix A.

We remark that the proof of Theorem 1, operationally demonstrates that the MMSE reconstruction can be converted to another reconstruction satisfying 0 -PLF-FMD with at-most a factor of 2 increase in distortion, generalizing the result in [16] for the single frame scenario (see also [21]).

We next consider the case when zero perception loss is satisfied under the PLF-JD metric. Analogous to Φ D 0 ( P M | X K ) in Definition 1, one can define Φ joint D 0 ( P M | X K ) to be the set of distortions associated with reconstruction functions that satisfy (3). The analysis of Φ joint D 0 ( P M | X K ) is discussed in Appendix B as it is more involved. In general, the factor of two bound as in Theorem 1 cannot be realized in this case as demonstrated by a counter-example in Appendix B. Nevertheless, for a special family of encoders we can obtain a counterpart of Theorem 1. In this family of encoders, the source X j at time j is nearly independent of the encoder outputs up to and including time j , i.e., we can express:

where µ is a sufficiently small constant and the distribution Q noisy ( · ) could be arbitrary conditional distribution with same marginal as P X j . We note that such encoders are studied in a variety of problems in information theory (see e.g., [27]) that correspond to the low rate operating regime. The following result states that the factor-two bound holds approximately for such encoders.

Theorem 2 For the class of encoders given by (7) , we have

We note that the low-rate operating regime is practically important, as at higher rates MMSE based reconstructions can suffice and the use of PLF metrics may be less relevant.

4 Rate-Distortion-Perception Region

In this section, we assume that both the encoder P M | X K as well as the reconstruction P ˆ X | M K can be optimized and study the associated rate-distortion-perception (RDP) tradeoff. We remind the reader that for PLF-JD and PLF-FMD, the PLFs are denoted by ϕ j ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) and ξ j ( P X j , P ˆ X j ) , respectively. In this case, the operational RDP region in the one-shot setting is defined as follows.

Definition 2 ( Operational RDP region) For a given P X , an RDP tuple ( R , D , P ) is said to be achievable for the one-shot setting if there exist an encoder P M | X K and a reconstruction P ˆ X | M K satisfying:

❊ [ ℓ ( M j )] ≤ R j , ❊ [ ∥ X j -ˆ X j ∥ 2 ] ≤ D j , ϕ j ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) ≤ P j , j = 1 , 2 , 3 , (9) where ℓ ( M j ) denotes the length of the message M j . The closure of the set of all achievable tuples, denoted by C o RDP , is the operational RDP region. Moreover, for a given ( D , P ) , the operational DP rate region, denoted by R o ( D , P ) , is the closure of the set of all tuples R such that ( R , D , P ) ∈ C o RDP .

The region C o RDP cannot be directly computed as it involves all possible one-shot encoders/decoders. But for first-order Markov source, it has a tractable approximation in terms of mutual information.

4.1 RDP Region of First-Order Markov Sources

We first define the first-order Markov sources and then introduce an iRDP region which is computable.

Definition 3 We call X as a first-order Markov source if the Markov chain X 1 → X 2 → X 3 holds.

Definition 4 (Information RDP region) For first-order Markov sources, the information RDP (iRDP) region, denoted by C RDP , is the set of all tuples ( R , D , P ) which satisfy the following

for some deterministic functions η 1 ( . ) and η 2 ( ., . ) . Moreover, for a given ( D , P ) , the information DP (iDP) rate region, denoted by R ( D , P ) , is the closure of the set of all tuples R that ( R , D , P ) ∈ C RDP .

The expression for the iRDP region involves a search over auxiliary random variables X r and ˆ X that satisfy (12)-(14) subject to the constraints in (10)-(11). For first-order Markov sources, the following theorem states that the operational DP rate region can be approximated by the iDP rate region.

Theorem 3 For first-order Markov sources, a given ( D , P ) and R ∈ R ( D , P ) , we have

From Theorem 3, it follows that R ( D , P ) with overhead log( R +1)+5 is an inner bound to R o ( D , P ) . On the other hand, R ( D , P ) provides an outer bound to R o ( D , P ) . The two bounds match with each other in high rates. It can be shown that the overhead also vanishes in the large-blocklength setting where multiple symbols are encoded at a time. In the remainder of this paper, we will approximate R o ( D , P ) with R ( D , P ) and use the latter region for our analysis.

Remark 1 (Encoded Representations): The proof of the inner bound in Theorem 3 in Appendix D provides an operational interpretation to the auxiliary random variables X r = ( X r, 1 , X r, 2 , X r, 3 ) defined in iRDP region in Definition 4. In particular, X r,j is a lossy version of the source sample X j generated by the encoder in step j . It is compressed and transmitted to the decoder at rate R j in (10) . We refer to X r as the encoded representation of the source X . The Markov chains (13) -(14) indicate that without loss of optimality, an encoded representation X r,j can be computed from the source X j and past reconstructions X r, 1 , . . . , X r,j -1 without using past source samples X 1 , . . . , X j -1 .

Remark 2 (Deterministic Reconstructions): Note that the reconstruction functions generating ˆ X in Definition 4 are deterministic functions of the encoded representations (c.f. (12) ). In particular, the shared randomness K is not required in the reconstruction functions. However, as the proof of the inner bound of Theorem 3 illustrates, the shared randomness is required in the compression and construction of X r,j . Moreover, by following the arguments in [28], one can set the reconstruction function of the last frame to be identity. Thus, in Definition 4, we have set ˆ X 3 = X r, 3 in (12) where T = 3 . In the sequel, for T frames we will set ˆ X T = X r,T .

Remark 3 The result in Theorem 3 also holds for the PLF-FMD. That is, one can replace the PLF in (9) and (11) by ξ j ( P X j , P ˆ X j ) and get a similar result (see Appendix D for the justification).

4.2 Gauss-Markov Source Model: RDP Region

In this section, we obtain practical insights through the analysis of the special case of first-order Gauss-Markov sources. We assume that X 1 ∼ N (0 , σ 2 1 ) ,

where N j is independent of X j with mean zero and variance (1 -ρ 2 j ) σ 2 j +1 for j = 1 , 2 . Note that the model extends naturally to the case of T time-steps. The perception metric is assumed to be the Wasserstein-2 distance, i.e., ϕ j ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) := W 2 2 ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) . For the PLF-FMD, the perception metric is given by ξ j ( P X j , P ˆ X j ) := W 2 2 ( P X j , P ˆ X j ) .

The following result states that for Gaussian sources, jointly Gaussian reconstructions are optimal. Thus, for a given tuple ( D , P ) , the characterization of R ( D , P ) becomes computable.

Theorem 4 For the Gauss-Markov source model, any tuple ( R , D , P ) ∈ C RDP can be attained by a jointly Gaussian distribution over X r and identity mappings for η j ( · ) in Definition 4.

Generally, the optimized distribution in the above theorem may not admit a simple form. In the special case of T = 2 frames, the optimal reconstructions are given in Appendix E. To obtain practical insights, we consider various asymptotic operating regimes and provide a detailed analysis in Appendix F for the case of T = 2 frames and with σ 2 1 = σ 2 2 . A summary of these results is provided in Table 2 in the same Appendix. We briefly summarize some of these results next.

4.3 Gauss-Markov Source Model: Extremal Rates

One of the key observations of this paper is that the choice of PLF has implication on the rate allocation across different frames. Specifically, first consider the case when both R 1 = R 2 are small i.e., R 1 = R 2 = ϵ (for small enough ϵ ). We discuss how each PLF affects the reconstruction in the second step. In the first step, we note that reconstruction in both cases must be identical and of the form ˆ X G 1 = √ 2 ϵ ln 2 X 1 + Z 1 where Z 1 ∼N (0 , (1 -2 ϵ ln 2) σ 2 ) is independent of X 1 ; the resulting distortion is given by D 1 =2(1 -√ 2 ϵ ln 2) σ 2 . However, the reconstructions in the second steps will be different for the two measures. For simplicity, we consider the extreme case when ρ =1 (i.e., when X 2 = X 1 ). Here, the PLF-JD metric is required to preserve perfect correlation and thus has to set ˆ X G 2 = ˆ X G 1 and results in D 2 = D 1 . In other words, the decoder in the second step is unable to use any information transmitted in the second step as 0-PLF-JD enforces the stringent constraint ˆ X G 2 = ˆ X G 1 . In contrast, for the PLF-FMD metric, it can be shown that the reconstruction in the second step for ρ =1 reduces to ˆ X G 2 = √ 2 √ 2 ϵ ln 2 X 1 + Z FMD and the associated distortion is given by D 2 =2(1 -√ 4 ϵ ln 2) σ 2 , which is lower than PLF-JD. Extending this example to T steps (with ρ =1 ), we note that PLF-JD will always be forced to output ˆ X 1 , while the reconstruction using PLF-FMD will successively improve. The following theorem formalizes this observation.

Theorem 5 For sufficiently small ϵ , let R j = ϵ and suppose that ρ j = ρ and σ j = σ , for j = 1 , . . . , T . The achievable distortions D FMD ,j (for 0 -PLF-FMD) and D JD ,j (for 0 -PLF-JD) are:

In particular, specializing to ρ = 1 , ∆ FMD ,j = 2 j -1 2 and ∆ JD ,j = 1 . This shows that the decrease in D FMD ,j is exponential at each step which implies the ability of decoder based on 0 -PLF-FMD in correcting mistakes and not propagating them in future reconstructions. However, as discussed previously the decoder which uses 0 -PLF-JD is stuck at ˆ X j = ˆ X 1 when ρ = 1 and results in no subsequent improvement in the distortion. We call this behaviour as permanence of error . This phenomenon is magnified in the case when R 1 → 0 and R 2 →∞ , treated in Table 2 (Appendix F) as the PLF-JD severely constrains the decoder to copy the previous noisy reconstruction while the flexibility provided by PLF-FMD reduces the distortion in the second step.

The case when R 1 → ∞ and R 2 = ϵ treated in Table 2 in Appendix F corresponds to the case when X 1 is sent at a sufficiently high rate (as is the case with some I-frames) while X 2 is sent at a low rate. Naturally, we have ˆ X G 1 ≈ X 1 for both PLFs. On the other hand, we once again see a qualitatively different behaviour in the reconstruction of X 2 . For the case of 0 -PLF-FMD, we have ˆ X G 2 ≈ (1 -O ( ϵ )) ˆ X G 1 + O ( ϵ ) X 2 , i.e., the decoder essentially copies the previous frame with

little contribution from the second step. In contrast, for the case of 0 -PLF-JD, it can be shown that ˆ X G 2 ≈ ρ 1 ˆ X G 1 + O ( √ ϵ ) X 2 + Z JD, where Z JD is independent Gaussian noise with variance close to 1 -ρ 2 . We note that the PLF-JD metric prevents the decoder from simply 'copying' the previous frame, but instead forces the decoder to generate a more diverse representation consistent with the joint distribution between the two frames.

4.4 Universal Representations for Gauss-Markov Source Model

In this section, we show that the Gauss-Markov source model admits universal encoded representations. Such representations can be transformed through appropriate reconstruction functions to achieve the entire DP rate region. This is the counterpart of the result for general sources in Theorem 1 where it is shown that the MMSE reconstructions can be transformed to some target reconstructions satisfying the 0 -PLF-FMD with at most a factor-2 increase in distortion. In contrast, we demonstrate that the Gauss-Markov model admits exact universality i.e., target reconstructions proposed in this section achieve all points in the iDP rate region. Interestingly, the transformation is linear with possibly some additive noise. First, we formalize the notion of universal representations.

Definition 5 (iDP-Tradeoff) For a given rate tuple R , the optimal iDP-tradeoff is the closure of the set of all tuples ( D , P ) such that ( R , D , P ) ∈ C RDP and is denoted by DP ( R ) .

Definition 6 (Universal Representation) A given encoded representation X r is called universal with respect to rate tuple R if it satisfies the rate constraints (10) and the Markov chains in (13) -(14) and for each ( D , P ) ∈ DP ( R ) , there exists a reconstruction ˆ X generated from P ˆ X | X r achieving it.

For the Gauss-Markov source model, we show that the MMSE reconstruction admits a universal representation. We consider the reconstruction ˆ X r that achieves minimum distortion in the DP ( R ) region. This point is explicitly characterized in Appendix H.1. Furthermore, following Theorem 4, since the reconstruction functions η j ( · ) are identity, the MMSE reconstruction is equivalent to MMSE representation i.e., ˆ X r = X RD r .The following theorem establishes that any point in DP ( R ) can be achieved from ˆ X r .

Theorem 6 For the Gauss-Markov source model and a given rate tuple R with strictly positive components, let the MMSE representation be denoted as X RD r = ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) . Let ( D , P ) ∈ DP ( R ) and let ˆ X = ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) be the corresponding reconstruction achieving it. Then there exist κ 1 , θ 1 , θ 2 , ψ 1 , ψ 2 and ψ 3 and noise variables ( Z 1 , Z 2 , Z 3 ) independent of ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) , which satisfy the following

The above theorem indicates that the MMSE representation can be linearly transformed to achieve any point in DP ( R ) . In general the MMSE representation may have to be degraded through additional noise terms. In the proof of Theorem 6 we identify conditions when such degradation is not needed.

As discussed in Appendix H.2, Theorem 6 holds for both PLFs. This suggests the idea that one can train an encoder to get MMSE representations which are oblivious to the choice of PLF. Then, the decoder can generate a reconstruction which satisfies either of PLFs by simply applying a linear transformation to the MMSE representation. Thus, the task of choosing the right PLF can be assigned to the decoder based on distortion and perception requirements. We conclude by noting that Appendix H.3 provides an example where the coefficients in Theorem 6 can be computed explicitly.

5 Experimental Results

We conduct experiments on the MovingMNIST dataset [29] (with 1 digit) using Wasserstein GAN [30], to verify the implications of our theoretical claims to perceptual video compression. Additional results on the KTH dataset [31] are available in Appendix J.3. Our compression network is built on the scale-space flow model [32] and conditional module [33]. For a given rate and PLF, we obtain different distortion-perception tradeoff points by optimizing the weighted sum between distortion

-

-

GroundTruth

MMSE

S

S

S

<

3

3

3

5

。

。

。

3

3

3

3

。

。

。

6

6

6

G

6

6

9

6

3

3

3

5

5

5

5

3

8

3

3

3

3

3

β

a

3

3

3

3

3

8

B

9

6

9

8

8

8

3

8

X1x2x3

X1x2X3

x1x2

1st (

R

1

= 12

)

2nd

3rd

0

.

0226

0

.

0228

0

.

0230

0

.

0232

0

.

0234

Distortion

R

2

=

R

3

= 4

, 0-PLF-JD

R

2

=

R

3

= 8

, 0-PLF-JD

R

2

=

R

3

= 12

, 0-PLF-JD

1st (

R

1

= 12

)

2nd

3rd

Frame

0

.

012

0

.

014

0

.

016

0

.

018

0

.

020

0

.

022

0

.

024

Distortion

R

2

=

R

3

= 4

, 0-PLF-FMD

R

2

=

R

3

= 8

, 0-PLF-FMD

R

2

=

R

3

= 12

, 0-PLF-FMD

(b)

Distortion per frame

(

X

i

-

ˆ

X

i

)

2

for

different rates with

i

= 1

,

2

,

3

.

and perception losses. Details about the architecture and training procedure are available in the Appendix J.1. The experimental setup is focused on validating our theory, rather than proposing stateof-the-art neural network architectures. Accordingly, we begin by (1) validating Theorems 1 and 2, which characterize the factor-of-two bounds on the distortion of 0-PLF reconstructions (2) empirically demonstrating the error permanence phenomenon of the PLF-JD in Section 4.3 and (3) computing the DP tradeoff function experimentally as well as confirming that the MMSE reconstruction provide near universal representations, as motivated by the results in Section 4.4.

As our first experimental result in Table 1, we validate the factor of two bounds in Theorems 1 and 2. We consider the compression of two frames X 1 and X 2 at rates R 1 and R 2 respectively. The compression of X 1 is performed without any prior reference and corresponds to the compression of the 'I-frame', while the compression of X 2 corresponds to the 'P-frame', using X 1 as the reference. We consider the cases when either R 1 = ∞ or R 1 =12 bits, where the former corresponds to lossless compression of X 1 and the latter corresponds to the low rate regime (see Appendix I for a justification). The average distortion for the

(a) Case 1: R 1 = ∞ bits

first frame when R 1 = 12 is 0 . 0124 for the MMSE reconstruction and 0 . 0235 for the 0 -PLF reconstruction, thus satisfying the factor of two bound. In compression of X 2 , we systematically vary the value of the rate R 2 ∈{ 4 , 8 , 12 , ∞} . Following Table 1b, for 0-PLF-JD reconstruction, only R 2 =4 bits (low rate) satisfies the factor of two bounds as expected. Intuitively, even as more bits are acquired, the 0-PLF-JD criteria actively restricts improving the reconstructions, resulting in persistently higher distortion. Even when R 2 = ∞ , the distortion remains non-zero as the decoder is forced to maintain temporal consistency with ˆ X 1 . In contrast, for FMD, the factor of 2 bound holds at all rates, consistent with Theorem 1.

In Fig. 2, we present our experimental results with a group of pictures (GOP) of size 3 (i.e. one I-frame followed by two P-frames). In Fig. 2a, we visualize sample reconstructions for MSE, 0PLF-FMD and 0-PLF-JD cases when operating in the low-rate regime with R j =12 bits for j =1 , 2 , 3 . Note that given an incorrect digit reconstruction in ˆ X 1 , the decoder with 0-PLF-JD consistently produces incorrect digits (or content) while the 0-PLF-FMD gradually 'corrects' it, which confirms the error permanence phenomenon discussed in the theoretical analysis in Section 4.3 and Table 2 in Appendix F. We also plot the framewise distortion in Fig. 2b to show the difference in achievable distortion two perception metrics across different values for R 2 and R 3 as a function of the frame index. Consistent with Theorem 5, the achievable distortion decreases much faster for 0-PLF-FMD than 0-PLF-JD for all selection of rates. Finally, we show similar results in Figure 3 for UVG dataset.

In Fig. 4, we plot the tradeoff curves between distortion and perception for the second reconstruction ˆ X 2 for both optimal (end-to-end) and universal representations for two cases: when R 1 = ∞ and R 1 =12 bits and for a range of values for R 2 . In general, the curves for both universal and optimal

X2

PLF-FMD:X2

PLF-JD:X2

X3

PLF-FMD:X3

PLF-JD:X3

0.42

AveragePixelIntensityPerFrame

0.38

Source

FMD

0.36

0.34

0.32

representations are relatively close to each other at all rate regimes. The general shape of every curve is relatively similar with the exception of the PLF-JD metric in Fig. 4b, where the curves for different rates seemingly converge since increasing the rate does not significantly improve the distortion in this case as noted previously. Finally, as the universal encoders are derived from MMSE solutions, these results imply that one can simply send the MMSE representation to the decoder and the user can flexibly change the DP tradeoff up to their requirements. We further note that even when the end-to-end model targets an operating point different from the MMSE reconstruction, the latter is still required to estimate the motion flow vectors best. The universal representation provides a natural way to reconstruct the MMSE reconstruction from the encoder output. In the plots of Fig. 4, we leverage on established universality results for I-frame compression in prior works [16] to construct the MMSE representation for motion compensation as we have a GOP of size 2 .

0

.

005

0

.

010

0

.

015

0

.

020

Distortion:

||

X

2

-

ˆ

X

2

||

2

2

2

4

6

PLF-JD:

W

1

(

PX

1

,X

2

, P

ˆ

X

1

,

ˆ

X

2

)

0

.

005

0

.

010

0

.

015

Distortion:

||

X

2

-

ˆ

X

2

||

2

2

0

2

4

6

PLF-FMD:

W

1

(

PX

2

, P

ˆ

X

2

)

3.17 bit universal

3.17 bit end-to-end

2 bit universal

2 bit end-to-end

1 bit universal

1 bit end-to-end

(a)

Case 1:

R

1

=

∞

,

R

2

=

{

1

,

2

,

3

.

17

}

bits

0

.

01

0

.

02

Distortion:

||

X

2

-

ˆ

X

2

||

2

2

2

4

6

PLF-JD:

W

1

(

PX

1

,X

2

, P

ˆ

X

1

,

ˆ

X

2

)

∞

bit

12 bit universal

8 bit universal

4 bit universal

12 bit end-to-end

8 bit end-to-end

4 bit end-to-end

0

.

010

0

.

015

0

.

020

Distortion:

||

X

2

-

ˆ

X

2

||

2

2

2

4

6

PLF-FMD:

W

1

(

PX

2

, P

ˆ

X

2

)

∞

bit

12 bit universal

8 bit universal

4 bit universal

12 bit end-to-end

8 bit end-to-end

4 bit end-to-end

(b)

Case 2:

R

1

=12

,

R

2

=

{

4

,

8

,

12

,

∞}

bits

Finally in Appendix J.5, we consider the ability of the decoder to generate diverse reconstructions when operating under either PLF-JD or PLF-FMD. We focus on the case when X 1 is transmitted losslessly and when X 2 is compressed at low rates. Consistent with the theoretical analysis in Section 4.2 and Table 2 in Appendix F, the decoder optimized for PLF-JD is capable of producing diverse reconstructions by mimicking the actual motion between the frames. The PLF-FMD leads to reconstructions that are highly correlated and less desirable.

6 Conclusions

This work examines different perception loss functions for causal video coding, establishing its key theoretical properties such as the operational RDP region and universality principle. Our analysis highlights that while 0-PLF-JD reconstruction preserves temporal correlation, it is susceptible to the error permanence phenomenon. Moreover, our investigation of universality reveals that the encoder can transform the MMSE representation to other points on the DP tradeoffs, irrespective of the PLF. We suggest future research directions such as exploring region-based perceptual metrics [34], incorporating image-aware bits allocation, and leveraging conditional perception metric [18].

References

E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, 'Scale-space flow for end-to-end optimized video compression,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2020, pp. 8503-8512.

R. Yang, Y. Yang, J. Marino, and S. Mandt, 'Hierarchical autoregressive modeling for neural video compression,' 2020. [Online]. Available: https://arxiv.org/pdf/2010.10258.pdf

O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle, and L. Bourdev, 'Elf-vc: Efficient learned flexible-rate video coding,' 2021. [Online]. Available: https://arxiv.org/abs/2104.14335

J. Li, B. Li, and Y . Lu, 'Deep contextual video compression,' in Advances in Neural Information Processing Systems , 2021, pp. 18 114-18 125.

A. Golinski, R. Pourreza, Y. Yang, G. Sautiere, and T. S. Cohen, 'Feedback recurrent autoencoder for video compression,' in Proceedings of the Asian Conference on Computer Vision , 2020.

S. Zhang, M. Mrak, L. Herranz, M. G. Blanch, S. Wan, and F. Yang, 'Dvc-p: Deep video compression with perceptual optimizations,' in 2021 International Conference on Visual Communications and Image Processing (VCIP) . IEEE, 2021, pp. 1-5.

F. Mentzer, E. Agustsson, J. Ballé, D. Minnen, N. Johnston, and G. Toderici, 'Neural video compression using gans for detail synthesis and propagation,' in European Conference on Computer Vision , 2022.

R. Yang, L. Van Gool, and R. Timofte, 'Perceptual learned video compression with recurrent conditional gan,' arXiv preprint arXiv:2109.03082 , vol. 1, 2021.

V. Veerabadran, R. Pourreza, A. Habibian, and T. Cohen, 'Adversarial distortion for learned video compression,' 2021. [Online]. Available: https://arxiv.org/pdf/2004.09508.pdf

Y. Wang, P. Bilinski, F. Bremond, and A. Dantcheva, 'G 3 an: Disentangling appearance and motion for video generation,' in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Y. Blau and T. Michaeli, 'Rethinking lossy compression: The rate-distortion-perception tradeoff,' in International Conference on Machine Learning . PMLR, 2019, pp. 675-685.

E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool, 'Generative adversarial networks for extreme learned image compression,' in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 221-231.

J. Ballé, V . Laparra, and E. P. Simoncelli, 'End-to-end optimized image compression,' in 5th International Conference on Learning Representations , 2017.

L. Theis, W. Shi, A. Cunningham, and F. Huszár, 'Lossy image compression with compressive autoencoders,' in 5th International Conference on Learning Representations , 2017.

F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. V. Gool, 'Conditional probability models for deep image compression,' in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

G. Zhang, J. Qian, J. Chen, and A. Khisti, 'Universal rate-distortion-perception representations for lossy compression,' in Advances in Neural Information Processing Systems , 2021, pp. 11 517-11 529.

C. Gao, T. Xu, D. He, Y. Wang, and H. Qin, 'Flexible neural image compression via code editing,' Advances in Neural Information Processing Systems , vol. 35, pp. 12 184-12 196, 2022.

F. Mentzer, G. Toderici, M. Tschannen, and E. Agustsson, 'High-fidelity generative image compression,' in Advances in Neural Information Processing Systems , 2020.

E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool, 'Generative adversarial networks for extreme learned image compression,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 221-231.

G. Lu, C. Cai, X. Zhang, L. Chen, W. Ouyang, D. Xu, and Z. Gao, 'Content adaptive and error propagation aware deep video compression,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 . Springer, 2020, pp. 456-472.

Y. Blau and T. Michaeli, 'The perception-distortion tradeoff,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 6228-6237.

D. Freirich, T. Michaeli, and R. Meir, 'A theory of the distortion-perception tradeoff in wasserstein space,' Advances in Neural Information Processing Systems , vol. 34, pp. 25 661-25 672, 2021.

N. Saldi, T. Linder, and S. Yüksel, 'Randomized quantization and optimal design with a marginal constraint,' in 2013 IEEE International Symposium on Information Theory . IEEE, 2013, pp. 2349-2353.

Z. Yan, F. Wen, R. Ying, C. Ma, and P. Liu, 'On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework,' in International Conference on Machine Learning . PMLR, 2021, pp. 11 682-11 692.

E. Agustsson, D. Minnen, G. Toderici, and F. Mentzer, 'Multi-realism image compression with a conditional generator,' arXiv preprint arXiv:2212.13824 , 2022.

V. M. Panaretos and Y. Zemel, An invitation to statistics in Wasserstein space . Springer, 2020.

A. Makur, Information contraction and decomposition . PhD Thesis, MIT, 2019.

N. Ma and P. Ishwar, 'On delayed sequential coding of correlated sources,' IEEE Trans. on Info. Theory , vol. 57, no. 6, pp. 3763-3782, 2011.

N. Srivastava, E. Mansimov, and R. Salakhudinov, 'Unsupervised learning of video representations using lstms,' in International conference on machine learning . PMLR, 2015, pp. 843-852.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, 'Improved training of wasserstein gans,' Advances in neural information processing systems , vol. 30, 2017.

C. Schuldt, I. Laptev, and B. Caputo, 'Recognizing human actions: a local svm approach,' in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. , vol. 3. IEEE, 2004, pp. 32-36.

E. Agustsson, D. Minnen, N. Johnston, J. Balle, S. J. Hwang, and G. Toderici, 'Scale-space flow for end-to-end optimized video compression,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 8503-8512.

J. Li, B. Li, and Y . Lu, 'Deep contextual video compression,' Advances in Neural Information Processing Systems , vol. 34, pp. 18 114-18 125, 2021.

E. Pergament, P. Tandon, O. Rippel, L. Bourdev, A. G. Anderson, B. Olshausen, T. Weissman, S. Katti, and K. Tatwawadi, 'Pim: Video coding using perceptual importance maps,' arXiv preprint arXiv:2212.10674 , 2022.

C. T. Li and A. El Gamal, 'Strong functional representation lemma and applications to coding theorems,' IEEE Trans. on Info. Theory , vol. 64, no. 11, pp. 6967-6978, 2018.

P. Stavrou, M. Skoglund, and T. Tanaka, 'Sequential source coding for stochastic systems subject to finite rate constraints,' IEEE Trans. on Automatic Control , vol. 67, no. 8, pp. 3822-3835, 2022.

A. Khina, V. Kostina, A. Khisti, and B. Hassibi, 'Tracking and control of gauss-markov processes over packet-drop channels with acknowledgments,' IEEE Trans. on Cont. of Net. Systems , vol. 6, no. 2, pp. 549-560, 2019.

A. El Gamal and Y. H. Kim, Network Information Theory . Cambridge University Press, 2011.

E. Denton and R. Fergus, 'Stochastic video generation with a learned prior,' in International conference on machine learning . PMLR, 2018, pp. 1174-1183.

Y.-H. Kwon and M.-G. Park, 'Predicting future frames using retrospective cycle gan,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 1811-1820.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, 'The unreasonable effectiveness of deep features as a perceptual metric,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 586-595.

S. Hong, D. Yang, Y. Jang, T. Zhao, and H. Lee, 'Diversity-sensitive conditional generative adversarial networks,' in 7th International Conference on Learning Representations, ICLR 2019 . International Conference on Learning Representations, ICLR, 2019.

R. Yang and S. Mandt, 'Lossy image compression with conditional diffusion models,' arXiv preprint arXiv:2209.06950 , 2022.

A Distortion Analysis for 0 -PLF-FMD

Recall the definition of Wasserstein-2 distance [26] as follows. For given distributions P X j and P ˜ X j , let

❊ where the infimum is over all joint distributions of ( X j , ˜ X j ) with marginals P X j and P ˜ X j .

Theorem 1 The set Φ D 0 ( P M | X K ) is characterized as follows:

Furthermore, we also have that:

Proof: Define

D 0 := { D : D j ≥ ❊ [ ∥ X j -˜ X j ∥ 2 ] + W 2 2 ( P ˜ X j , P X j ) , j = 1 , 2 , 3 } . (21) First, we show that Φ D 0 ( P M | X K ) ⊆ D 0 . For any D ∈ Φ D 0 ( P M | X K ) , there exists ˆ X D 0 = ( ˆ X D 0 1 , ˆ X D 0 2 , ˆ X D 0 3 ) jointly distributed with ( M , X , K ) such that

Then, for example, the analysis for the second frame is as follows

❊ ❊ ≥ ❊ [ ∥ X 2 -˜ X 2 ∥ 2 ] + W 2 2 ( P ˜ X 2 , P ˆ X D 0 2 ) (27) = ❊ [ ∥ X 2 -˜ X 2 ∥ 2 ] + W 2 2 ( P ˜ X 2 , P X 2 ) , (28) where (26) holds because both ˜ X 2 and ˆ X D 0 2 are functions of ( M 1 , M 2 , K ) and thus the MMSE ( X 2 -˜ X 2 ) is uncorrelated with ( ˆ X D 0 2 -˜ X 2 ) ; (28) follows because the 0 -PLF-FMD implies that P ˆ X D 0 2 = P X 2 . Following similar steps for other frames, we get Φ D 0 ( P M | X K ) ⊆ D 0 .

Next, we show that D 0 ⊆ Φ D 0 ( P M | X K ) . Assume that D ∈ D 0 . Let ˆ X ∗ 1 be an auxiliary random variable jointly distributed with ( M 1 , K ) such that it satisfies the following conditions

Moreover, let ˆ X ∗ 2 be an auxiliary random variable jointly distributed with ( M 1 , M 2 , K ) such that the following two conditions are satisfied

and and

Similarly, we define ˆ X ∗ 3 . Now, notice that since D ∈ D 0 , we have:

It then directly follows that

where

(34) follows because ˜ X 2 and ˆ X ∗ 2 are functions of ( M 1 , M 2 , K ) and thus the MMSE ( X 2 -˜ X 2 ) is uncorrelated with ( ˆ X ∗ 2 -˜ X 2 ) ;

(35) follows from (32);

(36) follows because P ˆ X ∗ 2 = P X 2 .

Following similar steps for other frames, we get D ∈ Φ D 0 ( P X r | X ) .

Now, notice that W 2 2 ( P ˜ X 2 , P X 2 ) ≤ ❊ [ ∥ X 2 -˜ X 2 ∥ 2 ] since the Wasserstein-2 distance takes the infimum over all possible joint distributions ( X 2 , ˜ X 2 ) , but the expectation in ❊ [ ∥ X 2 -˜ X 2 ∥ 2 ] is taken over the given P X 2 ˜ X 2 . Thus, we get

This concludes the proof.

B Distortion Analysis for 0 -PLF-JD

Let ˆ X ∗ 1 be defined as in (29)-(30). Moreover, let ˆ X ∗ 2 be an auxiliary random variable jointly distributed with ( M 1 , M 2 , K ) such that the following conditions are satisfied

and

Then, the following result holds.

Theorem 2 We have

Proof: Define

Now, assume that D ∈ D 0 joint . For the first frame, recall that ˆ X ∗ 1 is an auxiliary random variable jointly distributed with ( M 1 , K ) such that it satisfies (29)-(30). From similar steps to (34)-(36), it then follows that

For the second frame, since D ∈ D 0 joint , we have:

Recall that ˆ X ∗ 2 is an auxiliary random variable jointly distributed with ( M 1 , M 2 , K ) such that (39)-(40) hold. It then directly follows that

(46) follows because ˜ X 2 and ˆ X ∗ 2 are functions of ( M 1 , M 2 , K ) and thus the MMSE ( X 2 -˜ X 2 ) is uncorrelated with ( ˆ X ∗ 2 -˜ X 2 ) ,

(48) follows from (40),

(49) follows because P ˆ X ∗ 1 ˆ X ∗ 2 = P X 1 X 2 .

Following similar steps for the third frame, we get D ∈ Φ D 0 ( P M | X K ) . This concludes the proof.

B.1 A Counterexample for Factor-Two Bound in Case of 0 -PLF-JD

Assume that we have only two frames, i.e., D 3 →∞ . Let M 1 be independent of X 1 and M 2 = X 2 . Then, we have ˜ X 1 = ∅ and ˜ X 2 = X 2 . Consider the achievable distortion region of Theorem 2. The distortion of the first step is given by the following

For the second frame, we have

where (51) follows because ˜ X 2 = X 2 and (52) follows because X 2 is independent of ˆ X ∗ 1 ( M 1 is independent of X 1 , then ˆ X ∗ 1 , which is a function of ( M 1 , K ) , would be independent of X 1 and hence independent of X 2 ).

Now, notice that the MMSE distortion of the second step is zero since ˜ X 2 = X 2 . However, the achievable distortion of the second step for the reconstruction satisfying 0 -PLF JD is given in (52) which clearly does not satisfy the factor-two bound.

where

C Fixed Encoders Operating at Low rate regime

We consider the class of noisy encoders where the encoder distribution can be written as follows

where µ is a sufficiently small constant and the distribution Q noisy ( · ) could be arbitrary conditional distribution with same marginal as P X j .

Theorem 3 For the class of encoders given by (53) , we have

Denote the reconstruction of the second step by ˆ X ∗ 2 and consider the expected distortion. From a similar justification starting from (24) and leading to (26), we can write the distortion as follows

❊ ❊ Now, we study the expected term [ ∥ ˜ X 2 -ˆ X ∗ 2 ∥ 2 ] as follows

In order to analyze the above expression, we first approximate the MMSE reconstruction ˜ X 2 as follows

where (58) follows from (53). Moreover, notice that (59) implies that

Next, consider the expected term in (56) as follows

(64)

where

(62) follows from (59);

(63) follows because the 0 -PLF-JD implies that P ˆ X ∗ 2 | ˆ X ∗ 1 = P X 2 | X 1 and ❊ [ X 2 ] is just a constant; · (64) follows from 0 -PLF-JD where P ˆ X ∗ 1 = P X 1 ;

(66) follows from (61).

Considering (55) and (66), we get

The proof for the third frame follows similar steps.

Px,1|X1

Xr.1

Px1Xr,1

Representation 1

Reconstruction 1

1

1

4

Pxr,2|Xr,1X1X2

Xr.2

Px2|Xr,2Xr,1X1

Representation 2

Reconstruction2

X1

一

一

1

一

←-

一

一

X3

Xr,3

X3

Xr,3|Xr,2Xr,1X1X2X3

x3|x2X1Xr,3Xr,2Xr,1

Representation 3

Reconstruction3

D Operational RDP Region

Recall the definition of iRDP region C RDP for first-order Markov sources (Definition 4) as follows. It is the set of all tuples ( R , D , P ) satisfying

for auxiliary random variables ( X r, 1 , X r, 2 , X r, 3 ) and ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) such that

for some deterministic functions η 1 ( . ) and η 2 ( ., . ) .

Theorem 4 For first-order Markov sources, a given ( D , P ) and R ∈ R ( D , P ) , we have

Moreover, the following holds:

Proof: Before stating the achievable scheme, we first discuss the strong functional representation lemma [35]. It states that for jointly distributed random variables X and Y , there exists a random variable U independent of X , and function ϕ such that Y = ϕ ( X,U ) . Here, U is not necessarily unique. The strong functional representation lemma states further that there exists a U which has information of Y in the sense that

Notice that the strong functional representation lemma can be applied conditionally. Given P XY | W , we can represent Y as a function of ( X,W,U ) such that U is independent of ( X,W ) and

Proof of (77) (Inner bound) :

For a given ( D , P ) and R ∈ R ( D , P ) , let X r = ( X r, 1 , X r, 2 , X r, 3 ) be jointly distributed with X = ( X 1 , X 2 , X 3 ) where the Markov chains (74)-(76) hold and the rate constraints in (68)-(70)

X

2

V1

V1

X1

M1 ∈ {0,1}*

文1

q1(.)

compress

decompress

Recons.

1Encoder 1

Decoder 1

K

一

1

√

V2

Xr,2

M2 E {0,1}*

Xr,2

X2

q2(.)

compress

decompress

Recons.

Encoder 2

Decoder 2

are satisfied such that there exist ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) for which distortion-perception constraints (71)-(72) hold. Denote the joint distribution of ( X , X r , ˆ X ) by P XX r ˆ X and notice that according to the Markov chains in (74)-(76), it factorizes as the following

✶ ✶ ✶ For an illustration of encoded representations X r and reconstructions ˆ X in R ( D , P ) which are induced by distribution P XX r ˆ X , see Fig. 5.

Now, we show that R +log( R +1) + 5 ∈ R ( D , P ) . The achievable scheme is as follows. Fix the joint distribution P X r according to (81) which constructs the codebook, given by

From the strong functional representation lemma [35], we know that

there exist a random variable V 1 independent of X 1 and a deterministic function q 1 such that X r, 1 = q 1 ( X 1 , V 1 ) and

which means that the first encoder observes the source X 1 and applies the function q 1 to get X r, 1 whose distribution needs to be preserved according to (82) (see Fig. 6);

according to the conditional strong functional representation lemma, there exist a random variable V 2 independent of ( X 2 , X r, 1 ) and a deterministic function q 2 such that X r, 2 = q 2 ( X r, 1 , X 2 , V 2 ) and

At the second step, the representation X r, 1 is available at the second encoder. So, upon observing the source X 2 , it applies the function q 2 to get X r, 2 whose conditional distribution given X r, 1 needs to be preserved according to (82) (see Fig. 6);

according to the conditional strong functional representation lemma, there exist a random variable V 3 independent of ( X 3 , X r, 1 , X r, 2 ) and a deterministic function q 3 such that X r, 3 = q 3 ( X r, 1 , X r, 2 , X 3 , V 3 ) and

Now, the encoding and decoding are as follows

With V 1 available at all encoders and decoders, we can have a class of prefix-free binary codes indexed by V 1 with the expected codeword length not larger than I ( X 1 ; X r, 1 ) + log( I ( X 1 ; X r, 1 ) + 1) + 5 to represent X r, 1 , losslessly (see Fig. 6).

With V 2 available at the encoders and decoders, we can design a set of prefix-free binary codes indexed by ( V 2 , X r, 1 ) with expected codeword length not larger than I ( X 2 ; X r, 2 | X r, 1 ) + log( I ( X 2 ; X r, 2 | X r, 1 ) + 1) + 5 to represent X r, 2 , losslessly(see Fig. 6).

Similarly, one can represent X r, 3 losslessly with V 3 available at the third encoder and decoder.

The decoders can use functions ˆ X 1 = η 1 ( X r, 1 ) , ˆ X 2 = η 2 ( X r, 1 , X r, 2 ) and ˆ X 3 = X r, 3 to get the reconstruction ˆ X .

This shows that R +log( R +1) + 5 ∈ R o ( D , P ) .

Proof of (78) (Outer Bound) :

For any ( D , P ) , R ∈ R o ( D , P ) , shared randomness K , encoding functions f j : X 1 × . . . ×X j ×K → M j and decoding functions g j : M 1 ×M 2 × . . . ×M j ×K → ˆ X j such that and

we lower bound the expected length of the messages. Define

and recall that according to the decoding functions, we have

Now, consider the following set of inequalities

Notice that the definitions in (89)-(90) imply the following Markov chains

On the other hand, the decoding functions of the first and second steps imply that

where together with definitions in (89) and (90), we can write

such that η 1 ( . ) and η 2 ( ., . ) are deterministic functions.

We can write

Similarly, we have

Now, consider the fact that the set of constraints in (87)-(88), (95), (98), (101) with Markov chains in (102)-(103) and deterministic functions in (106)-(107) constitute an iRDP region, denoted by ¯ C RDP , which is the set of all tuples ( R , D , P ) such that

for auxiliary random variables ( X r, 1 , X r, 2 ) and ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) satisfying the following

for some deterministic functions η 1 ( . ) and η 2 ( ., . ) .

Comparing the two regions ¯ C RDP and C RDP , we identify the following differences. The Markov chain in (74) is more restricted comparing to (115). Moreover, the Markov chain (75) does not exist in ¯ C RDP . The following lemma states that ¯ C RDP = C RDP . Now, for a given ( D , P ) , let ¯ R ( D , P ) denote the set of rate tuples R such ( R , D , P ) ∈ ¯ C RDP , then this lemma implies that ¯ R ( D , P ) = R ( D , P ) which completes the proof of the outer bound. Moreover, notice that the above proof only deals with the statistics of the representations and reconstructions and does not depend on the choice of the PLF. So, it holds for both PLF-FMD and PLF-JD. This concludes the proof.

We conclude this section by the following lemma.

Lemma 1 For first-order Markov sources, we have

Proof: This result for the scenario without perception constraint has been similarly observed in [36, Eq. (12)]. The proof in this section is provided for completeness.

First, notice that the set of Markov chains in (74)-(76) is more restricted than the ones in (114)-(115), hence C RDP ⊆ ¯ C RDP . Now, it remains to prove that ¯ C RDP ⊆ C RDP . Consider the following facts

The distortion constraints in (111) depend only on the joint distribution of ( X j , ˆ X j ) , and thus on the joint distribution of ( X j , X r, 1 , . . . , X r,j ) . So, imposing the Markov chain X r, 2 → ( X 2 , X r, 1 ) → X 1 does not affect the expected distortion ❊ [ ∥ X 2 -ˆ X 2 ∥ 2 ] since it does not depend on the joint distribution of X 1 with ( X r, 1 , X r, 2 , X 2 ) . A similar argument holds for other frames;

The perception constraints in (112) depend on the joint distributions P X 1 ...X j and P ˆ X 1 ,..., ˆ X j (hence on P X r, 1 ...X r,j ). Thus, imposing X r, 2 → ( X 2 , X r, 1 ) → X 1 does not affect ϕ 2 ( P X 1 X 2 , P ˆ X 1 ˆ X 2 ) since it does not depend on the joint distribution of X 1 with ( X r, 1 , X r, 2 , X 2 ) . A similar argument holds for other frames;

Moreover, the rate constraints in (109) and (110) would be further lower bounded by

Thus, the set of rate constraints is optimized by the set of Markov chains (74)-(76).

The mutual information terms I ( X 1 ; X r, 1 ) , I ( X 2 ; X r, 2 | X r, 1 ) and I ( X 3 ; ˆ X 3 | X r, 1 , X r, 2 ) depend on distributions P X 1 X r, 1 , P X r, 1 X r, 2 X 2 and P X 3 ˆ X 3 X r, 1 X r, 2 , respectively. So, these distributions should be preserved by the set of Markov chains. The first two distributions are preserved by the choice of (73)-(74). Now, since we have first-order Markov sources (see Definition 3), preserving the joint distributions of P X r, 1 X 1 and P X r, 1 X r, 2 X 2 is sufficient to preserve the distribution P X r, 1 X r, 2 X 3 . So, preserving the joint distribution of P ˆ X 3 X r, 1 X r, 2 is sufficient to keep I ( X 3 ; ˆ X 3 | X r, 1 , X r, 2 ) unchanged.

Considering the above four facts, without loss of optimality, one can impose the following Markov chains

This concludes the proof for the PLF-JD. For the PLF-FMD, notice that the only difference is the second fact stated above. But, this also holds since the perception constraints depend only on P X j and P ˆ X j (hence on P X r, 1 ...,X r,j ).

E Gauss-Markov Source Model

We first remark that the Wasserstein-2 distance can also be replaced by the KL-divergence in most of the following analysis. The common properties between these two measures are convexity and the fact that they both depend on only second-order statistics when restricted to Gaussian source model.

Theorem 5 For the Gauss-Markov source model, any tuple ( R , D , P ) ∈ C RDP can be attained by a jointly Gaussian distribution over ( X r, 1 , X r, 2 , X r, 3 ) and identity mappings for η j ( · ) in Definition 4.

Proof: First, notice that a proof for the setting without perception constraint is provided in [37]. The following proof is different from [37] in some steps and also involves the perception constraint.

For a given tuple ( R , D , P ) ∈ C RDP , let X ∗ r, 1 , X ∗ r, 2 , ˆ X ∗ 1 = η 1 ( X ∗ r, 1 ) , ˆ X ∗ 2 = η 2 ( X ∗ r, 1 , X ∗ r, 2 ) and ˆ X ∗ 3 be random variables satisfying (73)-(75). Let P ˆ X G 1 | X 1 , P ˆ X G 2 | ˆ X G 1 X 2 and P ˆ X G 3 | ˆ X G 1 ˆ X G 2 X 3 be jointly Gaussian distributions such that the following conditions are satisfied.

In general, the Gaussian random variables which satisfy the constraints in (122)-(124) can be written in the following format

for some real ν , ω 1 , ω 2 , τ 1 , τ 2 , τ 3 where ˆ X G 1 ∼ N (0 , σ 2 ˆ X G 1 ) , ˆ X G 2 ∼ N (0 , σ 2 ˆ X G 2 ) , Z 1 , Z 2 and Z 3 are Gaussian random variables with zero mean and variances α 2 1 , α 2 2 , α 2 3 , independent of ˆ X G 1 , ( ˆ X G 1 , X 2 ) and ( ˆ X G 1 , ˆ X G 2 , X 3 ) , respectively.

We explicitly derive the coefficients ν , ω 1 , ω 2 , τ 1 , τ 2 and τ 3 in the following. Multiplying both sides of (125) by ˆ X G 1 and taking an expectation, we get

According to (122), the above equation can be written as follows

❊ ❊ Multiplying both sides of (126) by the vector [ ˆ X G 1 X 2 ] and taking an expectation, we have

❊ Considering the fact that ❊ [ X 2 ˆ X G 1 ] = ρ 1 ❊ [ X 1 ˆ X G 1 ] and according to (123), the above equation can be written as follows

Similarly, multiplying both sides of (127) by the vector [ ˆ X G 1 ˆ X G 2 X 3 ] , taking an expectation and considering (124), we get

For the third step, the coefficients and noise variance of (127) are given as follows

Now, we look at the rate constraints.

Rate Constraints :

Consider the rate constraint of the first step as follows

(143) follows because ˆ X ∗ 1 is a function of X ∗ r, 1 ;

(146) follows because for a given covariance matrix in (122), the Gaussian distribution maximizes the differential entropy;

(147) follows because the MMSE is uncorrelated from the data and since the random variables are Gaussian, the MMSE would be independent of the data.

where

Next, consider the rate constraint of the second step as the following

where

(151) follows because ˆ X ∗ 1 and ˆ X ∗ 2 are deterministic functions of X ∗ r, 1 and ( X ∗ r, 1 , X ∗ r, 2 ) , respectively;

(152) follows because for a given covariance matrix in (123), the Gaussian distribution maximizes the differential entropy;

(154) follows from entropy power inequality (EPI) [38, pp. 22];

(155) follows from (142).

Similarly, consider the rate constraint of the third frame as the following,

Next, we look at the distortion constraint.

Distortion Constraint : The choices in (122)-(124) imply that

Finally, we look at the perception constraint

Perception Constraint :

Define the following distribution

Now, define P U G V G to be a Gaussian joint distribution with the following covariance matrix

Then, we have the following set of inequalities:

where

(168) follows from the definition in (165);

(169) follows from (166) which implies that ( U ∗ , V ∗ ) and ( U G , V G ) have the same secondorder statistics;

(172) follows because P V G = P ˆ X G 1 which is justified in the following. First, notice that both P V G and P ˆ X G 1 are Gaussian distributions. Denote the variance of V G by σ 2 V G and recall that the variance of ˆ X G 1 is denoted by σ 2 ˆ X G 1 . According to (166), σ 2 V G is equal to the variance of V ∗ . Also, from (165), we know that P V ∗ = P ˆ X ∗ 1 , hence the variances of V ∗ and ˆ X ∗ 1 are the same. On the other side, according to (122), we know that the variance of ˆ X ∗ 1 is equal to σ 2 ˆ X G 1 . Thus, we conclude that σ 2 ˆ X G 1 = σ 2 V G , which yields P V G = P ˆ X G 1 . A similar argument shows that P U G = P X 1 .

A similar argument holds for the perception constraint of the second and third steps for both PLFs.

Thus, we have proved the set of Gaussian auxiliary random variables ( ˆ X G 1 , ˆ X G 2 , ˆ X G 3 ) given in (125)(127) where the coefficients are chosen according to distortion-perception constraints, provides an outer bound to C RDP which is the set of all tuples ( R , D , P ) such that

Now, we need to show that the above RDP region is also an inner bound to C RDP . This is simply verified by the following choice. In iRDP region of (68)-(76), choose the following:

where ( ˆ X G 1 , ˆ X G 2 , ˆ X G 3 ) satisfy (125)-(127) with coefficients chosen according to distortion-perception constraints. The lower bounds on distortion and perception constraints in (177) and (178) are immediately achieved by this choice. Now, we will look at the rate constraints. The achievable rate constraint of the first step can be written as follows

which immediately coincides with (174). The achievable rate of the second step can be written as follows

(184) follows because EPI holds with 'equality' for jointly Gaussian distributions [38, pp. 22];

(185) follows from (175).

Thus, the bound in (185) coincides with (155). A similar argument holds for the achievable rate of the third frame.

Notice that the above proof (both converse and achievability) can be extended to T frames using the sequential analysis that was presented. Thus, without loss of optimality, one can restrict to the jointly Gaussian distributions and identity functions η 1 ( . ) and η 2 ( ., . ) in iRDP region C RDP .

For a given rate R , the following corollary provides the optimization programs which lead to the characterization of the DP tradeoff DP ( R ) for the Gauss-Markov source model.

Corollary 1 For a given rate tuple R and T = 2 frames, the optimal reconstructions of the DPtradeoff DP ( R ) can be written as follows

where Z 1 (resp Z 2 ) is a Gaussian random variable independent of X 1 (resp ( ˆ X G 1 , X 2 ) ) and ˆ X G j ∼ N (0 , ˆ σ 2 j ) for j = 1 , 2 , and ν, ω 1 , ω 2 , ˆ σ 2 1 , ˆ σ 2 2 are the solutions of the following optimization program for the first step,

and the following minimization problem for the second step and PLF-FMD,

or the following minimization problem for the second step and PLF-JD,

where tr ( . ) denotes the trace of a matrix and

where

Proof: We obtain the optimization programs for T = 2 frames as follows.

For a given rate tuple R , the DP-tradeoff DP ( R ) is given by the set of all tuples ( D , P ) such that there exists ˆ X G satisfying the following Markov chains

In general, the set of reconstructions that satisfy (193)-(194) can be written as follows

Plugging the above into (195) and (196) yields the following rate expressions

Re-arranging the terms in the above constraints yields the conditions in (188b) and (190b). Considering (197) with (199)-(200) gives the following expressions for distortions

❊ ❊ which are the objective functions in (188a) and (190a). Now, we evaluate the perception constraint. Notice that the covariance matrices of ( X 1 , X 2 ) and ( ˆ X G 1 , ˆ X G 2 ) are given by Σ 12 and ˆ Σ 12 defined in (191) and (192), respectively. The Wasserstein-2 distance between two Gaussian distributions with covariance matrices Σ 12 and ˆ Σ 12 is given in (190c) as discussed in [26, pp. 18].

Similarly, the expressions in (189) for the decoder based on PLF-FMD can be obtained.

F Gauss-Markov Source Model: Extremal Rates

In this section, we derive the achievable reconstructions for some special cases. We assume that we have only two frames, i.e., D 3 , P 3 →∞ . Moreover, let σ 2 1 = σ 2 2 := σ 2 for simplicity. In general, the reconstructions can be written as follows

where ˆ X G j ∼ N (0 , ˆ σ 2 j ) for j = 1 , 2 . Recall the optimization program of the first step in (188) as follows

and the following conditions, and

For a given ˆ σ 2 1 , the objective function in (207a) is a monotonically deacreasing function of ν , hence one can restrict ν to be nonnegative, without loss of optimality. So, the above optimization program can be written as

Optimizing with respect to ν in the above program, we have

where the optimization program reduces to

Next, recall the optimization program of the second step for PLF-FMD in (189) as follows

Plugging (209) into the above program, we get

(212b)

The optimization program for the second step of PLF-JD is similar to the above program (212) when (212c) is replaced by (190c). In this section, we study different rate regimes and obtain the solutions of the above optimization programs. In particular, we are interested in two perception thresholds P 2 →∞ and P 2 = 0 where the former corresponds to the classical rate-distortion region and the latter is the case of 0 -PLF. For the 0 -PLF-FMD, we have ˆ σ 1 = ˆ σ 2 = σ . For the 0 -PLF-JD, in addition to preserving the marginals, the correlation ❊ [ ˆ X G 1 ˆ X G 2 ] = ρ 1 σ 2 should be satisfied. For each of these cases, the optimization program in (212) is simplified in the following.

Optimization Program of the Second Step for P →∞ : In this case, there is no perception constraint in the setting and the optimization program in (212) reduces to the following

This case corresponds to the classical rate-distortion tradeoff where it is shown that for a given rate, the MMSE reconstructions are indeed optimal [28,37]. The expressions for MMSE reconstructions are given in Appendix H.1.

Optimization Program of the Second Step for 0 -PLF-FMD : In this case, we have ˆ σ 1 = ˆ σ 2 = σ . So, the optimization program in (212) reduces to the following

Here, ω 1 and ω 2 only need to satisfy the rate constraint given in (214b) which represents a larger search space than that of 0 -PLF-JD which will be discussed in the following.

Optimization Program of the Second Step for 0 -PLF-JD : In this case, in addition to preserving marginals ˆ σ 1 = ˆ σ 2 = σ , we need to satisfy the constraint ❊ [ ˆ X G 1 ˆ X G 2 ] = ρ 1 σ 2 . Thus, the optimization program of this case has an extra condition ω 1 + νω 2 ρ 1 = ρ 1 comparing to (214) and it is given as follows

Comparing (215) with (214), we notice that the search space of the optimization program for 0 -PLFJD is smaller than that of 0 -PLF-FMD. Thus, a larger distortion is expected for 0 -PLF-JD.

Before studying each case of extremal rates, we introduce another constraint in the optimization program of all above three cases of perception metrics. We restrict to nonnegative ω 1 ω 2 ρ 1 and get an upper bound on the programs (213), (214) and (215). So, in further discussion on these programs, the constraint ω 1 ω 2 ρ 1 ≥ 0 will be also considered.

In the low-rate regime, notice that we can approximate the rate term as follows

Plugging the above into (209), we have

Also, inserting (216) into the rate constraint of the second step (211c) yields the following

Re-arranging the terms in the above inequality yields the following

So, in all of the optimization programs of the case R 1 = R 2 = ϵ , the above constraint (220) will replace the rate constraint of the second step.

Now, we consider different cases based on the perception measure.

a) Without a perception constraint : In this case, using (216), the optimization program of the first step in (210) simplifies to the following

which gives us the following optimal solution

Plugging the above solution into (217) and (221), we get

and

Now, we look at the optimization program of the second step (213). For a given ω 1 and ω 2 , the objective function is an increasing function of ˆ σ 2 2 , so optimizing over ˆ σ 2 2 yields the following

Thus, the optimization program (213) is further upper bounded by the following

The optimal solution of the above minimization is given by the following

Thus, considering the dominant terms of (223), (227) and (228), we have

and Z j ∼ N (0 , 2 ϵσ 2 ln 2) for j = 1 , 2 . Notice that

b) 0 -PLF-FMD : In this case, we have ˆ σ 1 = ˆ σ 2 = σ . For the optimization program of the first step, (209) reduces to the following

and D 1 is given in the following which is derived by (210)

Now, we study the optimization program of the second step. The optimization program of (214) is further upper bounded by the following

Now, we further simplify the inequality (235b) in the following. Considering the fact that ω 1 ω 2 ρ 1 ≥ 0 , this inequality implies that

So, using the above inequalities, the RHS of (235b) can be upper bounded as follows

Now, according to (238), the optimization program in (235) is further upper bounded by the following

For a given ω 1 (resp ω 2 ), the objective function (239a) is a monotonically decreasing function of ω 2 (resp ω 1 ), so the optimal solution is attained on the boundary, i.e.,

Thus, the program (239) further simplifies to the following

The optimal solution of the above program is given by

Thus, considering dominant terms of (233), (242) and (243), we get

where Z 1 ∼ N (0 , (1 -2 ϵ ln 2) σ 2 ) and

Notice that

For the special case of ρ 1 = 1 , the expressions in (244) and (245) simplify as follows

Define Z FMD := 1 √ 2 Z 1 + Z 2 and notice that Z FMD ∼ N (0 , (1 -4 ϵ ln 2) σ 2 ) . Moreover, we have

c) 0 -PLF-JD : In this case, the optimization program of the first step is similar to the previous case. The optimization program of the second step is given in (215) where the condition ω 1 + νω 2 ρ 1 = ρ 1 is introduced. According to (233), ν = O ( √ ϵ ) which suggests the following form for ω 1 ,

which together with (240) yields

for some small δ ϵ that goes to zero as ϵ → 0 . The parameter δ ϵ will be determined later. Plugging ω 1 = ρ 1 -δ ϵ into (240), we find out that only the constant term of ω 1 contributes to a dominant term for ω 2 which yields the following

Now, applying the constraint ❊ [ ˆ X G 1 ˆ X G 2 ] = ρ 1 σ 2 , we get

However, notice that since δ ϵ = O ( ϵ ) , it does not contribute to dominant terms of distortion. So, we can simply represent ˆ X G 1 and ˆ X G 2 as follows

where Z 1 ∼ N (0 , (1 -2 ϵ ln 2) σ 2 ) and Z 2 ∼ N (0 , (1 -ρ 2 1 -(1 -ρ 2 1 +2 ρ 2 1 √ 1 -ρ 2 1 )2 ϵ ln 2) σ 2 ) . The following distortions are also achievable

For the special case of ρ = 1 , according to (259) and (261), we have ˆ X G 2 = ˆ X G 1 and D 2 = D 1 .

2) R 1 → ∞ , R 2 = ϵ for small ϵ : In this case, since R 1 → ∞ , we have ˆ X G 1 = X 1 , D 1 = 0 , and we only need to solve the optimization program of the second step. Also, we have the following approximation

We consider three different cases based on the perception constraint.

a) Without a perception constraint : In this case, consider the optimization program (213). For a given ω 1 and ω 2 , the objective function is an increasing function of ˆ σ 2 2 , hence optimizing over ˆ σ 2 2 , we get

The program in (213) is further upper bounded by the following

The solution of the above optimization program is given by the following

where Z 2 ∼ N (0 , (1 -ρ 2 1 ) σ 2 2 ϵ ln 2) . So, the reconstruction of the second frame closely resembles the first frame. The distortions of the first and second frames are zero and (1 -ρ 2 1 -(1 -ρ 2 1 )2 ϵ ln 2) σ 2 , respectively.

Thus, we have

Thus, we have

b) 0 -PLF-FMD : In this case, ˆ σ 1 = ˆ σ 2 = σ . Thus, the optimization program in (214) is further upper bounded by the following

For a given ω 1 (resp ω 2 ), the objective function (269a) is a monotonically decreasing function of ω 2 (resp ω 1 ). So, the optimal solution is attained on the boundary, i.e., (269b) is satisfied with equality given as follows

It can be easily verified that the first-order terms of ω 1 and ω 2 which optimize the program are 1 and 0 , respectively. So, we write ω 1 and ω 2 in the following form

for some real δ 1 and δ 2 . Plugging the above (271) and (272) into (270) and considering the dominant terms, we get

On the other side, we can write the objective function in (269) as follows

2

σ

2

2

2

Differentiating the above expression with respect to δ 2 and letting it be zero, we have:

Thus, we have

where Z 2 ∼ N (0 , ( 1 -ρ 2 1 ρ 2 1 )2 ϵ ln 2) . Again, the reconstruction of the second frame is almost similar to the first frame and the distortion is 2(1 -ρ 1 -( 1 -ρ 2 1 2 ρ 1 )2 ϵ ln 2) σ 2 .

c) 0 -PLF-JD : First consider the case where ρ 1 = 1 . The optimization program is given in (215) where the constraint ω 1 + νρ 1 ω 2 = ρ 1 is introduced. Notice that ω 1 can be written in the following form

̸

for some δ ϵ that goes to zero as ϵ → 0 . The parameter δ ϵ will be determined later. Plugging ω 1 = ρ 1 + δ ϵ into (270) yields the following

which is derived only through the first-order term of ω 1 which is ρ 1 . Now, considering the fact that ❊ [ ˆ X G 1 ˆ X G 2 ] = ρ 1 σ 2 , we obtain δ ϵ = -ρ 1 √ 2 ϵ ln 2 . (282) Thus, we have

where Z 2 ∼ N (0 , (1 -ρ 2 1 ) σ 2 ) . Here, the reconstruction of the second frame closely resembles the first frame. The distortion of the second frame is 2(1 -ρ 2 1 -(1 -ρ 2 1 ) √ 2 ϵ ln 2) σ 2 .

If ρ 1 = 1 , we simply have ˆ X G 2 = ˆ X G 1 = X 1 = X 2 which can be derived from (283)-(284) by letting X 1 = X 2 .

The analysis for the case of R 1 = ϵ and R 2 →∞ is similar and is omitted for brevity. The results of this section are summarized in Table 2.

G Comparison of PLFs in Low-Rate Regime

Theorem 6 For sufficiently small ϵ , let R j = ϵ and suppose that ρ j = ρ and σ j = σ , for j = 1 , . . . , T . The achievable distortions D FMD ,j (for 0 -PLF-FMD), and D JD ,j (for 0 -PLF-JD) are:

where ∆ FMD ,j := √ 1 + ρ 2 (2 ρ 2 ) j -1 -1 2 ρ 2 -1 and ∆ JD ,j := ρ 2( j -1) + ✶ { j ≥ 2 } · √ 1 -ρ 2 ( ∑ j -2 i =0 ρ 2 i ) . Proof: We extend the proof in the previous section for the low-rate regime to T frames.

Distortion Analysis for 0 -PLF-FMD :

We follow similar steps to (233)-(248) for optimization problems of the third and fourth frames and then use induction to derive expressions for T frames. For simplicity, we assume that ρ j = ρ for all j . Notice that in the following proof, ( ˆ X G 1 , ˆ X G 2 ) are as in (205)-(206) where ν , ω 1 and ω 2 are already derived in (233)-(248).

Now, consider the reconstruction of the third frame as follows

for some τ 1 , τ 2 , τ 3 , where ˆ X G 3 ∼ N (0 , σ 2 ) and Z 3 is a Gaussian random variable independent of ( ˆ X G 1 , ˆ X G 2 , X 3 ) . The rate constraint of the third step is given by

Evaluating the above constraint with the choice of random variables ( ˆ X G 1 , ˆ X G 2 , ˆ X G 3 ) and re-arranging the terms, we get

Similar to (240), considering the dominant terms of the above rate constraint and the fact that the solution of the optimization problem is attained when the above inequality is satisfied with 'equality', we get

The distortion can be written as follows

❊ [ ∥ X 3 -ˆ X G 3 ∥ 2 ] = 2 σ 2 -2 τ 3 σ 2 -2 τ 2 ω 2 ρσ 2 -2 τ 2 ω 1 νρ 2 σ 2 -2 τ 1 νρ 2 σ 2 . (290) So, the goal is to solve the following optimization problem for the third step

We restrict the search space to τ 1 , τ 2 , τ 3 ≥ 0 and get an upper bound to the above optimization program as follows

The above optimization problem is equivalent to the following

We proceed with solving the above optimization program. Taking the derivative of the objective function with respect to η 1 and η 2 yields the following:

Solving the above set of equations, we get

Thus, considering the dominant terms, we get the following reconstruction for the third frame

The above reconstruction yields the following distortion for the third frame

Finally, consider the reconstruction of the fourth frame as follows

where ˆ X G (0 , σ 2 )

The distortion can be written as follows

We take the derivative of the above expression with respect to λ 1 , λ 2 and λ 3 and we get

Solving the above set of equations yields the following

Thus, considering the dominant terms, we can write

The distortion term then becomes:

Now, we use induction to derive the terms for T frames. Define

Thus, we have

where Z j is a Gaussian random variable independent of ( ˆ X G 1 , . . . , ˆ X G j -1 , X j ) and its variance is such that ❊ [( ˆ X G j ) 2 ] = σ 2 . The distortion is given by the following expression

For the special case where ρ = 1 , then the distortion simplifies to the following which shows an exponential decrease at each step.

Distortion Analysis for 0 -PLF-JD :

In this case, the proof for T frames is similar to (254)-(261). Thus, we have

where Z j is a Gaussian random variable independent of ( ˆ X G j -1 , X j ) and its variance is such that ❊ [( ˆ X G T ) 2 ] = σ 2 . It should be mentioned that preserving the correlation coefficients, e.g., ❊ [ ˆ X G j ˆ X G j -1 ] = ρ , needs some correction terms of O ( ϵ ) as discussed in (257). However, as shown in (261), these correction terms do not contribute to dominant terms of distortion and hence, they can be ignored in the presentation of (319). Now, define and notice that

For the special case of ρ = 1 , we get ∆ JD ,j = 1 which remains a constant across different steps.

H Universality Statement for Gauss-Markov Source Model

H.1 MMSERepresentations for a Given Rate

For a given rate tuple R , the minimum distortions achievable by MMSE representations are derived in [28, 37] and are given by

The above distortions are achieved by the following optimal reconstructions ˆ X r given in [28]. Notice that the MMSE representation is X RD r = ˆ X r , i.e., the functions η 1 ( . ) and η 2 ( ., . ) of iRDP region C RDP (Definition 4) are identity functions (this statement follows from Theorem 5). Now, we choose the reconstruction ˆ X r in the following.

The reconstruction ˆ X r, 1 is chosen such that ˆ X r, 1 → X 1 → ( X 2 , X 3 ) holds a Markov chain and

where ˆ X r, 1 ∼ N (0 , σ 2 1 -D min 1 ) and Z 1 ∼ N (0 , D min 1 ) are independent random variables. Then, the reconstruction ˆ X r, 2 is chosen as follows. Let

where

which is the innovation from ˆ X r, 1 to X 2 . Now, we find the random variables ˆ W 2 and Z 2 such that

where ˆ W 2 ∼ N (0 , ρ 2 1 σ 2 2 σ 2 1 D min 1 + σ 2 N 1 -D min 2 ) and Z 2 ∼ N (0 , D min 2 ) are independent from each other, and the Markov chain ˆ W 2 → ( X 2 , ˆ X r, 1 ) → ( X 1 , X 3 ) holds. Now, define

Finally, we choose the reconstruction ˆ X r, 3 as follows. Let

which is the innovation from ˆ X r, 2 to X 3 . Now, we find random variables ˆ W 3 and Z 3 such that

where ˆ W 3 ∼ N (0 , ρ 2 2 σ 2 3 σ 2 2 D min 2 + σ 2 N 2 -D min 3 ) and Z 2 ∼ N (0 , D min 3 ) are independent from each other, and the Markov chain ˆ W 3 → ( X 3 , ˆ X r, 1 , ˆ X r, 2 ) → ( X 1 , X 2 ) holds. Now, define

Thus, the optimal reconstruction ˆ X r is chosen and it satisfies the rate constraint R .

H.2 Universality Statement

Theorem 7 For a given rate tuple R with strictly positive components, let the MMSE representation be denoted as X RD r = ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) . Let ( D , P ) ∈ DP ( R ) and let ˆ X = ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) be the corresponding reconstruction achieving it. Then there exist κ 1 , θ 1 , θ 2 , ψ 1 , ψ 2 and ψ 3 and noise variables ( Z 1 , Z 2 , Z 3 ) independent of ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) , which satisfy the following

For a given positive rate tuple R , let the MMSE representation X RD r be in the set P RD ( R ) . Also, let ( D , P ) ∈ DP ( R ) and X r , ˆ X be the corresponding representation and reconstruction achieving it.

Proof: First, notice that according to the proof of Theorem 5 for the Gauss-Markov source model, one can set ˆ X = X r in iRDP region of C RDP , without loss of optimality. So, in the following proof, the reconstruction X r and representation ˆ X are used interchangeably, in some places.

We show the following statement. If

then, there exist κ 1 , θ 1 , θ 2 , ψ 1 , ψ 2 and ψ 3 and noise variables Z 1 , Z 2 , Z 3 independent of X RD r, 1 , ( X RD r, 1 , X RD r, 2 ) , ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) , respectively, which satisfy the following

If (338)-(340) are satisfied with equality, then the noise random variables in (341)-(343) do not exist and a linear combination is sufficient for converting ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) to ( ˆ X 1 , ˆ X 2 , ˆ X 3 ) .

First, we prove the statement when all of inequalities in (338)-(340) hold with 'equality'. We provide the proof for T = 2 frames. The extension to arbitrary number of frames is straightforward. To that end, we first prove the following two lemmas.

Lemma 2 Without loss of optimality, the reconstruction of the first step ˆ X 1 satisfies the following

where and W 1 is a Gaussian random variable that its statistics do not depend on the pair ( D 1 , P 1 ) .

Proof: According to Theorem 5, we know that ( X 1 , ˆ X 1 ) are jointly Gaussian. So, we can write X 1 as follows

where T 1 is a Gaussian random variable independent of ˆ X 1 with a constant variance σ 2 1 2 -2 R 1 . Notice that (346) can be written as follows

where Q is a Gaussian random variable independent of X 1 with a zero-mean and variance σ 2 1 2 -2 R 1 1 -2 -2 R 1 and

From (347), we get

1 -1

Now, defining W := (1 2 -2 R 1 )( X + Q ) yields the desired result.

Lemma 3 Without loss of optimality, the reconstructions of the first and second steps ( ˆ X 1 , ˆ X 2 ) satisfy the following

❊ and W 2 is a Gaussian random variable that its statistics do not depend on the pairs ( D 1 , P 1 ) and ( D 2 , P 2 ) .

Proof: According to Theorem 5, we know that ( X 1 , X 2 , ˆ X 1 , ˆ X 2 ) are jointly Gaussian. So, we can write X 2 as follows

where T 2 is a Gaussian random variable independent of ( ˆ X 1 , ˆ X 2 ) with a constant variance of σ 2 X 2 | ˆ X 1 2 -2 R 2 where

Notice that (353) can be written as follows

where Q ′ is a Gaussian random variable independent of X 2 with a zero-mean and variance σ 2 X 2 | ˆ X 1 2 -2 R 2 1 -2 -2 R 2 . Defining W 2 := (1 -2 -2 R 2 )( X 2 + Q ′ ) yields the desired result.

where

Now, we proceed with the proof of the theorem. According to Lemma 2, there exist real γ 1 and γ ′ 1 such that

Then, according to Lemma 3, there exist λ 1 , λ 2 , λ ′ 1 and λ ′ 2 such that

The above equation can be written as

A similar justification holds for the third frame.

Next, we prove the statement when at least one of the rate constraints in (338)-(340) hold with strict inequality. In the following, we construct new reconstructions ( ˆ X ′ 1 , ˆ X ′ 2 ) based on ( ˆ X 1 , ˆ X 2 ) such that they satisfy the rate constraints ( R 1 , R 2 ) with equality. Then, we will be able to apply the two lemmas we proved to show that ( ˆ X 1 , ˆ X 2 ) are linearly related to MMSE reconstructions ( X RD r, 1 , X RD r, 2 ) .

Construction of ˆ X ′ 1 :

Now, let

Define

Construction of ˆ X ′ 2 :

Next, consider the second step. Define where ˆ R 2 ≤ R 2 . Also, recall that

Define ˜ X 2 := ❊ [ X 2 | X RD r, 1 , X RD r, 2 ] to be the MMSE reconstruction and consider that R 2 = I ( X 2 ; X RD X RD )

where ˆ R 1 ≤ R 1 . Also, recall that

Now, let ˆ X ′ 1 such that ˆ X ′ 1 → X RD r, 1 → X 1 holds and

where W 1 ∼ N (0 , ν 2 1 ) independent of ˆ X 1 and ν 2 1 will be determined in the following. Notice that I ( X 1 ; ˆ X ′ 1 ) is a monotonically decreasing function of ν 2 1 . So, one choose ν 2 1 such that

Now, according to Lemma 2, since ˆ X ′ 1 and ˆ X 1 have the same rates, there exists a coefficient κ ′ 1 such that

Now, define Z 1 := κ ′ 1 W 1 and notice that

where the last equality follows because both Markov chains X 2 → ( X RD r, 1 , X RD r, 2 ) → ˜ X 2 and X 2 → ˜ X 2 → ( X RD r, 1 , X RD r, 2 ) hold where the latter one is satisfied for Gaussian random variables for which we can write X 2 = [ X 2 | X r, 1 , X r, 2 ] + W ′ such that W ′ is independent of ( X RD r, 1 , X RD r, 2 ) .

❊ Now, we show that I ( X 2 ; ˜ X 2 | X RD r, 1 ) ≤ I ( X 2 ; ˜ X 2 | ˆ X ′ 1 ) . This is justified in the following

where (377) follows because W 1 is independent of ( X 2 , X RD r, 1 ) and (378) follows from the Markov chain X 2 → ˜ X 2 → X RD r, 1 .

Define

and consider the fact that R ′ 2 ≥ R 2 . Now, we introduce ˆ X ′ 2 such that ˆ X ′ 2 → ( ˜ X 2 , ˆ X ′ 1 ) → X 2 forms a Markov chain and

where W 2 ∼ N (0 , ν 2 2 ) independent of ( ˜ X 2 , ˆ X 1 ) and ν 2 2 will be determined in the following. Since I ( X 2 ; ˆ X ′ 2 | ˆ X ′ 1 ) is a monotonically decreasing function of ν 2 2 , we can choose ν 2 2 such that

Then, according to Lemma 3, there exist λ ′ 1 , λ ′ 2 , ˆ λ 1 and ˆ λ 2 such that

Plugging (363), (367) and (381) into the above expression and letting ˜ X 2 = αX RD r, 1 + βX RD r, 2 for some α, β , we get

Now define

Thus, we have

Notice that the above proof only uses the information about reconstructions of the operating points in DP-tradeoff and it does not depend on the choice of PLF. So, it holds for both PLF-JD and PLF-FMD. This concludes the proof.

H.3 Gaussian Example

Assume that the sources are symmetric in the sense that σ 2 1 = σ 2 2 = σ 2 3 = 1 , ρ 1 = ρ 2 = ρ 3 := ρ for some 0 < ρ ≤ 1 . Also, suppose that the perception thresholds are symmetric, i.e., P 1 = P 2 = P 3 := P for some 0 < P ≤ 1 . We choose the rate tuple R such that the minimum distortions D min j = D for j ∈ { 1 , 2 , 3 } . According to Appendix H.1, such rates are given by

The covariance matrix of the MMSE representations cov ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) is given by (1 -D )Σ where

If we introduce the 0 -PLF while keeping the rates as those of MMSE reconstructions, it can be shown that the optimal distortions are all equal to D 1 = D 2 = D 3 = 2 -2 √ 1 -D . Denote the reconstructions by ( ˆ X 0 D 1 , ˆ X 0 D 2 , ˆ X 0 D 3 ) and notice that the covariance matrix of the reconstructions is equal to that of the sources and is given by Σ . Thus, the covariance matrix of ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) is (1 -D ) times the covariance matrix of ( ˆ X 0 D 1 , ˆ X 0 D 2 , ˆ X 0 D 3 ) . So, the reconstructions ( X RD r, 1 , X RD r, 2 , X RD r, 3 ) and ( ˆ X 0 D 1 , ˆ X 0 D 2 , ˆ X 0 D 3 ) can be transformed to each other by the scaling factor 1 √ 1 -D . This inspires the idea that reconstructions corresponding to different tuples ( D , P ) are linearly related to those of MMSE representations which is the essence of the following Theorem 6. Moreover, both PLFs either based on FMD or JD perform similarly in this example since individually scaling the reconstruction of each frame finally ends up in matching the covariance matrix of all frames.

I Justification of low-rate regime for Moving MNSIT

In the MovingMNIST dataset, the digit in I-frame is generated uniformly across the 32 × 32 center region in a 64 × 64 image, meaning that log(32 × 32)=10 bits are required to localize the digits and any lower rate would result in much less correlated reconstructions. As such, one can consider R 1 =12 bits (2 extra bits for content and style) as a low rate. For P-frames, the movement is uniformly constrained within a 10 × 10 region so any rate R 2 ≤ log 2 (10 × 10)=6 . 6 bits (excluding residual compensation) can be considered a low rate.

J Experiment Details

J.1 Training Setup and Overview

Our compression architecture is built on the scale-space flow model [32], which allows end-to-end training without relying on pre-trained optical flow estimators. For better compression efficiency, we replace the residual compression module with the conditioning one [33]. In the following, we will interchangeably refer X 1 as the I-frame and subsequent ones as P-frames. The annotation for the encoder, decoder, and critic (discriminator) will be referred to as f, g , and h respectively and their specific functionality (e.g motion compression, joint perception critic) will be described within context through a subscript/superscript.

Distortion and Perception Measurement: We follow the setup in prior works [16,21] for distortion and perception measurement. Specifically, we use MSE loss ❊ [ || X -ˆ X || 2 ] as a distortion metric and Wasserstein-1 distance as a perception metric, which can be estimated through the WGAN critics (following the Kanotorovich-Rubinstein duality). For the marginal perception metric, we optimize

:Fixed Encoder

J:Optimized Encoder/Decoder

(X1, X1)

91

9i

m

gi

C

文

gi

m

gi

C

[X,1]

[xm]

X{-1]

[xs,]

[X"]

X[i-1]

[x,]

f1

f

X1

(Xi-i, Xi)

(x, Xi)

(Xi-i1, Xi)

(x, Xi)

(a)I-frame

(b) P-frame universal

(c) P-frame end-to-end

our critics h m to classify between original image X and synthetic ones ˆ X . This will then allow us to measure W 1 ( P X , P ˆ X ) since:

where F is a set of all bounded 1-Lipschitz functions. Similarly, the joint perception metric is realized through W 1 ( P X 1 ...X j , P ˆ X 1 ... ˆ X j ) by training a critic h j that classifies between synthetic and authentic sequences:

In practice, the set of 1-Lipschitz functions is limited by the neural network architecture. Also, although our analysis employs the Wasserstein-2 distance as a perception metric, it is worth noting that the ideal reconstructions (0-PLF) for this metric and the one used in our study should be identical.

I-frame Compressor: We compress I-frames in a similar fashion as previous works [16, 21]. Our encoder f I and decoder g I in Figure 7a contain a series of convolution operations and we control the rate R 1 by varying the dimension and quantization level in the bottleneck. The model utilizes common randomness through the dithered quantization operation. For a given rate R 1 , we vary the amount of DP tradeoff by controlling the hyper-parameter λ marginal i in the following minimization objective L 1 :

L 1 = ❊ [ || X 1 -ˆ X 1 || 2 ] + λ marginal i W 1 ( P X 1 , P ˆ X 1 ) (395) Following the results from Zhang et al. [16], we fix the encoder after optimizing the encoder-decoder pair for MSE representations. We then fix the encoder and train another decoder to obtain the optimal reconstruction with perfect perception, i.e, W 1 ( P X , P ˆ X ) ≈ 0 . We will leverage these universal representation results to compress P-frames (both end-to-end and universal).

P-frame Compressor: We describe the loss functions before explaining our architectures. Given previous reconstructions ˆ X [ i -1] := { ˆ X 1 , ˆ X 2 , ..., ˆ X i -1 } , one can adjust the distortion-joint perception tradeoff by controlling the hyper-parameter λ joint i in the following objective L i .

L joint i = ❊ [ || X i -ˆ X i || 2 ] + λ joint i W 1 ( P X [ i ] , P ˆ X [ i ] ) (396) Note that in order to achieve 0-PLF-JD, previous reconstructions ˆ X [ i -1] must also achieve 0-PLF-JD, since it is impossible to reconstruct such ˆ X i if the previous ˆ X [ i -1] are not temporally consistent 3 . For the FMD metric, we use the loss function in (395).

In the universal model in Figure 7b, the motion encoder f i m compresses and sends the quantized flow fields [ X m r,i ] between the MMSE reconstruction ˜ X i -1 and X i . Given [ X m r,i ] , the flow decoder and warping module g i m will transform ˜ X i -1 into ˜ X w i (predicted frame). We use f c i to compress the

residual information [ X c r,i ] between X i and ˜ X w i 4 , which will be decoded by g c i . We note that for MMSE representation, g c i only requires ˜ X w i as a conditional input while an additional conditioning input ˆ X [ i -1] is required when perceptual optimization is involved. Together, f i m , g i m , f c i , and g c i are optimized for MMSE reconstructions. To train for different DP tradeoffs, we fix f i m , g i m , f c i and adapt the new decoder ˆ g c i (conditioning on ˜ X w i , ˆ X [ i -1] ). We note that fixing g i m for universal representation is essential since [ X c r,i ] is dependent on the outputs ˜ X w i of g i m .

In the end-to-end model in Figure 7c, we use an MMSE representation to estimate the motion vector, as in the case of the universal model. The only difference is that the encoder f c i also uses previous ˆ X i and the encoders will be jointly trained with the decoders.

J.2 Networks Architecture

In this section, we describe the network architecture for universal and end-to-end P-frame compressor models. 5 . In the architecture layout, we denote BN2D and SN for the Batchnorm2D and Spectral Normalization layers. Convolutional and transposed convolutional layer are denoted as 'conv' and 'upconv' respectively, which is accompanied by number of filters, kernel size, stride, and padding.

Motion Encoder and Decoder. The universal and optimized end-to-end model shares the same architecture for the motion encoder and decoder. ( f i m and g i m respectively). We follow the original implementations [32] and present the convolutional architecture in Table 3. Different from the original implementation, however, we replace the last layer with dithered quantization layer (as in [16]) in our implementation. The output dimension of the motion encoder is denoted as d m .

Encoder

(a)

f

i m

(b) Decoder g i m

Residual Encoder and Decoder. The architecture of the conditional residual encoder is shown in Table 4a, where we stack multiple frames along their channel dimension as an input. As described previously, in the residual encoder, the universal model requires only X i , ˜ X w i while the end-to-end model will receive X i , ˜ X w i and ˆ X [ i -1] . We denote the output dimension of this residual encoder as d r . In the decoding part, the decoder will first condition all the previous reconstructions ˆ X [ i -1] by projecting them into an embedding vector of size 192 (conditioning module in Table 4b). Then we concatenate this vector with the output of f r i . The concatenated vector will be fed into the decoder (Table 4c) to produce the reconstruction ˆ X i .

FMD and JD Critics. For the video critics, our PLF-JD critic architecture is inspired by the work of Kwon and Park [40], where we concatenate frames sequentially along their channel dimensions. For both PLF-FMD and PLF-JD critics, we add spectral normalization layers for better convergence. Their architecture is shown in Table 5.

Rate and output dimension The rate R is computed by log 2 ( d enc × L ) , where L is the number of quantization levels and d enc = d r + d m . Table 6 provides configurations of the rate, d m , d r , and L in the experiment.

Training Details: We use a batch size of 64, RMSProp optimizer with a learning rate of 5 × 10 -5 , and train each model with 360 epochs, where the training set contains 60000 images. To accelerate

(c) Decoder

training, we pre-train each model for 60 epochs with the MSE objective only. Under WGAN-GP framework [30], we use the gradient penalty of 10 and update the encoders/decoders for every 5 iterations. The parameters λ controlling the tradeoff are in Table.7. Training takes 2 days per model on a single NVIDIA P100 GPU. For the MovingMNIST factor of two bound and permanence of error experiments, we repeat the training 3 times.

J.3 Permanence of Error on KTH Datasets

The KTH dataset is a widely-used benchmark dataset in computer vision research, consisting of video sequences of human actions performed in various scenarios. We show more examples supporting our argument for the permanence of error on this realistic dataset. We use 16 bits for each frame. In general, the 0-PLF-JD decoder consistently outputs correlated but incorrect reconstructions due to the error induced by the first reconstructions, i.e., the P-frames will follow the wrong direction induced from the I-frame reconstruction. Besides the moving direction, we also notice that the type of actions (i.e. walking, jogging, and running) is also affected. On the other hand, while losing some temporal cohesion, MMSE and 0-PLF FMD decoders manage to fix the movement error.

Finally, we computed LPIPS [41] in Table 8, which is known as a full-reference perceptual metric for images. We compare LPIPS for each reconstruction (MMSE, 0-PLF-FMD and 0-PLF-JD) at each timeframe. This result aligns with our results on distortion loss, as the MMSE and 0-PLF-FMD tend to correct the reconstruction (so the ground truth and reconstruction look more similar over time). On the other hand, due to error permanence, the 0-PLF-JD reconstructions become different from their source sequence, causing the score to go up.

(b) P-frame encoder, R 1 = ϵ (12 bits).

GT

MMSE

0-PLF

FMD

0-PLF

GT

MMSE

0-PLF

FMD

0-PLF

GT

MMSE

0-PLF

FMD

0-PLF

GT

MMSE

0-PLF

FMD

0-PLF

Y

文

1

J.4 RDP Tradeoff for 3 frames

We extend our experimental results for the RDP-tradeoff and the principal of universality to the case of GOP size 3. As mentioned in the main paper, while the universal model only requires MMSE representations, the optimal end-to-end model also requires the MMSE reconstructions from previous frames to provide best estimates for motion flow vectors. Practically, this is challenging for our employed architecture since only previous ˆ X 1 , ˆ X 2 are available. As a result, to compare the RDP tradeoff between universal and end-to-end model, we also provide the end-to-end model with the MMSE estimate from previous frames while noting that this is unfeasible in practice. Interestingly, we show in Figure 9 the RDP tradeoff curves for the third frame X 3 and its reconstruction ˆ X 3 ,

0

.

010

0

.

015

0

.

020

Distortion:

||

X

3

-

ˆ

X

3

||

2

2

1

2

3

4

5

6

PLF-JD:

W

1

(

PX

1

,X

2

,X

3

, P

ˆ

X

1

,

ˆ

X

2

,

ˆ

X

3

)

0

.

010

0

.

015

Distortion:

||

X

3

-

ˆ

X

3

||

2

2

1

2

3

4

5

PLF-FMD:

W

1

(

PX

3

, P

ˆ

X

3

)

3.17 bits universal

3.17 bits end-to-end

2 bits universal

2 bits end-to-end

1 bit universal

1 bit end-to-end

(a)

Case 1:

R

1

=

∞

,

R

2

=

R

3

=

{

1

,

2

,

3

.

17

}

bits

0

.

010

0

.

015

0

.

020

Distortion:

||

X

3

-

ˆ

X

3

||

2

2

2

4

6

PLF-JD:

W

1

(

PX

1

,X

2

,X

3

, P

ˆ

X

1

,

ˆ

X

2

,

ˆ

X

3

)

0

.

010

0

.

015

0

.

020

Distortion:

||

X

3

-

ˆ

X

3

||

2

2

2

4

6

PLF-FMD:

W

1

(

PX

3

, P

ˆ

X

3

)

12 bits universal

12 bits end-to-end

8 bits universal

8 bits end-to-end

4 bits universal

4 bits end-to-end

(b)

Case 2:

R

1

=12

,

R

2

=

R

3

=

{

4

,

8

,

12

}

bits

observing that the universal and optimized curves are still relatively close to each other. When ( R 1 , R 2 , R 3 )=( ∞ , ϵ, ϵ ) , we note that the distortion for X 3 is larger than X 2 since the allocated rate is not enough to correct the motion. Finally, for the case ( R 1 , R 2 , R 3 )=( ϵ, ϵ, ϵ ) , we note that the curves again converge as in the case of ( R 1 , R 2 )=( ϵ, ϵ ) due to the incorrect reconstruction in the I-frame.

J.5 Diversity and Correlation

When ( R 1 , R 2 )=( ∞ , ϵ ) , our theoretical analysis predicted that the decoder optimized for JD is capable of producing diverse reconstructions. On the other hand, an optimized decoder for FMD will tend to produce reconstructions that are highly correlated with the previous reconstruction ˆ X 1 6 . In our experiment, we also observe such behavior, summarized in Table 9 and show several examples for R 2 = 2 bits in Figure 10. We observe that reconstructions from the joint metric deviate more randomly from X 1 than the marginal reconstructions. The marginal reconstructions, on the other hand, stay much closer to their original reconstruction ˆ X 1 .

X1

X2

X2(0-PLF JD)

X2 (0-PLF FMD)

9

9

9

9

9

7

7

7

7

7

2

2

9

We measure the diversity in ˆ X 2 reconstruction using E[ Var ( ˆ X 2 | X 1 , X 2 )] and the correlation with ˆ X 1 by E[ sim ( ˆ X 2 , X 1 )] , where sim ( u, v ) is the cosine distance between u, v . Table 9a shows that as we increase the number of bits in R 2 , the diversity decreases as the decoder can reconstruct the frame more precisely. In Table 9b, we see that the joint metric keeps the correlation relatively constant, showing that it actually preserves the temporal consistency. On the other hand, as the rate becomes larger, 0-PLF-FMD reconstruction tends to be less correlated with the previous frame X 1 . Finally, we note that our architecture innately utilizes common randomness to produce diverse reconstructions and does not suffer from mode-collapse behavior in general conditional GAN settings [42].

X1

X1

X2

PLF-FMD:X2

PLF-JD:X2

务

新

X1

X2

PLF-FMD:X2

PLF-JD:X2

X1

X2

PLF-FMD:X2

PLF-JD:X2

X1

X2

PLF-FMD:X2

PLF-JD:X2

X1

X1

X2

PLF-FMD:X2

PLF-JD :X2

K Error Permanence on UVG Dataset

We demonstrate the phenomenon of error permanence in a large-scale scenario using the UVG dataset. Our P-frame compressor is trained on the Vimeo-90k dataset. As illustrated in Figure 11, when reconstructing I-frames, an inaccurate color tone is introduced, which persists when employing PLF-JD. However, PLF-FMD effectively rectifies this issue within P-frames. Numerical results are

in Table. 10. We note that for the I-frame, we use the pretrained model in [43] that targets high perceptual quality and is also trained on Vimeo-90k dataset.

L Limitations

This work studies the effects of different perception loss functions, namely the PLF-JD and PLF-FMD, on the performance of lossy causal video compression. Our theoretical analysis and experiment reveal the error permanence phenomenon and show the universality principle, suggesting that MMSE representation can be transformed into other points on the DP tradeoffs.

In practice, one might want to combine these two losses, for example, perfect framewise realism (0-PLF FMD) while retaining some degree of temporal cohesion (PLF-JD small), which is not considered in this work. Furthermore, analysis for other types of video compression schemes, such as with B-frame, and scaling the universality compression architecture to high-definition videos are also desired.

Peer reviews:
comments: SUMMARY:
This paper studies the choice of perception loss for learned video compression and summarizes some valuable conclusions. The first one is the pros and cons of PLF-JD and PLF-FMD. The second one is the universality of MMSE reconstructions.

STRENGTHS:
This paper conduct experiments and give some theoretical analysis of the perception loss functions for learned video compression. It is valuable when designing and training a video compression pipeline.

WEAKNESSES:
1.  The datasets used in the paper are too simple. It is questionable that the conclusions still work for the datasets that are closer to the real world. It is suggested to provide more experiments on the widely used datasets in learned video compression methods, such as UVG, MCL-JCV, and HEVC Class B datasets. Most papers conduct their experiments on these datasets. So we can see more performance comparisons and verify the proposed method is not a toy example.

2.  Considering this paper focus on perception loss, why not try some metrics about perceptual quality? For example, the experiments can also provide widely used perceptual metrics such as  LPIPS.

QUESTIONS:
In the experiments, I do not know how the proposed method precisely controls the bit rate to make the rates of different methods to be identical.

LIMITATIONS:
See the weakness part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper consider theoretical considerations of generative video compression.\n\nWhile rate-distortion-perception theory (Blau & Michaeli, 2019) explains the trade-off between realism and distortion for generaive compression, the causal processing typically introduced for video models adds constraints that need to be considered.\n\nThe authors consider two settings: FMD (frame marginal distributions) and JD (joint distribution). In the FMD case, for full realism an observer cannot tell apart reconstructed frames and original frames when looking at one frame at a time, but there can be synthetic detail that is not temporally consistent. In the JD case, for full realism the joint distribution of reconstructed videos matches the originals.\n\nThe main results of the paper are:\n\nTheorem 1: in the FMD setting, we can achieve full realism (“perfect perception”) by at most doubling the distortion of an MSE optimized system. This generalizes a result from (Blau & Michaeli, 2019 [16]) for the single frame scenario.\n\nTheorem 2: In the JD setting, the authors obtain a similar result but for a more constrained setting. They need the source at time j to be nearly independent of the encoder outputs upto and including time j. Interestingly, they demonstrate an example in the appendix where the factor-of-two bound is not achieable. \n\nThe example can be summarized as follows: if you have some loss of information in reconstruction 1, you need reconstruction 2 to be consistent with it even if you have perfectly encoded the original frame. Hence you can’t achieve a zero reconstruction error in frame 2 despite having losslessly encoded it, violating the factor of two bound.\n\nTheorems 3-6: Here the analysis is done for the more general rate-distortion-perception setting, where the perceptual quality/realism is sacrificed to improve distortion. The authors consider both the setting of first-order Markov sources and do further analysis in the gaussian special case.\n\nSTRENGTHS:\n- Relatively readable paper\n- Valuable insights into the generative video compression field\n- The theoretical results are validated in experiments on small datasets (Moving MNIST and KTH).\n\nWEAKNESSES:\n- Intuitively, why would we only consider frame level marginal distributions? Just like pixel-level marginal distributions don’t make sense, this should lead to massive temporal artefacts/flickering in the low rate regime. While e.g. [7] used only a frame-level discriminator, they use warping for temporal consistency.\n- I found the proof of Theorem 1 to be quite similar to the proof of Thm. 2 in Appendix B in (Blau & Michaeli, 2019). It seems to me that proof also works as-is in the setting where you condition on previously encoded frames, as there are no assumptions on how the MSE optimized reconstruction is obtained (so whether it is using information from previous frames or not doesn’t seem to matter in the derivation). I would appreciate if the author could clarify differences here. \n- Experiments are limited to toy data.\n\n\nMinor (not affecting rating): *The supplement is very long* (30+ pages) and the main paper contains 24 references to it. It seems to me the authors have a lot more to say than fits into 9 pages. I'm not sure what to do with this but it feels slightly weird. I did not read everything in the supplement.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nToy data / causal setting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper examines the choice of perceptual loss for causal, low-latency video compression model with distortion-perception optimization. By using information theoretic analysis and deep-learning based experiments, the authors demonstrate that the choice of perceptual loss can have a significant effect on the decoded reconstruction, especially at low-bit rates.

STRENGTHS:
This paper demonstrates the proposed assertions through theoretical derivations and detailed experiments, which are convincing enough. In addition, these conclusions provide a good guideline basis for subjective optimization of learned video compression methods.

WEAKNESSES:
Since this paper is oriented to the discussion of learned video compression, the experimental setup may be too simple to have generalized conclusions. For example, the used sequences (e.g., MNIST and KTH) for evaluation are too homogeneous to present the characteristics of videos. It should be better to have some comparisons on more complex sequences such as the UVG dataset. It is critical to illustrate whether these findings would be useful in practical applications.

QUESTIONS:
As mentioned above, it is suggested to have some experiments on sequences with more complex motion and coded bits to examine the proposed findings in addition to those toy examples in the submission.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper systematically studies the rated-distortion-perception tradeoff in neural video compression. Since videos are made of consecutive frames, this paper considers two different perceptual metrics, including the joint distribution of all video frames (PLF-JD), and per-frame perceptual loss (framewise marginal distribution, PLF-FDM). Some conclusions are proposed regarding the choice of PLF-JD or PLF-FDM. \n\nIn addition, motivated by the universal encoded representations in previous RDP papers from the field of image compression, the authors in this paper demonstrate the video-version universal encoded representations hold when the perceptual constraint is PLF-FDM. While a similar result does not hold for PLF-JD in general, it is satisfied for a special class of encoders which operate in the low-rate regime. The universal encoded representations in video compression have several advantages for video compression such as when it needs to extract motion information from an MSE-based reconstruction.\n\nSTRENGTHS:\nOriginality and quality of this paper are pretty good. The PLF-JD and PLF-FDM are two different but both reasonable perceptual metrics used for optimization of video compression models. This is the first paper that studies the RDP issue in video compression and the results and conclusions are insightful. Some important conclusions regarding RDP include:\n\n(1) There is a significant penalty in distortion when using PLF-JD in the low-rate regime.\n\n(2) While PLF-JD preserves better temporal consistency across video frames, it suffers from the permanence of error phenomenon in which the mistakes in reconstructions propagate to future frames. \n\nThe universal encoded representations for video compression are studied as well, which are demonstrates to be always held when using PLF-FDM (similar to compressing images sequentially). \n\nBesides, the effectiveness of RDP in video compression models is well verified by theoretical analysis in Gaussian-Markov sources.\n\nWEAKNESSES:\n(1)\t(Perhaps not as a weakness) I am wondering whether these conclusions of RDP in video compression can be extended to distortion-perception tradeoff in other video generation / restoration tasks. The joint-frame loss PLF-JD and the per-frame loss PLF-FDM should hold similar conclusions for video generation.\n\n(2)\tThe state-of-the-art neural video compression framework contains a motion encoder/decoder and a residual encoder/decoder, which are effective for high-resolution videos. It seems this paper studies the other neural video compression framework that does not explicitly transmit motion information. How to apply the proposed theory to such motion-residual-separated video compression framework?\n\nQUESTIONS:\nSee the abovementioned “weaknesses”.\n\nLIMITATIONS:\nLimitations are discussed in the final section of Appendix. This paper is in the field of video compression which does not involve strong negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}