SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
W _ {2} = \hat {W} _ {2} + Z _ {2}, \tag {333}
$$
where  $\hat{W}_2\sim \mathcal{N}(0,\rho_1^2\frac{\sigma_2^2}{\sigma_1^2} D_1^{\mathrm{min}} + \sigma_{N_1}^2 -D_2^{\mathrm{min}})$  and  $Z_{2}\sim \mathcal{N}(0,D_{2}^{\mathrm{min}})$  are independent from each other, and the Markov chain  $\hat{W}_2\to (X_2,\hat{X}_{r,1})\to (X_1,X_3)$  holds. Now, define
$$
\hat {X} _ {r, 2} := \rho_ {1} \frac {\sigma_ {2}}{\sigma_ {1}} \hat {X} _ {r, 1} + \hat {W} _ {2}. \tag {334}
$$
Finally, we choose the reconstruction  $\hat{X}_{r,3}$  as follows. Let
$$
W _ {3} := \rho_ {2} \frac {\sigma_ {3}}{\sigma_ {2}} Z _ {2} + N _ {2}, \tag {335}
$$
which is the innovation from  $\hat{X}_{r,2}$  to  $X_{3}$ . Now, we find random variables  $\hat{W}_3$  and  $Z_{3}$  such that
$$
W _ {3} = \hat {W} _ {3} + Z _ {3}, \tag {336}
$$
where  $\hat{W}_3\sim \mathcal{N}(0,\rho_2^2\frac{\sigma_3^2}{\sigma_2^2} D_2^{\mathrm{min}} + \sigma_{N_2}^2 -D_3^{\mathrm{min}})$  and  $Z_{2}\sim \mathcal{N}(0,D_{3}^{\mathrm{min}})$  are independent from each other, and the Markov chain  $\hat{W}_3\rightarrow (X_3,\hat{X}_{r,1},\hat{X}_{r,2})\rightarrow (X_1,X_2)$  holds. Now, define
$$
\hat {X} _ {r, 3} := \rho_ {1} \frac {\sigma_ {3}}{\sigma_ {2}} \hat {X} _ {r, 2} + \hat {W} _ {3}. \tag {337}
$$
Thus, the optimal reconstruction  $\hat{\mathbf{X}}_r$  is chosen and it satisfies the rate constraint R.
H.2 Universality Statement
Theorem 7 For a given rate tuple  $\mathsf{R}$  with strictly positive components, let the MMSE representation be denoted as  $\mathsf{X}_r^{RD} = (X_{r,1}^{RD},X_{r,2}^{RD},X_{r,3}^{RD})$ . Let  $(\mathsf{D},\mathsf{P})\in \mathcal{DP}(\mathsf{R})$  and let  $\hat{\mathsf{X}} = (\hat{X}_1,\hat{X}_2,\hat{X}_3)$  be the corresponding reconstruction achieving it. Then there exist  $\kappa_{1}$ ,  $\theta_{1}$ ,  $\theta_{2}$ ,  $\psi_{1}$ ,  $\psi_{2}$  and  $\psi_{3}$  and noise variables  $(Z_{1},Z_{2},Z_{3})$  independent of  $(X_{r,1}^{RD},X_{r,2}^{RD},X_{r,3}^{RD})$ , which satisfy the following
For a given positive rate tuple  $\mathsf{R}$ , let the MMSE representation  $\mathsf{X}_r^{RD}$  be in the set  $\mathcal{P}^{RD}(\mathsf{R})$ . Also, let  $(\mathsf{D},\mathsf{P})\in \mathcal{DP}(\mathsf{R})$  and  $\mathsf{X}_r$ ,  $\hat{\mathbf{X}}$  be the corresponding representation and reconstruction achieving it.
Proof: First, notice that according to the proof of Theorem 5 for the Gauss-Markov source model, one can set  $\hat{\mathsf{X}} = \mathsf{X}_r$  in iRDP region of  $\mathcal{C}_{\mathsf{RDP}}$ , without loss of optimality. So, in the following proof, the reconstruction  $\mathsf{X}_r$  and representation  $\hat{\mathsf{X}}$  are used interchangeably, in some places.
We show the following statement. If
$$
R _ {1} \geq I \left(X _ {1}; X _ {r, 1}\right), \tag {338}
$$
$$
R _ {2} \geq I \left(X _ {2}; X _ {r, 2} \mid X _ {r, 1}\right), \tag {339}
$$
$$
R _ {3} \geq I \left(X _ {3}; X _ {r, 3} \mid X _ {r, 1}, X _ {r, 2}\right), \tag {340}
$$
then, there exist  $\kappa_{1}$ ,  $\theta_{1}$ ,  $\theta_{2}$ ,  $\psi_{1}$ ,  $\psi_{2}$  and  $\psi_{3}$  and noise variables  $Z_{1}$ ,  $Z_{2}$ ,  $Z_{3}$  independent of  $X_{r,1}^{\mathrm{RD}}$ ,  $(X_{r,1}^{\mathrm{RD}}, X_{r,2}^{\mathrm{RD}})$ ,  $(X_{r,1}^{\mathrm{RD}}, X_{r,2}^{\mathrm{RD}}, X_{r,3}^{\mathrm{RD}})$ , respectively, which satisfy the following
$$
\hat {X} _ {1} = \kappa_ {1} X _ {r, 1} ^ {\mathrm {R D}} + Z _ {1}, \tag {341}
$$
$$
\hat {X} _ {2} = \theta_ {1} X _ {r, 1} ^ {\mathrm {R D}} + \theta_ {2} X _ {r, 2} ^ {\mathrm {R D}} + Z _ {2}, \tag {342}
$$
$$
\hat {X} _ {3} = \psi_ {1} X _ {r, 1} ^ {\mathrm {R D}} + \psi_ {2} X _ {r, 2} ^ {\mathrm {R D}} + \psi_ {3} \hat {X} _ {r, 3} ^ {\mathrm {R D}} + Z _ {3}. \tag {343}
$$
If (338)-(340) are satisfied with equality, then the noise random variables in (341)-(343) do not exist and a linear combination is sufficient for converting  $(X_{r,1}^{\mathrm{RD}}, X_{r,2}^{\mathrm{RD}}, X_{r,3}^{\mathrm{RD}})$  to  $(\hat{X}_1, \hat{X}_2, \hat{X}_3)$ .
First, we prove the statement when all of inequalities in (338)-(340) hold with "equality". We provide the proof for  $T = 2$  frames. The extension to arbitrary number of frames is straightforward. To that end, we first prove the following two lemmas.
Lemma 2 Without loss of optimality, the reconstruction of the first step  $\hat{X}_1$  satisfies the following
$$
\gamma_ {1} \hat {X} _ {1} = W _ {1}, \tag {344}
$$
$$
\gamma_ {1} := \frac {\mathbb {E} \left[ X _ {1} \hat {X} _ {1} \right]}{\sigma_ {\hat {X} _ {1}} ^ {2}}, \tag {345}
$$
and  $W_{1}$  is a Gaussian random variable that its statistics do not depend on the pair  $(D_{1}, P_{1})$ .
Proof: According to Theorem 5, we know that  $(X_{1},\hat{X}_{1})$  are jointly Gaussian. So, we can write  $X_{1}$  as follows
$$
X _ {1} = \gamma_ {1} \hat {X} _ {1} + T _ {1}, \tag {346}
$$
where  $T_{1}$  is a Gaussian random variable independent of  $\hat{X}_{1}$  with a constant variance  $\sigma_1^2 2^{-2R_1}$ . Notice that (346) can be written as follows
$$
\hat {X} _ {1} = \alpha_ {1} (X _ {1} + Q), \tag {347}
$$
where  $Q$  is a Gaussian random variable independent of  $X_{1}$  with a zero-mean and variance  $\frac{\sigma_1^2 2^{-2R_1}}{1 - 2^{-2R_1}}$  and
$$
\alpha_ {1} := \frac {1}{\gamma_ {1}} \left(1 - 2 ^ {- 2 R _ {1}}\right). \tag {348}
$$
From (347), we get
$$
\gamma_ {1} \hat {X} _ {1} = (1 - 2 ^ {- 2 R _ {1}}) (X _ {1} + Q). \tag {349}
$$
Now, defining  $W_{1} \coloneqq (1 - 2^{-2R_{1}})(X_{1} + Q)$  yields the desired result.
Lemma 3 Without loss of optimality, the reconstructions of the first and second steps  $(\hat{X}_1,\hat{X}_2)$  satisfy the following
$$
\lambda_ {1} \hat {X} _ {1} + \lambda_ {2} \hat {X} _ {2} = W _ {2}, \tag {350}
$$
$$
\lambda_ {1} := \frac {\rho_ {1} \mathbb {E} \left[ X _ {1} \hat {X} _ {1} \right] \hat {\sigma} _ {X _ {2}} ^ {2} - \mathbb {E} \left[ \hat {X} _ {1} \hat {X} _ {2} \right] \mathbb {E} \left[ X _ {2} \hat {X} _ {2} \right]}{\hat {\sigma} _ {X _ {1}} ^ {2} \hat {\sigma} _ {X _ {2}} ^ {2} - \mathbb {E} ^ {2} \left[ \hat {X} _ {1} \hat {X} _ {2} \right]}, \tag {351}
$$
$$
\lambda_ {2} := \frac {\rho_ {1} \mathbb {E} \left[ X _ {1} \hat {X} _ {1} \right] \mathbb {E} \left[ \hat {X} _ {1} \hat {X} _ {2} \right] - \hat {\sigma} _ {X _ {1}} ^ {2} \mathbb {E} \left[ X _ {2} \hat {X} _ {2} \right]}{\hat {\sigma} _ {X _ {1}} ^ {2} \hat {\sigma} _ {X _ {2}} ^ {2} - \mathbb {E} ^ {2} \left[ \hat {X} _ {1} \hat {X} _ {2} \right]}, \tag {352}
$$
and  $W_{2}$  is a Gaussian random variable that its statistics do not depend on the pairs  $(D_{1}, P_{1})$  and  $(D_{2}, P_{2})$ .
Proof: According to Theorem 5, we know that  $(X_{1}, X_{2}, \hat{X}_{1}, \hat{X}_{2})$  are jointly Gaussian. So, we can write  $X_{2}$  as follows
$$
X _ {2} = \lambda_ {1} \hat {X} _ {1} + \lambda_ {2} \hat {X} _ {2} + T _ {2}, \tag {353}
$$
where  $T_{2}$  is a Gaussian random variable independent of  $(\hat{X}_1,\hat{X}_2)$  with a constant variance of  $\sigma_{X_2|\hat{X}_1}^2 2^{-2R_2}$  where
$$
\sigma_ {X _ {2} \mid \hat {X} _ {1}} ^ {2} := \frac {1}{2} \log \left(\rho_ {1} ^ {2} \sigma_ {1} ^ {2} 2 ^ {- 2 R _ {1}} + 2 ^ {2 H \left(N _ {1}\right)}\right). \tag {354}
$$
Notice that (353) can be written as follows
$$
\lambda_ {1} \hat {X} _ {1} + \lambda_ {2} \hat {X} _ {2} = (1 - 2 ^ {- 2 R _ {2}}) \left(X _ {2} + Q ^ {\prime}\right), \tag {355}
$$
where  $Q'$  is a Gaussian random variable independent of  $X_{2}$  with a zero-mean and variance  $\frac{\sigma_{X_2|\hat{X}_1}^2 2^{-2R_2}}{1 - 2^{-2R_2}}$ . Defining  $W_{2} \coloneqq (1 - 2^{-2R_{2}})(X_{2} + Q')$  yields the desired result.
Now, we proceed with the proof of the theorem. According to Lemma 2, there exist real  $\gamma_{1}$  and  $\gamma_1^\prime$  such that
$$
\gamma_ {1} \hat {X} _ {1} = \gamma_ {1} ^ {\prime} X _ {r, 1} ^ {\mathrm {R D}}. \tag {356}
$$
$$
\kappa_ {1} := \frac {\gamma_ {1} ^ {\prime}}{\gamma_ {1}}. \tag {357}
$$
Then, according to Lemma 3, there exist  $\lambda_1, \lambda_2, \lambda_1'$  and  $\lambda_2'$  such that
$$
\lambda_ {1} \hat {X} _ {1} + \lambda_ {2} \hat {X} _ {2} = \lambda_ {1} ^ {\prime} X _ {r, 1} ^ {\mathrm {R D}} + \lambda_ {2} ^ {\prime} X _ {r, 2} ^ {\mathrm {R D}}. \tag {358}
$$
The above equation can be written as
$$
\hat {X} _ {2} = \frac {\lambda_ {1} ^ {\prime} - \lambda_ {1} \kappa_ {1}}{\lambda_ {2}} X _ {r, 1} ^ {\mathrm {R D}} + \frac {\lambda_ {2} ^ {\prime}}{\lambda_ {2}} X _ {r, 2} ^ {\mathrm {R D}} \tag {359}
$$
$$
:= \theta_ {1} X _ {r, 1} ^ {\mathrm {R D}} + \theta_ {2} X _ {r, 2} ^ {\mathrm {R D}}. \tag {360}
$$
A similar justification holds for the third frame.
Next, we prove the statement when at least one of the rate constraints in (338)-(340) hold with strict inequality. In the following, we construct new reconstructions  $(\hat{X}_1', \hat{X}_2')$  based on  $(\hat{X}_1, \hat{X}_2)$  such that they satisfy the rate constraints  $(R_1, R_2)$  with equality. Then, we will be able to apply the two lemmas we proved to show that  $(\hat{X}_1, \hat{X}_2)$  are linearly related to MMSE reconstructions  $(X_{r,1}^{\mathrm{RD}}, X_{r,2}^{\mathrm{RD}})$ .
Construction of  $\hat{X}_1'$ :
$$
\hat {R} _ {1} := I \left(X _ {1}; \hat {X} _ {1}\right), \tag {361}
$$
where  $\hat{R}_1\leq R_1$  . Also, recall that
$$
R _ {1} = I \left(X _ {1}; X _ {r, 1} ^ {\mathrm {R D}}\right). \tag {362}
$$
Now, let  $\hat{X}_1^{\prime}$  such that  $\hat{X}_1^\prime \to X_{r,1}^{\mathrm{RD}}\to X_1$  holds and
$$
\hat {X} _ {1} ^ {\prime} = X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}, \tag {363}
$$
where  $W_{1} \sim \mathcal{N}(0, \nu_{1}^{2})$  independent of  $\hat{X}_{1}$  and  $\nu_{1}^{2}$  will be determined in the following. Notice that  $I(X_{1}; \hat{X}_{1}')$  is a monotonically decreasing function of  $\nu_{1}^{2}$ . So, one choose  $\nu_{1}^{2}$  such that
$$
I \left(\hat {X} _ {1} ^ {\prime}; X _ {1}\right) = I \left(X _ {1}; \hat {X} _ {1}\right) = \hat {R} _ {1}. \tag {364}
$$
Now, according to Lemma 2, since  $\hat{X}_1'$  and  $\hat{X}_1$  have the same rates, there exists a coefficient  $\kappa_1'$  such that
$$
\begin{array}{l} \hat {X} _ {1} = \kappa_ {1} ^ {\prime} \hat {X} _ {1} ^ {\prime} (365) \\ = \kappa_ {1} ^ {\prime} X _ {r, 1} ^ {\mathrm {R D}} + \kappa_ {1} ^ {\prime} W _ {1}. (366) \\ \end{array}
$$
Now, define  $Z_{1} \coloneqq \kappa_{1}^{\prime}W_{1}$  and notice that
$$
\hat {X} _ {1} = \kappa_ {1} ^ {\prime} X _ {r, 1} ^ {\mathrm {R D}} + Z _ {1}. \tag {367}
$$
Construction of  $\hat{X}_2'$ :
Next, consider the second step. Define
$$
\hat {R} _ {2} := I \left(X _ {2}; \hat {X} _ {2} \mid \hat {X} _ {1}\right), \tag {368}
$$
where  $\hat{R}_2\leq R_2$  . Also, recall that
$$
R _ {2} = I \left(X _ {2}; X _ {r, 2} ^ {\mathrm {R D}} \mid X _ {r, 1} ^ {\mathrm {R D}}\right). \tag {369}
$$
Define  $\tilde{X}_2\coloneqq \mathbb{E}[X_2|X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}}]$  to be the MMSE reconstruction and consider that
$$
\begin{array}{l} R _ {2} = I \left(X _ {2}; X _ {r, 2} ^ {\mathrm {R D}} \mid X _ {r, 1} ^ {\mathrm {R D}}\right) (370) \\ = I \left(X _ {2}; \tilde {X} _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}\right), (371) \\ \end{array}
$$
where the last equality follows because both Markov chains  $X_{2}\rightarrow (X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}})\to \tilde{X}_{2}$  and  $X_{2}\rightarrow$ $\tilde{X}_2\rightarrow (X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}})$  hold where the latter one is satisfied for Gaussian random variables for which we can write  $X_{2} = \mathbb{E}[X_{2}|X_{r,1},X_{r,2}] + W^{\prime}$  such that  $W^{\prime}$  is independent of  $(X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}})$ .
Now, we show that  $I(X_{2};\tilde{X}_{2}|X_{r,1}^{\mathrm{RD}})\leq I(X_{2};\tilde{X}_{2}|\hat{X}_{1}^{\prime})$ . This is justified in the following
$$
\begin{array}{l} I \left(X _ {2}; \tilde {X} _ {2} \mid \hat {X} _ {1} ^ {\prime}\right) = I \left(X _ {2}; \tilde {X} _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}\right) (372) \\ = H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}, X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}\right) (373) \\ \geq H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}, W _ {1}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}, X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}\right) (374) \\ = H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}, W _ {1}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}, X _ {r, 1} ^ {\mathrm {R D}} + W _ {1}\right) (375) \\ \geq H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}, W _ {1}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}\right) (376) \\ = H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}\right) (377) \\ = H \left(X _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}\right) - H \left(X _ {2} \mid \tilde {X} _ {2}, X _ {r, 1} ^ {\mathrm {R D}}\right) (378) \\ = I \left(X _ {2}; \tilde {X} _ {2} \mid X _ {r, 1} ^ {\mathrm {R D}}\right), (379) \\ \end{array}
$$
where (377) follows because  $W_{1}$  is independent of  $(X_{2}, X_{r,1}^{\mathrm{RD}})$  and (378) follows from the Markov chain  $X_{2} \rightarrow \tilde{X}_{2} \rightarrow X_{r,1}^{\mathrm{RD}}$ .
$$
R _ {2} ^ {\prime} := I \left(X _ {2}; \tilde {X} _ {2} \mid \hat {X} _ {1} ^ {\prime}\right), \tag {380}
$$
and consider the fact that  $R_2' \geq R_2$ . Now, we introduce  $\hat{X}_2'$  such that  $\hat{X}_2' \to (\tilde{X}_2, \hat{X}_1') \to X_2$  forms a Markov chain and
$$
\hat {X} _ {2} ^ {\prime} = \tilde {X} _ {2} + \hat {X} _ {1} ^ {\prime} + W _ {2}, \tag {381}
$$
where  $W_{2} \sim \mathcal{N}(0, \nu_{2}^{2})$  independent of  $(\tilde{X}_{2}, \tilde{X}_{1})$  and  $\nu_{2}^{2}$  will be determined in the following. Since  $I(X_{2}; \tilde{X}_{2}^{\prime} | \tilde{X}_{1}^{\prime})$  is a monotonically decreasing function of  $\nu_{2}^{2}$ , we can choose  $\nu_{2}^{2}$  such that
$$
I \left(X _ {2}; \hat {X} _ {2} ^ {\prime} \mid \hat {X} _ {1} ^ {\prime}\right) = I \left(X _ {2}; \hat {X} _ {2} \mid \hat {X} _ {1}\right) = \hat {R} _ {2}. \tag {382}
$$
Then, according to Lemma 3, there exist  $\lambda_1^\prime ,\lambda_2^\prime ,\hat{\lambda}_1$  and  $\hat{\lambda}_{2}$  such that
$$
\lambda_ {1} ^ {\prime} \hat {X} _ {1} ^ {\prime} + \lambda_ {2} ^ {\prime} \hat {X} _ {2} ^ {\prime} = \hat {\lambda} _ {1} \hat {X} _ {1} + \hat {\lambda} _ {2} \hat {X} _ {2}. \tag {383}
$$
Plugging (363), (367) and (381) into the above expression and letting  $\tilde{X}_2 = \alpha X_{r,1}^{\mathrm{RD}} + \beta X_{r,2}^{\mathrm{RD}}$  for some  $\alpha, \beta$ , we get
$$
\left(\lambda_ {1} ^ {\prime} + (1 + \alpha) \lambda_ {2} ^ {\prime} - \hat {\lambda} _ {1} \kappa^ {\prime}\right) X _ {r, 1} ^ {\mathrm {R D}} + \lambda_ {2} ^ {\prime} \beta X _ {r, 2} ^ {\mathrm {R D}} + \left(\lambda_ {1} ^ {\prime} + \lambda_ {2} ^ {\prime}\right) W _ {1} + \lambda_ {2} ^ {\prime} W _ {2} - \hat {\lambda} _ {1} Z _ {1} = \hat {\lambda} _ {2} \hat {X} _ {2}. \tag {384}
$$
$$
\theta_ {1} := \frac {\lambda_ {1} ^ {\prime} + (1 + \alpha) \lambda_ {2} ^ {\prime} - \hat {\lambda} _ {1} \kappa^ {\prime}}{\hat {\lambda} _ {2}}, \tag {385}
$$
$$
\theta_ {2} := \frac {\lambda_ {2} ^ {\prime} \beta}{\hat {\lambda} _ {2}}, \tag {386}
$$
$$
Z _ {2} := \frac {\left(\lambda_ {1} ^ {\prime} + \lambda_ {2} ^ {\prime}\right)}{\hat {\lambda} _ {2}} W _ {1} + \frac {\lambda_ {2} ^ {\prime}}{\hat {\lambda} _ {2}} W _ {2} - \frac {\hat {\lambda} _ {1}}{\hat {\lambda} _ {2}} Z _ {1}. \tag {387}
$$
$$
\hat {X} _ {2} = \theta_ {1} X _ {r, 1} ^ {\mathrm {R D}} + \theta_ {2} X _ {r, 2} ^ {\mathrm {R D}} + Z _ {2}. \tag {388}
$$
Notice that the above proof only uses the information about reconstructions of the operating points in DP-tradeoff and it does not depend on the choice of PLF. So, it holds for both PLF-JD and PLF-FMD. This concludes the proof.
H.3 Gaussian Example
Assume that the sources are symmetric in the sense that  $\sigma_1^2 = \sigma_2^2 = \sigma_3^2 = 1$ ,  $\rho_{1} = \rho_{2} = \rho_{3}\coloneqq \rho$  for some  $0 < \rho \leq 1$ . Also, suppose that the perception thresholds are symmetric, i.e.,  $P_{1} = P_{2} = P_{3}\coloneqq P$  for some  $0 < P\leq 1$ . We choose the rate tuple  $\mathsf{R}$  such that the minimum distortions  $D_{j}^{\mathrm{min}} = D$  for  $j\in \{1,2,3\}$ . According to Appendix H.1, such rates are given by
$$
R _ {1} = \frac {1}{2} \log \frac {1}{D}, \tag {389}
$$
$$
R _ {2} = \frac {1}{2} \log \frac {\rho^ {2} D + (1 - \rho)}{D}, \tag {390}
$$
$$
R _ {3} = \frac {1}{2} \log \frac {\rho^ {2} D + \left(1 - \rho^ {2}\right)}{D}. \tag {391}
$$
The covariance matrix of the MMSE representations  $\operatorname{cov}(X_{r,1}^{\mathrm{RD}}, X_{r,2}^{\mathrm{RD}}, X_{r,3}^{\mathrm{RD}})$  is given by  $(1 - D)\Sigma$  where
$$
\Sigma := \left( \begin{array}{c c c} 1 & \rho & \rho^ {2} \\ \rho & 1 & \rho \\ \rho^ {2} & \rho & 1 \end{array} \right). \tag {392}
$$
If we introduce the 0-PLF while keeping the rates as those of MMSE reconstructions, it can be shown that the optimal distortions are all equal to  $D_{1} = D_{2} = D_{3} = 2 - 2\sqrt{1 - D}$ . Denote the reconstructions by  $(\hat{X}_{D_1}^0,\hat{X}_{D_2}^0,\hat{X}_{D_3}^0)$  and notice that the covariance matrix of the reconstructions is equal to that of the sources and is given by  $\Sigma$ . Thus, the covariance matrix of  $(X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}},X_{r,3}^{\mathrm{RD}})$  is  $(1 - D)$  times the covariance matrix of  $(\hat{X}_{D_1}^0,\hat{X}_{D_2}^0,\hat{X}_{D_3}^0)$ . So, the reconstructions  $(X_{r,1}^{\mathrm{RD}},X_{r,2}^{\mathrm{RD}},X_{r,3}^{\mathrm{RD}})$  and  $(\hat{X}_{D_1}^0,\hat{X}_{D_2}^0,\hat{X}_{D_3}^0)$  can be transformed to each other by the scaling factor  $\frac{1}{\sqrt{1 - D}}$ . This inspires the idea that reconstructions corresponding to different tuples  $(\mathsf{D},\mathsf{P})$  are linearly related to those of MMSE representations which is the essence of the following Theorem 6. Moreover, both PLFs either based on FMD or JD perform similarly in this example since individually scaling the reconstruction of each frame finally ends up in matching the covariance matrix of all frames.
I Justification of low-rate regime for Moving MNSIT
In the MovingMNIST dataset, the digit in I-frame is generated uniformly across the  $32 \times 32$  center region in a  $64 \times 64$  image, meaning that  $\log (32 \times 32) = 10$  bits are required to localize the digits and any lower rate would result in much less correlated reconstructions. As such, one can consider  $R_{1} = 12$  bits (2 extra bits for content and style) as a low rate. For P-frames, the movement is uniformly constrained within a  $10 \times 10$  region so any rate  $R_{2} \leq \log_{2}(10 \times 10) = 6.6$  bits (excluding residual compensation) can be considered a low rate.
J Experiment Details
J.1 Training Setup and Overview
Our compression architecture is built on the scale-space flow model [32], which allows end-to-end training without relying on pre-trained optical flow estimators. For better compression efficiency, we replace the residual compression module with the conditioning one [33]. In the following, we will interchangeably refer  $X_{1}$  as the I-frame and subsequent ones as P-frames. The annotation for the encoder, decoder, and critic (discriminator) will be referred to as  $f, g$ , and  $h$  respectively and their specific functionality (e.g. motion compression, joint perception critic) will be described within context through a subscript/superscript.
Distortion and Perception Measurement: We follow the setup in prior works [16, 21] for distortion and perception measurement. Specifically, we use MSE loss  $\mathbb{E}[||X - \hat{X} ||^2]$  as a distortion metric and Wasserstein-1 distance as a perception metric, which can be estimated through the WGAN critics (following the Kanotorovich-Rubinstein duality). For the marginal perception metric, we optimize
Figure 7: Compression diagram for (a) I-frame (b) P-frame with universal representation and (c) P-frame with optimized representation. For simplicity, we do not show the shared randomness  $K$ .
our critics  $h_m$  to classify between original image  $X$  and synthetic ones  $\hat{X}$ . This will then allow us to measure  $W_1(P_X, P_{\hat{X}})$  since:
$$
W _ {1} \left(P _ {X}, P _ {\hat {X}}\right) = \sup  _ {h _ {m} \in \mathcal {F}} \mathbb {E} \left[ h _ {m} (X) \right] - \mathbb {E} \left[ h _ {m} (\hat {X}) \right] \tag {393}
$$
where  $\mathcal{F}$  is a set of all bounded 1-Lipschitz functions. Similarly, the joint perception metric is realized through  $W_{1}(P_{X_{1}\ldots X_{j}},P_{\hat{X}_{1}\ldots \hat{X}_{j}})$  by training a critic  $h_j$  that classifies between synthetic and authentic sequences:
$$
W _ {1} \left(P _ {X _ {1} \dots X _ {j}}, P _ {\hat {X} _ {1} \dots \hat {X} _ {j}}\right) = \sup  _ {h _ {j} \in \mathcal {F}} \mathbb {E} \left[ h _ {j} \left(X _ {1}, \dots , X _ {i}\right) \right] - \mathbb {E} \left[ h _ {j} \left(\hat {X} _ {1}, \dots , \hat {X} _ {i}\right) \right] \tag {394}
$$
In practice, the set of 1-Lipschitz functions is limited by the neural network architecture. Also, although our analysis employs the Wasserstein-2 distance as a perception metric, it is worth noting that the ideal reconstructions (0-PLF) for this metric and the one used in our study should be identical.
I-frame Compressor: We compress I-frames in a similar fashion as previous works [16, 21]. Our encoder  $f_{I}$  and decoder  $g_{I}$  in Figure 7a contain a series of convolution operations and we control the rate  $R_{1}$  by varying the dimension and quantization level in the bottleneck. The model utilizes common randomness through the dithered quantization operation. For a given rate  $R_{1}$ , we vary the amount of DP tradeoff by controlling the hyper-parameter  $\lambda_{i}^{\mathrm{marginal}}$  in the following minimization objective  $\mathcal{L}_1$ :
$$
\mathcal {L} _ {1} = \mathbb {E} [ \| X _ {1} - \hat {X} _ {1} \| ^ {2} ] + \lambda_ {i} ^ {\text {m a r g i n a l}} W _ {1} \left(P _ {X _ {1}}, P _ {\hat {X} _ {1}}\right) \tag {395}
$$
Following the results from Zhang et al. [16], we fix the encoder after optimizing the encoder-decoder pair for MSE representations. We then fix the encoder and train another decoder to obtain the optimal reconstruction with perfect perception, i.e.,  $W_{1}(P_{X},P_{\hat{X}})\approx 0$ . We will leverage these universal representation results to compress P-frames (both end-to-end and universal).
$P$ -frame Compressor: We describe the loss functions before explaining our architectures. Given previous reconstructions  $\hat{X}_{[i-1]} := \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{i-1}\}$ , one can adjust the distortion-joint perception tradeoff by controlling the hyper-parameter  $\lambda_i^{\mathrm{joint}}$  in the following objective  $\mathcal{L}_i$ .
$$
\mathcal {L} _ {i} ^ {\text {j o i n t}} = \mathbb {E} [ \| X _ {i} - \hat {X} _ {i} \| ^ {2} ] + \lambda_ {i} ^ {\text {j o i n t}} W _ {1} \left(P _ {X _ {[ i ]}}, P _ {\hat {X} _ {[ i ]}}\right) \tag {396}
$$
Note that in order to achieve 0-PLF-JD, previous reconstructions  $\hat{X}_{[i - 1]}$  must also achieve 0-PLF-JD, since it is impossible to reconstruct such  $\hat{X}_i$  if the previous  $\hat{X}_{[i - 1]}$  are not temporally consistent<sup>3</sup>. For the FMD metric, we use the loss function in (395).
In the universal model in Figure 7b, the motion encoder  $f_{i}^{m}$  compresses and sends the quantized flow fields  $[X_{r,i}^{m}]$  between the MMSE reconstruction  $\tilde{X}_{i-1}$  and  $X_{i}$ . Given  $[X_{r,i}^{m}]$ , the flow decoder and warping module  $g_{i}^{m}$  will transform  $\tilde{X}_{i-1}$  into  $\tilde{X}_{i}^{w}$  (predicted frame). We use  $f_{i}^{c}$  to compress the
3This follows from the inequality:  $W_2^2 (P_{X_1,X_2},P_{\hat{X}_1,\hat{X}_2})\geq W_2^2 (P_{X_1},P_{\hat{X}_1}) + W_2^2 (P_{X_2},P_{\hat{X}_2})$