SlowGuess/NeurIPS_2023_OCR huggingface dataset:
residual information  $[X_{r,i}^{c}]$  between  $X_{i}$  and  $\hat{X}_i^{w4}$ , which will be decoded by  $g_i^c$ . We note that for MMSE representation,  $g_i^c$  only requires  $\tilde{X}_i^w$  as a conditional input while an additional conditioning input  $\hat{X}_{[i - 1]}$  is required when perceptual optimization is involved. Together,  $f_{i}^{m},g_{i}^{m},f_{i}^{c}$ , and  $g_{i}^{c}$  are optimized for MMSE reconstructions. To train for different DP tradeoffs, we fix  $f_{i}^{m},g_{i}^{m},f_{i}^{c}$  and adapt the new decoder  $\hat{g}_i^c$  (conditioning on  $\tilde{X}_i^w,\hat{X}_{[i - 1]}$ ). We note that fixing  $g_i^m$  for universal representation is essential since  $[X_{r,i}^{c}]$  is dependent on the outputs  $\tilde{X}_i^w$  of  $g_i^m$ .
In the end-to-end model in Figure 7c, we use an MMSE representation to estimate the motion vector, as in the case of the universal model. The only difference is that the encoder  $f_{i}^{c}$  also uses previous  $\hat{X}_{i}$  and the encoders will be jointly trained with the decoders.
J.2 Networks Architecture
In this section, we describe the network architecture for universal and end-to-end P-frame compressor models. In the architecture layout, we denote BN2D and SN for the Batchnorm2D and Spectral Normalization layers. Convolutional and transposed convolutional layer are denoted as "conv" and "upconv" respectively, which is accompanied by number of filters, kernel size, stride, and padding.
Motion Encoder and Decoder. The universal and optimized end-to-end model shares the same architecture for the motion encoder and decoder.  $(f_{i}^{m}$  and  $g_{i}^{m}$  respectively). We follow the original implementations [32] and present the convolutional architecture in Table 3. Different from the original implementation, however, we replace the last layer with dithered quantization layer (as in [16]) in our implementation. The output dimension of the motion encoder is denoted as  $d_{m}$ .
Table 3: Motion Encoder  ${f}_{i}^{m}$  and Decoder  ${g}_{i}^{m}$  .
(a) Encoder  $f_{i}^{m}$
Input-64×64×(2×channels)
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (dm:4:2:0), BN2D
Quantizer
(b) Decoder  $g_{i}^{m}$
Input-dm
upconv (64:4:1:0), BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (3:5:2:0), BN2D
Residual Encoder and Decoder. The architecture of the conditional residual encoder is shown in Table 4a, where we stack multiple frames along their channel dimension as an input. As described previously, in the residual encoder, the universal model requires only  $X_{i}$ ,  $\tilde{X}_{i}^{w}$  while the end-to-end model will receive  $X_{i}$ ,  $\tilde{X}_{i}^{w}$  and  $\hat{X}_{[i-1]}$ . We denote the output dimension of this residual encoder as  $d_{r}$ . In the decoding part, the decoder will first condition all the previous reconstructions  $\hat{X}_{[i-1]}$  by projecting them into an embedding vector of size 192 (conditioning module in Table 4b). Then we concatenate this vector with the output of  $f_{i}^{r}$ . The concatenated vector will be fed into the decoder (Table 4c) to produce the reconstruction  $\hat{X}_{i}$ .
FMD and JD Critics. For the video critics, our PLF-JD critic architecture is inspired by the work of Kwon and Park [40], where we concatenate frames sequentially along their channel dimensions. For both PLF-FMD and PLF-JD critics, we add spectral normalization layers for better convergence. Their architecture is shown in Table 5.
Rate and output dimension The rate  $R$  is computed by  $\log_2(d_{enc} \times L)$ , where  $L$  is the number of quantization levels and  $d_{enc} = d_r + d_m$ . Table 6 provides configurations of the rate,  $d_m$ ,  $d_r$ , and  $L$  in the experiment.
Training Details: We use a batch size of 64, RMSProp optimizer with a learning rate of  $5 \times 10^{-5}$ , and train each model with 360 epochs, where the training set contains 60000 images. To accelerate
<sup>4</sup>Here, we use conditioning [33] instead of sending  $X_{i} - \tilde{X}_{i-1}^{w}$  as in the original work [32]
For the I-frame compressor, we follow the DCGAN implementation by Denton et al [39], adding the dithered quantization layer in the encoder's last layer (https://github.com/edenton/svg/blob/master/models/dcgan_64.py)
Table 4: Residual Encoder, Conditional Module, and Residual Decoder.
(a)Encoder  $f_{i}^{c}$
Input
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (dr:4:1:0), BN2D
Quantizer
(b)Conditional Module
Input
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (64:5:2:0), BN2D, l-ReLU
conv (192:4:1:0), BN2D
(c)Decoder
Input-(dr+192)
upconv (64:4:1:0) uc4s1, BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (64:5:2:0), BN2D, l-ReLU
upconv (channels:5:2:0), BN2D
Table 5: PLF-FMD and PLF-JD critic for frame  $i$  .
(a) PLF-FMD Critic
Input-64×64×channels
SN, conv (64:4:2:1), l-ReLU
SN, conv (128:4:2:1), l-ReLU
SN, conv (256:4:2:1), l-ReLU
conv (512:4:2:1), l-ReLU
Linear
(b) PLF-JD Critic
Input-64×64×(i×channels)
SN, conv (64:4:2:1), l-ReLU
SN, conv (128:4:2:1), l-ReLU
SN, conv (256:4:2:1), l-ReLU
conv (512:4:2:1), l-ReLU
Linear
training, we pre-train each model for 60 epochs with the MSE objective only. Under WGAN-GP framework [30], we use the gradient penalty of 10 and update the encoders/decoders for every 5 iterations. The parameters  $\lambda$  controlling the tradeoff are in Table.7. Training takes 2 days per model on a single NVIDIA P100 GPU. For the MovingMNIST factor of two bound and permanence of error experiments, we repeat the training 3 times.
Table 6: Rate, embedding dimension  ${d}_{m},{d}_{r}$  and quantization level  $L$  .
(a) P-frame encoder,  $R_{1} = \infty$
R2	dm	dr	L
1 bit	1	0	2
2 bits	1	1	2
3.17 bits	1	1	3
(b) P-frame encoder,  $R_{1} = \epsilon$  (12 bits).
R2	dm	dr	L
4 bit	2	2	2
8 bits	4	2	2
12 bits	6	2	2
J.3 Permanence of Error on KTH Datasets
The KTH dataset is a widely-used benchmark dataset in computer vision research, consisting of video sequences of human actions performed in various scenarios. We show more examples supporting our argument for the permanence of error on this realistic dataset. We use 16 bits for each frame. In general, the 0-PLF-JD decoder consistently outputs correlated but incorrect reconstructions due to the error induced by the first reconstructions, i.e., the P-frames will follow the wrong direction induced from the I-frame reconstruction. Besides the moving direction, we also notice that the type of actions (i.e. walking, jogging, and running) is also affected. On the other hand, while losing some temporal cohesion, MMSE and 0-PLF FMD decoders manage to fix the movement error.
Finally, we computed LPIPS [41] in Table 8, which is known as a full-reference perceptual metric for images. We compare LPIPS for each reconstruction (MMSE, 0-PLF-FMD and 0-PLF-JD) at each timeframe. This result aligns with our results on distortion loss, as the MMSE and 0-PLF-FMD tend to correct the reconstruction (so the ground truth and reconstruction look more similar over time). On the other hand, due to error permanence, the 0-PLF-JD reconstructions become different from their source sequence, causing the score to go up.
Table 7: Perception loss and their associated  $\lambda$
Perception Loss	λ × 10-3
Joint Distance (JD)	0.0, 0.7, 1.0, 1.15, 1.2, 1.25, 1.3, 1.5, 1.7
2.0, 3.0, 5.0, 8.0, 10.0, 40.0, 80.0
Frame Marginal Distance (FMD)	0.0, 0.4, 0.7, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 7.0, 10.0, 40.0
Figure 8: Additional Experimental Results for the Permanence of Error Phenomenon on KTH Dataset.
Table 8: LPIPS score on KTH dataset (lower is better).
MMSE	PLF-FMD	PLF-JD
1st frame	0.1036	0.0584	0.0584
2nd frame	0.0521	0.0313	0.0594
3rd frame	0.0413	0.0232	0.0613
J.4 RDP Tradeoff for 3 frames
We extend our experimental results for the RDP-tradeoff and the principal of universality to the case of GOP size 3. As mentioned in the main paper, while the universal model only requires MMSE representations, the optimal end-to-end model also requires the MMSE reconstructions from previous frames to provide best estimates for motion flow vectors. Practically, this is challenging for our employed architecture since only previous  $\hat{X}_1$ ,  $\hat{X}_2$  are available. As a result, to compare the RDP tradeoff between universal and end-to-end model, we also provide the end-to-end model with the MMSE estimate from previous frames while noting that this is unfeasible in practice. Interestingly, we show in Figure 9 the RDP tradeoff curves for the third frame  $X_3$  and its reconstruction  $\hat{X}_3$ ,
(a) Case 1:  $R_{1} = \infty$ ,  $R_{2} = R_{3} = \{1,2,3.17\}$  bits
(b) Case 2:  $R_{1} = 12$ ,  $R_{2} = R_{3} = \{4,8,12\}$  bits
Figure 9: RDP tradeoff curves for end-to-end and universal models. We plot the tradeoff for the two regimes:  $R_{1} = \infty$  and  $R_{1} = \epsilon$  in (a) and (b) respectively. The universal and optimal curves are close to each other.
observing that the universal and optimized curves are still relatively close to each other. When  $(R_{1}, R_{2}, R_{3}) = (\infty, \epsilon, \epsilon)$ , we note that the distortion for  $X_{3}$  is larger than  $X_{2}$  since the allocated rate is not enough to correct the motion. Finally, for the case  $(R_{1}, R_{2}, R_{3}) = (\epsilon, \epsilon, \epsilon)$ , we note that the curves again converge as in the case of  $(R_{1}, R_{2}) = (\epsilon, \epsilon)$  due to the incorrect reconstruction in the I-frame.
J.5 Diversity and Correlation
When  $(R_1, R_2) = (\infty, \epsilon)$ , our theoretical analysis predicted that the decoder optimized for JD is capable of producing diverse reconstructions. On the other hand, an optimized decoder for FMD will tend to produce reconstructions that are highly correlated with the previous reconstruction  $\hat{X}_1^6$ . In our experiment, we also observe such behavior, summarized in Table 9 and show several examples for  $R_2 = 2$  bits in Figure 10. We observe that reconstructions from the joint metric deviate more randomly from  $X_1$  than the marginal reconstructions. The marginal reconstructions, on the other hand, stay much closer to their original reconstruction  $\hat{X}_1$ .
Figure 10: Diversity in reconstruction  $\hat{X}_2$  for 0-PLF-JD and correlation with previous frames  $\hat{X}_1$  for 0-PLF-JMD. We show  $X_1$  in the first column. From the second column, the light-dark region represents  $X_1$  and the color digit represents  $X_2$ ,  $\hat{X}_2$ . For each perception metric, we show two samples.
We measure the diversity in  $\hat{X}_2$  reconstruction using  $\operatorname{E}[\operatorname{Var}(\hat{X}_2|X_1,X_2)]$  and the correlation with  $\hat{X}_1$  by  $\operatorname{E}[\operatorname{sim}(\hat{\mathrm{X}}_2,\mathrm{X}_1)]$ , where  $\operatorname{sim}(u,v)$  is the cosine distance between  $u,v$ . Table 9a shows that as we increase the number of bits in  $R_2$ , the diversity decreases as the decoder can reconstruct the frame more precisely. In Table 9b, we see that the joint metric keeps the correlation relatively constant, showing that it actually preserves the temporal consistency. On the other hand, as the rate becomes larger, 0-PLF-FMD reconstruction tends to be less correlated with the previous frame  $X_1$ . Finally, we note that our architecture innately utilizes common randomness to produce diverse reconstructions and does not suffer from mode-collapse behavior in general conditional GAN settings [42].
$X_{1} = \hat{X}_{1}$  in this regime.
Table 9: Diversity (a) between  ${\widehat{X}}_{2}$  and Correlation Measures (b) between  ${\widehat{X}}_{2}$  and  ${X}_{1}$  .
(a) Diversity Measures  $\uparrow$
R2	Joint	Marginal
1 bit	0.0096	0.0004
2 bits	0.0082	0.0029
3.17 bits	0.0042	0.0022
(b) Correlation Measures.
R2	Joint	Marginal
1 bit	0.5218	0.6202
2 bits	0.5190	0.5969
3.17 bits	0.5205	0.5508
PLF-FMD:
PLF-FMD:  $\hat{X}_2$
PLF-JD:X
$X_{2}$
PLF-FMD:  $\hat{X}_2$
PLF-JD:X
PLF-JD:X
X2
X
PLF-FMD:  $\hat{X}_2$
PLF-FMD:  $\hat{X}_2$
PLF-JD:X
PLF-JD:X2
X1
Figure 11: Visualization of error permanence. The PLF-JD reconstructions propagate the flaws in the color tone from the previous I-frame reconstruction while the PLF-FMD is able to fix such error. Compression rate for I-frame and P-frame are 0.144bpp (low rate) and 4.632bpp (high rate) respectively.
K Error Permanence on UVG Dataset
We demonstrate the phenomenon of error permanence in a large-scale scenario using the UVG dataset. Our P-frame compressor is trained on the Vimeo-90k dataset. As illustrated in Figure 11, when reconstructing I-frames, an inaccurate color tone is introduced, which persists when employing PLF-JD. However, PLF-FMD effectively rectifies this issue within P-frames. Numerical results are
Table 10: Distortion  $(X_{2} - \hat{X}_{2})^{2}$  evaluated across 3900 UVG frames. The PLF-JD reconstructions exhibit notably greater distortion compared to MMSE and PLF-FMD, aligning with our theoretical findings.
MMSE	PLF-FMD	PLF-JD
Distortion (MSE)	0.0026	0.0032	0.0168
in Table. 10. We note that for the I-frame, we use the pretrained model in [43] that targets high perceptual quality and is also trained on Vimeo-90k dataset.
This work studies the effects of different perception loss functions, namely the PLF-JD and PLF-FMD, on the performance of lossy causal video compression. Our theoretical analysis and experiment reveal the error permanence phenomenon and show the universality principle, suggesting that MMSE representation can be transformed into other points on the DP tradeoffs.
In practice, one might want to combine these two losses, for example, perfect framewise realism (0-PLF FMD) while retaining some degree of temporal cohesion (PLF-JD small), which is not considered in this work. Furthermore, analysis for other types of video compression schemes, such as with B-frame, and scaling the universality compression architecture to high-definition videos are also desired.