SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As we see, the choice of  $J_{k}$  affects current (and future) errors by its correlation with the two independent components,  $(A_{k}\hat{x}_{k - 1}^{*},K_{k}\mathcal{I}_{k})$ . Let us now consider filters of the form
$$
J _ {k} = \Phi_ {k} A _ {k} \Upsilon_ {k} + \Pi_ {k} K _ {k} \mathcal {I} _ {k} + w _ {k}, \quad w _ {k} \sim \mathcal {N} \left(0, \Sigma_ {w _ {k}}\right), \tag {18}
$$
where here, by slight abuse of notation, we define the process of unutilized information as
$$
\Upsilon_ {k} \triangleq \hat {x} _ {k - 1} ^ {*} - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | \hat {x} _ {0}, \dots , \hat {x} _ {k - 1} \right], \quad \Upsilon_ {0} = 0. \tag {19}
$$
The matrices  $\Pi_k, \varPhi_k \in \mathbb{R}^{n_x \times n_x}$  are coefficients such that
$$
\Sigma_ {w _ {k}} = Q _ {k} - \Phi_ {k} A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top} - \Pi_ {k} M _ {k} \Pi_ {k} ^ {\top} \succeq 0, \tag {20}
$$
where we denote the Kalman update covariance by  $M_{k} \triangleq K_{k}S_{k}K_{k}^{\top}$ . This guarantees that  $J_{k} \sim N(0, Q_{k})$ , as desired. Importantly, as opposed to  $v_{k}$ , the dimension of  $\Upsilon_{k}$  is fixed, namely it does not grow with time  $k$ . Note that since  $\hat{x}_{k-1}^{*}$  is a linear combination of  $(\mathcal{I}_0, \dots, \mathcal{I}_{k-1})$ , (18) is a special choice of  $\pi_{k}$  and  $\phi_{k}$  in (14) where coefficient size does not grow with  $k$  as well.  $\Upsilon_{k}$  and its covariance  $\Sigma_{\Upsilon_{k}}$  are given via a recursive form, illustrated in Fig. 2 (and derived in App. E):
$$
\Upsilon_ {k + 1} = A _ {k} \Upsilon_ {k} + K _ {k} \mathcal {I} _ {k} - \Psi_ {k} Q _ {k} ^ {\dagger} J _ {k}, \Sigma_ {\Upsilon_ {k + 1}} = A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} + M _ {k} - \Psi_ {k} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top}, \tag {21}
$$
$$
\varPsi_ {k} \triangleq M _ {k} \Pi_ {k} ^ {\top} + A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top}. \tag {22}
$$
Note again that unlike the innovation  $\mathcal{I}_k$ ,  $\Upsilon_k$  might not be a white process, but we have that  $\Upsilon_k$  is independent of the filter's output  $\hat{X}_0^{k-1}$  and  $\mathcal{I}_k$ . Equation (17) now takes the form
$$
D _ {k} = A _ {k} D _ {k - 1} A _ {k} ^ {\top} + Q _ {k} + M _ {k} - \Pi_ {k} M _ {k} - M _ {k} \Pi_ {k} ^ {\top} - A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top} - \Phi_ {k} A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top}, \tag {23}
$$
where we observe that  $\Sigma_{\Upsilon_k}$  may depend on the choice of  $\{\Pi_t, \varPhi_t\}_{t=0}^{k-1}$ . In order to retrieve an optimal filter, one should perform optimization of the desired objective over  $\{\Pi_t, \varPhi_t\}_{t=0}^T$ , under the constraints given in (20). From (16), minimizing the cost (5) boils down to minimizing  $\sum_{k=0}^T \alpha_k \operatorname{Tr}\{D_k\}$  subject to the constraints in (20), which is an optimization problem over only  $\mathcal{O}(n_x^2 T)$  parameters.
4.2 An Exactly solvable reduction: Perceptual Kalman Filter
We now consider an additional reduction, which allows to obtain a closed form solution for the filter's coefficients. Specifically, a reduced-size filter can be obtained by using the form (12) and (18) with the sub-optimal choice  $\varPhi_{k}\equiv 0$ , namely
$$
J _ {k} = \Pi_ {k} K _ {k} \mathcal {I} _ {k} + w _ {k}. \tag {24}
$$
The meaning of this choice is that only newly-observed information is used for updating estimation at each stage, while non-utilized information from previous time steps is discarded. We note that such a simplification should be used with discretion; while requiring only half of the computations, rejecting the unutilized information might lead to enhanced errors in some settings (e.g. when an observation is missing). However, in many cases where observations are informative and different timesteps are weakly correlated, past unutilized information rapidly becomes irrelevant and can be safely ignored. We demonstrate the utility of this reduction in Sec. 5. Here,  $\Pi_k$  is a  $n_x \times n_x$  coefficient matrix, and
Algorithm 1 Perceptual Kalman Filter (PKF)
Input: Kalman Filter outputs  $\{\mathcal{I}_k, K_k, S_k\}_{k=0}^T$ , weights  $\{\alpha_k\}_{k=0}^T$ , matrices  $A, P_0, \{Q_k\}_{k=1}^T$  satisfying  $\operatorname{Im}\{B_k M_k B_k\} \subseteq \operatorname{Im}\{Q_k\}$  for all  $k = 0, \dots, T$ .
initialize:  $\hat{x}_0 = \Pi_0K_0y_0 + w_0$ ,  $w_0 \sim \mathcal{N}\left(0, P_0 - \Pi_0M_0\Pi_0^\top\right)$ , where  $\Pi_0$  is given by (30) and  $B_0 = \sum_{t=0}^{T} \alpha_t(A^t)^\top A^t$ .
for  $k = 1$  to  $T$  do
calculate:  $M_{k} = K_{k}S_{k}K_{k}^{\top}$ $B_{k} = \sum_{t = k}^{T}\alpha_{t}(A^{t - k})^{\top}A^{t - k},M_{B} = B_{k}M_{k}B_{k}.$
compute optimal gain (30):  $\Pi_k = Q_k M_B^{\frac{1}{2}} \left(M_B^{\frac{1}{2}} Q_k M_B^{\frac{1}{2}}\right)^{\frac{1}{2} \dagger} M_B^{\dagger \frac{1}{2}} B_k M_k M_k^{\dagger}$ .
sample:  $w_{k}\sim \mathcal{N}\left(0,Q_{k} - \Pi_{k}M_{k}\Pi_{k}^{\top}\right)$
update state:  $\hat{x}_k = A\hat{x}_{k - 1} + \Pi_kK_k\mathcal{I}_k + w_k$
$w_{k}\sim \mathcal{N}(0,Q_{k} - \Pi_{k}M_{k}\Pi_{k}^{\top})$  is a Gaussian noise, uncorrelated with all other states, observations and noises in the system up to time  $k$ . Again, note that  $\mathcal{I}_k$  is independent of the measurements up to time  $k - 1$ , hence this choice makes  $J_{k}$  independent of  $\hat{X}_0^{k - 1}$ . These innovation-based corrections resemble the mechanism exploited in (8), hence we will refer to optimal filters of the form (24) as perceptual Kalman filters (PKF).
Now, by a straightforward substitution, (17) becomes
$$
D _ {k} = A _ {k} D _ {k - 1} A _ {k} ^ {\top} + Q _ {k} + M _ {k} - \Pi_ {k} M _ {k} - M _ {k} \Pi_ {k} ^ {\top}, \quad k = 0, \dots , T, \tag {25}
$$
where we consider  $Q_0 = P_0$ ,  $M_0 = \Sigma_{\hat{x}_0^*}$  and  $D_{-1} = 0$ . As before, minimizing (5) boils down to minimizing  $\sum_{k=0}^{T} \alpha_k \operatorname{Tr}\{D_k\}$ , and in order for (24) to be well-defined, we should enforce the constraints  $Q_k - \Pi_k M_k \Pi_k^\top \succeq 0$ ,  $k = 0, \dots, T$ . For simplicity, we consider the time-invariant case where  $A_k \equiv A$ , so that the optimization objective becomes
$$
\left\{ \begin{array}{l l} \min  _ {\left\{\Pi_ {k} \right\} _ {k = 0} ^ {T}} & \sum_ {k = 0} ^ {T} \alpha_ {k} \operatorname {T r} \left\{D _ {k} \right\} \\ \text {s . t .} & D _ {k} = \sum_ {t = 0} ^ {k} A ^ {k - t} \left[ \mathscr {Q} _ {t} - \Pi_ {t} M _ {t} - M _ {t} \Pi_ {t} ^ {\top} \right] \left(A ^ {\top}\right) ^ {k - t}, \quad k = 0, \dots , T, \\ & Q _ {k} - \Pi_ {k} M _ {k} \Pi_ {k} ^ {\top} \succeq 0, \quad k = 0, \dots , T, \end{array} \right. \tag {26}
$$
where we denoted  $\mathcal{Q}_k = Q_k + M_k$ . Substituting  $D_{k}$ , we can rewrite the objective as
$$
\sum_ {k = 0} ^ {T} \operatorname {T r} \left\{\sum_ {t = k} ^ {T} \alpha_ {t} A ^ {t - k} \left[ \mathcal {Q} _ {k} - 2 \Pi_ {k} M _ {k} \right] \left(A ^ {\top}\right) ^ {t - k} \right\}. \tag {27}
$$
As we can see, optimization over a particular coefficient  $\Pi_k$  does not affect other summands of the external sum. Therefore, each  $\Pi_k$  can be optimized separately. Minimizing the cost at the  $k$ -th step is equivalent to
$$
\max  _ {\Pi_ {k}} \operatorname {T r} \left\{\Pi_ {k} M _ {k} \sum_ {t = k} ^ {T} \alpha_ {t} \left(A ^ {t - k}\right) ^ {\top} A ^ {t - k} \right\} \quad \text {s . t .} \quad Q _ {k} - \Pi_ {k} M _ {k} \Pi_ {k} ^ {\top} \succeq 0. \tag {28}
$$
Let us denote  $B_{k} \triangleq \sum_{t=k}^{T} \alpha_{t}(A^{t-k})^{\top} A^{t-k} = \alpha_{k} I + A^{\top} B_{k+1} A$ . As we now show, this optimization problem possesses a closed-form solution under a mild assumption (which is satisfied e.g. when  $Q_{k} \succ 0$ ). The proof is given in Appendix F.
Theorem 4.3. Assume that  $\operatorname{Im}\left\{B_kM_kB_k\right\} \subseteq \operatorname{Im}\left\{Q_k\right\}$  for every  $k$ . Let  $M_B \triangleq B_kM_kB_k$  and denote  $\Omega = \left\{\Pi_k: Q_k - \Pi_kM_k\Pi_k^\top \succeq 0\right\}$ . Then the optimal value in (28) is given by
$$
\left. \max  _ {\Pi_ {k} \in \Omega} \operatorname {T r} \left\{\Pi_ {k} M _ {k} B _ {k} \right\} = \operatorname {T r} \left\{\left(M _ {B} ^ {1 / 2} Q _ {k} M _ {B} ^ {1 / 2}\right) ^ {\frac {1}{2}} \right\}, \right. \tag {29}
$$
and is achieved by the optimal coefficient (which is generally not unique)
$$
\Pi_ {k} ^ {*} = Q _ {k} M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q _ {k} M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {\dagger 1 / 2} B _ {k}. \tag {30}
$$
For a closed form solution under the alternative assumption that  $\operatorname{Im}\{M_k\} \subseteq \operatorname{Im}\{Q_k\}$ , as well as a discussion of stationary filters, please see the Appendix. The Perceptual Kalman filter (PKF) obtained from Thm. 4.3 is summarized in Alg. 1.
5 Numerical demonstrations
We now revisit our main question: what is the cost of temporal consistency in online restoration? In addition, as we have seen in Sec. 4.2, the relaxation  $\varPhi_{k}=0$ , yielding the Perceptual Kalman filters, reduces the complexity of computation, possibly at the cost of higher errors. It is natural, then, to ask what is the cost of this simplification. In the following experiments, we compare the performance of several filters;  $\hat{x}_{\mathrm{kal}}^{*}$  and  $\hat{x}_{\mathrm{tic}}$  correspond to the Kalman filter and the temporally
inconsistent filter (10) (which does not possess perfect-perceptual quality). The estimate  $\hat{x}_{\mathrm{opt}}$  is a perfect-perception filter obtained by numerically optimizing the coefficients in (18), where the cost is the MSE at termination time,  $\mathcal{C}_{\mathrm{T}} = \mathbb{E}[\| \hat{x}_T - x_T\|^2]$ .  $\hat{x}_{\mathrm{direct}}$  is a perfect-perception filter obtained by the optimization approach discussed in Appendix C, where we consider again the terminal cost. The estimates  $\hat{x}_{\mathrm{auc}},\hat{x}_{\mathrm{minT}}$  correspond to PKF outputs (Alg. 1) minimizing the total cost (area under curve)  $\mathcal{C}_{\mathrm{auc}} = \sum_{k = 0}^{T}\mathbb{E}[\| \hat{x}_k - x_k\|^2 ]$  and the terminal cost  $C_T$ , respectively. The filters are summarized in Table 1. Full details and additional experimental results are given in App. I.
Table 1: List of demonstrated filters.
description	def.	perfect-perception
per-sample	temporal
x*kal	Kalman filter	Alg.2	X	X
xtic	Per-sample quality(no temporal)	(10)	✓	X
xopt	Optimized filter	(18)	✓	✓
xdirect	Directly optimized	App. C	✓	✓
xauc	PKF(total cost)	Alg. 1	✓	✓
xminT	PKF(terminal cost)	Alg.1	✓	✓
5.1 Harmonic oscillator
We start with a simple 2-D example. Specifically, we consider a harmonic oscillator, where the state  $x_{k}\in \mathbb{R}^{2}$  corresponds to position and velocity, and the observation at time  $t$  is a noisy versions of the position at time  $t - \frac{1}{2}\Delta_t$ , where  $\Delta_t$  is the sampling period (see App. I for details). Figure 3 shows the MSE for  $\hat{x}_{\mathrm{opt}}$  and the sub-optimal PKF outputs  $\hat{x}_{\mathrm{auc}},\hat{x}_{\mathrm{minT}}$ . The estimates  $\hat{x}_{\mathrm{kcal}}^*$  and  $\hat{x}_{\mathrm{tic}}$  achieve lower MSE than  $\hat{x}_{\mathrm{opt}}$ , however they do not possess perfect-perceptual quality. The difference in MSE between the filters  $\hat{x}_{\mathrm{opt}}$  and  $\hat{x}_{\mathrm{tic}}$  is the cost of temporal consistency in online estimation for this setting.
In Figure 3, we also observe that the PKF estimations are indeed not MSE optimal at time  $T = 255$ . However, their RMSE is only about
Figure 3: MSE on Harmonic oscillator. We observe  $(i)$  the difference in distortion between the perfect-perceptual state  $\hat{x}_{\mathrm{opt}}$ , optimized according to (18), and  $\hat{x}_{\mathrm{tic}}$ . This additional cost is due to the perceptual constraint on the joint distribution. Also note the gap  $(ii)$  between MSE of the optimized estimator and  $\hat{x}_{\mathrm{minT}}$  which is due to the sub-optimal choice of coefficients,  $\varPhi_{k}=0$ .
$30\%$  higher than that of  $\hat{x}_{\mathrm{opt}}$  and they have the advantage that they can be solved analytically and require computing only half of the coefficients  $(\Pi_k)$ . The penalty related to this reduction may vary, depending on the exact setup. In Fig. 4 we demonstrate the relations between gaps due to temporal consistency (gap between  $\hat{x}_{\mathrm{direct}}$  and  $\hat{x}_{\mathrm{tic}}$ ) and due to the reduction  $\varPhi_{k}=0$  (gap between  $\hat{x}_{\mathrm{direct}}$  and  $\hat{x}_{\mathrm{minT}}$ ), in various settings based on the Harmonic Oscillator example. On the left pane, dynamics are driven by  $x_{k} = \rho Ax_{k - 1} + q_{k}$ , where  $A$  is a fixed marginally-stable matrix, and  $\rho$  is a scalar controlling the strength of correlation between timesteps. On the right pane,  $\rho = 1$  and observations are given only up to time  $\tau$  ( $y_{k}$  is only noise for  $k \geq \tau$ ). We observe that for systems where timesteps are strongly correlated, or in the absence of current information, discarding  $v_{k}$  leads
<sup>1</sup>Our code is publicly available at https://github.com/ML-group-il/perceptual-kalman-filters.
Figure 4: Different dynamics yield different MSE gaps. Here we use different dynamics, based on the Harmonic oscillator (matrices  $A, C, Q, R$  are given in Sec. I.1) to demonstrate behavior of gaps due to temporal consistency (gap between  $\hat{x}_{\mathrm{direct}}$  and  $\hat{x}_{\mathrm{tic}}$ ) and due to the reduction  $\varPhi_{k}=0$  (gap between  $\hat{x}_{\mathrm{direct}}$  and  $\hat{x}_{\mathrm{minT}}$ ). We present analytical errors at time  $T=255$ , normalized by the ground-truth state  $x_{T}$  variance. (Left) dynamics are driven by the stabilized matrix  $\rho A$  for different scalar values of  $\rho$ . Observe that when timesteps are more correlated (higher  $\rho$ ), both consistency and unutilized information play a major role, hence MSE gaps between filters grow. (Right) Here,  $\rho = 1$ , and observations are given only up to time  $\tau$  ( $y_{k}=$  noise, for  $k \geq \tau$ ).
to significantly higher errors. Whenever states are less correlated and more observations are given, unutilized information can be ignored with lower cost.
5.2 Dynamic texture
We now illustrate the qualitative effects of perceptual estimation in a simplified video restoration setting. Specifically, we consider a video of a "dynamic texture" of waves in a lake. Such dynamic textures are accurately modeled by linear dynamics of a Gaussian latent representation [5], whose parameters we learn from a real video. Here, frames are generated from a latent 128-dimensional state  $x_{k}^{FA}$  which corresponds to their Factor-Analysis (FA) decomposition (see e.g. [2, Sec. 12.2.4] for more details). Thus,  $512 \times 512 \times 3$  frames in the video domain are created through an affine transformation of  $x_{k}^{FA}$ . Linear observations  $y_{k} \in \mathbb{R}^{32 \times 32}$  are given in the frame (pixel) domain, by  $16 \times$  downsampling the  $Y$ -channel of the generated ground-truth frames, and adding white Gaussian noise. All filtering is done in the latent domain, and then transformed to the pixel domain. MSE is also calculated in the FA domain. The exact settings can be found in App. I.
In the first experiment, measurements are supplied up to frame  $k = 127$  and then stop (Fig. 5), letting the different filters predict the next, unobserved, frames of the sequence. We can see that until frame  $k = 127$ , all filters reconstruct the reference frames well. Starting from time  $k = 128$ , when measurements stop, the Kalman filter slowly fades into a static, blurry output which is the average frame value in this setting. This is a non-'realistic' video; Neither the individual frames nor the temporal (static) behavior are natural to the domain. Our perfect-perceptual filter,  $\hat{x}_{\mathrm{auc}}$ , keeps generating a 'natural' video, both spatially and temporally. This makes its MSE grow faster<sup>2</sup>.
We now perform a second experiment, where measurements are set to zero until frame  $k = 512$ . At times  $k > 512$  they are given again by the noisy, downsampled frames as described above. In Fig. 6 we present the outcomes of the different filters. We first note that up to frame  $k = 512$ , there is no observed information, hence outputs are actually being generated according to priors. The Kalman filter outputs a static, average frame. The filter  $\hat{x}_{\mathrm{tic}}$  randomizes each frame independently, leading to unnatural random movement with flickering features. At frame  $k = 513$ , when observations become available,  $\hat{x}_{\mathrm{kcal}}^*$  and  $\hat{x}_{\mathrm{tic}}$  get updated immediately, creating an inconsistent, non-smooth motion between frames 512 and 513. The PKF output  $\hat{x}_{\mathrm{auc}}$ , on the other hand, maintains a smooth motion. Since the outputs of inconsistent filters rapidly become similar to the ground-truth, their errors drop. The perfect-perceptual filter,  $\hat{x}_{\mathrm{auc}}$ , remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.
2More visual details, including ground-truth clips and empirical error can be found in the Appendix. Full video clips for both experiments are supplied with the supplementary material.
Figure 5: Frame prediction on a dynamic texture domain. In this experiment, measurements are supplied only up to frame  $k = 127$  and the filter's task is to predict the unobserved future frames. Observe that  $\hat{x}_{\mathrm{kal}}^*$  fades into a blurred average frame, while the perceptual filter  $\hat{x}_{\mathrm{auc}}$  generates a natural video, both spatially and temporally. This makes its MSE grow faster.
Figure 6: Frame generation on a dynamic texture domain. In the first half of the demo ( $k \leq 512$ ), there are no observations, hence the reference signal is restored according to prior distribution. The filters with no perfect-perceptual quality constraint in the temporal domain generate non-realistic frames (Kalman filter output  $\hat{x}_{\mathrm{kal}}^*$ ) or unnatural motion ( $\hat{x}_{\mathrm{tic}}$ ). Perceptual filter  $\hat{x}_{\mathrm{auc}}$  is constrained by previously generated frames and the natural dynamics of the domain, hence its MSE decays slower.
Conclusion We studied the problem of causal filtering of time sequences from corrupted or missing observations, where the filter's output process is constrained to possess perfect (spatio-temporal) perceptual-quality in the sense of being distributed like the original signal. Our theoretical derivations focused on Gauss-Markov state-space processes. We introduced the novel concept of an unutilized information process and established a special class of perceptual filters, coined Perceptual Kalman Filters (PKF), that are based on the innovation process alone. We demonstrated the qualitative effects of perfect perceptual quality estimation on a video reconstruction problem. To the best of our knowledge, this is the first work addressing the distortion-perception tradeoff in online restoration settings. This work paves the way toward understanding perceptual online filtering, and the cost of temporal consistency in sequential estimation problems.
Acknowledgements The work of RM was partially supported by the Israel Science Foundation grant 1693/22 and by the Skillman chair in biomedical sciences. The work of TM was partially supported by the Israel Science Foundation grant 2318/22 and by a gift from KLA. RM and TM were supported by the Ollendorff Minerva Center, ECE Faculty, Technion.
[1] Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks. advances in neural information processing systems, 30, 2017.
[2] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[3] Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6228-6237, 2018.
[4] Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixe, and Nils Thuerey. Learning temporal coherence via self-supervision for gan-based video generation. ACM Transactions on Graphics (TOG), 39(4):75-1, 2020.
[5] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. International Journal of Computer Vision, 51:91-109, 2003.
[6] Dror Freirich and Emilia Fridman. Decentralized networked control of systems with local networks: A time-delay approach. Automatica, 69:201-209, 2016.
[7] Dror Freirich and Emilia Fridman. Decentralized networked control of discrete-time systems with local networks. International Journal of Robust and Nonlinear Control, 28(1):365-380, 2018.
[8] Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the distortion-perception tradeoff in wasserstein space. Advances in Neural Information Processing Systems, 34:25661-25672, 2021.
[9] Robert M Freund. Introduction to semidefinite programming (sdp). Massachusetts Institute of Technology, pages 8-11, 2004.
[10] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.
[11] R.E. Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering. 82 (1) 35-45, 1960.
[12] Tae Hyun Kim, Mehdi S. M. Sajjadi, Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for video restoration. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[13] Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
[14] Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. Perceptual video super resolution with enhanced temporal consistency. arXiv preprint arXiv:1807.07930, 2018.
[15] Alexander Shapiro. Extremal problems on the set of nonnegative definite matrices. Linear Algebra and its Applications, 67:7-18, 1985.
[16] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
[17] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM review, 38(1): 49-95, 1996.
Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint - Supplementary Material
In App. A we provide a detailed theoretical background on the Kalman Filter, its properties and recursive calculation. In App. B we prove that under our perfect perceptual filtering setting, there exists a linear optimal filter (Thm. 4.1). In App. C we discuss a direct, non-recursive method for optimizing perceptual filter coefficients. In App. D we present the derivation of the Lyapunov equation (17) for the error of perceptual filters. In App. E we derive the recursive expression for the filter given in (18). In App. F we find a closed-form solution for PKF coefficients by proving Theorem 4.3. In this appendix, we also give some brief overview on the extremal problem of finding a minimal distance between distributions. App.G contains a discussion about stationary perceptual Kalman filters in the steady-state regime. We summarize all definitions and notations in App. H. Finally, in App.I we give full details for all numerical demonstrations, and present additional empirical and visual results. More results are provided in the supplementary video.
A The Kalman Filter algorithm
In this Section we supply a detailed reminder of the Kalman filter Algorithm. The celebrated Kalman filter [11] assumes a state  $x_{k} \in \mathbb{R}^{n_{x}}$ , where dynamics are modeled as deterministic linear functions perturbed by a Gaussian noise, and observations  $y_{k} \in \mathbb{R}^{n_{y}}$  are linear functions of  $x_{k}$  with an additive noise