SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 7: MSE distortion on Harmonic oscillator.  $\hat{x}_{\mathrm{opt}}$  is a numerically optimized perfect-perceptual quality filter's output (minimizing error at time  $T = 255$ , dashed horizontal line).  $\hat{x}_{\mathrm{auc}}, \hat{x}_{\mathrm{minT}}$  are PKF outputs minimizing different objectives. Observe that PKF estimations are not MSE optimal, but require less computations.  $\hat{x}_{\mathrm{kal}}^*$  and  $\hat{x}_{\mathrm{tic}}$  are not perfect-perceptual quality filters. (top) Empirical error, over  $N = 1024$  sampled trajectories. (bottom) Analytical error. The difference in distortion between the perfect-perceptual state  $\hat{x}_{\mathrm{opt}}$ , optimized according to (18), and  $\hat{x}_{\mathrm{tic}}$  is due to the perceptual constraint on the joint distribution. This is the cost of temporal consistency in online estimation for this setting. The gap between the MSE of the optimized estimator and  $\hat{x}_{\mathrm{minT}}$  is due to the sub-optimal choice of coefficients,  $\varPhi_k = 0$ .
I.2 Example: Two coupled inverted pendulums
Next, we demonstrate the quantitative behavior of perceptual Kalman filters, by comparing the MSE between the PKF outputs when minimizing different cost functions, and between non-perceptual filters outputs. More specifically, this experiment demonstrates:
1. How minimizing different objectives in Algorithm 1 leads to different filters.
2. The cost of perfect-perceptual quality filtering, given by Algorithm 1, over optimal filters.
We consider a higher-dimensional, well-studied example of two coupled inverted pendulums, mounted on carts [7, 6]. The cart positions, pendulum deviations, and their velocities (Fig. 8), are given by the discretized stable closed-loop system with perturbation
$$
x _ {k + 1} = A x _ {k} + q _ {k}, \quad q _ {k} \sim \mathcal {N} (0, Q), \tag {133}
$$
where  $x_{k}\in \mathbb{R}^{8}$  . The initial state is distributed as
$$
x _ {0} \sim \mathcal {N} (0, P _ {0}). \tag {134}
$$
The system matrices are given by
$$
A = I + A _ {c l} \cdot \Delta_ {t}, \tag {135}
$$
where  $\Delta_t = 5 \times 10^{-4}$  is the sampling interval and
$$
A _ {c l} = \left[ \begin{array}{c c} A _ {1} + B K _ {1} & F \\ F & A _ {2} + B K _ {2} \end{array} \right], \tag {136}
$$
$$
A _ {1} = A _ {2} = \left[ \begin{array}{c c c c} 0 & 1 & 0 & 0 \\ 2. 9 1 5 6 & 0 & - 0. 0 0 0 5 & 0 \\ 0 & 0 & 0 & 1 \\ - 1. 6 6 6 3 & 0 & 0. 0 0 0 2 & 0 \end{array} \right], \quad B = \left[ \begin{array}{c} 0 \\ - 0. 0 0 4 2 \\ 0 \\ 0. 0 1 6 7 \end{array} \right]. \tag {137}
$$
The coupling is given by
$$
F = \left[ \begin{array}{c c c c} 0 & 0 & 0 & 0 \\ 0. 0 0 1 1 & 0 & 0, 0. 0 0 0 5 & 0 \\ 0 & 0 & 0 & 0 \\ - 0. 0 0 0 3 & 0 & - 0. 0 0 0 2 & 0 \end{array} \right], \tag {138}
$$
and stabilizing state-feedback controllers (each acts on a single cart) are
$$
K _ {1} = \left[ \begin{array}{l l l l} 1 1 3 9 6. 0 & 7 1 9 6. 2 & 5 7 3. 9 6 & 1 1 9 9. 0 \end{array} \right], \quad K _ {2} = \left[ \begin{array}{l l l l} 2 9 2 4 1 & 1 8 1 3 5 & 2 8 7 5. 3 & 3 6 9 3. 9 \end{array} \right]. \tag {139}
$$
The partial measurements are given by  $y_{k} = Cx_{k} + r_{k}$ , where  $r_k\sim \mathcal{N}(0,R)$ , with coefficients
$$
C = \left[ \begin{array}{c c} \bar {C} _ {1} & 0 \\ 0 & \bar {C} _ {2} \end{array} \right], \quad \bar {C} _ {1} = \bar {C} _ {2} = \left[ \begin{array}{l l l l} 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{array} \right]. \tag {140}
$$
Namely, we observe only position and angle for each cart/pendulum, while velocities are not being measured.
The perturbation covariances are given by
$$
P _ {0} = \left[ \begin{array}{c c} \bar {P} _ {0} & 0 \\ 0 & \bar {P} _ {0} \end{array} \right], \quad Q = \left[ \begin{array}{c c} \bar {Q} & 0 \\ 0 & \bar {Q} \end{array} \right], \quad R = \left[ \begin{array}{c c} \bar {R} & \frac {1}{8} \bar {R} \\ \frac {1}{8} \bar {R} & \bar {R} \end{array} \right], \tag {141}
$$
$$
\bar {P} _ {0} = \left[ \begin{array}{c c c c} 0. 1 5 4 & 0. 1 4 2 & - 0. 1 4 3 & 0. 0 9 3 \\ 0. 1 4 2 & 0. 1 4 4 & - 0. 1 2 4 & 0. 0 5 8 \\ - 0. 1 4 3 & - 0. 1 2 4 & 0. 1 6 7 & - 0. 1 4 8 \\ 0. 0 9 3 & 0. 0 5 8 & - 0. 1 4 8 & 0. 1 9 2 \end{array} \right] \cdot 5 \times 1 0 ^ {- 4}, \tag {142}
$$
$$
\bar {Q} = 1 0 ^ {- 2} \cdot \left[ \begin{array}{c c c c} 0. 6 4 2 & - 0. 1 3 6 & 0. 7 8 & 0. 2 6 2 \\ - 0. 1 3 6 & 0. 8 9 4 & - 0. 2 4 8 & 0. 0 7 4 \\ 0. 7 8 & - 0. 2 4 8 & 1. 2 8 4 & - 0. 3 1 4 \\ 0. 2 6 2 & 0. 0 7 4 & - 0. 3 1 4 & 1. 7 6 6 \end{array} \right] \times \Delta_ {t}, \bar {R} = 1 0 ^ {- 2} \cdot \left[ \begin{array}{l l} 0. 3 7 5 & - 0. 3 3 \\ - 0. 3 3 & 0. 7 7 1 \end{array} \right] \times \Delta_ {t}. \tag {143}
$$
Figure 9: MSE distortion on Coupled inverted pendulums for perceptual and non-perceptual filters (near the time  $T$ ).  $\hat{x}_{\mathrm{auc}}, \hat{x}_{\mathrm{minT}}$  are PKF outputs minimizing different objectives. Observe that while both possess perfect-perceptual quality, they yield different estimations. Also, pay attention to the MSE gap between the MSE-optimal, but not perceptual, Kalman filter and the PKFs.
Figure 8: Coupled inverted pendulums.
We simulate the system for  $2^{10}$  time steps ( $T = 2^{10} - 1$ ), over  $N = 2^{10}$  independent experiments. In Figures 9 and 10 we show the MSE distortion as a function of time,  $\mathbb{E}\left[\| \hat{x}_k - x_k\|^2\right]$ , for the different filters of Table 3;  $\hat{x}_{\mathrm{kAL}}^*$  is the optimal Kalman filter.  $\hat{x}_{\mathrm{tic}}$  is the perceptual filter without consistency constraints, given in (10).  $\hat{x}_{\mathrm{auc}}$  is the PKF output minimizing the total cost (130).  $\hat{x}_{\mathrm{minT}}$  (marked by  $\star$ ) is the PKF output minimizing the terminal cost (129).
We observe that filters satisfying the perfect perceptual quality constraint  $(\hat{x}_{\mathrm{auc}}$  and  $\hat{x}_{\mathrm{minT}})$  achieve higher distortions compared to the per-sample only perceptual filter  $\hat{x}_{\mathrm{tic}}$ , which in turn attains MSE distortion slightly higher than that of the MSE-optimal Kalman filter. This demonstrates again the cost of temporal consistency in online estimation. Note also that PKFs minimizing different cumulative objectives, yield different estimations; while  $\hat{x}_{\mathrm{minT}}$  is optimal at termination time  $T$ ,  $\hat{x}_{\mathrm{auc}}$  achieves a lower MSE on average. As we will see next, both filters attain the same perceptual quality.
Figure 10: MSE distortion on Coupled inverted pendulums for perceptual and non-perceptual filters (full view).
In Fig. 11 we estimate the perceptual quality, given by the Wasserstein distance between the ground-truth distribution and the empirical Gaussian distributions of the different filters outputs. In Fig. 11(top) we estimate the distance between single-sample distributions, while in Fig. 11(bottom) we consider the joint distributions of 16 state-vectors,  $x_{t}, t \in [k - 15, k]$ . Observe that while each sample of  $\hat{x}_{\mathrm{tic}}$  is distributed similarly to its reference sample, it fails to attain perfect perceptual quality where we measure the distance from the real process distribution. PKF outputs attain low perceptual index (high quality) in both scenarios. We also present the perceptual quality measured for the ground-truth signal  $x_{\mathrm{gt}}$  empirical distribution, as a reference.
Figure 12 shows the asymptotic behavior (empirical error for large horizon  $T$ ) of  $\hat{x}_{stat}$ , the stationary PKL (125). The figure also presents the empirical errors for Kalman filter and its stationary version (multiplied by a factor of 2, which is an upper bound on the MSE distortion of perceptual estimators without temporal constraints, see [3]), and the theoretical steady-state error of (125), obtained by optimizing (128) (dashed horizontal line) for comparison. The error of the non-stationary perceptual filter  $\hat{x}_{\mathrm{auc}}$  is also shown.
I.3 Dynamic texture
Here we illustrate the qualitative effects of perceptual (temporally consistent) estimation in a simplified video restoration setting. Please see the supplementary video for the full videos. This setup visually demonstrates how:
1. Filters with no perfect perceptual quality tend to generate non-realistic images or atypical motion (random or slow movement, flickering artifacts etc.).
2. PKF outputs are natural to the domain, both spatially and temporally.
For this extent, we introduce the 'Dynamic Texture' domain. In this domain, video frames are generated from a latent state which represents their Factor-Analysis (FA) decomposition (see e.g. Bishop and Nasrabadi [2, Sec. 12.2.4] for more details). The dynamics in the FA domain are assumed to be linear, with a small Gaussian perturbation,
$$
x _ {k} ^ {\mathrm {F A}} = A ^ {\mathrm {F A}} x _ {k - 1} ^ {\mathrm {F A}} + q _ {k}, \quad x _ {0} ^ {\mathrm {F A}} \sim \mathcal {N} (0, I), \quad x _ {k} ^ {\mathrm {F A}} \in \mathbb {R} ^ {1 2 8}. \tag {144}
$$
Figure 11: Perceptual quality measured by estimated Wasserstein distance  $\hat{d}_P$  (lower is better). (top) Distance between distributions of single samples  $p_{x_k}$  and  $p_{\hat{x}_k}$ . (bottom) Distance between distributions of 16-state vectors (at times  $[k - 15, k]$ ),  $p_{X_{k-15}^k}$  and  $p_{\hat{X}_{k-15}^k}$ . Observe that  $\hat{x}_{\mathrm{tic}}$  single samples are distributed similarly to the ground-truth signal, but they fail to attain the reference joint distribution between timesteps. PKF outputs  $\hat{x}_{\mathrm{auc}}$  and  $\hat{x}_{\mathrm{minT}}$  attain high measured quality in both cases.
Figure 12: MSE distortion on Coupled inverted pendulums for stationary filters.
The state vector with the given dynamics creates frames of a wavy lake in the video domain  $^3$ , through an affine transformation.
$$
x _ {k} ^ {v i d} = W _ {\mathrm {F A} \rightarrow v i d} \left(x _ {k} ^ {\mathrm {F A}} + \varepsilon^ {\mathrm {F A}}\right). \tag {145}
$$
$W_{\mathrm{FA}\rightarrow vid}$  is a linear transformation from  $\mathbb{R}^{128}$  latent states to  $\mathbb{R}^{512\times 512\times 3}$  frames, and  $\varepsilon^{\mathrm{FA}}$  is a constant vector.  $A^{FA}$  and the noise  $q_{k}$  parameters are estimated similarly to [5]. Linear observations  $y_{k}\in \mathbb{R}^{32\times 32}$  are given in the frame (pixel) domain, by
$$
y _ {k} = C _ {k} x _ {k} ^ {\mathrm {F A}} + r _ {k}. \tag {146}
$$
At times where information is being observed,
$$
C _ {k} = C _ {\times 1 6} W _ {R G B \rightarrow y} W _ {\mathrm {F A} \rightarrow v i d}, \tag {147}
$$
where  $W_{RGB\to y}$  is a projection onto the  $Y$ -channel (grayscale) and  $C_{\times 16}$  is a matrix that performs  $16\times$  downsampling in both axes. At times where there is no observed information,  $C_k = 0$ . Here,  $r_k$  is a Gaussian noise.
In our first experiment, measurements are supplied as in (147) up to frame  $k = 127$  and then vanish  $(C_k = 0, k \geq 128)$ , letting the different filters predict the next, unobserved, frames of the sequence. We pass  $y_{k}$  as an input to the various filters (see Table 3);  $\hat{x}_{\mathrm{kAL}}^{*}$  is the Kalman filter output.  $\hat{x}_{\mathrm{tic}}$  is the perceptual filter in the spatial domain, given in (10).  $\hat{x}_{\mathrm{auc}}$  is our Algorithm (PKF) output reducing the total cost in the latent space,  $\mathcal{C}_{\mathrm{auc}} = \sum_{k=0}^{T} \mathbb{E}\left[\|x_k^{\mathrm{FA}} - \hat{x}_k\|^2\right]$ . All filtering is done in the latent domain, and then transformed to the pixel domain. MSE is also calculated in the FA domain. In (Fig. 13) we can see that until frame  $k = 127$ , all filters reconstruct the reference frames well. Starting at time  $k = 128$ , when measurements disappear, we observe that the Kalman filter slowly fades into a static, blurry output which is the average frame value in this setting. This is definitely a non-'realistic' video; Neither the individual frames nor the static behavior are natural to the domain. Our perfect-perceptual filter,  $\hat{x}_{\mathrm{auc}}$ , keeps generating a 'natural' video, both spatially and temporally. This makes its MSE grow faster.
We now perform a second experiment, where  $C_k$  is set to zero until frame  $k = 512$ . At times  $k \geq 513$  measurements are given again by the noisy, downsampled frames as described in (146)-(147). In Fig. 14 we present the outcomes of the different filters. We first observe that up to frame  $k = 512$ , there is no observed information, hence outputs are actually being generated according to priors.
<sup>3</sup>Original frames are taken from 'river-14205' by OjasweinGuptaOJG via pixabay.com, and are free to use under the content licence.
Figure 13: Frame prediction on a dynamic texture domain. In this experiment, measurements are supplied only up to frame  $k = 127$ . The filter's task here is to predict the unobserved future frames of the sequence. Observe that the  $\hat{x}_{\mathrm{kal}}^*$  fades into a blurred average frame, while the perceptual filter  $\hat{x}_{\mathrm{auc}}$  generates a natural video, both spatially and temporally. This makes its MSE grow faster,
Figure 14: Frame generation on Dynamic texture domain. In the first half of the demo ( $k \leq 512$ ), there are no observations, hence the reference signal is restored according to prior distribution. We observe that filters with no perfect-perceptual quality constraint in the temporal domain generate non-realistic frames (Kalman filter output  $\hat{x}_{\mathrm{kal}}^*$ ) or unnatural motion ( $\hat{x}_{\mathrm{tic}}$ ). Perceptual filter  $\hat{x}_{\mathrm{auc}}$  is constrained by previously generated frames and the natural dynamics of the domain, hence its MSE decays slower.
The Kalman filter outputs a static, average frame.  $\hat{x}_{\mathrm{tic}}$  randomizes each frame independently, which creates the impression of rapid, random movement with flickering features, which is unnatural to the reference domain. At frame  $k = 513$ , when observations become available, we can see that  $\hat{x}_{\mathrm{kal}}^*$  and  $\hat{x}_{\mathrm{tic}}$  are being updated immediately, creating an inconsistent, non-smooth motion between frames 512 and 513. PKF output  $\hat{x}_{\mathrm{auc}}$ , on the other hand, keeps maintaining a smooth motion. Since non-consistent filters outputs rapidly become similar to the ground-truth, their errors drop. The perfect-perceptual filter,  $\hat{x}_{\mathrm{auc}}$ , remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.