SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
x _ {k} = A _ {k} x _ {k - 1} + q _ {k}, \quad q _ {k} \sim \mathcal {N} (0, Q _ {k}), \quad k = 1, \dots , T, \tag {31}
$$
$$
y _ {k} = C _ {k} x _ {k} + r _ {k}, \quad r _ {k} \sim \mathcal {N} (0, R _ {k}), \quad k = 0, \dots , T. \tag {32}
$$
The noise terms  $q_{k}$  and  $r_{k}$  are independent white Gaussian processes with zero mean and covariances  $Q_{k}, R_{k}$ , respectively.  $x_{0}$  is assumed to have a zero-mean Gaussian distribution with covariance  $P_{0}$ , independent of  $q_{1}, r_{0}$ . For convenience, we will sometimes refer to  $P_{0}$  as  $Q_{0}$ . The matrices  $A_{k}, C_{k}, Q_{k}, R_{k}$  and  $P_{0}$  are system parameters with appropriate dimensions, and assumed to be known. Considering the MSE distortion, we denote
$$
\hat {x} _ {k \mid s} \triangleq \underset {\hat {x}} {\operatorname {a r g m i n}} \mathbb {E} \left[ \| x _ {k} - \hat {x} \| ^ {2} | y _ {0}, \dots , y _ {s} \right], \tag {33}
$$
namely the optimal MSE estimator of the state at time  $k$ , given measurements up to time  $s$ . Under the assumptions mentioned above, Kalman filters produce the mean state estimate  $\hat{x}_{k|k}$ , an MSE-optimal estimator of  $x_{k}$  given the observations up to time  $k$ . The Kalman optimal state  $\hat{x}_k^* \equiv \hat{x}_{k|k}$  is given by the recurrence
$$
\hat {x} _ {k} ^ {*} = A _ {k} \hat {x} _ {k - 1} ^ {*} + K _ {k} \mathcal {I} _ {k}, \tag {34}
$$
where  $K_{k}$  is the optimal Kalman gain [11], given explicitly in Algorithm 2.
The vector  $\mathcal{I}_k$  is the innovation process,
$$
\mathcal {I} _ {k} = y _ {k} - C _ {k} \hat {x} _ {k | k - 1}, \tag {35}
$$
describing the contribution of the new observation  $y_{k}$  over the optimal prediction based on previous observations. Since we are in the Linear-Gaussian setup, we have that the innovation state  $\mathcal{I}_k$  is orthogonal to the measurements  $y_0,\dots ,y_{k - 1}$ , guaranteeing the MSE optimality of the estimation. The calculation of Kalman state is summarized in Algorithm 2.
Algorithm 2 Kalman Filter
initialize:  $\hat{x}_0^* = K_0y_0 = P_0C_0^\top \Sigma_{Y_0}^{-1}y_0, P_{0|0} = P_0 - P_0C_0^\top \Sigma_{Y_0}^{-1}C_0P_0,\mathcal{I}_0 = y_0,S_0 =$
$C_0P_0C_0^\top +R_0$
calculate prior:  $\hat{x}_{k|k - 1} = A_k\hat{x}_{k - 1|k - 1},\quad P_{k|k - 1} = A_kP_{k - 1|k - 1}A_k^\top +Q_k$
calculate Innovation:  $\mathcal{I}_k = y_k - C_k\hat{x}_{k|k - 1}$ ,  $S_{k} = C_{k}P_{k|k - 1}C_{k}^{\top} + R_{k}$
Kalman gain:  $K_{k} = P_{k|k - 1}C_{k}^{\top}S_{k}^{-1}$
update (posterior):  $\hat{x}_k^* = \hat{x}_{k|k} = \hat{x}_{k|k - 1} + K_k\mathcal{I}_k$ $P_{k|k} = (I - K_kC_k)P_{k|k - 1}$
B Optimality of linear filters (proof of Thm. 4.1)
In this section we show that under a family of optimality criteria (5) and perfect-perceptual quality and causality constraints (6-7), linear filters of the form (14) are optimal. We start with the following.
Theorem B.1. Let  $Y_0^T = (y_0, \ldots, y_T)$  be the set of measurements (4), and let  $(J_0^T, Y_0^T)$  be a joint distribution s.t.  $J_k$  is independent of  $y_{k+n}$  given  $Y_0^k$  for all  $k \in [0, T]$  and  $n \geq 1$ . Then,
$$
\mathbb {E} \left[ J _ {k} \mathcal {I} _ {k + n} ^ {\top} \right] = 0. \tag {36}
$$
Proof. Denote  $\hat{J}_k = \mathbb{E}\left[J_k|\mathcal{I}_0^k\right]$ . We can write the measurements as a linear function of the innovations,  $y_{k} = \sum_{t = 0}^{k}H_{k,t}\mathcal{I}_{t}$ . We have
$$
\hat {y} _ {k + n} ^ {k + n - 1} \triangleq \mathbb {E} \left[ y _ {k + n} \mid Y _ {0} ^ {k + n - 1} \right] = \mathbb {E} \left[ y _ {k + n} \mid \mathcal {I} _ {0} ^ {k + n - 1} \right] = \sum_ {t = 0} ^ {k + n - 1} H _ {k + n, t} \mathcal {I} _ {t}, \tag {37}
$$
$$
\hat {y} _ {k + n} ^ {k} \triangleq \mathbb {E} \left[ y _ {k + n} \mid Y _ {0} ^ {k} \right] = \mathbb {E} \left[ y _ {k + n} \mid \mathcal {I} _ {0} ^ {k} \right] = \sum_ {t = 0} ^ {k} H _ {k + n, t} \mathcal {I} _ {t} = \mathbb {E} \left[ \hat {y} _ {k + n} ^ {k + n - 1} \mid \mathcal {I} _ {0} ^ {k} \right]. \tag {38}
$$
For any  $k$  and  $n = 1$ ,  $\mathcal{I}_{k + 1} = y_{k + 1} - \hat{y}_{k + 1}^k$ , and therefore
$$
\mathbb {E} \left[ J _ {k} \mathcal {I} _ {k + 1} ^ {\top} \right] = \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} \left[ y _ {k + 1} - \hat {y} _ {k + 1} ^ {k} \right] ^ {\top} \mid \mathcal {I} _ {0} ^ {k} \right] \right] = \mathbb {E} \left[ \hat {J} _ {k} \left[ \hat {y} _ {k + 1} ^ {k} \right] ^ {\top} - \mathbb {E} \left[ J _ {k} \mid \mathcal {I} _ {0} ^ {k} \right] \left[ \hat {y} _ {k + 1} ^ {k} \right] ^ {\top} \right] = 0. \tag {39}
$$
This is due to the facts that  $J_{k}$  and  $y_{k + 1}$  are independent given the condition, and that  $\hat{y}_{k + 1}^{k}$  is a deterministic function of  $\mathcal{I}_0^k$ .
Now, assume we know that  $\mathbb{E}\left[J_k\mathcal{I}_t^\top \right] = 0$  for  $k + 1\leq t\leq k + n - 1$ . We can write
$$
\begin{array}{l} \mathbb {E} \left[ J _ {k} \mathcal {I} _ {k + n} ^ {\top} \right] = \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} \left[ y _ {k + n} - \hat {y} _ {k + n} ^ {k + n - 1} \right] ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] \right] \\ = \mathbb {E} \left[ \hat {J} _ {k} \left[ \hat {y} _ {k + n} ^ {k} \right] ^ {\top} - \mathbb {E} \left[ J _ {k} \sum_ {t = 0} ^ {k + n - 1} \mathcal {I} _ {t} ^ {\top} H _ {k + n, t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] \right] \\ = \mathbb {E} \left[ \hat {J} _ {k} \left[ \hat {y} _ {k + n} ^ {k} \right] ^ {\top} - \mathbb {E} \left[ J _ {k} \sum_ {t = 0} ^ {k} \mathcal {I} _ {t} ^ {\top} H _ {k + n, t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] - \mathbb {E} \left[ J _ {k} \sum_ {t = k + 1} ^ {k + n - 1} \mathcal {I} _ {t} ^ {\top} H _ {k + n, t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] \right] \\ = \mathbb {E} \left[ \hat {J} _ {k} \left[ \hat {y} _ {k + n} ^ {k} \right] ^ {\top} - \hat {J} _ {k} \left[ \hat {y} _ {k + n} ^ {k} \right] ^ {\top} - \sum_ {t = k + 1} ^ {k + n - 1} \mathbb {E} \left[ J _ {k} \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] H _ {k + n, t} ^ {\top} \right] \\ = - \sum_ {t = k + 1} ^ {k + n - 1} \mathbb {E} \left[ J _ {k} \mathcal {I} _ {t} ^ {\top} \right] H _ {k + n, t} ^ {\top} \\ = 0. \tag {40} \\ \end{array}
$$
We now show that for every filter which is feasible under (6) and (7), one can find a linear filter, jointly Gaussian with the measurement set, attaining the same cost objective.
Theorem B.2. Let  $Y_0^T = (y_0, \ldots, y_T)$  be the set of measurements (4), and let  $\mathcal{J}_0^T = (\mathcal{J}_0, \ldots, \mathcal{J}_T)$  be jointly distributed with  $Y_0^T$  such that:
(i)  $\mathcal{J}_0^T\sim \mathcal{N}(0,\mathrm{diag}\{P_0,Q_1,\ldots ,Q_T\})$
(ii)  $\mathcal{J}_k$  is independent of  $y_{k + n}$  given  $Y_0^k$  for all  $k\in [0,T]$  and  $n\geq 1$
(iii)  $\begin{array}{r}\sum_{k = 0}^{T}\alpha_{k}\mathbb{E}\left[\| x_{k} - \chi_{k}\|^{2}\right] = \mathcal{C}, \end{array}$  where  $\chi_{k}$  is the process given by  $\chi_k = A_k\chi_{k - 1} + \mathcal{J}_k$  with  $\chi_0 = \mathcal{I}_0$
Then, there exists a joint Gaussian distribution  $\left(J_0^T, Y_0^T\right)$  in which (i) and (ii) hold, and the estimator given by
$$
\hat {x} _ {k} = A _ {k} \hat {x} _ {k - 1} + J _ {k}, \quad \hat {x} _ {0} = J _ {0} \tag {41}
$$
achieves the same cost (iii), namely  $\sum_{k=0}^{T} \alpha_k \mathbb{E}\left[\|x_k - \hat{x}_k\|^2\right] = C$ .
Furthermore, we can write
$$
J _ {k} = \pi_ {k} \mathcal {I} _ {k} + \phi_ {k} v _ {k} + w _ {k}, \tag {42}
$$
$$
v _ {k} = \mathcal {I} _ {0} ^ {k - 1} - \mathbb {E} \left[ \mathcal {I} _ {0} ^ {k - 1} \mid J _ {0} ^ {k - 1} \right] \tag {43}
$$
and  $w_{k}$  is a white Gaussian noise, independent of  $Y_0^T$  and  $J_0^{k - 1}$ .
Proof. Let  $(J_0^T, Y_0^T)$  be the Gaussian distribution defined by the moments of  $(\mathcal{I}_0^T, Y_0^T)$  up to second order. We observe that from Theorem B.1 above,  $J_k$  is independent of all future innovations  $\mathcal{I}_{k+n}$ , namely it is based only on measurements up to time  $k$ . Using the notions of Theorem B.1's proof,
$$
\begin{array}{l} \mathbb {E} \left[ (J _ {k} - \hat {J} _ {k}) (y _ {k + n} - \hat {y} _ {k + n} ^ {k}) ^ {\top} | Y _ {0} ^ {k} \right] = \mathbb {E} \left[ (J _ {k} - \hat {J} _ {k}) \sum_ {t = k + 1} ^ {k + n} \mathcal {I} _ {t} ^ {\top} H _ {k + n, t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] \\ = \sum_ {t = k + 1} ^ {k + n} \left[ \mathbb {E} \left[ J _ {k} \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] - \hat {J} _ {k} \mathbb {E} \left[ \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] \right] H _ {k + n, t} ^ {\top} \\ = \sum_ {t = k + 1} ^ {k + n} \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {t}, \mathcal {I} _ {0} ^ {k} \right] | \mathcal {I} _ {0} ^ {k} \right] H _ {k + n, t} ^ {\top} \\ = \sum_ {t = k + 1} ^ {k + n} \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} | \mathcal {I} _ {t}, \mathcal {I} _ {0} ^ {k} \right] \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] H _ {k + n, t} ^ {\top} \\ = \sum_ {t = k + 1} ^ {k + n} \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} | \mathcal {I} _ {0} ^ {k} \right] \mathcal {I} _ {t} ^ {\top} | \mathcal {I} _ {0} ^ {k} \right] H _ {k + n, t} ^ {\top} \\ = \sum_ {t = k + 1} ^ {k + n} \mathbb {E} \left[ J _ {k} \mid \mathcal {I} _ {0} ^ {k} \right] \mathbb {E} \left[ \mathcal {I} _ {t} ^ {\top} \mid \mathcal {I} _ {0} ^ {k} \right] H _ {k + n, t} ^ {\top} \\ = 0. \tag {44} \\ \end{array}
$$
This means that  $J_{k}$  and  $y_{k + n}$  are independent given  $Y_0^k$ , which proves (ii).
From (17) we see that the cost functional depends only on the second order statistics of  $(\mathcal{I}_0^T,\mathcal{I}_0^T)$  which are identical to those of  $(J_0^T,\mathcal{I}_0^T)$ , hence (iii) holds:
$$
\sum_ {k = 0} ^ {T} \alpha_ {k} \mathbb {E} \left[ \| x _ {k} - \hat {x} _ {k} \| ^ {2} \right] = \sum_ {k = 0} ^ {T} \alpha_ {k} \mathbb {E} \left[ \| x _ {k} - \chi_ {k} \| ^ {2} \right] = \mathcal {C}. \tag {45}
$$
To prove (42), we now write
$$
J _ {k} = \varepsilon_ {k} + w _ {k}, \tag {46}
$$
where  $\varepsilon_{k} = \mathbb{E}\left[J_{k}|Y_{0}^{T},J_{0}^{k - 1}\right]$ , and  $w_{k} = J_{k} - \mathbb{E}\left[J_{k}|Y_{0}^{T},J_{0}^{k - 1}\right]$  is independent of  $Y_0^T$  and  $J_0^{k - 1}$ . Now, since both  $J_{k}$  and  $J_0^{k - 1}$  are independent of  $\mathcal{I}_{k + 1}^T$ ,
$$
\varepsilon_ {k} = \mathbb {E} \left[ J _ {k} \mid Y _ {0} ^ {T}, J _ {0} ^ {k - 1} \right] = \mathbb {E} \left[ J _ {k} \mid \mathcal {I} _ {0} ^ {k}, J _ {0} ^ {k - 1} \right] = \sum_ {t = 0} ^ {k} \phi_ {k, t} \mathcal {I} _ {t} + \sum_ {t = 0} ^ {k - 1} \psi_ {k, t} J _ {t}. \tag {47}
$$
$J_{k}$  is independent of  $J_0^{k - 1}$ , thus
$$
\mathbb {E} \left[ J _ {k} \mid J _ {0} ^ {k - 1} \right] = \mathbb {E} \left[ \mathbb {E} \left[ J _ {k} \mid \mathcal {I} _ {0} ^ {T}, J _ {0} ^ {k - 1} \right] \mid J _ {0} ^ {k - 1} \right] = 0. \tag {48}
$$
Conditioning both sides of (47) on  $J_0^{k-1}$  and taking expectations,
$$
0 = \sum_ {t = 0} ^ {k} \phi_ {k, t} \mathbb {E} \left[ \mathcal {I} _ {t} \mid J _ {0} ^ {k - 1} \right] + \sum_ {t = 0} ^ {k - 1} \psi_ {k, t} J _ {t}. \tag {49}
$$
Note that  $\mathbb{E}\left[\mathcal{I}_k|J_0^{k - 1}\right] = 0$ , which together with (49) implies
$$
\varepsilon_ {k} = \phi_ {k, k} \mathcal {I} _ {k} + \sum_ {t = 0} ^ {k - 1} \phi_ {k, t} \left[ \mathcal {I} _ {t} - \mathbb {E} \left[ \mathcal {I} _ {t} \mid J _ {0} ^ {k - 1} \right] \right] = \pi_ {k} \mathcal {I} _ {k} + \phi_ {k} v _ {k}. \tag {50}
$$
Now, all we have left to show is that  $w_{k}$  is a white sequence. Since  $w_{k + n}$  ( $n \geq 1$ ) is independent of  $J_0^k$  and  $\mathcal{I}_0^T$  (which also constitute  $v_{k}$ ), it is easy to obtain
$$
\mathbb {E} \left[ w _ {k + n} w _ {k} ^ {\top} \right] = \mathbb {E} \left[ w _ {k + n} \left[ J _ {k} - \pi_ {k} \mathcal {I} _ {k} - \phi_ {k} v _ {k} \right] ^ {\top} \right] = 0. \tag {51}
$$
Corollary B.3. Given a cost objective of the form  $\mathcal{C} = \sum_{k=0}^{T} \alpha_k \mathbb{E}\left[\|x_k - \hat{x}_k\|^2\right]$ , there exists a linear filter of the form
$$
J _ {k} = \pi_ {k} \mathcal {I} _ {k} + \phi_ {k} v _ {k} + w _ {k}, \tag {52}
$$
$$
\hat {x} _ {0} = J _ {0} \tag {53}
$$
$$
\hat {x} _ {k} = A _ {k} \hat {x} _ {k - 1} + J _ {k}, k = 1, \dots , T \tag {54}
$$
is an optimal estimator under the perfect perceptual quality and causality constraints (6-7).
Proof. Under the perfect perceptual quality constraint, an estimate sequence  $\chi_{k}$  must satisfy that
$$
\mathcal {J} _ {k} = \chi_ {k} - A _ {k} \chi_ {k - 1} \tag {55}
$$
is a white Gaussian process with covariances  $Q_{k}$ . If, in addition,  $\chi_{k}$  satisfies the causality condition (6), so does  $\mathcal{I}_k$ . We conclude from Theorem B.2 that there exists a causal linear filter  $J_{k}$  that achieves the same expected objective  $\mathcal{C}$  as  $\chi_{k}$ .
Now, note again that from (17), for perfect-perceptual quality causal filters, the objective  $\mathcal{C}$  is a continuous function of the covariance matrix
$$
\mathbb {E} \left[ \mathcal {J} _ {0} ^ {T} \left(\mathcal {I} _ {0} ^ {T}\right) ^ {\top} \right] = \left[ \begin{array}{c c} \operatorname {d i a g} \left\{P _ {0}, Q _ {1}, \dots , Q _ {T} \right\} & L \\ L ^ {\top} & \operatorname {d i a g} \left\{S _ {0}, S _ {1}, \dots , S _ {T} \right\} \end{array} \right] \succeq 0, \tag {56}
$$
where, due to the causality demand,  $L$  is a quasi lower triangular matrix. The set of such feasible matrices is non-empty, closed (since it is the intersection of the closed cone of PSD matrices with a finite set of hyperplanes) and bounded. Hence,  $\mathcal{C}$  attains a minimal value on some joint distribution  $p_{\mathcal{J}_0^T, Y_0^T}$ , which can be chosen to be joint-Gaussian as we have seen.
C A Direct optimization approach to perfect-perceptual quality filtering
For the sake of completeness, we now discuss a method for optimizing non-recursive perfect-perceptual quality filter coefficients. This approach leads to convex programs. However, as we will see next, it might become impractical for large configurations.
Let  $J = J_0^T \sim \mathcal{N}(0, Q)$ , where  $Q = \mathrm{diag}\left\{\{Q_k\}_{k=0}^T\right\}$ , be a causal function of the measurements,  $J = \varPhi \mathcal{I} + W$ , where  $\mathcal{I} = \mathcal{I}_0^T$  is the innovation process with covariance  $S = \mathrm{diag}\{S_k\}$  and  $W$  is an independent noise. Now,  $\hat{X} = \hat{X}_0^T = A_J J$  is the filter's output, where
$$
A _ {J} = \left[ \begin{array}{c c c c} I & 0 & \dots & 0 \\ A _ {1} & I & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ \prod_ {k = 0} ^ {T - 1} A _ {T - k} & \prod_ {k = 1} ^ {T - 1} A _ {T - k} & \dots & I \end{array} \right]. \tag {57}
$$
Recall  $X^{*}$  is the Kalman filter output given by  $X^{*} = A_{J}K\mathcal{I}$ , where  $K = \mathrm{diag}\{K_k\}$ . Let  $\mathcal{W} = \mathrm{diag}\{\alpha_k\} \otimes I_{n_x}$  be a weighting matrix. The objective (5) is now given by
$$
\begin{array}{l} \mathcal {C} (\hat {X}) = \mathbb {E} \left[ \left(\hat {X} - X ^ {*}\right) ^ {\top} \mathcal {W} \left(\hat {X} - X ^ {*}\right) \right] \\ = \operatorname {T r} \left\{\mathcal {W} \mathbb {E} \left[ \hat {X} \hat {X} ^ {T} \right] + \mathcal {W} \mathbb {E} \left[ X ^ {*} X ^ {* \top} \right] - 2 \mathcal {W} \mathbb {E} \left[ \hat {X} X ^ {* \top} \right] \right\}. \tag {58} \\ \end{array}
$$
Hence, we have to maximize
$$
\begin{array}{l} \mathcal {C} (\Phi) = 2 \operatorname {T r} \left\{\mathscr {W} \mathbb {E} \left[ \hat {X} X ^ {* ^ {\top}} \right] \right\} \\ = 2 \operatorname {T r} \left\{\mathscr {W} A _ {J} \Phi S K ^ {\top} A _ {J} ^ {\top} \right\} \\ = 2 \operatorname {T r} \left\{\left(\Phi S\right) K ^ {\top} A _ {J} ^ {\top} \mathcal {W} A _ {J} \right\} \\ = 2 \operatorname {T r} \left\{\Phi S K ^ {\top} B \right\}, \tag {59} \\ \end{array}
$$
where  $B \triangleq A_J^\top \mathcal{W}A_J$ . This is subject to the perfect perceptual-quality constraint
$$
Q - \Phi S \Phi^ {\top} \succeq 0, \text {o r e q u i v a l e n t l y} \left[ \begin{array}{c c} Q & \Phi S \\ S \Phi^ {\top} & S \end{array} \right] \succeq 0, \tag {60}
$$
where  $\varPhi$  is a lower quasi-triangular matrix (causality constraint)
$$
\Phi = \left[ \begin{array}{c c c c} \Phi_ {0, 0} & 0 & \dots & 0 \\ \Phi_ {1, 0} & \Phi_ {1, 1} & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ \Phi_ {T, 0} & \Phi_ {T, 1} & \dots & \Phi_ {T, T} \end{array} \right]. \tag {61}
$$
Again, under this formulation,
$$
J = \Phi \mathcal {I} + W, \tag {62}
$$
where  $W \sim \mathcal{N}(0, Q - \varPhi S\varPhi^\top)$  is a Gaussian noise independent of  $\mathcal{I}$ . Note that  $W_0^T$  might not be a white sequence in this case, since its covariance might not be a block-diagonal matrix. As a result, the noise sequence has to be sampled dependently. Also note that this problem possesses the same memory complexity as (14). To conclude, this method leads to convex, but large optimization programs, and is impractical for high dimensional settings or long temporal sequences.
D Derivation of eq. (17)
Recall  $\hat{x}_k^*$  is the optimal Kalman state at time  $k$ , achieving MSE given by
$$
d _ {k} ^ {*} = \mathbb {E} \left[ \left\| \hat {x} _ {k} ^ {*} - x _ {k} \right\| ^ {2} \right] = \operatorname {T r} \left\{P _ {k \mid k} \right\}. \tag {63}
$$
$P_{k|k}$  is the error covariance, given explicitly in Algorithm 2. By the orthogonality principle, for any estimator  $\hat{x}_k$  based on the measurements  $y_0,\ldots ,y_k$  we have
$$
\mathbb {E} \left[ \| x _ {k} - \hat {x} _ {k} \| ^ {2} \right] = \mathbb {E} \left[ \| x _ {k} - \hat {x} _ {k} ^ {*} \| ^ {2} \right] + \mathbb {E} \left[ \| \hat {x} _ {k} ^ {*} - \hat {x} _ {k} \| ^ {2} \right] = d _ {k} ^ {*} + \mathbb {E} \left[ \| \hat {x} _ {k} - \hat {x} _ {k} ^ {*} \| ^ {2} \right]. \tag {64}
$$
Now, consider an estimator  $\hat{x}_k$  of the form (12), and recall
$$
D _ {k} \triangleq \mathbb {E} \left[ \left[ \hat {x} _ {k} ^ {*} - \hat {x} _ {k} \right] \left[ \hat {x} _ {k} ^ {*} - \hat {x} _ {k} \right] ^ {\top} \right]. \tag {65}
$$
Since we choose  $J_{k}\sim \mathcal{N}(0,Q_{k})$  to be independent of  $\hat{x}_{k - 1}$  and  $\mathcal{I}_k$  is independent of  $\hat{x}_{k - 1}$  and  $\hat{x}_{k - 1}^{*}$ , we write
$$
\begin{array}{l} {D _ {k}} = {\mathbb {E} \left[ \hat {x} _ {k} ^ {*} - \hat {x} _ {k} \right] \left[ \hat {x} _ {k} ^ {*} - \hat {x} _ {k} \right] ^ {\top}} \\ = \mathbb {E} \left[ A _ {k} \hat {x} _ {k - 1} - A _ {k} \hat {x} _ {k - 1} ^ {*} + J _ {k} - K _ {k} \mathcal {I} _ {k} \right] \left[ A _ {k} \hat {x} _ {k - 1} - A _ {k} \hat {x} _ {k - 1} ^ {*} + J _ {k} - K _ {k} \mathcal {I} _ {k} \right] ^ {\top} \\ { = } { A _ { k } \mathbb { E } \left[ \left[ \hat { x } _ { k - 1 } ^ { * } - \hat { x } _ { k - 1 } \right] \left[ \hat { x } _ { k - 1 } ^ { * } - \hat { x } _ { k - 1 } \right] ^ { \top } \right] A _ { k } ^ { \top } + \mathbb { E } \left[ J _ { k } J _ { K } ^ { \top } \right] + K _ { k } \mathbb { E } \left[ \mathcal { I } _ { k } \mathcal { I } _ { k } ^ { \top } \right] K _ { k } ^ { \top } } \\ - \mathbb {E} \left[ J _ {k} \left[ A _ {k} \hat {x} _ {k - 1} ^ {*} + K _ {k} \mathcal {I} _ {k} \right] ^ {\top} \right] - \mathbb {E} \left[ \left[ A _ {k} \hat {x} _ {k - 1} ^ {*} + K _ {k} \mathcal {I} _ {k} \right] J _ {k} ^ {\top} \right] \\ = A _ {k} D _ {k - 1} A _ {k} ^ {\top} + K _ {k} S _ {k} K _ {k} ^ {\top} + Q _ {k} \\ - \mathbb {E} \left[ J _ {k} \mathcal {I} _ {k} ^ {\top} \right] K _ {k} ^ {\top} - K _ {k} \mathbb {E} \left[ \mathcal {I} _ {k} J _ {k} ^ {\top} \right] - A _ {k} \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} J _ {k} ^ {\top} \right] - \mathbb {E} \left[ J _ {k} \hat {x} _ {k - 1} ^ {* \top} \right] A _ {k} ^ {\top}. \tag {66} \\ \end{array}
$$
E Derivation of recursive perfect-perceptual quality filters
We now derive the recursive expression (21)-(22) for the filter given in (18),
$$
\hat {x} _ {k} = A _ {k} \hat {x} _ {k - 1} + J _ {k}, \tag {67}
$$
$$
J _ {k} = \Phi_ {k} A _ {k} \Upsilon_ {k} + \Pi_ {k} K _ {k} \mathcal {I} _ {k} + w _ {k}, w _ {k} \sim \mathcal {N} \left(0, \Sigma_ {w _ {k}}\right), \tag {68}
$$
defined by the coefficients  $\{\Pi_k, \varPhi_t\}_{t=0}^T$  fulfilling the constraints (20). Recall
$$
\Upsilon_ {k} \triangleq \hat {x} _ {k - 1} ^ {*} - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | \hat {x} _ {0}, \dots , \hat {x} _ {k - 1} \right] = \hat {x} _ {k - 1} ^ {*} - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | J _ {0}, \dots , J _ {k - 1} \right] \tag {69}
$$
where  $\hat{x}_k^*$  is the Kalman state.  $J_0^{k - 1},\mathcal{Y}_k,\mathcal{I}_k,w_k$  are jointly-Gaussian and independent, and we have
$$
\mathbb {E} \left[ J _ {n} J _ {k} ^ {\top} \right] = Q _ {k} \delta_ {n = k}, \tag {70}
$$
$$
\mathbb {E} \left[ \mathcal {I} _ {k} J _ {k} ^ {\top} \right] = S _ {k} K _ {k} ^ {\top} \Pi_ {k} ^ {\top}, \tag {71}
$$
$$
\mathbb {E} \left[ \Upsilon_ {k} J _ {k} ^ {\top} \right] = \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top}. \tag {72}
$$
$$
\begin{array}{l} \Upsilon_ {k + 1} - A _ {k} \Upsilon_ {k} = \hat {x} _ {k} ^ {*} - A _ {k} \hat {x} _ {k - 1} ^ {*} - \left[ \mathbb {E} \left[ \hat {x} _ {k} ^ {*} | J _ {0} ^ {k} \right] - A _ {k} \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | J _ {0} ^ {k - 1} \right] \right] \\ = K _ {k} \mathcal {I} _ {k} - K _ {k} \mathbb {E} \left[ \mathcal {I} _ {k} \mid J _ {0} ^ {k} \right] - A _ {k} \left[ \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} \mid J _ {0} ^ {k} \right] - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} \mid J _ {0} ^ {k - 1} \right] \right] \tag {73} \\ \end{array}
$$
Since  $J_0^k$  is an independent sequence, and since  $\mathcal{I}_k$  depends only on  $J_{k}$
$$
K _ {k} \mathbb {E} \left[ \mathcal {I} _ {k} \mid J _ {0} ^ {k} \right] = K _ {k} \mathbb {E} \left[ \mathcal {I} _ {k} \mid J _ {k} \right] = K _ {k} S _ {k} K _ {k} ^ {\top} \Pi_ {k} ^ {\top} Q _ {k} ^ {\dagger} J _ {k}. \tag {74}
$$
We also have that  $\mathcal{Y}_k, J_k$  are independent of  $J_0^{k-1}$ , implying
$$
\begin{array}{l} \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | J _ {0} ^ {k} \right] - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | J _ {0} ^ {k - 1} \right] = \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} - \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} | J _ {0} ^ {k - 1} \right] | J _ {0} ^ {k} \right] \\ = \mathbb {E} \left[ \Upsilon_ {k} | J _ {0} ^ {k} \right] = \mathbb {E} \left[ \Upsilon_ {k} | J _ {k} \right] \\ = \Sigma \Upsilon_ {k} A _ {k} ^ {\top} \Phi_ {k} ^ {\top} Q _ {k} ^ {\dagger} J _ {k}. \tag {75} \\ \end{array}
$$
$$
\Upsilon_ {k + 1} = A _ {k} \Upsilon_ {k} + K _ {k} \mathcal {I} _ {k} - \Psi_ {k} Q _ {k} ^ {\dagger} J _ {k}, \tag {76}
$$
$$
\varPsi_ {k} \triangleq M _ {k} \Pi_ {k} ^ {\top} + A _ {k} \Sigma_ {\mathcal {Y} _ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top}. \tag {77}
$$
The covariance is then given by the recursive form
$$
\begin{array}{l} \Sigma_ {\Upsilon_ {k + 1}} = A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} + M _ {k} + \Psi_ {k} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top} \\ - A _ {k} \Sigma_ {\Upsilon_ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top} Q _ {k} ^ {\top} \Psi_ {k} ^ {\top} - K _ {k} S _ {k} K _ {k} ^ {\top} \Pi_ {k} ^ {\top} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top} (78) \\ - \left[ A _ {k} \Sigma_ {\mathcal {Y} _ {k}} A _ {k} ^ {\top} \Phi_ {k} ^ {\top} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top} \right] ^ {\top} - \left[ K _ {k} S _ {k} K _ {k} ^ {\top} \Pi_ {k} ^ {\top} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top} \right] ^ {\top} (79) \\ = A _ {k} \Sigma \gamma_ {k} A _ {k} ^ {\top} + M _ {k} - \Psi_ {k} Q _ {k} ^ {\dagger} \Psi_ {k} ^ {\top}. (80) \\ \end{array}
$$
At time  $k = 0$  we have  $\Upsilon_0 = 0$  and  $\Sigma \Upsilon_0 = 0$ .
Remark E.1 (The non-reduced case). For the full, non-reduced linear filter (14)-(15), we have the following similar formula
$$
v _ {k} = \left[ \begin{array}{c} \mathcal {I} _ {k - 1} \\ v _ {k - 1} \end{array} \right] - \left[ \begin{array}{c c} S _ {k - 1} & 0 \\ 0 & \Sigma_ {v _ {k - 1}} \end{array} \right] \left[ \begin{array}{c} \pi_ {k - 1} ^ {\top} \\ \phi_ {k - 1} ^ {\top} \end{array} \right] Q _ {k - 1} ^ {\dagger} J _ {k - 1} \tag {81}
$$