SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Sigma_ {v _ {k}} = \left[ \begin{array}{c c} S _ {k - 1} & 0 \\ 0 & \Sigma_ {v _ {k - 1}} \end{array} \right] - \left[ \begin{array}{c c} S _ {k - 1} & 0 \\ 0 & \Sigma_ {v _ {k - 1}} \end{array} \right] \left[ \begin{array}{c} \pi_ {k - 1} ^ {\top} \\ \phi_ {k - 1} ^ {\top} \end{array} \right] Q _ {k - 1} ^ {\dagger} \left[ \begin{array}{c} \pi_ {k - 1} ^ {\top} \\ \phi_ {k - 1} ^ {\top} \end{array} \right] ^ {\top} \left[ \begin{array}{c c} S _ {k - 1} & 0 \\ 0 & \Sigma_ {v _ {k - 1}} \end{array} \right]. \tag {82}
$$
Notice, however, that the dimension of  $v_{k}$  grows with time  $k$ .
F A Generalized extremal problem with semidefinite constraints (proof of Thm. 4.3)
In this section we prove Theorem 4.3. We start with a brief overview of the extremal problem of finding a minimal distance between distributions, and of general semi-definite programs.
To prove the Theorem we observe that (28), is a generalization of the extremal problem, and suggest a non-trivial dual form where, under our assumptions, strong duality holds.
F.1 Minimal distance between distributions
Consider two Gaussian distributions on  $\mathbb{R}^n$  with zero means and PSD covariance matrices  $\Sigma_1, \Sigma_2$  respectively. We consider the problem of constructing a Gaussian vector  $[X,Y]$  minimizing  $\mathbb{E}\| X - Y\|^2$  while inducing the given marginal distributions,  $X \sim \mathcal{N}(0, \Sigma_1)$ ,  $Y \sim \mathcal{N}(0, \Sigma_2)$ . This problem is equivalent to the following maximization of correlation [13]
$$
\operatorname {T r} \left\{2 \Pi \right\}\rightarrow \max  _ {\Pi} \quad \text {s . t .} \Sigma = \left[\begin{array}{c c}\Sigma_ {1}&\Pi\\\Pi^ {\top}&\Sigma_ {2}\end{array}\right] \succeq 0. \tag {83}
$$
We have the following results of Olkin and Pukelsheim [13].
Lemma F.1. [13, Lemma 1]. Let  $\Sigma_2^g$  be any generalized inverse of  $\Sigma_2$ . Then  $\Sigma \succeq 0$  iff
$$
\Sigma_ {2} \Sigma_ {2} ^ {g} \Pi^ {\top} = \Pi^ {\top} \text {a n d} \Sigma_ {1} - \Pi \Sigma_ {2} ^ {g} \Pi^ {\top} \succeq 0. \tag {84}
$$
Theorem F.2. [13, Thm. 4]. If  $\operatorname{Im}\{\Sigma_2\} \subseteq \operatorname{Im}\{\Sigma_1\}$ , then an optimal solution to (83) is given by
$$
\max  _ {\Pi} \operatorname {T r} \left\{2 \Pi \right\} = 2 \operatorname {T r} \left\{\left(\Sigma_ {2} ^ {1 / 2} \Sigma_ {1} \Sigma_ {2} ^ {1 / 2}\right) ^ {1 / 2} \right\}, \tag {85}
$$
achieved by the argument
$$
\Pi^ {*} = \Sigma_ {1} \Sigma_ {2} ^ {1 / 2} \left[ \left(\Sigma_ {2} ^ {1 / 2} \Sigma_ {1} \Sigma_ {2} ^ {1 / 2}\right) ^ {1 / 2} \right] ^ {g} \Sigma_ {2} ^ {1 / 2}. \tag {86}
$$
In the case where  $\operatorname{Im}\{\Sigma_2\} = \operatorname{Im}\{\Sigma_1\}$ ,  $\Pi^*$  is a unique optimal argument.
Under the setting discussed in Sec. 2, Theorem F.2 implies that in the more general case where  $\Sigma_x\succeq 0$ , the MSE-optimal perfect perceptual-quality estimator (2) is obtained by
$$
\hat {x} = \mathcal {T} ^ {*} x ^ {*} + w, \quad \mathcal {T} ^ {*} \triangleq \Sigma_ {x} \Sigma_ {x ^ {*}} ^ {\frac {1}{2}} \left(\Sigma_ {x ^ {*}} ^ {\frac {1}{2}} \Sigma_ {x} \Sigma_ {x ^ {*}} ^ {\frac {1}{2}}\right) ^ {\frac {1}{2} \dagger} \Sigma_ {x ^ {*}} ^ {\frac {1}{2} \dagger}. \tag {87}
$$
Here again,  $w$  is a zero-mean Gaussian noise with covariance  $\Sigma_w = \Sigma_x - \mathcal{T}^*\Sigma_{x^*}\mathcal{T}^{*\top}$ , independent of  $y$  and  $x$ , and  $\Sigma_{x^*}^\dagger$  is the Moore-Penrose inverse of  $\Sigma_{x^*}$ .
F.2 SDP Setting and duality - background
Semi-definite programming (SDP) [9, 17] is an optimization problem in  $X \in \mathbb{R}^{n \times n}$  of the form
$$
C \bullet X \rightarrow \max  _ {X} \tag {88}
$$
$$
\mathrm {s . t .} A _ {i} \bullet X = b _ {i}, i = 1, \dots , m, \tag {89}
$$
$$
X \succeq 0. \tag {90}
$$
Here,  $C, A_{i}$  are real symmetric matrices of appropriate dimensions, and  $A \bullet X = \operatorname{Tr}\{A^{\top}X\}$  is the Frobenius product. SDPs yield the Lagrangian
$$
\begin{array}{l} L (X, \lambda , \nu) = \nu^ {\top} b + \left(C - \sum_ {i = 0} ^ {m} \nu_ {i} A _ {i}\right) \bullet X + \lambda \rho_ {\min } (X) \\ = \nu^ {\top} b + \left(C - \sum_ {i = 0} ^ {m} \nu_ {i} A _ {i}\right) \bullet X + \min  _ {Y \succeq 0, \operatorname {T r} Y = \lambda} Y \bullet X, \tag {91} \\ \end{array}
$$
where  $\lambda \geq 0$  and  $\rho_{min}$  is the minimal eigenvalue. The Dual problem (DSP) is given by
$$
\nu^ {\top} b \rightarrow \min  _ {\nu} \text {s . t .} C - \sum_ {i = 0} ^ {m} \nu_ {i} A _ {i} \preceq 0. \tag {92}
$$
In this case, strong duality exists iff the SDP is strictly feasible, i.e. it has a feasible solution interior to the feasible set,  $X \succ 0$ . This condition is sometimes referred to as the Slater condition.
F.3 A generalized extremal problem with strong duality
Recall  $Q_{k}, M_{k}, B_{k}$  are real, symmetric positive semidefinite  $n_x \times n_x$  matrices, and the optimization problem (28),
$$
\operatorname {T r} \left\{\tilde {\Pi} _ {k} M _ {k} B _ {k} \right\} = \operatorname {T r} \left\{\tilde {\Pi} _ {k} M _ {k} M _ {k} ^ {\dagger} M _ {k} B _ {k} \right\}\rightarrow \max  _ {\tilde {\Pi} _ {k}} \text {, s . t .} Q _ {k} - \tilde {\Pi} _ {k} M _ {k} \tilde {\Pi} _ {k} ^ {\top} \succeq 0. \tag {93}
$$
Since (93) involves a single time step, we will omit the index  $k$ .
We consider  $\Pi = \tilde{\Pi} M$ , hence  $\Pi^{\top} = MM^{\dagger}\Pi^{\top}$ , and since  $M = MM^{\dagger}M$  we rewrite (93) as
$$
\operatorname {T r} \left\{\Pi B \right\} = \operatorname {T r} \left\{B \Pi \right\}\rightarrow \max  _ {\Pi} \text {, s . t . ,} Q - \Pi M ^ {\dagger} \Pi^ {\top} \succeq 0, \Pi^ {\top} = M M ^ {\dagger} \Pi^ {\top}. \tag {94}
$$
By Lemma F.1, the constraints in (94) are equivalent to
$$
X \triangleq \left[ \begin{array}{c c} Q & \Pi \\ \Pi^ {\top} & M \end{array} \right] \succeq 0. \tag {95}
$$
This can be formulated as the semi-definite program,
$$
C \bullet X \rightarrow \max  _ {X} \text {s . t .} \left\{\begin{array}{l l}A _ {i j} ^ {Q} \bullet X&= Q _ {i j}\\A _ {i j} ^ {M} \bullet X&= M _ {i j}\end{array}, 0 \leq i \leq j \leq n - 1, X \succeq 0, \right. \tag {96}
$$
where  $C = \frac{1}{2}\left[ \begin{array}{cc}0 & B\\ B & 0 \end{array} \right]$ , and  $A_{ij}^{Q} = \left[ \begin{array}{cc}\Lambda_{ij} & 0\\ 0 & 0 \end{array} \right], A_{ij}^{M} = \left[ \begin{array}{cc}0 & 0\\ 0 & \Lambda_{ij} \end{array} \right], \Lambda_{ij} = \frac{1}{2} (e_{ij} + e_{ji})$ .
Note that when  $B$  is a scalar matrix, (94) is similar to the problem studied in Olkin and Pukelsheim [13]. Their approach was later extended by Shapiro [15] to general linear objectives, where the Slater condition holds.
F.3.1 Strong duality
The SDP (96) yields the standard dual formulation
$$
Q \bullet \nu_ {Q} + M \bullet \nu_ {M} \rightarrow \min  _ {\nu_ {Q}, \nu_ {M}} \text {, s . t .} \left[\begin{array}{c c}\nu_ {Q}&- \frac {1}{2} B\\- \frac {1}{2} B&\nu_ {M}\end{array}\right] \succeq 0, \nu_ {Q}, \nu_ {M} \in \mathbb {R} ^ {n _ {x} \times n _ {x}}. \tag {97}
$$
This should give us a hint about the optimal solution to (94). Pay attention, however, that according to the theory, strong duality in (97) is guaranteed only if  $Q, M \succ 0$ , which might not be the case (see e.g. [15]). To get a tight bound for the general case  $Q, M \succeq 0$ , we now provide an alternative form of duality to (94).
The following is an adaptation of techniques used in Olkin and Pukelsheim [13]. We start with the following Lemma.
Lemma F.3. Let  $\Pi$  be a feasible solution to (94),  $R, G \in \mathbb{R}^{n_x \times n_x}$  are general matrices. Then,
$$
\operatorname {T r} \left\{Q R R ^ {\top} + B M B G G ^ {\top} \right\} \geq 2 \operatorname {T r} \left\{\Pi B G R ^ {\top} \right\}. \tag {98}
$$
Proof. From the non-negativity of  $X$  in (95) we have
$$
\left[ R ^ {\top}, - G ^ {\top} B \right] \left[ \begin{array}{c c} Q & \Pi \\ \Pi^ {\top} & M \end{array} \right] \left[ \begin{array}{c} R \\ - B G \end{array} \right] = \tag {99}
$$
$$
R ^ {\top} Q R + G ^ {\top} B M B G - R ^ {\top} \Pi B G - G ^ {\top} B \Pi^ {\top} R \succeq 0.
$$
The trace is nonnegative, hence we have the desired result.
Remark F.4. Similarly, we can obtain
$$
\operatorname {T r} \left\{Q B R R ^ {\top} B + M G G ^ {\top} \right\} \geq 2 \operatorname {T r} \left\{B \Pi G R ^ {\top} \right\}. \tag {100}
$$
Now, we suggest an alternative to (DSP) (97), where strong duality will hold.
Theorem F.5. [Strong duality]. Let
$$
\Omega = \left\{\Pi \in \mathbb {R} ^ {n _ {x} \times n _ {x}}: Q - \Pi M ^ {\dagger} \Pi^ {\top} \succeq 0, \Pi^ {\top} = M M ^ {\dagger} \Pi^ {\top} \right\}, \tag {101}
$$
$$
\mathcal {S} = \left\{\left(S, S ^ {-}\right): S, S ^ {-} \succeq 0, S S ^ {-} S = S, S ^ {-} S S ^ {-} = S ^ {-}, B M = S S ^ {-} B M \right\}, \tag {102}
$$
and denote  $M_B \triangleq BMB$ . Assume  $\operatorname{Im}\{M_B\} \subseteq \operatorname{Im}\{Q\}$ . Then,
$$
\begin{array}{l} \min  _ {(S, S ^ {-}) \in \mathcal {S}} \left\{Q \bullet S + M \bullet \left(B S ^ {-} B\right) \right\} = \max  _ {\Pi \in \Omega} \operatorname {T r} \left\{2 \Pi B \right\} \\ = 2 \operatorname {T r} \left\{\left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} \right\}. \tag {103} \\ \end{array}
$$
The extreme value is obtained for
$$
S ^ {*} = M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2}, \tag {104}
$$
$$
S ^ {- *} = M _ {B} ^ {1 / 2 \dagger} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} M _ {B} ^ {1 / 2 \dagger}, \tag {105}
$$
$$
\Pi^ {*} = Q S ^ {*} M _ {B} ^ {\dagger} B M = Q M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2 \dagger} B M. \tag {106}
$$
Optimal solution  $\Pi^{*}$  is generally not unique.
To prove strong duality, we will use the following lemmas.
Lemma F.6. Assume PSD matrices  $Q, M_B$  such that  $\operatorname{Im}\{M_B\} \subseteq \operatorname{Im}\{Q\}$ , then  $\operatorname{Im}\{M_B\} = \operatorname{Im}\left\{M_B^{1/2}QM_B^{1/2}\right\}$ .
Proof. Recall  $M_B, M_B^{1/2}QM_B^{1/2}$  are real symmetric matrices.
Let  $v \in \mathrm{Ker}\{M_B^{1/2}QM_B^{1/2}\}$ , we have  $\| Q^{1/2}M_B^{1/2}v \| = 0$  hence  $M_B^{1/2}v \in \mathrm{Ker}\{Q^{1/2}\} \subseteq \mathrm{Ker}\{M_B^{1/2}\}$ , which yields  $M_Bv = 0$ , implying  $\mathrm{Ker}\{M_B^{1/2}QM_B^{1/2}\} \subseteq \mathrm{Ker}\{M_B\}$ . Opposite relation is trivial.
$$
\operatorname {I m} \left\{M _ {B} \right\} = \operatorname {K e r} \left\{M _ {B} \right\} ^ {\perp} = \operatorname {K e r} \left\{M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2} \right\} ^ {\perp} = \operatorname {I m} \left\{M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2} \right\}. \tag {107}
$$
Lemma F.7. Im  $\{BM\} \subseteq \operatorname {Im}\{BMB\}$
Proof. Let  $v \in \mathrm{Ker}\{BMB\}$ , then  $\| M^{1/2}Bv \| = 0$  and  $Bv \in \mathrm{Ker}\{M^{1/2}\} = \mathrm{Ker}\{M\}$ . Hence  $\mathrm{Ker}\{BMB\} \subseteq \mathrm{Ker}\{MB\}$ . We have
$$
\operatorname {I m} \left\{B M \right\} = \operatorname {K e r} \left\{M B \right\} ^ {\perp} \subseteq \operatorname {K e r} \left\{B M B \right\} ^ {\perp} = \operatorname {I m} \left\{B M B \right\}. \tag {108}
$$
We are now ready to prove Theorem F.5.
Proof. [Theorem F.5]. Let  $\Pi \in \Omega$ , then  $X \succeq 0$  in (95). For any  $(S, S^{-}) \in S$  we can choose  $R = S^{1/2}$ ,  $G = S^{-}R$ . From the result of Lemma F.3 it follows that
$$
\begin{array}{l} Q \bullet S + M \bullet (B S ^ {-} B) = \operatorname {T r} \left\{Q R R ^ {\top} + B M B G G ^ {\top} \right\} \\ > 2 \mathrm {T r} \left\{\Pi_ {R C P} ^ {G} \right\} - 2 \mathrm {T r} \left\{\Pi_ {R S} ^ {G} - G \right\} - 2 \mathrm {T r} \left\{\Pi_ {R} \right\} \end{array} \tag {109}
$$
$$
\geq 2 \operatorname {T r} \left\{\Pi B G R ^ {\top} \right\} = 2 \operatorname {T r} \left\{\tilde {\Pi} B S ^ {-} S \right\} = 2 \operatorname {T r} \left\{\Pi B \right\}.
$$
The last equality holds since  $BM = SS^{-}BM$ , and  $\operatorname{Im}\left\{\Pi^T\right\} \subseteq \operatorname{Im}\{M\}$ .
We now prove that  $\Pi^{*}\in \Omega$
$$
\begin{array}{l} Q - \Pi^ {*} M ^ {\dagger} \Pi^ {* ^ {\top}} \\ = Q - Q M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {\dagger 1 / 2} B M M ^ {\dagger} M B M _ {B} ^ {\dagger 1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2} Q \\ = Q - Q M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {\dagger} M _ {B} ^ {1 / 2} Q \\ = Q ^ {1 / 2} \left[ I - Q ^ {1 / 2} M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {\dagger} M _ {B} ^ {1 / 2} Q ^ {1 / 2} \right] Q ^ {1 / 2} \\ = Q ^ {1 / 2} \left[ I - Q ^ {1 / 2} M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {\dagger} M _ {B} ^ {1 / 2} Q ^ {1 / 2} \right] ^ {2} Q ^ {1 / 2} \succeq 0. \tag {110} \\ \end{array}
$$
The last equality holds since it is easy to see that  $\left[I - Q^{1 / 2}M_B^{1 / 2}\left(M_B^{1 / 2}QM_B^{1 / 2}\right)^\dagger M_B^{1 / 2}Q^{1 / 2}\right]$  is a symmetric (orthogonal) projection.
We further prove that  $S^{*}, S^{-*} \in \mathcal{S}$ . It is easy to show that  $S^{*}, S^{-*}$  are symmetric generalized inverses, reflexive to each other ( $S^{-*}$  is in fact the Moore-Penrose inverse of  $S^{*}$ ):
$$
\begin{array}{l} S ^ {*} S ^ {- *} = M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2} M _ {B} ^ {1 / 2 \dagger} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} M _ {B} ^ {1 / 2 \dagger} (111) \\ = M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} \left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} M _ {B} ^ {1 / 2 \dagger} (112) \\ = M _ {B} ^ {1 / 2} M _ {B} ^ {1 / 2 \dagger} = M _ {B} ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2} (113) \\ = S ^ {- *} S ^ {*}. (114) \\ \end{array}
$$
The equalities hold since by Lemma F.6,
$$
\left. \operatorname {I m} \left\{M _ {B} ^ {1 / 2} \right\} = \operatorname {I m} \left\{M _ {B} \right\} = \operatorname {I m} \left\{M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2} \right\} = \operatorname {I m} \left\{\left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} \right\}, \right. \tag {115}
$$
and since for a PSD matrix  $R$ ,  $RR^{\dagger} = R^{\dagger}R$  is an orthogonal projection onto its image. Using Lemma F.7 we have
$$
S ^ {*} S ^ {- *} B M = M _ {B} ^ {1 / 2 \dagger} M _ {B} ^ {1 / 2} B M = B M. \tag {116}
$$
It is now easy to verify that
$$
Q \bullet S ^ {*} + M \bullet \left(B S ^ {- *} B\right) = 2 \operatorname {T r} \left\{\Pi^ {*} B \right\} = 2 \operatorname {T r} \left\{\left(M _ {B} ^ {1 / 2} Q M _ {B} ^ {1 / 2}\right) ^ {1 / 2} \right\}, \tag {117}
$$
Corollary F.8. Under the assumption,  $\operatorname{Im}\{M_B\} \subseteq \operatorname{Im}\{Q_k\}$ , the optimal gain in (28) is given by
$$
\Pi_ {k} ^ {*} = Q _ {k} M _ {B} ^ {1 / 2} \left(M _ {B} ^ {1 / 2} Q _ {k} M _ {B} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {B} ^ {\dagger 1 / 2} B _ {k}. \tag {118}
$$
Remark F.9. Under the alternative assumption,  $\operatorname{Im}\{M_k\} \subseteq \operatorname{Im}\{Q_k\}$ , the optimal gain in (28) is given by
$$
\Pi_ {k} ^ {*} = Q _ {k} \tilde {B} M _ {b} ^ {1 / 2} \left(M _ {b} ^ {1 / 2} Q _ {b} M _ {b} ^ {1 / 2}\right) ^ {1 / 2 \dagger} M _ {b} ^ {\dagger 1 / 2} \tilde {B}, \tag {119}
$$
where  $\tilde{B} = B_k^{1/2}$ ,  $Q_b = \tilde{B}Q_k\tilde{B}$ ,  $M_b = \tilde{B}M_k\tilde{B}$ .
Proof. Recall our goal in (28) is to maximize  $\operatorname{Tr}\{\Pi MB\} = \operatorname{Tr}\left\{\tilde{B}\Pi M\tilde{B}\right\}$  under the condition  $Q - \Pi M\Pi^{\top} \succeq 0$  (we omit the index  $k$ ). This is equivalent to minimizing  $\mathbb{E}\left[\| \tilde{B}X - \tilde{B}Y\|^2\right]$  w.r.t  $\Pi$ , where  $(X,Y) \sim \mathcal{N}(0,\Sigma)$  and  $\Sigma = \begin{bmatrix} Q & \Pi M \\ M\Pi^{\top} & M \end{bmatrix} \succeq 0$ .
In this case,  $(\tilde{B} X,\tilde{B} Y)\sim \mathcal{N}(0,\Sigma_b)$  where  $\Sigma_{b} = \left[ \begin{array}{cc}\tilde{B} Q\tilde{B} & \tilde{B}\Pi M\tilde{B}\\ \tilde{B} M\Pi^{\top}\tilde{B} & \tilde{B} M\tilde{B} \end{array} \right]$ . According to Thm. F.2, under the assumption  $\operatorname {Im}\left\{\tilde{B} M\tilde{B}\right\} \subseteq \operatorname {Im}\left\{\tilde{B} Q\tilde{B}\right\}$ , the minimal distance is achieved when
$$
\tilde {B} \Pi M \tilde {B} = \tilde {B} Q \tilde {B} M _ {b} ^ {\frac {1}{2}} \left(M _ {b} ^ {\frac {1}{2}} Q _ {b} M _ {b} ^ {\frac {1}{2}}\right) ^ {\frac {1}{2} \dagger} M _ {b} ^ {\frac {1}{2}}. \tag {120}
$$
Note that  $\operatorname{Im}\{M\} \subseteq \operatorname{Im}\{Q\}$  implies  $\operatorname{Im}\left\{\tilde{B} M\tilde{B}\right\} \subseteq \operatorname{Im}\left\{\tilde{B} Q\tilde{B}\right\}$ , and it is straightforward to verify that  $Q - \Pi^{*}M\Pi^{*\top} \succeq 0$ .
G Stationary settings
A note is in place regarding the stationary perceptual Kalman filter. In the Kalman steady-state regime, where dynamics (31)-(32) are time-invariant and  $T \to \infty$ , the matrices  $K$  and  $S$  in Algorithm 2 are determined by the covariance matrix  $P$ ,
$$
K = P C ^ {\top} \left(C P C ^ {\top} + R\right) ^ {- 1}, \quad S = C P C ^ {\top} + R. \tag {121}
$$
Here,  $C$  stands for the time-invariant observation matrix  $(y_{k} = Cx_{k} + r_{k})$  and  $P$  is a solution to the Discrete-Time Algebraic Riccati equation (DARE)
$$
P = A P A ^ {\top} - A P C ^ {\top} \left(C P C ^ {\top} + R\right) ^ {- 1} C P A ^ {\top} + Q. \tag {122}
$$
Similarly, under the steady-state regime, (26) becomes
$$
\left\{\begin{array}{l l}\operatorname {T r} \{D \}&\rightarrow \min  _ {\Pi}\\\text {s . t .}&D = A D A ^ {\top} + Q + M - \Pi M - M \Pi^ {\top}, M = K S K ^ {\top}, Q - \Pi M \Pi^ {\top} \succeq 0\end{array}\right. \tag {123}
$$
where  $D$  obeys an (Algebraic) Lyapunov equation. If  $A$  is stable,
$$
D (\Pi) = \sum_ {k = 0} ^ {\infty} A ^ {k} (Q + M - \Pi M - M \Pi^ {\top}) \left(A ^ {k}\right) ^ {\top}. \tag {124}
$$
Hence, stationary perceptual filter is of the form
$$
\hat {x} _ {k} = A \hat {x} _ {k - 1} + J _ {k}, \tag {125}
$$
$$
J _ {k} = \Pi K \mathcal {I} _ {k} + w _ {k}, \tag {126}
$$
$$
w _ {k} \sim \mathcal {N} (0, Q - \Pi M \Pi^ {\top}), \tag {127}
$$
and in order to find optimal gain  $\Pi$ , minimizing  $\operatorname{Tr}\{D(\Pi)\}$ , we have to solve
$$
\max  _ {\Pi} \operatorname {T r} \left\{\Pi M B \right\} \text {s . t .} Q - \Pi M \Pi^ {\top} \succeq 0, \tag {128}
$$
where we define  $B \triangleq \sum_{k=0}^{\infty}\left(A^{k}\right)^{\top}A^{k}$ , and the solution (under the assumption  $\operatorname{Im}\{BMB\} \subseteq \operatorname{Im}\{Q\}$ ) is given again by (30).
H List of notations
We summarize our notations in the following Table.
Table 2: Definitions and Notations
Notation	Description	Definition	Dimensions
nx	state dimension
ny	measurement dimension
Ak	system dynamics	nx × nx
Ck	measurement function	ny × nx
Qk,Rk	noise covariances	nx × nx,ny × ny
xk	system state (ground-truth)	nx
yk	measurements	ny
x̂k	state estimator	nx
x̂k*	optimal Kalman state	see Algorithm 2	nx
x̂k|s	best MSE state estimators, up to time s	nx
Ik	innovation process	see Algorithm 2	ny
Sk	innovation covariance	see Algorithm 2	ny × ny
Kk	Kalman gain	see Algorithm 2	nx × ny
Πk	innovation perceptual gain	nx × nx
Mk	Kalman update covariance	Mk = KkSskKk†	nx × nx
vk	unutilized information process	see (15)	kny
Yk	unutilized information process (recursive)	see (19)	nx
ΣYk	unutilized information covariance	nx × nx
Φk	unutilized information perceptual gain	nx × nx
Bk	weight matrix	Bk = ∑t=kT αt(At-k)T A t-k	nx × nx
Dk	deviation from MMSE	Dk = E[ x̂k* - x̂k] [ x̂k* - x̂k]†	nx × nx
T	Termination time (horizon)
C(Ŷ0T)	minimization objective	C = ∑k=0T αkE [||xk - x̂k||2]
I Numerical demonstrations
In this section we provide full details for the experimental settings of Sec. 5, with additional numerical and visual results. In the following, we compare the performance of several filters;  $\hat{x}_{\mathrm{kal}}^*$  and  $\hat{x}_{\mathrm{tic}}$  correspond to the Kalman filter and the temporally-inconsistent filter (10) (which does not possess perfect-perceptual quality). The estimate  $\hat{x}_{\mathrm{opt}}$  is generated by a perfect-perception filter obtained by numerically optimizing the coefficients in (18), where the cost is the MSE at termination time, i.e. the terminal cost
$$
\mathcal {C} _ {\mathrm {T}} = \mathbb {E} \left[ \| \hat {x} _ {T} - x _ {T} \| ^ {2} \right]. \tag {129}
$$
The estimates  $\hat{x}_{\mathrm{auc}}, \hat{x}_{\mathrm{minT}}$  correspond to PKF outputs (Alg. 1) minimizing the total cost (area under curve)
$$
\mathcal {C} _ {\mathrm {a u c}} = \sum_ {k = 0} ^ {T} \mathbb {E} \left[ \| \hat {x} _ {k} - x _ {k} \| ^ {2} \right], \tag {130}
$$
and the terminal cost, respectively. Finally,  $\hat{x}_{\mathrm{stat.}}$  is the stationary PKF, discussed in App. G. The filters are summarized in Table 3.
Table 3: List of demonstrated filters.
description	definition	perfect-perception
per-sample	temporal
x*kal	Kalman filter	Algorithm 2	X	X
xtic	Per-sample perceptual quality (temporally-inconsistent)	(10)	✓	X
xopt	Optimized perfect-perceptual quality filter	(18)	✓	✓
xauc	PKF with total cost	Algorithm 1	✓	✓
xminT	PKF with terminal cost	Algorithm 1	✓	✓
xstat.	Stationary PKF	(125)	X	X
I.1 Example: Harmonic oscillator
We start with a simple 2-D example, where we demonstrate the differences in MSE distortion between the optimized perfect-perceptual quality filter  $\hat{x}_{\mathrm{opt}}$ , the temporally inconsistent filter  $\hat{x}_{\mathrm{tic}}$  and the efficient sub-optimal (perceptual) PKF. Consider the harmonic oscillator, where the entries of the state  $x_{k} \in \mathbb{R}^{2}$  correspond to position and velocity, and evolve as
$$
x _ {k + 1} = A x _ {k} + q _ {k}, \quad q _ {k} \sim \mathcal {N} (0, I) \tag {131}
$$
$$
A = I + \left[ \begin{array}{l l} 0 & 1 \\ - 2 & 0 \end{array} \right] \times \Delta_ {t}, \tag {132}
$$
where  $\Delta_t = 5 \times 10^{-3}$  is the sampling interval. Assume we have access to noisy and delayed scalar observations of the position (corresponding to time  $t - \frac{1}{2}\Delta_t$ ) so that  $y_k = \left[1 - \frac{1}{2}\right]x_k + r_k$ , where  $r_k \sim \mathcal{N}(0,1)$  and  $x_0 \sim \mathcal{N}(0,0.8I)$ .
We numerically optimize the coefficients  $\{\Pi_k, \varPhi_k\}_{k=0}^T$  in (18), to minimize the terminal error (129) (Tr  $\{D_T\}$  in (23)) at time  $T = 255$  under the constraints (20). Figure 7 shows the MSE distortion for the optimized perfect-perception filter  $\hat{x}_{\mathrm{opt}}$  defined by (18) and  $\{\Pi_k, \varPhi_k\}_{k=0}^T$ , and the sub-optimal PKF outputs  $\hat{x}_{\mathrm{auc}}, \hat{x}_{\mathrm{minT}}$ , minimizing the total cost (130) and the terminal cost (129) (see Table 3). We observe that PKF estimations are indeed not MSE optimal at time  $T$ . However, their RMSE at time  $T$  is only  $\sim 30\%$  higher than that of  $\hat{x}_{\mathrm{opt}}$  and they have the advantage that they can be solved analytically and require computing only half of the coefficients  $(\Pi_k)$ .
The estimates  $\hat{x}_{\mathrm{kal}}^{*}$  and  $\hat{x}_{\mathrm{tic}}$  achieve lower MSE than  $\hat{x}_{\mathrm{opt}}$ , however they do not possess perfect-perceptual quality. We can see the difference in MSE distortion between the filters  $\hat{x}_{\mathrm{opt}}$  and  $\hat{x}_{\mathrm{tic}}$ , with and without perception constraint in the temporal domain. This is the cost of temporal consistency in online estimation for this setting.