SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dror Freirich
Technion - Israel Institute of Technology drorfrc@gmail.com
Technion - Israel Institute of Technology tomer.m@ee.technion.ac.il
Ron Meir  
Technion - Israel  
Institute of Technology  
rmeir@ee.technion.ac.il
Many practical settings call for the reconstruction of temporal signals from corrupted or missing data. Classic examples include decoding, tracking, signal enhancement and denoising. Since the reconstructed signals are ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception. Mathematically, perfect perceptual-quality is achieved when the distribution of restored signals is the same as that of natural signals, a requirement which has been heavily researched in static estimation settings (i.e. when a whole signal is processed at once). Here, we study the problem of optimal causal filtering under a perfect perceptual-quality constraint, which is a task of fundamentally different nature. Specifically, we analyze a Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to "knowingly" ignore new information revealed by the observations in order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.
In many settings, it is desired to reconstruct a temporal signal from corrupted or missing data. Examples include decoding of transmitted communications, tracking targets based on noisy measurements, enhancing audio signals, and denoising videos. Traditionally, restoration quality has been assessed by distortion measures such as MSE. As a result, numerous methods targeted the minimization of such measures, including the seminal work of Kalman [11]. However, in applications involving human perception, one may favor reconstructions that cannot be told apart from valid signals. Mathematically, such perfect perceptual quality can be achieved only if the distribution of restored signals is the same as that of "natural" signals.
Interestingly, it has been shown that good perceptual quality generally comes at the price of poor distortion and vice versa. This phenomenon, known as the perception-distortion tradeoff, was first studied in [3], and was later fully characterized in [8] for the particular setting where distortion is measured by MSE and perception is measured by the Wasserstein-2 distance between the distribution of estimated signals and the distribution of real signals. However, to date, all existing works addressed the static (non-temporal) setting, in which the entire corrupted signal is available for processing all at
once. This setting is fundamentally different from situations involving temporal signals, in which the corrupted signal is processed causally over time, such that each sample is reconstructed only based on observations up to the current time.
To illustrate the inherent difficulty in causal estimation, consider video restoration tasks like denoising, super-resolution, or frame completion (see Fig. 1). Achieving high perceptual quality in those tasks requires generating restored videos whose spatio-temporal distribution matches that of natural videos. Particularly, an incorrect temporal distribution may lead to flickering artifacts [14] or to unnaturally slow dynamics [4]. To comply with this requirement, the restoration method needs to 'hallucinate' motion whenever the dynamics cannot be accurately determined from the measurements. For example, it may be impossible to determine whether a car is standing still or moving slowly from just a few noisy frames, yet the restoration method must generate some (nearly) constant velocity in order to comply with the statistics of natural videos. However, as more measurements become avail-
able, the uncertainty may be reduced, and it may become evident that the hallucinated dynamics were in fact incorrect. When this happens, the method cannot suddenly change the motion in the output video, because such an abrupt change would deviate from natural video statistics. Thus, although the method "becomes aware" of its mistake, it may have to stick to its past decisions for a while. A natural question is, therefore:
Figure 1: The temporal consistency dilemma. Estimation cannot suddenly change the motion in the output video, because such an abrupt change would deviate from natural video statistics. Thus, although the method is aware of its mistake, it may have to stick to its past decisions.
What is the precise cost of temporal consistency in online restoration?
In this paper, we study this question in the setting where the signal to be restored,  $x_{t}$ , is a discrete-time Gaussian Markov process, and the measurements  $y_{t}$  are noisy linear transformations of the signal's state. We address the problem of designing a causal filter for estimating  $x_{t}$  from  $y_{t}$ , where the distribution law of the filter's output,  $\hat{x}_{t}$ , is constrained to be the same as that of  $x_{t}$  (perfect perceptual quality). We show that this temporal consistency constraint indeed comes at the cost of increased MSE compared to filters that only enforce the correct distribution per time step, but not joint distributions across time steps. To derive a recursive form for linear perceptual filters, we introduce the novel concept of an unutilized information process, which is the portion of accumulated information that does not depend on past estimates. We provide a closed-form expression for the MSE of such filters and show how to design their coefficients to minimize different objectives. We further establish a special class of perceptual filters, based on the classic innovation process, which has an explicit solution. We analyze the evolution of MSE over time for perceptual filters and for non-perceptual ones in several numerical setups. Finally, we demonstrate the qualitative effects of perfect perceptual-quality estimation on a simplified video reconstruction problem.
Related work Many works proposed practical algorithms for achieving high (spatio-temporal) perceptual quality in video restoration tasks. Bhattacharjee and Das [1] improved temporal coherence by using a loss that penalizes discontinuities between sequential frames. Pérez-Pellitero et al. [14] suggested a recurrent generator architecture whose inputs include the low-resolution current frame (at time  $t$ ), the high-resolution reconstruction of the previous one (at time  $t - 1$ ), and a low-resolution version of the previous frame, aligned to the current one. The model is trained using losses that encourage it to conserve the joint statistics between consecutive frames. Chu et al. [4] introduced a temporally coherent GAN architecture (TecoGAN). Their generator's input includes again a warped version of the previously generated frame, where each discriminator input consists of 3 consecutive frames, either generated or ground-truth. They also introduced a bi-directional loss that encourages long-term consistency by avoiding temporal accumulation of artifacts, and another loss which measures the similarity between motions. More recent progress in generating temporally coherent videos includes Make-a-video [16] that expands a text-to image model with spatio-temporal convolutional and attention layers, and Video-Diffusion Models [10] that use a
diffusion model with a network architecture adapted to video. In the context of online restoration, we mention the work of Kim et al. [12] which presented a GAN architecture for real-time video deblurring, where restoration is done sequentially. They introduce a network layer for dynamically (at test-time) blending features between consecutive frames. This mechanism enables generated features to propagate into future frames, thus improving consistency. We note, however, that our work is the first to provide a theoretical/mathematical framework, and a closed-form solution for a special case.
2 Preliminaries: The distortion-perception tradeoff
Let  $x, y$  be random vectors taking values in  $\mathbb{R}^{n_x}$  and  $\mathbb{R}^{n_y}$ , respectively, with joint probability  $p_{xy}$ . Suppose we want to estimate  $x$  based on  $y$ , such that the estimator  $\hat{x}$  satisfies two requirements: (i) It has a low distortion  $\mathbb{E}[d(x, \hat{x})]$ , where  $d(\cdot, \cdot)$  is some measure of discrepancy between signals; (ii) It has a good perceptual quality, i.e. it achieves a low value of  $d_p(p_x, p_{\hat{x}})$ , where  $d_p(\cdot, \cdot)$  is a divergence between probability measures. Blau and Michaeli [3] studied the best possible distortion that can be achieved under a given level of perceptual quality, by introducing the distortion-perception function
$$
D (P) = \min  _ {p _ {\hat {x} | y}} \left\{\mathbb {E} \left[ d \left(x, \hat {x}\right) \right]: d _ {p} \left(p _ {x}, p _ {\hat {x}}\right) \leq P \right\}. \tag {1}
$$
Freirich et al. [8] provided a complete characterization of  $D(P)$  for the case where  $d$  is the squared-error and  $d_p$  is the Wasserstein-2 distance. Particularly, in the Gaussian case, they developed a closed-form expression for the optimal estimator.
In this paper we discuss estimation with perfect perceptual quality, namely  $P = 0$ . In this case, [8, Thm. 4] implies that if  $x$  and  $y$  are zero-mean, jointly-Gaussian with covariances  $\Sigma_x, \Sigma_y \succ 0$ , and  $x^* = \mathbb{E}[x|y]$ , then a MSE-optimal perfect perceptual-quality estimator is obtained by
$$
\hat {x} = \mathcal {T} ^ {*} x ^ {*} + w, \quad \mathcal {T} ^ {*} \triangleq \Sigma_ {x} ^ {\frac {1}{2}} \left(\Sigma_ {x} ^ {\frac {1}{2}} \Sigma_ {x ^ {*}} \Sigma_ {x} ^ {\frac {1}{2}}\right) ^ {\frac {1}{2}} \Sigma_ {x} ^ {- \frac {1}{2}} \Sigma_ {x ^ {*}} ^ {\dagger}, \tag {2}
$$
where  $w$  is a zero-mean Gaussian noise with covariance  $\Sigma_w = \Sigma_x - \mathcal{T}^*\Sigma_{x^*}\mathcal{T}^{*\top}$ , independent of  $y$  and  $x$ , and  $\Sigma_{x^*}^\dagger$  is the Moore-Penrose inverse of  $\Sigma_{x^*}$ . For the more general case where  $\Sigma_x\succeq 0$ , a similar result can be obtained by using Theorem F.2 in the Appendix.
We consider a state  $x_{k} \in \mathbb{R}^{n_{x}}$  with linear dynamics driven by Gaussian noise, and observations  $y_{k} \in \mathbb{R}^{n_{y}}$  that are linear transformations of  $x_{k}$  perturbed by Gaussian noise,
$$
x _ {k} = A _ {k} x _ {k - 1} + q _ {k}, \quad k = 1, \dots , T, \tag {3}
$$
$$
y _ {k} = C _ {k} x _ {k} + r _ {k}, \quad k = 0, \dots , T. \tag {4}
$$
Here, the noise vectors  $q_{k} \sim \mathcal{N}(0, Q_{k})$  and  $r_{k} \sim \mathcal{N}(0, R_{k})$  are independent white Gaussian processes, and  $x_{0} \sim \mathcal{N}(0, P_{0})$  is independent of  $q_{1}, r_{0}$ . For convenience, we will sometimes refer to  $P_{0}$  as  $Q_{0}$ . The matrices  $A_{k}, C_{k}, Q_{k}, R_{k}$  and  $P_{0}$  are deterministic system parameters with appropriate dimensions, and assumed to be known.
Our goal is to construct an estimated sequence  $\hat{X}_0^T = (\hat{x}_0, \dots, \hat{x}_T)$  based on the measurements  $Y_0^T = (y_0, \dots, y_T)$ , which minimizes the cost
$$
\mathcal {C} \left(\hat {x} _ {0}, \dots , \hat {x} _ {T}\right) = \sum_ {k = 0} ^ {T} \alpha_ {k} \mathbb {E} \left[ \left\| x _ {k} - \hat {x} _ {k} \right\| ^ {2} \right], \tag {5}
$$
for some given weights  $\alpha_{k} \geq 0$ . Importantly, we want to do so under the following two constraints.
Temporal causality :  $\hat{x}_k\sim p_{\hat{x}_k}(\cdot |y_0,\ldots ,y_k,\hat{x}_0,\ldots ,\hat{x}_{k - 1}),$  (6)
Perfect perceptual quality:  $p_{\hat{X}_0^T} = p_{X_0^T}$  (7)
Condition (6) states that each prediction should depend only on past and present observations and on the past predictions. Note that Condition (7) requires not only that every estimated sample have the same distribution as the original one, but also that the joint distribution of every subset of reconstructed samples be identical to that of the corresponding subset of samples in the original sequence. In the context of video processing, this means that not only does every recovered frame have to look natural, but also that motion must look natural. This perfect perceptual quality constraint is what sets our problem apart from the classical Kalman filtering problem, which considers only the causality constraint. Since we will make use of the Kalman filter, let us briefly summarize it.
The Kalman filter (no perceptual quality constraint) Let  $\hat{x}_{k|s}^{*} \triangleq \mathbb{E}[x_{k}|y_{0},\ldots ,y_{s}]$  denote the estimator of  $x_{k}$  based on all observations up to time  $s$ , which minimizes the MSE. The celebrated Kalman filter [11] is an efficient method for calculating the Kalman optimal state  $\hat{x}_k^*\equiv \hat{x}_{k|k}^*$  recursively without having to store all observations up to time  $k$ . It is given by the recurrence
$$
\hat {x} _ {k} ^ {*} = A _ {k} \hat {x} _ {k - 1} ^ {*} + K _ {k} \mathcal {I} _ {k}, \tag {8}
$$
where  $K_{k}$  is the optimal Kalman gain [11], whose recursive calculation is given in Algo. 2 in the Appendix. The vector  $\mathcal{I}_k$  is the innovation process,
$$
\mathcal {I} _ {k} = y _ {k} - C _ {k} \hat {x} _ {k | k - 1} ^ {*}, \tag {9}
$$
describing the new information carried by the observation  $y_{k}$  over the optimal prediction based on the observations up to time  $k - 1$ , which is given by  $\hat{x}_{k|k - 1}^{*} = A_{k}\hat{x}_{k - 1}^{*}$ . The innovation  $\mathcal{I}_k$  is uncorrelated with all observations up to time  $k - 1$ , which guarantees the MSE optimality of the estimation. The calculation of the Kalman state is also summarized in Alg. 2. Pay attention to the innovation process  $\mathcal{I}_k$ , its covariance  $S_{k}$  and gain  $K_{k}$ , which we will build upon. Our notations are summarized in Table 2 in the Appendix. Note that since the Kalman filter minimizes the MSE at each timestep, it also minimizes (5) regardless of the choice of  $\alpha_{k}$ , but it generally fails to fulfill (7). As we will see later, when taking (7) into consideration, the choice of  $\alpha_{k}$  does affect the optimal filter.
Temporally-inconsistent perceptual filter A naive way to try to improve the perceptual quality of the Kalman filter would be to require that each  $\hat{x}_k$  be distributed like  $x_k$  (but without constraining the joint distribution of samples). In the context of video processing, each frame generated by such a filter would look natural, but motion would not necessarily look natural. This problem can be solved using the result (2), which gives the optimal "temporally-inconsistent" perceptual estimator
$$
\hat {x} _ {k} ^ {\mathrm {t i c}} = \mathcal {T} _ {k} ^ {*} \hat {x} _ {k} ^ {*} + w _ {k} = \mathcal {T} _ {k} ^ {*} \left(A _ {k} \hat {x} _ {k - 1} ^ {*} + K _ {k} \mathcal {I} _ {k}\right) + w _ {k}, \tag {10}
$$
with  $\mathcal{T}_k^*$  and  $w_{k}$  from (2). These quantities depend only on the covariances of  $x_{k},\hat{x}_{k}^{*}$ , which can be computed recursively using the Kalman method. The MSE of this estimator is given by (see [8])
$$
\mathbb {E} \left[ \| x _ {k} - \hat {x} _ {k} \| ^ {2} \right] = d _ {k} ^ {*} + \operatorname {T r} \left\{\Sigma_ {x _ {k}} + \Sigma_ {\hat {x} _ {k} ^ {*}} - 2 \left(\Sigma_ {x _ {k}} ^ {\frac {1}{2}} \Sigma_ {\hat {x} _ {k} ^ {*}} \Sigma_ {x _ {k}} ^ {\frac {1}{2}}\right) ^ {\frac {1}{2}} \right\}, \tag {11}
$$
where  $d_k^*$  is the MSE of the Kalman filter, which can also be computed recursively.
Our setting (with the perceptual quality constraint) Going back to our setting, one may readily recognize that perceptually reconstructing the signal  $X_0^T$  from the full measurement sequence  $Y_0^T$  is also a special case of the Gaussian perceptual restoration problem discussed in Section 2, only applied to the entire sequence of states and measurements. Generally, this estimate already achieves a higher MSE than the estimate that minimizes the MSE without the perceptual constraint. However, in our setting we have the additional causality constraint (6). Requiring both constraints (7) and (6) might incur an additional cost, as illustrated by the following example, where applying each one of them does not restrict the optimal solution, but together they result in a higher MSE.
Example 3.1. Let  $T = 1$  and consider the process  $(x_0, x_1) = (q_0, q_0)$ , where  $q_0 \sim \mathcal{N}(0, 1)$  with observations  $(y_0, y_1) = (0, x_1)$ . Assume we want to minimize the error at time  $k = 1$  (namely  $(\alpha_0, \alpha_1) = (0, 1)$  in (5)). Then, considering only the causality constraint (6), the estimator  $(\hat{x}_0, \hat{x}_1) = (y_0, y_1)$  is optimal. Indeed, it is causal and it achieves zero MSE. Similarly, considering only the perceptual quality constraint (7), the estimator  $(\hat{x}_0, \hat{x}_1) = (y_1, y_1)$  is optimal. Indeed, it is distributed like  $(x_0, x_1)$  and it also achieves zero MSE. However, when demanding both conditions,  $\hat{x}_0$  must be based on no information to obey (6) (as  $y_0 = 0$ ), and it must be drawn from the prior distribution  $\mathcal{N}(0, 1)$  in order to be distributed like  $x_0$  and obey (7). Furthermore, to satisfy (7), we must also have  $\hat{x}_1 = \hat{x}_0$ . Therefore, the optimal estimator in this case is  $(\hat{x}_0, \hat{x}_1) = (\tilde{q}_0, \tilde{q}_0)$ , where  $\tilde{q}_0 \sim \mathcal{N}(0, 1)$  is independent of  $q_0$ . The MSE achieved by this estimator is 2.
4 Perfect perceptual-quality filters
The perceptual constraint (7) dictates that the estimator must be of the form
$$
\hat {x} _ {k} = A _ {k} \hat {x} _ {k - 1} + J _ {k}, \quad \hat {x} _ {0} = J _ {0}, \tag {12}
$$
where  $J_{k} = \hat{x}_{k} - A_{k}\hat{x}_{k - 1}$  is distributed as  $\mathcal{N}(0,Q_k)$  and is independent of  $\hat{X}_0^{k - 1}$ . Note the similarity between (12) and the MSE-optimal state (8), in which  $J_{k} = K_{k}\mathcal{T}_{k}$ . Here, however, this choice is not valid due to the constraint on the output distribution. In terms of temporal consistency, an estimator of the form (12) guarantees that previously presented features obey the natural dynamics of the domain, while newly generated estimates do not contradict the previous ones. In order to maintain causality (6),  $J_{k}$  must be of the form
$$
J _ {k} \sim p _ {J _ {k}} (\cdot | y _ {0}, \dots , y _ {k}, \hat {x} _ {0}, \dots , \hat {x} _ {k - 1}), \tag {13}
$$
i.e.,  $J_{k}$  is independent of future observations  $Y_{k + 1}^{T}$  given  $Y_0^k$ . As a consequence,  $J_{k}$  is uncorrelated with  $\mathcal{I}_{k + n}$  for all  $n\geq 1$ .
We now discuss linear estimators, where  $Y_0^T$  and  $J_0^T$  (hence  $\hat{X}_0^T$ ) are jointly Gaussian. Our first result is as follows (see proof in App. B).
Theorem 4.1. Under the cost (5), there exists a linear optimal estimator of the form
$$
J _ {k} = \pi_ {k} \mathcal {I} _ {k} + \phi_ {k} v _ {k} + w _ {k}, \tag {14}
$$
$\pi_k \in \mathbb{R}^{n_x \times n_y}$  and  $\phi_k \in \mathbb{R}^{n_x \times (kn_y)}$  are the filter's coefficients,  $w_k$  is an independent Gaussian noise with covariance  $\Sigma_{w_k} = Q_k - \pi_k S_k \pi_k^\top - \phi_k \Sigma_{v_k} \phi_k^\top \succeq 0$ ,  $v_k$  is the process of unutilized information
$$
v _ {k} \triangleq \mathcal {I} _ {0} ^ {k - 1} - \mathbb {E} \left[ \mathcal {I} _ {0} ^ {k - 1} \mid \hat {X} _ {0} ^ {k - 1} \right], \mathcal {I} _ {0} ^ {k - 1} = \operatorname {C o l u m n} \left\{\mathcal {I} _ {0}, \dots , \mathcal {I} _ {k - 1} \right\}, v _ {0} = 0. \tag {15}
$$
Note that with this form for  $J_{k}$ , the state  $\hat{x}_k$  is indeed a function of the observations  $Y_0^k$  and the previous states  $\hat{X}_0^{k-1}$ . Intuitively,  $v_k$  is the part of the information in the observations, which has no correlation with the information used to construct the past estimates  $\hat{X}_0^{k-1}$ . Thus, from the standpoint of the filter's output, this information has not yet been introduced. As opposed to the innovation  $\mathcal{I}_k$ , the process  $v_k$  is not white, and it is affected by the choices of  $\pi_t$  and  $\phi_t$  up to time  $k-1$ . However,  $\mathcal{I}_k$  is always independent of  $v_k$ , since  $\mathcal{I}_k$  is independent of  $\mathcal{I}_0^{k-1}$  and  $J_0^{k-1}$ , which constitute  $v_k$ .
The filter of Theorem 4.1 is causal but not recursive. Specifically, although it is possible to obtain  $v_{k+1}, \Sigma_{v_{k+1}}$  given the coefficients  $\{\pi_t, \phi_t\}_{t=0}^k$  (see App. E), the dimension of  $v_k$  grows with time (it is  $kn_y$ ), thus increasing the cost of computing  $\phi_k v_k$ . Furthermore, determining the coefficients  $\{\pi_k, \phi_k\}_{k=0}^T$  that minimize the objective (5) (which is done a-priori in an offline manner) requires solving a large optimization problem, as the total size of all coefficients is  $\mathcal{O}(n_x n_y T^2)$ . To efficiently optimize these coefficients, we next suggest two simplified versions of this form, which may generally be sub-optimal but easier to optimize.
Remark 4.2. A remark is in place regarding objectives beyond the
Figure 2: Recursive perceptual filtering (Sec. 4.1). The state estimator  $\hat{x}_k$  consists of the previous state  $\hat{x}_{k - 1}$ , and the innovation and unutilized information processes. The unutilized information state  $\Upsilon_{k + 1}$  is updated using the previously unutilized information  $\Upsilon_{k}$  and the newly-arriving information  $\mathcal{I}_k$ . The currently utilized information, arriving from  $J_{k}$ , is then subtracted from  $\Upsilon_{k + 1}$ .
squared-error. While (14) forms an optimal filter under the cost (5), it can be considered as a representation for linear filters in general. The constraints on the coefficients  $(Q_{k} - \pi_{k}S_{k}\pi_{k}^{\top} - \phi_{k}\Sigma_{v_{k}}\phi_{k}^{\top}\succeq 0)$  are necessary and sufficient for perfect perception, regardless of the cost objective. It is therefore possible to optimize coefficients for objectives other than MSE under these constraints to obtain optimal perceptual linear filters. Optimization may be performed numerically (when the cost is tractable) or in an online fashion, assuming access to ground-truth samples. Note, however, that under general distortion measures, optimal filters might be non-linear.
4.1 Recursive-form filters
The optimal Kalman state  $\hat{x}_k^*$  achieves the minimal possible MSE, given by  $d_k^* = \mathbb{E}\left[\| \hat{x}_k^* -x_k\|^2\right] = \mathrm{Tr}\left\{P_{k|k}\right\}$ , where  $P_{k|k}$  is the error covariance, given explicitly in Alg. 2. By the orthogonality principle (see e.g. [8, Lemma 2]), any other estimator  $\hat{x}_k$  based on the observations  $Y_0^k$ , satisfies
$$
\mathbb {E} \left[ \| x _ {k} - \hat {x} _ {k} \| ^ {2} \right] = d _ {k} ^ {*} + \mathbb {E} \left[ \| \hat {x} _ {k} - \hat {x} _ {k} ^ {*} \| ^ {2} \right]. \tag {16}
$$
Now, consider an estimator  $\hat{x}_k$  of the form (12), and let  $D_k \triangleq \mathbb{E}[(\hat{x}_k^* - \hat{x}_k)(\hat{x}_k^* - \hat{x}_k)^\top]$ . Since we choose  $J_k$  to be normally distributed and independent of  $\hat{x}_{k-1}$ , it is easy to see that  $D_k$  obeys the Lyapunov difference equation
$$
\begin{array}{l} D _ {k} = A _ {k} D _ {k - 1} A _ {k} ^ {\top} + K _ {k} S _ {k} K _ {k} ^ {\top} + Q _ {k} \\ - \mathbb {E} \left[ J _ {k} \mathcal {I} _ {k} ^ {\top} \right] K _ {k} ^ {\top} - K _ {k} \mathbb {E} \left[ \mathcal {I} _ {k} J _ {k} ^ {\top} \right] - A _ {k} \mathbb {E} \left[ \hat {x} _ {k - 1} ^ {*} J _ {k} ^ {\top} \right] - \mathbb {E} \left[ J _ {k} \hat {x} _ {k - 1} ^ {* \top} \right] A _ {k} ^ {\top}. \tag {17} \\ \end{array}
$$