Title: Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint
Venue: NeurIPS 2023
Accepted: yes
Keywords: Kalman filter

estimation theory

causal filtering

signal processing

distortion-perception tradeoff
Abstract: Many practical settings call for the reconstruction of temporal signals from corrupted or missing data. Classic examples include decoding, tracking, signal enhancement and denoising. Since the reconstructed signals are ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception.
Mathematically, perfect perceptual-quality is achieved when the distribution of restored signals is the same as that of natural signals, a requirement which has been heavily researched in static estimation settings (i.e. when a whole signal is processed at once). 
Here, we study the problem of optimal causal  filtering under a perfect perceptual-quality constraint, which is a task of fundamentally different nature.  
Specifically, we analyze a Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to ``knowingly'' ignore new information revealed by the observations in order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.

Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint

Abstract

Many practical settings call for the reconstruction of temporal signals from corrupted or missing data. Classic examples include decoding, tracking, signal enhancement and denoising. Since the reconstructed signals are ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception. Mathematically, perfect perceptual-quality is achieved when the distribution of restored signals is the same as that of natural signals, a requirement which has been heavily researched in static estimation settings (i.e. when a whole signal is processed at once). Here, we study the problem of optimal causal filtering under a perfect perceptual-quality constraint, which is a task of fundamentally different nature. Specifically, we analyze a Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to 'knowingly' ignore new information revealed by the observations in order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.

1 Introduction

In many settings, it is desired to reconstruct a temporal signal from corrupted or missing data. Examples include decoding of transmitted communications, tracking targets based on noisy measurements, enhancing audio signals, and denoising videos. Traditionally, restoration quality has been assessed by distortion measures such as MSE. As a result, numerous methods targeted the minimization of such measures, including the seminal work of Kalman [11]. However, in applications involving human perception, one may favor reconstructions that cannot be told apart from valid signals. Mathematically, such perfect perceptual quality can be achieved only if the distribution of restored signals is the same as that of 'natural' signals.

Interestingly, it has been shown that good perceptual quality generally comes at the price of poor distortion and vice versa. This phenomenon, known as the perception-distortion tradeoff , was first studied in [3], and was later fully characterized in [8] for the particular setting where distortion is measured by MSE and perception is measured by the Wasserstein2 distance between the distribution of estimated signals and the distribution of real signals. However, to date, all existing works addressed the static (non-temporal) setting, in which the entire corrupted signal is available for processing all at

once. This setting is fundamentally different from situations involving temporal signals, in which the corrupted signal is processed causally over time, such that each sample is reconstructed only based on observations up to the current time.

To illustrate the inherent difficulty in causal estimation, consider video restoration tasks like denoising, super-resolution, or frame completion (see Fig. 1). Achieving high perceptual quality in those tasks requires generating restored videos whose spatio-temporal distribution matches that of natural videos. Particularly, an incorrect temporal distribution may lead to flickering artifacts [14] or to unnaturally slow dynamics [4]. To comply with this requirement, the restoration method needs to 'hallucinate' motion whenever the dynamics cannot be accurately determined from the measurements. For example, it may be impossible to determine whether a car is standing still or moving slowly from just a few noisy frames, yet the restoration method must generate some (nearly) constant velocity in order to comply with the statistics of natural videos. However, as more measurements become avail-

missing

observation

estimation

t=1

t=2

t=3

able, the uncertainty may be reduced, and it may become evident that the hallucinated dynamics were in fact incorrect. When this happens, the method cannot suddenly change the motion in the output video, because such an abrupt change would deviate from natural video statistics. Thus, although the method 'becomes aware' of its mistake, it may have to stick to its past decisions for a while. A natural question is, therefore:

What is the precise cost of temporal consistency in online restoration?

In this paper, we study this question in the setting where the signal to be restored, x t , is a discretetime Gaussian Markov process, and the measurements y t are noisy linear transformations of the signal's state. We address the problem of designing a causal filter for estimating x t from y t , where the distribution law of the filter's output, ˆ x t , is constrained to be the same as that of x t (perfect perceptual quality). We show that this temporal consistency constraint indeed comes at the cost of increased MSE compared to filters that only enforce the correct distribution per time step, but not joint distributions across time steps. To derive a recursive form for linear perceptual filters, we introduce the novel concept of an unutilized information process, which is the portion of accumulated information that does not depend on past estimates. We provide a closed-form expression for the MSE of such filters and show how to design their coefficients to minimize different objectives. We further establish a special class of perceptual filters, based on the classic innovation process, which has an explicit solution. We analyze the evolution of MSE over time for perceptual filters and for non-perceptual ones in several numerical setups. Finally, we demonstrate the qualitative effects of perfect perceptual-quality estimation on a simplified video reconstruction problem.

Related work Many works proposed practical algorithms for achieving high (spatio-temporal) perceptual quality in video restoration tasks. Bhattacharjee and Das [1] improved temporal coherence by using a loss that penalizes discontinuities between sequential frames. Pérez-Pellitero et al. [14] suggested a recurrent generator architecture whose inputs include the low-resolution current frame (at time t ), the high-resolution reconstruction of the previous one (at time t -1 ), and a low-resolution version of the previous frame, aligned to the current one. The model is trained using losses that encourage it to conserve the joint statistics between consecutive frames. Chu et al. [4] introduced a temporally coherent GAN architecture (TecoGAN). Their generator's input includes again a warped version of the previously generated frame, where each discriminator input consists of 3 consecutive frames, either generated or ground-truth. They also introduced a bi-directional loss that encourages long-term consistency by avoiding temporal accumulation of artifacts, and another loss which measures the similarity between motions. More recent progress in generating temporally coherent videos includes Make-a-video [16] that expands a text-to image model with spatio-temporal convolutional and attention layers, and Video-Diffusion Models [10] that use a

diffusion model with a network architecture adapted to video. In the context of online restoration, we mention the work of Kim et al. [12] which presented a GAN architecture for real-time video deblurring, where restoration is done sequentially. They introduce a network layer for dynamically (at test-time) blending features between consecutive frames. This mechanism enables generated features to propagate into future frames, thus improving consistency. We note, however, that our work is the first to provide a theoretical/mathematical framework, and a closed-form solution for a special case.

2 Preliminaries: The distortion-perception tradeoff

Let x, y be random vectors taking values in R n x and R n y , respectively, with joint probability p xy . Suppose we want to estimate x based on y , such that the estimator ˆ x satisfies two requirements: (i) It has a low distortion E [ d ( x, ˆ x )] , where d ( · , · ) is some measure of discrepancy between signals; (ii) It has a good perceptual quality, i.e. it achieves a low value of d p ( p x , p ˆ x ) , where d p ( · , · ) is a divergence between probability measures. Blau and Michaeli [3] studied the best possible distortion that can be achieved under a given level of perceptual quality, by introducing the distortion-perception function

Freirich et al. [8] provided a complete characterization of D ( P ) for the case where d is the squarederror and d p is the Wasserstein2 distance. Particularly, in the Gaussian case, they developed a closed-form expression for the optimal estimator.

In this paper we discuss estimation with perfect perceptual quality, namely P = 0 . In this case, [8, Thm. 4] implies that if x and y are zero-mean, jointly-Gaussian with covariances Σ x , Σ y ≻ 0 , and x ∗ = E [ x | y ] , then a MSE-optimal perfect perceptual-quality estimator is obtained by

where w is a zero-mean Gaussian noise with covariance Σ w = Σ x -T ∗ Σ x ∗ T ∗⊤ , independent of y and x , and Σ † x ∗ is the Moore-Penrose inverse of Σ x ∗ . For the more general case where Σ x ⪰ 0 , a similar result can be obtained by using Theorem F.2 in the Appendix.

3 Problem formulation

We consider a state x k ∈ R n x with linear dynamics driven by Gaussian noise, and observations y k ∈ R n y that are linear transformations of x k perturbed by Gaussian noise,

Here, the noise vectors q k ∼ N (0 , Q k ) and r k ∼ N (0 , R k ) are independent white Gaussian processes, and x 0 ∼ N (0 , P 0 ) is independent of q 1 , r 0 . For convenience, we will sometimes refer to P 0 as Q 0 . The matrices A k , C k , Q k , R k and P 0 are deterministic system parameters with appropriate dimensions, and assumed to be known.

Our goal is to construct an estimated sequence ˆ X T 0 = (ˆ x 0 , . . . , ˆ x T ) based on the measurements Y T 0 = ( y 0 , . . . , y T ) , which minimizes the cost

for some given weights α k ≥ 0 . Importantly, we want to do so under the following two constraints.

Condition (6) states that each prediction should depend only on past and present observations and on the past predictions. Note that Condition (7) requires not only that every estimated sample have the same distribution as the original one, but also that the joint distribution of every subset of reconstructed samples be identical to that of the corresponding subset of samples in the original sequence. In the context of video processing, this means that not only does every recovered frame have to look natural, but also that motion must look natural. This perfect perceptual quality constraint is what sets our problem apart from the classical Kalman filtering problem, which considers only the causality constraint. Since we will make use of the Kalman filter, let us briefly summarize it.

The Kalman filter (no perceptual quality constraint) Let ˆ x ∗ k | s ≜ E [ x k | y 0 , . . . , y s ] denote the estimator of x k based on all observations up to time s , which minimizes the MSE. The celebrated Kalman filter [11] is an efficient method for calculating the Kalman optimal state ˆ x ∗ k ≡ ˆ x ∗ k | k recursively without having to store all observations up to time k . It is given by the recurrence

where K k is the optimal Kalman gain [11], whose recursive calculation is given in Algo. 2 in the Appendix. The vector I k is the innovation process,

describing the new information carried by the observation y k over the optimal prediction based on the observations up to time k -1 , which is given by ˆ x ∗ k | k -1 = A k ˆ x ∗ k -1 . The innovation I k is uncorrelated with all observations up to time k -1 , which guaranties the MSE optimality of the estimation. The calculation of the Kalman state is also summarized in Alg. 2. Pay attention to the innovation process I k , its covariance S k and gain K k , which we will build upon. Our notations are summarized in Table 2 in the Appendix. Note that since the Kalman filter minimizes the MSE at each timestep, it also minimizes (5) regardless of the choice of α k , but it generally fails to fulfill (7). As we will see later, when taking (7) into consideration, the choice of α k does affect the optimal filter.

Temporally-inconsistent perceptual filter A naive way to try to improve the perceptual quality of the Kalman filter would be to require that each ˆ x k be distributed like x k (but without constraining the joint distribution of samples). In the context of video processing, each frame generated by such a filter would look natural, but motion would not necessarily look natural. This problem can be solved using the result (2), which gives the optimal 'temporally-inconsistent' perceptual estimator

with T ∗ k and w k from (2). These quantities depend only on the covariances of x k , ˆ x ∗ k , which can be computed recursively using the Kalman method. The MSE of this estimator is given by (see [8])

where d ∗ k is the MSE of the Kalman filter, which can also be computed recursively.

Our setting (with the perceptual quality constraint) Going back to our setting, one may readily recognize that perceptually reconstructing the signal X T 0 from the full measurement sequence Y T 0 is also a special case of the Gaussian perceptual restoration problem discussed in Section 2, only applied to the entire sequence of states and measurements. Generally, this estimate already achieves a higher MSE than the estimate that minimizes the MSE without the perceptual constraint. However, in our setting we have the additional causality constraint (6). Requiring both constraints (7) and (6) might incur an additional cost, as illustrated by the following example, where applying each one of them does not restrict the optimal solution, but together they result in a higher MSE.

Example 3.1 . Let T = 1 and consider the process ( x 0 , x 1 ) = ( q 0 , q 0 ) , where q 0 ∼ N (0 , 1) , with observations ( y 0 , y 1 ) = (0 , x 1 ) . Assume we want to minimize the error at time k = 1 (namely ( α 0 , α 1 ) = (0 , 1) in (5)). Then, considering only the causality constraint (6), the estimator (ˆ x 0 , ˆ x 1 ) = ( y 0 , y 1 ) is optimal. Indeed, it is causal and it achieves zero MSE. Similarly, considering only the perceptual quality constraint (7), the estimator (ˆ x 0 , ˆ x 1 ) = ( y 1 , y 1 ) is optimal. Indeed, it is distributed like ( x 0 , x 1 ) and it also achieves zero MSE. However, when demanding both conditions, ˆ x 0 must be based on no information to obey (6) (as y 0 = 0 ), and it must be drawn from the prior distribution N (0 , 1) in order to be distributed like x 0 and obey (7). Furthermore, to satisfy (7), we must also have ˆ x 1 = ˆ x 0 . Therefore, the optimal estimator in this case is (ˆ x 0 , ˆ x 1 ) = (˜ q 0 , ˜ q 0 ) , where ˜ q 0 ∼ N (0 , 1) is independent of q 0 . The MSE achieved by this estimator is 2 .

4 Perfect perceptual-quality filters

The perceptual constraint (7) dictates that the estimator must be of the form

where J k = ˆ x k -A k ˆ x k -1 is distributed as N (0 , Q k ) and is independent of ˆ X k -1 0 . Note the similarity between (12) and the MSE-optimal state (8), in which J k = K k I k . Here, however, this choice is not valid due to the constraint on the output distribution. In terms of temporal consistency, an estimator of the form (12) guarantees that previously presented features obey the natural dynamics of the domain, while newly generated estimates do not contradict the previous ones. In order to maintain causality (6), J k must be of the form

i.e. , J k is independent of future observations Y T k +1 given Y k 0 . As a consequence, J k is uncorrelated with I k + n for all n ≥ 1 .

We now discuss linear estimators, where Y T 0 and J T 0 (hence ˆ X T 0 ) are jointly Gaussian. Our first result is as follows (see proof in App. B).

Theorem 4.1. Under the cost (5) , there exists a linear optimal estimator of the form

π k ∈ R n x × n y and ϕ k ∈ R n x × ( kn y ) are the filter's coefficients, w k is an independent Gaussian noise with covariance Σ w k = Q k -π k S k π ⊤ k -ϕ k Σ υ k ϕ ⊤ k ⪰ 0 , υ k is the process of unutilized information

Note that with this form for J k , the state ˆ x k is indeed a function of the observations Y k 0 and the previous states ˆ X k -1 0 . Intuitively, υ k is the part of the information in the observations, which has no correlation with the information used to construct the past estimates ˆ X k -1 0 . Thus, from the standpoint of the filter's output, this information has not yet been introduced. As opposed to the innovation I k , the process υ k is not white, and it is affected by the choices of π t and ϕ t up to time k -1 . However, I k is always independent of υ k , since I k is independent of I k -1 0 and J k -1 0 , which constitute υ k .

The filter of Theorem 4.1 is causal but not recursive. Specifically, although it is possible to obtain υ k +1 , Σ υ k +1 given the coefficients { π t , ϕ t } k t =0 (see App. E), the dimension of υ k grows with time (it is kn y ), thus increasing the cost of computing ϕ k υ k . Furthermore, determining the coefficients { π k , ϕ k } T k =0 that minimize the objective (5) (which is done a-priori in an offline manner) requires solving a large optimization problem, as the total size of all coefficients is O ( n x n y T 2 ) . To efficiently optimize these coefficients, we next suggest two simplified versions of this form, which may generally be sub-optimal but easier to optimize.

Remark 4.2 . A remark is in place regarding objectives beyond the

Kalman filter

Ak

Unutilized

information

Ak

Perceptual filter

Φ𝑘

Π𝑘

squared-error. While (14) forms an optimal filter under the cost (5), it can be considered as a representation for linear filters in general. The constraints on the coefficients ( Q k -π k S k π ⊤ k -ϕ k Σ υ k ϕ ⊤ k ⪰ 0 ) are necessary and sufficient for perfect perception, regardless of the cost objective. It is therefore possible to optimize coefficients for objectives other than MSE under these constraints to obtain optimal perceptual linear filters. Optimization may be performed numerically (when the cost is tractable) or in an online fashion, assuming access to ground-truth samples. Note, however, that under general distortion measures, optimal filters might be non-linear.

4.1 Recursive-form filters

The optimal Kalman state ˆ x ∗ k achieves the minimal possible MSE, given by d ∗ k = E [ ∥ ˆ x ∗ k -x k ∥ 2 ] = Tr { P k | k } , where P k | k is the error covariance, given explicitly in Alg. 2. By the orthogonality principle (see e.g. [8, Lemma 2]), any other estimator ˆ x k based on the observations Y k 0 , satisfies

Now, consider an estimator ˆ x k of the form (12), and let D k ≜ E [(ˆ x ∗ k -ˆ x k )(ˆ x ∗ k -ˆ x k ) ⊤ ] . Since we choose J k to be normally distributed and independent of ˆ x k -1 , it is easy to see that D k obeys the Lyapunov difference equation

As we see, the choice of J k affects current (and future) errors by its correlation with the two independent components, ( A k ˆ x ∗ k -1 , K k I k ) . Let us now consider filters of the form

where here, by slight abuse of notation, we define the process of unutilized information as

The matrices Π k , Φ k ∈ R n x × n x are coefficients such that where we denote the Kalman update covariance by M k ≜ K k S k K ⊤ k . This guarantees that J k ∼ N (0 , Q k ) , as desired. Importantly, as opposed to υ k , the dimension of Υ k is fixed , namely it does not grow with time k . Note that since ˆ x ∗ k -1 is a linear combination of ( I 0 , . . . , I k -1 ) , (18) is a special choice of π k and ϕ k in (14) where coefficient size does not grow with k as well. Υ k and its covariance Σ Υ k are given via a recursive form, illustrated in Fig. 2 (and derived in App. E):

Note again that unlike the innovation I k , Υ k might not be a white process, but we have that Υ k is independent of the filter's output ˆ X k -1 0 and I k . Equation (17) now takes the form

where we observe that Σ Υ k may depend on the choice of { Π t , Φ t } k -1 t =0 . In order to retrieve an optimal filter, one should perform optimization of the desired objective over { Π t , Φ t } T t =0 , under the constraints given in (20). From (16), minimizing the cost (5) boils down to minimizing ∑ T k =0 α k Tr { D k } subject to the constraints in (20), which is an optimization problem over only O ( n 2 x T ) parameters.

4.2 An Exactly solvable reduction: Perceptual Kalman Filter

We now consider an additional reduction, which allows to obtain a closed form solution for the filter's coefficients. Specifically, a reduced-size filter can be obtained by using the form (12) and (18) with the sub-optimal choice Φ k ≡ 0 , namely

The meaning of this choice is that only newly-observed information is used for updating estimation at each stage, while non-utilized information from previous time steps is discarded. We note that such a simplification should be used with discretion; while requiring only half of the computations, rejecting the unutilized information might lead to enhanced errors in some settings ( e.g. when an observation is missing). However, in many cases where observations are informative and different timesteps are weakly correlated, past unutilized information rapidly becomes irrelevant and can be safely ignored. We demonstrate the utility of this reduction in Sec. 5. Here, Π k is a n x × n x coefficient matrix, and

Algorithm 1 Perceptual Kalman Filter (PKF)

Input : Kalman Filter outputs {I k , K k , S k } T k =0 , weights { α k } T k =0 , matrices A,P 0 , { Q k } T k =1 satisfying Im { B k M k B k } ⊆ Im { Q k } for all k = 0 . . . , T .

initialize : ˆ x 0 = Π 0 K 0 y 0 + w 0 , w 0 ∼ N ( 0 , P 0 -Π 0 M 0 Π ⊤ 0 ) , where Π 0 is given by (30) and B 0 = ∑ T t =0 α t ( A t ) ⊤ A t . for k = 1 to T do

w

k

0

, Q

k

Π

k

M

k

Π

⊤

.

calculate : M k = K k S k K ⊤ k , B k = ∑ T t = k α t ( A t -k ) ⊤ A t -k , M B = B k M k B k . compute optimal gain (30): Π k = Q k M 1 2 B ( M 1 2 B Q k M 1 2 B ) 1 2 † M † 1 2 B B k M k M † k . sample : ∼ N ( -k ) update state : ˆ x k = A ˆ x k -1 +Π k K k I k + w k .

end for w k ∼ N (0 , Q k -Π k M k Π ⊤ k ) is a Gaussian noise, uncorrelated with all other states, observations and noises in the system up to time k . Again, note that I k is independent of the measurements up to time k -1 , hence this choice makes J k independent of ˆ X k -1 0 . These innovation-based corrections resemble the mechanism exploited in (8), hence we will refer to optimal filters of the form (24) as perceptual Kalman filters (PKF).

Now, by a straightforward substitution, (17) becomes

where we consider Q 0 = P 0 , M 0 = Σ ˆ x ∗ 0 and D -1 = 0 . As before, minimizing (5) boils down to minimizing ∑ T k =0 α k Tr { D k } , and in order for (24) to be well-defined, we should enforce the constraints Q k -Π k M k Π ⊤ k ⪰ 0 , k = 0 , . . . , T . For simplicity, we consider the time-invariant case where A k ≡ A , so that the optimization objective becomes

As we can see, optimization over a particular coefficient Π k does not affect other summands of the external sum. Therefore, each Π k can be optimized separately. Minimizing the cost at the k -th step is equivalent to

Let us denote B k ≜ ∑ T t = k α t ( A t -k ) ⊤ A t -k = α k I + A ⊤ B k +1 A . As we now show, this optimization problem possesses a closed-form solution under a mild assumption (which is satisfied e.g. when Q k ≻ 0 ). The proof is given in Appendix F.

Theorem 4.3. Assume that Im { B k M k B k } ⊆ Im { Q k } for every k . Let M B ≜ B k M k B k and denote Ω = { Π k : Q k -Π k M k Π ⊤ k ⪰ 0 } . Then the optimal value in (28) is given by and is achieved by the optimal coefficient (which is generally not unique)

For a closed form solution under the alternative assumption that Im { M k } ⊆ Im { Q k } , as well as a discussion of stationary filters, please see the Appendix. The Perceptual Kalman filter (PKF) obtained from Thm. 4.3 is summarized in Alg. 1.

5 Numerical demonstrations 1

We now revisit our main question: what is the cost of temporal consistency in online restoration? In addition, as we have seen in Sec. 4.2, the relaxation Φ k = 0 , yielding the Perceptual Kalman filters, reduces the complexity of computation, possibly at the cost of higher errors. It is natural, then, to ask what is the cost of this simplification. In the following experiments, we compare the performance of several filters; ˆ x ∗ kal and ˆ x tic correspond to the Kalman filter and the temporally-

inconsistent filter (10) (which does not possess perfect-perceptual quality). The estimate ˆ x opt is a perfect-perception filter obtained by numerically optimizing the coefficients in (18), where the cost is the MSE at termination time, C T = E [ ∥ ˆ x T -x T ∥ 2 ] . ˆ x direct is a perfect-perception filter obtained by the optimization approach discussed in Appendix C, where we consider again the terminal cost. The estimates ˆ x auc , ˆ x minT correspond to PKF outputs (Alg. 1) minimizing the total cost (area under curve) C auc = ∑ T k =0 E [ ∥ ˆ x k -x k ∥ 2 ] and the terminal cost C T , respectively. The filters are summarized in Table 1. Full details and additional experimental results are given in App. I.

5.1 Harmonic oscillator

We start with a simple 2 -D example. Specifically, we consider a harmonic oscillator, where the state x k ∈ R 2 corresponds to position and velocity, and the observation at time t is a noisy versions of the position at time t -1 2 ∆ t , where ∆ t is the sampling period (see App. I for details). Figure 3 shows the MSE for ˆ x opt and the sub-optimal PKF outputs ˆ x auc , ˆ x minT . The estimates ˆ x ∗ kal and ˆ x tic achieve lower MSE than ˆ x opt , however they do not possess perfect-perceptual quality. The difference in MSE between the filters ˆ x opt and ˆ x tic is the cost of temporal consistency in online estimation for this setting .

In Figure 3, we also observe that the PKF estimations are indeed not MSE optimal at time T = 255 . However, their RMSE is only about

Ns = 1024

0

50

100

150

200

250

k

(time)

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

RMSE (analytical)

(i) Gap due to

temporal consistency

(i)

(ii) Gap due

to Φ

k

= 0

(ii)

ˆ

x

opt

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

30% higher than that of ˆ x opt and they have the advantage that they can be solved analytically and require computing only half of the coefficients ( Π k ). The penalty related to this reduction may vary, depending on the exact setup. In Fig. 4 we demonstrate the relations between gaps due to temporal consistency (gap between ˆ x direct and ˆ x tic ) and due to the reduction Φ k = 0 (gap between ˆ x direct and ˆ x minT ), in various settings based on the Harmonic Oscillator example. On the left pane, dynamics are driven by x k = ρAx k -1 + q k , where A is a fixed marginally-stable matrix, and ρ is a scalar controlling the strength of correlation between timesteps. On the right pane, ρ = 1 and observations are given only up to time τ ( y k is only noise for k ≥ τ ). We observe that for systems where timesteps are strongly correlated, or in the absence of current information, discarding υ k leads

terminal error at

T

=255, stabilized Harmonic oscillator

x

k

=

ρAx

k

1

+

q

k

terminal error at

T

=255, stabilized Harmonic oscillator

x

k

=

Ax

k

1

+

q

k

-

0

.

30

-

0

.

25

-

0

.

20

-

0

.

15

-

0

.

10

-

0

.

05

0

.

00

log

ρ

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

√

E

‖

ˆ

x

T

-

x

T

‖

2

Tr

{

Σ

x T

}

ˆ

x

minT

ˆ

x

direct

ˆ

x

tic

0

50

100

150

200

250

τ

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

√

E

‖

ˆ

x

T

-

x

T

‖

2

Tr

{

Σ

x T

}

ˆ

x

minT

ˆ

x

direct

ˆ

x

tic

-

-

to significantly higher errors. Whenever states are less correlated and more observations are given, unutilized information can be ignored with lower cost.

5.2 Dynamic texture

We now illustrate the qualitative effects of perceptual estimation in a simplified video restoration setting. Specifically, we consider a video of a 'dynamic texture' of waves in a lake. Such dynamic textures are accurately modeled by linear dynamics of a Gaussian latent representation [5], whose parameters we learn from a real video. Here, frames are generated from a latent 128 -dimensional state x FA k which corresponds to their Factor-Analysis (FA) decomposition (see e.g. [2, Sec. 12.2.4] for more details). Thus, 512 × 512 × 3 frames in the video domain are created through an affine transformation of x FA k . Linear observations y k ∈ R 32 × 32 are given in the frame (pixel) domain, by 16 × downsampling the Y -channel of the generated ground-truth frames, and adding white Gaussian noise. All filtering is done in the latent domain, and then transformed to the pixel domain. MSE is also calculated in the FA domain. The exact settings can be found in App. I.

In the first experiment, measurements are supplied up to frame k = 127 and then stop (Fig. 5), letting the different filters predict the next, unobserved, frames of the sequence. We can see that until frame k = 127 , all filters reconstruct the reference frames well. Starting from time k = 128 , when measurements stop, the Kalman filter slowly fades into a static, blurry output which is the average frame value in this setting. This is a non-'realistic' video; Neither the individual frames nor the temporal (static) behavior are natural to the domain. Our perfect-perceptual filter, ˆ x auc , keeps generating a 'natural' video, both spatially and temporally. This makes its MSE grow faster 2 .

We now perform a second experiment, where measurements are set to zero until frame k = 512 . At times k > 512 they are given again by the noisy, downsampled frames as described above. In Fig. 6 we present the outcomes of the different filters. We first note that up to frame k = 512 , there is no observed information, hence outputs are actually being generated according to priors. The Kalman filter outputs a static, average frame. The filter ˆ x tic randomizes each frame independently, leading to unnatural random movement with flickering features. At frame k = 513 , when observations become available, ˆ x ∗ kal and ˆ x tic get updated immediately, creating an inconsistent, non-smooth motion between frames 512 and 513 . The PKF output ˆ x auc , on the other hand, maintains a smooth motion. Since the outputs of inconsistent filters rapidly becomes similar to the ground-truth, their errors drop. The perfect-perceptual filter, ˆ x auc , remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.

'rsize':

(512, 512), 'n components':

128, 'dt':

1.0, 'fps':

0.1, 'ds':

)

.

gt

x

(obs

y

)

.

∗

ˆ

(obs

x

y

[16, 16], 'Wc':

N

s

=16

10, 'maxFrame': 1500, 'lmaxA': 0.0, 'cmdLine': ['./inp.py', '-wc', '5', '-T', '10', '-dataset', 'river', '-cmp', '128', '-mxfr', '1500', '-frsze', '512', '-ds', '16', '-nc', '3', '-ncy', '1', '-fani', '1', '-real1', '0', '-workers', '1', '-rpow', '1e-1']

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

(ours)

k=256

k=2047

ˆ

x

tic

ˆ

x

auc

(ours)

k=256

k=2047

0

500

1000

1500

2000

k

(time)

0

50

100

150

200

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

0

500

1000

1500

2000

k

(time)

0

50

100

150

200

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

5, 'id':

'2023-05-12 08 30 04s0r47 river', 'seed':

0, 'startFrame':

gt

y

ˆ

x

∗

ˆ

x

tic

k=511

ˆ

x

auc

(ours)

k=512

k=513

k=514

0

200

400

600

800

1000

k

(time)

0

50

100

150

200

250

300

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

0

200

400

600

800

1000

k

(time)

0

50

100

150

200

250

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

N

s

=16

Conclusion We studied the problem of causal filtering of time sequences from corrupted or missing observations, where the the filter's output process is constrained to possess perfect (spatio-temporal) perceptual-quality in the sense of being distributed like the original signal. Our theoretical derivations focused on Gauss-Markov state-space processes. We introduced the novel concept of an unutilized information process and established a special class of perceptual filters, coined Perceptual Kalman Filters (PKF), that are based on the innovation process alone. We demonstrated the qualitative effects of perfect perceptual quality estimation on a video reconstruction problem. To the best of our knowledge, this is the first work addressing the distortion-perception tradeoff in online restoration settings. This work paves the way toward understanding perceptual online filtering, and the cost of temporal consistency in sequential estimation problems.

Acknowledgements The work of RM was partially supported by the Israel Science Foundation grant 1693/22 and by the Skillman chair in biomedical sciences. The work of TM was partially supported by the Israel Science Foundation grant 2318/22 and by a gift from KLA. RM and TM were supported by the Ollendorff Minerva Center, ECE Faculty, Technion.

References

Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks. advances in neural information processing systems , 30, 2017.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6228-6237, 2018.

Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for gan-based video generation. ACM Transactions on Graphics (TOG) , 39(4):75-1, 2020.

Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. International Journal of Computer Vision , 51:91-109, 2003.

Dror Freirich and Emilia Fridman. Decentralized networked control of systems with local networks: A time-delay approach. Automatica , 69:201-209, 2016.

Dror Freirich and Emilia Fridman. Decentralized networked control of discrete-time systems with local networks. International Journal of Robust and Nonlinear Control , 28(1):365-380, 2018.

Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the distortion-perception tradeoff in wasserstein space. Advances in Neural Information Processing Systems , 34:25661-25672, 2021.

Robert M Freund. Introduction to semidefinite programming (sdp). Massachusetts Institute of Technology , pages 8-11, 2004.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

R.E. Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering. 82 (1) 35-45 , 1960.

Tae Hyun Kim, Mehdi S. M. Sajjadi, Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for video restoration. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications , 48:257-263, 1982.

Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. Perceptual video super resolution with enhanced temporal consistency. arXiv preprint arXiv:1807.07930 , 2018.

Alexander Shapiro. Extremal problems on the set of nonnegative definite matrices. Linear Algebra and its Applications , 67:7-18, 1985.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM review , 38(1): 49-95, 1996.

Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint Supplementary Material

In App. A we provide a detailed theoretical background on the Kalman Filter, its properties and recursive calculation. In App. B we prove that under our perfect perceptual filtering setting, there exists a linear optimal filter (Thm. 4.1). In App. C we discuss a direct, non-recursive method for optimizing perceptual filter coefficients. In App. D we present the derivation of the Lyapunov equation (17) for the error of perceptual filters. In App. E we derive the recursive expression for the filter given in (18). In App. F we find a closed-form solution for PKF coefficients by proving Theorem 4.3. In this appendix, we also give some brief overview on the extremal problem of finding a minimal distance between distributions. App.G contains a discussion about stationary perceptual Kalman filters in the steady-state regime. We summarize all definitions and notations in App. H. Finally, in App.I we give full details for all numerical demonstrations, and present additional empirical and visual results. More results are provided in the supplementary video.

A The Kalman Filter algorithm

In this Section we supply a detailed reminder of the Kalman filter Algorithm. The celebrated Kalman filter [11] assumes a state x k ∈ R n x , where dynamics are modeled as deterministic linear functions perturbed by a Gaussian noise, and observations y k ∈ R n y are linear functions of x k with an additive noise

The noise terms q k and r k are independent white Gaussian processes with zero mean and covariances Q k , R k , respectively. x 0 is assumed to have a zero-mean Gaussian distribution with covariance P 0 , independent of q 1 , r 0 . For convenience, we will sometimes refer to P 0 as Q 0 . The matrices A k , C k , Q k .R k and P 0 are system parameters with appropriate dimensions, and assumed to be known. Considering the MSE distortion, we denote

namely the optimal MSE estimator of the state at time k , given measurements up to time s . Under the assumptions mentioned above, Kalman filters produce the mean state estimate ˆ x k | k , an MSE-optimal estimator of x k given the observations up to time k . The Kalman optimal state ˆ x ∗ k ≡ ˆ x k | k is given by the recurrence

where K k is the optimal Kalman gain [11], given explicitly in Algorithm 2.

The vector I k is the innovation process,

describing the contribution of the new observation y k over the optimal prediction based on previous observations. Since we are in the Linear-Gaussian setup, we have that the innovation state I k is orthogonal to the measurements y 0 , . . . , y k -1 , guaranteeing the MSE optimality of the estimation. The calculation of Kalman state is summarized in Algorithm 2.

B Optimality of linear filters (proof of Thm. 4.1)

In this section we show that under a family of optimality criteria (5) and perfect-perceptual quality and causality constraints (6-7), linear filters of the form (14) are optimal. We start with the following. Theorem B.1. Let Y T 0 = ( y 0 , . . . , y T ) be the set of measurements (4) , and let ( J T 0 , Y T 0 ) be a joint distribution s.t. J k is independent of y k + n given Y k 0 for all k ∈ [0 , T ] and n ≥ 1 . Then,

Proof. Denote ˆ J k = E [ J k |I k 0 ] . We can write the measurements as a linear function of the innovations, y k = ∑ k t =0 H k,t I t . We have and

For any k and n = 1 , I k +1 = y k +1 -ˆ y k k +1 , and therefore

This is due to the facts that J k and y k +1 are independent given the condition, and that ˆ y k k +1 is a deterministic function of I k 0 .

We now show that for every filter which is feasible under (6) and (7), one can find a linear filter, jointly Gaussian with the measurement set, attaining the same cost objective.

Theorem B.2. Let Y T 0 = ( y 0 , . . . , y T ) be the set of measurements (4) , and let J T 0 = ( J 0 , . . . , J T ) be jointly distributed with Y T 0 such that:

(i) J T 0 ∼ N (0 , diag { P 0 , Q 1 , . . . , Q T } ) .

(ii) J k is independent of y k + n given Y k 0 for all k ∈ [0 , T ] and n ≥ 1 .

Then, there exists a joint Gaussian distribution ( J T 0 , Y T 0 ) in which (i) and (ii) hold, and the estimator given by

achieves the same cost (iii) , namely ∑ T k =0 α k E [ ∥ x k -ˆ x k ∥ 2 ] = C . Furthermore, we can write where

and w k is a white Gaussian noise, independent of Y T 0 and J k -1 0 .

Proof. Let ( J T 0 , Y T 0 ) be the Gaussian distribution defined by the moments of ( J T 0 , Y T 0 ) up to second order. We observe that from Theorem B.1 above, J k is independent of all future innovations I k + n , namely it is based only on measurements up to time k . Using the notions of Theorem B.1's proof,

This means that J k and y k + n are independent given Y k 0 , which proves (ii).

From (17) we see that the cost functional depends only on the second order statistics of ( J T 0 , I T 0 ) which are identical to those of ( J T 0 , I T 0 ) , hence (iii) holds:

To prove (42), we now write where ε k = E [ J k | Y T 0 , J k -1 0 ] , and w k = J k -E [ J k | Y T 0 , J k -1 0 ] is independent of Y T 0 and J k -1 0 . Now, since both J k and J k -1 0 are independent of I T k +1 ,

J k is independent of J k -1 0 , thus

Conditioning both sides of (47) on J k -1 0 and taking expectations,

Note that E [ I k | J k -1 0 ] = 0 , which together with (49) implies

Now, all we have left to show is that w k is a white sequence. Since w k + n ( n ≥ 1 ) is independent of J k 0 and I T 0 (which also constitute υ k ), it is easy to obtain

Corollary B.3. Given a cost objective of the form C = ∑ T k =0 α k E [ ∥ x k -ˆ x k ∥ 2 ] , there exists a linear filter of the form such that

is an optimal estimator under the perfect perceptual quality and causality constraints (6-7).

Proof. Under the perfect perceptual quality constraint, an estimate sequence χ k must satisfy that

is a white Gaussian process with covariances Q k . If, in addition, χ k satisfies the causality condition (6), so does J k . We conclude from Theorem B.2 that there exists a causal linear filter J k that achieves the same expected objective C as χ k .

Now, note again that from (17), for perfect-perceptual quality causal filters, the objective C is a continuous function of the covariance matrix where, due to the causality demand, L is a quasi lower triangular matrix. The set of such feasible matrices is non-empty, closed (since it is the intersection of the closed cone of PSD matrices with a finite set of hyperplanes) and bounded. Hence, C attains a minimal value on some joint distribution p J T 0 ,Y T 0 , which can be chosen to be joint-Gaussian as we have seen.

C A Direct optimization approach to perfect-perceptual quality filtering

For the sake of completeness, we now discuss a method for optimizing non-recursive perfectperceptual quality filter coefficients. This approach leads to convex programs. However, as we will see next, it might become impractical for large configurations.

Let J = J T 0 ∼ N (0 , Q ) , where Q = diag { { Q k } T k =0 } , be a causal function of the measurements, J = Φ I + W , where I = I T 0 is the innovation process with covariance S = diag { S k } and W is an independent noise. Now, ˆ X = ˆ X T 0 = A J J is the filter's output, where

Recall X ∗ is the Kalman filter output given by X ∗ = A J K I , where K = diag { K k } . Let W = diag { α k } ⊗ I n x be a weighting matrix. The objective (5) is now given by

Hence, we have to maximize

where B ≜ A ⊤ J W A J . This is subject to the perfect perceptual-quality constraint where Φ is a lower quasi-triangular matrix (causality constraint)

Again, under this formulation, where W ∼ N ( 0 , Q -ΦSΦ ⊤ ) is a Gaussian noise independent of I . Note that W T 0 might not be a white sequence in this case, since its covariance might not be a block-diagonal matrix. As a result, the noise sequence has to be sampled dependently. Also note that this problem possesses the same memory complexity as (14). To conclude, this method leads to convex, but large optimization programs, and is impractical for high dimensional settings or long temporal sequences.

D Derivation of eq. (17)

Recall ˆ x ∗ k is the optimal Kalman state at time k , achieving MSE given by

P k | k is the error covariance, given explicitly in Algorithm 2. By the orthogonality principle, for any estimator ˆ x k based on the measurements y 0 , . . . , y k we have

Since we choose J k ∼ N (0 , Q k ) to be independent of ˆ x k -1 and I k is indepenedent of ˆ x k -1 and ˆ x ∗ k -1 , we write

E Derivation of recursive perfect-perceptual quality filters

We now derive the recursive expression (21)-(22) for the filter given in (18),

defined by the coefficients { Π k , Φ t } T t =0 fulfilling the constraints (20). Recall where ˆ x ∗ k is the Kalman state. J k -1 0 , Υ k , I k , w k are jointly-Gaussian and independent, and we have

We can write

Since J k 0 is an independent sequence, and since I k depends only on J k ,

We also have that Υ k , J k are independent of J k -1 0 , implying

Hence, where we denote

The covariance is then given by the recursive form

At time k = 0 we have Υ 0 = 0 and Σ Υ 0 = 0 .

Remark E.1 (The non-reduced case) . For the full, non-reduced linear filter (14)- (15) , we have the following similar formula

and

Notice, however, that the dimension of υ k grows with time k .

F A Generalized extremal problem with semidefinite constraints (proof of Thm. 4.3)

In this section we prove Theorem 4.3. We start with a brief overview of the extremal problem of finding a minimal distance between distributions, and of general semi-definite programs.

To prove the Theorem we observe that (28), is a generalization of the extremal problem, and suggest a non-trivial dual form where, under our assumptions, strong duality holds.

F.1 Minimal distance between distributions

Consider two Gaussian distributions on R n with zero means and PSD covariance matrices Σ 1 , Σ 2 respectively. We consider the problem of constructing a Gaussian vector [ X,Y ] minimizing E ∥ X -Y ∥ 2 while inducing the given marginal distributions, X ∼ N (0 , Σ 1 ) , Y ∼ N (0 , Σ 2 ) . This problem is equivalent to the following maximization of correlation [13]

We have the following results of Olkin and Pukelsheim [13].

Lemma F.1. [13, Lemma 1]. Let Σ g 2 be any generalized inverse of Σ 2 . Then Σ ⪰ 0 iff

Theorem F.2. [13, Thm. 4]. If Im { Σ 2 } ⊆ Im { Σ 1 } , then an optimal solution to (83) is given by

achieved by the argument

In the case where Im { Σ 2 } = Im { Σ 1 } , Π ∗ is a unique optimal argument.

Under the setting discussed in Sec. 2, Theorem F.2 implies that in the more general case where Σ x ⪰ 0 , the MSE-optimal perfect perceptual-quality estimator (2) is obtained by

Here again, w is a zero-mean Gaussian noise with covariance Σ w = Σ x -T ∗ Σ x ∗ T ∗⊤ , independent of y and x , and Σ † x ∗ is the Moore-Penrose inverse of Σ x ∗ .

F.2 SDP Setting and duality - background

Semi-definite programming (SDP) [9, 17] is an optimization problem in X ∈ R n × n of the form

Here, C, A i are real symmetric matrices of appropriate dimensions, and A · X = Tr { A ⊤ X } is the Frobenius product. SDPs yield the Lagrangian

where λ ≥ 0 and ρ min is the minimal eigenvalue. The Dual problem (DSP) is given by

In this case, strong duality exists iff the SDP is strictly feasible, i.e. it has a feasible solution interior to the feasible set, X ≻ 0 . This condition is sometimes referred to as the Slater condition .

F.3 A generalized extremal problem with strong duality

Recall Q k , M k , B k are real, symmetric positive semidefinite n x × n x matrices, and the optimization problem (28),

Since (93) involves a single time step, we will omit the index k .

We consider Π = ˜ Π M , hence Π ⊤ = MM † Π ⊤ , and since M = MM † M we rewrite (93) as

By Lemma F.1, the constraints in (94) are equivalent to

This can be formulated as the semi-definite program,

Note that when B is a scalar matrix, (94) is similar to the problem studied in Olkin and Pukelsheim [13]. Their approach was later extended by Shapiro [15] to general linear objectives, where the Slater condition holds.

F.3.1 Strong duality

The SDP (96) yields the standard dual formulation

This should give us a hint about the optimal solution to (94). Pay attention, however, that according to the theory, strong duality in (97) is guaranteed only if Q,M ≻ 0 , which might not be the case (see e.g. [15]). To get a tight bound for the general case Q,M ⪰ 0 , we now provide an alternative form of duality to (94).

The following is an adaptation of techniques used in Olkin and Pukelsheim [13]. We start with the following Lemma.

Lemma F.3. Let Π be a feasible solution to (94), R,G ∈ R n x × n x are general matrices. Then,

Proof. From the non-negativity of X in (95) we have

The trace is nonegative, hence we have the desired result.

Remark F.4 . Similarly, we can obtain

Now, we suggest an alternative to (DSP) (97), where strong duality will hold.

Theorem F.5. [Strong duality]. Let

and denote M B ≜ BMB . Assume Im { M B } ⊆ Im { Q } . Then,

The extreme value is obtained for

Optimal solution Π ∗ is generally not unique.

To prove strong duality, we will use the following lemmas.

Lemma F.6. Assume PSD matrices Q,M B such that Im { M B } ⊆ Im { Q } , then Im { M B } = Im { M 1 / 2 B QM 1 / 2 B } .

Proof. Recall M B , M 1 / 2 B QM 1 / 2 B are real symmetric matrices.

Let v ∈ Ker { M 1 / 2 B QM 1 / 2 B } , we have ∥ Q 1 / 2 M 1 / 2 B v ∥ = 0 hence M 1 / 2 B v ∈ Ker { Q 1 / 2 } ⊆ Ker { M 1 / 2 B } , which yields M B v = 0 , implying Ker { M 1 / 2 B QM 1 / 2 B } ⊆ Ker { M B } . Opposite relation is trivial.

We have

Lemma F.7. Im { BM } ⊆ Im { BMB } .

Proof. Let v ∈ Ker { BMB } , then ∥ M 1 / 2 Bv ∥ = 0 and Bv ∈ Ker { M 1 / 2 } = Ker { M } . Hence Ker { BMB } ⊆ Ker { MB } . We have

We are now ready to prove Theorem F.5.

Proof. [Theorem F.5]. Let Π ∈ Ω , then X ⪰ 0 in (95). For any ( S, S -) ∈ S we can choose R = S 1 / 2 , G = S -R . From the result of Lemma F.3 it follows that

The last equality holds since BM = SS -BM , and Im { Π T } ⊆ Im { M } .

We now prove that Π ∗ ∈ Ω .

(110) The last equality holds since it is easy to see that [ I -Q 1 / 2 M 1 / 2 B ( M 1 / 2 B QM 1 / 2 B ) † M 1 / 2 B Q 1 / 2 ] is a symmetric (orthogonal) projection.

We further prove that S ∗ , S -∗ ∈ S . It is easy to show that S ∗ , S -∗ are symmetric generalized inverses, reflexive to each other ( S -∗ is in fact the Moore-Penrose inverse of S ∗ ):

The equalities hold since by Lemma F.6,

and since for a PSD matrix R , RR † = R † R is an orthogonal projection onto its image. Using Lemma F.7 we have

It is now easy to verify that

which completes the proof.

Corollary F.8. Under the assumption, Im { M B } ⊆ Im { Q k } , the optimal gain in (28) is given by

Remark F.9 . Under the alternative assumption, Im { M k } ⊆ Im { Q k } , the optimal gain in (28) is given by

where ˜ B = B 1 / 2 k , Q b = ˜ BQ k ˜ B, M b = ˜ BM k ˜ B .

Proof. Recall our goal in (28) is to maximize Tr { Π MB } = Tr ˜ B Π M ˜ B under the condition Q -Π M Π ⊤ ⪰ 0 (we omit the index k ). This is equivalent to minimizing E [ ∥ ˜ BX -˜ BY ∥ 2 ] w.r.t Π , where ( X,Y ) ∼ N (0 , Σ) and Σ = [ Q Π M M Π ⊤ M ] ⪰ 0 .

{ }

In this case, ( ˜ BX, ˜ BY ) ∼ N (0 , Σ b ) where Σ b = [ ˜ BQ ˜ B ˜ B Π M ˜ B ˜ BM Π ⊤ ˜ B ˜ BM ˜ B ] . According to Thm. F.2, under the assumption Im { ˜ BM ˜ B } ⊆ Im { ˜ BQ ˜ B } , the minimal distance is achieved when

Note that Im { M } ⊆ Im { Q } implies Im { ˜ BM ˜ B } ⊆ Im { ˜ BQ ˜ B } , and it is straightforward to verify that Q -Π ∗ M Π ∗⊤ ⪰ 0 .

G Stationary settings

Anote is in place regarding the stationary perceptual Kalman filter. In the Kalman steady-state regime, where dynamics (31) -(32) are time-invariant and T →∞ , the matrices K and S in Algorithm 2 are determined by the covariance matrix P ,

Here, C stands for the time-invariant observation matrix ( y k = Cx k + r k ) and P is a solution to the Discrete-Time Algebraic Riccati equation (DARE)

Similarly, under the steady-state regime, (26) becomes

where D obeys an (Algebraic) Lyapunov equation. If A is stable,

Hence, stationary perceptual filter is of the form

and in order to find optimal gain Π , minimizing Tr { D (Π) } , we have to solve

where we define B ≜ ∑ ∞ k =0 ( A k ) ⊤ A k , and the solution (under the assumption Im { BMB } ⊆ Im { Q } ) is given again by (30).

H List of notations

We summarize our notations in the following Table.

C

∑

C

[

∥

-

∥

]

I Numerical demonstrations

In this section we provide full details for the experimental settings of Sec. 5, with additional numerical and visual results. In the following, we compare the performance of several filters; ˆ x ∗ kal and ˆ x tic correspond to the Kalman filter and the temporally-inconsistent filter (10) (which does not possess perfect-perceptual quality). The estimate ˆ x opt is generated by a perfect-perception filter obtained by numerically optimizing the coefficients in (18), where the cost is the MSE at termination time, i.e. the terminal cost

The estimates ˆ x auc , ˆ x minT correspond to PKF outputs (Alg. 1) minimizing the total cost (area under curve)

and the terminal cost , respectively. Finally, ˆ x stat . is the stationary PKF, discussed in App. G. The filters are summarized in Table 3.

I.1 Example: Harmonic oscillator

We start with a simple 2 -D example, where we demonstrate the differences in MSE distortion between the optimized perfect-perceptual quality filter ˆ x opt , the temporally inconsistent filter ˆ x tic and the efficient sub-optimal (perceptual) PKF. Consider the harmonic oscillator, where the entries of the state x k ∈ R 2 correspond to position and velocity, and evolve as

with where ∆ t = 5 × 10 -3 is the sampling interval. Assume we have access to noisy and delayed scalar observations of the position (corresponding to time t -1 2 ∆ t ) so that y k = [ 1 -1 2 ] x k + r k , where r k ∼ N (0 , 1) and x 0 ∼ N (0 , 0 . 8 I ) .

We numerically optimize the coefficients { Π k , Φ k } T k =0 in (18), to minimize the terminal error (129) ( Tr { D T } in (23)) at time T = 255 under the constraints (20). Figure 7 shows the MSE distortion for the optimized perfect-perception filter ˆ x opt defined by (18) and { Π k , Φ k } T k =0 , and the sub-optimal PKF outputs ˆ x auc , ˆ x minT , minimizing the total cost (130) and the terminal cost (129) (see Table 3). We observe that PKF estimations are indeed not MSE optimal at time T , However, their RMSE at time T is only ∼ 30% higher than that of ˆ x opt and they have the advantage that they can be solved analytically and require computing only half of the coefficients ( Π k ).

The estimates ˆ x ∗ kal and ˆ x tic achieve lower MSE than ˆ x opt , however they do not possess perfectperceptual quality. We can see the difference in MSE distortion between the filters ˆ x opt and ˆ x tic , with and without perception constraint in the temporal domain. This is the cost of temporal consistency in online estimation for this setting .

N

0

50

100

150

200

250

k

(time)

0

100

200

300

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

ˆ

x

opt

0

50

100

150

200

250

k

(time)

0

50

100

150

200

250

300

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

ˆ

x

opt

ˆ

x

opt

0

50

100

150

200

250

k

(time)

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

RMSE (analytical)

(i) Gap due to

temporal consistency

(i)

(ii) Gap due

to Φ

k

= 0

(ii)

ˆ

x

opt

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

Ns = 1024

s

=1024

I.2 Example: Two coupled inverted pendulums

Next, we demonstrate the quantitative behavior of perceptual Kalman filters, by comparing the MSE between the PKF outputs when minimizing different cost functions, and between non-perceptual filters outputs. More specifically, this experiment demonstrates:

How minimizing different objectives in Algorithm 1 leads to different filters.

The cost of perfect-perceptual quality filtering, given by Algorithm 1, over optimal filters.

Weconsider a higher-dimensional, well-studied example of two coupled inverted pendulums, mounted on carts [7, 6]. The cart positions, pendulum deviations, and their velocities (Fig. 8), are given by the discretized stable closed-loop system with perturbation

where x k ∈ R 8 . The initial state is distributed as

The system matrices are given by

where ∆ t = 5 × 10 -4 is the sampling interval and

The coupling is given by and stabilizing state-feedback controllers (each acts on a single cart) are

K 1 = [11396 . 0 7196 . 2 573 . 96 1199 . 0] , K 2 = [29241 18135 2875 . 3 3693 . 9] . (139)

The partial measurements are given by y k = Cx k + r k , where r k ∼ N (0 , R ) , with coefficients

Namely, we observe only position and angle for each cart/pendulum, while velocities are not being measured.

The perturbation covariances are given by

where

N

s

=1024

500

600

700

800

900

1000

k

(time)

0

.

15

0

.

20

0

.

25

0

.

30

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

500

600

700

800

900

1000

k

(time)

0

.

15

0

.

20

0

.

25

0

.

30

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

Pos.1

angle 1

coupling

Pos.2

angle 2

We simulate the system for 2 10 time steps ( T = 2 10 -1 ), over N = 2 10 independent experiments. In Figures 9 and 10 we show the MSE distortion as a function of time, E [ ∥ ˆ x k -x k ∥ 2 ] , for the different filters of Table 3; ˆ x ∗ kal is the optimal Kalman filter. ˆ x tic is the perceptual filter without consistency constraints, given in (10). ˆ x auc is the PKF output minimizing the total cost (130). ˆ x minT (marked by ' ⋆ ') is the PKF output minimizing the terminal cost (129).

We observe that filters satisfying the perfect perceptual quality constraint ( ˆ x auc and ˆ x minT ) achieve higher distortions compared to the per-sample only perceptual filter ˆ x tic , which in turn attains MSE distortion slightly higher than that of the MSE-optimal Kalman filter. This demonstrates again the cost of temporal consistency in online estimation. Note also that PKFs minimizing different cumulative objectives, yield different estimations; while ˆ x minT is optimal at termination time T , ˆ x auc achieves a lower MSE on average. As we will see next, both filters attain the same perceptual quality.

N

s

=1024

0

200

400

600

800

1000

k

(time)

0

.

0

0

.

1

0

.

2

0

.

3

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

0

200

400

600

800

1000

k

(time)

0

.

0

0

.

1

0

.

2

0

.

3

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

ˆ

x

minT

In Fig. 11 we estimate the perceptual quality, given by the Wasserstein distance between the groundtruth distribution and the empirical Gaussian distributions of the different filters outputs. In Fig. 11(top) we estimate the distance between single-sample distributions, while in Fig. 11(bottom) we consider the joint distributions of 16 state-vectors, x t , t ∈ [ k -15 , k ] . Observe that while each sample of ˆ x tic is distributed similarly to its reference sample, it fails to attain perfect perceptual quality where we measure the distance from the real process distribution. PKF outputs attain low perceptual index (high quality) in both scenarios. We also present the perceptual quality measured for the ground-truth signal x gt empirical distribution, as a reference.

Figure 12 shows the asymptotic behavior (empirical error for large horizon T ) of ˆ x stat. , the stationary PKL (125). The figure also presents the empirical errors for Kalman filter and its stationary version (multiplied by a factor of 2 , which is an upper bound on the MSE distortion of perceptual estimators without temporal constraints, see [3]), and the theoretical steady-state error of (125), obtained by optimizing (128) (dashed horizontal line) for comparison. The error of the non-stationary perceptual filter ˆ x auc is also shown.

I.3 Dynamic texture

Here we illustrate the qualitative effects of perceptual (temporally consistent) estimation in a simplified video restoration setting. Please see the supplementary video for the full videos. This setup visually demonstrates how:

Filters with no perfect perceptual quality tend to generate non-realistic images or atypical motion (random or slow movement, flickering artifacts etc.).

PKF outputs are natural to the domain, both spatially and temporally.

For this extent, we introduce the 'Dynamic Texture' domain. In this domain, video frames are generated from a latent state which represents their Factor-Analysis (FA) decomposition (see e.g. Bishop and Nasrabadi [2, Sec. 12.2.4] for more details). The dynamics in the FA domain are assumed to be linear, with a small Gaussian perturbation,

vec.len 1, nparams 36.0, nsamples 65536

0

68

136

204

272

340

408

476

544

612

680

748

816

884

952

1020

k

(time)

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

dP

(

Xk

,

Xk

)

x

gt

x

*

x

auc

x

minT

x

tic

15

81

147

213

279

345

411

477

543

609

675

741

807

873

939

1005

k

(time)

0.00

0.02

0.04

0.06

0.08

dP

(

X k

k

15

,

X k

k

15

)

x

gt

x

*

x

auc

x

minT

x

tic

vec.len 16, nparams 8256.0, nsamples 65536

Ns

0

1000

2000

3000

4000

k

(time)

0.0

0.2

0.4

0.6

0.8

1.0

MSE

x

*

x

stat. (×2)

x

stat. (empirical)

x

stat. (theoretical)

x

auc

=

256

The state vector with the given dynamics creates frames of a wavy lake in the video domain 3 , through an affine transformation, x vid k = W FA → vid ( x FA k + ε FA ) . (145) W FA → vid is a linear transformation from R 128 latent states to R 512 × 512 × 3 frames, and ε FA is a constant vector. A FA and the noise q k parameters are estimated similarly to [5]. Linear observations y k ∈ R 32 × 32 are given in the frame (pixel) domain, by

At times where information is being observed,

where W RGB → y is a projection onto the Y -channel (grayscale) and C × 16 is a matrix that performs 16 × downsampling in both axes. At times where there is no observed information, C k = 0 . Here, r k is a Gaussian noise.

In our first experiment, measurements are supplied as in (147) up to frame k = 127 and then vanish ( C k = 0 , k ≥ 128 ), letting the different filters predict the next, unobserved, frames of the sequence. We pass y k as an input to the various filters (see Table 3); ˆ x ∗ kal is the Kalman filter output. ˆ x tic is the perceptual filter in the spatial domain, given in (10). ˆ x auc is our Algorithm (PKF) output reducing the total cost in the latent space, C auc = ∑ T k =0 E [ ∥ x FA k -ˆ x k ∥ 2 ] . All filtering is done in the latent domain, and then transformed to the pixel domain. MSE is also calculated in the FA domain. In (Fig. 13) we can see that until frame k = 127 , all filters reconstruct the reference frames well. Starting at time k = 128 , when measurements disappear, we observe that the Kalman filter slowly fades into a static, blurry output which is the average frame value in this setting. This is definitely a non-'realistic' video; Neither the individual frames nor the static behavior are natural to the domain. Our perfect-perceptual filter, ˆ x auc , keeps generating a 'natural' video, both spatially and temporally. This makes its MSE grow faster.

We now perform a second experiment, where C k is set to zero until frame k = 512 . At times k ≥ 513 measurements are given again by the noisy, downsampled frames as described in (146)-(147). In Fig. 14 we present the outcomes of the different filters. We first observe that up to frame k = 512 , there is no observed information, hence outputs are actually being generated according to priors.

, 'n components': 128, 'dt':

1.0, 'fps':

0.1, 'ds':

[16, 16], 'Wc':

x

gt

y

(obs

.

)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

(ours)

k=256

k=2047

0

500

1000

1500

2000

k

(time)

0

50

100

150

200

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

0

500

1000

1500

2000

k

(time)

0

50

100

150

200

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

N

s

=16

10, 'id':

'2023-05-12 09 07 35s0r18 river', 'seed':

0, 'startFrame':

10, 'maxFrame': 1500, 'lmaxA': 0.0, 'cmdLine': ['./inp.py', '-wc', '10', '-T', '11', '-dataset', 'river', '-cmp', '128', '-mxfr', '1500', '-frsze', '512', '-ds', '16', '-nc', '3', '-ncy', '1', '-fani', '0', '-real1', '0', '-workers', '1',

), 'n components': 128, 'dt':

1.0, 'fps':

0.1, 'ds':

[16, 16], 'Wc':

x

gt

y

(obs

.

)

ˆ

x

∗

ˆ

x

tic

k=511

ˆ

x

auc

(ours)

k=512

k=513

k=514

0

200

400

600

800

1000

k

(time)

0

50

100

150

200

250

300

MSE (empirical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

0

200

400

600

800

1000

k

(time)

0

50

100

150

200

250

MSE (analytical)

ˆ

x

∗

ˆ

x

tic

ˆ

x

auc

N

s

=16

5, 'id':

'2023-05-12 08 30 04s0r47 river', 'seed':

0, 'startFrame':

10, 'maxFrame': 1500, 'lmaxA': 0.0, 'cmdLine': ['./inp.py', '-wc', '5', '-T', '10', '-dataset', 'river', '-cmp', '128', '-mxfr', '1500', '-frsze', '512', '-ds', '16', '-nc', '3', '-ncy', '1', '-fani', '1', '-real1', '0', '-workers', '1', '

The Kalman filter outputs a static, average frame. ˆ x tic randomizes each frame independently, which creates the impression of rapid, random movement with flickering features, which is unnatural to the reference domain. At frame k = 513 , when observations become available, we can see that ˆ x ∗ kal and ˆ x tic are being updated immediately, creating an inconsistent, non-smooth motion between frames 512 and 513 . PKF output ˆ x auc , on the other hand, keeps maintaining a smooth motion. Since non-consistent filters outputs rapidly becomes similar to the ground-truth, their errors drop. The perfect-perceptual filter, ˆ x auc , remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.

Peer reviews:
comments: SUMMARY:
This work studies the problem of temporal singal reconstruction for corrupted data. Under a perfect perceptual-quality constraint, the previously regarded optimal model, i.e, the Kalman filter is shown to confronted with a fundamental dilemma. A recursive formula for perceptual filters is proposed and be empirically validated on a video reconstruction problem.

STRENGTHS:
+ This work is put in a proper context in the liteature, where the perception-distortion trade-off is an important theory that only attract few research interests.
+ Reasonable problem formulation, and detailed theoretical drivations.

WEAKNESSES:
- Is the proposed tool generalizable to other fidelity measures in addition to MSE?
- The effecrtiveness of the proposed tool is only validated on a single sample, more are expected.

QUESTIONS:
It's better to split Eq. 17 into multiple lines for improved formatting.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-20

comments: SUMMARY:
This article addresses the problem of optimal causal filtering under a perfect perceptual-quality constraint. The authors introduce the concept of an unutilized information process and present a recursive formula for perceptual filters. The study demonstrates the effects of perfect perceptual-quality estimation on video reconstruction. Overall, this article contributes to understanding optimal causal filtering under a perceptual-quality constraint and offers new insights for addressing this issue.

STRENGTHS:
Unfortunately, for me the paper was not easily accessible as I am not familiar with studies of the signal reconstruction.
·The overall problem is interesting and has practical significance.
·I find this work quite original to the best of my limited knowledge.
·The paper to me seems technically sound and theoretical assumptions are formally proved.
·The appendix materials have been carefully prepared, which can be of great help in understanding the paper. Meanwhile, the open-source code provides convenience for community reference.

WEAKNESSES:
·Experimental analysis is limited, is it necessary to showcase more diverse video scenarios?
·Is it necessary to add a conclusion section to summarize the conclusion of this paper and clarify its possible application scenarios, as well as the interests can be followed in the future research.

QUESTIONS:
As mentioned above.

LIMITATIONS:
The paper clearly describes societal impact and potential limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper aims to study the problem of optimal causal filtering under a perceptual-quality constraint. The main contribution of this paper is to provide a mathematical framework and a closed-form solution to the aforementioned problem under some mild conditions. The experimental results on a video reconstruction problem demonstrate the effectiveness of perceptual-quality estimation.

STRENGTHS:
1. The cost of temporal consistency investigated in this study holds significant importance in the field of online restoration.
2. The experimental results on a video reconstruction problem provide evidence for the qualitative effects of the proposed perceptual filtering.

WEAKNESSES:
1.	The readability of the paper is restricted due to the oversimplification of the formula derivation, such as in the case of Formula 16.

QUESTIONS:
1. The authors should concisely explain why they did not compare their method with other perceptual-quality constrained SOTA methods in online restoration. Adding a comparison with SOTA methods, rather than solely comparing to the Kalman filter, would enhance the overall persuasiveness.
   
2. When both constraints 6 and 7 are applied simultaneously, it may result in an additional MSE loss. Under what conditions can this loss be alleviated while still satisfying both constraints?

LIMITATIONS:
No limitations are addressed in this paper. It is recommended that the authors make an effort to improve the readability of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces the Perceptual Kalman filter as a tractable solution to the online state estimation problem under perfect perpetual-quality constraints. This means the authors review the problem of state estimation under perceptual constrains (i.e. the joint distribution of any sub-sequence of states should match the original data). Then they derive a tractable approximation of the perceptual filters which have an analytic solution. Experiments were conducted on a harmonic oscillator and a dynamic texture domain. The harmonic oscillator experiments capture the MSE due to perceptual consistency and also due to the approximation used for tractability. The dynamic prediction domain demonstrates the approach scales to high-dimensions.\n\nSTRENGTHS:\n1. Clear and thorough presentation of perceptual estimation and its relationship to prior work (in particular the Kalman Filter). \n2. Easy-to-understand derivation of the Perceptual Kalman filter and in particular the trade-offs made in order to obtain tractability.\n3. Experiments that clearly compare how the technique compares to prior work, the theoretical maximum performance and the cost of tractability. Scalability experiments also demonstrate that this works for large linear systems.\n\nWEAKNESSES:\n1. Although the linear systems and observations have wide applicability, the paper would be improved by mentioning this limitation and why it has or doesn't have a significant effect in the applications where perceptual quality of state-estimates is desired. This is mainly to explain how widely the linear results presented by the paper apply to the problem landscape of the general case.\n2. The experiments only show results for a harmonic oscillator. Although this is extremely useful as away to visualize the consistency gap and the gap due to $\\Phi_k = 0$, experiments showing how different dynamics can change these gaps would be very useful.\n\nQUESTIONS:\nHow bad can the $\\Phi_k = 0$ approximation hurt your performance? Is there some provable bound for the worst-case?\n\nLIMITATIONS:\nI think the authors present the limitations of the method clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}