Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Quality Constraint
Authors: Dror Freirich
Tomer Michaeli
Ron Meir
Venue: NeurIPS 2023 poster (2023)
Keywords: Kalman filter
estimation theory
causal filtering
signal processing
distortion-perception tradeoff
TL;DR: We study the problem of optimal causal filtering under a perfect perceptual-quality constraint.

Abstract: Many practical settings call for the reconstruction of temporal signals from corrupted or missing data. Classic examples include decoding, tracking, signal enhancement and denoising. Since the reconstructed signals are ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception.
Mathematically, perfect perceptual-quality is achieved when the distribution of restored signals is the same as that of natural signals, a requirement which has been heavily researched in static estimation settings (i.e. when a whole signal is processed at once). 
Here, we study the problem of optimal causal  filtering under a perfect perceptual-quality constraint, which is a task of fundamentally different nature.  
Specifically, we analyze a Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to ``knowingly'' ignore new information revealed by the observations in order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.

Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Qualit Constraint
Dror Freirich Technion Israel Institute Technology drorfrc@gmail. com
Tomer Michaeli Technion Israel Institute of Technology tomer m@ee. technion..... .c.
Ron Meir Technion Israel
Institute Technology rmeir@ee. technionnninnno ail
Abstract
Many practical settings call for the reconstruction of temporal signals from cor- rupted or missing data. Classic examples include decoding, tracking, signal en- hancement and denoising. Since the reconstructed signals ultimately viewed by humans, it is desirable to achieve reconstructions that are pleasing to human perception, Mathemattically, perfect perceptual-quality achieved when the dis- tribution of restored signals is the same as that of natural signals, requirement which has been heavily researched in static estimation settings (i.e. when whole signal is processed at once). Here, we study the problem of optimal causal filtering under perfect purceptual-quality constraint, which is a task of fundamentally different nature. Specifically, we analyze Gaussian Markov signal observed through a linear noisy transformation. In the absence of perceptual constraints, the Kalman filter is known to be optimal in the MSE sense for this setting. Here, we show that adding the perfect perceptual quality constraint (i.e. the requirement of temporal consistency), introduces a fundamental dilemma whereby the filter may have to "knowingly" ignore new information revealed by the observations order to conform to its past decisions. This often comes at the cost of a significant increase in the MSE (beyond that encountered in static settings). Our analysis goes beyond the classic innovation process of the Kalman filter, and introduces the novel concept of an unutilized information process. Using this tool, we present a recursive formula for perceptual filters, and demonstrate the qualitative effects of perfect perceptual-quality estimation on a video reconstruction problem.
Introduction
many settings, it desired reconstruct temporal signal from corrupted or missing data. Exam- ples include decoding of transmitted communications, tracking targets based on noisy measurements, enhancing audio signals, and denoising videos. Traditionally, restoration quality has been assessed by distortion measures such MSE. As result, numerous methods targeted the minimization of such measures, including the seminal work of Kalman [. However, in applications involving human perception, may favor reconstructions that cannot be told apart from valid signals. Mathemati- cally, such perfect perceptual quality can be achieved only if the distribution of restored signals is the same that of "natural" signals.
Interestingly, has been shown that good perceptual quality generally comes at the price of poor distortion and vice versa. This phenomenon, known as the perception-distoriinn tradeoff. was first studied in [3) and was later fully characterized measured by MSE and perception is measured by the Wasserstein-2 distance between the distribution of estimated signals and the distribution real signals. However, to date, all existing works addressed the static (non-temporal) setting, in which the entire corrupted signal is available for processing all at
for the particular setting where distortion
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

once. This setting is fundamentally different from situations involving temporal signals, in which the corrupted signal is processed causally over time, such that each sample is reconstructed only based on observations up to the current time.
To illustrate the inherent difficulty in causal es- timation, consider video restoration tasks like denoising, super-resolution, or frame comple- tion (see Fig. D. Achieving high perceptual quality in those tasks requires generating re- stored videos whose spatio-tens temporal distribution
estimation observation
missing
matches that of natural videos. Particularly, an incorrect temporal distribution may lead to flick-
ering artifacts [4] or to unnaturally slow dynam-
ies [4] To comply with this requirement, the restoration method needs to 'hallucinate" motion whenever the dynamics cannot be accurately de- termined from the measurements. For example, it may be impossible to determine whether a car is standing still or moving slowly from just a few noisy frames. yet the restoration method must generate some (nearly) constant velocity in order to comply with the statistics of natural videos. However, as more measurements become avail- able, the uncertainty may be reduced, and it may become evident that the hallucinated dynamics were in fact incorrect. When this happens, the method cannot suddenly change the motion in the output video, because such an abrupt change would deviate from natural video statistics. Thus, although the method "becomes aware" of its mistake, it may have to stick to its past decisions for a while. A natural question is, therefore:
Figure The temporal consistency dilemma. Estimation cannot suddenly change the motion in the output video, because such an abrupt change would deviate from natural video statistics. Thus, although the method is aware of its mistake, may have to stick to its past decisions.
What is the precise cost of temporal consistency in online restoration?
In this paper, we study this question in the setting where the signal to be restored, 22, is a discrete- time Gaussian Markov process, and the measurements y/. are noisy linear transformations of the signal's s state. We address the problem of designing a causal filter for estimating Ie from yt, where the distribution law of the filter's 's output, di, is constrained be the same as that of de (perfect perceptual quality). We show that this temporal consistency constraint indeed comes at the cost of increased MSE compared to filters that only enforce the correct distribution per time step, but not joint distributions across time steps. To derive a recursive form for linear perceptual filters, we introduce the novel concept of an unutilized information process, which is the portion of accumulated information that does not depend on past estimates. We provide closed-form expression for the MSE of such filters and show how to design their coefficients to minimize different objectives. We further establish a special class of perceptual filters, based on the classic innovation process, which has an explicit solution. We analyze the evolution of MSE over time for perceptual filters and for non-perceptual ones in several numerical setups. Finally, we demonstrate the qualitative effects of perfect perreptual-qualiine estimation on simplified video reconstruction problem.
Many works proposed practical algorithms for achieving high (spatio-temporal) perceptual quality in video restoration tasks. Bhattacharjee and Das [ improved temporal coherence by using loss that penalizes discontinuities between sequential frames. Pérez-Pelliten- al. [14 suggested a recurrent generator architecture whose inputs include the low-resolution current frame (at time t), the high-resolution reconstruction of the previous one (at time 1), and a low-resolution version of the previous frame, aligned to the current one. The model is trained using losses that encourage it conserve the joint statistics between consecutive frames. Chu al. introduced a temporally coherent GAN architecture (TecoGAN). Their generator's input includes again a warped version of the previously generated frame, where each discriminator input consists of 3 consecutive frames, either generated or ground-truth. They also introduced a bi-directional loss that encourages long-term consistency by avoiding temporal accumulation of artifacts, and another loss which measures the similarity between motions. More recent progress in generating temporally coherent videos includes Make-a-video [16] that expands a text-to image model with spatio-temporal convolutional and attention layers, and
Related work
Models [10] that use

---

diffusion model with network architecture adapted to video. In the context of online restoration, we mention the work of Kim al. [[2] which presented GAN architecture for real-time video deblurring, where restoration is done sequentially. They introduce a network layer for dynamically (at test-time) blending features between consecutive frames. This mechanism enables generated features to propagate into future frames, thus improving consistency. We note, however, that our work is the first to provide a theoretical/mathematic. framework, and a closed-form solution for a special case.
2 Preliminaries: The distortion-perception tradeoff
Let z, y be random vectors taking values in IR" and RRy,, respectively, with joint probability Pay- Suppose we want estimate x: based on y, such that the estimator & satisfies two requirements: (i) It has a low distortion E[d(x, $)}, where :( is some measure of discrepancy between signals; (ii) has good perceptual quality, i.e. it achieves low value of (Pz: Ps), where dz(::-) is divergence between probability measures, Blau and Michaeli [3] studied the best possible distortion that can be achieved under given level of perceptual quality, by introducing the distortion-perrcption function
D(P) = min {E (d)(( æ)] (p ps) - P).
(1)
Pilo
Freirich al. [8] provided a complete characterization of (PP) for the where d is the squared- error and dp is the Wasserstein-2 distance. Particularly, in the Gaussian case, they developed a closed-form expression for the optimal estimator.
In this paper we discuss estimation with perfect perceptual quality, namely P= In this case, g
Thm. 4] implies that if and y
mean, jointly-Gaussian Gaussian with covariances Σg, Ey > 0, and xx' = E [y]. then a MSE-optimal perfect perceptual-quall estimator is obtained by
g w, EE
(2)
where w is zero-mean Gaussian noise with covariance Ene -'E, independent of and v, and Σ the Moore-Penross inverse of Σg., For the more general case where En > 0, similar result can be obtained by using Theorem F2 in the Appendix.
3 Problem formulation
We consider state ny E TR= with linear dynamics driven by Gaussian noise, and observations yk E R that are linear transformations of my perturbed by Gaussian noise,
A + 9k.
k == I..,
y/ Caar rrk
O....
Here, the noise vectors Ql ~ NV(,, and T ~ NO,, R&)) are independent white Gaussian processes, and Zo (0, Pa) is independent of q1, To- For convenience, will sometimes refer to Po as Qo- The matrices As. CR. Qk, R2 and Po are deterministic system parameters with appropriate dimensions, and assumed be known.
Our goal is to construct an estimated sequence X3 (foo...,āy .rr) based on the measurements Y (yo. .......... BT). which minimizes the cost
C(tu. ir)
Σ A==
a [1< 1
(5)
for some given weights ag 2 0. Importantly, we want to do so under the following two constraints.
Temporal causality:
S Ps, (-lvo...... M/s 1)=1),,
(6)
Perfect perceptual quality: : Pxg = Px..
Condition (6) states that each prediction should depend only on past and present observations and on the past predictions. Note that Condition ? requires not only that every estimated sample have the same distribution as the original one, but also that the joint distribution of every subset of reconstructed samples be identical to that of the corresponding subset of samples in the original sequence. the context video processing, this means that not only does every recovered frame have to look natural, but also that motion must look natural. This perfect perceptual quality constraint what sets our problem apart from the classical Kalman filtering problem, which considers only the causality constraint. Since we will make use of the Kalman filter, let us briefly summarize it.
3

---

The Kalman filter (no perceptual quality constraint) Let 2 - [zx|Boo. Yal denote the estimator of 25. based on all observations up to time which minimizes the MSE. The celebrated Kalman filter [ is an efficient method for calculating the Kalman optimal state & a recursively without having to store all observations up to time k. is given by the recurrence
$ A + K&IN.
(8)
where Kg is the optimal Kalman gain 11I whose recursive calculation is given in Algo. 2 in the Appendix. The vector I is the innovation process,
Is = yk C2 1/K-1
(9)
describing the new information carried by the observation ys over the optimal prediction based on the observations up to time Is - 1, which is given by fk|k-1 = - The innovation Iş is uncorrelated with all observations up to time k - 1, which guaranties the MSE optimality of the estimation. The calculation of the Kalman state is also summarized in Alg. 2 Pay attention the innovation process Ig. its covariance S₁ and gain Kg, which we will build upon. Our notations are summarized in Table in the Appendix. Note that since the Kalman filter minimizes the MSE at each timestep. it also minmiizs (5) regardless of the choice of agg, but it generally fails to fulfill (7). As we will see later, when taking into consideration, the choice of ag does affect the optimal filter.
cumporally-ineonsisten perceptual filter A naive way to try to improve the perceptual quality of the Kalman filter would be to require that each by: be distributed like rı (but without constraining the joint distribution of samples). In the context of video processing, each frame generated by such a filter would look natural, but motion would not necessarily look natural. This problem can be solved using the result (2) which gives the optimal "temporally-incoonsstet perceptual estimator
stic I'k
J wk = Fe (A + KgTa) + wk:
(10)
with Tht and Wyy from 2 These quantities depend only on the covariances 2yy Rt. which can be computed recursively using the Kalman method. The MSE of this estimator is given by (see (8)
E 1| dx 12] di Tr Ez, + Ex; - 2 Es Ian
(11)
where di is the MSE of the Kalman filter, which can also be computed recursively.
Our setting (with the perceptual quality constraint) Going back our setting, one may readily recognize that perceptually reconstructing the signal X from the full measurement sequence Yo
is also special case of the Gaussian perceptual restoration problem discussed in Section 2 only applied to the entire sequence of states and measurements. Generally, this estimate already achieves a higher MSE than the estimate that minimizes the MSE without the perceptual constraint. However.
in our setting we have the additional causality constraint (6). Requiring both constraints 77) and (6)
might incur an additional cost, as illustrated by the following example, where applying each one of them does not restrict the optimal solution, but together they result in a higher MSE.
and consider the process (xo, 21) (go. 9o). where qu NN 1). 21). Assume we want to minimize the error at time k = 1 (namely (ag, a1) = (0. 1) in 5 ). Then, considering only the causality constraint (6). the estimator (2o, )1) = (yo. y1) is optimal. Indeed, it is causal and it achieves zero MSE. Similarly, considering only the perceptual quality constraint 7 the estimator (fo. .1) (y1, y1) is optimal. Indeed, it is distributed like (ma, 21) and it also achieves zero MSE. However, when demanding both conditions, to must be based on no information obey 6 (as yo 0), and it must be drawn from the prior distribution N(0, 1) in order to be distributed like xo and obey (7). Furthermore, to satisfy 7 we must also have ét = do. Therefore, the optimal estimator in this case is (.0. = (ğo, To). where To ~ (0, 1) is independent of qo- The MSE achieved by this estimator is 2.
Example 3.1. Let T
with observations (yo. 8/1
4 Perfect perceptual-quality filters
The perceptual constraint ? dictates that the estimator must be of the form
Aq + Jk. to Jo.
(12)

---

where J4 = 2x A 1 is distributed as N(, Qx.) and is independent of X Note the similarity between (12) and the MSE-optimal state in which J K,I,. Here, however, this choice not valid due to the constraint on the output distribution. In terms of temporal consistency, an estimator of the form (12) guarantees that previously presented features obey the natural dynamics of the domain, while newly generated estimates do not contradict the previous ones. In order to maintain causality 66. Ju must be the form
Jk PJu (-1yo..... -......., Uk S0 - Fx-1);
(13)
i... Ju is independent of future observations Y given YD As consequence, Js is uncorrelated with Iatn for all n 2 1.
We now discuss linear estimators, where Yo and Th (hence x)) are jointly Gaussian. Our first result is follows (see proof in App. B).
Theorem 4.1. Under the cost 3 there exists a linear optimal estimator of the form
J& III + UR wky
(14)
TA E Xy and Φk E R (km,) ((mm)
are the filter's coefficients, wy is an independent Gaussian noise 2x Lu 0x ž Up the process of unutilized information
with covariance Enp Qu S4
2 I Uk
8
I Column IIo....III-1)- -0= = O.
II-13,,
(15)
Note that with this form for JR. the state && is indeed a function of the observations Y² and the previous states & Ak-1, Intuitively, ug is the part of the information in the observations, which has no correlation with the information used to construct the past estimates X6 k-1 Thus, from the standpoint of the filter's s output, this information has not yet been introduced. As opposed to the innovation Is. the process V2 is not white, and it is affected by the choices of ग. and &i up to time
However, I& is always independent of ug, since I& is independent of I6-1 and J which constitute Uk.
The filter of Theorem 4.1 is causal but not recursive. Specifically, although is possible obtain De+1,Eous, given the coefficients (Te, 6b) (see App. E). the di- mension of Us grows with time (it is kny), thus increasing the cost of computing ΦgUK. Further- more, determining the coefficients (TK, du To that minimize the ob- jective (5) (which is done a-priori in an offline manner) requires solv- ing large optimization problem, as the total size of all coefficients On,n,,T To efficiently op- timize these coefficients, we next suggest two simplified versions of this form, which may generally be sub-optimal but easier to optimize.
Wikk
1l
42
I
Y;
yk
Unutilized information
Kalman filter
Perceptual filter)
Y
The state estimator 2t consists of the previous state - and the inno- vation and unutilized information processes. The unutilized information state 7 is updated using the previously unuti- lized information Y3 and the newly-arriving information Ia- The currently utilized information, arriving from Jr. is then subtracted from Tx+1-
Figure 2: Recursive perceptual filtering (Sec. 4.1
Remark 4.2. A remark is in place regarding objectives beyond the squared-error. While (14) forms an optimal filter under the cost 5). it can be considered as represen- tation for linear filters general. The constraints on the coefficients (Qx S. Eu of > 0) are necessary and sufficient for perfect perception, regardless of the cost objective. It is therefore possible to optimize coefficients for objectives other than MSE under these constraints to obtain optimal perceptual linear filters. Optimization may be performed numerically (when the cost is tractable) an online fashion, assuming access to ground-truft samples. Note, however, that under general distortion measures, optimal filters might be nin-linear.
5

---

4.1 Recursive-form filters
The optimal Kalman state 2g achieves the minimal possible MSE, given by d E [1 I ||
Tr { where Pajk is the error covariance, given explicitly in Alg. 2 By the orthogonality
principle (see e.g. g Lemma 2)), any other estimator 2y based on the observations Yo satisfies
E || - .x 12] = dx || +
(16)
Now, consider estimator ât of the form 12 and let D₁ - Bl 2x) 6x) Since choose J3 to be normally distributed and independent - it is easy to see that D3 obeys the Lyapunov difference equation
D3 A + K + +0k
K [ - B[
(17)
As we see, the choice of Ju affects current (and future) errors by its correlation with the two independent components, (A KgIx). Let us now consider filters of the form
Jx A + IIgKII + wks wk N (0, Ewe),
(18)
where here, by slight abuse of notation, we define the process of unutilized information as
T f - [ ... (....li.....fA..... To = 0.
(19)
The matrices g1. 4s E 1 are coefficients such that
Ewg = Q t Ar A o M& 2 O,
(20)
where we denote the Kalman update covariance by M₁ & K,S.KT This guarantees that
Jg, Qk). desired. Importantly, opposed to th. the dimension of Y₂ is fixed, namely
does not grow with time k. Note that since شي , is a linear combination of (Io...., ..... (18) is
special choice of मe and on in (14) where coefficient size does not grow with i as well. Tr and its
covariance Σr, are given via a recursive form, illustrated in Fig. (and derived App. E
T A&T + KRIN Vn ΣT.I Ar Al + Mr. V.Q.V
(21)
Vx A M&III AgEr, A,
(22)
Note again that unlike the innovation It. Yx might not be a white process, but we have that Tę is independent of the filter's 's output X &--1 and In Equation (17 now takes the form
D2 A A Qu + M Ms Mell AR AT o1 A
(23)
where we observe that Er, may depend on the choice of (11, 21)- In order to retrieve an optimal filter, one should perform optimization of the desired objective over {II, 1 under the constraints given in (20 From (16). minimizing the cost (5) boils down to minimizing Ek-o Tr {Dx) subject
the constraints (20 which is an optimization problem over only On parameters.
4.2 An Exactly solvable reduction: Perceptual Kalman Filter
We now consider an additional reduction, which allows to obtain a closed form solution for the filter's coefficients. Specifically, reduced-size filter can be obtained by using the form (12) and (18) with the sub-optimal choice & = 0, namely
Jx = II1KKII + wi.-
(24)
The meaning of this choice is that only nowly-observed information is used for updating estimation at each stage, while non-utilized information from previous time steps is discarded. We note that such simplification should be used with discretion; while requiring only half of the computations, rejecting the unutilized information might lead enhanced errors in some settings (e.g. when an observation is missing). However, in many cases where observations are informative and different timesteps are weakly correlated, past unutilized information rapidly becomes irrelevant and can safely ignored. We demonstrate the utility of this reduction in Sec. Here, 11, is F2 X ny coefficient matrix, and

---

Algorithm Perceptual Kalman Filter (PKF)
Input: Kalman Filter outputs {Ig. Kg, Ss 17 weights {ax (o= matrices A, Po. (Qu satisfying Im Bxk Im {Qx} for all k 0 D.
initialize: 5o To Koyo + wp. wo N (0, Po ))) where Ho is given by (30) and
Bu = Ei= ar (AA) for k = to T do
calculate: M₁ = KES BR = Ei= at (A A , BBM,
compute optimal gain (30): 1 = Q (MBQuMB) M M B.M.
sample: / (0, Qx II M
update state: it - + IT K + wk-
end for
wg (0, Qs MIM is Gaussian noise, uncorrelated with all other states, observations and noises in the system up to time k. Again, note that Zę is independent of the measurements to
time k - 1, hence this choice makes Jj independent of X These innovation prraatoon-baedddd corrections resemble the mechanism exploited in (8. hence we will refer to optimal filters the form (24) as
perceptual Kalman filters (PKF).
Now, by straightforward substitution,
becomes
D3 = D + Qk + MR Mg MLTTE ==
(25)
where consider Qo Po. Mo Egg and D_1 0. As before, minimizing boils down to minimizing Σi= Tr {D and in order for 24 to be well-defined, we should enforce the constraints Q2 11 M > 0, k O, o....,T. T. For simplicity, we consider the time-invariant case where A A, so that the optimization objective becomes
min fI
E Tr {D+}
8.t.
D₁ Et=o A 2, - 2.-II,M,
-M-MI,,,AAA - M (AT)*=1 k 0
(26)
Q₁ 11I.MlI. = k= O.....,,T
where we denoted 2x Qs + Mr. Substituting DE. we can rewrite the objective
ΣTH Tr Σ a.A'
- Mc] (AT)
(27)
As see, optimization over particular coefficient IT, does not affect other summands of the external sum. Therefore, each ПI can be optimized separately. Minimizing the cost at the k-th step is equivalent to
max Tr TT
M& Σ at (A A'
s.t. Qs M =
(28)
Let us denote B& Et=R az (A' - =ax++ A BB+1A. A As we now show, this optimization problem possesses closed-form solution under a mild assumption (which is satisfied e.g. when Q 0). The proof is given in Appendix F
Theorem 4.3. Assume that Im BB B&} C Im (Qk) for every k. Let MB & B& and
denote Ω {Ilx Qk -II. > 0}. Then the optimal value in 288 is given by
max Tr (II, IH{ (M 2 1/2 2).
Q
}
(29)
HeE
and is achieved by the optimal coefficient (which is generally not unique)
m 122 (M !22 1/2t 11/2BBn Bk-
M/
(30)
For closed form solution under the alternative assumption that Im [M&} C Im (Qx). as well as discussion of stationary filters, please see the Appendix. The Perceptual Kalman filter (PKF) obtained from Thm. 4.3 is summarized in Alg. |
7

---

Numerical demonstrations
We revisit main question: what is the cost of temporal consis- tency online restoration? In ad- dition, as we have seen in Sec. 4.2 the relaxation Øx = 0. yielding the Perceptual Kalman filters, reduces the complexity of computation, pos- sibly at the cost of higher errors. is natural, then, to ask what is the cost of this simplification. In the following experiments, we com- pare the performance of several fil- ters; Fial and ætie correspond to the Kalman filter and the temporally-
Table
List of demonstrated filters.
perfect-perception
description
def.
per-sample
temporal
kal
Kalman filter
Alg
X
X
Per-sample quality
Pric
(no temporal)
Popt
Optimized filter
Edirect
Directly optimized
App.
PKF
Pune
Alg.
(total cost)
PKF
imin'
Alg
(terminal cost)
quality). The estimate Łopt filter obtained by numerically optimizing the coefficients in (18), where the cost is the MSE at termination time, CT = Billăr - ZT
inconsistent filter (1O) (which does not possess
perfect-perception filter obtained by the optimization approach discussed in Appendix C where we consider again the terminal cost. The estimates Panc; minn' correspond to PKF outputs (Alg. [ minimizing the total cost (area
-r|1 [|]] Fdirect
under curve) Caue = "I= k= Billa xx and the terminal cost Cr,, respectively. The filters are summarized in Table I Full details and additional experimental results are given in App.
5.1 Harmonic oscillator
We start with a simple 2-D exam- ple. Specifically, consider har- monic oscillator, where the state Ia R² corresponds to position and velocity, and the observation at time is noisy versions of the po- sition at time - A,, where A, is the sampling period (see App. for
12.5
15.0
RMSE (saniyrical)
12.5
Asp
20
Gap to empporal consistercc
details). Figure 3 shows the MSE for Łopt and the sub-optimal PKF outputs Bauc: T. The estimates fial and Peic achieve lower MSE than Topt. however they do not pos- sess perfect-perceptual quality. The difference in MSE between the fil- ters Copt and îrie is the cost of tem- poral consistency in online estima- tion for this setting.
5.0
Aus
(i) Gap due
Fax
4
SuilT
[tiime)
Figure MSE on Harmonic oscillator. We observe (i) the difference in distortion between the cope. optimized according to (18). and étie- This additional cost is due to the perceptual constraint on the joint distribu- tion. Also note the gap (it) between MSE of the optimized estimator and Tmint which is due to the sub-optimal choice of coefficients, dx
state
In Figure 3 we also observe that the PKF estimations are indeed not MSE optimal at time T = 255. However, their RMSE only about 30% higher than that of &opt and they have the advantage that they can be solved analytically and require computing only half of the coefficients (TT,). The penalty related to this reduction may vary, depending on the exact setup. In Fig. 4 we demonstrate the relations between gaps due to temporal consistency (gap between dddrectt and áue) and due to the reduction & (gap between Ltirect and & minn)), in various settings based on the Harmonic Oscillator example. On the left pane, dynamics are driven by I = pAæ + gk. where scalar controlling the strength of correlation between timesteps. On the right pane, P 1 and observations are given only up to time 7 (iik is only noise for l 2 T). We observe that for systems where timesteps strongly correlated, or in the absence of current information, discarding U% leads
a fixed marginally-stable matrix, and is
Our code is publicly available at http:://iitbbboom/toooguu//////p -ateerrreepptaaaaaaaanaaaaaaa talten-filters

---

14
Pellenas
12
10-
SminT
Fditect
46
63- Sike
0.05 603
log
Figure 4: Different dynamics yield different MSE gaps. Here we use different dynamics, based on the Harmonic oscillator (matrices A, C, Q R are given in Sec. [.T. to demonstrate behavior of gaps due to temporal consistency (gap between ddirect and Ftie) and due to the reduction & (gap between &direct and m)n We present analytical errors at time T = 255, normalized by the ground-truth state Øт variance. (Left) dynamies are driven by the stabilized matrix pA for different scalar values of p. Observe that when timesteps are more correlated (higher p), both consistency and unutilized information play major role, hence MSE gaps between filters grow. (Right) Here, p and observations are given only up to time (yk = noise, for ls T).
to significantly higher errors. Whenever states are less correlated and more observations are given, unutilized information can be ignored with lower cost.
5.2 Dynamic texture
We now illustrate the qualitative effects of perceptual estimation simplified video restoration setting. Specifically, consider video of "dynamic texture" of waves in a lake. Such dynamic textures are accurately modeled by linear dynamics of a Gaussian latent representation (5. whose parameters lear from real video. Here, frames generated from latent 128-dimensional state = which corresponds to their Factor-Analysis for more details). Thus, 512 x 312 x 3 frames in the video domain created through affine transformation of =f Linear observations
(FA) decomposition (see e.g. [2] Sec. 12.2.4]
E x2 are given in the frame (pixel) domain, by 16x downsampling the Y-channel of the generated ground-truth frames, and adding white Gaussian noise. All filtering is done in the latent domain, and then transformed to the pixel domain. MSE is also calculated in the FA domain. The exact settings can be found in App. [
In the first experiment, measurements are supplied up to frame = 127 and then stop (Fig. 5). letting the different filters predict the next, unobserved, frames of the sequence. We can that until frame A = 127, all filters reconstruct the reference frames well. Starting from time A = 128, when measurements stop, the Kalman filter slowly fades into static, blurry output which is the average frame value in this setting. This is non- 'realistic' video; Neither the individual frames nor the temporal (static) behavior are natural to the domain. Our perfect-perceptual filter, Sangs keeps generating a 'natural' video, both spatially and temporally. This makes its MSE grow faster"
We now perform second experiment, where measurements set to zero until frame A 512. At times k > 512 they are given again by the noisy, downsampled frames as described above. Fig. present the outcomes of the different filters. We first note that up to frame & 512, there is no observed information, hence outputs are actually being generated according to priors. The Kalman filter outputs static, average frame. The filter tric randomizes each frame independently, leading to unnatural random movement with flickering features. At frame à 513, when observations become available, t and tūe get updated immediately, creating inconsistent, non-smooth motion between frames 512 and 513. The PKF output fance on the other hand, maintains a smooth motion. Since the outputs of inconsistent filters rapidly becomes similar to the ground-truth, their errors drop. The perfect-percepten filter, Lanc, remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.
'More visual details, including ground clips and empirical error can be found in the Appendix. Full video clips for both experiments supplied with the supplementary material.

---

En (onrs)
k=127
256
k-2047
MSE (smndytical) ballyttiaal
(time)
Figure 5: Frame prediction dynamic texture domain. In this experiment, measurements supplied only up to frame & 127 and the filter's task is to predict the unobserved future frames. Observe that Gial fades into blurred average frame, while the perceptual filter aue generates natural video, both spatially and temporally. This makes its MSE grow faster.
Saor (ours)
k=511
512
k=513
k=514
MSE (anslytiosn beal)
20
2001 1003
(time)
Figure 6: Frame generation dynamic texture domain. In the first half of the demo (k S 512), there are no observations, hence the reference signal is restored according to prior distribution. The filters with no frames (Kalman filter output ( or unnatural motion (&tie). Perceptual filter Zane is constrained by previously generated frames and the natural dynamics of the domain, hence its MSE decays slower.
quality constraint the temporal domain generate non-realistic
Conclusion We studied the problem of causal filtering time sequences from corrupted missing observations, where the the filter's output process is constrained to possess perfect (spatio-temporal) perceptual-qualitty in the sense of being distributed like the original signal. Our theoretical derivations focused on Markov state-space processes. We introduced the novel concept of an unutilized information process and established a special class of perceptual filters. coined Perceptual Kalman Filters (PKF), that are based on the innovation process alone. We demonstrated the qualitative effects of perfect perceptual quality estimation on a video reconstruction problem. To the best of our knowledge, this is the first work addressing the settings. This work paves the way toward understanding perceptual online filtering, and the cost of temporal consistency in sequential estimation problems.
tradeoff in online restoration
10

---

Acknowledgements The work of RM was partially supported by the Israel Science Foundation grant 1693/22 and by the Skillman chair in biomedical sciences. The work of TM was partially supported by the Israel Science Foundation grant 2318/22 and by a gift from KLA. RM and TM were supported by the Ollendorff Minerva Center, ECE Faculty, Technion.
References
[1] Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep mtati-stage generative adversarial networks. advances in neural information processing systems, 30, 2017.
[2] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[3] Yochai Blau and Tomer Michaeli. The perseption-diitortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6228 2237,,23, 2018.
[4] Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for gan- ban-based video generation. ACM Transactions on Graphics (TOG), 39(4):75-1, 2020.
[5] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. International Journal of Computer Vision, 51:91-109, 2003.
[6] Dror Freirich and Emilia Fridman. Decentralized networked control of systems with local networks: A time-delay approach. Automatica, 69:201-209, 2016.
[7] Dror Freirich and Emilia Fridman. Decentralized networked control of discrete-time systems with local networks. International Journal of Robust and Nonlinear Control, 28(1):365-380, 2018.
[8] Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the distortion tradeoff in wasserstein space. Advances in Neural Information Processing Systems, 34:25661-25672, 2021.
[9] Robert M Freund. Introduction to semidefinite programming (sdp). Massachusetts Institute of Technology, pages 8-11, 2004.
[10] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprini arXiv:2204.0344 2022.
[11] R.E. Kalman, A new approach to linear filtering and prediction problems. Journal of Basic Engineering. 82 (1) 35-45, 1960.
[12] Tae Hyun Kim, Mehdi S. M. Sajjadi, Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for video restoration. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[13] Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
[14] Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. Perceptual video super resolution with enhanced temporal consistency. arXiv preprint arXiv:1807.07938 2018
[15] Alexander Shapiro. Extremal problems on the set of nonnegative definite matrices. Linear Algebra and its Applications, 67:7-18. 1985.
[16] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. a-iee:- Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[17] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM review, 38(1): 49-95, 1996.

---

Perceptual Kalman Filters: Online State Estimation under a Perfect Perceptual-Qualit, Constraint Supplementary Material
In App. A we provide detailed theoretical background on the Kalman Filter, its properties and recursive calculation. In App. prove that under perfect perceptual filtering setting, there exists linear optimal filter (Thm. 4.1). In App. C we discuss a direct, non-recursive method for optimizing perceptual filter coefficients. In App. D we present the derivation of the Lyapunov equation we derive the recursive expression for the filter given in (18). In A we find closed-form solution for PKF coefficients by proving Theorem 4.3 In this appendix, we also give some brief overview on the extremal problem of finding a minimal distance between distributions. App(aioonaans discussion about stationary perceptual Kalman filters in the steady-state regime. We summarize all definitions and notations in App. H Finally, in A we give full details for all numerical demonstrations, and present additional empirical and visual results. More results are provided in the supplementary video.
(17) for the error of perceptual filters. In App. E
A The Kalman Filter algorithm
In this Section supply a detailed reminder of the Kalman filter Algorithm. The celebrated Kalman filter [T]] assumes a state n R where dynamics are modeled deterministic linear functions perturbed by a Gaussian noise, and observations E R"> are linear functions of I& with an additive noise
It -A + 4k.
4s NO Q&), Th ~ N(O,
k == 1, T. k: 0, ....
(31)
3/k =CkIk Tk:
(32)
The noise terms Q& and rk are independent white Gaussian processes with zero mean and covariances Qk, Rs. respectively. vp is assumed to have zero-mean Gaussian distribution with covariance Po. independent of 91, ru- For convenience, we will sometimes refer Po as Qu- The matrices Ag. Cr. RR and Po are system parameters with appropriate dimensions, and assumed to be known. Considering the MSE distortion, we denote
sxje argmin E [| - 131 ... yo
(33)
namely the optimal MSE estimator of the state at time k. given measurements up time 8. Under the assumptions mentioned above, Kalman filters produce the mean state estimate &gik. an MSE-optimal estimator of zg given the observations up to time k. The Kalman optimal state st Sypk is given by the recurrence
i A + KRIs.
(34)
where Kş is the optimal Kalman gain given explicitly in Algorithm 2
The vector Ig is the innovation process.
I = C4R
(35)
describing the contribution of the new observation yı over the optimal prediction based on previous observations. Since are in the orthogonal to the measurements Vo, Wh-1- 1, guaranteeing MSE optimality of the estimation. The calculation of Kalman state is summarized in Algorithm 2
setup, we have that the innovation state I, is
12

---

Algorithm 2 Kalman Filter
initialize: 20
Koyo
PoCo yo,
Pojo = Po PoC Eyg PoPo. = you So
CoPoCo + Rg.
for 1 do
calculate prior: fxjk-1 1IR
PaR-1 A&Pk- 1|k + +xx
calculate Innovation: IĄ yk CKE
Sk = Ck PRIE--11C Rs
Kalman gain: K₂ = k
k- C/ Sk
PAJR (I -I- (K-KaCk)PABB update (posterior): 1 02/2 Bb/N-11 + KEIK,
end for
13

---

B Optimality of linear filters (proof of Thm. 4.1
In this section we show that under family of optimality criteria (5) and perfect-percepuuall quality and causality constraints 7 linear filters of the form (14) are optimal. We start with the following.
Theorem B.I. Let Yo (yo....., -y) be the set of measurements 4 and let (Jb Yo") be a joint distribution s.t. Jg is independent of tk given Y for all & E [o. T and n Then,
B [JII 0.
(36)
Proof. Denote J/ = E [Jx [ We can write the measurements as linear function of the innova- tions, yk: = Σt- HI. We have
k+n-1
ktn-1 2 Mo+n 1 yktn-1
E
Wktn
[
Hk+n,eh,
(37)
and
ohtn A E (o [vt+n IZ
Σ Hatn,
[ott 1
(38)
For any h and n Ix+1 yk+1 1 and therefore
E [
E Ju [UK+1 - zh [Z
=
IE
Ju [ohh
[Jx[Z]]]
[ = 0.
(39)
This is due to the facts that J/ and yk++1 are independent given the condition, and that sh is deterministic function of Z
Now, assume we know that [ for 1 1515 n
We can write
J
E
JR
[z
k+n-1
E
8& [8k
T -
E
Jk
I H 1 I
1=0
k+n-1
E
J& [uk [
E
Jk
I
H ktn,
- E
Jj Σ I
H
ktn
IT
t=0
tim k+1
k+n-1
E
Jn [
- Wktn
Σ 1=++1
E
JRIt [Z
Hatml
k
Σ
E [JAI]]]
1=++1
= 0.
(40)
We now show that for every filter which is feasible under (6) and (7). one can find linear filter, jointly Gaussian with the measurement set, attaining the same cost objective.
Theorem B.2. Let YJ" (yg.... "y......yr) yr) be the set of measurements (4, and let JJ (Jo Jr) be jointly distributed with Yo such that:
(i) J. N (0, diag(Po. Qı.... QrI).
(ii) Jk is independent of yktn given Ya for all ls [0, and n 2 1.
(ii) Σ ag. [||ar - [2] C. where Xk is the process given by Xk AaXk-1 + Ts with
Xo = To.
14

---

Then, there exists a joint Gaussian distribution (Jo Yoy) in which and 1 hold, and the estimator given by
nx A + Jkr to Jo
(41)
achieves the same cost namely Σk=0 0rk . [1|11x 1x C. Furthermore, we can write
Ju = TTIxx + Uk + wwk
(42)
where
Uk =
(43)
and Wy. is white Gaussian noise, independent of Yo² and Jú 1
Proof. Let (JT Yo") be the Gaussian distribution defined by the moments of (To Yo up to second order. We observe that from Theorem B.I above. Jhe is independent of all future innovations I=+n. namely it is based only on measurements up to time k. Using the notions of Theorem B.I proof.
E (Jk - (yn+ vhtn) 180
E
(J& Jk)
I H
t= k+1
k+n
E J
I
J&B Ing [281
k=
k+t
Σ k=++1
E [
[Te Ib [ H [
Σ
E
Io I IIO Harao [
Σ k=++1
IJ [zt]] Hatno [J
k+n
E [J
[
[Z8] H
H
1=
0.
(44)
This means that Ju and yk++ are independent given Y which proves
From (17) we see that the cost functional depends only on the second order statisties of (Jr ITg'
which are identical to those of (JT I hence (il) holds:
T
E
11 - 2x
Σ k=
agE
1| - x =C C.
(45)
To prove 44). we now write
J3 = 5k + Wk:
(46)
where EN
E [Jalo JK-11 and wA Ju - E [J Jb is independent of Y and JØ-!. Now, since both Ju and J j are independent of I
k
R-
Es = E. [JIVo Jo - [ J 21
Σ 1=
dr I, +
(47)
1=0
J& is independent of J thus
E [ k -1 = [JA[I] J 11 1 0.
(48)
15

---

Conditioning both sides of 47 on J0 Jk-1 and taking expectations,
whare.
(49)
Note that E [ā = 0, which together with (49) implies
Ek = D *+
[I
[6 '11 = TT + UEUK
(50)
Now, all we have left to show is that wy white sequence. Since wh+n (n 1) is independent of Jó and Zo (which also constitute Uk), it is easy to obtain
E Wk+ weg = wk+n [Jx - aRIN - -brekl =
(51)
Corollary B.3. Given a cost objective of the form C Σ oz [1|:0 fx 1... linear filter of the form
there exists
Jk II + ØRUK + wk:
(52)
such that
Bo Jo
(53)
5t A + Jk, k=...,,
(54)
is an optimal estimator under the perfect perceptual quality and causality constraints 6
Proof. Under the perfect perceptual quality constraint, an estimate sequence X must satisfy that
Js = X AsXk-1
(55)
a white Gaussian process with covariances Qs. If, in addition, X satisfies the causality condition (6). so does Js. We conclude from Theorem B.2 that there exists causal linear filter J4 that achieves the same expected objective C XR-
Now, note again that from [1]), for perfect-percpptual quality causal filters, the objective is continuous function of the covariance matrix
(1b) [digg[ (Ps Q1. ........0 Qr} 7,)
L diag S, , Sr) = 0,
To (I6)
(56)
where, due to the causality demand, L is quasi lower triangular matrix. The set of such feasible matrices is non-empty, closed (since it is the intersection of the closed cone of PSD matrices with finite set of hyperplanes) and bounded. Hence, attains minimal value on some joint distribution PJz YT which can be chosen to be loint-Gaussin as we have seen.
16

---

C A Direct optimization approach to perfect-perceptune quality filtering
For the sake of completeness, now discuss method for optimizing non-recursive perfect- perceptual quality filter coefficients. This approach leads to convex programs. However, we will see next, might become impractical for large configurations.
Let J= Jo ~ N (0, Q), where Q = diag {{Qx) be causal function of the measurements, == + W, where I = Io is the innovation process with covariance = diag {Sk} and W is an independent noise. Now, X X A, is the filter's output, where
7
0
A
1
A,
(57)
T-1
IT Ar-k
Ar-k
Recall X is the Kalman filter output given by X = A,KI, where diag {Ka}. Let W
diag {ax @ In, be weighting matrix. The objective (5) is now given by
C(X) = E (X
(X x*)
Tr
WE
X8
+ W [x
2WE {
(58)
Hence, we have to maximize
c(Φ) = 2Tr WE [
2Tr {W ASKST A]
-ITh ((DS)K] A]
= 2TT (T5K B}.
(59)
where A A]W Ay. This subject the perfect perceptual -ualityy constraint
Q
s
Q - $SΦ1 0, or equivalently
0,
(60)
S$
s
where Φ is lower quasi-triangular matrix (causality constraint)
P0.0
0
11.0
11,1
(61)
TT.0
Φ
Again, under this formulation,
J = +I W.
(62)
where W N (0.Q T5Φ is Gaussian noise independent of I. Note that W might not be white sequence in this case, since covariance might not be block-diagonal matrix. As result, the noise sequence has to be sampled dependently. Also note that this problem possesses the same memory complexity as (14). To conclude, this method leads to convex, but large optimization programs, and is impractical for high dimensional settings long temporal sequences.
17

---

D Derivation of eq.
(17
Recall o is the optimal Kalman state at time achieving MSE given by
di = | Ix | = Tr { {
(63)
Pals is the enror covariance, given explicitly in Algorithm 2 By the orthogonality principle, for any estimator S& based on the measurements yu. V/ we have
[ i사 ||]] = [|
[1] + IE [1| *x = dx + [| -x [2]
(64)
Now, consider an estimator Ty of the form 12 and recall
& E
(65)
Since we choose Js NO, Qk) be independent of ă and Iy is indepenedent of dx-1 and oh k-11 we write
DR =
E [ Txl 10% ]x4]
[A A + Js - KIIn] [A A + J. KIIN]]
AGE [ x 1] [ - xx-1 1]
A + [ + K&E [III ] KE
-E Jk [A 1 + KI ']
[[
[
K JK
DA- A + Kg + Qk
-E [JIN K K [T AgE 1 J+
E [J ]] A
(66)
18

---

E Derivation of recursive perfect-perceptual quality filters
We now derive the recursive expression
for the filter given in (8).
K A +
Jk:
(67)
J& = Φ A &x +
KLI + wk. wh ~ N (0, )w,),
(68)
defined by the coefficients
{ tt 1 fulfilling the constraints
Recall
Y 1 E [
x Jou Jk
(69)
where 27 is the Kalman state.
JØ-1 Th, Ias wy are
and independent, and we have
E Jn
2n
(70)
E J S K mIp
(71)
[
= En, Al OT.
(72)
We can write
T ArT = =;
A k-
[E | AglE [ 1.0
K K₁ [I
A&
[ [ |J6]
E
11 "1]
(73)
Since Jh is an independent sequence, and since I depends only on Jk.
K& I& J K3 Ix J&]
S mI Q.
(74)
We also have that nh Js are independent of _ implying
1 E [
E [ E - 155-111-111 [5]]
= =E [Tc|][6] = E JAc
-E 0 Jh.
(75)
Hence,
M A& + KAI V. Q.
(76)
where we denote
U A MRLl AsErr A7
(77)
The covariance is then given by the recursive form
Σr--1 ArEr, AT + Mr Q
ArEr, AT Q/ K&S mI Q.
(78)
[ArEn AT VTvTT]] J
- -[KuSAKK )
(79)
Av A + Mr VVQ.VT
(80)
At time k: = we have Y, = 0 and Σx, = 0. Remark E. (The non-reduced case). For the full, non-reduced linear filter 14 following similar formula
15
we have the
Uk
Ix-1
-k-1
Sh-1
0
x1-
n
Q
Jk-1
(81)
and
Eny,
S
Sk-1
n T
Q
T @
S
0
(82)
Σpe-1
0
Eve-1
0
Σ
Notice, however, that the dimension of Uk grows with time k.
19

---

A Generalized extremal problem with semidefinite constraints (proof of Thm. 4.3
In this section we prove Theorem 4.3 We start with a brief overview of the extremal problem of finding minimal distance between distributions, and of general semi-definite programs.
To prove the Theorem we observe that (28), is generalization the extremal problem, and suggest non-trivial dual form where, under our assumptions, strong duality holds.
F1 Minimal distance between distributions
Consider two Gaussian distributions R with zero means and PSD covariance matrices Σ1, Σ₂ respectively. consider the problem constructing a Gaussian vector [X, Y] minimizing B|| Y|l while inducing the given marginal distributions, X ~ / (0, Σ1) Y N (0, Σ2). This problem
equivalent to the following maximization of correlation [3
Σ
II
Tr (2 + max, S.L.
0.
(83)
II
Σ2
We have the following results of Olkin and Pukelsheim [3]
Lemma F.1. [13 Lemma 1J. Let 2g be any generalized inverse of E2. Then E 2
iff
IT and E. - -TT 00
(84)
Theorem E2. 13 Thm. 41. If Im [E2} } C Im {Σi} J. then an optimal solution to
83
is given by
1/2
max Tr [211} 2Tr
(85)
achieved by the argument
I = E,E//3
- En 1/2
11/2
9/2
E//
(86)
In the case where Im (Σ2) = Im {E1}. IT" is a unique optimal argument.
Under the setting discussed in Sec. Σg O, the MSE-optimal perfect perceptual-quaite estimator (2) is obtained by
2 Theorem
F.2 implies that in the more general case where
i = -" + w, T & E (2.
(87)
Here again, w is zero-mean Gaussian noise with covariance En, = En g' geT, independent of y and x. and Ej. is the Monrr-Penrose inverse Eg-.
F.2 SDP Setting and duality background
Semi-definite programming (SDP) 9 is an optimization problem in X E RRax of the form
X
(88)
s.t. A; X b; m,
(89)
X > 0.
(90)
Here, A, are real symmetric matrices of appropriate dimensions, and x -Tr(ATa x} the Frobenius product. SDPs yield the Lagrangian
m
v(x,A,v)=v' -
m
X + Apmin (X)
1=0
m
= -
144
min Try
X,
(91)
1=0
where X > 0 and Pmin is the minimal eigenvalue. The Dual problem (DSP) is given by
Tb min, s.t. s.t.C
44 3
(92)
1=0
In this case, strong duality exists iff the SDP strictly feasible, i.e. it has feasible solution interior to the feasible set, X 0. This condition is sometimes referred to as the Slater condition.
20

---

F.3
A generalized extremal problem with strong duality
Recall Qs, MR, Bg are real, symmetric positive semidefinite ny x n2 matrices, and the optimization
problem 28
Tr [T.MM.BBa Tr ( (MMM max, s.t. Qk M&M O. l1s
(93)
Since (93 involves a single time step, we will omit the index k.
We consider II I1M, hence II" MM and since M MM M we rewrite (93) as
Tr [1BB} Tr [BI]] + max, s.t., Q -IIM'III = I1 -M
(94)
By Lemma F.1 the constraints in (94) are equivalent to
Q I
π M
X
= 0.
(95)
This can be formulated as the semi-definite program,
X
=Qis
=
CoX max, s.t.
(96)
AM
X
Mij
where C'
0 B B
and 1
Aij 0
0
A =
0
A.s = (e + czr)-
2
0
Agj
Note that when B is a scalar matrix, (94) is similar to the problem studied in Olkin and Pukelsheim [3]. Their approach was later extended by Shapiro [15] to general linear objectives, where the Slater condition holds.
F3311 Strong duality
The SDP (96) yields the standard dual formulation
PQ + M VM
s.t. PQ.
min
VQ -
1
0, MM X=
(97)
This should give hint about the optimal solution to (94). Pay attention, however, that according to the theory, strong duality in (97) guaranteed only Q. M > 0, which might not be the case (see
e.g. [15]). To get tight bound for the general case Q .M = 0, we provide alternative form of duality to (94).
The following is an adaptation of techniques used in Olkin and Pukelsheim following Lemma.
13
We start with the
Lemma F.3. Let be a feasible solution to 94 R,G nh X are general matrices. Then,
Tr [QRQ + BMBGG 2 2Tr [IBGR
(98)
Proof. From the non-negativity of X in 95) have
-
Q II
R
n
M
BG
(99)
R TQR+G BMBG- R TBB GT
BIT
R 0.
The trace is nonegative, hence we have the desired result.
Remark F.4. Similarly, we can obtain
Tr (QBRRR B MGG") 2Tr (BIGRT),
(100)
Now, suggest alternative to (DSP) 97 where strong duality will hold.
21

---

Theorem F5. [Strong duality]. Let
Q= -([1 R 1Raxx xnx, :Q -Q- > 0, = =M
(101)
- 5 S = BM = SS BM),
(102)
and denote MB - BMB. Assume Im [MB] C Im (Q). Then,
(in (Q ( M (BS B)) maxnen Tr (2I1B)
2TT M 1/2 QM 1/2 1/2
(103)
8
The extreme value obtained for
1 [M (M QM
1/21
M
(104)
S M [M (M/ 1/2 1/2
1//21
(105)
1/2f
II" QS* BM = QMM (M QM?
M BM.
(106)
Optimal solution IIo is generally not unique.
To prove strong duality, we will use the following lemmas.
Lemma F.6. Assume PSD matrices Q, MB such that Im {MB} S Im (Q), then Im {MB} =
Im M 1/2 1 [ B
Proof. Recall MB, M QM 1/2
1/2
B are real symmetric matrices.
Let v E [M M
1/2
2/2 ell 1/2 }, we have /|Q//MM32
0 hence M /er(qu/3 s
Ker ( which yields 2 0, implying Ker 1/2 / S Ra) Opposite relation (
trivial.
We have
Im (MB) = [M) = /MM Im /
(107)
Lemma F.7. Im (BM) S Im (BMB).
Proof. Let v E Ker (cr(BMB). BMB). then | = 0 and Bu E Ker(M = Ker(M), Hence
(BM C Ker (BB. We have
Im (BM) Ker[MB)+ C Ker[BMB)+ Im (BMB).
(108)
L
We now ready prove Theorem
Proof.
[hhoreem E.5 Let II Ω, then X 0 in 995 For any (8
we can choose
R = S
G S R. From the result of Lemma F3 it follows that
Q S (BS B)
Tr (QRRT= BBG
(109)
2 2Tr [IBGRT] 2Tr (BBS S} 2Tr (IIB).
The last equality holds since BM SS- BM, and Im {uT)} Im (M).
22

---

We now prove that II" E L2
Q M
Q QMB 12 QME 1/2
2 BMMMMBM" (M 1/21 2
Q - QM (M (M Q
Q1/2
T- T M @
2 Q1/2
Q1/2
2
= Q1/² I - Q
M QM!!"
M Q112
Q1/2 =
(110)
The last equality holds since it is easy to see that 1 - Q MMPQMMH M Q1/² is
symmetric (orthogonal) projection.
We further prove that S E S. is easy to show that S S are symmetric generalized inverses, reflexive each other (S-* is in fact the Peoor-Penrooe inverse of S*)):
S M !
1/21
1/21 (M (M!
1/2 M
(111)
M/ " QM! 1/21
! 1/2 M
(112)
=M 1/21 MB /22 1/2
(113)
=S*
(114)
The equalities hold since by Lemma F6
Im { (M [m) Im { My / /M Im (Mg 1/2
(115)
and since for a PSD matrix R, RR Rt R orthogonal projection onto its image. Using Lemma we have
S BB = M M BM = BM.
(116)
It is now easy to verify that
(M!?QM!! 1/2)
St + (BS (BS-"B) B) 2Th (II' B) 2Tr
(117)
which completes the proof.
Corollary F.8. Under the assumption, Im {Ms} S Im (Qx), the optimal gain in 28 is given by
II = QuM " 1/21
M/ B
11/2
(118)
Remark F.9. Under the alternative assumption, Im (M&} S Tm (Qx). the optimal gain in 28 is given by
II = BQ.BM (M. (222 1/21
M5
(119)
where B = B122 Qb B, Mo B.
Proof. Recall our goal in is to maximize (IIMB) Tr {BIMB) under the condition
Q - TMMT 2 (we omit the index k)). This is equivalent to minimizing E [ SY|1]] weet
Q
ILM
11, where (X N (0, )) and Σ = MI1
> 0.
M
23

---

BQB BMIT
BIMM
In this case, (Bx, ~N (0, Es) where Eσ
According to Thm. F.2
BMB
}. the minimal distance is achieved when under the assumption Tm [ B3 } _ Im { BQB]
BUMB BQBM (MÉQuM²) št Mg.
(120)
Note that Im {M} S Im [Q} implies Im (BMB} S Im [QB]. and straightforward to verify that Q - II TII m 0.
24

---

G
Stationary settings
note place regarding the stationary perceptual Kalman filter. In the Kalman steady-state regime, where dynamics (31) B2 are time-invariant and T - oo, the matrices K and S in Algorithm are
determined by the covariance matrix P.
= PC PCPCC + +R) S CPC R.
(121)
Here, stands for the time-invariant observation matrix (y) Cay + rk) and P solution the Discrete- Tisceete-Time Algebraic Riccati equation (DARE)
= APA APC (CPCT CR) +Q.
(122)
Similarly, under the steady-state regime,
26
becomes
Tr [D} minn
(123)
s.t.
D ADA + M I1M MII .M KSKT, Q ILMII ž
where obeys an (Algebraic) Lyapunov equation. If A is stable,
D(I) = (Q -IM - MII") (4*) T
(124)
k=
Hence, stationary perceptual filter is of the form
f = A + Jk:
(125)
J₁ IKL + Wkk
(126)
w/ N (0, IIMII",
(127)
and in order to find optimal gain II, minimizing Tr (D(IT)), we have solve
max Tr (IIMB) s.t. Q - IIMI ε 0,
(128)
where define B A E (A*) A and the solution (under the assumption Im (BMB) C Im (Q)) is given again 30
25

---

H List of notations
We summarize our notations in the following Table.
Table 2: Definitions and Notations
Notation
Description
Definition
Dimensions
F2
state dimension
nl
measurement dimension
A
system dynamics
Th f2
C2
measurement function
Vly Tbz
Qk, Rk
noise covariances
11 X D2 X Bhp
system state (ground-truth)
The
Vk
measurements
fly
state estimator
Thg
optimal Kalman state
Algorithm 2
best MSE state esimators, up to time
innovation process
see Algorithm
fly
Sk
innovation covariance
see Algorithm
By Byy
K
Kalman gain
see Algorithm
1h fly
II
innovation perceptual gain
Ty X thr
Mg
Kalman update covariance
Mr SrK,K
Tg X The
Uk
unutilized information process
115
kmy
N
unutilized information process (recursive)
see (19
Yhg
Σr.
unutilized information covariance
The Wh
unutilized information perceptual gain
Th f2
weight matrix
B₁ E (A' A
Th Thg
Dy
deviation from MMSE
D2 = E E
24 [ fx]
X T2g
T
Termination time (horizon)
C(X))
minimization objective
C= k=0 a
I 3/ 1
26

---

I Numerical demonstrations
In this section we provide full details for the experimental settings of Sec. with additional numerical and visual results. In the following, we compare the performance of several filters; aa and Pric correspond to the Kalman filter and the temporally- perfect-percepplee quality). The estimate oppt is generated by perfect-perceplee by numerically optimizing the coefficients in (18). where the cost is the MSE at termination time, Le. the terminal cost
filter (TO) (which does not possess
filter obtained
CT E | xT xT|??
(129)
The estimates Rauc; Rmin't correspond to PKF outputs (Alg. minimizing the total cost (area under curve)
T
Cauc
E [| - .x
(130)
and the terminal cost, respectively. Finally, Retat. is the stationary PKF, discussed in App. G filters are summarized in Table K
The
Table 3: List of demonstrated filters.
perfect-perception
description
definition
per-sample
temporal
Kalman filter
Algorithm
X
Per-sample perceptual quality
Pric
x
(temporallyinnconsstten
Optimized pprretuuall
Copt
188
quality filter
Base
PKF with total cost
Algorithm
T
PKF with terminal cost
Algorithm
Sstat.
Stationary PKF
125
X
L.1 Example: Harmonic oscillator
We start with a simple 2-D example, where we demonstrate differences in MSE distortion between the optimized perfect-perceptual quality filter Copt. the temporally inconsistent filter -tic and the efficient sub-optimal (perceptual) PKE Consider the harmonic oscillator, where the entries of state 24 I correspond to position and velocity, and evolve
Øk+1 Ar + Ok:
Ls (0,I)
(131)
with
0
1 0
A= 1+
x
A.:
(132)
-2
where A, = 5 10-3 observations of the position (corresponding to time t A1) that y/h = [1 Zg + Tk. where rk / (0, 1) and xo A (0, 0.87.
3 is the sampling interval. Assume we have access to noisy and delayed scalar
We numerically optimize the coefficients 4 2x) in 18 minimize the terminal error 129 (Tr {D in (23) at time T = 255 under the constraints (20). Figure shows the MSE distortion for the optimized perfect-perreption filter åopt defined by 18 and {11 ) and the sub-optimal PKF outputs aauc TminT, minimizing the total cost (03) and the terminal cost (129) (see Table 3). We observe that PKF estimations are indeed not MSE optimal at time T', However, their RMSE at time T is only ~ 30% higher than that of &opt and they have the advantage that they can be solved analytically and require computing only half of the coefficients (II,).
The estimates digt and ètic achieve lower MSE than Stpt, however they do not possess perfect- perceptual quality. We can see the difference in MSE distortion between the filters Tops and Stics with and without perception constraint in the temporal domain. This the cost of temporal consistency in online estimation for this setting.
27

---

200
MSE (emppizical)
Iix
200)
200
A: (Lime)
MSE (snmlyttoel)
200
20
k (time)
17.5
15.0
RMSE (analytical)
12.5-
10.0-
Top
Gap due to temporal consistency
7.5
5.0
Late
(ii) Gap due
Louc
to L 0
2.5-
Tmin'T
50
100
150
200
250
(time)
Figure 7: MSE distortion on Harmonic oscillator. Popt is numerically optimized perfect- perceptual quality filter's output (minimizing eπror at time &min are PKF outputs minimizing different objectives. Observe that PKF estimations are not MSE optimal, but require less computations. dtt and Siiie are not perfect- perrect-percpptuall quality filters. (top) Empirical error, over N = 1024 sampled trajectories. (bottom) Analytical error. The difference distortion between the perfect-percepee the joint distribution. This is the cost of temporal consistency in online
255, dashed horizontal line).
state &opt, optimized according to (18), and êtic is due to
the perceptual constraint estimation for this setting. The gap between the MSE optimized estimator and minnT due to the sub-optimal choice of coefficients, & 0.
28

---

1.2 Example: Two coupled inverted pendulums
Next, we demonstrate the quantitative behavior of perceptual Kalman filters, by comparing the MSE between the PKF outputs when minimizing different cost functions, and between non-perceptual filters outputs. More specifically, this experiment demonstrates:
1. How minimizing different objectives in Algorithm leads to different filters.
2. The cost of
quality filtering, given by Algorithm ! over optimal filters.
We consider dither-ddimensonal,, well-studied example of two coupled inverted pendulums, mounted on carts 6]. The cart positions, pendulum deviations, and their velocities (Fig. 8). are given by the discretized stable closed-loop system with perturbation
Tk+1 = Az + 4k.
qk ~ N (0,
(133)
where ny R The initial state is distributed
20 (0,
(134)
The system matrices are given by
= 1=I+Aal -
(135)
where A, = 5 x 10 is the sampling interval and
A + BK F
F BK2
Act
(136)
A2
O -0.0005
07
0 -0.0042
2.9156
0
11101
A1 A2
B
(137)
0
O 0.0002
0 0.0167
1.6663
The coupling is given by
0.0011
0, 0.0005
F
(138)
-0.0003
-0.0002
and stabilizing state-feedback controllers (each acts on single cart) are
K1 [11396.0 7196.2
573.96 1199.0], K₂ = [29211 18135
2875.3 3693.9). (139)
The partial measurements are given by Cr. + Tk. where Th ~ N (0, R), with coefficients
=
C2
0
C - C2
1
0
0 0
(140)
Namely, we observe only position and angle for each cart/pendulum, while velocities are not being measured.
The perturbation covariances given by
Po
0
Q
R
R
Po =
Po
Q
=
0
Q
R
=
1
(141)
where
0.154
0.142
-0.143
0.093
0.142
0.144
-0.124
0.058
Po
.5 x 10
(142)
-0.143
-0.124
0.167
-0.148
0.093
0.058
-0.148
0.192
0.642
0.136
0.78
0.262
-0.136
0.894
-0.248
0.074
0.375
-0.33
10
Ar, R 10
A
0.78
0.248
1.284
-0.314
-0.33
0.771
0.262
0.074
-0.314
1.766
(143)
29

---

MSE (empirical
0.25
0.21
0.15
500
700
800
9011
1000
I: (time)
MSE (analytical)
0.30+
0.25
Fije
0.20-
0.15-
300
1300
VO0
800
9011
1000
f: (time)
Figure 9: MSE distortion on Coupled inverted pendulums for perceptual and non-perceptual filters (near the time 7), Rauc, T PKF outputs minimizing different objectives. Observe that while both possess perfect-perceptual quality, they yield different estimations. Also, pay attention to the MSE gap between the MSE-optimal, but perceptual, Kalman filter and the PKFs.
coupling
angle
angle
Pos.1
Pos.2
Figure 8: Coupled inverted pendulums.
We simulate the system for 200 time steps (7 = 2 1), over N = 21 independent experiments. In
Figures and 10 we show the MSE distortion as a function of time, E [|] - xx|]²] for the different filters of Table &ig is the optimal Kalman filter. Ltic is the perceptual filter without consistency constraints, given in (10p. Laue the PKF output minimizing the total cost (130). TminT (marked by 'x') is the PKF output minimizing the terminal cost (129).
We observe that filters satisfying the perfect perceptual quality constraint (&auc and & achieve higher distortions compared to the per-sample only perceptual filter êtic. which in turn attains MSE distortion slightly higher than that of the MSE-optimal Kalman filter. This demonstrates again the cost of temporal consistency in online estimation. Note also that PKFs minimizing different cumulative objectives, yield different estimations; while #minT is optimal at termination time .ane achieves lower MSE average. As we will next, both filters attain the same perceptual quality.
30

---

MSE (empirical)
Auc
Fux
0.21
FmicT
0.11
2000
400
0X0
800
1000
(time)
0.3-
MSE (analytical)
Ruc
Fux
0.7-
PuicT
0.1
0.1
200
400
EXII
ald 1000
(time)
Figure 10: MSE distortion on Coupled inverted pendulums for perceptual and non-perceptual
filters (full view).
In Fig. 11 we estimate the perceptual quality, given by the Wasserstein distance between the ground- truth distribution and the empirical Gaussian distributions of the different filters outputs. Fig. 11 (top) we estimate the distance between single-sample distributions, while in Fig. [[oottom) we consider the joint distributions of state-vectors, Zz, t [k - 15, k). Observe that while each sample of êtic distributed similarly to its reference sample, it fails to attain perfect perceptual quality where we measure the distance from the real process distribution. PKF outputs attain low perceptual index (high quality) in both scenarios. We also present the perceptual quality measured for the ground-truth signal Egt empirical distribution, as a reference.
Figure 12 shows the asymptotic behavior (empirical error for large horizon 7) of Estat., the stationary PKI. (25 The figure also presents the empirical errors for Kalman filter and its stationary version (multiplied by a factor of 2, which is upper bound on the MSE distortion of perceptual estimators without temporal constraints, see R ), and the theoretical steady-state error of (125), obtained by
optimizing (128) (dashed horizontal line) for comparison. The error of the non-stationary perceptual filter auc is also shown.
I.3 Dynamic texture
Here we illustrate the qualitative effects of perceptual (temporally consistent) estimation sim- plified video restoration setting. Please see the supplementary video for the full videos. This setup visually demonstrates how:
Filters with no perfect perceptual quality tend to generate non non-realistic images or atypical motion (random or slow movement, flickering artifacts etc.).
2. PKF outputs are natural to the domain, both spatially and temporally.
For this extent, we introduce the 'Dynamic Texture' domain. In this domain, video frames are generated from a latent state which represents their Factor-Analysis (FA) decomposition (see e.g. Bishop and Nasrabadi (2) Sec. 12.2.4] for more details). The dynamics in the FA domain are assumed be linear, with small Gaussian perturbation,
FA Ik
FA FA
gk:
FA
(0,1),
FA E R 128. 2
(144)
31

---

X kal
175
Xgt
Raue
Xtic
6.0
minT
68
136
204
272
340
408
476
544
612
680
748
816
864
952
1020
k(time)
600
&
15
Xtic
15 A
Xx
de
Xgt
Xavc
minTT
15
81
147
213
279
345
411
477
543 k(time)
609
675
741
807
873
939
1005
Figure 11: Perceptual quality measured by estimated Wasserstein distance dp (lower is better). (top) Distance between distributions of single samples Pz, and pa,- (bottom) Distance between distributions of 16-state vectors (at times [ 15, k)), Pxt_,, and Pxt_ Observe that âtic single samples distributed similarly the ground-truth signal, but they fail to attain the reference joint distribution between timesteps. PKF outputs hane and T attain high measured quality in both cases.
32

---

Rig(X2)
Real (x2)
1.0
Ssta. (empiric))))
Star. (theoretical)
Rau
0.8
MSE
0.6
0.4
0.2
0.0
1000
2000
3000
4000
k(time)
Figure 12: MSE distortion on Coupled inverted pendulums for stationary filters.
The state vector with the given dynamics creates frames wavy lake in the video domain 5] through an affine transformation,
W Wiid (x FA).
(145)
WPA-wodd is linear transformation from R 28 latent states to R512x512x3 frames, and gPA is constant vector. FA and the noise ql parameters are estimated similarly to [5]. Linear observations yı RR32 are given in the frame (pixel) domain, by
yk Ckac Tk-
(146)
At times where information is being observed,
C₂ Cx1e WeRiccBBByWFA... WFA-ridd
(147)
where WRGB-y is projection onto the Y-channel (grayscale) and Cx is matrix that performs 16x downsampling both axes. At times where there no observed information, C+ = Here, Tk is a Gaussian noise.
In our first experiment, measurements are supplied in (177) up frame 127 and then vanish (O2 0, 2 128), letting the different filters predict the next, unobserved, frames of the sequence. We pass y/ as input the various filters (see Table 3 -ha the Kalman filter output. &ric the perceptual filter in the spatial domain, given in (1O). Pauc is our Algorithm (PKF) output reducing
the total cost in the latent space, Cane = DT k E [1 domain, and then transformed to the pixel domain. MSE is also calculated the FA domain. In (Fig. 13) we can see that until frame = 127, all filters reconstruct the reference frames well. Starting at time k 128, when measurements disappear, we observe that the Kalman filter slowly fades into static, blurry output which is the average frame value in this setting. This is definitely non- 'ralliticc' video; Neither the individual frames nor the static behavior are natural to the domain. Our perfect-perceptuse filter, L keeps generating 'natural' video, both spatially and temporally. This makes its MSE grow faster.
FA 4p. I²]. All filtering is done in the latent
We now perform second experiment, where Ci. is set to zero until frame A: 512. At times A: > 513 measurements are given again by the noisy, downsampled frames as described in (16) 117 In Fig. 14 present the outcomes of the different filters. We first observe that up frame k 512, there is no observed information, hence outputs are actually being generated according to priors.
Original frames are taken from 'river-14200 by OjasweinGuptaOIG via pixabay.com, and are free to use under the content licence.
33

---

y (obs..)
30
Fa (ours)
k=125 127
256
20407
200
MSE (empitical)
160
100
1000
1560
2000
1: (time)
MSD (analytical)
100
50-
100
1000 1: (time)
1560
adoe
Frame prediction dynamic texture domain. In this experiment, measurements supplied only up to frame 127. The filter's task here to predict the unobserved future frames of the sequence. Observe that the a fades into blurred average frame, while the perceptual filter faue generates a natural video, both spatially and temporally. This makes its MSE grow faster,
Figure 13:
34

---

3 (olos.)
Fax {ours)
5151
k=512
k=513
514
300
MSE (empittecall
250-
3
150-
1: (time)
250
MSE (anslytical)
200
100
50-
3'0
400
600
1000
1: (time)
Figure 14: Frame generation on Dynamic texture domain. In the first half the demo < 512). there are no observations, hence the reference signal is restored according to prior distribution. We observe that filters with no perfect-perceptual quality constraint in the temporal domain generate non-realistic frames (Kalman filter output ( or unnatural motion (Zise). Perceptual filter &aue is constrained by previously generated frames and the natural dynamics of the domain, hence its MSE decays slower.
The Kalman filter outputs static, average frame. Teic randomizes each frame independently. which creates the impression of rapid, random movement with flickering features, which is unnatural to the reference domain. At frame A: 513, when observations become available. we can that Gi. and frie are being updated immediately, creating an inconsistent, non-smooth motion between frames 512 and 513. PKF output aauc. on the other hand, keeps maintaining a smooth motion. Since non-consistent filters outputs rapidly becomes similar to the ground-trutt their errors drop. The perfect-pereeptual filter, aauc. remains consistent with its previously generated frames and the natural dynamics of the model, hence its error decays more slowly.
35


Reviews:
[{"review_id": "4aa0CL4N3u", "reviewer": "Reviewer_j8n4", "summary": "This submission introduces the Perceptual Kalman filter as a tractable solution to the online state estimation problem under perfect perpetual-quality constraints. This means the authors review the problem of state estimation under perceptual constrains (i.e. the joint distribution of any sub-sequence of states should match the original data). Then they derive a tractable approximation of the perceptual filters which have an analytic solution. Experiments were conducted on a harmonic oscillator and a dynamic texture domain. The harmonic oscillator experiments capture the MSE due to perceptual consistency and also due to the approximation used for tractability. The dynamic prediction domain demonstrates the approach scales to high-dimensions.", "questions": "How bad can the $\\Phi_k = 0$ approximation hurt your performance? Is there some provable bound for the worst-case?", "limitations": "I think the authors present the limitations of the method clearly.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Clear and thorough presentation of perceptual estimation and its relationship to prior work (in particular the Kalman Filter). \n2. Easy-to-understand derivation of the Perceptual Kalman filter and in particular the trade-offs made in order to obtain tractability.\n3. Experiments that clearly compare how the technique compares to prior work, the theoretical maximum performance and the cost of tractability. Scalability experiments also demonstrate that this works for large linear systems.", "weaknesses": "1. Although the linear systems and observations have wide applicability, the paper would be improved by mentioning this limitation and why it has or doesn't have a significant effect in the applications where perceptual quality of state-estimates is desired. This is mainly to explain how widely the linear results presented by the paper apply to the problem landscape of the general case.\n2. The experiments only show results for a harmonic oscillator. Although this is extremely useful as away to visualize the consistency gap and the gap due to $\\Phi_k = 0$, experiments showing how different dynamics can change these gaps would be very useful. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\nNonlinear filtering is in general analytically intractable even in the standard setting of classic state estimation without constraints. Given the absence of current theoretical analysis of perceptual constraints in filtering, it is natural to focus on the linear setting at the current state of knowledge and gain analytical insight. Even this has proved to be challenging, and a clear goal is to extend this work to more general settings. While temporal consistency is an emerging topic in the pracical design of deep models and architectures, none of the existing methods are designed to achieve perfect perception as in our work, hence empirical comparisons are hard. We will add a discussion on this issue to the paper.\n\nAn interesting direction for future work can be a study of domains where there are hidden linear-Gauss dynamics in some latent space (e.g. using GAN, VAE etc.) while transformation back to signal space is non linear. We will mention this as a future work\n\nRegarding a bound on the degradation due to $\\varPhi_k=0$ reduction: In fact, there is no such general bound. There are possible cases where this reduction may lead to worse error, while in other scenarios it might be optimal. Let us examine two extreme cases. In one extreme, when an observation is missing, the corresponding innovation is uncorrelated with the estimand, hence discarding past unutilized information means that the state update step in Algorithm 1 is completely random, which results in high MSE. In the other extreme, for processes where states at different timesteps are weakly correlated ($\\rho(A)\\ll 1$ or $\\rho(Q) \\gg \\rho(P_0))$, unutilized information rapidly becomes irrelevant and can be safely discarded.\n\nWe empirically demonstrate the efficiency of the reduced filters in our numerical Section (Fig. 3) and in Appendix H.2. We’ll add a discussion about the intuition behind the influence of  $\\varPhi_k$. Thank you for this interesting remark.\n\nRegarding your request to show gaps for different dynamics. We conducted an experiment that demonstrates the MSE gaps for different settings. The experimental details and numerical results appear in our \"global\" response and related PDF. We will add this demonstration to the final version.\n\n\n\n"}, {"review_id": "yA3jTzN2Ou", "reviewer": "Reviewer_3Cww", "summary": "This paper aims to study the problem of optimal causal filtering under a perceptual-quality constraint. The main contribution of this paper is to provide a mathematical framework and a closed-form solution to the aforementioned problem under some mild conditions. The experimental results on a video reconstruction problem demonstrate the effectiveness of perceptual-quality estimation.", "questions": "1. The authors should concisely explain why they did not compare their method with other perceptual-quality constrained SOTA methods in online restoration. Adding a comparison with SOTA methods, rather than solely comparing to the Kalman filter, would enhance the overall persuasiveness.\n   \n2. When both constraints 6 and 7 are applied simultaneously, it may result in an additional MSE loss. Under what conditions can this loss be alleviated while still satisfying both constraints?", "limitations": "No limitations are addressed in this paper. It is recommended that the authors make an effort to improve the readability of the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The cost of temporal consistency investigated in this study holds significant importance in the field of online restoration.\n2. The experimental results on a video reconstruction problem provide evidence for the qualitative effects of the proposed perceptual filtering.", "weaknesses": "1.\tThe readability of the paper is restricted due to the oversimplification of the formula derivation, such as in the case of Formula 16.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\nWe did our best to make our work readable. We will make an effort to clarify equation derivations and improve readability in the final version. \nAs for Eq.16, it is a direct consequence of the MMSE orthogonality property, see e.g. [8]. We will clarify this in the text.\n\nTo the best of our knowledge, there is no other work that addresses perfect-perception reconstruction (in the mathematical sense) in causal filtering problems. While temporal consistency is an emerging topic in the design of deep models and architectures, none of the existing methods are designed to achieve perfect perception as in our work. We will add a discussion on this issue to the paper. \n\nUnder the causality constraint (6), the deterministic Kalman filter is known to be MSE optimal. Now considering (7), the MSE is expected to grow, as shown analytically in the static case in [8]. This cost is alleviated in the extremely degenerate cases, where KF perfectly estimates the trajectory (e.g. observations are full or process is deterministic).\n"}, {"review_id": "XisONkSFCi", "reviewer": "Reviewer_ZP1L", "summary": "This article addresses the problem of optimal causal filtering under a perfect perceptual-quality constraint. The authors introduce the concept of an unutilized information process and present a recursive formula for perceptual filters. The study demonstrates the effects of perfect perceptual-quality estimation on video reconstruction. Overall, this article contributes to understanding optimal causal filtering under a perceptual-quality constraint and offers new insights for addressing this issue.", "questions": "As mentioned above.", "limitations": "The paper clearly describes societal impact and potential limitations of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Unfortunately, for me the paper was not easily accessible as I am not familiar with studies of the signal reconstruction.\n·The overall problem is interesting and has practical significance.\n·I find this work quite original to the best of my limited knowledge.\n·The paper to me seems technically sound and theoretical assumptions are formally proved.\n·The appendix materials have been carefully prepared, which can be of great help in understanding the paper. Meanwhile, the open-source code provides convenience for community reference.", "weaknesses": "·Experimental analysis is limited, is it necessary to showcase more diverse video scenarios?\n·Is it necessary to add a conclusion section to summarize the conclusion of this paper and clarify its possible application scenarios, as well as the interests can be followed in the future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive response. \n\nOur focus in this work was on analytic closed form results. To achieve this we applied our algorithms and analysis to the Gauss-Markov setting, where we also focus our empirical efforts. Future work, extending our results to more general domains, will be able to compare empirical results in more diverse settings. In this respect, please also see our response to reviewer 3Cww.\n\nWe did not include a conclusion section due to the limited space. We will add it to the final version. \n\nPay attention that additional experimental results can be found in the Appendix, as well as in our \"global\" response..\n"}, {"review_id": "kwmroy7hBV", "reviewer": "Reviewer_c5sG", "summary": "This work studies the problem of temporal singal reconstruction for corrupted data. Under a perfect perceptual-quality constraint, the previously regarded optimal model, i.e, the Kalman filter is shown to confronted with a fundamental dilemma. A recursive formula for perceptual filters is proposed and be empirically validated on a video reconstruction problem.\n", "questions": "It's better to split Eq. 17 into multiple lines for improved formatting.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ This work is put in a proper context in the liteature, where the perception-distortion trade-off is an important theory that only attract few research interests.\n+ Reasonable problem formulation, and detailed theoretical drivations.\n", "weaknesses": "- Is the proposed tool generalizable to other fidelity measures in addition to MSE?\n- The effecrtiveness of the proposed tool is only validated on a single sample, more are expected.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n*Is the method can be generalized beyond MSE?*\n\nPlease note that in Thm. 4.1 we have a general form for linear perfect-perceptual quality filters (Eq.14). While the form (14) is optimal for MSE, it can be used as a representation for linear filters in general (but see restriction below).\nThe constraints on the coefficients ($Q_k-\\pi_k S_k \\pi_k^\\top - \\phi_k \\Sigma \\phi_k^\\top \\succeq 0$) are necessary and sufficient for perfect perception, again, regardless of the cost objective. It indeed might be possible to optimize coefficients for objectives other than MSE under these constraints to gain optimal perceptual *linear* filters. Note, however, that under general distortion measures, optimal filters might be non-linear. We will add this discussion to the paper. Thank you for the interesting remark.\n\n*More validation is expected:*\n\nWe validate our method on a 2d oscillator example, and two video reconstruction scenarios, which were included in the main text within the space limitations. Additional experimental results can be found in  Appendix H, including inverted pendulums and further results for the video reconstruction. \n\nPlease refer to our \"global\" response, where we add an experiment for demonstrating many additional different dynamics. We will add this demonstration to the final version.\n\n*Formatting Eq. 17:*\n\nThank you for your suggestions, we will format eq. 17 as you suggested, \n"}]

Author rebuttal:
We thank all reviewers for their comments.

Following some of your remarks, we conducted an additional experiment to demonstrate different MSE gaps between temporally consistent and inconsistent filters (based on harmonic oscillators with different dynamics). Detailed description and results appear in the attached PDF.

Specifically, this experiment demonstrates how different dynamics yield different gaps (due to temporal consistency, and due to $\varPhi_k = 0$). We used different harmonic oscillator dynamics (Section 5.1), where the marginally stable matrix $A$ is multiplied by a varying factor of $\rho$, and observations are given only up to a certain time.
We plot the terminal mse, at time $T$ for each filter, normalized by the variance of $x_T$. The estimator $\hat X_{direct} $ is established using the direct optimization approach (Appendix C).

We observe that when timesteps are more correlated, both consistency and unutilized info play a major role, hence MSE gaps between filters get bigger. This experiment also demonstrates that past unutilized information is helpful in the absence of current obsetvations.

We will add this experiment to the paper.


Meta-review: Reviewers and the AC read the rebuttal and took that into consideration for their final recommendation. All reviewers participated in the discussion and their initial concerns were adequately addressed. Several suggestions for improving the exposition were made, which should be incorporated into the camera-ready paper.
Decision: Accept (poster)