A polar prediction model for learning to represent visual transformations
Authors: Pierre-Etienne H Fiquet
Eero P Simoncelli
Venue: NeurIPS 2023 poster (2023)
Keywords: video prediction
neural coding
symmetry discovery
self-supervised representation-learning

Abstract: All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.

A polar prediction model for learning to represent visual transformations
Pierre-Étienne H. Fiquet'
Eero P. Simoncelli 1.2
Center for Neural Science, New York University
2 Center for Computational Neuroscience, Flatiron Institute {pef246, eero eero. aero.simonnellllynyy..c.
Abstract
All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose self-supervised representatioon-laariig framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and group-theorethe generalization, and we optimize its parameters on next-framm prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selectivee complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in form that simplifies temporal prediction.
Introduction
The fundamental problem of vision can be framed as that of representing images in a form that is more useful for performing visual tasks, be they estimation, recognition, motor action. Perhaps the most general "task" is that of temporal prediction, which has been proposed as fundamental goal for unsupervised learning of visual representations [1]]. Previous research along these lines has generally focused on estimating stable representations rather than using them to predict: for example, extracting slow features [2], or finding sparse codes that have slow amplitudes and phases [3].
In video processing and computer vision, a common strategy for temporal prediction is to first estimate local translational motion, and then (assuming no acceleration) use this to warp and/or copy previous content to predict the next frame. Such motion compensation is a fundamental component in video compression schemes MPEG [4]. These video coding standards the result of decades of engineering effort [5], and have enabled reliable and efficient digital video communication that now commonplace. But motion estimation a difficult nonlinear problem, and existing methods fail in regions where temporal evolution is not translational and smooth: for example, expanding or rotating motions, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi- sumi-transparent surfaces (c.g., viewing the world through a dirty pane of glass). In compression schemes, these failures of motion estimation lead prediction errors, which must then be adjusted by sending additional corrective bits.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Human perception does not seem to suffer from such failures subjectivell we can anticipate the time-evolution of visual imput even the vicinity of these commonly occurring non non-translational changes. In fact, those changes are often highly informative, revealing object boundaries, and providing ordinal depth and shape cues and other information about the visual scene. This suggests that the human visual system uses a different strategy, perhaps bypassing altogether the explicit estimation local motion, to represent and predict evolving visual input. Toward this end, and inspired by the recent hypothesis stating that primate visual representations support prediction by "straightening" the temporal trajectories of naturally-occurring input [6], we formulate an objective for learning image representation that facilitates prediction by linearizing the temporal trajectories of frames in natural videos.
To motivate the separation of spatial representation and temporal prediction, we first consider the special case of rigidly translating signals one dimension. the frequency domain, translation corresponds to phase advance (section 2.1), which reduces prediction to phase extrapolation (section 2.2). We invoke basic arguments from group theory to motivate the search for generalized representa- tions (section 2.3). We propose a neural network architecture that maps individual video frames to latent space where prediction can be computed more readily and then mapped back to generate an estimated frame (section 3). We train the entire system end-to-end to minimize next frame prediction errors and verify that, in controlled experiments, the learned weights recover the representation of the group used to generate synthetic data. On natural video datasets, our framework consistently outperforms conventional motion compensation methods and is competitive with deep predictive neural networks (section 4). We establish connections between each element of our framework and the modeling of early visual processing (section 4.3).
2 Background
2.1 Base case: the Fourier shift theorem
Our approach is motivated by the well-known behavior of Fourier representations with respect to signal translation. Specifically, the complex exponentials that constitute the Fourier basis are the eigenfunctions of the translation operator, and translation of inputs produces systematic phase IN=1] R be discrete signal indexed by advances of frequency coefficients. Let CN be its Fourier transform indexed by h E [0, N 1]. spatial location n E N 1]. and let x E
We write xt", the circular shift of by v, art"" = In-e (with translation modulo N). Let denote the primitive 4h root of unity, é cd2a/N and let F N*N denote the Fourier matrix, Fak = VN Si Multiplication by the adjoint (i.e. the conjugate transpose), F*, gives the Discrete Fourier Transform (DFT), and by F the inverse DFT. We can express the Fourier shift theorem' as: x!" = Fdiag F*x, where & = [00, ,0" 6 2v o (n-1)u]T is the vector of phase shift by See Appx. A for consolidated list of notation used throughout this work.
This relationship can be depicted in a compact diagram:
Kk
Φ ku Tk
phase shift
F
(1)
F
Fn-v
spatial shift
This diagram illustrates how transforming to the frequency domain renders translation a "simpler" operation: phase shift acts as rotation in each frequency subspace, i.e. it is diagonalized.
1 Proof by substituting m = 7 w:
N-1
N 1--
N-1
2 &
Σ nmD
0
Σ
0 Φ mm
kv
0
kn
En 0
Lke
n=

---

Figure 1: Straightening translations.
(a) Three snapshots of translating sig- nal consisting of two superimposed sinu- soidal components: Zn,t sin (on2
(a) translating signal
projected trajectory
PCA
1)) sin (2 ()) (b) Projection of the signal into the space of the top three principal components. The colored points correspond to the three snapshots in panel (a). In signal space, the tem- poral trajectory is highly curved-linear extrapolation fails. (c) Complex-valus Fourier coefficients of the signal as a function of frequency. The temporal tra- jectory of the frequency representation is the phase advance of each sinusoidal component. (d) Trajectory of one ampli- tude and both (unwrapped) phases com- ponents. The conversion from rectan- gular to polar coordinates reduces the trajectory to a straight line-which is predictable via linear extrapolation.
2.5
on
sa
-34
21
0.0
PC2
F
1pose
PC1
(c) frequency coefficients
amplitude/ phuse
rect2pol
10
15
roal
30
0g
hrquency
0
2.2 Prediction via phase extrapolation
Consider a signal, the N-dimensional vector x,, that translates at a constant velocity over time: In,t = 2n-et,0- This sequence traces a highly non-linear trajectory in signal space, i.e. the vector space where each dimension corresponds to the signal value at one location. In this space, linear extrapolation fails. As an example, Figure I shows a signal consisting of a sum of two sinusoidal components in one spatial dimension. Mapping the signal to the frequency domain simplifies the description. In particular, the translational motion now corresponds to circular motion of the two coomple--valued Fourier coefficients associated with the constituent sinusoids. In polar coordinates, the trajectory of these coefficients is straight, with both phases advancing lincarly (at a rate proportional to their frequency)), and both amplitudes constant.
2.3 Generalization: representing transformation groups
Streaming visual signals are replete with structured transformations, such as object displacements and surface deformations. While these not be captured by the Fourier representation. which only handles global translation, the concept of representing transformations in their eigen-basis basis generalizes. Indeed, representation theory describes elements of general groups as linear transformations in vector spaces, and decomposes them into basic building blocks [7]. However, the transformation groups acting in image sequences not known priori, and it can be difficult to give explicit representation of general group actions. In this work, we aim to find structures that can be modeled groups in image sequences, and we learn their corresponding representations from unlabeled data.
In harmonic analysis, the Peter- Weyl Theorem (1927) establishes the completeness of the unitary irreducible representations for compact topological groups (an irreducible representation is subspace that is invariant to group action and that can not be further decomposed). Furthermore, every compact Lie group admits faithful (i.c. injective) representation given by explicit complete orthogonal basis, constructed from finite-dimensional irreducible representations [7]. Accordingly, the action of a compact Lie group can be expressed as rotation within each irreducible representation (an example is the construction of steerable filters [8] in the computational vision literature).
In the special case of compact commutative Lie groups, the irreducible representations are one- dimensional and complex-vail valued (alternatively, pairs of real valued basis functions). These groups have a toroidal topology and, in this representation, their action can be described as advances of the phases. This suggests strategy for learning a representation: seek pairs of basis functions for which phase extrapolation yields accurate prediction of upcoming images stream of visual signals.
3

---

3 Methods
3.1 Objective function
We aim to learn a representation of video frames that enables next frame prediction. Specifically, we optimize a cascade of three parameterized mappings: an analysis transform (S.)) that maps each frame to a latent representation, prediction in the latent space (Pu),, and synthesis transform (gw) that maps the predicted latent values back to the image domain. The parameters w of these mapping are learned by minimizing the average squared prediction error:
min
|x+1
Iw (2(2(+1)ll
where 2,+1 = Pul 21-1), and Z = fw (Xt).
(2)
An instantiation of this framework is illustrated in Figure 2. Here the analysis and synthesis transforms are adjoint linear operators, and the predictor is a diagonal phase extrapolation.
MSE
R141
Xt+1
W 6,
W
Figure 2: Polar prediction model. The previous and current images in sequence (X;-1 and x²) are convolved with pairs of filters (W*), each yielding complex-valuee coefficients. For a given spatial location in the image, the coefficients for each pair of filters are depicted in complex planes with colors corresponding to time step. The coefficients at time + are predicted from those at times t- 1 and É by extrapolating the phase (6t). These predicted coefficients are then convolved with the adjoint filters (W) to generate a prediction of the next image in the sequence ($1+1). This prediction compared to the next frame (x++1) by computing the mean squared error (MSE) and the filters learned by minimizing this error. Notice that, at coarser scales, the coefficient amplitudes tend to be larger and the phase advance smaller, compared to finer scales.
3.2
Analysis-synthesis transforms
Local processing When focusing on small spatial region in an image sequence, the transformation local translation. That is to say, in
observed time passes can often be well approximated spatial neighborhood around position n, m E [n),, we have: Fm,4+1 R Im-v.t- We can use the decomposition described for global rigid translation, replacing the Fourier transform with learned local convolutional operator [9], processing each spatial neighborhood of the image independently and in parallel.
At every position in the image (spatial indices are omitted for clarity of notation), each pair of coefficients is computed as inner product between the input and the filter weights of each pair of channels. Specifically for E [0, K]. where is the number of pairs of channels, we have: y22.1 = v x and 1,t 12k++1 X4- Correspondingly, an estimated next frame is generated by applying the transposed convolution Iu. to advanced coefficients (see section 3.3). We use the same weights for the encoding and decoding stages, that is to say the analysis operator is the conjugate transpose of the synthesis operator (as is the case for for the Fourier transform and its inverse). Sharing these weights reduces the number of parameters and simplifies interpretation of the learned solution.

---

Multiscale processing Transformations such as translation act on spatial neighborhoods of various sizes. To account for this, the image first expanded at multiple resolutions fixed overcomplete Laplacian pyramid [10]; then learned spatial filtering (see previous paragraph) and temporal pro- cessing (see section 3.3) are applied on these coefficients; and finally, the modified coefficients are recombined across scales to generate the predicted next frame (see details in the caption of Figure 3).
Figure 3: Laplacian pyramid. An image is recursively split into low frequency approximation and high frequency details. Given the initial image x = xj=0 E R., the low frequency approximation (aka. Gaussian pyramid coefficients) is computed via blurring (convolution with a fixed filter B) and downsampling ("stride" of 2, denoted 2.):
Xj (B X,-1 E R2 for levels E [1, J]; and the high frequency details (aka. Laplacian pyramid coefficients) are computed via upsampling (put one zero between each sample, 2n) and blurring: Ax; x; B (2 (2'x++1). These coefficients, (Ax, rosj<J, as well as the lowpass, æ. can then be further processed. A new image is constructed recursively on these processed coefficients. First by upsampling the lowest resolution, and then by adding the corresponding details until the initial scale j 0 as: Xj B (2xx++) + Axy.
2
Xj-1
Ax;
2
3.3 Prediction mechanism
Polar predictor In order to obtain phases, we group coefficients in pairs, express them as complex- valued: zk,t 3/2 11/22+11, E C, and convert polar coordinates: zk, t0,, With this notation, linear phase extrapolation amounts to εx,++1 = ( +00k.)) where the phase advance A0k, is equal to the phase difference over the interval from - to t: A0k, Og 0g.1-1- Note that we assume no phase acceleration and constant amplitudes. The phase-advanced coefficients can be expressed in more direct way, using complex arithmetic, as:
2k
21,1+1 Sk,4 where Sk.F 12x, ||2,,-11|
(3)
with 7 and |2| respectively denoting complex conjugation and complex modulus of This formulation in terms of products of complex coefficients has the benefit of handling phases implicitly, which makes phase processing computationally feasible, as previously noted in the texture modeling literature [11, 12]. Optimization over circular variables suffers from discontinuity if one represents the variable over a finite interval (e.g. [-T, .r). Alternatively, procedures for "unwrapping" the phase generally unstable and sensitive to noise.
In summary, given video dataset = .x....,, xTl E RN the convolutional filters of a polar prediction model are learned by minimizing the average squared prediction error:
T
|x+1 - Wdiag(6,) W x2 Σ
min
WECNX NK
where 8, = (Ze O 2 Z ([ze ④ [z),)),, and z. = W' X
(4)
The columns of the convolutional matrix W contain the K complex-valued filters, wi V2k iV2x++1 CN (repeated at N locations) and multiplication, division, amplitude, and complex conjugation are computed pointwise.
This "polar predictor" (hereafter, PP) is depicted in figure 2. Note that the conversion to polar coordinates is the only non-linear step used in the architecture. This bivariate non- non-linear activation function differs markedly from the typical (pointwise) rectification operations found in convolutional neural networks. Note that the polar predictor homogeneous of degree one, since it is computed as the ratio of a cubic over quadratic.
Using "Gauss's trick". each complex multiplication can be computed with only three real multiplications:
(a (b)(c + id) ac bd i((a + () + d) bd).
5

---

Quadratic predictor Rather than building in the phase extrapolation mechanism, we now consider a more expressive parametrization of the predictor (pw:) that can be learned on data jointly with the analysis and synthesis mappings. This "quadratic predictor" (hereafter, QP) generalizes the polar extrapolation mechanism and can accommodate groups of channels of size larger than two (as for the real and imaginary part in the polar predictor).
Learnable quadratic prediction mecha- nism. Groups of coefficients (yx,)) at the previous and current time-step are normalized (Ug,,) and then passed M through in Linear-Square-Linne
Figure 4:
L1
L2
cascade to produce prediction matrix (Ms,). ) This matrix applied the current vector of coefficients to predict the next one. The linear transforms (L1 and L2) are learned. This quadratic prediction module contains phase extrapola- tion special case and handles the more general case of groups of coefficients beyond pairs.
ug
Yg
1-1
L+1
First, rewrite the phase extrapolation mechanism of equation using only real-valued elements:
$2k,4+1 Coos A 022+1..41
sin A0k,t V2k,t cos Atk. (2k+1.1
(5)
sin APR,
where the 2 prediction matrix, MA,r, is rotation by angle 20,,.- Using an elementary trigono- metric identity: cos(a- b) cos(a) cos(b) + sin(a) sin(b) and recalling that yk., V2g ax,t cos(Ou,r). the elements of this prediction matrix can be made explicit. They are quadratic function of the normalized response, which we write u2k (for unit vector), as:
92k.
cos 2O 122,:-11 + 126-1, where U2R,3 = (U + V 21 1/22
(6)
This quadratic function be expressed in terms of squared quantities (without cross terms) using the polarisation identity: aB = ((a + B)² - -a3-3)3 /)4 Specifically:
Cos A0x = (u22,1 + u2x,0-1) - (u22, U2x,t-1)
(U2++1,7 + 122+111-1)2 (u25+1. - 122+1,1-1)3)
An analogous expression can be derived for sin A0R.1- These operations are depicted in Figure 4: current and previous pairs of coefficients (yx. = [y2a, 121++11, 32k,e-1, B28+1.11] E I²) are normalized (UR,e E IR then linearly combined (L, E Rdxd), pointwise squared, and linearly combined again (I2 E 1xd) to produce a prediction matrix (Ma,e E R2x2) that is applied to the current coefficients to produce a prediction (Ūæx..+1, (28++1+441 E TR?)). These linear combination can be learned jointly with the analysis and synthesis weights by minimizing the prediction error.
In summary, the quadratic predictor is learned as:
min L.
|x++1
VA, V Txx where A, = blockdiag MI....... MK.),
M LL 1 U
Yk.t y 172
and Y: = V
(8)
The columns of the convolutional matrix V E N contain the groups of y filters (repeated
at N locations). The number of channels in group no longer limited to pairs (ggl 2 2
parameerr....rr...... The prediction matrices. M R5*9, computed as Linear-Square Linear cascade on normalized activity from group of coefficients at the previous two time points:
ug [uk,t, L..., Ub+g-1.1] E R2 and ug The linear combinations are learnable matrices L, R and L2 x where the number of quadratic units and squaring computed pointwise. Note that in the case of pairs (g 2), six quadratic units suffice (d = 6).

---

4 Results
4.1 Recovery of planted symmetries
To experimentally validate our approach, we first verified that both the PP and the QP models can robustly recover known symmetries in small synthetic datasets consisting of translating or rotating image patches. For these experiments, the analysis and synthesis transforms are applied to the entire patch (i.e., no convolution). Learned filters for each of these cases are displayed in Figure 7, Appendix B. When trained on translating image patches, the learned filters are pairs of plane waves, shifted in phase by x 2/2. Similarly, when trained on rotating patches, the learned filters are conjugate pairs of circular harmonics. This demonstrates that theses models can recover the (regular) representation of some simple groups from observations of signals where their transformations are acting. When the transformation is not perfectly translational (e.g. translation with open boundary condition), the learned filters are localized Fourier modes. Note that when multiple kinds of transformations are acting in data (e.g., mixtures of both translations and rotations), the PP model is forced to compromise on single representations. Indeed, the phase extrapolation mechanism is adaptive but the basis in which is computed is fixed and optimized. A more expressive model would also allow for adaptation of the basis itself.
4.2 Prediction performance on natural videos
We compare our multiscale polar predictor (mPP) and quadratic predictor (mQP) methods to a causal implementation of the traditional notion-compenstted coding (cMC) approach. For each block in a frame, the coder searches for the most similar spatially displaced block in the previous frame, and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We also compare to phase-extrapolation within a steerable pyramid [13], an overcomplete multi-scale decomposition into oriented channels (SPyr). We also implemented a deep convolutional neural network predictor (CNN), that maps two successive observed frames to estimate of the next frame [14]. Specifically, we use CNN composed of 20 non-linear stages, each consisting of 64 channels, and computed with 3 x filters without additive constants, followed by half-war wave rectification. Finally, we also consider U-net [15] which is CNN that processes images at multiple resolutions (Unet). The number of non-linear stages, the number of channels and the filter size match that of the basic CNN. See descriptions in Appendix C for architectures and Appendix D for dataset and training procedures.
Table 1: Performance comparison. Prediction error computed the DAVIS dataset. Values indicate mean Peak Signal to Noise Ratio (PSNR in dB) and standard deviation computed over 10 random seeds (in parentheses).
Method
split train 21.32
Copy
cMC
SPyr
mPP
mQP
CNN
Unet
23.83
25.13
25.31 (0.04)
25.38 (0.11)
25.78 (0.18)
26.91 (0.38)
test
20.02
22.37
23.82
24.11 (0.01)
24.04 (0.06)
23.58 (0.05)
23.94 (0.06)
Prediction results on the DAVIS dataset [16] are summarized in Table 1. First, observe that the predictive algorithms considered in this study perform significantly better than baselines obtained by simply copying the last frame, causal motion compensation, or phase extrapolation in a steerable pyramid. Second, the multiscale polar predictor performs better than the convolutional neural networks on test data (both CNN and Unet are overfit). This demonstrates the efficiency of the polar predictor: the PP model has roughly 30 times fewer parameters than the CNN and uses a single non-linearity, while the CNN and Unet contain 20 non-linear layers. Appendix E contains additional a comparison of computational costs for these algorithms: number of trainable parameters, training and inference time. Note that on this dataset, the Unet seems to overfit to the training set. Moreover, the added expressivity afforded by the multiscale quadratic predictor did not result significant performance gains. A representative example image sequence and the corresponding predictions are displayed Figure 5. Additional results on a second natural video dataset are detailed in Appendix and confirm these trends.
7

---

x(r 1)
x(i
x 1)
eMC
SPyr
mPP
mQp
CNN
PSNR
21.67
25.70
26.49
26.98
27.28
26.44
Figure 5: Example image sequence and predictions. A typical example image sequence from the DAVIS test set. The first three frames on the top display the unprocessed images, and last five frames show the respective prediction for each method. The bottom row displays error maps computed the difference between the target image and each predicted next frame the corresponding position in the first row. All subfigures are shown on the same scale.
4.3 Biological modeling
The polar prediction model provides a hypothesis for how cortical circuits in the primate visual system might compute predictions of their inputs ("predictive processing") [17-20]. This framework is agnostic to how predictions are used and is complementary to candidate algorithms for signaling predictions and prediction errors across the visual hierarchy ("predictive coding") [21-23].
First, the learned convolutional filters of a polar prediction model resemble receptive fields of neurons area V1 (primary visual cortex). They are selective for orientation and spatial frequency, they tile the frequency domain (making efficient use of limited resources), and filters in each pair have similar frequency selectivity and are related by shift in spatial phase. Representative filters are displayed in Figure 6.
Second, the quadratic prediction mechanism derived in section 3.3 suggests qualitative bridge to physiology. Computations for this model are expressed in terms of canonical computational elements of early visual processing in primates. The normalized responses u2x (4) in equation 6 are linear projections of the visual input divided by the energy of related cells, similar to the normalization behavior observed in simple cells [24]. The quadratic units my in equation 7 sensitive temporal change in spatial phase. This selectivity for speed in a given orientation and spatial frequency is reminiscent of selletivn complex cells which are thought to constitute the first stage of motion estimation [25, 26].
5 Related work
The polar prediction model is conceptually related to representation learning methods that aim to factorize visual signals. In particular, sparse coding with complex-valued coefficients [3] aims to factorize form and motion. More generally, several methods adopt a Lie group formalism to factorize invariance and equivariance. Since the seminal work that proposed learning group generators from dynamic signals [27], a polar parametrization was explored in [28] to identify irreducible representations in synthetic dataset, and corresponding neural circuit was proposed in [29]. The Lie group formalism has also been combined with sparse coding [30, 31] []]]]]]@ to model natural images as points on latent manifold. More recently, bispectral neural networks [32] have been shown to learn image representations invariant to given global transformation. In a related formalism, factored Boltzmann machines have been proposed learn relational features [33]. The polar prediction model differs in two important ways: (1) unlike the coding approach that operates on iid data, it focuses on predicting, not representing, the signal; (2) the prediction objective does not promote sparsity of either amplitude or phase components and does not rely on explicit regularization. The discontinuity arising from selection of sparse subsets of coefficients seems odds with the representation of continuous group actions [34]. The polar prediction model relies
smooth and continuous parameterization to jointly discover and exploit the transformations acting in sequential data. The polar prediction model is convolutional and scales to natural video data, it can adapt the multiple unknown and noisy

---

(a) spatial domain filters
(b) corresponding Fourier amplitude spectra
Figure 6: Learned filters. A single stage polar predictor was trained to predict videos from the DAVIS dataset. (a) Filters in each pair are displayed side by side and sorted by their norm. (b) Their amplitude spectra are displayed at corresponding locations. Observe that the filters are selective for orientation and spatial frequency, tile the frequency spectrum, and form quadrature pairs.
transformations that act in different spatial position. The literature on motion microscopy describes temporal processing of local phases in fixed complex-valued wavelet representation to interpolate between video frames and magnify imperceptible movements [35]. Polar prediction processes phase in learned representation extrapolate to future frames.
The polar prediction model is related other representation learning methods that also rely on temporal prediction. Temporal stability was used to learn visual representations invariant to geometric transformations occurring in image sequences [36]. The idea of learning a temporally straightened representation from image sequences was explored using a heuristic extrapolation mechanism [37] (specialized "soft max-pooling" and "soft argmax-pond pooooinoonn modules). at finding video representations which decompose content and pose and identify "ccomponents" in order facilitate prediction of synthetic image sequences [38]. To tackle the challenge of natural video prediction, more sophisticated architectures have been developed for decomposing images into predictable objects [39]. A recurrent instantiation predictive coding through time relying on a stacked convolutional LSTM architecture proposed [40] and shown to relate to biological vision [41]. In contrast, the polar prediction model scales to prediction of natural videos while remaining interpretable. Another related approach, originating in the fluid mechanics literature, focuses on the Koopman operator [42]. This approach is a dynamical analog of the kernel trick: it lifts a system from its original state-space into higher dimensional representation space where the dynamics can be linearized (i.e. represented by fixed dynamics matrix). This formalism has inspired a line of work in machine learning: predictive auto-encoders learn coordinate systems that approximately linearize a system's 's dynamics [43]. Auxiliary networks have been introduced to adjust the dynamics matrix velocity [44]. In contrast, the polar prediction model learns representation that (implicitly) straightens the temporal evolution in an adaptive representation.
related approach aimed
Discussion
We'n ve introduced a polar prediction model and optimized it to represent visual transformations. Using adaptive phase extrapolation in fixed shiftable basis, the model jointly discovers and exploits the approximate symmetries image sequences which are due local content deformation. The basis is optimized to best diagonalize video dynamics by minimizing mean squared prediction error. The phase relationships are exploited implicitly in bundled computation, bypassing the instabilities and discontinuities of angular phase variables. By starting from mathematical fundamentals and considering an abstract formulation in terms of learning the representation of transformation groups, we formulated a framework that makes three major contributions. First, it provides a method for discovering the approximate symmetries implicit sequential data and complements methods for imposing known invariants. Second, it achieves accurate next-frame video prediction within

---

principled framework and provides an interpretable alternative to standard architectures. Third, it offers framework to understand the nonlinear response properties of neurons in primate visual systems, potentially offering functional explanation for perceptual straightening.
The polar prediction model makes several strong assumptions. First, it is inertial, assuming that amplitude is unchanged and phase evolves linearly with no acceleration. Second, it separates spatial and temporal processing, which seems odds with the spatiotemporal selectivities of visual neurons [45], but could enable downstream image based tasks (e.g. object segmentation, heading direction estimation). Third, it acts independently on each coefficient, although the representation does not perfectly diagonalize the dynamics of natural videos. This is analogous to the situation in modern nonlinear signal processing where diagonal adaptive operators in appropriate bases have had major impact in compression. denoising. and linear inverse problems [46]. Our empirical results demonstrate that these assumptions provide a reasonable description of image sequences. The standard deep networks considered here could in principle have discovered similar solution, but they seem not to. This exemplifies fundamental theme in computational vision and machine learning: when possible, let the representation do the analysis. An important limitation of the framework comes from the use of mean squared error, which minimized by the posterior mean and tends to result in blurry predictions. Since temporal prediction is inherently uncertain, predictive processing should be probabilistic and exploit prior information.
The polar prediction framework suggests many interesting future directions, both for the study of visual perception (e.g. object constancy occlusion, and object grouping from common fate) and for the development of engineering applications (e.g. building flow-free video compression standard). To expand expressivity and better represent signal geometry, may be possible to design a polar prediction architecture that also adapts the basis, potentially extending to the case of noncommutative transformations (e.g. the two dimensional retinal projection of three dimensional spatial rotation). Finally, it is worth considering the extension of the principles described here to prediction at longer temporal scales, which will likely require learning more abstract representations.
Acknowledgments
We thank members of the Laboratory for Computational Vision at NYU, and of the Center for Computational Neuroscience at the Flatiron Institute for helpful discussions. This work was supported in part by the Simons Foundation.
References
[1] Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 1991.
[2] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 2002.
[3] Charles F Cadieu and Bruno Olshausen. Learning intermediate-level representations of form and motion from natural movies. Neural computation, 2012. 1,
[4] Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 2003.
[5] Peter Elias. Predictive coding-i. IRE transactions on information theory, 1955.
[6] Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 2019. 2
[7] Brian Hall. Lie groups, lie algebras, and representations. In Quantum Theory for Mathemati- cians. Springer, 2013. 3
[8] William T Freeman, Edward H Adelson, et The design and use of steerable filters. IFEE Transactions Pattern analysis and machine intelligence, 1991. 3
10

---

[9] David Fleet and Allan D Jepson. Computation of component image velocity from local phase information. International journal of computer vision, 1990.
[10] Peter Burt and Edward Adelson. The laplacian pyramid as a compact image code. IEEE Transactions communications, (1(4):532-54) 1983. 5, 15
[11] Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 2000. 5, 15
[12] Sixin Zhang and Stéphane Mallat. Maximum entropy models from phase harmonic covariances. Applied and Computational Harmonic Analysis, 2021.
[13] Eero P Simoncelli and William T Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. Proceedings., International Conference on Image Processing, 1995. 7, 15
[14] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016. 7, 15
[15] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Interven- tion, 2015. 7, 15
[16] Jordi Pont-Tuset Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 7, 16
[17] Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 1982.
[18] Michael J Berry II, Iman H Brivanlou, Thomas Jordan, and Markus Meister. Anticipation of moving stimuli by the retina. Nature, (98(5533)3334, 1999.
[19] Stephanie Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in sensory population. Proceedings of the National Academy of Sciences, 2015.
[20] Olivier J Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero P Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications, 12(1):5982, 2021.
[21] David Mumford. On the computational architecture of the neocortex: Ii the role of cortico- cortical loops. Biological cybernetics, 1992.
[22] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some cxtra-classical receptive-ficld effects. Nature neuroscience, 1999.
[23] Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 2010.
[24] David Heeger. Normalization of cell responses car striate cortex. Visual neuroscience, 1992. 8
[25] Edward Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a, 1985.
[26] Eero P Simoncelli and David Heeger. A model of neuronal responses in visual area mt. Vision research, 3835),4, 1998. 8
[27] Rajesh Rao and Daniel Ruderman. Learning lie groups for invariant visual perception. Advances in neural information processing systems, 1998. 8
[28] Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. In International Conference Machine Learning. PMLR, 2014. 8

---

[29] Alexander Genkin, David Lipshutz, Siavash Golkar, Tiberiu Tesileanu, and Dmitri Chklovskii. Biological learning of irreducible representations of commuting transformations, Advances in Neural Information Processing Systems, 2022. 8
[30] Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018. 8
[31] Ho Yin Chau, Frank Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lie group transformations and sparse coding, arXiv preprint arXiv:20/2.12071,, 2020. 8
[32] Sophia Sanborn, Christian Shewmake, Bruno Olshausen, and Christopher Hillar. Bispectral neural networks. arXiv preprint (rXiv:2209.03416: 2022.
[33] Roland Memisevic and Geoffrey E Hinton. Learning to represent spatial transformations with factored higher-order boltzmann machines. Neural computation, 2010.
[34] Jörn-Philipp Lies, Ralf M Häfner, and Matthias Bethge. Slowness and sparseness have diverging
effects on complex cell learning. PLoS computational biology, 10(3):e 2014.
[35] Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T Freeman. Phase-based video motion processing. ACM Transactions on Graphics, 2013. 9
[36] Matthias Bethge, Sebastian Gerwinn, and Jakob H Macke. Unsupervised learning of a steerable basis for invariant image representations. In Human Vision and Electronic Imaging XII. SPIE, 2007.
[37] Ross Goroshin, Michael F Mathieu, and Yann LeCun. Learning to lincarize under uncertainty. In Advances in Neural Information Processing Systems, 2015. 9
[38] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances neural information processing systems, 2018.
[39] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Waldo: Future video synthesis using object layer decomposition and parametric flow prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 9
[40] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning, International Conference on Learning Representations, 2017. 9
[41] William Lotter, Gabriel Kreiman, and David Cox. A neural network trained for prediction mimics diverse features of biological neurons and perception. Nature machine intelligence, 2(4):210-219, 2020. 9
[42] Igor Mezic. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 2005. 9
[43] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting se- quential data using consistent koopman autoencoders. In International Conference on Machine Learning, 2020. 9
[44] Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 2018. 9
[45] Gregory C DeAngelis, Izumi Ohzawa, and Ralph D Freeman. Receptive-field dynamics in the central visual pathways. Trends in neurosciences, 18(10):451-458 1995. 10
[46] Stéphane Mallat. A wavelet tour of signal processing: The sparse way. Academic Press, 2008
10
[47] Shan 7hu and Kai-Kuang Ma. new diamond search algorithm for fast mltck- motion estimation. IEEE transactions Image Processing, 2000. 15
12

---

[48] J Hans van Hateren and Dan L Ruderman. Independent component analysis of natural im- age sequences yields ppatio-tempore filters similar simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 1998. 16
[49] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
16
13

---

A Notation
Let R. denote the N-dimensional Euclidean space equipped with the usual Euclidean norm || and let CK denote the K-dimensional complex vector space. Let CN* denote the set of N x K complex-valued matrices, Let denote the adjoint (ie. the conjugate transpose) of F.
Given vectors u, v E CK let u @ = [urvi,..., uKUK]] E C denote the elementwise (aka. Hadamard) product of u and v. Let u 2 v = [u, ux /ul C denote the elementwise division of u and v. Let diag(u) u) denote the K diagonal matrix whose (k, k) entry ug.
Let the complex number E C be expressed in rectangular coordinates as 2 x + by where (xr, v) R2, in polar coordinates as ae where (a, 0) R [-T,, m|. Its complex conjugate u cos and sin(()); and
by ae- The rectangular coordinates the polar coordinates are |2| vz² + y², and Le atan2 x). We overload these notations denote element-wise operations on vector CK EI..... 2 C and lz] [|>]] ..... x | R
B Planted symmetries
(a) Translation (cyclic boundary)
(b) Rotation
(c) Translation and rotation
(d) Translation (open boundary)
Figure 7: Learned filters on planted symmetries.
Filters of polar predictor networks trained to predict small synthetic sequences. We randomly select 100 image patches of size 16 x 16 from the DAVIS dataset and generate training data by manually transforming them-applying translations or rotations. We verify that recovers corresponding harmonic functions: Fourier modes for translation (panel a). and disk harmonics for rotation (panel b). To show that the recovery harmonics robust, we design two additional synthetic datasets. i) the combination of translational and rotational sequences. In this case, PP learns some filters
14

---

that correspond to either transformation. But the model does not have the expressivity required to dynamically adapt the representation to either transformation (panel c); ii) generalized translation sequences: spatially sliding a square window on a large image (ie. new content creeps in and falls off boundaries), instead of using cyclic boundary condition (ie. content wraps around the edges). In this case, PP learns localized Fourier-like modes (panel d), indicating that approximate group actions still provide meaningful training signal- some of the filters are not structured. In each panel, the 32 pairs of filters are sorted by their norm. Notice that some of high frequency harmonics are missing, This is due to the spectral properties of the datasets, which have more power at lower frequencies.
C Description of architectures
Motion Compensation We compare our method to the traditional motion--oomennstted coding approach that forms the core of inter-picture coding in well established compression standards such as MPEG. Block matching is an essential component of these standards, allowing the compression of video content by up to three orders of magnitude with moderate loss of information. For each block in frame, typical coders search for the most similar spatially displaced block in the previous frame (typically measured with MSE), and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We implemented a "diamond search" algorithm [47] operating on blocks of 8 x 8 pixels, with a maximal search distance of 8 pixels which balances accuracy of motion estimates and speed of estimation (the search step is computationally intensive). We use the estimated displacements to perform causal motion compensation (cMC), using displacement vectors estimated from the previous two observed frames (x--1 and x,) to predict the next frame (X++1) rather than the current one (as in MPEG).
Complex Steerable Pyramid We consider fixed multiscale oriented representation of image content: a steerable pyramid [13, 11] covering 16 orientations and scales on the DAVIS dataset (resp. 16 orientations and 4 scales VanHateren dataset). This choice of number of orientations and number of sacles maximizes prediction performance on the corresponding datasets.
Polar Predictor We use 32 pairs of convolutional channels with filters of size 17 x 17 pixels, without biases (no additive constants) and no padding (ic. "valid" boundary condition). For the multiscale version (mPP), we use 16 pairs of convolutional channels with filters of size 11 x 11 pixels, without biases (no additive constants). The representation is computed inside fixed Laplacian pyramid [10]. We used 4 scales for the DAVIS dataset (and respectively 3 scales for the VANH dataset). Within this multiscale representation. the learned filters are applied with zero padding (ic. "same" boundary condition).
Quadratic Predictor For the multiscale version (mQP), we use the same analysis (fuz) and synthesis (gw) hyperparameters as mPP. For the quadratic prediction mechanism, we use 16 groups of convolutional filters (twice as many as for mPP). The quadratic predictor (Pw)) operates on groups of 4 coefficients and contains 12 quadratic units. It is more expressive than the multiscale Polar Predictor architecture and contains phase advance as special case.
Vanilla CNN Finally, we implemented more direct convolutional neural network predictor (CNN), that maps two successive observed frames to an estimate of the next frame [14]. For this, we used CNN composed of 20 stages, each consisting of 64 channels, and computed with 3 x 3 filters without additive constants, followed by half-wave rectification (ie. ReLU). To facilitate learning, a skip connection copies the current frame x; and the network only outputs residuals that get added to the current frame in order to predict the next frame: 8 Xx + fu ([x,,, x().)). This model jointly transforms and processes pairs of frames to generate predictions, while both polar predictor (PP) and quadratic predictor (QP) separate spatial processing and temporal extrapolation.
The Unet architecture [15] is commonly used for to-gge tasks. It has an analysis- synthesis structure with downsampling, upsampling, and skip connections between levels at the same resolution. We used a 5 levels architecture, each block consists of two convolutions, batch norm and rectification. The convolutions have filters of size x 3, comprise 64 channels, and no additive bias. The end-to-end network comprises 20 non-linear stages with ReLU activations.
Unet
15

---

D Description of datasets and optimization
To train, test and compare these models, use the DAVIS dataset [16], which was originally designed as a benchmark for video object segmentation. Image sequences in this dataset contain diverse motion of scenes and objects (eg., with fixed or moving camera, and objects moving different speeds and directions), which make next frame prediction challenging. Each clip is sampled 25 frames per second, and is approximately 3 seconds long. The is subdivided into 60 training videos (4741 frames) and 30 test videos (2591 frames). We pre-processed the data, converting all frames to monochrome luminance values, and scaling their range to the interval [-1, 1]. Frames cropped to a 256 x 256 central region, where most of the motion tends to occur, and then spatially down-sampled 128 128 pixels.
DAVIS
VanHateren We also consider a smaller video dataset consisting in footage of animals in the wild [48] which contains variety of motions (animals in the scene, camera motion, etc.) and occlusions. The missing band at the top the frame is cropped, reducing the image size from 128 x 128 pixels to 112 x 128 pixels. The dataset is standardized to mean and unit variance, it is split into snippets of 11 frames, 292 snippets are used for training and 33 for testing. There is no spatial downsampling or whitening.
Boundary handling The computation of this prediction error restricted to the center of the image because moving content that enters from outside the video frame is inherently unpredictable. Specifically, we trim a 17-pixel strip from each side, yielding frames of size 94 x 94 pixels for the DAVIS dataset and 78 x 94 for the VanHateren dataset. Convolutions are computed with zero padding so that the outputs have the same size as inputs (the only exception is for the plain Polar Predictor, shown in Figure 6, where valid convolutions were performed).
Training procedure We assume the temporal evolution of natural signals to be sufficiently and appropriately diverse for training, and do not apply any additional data augmentation procedures. We train on brief temporal segments containing and process these in batches of size 4. We train each model for 200 epochs on DAVIS using the Adam optimizer [49] with default parameters and learning rate of 3 10-4 The learning rate is automatically halved when the test loss plateaus. In the CNN, we use batch normalization before every half-wave rectification, rescaling by the standard deviation of channel coefficients (but with no additive bias terms).
frames (which allows for prediction of frames),
E Additional results
Additional dataset Note that the PSNR is the logarithm of the MSE in units of the signal: PSNR 10 1og (MAX² /MSE), where MAX is the maximum possible pixel value of the image. A PSNR of OdB means that the squared peak signal and the mean squared error are equal: a PSNR of 10dB (resp. 20dB) means that the squared peak signal is 10 times (resp. 100 times) bigger than the MSE
Table 2: Performance comparison. Prediction error computed on the VanHateren dataset. Values indicate mean (standard deviation) of Peak Signal Noise Ratio (PSNR in dB) computed over 10 random seeds.
Method
split train
Copy
cMC
SPyr
mPP
mQP
CNN
Unet
27.10
28.56
29.50
30.16 (0.02)
30.28 (0.04)
31.16 (0.10)
31.52 (0.42)
test
26.41
27.93
28.83
29.11 (0.01)
29.07 (0.04)
29.09 (0.05)
29.26 (0.06)
Computational costs The polar predictor described in this paper is lightweight: it runs rapidly and contains few parameters (two orders of magnitude less than CNN and Unet). The polar predictor is designed as an online method that could be applied to streaming data. Parameter counts, as well as training and inference time are reported in Table 3.
Notice that the Unet is also very efficient: runs fast (most of the computation applied to spatially downsampled coefficients). Note that the Quadratic Predictor (mQP) is slower. This model was
16

---

Table 3: Number of trainable parameters and run times. Mean training time (standard deviation) for 200 epochs on the VanHateren dataset (mean in min:sec, std in sec). Mean inference time (standard deviation) for a single video snippet containing 11 frames of size 112 128 pixels (mean in ms, std in ps).. Training and inference time are computed on NVIDIA A100 GPU.
Method
dataset
mPP
mQP
CNN
Unet
# parameters
7,744
15,776
666,496
591,041
training time
10:24 (20)
46: (29)
32:17 (50)
11:14 (17)
inference time
9.71 (11)
19.8 (8)
9.57 (8.4)
2.7 (8.2)
developed to suggest a connection with physiology (each element recapitulates know functional building blocks primate visual physiology), not to improve performance or efficiency.
Decoupling analysis and synthesis When decoupling the analysis and synthesis transforms (ie. untying the weights), Polar Predictor (not multiscale) achieved similar prediction performance on the VanHateren dataset (tied: train 29.87/ test 28.83; vs. untied: train 29.99/ test 28.87 dB). The projection and basis vectors (ie. the filters in the analysis and synthesis transforms) align as training progresses.
17


Reviews:
[{"review_id": "npVB0VytUX", "reviewer": "Reviewer_Zhuh", "summary": "The authors present a self-supervised representation-learning framework inspired from the idea of continuous deformations of objects in videos. The framework allows for next-frame prediction given previous ones, and borrows ideas from the Fourier shift theorem. The framework is composed of three stages - an analysis transformation $f_w$ of given frames, a prediction transformation $p_w$ that predicts the next frame representation given previous ones, and a synthesis transformation $g_w$ which transform the latent representation to the actual frame. $g_w$ is implemented as the inverse of $f_w$, and the two share parameters. The authors present two variants of their framework, the polar predictor (PP) where $p_w$ is a fixed polar extrapolation, and a learnt $p_w$ variant termed the quadratic predictor (QP). The authors also detail results on next-frame prediction task, and show that their method surpasses baseline techniques.", "questions": "- Have you tried using the proposed framework to predict additional frames, as opposed to only the next one? It seems quite intuitive and straightforward using the proposed method since it only requires an additional step in the polar extrapolation.\n\n- Are there any latency constraints to the proposed approach? For example in the transformation to polar coordinates.", "limitations": "After reading through the paper a few times, I couldn't find any addressed limitations (although I can't point to any unaddressed limitations).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of exploiting the temporal symmetries and local deformations in videos to derive a natural framework for next-step prediction is interesting and has the potential to provide interpretability for sequence prediction tasks.\n\n- The idea of sharing parameters between the synthesis and analysis operators, contributes clarity to the framework's underlying process.\n\n- Multiscale processing using a Laplacian pyramid is an interesting solution to overcome video transformations of different scales.\n\n- The proposed framework shows promising results in the provided evaluation.", "weaknesses": "- Technical details regarding the implementation of the learned transformations is missing from the main paper.\n\n- The section on multiscale processing using a Laplacian pyramid, in my opinion, is hard to follow. A figure illustrating the process would greatly help.\n\n- The paper could benefit from some ablation tests, for example - using shared weights for $f_w$ and $g_w$ as opposed to separate weights, using different network architectures/depths and so on.\n\n- The authors present the method as a representation-learning framework, but it's not clear how the learnt representations could assist with tasks beyond next-frame prediction? I would appreciate listing some additional possible applications. Video compression is briefly mentioned as a possibility, but no details are given (are the representations compact relative to the original frame?)\n\n- Regarding the comparison to baseline methods - the baseline CNN architecture chosen for comparison seems quite arbitrary. On the one hand, the authors claim their method achieves comparable or better results using less parameters than the CNN, but on the other hand - no baseline CNN with the same number of parameters is given for comparison. In addition, the architecture seems arbitrary - 20 layers with the same number of channels, no skip connections in between (only one that wraps the entire network so that the predictions are residuals) and no information regarding non-linearities and/or normalization layers. It seems to me that choosing an of-the-shelf architecture for general image-to-image tasks (and slightly modifying the input/output) would have been a much better and competitive baseline. In addition, the realted work section mentions additional works on next-frame prediction (for example using LSTMs), which could contribute to the comparison. I would appreciate the author's response on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n* **Implementation details**: the learned transformations and their rationale are introduced in the main text; but, due to space limitation, their full description (architectures, datasets and optimization) is relegated to the supplementary material. We improved the readability of these sections, and in the rebuttal pdf, we included equations to summarize the polar prediction method.\n* **Laplacian pyramid**: we drew a multiscale diagram and included it as a figure in the rebuttal pdf, we also clarified the notation and text in the corresponding caption. The multiscale processing in mPP and mQP can be thought of as a preprocessing step: the analysis ($f_w$), prediction ($p_w$) and synthesis ($g_w$) steps are applied to the Laplacian coefficients ($\\Delta \\mathbf{x}$) instead of the image itself ($\\mathbf{x}$).\n* **Ablations**: we have explored decoupling the analysis and synthesis transformations (ie. untying feedforward and feedback weights) of a Polar Predictor (not multiscale). It achieved a similar prediction performance on the VanHateren dataset (tied: train 29.87/ test 28.83; vs. untied: train 29.99 / test 28.87 dB) and observed that the learned projection and basis vectors align as training progresses.\n* **Applications**: compression and segmentation are two downstream tasks where the proposed method may prove effective, see global response for a brief discussion.\n* **Comparison methods**: Although, we have not systematically optimized the hyperparameters of the CNN (we are reusing the dnCNN [1] architecture which has proven successful in other image-to-image tasks), we have added a U-net baseline - see global response. We have not studied fully recurrent architectures (such as LSTMs). \n* **Beyond next-frame**: We have observed that, as expected, the quality of the prediction quickly degrades with the temporal horizon, but we have not carefully compared the performance decay rate of different prediction methods. Computing further predictions in the latent space and then generating the corresponding frames using the synthesis transform is an interesting suggestion. Now that you mention it, it occurs to us that such predictions could be compared to a recursive approach where the next frame is predicted and then fed back in as an input - we intend to run these experiments.\n* **Speed**: For parameter count and latency, see global response and Table 2 in the rebuttals pdf.\n\n[1] Zhang, K., Zuo, W., Chen, Y., Meng, D. and Zhang, L., 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing"}, {"review_id": "Q45iX2SLGc", "reviewer": "Reviewer_HG1W", "summary": "The authors in this work propose a new self-supervised learning technique that is aimed to perform predictive processing of natural videos (although the approach seems broad enough to be applicable to other sequential signals with similar inductive priors to vision). The authors develop two parameter-efficient architectures (multiscale quadratic and polar prediction mechanisms, mQP, mPP) based on Fourier shift theorem and its group theoretic generalization and train them to perform next-frame prediction from natural videos. Experimental results show that their proposed abovementioned mQP and mPP architectures outperform relevant conventional motion coding baselines and a supervised CNN on video reconstruction on the two video datasets of VanHateren and DAVIS. It is very intriguing that an interpretation of the learned filters in their architecture shows similarity to canonical receptive fields observed in V1 cells, hence providing a new theoretical hypothesis for a potential function of V1 cells in performing future prediction.", "questions": "- Please see the weaknesses section above for my suggestions. ", "limitations": "The authors have addressed limitations of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The proposed work seems theoretically very sound, the proposed architectures mQP and mPP are designed ground up based on the extension of the Fourier shift theorem to account for transformations that occur in natural videos. Adequate mathematical and illustrative derivation of the proposed algorithm is very useful to grasp the underlying mechanism even for non-experts in this specific area such as myself.\n+ It is super interesting that canonical receptive field structure identified in V1 cells is emergent from the proposed self-supervised architecture and this serves as a foundation for the hypothesis of V1 functioning in temporal straightening and future prediction.\n+ Experimental results show that the proposed approach is clearly well-performing in terms of future frame prediction. These evaluations aren't very extensive like typical machine learning papers but the message from the paper is clearly conveyed in this demonstration. Additionally the link to biological modeling that I mentioned previously makes the paper a fresh read with many interesting avenues.\n+ The writing is great overall and related prior work has been covered quite comprehensively.", "weaknesses": "- In the experimental evaluation, it maybe good to include the number of parameters and runtime (training and inference if possible, but certainly for inference) for each algorithm. I believe this will provide a more full picture and show how much more efficient the proposed architecture is.\n- It would be great if the authors also add measures of variance for the performance of various algorithms on the two datasets compared in the evaluation section.\n- Although the authors have briefly touched upon potential alternate usecases of the proposed algorithm in other vision tasks (segmentation, direction estimation etc.) it would be good for readers in the broader audience to elaborate further on this. \n- Limitations of the proposed work aren't discussed in much detail, I only see that one potential limitation listed here is the inability of current proposed algorithms to generalize to non-commutative groups which is left for future scope. I encourage the authors to further discuss this in more detail and add other directions they see for extension of this work to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n* **Costs**: see global response and Table 2 of rebuttal pdf.\n* **Error bars**: see global response and Table 1 of rebuttal pdf.\n* **Applications**: see global response.\n* **Limitations**: see global response.\n"}, {"review_id": "YbKy94QbMe", "reviewer": "Reviewer_i4iN", "summary": "This research develops a self-supervised representation-learning framework that uses the regularities of natural videos to make accurate predictions. The architecture is inspired by the Fourier shift theorem and trained for next-frame prediction. The approach can discover irreducible representations of smooth commutative groups in data. It achieves similar performance comparing to the traditional motion compensation and conventional deep networks, while being interpretable and fast. The framework is implemented using normalized simple and direction-selective complex cell-like units, similar to primate V1 responses, demonstrating its potential to explain how the visual system makes temporal predictions.\n", "questions": "-\tDetailed experimental setup and dataset description are missing in the main text. \n-\tWould be beneficial to see other evaluation metrics on compression decoding quality and efficiency would be beneficial other than only PSNR metrics, e.g. bitrate, inference time etc.    \n-\tWill the learned representation also be useful for image classification or other static tasks?\n\n", "limitations": "The authors have drawn parallels between the computational steps of this approach and classical models of early visual processing in primates. However, the evidence supporting these parallels could be better substantiated. Quantitative comparison to physiological data may not be a straightforward task, and potential challenges in doing so should be discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper leverages the temporal regularity of the natural video and propose an video compression algorithm for temporal linearization and compression. The model exploits the local symmetries present in the temporal evolution of images therefore save computational resource comparing to the end-to-end CNN models. This is novel and original for video compression and image representation learning.\n-\tThe paper suggests an important research direction: instead of producing an end-to-end generic deep learning framework, the authors leverages the temporal regularity of the natural image sequence to reduce the computational cost. Human mind utilizes certain evolutionary bias to improve its efficiency, so should we exploit these shortcut and apply them into the computer vision research. \n-\tThe paper’s method is elaborated clearly with precise mathematical notation. The presentation flow is clear although it would be benefit to draw illustrative figures to show intuitively show the main idea of the paper. \n", "weaknesses": "-\tDespite the promising direction of research, the comparison between CNN models and the proposed method doesn’t demonstrate a significant improvement in terms of numbers. Further error bar would provide more concrete idea of how much the method has improved. \n-\tOther evaluation metrics could be useful such as the time of performing compression, as well as the computational resource. It would still be a good contribution if the model can achieve the same performance but using significantly less resources. Although the paper briefly mentioned this, it would be more thorough to outline the resource saving using the proposed method. \n-\tA description of the overall algorithm would make the presentation more clear. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n* **Compression**: The prediction method presented in this paper does not constitute a full video compression engine. Indeed the error are not quantized and we have not considered a full rate-distortion tradeoff. But we envision a possible application of the ideas developed in this study to coding because prediction is a key step in video coding.\n* **Error bars**: we added standard deviation of prediction error over multiple runs which helps interpret the relative performance of the algorithms, see Table 1 in rebuttal pdf.\n* **Costs**: our claim is that the performance of the proposed methods is on par with that of more complex models while using significantly fewer parameters and remaining interpretable, see Table 2 in rebuttal pdf.\n* **Algorithm description**: we added equations to describe the polar predictor algorithm, using concise vector notation, see rebuttal pdf.\n* **Detailed setup**: we omitted implementation details from main text due to space limitations, this information is available in the supplementary material of the original submission and we have improved the readability of the corresponding section.\n* **Metrics**: we computed SSIM, the results come out similar and do not change our initial interpretation, see global response.\n* **Downstream task**: we anticipate that segmentation, rather than recognition, would potentially benefit from the representation learned on the prediction task, a more detailed study is left for future work, see global response.\n* **Comparing with physiology**: We agree that quantitative comparison with physiological data is often challenging, especially given the amount of noise and the limited number of trials. In this study we operationalized a construction of visual signal prediction that can be cast in the same modeling framework that has been used to describe neural responses - and we aim to apply it to data in a future study.\n"}, {"review_id": "ZYBGqXznG8", "reviewer": "Reviewer_nLAh", "summary": "Motivated by Fourier shift theorem and its group-theoretic generalization, this paper proposed a new video prediction model.", "questions": "I am not familiar with computational neuroscience. Therefore, I can only provide useful feedback with respect to generative performance.", "limitations": "No limitations were discussed in the paper.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Biological modeling: computational elements can be used to describe primate V1 responses. \n2. Fourier shift theorem and its group-theoretic generalization were incorporated into future prediction", "weaknesses": "1. Evaluation: only simple baselines like CNN were included for comparison. Only one quantitative metric, PSNR, was used. Other metrics like SSIM or FVD can be more informative. No demonstration of any qualitative results.\n2. Loss: this framework used MSE as loss for optimization, which indicated this setting didn't consider any uncertainty factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n* **Evaluation**: we computed SSIM, the results come out similar and do not change our initial interpretation, see global response. A qualitative example and its interpretation is included in Figure 7 of the supplementary material in the original submission. We intend to move it to the main text and to add more examples in the appendix.\n* **Loss:** we used MSE for convenience because it is standard and amenable to optimization via gradient descent - although it is well known that image signals are not well captured by a gaussian model. If you have any suggestion, I would be curious to know what you are thinking about.\n* **Computational neuroscience:** one of the main contributions of this paper is to connect predictive video processing with modeling of the early visual system in computational neuroscience. By starting from fundamentals (Fourier shift theorem) and considering an abstract formulation (learning the representation of group transformations), we have exposed the unity of these two approaches. In particular, normative models of V1 receptive fields are typically based on coding efficiency and sparse coding, while our approach relies on the prediction principle instead and naturally accounts for several well documented phenomena such as normalized simple cell responses and direction selective complex cell responses."}, {"review_id": "pt94BcSxbn", "reviewer": "Reviewer_7L6B", "summary": "The authors create self-supervised video prediction model that relies on principles of the Fourier shift theorem. The approach is inspired by work suggesting that humans rely on perceptual straightened representations to support prediction. The authors first validate their method on toy cases of simple translation and rotation and later show that their model outperforms motion compensation and a some standard deep learning learning models at next frame prediction. Finally, they draw connections between their model and early visual processing in biological vision and discuss their model's impact more widely to computer vision (a simpler and more principled way to get natural video representations).", "questions": "I generally found the paper clear and interesting, but I would like to hear more on the following points:\n\nCould the authors discuss how their model might compare new architectures ViT and diffusion models for video prediction? Could the authors report MSE, SSIM, and lpips scores to help me get a better sense of how their model compares to other approaches? To that end, do the authors have any performance results on other datasets that are used in video prediction (for example moving MNIST or Kinetics)?\n\nDo the authors have any thoughts on other downstream tasks their video representation learning framework could be useful for? Or are there even aspects of video prediction that your model might do better at being grounded in a straightening representation space (for example long range prediction vs short range prediction)?\n", "limitations": "Limitations are not widely discussed, but aside from the weaknesses I highlighted I do not see a glaring limitation that needs to be addressed. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The methods are well thought-out and of good quality. The connection between perceptual straightening and the Fourier shift theorem is clever and novel. The use of both local and multi-scale processing in the model's design is is thoughtful and clear grounded in principles of human visual processing. In general, the authors make it very clear how their method relates to prior work -- this makes it the novelty and significance of the work clear.\n\nThe analyses of the model are good. I appreciate that the authors test their approach in synthetic data and then move to larger natural video datasets like DAVIS. Some good baselines are included in the analysis (copying frames, basic CNN), and there is an analysis of the filters learned by the model.", "weaknesses": "The main weakness I see not having enough baseline comparisons to existing methods in video prediction or video representation learning. The work only compares their model to a basic CNN which makes it difficult to understand how their model compares to more state of the art approaches or even older methods grounded in neuroscience like PredNet. Reporting scores for more models or adding another dataset (like Kinetics) with more performance measures like mse or lpips would address this point. In addition, more detail on compute and training requirements would help with understanding the significance of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n* **Architectures, metrics and datasets**: we have added a Unet architecture to our comparisons, see global response. We could not find a reference PyTorch implementation of the PredNet architecture, but to ease comparison in the future, we intend to release our code upon acceptance. Since a main motivation of our study was to develop an interpretable prediction method, we did not consider architectures like ViT or Diffusion models, although we do expect that such expressive architectures would perform very well (given enough data). We also computed SSIM and observed that the results come out similar and do not change our initial interpretation, see global response. We have also applied our method to the UCF-101 dataset and again the trend in the results was similar and chose not to include it to reduce clutter. We are currently designing toy datasets to tests specific hypothesis regarding video prediction (especially at occlusion boundaries).\n* **Downstream**: see applications in global response.\n* **Limitations**: see global response.\n"}]

Author rebuttal:
We thank the reviewers for their comments and questions.
The points that were raised by multiple reviewers are addressed in this global response
and the other questions are addressed in individual responses.

**Additions/extensions:**
* **Error bars**: single run prediction errors were reported in the initial submission, we now include average prediction error (and standard deviation) computed over ten random seeds - the results are assembled in Table 1 of the rebuttal pdf. Notice that the methods we intoduced (mPP and mQP) outperform simple baselines (Copy, cMC and SPyr) and rival more complex architectures (CNN and Unet) while being less variable (and also more interpretable).
* **New comparison algorithm**: we trained a Unet [1] for video prediction, it is a standard architecture for image-to-image tasks like segmentation. It has an encoder-decoder like structure with downsampling, upsampling, and skip connections between levels at the same resolution. We used a 5 levels architecture, each block consists of two convolutions, batch norm and rectification. The convolutions have filters of size 3 by 3, comprise 64 channels, and no additive bias. As expected, this architecture is very efficient: it runs fast (most of the computation is applied to spatially downsampled coefficients), it is the most performant on the VanHateren dataset, but it overfits on DAVIS (even though we limited the size of the model by using only 64 channels). This architecture comprises 20 non-linear stages (ReLU), which is to be compared to the single polar non-linearity in the polar predictor (mPP).
* **Costs**: The polar predictor described in this paper is lightweight: it runs very fast and contains few parameters (two orders of magnitude less than CNN and Unet). The polar predictor is designed as an online method that could be applied to streaming data and it is indeed very quick to train and to run. Parameter counts, as well as training and inference time are reported in Table 2 of the rebuttal pdf.
    * This project was developed on a NVIDIA A100 GPU, and used small datasets: 3575 frames of size 112 by 128 for VanHateren, 7332 frames of size 128 by 128 for DAVIS.
    * Note that the Quadratic Predictor (mQP) is slower, indeed we used a naive implementation of this model as it is not intended for performance but to build a connection with the computational neuroscience literature by showing how each computational element recapitulates know functional building blocks of primate V1 physiology.
* **Performance metric**: We report performance measured by PSNR (which is the logarithm of MSE in units of the signal). We also computed SSIM and observed that it did not change the trend or the interpretation of the results. A note on interpreting PSNR results: a PSNR of 0dB means that the signal and the error have the same variance, a PSNR of 20dB (resp. 40dB) means that the signal variance is 10 times (resp. 100 times) bigger than the error variance.

**Writing improvements:**
* **Unify algorithm**: Each element of the prediction computation and loss function were introduced gradually through the development of main text. To enhance clarity, we gather the equations that precisely describe the architecture and objective, using compact vector notation. These equations are at the top of rebuttal pdf.
* **Applications**: We briefly outline potential use of the learned mPP representation for other tasks.
    * Compression: in typical video compression engines like MPEG, the encoder computes and transmits motion vectors (slow to compute for the encoder but fast to apply for the decoder) as well as correction bits. Since our polar predictor can very rapidly compute a predicted frame, we envision a compression method that only sends prediction errors - thereby potentially saving substantial bitrate on motion vector. In practice, the feasibility of this approach would depend on the propagation of error quantization. Actually testing these ideas at different quantization levels is a large endeavor that is well beyond the scope of the present study.
    * Segmentation: visual content that moves together might belong to the same object, therefore the equivariant part of the polar predictor could be used to infer object boundaries by clustering phases changes. We expect that heading direction estimation could be approach in a similar way and leave this.
* **Limitations**: Throughout the main text, we have formulated assumptions and expressed the limitations of our prediction methods. We will consolidate them into a paragraph to be included in the discussion section of the main paper.
   * Our method has several limitations. The polar and quadratic prediction mechanisms assume fixed amplitude and linear phase increments in successive frames. Therefore these parameterizations cannot capture acceleration or longer temporal dependencies, and they also cannot precisely handle fast and long range displacements or deformations. In practice, and as can be seen in the qualitative prediction examples, error often occur at occlusion boundaries. We rely on a simple MSE objective function which corresponds to a simplistic gaussian noise model on the pixels, which we know to be an inadequate description of visual signals. The synthetic translation and rotation examples show that different representations are required to capture different group transformations, but the polar predictor architecture is forced to find a compromise between the two. We are now instead considering representations that can modulate their gain in order to adapt to the transformation currently acting in the data. Finally, recall that our choice of parameterization was motivated by the representation theory of commutative Lie groups, it can therefore not handle non-Abelian transformations and we leave exploration of QP mechanism's expressivity for future work.

[1] Ronneberger, Fischer, Brox 2015 U-net: Convolutional networks for biomedical image segmentation. In MICCAI

Meta-review: The paper proposes a new self-supervised learning technique for predictive processing of natural videos. The reviewers agree that the the ideas are novel and elegant. Several reviewers raised concerns about insufficient evaluation, which the authors addressed by adding another baseline and reporting that their results also hold with a different evaluation metric (SSIM; however, no numbers presented). Although the paper could probably be further improved, it appears to make an innovative contribution already in its current form.
Decision: Accept (poster)