Learning to represent and predict evolving visual signals via polar straightening
Authors: Pierre-Etienne H Fiquet
Eero P Simoncelli
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video prediction
self-supervised representation learning
phase prediction
invariance / equivariance factorization

Abstract: Observer motion and continuous deformations of objects and textures imbue natural videos with distinct temporal structures, enabling the prediction of future frames from past ones.  Conventional methods proceed by estimating local motion, or optic flow, and then using this to predict future frames by warping and copying content.  Here, we explore a more direct methodology, in which frames are transformed into an alternative representation where temporal structure and evolution are more readily accessible. As a base case, a rigidly translating pattern can be described in the frequency domain as a linear combination of sinusoids, each with constant amplitude and phase that cycles at a rate proportional to its frequency. This fundamental property of Fourier representation reduces prediction to angular extrapolation. Motivated by the geometry of this well-known case, we formulate a self-supervised learning problem which seeks a transformation of video frames  to facilitate next-frame prediction in these natural polar coordinates. We construct a network architecture in which pairs of convolutional channels are used to factorize signals into slowly evolving amplitudes and linearly advancing phases. We train this network to predict future frames, and compare its performance with that of conventional methods using optic flow, and other learned predictive neural networks, evaluated on natural videos from the DAVIS dataset. We find that the polar predictor achieves high prediction performance while remaining interpretable and fast, thereby demonstrating the potential of a flow-free video processing methodology that is trained end-to-end to predict natural video content.

Under review as a conference paper at ICLR 2023
LEARNING TO REPRESENT AND PREDICT IMAGE SE- QUENCES VIA POLAR STRAIGHTENING
Anonymous authors Paper under double-blind review
ABSTRACT
Observer motion and continuous deformations of objects and textures imbue nat- ural videos with distinct temporal structures, enabling partial prediction of future frames from past ones. Conventional methods first estimate local motion, or op- tic flow, and then use it to predict future frames by warping and copying content. Here, we explore more direct methodology, in which frames are mapped into learned representation space where the structure of temporal evolution is more readily accessible. Motivated by the geometry the Fourier shift theorem and its group-theoretic generalization, video frames learned polar coordinates to facilitate prediction. Specifically, we construct networks in which pairs of convolutional channel coefficients interpreted as complex-valued, and are expected to evolve with slowly varying amplitudes and linearly advancing phases. We train these models on next-frame prediction, and compare their performance with that of conventional methods us- ing optic flow, and other learned predictive networks, evaluated on natural videos from two datasets. We find that the polar predictor achieves high prediction perfor- mance while remaining interpretable and fast, thereby demonstrating the potential of flow-free video processing methodology that is trained end-to-end to predict natural video content.
formulate simple architecture that represents
INTRODUCTION
One way to frame the fundamental problem of vision is that of representing the signal in form that more useful for performing visual tasks, be they estimation, recognition, or guiding motor actions. Perhaps the most general "task" is that of temporal prediction, which has been proposed as fundamental goal for unsupervised learning of visual representations (Földāák] 1991]. But previous research along these lines has generally focused estimating these transtormations rather than using them to predict: extracting slow features (Wiskott & Sejnowski 2002), and finding dictionaries and sparse codes that have slow amplitudes and phases (Cadieu & Olshausen 2012).
In video processing and computer vision, a common strategy for temporal prediction is to first estimate local translational motion, and then copy and paste content to predict the next frame. Such motion compensation is an important component in making compression schemes like MPEG successful (Wiegand et al. 2003). These video coding standards are the fruit of major engineering efforts, they make digital video communication feasible and are widely used. But motion estimation difficult nonlinear problem, and existing methods fail in regions where temporal evolution is not translational. For example, in cases of expanding or rotating motion, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi tami-transparent surfaces (e.g.. viewing the world through dirty pane of glass). In compression schemes, these failures of motion estimation lead to prediction errors, which are then fixed by sending additional corrective bits.
Human perception does not seem to suffer from such failures at least, our subjective sense is that we can anticipate the time-evolution visual input even in the vicinity these commonly occurring non-translational changes. In fact, those changes are often the most informative ones as they reveal object boundaries, provide ordinal depth and other information about the visual scene. This may imply that humans make of different strategy, perhaps bypassing altogether the estimation of motion, to represent and predict evolving visual input. Toward this end, and inspired by recent hypotheses that primate visual representations support prediction by "straightening" the temporal

---

Under review as a conference paper at ICLR 2023
input (Hénaff et al. 2019), formulate objective for learning an image representation that facilitates prediction by linearizing the temporal trajectories frames of natural video.
trajectories of
This separation of the instantancous representation and the temporal prediction is best motivated by considering the behavior of rigidly translating video content when viewed in the frequency domain. First in section I.I we review how translation corresponds to steady phase advancement in the frequency domain, and then in section 1.2 we explain how this relationship reduces prediction of rigidly translating content to angular ex extrapolation. We place this observation in the general context of group representation theory in section 1.3 Next in section 2 describe how to fit parameterized
mappings of individual video frames into complex coefficients which can be temporally predicted by phase advancement. These predicted representations are then used to synthesize an estimated frame, and the entire systems are trained end-to-end to minimize next frame prediction errors. In section 5] we report training results of several such systems, and show that they produce systematic improvements in predictive performance over conventional motion compensation methods, or direct predictive neural networks. Finally, in section El we relate our approach existing work and then in section S discuss its significance and implications.
1.1
BASE CASE: THE FOURIER SHIFT THROREM
Our approach motivated by the well-known behavior of Fourier representations with respect to signal translation. Specifically, the complex exponentials that make up the Fourier basis are the eigenfunctions of the translation operator, and translation of inputs produces systematic phase ad- vances of frequency coefficients. Let z RN be discrete signal indexed by spatial location n E [0, 1]. and let ã CN be its Fourier transform indexed by E [0, 1]. We write xx (n) æ(n v), the translation of by modulo N (ie. circular shift with period N). Defining
/ the primitive N-th root of unity,
express the Fourier shift theorem
N-1-v Σ
(k)
Σ æ(n o)o kn
x(m) km t-ke o kv
z(n) kn = -kk-E(())
n=0
m=
This relationship may be depicted in compact diagram:
F(k)
(1k)
advance phase
p-1
(nn)
x -
shift
where indicates the Fourier transform. In the context of our goals, the diagram illustrates the point that transforming to the Fourier domain renders translation "simpler" operation: phase advance a rotation in the two dimensional (complex) plane.
1.2 PREDICTION VIA ANGULAR EXTRAPOLATION
Now consider observations of signal that translates at a constant velocity over time, x(n,t) (nn 1---5 ut). Although the temporal evolution easy describe, the trajectory of the signal is quite complicated, rendering prediction difficult. As an example, Figure [ shows signal consisting of a sum of two sinusoidal components. Transforming the signal to the Fourier domain simplifies the description. In particular, the translational motion now corresponds circular motion of the two (complex-valued) Fourier coefficients associated with the constituent sinusoids.
The motion is further simplified by polar coordinate transform to extract phase and amplitude of each Fourier coefficient. Specifically, the motion is now along a straight trajectory, with both phases advancing linearly (but at different rates), and both amplitudes constant. Note that this is geometric property that holds for any rigidly translating signal, and offers a simple means of predicting content over time. Indeed, can use the shift property on x(n, 1+1) 1)= x. a²(n,) () and observe that prediction
2

---

Under review as a conference paper at ICLR 2023
52
O iima
5.0
4.9
4.8
15
rcal
10
0g
frequency
01
F
14h
2.5
0.0 PC3
60
2.5
100
PC2
2.5
0.25
1.00
PCI
2.5
Figure 1: Translation of ID signal consisting of sum of two sinusoidal components: 2(n, sin (2 - - (n t)) Lower left: three snapshots of the signal as translates. Lower right: In the figh-dimensional space representing the signal (each axis corresponding the signal value at one location), the temporal trajectory is highly curved. Shown is the projection of the signal vector into the 3D space of the top three principal components. Three colored points indicate the tree snapshots in lower left panel. Upper left: Fourier transform of the signal, showing complex-value this representation the temporal trajectory corresponds
coefficients function of frequency. to linearly increasing phase of the two sinusoidal components, each at rate proportional to its frequency. Upper right: polar coordinate transform to amplitude and phase of each frequency component leads to representation that evolves along straight line, and is thus readily predictable (phases are unwrapped for display purposes).
now reduced to linear extrapolation of each coefficient's phase. We have the three step process:
T(k,1)
Σ
kn x(n, (),
(analyze)
2=
T(k, 1 +
< !),
(advance phase)
N
s(n, (n,
An jn 1).
(synthesize)
Since we assume that the motion from time f to t+ is identical to that from time t- to t, the phase advance fv can be computed from the past two representations as fv Lā(k, t) - Lā(k, 1), where Ze is the phase of the complex number 2. Thus, polar coordinate transformation in the Fourier domain converts translational motion into straight trajectories which are predictable via linear phase extrapolation.
1.3 GENERALIZATION: REPRESENTING COMMUTATIVE CONTINUOUS GROUPS
Fourier analysis be seen special case of the representation theory of compact commuta- tive Lie groups (Mackey 1980). In harmonic analysis, the celebrated Peter- Peter-Weyl Theorem (1927) guarantees the existence of harmonic basis functions that generalize the Fourier shift property for
3

---

Under review as a conference paper at ICLR 2023
fw
fw
Fw
t
Figure 2: Unsupervised predictive representation learning framework. Each frame is transformed using a parametric mapping far to an internal space consisting of pairs of spatial channels. Pre- dictions of individual complex coefficients at time & by an amount equal to the phase advance over the interval from t to f (at each time step, one such coefficient is depicted as vector in two dimensions). Predicted frames are then constructed by applying the parameterized inverse mapping on the advanced coefficients. Both forward and inverse mappings are trained minimize the prediction error (typically, the mean squared error be- tween the predicted and actual frame at 1).
computed by advancing their phase
arbitrary compact Lie groups. Specifically, the linear unitary representation of G is given by plicit complete orthogonal basis, constructed from the finite-dimensional irreducible representations of G (Hall 2013). The defining property of irreducible representation is that its span is invariant under group action, so that arbitrary shifts can be expressed as linear combination of basis func- tions (an example is the construction of steerable filters (Freeman et al. 1991) in the computational vision literature). In the case of compact commutative Lie groups, complex irreducible represen- tations one-dimensional, meaning that harmonic basis functions come in pairs. Therefore, the angular extrapolation mechanism of the previous section can be employed for prediction in much wider setting than that of translational motion. Since the groups at play in natural videos (and their corresponding harmonic basis functions) are unknown, we aim to learn them from data.
LEARNING TO PREDICT WITH ANGULAR EXTRAPOLATION
To generalize beyond translation and the Fourier transform, we aim to learn a representation of video frames that enables prediction via angular extrapolation. Specifically, we focus next frame prediction, and optimize two parameterized mappings: one for the analysis and one for the synthesis transform. This framework is illustrated in Figure 2 notice its similarity with diagram 1
2.1 ARCHITECTURE AND OBJECTIVE FUNCTION
When focusing on small region in an image sequence, the transformation observed as time passes can be approximated
local translation. That is to say, in spatial neighborhood around position n, we have: æ(n, t + 1) ze r(n - v, t). We can reuse the setup described for global rigid translation, replacing the Fourier transform with local convolutional operator (Fleet & Jepson 1990). pro-
cessing each spatial neighborhood of the image independently and in parallel, and applying angular extrapolation the coefficients computed each position.
4

---

Under review as a conference paper at ICLR 2023
Analogous to the Fourier transform, use the same weights for the encoding and decoding stages, that is to say the analysis operator is the transpose of the synthesis operator. Sharing these weights reduces the number of parameters and simplifies interpretation of the learned solution. This "polar predictor" (hereafter PP) is consistent with the general scheme described in figure 2 where fu is taken be linear and convolutional, and fu is its transpose. In practice, we assumed 64 convolu- tional channels with filters of size 17 x 17 pixels, with no additive constants.
At every position in the image (spatial indices are omitted for clarity of notation), each coefficient (i(()) is computed as an inner product between the input xx() and the filter weights wy of each channel j E [0, 63] y5(() = w x((). In order to obtain phases, we combine coefficients in pairs, indexed by k E [0, which can be written as single complex coefficient as: (x(()
y2u (4) + iz (4) E C, and expressed in polar coordinates: =x(() = as (1)())))220) This polar coordinate transformation the only non-linear step used in the polar predictor architecture, pairs of coefficients and differs from more typical activation functions which operate on each coefficient independently.
With this notation, linear phase extrapolation reduces to (t + (x(t)c(( where the phase advance x0²(1) is equal to the phase difference over the interval from (-1 to l: A0x (l)()) (l 1). This can be written in a more compact form, using complex arithmetic, as:
2x (t
(t + 1)
(2)
(tx(t)| 1)1* 1)
where denotes complex conjugation and |a| the complex modulus. Note that the intermediate phase variable drops-out out when the whole computation is expressed using complex coefficients. This formulation bundles together phase and amplitude computations and has the benefit of handling phases implicitly, bypassing the phase unwrapping difficulty and avoiding the notorious instability of angular variables (indeed phase is unstable when amplitude is low), while still computing the desired angular extrapolation. In practice, this phase advance can be implemented using real valued elements only, as:
U2u (t + 1) 122+ (1 + sin (1) COS (!)
cos (tx -sin (f2
3/2x (t) |/22++1 (()
where trigonometric identities can be used calculate cos A0g (f) and sin A0 (t) without explicitly evaluating phases. We find that such an indirect formulation of phase processing is necessary for the stability of training, as previously noted in the texture modeling literature (Portilla & Simoncelli 2000).
non-linear representation implemented with deep convolutional neural networks for both for the encoder fw and the decoder Jw. The goal is to learn a transformation in which the coefficients can be predicted using the polar extrapolation mechanism. This "deep polar predictor" (deepPP) model can be written as 2(t + 1) Iw f (x(())), where A(t) depends on both xx) and ætt 1) and implements angular extrapolation in the code space as described in equation 3 This model amounts to a signal factorization into local amplitude and phase in a non- linear representation. We aim to study the effect of nin-liny separately from that of spatial scale, therefore we chose the number of layers and the kernel sizes to match the effective receptive field size of the shallow polar predictor, that is to say units in both models operate on the same size input region in the image. In practice, both the encoder and the decoder are composed of 4 convolutional layers, each with 64 channels, and using filter kernels of size 5x followed by half-wave wave rectification (ReLU). In this architecture, the encoder and the decoder are not tied together.
As second generalization
The convolutional kernels w are learned by minimizing the average prediction error, measured as squared error:
min E,|| #(t + 1)11.
We restrict the computation of this prediction error to the center of the image because moving content that enters from outside the video frame is inherently unpredictable. In practice, trim a 17-pixll strip from each side. Note that we only perform valid convolutions to avoid artificial interference with prediction, indeed zero-padding creates undesirable boundary artifacts.
5

---

Under review as a conference paper at ICLR 2023
2.2 COMPARISON MODELS
We compare our method to the traditional motion compensated coding approach that forms the core of inter-picture coding in well established compression standards such as MPEG. Block matching is an essential component of these standards, allowing the compression of video content by three orders of magnitude with moderate loss of information. For each block in a frame, typical coders search for the most similar spatially displaced block in the previous frame (typically measured with MSE), and communicate the displacement coordinates to allow prediction of frame content by trans- lating blocks of the (already transmitted) previous frame. Specifically, we implemented "diamond search" algorithm ZZuu & Ma 2000 operating on blocks of x pixels, with a maximal search distance of 8 pixels which balances accuracy of motion estimates and speed of estimation (the search step is computationally intensive). Here, we use the estimated displacements to perform causal mo- tion compensation (cMC), using displacement vectors estimated from the last two observed frames (I and 2τ) to predict the next frame (x++1) rather than the current one (as in MPEG).
To isolate the effects of the prediction in polar coordinates, we also implemented a predictor that uses linear extrapolation on the responses of a deep neural network (deepL), with architecture identical to that of the deep polar predictor. That is to say, we replace equation by: O 1), which amounts to enforcing linear dynamics in the latent space of a non-linear representation.
= 2y, (t) yy
We also implemented a more direct convolutional neural network predictor (CNN), that maps pairs of observed frames to an estimate of the next frame (Mathieu et al., 2016). Such mapping jointly transforms and predicts visual signals without explicitly partitioning spatial content representation and temporal feature extrapolation. Specifically we used CNN composed of 20 stages, each con- sisting of 64 channels, and computed with 3 x filters without additive constants, followed by half- wave rectification. Unlike the previous architectures, this model jointly processes pairs of frames.
2.3 DATASETS AND TRAINING
To train, test and compare these models, we use the DAVIS dataset (Pont- Tust-Tuseteen et al. 2017p, which was originally designed as a benchmark for video object segmentation. Image sequences in this dataset contain diverse motion of scenes and objects (at different magnitudes and speeds, with fixed and moving camera, etc.), which make next frame prediction challenging. Each clip is sampled at 25 frames per second, and is approximately 3 seconds long. The set is subdivided into 60 training videos (4741 frames) and 30 test videos (2591 frames). We pre-processed the data, converting all frames to monochrome luminance values, and scaling their range to the interval [-1, 1]. Frames reduced by cropping 256 x 256 central region, where most of the motion tends to occur, and by wown-sampling by a factor of two down to size 128 x 128 pixels. We hypothesize the temporal evolution present in natural signals to be sufficiently diverse, therefore we do not apply any data augmentation procedures. We crop videos temporally into brief snippets of 11 frames, which allows for prediction of 9 frames, and process them in batches of size 4. We train each model for one hundred epochs using the Adam optimizer with default parameters and a learning rate of 3 10-4 Kingma & Bal 2015). The learning rate is halved at epochs 50, 60, 70, 80, 90, 100. We use batch normalization with no additive terms before every half-wave rectification, rescaling by the standard deviation of channel coefficients. Similarly, we also trained on the larger UCF-101 dataset (Soomro et al.] 2012). This dataset, initially designed for action recognition, contains about 2.5 million frames, which amounts to over 27 hours of video data. Note that the clips are only available in compressed formats and may contain motion artifacts (due to frame coding), unlike the DAVIS dataset that is available as individual frames. We used the same pre- pre-prceesiig procedure, except that we only reduced by cropping 128 x 128 central region, without any down-sampling. We employ the same training procedure, except that we only run training for 25 epochs.
3 UNSUPERVISED REPRESENTATION LEARNING
3.1 PERFORMANCE
We summarize the prediction results in Table First, observe that the predictive algorithms consid- ered in this study perform significantly better than the baseline obtained by simply copying the last frame. Second, the polar predictor (PP) performs about as well as the vanilla convolutional neural

---

Under review as a conference paper at ICLR 2023
test RMSE
testl R RMSH
test RMSE
test MMSI:
0.2
PP CNN
0.1-
daxppp PP
0.0
deepiPP deepl.
0.0-
0.0
w-.
0.:0
-0.1
-0.2
0.00
0.25
0.50
0.00
0.25
0.30
0.0
0.2
0.4
0.2
0.4
PP
PP
deepPP
deepPP
(a)
(b)
(c)
(d)
Figure 3: Detailed performance comparison (in Root Mean Squared Error) of predictive algorithms.
Each point corresponds to frame in the test set. Vertical axes represent difference in performance: For points lying below the horizontal axis, the method whose performance is plotted on the hori- zontal axis achieves lower RMSE than the comparison method. Black crosses indicate average RMSE. Red point corresponds to example in Figure green to Figure 5 and blue to Figure
network (CNN) in terms of test mean squared error on DAVIS. This is remarkable because the PP model has roughly 30 times fewer parameters and uses a single non-linearity, while the CNN is composed of 20 non-linear layers. This shows that using a polar non-linearity contributes to perfor- mance. Finally, observe that deepPP achieves the lowest mean squared error, notably it outperforms the deepL model which is identical in architecture but employs a linear prediction mechanism. It also outperforms the CNN on the UCF-101 test dataset while remaining significantly simpler. This is further evidence that polar coordinates provide an inductive bias beneficial to the prediction task.
Table 1: Prediction enrrr computed on the DAVIS and UCF-101 datasets. Values indicate average Mean Squared Error.
Algorithm
dataset
split Copy
cMC
deepL
CNN
PP
deepPP
DAVIS
train 0.064
0.048
0.034
0.031
0.036
0.028
test
0.065
0.049
0.037
0.035
0.035
0.032
UCF-101
train 0.0302
0.0220
0.0210
0.0245
0.0216
test
0.0286
0.0299
0.0217
0.0215
0.0229
0.0210
# param
0
2
665,856
666,496
18,496
665,856
While average performance values provide compact summary, it is also informative to examine the distribution of prediction errors on individual frames from the test set. Figure ssows pairwise comparison of the predictive algorithms for each frame in the DAVIS dataset. To make the contrast more apparent, we display performance difference on the vertical axis. Note that while the models have been optimized to reduce mean squared error, show root mean squared error (RMSE) in order to facilitate visual inspection of the results (the concavity of the square root spreads out small differences). We see that (a) The polar predictor representation systematically outperforms causal motion compensation, especially on difficult examples. CNN on the bulk of easy medium cases but this advantage reversed for harder examples. (c) The deep polar predictor outperforms the single layer polar predictor overall, indicating that the non-linearity in representation can help, but there are many exceptions. (d) The deep polar predictor clearly outperforms the deep linear predictor, revealing the strong benefit of using polar extrapolation mechanism over a linear one.
(b) The polar predictor outperforms the
3.2 LEARNED FILTERS
In order better understand these results, visualized the learned filters of the polar predictor trained on the DAVIS dataset and observed that the learned filter are Gabor-like (selective for orien- tation and spatial frequency) and that that they tile the frequency domain. Interestingly, filters in
7

---

Under review as a conference paper at ICLR 2023
x(b-))
x(++1)
cMC
deopl.
CNN
PP
deepPP
target
prediction
MSE SSIM
0,0678
0.0387
0.0226
0.0197
0.0231
0.0187
0.35
0.48
0.51
0.55
0.56
0.59
Figure 4: A typical example image sequence from the DAVIS test set. The first three frames on the top row display the unprocessed images, and last five frames show the respective prediction for each method (with their shorthand above). The bottom row displays error maps computed as the difference between the target image ztt 1) and each predicted next frame the corresponding position in the first row. Images, predictions and error maps are shown on the same scale.
x(t-)
X(0
x(++1)
cMC
deepl.
CNN
PP
deepPP
MSE SSIM
0,0917
0.0798
0.043 0.24
0,0409 0.23
0,0394 0.27
0,0402
0.13
0.24
Figure 5: Another example image sequence see caption of Fig.
pair have similar frequency amplitude and, in the spatial domain, they are related by a 90 degrees phase shift. In other words, optimizing for angular extrapolation produces pairs filters that are in quadrature, see in Figure Va and 9b in the Appendix. This relationship is analogous to that of sines and cosines and is consistent with the structure of the angular extrapolation described in equation B
3.3 EXAMPLES
Consider set of example videos, chosen to illustrate behaviors of the methods being compared. In Figure $ we see a wall, its shadow and their sharp boundaries on a grass background as the camera moves. Both PP and deepPP generate good results, cMC produces a sharp prediction, but with significant blocking artifacts, and both the CNN and deepL tend toward excessive blurring. Figure S shows an example with nonrigid motions/deformations. As the biker advances to the right, the camera tracks and leads its displacement. Here, the cMC is is significantly sharper than the others, but introduces substantial artifacts. Again, the PP methods produce sharper results than either the CNN deepL methods. A couple additional informative examples are displayed in the appendix.
4 RELATED WORK
Our method is conceptually related to sparse coding with complex- representations (Cadieu & Oishausen 2012) in that factorizes natural videos into form and motion. But differs in a number of important ways: (1) sparse coding focuses representing, not predicting, the signal; (2) we do not promote sparsity of either amplitude or phase components; (3) finally, the discontinuity arising from selection of sparse subsets of coefficients seems odds with the representation of continuous group actions, while our explicit mapping into polar coefficients aims for a smooth and continuous parameterization of the transformations that occur in natural videos. Several other studies have
8

---

Under review as a conference paper at ICLR 2023
aimed to learn representations that decompose signal identity and attribute (ie. a what-where, or equiivariance factorization). In particular learning lincarized features from video was explored using heuristic extrapolation mechanism a toy dataset (Goroshin et al. 2015). The authors developed specialized "soft max-pooling" and "soft argmax-pooling" modules while we rely on principles of signal processing and representation theory chose the polar non-linearity and describe an effective and stable implementation thereof.
Our method is also related to work that adopts a Lie group formalism in representation learning. Since the seminal work of (Rao & Ruderman, 1998) that proposed learning Lie group generators from dynamic signals, the polar parametrization was explored in (Cohen & Welling 2014) iden- tify irreducible representations in synthetic dataset. The continuous group formalism has also been used in complement to to sparse coding (Chen et al., 2018 Chau al. 2020 to model natural images as a manifold in the space of pixels. In contrast, our formulation relies on predictive for- mulation rather than coding. Most importantly our work focuses on local symmetries rather than global signal transformations: polar predictor is able to adaptively capture and exploit different transformations in different regions of complex high-dimennional signals.
Finally, in the fluid mechanics literature, the Koopman operator approach (Mezić 2005 has been used to lift system from its original state-space to a higher dimensional representation space where its dynamics can be linearized dynamical analog of the well known kernel trick. This formalism has spurred a line of work in machine learning that relies on autoencoders to learn coordinate systems that approximately linearize dynamics (Lusch et al. 2018). In this perspective, our work also be interpreted as learning the spectral properties of an abstract Koopman operator operating on video data, specifically estimating its complex eigenvectors.
5 DISCUSSION
We have presented self-supervised representation-llarnig framework based on temporal predic- tion of image sequences via polar straightening. The optimized mappings validate the idea that pre- dictable features be extracted from natural video using an angular extrapolation in representation space. Our results also demonstrate that building structure into the architecture can be advantageous, provided that the said structure is well matched to the task. Specifically, we rely on the polar co- ordinate transformation as a bivariate non-linear activation function to pair together coefficients of the representation and show that such an activation function, all else being equal, induces perfor- mance benefits. This exemplifies a fundamental theme in computational vision: when possible, let the representation do the analysis. More concretely, this effect was enabled by bundling angular computations, such that the phase manipulation remains implicit (angular variables are not directly represented in our implementation). This choice of prediction mechanism, which was motivated by principles of signal processing and harmonic analysis, acts as a structural prior and embodies the general hypothesis that evolving content will continue to evolve in the same way. Although the con- ventional deep convolutional network (CNN) considered here could in principle have discovered this solution, it failed to do so (within the constraints of our setup), which emphasizes the importance of architectural design choices that act as inductive biases. Using terminology from group theory, our polar models factorize signals into an invariant part, which stable in time, and an equivariant part, which evolves linearly.
This approach to prediction has the advantage of being motion-informed while not relying on explicit motion estimation. Because it is not constrained assigning a single motion vector every location and instead relies on a distribution of phases, this method outperforms the motion compensated algorithm (it also runs several order of magnitude faster). The polar predictor is not only able to bypass known difficulties of motion estimation, but also to generate a representation of each frame. would be interesting to validate the generality of that representation by evaluating it on other tasks like object categorization or segmentation, as well as estimation of heading direction of a moving observer. Several natural extensions of the work presented here can be further explored: (i) treating the angular extrapolation prediction mechanism as a more general building block that is cascaded in a multi-layer architecture; (ii) predicting at longer timescales from representations deeper in the hierarchy; (ii) measuring prediction eиror directly in the representation domain, while avoiding representation collapse this would allow a potential connection with biological neural architecture and to human visual perception.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Charles F Cadieu and Bruno A Olshausen. Learning intermediate-lever representations form and motion from natural movies. Neural computation, 2012.
Ho Yin Chau, Frank Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lie group transformations and sparse coding. arXiv preprint arXiv:2012.12077 2020.
Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018.
Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning. PMLR, 2014.
David J Fleet and Allan Jepson. Computation of component image velocity from local phase information. International journal of computer vision, 1990.
Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 1991.
William Freeman, Edward Adelson, et al. The design and use of steerable filters. IEEE Trans- actions on Pattern analysis and machine intelligence, 1991.
Ross Goroshin, Michael F Mathieu, and Yann LeCun. Learning to linearize under uncertainty. In Advances in Neural Information Processing Systems, 2015.
Brian Hall. Lie groups, lie algebras, and representations, In Quantum Theory for Mathematicians. Springer, 2013.
Olivier Hénaff, Robbe LT Goris, and Eero Simoncelli. Perceptual straightening of natural videos.
Nature neuroscience, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamies. Nature communications, 2018.
George W Mackey. Harmonic analysis as the exploitation of symmetry-a historical survey. Bulletin (New Series) of the American Mathematical Sociery, 3(1. P1):543 6988, 1980.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016.
Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Non- linear Dynamics, 2005.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arheláez, Alexander Sorkine- and Luc Van Gool. The 2017 davis challenge video object segmentation. urX4v:/704.0000 2017.
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of com- plex wavelet coefficients. International journal of computer vision, 2000.
Rajesh Rao and Daniel Ruderman. Learning lie groups for invariant visual perception. Advances in neural information processing systems, 1998.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint 12f.:1222204222 2012.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from enror visibility to structural similarity. JEEE transactions on image processing, 2004.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 2003.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 2002.
Shan Zhu and Kai-Kut Kui-Kuang Ma. A new diamond search algorithm for fast matchin motion estimation. IEEE transactions on Image Processing, 2000.
10

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Table Prediction error computed on the DAVIS dataset. Values indicate average PSNR and SSIM. All methods obtain relatively low structural similarity scores (Wang et al. 2004). perceptual mea- sure of similarity that equals one when both images are identical. This indicates that prediction on this dataset is quite challenging.
Algorithm
metric
set
Copy
cMC
deepl.
CNN
PP
deepPP
PSNR
train 21.32
23.82
23.18
23.84
24.49
24.52
test
20.06
22.37
22.30
22.82
23.46
23.35
SSIM 1
train 0.52
0.64
0.58
0.62
0.65
0.64
test
0.50
0.62
0.55
0.59
0.63
0.61
test RMSE
despPPp- CNN
0.00
00.05
0.0
deepPP
Figure 6: Another comparison caption of Fig. forms the CNN over the whole range of difficulties.
The deep polar predictor generally outper-
x(t-1)
x(t+l)
deepl.
deepPP
SSIM
0.32
0.77
0.46
0.51
0.72
0.61
Figure 7: Another example image sequence caption of Fig.
A video with portion not
moving. In these regions the cMC is most effective, and PP and deep outperform CNN and deepl..

---

Under review as a conference paper at ICLR 2023
x(++)
cMC
deopl.
CNN
PP
deepP'P
MSE SSIM
0,0365
0.0454
0,0300
0,0315 0.27
0.0290
0.0313
0.23
0.27
0.27
Figure 8: Another image sequence see caption Fig. As expected, in presence of large non- rigid motion, with texture and object deformations, prediction is very difficult and none of the tested methods performs well.
spatial domain filters
(b) corresponding Fourier amplitude spectra
Figure 9: Optimized filters of polar predictor: The 32 pairs of filters are sorted by their norm and their amplitude spectrum is displayed at corresponding locations on the right panel. Observe that the filters selective for orientation and spatial frequency, tile the frequency spectrum, and form quadrature pairs. interesting that such structured results were obtained from optimizing filters jointly and from random initial condition, but notice that some of the learned filters do not exactly conform to the idealized description just given.
Figure 10: Another set of optimized filters of size 17 x 17, these trained in PP architecture synthetic sequences generated by rotating image patches.
12


Reviews:
[{"review_id": "x3TjxlI1gl", "reviewer": "Reviewer_gEGq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a self-supervised learning objective based on next-frame prediction that uses angular extrapolation in polar coordinates while keeping amplitudes constant. They show that this inductive bias of using angular instead of linear extrapolation improves next-frame prediction performance in terms of MSE and SSIM on video snippets taken from the DAVIS dataset.\n", "summary_of_the_review": "Nice and simple paper, but somewhat weak and shallow on the experimental side.\n", "strength_and_weaknesses": "### Strengths\n\n + Conceptually well motivated idea\n + Paper is well written and easy to read\n\n\n### Weaknesses\n\n 1. Only small dataset and shallow architecture\n 1. Unclear how robust improvements are\n\n\n### 1. Small dataset and shallow architecture\n\nIt is great to see that the well-motivated inductive bias helps in the relatively data-limited regime the authors study. However, as video data is abundant and easy to obtain in masses, it is not clear to me how relevant such an inductive bias is. Would the advantage also hold on a much larger dataset with a much deeper network architecture?\n\nThe fact that the linear encoder/decoder works so well suggests to me that not much of relevance can be learned from such a small dataset. Presumably a deep architecture trained on a large dataset would learn something about objects and their 3d motion patterns and be able to outperform a linear encoder/decoder by a large margin.\n\n\n### 2. Robustness of improvements\n\nRelated to the previous point that the linear method is so good, I think it would be important to provide error bars on Table 1. How confident are we that the differences between the methods are indeed significant? How much variability is there between multiple differently initialized models? What does it tell us that SSIM is more or less the same across cMC, PP and deepPP?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GayXR4swsJB", "reviewer": "Reviewer_67e4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the Fourier Shift Theorem, the paper tries to learn a representation space where the temporal evolution is equivalent to a matrix multiplication by a dynamics matrix A(t). This dynamics matrix is learned by minimizing the one step prediction error in an end to end fashion. More specifically, a frame x(t) is encoded into representation z, where it is advanced through the dynamics matrix and then decoded back. The decoded should be same as the next frame x(t+1). Through experiments on DAVIS dataset, the paper shows that this kind of representation is powerful and outperforms a couple of baselines like convolutional network based prediction and causal motion compensation.", "summary_of_the_review": "I liked the fresh perspective presented in the paper. However, the paper has some rooms for improvement as discussed earlier. Hence, I am on the side of rejection at the moment.", "strength_and_weaknesses": "I appreciate that the paper tries to develop a simple, lightweight and interpretable method and provides insights on the algorithm. Some of the ideas in the paper like learning an appropriate representation is really powerful is an important message.\n\nHowever, the paper needs strengthening in several fronts. The first and the most important question is a theoretical and a conceptual one. Even though the inspiration from Fourier shift theorem is interesting, it is hard to justify the method based just on that since in real videos, frame to frame transformation is rarely just a translation. Second, even though conceptually the paper discusses Fourier transform, the implementation requires a function (W or g) to be learned which could very well be different from the Fourier transform. However, this limitation can be easily mitigated if the authors choose to develop their work from the perspective of Koopman embedding [1,3,4]. Koopman operator theory is precisely the generalization that authors are looking for. Koopman embedding is the representation space where the complicated nonlinear dynamics are represented as linear dynamics. The dynamics matrix A discussed in the paper could actually be thought as the Koopman operator. \n\nInterestingly, there already exist a couple of papers that try to use Koopman embedding in video prediction. For example, Comas et al [2] is conceptually doing something very similar to this paper, although they are separating objects from the video and applying linear dynamics to the individual objects in some representation space.\n\nSecond important front where the paper needs improvement is its experiments or empirical validation. Although some of the insights in the paper are interesting, they need to be validated in a couple of dataset to support the calim that those results and improvements are indeed general. At the moment, the paper only validates results in one dataset.\n\nThird important improvement could come from comparison with stronger deep learning methods in video prediction. It is interesting that the method beats vanilla CNN or deep network without nonlinearity, but there are other specialized works for the purpose of video prediction (e.g. DDPAE [5]). Comparison to those will tell us where the method stands. \n\nReferences:\n1. Bethany Lusch, J. N. Kutz, and S. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9, 2018.\n2. Comas, A., Ghimire, S., Li, H., Sznaier, M. and Camps, O., 2021. Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspective. arXiv preprint arXiv:2110.00547.\n3. Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning (ICML), 2020\n4. Jeremy Morton, F. Witherden, and Mykel J. Kochenderfer. Deep variational koopman models: Inferring koopman observations for uncertainty-aware dynamics modeling and control. In International Joint Conferences on Artificial Intelligence (IJCAI), 2019\n5. Hsieh, Jun-Ting, Bingbin Liu, De-An Huang, Li F. Fei-Fei, and Juan Carlos Niebles. \"Learning to decompose and disentangle representations for video prediction.\" Advances in neural information processing systems 31 (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CiDNJc__X9", "reviewer": "Reviewer_zsxR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new way to predict video frames inspired by how rigid objects translate in the Fourier domain, where translation becomes angular progression in a straight line (in polar coordinates). The model simulates (as I understand it) the fourier transform using a particular non-linearity that projects the data into a space similar to the fourier domain, transforms it, and then maps it back. The method is compared to a number of other methods and shows superior performance in terms of MSE and SSIM scores. One version uses orders of magnitude fewer parameters and still outperforms the deep learning baseline methods in terms of MSE. On the other hand, it just barely outperforms a 2 parameter method in terms of SSIM. \n", "summary_of_the_review": "To the extent that my knowledge about this area is limited to what is in the paper, this is a novel approach to video prediction, and performs well against the baselines. However, I recall a paper or talk by Yann LeCun and NeurIPS a few years ago that gave very good predictions of video frames by assuming a latent variable that somehow influenced the prediction and \"chose\" one future out of the possible ones. The predictions he showed were very clear, but I'm not sure how that relates to this work. Also, the straight convolutional network predictor approach is from 2016; surely more has happened in this area in the last 6 years?", "strength_and_weaknesses": "Strengths:\n\n+ The inspiration for this method is good.\n\n+ I am not an expert in this area, but it appears to be a completely novel method.\n\n+ The performance is good compared to the baseline methods.\n\n+ The writing is very clear.\n\nWeaknesses, with concrete, actionable feedback\n\n- It is hard to evaluate the qualitative results in Figures 5-8.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject