SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The columns of the convolutional matrix  $\mathbf{V} \in \mathbb{R}^{N \times gNK}$  contain the groups of  $g$  filters (repeated at N locations). The number of channels in a group is no longer limited to pairs ( $|g| \geq 2$  is a hyper-parameter). The prediction matrices,  $\mathbf{M}_{k,t} \in \mathbb{R}^{g \times g}$ , are computed as a Linear-Square-Linear cascade on normalized activity from group of coefficients at the previous two time points:  $\mathbf{u}_{k,t} = [u_{k,t}, u_{k+1,t}, \ldots, u_{k+g-1,t}]^\top \in \mathbb{R}^g$  and  $\mathbf{u}_{k,t-1}$ . The linear combinations are learnable matrices  $\mathbf{L}_1 \in \mathbb{R}^{2g \times d}$  and  $\mathbf{L}_2 \in \mathbb{R}^{g^2 \times d}$ , where  $d$  is the number of quadratic units and squaring is computed pointwise. Note that in the case of pairs ( $g = 2$ ), six quadratic units suffice ( $d = 6$ ).
4.1 Recovery of planted symmetries
To experimentally validate our approach, we first verified that both the PP and the QP models can robustly recover known symmetries in small synthetic datasets consisting of translating or rotating image patches. For these experiments, the analysis and synthesis transforms are applied to the entire patch (i.e., no convolution). Learned filters for each of these cases are displayed in Figure 7, Appendix B. When trained on translating image patches, the learned filters are pairs of plane waves, shifted in phase by  $\pi /2$ . Similarly, when trained on rotating patches, the learned filters are conjugate pairs of circular harmonics. This demonstrates that theses models can recover the (regular) representation of some simple groups from observations of signals where their transformations are acting. When the transformation is not perfectly translational (e.g. translation with open boundary condition), the learned filters are localized Fourier modes. Note that when multiple kinds of transformations are acting in data (e.g., mixtures of both translations and rotations), the PP model is forced to compromise on a single representations. Indeed, the phase extrapolation mechanism is adaptive but the basis in which it is computed is fixed and optimized. A more expressive model would also allow for adaptation of the basis itself.
4.2 Prediction performance on natural videos
We compare our multiscale polar predictor (mPP) and quadratic predictor (mQP) methods to a causal implementation of the traditional motion-compensated coding (cMC) approach. For each block in a frame, the coder searches for the most similar spatially displaced block in the previous frame, and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We also compare to phase-extrapolation within a steerable pyramid [13], an overcomplete multi-scale decomposition into oriented channels (SPyr). We also implemented a deep convolutional neural network predictor (CNN), that maps two successive observed frames to an estimate of the next frame [14]. Specifically, we use a CNN composed of 20 non-linear stages, each consisting of 64 channels, and computed with  $3 \times 3$  filters without additive constants, followed by half-wave rectification. Finally, we also consider a U-net [15] which is a CNN that processes images at multiple resolutions (Unet). The number of non-linear stages, the number of channels and the filter size match that of the basic CNN. See descriptions in Appendix C for architectures and Appendix D for dataset and training procedures.
Table 1: Performance comparison. Prediction error computed on the DAVIS dataset. Values indicate mean Peak Signal to Noise Ratio (PSNR in dB) and standard deviation computed over 10 random seeds (in parentheses).
Method
split	Copy	cMC	SPyr	mPP	mQP	CNN	Unet
train	21.32	23.83	25.13	25.31 (0.04)	25.38 (0.11)	25.78 (0.18)	26.91 (0.38)
test	20.02	22.37	23.82	24.11 (0.01)	24.04 (0.06)	23.58 (0.05)	23.94 (0.06)
Prediction results on the DAVIS dataset [16] are summarized in Table 1. First, observe that the predictive algorithms considered in this study perform significantly better than baselines obtained by simply copying the last frame, causal motion compensation, or phase extrapolation in a steerable pyramid. Second, the multiscale polar predictor performs better than the convolutional neural networks on test data (both CNN and Unet are overfit). This demonstrates the efficiency of the polar predictor: the PP model has roughly 30 times fewer parameters than the CNN and uses a single non-linearity, while the CNN and Unet contain 20 non-linear layers. Appendix E contains additional a comparison of computational costs for these algorithms: number of trainable parameters, training and inference time. Note that on this dataset, the Unet seems to overfit to the training set. Moreover, the added expressivity afforded by the multiscale quadratic predictor did not result in significant performance gains. A representative example image sequence and the corresponding predictions are displayed in Figure 5. Additional results on a second natural video dataset are detailed in Appendix E and confirm these trends.
Figure 5: Example image sequence and predictions. A typical example image sequence from the DAVIS test set. The first three frames on the top row display the unprocessed images, and last five frames show the respective prediction for each method. The bottom row displays error maps computed as the difference between the target image and each predicted next frame on the corresponding position in the first row. All subfigures are shown on the same scale.
4.3 Biological modeling
The polar prediction model provides a hypothesis for how cortical circuits in the primate visual system might compute predictions of their inputs ("predictive processing") [17-20]. This framework is agnostic to how predictions are used and is complementary to candidate algorithms for signaling predictions and prediction errors across the visual hierarchy ("predictive coding") [21-23].
First, the learned convolutional filters of a polar prediction model resemble receptive fields of neurons in area V1 (primary visual cortex). They are selective for orientation and spatial frequency, they tile the frequency domain (making efficient use of limited resources), and filters in each pair have similar frequency selectivity and are related by a shift in spatial phase. Representative filters are displayed in Figure 6.
Second, the quadratic prediction mechanism derived in section 3.3 suggests a qualitative bridge to physiology. Computations for this model are expressed in terms of canonical computational elements of early visual processing in primates. The normalized responses  $u_{2k}(t)$  in equation 6 are linear projections of the visual input divided by the energy of related cells, similar to the normalization behavior observed in simple cells [24]. The quadratic units  $m_g$  in equation 7 are sensitive to temporal change in spatial phase. This selectivity for speed in a given orientation and spatial frequency is reminiscent of direction-selective complex cells which are thought to constitute the first stage of motion estimation [25, 26].
The polar prediction model is conceptually related to representation learning methods that aim to factorize visual signals. In particular, sparse coding with complex-valued coefficients [3] aims to factorize form and motion. More generally, several methods adopt a Lie group formalism to factorize invariance and equivariance. Since the seminal work that proposed learning group generators from dynamic signals [27], a polar parametrization was explored in [28] to identify irreducible representations in a synthetic dataset, and a corresponding neural circuit was proposed in [29]. The Lie group formalism has also been combined with sparse coding [30, 31] to model natural images as points on a latent manifold. More recently, bispectral neural networks [32] have been shown to learn image representations invariant to a given global transformation. In a related formalism, factored Boltzmann machines have been proposed to learn relational features [33]. The polar prediction model differs in two important ways: (1) unlike the coding approach that operates on  $i$ d data, it focuses on predicting, not representing, the signal; (2) the prediction objective does not promote sparsity of either amplitude or phase components and does not rely on explicit regularization. The discontinuity arising from selection of sparse subsets of coefficients seems at odds with the representation of continuous group actions [34]. The polar prediction model relies on a smooth and continuous parameterization to jointly discover and exploit the transformations acting in sequential data. The polar prediction model is convolutional and scales to natural video data, it can adapt to the multiple unknown and noisy
(a) spatial domain filters
(b) corresponding Fourier amplitude spectra
Figure 6: Learned filters. A single stage polar predictor was trained to predict videos from the DAVIS dataset. (a) Filters in each pair are displayed side by side and sorted by their norm. (b) Their amplitude spectra are displayed at corresponding locations. Observe that the filters are selective for orientation and spatial frequency, tile the frequency spectrum, and form quadrature pairs.
transformations that act in different spatial position. The literature on motion microscopy describes temporal processing of local phases in a fixed complex-valued wavelet representation to interpolate between video frames and magnify imperceptible movements [35]. Polar prediction processes phase in a learned representation to extrapolate to future frames.
The polar prediction model is related to other representation learning methods that also rely on temporal prediction. Temporal stability was used to learn visual representations invariant to geometric transformations occurring in image sequences [36]. The idea of learning a temporally straightened representation from image sequences was explored using a heuristic extrapolation mechanism [37] (specialized "soft max-pooling" and "soft argmax-pooling" modules). A related approach aimed at finding video representations which decompose content and pose and identify "components" in order to facilitate prediction of synthetic image sequences [38]. To tackle the challenge of natural video prediction, more sophisticated architectures have been developed for decomposing images into predictable objects [39]. A recurrent instantiation of predictive coding through time relying on a stacked convolutional LSTM architecture was proposed [40] and shown to relate to biological vision [41]. In contrast, the polar prediction model scales to prediction of natural videos while remaining interpretable. Another related approach, originating in the fluid mechanics literature, focuses on the Koopman operator [42]. This approach is a dynamical analog of the kernel trick: it lifts a system from its original state-space into a higher dimensional representation space where the dynamics can be linearized (i.e. represented by a fixed dynamics matrix). This formalism has inspired a line of work in machine learning: predictive auto-encoders learn coordinate systems that approximately linearize a system's dynamics [43]. Auxiliary networks have been introduced to adjust the dynamics matrix to velocity [44]. In contrast, the polar prediction model learns a representation that (implicitly) straightens the temporal evolution in an adaptive representation.
We've introduced a polar prediction model and optimized it to represent visual transformations. Using adaptive phase extrapolation in a fixed shiftable basis, the model jointly discovers and exploits the approximate symmetries in image sequences which are due to local content deformation. The basis is optimized to best diagonalize video dynamics by minimizing mean squared prediction error. The phase relationships are exploited implicitly in a bundled computation, bypassing the instabilities and discontinuities of angular phase variables. By starting from mathematical fundamentals and considering an abstract formulation in terms of learning the representation of transformation groups, we formulated a framework that makes three major contributions. First, it provides a method for discovering the approximate symmetries implicit in sequential data and complements methods for imposing known invariants. Second, it achieves accurate next-frame video prediction within a
principled framework and provides an interpretable alternative to standard architectures. Third, it offers a framework to understand the nonlinear response properties of neurons in primate visual systems, potentially offering a functional explanation for perceptual straightening.
The polar prediction model makes several strong assumptions. First, it is inertial, assuming that amplitude is unchanged and phase evolves linearly with no acceleration. Second, it separates spatial and temporal processing, which seems at odds with the spatiotemporal selectivities of visual neurons [45], but could enable downstream image based tasks (e.g. object segmentation, heading direction estimation). Third, it acts independently on each coefficient, although the representation does not perfectly diagonalize the dynamics of natural videos. This is analogous to the situation in modern nonlinear signal processing where diagonal adaptive operators in appropriate bases have had a major impact in compression, denoising, and linear inverse problems [46]. Our empirical results demonstrate that these assumptions provide a reasonable description of image sequences. The standard deep networks considered here could in principle have discovered a similar solution, but they seem not to. This exemplifies a fundamental theme in computational vision and machine learning: when possible, let the representation do the analysis. An important limitation of the framework comes from the use of mean squared error, which is minimized by the posterior mean and tends to result in blurry predictions. Since temporal prediction is inherently uncertain, predictive processing should be probabilistic and exploit prior information.
The polar prediction framework suggests many interesting future directions, both for the study of visual perception (e.g. object constancy at occlusion, and object grouping from common fate) and for the development of engineering applications (e.g. building a flow-free video compression standard). To expand expressivity and better represent signal geometry, it may be possible to design a polar prediction architecture that also adapts the basis, potentially extending to the case of noncommutative transformations (e.g. the two dimensional retinal projection of three dimensional spatial rotation). Finally, it is worth considering the extension of the principles described here to prediction at longer temporal scales, which will likely require learning more abstract representations.
We thank members of the Laboratory for Computational Vision at NYU, and of the Center for Computational Neuroscience at the Flatiron Institute for helpful discussions. This work was supported in part by the Simons Foundation.
[1] Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 1991. 1
[2] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 2002. 1
[3] Charles F Cadieu and Bruno A Olshausen. Learning intermediate-level representations of form and motion from natural movies. Neural computation, 2012. 1, 8
[4] Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc video coding standard. IEEE Transactions on circuits and systems for video technology, 2003. 1
[5] Peter Elias. Predictive coding-i. IRE transactions on information theory, 1955. 1
[6] Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 2019. 2
[7] Brian C Hall. Lie groups, lie algebras, and representations. In Quantum Theory for Mathematicians. Springer, 2013. 3
[8] William T Freeman, Edward H Adelson, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence, 1991. 3
[48] J Hans van Hateren and Dan L Ruderman. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 1998. 16
[49] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *ICLR*, 2015. 16
Let  $\mathbb{R}^N$  denote the  $N$ -dimensional Euclidean space equipped with the usual Euclidean norm  $||\cdot||$  and let  $\mathbb{C}^K$  denote the  $K$ -dimensional complex vector space. Let  $\mathbb{C}^{N\times K}$  denote the set of  $N\times K$  complex-valued matrices. Let  $\mathbf{F}^*$  denote the adjoint (ie. the conjugate transpose) of  $\mathbf{F}$ .
Given vectors  $\mathbf{u},\mathbf{v}\in \mathbb{C}^{K}$ , let  $\mathbf{u}\odot \mathbf{v} = [u_1v_1,\ldots ,u_Kv_K]^\top \in \mathbb{C}^K$  denote the elementwise (aka. Hadamard) product of  $\mathbf{u}$  and  $\mathbf{v}$ . Let  $\mathbf{u}\oslash \mathbf{v} = [u_{1} / v_{1},\dots ,u_{K} / v_{K}]^{\top}\in \mathbb{C}^{K}$  denote the elementwise division of  $\mathbf{u}$  and  $\mathbf{v}$ . Let  $\mathrm{diag}(\mathbf{u})$  denote the  $K\times K$  diagonal matrix whose  $(k,k)^{th}$  entry is  $u_{k}$ .
Let the complex number  $z \in \mathbb{C}$  be expressed in rectangular coordinates as  $z = x + iy$  where  $(x, y) \in \mathbb{R}^2$ , or in polar coordinates as  $z = ae^{i\theta}$  where  $(a, \theta) \in \mathbb{R}_+ \times [-\pi, \pi]$ . Its complex conjugate is  $\overline{z} = x - iy = ae^{-i\theta}$ . The rectangular coordinates are  $u = a\cos(\theta)$  and  $v = a\sin(\theta)$ ; and the polar coordinates are  $a = |z| = \sqrt{x^2 + y^2}$ , and  $\theta = \angle z = \mathrm{atan2}(y, x)$ . We overload these notations to denote element-wise operations on a vector  $\mathbf{z} \in \mathbb{C}^K$ :  $\overline{\mathbf{z}} = [\overline{z_1}, \dots, \overline{z_K}]^\top \in \mathbb{C}^K$  and  $|\mathbf{z}| = [|z_1|, \dots, |z_K|]^\top \in \mathbb{R}_+^K$ .
B Planted symmetries
(a) Translation (cyclic boundary)
(b) Rotation
(c) Translation and rotation
Figure 7: Learned filters on planted symmetries.
(d) Translation (open boundary)
Filters of polar predictor networks trained to predict small synthetic sequences. We randomly select 100 image patches of size  $16 \times 16$  from the DAVIS dataset and generate training data by manually transforming them—applying translations or rotations. We verify that PP recovers the corresponding harmonic functions: Fourier modes for translation (panel a), and disk harmonics for rotation (panel b). To show that the recovery of harmonics is robust, we design two additional synthetic datasets. i) the combination of translational and rotational sequences. In this case, PP learns some filters
that correspond to either transformation. But the model does not have the expressivity required to dynamically adapt the representation to either transformation (panel c); ii) generalized translation sequences: spatially sliding a square window on a large image (ie. new content creeps in and falls off at boundaries), instead of using cyclic boundary condition (ie. content wraps around the edges). In this case, PP learns localized Fourier-like modes (panel d), indicating that approximate group actions still provide meaningful training signal—but some of the filters are not structured. In each panel, the 32 pairs of filters are sorted by their norm. Notice that some of high frequency harmonics are missing. This is due to the spectral properties of the datasets, which have more power at lower frequencies.
C Description of architectures
Motion Compensation We compare our method to the traditional motion-compensated coding approach that forms the core of inter-picture coding in well established compression standards such as MPEG. Block matching is an essential component of these standards, allowing the compression of video content by up to three orders of magnitude with moderate loss of information. For each block in a frame, typical coders search for the most similar spatially displaced block in the previous frame (typically measured with MSE), and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We implemented a "diamond search" algorithm [47] operating on blocks of  $8 \times 8$  pixels, with a maximal search distance of 8 pixels which balances accuracy of motion estimates and speed of estimation (the search step is computationally intensive). We use the estimated displacements to perform causal motion compensation (cMC), using displacement vectors estimated from the previous two observed frames  $(\mathbf{x}_{t-1}$  and  $\mathbf{x}_t)$  to predict the next frame  $(\mathbf{x}_{t+1})$  rather than the current one (as in MPEG).
Complex Steerable Pyramid We consider a fixed multiscale oriented representation of image content: a steerable pyramid [13, 11] covering 16 orientations and 5 scales on the DAVIS dataset (resp. 16 orientations and 4 scales on VanHateren dataset). This choice of number of orientations and number of sacles maximizes prediction performance on the corresponding datasets.
Polar Predictor We use 32 pairs of convolutional channels with filters of size  $17 \times 17$  pixels, without biases (no additive constants) and no padding (ie. "valid" boundary condition). For the multiscale version (mPP), we use 16 pairs of convolutional channels with filters of size  $11 \times 11$  pixels, without biases (no additive constants). The representation is computed inside a fixed Laplacian pyramid [10]. We used 4 scales for the DAVIS dataset (and respectively 3 scales for the VANH dataset). Within this multiscale representation, the learned filters are applied with zero padding (ie. "same" boundary condition).
Quadratic Predictor For the multiscale version (mQP), we use the same analysis  $(f_w)$  and synthesis  $(g_w)$  hyperparameters as mPP. For the quadratic prediction mechanism, we use 16 groups of 4 convolutional filters (twice as many as for mPP). The quadratic predictor  $(p_w)$  operates on groups of 4 coefficients and contains 12 quadratic units. It is more expressive than the multiscale Polar Predictor architecture and contains phase advance as a special case.
Vanilla CNN Finally, we implemented a more direct convolutional neural network predictor (CNN), that maps two successive observed frames to an estimate of the next frame [14]. For this, we used a CNN composed of 20 stages, each consisting of 64 channels, and computed with  $3 \times 3$  filters without additive constants, followed by half-wave rectification (ie. ReLU). To facilitate learning, a skip connection copies the current frame  $\mathbf{x}_t$  and the network only outputs residuals that get added to the current frame in order to predict the next frame:  $\hat{\mathbf{x}}_{t+1} = \mathbf{x}_t + f_w([\mathbf{x}_t, \mathbf{x}_{t-1}])$ . This model jointly transforms and processes pairs of frames to generate predictions, while both polar predictor (PP) and quadratic predictor (QP) separate spatial processing and temporal extrapolation.
Unet The Unet architecture [15] is commonly used for image-to-image tasks. It has an analysis-synthesis structure with downsampling, upsampling, and skip connections between levels at the same resolution. We used a 5 levels architecture, each block consists of two convolutions, batch norm and rectification. The convolutions have filters of size  $3 \times 3$ , comprise 64 channels, and no additive bias. The end-to-end network comprises 20 non-linear stages with ReLU activations.
D Description of datasets and optimization