SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pierre-Étienne H. Fiquet<sup>1</sup> Eero P. Simoncelli<sup>1,2</sup>
<sup>1</sup> Center for Neural Science, New York University  
<sup>2</sup> Center for Computational Neuroscience, Flatiron Institute {pef246, eero.simoncelli}@nyu.edu
All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.
The fundamental problem of vision can be framed as that of representing images in a form that is more useful for performing visual tasks, be they estimation, recognition, or motor action. Perhaps the most general "task" is that of temporal prediction, which has been proposed as a fundamental goal for unsupervised learning of visual representations [1]. Previous research along these lines has generally focused on estimating stable representations rather than using them to predict: for example, extracting slow features [2], or finding sparse codes that have slow amplitudes and phases [3].
In video processing and computer vision, a common strategy for temporal prediction is to first estimate local translational motion, and then (assuming no acceleration) use this to warp and/or copy previous content to predict the next frame. Such motion compensation is a fundamental component in video compression schemes as MPEG [4]. These video coding standards are the result of decades of engineering effort [5], and have enabled reliable and efficient digital video communication that is now commonplace. But motion estimation is a difficult nonlinear problem, and existing methods fail in regions where temporal evolution is not translational and smooth: for example, expanding or rotating motions, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi-transparent surfaces (e.g., viewing the world through a dirty pane of glass). In compression schemes, these failures of motion estimation lead to prediction errors, which must then be adjusted by sending additional corrective bits.
Human perception does not seem to suffer from such failures—subjectively, we can anticipate the time-evolution of visual input even in the vicinity of these commonly occurring non-translational changes. In fact, those changes are often highly informative, revealing object boundaries, and providing ordinal depth and shape cues and other information about the visual scene. This suggests that the human visual system uses a different strategy, perhaps bypassing altogether the explicit estimation of local motion, to represent and predict evolving visual input. Toward this end, and inspired by the recent hypothesis stating that primate visual representations support prediction by "straightening" the temporal trajectories of naturally-occurring input [6], we formulate an objective for learning an image representation that facilitates prediction by linearizing the temporal trajectories of frames in natural videos.
To motivate the separation of spatial representation and temporal prediction, we first consider the special case of rigidly translating signals in one dimension. In the frequency domain, translation corresponds to phase advance (section 2.1), which reduces prediction to phase extrapolation (section 2.2). We invoke basic arguments from group theory to motivate the search for generalized representations (section 2.3). We propose a neural network architecture that maps individual video frames to a latent space where prediction can be computed more readily and then mapped back to generate an estimated frame (section 3). We train the entire system end-to-end to minimize next frame prediction errors and verify that, in controlled experiments, the learned weights recover the representation of the group used to generate synthetic data. On natural video datasets, our framework consistently outperforms conventional motion compensation methods and is competitive with deep predictive neural networks (section 4). We establish connections between each element of our framework and the modeling of early visual processing (section 4.3).
2.1 Base case: the Fourier shift theorem
Our approach is motivated by the well-known behavior of Fourier representations with respect to signal translation. Specifically, the complex exponentials that constitute the Fourier basis are the eigenfunctions of the translation operator, and translation of inputs produces systematic phase advances of frequency coefficients. Let  $\mathbf{x} = [x_0,\dots ,x_{N - 1}]^\top \in \mathbb{R}^N$  be a discrete signal indexed by spatial location  $n\in [0,N - 1]$ , and let  $\widetilde{\mathbf{x}}\in \mathbb{C}^N$  be its Fourier transform indexed by  $k\in [0,N - 1]$ . We write  $\mathbf{x}^{\downarrow v}$ , the circular shift of  $\mathbf{x}$  by  $v$ ,  $x_{n}^{\downarrow v} = x_{n - v}$  (with translation modulo  $N$ ). Let  $\phi$  denote the primitive  $N^{\mathrm{th}}$  root of unity,  $\phi = e^{i2\pi /N}$ , and let  $\mathbf{F}\in \mathbb{C}^{N\times N}$  denote the Fourier matrix,  $F_{nk} = \frac{1}{\sqrt{N}}\phi^{nk}$ . Multiplication by the adjoint (i.e. the conjugate transpose),  $\mathbf{F}^*$ , gives the Discrete Fourier Transform (DFT), and by  $\mathbf{F}$  the inverse DFT. We can express the Fourier shift theorem as:  $\mathbf{x}^{\downarrow v} = \mathbf{F}\mathrm{diag}(\phi_v)\mathbf{F}^*\mathbf{x}$ , where  $\phi_v = [\phi^0,\phi^{-v},\phi^{-2v},\ldots ,\phi^{-(n - 1)v}]^\top$  is the vector of phase shift by v. See Appx. A for a consolidated list of notation used throughout this work.
This relationship can be depicted in a compact diagram:
$$
\begin{array}{c} \widetilde {x} _ {k} \xrightarrow [ \text {p h a s e s h i f t} ]{} \phi^ {- k v} \widetilde {x} _ {k} \\ \left( \begin{array}{l l} \mathbf {F} ^ {*} & \mathbf {F} \\ x _ {n} & \text {s p a t i a l s h i f t} \end{array} \right) \xrightarrow {} x _ {n - v} \end{array} \tag {1}
$$
This diagram illustrates how transforming to the frequency domain renders translation a "simpler" operation: phase shift acts as rotation in each frequency subspace, i.e. it is diagonalized.
Proof by substituting  $m = n - v$ :
$\widetilde{x^{\downarrow v}}_k = \sum_{n = 0}^{N - 1}\phi^{-kn}x_{n - v} = \sum_{m = -v}^{N - 1 - v}\phi^{-kv}\phi^{-km}x_m = \phi^{-kv}\sum_{n = 0}^{N - 1}\phi^{-kn}x_n = \phi^{-kv}\widetilde{x}_k.$
Figure 1: Straightening translations. (a) Three snapshots of a translating signal consisting of two superimposed sinusoidal components:  $x_{n,t} = \sin (2\pi (n - t)) + \sin (2\pi 3(n - t)) / 2$ . (b) Projection of the signal into the space of the top three principal components. The colored points correspond to the three snapshots in panel (a). In signal space, the temporal trajectory is highly curved—linear extrapolation fails. (c) Complex-valued Fourier coefficients of the signal as a function of frequency. The temporal trajectory of the frequency representation is the phase advance of each sinusoidal component. (d) Trajectory of one amplitude and both (unwrapped) phases components. The conversion from rectangular to polar coordinates reduces the trajectory to a straight line—which is predictable via linear extrapolation.
2.2 Prediction via phase extrapolation
Consider a signal, the  $N$ -dimensional vector  $\mathbf{x}_t$ , that translates at a constant velocity  $v$  over time:  $x_{n,t} = x_{n-vt,0}$ . This sequence traces a highly non-linear trajectory in signal space, i.e. the vector space where each dimension corresponds to the signal value at one location. In this space, linear extrapolation fails. As an example, Figure 1 shows a signal consisting of a sum of two sinusoidal components in one spatial dimension. Mapping the signal to the frequency domain simplifies the description. In particular, the translational motion now corresponds to circular motion of the two (complex-valued) Fourier coefficients associated with the constituent sinusoids. In polar coordinates, the trajectory of these coefficients is straight, with both phases advancing linearly (at a rate proportional to their frequency), and both amplitudes constant.
2.3 Generalization: representing transformation groups
Streaming visual signals are replete with structured transformations, such as object displacements and surface deformations. While these can not be captured by the Fourier representation, which only handles global translation, the concept of representing transformations in their eigen-basis generalizes. Indeed, representation theory describes elements of general groups as linear transformations in vector spaces, and decomposes them into basic building blocks [7]. However, the transformation groups acting in image sequences are not known a priori, and it can be difficult to give an explicit representation of general group actions. In this work, we aim to find structures that can be modeled as groups in image sequences, and we learn their corresponding representations from unlabeled data.
In harmonic analysis, the Peter-Weyl Theorem (1927) establishes the completeness of the unitary irreducible representations for compact topological groups (an irreducible representation is a subspace that is invariant to group action and that can not be further decomposed). Furthermore, every compact Lie group admits a faithful (i.e. injective) representation given by an explicit complete orthogonal basis, constructed from finite-dimensional irreducible representations [7]. Accordingly, the action of a compact Lie group can be expressed as a rotation within each irreducible representation (an example is the construction of steerable filters [8] in the computational vision literature).
In the special case of compact commutative Lie groups, the irreducible representations are one-dimensional and complex-valued (alternatively, pairs of real valued basis functions). These groups have a toroidal topology and, in this representation, their action can be described as advances of the phases. This suggests a strategy for learning a representation: seek pairs of basis functions for which phase extrapolation yields accurate prediction of upcoming images in a stream of visual signals.
3.1 Objective function
We aim to learn a representation of video frames that enables next frame prediction. Specifically, we optimize a cascade of three parameterized mappings: an analysis transform  $(f_w)$  that maps each frame to a latent representation, a prediction in the latent space  $(p_w)$ , and a synthesis transform  $(g_w)$  that maps the predicted latent values back to the image domain. The parameters  $w$  of these mapping are learned by minimizing the average squared prediction error:
$$
\min  _ {w} \sum_ {t} \| \mathbf {x} _ {t + 1} - g _ {w} \left(\hat {\mathbf {z}} _ {t + 1}\right) \| ^ {2}; \quad \text {w h e r e} \hat {\mathbf {z}} _ {t + 1} = p _ {w} \left(\mathbf {z} _ {t}, \mathbf {z} _ {t - 1}\right), \text {a n d} \mathbf {z} _ {t} = f _ {w} \left(\mathbf {x} _ {t}\right). \tag {2}
$$
An instantiation of this framework is illustrated in Figure 2. Here the analysis and synthesis transforms are adjoint linear operators, and the predictor is a diagonal phase extrapolation.
Figure 2: Polar prediction model. The previous and current images in a sequence  $(\mathbf{x}_{t-1}$  and  $\mathbf{x}_t)$  are convolved with pairs of filters  $(\mathbf{W}^*)$ , each yielding complex-valued coefficients. For a given spatial location in the image, the coefficients for each pair of filters are depicted in complex planes with colors corresponding to time step. The coefficients at time  $t+1$  are predicted from those at times  $t-1$  and  $t$  by extrapolating the phase  $(\delta_t)$ . These predicted coefficients are then convolved with the adjoint filters  $(\mathbf{W})$  to generate a prediction of the next image in the sequence  $(\hat{\mathbf{x}}_{t+1})$ . This prediction is compared to the next frame  $(\mathbf{x}_{t+1})$  by computing the mean squared error (MSE) and the filters are learned by minimizing this error. Notice that, at coarser scales, the coefficient amplitudes tend to be larger and the phase advance smaller, compared to finer scales.
3.2 Analysis-synthesis transforms
Local processing When focusing on a small spatial region in an image sequence, the transformation observed as time passes can often be well approximated as a local translation. That is to say, in a spatial neighborhood around position  $n$ ,  $m \in N(n)$ , we have:  $x_{m,t+1} \approx x_{m-v,t}$ . We can use the decomposition described for global rigid translation, replacing the Fourier transform with a learned local convolutional operator [9], processing each spatial neighborhood of the image independently and in parallel.
At every position in the image (spatial indices are omitted for clarity of notation), each pair of coefficients is computed as an inner product between the input and the filter weights of each pair of channels. Specifically for  $k \in [0, K]$ , where  $K$  is the number of pairs of channels, we have:  $y_{2k,t} = \mathbf{v}_{2k}^{\top} \mathbf{x}_t$  and  $y_{2k+1,t} = \mathbf{v}_{2k+1}^{\top} \mathbf{x}_t$ . Correspondingly, an estimated next frame is generated by applying the transposed convolution  $g_w$  to advanced coefficients (see section 3.3). We use the same weights for the encoding and decoding stages, that is to say the analysis operator is the conjugate transpose of the synthesis operator (as is the case for the Fourier transform and its inverse). Sharing these weights reduces the number of parameters and simplifies interpretation of the learned solution.
Multiscale processing Transformations such as translation act on spatial neighborhoods of various sizes. To account for this, the image is first expanded at multiple resolutions in a fixed overcomplete Laplacian pyramid [10]; then learned spatial filtering (see previous paragraph) and temporal processing (see section 3.3) are applied on these coefficients; and finally, the modified coefficients are recombined across scales to generate the predicted next frame (see details in the caption of Figure 3).
Figure 3: Laplacian pyramid. An image is recursively split into low frequency approximation and high frequency details. Given the initial image  $\mathbf{x} = \mathbf{x}_{j=0} \in \mathbb{R}^N$ , the low frequency approximation (aka. Gaussian pyramid coefficients) is computed via blurring (convolution with a fixed filter  $B$ ) and downsampling ("stride" of 2, denoted  $2_{\downarrow}$ ):  $\mathbf{x}_j = 2_{\downarrow}(B \star \mathbf{x}_{j-1} \in \mathbb{R}^{2^{-j}N})$ , for levels  $j \in [1, J]$ ; and the high frequency details (aka. Laplacian pyramid coefficients) are computed via upsampling (put one zero between each sample,  $2^{\uparrow}$ ) and blurring:  $\Delta \mathbf{x}_j = \mathbf{x}_j - B \star (2^{\uparrow} \mathbf{x}_{j+1})$ . These coefficients,  $\{\Delta \mathbf{x}_j\}_{0 \leq j < J}$ , as well as the lowpass,  $x_J$ , can then be further processed. A new image is constructed recursively on these processed coefficients. First by upsampling the lowest resolution, and then by adding the corresponding details until the initial scale  $j = 0$  as:  $\mathbf{x}_j = B \star (2^{\uparrow} \mathbf{x}_{j+1}) + \Delta \mathbf{x}_j$ .
3.3 Prediction mechanism
Polar predictor In order to obtain phases, we group coefficients in pairs, express them as complex-valued:  $z_{k,t} = y_{2k,t} + iy_{2k+1,t} \in \mathbb{C}$ , and convert to polar coordinates:  $z_{k,t} = a_{k,t}e^{i\theta_{k,t}}$ . With this notation, linear phase extrapolation amounts to  $\hat{z}_{k,t+1} = a_{k,t}e^{i(\theta_{k,t} + \Delta \theta_{k,t})}$ , where the phase advance  $\Delta \theta_{k,t}$  is equal to the phase difference over the interval from  $t-1$  to  $t$ :  $\Delta \theta_{k,t} = \theta_{k,t} - \theta_{k,t-1}$ . Note that we assume no phase acceleration and constant amplitudes. The phase-advanced coefficients can be expressed in a more direct way, using complex arithmetic, as:
$$
\hat {z} _ {k, t + 1} = \delta_ {k, t} z _ {k, t}, \text {w h e r e} \delta_ {k, t} = \frac {z _ {k , t} \bar {z} _ {k , t - 1}}{\left| z _ {k , t} \right| \left| z _ {k , t - 1} \right|}, \tag {3}
$$
with  $\overline{z}$  and  $|z|$  respectively denoting complex conjugation and complex modulus of  $z$ . This formulation in terms of products of complex coefficients has the benefit of handling phases implicitly, which makes phase processing computationally feasible, as previously noted in the texture modeling literature [11, 12]. Optimization over circular variables suffers from a discontinuity if one represents the variable over a finite interval (e.g.  $[- \pi, \pi]$ ). Alternatively, procedures for "unwrapping" the phase are generally unstable and sensitive to noise.
In summary, given a video dataset  $X = [\mathbf{x}_1,\dots ,\mathbf{x}_T]\in \mathbb{R}^{N\times T}$ , the convolutional filters of a polar prediction model are learned by minimizing the average squared prediction error:
$$
\min  _ {\mathbf {W} \in \mathbb {C} ^ {N \times N K}} \sum_ {t = 1} ^ {T} \left| \left| \mathbf {x} _ {t + 1} - \mathbf {W} \operatorname {d i a g} \left(\delta_ {t}\right) \mathbf {W} ^ {*} \mathbf {x} _ {t} \right| \right| ^ {2};
$$
$$
\text {w h e r e} \boldsymbol {\delta} _ {t} = \left(\mathbf {z} _ {t} \odot \overline {{\mathbf {z} _ {t - 1}}}\right) \odot \left(\left| \mathbf {z} _ {t} \right| \odot \left| \mathbf {z} _ {t - 1} \right|\right), \text {a n d} \mathbf {z} _ {t} = \mathbf {W} ^ {*} \mathbf {x} _ {t}. \tag {4}
$$
The columns of the convolutional matrix  $\mathbf{W}$  contain the  $K$  complex-valued filters,  $\mathbf{w}_k = \mathbf{v}_{2k} + i\mathbf{v}_{2k+1} \in \mathbb{C}^N$  (repeated at  $N$  locations) and multiplication, division, amplitude, and complex conjugation are computed pointwise.
This "polar predictor" (hereafter, PP) is depicted in figure 2. Note that the conversion to polar coordinates is the only non-linear step used in the architecture. This bivariate non-linear activation function differs markedly from the typical (pointwise) rectification operations found in convolutional neural networks. Note that the polar predictor is homogeneous of degree one, since it is computed as the ratio of a cubic over a quadratic.
2Using "Gauss's trick", each complex multiplication can be computed with only three real multiplications:
$(a + ib)(c + id) = ac - bd + i((a + b)(c + d) - ac - bd)$ .
Quadratic predictor Rather than building in the phase extrapolation mechanism, we now consider a more expressive parametrization of the predictor  $(p_w)$  that can be learned on data jointly with the analysis and synthesis mappings. This "quadratic predictor" (hereafter, QP) generalizes the polar extrapolation mechanism and can accommodate groups of channels of size larger than two (as for the real and imaginary part in the polar predictor).
Figure 4: Learnable quadratic prediction mechanism. Groups of coefficients  $(\mathbf{y}_{k,t})$  at the previous and current time-step are normalized  $(\mathbf{u}_{k,t})$  and then passed through in a Linear-Square-Linear cascade to produce a prediction matrix  $(\mathbf{M}_{k,t})$ . This matrix is applied to the current vector of coefficients to predict the next one. The linear transforms  $(\mathbf{L}_1$  and  $\mathbf{L}_2)$  are learned. This quadratic prediction module contains phase extrapolation as a special case and handles the more general case of groups of coefficients beyond pairs.
First, we rewrite the phase extrapolation mechanism of equation 3 using only real-valued elements:
$$
\left[ \begin{array}{l} \hat {y} _ {2 k, t + 1} \\ \hat {y} _ {2 k + 1, t + 1} \end{array} \right] = \left[ \begin{array}{l l} \cos \Delta \theta_ {k, t} & - \sin \Delta \theta_ {k, t} \\ \sin \Delta \theta_ {k, t} & \cos \Delta \theta_ {k, t} \end{array} \right] \left[ \begin{array}{l} y _ {2 k, t} \\ y _ {2 k + 1, t} \end{array} \right], \tag {5}
$$
where the  $2 \times 2$  prediction matrix,  $\mathbf{M}_{k,t}$ , is a rotation by angle  $\Delta \theta_{k,t}$ . Using an elementary trigonometric identity:  $\cos(a - b) = \cos(a)\cos(b) + \sin(a)\sin(b)$  and recalling that  $y_{k,t} = \mathbf{v}_{2k}^{\top}\mathbf{x}_t = a_{k,t}\cos(\theta_{k,t})$ , the elements of this prediction matrix can be made explicit. They are a quadratic function of the normalized response, which we write  $u_{2k,t}$  (for unit vector), as:
$$
\cos \Delta \theta_ {k, t} = u _ {2 k, t} u _ {2 k, t - 1} + u _ {2 k + 1, t} u _ {2 k + 1, t - 1}, \text {w h e r e} u _ {2 k, t} = \frac {y _ {2 k , t}}{\left(y _ {2 k , t} ^ {2} + y _ {2 k + 1 , t} ^ {2}\right) ^ {1 / 2}}. \tag {6}
$$
This quadratic function can be expressed in terms of squared quantities (without cross terms) using the polarisation identity:  $\alpha \beta = ((\alpha + \beta)^2 - (\alpha - \beta)^2)/4$ . Specifically:
$$
\cos \Delta \theta_ {k, t} = \frac {1}{4} \left(\left(u _ {2 k, t} + u _ {2 k, t - 1}\right) ^ {2} - \left(u _ {2 k, t} - u _ {2 k, t - 1}\right) ^ {2} + \right.
$$
$$
\left. \left(u _ {2 k + 1, t} + u _ {2 k + 1, t - 1}\right) ^ {2} - \left(u _ {2 k + 1, t} - u _ {2 k + 1, t - 1}\right) ^ {2}\right). \tag {7}
$$
An analogous expression can be derived for  $\sin \Delta \theta_{k,t}$ . These operations are depicted in Figure 4: current and previous pairs of coefficients  $(\mathbf{y}_{k,t} = [y_{2k,t}, y_{2k+1,t}, y_{2k,t-1}, y_{2k+1,t-1}]^\top \in \mathbb{R}^4)$  are normalized  $(\mathbf{u}_{k,t} \in \mathbb{R}^4)$ , then linearly combined  $(\mathbf{L}_1 \in \mathbb{R}^{4 \times d})$ , pointwise squared, and linearly combined again  $(\mathbf{L}_2 \in \mathbb{R}^{4 \times d})$  to produce a prediction matrix  $(\mathbf{M}_{k,t} \in \mathbb{R}^{2 \times 2})$  that is applied to the current coefficients to produce a prediction  $([ \hat{y}_{2k,t+1}, \hat{y}_{2k+1,t+1}]^\top \in \mathbb{R}^2)$ . These linear combinations can be learned jointly with the analysis and synthesis weights by minimizing the prediction error.
In summary, the quadratic predictor is learned as:
$$
\min  _ {\mathbf {V}, \mathbf {L} _ {1}, \mathbf {L} _ {2}} \sum_ {t = 1} ^ {T} | | \mathbf {x} _ {t + 1} - \mathbf {V} \boldsymbol {\Lambda} _ {t} \mathbf {V} ^ {\top} \mathbf {x} _ {t} | | ^ {2}; \quad \text {w h e r e} \boldsymbol {\Lambda} _ {t} = \operatorname {b l o c k d i a g} (\mathbf {M} _ {1, t}, \dots , \mathbf {M} _ {K, t}),
$$
$$
\mathbf {M} _ {k, t} = \mathbf {L} _ {2} \left(\mathbf {L} _ {1} ^ {\top} \mathbf {u} _ {k, t}\right) ^ {\odot 2}, \quad \mathbf {u} _ {k, t} = \frac {\mathbf {y} _ {k , t}}{\left(\mathbf {1} _ {g} ^ {\top} \mathbf {y} _ {k , t} ^ {2}\right) ^ {1 / 2}}, \quad \text {a n d} \quad \mathbf {y} _ {t} = \mathbf {V} ^ {\top} \mathbf {x} _ {t}. \tag {8}
$$