Title: A polar prediction model for learning to represent visual transformations
Venue: NeurIPS 2023
Accepted: yes
Keywords: video prediction

neural coding

symmetry discovery

self-supervised representation-learning
Abstract: All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.

A polar prediction model for learning to represent visual transformations

Abstract

All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.

1 Introduction

The fundamental problem of vision can be framed as that of representing images in a form that is more useful for performing visual tasks, be they estimation, recognition, or motor action. Perhaps the most general 'task' is that of temporal prediction, which has been proposed as a fundamental goal for unsupervised learning of visual representations [1]. Previous research along these lines has generally focused on estimating stable representations rather than using them to predict: for example, extracting slow features [2], or finding sparse codes that have slow amplitudes and phases [3].

In video processing and computer vision, a common strategy for temporal prediction is to first estimate local translational motion, and then (assuming no acceleration) use this to warp and/or copy previous content to predict the next frame. Such motion compensation is a fundamental component in video compression schemes as MPEG [4]. These video coding standards are the result of decades of engineering effort [5], and have enabled reliable and efficient digital video communication that is now commonplace. But motion estimation is a difficult nonlinear problem, and existing methods fail in regions where temporal evolution is not translational and smooth: for example, expanding or rotating motions, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi-transparent surfaces (e.g., viewing the world through a dirty pane of glass). In compression schemes, these failures of motion estimation lead to prediction errors, which must then be adjusted by sending additional corrective bits.

Human perception does not seem to suffer from such failures-subjectively, we can anticipate the time-evolution of visual input even in the vicinity of these commonly occurring non-translational changes. In fact, those changes are often highly informative, revealing object boundaries, and providing ordinal depth and shape cues and other information about the visual scene. This suggests that the human visual system uses a different strategy, perhaps bypassing altogether the explicit estimation of local motion, to represent and predict evolving visual input. Toward this end, and inspired by the recent hypothesis stating that primate visual representations support prediction by 'straightening' the temporal trajectories of naturally-occurring input [6], we formulate an objective for learning an image representation that facilitates prediction by linearizing the temporal trajectories of frames in natural videos.

To motivate the separation of spatial representation and temporal prediction, we first consider the special case of rigidly translating signals in one dimension. In the frequency domain, translation corresponds to phase advance (section 2.1), which reduces prediction to phase extrapolation (section 2.2). We invoke basic arguments from group theory to motivate the search for generalized representations (section 2.3). We propose a neural network architecture that maps individual video frames to a latent space where prediction can be computed more readily and then mapped back to generate an estimated frame (section 3). We train the entire system end-to-end to minimize next frame prediction errors and verify that, in controlled experiments, the learned weights recover the representation of the group used to generate synthetic data. On natural video datasets, our framework consistently outperforms conventional motion compensation methods and is competitive with deep predictive neural networks (section 4). We establish connections between each element of our framework and the modeling of early visual processing (section 4.3).

2 Background

2.1 Base case: the Fourier shift theorem

Our approach is motivated by the well-known behavior of Fourier representations with respect to signal translation. Specifically, the complex exponentials that constitute the Fourier basis are the eigenfunctions of the translation operator, and translation of inputs produces systematic phase advances of frequency coefficients. Let x = [ x 0 , . . . , x N -1 ] ⊤ ∈ R N be a discrete signal indexed by spatial location n ∈ [0 , N -1] , and let ˜ x ∈ C N be its Fourier transform indexed by k ∈ [0 , N -1] . We write x ↓ v , the circular shift of x by v , x ↓ v n = x n -v (with translation modulo N ). Let ϕ denote the primitive N th root of unity, ϕ = e i 2 π/N , and let F ∈ C N × N denote the Fourier matrix, F nk = 1 √ N ϕ nk . Multiplication by the adjoint (i.e. the conjugate transpose), F ∗ , gives the Discrete Fourier Transform (DFT), and by F the inverse DFT. We can express the Fourier shift theorem 1 as: x ↓ v = F diag ( ϕ v ) F ∗ x , where ϕ v = [ ϕ 0 , ϕ -v , ϕ -2 v , . . . , ϕ -( n -1) v ] ⊤ is the vector of phase shift by v. See Appx. A for a consolidated list of notation used throughout this work.

This relationship can be depicted in a compact diagram:

This diagram illustrates how transforming to the frequency domain renders translation a 'simpler' operation: phase shift acts as rotation in each frequency subspace, i.e. it is diagonalized.

-

-

<latexi sh

1_b

64="+SnU

I

g9H

Z

QLY

B

Eu

>A

CO

c

V

J

8N

G

3

r

v

KX

T

f

0

y

o

P

/

F

D

7

M

RqW

j

5

2

w

d

p

0

0

2

-

0

4

2

0

-

2

-

2

10

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

space

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

6

8

-

4

6

8

5

t

+1

t

t

-1

(a) translating signal

space

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

space

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

(b) projected trajectory

PC1

PC2

PCA

0

2

4

6

8

-

4

-

2

0

2

4

-

4

-

2

0

2

4

2

4

6

8

5

10

15

20

4

.

8

4

.

9

5

.

0

5

.

1

5

.

2

(c) frequency coefficients

frequency

real

imag

F

∗

0

2

4

6

8

-

4

-

2

0

2

4

-

4

-

2

0

2

4

2

4

6

8

5

10

15

20

4

.

8

4

.

9

5

.

0

5

.

1

5

.

2

(d) amplitude / phase

rect2pol

1

0

1

0

.

.

0

5

.

.

0

5

2.2 Prediction via phase extrapolation

0

0

.

0

5

.

2

.

0

0

5

.

-

0

2

.

5

5

.

2

.

0

0

5

.

-

0

2

5

.

.

5

0

.

0

0 . 00 0 . 25 0 . 50 0 . 75 1 . 00 space 1 . 0 0 . 5 -5 . 0 -2 . 5 0 . 0 2 . 5 5 . 0 -5 . 0 -2 . 5 0 . 0 2 . 5 5 . 0 -5 . 0 0 . 00 0 . 25 0 . 50 0 . 75 1 . 00 space -1 . 0 -0 . 5 -5 . 0 -2 . 5 0 . 0 2 . 5 5 . 0 -5 . 0 -2 . 5 0 . 0 2 . 5 5 . 0 -Consider a signal, the N -dimensional vector x t , that translates at a constant velocity v over time: x n,t = x n -vt, 0 . This sequence traces a highly non-linear trajectory in signal space, i.e. the vector space where each dimension corresponds to the signal value at one location. In this space, linear extrapolation fails. As an example, Figure 1 shows a signal consisting of a sum of two sinusoidal components in one spatial dimension. Mapping the signal to the frequency domain simplifies the description. In particular, the translational motion now corresponds to circular motion of the two (complex-valued) Fourier coefficients associated with the constituent sinusoids. In polar coordinates, the trajectory of these coefficients is straight, with both phases advancing linearly (at a rate proportional to their frequency), and both amplitudes constant.

2.3 Generalization: representing transformation groups

Streaming visual signals are replete with structured transformations, such as object displacements and surface deformations. While these can not be captured by the Fourier representation, which only handles global translation, the concept of representing transformations in their eigen-basis generalizes. Indeed, representation theory describes elements of general groups as linear transformations in vector spaces, and decomposes them into basic building blocks [7]. However, the transformation groups acting in image sequences are not known a priori, and it can be difficult to give an explicit representation of general group actions. In this work, we aim to find structures that can be modeled as groups in image sequences, and we learn their corresponding representations from unlabeled data.

In harmonic analysis, the Peter-Weyl Theorem (1927) establishes the completeness of the unitary irreducible representations for compact topological groups (an irreducible representation is a subspace that is invariant to group action and that can not be further decomposed). Furthermore, every compact Lie group admits a faithful (i.e. injective) representation given by an explicit complete orthogonal basis, constructed from finite-dimensional irreducible representations [7]. Accordingly, the action of a compact Lie group can be expressed as a rotation within each irreducible representation (an example is the construction of steerable filters [8] in the computational vision literature).

In the special case of compact commutative Lie groups, the irreducible representations are onedimensional and complex-valued (alternatively, pairs of real valued basis functions). These groups have a toroidal topology and, in this representation, their action can be described as advances of the phases. This suggests a strategy for learning a representation: seek pairs of basis functions for which phase extrapolation yields accurate prediction of upcoming images in a stream of visual signals.

4

2

4

4

6

8

-

2

-

4

2

4

2

0

-

2

-

4

0

4

2

2

4

4

2

4

5

.

5

2

.

5

.

4

.

4

6

.

20

15

1

0

9

8

8

5

10

15

5

.

5

.

5

.

4

.

.

4

20

2

1

0

9

8

PC3

Figure 1: Straightening translations.

1 (a) Three snapshots of a translating signal consisting of two superimposed sinusoidal components: x n,t = sin(2 π ( n -t )) + sin(2 π 3( n -t )) / 2 . (b) Projection of the signal into the space of the top three principal components. The colored points correspond to the three snapshots in panel (a). In signal space, the temporal trajectory is highly curved-linear extrapolation fails. (c) Complex-valued Fourier coefficients of the signal as a function of frequency. The temporal trajectory of the frequency representation is the phase advance of each sinusoidal component. (d) Trajectory of one amplitude and both (unwrapped) phases components. The conversion from rectangular to polar coordinates reduces the trajectory to a straight line-which is predictable via linear extrapolation.

3 Methods

3.1 Objective function

We aim to learn a representation of video frames that enables next frame prediction. Specifically, we optimize a cascade of three parameterized mappings: an analysis transform ( f w ) that maps each frame to a latent representation, a prediction in the latent space ( p w ), and a synthesis transform ( g w ) that maps the predicted latent values back to the image domain. The parameters w of these mapping are learned by minimizing the average squared prediction error:

An instantiation of this framework is illustrated in Figure 2. Here the analysis and synthesis transforms are adjoint linear operators, and the predictor is a diagonal phase extrapolation.

x

t

-

1

x

t

̂

x

t

+1

x

t

+1

MSE

. . .

. . .

δ

t

W

∗

W

3.2 Analysis-synthesis transforms

Local processing When focusing on a small spatial region in an image sequence, the transformation observed as time passes can often be well approximated as a local translation. That is to say, in a spatial neighborhood around position n , m ∈ N ( n ) , we have: x m,t +1 ≈ x m -v,t . We can use the decomposition described for global rigid translation, replacing the Fourier transform with a learned local convolutional operator [9], processing each spatial neighborhood of the image independently and in parallel.

At every position in the image (spatial indices are omitted for clarity of notation), each pair of coefficients is computed as an inner product between the input and the filter weights of each pair of channels. Specifically for k ∈ [0 , K ] , where K is the number of pairs of channels, we have: y 2 k,t = v ⊤ 2 k x t and y 2 k +1 ,t = v ⊤ 2 k +1 x t . Correspondingly, an estimated next frame is generated by applying the transposed convolution g w to advanced coefficients (see section 3.3). We use the same weights for the encoding and decoding stages, that is to say the analysis operator is the conjugate transpose of the synthesis operator (as is the case for for the Fourier transform and its inverse). Sharing these weights reduces the number of parameters and simplifies interpretation of the learned solution.

<latexi sh

1_b

64="

v

2WT8uFR

zgS9

3c

>A

CfH

V

Mw

q

d

m

+

O

UE

75

r

n

LZ

J

DX

p

P

j

Y

0

K

Q

N

k

<latexi sh

1_b

64="F

Dj

N

yE2

V

mk

U

c

8

PY

>A

C

3

BfS+

wHM

qXr

QJ

G9

np

O

K

o

z

0

/

7

R

u

W

Z

d

g

I

5

L

T

v

<latexi sh

1_b

64="NwCkmH

5fq+QYKP

F

IjUc

V

8

>A

On

BJS

G

2

r

D

u

R

E

X

p

d

WT

z

o

/

L

79v

3

0

M

g

y

Z

<latexi sh

1_b

64="W

q9Eyz

FS

Kj

N

UHI

>A

B7

VDL

2p

f

Zd

g

T

0

C

Y

M

8G

X

+

/

n

R

m

w

P

3

J

5

O

Q

<latexi sh

1_b

64="ZXFv n0m

q

OUzfLcRWV

78

D

I

>A

C

3

SgM

d

Q

u

y

r

K

o

2

J

H

k

G5

+

P

B

N

p

/

E

j

T

9

Y

w

<latexi sh

1_b

64="y

0

3L

w

cjp

u

g

7K/8

>A

COH

VD

S

M

F

U

frW

EN

m

XB

G

2

k

oZ

J

v

IP

Q

59

+

Y

d

n

q

R

T

z

<latexi sh

1_b

64="W

q9Eyz

FS

Kj

N

UHI

>A

B7

VDL

2p

f

Zd

g

T

0

C

Y

M

8G

X

+

/

n

R

m

w

P

3

J

5

O

Q

<latexi sh

1_b

64="v yOGuNop

mgd

HM

J3

0

Yq

>A

C

c

VB

9

wF

W8

f

T

S2E

Lk

Pn

Q

U5

Z

I

r

/K

z

R

7

+

X

D

<latexi sh

1_b

64="

AjX/0cK

2

y

u

G8L

D

v

q

>

CQH

VBJS

N

3Ur

Y

gZ

Ek

9

F

m

7d

I

M5

f

n

W

+

w

o

R

p

T

O

P

z

Multiscale processing Transformations such as translation act on spatial neighborhoods of various sizes. To account for this, the image is first expanded at multiple resolutions in a fixed overcomplete Laplacian pyramid [10]; then learned spatial filtering (see previous paragraph) and temporal processing (see section 3.3) are applied on these coefficients; and finally, the modified coefficients are recombined across scales to generate the predicted next frame (see details in the caption of Figure 3). Laplacian Pyramid

+

-

∆

x

j

x

j

x

j

+1

2

↓

2

↑

2

↑

3.3 Prediction mechanism

Polar predictor In order to obtain phases, we group coefficients in pairs, express them as complexvalued: z k,t = y 2 k,t + iy 2 k +1 ,t ∈ C , and convert to polar coordinates: z k,t = a k,t e iθ k,t . With this notation, linear phase extrapolation amounts to ˆ z k,t +1 = a k,t e i ( θ k,t +∆ θ k,t ) , where the phase advance ∆ θ k,t is equal to the phase difference over the interval from t -1 to t : ∆ θ k,t = θ k,t -θ k,t -1 . Note that we assume no phase acceleration and constant amplitudes. The phase-advanced coefficients can be expressed in a more direct way, using complex arithmetic, as:

with z and | z | respectively denoting complex conjugation and complex modulus of z . This formulation in terms of products of complex coefficients 2 has the benefit of handling phases implicitly, which makes phase processing computationally feasible, as previously noted in the texture modeling literature [11, 12]. Optimization over circular variables suffers from a discontinuity if one represents the variable over a finite interval (e.g. [ -π, π ] ). Alternatively, procedures for 'unwrapping' the phase are generally unstable and sensitive to noise.

In summary, given a video dataset X = [ x 1 , . . . , x T ] ∈ R N × T , the convolutional filters of a polar prediction model are learned by minimizing the average squared prediction error:

The columns of the convolutional matrix W contain the K complex-valued filters, w k = v 2 k + i v 2 k +1 ∈ C N (repeated at N locations) and multiplication, division, amplitude, and complex conjugation are computed pointwise.

This 'polar predictor' (hereafter, PP ) is depicted in figure 2. Note that the conversion to polar coordinates is the only non-linear step used in the architecture. This bivariate non-linear activation function differs markedly from the typical (pointwise) rectification operations found in convolutional neural networks. Note that the polar predictor is homogeneous of degree one, since it is computed as the ratio of a cubic over a quadratic.

Figure 3: Laplacian pyramid. An image is recursively split into low frequency approximation and high frequency details. Given the initial image x = x j =0 ∈ R N , the low frequency approximation (aka. Gaussian pyramid coefficients) is computed via blurring (convolution with a fixed filter B ) and downsampling ('stride' of 2, denoted 2 ↓ ): x j = 2 ↓ ( B ⋆ x j -1 ∈ R 2 -j N ), for levels j ∈ [1 , J ] ; and the high frequency details (aka. Laplacian pyramid coefficients) are computed via upsampling (put one zero between each sample, 2 ↑ ) and blurring: ∆ x j = x j -B ⋆ (2 ↑ x j +1 ) . These coefficients, { ∆ x j } 0 ≤ j<J , as well as the lowpass, x J , can then be further processed. A new image is constructed recursively on these processed coefficients. First by upsampling the lowest resolution, and then by adding the corresponding details until the initial scale j = 0 as: x j = B ⋆ (2 ↑ x j +1 ) + ∆ x j .

<latexi sh

1_b

64="FGjE

p

9

5

H

+P7VJdo

Bg

>A

X

c

DLS

N

3

W

r

Y

T

8uC

U

2M

kz

I

R

w

n

Z

f

K

/

v

Qm

q

0

O

y

<latexi sh

1_b

64="DIRq0Hpn

O+

g

f

7

U

>A

83

c

LS

F

TX

W

d

v

w

Y

/Q

Z

r

kG

u

C

9

5

P

o

m

y

z

K

N

J

E

2

<latexi sh

1_b

64="mpcTFZvwRzCg

QG

yM

B7+S

>A

8X

V

N

I

EJ3

L

KO

UY

k

j

W

o

u2q

/

D

r

9

0

f

dn

5

P

H

<latexi sh

1_b

64="mpcTFZvwRzCg

QG

yM

B7+S

>A

8X

V

N

I

EJ3

L

KO

UY

k

j

W

o

u2q

/

D

r

9

0

f

dn

5

P

H

<latexi sh

1_b

64="U

JGC

9o3

8

QSgE

vn

M

>A

B

H

c

VDL

N

Oy

r

f

Y

P

I

TZ

W

2m

k

X

F

/

R

ju

5

p

d

7

q

w

0

+

K

z

<latexi sh

1_b

64="2wHX

EmMG

dFPz

DS

O

k

R

>A

B

c

V

L

gN

y

r

fUY9

o

C

Q

8

I

Z

5

j

J

v

+

/

0

qu

n

3

7

p

W

K

T

<latexi sh

1_b

64="ERA/r2

j

Su7NWd

0

L

+Z

YUw

>

B

H

c

VD

FJ3

v

o

gq

T

8

C

9

Qp

M

y

O

Gz

X

5

P

f

K

k

I

m

n

<latexi sh

1_b

64="

nUP

w

jF

BG9K7

IpRmZ

>A

83

c

VDLSgM

X

W

Y

0

do

T

u

k

5

v+H

J

/

2

O

E

y

qQ

r

z

N

f

C

Quadratic predictor Rather than building in the phase extrapolation mechanism, we now consider a more expressive parametrization of the predictor ( p w ) that can be learned on data jointly with the analysis and synthesis mappings. This 'quadratic predictor' (hereafter, QP ) generalizes the polar extrapolation mechanism and can accommodate groups of channels of size larger than two (as for the real and imaginary part in the polar predictor). Quadratic prediction mechanism

×

(

.

)

2

L

1

L

2

t

1

t

+1

÷

÷

(

.

)

2

√

.

Σ

u

g

y

g

M

g

t

Figure 4: Learnable quadratic prediction mechanism. Groups of coefficients ( y k,t ) at the previous and current time-step are normalized ( u k,t ) and then passed through in a Linear-Square-Linear cascade to produce a prediction matrix ( M k,t ). This matrix is applied to the current vector of coefficients to predict the next one. The linear transforms ( L 1 and L 2 ) are learned. This quadratic prediction module contains phase extrapolation as a special case and handles the more general case of groups of coefficients beyond pairs.

-

First, we rewrite the phase extrapolation mechanism of equation 3 using only real-valued elements:

where the 2 × 2 prediction matrix, M k,t , is a rotation by angle ∆ θ k,t . Using an elementary trigonometric identity: cos( a -b ) = cos( a ) cos( b ) + sin( a ) sin( b ) and recalling that y k,t = v ⊤ 2 k x t = a k,t cos( θ k,t ) , the elements of this prediction matrix can be made explicit. They are a quadratic function of the normalized response, which we write u 2 k,t (for unit vector), as:

This quadratic function can be expressed in terms of squared quantities (without cross terms) using the polarisation identity: αβ = (( α + β ) 2 -( α -β ) 2 ) / 4 . Specifically:

An analogous expression can be derived for sin ∆ θ k,t . These operations are depicted in Figure 4: current and previous pairs of coefficients ( y k,t = [ y 2 k,t , y 2 k +1 ,t , y 2 k,t -1 , y 2 k +1 ,t -1 ] ⊤ ∈ R 4 ) are normalized ( u k,t ∈ R 4 ), then linearly combined ( L 1 ∈ R 4 × d ), pointwise squared, and linearly combined again ( L 2 ∈ R 4 × d ) to produce a prediction matrix ( M k,t ∈ R 2 × 2 ) that is applied to the current coefficients to produce a prediction ( [ˆ y 2 k,t +1 , ˆ y 2 k +1 ,t +1 ] ⊤ ∈ R 2 ). These linear combination can be learned jointly with the analysis and synthesis weights by minimizing the prediction error.

In summary, the quadratic predictor is learned as:

The columns of the convolutional matrix V ∈ R N × gNK contain the groups of g filters (repeated at N locations). The number of channels in a group is no longer limited to pairs ( | g | ≥ 2 is a hyper-parameter). The prediction matrices, M k,t ∈ R g × g , are computed as a Linear-SquareLinear cascade on normalized activity from group of coefficients at the previous two time points: u k,t = [ u k,t , u k +1 ,t , . . . , u k + g -1 ,t ] ⊤ ∈ R g and u k,t -1 . The linear combinations are learnable matrices L 1 ∈ R 2 g × d and L 2 ∈ R g 2 × d , where d is the number of quadratic units and squaring is computed pointwise. Note that in the case of pairs ( g = 2 ), six quadratic units suffice ( d = 6 ).

<latexi sh

1_b

64="WvEwquP

A+N0KB

gJ5

D

of

Q

>

83

c

V

LS

F

2pr

Zd

R

CG

7

O

U

T

M

nz

Y

k

mX

9

y

I

H

/

j

<latexi sh

1_b

64="pVj+f07

M

85I

wYOy

WE

>A

B

3

c

DLSg

F

TX

q

d

v

/o

C

Z

rQ

kG

u

9

zn

K

m

P

R

H

J

N

U

<latexi sh

1_b

64="

HcQJI+

8R7LgBjk

FGmwr0Y

>A

n

VD

S

N

EOz

M

C

5

d

Z

X

P

U

o

f

v

qK

u

y

W

9

3

p

T

/

2

<latexi sh

1_b

64="8yfnr59D

A

PoX

HI

pU

FZc0

>

B

3

V

LS

N

2

du vgq

T

CG

7

Jm

+

j

Q

M

z

w

R

W

Y

E

/

K

O

k

<latexi sh

1_b

64="AU

Cg prJqz9

ZK

wjME

>

B7H

c

V

NT8

3

F+IX

LRm

Y

v

d

u2

y

k

D

5OW

P

0

n

Q

o

G

/

f

S

<latexi sh

1_b

64="N

73n+

dS9V

TG

HzUBEmA

>

X

c

DJ

g

K

j

L

v

RPY

Z

8

Mo

f

k

p

u

Q

/

O

F

w

q

r

y

W

2

5

0

I

C

<latexi sh

1_b

64="

NHF

p

W

mQI

y

oZn

>A

B7

c

V

S8

EJ

q/

9L

0

j

X

2

O

K

w

R

d

v

g

P

D

z

T

f

u

Y

5+

G

M

<latexi sh

1_b

64="Q

yH

zuZ

58v

g

n+dXw

k

R

>A

B

c

VDLS

N

EO

r

fUY9

oP

K

j

G

C

Jm

W

7

M

T

3

F

/

I

0

p

q

2

<latexi sh

1_b

64="8yfnr59D

A

PoX

HI

pU

FZc0

>

B

3

V

LS

N

2

du vgq

T

CG

7

Jm

+

j

Q

M

z

w

R

W

Y

E

/

K

O

k

<latexi sh

1_b

64="RuUZ

g

E9fSCk7w

T

X

>A

B

H

c

VDL

N

Oz

G

Mr

Y

8

3

W

J

K

Q

ny

F

v

I

d

0

/

j

2

q

m

5

o

<latexi sh

1_b

64="AU

Cg prJqz9

ZK

wjME

>

B7H

c

V

NT8

3

F+IX

LRm

Y

v

d

u2

y

k

D

5OW

P

0

n

Q

o

G

/

f

S

<latexi sh

1_b

64="

rCB/qKwLjzQDn

RO

c

y

>A

V

SgN

E

fUY9

P

GH

v

8IF

7

3

T

u

M

+

m

J

W

Z

d

p

5

2

<latexi sh

1_b

64="2

ZBU

MA

/Wz vF

rmKqQPLHN

>

n

c

VD

EO

G

Y

I

y

j

w

X

9

7

u

C

J

o

8

f

k

d

3

0

p

T

5

+

R

<latexi sh

1_b

64="0

7

Iyw p5Mj

n

Tk

F

C

o

>A

B

X

c

VDLSgN

EO

r

fUY9

P

K

G

m

3

z

W

/8

2

dQ

u

v

+

Z

q

J

R

H

<latexi sh

1_b

64="vdyYO5

M2

N0

8IF

7

A

>

B

3

c

VDLSg

TX

W+q

j

w

kPJp

U

oQ

/

z

H

E

Z

R

m

K

9

C

G

4 Results

4.1 Recovery of planted symmetries

To experimentally validate our approach, we first verified that both the PP and the QP models can robustly recover known symmetries in small synthetic datasets consisting of translating or rotating image patches. For these experiments, the analysis and synthesis transforms are applied to the entire patch (i.e., no convolution). Learned filters for each of these cases are displayed in Figure 7, Appendix B. When trained on translating image patches, the learned filters are pairs of plane waves, shifted in phase by π/ 2 . Similarly, when trained on rotating patches, the learned filters are conjugate pairs of circular harmonics. This demonstrates that theses models can recover the (regular) representation of some simple groups from observations of signals where their transformations are acting. When the transformation is not perfectly translational (e.g. translation with open boundary condition), the learned filters are localized Fourier modes. Note that when multiple kinds of transformations are acting in data (e.g., mixtures of both translations and rotations), the PP model is forced to compromise on a single representations. Indeed, the phase extrapolation mechanism is adaptive but the basis in which it is computed is fixed and optimized. A more expressive model would also allow for adaptation of the basis itself.

4.2 Prediction performance on natural videos

We compare our multiscale polar predictor ( mPP ) and quadratic predictor ( mQP ) methods to a causal implementation of the traditional motion-compensated coding ( cMC ) approach. For each block in a frame, the coder searches for the most similar spatially displaced block in the previous frame, and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We also compare to phase-extrapolation within a steerable pyramid [13], an overcomplete multi-scale decomposition into oriented channels ( SPyr ). We also implemented a deep convolutional neural network predictor ( CNN ), that maps two successive observed frames to an estimate of the next frame [14]. Specifically, we use a CNN composed of 20 non-linear stages, each consisting of 64 channels, and computed with 3 × 3 filters without additive constants, followed by half-wave rectification. Finally, we also consider a U-net [15] which is a CNN that processes images at multiple resolutions ( Unet ). The number of non-linear stages, the number of channels and the filter size match that of the basic CNN. See descriptions in Appendix C for architectures and Appendix D for dataset and training procedures.

Method

Prediction results on the DAVIS dataset [16] are summarized in Table 1. First, observe that the predictive algorithms considered in this study perform significantly better than baselines obtained by simply copying the last frame, causal motion compensation, or phase extrapolation in a steerable pyramid. Second, the multiscale polar predictor performs better than the convolutional neural networks on test data (both CNN and Unet are overfit). This demonstrates the efficiency of the polar predictor: the PP model has roughly 30 times fewer parameters than the CNN and uses a single non-linearity, while the CNN and Unet contain 20 non-linear layers. Appendix E contains additional a comparison of computational costs for these algorithms: number of trainable parameters, training and inference time. Note that on this dataset, the Unet seems to overfit to the training set. Moreover, the added expressivity afforded by the multiscale quadratic predictor did not result in significant performance gains. A representative example image sequence and the corresponding predictions are displayed in Figure 5. Additional results on a second natural video dataset are detailed in Appendix E and confirm these trends.

(

Example predictions

cMC

25.70

x

t

x

t

x

t

PSNR

21.67

SPyr

26.49

mPP

26.98

mQP

27.28

CNN

26.44

-1)

(

)

(

+1)

13

4.3 Biological modeling

The polar prediction model provides a hypothesis for how cortical circuits in the primate visual system might compute predictions of their inputs ('predictive processing') [17-20]. This framework is agnostic to how predictions are used and is complementary to candidate algorithms for signaling predictions and prediction errors across the visual hierarchy ('predictive coding') [21-23].

First, the learned convolutional filters of a polar prediction model resemble receptive fields of neurons in area V1 (primary visual cortex). They are selective for orientation and spatial frequency, they tile the frequency domain (making efficient use of limited resources), and filters in each pair have similar frequency selectivity and are related by a shift in spatial phase. Representative filters are displayed in Figure 6.

Second, the quadratic prediction mechanism derived in section 3.3 suggests a qualitative bridge to physiology. Computations for this model are expressed in terms of canonical computational elements of early visual processing in primates. The normalized responses u 2 k ( t ) in equation 6 are linear projections of the visual input divided by the energy of related cells, similar to the normalization behavior observed in simple cells [24]. The quadratic units m g in equation 7 are sensitive to temporal change in spatial phase. This selectivity for speed in a given orientation and spatial frequency is reminiscent of direction-selective complex cells which are thought to constitute the first stage of motion estimation [25, 26].

5 Related work

The polar prediction model is conceptually related to representation learning methods that aim to factorize visual signals. In particular, sparse coding with complex-valued coefficients [3] aims to factorize form and motion. More generally, several methods adopt a Lie group formalism to factorize invariance and equivariance . Since the seminal work that proposed learning group generators from dynamic signals [27], a polar parametrization was explored in [28] to identify irreducible representations in a synthetic dataset, and a corresponding neural circuit was proposed in [29]. The Lie group formalism has also been combined with sparse coding [30, 31] to model natural images as points on a latent manifold. More recently, bispectral neural networks [32] have been shown to learn image representations invariant to a given global transformation. In a related formalism, factored Boltzmann machines have been proposed to learn relational features [33]. The polar prediction model differs in two important ways: (1) unlike the coding approach that operates on iid data, it focuses on predicting, not representing, the signal; (2) the prediction objective does not promote sparsity of either amplitude or phase components and does not rely on explicit regularization. The discontinuity arising from selection of sparse subsets of coefficients seems at odds with the representation of continuous group actions [34]. The polar prediction model relies on a smooth and continuous parameterization to jointly discover and exploit the transformations acting in sequential data. The polar prediction model is convolutional and scales to natural video data, it can adapt to the multiple unknown and noisy

(a) spatial domain filters

(b) corresponding Fourier amplitude spectra

transformations that act in different spatial position. The literature on motion microscopy describes temporal processing of local phases in a fixed complex-valued wavelet representation to interpolate between video frames and magnify imperceptible movements [35]. Polar prediction processes phase in a learned representation to extrapolate to future frames.

The polar prediction model is related to other representation learning methods that also rely on temporal prediction. Temporal stability was used to learn visual representations invariant to geometric transformations occurring in image sequences [36]. The idea of learning a temporally straightened representation from image sequences was explored using a heuristic extrapolation mechanism [37] (specialized 'soft max-pooling' and 'soft argmax-pooling' modules). A related approach aimed at finding video representations which decompose content and pose and identify 'components' in order to facilitate prediction of synthetic image sequences [38]. To tackle the challenge of natural video prediction, more sophisticated architectures have been developed for decomposing images into predictable objects [39]. A recurrent instantiation of predictive coding through time relying on a stacked convolutional LSTM architecture was proposed [40] and shown to relate to biological vision [41]. In contrast, the polar prediction model scales to prediction of natural videos while remaining interpretable. Another related approach, originating in the fluid mechanics literature, focuses on the Koopman operator [42]. This approach is a dynamical analog of the kernel trick: it lifts a system from its original state-space into a higher dimensional representation space where the dynamics can be linearized (i.e. represented by a fixed dynamics matrix). This formalism has inspired a line of work in machine learning: predictive auto-encoders learn coordinate systems that approximately linearize a system's dynamics [43]. Auxiliary networks have been introduced to adjust the dynamics matrix to velocity [44]. In contrast, the polar prediction model learns a representation that (implicitly) straightens the temporal evolution in an adaptive representation.

6 Discussion

We've introduced a polar prediction model and optimized it to represent visual transformations. Using adaptive phase extrapolation in a fixed shiftable basis, the model jointly discovers and exploits the approximate symmetries in image sequences which are due to local content deformation. The basis is optimized to best diagonalize video dynamics by minimizing mean squared prediction error. The phase relationships are exploited implicitly in a bundled computation, bypassing the instabilities and discontinuities of angular phase variables. By starting from mathematical fundamentals and considering an abstract formulation in terms of learning the representation of transformation groups, we formulated a framework that makes three major contributions. First, it provides a method for discovering the approximate symmetries implicit in sequential data and complements methods for imposing known invariants. Second, it achieves accurate next-frame video prediction within a

principled framework and provides an interpretable alternative to standard architectures. Third, it offers a framework to understand the nonlinear response properties of neurons in primate visual systems, potentially offering a functional explanation for perceptual straightening.

The polar prediction model makes several strong assumptions. First, it is inertial, assuming that amplitude is unchanged and phase evolves linearly with no acceleration. Second, it separates spatial and temporal processing, which seems at odds with the spatiotemporal selectivities of visual neurons [45], but could enable downstream image based tasks (e.g. object segmentation, heading direction estimation). Third, it acts independently on each coefficient, although the representation does not perfectly diagonalize the dynamics of natural videos. This is analogous to the situation in modern nonlinear signal processing where diagonal adaptive operators in appropriate bases have had a major impact in compression, denoising, and linear inverse problems [46]. Our empirical results demonstrate that these assumptions provide a reasonable description of image sequences. The standard deep networks considered here could in principle have discovered a similar solution, but they seem not to. This exemplifies a fundamental theme in computational vision and machine learning: when possible, let the representation do the analysis. An important limitation of the framework comes from the use of mean squared error, which is minimized by the posterior mean and tends to result in blurry predictions. Since temporal prediction is inherently uncertain, predictive processing should be probabilistic and exploit prior information.

The polar prediction framework suggests many interesting future directions, both for the study of visual perception (e.g. object constancy at occlusion, and object grouping from common fate) and for the development of engineering applications (e.g. building a flow-free video compression standard). To expand expressivity and better represent signal geometry, it may be possible to design a polar prediction architecture that also adapts the basis, potentially extending to the case of noncommutative transformations (e.g. the two dimensional retinal projection of three dimensional spatial rotation). Finally, it is worth considering the extension of the principles described here to prediction at longer temporal scales, which will likely require learning more abstract representations.

Acknowledgments

We thank members of the Laboratory for Computational Vision at NYU, and of the Center for Computational Neuroscience at the Flatiron Institute for helpful discussions. This work was supported in part by the Simons Foundation.

References

Peter Földiák. Learning invariance from transformation sequences. Neural Computation , 1991. 1

Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation , 2002. 1

Charles F Cadieu and Bruno A Olshausen. Learning intermediate-level representations of form and motion from natural movies. Neural computation , 2012. 1, 8

Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc video coding standard. IEEE Transactions on circuits and systems for video technology , 2003. 1

Peter Elias. Predictive coding-i. IRE transactions on information theory , 1955. 1

Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience , 2019. 2

Brian C Hall. Lie groups, lie algebras, and representations. In Quantum Theory for Mathematicians . Springer, 2013. 3

William T Freeman, Edward H Adelson, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence , 1991. 3

David J Fleet and Allan D Jepson. Computation of component image velocity from local phase information. International journal of computer vision , 1990. 4

Peter Burt and Edward Adelson. The laplacian pyramid as a compact image code. IEEE Transactions on communications , 31(4):532-540, 1983. 5, 15

Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision , 2000. 5, 15

Sixin Zhang and Stéphane Mallat. Maximum entropy models from phase harmonic covariances. Applied and Computational Harmonic Analysis , 2021. 5

Eero P Simoncelli and William T Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. Proceedings., International Conference on Image Processing , 1995. 7, 15

Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. ICLR , 2016. 7, 15

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention , 2015. 7, 15

Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675 , 2017. 7, 16

Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences , 1982. 8

Michael J Berry II, Iman H Brivanlou, Thomas A Jordan, and Markus Meister. Anticipation of moving stimuli by the retina. Nature , 398(6725):334, 1999.

Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proceedings of the National Academy of Sciences , 2015.

Olivier J Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero P Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications , 12(1):5982, 2021. 8

David Mumford. On the computational architecture of the neocortex: Ii the role of corticocortical loops. Biological cybernetics , 1992. 8

Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience , 1999.

Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience , 2010. 8

David J Heeger. Normalization of cell responses in cat striate cortex. Visual neuroscience , 1992. 8

Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a , 1985. 8

Eero P Simoncelli and David J Heeger. A model of neuronal responses in visual area mt. Vision research , 38(5):743-761, 1998. 8

Rajesh Rao and Daniel Ruderman. Learning lie groups for invariant visual perception. Advances in neural information processing systems , 1998. 8

Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning . PMLR, 2014. 8

Alexander Genkin, David Lipshutz, Siavash Golkar, Tiberiu Tesileanu, and Dmitri Chklovskii. Biological learning of irreducible representations of commuting transformations. Advances in Neural Information Processing Systems , 2022. 8

Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems , 31, 2018. 8

Ho Yin Chau, Frank Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lie group transformations and sparse coding. arXiv preprint arXiv:2012.12071 , 2020. 8

Sophia Sanborn, Christian Shewmake, Bruno Olshausen, and Christopher Hillar. Bispectral neural networks. arXiv preprint arXiv:2209.03416 , 2022. 8

Roland Memisevic and Geoffrey E Hinton. Learning to represent spatial transformations with factored higher-order boltzmann machines. Neural computation , 2010. 8

Jörn-Philipp Lies, Ralf M Häfner, and Matthias Bethge. Slowness and sparseness have diverging effects on complex cell learning. PLoS computational biology , 10(3):e1003468, 2014. 8

Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T Freeman. Phase-based video motion processing. ACM Transactions on Graphics , 2013. 9

Matthias Bethge, Sebastian Gerwinn, and Jakob H Macke. Unsupervised learning of a steerable basis for invariant image representations. In Human Vision and Electronic Imaging XII . SPIE, 2007. 9

Ross Goroshin, Michael F Mathieu, and Yann LeCun. Learning to linearize under uncertainty. In Advances in Neural Information Processing Systems , 2015. 9

Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems , 2018. 9

Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Waldo: Future video synthesis using object layer decomposition and parametric flow prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 9

William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. International Conference on Learning Representations , 2017. 9

William Lotter, Gabriel Kreiman, and David Cox. A neural network trained for prediction mimics diverse features of biological neurons and perception. Nature machine intelligence , 2(4):210-219, 2020. 9

Igor Mezi´ c. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics , 2005. 9

Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning , 2020. 9

Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications , 2018. 9

Gregory C DeAngelis, Izumi Ohzawa, and Ralph D Freeman. Receptive-field dynamics in the central visual pathways. Trends in neurosciences , 18(10):451-458, 1995. 10

Stéphane Mallat. A wavelet tour of signal processing: The sparse way. Academic Press, 2008. 10

Shan Zhu and Kai-Kuang Ma. A new diamond search algorithm for fast block-matching motion estimation. IEEE transactions on Image Processing , 2000. 15

J Hans van Hateren and Dan L Ruderman. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences , 1998. 16

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015. 16

A Notation

Let R N denote the N -dimensional Euclidean space equipped with the usual Euclidean norm || · || and let C K denote the K -dimensional complex vector space. Let C N × K denote the set of N × K complex-valued matrices. Let F ∗ denote the adjoint (ie. the conjugate transpose) of F .

Given vectors u , v ∈ C K , let u ⊙ v = [ u 1 v 1 , . . . , u K v K ] ⊤ ∈ C K denote the elementwise (aka. Hadamard) product of u and v . Let u ⊘ v = [ u 1 /v 1 , . . . , u K /v K ] ⊤ ∈ C K denote the elementwise division of u and v . Let diag ( u ) denote the K × K diagonal matrix whose ( k, k ) th entry is u k .

Let the complex number z ∈ C be expressed in rectangular coordinates as z = x + iy where ( x, y ) ∈ R 2 , or in polar coordinates as z = ae iθ where ( a, θ ) ∈ R + × [ -π, π ] . Its complex conjugate is z = x -iy = ae -iθ . The rectangular coordinates are u = a cos( θ ) and v = a sin( θ ) ; and the polar coordinates are a = | z | = √ x 2 + y 2 , and θ = ∠ z = atan2 ( y, x ) . We overload these notations to denote element-wise operations on a vector z ∈ C K : z = [ z 1 , . . . , z K ] ⊤ ∈ C K and | z | = [ | z 1 | , . . . , | z K | ] ⊤ ∈ R K + .

B Planted symmetries

(a)

Translation (cyclic boundary)

(b)

Rotation

(c)

Translation and rotation

(d)

Translation (open boundary)

Filters of polar predictor networks trained to predict small synthetic sequences. We randomly select 100 image patches of size 16 × 16 from the DAVIS dataset and generate training data by manually transforming them-applying translations or rotations. We verify that PP recovers the corresponding harmonic functions: Fourier modes for translation (panel a), and disk harmonics for rotation (panel b). To show that the recovery of harmonics is robust, we design two additional synthetic datasets. i) the combination of translational and rotational sequences. In this case, PP learns some filters

that correspond to either transformation. But the model does not have the expressivity required to dynamically adapt the representation to either transformation (panel c); ii) generalized translation sequences: spatially sliding a square window on a large image (ie. new content creeps in and falls off at boundaries), instead of using cyclic boundary condition (ie. content wraps around the edges). In this case, PP learns localized Fourier-like modes (panel d), indicating that approximate group actions still provide meaningful training signal-but some of the filters are not structured. In each panel, the 32 pairs of filters are sorted by their norm. Notice that some of high frequency harmonics are missing. This is due to the spectral properties of the datasets, which have more power at lower frequencies.

C Description of architectures

Motion Compensation We compare our method to the traditional motion-compensated coding approach that forms the core of inter-picture coding in well established compression standards such as MPEG. Block matching is an essential component of these standards, allowing the compression of video content by up to three orders of magnitude with moderate loss of information. For each block in a frame, typical coders search for the most similar spatially displaced block in the previous frame (typically measured with MSE), and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame. We implemented a 'diamond search' algorithm [47] operating on blocks of 8 × 8 pixels, with a maximal search distance of 8 pixels which balances accuracy of motion estimates and speed of estimation (the search step is computationally intensive). We use the estimated displacements to perform causal motion compensation ( cMC ), using displacement vectors estimated from the previous two observed frames ( x t -1 and x t ) to predict the next frame ( x t +1 ) rather than the current one (as in MPEG).

Complex Steerable Pyramid We consider a fixed multiscale oriented representation of image content: a steerable pyramid [13, 11] covering 16 orientations and 5 scales on the DAVIS dataset (resp. 16 orientations and 4 scales on VanHateren dataset). This choice of number of orientations and number of sacles maximizes prediction performance on the corresponding datasets.

Polar Predictor We use 32 pairs of convolutional channels with filters of size 17 × 17 pixels, without biases (no additive constants) and no padding (ie. 'valid' boundary condition). For the multiscale version (mPP), we use 16 pairs of convolutional channels with filters of size 11 × 11 pixels, without biases (no additive constants). The representation is computed inside a fixed Laplacian pyramid [10]. We used 4 scales for the DAVIS dataset (and respectively 3 scales for the VANH dataset). Within this multiscale representation, the learned filters are applied with zero padding (ie. 'same' boundary condition).

Quadratic Predictor For the multiscale version (mQP), we use the same analysis ( f w ) and synthesis ( g w ) hyperparameters as mPP. For the quadratic prediction mechanism, we use 16 groups of 4 convolutional filters (twice as many as for mPP). The quadratic predictor ( p w ) operates on groups of 4 coefficients and contains 12 quadratic units. It is more expressive than the multiscale Polar Predictor architecture and contains phase advance as a special case.

Vanilla CNN Finally, we implemented a more direct convolutional neural network predictor ( CNN ), that maps two successive observed frames to an estimate of the next frame [14]. For this, we used a CNN composed of 20 stages, each consisting of 64 channels, and computed with 3 × 3 filters without additive constants, followed by half-wave rectification (ie. ReLU). To facilitate learning, a skip connection copies the current frame x t and the network only outputs residuals that get added to the current frame in order to predict the next frame: ˆ x t +1 = x t + f w ([ x t , x t -1 ]) . This model jointly transforms and processes pairs of frames to generate predictions, while both polar predictor (PP) and quadratic predictor (QP) separate spatial processing and temporal extrapolation.

Unet The Unet architecture [15] is commonly used for image-to-image tasks. It has an analysissynthesis structure with downsampling, upsampling, and skip connections between levels at the same resolution. We used a 5 levels architecture, each block consists of two convolutions, batch norm and rectification. The convolutions have filters of size 3 × 3 , comprise 64 channels, and no additive bias. The end-to-end network comprises 20 non-linear stages with ReLU activations.

D Description of datasets and optimization

DAVIS To train, test and compare these models, we use the DAVIS dataset [16], which was originally designed as a benchmark for video object segmentation. Image sequences in this dataset contain diverse motion of scenes and objects (eg., with fixed or moving camera, and objects moving at different speeds and directions), which make next frame prediction challenging. Each clip is sampled at 25 frames per second, and is approximately 3 seconds long. The set is subdivided into 60 training videos (4741 frames) and 30 test videos (2591 frames). We pre-processed the data, converting all frames to monochrome luminance values, and scaling their range to the interval [ -1 , 1] . Frames are cropped to a 256 × 256 central region, where most of the motion tends to occur, and then spatially down-sampled to 128 × 128 pixels.

VanHateren We also consider a smaller video dataset consisting in footage of animals in the wild [48] which contains a variety of motions (animals in the scene, camera motion, etc.) and occlusions. The missing band at the top the frame is cropped, reducing the image size from 128 × 128 pixels to 112 × 128 pixels. The dataset is standardized to zero mean and unit variance, it is split into snippets of 11 frames, 292 snippets are used for training and 33 for testing. There is no spatial downsampling or whitening.

Boundary handling The computation of this prediction error is restricted to the center of the image because moving content that enters from outside the video frame is inherently unpredictable. Specifically, we trim a 17-pixel strip from each side, yielding frames of size 94 × 94 pixels for the DAVIS dataset and 78 × 94 for the VanHateren dataset. Convolutions are computed with zero padding so that the outputs have the same size as inputs (the only exception is for the plain Polar Predictor, shown in Figure 6, where valid convolutions were performed).

Training procedure We assume the temporal evolution of natural signals to be sufficiently and appropriately diverse for training, and do not apply any additional data augmentation procedures. We train on brief temporal segments containing 11 frames (which allows for prediction of 9 frames), and process these in batches of size 4. We train each model for 200 epochs on DAVIS using the Adam optimizer [49] with default parameters and a learning rate of 3 · 10 -4 . The learning rate is automatically halved when the test loss plateaus. In the CNN, we use batch normalization before every half-wave rectification, rescaling by the standard deviation of channel coefficients (but with no additive bias terms).

E Additional results

Additional dataset Note that the PSNR is the logarithm of the MSE in units of the signal: PSNR = 10 log 10 ( MAX 2 / MSE ) , where MAX is the maximum possible pixel value of the image. A PSNR of 0dB means that the squared peak signal and the mean squared error are equal; a PSNR of 10dB (resp. 20dB) means that the squared peak signal is 10 times (resp. 100 times) bigger than the MSE.

Computational costs The polar predictor described in this paper is lightweight: it runs rapidly and contains few parameters (two orders of magnitude less than CNN and Unet). The polar predictor is designed as an online method that could be applied to streaming data. Parameter counts, as well as training and inference time are reported in Table 3.

Notice that the Unet is also very efficient: it runs fast (most of the computation is applied to spatially downsampled coefficients). Note that the Quadratic Predictor (mQP) is slower. This model was

Method

developed to suggest a connection with physiology (each element recapitulates know functional building blocks of primate visual physiology), not to improve performance or efficiency.

Decoupling analysis and synthesis When decoupling the analysis and synthesis transforms (ie. untying the weights), a Polar Predictor (not multiscale) achieved a similar prediction performance on the VanHateren dataset (tied: train 29.87/ test 28.83; vs. untied: train 29.99 / test 28.87 dB). The projection and basis vectors (ie. the filters in the analysis and synthesis transforms) align as training progresses.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors create self-supervised video prediction model that relies on principles of the Fourier shift theorem. The approach is inspired by work suggesting that humans rely on perceptual straightened representations to support prediction. The authors first validate their method on toy cases of simple translation and rotation and later show that their model outperforms motion compensation and a some standard deep learning learning models at next frame prediction. Finally, they draw connections between their model and early visual processing in biological vision and discuss their model's impact more widely to computer vision (a simpler and more principled way to get natural video representations).\n\nSTRENGTHS:\nThe methods are well thought-out and of good quality. The connection between perceptual straightening and the Fourier shift theorem is clever and novel. The use of both local and multi-scale processing in the model's design is is thoughtful and clear grounded in principles of human visual processing. In general, the authors make it very clear how their method relates to prior work -- this makes it the novelty and significance of the work clear.\n\nThe analyses of the model are good. I appreciate that the authors test their approach in synthetic data and then move to larger natural video datasets like DAVIS. Some good baselines are included in the analysis (copying frames, basic CNN), and there is an analysis of the filters learned by the model.\n\nWEAKNESSES:\nThe main weakness I see not having enough baseline comparisons to existing methods in video prediction or video representation learning. The work only compares their model to a basic CNN which makes it difficult to understand how their model compares to more state of the art approaches or even older methods grounded in neuroscience like PredNet. Reporting scores for more models or adding another dataset (like Kinetics) with more performance measures like mse or lpips would address this point. In addition, more detail on compute and training requirements would help with understanding the significance of their approach.\n\nQUESTIONS:\nI generally found the paper clear and interesting, but I would like to hear more on the following points:\n\nCould the authors discuss how their model might compare new architectures ViT and diffusion models for video prediction? Could the authors report MSE, SSIM, and lpips scores to help me get a better sense of how their model compares to other approaches? To that end, do the authors have any performance results on other datasets that are used in video prediction (for example moving MNIST or Kinetics)?\n\nDo the authors have any thoughts on other downstream tasks their video representation learning framework could be useful for? Or are there even aspects of video prediction that your model might do better at being grounded in a straightening representation space (for example long range prediction vs short range prediction)?\n\nLIMITATIONS:\nLimitations are not widely discussed, but aside from the weaknesses I highlighted I do not see a glaring limitation that needs to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Motivated by Fourier shift theorem and its group-theoretic generalization, this paper proposed a new video prediction model.

STRENGTHS:
1. Biological modeling: computational elements can be used to describe primate V1 responses. 
2. Fourier shift theorem and its group-theoretic generalization were incorporated into future prediction

WEAKNESSES:
1. Evaluation: only simple baselines like CNN were included for comparison. Only one quantitative metric, PSNR, was used. Other metrics like SSIM or FVD can be more informative. No demonstration of any qualitative results.
2. Loss: this framework used MSE as loss for optimization, which indicated this setting didn't consider any uncertainty factors.

QUESTIONS:
I am not familiar with computational neuroscience. Therefore, I can only provide useful feedback with respect to generative performance.

LIMITATIONS:
No limitations were discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research develops a self-supervised representation-learning framework that uses the regularities of natural videos to make accurate predictions. The architecture is inspired by the Fourier shift theorem and trained for next-frame prediction. The approach can discover irreducible representations of smooth commutative groups in data. It achieves similar performance comparing to the traditional motion compensation and conventional deep networks, while being interpretable and fast. The framework is implemented using normalized simple and direction-selective complex cell-like units, similar to primate V1 responses, demonstrating its potential to explain how the visual system makes temporal predictions.\n\nSTRENGTHS:\n-\tThe paper leverages the temporal regularity of the natural video and propose an video compression algorithm for temporal linearization and compression. The model exploits the local symmetries present in the temporal evolution of images therefore save computational resource comparing to the end-to-end CNN models. This is novel and original for video compression and image representation learning.\n-\tThe paper suggests an important research direction: instead of producing an end-to-end generic deep learning framework, the authors leverages the temporal regularity of the natural image sequence to reduce the computational cost. Human mind utilizes certain evolutionary bias to improve its efficiency, so should we exploit these shortcut and apply them into the computer vision research. \n-\tThe paper’s method is elaborated clearly with precise mathematical notation. The presentation flow is clear although it would be benefit to draw illustrative figures to show intuitively show the main idea of the paper.\n\nWEAKNESSES:\n-\tDespite the promising direction of research, the comparison between CNN models and the proposed method doesn’t demonstrate a significant improvement in terms of numbers. Further error bar would provide more concrete idea of how much the method has improved. \n-\tOther evaluation metrics could be useful such as the time of performing compression, as well as the computational resource. It would still be a good contribution if the model can achieve the same performance but using significantly less resources. Although the paper briefly mentioned this, it would be more thorough to outline the resource saving using the proposed method. \n-\tA description of the overall algorithm would make the presentation more clear.\n\nQUESTIONS:\n-\tDetailed experimental setup and dataset description are missing in the main text. \n-\tWould be beneficial to see other evaluation metrics on compression decoding quality and efficiency would be beneficial other than only PSNR metrics, e.g. bitrate, inference time etc.    \n-\tWill the learned representation also be useful for image classification or other static tasks?\n\nLIMITATIONS:\nThe authors have drawn parallels between the computational steps of this approach and classical models of early visual processing in primates. However, the evidence supporting these parallels could be better substantiated. Quantitative comparison to physiological data may not be a straightforward task, and potential challenges in doing so should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this work propose a new self-supervised learning technique that is aimed to perform predictive processing of natural videos (although the approach seems broad enough to be applicable to other sequential signals with similar inductive priors to vision). The authors develop two parameter-efficient architectures (multiscale quadratic and polar prediction mechanisms, mQP, mPP) based on Fourier shift theorem and its group theoretic generalization and train them to perform next-frame prediction from natural videos. Experimental results show that their proposed abovementioned mQP and mPP architectures outperform relevant conventional motion coding baselines and a supervised CNN on video reconstruction on the two video datasets of VanHateren and DAVIS. It is very intriguing that an interpretation of the learned filters in their architecture shows similarity to canonical receptive fields observed in V1 cells, hence providing a new theoretical hypothesis for a potential function of V1 cells in performing future prediction.\n\nSTRENGTHS:\n+ The proposed work seems theoretically very sound, the proposed architectures mQP and mPP are designed ground up based on the extension of the Fourier shift theorem to account for transformations that occur in natural videos. Adequate mathematical and illustrative derivation of the proposed algorithm is very useful to grasp the underlying mechanism even for non-experts in this specific area such as myself.\n+ It is super interesting that canonical receptive field structure identified in V1 cells is emergent from the proposed self-supervised architecture and this serves as a foundation for the hypothesis of V1 functioning in temporal straightening and future prediction.\n+ Experimental results show that the proposed approach is clearly well-performing in terms of future frame prediction. These evaluations aren't very extensive like typical machine learning papers but the message from the paper is clearly conveyed in this demonstration. Additionally the link to biological modeling that I mentioned previously makes the paper a fresh read with many interesting avenues.\n+ The writing is great overall and related prior work has been covered quite comprehensively.\n\nWEAKNESSES:\n- In the experimental evaluation, it maybe good to include the number of parameters and runtime (training and inference if possible, but certainly for inference) for each algorithm. I believe this will provide a more full picture and show how much more efficient the proposed architecture is.\n- It would be great if the authors also add measures of variance for the performance of various algorithms on the two datasets compared in the evaluation section.\n- Although the authors have briefly touched upon potential alternate usecases of the proposed algorithm in other vision tasks (segmentation, direction estimation etc.) it would be good for readers in the broader audience to elaborate further on this. \n- Limitations of the proposed work aren't discussed in much detail, I only see that one potential limitation listed here is the inability of current proposed algorithms to generalize to non-commutative groups which is left for future scope. I encourage the authors to further discuss this in more detail and add other directions they see for extension of this work to the paper.\n\nQUESTIONS:\n- Please see the weaknesses section above for my suggestions.\n\nLIMITATIONS:\nThe authors have addressed limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a self-supervised representation-learning framework inspired from the idea of continuous deformations of objects in videos. The framework allows for next-frame prediction given previous ones, and borrows ideas from the Fourier shift theorem. The framework is composed of three stages - an analysis transformation $f_w$ of given frames, a prediction transformation $p_w$ that predicts the next frame representation given previous ones, and a synthesis transformation $g_w$ which transform the latent representation to the actual frame. $g_w$ is implemented as the inverse of $f_w$, and the two share parameters. The authors present two variants of their framework, the polar predictor (PP) where $p_w$ is a fixed polar extrapolation, and a learnt $p_w$ variant termed the quadratic predictor (QP). The authors also detail results on next-frame prediction task, and show that their method surpasses baseline techniques.\n\nSTRENGTHS:\n- The idea of exploiting the temporal symmetries and local deformations in videos to derive a natural framework for next-step prediction is interesting and has the potential to provide interpretability for sequence prediction tasks.\n\n- The idea of sharing parameters between the synthesis and analysis operators, contributes clarity to the framework's underlying process.\n\n- Multiscale processing using a Laplacian pyramid is an interesting solution to overcome video transformations of different scales.\n\n- The proposed framework shows promising results in the provided evaluation.\n\nWEAKNESSES:\n- Technical details regarding the implementation of the learned transformations is missing from the main paper.\n\n- The section on multiscale processing using a Laplacian pyramid, in my opinion, is hard to follow. A figure illustrating the process would greatly help.\n\n- The paper could benefit from some ablation tests, for example - using shared weights for $f_w$ and $g_w$ as opposed to separate weights, using different network architectures/depths and so on.\n\n- The authors present the method as a representation-learning framework, but it's not clear how the learnt representations could assist with tasks beyond next-frame prediction? I would appreciate listing some additional possible applications. Video compression is briefly mentioned as a possibility, but no details are given (are the representations compact relative to the original frame?)\n\n- Regarding the comparison to baseline methods - the baseline CNN architecture chosen for comparison seems quite arbitrary. On the one hand, the authors claim their method achieves comparable or better results using less parameters than the CNN, but on the other hand - no baseline CNN with the same number of parameters is given for comparison. In addition, the architecture seems arbitrary - 20 layers with the same number of channels, no skip connections in between (only one that wraps the entire network so that the predictions are residuals) and no information regarding non-linearities and/or normalization layers. It seems to me that choosing an of-the-shelf architecture for general image-to-image tasks (and slightly modifying the input/output) would have been a much better and competitive baseline. In addition, the realted work section mentions additional works on next-frame prediction (for example using LSTMs), which could contribute to the comparison. I would appreciate the author's response on this.\n\nQUESTIONS:\n- Have you tried using the proposed framework to predict additional frames, as opposed to only the next one? It seems quite intuitive and straightforward using the proposed method since it only requires an additional step in the polar extrapolation.\n\n- Are there any latency constraints to the proposed approach? For example in the transformation to polar coordinates.\n\nLIMITATIONS:\nAfter reading through the paper a few times, I couldn't find any addressed limitations (although I can't point to any unaddressed limitations).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}