Learning vector representation of local content and matrix representation of local motion, with implications for V1
Authors: Ruiqi Gao
Jianwen Xie
Siyuan Huang
Yufan Ren
Song-Chun Zhu
Ying Nian Wu
Venue:  (2020)
Keywords: Representation learning
V1
neuroscience

Abstract: This paper proposes a representational model for image pair such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1). The model couples the following two components. (1) The vector representations of local contents of images. (2) The matrix representations of local pixel displacements caused by the relative motions between the agent and the objects in the 3D scene. When the image frame undergoes changes due to local pixel displacements, the vectors are multiplied by the matrices that represent the local displacements. Our experiments show that our model can learn to infer local motions. Moreover, the model can learn Gabor-like filter pairs of quadrature phases.

Under review as a conference paper at ICLR 2020
LEARNING VECTOR REPRESENTATION OF LOCAL CON- TENT AND MATRIX REPRESENTATION OF LOCAL MO- TION, WITH IMPLICATIONS FOR V1
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes a representational model for image pair such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1). The model couples the following two components.
(1) The vector representations of local (2) The matrix representations of local pixel displacements caused by the relative motions between the agent and the objects in the 3D scene. When the image frame undergoes changes due to local pixel displacements, the vectors are multiplied by the matrices that represent the local displacements. Our experiments show that our model can learn to infer local motions. Moreover, the model can learn Gabor-like filter pairs of quadrature phases.
contents of images.
INTRODUCTION
Our understanding of the primary visual cortex or V1 (Hubel & Wiesel, 1959) is still very limited (Olshausen & Field 2005). In particular, the mathematical and representational models for V1 are still in short supply. Two prominent examples of such models are sparse coding (Olshausen & Field 1997) and independent component analysis (ICA) (Bell Sejnowski 1997). Although such models do not provide detailed explanations of V1 at the level of neuronal dynamics, they help understand the computational problems being solved by V1.
In this article, propose model of this sort. It is representational model of natural image pair that are related by local pixel displacements. The image pair be consecutive frames of video sequence, where the local pixel displacements are caused by the relative motions between the agent and the objects in the 3D environment. Perceiving such local motions can be crucial for inferring ego-motion, object motions, and 3D depth information.
M(
(x(rr
(
As is the with existing models, expect our model explain only limited aspects of V1, some of which are: (1) The receptive fields of VI simple cells resemble Gabor filters (Daugman 1985). (2) Adjacent simple cells have quadrature phase relationship (Pollen & Ronner 1981). (3) The V1 cells are capable of perceiving local motions. While existing models can all explain (1), our model can also account for (2) and (3) naturally. Compared to models such as sparse coding and ICA, our model serves a more direct purpose of perceiving local motions.
Figure 1: Scheme of representation
Our model consists of the following two components. See Figure T for an illustration, where the image is illustrated by the big rectangle. A pixel is illustrated by dot. The local image content is illustrated by a small square around it. The displacement of the pixel illustrated by a short arrow, which is within the small square. The vector representation of the local image content is represented by a long vector, which rotates as the image undergoes deformation due to the pixel displacements. Section explains the notation.
(1) Vector representation local image content. The local content around each pixel is represented by high dimensional vector. Each unit in the vector is obtained by linear filter. These local filters

---

Under review as a conference paper at ICLR 2020
wavelets assumed to form normalized tight frame, i.., the image can be reconstructed from the vectors using the linear filters as the basis functions.
(2) Matrix representation of local displacement. The change of the image from the current time frame to the next time frame is caused by the displacements of the pixels. Each possible displacement is represented by matrix that acts on the vector. When the image changes according the displace- ments, the vector at each pixel is multiplied by the matrix that represents the local displacement, in other words, the vector each pixel is rotated by the matrix representation of the displacement of this pixel.
One motivation of work comes from Fourier analysis. An image patch I can be expressed by the Fourier decomposition I Σx (gel(wa.)) Assuming the image patch undergoes smooth motion so that all the pixels are shifted by a constant displacement dz, the shifted image patch J(x) I(x - dx) = E. C =()))d @((wx=), The change from the complex number C to e&e =( corresponds to rotating 2D vector by 2 2 matrix. However, we emphasize that our model does not assume Fourier basis or its localized version such as Gabor filters. The model figures it out with generic vector and matrix representations.
We train this representational model on image pairs where in each pair, the second image is a deformed version of the first image, and the deformation is known. We learn the encoding matrices for vector representation and the matrices that represent the pixel displacements from the training data.
Our experiments show that our method can learn VI-like units that can be well approximated by Gabor filters with quadrature phase relationship. After learning the encoding matrices for vector representation and the matrix representations of the displacements, we can infer the displacement field using the learned model. Compared to current optical flow estimation methods (Dosovitskly jtt al. 2015 Ilg et al. 2017). which use complex deep neural networks to predict the optical flow, our model is much simpler and is based on explicit vector and matrix representations. We also demonstrate comparable results to these methods, terms of the inference of displacement field.
In terms of biological interpretation, the vectors can be interpreted as activities of groups of neurons, and the matrices can be interpreted as synaptic connections. See subsections 4.3 and 4.4 for details.
CONTRIBUTIONS AND RELATED WORK
This paper proposes a simple representational model that couples the vector representations of local image contents and matrix representations of local pixel displacements. The model is new and different from existing models for V1. It explains some aspects of V1 simple cells such as Gabor- like receptive fields and quadrature phase relationship. It adds to understanding of V1 motion perception in terms of representational and relational model.
The following are two themes of related work.
(1) V1 models. Most well known models for V1 are concerned with statistical properties of natural images video sequences. Examples include sparse coding model (Olshausen & Field 1997]
Lewicki & Olshausen 1999, Olshausen 2003). independent component analysis (ICA) (Hy
al. 2004 Bell & Sejnowski] [997| van Hateren & Ruderman, 1998), slowness criterion (Hyvärinen
et al. 2003 Wiskott & Sejnowski 2002). ) and prediction (Singer et al. 2018). While these models are very compelling, they do not serve direct purpose of perceptual inference. Our model is learned for the direct purpose of perceiving local motions caused by relative motion between the agent and the surrounding 3D environment.
We want to emphasize that our model is complementary to the existing models for V1. Similar to existing models, our work assumes a linear generative model for image frames, but our model adds relational component with matrix representation that relates the consecutive image frames. Our model is also complementary to slowness criterion in that when the vectors are rotated by matrices, the norms of the vectors may remain constant.
(2) Matrix representation. In representation learning, it is a common practice to encode the signals or states vectors. However, it is much less explored theme to represent the motions, actions or relations by matrices that act on the vectors. An early work in this theme is (Paccanaro & Hinton 2001), which leams matrices represent relations. More recently, (Jayaraman & Grauman 2015)
2

---

Under review as a conference paper at ICLR 2020
learns matrix representation for ego-motion. (Gao al., 2018) learns vector representation for self-position and matrix representation for self-motion in a representational model for grid cells. Our work constitutes a new development along this theme.
The matrix representation of local displacements in our work is partially inspired by the group representation theory, where the group elements represented by matrices acting the vectors (Fulton & Harris, 2013). In our work, local displacements belong to 2D Euclidean group. Our modeling of local motion dx is similar to the treatment of Lie group via Lie algebra by analyzing infinitesimal changes. The objects in the image may undergo more complex motions which form more complex Lie groups (e.g., rotations and translations). We can again represent the objects (e.g.. their poses) in the scene by vectors, and represent the motions of the objects by matrices. The representation theory underlies much of modern mathematics and holds the key the quantum theory (Zee 2016). Perhaps it also underlies the visual and motor cortex, where the neuron activities encode vectors, and the synaptic connections encode the matrices that rotate them, with the matrices representing motions, actions, and relations.
REPRESENTATIONAL MODEL.
3.1 VECTOR REPRESENTATION
Let (I(r), E D} be an image observed at a certain instant, where æ (x1, is the 2D coordinates of pixel. D is the image domain (c.g., 128 x 128). We represent the image I by vectors (u(x), x E D-}. where each v(x) vector defined pixel x, and D_ may consist of sub-sampled set of pixels in D (e.g., sub-sampled every 8 pixels). V {w(x), E D-} forms vector representation of the whole image.
We assume the vector encoding is linear and convolutional. Specifically, let [[x]] be a squared patch (e.g., 16 x 16) of centered at æ. We can make [[x] into vector (e.g., 256 dimensional). Let
(1) he the linear encoder, where W is the encoding matrix that encodes [[x] into a vector v(x), and W is the same for all x, i.e., convolutional. The rows of W are the linear filters and can be displayed as local image patches of the same size as the image patch [xx]. We can write V = WI, if we treat I as vector, and the rows of W the shifted or translated versions of W.
rxx) = WT[z], x E D
3.2 NORMALIZED TIGHT FRAME AND ISOMETRY
We assume that W is an auto-encoding normalized tight frame, i.e.,
I= WV
Thus, the linear filters for bottom-up encoding also serve as basis functions for top-down decoding. Both the encoder and decoder can be implemented by convolutional linear neural networks.
The normalized tight frame assumption can be justified by the fact that for two images and J, we have (WI, WJ) = I (TWT WJ = (I.J), that is, the vector representation preserves the inner product. As result, ||W]] [I], |WJ]| [J]], thus the vector representation also preserves the angle and has the isometry property.
When the image I changes from I, to I,+1, its vector representation V changes from V4 to V4+1, and the angle between I, and I441 is the same as the angle between V, and V4+1-
3.3 SUB-VECTORS
The vector v(x) can be high-dimensional. We further divide v(x) into K sub-vectors, v(r) = (x), k = =1, 1,... K') Each sub- vectvr is obtained by an encoding sub-matrix W(A), i.e., (x)())
[[]], to = 1,.... K, where ([k)) consists of the rows of W that correspond to ((k) According to the normalized tight frame assumption, have I = ExED_ Σ W(I) ((t) (z). In practice, find that this assumption is necessary for the emergence of V1-like receptive field.
3

---

Under review as a conference paper at ICLR 2020
3.4 MATRIX REPRESENTATION
Let I, be the image at time frame I. Suppose the pixels of I, undergo local displacements, where the displacement at pixel x is ((r). We assume that (xx) is within a squared range (e.g. 6) x [-6, 6) pixels) that is inside the range of I, (e.g., 16 x 16 pixels). Let 4+1 be the resulting image. Let (x()) be the vector representation of I [z), and let (+4( (z) be the vector representation of [
Then (2)) = (vr (k) (z:), k 1,.., K). and u(41 (2) (0++1 (4), 1,..., K).
(16)
The transition from I to I++1 illustrated by the following diagram:
(o(e)))
(9) (x)
"41 (x)
(3)
(K))
((K)
((x)
I
I++1
Specifically, we assume that
(k) (x), Vx D_, I..., 4 (x) (8xx))) (z),
(4)
That is, when I changes from I, to I (x) undergoes linear transformation, driven by matrix (5xz), which depends on the local displacement 8(z). In terms of the whole vector v(x) (v(() 1,..., K), we have (z) M (6(x)))) where M(((r)
diag M (8(x)), k I,..., K) is the matrix representation of the local displacement d(x).
3.5 DISENTANGLED ROTATIONS
The linear transformations the sub-vectors (z) can be considered as rotations. Here we use the word "rotation" " in the loose sense without strictly enforcing M (R) to be orthogonal. v(x) is like a multi-arm clock, with each arm ()() (z) rotated by (k) (8(x)) The rotations of ((*) (x) for different & and disentangled. Here disentanglement means that the rotation of a sub-vector does not depend on other sub-vectors.
The disentanglement between different positions is the key feature of our model. Recall the change of image I is caused by the displacement of pixels, yet the rotations of sub-vectors ()()() at different pixels r are disentangled. This enables the agent to sense the displacement of a pixel only by sensing the rotations of the sub-vectors at this pixel without having to establish the correspondences between the pixels of consecutive frames.
3.6 PARAMETRIZATION
We can discretize the displacement ((x) into finite set of possible values {6}, and we learn separate
M (*)()) (6) for each d. We can also learn parametric version of (*) (8) the second order Taylor
expansion a matrix-valued function of (61,62). (8) = I + B (k] 5, + (B²h) (2) 8, + B 8
B 82 B(b)) 8102, where I is the identity matrix, and ((k) (B(*) B (k) B(( B B are matrices of coefficients of the same dimensionality as ((k) (8).
(k)
3.7 LOCAL MIXING
((x:) is large, u (I) may contain information from adjacent image patches of I, in addition to [x]]] We can generalize the motion model in Equation allow local mixing encoded vectors.
Let S be a local support centered at O. We assume that
Σ doES
(5(x) (1) (1) + da))
(5)
In the learning algorithm, we discretize dr and learn separate (+) (5, dr) for each dx.
4

---

Under review as a conference paper at ICLR 2020
4 LEARNING AND INFERENCE
The input data consist of the triplets (Ie, (0(x), D-), It+1), where (ō(z)) the given displace- ment field. The learned model consists of matrices (W(K), (f)) 1,.., K E A), where A is the range of 8. In the case of parametric M(K) learn the B matrices in the second order Taylor expansion in subsection 3.6
4.1 LOSS FUNCTIONS FOR LEARNING
We use the following loss functions:
(1) Rotation loss
| M (6
Lr,ak
(6)
For local mixing generalization, L1,x, I+t1 Exves dz + dx)
dd))|
(2) Reconstruction loss
2 Σ W WL| + |++1 RED_
L2
I,
Σ w WL++ SED_
In the learning algorithm, we learn the model by a weighted sum of the expectations of ExEDD Lı,z, and L2, where the expectations are taken over the training pairs of images and the corresponding displacement fields.
4.2 INFERENCE OF MOTION
After learning (W(),, (8). Yk, Y8), for a testing pair (I, I441), we can infer the pixel displace- ment field (6(z), D-) by minimizing the rotation loss: 8(x) arg maNse (o), where
(o)
Σ w w [+1 (8)
1 [ ||= = [I [a] M(D)WI
(8)
This algorithm is efficient because it can be parallelized for all D_ and for all 8 A.
If we learn parametric model for (o), we can infer the displacement field (6(x), Vxr) by minimizing Σ, (fxx)) using gradient descent with an initialization of (f(xr) from random small values. To encourage the smoothness of the displacement field, add the penalty term [[=
4.3 BIOLOGICAL INTERPRETATIONS OF CELLS AND SYNAPTIC CONNECTIONS
The learned (R),), (o)) Ak. d) can be interpreted as synaptic connections. For each W() corresponds to one set of connection weights. Suppose 8 E A is discretized, then for each 5, (+) (5) corresponds to one set of connection weights, and (M (8), A) corresponds to multiple sets of
connection weights. After computing ur
(k) ( I, [z). (6)u,, (k) is computed simultaneously
for every E A. Then 8(x) is inferred by max pooling according to Equation (8)
(k) vt can be interpreted activities of simple cells, and [| can be interpreted activity of
complex cell. we enforce norm stability so that [|,! [| then the complex cell response is invariant to the local motion and is related to the slowness property (Hyvärinen et al. [0033 Wiskott
& Sejnowski 2002), which is a by-product of our model if (A) (6) a rotation matrix, which is covariant with the local motion.

---

Under review as a conference paper at ICLR 2020
4.4 SPATIOTEMPORAL FILTERS AND RECURRENT IMPLEMENTATION
If we enforce norm stability or the orthogonality of (o)) (6), then minimizing ||Ue=1,> M M (6) over 8 E A is equivalent to maximizing (1++1,,,, M (o).,,)) which in turn is equivalent maximizing l|1++1.. + M (=)2,.=11
that U4+1, and (6) aligned. This alignment criterion can be conveniently generalized to multiple consecutive frames, so that we can estimate the velocity at a by maximizing the m-step alignment score ||ul|? where
m M(6) 24tL Σ M(5) WI
U
consists of responses of spatiotemporal filters, and |ul corresponds to the energy of motion 5 in the motion energy model (Adelson & Bergen 1985) for direction selective cells. Thus our model connected with the motion energy model. Moreover, our model enables a recurrent network for computing ti by u; = Ut+i,z + (ō)u,;1 for i = O..... m, with u = 0, and и = Unn This recurrent implementation is much more efficient and biologically plausible than the plain implementation of spatiotemporal filtering which requires memorizing all the I for i 0, ..... m. See (Pachitariu & Sahani (017) for a discussion of biological plausibility of recurrent implementation of spatiotemporal filtering in general.
5 EXPERIMENTS
First, in section 5.1 we introduce the datasets used to learn the models. Then in section 5.2 we show the learned Gabor-like units, and make connection with the spatial profile of simple cells cat and Macaque monkey in terms of neuroscience metrics, indicating the biological plausibility of the learned units. Then in sections 5.3 5..4 we show the learned representations can be applied to infer displacement field reasonably well, and the representations can be trained either in supervised or unsupervised manner. Please refer to appendix A for the implementation details. In appendices B and we illustrate that the learned representations are capable of two extra tasks, frame animation and interpolation.
5.1 SYNTHETIC AND PUBLIC DATASETS
ViDeform. Usually is difficult get ground truth motions from natural video frames. Thus we consider learning from image pairs with synthetic motions. First we consider random smooth deformations for natural images. Specifically, We can obtain the training data by collecting static images for (I,) and simulate the displacement field (8(z)). The simulated displacement field is then used to transform L, to obtain I,+1- We retrieve natural images I, from MIT places365 dataset (Zhou et al.) 2016). The images are scaled to 128 x 128. We sub-sample the pixels of images into a m X mn grid (m = in the experiments), and randomly generate displacements on the grid points, which serve as the control points for deformation. Then 8(z) for D can be obtained by spline interpolation of the displacements on the control points. We get I1+1 by warping I using 6(x) (Jaderberg et al. 2015). When generating displacement 8 (ō1, 62), both ō and 52 randomly sampled from range of [-6, +6]. We generate 20, 000 pairs for training and 3, 000 pairs for testing. We name this dataset VIDeform.
Vibjeing Next we consider separating the displacement field into motions of the background and foreground, to jointly simulate the self-motion of the agent and the motion of the objects in the natural 3D scenes. To this end, we create a synthetic dataset, by applying affine transformations to background images collected from MIT places365 (Zhou et al., 2016) and foreground objects from a public 2D object dataset COIL- (Nene et 1996). The background images are scaled to 128 x 128, and the foreground images are randomly rescaled. To generate motion, we randomly sample affine parameters of translation, rotation, and scaling for both the foreground and background images. The motion of the foreground objects are relative to the background images, which can be explained as the relative motion between the moving object and agent. We tune the distribution of the affine parameters to keep the range of the displacement fields within [-6, +6], which is consistent with the VIDeform dataset. Together with the mask of the foreground object and the sampled transformation parameters, we render the image pair (I, I.1) and its displacement field (6(z)) for each pair of background image and foreground image.

---

Under review as a conference paper at ICLR 2020
Specifically, obtain the estimated masks from (tev 2006) for the 2D foreground objects and remove some textureless objects, resulting in 96 objects with 72 views per object available. We generate 14, 411 synthetic image pairs with their corresponding displacement fields and further split 12,411 411 pairs for training and 2, 000 pairs for testing. We name this dataset V VIFlyingObjects Compared with previous optical flow dataset like Flying Chairs (Dosovisskiy et al. 2013) and scene flow dataset like FlyingThings23 Things3D (Mayer et al., (216)). the proposed VIFlyingObjects. dataset has various foreground objects with more realistic texture and smoother displacement fields, which simulates more realistic environments. We shall release this dataset.
MPI-Sintel. MPI-Sintel (Butler et al. 2012; Wulff et al., 2012) is a public dataset designed for the evaluation of optical flow derived from rendered aritificial scenes, with special attention to realistic image properties. Since MPI-Sintel relatively small which contains around a thousand image pairs, we use it only for testing the learned models in the inference of displacement field and frame animation, as described in details in sections 553 and Appendix [ We use the final version of MPI-Sintel and resize each frame into size 128 x 128. We select frame pairs whose motions are within the range of [-6, +6], resulting in 384 frame pairs in total.
MUG Facial Expression. MUG Facial Expression dataset (Aifanti et al., 2010) records natural facial expression videos of 86 subjects sitting in front of one camera. This dataset has no ground truth of displacement field, which we use for unsupervised learning stated in details in section [5.4] 200 videos with 30 frames are randomly selected for training, and anther 100 videos are sampled for testing. The frame images are resized to 64 x 64.
5.2 LEARNED GABOR-LIKE UNITS WITH QUADRATURE PHASE RELATIONSHIP
In this section we show and analyze the learned units. Figure a) displays the learned units, i.e., rows of WR),, on VIDeform. The units are learned with non-paramttric (,), i.e., we learn a separate ((5) for each displacement. 5(x) is discretized with an interval of 0.5. Similar patterns can be obtained by using parametric version of (8). Please refer to the supplementary and D for more results, including animation of filters, filters learned with local mixing motion model (eqn. (5). with different block sizes, and learned on V VlJlyingObjects. VI-like patterns emerge from the learned units. Moreover, within each sub-vector, the orientations and frequencies of learned units are similar, while the phases are different.
To further analyze the spatial profile of the learned units, we fit every unit by two dimensional Gabor //22a cos(2n + +). where (ar 3/) is obtained by translating and rotating the original coordinate system (æo, yo): x' (x= - Io) cos 0 + (y - yo) sin f y -xx xa) sin 0 + (y yo) cos O. The fitted Gabor patterns are shown in figure B(b.. with the average fitting r2 equal to 0.96 (std 0.04). The average spatial-frequency bandwidth is 1.13 octaves, with range of 0.12 to 4.67. Figure(c) shows the distribution of the opatial-frqqueet
function (Jones& & Palmer 1987h: haa', v/) A exp( 32 -3--0'y (v'
bandwidth, where the majority falls within range of 0.5 to 2.5. The characteristics are reasonably similar to those of simple-cell receptive fields in the cat (Issa et al. 2000) (weighted mean 1.32 octaves, range of 0.5 to 2.5) and the macaque monkey (Foster et al. 1985) (median 1.4 octaves, range of 0.4 to 2.6). To analyze the distribution of the spatial phase ø we follow the method in (Ringach) 2002) to transform the parameter á into an effective range of O to π 7/2, and plot the histogram of the transformed & in figure Cc). The strong bimodal with phases clustering near 0 and T T/2 is consistent with those of the macaque monkey (Ringach) 2002).
In the above experiment, we fix the size of the convolutional filters (16 x 16 pixels). A more reasonable model is to have different sizes of convolutional filters, with small size filters capturing high frequency content and big size filters capturing low frequency content. For fixed size filters, they should only account for the image content within frequency band. To this end, we smooth every image by two Gaussian smoothing kenels (kernel size 8, a = 1,4), and take the difference between the two smoothed images as the input image of the model. The effect of the two smoothing kernels similar to a bandpass filter, so that the input images are constrained within a certain range of frequencies. The learned filters on VIDeform are shown in aa). Again for every unit, we fit it by a two dimensional Gabor function, resulting in an average fitting x2 = 0.83 (std- = 0.12). Following the amalysis of (Ringach) 2002 Rehn & Sommer 2007). a scatter plot of n. ozf f versus ny = oyf is constructed in Figure b(bb based on the fitted parameters, where ng and n,, represent the width and length of the Gabor envelopes measured in periods of the cosine waves. Compared to Sparsenet
7

---

Under review as a conference paper at ICLR 2020
33
Spotii 65-1 11-525 025
23
Spotial phase
(a) Learned units
(b) Fitted Gabor patterns
(c) Frequency and phase
Figure 2: Learned results VIDeform. Learned units. Each block shows two learned units within the bandwidth (in octaves) and
same sub-vector. (b) Fitted Gabor patterns. (c) Distributions spatial phase Φb
(Olshausen & Field 1996 1997), the learned units by our model have more similar structure to the receptive fields of macaque monkey.
We also show profile of the learned units within each sub-vector in Figure C(c). Within each sub- vector, the frequency f and orientation 0 of the paired units tends be the same. More importantly, most of the paired units differ in phase á by approximately π //, consistent with the quadratic phase relationship between adjacent simple cells (Pollen & Ronner 1981 Emerson & Huang 1997).
2.5
Curmedel
Scartene:
witth
width
(b) Gabor envelope shapes of the learned units
Jaherence onterraiinn
O/fer ton plase
(a) Learned units
(c) Profile of paired units within each sub-vector
Learned results band-pass image pairs from VIDeform. (a) Learned units. Each block shows learned units within the sub-vector. (b) Distribution the Gabor envelope shapes in the width and length plane. (c) Difference frequency f. orientation and phase paired units within each sub-vector.
Figure 3:
5.3 INFERENCE DISPLACEMENT FIELD
We then test the learned representations in terms of inferring the displacement field (8(z)) be- tween pairs of frames (I, 1.). To get valid image patches for the inference, we leave those displacements at image border (8 pixels at each side).
We infer the displacement field (6(x)) using the learned vector and matrix representation. On top of that, we also train CNN model with ResNet blocks (He et al. 2016) to refine the inferred displacement field. In training this CNN, the input is the inferred displacement field, and the output is the ground truth displacement field. with least squares regression loss. The detailed model structure in appendix]] ! For V ViDeform, we train the representational model without refinement and test on the testing of ViDeform. For v VljjyingObjeetss, train both the representational model and the
8

---

Under review as a conference paper at ICLR 2020
refinement CNN the training set, and test the testing set of VIFlyingObjeets and MPI-Sintel datasets. The refinement CNN is to approximate the processing in visual areas V2- V6 that integrates
and refines the motion perception in V1 (Gazzaniga et al., 2002 Lyon & Kaas 2002: Moran &
Desimone 1985
Born & Bradley 2005 Allman & Kass 1975).
Figure displays several examples of the inferred displacement field, learned with non-parametrie M()), using the local mixing motion model (eqn. b where the local support is in a range of 1-4,+-4, and dæ is taken with sub-sampling pre-trained NeowNet 2.0 (lg et al. 2017) model as comparison. In Table we report the average endpoint eπrоr (EPE) of the inferred results. We compare with some baseline methods, such as the FlowNet and its variants (Dosovitskiy et al. 2015] Ilg et al. (017). by obtaining the pre-trained models and testing on the corresponding datasets. Note that those methods train deep and complicated neural networks with large scale datasets to predict optical flows in supervised manners, while our model can be treated simple one-layer auto-encoder network, accompanied by weight matrices representing motions. We achieve competitive results to these methods.
of 2. We also show the inferred results from
I,
Gt
FN2
Ours
L
I
Gt
FN2
Ours
I
I++1
Gt
FN2
Ours
I,
I+
Gt
FN2
Ours f
Figure 4: Examples of inference of displacement each block, from left to right are I,, I.+1. ground truth displacement field and inferred displacement field by pre-trained FlowNet 2.0 model and our learned model respectively. indicates that the results are refined by the refinement CNN. The displacement fields are color coded. See supplementary for the color code (Liu al.) 2010).
on VIDeform, Vbjeeing and MPI-Sintel.
Table 1: Average endpoint error of the inferred displacement. (FN stands for FlowNet)
FN-C.
FN-S
FN-CS
FN-CSS
Ours
Ours Refine
VIDeform
1.324
1.316
0.713
0.629
0.686
0.444
ViFiyingObjects
0.852
0.865
0.362
0.299
0.285
0.442
0.202
MPI-Sintel
0.363
0.410
0.266
0.234
0.146
0.337
0.212

---

Under review as a conference paper at ICLR 2020
5.4 UNSUPERVISED LEARNING
Assume there is dataset of frame sequences, where the ground truth displacement fields are unknown. We can learn the model by the following steps: (1) first we take the frames as static images, deform the images like what we did for VIDeform get image pairs, and learn the model as initialization; (2) then we infer the displacement fields between adjacent pair of frames using the initialized model; (3) using adjacent pair of frames as training data, we alternatively update the model parameters and re-infer displacement fields. In this task, we use the parametric M and infer the displacement field by gradient descent on a weighted sum of Σ (6))) and ||=](=)] At each iteration, we start the inference from the inferred displacement field from the last iteration.
We test the unsupervised learning on MUG Facial Expression dataset (Aifanti et al., 2010). Figure shows some examples of inferred displacement fields by the unsupervised learning. The inference results are reasonable, which capture the motions around eyes, eyebrows, chin or mouth. See supplementary D and H for the learned filters and more inferred examples.
Figure 5: Examples of inferred displacement fields by unsupervised learning. The top row shows the observed image sequences, while the bottom row shows the inferred color coded displacement field (Liu et al. 2010).
We perform ablation studies to analyze the effect of two components of the proposed model: (1) dimensionality sub-vectors; (2) sub-sampling rate. Please refer to supplementary for the details.
6 CONCLUSION
This paper proposes simple representational model that couples vector representations of local image contents and matrix representations of local motions. Unlike existing models for VI that focus on statistical properties of natural images videos, our model serves direct purpose of perception of local motions caused by the relative motions between the agent and the 3D environment. Our model learns Gabor-like units with quadrature phases. We also give biological interpretations of the learned model and connect it to the spatiotemporal energy model. Our model is novel, and it is our hope that it adds to our understanding of motion perception in VI in terms of modeling and inference.
In our future work, we shall study the inference of ego-motion, object motions and 3D depth information based on local pixel displacements by expanding our model. We shall also extend our model to stereo in binocular vision by allowing separate encoding matrices for the pair of input images to the two eyes related by pixel displacements caused by depths.
REFERENCES
1ooundrruth100-for-comm 100 labelled images for object recognition tests, 2006. URL http: 7/tev. fbk. eu aaaaaaaaaassssssseaaaaaeaseaaeee rutt///ttuuuuutttooootoo for-oooofooo
Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a, 2(2):284-299, 1985.
10

---

Under review as a conference paper at ICLR 2020
Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The mug facial expression database. In 1 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1-4. IEEE, 2010.
Allman and JH Kass. The dorsomedial cortical visual area: a third tier area in the occipital lobe of the owl monkey (aotus trivirgatus). Brain research, 100(3):473-477 1975.
Anthony Bell and Terrence J Sejnowski. The independent components of natural scenes are edge filters. Vision research, 37(23):3327-3338 1997.
Richard T Born and David C Bradley. Structure and function of visual area Annu, Rev. Neurosci., 28: 18-119, 2005.
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. A. Fitzgibbon et al. (Eds.) (ed.), European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pp. 61 1-25.. Springer-Verlag. October 2012.
John G Daugman. Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. JOSA A, 2(7):1160-1100)) 1985.
A. Dosovitskiy, P. Fischer, E. 1lg. Häusser, C. Hazırbas, V. Golkov, v.d. Smagt, D. Cremers, and
T. Brox. Flownet: Learning optical flow with convolutional networks. In /EEE International Confer-
ence on Computer Vision (ICCV), 2015. URL http::/ttmmmb.... informatiiiii uni freiburg IB15
Ge/Publ] wwt.iiiatnnnn/11111
Robert C Emerson and Morgan C Huang. Quadrature subunits in directionally selective simple cells: counterphase and drifting grating responses. Visual neuroscience, 14(2):373-33) -3855, 1997.
KH Foster, James P Gaska, M Nagler, and DA Pollen. Spatial and temporal frequency selectivity of neurones in visual cortical areas v1 and v2 of the macaque monkey. The Journal of physiology, 365(1):331-363, 1985.
William Fulton and Joe Harris. Representation theory: a first course, volume 129. Springer Science & Business Media, 2013.
Ruiqi Gao, Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning grid-like units with vector representation of self self-position and matrix representation of self-motion. arXiv preprint 1rXiv:1810.05597, 2018
Michael S Gazzaniga, RB Ivry, and GR Mangun. Cognitive neuroscience, new york: W. w, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
David H Hubel and Torsten N Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 14843)5574-511, 1959.
Aapo Hyvärinen, Jarmo Hurri, and Jaakko Väyrynen. Bubbles: unifying framework for low-level statistical properties of natural image sequences. JOSA A, 2077):127--1252, 2003.
Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent component analysis, volume 46. John Wiley & Sons, 2004.
E. Ilg, N. Mayer, T. Saikia, M. Keuper. A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. URL http:://mm. informatii uuni-fre fre bubrbburg.... de/ iPublications// 2ttp//2211//MMBB11
Naoum P Issa, Christopher Trepel, and Michael P Stryker. Spatial frequency maps in cat visual cortex. Journal of Neuroscience. 20(22):8504-851 2000.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
11

---

Under review as a conference paper at ICLR 2020
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1413-1421, 2015.
Judson P Jones and Larry A Palmer. An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. Journal of neurophysiology. 123 1987.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv:1412.6990 2014.
Michael Lewicki and Bruno Olshausen. Probabilistic framework for the adaptation and compari- son of image codes. JOSA A, 16(7):1587-1601, 1999.
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5)9978-994, 2010.
David C Lyon and Jon H Kaas. Evidence for modified v3 with dorsal and ventral halves in macaque monkeys. Neuron, 33(3):453-46 2002.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4040-4048, 2016.
Jeffrey Moran and Robert Desimone. Selective attention gates visual processing in the extrastriate cortex. Science, 229(4715);782-78, 1985.
Sameer Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). 1996.
Bruno Olshausen. Learning sparse, overcomplete representations of time-varying natural images. In Image Processing. 2003. ICIP 2003. Proceedings. 2003 International Conference on, volume 1, pp. 41. IEEE, 2003.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583)))6 1996.
Bruno A Olshausen and David Field. Sparse coding with overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311-3225 1997.
Bruno A Olshausen and David J Field. How close are understanding v1? Neural computation, 1768):1665-16999 2005.
Alberto Paccanaro and Geoffrey E. Hinton. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions on Knowledge and Data Engineering, 13(2): 232-244, 2001.
Marius Pachitariu and Maneesh Sahani. Visual motion computation in recurrent neural networks. bioRxiv, pp. 099101, 2017.
Daniel A Pollen and Steven F Ronner. Phase relationships between adjacent simple cells in the visual cortex. Science, 212(4501): 1204500):1409-1411 140--1411, 1981.
Martin Rehn and Friedrich T Sommer. A network that uses few active neurones code visual input predicts the diverse shapes of cortical receptive fields. Journal of computational neuroscience, 22 (2):135-146, 2007.
Dario Ringach. Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. Journal of neurophysiology, 88(1):455-44 463, 2002.
Yosef Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp. Andrew J King, and Nicol Harper. Sensory cortex is optimized for prediction of future input. Elife, 7:e31557, 2018.
12

---

Under review as a conference paper at ICLR 2020
J Hans van Hateren and Dan L Ruderman. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society London B: Biological Sciences, 265(1412):2315-233 1998.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
Wulff, D. J. Butler, G. B. Stanley, and M. J. Black. Lessons and insights from creating a synthetic optical flow benchmark. In A. Fusiello al. (Eds.) (ed.), ECCV Workshop on Unsolved Problems in Optical Flow and Stereo Estimation, Part II, LNCS 7584, pp. 168-177. Sprin777-rrn Octoher 2012.
Anthony Zec. Group theory in nutshell for physicists. Princeton University Press, 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding, arXiv preprint
2016.
A IMPLEMENTATION DETAILS
K) from image pairs (I, (8(x)), (aae),,[ee I4+1). The number of sub-vectory K = 40, and the number of units in each sub-vector y ()()) (x)) is 2. We also try other dimensionalities sub-vector, e.g., and See supplementary materials. Each row of the encoding matrix W*k) a filter. The size of the filter is 16 x 16, with a sub-sampling rate of pixels in order to get D We learn the model using stochastic gradient descent implemented by Adam (Kingma & 2014. with learning rate 0.0008.
We learn our model (W(A), (8),
For unsupervised learning in section 5.4 since the image size reduces to 64, we use kernel size 8 with sub-sampling rate of pixels. stage (1) for model initialization, we the range of displacement to 1-3, +3]. Displacements at image border are left out.
B COLOR CODE OF DISPLACEMENT FIELD
Figure 6 shows the color map for the color coded displacement fields used in this paper (Liu et al.) 2010
Figure 6: Color map for the color coded displacement fields. The displacement of every pixel in this map is the vector from the center of the square to this pixel. The center pixel does not move. The range of color is taken according to the maximum length of flows in each displacement field.
C ANIMATION OF LEARNED UNITS: MOVING VI-LIKE UNITS
We have (Kk (0) (2) (w) (1k (OW where each of the encoding matrix Wkk) serves filter. Let W (8) ()) By changing values of 5, move. Figure 7 shows several examples of the animation. Each block shows a certain (k)) animated by fixed 8. Each column shows the units in the same (8). As 8 changes, the orientations of learned units remain the same, while the phases change, and the units belonging to the same sub-vector tend have similar movements.
animate (R)) to make it
13

---

Under review as a conference paper at ICLR 2020
Figure 7: Animation of the learned filters
D LEARNED FILTERS
Figure 8 shows the learned filters under different settings, including learned on VIFlyingObjects, learned with parametric M, learned with local mixing model (eqn [5) and learned unsupervisedly.
VillyingObjects
(b) Parametric M
(c) Local mixing
(d) Unsupervised learning
Figure 8: Filters learned from under different settings: (a) filters learned on ViFlyingObjects with non-parametric M: (b) filters learned with parametric M: (c) filters learned with non-parametric
M and local mixing motion model (model used in section 5.3); (d) filters learned on MUG Facial expression dataset with unsupervised learning (model used in section 5.4).
E MULTI-STEP FRAME ANIMATION
Given the starting frame [1( sequence of displacement fields (x), (x), Vx), we can animate the subsequent multiple frames {I, (z), .... Ir(x)) using the learned model. We use the model with local mixing with the same setting as in section [.3 We introduce a re-encoding process when performing multi-step animation. At time after get the next animated frame I,+1, take as the observed frame at time t + 1, and re-encode it to obtain the latent vector 24+1 at time t 1.
Figure 9 displays several examples of 6-step animation, learned with non panaameetticc version of M on ViDeform and ViFlyingOhjects. The animated frames match the ground truth frames well. As quantitative evaluation, compute the per pixel distance between the predicted frames and observed frames, which 9.032 in the testing dataset for V1Deform and 12.076 for VIFlyingObjects.
Figure 9: Examples of multi-step animation. For each block, the first row shows the ground truth frame sequences, while the second row shows the animated frame sequences.
F FRAME INTERPOLATION
Inspired by the animation and inference results, we show that our model can also perform frame interpolation, by combining the animation and inference together. Specifically, given pair of starting
14

---

Under review as a conference paper at ICLR 2020
frame Iy and end frame Ir, want to derive sequence of frames (Ig, Ii...,, I.., Ir) that changes
smoothly. Let to (z) = WIffa) and (r(x) = WIr[2] for each æ E D. At time step t + 1, like the inference, we can infer displacement field d (z) by
(x;
- M (6. dr)uz (1) (x + dr), VO EAY
(10)
(2)
arg min SE
UT - 214 (2,
4x .V
(11)
Like the animation, we get the animated frame I++1 by decoding (141 ((+1(2)), and then re-encode it obtain the latent vector Ut+1 (z).
The algorithm stops when I, is close enough to Ir (mean pixel enror < 10). Figure 10 shows several examples, learned with non-parametric M on VIDeform and ViFlyingObjects. For 96.0% of the testing pairs, the algorithm can accomplish the frame interpolation within 10 steps. With this algorithm, we are also able to infer displacements larger than the acceptable range of
Figure 10: Examples of frame interpolation, learned with non con-parametric M. For each block, the first frame and frame are given, while the frames between them are interpolated frames.
G ABLATION STUDY
We perform an ablation study to analyze the effect of several components the proposed model. All the models the ablation study are trained with non-parametric (5) on VIDeform.
Dimensionality of sub-vectors. In the experiments, we assume that the number of units in each sub-vector (4) (z) is 2, so that within each vector,, a pair of V1-like patterns are learned. However, show that the dimensionality of sub-vectors does not have to be 2. In figure I1 () we show the learned filters with dimension of sub-vectors equal to 4 and 6. For fair comparison, we fix the total number of units in the whole vector 96, and change the number of units in each sub-vector. Table 2 summarizes the quantitative analysis of the models learned with different dimensionalities of sub-vectors, terms of the performances of multi-step animation and inference of displacement field. As the dimensionality of vectors increases, the error rates of the two tasks decrease first and then increase. Besides, in figure bT)b) we show the learned filters without the assumption of sub-vectors.
Table 2: Quantitative analysis of the models learned with different dimensionalities of sub-vectors.
Sub-vector dim
animation MSE
8.684
8.387
7.486
7.926
8.412
inference EPE
0.554
0.520
0.496
0.500
0.528
Sub-sampling rate. Another factor that may affect the learned model is the sub-sampling rate in order to get D_. In the experiments, we use sub-sampling rate 8, which half of the filter size. We also increase decrease the sub-sampling rate to make the adjacent image patches connected with each other more loosely or tightly. Jumbmarzzsssss the performance learned models with different sub-sampling rates, in terms of multi-step animation and inference of displacement field.
15

---

Under review as a conference paper at ICLR 2020
(b)
Figure 11: (a) Filters learned with higher dimension of sub-vectors. The total number of units in the whole vector is fixed to 96. Each block shows the learned units within the same sub-vectors. (b) Filters learned without sub-vector assumption.
Table 3: Quantitative analysis of the models learned with different sub-sampling rates.
Sub-sampling rate
4
8
16
animation MSE
7.492
8.094
10.808
inference 0.658
0.505
0.565
H UNSUPERVISED LEARNING: MORE RESULTS
See figure 12 for more inference results by unsupervised learning.
Figure 12:
More examples of inferred displacement fields by unsupervised learning. The top shows the observed image soquences, while the bottom row shows the inferred color coded displacement field.
1 REFINEMENT CNN MODEL STRUCTURE
16

---

Under review as a conference paper at ICLR 2020
Table 4: Refinement CNN model structure
3 conv. 8 ReI.U, stride 1 3 conv. 16 ReLU, stride 1
Residual blocks
{
3 3 conv. 16 BN RELU, stride
} 4
3 x 3 conv. 16 BN, stride 1
3 conv. 8 ReLU, stride 1 3 x 3 conv. 2 ReLU,, stride 1
17


Reviews:
[{"review_id": "r1gWl4Ksqr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a model for learning local pixel motions between pairs of frames using local image representations and relative pixel displacements between agents and objects.  The model learned is compared to the ability of the primary visual cortex where adjacent simple cells share quadrature relationships and capture local motion.\n\n\"The representation theory underlies much of modern mathematics and holds the key to the quantum\ntheory (Zee, 2016).\"\nCan the relevance of this claim be elaborated on?\n\n\"Figure 1 illustrates the scheme of representation.\"\nPlease provide more detail here on what is happening in the figure.  The caption and reference here are not informative to what the figure is representing.\n\n\"We obtain the training data by collecting static images for (It) and simulate the\ndisplacement field ...  We refer to this method as self-supervised learning\"\nThis is not self-supervised learning.  In self-supervised learning the training label/signal is generated by the system.  In this case artificial data is being generated as the displacement between images is sampled.\n\nSince the motion between images is artificially generated what guarantees are there that the model is learning to capture realistic motion behavior?  Why not use adjacent video frames?\n\n\"Note that those methods train deep and complicated neural networks with large scale datasets to\npredict optical flows in supervised manners, while our model can be treated as a simple one-layer\nnetwork, accompanied by weight matrices representing motions.\"\nIs there a comparison on execution times of the different approaches?\n\n\"by obtaining the pre-trained models and testing on V1Deform testing data\"\nIs this a fair comparison if the proposed approach was trained on V1Deform training data and the comparison methods were not.  A more appropriate comparison would be to apply all the methods to infer the displacement fields between video frames which is also a more natural application.  This can be controlled to contain small motions if needed.  Why nt use the MUG dataset here?\n\n\"Displacements at image border are leaved out\" -> left out\n\nSections 5.4, 5.5 and 5.6 show only qualitative results with no comparison methods.  Can the authors provide reasons that other methods could not be used for evaluation?\n\nI am not sure I understand the motivation for the approach.  Why do we need this over other methods that can better capture larger motions.  This needs to be more clear from the introduction.  Why do we care if the approach captures aspects of V1 for the tasks presented?\n\nThe work is sensible and the approach is clear but I found the evaluation and motivation lacking in key areas that I mention above.  The authors should revise and make it clear to the reader why we should care about this problem.  Aligning with V1 is interesting but it does not come into play in the applications of the approach or the analysis so I am not sure why I should care.  The evaluation also needs to be much more convincing before I could recommend acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyghDmnsqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The hypothesis in this paper is that the primary purpose of the cells of the V1 cortex is to perceive motions and predict changes in the local image contents. The authors show that by learning from image pairs from a continuous sequence, both V1-like features and motion operators can be learned. I found the hypothesis and formulation reasonable, the numerical results are supportive, it's actually interesting to see that the proposed model's motion prediction outperforms the other dedicated models. Further, the authors used inference to infer the motion during learning, I think this is quite a novel topic to work on. Overall, this makes a good submission.\n\nHere are some issues could be addressed further:\n\n1. Section 3.3 introduces subvectors. This implicitly introduces an independence assumption when combined with a motion operator. Then in section 5, the authors studied the dimensionality of subvectors. If the subspaces are assumed to be 2, then this independence regularization is quite strong. This may not support the authors' claim that the prediction of motion is enough to achieve V1-like features and I tend to conclude the V1-like receptive fields come from the implicit independence constraint. I'd suggest an additional ablation experiment to verify the impact of the subspace assumption. \n\n2. To model the motion,  we can directly use lie operators, the authors may want to discuss the connection between the suggested method and the Lie group approach. \n\n3. I found some minor issues, e.g.:\n       3.1 In Section 3.2 it's normalized tight frame (Parseval frame).\n       3.2 In Equation 2 I understand it's a deconvolution, however, the notation is still not ideal.\n       3.3 In the section paragraph of Section 3.2,  'the representation has the isometry property' and 'the vector representation also preserves the angle' should be switched?\n       3.4 small typos like 'mortar cortex' -> 'motor cortex'.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJecGU-oKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a representation model for describing local pixel displacement. The proposed model uses matrix multiplication for optical flow estimation, where an image is transformed into a vector and the local motion is modeled by a matrix. \nThe recommendation of this work is based on the following reasons. First, the motivation of the proposed method is not convincing. While the proposed ideas are interesting, it is not clear why this approach sheds light on our understanding of motion perception. Is there any psychological evidence to support the proposed model? Or the authors simply take some ideas form V1 model and add a module to “explain” motion?  Second, the experimental results are not sufficient to demonstrate the effectiveness of the proposed model.\nMajor issues:\nFirst, while it is interesting to use matrix multiplication to model motion, it is not clear why the motion between patches I_t[x] and I_{t+1}[x] can be approximated with linear transformation (Section 3.4). Furthermore, it is not clear why the transformation M only depends on the displacement of the center pixel whereas different pixels in a patch I_t[x] could have different displacements. \nSecond, the proposed model for optical flow estimation is only evaluated on the proposed V1Deform dataset. If the authors position this paper “may shed light on our motion perception in primary visual cortex”, the authors certainly need to carry out sufficient psychophysical experiments. \nMinor issues:\nFirst, Eq. 2 does not seem correct to me. The left and right sides of Eq. 2 have different dimensions.\nSecond, the authors may consider using {} instead of () to define a set of pixels or vectors in Section 3.1.\nThird, while the reconstruction loss (Eq. 7) is used in this paper, I wonder what the results would be like if the authors simply enforce W’W=I instead.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews. On one hand, there is interesting novelty in relation to biological vision systems. On the other hand, there are some serious experimental issues with the machine learning model. While reviewers initially raised concerns about the motivation of the work, the rebuttal addressed those concerns. However, concerns about experiments remained. 
Decision: Reject