Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The hypothesis in this paper is that the primary purpose of the cells of the V1 cortex is to perceive motions and predict changes in the local image contents. The authors show that by learning from image pairs from a continuous sequence, both V1-like features and motion operators can be learned. I found the hypothesis and formulation reasonable, the numerical results are supportive, it's actually interesting to see that the proposed model's motion prediction outperforms the other dedicated models. Further, the authors used inference to infer the motion during learning, I think this is quite a novel topic to work on. Overall, this makes a good submission.

Here are some issues could be addressed further:

1. Section 3.3 introduces subvectors. This implicitly introduces an independence assumption when combined with a motion operator. Then in section 5, the authors studied the dimensionality of subvectors. If the subspaces are assumed to be 2, then this independence regularization is quite strong. This may not support the authors' claim that the prediction of motion is enough to achieve V1-like features and I tend to conclude the V1-like receptive fields come from the implicit independence constraint. I'd suggest an additional ablation experiment to verify the impact of the subspace assumption. 

2. To model the motion,  we can directly use lie operators, the authors may want to discuss the connection between the suggested method and the Lie group approach. 

3. I found some minor issues, e.g.:
       3.1 In Section 3.2 it's normalized tight frame (Parseval frame).
       3.2 In Equation 2 I understand it's a deconvolution, however, the notation is still not ideal.
       3.3 In the section paragraph of Section 3.2,  'the representation has the isometry property' and 'the vector representation also preserves the angle' should be switched?
       3.4 small typos like 'mortar cortex' -> 'motor cortex'.
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: The authors propose a model for learning local pixel motions between pairs of frames using local image representations and relative pixel displacements between agents and objects.  The model learned is compared to the ability of the primary visual cortex where adjacent simple cells share quadrature relationships and capture local motion.

"The representation theory underlies much of modern mathematics and holds the key to the quantum
theory (Zee, 2016)."
Can the relevance of this claim be elaborated on?

"Figure 1 illustrates the scheme of representation."
Please provide more detail here on what is happening in the figure.  The caption and reference here are not informative to what the figure is representing.

"We obtain the training data by collecting static images for (It) and simulate the
displacement field ...  We refer to this method as self-supervised learning"
This is not self-supervised learning.  In self-supervised learning the training label/signal is generated by the system.  In this case artificial data is being generated as the displacement between images is sampled.

Since the motion between images is artificially generated what guarantees are there that the model is learning to capture realistic motion behavior?  Why not use adjacent video frames?

"Note that those methods train deep and complicated neural networks with large scale datasets to
predict optical flows in supervised manners, while our model can be treated as a simple one-layer
network, accompanied by weight matrices representing motions."
Is there a comparison on execution times of the different approaches?

"by obtaining the pre-trained models and testing on V1Deform testing data"
Is this a fair comparison if the proposed approach was trained on V1Deform training data and the comparison methods were not.  A more appropriate comparison would be to apply all the methods to infer the displacement fields between video frames which is also a more natural application.  This can be controlled to contain small motions if needed.  Why nt use the MUG dataset here?

"Displacements at image border are leaved out" -> left out

Sections 5.4, 5.5 and 5.6 show only qualitative results with no comparison methods.  Can the authors provide reasons that other methods could not be used for evaluation?

I am not sure I understand the motivation for the approach.  Why do we need this over other methods that can better capture larger motions.  This needs to be more clear from the introduction.  Why do we care if the approach captures aspects of V1 for the tasks presented?

The work is sensible and the approach is clear but I found the evaluation and motivation lacking in key areas that I mention above.  The authors should revise and make it clear to the reader why we should care about this problem.  Aligning with V1 is interesting but it does not come into play in the applications of the approach or the analysis so I am not sure why I should care.  The evaluation also needs to be much more convincing before I could recommend acceptance.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a representation model for describing local pixel displacement. The proposed model uses matrix multiplication for optical flow estimation, where an image is transformed into a vector and the local motion is modeled by a matrix. 
The recommendation of this work is based on the following reasons. First, the motivation of the proposed method is not convincing. While the proposed ideas are interesting, it is not clear why this approach sheds light on our understanding of motion perception. Is there any psychological evidence to support the proposed model? Or the authors simply take some ideas form V1 model and add a module to “explain” motion?  Second, the experimental results are not sufficient to demonstrate the effectiveness of the proposed model.
Major issues:
First, while it is interesting to use matrix multiplication to model motion, it is not clear why the motion between patches I_t[x] and I_{t+1}[x] can be approximated with linear transformation (Section 3.4). Furthermore, it is not clear why the transformation M only depends on the displacement of the center pixel whereas different pixels in a patch I_t[x] could have different displacements. 
Second, the proposed model for optical flow estimation is only evaluated on the proposed V1Deform dataset. If the authors position this paper “may shed light on our motion perception in primary visual cortex”, the authors certainly need to carry out sufficient psychophysical experiments. 
Minor issues:
First, Eq. 2 does not seem correct to me. The left and right sides of Eq. 2 have different dimensions.
Second, the authors may consider using {} instead of () to define a set of pixels or vectors in Section 3.1.
Third, while the reconstruction loss (Eq. 7) is used in this paper, I wonder what the results would be like if the authors simply enforce W’W=I instead.

review_title: Official Blind Review #1

Meta-review: The paper received mixed reviews. On one hand, there is interesting novelty in relation to biological vision systems. On the other hand, there are some serious experimental issues with the machine learning model. While reviewers initially raised concerns about the motivation of the work, the rebuttal addressed those concerns. However, concerns about experiments remained. 
Decision: Reject