Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new way to predict video frames inspired by how rigid objects translate in the Fourier domain, where translation becomes angular progression in a straight line (in polar coordinates). The model simulates (as I understand it) the fourier transform using a particular non-linearity that projects the data into a space similar to the fourier domain, transforms it, and then maps it back. The method is compared to a number of other methods and shows superior performance in terms of MSE and SSIM scores. One version uses orders of magnitude fewer parameters and still outperforms the deep learning baseline methods in terms of MSE. On the other hand, it just barely outperforms a 2 parameter method in terms of SSIM. 

strength_and_weaknesses: Strengths:

+ The inspiration for this method is good.

+ I am not an expert in this area, but it appears to be a completely novel method.

+ The performance is good compared to the baseline methods.

+ The writing is very clear.

Weaknesses, with concrete, actionable feedback

- It is hard to evaluate the qualitative results in Figures 5-8.

clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is clear, but not being an expert in this area (and not being a EE guy, Fourier transforms are still somewhat mysterious to me!), I'm sure I didn't understand parts of the paper.

Quality: The quality of the results appear to be good. However, not being familiar with the area, I don't know if the systems used as baselines are state of the art. Hopefully other reviewers will know. 

Novelty: As far as I know, this is completely novel.

Reproducibility: I assume they will share their code if the paper is accepted.

Typos, wording suggestions:

p1: "For example, in cases of expanding or rotating motion, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi-transparent surfaces (e.g., viewing the world through a dirty pane of glass)." This sentence no verb.

p4: equation in 4th line from the bottom: In order for the indexes of the y variables to take on values from 1:64, if k is 1:32, then the subscripts for the two instances of y here should be 2k-1 and 2k. 

p6, 6 lines up from section 3: You say elsewhere (second-to-last sentence in the bottom full paragraph on page 5 - or is this only for the cMC method?) that you will use frames t-1 and t-2 to predict frame t+1, if I understood this correctly. In that case, with 11 frames, you can only predict 8 frames, not 9. But I guess from elsewhere in the paper that this only applies to cMC, so never mind if so!

Section 3.1: 1st pp, second sentence. I don't understand this sentence - is there a noun missing at the end? half of what?

Same paragraph, 3rd line from the bottom: auto-encoder -> autoencoder

3rd line from the bottom: It would help a bit to note that difficulty increases from left to right in the plots in Figure 3.

Figures 5-8: What is the first image in the second row of these figures? Is that just using x(t) as the prediction? of x(t+1)? Also, in Figure 5, I can't tell what this is a video of. Perhaps a bit of explanation would help? Also, in Figure 6, it looks like you are going backwards in time rather than forwards - the cyclist seems to be moving backwards. The same comment applies to Figure 7 - it looks like the person is moving backwards?
summary_of_the_review: To the extent that my knowledge about this area is limited to what is in the paper, this is a novel approach to video prediction, and performs well against the baselines. However, I recall a paper or talk by Yann LeCun and NeurIPS a few years ago that gave very good predictions of video frames by assuming a latent variable that somehow influenced the prediction and "chose" one future out of the possible ones. The predictions he showed were very clear, but I'm not sure how that relates to this work. Also, the straight convolutional network predictor approach is from 2016; surely more has happened in this area in the last 6 years?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a self-supervised learning objective based on next-frame prediction that uses angular extrapolation in polar coordinates while keeping amplitudes constant. They show that this inductive bias of using angular instead of linear extrapolation improves next-frame prediction performance in terms of MSE and SSIM on video snippets taken from the DAVIS dataset.

strength_and_weaknesses: ### Strengths

 + Conceptually well motivated idea
 + Paper is well written and easy to read


### Weaknesses

 1. Only small dataset and shallow architecture
 1. Unclear how robust improvements are


### 1. Small dataset and shallow architecture

It is great to see that the well-motivated inductive bias helps in the relatively data-limited regime the authors study. However, as video data is abundant and easy to obtain in masses, it is not clear to me how relevant such an inductive bias is. Would the advantage also hold on a much larger dataset with a much deeper network architecture?

The fact that the linear encoder/decoder works so well suggests to me that not much of relevance can be learned from such a small dataset. Presumably a deep architecture trained on a large dataset would learn something about objects and their 3d motion patterns and be able to outperform a linear encoder/decoder by a large margin.


### 2. Robustness of improvements

Related to the previous point that the linear method is so good, I think it would be important to provide error bars on Table 1. How confident are we that the differences between the methods are indeed significant? How much variability is there between multiple differently initialized models? What does it tell us that SSIM is more or less the same across cMC, PP and deepPP?

clarity,_quality,_novelty_and_reproducibility: Overall I think it is a nice and novel idea that is well motivated and well explained. The paper is well written and easy to follow, but the experiments are somewhat underwhelming due to the points discussed above.

The results can probably be reproduced relatively easily as the experiments are relatively straightforward and well explained. Unfortunately I could not find a statement about availability of code.

summary_of_the_review: Nice and simple paper, but somewhat weak and shallow on the experimental side.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Inspired by the Fourier Shift Theorem, the paper tries to learn a representation space where the temporal evolution is equivalent to a matrix multiplication by a dynamics matrix A(t). This dynamics matrix is learned by minimizing the one step prediction error in an end to end fashion. More specifically, a frame x(t) is encoded into representation z, where it is advanced through the dynamics matrix and then decoded back. The decoded should be same as the next frame x(t+1). Through experiments on DAVIS dataset, the paper shows that this kind of representation is powerful and outperforms a couple of baselines like convolutional network based prediction and causal motion compensation.
strength_and_weaknesses: I appreciate that the paper tries to develop a simple, lightweight and interpretable method and provides insights on the algorithm. Some of the ideas in the paper like learning an appropriate representation is really powerful is an important message.

However, the paper needs strengthening in several fronts. The first and the most important question is a theoretical and a conceptual one. Even though the inspiration from Fourier shift theorem is interesting, it is hard to justify the method based just on that since in real videos, frame to frame transformation is rarely just a translation. Second, even though conceptually the paper discusses Fourier transform, the implementation requires a function (W or g) to be learned which could very well be different from the Fourier transform. However, this limitation can be easily mitigated if the authors choose to develop their work from the perspective of Koopman embedding [1,3,4]. Koopman operator theory is precisely the generalization that authors are looking for. Koopman embedding is the representation space where the complicated nonlinear dynamics are represented as linear dynamics. The dynamics matrix A discussed in the paper could actually be thought as the Koopman operator. 

Interestingly, there already exist a couple of papers that try to use Koopman embedding in video prediction. For example, Comas et al [2] is conceptually doing something very similar to this paper, although they are separating objects from the video and applying linear dynamics to the individual objects in some representation space.

Second important front where the paper needs improvement is its experiments or empirical validation. Although some of the insights in the paper are interesting, they need to be validated in a couple of dataset to support the calim that those results and improvements are indeed general. At the moment, the paper only validates results in one dataset.

Third important improvement could come from comparison with stronger deep learning methods in video prediction. It is interesting that the method beats vanilla CNN or deep network without nonlinearity, but there are other specialized works for the purpose of video prediction (e.g. DDPAE [5]). Comparison to those will tell us where the method stands. 

References:
1. Bethany Lusch, J. N. Kutz, and S. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9, 2018.
2. Comas, A., Ghimire, S., Li, H., Sznaier, M. and Camps, O., 2021. Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspective. arXiv preprint arXiv:2110.00547.
3. Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning (ICML), 2020
4. Jeremy Morton, F. Witherden, and Mykel J. Kochenderfer. Deep variational koopman models: Inferring koopman observations for uncertainty-aware dynamics modeling and control. In International Joint Conferences on Artificial Intelligence (IJCAI), 2019
5. Hsieh, Jun-Ting, Bingbin Liu, De-An Huang, Li F. Fei-Fei, and Juan Carlos Niebles. "Learning to decompose and disentangle representations for video prediction." Advances in neural information processing systems 31 (2018).
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The idea of the paper is new in the sense that it is trying to develop a new dynamic model that is lightweight and leverages the ideas from signal processing to learn appropriate representation. However, the paper needs some strong improvements in a couple of fronts to be ready for publication. The paper contains some information about reproducibility, although that could also be improved. 
summary_of_the_review: I liked the fresh perspective presented in the paper. However, the paper has some rooms for improvement as discussed earlier. Hence, I am on the side of rejection at the moment.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a novel approach to next-frame prediction  in video sequences using natural polar coordinates. Compared to a linear approach, angular extrapolation in polar coordinates achieves higher accuracy on video snippets taken from the DAVIS dataset. There was consensus that the proposed framework was interesting but one of the reviewers mentioned a possible connection to Koopman operators which the authors acknowledged in the rebuttal but failed to adequately address. Additional concerns included the fact that the linear baseline performed so well and this point was not adequately addressed in the the rebuttal.

justification_for_why_not_higher_score: No real support for acceptance by any of the reviewers. 
justification_for_why_not_lower_score: NA
Decision: Reject