Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the problem of learning symmetry from sequential data with a certain stationarity property. In particular, from time sequences of length at least three, meta-learning is used to learn representations such that a future observation can be predicted well by a linear transition. It is shown that doing so can tease out the disentangled structure by simultaneous block-diagonalization. Each block then corresponds to a disentangled feature. 

The meta-learning framework is very straightforward: We learn an injective mapping \phi for each time step, such that a linear transition relates time step t to time step t+1. The loss function involves a decoder that maps the linear transition \phi_t-1 to \phi_t. The transition can be parameterized in various complex ways, but as mentioned above only a linear map is used. The inner loop consists of minimizing the prediction loss, while the outer loop updates the encoder. The main assumption is that the velocity/acceleration is preserved within each observation. Experiments on Sequential MNIST, 3D Shapes, and SmallNORB show the efficacy of the proposed method. 



strengths_and_weaknesses: Strengths:
- The basic idea of the paper is quite simple: That we can learn some underlying algebraic structure in an unsupervised way when the transitions can be represented linearly in some latent space. It follows the line of work on Slow Feature Analaysis, ICA and more recent work by Keller and Welling, and also seems reminiscent of work on Koopman analysis (although multiple dynamics are considered here). The paper proposes to use meta-learning to predict a future observation and model only linear transitions. The meta-learning setup is also quite simple. 
- The experiments are carried out on reasonable datasets: Sequential MNIST, 3D Shapes and SmallNorb. The overall results and quality of ablation is good. 
- The extension to constant acceleration is also reasonable, as are the discussion of limitations. 

Weaknesses: 
- The paper is actually quite hard to read. I think it can be made really good with a smoother presentation. Even though the main idea is quite simple, I often had difficulty following the details and trying to decode what the authors might have meant. 
- The experiments are good in the sense that they cover some good datasets, and have good ablations. However, I have difficulty in placing their significance, given the authors only report measures such as equivariance error. Can we have another downstream task in which such learnt representations are shown to be helpful? If not, is it possible to have some reasonable baselines to get a sense of how good the variants proposed here are? Overall I like the main idea -- it is simple and elegant, but I am not quite sure what to make of the results, and I think the paper could do a better job at presenting them. 
questions: See above: 

Minor comments:

There are quite a few typos and somewhat awkward sentence constructions. Below are a few examples:
- Typo, line 16: *in machine learning? 
- Line 20: should be "has succeeded"
- Line 34-35: "in the way of meta-learning" -> by meta-learning/by the way of meta-learning/by means of meta-learning
- Line 44-45: "There are rich literatures in unsupervised/weakly supervised" -> There is a rich literature...
- Line 63: "Many studies impose algebraic constraints that reflects some form of geometrical assumption." -> that reflect some form of..
- Line 86: identifiebility -> identifiability 


limitations: Yes, adequately addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to discover structural properties of the data by solving a sequential prediction tasks.
The idea is that solving the prediction tasks produces equivariant models, which by guarantees from representation theory, means that the learned representations are simultaneously block-diagonalized, with blocks corresponding to disentangled factors.

Specifically, the paper learns an encoder $\Phi$, a decoder $\Psi$, and a matrix-valued function $M$, such that given adjacent observations $s_t, s_{t+1}$, the difference $\||\Psi (M\Phi(s_t)) - s_{t+1}\||_2^2$ is minimized. The training borrows idea from meta learning, where the algorithm first estimates $M$ based on a part of the sequence (denoted $\mathbf{s}_c$), and $\Phi$ is then estimated using this $M$ and the rest of the sequence (denoted $\mathbf{s}_p$).

Empirical results show that 1) the proposed method is able to train on only adjacent time frames and extrapolate to unseen steps in the future, and that 2) the learned linear transitions can be simultaneously block-diagonalized, with blocks corresponding to 1 factor of variation.
strengths_and_weaknesses: Strengths
- It's an interesting idea to learn disentangled representations by the simple prediction task.
- The paper presents a variety of empirical results and is clear on implementation details.
- The paper provides hypotheses on some empirical results (Sec 6) and is open about the limitations of the proposed methods, e.g. failing to work on non-invertible transitions.

Concerns
- [Clarified in the response] While inductive biases are necessary as established by prior work, the proposed method seems too tailored to the specific data structure, and it seems unfair to compare with baseline models that do not have such knowledge about the data.
- [Draft updated] The messages from some empirical results could be made clearer; please see detailed questions below.
- [Clarified in the response] Please discuss the relation to the line of work about incorporating physics into the prediction model, e.g. PhyDNet (Guen & Thome 20), and disentangled representation learning in videos in general.

=== **Update** ===

The authors' responses have address most of my concerns. Many of my previous questions were because I didn't get the main point the paper, since I had trouble parsing the paper as I mentioned in comments on clarity.
The writing is now improved and my review has been updated accordingly.

questions: Questions:
- The choice of $M$ seems to rely on knowledge of the true dynamics. How to obtain such knowledge in general? How robust is the method to "model mismatch", e.g. when internal optimization is performed on sequences with noises or with varying velocity? How can the model be extended to handle some amount of model mismatch?
- Would the choice of $T_c, T_p$ affect the performance? 
    - Choosing $T_c=2$ should be sufficient if the data perfectly follows the model assumption (e.g. constant velocity), but this may not be the case when there's model mismatch.
- Would the proposed method be able to handle multiple objects?
- Sec 5.2, swapping $M^*$: my understanding is that $M^*$ should depend on the sequence-specific action $g$. However in Fig 4(a), the two sequences clearly have different actions (e.g. the bottom row seems to be doing both color rotation and shape rotation, while the top row only has shape rotation) and different orbits (since the digits are not the same). How could $M^*$ be the same then?
- Fig 8 in the appendix: line 255 in the main text says that the proposed method gives "better representations" but it's unclear how so. Please explain the success/failure modes of the proposed method and the reasons.
- Line 276, about simultaneously block-diagonalizable: please provide reference for this.


Clarifications:
- Line 124: is $\circ_{latent}$ different from $\circ$?
- Line 220: "digit4 only training": does it mean that the encoder/decoder have not seen other digits during training?
- Line 233: what's the reason for splitting $H$ into 8 (and not some other number of) subtensors?
- Line 250: what does "fixed block structure" mean? Does dividing $H$ into 8 subtensors count at fixing the block structure?
- Line 303: what does "sufficiently large" mean for $m$?


Writing: overall I find the paper a bit hard to parse, possibly due some inconsistent notations and my own in-familiarity to the topic.
Some concrete things:
- Please fix citation [13].
- Line 115: do you mean members of $S$ (rather than $s$, which is itself a member of $S$)?
- Line 143: maybe consider a different notion than $M(g, s_1)$, since $M$ has been defined to be a single-argument function from $\mathcal{G}$ to $\mathbb{R}^{a \times a}$.
- Line 179: should it be "optimized on $s_p$"?
- Inconsistent notations:
    - $H_{+1} H_{+0}^\dagger$ in eq (4) and $H_2H_1^\dagger$ in Fig 1.
    - $T_p$ on line 227, $t_p$ on line 245.
- Line 233 & 234: it would be more consistent to use either 256 or $m$ (but not both) in defining $H, H_k$.
- The paragraph on line 252: it would be better to briefly explain what "better representation" means.
- Line 289: there seems to be a typo in the definition of $\hat{M}^*$.

limitations: The paper discusses the limitations of the proposed method, such as failing to work with non-invertible transitions, e.g. ShapeNet dataset.

The paper also discusses potential societal impact if the method were to employed in real-world sequential prediction problems.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors of this paper investigate the time series models with certain stationary properties, and use the learned symmetry structure for predicting the future in time series.  They consider the type of time series where each sequence is generated by initial observation and some fixed transition operator. Then they propose to optimize a homeomorphic function with equivariance property w.r.t. a group of transition operators. In the experiments, they evaluate the proposed approach on sequences generated based on 3 image datasets.The results show that their method can successfully learn transition operators such object rotation, hue rotation, and object translation. By applying these operators to the initial time step, they show that they could predict the future sequence for a few number of time steps.
strengths_and_weaknesses: Originality: In my opinion, this is an interesting paper because they try to investigate the equivariance property in stationary time series models. From this perspective this idea is novel, though I believe the problem defined in this paper can be alternatively tackled using existing methods on disentangled representation learning. (I will think of this weakness more as insignificance rather than originality, because in general studying equivariance in time series is a great topic, and it is not reasonable to argue about originality merely because some existing methods can solve the same type of problems.)

Significance: Regarding disentangled representation learning. I mainly wonder what we additionally achieved with this proposed method, compared to the existing methods on disentangled representation learning. The reason is that, the authors specifically consider time series with fixed transition operators, e.g. rotation, translation, and the goal is to predict the future observations in the sequences. Because of the strong assumption on fixed transition operators, these prediction tasks, in my opinion, can also be framed as generating examples by varying factors of variation using models in [1,2,3]. 
While the authors have explained how their work differs from the prior work on disentangled representation learning, I would appreciate it if they do some comparison against some of those methods such as [1,2,3].For both learning disentangled representations and generation tasks, those VAE-based methods will also achieve impressive results in datasets such as MNIST, FashionMNIST, Dsprites. On the other hand, this proposed method can only do point-estimation, while those VAE-based models can characterize the prediction uncertainty in a variational manner. This is my main concern about the significance of this work.

Quality: This paper is well structured in the sense that they provide proof of theory that supports their motivation and sufficient empirical evaluations as well. 

Clarity: This paper is well written and clearly conveys the idea in the paper.

[1] Chen, Ricky TQ, et al. "Isolating sources of disentanglement in variational autoencoders." Advances in neural information processing systems 31 (2018).

[2] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International Conference on Machine Learning. PMLR, 2018.

[3] Esmaeili, Babak, et al. "Structured disentangled representations." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

questions: I have some questions:

- Can you motivate the ablation study in the experiments? I don’t understand why you are doing it. 

- Regarding the comparison against SimCLR and CPC, I find it a bit hard to understand the experiments and interpret the results. Can you please provide more details?

- With noisy background in the image examples, do you find the optimization more difficult? Intuitively it adds challenges to learn the representations for the objects there, but I wonder how harder it becomes in your case, since you assume a purely static background.

General suggestion:

- I think it is interesting to study the equivariance property in time series in general. On the other hand, I feel it is also important to showcase the robustness / generalization when tackling trajectory prediction tasks. In other words, the point estimations are not that great in many use cases, and we would want to characterize the model uncertainty. So I wonder if you could do some reasoning about the prediction uncertainty in this proposed method.

limitations: To my best knowledge, the authors have comprehensively discussed the limitations of their work and the potential negative impact.
review_title: null

Meta-review: metareview: 
While there was a certain lack of enthusiasm in the scores of the reviewers, the author's answers cleared the concerns of the reviewers participating in the discussion and overall the recommendation leans towards acceptance. This paper is, in the reviewers' opinions, sound and adds to the literature on unsupervised learning of symmetry. The formulation (of learning symmetry by only modelling linear transitions) is nicely simple. Experiments and evaluations generally were considered of adequate quality.


confidence: Less certain
Decision: Accept