Unsupervised Learning of Equivariant Structure from Sequences
Authors: Takeru Miyato
Masanori Koyama
Kenji Fukumizu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Disentanglement
Symmetry
Unsupervised Learning
Equivariance

Abstract: In this study, we present \textit{meta-sequential prediction} (MSP), an unsupervised framework to learn the symmetry from the time sequence of length at least three. 
Our method leverages the stationary property~(e.g. constant velocity, constant acceleration) of the time sequence to learn the underlying equivariant structure of the dataset by simply training the encoder-decoder model to be able to predict the future observations. 
We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product by applying \textit{simultaneous block-diagonalization} to the transition operators in the latent space, the procedure which is commonly used in representation theory to decompose the feature-space based on the type of response to group actions.
We will showcase our method from both empirical and theoretical perspectives.
Our result suggests that finding a simple structured relation and learning a model with extrapolation capability are two sides of the same coin. The code is available at https://github.com/takerum/meta_sequential_prediction.

Unsupervised Learning of Equivariant Structure from Sequences
Takeru Miyato 1.2 Masanori Koyama" " Kenji 3ukumizu eqqul contribution
Preferred Networks, Inc. University of Tübingen 'hhe Institute of Statistical Mathematics
Abstract
In this study, present meta-sequential prediction (MSP), an unsupervised framework to learn the symmetry from the time sequence of length at least three. Our method leverages the stationary property (e.g. constant velocity, constant ac- celeration) of the time sequence to learn the underlying equivariant structure of the dataset by simply training the encoder-decoder model to be able to predict the future observations. We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product by apply- ing simultaneous bacckdiagonalization to the transition operators in the latent space, the procedure which is commonly used in representation theory to decom- pose the feature-space based the type of response to group actions. We will showcase our method from both empirical and theoretical perspectives. Our re- sult suggests that finding simple structured relation and learning model with extrapolation capability are two sides of the same coin. The code is available at bttps://githan com/ http/takerenn wackerum/nnee met sequential prediction
Introduction
The recent evolution and successes of neural networks in machine learning fields have shown the importance of symmetry-aware neural network models [18, 45, 38, 63]. particular, symmetries in the form of geometric/algebral constraints have been proven useful in various applications involy- ing high-dimensional, highly-structure observations. For example, recent literature of roboties and reinforcement learning has succeeded in exploiting the knowledge of geometrical symmetries to im- prove the sample efficiency [62, 65] or to train a model that generalizes to unseen observations [58].
However, building an inductive bias that matches the given dataset of interest is challenging, and re- cent studies have been exploring the ways to learn symmetries itself from observational sequences. Many of these approaches consider settings with relatively restrictive assumptions or weak supervi- sion. For example. [56] allows the trainer use the knowledge of the identities of the actions used in making the transition. Meanwhile, [4, 35, 34] essentially assume that the transition velocity of all sequences the datasets are the same.
These studies indicate that there is still much room left for the question of "what required for dataset/model enable the unsupervised learning of the equivariance relation corresponding the underlying symmetry". This paper advances this investigation by showing that if the sequential dataset consists of time series with certain stationary property (constant velocity, constant accel- eration). we can learn the underlying symmetries by simply training the model to be able to predict the future with linear transition in the latent space. Our theory in Section 3 shows that this strat- egy can learn model that is almost equivariant. Moreover, we will experimentally show that, by training an encoder-decoder model in framework meta-learning which we call meta-sequential prediction (MSP), we can actually learn an equivariant model. In particular, we show that we can
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

learn hidden equivariance structure in the dataset by splitting (1) the internal training step to com- pute the prediction loss of linear transition in the latent space from (2) the external training step to update the encoder and decoder. We will also empirically show that, in alignment with group rep- resentation theory [42], the learned linear latent transitions in our framework can be simultaneously block-diagonalized, and that each block corresponds to a disentangled factor of variation.
Related works
Recently, numerous studies have explored the ways to learn symmetry in data-driven manner. There is rich literature in unsupervised/weakly supervised approaches that use sequential datasets to exploit the structure that shared across time, and they all differ by the types of inductive bias. For example, the object-centric approach introduces inductive bias in the form of architecture, and equips the model with pre-defined slots to be allocated to objects [39, 33, 40J. Meanwhile, [22. I] assumes that the symmetry to be found takes the form of the energy conservation law, and learns each variable in the law as a function of observations. While this approach assumes that some energy is preserved in each observation, we assume that the transition parameters like velocity and accel- eration are preserved within each observation. Other more indirect forms of inductive bias include those relevant to distributional sparseness and symmetry defined through algebraic constraints. [31] for instance assumed that every stationary component of a given time series is generated by a finite and independent latent time series. [41] proposed sparsely model the transition with a distribu- tion of large kurtosis. Our work belongs to a family of unsupervised learning that seeks to find the underlying symmetry of the dataset based on an algebraic inductive bias that the transitions can be represented linearly in some latent space. In this sense, our inductive bias also has a connection to Koopman operator [43, 25, 3]. We are different from these studies in that we are aiming learn common encoding function (i.e. lifting function) under which the set of sequences following differ- ent dynamics can be described linearly. Also [70] applied Koopman operator on pedestrian walking sequences, and [23] used Koopman operator to separate the foreground from the background. How- ever, [70] does not set out their model to solve the extrapolation, and neither [70] nor [23] discusses the natural algebraic decomposition of the latent space that results solely from the objective to pre- dict the unseen future.
Unsupervised learning with algebraic/geometrical constraints Many studies impose algebraic constraints that reflect some form of geometrical assumptions. [16] uses a known coordinate map parametrization of Lie group family to construct a posterior distribution on the manifold. [54] assumes that the observations are dense enough on the data-manifold to describe its tangent space, and exploits a property of random walks on the product manifold to decompose the data space. In the analysis of sequential datasets, [13, 59, 10, 56, 12] make some Lie group type assumptions about the transition. [56] also assumes that the identity of the actions in the sequences is known. As for the approaches with less explicitly geometrical touch, [35] uses capsule structure in their probabilistic framework to model a finitely cyclic structure while retaining the computability of posterior distri- bution. By design, [35] assumes that all sequences in the dataset transition with the same cyclic velocity. [69] enforces the underlying transition action to be commutative. Some of these studies learn the representation
that the linear transition in the latent space can be explicitly computed [56, 12, 4]. In particular, [4] presents a theory that suggests that a representation without this feature would have topological defects, such as discontinuity. Our approach shares a similar philosophy with these works except that, instead of imposing strong assumption about the underlying symmetry, we only make a relatively weak stationarity assumption about the dataset; although we assume each sequence to be transitioning with constant velocity/acceleration, we allow the velocity/acceleration to vary across different seeeeteeeeeeeeee sequences.
Disentangled Representation Learning Disentangled structure [29, 30] is a form of symmetry that has also been actively studied. It is known that, under the i.i.d assumption of examples, unsu- pervised learning of disentanglement representation is not achievable without some inductive biases encoded in models and datasets [49]. In response to this work, subsequent works have explored different frameworks such as weakky-/semi--uperviiedd settings [51, 50] and learning on sequential examples [41] to learn disentangled representations. For example, PhyDNet [24] disentangles the known physical dynamics from the unknown factors by preparing an explicit module called PhyCell. ICA [32] and recent works [71, 64] also discuss the identifiability property of learned representa-
2

---

tions. Classical methods like [28, 6, 36] take an approach of incorporating the inductive bias in the form of a probabilistic model.
We are different from many previous methods in that we do not equip our model with an explicit disentanglement framework. Our method achieves disentanglement as a by-product of training model that can predict the future linearly in the latent space. The set of latent linear transformations estimated by our method for different time sequences can be simultaneously block-diagonalizedd and the latent space of each block corresponds to a disentangled feature. Our data assumption about constant velocity/acceeeraaion might be similar in taste to the setting used by [32], in which the observed time series can be split into the finite number of stationary components.
3 Learning of equivariant structure from stationary time sequences
Our goal is to learn the underlying symmetry structure of a dataset in an unsupervised way that helps us predict the future. What do us humans require for the dataset when we are tasked to, for example, predict where a thrown ball would in the next second? We hypothesize that we solve such a prediction task by analyzing short, past time-frame with certain stationary property (e.g., constant velocity/accelerationn.. Indeed, people with good dynamic visual acuity can chase a fast- moving object, because they can identify such short stationary fime-frame and use it to predict the near future linearly in their latent space. Based on this intuition, we propose to provide the trainer with dataset consisting of constant velocity/acceleratio sequences. We formalize this idea below.
Dataset structure Our dataset S consists of sequences in some ambient space X, so that each member S takes the form [st X; I.... T] E S. Because a sequence that transitions with constant velocity, we assume that all 8t in a given instance of e S are related by fixed transition operation g E G so that 8t+1 go St for all t, where G is the set of transition operators on and each E 9 acts on æ E XX by sending x to g o x. We assume that X' is closed under G; that is, g x X' for all X, g E G. We allow g to be continuous as well, that g might not have a finite order (For instance, if g is a rotation with speed 2Tr with irrational r, any finite repetition of g would not agree with identity mapping). This way, our setting is different from those used in [35] that explores cyclic structure using the capsules of same size. We emphasize that the transition action g is generally assumed to differ across the different members of S. For example, if 9 is set of rotations and X' a set of images, then the rotational speed, direction and the initial image may all be different for any two distinct sequences, s and s'. Because each instance of S is characterized by $ and g, we may write s($1.g) g) to denote a sequence that begins with initial frame 8i and transitions with g. Summarizing, S a subset of {lgt osit 0, O..... T' 1]; $1 E X, E 9).
want to be describing
Prediction framework through equivariance Our strategy is to exploit the stationary property of each E S to seek an invertible continuous function $ X Raxm such that there exists some M : G + Raxa satisfying
M, (z) = ((g x) for all x E X' and g E G.
This relation is known equivariance [1I], and this type of tensorial latent space has also been used [44] as well for the unsupervised learning of the structure underlying the dataset. other words, we seek model in which X' and Raxm are invertibly related by an equivariance relation with respect to G, where g E G acts on (xx) E Raxm via the map ((x) ト (xx(x) with M, E Raxa We assume m > in our study!. In this framework, we predict the sequence s($).g) with the relation o - (M, (s) = St. When G is group, (1) would imply Mat ((r) L(gh o x) = M, = M, M, (xx) for all x, and the map + M, is called a representation of & 19,67,10111 10).
Because we are aiming to establish the framework in which the representation of each action can be explicitly computed, our philosophy has much in common with the proposition of [4]. This approach is in contrast to [35], which encodes a predefined cyclic structure in the model.
We note that, when m > 1, the action of M, Raxm can be realized by applying the same M, to m-copies of IR.. In other words, our prediction framework assumes that there are m number of subspaces that react to g in the same way. This m assumption is also considered [4]. The case which there are no copies of subspaces that act in the same way is called multiplicity- free in the literature of representation theory 19. 19), and is known to be a special case that happens only under a restrictive condition on the dimension of the observations space [46]. A similar idea has been used in model architecture well [15].
3

---

Training
Intra-orbital homogeneity
Full equivariance
M(g.s)) = Míg, gs
Míg, x)= Mgg. hx)
Míg,
= y)
(gs+)
(gx)
((hx)
Diightx)
Oigh
ogy
(s))
(
g
(GPO
gel
h
Figure 1: Visualization of intra-orbital homogeneity vs full equivariance. During the training, the model was trained satisfy M(g.81 81) Mgg. o 81) for all and 81. When intra-orbital homo- geneity holds, M(g.z) x) = (g. h o x)) for all h, E G and x. When the full equivariance holds, (g, x) is invariant across different orbits.
3.1 Learning equivariance relation from stationary sequential dataset
However, training the model satisfying (1) with just the constant velocity assumption not triv- ial task, because this model assumption only assures that, for each sequence s(81,g) there is a sequence-specífic operator ((g, 81) that is guaranteed only be able to predict the sequence that transitions with g and begins from 81 in the way of M (1 = D(5++1) = ( st) (the left most panel in Figure 1). In order satisfy the full equivariance (()) or the right most panel Figure 1), (gg 81) shall not depend on 81 (i.e. homogeneous with respect to 81). At the same time, because the constant velocity assumption applies to each sequence over all time inter- vals, it at least assures that the latent transition M is well defined within each sequence; that is, M .I) M (g.g o $1) M (g. g2 o $1) ... and so on. turns out that, with some regularity as- sumptions on the model and the choice of G, we can extend this observation to say that M satisfies intra-orbital homogeneity (the middle panel Figure 1) that is, M(g, x) is constant on the orbit G o {g x; g E 9) for each x.
Proposition 3.1. Suppose that D(s:) M $1) (1)(((s)1))))))))))))))))))))()) for all and t. If m >> and if G is compact commutative Lie group, then M satisfies intra-orbital homogeneity.
Also, if M satisfies intra-orbital homogeneity, M(g, x)) and M x') for any pair (x,x') in different orbits G o x + G x' can be shown to be at least similar.
Proposition 3.2. Suppose that M (g.x) satisfies intra-orbital homogeneity, and suppose that & is compact connected group. If M (g, x) is continuous with respect x, then for all (x,x²) x!), there exists some P such that PM x.) P M(g 2').
Thus, much of the equivariance property can be satisfied automatically by training the representation on the set of stationary sequences. Interestingly, as we experimentally demonstrate later, our training method in the next section successfully learns a fully equivariant & without explicitly enforcing the change of basis to be I.
3.2 Learning + via solving meta-sequential prediction task
We propose meta-learning way to learn a homeomorphic function $ X + Raxm with equivari- ance property by seeking an injective & such that M (g, $1 (1))) ( for all E G, $1 E X. We learn such by casting this problem as meta-learning problem in which M (g. $1) to be internally estimated for each s. In other words, we seek pair of an encoder & and a decoder &i such that C(Ø, d v ss) min Ex (M+))) s is optimized for each
We conduct this optimization by splitting each s (s1, into conditional time sequence 8. (8,, } and validation time sequence Sp = (8T.+1, ST), while using the former for the internal optimization of M to force the linear algebraic relation the latent space and using the latter for the prediction loss. More precisely, we solve the following optimization problem about &b

---

and V:
( :- - PM (sc (b)
se12
where M (). arg min M E | - (((+1)1
Since M is obtained from the latent sequence in the internal optimization, we call this learning framework the meta-sequential prediction (MSP). It might appear as if we can also set s = S. and optimize the following reconstruction version of (2):
( :) = E. (M s
(3)
However, as we will see in the experiment section, the use of the validation sequence 8p makes substantial difference in the learned representation. This is most likely because the minimization of the validation error of M on Sp would encourage transition M We will illustrate this effect in the experiment section. We shall note that we can also parameterize M as M exp(A), where A E Raax is Lie algebra element to be internally optimized. This type of approach was used in [14] for building Lic group convolutional network and [59, 12] for predicting a sequence that is not necessarily stationary. In order to train their model, [59] used additional parameters to diagonalize each algebra element as well as hyperparameters to stabilize the training. We also experimented with a Lie-algebra style representation of M and used SGD to internally optimize the exponent parameters, but we needed to carefully tune the hyper- parameter for the norm regularization term to stabilize the training and never really succeeded to train the model without collapsing. Our internal optimization procedure is free of such a parameter tuning.
exclude the s.-Specific information from the
Internal Optimization of M Because the internal optimization in (2) is a lin- ear problem, it can be solved analytically as
2. Internally optimize M in latent space:
M (s:sil((ss)ss))------)) MD(s))
1. Encode
$1, S2
M 1) H+1HH H
(4)
((3))
(Dss)
M
(M- ()))))))))
where H40 (s1); ((8.
3. Rollout and decode by: (((MM) (s:))
pa. and H+1 1a)
S
[0(s)):...: are the horizontal concatenations of the encoded frames and H is the Moore- Penrose pseudo inverse of H Because (s. ( is closed form with respect T the loss (2) can be directly optimized by differentiating it with respect to the parameters of both & and V. Thus, the training done in an end-to-end manner. Figure 2 summarizes the overall procedure make prediction on given sequence when T- T, = t. We note that, although we have assumed the dataset to consist of constant-velocity sequences, we can readily extend our method to the dataset consisting of the time series with higher-order stationarity, such as constant acceleration. See Section 4.4 for the detailed explanation of the model extension and the experimental results.
S.2
Figure 2: The overview of meta-sequential predic- tion (MSP) when Tç = 2 and Tр = t. After the en- coder encodes the observations into tensor representa- tions (ss1) ((22). the method solves the least square problem: M arg min as M The model then predicts the future observations by Rt+2 ((AA>)<<<s))) These processes (including the linear problem) are all differentiable.
((s2) (22)1 l|
3.3 Irreducible decomposition of M
Representation theory guarantees that, if is compact connected group, any representation D g + Rax can be simultaneously block-diagonalized; that is, there is a common change of basis U such that V := ===UDA UDJUT θ vi)), where vy is called irreducible representation that cannot be block- diogodiiggn any further [42, 10, 67]. The equivariance of our & which show in the experimental section suggests that (s((s))) may be simultaneously block-diagonalizable as well. This block- -inoooaagnnliinnoonn sometimes reveals disentanglement structure because any irreducible representation of G x G2 is of form (1) X (2),, where (k) is an irreducible representation of
5

---

M *
V*
X
Figure 3: Simutaneous block diagonalization (SBD) applied to the set of Mss obtained from 3DShapes sequences. SBD finds the common change of basis under which all matrices V UM- simultancously take the form of block diagonal matrices with the same block positions.
For clarity, we provide in this figure the visualizations of M == M - and V = V / instead of M and V*.
Gk. In particular, if M irreducible representations have the form v() @ or 1 @ V then each block would either corresponds to the action of 91 or of G2.
To find U that simultaneously block-diagonalizes all (s) we optimized U based on the fol- lowing objective function that measures the block-ness of (s) UM (s/)) based on the normalized graph Laplacian operator A:
(V=(()))) = (A(A(V Irane - -E- -A
(5)
with abs(V"(s)) representing the matrix such that abs(V (S)(() - IV Our objective function is based on the fact that, if we are given adja- cency matrix A of a graph, then the number of connected components in the graph can be identified by looking at the rank of the graph Laplacian. For the derivation, please see Appendix E. Through this decomposition, we are able to uncover the hidden block structure of M's. See Figure 3 for the actual block decoopossoossss of M through our simultaneous block diagonalization. We show in Section 4.3 that each block component of V with optimized U corresponds to the disentangled factor of variations in dataset.
where A(V*(s)) =
4 Experiments
We conducted several experiments investigate the efficacy of our framework. In this section, we briefly explain the experimental settings. For more details, please see Appendix D. We tested our framework on Sequential MNIST, 3DShapes [5], and SmalINORB [48]. Sequential MNIST is created from MNIST dataset [47]. For all experiments, we used a ResNet [26]-based encoder- decoder architecture and we set 16 and m 256 that the latent space lives in IR 16x256
For Sequential MNIST, we chose our G be the set of all combinations of three types of trans- formations: shape rotation, hue rotation, and translation, and randomly sampled single instance of g E G for each sequence(See Appendix D for the examples of sequences). To create each se- quence, we first resized the MNIST image to 24x 24, applied repetitions of randomly sampled, fixed member of E G and embedded the results to 32 x 32 images. For shape and hue rotations, randomly sampled the velocity of angles from uniform distribution on the interval [-T/2, 'T /2) for each sequence. For translation, we randomly sampled the start point and end point in the range of [-10, 10], and then moved the digit images also experimented on sequential MNIST with background (Sequential MNIST-bg). For Sequan- tial MNIST-bg, we used the same generation rule as Sequential MNIST but added background images behind the moving digits. For the background, we used a randomly sampled images from ImageNet [57], which were all resized to 32x32. Also, we only used the images of digit 4 for most of the experiments on Sequential MNIST/MNISTT-ng Unless otherwise noted, all evaluations in this paper for the Sequential MNIST are based training with only digit 4.
straight line between the sampled points. We
3DShapes and SmalINORB are datasets with multiple factors of variation. We created a set of constant-velocity sequences from these datasets by varying fixed combination of factors for each sequence. That is, on these datasets, we chose our G to be the set of variations of factors, and sampled each g as II. 8. where 9; represents the increase of i-th factor by one unit and & E Z. Thus, the value of e, represents the velocity in the direction of the i-th factor the grid. For 3DShapes, we chose wall hue, floor hue, object hue, scale, and orientation as the factors to vary. We varied elevation and azimuth for SmalINORB. For the split of each sequence into (Sc, 8p), set T. = 2 and Tp = 1 on all of the constant velocity experiments.
² g + {1) is valid irreducible representation for any G.

---

#
3
X
#
(a) Generated examples with proposed method Sequential MNIST and MNIST-bg
(1)
s
(3
(4)
(b) Comparison of variant methods
Figure 4: (a): Predictions made by meta-sequential prediction Sequential MNIST and MNIST-bg. The ground truth sequence placed below the predictions, with the first two images representing Se- (b): Typical failure examples generated by the comparative methods. (1)(2)3) and (4) M Neural transition, Rec. model and Ours w/ fixed block, respectively. See Appendix B for more examples.
200
2500
Neural transition
2000
Noura
Neura transition
Rec. mode!
Rec: model
80-
NeuraiM"
wifiked blocks
wiflied blocks
Ours wiliied blocks
20
Curs
(a) Sequential MNIST
(b) Sequential MNIST-bg
(c) 3DShapes
Figure 5: Prediction errors C with T. and T, 1,..., 18. During the training phase, models are trained to predict the observations only at T. = 1. The prediction errors at Tp > 1 indicate the extrapolation performance. The results on SmaliNORB can be found in Appnedix A.
We also conducted experiments for the sequences with constant acceleration on Sequential MNIST. To create a sequence with constant acceleration, we chose a pair 9a 9. E G for each sequence, and generated by setting $141 I.g.s,. We claborate the detail of this extension in 4.4.
As ablations, we tested several variants of our method: fixed 2x2 blocks (abbreviated as fixed blocks), N Reconstruction model (abbreviated as Rec. Model), and Neural transition. For the method of fixed 2x2 blocks, we separated the latent tensor ((s) E IR 16x256 into subtensors (*(()( (s) E 2x22256) k= and calculated pseudo inverse for each 2222256 dimensional space. This variant yields M direct sum of eight 2 2 matrices. We tested this variant to see the effect of introducing predetermined representation theoretic structure as in [10]. For Rec. model, we trained & and based on L Eq. with T. to see the effect of our use of T.. For M we trained an additional network Mo that maps s. transition matrix, replaced M with Mg in (4), and optimized and (Φ, v simultaneously. We may see N as variant in which the meta part of the internal and external training is removed from our method. For Neural transition, we trained 1xl ID-convolutional networks to be applied to latent sequences in the past to produce the latent tensor in the next time step; for instance, $++1 = (1DCCNN((ss)) (ss--))) when T. = 2 The 1DCNN was applied along the multiplicity dimension m. In this variant, the relation between (sss) and ((s++)) is not necessarily linear. Section D. Appendix describes each of the comparison methods more detail. In testing all of these variants, used the same pair of encoder and decoder architecture as the proposed method.
compute transition each
We tested this variant
4.1 Qualitative and quantitative results on the prediction
Figure shows the example sequences generated by the proposed model and comparative models. Figure presents the prediction performance at T. + Tp when T, = 2. To produce this result, we
"This model be simplified version of [60]

---

N
(ii) Ours
(True sequences)
swap M
swap
M
(a) Dependency of M sequence. The 2x3 tiled images in the leftmost panel represents sequences ((1) a with the same transition action g. We consider the effects of M* and M*() inferred respectively
from ((1) and s NeuralM fails in prediction when M M*(2) is used to predict ( Our method does not fail by this swap, indicating (2) = M)
MNIST
MNIST-bg
3DShapes
SmalINORB
Method
CP
CP
CP
Soquiv
C
L
C
CD
equiv
cquiv
Rec. Model
48.91
64.22
87.05
95.66
153.39
258.20
57.01
78. 13
N
4.99
64.25
20.60
83.18
2.09
217.73
28.98
53.24
MSP (Ours)
6.42
15.91
27.38
36.41
2.74
2.87
31.14
44.77
(b) Equivariance performance based on (2). and C Cieggiv (6) with T. 2 and Tμ 1. To evaluate cquivariance errors on more difficult settings, used all of digits Scquential MNIST-bg for both training and test sets.
(Fq.
Figure 6: Quantitative and qualitative evaluation of learned equivariance.
pook-propagated the prediction error at Tр = to the encoder during the training, and the prediction at T; > 1 was used to evaluate the extrapolation performance. Our method successfully predicts the images for T 2 1. Neural transition and NeuralM had almost the same prediction performance at Tp = 1. but they both failed in extrapolation. Our fixed 2 x blocks variant failed in extrapolation as well. This might be because the over-regularized structure of 2x2 block hindered with the training of the SGD optimization [17].
To evaluate how our learned representation relates to the structural features the dataset, we also regressed the factors of transition from M and regressed the class of the digits from ((s1) (Fig- ures 10 and 11 in Appendix A). SimCLR [8] and contrastive predictive coding (CPC) [61, 27] are tested as baselines. Please see Appendix D for the detailed experimental settings for SimCLR and CPC. Our method yields the representation with better prediction performance than the comparative methods on the test datasets.
4.2 Equivariance performance
As we have described through Section 3.1 and 3.2, the equivariance is achieved when M* (s(ss ())))) in (2) does not depend on 81, where we recall that s(81,9) .9) represents the sequence that begins with 81 and transitions with g. see how much the trained model is equivariant to the transformations in the sequential dataset, we therefore calculated the equivariance error, which is the prediction error from applying M (s}(b) to ((s/r:) for a pair s * s' that transitions with the same g. In other words, when T. 2. we compute the following;
Caquiv == E., S (M (s| ((55))
(6)
where S(g) represents the set of all sequences that transition with For each pair of s * s' we set T. and Tp as done in the experiments in the previous sections. Table 6b compares the M method against our method in terms of the equivariance error. Figure 6a shows the result of applying M M (s| on (s(s)) and applying (s ()) on ((s2). We see that when we swap M this way, N also swaps the digits; this implies that M learned by NeuralM encodes the sequence specific information together with the transition. On the other hand, swapping of M does not affect the prediction for our method, suggesting that our method is succeeding to learn an equivariant model. This somewhat surprising, because our model does not have the explicit mechanism to enforce the full equivariance (P 7 in Section 3.1).

---

Belore SBD
After SBD
(2)
(3)
(1) Flooor PaLe
221 Welt huse
Objeect hue
(4) Scale
(5) Orientation
(4)
(5)
(a) Simeltaneously bock-diagonalizer matrices
(b) Disentangled transition representation on 3DShapes
Figure 7: (a) Simultaneous block-diagonalization (SBD) of M The top right matrix is the visual- ization of abs(V' I) averaged over all of the training sequences. Each the five matrices below is the visualization of abs(V* I) averaged over the set of sequences which only a single factor was varied. Coordinates are permuted for better visibility.
(b) Sequences generated by applying the transformation of just one block. To produce the disentangled sequences in each row from the leftmost two images in the bottom row, we performed the internal optimization of M while setting all but the block positions corresponding to each factor of variation to be identity. We elaborate this result and the results for Sequential MNIST, MNIST-bg and SmalINORB in Appendix A.
4.3 Structures found by simultaneous block-diagonallzzaton of M
We have seen in the previous section that the trained + is fully equivariant to transformations G, which implies each A1* is representation of the corresponding transformation of g E G. As we describe in Section 3.3, we apply simultaneous block-diaognalization uncover the symmetry struc- ture captured by * Figure 7a shows the structure revealed by simultaneous block-diagonalization through the change of basis U trained by minimizing the average of Lba in eq.(5) over all s. Fig- ure 7b shows the results of applying transformation of only one block. We can see that each block only alters one factor of variation. Our results suggest that the learned M captures the hidden disentangled structure of the group actions behind the datasets.
4.4 Extension to the sequences with constant acceleration
We have that seetesseeultial prediction successfully learns equivariant structure from the set of constant-velos constant-velocity sequences. In this section, we show that we can extend our concept to the set of sequences sharing the stationarity of higher order (constant acceleration). By definition, the pair of (s,)) and ((s,-1) encodes the information about the velocity at t. When the multiplicity m sufficiently large*, the velocity can be estimated by: 1 *M; = ((s;) (st-))) Because this would yield sequence of velocities, we can simply apply our method again estimate the constant acceleration by 2M ',, Mi, where '4 M2:....: Mr. | E Rax 22) ',,
[LM3;-.. Mr.] E (=x We can then predict the future representation & for To + 1,...,T, Tp by
= ((TT MM)))))))))))))))))))) (8T2) where 'M = 2 M
where II represents multiplications from left. We train and by minimizing the mean squared error between 5, and s, fort = T. 1, T as in (2). To create sequence of constant acceler- ation from MNIST dataset, we only used shape and color rotations. We chose the initial velocity for these rotations randomly on the interval [- π /40, 7 //0)). The results are shown in Figure 8. The Neural transition and the constant-velocity version of our method failed predict the accelerated sequence, while the 2nd or- der model succeeded in predicting the sequence even after Tp > 5. Also, Figure 15f and Table 16c
the interval [- 1/ /5) for each sequence, and chose the acceleration
'If m is less than a, we cannot obtain the pseudo inverse because of the rank deficient in (s: (s)r)))(ss-)))
Thus m should he at least larger than

---

Neural transition
(a) Neural transition
3
/
60-
Ours(2nd orderi
Oursilst order)
(b) Ours (1st order)
&
X
30
(c) Ours (2nd order)
3
4
4
5
13
15
Tp
(a) Generated images accelerated Sequential MNIST.
(b) Prediction enror.
Figure 8: Results on accelerated Sequential MNIST. Every model was trained with T. = 5, Tp = 5.
Neural transition overfitted and collapsed from Tр 5 (beyond the training horizon).
Appendix shows that the accelerated version of our proposed model again achieves learning the equivariance relation.
5 Discussion & Limitations
How full equivariance achieved in our method? The theoretical results we provided in section 3.1 only assure that M a) and M (g, x²) are similar when the underlying group is commutative, compact and connected. However, as we have shown experimentally, our method seems to learn- ing & for which the estimators of M satisfy (s(s(g x)($) = M (s(g, a:) ()). This can be happen- ing because our framework and the training method based on the internal optimization in the latent space is somehow encouraging M be orthogonal (See the loss curve of orthogonality of M in Appendix A). Maybe this is forcing the change of matrix such that PM P-1 (g, x') to be also rotations well, which commutes with (g, x) itself. Also, Figure 4b and 5 show that, as reported in [34], the models trained with reconstruction loss like (3) does not well capture the group transformation behind the sequences: the encoder representation was found be significantly worse than that the model trained with (2). We hypothesize that (3) fails to remove the sequence-specifiee information from M while (3) succeeds do by training the model be able predict the unseen images.
Towards learning symmetries from more realistic observations As making connection between our prediction framework and group equivariance, we are essentially assuming that the transitions always invertible, because group is closed under inversion. However, this might not be always the case in real world applications; for instance, if the image sequences are the sequential generally not invertible because only part
renderings of rotating 3D object, the transitions of the object is visible at each time step. We experimented Sequential ShapeNet, which is created from Shape [7] dataset. A series of rendered images is generated by sequentially applying 3D) rotations of different speeds for each axis. Generated results on Sequential ShapeNet (See 26 in Appendix B show that actually our current method was not able to generate the images on 3D rotated datasets. If the transitions are not invertible, some measures must be taken in order to resolve the indeterminacy, such as probabilistic modeling or additional structural inductive bias.
Broader impact Because our study generally contributes to predictions and extrapolation, it has much potential to negatively affect the society as most other prediction methods. In particular, applications of our method to image sequence can be potentially integrated into weapon systems, for example. At the same time, our unsupervised learning the symmetrical structure from sequential datasets may also contribute to new discoveries in the systems of finance, medical science, physics and other fields of ML. such as reinforcement learning.
References
[1] F. Alet, D. Dohlar, A. Zhou, J. Tenenbbum, K. Kawaguchi, and C. Finn. Noether networks: meta- meta-lcarning useful conserved quantities. NeurlPS, pages 16384-16397, 2021.
[2] L. Ba, J. R. Kiros, and E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06499 2016.
[3] P. Bevanda, S. Sosnowski. and S. Hirche. Koopman operator dynamical models: Learning, analysis and control. Annual Reviews in Control, 52:197-212,
2021. 2021.
10

---

[4] D. Bouchacourt, M. Ibrahim, and S. Deny. Addressing the topological defects of disentanglement via distributed operators. arXiv preprini arXiv:2102.05623, 2021.
[5] C. Burgess and H. Kim. 3d shapes dataset. https///ihhbb.oooeeeepmmi//aaaas
2018.
[6] C. P. Burgess. Higgins. A. Pal. L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Understanding disentangling B-vae. In NeurlPS Workshop of Learning Disentangled Features, 2017.
[7] A. X. Chang. T. Funkhouser. L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese. M. Savva, S. Song,
H. Su, Xiao, L. Yi, and F. Yu. ShapeNet: An Information- Rich 3D Model Repository. Technical Report arXiv: 151 152003012 [es.GR) GR Stanford University Princcton University Toyota Technological Institute at Chicago, 2015, URL ht atttps: ssapppett....e// org/ org/terme ttrms.
[8] I. Chen, S. Kornblith. M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In 1597- 2020.
19] M. Clausen and U. Baum. Fast Fourier Transforms. Wissenschaftsverlag, 1993.
[10] T. Cohen and M. Welling. Learning the irreducible representations commutative lie groups. In ICML, pages 1755 100-17655 1763, 2014.
[11] T. Cohen and M. Welling. Group equivariant convolutional networks. In ICMI, pages 2990-2999, 2016.
[12] M. Connor and C. Rozell. Representing closed transformation paths in encoded network latent space. In AAAI, pages 3666-3675, 2020.
[13] B. Culpepper and B. Olshausen. Learning transport operators image manifolds. NeurIPS, 22:423 431,
2009.
[14] N. Dehmamy, R. Walters, Y. Liu, D. Wang, and R. Yu. Automatic symmetry discovery with lie algebra convolutional network. NewrlPS, pages 2503-2515, 2021.
[15] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and J. Guibas. Vector ncurons: A general framework for -ouui-equiaa networks. In ICCV, pages 22180-12189, 2021.
[16] L. Falorsi, de Haan, R. Davidson, N. De Cao, M. Weiler, Forré, and T. S. Cohen. Explorations in homeomorphic variational auto-encoding. arXiv preprint McXiv:/807.046N9 2018.
[17] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
[18] K. Fukushima and S. Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.
[19] W. Fulton and J. Harris. Representation Theory: A First Course, volume 129. Springer Science & Business Media, 1991.
[20] X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In A/STATS, pages 315-323,
2011.
[21] Greff, F. Belletti, Beyer, C. Doersch, Du, D. Duckworth, D. Fleet, Gnanapragasam, F. Golemo, C. Herrmann, T. Kipf. A. Kundu, D. Lagun, I. Laradji. H.-T. D. Liu, H. Meyer, Y. Miao, D. Nowrouzezahrai, C. Oztireli, E. Pot, N. Radwan, D. Rebain, S. Sabour, M. S. M. Sajjadi, M. Sela, V. Sizzmann, A. Stone, D. Sun, S. Vora, Z. Wang, T. Wu, K. M. Yi, F. Zhong, and A. Tagliasacchi. Kubric: scalable dataset generator. In CVPR, 2022.
[22] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In NeurIPS, pages 15353- 15363, 2019.
[23] Grosek and J. N. Kutz. Dynamic mode decomposition for real-time background/foreground separation in video. arXiv preprint 1rXiv:1404.7590 2014.
[24] V. L. Guen and N. Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In CVPR, pages 11474-11484, 2020.
[25] Y. Han. W. Hao, and U. Vaidya. Deep learning of koopman representation for control. In 2020 59th IEEE Conference on Decision and Control (CDC). pages 1890-1895. IEEE, 2020.
[26] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[27] O. Hênaff, A. Srinivas. J. Fauw, A. Razavi, Doersch, M. A. Eslami, and A. van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML, pages 4182-4192, 2020. [28] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vac: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[29] 1. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner. Towards defini- tion of disentangled representations. arXiv preprint Xiv: 18/2.02230, 2018
11

---

[30] I. Higgins, S. Racanière, and D. Rezende. Symmetry- representations for artificial and biological general intelligence. Frontiers in Computational Neuroscience, page 28, 2022.
[31] A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear
ica. NewrlPS, 2016.
[32] A. Hyvärinen and F. Oja. Independent component analysis: algorithms and applications. Neural net- works, ((4-5))411-4360 2000.
[33] R. Kabra, Zoran, G. Erdogan, L. Matthey, Creswell, M. Botvinick, Lerehner, and C. Burgess. Simone:
View-invariant, comporall--asstrant object representations via unsupervised video decompo- sition. In NeurlPS, pages 20146-20159, 2021.
[34] T. Keller and M. Welling. Predictive coding with topographic variational autoencoders, In ICCV work- shop, pages 1086-1099 106610991222000 2021.
[35] T. Keller and M. Welling. Topographic vaes learn equivariant capsules. In NeurlPS, pages 28585-28597, 2021.
[36] H. Kim and A. Mnih. Disentangling by factorising, In JCML, pages 2649-2658, 2018.
[37] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In /CLR, 2015.
[38] T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR,
2016.
[39] T. Kipf, E. van der Pol, and M. Welling, Contrastive learning of structured world models. In ICLR, 2020.
[40] T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional object-centric learning from vidco. In ICLR, 2021.
[41] D. Klindt, L. Schott, Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In ICLR, 2021.
[42] 1. R. Kondor. Group theoretical methods in machine learning. Columbia University, 2008.
[43] B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931.
[44] M. Koyama, T. Miyato, and K. Fukumizu. Invariance-adage adapted decomposition and lasso-type contrastive learning. In ICML 2022 Workshop on Topology, Algebra, and Geometry in Machine Learning, 2022.
[45] A. Krizhevsky, Sutskever, and E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 25, 2012.
[46] A. S. Leahy. A classification of multiplicity free representations. Journal Lie Theory, 8:367-391, 1998.
[47] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recogni- tion. Proceedings of the IEEE, 86(1 1):2278-23244, 1998. URL http: / /yann. lecun. com/ /xdb// amist/Licenss
[48] Y. LeCun, F. J. Huang, and I.. Bottou. Learning methods for generic ohject recognition with invariance to pose and lighting. In Proceedings of the 2004 IEFE Computer Sociery Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2. pages 11-104. IEEE, 2004. URL https://cs. nyu. byy.edu/// edueeydulaeaoeee cestta/yyllbbbbce cataaaa//aaa//eeee wotttaannooovv..o.ooe wmw.al/....m.......mmmmm.m...m.1
[49] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem, Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, pages 41 14- 4124, 2019.
[50] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. In ICMI., pages 6348-6359, 2020.
[51] F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of
variation using few labels. In ICLR, 2020.
[52] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier monlincarities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
[53] V. Nair and G. E, Hinton. Rectified linear units improve restricted boltzmann machines. ICML. pages 807-814. 2010.
[54] D. Pfau, T. Higgins, Botev, and S. Racanière. Disentangling by subspace diffusion. NeurlPS, pages
17403-17415, 2020.
[55] S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille. Weight standardization. arXiv preprint veXiv:/933.100 2019.
[56] R. Quessard, T. Barrett, and W. Clements. Learning disentangled representations and group structure of dynamical environments. NewrlPS, pages 19727-19737, 2020.
12

---

[57] O. Russakovsky, J. Deng, Н. Su, J. Krause. S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision. 115(3):11-252,, 2015.
[58] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann. Neural descriptor fields: Se e3uuqivrrri object representations for manipulation. arXiv preprint
arXiv:2112.05124, 2021.
[59] J. Sohl-Dickstein, C. M. Wang, and B. A. Olshausen. An unsupervised algorithm for learning lie group transformations. arXiv preprint arXiv:/001.0.0.0 1001. 2010.
[60] A. Van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (XXv::609.03499 2016.
[61] A. Van Oord, Y. and O. Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, 2018.
[62] E. van der Pol, D. Worrall, van Hoof, Olichock, and M. Welling. Mdp homomorphic networks: Group symmetrics in reinforcement leaming. NeurlPS, pages 4199-4210, 2020.
[63] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E., Kaiser, and I. Polosukhin. Attention all you need. NeurlPS, 30, 2017.
[64] J. Von Kügelgen, Y. Sharma, I.. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self- supervised learning with data augmentations provably isolates content from style. NewrIPS, 34, 2021.
[65] D. Wang. R. Walters, and R. Platt. So(2) -oquivariant reinforcement learning. arXiv preprint ar arXiv:2203.044 044399 2022.
[66] Z. Wang and J.-C. Liu. Translating math formula images latex sequences using deep neural networks with sequence-level training. International Journal on Document Analysis and Recognition (IJDAR), 24 (1):63-75, 2021.
[67] S. H. Weintraub. Representation Theory of Finite Groups: Algebra and Arithmetic,, volume 59, American Mathematical Society, 2003.
[68] Y. Wu and K. He. Group normalization. In ECCV, pages 3-19, 2018.
[69] T. Yang, X. Ren, Y. Wang, W. Zeng. and N. Zheng. Towards building a group-based unsupervised repre- sentation disentanglement framework. In ICLR, 2021.
[70] S. Zhang, Wang. and A. Li. Cross-view gait recognition with deep universal linear embedklings. In CVPR, pages 9095-9104, 2021.
[71] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel, Contrastive learning inverts the data generating process. In ICML, pages 12979-12990, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's con- tributions and scope? [Yes] The abstract and introduction reflects the contributions this paper.
(b) Did you describe the limitations of your work? [Yes] In the Discussion & Limitations sec- tion, we discuss the type of dataset to which our work cannot be directly applied, as well as the theoretical problem that has not been solved.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We included the broader impact discussion in Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes] We read the guidelines and we confirmed that the contents in our paper conformed to them.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We included the full set of assumptions. We provide the details of the assumptions in the supplementary material.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide complete proofs of theoretical results in Supplementary material.
3. If you ran experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main experimen- tal results (either in the supplemental material or as a URL)? [Yes] We provided code for reproducing the experimental results as supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We explained how we chose the hyperparameters and how we split the dataset into training and test sets in section 4 and appendix section D.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We include the standard deviation values of equivariance perfor- mance in Appendix A (We omitted them in the main submission because of the space limi- tation) and the linear classification performance of the transition parameters in Appendix A.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We put the information GPUs which used to train the models. Also we included approximated costs for reproducing the full results of experiments in D.
you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited [47] for MNIST,
[7] for ShapeNet, [36] for 3Dshapes and [48] for SmalINORB.
(b) Did you mention the license of the assets? [Yes] For each dataaset, we included the URL to the license information in the reference section.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] For ShapeNet dataset, we are approved download the assets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable in- formation offensive content? [Yes] We don't 't find any personally identifiable information the images used our experiments.
you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applica- ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "RknCLq83Z6e", "reviewer": "Reviewer_hTb6", "summary": "The authors of this paper investigate the time series models with certain stationary properties, and use the learned symmetry structure for predicting the future in time series.  They consider the type of time series where each sequence is generated by initial observation and some fixed transition operator. Then they propose to optimize a homeomorphic function with equivariance property w.r.t. a group of transition operators. In the experiments, they evaluate the proposed approach on sequences generated based on 3 image datasets.The results show that their method can successfully learn transition operators such object rotation, hue rotation, and object translation. By applying these operators to the initial time step, they show that they could predict the future sequence for a few number of time steps.", "questions": "I have some questions:\n\n- Can you motivate the ablation study in the experiments? I don’t understand why you are doing it. \n\n- Regarding the comparison against SimCLR and CPC, I find it a bit hard to understand the experiments and interpret the results. Can you please provide more details?\n\n- With noisy background in the image examples, do you find the optimization more difficult? Intuitively it adds challenges to learn the representations for the objects there, but I wonder how harder it becomes in your case, since you assume a purely static background.\n\nGeneral suggestion:\n\n- I think it is interesting to study the equivariance property in time series in general. On the other hand, I feel it is also important to showcase the robustness / generalization when tackling trajectory prediction tasks. In other words, the point estimations are not that great in many use cases, and we would want to characterize the model uncertainty. So I wonder if you could do some reasoning about the prediction uncertainty in this proposed method.\n", "limitations": "To my best knowledge, the authors have comprehensively discussed the limitations of their work and the potential negative impact.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: In my opinion, this is an interesting paper because they try to investigate the equivariance property in stationary time series models. From this perspective this idea is novel, though I believe the problem defined in this paper can be alternatively tackled using existing methods on disentangled representation learning. (I will think of this weakness more as insignificance rather than originality, because in general studying equivariance in time series is a great topic, and it is not reasonable to argue about originality merely because some existing methods can solve the same type of problems.)\n\nSignificance: Regarding disentangled representation learning. I mainly wonder what we additionally achieved with this proposed method, compared to the existing methods on disentangled representation learning. The reason is that, the authors specifically consider time series with fixed transition operators, e.g. rotation, translation, and the goal is to predict the future observations in the sequences. Because of the strong assumption on fixed transition operators, these prediction tasks, in my opinion, can also be framed as generating examples by varying factors of variation using models in [1,2,3]. \nWhile the authors have explained how their work differs from the prior work on disentangled representation learning, I would appreciate it if they do some comparison against some of those methods such as [1,2,3].For both learning disentangled representations and generation tasks, those VAE-based methods will also achieve impressive results in datasets such as MNIST, FashionMNIST, Dsprites. On the other hand, this proposed method can only do point-estimation, while those VAE-based models can characterize the prediction uncertainty in a variational manner. This is my main concern about the significance of this work.\n\nQuality: This paper is well structured in the sense that they provide proof of theory that supports their motivation and sufficient empirical evaluations as well. \n\nClarity: This paper is well written and clearly conveys the idea in the paper.\n\n[1] Chen, Ricky TQ, et al. \"Isolating sources of disentanglement in variational autoencoders.\" Advances in neural information processing systems 31 (2018).\n\n[2] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International Conference on Machine Learning. PMLR, 2018.\n\n[3] Esmaeili, Babak, et al. \"Structured disentangled representations.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FjO0jJwnbpr", "reviewer": "Reviewer_dgwX", "summary": "The paper studies the problem of learning symmetry from sequential data with a certain stationarity property. In particular, from time sequences of length at least three, meta-learning is used to learn representations such that a future observation can be predicted well by a linear transition. It is shown that doing so can tease out the disentangled structure by simultaneous block-diagonalization. Each block then corresponds to a disentangled feature. \n\nThe meta-learning framework is very straightforward: We learn an injective mapping \\phi for each time step, such that a linear transition relates time step t to time step t+1. The loss function involves a decoder that maps the linear transition \\phi_t-1 to \\phi_t. The transition can be parameterized in various complex ways, but as mentioned above only a linear map is used. The inner loop consists of minimizing the prediction loss, while the outer loop updates the encoder. The main assumption is that the velocity/acceleration is preserved within each observation. Experiments on Sequential MNIST, 3D Shapes, and SmallNORB show the efficacy of the proposed method. \n\n\n", "questions": "See above: \n\nMinor comments:\n\nThere are quite a few typos and somewhat awkward sentence constructions. Below are a few examples:\n- Typo, line 16: *in machine learning? \n- Line 20: should be \"has succeeded\"\n- Line 34-35: \"in the way of meta-learning\" -> by meta-learning/by the way of meta-learning/by means of meta-learning\n- Line 44-45: \"There are rich literatures in unsupervised/weakly supervised\" -> There is a rich literature...\n- Line 63: \"Many studies impose algebraic constraints that reflects some form of geometrical assumption.\" -> that reflect some form of..\n- Line 86: identifiebility -> identifiability \n\n", "limitations": "Yes, adequately addressed. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The basic idea of the paper is quite simple: That we can learn some underlying algebraic structure in an unsupervised way when the transitions can be represented linearly in some latent space. It follows the line of work on Slow Feature Analaysis, ICA and more recent work by Keller and Welling, and also seems reminiscent of work on Koopman analysis (although multiple dynamics are considered here). The paper proposes to use meta-learning to predict a future observation and model only linear transitions. The meta-learning setup is also quite simple. \n- The experiments are carried out on reasonable datasets: Sequential MNIST, 3D Shapes and SmallNorb. The overall results and quality of ablation is good. \n- The extension to constant acceleration is also reasonable, as are the discussion of limitations. \n\nWeaknesses: \n- The paper is actually quite hard to read. I think it can be made really good with a smoother presentation. Even though the main idea is quite simple, I often had difficulty following the details and trying to decode what the authors might have meant. \n- The experiments are good in the sense that they cover some good datasets, and have good ablations. However, I have difficulty in placing their significance, given the authors only report measures such as equivariance error. Can we have another downstream task in which such learnt representations are shown to be helpful? If not, is it possible to have some reasonable baselines to get a sense of how good the variants proposed here are? Overall I like the main idea -- it is simple and elegant, but I am not quite sure what to make of the results, and I think the paper could do a better job at presenting them. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1bvMFKhj01", "reviewer": "Reviewer_XtDt", "summary": "This paper proposes to discover structural properties of the data by solving a sequential prediction tasks.\nThe idea is that solving the prediction tasks produces equivariant models, which by guarantees from representation theory, means that the learned representations are simultaneously block-diagonalized, with blocks corresponding to disentangled factors.\n\nSpecifically, the paper learns an encoder $\\Phi$, a decoder $\\Psi$, and a matrix-valued function $M$, such that given adjacent observations $s_t, s_{t+1}$, the difference $\\||\\Psi (M\\Phi(s_t)) - s_{t+1}\\||_2^2$ is minimized. The training borrows idea from meta learning, where the algorithm first estimates $M$ based on a part of the sequence (denoted $\\mathbf{s}_c$), and $\\Phi$ is then estimated using this $M$ and the rest of the sequence (denoted $\\mathbf{s}_p$).\n\nEmpirical results show that 1) the proposed method is able to train on only adjacent time frames and extrapolate to unseen steps in the future, and that 2) the learned linear transitions can be simultaneously block-diagonalized, with blocks corresponding to 1 factor of variation.", "questions": "Questions:\n- The choice of $M$ seems to rely on knowledge of the true dynamics. How to obtain such knowledge in general? How robust is the method to \"model mismatch\", e.g. when internal optimization is performed on sequences with noises or with varying velocity? How can the model be extended to handle some amount of model mismatch?\n- Would the choice of $T_c, T_p$ affect the performance? \n    - Choosing $T_c=2$ should be sufficient if the data perfectly follows the model assumption (e.g. constant velocity), but this may not be the case when there's model mismatch.\n- Would the proposed method be able to handle multiple objects?\n- Sec 5.2, swapping $M^*$: my understanding is that $M^*$ should depend on the sequence-specific action $g$. However in Fig 4(a), the two sequences clearly have different actions (e.g. the bottom row seems to be doing both color rotation and shape rotation, while the top row only has shape rotation) and different orbits (since the digits are not the same). How could $M^*$ be the same then?\n- Fig 8 in the appendix: line 255 in the main text says that the proposed method gives \"better representations\" but it's unclear how so. Please explain the success/failure modes of the proposed method and the reasons.\n- Line 276, about simultaneously block-diagonalizable: please provide reference for this.\n\n\nClarifications:\n- Line 124: is $\\circ_{latent}$ different from $\\circ$?\n- Line 220: \"digit4 only training\": does it mean that the encoder/decoder have not seen other digits during training?\n- Line 233: what's the reason for splitting $H$ into 8 (and not some other number of) subtensors?\n- Line 250: what does \"fixed block structure\" mean? Does dividing $H$ into 8 subtensors count at fixing the block structure?\n- Line 303: what does \"sufficiently large\" mean for $m$?\n\n\nWriting: overall I find the paper a bit hard to parse, possibly due some inconsistent notations and my own in-familiarity to the topic.\nSome concrete things:\n- Please fix citation [13].\n- Line 115: do you mean members of $S$ (rather than $s$, which is itself a member of $S$)?\n- Line 143: maybe consider a different notion than $M(g, s_1)$, since $M$ has been defined to be a single-argument function from $\\mathcal{G}$ to $\\mathbb{R}^{a \\times a}$.\n- Line 179: should it be \"optimized on $s_p$\"?\n- Inconsistent notations:\n    - $H_{+1} H_{+0}^\\dagger$ in eq (4) and $H_2H_1^\\dagger$ in Fig 1.\n    - $T_p$ on line 227, $t_p$ on line 245.\n- Line 233 & 234: it would be more consistent to use either 256 or $m$ (but not both) in defining $H, H_k$.\n- The paragraph on line 252: it would be better to briefly explain what \"better representation\" means.\n- Line 289: there seems to be a typo in the definition of $\\hat{M}^*$.\n", "limitations": "The paper discusses the limitations of the proposed method, such as failing to work with non-invertible transitions, e.g. ShapeNet dataset.\n\nThe paper also discusses potential societal impact if the method were to employed in real-world sequential prediction problems.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- It's an interesting idea to learn disentangled representations by the simple prediction task.\n- The paper presents a variety of empirical results and is clear on implementation details.\n- The paper provides hypotheses on some empirical results (Sec 6) and is open about the limitations of the proposed methods, e.g. failing to work on non-invertible transitions.\n\nConcerns\n- [Clarified in the response] While inductive biases are necessary as established by prior work, the proposed method seems too tailored to the specific data structure, and it seems unfair to compare with baseline models that do not have such knowledge about the data.\n- [Draft updated] The messages from some empirical results could be made clearer; please see detailed questions below.\n- [Clarified in the response] Please discuss the relation to the line of work about incorporating physics into the prediction model, e.g. PhyDNet (Guen & Thome 20), and disentangled representation learning in videos in general.\n\n=== **Update** ===\n\nThe authors' responses have address most of my concerns. Many of my previous questions were because I didn't get the main point the paper, since I had trouble parsing the paper as I mentioned in comments on clarity.\nThe writing is now improved and my review has been updated accordingly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept