Multifactor Sequential Disentanglement via Structured Koopman Autoencoders
Authors: Nimrod Berman
Ilan Naiman
Omri Azencot
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Koopman methods
Sequential Disentanglement

Abstract: Disentangling complex data to its latent factors of variation is a fundamental task in representation learning. Existing work on sequential disentanglement mostly provides two factor representations, i.e., it separates the data to time-varying and time-invariant factors. In contrast, we consider multifactor disentanglement in which multiple (more than two) semantic disentangled components are generated. Key to our approach is a strong inductive bias where we assume that the underlying dynamics can be represented linearly in the latent space. Under this assumption, it becomes natural to exploit the recently introduced Koopman autoencoder models. However, disentangled representations are not guaranteed in Koopman approaches, and thus we propose a novel spectral loss term which leads to structured Koopman matrices and disentanglement. Overall, we propose a simple and easy to code new deep model that is fully unsupervised and it supports multifactor disentanglement. We showcase new disentangling abilities such as swapping of individual static factors between characters, and an incremental swap of disentangled factors from the source to the target. Moreover, we evaluate our method extensively on two factor standard benchmark tasks where we significantly improve over competing unsupervised approaches, and we perform competitively in comparison to weakly- and self-supervised state-of-the-art approaches. The code is available at https://github.com/azencot-group/SKD.

Published as a conference paper at ICLR 2023
MULTIFACTOR SEQUENTIAL DISENTANGLEMENT VIA STRUCTURED KOOPMAN AUTOENCODERS
Nimrod Berman', Ilan Naiman", Omri Azencot Department of Computer Science
Ben-Gurion University of the Negev
(bermann, na imanni @poost............. bgu.bg.u... ac. i1, azencotecc. ac
ABSTRACT
Disentangling complex data to its latent factors of variation is a fundamental task in representation learning. Existing work on sequential disentanglement mostly provides two factor representations, i.e., it separates the data to time- varr-ia and time-invariant factors. In contrast, we consider multifactor disentanglement which multiple (more than two) semantic disentangled components are generated. Key to our approach a strong inductive bias where we assume that the underlying dynamics can be represented linearly in the latent space. Under this assumption, it becomes natural to exploit the recently introduced Koopman autoencoder models. not guaranteed in Koopman approaches,
However, disentangled representations and thus we propose novel spectral loss term which leads to structured Koopman matrices and disentanglement. Overall, we propose simple and easy to code deep model that is fully unsupervised and it supports multifactor disentanglement. We showcase new disentangling abilities such as swapping of individual static factors between characters, and an incremental swap of disentangled factors from the source to the target. Moreover, we evaluate our method extensively on two factor standard benchmark tasks where we significantly improve over competing unsupervised approaches, and we perform competitively in comparison to weakly- and self-supervised state-of-the-art approaches. The code is available at GitHub
INTRODUCTION
Representation learning deals with the study of encoding complex and typically high-dimensional data in meaningful way for various downstream tasks (Goodfellow et al. (016). Deciding whether a certain representation is better than others is often task- and domain-dependerd tangling data its underlying explanatory factors is viewed by many fundamental challenge in representation learning that may lead to preferred encodings (Bengio et al.) 2013). Recently, several works considered two factor disentanglement of sequential data which time-varying features and invarintaiinnt features are encoded in two separate sub-spaces, In this work, we contribute to the latter line work by proposing simple and efficient unsupervised deep learning model that performs multiaccorr disentanglement of sequential data. Namely, our method disentangles sequential data to more than two semantic components.
However, disen-
One of the main challenges in disentanglement learning is the limited access to labeled samples, particularly in real-world scenarios. Thus, prior work on sequential disentanglement focused on unsupervised models which uncover the fime-varying and time invarinat features with available labels (Hsu et al. 2017 Li & Mandt 2018).. Specifically, two feature vectors are produced, represent- ing the dynamic and static components in the data, e.g., the motion of a character and its identity, respectively. Subsequent works introduce two factor self-supervised models which incorporate supervisory signals and mutual information loss (Zhu al. 2020) data augmentation and contrastive penalty (Bai et al. 2021], and thus improve the disentanglement abilities of prior baseline models. Yamada et al. (2020) proposed a probabilistic model with ladder module, allowing certain multifactor disentanglement capabilities. Still, to the best of our knowledge, the majority of existing work do not explore the problem of unsupervised multifactor sequential disentanglement.
joint first authors

---

Published as a conference paper at ICLR 2023
In the case of static images, multiple disentanglement approaches have been proposed KKulkarni
al. [2015] Higgins et al., |017]] Kim & Mnih| 2018; Chen et 2018: 2016 Burgess et al. 2018
Kumar 2017] Bouchacuurt al. 2018). In addition, there several approaches that support disentanglement of the image to multiple distinct factors. For instance, Li et al. (2020) design an architecture which learns the shape, pose, texture and background of natural images, allowing to generate new images based on combinations of disentangled factors. In (Xiang et al. (021), the authors introduce a weakly-supervised framework where N factors can be disentangled, given N 1 labels. In comparison, our approach is fully unsupervised, deals with sequential data and the number of distinct components determined by hyperparameter.
Recently, Locatello et al. (2019) showed that unsupervised disentanglement is impossible without inductive biases on models and datasets. While exploiting the underlying temporal structure had been shown as strong inductive bias in existing disentanglement approaches, we argue in this work that a stronger assumption should be considered. Specifically, based on Koopman theory (Koopman) 1931) and practice (Budišić et al. 2012 Brunton et al. 2021), we assume that there exists learnable representation where the dynamics of input sequences becomes linear. Namely, the temporal change between subsequent latent feature vectors can be encoded with matrix that approximates the Koopman operator. Indeed, the same assumption was shown to be effective in challenging scenarios such fluid flows (Rowley al.. 2009) well as other application domains (Rustamov et al.) 2013 Kutz et al. 2016). However, it has been barely explored in the context of disentangled representations.
In this paper, we design an autoencoder network (Hinton & Zemel 1993) that is similar previous Koopman methods (Takeishi et al. 2017, Morton et al. |l.] 2018)), and which facilitates the learning of linear temporal representations. However, while the dynamics encoded in Koopman operator, dis- entanglement is nol guaranteed. To promote disentanglement, we make the following key observation: eigenvectors of the approximate Koopman operator represent time-invarian and variant factors. Motivated by this understanding, we propose a novel spectral penalty term which splits the operator's s spectrum to separate and clearly-defined sets of static and dynamic eigenvectors. Importantly, our framework naturally supports multifactor disentanglement: every eigenvector represents a unique disentangled factor, and it is considered static or dynamic based on its eigenvalue.
Contributions. Our main contributions can be summarized as follows.
1. We introduce strong inductive bias for disentanglement tasks, namely, the dynamics of input sequences can be encapsulated in a matrix. This assumption is backed by the rich Koopman theory and practice.
2. We propose new unsupervised Koopman autoencoder learning model with novel spectral penalty on the eigenvalues of the Koopman operator. Our approach allows straightforward multifactor disentanglement via the eigendecoomossition of the Koopman operator.
3. We extensively evaluate our method on new multifactor disentanglement tasks, and on several two factor benchmark tasks, and we compare our work to state-of- state-of-the-art the-art unsupervised and weakly-supervisee techniques. The results show that our approach outperforms baseline methods in various quantitative metrics and computational resources aspects.
2 RELATED WORK
Sequential Disentanglement. Most existing work on sequential disentanglement is based on the dynamical variational autoencoder (VAE) architecture (Girin et al., 2020). Initial attempts focused on probabilistic models that separate between static and dynamic factors, where in (Hsu et al. 2017) the joint distribution is conditioned on the mean, and in (Li & Mandt 2018) conditioning is defined on past features. Subsequent works proposed self-supervised approaches that depend on auxiliary tasks and supervisory signals (Zhu et al., 2020), or on additional data and contrastive penalty terms (Bai et al. 2021). In Han et al. (2021a) 1a), the authors replace the common Kullback-Leibhe the Wasserstein distance between distributions. Some approaches tailored to video disentanglement
divergence with
use generative adversarial network (GAN) architectures (Villegas et al. 2017: Tulyakov al. 2018)
and a recurrent model with adversarial loss (Denton & Birodkar 20.7). Finally, Yamada et al. 2020) proposed variational autoencoder model including ladder module (Zhao al. 2017). which allows to disentangle multiple factors. The authors demonstrated qualitative results of multifactor latent traversal between various two static features and three dynamic features the Sprites dataset.
2

---

Published as a conference paper at ICLR 2023
Koopman Module
X rec
Lree
by
X
Z
Z, Z
Z
C
Eped
Xenc
Xdec
Drig
Figure 1: Our architecture is based on a Koopman autoencoder network which includes encoder Xeue. decoder Xdec, and Koopman module that computes the Koopman operator C' via least squares solves. We augment this model with a novel spectral penalty term Leig which facilitates the learning of spectrally structured C' matrices, and thus supporting multifactor disentanglement by construction.
Dynamics Learning. Over the past few years, an increasing interest was geared towards learning and representing dynamical systems using deep learning techniques. Two factor disentanglement methods based on Kalman filter (Fraccaro et al. 2017). and state-space models (Miladinović et al.
2019) focus on ordinary differential equation systems. Other methods utilize the mutual information between past and future estimate predictive information Clark et al.
(2019); Bai et al. (2020).
Mostly related to our approach are Koopman autoencoders (Lusch et al. 2018: Yeung et al., 2019
Otto & Rowley 2019 Li al. 2019 Azencot al., 2020 Han al. (02Tb), related to classical learning methods, e.g.. Azencot et al. (2019); Cohen
Cohen et al. (2021). Specifically, in (Takeishi et al.
2017: Morton al./ 2018 Iwata Kawahara 2020) the Koopman operator learned via least squares solve per batch, allowing to train a single neural model on multiple initial conditions. We base our architecture on the latter works, and we augment it with a novel spectral loss term which promotes disentanglement. Recently, an intricate model for video disentanglement was proposed in (Comas et al., 202I). While the authors employ Koopman techniques that work, only partially related to our work since they explicitly model pose and appearance components, whereas our approach can model an arbitrary number of disentangled factors. In addition, their architecture is based the attention network (Bahdanau al., 2014), where the Koopman module is mostly related to prediction. In comparison, in our work the Koopman module is directly responsible for unsupervised disentanglement of sequential data.
Koopman Spectral Analysis. Our method is based learning Koopman operators with structured
spectra. Spectral analysis of Koopman operators is an active research topic (Mezič 2013 Arbabi
Mezic 2017 Mezic 2017, Das & Giannakis, 2019 Naiman & Azencot 2023) We explore
Koopman eigenfunctions associated with the eigenvalue I. These eigenfunctions are related to global
stability (Mauroy Mezic 2016), and to orbits of the system (Mauroy Mezić 2013) Azencot et
2013|2004), Other attempts focused on computing eigenfunctions for a known spectrum (Mohr & Mezic 2014). Recently, pruning weights of neural networks using eigenfunctions with eigenvalue I was introduced in (Redman al., 2. However, to the best of our knowledge, our work is among a few to propose deep learning model for generating spectrally-structne Koopman operators.
3 KOOPMAN AUTOENCODER MODELS
We recall the Koopman autoencoder (KAE) architecture introduced in (Takeishi et al.) (017) it is the basis of our model. The KAE model consists an encoder and decoder modules, similarly to standard autoencoders, and in between, there a Koopman module. The general idea behind this architecture is that the encoder and decoder are responsible to generate effective representations and their reconstructions, driven by the Koopman layer which penalizes for nonlinear encodings.
We denote by X (bx batch of sequence data {x.,) C I. where {1...., b) and j E (1...., ++ 1} represent the batch sample and time indices, respectively. The tensor X is encoded to its latent representation Z e (x* via = Xene (X). The Koopman layer splits the latent variables to past Zp and future 7y observations, and then, it finds the best linear map such that
3

---

Published as a conference paper at ICLR 2023
Zp CC = y, Formally, Zp (22g) R for j E {1, {l,...,t) .t and any i, and 2 (2iz) R for E f2....,t .t....,,+ ++1] and any i, ie., Zp holds the first latent variables per sample, and 2 holds the last È variables, Then, C = arg min& 12p-C 25ト² = 2, where A denotes the pseudo-inverse of the matrix A. Importantly, the matrix is computed per Z during both training and inference, and in particular, C' is not parameterized by network weights. Additionally, the pseudo- inverse computation supports backpropagation, and thus it can be used during training (lonescu et 2015). Lastly, the latent samples are reconstructed with the decoder X Xder (2).
Z,,
The above architecture employs reconstruction and prediction loss terms: the reconstruction loss promotes an autoencoder learning, and the prediction loss aims to capture the dynamics in C. We use the notation ( Y) = b4 Eig Y (i, - (i,,)] for the average distance between tensors x, Y E Rbxeex for i E (I,...,b) and j E (1..... .... (ee X)= CMss(XXres (X) Spred ( ( yz x x,) = LMSE (2,, (2,, ) + ,
1). Then, the losses are given by
(2)
where 2; Zp CC, Xdec ( 2),, and X are the inputs corresponding to Z latent variables. The network loss taken to L = Aree Cree + Apredd where Arees Apred R balance between the reconstruction and prediction contributions. We show in Fig. 1 an illustration of the Koopman autoencoder architecture using the notations above.
MULTIFACTOR DISENTANGLING KOOPMAN AUTOENCODERS
How disentanglement can be achieved given the Koopman autoencoder architecture? For comparison, other disentanglement approaches typically represent the disentangled factors explicitly. In contrast the batch dynamics KAE models encoded the approximate Koopman operator matrix C, where C' propagates latent variables through time while carrying the static as well as dynamic information. Thus, the time-varying and time-invariant factors are still entangled in the Koopman matrix. We now show that KAE theoretically enables disentanglement under the following analysis.
Koopman disentanglement. In general, one of the key advantages of Koopman theory and practice the linearity of the Koopman operator, allowing to exploit tools from linear analysis. Specifically, our approach depends heavily on the spectral analysis of the Koopman operator (Mezić (2055). In what follows, we perform our analysis directly on C, and we refer the reader App. and the references therein for a detailed treatment of the full Koopman operator. The eigendecomposition of C' consists of a set of left eigenvectors (Φ) E C+ and set of eigenvalues {)₂ E C} such that
=-== i=
The eigenvectors can be viewed as approximate Koopman eigenfunctions, and thus the eigenvectors hold fundamental information related to the underlying dynamics. For instance, the eigenvectors describe the temporal change in latent variables. Formally,
2 2fT1 j= i.,,,..,t
where =)== (xff, o4 is the projection of z the eigenvector of The approximation follows from C' being the best (and not necessarily exact) linear fit between past and future features. Moreover, follows that predicting step j + from j is achieved simply by applying powers of the Koopman ~ =fr
matrix on 23. i.c., 2 C" Σ, N
Our approach to multifactor disentangiement is based on the following key observation: eigenvectors of the matrix C' whose eigenvalue is 1 represent time inve-invaiant factors. For instance, assume C' has single eigenvector Φn with \ and X; F 1 for f 1, then it follows from Eq. that
c- 2 of
Σ ooNT
(5)
Essentially, the contribution of oi is not affected by the dynamics and remains constant, and thus the first addend remains constant throughout time, and it related to static features of the dynamics.
4

---

Published as a conference paper at ICLR 2023
In contrast, every element in the sum in Eq. i scaled by its respective X, and thus the sum changes throughout time, and these cigenvectors are related to dynamic features. We conclude that the KAE architecture virtually allows disentanglement via eigendecomposition of the Koopman matrix where the static factors are eigenvectors with eigenvalue 1, and the rest are dynamic factors.
Multifactor Koopman Disentanglement. Unfortunately, the vanilla KAE model is not suitable for disentanglement as the learned Koopman matrices can generally have arbitrary spectra, with multiple static factors or no static components at all. Moreover, KAE does not allow to explicitly balance the number of static vs. dynamic factors. To alleviate the shortcomings of KAF, we propose to augment the Koopman autoencoder with a spectral loss term Leig which explicitly manipulates the structure of the Koopman spectrum, and its separation to static and dynamic factors. Formally,
k. A
1
Letat
(1 +0)
(6)
Kig
linin
ha Σ E(1A c)
Layu
(7)
ki
Leig -Esaat + Layn
(8)
-1 real
where h, and ka represent the number of static and dynamic components, respectively, and thus = kg + ka- The term Latat measures the average distance every static eigenvalue from the complex value 1. The role of Layn is to encourage separation between the static and dynamic factors. In practice, this is achieved with a threshold function & which takes the modulus of X and a user parameter E 1), and it returns [Λe] if [Ag] e, and zero otherwise. Thus, Layn penalizes dynamic factors whose modulus is outside an e-ball. The inset figure shows an example spectrum we obtain using our loss penalties, where blue and red denote static and dynamic factors, respectively.
Method Summary. Given batch X Bbxtxm, we feed it the encoder. Our encoder is similar to the one used in C-DSVAE (Bai et al.] 202T) having five convolutional layers, followed by a uni- directional LSTM module. The output of the encoder is denoted by Z E gxxtxk and it is passed to the Koopman module. Then, Z is split to past Z, and future 2, observations, allowing to compute the approximate Koopman operator via C = 2 2 In addition, we compute 2, Zp C which will be used to compute Cpred- After the Koopman module, we apply the decoder whose structure mimics the encoder but in reverse having LSTM component and de-convolutional layers. Additional details on the encoder and decoder are detailed in Tab. 5 We decode Z to obtain the reconstructed signal Xrec, and we decode 2 to approximate the future recovered signals Xy. The total loss is given by L = >xreeCee + Apred + >eiggooig where the balance weights Aree, Apred and Aeig scale the loss penalty terms and the exact values are given in Tab. 6 To compute Leig, we identify the static and dynamic subspaces. This is done by simply sorting the eigenvalues based on their modulus, and taking the last K2 eigenvectors, whereas the rest ka are dynamic factors. Identifying multiple factors is more involved and can be obtained by manual inspection or via an automatic procedure using a pre-trained classifier, see App. B.5
Multifactor Static and Dynamic Swap. Similar to previous methods our approach allows to swap between e.g., the static factors of two different input samples. In addition, our framework naturally supports multifactor swap as we describe next. For simplicity, we first consider the swap of single factor (e.g., hair color in Sprites (Reed et al. (015)) for the given latent codes of two samples, 2j (u) and 2 J 1,...,t + Denote by Φ the eigenvector of the factor we wish swap, then single swap is obtained by switching the Koopman projection coefficients of Φ1, i.c.
& ()) 2)(w) =))))))()00))) (w) Σ Z(()))
(w) = 2))xc
where 2, (u) denotes the new code of 2u) using the swapped factor from the v sample, and similarly for 2, (v). If several factors are to be swapped, then 2, (u) = Σier ()) + Z] (u)o,, where denotes the set of eigenvector indices swap, and 7 is the complement set. The above formulation
5

---

Published as a conference paper at ICLR 2023
source
target
Figure 2: In the factorial swap experiment we modify individual static factors of the source character
to match those of the target. The top shows the gradual change of the hair, skin, and top colors.
is equivalent to the simpler tensor notation (* M = 2/u. I and 2* - Zi, n. where Z Caxtxk is the Koopman projection coefficients of a batch with n samples, and Z represents the swapped coefficients. Thus, the swapped latent code given by 2 Z* O;, where = (6,) is the matrix of eigenvectors organized in columns. A more detailed description of the swaps and how to obtain the disentangled subspaces representations is provided in App. B
5 RESULTS
We evaluate our model on several two- and multi-factor disentanglement tasks. For every dataset, we train our model, and for evaluation, we additionally train a vanilla classifier on the label sequences. In all experiments, we apply our model on mini-batches, extracting the latent codes Z and the Koopman matrix C. Disentanglement tests use the eigendecomposition of C, where we identify the subspaces corresponding to dynamic and static factors, denoted by Jayn and Istar, respectively. We may label other subspaces such as to note they correspond to e.g., hair color change in Sprites. To identify the subspace corresponding to a specific factor we perform manual or automatic approaches (App. (B). Importantly, subspace's dimension of single factor may be larger than one. We provide further details regarding the network architectures, hyperparameters, datasets, data pre-processing, and comparison of computational resources (App. B. Additional results provided in App.
5.1 MULTIFACTOR DISENTANGLEMENT
We will demonstrate that our method disentangles sequential data to multiple distinct factors, and thus extends the toolbox introduced in competitive sequential disentanglement approaches which only supports two factor disentanglement. Specifically, while prior techniques separate to static and dynamic factors, show that our model identifies several semantic static factors, allowing a finer control over the factored items for downstream tasks. We perform qualitative and quantitative tasks on the Sprites (Reed et al. 2015) and MUG (Aifanti al. 2010) datasets to show those advantages.
Factorial swap. This experiment demonstrates that our method is capable of swapping individual content components between sprite characters. We extract batch with 32 samples, and we identify by manual inspection the subspaces responsible for hair color, skin color, and top color, labeled by In Le, Lu. We select two samples from the test batch, shown as the source and target in Fig. 2 To swap individual static factors between the source and target, we follow Eq. Specifically, change the static features the source to be those of the target. For example, the top row in Fig. 2 shows the source being modified to have the hair color, followed by skin color, and then top color of the target, from left to right. In practice, this is achieved via setting 7hh - Zhs Z Zere and assigning [= In] Ziglls In], Zas Ina] Zugt Ina), and 7hzt [ns=] Zege (:. Inst]. where Zurc, Zigt C x40 are the Koopman projection values of the source and target, respectively. The set L.. In U,, and similarly for /nat The tensor Za represents the new character obtained by borrowing the hair color of the target, and similarly for ths and hat In total, we demonstrate in Fig. 2he changes: h- (top), h (middle), and -- (bottom). We additionally show [2 [2]an an example of individual swaps including all possible combinations. Our results display good multifactor separation and transfer of individual static factors between different characters.
gradually
in Fig.

---

Published as a conference paper at ICLR 2023
Factor by Skin & Hair
Skin
peach
50
beige
gray
light gray
light brows
brown
Hair
green
blue
yellow
50
white
pupple
50
Figure 3: We show the SNE plot of the 4D Koopman static subspace which encodes the skin and hair colors. The embedding perfectly clusters all (skin, hair) color combinations.
To quantitatively assess the performance of our approach in the factorial swap task, we consider the following experiment. We iterate over test batches of size 256, and for every batch we automatically identify its hair color and skin color subspaces, In, Ig. Then, we compute a random sampling of Z denoted by J, and separately swap the hair color and the skin color. In practice, this boils down to Th = 2 = Z and setting [: Jn] = 2[J, :, Jn] and similarly, [= Te] = Z :, Tz]- The new latent codes are reconstructed and fed the pre-trained classifier, and we compare the predicted labels to the true labels of [[J]. The results are reported in Tab. [ where we list the accuracy measures for every factor. For most non-swapped factors, we obtain an accuracy score close to random guess, e.g., the skin accuracy in the hair swap is 16.25 65% which is very close to 1/6.. Moreover, the swapped factors yield high accuracy scores marked in bold, validating the successful swap of individual factors.
Table 1: Accuracy measures of factorial swap experiments.
Test
action
skin
top
pants
hair
hair swap
10.51%
16.25% 73.01%
16.33%
35.51%
90.59%
skin swap
10.55%
16.29%
30.55%
17.70%
Latent Embedding. We now explore the effect of model the latent representation of samples. To this end, we consider a batch X of sprites where the motion, skin and hair colors are arbitrary, and the top and pants colors are fixed, for total of 324 examples. Following the above experiment, we automatically identify the subspaces responsible for changing the hair and skin color, In. 4g. To explore the distribution of the latent code, we visualize the Koopman projection coefficients of the Sooppannpror
4- 4-dimensional subspace Ins = I, U Ig given by Z|:. : Ins] E C324x8x4 We plot in Fig. 3 the 2D embedding obtained using E-SNE (Van der Maaten Hinton 2008). To distinguish between skin and hair labels, we paint the face of every 2D point based on its true hair label, and we paint the point's edge with the true skin color. The plot resembles a grid-like pattern, showing a perfect separation to all 36 unique combinations of (skin, hair) colors. We conclude that the Koopman subspace Ina indeed disentangles the samples based on either their skin or hair.
Incremental Swap. In this test we explore multifactor features of time vare-varying Koopman subspaces on the MUG dataset. Given a source image u, we gradually modify its dynamic factors to be those of the target v. In practice, we compute Z ju, Ia] = Iv, :, Ie]. where I, C I Jay is an index set from Jayn such that q {1, 3) and I I2 C Is Layn- Specifically. |41 4, [I2| 6, Iz 32. Fig. Iishww the incremental swap results of two examples changing from disgust to happiness (left), and happiness to anger (right). The three rows below the source row the reconstructions of the gradual swap denoted by (Tg)) == (Z/u, :, Ig]: T). Our results demonstrate in both cases non-trivial gradual change from the source expression to the target, as more dynamic features are swapped. For instance, the left source is mapped to a smiling character over all time samples in x (T2), and then it is fixed to better match the happiness trajectory source in X
7

---

Published as a conference paper at ICLR 2023
disgust to happiness
happiness to anger
source
(I)
(
((1))
target
Figure 4: Our method allows to swap the dynamic features incrementally, and thus it achieves a relatively smooth transition between the source and target expressions.
5.2 TWo FACTOR DISENTANGLEMENT OF IMAGE DATA
We perform two factor disentanglement on Sprites and MUG datasets, and we compare with state-of- the-art methods. Evaluation is performed by fixing the time-varying features of a test batch while randomly sampling its time-invariant features. Then, a pre-trained classifier generates predicted labels for the new samples while comparing them to the true labels. We use metrics such accuracy (Acc), inception score (IS), intra-entropy H (ylær) and inter-entropy (v) (Bai et al. (021). J. We extract batches of size 256, and we identify their static and dynamic subspaces automatically. In contrast to most existing work, our approach is not based on variational autoencoder model, and thus the sampling process in our approach performed differently. Specifically, for every test sequence, we randomly sample static features by generating new latent code based on a random sampling in the convex hull of the batch. That is, we generate random coefficients {az} for every sample in the batch such that they form partition of unity and a, E 1]. Then, we swap the static features of the batch with those of the new samples, Z|:, = epochs of random sampling, and report the average results in Tab. 2 3 Notably,
a;Zli, : Istat.] We perform 300 method outperforms previous SOTA methods on the Sprites dataset across all metrics. On the MUG dataset, achieve competitive accuracy results and better results IS and H (y) metrics. In comparison to unsupervised methods MoCoGAN, DSVAE and R- E-W,,, our results are the best on all metrics.
5.3 TWo FACTOR DISENTANGLEMENT OF AUDIO DATA
We additionally evaluate our model on diffcrent data modality, utilizing a benchmark downstream speaker verification task (Hsu al. 2017 on the TIMIT dataset (Garofolo et al. 1992). In this task, we aim to distinguish between speakers, independently of the text they read. We compute for each test sample latent representation Z, and its dynamic and static sub-representations Zayn: Zstat. respectively. In an ideal two factor disentanglement, we expect Zstat encode the speaker identity, whereas Zayn should be agnostic to this data. To quantify the disentanglement we employ the Equal Error Rate (EER) test. Namely, we compute the cosine similarity between all pairs of latent sub- representations in Zgtat. The pair is assumed to encode the same speaker if their cosine similarity is higher than threshold j0, 1). and the pair has different speakers otherwise. The threshold needs to be calibrated to receive the EER (Chenafa et al. 2008). If Zutat indeed holds the speaker identity,
Table 2: Disentanglement metrics on Sprites.
Table 3: Disentanglement metrics on MUG.
Method
Acct
ISt
Hy]
(v)t
Method
Acct
ISt
(y]rr) (
MoCoGAN
92.89%
8.461
0.090
2.192
MoCoGAN
63.12%
4.332
0.183
1.721
DSVAE
90.73%
8.384
0.072
2.192
DSVAE
54.29%
3.608
0.374
1.657
R-WAE
98.98%
8.516
0.055
2.197
R-WAE
71.25%
5.149
0.131
1.771
S3VAE
99.49%
8.637
0.041
2,197
S3VAE
70.51%
5.136
0.135
1.760
C-DSVAE
99.99%
8.871
0.014
2.197
C-DSVAE
81.16%
5.341
0.092
1.775
Ours
100%
8.999
1.6e-7 2.197
Ours
77.45%
5.569
0.052
1.769
8

---

Published as a conference paper at ICLR 2023
SOurce
target
static swap
dynamic swap
Leig
Latat
Layu
KAE
Figure 5: Our ablation study shows that the full model Leig disentangles data well, whereas models using only Lstat loss only Ldya loss Leig loss at all struggle with swapping static features.
then its EER score should be low. The same test is also repeated on Zayz for which expect high EER scores as should not contain speaker information. We report the results in Tab. 4 Our method achieves the third best overall EER the static and dynamic tests. However, S3VAE and C-DSVAE either use significantly more data or self-supervision signals. We label by C-DSVAE" and C-DSVAE the approach C-DSVAE without content and dynamic augmentation, respectively. When comparing to unsupervised approaches that do not use additional data (FHVAE, DSVAE, and R-WAE), we achieve the best results with a margin of 0.27% and 3.37% static and dynamic, respectively.
Table 4: Disentanglement metries TIMIT.
Method
FHVAE DSVAE R-WAE S3VAE C-DSVAE" C-DSVAE C-DSVAE
Ours
Static EER
5.06%
5.65%
4.73% 5.02%
5.09%
4.31%
4.03%
4.46% 26.78%
Dynamic EERT
22.77% 19.20% 23.41% 25.51%
24.30%
31.09%
31.81%
5.4 ABLATION STUDY
We train different models to evaluate the effect of our loss term on the KAE architecture: full model with Leig. KAE + fistat KAE Layn: and baseline KAE without Leig. All other parameters are left fixed. In Fig. [5] we show a qualitative example of static and dynamic swaps between the source and the target. Each of the bottom four rows in the plot is associated with a different model. The full model (Leig) yields clean disentanglement results on both swaps. In contrast, the static features are not perfectly swapped when removing the dynamic penalty (Lstac). Moreover, the model without static loss (Laya) does not swap the static features at all. Finally. the baseline KAE model generates somewhat random samples. We note that all cases (even for the KAE model), the motion is swapped relatively well which can be attributed to the good encoding of the dynamics via the Koopman matrix.
DISCUSSION
We have proposed novel approach for multifactor disentanglement of sequential data, extending existing two factor methods. Our model is based on a strong inductive bias where we assumed that the underlying dynamics can be encoded linearly. The latter assumption calls for exploiting recent Koopman autoencoders which we further enhance with a novel spectral loss term, leading to an effective disentangling model. Throughout an extensive evaluation, we have shown new disentanglement sequential tasks such as factorial swap and incremental swap. In addition, our approach achieves baae-offthh--a results on two factor tasks comparison to baseline unsupervised approaches, and it performs similarly to self-supervised and weakly- techniques.
There are multiple directions for future research. First, our approach is complementary to most existing VAE approaches, and thus merging features of our method with variational sampling, mutual information and contrastive losses could be fruitful. Second, theoretical aspects such as disentanglement guarantees could be potentially shown in our framework using Koopman theory.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was partially supported by the Lynn and William Frankel Center of the Computer Science Department, Ben-Gurion University of the Negev, an ISF grant 668/21, ISF equipment grant, and by the Israeli Council for Higher Education (CHE) via the Data Science Research Center, Cen-Gurion University of the Negev, Israel.
REFERENCES
Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The MUG facial expression database. In 1 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1-4, 2010.
Hassan Arbabi and Igor Mezic. Ergodic theory, dynamic mode decomposition, and computation of spectral properties of the Koopman operator. SIAM Journal on Applied Dynamical Systems, 16(4): 1096-21266 2017.
Omri Azencot, Mirela Ben-Chen, Frédéric Chazal, and Maks Ovsjanikov. An operator approach to tangent vector field processing. In Computer Graphics Forum, volume 32, pp. 73-82. Wiley Online Library, 2013.
Omri Azencot, Steffen Weißmann, Maks Ovsjanikov, Max Wardetzky, and Mirela Ben-Chen. Func- tional fluids surfaces. In Computer Graphics Forum, volume 33, pp. 237-246. Wiley Online Library, 2014.
Omri Azencot, Wotao Yin, and Andrea Bertozzi. Consistent dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, 18(3):1565-1555 2019.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning, pp. 475-485, PMLR, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.044 2014.
Junwen Bai, Weiran Wang, Yingbo Zhou, and Caiming Xiong. Representation learning for sequence data with deep autoencoding predictive components. arXiv preprint a:Xiv:2010.03/.5, 2020.
Junwen Bai, Weiran Wang, and Carla Gomes. Contrastively disentangled sequential variational autoencoder. Advances Neural Information Processing Systems, 34, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3598):1798-1825 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Steven I. Brunton, Marko Budišié, Furika Kaiser, and Nathan Kutz. Modern Koopman theory for dynamical systems. arXiv preprint arXiv:2102. /2086, 2021.
Marko Budišić, Ryan Mohr, and Igor Mezić. Applied Koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22(4):047510, 2012.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint 1rXiv:1804.03599 2018
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances neural information processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2023
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Mohamed Chenafa, Dan Istrate, Valeriu Vrabie, and Michel Herbin. Biometric system based on voice recognition using multiclassifiers. In European Workshop on Biometrics and Identity Management, pp. 206-215. Springer, 2008.
David Clark, Jesse Livezey, and Kristofer Bouchard. Unsupervised discovery of temporal structure in noisy data with dynamical components analysis. Advances in Neural Information Processing Systems, 32, 2019.
Ido Cohen, Omri Azencot, Pavel Lifshits, and Guy Gilboa. Modes of homogeneous gradient flows. SIAM Journal on Imaging Sciences, 14(3):913 13-999,,,945, 2021.
Armand Comas, Sandesh Ghimire, Haolin Li, Mario Sznaier, and Octavia Camps. Self-supervised decomposition, disentanglement and prediction of video sequences while interpreting dynamics: A Koopman perspective. arXiv preprint arXiv:2 2021.
Suddhasattwa Das and Dimitrios Giannakis. Delay-coordinan maps and the spectra of Koopman operators. Journal of Statistical Physics, 175(6):1107-111 2019.
Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30, 2017.
Tanja Eisner, Bálint Farkas, Markus Haase, and Rainer Nagel. Operator theoretic aspects of ergodic theory, volume 272. Springer, 2015.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. Advances neural information processing systems, 30, 2017.
Garofolo, Lori Lamel, W. Fisher, Jonathan Fiscus, D. Pallett, N. Dahlgren, and V. Zue. TIMIT acoustic-phonetic continuous speech corpus. Linguistic Data Consortium, 11 1992.
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical Variational Autoencoders: A comprehensive review. arXiv preprint arXiv:2008. 22595, 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled Recurrent Wasserstein Autoencoder. arXiv preprint arXiv:2101.07496, 2021a.
Minghao Han, Jacob Euler-Rolks and Robert K Katzschmann. DeSKO: Stability robust control with deep stochastic Koopman operator. In International Conference Learning Representations, 2021b.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with a constrained variational framework. In International Conference Learning Representations, ICLR, 2017.
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and Helmholtz free energy. Advances neural information processing systems, 6, 1993.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In Proceedings of the IEEE international conference on computer vision, pp. 2965-2973 2015.
11

---

Published as a conference paper at ICLR 2023
Tomoharu Iwata and Yoshinobu Kawahara. Neural dynamic mode decomposition for end-to-ena modeling of nonlinear dynamics. arXiv preprint arXiv:2012.06191, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658 PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint (rXivv1412.6980, 2014.
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931.
Tejas Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. Advances in neural information processing systems, 28, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv: 2017.
J Nathan Kutz, Steven I. Brunton, Bingni W Brunton, and Joshua I. Proctor. Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.
Yueheng Lan and Igor Mezić. Linearization in the large of nonlinear systems and Koopman operator spectrum. Physica D: Nonlinear Phenomena, 242(1):4 2013.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv preprint
arXiv: 02991, 2018
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. MixNMatch: Multifactor
disentanglement and encoding for conditional image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8039-8048, 2020.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional Koopman operators for model-based control. arXiv preprint /xxiv::910.0820 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. international conference on machine learning, pp. 4114-4124, PMLR, 2019.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):1 2018.
Alexandre Mauroy and Igor Mezić. spectral operator-theoreticc framework for global stability. In 52nd IEEE Conference Decision and Control, pp. 5234-5239. IEEE, 2013.
Alexandre Mauroy and Igor Mezić. Global stability analysis using the eigenfunctions of the Koopman operator. IEEÉ Transactions on Automatic Control, 61111):3356-336 2016.
Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlin- Dynamics, 41 (1))309-3325 2005.
Igor Mezić. Analysis of fluid flows via spectral properties of the Koopman operator. Annual Review of Fluid Mechanics, 45:357-378, 2013.
Igor Mezic. Koopman operator spectrum and data analysis. arXiv preprint arXiv:1002.07597, 2017.
Dorde Miladinović, Muhammad Waleed Gondal, Bernhard Schölkopf, Joachim M Buhmann, and Stefan Bauer. Disentangled state space representations. arXiv preprint arXiv: 03355 2019.
Ryan Mohr and Igor Mezić. Construction of eigenfunctions for scalar-type operators via Laplace averages with connections the Koopman operator. arXiv preprint 1rX3v:1403.6559, 2014.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. Advances in Neural Information Processing Systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2023
Ilan Naiman and Omri Azencot. An operator theoretic approach for analyzing sequence neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2023.
Samuel Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning
dynamics. SIAM Journal Applied Dynamical Systems, 18(1):558-593, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. Associates, Inc., 2019.
Curran
William T Redman, Maria Fonoberova, Ryan Mohr, Yannis Kevrekidis, and Igor Mezic. An operator thcoretic view on pruning deep neural networks. In International Conference on Learning Representations, 2021.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in neural information processing systems, 28, 2015.
Clarence W Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-127, 2009.
Raif M Rustamov, Maks Ovsjanikov, Omri Azencot, Mirela Ben-Chen, Frédéric Chazal, and Leonidas Guibas. Map-based exploration of intrinsic shape differences and variability, ACM Transactions on Graphics (TOG), 32(4):1-12, 12, 2013.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman invariant subspaces for dynamic mode decomposition. Advances in Neural Information Processing Systems, 30, 2017.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526-1535, 2018
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(1)), 2008
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint vrXiv:1706. 08033, 2017.
Stephen Wiggins, Stephen Wiggins, and Martin Golubitsky. Introduction to applied nonlinear dynamical systems and chaos, volume 2. Springer, 2003.
Sitao Xiang, Yuming Gu, Pengda Xiang, Menglei Chai, Hao Li, Yajie Zhao, and Mingming He. DisUnknown: Distilling unknown factors for disentanglement learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision pp. 14810-14819, 2021.
Masanori Yamada, Heecheol Kim, Kosuke Miyoshi, Tomoharu Iwata, and Hiroshi Yamakawa.
Disentangled representations for sequence data using information bottleneck principle. In Asian Conference on Machine Learning, pp. 305-320. PMLR, 2020.
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pp. 4832 4839. IEEE, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. arXiv preprint arXiv: /002.08296 2017.
Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6538 6547, 2020.
13

---

Published as a conference paper at ICLR 2023
A KOOPMAN THEORY
We briefly introduce the key ingredients of Koopman theory (Koopman, 1931) which are related to work. Consider a dynamical system : + M over the domain M given via the update rule
T++1 = p
where It E M C. R", and & E N is the time index. Koopman theory proposes an alternative representation of the dynamical system y by linear yet dimensonai Koopman operator Kip- Formally,
(xp) fo p((x))
where f: M + C is an observable complex-valuess function, and f o yp denotes composition of transformations. Due to the linearity of Kup, we can discuss its eigendecomposstion, when it exists. Specifically, let Ay E C, tey : M + C pair of eigenvalue and eigenfunction respectively of K.p- i.., it holds that
KoØ = A50j for any j
From a theoretical viewpoint, there is no loss of information to represent the dynamics with sp or with K.g (Eisner al., 2015). Namely, one can recover the dynamics yp from given K.,, operator. Hartman-Grroban Grobman Theorem states that the linearization around hyperbolic fixed points conjugate the full, nonlinear system WWiggins et al. 2003). The latter result was further extended to the entirety of the basin (Lan & Mezić 2013). In practice, various tools were recently developed to approximate the infinite-dimensionall Koopman operator using a fnite-dimensional Koopman matrix. In particular, the Dynamic Mode Decomposition (DMD) (Schmid) 2010) is a popular technique for approximating dynamical systems and their modes. DMD was shown to be intimately related to Koopman mode decomposition in (Rowley et al. (2009). which deals with the extraction of Koopman eigenvalues and eigenfunctions in a data-driven setting. Thus, the above discussion establishes the link between work and Koopman theory since in practice, Koopman module is similar in spirit to DMD. Moreover, it justifies our use of the Koopman matrix to encode the dynamics as well as disentangle
Moreover, the Hartman-
B EXPERIMENTAL SETUP: ARCHITECTURE, DATASETS, HYPERPARAMETERS, AND MORE
B. DATASETS
Sprites. Reed et al. 2015) introduced dataset of animated cartoon characters. Each character is composed of static and dynamic attributes. The static attributes include the color of skin, tops, pants and hair; each contains six possible variants. The dynamic attributes include three different motions: walking, casting spells and slashing, where each motion admits three different orientations: left, right, and forward. In total there nine motions character can perform and 1296 unique characters. sequence is composed of eight RGB image frames of size of 64 x 64. We use 9000 samples for training and 2664 samples for testing.
MUG. Aifanti et al. (210)) share a facial expression dataset which contains image sequences of 52 subjects. Fach subject performs six facial expressions: anger, fear, disgust, happiness, sadness and surprise. Each video in the dataset consists of 50 to 160 frames. To create sequences of length 15 as described previous work (Bai al. 202T) Db, we randomly sample 15 frames from the original sequence. Then, we crop the faces using Haar Cascades face detection, and we resize to 64 x 64 resulting in sequences
11x33x64xx4 for total of 3429 samples. Finally, we split the dataset such that 75% of it is used for the train set, and 25% for the test set.
TIMIT. Garofolo et al. (1992) made TIMIT available which contains 16kHz audio recordings of American English speakers reading short texts. In total, the dataset has 6300 utterances (5.4 hours) aggregated from 630 speakers reading 10 phonetically rich sentences each. For each batch of samples the data pre-processing procedure goes as follows: First, we take the maximum raw audio length in the batch, and we zero pad all samples to match that length. Second, we calculate for each sample its log spectrogram with 201 frequency features calculated by window of 10ms, using Short Time Fourier Transform (STFT). Thus, each batch has its own t (time steps) length, with an average length after padding of = 450. The resulting sequences of dimension x: E R tx201
14

---

Published as a conference paper at ICLR 2023
B.2 DISENTANGLEMENT METRICS
Accuracy (Acc) measures how well a model preserves the fixed features while sampling the others. We compute it using pre-trained classifier (also called judge) which trained on the same train set and tested on the same test set as our model. The classifier outputs the probability measures per feature the dataset. For instance, E outputs one label for the pose and additional labels for each of the static factors (hair, skin, top and pants) for the Sprites dataset.
Inception Score (IS) measures the performance of generator. The score is calculated by first applying the judge on every generated sequence Zi:t. yielding the conditional predicted label distribution p(y (1:)). Then, given the marginal predicted label distribution p(y) compute the Kullback- Leibler (KL) divergence KL (p(y (1:+) p(y)). The inception score is given by:
IS = exp(E, [KL (x(()2))1)) p(y)))
Intra-Entropy (ylzz measures the conditional predicted label entropy of all generated sequences. To obtain the predicted labels use the judge and compute E H (p(y)=t..) where b the number generated sequences. Lower intra-entropy score reflects higher confidence of the classifier C.
Inter-Entropy (1) measures the marginal predicted label entropy of all generated sequences. We can compute Hp(()) using the judge's 's output on the predicted labels {v). Higher inter-entropy score reflects higher diversity among the generated sequences.
Equal Error Rate (EER) is used in the speaker verification task on the TIMIT dataset. is the value of false positive rate or false negative rate of a model over the speaker verification task, when the rates are equal.
B.3 ARCHITECTURE AND HYPERPARAMETERS
Our models are implemented in the PyTorch (Paszke et al. 2019) framework. We used Adam optimizer (Kingma & Ba 2014) and learning rate of 0.001 for all models, with weight decay. Regarding hyper-parameters in our experiments, and Aeig are tuned over {1, 10, 15, 20). .. k2 tuned between and 20, and the threshold for the dynamic loss is tuned over {0.4, 0.5, 0.55, 0.6, 0.65J. The hyper- parameters are chosen through standard grid search.
tuned between 40 and 200 and Areey Apred
B.3.1 ENCODER AND DECODER
Sprites and MUG. Our encoder and decoder follow the same general structure as Bai et al. (202I). First we have the same convolutional encoder in C-DSVAE. Then have uni-directional LSTM. The architecture is described in detail in Tab. where Conv2D and Conv2DT denote a 2D convolution layer and its transpose, and BN2D is 2D batch normalization layer. Additionally, the hyperparameters are listed in Tab. 6, where b is the batch size, k is the size of Koopman matrix, is the dimension of the LSTM hidden state, and #epochs is the number of epochs we used for training. The balance weights Xreey Apred and Xvig scale the loss penalty terms of the Koopman layer, Lror; Lpred and Lrig: respectively. Finally, hi is the amount of static factors, and the dynamic threshold, see Eqs. aand in the main text.
Table 5: Architecture details.
Encoder
Decoder
64 image
Z LSTM(k.A)
Conv2D(3, (2, 2,2, 1) BN2D(32) - LeakyReLU
2w.2D)32,6444.2.. BN2D(64) LeakyRel.U
Comv2DT(A, 256,0 BN2D(256) LeakyReLUU
Conv2D(64,12) 4,2, BN2D(128) UUUU
C2n5 1,0) BN2D(128) LeL
Conv2D/ 128, 2,6, 1) HHN2D(256) LenkyRel.U
Com=2DT:128, 64,4,1 24,
BN2D(644) - LeakyRel.U
Conv2D(256,
BN2D(k) -LeakyReLU
Conv2DT 32, - BN2D(33) LeakyReLU
LSTM(K, k)
3, 1, Sigmoid
15

---

Published as a conference paper at ICLR 2023
Table 6: Hyperparameter details.
Dataset
b
k
h
#epochs
Avee
Apred
Acig
kg
Sprites
32
40
40
800
15
1
8
0.5
MUG
16
40
100
1000
20
1
5
0.5
TIMIT
30
165
400
15
3
1
15
0
uni-directional LSTM module instead of a bi-directional layer. The encoder LSTM input dimension is 201 which the spectrogram features dimension and its output dimension k. The decoder LSTM input dimension is and its output dimension is 201. The hyperparameter values are detailed in Tab, 6
TIMIT.
We design neural network related to DSVAE architecture, but
B.3.2 KOOPMAN LAYER
The Koopman layer in our architecture is responsible for calculating the Koopman matrix C, and it is associated with the accompanying losses Frecy preed Leig. It may happen that the latent codes provided to the Koopman module are very similar, leading to numerically unstable computations. To alleviate this issue, we consider two possibilities. One, use blur filter on the image before inserting to the encoder (used for the Sprites datasets). Two, add small random uniform noise sampled from [0, to the latent code Z, i.e., Z 0.005.,, where N denotes the noise (used on TIMIT). Both options yield more diverse latent encodings, which in turn, stabilize the computation of C' and the training procedure. Finally, we note that our spectral penalty terms Lstat and Layn which compose Leig are stable for a large regime of hyperparameter ranges.
B.3.3 ADDITIONAL DYNAMIC LOss OPTIONS
The proposed form of Layn in Eq. constrains the dynamic factor modulus to an e-ball to promote separation between the static factors located on the point 1 + and the dynamic factors. However, there settings for which Layn may be not optimal. For instance, a dataset may contain measure- preserving dynamic factors, e.g., as in the motion of a pendulum. Another example includes growing dynamic factors, e.g., in ball moving from the center of the frame towards the boundaries of the frame. one has additional knowledge regarding the underlying dynamics, one may adapt Layn accordingly. We consider the following options:
1. Set e = while adding the dynamic loss term to Leig. In this case, Layn penalizes dynamic factors that are inside 8-ball around the point + This option addresses measure- preserving dynamic oscillations in the data.
2. Set = 1 7: 7 O while adding the dynamic loss term to Leig- In this case, Layn penalizes dynamic factors that are inside 8-ball around the point + This option addresses growing dynamic factors.
B.4 DISENTANGLEMENNT PROCESS USING MULTIFACTOR DISENTANGLING KOOPMAN ACTOENCODERS
In what follows, we detail the process of extracting the multifactor latent representation of sample. and in addition, we will demonstrate general swap of a factor between two arbitrary samples. We let X E b be our input batch and x R. be a single sample that we want calculate its multifactor disentangled latent representation. The disentanglement process of into its multiple latent factors representations using our model contains the following steps:
1. We insert X into the model encoder and get the encoder output Z [
2. We compute the Koopman matrix C' for the batch X using the Koopman layer as described in the main text.
3. We compute the eigendecomposition of to get the eigenvectors matrix E Clack In addition, we calculate U V- Now calculate Σ" := 2 V for every E RR 2 stores the coefficients in the Koopman space and they are the disentangled latent representation in our method. Notice that z" = VVVU =
16

---

Published as a conference paper at ICLR 2023
4. We identify the indices that correspond to each latent factor. It may be that several indices represent one factor. We use the identification method of subspaces described in B.3 to extract the indices set. Let I be some latent factor index set. Then, the latent representation of factor 1 for the input x is =[Tx]- For instance, J can be the hair color factor. If we want aggregate a few factors together 7 T U Tn U UTp. where 1 = skin indices, 1, = top indices, LA. hair indices, 1p pants indices. In practice I encodes character identity on the Sprites dataset.
take group of factors,
To conclude, these four steps describe the process of disentangling arbitrary factors in our setup. To demonstrate swap, let us assume we use the Sprites dataset. Let 21; 22 be two samples in X and let us assume we want swap their hair and skin attributes. We will use steps 1, 2 and to extract 1 22 multifactor latent representation ž1, 22. Then, using Step 4, we will identify and extract Ix = I, In. were /, hair indices and I skin indices. Now, we want to swap the latent representations of the hair and skin factors between the sample. To do so, we simply preform & [Ix] = Bg [x] and vice versa 82 [xx] 81
[ in parallel.
To get back to the pixel space, we need to repeat our steps backward. First we need to compute the new 2; after the swap. We will do it using the V matrix calculated in step 3. We compute 27 = = 2 = VV Finally, we can insert Re(21 (1) (e(²)) as inputs to the model decoder and get the desired swapped new samples ãi, 22. Last note, if is some input for the model decoder then must be real-valued, however, is typically complex-valuee since V, U E C Thus, we keep the real part of 2, and we eliminate its imaginary component.
In what follows, we show that Rc(="VL) = 2², and thus feeding the real part to the decoder as mentioned above is well justified. Moreover, similar proof holds for swapped latent vectors, i.e., lm(2) = O. Finally, we validated that standard numerical packages such as Numpy and py Torch satisfy this property up to machine precision.
Theorem 1. IfC E Rx
k is full rank, then Re(=-VU) = x² for any E R where V is the matrix of eigenvectors of C, and U V-1
Proof. It follows that
F VU Σ (2,0,)u3
where Vy is the j-th column of V, and uj is the j-row of U. To prove that Im(= VV)) = 0, is sufficient to show that if 01 and U2 are complex conjugate pair of vectors from V, i.e. Ui V/g then (z, ui vilui is the complex conjugate of (z, U2)u2. First, we have that
k
a1 = (2, = Σ 2/1/u10d Σ
(z, 22
02
where the third equality holds since 01 U2 and the last equality holds since is real-valued. The proof is complete if we show that #1 Ty. since then we have (1,v1) {2, u2)uq. u2. To verify that complex conjugate column pairs transform to complex conjugate row pairs, we assume w.l.o.g that the matrix V can be organized such that nearby columns complex conjugates, i.e., v1 = T2 2ty = Vg. and so on. Let P be the permutation matrix that exchanges the columns of V to their complex conjugates, i.e., it switches the i-th column with the (i 1)-th column, where i is odd. Then VP V. It follows that
(VP)-1= pTV-1 PTU U
namely, the i-th row is the complex conjugate of the (i + 1)-th row, where is an odd number.
B.5 IDENTIFICATION OF SUBSPACES
There are two scenarios in which we need identify semantic Koopman subspaces in the eigenvectors of the Koopman matrix C:
separate between static and dynamic information (two factor separation). 2. identify individual factors, e.g., hair color in sprites (multifactor separation).
17

---

Published as a conference paper at ICLR 2023
Table 7: Accuracy measures of factorial swap experiments, see Tab.
Test
action
skin
top
pants
hair
hair swap 11.35% + 0.65%
17. 17.40% = 7.7% 17.07% + 0.77% 36 36.29% 0.88% 90.20% 0.52% 72. + 0.68% 17.23% + 899% 31.22% 0.84% 16.92% + 0.77%
skin swap 11.35% + 0.65%
Two factor separation. To distinguish between time-invariant and time-varying factors, we sort the eigenvalues based on their distance from the complex value + =0. Then, the subspace of static features Iatt is defined as the eigenvalues' indices of the first k, elements in the sorted array. Then, the dynamic features subspace fayn holds the remaining indices, ie., Layn set of all indices, and S1 S2 generates the set difference of the sets S1 and S2.
Istes. where I is the
Multifactor separation. The identification of individual features such the hair color skin color in Sprites is less straightforward, unfortunately. Essentially, the key difficulty lies in that the Koopman matrix may encode individual factor using subspace whose dimension is unknown priori. In addition, the subspace related to e.g., hair color may depend on the particular batch sample. For instance, we observed cases where the hair color subspace was of dimension 2 and 3 for three different batches. Nevertheless, manual inspection of at typically reveals the role of the eigenfunctions, and it can achieved efficiently as k; I 15 in our experiments. Still, we opt for an automatic approach, and thus we propose the following simple procedure. We consider the power set of Lgsat. denoted by Istat. Let be an element of Iglat. then we swap the content of the batch with respect to J, and check the accuracy of the factor in question (e.g., hair color) using the pre-trained classifier. The subspace which corresponds to a single factor change the one for which the accuracy of the factor decreases the most with respect to the original samples. In practice, we noticed that often the subspace of factor is composed of subsequent eigenvectors in the sorting described for the two factor separation. Thus, many subsets J of the power set Lenat be ignored. We leave further exploration of this aspect for future work.
B.6 SPEAKER VERIFICATION EXPERIMENT DETAILS
The speaker verification task in Sec. 5.3 is performed as follows. We the test set of TIMIT which contains 24 unique speakers, with eight different sentences per speaker. In total there are 192 audio samples. We compute the latent representation Z for this data, and its Koopman matrix C. Using the eigendecomposition of C, we identify the static and dynamic subspaces Lttat and I layn- We denote by Zstat. Zaya the latent codes obtained when projecting Z to Lerat, fayns respectively. Formally, this is computed via Zstat = 7 t -- Totat] ( and similarly for the dynamic features. To perform the speaker verification task we calculate the identity representation code for the batch given by
Zatas
Σ Zstar [: j. :]
Zayn
Σ
Zataz l. j. :], Zayns Zoyn E R 192xx165
j=1
The EER calculations are performed separately for Zerat and Zeyn for all of their 1922 18336 pair combinations.
C ADDITIONAL RESULTS
C. MEAN AND STANDARD DEVIATION MEASURES
We report the mean and standard deviation measures computed over 300 runs for the results reported in Tab. 2 3 in the main text. The results are detailed in Tab. 7 The low standard deviation highlights the robustness of our method to various seed numbers, and the overall stability of our trained models.
C.2 DATA GENERATION
We present qualitative results of our model's unconditional generation capabilities. To this end, we randomly sample static and dynamic features by producing latent code based random
18

---

Published as a conference paper at ICLR 2023
Table 8: Disentanglement metrics Sprites and MUG, see Tabs,
Method
Acct
1St
H(ylx)
H(y)
Sprites 100% + 0% 8.999 + 2.3e 6 1.6e- 2.2e- 2.2e-1
2.197
MUG
77.45% 0.62%
5.569 + 0.026 0.052 0.004
1.769 0
sampling in the convex hull of two randomly chosen samples from the batch. That is, for every sample in the batch we generate random coefficients {a, E [0. I]} which form partition of unity Eje, 02 1, where denotes the sample indices, and |J] combination. Then, we swap the static dynamic features of the source (src) sample using the convex combination, Z[sre, Isaat = Σje, Ot Z Iscee]. 2(src, :, Lynnl Σje, a2 Zi, aynn respectively. The reconstruction of the latent codes for which static or dynamic factors are swapped are shown on the right panels in Figs. datasets demonstrate a non-trivial generation of Tactors while preserving the dynamic/static factors shown on the left panels.
2 the number of samples in the
[3 respectively. Our results on both Sprites and MUG
C.3 TWO FACTORS AND MULTIFACTOR SWAPS
We present several qualitative results of two factor swapping between static and dynamic factors of two given samples. In Figs. andd each odd indexed row E (1, 3,5,77 7 shows the source sequence on the left and the target sequence the right. Even indexed rows j E {2, 4, 6, 8} represent the reconstructed samples after the swap where the left we show the static swap, and the right the dynamic swap. Notably, all examples show clean swaps while preserving non-swapped features.
Additionally, we extend the result in Fig. 2 to show example in which we swap all multifactor combinations. Specifically, we show in Fig. 12 several multifactor swap from the source sequence (top row) to the target sequence (bottom row). The text the left of every sequence in between denotes the swapped factor(s). For instance, the second row with the text shows how the pants color of the target is swapped to the source character. Similarly, the row with the text s+tth is related to the swap of the skin, top, and hair colors.
Figure 6: Unconditional generation of Sprite characters. The left panel shows the source sequences, and the right panel demonstrates the sampled characters where time-varying features are preserved.
19

---

Published as a conference paper at ICLR 2023
Figure 7: Unconditional generation for the MUG dataset. The left panel shows the source sequences, and the right panel demonstrates the sampled identities where time-varying features are preserved.
C.4 STATIC INCREMENTAL SWAP ON MUG
Similarly to Fig. 4 in the main text, we now show an incremental swap example on the MUG dataset where the static features are swapped gradually, see Fig. [O] The multifactor subspaces used in this experiment are of sizes |/1| 1, | 2, |49 5 where 41 I2 C 13 C Lstat. We observe non-trivial gradual change from the source sequence (top row) to the target sequence (bottom row). In each incremental step, more static features changing towards the target samples. Specifically, the skin color, hair color and density, ears structure, nose structure, chicks structure, chicks texture, lips and more other physical characteristics change gradually to better match the physical appearance of the target. Additionally, we observe that the source expression of the source is not altered during the transformation, highlighting the disentanglement capabilities of our approach.
Figure 8: Several static and dynamic swap results on the Sprites dataset.
20

---

Published as a conference paper at ICLR 2023
Figure 9: Several static and dynamic swap results on the MUG dataset.
C.5 KOOPMAN MATRIX SPECTRUM ABLATION STUDY
We would like to explore the impact of our spectral loss on the spectrum and the eigenvalues scattering of the Koopman matrix C. To this end, we train four different models: full model with Leig. KAE + Cetat. KAE Ldyn. and baseline KAE without Leig. We show in Fig. IT the obtained spectra for the various models, where eigenvalues associated with static factors are marked in blue, and the dynamic components are highlighted in red. Our model shows clear separation between the static and dynamic factors, allowing to easily disentangle the data in practice. In contrast, the models KAE and at yield spectra in which the static and dynamic components are very close to each other, leading to challenging disentanglement. Finally, the model Layn shows separation in its spectrum, however, some of the static factors drift away from the eigenvalue 1.
C.6 COMPUTATIONAL RESOURCES COMPARISON
We compare our method in terms of network memory footprint and the amount of data used for the Sprites dataset. We show in Tab. 9 the comparison of method with respect to the other methods. All other approaches use significantly more parameters than our method, which uses 2 million weights. In addition, S3VAE and C-DSVAE utilize additional information during training. S3VAE exploits supervisory signals to an unknown extent as the details do not appear in the paper, and the code is proprietary. C-DSVAE uses data augmentation of size sixteen times the train set, that is, for content augmentation they generated eight times more train data, and the same amount for the motion augmentation. In comparison, our method and DSVAE do not use any additional data on top of the train set.
The time complexity analysis of our method governed by the complexities of the encoder, decoder, the Koopman layer and the loss function. The encoder and decoder can be chosen freely and are
source
(T:)
X(12)
X(1s)
target
F
R
6
Figure 10: An incremental swap result of the static features on the MUG dataset.
21

---

Published as a conference paper at ICLR 2023
Ours
Latas
Ddyn
KAE
imag
a
-1
1
-1
-1
real
real
real
real
Figure 11: The Koopman matrix spectrum of different models.
typically similar to prior work (Hsu al.
2017 Li Mandi 2018 Zhu 2020 Bai et al. 2021
and thus we focus our analysis on the Koopman layer and the loss function. The dominant operation the Koopman layer in terms of complexity is the computation of the pseudo-inverse of Zp (please Section B Computing the pseudo-inverse of matrix is implemented in high-level deep learning frameworks such as PyTorch via SVD. The textbook complexity of SVD is Ominnmmn' m²n)) for x matrix. In addition, computing the loss function involves eigendecomposition. The theoretic complexity of cigendecomposition is equivalent to that of matrix multiplication, which in case is (k22337)), where the Koopman operator is of size k x k. In comparison, the matrices Zp for which we compute pseudo-inverse are of size b 1 x k; and typically k < b t. Thus, the pseudo-inverse operation governs the complexity of the algorithm. The development of efficient SVD algorithms for the GPU is an ongoing research topic in itself. As far as we know, there is some parallelization torch SVD computation, mainly affecting the decomposition Koopman matrices we use are typically small (e.g., 100 x 100), and thus the effective computation time is short.
large matrices. The
Table Computational resources comparison.
Method
DSVAE
WAW
S3VAE self-supervised
C-DSVAE
Ours unsupervised
Type
unsupervised (weakly) unsupervised
self-supervised
Params 21M
121M
11M
11M
2M
Data
labels
supervisory signals data augmentation (x16)
22

---

Published as a conference paper at ICLR 2023
source
t+ss
p+t
h+t
25.
pth
8th
s+p
s+t+h
p+t+s
2R
p+h+t
pth+s
s+t+h+p
target
Figure 12: Multifactor swap of individual static factors and their combinations on the Sprites dataset.
23

---

Published as a conference paper at ICLR 2023
Figure 13: Unconditional generation of Sprite characters where the static factors are kept fixed, and the dynamic features are randomly sampled.
25
25
F
Figure 14: Unconditional generation of MUG images where the static factors are kept fixed, and the dynamic features are randomly sampled.
24


Reviews:
[{"review_id": "cPEF333OOM", "reviewer": "Reviewer_tz1i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A method for disentangled representation learning from sequential data is proposed. It is based on the eigendecomposition of an estimation of the Koopman operator of dynamics. The method is demonstrated with standard benchmark datasets.", "summary_of_the_review": "This is a nice paper introducing an interesting idea for static/dynamic disentanglement of sequential data. While the discussion could be deepened in a few aspects, the current paper looks sufficiently good for publication in ICLR.", "strength_and_weaknesses": "### Strengths\n\n- The idea of using Koopman operator's spectrum for static/dynamic disentanglement is very interesting and reasonable.\n- The proposed method is examined against different datasets with different evaluation metrics.\n\n### Weaknesses\n\n1. Despite the nice reasoning for static/dynamic disentanglement, multifactor disentanglement is not well supported by theoretical arguments. I do not think it is a fatal flaw, but it is certainly a weakness as one of the main claims of the paper is the capability of multifactor disentanglement.\n\n2. The definition of $\\mathcal{L}_\\text{dyn}$ penalize $| \\lambda |$ regardless of $\\angle \\lambda$. This sounds strange because even if $\\vert \\lambda \\vert=1$, dynamic modes with $\\angle \\lambda \\neq 0$ represent (energy-preserving) oscillations, which would be regarded as dynamic factors. I guess, in the presented experiments, the current definition of $\\mathcal{L}_\\text{dyn}$ worked as expected only because such energy-preserving oscillations were not dominant in the datasets.\n\n### Other comments, not necessarily weaknesses\n\n3. Multifactor-ness only holds for static factors. While it might be a limit of the proposed method, it is understandable because, in many datasets, dynamic factors correspond to dissipative dynamic modes (with, say, $\\vert \\lambda \\vert < 0.9$) that disappear rapidly, and thus considering each component of possibly-multiple dynamic factors is very difficult.\n\n4. The subspace identification process in Section B.5 should be, even if very briefly, mentioned somewhere around Eqs. (6)--(8) because it would be a big question when reading there.\n\n5. Just before Eq. (3), $\\phi$ should be called as a *left* eigenvector, instead of just saying eigenvector.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aducg51g2_", "reviewer": "Reviewer_oLc4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new autoencoder method for sequence disentanglement. This work extends the prior works to handle multi-factor sequential disentanglement. While keeping the general encoder and decoder structure, the model adds a Koopman module for the latent space. This work holds the assumption that the underlying dynamics can be represented linearly in the latent space. The Koopman layer computes the Koopman matrix and uses eigenvectors as static (eigenvalue 1) or dynamic factors (others).", "summary_of_the_review": "This is a novel work with noticeable contributions. The goal is clear and the work improves over existing works. But the computation cost and some details (hyper-params, thresholds) can be elaborated on.", "strength_and_weaknesses": "Pros:\n1. Handling multi-factors is a very tough challenge. This paper proposes a very interesting idea to address this problem.\n2. The experiments are thorough and the performance looks promising.\n3. The structure of the paper is clear and informative.\n\nCons:\n1. How do you decide length T?\n2. While I think the claim \"eigenvectors of the matrix C whose eigenvalue is 1 represent time-invariant factors\" does make sense, how do you set thresholds and decide whether the eigenvalue is close to 1 or not?\n3. How costly is the whole eigendecomposition?\n4. Deriving a new C for each Z might cause overfitting. Have you considered stationary assumptions?\n5. For more real-world problems where there might not be explicit semantic meanings, how do you map the eigenvectors to semantics?\n6. Exploring the predictiveness of past for future has been seen in some prior works like predictive information [1,2]. You can also discuss the similarities and differences.\n\n[1] Clark, D., Livezey, J. and Bouchard, K., 2019. Unsupervised discovery of temporal structure in noisy data with dynamical components analysis. Advances in Neural Information Processing Systems, 32.\n\n[2] Bai, J., Wang, W., Zhou, Y. and Xiong, C., 2020, September. Representation Learning for Sequence Data with Deep Autoencoding Predictive Components. In International Conference on Learning Representations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2ikkLImAikA", "reviewer": "Reviewer_WaMN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for disentangling data into latent factors of variation by enforcing structure within Koopman matrices used to model the data's time evolution. This procedure allows for disentangling the static and dynamic components of the data, and furthermore disentangling features within those components (e.g. hair color, skin color, etc.). Experiments show the ability to use trained models to swap features within images and improve on existing approaches in benchmark tasks.", "summary_of_the_review": "I found this paper to be interesting and of good quality. I thought this was a good extension to prior Koopman-based modeling approaches and seems to provide value within the field of data disentangling.\n\nA few comments/questions:\n- In the loss penalty applied to the dynamic eigenvalues we seem to be explicitly imposing that the eigenvalues are less than one. In a dynamical system, this corresponds to having no unstable modes. For image data, how would having an \"unstable\" mode with eigenvalue magnitude >1 manifest itself? Do you ever see this in cases where the loss penalty isn't applied?\n- This paper is working with datasets where there are clear distinctions between attributes within the dataset (especially in the sprite dataset). Can you set the number of static modes to be the number of factors that vary within the dataset (i.e. if only hair color, skin color, and shirt color vary, can you get away with only using three static modes)? If not, why do you think that is?\n- In dimensionality reduction techniques that rely upon SVD, it is usually assumed that the modes with the largest eigenvalues are most important, i.e. they explain most of the variance in the data. Did you see anything similar in your experiments? If you were working with a dataset for which you had less a priori knowledge about important attributes, do you think you would be able to discover them through looking at the dominant modes?\n- What role does the LSTM play in the encoder architecture? It would be possible to perform dimensionality reduction with the convolutional layers alone. Given that the temporal relationship between encodings is being modeled by the Koopman operator and not the LSTM, why do you need to run the encodings through an LSTM first?\n- In the appendix it states \"Thus, we keep the real part of z, and we eliminate its imaginary component\". This seems like it would be throwing out some information? Why do you think this is justified, and did you try alternatives like calculating the magnitude, feeding both the real and imaginary components to the decoder, etc.?", "strength_and_weaknesses": "Strengths:\n- To my knowledge the work is a novel contribution to two field (Koopman-based modeling and disentangling of complex data)\n- Experiments were interesting, relatively extensive, and served as a good illustration of the benefits of the proposed method\n\nWeaknesses:\n- The process of multifactor separation seems potentially cumbersome and I'm not sure how easily it could be extended to different problems.\n- A few questions about the algorithm itself that should be clarified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%