Continual Unsupervised Disentangling of Self-Organizing Representations
Authors: Zhiyuan Li
Xiajun Jiang
Ryan Missel
Prashnna Kumar Gyawali
Nilesh Kumar
Linwei Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: continual disentanglment
generative model
VAE
SOM

Abstract: Limited progress has been made in continual unsupervised learning of representations, especially in reusing, expanding, and continually disentangling learned semantic factors across data environments. We argue that this is because existing approaches treat continually-arrived data independently, without considering how they are related based on the underlying semantic factors. We address this by a new generative model describing a topologically-connected mixture of spike-and-slab distributions in the latent space, learned end-to-end in a continual fashion via principled variational inference. The learned mixture is able to automatically discover the active semantic factors underlying each data environment and to accumulate their relational structure based on that. This distilled knowledge of different data environments can further be used for generative replay and guiding continual disentangling of new semantic factors. We tested the presented method on a split version of 3DShapes to provide the first quantitative disentanglement evaluation of continually learned representations, and further demonstrated its ability to continually disentangle new representations in benchmark datasets.

Published as a conference paper at ICLR 2023
CONTINUAL UNSUPERVISED DISENTANGLING OF SELF-ORGANIZIN REPRESENTATIONS
Zhiyuan Li', Xiajun Jiang', Ryan Missel', Prashnna Kumar Gyawali", Nilesh Kumar', Linwei Wang'
Rochester Institute of Technology', Stanford University?
(217904, xj7056, rxm 2 44, nk4856, Linwei pgyawali@staanfrdd.ccm
ABSTRACT
Limited progress has been made in continual unsupervised learning of represen- tations, especially in reusing, expanding, and continually disentangling learned semantic factors across data environments. We argue that this is because existing approaches treat continually-arm they are related based their underlying semantic factors. We address this by a new generative model describing a topologically-connee and-slab distributions in the latent space, learned end-to-end in continual fashion via principled variational inference. The learned mixture automatically discovers the active semantic factors underlying each data environment, and accordingly accumulate their relational structure. This distilled knowledge can further be used for generative replay and guiding continual disentangling of sequentially-ang arrived semantic factors. We tested the presented method on a split version of 3DShapes to provide the quantitative disentanglement evaluation of continually learned rep- resentations, and further demtedtrated its ability to continually disentangle new representations and improve shared downstream tasks in benchmark datasets.
data independently, without considering how
mixture of spike-
INTRODUCTION
The progress in continual learning has been mostly made for supervised discriminative learning. whereas continual unsupervised representation learning remains relatively under-explored (Rama- puram et al., 2020 Achille et al. 2018 Rao et al., 2019). The few existing works have primarily focused on battling catastrophic forgetting in the generative performance of a model: for instance, a common approach known as generative-relaay synthesizes past samples using a snapshot of the generative model trained from past data, and then continually trains the model to generate both new data and synthesized past samples (Achille et al. 2018 Rao et al. 2019 Ramapuram et al. 2020).
There is however another important yet under-explored question in continual unsupervised repre- sentation learning: how to reuse, expand, and continually disentangle latent semantic factors across different data environments? These inherent in the human learning process: while learning from new data (e.g., learning cars after bicycles), we are naturally able to reuse shared semantic factors without re-learning (e.g., wheels), expand and disentangle new semantic factors (e.g., the shape of cars), while accumulating knowledge about the relationship among data environments based on these semantic factors (e.g., bicycles and cars both have wheels but are different in shapes). Disentangled representation learning, as Long-standing research topic, has demonstrated various benefits in
generative modeling and downstream tasks (Higgins et al. 2017
Kumar et al.) 2017 Kim & Mnih
2018, Liu et al. 2021 Rhodes & Lee 2021 Horan et al. 202T
With increasing recent interests
in unsupervised representation learning a continual learning setting (Rao et al. 2019 Madaan al., 2021), it is important to investigate the challenges and solutions achieve disentanglement of scquentially-arre arriived semantic factors in streaming data.
Reusing latent dimensions for learned semantic factors has mainly been attempted by teacher-student like approach where the student model taught to infer and generate similarly to a snapshot of the past models (teacher) on replayed data (Achille et al., 2018; Ramapuram al. 2020). In Achille et al. (2018), this is further facilitated by explicilly masking out latent dimensions that are not actively used in data environment, Such masks however have be heuristically defined before training on

---

Published as a conference paper at ICLR 2023
Continual
Encoder
Decodar
Topplegicllly-oomnette mixture of splkk--an-ssaa distributions
active
factore
relational atructure On SON
1O000000
Figure I: Through a self-organizing spike-and-slan mixture, CUDOS continually distills knowledge about the relational structure of data environments with their shared and distinct semantic factors.
the new environment. How to automatically discover latent dimensions explaining active semantic factors underlying each data environment, in continual fashion, remains open question.
Expanding learned semantic factors, in part, results naturally from continually optimizing a generative model to newly arrived data (Achille et al. 2018 Ramapuram et al. 2020). or even progressively
increasing the model's latent capacity on the same data (Burgess et al. 2017 Li et al. 2020). These alone however would not uncover the relational structure among data environments in the latent space. In Rao et al. (2019). a mixture of Gaussians has been used and continually expanded such that newly arrived data clustered to existing expanding data distributions, it does not consider the reuse of semantic factors among data clusters.
mixture component. While this captures the
Continually disentangling semantic factors, until now, is limited to the native disentangling ability inherent in VAE, or promoting the reusing of shared semantic factors (Achille et al. 2018 Ramapuram al. 2020 While the common strategy of generative replay teaches model what latent dimensions to use for shared semantic factors on the replayed data (Achille et al.) 2018 Raaaauram et al. 2020). no such guidance is available on new data. As result, as we will show, none of the existing approaches can prevent newly-learned semantic factors to be entangled with re-used ones.
In this work, we show that the above limitations boil down to a fundamental bottleneck in existing continual unsupervised learning of representations: that the learner is asked to treat continually-arrive data independently, without knowing how they are related based on the underlying semantic factors. To overcome this, we argue that the model needs to learn two critical knowledge: latent dimensions explaining active semantic factors underlying each data environment, and the relationship among the latter based on the former. We present Continual Unsupervised Disentangling of self self-OOranizzin representations (CUDOS) that is able to accumulate the relational structure of continually-arrive data based their underlying active semantic factors, and exploiting this knowledge guide disentangling of sequentially- -arrived semantic factors. As illustrated in Fig. 1 to accumulate the relational structure of the data, we model the latent representations with topologically-comee mixture of distributions via Bayesian self-organizing maps (SOM) (Kohonen) 1990] Yin & Allinson 2001). To automatically discover active semantic factors underlying each data environment, we model each component of the SOM mixture with boike-and-slate distribution (Titsias & Lazaro-Grede 2011 MoITTTonnolin et al. 2020), such that the sparse spike variable identifies latent dimensions explaining active semantic factors. This results in a generative model with self-organizing mixture of slab- and-spike distributions, where the distilled knowledge the relational structure of data environments and their associated active semantic factors supports 1) mixture-based generative replay and 2) semantic factors.
continual disentangling of sequentially-
We evaluated CUDOS on both benchmark datasets for continual representation learning, and split version of 3DShapes (Burgess & Kim 2018) designed for quantitative evaluation of disentangling siieentially-arrivee semantic factors. In comparison to existing works, we showed that CUDOS not only addressed catastrophic forgetting, but also improved both quantitatively and qualitatively - continual disentanglement of latent semantic factors and thereby downstream discriminative tasks.
2

---

Published as a conference paper at ICLR 2023
2
RELATED WORKS
Deep Learning with SOM:The use of SOM has been explored within deep learning of image classification (Liu et al. 2015) image clustering (Manduchi et al. 2019 Forest al., 2000), and time-series prediction (Jin et al. 2015 Fortuin et al. 2019). None of these works, however, considered continual learning of representations. In the context of continual learning, SOM was used to first learn the relationship among all discriminative tasks, and then used to mask the neurons of a fully- connected layer for each task (Bashivan al. 2018). A dual cuaa-m self-organizing architecture was also presented for learning abject instances and categories in life-long object recognition (Parisi jet al., 2)18). While both works shared our motivation to leverage the ability of SOM to maintain and expand a memory of data distributions across environments, neither considered continual unsupervised representation learning, or utilized SOM as a topologically-comeee
Bayesian mixture model.
Inferring Active Semantic Factors in VAE: Learning meaningful representations is vital task for VAE, where large latent space often leads to latent dimensions that carry little information (Chen et al. 2017). Sparse coding and discrete latent space have proved to be elegant solutions (Oord et al. 2017 Tonolini et al.) 2020). Non- Non-parametric discrete densities have been explored for stochastic latent activation improve disentanglement (Gyawali et al., 2019). A discrete latent space based on vector quantization was shown to solve posterior collapse (Oord et al., 2017), whereas sparsity was directly modeled in continuous latent space using apike-and-slab priors (Tonolini et al. (2020) None of these concepts has been extended to continual unsupervised learning of representations.
Continual Unsupervised Representation Learning: Most existing works in continual unsupervised
representation learning relied on enhanced generative replay to combat catastrophic forgetting of generation using learned semantic factors (Achille et al.) 2018 Ramapuram al., 2020 Ye & Bors] 2020). The two most related works are those presented in Achille et al. al.j(2018)) 8) and Rao et al. (2019). In Achille et al. (2018), to reuse latent dimensions explaining semantic factors shared with past data, mask for active latent dimensions specific to each data environment was used. This mask was heuristically determined before training on the new data, which will affect the learning and sharing of latent dimensions on the new data. Furthermore, while this strategy protects latent dimensions specific to past data environments (not shared and thus turned off), it does not prevent new semantic factors from being entangled with the shared dimensions. CUDOS presents fundamental solutions to these problems by 1) principled variational inference of active latent dimensions leveraging alab-and-spike priors, and 2) guiding continual disentangling of sequentially-arrive semantic factors by exploiting the relational structure of data.
In Rao et al. (2019), , mixture of Gaussians was continually expanded as new data arrive. This shares our intuition that continual learning of representations can benefit from accumulating an evolving summary of data in the latent space. How the different data clusters are related in terms of shared and distinct semantic factors, however, was not exploited in Rao et al. (2019). In contrast, CUDOS exploits the relationship among data clusters based on their underlying active semantic factors, and uses that to facilitate continual reuse and disentangling of sequentially-anni arrived semantic factors.
3 METHODS
We first establish the foundation for CUDOS: a VAE with a self-organizing mixture of spike-and-slab priors to learn the relational structure of data based on their active semantic factors (Section We then describe how to use this data summary for generative replay (Section 3.2.2). and use the relation between new and past data to improve continual disentangling of representations (Section 3.2.3..
3.1 LEARNING THE RELATIONAL STRUCTURE OF DATA VIA ACTIVE SEMANTIC FACTORS
Fig. [ outlines the foundation model underlying CUDOS. Given data x, are interested in learning representations of meaningful semantic factors within a latent vector z E R'. While doing so we encourage sparse coding to discover latent dimensions explaining active semantic factors in VAE (Tonolini et al. 2020). while learning the relational structure of data based on these semantic factors.
Generative Model: To accumulate the relational structure of data, we model the latent space by mixture of distributions using Bayesian-SOM (Yin & Allinson 2001) with K nodes. Additionally, to discover the active semantic factors for each data environment, model each mixture component as
3

---

Published as a conference paper at ICLR 2023
spike-and-slaie distribution that encourages sparsity in the latent dimensions This gives rise to the following generative process:
(Tnoliii et al. 2020).
w Cat(T)) Por u = 1) = II
Ia (o1))) - a=))((()22)
j=1
Po (z) Σ [(lz)wa)p(un) x~ po(zxx))) k=1
Pa.v (x, 2x = pe ( (x/z)pp (z)w)
where * denotes the j-th element of x, and 8(x) Dirac delta function centered at zero. The mixing prior p(w) is parameterized by æ, n; 2 and Σ n. The latent variable z from the k-th mixture component is parametrized by 4/x (14; of. OC2, } where parameter 0r introduces sparsity to mask out inactive dimensions. Data x are generated from z via a neural network parametrized by
Inference Model: We define variational approximations of the posterior density p(z, w/ x) as:
q(z, w|x) (((z)))) go(z|x) II o N + j=1
(w = 1/2) p(z wk. 4h) = =)
E w vo' )p( ww = 1)
where py (wn = 1|z) is the posterior probability of the k-th mixture component. For parameters ju and logø² of the spike-and-slab distribution, their inference amortized as the output of an encoding network parameterized by Φ. For parameter ã, considering that similar data should share latent dimensions for common semantic factors, we infer at a set level as inspired by the neural statistician (Edwards & Storkey 2017, Hewitt et al. 2018). We discuss how to determine the set in Section 3.2.1
Variational Inference: The parameters v/, 8, and & are optimized by the ELBO loss as:
p8, p w) (x, [log po p (z)w))(w)) (2(2, wxx)
log log 2 CELBO (e, [log
4o P (wlz) (z) [log (x|22) E (x) (z) (2) E (x DxL (p (10 (p)))))(((())))))
(3)
The first reconstruction term is similar that in the vanilla VAE. The second term measures the -i--divegeence between (z(x)) and its conditional prior, measured over the posterior distribution of the SOM mixture Pv (w) (w)z). Specifically, we estimate the expectation over Pyp (w|z) as:
K
(mlp)) (Potmlo)DDaaqqa (
Σ (ww = Dx)e) 1)], km
(4)
where Pe (wxlz) be computed in batch during forward propagation, and DKI(go (z) (z/wk= = 1)] can be derived following (Tonolini et al. 2020 as:
Σ Lld (log of + (id' -m1)) + (60)2 2(02)2
}
+11 ā²) log 1 as
+ ā log
gd
ag.
This KL loss encourages (z(xx) follow mixture density with the mixing probability determined by the posterior probability of each component given z. Note that the latter automatically considers sharing of semantic factors between the inferred z and each mixture component (via spike variable ā and ).)) The third term in Eqn. 33) measures the KL-divergence between the posterior density of w and its prior (set to be uniform in this work). The expectation estimated by Monte Carlo samples.
Iterative Optimization: We maximize the ELBO loss as defined in Eqn. 3 by iterative optimization. In each iteration, we first fix &b of the SOM mixture and maximize Eqn. (3) with respect to the VAE's parameters 0 and à by stochastic gradient descent with reparameterization trick (Kingma & Welling 2014 Tonolini et al. y0200): at the first iteration, the SOM-mixture is initialized as a uniform mixture of m (0, 7, 0.2) and the optimization becomes standard ELBOO with spike-and-slab priors.
4

---

Published as a conference paper at ICLR 2023
A (latizd spare)
Parv
dalaa
E.s
SOM
Gatt
Gzaer
Figure 2: (a) Combating catastrophic forgetting by generative replays from Continually disentangling by using the relation between new and past data based on their underlying shared semantic dimensions.
and (b)
With the updated / and 6, we then maximize Eqn. K with respect to the SOM-mixture parameter & which, as derived in Appendix-l, amounts to maximizing the expectation of the log-likelihood of Py (z) over the variational posterior distribution of (z(xx)) o0 = yx.. [log (z)] We follow the theory Gepperth & Pfülb (2021 to optimize vp using stochastic gradient descent.
3.2 CONTINUAL. LEARNING WITH CUDOS
We now consider setting of continual unsupervised learning where the label of the underlying data environments is unknown. Following existing approaches (Achille et al.) 2018 Rao et al.) 2019 Ramapuram al. 2020), maintain snapshot of the model parameters (olld Baid Potal every training steps. These model snapshots are used to guide (1) synthesizing replayed samples to teach the model to perform consistently on past data (Section 3.2.2), and (2) continually disentangling sequentially- arrived semantic factors using the relationship between past and new data (32.3)).
3.2.1 INFERRING a FOR STREAMING DATA
As mentioned in Section 3.1 we choose to infer at a set level to leverage shared information underpinning set of data. If the boundary between data environments is known, & can be shared for all data within the same data environment. Alternatively, if the boundary of data environments is not known, assume â to be shared within each mini-batch Fbach- Specifically, we maintain 6a = {ā1, á:2, āa) ) for d number of sets with distinct underlying semantic factors. Given a new batch of Thanch, first determine if it can be described by an existing d or new d I has to be allocated if the existing d. fails to describe Bbatch well:
à,
if Ep(zj,, aa) [c,] s Ta:
new
( otherwise,
where à = argmin ge Ep(zja, [C,], Ta is a threshold for the reconstruction error C, of Tbathh averaging over pixels. Note that Equation (5) only determines how data batch Whatch is associated with a variable & The values of these variables are optimized during variational inference.
3.2.2 GENERATIVE REPLAY WITH CUDOS
To combat forgetting, we synthesize data samples following the generation process as defined in Eqn. ( using the snapshot of the SOM mixture with parameters Vooaa As illustrated in Fig. 2 a), on the synthesized samples Xald and their corresponding latent samples Zp. we encourage the model to be consistent with its past snapshot (Ramapuram et al. 2020 Achille al.) 2018 Lezamal 2019
Cold [S (x)ppe(()) ()) Xola] + L.[S( Rota)) zp].
where C. is mean squared error and S(-) is sampling process. This strategy combines the two key existing concepts replay mechanism: as generative replay (Shin al. 2017 Ramapuram et al.) 2020), data are synthesized quickly and readily with limited burden on storage; as in core-set methods (Nguyen et al., (2018) Borsos et al., 2020), the Mom-mixture provides representative summary of data ensuring that more important mixture components are more frequently re-used in future training.
3.2.3 GUIDING CONTINUAL. DISENTANGLEMENT
Generative replay lacks mechanisms to teach the model how to disentangle newly-arrived semantic factors from latent dimensions already used for previously-learned semantic factors. CUDOS provides
5

---

Published as a conference paper at ICLR 2023
unique opportunity to address this issue by its ability to describe the relation between and past data. For new data x, its shared latent dimensions with SOM summary of past data (parameterized by vatal is determined by mask Saha computed by scaled and displaced Sigmoid function:
Sigmoid(b(á') o Trguu 0.5))
where c is the spike variable inferred from x and Kenar.var the spike variable of the best-matching unit UBMU (the component with the largest posterior) on the snapshot of the past SOM. b scales and sharpens the Sigmoid function towards gated function (Toollii et al. 2020). Note that neither Saha or UBmU is fixed; they are functions of the unknown spike variables di and aeemm-vas- We now use this (1) maintain consistency on latent dimensions for shared semantic factors (if any), and (2)
prevent entangling new semantic factors into the shared latent dimensions.
Reusing Shared Semantic Factors: To teach the model reuse latent dimensions corresponding to previously- semantic factors on new data x, we ask the VAE encoder 4s to be consistent with its past snapshot gosu in the shared dimensions when inferring from the new data:
Cnewz = Le(5(g9(z)x)) O Schwer (qgoua (z|x)) C Syaall
(8)
where is element-wise multiplication.
Disentangling New Semantic Factors: Now since the past model does not necessarily know how to generate new data how does it teach the new model? Intuitively, given the relation between x and the past data as determined in Eqn. 7 even though the past model does not know how to generate the new would know how to generate from those shared semantic dimensions in Therefore, as illustrated in Fig. b)) if we take a sample zuamu, from the mixture component UBMU, and replace its latent values with those inferred from x at the shared latent dimensions, the model should know how to generate from this combined latent vector Zoom in a way consistent to the past generator:
ZugMU Aass
S(p (z) (B(B)))) Zeom Zuguus Vou ? (1 Swan) + S(qo(z)x))
Swar
Cnowx = (e(S (x) (x)z)mm)) )), (x(2<o)))))
(10)
Intuitively, if the model entangles new semantic factors into the shared dimensions, it will be penalized as the past decoder does not know how to generate from these new factors. As we will demonstrate, this constraint - uniquely made possible by CUDOS is critical in continual disentanglement.
Summary: summary, the overall loss for CUDOS is:
C = CELBO + 7i Cold + Tnewe + 7a
(11)
where 71, 72, and 73 weighting hyperparameters. This extends the foundation objective function to continual learning settings, which promotes the sharing of shared semantic factors between new and past data environments, and the disentangling of new semantic factors.
EXPERIMENTS AND RESULTS
We evaluated CUDOS (1) split version of 3DShapes (Burgess & Kim 2018) for quantitative evaluation of continual disentanglement, (2) (NIST((eCCn) et al. 1998). Mushion-MNISTT [iao et al. 2017), and their moving versions in (Achille al. (018). and split-CelebA (Liu et 2015).
Split-3DShapes: We quantitatively evaluated the continual disentanglement of past and new repre- sentations in a split version of 3Dshapes with two sub-sets: The first one only had red floor and wall, and the second added all floor colors except red. A successful continual learning of representations is expected to continually learn the new factor of floor color while reusing the others learned in the first set. We compared CUDOS to four groups of baselines: (1) naive VAE (Kingma & Welling 2014). naive VAE (Chen et al. 2018), and VAE with gradually increased capacity (Burgess al. 2017) without an explicit mechanism to combat catastrophic forgetting, (2) unsupervised continual learning reliant on generative replay (Achille et al. 2018:11 Ramapuram et al. 2020) and deuristically-definee
masks of active latent dimensions (Achille et al.) 2018). (3) unsupervised continual learning using a mixture of Gaussian in the latent space (Rao et al. 2019), and (4) a continual learning version of
VQ- VQ-VAE (Oord et al. 2017) (a prototype-basee method similar to SOM) and IC-VAE (Chen et al.)
2018) (a representative disentangling VAE) with replay mechanism Lola- Experiments on each model executed at least 5 times.

---

Published as a conference paper at ICLR 2023
Na variation of floor color and wall color
Continual learning added semantic factor of floor color only Luy
Inactive activve dimeessons
(A)
Figure 3: Continually learning a split version of 3DShapes where the variations of floor colors were absent initially and appeared later. Black bounding boxes annotate which latent dimensions the new semantic factors are learnt. (a): Traversing each latent dimension after training on data with no floor color variations.
(b): Traversing each dimension after training on the data with floor color variations, where comparison of CUDOS with baseline methods shows the improved ability to disentangle new semantic factors into previously inactive dimensions.
Method
MIG f
MIG-sup 1
[Zpui factors
reconstruction loss
Naive VAE
0.229
0.228=0.0066
1.220=00.29
0.351=0.044
Naive TC-VAE
0.295 141
0.377=0.169
1.113=+0.15
0.355 144
Burgess al.
2017
0.192.10.09
0.101
1.154=10.02
0.300=10.037
Continual TC- VAE
0.138
0.254:10.109
1.312-10.668
0.006:10.004
Continual VQ-
0.115:10.045
0.226-10.045
1.470 +:1233
0.008
Achille et al.
2018
0.162+40.057
0.234:10.078
0.639-10.200
0.025-10.110
Rao et al. (011
0.095
0.091
0.886 084
0,042=1.015
Ramapuram al. 2020
0.1056
0.296=10.073
1.115=0.222
0.003-0.003
CUDOS
0.242.10.047
0.32610.053
0.024=0.029
0.021=0.013
Table 1: Quantitative metries for disentanglement. MIG: mutual information gap (Chen et al.
2018). MIG-sup: supplement mutual information gap (Li al. 2020). ; factorS mutual information between already active dimensions and the new factors. reconstruction loss: the relative change for reconstructing past data.
Choosing suitable disentanglement metrics vital as different metrics may measure different aspects of the disentanglement (Locatello et al.) 2020 Zaidi et al. 2022). Here we consider two types of metrics. First, to form a complete measurement of the one-to-one relationship between latent dimensions and semantie factors, chose MIG-sup (Li et al. (220)) that penalizes learning multiple semantic factors into the same dimensions in combination with the complementary MIG (Chen et al.) 2018). Second, focus the disentanglement of eequentiallyaariieee semantic factors, compute the mutual information / (Zppst: factors between active dimensions for past data and new factors the new data. Ideally, if model manages disentangle new semantic factors into dimensions not used by previous data, / (Zpasi factors should be close to 0. Finally, we also compute the relative change of reconstruction loss for past data to measure forgetting.
As shown in FigB: 3(a, major semantic factors such as object shape, size, and color were learned from the first subset. When new data stream introduced (shown in Fig while all models were able to reuse most of the latent dimensions corresponding to previously learned semantic factors, all baseline models entangled new semantic factors the floor color with these dimensions. In contrast, CUDOS was able to not only reuse shared latent dimensions. but also disentangle previously inactive dimensions. Note that we include results of Ramapuram et al. (2020) in Fig as an example of the baseline models that have constraints traversing results of other comparison models can be found in Appendix [ Additional results on different sequences of split-3DShapes can be found in Appendix !I]
ones into
replayed data, annotated by Loss- Visual
Quantitatively, as summarized in Table. CUDOS witnessed the lowest I I(Zpasi factorsnew) among all comparison models, well as the best disentanglement performance measured by MIG and MIG-

---

Published as a conference paper at ICLR 2023
(b)
Figure 4: SOM results on plitt3DDhhaess (a) SOM prototypes learnt after the first (left) and second (right) data environments. Black boxes label prototypes associated with current (black) and replayed data samples (blue), while the rest the prototypes are mixture-interpolltee. (b) Average spike parameter 0g.. Prototypes of the second data environment (black) shared active dimensions from the first data environment (blue) with added dimensions. Gray represents inactive dimensions.
Moving Fashisn-MN1IS
Moving MNIST
MNIST
3DShapes
baselines
CUDOS
Scale R
0.05++0.07
0.58:-0.13
Orientation R²
0.08+-0.05
0.93
A Scale
-0.92:10.10
-0.13:0..13
A Orientation
-..1110.15
-0.0010.011
Moving-MNIS
baselines
CUDOS
2
2
X-pos R f
0.67+10.9
0.70=10.06
2
2
Y-pos R f
0.75+10.05
0.66:10.23
A X-pos
13
0.054=
A Y-pos
0.01
-0.15+00.27
Figure 5: Continual learning of Moving Fashion- MNIST and MNIST.
Table 2: N2 score on new data & its change A from that old data prior to continual learing.
sup. This carefully designed experiment provided clear evidence that CUDOS able to continually disentangle new semantic factors without entangling them with shared ones learned from the past. also showed that this cannot achieved by naively using generative replay to extend existing disentangling or prototype-basee VAE (e.g.. TC-VAE and VQ- VAE) into continual versions. More discussion of differences between static and continual disentanglement can be found in Appendix
Fig. 4 (a) shows that the SOM-mixture continually updated summary of the relation between past (blue box) and new data environments (black box) based on their active latent dimensions (b). T-SNE plots of the learned latent representations after continual learning as alternative way visualize
the relationships among data are provided in Appendix G
Moving MNIST & Fashion-MNIST: We then tested CUDOS on sequences of MaShi.n-MNISTT. MNIST, and moving versions of them similar to that presented in Achille et al. (2018). 5ig.5 presented results for one sequence: Moving Bushion-MNIS, As shown, CUDOS was able to disentangle new semantic factors (green boxes), while re-using those learned from the past (red boxes). For instance, the positional semantic factors learned from Moving FasHion-MNIST were reused while learning Moving MNIST.
+ Moving MNIST + Fashion-MNIST
CelebA: We further designed experiments for continually learning over split versions of CelebA data. Fig 6 provide examples of results obtained by two different splits: by age (top), and by bangs (bottom). Traversing results on selected dimensions showed that CUDOS was able to reuse latent dimensions for previous semantic factors while learning new ones related to the new attributes.
Benefits on downstream tasks: To evaluate the usefulness of the disentangled representations learnt by CUDOS, we focused on shared tasks related the continually- semantic factors shared among past and new data environments, including predicting the scale and orientation of 3DShapes. and predicting the X- and Y- positions of Moving-M. MNIST. For each task, identified the corresponding active latent dimensions after learning the first data environment, and continually trained linear regressors predict the ground-truth labels using these active latent dimensions: the
-NNIT.
8

---

Published as a conference paper at ICLR 2023
Young
Old
skin color
groder
face aging
Bald
Hairy
skin
hair amouunt
Figure 6: CelebA split by age (top row) and bangs (bottom row). The green boxes highlight the newly learned semantic factors, and the red boxes highlight the reused ones.
moving fashion
moving MNIST
fashion
active inactive
0.74
0.85
0.86
0.4
0.17
0.35
Method
MIG 1
MIG-sup
[[Zua factorS 4
reconstruction +
VAE+SOM+Col
0.128+00.026
0.125++0.029
0.969
0.011+0.001
VAE+SS+Lod
0.302=#0121
0,293++0.119
0.782++0.766
0.005:+0.03
VAE+SOM-SS+Call
0.217=0.072
0,240=+0.082
0.648+00266
0.024+0.004
SSE=SOM+SS+ Lnowz
0.216110.036
0.320110,086
0.072.10.050
0.0241100004
CUDOS above++Coon
0.242-10.047
0.326.10.033
0.024.10.029
0.021100003
Table 4: Top:
Continual digit classification accuracy (testing) based on active or inactive dimensions.
Bottom: Ablation study. SS: spike-and-slab density. Losal/ Lnew: constraints on replayed/new data.
intuition is that, if new semantic factors are entangled into these shared dimensions, the performance of the regressor will decrease during continual learning. For comparisons, /Achille al. (2018) and Ramapuram et al. (2020) were used as baselines and their results aggregated for 3Dshapes, and Ramapuram al. (020)) was used the baseline for MoVing-MNIST. Table 2 summarizes the li2 scores of each task obtained on new data along with their changes A from the R² scores obtained on old data prior continual learning. As shown, CUDOS achieved best final R² score for scale and orientation regression, along with minimum drop of performance over the course of continual learning. For Moving-MNIS, MNIST, CUDOS and baselines achieved similar results on X-Y position regression, with in general less significant performance drop in comparison to 3DShapes. We argue that this is because X-Y position regression on clean background is an easier task, than scale and orientation regression (top) shows digit-classen classifcassiiiaaiionn performance on Moving-MNN which suggested that active semantic factors for the digit data environments
more complex data environment like 3DShapes. Additionally, Table 4 using active and inactive dimensions, learned properly.
Ablation study: Table. 4 (bottom) presents a detailed ablation study on the contribution brought by the different ingredients within CUDOS. We did not include results from VAE+ Lotd because they represented by the work of Ramapuram et al. 2020) as reported in Table. D As shown, the sparsity introduced by spike-and-slab distribution plays a significant role in improving the disentanglement ability of CUDOS. While SOM alone does not appear to improve the general disentangling ability of the model, does seem to help disentangle new semantic factors from previously learned ones; more importantly, it is a necessary component for learning the relational structure of data to guide disentanglement, i.e., for enabling Luowx in Equation (IO). Indeed, the combined introduction of SOM and Lnewx brings significant improvement in the ability of CUDOS to disentangle new semantic factors from previously learned ones, as it is evident the improvement achieved in MIG-sup and I (Zpas factors ). More implementation details can be found in Appendix C
In this paper, we demonstrated that an overlooked key ingredient to continual unsuper- vised learning of representations is to exploit the relational structure of data based on their underlying active semantic factors. We presented CUDOS, a novel VAE with self-organizing spike-and-se mixtures, address this. limitation of the present form of CUDOS is that continual disentangling partly depends on the teacher-stulee stuuent strategy, leaving the snapshot timing a will investigate sequential Bayesian inference potential solution to this problem as future work.
Conclusion:
We

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by National Heart, Lung, and Blood Institute (NHLBI) grant R01HL145590, and National Institute of Nursing Research (NINR) grant ROINR018301.
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-domaii latent homologies. In Advances in Neural Information Processing Systems, pp. 9873-9883, 2018.
Pouya Bashivan, Martin Schrimpf, Robert Ajemian, Irina Rish, Matthew Riemer, and Yuhai Tu. Continual learning with self self-organizing maps. In Continual Learning Workshop, 32nd Conference on Neural Information Processing Systems, 2018.
Zalán Borsos, Mojmír Mutný, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 2020.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://siihhcccommmeemmmmm//lles 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in beta-vac. Advances in Neural Information Processing Systems, 2017.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In International Conference on Learning Representations, 2017.
Harrison Edwards and Amos Storkey. Towards a neural statistician. In International Conference on Learning Representations, 2017.
Florent Forest, Mustapha Lebbah, Hanene Azzag, and Jérôme Lacaille. Deep embedded som: Joint representation learning and self-organization. Reconstruction, 500:500, 2000.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- vae: Interpretable discrete representation learning on time series. In International Conference on Learning Representations, 2019.
Alexander Gepperth and Benedikt Pfülb. Gradient-based training of gaussian mixture models for high-dimensional streaming data. Neural Processing Letters, 53(6):4331 430044443184488, 2021.
Prashnna Gyawali, Zhiyuan Li, Cameron Knight, Sandesh Ghimire, B Milan Horacek, John Sapp. and Linwei Wang. Improving disentangled representation learning with the beta bernoulli process. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 1078-1083, IEEE, 2019.
Luke B Hewitt, Maxwell I Nye, Andreea Gane, Tommi Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. In The Conference on Uncertainty in Artificial Intelligence, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Daniella Horan, Eitan Richardson, and Yair Weiss. When unsupervised disentanglement possible? In Advances in Neural Information Processing Systems, volume 34, pp. 5150-5161, 2021.
10

---

Published as a conference paper at ICLR 2023
Cheng Hao Jin, Gouchol Pok, Yongmi Lee, Hyun- Woo Park, Kwang Deuk Kim, Unil Yun, and Keun Ho Ryu. A som clustering pattern sequence-basee based next symbol prediction method for day-ahead direct electricity load and price forecasting. Energy conversion and management, 90: 84-92, 2015.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, 2018
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464-1483 14800 1990.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentangled latent concepts from unlabeled observations. arXiv preprint 4cXiv:1711.00845 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11 96(11))2278-2324, 1998.
José Lezama. Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision. In International Conference Learning Representations, 2019.
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive
learning and disentanglement hierarchical representations. In International Conference on Learning Representations, 2020.
Nan Liu, Jinjun Wang, and Yihong Gong. Deep self-organizing map for visual classification. In 2015 international joint conference on neural networks (ICCNN, pp. 1-6. IEEE, 2015.
Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. In Advances in Neural Information Processing Systems, volume 34, pp. 10587- 10599, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research, 2020.
Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. In International Conference on Learning Representations, 2021.
Laura Manduchi, Matthias Hüser, Julia Vogt, Gunnar Rätsch, and Vincent Fortuin. Dpsom: Deep probabilistic clustering with self-organizing maps. arXiv preprint arXiv:1910.01590 2019.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. In International Conference Learning Representations, 2018.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
German Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Frontiers in Neurorobotics, 12:78, 2018.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neurocomputing, 404:381-400, 2020.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. Advances in Neural Information Processing Systems, pp. 7647-7657, 2019.
Travers Rhodes and Daniel Lee. Local disentanglement in variational auto-encoders using jacobian regularization. In Advances Neural Information Processing Systems, volume 34, pp. 22708 22719, 2021.
11

---

Published as a conference paper at ICLR 2023
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances Neural Information Processing Systems, pp. 2990-2999, 2017.
Michalis Titsias and Miguel Lazaro-Gredilla. Spike and slab variational inference for multi-task and multiple kernel learning. In Advances in Neural Information Processing Systems, 2011.
Francesco Tonolini, Bjørn Sand Jensen, and Roderick Murray-Smithh Variational sparse coding. In Uncertainty in Artificial Intelligence, pp. 690-700. PMLR, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1IXiv:1708.07720 2017.
Fei Ye and Adrian Bors. Learning latent representations across multiple data domains using lifelong vaegan. In European Conference Computer Vision, pp. 777-795. Springer, 2020.
Yin and NM Allinson. Bayesian self-organising map for gaussian mixtures. Proceedings- Vision, Image and Signal Processing, 148(4):233 240, 2001.
IEE
Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disen- tanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems, 2022.
12

---

Published as a conference paper at ICLR 2023
A OPTIMIZING v FROM ELBO
To make the partial derivative of xb clearer, we first rewrote the last two terms of ELBO in Eqn. as:
pu (
(xl()) DDiI (pro (10))) [p((w))
- pol(ll)) Pu(w)z)
ao (z) Po (z(w) dzdw + (xl / (o(wlz)oog p(w) dw
go (z/x)
Pv (wiz)
[f (0(2|x) / p dzddw
[ Polw)z)) /
g (s(x)x)logg (2) (z)x)
+ Bgo(alx)) / (u)l p(w) dw (w ())
I po (ull))log Po po(z(w)p((w) dw (aw Pe (w 2)
(xlx))
(xlx)
(logp) (x (logao (2/x)]
(12)
Therefore the optimized u can solved by:
argmax, :etimax,EBBO ELBO Bag
(13)
B MOVING MNIST AND MOVING FASHION-MNIS DATASET
Based the original MNIST and fashion-MNIST dataset, we created the moving version of them by: (1) Resize the original 28*28 images to 36*36 images, (2) Place the original 36*36 images image at the top-left of 64*64 black background. (2) Apply translation for both x and y axis with values 15,10,1,,
C EXPERIMENTS DETAIL
Hyper-parametee settings and implementation strategy: We set = 0.25, = 7, = 0.35, b 10 in all experiments. Snapshot of the model is updated every 7 1500 iteration steps. Regarding weights in Eqn TT, in our experiments, generally, the loss function becoming too large or small, and we found a rule-of-thumb weight value described above across different datasets. The most tricky part is the weighting for the constraints of old data and new data. We found a slightly higher weight on new data can achieve better continual disentanglement results. We reason that the constraint new factors more difficult compared with reconstructing old data, and therefore emphasizing more on that (higher weight) can be helpful. In general, we didn't 't find obvious differences within around 30% percent changes of each weight. Putting them too high (larger than the ELBO term) will affect the original continual learning of new data. Metries in Table. are calculated in setting where the boundary of data environments is known.
are trying to avoid certain parts of
Network-architecture Table. Table.5]shoov. shows the network architecture for all our experiments, where BN stands for batch normalization, Conv stands for convolution layer, T-Conv stands for transposed convolution layer, FC stands for fully-connected layer. ELU stands for Exponential linear unit activation.
Code is available atttppr//gthub..ccw///yyy//////.
D TRAVERSING RESULTS ON SPLIT-3DSHADE
Here we presented additional traversing results for baseline methods, along with their
I (Zgpst; factors value.
13

---

Published as a conference paper at ICLR 2023
0
0
2
2
3
5
5
7
8
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
Figure 7: Burgess et al. 2017) I (Zzuss factorsnew ): 1.185
6
6
10
10
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
Figure 8: Achille et al. (2018),
(Zgustt factors 0.602.
14

---

Published as a conference paper at ICLR 2023
3
9
10
10
11
11
12
12
13
13
14
14
15
15
16
16
17
17
18
18
19
19
Figure 9;
Ramapuram et al.
2020
I (Zpast) factors ): 0.877
0
1
2
3
5
6
8
9
10
10
1
11
12
12
13
13
14
14
15
15
16
17
18
19
19
Figure 10: Continual VQ-VAE, / (Zipasti factorsnew): 1.402
15

---

Published as a conference paper at ICLR 2023
Encoder Layer
Encoder parameters
Decoder Layer
Decoder parameters
Conv
32,4,4, strides=2, BN, ELU 64,4,4, strides=2. BN ELU
FC1
1024, BN, ELU 4944256, BN, ELU
Conv2
FC2
Conv3
128,4,4, strides=2, BN, ELU
T-Conv
128,4,4, strides=2, BN ELU
Conv4
256,4,4, strides=2, BN, ELU
T-Conv2
64,4,4, strides=2, BN, ELU
FC1
1024, BN, ELU
T-Conv3
32,4,4, strides=2, BN, ELU
FC2
1024, BN, ELU
T-Conv4
image channels,4,4, strides=2
FC3
latent dims*2
Table 5: Network architecture
8
9
8
0
6
6
9
Figure 11: Prototypes learnt for continual learning on Foinig-FanshhionMMNISTT to Moving-MNIST MNIST.
E SOM PROTOTYPES
Additional SOM prototypes learnt for Moving-MNIST are shown in Fig In the SOM, the presented model was able remember and accumulate old representations, e.g., fashion digits, while learning new representations, e.g., number digits, Additionally, the shared representations, the X- Y position, were changing smoothly among prototypes.
16

---

Published as a conference paper at ICLR 2023
750 iters
1125 iters
iters
1875 Iters
2250 iters
2625
3000 iters
Figure 12: Mapping density SOM of the first data environment of split-3DShapes during training.
375 iters
750 iters
1325 Ieer'
1500 Iters
Berr
2250
2625
3000
Figure 13: Mapping density on SOM of the second data environment of split-3DShapes during training. Dots without black boundaries replay data.
F DATA ENVIRONMENTS MAPPING ON SOM DURING TRAINING
In Fig 12 and Fig 13
we presented additional data environments' mapping density SOM during
continual training.
G T-SNE FOR 3DSHAPES FACTORS
Here we presented additional T-SNE plots for the learned latent representations of our model (colored by the generative factors of 3DShape) after continual learning of split-3DShape shown in Fig 3 As shown in Fig 14, some factors such as floor color (10 classes) and shape (4 classes) formed good clusters while some more difficult factors such as orientation (15 classes) and scale classes) formed fewer discriminative clusters.
H TRAVERSING RESULTS ON MOVING-MNIST
Here presented additional traversing results for baseline methods Moving-MNIST.
17

---

Published as a conference paper at ICLR 2023
floor color
object
orammationn
Figure 14: T-SNE plots for 3DShapes factors.
0
0
f
2
f
3
f
3
4
f
f
f
5
8
6
7
1
1
T
T
7
9
8
f
f
8
5
6
0
9
9
4
4
4
4
0
0
Figure 15:
Burgess al. 2017
18

---

Published as a conference paper at ICLR 2023
0
0
D
0
1
0
0
0
0
0
0
2
2
2
2
0
3
7
3
2
3
0
4
1
4
5
5
6
T
6
3
0
7
T
7
9
0
0
0
5
8
8
1
1
1
8
3
0
0
0
9
9
4
0
0
0
8
Figure 16: Achille et al. (2018).
0
0
0
D
1110
1
1
0
1
2
2
2
3
544
3
3
4
5
5
6
6
m
3
0
7
7
0
0
11
8
8
0
0
0
0
9
9
0
0
O
Figure 17:
Ramapuram al. (202)).
19

---

Published as a conference paper at ICLR 2023
0
0
1
2
3
3
4
5
5
8
1
6
6
1
8
8
1
7
7
1
8
8
i
1
1
9
Figure 18: Continual VQ-VAE.
I ADDITIONAL EXPERIMENTS OF
Here we first presented additional experiments of split-3DShapes with two splitting versions that each has three data environments. The first version, as shown in [9 was starting with no floor and wall color variations and then continually added them. The second version, as shown in Fig 20 was starting with no scale and wall color variations and then continually added them. For most latent dimensions during continual learning, CUDOS was able to reuse those corresponding to previously learned semantic factors and disentangle new ones into previously inactive dimensions.
Next, we presented reversed sequence of split-3DShape, where the first data environment includes all semantic factors and the second one includes a subset that doesn't have the floor color variations (only red remained). As shown in fig 21 our model was able reuse all previous semantic factors without any expansion of latent space. Furthermore, our model was able to remember how to generate the variations of the missing floor color in the second data environment after continual training.
J MORE DISCUSSION OF DIFFERENCES BETWEEN STATIC AND CONTINUAL DISENTANGLEMENT
Continually disentangling sequentially-arien arriived semantic factors is fundamentally different from disentangling static dataset where the model all semantic factors at once. When semantic factors are presented sequentially, the model needs to make sure that it 1) does not forget previously learned factors, 2) is able re-use latent dimensions corresponding to shared factors, and 3) prevents new ones to be entangled into the shared ones. None of these challenges are addressed by state-of- the-art (SOTA) disentangling VAEs such as FactorVAE (Kim & Mnih 2018 and TC- VAE (Chen) et al. 2018). The continual setting sees significant challenges such as forgetting (including forgetting learned semantic factors) and, as in other continual learning problems, cannot be expected to see similar disentanglement performance to those reported on static settings. That's 's why the non-continual baselines in Table [ including naive VAE, naive TC-VAE, and Burgess et al. (2017). all showed weaker performance than what would have been expected in static learning setting. Furthermore, because these models are not designed for continual learning, they will also face challenges such as catastrophic forgetting. We can take closer look at this by taking naive TC- VAE versus continual TC- VCE (with generative replay mechanism) as an example. As shown in Table ! naive TC- VAE achieved overall better MIG and MIG-sup scores because they forget about previous data and
20

---

Published as a conference paper at ICLR 2023
No variation of
Continual learning added semantic factor of floor color
Continual learning added semantic factor wall color
Figure 19: Continually learning split version of 3DShapes where the variations of floor and wall colors were absent initially and appeared later. Each row images is traversing each latent dimension after training data environments titled above. Black bounding boxes annotate where the semantic factors are learnt.
No variation of scale and wall color
Continual learning of added semantic factor of scale
Continual learning of added semantic factor of wall color
Figure 20: Continually learning split version of 3DShapes where the variations of scale and wall colors were absent initially and appeared later. Fach row of images is traversing each latent dimension after training on data environments titled above. Black bounding boxes annotate where the new semantic factors are learnt.
21

---

Published as a conference paper at ICLR 2023
Continual learning of removed semantic factor of floor color
semantic factors
Figure 21: Continually learning a reversed sequence of -Dhhae where all semantic factors presented in the beginning but later the variation of floor color removed (only red remained). Each row of images is traversing each latent dimension after training on data environments titled above.
focus on disentangling the new data (in split 3DShapes, the two sequentially-preseee datasets share many latent factors but one). Therefore, forgetting previous semantic factors did not create large performance drop in MIG and MIG-sup scores. This catastrophic forgetting however is reflected in the reconstruction loss, which increased from 2365 to 3207 (around 35% change) on the previous data. In addition, the / (Zipast factorsnew) metric further shows that naive TC-VAE is not able separate new semantic factors from previously-Iearmn latent dimensions, rather achieved relatively high disentangling by simply forgetting previous factors and learning the new data itself. This is similar to naive VAE's performance.
By extending TC-VAE to continual learning setting with generative replay, the continual TC-VAE was able to remember how to reconstruct the previous data, where the reconstruction loss only increased from 2376 2395 (0.7% change). However, its disentanglement performance including MIG and MIG-sup dropped. This demonstrates the aforementioned challenges of disentangling sequentially-arde
semantic factors (while remembering the previous factors the same time) that are fundamentally different from disentangling all seen factors at once. It also shows that such challenges cannot be addressed by either naive SOTA disentangling VAEs, or simply extending these SOTA disentangling VAEs into a continual setting (via standard techniques such as generative replay). We believe these provide further evidence for the contribution of the presented work.
22


Reviews:
[{"review_id": "tQ9FiGjgNfb", "reviewer": "Reviewer_ZXCc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author shows that an overlooked key ingredient to continual unsupervised learning of representations is to exploit the relational structure of data based on their underlying active semantic factors. This paper proposes a novel VAE with self-organizing spike-and-slab mixtures called CUDOS. ", "summary_of_the_review": "This paper presents CUDOS, a novel VAE with self-organizing spike-and-slab mixtures. ", "strength_and_weaknesses": "Strength:\n\n- The CUDOS can treat continually-arrived data independently, without knowing how they are related based on the underlying semantic factors. \n\n- This paper has conducted a large number of experiments to testify to the disentangling ability of CUDOS.\n\n- This work mitigates the catastrophic forgetting from a novel perspective of generative replays from SOM-mixture. \n\n\nWeakness:\n\n- More visualization like TSNE can further clarify the disentangling ability of CUDOS.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "el8f7jLSvWt", "reviewer": "Reviewer_ARpk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to perform unsupervised disentanglement with continual representation learning methods. Specifically, they designed a variational autoencoder (VAE) that exploits the relationships between datapoints based on their latent factors. The model learns the latent representations of data by applying a self-organizing mixture of spike-and-slab priors (which is topologically informed) to the VAE. The proposed method is validated on many disentanglement datasets (3DShapes, variants of MNIST, and celebA), where the proposed method showed superior disentanglement performance when compared to other continual learning algorithms.\n", "summary_of_the_review": "I suggest rejection for this paper. Although there are many interesting and innovative components of the proposed approach, the evaluation of the proposed method isn’t convincing as many benchmark models that should be considered are not considered. The introduction (motivation) and the related work section could also use major editions to improve clarity. I am willing to increase the score for this paper if sufficient revision is made accordingly to address my major concerns.", "strength_and_weaknesses": "Strength\n- It is interesting to utilize sample-2-sample relationship for disentanglement, and it is interesting to see how it can be applied in the continual learning regime, where semantic factors can possibly be expanded. \n- The paper presented solid mathematical/theoretical contributions by formulating the variational inference process with Bayesian-SOM and spike-and-slab distribution. The presented method seems to have several unique advantages because of the selected priors.\n\nWeaknesses\n\n1. Major concern: experiments and methods for benchmarking aren't sufficient for demonstrating the strength of the method.\n- The proposed method seems to have relatively weak disentanglement performance when compared with previously existing methods. Specifically, [1] reported several models’ MIG scores on 3Dshapes (the non-splitting version), where FactorVAE has an average of 0.3, AnnealedVAE has an average of 0.4. They also reported relatively high model variance, where a naive VAE can achieve a MIG score of as high as 0.8. It is understandable that the paper reported relatively low variance as each model is run 5 times and there might exist different settings for randomness. However, it is not convincing that the model actually disentangles well, as it does not outperform naive VAE by a large margin, and some other VAE methods that are designed for disentanglement study can easily outperform the proposed method.\n- The benchmarking models are selected with certain bias. Only model2 (beta-VAE by Burgess) and model4 (life-long disentanglement by Achille) are specifically designed for disentanglement (correct me if I am wrong). All other methods are continual learning methods. It is understandable to include many continual learning methods, as the proposed method is also based on continual learning techniques. However, as the method is motivated to address the disentanglement problem, it should consider more methods (e.g. VAEs), and especially state-of-the-art methods, that are specifically designed for disentanglement (e.g. FactorVAE, TC-VAE, DIP-VAE) to have a fair comparison.\n- Although the evaluation of disentanglement has been a long-standing problem [1, 2], using the mutual information I(z_{past}; factors_{new}) seems to be a particularly biased one as the proposed method is optimized on similar components.\n\n2. Related work related to disentanglement learning is lacking\n- There are many different variants of VAE that are designed for disentanglement that are not considered and mentioned. The original beta-VAE paper [3] isn’t cited. [4-6] are VAEs that are mentioned above. [7-9] are more recent unsupervised methods that consider sparse coding or separatable latent factors.\n- When doing unsupervised disentanglement learning, it is also recommended to clearly demonstrate the assumptions [1].\n\n3. Motivation isn’t strong enough\n- It isn’t totally convincing to me why one would like to perform disentanglement under the continual learning setting. I think it would be beneficial if the authors could stress less about how it is the first work that evaluates the disentanglement scores for continually learned latents, but stress why it would be important instead.\n\n4. Minor comments:\n- The details of the Split-3DShapes dataset aren’t clear. How many data points are within each subset? Why it is separated in this particular way?\n- How are the hyper-parameters selected? I did not find any details about the model selection procedure. What are the training/validation/testing sets? (also for the benchmark models)\n- I am not sure if the important details for reproducing the models are included. What is the model learning rate?\n- Does the network architecture apply to all benchmark models?\n\n\n[1] Locatello, Francesco, et al. \"Challenging common assumptions in the unsupervised learning of disentangled representations.\" international conference on machine learning. PMLR, 2019.\n\n[2] Mathieu, Emile, et al. \"Disentangling disentanglement in variational autoencoders.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Higgins, Irina, et al. \"beta-vae: Learning basic visual concepts with a constrained variational framework.\" (2016).\n\n[4] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International Conference on Machine Learning. PMLR, 2018.\n\n[5] Chen, Ricky TQ, et al. \"Isolating sources of disentanglement in variational autoencoders.\" Advances in neural information processing systems 31 (2018).\n\n[6] Kumar, Abhishek, Prasanna Sattigeri, and Avinash Balakrishnan. \"Variational inference of disentangled latent concepts from unlabeled observations.\" arXiv preprint arXiv:1711.00848 (2017).\n\n[7] Rhodes, Travers, and Daniel Lee. \"Local Disentanglement in Variational Auto-Encoders Using Jacobian $ L_1 $ Regularization.\" Advances in Neural Information Processing Systems 34 (2021): 22708-22719.\n\n[8] Liu, Ran, et al. \"Drop, swap, and generate: A self-supervised approach for generating neural activity.\" Advances in Neural Information Processing Systems 34 (2021): 10587-10599.\n\n[9] ​​Horan, Daniella, Eitan Richardson, and Yair Weiss. \"When Is Unsupervised Disentanglement Possible?.\" Advances in Neural Information Processing Systems 34 (2021): 5150-5161.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NRHULgPGhXu", "reviewer": "Reviewer_z1ce", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to achieve continual, disentangled, and unsupervised representation learning where the representations can be reused and expanded upon newly arriving data. For this, the authors propose to model the relational structure of continuously arriving data based on their \"active semantic factors\", which allows for the discovery of new semantic factors while recycling (the old) shared ones. The two pillars of the methodology are (i) self-organizing maps (SOMs) to maintain a relational data structure and (ii) sparse spike variables to model the underlying active semantic factors in each mixture component. The method is shown to achieve outstanding continual disentanglement of latent semantic factors.", "summary_of_the_review": "The paper presents a novel, well-motivated method that tackles an important and under-explored aspect of continual learning. While the qualitative evaluation could be more thorough, the submission in the current form would already be a valid contribution to the continual learning and representation learning community. I recommend acceptance.", "strength_and_weaknesses": "### Strengths\n- It is a thoughtful and careful approach to designing the model architecture. Authors start with a well-defined problem, identify a problem in existing literature (the independence assumption) and propose a creative solution. The use of specific components is well-motivated, and the intuitions are verified through ablation studies. The loss formulation is principled and well-argued.\n- The paper is very well-written. The sentences are crystal clear and the ideas are communicated in a great level of detail. The transitions are also very smooth. I really enjoyed reading the work! \n\n### Weaknesses\n- The final loss term (11) is rather complicated, involving many moving parts. Although each term is individually motivated, ablation studies displaying their impact are needed.\n- Qualitative evaluation can be improved. The authors provide quantitative evidence that their method performs continual disentanglement, but it would be informative to dig deeper into the results shown in Figure 3. In particular, introducing novel factors (not limited to floor colour) in sequence would shed more light on the learning dynamics and make a more convincing argument that the model discovers, reuses, and expands latent semantic factors. Another extension (arguably out of the scope of the submission) would be to use the model as a generative replay module in existing continual learning methods and check if the disentangled representation provides benefits for downstream tasks like classification or semantic segmentation.\n\n\n### Questions\n- I don't immediately see why sparse coding enables the discovery of \"active semantic factors\" in an auto-encoder. In particular, why are these \"semantic factors\" instead of, e.g., \"active features\"? This could be elaborated more.\n- I'm not sure how exactly $\\alpha$'s are inferred for streaming data. Precisely, why do we maintain a set? Here, does each \"semantic factor\" correspond to one latent dimension? How does this connect to Figure 1?\n- Why do we need to address the fact that \"generative replay lacks mechanisms to constrain newly arrived data\"?\n- Does the method maintains a fixed number of SOM snapshots? How are they updated as learning proceeds?\n- If eq.8 is optimized perfectly, then would we still need to optimize eq.9? It seems to me that both objectives serve the same purpose? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hu0xThKUm2", "reviewer": "Reviewer_NqT4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for disentangling semantic factors (a.k.a. generative factors) in a continual setting where the observed data from different environments come in a stream. This paper studies continual disentanglement in unsupervised learning, an under-explored area. This paper proposes principled solutions to reusing, expanding, and continually disentangling latent dimensions for each data environment coming in a continual fashion. The overall method is called the Continual Unsupervised Disentangling of self-Organizing representations (CUDOS). CUDOS learns the relationships among active latent dimensions for each data environment and reuses appropriate latent dimensions while training on new data environments. Latent space is modeled as a mixture of self-organizing maps (SOM) where each mixture component is modeled as a spike-and-slab distribution to encourage sparse representation in latent space. This paper proposes separate objective functions for each desideratum of continual disentanglement. Empirical studies show that CUDOS performs better than existing methods on various datasets w.r.t. multiple metrics.", "summary_of_the_review": "This paper addresses an important and under-explored idea of unsupervised continual disentanglement of the data coming from different data environments. As discussed in the weaknesses, it is unclear what real-world advantage CUDOS brings by expanding latent dimensions (which is a major contribution of CUDOS) for each data environment. I feel experimental results are sufficient to verify the theoretical claims of the paper. Considering the novel contributions, I provide my rating as ‘marginally above the acceptance threshold’.", "strength_and_weaknesses": "Strengths:\n------------\n+ **Motivation:** Each of the issues related to continual disentanglement: catastrophic forgetting, expanding, and reusing latent dimensions are dealt with separately using three objective functions that are added to the traditional ELBO objective in a Variational Auto Encoder. Such isolation gives more control over the latent space to achieve better disentanglement.\n+ **Methodology:** When training on new data environment, CUDOS expands the latent dimension by using new semantic factors that are not entangled with existing active semantic factors (Equation 7). Such expansion is a good addition to the CUDOS methodology to improve the disentanglement score in continual learning.\n+ **Experiments:** Experimental results are promising in confirming to the theoretical claims of the paper.\n\nWeaknesses:\n----------------\n- **Readability:** The paper is challenging to understand in some places. Especially, Sub-Section 3.2 could be more detailed. For example, does the line ‘Generative replay lacks mechanisms to constrain newly arrived data.’ mean CUDOS wants to generate a replay buffer using old semantic factors? Also, some preliminaries on existing work, such as how SOMs are used in VAEs, would make the paper more self-contained. \n- **Methodology:** Why is it essential to expand latent dimensions while training in new environments? Why cannot existing/already active semantic factors capture new variations in the data? For example, in Figure 3.b and page 7 of the results section, while I agree that CUDOS uses previous inactive dimensions for learning new semantic factors, naive VAE also generate variations w.r.t. one semantic factor while keeping other semantic factors fixed. That is, naive VAE and CUDOS both produce the desired behavior of disentangling semantic factors. What advantage does CUDOS bring by using previously inactive dimensions?\n- **Methodology:** Are semantic factors continually expanded when training in a new data environment? What if active semantic factors of an environment describe a new data environment? Is it guaranteed that CUDOS will not learn/expand the existing semantic factors? How does it affect the training procedure? If it is assumed that a new data environment leads to expanding latent space, the authors should clarify it in the paper and its effect on the training procedure.\n- **Experiments:** In the experiments, the reconstruction loss of CUDOS is not the best compared to baselines. Since good generation is an essential aspect of any generative model, this could be a negative aspect of CUDOS. Is this because of multiple objective functions in Equation 11? What is the effect of weighting hyperparameters of Equation 11 on the final output? \n- A typo: in the results of Moving MNIST & Fashion-MNIST, it should be Fig. 5 instead of Fig. 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%