Disentangling Action Sequences: Discovering Correlated Samples
Authors: Jiantao Wu
Chunxiuzi Liu
Lin Wang
Venue:  (2021)
Keywords: disentanglement
representation learning
unsupervised
inductive bias

Abstract: Disentanglement is a highly desirable property of representation due to its similarity with human’s understanding and reasoning. This improves interpretability, enables the performance of down-stream tasks, and enables controllable generative models.However, this domain is challenged by the abstract notion and incomplete theories to support unsupervised disentanglement learning. We demonstrate the data itself, such as the orientation of images, plays a crucial role in disentanglement and instead of the factors, and the disentangled representations align the latent variables with the action sequences. We further introduce the concept of disentangling action sequences which facilitates the description of the behaviours of the existing disentangling approaches. An analogy for this process is to discover the commonality between the things and categorizing them. 

Furthermore, we analyze the inductive biases on the data and find that the latent information thresholds are correlated with the significance of the actions. For the supervised and unsupervised settings, we respectively introduce two methods to measure the thresholds. We further propose a novel framework, fractional variational autoencoder (FVAE), to disentangle the action sequences with different significance step-by-step. Experimental results on dSprites and 3D Chairs show that FVAE improves the stability of disentanglement.

Under review as conference paper at ICLR 2021
DISENTANGLING ACTION SEQUENCES: FINDING COR- RELATED IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Disentanglement is highly desirable property of representation due its similarity with human's 's understanding and reasoning. This improves interpretability, enables the performance of down-stream tasks, and enables controllable generative models. However, this domain is challenged by the abstract notion and incomplete theories to support unsupervised disentanglement learning. We demonstrate the data itself, such the orientation of images, plays crucial role in disentanglement instead of the ground-truth factors, and the disentangled representations align the latent variables with the action sequences. We further introduce the concept of disentangling action sequences which facilitates the description of the behaviours of the existing disentangling approaches. An analogy for this process is to discover the commonality between the things and categorizing them.
Furthermore, we analyze the inductive biases on the data and find that the latent information thresholds are correlated with the significance of the actions. For the supervised and unsupervised settings, we respectively introduce two methods to measure the thresholds. We further propose a novel framework, fractional variational autoencoder (FVAE), disentangle the action sequences with different significance step-by-stey Experimental results on dSprites and 3D Chairs show that FVAE improves the stability of disentanglement.
INTRODUCTION
The basis of artificial intelligence is understand and reason about the world based limited set of observations. Unsupervised disentanglement learning is highly desirable due to its similarity with the way we human think. For instance, we can infer the movement of running ball based on single glance. This is because the human brain is capable of disentangling the positions from set of images. has been suggested that disentangled representation is helpful for a large variety of downstream tasks (Schölkopf et al. 2012 Peters et al. 2017). According to Kim & M (2018), disentangled representation promotes interpretable semantic information. That brings substantial advancement, including but not limited to reducing the performance gap between humans and AI approaches (Lake et al. 2017 Higgins et al. 2018). Other instances of disentangled representation
include semantic image understanding and generation (Lample al.) [017 Zhu al. 2018 E gammal et al. 2017). zero-shot learning (Zhu et al. (019)). and reinforcement learning (Higgins et al. 2017b). Despite the advantageous of the disentangling representation approaches, there still two issues to be addressed including the abstract notion and the weak explanations.
Notion The conception of disentangling factors of variation first proposed in 2013. It claimed in Bengio et al. (2113) that for observations the considered factors should be explanatory and independent of each other. The explanatory factors are however hard to formalize and measure. An alternative way is to disentangle the ground-truth factors (Ridgeway/ 2016; Do & Tran 2020). However, if we consider the uniqueness of the ground-truth factors, a question which arises here is how discover it from multiple equivalent representations? As proverb "one cannot make bricks without straw". Locatello et al. (2019) prove the impossibility of disentangling factors without the help of inductive biases in the unsupervised setting.
Explanation There are mainly two types of explanations for unsupervised disentanglement: information bottleneck, and independence assumption. The ground-truth factors affect the data

---

Under review as conference paper at ICLR 2021
independently, therefore, the disentangled representations must follow the same structure. The approaches. holding the independence assumption, encourage independence between the latent variables (Schmidhuber 1992; Chen et al.) 2018 Kim & Mnih 2018 Kumar et al. 2018 Lopez et al., 2018). However, the real real-world problems problees have no strict constraint on the independence assumption, and the factors may be correlative. The other explanation incorporates information theory into disentanglement. Burgess et al.) Higgins et al., Insu Jeon et al., |axx et al. jsuggest that a limit on the capacity of the latent information channel promotes disentanglement by enforcing the model acquire the most significant latent representation. They further hypothesize that the information bottleneck enforces the model find the significant improvement.
In this paper, we first demonstrate that instead of the ground-trum truth factors the disentangling approaches learn actions of translating based on the orientation of the images. We then propose the concept of disentangling actions which discover the commonalities between the images and categorizes them into sequences. We treat disentangling action sequences as a necessary step toward disentangling factors, which can capture the internal relationships between the data, and make it possible to analyze the inductive biases from the data perspective. Furthermore, the results on a toy example show that the significance of actions is positively correlated with the threshold of latent information. Then, promote that conclusion to complex problems. Our contributions are summarized in the following:
We show that the significance of action related to the capacity of learned latent information, resulting in the different thresholds of factors.
We propose a novel framework, fractional variational autoencoder (FVAE) to extracts explanatory action sequences step-by-step, and each step, learns specific actions by blocking others' information.
We organize the rest of this paper as follows. Sec.2 describes the development of unsupervised disentanglement learning and the proposed methods based on VAEs. In Sec.3, through example, show that the disentangled representations are relative to the data itself and further introduce a novel concept, i.e., disentangling action sequences. Then, we investigate the inductive biases on the data and find that the significant action has a high threshold of latent information. In 4,c.4, we propose a step-by- step disentangling framework, namely fractional VAE (FVAF), to disentangle action sequences. For the labelled and unlabelled tasks, we respectively introduce two methods to measure their thresholds. We then evaluate FVAE on a labelled dataset I'SSrites,MMattiieee (3D Chairs, Chairs,AAubry et al. (2014). Finally, we conclude the paper and discuss the future work in Sec.5
al. (2017) and an unlabelled dataset
2 UNSUPERVISED
LEANTLEANN LEARNING
We first introduce the abstract concepts and the basic definitions, followed by the explanations based on information theory and other related works. This article focuses on the explanation of information theory and the proposed models based on VABBs.
2.1 THE CONCEPT
Disentanglement learning is fascinating and challenging because of its intrinsic similarity human intelligence. As depicted in the seminal paper Bengio et al., humans can understand and reason from a complex observation the explanatory factors. common modeling assumption of disentanglement learning is that the observed data is generated by set of ground-truth factors. Usually, the data has a high number of dimensions; hence is hard to understand, whereas the factors have a low number of dimensions, thus simpler and easier to be understood. The task of disentanglement learning is to uncover the ground -oound-truth factors. Such factors are invisible to the training process in an unsupervised setting. The invisibility of factors makes it hard to define and measure disentanglement (Do & Tran 2020).
Furthermore, it is shown in Locatello et al. (0019) that it is impossible to unsupervised disentangle the underlying factors for the arbitrary generative models without inductive biases. In particular, they suggest that the inductive biases on the models and the data should be exploited. However, they do not provide formal definition of the inductive bias and such definition is still unavailable.
2

---

Under review as conference paper at ICLR 2021
2.2
INFORMATION BOTTLENECK
Most of the dominant disentangling approaches are the variants of variational autoencoder (VAE). The variational autoencoder (VAE) is a popular generative model, assuming that the latent variables obey specific prior (normal distribution in practice). The key idea of VAE is maximizing the likelihood objective by the following approximation:
[(0,0; E,,, (x/x) (oopoo/ - (alllDkkl(gg((())))) (90(2/=)llp()))
(1)
which known as the evidence lower bound (ELBO); where the conditional probability ((x))))((z(aa)
are parameterized with deep neural networks.
Higgins et al. find that the KL term of VAEs encourages disentanglement and introduce a hyperparame- ter in front of the KL term. They propose the 3-VAE maximizing the following expression:
((0,0;0,2) (x))) [logpa [z - (go (2<{
(2)
B controls the pressure for the posterior Q0(z(z) to match the factorized unit Gaussian prior p(z). Higher values of 3 lead to lower implicit capacity of the latent information and ambiguous reconstruc- tions. Burgess et al. propose the Annealed- VAE that progressively increases the information capacity of the latent code while training:
L( C(0, (2
[lggoo<]]=)] (oggoo
- (2o(<)x) (p())) -c
where is large enough constant to constrain the latent information, is value gradually increased from zero to a large number to produce high reconstruction quality. As the total information bottleneck gradually increasing, they hypothesize that the model will allocate the capacity of the most improvement of the reconstruction log- ligliikl to the encoding axes of the corresponding factor. However, they did not exploit why each factor makes different contributions the reconstruction log- ligeeik.o
2.3 OTHER RELATED WORK
The other dominant direction initiates from the prior of factors. They assume that the ground-truth factors are independent of each other, and series of methods enforce the latent variables have the same structure as the factors. Factor VAE (Kim & Mnih 2018) applies a discriminator to approximately calcu- late the total correlation (TC, Watanabe (1960); ); 3 3-TCVAE (Chen et al. (018)) promotes the TC penelty by decomposing the KL term; DIP-VAE (Kumar et al. 2018) identifies the covariance matrix of g(2).
3 DISENTANGLING ACTION SEQUENCES
For machines and humans, disentangling the underlying factors is challenging task. For instance, there are more than 1, 1,400 breeds of dogs in the world, and seems impossible for an ordinary person to distin- guish all of them just by looking their pictures. The challenge disentangling the underlying factors is mainly due to the complexity of establishing relationships without supervision, where the correspond- ing models should contain some level of prior knowledge inductive biases. However, is possible to determine the differences without having extensive knowledge. For example, one may mistakenly identify the breed of dog by looking at a picture but is almost impossible to misrecognize a dog as cat. Therefore, in practice, discovering the differences similarities is often much easier task than that of uncovering the underlying factors, and it does not also need much prior knowledge. One may conclude that discovering the commonalities between things is an important step toward disentanglement.
3.1
ACTION SEQUENCHS
Action The continuous set of images over certain direction.
The observed data consists of a set of actions such as rotation, translation, and scaling. Such actions meaningful and underlying. Therefore, the goal of disentanglement learning is to separate the underlying actions of observation. Usually, the continuous actions are infeasible for the machine learning system, and we have convert them into some discrete and consistent sequences.
3

---

Under review conference paper at ICLR 2021
Dataset
Projection
Dataset
Projection
2
0
-2
-2
-2.5
0.0
2.5
(a) A1
(b) A2
Dataset
Projection
Dataset
Projection
19:00
-2-
-4-
(c) A3
(d) A4
Figure 1: Examples of the influences of the data on the learned representations. (a) Translation perpendicular the coordinate axis. (b) Translation along the polar axis. (c) Translation of rotated images along the coordinate axis. (d) The learned presentation of three shapes on dSprites. Each line denotes the projections of the rotation angles into the latent variables w.r.t. three shapes.
Given dataset S, we assume it consists of parameterized action sequences, and each action sequence a subset of the dataset. We can model a parameterized action sequence as:
S' (mjj)))) Tmm ({t)),
where St E denotes action sequence, denotes the type of actions, denotes parameter vector, denotes a transformation, denotes the step of the action. The procedure is similar to the latent traversal Higgins al. (2)77). Differently, action sequences describe both the real observed data and the reconstructed data: For the real observed data, m is the ground-truth factor vector, and T is the generating function; For the reconstructed data, m is the latent variable vector, and T is a neural network. For clarity, we denote generating actions as sequences of images controlled by the ground-truth factors and action sequences as the approximation of these actions generated by neural network.
However, the neural networks are flexible enough to approximate any actions, and it's tricky to infer and learn the generating actions. The marvelous thing about disentanglement learning that the models seem to learn these actions unsupervisedly. A popular view of disentanglement believes the ground-truth factors should be separated into independent latent variables Locatello et al. (2019): Chen et al. (2088). That means the learned action sequences have to match the generating actions precisely. However, (Rolinck et al., 2019) shows that the VAEs conduct a PCA-like behavior, and the models prefer to extract the principal component. Hence, the models may select some action sequences having more considerable variations and learn the significant action sequences rather than the generating actions. Using VAEs, minimizing the objective increases the overlap between the posterior distributions across the dataset Burgess et al. al. (2018), and it then leads to the learned action sequences showing some internal relationships. Although (Locatello al. 2019) suggest that the inductive biases on the models and the data should be exploited. the current researches focus on the role of the models disentanglement(Burrrg Burgess al. (2018): Chen et al. (2018); Rolinek Rolinek al. (2019).
3.2 INDUCTIVE BIAS ON THE DATA
We believe the data itself containing the vital information for disentanglement. In other words, keeping the applied model the same, the learned representations change corresponding to some modifications on the data. Therefore, there exist some data clues inside the data guiding the models disentangle. We create toy dataset family- each dataset contains 40x40 images which are generated from an original image of an 1 1xx5 rectangle by translating 64x64 canvas. Each image on the dataset has unique
4

---

Under review as conference paper at ICLR 2021
Ordered Unardered
S=21, z2,
23}
Binding
Passible Sequences
Figure 2: An example of action sequences. There are three images with differing sizes on the dataset (left box), and the AE learns representation with one dimension of latent space (the double-arrow lines). So the total number of possible sequences is six. Two of them are meaningful, and the others are somewhat random.
label (position of X. position of Y) to describe the ground-truth factors. In this dataset family, there are two variables: the orientation the rectangle and way to determine the two factors. There are infinite solutions to determine these two factors; the polar coordinate system and the Cartesian coordinate system are the most common solution. We then create a baseline dataset, A1, with a horizontal rectanag rectangle in the Cartesian coordinate system and obtain its variants. A2 differs in the positions determined by the polar coordinate system, and A3 differs in the orientation (45 degrees) of the images. A4 uses the results of dSprites, and we only show the rotation parts w.r.t. three shapes. For the experiment settings in this section, we choose the well-examined baseline model, 3- VAE (3 = 50), and the other settings such as the backbone network, learning rate, and optimizer refers the settings in Locatello et (2019).
As it is shown in Fig. 1, we visualize the learned representations in the latent space. One can see that Al and A3 have the same generating actions but different in the orientation of the rectangles, and the difference of the rectangles causes the rotation of the learned representations; the model fails to learn the generating actions, however, the learned action sequences are explanatory and similar to Al's (see in A.44. We argue that current approaches don't 't guarantee to separate the ground-truth factors into isolating dimension (the same conclusion as Locatello et al. (2019)) because A2 and A3 fail to learn the generating action sequences. However, the data guarantee the learned action sequences (Al, A3). One can see that the invariant A A2, and A3 is that they have learned two actions moving along the direction of the long side of the rectangle and the orthogonality direction (see in Fig. 10).
3.3 SIGNIFICANCE OF ACTION SEQUENCES
It is suggested in Burgess et al. (2018) that the underlying components have different contributions to the objective function. (Rolinek et al. 2019) addressed that the VAEs conduct PCA-like behaviors. However, they didn't 't exploit the inductive biases on the data explicitly. Therefore, we use entropy to measure the information among this action:
H()) - IEE"" p( p(zr)lo((((z)))d
S.
where is an action, x is image belonging to this action, p(x) the probability of the image occupation in this action. However, this formula cannot be applied directly. For the discrete situation, assume each image is sample from an action distribution which obeys the Gaussian Drstribution N N(u.o²) and use X to estimate the action distribution. Hence. obtain the approximate entropy by sampling:
H(S') - N
Σ log( 2
exp
(8,
26
)
2x
where S' is the set of an action, X ~
In this part, build new translation family (A5) with two controllable parameters O,L, where 0 the orientation of the rectangle, is the translating distance of the rectangle. A5 has only one factor and indicates an action moving from the left the right, and Oandl. controls the entropy of this action.
5

---

Under review conference paper at ICLR 2021
180.0
4.0
180.0
3.25
3.5
3.00
135.0
135.0
2.50
90.0
90.0
2.25
2.5
2.00
2.0
1.75
0.0
10.0
20.0
30.0
40.0
50.0
0.0
10.0
20.0
30
40.0
50.0
(a) Entropy vs. 0 and
(b) KL diverge vs. and L
Figure Comparison between the entropy of actions and KI. divergence of latent variables.
As shown in Fig. B(a), a high value of L means long traveling area and a small overlap, which has large entropy. If the value of L is small enough, the images in the sequence action has small entropy or insignificant. Similarly, the action has more overlap when 0 90. Note that the maximum for both reached when f= or 180 and L is at its maximum. To measure how much information the model has learned, we select KL divergence to indicate the latent information. The experimental results reveal similar trend between the entropy and the KL diverge in Fig. Bb). Therefore, higher significance of the action ends in more information on the latent variables. In contrast, gradually increasing the pressure on the KL term, the latent information decreases until it reaches zero. One can infer that there exists a critical point that the model can learn information from the action. We call that the threshold of latent information. We hypothesize the significance of actions and latent information thresholds are positive correlation
almost the same, the
A.5
4 PLAIN IMPLEMENTATION ACCORDING TO THRESHOLDS
Endccer
Dataset
Tranabtioon
Deooder
RAII
High EResSre Lsa pecoure
BMT
FVAE
Mixed label
Figure 4: (a) The architecture of FVAE. Though the samples distribute in the dataset randomly, they have intrinsic significance. Under a high pressure (9), the significant actions can pass information alone the red line to itself along, and the information of insignificant actions is blocked. (b) The decoder receives the label information except for the target action.
We have discussed the situation with only one action, and there are various thresholds for different significant actions. Particularly, if 3 is large enough, information of the insignificant actions will be blocked, decaying to single factor discovery problem. From the modeling perspective, the learning process is similar to single action learning problem. However, the difficulty of disentanglement is that different kinds of ground-tru -round-truth actions are mixed, and single and a fixed parameter is unable to separate them. Therefore, the plain idea set different thresholds the learning phases, and then in

---

Under review as conference paper at ICLR 2021
JESN
dtacrete.ng mig
eppicionn scoPe
modulartyyooo
betall
loca
betad
bxtal
X3
22
682
Figure Re-entanglement phonomenon.
2.00
scale
rotation
1.75
pasx
1.50
1.25
1.00
20
40
60
80
100
120
80
100
(a) dSprites
(b) 3D Chairs
Figure 6: B vs. KL divergence on dSprites (left) and 3D Chairs (right). Each line denotes the dimensional KL diverge over /3 increasing.
each phase, we enforce the model to learn specific actions by blocking information of the secondarily significant actions. We propose fractional variational autoencoder (FVAE) which disentangles the action sequences step-by-step. The architecture of FVAE is shown in Fig. 4(a). The encoder consists of several groups of sub-encoders, and the inputs of the decoder are the concatenated codes of all sub-encoders. Besides, to prevent re-entangling the learned actions (see in Fif. 5). set different learning rates for the sub-encoders, which reduces the learning rate for the reset of N-1 groups and prevent the model from allocating the learned codes. The training process of FVAE similar the common operation in chemistry for separating mixtures-distillant we repeat the process of heating the liquid with different boiling points, and in each step, the heating temperature is different to ensure that only one component is being collected.
To separate a mixture of liquid,
Discussion Although Annealed follows the same principles the FVAE, it differs in the interpretation of the effects of beta, and it does not explicitly prevent mixing the factors. Moreover, the performance of AnnealedVAE depends on the choice of hyperparameter practice Locatello et al. )2019). "A large enough value" is hard to determine, and the disentangled representation re-entangled led for an extremely large C. To address this issue, here we introduce two methods to determine the thresholds in each phase for the labeled and unlabelled tasks.
4.1 LABELLED TASK
For the labeled setting, we focus on one type of action and clip the rest of them at first. However, the samples of one action are usually insufficient. For example, there are only three types of shapes on dSprites. Besides, the label information may be corrupted, and only some parts of the dataset are labeled.
To address these issues, we introduce the architecture shown in Fig. 4(b), in which the label information excepted for the target actions are directly provided to the decoder. We evaluate FVAE on the dSprites dataset (involving five actions: translating, rotating, scaling, and shaping). We first measure the threshold of each action, and the result is shown in Fig. 6(a) One
that the thresholds of translating and scaling are higher than the others. This suggests that these actions are significant and easy to be disentangled. This is in line with the results in Burgess al. (2018) Higgins et al. (201aa).
7

---

Under review as conference paper at ICLR 2021
Disentanglement on dSprites
0.40
WV
0.35
MIG metric
Leameed Latene Menabbe
Learned Later Later-Vereeiles Werattereetees
0.30
0.25
0.20
B-VAE
FVAE AnnealedVAEE--CCVEE
Ground Truth
(a) 8-VAE
(b) FVAE
(c) Disentanglement
Figure 7: Comparison between (a) 3- VAE and (b) FVAE. We visualize the average projections of the factors into the most informative latent variable on dSprites (only active units are shown). The colored lines indicate three types of shapes (red: eclipse, green: square, blue: heart). (c) The disentanglement scores of 3- 3-VAE and FVAE (MIG, [Chen et al. 2018)).
(MIG,[Chen
According to these thresholds, we then arrange three stages for dSprites. At each stage, we set up bigger number of 3 than the threshold of the secondary action. The pressure on the KL term also prevents the insignificant actions from being disentangled and ensures that the model only learns from the information of the target action. The training of each stage can be found in the Appendix. As shown in Fig. 7(a), the translation factor disentangled at first easily, while it is hard to distinguish the shape, orientation, and scale. Gradually, scaling and orientation also emerge in order. Nevertheless, should be noted that the shaping is still hard to be separated. This could be attributed to the lack of commonalities between these three shapes dSprites and motion compensation for smooth transition. In other words, in terms of shape, the lack of intermediate states between different shapes an inevitable hurdle for its disentanglement. Fig. 7 shows more substantial difference between the 8- VAE and the FVAE. B- V-V has an unstable performance compared to FVAE, and position information entangles with orientation on some dimension.
4.2 UNLABELLED TASK
For the unlabelled setting, we introduce the annealing test to detect the potential components. In the beginning, very large value for 3 is set to ensure that no action is learned. Then, we gradually decrease 3 to disentangle the significant actions. There exists a critical point in which the latent information starts increasing, and that point approximates the threshold of the corresponding action.
3D Chairs is an unlabelled dataset containing 1394 3D models from the Internet. Fig. 6(b) shows the result of the annealing test on 3D Chairs. One can recognize three points where the latent information suddenly increases: 60, 20, 4. Therefore, we arrange a three-stage training process for 3D Chairs (more details in the Appendix). As shown in Fig. 7(b),
the change of azimuth in the first stage. In the second stage, one can see the change of size, and in the third stage, one can see the change of the leg style, backrest, and material used in the making of the chair.
CONCLUSION
We demonstrated an example the effects of images' orientation on the disentangled representations. We have further investigated the inductive biases on the data by introducing the concept of disentangling action sequences, and we regarded that as discovering the commonality between the things, which is essential for disentanglement. The experimental results revealed that the actions with higher significance have larger value of thresholds of latent information. We further proposed the fractional variational autoencoder (FVAE) disentangle the action sequences with different significance step-by-step. We then evaluated the performance of FVAE on dSprites and 3D Chairs. The results suggested robust disentanglement where re-entangling prevented.
8

---

Under review conference paper at ICLR 2021
Stage (1)
Stage
Stage (2)
Pr
Stage
Pp
Stage (3) (a) dSprites
Stage (3)
(b) 3D Chairs
Figure 8: FVAE disentangles action sequences step-by-step. Latent traversals at each stage. The left the results on dSprites. The right is the results on 3D Chairs.
This paper proposed a novel tool to study the inductive biases by action sequences. However, other properties of inductive biases on the data remain be exploited. The current work focuses on an alternative explanation for disentanglement from the perspective of information theory. In the future, the influence of independence on disentanglement requires further investigation.
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, and Josef Sivic. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using large dataset of cad models. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3762-3769, 2014. 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 3588) 2013.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner, 2018. / http://://.aaaavooooo//////o//// 18099
Tian Qi Chen, X. Li, Roger B. Grosse, and D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. ArXiv, abs/ 1et1110022.099 2018.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations.
In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. Can: Creative adversarial networks, generating art by learning about styles and deviating from style norms. arXiv preprint arXiv:1706.0706 2017.
I. Higgins, Nicolas Sonnerat, Loic Matthey, A. Pal, C. Burgess, Matko Bosnjak, M. Shanahan, M. Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv: Machine Learning, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained

---

Under review as conference paper at ICLR 2021
variational framework. In 5th International Conference on Learning Representations, ICLR 2017 pp. 1-13, 2017a.
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: improving zero-shot transfer in reinforcement learning. In Doina Precup and Yee Whye Teh (eds.). Proceedings of 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1480-1490. PMLR, 2017b.
Insu Jeon, Wonkwang Lee, and Gunhee Kim. Ib-gan: Disentangled representation learning with information bottleneck gan, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. 35th International Conference on Machine Learning, JCML 2018, 6:4153 4,,,-111,,,000 2018.
Abhishek Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. ArXiv, abs/1711.0088 100848, 2018.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Guillaume Lample, Neil Zeghidour, Nicolas Usumier, Antoine Bordes, Ludovie DENOYER, and Marc textuutsitoe Aurelio Ranzato. Fader networks:manipulating images by sliding attributes. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.),NPPS 30, pp. 5967-5976. Curran Associates, Inc, 2017.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. 36th International Conference on Machine Learning, ICMI. 2019, 2019-June: 7247-7283, 2019.
Romain Lopez, Jeffrey Regier, N. Yosef, and Michael Jordan. auto-encoding variational bayes. In NeurlPS, 2018.
Information constraints on
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://hww....oeeee/e///////ssss
2017.
J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: Foundations and learning algorithms, 2017.
K. Ridgeway, A survey of inductvee biases for factorial rerresenattnnnnnnnnnneeeee leaoonnn--aannnnn..ee ArXiv, hbtp:/aa///11122222.0 2016.
Michal Rolinek, Dominik Zietlow, and G. Martius. Variational autoencoders pursue pea directions (by accident). 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22398-1247,,2000
Andrew Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):12402) 2019.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863-879, 1992. ISSN 08997667. doi: 10.1 162/neco. http:/eco..192.4.666 1992.4.6.863.
B. Schölkopf, D. Janzing, Peters, Eleni Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In /CML/, 2012.
Satosi Watanabe.
Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66-82, 82, 1960.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 1004-1013, 2018.
Yizhe Zhu, Jianwen Xic, Bingchen Liu, and Ahmed Elgammal. Learning feature-to-featuree translator by alternating back-propagatis for zero-shot learning. arXiv preprint arXiv: 97X
10

---

Under review conference paper at ICLR 2021
Phase
2
F1
5e-4
5e-5
5e-5
B2
5e-4
5e-5
E3
0
5e-4
31
100
40
32
60
20
2
Table 1: Training settings on dSprites and 3D Chairs. E; denotes the learning rate of i-th group of sub-encoder. denotes the regularisation coefficient before the KL divergence, 81 for dSprites, 32 for 3D Chairs
Factor
Threshold
Factor
Threshold
Shape
32
Chair size
60
Scale
100
Leg style
20
Orientation
5
Swivel
2
Position 120+
Unknown
2
Position 120+
Table 3: Thresholds of actions on 3D Chairs.
Table 2: Thresholds of actions on dSprites.
A APPENDIX
A.1 DATASETS
A.2 DSPRITES AND 3D CHAIRS
H
A
*4
(a)
(b)
Figure 9; Real samples from the training dataset.
A.3 TRAINING DETAILS
The basic architecture for all experiments follows the settings on Locatello et al. (2019). The hyperparameters of our proposed methods are listed in Tab. [] Tab. and show the measured thresholds of the intrinsic action sequences.
A.4 LEARNED ACTION SEQUENCES
Fig. T0 is the supplemental results of the experiments Sec 3.2.
11

---

Under review as conference paper at ICLR 2021
(a) A1
(b) A2
(c)
Figure 10: Latent traversal of the trained models on A1, A2, and A3.
A.5
THE HYPOTHESIS OF KI.
We hypothesize the KL diverge is inversely proportional to and positively proportional to H:
HS))
KL
C22
33
where C1,C2 are constant.
We examine this in Fig. 11
A.6 SAMPLES
12

---

Under review conference paper at ICLR 2021
5
45
3.5
3
2
1.
0.5
9000
10
20
50
60
70
Figure 11: KL vs. B.
Traversal: -2-2
Position
Position Y
Scale
Orientation
Figure 12: Latent traversal plots for FVAE dSprites. The top row show the real samples from dSprites.
13

---

Under review as conference paper at ICLR 2021
F
Traversal: -2 ~ 2
Chair size
Azimuth
Leg style
Material
Figure 13: Latent traversal plots for FVAE on 3D Chairs. The top row show the real samples from 3D Chairs.
14


Reviews:
[{"review_id": "uCwzsSi3Am5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. This paper gives a new comprehension of existing unsupervised disentangled representation learning method, regarding it as finding commonality of input data and disentangling action sequence information in factors.\n2.  This paper gives a new idea that \\beta has positive relationship with action significance and conduct an experiment to validate it.\n3.  This paper proposes a new variant of VAE called FVAE which learns the disentangled action sequence step by step.\nCons:\n1.  This paper mainly gives descriptions of insights while lacking some formulations to explain the settings and methods better.\n2.  Experimental results are not well organized, some axis lack corresponding labels, like Figure 5. Figures are not clear, like numbers in Figure 1 (a). \n3.  Some definitions are ambiguous, like x in equation 5. \n4.  Some descriptions in the paper are confusing, e.g. “We argue that the factors are not the key to disentanglement since the learned representations are changed while the factors are unchanged (A1, A3), and the learned representations do not change while the factors are changed (A1, A2).” This experiment, from my perspective, shows that the learned factors are disentangled in a particular form which is not consistent with the preset ground truth. And different action sequences are also different factors. Section 3.1 might be described in a more considerate way to show what the experiment results really indicate. \n5. There exist some typos in this paper, like “leaned” for “learned”.\n\nOverall review:\nThis paper gives a new comprehension of existing disentangled representation learning by regarding it as finding disentangled action sequences, which is interesting and has some good insights. However, some ideas should be supported by clearer formulations and some conclusions of experiments are not valid. Moreover, the logic of this paper is a little unclear, and experimental figures are incomplete. With some modifications, this paper could be an excellent paper..\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "t4Js5vRgHqo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper addresses the problem of disentangling representations using Variational Autoencoders. In particular, the authors introduce the concept of disentangling action sequences and propose the fractional variational autoencoder framework to disentangle them step-by-step. To this end, they analyze the inductive biases on the data and define latent information thresholds which are correlated with the significance of the actions.\n\n##################################################################\n\nStrengths:\n- The paper tackles the important problem of disentangling representations.\n- Overall, the paper is well structured. In particular, the introduction section clearly motivates the problem and summarises existing approaches.\n- The idea of disentangling action sequences is interesting as it allows to analyse the inductive bias on the data.\n\n##################################################################\n\nWeaknesses:\n- Fractional variational autoencoder (FVAE) proposed in this paper is closely related to the work of Burgess et al. (2018). Although authors include a brief discussion comparing both methods, the main novelty of FVAE (i.e. explicitly avoid mixing the factors and defining thresholds to prevent re-entanglement for extremely high capacity) is still not sufficiently emphasised throughout the paper. Moreover, it would be good to include experimental comparisons to Annealed VAE (for instance in Figure 6) to give more insights on the relevance of the proposed approach.\n- Description of the toy dataset family is not easily understood and it would be good to clarify annotations in Figure 1(a). Since Figure 9 of Appendix is clear enough, it might be nice to include it in the main paper to help the reader follow the analysis of the corresponding experiment. In the latter, it is shown that the disentangled representations are not invariant to orientation of rectangles (A1, A3). Here, one can assume that positions x and y and orientation of rectangles contribute differently to reconstruction. Hence, it would be interesting to see the effect of progressively increasing the bottleneck capacity on the obtained representations, as proposed in Burgess et al. (2018). Would this help disentangle position at first then orientation of rectangles?\n- While a quantitative analysis has been provided in Figure 6 using the MIG metric to compare FVAE and Beta-VAE, it is still insufficient to make clear conclusions on the performance of the proposed method. Several metrics (e.g. Mutual Information Gap, Modularity, etc.) and evaluation benchmarks have been integrated in DisLib (Locatello et al. (2019)) allowing easy evaluations of disentangling approaches. I recommend using it for further quantitative analysis. \n- In Section 2, authors present the concept of disentangling representations and describe the work of Locatello et al. (2019) which shows the necessity of inductive bias to unsupervisedly disentangle the underlying factors. After mentioning that “a formal definition of the inductive bias is still unavailable”, this point would be more clear with some examples, for instance the assumption in Burgess et al. (2018) that Beta-VAE aligns latent dimensions with components that make different contributions to reconstruction.\n- In Section 3.1, authors mention that existing models disentangle the ground-truth factors by accident. This would be a little misleading to previous claims on the role of inductive bias on the data (or the model) which allows to achieve disentanglement  Locatello et al. (2019). I suggest more clarification to this point.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mwf_pCynhc9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors proposed fractional variational autoencoder (FVAE) for the learning of disentangled representation where the action sequences can be extracted step-by-step. Experiments are shown to illustrate how the algorithm works.\n\n#################\n\n\n. The authors proposed FVAE but the associated objective function is not introduced explicitly, which is confusing. Is it the same as the objective of \\beta-VAE?\n\n. Fig.3: 1) What's the KL divergence here? Is it between the posterior and the prior? 2) It's claimed that the trend of KL divergence is consistent with that of entropy. But it is hard to see from Fig. 3. 3) It is claimed that the significance of action is related to the capacity of learned latent information. Based on Fig. 3, this conclusion is not convincing. Also, Fig. 3 is obtained based on a toy dataset. To claim it as a main contribution, the conclusion needs to be verified on other datasets as well.  \n\n. Section 4.1: What's definition of the label here? It's not clear. Is it like the types of shapes on dSprites?\n\n. Section 4.1: The training on dSprites includes two phases: find thresholds and then train different stages. 1) The authors arranged  three stages for dSprites. This seems arbitrary. Why not four or five stages? 2) What's the training objective function of each stage? 3) How are the curves in Fig. 5 derived? More explanation is required.\n\n. Section 4.2: It is claimed that ``One can recognize three points where the latent information suddenly increases: 60, 20, 4.'' This is hard to see from Fig. 5b) as all curves look smooth. Thus, the following three-stage training process is questionable. The training for unlabeled task needs more study.\n\n. The experiments are limited. There are a lot of papers regarding disentangled representation, and the authors only compared with \\beta-VAE. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "e2zns8ME8EK", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Manuscript Summary\n====\n\nThis paper constructs a disentanglement problem from a temporally causal view, where data are observed in sequences, and where actions cause those observations to change as the sequence progresses in time. Their stated objective is to recover the specific actions and their parameters (e.g. rotation and translation, and their magnitudes/signs).\n\nThe authors thus construct a \"Fractional VAE\" (FVAE), and then construct sequences from Dsprites and Chairs based on their statement of the problem.\n\nInitial Decision (from this reviewer), Review, and Reasoning\n====\n\nI think this paper should be rejected; were this a journal, I would suggest at least major revision.\n\nOverall the concept of bringing temporal causality (which for some cases *is* valid as the causal diagram/frame) into the disentanglement problem statement is a good idea. However, after that point in the manuscript, I cannot understand what has done. For example, section 2 is a restatement of previous work, and section 3 begins with an explanation of the dataset construction. Section 3.3, section 4, and Figure 4A I think describe the paired asymmetric autoencoder method, but a sparse few paragraphs are given at this point. What they _do_ describe is a set of \"sub-encoders\" with varying compression rates $\\beta$. However, beyond varying the rates, it's not clear how particular \"ground truth\" factors (e.g. $\\theta, L$) can be selected for and locked in to specific latent factors in an unsupervised manner, or even if this should happen.\n\nBy varying $\\beta$ we receive different amounts of information in the representation, but how can we ensure across \"learning phases\" disentanglement? Further, if these KL divergences are set to different $\\beta$, this means we don't have a divergence for the joint representation? (the concatenation of the sub-encoders) So how can we ensure that these are disentangled themselves? While successively learned encodings would optimally not include previously encoded data, why would these encoders learn separate concepts instead of coarse grained representation with all concepts to successively finer representations (or refinements to those coarse grained representations) with each successive sub-encoder.  Or are these separate phases repeated?\n\nPerhaps these questions have answers in the positive, but they should be answered by the manuscript.\n\nI further cannot make a connection between the actions sequences and the training methods/arch. I think I have understood both (...save for the above highlighted problems), but I cannot understand where the sequences come in practically speaking, even modulo the aforementioned issues. How does the FVAE or its training scheme use this information? Does it use this information?\n\nI think the positive experimental results in Figure 6c mean that there is something here. However, I cannot tell given section 4 what is actually being done.\n\nSuggestions\n====\n\nI suggest a clear procedure section with numbered steps. It is of vital importance that the reader understand what has been done. If it is already there, it should be made much more obvious/clear.\n\nI think the connection between sequences of images under actions and the proposed method needs to be made, or, if I missed this connection, should be made clear.\n\nThe initial portion of section 3 concerning dataset construction might also be moved to much later.\n\nThere are philosophically challenging sections which I did not comment on in the review portion. I think these are approximately orthogonal to the method due to the scope of the problem: rotation/translation may be disentangled. Can dog breeds *be* disentangled, even in theory (example from Section 3)? Disentanglement of simple mechanisms/\"actions\" are perfectly acceptable at least in my opinion for the state of the field at this moment. Using more complex examples may not be helpful. Similarly, the discussion in section 1 raises questions that are unrelated to the later method. Since a literature review is undertaken in Section 2, the paper could have started at \"In this paper, we first demonstrate that instead of the ground-truth factors the disentangling approaches [should] learn [disentangled] actions.\", with an update for phrasing.\n\nI would also give the paper another read through for grammar.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Lnoczyks-X", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nIn this submission, a common modelling assumption for unsupervised disentanglement is challenged: that the disentangled representation follows the independence structure of the underlying (data generating) factors. Instead, the paper proposes to consider *action sequences* which describe how datapoints are interrelated. The paper provides evidence that the capacity of the latent representation (controlled by Lagrange parameter beta in beta-VAE related models) is related to the significance of particular action sequence for disentanglement. To leverage this insight, the fractional VAE (FVAE) is proposed, consisting of several sub-encoders and different training stages. The disentangling properties of the FVAE is demonstrated on the dSprites and 3D chairs datasets, with the FVAE performing favourably to the beta-VAE w.r.t. the Mutual Information Gap (MIG) disentanglement metric on dSprites.\n\n### Strengths:\n- Novelty / relevance: The submission addresses the important topics of inductive biases and disentangling factors in learning disentangled representations and suggest the interesting and novel concept of action sequences which seems to be related to the general idea of uncovering symmetries with deep latent variable models. In particular, the annealing approach with respect to the KL-divergence Lagrange parameter beta in the FVAE setting to separate “significant modes” might pose a relevant insight useful in other related approaches and to a more broader audience. \n\n### Weaknesses:\n- Technical quality / significance: The submission mentions the similarities to approaches like AnnealedVAE by Burgess et al. and qualitatively discusses differences and relates some results to this competing approach, but an empirical evaluation of the proposed approach to the competing method is missing. This is quite important, as the technical details of annealing the capacity of the latent representation seem very much alike. Also comparing the disentangling scores to more state-of-the-art approaches like FactorVAE (Kim and Minh) would be important. The evaluation is solely done with respect to the beta-VAE which might not be the most relevant competitor here. For instance, figure 3 in Burgess et al. reports a similar finding as provided in figure 7a in the submission, i.e. controlling the information capacity disentangles first positional / translational factors, then scale and then orientation / shape. Therefore, it is difficult to assess the validity of the claims of the proposed approach and whether a significantly different contribution than in Burgess et al. is made.\n- Figure 5a suggests for dSprites that position/translation, scale and shape are the relevant actions in that order. However, the result in figure 7a suggest, that first translation, then scale and lastly rotation are gradually disentangled which seems to contradict the first result in figure 5a. Shouldn’t these be the same?\n- Figure 6a and 6b are not explained or discussed and their interpretation is not clear. A reader might be familiar with similar plots e.g. in the paper by Higgins et al., but still the key insight should be stated somewhat more clearly in the paper.\n- Clarity: At times it is difficult to follow the presentation of the content in the paper and in some cases I find it hard to follow the statements and conclusions. For instance:\n- Toy example in section 3.1, especially last paragraph: I believe the interpretation of the results in figure 1 requires a little bit more explanation. As I understand it, the ground-truth factors here are the positions X, Y of the rectangles. The dataset provides the variables (i) orientation of the rectangle, (ii) coordinates in either Cartesian or polar coordinate system. I do not quite follow the statement that *“[…] learned representations are changed while the factors are unchanged (A1, A3), and the learned representations do not change while the factors are changed (A1, A2).”* In case (A1, A3) I would say the latent representation is the same up to permutation of coordinate axes / rotations, which is inherent to VAE / PCA approaches. I.e. the meaning of the axes would be still the same (up to these transformations). Therefore, I am also not quite sure about the statement: *“As we have shown in Sec. 3.1, the orientation of the rectangle can affect the direction of the disentangled representation”* (p. 5). The “direction” of the representation is less relevant, as the interpretation of the axes is still the same. However, I might miss the point which is tried to be made here. Could the authors comment on that?\n- Definition of action sequence, section 3.2: The paper tries to motivate “action sequences” but in my opinion the notion remains somewhat unclear in an abstract setting. A “meaningful action sequence” is defined as *“a sequence / ordered permutation of elements from a subset of the dataset, which reveals the relationship among the elements”*, with elements being images here. In a simple example as scaling or translating objects, this notion and the distinction to “ground-truth factors” might be clearer. However, in more complex / less structured examples, say images of faces, the difference between “action sequence” and “ground-truth factors” is not very clear to me. The paper suggests for a more formal definition to consider Higgins et al. but, in order to be self-contained and clear, a more explicit and formal definition of this notion is required in the paper, in my opinion. Could the authors maybe provide a more formal definition? \n\n### Additional Feedback:\n- Page 6 and figure 3: *“Please note that the maximum for both are reached when theta=90 and L is at its maximum.”* Figure 3 suggests that the maximum (yellow region) is reached for large L and theta close to 0 or about 180. It seems that there is a discrepancy between the description and the figure.\n- Figure 5, page 7: In 7b the legend specifies integers, but it is not clear, what these integers encode. And is it maybe *“KL divergence vs beta”* (-> *”y against x”*) in the caption?\n\n- Abstract (p. 1): I would suggest rephrasing the following sentence:  *“We demonstrate the data itself, such as the orientation of images, plays a crucial role in disentanglement and instead of the factors, and the disentangled representations align the latent variables with the action sequences.”*\nMaybe get rid of the first *“and”* as well as making clear what *“factors”* (maybe rather *“ground-truth / separating factors”*?) are meant. On the first read, this sentence was quite confusing to me.\n- Introduction (p. 1): Second sentence, *“thinking”* -> *“think”*.\n- Introduction (p. 1): Third sentence, *“[…] single glance this is because […]”* -> *“[…] single glance. This is […]”*.\n- Introduction (p. 1): Notion paragraph first word, *“the”* -> *“The”*.\n- Introduction (p. 1): Notion paragraph, *“[…] a question arise here is […]”* -> *“[…] a question which arises here is: […]”*.\n- Figure 1, caption: *“leaned”* -> *“learned”*.\n- Section 3.3, incomplete sentence after equation 5 or unnecessary *“,”*.\n- Section 4, page 6: *“[…] leading to the disentangling process decays […]”* -> *”[…] decaying […]”*\n- Section 4, page 6: *“[…] targeted action into the leaned codes.“* -> *“[…] learned […]”*\n- Figure 7, page 8: Full stop *“.”* missing in the last sentence of the caption.\n\n### Recommendation:\nIn general, the paper deals with relevant issues in learning disentangled representations and provides interesting tools to address some of these aspects. In particular, the annealing procedure in the FVAE is potentially a relevant contribution. However, the relation to similar approaches is not evaluate adequately, in my opinion, which makes it difficult to assess the justification of some claims. Also, a careful revision of the submission seems advisable which might clarify some of the aspects raised above. In the current form, I believe that the paper is not ready for publication and I would rather see this submission rejected. Nevertheless, I am willing to reconsider my rating if the authors are able to address some of the concerns and questions raised above.\n\n### Post-Rebuttal:\nI want to thank the authors for their responses and clarifications. I think the revision already improved the quality of the submission quite a bit. However, I still believe that there are some aspects which need a better presentation and clearer discussion. \n\nFor example, a more direct discussion and (empirical) comparison to other approaches like AnnealedVAE is necessary, as also other reviewers pointed out, to justify the points made (qualitatively) in the paper. The added results in figure 6c already provide results in that direction.  \n\nI appreciate the clarifications in the notions of action and action sequence. Although I agree that the notions are comprehensible in the toy example and dSprites setting, I still think that the point I raised in my initial review applies. In order to provide a well-defined notion a more formal definition is required. To me it is still unclear what an action sequence in the case of e.g. images of faces should be.\n\nI genuinely believe that the proposed approach might pose a relevant contribution but the paper lacks an adequate presentation at the moment, in my opinion. Therefore, I stand with my initial recommendation that this submission is not ready for publication and I endorse rejecting the paper. However, I would like to encourage the authors to do a major revision taking the issues raised by the reviewers into consideration and to submit again.\n\n\n### References: \n- Higgins et al., “beta-VAE: Learning basic visual concepts with a constrained\nvariational framework”, ICLR 2017.\n- Kim and Mnih, “Disentangling by factorising”, ICML 2018.\n- Burgess et al., ”Understanding disentangling in beta-VAE”, NeurIPS 2018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial round of reviews showed a consensus among the reviewers that the presentation of the paper was poor, the novelty was unclear, claims were not properly justified, and the experimental evaluation and discussion were quite insufficient. The authors provided a rebuttal and an updated version of the paper. Although the updated paper demonstrated that the proposed approach indeed provides some benefits, it appears that the authors were not successful to address the numerous but constructive reviewers' comments.

The paper is not ready for publication in ICLR 2021 and can benefit from major revisions and careful proofreading. 
Decision: Reject