On the Necessity of Disentangled Representations for Downstream Tasks
Authors: Ruiqian Nai
Zixin Wen
Ji Li
Yuanzhi Li
Yang Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation disentanglement
representation learning downstream task

Abstract: A disentangled representation encodes generative factors of data in a separable and compact pattern. Thus it is widely believed that such a representation format benefits downstream tasks. In this paper, we challenge the necessity of disentangled representation in downstream applications. Specifically,  we show that dimension-wise disentangled representations are not necessary for downstream tasks using neural networks that take learned representations as input. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network architectures. Moreover, our study reveals that the informativeness of representations best accounts for downstream performance. The positive correlation between informativeness and disentanglement explains the claimed usefulness of disentangled representations in previous works. 

Under review as a conference paper at ICLR 2023
ON THE NECESSITY OF DISENTANGLED REPRESENTA- TIONS FOR DOWNSTREAM TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
A disentangled representation encodes generative factors of data in separable and compact pattern. Thus it is widely believed that such a representation for- benefits downstream tasks. In this paper, we challenge the necessity of dis- entangled representation in downstream applications. Specifically, we show that dimension-wind wise disentangled representations are not necessary for downstream tasks using neural networks that take learned representations as input. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network ar- chitectures. Moreover, our study reveals that informativeness of representations best accounts for downstream performance. The positive correlation between the informativeness and disentanglement explains the claimed usefulness of disentan- gled representations in previous works.
INTRODUCTION
Disentanglement has been considered an essential property of representation learning (Bengio et al.,
2013 Peters et al. 2017; Goodfellow et al., 2016; Bengio et al. [2007] [Scmiddhuber [9922] Lake ct al. 2017: Tschannen et al. 2018). Though there is no widely accepted formal definition yet, the fundamental intuition that a disentangled representation should separately and distinctly capture information from generative data factors (Bengio et al. 2013). In practice, disentanglement is often implemented emphasize dimension-win relationship. ie.., a representation dimension should capture information from exactly one factor and vice versa (Locatello et al. 2019b] Higgins et al., 2016 Kim & Mnih| 2018 Chen et al., 2018 Eastwood & Williams] [2088] Ridgeway & Mozer,
2018 Kumar et a 2011: & Tran 2019). Disentangled representations offer numan--nterppeee
factor dependencies. Therefore, in theory, they are robust to variations in the natural data and are expected to benefit downstream performances (Bengio al. 2013).
Researchers are interested in empirically verifying these purported advantages. Especially, they fo- the following two-staged tasks: (1) extracting representations in an unsupervised manner from data, (2) then performing downstream neural networks training based on learned representa-
tions (van Steenkiste et al. 2019 Locatello et al.) 2019a] Dittadi et al. 20200) Locatello al.. 22200) Among various downstream tasks, except the ones that explicitly require disentanglement (Higgins al. 2018b Gabbay Hoshen [202T] Schölkopf al. (222). I). abstract visual reasoning widely
recognized as popular testbed (van Steenkiste et al. Locatool
Locatello et al. 2020) /Schölkopl et al.
2021 The premise behind aligns with the goals of machine intelligence (Snow al., 1984,
Carpenter et al. 1990). Moreover, its mechanism ensures valid measurement of representations downstream performance (Fleuret et al. 2011 Barrett et al. 2018).
In the abstract visual reasoning task, intelligent agents are asked to take human IQ tests, i.e., predict
the missing panel of Raven's Progressive Matrices (RPMs) (Raven) 1941). Indeed it a challenging task for representation learning (Barrett et al.) 2018: van Steenkiste et al. 2019). Disentanglement literature often takes this task as an encouraging example to show that disentanglement leads to
quicker learning and better final performance (van Steenkiste al. 2019 Locatello et al. 2020
Schölkopf et al. (02T).
However, the abstract visual reasoning task, find that rotating disentangled representations, ie., multiplying the representations by an orthonormal matrix, has no impact on sample efficiency and final accuracy. We construct the most disentangled representations, i.e., normalized true factors.

---

Under review as a conference paper at ICLR 2023
Then we solve the downstream tasks from them and their rotated variants. As shown in Figure there is little difference between the accuracy curves of original and rotated representations through- out the learning process. On one hand, this phenomenon is surprising since the rotation decreases dimension-wine wise disentanglement by destroying axis alignment (Locatello et al.] 2019b). Indeed, in Figure|2bjwe we can observe notable drops in disentanglement metric scores (first 5 columns). Our find- ing demonstrates that disentanglement does not affect the downstream learning trajectory. which is against the commonly believed usefulness of disentanglement. On the other hand, it is not surprising since we apply an invertible linear transform. We can observe that Logistic Regression (LR) accu- racy remains 100% before and after rotation, indicating that a simple linear layer could eliminate the effects of rotation.
2a
Per such facts, some questions arise: Are disentangled representations necessary for two-staged tasks? If not, which property matters? To address them, we conduct an extensive empirical study based on abstract reasoning tasks. Our contributions are as follows.
We challenge the necessity of disentanglement for abstract reasoning tasks. We find that (1) entan- gling representations by random rotation has little impact, and (2) general-purpose representation learning methods could reach better or competitive performance than disentanglement methods.
Following Eastwood & Williams 2018 we term what information the representation has learned as informativeness. We show that informativeness matters downstream performance most, (1) Logistic regression (LR) accuracy on factor classification correlates most with downstream per- formance, comparing with disentanglement metrics. (2) Conditioning on close LR accuracy, dis- entanglement still correlates mildly. (3) The informativeness behind the previously argued usefulness of disentanglement since we observe positive correlation between LR and disentan- glement metrics.
We conduct a large-scale empirical study supporting our claim. We train 720 representation learn- ing models covering two datasets, including both disentanglement and general-purposs methods. Then we train WReNs (Barrett et al.J 2018)) and Transformers (Vaswani et al., 2017 Hahne et al. 2019) using the outputs of each representation learning model perform abstract reasoning, yielding a total of 7200 abstract reasoning models.
2 RELATED WORK
Disentangled representation learning. There is no agreed-upon formal definition of disentangle- ment. Therefore, in practice, disentanglement is often interpreted as a one-to-one mapping between representation dimensions and generative factors of data, which we term "dimension-win wise disen- tanglement". It requires that the representation dimension encode only one factor and vice versa (Locatello et al., 2019b Eastwood & Williams) 2018: Kumar al. 2017, Do & Tran 2019). Be- sides dimension-win wise disentanglement, Higgins et al. (2018a) propose a definition from the group theory perspective. However, its requirement in interaction with the environment prevents applicable learning methods for existing disentanglement benchmarks (Caselles-Dupré et al. 2019).
Adopting the dimension definition, researchers develop methods and metrics. SOTA disentan- glement methods are mainly variants of generative methods (Higgins et al. 2016)] Kim & Mnih
2018 Burgess et al. [2018: Kumar al. 20177; Chen al., 2018 [ Jeon et al. 2018 Lin al.
2020 Corresponding metrics are designed in the following ways (Zaidi et al. 2020): interven-
ing factors (Higgins al., 2016, Kim & Mnih 20088) estimating mutual information (Chen al.
2018), and developing classifiers (Eastwood & Williams 2018 Kumar et al. 2017). Another line of work related to disentangled representation learning the Independent Component Analysis (ICA) [Comon] 1994). ICA aims to recover independent components of the data, using the mean corre- lation coefficient (MCC) as the metric. However, ICA models require access to auxiliary variables (Hyvainnen et al. 2019). leading to inevitable supervision for image datasets training (Hyvarinen & Morioka| 2016 Khemakhem et al. 2020aa b Klindt et al. 2020). In this paper, we focus on the downstream performance of unsupervised representation learning.
Downstream tasks. is widely believed that disentangled representations benefit downstream tasks. Intuitively, they offer a human-understare understandable structure with ready access to salient factors, hence should be enjoying robust generalization capacity (Bengio et al.] (013)] Do & Tran 2019). Several works conduct empirical studies on downstream tasks to support the notions above, includ-
2

---

Under review as a conference paper at ICLR 2023
ing abstract reasoning (van Steenkiste et al. 2019). fairness (Locatello et al. 2019a). and sim2real transfer (Dittadi et al. 2020). Among these works, van Steenkiste et al. (2019) provide the most en- couraging evidence from abstract reasoning tasks. We adopt their settings and investigate the same tasks. However, their results are questionable. Firstly, it underestimates factors' linear classification accuracy, yielding weaker correlation between informativeness and downstream performance (see Figure g in Appendix A.3). Moreover, only variants of VAEs are considered. We address these issues and achieve opposite conclusions.
Abstract visual reasoning has been a popular benchmark to measure the representation's down- stream performance, especially in disentanglement literature (Steenbrugge et al. 2018 van et al. 20200 Locatello et al. [2200] [Schölkopf et al. [021]] The most common type is the Raven's Progressive Matrices (RPMs) (Raven) 1941). which highly emphasize abstract and relational reasoning capacities and effectively represent human intelligence (Snow et al., 1984 Carpenter et al. 1990). To solve RPMs, one is asked to complete the missing panel of a x grid by exploring the logical relationships of 8 context panels. Moreover, abstract visual reasoning a well-developed benchmark for representation learning. Given that is coupled with a principle treatment of generalization (Fleuret et al.) 2011), neural network can not solve reasoning tasks by simply memorizing superficial statistical features. Besides, it can avoid pitfalls where test-specific heuristics learned by downstream models obscures the original properties of representations (Barret) et al,, 2018). To summarize, (1) the goal of abstract visual reasoning highlights our requirements for representation learning, and (2) its mechanism ensures valid measurements. For these reasons, we focus on the necessity of disentanglement for the abstract reasoning task.
Steenkiste et al.
3 DOWNSTREAM BENCHMARK: ABSTRACT VISUAL REASONING
This section contains background on the downstream benchmark framework. We first introduce the definition of the abstract visual reasoning task. Then we present the framework's 's ingredients: representation learning methods, metrics, and abstract reasoning models.
3.1 ABSTRACT VISUAL REASONING AS TWO-STAGED TASK
The abstract visual reasoning tasks are highly inspired by the famous human IQ test, Raven's 's Progressive Matrices (RPMs) (Raven) 1. Figure shows an RPM question in our evaluation dataset. There are eight context panels and one missing panel in the left part of the figure. The context panels are arranged following some logical rules across rows. During the test, the subject must pick one of the six candidates listed in the right part to fix the missing panel. The goal is to maintain the logical relationships given by the contexts. More details of RPMs are available in Appendix A.4
?
Figure 1: An example of RPM
3DShapes from van Steenkiste et al. 2019
We adopt RPMs as a downstream benchmark following van Steenkiste al. (2019). To measure the necessity of disentanglement for downstream tasks, we separate the evaluation process into two stages: (1) In Stage-1, representation learning models extract representations from images of which RPMs consist, and (2) in Stage-2, abstract reasoning models predict the missing panels from the frozen representations of contexts and answer candi- dates. Correspondingly, we denote representation learning models as Stage-1 models while abstract reasoning models as Stage-2 models. For Stage-1, we measure the disentanglement properties of representations. diverse set of Stage- and Stage-2 models are trained, yielding multiple sam- ples from the joint distribution of representation metric scores and downstream accuracy. Finally, we study the relationships between representation qualities and downstream performance. We aim to investigate whether more disentangled representations perform better abstract reasoning tasks.
The two-staged framework leverages large-scala experiments to reveal connections between the dis- entanglement of representations and their downstream performance. provides a precise measure- ment of the importance of disentanglement. Therefore the two-staged framework bidely-accepted (van Steenkiste et al. 2019 Locatello et al. 2019a Dittadi et al. 2020 Locatello et al. 2020).
3

---

Under review as a conference paper at ICLR 2023
3.2 BACKGROUND OF REPRESENTATION LEARNING
Disentangled representation learning methods. The seminal works of Higgins et al. (0116) and Chen al. (2016) embody disentanglement by augmenting deep generative models (Kingma & Welling 2013; Goodfellow et al.. 2014). For disentangled representation learning methods, we Tocus variants of VAH. Namely, 3-VAE (Higgins et al. (016), AnnealedVAR (Burgess et al., 3-TCVAE (Chen et al. 2018), FactorVAE (Kim & Mnih 20188) and DIP- MAE (Kumar et al,,
2018
2017 They achieve disentanglement mainly by encouraging independence between representation
dimensions. Please refer to Appendix A.2 for details.
General-purpose representation learning methods. In our study, methods not (explicitly) encour- aging disentanglement are called general-purposs purpose methods. We take BYOL (Grill et al.) (2200 as representative. YOL is negative-free contrastive learning method. creates different "views" of an image by data augmentation and pulls together their distance in representation space. To avoid collapsing to trivial representations, a predictor appending to one of the siamese encoders and ex- ponential moving average update strategy (He al. 2020) employed. It does not encourage disentanglement due to the lack of regularizers. Indeed, the empirical evidence Cao et al. (2022) demonstrates that representations learned by BOL have weak disentanglement.
Representation property metrics. Considered properties of representations cover two axes met- rics: disentanglement metrics and informativeness metrics (Eastwood & Williams 2018). We in- clude Beta VAE score (Higgins et al., 2016), FactorVAB score (Kim & Mnih] 2018). Mutual Infor- mation Gap (Chen al. 2018)_ SAP (Kumar al. 2017), and IDCT Disentanglement (Eastwood & Williams 2018). Locatello et al. (2019b) proves their agreement on VAE methods with exten- sive experiments. Though their measurements are different, their results are positively correlated. On the other hand, informativeness requires representations to encode enough information about factors. In this work, we employ Logistic Regression (LR).
a favorable metric adopted by un- supervised pretraining literature (He et al., 2020; Grill et al. 2020 Caron et al., 2021). Given the weak capacity of linear models, a higher LR accuracy ensures that sufficient information explicitly encoded. However, it does not emphasize dimension-wise encoding pattern like disentanglement. To distinguish, we term the property indicated by LR as informativeness.
3.3 BACKGROUND OF METHODS FOR ABSTRACT REASONING
In Stage-1, I, we extract representations of eight context panels (the left part of Figure D and six answer candidates (the right part of [i. Then in Stage- 2. downstream models perform abstract reasoning from the (frozen) representations. Abstract reasoning models evaluate whether filling the blank panel by a candidate follows the logical rules given by contexts. For a trial T; of one candidate a, {a1.-.. 06) and eight context panels C {(i,-.. 6s), its score is calculated as follows:
Y = Stage
Stagel (T:) = {Stagel(ci), ..., Stage (cx)) U (Stagel(a.),, (I)
where Y; is the score of trial T,, Stagel Stage2(-) denote the forward process of the Stage- I and Stage-2 models, and Stagel (T;) the representations contexts and candidate a, After evaluating all trials (Ti, , T.)., the output answer is arg max; Y,.
We implement two different well-defined structures of Stage-2 models, namely, WReN (Barrett et al., (2188) and Transformer (Vaswani et al.) 22177: Hahne et al., 2019). First, they employ an MLP or a Transformer to embed an RPM trial. Then an MLP head predicts a scalar score from the embeddings.
4 EXPERIMENTS
In this Section, we conduct systematic empirical study about representation properties' impacts on downstream performance. First, we introduce our experimental conditions in Section [.. Then we provide empirical evidence to challenge the necessity of disentanglement (Section 2 and to tell which property matters (Section 4.3)
4.1 EXPERIMENTS SETUP
We build upon the experiment conditions of van Steenkiste et al. 2009 tasks, i.e., RPMs, are solved through two-stage process:
Abstract visual reasoning representations
Stage-1
Stage-2
data
4

---

Under review as a conference paper at ICLR 2023
RPM answers. We first train Stage-1 I models in unsupervised manner and evaluate their disen- tanglement and informativeness. Then Stage-2 models are trained and evaluated on downstream tasks, yielding an abstract reasoning accuracy of representation. Provided with large amount of (representation property score, downstream performance) pairs, we conduct a systematic study to investigate the necessity of disentanglement. More implementation details are available in Ap- pendix [
Datasets. We replicate the RPM generation protocol in van Steenkiste et al. (2019). The panel images consist of disentanglement benchmark image datasets, namely, Abstract dSprites (Matthey al. 2017]] van Steenkiste et al., 2019) and 3DShapes (Burgess & Kim 2018). The rows of RPMs are arranged following the logical AND of ground truth factors. As for hardness, we only reserve hard-mixed, whose contexts and candidates are more confusing. According to the generation pro- cess, the size of generated RPMs is sufficiently large (about 10L1), allowing us to produce fresh samples throughout training.
Reference models. Stage-1 models extract representations from RPM's panels. To ensure the gener- alizability the results, we include 360 disentangled VABs (denoted as DisVAEs) and 360 B BYOLs. Our choices of Stage-1 models cover both disentangled and genrroe-purpose representation learning methods. Moreover, we are interested in the overall relationship between representation properties and downstream performance. Therefore need to study the correlation between two distributions, i.e, representation metric scores and downstream performance. For this, we include various sam- ples for both Stage-1 1 and Stage-2 ensure they representative enough. For Stage- diverse set of configurations are included for each type of representation learning model. According to the histograms in Appendix C.4] our choices span various disentanglement and informativeness scores. For St to better estimate the downstream performance distribution, we use multiple Stage-2 configurations for each representation instead of searching for the best one. Specifically, we train 10 Stage-2 models (5 WReNs and 5 Transformers) for every Stage-1 I model. Stage-2 configurations are randomly sampled from search space described Appendix AA.3 and shared across Stage-1 models. By this, we ensure fair comparisons across representations.
Training protocol. Training conducted two-staged. Firstly, we train Stage-1 models in an unsu- pervised manner on the dataset consisting of RPMs' panels, i.e., Abstract dSprites or 3DShapes. For DiE models, we use the training protocol of van Steenkiste et al. 2019), while for BYO mod- els, follow Cao et al. (2022). In Stage-2, all models trained for 10K iterations with batch size of 32. After every 100 iterations, we evaluate the accuracy on newly generated 50 of unseen RPM samples for validation and another 50 mini-batcher for testing.
Evaluation protocol. We first evaluate the two stages separately. Then we analyze the relationship between the two stages, i.e., representation properties and downstream performance. Specifically, to challenge the necessity of disentanglement, we are interested in whether more disentangled repre- sentations lead better downstream performance. Further, if it turns out that disentanglement is of limited importance, can we find another metric that better accounts for downstream performance? Therefore, for Stage- we employ representation metrics described in Section 3.2 to measure two aspects: disentanglement and informativeness. For all Stage-1 models, we compute the following metric scores: BetaVAE score, FactorVAE score, MIG, SAP, and I.R accuracy. DCI Disentanglement is only evaluated for DisVAEs since it takes hours to develop the Gradient Boosting Trees required during the evaluation process on high- high-dimensional representations of BYOLs (Cao et al., 2022). For Stage-2, we inspect accuracy on newly generated test sets every 100 iterations, yielding accuracy for multiple training steps. Since every step sees fresh samples, we employ training curves to measure sample efficiency. We also report accuracy Hscuracy-Hsamples curves in Appendix C.2
To summarize the downstream performance of a Stage-1 model, over 5 WReNs or Transformers in Stage-2, we report the mean accuracy denoted WReN or Trans., and max accuracy denoted as WReN' or Trans.". Finally, we calculate the rank correlation (Spearman) between the mean per- formance of Stage-1 models (WReN and Trans.) certain Stage-2 steps and their Stage-1 metric scores. A larger correlation indicates a higher significance of the representation property on down- stream performance.
5

---

Under review as a conference paper at ICLR 2023
0.6
1.0
0.6
0.5
0.
0.9
Scort
Stags
(a)
(b)
Figure 2: Downstream accuracy and metric scores of original/rotated true factor values with 5 con- figurations of rotation matrices and downstream model structures. (a) Average downstream accuracy throughout training. The shaded area indicates max and min values. (b) The violin plot of metric scores of original/rotated true factor values. Scores with deterministic values collapse to a line. The first 5 metrics measure disentanglement while LR measures informativeness.
4.2 ARE DISENTANGLED REPRESENTATIONS NECESSARY?
Hereafter we challenge the necessity of disentanglement. We begin by comparing disentangled representation v.8. deliberately designed, entangled representation the downstream perfor- mance. Then we discuss the necessity of disentanglement inductive bias by evaluating the perfor- mance of disentanglement and general-purpose representation learning methods.
Effeets of attenuating disentanglement. We first construct the most disentangled representations, i.., the normalized true factor values, We normalize the true factor values to have zero means and unit standard deviations, yielding 6-d representations (note that Abstract dSprites and 3DShapes both labeled with ground truth factors). Then rotate the constructed representations by multiplying randomly generated orthonormal matrices. Afterward, each dimension of the rotated feature captures combination of factors, thus destroying disentanglement. Finally, perform abstract reasoning training from true factors before and after rotations. We also conduct rotations on representations learned by VAES.
We run 5 seeds defining the randomly gener- ated rotation matrices and Stage-2 model con- figurations. We report results on 3DShapes with original/rotated true factors represen- tations and WReNs Stage-2 models in Fig- ure 2 depicted in Figure 2a there is lit-
0.8
Accuracy
0.6
0.4
tle difference between performance before and after rotation throughout the training process. Yet Figure 2b shows significant drops in disen- tanglement metric scores. This surprising phe- nomenon suggests that even though dras- tically entangle the representations, the down- stream performance remains unchanged, firmly against the necessity of disentanglement, How- ever, we can see from Figure 2b that LR scores are 100% before and after rotation. It is easy understand because the rotation we applied is just an invertible linear transform, which simple LR can recover, not to mention more capa- ble Stage-2 models. Moreover, we observe similar results for learned representations (Figure 3). We select the most disentangled DisVAE measured by FactorVAE score among the 180 VAE models trained on 3DShapes (recall Section 4.1). I). As shown in Figure 3 rotation does not hurt the performance of representations learned by DiEV backing up claim that disentanglement representations might not be necessary to achieve good downstream performance. More results of
Representazions
Rotated represeniations
0.2
0
2000
4000
6000
8000
10000
Steps
Figure 3: Average downstream accuracy Dis- VAES' representations w/ and w/o rotation on 3DShapes with WReN Stage-2 models. We select the most disentangled DisVAE in terms of FactorVAE score. The two curves are almost iden- tical.
rotation experiments other datasets are reported in Appendix C.3
Summary: Destroying disentanglement (by random rotation) representations does not have a noticeable impact downstream performance throughout training.
Advantages of disentanglement inductive bias. From previous results, we demonstrate that both high performance and high sample efficiency be achieved even if we deliberately destroy disen-

---

Under review as a conference paper at ICLR 2023
0.5
Accuecy
Asouncy
0.6
04
BYOL
BYOL
DisVAEs
DisVAEs
0.2
0.2
2000
4003
(UTO)
5000
1000]
0
2000
4002
(UTN)
5000
10000
Staps
Stops
(a) Stage-2=WReN
(b) Srage-2=Transformer
Figure 4: Average test accuracy on 3DShapes throughout the training. We select the Stage-I models with best WReN Trans. among 3600 checkpoints 3DShapes. Stage-1 I models with disentan- glement inductive bias (DisVAEs are not necessarily better than those without such bias (B YOL) in terms of sample efficiency and final accuracy.
Table Best downstream test performance (%) of different stagel models. The step with the highest validation accuracy is reported. The numbers in the parentheses are STDs.
Dataset
Stagel
WReN*
WReN
Trans."
Trans.
DisVAEs
89.9
84.8(0.91)
96.4
87. 87.0(6.36)
3Dhapes
BYOL
93.9
87.1(4.68)
95.0
88.0(2.62)
Abstract
Dis VASS
76.5
68.7(1.39)
72.5
66.4(7.00) (76.06)))
dSprites
BYOL
78.3
72.2(3.11)
81.4
78.1(1.75)
tanglement. Further, are interested in the inductive biases of Stage-1 I models: Do disentangled representation learning models have absolute advantages on downstream performance over general- purpose models? For this, we compare the downstream performance of different families of learning models described in Section 4.] including BYOL, 3-VAE, AnnealedVAE, 3-TCVAE, Factor FAC, DIP-VAE-I, and DIP- CEP--AE-IL Among them, BYOL does not explicitly encourage disentanglement. On the other hand, all DisVAEs are disentangled representation learning methods. From a large pool of 7200 checkpoints, we report the best performance for each model family.
Figure shows overviews of training trajectories of Stage-I models with the highest performing WReN and Trans. on 3DShpaes for multiple training steps. For WReN as Stage-2 models (Fig- ure 4a BYOL leads at the beginning, then DisVAEs catch up, and finally, BYOL converges at higher accuracy. contrast, when Stage-2 models are Transformers, BYOL's curve grows faster, but DisVAEs and BYOL converge with comparable performance. In general, the two curves evolve in almost identical patterns with small gaps, indicating that disentanglement inductive bias is of lim- ited utility improving downstream sample efficiency. Corresponding analysis on Abstract dSprites is available in Appendix C.3] where we reach the same conclusions. As for final performance, we report maximal WReN, WReN", Trans. and Trans." across different Stage-2 models and datasets in Table [ We select checkpoints to evaluate based on validation accuracy. In particular, the best higher than that of VAES. In addition, it appears that BYOL
WReN and Trans. of B YOL performs better than or on par with VAES in terms of WReN* and Trans.". Especially, BYOL outperforms Dis VAEs on Abstract dSprites with a considerable margin.
Summary:
Models not intended for disentangled representation learning can reach superior or com- parable downstream performance. Therefore disentanglement inductive bias does not necessarily lead to better sample efficiency or final accuracy.
4.3 WHICH PROPERTY MATTERS DOWNSTREAM PERFORMANCE?
The results in Section provide encouraging cases against the necessity of disentanglement.. Additionally, we are interested in several further issues: (1) Which property matters downstream performance most? (2) How can we interpret the previously claimed benefits from disentangle-
ment(Bengio et al. 2013 Higgins et al. 2016 van Steenkiste et al. 2019 Locatello et al. 2019a
Dittadi al. 2020 On account of these questions, start by investigating how different repre- sentation properties influence downstream accuracy. We include informativeness and various disen- tanglement metrics.
7

---

Under review as a conference paper at ICLR 2023
BesaV Scoee
39.00
S8.
50.8
49.3
BetaVAE
-5.7
-8.5
-20.4
90
PucteVVE AE Score
55.5
65
51.7
57.3
$7.0
-00
Pacted-All Score
-51.7
-11.4
-37.6
-44.5
-39.5
25
Bisnsanllmani
12.5
-3.0
-186
-2.33
-45.4
-40.
MIG
16.8
17.5
4.4
-04
0.2
SAP
25.8
26,3
64.5
38.6
-25
EX
$6.6
66.3
91.3
85.9
84.0
1R
72.3
71.4
70.8
60.7
59.4
-50
1000
2000
5000 step
10000
Best
1000
2000
5000
10000
Doxt
skep
(a) Dis WREN
-90
57.2
$7.1
13.9
-2.1
Facto VAE Seare
43.5
36.9
56.3
50.1
60.3
-62
PacterVAlS Scor
22.6
6.
22
-245
-24.0
40
DCI Disarangooeee
27.4
20
MIG
104
34
9.5
31.9
32.5
MIG
-1.3
-10.3
14.1
15.7
16.4
-0
22.4
21.5
24.7
30.2
37.0
37.6
LR
75.1
65.5
74.5
50.1
80.8
LK
53.3
-0
stop
(c) DisVAE-Transforme
(d) BoomTransoooeee
Figure 5: Rank correlations between WReN Trans. and representation metries on 3DShapes. We denote the step with the highest validation accuracy as "Best". The brighter the panel, the more correlated the representation metric is with the downstream performance.
VetaVAE Score
AAE Score
MIG
SAP
LR
Store value
CAt
(4)
Avg accuracy
Avg accurncy
Avg accuracy
Avg occurney
Aug accuracy
Figure 6: Representation metrics versus WReN at step 10000, where Stage- models are BYOL, and the dataset is 3DShapes. We can observe strong positive correlation between the informativeness metric scores and downstream accuracy.
Recall that we train 720 Stage- and 7200 Stage-2 2 models (see Section 4.1b. By taking WReN and Trans. as measurements (average reasoning accuracy over 5 WReNs or 5 Transformers), we yield 720 representations paired with their downstream performance. Generally, our analysis is based on rank correlation (Spearman) between representation metric scores and downstream performance. If the correlation score is high, we can conclude that the representation property measured by the considered metric score is significant to downstream performance.
The representation property of the most significance. We calculate the rank correlation between downstream accuracy and disentanglement and informativeness scores. Meanwhile, we report rank correlation steps 1K, 2K, 5K, and 10K, and the step with the highest validation accuracy. From correlations at different training steps, we can tell how a representation property affects sample efficiency.
Figure 5) displays rank correlations between representation metric scores and abstract reasoning test accuracy 3DShapes. Firstly we find that Logistic Regression accuracy (LR) correlates most with downstream performance. The strong correlation is exploited for all considered models at mul- tiple steps. Since LR requires sufficient information be captured and extracted easily from repre- sentations, we can conclude that the informativeness matters most in broad conditions. In contrast, we observe that the importance of disentangiement varies among Stage-1 1 model families. Disen- tangled representation learning models (AEs) exhibit strong positive correlations for several disentanglement metries (but weaker than LR), such as FactorVAE score and DCI Disentanglement However, their significance does not apply to BYOL, where the correlation of disentanglement is mild or even negative. In Figure we plot the (WReN, metric score) pairs at step 10000. Indeed, for BYOL-WReN on 3DShapes, we can see the linear regression provides a good fit of downstream accuracy and informativeness metrics. As for disentanglement metrics, we can see that Beta VAE score and Factor VAE score suffer from narrow spreads. For MIG and SAP, the regression lines have negative slopes. We conduct a similar analysis on Abstract dSprites and take the same observations. Please refer to Appendix for more details.
8

---

truly Under review as a conference paper at ICLR 2023
NelaVll-Sooo (D1)
1966
514
6/4
611
Add Beltae
9.8
PactoY Score (D2)
X7
16.9
04.
911
50
Ady Paccurk All Some
26.1
18:5
9.9
9.8
10.4
Deeentanglmmerrroo (D5)
60
DCT Divettangleeees
18.7
8.5
-12
+7.5
-9.3
10
MIG
40
MLLMG
8.1
SAP (D5)
979
1229
11
AALSAP
21,6
5.0
-5.8
-3.0
8.4
(1)
511
645
244
57
356
1940
1000
2000
5000
10000
Bat
D5
(a) Overall rank corrolations.
(b) Correlations of adjusted metrics.
Figure 7: (a) Correlations between metrics and (b) correlations between adjusted metrics and down- stream accuracy when using LEVVE---Reee pipeline on 3DShapes. Disentanglement metries ex- hibit positive correlations with LR. Yet when conditioned on close informativeness, their adjusted versions show mild correlations.
Summary: The informativeness influences downstream performance most. The results are consis- tent across datasets and model structures.
Understanding for the previously claimed success of disentanglement. Previous works (van
Steenkiste et al., 2019 Locatello al. [019a] Dittadi et al. 2020 Locatello al. 2020) have re- ported empirical evidence backing up the advantages of disentangled representations. Consistently, we observe relatively strong correlations with disentanglement metrics, especially when Stage-1 models are VAES in Figure Based on our conclusion the significance of the informativeness. we study the Dis DisVAE-MY WReN case and provide some insights to explain why the disentanglement metrics have high correlation to downstream performance in some cases.
We compute the overall correlations between metrics. The results are shown in Figure 7 For Dis- VAEs, we find that informativeness and disentanglement have high correlation scores. In particular, we can observe relatively strong correlations between LR and Factor VAE score and BetaVAE score. Accordingly, these disentanglement metrics exhibit relatively strong correlations with downstream performance in Figure 5a In contrast, other disentanglement metrics correlate mildly with LR. And they are ineffective for downstream performance. Therefore, disentanglement metrics
predictable for downstream performance, but LR is.
To "purify" the effect of disentanglement, natural question is: informativeness, does the more disentangled one more helpful for downstream tasks? For this, we employ adjusted metrics in Locatello et (2019a):
If representations are of close
Adj. Met Metric-= = Metric
Σ
Metrich
(2)
KEN(LR)
For a representation and certain metric (we care more about disentanglement metrics), we de- note its original metric score as Metric. Then find its 5 nearest neighbors in terms of LR, which we write as .N (LR). Finally, the difference between the original metric score and the mean score of the nearest neighbors reported adjusted metrics. Intuitively, we calculate the relative disentanglement for representations with close L.R.
Figure displaas correlations between adjusted metrics and downstream performance. We find that all adjusted disentanglement metrics correlate mildly with downstream performance. From this, can that when informativeness close, being disentangled contributes only small portion to the downstream performance when the downstream training steps are limited (In our case, less than or equal to 2000 steps, see Figure S and Figure 7
Summary: The informativeness is the most predictable metric for downstream performance. Dis- entanglement only brings small extra benefits at the very beginning of downstream training.
CONCLUSION
In this paper, we challenge the necessity of dimension-wist disentanglement for downstream tasks. We conduct a large-scale empirical study the abstract visual reasoning task. We start by showing that high downstream perforrmane performance can be achieved by less disentangled representations. In addition, identify that the informativeness is of the most significance. Finally, conclude that dimension- wise disentanglement is unnecessary for downstream tasks using deep neural networks with learned representations as input.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide information to reproduce our results in Appendix A We commit to making our codes publicly available.
REFERENCES
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning neural networks. In International conference on machine learning, pp. 511-520. PMLR, 2018.
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788)887, 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://ghhhlbcommmmeemmm////h/as 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv: 1804.9399, 2018.
Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, and Yang Gao. An empirical study on disentan- glement of negative-free free contrastive learning. arXiv preprint arXivv2206.0413 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theo- retical account of the processing the raven progressive matrices test. Psychological review, 97 (3):404 1990.
Hugo Caselles-Dupte Michael Garcia Ortiz, and David Filliat. Symmetry-basee dis- entangled representation learning requires interaction with environments. In H. Wal- lach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates. Inc., 2019. URL ltttps:/ wwwp:iiiiiinnnnnninneeeee cc/paperr2019/Eile) 36e729ec1
Paper. pdf
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp- 2180-2188, 2016.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287314, 1994.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther. Stefan Bauer, and Bernhard Schölkopf. On the transfer disentangled representations in realistic settings. In International Conference Learning Representations, 2020.
10

---

Under review as a conference paper at ICLR 2023
Kien Do and Truyen Tran. Theory and evaluation metries for learning disentangled representations. In Interaational Conference on Learning Representations, 2019.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation disen- tangled representations. In ICML, 2018.
François Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans on visual categorization test. Proceedings of the National Academy of Sciences, 108(43):17621-1
2011.
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6783-6792, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Lukas Hahne, Timo Lüddecke, Florentin Wörgötter, and David Kappel. Attention abstract visual reasoning. arXiv preprint 191iv: 1.05990, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. arXiv preprint, 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerehner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02233., 2018a.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bošnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. International Conference on Learning Represen- tations, 2018b.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29:3765-3775 2016.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Insu Jeon, Wonkwang Lee, and Gunhee Kim. Ib-gan: Disentangled representation learning with information bottleneck gan. arXiv preprint, 2018.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference on Artificial Intel- ligence and Staristics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Pio Monti, Diederik Kingma, and Aapo Hyvärinen. Ice-beem: Identifiable conditional based deep models based on nonlinear ica. arXiv preprint arXiv:2002.11533 15537, 2020b.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658 PMLR, 2018.

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational
bayes.
arXiv preprint
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint 1rXxiv2007.10930 2020.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv: 000848, 2017.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Zinan Lin, Kiran Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning, pp. 6127-6139. PMLR, 2020.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairesss of disentangled representations. Advances in Neural Information Processing Systems, 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions the unsupervised learn- ing of disentangled representations. In ICML, 2019b.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-superviee disentanglement without compromises. International Confer- on Machine Learning, pp.
PMLR, 2020.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 2017.
testing sprites dataset. http://iww...oomddeeeeiiid/sssss
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
John Raven. Standardization of progressive matrices, 1938. British Journal of Medical Psychol- ogy, 1941.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv:1802.05312, 2018.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6):8633879, 1992.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965):61--634, 2021.
Richard E Snow, Patrick C Kyllonen, Brachia Marshalek, et al. The topography of ability and learning correlations. Advances the psychology of human intelligence, 2(S 47):103, 1984.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generaliza- tion for abstract reasoning tasks using disentangled feature representations. arXiv preprint 1rXiv:1811.04784 2018
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning, arXiv preprint arXiv: 2018.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? arXiv: /2506, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
12

---

Under review as a conference paper at ICLR 2023
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of re- search and development, 4(1):66- 82, 1960.
Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disen- tanglement: A review of metrics. arXiv preprint arXiv:2012.09276, 2020.
A REPRODUCIBILITY
In this Section, we provide implementation details to ensure reproducibility. In addition, we commit making our codes, configurations, and running logs publicly available. All experiments are run on a machine with 2 Intel Xeon Gold 5218R 20-core processors and 4 Nvidia GeForce RTX 3090 GPUs.
A.1 REPRESENTATION LEARNING METHODS
We include both disentangled representation learning methods and general-purpose representation learning methods. i.e., VAES and BYOL (Grill et al., 2020
DisS implementation. The DisVAEs include 3- VAE (Higgins et al., 2016)) Annealed-
VAE (Burgess et al. 2018). 3-TCVAE (Chen et al., 2018). FactorVAE (Kim & Mnih 2018). and DIP-VAEL and DIP- VAE-II (Kumar et al.) 2017). We use the output of the encoder, the mean of (=(2))) representations. Hereafter, we introduce details for each method. The above methods encourage disentanglement by adding regularizers to ELBO. Adopting the notation in Tschannen following unified form:
al. 2018), their objectives can be written
E pp(r) E (=(aa) log (=(x))]]] A1 (R (Rg((())))) + AeRo(go(2)),
where (2|xx) is the posterior parameterized by the output of the encoder, p=(x]2) is induced by decoder output, R2 R2 are the regularizer applying to the posterior and aggregate posterior, and '1, A2 the coefficients controlling regularization. In the objective of 3-VAE, 3 À > 1, A2 = Taking R1 (90 (2[2)) == (x=])<(2|=>]1 [[))] forces the posterior to be close to the prior (usually unit gaussian), hence penalizing the capacity of the information bottleneck and encourage disentanglement. Factor and 3-TCVAE takes \ = 0, A2 1. With R2(4a(2)) == (C)(g(=)) they penalize the Total Correlation (TC) (Watanabe) 1960). FactorVAE estimates TC by adversarial training, while 3-TCVAE estimates TC by biased Monte Carlo sampling, Finally, DIP-VAE-I and VAE-I take A O, A2 1 and R2)4s(2)) = |ov I 2
penalizing the distance
between aggregated posterior and factorized prior.
We use the code and configurations from the DisLib Locatello al. 2019b). As for parameters, we use the same sweep as van Steenkiste et al. 2019): for each one of the 6 DisVAEs, we use configurations. We train each model using different random seeds. Since we consider datasets (3DShapes and Abstract dSprites), finally, we yield 6 6 + 360 DisVAE checkpoints.
X;
representation
projection
prediction
online
ema
ema
loss:
(r
||ull
X2
representation
projection
target
Figure 8: The pipeline of BYOL (Grill et al. 2020
BYOL implementation. BYOL Grill et al. 2020 is a contrastive learning method. Figure shows its pipeline. For each image
first create two "views" of it by data augmentation, i.c., Ji and Ø2. Then they are input to the siamese encoders: the online encoder and the target encoder. Specifically, is fed to the online encoder, while 2a is fed to the target encoder, yielding the output
https:: thub. com gow. resea / sentangl 1iaeent___bggg.

---

Under review as a conference paper at ICLR 2023
Table The representation network architecture of our BYOL implementation, following Cao al. (022). Besides, there is ReLU activation layer and possible normalization layer following each convolutional layer to create stack of (Conv-ReLU-Nowe norm and representation dimension D are parameters to be set.
blocks. The normalization stratege
Representation Network
input: 64 x 64 images pipeline:
4x4 conv, stride 2, 32-channel
4x4 conv, stride 2, 32-channel
4x4 conv, stride 64-channel
4x4 conv, stride 2, 64-channel
4x4 conv, stride 128-channel
1x1 conv, stride 1, D-channel
21 and 22, respectively. As for architectures, both encoders share the same representation network and projection MLP. The prediction MLP is appended to the online encoder in order to avoid BYOL learning trivial representations. The objective of BYOL
{21, 22)
L
(4)
11 |2112||22112
We are pulling the representations of the two "views" close. While training, the online encoder's parameters are updated by gradient descent. However, the target encoder's parameters are updated by the online parameters' Exponential Moving Average (EMA) (He et al., (020). After training, we only keep the online encoder and use the output of the representation network as representations.
We use the PyTorch implementation of BYOL We use the representation network architecture as shown in Table 2 where the representation dimension is a parameter to be set. Except for normalization and output dimensions, the representation network architecture of BYOL and the encoder architecture DisVAEs are similar. As for predictor and projector, we use the pipeline Linear- BN ReLU Linear with 256 hidden neurons. We train the BYOLs for 105 epochs using the Adam optimizer with 31 = 0.9, B2 = 0.999, = 10 and learning rate (Ir) as a variable parameter. For augmentation, use the pipeline of Cao al. 2022 (in PyTorch-ttyle):
1. SamddomAppyytrrmmffooosCooo///rt Tja Bjs 0.2), p=0.8)
2. RandomGGrayScall(/==/eoo))
3. RundomttorizomaFFinn))
4. htw.ppppwworooooooosssssssaaaaae (1.0, 2.0)), p=0.2)
5. RandommRestaeCroplliieee 64), scale=(Xeemp 1.0))
The Zja- Pgna. and Terop are parameters to be set. Zja controls how much to jitter brightness, contrast, and saturation. Pgay controls the probability to convert the image grayscale. Terop defines the lower bound for the random area of the crop.
We perform parameter sweep on the cross product of intervals of parameters D, norm, Tja Paray and Temp- On 3DShapes, we use the following parameter grid (in scikit-learn style):
('D: [32, 64, 128], 'Ir': 1,e-2, 3e-3], 'norm': (BatchNorm() ()] 'x (0.6, 0.81, '9_gray': 10.5, 0.7, 0.9), 'x_crop': [1.0]), ('D': [256], 'Ir'; 13e-4, 3e-5], 'norm': [BatchNorm(), GroupNorm (atpuuuggrossa)) 1, 'xiitts 10.4, 0.81,
'p_gray": [0.3, 0.5, 0.71, 'x_crpp': [1.0]}
]
On Abstract dSprites, we use the following parameter grid:
14

---

Under review as a conference paper at ICLR 2023
('D': [32, 64, 1281, '1r': [3e-3, 3e-4], 'norm": [BatchNorm ()., "xjit'; [0.6, 0,8], 'p_gray'; I0.0, 0.1, 0.21, 'x_crop': [0.6]), ('D': [256], '1r'; 13e-4, 3c-5], 'norm': [BatchNorm(), (), GroupNorm (num (nrrouooo/mm (x_jith: 10.4, 0.81, 'p_gray": [0.0, 0.1, 0.2], 'x_crep': [0.6])
For each parameter configuration, we run it with 3 random seeds. Finally, we trained 360 BOOLs in total.
A.2 ABSTRACT REASONING METHODS
We include two abstract reasoning network architectures: WReN (Barrett al. 2018 van Steenkiste et al. 2019) and Transformer (Vaswani et al. 2017 Hahne et al. 2019).
WReN implementation. WReN consists of two parts: graph MLP and edge MLP. Here we the same notations as in Section 3.3 For the representations of a trial Stagel (T,), edge MLP takes a pair of representations in IT.) input and embed them to edge embeddings. Then all edge embeddings of Stagel (T:) (in total Cg=36) are added up and input to the graph MLP. Finally, the graph MLP output a scalar score, predicting the correctness of the trial T..
We use the code (van Steenkiste et al. 2019) to implement WReN, And we use the same parameter searching spaces as them. All WReNs are trained in 10K steps with a batch size of 32. The learning rate for the Adam optimizer is sampled from the set f0.01. 0.001, 0.0001} while 31 = 0.9, B2 = 0.999, and 10 8 For the edge MLP in the WReN model, we uniformly sample its hidden units in 256 or 512, and we uniformly choose its number of hidden layers in 2, 3. or 4. Similarly, for the graph MLP the WReN model, we uniformly sample its hidden units 128 or 512, and we uniformly choose its number of hidden layers in 1 or 2 before the final linear layer to predict the final score. We also uniformly sample whether we apply no dropout, dropout of 0.25, dropout of 0.5, dropout of 0.75 units before this last layer.
Transformer implementation. We simplify the architecture of Hahne et al. (2019). Here we treat Stagel(T.) sequence. We first linear project all representations and prepend them with learn- able [class] token. We add them with learnable positional embeddings. Then they are input into stack of Transformer blocks (Vaswani et al. 2017). Finally, an MLP predicts scalar score from the class embedding of the final Transformer block.
We implement the Transformer architecture ourselves with utilities of the DisLib code base. All Transformers are trained for the same steps and same batch size as WReN, i..., 10K steps with batch size of 32. We use the Adam optimizer with weight decay and cosine learning rate scheduler. The learning rate for the Adam optimizer is uniformly selected from {5e - 4, Ge 4, 7e - 4]. The depth of Transformer blocks is uniformly set to be 2, 3, or 4. The dimensions g, k, of the self- attention model are uniformly 32 or 64. The MLP head uses the same architecture and parameter space as the graph MLP WReN. For other fixed parameters, please refer to our codes for details.
A.3 REPRESENTATION METRICS
In the main text, we employ disentanglement and informativeness metrics to measure the properties of representations. Here we provide more details.
Disentanglement metrics. We use the setup and implementation of Locatello et al. (2019b). Here we briefly introduce the details of our considered metrics. Namely, Beta VAE score (Higgins al.)
2016). VAE score (Kim & Mnih 2018), Mutual Information Gap (Chen et 20.1) SAP (Kumar et al. 2017). and DCI Disentanglement (Eastwood & Williams, 2018). The BetaVAE score and the Factor VAE score predict the intervened factor from representations to measure disentangle- ment. The Mutual Information Gap and SAP compute the gap in response for each factor between the two highest representation dimensions. The difference is that MIG measures mutual informa- tion while SAP measures classification accuracy. The DCI Disentanglement calculates the entropy of the relative importance of latent dimension in predicting factors. We follow previous studies (Locatello et al. (219bb] yan Steenkiste et al., 2019 Locatello et al., 2019a Dittadi et al. 2020) to develop a Gradient Boosting Tree (GBT) for prediction during the DCI Disentanglement evaluation.
15

---

Under review as a conference paper at ICLR 2023
Though according Eastwood & Williams 2018 any classifier could be used. As reported by Cao et al. (022), , the GBT takes hours to train from high- high-dimensional representations learned by BYOL. Thus only report DCI Disentanglement score for Dis VAES.
LR (mc)
LR (oxz)
-80
56.6
86,9
91.3
85.9
84.0
LR
Coun
60
30
20
LR
25.9
57.3
67.0
63.3
61.0
40
1000
2000
5000
10000
Best
step
(b)
Figure 9: (a) Prediction accuracy by "multinomial" LR, denoted LR (mn), and "one rest" LR, denoted as LR (ovr) of VisS on 3DShapes. (b) LR (mn) and LR (ovr)'s correlations with the downstream performance of the DisVAEs- WReN pipeline.
Informativeness metrics. We use LR to measure the informativeness representations. We train Logistic Regression model to predict factor values from representations. We use 10000 samples to train LR. Unlike van Steenkiste et al. (2019), we use "multinomial" instead of "one v.s. rest" as the multi-class classification scheme. As shown in Figure Pal for the same set of representations, "one v.s. Rest" LR has inferior prediction accuracy. Moreover, ranking by scores these two LRs yields different results. In Figure 9b we can observe different correlations of the "one Rest" LR. To better estimate informativeness, we use "multinomial" LR as the measurement.
A.4 ABSTRACT VISUAL REASONING DATASETS
We use the abstract visual reasoning datasets developed by van Steenkiste et al. (2019) i.e., Ravens' Progressive Matrices created from 3DShapes (Burgess & Kim 2018) and Abstract dSprites
Matthey et al. 2017 van Steenkiste al. 2019).
We sketch the rules here by taking the RPM in Figure T as an example. The reasoning attributes the ground truth factors of 3DShpaes. i.c., floor hue, wall hue, object hue. scale, shape, and orientation. Each row in the 3 x 3 matrix has 1, 2, or 3 ground truth factors taking fixed value. And the rows have the same fixed ground truth factors, though they might take different values. From the context panels, one should discover the underlying logical relationship. Finally, one is asked to the missing panel by one of the candidates. For the RPM in Figure [[ can infer that the fixed factors are: wall hue, shape, and orientation. Then for the third row, from the first 2 panels, we know that the values for the shared factors are: the wall hue is blue, shape cylinder, and the orientation is the azimuth that makes the wall corner appears in the righter part of the image. So we choose the candidate with these factor values as the solution, as shown in Figure T0b shows sample of RPMs with answers Abstract dSprites.
from the contexts,
Figure T0al
B ABLATIONS ON GENERAL--PURPSE PURPOSE REPRESENTATION LEARNING METHODS
In the main text, use BYOL representative of general-purposs purpose representation learning meth- ods. For completeness, here we introduce another general-
sencral-purpose purpose method, SimSiam (Chen He 2021 We modify the code of B BYOLFoo train SimSiams 3DShapes with the following parameter grid:
D'D': [512], 'Ir': 13e-4, 3c-5], nnomm [BatchNorm 'x (0.4, 0.8),
16

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Figure 10: RPM questions with solutions on (a) 3DShapes and (b) Abstract dSprites.
BetaVAE Score
84.4
82.7
64.2
48.3
51.6
75
FactorVAE Score
-52.0
-52.0
-52.0
-52.0
-51.8
MIG
52.3
44.6
31.6
31.0
36.8
25
-0
SAP
16.3
15.9
21.55
22.7
20.3
25
90.3
54.8
1000
2000
5000
10000
Best
step
Figure 11: Correlations of SimSiam- on 3DShapes.
0.8
1.0
9.5
0.3
2003
8000
10000
IKSD
NIN
SA.
Matri
(b) Figure 12: (a) Average downstream accuracy throughout training from ground truth factors with and without rotation. (b) The violin plot of metric scores of original/rotated true factor values.
(a)
'p-gray's [0.3, 0.5, 0.71, 'x_crop': [0.6, 6, 1.01}
]
For each configuration, same WReNs for DisVAEs and BYOLs as Stage-2 models.
with 3 seeds. So finally, yield 72 SimSiams. Then the
The results of -RRN agree with conclusions in the main text. As for the best per- formance, we have WReN=85. % and =94 %, which is better than DisVAEs'. Figure I1 shows the correlations of downstream performance and representation properties. LR still correlates most for all considered steps.
C ADDITIONAL RESULTS
17

---

Under review as a conference paper at ICLR 2023
Table 3: The model type and the step achieving the performance in Table of model @ step. For WReN and Trans., we report the mean steps and STDs.
We report in the form
Dataset
Stage1
WReN"
WReN
Trans."
Trans.
AnnealedVAE
3-VAE
DIP-VAE-I
3-TCVAE
3DShapes
DisVAEs
@9600
(8400(712)
@9900
@9100(901)
BYOL
BYOL
BYOL
BYOL
3DShapes
BYOL
@ 10000
@8900(849)
@8600
@8860(933)
3-VAE
DIP-VAE-I
DIP-VAE-I
DIP-VAE-I
Abstract dSprites
DisVAEs
@9900
@8920(898)
@9400
(99300(12))
BYOL
BYOL
BYOL
BYOL
Abstract dSprites
BYOL
@8100
@8320(1 195)
@8800
@8100(725)
0.9
0.8
Accuracy
0.6
0.5
0.3
Represantation
Represennzions
0.2
0
2000
4000
6000
R000
10000
Sample Size (#Batches)
Figure 13: Accuracy v.s. #samples curves of the most disentangled VAES before and after rota- tion. It consistent with Figure
1.U
0.8
0.8
0.6
0.6
0.4
0.4
BYCL DaVAbb
Brot
DaVAb
0.2
0.2
4000
10000
Sumple (1Butches)
Sumple (4Baishes)
(a) Stage-2=WReN
(b) 2
Figure 14: Accuracy v.s. #samples curves of the Stage-I models with the best WReN or Trans.. It
consistent with Figure
C.I ADDITIONAL RESULTS OF FINAL PERFORMANCE
In Table D we report the best final performance of DisS and BYOLs. Here we provide more details on which type of VAES at which steps achieve the reported performance in Table [ We can observe that the best Dis VAEs vary with different datasets and Stage-2 models. As for the best steps, except 3DShhheSShhaaSE... WRRN, BYOL achieves the best performance earlier than VAES.
C.2 ACCURACY-HSAMPLL CURVES
We employ training curves (accuracy-step) in the main text to evaluate sample efficiency following van Steenkiste et al. 2019). For completeness, here we show accuracy-tsamples curves.
We present the accuracy accuracy+#samplec #samples versions of Figure 3 and Figure 4 i.c., Figure B3 and Figure 14
We train the same models as in the main text until convergence with fixed training data sizes of 100, 1000, 5000, 7000, and 10000 batches. Then for each sample size, plot the performance the
18

---

Under review as a conference paper at ICLR 2023
Table Mean metric scores with STDs of different Stage-I models.
Dataset
Stagel
BetaVAE
FactorVAE
MIG
SAP
LR
3DShapes
DisVAEs
93.7(7.7)
82.3(11.1)
25.5(15.1)
6.5(3.8)
78 78.0(9.6)
3DShapes
BYOL
99.9(0.3)
96.1 (4.6)
8.1(5.3)
1.2(0.9)
96.6(1.8)
Abstract dSprites
DisSAEs
62.3(14.1)
49.1(10.5)
13.3(7.0)
6.8(3.4)
36.8(4.4)
Abstract dSprites
BYOL.
63.6(17.0)
62.4(11.8)
2.6(1.8)
0.5(0.3)
43.0(8.2)
step with the highest validation accuracy. We can see the ranking of representations and evolving patterns of both types of curves agree well.
C.3 ADDITIONAL RESULTS OF RANDOM ROTATION EXPERIMENTS
This section contains additional results of the random rotation experiments. Here we report the downstream performance of deliberately entangled (by random rotation) representations.
0.6
Accuracy
0.5
0.4
0.3
Reprasentations
Rotated represeniations
0.2
0
2000
4000
6000
8000
10000
Steps
Figure 15: Test accuracy curves through training for representations before and after rotation Abstract dSprites. Like for 3DShapes, we select the DisVAEs with the highest FactorVAE score. The two curves are almost identical.
Figure 12 shows the same experiments as Figure 2 on Abstract dSprites. We can observe that the two curves in Figure 12a are almost identical. And Figure 12b) can observe that disentangle- ment metric scores drop drastically while LR remains the same. We notice that LR is not 100%. This is because some factors of Abstract dSprites have too many support values. e.g., the and positions both have 32 possible values. However, our conclusion in the main text still holds as we observe that LR is invariant random rotation. On Abstract dSprites, disentangled representations from Dis VAES (measured by FactorVAE score). In Figure [5] we can see that rotation has little impact on the training trajectories. So our conclusion is similar across datasets.
randomly rotate the most
C.4 ADDITIONAL RESULTS OF CORRELATIONS
In this part, we report additional results related to the correlations between representation metrics and downstream performance.
Absolute values of metric scores and downstream accuracy. We show the histograms sanity check of the distribution of metric scores and downstream accuracy. Figure T6 presents the score distributions of each metric. We report the mean metric scores with STDs to depict the overall properties for Stage-1 models Table Figure 17 and Figure display the distributions of downstream performance.
Rank correlations. This part contains additional results of rank correlations. On 3DShapes, Fig- 19 displays rank correlations between adjusted metries and downstream accuracy, Figure 20 shows the overall correlation between metrics. On Abstract dSprites, Figure 2T shows correlations between metrics and downstream performance. Then Figure 22 presents correlations between ad-
19

---

Under review as a conference paper at ICLR 2023
justed metrics and downstream performance. Finally, Figure 23 displays the overall correlations between metrics.
Plots of (metric score, downstream accuracy) pairs. Figures 24 provide an in-depth view of the correlations, where we plot (metrics, downstream accuracy pairs.
25 26
29
30
and
31
ReaVAR
FactorVAS
MIG
1.R
92
30
80
42
60
40
23
9.8
0.6
1.0
1.0
0.0
6.1
9.2
0.6
LO
(a) DisVAEs on 3DShapes
BeleVAE
FacturNA
MIG
175
175
175
175
175
320
100
100
Coure
100
23
0.95
0.59
1.0
0.0
0.5
0.6
0.2
0.50
602
0.06
0.06
0.50
0.55
(b) BYOL on 3DShapes
BetaVAB
EactorVAE
DC3-D
MIG
SAP
ER
33
30
30
30
30
CouaR
Couss
33
10
10
10
10
Dis Abstract dSprites
BeuVAE
FacteVVAE
MIG
SAP
1.R
32
20
32
32
30
Coast
0.4
0090
8.655
0.055
3.100
3.015
0.5
(d) BYOL Abstract dSprites
Figure 16: Histograms of metric scores of Dis VAEs and BYOL on 3dShapes and Abstract dSprites.
20

---

Under review as a conference paper at ICLR 2023
1000
2000
5000
10000
Best
20
30
Chazz
32
10
10
10
10
0.5
0.4
(a) Dis WReN
1000 sec
20000 and
5000.aec
10000sacz
Rest acd
42
40
40
30
20
20
(b) DisVAES- Transformer
1000 acc
2000 ace
5000.00 ax
10000 acc
Destaacc
40
40
Colce
22
20
20
mm
0.65
0.75
0.70
6.75
0.750
0.375
0.000
0.025
0.72
0.80
085
0.75
0.20
0.88
(c) BYOL-WReN
1000
5000
1000ttacz
30
30
30
22
20
25
0.50
0.35
0.58
0.35
0.50
0.65
0.65
0.70
0.75
0.70
0:75
8.50
0.35
0.75
6.00
0.55
(d) -Yon-Transformer
Figure 17: Histograms of downstream accuracy for multiple steps on 3DShapes.
21

---

Under review as a conference paper at ICLR 2023
1000,acc
2000
5000
10000
Best aestacc
Cuall
CMill
20
Dis WReN
1000 soc
2000 auC
5000.acc
10000sace
Bes ace
40
20
20
(b) DisVAEs-Transfomeee Transformer
1000. a0e
a0c0.ace
5000.acc
10000 aoc
Best aoc
Colsk
22
0.25
0.59
0.350
0.53
06000
0.523
0.60
0.00
0.05
0.60
(c) BYOL-WReN
1000ttacz
25
25
25
32
10
21
0.2
64
65
3.3
0.4
0.5
05
0.6
0.5
0.8
0.4
0.6
0.5
(d) LYOL-Transformer
Figure 18: Histograms of downstream accuracy for multiple steps Abstract dSprites.
22

---

Under review as a conference paper at ICLR 2023
A-j
23.3
25.1
29.3
Ag BeR Secre
2.6
9.2
13.8
9.5
9.5
-00
3.3
16.1
Add MMB.Fad VAE Scou
-32.8
-30.6
-17.2
-27.4
-25
-10
Adi Diserangloone
-3.7
MIMMG
25.4
166.8
-20
A4JMG
-27.7
-17.0
10.8
11.6
12.4
10
AGESA
-10.4
A4JSAP
-246
-16.1
8.6
K.1
1000
2000
5400
10000
Bent
1000
2000
5000 stop
10000
(a) BYOL-WReN
(b) DiaVAE-Transformer
20
Adg PlateeAAAE
9
-17
-17
AEMIG
33.2
22.0
5.2
9.7
9.5
110
11.4
1000
2000
5000 s0ap
10000
Best
(c) BOLBBYYLTTmmma Transformer
Figure 19: Rank correlation between WReN or Trans. and adjusted metric scores on 3DShapes.
BeiVAE Score (D1)
T7A
(D1)
Faste Seore (D2)
6+7
140
80
PacleVVAl Score (03)
4644
1894
16.
337
-85
50
75
Dianinglammii (D3)
S64
60
(D-)
25
MBG
40
(D5(
174
SAP (D5)
86
(R
S11
445
246
52
150
1800
(1))
11
(a) ViE
(b) BYOL
Figure 20: Overall correlation between metric scores 3DShapes.
BeiVAE
61.3
56.1
42.5
29.1
25.7
BasNAE
49.8
43.0
37.6
53.3
51.7
PacteVVAE
63.4
54.7
-0
Pacter/Al:
60.0
DCLDD
45.0
17.4
10.9
24.8
27.6
40
MIG
10.8
15.7
15.4
1.3
-1.3
40
SAP
21.8
12.9
12.1
11.8
12.2
SAP
10.8
7.9
11.2
4.2
0.5
-20
1.X
200
5733
67.7
64.1
1000
2000
5100
10000
Bew
1000
2040)
5005
10000
Tles
(a) Dis WReN
(b) BYOL-WReN
BesVAll
BeEVAE
56.1
62.1
68.4
67.8
66.9
PackeVVAll
61.5
65.3
Factu/VAE
35.5
50
DCLDD
24.6
57.2
47.7
52.8
$6.1
-0
25
19.5
52.1
SAP
12.
16.5
28.5
34.3
55.00
9
SAP
-51.7
-40K
-65
-44.
-45
25
77.1
13K
74.7
76.5
79.0
77.9
-00
1000
2500
5003
12000
Best
1000
2000
5005
15003
Best
(c)
(d) BYOL- -ranf-Trasformer
Figure 21: Rank correlation between WReN or Trans, and metric scores on Abstract dSprites.
23

---

Under review as a conference paper at ICLR 2023
Agg BenKAE
-1.9
-18.6
-30.9
-1..9
-12.
14.5
16.5
13.5
5.3
ACDDCDD
-5.0
-0..5
-14.3
-6.0
-6.8
-0
AUMMG
15.5
21.3
18.9
8.3
5.3
AdIMG
-10
A4JSAP
-20.9
-15.8
-8.3
AAJAA
4.9
4.9
6.2
4.4
-2.2
stop
step
(a) DiS WReN
SGYOL-WRei.
BAE
20
Adg DaaVAA
18.9
6.5
15.8
13.9
13.7
-19
Adj PastorVAE
4.7
22.6
-1.7
AgjDC
-15.0
-1.00
-.9
+0.8
0.2
-3.1
-5.0
AMLMNG
12.9
+..
A4J.SA
AgSAP
-22.5
-16.K
-13.7
-11.0
-11.2
(c) DisVAAETranssormee
(d) BOOL-Trannfoomme
Figure 22: Rank correlation between WReN dSprites.
Trans. and adjusted metric scores Abstract
BauEV Scar
122
A1.
Scare (DI)
75
VAE Score (D2)
612
300
26.
+2
283
15
VACUAVAL Soore (D2)
X*
-1.9
50
DCED (05)
814
8:5
MBG (D4)
25
SAP(
-1.5
LR
113
02
1350
(1))
400
-25
D1
02
13
D5
D1
122
D5
(a) Dis VAE
(b) BYOL
Figure 23: Overall correlation between metric on Abstract dSprites.
24

---

Under review as a conference paper at ICLR 2023
Scot:
Step
Score BaiVAE (Sep
BeiVAE (Sep
Scure
Step 10036
Seare
Best
1.8
1.0
0.5
FadueVAEl Siep
FactorVAEl Suap
Scoee FactorV AEl Step 5036
FactorVAEl Sxpp
FasterVAll SSop
Q.8
6.8
6.6
06
0.6
Stop
Scoue
DCSDD
Scare
DCLD Step
Seare
Step
DCS-D Step
1.8
1.0
0.4
64
06
0.4
0.2
Skep
Score MBG Step
Step 10000
0.8
6.8
us
0.8
8:50
8.5
(Sep
(Sep
Score (Stp
Step
10030
Step
0.0
Scatc Stop 1200
Score LR Stop
LRI
Score
LR Step
Secre
(ttp
10
6.5
08
0.8
0.4
Avx
Ang
Avg
Figure 24: DisVAE: metric scores v.s. WReN on 3DShapes.
25

---

Under review as a conference paper at ICLR 2023
Scvrr Bel/VAl! 1000
Scar BeaaV 2000
Score Thep VAIl 1000
Soam BAtaVAAl 10000
Some BeseV Bost
1.0
Scate
0.9
0.5
0.7
0,
0.2
0.8
FacturVAE
VisterVAE
FudteVAE Step
YaatsVVAEl
FactorVAll
Score viiz
9.8
0,6
0.
04
0.4
DXS5-1 (Stp
Stop
Step
1.00
0.4
Soure
0.50
0.4
04
010
Step 1000
Score -XIO) Stop 2200
Scare MIG 13000
Souc Boi
0.5
0,8
0.5
06
0.50
Scare
Secrcw SAP. Step 1000
Score SAI (Sup 2000
SAl Skep -
Sexec SAl! Stop 10000
Score Step Best
Scate
0.1
0,0
8.0
Scare L.R
Step
1000
Scure LR SSop 2000
Score L.R Step 5000
Score LR Slap
10000
Scoce
LR
Stop Best
0.8
0.
0.6
06
04
84
0
04
0.3
6.5
0.
Ang sccumcy
Aug serivry
Ang scorscy
Aug secimey
Avgaoceeayy
Figure 25: Dis VAES' metric scores v.s. Trans. on 3DShapes.
26

---

Under review as a conference paper at ICLR 2023
Soare Bes Step - 1000
Soate - Bww.aaaa.. ATI Stup - 2000
Scorcm VAll Step 5000
Scorc VAa VAC Stop 10000
Scvec Dera VAIl Skep Disd
1,000
1.000
1.000
0,995
0.955
0.995
Soor
8.500
0.992
0.903
0.900
0
0085
0.985
8.000
0.65
0.70
9.75
8.70
6:75
3.759
0.775
3,800
0665
0.75
0.80
9.85
6.75
0.80
0.35
Score - All Step 1000
Scort VatteVAll Step - 2000
Scor [baneVAEl Stop 5000
Seom - Ib.ueeVlll SSip (0000
Scor - VaateVVAl Stop - Bec
1.16
1.00
1:00
1.36
3.90
0.85
0.95
0.90
0.90
0.85
685
0.50
0.80
080
0.50
0.00
0.65
0.70
0.75
0.70
0.75
0.750
0,775
0.000
1025
0.75
0.40
0.40
6.75
0.50
035
Soure MXG (Sep 1003
Scote MIG Stap 2000
Score
MIG Skep 5000
Score
MIG (leg 10000
Score MIG Stop Bot
0.20
0.20
0.00
0.0
1.15
0.55
615
0.15
0.15
2.10
11000
6.10
0.10
0.16
0.05
0:85
005
0.66
085
0.00
11:02
600
0.00
020
0.65
8.70
0.75
0.70
0:75
0.700
0,775
0.00
1005
0.75
0.48
0.40
6.75
0.903
0.55
Scorpw SA2 1000
Score SAl SStp 2000
Somtz -SAP Sttp 5000
Somt SAP Stap 10000
Sarzz SAP Dood
0.06
0.85
605
3.04
60
084
3.10
0.82
602
0.00
0.50
6.00
0.65
0.70
0.75
0.70
6:75
0.759
0.755
0,000
6625
0.75
0.38
9.85
6.75
0.90
0.55
Score L.X Step 1000
Soure = L.X Step 2000
Sccrc L. (Stpp 5000
Score =Lt (Sup 10000
Scvee .IIX| Sttp
0.98
2.06
8.955
0.95
2.54
1:544
604
0.92
692
8.955
0.005
0.90
5.000
0.38
0.55
2.70
0.75
0.70
6:75
0.759
0,755
3,200
6625
0.75
0.30
9
6:55
0.90
0.55
Aug secumey
Aug 2002
Ang awuray
Avg socusny
Aug sccuracy
Figure 26: BYOLs' metric scores v.s. WReN on 3DShapes.
27

---

Under review as a conference paper at ICLR 2023
Sooce BeaaVEE Shep
1000
Score
BAEVAE (Sepp
2000
Scuce
BeaVVEE Ship
5000
Score
BAEVAE Step
10000
Score
Step
Bost
1.000
1,000
0.985
0.545
0,905
0.595
8.500
0.040
0,903
0.000
8.65
0.955
6565
0,985
0.985
8.900
0.980
0.500
0.35
0.70
0.75
0.36
Scrrc PaciorV/ll: Step 1000
Soare
VAIl Stop- 2000
Screc Val Step - 5000
Scem /.atteVV All] Stop (0000
Score - V'ateeVAlll Stop - lec
1.00
1.00
1.50
3.95
0.65
695
2.55
3.00
090
0.50
2.85
0.55
0.85
2.85
0.30
0.00
6.0
0.80
0.50
05
0.60
0.55
0.60
6.6
0.55
0.70
0.35
6.70
6:75
0.90
0.55
0.75
0.90
0.85
Soos MXG Step 1000
Score
MIG Skep 2000
Sous MXI Step 5000
Scoce MIG SSep
10000
Sccre
MIGI Stop Best
0.20
0.23
0.20
0.00
1.15
1.55
015
615
2.10
410
6.10
010
0.05
0.85
0.4
005
0.5
0.00
11:0
0.00
600
0.50
0:55
0.60
6:55
0.02
6.65
0.55
0.70
0:75
6.70
0.75
0.90
0.55
0.75
11.03
0.85
Score SAl Stop 1000
Somt SAP Stap 2000
Score SAL (Sup 5000
Score SAP Skpp 10000
Sortz SAP Dood
0.06
0.85
0.66
605
2.04
n.01
0.44
2.00
6.82
0.62
602
2.00
0.50
6.00
0.50
0.55
0.60
0.55
0.
6.65
0.55
0.70
0:75
6.70
6:75
6.30
0.35
0.75
0.90
0.85
L.R.
1000
LR
Scoue -.R| Sop 10000
Scou
(Sep
1.000
1.000
0.59
0.9
0.915
8.50
0.44
094
8.925
0925
0.92
692
0.903
0.88
088
0.55
0.60
0.62
6/65
0.55
0.70
0.75
0,9)
0.65
0.55
0.35
Aug RECATREY
Avg 3c0rar
Aug Recureey
Avg Rocurny
Aug stouncy
Figure 27: BYOLs metric scores v.s. Trans, on 3DShapes.
28

---

Under review as a conference paper at ICLR 2023
Seare BeaaVAE Step 1002
Scoce VAG Step 2003
Scoce BauVAE Step 5000
Scare BeaVAE Siep
10000
Scate
BAIVAE
Step
Bost
66
64
64
Scoce TacterVASS Step
Soom - FacteVAll SSp 2000
Searo PeceeVAAll Stop 5000
Soor ParseVAE Sup- 10000
Soore - FacteVAlll (Sp
66
66
Screow DDC3-1 Step- 1000
DEC34 Step 2000
DC3D Step 5000
DC1-0 Stop 10000
Scorc DCCI-D Stup Bar
Score Stop 1002
(Stp
Skep
[Sep- 10020
Scorc-
2.2
6.2
62
Step
(Sep
(Sep
10030
8.10
0,99
8.65
9.05
606
0.85
0.55
000
Step
(Sop-
LR (Sup-
(Stp
4.4
6A
64
64
04
83
D3
63
0.5 Ang occunay
0.5
6.5 Aug Sc0nar
Avg occomny
Avg cocksacy
Aug soarncy
Figure 28: DisVAEs metric scores v.s. WReN on Abstract dSprites.
29

---

Under review as a conference paper at ICLR 2023
Score
Step
Score
BoiVAE (See
BeiVVE (Sep
Scure (Sep
Seare
Best
18
2.5
0.6
0.4
6.4
66
0.4
FacueVAEl Supp
FactorVAEl Sopp 2000
Score FactorV Step 5006
FastorVAEl Stap
FastorVAl Stop
2.6
0.8
6.
0.6
64
04
Stop
Scoue
Scare
DCLD
Seare
Step
Siep
Score MBG Step 5000
Step 10000
0.2
6.2
02
0.2
(Ste
Score
(Stp-
SSep
10030
0.65
6.08
8.60
0.00
000
8.00
8.00
Scate Supp 1200
Score LR Stpp 2000
LR Step
Score LR|| Step
Secre
Step
2.5
0.5
0.50
0.
0.4
Ave aoMmaay
Arr
Aug
Avg
Figure 29: Dis VAES' metric scores v.s. Trans. on Abstract dSprites.
30

---

Under review as a conference paper at ICLR 2023
Souce BeiaVEl Shap 1000
Score BaiVAE Step 2000
Souce BeiVVEEI Ship
5000
Score BaaVAE AE Step 10000
Score
BAIVAE Step
Best
1.0
1.0
1.0
1.0
Sorevalie
0.5
6
6.5
0.90
0:90
6.70
0.00
D.5
Scrre PaccorV/Al Step 1000
Score - VaateVAlll Stup 2000
Scrrc PacaeVAll Step - 5000
Scom Ulameee Stop (0000
Score - F'ateeVAll Stop Bea
0.2
0.8
9.6
0,6
0.55
0.59
8.65
0.55
0:50
8.6
0.70
3.60
6.6
0,70
0.55
0.50
0.55
0.70
Soou
MBG Step 1000
Score
MIG Skep 2000
Soup
MXG Step
5000
Scoce
MIG Step
10000
Sccre
MIG Stop
Best
3.08
0.61
3.68
6.9
0.68
3.06
0.05
0.06
605
2.66
3.00
0.54
0.64
604
082
0.5
00
0.55
0.90
0.65
0.55
0:50
6.6
0.70
0.50
6.65
6070
0.55
0.50
0.55
6.00
Score SAU Stop 1000
Sortz SAA! Stap 2000
Score SAL Stup 5000
Scare SAP Skpp 10000
Scare SAP
Dood
0.015
0213
0A05
0015
0.505
Scare rale
8.000
0233
8AC0
0013
0.000
no5s
6.005
0005
0.065
0.5
9.55
0.60
6.65
0.55
0.50
6.65
9.70
3.60
6.65
6,70
0.55
0.50
0:55
6:70
L.R.
1003
LR
LR
Scuu
L.R
10000
Scoo
SSep
0.6
6.5
Sorralle
0.5
0.5
64
0.4
0.4
06
0.55
0.00 Avg. 300rtr
6.65
0.00
070
0.00 Avg.ooouryy
0.0
0.55
6.00
Aug cocureey
Aug REOUEeE
Aug seouney
Figure 30: BYOLs metric scores v.s. WReN on Abstruct dSprites.
31

---

Under review as a conference paper at ICLR 2023
Souce BeaVVEl Shap 1000
Score BaiVAE Step 2000
Souce BeiVVE Ship 5000
Score BaaVAE AE Step
10000
Score BAiVAE Step
Bost
1.
Soe
0.6
0.2
64
Scrrew PactorV/Al: Step 1000
Soar VaateVAAll Stup 2000
Score Vac Step - 5000
Scor Eaameee Stop (C000
Score - V'ateeVAll Stop Bea
0.2
8.4
0.5
0.3
Saat vdde
9.7
6:7
0.7
0.3
0.5
0.5
0.5
0.5
0.4
0.2
0.5
9.4
0.5
0.3
0.4
9.5
0.5
04
3.6
0.5
0.4
0.6
0.8
Soou MXG Step 1000
Score
MIG Slep 2000
Souz MXG Step 5000
Scoce MIG Step
10000
Score MIG Stop
Best
2.08
0.81
3.68
6.9
0.68
2.06
0.06
605
2.66
2.04
0.54
0.64
604
0.64
0.10
0.02
0.00
0.2
0.3
6.4
6d
0.5
0.4
0.5
0.6
644
0.6
0.4
0.6
0.
0.4
0.6
0.
Score SAl Stop 1000
Soatz SAP Step 2000
Score SAL Stup 5000
Scare SAP Skop 10200
Scare SAP Dood
0.055
0213
0.05
0015
0.515
Scare rabe
8.10
0333
8,00
0013
0.000
0085
8.605
0005
0.065
0.2
0.3
9.4
05
0.3
0.4
9.5
0.6
64
3.6
0.4
0.5
0.8
0.4
0.6
0.8
LR
1003
LR
1B
Scue L.R Stp 10000
Scoo
(Sep
0.6
0.6
Sucrvile
0.5
0.5
0.5
0.
0.4
0.4
0.4
0.2
6.4
0.5
6.4
044 Aug Rocureey
114
0.6
0k
Aug eocureey
Avg 3c0eaiyy
ArggooouRy
Ang seouncy
Figure 31: BYOLs' metric scores v.s. Trans. on Abstract dSprites.
32


Reviews:
[{"review_id": "hUeMJd5NFGW", "reviewer": "Reviewer_hRSM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies dimension-wise disentangled representations for downstream applications. Through extensive experiments, the authors conclude that disentanglement is not a necessity for achieving good performance in downstream tasks, and general-purpose representation learning methods could achieve better (or at least competitive performance) than disentanglement methods.\n\nThe authors show that Logistic regression accuracy on factor classification well-correlates with downstream task performance. The reason that we feel disentanglement is useful for downstream tasks is presumably due to the positive correlation between LR and disentanglement metrics.\n", "summary_of_the_review": "Overall, I think the paper shows something interesting, and I’m inclined to recommend this paper.\n\n1.     One thing I’m not sure about is how thoroughly that people have studied the importance of disentanglement for downstream tasks. This work is related to one previous ICLR submission (https://openreview.net/pdf?id=1JN7MepVDFv) where the authors studied the relationship between multi-task learning and disentanglement. The authors showed that disentanglement emerges naturally from MTL. The paper was rejected due to the main issue that MTL results in more extraction of information and that it is hard to disentanglement from the disentanglement metrics used. The authors (of the MTL work) also claimed that it is inconclusive whether disentangled representations have a clear positive impact on the model performance. In this work, the authors give a clearer explanation.\n\n2.     Recent SSL framework also shows that a linear head on top of a pre-trained general-purpose encoder can achieve near-SOTA performance. \n", "strength_and_weaknesses": "Strength:\n\n--The authors challenge the necessity of dimension-wise disentanglement for downstream tasks via extensive amounts of ablation studies. Through experiments, the authors examined 1) effects of attenuating disentanglement, 2) general-purpose vs disentangled training, 3) How different disentanglement/informativeness metrics correlate with downstream tasks, and 4) the correlation between LR and some disentanglement metrics. Every claim/reasoning is supported by experiments.\n\n--The authors conduct multiple runs of experiments, and cover different kinds of model architectures and disentanglement training methods and metrics. The exploration is pretty thorough. \n \nWeakness:\n\n--The exploration should be expanded to other tasks and domains.\n\n--There are a lot of general-purpose pre-training algorithms, but the authors mostly focus on BOYL.\n\n--Though two-stage training is acceptable, what if the experiments are conducted in a joint-training setup where disentangling losses are used to regularize the supervised loss?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YYHLNW-Z6z", "reviewer": "Reviewer_r3zU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the correlation between dimension-wise disentanglement scores and downstream performance. In particular, it does so when using MLPs or Transformers to perform the task of abstract visual reasoning using the learned representation. After observing a poor correlation on this task, it concludes that disentanglement is not *necessary* for good downstream performance.", "summary_of_the_review": "While this paper is well written and explores a question that is both open and important, it ultimately fails to meet the standards required for acceptance. In particular: (1) its central claim---disentanglement is not *necessary* for downstream tasks---seems trivial; (2) its evaluation of sample efficiency is incorrect (this is one of the main purported benefits of disentanglement); and (3) its novelty/difference in comparison to prior such evaluations is not made sufficiently clear.", "strength_and_weaknesses": "**Strengths:**\n- **Clear writing:** this paper was easy to read and understand, communication was clear.\n- **Important question/investigation:** The usefulness of disentanglement for downstream tasks is an open and important question.\n\n**Weaknesses:**\n- **Not clear who claimed disentanglement is *necessary* for downstream tasks:** \n  - I agree with the authors that it is often said/believed that disentangled representations *can* be beneficial for downstream tasks. \n  - However, throughout this paper, including in the title, the authors speak of the *necessity* of disentanglement representations for downstream tasks, claiming that they \"challenge the necessity of disentanglement for downstream tasks\". It is not clear to me who has claimed that disentanglement is *necessary*, i.e. why it should be surprising that there exists a downstream task for which disentanglement does not help---this seems like a trivial statement.\n  - This is a major weakness since \"challenging the necessity of disentanglement\" is perhaps the central contribution of this paper.\n  - A more interesting and non-trivial alternative question could be: when does disentanglement help, and when does it not? This is what prior works investigated [2,3,4], Naturally, there will be tasks for which it helps and tasks for which it does not.\n- **Incorrect evaluation of sample efficiency:**\n  - The authors use learning curves (gradient step vs. accuracy) to evaluate \"sample efficiency\". While each step sees new samples, this is fundamentally flawed since it convolutes sample efficiency (the performance with N samples) and update/step efficiency (the performance with M gradient updates).\n  - This is a major weakness since sample efficiency is one of the most commonly-purported downstream benefits of disentanglement [1,2,3,4], making it central to the authors' claims.\n- **Insufficient comparison to related studies on disentanglement and downstream performance:**\n  - Many works have thoroughly investigated the correlation between disentanglement and downstream performance [2,3,4]. These works were much wider in their scope (tasks, datasets, representations) and reached different conclusions. Despite the attempt to undermine these studies in the related work and on page 9, I was left unconvinced that the results in this paper are novel or should underline/question those that came before. A better comparison and explanation would help, or perhaps a more specific claim could be made (e.g. relating to this one reasoning task with neural-net architectures).\n- **Minor:**\n  - *Rotation-of-factors issue is well-known and studied:* The authors claim to \"find that rotating disentangled representations [...] has no impact on [...] final accuracy\". This seems unsurprising given the well-known issue of rotation of factors in a linear factor-analysis model [5, sec. 9.6], which also leads to the condition in independent components analysis (ICA) that at most one of the factors can be Gaussian [6]. It was also discussed in [7]. Finally, note that it has also been investigated by a very recent (perhaps concurrent) work [8] which proposes a new notion of disentanglement that is unaffected by such rotations [8].\n  - *The term \"informativeness\" is either overloaded or not cited:* The authors seem to use the term \"informativeness\" to refer to the linear-classifier performance in classifying the ground-truth factors from the learned (disentangled) representation. If so, this is precisely the definition of the \"informativeness\" metric in [7], but the authors never mention this relation. If this is in fact the same metric, the authors should appropriately cite, or if not, they should use a different name/term to avoid overloading an existing metric for evaluating disentangled representations.\n  - *Unclear to me why disentanglement should help the specific abstract-reasoning task used:* Are only a subset of the factors needed? Do only a subset of the factors change? Is it surprising that disentanglement does not help?\n  - *Incorrect and imprecise statements:*\n    - In the second paragraph of the introduction, the authors claim that the abstract-reasoning task is general and widely-adopted, while other downstream evaluation tasks are \"trivial or domain-specific\", citing many past evaluations. I have to disagree with this presentation of abstract visual-reasoning as the holy-grail of downstream evaluations, and suggest that the wording be toned down.\n    - *\"Locatello et al. (2019b) proves their agreement on VAE methods\"* -- Locatello et al. do not prove the agreement of different disentanglement metrics -- many of them measure different things.\n   - *\"it takes hours to develop the Gradient Boosting Trees required [to evaluate DCI disentanglement]\"*: GBTs are not _required_ to evaluate DCI disentanglement---any classifier can be used, including those with a lower cost (e.g. random forests).\n\n[1] Bengio, Y., Courville, A., \\& Vincent, P. (2013). Representation learning: A review and new perspectives. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 35(8), 1798-1828.\n\n[2] Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., \\& Bachem, O. (2020). A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation. _Journal of Machine Learning Research_, 21, 1-62.\n\n[3] Van Steenkiste, S., Locatello, F., Schmidhuber, J., \\& Bachem, O. (2019). Are disentangled representations helpful for abstract visual reasoning?. _Advances in Neural Information Processing Systems_, 32.\n\n[4] Dittadi, A., Träuble, F., Locatello, F., Wüthrich, M., Agrawal, V., Winther, O., ... \\& Schölkopf, B. (2021). On the Transfer of Disentangled Representations in Realistic Settings. In _International Conference on Learning Representations_.\n\n[5] Mardia, K. V., Kent, J. T., \\& Bibby, J. M. (1979). _Multivariate Analysis_. Academic Press, London.\n\n[6] Hyvarinen, A., Karhunen, J., \\& Oja, E. (2001). _Independent Component Analysis_. Wiley.\n\n[7] Eastwood, C., \\& Williams, C. K. I. (2018). A framework for the quantitative evaluation of disentangled representations. In _International Conference on Learning Representations_.\n\n[8] Eastwood, C., Nicolicioiu, A. L., von Kügelgen, J., Kekić, A., Träuble, F., Dittadi, A., \\& Schölkopf, B. (2022). DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability. _arXiv preprint arXiv:2210.00364_.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "T77WPX8OMo", "reviewer": "Reviewer_RvuP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors call into question the conventional thinking that downstream tasks (the take a representation as input) benefit from a disentangled representation. Their results are that the informativeness of the representation not the disentangled nature of is what results in improved downstream performance.", "summary_of_the_review": "The authors demonstrate for RPM downstream tasks the informativeness (the ability for the logistic regression to achieve a level of accuracy using the representation) is more indicative of downstream performance that the disentanglement of the representation. My biggest issue with the paper is their use of the orthonormal matrix multiplication to re-entangle the representation. ", "strength_and_weaknesses": "As I was reading the paper I was confused by these two statements:\n\n\"However, on the abstract visual reasoning task, we find that rotating disentangled representations, i.e., multiplying the representations by an orthonormal matrix, has no impact on sample efficiency and final accuracy.\"\n\n-and-\n\n\"Our finding demonstrates that disentanglement does not affect the downstream learning trajectory, which is against the commonly believed usefulness of disentanglement. On the other hand, it is not surprising since we apply an invertible linear transform. We can observe that Logistic Regression (LR) accuracy remains 100% before and after rotation, indicating that a simple linear layer could eliminate the effects of rotation.\"\n\nIt appeared the authors believed that the multiplication would destroy the disentangled representation, and used this to prove their thesis. But, they later acknowledged that LR is capable of reversing the multiplication. I am at a loss for why to include this in the paper, and why to have it so early in the text.\n\nThe authors have a good experiment design, using learners that provide a disentangled representation (various VAE based approached) and compare against learned representations created by BYOL (which makes no claim to disentanglement). These representations are used to perform abstract visual reasoning tasks. They use the accuracy of using a logistic regression on the learned representation (informativeness) as a metric to show the usefulness of the representation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D_TrinH0H4a", "reviewer": "Reviewer_ZFjG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper challenges a common belief that a disentangled representation is useful for downstream tasks. Following up Steenkiste et al., 2019 and Locatello et al. 2019b,  the authors focused on the informativeness of the representation and its correlation with the performance of downstream tasks.", "summary_of_the_review": "Adding logistic regression as a measure of informativeness to previous measures of disentanglement, it is clear from experiments that this measure should be of value in evaluating learned representations in connection with downstream accuracy.", "strength_and_weaknesses": "Strong points:\n\n-extensive experimentation and evaluation\n\n-effective use of logistic regression as a measure of informativeness\n\n-a comparison between the effects of disentangled representation vs. deliberate entanglement of disentangled representations on downstream task\n\nWeak points:\n\n-lack of theoretical contribution\n\n-there are far more types of disentangling VAEs against one general-purpose representation learning method (BYOL). Since downstream test performance is averaged over different VAEs, is this a fair comparison?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5sQLUgt6_K", "reviewer": "Reviewer_Pb5b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper performs a large scale empirical study to investigate whether disentangled representations provide a clear benefit for the final performance on downstream tasks. First, the ground-truth disentangled representation (normalized true factors) are compared to a rotated version of the same representations. The authors show that the two types of representations yields no significant difference in the final downstream performance. The second paper contribution compares the final performance of two models (a WReN and a Transformer) on an abstract reasoning tasks, using representations learned both via disentanglement-oriented learning methods (DisVAEs) and an entangled representation learning method (BYOL). They report a better downstream performance using representations based on BYOL. Finally, the authors show that the Informativeness is the metric that correlates the most with downstream performance on both DisVAE and BYOL representations, with disentanglement bringing only a small extra benefit.", "summary_of_the_review": "This paper tries to tackle some relevant problems of the representation learning community. While the experiments are generally well-designed, it does not seem to me that the presented evidence is enough to support the major claims of this paper.  Some additional experiments are needed, and some concerns in the experimental settings need to be addressed, in order to make this manuscript suitable for publication.\n\nMinor points:\n - The term “FactorVAE” is often spelled incorrectly on page 8.\n - The second point of contributions in the introduction start with the sentence “We show that what information...” that is a bit hard to read. I would recommend rephrasing that sentence.\n", "strength_and_weaknesses": "STRENGHTS\n\n(+) The experimental setting, is for the most part, reasonable and well-designed.  The paper is well-written and easy to follow. The experiments try to tackle relevant questions in the representation learning community.\n\nWEAKNESSES\n\nThe evidence presented is not enough to back up all claims of this paper. In particular, I have the following concerns:\n\n(-) In the first contribution, the true disentangled representations are compared only with a rotated version of themselves. Even if the rotated representations are not disentangled according to the metrics, I think that they are still a much more similar to a disentangled representation than the typical representations learned by deep learning models. Therefore, comparing between the two is not enough to claim that disentanglement is not beneficial for downstream performance in general. A fairer comparison would be between true disentangled representations and entangled representations learned by a standard VAE, where \\lambda_1 and \\lambda_2  of Equation 3 are both set to 0. \n\n(-) Table 1. Two concerns: first, for which DisVAEs model does the reported accuracy refer to? Is it an average between the models? Second, reporting the step with the highest validation accuracy for both DisVAE and BYOL can be misleading, since it hides information about *when* that accuracy is achieved. It might be the case where BYOL achieves an overall better accuracy, but much later than DisVAEs. In that case, using a DisVAE model for representation learning could still be valuable. It is generally fairer to fix a specific number of training steps, or to run an early stopping strategy. More generally, it can be interesting to have more information about the speed of convergence of these model. Furthermore, I would have expected to see the standard deviations of performance of the 5 different runs of WReNs and Transformers reported in Table 1.\n\n(-) Figure 1 should be moved closer to section 4.2, for the sake of readability.The results of  Figures 5 and 6 show that informativeness has a strong correlation with downstream tasks. However, it seems that disentanglement somewhat implies informativeness, and that the representations trained with BYOLS exhibit high disentanglement scores on dSprites, at least in the case of the beta-VAE and FactorVAE scores. I am not sure that correlations alone are enough to make any conclusions about the usefulness of disentanglement. In order to conclude that informativeness is more beneficial than disentanglement on downstream tasks, I would like to see a comparison of the absolute values of disentanglement scores and informativeness scores for the two representations, showing that BYOL representations have higher informativeness, lower disentanglement, and higher downstream performance than DisVAE representations. Finally, it is still not very clear to me how the authors selected the final DisVAEs representation to be used in the figures.\n\nOn the minors side, Figure 1 should be moved closer to section 4.2, for the sake of readability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject