On the relationship between disentanglement and multi-task learning
Authors: Lukasz Maziarka
Aleksandra Nowak
Maciej Wolczyk
Andrzej Bedychaj
Venue: ICLR 2022 Submitted (2022)
Keywords: disentanglement representations
multi-task learning

Abstract: One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks. At the same time finding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting. In this paper, we take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We perform a thorough empirical study of the representations obtained by neural networks trained on automatically generated supervised tasks. Using a set of standard metrics we show that disentanglement appears in a natural way during the process of multi-task neural network training.

Under review as a conference paper at ICLR 2022
ON THE RELATIONSHIP BETWEEN DISENTANGLEMENT AND MUSTI-TASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks. At the same time finding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting. In this paper, we take a closer look at the relation- ship between disentanglement and multi-task learning based hard parameter sharing. We perform a thorough empirical study of the representations obtained by neural networks trained on automatically generated supervised tasks. Using a set of standard metrics we show that disentanglement appears naturally during the process of multi-task neural network training.
INTRODUCTION
Disentangled representations have recently become an important topic in the deep learning commu- nity (Eastwood & Williams, 2018; Locatello al., 2019a; Ma et al., 2019; Sanchez al., 2019; Do & Tran, 2020). The main assumption in this problem is that the data encountered in the real world is generated by few independent and explanatory factors of variation. It is commonly accepted that such representations are not only more interpretable and robust but also perform better in tasks re- lated to transfer learning and one-shot learning (Bengio, 2013; Lake al., 2017: Schölkopf al., 2012; Locatello et al., 2019c).
Intuitively, a disentangled representation encompasses all the factors of variation and as such can be used for various tasks based on the same input space. On the other hand, non-disentangled representations, such as those learned by vanilla neural networks, might focus only on one or a few factors of variations that are relevant for the current task, while discarding the rest. Such representation may fail when encountering different tasks that rely on distant aspects of variation which have not been captured.
Exploiting prevalent features and differences across tasks is also the paradigm of multi-task learn- ing. In a standard formulation of multi-task setting, model is given input and has return predictions for multiple tasks at once. The neural network might be therefore implicitly regularized to capture more factors of variation than a network that learns only single task. Based on this intuition, we hypothesize that disentanglement is likely to occur in the latent representations in this type of problem.
This paper aims to test this hypothesis empirically. We investigate whether the use of disentangled representations improves the performance of a multi-task neural network and whether disentangle- ment itself is achieved naturally during the training process in such a setting.
Our key contributions are:
Construction of synthetic datasets that allow studying the relationship between multi-task and disentanglement learning.
Study of the effect multi-task learning with hard parameter sharing on the level of dis- entanglement obtained in the latent representation of the model.
Analysis of the informativeness of the latent representation obtained in the single- and multi-task training.
Inspection of the effect of disentangled representations on the performance of a multi-task model.

---

Under review as a conference paper at ICLR 2022
We verify our hypotheses by training multiple models in single- and multi-task settings and investi- gating the level of disentanglement achieved in their latent representations. In our experiments, we find that in a hard parameeer sharing scenario multi-tasi learning indeed seems encourage disen- tanglement. However, it is inconclusive whether disentangled representations have a clear positive impact on the models performance, the obtained by us results in this matter vary for different datasets.
2
RELATED WORK
2.1 DISENTANGLEMENT
Over the recent years, many methods that directly encourage disentanglement have been proposed. This includes algorithms based on variational and Wasserstein auto-encoders (Kim & Mnih, 2018; Higgins al., 2017: Kumar et al., 2017: Brakel & Bengio, 2017: Spurek et al., 2020), flow networks (Dinh et al., 2014; Sorrenson et al., 2020) or generative adversarial networks (Chen et al., 2016). The main interest behind disentanglement learning lays in the assumption that such transformation unravels the semantically meaningful factors of variation present in the observations and thus it is desired in training deep learning models. particular, disentanglement believed to allow for informative compression of the data that results in a structural, interpretable representation, which is casily adaptable for new tasks (Bengio, 2013; Lake et al., 2017; Schmidhuber, 1992; Lipton, 2018).
Several of these properties have been experimentally proven in applications in many domains, in- cluding video processing tasks (Hsich et al., 2018), recommendation systems (Ma et al., 2019) or ab- stract reasoning (Van Steenkiste al., 2019; Steenbrugge et al., 2018). Morcover, recent research in reinforcement learning concludes that disentangling embeddings of skills allows for faster retraining and better generalization (Petangoda et al., 2019). Finally, disentanglement seems also be pos- itively correlated with fairness when sensitive variables are not observed (I (Locatello et al., 201 9a). On the other hand, some empirical studies suggest that one should be cautious while interpreting the properties of disentangled representations. For instance, the latest studies in the unsupervised learn- ing domain point that increased disentanglement does not lead to decreased sample complexity in downstream tasks (Locatello et al., 2019b).
Another key challenge in studying disentangled representations is the fact that measuring the quality of the disentanglement is nontrivial task (Do & Tran, 2020; Eastwood & Williams, 2018: Kim Mnih, 2018), especially unsupervised setting (Locatello et al., 2019b). This motivates the research on practical advantages of disentanglement representations and their impact on the studied problem in possible future applications, which is the main focus of our work in the case of multi-task learning.
2.2 MULTI-TASK LEARNING
Multi-task learning aims simultancously solving multiple tasks by exploiting common information (Ruder, 2017). The approaches used predominantly to this problem are soft (Duong et al., 2015) and hard (Caruana, 1993) parameter sharing. In hard parameter sharing the weights of the model are divided into those shared by all tasks, and task-specific. In deep learning, this idea is typically implemented by sharing consecutive layers of the network, which are responsible for learning joint data representation. In soft parameter sharing each task is given set of separate parameters. The limitations are then imposed by information-sharing or regularizing the distance between the parameters by adding applicable loss to the optimization objective.
Multi-task learning is widely used in the Deep Learning community, for instance in applications related to natural language processing (Liu et al., 2019), computer vision (Misra et al., 2016) or molecular property prediction modeled by graph neural networks (Capela et al., 2019). One may observe that the premises of multi-task and disentanglement learning are related to each other and thus it is interesting to investigate whether the joint data representation obtained in a multi-task problem exhibits some disentanglemennt-elaadde properties.
2

---

Under review as a conference paper at ICLR 2022
3
METHODS
In this section, we describe the methods and datasets used for conducting the experiments.
3.1 DATASET CREATION
In order to investigate the relationship between multi-task learning and disentanglement, we require a dataset that fulfills two conditions:
1. It provides access to the true (disentangled) generative factors from which the observa- tions x are created.
2. It proposes multiple tasks for supervised learner by providing labels ys which non-linearly depend on the true factors 2.
The first condition is required in order to mea- sure how well the learned representations ap- proximate the true latent factors 2. Access to the true factors allows for full control over the experimental settings and permits a fair com- parison through the use of supervised disentan- glement metrics. Note that even though unsu- pervised metrics have been proposed in the lit- erature well, they typically yield less reliable results, as we further discuss in section 3.3.
Randomy inttalized networkss his)
31
Shooe:
Positan 10pe
xx
Postinn: p0pe
Roadonn 3.2 n Scale:10
Generative function g(t)
Supervised learner
The setting of our experiments. Given a dataset of pairs (z, 2) of observations and their true generative factors, we generate a of func- tions h(=); which are aimed to approximate real- world supervised tasks. Then, we train neural network (x) in multi-task regression setting on pairs (x, h(z)). After the training, we investi- gate the hidden representations learned by fg and explore their relation to true factors 2.
Figure 1:
The second condition is needed train net- work on multiple nontrivial tasks to approxi- mate the real-world setting of multi-task learn- ing.
To our best knowledge, no nontrivial datasets exist that would abide by both those require- ments. Most of the available disentanglement datasets, such dSprites, Shapes3D, and MPI3D do fulfiil the first condition, as they provide pairs (I, 2) of observations and their true generative factors. However, those datasets do not offer any type of challenging task which model could be trained. On the other hand, many datasets used for supervised multi-task learning fulfill the second condition by providing pairs (I, y). but do not equip the researcher with the latent factors (ground truth), failing the first condition.
Thus, we aim to create our own datasets which fulfill both conditions by incorporating nontrivial tasks into standard disentanglement datasets. Since in multi-task approaches one often tries solve tens of tasks at once, designing them by hand is infeasible and as such we decide to generate them automatically in a principled way. In particular, since supervised learning tasks might be formalized as finding a good approximation to an unknown function f(x)) given a set of points (z, h(x)), we generate random functions h(=) which are then used to obtain targets for our dataset (see Figure 1).
We require h22) to be both nontrivial (i.e. non-linear and non-convex) and sufficiently smooth to approximate the nature of real-life tasks. In order to find a family of functions that fulfills those conditions, we take inspiration from the field of extreme learning, which finds that features obtained from randomly initialized neural networks are useful for training linear models on various real- world problems (Huang et al., 2011). As such, randomly initialized networks should be able to approximate these tasks up to a linear operation.
In particular, in order to generate the dataset, we define neural network architecture h(z, 0). For this purpose, we used an MLP with four hidden layers with 300 units, tanh activations, and an output layer which returns single number. Then sample weight initializations of this network from the Gaussian distribution 0; ~ (O, 1), where E {1....,mi, n.. Each of the networks h(2,, 0.) obtained by random initialization defines a single task in our approach. Thus, for given dataset D = (xt, =) containing observations and their true generative factors, we obtain dataset for multi- task supervised learning by applying:
D {1x, h(2)) (x) D) (Λx,
3

---

Under review as a conference paper at ICLR 2022
where h(2) is a vector of stacked target values for each task, whose element i is given by h(*); h(2, 8.).
We use this data regression task, i... for given neural network fø parameterized by é the goal is to find:
arg min
Σ
(lf -ylé.
(x,))D
We use this process to create multi-task supervised versions of dSprites, Shapes3D, and MPI3D,
with 10 tasks for each dataset.
3.2 MODELS
3.2.1 MULTI-TASK MODEL
We investigate the relation between disentanglement and multi-task learning based hard pa- rameter sharing approach. In this setting, several consecutive hidden layers the model are shared across all tasks in order to produce joint data representation. This representation is then propagated to separate task-specific layers which are responsible for computing the final predictions.
In particular, a shared convolutional encoder and separate fully-connected heads for each of the tasks. The encoder learns the joint representation by trans- forming the inputs into d-dimensional latent space. The heads are implemented by 4-layer MLPs with ReLU activations, order to match the capacity of the networks used for task gen- erating functions (x)x). This overview of the model is illustrated in Figure 2.
network consisting of
St
Input
convolutional enceder Fis)
setert eek:
(FCN)
9z
Tasc a iiteed (FER)
Figure The model used for multi-task train- ing. The convolutional encoder E(x) transforms the input data x to a latent representation parameters of the encoder are shared across all tasks. Next, the produced representation is passed the task-specific heads, which are implemented by fully-connected networks (FCN).
The
3.2.2 AUTO-ENCODER MODEL
In the second part of our experiments we want to understand if disentangled represen- tation provides some benefits for the multi- task problem. In order produce disentangled representations, decided ferent prresentationnIeaamne encoder (Kingma & Welling, 2013; Higgins et al., 2017) and FactorVAE (Kim Mnih, 2018).
three dif- algorithms: vanilla auto-encoder, the (beta)-variational auto-
All these variants of the auto-encoder architecture encompass a similar framework. An auto-encoder imposes a bottleneck in the network which forces a compressed knowledge representation of the original input. In some variants of those models, we additionally try to constrain the latent variables to be highly informative and independent which further correlates to disentanglement, e.g. in models like 3-VAE and VAC. We use latent representations from these models to train task-specific heads and evaluate disentanglement helped to decrease an error that task.
The vanilla auto-encoder is also used in Section 4.2, where we add decoder with transposed con- volutions to pre-trained encoders from Section 4. This treatment is aimed to decode information for particular encoders in the most efficient way. As such, we find auto-encoders to be useful tool for investigating disentanglement.
3.3 DISENTANGLEMENT METRICS
Measuring the qualitative and quantitative properties of the disentanglement representation discov- ered by the model nontrivial task. Due to the fact that the true generating factors given
We provide the full model summary in Appendix A. The architecture of the encoder follows the one from (Abdi et al., 2019), which adopts the work of (Locatello et al., 2019b) for the pyterch package. We use the implementations from https: // thub. com/ ami abdi / /di sentangl ement pepentppytee -pytorch.

---

Under review as a conference paper at ICLR 2022
dataset usually unknown, one may assume that decomposition can be obtained only to some extend.
Commonly used unsupervised metrics are based on correlation coefficients which measure the in- trinsic dependencies between the latent components. Such measures are widely used the inde- pendent component analysis (Hyvarinen & Morioka, 2016; 2017; Hirayama et al., 2017; Brakel & Bengio, 2017; Spurek et al., 2020; Bedychaj et al., 2020). However, uncorrelatedness does not imply stochastical independence. Furthermore, metries based on linear correlations may not be able to cap- ture higher-order dependencies and are often ineffective in large dimensional or in over-determined spaces. All this makes the use of such unsupervised metrics questionable.
An alternative solution would be to use supervised metries, which usually are more reliable (Lo- catello et al., 2019b). This is of course only possible after assuming access to the true generative factors. Such an assumption is rarely valid for real-world datasets, however, it is satisfied for syn- thetic datasets. Synthetic datasets present therefore reasonable baseline for benchmarking disen- tanglement algorithms.
Frequently used metrics which use supervision are mutual information gap (MIG) (Chen et al., 2018), the FactorVAF metric (Kim & Mnih, 2018), Separated Attribute Predictability (SAP) score (Kumar et al., 2018) and saapaanggleeenncoopppeenss--iiii Williams, 2018). In order to comprehensively assess the level of disentanglement in our experi- ments, we have decided to use all of the above-mentioned metrics to validate our results. A more detailed description of those metrics is available in Appendix B.
(DCI) (Fastwood &
RESULTS AND DISCUSSION
In this section, we describe the performed experiments and discuss the obtained results. For more details on the training regime and experimental setup please refer Appendix C.
4.1 DOES HARD PARAMETER SHARING ENCOURAGE DISENTANGLEMENT
One of the most common approaches to multi-task learning is hard parameter sharing. The key challenge in this method is to learn a joint representation of the data which is at the same time infor- mative about the input and be easily processed in more than one task. It is therefore tempting to verify whether disentanglement arises in those representations implicitly, as a consequence of hard parameter sharing.
In order to investigate this problem we build a simple multi-task model described in Section 3.2 and evaluate on the three datasets discussed in Section 3.1: dSprites, Shapes3D, and MPI3D, each with 10 artificial tasks. After the training is complete, we calculate each of the disentanglement metries described in Section 3.3 on the latent representation of the input data?. We compare the obtained results with the same metrics computed for an untrained (randomly initialized) network and for single-task models. In all the cases we use the same architecture and training regime. Note that in the mongle-model scenario we train separate model for each of the 10 tasks, which is implemented by utilizing only one, dedicated head in the optimization process. We train all models three times, using a different random seed in the parameters initialization procedure. We report the mean results and standard deviations in Figure 3.
We observe that disentanglement metries computed for the representations obtained in the multi-task setting are typically significantly better than the values obtained for single-task or random represen- tations. Note that even the maximum mean result over all ten single-task models is in almost every case further than one standard deviation from the multitask mean. Moreover, this is true for all the tested datasets.
Let us also point out that instead of using separate heads for each of the tasks in the multi-task model one could simply use one head with the output dimension equal the number of tasks and perform standard multivariate regression (with no parameter sharing). As presented in Figure 4, the latent representations emerging such a scenario are less disentangled (in terms of the considered metrics)
We use the implementations of Locatello et al. (2019b), which are available at httph:t////// /tttp://giii thub. google research/cee rch/ sentangl ement_ 1ib
5

---

Under review as a conference paper at ICLR 2022
MIG
factor VAE
completeness
disentanglement
informativeness,
SAP score
0.5
0.18
TAY
nea
mn
mih
0.15
69 Mi
WSP
0.02
o5prites
0.06
0.15
0.01
0.00
0.4
0.30
0.4
0.02
0.04
0004
0.07
6.02
6003
0.2
0.05
0.03
1115
0.1
0.022
1.100
0003
0.00
6.10
man
mean
Yean
meo
0.35
Shapes.30
6.08
0.8
2.06
0.4
10
0.03
0.2
0.02
0.01
0.01
0.50
0.0
8.99
0.00
6.00
0.8
0.38
11.00
1m8
mear
meen
X00 Year
0.03
MP MPI3D
0.5
0.01
0.00
0.10
0.10
0.0
0.50
0003
8.04
0006
0.2
2.201
0.05
6607
6.02
0001
0.m3
6.00
random
muthi
0.0
random
single
mutii
0.00
random
single
muthi
0.00
randum
mutii
0gg
randorm
single
muli
0.00
randora
single
min
Figure 3: Different disentanglement metrics computed for random (untrained), single-task and multi-task models evaluated on the three datasets described in Section 3.1. The higher the value the better. For the single-task scenario, we report the mean over all task-specific models. Note that in almost every case the multi-task representations (red bars) outperform the random or single- task representations (dark-gray bars and light gray bars, respectively). Additionally, for single-task models, we report the maximal and minimal values over all tasks to show that the performance on multi-task does not rely on any single 'lucky' task. For tabulated results please refer to Appendix E.
0.10
MIG
factor VAE
0.20
completeness
0.20
disentanglement
0.6
informativeness
0.06
SAP score
0.6
0.15
0.15
0.04
0.4
0.4
0.10
0.03
0.04
0.2
0.05
0,2
0.01
0.00
Sapes
NP J0
0.0
65p7
0.00
65amees
HP33
0.00
cSprbes
SrapesI
NP330
0.0
0597005
Shapsc10
WP-30
0.00
WFT0
Figure 4: Different disentanglement metrics computed for the multi-task setting with one head
shared between all tasks (one-head) and separate head for every task (multi-head), evaluated on the three datasets described in Section 3.1. The higher the value the better. One may observe that melti-head representations perform better than the ones obtained in the standard, one-head multi- variate regression task. For tabulated results please refer to Appendix E.
than the representations obtained when utilizing hard parameter sharing. However, the achieved values are still better than in single-task models. This suggests that even though the increase in the metrics may be partially caused by simply training the network on higher-dimensional targets, the positive influence of hard parameter sharing cannot be ignored. This advocates in favor of the hypothesis that multi-task representations indeed more disentangled than the ones arising in single-task learning.
4.2 WHAT ARE THE PROPERTIES OF THE LEARNED REPRESENTATIONS?
The previous section discussed the obtained representations by analyzing quantitative disentangle- ment metrics. Here, we provide more insights into the characteristics of latent encodings.

---

Under review as a conference paper at ICLR 2022
floor hue
hue
object hue
scale
shape
orientation
multittakk
A
sing single-task
random
120
Figure 5: UMAP embeddings of the latent representations of the Shapes3D test dataset obtained for different models. Change the color within one subplot presents the change in one particu- lar ground truth component. The embeddings obtained by the multi-task model seem to be most semantically meaningful. See Appendix for plots for other datasets.
4.2.1 UMAP EMBEDDINGS
In order to gain intuition behind the differences between the representations obtained in the pre- vious experiment compute 2D-embedding of the latent encodings using the UMAP algo- rithm (Mcinnes et al., 2018). The results are presented in Figure 5.
The embeddings obtained for the multi-task representations much more semantically meaning- ful, with easily distinguishable separate clusters. Moreover, the position and internal structure of the clusters correspond different values of the true factors. This cannot be observed for the un- trained or single-task representations, suggesting that the multi-task representations are indeed more successful in encompassing the information about the real values of the generative sources of the data.
imput
random
single-task
multi-task
Shaap (essasii
Figure 6: Reconstructions obtained by the decoders trained on random, single-task, and multi-task encoding. For reference, we provide the original input images in the first row. The quality of the reconstruction for the random and single-task representation is very poor. Contrary, the multi-task encoder provided latent space that can be successfully decoded into images that closely resemble the corresponding examples from the input. Thus, conclude that the multi-task representations are more informative about the data and provide better compression. See Appendix D.2 for recon- structions for other datasets.
4.2.2 LATENT SPACE TRAVERSAL
Providing qualitative results of the retrieved factors a common practice in disentanglement learn- ing (Locatello et al., 2019e; Kumar et al., 2017: Sanchez al., 2019; Sorrenson et al., 2020; Lo- catello et al., 2019b). In particular, visual presentation of the interpolations over the latent space allows assessing from human perspective the informativeness and decomposition of the ob-

---

Under review as a conference paper at ICLR 2022
tained representations. Note that such analysis is possible only after adding and training suitable decoder network, which maps the retrieved factors back to the image space.
In our setting, the decoder mirrors the archi- tecture of the encoder (the convolutions are re- placed by transposed convolutions of the same size - see Appendix A). Given the latent rep- resentations as an input, the decoder optimizes the reconstruction error (as measured by MSE) between its outputs and the original images. We train three separate decoders corresponding to the different encoders from the previous section randomly initialized encoder, an encoder produced by one of the single-task models, and multi-task encoder.
(a) Random encoder
First, let us discuss the reconstruction quality achieved by each of the tested decoders. Re- sults of this experiment are presented in Fig- ure 6². Reconstructions produced for the multi- task encodings are clearly superior to the ones obtained for the single-task encodings. In the first case, the resulting images are sharp and contain almost no noise. In contrast, the sin- gle task reconstructions are blurry and similar to the ones produced for the randomly initial- ized encoder. We would like to emphasise that all the decoders used the same architecture and that during their optimization the parameters of the corresponding encoders were kept fixed. Therefore the quality of the reconstruction is an important property of latent representation, as it allows us to assess the compression capacity of the representation. From this perspective, the compression obtained in the multi-task scenario is much more informative about the input than in the single-task scenario.
(b) Single task encoder
(c) Multi-task encoder
Figure 7: Traverses over latent variable produced for a given architecture. The same example was used in all three traverses. The second row of each image shows how the decoder reconstructed this example in a particular setting. The rest of the factors come from latent space generated from each encoder. Visualization of components from the multi-task encoder are sharp and distinguish the generating factors distinctly. The same cannot be said about the latent factors in single-task and random encoders. which are blurry and discon- nected from any interpretable ground truth factors. Please refer to Appendix for the results of the traversals over other datasets.
Another approach to the visualisation of the latent variables is to perform interpolations (traversals) in the latent space. We start by se- lecting a random sample from the dataset and compute its encoding ă E Rd. By modifying one of the components of vector z from -1 to with 0.1 step and leaving the d - 1 unchanged, we produce a traversal along that particular fac- tor. We repeat this procedure for all the fac- tors in order to capture their impact on the de- coded example. Results of such traverses for the dSprites dataset are shown in Figure 7.
Note that since the models were not trained directly for disentanglement but only solve a super- vision task, it is not surprising that the representations are not as clearly factorized as in specialized methods such as VactorVAE. However, for the multi-task model, certain latent dimensions still ap- pear to be disentangled and one can casily spot the difference in quality between the single and multi-task representations. In the multi-task traversals, we can notice components that are respon- sible for the position and scale of a given figure (in Figure 7c, consider the 5th and 7th factors, respectively). In contrast, the results for single task representations demonstrate that even slight change in any of the single latent dimensions leads to degradation of the reconstructed examples.
Numerical values for reconstruction errors are presented in Appendix D.2.
8

---

Under review as a conference paper at ICLR 2022
As expected, this effect is more evident for the random (untrained) representations, where the corruption over latent factor is even more prevalent than in the case of a single-task traversal.
4.3 DOES DISENTANGLEMENT HELP IN TRAINING MULTI-TASK MODELS?
In the previous sections, we studied whether multi-task learning encourages disentanglement. Here we consider an inverse problem by asking whether using disentangled representation helps multi- task learning. To investigate this issue, we train an nuto-encoder-basee model devised specifically to produce disentangled latent representations without access to the true latent factors. Next, we freeze its parameters and use the encoder function to transform the inputs. The obtained latent encodings are then passed directly to the heads of a multi-task network which minimizes the average regression loss given the target values of the artificial tasks.
We consider three different auto-encoder-baee based algorithms described in Section 3.2.2: a vanilla auto- encoder (AE), variational auto-encoder (VAE), and the FactorVAB: The vanilla auto-encoder does not directly enforce latent disentanglement during the training. In the VAE model, the prior normal distribution with identity covariance matrix implies some disentanglement, Finally, FactorVAB in- troduces a new module to the VAE architecture that explicitly induces informative decomposition. Therefore, the representations obtained for each subsequent model should be also naturally ordered by the level of the achieved disentanglement. For the exact values of the calculated metrics please refer to Appendix
In addition, we also study scenario in which we explicitly provide the true source factors. We trained all regression models three times, using a different random seed in the parameters initialization procedure.
Table 1: RMSE of multi-task networks trained on latent representations obtained by different auto-encoder-baase methods. For comparison, we added the model trained on ground truth factors. The best results are bolded, and best out of auto-encoder architectures underlined.
Table summarizes the performance of the multi-task model trained on the representations obtained for the above- discussed methods. Although the rep- resentations obtained from Factor VAE are better (see, for instance, MIG or DCI measures in Appendix F) than those from VAE and AE, the encod- ings produced by the vanilla AE the best among the tested, exceeding the others on Shapes3D and MPI3D and being second on dSprites. Note that these results coincide with observations presented in the litera- ture. For example, (Locatello et al., 2019b) compared different models that enforce disentanglement during the training and showed that even high value of that property within the factors do not constitute better model performance. However, two out of three datasets, the use of the ground true factors seems to significantly improve the obtained results. This may suggest that the represen- tations produced by the considered disentanglement methods are not fully factorized. It is therefore inconclusive whether the discrepancy between the obtained results is due to the shortcomings of the used methods or a manifestation of the impracticality of disentanglement.
Dataset Ground Truth
dSprites
Shapes3
MPB3D
150.235 +: 3.754
72.979 + 0.193
108.568 +: 0.285
AE VAE VALE
80,062 1: 0,341
114.939 1 0.160
150.190 +: 0.097
63.260 + 0.260
132.072 + 0.169
194.865 + 15.61
91,937 1 0.199
118.396 1 0.423
151.646 + 0.336
CONCLUSIONS
In this paper, we studied the relationship between multi-task and disentanglement representation learning. A fair evaluation of our hypothesis impossible on real-world datasets, without provided ground truth factors. To evaluate our results we had to introduce synthetic datasets that contain all necessary properties to be seen as a benchmark in this field. Next, we studied the effects of multi-task learning with hard parameter sharing on representation learning. We found that nontrivial disentanglement appears in the representations learned in a multi-task setting. Obtained factors have intuitive interpretations and correspond to the actual ground truth components. Finally, we inverted the question and investigated the hypothesis that disentangled representation is needed for multi- task learning, the results however are not conclusive. We found out that multi-task models benefit from disentanglement only on specific datasets. However, we cannot name an indicator of when this unambiguously applies.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Amir H. Abdi, Purang Abolmacsumi, and Sidney Fels. Variational learning with disentanglement- pytorch. arXiv preprint arXiv: 1912.05914 2019.
Andrzej Bedychaj, Przemystaw Spurek, Aleksandra Nowak, and Jacek Tabor. Wica: nonlinear
weighted ica, 2020.
Yoshua Bengio. Deep learning of representations: Looking forward, 2013.
Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non- linear ica, 2017.
Fabio Capela, Vincent Nouchi, Ruud Van Deursen, Igor V Tetko, and Guillaume Godin. Multitask learning on graph neural networks applied to molecular property predictions. arXiv preprint 1910vv1910. 19t.:1190..114.... 2019.
Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pp. 41-44,1488 Morgan Kaufmann, 1993.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K. Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in neural information processing systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29:2172-2180, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516 2014.
Kien Do and Truyen Tran. Theory and evaluation metries for learning represtttio..
In International Conference on Learning Representations, 2020. URL https://openneview.nee/ forum'id-HJgKOh44
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross-
lingual parameter sharing in neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 845-850, 2015.
Cian Eastwood and Christopher K. I. Williams, framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
URL http:://ppeneeiie..e///ouuuiiiii
Higgins, Loie Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Junichiro Hirayama, AJ Hyvarinen, and Motoaki Kawanabe. Splice: Fully tractable hierarchical ex- tension of ica with pooling. In Proceedings of the International Conference Machine Learning, volume 70, pp. 1491-1500. Machine Learning Research, 2017.
Jun-Ting Hsich, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to de- compose and disentangle representations for video prediction. In Advances Neural Information Processing Systems, pp. 517-526, 2018.
Guang-Bin Huang, Dian Hui Wang, and Yuan Lan. Extreme learning machines: a survey, Interna- tional journal of machine learning and cybernetics, 2(2):107-122, 2011.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances Neural Information Processing Systems, pp. 3765-3773, 2016.
AJ Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. Proceedings of Machine Learning Research, 2017.
10

---

Under review as a conference paper at ICLR 2022
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: ar2.v:180..05983, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44226690, 2014.
Diederik P Kingma and Max Welling. arXiv:1312.611, 2013.
Auto-encoding variational bayes.
arXiv preprint
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.0085: 2017.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations, 2018
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Zachary C Lipton. The mythos of model interpretability. Queue, 16(3):31-57,0011
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:/901. 2019.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In Advances Neural Informa- fion Processing Systems, pp. 14611- 1-14624, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations, 2019b.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.0/258, 2019c.
Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled rep- resentations for recommendation. In Advances in neural information processing systems, pp- 5711-5722, 2019.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4000 2016.
Janith C. Petangoda, Sergio Pascual-Dias Vincent Adam, Peter Vrancx, and Jordi Grau-Moya. Disentangled skill embeddings for reinforcement learning, 2019.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 2017.
Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representa- tions via mutual information estimation, 2019.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6):863-879, 1992.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arX1v.120..6471, 2012.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-fton networks (gin), 2020.

---

Under review as a conference paper at ICLR 2022
Przemyslaw Spurek, Aleksandra Nowak, Jacek Tabor, Lukasz Maziarka, and Stanistaw Jastrzebski. Non-linear ica based on cramer wold metric. In International Conference on Neural Information Processing, pp. 294-305. Springer, 2020.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generaliza- tion for abstract reasoning tasks using disentangled feature representations. arXiv preprint arXiv:1811.04784. 2018.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems. pp. 14245-14258, 2019.
12

---

Under review as a conference paper at ICLR 2022
A SUMMARY OF THE ARCHITECTURE OF THE MULTI-TASK MODEL
The architecture of the convolutional encoder (x)) is provided in Table 2, together with the ar- chitecture of the corresponding decoder, which used in experiments in Section 4.2. For the fully-connected heads, we used the same architecture as the one utilized during dataset creation, which is presented in Table
Table 2: The architecture of auto-encoder-based methods. Non-linearity in all layers is given by ReLU function.
Encoder
Decoder Kernel
Type
Kernel
Stride
Outputs
Тype
Stride
Outputs
Conv 2d
32
Conv 2d
256
Conv 2d
4
32
Conv Transpose 2d
4
256
Conv 2d
64
Conv Transpose 2d
128
Conv 2d
4
2
128
Conv Transpose 2d
4
2
128
Conv
256
Conv Transpose 2d
64
Conv 2d
256
Conv Transpose 2d
4
64
Dense
output.dim
Conv Transpose 2d
num num.channels
Table 3: The architecture of single fully-connected head in the single- and multi-task neural net- work. We apply non- non-lineane linearity (given by the ReLU function) after all layers except the last one.
Type
Output shape
Dense
300
Dense
300
Dense
300
Dense 10
B DISENTANGLEMENT METRICS
In our experiments, we decided to use four measures of disentanglement to comprehensively validate our results. For the convenience of the reader, this part of the appendix, we shortly describe the used measures (for wider context we encourage the reader to refer to the original papers).
B. MUTUAL INFORMATION GAP (MIG)
MIG computes the mutual information between each of the ground truth components 2, and disentangled factor žj. The mutual information between 2, and is denoted by I(2,, Next, the latent dimension with maximum mutual information score identified for each of the retrieved factor (denoted by [(2, Amax, )). along with the second-best result of the same score (denoted by [2;, Emuxg) )). The difference between those values gives a gap, which finally is normalized with respect the total mutual information associated with the studied factor:
T(zi; Emanyy I(25, Emuxz)
MIG
Σ [2,, 2,)
Where m is the dimension of ground truth components space. To report one score we average the MIG scores of all factors.
B.2 FACTORVAE METRIC
We start by normalizing retrieved factors by their respective standard deviation computed over the dataset. For a subset of the dataset, a ground truth component is then randomly selected and fixed at random value. Variance is then computed over normalized retrieved factors in this subset. Next,
13

---

Under review as a conference paper at ICLR 2022
the lowest variance factor - the that should mostly resemble the fixed ground truth component - associated with that ground truth component.
This procedure with selecting the subsets and fixing of its ground truth components is then repeated multiple times (in our experiments 10000 times). As a result, the associations between disentangled factor and ground truth component used inputs in majority vote classifier. Factor VAE metric is the mean accuracy of the classifier.
B.3 SEPARATED ATTRIBUTE PREDICTABILITY (SAP)
SAP attributes score Sij all pairs of ground truth components 2; and disentangled factors ěj. For continuous components, linear regression predicts the disentangled factors, and Sig. is the coefficient of determination (It²) of the regression. the case of categorical features, SAP fits decision tree ground truth components and reports the balanced classification accuracy. The final SAP score is achieved by computing the difference between the two highest Siy values for all factors:
SAP
S iimux: Simuxg
where n is the dimension of ground truth components space, Simax, is the highest score for compo- 2; and Simaxg the second highest for the same component.
B.4 DISENTANGLEMENT COMPLETENESS, AND INFORMATIVENES, (DCI)
Unlike previous measures, DCI is a complete framework that allows verifying several properties of the achieved representation. Disentanglement and completeness are estimated by inspecting the regressor's parameters to derive predictive importance weights Rig for each pair (zi, Zz) of ground truth 2; and retrieved ž components.
The completeness for ground truth component 2; is given by
C 1+
Σ Pig log, Pij.
where stands for ground truth dimension and Pij is the probability that disentangled factor ž is important to predict 2, These probabilities are obtained by dividing each importance weight by the sum of all importance weights related to given component:
Raj
Pij
Risk
The final compactness score is average of compactness scores all components.
Disentanglement for retrieved factor žj is given by
d
D, 1+ Σ
Pij loga Pcj
where d is the dimension of the latent space and Pij is the probability that the latent factor žÿ is important to predict only the component z, Analogously to completeness, those probabilities are normalized with respect potentially disentangled factors:
Rig
Psj
Ei=1 Rkj
The final disentanglement score is a weighted average of the individual disentanglement scores:
Σd R.j
D Σ Dg, where H
Σk E Ra
If disentangled variable is irrelevant for predicting 2j. then its μ4 (and thus contribution to the overall disentanglement) will be near zero.
Finally, the prediction error of the regressor measures the informativeness of the representation. Nor- malized inputs and outputs allow to compute the estimation error for a completely random mapping and use it to normalize the score between ( and 1.
14

---

Under review as a conference paper at ICLR 2022
C TRAINING REGIME AND EXPERIMENTAL SETUP
C.I THE MULTI-TASK MODEL EXPERIMENT 4.1
We train the multi- task model to minimize the sum of the task errors. The training is performed for 200 epochs with learning rate 0.001 and batch size 256, by using the AdaM optimizer (Kingma & Ba, 2014) with 31 = 0.9 and 32 0.999. We repeat this procedure three times, changing the random seed initialization, and report the mean and average values of the disentanglement metries.
C.2 LATENT VISUALISATIONS - EXPERIMENT 4.2
The encoder architecture was taken from the experiments in Section 4.3. The multi-task model for each experiment was randomly selected from one of the seeds from the 10 tasks setting. Addition- ally, one of the single-task encoders was selected out of the trained ones for the same seed. The random encoder was initialized by the default initialization used by the pytorch library.
The decoder architecture was optimized by minimizing the mean square error between the decoded and input image. The training was performed over 500 epochs. We used mini-batches of 64 images and gradually reduced learning rate starting from 0.0002, with a reduction of 50% every 100 epochs.
C.3 CLASSIFICATION BASED ON LATENT FACTORS - EXPERIMENT 4.3
We used the same auto-encoder and multi-task architectures like the one used in previous experi- ments (and defined in Section A), however with non-linearity given by tanh function. We trained all auto-encoders for 100 epochs, using batch size 64, learning rate 0.0001, Adam optimizer (Kingma & Ba, 2014) and latent dimension equal 8. Other hyperparameters settings were adapted from (Abdi et al., 2019). Multi-task networks were trained for 30 epochs, using batch size 64, learning rate 0.0001, and adam optimizer. In order average the scores over different runs, repeated the multi-task network training 3 times.
D VISUALISATIONS OF DECODED REPRESENTATIONS
D.1 UMAP EMBEDDINGS
In order to visualize the latent representations obtained for the random (untrained), single-task, and multi-task models we embed them into two-dimensional space by using the UMAP algorithm. The results are shown in Figure 8. It may be observed that the embeddings obtained for the multi-task representations are much more semantically meaningful. This especially evident for the dSprites and Shapes3D datasets. The MPI3D dataset is significantly more difficult problem, and although the multi-task embeddings seem to be correlated to some of the true factors, the difference is not as visible in this case.
D.2 RECONSTRUCTIONS
As described in Section 4.2, trained decoders various latent spaces produced by the encoders in the experiment from Section 4.1. 1. We provide the numerical values of the reconstruction error in Table 4 and qualitative images of the reconstructed examples in Figure 9. It may be observed that the latent representations produced by random and single task encoders do not allow the decoder to successfully restore the input examples. Moreover, the decoder trained on single-task latent even worse (in the case of reconstruction) than the random one.
Table 4: Test reconstruction enror between the decoded images and the original input images.
random
single-task
multi-task
dSprites
308.0 008.4
326.30
35,97
Shapes3D
0.044
0.082
0.008
MPI3D
0.0021
0.0061
0.0009
15

---

Under review as a conference paper at ICLR 2022
Table 5: The exact values of the metrics computed in the experiment from Section 4.1.
(a) MIG
(b) Factor VAE metric
dSprites
ShapesāD
MPL3D
dSprites
Shapes3D
MPI3D
model
model
random
0.01 + 0.01
0.02 = 0.01
0.01 # 0.00
random
0.00 0.00
0.00 + 0.00
0.00 1: 0.00
singlemman
0.01 + 0.01
0.01 0.00
0.01 0.00
single-mean
0.31 0.03
0.27 4 0.03
0.21 2: 0.02
single
max
0.04
single-min
0.01 + 0.00
0.00 0.00
0.00 + 0.00
single-min
0.26 0.01
0.23 0.00
0.18 1: 0.01
multi-head
0.04 + 0.02
0.08 0.02
0,04 + 0.02
multi-haad
0.50 0.11
0.59 0.04
0.36 0.04
0.02
(c) completeness (DCI)
(d) disentanglement (DCT)
dSprites
Sbapes3D
MPL3D
dSprites
Shapes3D
MPI3
model
model
random
0.02 + 0.01
0.03 0.01
0.05 0.01
nandem
0.02 0.01
0.03 0.01
0.05 :: 0.01
single
0.01
single-max
0.05 + 0.02
0.03 0.01
0.06 + 0.01
single-max
0.04 = 0.01
0.03 0.01
0.04 1: 0.00
single-min
0.02 + 0.00
0.01 0.00
0.02 0:0.000 0.00
singl--min
0.02 0.00
0.01 + 0.00
0.02 :: 0.00
multi-head
0.08
one-head
0.06 1:0:::00 0.02
0.05 0.02
0.05 + 0.01
one-head
0.05 0.01
0.05 0.03
0.05 0.01
(e) informativeness (DCI)
(I) SAP score
dSprites
Shapes3D
MPB3D
dSpritts
Shapes3D
MPI31>
model
mudel
+
0.01
0.43
naddom
+
0.00
single-meun
0.25 + 0.03
0.28 0.01
0.30 + 0.01
single-mean mean
0.01 0.01
0.01 + 0.00
0.01 0.00
single-max
0.30 + 0.02
0.30 0.01
0.31 + 0.01
single max
0.02 0.00
0.01 + 0.00
0.01 + 0.00
singlo-min
single-min
malli-heuu
0.41 + 0.01
0.53 0.03
0.53
0.04
multi-hcol head
0.01 0.01
0.04 0.01
0.02 0.01
one head
0.36 +: 0.01
0.44 0.03
0.47 + 0.04
one head
0.02 0.02
0.02 4: 0.01
0.01 è 0.01
D.3 TRAVERSALS LATENT SPACE
In parallel to the study of the quality of the reconstructions, have also explored the traversals in latent spaces. Given a latent representation ă of an arbitrary image we compute the traverse along each one the components ž, described in Section 4.2. This traversal represents how the image changes if only one component is slightly modified. This procedure provides visually qualitative way of assessing the level of disentangled the obtained representations.
In order to complement the discussion conducted in Section 4.2 we present here also the traversals for the Shapes3D and MPI3D datasets (in Figures 11 and 12, respectively). One may observe that the results align with the quantitative studies of disentangled metrics from Figure - where showed that the most disentangled representation obtained in the multi-task scenario. Note that the most informative changes of particular feature for given object may be observed in multi- task traversals. One may spot that object factors - although not totally disentangled - change independently from each other.
The same is not true for single-task traversals. In the example from Shapes3D dataset (Figure 1 11), observe that the single-task traversals capture only the color change of the background wall. It is also not surprising that the least informative traversal comes from the randomly initialized encoder.
E DISENTANGLEMENT AND HARD PARAMETER SHARING
In Section 4.1 we discuss the influence of hard parameter sharing on disentanglement learning. Here we present the computed metrics for all models (including regression) in a tabulated manner in Table 5. In addition, we also present the average MSE loss the test dataset in Figure 13.
F DISENTANGLED REPRESENTATION AS BASES FOR MULTI-TASK TRAINING
In Section 4.3 took the opportunity to discuss how disentanglement influences multi-task train- ing. this section, we present numerical results of all computed disentanglement metrics across trained encoders. is not surprising that FactorVAE representations most disentangled in the
16

---

Under review as a conference paper at ICLR 2022
Table 6: Numerical results of disentanglement metries for latent on which multi-task training was performed.
(a) MIG
(b) SAP score
Dataset
dSprites
Shapes31>
MPI3D
Dataset
dSprites
Shapes31D
MPI3D
AE
0.028
0.028
0,023
AE
0.006
0.020
0.009
VAE
0.117
0.041
0.011
VAE
0.032
0.020
0.017
FactorVAE
0.272
0.251
0.040
Factor VAE
0.068
0.020
0.011
(c) Factor VAE metric
(d) informativeness (DCI)
Dataset
dSprites
Shapes3I)
MPI3D
Dataset
dSprites
Shapes31D
MPI3D
AE
0.566
0.565
0.297
AE
0.395
0.493
0.473
VAE
0.710
0.564
0.323
VAE
0.579
0.533
0.484
FactorVAL
0.622
0.690
0.310
Factor
0.664
0.610
0.482
(e) disentanglement (DCI)
(f) completeness (DCI)
Dataset
dSprites
Shapes3D
MPI3D
Dataset
dSprites
Shapes3D
MPI3D
AE
0.052
0.081
0.070
AE
0.046
0.078
0.078
VAE
0,257
0.124
0.119
VAE
0.271
0.120
0.128
Factor VAE
0.356
0.342
0.079
Factor VAE
0.407
0.331
0.091
predominant number of cases. What can be read as a surprise that FactorVAB representations are never the best in terms of the root mean square error metric of the model that was trained them.
G INCREASING THE NUMBER OF TASKS
Apart from the tested in the main paper scenario with tasks, we also conducted experiments with varying number of tasks from the list of [5, 10, 20, 30, 40, 50). The results are presented in Figure 14. is impossible to draw any clear conclusions from this results, as the results vary a lot. may be observed that in some cases increasing the number of tasks up 30 leads to higher values of selected metrics, but at the same time having a negative impact on the others (for instance, Shapes3D and Factor VAE versus DCI disentanglement not consistent between datasets (consider the top row for the dSprites dataset versus the Shapes3D).
DCI completeness). These discrepancies also
H VARYING THE NUMBER OF USED GENERATING FACTORS
Apart from the presented in Section 4.3 approach which uses all of the factors generate task, we also considered scenario in which random subset of the factors is sampled for each task, and a scenario in which the tasks generated from disjoint subsets (every odd task depends only on the first half of the factors and every even task on the other half). We compare these approaches Figure 15. The computed disentanglement measures vary and the precise subset of incorporated factors in the task generating procedure does not have any conclusive impact on the final quality of the learned representation.
I NUMBER OF RETRIEVED COMPONENTS
In addition to the results presented in Section 4.3 we also compute the number of retrieved compo- nents and the mean correlation values between the retrieved components and the ground truth fac- tors in Table 7. The results are computed for the representations obtained on the test splits for each datasets used in the UMAP embedding experiment in Section 4.3. To get the number of retrieved components for each of the component of the representation with each of the ground true factor and choose the one for which the correlation is the largest and statistically significant. We next return the number of unique components matched in this way.
compute the spearman correlation
17

---

Under review as a conference paper at ICLR 2022
dataset
factors retrived
mean corr
std corr
min corr
max corr
dsprites multi dsprites single
0.385113
0.166741
0.079401
0.616765
4
0.165147
0.057721
0.085172
0.270090
shapes3d _multi shapes3d_shn _ingle mpi3d mullt mpi3d single
5
0.518252
0.307997
0.041420
0.903151
4
0.311432
0.287550
0.000000
0.793141
6
0.317056
0,142792
0.111870
0,585552
0,202390
0.111612
0.061758
0.363220
Table 7: The number of factors retrived by each method (mu 1 it for multi-task models and s ngle for a single task models) and the average/std/min and max correlation of the retrieved components with the ground truth factors.
PERFORMANCE ON SINGLE TASKS
In this section we provide the losses on all tasks Tables and for the dSprites, Shapes3D, and MPI3D datasets, respectively.
18

---

Under review as a conference paper at ICLR 2022
lossO
loss1
loss2
loss3
loss4
loss5
loss6
loss?
Loss8
loss9
total
lask
77.70
426.78
229.77
582.61
206.75
252.31
300.37
183.85
184.42
175.34
261.99
3 256.64
76.76
229.58
582.95
206.88
252.19
300.19
183.89
184.38
175.47
244.89
256.57
426.46
86.85
582.97
206.77
252.34
300.20
183.90
184.24
175.44
265.57
252.30
5 256.76
426.27
229.83
582.84
68.08
252.19
300.53
184.05
184.56
175.52
266.06
6
256.65
426.59
229.90
582.76
206.93
111.47
300.55
184.04
184.56
175.59
265.90
7
256.32
426.19
229.88
583.29
206.79
252.33
79.23
183.94
184.29
175.65
257.79
8
256.34
426.81
229.73
582.06
206.82
252.37
300.31
75.45
184.32
175.37
268.96
229.68
583.59
252.41
74.28
175.42
10
256.48
426.91
229.85 67
582.41
206.78
252.36
300.38
183.89
184.32
84.22
270.74
51.96
55.18
51.01
57.34
53.25
Table 8:
The test MSE for the experiments from Section 4.3 for dSprites dataset.
InssO
lossl
loss2
loss3
Inss4
loss5
Inss6
loss7
loss8
loss9
total_Joss
task
21.23
333.18
230.44
422.14
212.84
235.46
152.17
143.39
227.64
209.95
2
199.57
22.85
121.07
230.50
422.43
212.75
235.65
152.17
143.40
227.78
196.82
199.50
333,30
13.18
230.46
422.27
212.83
235.42
152.16
143.39
227.68
217.02
4
199.47
333.41
121.01
15.90
422.50
212.79
235.37
152.15
143.38
227.63
206.36
5
199.44
333.21
121.05
230.33
17.24
213.09
235.59
152.14
143.49
227.69
187.33
199.50
333.14
121.04 14.04
230.46
422.13
235.45
152.16
143.41
227.62
208.19
7
199.52
333.24
121.02
230.49
422.26
212.83
22.49
152.17
143.43
227.62
206.51
9 199.57
333.28
121 .000
230.45
422.24
212.87
235.47
152.15
19.97
227.70
215.47
199.57
333.27
121.04
230.39
422.54
212.91
235.44
152.14
143.41
24.79
207.55
multi 10
26.50
27.65
17.51
19.41
22.95
21.54
27.80
23.71
24.30
28.92
24.03
Table 9: The test MSE for the experiments from Section 4.3 for Shapes3D dataset.
lossO
loss
loss2
loss3
loss4
loss6
loss8
loss9
loss
task
1
132.96
293.81
237.19 19
255.46
218.69
201.70
407.40 .0
279.86
181.91
433.82
264.28
172.04
165.08
237.15
255.15
218.70
201.74
407.28
279.80
181.91
433.36
255.22
3
171.97
293.79
130.41
255.34
218.68
201.74
407.48
279.96
181.91
433.63
257.49
172.04
293.48
237
133.63
218.73
201.80
407 44
279.93
181 89
433.71
255.99
172.03
293.67
237.29
255.34
201.82
407.38
279.89
181.90
433.51
262.12
6
171.94
293.77
237.15
255.31
218.83
117.03
407.56
279.66
181.91
433.91
259.71
171.98
293.75
237.25
255.36
218.76
201.64
149.68
279.85
181.93
433.50
242.37
8
171.97
293.77
237.18
255.44
218.69
201.79
407.22
145.36
181.90
433.69
254.70
9
171.98
293.72
237.18
255.32
218.69
201.73
407.43
279.94
156.73
433.62
265.64
171.93
293.76
237.23
255.27
218.69
201.72
407.49
279.92
181.92
131
237.98
multi-10 10
77.60
102.11
76.46
82.99
93.97
71.64
91.93
87.51
89.75
80.52
85.48
Table 10: The test MSE for the experiments from Section 4.3 for MP13D dataset.
19

---

Under review as a conference paper at ICLR 2022
color
shape
scale
orientation
position
position
mult-task,
single task
rarcom
(a) dSprites
objact
scale
shape
orientation
single-task #
A
random
10
xa
d6
120
46
xto
(b) S
object hue
object shape
object size
camera height
background
horizontal axis
multi
single
random
(c) MP13D
Figure 8: The UMAP embeddings obtained for untrained, single-task, and multi-task models on different datasets (computed the test splits). The change in color corresponds change in the value selected true factor.
20

---

Under review as a conference paper at ICLR 2022
dSprites
S
MPI3D
input
random
single- task
multi-task
Figure 9: Reconstructions obtained during the experiments described in Section 4.2. The quality of the reconstruction for all datasets behaves similarly. One may easily observe that the multi-task encoder provided latent space that can be successfully decoded into images that elosely resemble the corresponding examples from the input. This is not the case in single-task or random encoders.
21

---

Under review as a conference paper at ICLR 2022
recon
Octor
foctor
Detrr
fucto
(a) Random encoder
tecon
Outorr
fector
foctx
fector
(b) Single task encoder
Insut
foctor
focto
fecto
Iectrr
(c) Multi-task encoder
Figure 10: Traverses for dSprites dataset over latent variable produced for a given architecture.
22

---

Under review as a conference paper at ICLR 2022
recon
Oetor
foctor
Dettrr
fuctor
(a) Random encoder
tecon
Outorr
foctor
foct
fector
(b) Single task encoder
Insu
factor
focto
Incter
fecto
Inctor
(c) Multi-task encoder
Figure 11: Traverses for MP13D dataset over latent variable produced for given architecture.
23

---

Under review as a conference paper at ICLR 2022
TecOn
foctor
Detre
foctor
(a) Random encoder
1154
fecto
fector
(b) Single task encoder
10000
Foctor
fxtor
foto
Iactor
(c) Multi-task encoder
Figure 12: Traverses for Sh datset over latent variable produced for given architecture.
24

---

Under review as a conference paper at ICLR 2022
dSprites
Shapes3D
MPI3D
300
300
300
2/2.55
200.150
1255.50
150
150
50
50
50
random
single
multi head
random
sing le
multi-head
one. head
random
single
multi-heal
one one-heao
one-
Figure 13: The average MSE on the test set computed for the random (untrained) model, a single- task model, and the multi-task models: multi-hess head and one-head In the single-task case, we report the mean over all models for each task. The lower the value the better. As expected, the methods which jointly optimize the tasks achieve the best results.
mig
vae
completeness
disentanglement
informativeness
0.20
0.20
0.08
deprites
0.15
0.15
0.06
0.04
0.10
0.10
0.7
0.02
0.05
0.05
0.00
0.00
0.00
0.30
0.20
0.20
0.5
0.08
Pagessa
0.15
0.15
0.06
0.04
0.10
0.10
0.2
0.02
0.05
0.05
0.10
0.20
0.20
0.5
0.08
0.15
0.15
mpi3d
0.06
0.10
0.10
0.7
0.02
muit-10
mult-20
mull-30
mult-100
mult-50
mutt-10
mult 20
mult-30
mult-100
mult-50
mut-100
mult-20
mutt-30 30
multi
mutt-50
mutt
mult
mult 30
mult-10
mult-50
mult
mutt-10
mult 20
mutt-30 30
mult
mutt-50
Figure 14: The disentanglement metrics computed for the multi-task model for different number of
tasks presented on the x-axis. Experiments for the mpi3d dataset with 40 tasks did not converge
(thus observe significantly lower values for this bar).
MIG
factor VAE
completeness
disenntanglement
informativeness m.90
SAP score
0.5
0.6
6/9
013
6.00
5.30
0.0
0.30
0.313
10.
0.30
091
0.2
0:27
00300
rundon
rencom
degord
rancom
Figure 15: The disentanglement metrics different factors splits in the multi-task seeting
25


Reviews:
[{"review_id": "PkBfyABDuQz", "reviewer": "Reviewer_ZyfG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1. The paper is well-written and easy to read.\n\n2. Authors attempted at analyzing how MTL affects the disentanglement quality, which is an interesting fundamental question that can be helpful to the community. \n\n**Weaknesses:**\n1. The most important issue with this paper is the experimental setup. It is absolutely clear that the amount of information about the generative factors encoded in the latent embedding will correlate with all of the disentanglement metrics used in this paper, regardless of whether the embedding is actually disentangled (disentanglement means that each factor is encoded by a subset of dimensions, and each dimension in the latent embedding depends only on one factor). All of the reported metrics compute an average across all generative factors, therefore, the more information is actually embedded into the latent representation, the higher are all of the disentanglement metrics. And, it is very clear, and even evident from the results illustrated in Figure 6, the MTL model encodes substantially more information about the input images than the single-task models. Therefore, the disentanglement results reported in Figure 3 are misleading, as it is impossible to compare the average disentanglement quality of the embeddings with various bandwidths. Even the illustration of embeddings in Figure 5 shows that the embeddings obtained by the multi-head model are still entangled, but it just has a different distribution.  A more conclusive metric should compute *the average number of factors* disentangled.\n\n2. Another issue with the experimental setup is the experiment with single-head vs multi-head multi-task learning. First of all, it is not discussed that the single-head model might perform the tasks significantly worse, as it has much fewer parameters than the multi-head model. Basically, the single-head model will learn a representation that gives a good task-average performance, which will also lead to the loss of some amount of information. Additionally, since the task separation happens in the last layer of the head in the single-head case, the representation obtained by the layer before the last one might be more disentangled than the one before the head.\n\n3.  The experimental setup with VAE-based embeddings for MTL is also inconclusive. The VAE-based multitask prediction quality should be compared with the vanilla hard sharing MTL model with a shared backbone that is trained from scratch on the input images. It is not clear how to interpret the results in Table 1, as it only implicitly tells us about the disentanglement quality of various VAEs, which seems out of the scope of this paper.\n\n", "summary_of_the_paper": "The paper provides an empirical analysis of the effect of multi-task learning on the disentanglement of the learned latent representation. Authors use the existing synthetic datasets for the evaluation of disentanglement and create a set of tasks by predicting label values using randomly initialized MLPs of a fixed architecture. Authors report the experimental results on three synthetic datasets, comparing a hard parameter sharing strategy with a shared backbone and separate task-specific heads with the single-task case and randomly initialized networks. Additionally, the paper includes experimental results of MTL using the latent representations obtained by the three common disentanglement methods.", "summary_of_the_review": "The paper aims to show that MTL implicitly improves the disentanglement quality of the learned representations, however, the crucial flaws in the experimental setup make all of the major results presented in this paper inconclusive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0jAPXxzkaMN", "reviewer": "Reviewer_NYHm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major concerns:\n\nI think the relationship between multi-task learning and disentanglement can be more carefully discussed. The number of learning tasks and also the relevance of different tasks could matter.\n\nThroughout the paper, the authors use ten different random tasks. Did the authors try to vary the number of multi-tasks and see how that affects disentanglement? Consider a discrete generative factor (with m values); how does the number of different random tasks (n) help with disentanglement given n<m, n=m, and n>m?\n\nAlso, in the paper, there is no assumption on the relevance among different tasks. Given three discrete generative factors z_1, z_2 and z_3, if one task is classification with respect to z_1 and z_2, while the second task is the classification for z_2, the learned representation (probably) would not show clear disentanglement property on z_3. Also, in such kind of scenario, removing the second task probably would not hurt the quality of the learned representation in terms of disentanglement.", "summary_of_the_paper": "This paper studies the relationship between disentanglement and multi-task learning (hard parameter sharing) via empirical study. The authors very carefully examined if multi-task learning encourages disentanglement. The authors performed an extensive empirical study and looked at different metrics on a couple of datasets.\n\nWhat’s more, the authors also provided synthetic datasets that can help study the relationship between feature disentanglement and multi-task learning. The authors described the right approach for such kind of dataset synthesis.", "summary_of_the_review": "After initial review, I am inclined to accept this paper; I am happy to update my score after discussing it with authors and other fellow reviewers.\n------------------------------Post-rebuttal---------------------------------\n\nI read the authors' responses and went through the authors' new experiments. I also read the reviews from other fellow reviewers and the authors' responses to them. At this moment, I am inclined to keep my score unchanged.\n\nI think each task would encourage representations to respect specific transformations and thus may implicitly encourage disentanglement. Therefore, I believe the number of transformations and their relatedness matter; this is the motivation of my questions.\n\nThe authors try to address one of my questions on the number of tasks, but the conclusion is inconclusive. Regarding the experiments on exploring the effect of the number of tasks, it's unclear why 40 tasks is always a bad choice in all three datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0YFWNwOVHej", "reviewer": "Reviewer_r7PG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper presents a reasonable evaluation setup (e.g. they include lots of different metrics, which I liked)\n- They introduce a new dataset that might be interesting to the research community. (It's perhaps a little simplistic, but it seems like a fine starting point.)\n- They show some fairly interesting findings -- most importantly, that multi-task learning seems to result in more disentangled representations (across every quantitative metric, and apparently wrt qualitative evaluations as well), but that the reverse isn't true (disentanglement doesn't clearly help with learning good multi-task representations) -- which I found fairly compelling.\n\nWeaknesses:\n- The paper makes some claims about the relationship between disentanglement and multi-task learning that I found somewhat uncompelling, or at least confusing. For example, they say that \"Intuitively, a disentangled representation encompasses all the factors of variation and as such can be used for various tasks based on the same input space\". However, I don't think of disentanglement as necessarily recovering *all* of the factors of variation. For example, presumably beta-VAE only disentangles a small fraction of the factors of variation in natural images, not all of them. So I think it's intuitive that multi-task models should \"discard\" less than single-task models, but I don't see how that intuitively connects to disentanglement. I suggest clarifying these sorts of claims.\n- I thought the traversals experiment (figure 7) was kind of weak. I think the claim is very plausible, but I think it's hard to interpret this comparison because the reconstructions for the random and single-task encoders are much messier than for the multi-task encoder, and because it's just a single qualitative example (that's plausibly cherry picked).\n- I would have liked some more realistic, large-scale, practical experiments in addition to the synthetic experiments they included in the paper (e.g. can we actually improve the disentanglement of large image generation models by doing multi-task training? What can we say about the features of \"naturally\" multi-task models, such as large pre-trained language models?)\n\nOverall I think this paper has a lot of room for improvement, but is already interesting and reasonably thorough, so overall I would lean slightly towards acceptance.", "summary_of_the_paper": "This paper studies the connection between disentanglement and multi-task learning. They present several sources of evidence that multi-task learning makes features quite a bit more disentangled. They also show some additional findings, such as that the reverse is not clearly true: make features more disentangled does not robustly help with multi-task learning.", "summary_of_the_review": "Overall the main claim of the paper is interesting and moderately well supported, so while the paper clearly has room for improvement, I would still lean slightly towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0LKpQ0n58Y", "reviewer": "Reviewer_zzap", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is quite interesting, and I think fundamental analyses of multitask learning representations like this are important within multitask learning research. The analysis goes through multiple different network architectures and tabulates different metrics, and is reasonably thorough. The visualizations are also clear and fairly convincing. \n\nMy concerns are mostly in the problem setup. Currently my rating will be a weak reject but I am happy to upgrade my score if authors can address some of these concerns satisfactorily.\n\n(1) In the vast majority of multitask settings, the output tasks are correlated. However, since the output regression tasks here are generated through multiple IID randomly initialized networks, this seems to imply no correlation between these tasks. How would stronger correlation between multitask outputs affect the results? Happy to see either theoretical or empirical analysis on this. And why do the authors think the current results would apply to real applications of multitask learning, given this discrepancy? \n\n(2) Why are the single-task network results so weak, even compared to a randomly initialized network? This seems like a weird result to me, since a random network should be just outputting noise, and yet the claim here is that the single-task network is actually performing worse in terms of the disentanglement metrics. Could authors provide loss curves for the multitask regression tasks and show that the single-task networks are at least training? I feel like the weird random vs single results might suggest the single-task network is not well tuned or the synthetic tasks are too noisy.\n\n(3) As an addendum to the above, often for multitask networks the single-task networks perform on-par or even slightly better than the multitask network on pure loss metrics. Can authors provide an explanation as to why there's such a huge discrepancy here?\n\n(4) Why did authors choose not to compare to the approach of regressing the latent variables z directly? I would expect such a network would perform quite well. In a related question, is it plausible that the improved performance with multiple tasks is more a function of noise reduction given the extra data? Have authors tried to dramatically reduce the size of the generating functions for the synthetic labels and see if the conclusions still hold?\n\n(5) Why are the randomly initialized networks still producing reasonable reconstruction results? I was under the impression that the encoder is a straightforward convnet with no skip connections but am I wrong on that?\n\n(6) Authors should greatly expand their related work section. There is a rich abundance of research out there in regards to multitask learning within a deep learning context and authors only touched the surface. In particular, how do the current disentanglement ideas relate to recent work in improving deep multitask networks?", "summary_of_the_paper": "Authors provide disentanglement analysis of multitask learning by creating a semi-synthetic dataset based on latent information in other simple datasets. Authors run the latent information through randomly initialized FC layers to create auxiliary tasks, and then train a convolutional network to produce a bottlenecked representation which is then fit to these auxiliary tasks. Authors also train various autoencoder models to see if disentangled representations help with multitask training, with results inconclusive. ", "summary_of_the_review": "The idea is interesting and I would love to see more papers try to tackle fundamental analysis of multitask models like this, but there are enough confusing and counterintuitive results that I would like to clarify things before endorsing this work. I very much look forward to the author's response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to look at the relationship between disentanglement
and multi-task learning.  The authors claim to show that disentanglement
emerges naturally from MTL.

The main discussion was whether the claim that disentanglement emerges
naturally from MTL has been adequately demonstrated.  The main  
issue is that MTL results in more extraction of information and that
is hard to disentangle from the disentanglement metrics used.

Reviewers agreed the work was interesting but not as complete as would
be desirable.  I also feel it is not ready for ICLR presentation, but  
with further work could be a nice future contribution.
Decision: Reject