Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the relationship between disentanglement and multi-task learning (hard parameter sharing) via empirical study. The authors very carefully examined if multi-task learning encourages disentanglement. The authors performed an extensive empirical study and looked at different metrics on a couple of datasets.

What’s more, the authors also provided synthetic datasets that can help study the relationship between feature disentanglement and multi-task learning. The authors described the right approach for such kind of dataset synthesis.
main_review: Major concerns:

I think the relationship between multi-task learning and disentanglement can be more carefully discussed. The number of learning tasks and also the relevance of different tasks could matter.

Throughout the paper, the authors use ten different random tasks. Did the authors try to vary the number of multi-tasks and see how that affects disentanglement? Consider a discrete generative factor (with m values); how does the number of different random tasks (n) help with disentanglement given n<m, n=m, and n>m?

Also, in the paper, there is no assumption on the relevance among different tasks. Given three discrete generative factors z_1, z_2 and z_3, if one task is classification with respect to z_1 and z_2, while the second task is the classification for z_2, the learned representation (probably) would not show clear disentanglement property on z_3. Also, in such kind of scenario, removing the second task probably would not hurt the quality of the learned representation in terms of disentanglement.
summary_of_the_review: After initial review, I am inclined to accept this paper; I am happy to update my score after discussing it with authors and other fellow reviewers.
------------------------------Post-rebuttal---------------------------------

I read the authors' responses and went through the authors' new experiments. I also read the reviews from other fellow reviewers and the authors' responses to them. At this moment, I am inclined to keep my score unchanged.

I think each task would encourage representations to respect specific transformations and thus may implicitly encourage disentanglement. Therefore, I believe the number of transformations and their relatedness matter; this is the motivation of my questions.

The authors try to address one of my questions on the number of tasks, but the conclusion is inconclusive. Regarding the experiments on exploring the effect of the number of tasks, it's unclear why 40 tasks is always a bad choice in all three datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors provide disentanglement analysis of multitask learning by creating a semi-synthetic dataset based on latent information in other simple datasets. Authors run the latent information through randomly initialized FC layers to create auxiliary tasks, and then train a convolutional network to produce a bottlenecked representation which is then fit to these auxiliary tasks. Authors also train various autoencoder models to see if disentangled representations help with multitask training, with results inconclusive. 
main_review: The work is quite interesting, and I think fundamental analyses of multitask learning representations like this are important within multitask learning research. The analysis goes through multiple different network architectures and tabulates different metrics, and is reasonably thorough. The visualizations are also clear and fairly convincing. 

My concerns are mostly in the problem setup. Currently my rating will be a weak reject but I am happy to upgrade my score if authors can address some of these concerns satisfactorily.

(1) In the vast majority of multitask settings, the output tasks are correlated. However, since the output regression tasks here are generated through multiple IID randomly initialized networks, this seems to imply no correlation between these tasks. How would stronger correlation between multitask outputs affect the results? Happy to see either theoretical or empirical analysis on this. And why do the authors think the current results would apply to real applications of multitask learning, given this discrepancy? 

(2) Why are the single-task network results so weak, even compared to a randomly initialized network? This seems like a weird result to me, since a random network should be just outputting noise, and yet the claim here is that the single-task network is actually performing worse in terms of the disentanglement metrics. Could authors provide loss curves for the multitask regression tasks and show that the single-task networks are at least training? I feel like the weird random vs single results might suggest the single-task network is not well tuned or the synthetic tasks are too noisy.

(3) As an addendum to the above, often for multitask networks the single-task networks perform on-par or even slightly better than the multitask network on pure loss metrics. Can authors provide an explanation as to why there's such a huge discrepancy here?

(4) Why did authors choose not to compare to the approach of regressing the latent variables z directly? I would expect such a network would perform quite well. In a related question, is it plausible that the improved performance with multiple tasks is more a function of noise reduction given the extra data? Have authors tried to dramatically reduce the size of the generating functions for the synthetic labels and see if the conclusions still hold?

(5) Why are the randomly initialized networks still producing reasonable reconstruction results? I was under the impression that the encoder is a straightforward convnet with no skip connections but am I wrong on that?

(6) Authors should greatly expand their related work section. There is a rich abundance of research out there in regards to multitask learning within a deep learning context and authors only touched the surface. In particular, how do the current disentanglement ideas relate to recent work in improving deep multitask networks?
summary_of_the_review: The idea is interesting and I would love to see more papers try to tackle fundamental analysis of multitask models like this, but there are enough confusing and counterintuitive results that I would like to clarify things before endorsing this work. I very much look forward to the author's response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the connection between disentanglement and multi-task learning. They present several sources of evidence that multi-task learning makes features quite a bit more disentangled. They also show some additional findings, such as that the reverse is not clearly true: make features more disentangled does not robustly help with multi-task learning.
main_review: Strengths:
- The paper presents a reasonable evaluation setup (e.g. they include lots of different metrics, which I liked)
- They introduce a new dataset that might be interesting to the research community. (It's perhaps a little simplistic, but it seems like a fine starting point.)
- They show some fairly interesting findings -- most importantly, that multi-task learning seems to result in more disentangled representations (across every quantitative metric, and apparently wrt qualitative evaluations as well), but that the reverse isn't true (disentanglement doesn't clearly help with learning good multi-task representations) -- which I found fairly compelling.

Weaknesses:
- The paper makes some claims about the relationship between disentanglement and multi-task learning that I found somewhat uncompelling, or at least confusing. For example, they say that "Intuitively, a disentangled representation encompasses all the factors of variation and as such can be used for various tasks based on the same input space". However, I don't think of disentanglement as necessarily recovering *all* of the factors of variation. For example, presumably beta-VAE only disentangles a small fraction of the factors of variation in natural images, not all of them. So I think it's intuitive that multi-task models should "discard" less than single-task models, but I don't see how that intuitively connects to disentanglement. I suggest clarifying these sorts of claims.
- I thought the traversals experiment (figure 7) was kind of weak. I think the claim is very plausible, but I think it's hard to interpret this comparison because the reconstructions for the random and single-task encoders are much messier than for the multi-task encoder, and because it's just a single qualitative example (that's plausibly cherry picked).
- I would have liked some more realistic, large-scale, practical experiments in addition to the synthetic experiments they included in the paper (e.g. can we actually improve the disentanglement of large image generation models by doing multi-task training? What can we say about the features of "naturally" multi-task models, such as large pre-trained language models?)

Overall I think this paper has a lot of room for improvement, but is already interesting and reasonably thorough, so overall I would lean slightly towards acceptance.
summary_of_the_review: Overall the main claim of the paper is interesting and moderately well supported, so while the paper clearly has room for improvement, I would still lean slightly towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides an empirical analysis of the effect of multi-task learning on the disentanglement of the learned latent representation. Authors use the existing synthetic datasets for the evaluation of disentanglement and create a set of tasks by predicting label values using randomly initialized MLPs of a fixed architecture. Authors report the experimental results on three synthetic datasets, comparing a hard parameter sharing strategy with a shared backbone and separate task-specific heads with the single-task case and randomly initialized networks. Additionally, the paper includes experimental results of MTL using the latent representations obtained by the three common disentanglement methods.
main_review: **Strengths:**
1. The paper is well-written and easy to read.

2. Authors attempted at analyzing how MTL affects the disentanglement quality, which is an interesting fundamental question that can be helpful to the community. 

**Weaknesses:**
1. The most important issue with this paper is the experimental setup. It is absolutely clear that the amount of information about the generative factors encoded in the latent embedding will correlate with all of the disentanglement metrics used in this paper, regardless of whether the embedding is actually disentangled (disentanglement means that each factor is encoded by a subset of dimensions, and each dimension in the latent embedding depends only on one factor). All of the reported metrics compute an average across all generative factors, therefore, the more information is actually embedded into the latent representation, the higher are all of the disentanglement metrics. And, it is very clear, and even evident from the results illustrated in Figure 6, the MTL model encodes substantially more information about the input images than the single-task models. Therefore, the disentanglement results reported in Figure 3 are misleading, as it is impossible to compare the average disentanglement quality of the embeddings with various bandwidths. Even the illustration of embeddings in Figure 5 shows that the embeddings obtained by the multi-head model are still entangled, but it just has a different distribution.  A more conclusive metric should compute *the average number of factors* disentangled.

2. Another issue with the experimental setup is the experiment with single-head vs multi-head multi-task learning. First of all, it is not discussed that the single-head model might perform the tasks significantly worse, as it has much fewer parameters than the multi-head model. Basically, the single-head model will learn a representation that gives a good task-average performance, which will also lead to the loss of some amount of information. Additionally, since the task separation happens in the last layer of the head in the single-head case, the representation obtained by the layer before the last one might be more disentangled than the one before the head.

3.  The experimental setup with VAE-based embeddings for MTL is also inconclusive. The VAE-based multitask prediction quality should be compared with the vanilla hard sharing MTL model with a shared backbone that is trained from scratch on the input images. It is not clear how to interpret the results in Table 1, as it only implicitly tells us about the disentanglement quality of various VAEs, which seems out of the scope of this paper.


summary_of_the_review: The paper aims to show that MTL implicitly improves the disentanglement quality of the learned representations, however, the crucial flaws in the experimental setup make all of the major results presented in this paper inconclusive.
review_title: null

Meta-review: This paper aims to look at the relationship between disentanglement
and multi-task learning.  The authors claim to show that disentanglement
emerges naturally from MTL.

The main discussion was whether the claim that disentanglement emerges
naturally from MTL has been adequately demonstrated.  The main  
issue is that MTL results in more extraction of information and that
is hard to disentangle from the disentanglement metrics used.

Reviewers agreed the work was interesting but not as complete as would
be desirable.  I also feel it is not ready for ICLR presentation, but  
with further work could be a nice future contribution.
Decision: Reject