Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of neural representation learning in multi-task learning. The authors compared three types of model architectures: individual network, parallel network and task switching network, on MNIST dataset with a multitask learning setting. To be more specific, the authors conducted analyses on the representations learned from these model architectures. In the discussion, the authors pointed out that task-switching networks can encode congruency to resolve multi-tasking, and the context of tasks can improve multi-task learning differently when being used at different positions (layer depth).
strength_and_weaknesses: Strengths
- This paper studies a very meaningful research problem of understanding the representation learned for multi-task models. 
- The authors utilized different statistical tools to conduct analyses on understanding the representation learned and importance of context for different multi-task learning model architectures.

Weaknesses
- Presentation. The presentation and organization of this paper can be improved. I found it hard to connect different analyses done in section 3. It might be good to have an overview of different research hypotheses or questions being answered in each of the analyses.
 
- Novelty. The technical novelty of this paper is relatively limited. The model architectures used in this paper to conduct analysis don’t reflect recent advances in multi-task deep learning. Also the analyses being done here is similar to other existing work that studies the understanding of multi-task deep learning. I also want to see if the authors can propose some novel improved techniques based on their findings. 

- Related work. This paper doesn’t include recent advances in multi-task deep learning, either in understanding MTL or MTL model architectures, which limits its contribution and significance. For example, [1] conduct analysis on multi-task deep learning for its generalization ability related to representation learning and model capacity. Also, besides the task switching network, Mixture-of-experts [2,3] based approaches, and attention modules [4] have been widely used for multi-task learning.

[1] https://arxiv.org/abs/2005.00944
[2] https://proceedings.neurips.cc/paper/2021/hash/f5ac21cd0ef1b88e9848571aeb53551a-Abstract.html
[3] https://dl.acm.org/doi/pdf/10.1145/3219819.3220007
[4] https://arxiv.org/abs/1910.10683

- Experiment dataset. The dataset used in conducting the experiment is quite small. Multi-task learning generalization benefits can be better evaluated in different scenarios, such as: (1) tasks have similar amounts of data, (2) some tasks have much fewer data (3) few-shot tasks. These scenarios cannot be fully captured by MNIST dataset.

clarity,_quality,_novelty_and_reproducibility: The clarity and organization of this paper needs to be improved. Quality and technical novelty of this paper is relatively limited. 
summary_of_the_review: This paper conducted experiments and analyses on understanding neural representations in multi-task learning networks. There are some very interesting results and discussions. However, I think this paper can be further improved by including more recent advances in multi-task deep learning and discussion related to their findings. Meanwhile, the experiment section can be improved by including larger benchmark datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper compares three network architectures for multi-task learning. The experiment uses the MNIST dataset for two binary classification tasks. The authors categorized the output activation of each layer according to their input number to show their (un)correlation with the task. The authors also analyzed the structure of the task-switching network by changing the additional method or location, order, or strength of context vectors. Based on the results of these experiment and various analyzing methods, they present special features of task-switching networks. This paper proposes that adding context helps the network to generate expressive features and map tasks to appropriate outputs.
strength_and_weaknesses: Strengths
- Various analysis technique
- Well visualized results

Weaknesses
- Miss guided figure numbers make readers confusing. Correction is must needed, especially in the first half.
- I think tasks are too easy that switching-task might be not meaningful. As you showed in Figure 2(c), all models work well in both tasks. So Figure 2(a) has no meaning because outputs are too obvious. If you want to claim parameter sharing of networks, tasks should not be correlated, like the number of input and average value of input pixels.
- Why did you use one-hot vector as context vector? Adding not trainable one-hot vector as bias makes 'biased' networks. I want to see the activation values and its means. Dose adding 1 make too big changes in its output? If it is, the experiments about the context strengths is just come from the added bias, not the context. Using embedding vectors is reasonable choice if you care about the gap between value 1 and activation.
- In figure 5, you should show magnitude of individual network trained with one-hot vector used as bias to prove magnitude difference come from context.

clarity,_quality,_novelty_and_reproducibility: Almost of figure numbers are incorrect. I should guess the matching figures.
A lot of typos make readers confusing.
A logical leap is seen between the result and the argument.
It is hard to agree with their proposes, I can't give much novelty.
The networks dimensions, optimizing parameters, regularization techniques, and other training hyper-parameters are not mentioned. difficult to reproduce without supplementary material.

summary_of_the_review: The authors did many experiments. But experimental results and their findings is not strongly connected.
The tasks they defined are highly correlated, so it is not possible to be sure that the results came from the context or the task, which is insufficient to supporting their findings.
Too many typos make uncomfortable experience.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates the effect of task-dependent context in multi-task learning for the task-switching networks. Particularly, the authors show the task-switching networks operate in a regime between two extreme cases of individual and parallel networks. The authors also study the shared representations, the role of context and its magnitude in simple multi-task learning experimental settings. 
strength_and_weaknesses: The paper studies an important problem which I believe it is not well studied yet. However, I have the following questions and comments for improving the paper:
- For the task switching network, having separate bias terms for each task makes sense but might be too simplistic. It would be interesting to do similar studies to what you already have with more flexible transformations such as the one proposed in Sun et al., 2021. 
- minor: colorbar in Figure 2(a) seems to be incorrect (all red).
- Figure 3.1b in the text seems to refer to Figure 2.
- “switching networks, behave similarly as parallel in the first tasks” it should be in the first “layers”. Also the section is missing in “After observing the impact of context in the first layer (section ), “. Generally, the paper can benefit from some careful proofreading. 
- The title for section 3.4 might be misleading as Figure 5.C doesn’t support that. Also I’m surprised that having a single layer with context is doing so well in terms of accuracy. This may be a hint that the task is too simple for an interesting analysis. A dataset with more diverse images may help in this experiment. 
- How does the analysis change if done on a more challenging dataset like CIFAR-10 and also with a slightly more complicated network architecture like a CNN. 
- I found some results to be interesting but it’s unclear how they can be used for designing better context-dependent multi-task learning algorithms. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear but can benefit from proofreading. The quality of the analysis can be further improved as suggested above. 
summary_of_the_review: Overall, I found the exploratory analysis of context dependent multi-task learning an important analysis to be done. However, I believe the paper needs some work to either show the results hold in more general settings and difficult tasks, or provides some guideline on how to utilize the insights for improving multi-task learning algorithms.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates neural network representations under multi-task learning based on five binary classifications tasks constructed from MNIST.  The authors compare using independent, individual multi-layer perceptrons (MLPs) per task with parallel MLPs that process tasks simultaneously and with task-switching MLPs that share weights between the individually processed tasks. They show that task-switching networks learn disentangled shared representations that generalize between tasks and encode congruency.
strength_and_weaknesses: The work investigates an important domain, is improving the understanding of neural network representations under multi-task learning may be helpful for improving multi-task learning and explaining model predictions. It remains however unconvincing as 1) the authors do not verify if their results hold across multiple runs with different random initializations of the networks or are only random artifacts 2) MNIST is a narrow and small dataset, findings based which may not transfer to state-of-the-art applications of neural networks for example in computer vision or natural language processing which were found to benefit from multi-task learning.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, though the authors make undue allusions between their work and multi-task handling in the human brain. 
summary_of_the_review: The work is interesting but the limitations in choice of networks and datasets make it not strong enough for ICLR. Follow-up work that addresses the limitations will help to qualify the paper,
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper analyzes neural multi-task learning from a representation perspective. The authors compare three types of networks: individual (separate weights for each task), parallel networks (all weights but output layer shared), and task-switching networks (all weights but context vectors shared) using analyses such as RSA (Representational Similarity Analysis). They show that task-switching networks can encode congruency to resolve multi-tasking, and task context can improve multi-task learning when provided at different layers.

Strengths: Clearly this is an interesting and important problem and the authors have provided some insight in their analysis. The analyses presented are novel in this context as far as I can tell.

Weaknesses: Aside from typos and clarity which I assume the authors have addressed in their revision, there were two main weaknesses brought up: (1) lack of engagement with the large body of MTL literature developed since Caruana (see below for another reference); (2) too narrow and descriptive an analysis of too simple a model.

I understand the author's perspective on starting simple where it's easier to tease out and control the relevant factors. But the conclusion, for example, that weight sharing improves transfer, has been widely discussed and analyzed elsewhere and felt a bit unsurprising to the reviewers. The analysis is interesting but perhaps lacks some generality and is a bit narrowly focused on this particular setting.

Another reference https://arxiv.org/pdf/1711.01239.pdf.  

justification_for_why_not_higher_score: The paper got straight 3's and was unconvincing to any of the reviewers
Decision: Reject