Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a new perspective on solving the conflicting gradients problem in MTL. It checks the conflicting level at layer scale, and turns the layers with high conflict scores into task-specific ones. Some theoretical analysis and experimental verifications are provided. 
strength_and_weaknesses: Strengths:

+ The perspective on identifying conflicting gradients at layer scale is interesting. The proposed method that turn layers with high conflicts to task-specific ones seems to work well.
+ The overall structure of this paper is easy to follow.

Weaknesses:
+ I am not convinced by the claim that "gradient manipulation can not reduce the occurrence of conflicting gradients". In my practice, I find these techniques can alleviate the conflicting gradients. Your results are only based on one dataset. It's surely not enough. Moreover, how do you calculate the conflicting angles? I think Figure 1 and Section 3.3 need more in-depth explanations. Otherwise, this claim seems to be exaggerated.
+ The key algorithmic design "turn the layers into task-specific ones" is hard to figure what its exact meaning. I struggled to understand this point very concretely. Some figures with network structures may help better illustrate this clearly. 
+ The applicability of this method to other commonly used MTL structure such as MMoE is not provided.
clarity,_quality,_novelty_and_reproducibility: Clarity: Generally, this paper is easy to follow. But the key part of the algorithmic design is not clear. Readers will find it hard to understand what are the precise actions of "turn the layers into task-specific ones".

Quality and Novelty: Fairly good

Reproducibility: The clarity affects the reproducibility to some extent.
summary_of_the_review: The paper is in general interesting. But in the current form, some parts of this paper are questionable or not clear. The authors need to clarify two main points raised in the review. 

(1) The claim on the effects  of gradient manipulation to conflicting gradients. 

(2) What does "set these layers task-specific" mean?

(3) More experiments or discussions are needed to see whether the proposed applies to typical MTL structures such as MMoE.

If the concerns are well addressed, I will raise my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to mitigate the negative transfer, i.e., the conflicting gradients, of different tasks on the shared layers. After showing that the gradient surgery methods cannot reduce the occurrence of conflict by investigating the gradient angle distributions of different tasks, the authors propose to split those layers with the most conflicting gradients. 

Specifically, the authors train the network for some initial steps to calculate an S score to indicate the gradient conflict, where for a specific layer, its S score is the number of tasks whose gradient angle is larger than a threshold. Then, the shared layer with a higher S score (i.e., higher gradients conflict) is split for T times for each of T tasks. Finally, the new network with split task-specific layers is continued to train until convergence. Theoretical analysis is also provided to guarantee that applying the proposed Recon reduces the training loss.

The experiments are conducted on Multi-Fashion+MNIST, CityScapes, NYUv2, and PASCAL-Context with ablations validating the promising performance.
strength_and_weaknesses: Strength:

1. Important problem with the pilot study probing existing issues and the concise solution addressing them.
2. The theoretical analysis is also carried out that guarantees a lower training loss.
3. This paper is well-organized and easy to follow.

Weaknesses:

1. In order to avoid , there is another category of MTL methods aims to avoid the negative transfer/gradient conflict by designing proper network architectures. For example, [1-4] perform layerwise feature fusing between tasks, which do not consider splitting from a shared network, but starting with several individual networks to merge (fuse features). Can the authors discuss, and preferably compare with, those methods?
2. Would it help to control the starting iteration step when calculating (i.e., summing up) the s-scores? In other words, currently s^(k) = \sum_{i=1}^{I} s_i^(k), would it help if i does not start with 1 as the s-scores of the very initial iterations can be noisy?
3. This could be a minor issue: there exists a much larger MTL dataset, i.e., Taskonomy [5], to validate the proposed method.

[1] Misra et al., Cross-stitch networks for multi-task learning. CVPR 2016.

[2] Gao et al., NDDR-CNN: Layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction. CVPR 2019.

[3] Ruder et al., Latent multi-task architecture learning. AAAI 2019.

[4] Gao et al., MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning. CVPR 2020.

[5] Zamir et al., Taskonomy: Disentangling task transfer learning. CVPR 2019.
clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is clearly written with good organization.

Quality, Novelty: this paper probed an important issue of the existing methods and proposed original solutions.

Reproducibility: though the proposed method is fairly concise and should be easy to implement, the reproducibility still depends on if the authors release the codes.
summary_of_the_review: Overall I think this is a good paper with clear (empirical) problem analysis and concise solution. For those further possible improvements, please see the Weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task learning method to reduce conflict gradients during multi-task optimisation. The proposed method, name Recon is very simple in design and intuitive – it first computes the cosine similarity score for all combinations of task pairs in each shared layer, and sets this shared layer to be task-specific if its cosine similarity score is negative for multiple task pairs. As such, the proposed method can be combined with any gradient-based multi-task optimisation methods and can be applied on any multi-task architecture. The paper verifies this design on a range of multi-task benchmarks with different architectures and different optimisation methods.
strength_and_weaknesses: Strength. 
+ The paper is very well written and clearly motivated, supported with a wide range of experiments.
+ The proposed method is general in design and can be attached to any optimisation methods and any architecture.
+  The proposed method can achieve smaller loss supported with theoretical analysis.

Limitations.
+ Scalability and Running Time. Clearly, one inevitable problem for the proposed method is to compute cosine similarity for all combination of task pairs which might be unscalable when the number of training tasks is large. Though I agree that the newly initialised task-specific parameters might be small compared to the total network parameters, but the running time might be much slower. It would be clearer if the authors could the list the increased running time for such method and how it scales with increasing number of tasks. One paragraph that related to this calming that “requires extra 25% of training iterations” is clearly not sufficient and definitely not fast considering the fact that we only 2 tasks.

+ Conflict from Architecture Design. It’s very interesting to see that the authors found out that the conflict layers are consistent across multiple methods. This observation makes me wonder that the conflict itself was mainly due to the multi-task architecture design. As such, I am wondering whether these conflicts would reduce if we re-train the model from scratch after we set these conflict layers to be task-specific. If so, the running time would not be a problem since we only need to run this method once.  Just to be clear, the consistency is from different methods on the same dataset or different methods on different datasets as well?
+ Severity criteria. Is this hyper-parameter needs to be tuned for each dataset as the paper lists the percentage of increased parameters vary quite a lot from different datasets, which will make the method a bit less general.
+ Minor visualisation suggestion. Please add the attached method before (w/ Recon) or have a midrule to separate other methods. This would make the comparison much clearer.

clarity,_quality,_novelty_and_reproducibility: The paper is very clear and should be easy to reproduce.
summary_of_the_review: The paper proposes a simple multi-task optimisation method: to make task-shared layers to be task-specific if the gradient conflict is severe. The paper in general is very well written with comprehensive empirical and theoretical support.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles conflicting gradients in multi-task learning (MTL). Specifically, the idea is to identify the shared layers of the multi-task model that exhibit large degrees of conflicting gradients, and replace these layers with task-specific parameters. The reported experiments suggest that only a small number of additional task parameters are necessary to improve performance, and reduce the occurrence of conflicting gradients in the remaining shared layers.

The experiments cover four multi-task benchmarks: multi-fashion+MNIST, CityScapes, NYUv2, and PASCAL-Context. The main finding is that the proposed “Recon” MLT approach improves the performance of all base models. Notably, the performance increase can be obtained with only a small increase in the total number of parameters (1.42%, 12.77%, 0.26%, and 9.8%, respectively for each benchmark).

Further analysis is provided in the way of an ablation study, showing that Recon significantly reduces the occurrence of conflicting gradients. Additionally, a theoretical analysis is presented, showing that a single gradient update on the model parameters of Recon achieves lower loss than on the original model parameters.

strength_and_weaknesses: While the approach is straightforward, the simplicity and effectiveness of the proposed Recon method are appealing.

A potential concern with resolving conflicting gradients at training time is that they may, in some cases, provide a regularisation benefit that leads to positive transfer on held-out data. I would have liked to see more discussion on this issue.

The experiments are limited in two respects: (1) they focus only on vision problems; and (2) three out of four multi-task settings involve semantic segmentation. I would have liked to see more diversity of benchmarks as well as benchmarks with larger numbers of tasks.

A downside of the proposed approach is that the resulting model is not actually multi-task anymore, as it contains task-specific weights, which could be problematic in some settings for deployment.

Finally, the hyperparameters (S, K) appear to be quite sensitive, which increases the cost of the approach as they require fine-tuning for the different domains / datasets.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and proposes a novel approach. Although the approach is simple, reproducibility would be easier if code can be shared. 

Definition 1 has a typo: “angel” -> angle

summary_of_the_review: Overall, this paper describes a simple idea to improve MTL by introducing a relatively small number of task-specific parameters, in order to reduce the impact of gradient conflict. The experiments are fairly comprehensive.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method to improve the performance of multi-task learning methods, by reducing the conflict amongst gradients for different tasks. The key findings are that naive "gradient surgery" may not effectively solve this problem, while a simple per-layer conflict resolution is effective.

Three reviewers found the paper to be interesting, with a simple idea shown to be effective on a broad range of empirical settings. The careful study of why existing strategies may fail was also found to be illuminating, and the overall writing was seen to be of good quality. One reviewer raised some questions regarding the ineffectiveness of gradient manipulation, and the potential applicability to MMoE. The author response on these points is, in my reading, satisfactory. Thus, I believe the paper is suitable for publication, and would be of interest to the community.
justification_for_why_not_higher_score: Paper is not quite of an award quality, but the scores do indicate broad consensus that the idea is interesting and well-executed.
justification_for_why_not_lower_score: Reviewers were generally positive, and critical reviewer's comments appear to be adequately addressed. From my reading the paper is indeed of a good quality.
Decision: Accept: poster