Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a method for grouping tasks to achieve stronger multi-task learning (MTL) performance. The approach is relatively simple and quite efficient in comparison to alternatives: during a single pass of learning, measure the impact that one task's gradient updates would have on other tasks. A theoretical analysis is conducted in a simplified (strongly convex) setting, and experiments show that this approach is as effective as alternatives that are much more computationally intensive. Overall this appears to be a useful technique that could be useful for many other researchers and also potentially for product teams.

main_review: Strengths:
1. The approach is relatively simple, easy to understand, and (relatively) easy to implement.
2. The approach appears highly effective in the setting where the number of parameters total (across all models) is not constrained.
3. The experiments are conducted well, and the results are convincing.
 
Weaknesses:
1. While the paper makes an argument on the basis of deployed systems that it is reasonable to allow the number of parameters to grow as a function of the number of task groupings, this does introduce a confounding effect in analyzing the results. It also was unclear to me whether, for the comparison systems, if they were also allowed to have increasing numbers of parameters for the number of groups.
2. The assumption of convexity in the theoretical analysis, while probably the only way to make the result go through, is fairly unsatisfying in MTL settings, especially when the whole motivation is to learn good shared representations.



I only have a few additional questions/comments:

 - I would avoid the term "co-training" -- this is its own field of ML, quite different from MTL, and overloading the term is confusing.

 - I was confused throughout much of the paper about why the paper claims there are 2^T-1 possible groupings; I thought it should be B_T (the T-th Bell number). It turns out the answer is because tasks can contribute to the training of networks that do not make predictions for that task. This should be made clearer much sooner (eg around line 38).
 
 - l.103 "combined task performance" -- I assume this means "average task performance" (later text confirms this) but it would be good to be specific.
 
 - I find the terminology "inference-time budget" confusing because it's not clear what it's a budget over. I would call it "parameter budget" or "memory budget" or something like that.
 
 - l.107 I don't understand what "are not served" means.
 
 - In the theoretical analysis (4.3) it wasn't clear to me how this interacts with the affinity averaging described around line 168. It would be nice if this were clarified.
 
 - I'm very confused by Figure 2. My understanding is that what is plotted here is the runtime required to find groups (this is also what the caption says). But then why is STL 8.5x and 3.8x, respectively? There is no time required to find groups in STL. Similarly, why does "Full MTL" have a non-zero amount of time to "find groups"? With those as baselines, I would expect everything else to be infinitely longer. I'm clearly not understanding what is being plotted here, but l.237 syas "we report the time to determine task groupings, not determine task groupings and train the resultant multi-task networks."
 
 - I'd ideally like to see figures showing the full training time (including groupings and training) for all results. The proof of the theorem can be moved to the appendix to make space for this. I understand that the point of this paper is a method to find good groupings, but given that the paper stresses the practical relevance of this approach, what practitioners will really care about is the full time.
 
 - l.246 I don't understand the discussion of 2-splits, 3-splits, 4-splits here. I assume that this means that you're allowed to have 2* as many parameters, 3*, and 4*? Is that right? Please clarify.
 
 - Table 3 body and caption says "b=128x" but the text on l.346 and l.355 says "b=0.5x". Which is it?


limitations_and_societal_impact: Limitations are not discussed. Societal impact is reasonable.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Authors propose a single run training method to find out what group of tasks should be learnt together based on the network capacity and computational budget. They evaluate TAG on CelebA and Taskonomy datasets.
main_review: The paper is well written and easy to understand. 

The proposed method solves the task grouping problem very efficiently, which is a very challenging topic in MTL. Previous methods require training many networks to determine the task group while this method only requires a single run, which is very innovative and exciting. I like the method. 

The method accumulates training gradients of different tasks during the single run and computes the task affinity, which sheds some light on the challenging task affinity computation. It is also very insightful in transfer learning.

In experiment, they evaluate on CelebA and Taskonomy datasets. TAG yields the best relative runtime and total test loss in both scenarios. They also carefully analyze the drawback of their method in Sec5.2 and also carefully discuss the frequency of computing the gradient and which data split should be used. 

For Rebuttal,
It would be very interesting that the authors could provide some grouping examples, and show the difference when it is 2 groups, 3 groups and 4 groups. Task affinity matrix will also demonstrate the effectiveness of the method.


limitations_and_societal_impact: Yes, they have mentioned and analyzed very carefully in sec 5.2
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Given a large set of tasks, the goal of this work is to identify subsets of these tasks that can be trained together to improve performance and avoid harmful interference. In contrast to multi-task research that focuses on building a model that handles _all_ tasks as well as possible, this work looks at the cues acquired from training such a model to quickly assess which tasks to group together. This is done by seeing whether a task's gradient update helps or hurts the losses of other tasks. 
main_review: Main thoughts:

- (+) the proposed strategy for measuring task affinity is well justified and straightforward - would be easy enough to implement and replicate. From a first impression it is unclear that it would necessarily correlate with subsequent multi-task performance, but the experiments back up its utility as a grouping signal
- (+) the case made for efficiency (that this method only requires a single training run of a model with all tasks) is also a strong advantage relative to the cost of other ways to find task groupings
- (+) clear and well written work with thoughtful analysis throughout paper and in ablations
- (-) I do not find 9 attributes on CelebA the most compelling multi-task benchmark (though I understand CelebA is a go-to setting in the multi-task literature). The difference in test error is marginal across these methods and facial attribute classification is such a restricted setting. I think it would be better to see results on diverse multi-domain multi-task settings like Visual Decathlon or VTAB. There is extra overhead in using the proposed method in those settings since all task losses cannot be evaluated in a single inference pass, but seems as though it would still be useful.

I lean positive for this paper, I think that the authors propose a nice, simple strategy for grouping tasks in multi-task training and provide a solid set of experiments and analysis to justify its use.

Additional comments:

- There's an ablation looking at task affinity at different stages of training, the dynamics here seem like they'd be interesting to have a clearer picture of. How does affinity typically change over the course of training, what does it look like for the tasks that match up best, are there different patterns for different pairs of tasks (moving from high -> low affinity or vice versa)
- Are task affinities robust to the presence of other sets of tasks. Say I trained a model with Tasks A, B, C, D and a model with Tasks B, C, E, F. Would the affinity between B + C be similar in both cases?
- Is it possible to see the distribution in performance of random groupings? Were the optimal and worst values derived from an exhaustive search or were they the best/worst to appear in the random sampling?

Typo L224 "hyperparamters", L263 "Segmentic Segmentation"


-----
**Update:** After having read the other reviews and author responses I maintain my original recommendation to accept. I really appreciate the effort in the response along with the attached figures and details that addressed my questions.
limitations_and_societal_impact: I found the "Broader Impact" statement solid, much better than what I'm used to coming across in submissions
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a task grouping approach for multi-task learning performed in a single training step. It measures an inter-task affinity by quantifying the effect of a task’s gradient on other tasks. This work also shows a theoretical analysis of the task grouping by the inter-task affinity under some conditions. Experimental results show that the presented approach gives competitive results compared to other multi-task learning counterparts using two benchmark datasets.
main_review: This work focuses on suggesting an inter-class affinity measure rather than proposing other strategies for multi-task learning such as analyzing dataset characteristics, model architectures, etc. Since this work assumes learning tasks in a single architecture (hard-parameter sharing), identifying similar tasks and grouping them are important tasks to improve learning efficiency. It presents a simple approach to identify task groups based on the response of gradient updates in the shared parameters. This strategy leads to comparable performance for two multi-task learning scenarios.
limitations_and_societal_impact: The idea of measuring inter-task affinity is simple and the technical novelty is limited. The pairwise affinity is measured by comparing the losses before and after applying the gradient update in the shared body of the network. Since it computes all the pairwise task affinities, I think it takes heavy complexity.

It is unclear and not well described how to split tasks into several groups using the affinity. Based on the score between L152 and L153, how to group tasks, and in L170, how to set k in the method? 

The theoretical analysis seems not practical because it assumes a convex setting with a smooth loss function. If using popular deep neural networks, it is questionable the analysis can still be valid.

In L242, how the 9 attributes were selected out of 40 attributes in CelebA?

In Figure 2, I wonder how to collect the results (error, runtime) of the compared methods and whether they are collected under the same experimental setup.

In Table 2, it is unclear what the value means. An exact measure for the relative performance should be provided to reproduce.

In Table 3, it is insufficient to show another set of hyperparameters. A control experiment with respect to multiple different values of hyperparameters (batch size, learning rate) should be provided to show the stability of the proposal.

--

Update:
I have read the authors' responses and other reviewers' comments. 
Many of my concerns have been addressed, but reducing the expensive cost for computing pairwise affinity is rather heuristic which is performed by a simple trick, not from the algorithm itself. It would be better to address this point to make the paper stronger.
By considering other reviewers' comments and their views on the work, I agree with them, and my comment is not much deviated from others' opinions. The proposed method can be helpful for practitioners due to its simplicity and promising results.
So, I raise my rating and wouldn't mind if the paper gets accepted.
review_title: null

Meta-review: This paper proposes a method for grouping tasks to achieve stronger multi-task learning performance.  All four reviewers believe this is a well-written paper and should get accepted.   Authors need to modify their final version based on reviewers' comments.  
Decision: Accept (Spotlight)