Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method to train deep multi-task networks using gradient normalization. The key idea is to enforce the gradients from multi tasks balanced so that no tasks are ignored in the training. The authors also demonstrated that the technique can improve test errors over single task learning and uncertainty weighting on a large real-world dataset.

It is an interesting paper with a novel approach to multi-task learning. To improve the paper, it would be helpful to evaluate the method under various settings. My detailed comments are below.

1. Multi-task learning can have various settings. For example, we may have multiple groups of tasks, where tasks are correlated within groups but tasks in different groups are not much correlated. Also, tasks may have hierarchical correlation structures. These patterns often appear in biological datasets. I am wondering how a variety of multi-task settings can be handled by the proposed approach. It would be helpful to discuss the conditions where we can benefit from the proposed method.

2. One intuitive approach to task balancing would be to weight each task objective based on the variance of each task.  It would be helpful to add a few simple and intuitive baselines in the experiments. 

3. In Section 4, it would be great to have more in-depth simulations (e.g., multi-task learning in various settings). Also, in the bottom right panel in Figure 2, GrandNorm and equal weighting decrease test errors effectively even after 15000 steps but uncertainty weighting seems to reach a plateau. Discussions on this would be useful.

4. It would be useful to discuss the implementation of the method as well. 










review_title: Normalizing gradients for efficient multi-task network learning 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper summary:
Existing works on multi-task neural networks typically use hand-tuned weights for weighing losses across different tasks. This work proposes a dynamic weight update scheme that updates weights for different task losses during training time by making use of the loss ratios of different tasks. Experiments on two different network indicate that the proposed scheme is better than using hand-tuned weights for multi-task neural networks.


Paper Strengths:
- The proposed technique seems simple yet effective for multi-task learning.
- Experiments on two different network architectures showcasing the generality of the proposed method.


Major Weaknesses:
- The main weakness of this work is the unclear exposition of the proposed technique. Entire technique is explained in a short section-3.1 with many important details missing. There is no clear basis for the main equations 1 and 2. How does equation-2 follow from equation-1? Where is the expectation coming from? What exactly does ‘F’ refer to? There is dependency of ‘F’ on only one of sides in equations 1 and 2? More importantly, how does the gradient normalization relate to loss weight update? It is very difficult to decipher these details from the short descriptions given in the paper.
- Also, several details are missing in toy experiments. What is the task here? What are input and output distributions and what is the relation between input and output? Are they just random noises? If so, is the network learning to overfit to the data as there is no relationship between input and output? 


Minor Weaknesses:
- There are no training time comparisons between the proposed technique and the standard fixed loss learning.
- Authors claim that they operate directly on the gradients inside the network. But, as far as I understood, the authors only update loss weights in this paper. Did authors also experiment with gradient normalization in the intermediate CNN layers?
- No comparison with state-of-the-art techniques on the experimented tasks and datasets.


Clarifications:
- See the above mentioned issues with the exposition of the technique.
- In the experiments, why are the input images downsampled to 320x320?
- What does it mean by ‘unofficial dataset’ (page-4). Any references here?
- Why is 'task normalized' test-time loss as good measure for comparison between models in the toy example (Section 4)? The loss ratios depend on initial loss, which is not important for the final performance of the system.


Suggestions:
- I strongly suggest the authors to clearly explain the proposed technique to get this into a publishable state. 
- The term ’GradNorm’ seem to be not defined anywhere in the paper.


Review Summary:
Despite promising results, the proposed technique is quite unclear from the paper. With its poor exposition of the technique, it is difficult to recommend this paper for publication.
review_title: Technique seems interesting and useful. But, the exposition of the technique is poor and several important details are missing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper addresses an important problem in multitask learning. But its current form has several serious issues. 

Although I get the high-level goal of the paper, I find Sec. 3.1, which describes the technical approach, nearly incomprehensible. There are many things unclear. For example:

-  it starts with talking about multiple tasks, and then immediately talks about a "filter F", without defining what the kind of network is being addressed. 

- Also it is not clear what L_grad is. It looks like a loss, but Equation 2 seems to define it to be the difference between the gradient norm of a task and the average over all tasks. It is not clear how it is used. In particular, it is not clear how it is used to "update the task weights"

- Equation 2 seems sloppy. “j” appears as a free index on the right side, but it doesn’t appear on the left side. 

As a result, I am unable to understand how the method works exactly, and unable to judge its quality and originality.

The toy experiment is not convincing. 

- the evaluation metric is the sum of the relative losses, that is, the sum of the original losses weighted by the inverse of the initial loss of each task. This is different from the sum of the original losses, which seems to be the one used to train the “equal weight” baseline. A more fair baseline is to directly use the evaluation metric as the training loss. 
- the curves seem to have not converged.

The experiments on NYUv2 involves non-standard settings, without a good justification. So it is not clear if the proposed method can make a real difference on state of the art systems. 

And the reason that the proposed method outperforms the equal weight baseline seems to be that the method prevents overfitting on some tasks (e.g. depth). However, the method works by normalizing the norms of the gradients, which does not necessarily prevent overfitting — it can in fact magnify gradients of certain tasks and cause over-training and over-fitting. So the performance gain is likely dataset dependent, and what happens on NYU depth can be a fluke and does not necessarily generalize to other datasets. 
review_title: Lacking in clarity; experiments not convincing

Meta-review: This paper proposes a way to automatically weight different tasks in a multi-task setting.  The problem is a bit niche, and the paper had a lot of problems with clarity, as well as the motivation for the experimental setup and evaluation.
Decision: Reject