Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper applies the Gumbel-softmax to optimizing task-specific routing in deep multi-task learning. Experiments demonstrate improvements of the method over no sharing or full sharing, and it is used to achieve s-o-t-a results in the Omniglot MTL benchmark.

Although the end results are good, and the approach is well-motivated, I am leaning to reject, because the experiments have not made clear when the method works and how it behaves. The improvements over the full-sharing baselines appear fairly small, and in the analysis it appears the model is mainly discarding unneeded pooling layers. Is there some real task-specific routing that the method is able to take advantage of? Maybe an experiment where full-sharing is detrimental, i.e., because there are some highly unrelated tasks, would help to highlight how the approach selects appropriate module subsets for each task. E.g., what are the routing patterns in Section 6.1 that are the same within each pair of MNIST tasks, but different across task pairs? Is there a way to visualize differences between routing of different Omniglot tasks?

Similarly, the experiment in Section 2 is interesting, but the conclusion that negative transfer exists is not novel. Is there a way to include the Gumbel approach in these synthetic experiments to show that it addresses this issue? E.g., something like the result in A.3 could be promoted to Section 2. More compelling synthetic datasets could be generated by the method in A.1. for the case where tasks are somewhat related, in which case we can actually see if how the sharing occurs. Could Gumbel see a bigger boost in these synthetic experiments if training data were limited and generalization was tested instead of training loss? 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In many ways this work is well presented. However, I have major concerns regarding the novelty of the proposed method and the theoretical rationale for the key design choices. Although the authors do cite and discuss (Rosenbaum et al., 2019), what is very much not clear to me is how the Gumbel-Matrix Routing proposed in this work differs from past work using the Gumbel Softmax within routing networks. It seems like past work even focused on using only the task for routing, so it is not clear to me how the approach here is really novel in comparison. Even if there is some distinction I am missing, the high level idea is clearly not that new. Additionally, there is not much theoretical discussion about what the Gumbel Softmax adds to routing networks. 

The bias/variance tradeoff of Gumbel Softmax / RELAX / REINFORCE was already highlighted in (Rosenbaum et al., 2019). Can the performance of the model on the settings tested be attributed to this tradeoff? If so, would a RELAX model perform even better? Moreover, there is not much discussion of important implications of using the Gumbel Softmax trick in the context of routing. First, as the authors acknowledge, but don't really elaborate on, using the Gumbel Softmax means we must backprop through every possible routing choice in each layer. As a result, the Gumbel approach results in a large scaling of computation with the number of modules, limiting the applicability to more ambitious settings. Moreover, while a clear motivation of this work is eliminating interference between tasks, it is not really explained how Gumbel Softmax does this and how it compares to hard routing decisions in this respect. During backprop, the computation it very similar to mixtures of experts models, and should contain more interference than hard routing. Can you explicitly show that the shape of the Gumbel distribution results in less interference between modules during learning than the standard mixtures of experts softmax approach? 

Furthermore, (Rosenbaum et al., 2019) found that a number of RL based models outperform Gumbel Softmax when routing on multi-task settings of CIFAR-100 and the Stanford Corpus of Implicatives. The authors do not provide any explanation for why this approach did not succeed in their settings. This also leads me to doubt how impressive the results presented here are as there is really not any apples to apples comparison with the same architecture and different routing decisions. In Tables 1 and 2 the best baseline is full sharing. This indicates to me that the performance difference with other cited baselines has to do with different architecture choices and not changes in the routing policy itself. The experiments can be much improved by discussing why past approaches to Gumbel based routing have failed and by thoroughly comparing to other methods for just the routing decisions with the same base architecture as done in prior work.  Unfortunately, in its current form, there is not enough context provided for the community to understand the implications of the proposed approach in the submitted draft even though it achieves good performance. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes to learn the routing matrix in routing networks for multi-task learning (MTL) using the gumbel softmax trick for binary random variables. It makes the model amenable for training the network and the routing matrix simultaneously, which is a relatively easier and unified training procedure compared to the original routing networks. The gumbel softmax trick technique is pretty standard. The proposed method is evaluated on two MTL datasets with comparisons to baselines on one of them. 

In terms of methodology, using gumbel trick for learning routing matrix seems new to my knowledge. Although the trick has been applied to other problems and is used in a standard way. I like the idea of using this trick to make the learning of routing network unified under optimization compared to the learning in the original routing network. 

However, the experiments seem not extensive enough to demonstrate its superiority and efficiency. The method is only compared with other state of the art methods on one dataset. More experiments on various datasets and neural network architectures will be more convincing to me. I am also interested in how does the sparsity of the different routing models compare to each other? It would be unfair if some models trade performance for sparsity compared to the method proposed in this paper. Also it would be interesting to see how the learned routing matrix pattern could say something about the relatedness of different tasks.
Regarding "full sharing", is it different tasks trained together with the same network? 
And another minor question for the experiments on MNIST, what are the accuracies for single task learning using same architecture?

Overall, I find the idea of using gumbel trick for learning routing networks interesting. However, I feel the experiments are not sufficient and I would encourage the authors to conduct more experiments and comparisons.

review_title: Official Blind Review #2

Meta-review: This paper proposed to use Gumbel softmax to optimize the routing matrix in routing network for multitask learning.  All reviewers have a consensus on rejecting this paper.  The paper did not clearly explain how and why this method works, and the experiments are not sufficient.
Decision: Reject