Gumbel-Matrix Routing for Flexible Multi-task Learning
Authors: Krzysztof Maziarz
Efi Kokiopoulou
Andrea Gesmundo
Luciano Sbaiz
Gabor Bartok
Jesse Berent
Venue:  (2020)

Abstract: This paper proposes a novel per-task routing method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharing. However, sharing parameters between unrelated tasks can hurt performance. To address this issue, routing networks can be applied to learn to share each group of parameters with a different subset of tasks to better leverage tasks relatedness. However, this use of routing methods requires to address the challenge of learning the routing jointly with the parameters of a modular multi-task neural network. We propose the Gumbel-Matrix routing, a novel multi-task routing method based on the Gumbel-Softmax, that is designed to learn fine-grained parameter sharing. When applied to the Omniglot benchmark, the proposed method improves the state-of-the-art error rate by 17%.

Under review as a conference paper at ICLR 2020
GUMBEL-MATRIX ROUTING FOR FLEXIBLE MULTI-TASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes a novel per-task routing method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharing. However, sharing parameters between unrelated tasks can hurt performance. To address this issue, routing networks can be applied to learn to share each group of parameters with different subset of tasks to bet- ter leverage tasks relatedness. However, this use of routing methods requires address the challenge learning the routing jointly with the parameters of a mod- ular multi-task neural network. We propose the Gumbel- routing, novel multi-task routing method based on the Gumbel-Softmax, that is designed to learn fine- fine-grained parameter sharing. When applied the Omniglot benchmark, the proposed method improves the state-of-the ofute-of-thh-ar error rate by 17%.
INTRODUCTION
Multi-task learning (Caruana 1998 1993 based on neural networks has attracted lots of research interest in the past years and has been successfully applied to several application domains, such as recommender systems (Bansal et al.) 2016) and real-time object detection (Girshick] [0115.. For instance, a movie recommendation system may optimize not only the likelihood of the user clicking on suggested movie, but also the likelihood that the user is going to watch it.
The most common architecture used in practice for multi-task learning is the so-called shared bot- tom, where the tasks share parameters in the early layers of the model, which are followed by task-specific heads. However, as our experiments on synthetic data show, when the tasks are un- related, parameter sharing may actually hurt individual tasks performance. Therefore, resorting to flexible parameter sharing becomes very important. This can be achieved by manually trying sev- eral different static sharing patterns. However, this option effort. Instead, an approach where the sharing pattern is learned and adapted to the task relatedness, preferable.
scalable, since it requires significant
At the same time, routing networks (Rosenbaum et al. 2018) have been introduced as powerful models, which route each input sample through its own path, selectively activating only parts of the network. They have shown strong performance in various settings thanks to their high flexibility. Routing networks lend themselves as natural choice for learning sharing patterns in multi-tase modeling. However, they are typically rather hard to train in practice (see e.g., (Rosenbaum et al.) 2019) for the challenges of routing networks and references there in).
In this work, we introduce novel method, that learns the sharing pattern jointly with the model parameters using standard propapropagttion. Assuming that the network consists of several layers, where each layer consists of several components, the core idca of the proposed method is to learn, for each component, set of binary allocation variables indicating which tasks use this component. We rely the Gumbel-softmax reparameterization method (Jang al. 2017) in order to train these binary variables jointly with the parameters of the components.
We provide experiments synthetic data as well real-world data showing that the proposed method can adapt the sharing pattern to the task relatedness, outperforming strong baselines and previous sate-of-the-ar methods. In summary, the paper contributions are the following:

---

Under review as a conference paper at ICLR 2020
We analyze positive and negative transfer effeets synthetic scenarios. This analysis motivates the need for task dependent parameter sharing multi-task networks.
We propose novel routing method based on Gumbel binary variables, that allows for learning flexible parameter sharing which adapts to the task relatedness, and can be opti- mized with standard back-propagation.
The source code implementing the proposed method and the benchmarks described in this paper is publicly available at lttp:: github. com/ anonymous
POSITIVE AND NEGATIVE TRANSFER BETWEEN TASKS
We start with a practical example showing that besides positive transfer, negative transfer may as well occur in practice. That is, when the tasks are unrelated, allowing them to interact in bigger model instead of training them separately harms the model performance. To show that both positive and negative transfer occurs, generate two synthetic tasks, where the task relatedness p can be explicitly controlled. Our synthetic data generation process is based on that of (Ma et al. 2018), and describe it in detail in Appendix A.I We consider two edge cases: two unrelated tasks (p 0), and two tasks that are the same up to noise (p = 1).
We create simple multi-task network. This network architecture consists of 4 parallel components, and each component contains a stack of fully connected layers. Each input example can be provided input any subset of the 4 parallel components; the outputs of the components fore being passed to a task-specific linear head. We chose this network to have low enough capacity, so that there is visible competition between tasks, For more information about the architecture, refer to Appendix A.2
averaged be-
For this experiment, that both tasks use all components, while 'no sharing' means that tasks use disjoint halves of all components. Therefore, in the sharing' pattern, the tasks are completely independent. Note that regardless of the pattern, the total amount of parameters in the model remains the same; the only difference is in which parameters get used by which tasks. other words, 'no sharing' corresponds to imposing constraint that the network should be evenly divided between the tasks, while 'full sharing' leaves that to the optimization algorithm to decide.
hard-coded sharing patterns. The 'full sharing' pattern means
Tasks unrelated
Tasks the up
No sharing
No sharing
Full sharing
Full sharing
3--
1-
0
1000 2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Figure 1: Comparison of the 'ull sharing' and 'no sharing' patterns for the case of unrelated tasks (left) and almost equal tasks (right). The plots show loss over time (averaged over the two tasks and smoothed over window of 100 steps). We ran each experiment 30 times, and the shaded area corresponds to the 90% confidence interval.
We run four experiments: one for every combination of sharing pattern ('full sharing' and 'no shar- ing'), and task relatedness (p E {0, 11). For each experiment, we report the 1.2 loss over time, averaged over the two tasks. The results are shown in Figure [ Since for 'no sharing" there is no interaction between the tasks, the average loss behaves in the same way irrespective of the task re- latedness. For 'full sharing". both tasks are allowed to update all parameters, and it improves performance the tasks are related, it actually hurts for two completely unrelated tasks.
that while
Motivated by this example, argue that general multi-task models should be able learn flexible sharing patterns that are able to adapt to the task relatedness. In Section we introduce a framework which flexibly learn parameter sharing patterns, including but not limited 'no sharing' and
2

---

Under review as a conference paper at ICLR 2020
'ull sharing'. Since method is based on routing networks, provide some background about them first.
3 ROUTING NETWORKS
Standard neural networks process every input example in the way. Routing networks provide a more flexible alternative, where every input is passed through a subgraph of the entire model (also referred to the supernetwork). Typically, the model is divided into layers, and some layers contain multiple parallel components (also called modules or experts). Inputs to the large network are routed through some sets of components in subsequent layers. This general framework was used by multiple recent works (Fernando et al. 2017,, Rosenbaum et al.) [0188] Ramachandran & Le 2019). although with some small differences or additional constraints depending on the routing method.
Component #1
Component #5
Component #2
Component #6
Aggregate
Component #3
Component #7
Component #4
Component #8
Figure 2: A general example of a routing model. The input passed through two components in the first layer, and one component in the second layer.
In this paper, task and multi-task setups. Note that we do not impose any constraints on the set components used for specific input; in particular, the number of components used can vary between layers and between individual inputs. After passing a sample through single routed layer, the outputs of all activated components are aggregated form an input to the next layer. In all experiments in this paper we simply use the average as the aggregation.
relatively general view depicted in Figure which is relevant to both single-
GUMBEL-MATRIX ROUTING FRAMEWORK
We design our framework so that is able learn to select task-conditioned subgraphs larger computational graph. In Figure 3 we show an example of routing model, where the routing is conditioned on the task only. In that example, there are two tasks, and two routed layers. Each task selects some subset of components within every layer. For every routed layer, encode the component assignment as a binary matrix. Concretely, if there are T tasks, and C' components in a given layer, the matrix has shape T C. where the entry in the i-th row and j-th column is 1 if the i-th task uses the j-th component. We show these matrices at the bottom of Figure 3 Our goal is to learn the routing matrices in the way that maximizes the average per-task performance of the model. In order to learn the routing matrices we condition on the task id, which implies that all samples from the same task will go through the same path in the network. We refer to our framework as the Gumbel-matre routing framework. While at training time our method samples many different routing matrices, at the end single matrix selected per layer. At inference time, the matrices are fixed, and thus our routing does not add any overhead over the underlying non-routed model.
4.1 TRAINING
For each layer, want to maintain probability distribution over all possible binary routing matri- ces. To that end, we assume this distribution to be factorized, and we explicitly maintain a matrix of per-connection probabilities. Each probability is represented pair of two complementary logits.
3

---

Under review as a conference paper at ICLR 2020
Layer 1
Layer
Task-specific heads
(tank
Conv
Average
Conv 2x3
#11)
(tnoe
Conv Sec
Canv Bxc
Conv 7X7
Conv 7x7
In (aak #21
Idendity
Average
Identiyy
Aweroge
Linear projection (baok #2)
Out (lask #2)
Components
Companents
Tas
Figure 3: An example routing network with tasks. Some components are used by both tasks (purple), some by only one of the tasks (red or blue, respectively), and one identity component is completely unused (white). Below each layer we show the corresponding routing matrix.
To perform forward pass, we sample all binary connections independently according to the proba- bility matrix. In principle, is possible that for given task, all connections sampled to O. that case, the output of the routed layer would be a zero vector, independently of the input. In practice, we found that this happens very rarely, and mostly at the beginning of training, since usually one of the connection probabilities quickly becomes close to 1. Therefore, we did not try to devise method which would artificially prevent an all-zeros connection pattern from being sampled.
To initialize our method, we have to set the connection probabilities to some initial values. In princi- plc, it is possible to introduce prior knowledge, and set these probabilitics in way that encourages or discourages certain patterns. However, here we consider the most general approach, where all connection probabitlies are initialized to the same constant value Pinit. Setting Pait = 0.5 gives the highest routing entropy, and corresponds the weakest prior, therefore choice. However, in our experiments, we also found that for routing in large and deep networks, it is beneficial to set Pinit. closer to in order to enhance the trainability of the components and to stabilize the initial learning phases.
considered it as a default
In the backwards pass, only the components that activated will get gradients, the inactive components do not contribute to the final output of the network. However, in order to get gradient for the connection probabilities, we would have to backpropagate through sampling. In the next section, we describe a method which we use to accomplish that.
4.2 GUMBEL-SOFTMAX TRICK
In order to get gradients to the connections probabilites, we follow (Jang et al. 2017), and repa- rameterize sampling from a Bernoulli distribution by using the Gumbel-Softmax trick. The Gumbel distribution can be defined by the following forward sampling procedure:
u ~ Uniform(0. 1) = -log(- log(u)) ~ Gumbel.
Instead of using the logits to directly sample binary value, we add independent noise from the Gumbel distribution to each of the logits, and then select the binary value with the highest logit (i.e. argmax) as the sample z. Formally, to sample from Bernuull(pp. we use the following procedure. Let p); draw go and 91 from the Gumbel distribution, and produce the sample
arg Us, where == log(T) [go, 91]. (E(0,1)
The argmax operation is not differentiable, but can be approximated by a softmax with annealing temperature. Therefore, the forward pass, use the argmax to obtain binary connection value, while on the backwards pass, we approximate it with softmax, similarly to (Guo et al. 2018). This approach is known the Straight-Throug! Gumbel- Softmmx estimator (Jang al. 2017). Note that
4

---

Under review as a conference paper at ICLR 2020
the backwards pass actually requires all components to be evaluated, irrespective of whether they are used the forward pass or not. Therefore, if a connection is sampled to be inactive, then the corresponding component will not get gradients, but its output will be used to compute the gradient for the connection probability.
4.3 INFERENCE
At inference time, it is possible follow the same procedure as training time, i.c. sample the connection pattern for every test batch. In our experiments we found that this works well, and does not introduce large amount of noise in the evaluation result, since the connection probabilities naturally tend to converge to either 0 or during training. An alternative approach is to fix the connections to their maximum likelihood variants, and that pattern for every forward pass. We do that in the evaluation phase of all our experiments, since we believe this is closer to how the Gumbel- Matrix framework should be used practice. Note that for the maximum likelihood approach, we can discard all connection probabilities after the training has completed. The probabilities are used only to describe how to select subgraph of the network for each task.
4.4 BUDGET PENALTY
We have found that Gumbel-Matrix routing generally trains well in its vanilla form. However, we note some ways in which it possible alter its default behavior. For example, one might want to learn routing pattern with certain degree of sparsity. To that end, we introduce the budget penalty, which penalizes the model from exceeding given computational budget. Let us assume we define the budget as maximum percentage of active connections. Since we explicitly know all connection probabilities, by summing them up over all layers we obtain the expected number of connections ea for forward pass given point in time. Therefore, we can set a budget b E (0, 1), corresponding to the maximum allowed fraction of active connections, and define the budget auxiliary loss as: A max(0, e. - b). where À is constant that controls the strength of the penalty. For sufficiently large X, this penalty can be viewed as a hard constraint in practice.
5 RELATED WORK
Traditionally, work on multi-task learning includes hand-designing the sharing pattern, in order to strike good balance between shared and task-specific parameters. Our method is more related to works in the intersection of routing models and multi-task learning, where the sharing pattern is learned jointly with the model parameters. These works can be mainly divided based the algorithm that used to learn the routing.
Some methods cast learning the routing (Rosenbaum et al. 2018) propose framework based on multi-agent Reinforcement Learning, where the nnsitive-negative transfer problem is taken of by finding Nash equilibrium. In contrast, our work does not rely on Reinforcement Learning, and can be trained with standard back-propagatin notion of task-specific paths is also present in (Fernando et al. 2017); these paths are learned using evolutionary algorithms.
Reinforcement Learning problem. The authors in
Many other works use the Sparsely-Gated Mixture-of-Expert (Shazeer et al. 2017). initially devel- oped for a single-task model. This idea is extended in (Ma et al., 2018) by introducing a separate gating function per task, and in (Ramachandran & Le (019) by using architecturally diverse experts and increasing the routing depth. However, (Ramachandran & Le 2019) report that their models are often hard to train. In contrast, we show that our method can improve accuracy even when routing in large and deep neural networks.
Sub-Network Routing (Ma et al., 2019) proposes a routing method for multitask networks. This approach is not task-dependent, as it learns routing that is applied to all the tasks. It is based on learned binary variables, that control the connections between modules, rather than controlling the activation of module itself. Another related approach relies on cross-stitch models (Misra al. 2016). It uses single-tast models that are stitched together with cross-stitch units, which learn shared representations by linearly combining intermediate representations from the single task models.
5

---

Under review as a conference paper at ICLR 2020
Additionally, some works use concepts similar the ones use, but not for multi-task learning- In particular, Guo et al. (2018) also rely on the Gumbel Softmax trick to learn binary variables, with focus fine-tuning for transfer learning applications. The binary variables
used to decide which layers of a pre-trained model should he fine-tuned on the target task. Another related method (Bengio et al.] (016) learns binary variables that mask the outputs of each layer, conditioning on the activations of the previous layers. In order to learn these binary variables, the REINFORCE algorithm (Williams) 1992) is used. similar notion of adaptive inference graphs has been pro- posed in Veii & Belongie 2019) that uses convolutional neural nets for image classification based on ResNet-type of architecture, where some layers are skipped using learned gating functions. However, note that these methods have not been designed for multi-task learning.
Finally, our approach related to methods for Neural Architecture Search (NAS) (Zoph & Le 2017 Real et al. (017), which automatically design neural network architectures for a given task. Our method is similar spirit as the efficient NAS methods (Pham et al., 2018 [iu et al. (019) in the
sense that the architecture parameters are jointly optimized with the model parameters. Our method searches for a multi-task architecture that learns a flexible parameter sharing pattern according to the task relatedness, and uses simpler architecture encoding based on binary variables.
6 EXPERIMENTS
In all our experiments, we impose an additional constraint that each input batch contains samples for only one task. Since the routing is conditioned on the task only, this allows us to sample the connection pattern once per forward pass. To train network in multi-task setting, batch of input samples per task, pass them through the network in random order, and repeat that process for a predefined number of steps. To start, have tested our method on the same synthetic data as in Section Due to lack of space, we provide the experimental results in Appendix A.3
draw one
6.1 MNIST
Experimental setup To test our routing method in a controlled environment where we know which pairs of tasks are more related, we create the following 4-MNISTs setup based on the MNIST dataset.
We first define the MNIST-rot task, by taking the input-output pairs of MNIST, and rotating all input images clockwise by 90 degrees. We run experiments on 4 tasks, where the first two tasks are copies of MNIST, and the next two are copies of MNIST-rot. Note that two copies of the same task have the same training and test datasets, but the order of batches is different. In order to make the setup difficult, we use relatively small routed network. It consists of three routed layers, containing four components each. The components in the first routed layer are 5x5 convolutions, while in the second and third layers are 3x3 convolutions. After the last routed layer, the output feature map is flattened, and passed through task-specific linear head. For more details, see Appendix B
Results We first run two baselines, corresponding to the 'no sharing' and 'full sharing' patterns introduced in Section 2 In this case, 'no sharing' corresponds the i-th of the four tasks using only the i-th component in every layer- again, this means that there is no interaction between tasks. 'Full sharing' means that all tasks use all components. We see that 'full sharing' strongly outperforms 'no sharing', which shows that this routed network is small even for MNIST, and using one component per layer is not enough to reliably learn the task.
Next, we train two variants of Gumbel-Matrix routing: one without any auxiliary penalties, and one with the budget constraint set to 0.75. We report all of these results in Table We found that the two copies of MNIST end up using the same routing patterns, as well the two are copies of -ot. However, patterns used by the copies of MNIST are different from the ones used by MNIST-rot. As seen in the results, this gives better performance, since the processing is task- dependent. Furthermore, we see that by using the budget penalty, we can reduce the number of active connection without sacrificing the test accuracy.

---

Under review as a conference paper at ICLR 2020
Table 1: Results the 4-MNISTs multitask setup. Each experiment was run 30 times, we report
mean and standard deviation of the error.
Method
Test accuracy (%)
Active connections (%)
No sharing
93.5 + 5.1
25
Full sharing
95.7 = 0.5
100
Gumbel- Matriix
96.8 = 0.2
96
Gumbel--Matrx (budget 0.75)
96.7 + 0.3
75
6.2 OMNIGLOT
Next, we test our method on the Omniglot multi-task setup (Lake et al. (013).. The Omniglot dataset consists of 50 different alphabets, each containing some number characters. Input samples for each of the characters are handwritten grayscale images of size 105 x 105.
Experimental setup We create our setup following Meyerson Miikkulainen (2018), where each alphabet is treated as separate task of predicting the character class. We follow previous works (Liang et al, 2018:) Ramachandran & Le 2019) and use a fixed random subset of 20 alpha- bets, splitting every alphabet into training/valldattonnesst sets with proportions 50%/20%/300 For completeness, we list these 20 alphabets in Appendix C.I C.I
In order to have direct comparison with the state-of-the of Ramachandran & Le 2019),
the same underlying network, optimizer, and regularization techniques, and only change the routing algorithm. We have reached out the authors of (Ramachandran & Le| 2019) obtain various details and make sure the setups match. We report the architecture here for completeness.
The network consists of: one shared Ixl convolution, then routed layers, and finally linear task specific heads. Each routed layer contains 7 different components: conv 3x3 + conv 3x3, conv 5x5 - conv 5x5, conv 7x7 + conv 7x7, conv 1x7 + conv 7x1, 3x3 max pooling, 3x3 average pooling, identity. The number of channels is 48 throughout the network. All components use padding to make sure the output shape the same the imput shape: the spatial dimensions are reduced by adding a stride of to 5 of the routed layers. We use GroupNorm (Wu & He 2018) and ReI.U after each convolution and after each routed layer.
We regularize the model with Dropout and 12-regularization. For training, we use the Adam opti- mizer. Since the routing logits are updated only once every T' steps (where T is the number of tasks), we have found that for T 20 it is beneficial to use larger learning rate for the routing logits than for the components. Therefore, we set the learning rate for the routing logits to be T times larger than the one for the other weights, and found this rule of thumb to work well in practice. We set the training length to be larger than needed for the methods to attain their peak performance, select the best checkpoint for each method based validation accuracy, and evaluate that single checkpoint the test set. For hyperparameter values, as well as more details about the network, see Appendix C.2
Results Before training model based on Gumbel-Matrin routing, we train a 'full sharing' variant, where all tasks use all components. We do not evaluate a 'no sharing' variant, since the number of tasks T is larger than the number of components per layer. Then, train a routing model, where we use of the experiments in Table 2
routing to model the connections in each routed layer. We show the results
We see that the underlying non-routed model actually outperforms the Mexture-oo-Experee routing of Ramachandran & Le (201)), which we found to very surprising. We conjecture that even though the cixture-of-Experts variant is more powerful, in the case of multi-task learning on Omniglot optimization difficulties outweigh that benefit. In contrast to our method, the Mixture-of- framework hard-codes the required sparsity for each layer. This can bring immense computational savings (Shazeer et al.) 2017). but may also sacrifice accuracy. In some cases, like the one of Shazeer et al. (2017). the 'full sharing' variant would be prohibitively expensive to run, making the comparison infeasible. However, we encourage routing models researchers to compare their routed networks with their 'full sharing' counterparts whenever possible.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Results on multitask Omniglot setup. Each experiment was 10 times, we report mean and standard deviation of the error.
Method
Valid. error (%)
Test error (%)
Single Task (Meyerson Miikkulainen, 2018)
36,41 + 0.53
39.19 + 0,50
Soft Ordering (Meverson & Mukkulaınen 2018
32.33 = 0.74
33,41 = 0.71
CMTR (Liang et al. 2018)
11.80 + 1.02
12.81 = 1.02
MoE (Ramachandran Le 2019)
7.95 + 0.37
7.81 + 0.54
Full sharing
6.16 = 0.50
6.75 = 0.33
Gunbel-Matrix
5.69 + 0.22
6.48 + 0.28
Next, we train a routed model based on Gumbel-Matrix routing. We do not use any auxiliary losses, and find that the model naturally removes some of the connections to allow for task-specific pro- cessing. Even though the network is not explicitly penalized for high routing entropy, connection probabilities still converge be either close to close to 1. We report the resulting accuracy in T We see that the Mambel-Matrix routing improves the accuracy over a very strong 'full sharing' baseline.
Analysis In order to analyze the routing patterns learned by our network, we compute several statistics. First, for each task, we concatenate the routing matrices from all routed layers to form a larger matrix, which we call global pattern. If two tasks have exactly the same global pattern, the network will process them in the same way (up to the task-specific linear heads). We find that, on average, the model converges to having 10.1 different global patterns. That means that tasks are clustered into groups that are processed in the same way, and the average size of such group is approximately 2.
Next, we analyze what kinds of components are used at every layer in the network. Recall that each layer in the model consists of 7 components, out of which 4 are different kinds of convolutions, 2 are pooling (max and average), and the last one is identity, which can be though of as a skip connection. We found that the usage trends are consistent across experimental runs, and mostly depend on the layer in question. The trends are the following (please see Appendix C.3 for relevant illustrations):
In layers 4, we found that the connections all components are being dropped a similar rate; most of the diversity in processing of different tasks concentrates in these layers.
In layers 5 - 7, almost all components are always used, and the only source of differences between tasks is that some of them use the average pooling component, while others don't
In layer the average pooling component is never used, and the other components are always used.
Our interpretation of this result is that the Hambeel-Matrn routing shows a mixture of two behaviors. First, the connections can be dropped in order to obtain task-specific processing; this trend is visible the first layers of the model. Second, which corresponds to form of network pruning. We speculate that the task-specific processing in the initial layers "aligns' the feature spaces of samples coming from different tasks. the final layers, the tasks processed in uniform way, but some components pruned to improve the model quality.
connections to specific components can be removed,
7 CONCLUSIONS AND FUTURE WORK
We introduced new method for multi-task learning that learns the pattern parameter sharing together with the model parameters using standard back-propagation. We provided experimental re- sults showing that our method can learn flexible sharing patterns, and adapt to the task relatedness, which results in significantly improved performances over the state-of-the-an Right now, the rout- ing decision in our method is conditioned only on the task id. Conditioning on richer information such task embedding will be the focus of future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
T. Bansal, D. Belanger, and A. McCallum. Ask the GRU: Multi-task learning for deep text recom- mendations. In 10th ACM Conference on Recommender Systems, pp. 107-114, ACM, 2016.
E. Bengio, P.-L. Bacon, Joelle Pineau, and D. Precup. Conditional computation neural networks for faster models. arXiv preprint arXiv:1511.06299 2016.
R. Caruana. Multitask learning: A knowledge- source of inductive bias. Machine Learning: Proceedings of the Tenth International Conference, pp. 41 41-48, 1993.
R. Caruana. Multitask learning. Learning learn, 1998.
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. Rusu, A. Pritzel, and D. Wier- stra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.
R. Girshick. Fast R-CNN. In International Conference on Computer Vision (ICCV), pp. 1440-1448. IEEE, 2015.
Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, and R. Feris. SpotTune: Transfer Learning through Adaptive Fine-tuning. Conference on Computer Vision and Pattern Recognition (CVPR), 2018. URLpttpss: / arxiv. www.i..wrg/bb///l11ll.ooo
E. Jang, S. Gu, and B. Poole. Categorical reparametrization with Gumbel-softtax. International Conference Learning Representations (ICLR), 2017.
M. B. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 1550:13321331
2015. 2015.
J. Liang, E. Meyerson, and R. Mikkuulannnn Evolutionary architecture search for deep multitask networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 466- 473. ACM, 2018.
H. Liu, K. Simonyan, and Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representutions (ICLR), 2019.
J. Ma. Z. Zhao, Yi X., J. Chen, L. Hong, and Chi. Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts KDD, 2018.
J. Ma, Z. Zhao, J. Chen, A. Li, L. Hong, and E. Chi. SNR: Sub-Network Routing for Flexible Parameter Sharing Multi-task Learning. AAAI Conference on Artificial Intelligence, 2019.
URL URthttpp: // /wwww aiagima.......m................a wwt.aaa.cc.op.paaaaaaaaaaapaa..a wwwoom/pprr////e/ooooommooa SNR.pdE
E. Meyerson and Risto Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. International Conference on Learning Representations (ICLR), 2018.
I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, IEEE, 2016.
H. Pham, M. Guan, B. Zoph, Q. Le, and Dean. Efficient neural architecture search via parameters sharing. International Conference on Machine Learning (ICML), 2018.
Ramachandran and Q. V. Le. Diversity and Depth in Per-Example Routing Models. International Conference on Learning Representations (ICLR), 2019. httpss://ppenreviee... pdf?iddBBxNJnnce
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. Large- Scale Evolution of Image Classifiers. International Conference on Machine Learning (ICML), 2017.
C. Rosenbaum, T. Klinger, and M. Riemer. Routing Networks: Adaptive Selection of Non- Linear Functions for Multi Task Learning, International Conference on Learning Representations (ICLR), 2018. URL httpp:::/ppppmeeeeeeeeeee/////pp

---

Under review as a conference paper at ICLR 2020
C. Rosenbaum, Cases, M. Riemer, and T. Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint 11Xi::190..12774, 2019.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 2017.
A. Veit and S. Belongie. Convolutional Networks with Adaptive Inference Graphs. In International Journal of Computer Vision (IJC).. pp. 1-12, 2019.
R. Williams. Simple statistical mudieentffollowig algorithms for connectionist reinforcement learning. In Machine Learning, volume 8(3-4), pp. 229 229-256, 1992.
Y. Wu and K. He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
B. Zoph and Q V. Le. Neural Architecture Search with Reinforcement Learning. International Conference on Learning Representations (ICLR), 2017.
10

---

Under review as a conference paper at ICLR 2020
A SYNTHETIC DATA
A.1 GENERATION
We follow a process for synthetic data generation based (Ma et al. 2018), which allows for explicit control of the relatedness between tasks. In particular, we generate the data as follows:
1. We generate two orthogonal unit vectors u1, U2 R That is, uj and satisfy in u2 = and || | 1.
2. Given desired score of task relatedness 0 S p S 1, we generate two weight vectors W1, W2 such that
W1 = cu1, W2 = c(pu1 + v1 - v1=2²u
(1)
3. We randomly sample a data point Rd where each dimension is sampled from (N0, 1).
4. Generate two labels y1, U/ for the two tasks as follows:
y1 = w x +
sin a x + 3;) +
(2)
m
2/2 = w x +
sin(a, x + +
(3)
where 61, 62 ~ 0.0,0.01.
In the procedure above, the values of d, and the sequences 0g, 83 are hyperparameters. For
all of our experiments with synthetic data, we set 128, c = 1, m = 6, a; = B3 (i 1)2 We found m be the most important of these hyperparameters, it controls the non-linearity (and hence the difficulty) of the tasks. In particular, for m the input-output relation for the produced task will be linear (up noise).
A.2 ARCHITECTURE FOR THE 'POSITIVE AND NEGATIVE TRANSFER' EXPERIMENT
Here we describe in detail the network used for the experiment in Section
The network starts with a layer containing parallel fully connected components. Two of these components components consist of fully connected layer, and the other two consist of 2 layers with 4 hidden units. All components have 4 output units, and use ReLU activations on the hidden and output units. Note that specific input can be passed through multiple components; the outputs of the components are averaged before being passed to a task-specific linear head. We present this architecture in Figure
Routed layer
Linear projections
In (A1)
FC ReLU
Projection (A1)
Out (A1)
(A2)
ReLU
Projection (A2)
(A2)
Aggregate
(B1)
FC ReLU
FC ReLU
Projection (B1)
Out (B
In (B2)
FC ReLU
Projection (B2)
(B2)
Figure 4: Multi-task model architecture for the experiment on the synthetic data. We denote fully connected layers by FC, and name the two pairs of tasks A1, A2 and B1, B2, respectively.
11

---

Under review as a conference paper at ICLR 2020
A.3 CUMBEL-MATRIA EXPERIMENT
We additionally test the Gumbel-Matrix routing on synthetic data using a setup that emphasizes the routing pattern. To that end, we create routed network that does not contain task specific heads. Instead, consists of a routed layer with components, each with output unit. The components are relatively large for this kind of simple data: each is 3 layers deep, with 16 hidden units in every hidden layer, ReLU activations on the hidden units, and no activation on the output units.
We use batch size of 64, learning rate 0.01, and we clip the gradient norm to 1.0.
We generate two pairs of tasks according to A. so that each pair contains tasks that are the same up to noise, but tasks in different pairs are unrelated. Since we use network with no task-specifis heads, if two tasks get routed through the same set components, the network will implement the same function for these tasks. Therefore, if the routing is incorrect (i.e. routes two tasks from different pairs through the same subset of components), the rest of the network cannot make up for it, making it good sanity check for our method.
Note that in this network, each component capable of learning the synthetic task on its own, and using 'no sharing' pattern is solid baseline. However, routing should still be able improve performance: if it discovers the related pairs of tasks, they can be routed through a shared set of components, and parameters these components would get twice as much training data compared to the 'no sharing' case.
We train for 3000 batches per task, and track the average loss over tasks. The result is shown in Figure We found that the related tasks converge to using the same subset of components, but the subsets used by the unrelated tasks are distinct. Because of that, the routed model outperforms the 'no sharing' variant. In this setup, the 'full sharing' setup works very poorly, since it is unable to learn both pairs of tasks at the same time. Hence, we omit the results for the 'full sharing' baseline.
5
sharing Gumbel matrix
3
L003
1
1000
2000
3000
Training steps
Figure 5: Comparison of the 'no sharing' pattern with the learned Gumbel-Matrix routing on the synthetic data experiment. The plot shows loss time (averaged over the four tasks and smoothed over a window of 100 steps). We ran each experiment 20 times, and the shaded area corresponds to the 90% confidence interval.
B ARCHITECTURE FOR THE MULTI-TASK MNIST EXPERIMENT
Here we report additional details on the network used for the MNIST experiments.
As stated before, the network consists of three routed layers, containing convolutions with kernels 5x5, 3x3 and 3x3, respectively. We do not use padding in any of the convolutions, which means spa- tial dimensions are slightly reduced in each routed layer. After the first and second routed layer, further reduce the spatial dimensions by applying 2x2 average pooling. All convolutions throughout the network have 4 filters.
12

---

Under review as a conference paper at ICLR 2020
During training, regularize the model with Dropout (dropout probability 0.5, applied right before the tasks-specific heads). We set the training length to 10 epochs per task, and use a batch size of 16. We present this network in Figure 6
Layer
Layer 2
Layer
Linear projectons
In IMNIST #1)
Gony fad
Conv Sc3
Coru 3c3
Propztion (MNST #1)
(MNIST #1)
IMNIST W2)
Cany
Cone
3c3
Projection (MNIST 92)
OUI R2)
AngPaol 2x
ArePuol 2G
Corv
3x3
IMNIST
In DMNIST-SE-R21
Cony 5a5
Cony 3x8
Corv 3x3
Projection (MN Teal #2)
Gut (KNI5T-vol #2)
Figure MNIST multi-task network.
C OMNIGLOT
C.1 ALPHABETS
In our Omniglot experiments, we followed prior works (Liang et al. 2018 Ramachandran & Le 2019) and used a fixed set 20 (out of 50) Omniglot alphabets. These alphabets are the first in the order originally reported by (Meyerson & Miikkulainen 2018). The names of these alphabets are following: Gujarati, Sylheti, Arcadian, Tibetan, Old Church Slavonic (Cyrillic), Angelic, Malay (Jawi-Arabic), Sanskrit, Cyrillic, Anglo-Saxon Futhore, Syriac (Estrangelo), Ge ez, Japanese (katakana), Keble, Manipuri, Alphabet of the Magi, Gurmukhi, Korean, Early Aramaic, Atemayar Qelisayer.
C.2 ARCHITECTURE AND HYPERPARAMETER
For the Omniglot experiments, we use a large convolutional network, developed by the authors of Diversity and Depth Per-Example Routing Models (Ramachandran & Le 2019
One detail that we copy from Ramachandran & Le 2019) is that the identity component, in the case when it belongs routed layer with stride larger than 1, is actually implemented strided 1x1 convolution. While it makes the name 'identity component' slightly inappropriate, we follow this to be directly comparable.
We tuned the network hyperparameters using the validation set, and used the following values for all of our Omniglot experiments. Dropout probability is 0.5, I.2-regularization strength 0.0003, and the learning rate is set to 0.0001. We train with batch size of 16.
For the Gumbel-Matris routing, we set Pinit 0.97. We found that increasing Pinst does slightly increase the average number of active connections at convergence, however, even with initialization as high as 0.97, the model still confidently removes many connections. On other hand, we found Panit 0.97 trains much better than Pinit 0.5. We believe that in deep routed network with many components in every layer setting Pinit to 0.5 introduces too much noise during training.
We present the overview of the architecture in Figure 7 Additionally, in Figure g show the components within a single routed layer.
13

---

Under review as a conference paper at ICLR 2020
Linear projections
(Omnigiot !)
Projection (Omnigior
(Omniglot 1)
In (Omniglor 2)
Projection (Omnigiot 2)
Out (Omnigiot 2)
Flasten Dropout
Routed layers
(Omnigiot 19)
Projection (Omnigiot 19)
(Omniglot 19)
(Omniggoot 20)
Projaction (Omnigiot 20)
Out (Omniglat 20)
Layer [stridee 2)
Layer #2
Layer (etride 1)
Layer (stride 2)
Layer (stride
Layer (stride 1)
Layer (stride 2)
Layer (stride 1)
(stride 2)
Figure Omniglot multi-tast network.
layer s)
Conv 3x3 (stride 1)
GN & ReLU
Conv 3x3 (stride
GN ReLU
Conv 5x5 1)
GN ReLUU
Conv 5x5
GN RaLU
Conv (stride
ReLLU
Conv (stride
GN ReLU
(stride 1)
Conv (stride
ReLU
MaxPool (stride s)
AugPaol 3x3 (stride s)
Idenlity (stride s)
Figure 8: Components inside routed layer the Omniglot multi-task network. We denote Group
Norm by GN, and the layer stride s. Note that for this specific architecture have {1,
14

---

Under review as a conference paper at ICLR 2020
C.3 ADDITIONAL ANA
As we have discussed in Subsection 6.2 the connection to the average pooling component is being dropped most often, while other connections are only dropped in the early layers.
Figure 9 iillustrate these trends and shows how the connection probabilities are evolving over time for a certain task (the 'Syriac' alphabet). Note that for the experiments in Subsection 6.2 we ran 10 experiments for every method; here, for ease of presentation, we only show one of the runs. We see for example that for this specific task, the connection to the 3 3 convolutional component gets dropped in the first layer, while connections to average pooling are dropped in layers 3, 6 and 8, In the remaining three layers, the task uses all components.
Layer
Layer
Conmection probaaiitg
0.8
Concict >s Convlies
0.8
Convict -> Canvied
Cone5e5 > Candix5
Conv5i5 -> Canu5a5
0.6
Convix7
0.6
Convix? la fonvldd
Convlxd -a Conu7xt
0.4-
Max/col 3x3
MaxPael 3x3
AvgPool 3x3 Idartity
AvgPool 3x3
ldeetityy
0.0
0.0
1.0
Layer 3
10-
Layer 4
Connerton probeeity
Convixt > Convxxs
Comvxi Convoxs
Convix7 > Convix7
Convix? - ConvTx?
Convixt -a tonvhxl
Condlx Conuixt
0.4
Maxhool
94.
MasPael
AvgPsol 3x3
AvgPuol 3x3
Idertity
ideniiiy
0.2
02
Layer
Layer
0.8
Convax3 > Convax3
0.8
Convaxa Convaxa
Cornection pmmb
Convix5 Conv5xs
Comsis Convsis
0.6
Convix7 Convix7
0.6
ConVTK7 ConvTx7
Convixt is conv7k3
Convixt -> Convixt
Maxhool lxl
MaxPoel lx?
Augisul 3x3
AvgPoel 3x3
Idartity
Icentity
0.0
0.
1.0
Layer 7
10-
Layor 8
Connection probabiity
Convax3 Conaxx3
Convaxa Convax3
Convix5 > Cond5x5
Comi5is ConvSxS
Convix7 Conv7x7
Convix7 ConvTk7
Convixt conv7xd
Convlx? Conv7xt
0.4
04
Augitel 3c3
AvgPaol 3x3
Idertkt
Idertity
0.2
02-
Training
Training time
Figure 9: Connection probabilities over time (for one run, and one task). Each plot above shows the connection probabilities for all components of certain routed layer in the network.
15


Reviews:
[{"review_id": "rker7VmRKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies the Gumbel-softmax to optimizing task-specific routing in deep multi-task learning. Experiments demonstrate improvements of the method over no sharing or full sharing, and it is used to achieve s-o-t-a results in the Omniglot MTL benchmark.\n\nAlthough the end results are good, and the approach is well-motivated, I am leaning to reject, because the experiments have not made clear when the method works and how it behaves. The improvements over the full-sharing baselines appear fairly small, and in the analysis it appears the model is mainly discarding unneeded pooling layers. Is there some real task-specific routing that the method is able to take advantage of? Maybe an experiment where full-sharing is detrimental, i.e., because there are some highly unrelated tasks, would help to highlight how the approach selects appropriate module subsets for each task. E.g., what are the routing patterns in Section 6.1 that are the same within each pair of MNIST tasks, but different across task pairs? Is there a way to visualize differences between routing of different Omniglot tasks?\n\nSimilarly, the experiment in Section 2 is interesting, but the conclusion that negative transfer exists is not novel. Is there a way to include the Gumbel approach in these synthetic experiments to show that it addresses this issue? E.g., something like the result in A.3 could be promoted to Section 2. More compelling synthetic datasets could be generated by the method in A.1. for the case where tasks are somewhat related, in which case we can actually see if how the sharing occurs. Could Gumbel see a bigger boost in these synthetic experiments if training data were limited and generalization was tested instead of training loss? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syl8rm7atB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to learn the routing matrix in routing networks for multi-task learning (MTL) using the gumbel softmax trick for binary random variables. It makes the model amenable for training the network and the routing matrix simultaneously, which is a relatively easier and unified training procedure compared to the original routing networks. The gumbel softmax trick technique is pretty standard. The proposed method is evaluated on two MTL datasets with comparisons to baselines on one of them. \n\nIn terms of methodology, using gumbel trick for learning routing matrix seems new to my knowledge. Although the trick has been applied to other problems and is used in a standard way. I like the idea of using this trick to make the learning of routing network unified under optimization compared to the learning in the original routing network. \n\nHowever, the experiments seem not extensive enough to demonstrate its superiority and efficiency. The method is only compared with other state of the art methods on one dataset. More experiments on various datasets and neural network architectures will be more convincing to me. I am also interested in how does the sparsity of the different routing models compare to each other? It would be unfair if some models trade performance for sparsity compared to the method proposed in this paper. Also it would be interesting to see how the learned routing matrix pattern could say something about the relatedness of different tasks.\nRegarding \"full sharing\", is it different tasks trained together with the same network? \nAnd another minor question for the experiments on MNIST, what are the accuracies for single task learning using same architecture?\n\nOverall, I find the idea of using gumbel trick for learning routing networks interesting. However, I feel the experiments are not sufficient and I would encourage the authors to conduct more experiments and comparisons.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlvY-bRYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In many ways this work is well presented. However, I have major concerns regarding the novelty of the proposed method and the theoretical rationale for the key design choices. Although the authors do cite and discuss (Rosenbaum et al., 2019), what is very much not clear to me is how the Gumbel-Matrix Routing proposed in this work differs from past work using the Gumbel Softmax within routing networks. It seems like past work even focused on using only the task for routing, so it is not clear to me how the approach here is really novel in comparison. Even if there is some distinction I am missing, the high level idea is clearly not that new. Additionally, there is not much theoretical discussion about what the Gumbel Softmax adds to routing networks. \n\nThe bias/variance tradeoff of Gumbel Softmax / RELAX / REINFORCE was already highlighted in (Rosenbaum et al., 2019). Can the performance of the model on the settings tested be attributed to this tradeoff? If so, would a RELAX model perform even better? Moreover, there is not much discussion of important implications of using the Gumbel Softmax trick in the context of routing. First, as the authors acknowledge, but don't really elaborate on, using the Gumbel Softmax means we must backprop through every possible routing choice in each layer. As a result, the Gumbel approach results in a large scaling of computation with the number of modules, limiting the applicability to more ambitious settings. Moreover, while a clear motivation of this work is eliminating interference between tasks, it is not really explained how Gumbel Softmax does this and how it compares to hard routing decisions in this respect. During backprop, the computation it very similar to mixtures of experts models, and should contain more interference than hard routing. Can you explicitly show that the shape of the Gumbel distribution results in less interference between modules during learning than the standard mixtures of experts softmax approach? \n\nFurthermore, (Rosenbaum et al., 2019) found that a number of RL based models outperform Gumbel Softmax when routing on multi-task settings of CIFAR-100 and the Stanford Corpus of Implicatives. The authors do not provide any explanation for why this approach did not succeed in their settings. This also leads me to doubt how impressive the results presented here are as there is really not any apples to apples comparison with the same architecture and different routing decisions. In Tables 1 and 2 the best baseline is full sharing. This indicates to me that the performance difference with other cited baselines has to do with different architecture choices and not changes in the routing policy itself. The experiments can be much improved by discussing why past approaches to Gumbel based routing have failed and by thoroughly comparing to other methods for just the routing decisions with the same base architecture as done in prior work.  Unfortunately, in its current form, there is not enough context provided for the community to understand the implications of the proposed approach in the submitted draft even though it achieves good performance. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed to use Gumbel softmax to optimize the routing matrix in routing network for multitask learning.  All reviewers have a consensus on rejecting this paper.  The paper did not clearly explain how and why this method works, and the experiments are not sufficient.
Decision: Reject