Mint: Matrix-Interleaving for Multi-Task Learning
Authors: Tianhe Yu
Saurabh Kumar
Eric Mitchell
Abhishek Gupta
Karol Hausman
Sergey Levine
Chelsea Finn
Venue:  (2020)
Keywords: multi-task learning

Abstract: Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data. Applications of neural networks often consider learning in the context of a single task. However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks. Such multi-task learning settings have the potential to improve data efficiency and generalization by sharing data and representations across tasks. However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is very difficult to learn a single model that can solve all the tasks while realizing data efficiency and performance benefits. Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide. In this work, we develop an approach that endows a single model with the ability to represent both extremes: joint training and independent training. To this end, we introduce matrix-interleaving (Mint), a modification to standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix. By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks. On three challenging multi-task supervised learning and reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both.

Under review as a conference paper at ICLR 2020
MINT: MATRIX-INTERLEAVING FOR MULTI-TASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data. Applications of neural networks often consider learning in the context of single task. However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks. Such multi-task learning settings have the potential improve data efficiency and generalization by sharing data and representations across tasks. However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is difficult to learn single model that can solve all the tasks while realizing data efficiency and performance benefits. Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide. In this work, we develop approach that endows single model with the ability to represent both extremes: joint training and independent training. To this end, introduce matrix-interleaving (Mint), modification standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix. By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks. On three challenging multi-task reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both.
INTRODUCTION
While deep learning has enabled remarkable levels of generalization through the use of function approximators, this comes at the cost of large amounts of data, which remains a critical challenge in deploying deep learning to a number of domains. When combined with deep networks, multi- task learning offers the promise of building more powerful representations using less data per task, leading to greater performance and data efficiency. However, multi-task deep learning has also posed considerable challenges. Numerous works have observed that joint training on multiple tasks can actually decrease task performance due the negative influence of other tasks (Parisotto et al.) 2015 Rusu et al.] 2a) Indeed, training networks entirely independently on each task has remained a strong approach, to the point that multiple multi-task methods have first trained models independently before using them to train a multi-tasking model (Parisotto et al.] 2015) Rusu et al. 2016aa Ghosh al. 2017 Teh et al. 2017; Czarnecki al. 2019).. Moreover, our experiments in Section indicate that three recently proposed methods for multi-task learning are all surpassed by training models independently per task. However, training independent models will only work well when provided enough data per task, and precludes potential positive data-efficiency gains from multi-task learning, only providing protection against negative transfer. Further, while a number of works have successfully shared parameters, finding an architecture with the appropriate level of parameter sharing for a given problem domain can require a considerable amount of manual engineering. In this work, we aim to develop a multi-task learning method that can perform well both when tasks share very little and when they share a large amount of structure.
To address this problem, we consider how single neural network model can represent two extremes: independent models, when optimization challenges prevail, or a single model with shared weights, when sharing is beneficial. Further, would like such model be able to represent intermediate

---

Under review as a conference paper at ICLR 2020
levels of model sharing, when appliable. One option for performing independent training within a single model is to put separate networks with independent weights into a single model, using the task ID to select which network prediction to output. However, this prevents any sharing. An alternative approach is to condition the model on the task ID, through various conditioning approaches, including additive and multiplicative approaches such as FiLM (Perez et al. 2018). In fact, point-wise multiplicative conditioning, as proposed in FiLM, can indeed represent separate networks by selecting which parts of the network to be used for different tasks, as can number of other approaches in multi-task learning (Rosenbaum et al. 2017 2019 Fernando et al. 2017). Yet, these approaches still require an optimization over shared parameters in order to select which parameters are used for each task. These shared parameters can introduce significant optimization challenges.
We instead consider how to allow a model to perform optimization on only shared parameters, only disjoint parameters, or any combination thereof. We can achieve this by simply interleaving learned per-task matrices at each layer of a jointly-trained neural network. When optimization over shared parameters is ineffective, the model can still represent full neural network per task using only the per-task matrices, resulting in independent training; while using identical per-task matrices results in standard joint training. Intermediately, mix of shared and per-task parameters may be used. In effect, by incorporating these matrices into the network, the optimizer itself can automatically and dynamically modulate the degree to which representation is shared between tasks, depending on the problem domain and the optimization progress, and can do so without having to optimize shared parameters.
The primary contribution of this paper is a simple yet effective approach for multi-task learning that can represent and smoothly interpolate between independent training and joint training, via matrix interleaving (Mint). We describe how we can implement Mint in deep multi-task models and show its effectiveness in improving data efficiency and generalization in multi-task settings while providing intuition about the reasons why this architecture performs so well. Further, we show that the model can be extended to goal-conditioned reinforcement learning in a straightforward manner by allowing the model to generate the interleaved matrices conditioned on task information such as the goal. We evaluate Mint on sets of tasks with both high and low levels of shared structure and find that performs well in both settings, performing comparably to outperforming both joint training and independent training, effectively combining the best elements of both. Further, in comparison to previous methods that use multiplicative interactions for continual learning (Cheung et al., 2019) and for general conditioning (Perez et al.] 2018). Mint is better able to separate tasks by avoiding the need optimize over shared parameters and can empirically produce substantially better performance range of challenging multi-task problems. Finally, Mint also outperforms state-of-the-arr approaches for multi-task learning while being significantly simpler implement.
2 PRELIMINARIES
In multi-task learning, the goal is to find a e-parameterized model fo that reaches high performance across all training tasks drawn from task distribution To p(T), i.e. min minET..P(T) (fo(l
where La denotes the loss function for task Tg. In this work, we study the setting in which we have a fixed set of K tasks {T.} and we wish to obtain high performance on all of the tasks in this set. In Section 4 we will study this multi-task problem in the context of a reinforcement learning setting. In our multi-task learning setup, we train model that is conditioned on a task indicator 25 which is used to specify a task Ty that should be performed. The task indicator can be represented in a variety of ways, from simple categorical variables to learned task embeddings (Hausman et al. 20188). This formulation can be readily extended to a goal-conditioned reinforcement learning setting, where 2 indicates the desired goal. The multi-tast learning model learns to optimize objectives of all the tasks Ty that the model is trained on.
K
Joint and fully independent training the two extremes of multi-task learning. Assuming of training tasks, we characterize multi-task training as independent if it optimizes a set of task-specific, disjoint parameters {0* ["= that parameterize the model for. We define joint training as finding single set of task- lask-independent, shared parameters 0 of fa. Note that joint training utilizes the fact that the parameters are shared throughout learning. While joint training has a number of data-efficiency and generalization benefits, it can be difficult to train effectively.
2

---

Under review as a conference paper at ICLR 2020
3 MULTI-TASKING WITH INTERLEAVED MATRICES
Considering fully independent and fully joint training as two ends of the spectrum in multi-task learning problems, we want to design an algorithm to get the best of both worlds the stability of independent training and the parameter-shan sharing efficiency of jointly trained models. To this end, we propose to build models that allow the neural network to learn how much information should be shared and between which tasks throughout learning, in an adaptive fashion. We describe the details of our approach below.
3.1 MINT: INTRRL-VITEREE NETWORKS
In the case of neural network models, we view representations as network activations at different layers of the network. We aim to introduce modification to the neural network that would allow the model to either form those activations in completely task-specific way, a completely shared way, or in a way that shares an intermediate degree. To achieve this, we propose model architecture that transforms the previous layer's representation both in a task-general way and in task-specific way, sequence. When two tasks share very little, the network can optimize task-specific weights, while when the tasks share a considerable degree of structure, the network can leverage the shared weights. Since these transformations are task specific and can be introduced various layers of the network, they allow for a different amounts of representation shared at different levels of the neural network model.
To understand the practical implementation, we consider the activations at layer I of a fully-connected neural network: ( (W() ((-1) + =b)), where WF() is the weight matrix for layer 1, b(9) is the bias vector for layer L, and a is the non-linearity at each layer. The Mint layer augments the traditional
fully-connected layer with task-specific weight matrix Mr and bias vector 3x. where k indexes the
task. The forward pass of a Mint layer for some vector of activations is presented Definition [
Definition 1. A Mint layer applies affine transformation to activations y E R. follows, yielding new activations a:
Mint(y) M&y + 3k
where M₁ and 3g are per-layer task task-specific matrix and bias, respectively. A neural network aug- mented with Mint thus contains parameters that both shared across all tasks well parameters that are only used for a particular task for each layer 1, i.e. 0 = (W(), 80), Mr()) a(")). See Figure for visual depiction of the application of Mint. We show how the regular conneccd layers and Mint layers can be interleaved in Equation 2 and 33n33belom below:
( (1 (1 + +801)
(2)
a 1 M (0 + +kx
(3)
Because this layer is dully-differentiable, we can learn the task-specific matrices M and biases / jointly with the shared parameters of the model.
When we apply Mint to tasks with very large numbers tasks, or arbitrary task descriptors (e.g.. goal-conditioning),, we can train separate neural networks The T - M S.. to output the Mint matrices and biases at every layer, instead of storing independent matrices for each task. In this case, Mint resembles FiLM Perez et (2018), another method which performs transformation-bat based task conditioning. In contrast to FiLM, Mint uses matrix transforming the activations at each layer instead point-wise multiplication by a vector. the next section, we study a theoretical property of Mint that motivates the chosen definition of the Mint layer. We validate the benefits of Mint in our experimental evaluation in Section
3.2 THE EXPRESSIVE POWER OF MINT
We next aim to theoretically study the expressive power of multi-task learning architectures when the shared parameters cannot be optimized effectively (e.g. due to optimization challenges arising from learning tasks that share little or structure). While viable approach in this regime is simply use separate task specifce networks fa, with no shared parameters, this approach makes the strong assumption that the tasks indeed share no exploitable structure. This is often not known priori.
3

---

Under review as a conference paper at ICLR 2020
fo fo(
Mad + Be
Mint Layer
Mint Layer
Bk
fo-layer
M
Mint Layer
fe-layer
T
input
a
(b)
Figure I: Mint architecture. (a) A Mint Layer applies task-specific matrix multiplication and vector addition an input vector x. The matrix and vector make up an affine transformation to the input vector, given by task network. (b) The modified neural network model with distinct Mint layer added after each fully-connected layer.
We show in this section that
'worst case" where shared parameters are not useful, the class of functions defined by a Mint network is still as rich that defined by a set of purely task-specific networks. Thus, Mint makes no sacrifices in terms of expressiveness with respect to the optimal attainable parameters of and retains the possibility of exploiting shared structure if it does exist. Further, we show that the same not true of existing methods that are related to Mint. Specifically, compare the class of functions representable by Mint with
1. The class of functions defined by FiLM network (Perez et al., 2018). where a layer's activations are 'tilted' through element-wise multiplication with lask-specific vector " and addition with a task-specific bias 3x.
2. The class of functions expressable by a task indicator concatenation approach, where one-hot task indicator vector 25 for task Ts is concatenated to each shared layer's input.
We begin by considering the optimal parameters for task-specific networks føz- where of arg min Lo))), the parameters that minimize the loss function L. for task T, We assume that L. sequential
føz is an I.-layer MLP. We also consider I-layer Mint networks fo, (consisting pairs of Mint transform and shared parameter layer as in Equation [ and L-layer FiLM networks (consisting of L sequential pairs of FiLM transform and shared parameter layer). We state Definition to complete our setup.
Definition 2. Let (1-1) the activations of layer L 1-1 1, and let as = (W()), buy be the shared parameters (weight matrix and bias vector) at the 1-th layer. We assume that W()) an arbitrary, fixed invertible matrix.
With this setup, we state Lemmas D and 2, which highlight key difference in the layer-wise expressiveness of Mint and methods such as FiLM and task indicator concatenation.
Lemma 1. For a given as applying Mint to y (1-1) can express an arbitrary affine transformation at layer I for each task.
Lemma 2. (a) For a given au, there exist affine transformations that cannot be expressed by applying FiLM to (1--1) (b) Similarly, for a given as there exist affine transformations that cannot be expressed by network which concatenates the inpus (1-1) of each layer with a task indicator Zk.-
The proof can be found in Appendix A We assume only that the shared parameter matrices WIl) invertible, such that they do not lose information about the input. This assumption critically does not require the optimizer productively update the shared parameters.
We use Lemmas D and 2 to extend the above laser-wise expressiveness properties of Mint and FiLM compare the function classes defined by T--layer Mint networks fo, and FiL.M networks fe, in the
4

---

Under review as a conference paper at ICLR 2020
context of optimal L-layer MLP for We state this comparison in Theorem 1 We to denote the subset of 0g and bh that corresponds to the parameters of the network that are shared across tasks.
Theorem 1. For all a, there exists Of such that for (x) = Soz (1) for all x. Further, there exists or and E such that fox (3) + loz [ all thi.
Because for is an MLP (a composition of affine transformations and activation functions), the proof of the first statement follows from applying Lemma to each layer of fø; and fo, (the task specific and Mint networks, respectively). Similarly, Lemma I 2 implies that we can construct an and C; (and therefore of) such that we can find an ž satisfying the second statement.
That is, for any set of shared parameters in a Mint network and any set of optimal 'target' MLP, there exist weights for Mint that are effectively equivalent the target MLP. On the other hand, there exist settings of the shared parameters that make matching the outputs of the target MLP unattainable we reach an equivalent
in the FiLM and task indicator concatenation regimes. Using Lemma 2 conclusion regarding the limited expressiveness of the task indicator concatenation approach.
FLAVORS OF MINT
The general idea of Mint is implemented as follows in the supervised and reinforcement learning settings.
Multi- Task Supervised Learning In the case of supervised learning models, we simply apply Mint layers after every fully connected layer of the model. Specifically, for task identifier 2g R K where K is the number of tasks and for every layer / with activations (!)) E IR. nonlinearity and weight W(D and bias b()) we represent the transformation using two matrices TW E R=*** and T Baa that take in the task identifier and output the per-layer task-specific matrices M (9) and biases B² respectively. The transformation can be summarized as follows:
a = -TTi 2a (W (-1) + (00) T 2k = M₁ ( (WWW + b²)) + 30
(4)
Multi-Task Reinforcement Learning For multi-task reinforcement learning, we implement the architecture similarity to the supervised learning case but we combine this with actor-critic RL algorithms by introducing this architecture into both the critic (s,, 0, 24) and the actor x <al|,, 2x).
Goal-Conditioned Reinforcement Learning For the case of goal conditioned RL, we introduce slightly modified Mint architecture into both the actor and the critic conditioned on the task goal
g. Specifically,, for every layer / with activations (3)), nonlinearity a, weight W'I), and bias b() we represent two transformation function TT(() and T(()) by two 2-layer ReLU networks that take in the goal and produces per-layer goal- specifcc matrix M' (g) and the bias (00) respectively. The transformation can be summarized as:
1 = -T"" + 600)+ +T)))(10)))))))) M(g)) ((W +b41) + () (5)
RELATED WORK
Multi-task learning Caruana 1997: Bakker & Heskes 2003 Ruder 2017) focuses on the problem of
finding a single model that can solve multiple different tasks. This formulation can be readily applied
to variety of learning domains, such supervised learning (Zhang et al. 2014 Long & Wang 2015
Yang & Hospedales 2016b Sener Koltun 2018 8 Zamir et 2018), and multi-task (Espeholt et al. 2018 Wilson et al. 2007 Hessel et al.] 2019) and goal-conditioned reinforcement learning (Kaeibling
1993 Andrychowicz et al. 2017 Pong et al. 2018). While multi-task learning offers the promise of efficient training of shared representations, naïvely training single model on multiple tasks often does not result in these desired benefits, due to the optimization challenges introduced by the
multi-task setting (Teh al. 2017 Ghosh et al. 2017 Rusu et al. 2016a).
5

---

Under review as a conference paper at ICLR 2020
In order eliminate potential negative interference between different tasks during multi-task learning using a single model, many approaches propose to learn each task separately, to later combine their
solutions into single multi-task model (Levine et al. 2016 Teh et al. 2017 Ghosh al. 2017] Rusu et al. 60 Czarnecki et al., 2019) Parisotto et al. 2015). In contrast to these works, we present a method that is able train single model on multiple tasks and is able interpolate between the extremes of joint and independent training.
More closely related our approach, various architectural solutions have been proposed to increase the multi-task learning capability of the model. Example approaches include architectural changes
that allow multiple modules or paths within the same network (Fernando al. 2017 Devin
2016; Misra et al. 2016;; Rusu et al.) [0166)) Vandenhende et al.) [219]] Rosenbaum et al.) [00
transformation-bee
task conditioning (Perez et al. 2018), attention-basee architectures (Liu et al. Maninis et al. 2019), multi-headed network solutions (Long & Wang 2015 Riedmiller et al.
2018:
2018 Wulfmeier et al. 2019), and a variety other approaches (Hashimoto et al. 2016] Ruder12
al. 2017). We demonstrate an approach that allows for middle ground between the conceptual extremes of fully independent training at one end and single-model joint training the other. This added flexibility enables to sidestep the negative effects of potential task interference while the same time share parameters between the tasks when appropriate. Prior approaches (Yang & Hospedales 2016a Long al. 2017) have investigated this ability by factorizing layers in the neural network across independent and shared components. In contrast, our method is simpler to implement, less computationally intensive, and empirically casier optimize. In our experiments, we provide direct comparisons of our method to cross-stich (Misra et al. 2016), routing networks (Rosenbaum al. 2019), the FiLM architecture (Perez al. 2018), rotational superposition (Cheung et al. 2019 and multi- models.
EXPERIMENTS
The goal of experimental evaluation is answer the following questions: (1) does method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there little overlap?, (2) how does our method compare independent training and joint training in those settings?, (3) how does our method compare to prior statc-of-the of-th-rrtt approaches?
sam feacet
Ereketbell
sowees into hole
S. furn off leucwi
puih
lever dial pushwith 10, hand insert
11. get coffee
pul handio side
noombble
14 pull with stick
15, pick ouite. dimartike of hole
17, place anto theif
18, push mug
19 press 20. close box
handle side
21.
23, slide plata
pull
27 28. retrieve retrieve lock door plate side
plata
31. close drawser
37. press buttion top
33. reach
top will
35. meach with wall
36 insert
sida
37. push
38. push with wall
pit
40. unlock door
pull
44 unplug
45. window
46. open
open
49. open
SC. pick bin
button
place
mug
pag
door
Figure 2: Visualization of the 50 tasks from Meta-World used the MT50 evaluation. Mint is able to learn
about 30 of these tasks.
To answer the above questions, we conduct experiments multi-task reinforcement learning domains. For multi-task RL doamins, we perform experiments on two multi-task RL benchmark variants MT10 and MT50 (as showcased in Figure] 2) proposed by Yu al. 2011). Finally, to test if Mint can excel in continuous task distributions, we also evaluate the method on a goal-conditioned RL domain where Sawyer robot arm is required push randomly initialized puck to different goal positions. For all RL experiments, we use the popular off- off-policy RL algorithm, soft actor-critic (SAC) (Haamnja et al. 2018), which has shown to solve many RI. benchmarks with great data-efficiency.

---

Under review as a conference paper at ICLR 2020
On the multi-task RL domains, we compare Mint to the following methods:
SAC: train a vanilla SAC agent with task identifier as part of the input.
Multi-head SAC: train SAC agent where both the actor and critic are represented as multi-
head feedforward neural networks where the number of heads is the number of tasks.
SAC (concat all fe): train a SAC agent where the task identifier is concatenated with the activation at each layer and passed as inputs to the next layer.
FiI.M (Perez et al.. 2018): the actor and critic are learned with neural networks combined with FiL.M.
Superposition (Cheung et al., 2019): the actor and critic are learned with neural networks combined with superposition.
independent: learn separate actor and critic per task.
We provide details of architecture design in each domain well as environment set-up in the Appendix B
6.1 MULTI-TASK REINFORCEMENT LEARNING
On the RL domain, we first investigate the ability of Mint to perform a set of distinct RL tasks. As discussed Yu et al. 2019). MT10 and MT50 serve as representative benchmarks evaluate multi-task learning al algorithms on learning diverse range of robotics manipulation tasks. We present the results Figure 3 The success rates are averaged across tasks and we adopt the success metrics used in the Meta-Woo World benchmark. We design appropriate architectures of the actor and the critic of the SAC algorithm for each method such that the number of parameters of each approach is around the same magnitude (see Appendix B for details).
For MT10, Mint learns all tasks with the best data efficiency, while independent networks also learn all of the tasks with slightly worse data-efficiency. The other methods are unable acquire half of the skills. Mint, on the other hand, enjoys the expressive power to interpolate between independent learning and sharing, while mitigating optimization challenges, to attain the best results between the two extremes.
For MT50, where the evaluations done all 50 of the Meta-World environments, shown the right in Figure 3 Mint quickly learns to solve more than 60% of tasks in 20 million environment steps while SAC and SAC with multi-head architectures struggled in solve 40% of the tasks after 35 million steps. Independent networks learn to solve the tasks slower than Mint but eventually surpasses it! This result also validates the expressive power of Mint to represent both separate learning and learning with shared networks.
MT10
MT50
10
iouri
Independent
0.8
Mulii head SAC
C.6
Rales
Superposition F2
Success Reles
c.5
0.6
C.4
0.3
C.2
0.1
3000 10000 15000 20000 25000 30000 35000 Number thousand env steps Figure 3: Learning curves on MT10 (left) and MT50 (right). We observe that independent training performs well on both benchmarks. Mint, unlike prior multi-task lcarning approaches, is able to perform at a similar level to independent training.
1000
2000 3000
4000
5000
Number of thousand env steps
We also examine the learned Mint layers in the MT10 setting to analyze whether they capture task- specific information. Specifically, we replace the 'close drawer" task in MT10 with a duplicated 'press button top' task and thus we have two 'press button top' tasks in MT10 (see Figure Z) We
We did not evaluate superposition and FiL.M on MT50 given the poor performance of these methods on MT10.
7

---

Under review as a conference paper at ICLR 2020
train Mint on this set of tasks. Concretely, if we index the tasks from to 9, then we have two copies of task Ti and leam two separate Mint matrices: 1f,1 and M2
We then compute the pairwise & distance between the Mint layer Iearned for one of the duplicate 'press button top' tasks and the Mint layer learned for any other task such as 'insert peg side' in the set. Specifically, we compute d( d(Mr, 1, Mr,) for all i 1, where d is the I distance. We compute the percent difference between each d( d(Mr, Mr. and d(MT, My, and present it in Figure the left. For each pair Mr. # we plot 100d(Mr, 1.Mr, ((d))dd(Mp L., From the figure, the two di d(Mr, 1.M2)
Mint layers learned for the duplicated tasks have the smallest relative l distance except the distance between the Mint layers of 'press button top' and 'reach', which is reasonable since 'press button top' is structurally similar to 'reaching'.
Finally, we provide a comparison between Mint and other methods where the non-Mint methods are trained with the same number layers as Mint as opposed to the similar number of parameters in Figure 3 As shown on the right in Figure Mint outperforms all of the other approaches this setting.
MT10
MT10
1.0
Matt-
2.0
Success
00
0.6
0.0
pasn
open CraNe Rawting JKx Taxice
Number of thousaand nnv stops
1002 2000 3000 4000 5000
Figure 4: On the left, we show the percent increase in & distance between Mint layers learned for one of the duplicate tasks and each the other tasks MT10 as compared to the distance between the Mint layers learned for the two duplicate tasks. We can see that for most of the tasks, the percent increase in & distance is approximately 10%, except that the distance between 'reach' and "press button top' is smaller, which could be explained by the fact that 'press button top' inherently just reaching task. On the right, compare Mint a list of other methods with the same number of layers and find Mint achieves significantly higher success rate than any of the other approaches.
6.2 GOAL- COAL-CONDIIION REINFORCEMENT LEARNING
Next, we consider the question if Mint can be applied to a set of RL goals that have considerable shared structure. Hence, evaluate all methods the goal-conditioned sawyer pushing domain. In this environment, the goal space consists of the initial positions of the puck and the goal positions. For details of the goal-conditioned environment, see Appendix B At each policy rollout step, we sample a batch of 9 goals and collect 3 paths for each goal, where all the paths are stored in the task- replay buffers. At each training step, we sample a batch of goals and 128 samples per goal from their corresponding replay buffers. To prevent creating infinite number of replay buffers, we discretize the goal space into 200 goals. Given that it impractical to train 200 independent agents, we sample 10 goals from the goal space and train 10 independent SAC agents for estimating the performance of independent training in goal-conditioned pushing. As shown in Figure 5 Mint outperforms all methods both in terms of data efficiency and distance to the goal. SAC (concat all fe) also achieves comparable performance while independent networks fail to learn the task without sufficient amounts of data, suggesting that the ability of Mint to represent both joint training and independent networks per task is crucial in multi-task learning and can lead to considerable improvement.
CONCLUSION
Simultaneous optimization of multiple, potentially unrelated tasks can prove challenging for deep neural networks. Recent multi-task learning architectures attempt to mitigate this issue by providing alternative pathways for information to flow through a neural network for each task. In this paper, we introduce new multi-task learning module, Mint, which provides theoretical guarantees of universal approximation even for multi-tase settings with no shared structure. We conjecture that this property, not shared by similar multi-task architectures, enables Mint outperform other multi-task approaches
8

---

Under review as a conference paper at ICLR 2020
Goal-Conditioned Pushing
0.350
Distance to the goal
0.325
0.300
0.275
0.250
Mint
Independent
0.225
SAC
SAC (concat fc)
0.200
Superposition
0.175
FiLM
1000
2000
3000
4000
5000
Number of thousand env steps
Figure 5: Learning curves on goal-conditioned pushing. Mint is able to outperform all other methods in terms of both distance to the goal and the required number of environment steps.
on variety of reinforcement learning benchmarks. We also observe that Mint is able to match or improve upon the performance of independent training.
While Mint exhibits strong performance gains over previous methods, one potential limitation is that the task matrices may introduce significant number of parameters, particularly as the number of tasks increases. As discussed, this can be alleviated for problem domains with many tasks, by learning single neural network that produces the matrices and biases conditioned on the task descriptor. Further, in our experiments, we find that Mint-based networks can outperform prior methods while using comparable or fewer parameters.
In summary, Mint is simple, yet effective approach for deep multi-task learning. Its implementa- tion requires minimal modifications over standard deep networks. As a result, we expect it to be straightforward for future work to build upon use Mint for more effective multi-task learning in deep networks.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. J. Mach. Learn. Res., 2003.
Rich Caruana. Multitask learning. Machine Learning, 1997.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno A. Olshausen. Superposition of many models into one. CoRR, abs/ abss/990.00552. 2019.
Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS, 2019.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. CoRR, 0es/1609.070000 2016.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA:

---

Under review as a conference paper at ICLR 2020
scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, htsp/1701..87344 2017.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. CoRR, abs/1711.09874 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning. 2018
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. joint many- -ask model: Growing neural network for multiple nlp tasks. arXiv preprint 16Xiv: 1.11:11871.777 2016.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Leslic Pack Kaelbling. Learning achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016. 2016,
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. CoRR, 103.04403, 2018.
Mingsheng Long and Jianmin Wang. Learning multiple tasks with deep relationship networks. arXiv preprint arXv: 07 2, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multilinear relationship networks. In Advances in neural information processing systems, pp. 1693-1603, 2017.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. CoRR, abs/1994.08918, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert, Cross-stitch networks for multi-task learning. In Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint 15I 2015.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep rl for model-based control. arXiv preprint 1rXiv:1802.09088 2018
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint Xoxiv:1802.105536 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. CoRR, TttN//771.01239, 2017.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv: /904. /2774, 2019.
10

---

Under review as a conference paper at ICLR 2020
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
Sebastian Ruder12, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Sluice networks: Learning what to share between loosely related tasks, stat, 1050:23, 2017.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülgehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. In 4th International Conference on Leurning Representations, ICLR, 2016a.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606. 04671, 2016b.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectivv optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496 4506, 2017.
Simon Vandenhende, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: Deciding what layers to share. CoRR, abs/ toss/9944.0220,, 2019.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp. 1015-1022. ACM, 2007.
Markus Wulfieeer, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Regularized hierarchical policies for compositional transfer in robotics. arXiv preprint arXiv:1906. 1228, 2019.
Yongxin Yang and Timothy Hospedales. Deep multi-task representation learning: A tensor factorisa- tion approach. arXiv preprint arXiv: 1605.0639, 2016a. 6a.
Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-tass learning. arXiv preprint ar
2016b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Sergey Levine, and Chelsea Finn. Meta-world: A benchmark and evaluation for multi-task and meta-reinforcement learning, 2019. URL /tLl/tt/////tttuuu..
com/ wwtp://ooooooo/o/ett///aaaoaoooa
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference computer vision, pp. 94-108. Springer, 2014.
A PROOF OF THEOREM
Lemma 1. For a given oi, applying Mint to (11-1) can express an arbitrary offine transformation at layer for each task.
Proof
Let (i) and (()) be arbitrary weight matrix and bias vector. Suppose that for task we wish to represent the affine transformation Wx (1) (t-1)) + by (t) layer I of the network using the combination of Mint and the affine transformation described by applying (!)) multiplicatively

---

Under review as a conference paper at ICLR 2020
and b()) additively. Concretely, wish to determine whether there exist task-specific matrix ((a)() and task-specific bias vector ((x) such that:
(1 (t + bk (1) M (M y +(())))))))) 6())
WK
(6)
Let M())() ((O) -1 WK and w' (bx b(1). Then, the above equality holds.
w
Lemma 2. (a) For a given ax, there exist affine transformations that cannot be expressed by applying FiLM to y(--1), (b) Similarly, for a given Os, there exist affine transformations that cannot be expressed by network which concatenates the input y (1-11) of each layer with a task indicator zμ-
Proof of (a).
We wish show similar existence proof in Equation 6 except instead of per-task weight matrix ((x)()) each layer, FiLM uses a per-task modulation vector ((k) Concretely, we wish to determine whether there exist task-specific feature modulation vector e(k) and task-specific bias vector ((k) such that:
W 1 (1-1) + by ( = w (v(k)) @ y (1-1) + (kk) - 80)
where @ denotes the Hadamard product. This is equivalent representing (k) as diagonal matrix ((k) whose entries along the diagonal are the entries in v(k). Equation is equivalent to
Wk y (1-1) + bk (1) w (V ((-1) ((k) 6()
W (K) y + w 3(A)) 60)
Equating the corresponding terms both sides of the equation (those which include (1-1) and those which do not), equality holds only if:
V (1) (9) y (011- = W& @ (1--
+ V(K) ( w W (1
Since this equality must hold for all possible (1-1)
and ((k) ((!) -1 (bx (1) 6(0). However, since (AA) and Wk (1) is arbitrary, there is no
guarantee that (()) W (1) will be a diagonal matrix.
Proof of (b).
Let W and (1) be an arbitrary weight matrix and bias vector. Suppose that for task
we wish to represent the affine transformation W, Wx (1-1( + by (1) at layer / of the network by concatenating a task indicator to ( and extending W to apply an affine transformation to the concatenated input. This amounts to equating the following transformations:
Wx y + by (1) Wi. (3) (1-())) + b, (1) + Wr
where Wt. (4) is a matrix which is applied to zg. This equation only holds for all possible inputs y (t-1) Wx (1) Wn (1), Since W; (0) is fixed, we are not able to satisfy this equality.
B EXPERIMENT DETAILS
We SAC (Haarnoja al. 2018) algorithm for all RL domains. For Mint, We use 3-layer fully- connected neural network interleaved with per-task Mint matrices and biases at each layer for both the actor and critic. All the other methods 6-layer fully-connected neural networks for the actor and critic such that they maintain about the same number of total parameters as Mint. The number of hidden units per layer is 200, 160, and 150 for conditioo pushing, MTI0, and MT50 respectively for all methods.
On the goal-conditioned pushing experiment, we sample the (z, y) positions of the puck uniformly from the range [-0.2, 0.6] to [0.2, 0.7] and the positions of the goal uniformly from [-0.2, 0.85) to [0.2, 0.95).
12


Reviews:
[{"review_id": "rygCTmYRtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a single-network approach to multi-task learning by adding a task-specific linear transformation layer after each fully connected layer. The authors prove that the addition of such a layer keeps the expressive power of the network for each task. They also discuss how the linear transformation (parameterized by a transformation matrix and a bias vector) can be represented in a discrete manner in usual multi-task supervised learning and in a continuous manner (by two other neural networks) in goal-conditioned reinforcement learning. Experiments demonstrate the superiority of the proposed single-network approach.\n\nThe proposed approach is single and elegant. It is recommended to weak-reject the paper because of the following key reasons.\n\n(1) Problem formulation is far from clear, perhaps because of the lack of clarity in writing. In particular, the super short Section 2 did not clearly illustrate what's being trained and what's being tested, and whether we care about the generalization performance of each task, or the generalization performance to new tasks (generated from P(T)). Some notations are confusing there---for instance, i seems to be indicating tasks, but then there is a z_k as task indicator. Even for the main proposed approach in Section 3.1, the notations are loosely used in nature. For instance, it is hard to understand what the authors mean by \"train separate neural networks to output the Mint matrices and biases\"---there is no information about the \"training data\" for learning those neural networks.\n\n(2) Theoretical justification is at best shallow, or at least in the context that the authors have put it. While having an universal expressive power is good, it is easily achieved by adding an indicator variable (z_k) per layer (similar to task-specific-all-fc in the experiments). So the guarantee does not seem to be closely related to explaining the proposed approach (though the guarantee is nice to have). The authors contrast the guarantee with what FiLM (a competitor approach) can do, but in the experiments FiLM is not taken as a competitor in multi-task supervised learning, leaving a big gap between theory and practice. In the flow presented by the authors, it is strongly suggested to introduce FiLM in more detail and compare it with the proposed approach more clearly in design, theory and experiments.\n\n(3) It is hard to understand whether the experiments are reasonably designed. In particular, the two settings take different sets of competitors, and there is little information on why those competitors are selected, whether they represent state-of-the-art, etc.. The authors highlight that the proposed approach uses much fewer parameters but other than that it is hard to infer why the proposed approach is better. Is it better because there is more overfitting for the competitor's approaches given more parameters? Is it better because it is easier to tune? The task-specific-all-fc (which is of similar # parameters to the proposed approach) result particularly looks suspicious to me but there is no other information to double-check on why the proposed approach is better. In particular, I believe the authors have *not* answered their first proposed question \"does our method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there is little overlap\" properly---their best evidence may have been MT10 and MT50 experiments, but even in those experiments, I am not sure whether the authors want to take the results as suggesting there are \"substantial overlap\" or \"little overlap.\" \n\nSome other suggestions:\n\n(4) It is suggested to analyze the matrices learned by the proposed approach. Do the matrices contain reasonable task correlations (i.e. for two similar tasks, are the matrices somewhat similar) to understand more about the proposed approach.\n\n(5) It looks a bit strange to me that there is no discussion on regularizing the linear transformation matrices, as it seems possible to embed the task relations through the regularization. Have the authors considered the possibility?\n\n(6) The authors are overly-emphasizing what they want to do (interpolating between independent networks and shared network). This occupies multiple redundant paragraphs in the early sections. It is better to remove some of those and use the space for more solid results, such as clarifying the notations.\n\n(7) One baseline that could have been considered is to just train a fully-shared network (without z_k), and a fully-independent one. Then, use validation to select the better network and compare with the proposed approach.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxF5WxU9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a matrix-interleaving (Mint) based on neural networks for multi-task learning. The Mint contains a share parameter matrix and a task-specific parameter matrix. \n\nAuthors claim that the proposed Mint have the ability to represent both extremes: joint training and independent training. However, if the relations among tasks make the model between the both extreme cases, how is the performance of the proposed model?\n\nWhat does \"when the shared weight matrices are not learned\" mean? Are the shared weight matrices randomly initialized and then fixed without updating?\n\nTheorem 1 requires that each W^(l) is invertible, which implies that W^(l) is a square matrix. This requirement may not be satisfied in many neural networks. In this case, does Theorem 1 still hold? If not, Theorem 1 is not so useful.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlCxtZ59S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a simple but effective matrix-interleaving method (mint) for multi-task learning, which aims to represent both joint training and independent training.\nThe model achieves good performance on several supervised and reinforced learning datasets. Though the model resembles FiLM(Perez et al., 2018), it outperforms FiLM by a larger margin in three dataset.\n\nIt would be better for authors to give more detailed comparisons with models that work on combining both joint training and independent training.\n\nI would like to see the paper to be accepted for its simplicity and effectiveness.\n\nTypos: chnages -> changes?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BylK3m7RtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a factorization method for learning to share effectively in deep multitask learning (DMTL). The approach has some very satisfying properties: it forces all sharable structure to be used by all tasks; it theoretically captures the extremes of total sharing and total task independence; it is easy to implement, so would be a very useful baseline for future methods; and it is able to effectively exploit task similarities in experiments, and outperforms some alternative DMTL methods.\n\nI have two main concerns with the work: (1) It is most closely related to MTL factorization methods, but does not discuss this literature, or provide these experimental comparisons; (2) the interpretation of why Mint works is not clear: it is not clear that the universality is what makes it work, and there are no experimental analyses of what Mint learns. \n\nW.r.t. (1), there are several DMTL approaches that factorize layers across shared and task-specific components, e.g., [1], [2]. Such approaches are extensions of factorization approaches in the linear setting, e.g., [3], [4]. Compared to previous DMTL approaches, Mint is more closely related to these linear methods, as it takes the idea of factorizing each model matrix into two components and applies it to every applicable layer. In particular, the formal definition (i.e., without nonlinear activation between M and W) of Mint appears to be a special case of the more general factorizations in [1]; an experimental  comparison [1] would make the conclusions more convincing, e.g., that universality is important.\n\nHowever, in the Mint experiments, a non-linear activation is added between the two components of each layer. This could void the universality property. Is there some reason why this is not an issue in practice? \n\nMore generally, it is not clear that universality is the important advantage of Mint. Some existing DMTL methods already have this property, including Cross-stitch, which is compared to in the paper. The intriguing difference with Mint is that shared and unshared structure are applied sequentially instead of in parallel. Could there be an advantage in in this difference? E.g., is Mint a stronger regularizer because it forces all tasks to use all shared layers (learning the identity function for shared layers is hard), while something like cross-stitch could more easily degenerate to only use task-specific layers even when tasks are related?\n\nBeyond performance, analysis on what Mint actually learns would be clarifying. Can the sharing behavior be analyzed by looking at the trained Mint layers? Is Mint actually able to learn both of the extreme settings in practice? The non-synthetic experiments in the paper are only performed on tasks that are closely related. \n\nAs a final note, adding layers to non-Mint models to make the topologically more similar to Mint models may not help these other models. It may make them more difficult to train or overfit more easily, since they are deeper, but do not have Mint method to assist in training. Comparisons without these extra layers would make the experiments more complete. Do cross-stitch and WPL share the conv layers across all tasks like Mint in Table 1? They should to make it a clear comparison.\n\nOther questions:\n-\tWhat exactly are the “two simple neural networks” that produce the goal-specific parameters for goal-conditioned RL? Do these maintain the universality property?\n-\tCan Mint be readily extended to layer types beyond FC layers? This may be necessary when applying to more complex models.\n\n\n[1] Yang, Y. & Hospedales, T. M. “Deep Multi-task Representation Learning: A Tensor Factorisation Approach,” ICLR 2017.\n[2] Long, M., Cao, Z., Wang, J., & Philip, S. Y. “Learning multiple tasks with multilinear relationship networks”, NIPS 2017.\n[3] Argyriou, A., Evgeniou, T., & Pontil, M. “Multi-task feature learning,” NIPS, 2007.\n[4] Kang, Z., Grauman, K., & Sha, F. “Learning with Whom to Share in Multi-task Feature Learning,” ICML 2011.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers put this paper in the lower half and question the theoretical motivation and the experimental design. On the other hand, this seems like an alternative general framework for solving large-scale multi-task learning problems. In the future, I would encourage the authors to evaluate on multi-task benchmarks such as SuperGLUE, decaNLP and C4. Note: It seems there's more similarities with Ruder et al. (2019) [0] than the paper suggests. 

[0] https://arxiv.org/abs/1705.08142
Decision: Reject