Latent Multi-task Architecture Learning	Multi-task learning is motivated by the observation that humans bring to bear what they know about related problems when solving new ones. Similarly, deep neural networks can profit from related tasks by sharing parameters with other networks. However, humans do not consciously decide to transfer knowledge between tasks. In Natural Language Processing (NLP), it is hard to predict if sharing will lead to improvements, particularly if tasks are only loosely related. To overcome this, we introduce Sluice Networks, a general framework for multi-task learning where trainable parameters control the amount of sharing. Our framework generalizes previous proposals in enabling sharing of all combinations of subspaces, layers, and skip connections. We perform experiments on three task pairs, and across seven different domains, using data from OntoNotes 5.0, and achieve up to 15% average error reductions over common approaches to multi-task learning. We show that a) label entropy is predictive of gains in sluice networks, confirming findings for hard parameter sharing and b) while sluice networks easily fit noise, they are robust across domains in practice.

Prior Work as Instances of Sluice Networks	Our meta-architecture is very flexible and can be seen as a generalization over several existing algorithms for transfer and multi-task learning, including  ,  ,  ,  . We show how to derive each of these below. In our experiments, we include hard parameter sharing, low supervision, and cross-stitch networks as baselines. We do not report results for group lasso and frustratingly easy domain adaptation, which were consistently inferior, by some margin, on development data.

Another line of work looks into separating the learned space into a private (i.e. task-specific) and shared space  ,  to more explicitly capture the difference between task-specific and cross-task features. Constraints are enforced to prevent the models from duplicating information.  Bousmalis2016 use shared and private encoders regularized with orthogonality and similarity constraints for domain adaptation for computer vision.  Liu2017 use a similar technique for sentiment analysis. In contrast, we do not limit ourselves to a predefined way of sharing, but let the model learn which parts of the network to share using latent variables, the weights of which are learned in an end-to-end fashion.  Misra2016, focusing on applications in computer vision, consider a small subset of the sharing architectures that are learnable in sluice networks, i.e., split architectures, in which two $n$ -layer networks share the innermost $k$ layers with $0\le k\le n$ , but learn $k$ with a mechanism very similar to $\alpha $ -values. Our method is also related to the classic mixture-of-experts layer  .

In contrast to architecture search, the overall meta-architecture is fixed and the model learns the optimal latent connections and pathways for each task. Recently, a few authors have considered multi-task architecture learning  ,  , but these papers only address a subspace of the possible architectures typically considered in neural multi-task learning, while other approaches at most consider a couple of architectures for sharing  ,  ,  . In contrast, we introduce a framework that unifies previous approaches by introducing trainable parameters for all the components that differentiate multi-task learning approaches along the above dimensions.

Multi-task Architecture Learning	We introduce a meta-architecture for multi-task architecture learning, which we refer to as a sluice network, sketched in Figure 1 for the case of two tasks. The network learns to share parameters between $M$ neural networks—in our case, two deep recurrent neural networks (RNNs)  . The network can be seen as an end-to-end differentiable union of a set of sharing architectures with parameters controlling the sharing. By learning the weights of those sharing parameters (sluices) jointly with the rest of the model, we arrive at a task-specific MTL architecture over the course of training. The two networks $A$ and $B$ share an embedding layer associating the elements of an input sequence, in our case English words, with vector representations via word and character embeddings. The two sequences of vectors are then passed on to their respective inner recurrent layers. Each layer is divided into subspaces (by splitting the matrices in half), e.g., for network $A$ into $G_{A,1}$ and $G_{A,2}$ , which allow the sluice network to learn task-specific and shared representations, if beneficial. The subspaces have different weights.

The output of the inner layer of network $A$ is then passed to its second layer, as well as to the second layer of network $B$ . This traffic of information is mediated by a set of $\alpha $ and $\beta $ parameters similar to the way a sluice controls the flow of water. Specifically, the second layer of each network receives a combination of the output of the two inner layers weighted by the $\alpha $ parameters. Importantly, these $\alpha $ parameters are trainable and allow the model to learn whether to share or to focus on task-specific features in a subspace. Finally, a weighted combination of the outputs of the outer recurrent layers $G_{\cdot ,3,\cdot }$ as well as the weighted outputs of the inner layers are mediated through $\beta $ parameters, which reflect a mixture over the representations at various depths of the multi-task architecture. In sum, sluice networks have the capacity to learn what layers and subspaces should be shared, and how much, as well as at what layers the meta-network has learned the best representations of the input sequences.

Introduction	Multi-task learning (MTL) in deep neural networks is typically a result of parameter sharing between two networks (of usually the same dimensions)  . If you have two three-layered, recurrent neural networks, both with an embedding inner layer and each recurrent layer feeding the task-specific classifier function through a feed-forward neural network, we have 19 pairs of layers that could share parameters. With the option of having private spaces, this gives us $5^{19}=$ 19,073,486,328,125 possible MTL architectures. If we additionally consider soft sharing of parameters, the number of possible architectures grows infinite. It is obviously not feasible to search this space. Neural architecture search (NAS)  typically requires learning from a large pool of experiments with different architectures. Searching for multi-task architectures via reinforcement learning  or evolutionary approaches  can therefore be quite expensive. In this paper, we jointly learn a latent multi-task architecture and task-specific models, paying a minimal computational cost over single task learning and standard multi-task learning (5-7% training time). We refer to this problem as multi-task architecture learning.

In contrast to this approach, our method is designed for multi-task learning and thus encourages a) the sharing of parameters between different task “experts” if this is beneficial as well as b) differentiating between low-level and high-level representations.

Conclusion	We introduced sluice networks, a meta-architecture for multi-task architecture search. In our experiments across four tasks and seven different domains, the meta-architecture consistently improved over strong single-task learning, architecture learning, and multi-task learning baselines. We also showed how our meta-architecture can learn previously proposed architectures for multi-task learning and domain adaptation.

Related Work	Hard parameter sharing  is easy to implement, reduces overfitting, but is only guaranteed to work for (certain types of) closely related tasks  ,  .  Peng2016b apply a variation of hard parameter sharing to multi-domain multi-task sequence tagging with a shared CRF layer and domain-specific projection layers.  Yang2016 use hard parameter sharing to jointly learn different sequence-tagging tasks across languages.  Alonso:Plank:17 explore a similar set-up, but sharing is limited to the initial layer. In all three papers, the amount of sharing between the networks is fixed in advance. In soft parameter sharing  , each task has separate parameters and separate hidden layers, as in our architecture, but the loss at the outer layer is regularized by the current distance between the models.  Kumar2012 and  Maurer2013 enable selective sharing by allowing task predictors to select from sparse parameter bases for homogeneous tasks.  Soegaard:Goldberg:16 show that low-level tasks, i.e. syntactic tasks typically used for preprocessing such as POS tagging and NER, should be supervised at lower layers when used as auxiliary tasks.

Acknowledgements	Sebastian is supported by Irish Research Council Grant Number EBPPG/2014/30 and Science Foundation Ireland Grant Number SFI/12/RC/2289, co-funded by the European Regional Development Fund.

Analysis	To better understand the properties and behavior of our meta-architecture, we conduct a series of analyses and ablations.

Question: Do sluice networks outperform non-transfer learning approaches?
Answer: Yes