Multi-task learning (MTL) in deep neural networks is typically a result of parameter sharing between two networks (of usually the same dimensions) BIBREF0 . If you have two three-layered, recurrent neural networks, both with an embedding inner layer and each recurrent layer feeding the task-specific classifier function through a feed-forward neural network, we have 19 pairs of layers that could share parameters. With the option of having private spaces, this gives us $5^{19}=$ 19,073,486,328,125 possible MTL architectures. If we additionally consider soft sharing of parameters, the number of possible architectures grows infinite. It is obviously not feasible to search this space. Neural architecture search (NAS) BIBREF1 typically requires learning from a large pool of experiments with different architectures. Searching for multi-task architectures via reinforcement learning BIBREF2 or evolutionary approaches BIBREF3 can therefore be quite expensive. In this paper, we jointly learn a latent multi-task architecture and task-specific models, paying a minimal computational cost over single task learning and standard multi-task learning (5-7% training time). We refer to this problem as multi-task architecture learning. In contrast to architecture search, the overall meta-architecture is fixed and the model learns the optimal latent connections and pathways for each task. Recently, a few authors have considered multi-task architecture learning BIBREF4 , BIBREF5 , but these papers only address a subspace of the possible architectures typically considered in neural multi-task learning, while other approaches at most consider a couple of architectures for sharing BIBREF6 , BIBREF7 , BIBREF8 . In contrast, we introduce a framework that unifies previous approaches by introducing trainable parameters for all the components that differentiate multi-task learning approaches along the above dimensions.We introduce a meta-architecture for multi-task architecture learning, which we refer to as a sluice network, sketched in Figure 1 for the case of two tasks. The network learns to share parameters between $M$ neural networks—in our case, two deep recurrent neural networks (RNNs) BIBREF12 . The network can be seen as an end-to-end differentiable union of a set of sharing architectures with parameters controlling the sharing. By learning the weights of those sharing parameters (sluices) jointly with the rest of the model, we arrive at a task-specific MTL architecture over the course of training.The two networks $A$ and $B$ share an embedding layer associating the elements of an input sequence, in our case English words, with vector representations via word and character embeddings. The two sequences of vectors are then passed on to their respective inner recurrent layers. Each layer is divided into subspaces (by splitting the matrices in half), e.g., for network $A$ into $G_{A,1}$ and $G_{A,2}$ , which allow the sluice network to learn task-specific and shared representations, if beneficial. The subspaces have different weights.The output of the inner layer of network $A$ is then passed to its second layer, as well as to the second layer of network $B$ . This traffic of information is mediated by a set of $\alpha $ and $\beta $ parameters similar to the way a sluice controls the flow of water. Specifically, the second layer of each network receives a combination of the output of the two inner layers weighted by the $\alpha $ parameters. Importantly, these $\alpha $ parameters are trainable and allow the model to learn whether to share or to focus on task-specific features in a subspace. Finally, a weighted combination of the outputs of the outer recurrent layers $G_{\cdot ,3,\cdot }$ as well as the weighted outputs of the inner layers are mediated through $\beta $ parameters, which reflect a mixture over the representations at various depths of the multi-task architecture. In sum, sluice networks have the capacity to learn what layers and subspaces should be shared, and how much, as well as at what layers the meta-network has learned the best representations of the input sequences.Our meta-architecture is very flexible and can be seen as a generalization over several existing algorithms for transfer and multi-task learning, including BIBREF9 , BIBREF10 , BIBREF6 , BIBREF4 . We show how to derive each of these below.In our experiments, we include hard parameter sharing, low supervision, and cross-stitch networks as baselines. We do not report results for group lasso and frustratingly easy domain adaptation, which were consistently inferior, by some margin, on development data.To better understand the properties and behavior of our meta-architecture, we conduct a series of analyses and ablations.Hard parameter sharing BIBREF0 is easy to implement, reduces overfitting, but is only guaranteed to work for (certain types of) closely related tasks BIBREF30 , BIBREF31 . BIBREF7 Peng2016b apply a variation of hard parameter sharing to multi-domain multi-task sequence tagging with a shared CRF layer and domain-specific projection layers. BIBREF32 Yang2016 use hard parameter sharing to jointly learn different sequence-tagging tasks across languages. BIBREF8 Alonso:Plank:17 explore a similar set-up, but sharing is limited to the initial layer. In all three papers, the amount of sharing between the networks is fixed in advance.In soft parameter sharing BIBREF33 , each task has separate parameters and separate hidden layers, as in our architecture, but the loss at the outer layer is regularized by the current distance between the models. BIBREF34 Kumar2012 and BIBREF35 Maurer2013 enable selective sharing by allowing task predictors to select from sparse parameter bases for homogeneous tasks. BIBREF6 Soegaard:Goldberg:16 show that low-level tasks, i.e. syntactic tasks typically used for preprocessing such as POS tagging and NER, should be supervised at lower layers when used as auxiliary tasks.Another line of work looks into separating the learned space into a private (i.e. task-specific) and shared space BIBREF17 , BIBREF15 to more explicitly capture the difference between task-specific and cross-task features. Constraints are enforced to prevent the models from duplicating information. BIBREF16 Bousmalis2016 use shared and private encoders regularized with orthogonality and similarity constraints for domain adaptation for computer vision. BIBREF28 Liu2017 use a similar technique for sentiment analysis. In contrast, we do not limit ourselves to a predefined way of sharing, but let the model learn which parts of the network to share using latent variables, the weights of which are learned in an end-to-end fashion. BIBREF4 Misra2016, focusing on applications in computer vision, consider a small subset of the sharing architectures that are learnable in sluice networks, i.e., split architectures, in which two $n$ -layer networks share the innermost $k$ layers with $0\le k\le n$ , but learn $k$ with a mechanism very similar to $\alpha $ -values. Our method is also related to the classic mixture-of-experts layer BIBREF36 . In contrast to this approach, our method is designed for multi-task learning and thus encourages a) the sharing of parameters between different task “experts” if this is beneficial as well as b) differentiating between low-level and high-level representations.We introduced sluice networks, a meta-architecture for multi-task architecture search. In our experiments across four tasks and seven different domains, the meta-architecture consistently improved over strong single-task learning, architecture learning, and multi-task learning baselines. We also showed how our meta-architecture can learn previously proposed architectures for multi-task learning and domain adaptation.Sebastian is supported by Irish Research Council Grant Number EBPPG/2014/30 and Science Foundation Ireland Grant Number SFI/12/RC/2289, co-funded by the European Regional Development Fund.

Q: Do sluice networks outperform non-transfer learning approaches?

Q: What is hard parameter sharing?