TENSOR RING NETS ADAPTED DEEP MULTI-TASK LEARNING

1 INTRODUCTION . Multi-task learning ( MTL ) ( Caruana , 1997 ; Maurer et al. , 2016 ) is an approach for boosting the overall performance in each individual task by learning multiple related tasks simultaneously . In the deep learning context , jointly fitting sufficiently flexible deep neural networks ( DNNs ) to data of multiple tasks can be seen as adding an inductive bias to the deep models , which could be beneficial to learn feature representations preferable by all tasks . Recently , the deep MTL has gained much popularity and been successfully explored in an abroad range of applications , such as computer vision ( Zhang et al. , 2014 ; Misra et al. , 2016 ) , natural language processing ( Luong et al. , 2015 ; Liu et al. , 2017 ) , speech recognition ( Wu et al. , 2015 ; Huang et al. , 2015 ) and so on . However , a number of key challenges posed by the issues of ineffectiveness , inefficiency and inflexibility in deep MTL are left largely unaddressed . One major challenge is how to seek effective information sharing mechanisms across related tasks , which is equivalent to designing better parameter sharing patterns in the deep networks . Some previous work ( Zhang et al. , 2014 ; Yin & Liu , 2017 ) tried to solve this problem by means of hard parameter sharing ( Ruder , 2017 ) , where the bottom layers are all shared except with one branch per task at the top layers . Although being simple and robust to over-fitting ( Baxter , 1997 ) , this kind of architecture can be harmful when learning high-level task-specific features , since it focuses only on common low-level features of all tasks . Moreover , these common features may be polluted by some noxious tasks , leading to the negative transfer in low-level features among tasks ( Yosinski et al. , 2014 ) . An alternative line of work mitigate this issue to some extent by following the soft parameter sharing strategy ( Ruder , 2017 ) , under which one separate DNN is learned for each task with its own set of parameters , and the individual DNNs are implicitly linked by imposing constraints on the aligned weights . The deep MTL models of this type include using ` 2 norm regularization ( Duong et al. , 2015 ) , trace norm regularization ( Yang & Hospedales , 2016 ) and tensor norm priors ( Long & Wang , 2015 ; Long et al. , 2017 ) . The lack of efficiency in model complexity gives rise to another great challenge for current deep MTL . The above soft-sharing based deep models ( one set of parameters per task ) typically involve enormous number of trainable parameters and require extremely large storage and memory . It is thus usually infeasible to deploy those deep MTL models on resource-constrained devices such as mobile phones and wearable computers . Yang & Hospedales ( 2017 ) alleviated the issue by integrating tensor factorization with deep MTL and proposed deep multi-task representation learning ( DMTRL ) . Specifically , they first stack up the layer-wise weights from all tasks and then decompose them into low-rank factors , yielding a succinct deep MTL model with fewer parameters . Despite the compactness of the model , DMTRL turns out to be rather restricted on sharing knowledge effectively . This is because , as shown in Figure 1 , DMTRL ( TT or Tucker ) shares almost all fractions of layer-wise weights as common factors , leaving only a tiny portion of weights to encode the task-specific information . Even worse , such pattern of sharing must be identical across all hidden layers , which is vulnerable to the negative transfer of the features . As an effect , the common factors become highly dominant at each layer and greatly suppress model ’ s capability in expressing task-specific variations . The last challenge arises from the flexibility of architecture in deep MTL . Most of deep MTL models force tasks to have the equal-sized layer-wise weights or input dimensionality . This restriction makes little sense for the case of loosely-related tasks , since individual tasks ’ features ( input data ) can be quite different and the sizes of layer-wise features ( input data ) may vary a lot from task to task . In this work , we provide a generalized latent-subspace based solution to addressing aforementioned difficulties of deep MTL , from all aspects of effectiveness , efficiency and flexibility . Regarding the effectiveness , we propose to share different portions of weights as common knowledge at distinct layers , so that each individual task can better convey its private knowledge . As for the efficiency , our proposal shares knowledge in the latent subspace instead of original space by utilizing a general tensor ring ( TR ) representation with a sequence of latent cores ( Zhao et al. , 2016 ; 2017 ) . One motivation of TR for MTL is it generalizes other chain structured tensor networks ( Cichocki et al. , 2016 ) , especially tensor train ( TT ) ( Oseledets , 2011 ) , in terms of model expressivity power , as TR can be formulated as a sum of TT networks . On the other hand , TR is able to approximate tensors using lower overall ranks than TT does ( Zhao et al. , 2016 ) , thus yielding a more compact and sparselyconnected model with significantly less parameters for deep MTL . Adopting TR-format with much lower ranks could bring more benefits to deep MTL if we tensorize a layer-wise weight of each task into a higher-order weight tensor , since the weight can be decomposed into a relatively larger number but smaller-sized cores . This in turn facilitates the sharing of cores at a finer granularity and further enhances the effectiveness of sharing . Additionally , Zhao et al . ( 2017 ) observed that different cores control different levels of correlations in tensor data , e.g . for a tensorized image , each core affects one specific scale of the resolution . Such observation also provides a natural inspiration for encoding weights of deep MTL via chain based tensor networks and then sharing the cores across tasks . For the last challenge , the flexibility of deep MTL networks is maximally retained in our proposal by parameterizing one DNN per task , while the discrepancy between tasks ’ features is also taken into account by encoding layer-wise weights of different tasks using distinct number of cores . In this way , the network of each task may possess its own size of weight ( or input dimensionality ) . We refer to our framework as tensor ring multi-task learning ( TRMTL ) , as depicted in Figure 1 . With above properties , TRMTL achieves the state-of-the-art performance on a variety of datasets and we validate that each individual task can gain much benefit from the proposed architecture . 2 RELATED WORK . The classical matrix factorization based MTL ( Kumar & Daume III , 2012 ; Romera-Paredes et al. , 2013 ; Wimalawarne et al. , 2014 ) requires the dimensionality of weight vectors { wt ∈ RM } Tt=1 of T tasks to be equal-sized , so that these weights could be stacked up into one weight matrix W ∈ RM×T . Kumar & Daume III ( 2012 ) assumes W to be low-rank and factorizes it as W = LS . Here , L ∈ RM×K consists of K task-independent latent basis vectors , whereas each column vector of S ∈ RK×T is task-specific and contains the mixing coefficients of these common latent bases . ( Yang & Hospedales , 2017 ) extended this matrix based MTL to its tensorial counterpart DMTRL by making use of tensor factorization . Likewise , DMTRL starts by putting the equal-sized weight matrices { Wt ∈ RM×N } Tt=1 side by side along the ‘ task ’ mode to form a 3rd-order weight tensor W ∈ RM×N×T . In the case of CNN , this weight tensor corresponds to a 5th-order filter tensor K ∈ RH×W×U×V×T . DMTRL then factorizes W ( or K ) , for instance via TT-format , into 3 TTcores ( or 5 TT-cores for K ) ( Yang & Hospedales , 2017 ) . Analogously , the first 2 TT-cores ( or the first 4 TT-cores ) play exactly the same role as L for the common knowledge ; the very last TT-core is in fact a matrix ( similar to S ) , with each column representing the task-specific information . Our TRMTL differs widely with DMTRL and generalizes DMTRL from a variety of aspects . In order to reach TRMTL from DMTRL-TT , one needs to take four major types of generalizations ( G1-G4 ) , as demonstrated in Figure 2 . Firstly ( in G1 ) , TRMTL tensorizes the weight into a higherorder weight tensor before factorizing it . By doing so , the weight can be embedded into more latent cores than that of just 3 cores ( or 5 cores ) in DMTRL , which yields a more compact model and makes the sharing at a finer granularity feasible . Secondly ( in G2 ) , DMTRL stringently requires that the first D-1 cores ( D is weight tensor ’ s order ) must be all shared at every hidden layer , only the last vector is kept for private knowledge . By contrast , TRMTL allows for any sharing pattern at distinct layer . Thirdly ( in G3 ) , there is no need for layer-wise weights to be equal-sized and stacked into one big tensor as in TRMTL , each task may have its individual input dimensionality . Finally ( in G4 ) , TRMTL further generalizes TT to TR-format . For each task in DMTRL , the first core must be a matrix and the last core must be a vector ( with both border rank and outer mode size being 1 ) . Notice that our TRMTL also conceptually subsumes DMTRL-Tucker in terms of the first three aspects of generalizations ( G1-G3 ) . It is also worth mentioning that Wang et al . ( 2018 ) only applies TR-format for weight compression in a single deep net , whereas ours incorporates a more general tensor network into the deep MTL context . The two methods differ in goals and applications . Long et al . ( 2017 ) lately proposed MRN which incorporates tensor normal priors over the parameter tensors of the task-specific layers . MRN jointly learns the transferable features as well as multilinear relationship among tasks , with the objective to alleviate both under-transfer and negative-transfer of the knowledge . However , like methods ( Zhang et al. , 2014 ; Ouyang et al. , 2014 ; Chu et al. , 2015 ) , MRN follows the architecture where all the lower layers are shared , which may harm the transferability if tasks are loosely correlated . In addition , the relatedness of tasks is captured by the covariance structures over features , classes and tasks . Constantly updating these covariance matrices ( via SVD in ( Long et al. , 2017 ) ) becomes computationally prohibitive for large scale networks . Compared to above mentioned non-latent-subspace methods , TRMTL is highly compact and hence needs much fewer parameters , which is obviously advantageous in tasks with small sample size . 3 TENSOR PRELIMINARIES . High-order tensors ( Kolda & Bader , 2009 ) are referred to as multi-way arrays of real numbers . Let W ∈ RN1×···×ND be a Dth-order tensor in calligraphy letter , where D is called mode or way . Some very original work have successfully applied tensor decompositions to applications such as imaging analysis ( Vasilescu & Terzopoulos , 2002 ; 2003 ) and computer vision ( Vasilescu , 2011 ) . A recent tensor ring decomposition ( TR ) ( Zhao et al. , 2016 ) decomposes a tensor W into a sequence 3rdorder latent cores that are multiplied circularly . An example of TR-format is illustrated in Figure 3 . In TR-format , any two adjacent latent cores are ‘ linked ’ by a common dimension of size Rk+1 , k ∈ { 1 , ... , D } . In particular , the last core is connected back to the first core by satisfying the border rank condition RD+1 = R1 . The collection of [ R1 , R2 , ... , RD ] is defined as TR-rank . Under TR-format , merely ∑D k=1NkRkRk+1 parameters are needed to represent the original tensorW of size ∏D k=1Nk . Compared with TT-format ( Oseledets , 2011 ) , TR generalizes TT by relaxing the border rank condition . Zhao et al . ( 2016 ) conclude that TR is more flexible than TT w.r.t . low-rank approximation . The authors observe the pattern of ranks distribution on cores tend to be fixed in TT . In TT , the ranks of middle cores are often much larger than those of the side cores , while TR-ranks has no such drawbacks and can be equally distributed on cores . The authors also claim that , under the same approximation accuracy , the overall ranks in TR are usually much smaller than those in TT , which makes TR a more compact model than TT . For more favorable properties , such as TR is invariant under circular dimensional permutation , we refer readers to ( Zhao et al. , 2016 ; 2017 ) .

Summary: This paper studies deep multi-task learning. Prior papers have studied various knowledge sharing approaches for deep multi-task learning including hard and soft sharing schemes. And some soft sharing schemes have used tensor decompositions (including TT, and Tucker). This paper fuses this line of work with the recently proposed Tensor-Ring decomposition in order to obtain Tensor Ring (TR)-based soft sharing for multi-task learning. The results show some improvement over prior deep MTL methods based on other tensor factorisation methods.