Title: GENERALIZING DEEP MULTI-TASK LEARNING WITH HETEROGENEOUS STRUCTURED NETWORKS

Abstract: Many real applications show a great deal of interest in learning multiple tasks from different data sources/modalities with unbalanced samples and dimensions. Unfortunately, existing cutting-edge deep multi-task learning (MTL) approaches cannot be directly applied to these settings, due to either heterogeneous input dimensions or the heterogeneity in the optimal network architectures of different tasks. It is thus demanding to develop knowledge-sharing mechanism to handle the intrinsic discrepancies among network architectures across tasks. To this end, we propose a flexible knowledge-sharing framework for jointly learning multiple tasks from distinct data sources/modalities. The proposed framework allows each task to own its task (data)-specific network design, via utilizing a compact tensor representation, while the sharing is achieved through the partially shared latent cores. By providing more elaborate sharing control with latent cores, our framework is effective in transferring task-invariant knowledge, yet also being efficient in learning task-specific features. Experiments on both single and multiple data sources/modalities settings display the promising results of the proposed method, especially favourable in insufficient data scenarios.

1. INTRODUCTION:

Multi-task learning (MTL) (Caruana, 1997; Maurer et al., 2016) is an approach for boosting the overall performance of each individual task by learning multiple related tasks simultaneously. In the deep learning setting, jointly fitting sufficiently flexible deep neural networks (DNNs) to data of multiple tasks can be seen as adding an inductive bias to the deep models, which can facilitate the learning of feature representations that are preferable by all tasks. Recently, the deep MTL has been successfully explored in a broad range of applications, such as computer vision (Zhang et al., 2014; Misra et al., 2016) , natural language processing (Luong et al., 2015; Liu et al., 2017) , speech recognition (Wu et al., 2015; Huang et al., 2015) and so on.

Nevertheless, one key challenge in deep MTL remains largely unaddressed, that is, almost all existing deep MTL approaches (Yang & Hospedales, 2017; Long et al., 2017) restrict themselves only to the setting of multi-label learning (or multi-output regression) (Zhang & Yang, 2017) . In other words, different tasks must be fed with input data from the same source (or domain). Such requirement, however, seriously limits the applicability of those models to a more realistic scenario of deep MTL, where the tasks involve distinct data sources (domains) with unbalanced sample sizes or dimensions.

More specifically, tasks from some domains with abundant samples or small input dimensions are relatively easy to handle, whereas tasks from other domains are quite challenging due to the insufficient training data and large dimensionality. For instance, classifying hand-written digits (MNIST dataset (LeCun et al., 1998) ) is somewhat similar to the recognition of hand-drawn characters (Omniglot dataset (Lake et al., 2015) ). The Omniglot task is much harder than the MNIST task, as each character in Omniglot has only 20 training samples, while the input dimensionality is about 15 times larger than MNIST digit. As another example, predicting binary attributes (i.e., 'young', 'bald', 'receding hairline') from human face images (CelebA dataset (Liu et al., 2015) ) ought to be related to the age group classification using human photos taken in the wild (Adience dataset (Eidinger et al., 2014) ). The Adience task turns out to be the more difficult one since the wild images are not preprocessed and 7.6 times fewer than CelebA samples. Hence, it makes good sense to jointly learn these tasks to extract better feature representations, especially for the hard tasks, which could be achieved through transferring domain-specific knowledge from easy tasks.

Unfortunately, existing cutting-edge deep MTL models are only suited for the multi-label learning where different tasks share the same training inputs (i.e., X i = X j for i = j, where X i denotes the input for task T i ), and thus cannot be directly applied to above learning scenarios. This is due to those models fail to provide knowledge-sharing mechanisms that can cope with the intrinsic discrepancies among network architectures across tasks. Such discrepancies either arise from the heterogeneous dimensions of input data or from the heterogeneous designs of layer-wise structures.

Conventionally, knowledge-sharing mechanisms of deep MTL can be hard or soft parameter sharing (Ruder, 2017) . Hard sharing models (Zhang et al., 2014; Yin & Liu, 2017) share all parameters at the lower layers but with no parameters being shared at the upper layers across tasks. Soft sharing models (Duong et al., 2015; Yang & Hospedales, 2016; Long & Wang, 2015) , on the other hand, learn one DNN per task with its own set of parameters, and the tasks are implicitly connected through imposing regularization terms on the aligned weights. The common issue with above mechanisms is that, for the sharing part, the network architectures of all tasks are strictly required to be identical. It turns out that some of the tasks have to compromise on a sub-optimal network architecture, which may lead to the deterioration in the overall performance. Ideally, at all potentially shared layers, each task should be capable of encoding both task-specific and task-independent portions of variation.

To overcome this limitation, we propose a latent-subspace knowledge-sharing mechanism that allows to associate each task with distinct source (domain) of data. By utilizing tensor representation, different portions of parameters can be shared via latent cores as common knowledge at distinct layers, so that each task can better convey its private knowledge. In this work, we realize our proposed framework via tensor ring (TR) format (Zhao et al., 2016) and refer it as tensor ring multi-task learning (TRMTL), as shown in Figure 1 .

Our main contributions are twofold: (1) we offer a new distributed knowledge-sharing mechanism that can address the discrepancies of network architectures among tasks. Compared to existing deep MTL models that are only for multi-label learning, the joint learning of tasks from multi-datasets (multi-domains) with heterogeneous architectures becomes feasible.

(2) we provide a TR-based implementation of the proposed framework, which further enhances the performance of deep MTL models in terms of both compactness and expressive power. 

2. TENSOR PRELIMINARIES:

High-order tensors (Kolda & Bader, 2009) are referred to as multi-way arrays of real numbers. Let W ∈ R N1×•••×N D be a Dth-order tensor in calligraphy letter, where D is called mode or way. Some original work have successfully applied tensor decompositions to applications such as imaging analysis and computer vision (Vasilescu & Terzopoulos, 2002; 2003) . As a special case of tensor networks, the recent TR decomposition (Zhao et al., 2016) decomposes a tensor W into a sequence of 3rd-order latent cores that are multiplied circularly. An example of TR-format is illustrated in Figure 2 . In TR-format, any two adjacent latent cores are 'linked' by a common dimension of size R k+1 , k ∈ {1, ..., D}. In particular, the last core is connected back to the first core by satisfying the border rank condition

R D+1 = R 1 . The collection of [R 1 , R 2 , ..., R D ] is defined as TR-rank. Under TR-format, merely D k=1 N k R k R k+1 parameters are needed to represent the original tensor W of size D k=1 N k .

Compared with tensor train (TT) format (Oseledets, 2011) , TR generalizes TT by relaxing the border rank condition. (Zhao et al., 2016) concludes that TR is more flexible than TT w.r.t. low-rank approximation. The authors observe the pattern of ranks distribution on cores tend to be fixed in TT. In TT, the ranks of middle cores are often much larger than those of the side cores, while TR-ranks has no such drawbacks and can be equally distributed on cores. The authors find that, under the same approximation accuracy, the overall ranks in TR are usually much smaller than those in TT, which makes TR a more compact model than TT.

3. METHODOLOGY:

Our general framework learns one DNN per task by representing the original weight of each layer with a tensor representation layer, i.e., utilizing a sequence of latent cores. Then, a subset of cores are tied across multiple tasks to encode the task-independent knowledge, while the rest cores of each task are treated as private cores for task-specific knowledge.

3.1. TENSOR REPRESENTATION LAYER IMPLEMENTED VIA TENSOR RING FORMAT:

We start the section by describing the tensor representation layer, which lays a groundwork for our deep MTL approach. Our TR-based implementation is called tensor ring representation layer (TRRL). Following the spirit of TT-matrix (Novikov et al., 2015) representation, TR is able to represent a large matrix more compactly via TR-matrix format. Specifically, let W be a matrix of size

M × N with M = D k=1 M k , N = D k=1 N k , which can be reshaped into a Dth-order tensor W ∈ R M1N1×M2N2•••×M D N D via bijective mappings φ(•) and ψ(•). The map φ(i) = (φ 1 (i), ..., φ D (i)) transforms the row index i ∈ {1, ..., M } into a D-dimensional vector index (φ 1 (i), ..., φ D (i)) with φ k (i) ∈ {1, ..., M k };

similarly, the map ψ(•) converts the column index j ∈ {1, ..., N } also into a D-dimensional vector index (ψ 1 (j), ..., ψ D (j)) where ψ k (j) ∈ {1, ..., N k }. In this way, one can establish a one-to-one correspondence between a matrix element W(i, j) and a tensor element W((φ 1 (i), ψ 1 (j)), ..., (φ D (i), ψ D (j))) using the compound index (φ k (•), ψ k (•)) for mode k ∈ {1, ..., D}. We formulate the TR-matrix format as

EQUATION

where 'Tr' is the trace operation. G (k) denotes the kth latent core, while

G (k) [(φ k (i), ψ k (j))] ∈ R R k ×R k+1 corresponds to the (φ k (i), ψ k (j))th slice matrix of G (k) . g (1)T r1 [(φ 1 (i), ψ 1 (j))] represents the r 1 th row vector of the G (1) [(φ 1 (i), ψ 1 (j))] and g (D) r1 [(φ D (i), ψ D (j))] is the r 1 th column vector of G (D) [(φ D (i), ψ D (j))].

Notice that the third line in equation 1 implies TRRL is more powerful than TT layer in terms of model expressivity, as TRRL can in fact be written as a sum of R 1 TT layers. In the deep MTL context, the benefits of tensorization in our TRRL are twofold : a sparser, more compact tensor network format for each task and a potentially finer sharing granularity across the tasks.

3.2. THE PROPOSED KNOWLEDGE-SHARING FRAMEWORK:



3.2.1. THE GENERAL FORMULATION:

Our sharing strategy is to partition each layer's parameters into task-independent TR-cores as well as task-specific TR-cores. More specifically, for some hidden layer of an individual task t ∈ {1, ..., T }, we begin with reformulating the layer's weights W t ∈ R Ut×Vt in terms of TR-cores by means of TRRL, where

U t = Dt k=1 U k t , V t = Dt k=1 V k t . We thereafter reshape a layer's input h t ∈ R Ut into a D t th-order tensor H t ∈ R U 1 t ×•••×U D t t

. Next, the layer's input tensor H t can be transformed into layer's output tensor

EQUATION

where the common TR-cores subset {G (•) com } has C elements which can be arbitrarily chosen from the set of all D t cores, leaving the rest cores {G (•) t } as task-specific TR-cores. Pay attention that our TRMTL neither restricts on which cores to share, nor restricts the shared cores to be in a consecutive order. Finally, we reshape tensor Y t back into a vector output y t ∈ R Vt . Note that the portion of sharing, which is mainly measured by C, can be set to different values from layer to layer. According to equation 2, TRMTL represents each weight element in weight matrix as function of a sequence product of the slice matrices of the corresponding shared cores and private cores. Intuitively, this strategy suggests the value of each weight element is partially determined by some common latent factors, and meanwhile, also partially affected by some private latent factors. Thus, our sharing is carried out in an distributed fashion. This is more efficient than conventional sharing strategies in which each weight element is either 100% shared or 100% not shared.

Although we describe our general framework in terms of TR format, it is straightforward to implement our framework with other tensor network representations, such as Tucker (Tucker, 1966) , TT (Novikov et al., 2015) , projected entangled pair states (PEPS) and entanglement renormalization ansatz (MERA) (Cichocki et al., 2016) , as long as each layer-wise weight matrix is tensorized and decomposed into a sequences latent cores.

3.2.2. SPECIAL CASE FOR CNN SETTING:

Our model can be easily extended to convolutional kernel K ∈ R H×W ×I×O , where H × W is the spatial sizes and I and O are the input and output channels. Note that here TRRL is similar to TR based weight compression Wang et al. (2018) , but we use a different 4th-order latent cores in TR-matrix. As one special case of our general framework (TRMTL-CNN), we just share the spatial cores (height cores and width cores) in the tensorized kernel (via TRRL), while cores corresponding to input/output channels may differ from task to task :

EQUATION

where C is typically 1 for small-sized spatial dimensions. Thus, there is no need to specify how many and which cores to share for TRMTL-CNN. 4 EXPERIMENTAL RESULTS

4.1. EXPERIMENTAL SETTINGS:

We compare our TRMTL with single task learning (STL), MRN (Long et al., 2017) , two variants of DMTRL (Yang & Hospedales, 2017) . We repeat the experiments 5 times and record the average accuracy. The detailed settings and more experimental results are in the supplementary material.

4.1.1. TENSOR REPRESENTATION:

Before the sharing, we first tensorize the layer-wise weight into a Dth-order tensor, whose D modes have roughly the same dimensionality, such that the cores have approximately equal sizes if we assume the same TR-ranks. In this manner, we may measure the fraction of knowledge sharing by the number of shared cores. D is empirically set to be from 4 to 6 in most of our tests. For simplicity, we assume TR-ranks to be identical for all TR-cores across layers for each task. We choose TR-ranks by cross validation on a range of values among 5, 10 and 15. Note that there is no tensorization step in DMTRL in (Yang & Hospedales, 2017) , and DMTRL selects TT-ranks via tensor decomposition according to some specified threshold.

4.1.2. SHARED CORES SELECTION:

Our general TRMTL is highly flexible as we impose no restrictions on which cores to be shared and where to share across tasks. In practical implementation, we may need to trade-off some model flexibility for the ease of sharing-pattern selection by introducing some useful prior knowledge about the domain. For instance, many vision tasks tend to share more cores at the lower layers than upper layers. There are various strategies on how to select the shard cores w.r.t. both the location and the number. Authors of (Zhao et al., 2017) discover that distinct cores control an image at different scales of resolution. The authors demonstrate this by decomposing a tensorized 2D image and then adding noise to one specific core at a time. They show the core in the first location controls smallscale patches while the core in the last location influences on large-scale partitions. Motivated by this, under the general formulation 3.2.1, we preferentially share the features from the detailed scale to the coarse scale, which means we follow a natural left-to-right order in location to select different C number of cores at distinct layers. C is needed to tune via cross validation. In practice, we apply a greedy search on C layer by layer to effectively reduce the searching space. Another practical option is to prune the searching space by following the very useful guidance that C tends to decrease as the layers increase. For certain CNN based architectures, we adopt the special case TRMTL-CNN. Since the cores produced by tensorized convolutional kernel have their specific roles, we just share the cores that are associated to the spatial dimensions (height and width cores), leaving input/output cores being task-specific. In our tests, C is just 1 due to the small spatial kernels, thus eliminating the need of the tuning of this hyper-parameter.

4.2. DEEP MTL WITH SINGLE DOMAIN SOURCE:

We begin our test with data from single domain source to validate the basic properties of our model.

4.2.1. EFFECT OF SHARING PATTERNS:

Our first validation test is conducted on MNIST, where the task A is to classify the odd digits and the task B is to classify the even ones. To see how sharing styles and hyper-parameter C can affect the performance, we examine various patterns from three representative categories, as shown in Figure 3 . For instance, the patterns in 'bottom-heavy' category mean more parameters are shared at the bottom layers than the top layers, while 'top-heavy' indicates the opposite style. The validation is conducted on MNIST using multi-layer perceptron (MLP) with three tensorized hidden layers, each of which is encoded using 4 TR-cores. The pattern '014', for example, means the C are 0, 1 and 4 from lower to higher layers, respectively. We gauge the transferability between tasks with unbalanced training samples by the averaged accuracy on the small-sample tasks. Clearly, the 'bottom-heavy' patterns achieve significantly better results than those from the other two categories.

The pattern '420' is reasonable and obviously outperforms the pattern '044' in Figure 3 , since '044' overlaps all weights at the top layers but shares nothing at the bottom layer. Within each category, TRMTL is robust to small perturbation of C for pattern selection, both '410' and '420' obtain similarly good performance.

4.2.2. EFFECT OF MODEL COMPLEXITY:

We also examine the complexity of the compared models on MNIST. In Table 1 , STL and MRN have enormous 6, 060K and 3, 096K parameters, since they share weights in the original space. DMTRL-Tucker and TT with pre-train trick (Yang & Hospedales, 2017) are parameterized by 1, 194K and 1, 522K parameters. In contrast, TRMTL achieves the best accuracies while the numbers of parameters are significantly down to 16K and 13K. The huge reduction is due to the tensorization and the resulting more sparser TRRL with overall lower ranks.

4.2.3. EFFECT OF UNBALANCED SAMPLE SIZES:

Our next validation is carried out on Omniglot dataset (Krizhevsky & Hinton, 2009) to verify efficacy of knowledge transfer from data-abundance tasks to data-scarcity ones within one source of data domain. Omniglot data Lake et al. ( 2015) consists of 1, 623 unique characters from 50 alphabets with resolution of 105 × 105. We divide the whole alphabets into 5 tasks (task A to task E), each of which links to the alphabets from 10 languages. We now test a more challenging case, where only 1 task (task C) has sufficient samples while the samples of the other 4 tasks are limited. Figure 4 demonstrates the amount of the accuracy changes for each task, both with and without the aid of the data-rich task. We observe our TRMTL is able to make the most of the useful knowledge from task C and significantly boosts the accuracies of all other tasks.

4.2.4. VALIDATION ON RECURRENT NEURAL NETWORKS:

In our last validation, we like to explore whether the proposed sharing mechanism also works for recurrent neural networks. Hence, we test on UFC11 dataset (Liu et al., 2009) that contains 1, 651 Youtube video clips from 11 actions, which are converted to the resolution of 120 × 180 × 3. We assign 5 actions ('basketball', 'biking', 'diving', 'golf swinging' and 'horse back riding') to the task A and leave the rest 6 actions ('soccer juggling', 'swinging', 'tennis swinging', 'trampoline jumping', 'volleyball spiking' and 'walking') as the task B. The RNN is implemented using onelayer long short-term memory (LSTM) with input length of 190. The weights corresponding to the input video are tensorized and encoded into 4 TR-cores, whose input and output dimensions are [8, 20, 20, 18] and [4, 4, 4, 4] , respectively. The TR-rank is set to [2, 2, 2, 2, 2]. Only one layer of cores need to be shared and they are shared in a left-to-right order. The recognition precisions w.r.t. the number of shared cores are recorded in Table 2 . We find that sharing TR-cores between tasks via our TRMTL significantly improves the performance comparing to no sharing case, and sharing all 4 TR-cores achieves the best results for this RNN situation.

4.3. DEEP MTL WITH MULTIPLE DOMAIN SOURCES:

In this section, we show the key advantage of our method in handling multiple tasks defined on distinct data domains, where the optimal network architectures of the tasks could be different. We first verify on Omniglot and MNIST combination, where task A is to classify hand-drawn characters from first 10 alphabets, while task B is to recognize 10 hand-written digits. Task A is much harder than task B, as each character in task A has a very fewer training samples (only 20 per character). Table 3 shows the architecture specification of TRMTL using 4 layers MLP, we can see task A and task B possess their respective layer-wise network structures, while different portions of cores could be partially shared across layers. In contrast, to apply DMTRL, one has to first convert the heterogeneous inputs into equal-sized features using one hidden layer with totally unshared weights, so that the weights in following layers with same shape can be stacked up. In Table 4 , TRMTL obtains similar results to its competitors for the easier MNIST task, while both TRMTL-200 and 211 significantly outperform STL and DMTRL by a large margin for the more difficult Omniglot task. The poor performance of DMTRL due to its architecture's not being able to share any feature at the bottom hidden layer but has to share almost all the features at upper layers. We also conduct experiments on the challenging Office-Home dataset (Venkateswara et al., 2017) to evaluate the effectiveness of TRMTL in handling data from distinct domains. The dataset contains over 10,000 images collected from different domains including Art, Clipart, Product and Real World, which forms task A to task D, respectively. Each task is assigned to recognize 65 object categories presenting in offices or homes. The image styles and the levels of difficulty vary from task to task, e.g., images from Product (task C) have empty backgrounds while images of Art (task A) have complicated backgrounds. We train three FC layers on the features extracted from images of each task using pre-trained VGG-16 (Simonyan & Zisserman, 2014) . In Figure 5 , our TRMTL variants consistently outperform other competitors by a large margin, i.e., over 5% in accuracy for the toughest task A when 80% samples are available. The noticeable improvements are mainly credited to our sharing mechanism, which effectively shares the common signature of object identity across tasks regardless of their individual image styles. For TRMTL, we observe TRMTL-HT exceeds TRMTL-HM by at least 2% in the averaged accuracy and by 1% in the hardest task A, showing the efficacy of employing non-identical architectures on sharing high-level features.

4.3.2. RESULTS ON HETEROGENEOUS CNNS:

To further illustrate the merit of sharing knowledge using heterogeneous architectures, we next apply our TRMTL directly to the raw images via CNNs. We test on two large-scale human face datasets: 5 , in which we adopt the special case of TRMTL by sharing the spatial cores (e.g., [7, 7] , [5, 5] and [3, 3] ) in convolutional kernel yet preserving the differences w.r.t. input and output channel cores.

We focus on comparing the heterogeneous case with the homogeneous case in Table 5 where the shared structures are identical. As expected, in Figure 5 , our TRMTL significantly outperforms other methods on the hard task A. In the meantime, TRMTL obtains the best averaged accuracies on two tasks in nearly all cases, indicating the data-scarcity task A has little harmful impact on the data-abundant task B. For our TRMTL, we also observe TRMTL-HM exhibits worse accuracies than TRMTL-HT, which implies that a comprise on an identical CNN design for all tasks, such as input/output channel core and stride size, lead to deteriorated overall performance. The test also shows the effectiveness of TRMTL in sharing low-level features with heterogeneous architectures.

5. DISCUSSION:

Our general TRMTL framework relies on the manual selection of shared cores, i.e., one need to specify the number of shared cores C at each layer if we choose to share the cores in a left-to-right order across tasks. Although we can employ some efficient heuristics, the search space of this hyperparameter may grow rapidly as number of the layers increase. Besides the greedy search, a more sophisticated and possible option is to automatically select sharable core pairs that have the highest similarity. We may consider two cores as a candidate pair if the same perturbation of the two cores induces similar changes in the errors of respective tasks. In this way, one can adaptively select most similar cores from tasks according to a certain threshold, leaving the rest as private cores.

We should also point out that tensorization operation plays a key role in our proposed sharing mechanism. Due to the tensorization, the cores can be shared in a much finer granularity via our TRMTL framework. Furthermore, tensorizing weight matrix into high-order weight tensor yields more compact tensor network format (with much lower overall ranks), and thus a higher compression ratio for parameters. In contrast, DMTRL tends to produce a lot more parameters without tensorization.

6. CONCLUSION:

In this work, we have extended the conventional deep MTL to a broader paradigm where multiple tasks may involve more than one source data domain. To resolve the issues caused by the discrepancies among different tasks' network structures, we have introduced a novel knowledge sharing framework for deep MTL, by partially sharing latent cores via tensor network format. Our method is empirically verified on various learning settings and achieves the state-of-the-art results in helping tasks to improve their overall performance. of T tasks to be equal-sized, so that these weights could be stacked up into one weight matrix W ∈ R M ×T . The work (Kumar & Daume III, 2012 ) assumes W to be low-rank and factorizes it as W = LS. Here, L ∈ R M ×K consists of K task-independent latent basis vectors, whereas each column vector of S ∈ R K×T is task-specific and contains the mixing coefficients of these common latent bases. Yang & Hospedales (2017) extended this to its tensorial counterpart deep multi-task representation learning (DMTRL) by making use of tensor factorization. Likewise, DMTRL starts by putting the equal-shaped weight matrices {W t ∈ R M ×N } T t=1 side by side along the 'task' mode to form a 3rd-order weight tensor W ∈ R M ×N ×T . In the case of CNN, this weight tensor corresponds to a 5th-order filter tensor K ∈ R H×W ×U ×V ×T . DMTRL then factorizes W (or K), for instance via TT-format, into 3 TT-cores (or 5 TT-cores for K) Yang & Hospedales (2017) . Analogously, the first 2 TT-cores (or the first 4 TT-cores) play exactly the same role as L for the common knowledge; the very last TT-core is in fact a matrix (similar to S), with each column representing the task-specific information.

The fundamental difference between our TRMTL and DMTRL is that ours can tailor heterogeneous network structures to various tasks. In contrast, DMTRL is not flexible enough to deal with such variations with tasks. Specifically, our TRMTL differs widely with DMTRL and generalizes DMTRL from a variety of aspects. In order to reach TRMTL from DMTRL-TT, one needs to take four major types of generalizations (G1-G4), as shown in Figure 6 . Firstly (in G1), TRMTL tensorizes the weight into a higher-order weight tensor before factorizing it. By doing so, the weight can be embedded into more latent cores than that of just 3 cores (or 5 cores) in DMTRL, which yields a more compact model and makes the sharing at a finer granularity feasible. Secondly (in G2), DMTRL stringently requires that the first D-1 cores (D is weight tensor's order) must be all shared at every hidden layer, only the last vector is kept for private knowledge. By contrast, TRMTL allows for any sharing pattern at distinct layer. Thirdly (in G3), there is no need for layerwise weights to be equal-sized and stacked into one big tensor as in TRMTL, each task may have its individual input domains. Finally (in G4), TRMTL further generalizes TT to TR-format. For each task in DMTRL, the first core must be a matrix and the last core must be a vector (with both border rank and outer mode size being 1). Notice that our TRMTL also conceptually subsumes DMTRL-Tucker in terms of the first three aspects of generalizations (G1-G3). It is also worth mentioning that (Wang et al., 2018) only applies TR-format for weight compression in a single deep net, whereas ours incorporates a more general tensor network framework into the deep MTL context. The authors of (Long et al., 2017) lately proposed multilinear relationship network (MRN) which incorporates tensor normal priors over the parameter tensors of the task-specific layers. However, like methods (Zhang et al., 2014; Ouyang et al., 2014; Chu et al., 2015) , MRN follows the architecture where all the lower layers are shared, which is also not tailored for the extended MTL paradigm, and may harm the transferability if tasks are not that tightly correlated. In addition, the relatedness of tasks is captured by the covariance structures over features, classes and tasks. Constantly updating these covariance matrices (via SVD in (Long et al., 2017) ) becomes computationally prohibitive for large scale networks. Compared to these non-latent-subspace methods, TRMTL is highly compact and needs much fewer parameters, which is obviously advantageous in tasks with small sample size. In this section, we also conduct more experiments on CIFAR-10 dataset. We assign 10 classes into 3 tasks, in which task A relates to non-animals; task B comprises 4 animal classes including 'cat', 'dog', 'deer' and 'horse'; task C contains the remaining 2 classes. We like to verify the performance of different models in transferring the useful knowledge from data-abundant task to data-scarcity task within one source of data domain. To this end, we first test on CIFAR dataset with settings where each task may have insufficient training samples like 5%, 10% or 50%. For this test, we adopt the following architecture:

(3 × 64 C3) -(64 × 128 C3) -(128 × 256 C3) -(256 × 512 C3) -(8192 × 1024 FC) -(1024 × 512 FC) -(512 × 10 FC)

, where C3 stands for a 3 × 3 convolutional layer. We employ the general form of TRL on the last two CNN layers and first two FC layers where the most of the parameters concentrate, yielding 4 TR-cores per layer.

Figure 8 illustrates how the accuracies of one task (two tasks) vary with sample fractions, given the remaining two tasks (one task) get access to the full data. We observe the trends in which the accuracies of our model exceed the other competitors by a relatively large margin (shown in solid lines), in the cases of limited training samples, e.g., 5% or 10%. In the mean time, the advantage of our TRMTL is still significant in terms of the averaged accuracies of three tasks (shown in dash lines), which implies the data-scarcity task has little bad influence on the data-abundant tasks.

Table 10 reports the results of our two best patterns ('4431' and '4421') , as well as the one with 'bad' pattern '4444'. Clearly, TRMTL ('4431' and '4421') outperforms other methods in nearly all the cases. As for task A , for instance, the precision of TRMTL-4431 is increased by 1.7% when the data of task C becomes 100%. Even more, such enhancement further grows up to 5.5% in the situation that both task B and C's training samples are fully available. This is in contrast to MRN whose precision improvements are merely 0.4% and 3.0% in the corresponding scenarios. Again, the performance of TRMTL-4431 is superior to that of TRMTL-4444, indicating sharing all nodes like '4444' is not a desirable style.

It is also interesting to get an idea on what our model has learned via the visualization of the high level features. Figure 9 illustrates the task-specific features of our TRMTL (and DMTRL) using t-SNE for the dimensionality reduction. We can see a clear pattern of the clustered features produced by our model that are separated for different classes, which could be more beneficial the downstream classification tasks.

B.1.3 EFFECT OF INPUTS WITH HETEROGENEOUS DIMENSIONS:

In this section, we show the advantage of our method in handling tasks with heterogeneous inputs within single source of data domain. For this test, the tasks are assigned to input images with different spatial sizes or distinct channels (i.e. RGB or grayscale) on CIFAR-10 dataset. In order to apply DMTRL, one has to first convert the heterogeneous inputs into equal-sized features using one hidden layer with totally unshared weights, so that the weights in following layers can be stacked up and factorized. To better show the influence of heterogeneous inputs on the competitors, we adopt MLP with 4 hidden layers. The architectures for the heterogenous spatial sizes case and distinct channels case are shown in Table 11 and 12, respectively. For a good pattern of our TRMTL, such

:



Figures and Tables:

Figure1: The overall sharing mechanisms of multilinear relationship network (MRN), two variants of deep multi-task representation learning (DMTRL) for CNN setting and our TRMTL (general setting and CNN setting) w.r.t. two tasks. The shared portion is depicted in yellow. MRN: original weights are totally shared at the lower layers and the relatedness between tasks at the top layers is modelled by tensor normal priors. DMTRL (TT or Tucker): all layer-wise weights must be equal-shape so as to be stacked and decomposed into factors. For each task, almost all the factors are shard at each layer except the very last 1D vector. Such pattern of sharing is identical at all layers. TRMTL (General): layer-wise weights are separately encoded into TR-formats for different tasks, and a subset of latent cores are selected to be tied across two tasks. The portions of sharing can be different from layer to layer. TRMTL (CNN): spatial cores (height and width cores) in the tensorized convolutional kernel are shared, while cores of input/output channels of the kernel are task-specific.

Figure 2: The diagrams of a 4th-order tensor in TT and TR-format.

Figure 3: Validation test w.r.t sharing patterns and 'C'. The averaged accuracy of two tasks involved with 50 samples. The training samples for task A vs task B are '1800 vs 50' and '50 vs 1800'.

Figure 5: Left two figures: Performance comparison on Home-Office datasets with different fractions of training data. Right two figures: Performance comparison on Adience-CelebA datasets. In all figures, TRMTL-HT and TRMTL-HM correspond to the heterogeneous architectures and homogeneous architectures, respectively.

Figure 6: The demonstration of four types generalizations from DMTRL-TT to our TRMTL.A RELATED WORK

Figure 7: The results of accuracy changes of task A, B, D and E, when the faction of the data for training for task C is increased from 10% to 90%. '+' corresponds to the results after the samples augmentation of task C. 50% data for training for task A, B, D and E. The best sharing pattern of TRMTL is '432'.

Figure 8: Performance comparison of MRN, DMTRL-Tucker, DMTRL-TT and our TRMTL-4431 on CIFAR-10 with different fractions of training data. Top row: 100% data for two of the three tasks, and show the accuracy for the other one task (in solid lines) as well as the averaged accuracy of all three tasks (in dotted lines). Bottom row: 100% data for one of the three tasks, and show the accuracies for the other two tasks (in dashed and solid lines).

Performance and model complexity comparison on MNIST dataset.

Table content: <table><tr><td>Samples 1800 vs 1800</td><td>A</td><td>STL</td><td>B</td><td>MRN A</td><td>B</td><td colspan="3">DMTRL-Tucker DMTRL-TT A B A B</td><td>Ours-410 A B</td><td>Ours-420 A B</td></tr><tr><td>Accuracy</td><td colspan="6">96.8 96.9 96.4 96.6 95.2</td><td>96.2</td><td colspan="2">96.2 96.7 97.5 97.7 97.4 97.6</td></tr><tr><td>Number of parameters</td><td colspan="3">6,060K</td><td colspan="2">3,096K</td><td/><td>1,194K</td><td>1,522K</td><td>16K</td><td>13K</td></tr></table>

Validation of TRMTL for RNN on UFC11.

Table content: <table><tr><td>UFC11</td><td>0</td><td>Number of Shared Cores 1 2 3</td><td>4</td></tr><tr><td colspan="4">Task A (5 actions) 58.5 55.4 55.5 61.2 63.1</td></tr><tr><td colspan="4">Task B (6 actions) 48.0 54.9 56.6 56.0 58.3</td></tr></table>

Specification of network architecture of TRMTL for the Omiglot-MNIST datasets.

Table content: <table><tr><td/><td>Layer</td><td colspan="2">Omniglot (105×105) MNIST (28×28)</td></tr><tr><td>FC1</td><td>in modes out modes</td><td>[7, 7, 5, 5, 3, 3] [7, 7, 5, 5, 3, 3]</td><td>[7, 7, 4, 4] [7, 7, 5, 5]</td></tr><tr><td>FC2</td><td>in modes out modes</td><td>[7, 7, 5, 5, 3, 3] [7, 7, 5, 5, 3, 3]</td><td>[7, 7, 5, 5] [7, 7, 5, 5]</td></tr><tr><td>FC3</td><td>in modes out modes</td><td>[7, 7, 5, 5, 3, 3] [2, 2, 2, 2, 2, 2]</td><td>[7, 7, 5, 5] [2, 2, 2, 2]</td></tr><tr><td>FC4</td><td>in modes out modes</td><td>[64] [10]</td><td>[16] [10]</td></tr></table>

Performance comparison of STL, DMTRL and our TRMTL on Omniglot-MNIST datasets.

Table content: <table><tr><td/><td/><td colspan="4">Omniglot A vs MNIST B</td><td>A</td><td>STL</td><td>B</td><td colspan="6">DMTRL-Tucker DMTRL-TT TRMTL-200 TRMTL-211 A B A B A B A B</td></tr><tr><td/><td/><td/><td colspan="2">50% vs 100%</td><td/><td colspan="4">55.0 98.1 47.3</td><td/><td colspan="3">98.5</td><td>50.0 98.4 58.4 98.3 59.7 98.3</td></tr><tr><td/><td/><td/><td colspan="2">70% vs 100%</td><td/><td colspan="4">60.5 98.1 46.1</td><td/><td colspan="3">98.3</td><td>50.9 98.6 62.9 98.3 65.4 98.3</td></tr><tr><td/><td/><td colspan="3">100% vs 100%</td><td/><td colspan="4">63.3 98.1 50.7</td><td/><td colspan="3">98.5</td><td>52.3 98.5 66.8 98.3 67.5 98.3</td></tr><tr><td/><td>0.58</td><td colspan="2">Office-Home Task A performance</td><td/><td colspan="5">Office-Home Average performance</td><td/><td/><td>0.39</td><td colspan="2">Adience-CelebA Task A performance</td><td>0.65</td><td>Adience-CelebA Average performance</td></tr><tr><td/><td>0.54</td><td/><td/><td/><td>0.72</td><td/><td/><td/><td/><td/><td/><td>0.36</td><td/><td>0.62</td></tr><tr><td>Accuracy</td><td>0.38 0.42 0.46 0.50 0.34</td><td/><td/><td>Accuracy</td><td>0.63 0.66 0.69 0.60 0.57</td><td/><td/><td/><td colspan="2">TRMTL-HT TRMTL-HM DMTRL-TT DMTRL-Tucker STL</td><td>Accuracy</td><td>0.30 0.33 0.27 0.24 0.21</td><td/><td>Accuracy</td><td>0.56 0.59 0.53</td><td>TRMTL-HT TRMTL-HM DMTRL STL</td></tr><tr><td/><td>50%</td><td>60%</td><td>70%</td><td>80%</td><td>50%</td><td/><td>60%</td><td/><td>70%</td><td colspan="2">80%</td><td colspan="2">10%</td><td>50%</td><td>80%</td><td>10%</td><td>50%</td><td>80%</td></tr><tr><td/><td/><td colspan="2">Fraction of Training Samples</td><td/><td/><td colspan="4">Fraction of Training Samples</td><td/><td/><td/><td/><td>Fraction of Training Samples</td><td>Fraction of Training Samples</td></tr></table>

AdienceEidinger et al. (2014) and CelebALiu et al. (2015). Adience dataset contains 26, 580 unfiltered 227 × 227 face photos taken in the wild with variation in appearance, pose, lighting and etc; CelebA has a total number of 202, 599 preprocessed face images of resolution 218 × 178. For this test, the task A assigned to Adience data is to predict the label of age group that a person belongs to (8 classes), and we associate the task B to CelebA data to classify 40 binary facial attributes. Note that task A is much harder than task B, as the number of samples in Adience (with face images in the wild) is about 7.6 times fewer than that of CelebA (with cropped and aligned face images). Since Adience bears similarity to CelebA, we are interested to see whether the performance of the tough task (task A) can be enhanced by jointly learning on two domains of data. The heterogeneous architectures of TRMTL-CNN are shown in Table

Table content: <table/>

Performance comparison of STL, MRN, DMTRL and our TRMTL on Omniglot dataset.

Table content: <table><tr><td>Samples A vs B vs C vs D vs E</td><td>Task</td><td colspan="4">STL MRN DMTRL Tucker Ours-432</td></tr><tr><td/><td>A</td><td>30.4 31.2</td><td>30.5</td><td>28.9</td><td>31.6</td></tr><tr><td/><td>B</td><td>32.4 35.4</td><td>35.3</td><td>32.9</td><td>38.9</td></tr><tr><td>10% vs 10% vs 10% vs 10% vs 10%</td><td>C D</td><td>47.5 47.8 29.2 29.5</td><td>44.1 27.8</td><td>47.9 28.4</td><td>48.2 29.9</td></tr><tr><td/><td>E</td><td>40.5 41.2</td><td>38.7</td><td>43.0</td><td>42.7</td></tr><tr><td/><td colspan="2">Average 36.0 37.0</td><td>35.3</td><td>36.2</td><td>38.3</td></tr><tr><td/><td>A</td><td>61.1 70.1</td><td>63.6</td><td>59.0</td><td>73.6</td></tr><tr><td/><td>B</td><td>66.4 71.7</td><td>69.5</td><td>67.3</td><td>73.0</td></tr><tr><td>50% vs 50% vs 50% vs 50% vs 50%</td><td>C D</td><td>73.1 77.8 55.8 62.1</td><td>75.3 56.8</td><td>70.9 55.8</td><td>80.5 61.0</td></tr><tr><td/><td>E</td><td>68.8 73.0</td><td>70.9</td><td>71.0</td><td>75.4</td></tr><tr><td/><td colspan="2">Average 65.0 70.9</td><td>67.2</td><td>64.8</td><td>72.7</td></tr><tr><td/><td>A</td><td>72.2 78.6</td><td>74.0</td><td>75.5</td><td>80.5</td></tr><tr><td/><td>B</td><td>75.4 80.7</td><td>77.9</td><td>76.4</td><td>79.5</td></tr><tr><td>90% vs 90% vs 90% vs 90% vs 90%</td><td>C D</td><td>82.7 86.5 60.5 69.7</td><td>81.7 65.3</td><td>82.5 62.7</td><td>88.7 72.2</td></tr><tr><td/><td>E</td><td>74.9 82.1</td><td>76.7</td><td>75.4</td><td>80.7</td></tr><tr><td/><td colspan="2">Average 73.1 79.5</td><td>75.1</td><td>74.5</td><td>80.3</td></tr></table>

Performance comparison of STL, MRN, DMTRL and our TRMTL on CIFAR-10 with unbalanced training samples, e.g., '5% vs 5% vs 5%' means 5% of training samples are available for the respective task A, task B and task A. TR-ranks R = 10 for TRMTL.

Table content: <table><tr><td>Samples A vs B vs C</td><td>Task</td><td colspan="5">STL MRN DMTRL Ours-4444 Ours-4431 Ours-4421</td></tr><tr><td/><td>A</td><td>91.4 91.8</td><td>92.2</td><td>90.6</td><td>92.1</td><td>92.2</td></tr><tr><td>100% vs 100% vs 100%</td><td>B C</td><td>80.9 82.3 91.8 93.9</td><td>82.3 92.3</td><td>81.6 93.4</td><td>83.0 94.1</td><td>82.6 93.9</td></tr><tr><td/><td colspan="2">Average 88.0 89.3</td><td>88.9</td><td>88.5</td><td>89.8</td><td>89.6</td></tr><tr><td/><td>A</td><td>72.7 72.9</td><td>73.7</td><td>72.4</td><td>74.4</td><td>74.7</td></tr><tr><td>5% vs 5% vs 5%</td><td>B C</td><td>57.0 60.3 80.6 82.7</td><td>55.5 79.5</td><td>59.0 82.7</td><td>61.3 82.9</td><td>61.5 84.4</td></tr><tr><td/><td colspan="2">Average 70.1 72.0</td><td>69.6</td><td>71.4</td><td>72.9</td><td>73.5</td></tr><tr><td/><td>A</td><td>72.7 73.3</td><td>74.2</td><td>73.6</td><td>76.1</td><td>76.4</td></tr><tr><td>5% vs 5% vs 100%</td><td>B C</td><td>57.0 61.2 91.8 92.1</td><td>56.3 91.5</td><td>60.2 91.8</td><td>62.3 93.1</td><td>63.0 93.0</td></tr><tr><td/><td colspan="2">Average 73.8 75.5</td><td>74.0</td><td>75.2</td><td>77.2</td><td>77.4</td></tr><tr><td/><td>A</td><td>72.7 75.9</td><td>74.3</td><td>76.9</td><td>79.9</td><td>79.8</td></tr><tr><td>5% vs 100% vs 100%</td><td>B C</td><td>80.9 80.2 91.8 93.3</td><td>79.7 92.1</td><td>79.5 92.7</td><td>81.2 93.9</td><td>.81.1 93.9</td></tr><tr><td/><td colspan="2">Average 81.8 83.1</td><td>82.0</td><td>83.0</td><td>85.0</td><td>84.9</td></tr></table>

Peer review (iclr2020):
*Review In this paper, the authors proposed a new framework to jointly learn multiple tasks from heterogeneous sources. Most existing multitask learning frameworks require the input data to come from the same source. To overcome the limitation of those multi-task learning frameworks, the authors proposed to use tensor ring decomposition to decompose the weights of each layer into a sequence of 3rd-order latent cores. Then, some cores were selected to be shared for different tasks. The other cores were the task-specific cores that capture the task-specific features. The number of shared cores could be different for each layer. Experiments were done on multiple datasets. Detail comments: 1. The writing of the paper is not good. The methodology needs to be more detailed. It is hard to follow. 2. The novelty of the paper is not enough for an ICLR paper. It just applied an existing tensor decomposition method to multitask deep neural networks. 2. In the first experiment, the authors divided the MNIST into two tasks. The first one was to classify the odd digits. The second one was to classify the even ones. I didn*t see the reason for splitting MNIST into those two tasks. 3. In figure 5, the authors only provided the performance of the difficult task and the average performance. Please add the performance of the easy task to show how much the performance drops for the easy task. Also, please report the performance on task C for monoglot dataset for 4.2.3. 4. The setting can also be seen as transfer learning between different domains. In the experiments, the authors addressed the difficulty tasks* performance was enhanced. This is the contribution of the proposed method. However, we could see from the experiments that the performance of the easy task dropped. Why not compare with the transfer learning methods that do not affect the performance of the source domain? If the proposed method cannot get better performance compared with those transfer learning methods for difficult tasks, the existing transfer learning methods would be good enough to deal with the settings in this paper. 5. Some baselines are missing. There is no single task performance shown in Figure 4. 6. For 4.2.4, the authors just compared the results of their proposed framework under different settings. Please also provide the performance of other baselines to prove the proposed method is better than others.