Title: Deep Multi-task Representation Learning: A Tensor Factorisation Approach
Authors: Yongxin Yang
Timothy M. Hospedales
Venue: ICLR 2017

Abstract: Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.

TL;DR: A multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network.

Review discussion:
1) As the proposed soft-sharing utilizes task dependent weights, it would be great to discuss the increase of model size compared to the conventional hard-sharing.
Furthermore, experiments that shown the gains of the proposed method are not coming from the extra params would make the paper more convincing.
2) The paper focus on MTL, but the factorization approach can be used more generically, not restricted to MTL tasks. A simple way is to cluster the training data and each cluster can be your "task". In speech community, there's some work on cluster adaptive training of DNN, which does similar factorization for each DNN layer, such as "Cluster Adaptive Training for Deep Neural Network Based Acoustic Model" (http://ieeexplore.ieee.org/document/7364207/)
Discussions on the similarities or differences between them would be interesting.
Question 1: Model capacity and performance.
We list the number of parameters for each model in the first experiment (MNIST, binary one-vs-rest classification, 1% training data) and the performance (mean of error rates for 10-fold).
| Model        | Error Rate | # of param   | Ratio |
| STL          | 11.52      | 4351K        | 1.0   |
| DMTRL-LAF    | 8.25       | 1632K        | 0.38  |
| DMTRL-Tucker | 9.24       | 1740K        | 0.40  |
| DMTRL-TT     | 7.31       | 2187K        | 0.50  |
| UD-MTL       | 9.34       | 436K         | 0.10  |
| UD-MTL-Large | 9.39       | 1644K        | 0.38  |
The conventional hard-sharing method (UD-MTL) design is to share all layers except the top layer. Its number of parameter is roughly the 10% of single task learning method (STL), as most parameters are shared across the 10 tasks corresponding to 10 digits.
Similarly, all our soft-sharing methods significantly reduce the number of parameters compared to STL.
To directly compare our method to UD-MTL, while controlling for network capacity, we expanded UD-MDL by adding more hidden neurons so its number of parameter is close to our methods (Noted as UD-MTL-Large).
However UD-MDL performance does not increase. This is evidence that our model's good performance is not simply due to greater capacity than UD-MTL.
For most contemporary deep learning methods, the number of parameters per-se is not the key factor determining performance, but whether your method can effectively train all those parameters without overfitting, etc.
Therefore another way to interpret this result is that DMTRL provides a more effective approach to training many parameters than UD-MTL.
Nevertheless we emphasise that our focus is on comparison to STL, compared to which our model is much smaller. Recall that our UD-MTL architecture is chosen by brute force search over many possible architectures.
Thus our perspective is that surpassing STL - and even just matching UD-MTL performance - is an interesting achievement for DMTRL, since the sharing is found automatically rather than by brute force search, which is not scalable for larger problems.
Question 2: Re: "Cluster Adaptive Training for Deep Neural Network Based Acoustic Model".
Formalisation: Their formulation is similar to our simplest model, DMTRL-LAF. Compare their model in Page 4, Eq.19 with our Eq.2 in Page 3.
However, we go further by modelling the structure of higher-order tensors, rather than simply flattening the tensor and treating it as a vector.
Thus we proposed DMTRL-Tucker and DMTRL-TT which usually outperform DMTRL-LAF.
Motivation + Application: Our factorisation approach can indeed be used beyond MTL.
For example, in the mentioned paper, they demonstrated that, for a new task, it is possible to only train the coefficients of base models (\lambda in their Eq.19, or "S" in our Eq.2.
In fact we showed something similar but even more interesting in our previous ICLR'15 paper on (shallow) MTL, "A Unified Perspective on Multi-Domain and Multi-Task Learning". By encoding tasks as semantically descriptive vectors, we can generate a model for unseen problem on-the-fly (i.e., without training) by building a parameterization network that maps the task description vector to "S" (Eq.1 in page 3 of https://arxiv.org/pdf/1412.7489v3.pdf).
Thus the deep models developed in this submission are compatible with both these ideas of generating models with minimal or no effort. We didn't write this into our submission because of the volume limit and to keep the paper focused.
This paper proposed a deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network with tensor factorization and end-to-end knowledge sharing. This approach removed the requirement of a user-deﬁned multi-task sharing strategy in conventional approach. Their experimental results indicate that their approach can achieve higher accuracy with fewer design choices.
Although factorization ideas have been exploited in the past for other tasks I think applying it to MTL is interesting. The only thing I want to point out is that the saving of parameter is from the low-rank factorization. In the conventional MTL each layer's weight size can also be reduced if SVD is used.
BTW, recent neural network MTL was explored first (earlier than 2014, 2015 work cited) in speech recognition community. see, e.g.,
Huang, J.T., Li, J., Yu, D., Deng, L. and Gong, Y., 2013, May. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 7304-7308). IEEE.
--- SVD in the conventional MTL
It is possible to reduce the number of parameters in the conventional MTL via SVD or other factorisation techniques, while it is for neural network compression purpose, not for knowledge sharing among tasks.
--- Earlier work
Thanks for mentioning the earlier work in neural network MTL from speech recognition community, we will add the reference in the revised version.
The paper proposed a tensor factorization approach for MTL to learn cross task structures for better generalization. The presentation is clean and clear and experimental justification is convincing.
As mentioned, including discussions on the effect of model size vs. performance would be useful in the final version and also work in other fields related to this.
One question on Sec. 3.3, to build the DMTRL, one DNN per-task is trained with the same architecture. How important is this pretraining? Would random initialization also work here? If the data is unbalanced, namely, some classes have very few examples, how would that affect the model?
--- Random initialisation
In our experiments, we found that the performance is not sensitive to STL-based vs Random initialisation, thus both can be used with our framework. The important aspect of making random initialisation work well is as follows:
Recall that what our model learned is *NOT* the tensor itself, but the factors of tensor, so we are always talking about initialising factors, from which the tensor is constructed by multiplying them out. Randomly initialising those factors is not a good idea, because it is not straightforward to control the distribution of resulting tensor. E.g., you initialise two weight matrices from a specific Gaussian distribution, but the dot-product of them is clearly not the same distribution. It would have some extremely large/small values, which is a very bad starting point for the first round of forward-backward pass.
Therefore, if random initialisation is desired, a solution is to initialise the whole tensor to a suitable distribution of weights first, then do decomposition, and use the decomposed values for initialising the factors. In this way, the resulting re-composed tensor after multiplication has approximately the intended distribution of weight values.
While STL and the above random-based initialisation strategies perform similarly, our STL-based strategy has the advantage that it makes the choice of rank easier and more intuitive. If random initialisation is used, every tensor's low-rank choice(s) must be specified by the user. (Because there is no low-rank structure in a random tensor that can be found automatically).
However, if you stack the STL models, the formed tensor already has a low-rank structure. E.g., the first convolutional layers are likely to look similar for different tasks/domains (like Gabor filters).
Thus, analysing the rank of the stacked STL tensor gives you a good heuristic to set the ranks of the entire network. There is then only one hyper parameter of the entire process, the reconstruction error threshold used to choose the ranks, which has the straightforward interpretation of "prior on sharing strength".
--- Unbalanced tasks/classes
Tasks: The implementation of our loss function is the average of mean error of each task. For each task, we get the mean of its instances' errors, and then we take the average of all those mean errors. This protects the tasks with few instances, because if the loss function was designed to be the mean error of all instances in all tasks, tasks with very small data would be down-weighted.
By default, all our tasks are equally weighted.
However, the user is free to specify a non-uniform per-task weight, so they can make something like "one main task, several less important side tasks". It is also good practice to normalise the errors from different tasks so they are on a similar scale. Especially when your tasks are very different (e.g., one task is classification, another is regression).
Classes: For unbalanced classification problem, there are various classic solutions like re-sampling, class-wise re-weighting, optimisation on AUC instead of accuracy, etc. As our method is about modelling the weights of a neural network, it can work with any of these specially designed losses.
All of these are good-practice engineering points, however, they are not key to our methodology.
--- Revised version
We will add more discussions that address the questions you mentioned before in the revised version.
The paper proposed a nice framework leveraging Tucker and Tensor train low-rank tensor factorization to induce parameter sharing for multi-task learning.
The framework is nice and appealing.
However, MTL is a very well studied problem and the paper considers simple task for different classification, and it is not clear if we really need ``Deep Learning" for these simple datasets. A comparison with existing shallow MTL is necessary to show the benefits of the proposed methods (and in particular being deep) on the dataset. The authors ignore them on the basis of speculation and it is not clear if the proposed framework is really superior to simple regularizations like the nuclear norm. The idea of nuclear norm regularization can also be extended to deep learning as gradient descent are popular in all methods.
--- Comparison with classic (shallow) MTL methods
As the reviewer asked, we provide a comparison with classic MTL methods for the first experiment (MNIST, binary one-vs-rest classification, 1% training data, mean of error rates for 10-fold). PCA feature is obtained by flattening the image then dimension reduction by PCA. CNN feature is obtained by the penultimate layer of the CNN network used in our experiment (a variant of LeNet).
|                          | PCA Feature | CNN Feature |
| Single Task Learning     | 16.89       | 11.52       |
| Evgeniou & Pontil, 2004  | 15.27       | 10.32       |
| Argyriou et al., 2008    | 15.64       | 9.56        |
| Kumar & Daume III, 2012  | 14.08       | 9.41        |
| DMTRL-LAF                | N.A.        | 8.25        |
| DMTRL-Tucker             | N.A.        | 9.24        |
| DMTRL-TT                 | N.A.        | 7.31        |
| UD-MTL                   | N.A.        | 9.34        |
We have previously tried classic MTL methods in our experiments, and DMTRL decisively outperforms them throughout.
Despite these positive results, this comparison was deliberately excluded in the paper, and we briefly explained why in Section 4.1. Here we detail the reasons:
1) As seen above, PCA feature is not competitive with deep learning. This also answers the reviewer's question about if deep learning is necessary here.
2) From an experimental design view, it is not a very scientific comparison to make. If our DMTRL outperforms classic MTL methods with PCA feature: it is not clear if the difference is due to better MTL strategy (ours v.s. others) or more discriminative feature (CNN v.s. PCA).
Alternatively, we can replace the PCA feature by CNN feature (second column of above table).
Then classic MTL methods have two steps: feature extraction via CNN and classifier training by the MTL method. Again we can not conclude that our method is better because the difference could be due to either the MTL method itself, or the optimisation procedure: ours is end-to-end training, and classic MTL + CNN feature approaches have two independent steps.
Therefore, when we designed the experiments, we made sure that all competitors are (i) trained in an end-to-end fashion (ii) all based on the same CNN architecture.
Then we can confidently say that any performance difference is due to different MTL methods, rather than other factors.
3) Classic MTL methods can be a valid baseline for the first experiment only because they are binary classification problems. In the first experiment we cast a multi-class classification problem into multiple binary classification problems, so each class is a task.
Though this is a well established setting in MTL benchmarking, for many practical problems, esp. those needing deep models, each task itself is a multi-class classification problem. We denote this setting as heterogeneous MTL (i.e., our second and third experiment). This setting is usually addressed by some user-defined neural network architecture, and a typical design of which is to keep the bottom layers' parameters tied, and each task has its own top layer(s) parameters. One of the key motivations of this work is to eliminate the need for manually designing such a topology;
instead finding it in a data-driven way. Classic MTL methods, in general, can not apply to this setting of multiple tasks where each task has multiple classes.
Despite these caveats, we can easily add comparisons to the classic MTL methods in the final version.
--- Positioning of this submission
We agree if a problem is well solved by shallow methods, it is not necessary to get deep learning involved. The reviewer may be aware that some very classic MTL problems (datasets) are not evaluated in this submission, for example, School Dataset, Computer Survey Dataset. The reason is exactly what the reviewer mentioned: you do not need deep learning for these datasets. The experiments in this submission are object recognition problems from computer vision community, and for those problems, deep learning (CNN) has a clear advantage.
Then it is desirable to find a systematic way to add MTL upon deep learning.
In summary, the need of deep learning comes first.
--- Extending nuclear norm to deep learning
We appreciate the reviewer mentioned this point, and coincidentally it is our on-going project.
Even though it is totally out of scope of reviewing this submission, we can share some thoughts on this direction.
The nuclear (trace) norm is defined for matrix, and has a close connection to matrix rank.
Thus, some work based on the assumption of low-rank matrix, e.g., multi-task learning and matrix completion, use it as a regularisation term.
The parameters in deep learning or deep MTL are usually in a form of higher-order (>=3) tensors.
But it is not straightforward to extend nuclear norm from matrices to tensors, because tensor rank is non-uniquely defined. We have at least three: CP-, Tucker-, Tensor_Train-rank. For these different definitions of rank, there are correspondingly different ways to extend the nuclear norm. Some of those are easy, e.g., Tucker-rank and Tensor_Train-rank, for which we can define the tensor nuclear norm as a sum of nuclear norms of unfolded matrices from the tensor. CP-rank is tricky. We have not figured out an analytical form of nuclear norm corresponding to it, unless we make some (very strong) assumptions, e.g., assuming the tensor is orthogonally decomposable.
Another challenge is optimisation.
It is true that we can derive the (sub-)gradient of nuclear norm, thus technically we can apply a gradient descent method for it.
However, optimisation experts advised us not to do so because the nuclear norm is essentially non-differentiable. A better alternative is semidefinite programming.
However, deep neural networks are usually trained by gradient descent, and it is challenging to marry these two different optimisation strategies.
Nevertheless, we tried some easy solutions and obtained some interesting results. If the reviewer is interested in this topic, the code and technical report can be found in https://github.com/wOOL/TNRDMTL.
We are keen to hear your views on this direction, as you mention "nuclear norm regularization can also be extended to deep learning as gradient descent are popular".
Dear R1, hopefully we have answered your question satisfactorily. Please let us know if we can provide any further clarification.
Dear readers and reviewers,
We would like to thank all the reviewers that helped us to improve this submission. Based on their comments, we have revised the submission.
The main changed parts are:
1. We discuss some related work in multi-task learning from speech recognition community. (Sec 2, Comment from Reviewer 3).
2. We add a new section in related work to discuss the literature in the line of constructing a MTL-like problem via clustering training data. (Sec 2, Pre-review question from Reviewer2)
3. We give more details about initialisation, particularly how to make the random initialisation condition work in practice. (Sec 4, Comment from Reviewer 2)
Finally, we add two more experiments. Due to page limits, they are placed in the appendix.
A) We compare our methods with some classic (shallow, matrix-based) MTL methods using both classic and CNN features. (Appendix A, Comment from Reviewer 1)
B) We add details about model size in terms of number of parameters, and demonstrate what happens if we make an almost equal sized UD-MTL competitor. (Appendix B, Pre-review question from Reviewer2)

Meta-review:
role: PC
data: comment: The reviews for this paper were quite mixed, with one strong accept and a marginal reject. A fourth reviewer with strong expertise in multi-task learning and deep learning was brought in to read the latest manuscript. Due to time constraints, this fourth review was not entered in the system but communicated through personal communication. 
 
 Pros:
 - Reviewers in general found the paper clear and well written.
 - Multi-task learning in deep models is of interest to the community
 - The approach is sensible and the experiments show that it seems to work
 
 Cons:
 - Factorization methods have been used extensively in deep learning, so the reviewers may have found the approach incremental
 - One reviewer was not convinced that the proposed method would work better than existing multi-task learning methods
 - Not all reviewers were convinced by the experiments
 - The fourth reviewer found the approach very sensible but was not excited enough to champion the paper
 
 The paper was highly regarded by at least one reviewer and two thought it should be accepted. The PCs also agree that this paper deserves to appear at the conference.

Decision: Accept (Poster)