Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper presents a variational framework for multi-task learning, where for each image classification task, the feature representation z and the classification parameter w of the other tasks serve as the prior for the counterparts of the task. The priors are engaged as the regulariser in the variational inference framework, making the features and the classification parameters of the tasks close to each other. The proposed method outperforms several baselines on several image classification benchmark datasets.

Pros:

- The proposed way of using a Bayesian (variational) approach where the feature and classification parameters of other tasks are leveraged as priors is quite intuitive. As priors naturally serve as regularisations in Bayesian inference, it is natural and might be a principled way of sharing information across multiple tasks. The formulation of multi-task learning with the Bayesian framework might not be completely new in the conventional Bayesian settings, but it seems to be new in the auto-encoding/amortized variational inference settings. Although I'm not an expert in the domain of multi-task learning, I didn't find a similar point of view in the literature.

- I also think the proposal of the prior construction in Eq (9) and Eq (11) is interesting and clever, which is conducted by weighting the posteriors of the other tasks.

- It is also a plus that the authors have done comprehensive oblation study of the proposed framework, showing the different contributions of the components.  To me, the components form an elegant model.

- The paper is easy to follow.

Cons:

- The use of the Gumbel softmax in Eq (9) and Eq (10) is confusing to me. Motivation, intuition, and the learning of the A_ti are a bit unclear. Which parameter is learned in the end? Is it \pi_ti? If so, the \pi_ti is a kind of free parameter to learn. Then why is \pi_ti put through a Gumbel softmax?

- I also feel that the datasets used in the paper are a bit small-scaled in today's machine learning/deep learning areas.

---------------------------------------------------------------------------------------------------------------------------

The authors have addressed my previous concerns. The experiments on a larger dataset are a plus. Therefore, I would recommend an acceptance of the paper.
review_title: A variational framework for multi-task learning

final_score: rating: 7: Good paper, accept
review_content: This paper presents a variational based approach for multi-task learning for the setting where there is a limited amount of training data and for each related task. Prior distributions on model parameters are defined using weighted approximate posterior distributions of other related tasks. This has been done both for classification parameter (w) and latent features (z). The mixing weights are learned in the main optimization using Gumble-softmax prior for each task. The classifier parameters are learned with amortized inference to make posterior inference more effective. The ELBO is learned using MCMC samples using samples partially taken using the reparametrization trick. The experiment section compares the performance of the proposed method against previous multi-task learning approaches. 

The idea of using a variational approach for multi-task learning is novel and interesting. Using posterior distributions of related tasks in a weighted sum for defining prior distributions for parameters of each task is an interesting method that could also inspire more future work in this area.  In terms of the learning algorithm, there are no significant contributions related to novelty, but incorporating various details such as a Gumble-softmax weighting approach, amortized learning, and reparametrization trick have helped this proposed approach to form an overall good performance. 

Proposing this model for multi-task learning problems in which there are limited input samples for each of the related tasks is a problem that has not been explored enough before in the literature despite its importance for many domain applications. Focusing on this problem is an aspect that makes this paper have a high impact on the field. 

The related work section gives a good overview of the previous and current approaches for multi-task learning from classical penalty driven models, to Bayesian approaches and deep learning models. 

The experimental results show a clear improvement of VMTL over the current baselines. I liked that the authors have analyzed the importance of each of the pieces that are used in the model or in the learning algorithm such as the effectiveness of the Bayesian approximation and the choice of softmax for priors.

The paper is clearly written and organized. There are a couple of typos (takes instead of task in section 2.1 and B.3),.. 

Aside from a brief mentioning of cost reduction due to amortized learning, I don't see any computational time analyst. I think it would be very helpful if the authors could provide more information regarding running times compared to the baselines. 

Also, MCMC methods tend to result in high variances especially in the limited data domain. Seeing some analysis on how the variance is affected by the hyperparameter choices and the temperature parameter would be very informative. 

I would be interested to know the results for other multi-task learning-related tasks such as representation learning or robustness against adversarial attacks as the scope of this work doesn't have to be limited to the classification accuracies. Other applications such as NLP related ones could be interesting too as many multi-task learning approaches have been successfully applied there, and it would be nice to see how this could compare to those.

Overall, in my opinion, this paper presents a novel approach for a significant problem, and the experimental results support the claims. That is why I accept this paper.

review_title: Novel approach for multi-task learning using Bayesian inference framework - Good experimental results

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
### Summary
The paper proposes a Bayesian formulation of multi-task learning in the classification scenario.  Specifically, the problem is cast at a variational Bayesian inference problem. The inter-dependency of the tasks is enforced through Gumbbel-softmax priors, with the weights learned from the data. The Bayesian formulation leads to better performance when the amount of training data is low, as shown on three benchmark datasets for image classification.

##########################################################################
### Reasons for score
 Overall, I vote for reject. The methodology is not formulated in a proper Bayesian way as claimed. 
 The experiments, while covering a wide range of datasets and scenarios, do not have any error bars 
 and hence have very low statistical significance.

##########################################################################
### Pros 
1) The use of the Bayesian approach is well motivated for the case of limited data.
2) The explicit inter-task dependency of the classifier weights is introduced, in addition to the 
mode standard approach of interdependent features.
3) The related work section is quite thorough.
4) The experiments cover a wide range of datasets and analyse different aspects of the model: effect 
of variational approximations on latent representations and the classifier weights, particular 
choice of the prior, different amounts of training data available.

##########################################################################
### Cons
1) While other Bayesian multi-task models are mentioned on the related work, their applicability 
for the task at hand is not addressed and the need for this particular approach is not motivated.
2) The model is not formulated in a standard Bayesian way. There is no clear separation between the model and the inference scheme. Specifically, the prior for one task is dependent on the variational posterior for the other tasks and is learned, hence it is not a proper prior. 
3) The experiments do not have any error bars, only one value (i.e. one run). Thus the experiments 
only weakly support the claims and are not statistically significant.

##########################################################################
### Some typos/ minor comments
Eq. (2): Use a different letter in the product, not t

Fig. 1: What do the dashed lines show? Is this an illustration of the model or the inference?

Sec. 4.2 Implementation: You say about using VGGnet as a  "feature extractor in your architecture". 
The phrasing suggests you use it to extract z. I assume you pre-extract x. Clarify, please.

Supplement: Eq. (15): The very last term should have expectation over both q(w) and q(z|x)

Supplement B.3: "the prior of current takes is " -> "the prior of current tasks is"
review_title: Improper Bayesian model formulation, experiments do not have confidence intervals

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: [Summary]

In this paper, the authors propose to perform multi-task learning by casting as a variational inference problem. The key of the technique lies in how task relationships are modelled, through clever use of priors. In this paper, the priors for each task are conditioned on other tasks to enable knowledge transfer. The authors test their method in the small data-regime and show consistently better performance than the baselines.

[Decision]
I think this is a strong paper. The novelty is clear, the method well presented and the experimental section is good. Whilst grading an initial "7", I would potentially like to see (if possible) more results on bigger datasets, instead of restricting to 20%, to better understand the strength of the method when compared to other MTL baselines.

[Decision after reading rebuttal]
The authors appropriately addressed my concerns. The added experiments definitely reinforce the results of the paper and the added text help clarity. I recommend acceptance of this paper and would argue that its quality is reflect by an 8, up from my original score of 7.

[Main points]
1. The decision to cast multi-task learning as a variational inference problem where task relatedness is modelled through use of priors and posteriors is very interesting and well developed.
2. This paper is very well written, and whilst technical, is mostly easy to follow.
3. The ablation studies in addition to the experiments are extensive and sufficient to demonstrate: i) the advantages of the Bayesian modelling and ii) strengths of the method

[Questions]
1.  Why is a Gumbel-Softmax necessary to learn the mixing weights when updating the priors based on the other tasks? From what I can see, this is not a stochastic process that requires sampling from a Categorical. Why not learn $\mathcal{A}$ simply through backprop?
2. Modelling task relatedness is performed through the other task posteriors in Equation 9/10. In cases where they may only be task interference such that tasks should not be shared, how is this dealt with? Would $\mathcal{A}$ collapse to a uniform prior?
3. I am confused about the implementation (Section 4.2.). Is the VGGNet used as the backbone of the inference network for $z$ whilst classifiers on top of this are $w$? You state in Section 2.3. that an MLP is used for amortised inference. A more detailed explanation of the implementation is required to gain a better understanding of how everything is set up.
4. I am a bit confused about how the amortised inference is performed. For a given task $t$, there are $C$ specific labels. In Equation 12, do you mean that that weights for different labels are drawn from different distributions?
5. Whilst not the focus of the paper, how does the method perform when there is more data accessible? It would be interesting to gauge the competitiveness of the method then. In Tables 3 and 4, if we compare VMTL vs MRN, the difference in performance is not so significant. This might point that Bayesian MTL methods, with clever prior construction are the way to go. Results on larger datasets might help ascertain better performance of the method. 

[Other comments]

1. One notable reference is missing from the Related Works section; one that first proposed to use the Gumbel-Softmax for learning task relatedness in MTL in [1].
2. I would like to see a more in depth discussion between your method and [2, 3, etc.]. As your method works comparably to MRN and can be classified as MTL papers with clever priors, I would like to understand more the differences + pros/cons of techniques.
3. Is there a way to inspect what was learned to be shared?
4. It is mentioned in the introduction that a Bayesian framework would allow you to deliver uncertainty estimates over the predictions. Were you able to analyse this?
5. Would your method be amenable to a continual learning scenario where the posterior over all prior tasks could be used for a new task without suffering catastrophic forgetting?


[1] Bragman et al. Stochastic Filter Groups - https://openaccess.thecvf.com/content_ICCV_2019/papers/Bragman_Stochastic_Filter_Groups_for_Multi-Task_CNNs_Learning_Specialist_and_Generalist_ICCV_2019_paper.pdf
[2] MRN paper - https://arxiv.org/abs/1506.02117
[3] MTL with Dirichlet Process Priors - https://www.jmlr.org/papers/volume8/xue07a/xue07a.pdf

 
review_title: Very nice paper that looks at MTL from a variational inference perspective motivated for the small-data regime

Meta-review: This paper presents a probabilistic model for multitask learning with representation learning. The basic idea is to share information across tasks by making the prior over the model parameters of one task conditioned on a convex combination of the variational posteriors of the other tasks.

While some of the reviewers gave high scores and recommended acceptance, one of the reviewers (AnonReviewer3) had some pertinent concerns which lingered even after author response. In particular, AnonReviewer3 mentions that since the prior of one task is conditioned of the variational posteriors of the other tasks, the method is not a proper Bayesian approach. I also read the paper and agree with the assessment. Indeed, the common Bayesian way for multitask learning is to couple the tasks purely based on a prior that encouraging sharing across tasks instead of having task-specific prior that depend on the variational posterior of other tasks as is being done in this paper. 

I also read the reviews and the author response and have some other concerns as well:

- There is a huge amount of prior work on multitask learning, both non-Bayesian as well as Bayesian. Although the paper cites several of those it is disappointing that none of the baselines are Bayesian. Even the non-Bayesian baselines aren't the state-of-the-art recent methods, which is disappointing given the extensive body of prior work in this area.

- The rebuttal wrongly claims MTRL and MRN to be Bayesian methods (included in Table 14 as baselines) whereas they only have a probabilistic formulation and only do point estimation. At a minimum, the paper should show comparison with some Bayesian multitask learning approaches (e.g., shared hierarchical priors, or task clustering, etc). The baselines such as MTRL and MRN aren't among the strongest ones out there. 

- The paper's title is way too generic. There are several multitask learning papers that use variational inference for a Bayesian model. Moreover, given that the basic formulation itself is a bit problematic to be called Bayesian, the title in some sense is also misleading.

Due to the above issues, I don't think the paper can be accepted in its current form.
Decision: Reject