Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a variational multi-task learning method that uses Gumbel-Softmax priors for both the weight (w) of each task and the latent representation (z) of each data point. The Gumbel-Softmax trick is used for learning the relatedness between tasks. Amortized inference is also used for faster inference of the posteriors of the weight vectors and also for the latent representations. Experiments on different benchmark datasets show improvement over baselines especially in the low-data regime.
main_review: The paper is in general well-organized and easy to follow. The notations are clear and consistent throughout. Introducing a learnable Gumbel-Softmax prior for learning from relevant tasks is original and interesting. I like the extensive ablation studies on handling limited amounts of data, using gumbel-softmax prior, and amortization. Also the paper considers multiple baselines (both Bayesian and non-Bayesian) for comparison. In particular, the proposed method is able to perform relatively well with limited data.

My major remaining concern about the paper is about the use of different Gumbel-Softmax weights for the weight (w) and latent representation (z). Given that the weight and latent representations are coupled, maybe it makes more sense to have them to be the same? 

I am also curious if there is any other suitable metric for comparing the uncertainty prediction of the Bayesian MTL methods. Currently it seems only accuracy or NSME is used as a metric for evaluation.

limitations_and_societal_impact: Yes. Maybe can also warn about possible negative transfer if tasks are not very related.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors propose to use Gumbel-Softmax as the prior of a task for multi-task learning under the variational Bayesian inference setup. Instead of exploiting task relatedness for representations only, the proposed method jointly infers posteriors for both representations and classifiers. State-of-the-art results are achieved in the multi-input multi-output setting with a small training data regime. 
main_review: Strengths:
* Thorough literature review in terms of deterministic and probabilistic approaches.
* Leverage the advantages of probabilistic-based models and achieve good results on the small data regime.
* Various ablation studies demonstrate the effectiveness of Gumbel-Softmax in MTL environments and effectiveness of the proposed variational Bayesian representation z and classifier w.
* Although Gumbel-Softmax has been widely used in multi-task learning [7,19,42], to the best of my knowledge it is the first time being applied to the variational multi-task learning setup.

Weaknesses:
* It would be great to discuss the actual training and inference time of the proposed method compared to deterministic approaches to see the runtime impact of the additional sampling steps.
* The benefit of a probabilistic-based approach on small data regimes is clearly demonstrated in the experiments, where 3-12 samples per category per task are provided. Can the authors shed some light on the scenario when training with large amounts of data? Is varitionable-based approaches still favorable in that setting?

limitations_and_societal_impact: The main limitation I can think of is the performance of the proposed variational Bayesian approach in the large data regime as mentioned above. I was wondering if there is a way to combine the advantages of the proposed approach with deterministic approaches so that it can generalize well in more settings?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a unified few-shot multi-task learning framework which learns latent representations of data points and latent classifier weights through an amortized variational inference. Specifically, the proposed method regularizes the inference networks by minimizing KL divergence between the target variational distribution and the mixture of the non-target variational distribution. The mixing probability of the non-target variational distribution is sampled through Gumbel-Softmax distribution with learnable parameters. The intuition behind this regularization is that a model is able to learn from related tasks to improve the inference of latent representations and classifier weights (especially useful for the few-shot multi-task learning setting). The extensive experiments demonstrates that the proposed method is more effective than other methods for the few-shot multi-task learning.
main_review: - Originality

This paper tackles the uncertainty of few-shot multi-task learning through a unified probabilistic inference of representation and classifier weights, which is novel in the literature of few-shot multi-task learning. The difference from the related work is also well-explained in Section 3. 

- Quality

The technical quality of this paper is intuitive and well-demonstrated, however, there is some missing part for technical details, such as, KL regularization (see the below questions).  The experimental results are very impressive and coherent to the motivation and design purpose, for example, the proposed method is most useful for the most limited data setting.

- Clarity

This paper is well-written, but there are equations that can lead misunderstandings.

---
I appreciate the authors to address my concern, and I will raise my score.



limitations_and_societal_impact: The below is my and questions and suggestions to improve this paper.

- Although the idea of using an aggregated (or mixture of) variational posterior as a prior distribution is new for multi-task learning, it is already introduced in the literature of improving Variational Autoencoder (https://arxiv.org/pdf/1809.05284.pdf). It would be better if any discussion of this paper is included in the paper.

- This paper regularizes a model to use the data of not-target tasks by minimizing KL divergence between the variational distribution of target tasks and the mixture of the non-target distributions. I am not enthusiastic for the theory, but as far as I know any analytic solution of KL divergence between Gaussian distribution and a mixture of Gaussian distributions does not exists, and only bound exists. Can you elaborate how to compute KL divergence in your framework?

- The function f is introduced in equation 10, however, its usage is not explained in relation to equation 9. Is this function q_\phi in equation 9?

- I didn't understand how to select the class c in equation 10. Is this class is the same class of x_{t, n}? if it is, how it work when we do not know the label of x_{t, n} as equation 9 and 10?

- I have checked the implementation details of supplementary files to understand the above question. As I read, the cross attention f in equation 10 is used for the prior distribution of the latent representation, however, the cross attention layer appears only for the inference network \theta of classifier weights (in Table B.3). This is very different as I read in the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes a probabilistic method (variational inference) to exploit task relatedness and ignore harmful interference in multi-task learning paradigms. In particular, the authors focus on a unique MTL setting (training data is extremely limited; tasks have different input spaces; tasks have the same target space) to highlight the utility of their method. Experimental results on 5 datasets and compared to 6 related work is highly compelling to establishing the efficacy of the proposed method.
main_review: This work is grounded in the probabilistic multi-task learning domain but presents a novel approach: using variational inference to model task relatedness. This method appears highly effective when training data is extremely limited as it augments each task with a strong inductive bias by learning correlation patterns among tasks (Figure 3). Moreover, these correlation patterns appear to be sparse (good) which the authors attribute to using a Gumbel-Softmax to determine task relatedness. 

The submission is clearly written and cogent. The authors motivate their proposed method, outline the situation in which it is useful, offer intuition and explanation in defining methodology and comprehensively present compelling experimental results which adhere to their introduction. I believe the approach is technically sound; although, I did not carefully check for conflicting assumptions or unsupported jumps in the methodology. The extensive empirical evaluation significantly bolsters the author's proposed method.

That said, the work addresses a very limited use case of MTL, and I have not encountered a use case -- and have difficulty imagining a real world scenario -- where this technique would be used. The datasets used in the empirical evaluation are quite unusual: "small" datasets with an even smaller 0.1% -> 20% of the available training data being used for the method to be effective. Moreover, I would posit a transfer learning algorithm or meta learning algorithm would be significantly more suitable to this paradigm than a multi-task learning approach. Additionally, the method is quite involved and complex which reduces its utility and adoption. With regards to presenting novel research concepts, this work suggests their contribution is jointly exploring learned representations and classifiers; however, I would suggest the domain of soft-parameter sharing and even neural architecture search in MTL systems presently address this domain, but with significantly more utility on a variety of MTL applications. Integrating variational inference into multi-task learning is novel, but again, the extensibility of this contribution and application to MTL systems seems extremely limited.  

In summary, this is a very high quality and well-written work, but I struggle to see the significance of the proposed method to either real world systems or as a building block in future work. 
limitations_and_societal_impact: I cannot recall this work containing meaningful discussion of its limitations (aside from the limited scope of its applicability) and societal impact is not addressed.
review_title: null

Meta-review: The reviewers all found the approach to be “original and interesting,” with their main concerns addressed in the discussion. The technique looks to be very useful in the multi-task setting with limited data. During the discussion period, additional experiments showed that it doesn’t hurt in the large-data setting either, although the improvement is less pronounced. To me, this shows that the Bayesian approach seems to be doing the right thing: not getting in the way of single-task performance when enough data is available, but leveraging multiple tasks when data is scarce. One reviewer has concerns that the approach is too complex to be widely adopted, but given that code was made available as part of the submission, I am less concerned about this.
Decision: Accept (Poster)