Variational Multi-Task Learning
Authors: Jiayi Shen
Xiantong Zhen
Marcel Worring
Ling Shao
Venue:  (2021)
Keywords: multi-task learning
variational Bayesian inference
Gumbel-softmax priors

Abstract: Multi-task learning aims to improve the overall performance of a set of tasks by leveraging their relatedness. When training data is limited using priors is pivotal, but currently this is done in ad-hoc ways. In this paper, we develop variational multi-task learning - VMTL, a general probabilistic inference framework for simultaneously learning multiple related tasks. We cast multi-task learning as a variational Bayesian inference problem, which enables task relatedness to be explored in a principled way by specifying priors. We introduce Gumbel-softmax priors to condition the prior of each task on related tasks. Each prior is represented as a mixture of variational posteriors of other related tasks and the mixing weights are learned in a data-driven manner for each individual task. The posteriors over representations and classifiers are inferred jointly for all tasks and individual tasks are able to improve their performance by using the shared inductive bias. Experimental results demonstrate that VMTL is able to tackle challenging multi-task learning with limited training data well, and it achieves state-of-the-art performance on four benchmarks, consistently surpassing previous methods.

Under review as a conference paper at ICLR 2021
VARIATIONAL MULTI-TASK LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Multi-task learning aims to improve the overall performance of set of tasks by leveraging their relatedness. When training data is limited using priors is pivotal, but currently this is done in ad-hoc ways. In this paper, develop variational multi-task learning- VMTL, general probabilistic inference framework for si- multaneously learning multiple related tasks. We cast multi-task learning as a variational Bayesian inference problem, which enables task relatedness to be ex- plored principled way by specifying priors. We introduce Gumbel-softmax priors to condition the prior of each task on related tasks. Each prior is represented as a mixture of variational posteriors of other related tasks and the mixing weights are learned in data-driven manner for each individual task. The posteriors over representations and classifiers are inferred jointly for all tasks and individual tasks are able to improve their performance by using the shared inductive bias. Exper- imental results demonstrate that VMTL is able to tackle challenging multi-task learning with limited training data well, and it achieves state-of-the-ar perfor- mance on four benchmark datasets consistently surpassing previous methods.
INTRODUCTION
Multi-task learning (Caruana, 1997) is fundamental learning paradigm for machine learning, which aims to simultaneously solve multiple related tasks to improve the performance of the individual tasks by sharing knowledge. The crux of multi-task learning is how explore task relatedness (Ar- gyriou et al., 2007; Zhang & Yeung, 2012). which is non-trivial since the underlying relationship among tasks can be complicated and highly nonlinear. This has been extensively investigated in pre- vious work by learning shared features, designing regularizers imposed on parameters (Pong et al., 2010; Kang et al., 2011; Jawanpuria et al., 2015; Jalali et al., 2010) or exploring priors over param- eters (Heskes, 2000; Bakker & Heskes, 2003; Xue et al., 2007; Zhang & Yeung, 2012; Long et al., 2017). Recently, deep neural networks have been developed learning shared representations in the feature layers while keeping the classifier layers independent (Yang & Hospedales, 2016; Hashimoto et al., 2016; Ruder et al., 2017). It would be beneficial to learn them jointly by fully leveraging the shared knowledge related tasks, which however remains an open problem.
In our work, we consider a particularly challenging setting, where each task contains limited training data. Even more challenging, have only handful of related tasks gain shared knowledge from. This is in stark contrast to few-shot learning (Gordon et al., 2018; Finn et al., 2017; Vinyals et al., 2016) that also suffers from limited data for each task, but usually have a large number of related tasks. Therefore, in our scenario, it is difficult to learn a proper model for each task independently without overfitting (Long al., 2017: Zhang& & Yang, 2017) and is crucial leverage the inductive bias (Baxter, 2000) provided by various other related tasks that are learned simultaneously. To do so, we employ the Bayesian framework as it is able to deliver uncertainty estimates on predictions and automatic model regularization (MacKay, 1992; Graves, 2011), which makes it well suited for multi-task learning with limited training data. The major motivation of our work to leverage the Bayesian learning framework to handle the great challenges of limited data in multi-task learning.
In this paper, we introduce variational multi-task learning VMTL, a novel variational Bayesian inference approach that can explore task relatedness in a principled way. In order to fully utilize the shared knowledge from related tasks, we explore task relationships in both the feature representation and the classifier by placing prior distributions over them in Bayesian framework. Thus, multi- learning is cast as a variational inference problem for feature representations and classifiers jointly. The introduced variational inference allows us specify the priors by depending on variational pos-

---

Under review as a conference paper at ICLR 2021
teriors of related tasks. To further leverage the shared knowledge from related tasks, introduce the Gumbel-softmar prior to each task, which is a mixture of variational posteriors of other related tasks. We apply the optimization technique (Jang al., 2016) to learn the mixing weights jointly with the probabilistic modelling parameters by hack-propagation. The Gumbel-softmax priors are incorporated into the inference of posteriors to leverage the shared knowledge. We validate the effectiveness of the proposed VMTL by exten- sive evaluation on four challenging benchmarks for multi-task learning. The results demonstrate the benefit of variational Bayesian inference for modeling multi-task learning. VMTL consistently outperforms previous methods in terms of the average accuracy of all tasks.
representations and classifiers, which enable them
METHODOLOGY
In this work, we tackle the challenging multi-task learning setting where only few training sam- ples are available for each task, and only a limited number of related tasks to share knowledge. We investigate multi-task learning under the Bayesian learning framework, where learn the task relationship in a principled way by exploring priors. We cast multi-task learning as a variational inference problem. which offers unified framework learn task relatedness in both feature repre- sentations and task-specific classifiers.
2.1 MULTI-TASK VARIATIONAL INFERENCE
In our setting the tasks are classification problems which share the same label space, but where the samples are drawn from different data distributions. Given set of related tasks (D=) and each task D. {xT yf Jn N, is the number of training samples in the t-th task, the goal under this setting is to predict the label y of the test sample x for all tasks simultancously, using the shared information extracted from other related tasks. We note that the main challenge is the limited number of labeled samples for each task, which makes it difficult to learn a proper model for each task independently (Long al., 2017; Zhang & Yang, 2017).
Under this multi-task learning setting, we consider the Bayesian treatment. For a single task without knowledge sharing from related tasks, place prior over its classifier parameter w, which gives rise to the following data log-likelihood to maximize:
log ((D) = log / p(D(w)p(w)ddv
For multi-task learning, we solve T tasks simultaneously with knowledge sharing among tasks. Thus, after observing data from all T tasks, the true posterior p( p(w,]D) (D,) of a single task becomes (Parr. Using Bayes' rule, we have the posterior for task t follows:
T
p(w,|Dier) x p(w.) II p(D((ss) x pl (Dir P.)pp(,ww))))
(2)
We introduce variational distribution g(w;; 0,) parameterized by 9 for current task to approxi- mate the true posterior, which involves minimizing the Kullback-Leible (KL) divergence between the variational distribution and the true posterior:
arg min DKI. (g(w:; 0.) (Dir))
Generally, the approximate posterior is defined as fully-factorized Gaussian distribution, i.e. q(w;; 0e) A M(u,, af)) (Nguyen et al., 2017; Kingma et al., 2015; Molchanov al., 2017). By ap- plying Eq. (2) into (3) and extending them to all tasks, we obtain evidence lower bound (ELBO) for multi-task learning:
LELRO
1 +>
Σ
Ewiigg (aaoooooogppP2 [log((..ww.) DRr. [o(we: www))>lllww//P..../pa
(4)
where 0 = ={0,T=, is the set of parameters for all -ask-specific classifiers. We maximize the ELBO optimize the model parameters for multi-task learning. is worth noting that for MTL the prior of
2

---

Under review as a conference paper at ICLR 2021
y
W
Wt
Y+
z
x
x
N, ic (1:T)M
M.
Fig. 1. graphical illustration of the proposed model, variational multi-task learning (VMTL). The two dashed lines show the prior of current task depends on the posteriors of other tasks for classifiers and representations. VMTL offers a principled way to explore task relationships: for each task, the priors over the classifiers and feature representations are conditioned on other tasks.
each task is conditioned on other tasks which allows knowledge sharing between them, in contrast to single-task learning without knowledge sharing where uninformative Gaussian priors are generally applicd. Actually, this multi-task ELBO in Eq. (4) provides a general probabilistic inference frame- work that enables task relatedness to be explored in a principled way by leveraging the inductive bias provided by other related tasks (Ruder, 2017). For each task, the conditional prior p( (w (D (Dt) will serve as regularizer for the posterior inference.
In order to fully leverage the shared knowledge from related tasks to improve each individual task, in addition to the classifiers, we would also like share knowledge among the feature representations of samples from different tasks. To this end, we introduce the conditional prior p(z," xx.) over the feature representation zā for each sample x² task relatedness among feature representations in a unified way, as we do for classifiers.
doing so, we are able to explore task
To this end, rewrite the log-likelihood as a sum over the marginal likelihoods of individual data points as follows:
N log ply" Wr, x")
log logp(D:(ww)
(5)
No
The variational Bayesian inference framework developed in Eq. (4) allows the latent representation to be seamlessly incorporated into the log-likelihood in Eq. (5). Under the assumption that w, and z are conditionally independent, we therefore obtain new marginal conditional log- likeliholhoo as follows:
Ogg(y?/we: xf) log / ply?. zi We, xf) = llog| / ply" Iz w. dz?"
(6)
The posterior p(z? xx, y")) of latent representations z, is intractable. Therefore, like VAEs (Kingma & Welling, 2013) we introduce again variational posterior namely g(<:|x;; Ø), which is made conditional on the sample xf. We note that ø is the parameters of the inference network for latent representations, which is optimized jointly with other parameters in our model. By introducing q(z)) x Φ) into Eq. (6) and applying Jensen's inequality, we have
log z w:) lDk [9(2.) x²: 1)(((( xx²))]
logp(y? W., x²)
E Exp-aan
After integrating Eq. (4) and (7), we obtain the following variational objective for multi-task learn- ing:
N; Eigw Enng
1 LVMTL (0, = T
w.]] - DKL. Drt.
DKL (l
(8)
The objective function provides a general probabilistic inference framework for multi-task learning, which allows
jointly explore shared knowledge among representations and classifiers in
3

---

Under review as a conference paper at ICLR 2021
unified way by specifying priors. The detailed derivation of the ELBO for z and w is given in the Appendix A.1. The graphical illustration of our VMTL is shown in Fig. 1.
2.2 LEARNING TASK RELATEDNESS VIA GUMBEL-SOFTMAX PRIORS
The proposed variational multi-task inference framework enables the relationship among tasks be explored by designing priors for both latent representations and classifiers. In Bayesian inference, priors serve as regularisation, which provides principled way of sharing information across multi- ple tasks. We introduce the Gumbel-softmal prior for each task, which is mixture of variational posteriors of other related tasks. In the case of latent representations, we define the prior by using other tasks posteriors of latent representations:
p(z((x))
Σ
Auq(zi|ma;
(9)
SE(L,-.TM
where m, is the mean feature representation of samples from the same class in the i-th task. The mixing weight Au is defined as a binary value to indicate whether two tasks are correlated or not. To enable learning this binary Ari with back propagation, we introduce the Gumbel-softme technique (Jang et al., 2016):
exp((log T. + gx)/TT
Au exp((log Te + gx)/T) r- + cxp((1og(1 - nx)) + gw)/T)
(10)
where gei and Dri are samples taken from a Gumbel distribution, using inverse transform sampling by drawing ~ Uniform(D, 1) and computing g log(- log(u)). Te is the learnable parameter in the Gumbel-softmax technique, which denotes the probability of two tasks are correlated. Parameter 7 is the softmax temperature.
Likewise, we specify the prior over classifier parameters w. in the same way as in Zz:
p(w.(D.T\DD): ==
Σ
(w) A q( ww;;;; 0.),
(11)
VE(I."" TT)
where A(()) is obtained in similar way as in Eq. (10). Note that we use different mixing weights in designing the priors for representations and classifiers, which gave better results than a shared one in our preliminary experiments. This is likely due to the fact that the representations and classifiers leverage different correlation patterns among different tasks.
worth mentioning that the fundamental assumption in multi-task learning is that tasks are related and there is always positive transfer among them. Since tasks share the same label space in our setting, the case with only task interference would hardly happen. case only task interference, the KL term in Eq. (8) will degenerate to an l2 regularization on the representation and classifiers.
2.3 AMORTIZED INFERENCE
We leverage an amortization technique (Gershman & Goodman, 2014), in which we amortize the computational cost of inferring the posterior of the latent representation, as done in VAEs (Kingma & Welling, 2013). In effect, the amortized inference can also be adopted to learn classifier parameters similar the probabilistic prediction in few-shot learning (Gordon al., 2018).
To this end, we design the variational posterior g(w,l') each weight vector w depends only on samples from class of the current task t:
(Dr) in a context-independen manner such that
C
q( (D.) II a(wl)
II ( diag((o?)))).
(12)
where De are the samples from the c-th class, is the size of the shared label space, and each poste- rior is parameterized as diagonal Gaussian distribution. The amortized inference of these posteriors is implemented by multi-layer perceptrons (MLPs) (Gordon et al., 2018; Edwards & Storkey, 2016; Kingma & Welling, 2013) and the parameters of the inference networks are jointly optimized in the end-to-end learning. For a given task, we use the amortized inference to generate the classi- fier weight for each specific class by using the mean feature representations in this class. Thus,
4

---

Under review as a conference paper at ICLR 2021
the weight for different classes drawn from different distributions. In contrast the amortized inference for latent representations, the amortized classifier inference enables the cost to be shared across classes, which reduces the overall cost. Therefore, it offers an effective way handle huge number of object classes and can still produce competitive performance even in the existence of the amortization gap (Cremer al., 2018).
2.4 EMPIRICAL OBJECTIVE FUNCTION
In our implementation for the variational objective for multi-task learning Eq. (8), we adopt Monte Carlo sampling, and obtain the empirical objective function as follows:
N
M
LVMTII (0,
1 +
Σ
[lopply1k m.(() w. []]])]]]]]]]]]]]
MT
mmm
(13)
- DKLL [q xeP: ()(((( xx ]]]
DEL [o(w:::))lp((w Dir WDD]
where z, z,,(() q(2,) x d), w (m) q( f). I and M are the number of Monte Carlo samples. In practice, L and M are set to 10, which performs well while being computationally efficient. We maximize this empirical function to optimize the model's parameters: the log-likelihood term is im- plemented as the cross cross-entropy loss and the KL terms can be computed in closed forms. To sample from the variational posteriors, we adopt the reparameterization trick (Kingma & Welling, 2013). In the posterior inference of classifiers without amortization, we use the local reparameterization trick to reduce the gradient variance (Kingma et al., 2015). The priors p(zz) xx) and p(w, (wrlDrrr)D) implemented with Gumbel-softmax priors provided in Eq. (9) and Eq. (11), respectively, For amortized classifiers, the variational posterior q(w,; 0) is implemented using Eq. (12).
3 RELATED WORKS
Multi-task learning (Caruana, 1997) is a machine learning paradigm that aims to leverage shared knowledge from multiple related tasks to improve the generalization performance of all the tasks simultaneously. The core of multi-task learning is how to explore the task relationship, which has been extensively investigated in the literature.
Early works design feature-based based or parameter- baradeter-based regularizations to explore task relationships (Liu et al., 2009; Obozinski al., 2010; Pong et al., 2010; Jalali et al., 2010; Kang et al., 2011; Jawanpuria et al., 2015; Long et al., 2017; Zhang et al., 2020). Obozinski et al. (2010) are the first to study the multi-task feature selection (MTFS) problem based on the 2,, norm. Liu et al. (2009) propose to use the 1 norm with the objective function to select important features. Kang et al. (2011) design multiple task clusters, aiming to minimize the squared trace norm of the classifier parameters in each cluster.
Bayesian methods (Heskes, 2000; Bakker & Heskes, 2003; Yu et al., 2005; Xue et al., 2007; Guo et al., 2011; Titsias & Lázaro-Gredilla 2011; Zhang & Yeung, 2012; Long et al., 2017) are de- veloped for multi-task learning under probabilistic frameworks, where the regularization usually corresponds to prior. Heskes (2000) proposes Bayesian neural network for multi-task learning and analyse it with an empirical Bayesian framework. Yu et al. (2005) investigate Gaussian pro- cesses for multi-task learning assuming that all models are sampled from common prior. Zhang & Yeung (2012) reformulate the (p.g norm regularizer as a matrix-variate generalized normal prior and utilize the prior information explore task relations. Long et al. (2017) explores tensor normal distribution as priors of network parameters in different layers, which explicitly models the positive and negative relations across features and tasks. Xue et al. (2007) proposes non-parametric hi- erarchical Bayesian model to avoid the high complexity of model parameters and are implemented with a deterministic inference method. Further, Qian et al. (2020) adopt a variational information bottleneck method (Alemi et al., 2016) with uninformative prior distribution to improve the latent probabilistic representation. An important conclusion drawn by (Qian et al., 2020) is that, under ad- versarial attacks, variational latent representations are regularized and thereby expected to be more robust to noise than deterministic latent representations. The idea of conditioning priors on poste- riors is also amenable continual learning in that the posterior of the previous task can be used
5

---

Under review as a conference paper at ICLR 2021
as the prior of the current task to reduce catastrophic forgetting (Nguyen al., 2017; Adel et al., 2019; Ebrahimi et al., 2019). In addition, Bragman et al. (2019) for the first time apply the Gumbel- Softmax to learn task relatedness in multi-task learning. Although these methods also applicable to the data setting in this work and achieve encouraging improvements, they under-perform with very limited training data.
Deep learning has recently been explored for multi-task learning (Misra et al., 2016; Yang & Hospedales, 2016; Hashimoto et al., 2016; Ruder et al., 2017: Long et al., 2017; Meyerson & Mi- ikkulainen, 2017; Chen et al., 2018; Maziarz et al., 2019) by designing different deep architectures to explore task relationships. Liu et al. (2019); Zhang et al. (2020) use hard purameter-sharinn encoder to extract the shared representations, while learning task-specific decoder to obtain the pixel-level predictions. Based on soft parameter sharing, Misra et al. (2016) propose cross-stitch units to allow the model to leverage the shared knowledge from another task. Gao et al. (2020) fol- low the soft parameter sharing mechanism and incorporate neural architecture search into general- purpose multi-task learning. Meyerson & Miikkulainen (2017); Rosenbaum et al. (2017); Maziarz et al. (2019); Strezoski et al. (2019) develop flexible soft-ordering approaches to enable more effee- tive sharing among tasks. Instead of learning the structure of sharing, Kendall et al. (2018) propose to weight multiple loss functions by considering the homoscedastic uncertainty of each task. Chen al. (2018) present gradient normalization algorithm that automatically balances training in deep multi-task models by dynamically tuning gradient magnitudes.
In work, we address multi-task learning in probabilistic inference framework by casting it as a variational Bayesian inference problem. We explore task relationships in a principled way by specifying priors conditioned on other tasks, which enables the model share knowledge among related tasks for learning both representations and classifiers.
4 EXPERIMENTS
We conduct experiments on four benchmark datasets for multi-task learning with limited training data. The results demonstrate the benefits of variational Bayesian approximation to representations and classifiers for multi-task learning. Our VMTL achieves the best performance and consistently surpasses previous methods. We provide more experimental results in the Appendix B.
4.1 DATASETS
We evaluate the proposed VMTL under a challenging multi-task learning setting, where each task has limited training data and only a handful of related tasks that can be learned simultaneously to leverage the shared knowledge. To benchmark our model with previous methods, we conduct experiments on four benchmark datasets, where tasks are defined as classification problems from distinctive domains with shared label space.
Office-Home (Venkateswara et al., 2017) contains images from four domains/tasks: Artistic (A), Clipart (C), Product (P) and Real-world (R). Each task contains images from 65 object categories collected in the office and home settings. There are about 15,500 images in total.
Affice-Caltech (Gong al., 2012) was constructed by selecting the ten categories shared between Office-31 (Saenko et al., 2010) and Caltech-256 datasets (Griffin et al., 2007). One task consists of data from Caltech-256 (C), and the other three tasks consist of data from Office-31 whose images were collected from three distinct domains/tasks, e.g., Amazon (A), Webcam (W) and DSLR (D). There are 8-151 samples per category per task, and 2,533 images in total.
ImageCLEF (Long et al., 2017), the benchmark for the ImageCLEF domain adaptation challenge, contains 12 common categories shared by four public datasets/tasks: Caltech-256 (C), ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Bing (B). There are 2,400 images in total.
DomainNet (Peng et al., 2019), is a large-scale dataset with approximately 0.6 million images dis- tributed among 345 categories. It contains 6 distinct domains: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R) and Sketch (S). This dataset provides an extremely challenging bench- mark, which has never been introduced for multi- learning
We follow the standard evaluation protocol (Zhang & Yeung, 2012; Zhang & Yang, 2017) for multi- task learning and randomly select 5%, 10%, and 20% of samples from each task as the training set

---

Under review as a conference paper at ICLR 2021
Cliprr
Real-world (R)
Awerage
60
ANC Mcc-roo///
Accumoff
VMHTLAC
3256
40%
35%
2000
42%
35%
105
32%
10%
35%
30%
55%
35%
20%
30%
45%
59%
Train Spit
Fig. 2. The performance under different proportions of training data on the Office-Home dataset.
Table 1. Effectiveness of variational Bayesian approximation for representations and classifiers.
Representation (z)
Classifier (w)
5%
10%
20%
X
50.4=0.1
59.510.1
65.6.10.1
51.3=0.2
60.4
66.7=10.2
56.8=0.1
63.7=0.1
68.5=0.1
58.2 0.2
6..0+
69.1+0.1
Table Performance comparison of VMTL with different priors. vidual task are provided in the Appendix B.3. Our Gumbel-softmax produces the best performance.
The detailed results each indi-
Priors
5%
10%
20%
Mean
57.2+0.11
64.5+0.2
68.6=0.2
Learnable weighted
57.0++0.2
6..4
68.6
Gumbel-softtax
58.2+-0.2
6..0
69.1 + 0.1
and use the remaining samples the test set (Long al., 2017). Note that when and 20% labeled data for training, there are on average respectively 3, 6 and 12 samples per category per task. Hence, each task is provided with limited amount of labeled data, which is insufficient for building reliable classifiers without overfitting. In order to maintain this data setting of limited training data, for the large-scale dataset DomainNet, we the split 1%, 2% and 4%, which results in an average of 3, 6 and 12 samples per category per task, respectively. This poses great challenges due the huge number object categories.
5%, 10%
4.2 EXPERIMENTAL SETTINGS
Implementation Details Following the experimental settings in (Long al. 2017), we remove the final classifier layer of VGGnet (Simonyan Zisserman, 2014) and apply the remaining model to extract the feature representation for each sample, from which using amortized inference by MLPs (Kingma & Welling, 2013). 3). In our experiments, the temperature is annealed using the same schedule as applied in (Jang et al., 2016): and gradually anneal it to a small but non-zero value. For the KL-divergence we use the annealing scheme from (Bowman et al., 2015), increasing the weight of the KL-divergence by rate of lc-6 per iteration. The dimension of the latent variable is set to 512. We adopt Adam optimizer (Kingma & Ba, 2014) with learning rate of 1e-4 for training. All the results are obtained based on the 95% confidence interval from five runs.
infer the latent representation
start with high temperature
Comparison Methods We compare our method with single-task learning (STL) as well as a vari- ational version STL (VSTL) implemented by introducing variational Bayesian inference to repre- sentations and classifiers without using task relationship (Details can be found in the Appendix A.2). We also define a basic multi-task learning (bMTL) model, which a deep learning model with shared feature extractor and task-specific classifiers. The bMTL serves as baseline model to demonstrate the benefits of the probabilistic modeling of multi-task learning based on variational Bayesian inference. VMTL is our basic proposed method. VMTL-AC is our proposed method with amortized classifiers. We compare with multilinear relationship network (MRN) (Long et al., 2017), which holds the best performance among previous methods the four benchmarks.
7

---

Under review as a conference paper at ICLR 2021
4.3 EXPERIMENTAL RESULTS
Effectiveness in Handling Limited Data We conduct ablation studies to demonstrate the benefits of the proposed variational multi-task learning in exploring task relatedness. We conduct exper- iments under a large range of train-test splits from 5% to 50%. The results on the Office-Home dataset for different tasks and the average accuracy of all tasks are illustrated in Fig. More de- tailed experimental results are provided in the Appendix B.I. The performance advantage of our VMTL is larger the settings with less training data (S 20%), which demonstrates its effectiveness in handling challenging scenarios with very limited training data.
Effectiveness of Variational Bayesian Approximation We investigate the effect of variational Bayesian inference for representations and classifiers, separately. We conduct these experiments on the Office-Home dataset. The results with different train-test splits are shown in Table 1. More detailed experimental results are provided in the Appendix B.2. As can be seen, both variational Bayesian representations and classifiers can benefit performance. This benefit becomes more sig- nificant when training data is very limited, which indicates the effectiveness of leveraging shared knowledge by conditioning priors on related tasks in our VMTL. In addition, in Fig. 2, the VSTL is shown to strong learning model, which again demonstrates the benefits of variational Bayesian representations and classifiers compared to STL.
Table 3.
Performance comparison of different methods on the Office- dataset for multiple tasks: Artistic (A), Clipart (C), Product (P) and Real-world (R).
Methods
5%
10%
R
Avg.
A
F
Avg.
STL
36.7
30.8
67,5
61.7
49.2
50.4
40.8
74.4
67.5
58.3
VSTT.
37.9
31.3
64.0
52.0
76.2
MRN
53.3
36.4
70.5
67.7
57.0
59.9
42.7
76.3
73.0
63.0
bMIL
37.6:204
31.520.3
68.510.2
63.810.2
50.4=0.1
51.0=022
41.6200.1
76.010.3
69.2:10.3
59.5=0.11
VMTL-AC
52.3.1.4
37.510.5
70.1.00.3
66.7.10.2
56.7_0.2
58.4....5
46.5.10.3
76.910.2
73.1.10.3
63.710.11
VMIL
53.8:0.6
38.62002
71.410..3
68.8:1022
58.2-0.2
60.3=0.5
47.5100.2
78.1100.2
74.2:20.1
6..0=000
Table 4. Performance comparison of different methods on the Office-Caltech Caltech dataset for multiple tasks: Amazon (A), Webcam (W), DSLR (D) and Caltech- (C).
Methods
5%
10%
Avg.
Aug.
STL 87.4
87.9
96.4
82,8
88.6
92.8
97.7
87.8
84.3
90.7
VSTL
88.3
89.1
97.0
81.4
89.0
93.1
96.6
90.0
84.5
91.1
MRN
94.3
89.2
95.0
95.0
bMTL
90.0=20.7
89.4=0.88
95.0+11.1
83.5:0055
89.5-033
93.6=0.1
97.0##0.6
92.1+0.7
86.3#0..4
92.3=0.2
VMTL-AC
93.2-40.3
95.0--0.3
96.1+00.3
89.7440.5
93.5-0.1
94.8-033
96.8-+0..4
97.74-03
90.1++0.3
94.9-022
VMTL
93.8-0.33
95.5=20.4
96.42:044
90.0:20.3
93.9-022
95.5=0.1
97.0##0.1
97,940.3
91.0=+0.1
95.3=0.1
Effectiveness of Gumbel-Softmax Priors The introduced Gumbel-softmax prior provides an ef- fective way learn data-driven task relationships. To demonstrate their effectiveness, compare with several alternatives, including the mean and the learnable weighted posteriors of other tasks. The comparison results are shown in Table 2. The proposed Gumbel-softmax priors perform the best, consistently surpassing other alternatives. It is also worth noting that the advantage of Gumbel- softmax priors is ever larger for very limited training data, e.g., 5%, a challenging scenario where is crucial to leverage task relatedness.
Comparison with other methods The comparison results on the small-scale datasets Office- Home, Office- Caree-CCaatechmmaeaee ILag datasets and large-scale DomainNet datasets are shown in Ta- bles 3, 4, and 6, respectively. The average accuracy of all tasks used for performance measure- ment. The best results of average accuracy Due to the space limitation, we show the experimental results with more data accessible in Table 15 and Table 16 of the Appendix B.4. A comprehensive comparison with more other methods is pro- vided in the Appendix B.4.
marked in bold, while the second-best by underline.
Our VMTL consistently achieves the best performance all small-scale and large-scale datasets with all the defined train-test split settings. It is worth highlighting that on the most challenging set- ting of 5% training data, our VMTL shows large performance advantage compared methods.
8

---

Under review as a conference paper at ICLR 2021
vWm.ac
Training loss
20
10
250
500
1500
1750
2000
Iteration
Fig. 3. The illustration of training loss with iterations. VMTL-AC converges faster than VMTL, which demonstrates its computation benefit by amortized learning.
Table 5. Performance comparison of different methods the ImageCLEF dataset for multiple tasks: Caltech-256 (C), ImageNet II VLS 2012 (I), Pascal VOC 2012 (P), and Bing (B).
Methods
10%
B
F
STL
85.4
71.4
57.7
36.0
62.6
88.9
77.8
64.3
47.6
69.7
VSTL
87.0
73.2
60.5
39.0
89.6
79.1
66.6
48.0
70.8
MRN
90.1
76.5
72.8
54,9
73.7
93.3
83.2
70.4
56.3
75.8
73.2440.5
40.0-10.8
90.6-0.8
66.340.55
72.0=-03
VMTL-AC
89.1=004
81,4+0.9
71.2+00.4
56.+1111
74.5-05
92.3-0.4
83.9+0.8
71.8+0.8
58.9++0.8
76.7=0.2
83.24-0.6
58.340.8
7..9-0.2
Table 6.
Performance comparison of different methods on the large-scaled dataset DomainNer for multiple tasks: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R) and Sketch (S).
1%
Methods
P
Q
R
Avg.
STL
15.0
4.0
19.7
7.5
50.6
9.6
17.7
VSTL
18.9
5.4
23.5
15.2
54.5
12.2
21.6
bMTL
18.3+0..1
5.2+0.2
22.3++0.1
15.0+0.1
53.3+00.1
11.8=0.2
21.0+00.1
VMTL-AC
18.6+00.3
5.7+00.2
23.0++0.2
12.7+0.1
51.6+00.
12.5+0.2
20.7+00.1
VMTL
8.510.1
29.911.0
12.7.0.2
56.910.1
16.9.0.1
25.0.10.0
Specifically, on ImageCLEF, under the 5% setting, our VMTL surpasses the second best by a phe- nomenal margin up 2.3%. This demonstrates the effectiveness of VMTL in exploring relatedness to improve the performance of each task. In addition, our VMTL-AC can also produce compara- ble performance and is better than most previous methods. It is worth mentioning that VMTL-AC demonstrates computation advantages with faster convergence compared to VMTL due to the amor- tized learning as shown in Fig. 3. Besides, we found that VMTL-AC demonstrates good robustness against adversarial attacks. This could be due to that amortized learning applies the mean feature representations to generate classifiers, which is more robust to attacks. The detailed discussions are given in Appendix B.5. Finally, the improvement of VSTL over STL also indicates the benefits of variational Bayesian approximation for representations and classifiers.
5 CONCLUSION
In this paper, we address the multi-task learning problem and tackle challenging setting where each task has very limited amount of training data, with only a handful of related tasks. this end, we develop variational multi-task learning VMTI,, a general probabilistic inference framework for simultancously learning multiple tasks. We cast multi-task lcarning as a variational inference problem, which enables task relationships to be explored in principled way by specifying priors. Specifically, we introduce the Gumbee--offtax priors, which offer an effective way to learn the task relatedness in data-driven manner for each task. We evaluate VMTL four benchmark datasets for multi-task learning. Results demonstrate that our VMTL consistently achieves better or comparable performance with state-of-the-art multi-task learning approaches.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tameem Adel, Han Zhao, and Richard E Turner. Continual learning with adaptive weights (claw). arXiv preprint 1wIiv:1911.099 00551444 2019, 6
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.004100 2016.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In Advances in neural information processing systems, pp. 41-48, 2007. 1, 15
Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4(May):83 2003. 1,5
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from continuous space. arXiv preprint arXiv:/511.06345 2015.
Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In Proceedings of the IEEE International Conference Computer Vision, pp. 1385-1394, 2019.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75. 1997. 1,5
Jianhui Chen, Jiayu Zhou, and Jieping Ye. Integrating low-rank and group-sparse structures for robust multi-task learning. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 42-50, 2011. 15
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pp. 794-803, PMLR, 2018.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality variational autoen- coders. arXiv preprint 1.Xiv:8801.03550 2018. 5
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. arXiv preprint (956::9990022222 2019. 6
Harrison Edwards and Amos Storkey.
Towards
neural statistician.
arXiv preprint
16X::16606001885 2016. 4
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mil-nas: Task-agnostic neural architecture search towards general-purpos multi-task learning. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 11543- 11552, 2020. 6
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Pro-
ceedings of the annual meeting of the cognitive science society, volume 36, 2014.
Boqing Gong, Yuan Shi. Fei Sha, and Kristen Grauman. Geodesic flow kemel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2066-2073. IEEE, 2012.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:14/2.6572. 2014.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint 1805.0900 2018. 1,
10

---

Under review as a conference paper at ICLR 2021
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2348-1241 2356, 2011.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007. 6
Shengbo Guo, Onno Zoeter, and Cédric Archambeau. Sparse bayesian multi-task learning. In Advances in Neural Information Processing Systems, pp. 1755-1763, 2011.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing neural network for multiple nlp tasks. arXiv preprini 1.7.01877 2016.
1, 6
TM Heskes. Empirical bayes for learning to learn. 2000. 1,5
Ali Jalali, Sujay Sanghavi, Chao Ruan, and Pradeep K Ravikumar. A dirty model for multi-task learning. Advances in neural information processing systems, pp. 964-972, 2010. 1,5
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriiztioo with gumbel-softmax, arXiv preprint arXiv:161 1.00 144, 2016. 2, 4,
Pratik Kumar Jawanpuria, Maksim Lapin, Matthias Hein, and Bernt Schiele. Efficient output kernel learning for multiple tasks. In Advances in neural information processing systems. pp. 1189-1197, 2015. 5,5
Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share multi-task feature learning. In ICMI., volume pp. 4, 2011. 1,5
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018. 6
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rXiv:1412.6990 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 2013. 3, 3,5,
arXiv preprint
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in neural information processing systems, pp. 2575-2583, 2015. 2, 5,
Han Liu, Mark Palatueci, and Jian Zhang. Blockwise coordinate descent procedures for the multi- task lasso, with applications to neural semantic basis discovery. In Proceedings of the 26th Annual International Conference Machine Learning, pp. 649 6666,6666200 2009. 5
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871- 1880, 2019. 6
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Yu Philip. Learning multiple tasks with multilinear relationship networks. In Advances in neural information processing systems, pp. 1594-1603, 2017. 1, 2, 5, 6, 15
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- lution, 4(3):4884477 1992.
Krzysztof Maziarz, Ef Kokiopoulou, Andrea Gesmundo, Luciano Sbaiz, Gabor Bartok, and Jesse Berent. Gumbel-matrix routing for flexible multi-task learning. arXiv preprint arXiv:1910.04915, 2019. 6
Elliot Meyerson and Risto Mükkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint 1710001711.00008, 2017. 6

---

Under review as a conference paper at ICLR 2021
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4994-4003, 2016. 6
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. arXiv preprini arXiv: 707700900,,,, 2017. 2, 14
Cuong V Nguyen, Yingzhen Li. Thang D Bui, and Richard Turner. Variational continual learning. arXiv preprint 1exXiv:1100.16600 2017. 2, 6
Guillaume Obozinski, Ben Taskar, and Michael Jordan. Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20(2):231-252, 2010.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019. 6
Ting Kei Pong, Paul Tseng, Shuiwang Ji, and Jieping Ye. Trace norm regularization: Reformu- lations, algorithms, and multi-task learning. SIAM Journal Optimization, 20K6):3465-3444 3489, 2010. 1,5
Weizhu Qian, Bowei Chen, and Franck Gechter. Multi-task variational information bottleneck.
arXiv preprint wrXiv:2007.00339, 2020. 5
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239 2017. 6
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05000 05098, 2017.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Sluice networks: Learning what share between loosely related tasks. star, 1050:23, 2017. 6,
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pp. 213-226. Springer, 2010. 6
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014. 7
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491, 2015, 14
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1375-1384, 2019, 6
Michalis Titsias and Miguel Lárzar-Grediill. Spike and slab variational inference for multi-task and multiple kernel learning. In Advances in neural information processing systems, pp. 2339- 2347, 2011. 5
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 6
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classifica- tion with dirichlet process priors. Journal of Machine Learning Research, 8(Jam):35663, 2007. 1,
Yongxin Yang and Timothy Hospedales. Deep multi-task representation learning: A tensor factori- sation approach. arXiv preprini 160iv:1605.0633) 063391 2016. 1, 6, 15
12

---

Under review as a conference paper at ICLR 2021
Kai Yu, Volker Tresp, and Anton Schwaighofer. Learning gaussian processes from multiple tasks. In Proceedings of the 22nd international conference on Machine learning, pp. 1012-1019, 2005.
5
Yi Zhang, Yu Zhang, and Wei Wang. Deep multi-task learning via generalized tensor trace norm.
arXiv preprint arXiv:2002.04799 2020. 5, 6
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv: 1707,08144, 2017. 1, 2,
Yu Zhang and Dit-Yan Yeung. A convex formulation for learning task relationships in multi-task learning. arXiv preprint arXiv: 2012. 1,5, 6, 15
13

---

Under review as a conference paper at ICLR 2021
A DERIVATION
A.1 DERIVATION OF EVIDENCE LOWER BOUND FOR MULTI-TASK LEARNING
We provide derivation of the evidence lower bound with and jointly The log-likelihood of task / is conditioned on the data from other related tasks.
log p(y//x, DErAA:) log / ply1, Zu, we, |x., ( dz)
(14)
Under the assumption that w, and zr are conditionally independent, we therefore obtain
log plypp(y/xxr (x,, (D.)
/ plyclze, ww)pii,/ ,dw,
=log
log / [/ p(y, iz, ww)p(z 1 \ Ddw
[ [ wx)p(z,(xx pl 1.T' D.)g(wi) dw,
log
((w))
KKL (gwellllllllww..... PPrr [D)]] + E Bg((.)) log
p(yelze, we we)p(zz dzz
Klllg(w [:)] + B. log /
plyslze, we)p(zi x ) (xe) dz.
((2((2())
2 Kllgg(w:)ll((w//D:rrv KL[g(2 xx (x) + By(we,) E Dog ply: Ze, wt)
(15)
A.2 DERIVATION OF EVIDENCE LOWER BOUND FOR SINGLE TASK LEARNING
Generally, the proposed Bayesian inference framework which infers the posteriors of presentations z and classifiers w jointly can be widely applied in other research fields. To be simple, we introduce a variational version of single-task learning (VSTL), and provide the derivation of its evidence lower bound. It is worth noting that single-task learning does not share knowledge among tasks. Thus, the log- logliiill for single-task learning is not conditioned on the data from other related tasks.
log pxy(x) = log / / ply, w, x))wdz
/ p(ylz, w))(w))(z))))ww
log
/ p(ylz, d))pz)))ddp(()dd
log
[.] p(ylz, w)p(z)x)dd) (w(w) dw
(16)
log
q(w)
KLL + ( [log
p(ylz, w)p(z(x)dz)
p(ylz, w () dz)
KLL[(())lll(w) + B (w()) Dog
((2(x)
KKLg(())lp((w)) - KLL(g()))) [(z + By(w) Be(2)x)) (logg (loggly/z. w]]
Usually, the approximate posteriors q(w) and (z(xx) are defined as a fully-factorized Gaussian distribution. Due to lack of extracted information offered by other related tasks, the priors p( o() and p(z[x) are set to a standard Gaussian distribution, as applied in (Sohn et al., 2015; Kingma et al., 2015; Molchanov et al., 2017).
14

---

Under review as a conference paper at ICLR 2021
B EXTRA EXPERIMENTAL RESULTS
B. EFFECTIVENESS IN HANDLING LIMITED DATA
We further provide detailed information in Table about average accuracy in Fig. 2. Our pro- posed probabilistic models, i.e., VMTI, and VMTI,-CC outperform the deterministic baseline multi- task learning model (bMTL), which demonstrates the benefits of our proposed variational Bayesian framework. Given limited amount of training data, STL and VSTL not train proper model for each task. As the training data decreases, our methods based on the variational Bayesian framework are able to better handle this challenging case by incorporating the shared knowledge into the prior of each tasks. The best results of average accuracy are marked in bold, while the second- best by underline.
Table 7. Performance of average accuracy under different proportions of training data on Office-
Home.
Methods
10%
25%
35%
45%
STI.
49.2
58.3
61.3
64.9
66.4
67.7
68.3
70.3
70.4
71.9
VSTL
51.1
60.2
63.0
65.8
67.9
69.6
70.3
72.4
72.3
73.8
bMTL
50.4100.1
59.510.1
62.4100.1
65.6:10.1
66.810.11
68.310.2
68.910.11
70.520.1
70.710.2
72.3::02
VMTL--AC
56.6.10.2
63.7.10.1
65.4
68.1
68.7.10.1
69.4.0.22
70.0.10.1
70.910.11
70.4..0.11
71.5
VMTL
58.2:00.2
65.0:1000
66.4.10.1
69.11001
69.9.1.11
70.4:0.1
71.3:10.1
72.2:0.11
72.02:0.1
73.32:0.1
B.2 EFFECTIVENESS VARIATIONAL BAYESIAN APPROXIMATION
The comparison results on performance Bayesian approximation for representations z and clas- sifiers on the Office-Home, Office-Caltech and ImageCIEF datasets are shown in Tables 8, 9 and 10, respectively. Both variational Bayesian representations and classifiers can benefit perfor- mance. And we find that Bayesian classifiers in the variational inference framework contribute more to the performance than Bayesian representations. It is likely due to the fact that Bayesian classifiers can better improve the model's 's discriminative ability. Our method jointly infers the posteriors over feature representations and classifiers in a Bayesian framework, which consistently outperforms its variants on three benchmarks.
B.3 EFFECTIVENESS GUMBEL-SOFTMAN PRIORS
The performance comparison of the proposed VMTL with different priors on the Office- Office-Caltech and ImageCLEF datasets is shown in Tables 11, 12 and 13, respectively. "Mean" denotes that the prior of the current task is the mean of variational posteriors of other related tasks. "Learnable weighted" denotes that weights of mixing the variational posteriors of other related tasks are learnable. Our cumbel-ssfftax Priors apply the Gumbel-softmat technique to learn the mixing weights, which introduces uncertainty to the relationships among tasks in order to explore sufficient transferable information from other tasks. In the three datasets, our designed priors outperform other methods consistently.
B.4 A COMPREHENSIVE COMPARISON WITH OTHER METHODS
The comprehensive comparison with state-of- teff-the-r art methods, including multi-task feature learning (MTFL) (Argyriou et al., 2007), robust multi-task learning (RMTL) (Chen et al., 2011), multi-task relationship learning (MTRL) Zhang Yeung (2012), deep multi-task learning with tensor factor- ization (DMRL-TF) (Yang & Hospedales, 2016) and multilinear relationship network (MRN) (Long et al., 2017) is shown in Table 14. The results of the above state-of-the-art methods are taken from paper (Long et al., 2017). The results of three datasets under the 20% train-test split and the results of DomainNer under the 2% and 4% train-test split are is provided in Table 15 and Table 16, re- spectively. The proposed VMTL consistently achieves the best performance all datasets with all train-test split settings. VMTL with amortized classifiers (VMTL-AC) can produce competitive performance better than most of previous methods.
15

---

Under review as a conference paper at ICLR 2021
B.5 ROBUSTNESS OF OUR METHODS
We conduct some experiments to show the robustness of our methods against adversarial attacks. In our experiments, the adversarial attack is implemented by the fast gradient sign method (Goodfellow et al,, 2014) where e denotes the noise level. We evaluate our proposed VMTL, VMTL-AC, and the basic multi-task (eal) on the Office-home dataset. As shown in Fig. 4, under different noise levels, VMTL outperforms bMTL. As the noise level increases, the variant of our method VMTL-AC is more robust and significantly outperforms the baseline multi-task learning model.
Art (A]
Cipart
Product (P)
Realoworid
Awerage
VMTI
MNTT
20
62
03
24
55
63
83
32
63
04
15
08
53
23
02
03
04
03
86
96
01
02
03
54
05
03
Fig. 4. The performance for each task under different noise level on the Office-Home dataset.
16

---

Under review as a conference paper at ICLR 2021
65.6 65.62:01
06.7 06.7:00.2
68 33.51.0.1
69.1
93.1
93.3-022
95.3++022
95.3-10.1
76.8+0.3
78.3 78.3:1022
79 79.5-0222
80.3
Avg.
Avg.
Avg.
72.910.4
76.010..3
76.6
8688102
87.540.3
89
90.6:0.5
56.+0..8
57.5108
50.0++04
39.710.3
74.
Table 9. Detailed results on performance of Bayesian approximation for representation Z and classifier W on Office Caltech.
Table 8. Detailed results on performance of Bayesian approximation for representation z and classifier w on Office Home.
80.910.3
81 81.8:00.2
82.010..
82.5+0.22
96.0.1.22
95.210.88
99.2+0.3
99.240.6
Table 10. Detailed results on performance of Bayesian approximation for representation and classifier W on ImageCLER
71.940.8
74.010.2
77.1+0.8
77.9004
20%
20%
20%
P
51.8
53,2:10.3
53.5110.3
53.3-1033
94.5
94,9:10.2
96.6--0.3
95,82044
46.5
88. 1::0.5
88. 1=005
89.71003
W
56.610.33
57.8
62.5.:04
64.0=00.3
92.310.2 95.0:10.2
95.7-10.1
95.6-+0.2
95.6-10.1
92.9+0.5
93.5-10.2
93.8-0.5
94.0210.2
60.5
63.710.11
65.0=000
93,2-02
95.1+0.11
95.3101
72.0=n.3
72.6-0.33
77.8-03
77.9
Avg.
59.5
Avg.
Avg.
69. 09.2..033
70.2:004
72.51.0.1
74.2+0.1
86.3 36.3
87.11044
90.6-+0.3
91.010.1
51.9
$1.5
60.0+0.6
59.51006
R
76.0200.
76.6:10.3
77410..3
78.1+0.2
92.11107
94.3
97.8+04
97.9403
66.3+0.5
68108
72.0+066
71.8204
10%
10%
10%
P
41.620.1
42.5=00.2
47.2.10.2
47.5+0.2
97.010.6
97.020.2
97.3+0.4
97.0=0.1
79.3+0.22
79.8 19.3
86.3 06.3+0..4
86.520.4
w
51.010.2
52.7=00.4
57.61.0.3
60.3+0.55
93.61.011
94.41.0.1
94.8+-0.2
95.540.11
90.6++0.8
91.3-10.66
93.0--0.6
93.7100
50.4 50.410.1
51.4+00.2
56 1.0.1
58.2+-0.2
89.5 0.33
91.21104
93.5+0.11
93,911022
65.740.4
66.9 66.91002
75.94-0.2
76.01002
Avg.
Avg.
Avg.
63.810.2
64.4
66.9.0.22
68.8
83.5.0.5
84.510.3
89.6+0.2
90.01033
40.0++088
42.4.10.8
58.4+0.5
58.3
R
B
68. 58.5
69. 19.1
70.7:104
71.400.3
95.01111
97. 97.7
95.7-105
96.4.04
61.2+4..9
62. 62.6.10.7
71.8+006
71.4044
5%
5%
5%
31.5.003
32.74:0.3
17.81.0.1
38.6-40.2
89.4 89.410088
91.31007
95.540.3
95.5104
73.2+-0.5
73.5.10.8
$1.9+0.5
63.210.66
37.61:04
39.2400.3
$1.610.3
53.800.6
90.010.7
91.01:033
93.240.1
938403
88.3 38.3+0.5
89.01:44
91.4+111
91,1110.3
X
X
x
x
X
17

---

Under review as a conference paper at ICLR 2021
68.6=0.2
68.6=0.1
69.1 69.1=0.1
95. 10
95.1=0.1
95.3=0.11
79.9-0.3
80.3
Avg
Avg
Avg.
80.
76.5+0.2
75.9:0.3
76.620.
90,000.3
90.7
90.6203
58.810.7
59.210.
59.7
82.0+0.2
82.2:10.3
82.5+00.2
98-4+0.6
99. 99.2:10.9
99.2:0.66
78.8
77.740.
779404
20%
20%
20%
P
53.0+0.4
53.3:10.3
53.340.3
96.2440.4
94.51:0.3
95.8:10.4
89.6-10.5
89.6100.3
89,7+0.3
Detailed results on performance of VMTL with different priors on Office Home.
Table 12. Detailed results on performance of VMTL with different priors on Office- Caltech.
VMTL with different priors on ImageCLEF.
63.0=00.5
63.1 63.1+0..4
64.0=00.3
95.7-0.3
95.8
95.620..1
93.3
94.0-20.2
93.1
64.5+0.2
64.41.0.1
65.0::0.0
95.1 10.00
95.2100.1
95.3:10.1
78.0-:0.2
77.81002
779::0.2
Avg
73.5-00.3
72.7-10.3
74.2
90.420.2
90.2-00.2
91.0.0.1
60.2-004
59.3100.6
59.5-10.6
R
71.9
778=:011
78.1#02
97.1-03
97.9=0.0
97.920.3
71.7-04
719-0.8
718=044
10%
10%
47.040.3
47,2103
475540
97,710.2
97.01033
97.0
86.5403
86.1 10.1
8655
Table 13. Detailed results on performance
59.7+04
59.8:0033
60:34005
95.34-0.1
95.5.10.11
95.5100.1
93.54004
93.71-0.3
93,7:104
57.240.1
57.0+00.2
58.240.
93 8102
03.510.11
93,9:0,2
75.5100
76.0:0.2
Avg.
Avg.
Avg.
75.
68.0 08.0
67.71:0.2
68.84002
88.5 08.54-0.4
89.91-0.3
90.0:10.3
57. 97.41.0.8
58. 38.3
Table 11.
70.6:b0.1
70.7.00.2
71.4+0.
97. 1:0.5
96. 06.4
96.420.4
71.410.11
71.910,7
71.420.4
5% P
56k
5%
38, 08.0
38 0.2
38 08.
95.8 05.
94.20022
95.5 05.
03.0
81.8 0.4
83.2 03.2
W
52.0=0.4
51 $1.4100.4
33.8+066
93 8+005
92.9 0..910.4
93. 03.
90.7 00.7.
91 1103
89.
Leanable weighted
Gumbel -Sfftna
Learable weighred
Gumbel-son -softmax
Learnable weighted
Gumbell softmax
Priors
Mean
Priors
Mean
Priors
Meun
18

---

Under review as a conference paper at ICLR 2021
Table 14. Performance comparison of different methods on the Office Office-Home Home dataset for multiple tasks; Artistic (A), Clipart (C), Product (P) and Real Real-worlk -orldd (R).
08 08.21.1111
69. 00.1::0111
16.82-0.33
79 99.44--02
80 80.31:00.2
Table 16. Performance comparison of different methods on the large-scaled dataset DomainNet multiple tasks: Clipart (C), Infograph (O) Painting (P),
27.2::0.1
25.8 25.8
29.6
Avg.
58.3
58.6
59.3
66.5
66.9
Avg.
76.2
77.2
79.7
Avg.
23.5
27.5
76.
70.0
70.5
71.2
72.
72.5 8
75.9
00.6-0..3
Table 15. Performance comparison of different methods on the three dataset with 20% train-split: Office Home, Office- Caltechh ImageCLEP
54.6
55.6
59 9..4
56 56:0=088
59.7=0.5
S
16.3
17.8
1788:0022
16.3
21.8:00.2
76
60
ImageCI.EF (2EFF)20%))
82.010022
82.51:0.2
71.90:0
77.0+-077
77.910.4
20%
69.
69.6
20.2
79.3
80,7
72.5
72.7
75.8
58.7
60.7
59.6.10.2
58. 58.1 1::00
61.8:10.1
52.4100.3
53.3100.3
86.51:022
87.71002
89.71.0.3
38.8
39.2
56.
53.6
84.6
87.3
89.2
4%
5.2
20.2
20.7
18.2
15.7
62.3.0.11
64.0=20.3
92.9=1055
92.9-1.33
94.0-0.2
A
55.2
55.2
58.3
58.5
92.9
93.3
94.4
30.3
29.0=10.1
35.4100.
30.4
30.9
P
63.7-011
65.0=0.0
93.1 93.1=0.1
95.2-00.2
95.3-001
Avg.
54.2
53.7
55.5
63.0
Avg.
92.4
934
95.1
8.7--0.2
8.0=
11.91:00
7.1
8.9
73.1110.3
74.2:10.1
96.8:10.2
90.6:20.5
65.0
64.6
06.3
69.9
73.0
86.7
87.7
91.0
R
26.2+0..
24.9=10.0
30.9=0.1
Office-Cniee Calteell 20%)
C
23.0
26.4
76.91002
78.120.2
96.01:12
99.4-0.3
99.2+0.6
10%
P
66.3
65.9
67.7
73.6
76.3
D
95.2
96.0
99.2
32.6::0.1
30.5=:0.2
33.2-
Avg.
21.8
32.6
46.5 5.33
47.5=0.2
20.7
95.7-0.5
95.8:0.4
35.0
34.6
36.3
42.3
42.7
W
92.8
94.5
94.9
23.1 23.1
214110.2
25.5
14.0
23.2
58.410.5
60.3100.5
95.04:22
95 25.24003
95.6110.1
A
50.3
49.7
51.6
57.2
59.9
A
94.9 95.5
95.5
64.910022
62.7110.1
65.410.2
56.7 26.7.0..2
58.2100.2
63.6100.1
69.110.11
R
56.5
65.3
Ave.
47.9
49.5
50.1
53.4
$7.0
Avg.
64.9
65.8
66.9
24.7+10.2
21.30..2
18.5
66.7100
68.8102
72.9004
75.94004
76.6103
2%
7.2
23.7
59.5
60.6
61.3
62.9
67.7
73.1
73.8
72.8
Office Home (20%)
3688=0.1
35.3-10.2
40.0=:0.2
70. 0.3
71.420.3
80.9=033
82.0=02
82.
Quickdraw (Q), Real (R) and Sketch (S).
26.0
37.1
5%
61.5
62.3
62.9
67.1
70.5
81.3
81.2
80.7
11.91:0.3
10.4::0.3
14.3::0.2
37.5 0.5
08.6=0.2
0.5
52.4-033
53.3=0.3
6.6
12.3
30.4
32.8
33.3
34.5
36.4
50.6
$2.0
55.6
38
31.7.0.2
35.7=10.1
40.1
42.3
42.7
49.2
53.3
52.3:104
53.820.6
54.6
55.9
58.5
56 6::03.33
64.01:.3
C
20.5
33.8
34100.1
62
Methods
MTFL
RMTL
MTRL
DMTL-TF
MRN
VMIL-AC
VMIT
Methods
STI.
VSTL
MRN
bMTL
VMTL-AC
VMTL
Methods
STL
VSTL
bMTL
FMTL-AC
VMTL
19


Reviews:
[{"review_id": "jjNd0oCcDrO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper presents a variational framework for multi-task learning, where for each image classification task, the feature representation z and the classification parameter w of the other tasks serve as the prior for the counterparts of the task. The priors are engaged as the regulariser in the variational inference framework, making the features and the classification parameters of the tasks close to each other. The proposed method outperforms several baselines on several image classification benchmark datasets.\n\nPros:\n\n- The proposed way of using a Bayesian (variational) approach where the feature and classification parameters of other tasks are leveraged as priors is quite intuitive. As priors naturally serve as regularisations in Bayesian inference, it is natural and might be a principled way of sharing information across multiple tasks. The formulation of multi-task learning with the Bayesian framework might not be completely new in the conventional Bayesian settings, but it seems to be new in the auto-encoding/amortized variational inference settings. Although I'm not an expert in the domain of multi-task learning, I didn't find a similar point of view in the literature.\n\n- I also think the proposal of the prior construction in Eq (9) and Eq (11) is interesting and clever, which is conducted by weighting the posteriors of the other tasks.\n\n- It is also a plus that the authors have done comprehensive oblation study of the proposed framework, showing the different contributions of the components.  To me, the components form an elegant model.\n\n- The paper is easy to follow.\n\nCons:\n\n- The use of the Gumbel softmax in Eq (9) and Eq (10) is confusing to me. Motivation, intuition, and the learning of the A_ti are a bit unclear. Which parameter is learned in the end? Is it \\pi_ti? If so, the \\pi_ti is a kind of free parameter to learn. Then why is \\pi_ti put through a Gumbel softmax?\n\n- I also feel that the datasets used in the paper are a bit small-scaled in today's machine learning/deep learning areas.\n\n---------------------------------------------------------------------------------------------------------------------------\n\nThe authors have addressed my previous concerns. The experiments on a larger dataset are a plus. Therefore, I would recommend an acceptance of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PQ4GuZq1eJF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a variational based approach for multi-task learning for the setting where there is a limited amount of training data and for each related task. Prior distributions on model parameters are defined using weighted approximate posterior distributions of other related tasks. This has been done both for classification parameter (w) and latent features (z). The mixing weights are learned in the main optimization using Gumble-softmax prior for each task. The classifier parameters are learned with amortized inference to make posterior inference more effective. The ELBO is learned using MCMC samples using samples partially taken using the reparametrization trick. The experiment section compares the performance of the proposed method against previous multi-task learning approaches. \n\nThe idea of using a variational approach for multi-task learning is novel and interesting. Using posterior distributions of related tasks in a weighted sum for defining prior distributions for parameters of each task is an interesting method that could also inspire more future work in this area.  In terms of the learning algorithm, there are no significant contributions related to novelty, but incorporating various details such as a Gumble-softmax weighting approach, amortized learning, and reparametrization trick have helped this proposed approach to form an overall good performance. \n\nProposing this model for multi-task learning problems in which there are limited input samples for each of the related tasks is a problem that has not been explored enough before in the literature despite its importance for many domain applications. Focusing on this problem is an aspect that makes this paper have a high impact on the field. \n\nThe related work section gives a good overview of the previous and current approaches for multi-task learning from classical penalty driven models, to Bayesian approaches and deep learning models. \n\nThe experimental results show a clear improvement of VMTL over the current baselines. I liked that the authors have analyzed the importance of each of the pieces that are used in the model or in the learning algorithm such as the effectiveness of the Bayesian approximation and the choice of softmax for priors.\n\nThe paper is clearly written and organized. There are a couple of typos (takes instead of task in section 2.1 and B.3),.. \n\nAside from a brief mentioning of cost reduction due to amortized learning, I don't see any computational time analyst. I think it would be very helpful if the authors could provide more information regarding running times compared to the baselines. \n\nAlso, MCMC methods tend to result in high variances especially in the limited data domain. Seeing some analysis on how the variance is affected by the hyperparameter choices and the temperature parameter would be very informative. \n\nI would be interested to know the results for other multi-task learning-related tasks such as representation learning or robustness against adversarial attacks as the scope of this work doesn't have to be limited to the classification accuracies. Other applications such as NLP related ones could be interesting too as many multi-task learning approaches have been successfully applied there, and it would be nice to see how this could compare to those.\n\nOverall, in my opinion, this paper presents a novel approach for a significant problem, and the experimental results support the claims. That is why I accept this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E3dF6RsPa-v", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n### Summary\nThe paper proposes a Bayesian formulation of multi-task learning in the classification scenario.  Specifically, the problem is cast at a variational Bayesian inference problem. The inter-dependency of the tasks is enforced through Gumbbel-softmax priors, with the weights learned from the data. The Bayesian formulation leads to better performance when the amount of training data is low, as shown on three benchmark datasets for image classification.\n\n##########################################################################\n### Reasons for score\n Overall, I vote for reject. The methodology is not formulated in a proper Bayesian way as claimed. \n The experiments, while covering a wide range of datasets and scenarios, do not have any error bars \n and hence have very low statistical significance.\n\n##########################################################################\n### Pros \n1) The use of the Bayesian approach is well motivated for the case of limited data.\n2) The explicit inter-task dependency of the classifier weights is introduced, in addition to the \nmode standard approach of interdependent features.\n3) The related work section is quite thorough.\n4) The experiments cover a wide range of datasets and analyse different aspects of the model: effect \nof variational approximations on latent representations and the classifier weights, particular \nchoice of the prior, different amounts of training data available.\n\n##########################################################################\n### Cons\n1) While other Bayesian multi-task models are mentioned on the related work, their applicability \nfor the task at hand is not addressed and the need for this particular approach is not motivated.\n2) The model is not formulated in a standard Bayesian way. There is no clear separation between the model and the inference scheme. Specifically, the prior for one task is dependent on the variational posterior for the other tasks and is learned, hence it is not a proper prior. \n3) The experiments do not have any error bars, only one value (i.e. one run). Thus the experiments \nonly weakly support the claims and are not statistically significant.\n\n##########################################################################\n### Some typos/ minor comments\nEq. (2): Use a different letter in the product, not t\n\nFig. 1: What do the dashed lines show? Is this an illustration of the model or the inference?\n\nSec. 4.2 Implementation: You say about using VGGnet as a  \"feature extractor in your architecture\". \nThe phrasing suggests you use it to extract z. I assume you pre-extract x. Clarify, please.\n\nSupplement: Eq. (15): The very last term should have expectation over both q(w) and q(z|x)\n\nSupplement B.3: \"the prior of current takes is \" -> \"the prior of current tasks is\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "2L7uorYRkl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nIn this paper, the authors propose to perform multi-task learning by casting as a variational inference problem. The key of the technique lies in how task relationships are modelled, through clever use of priors. In this paper, the priors for each task are conditioned on other tasks to enable knowledge transfer. The authors test their method in the small data-regime and show consistently better performance than the baselines.\n\n[Decision]\nI think this is a strong paper. The novelty is clear, the method well presented and the experimental section is good. Whilst grading an initial \"7\", I would potentially like to see (if possible) more results on bigger datasets, instead of restricting to 20%, to better understand the strength of the method when compared to other MTL baselines.\n\n[Decision after reading rebuttal]\nThe authors appropriately addressed my concerns. The added experiments definitely reinforce the results of the paper and the added text help clarity. I recommend acceptance of this paper and would argue that its quality is reflect by an 8, up from my original score of 7.\n\n[Main points]\n1. The decision to cast multi-task learning as a variational inference problem where task relatedness is modelled through use of priors and posteriors is very interesting and well developed.\n2. This paper is very well written, and whilst technical, is mostly easy to follow.\n3. The ablation studies in addition to the experiments are extensive and sufficient to demonstrate: i) the advantages of the Bayesian modelling and ii) strengths of the method\n\n[Questions]\n1.  Why is a Gumbel-Softmax necessary to learn the mixing weights when updating the priors based on the other tasks? From what I can see, this is not a stochastic process that requires sampling from a Categorical. Why not learn $\\mathcal{A}$ simply through backprop?\n2. Modelling task relatedness is performed through the other task posteriors in Equation 9/10. In cases where they may only be task interference such that tasks should not be shared, how is this dealt with? Would $\\mathcal{A}$ collapse to a uniform prior?\n3. I am confused about the implementation (Section 4.2.). Is the VGGNet used as the backbone of the inference network for $z$ whilst classifiers on top of this are $w$? You state in Section 2.3. that an MLP is used for amortised inference. A more detailed explanation of the implementation is required to gain a better understanding of how everything is set up.\n4. I am a bit confused about how the amortised inference is performed. For a given task $t$, there are $C$ specific labels. In Equation 12, do you mean that that weights for different labels are drawn from different distributions?\n5. Whilst not the focus of the paper, how does the method perform when there is more data accessible? It would be interesting to gauge the competitiveness of the method then. In Tables 3 and 4, if we compare VMTL vs MRN, the difference in performance is not so significant. This might point that Bayesian MTL methods, with clever prior construction are the way to go. Results on larger datasets might help ascertain better performance of the method. \n\n[Other comments]\n\n1. One notable reference is missing from the Related Works section; one that first proposed to use the Gumbel-Softmax for learning task relatedness in MTL in [1].\n2. I would like to see a more in depth discussion between your method and [2, 3, etc.]. As your method works comparably to MRN and can be classified as MTL papers with clever priors, I would like to understand more the differences + pros/cons of techniques.\n3. Is there a way to inspect what was learned to be shared?\n4. It is mentioned in the introduction that a Bayesian framework would allow you to deliver uncertainty estimates over the predictions. Were you able to analyse this?\n5. Would your method be amenable to a continual learning scenario where the posterior over all prior tasks could be used for a new task without suffering catastrophic forgetting?\n\n\n[1] Bragman et al. Stochastic Filter Groups - https://openaccess.thecvf.com/content_ICCV_2019/papers/Bragman_Stochastic_Filter_Groups_for_Multi-Task_CNNs_Learning_Specialist_and_Generalist_ICCV_2019_paper.pdf\n[2] MRN paper - https://arxiv.org/abs/1506.02117\n[3] MTL with Dirichlet Process Priors - https://www.jmlr.org/papers/volume8/xue07a/xue07a.pdf\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a probabilistic model for multitask learning with representation learning. The basic idea is to share information across tasks by making the prior over the model parameters of one task conditioned on a convex combination of the variational posteriors of the other tasks.

While some of the reviewers gave high scores and recommended acceptance, one of the reviewers (AnonReviewer3) had some pertinent concerns which lingered even after author response. In particular, AnonReviewer3 mentions that since the prior of one task is conditioned of the variational posteriors of the other tasks, the method is not a proper Bayesian approach. I also read the paper and agree with the assessment. Indeed, the common Bayesian way for multitask learning is to couple the tasks purely based on a prior that encouraging sharing across tasks instead of having task-specific prior that depend on the variational posterior of other tasks as is being done in this paper. 

I also read the reviews and the author response and have some other concerns as well:

- There is a huge amount of prior work on multitask learning, both non-Bayesian as well as Bayesian. Although the paper cites several of those it is disappointing that none of the baselines are Bayesian. Even the non-Bayesian baselines aren't the state-of-the-art recent methods, which is disappointing given the extensive body of prior work in this area.

- The rebuttal wrongly claims MTRL and MRN to be Bayesian methods (included in Table 14 as baselines) whereas they only have a probabilistic formulation and only do point estimation. At a minimum, the paper should show comparison with some Bayesian multitask learning approaches (e.g., shared hierarchical priors, or task clustering, etc). The baselines such as MTRL and MRN aren't among the strongest ones out there. 

- The paper's title is way too generic. There are several multitask learning papers that use variational inference for a Bayesian model. Moreover, given that the basic formulation itself is a bit problematic to be called Bayesian, the title in some sense is also misleading.

Due to the above issues, I don't think the paper can be accepted in its current form.
Decision: Reject