Title: UNCERTAINTY IN MULTITASK TRANSFER LEARNING

Abstract: Using variational Bayes with neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. This results in a rich prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them. Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.
Recently, significant progress has been made to scale Bayesian neural networks to large tasks and to provide better approximations of the posterior distribution (Blundell et al., 2015) . Recent works extend fully factorized posterior distributions to more general families (Louizos and Welling, 2017; Krueger et al., 2017; Sun et al., 2017) . It is also possible to sample from the posterior distribution trough mini-batch updates (Mandt et al., 2017; Zhang et al., 2017) .
However, for neural networks, the prior is often chosen for convenience. This may become a problem when the number of observations is insufficient to overcome the choice of the prior. In this regime, the prior must express our current knowledge on the task and, most importantly, our lack of knowledge on it. In addition to that, a good approximation of the posterior under the small sample size regime is required, including the ability to model multiple modes. This is indeed the case for Bayesian optimization (Snoek et al., 2012) , Bayesian active learning (Gal et al., 2017) , continual learning (Kirkpatrick et al., 2017) , safe reinforcement learning (Berkenkamp et al., 2017) , exploration-exploitation trade-off in reinforcement learning (Houthooft et al., 2016) . Gaussian processes (Rasmussen, 2004) have historically been used for these applications, but an RBF kernel constitute a prior that is unsuited for many tasks. More recent tools such as deep Gaussian processes (Damianou and Lawrence, 2013) show great potential and yet their scalability whilst learning from multiple tasks needs to be improved.
Our contributions are as follow:
1. We provide a simple and scalable procedure to learn an expressive prior and posterior over models from multiple tasks.
2. We reach state of the art performances on mini-imagenet.
3. We propose two new benchmarks, each exposing a failure mode of popular meta learning algorithms. In contrast, our method perform well on these benchmarks.
• MAML (Finn et al., 2017) does not perform well on a collection of sinus tasks when the frequency varies. • Prototypical Network (Snell et al., 2017)'s performance decrease considerably when the diversity of tasks increases.
Outline: We first describe the proposed approach in Section 2. In Section 3, we extend to three level of hierarchies and obtain a model more suited for classification. Section 4 review related methods and outline the key differences. Finally, In Section 5, we conduct experiments on three different benchmarks to gain insight in the behavior of our algorithm.

2. LEARNING A DEEP PRIOR:

By leveraging the variational Bayesian approach, we show how we can learn a prior over models with neural networks. We start our analysis with the goal of learning a prior p(w|α) over the weights w of neural networks across multiple tasks. We then provide a reduction of the Evidence Lower BOund (ELBO) showing that it is not necessary to explicitly model a distribution in the very high dimension of the weight space of neural networks. Instead the algorithm learns a subspace suitable for expressing model uncertainty within the distributions of tasks considered in the multi-task environment. This simplification results in a scalable algorithm which we refer to as deep prior.

2.1. HIERARCHICAL BAYES:

To learn a probability distribution p(w|α) over the weights w of a network parameterized by α, we use a hierarchical Bayes approach across N tasks, with hyper-prior p(α). Each task has its own parameters w j , with W = {w j } N j=1 . Using all datasets D = {S j } N j=1 , we have the following posterior:1 p(W, α|D) = p(α|D) j p(w j |α, S j ) ∝ p(D|W)p(W|α)p(α)

= j i

p(y ij |x ij , w j )p(w j |α)p(α),

The term p(y ij |x ij , w j ) corresponds to the likelihood of sample i of task j given a model parameterized by w j e.g. the probability of class y ij from the softmax of a neural network parameterized by w j with input x ij . For the posterior p(α|D), we assume that the large amount of data available across multiple tasks will be enough to overcome a generic prior p(α), such as an isotropic Normal distribution. Hence, we consider a point estimate of the posterior p(α|D) using maximum a posteriori2 .

We can now focus on the remaining term: p(w j |α). Since w j is potentially high dimensional with intricate correlations among the different dimensions, we cannot use a simple Gaussian distribution. Following inspiration from generative models such as GANs (Goodfellow et al., 2014) and VAE (Kingma and Welling, 2013) , we use an auxiliary variable z ∼ N (0, I dz ) and a deterministic function projecting the noise z to the space of w i.e. w = h α (z). Marginalizing z, we have: p(w|α) = z p(z)p(w|z, α)dz = z p(z)δ hα(z)-w dz, where δ is the Dirac delta function. Unfortunately, directly marginalizing z is intractable for general h α . To overcome this issue, we add z to the joint inference and marginalize it at inference time. Considering the point estimation of α, the full posterior is factorized as follows:

N j=1 p(w j , z j |α, S j ) (1) = N j=1 p(w j |z j , α, S j )p(z j |α, S j ) ∝ N j=1 p(w j |z j , α)p(z j ) nj i=1 p(y ij |x ij , w j ),

where p(y ij |x ij , w j ) is the conventional likelihood function of a neural network with weight matrices generated from the function h α i.e.: w j = h α (z j ). Similar architecture has been used in Krueger et al. (2017) and Louizos and Welling (2017) , but we will soon show that it can be reduced to a simpler architecture in the context of multi-task learning. The other terms are defined as follows:

EQUATION

The task will consist of jointly learning a function h α common to all tasks and a posterior distribution p(z j |α, S j ) for each task. p(y|x, h α (z j )).

2.2. REDUCTION OF THE EVIDENCE LOWER BOUND.:

In the previous section, we described the different components for expressing the posterior distribution of Equation 4. While all these components are tractable, the normalization factor is still intractable.

To address this issue, we follow the Variational Bayes approach (Blundell et al., 2015) .

Conditioning on α, we saw in Equation 1 that the posterior factorizes independently for all tasks. This reduces the joint Evidence Lower BOund (ELBO) to a sum of individual ELBO for each task.

Given a family of distributions q θj (z j |S j , α), parameterized by {θ j } N j=1 and α, the Evidence Lower Bound for task j is:

ln p(Sj) ≥ E q(z j ,w j |S j ,α) n j i=1 ln p(yij|xij, wj) -KLj = E q θ j (z j |S j ,α) n j i=1 ln p(yij|xij, hα(zj)) -KLj (5) = ELBOj,

where,

EQUATION

= KL q θj (z j |S j , α) p(z j |α)

Notice that after simplification3 , KL j is no longer over the space of w j but only over the space z j . Namely, the posterior distribution is factored into two components, one that is task specific and one that is task agnostic and can be shared with the prior. This amounts to finding a low dimensional manifold in the parameter space where the different tasks can be distinguished. Then, the posterior p(z j |S j , α) only has to model which of the possible tasks are likely, given observations S j instead of modeling the high dimensional p(w j |S j , α).

But, most importantly, any explicit reference to w has now vanished from both Equation 5 and Equation 6. This simplification has an important positive impact on the scalability of the proposed approach. Since we no longer need to explicitly calculate the KL on the space of w, we can simplify the likelihood function to p(y ij |x ij , z j , α), which can be a deep network parameterized by α, taking both x ij and z j as inputs. This contrasts with the previous formulation, where h α (z j ) produces all the weights of a network, yielding an extremely high dimensional representation and slow training.

2.3. POSTERIOR DISTRIBUTION:

For modeling q θj (z j |S j , α), we can use N (µ j , σ j ), where µ j and σ j can be learned individually for each task. This, however limits the posterior family to express a single mode. For more flexibility, we also explore the usage of more expressive posterior, such as Inverse Autoregressive Flow (IAF) (Kingma et al., 2016) or Neural Autoregressive Flow (Huang et al., 2018) . This gives a flexible tool for learning a rich variety of multivariate distributions. In principle, we can use a different IAF for each task, but for memory and computational reasons, we use a single IAF for all tasks and we condition4 on an additional task specific context c j .

Note that with IAF, we cannot evaluate q θj (z j |S j , α) for any values of z efficiently, only for these which we just sampled, but this is sufficient for estimating the KL term with a Monte-Carlo approxi-mation i.e.:

KL j ≈ 1 n mc nmc i=1 ln q θj (z (i) j |S j , α) -ln N z (i) j 0, 1 ,

where z (i) j ∼ q θj (z j |S j , α). It is common to approximate KL j with a single sample and let the mini-batch average the noise incurred on the gradient. We experimented with n mc = 10, but this did not significantly improve the rate of convergence.

2.4. TRAINING PROCEDURE:

In order to compute the loss proposed in Equation 5, we would need to evaluate every sample of every task. To accelerate the training, we use a Monte-Carlo approximation as is commonly done through the mini-batch procedure. First we replace summations with expectations:

ELBO = N j=1 E zj ∼qj nj i=1 ln p(y ij |x ij , z j ) -KL j = E j∼U N N n j E zj ∼qj E i∼Un j ln p(y ij |x ij , z j )-KL j (7)

Now it suffices to approximate the gradient with n mb samples across all tasks. Thus, we simply concatenate all datasets into a meta-dataset and added j as an extra field. Then, we sample uniformly5 n mb times with replacement from the meta-dataset. Notice the term n j appearing in front of the likelihood in Equation 7, this indicates that individually for each task it finds the appropriate trade-off between the prior and the observations. Refer to Algorithm 1 for more details on the procedure.

1: for i in 1 .. n mb :

2: sample x, y and j uniformly from the meta dataset 3:

z j , ln q(z j ) = IAF α (µ j , σ j , c j )

4:

KL j ≈ ln q(z j ) -ln N (z j |0, I dz )

5::

L i = n j ln p(y|x, z j , α) + KL j Algorithm 1: Calculating the loss for a mini-batch

3. EXTENDING TO LEVEL OF HIERARCHIES:

Deep prior gives rise to a very flexible way to transfer knowledge from multiple tasks. However, there is still an important assumption at the heart of deep prior (and other VAE-based approach such as Edwards and Storkey ( 2016)): the task information must be encoded in a low dimensional variable z. In Section 5, we show that it is appropriate for regression, but for image classification, it is not the most natural assumption. Hence, we propose to extend to a third level of hierarchy by introducing a latent classifier on the obtained representation. This provides a simple way to enhance existing algorithm such as Prototypical Networks (Proto Net) (Snell et al., 2017) .

In Equation 5, for a given6 task j, we decomposed the likelihood p(S|z) into n i=1 p(y i |x i , z) by assuming that the neural network is directly predicting p(y i |x i , z). Here, we introduce a latent variable v to make the prediction p(y i |x i , v). This can be, for example, a Gaussian linear regression on the representation φ α (x, z) produced by the neural network. The general form now factorizes as follow:

p(S|z) = E v∼p(v|z) i p(y i |v, x i )p(x i )

, which is commonly called the marginal likelihood.

To compute ELBO j in 5 and update the parameters α, the only requirement is to be able to compute the marginal likelihood p(S|z). There are closed form solutions for, e.g., linear regression with Gaussian prior, but our aim is to compare with algorithms such as Prototypical Networks on a classification benchmark. Alternatively, we can factor the marginal likelihood as follow p(S|z) = n i=1 p(y i |x i , S 0..i-1 , z). If a well calibrated task uncertainty is not required, one can also use a leave-one-out procedure n i=1 p(y i |x i , S \ {x i , y i }, z). Both of these factorizations correspond to training n times the latent classifier on a subset of the training set and evaluating on a sample left out. We refer the reader to Rasmussen (2004, Chapter 5) for a discussion on the difference between leave-one-out cross-validation and marginal likelihood.

For a practical algorithm, we propose a closed form solution for leave-one-out in prototypical networks. In its standard form, the prototypical network produces a prototype c k by averaging all representations

γ i = φ α (x i , z) of class k i.e. c k = 1 |K| i∈K γ i , where K = {i : y i = k}. Then, predictions are made using p(y = k|x, α, z) ∝ exp (-c k -γ i 2 ). Theorem 1. Let c -i

k ∀k be the prototypes computed without example x i , y i in the training set. Then,

EQUATION

We defer the proof to supplementary materials. Hence, we only need to compute prototypes once and rescale the Euclidean distance when comparing with a sample that was used for computing the current prototype. This gives an efficient algorithm with the same complexity as the original one and a good proxy for the marginal likelihood.

4. RELATED WORK:

Hierarchical Bayes algorithms for multitask learning has a long history (Daumé III, 2009; Wan et al., 2012; Bakker and Heskes, 2003) . However most of the literature focuses on simple statistical models and does not consider transferring on new tasks.

More recently, Edwards and Storkey (2016) and Bouchacourt et al. (2017) explore hierarchical Bayesian inference with neural networks and evaluate on new tasks. Both papers use a two-level Hierarchical VAE for modeling the observations. While similar, our approach differs in a few different ways. We use a discriminative approach and focus on model uncertainty. We show that we can obtain a posterior on z without having to explicitly encode S j . We also explore the usage of more complex posterior family such as IAF. these differences make our algorithm simpler to implement, and easier to scale to larger datasets. Other works consider neural networks with latent variables (Tang and Salakhutdinov, 2013; Depeweg et al., 2017; Turner et al., 2018) but does not explore the ability to learn across multiple tasks.

Some recent works on meta-learning are also targeting transfer learning from multiple tasks. Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) finds a shared parameter θ such that for a given task, one gradient step on θ using the training set will yield a model with good predictions on the test set. Then, a meta-gradient update is performed from the test error through the one gradient step in the training set, to update θ. This yields a simple and scalable procedure which learns to generalize. Recently Grant et al. (2018) considers a Bayesian version of MAML. Additionally, (Ravi and Larochelle, 2016 ) also consider a meta-learning approach where an encoding network reads the training set and generates the parameters of a model, which is trained to perform well on the test set.

Finally, some recent interest in few-shot learning give rise to various algorithms capable of transferring from multiple tasks. Many of these approaches (Vinyals et al., 2016; Snell et al., 2017) find a representation where a simple algorithm can produce a classifier from a small training set. Bauer et al. (2017) use a neural network pre-trained on a standard multi-class dataset to obtain a good representation and use classes statistics to transfer prior knowledge to new classes.

5. EXPERIMENTAL RESULTS:

Through experiments, we want to answer i) Can deep prior learn a meaningful prior on tasks? ii) Can it compete against state of the art on a strong benchmark? iii) In which situations does deep prior and other approaches fail?

5.1. REGRESSION ON ONE DIMENSIONAL HARMONIC SIGNALS:

To gain a good insight into the behavior of the prior and posterior, we choose a collection of one dimensional regression tasks. We also want to test the ability of the method to learn the task and not just match the observed points. For this, we will use periodic functions and test the ability of the regressor to extrapolate outside of its domain.

Specifically, each dataset consists of (x, y) pairs (noisily) sampled from a sum of two sine waves with different phase and amplitude and a frequency ratio of 2:

f (x) = a 1 sin(ω•x+b 1 )+a 2 sin(2•ω•x+b 2 ),

where y ∼ N (f (x), σ 2 y ). We construct a meta-training set of 5000 tasks, sampling ω ∼ U(5, 7), (b 1 , b 2 ) ∼ U(0, 2π) 2 and (a 1 , a 2 ) ∼ N (0, 1) 2 independently for each task. To evaluate the ability to extrapolate outside of the task's domain, we make sure that each task has a different domain. Specifically, x values are sampled according to N (µ x , 1), where µ x is sample from the meta-domain U(-4, 4). The number of training samples ranges from 4 to 50 for each task and, evaluation is performed on 100 samples from tasks never seen during training.

Model Once z is sampled from IAF, we simply concatenate it with x and use 12 densely connected layers of 128 neurons with residual connections between every other layer. The final layer linearly projects to 2 outputs µ y and s, where s is used to produce a heteroskedastic noise, σ y = sigmoid(s) • 0.1 + 0.001. Finally, we use p(y|x, z) = N (µ y (x, z), σ y (x, z) 2 ) to express the likelihood of the training set. To help gradient flow, we use ReLU activation functions and Layer Normalization7 (Ba et al., 2016) .

Results Figure 1a depicts examples of tasks with 1, 2, 8, and 64 samples. The true underlying function is in blue while 10 samples from the posterior distributions are faded in the background. The thickness of the line represent 2 standard deviations. The first plot has only one single data point and mostly represents samples from the prior, passing near this observed point. Interestingly, all samples are close to some parametrization of Equation 5.1. Next with only 2 points, the posterior is starting to predict curves highly correlated with the true function. However, note that the uncertainty is over optimistic and that the posterior failed to fully represent all possible harmonics fitting these two points. We discuss this issue more in depth in supplementary materials. Next, with 8 points, it managed to mostly capture the task, with reasonable uncertainty. Finally, with 64 points the model is certain of the task.

To add a strong baseline, we experimented with MAML (Finn et al., 2017) . After exploring a variety of values for hyper-parameter and architecture design we couldn't make it work for our two harmonics meta-task. We thus reduced the meta-task to a single harmonic and reduced the base frequency range by a factor of two. With these simplifications, we managed to make it converge, but the results are far behind that of deep prior even in this simplified setup. Figure 1b shows some form of adaptation with 16 samples per task but the result is jittery and the extrapolation capacity is very limited. these results were obtained with a densely connected network of 8 hidden layers of 64 units8 , with residual connections every other layer. The training is performed with two gradient steps and the evaluation with 5 steps. To make sure our implementation is valid, we first replicated their regression result with a fixed frequency as reported in (Finn et al., 2017) .

Finally, to provide a stronger baseline, we remove the KL regularizer of deep prior and reduced the posterior q θj (z j |S j , α) to a deterministic distribution centered on µ j . The mean square error is reported in Figure 2 for an increasing dataset size. This highlights how the uncertainty provided by deep prior yields a systematic improvement. Vinyals et al. (2016) proposed to use a subset of Imagenet to generate a benchmark for few-shot learning. Each task is generated by sampling 5 classes uniformly and 5 training samples per class, the remaining images from the 5 classes are used as query images to compute accuracy. The number of unique classes sums to 100, each having 600 examples of 84 × 84 images. To perform meta-validation and meta-test on unseen tasks (and classes), we isolate 16 and 20 classes respectively from the original right: 4 sample tasks from the Synbols dataset. Each row is a class and each column is a sample from the classes. In the 2 left tasks, the symbol have to be predicted while in the two right tasks, the font has to be predicted. set of 100, leaving 64 classes for the training tasks. This follows the procedure suggested in Ravi and Larochelle (2016) .

5.2. MINI-IMAGENET EXPERIMENT:

The training procedure proposed in Section 2 requires training on a fixed set of tasks. We found that 1000 tasks yields enough diversity and that over 9000 tasks, the embeddings are not being visited often enough over the course of the training. To increase diversity during training, the 5 × 5 training and test sets are re-sampled every time from a fixed train-test split of the given task9 .

We first experimented with the vanilla version of deep prior (2). In this formulation, we use a ResNet (He et al., 2016) network, where we inserted FILM layers (Perez et al., 2017; de Vries et al., 2017) between each residual block to condition on the task. Then, after flattening the output of the final convolution layer and reducing to 64 hidden units, we apply a 64 × 5 matrix generated from a transformation of z. Finally, predictions are made through a softmax layer. We found this architecture to be slow to train as the generated last layer is noisy for a long time and prevent the rest of the Matching Networks (Vinyals et al., 2016) 60.0 % Meta-Learner (Ravi and Larochelle, 2016) 60.6 % MAML (Finn et al., 2017) 63.2% Prototypical Networks (Snell et al., 2017) 68.2 % SNAIL (Mishra et al., 2018) 68.9 % Discriminative k-shot (Bauer et al., 2017) 73.9 % adaResNet (Munkhdalai et al., 2018) 71.9 % Deep Prior (Ours) 62.7 % Deep Prior + Proto Net (Ours) 74.5 % To enhance the model, we combine task conditioning with prototypical networks as proposed in Section 3. This approach alleviates the need to generate the final layer of the network, thus accelerating training and increasing generalization performances. While we no longer have a well calibrated task uncertainty, the KL term still acts as an effective regularizer and prevents overfitting on small datasets 10 . With this improvement, we are now the new state of the art with 74.5% (Table 1 ). In Table 2 , we perform an ablation study to highlight the contributions of the different components of the model. In sum, a deeper network with residual connections yields major improvements. Also, task conditioning does not yield improvement if the leave-one-out procedure is not used. Finally, the KL regularizer is the final touch to obtain state of the art.

5.3. HETEROGENEOUS COLLECTION OF TASKS:

In Section 5.2, we saw that conditioning helps, but only yields a minor improvement. This is due to the fact that Mini-Imagenet is a very homogeneous collection of tasks where a single representation is sufficient to obtain good results. To support this claim, we provide a new benchmark 11 of synthetic symbols which we refer to as Synbols. Images are generated using various font family on different alphabets (Latin, Greek, Cyrillic, Chinese) and background noise (Figure 2 , right). For each task we have to predict either a subset of 4 font families or 4 symbols with only 4 examples. Predicting either fonts or symbols with two separate Prototypical Networks, yields 84.2% and 92.3% accuracy respectively, with an average of 88.3%. However, blending the two collections of tasks in a single benchmark, brings prototypical network down to 76.8%. Now, conditioning on the task with deep prior brings back the accuracy to 83.5%. While there is still room for improvement, this supports the claim that a single representation will only work on homogeneous collection of tasks and that task conditioning helps learning a family of representations suitable for heterogeneous benchmarks.

6. CONCLUSION:

Using a variational Bayes framework, we developed a scalable algorithm for hierarchical Bayesian learning of neural networks, called deep prior. This algorithm is capable of transferring information from tasks that are potentially remarkably different. Results on the Harmonics dataset shows that the learned manifold across tasks exhibits the properties of a meaningful prior. Finally, we found that MAML, while very general, will have a hard time adapting when tasks are too different. Also, we found that algorithms based on a single image representation only works well when all tasks can succeed with a very similar set of features. Together, these findings allowed us to reach the state of the art on Mini-Imagenet.

7. APPENDIX:

7.1 PROOF OF LEAVE-ONE-OUT Theorem 1. Let c -i k ∀k be the prototypes computed without example x i , y i in the training set. Then,

EQUATION

Proof. Let γ i = φ α (x i ), n = |K| and assume y i = k then,

EQUATION

)

When When experimenting with the Harmonics toy dataset in Section 5.1, we observed issues with repeatability, most likely due to local minima. We decided to investigate further on the multimodality of posterior distributions with small sample size and the capacity of IAF to model them. For this purpose we simplified the problem to a single sine function and removed the burden of learning the prior. The likelihood of the observations is defined as follows:

EQUATION

f (x) = sin(5(ω • x + b)); y ∼ N (f (x), σ 2 y ),

where σ y = 0.1 is given and p(ω) = p(b) = N (0, 1). Only the frequency ω and the bias b are unknown12 , yielding a bi-dimensional problem that is easy to visualize and quick to train. We use a dataset of 2 points at x = 1.5 and x = 3 and the corresponding posterior distribution is depicted in Figure 3 -middle, with an orange point at the location of the true underlying function. Some samples from the posterior distribution can be observed in Figure 3 -top.

We observe a high amount of multi-modality on the posterior distribution (Figure 3 -middle). Some of the modes are just the mirror of another mode and correspond to the same functions e.g. b + 2π or -f ; b + π. But most of the time they correspond to different functions and modeling them is crucial for some application. The number of modes varies a lot with the choice of observed dataset, ranging from a few to several dozens. Now, the question is: "How many of those modes can IAF model?". Unfortunately, Figure 3 -bottom reveals poor capability for this particular case.

After carefully adjusting the hyperparameters13 of IAF, exploring different initialization schemes and running multiple restarts, we rarely capture more than two modes (sometimes 4). Moreover, it will not be able to fully separate the two modes. There is systematically a thin path of density connecting each modes as a chain. With longer training, the path becomes thinner but never vanishes and the magnitude stays significant.

:

p(xij) cancelled with itself from the denominator since it does not depend on wj nor α. This would have been different for a generative approach.

This can be done through simply minimizing the cross entropy of a neural network with L2 regularization.

We can justify the cancellation of the Dirac delta functions by instead considering a Gaussian with finite variance, . For all > 0, the cancellation is valid, so letting → 0, we recover the result.

We follow the architecture proposed inKingma et al. (2016).

We also explored a sampling scheme that always make sure to have at least k samples from the same task. The aim was to reduce gradient variance on task specific parameters but, we did not observed any benefits.

We removed j from equations to alleviate the notation.

Layer norm only marginally helped.

We also experimented with various other architectures.

If the train and test split is not fixed for a given task, one could leak the test information through the task embeddings across different resampling of the task.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.10 We had to cross validate the weight of the kl term and obtained our best results using values around 0.1 11 Code and dataset will be provided.

We scale ω and b by a factor of 5 so that the range of interesting values fits well in the interval (-1, 1). This makes it more approachable by IAF.

12 layers with 64 hidden units MADE network for each layer, learned with Adam at a learning rate of 0.0002.

Figures and Tables:

Figure 1: Preview of a few tasks (blue line) with increasing amount of training samples (red dots). Samples from the posterior distribution are shown in semi-transparent colors. The width of each samples is two standard deviations (provided by the predicted heteroskedastic noise).

Figure 3: top: True function in the original space with 2 observed data points. middle: True posterior distribution, where the orange dot corresponds to the location of the true underlying function. bottom: Samples from IAF's learned posterior.

Average classification accuracy on 5-shot Mini-Imagenet benchmark.

Table content: <table><tr><td/><td>5-way, 5-shot</td><td>4-way, 4-shot</td></tr><tr><td/><td>Mini-Imagenet</td><td>Synbols</td></tr><tr><td>Proto Net (ours)</td><td>68.6 ± 0.5%</td><td>69.6 ± 0.8%</td></tr><tr><td>+ ResNet(12)</td><td>72.4 ± 1.0%</td><td>76.8 ± 0.4%</td></tr><tr><td>+ Conditioning</td><td>72.3 ± 0.6%</td><td>80.1 ± 0.9%</td></tr><tr><td>+ Leave-One-Out</td><td>73.9 ± 0.4%</td><td>82.7 ± 0.2%</td></tr><tr><td>+ KL</td><td>74.5 ± 0.5%</td><td>83.5 ± 0.4%</td></tr></table>



Table content: <table><tr><td>: Ablation Study of our model. Accuracy is</td></tr><tr><td>shown with 90% confidence interval over bootstrap</td></tr><tr><td>of the validation set.</td></tr></table>

Peer review (iclr2019):
 The authors state that their goal with this paper is manifold: They want to learn a prior over neural networks for multiple tasks. The posterior should go beyond mean field inference and yield good results. The authors claim in their paper that they learn an *expressive transferable prior over the weights of a network* for multi-task settings, which they denote with the unfortunate term *deep prior*. In sec. 2.1 the authors introduce the idea of a hierarchical probabilistic model of weights for a neural network p(W|a) conditioned on task latent variables p(a). They realize that one might want to generate those weights with a function which conditions on variable "z" and has parameters "a". They continue their argument in Sec 2.2 that since the weight scoring can be canceled out in the ELBO, the score of the model does not depend on weights "w" explicitly anymore. This, of course, is wrong, since the likelihood term in the ELBO still is an expectation over the posterior of q(w|z)q(z). However, the authors also realize this and continue their argumentation as follows: In this case -according to the authors- one may drop the entire idea about learning distributions over weights entirely. The math says: p(y|x ; a) = int_z p(z) int_w p(w|z ; a) p(y|x, w)dw dz. So the authors claim that a model p(y|x, z) which only conditions on *z* is the same as the full Bayesian Model with marginalized weights. They then suggest to just use any neural network with parameters "a" to model this p(y|x, z ;a) directly with z being used as an auxiliary input variable to the network with parameters "a" and claim this is doing the same. This is of course utterly misleading, as the parameter "a" in the original model indicated a model mapping a low dimensional latent variable to weights, but now a maps to a neural network mapping a latent variable and an input vector x to an output vector y. As such, these quantities are different and the argument does not hold. Also a point estimate of said mapping will not be comparable to the marginalized p(y|x). What is more concerning is that the authors claim this procedure is equivalent to learning a distribution over weights and call the whole thing a deep prior, while this paper contains no work on trying to perform the hard task of successfully parametrizing a high-dimensional conditional distribution over weights p(w|z) (apart from a trivial experiment generating all of them at once from a neural network for a single layer in a failed experiment) but claims to succeed in doing so by circumventing it entirely. In their experiments, the authors also do not actually successfully try to really learn a full distribution over the weights of a neural network. This alone suffices to realize that the paper appears to be purposefully positioned in a highly misleading way and makes claims about weight priors that are superficially discussed in various sections but never actually executed on properly in the paper. This is a disservice to the hard work many recent and older papers are doing in actually trying to derive structured hierarchical weight distributions for deep networks, which this paper claims is a problem they find to be *high dimensional and noisy*, which is exactly why it is a valid research avenue to begin with that should not be trivially subsumed by work such as this. When reducing this paper to the actual components it provides, it is a simple object: A deterministic neural network with an auxiliary, task-dependent latent variable which provides extra inputs to model conditional densities. Such ideas have been around for a while and the authors do not do a good job of surveying the landscape of such networks with additional stochastic input variables. One example is "Learning Stochastic Feedforward Neural Networks" by Tang and Salakhutdinov, NIPS 2013, a more recent one is "Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables" by Depeweg et al 2017. An obvious recent example of multi-task/meta/continual learning comparators would be "VARIATIONAL CONTINUAL LEARNING" by Nguyen et al. and other work from the Cambridge group that deals with multi-task and meta-learning and priors for neural networks. Another weakness of the paper is that the main driver of success in the paper*s experiment regarding classification is the prototypical network idea, rather than anything else regarding weight uncertainty which seems entirely disentangled from the core theoretical statements of the paper. All in all, I find this paper unacceptably phrased with promises it simply does not even attempt to keep and a misleading technical section that would distort the machine learning literature without actually contributing to a solution to the technical problems it claims to tackle (in relation to modeling weight uncertainty/priors on NN). Paired with the apparent disinterest of the authors to cite recent and older literature executing strongly related underlying ideas combining neural networks with auxiliary latent variables, I can only recommend that the authors significantly change the writing and the attribution of ideas in this paper for a potential next submission focusing on multi-task learning and clarify and align the core ideas in the theory sections and the experiment sections.