Uncertainty in Neural Processes
Authors: Saeid Naderiparizi
Kenny Chiu
Benjamin Bloem-Reddy
Frank Wood
Venue:  (2021)
Keywords: Neural Processes
Meta-learning
Variational Inference

Abstract: We explore the effects of architecture and training objective choice on amortized posterior predictive inference in probabilistic conditional generative models.  We aim this work to be a counterpoint to a recent trend in the literature that stresses achieving good samples when the amount of conditioning data is large.  We instead focus our attention on the case where the amount of conditioning data is small.  We highlight specific architecture and objective choices that we find lead to qualitative and quantitative improvement to posterior inference in this low data regime.  Specifically we explore the effects of choices of pooling operator and variational family on posterior quality in neural processes.  Superior posterior predictive samples drawn from our novel neural process architectures are demonstrated via image completion/in-painting experiments.

Under review as a conference paper at ICLR 2021
UNCERTAINTY IN NEURAL PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
We explore the effects of architecture and training objective choice amortized posterior predictive inference in probabilistic conditional generative models. We aim this work to be counterpoint to recent trend in the literature that stresses achieving good samples when the amount of conditioning data is large. We instead focus our attention on the case where the amount of conditioning data is small. We highlight specific architecture and objective choices that we find lead to qualita- tive and quantitative improvement posterior inference in this low data regime. Specifically we explore the effects of choices of pooling operator and variational family on posterior quality in neural processes. Superior posterior predictive sam- ples drawn from our novel neural process architectures are demonstrated via image completion/in-painting experiments.
INTRODUCTION
What makes a probabilistic conditional generative model good? The belief that a generative model good if it produces samples that are indistinguishable from those that it was trained on (Hinton) 2007) is widely accepted, and understandably so. This belief also applies when the generator is conditional, though the standard becomes higher: conditional samples must be indistinguishable from training samples for each value of the condition.
Consider an amortized image in- in-painting task in which the objective is to fill in missing pixel values given a subset of observed pixel values. If the number and location of observed pixels is fixed, then a good conditional generative model should produce sharp-looking sample images, all of which should be compatible with the observed pixel values. If the number and location of observed pixels allowed to vary, the same should remain true for each set of observed pixels. Recent work on this problem has focused reconstructing entire image from as small conditioning set as possible. As shown in Fig. I. state-of-the-ar art methods (Kim et al. 2018) achieve high-quality reconstruction from as few as 30 conditioning pixels in 1024-pixel image.
Our work starts by questioning whether reconstructing an image from a small subset of pixels is always the right objective. To illustrate, consider the image completion task on handwritten digits. A small set of pixels might, depending on their locations, rule out the possibility that the full image is, say, 5, 6. Human-like performance in this case would generate sharp-looking sample images for all digits that are consistent with the observed pixels (i.c.. 0, 2-4, and 7-9). Observing additional pixels will rule out successively more digits until the only remaining uncertainty pertains stylistic details. The bottom-right panel of Fig. [ demonstrates this type of "calibrated" uncertainty.
We argue that in addition to high-quality reconstruction based on large conditioning sets, amortized conditional inference methods should aim for meaningful, calibrated uncertainty, particularly for small conditioning sets. For different problems, this may mean different things (see discussion in Section 3. In this work, we focus on the image in-painting problem, and define well calibrated uncertainty to be combination of two qualities: high sample diversity for small conditioning sets; and sharp-looking, realistic images for any size of conditioning set. As the size of the conditioning set grows, we expect the sample diversity to decrease and the quality of the images to increase. We note that this emphasis is different from the current trend in the literature, which has focused primarily on making sharp and accurate image completions when the size of the conditioning context large (Kim et al.) 2018).
To better understand and make progress toward our aim, we employ posterior predictive inference in conditional generative latent-variabl model, with a particular focus neural processes (NPs)

---

Under review as a conference paper at ICLR 2021
2
2
3
3
7
2
2
9
7
7
2
2
7
9
3
2 2
7
2
5 3 2
9
7
2
2
2
7
2 3
9
2
2
3
D
2
2
7
7
7
2
2
6
3
3
2
2
Figure 1: Representative image in-painting results for CelebA and MNIST. From left to right, neural process (NP) (Garnelo et al. 2018b). attentive neural process (ANP) (Kim et al.) 2018), and ours. Top rows show context sets of given pixels, ranging from very few pixels to all pixels. In each panel the ground truth image (all pixels) is in the upper right corner. The rows correspond to i.dd. samples from the corresponding image completion model given only the pixels shown in the top row of the same column. Our neural process with semi-implicit variational inference and max pooling produces results with the following characteristics: 1) the images generated with a small amount of contextual information are "sharper" and more face- and digit-like than NP results and 2) there is greater sample diversity across the i.i.d. samples than those from the ANP. This kind of "calibrated uncertainty" is what we target throughout.
Garnelo et al. 2018a b). find that particular architecture choices can result in markedly different performance. In order to understand this, we investigate posterior uncertainty in NP models (Sec- iion4. and we use our findings to establish new best practices for NP amortized inference artifacts with well-calibrated uncertainty. In particular, demonstrate improvements arising from a com- bination max pooling, a mixture variational distribution, and a "normal" amortized variational inference objective.
The rest of this paper is organized as follows. EectionAandd Section present background material on amortized inference for generative models and calibrated uncertainty, respectively. Section discusses and presents empirical evidence for how NP models handle uncertainty. Section 5 intro- duces our proposed network architecture and objective. Section @rpport our results on the MNIST, FashionMNIST and CelebA datasets. Finally, Section 7 presents our conclusions.
2
AMORTIZED INFERENCE FOR CONDITIONAL GENERATIVE MODELS
Our work builds on amortized inference (Gershman & Goodman 2014 Kingma & Welling 2014).
probabilistic meta-lcarning (Gordon et al. 2019). and conditional generative models in the form of neural processes (Gamelo et al. 201 8b: Kim et al. 2018). This section provides background.
Let (æc, 3c) = (x, w/ } and (æT, yT) = (a/f, 2/)) be a context set and target set respec- tively. In image in- painting, the context set input ze subset of image's pixel coordinates, the context set output ye are the corresponding pixel values (greyscale intensity or colors), the target set input 2y is set of pixel coordinates requiring in in-painting, and the target set output zr is the corresponding set of target pixel values. The corresponding graphical model shown in Fig. 2
The goal of amortized conditional inference is to rapidly approximate, "test time," the posterior predictive distribution
=)po po (yTT Øc. uc) ( vc)dz.
(I)
2

---

Under review as a conference paper at ICLR 2021
2
y/i
2
C
Figure 2: Generative graphical model for a single neural process task. the task "context" set of input/output pairs (I4, y4) and T is target set in which only the input values are known.
We can think of the latent variable representing problem-specific task-encoding. The like- lihood term (or æT,2) shows that the encoding parameterizes a regression model linking the target imputs the target outputs. In the NP perspective, is function and Eq. (T) can be seen as integrating over the regression function itself, as in Gaussian process regression (Rasmussen) 2003
fundamental aims for amortized inference for conditional generative models: learning the model, parameterized by 0, that produces good samples, and pro- ducing an amortization artifact, parameterized by Ø, that can be used approximately solve Eq. (T) quickly at test time. Variational inference techniques couple the two learning problems. Let and be task sesk-specifii output and input sets, respectively, and assume that training time we know the values of y. We can construct the usual Lingle-raiinin--ee task evidence lower bound (ELBO) as
Variational inference There
py x)) (z-gg(=(m. log po(vls.w)poos)) (2(2(
(2)
Summing over all training examples and optimizing Eq. (2) with respect ø learns an amortized inference artifact that takes a context set and returns a task embedding; optimizing with respect to learns
generative model. Optimizing both simultancously results in an amortized inference artifact bespoke to the overall problem domain.
At test time the learned model and inference artifacts can be combined perform amortized poste- rior predictive inference, approximating Eq. with
po (yTTlaT æe, yc) Ro po poUTTlTT =) DC) dd
(3)
Crucially, given an input (xc, Me). sampling from this distribution is as simple as sampling a task embedding from (s(z|ar,, yC) and then passing the sampled to the generative model pe(yr|æ7, =) to produce samples from the conditional generative model.
Meta-learning The task-specific problem becomes meta-learning problem when learning gression model 0 that performs well on multiple tasks with the same graphical structure, trained on data for which the target outputs {v(} are observed well. In training our in-painting models, following conventions in the literature (Garnelo et al. 2018a b). tasks are simply random-size sub- sets of random pixel locations and values y from training set images. This random subsetting of training images into context and target sets transforms this into meta-learning problem, and the "encoder" go (= |æ., y) must learn to generalize over different context set sizes, with less posterior uncertainty as the context set size grows.
Neural processes Our work builds on neural processes (NPs) (Garnelo et al. 2018a b). NPs are deep neural network conditional generative models. Multiple variants of NPs have been proposed (Garnelo et al. 2018a|b] Kim et al. 2018), and careful empirical comparisons between them appear in the literature Grrver et al. 2019 Le et al. 2018).
NPs employ an alternative training objective to Eq. arising from the fact that the graphical model in Fig. allows Bayesian update the distribution of 2, conditioning the entire context set to produce a posterior Po (2|æc,, yc). the generative model is tractable family that allows analytic updates of this kind, then the NP objective corresponds maximizing
Es log Palyri www. (ola)ar (T
E Ez=qu(=larrer) log pe(brlz.er)a
(u(= (T)
where replacing Po(2|zz Bc) with its variational approximation is typically necessary because most deep neural generative models have computationally inaccessible posterior. This "NP objective"
3

---

Under review as a conference paper at ICLR 2021
-100
Classifier prediction
1000
Higa (afare, (ie)
600
-430
-440
-450
1000
Context set size
Context set size
(a) Variational posterior entropy
(b) Classifier prediction
Figure 3: Posterior contraction of (2| yc) in a NP+max pooling model. (a) The entropy of (= yC)) as a function of context set size, averaged over different tasks (images) and context sets. The gray shaded area both plots indicates context set sizes that did not appear the training data for the amortization artifact. (b) Predictions of classifier trained infer the context set size given only sc, the pooled embedding context set. Equivalent results for the standard NP+mean pooling encoder and for ANP appear in the Supplementary Material.
can be trained end-to-end, optimizing for both & and simultaneously, where the split of training data into context and target sets must vary in terms of context set size. The choice of optimizing Eq. (4p instead of Eq. (2) is largely empirical (Le 2018).
3 CALIBRATED UNCERTAINTY
Quantifying and calibrating uncertainty in generative models remains an open problem, particularly in the context of amortized inference. Previous work on uncertainty calibration has focused on problems with relatively simpler structure. For example, in classification and regression problems with single dataset, prior work framed the problem as predicting a cumulative distribution function that is close to the data-generating distribution, first as model diagnostic (Gneiting et al. (207)) and subsequently as post-hoe adjustment learned predictor (Kuleshov et al. [018)). version of the latter approach was also applied structured prediction problems (Kuleshov & Liang 2015).
Previous approaches are conceptually similar to our working definition of calibrated uncertainty. However, we seek calibrated uncertainty on per-image, per-conditioning set basis, which is fun- damentally different from previous settings. Quantification of aspects of generative model per- formance is area of ongoing research, with uncertainty quantification a particularly challenging problem.
4 UNCERTAINTY IN NEURAL PROCESS MODELS
In this section, we investigate how NP models handle uncertainty. A striking property of NP models is that as the size of the (random) context set increases, there is less sampling variation in target samples generated by passing
(=(2( Me) through the decoder. The samples shown in Fig. are the likelihood mean (hence deterministic function of 2), and so the reduced sampling variation can only be produced by decreased posterior uncertainty. Our experiments confirm this, as shown in Fig. 3a posterior uncertainty (as measured by entropy) decreases for increasing context size, even well-studied property
beyond the maximum training context size. Such posterior contraction of classical Bayesian inference and is consequence of the inductive bias of exchangeable models. However, NP models do not have the same inductive bias explicitly built in. How do trained NP models exhibit posterior contraction without being explicitly designed to do so? How do they learn to do so during training?
A simple hypothesis is that the network somehow transfers the context size through the pooling op- eration and into Po(s). which uses that information to set the posterior uncertainty. That hypothesis is supported by Fig. 3b which shows the results of training classifier to infer the context size given
4

---

Under review as a conference paper at ICLR 2021
-410
greedy-entrop
groedy-KI. random
10
50
100
430
-450
50
100
150
200
250
300
Context set size
(Left) The first {10, 50, 100} pixels greedily chosen to minimize (go((azzza y) (2a yc)). These pixels are highly informative about 2, but only subset of them will appear in the majority of random context sets. (Right) Posterior entropy decreasing as context size increases, for different methods of generating a context set: green is the average over 100 random context sets of each size; blue greedily chooses context pixels to minimize posterior entropy; and orange greedily minimizes DKL. (go( (2) bl, y) Uc)). The black dashed line represents the posterior entropy when conditioned on the full image.
Figure 4:
only se. However, consider that within randomly generated context set, some observations more informative than others. For example, Fig. F shows the first {10, 50, 100} pixels of an MNIST digit 2, greedily chosen minimize (gx y) y))lgl=ll((a))) yc)). If is interpreted to represent, amongst other things, which digit the image contains, then a small subset of pixels determine which digits are possible.
It is these highly informative pixels that drive posterior contraction in a trained NP. In a random con- text set, the number of highly informative pixels random. For example, max-pooled embedding saturates with the M most highly informative context pixels, where M < d, the dimension of em- bedding space. average, random context contain only nM/N of the informative pixels. In truth, Fig. Jjdsplays how the information content of a context depends, on average, on the size of that context. Indeed, greedily choosing context pixels results in much faster contraction (Fig.
size n. taken from an image with N pixels, will
Learning to contract It can be rewritten as
Posterior contraction implicitly encouraged by the NP objective Eq. (4)
E Eg=qu(=lar [log pa (yrll2, 2r] (KL( (t (l9 vc))
(5)
The first term encourages perfect reconstruction of yr. and discourages large variations in ~ (=(=|æT, y/),, which would result in large variations predictive log-likelihood. This effect is stronger for larger target sets since there are more target pixels to predict. In practice, C C T, so the first term also (indirectly) encourages posterior contraction for increasing context sizes. The sec- ond term, DKL(gs posterior to close the target posterior.
(2) (2) æy, BT) (20(2. Uc)), reinforces the contraction by encouraging the context
Although the objective encourages posterior contraction, the network mechanisms for achieving contraction are immediately clear. Ultimately, the details depend on interplay between the pixel embedding function, hgg the pooling operation ④ and Pa- We focus mean and max pooling.
Max pooling As the size of the context set increases, the max-pooled embedding sc + is non-decreasing in in a trained NP model, ||sc|| will increase each time informative pixel added the context set; will continue increasing until the context embedding saturates at the full image embedding. At high level, this property of max max-pooling means that the ac component of (*c) has a relatively simple task: represent function such that the posterior entropy a decreasing function of all dimensions of the embedding space. An empirical demonstration that Ps achieves this can found in the Supplementary Material.
Mean pooling For fixed image, the size of random context increases, its mean-pooled embedding will, on average, become closer to the full image embedding. Moreover, the mean- pooled embeddings of all possible context sets of the image contained in the convex whose
5

---

Under review as a conference paper at ICLR 2021
S/u
hp
P6
Biu
Ap
ge
3/2
gii
Ap
Min
(a) encoder
(b) decoder
Figure 5: Our modified neural process architecture. The encoder produces a permutation invariant embedding that parameterizes stochastic task encoding as follows: features extracted from each element of the context set using neural net he are pooled, then passed to other neural networks ps and Me that control the distribution over task embedding The decoder such task encoding along with embeddings of target inputs to produce the output distribution for each target input.
hull is formed by (a subset of) the individual pixel embeddings. The de component of Pa (sc). then, must approximate a function such that the posterior entropy is a convex function on the convex set formed by individual pixel embeddings, with minimum at near the full image embedding. Learning such a function across the embeddings of many training images seems a much harder learning task than that required by max pooling, which may explain the better performance of max pooling relative to mean pooling in NPs (see Section 6
Generalizing posterior contraction Remarkably, trained NP-based models generalize their pos- terior contraction to context and target sizes not seen during training (see Fig. 3 The discussion of posterior contraction in NPs using mean and max pooling the previous paragraphs highlights a shared property: for both models, the pooled embeddings of all possible context sets that can be obtained from an image are in a convex set that determined by a subset possible context set embeddings. For max-pooling, the convex set is formed by the max-pooled embedding of the M "activation" pixels. For mean-pooling,
convex set is obtained from the convex hull of the indi- vidual pixel embeddings. Furthermore, the full image embedding in both cases is contained in the convex set. We conjecture that a sufficient condition for an NP image completion model to yield posterior contraction that generalizes to context of unseen size is follows: For any image, the pooled embedding of every possible context set (which includes the full image) lies in a convex subset of the embedding space.
NETWORK ARCHITECTURE
The network architectures employ for our experiments build on NPs, inspired by our findings from Section We describe them detail in this section.
Encoder The encoder (2|20, BC) takes input observations from an i.i.d. model (see Fig. 2 plate over C), and therefore its encoding of those observations must be permutation invariant it is to be learned efficiently. Our gø. in related NP work, has a permutation-invarian architecture,
(,(,,, W:), Siis
SC 0
(u) po(sc); (=(are yc) N' oz)
Here Ps and h& are neural networks and + contains diagrams of a generalization of this encoder architecture (see below). The standard NP architecture uses mean pooling: motivated by our findings in Section we also employ max pooling-
cormutation-invariii pooling operator. Fig.
Hierarchical Variational Inference In order to achieve better calibrated uncertainty in small con- text size regimes, a more flexible approximate posterior could be beneficial. (Consider the MNIST experiment shown in Fig. 6 Intuitively, an encoder could learn to map from the context set to one-dimensional discrete 2 value that lends support only to those digits that are compatible with the context pixel values at the given context pixel locations (xc,ye). This suggests that do should be flexible enough to produce a multimodal distribution over 2, which can encouraged by mak- ing do mixture and corresponds to hierarchical variational distribution (Ranganath et al. 2016

---

Under review as a conference paper at ICLR 2021
2
2
4
4
2
7
7
9
3
2
2
2
9
3
3
2
7
7
0
2
2
2
5
7
7
3
7
7
2
2
2
7
2
2
NP objective with mean pooling
(h) NP objective with average pooling
5
7
9
9
9
7
2 3 2
9
7
7
9
9
2
2
2
7
3
9
9
2
2
2
CCI
15
15
C0
431
319
(c) ANP
ANP
0
0
8
7
4
7
3
2
2 2
6
2
6
2
9
3
?
2
2
2
4
2
7
4
3
3
2
(c) NP+SIVI objective with max pooling
(f) NP+SIVI objective with max pooling
Figure 6: Example MNIST and CelebA image completion tasks, for each of three NP methods. The following guide applies to each block. The top row shows context sets of different sizes (context sets are exactly the same for all methods), i.e., one task per column. The ground truth image in the upper right corner. The rows correspond to the mean function produced by go for different sampled values of z. The bottom row shows an empirical estimate of the standard deviation of the mean function from 1000 draws of a direct visualization of the uncertainty encoding.
Yin & Zhou 2018 Sobolev & Vetrov 2019). Specifically, the encoder structure described above, augmented with mixture variable is
(=(= y) = goll, () a. y)db
(6)
This is shown in Fig. [ For parameter- learninn.. the ELBO in Eq. (2) is targeted. However, the hierarchical structure of the encoder makes this objective intractable. Therefore, tractable lower bound to the ELBO used as the objective instead. particular, the objective based on semi- implicit variational inference (SIVI) (See Appendix A.3)
Decoder The deep neural network stochastic decoder in our work standard and not focus. Like other NP work, the data generating conditional likelihood in our decoder is assumed to factorize in a conditionally independent way, Po (277 2 = II. Pe (v(le, where m is the size of the target set and z! and y target set input and output respectively. Fig. [bb shows the decoder architecture, with the neural network go the link function to per pixel likelihood.
6 EXPERIMENTAL EVALUATION
We follow the experimental setup Garello et al. (018b)), where images are interpreted as func- tions that map pixel locations color values, and image in-painting is framed amortized
7

---

Under review as a conference paper at ICLR 2021
Method
MNIST
FashionMNIST
CelebA
NP+mean
0.96 + 0.12
0.93 + 0.15
2.91 + 0.30
mmP+
0.55 = 0.12
0.57 0.11
1.81 = 0.18
NP+max
1,07 = 0.11
=
3.17 = 0.30
SIVI+max
0.99 + 0.25
0.96 = 0.16
2.99 0.39
Table 1: Predictive held-out test log-likelihood
NP mean
NP max
ANP
SIVI max
MNIST
NIST
CelebA
Inception score
10
7.5
2.5
2
100
10a
102
10a
100
10a
10a
20
10
10
102
103
Context set size
Context set size
Context set size
Figure 7: Inception scores of conditional samples.
predictive inference task where the latent image-specific regression function needs to be inferred from a small context set of provided pixel values and locations. For ease of comparison to prior work, the same MNIST (LeCun et al., 1998) and CelebA (Liu et al. 2015) datasets. Addi- tionally, we run an experiment on FashionMNIST dataset (Xiao et al., 2017). Specific architecture details for all networks are provided in Appendix A and open-source code for all experiments will be released at the time of publication.
Qualitative Results Fig. 6 shows qualitative image in-painting results for MNIST and CelebA images. Qualitative results for FashionMNIST are shown in Appendix D It is apparent in all three contexts that ANPs perform poorly when the context set small, despite the superior sharpness of their reconstructions when given large context sets. The sets of digits and faces that ANPs produce are not sharp, realistic, nor diverse. On the other hand, their predecessor, NP (with mean pooling), arguably exhibits more diversity but suffers at all context sizes in terms of realism of the images. Our NP+SIVI with max pooling approach produces results with two important characteristics: 1) the images generated with a small amount of contextual information are sharper and more realistic; and variability across the i.i.d. samples. These qualitative results
2) there is high context-set-compatie demonstrate that max pooling plus the SIVI objective result in posterior mean functions that are sharper and more appropriately diverse, except in the high context set size regime where diversity does not matter and ANP produces much sharper images. Space limitations prohibit showing large collections of samples where the qualitative differences are even more readily apparent. Appendix L contains more comprehensive examples with greater numbers of samples.
Quantitative Results Quantitatively assessing posterior predictive calibration is an open prob- lem (Salimans et al. 2016 Heusel al.) 2017). Table reports, for the different architectures we consider, predictive held out test-data log leeliikllihoods averaged over 10,000 MNIST, 10,000 Fash- ionMNIST and 19,962 CelebA test images respectively. While the reported results make it clear that max pooling improves held-out test likelihood, likelihood alone does not provide direct measure of sample quality nor diversity. It simply measures how much mass is put on each ground-truth completion. It is also important to note that in our implementation of ANP, in contrast to its orig- inal paper, the observation variance is fixed and that is why ANP performs poorly in Table [ ANP model with learned observation variance outperforms all the other models in terms of held-out test likelihood. However, it is empirically shown that learning the observation variance NP mod- els with deterministic path (including ANPs) hurts the diversity of generated samples (Le et al. 2018) (see Appendix C for a detailed discussion and additional results for ANP model with learned variance).
Borrowing from the generative adversarial networks community, who have faced the similar prob- lems of how quantitatively evaluate models via examination of the samples they generate,
8

---

Under review as a conference paper at ICLR 2021
compute inception scores (Salimans al. 2016) using conditionally generated samples for different context set sizes for all of the considered NP architectures and report them in Fig. 7 Inception score is the mutual information between the generated images and their class labels predicted by clas- sifier, in particular, inception network (Szegedy et al. 2016). However, since inception network is an ImageNet (Deng et al. 2009) classifier, it is known to lead misleading inception scores when applied to other image domains (Barratt & Sharma 2018). We therefore use trained MNIST, Fash- ionMNIST, and CelebA classifiers place of inception network (He et at. 2016). (See Appendix H for details.) The images used to create the results in Fig. are the same as in Figs. 6 and [T For each context set size, the reported inception scores are aggregated over 10 different randomly chosen context sets. The dark gray dashed lines are the inception scores of training samples and represent maximum one might hope to achieve at context set size of zero (these plots start at one).
For small context sets, an optimally calibrated model should have high uncertainty and therefore generate samples with high diversity, resulting high inception scores as observed. As the context set grows, sample diversity should be reduced, resulting lower scores. Here again, architectures using max pooling produce large gains in inception score low-context size settings. Whether the addition of SIVI is helpful is less clear here (see Appendix for discussion the addition of SIVT). Nonetheless, the inception score is again only correlated with the qualitative gains we observe in Fig.
CONCLUSION
The contributions we report in this paper include suggested neural process architectures (max pool- ing, no deterministic path) and objectives (regular amortized inference versus the heuristic NP objec- tive, SIVI versus non-mixture variational family) that produce qualitatively better calibrated poste- riors, particularly in low context cardinality settings. We provide empirical evidence of how natural posterior contraction may be facilitated by the neural process architecture. Finally, we establish quantitative evidence that shows improvements in neural process posterior predictive performance and highlight the need for better metrics for quantitatively evaluating posterior calibration.
We remind the reader that this work, like most other deep learning work, highlights the impact of varying only a small subset of the dimensions of architecture and objective degrees of freedom. We found that, for instance, simply making Ps deeper than that reported in the literature improved baseline results substantially. The choice learning rate also had a large impact on the relative gap between the reported alternatives. We report what we believe to be the most robust configu- ration across all the configurations that we explored: max pooling and SIVI consistently improve performance.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shane Barratt and Rishi Sharma. A note on the inception score, 2018.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR, 2016.
Deng, W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1704-1715 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018b.
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Pro- ceedings of the annual meeting of the Cognitive Science Society, volume 36, 2014.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 69(2): 243-268, 2007.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019.
Aditya Grover, Dustin Tran, Rui Shu, Ben Poole, and Kevin Murphy. Probing uncertainty estimates of neural processes. 2019,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Geoffrey E Hinton. To recognize shapes, first learn generate images. Progress in brain research, 165:535-547, 2007.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference Learning Representations, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Volodymyr Kuleshov and Percy S Liing. Calibrated structured prediction. In C. Cortes, N. Lawrence, Lee, M. Sugiyama, and R. Garnett (eds.). Advances Neural Information Pro- cessing Systems, volume 28, pp. 3474-3482, 2015.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), International Conference on Machine Learning (ICML), volume 80, pp. 2796-2804, 2018.
Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation neural process objectives. NewrIPS workshop on Bayesian Deep Learning, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) 86(11):2278-2) 1998.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
10

---

Under review as a conference paper at ICLR 2021
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333, 2016.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pp. 63-71. Springer, 2003.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234 2242, 2016.
Artem Sobolev and Dmitry P Vetrov. Importance weighted hicrarchical variational inference. In Advances in Neural Information Processing Systems, pp. 603-615, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 1708.07777 2017.
Mingzhang Yin and Mingyuan Zhou arXiv:1805./1183, 2018.
Semi-implicit variational inference.
arXiv preprint


Reviews:
[{"review_id": "vbuB6iBc4Z6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improvement of the standard NP by using a mixture distribution \\q_{\\phi}, semi-implicit variational inference, and max pooling to capture the multimodel structure of the posterior distribution. Replacing one normal Gaussian distribution with a mixture (of Gaussians, normally) is a widely-adopted idea in latent variable models including NP; the adopted semi-implicit variational inference was originally developed in Yin and Zhou ICML 2018, and no further improvement on this inference method is proposed in this manuscript; max pooling is one of three commonly used pooling methods, i.e., max, min, and mean pooling. using one of them to replace another is simple but the explanation of the reason why max pooling is better is interesting and profound. So, the improvement is weak although it is shown to be effective by the empirical study. More importantly, the authors have investigated the posterior contraction of NP. It is interesting. The relationship between the two parts of the objective function of NP has been discussed related to the posterior contraction, both parts have contributed to the contraction apart from their classical explanation on reconstruction and regularization. To my best knowledge, it is the first work to discuss the posterior contraction of NP. It is a classical property in Bayesian and this link will enable further theoretical analysis for NP. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nZUeu9cKub", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an extension to Neural Processes (NPs), where they use max-pooling instead of mean-pooling as the aggregation function and use a mixture distribution with hierarchical VI for the decoder. They show that this slightly improves the predictive likelihoods, but crucially strongly improves the diversity of posterior samples when the conditioning set is small, which they argue is an important feature of such models.\n\nMajor comments:\n- What is the actual impact of using the SIVI? In most experiments, it looks like NP+max performs just as well.\n- What would happen if one would use mean-pooling AND max-pooling and just concatenate the two to yield an aggregated representation? Wouldn't that combine the best of both worlds and the downstream decoder could learn which representation to use for the mean and the variance prediction?\n- Could these ideas (SIVI, max-pooling) also be combined with more modern NP architectures (like Attentive NPs or Convolutional NPs)?\n\nMinor comments:\n- It is argued that the max-pooling is naturally better at capturing useful information for estimating the posterior variances. But what about the posterior mean? Shouldn't the mean-pooling be better for that?\n- In Tab. 1, \"NP+max\" seems to be the best-performing model. Why is it not shown in Tab. 6?\n\nSummary:\nI think the focus on the diversity of posterior samples is very interesting and highlights some important property of these kinds of models. However, given the relative simplicity of the proposed extensions, I feel like they are not studied extensively enough. For the paper to provide a clear value for the community, I think it would be good to extend the experiments to cover the whole combination space of {NP, ANP, SIVI} x {mean-pooling, max-pooling, mean+max-pooling}, so that it becomes clearer what the influence of the different design choices are in combination with each other.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ftdzTUyETGf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims at increasing the sample diversity of neural processes when the condition set is small, while maintaining visual fidelity. The low-data regime is arguably where neural processes are most interesting, and in that regard the paper is right to turn to this setting. The discussion on how different aggregation functions affect the predictive uncertainty of the neural process is also appreciated, as is the experiment on regressing the size of the condition set based on the latent embedding.\n\nUnfortunately, the experiments section does not paint a clear enough picture. While the experiments show us that the proposed modifications have some benefits, it is not so clear how much each part contributes. Especially the contribution of the SIVI bound is hard to judge. As it stands the paper feels a bit incomplete in this regard. For that reason I cannot recommend accepting the paper at this stage, though I am willing to revise my score based on the authors' response. Specifically, I'd appreciate if the paper could make a clear case for adopting the hierarchical latent variable structure and the SIVI bound, as these add complexity to the method (while the max-aggregator does not).\n\n### Pros\n\n* The paper deals with a relevant issue. Neural processes are most interesting when the condition set is small, and this scenario has so far been largely ignored.\n* The discussion on the choice of aggregator is useful, as are the experiments on the variational posterior entropy and the prediction of the context set size.\n\n### Cons\n\n* It is unclear how much each modification (SIVI and max-pooling) contribute. The experimental results compare SIVI+max pooling with NP+max pooling, but SIVI+mean is omitted. It's also notable that NP+max seems to work better than SIVI+max in the CelebA dataset. Some discussion would be helpful here, as I don't see any reason why a hierarchical latent structure should hurt in any case, barring optimization difficulties.\n* I am not familiar with SIVI and I don't expect the average reader to be either. I'd appreciate some discussion on the choice of using it.\n\n### Other comments\n\n* The inception score sounds like the mutual information between the class label and the generated image. I expect that stating this would help some readers.\n* Perhaps it would help to look at each component in isolation and in different settings, e.g. outside the image domain. I can see how max-pooling might be good for images, while other aggregation methods might have an edge in, e.g. a dataset of robot joint trajectories. Other people have investigated the choice of aggregation method, and reading this work reminded me of work by [Soelch et al.](https://arxiv.org/abs/1903.07348), which might be interesting to the authors.", "extra_scores": {}, "extra_text": {}}, {"review_id": "57Le8gn2iQj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper is an empirical investigation into the role of architecture and objective choices in Neural Process (NP) models when the amount of conditioning data is limited. Specifically, they investigate the question of well-calibrated uncertainty. \n\nClarity: The overall quality of the writing is clear, but missing details/explanations leave some claims unjustified and therefore lines of argument difficult to follow.\n\nOriginality: Limited -- the paper is mostly an empirical investigation, with the modifications to existing NPs being (a) max-pooling and (b) SIVI on the posterior z’s.\n\nSignificance: Seems limited (see more on “Cons” section and “Questions/comments”), though I am wondering whether the authors could have done more to emphasize the main contributions of this work. The paper’s significance for me has been hampered by lack of details and clear takeaway/implications of the results.\n\nPros: There’s been a lot of work on NPs and their variants, but less so on empirically investigating how and when they work well. There is also a range of systematic empirical evaluations both in the main text and supplement, which I appreciated.\n\nCons: There are some assumptions/statements that would be beneficial to elaborate upon in the paper. First, calibrated uncertainty is defined to be “high sample diversity for small conditioning sets; and sharp-looking, realistic images for any size of conditioning set.” This statement, introduced early on in the paper without much justification, is a point that the authors repeatedly return to, and I found myself wondering why (especially since NPs are built for prediction/regression, so a lot of the prior work on calibration for classification models should hold here such as (Murphy 1973), (Gneiting et. al 2007), (Kuleshov et al., 2018)). What is the benefit of reasoning about calibration in the generative modeling case (e.g. with Inception Scores)?\n\nAdditionally, why should a more flexible approximate posterior be more beneficial for better calibrated uncertainty? (Is this a log-likelihood argument, since the log-likelihood decomposes to calibration + “sharpness”?) More broadly, my biggest problem was that the paper makes claims about improving calibration (e.g. samples are obtained from “well calibrated posteriors” in Figure 6) without formally defining how to evaluate “good calibration,” what it means to have “calibrated uncertainty,” etc. I would appreciate if the authors could clear up any misunderstandings I may have had about the work.\n\nQuestions/comments:\n- (Heusel et. al 2017) show that the inception score (IS) is not a valid metric for CelebA -- would the authors report FID scores instead?\n- Regarding the comment in Section 4 about posterior contraction: aren’t NPs exchangeable by design via the iid latent variable z (Korshunova et. al 2019, among many others)? I thought that invoking (conditional) de Finetti via the iid latent variable is what allows NPs to model (exchangeable) stochastic processes.\n- It would be helpful to formally define “posterior contraction” for the reader -- is it referring to a reduction in posterior uncertainty?\n- Intuitively, why would sample diversity decrease as the size of the conditioning set grows? For example, if I have a dataset of 10 examples of only cats and dogs and increase its size to 1000 (which say, also includes examples of sheep), shouldn’t that also increase my sample diversity as well?\n- Shouldn’t the arrow going from z -> y_i in Figure 2 be reversed?\n\n-------------\nUPDATE: I have read the authors' rebuttal and revised draft and have raised my score to a 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes some design choices for neural processes, paying particular attention to their small-data performance, uncertainty, and posterior contraction.  This is certainly a worthwhile project, and R3 found the analysis interesting, giving the paper a score of 8.  However, R1, R2, and R4 found the experimental validation to be incomplete and insufficient to support the paper's broader recommendations.  As the paper is investigating the various combinations of implementations, I tend to agree with R1, R2, and R4 that this paper---while having some interesting ideas---needs a bit more precision and breadth to its experiments.
Decision: Reject