Bridge the Inference Gaps of Neural Processes via Expectation Maximization
Authors: Qi Wang
Marco Federici
Herke van Hoof
Venue: ICLR 2023 poster (2023)

Abstract: The neural process (NP) is a family of computationally efficient models for learning distributions over functions. However, it suffers from under-fitting and shows suboptimal performance in practice. Researchers have primarily focused on incorporating diverse structural inductive biases, e.g. attention or convolution, in modeling. The topic of inference suboptimality and an analysis of the NP from the optimization objective perspective has hardly been studied in earlier work. To fix this issue, we propose a surrogate objective of the target log-likelihood of the meta dataset within the expectation maximization framework. The resulting model, referred to as the Self-normalized Importance weighted Neural Process (SI-NP), can learn a more accurate functional prior and has an improvement guarantee concerning the target log-likelihood. Experimental results show the competitive performance of SI-NP over other NPs objectives and illustrate that structural inductive biases, such as attention modules, can also augment our method to achieve SOTA performance.

Published as a conference paper at ICLR 2023
BRIDGE THE INFERENCE GAPS OF NEURAL PRO- CESSES VIA EXPECTATION MAXIMIZATION
Qi Wang ; Marco Federici, Herke Hoof
AMLab, University of Amsterdam, 1098XH, Amsterdam, the Netherlands hhg1 23gel gmail 1ccom federeriiie i @uva. nl, h. vanhoof
ABSTRACT
The neural process (NP) is family of computationally efficient models for learn- ing distributions over functions. However, it suffers from under-fitingg and shows suboptimal performance in practice. Researchers have primarily focused in- corporating diverse structural inductive biases, e-g. attention or convolution, modeling. The topic of inference suboptimality and an analysis of the NP from the optimization objective perspective has hardly been studied in earlier work. To fix this issue, we propose a surrogate objective of the target ligeliiel of the meta dataset within the expectation maximization framework. The resulting model, referred to as the Self-normalized Importance weighted Neural Process (SI-NP), can learn a more accurate functional prior and has an improvement guar- antee concerning the target log-likelihood. Experimental results show the com- petitive performance of SI-NP over other NPs objectives and illustrate that struc- tural inductive biases, such as attention modules, can also augment our method to achieve SOTA performance.
INTRODUCTION
The combination of deep neural networks and stochastic processes provides a promising framework for modeling data points with cor- relations (Ghahramani, 2015). exploits the high capacity deep neural networks and enables uncertainty quantification for distribu- tions over functions.
1
D
z
D
look at the deep Gaussian process (Dami- anou & Lawrence, 2013). However, the run-time complexity predictive distributions in Gaussian processes cubic w.r.f. the number predicted data points. To circumvent this, Garnelo et al. (2018a;b) developed the family of neural processes (NPs) the al- ternative, which can model more flexible function distributions and capture predictive uncertainty
As example,
;(z[DP;0)
s(D" (;
Figure 1: Deep Latent Vari- able Models for Neural Pro- cesses. Here DC and D re- spectively denote the context points for the functional prior inference and the target points for the function prediction. The global latent variable is to summarize function prop- erties. The model involves functional prior distribu- tion p(2|DC, () +) and func- tional generative distribution p p(D" [2; 0)) Please refer to Section (2) for detailed nota- tion descriptions.
lower computational cost.
In this paper, we study the vanilla NP as deep latent variable model and show the generative process in Fig. (1). In particular, let us recap the inference methods used in vanilla NPs: It learns approximate the functional posterior (0(2) P(=[D"; ()) and a functional prior (2)))CC) = p(2|DC; 1), which are permuta- tion invariant to the order of data points. Then the predictive dis- tribution for data point [=.,y-] can be formulated in the form (e)(=)D)) [p(u- [ =]: d)).
While the NP provides computationally efficient framework for modeling exchangeable stochastic processes, it exhibits underfitting and fails to capture accurate uncertainty (Garnelo et al., 20;8b; Kim et al., 2019) in practice. To improve its generalization capability, researchers have focused much attention on finding appropriate inductive biases, egg. attention (Kim
"Correspondence Author.

---

Published as a conference paper at ICLR 2023
al., 2019) and convolutional modules (Gordon et al., 2019; Kawano et al., 2020), Bayesian mixture structures (Wang& & van Hoof, 2022) or Bayesian hierarchical structures (Naderiparizi et al., 2020), to incorporate modeling.
Research Motivations. Most previous work (Gamelo et al., 2018a;b; Kim et al., 2019; Gordon al., 2019; Wang & van Hoof, 2022) ignores the reason why the vanilla NP suffers the perfor- mance bottleneck and what kind of functional priors the vanilla NPs can represent. In particular, point the remaining crucial issues that have been sufficiently investigated in this domain, respectively: (i) understanding the inference suboptimality of vanilla NPs (ii) quantifying statistical traits of learned functional priors. To this end, we try to diagnose the vanilla NP from its optimiza- tion objective. Our primary interest is to find tractable way to optimize NPs and examine the statistics of learned functional priors from diverse optimization objectives.
Developed Methods. To understand the inference suboptimality of vanilla NPs, we establish con- nections among a collection of optimization objectives, e.g. approximate evidence lower bounds (ELBOs) and Monte Carlo estimates of log-likelihoods, in Section (3). Then we formulate tractable optimization objective within the variational expectation maximization framework and ob- tain the Self-normalized Importance weighted neural process (SI-NP) in Section (4).
Contributions. To summarize, our primary contributions are three-fold: (i) we analyze the inherent inference sub-optimality of NPs from optimization objective optimization perspective: (ii) we demonstrate the equivalence of conditional NPs (Garnelo et al., 2a) and SI-NPs with one Monte Carlo sample estimate, which closely relates the prior collapse in Definition (3.1); (iii) our devel- oped SI-NPs have an improvement guarantee to the likelihood of meta dataset in optimization and show significant advantage over baselines with other objectives.
PRELIMINARIES
General Notations. We study NPs in meta learning setup, T defines a set of tasks with т a sampled task. Let DE {( })){ and DJ-- {( yz) += denote the context points for the functional prior inference and the target points for the function prediction. The latent variable is functional representation of task 7 with observed data points.
We refer to 9 E θ as the parameters of the deep latent variable model for NPs. In detail, & consists of encoder parameters in a functional prior p(a|DC; (f) and decoder parameters in a gen- erative distribution p(D] 19). 6 refer to the parameters of a variational posterior distribution (x(2) (,(2) while 7) refer the parameters of a proposal distribution (2(2) ) in the fol- lowing self-normalized importance sampling. Gaussian distributions with diagonal covariance ma- trices are the default choice for these distributions, e.g. p(=(DC;;)
NN; (DC) (DF)),
(2) = (2; ;; us (.,".,, and (x(DF) = N (D.),,
NPs as Exchangeable Stochastic Processes. In vanilla NPs, the element-wise generative process can be translated into Eq. (I). Here the mean and variance functions are respectively denoted by and Σ.
Prim (1im)mm) = -f p(z) II N(npp(1 E(r,, 2))dz
(I)
Based on the Kolmogorov extension theorem (Klenke, 2013) and Finneti's theorem (Kerns & Székely, 2006), the above equation Printm (11:n+m)) is verified to be well-defined exchangeable stochastic process.
NPs in Meta Learning Tasks. Given collection of tasks T. can decompose the marginal distribution p P(DFIDD; with global latent variable z in Eq. (2). Here the conditional distribution p(z|DS; ;) with r E T is permutation invariant w.z.. the order of data points and encodes the functional prior in the generative process.
P(DF = II
p(D?12;0) ;(
TET
Marginal Likelihood
Generative Likelihood Functional Prior
2

---

Published as a conference paper at ICLR 2023
Throughout the paper, the optimization objective of our interest is the marginal log-likelihood of meta learning dataset in Eq. (3). Furthermore, this applies to all NP variants.
max Σ In [ Dp)
(3)
FET
For the sake of simplicity, we consider one task to derive equations in the following section, which corresponds to maximizing the following objective'.
[(() In / p(D l2 Os);
In the NP family (Garnelo et al., 201 8a:b); the target data points are conditionally independent given the global latent variable p(D² +) = 11;-1 p(y:|<z,, [[;;, d). The marginal distribution can be
interpreted the infinite mixture of distributions when z is defined continuous domain. Now learning distributions over functions is reduced to a probabilistic inference problem.
OPTIMIZATION GAPS AND STATISTICAL TRAITS
3.1 INFERENCE SUBOPTIMALITY IN VANILLA NPs
Previously, variational auto-encoder (VAE) models (Kingma & Welling, 2013; Rezende et al., 2014) mostly set prior distribution fixed, e.g. (O, I), as the default to approximate the posterior. This differs significantly from NPs family settings. On the one hand, the functional prior is learned in NPs. On the other hand, the functional prior participates in the performance evaluation.
Exact ELBO for NPs. Following the essential variational inference operation, connections between the exact ELBO and the log-likelihood in Eq. (5). Given the functional prior p(2|DS; ()) and the generative distribution pDD [2; 1),, , the exact functional posterior can be obtained by the Bayes rule
establish
P== =
The denominator p( DD ) makes exact inference infeasible, and the family of variational poste- riors is introduced to approximate ; Here the variational posterior family is defined in parameterized set 2+ (=)l }).
2
((0) = (nD(D
(x)
DEL (g0(2) || p(2 DJ: 0)
g
(5)
Approximation Cop
Exact ELBO
When the variational posterior family is flexible enough, e.g. p(a[DF;;) approximation gap can be reduced to an arbitrarily small quantity. In this case, maximizing the exact ELBO in Eq. (6) increases the likelihood in Eq. (5) accordingly.
0) E Qa. the posterior
(1, Φ) = Bg)(=) [In 22 0)] DKL (40(=) [[=
(6)
Approximate ELBO for NPs. As previously mentioned, the inference complicated since the functional prior and the posterior in the exact ELBO are unknown. To this end, Garnelo et al. (2018b) proposes surrogate objective as an approximate ELBO for NPs. This is defined as Eq. (7) to maximize.
CNP(0, 6) Ego(2)
In p pD t)
DEL (2)(2) || (o
Generative Likelibood
Consistent Regularizer
The Kullback-Leibler divergence between the approximate posterior (z(2) and the approximate prior 4s (=(= is referred the consistent regularizer in this paper. We claim that the consistent regularizer is the source of the inference suboptimality of vanilla NPs, and this is shown in Appendix (D.2) the proof of Remark (1).
Meta training and testing phases implemented in batch of tasks consistent with Eq. (2M(3).
3

---

Published as a conference paper at ICLR 2023
Remark 1 Eq. (7) is invalid variational inference objective, and optimizing it cannot guarantee to find optimal or locally optimal solutions for the maximization over Σ.eT in p(,; [D; 1))
Other Available Objectives in NPs Family. Now we turn to other tractable optimization objectives in NPs. These include conditional neural processes (CNPs) (Garnelo al., 201 8a) and convolutional neural processes (ConvNPs) (Foong et al., 2020) (or VERSA (Gordon et al., 2018)).
The CNP is also typical model of the NPs family. The objective LCNP(1) (1) can be obtained when the functional prior collapses into a Dirac delta distribution with $ a fixed real-value vector.
LNNp ()) (p)
[Im 2; 0) with p(=|Df; 0) 5(1= 21)
For the ConvNP, we do not focus on the convolutional structural inductive bias and concentrate more on the optimization objective itself. Its objective in Eq. (9) is a biased Monte Carlo estimate of Eq. (3), so we can maximize the tog-likelihoon of marginal distributions straightforwardly.
(P) = In
1
E
exp (Inp(D+)=():0) 1=1
B
:1)) with 200) ~ p(z|D,;
(9)
B
This is termed as Monte Carlo Maximum Likelihood (ML-NP (0), and B is the number of used Monte Carlo samples. Without the involvement of the consistent regularizer, both LCNP(1) and (P() (1) will not encounter the approximation gap in practice.
3.2 EVALUATION CRITERIA & ASYMPTOTIC PERFORMANCE
As in (Le et al., 2018; Foong et al., 2020), we take multi-sample Monte Carlo method to evaluate the performance. This applies to both meta training and meta testing processes. detail, NP models need to run times stochastic forward pass by sampling 2b) p(=[DD; () and then compute the log-likelihoods as In ((D?)
After describing the evaluation criteria, we turn to a phenomenon of our interest. In Gaussian pro- cesses (Ghahramani, 2015), with more observed context points, the epistemic uncertainty can be decreased, and the predictive mean function is closer to the ground truth.
Similarly, this trait is also reflected in NPs family and can be quantitatively described follows. Given a measure of average predictive errors 3, the number of context points and the evaluated dataset, DJd the introduced metries 3(D";; :n) are decreased when increasing n in prediction. In our paper, we refer to this trait as the asymptotic behavior.
Definition 3.1 (Prior Collapse) The functional prior p(=(DG;0) (oz ).E is said to collapse in learning when the trace of the covariance matrix satisfies Tr(Eo(D() = of ~ 0 with ED(D)) diaglo?......
As previously mentioned, we can more precisely keep track of measures (DD n)), such as predic- tive log- logllikelshoods mean square errors of data points, to assess the asymptotic behavior. The role of latent variables z is to propagate the uncertainty about the partial observations in functions. And Definition (3.1) provides quantitative way to examine the extent of prior collapse in ML-NPs and SI-NPs.
4 TRACTABLE OPTIMIZATION VIA EXPECTATION MAXIMIZATION
In this section, we propose alleviating the inference suboptimality of NPs with the help the vari- ational expectation maximization algorithm. The strategy is to formulate surrogate optimization objective and then execute the EM-steps in optimization. The henefit our method to guarantee performance improvement w.r.t. the likelihood of meta dataset in iterations and finally result in at least a local optimum.
Thh Monte Carlo maximum likelihood corresponds to the optimization objective that in (Foong et al., 2020) except that the convolutional inductive bias is removed.
4

---

Published as a conference paper at ICLR 2023
Optimization Gap
((01)
(1)2;11)
[((a)
x) ((0x11i0x) (((x)))
C(dz)
[()")
lteration
Convergence
Figure 2: Illustration of Expectation Maximization for NPs. Green lines indicate the results after the E-steps while the red lines are for the M-steps in Algorithm (1). In the convergence iteration, the performance gap L)0 L(0 1-) is close zero and the algorithm results in at least local optimal solution. Values of these quantities are increased from the left to the right.
4.1 VARIATIONAL EXPECTATION MAXIMIZATION FOR NPS
This part is to avoid the inference suboptimality of vanilla NPs mentioned earlier. We retain the neural architectures used in NPs. The basic idea is illustrated in Fig. (2). In detail, we iteratively construct the lower bound L(Ox) and maximize the surrogate function L(0: 9x). The referred optimization gap is due to the complexity of objectives or the choice of optimizers and measures the difference between converged (local) optimal functional prior and the theoretical optimal functional prior. The general pseudo code is Algorithm (1).
Algorithm 1:
Variational Expectation Maximization for NPs.
Input: Task distribution p(T); Task batch size, Number of particles, Initialized and vy. Output: Meta-trained parameters 8 and 1j.
for & = 1 to do
E-step #1: t- k + and reset the variational posterior (s(2) p(=[D]; 0k) in Eq. (5); if Use the Functional Prior as the Proposal then
Reset (>()) = pC: 0k);
else
E-step #2: update the proposal Vl = arg min, (,,,), in Eq. (29) according to operations in Appendix (E.3.1); end
M-step: optimize surrogate functions dk+1 arg max. CSINN 0: 02) in Eq. (12);
end
4.1.1 SURROGATE FUNCTION FOR EXACT NPs
To make the optimization of meta dataset log-likelihood feasible, we construct surrogate functions as the proxy in each iteration step. These meta learning surrogate functions with special properties are closely connected with the original objective, and we leave this discussion in Appendix (E).
Here 0R denotes the parameter the latent variable model for NPs in the k-th iteration of variational expectation maximization. Following the Algorithm (1), we take the E-step #1 by replacing the approximate posterior in Eq. (6) with the last time updated p(z[D?" 02). And this results in the following equation.
(();0+) = Ep(=Ipp :N)) 1np(DD,2(Dff. - Inp(=/D) [w)]
(10)
where ; is the posterior distribution.
Proposition The proposed meta learning function C(0,0s) in Eq. (10) is surrogate function w.r.t. the log-likelinee ligeliiel of the meta learning dataset.
The above proposition is examined based on the definition in Appendix (E.1),
4.1.2 TRACTABLE OPTIMIZATION WITH SELF-NORMALIZED IMPORTANCE SAMPLING
Since the second term in Eq. (10) is constant in the iteration, we can drop to simplify the surrogate objective the right side of the following equation.
5

---

Published as a conference paper at ICLR 2023
max ((0:0)) + max (D: :0()) [Imp(D" ;
(11)
Proposition 2 Optimizing this surrogate function of batch of tasks via the variational expectation
maxinization leads to an improvement guarantee w.H.f. the log- aagliikllhoodd EreihooddEee-e in pDD (D? ;
Still we cannot optimize (1: 0x) since the expectation has analytical solution and it is in- tractable to sample from ; for Monte Carlo estimates'. Remember that the marginal distribution p( p [ 0x) is task dependent and can not be ignored in computing the posterior pl p(z|DF; 0x)) To circumvent this, we introduce proposal distribution qn (2| and optimize the
objective via self- normalieddd importance sampling (Tokdar & Kass, 2010). The resulting meta learn- ing surrogate function is as follows:
p(=|D"" U2)
(EM 0x) Esg,,
Σ
w In pn =00,;;;
(=((
(12)
(gth)
In p(D 10 Generative Likelihood
p(2 DJ
CSINN((D:TT
Importance Weight
Functional Prior Likelihood
where z 2b) ~ Ona (2|D7), w(t) = =xp (In 0x) )) + + 0k) In Im (=(o))??) w/b
and (b) ΣΦ 20=1 w.
In terms of the first conditional term in Eq. (12). all the data points are conditional independent and this is further expressed as p(DT) (0); +) Σn= In p(p (=)] 9). practice, the selection of proposal distributions is empirically tricky, so we make the update of proposal distribu- tions optional in implementations. In our experimental settings, we simply use the functional prior p( ( ()) as the default proposal distribution, which is competitive enough in performance.
Proposition 3 With one Monte Carlo sample used in Eq. (12), the presumed diagonal Gaussian prior pf will collapse into a Dirac delta distribution in convergence. In this case, SI-NP with the one sample Monte Carlo estimate in Eq. (13) is equivalent with CNP in Eq. (8).
CSI-Ne(:: me. 0k) 2e (p())Ds)a)) In (D: 0)
+ Ept=lpgon) [inp(>|Df
(13)
Generative Likelihood
Prior Collapse Term
The Proposition (3) establishes connections between SI-NPs and CNPs in optimization and we attribute this to the collapse term in Eq. (13). Hence, CNP can be viewed as particular example in SI-NPs. The complete proof is based on limit analysis and can be found in Appendix (F.1).
4.2 SCALABLE TRAINING AND TESTING
As shown in Algorithm (I), the meta training process consists of two steps. We skip E-step #2 to avoid unstable optimization observed in empirical results. By repeating E-step # 1/M-sstp iterations until convergence, the method theoretically find least local optimal w.E.. the likelihodood of meta learning dataset based on the Proposition (2).
Once the learning progress reaches the final convergence, prior to obtain the predictive distribution. With B particles in prediction, the distribution can be expressed as follows.
make use of the learned functional
p2y|æ, 0) = EpteIDe Eptr 2]:
~
1
Σ
plyllz, 2001z,22000 200) 8) with (b) ~ p(z(Df;00 (14)
B
b=1
'Though the exact posterior distribution pDD ()p( DJ: d)dz is available.
p(= can be inferred by Bayes rule, the denominator

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS AND ANALYSIS
In this section, two central questions are answered: (i) can variational EM based models SI-NPs, achieve better local optimum than vanilla NPs? (ii) what is the role of randomness in functional priors? Specifically, we examine the influence of NPs optimizatio objectives on typical downstream tasks and understand the functional prior quantitatively.
Baselines & Evaluations. Since our concentration is on optimization objectives in NPs family, we compare to NP (Garnelo al., 2018b), and CNP (Garnelo et al., 2018a), ML-NP (Foong et al., 2020) in experiments. Note that our developed SI-NP and ML-NP (Foong et al., 2020) are impor- tance weighted models, but the mechanisms in estimating weights are significantly different. As for evaluations, refer the reader to Sec. (3.2) for more information. Due to the page limit, additional experimental results are attached Appendix (H.5), which shows that SI-NPs can achieve SOTA performance by adding attention networks.
5.1
SYNTHETICC REGRESSION
Curve Fitting with SI NPs
Curve Fitting with NP:
Curve Fitting with CNPs
Curve Fitting with ML-NPs
Groand
3.50
32
2.
-325
-10
Ground Trui
Prodit Moun
Groud Berkiter Mear
-26
Greand Tuah Preditt un Moan
-225
-28
Figure 3: Examples of Curve Fitting in RBF Kernel Cases. The plots report predictive mean functions with 33 standard deviations.
We create synthetic regression tasks by sampling functions from Gaussian processes. Three types of kernels, respectively Matern }. RBF, and Periodic, are used to generate diverse function distribu- tions. In each iteration, batch of data points from functions is randomly processed into the context and the target for training models.
In testing. the same strategy is used to generate functions and data points. We report the average log-likelihood results in Table (1). It shows SI-NPs outperform the vanilla NPs in all kernel cases and are least competitive with other baselines. ML-NPs are also superior NPs, but they cannot beat CNPs in RBF and Periodic cases. An illustration of fitted curves is given in Fig. (3): compared to vanilla NPs, we notice that SI-NPs and ML-NPs can better match the ground truth and capture the uncertainty between context data points.
Table I:
Test average log-likelihoods target data points for I-dimensional Gaussian process dataset with various kernels (reported standard deviations in runs). For each run, we randomly sample 1000 functions as tasks to evaluate.
Matern
RBF
Periodic
Lee (Oracle)
0,8211000
1.18
0.833
LNP (Gamelo et al., 201 2018b)
-0,225
-0.183+4o8
+0004
LENP (Garnelo et al., 201 8a)
00295
-0.533
LMILNe (Foong et al., 2020)
0.303
0.439910009
-0.547:0036
LSINP (ours)
0.305
0.493
-0.53240036
5.2
IMAGE COMPLETION
Similar to experiments in (Garnelo et al., 2018b; Kim et al., 2019), we perform image completion experiments in this section. We implement NPs baselines in four commonly used datasets, namely MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFARIO (Krizhevsky et al., 2009) and SVHN (Sermanet al., 2012). During the meta training. the goal is to complete images in which some pixels have been randomly masked out. More precisely, we learn a latent variable model that maps all pixel locations x E [0, 1]2 to Gaussian distribution parameters of pixel values based on the context pixel locations and values. Fig. (4) is an example of completion results from sampled images. For implementation details, please refer to Appendix (G).
7

---

Published as a conference paper at ICLR 2023
Table 2: Test average log-likelihoods with reported standard deviations for image completion in ANIIST/MNIITISVVIIIIIIAAS
(5 runs). We test the performance of different optimization ob- jectives in both context data points and target data points. Except CNPs, samples from the functional prior to evaluate the average log- leggliiklihoods...
32 Monte Carlo
MNIST
FMNIST
SVHN
CIFARIO
context
target
context
target
context
target
context
target
CNP
0.81 +0000
0.73+
t.83
0.73+0009
3.19-a.2
3.07=0az
2.35
2.03
LeNp
1. +000s
4.09
0.95
0,90+0009
3,48 3.57
2.71 +0002
2.53
CML-ND
1. 4.0040004
0.99
0.94
+0097
3.50
3.43
2.60 2.60++000s
2..00
CS (ours)
1.09
0.02
0.98
0.94s
3.57=2.003
3.50=0.00
2.75
2.60
In the evaluation, the number of context pixels is randomly selected for each image in the dataset. We examine the perfor- mance in a testing set of the above image datasets and report the average log-likelihoods in Table (2). We can see that SI- NP achieves the best performance in all image datasets. The performance gaps between SI-NPs and NPs are remarkable. Furthermore, the asympiotic behaviors of all baselines are il- lustrated in Fig. (5). All models exhibit asymptotic behaviors by varying the number of context points, and NPs mainly sult in clearly lower log-likelihoods with 10 context points. The observations with different number of context pixels consistent with the conclusion in Table (2).
Figure 4: SI-NP Completed Im- ages. From top to bottom in rows original images, context points, learned predictive means and vari- ances of sampled images.
3.0
CEPARLO
3.6
1.0
2.5
3.4
Log Likelhoodd
0.0
3.2
2.5
3.0
0.88
2.0
NP
NP
NP
2.6
NP
CNP
0.5
CNP
CNP
CNP
ML-NP
ML-NP
ML-NP
ML-NP
SI-NP
SI-NP
1000
Number of Context Pirels
Number Context Pixels
Number Context Pixnls
Number Context Pixels
Figure 5: Asymptotic Performance in Image Completion. We meta test pixel average log- likelihoods with varying number of context points in image datasets. Context points are randomly selected for each image in testing processes. For MNIST/FMNIST datasets, the numbers of context 100, pixels in testing are {10, 100, 300, 500, 700). For CIFARIO/SVHN datasets, the numbers of context
pixels in testing are {10, 100, 300, 500, 800, 1000).
Another critical finding is reported in Fig. (6), which examines whether the learned functional priors collapse into the deterministic ones. This is based on the average computed trace of covariance matrices learned functional priors. In MNIST dataset, SI-NPs encounter the prior collapse, and ML-NPs also obtain extremely lower trace values of covariance matrices. We attribute the prior collapse in MNIST to its most superficial structures and limited semantic diversity, which results in lower prior uncertainty. For image datasets with more complicated semantics, the computed trace values in SI-NPs retain a reasonable interval. The scale of SI-NPs" trace values coincides with the semantics complexity of image datasets: STAARROOSSINN-FMNITTTMOO observation is that with context points increasing, both models' trace values of functional priors in most datasets decrease to certain level.
Another common
The above reveals the meaning of the learned functional priors in SI-NPs: hibits higher uncertainty with more complicated semantics. Hence, it has the potential of generating functional samples with more diversity. For datasets with less rich semantics, the randomness of the leamed functional priors plays less critical role, and the deterministic functional representation is capable of function generation. Also, with more observations, the randomness of functional priors can be reduced.
The functional prior ex-
8

---

Published as a conference paper at ICLR 2023
26-4
2e-6
(MNIST)
MI-NP (FMNIST)
6.0
ML-NP (CFARROO)
Cous Trace
5.3
355
3.0
200
400
E00
200
403
630
0-
205
400
100
850
1003
205
400
600
800
1000
Coustince Trace
(MNIST)
8.6-
(FMNIST)
2.30
(AFAR10)
(SVHN)
1.52
8.4
2.20
1.50
6.2
2:15
2.10
1.46
5.0
1.46
200
402
500
402
602
202
400
500
850
1000
202
400
6000
600
2000
Number of Context Piwats
Number Context Fixels
Number Context Pixcls
Number Context Poxets
Figure 6: Statisties of Learned Functional Priors in ML-NPs/SI-NPs. In meta testing, we still vary the number of context points in image datasets. The trace of learned functional priors' covari- ance matrices Tr(E,(D;)) ] is computed based on p(z 1) = N22 (o Eo({))
RELATED WORK
NPs Family. NPs are simple and flexible in model formulations, but they may suffer from un- derfitting. Previous research focuses more on the use of structural inductive biases NPs. (Kim al., 2019; 2021) improves the predictive performance by adding attention based local variables. Translation equivariance and invariance are incorporated in modeling NPs with help of convolutions (Gordon al., 2019; Foong al., 2020; Kawano et al., 2020). To find more compact functional representations, the contrastive loss is used to regularize (C)NPs' training (Gondal et al., 2021). Hi- erarchical and mixture structures are also explored to induce diverse latent variables in NPs (Wang & Van Hoof, 2020; Wang & van Hoof, 2022). Lee et al. (2020) combines boostrapping tricks with neural processes to improve expressiveness of latent variables. Besides, NPs are applied address sequential decision-makinn problems (Galashov et al., 2019; Wang & Van Hoof, 2022). A summary of these models given Appendix (D.3).
Expectation Maximization & Wake-Sleep Algorithms. For log-likelihood maximization prob- lems with incomplete observations, expectation maximization (Bishop & Nasrabadi, 2006; Balestriero et al., 2020) is a tractable approach in optimization. It consists of E-step to opti- mize the distribution of latent variables and M-step to maximize the likelihood parameters. The optimization of VAEs and variants is also built upon EM framework (Ghojogh et al., 2021; Gao et al., 2021). Our developed algorithm can be interpreted as EM for NPs. Another family algo- rithms related our method is wake-sleep algorithms (Bornschein & Bengio, 2014; Eslami al., 2016; Dieng & Paisley, 2019; Le et al., 2020), where self-normalized importance sampling is used in Monte Carlo estimates. In this case, our method be also viewed as the extension of the reweighted wake-sleep (RWS) algorithm to NPs with an improvement guarantee. Another differ- ence lies in that optimizing a learnable functional prior of most importance in NPs, while RWS algorithms are mostly used in scenarios with fixed prior.
7 CONCLUSION
Technical Discussions. In this paper, we study NPs family from an optimization objective per- spective and analyze the inference suboptimality of vanilla NPs. Within the variational expectation maximization framework, our developed SI-NP improves the target likelihood step by step to obtain a more optimal functional prior. Besides, experimental results tell us the learned functional prior has close connections with the number of context points and complexity of function families.
Existing Limitations. Compared vanilla NPs, SI-NP requires more than one Monte Carlo sample from the functional prior in training, which consumes more computations. Though intuitively, the uncertainty of the functional prior can be forward propagated to the output distribution, the influence of such uncertainty has not been mathematically studied.
Future Extensions. The SI-NP objective is regardless of neural architectures so that any structural inductive bias be incorporated in modeling. The combination theoretically produce supe- rior functional representation baselines in the domain (Please refer to Appendix (H.5) for extensive experiments).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Dharmesh Tailor for the helpful discussions. We thank Eric Nalisnick and Tim Bakker for the helpful feedback in the AMLab Seminar talks.
REFERENCES
Randall Balestriero, Sébastien Paris, and Richard Baraniuk. Analytical probability distributions and exact
for deep generative networks. Advances in neural information processing systems, 13:14938-1444 2020.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. arXiv preprint arXiv:1406.2751, 2014.
Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Larry D Jackel, Yann LeCun, Urs A Muller, Edward Sackinger, Patrice Simard, et al. Comparison of classifier methods: case study in handwritten digit recognition. In Proceedings of the 12th IAPR International Conference Pattern Recognition, Vol. 3-Conference C: Signal Processing (Cat. No. 94CH3440- 5), volume 2. pp. 77-82. IEEE, 1994.
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimalitty variational autoen- coders. In International Conference Machine Learning, pp. 1078-1086. PMLR, 2018.
Zhenwen Dai, Andreas Damianou, Javier González, and Neil D Lawrence. Variational auto- encoded deep gaussian processes. In ICLR (Poster), 2016.
Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Adji B Dieng and John Paisley. Reweighted expectation maximization. arXiv:1906.058800 2019.
arXiv preprint
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29, 2016.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-leaming stationary stochastic process prediction with convolutional neural pro- cesses. Advances in Neural Information Processing Systems, 33:8284 8295, 2020.
Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint 1903v 1997
Angela Gao, Jorge Castellanos, Yisong Yue, Zachary Ross, and Katherine Bouman. Deepgem: Generalized ixspectation-maxiiizza cessing Systems, 34:1 1592-11603, 2021.
for blind inversion. Advances in Neural Information Pro-
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 7704-1715 PMLR, 2018a.
Marta Gamelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01623 2018b.
Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553):
452-459, 2015.
10

---

Published as a conference paper at ICLR 2023
Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. Factor analysis. probabilistic principal component analysis. variational inference, and variational autoencoder: Tutorial and survey. arXiv preprint arXiv:2101.00753 2021.
Muhammad Waleed Gondal, Shruti Joshi, Nasim Rahaman, Stefan Bauer, Manuel Wuthrich, and Bernhard Schölkopf. Function contrastive learning of transferable meta-reprrssetations. In Inter- national Conference Machine Learning, pp. 3755-3765. PMLR, 2021.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2018.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Representations, 2019.
I. Hospital, Analyse des infiniment petits. 1696.
David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58 (1):30-37, 2004.
Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. In International Conference Learning Representa- fions, 2020.
Jay Kerns and Gábor J Székely. Definetti's theorem for abstract finite exchangeable sequences. Journal of Theoretical Probability, 19(3):589-608 2006.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garell,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
Mingyu Kim, Kyeong Ryeol Go, and Se- Young Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In International Conference on Learning Represen- tations, 2021.
Diederik Kingma and Max Welling. 1312611312.61 2013.
Auto-encoding
variational bayes. arXiv preprint
Achim Klenke. Probability theory: a comprehensive course. Springer Science & Business Media, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tuan Anh Lc, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation of neural process objectives. In NeurlPS workshop Bayesian Deep Learning, volume 4, 2018
Tuan Anh Le, Adam R Kosiorek. N Siddharth, Yee Whye Tch, and Frank Wood. Revisiting reweighted wake-sleep for models with stochastic control flow. In Uncertainty in Artificial In- telligence, pp. 1039-1049. PMLR, 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee whye Teh. Boot- strapping neural processes. Advances in neural information processing systems, 33:6606-6615. 2020.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. Advances in Neural Information Processing Systems, 32, 2019.
Lars Maaloe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep gen- erative models. In International conference on machine learning, pp. 1445-1453. PMLR, 2016.
Saeid Naderiparizi, Kenny Chiu, Benjamin Bloem- Reedd, and Frank Wood. Uncertainty in neural processes. arXiv preprint arx.v:2010.03353, 2020.

---

Published as a conference paper at ICLR 2023
Art B. Owen. Monte carlo theory, methods and examples. 2013.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International conference on machine learning. pp. 1278-1286. PMLR, 2014.
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference on pattern recognition (ICPR2012), pp. 3288-388822 -291.. IEEE, 2012.
Surya Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinn Reviews: Computational Statistics, 2(1))54-60, 2010.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223. PMLR, 2018
Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with erarchical latent variables. In International Conference on Machine Learning, pp. 10018-10028. PMLR, 2020.
Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference Machine Learning, pp. 23055-23077. PMLR, 2022.
Qi Wang and Herke van Hoof. Learning expressive meta-repreesentations with mixture of expert neural processes. Advances in neural information processing systems, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist, a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 170.0077777 2017.
Heiko Zimmermann, Hao Wu, Babak Esmaeili, and Jan-Willem van de Meent. Nested variational inference. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2023
CONTENTS
Introduction
2 Preliminaries
3
Optimization Gaps and Statistical Traits
3.1
Inference Suboptimality in vanilla NPs
3.2
Evaluation Criteria & Asymptotic Performance
4
Tractable Optimization via Expectation Maximization
4.1
Variational Expectation Maximization for NPs
4.1.1
Surrogate Function for Exact NPs
4.1.2
Tractable Optimization with Self-Normalized Importance Sampling
4.2
Scalable Training and Testing
5
Experiments and Analysis
5.1
Synthetic Regression
5.2 Image Completion
Related Work
7 Conclusion
Frequently Asked Questions
15
Probabilistic Generative Process in NPs
16
C Predictive Distributions in GPs & NPs
16
D NPs Formulation & Structural Inductive Biases
16
D.1 Prior, Posterior & Proposal Distributions
16
D.2 Approximate ELBOs in NPs & Proof of Remark 1
18
D.3 Summary of NPs Family
18
D.4 Inference Gaps
19
E
Formulation of Variational Expectation Maximization Method
19
E.1
Proof of Improvement Guarantee using Variational Expectation Maximization
20
E.1.1 Meta Learning Surrogate Function
20
E.1.2
Improvement Guarantee
20
E.2
Importance Sampling in Variational EM Algorithm
20
E.3
Optimization Objective with Proposal Distributions
E.3.1
Update of Proposal Distributions (Optional)
21
13

---

Published as a conference paper at ICLR 2023
E.3.2
Derivation of the Proposal Update Objective
22
E.4
Gradient Estimates in Variational EM
22
F
Connection with CNPs
23
F11 Prior Collapse in SI-NPs with One Monte Carlo Sample
23
F.2 Effect of Adjusting Coefficients of the Prior Regularizer
23
G Experimental Setup & Implementation Details
24
G.I Meta Learning Datasets ................................
24
G.2 Neural Architectures & Optimizations & Evaluation Set-up
25
More Experimental Results
26
H.1 SI-NPs with A Learnable Proposal Distribution
26
H.2 Evaluation with More Monte Carlo Particles
26
H.3 Influence of Dimensions of Latent Variables
26
H.4
Sim2Real Experimental Results
................................
27
H.5
Augmenting SI-NPs with Attention Networks
28
H.5.1
Synthetic Regression
................................
28
H.5.2
Image Completion
28
H.5.3
Sim2Real
................................
29
H.6
Additional Visualizations
30
H.6.1
More Synthetic Regression Related Results
30
H.6.2
More Image Completion Related Results
30
I
Computation Tools
33
14

---

Published as a conference paper at ICLR 2023
A FREQUENTLY ASKED QUESTIONS
In this section, we list frequently asked questions from researchers who help proofread this manuscript. These raised questions are crucial, so we include more detailed discussions here. Meanwhile, we make extra efforts for readers quickly understand work and make slides as follows submi www.sssssiinnnsssseeeeeeeeseeeee pdf.
7C9iiddes-799//000 /icrr
Learnable Proposal Distributions. The reason why we set the update of learnable proposal distri- bution an optional choice is that we find it difficult to balance the optimization of Eq. (12) and Eq. (31). This is non-trivial and assigning equal weights in optimization results in unstable performance in experiments (Please refer to Section (H.1) for more discussion.). So, we leave the improvement of the proposal distribution future research. Suppose a reasonable objective or at least a weight scheduling mechanism for two objectives in two E-steps can be developed to stabilize the training. In that case, SI-NPs are likely to achieve even better performance.
Influence of Functional Prior Collapse. Generally, the generated functions are more diverse when the functional prior does not collapse: More randomness can be reflected from the functional prior, and the uncertainty of partial observation can be propagated to the output space. However, in ex- periments, we observe that for functions with more superficial structures, e.g. MNIST dataset, the deterministic functional representation in CNPs is sufficient to generate functions of high quality. So there is no consensus the influence of the functional prior collapse on the generation performance.
Influence of the Number of Inference Particles in SI-NPs. With more inference particles (latent variables), SI-NPs can avoid the prior collapse in some cases, but the cost is expensive for computa- tions. To trade-off performance and computations, we set the required number of particles for meta training in small quantity, e.g. 8 particles for image completion tasks.
between Different Optimization Objectives. In Table (3), we summarize the con- nections of different objectives with the meta dataset log-likelihood L()). Note that Lxp(0, o) is not the exact ELBO of (()). (M)) can also be viewed as the importance weighted method with equal weights for all particles. Since the estimates importance weights SI-NPs exploit the target observations and consider the difference in particles, this helps improve the model's 's general- ization capability. Though the self-normalized importance sampling makes SI-NPs' estimates over L()) biased, there is probabilistic convergence guarantee towards the true integral quantity of our interest (Please refer to Chapter 9 in the Book "Monte Carlo Theory, Methods and Examples" (Owen, 2013)).
Connections
Combination with Structural Inductive Biases. The primary research focus is on the optimality of the different optimization objectives, and this study applies more general stochastic processes (stationary or nonstationary cases). Theoretically, the objective of SI-NPs can be combined with most structural inductive biases in Table (4), especially when the function family is complicated. For example, the scale parameter of Matern kernels also varies with kernel values, bringing more variations in realizations. In this case, guess global latent variable is difficult to handle: SI-NPs and ML-NPs might encounter a similar performance bottleneck from the empirical observations in the main paper. To further improve the performance, we take the attention module (Kim al., 2019) as an example to conduct extensive experiments in Section (H.5). Since the meta learning exper- iment is computationally expensive and time-consune consuming in training processes, we do not examine combinations with other inductive biases in this paper.
GP Oracles in Synthetic Regression. Note that the GP oracle is computed in the form In pm Wi / mm) where the predictive distribution is mainly with a non-diagonal covari- ance matrix (suggesting Vn+m not independent in statistics). In comparison, the ligeliieldhood of NPs family is computed in the form Σ In ( dent w.r.t. the global latent variable 2. Such a difference in computations causes the mentioned gap. However, with a large neural model, such as the integration of attention networks in Section (H.5), the quantified performance gap of GP Oracles and the logelikelioood of NPs family is well reduced.
where y1:n+m is conditional indepen-
15

---

Published as a conference paper at ICLR 2023
Table 3: A Summary of Optimization Objectives in NPs Family. We list the available optimization
objectives in Section (3)((4). For Importance Weighted Estimates, multiple Monte Carlo samples are required in meta training.
Optimization Objective
Connection with (()) in Eq. (4)
Importance Weighted Estimates
LNP(1, ()
Approximate ELBO
LCNP(1)
Biased Estimate
X
LML-NP(()
Biased Estimate
V
NP (Ours)
Biased Estimate
/
B PROBABILISTIC GENERATIVE PROCESS IN NPS
Definition B.1 (Exchangeable Stochastic Processes) We denote a probability space of functions by (D, FP). P). Let Vny... be a probability measure Rd with [Z1,..., a finite index set. The process is called exchangeable stochastic process S XxX Rdd such that Vay.. F x Fx) = P(S,, E Fi,..... FF,....,zzy E FN) if it satisfies exchangeable consistency and marginalization consistency.
Here can translate the generative process of NPs in the following mathematical way.
T~ (T), 2 N(x pa (D, p(x), y~ ~~ (,,,)1,,22:0)) Vi f1,2..... m]
(15)
C PREDICTIVE DISTRIBUTIONS IN GPs & NPs
Here we take mieeiimesio deep Gaussian processes (Dai et al., 2016) as an example. With context points DC {(z,, 1 and target points D = {z, y1 1a+m = (er,yr), the key to applications is the predictive distribution p(f(xT) ,IT) = W Er). The conditional mean pr and covariance 2 functions Eq. (16) are permutation invariant to the order of context points.
PT mo( + ET. (vc ma(xc))
(16)
Ex = E ET,
Here the covariance matrix denoted by is computed with the context input TC = (II,..., x..) E
gaxd the target input IT (II,..., In+m) E [[++m)xd and a kernel function 1, e.g. [Ec,oli. (,;,,,,,, my), me is the mean function ma. and the context output is yc (yi,.... yn
R". Nevertheless, the computation of matrix inversion in Eq. (16) makes the runtime complexity as expensive as O(n + m)
D NPs FORMULATION & STRUCTURAL INDUCTIVE BIASES
D.1 PRIOR, POSTERIOR PROPOSAL DISTRIBUTIONS
Since fast adaptation is achieved in an amortized way, which reduces the gradient updates w.r.f model parameters learning function specifie latent variables with meta trained neural networks.
Definition D.1 (Permutation Invariant Functions) Let Sn be an n-element permutation group. For any permutation operator g Sn, the function is bijective mapping from the order set (1,2...., n) to itself:
g [1,2, ... n) + 191, 02..9......00 9n].
Then given a set of data points {x1, I2...... ;n), }, the function &b is said to be permutation invariant function if the following equation is satisfied Vg E Sn
(gg [21, xm)) (1 Tog...., F. Φ[[21, .2...., xn)))
16

---

Published as a conference paper at ICLR 2023
The context points and the target points treated as sets, so the amortized network should be permutation invariant W.Z.t. the order of data points.
Approximate Posterior Distribution. This is denoted by (=(=) in this paper. Usually, the approximate posterior used NPs (Garnelo et al., 2018b; Kim et al., 2019) and works as a proxy for the non-analytical exact posterior p[z]D?; ; :0).
Prior Distribution. This denoted by p(2 ( () in this paper. Unlike the approximate prior (2() used in NPs, we
exact functional prior in SI-NPs.
Proposal Distribution. This is denoted by 9n (= in this paper. The role of the proposal distri- bution resembles that of the approximate posterior in NPs. is used to sample latent variables and enables the computation of the importance weights in NPs.
Eembedding
21
1/1
MLP
T1
Kembedding
Pc
1/2
MLP
12
MLP
#
Σc
Kembeddinn
y/+
MLP
(x(2)D))
Mcan Pooling
or p<
Approximate Learned Functional Priors
2+
12-4-776
MLP
y
p(D
2;
II
p(un [=,,
Σ,
(b) Generative Distributions
Figure 7: Computational Diagram of Vanilla NPs. The blue one and the pink are, respectively, the encoder and the decoder structures vanilla NPs. The approximate posterior, the approximate prior, the learned prior, and the proposal distributions are in the same neural structure the encoder. For the generative distribution in the form of the decoder, the diagram shows one instance (I.. V/) E DJ in generation and the sampled global latent variable p(2[D,; () or z {=(=| shared across all data points in one realization.
In vanilla NPs, the latent variable is inferred from set of data points, and the distributions (a(2) and go (2) are learned via functions permutation invariant to the order of data points. An exam- ple module to parameterize (=<=(De) (2; Me) can be Eq. (17) with + mean pooling operator and (ho, } encoder networks.
N
r; = ha yal) V(2, v.) DC,
T/ [Pc. Ecl (s()))
(17)
The same with that in traditional stochastic processes, a realisation corresponds to a sampled func- tion f(X) generated in sequential way: (=(2)DD), f(X) p(Y|
Since this paper aims to study the influence of different optimization objectives concerning the NP models, retain the basic model set-up for all baselines. The Fig. (7) exhibits the neural chitecture the vanilla NPs, and such a structure is also shared with ML-NPs and SI-NPs in our paper.
17

---

Published as a conference paper at ICLR 2023
As previously mentioned, these methods be straightforwardly incorporated with other structural inductive biases (Kim et al., 2019; Gordon et al., 2019; Gondal et al., 2021; Wang & van Hoof, 2022). For example, in the following section, augment all methods with attention networks (Kim et al., 2019) and examine the resulting performance.
D.2 APPROXIMATE ELBOS IN NPS & PROOF OF REMARK 1
Inp(D? In p(DT (((
(18a)
2 (x) [mn [mpp(D]]=;0] DEL
(0(2)
;
LESOO (1,
(18b)
Approximate Posterior
Functional Prior
2 Bys(+) [Inp(D>]2; 2);
DKL
(0(2)
go(2|Dg)
(18c)
Approximate Posterior
Approximate Prior
For Eq. (18.b), remember that w.z.. these VAE-like methods, it is hard to guarantee the improve- ment of the evidence in each iteration when optimizing ELBO due to the existence of a posterior approximation gap. Meanwhile, the form of the functional prior unknown.
Proof D.1 (Remark 1) Vanilla NPs directly replace the real functional prior by the approximate one (z and introduce the consistent regularizer in Eq. (7). We further introduce the prior approximation gap in Eq. (19), which the sign is undetermined.
(x( p(
Lxp(O, 6) LELBO(d, o) Bgo(z) In
(19)
Prior Approximation Gap
Based on decomposition in Eq. (6V(19), we can find that there exists no consistent monotonic rela- tionship between L()) and (0 Φ). The invalid ELBO makes the optimization w.r.t. LNr(0, 0,,00 Φ) not always improve log likelihood L()).
((0) (0, d), ((0) & ())( YO E θ and & +
(20)
On the left side of Eq. (20), the ELBO of NPs can be bounded by the log-likelihood of meta learning dataset. Optimizing the ELBO can guarantee the optimal or local optimal solution when the approx- imation gap and the amortization gap are well closed. However, the right side of Eq. (20) implies the previous commonny-used strategies, e.g. normalizing flows for richer variational posterior dis- tributions (Rezende & Mohamed, 2015), auxiliary variables for augmented variational posterior distributions (Maalée al., 2016) and more flexible prior distributions (Tomezak & Welling, 2018), to close VAΣs inference gaps (Cremer el al., 2018) will not guarantee the performance improvement in theoretical sense. In other words, the consistent regularizer in NPs is ill-posed for optimization.
D.3 SUMMARY OF NPS FAMILY
In this part, we summarize the encoder and decoders for typical NP variants and point out the inductive biases behind them. The information is summarized in Table (4). We see variety of inductive biases can be used to improve NPs, most them from the model structure perspective rather than the optimization objective.
18

---

Published as a conference paper at ICLR 2023
Table 4: Summary of Typical Neural Process Related Models (Meta-Testing Scenarios). The recog- nition model and the generative model respectively correspond to the encoder and the decoder in the family of neural processes. Here [xc, ycl are context data points and we consider (I=,, y-) data point in target dataset.
Models
Recognition Model
Generative Model
Inductive Bias
CNP (Garnelo et al., 2018a)
fo(wes
conditional functional
NP (Garnelo et al., 2018b)
[0(2|(ze,)el Vcl)
pa(u- [2., x])
global functional
ANP (Kim et al., 2019; 2021)
4a (=l(xr, ycl) fag (2. (itc Mc].
(y [x=,, 2, z-])
global functional
local embedding
FCRL (Gondal et al., 2021)
fo [=( Ncl)
po [xt., z))
contrastive functional
ConvNP (Foong et al., 2020)
P=(=||<rc vcl)
po (M
convolutional functional
Conv-CNP (Gordon et al., 2019)
[z [ vc).
pa(u- po(y-1[EE, z=],
convolutional functional
FNP (Louizos al., 2019)
fo [no, x..)
po(1-12.)
latent DAG
D.4 INFERENCE GAPS
In this section, we apply the trick inference gap decomposition (Cremer al., 2018) understand vanilla NPs. Here we denote the approximate inference gap by D KT.. and the posterior approximation gap by DKL in Table (5).
Since is infeasible obtain the exact form for the functional posterior, cannot directly close the mentioned approximate gap, and approximate methods are used to learn them. In vanilla NPs, the consistent regularizer Eq. (7) works as the surrogate for the ELBO. But based on our analysis, closing this surrogate gap cannot lead to a theoretically optimal solution of functional priors.
Table 5: Inference Gaps in vanilla NPs. The + indicates the minimization to obtain the optimal inference solution. The sign means not applicable in deriving equivalent KL divergence form. The sign indicates the optimal posterior approximation in the family of variational distributions d² = arg mingeo Dx (x) || p(=[DF; v²)] Here H* consists of the optimal parameters in priors (( and the conditional distribution p(DT) 2; 0*).
Terms
Optimization Objective
KL Divergence or Gaps
Approximate Inference Posterior Approximation
C(0*) LELBO(U" Φ) ((0") - L8 +0))
DKL [g0(2) || pD= 0)
DxL. [9- (z) p(2 D( ;t))
Amortization
LELBO (0=, ,6) - LFI.OO (1" ;0,
DAL D
NP Prior Approximation
CELBO (0,, (0" )5-) - LND( ,0))
Surrogate Likelihood
C(D*) L(V; 0x)
2(0") - ((('))(((H)
E FORMULATION OF VARIATIONAL EXPECTATION MAXIMIZATION METHOD
In this section, we detail the progress of optimizing the NP model with the help of variational ex- pectation maximization algorithms. The concept of meta learning surrogate functions is introduced, and NPs are verified. Meanwhile, the improvement guarantee as well as other concerning technical points are included to better understand our method.
19

---

Published as a conference paper at ICLR 2023
E.1 PROOF OF IMPROVEMENT GUARANTEE USING VARIATIONAL EXPECTATION MAXIMIZATION
E.1.1 META LEARNING SURROGATE FUNCTION
Definition E.1 (Surrogate Function) Given the objective function f(0) to maximize, surrogate functions g 12.) w.r.t f(0) are a famity of functions with the following properties.
Both f(() and g 0k) are CC-sunctions which means f()) has first order and second order derivatives of those functions exist and these are smooth the domain of the function.
The following two formulas hold:
g(0; 0x) S f(0) vo; g(0x:0a) f(02).
(21)
When the objective function f(0) to maximize complicated, egg. multi-modal likelihood func- tions, surrogate function g(0; 0&)) enables casier proxy implementation with convergence guarantee to at least the local optimal. To see this point, recall the properties of the surrogate function (10x; 02) = f(0x). The update rule for the surrogate function follows that dx+1 arg max; g(0; 0g.. And this results in f (0x++1) 2 g(0x+1; 0x) 2 f(0)).
Recall the following function [(0; 0&) in the main paper.
[(0;0x) = Σ EPCEDTa 2np(DD22DDf;00 In p(=))
(22)
TET
The expectation operation in the k-th iteration step corresponds to E-step : (2( (2 (2) while the maximization operation in the k-th iteration step updates the parameter as M-step Ox+1 arg max, C(D; 1k).
Inp(D.ID,;d. F-step
((Dx:0s)
[(0++1:04)
(23a)
M-stop
Model Evidence
< S(00+1;; Drr.p(=/D,; [ In 0e+1)
(23b)
Model Evidence
E.1.2 IMPROVEMENT GUARANTEE
As illustrated in Eq.s (23), the surrogate function L(D; 04x) bounded by two log-likelihoods. Over the process of iterations, the log-likelihood is gradually increased to the final convergence.
:n S L(02;01) < " ((0
...
0n)
(24)
This indicates that the finally updated parameters of the surrogate function are exactly (sub-Joptimal ones for the evidence. Based on these rules, directly optimizing the surrogate function step by step can theoretically guarantec the finding of optimal or at least a local optimal parameters.
E.2 IMPORTANCE SAMPLING A VARIATIONAL EM ALGORITHM
Though the conditional marginal distribution p( D DDC pling trick can be used estimate the result with the help of proposal distribution (x(
2x) is not analytical, the importance sam-
'Note that the surrogate function above is defined in the Aimorize-Maximizen (MM) algorithm sense (Hunter & Lange, 2004).
20

---

Published as a conference paper at ICLR 2023
p(D 0k) p(D 21D;; 02)d2 / ()) p(D =IDS; 02).
de
(x(2)
2n
p(D 200) (x)
Σ
w(b),
(25)
B
(,(2(h) (
B b=1
with (6) (, D and with) p(D z/00
DD; 1kk)
2n (= (Dt)
Especially, the joint distribution is computed via the decomposition that p(D (1) DD; 0k.)
p(=(0)/ DD; (D 0z)) with p(D² |=(0) 02) III=1 p(yil<=;, [00]; 0n.)
With the above equation, the intractable optimization objective is transformed into a feasible one.
(2x) p(D; (R)
(FM(0:0k) (oIITT))))) Inp(DD,
[np(D 0)d=
(26a)
= =f (n(=
p(D DC;
;
02)
In x|D 0)d=
4n D ( [D; 0x)
(26b)
(
(D4 0p) In Inp (b)
ID+; (0)
(26c)
B
(bb
=ー w(br) In Inp(D
DF; ((;00)
200)
Σ (1)
lD , 21() DC; 0) (sI
(26d)
We can expand the term inside the expectation in Eg (26.a) follows.
2 (D -D) 9)
(27)
As for the exact posterior p(=IDT; 0z), get the following expansion.
p(2, :w) D(2 Ux) Ux)
p<=[D::0a) 0x) = Jp(z, D4:; 8x)dz J p( ;+ 2; 0x)dz
(28)
The distribution has the complicated denominator and this makes it infeasible to directly sample from the conditional distribution.
E.3
OPTIMIZATION OBJECTIVE WITH PROPOSAL DISTRIBUTIONS
This subsection shows the mentioned optional optimization step E-step #2 in Algorithm (1),
E.3.1 UPDATE OF PROPOSAL DISTRIBUTIONS (OPTIONAL)
The of proposal distribution y enables sampling for Monte Carlo estimates. Another role of the proposal distribution is work as proxy for the posterior p[=|DT; 0z), and the variance
21

---

Published as a conference paper at ICLR 2023
expected to be lower for stable training. So a reasonable optimization objective is to get two distributions close each other, egg. minimizing the Kull- back-Leibler divergence Dxr.pp(= ; (=
V 4n 0,(= D In (
:oa) p(a
This treatment equivalent wake phase updates in (Bornschein & Bengio, 2014). We ob- tain the optimization objective on the right side of Eq. (29) with the help of the self-normalized importance sampling.
min D Dxilo(e/D(;: (n [=]
(29)
+ min Lxi 1k- 0z)
Σ w In (x(x0))
Here the self-normalized importance weights [ inside the above equation are the same as that in Eq. (12). Also note that the the denominator B Ev= whh)) of the weight relates to the estimate of p(D² (DF),, which has biases with limited samples at the beginning of training. However, with the improvement of approximation, the bias can decreased accordingly (Zimmermann al., 2021).
E.3.2 DERIVATION OF THE PROPOSAL UPDATE OBJECTIVE
is trivial to see the following equation since the term E EpIIDTTO)) [p(=|D?; 9x)] is a constant.
min DKulp(=/DDD.... 0x.) (=()D e min -
(30)
Once again, apply self-normalized importance sampling to the right side of Eq. (30). With the same set of sampled latent variables, the reweighted objective w.z. the proposal distribution can be derived.
min - (nn
(b) In (2() CKL 7- Ux)
(31)
E.4 GRADIENT ESTIMATES IN VARIATIONAL EM
In the E-step #2, note that the model parameter is fixed as dx. estimate the gradient of w.r.t CKL(T), 7/--1, 8x) in the following way. This operation is to close the divergence between the proposal distribution and the exact posterior distribution.
0La (T (T
Σ (0) (0
(32)
On
On
In the M-step, note that the normalized importance weights are constant, the proposal distribu- tion fixed, and the gradient w.r.t (SINP(0; me, 0x) can be estimated in a straightforward way as follows.
(L 0z)
B =>at (1))) ( 20
(33)
00
We can see the role of normalized importance weights in gradient estimates from Eq. (33): remem- ber that (b) x pDD7 (b); ( ;x)) in the k-th iteration when the functional prior works as the proposal distribution, the more gradients will be allocated to those particles with higher generative likeli- hoods. Meanwhile, the normalized importance weights influence the optimization the functional prior distribution that the functional prior match the best of particles well.
22

---

Published as a conference paper at ICLR 2023
F CONNECTION WITH CNPS
F.1 PRIOR COLLAPSE IN SI-NPS WITH ONE MONTE CARLO SAMPLE
Theorem (L'Hôpital's Rule (Hospital, 1696)) Let f(x) and g(x) ) be two functions differentiable on an open interval I except possibly at point contained in I. If le f(x) lim g(2)
co, and (oce))(e)) + with Vo E I and # can have the following limit equation.
lim f(x) - g(x) x-
lim
g(
F(a)
lim --
((x)
f(x)
(34)
x x-
g(z)f())
g(x){(())
Note that in our default setup of SI-NPs, the functional prior works as the proposal distribution to sample the latent variable.
Proof F.1 (Proposition 3)
2 E Red be the latent variable for a diagonal Gaussian conditional prior p(=)D;; () (2; P (D.), (D()). Here the learned mean and the covariance matrix are simply denoted by μa = (p1............ E BRd and Eg diag [oi....,
from the conditional prior; it can be written as 2 = ps + E No),Ia) with help of a reparameterization trick (Kingma & Welling, 2013).
With one Monte Carlo sample &
+ 1n ln(TT) -
Σ Ino; (Mi - 2,)
d
()())) [Imp(2/Df;0)] se =
(35)
20
Eq. (35) is the result of one Monte Carlo estimate, termed as the collapse term in the main paper.
Built up these, rewrite the SI-NP optimization objective maximize Eq. (36).
LSINP Ep(=(De 12; Eptt DE:W)) [Inp(=|D{; 0)]
In ua + ED); - 2 Σ Ino, + (44-2;)3
(36)
ln(2T) +
20
Now we prove that when the learned variance parameter for collapse into the value zero, the optimization objective in Eq. (35) and Eq. (36) can be maximized. To simplify the notation, we put the mean variable pag aside, focus more on the variance variable Eg and let the value 2 denoted by Kg. We directly apply I.Hôôital''s Rule in Theorem (l) these quantities and obtain the following equations.
2
o²
lim oc
Ino;
Ri
lim
Inc;
lim
o² 0n + Ki
o.
of
o--20
-0?
K; Ina,
(37)
(o? lo? + +1))
20: In o; + G;
lim o.-50
lim o.
lim 0.
2 = +00
(-o?)
-20:
o.
Putting them together, the Gaussian latent variable will finally collapse into Dirac delta distri- bution and this demonstrates the equivalence between SI-NP with one Monte Carlo sample and CNP.
F.2 EFFECT OF ADJUSTING COFFFICIENTS OF THE PRIOR REGULARIZER
Remark Either increasing the number of Monte Carlo samples or lower the weight of the collapse term in SI-NPs effectively avoid the prior collapse.
23

---

Published as a conference paper at ICLR 2023
With increase of Monte Carlo samples, that of the collapse term, which indicates more weights are put in the gradient w.r.t the variance parameters to maximize the generative log- likelihood. In this way, it can naturally avoid the prior collapse caused by the second term.
the scale of the generative term outweights
Next, we analyze the influence of the coefficients with a E (0. 1) for the generative and functional prior likelihoods. To this end, we go back to the original meta learning objective and introduce the a-dependent one [(); a)) in Eq. (38).
max E p())DF)))) [In (D 0)] + (1 - a) (P(a [inp( 0)] = (();a)
In p(2|D); 0)
+ max p( [Im 2; 0 (1 2B (6)
In (2) v2x))
p(21DD;; Un)
max max puppD [mn((D] + (1 aa) [p(z|D] 0x) p(z
+(1 a) E ( [p<2|D?; 0s)
Constant
maxBpuupT:o [In((D] [2; + (1 aa)D [p(zIDF |l ((=
(38)
As noticed in L)):a), the KL divergence constraint enforces the functional prior p(=[D?; () to be close to the learned functional posterior p(2[D?" 12)) Since the functional posterior is derived from the full observation of function, this suggests that the relation of conditional entropy (x()))) < H() This trait is also empirically supported by Fig. (6), where the functional priors with fewer context points exhibit higher values in the trace of the covariance matrices. As consequence, larger a values impose less constraint on the KL divergence term and results in higher entropy of p(2 p(2)Df;00)
G EXPERIMENTAL SETUP& & IMPLEMENTATION DETAILS
In all experiments, we use Adam (Kingma & Welling, 2013) as the default optimizer for all experi- ments. Pytorch works as the toolkit program and run experiments.
The implementation of vanilla NPs/CNPs/ANPs follows the repository and related work (Garnelo et al., 201 80;;; Kim et al., 2019) in DeepMind. The implementation of ML-NPs is based on that in (Foong et al., 2020) except that the convolution modules are removed for the fair comparison since the inference objective is our research focus. To enable researchers method in studies, we leave the anonymous Github link here: ht attps:: : anonymous. 4open. scises science/ ///SII/SIII. S/SI_NPs C832, where we provide an example of SI-NPs. The full code implemen- tations of our method will be released in the final version.
implement our developed
G.1 META LEARNING DATASETS
Synthetic Regression. We use the Gaussian process simulator to generate different stochastic func- tions. Three types of kernels used to formulate diverse Gaussian processes. This the same as that in (Lee et al., 2020; Kawano et al., 2020).
For each task, generate from the uniform distribution 2[-2.0,2.0) and the mean function is zero. The number of context points is drawn from ~ U[3, 47] and the number of target points is + m with 03, 50 - ni]. The following kernel functions specify different Gaussian processes in this paper.
'https://pytorcc..om hhtpps: thub. com com/deppm nd/ /neur processess
24

---

Published as a conference paper at ICLR 2023
Matern
kernel:
v5d
5d
exp ( V5d
M(x = 1 1 +
312
with d = 4/2r 2/ 1 1.0) and 1 ~ (001,,066)
RBF kernel:
k(x,x') = 8² exp (- (xx
z²)2
212
with 8 U[0.1, 1.0] and 1 0.66
Periodic kernel:
-2 sin² ألللتعععاا
k(r, = exp
12
with 1.0,, (0011, 0.. and p U (0 1..10..5
Image Datasets. Benchmark image datasets include MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFARIO 10 (Krizhevsky et al., 2009) and SVHN (Sermanet et al., 2012). In meta training and lesting, we randomly select the number of context pixels n for each sampled batch of images U[1, 1023] in CIFAR (OFARIO/SVH), For pixel values,
(n ~ U11, 784] in MNIST/EMNIST and they are transformed to normalized Tensors via pytorch package. All other set-ups are the same as in (Garnelo et al., 8a;b;; Kim et al., 2019).
Sim2Real Dataset. This a part of additional experiments. We retain the preprocessing and meta training set-up in (Gordon al., 2019). The meta training datasets include the Lotka- Voltera sim- ulation samples and Predator-Prey' real-world dataset. All datasets are normalized before the meta training process. Please refer to (Gordon et al., 2019) for more details.
G.2 NEURAL ARCHITECTURES & OPTIMIZATIONS & EVALUATION SET-UP
Synthetic Regression. In terms of neural architectures. we use the same setup as that in (Gordon et al., 2019; Lee et al., 2020) for all baselines. The dimension of latent variables is 128. The Encoder is a two hidden layer neural network with 128 neuron units for each layer. The Decoder one hidden layer neural network with 128 neuron units. The optimizer's 's learning rate is 5e - 4. For all methods, we sample 100 tasks as one batch to train in each iteration, and the number of iteration steps in meta training is 100000. In meta training, the numbers of Monte Carlo samples for ML-NPs and SI-NPs are 16, while those in meta testing are 32.
Image Completion. The setup is the same with that in (Garnelo al., 2018a) and works for all NPs variants. As default, we set the dimension of latent variables z as 128 for all baselines. For all baselines, the Encoder is constituted with three hidden layers (128 neuron units each). The Decoder has five hidden layers (128 neuron units each) as well. The learning rate for the optimizer 5e 4. The training batch size for all images is and we meta train the model until convergence (the maximum epoch number for MNIST/FMNNST is 100, and that for CIFAR 10/SVHN is 200, and early stop used when it reaches convergence). samples are 16 for ML-NPs and 8 for SI-NPs (We find that 8 Monte Carlo samples are enough for SI-NPs to obtain competitive performance and increasing the number of particles will require more computations but the performance improvement is minor). In meta testing, the Monte Carlo sample numbers are 32 for latent variable models.
meta training. the numbers of Monte Carlo
Sim2Real. We use the same setup that in (Gordon et al., 2019; Lee et al., 2020) for all baselines. The neural architectures are the same as the Synthetic regression settings, except the dimension of output variables changes to 2. For other settings, please refer to (Gordon et al., 2019) for more details.
As in (Garnelo et al., 20188), for the Decoder in all models, the modified standard deviation variable d; = 0.1 + 0.9 * or for the output distribution p(y m, 2; 19) in all benchmarks, where a, ML.P, ( 21).
25

---

Published as a conference paper at ICLR 2023
Evaluation Set-up of Datasets. For Table (1)/(2 in meta testing, we keep the same set-up (Foong et al., 2020; Gordon et al., 2019; Lee et al., 2020; Kawano et al., 2020), which randomly select the number of context points in each batch and average the testing results of all batches as the final result. The range of the number of context points can be found in the above subsection.
H MORE EXPERIMENTAL RESULTS
H.1 SI-NPS WITH LEARNABLE PROPOSAL DISTRIBUTION
We have investigated the use of learnable proposal distribution in SI-NPs. this case, an additional proposal distribution (n(2)) N (2z Mr (,,(2) E. (Df )) is introduced to sample the latent variables. The neural architecture of such a proposal distribution the same as the approximate posterior in vanilla NPs, but the role is entirely distinguished.
Unfortunately, we find that it is difficult to stabilize performance even though coefficients of the generative likeliool In p(D" (2 (), the functional prior log- liglliielhhood In p(z((() [DC; ; ()) and the proposal likelihood in (z (D²)) are tuned in lot of trials. Since there is no convergence
solution when assigning the same weights to minimize Eq. (31) and the negative of Eq. (12) with shared optimizer, we do not report the result here.
H.2 EVALUATION WITH MORE MONTE CARLO PARTICLES
In the main paper, evaluating the exact likelihood ( In [Sp(D]] D)p(=/D;: ; is in- tractable for the studied latent variable models, so we report the evaluation results by setting the number of the Monte Carlo particles in Eq. (39) fixed.
1
B Σ
exp (Inp(DF1>(), .01] with 200) ~(=|Dff;
Lmc B) In
(39)
B
Empirical Explanation. We set B 32 in evaluation based on the empirical observations be- cause the evaluated likelihood for all experiments nearly reaches the convergence or does sig- nificantly increase. To see this point, we give the illustration of the evaluation results by varying the number of Monte Carlo samples and analyzing the test log ligeliielihodds in image completion tasks. As displayed in Fig. (8), SI-NPs show performance improvement with more particles in FMNIST/CIFAR 10/VHHN, and B 32 is sufficient enough evaluation", For CNPs, the perfor- mance does not change with more particles since the functional prior is collapsed. While for NPs, the performance goes far worse than amportance-weightee ones and does not significantly improve with more particles. So we only show the ML-NPs and SI-NPs for better visual comparison.
Theoretical Explanation. Note that ML-NPs and SI-NPs are importance weighted methods, the number of Monte Carlo samples matters. It is theoretically proved in Theorem in the paper impor- tance weighted autoencoders (Burda et al., 2016) that (Mc(V: B1) 2 (Mc B2) with B, 2 B2- Our developed SI-NPs be viewed the conditional version of importance weighted autoen- coders, which explains the empirical observations in Fig. (8). However, when the prior is collapsed deterministic embedding, we do not expect this effect.
H.3 INFLUENCE OF DIMENSIONS OF LATENT VARIABLES
It is unrealistic to explore all hyper-parameters' influence: e.g., learning rates, numbers of layers, dimensions of each layer, types of activation function, batch size in training, dimensions of latent variables, the Cartesian of these hyper-parameters causes dimension explosion, and the required time of running experiments can be more than one year with limited GPUs. So keep most of the set-up of the above hyper-parameters the same as that in original papers of CNP/NP/ANPs.
Even though in FMNIST, more performance gain is observed using SI-NPs with more particles, B = 32 sufficient and fair enough for all benchmarks.
26

---

Published as a conference paper at ICLR 2023
MNIST
Oat
CIFARIO
140
825
2-435
S1-MP
SHAP
255
S1-NP
250
990
Targst
2.000
250
2.4
2.64
0.875
100
120
120
2.45
20
20
90
191
100 120
Log Like
3.000
MLNP
NL-NP
2.150
MLNP
3.2
2.50
ML
2.075
2.100
0.375
2.60
203
A.2
1.060
20
23
80
100
320
300 120
20
40
50
80
1.00
327
20
45 60
60
300 120
Figure 8: Evaluation of Image Completion with Varying Number of Monte Carlo Particles. The x- axis records the number of Monte Carlo particles B in the {1, 4, 8, 16, 32, 64, 96, 128). The first row is to report the log-li oog-likelihood of the target points, while the second row is for the log-likelihood of the context points.
In this subsection, study the influence of dimensions of latent variables on SI-NPs. Considering that the meta training process is time expensive, we report the result on FMNIST image completion in Table (6). It can be observed that this factor rarely influences test performance.
Table 6: Test average log-likelihoods with reported standard deviations for image completion in FMNIST (5 runs). We test the performance of different optimization objectives in both context data points and target data points. We use 32 Monte Carlo samples from the functional prior to evaluate the average logllikeissoods.
dim.lat = 32
dim.lat = 64
dim_lat = 128 dim.lat = 256
#
context
target
context
target
context
target
context
target
CP (ours)
0.98
0.98 0.95
0.93
0.98 0.94
0.98 0.93
H.4 SIM2REAL EXPERIMENTAL RESULTS
Following that in (Gordon et al., 2019; Lee et al., 2020), We conduct experiments in Lotka-Volterra dynamical systems. The transition dataset is collected in the mentioned simulator for meta training. The testing scenarios include Lotka- Volterra simulators and a Predator- 's real-world dataset Hudson's 's Baye hare-lynx.
The meta testing results are reported in Table (7). It can be seen that SI-NPs can achieve the best performance in Lotka- Volterra and Predator- Prey datasets. Since the Predator-Prey dataset is out of the meta training distribution, the log- loglliielsooods are significantly lower than the Lotta-Volterra ones.
Fitling Besults for the Predator (Si-PPs)
Fitling Besults for the Pray (SINP)4
Pradace
Certad
Canae:
Tergel
tepulstton
Topuiadon
-25
-35
0.
1.
2.5
05
Figure 9: From the Left to the Right are population fitting results with #1 standard deviation in the predator and prey datasets with the meta trained SI-NPs. The r-axis corresponds to normalized years from 1845 to 1935 in order.
27

---

Published as a conference paper at ICLR 2023
Meanwhile, plot the predicted predator and prey population evolution in Fig. (9). As illustrated, meta trained SI-NPs can roughly characterize the trends and they can capture critical turning points.
Table 7: Test average log-likelihoods of target data points with reported standard deviations for Sim2Real (4 runs). We test the performance of different optimization objectives in both context data points and target data points. For each rum, we randomly sample 1000 functions as tasks to evaluate.
Sim (Lotka- Volterr)l
Rcal(Predator-PPee
CNP
-0.457+t005
-3.275
LeNe
-0.27
-3,012:10034
LMLNNP
-1.381 1..3811-0.999
-2.969210.66
LSLNP (ours)
-0.240
-2.334
H.5 AUGMENTING SI-NPS WITH ATTENTION NETWORKS
Theoretically, we can combine different optimization objectives of NPs with various structural in- ductive biases. Take the attention inductive bias an example; we augment the vanilla SI-NP with attention networks the same as in (Kim et al., 2019) and compare the augmented one with other augmented baselines. To enable fair comparison, networks. We apply the modification to all methods, and this operation results in ANP (Kim et al., 2019), ML-ANP and SI-ANPs.
also augment other baselines with attention
Neural Architectures. We retain all neural architectures the same as in Appendix (G.2), except for adding the deterministic path to obtain local deterministic variable (Kim et al., 2019). The deterministic path is built with cross attention encoder. Due to memory restriction, one head is used obtain 128 dimensional local variable, and the neural network set-up for query/key/value is the same as (Kim et al., 2019)..
Optimization. We retain the optimization step the same as those in Appendix (G.2).
H.5.1 SYNTHETIC REGRESSION
In Table (8), can be seen SI-ANPs outperform ANPs in all kernel cases. SI-ANPs show a slight advantage over MI.-ANPs in Marten and RBF kernels. We also observe anomaly results of ANPs in the Periodic kernel case, which illustrates that the use of attention might deteriorate the performance.
Table 8: Test average log-likelihoods of target data points with reported standard deviations for 1 1-dimensional Gaussian process dataset with various kernels (5 runs). For each run, randomly sample 1000 functions as tasks to evaluate. All NP models are augmented with attention networks. Settings are same as in the main paper.
Matern
RBF
Periodic
LANP (Kim et al., 2019)
0.98
1.06_0am0
-1.3462a.0z
LMIL-ANP (Foong et al., 2020)
0.985
1.062
0.574_0.z2
ANL (ours)
0.995
1.071
0.56
H.5.2 IMAGE COMPLETION
In Table (9), we report the results in image completion. We notice that the SI-ANP significantly beats other models in FMNIST/SVIN/CIFARII and is comparable with the ML-ANP in MNIST. Mcanwhile, the performance gap between vanilla NPs and SI-NPs in Table (2) is quite huge, while this situation is alleviated after adding attention modules. All of these indicate that incorporating structural inductive biases in complicated tasks is also necessary. Combining SI-NPs with more powerful structural inductive biases is the top choice for boosting performance.
hhtpps://gi thub. com/deepmiiiooooooeeeeeeee wn/ /nural a wwwerrllpprsee -processes
28

---

Published as a conference paper at ICLR 2023
Table 9: Test average log-likelihoods of target data points with reported standard deviations for im- age completion in WIIIST/EMIIS/SSSIN/IAAARR
(4 runs). All NP models are augmented with attention networks. Same in the main paper, we test the performance of different optimization objectives in target data points. We use 32 Monte Carlo samples from the functional prior to evaluate the average log- oogliiklisoods.
MNIST
FMNIST
SVHN
CIFARIO
LAN (Kim et al., 2019)
1.173 10.08
1.101 0000
4.01 I 1:0005
3.605
0.016
LML-ANP (Foong et al., 2020)
1.216
1.172 10,009
1.01710000
3.545
LIS-ANN (ours)
1.21120004
1.174_0.000
4.040
3.710
H.5.3 SIM2REAL
In Table (10), we report the results with the attention module augmentation. As observed, ANPs, ML-ANPs, and SI-ANPs exhibit comparable performance in Lotka- Voltera simulation. As for the Predatory-Prey testing results, the conclusion similar to that in Table (7) without attention augmentations.
Test average log-likelihoods of target data points with reported standard deviations for Sim2Real (4 runs). We test the performance of different optimization objectives augmented by attention inductive bias (Kim et al., 2019) in both context data points and target data points. For each run, we randomly sample 1000 functions as tasks to evaluate.
Table 10:
Sim (Lotka- Volterra)
Real Real(Predator-Pre)
CNP
2.21 I 40017
-3.174+0.311
LMI-ANP
2.203
-3.624+0.a52
LSI-ANP (ours)
2.203
-2.822+0316
In this case, we guess the local deterministic embedding from the attention network plays a more critical role than the global latent variable in fitting these two datasets. The visualization of fitting Predator-Prey samples with SI-ANPs is given in Fig. (10). The real-world samples are well fitted with well quantified standard deviations. We notice that the measured predictive mean square erros and the negative log-likelihoods of the target data points in the Predator-Pae testing dataset are comparable using SI-NPs and SI-ANPs. However, the fitting result of the context data points are distinguished a lot: the mean square error of SI-NPs in 4 runs is 0.285-10.007 and that of SI-ANPs 0. 0.029 0.044. This implies that the attention network in the real-world dataset focuses more on the context point fitting while the global latent variable is for the general trend characterization.
Fitting Results for the Predstor (SHANNP5)
Fitting Results (SIANPs)
Cretiths Near
Treakove Moin
Tagu
Terge:
Population
Population
05
Figure 10: From the Left to the Right are population fitting results with +1 standard deviation the predator and prey datasets with the meta trained SI-ANPs. The æ-axis corresponds to normalized years from 1845 to 1935 in order.
29

---

Published as a conference paper at ICLR 2023
Curve Fitting with NPs
Fitting MPs
Curve Fitting MLNPs
Droun Truts
Truss
Girur Trat
03
-24
-11
38
-2
Curve fitting with St-ils
Eurve Fitling with NPS
Curve Fitling with ChPs
Curve Fitting with NL-IS
Producuvs Moce
-02
-04
-12
Gound Tlun
-26
Graudd Neat
Fradet
-38
Fitting with
fitting
Fitting
Fitting
Oeeure Tree Prddctiva
Preditive
Marr
-35
-15
-26
-55
Frodict Heass
Figure 11: Examples of curve fitting in all kernel cases. From the up to the down rows are respectively Marten, RBF, and Periodic cases.
H.6 ADDITIONAL VISUALIZATIONS
H.6.1 MORE SYNTHETIC REGRESSION RELATED RESULTS
We include more visualized results with the meta-traines models for Gaussian process datasets in Fig. (11). An illustrated, periodic kernel cases are most challenging, and SI-NPs can learn faint but crucial fluctuation signals from mean functions, while vanilla NPs fail to capture them. For other cases, the behaviors of these models are similar characterizing trends: vanilla NPs tend show higher variance in Marten kernel cases. SI-NPs and ML-NPs are comparable in Marten and RBF kernel cases. CNPs seem to best match the context points both cases.
In Fig.s (12)(1))(144,, we respectively plot the curve fitting results in all kernel cases when all la- tent variable models are augmented by attention networks. In Marten and RBF cases, all attention augmented models well fit data points. Nevertheless, in RBF cases, SI-NPs can better capture data point-dependent deviations. In Periodic cases, it can be observed that SI-ANPs can precisely cap- ture fluctuations and quantify more convineing uncertainty. Sometimes, ANPs underestimate the uncertainty while ML-ANPs fail to show data point distinguished uncertainty.
H.6.2 MORE IMAGE COMPLETION RELATED RESULTS
In Fig. (15), the scale of computed KI. divergence values in vanilla NPs is positively correlated with the semantic complexity. Also, note that approximate functional prior vanilla NPs seldom collapses, but it has theoretical bias away from the optimal functional prior according to Remark (I). We can also find with more context points, the approximate prior is closer to the approximate posterior, the value decreases accordingly.
In Fig. (16), we sample a collection of images from MNNSTIAANISTISSSNCIIIARR to visualize the completed results. We can find that without structural inductive biases, SI-NPs reasonably complete the images in MNIST/EMNIST//SVE tions. The learned functional prior, such that in SVHN, generate images that are different from
and exhibit the uncertainty from partial observa-
30

---

Published as a conference paper at ICLR 2023
Curve Filting St AMP
Curve Bing with AMP
Gurve FiLig with NLANPS
Predctiv Near
64
Pedlctira Fair
Prettlee mas
60
02-
5.
-44
-13
-13
05
10
-25
15
-15 -23
Curve Fitling with S-ANPS
Curw no with ANP
Curve Filkin with NL-APPs
64-
Ferete
54
62
60
-.2
-03
-04
-46
-30 1.5 -ir-dd
16 35
-30
00
11
18
15
-32
-35
Curee Fitingeth SLAMPs
Cunve Fitin with
Ourve Fithns NL
60
43
-24
-46
TruedTras
40 45
-1
10
40
-5
-55
60
35
14
25
36
25
Figure 12: Examples of Curve Fitting in Matern Kernel Cases.
Curve Ritting SLAMPS
Curve Eit with
Filtin NL -N-N
02
-42
Curve Riting S1 ANPS
Curve Fgg with ANP
Curce FiLog with MLANPS
Padcoseer
the Farr
02
60
-22
-02
-54
-64
Curae fliting SLAMPS
Curse Eltt with ANP
Filt MI -MPs
50
-02
J02
-24
-46
-04
220
Figure 13: Examples of Curve Fitting in RBF Kernel Cases.
31

---

Published as a conference paper at ICLR 2023
Curse Fisting SL-MP
Curoe Fittin with AMPs
Dunver Fil with ML-APPs
-1A
m
Gound Teat Prddo
15
Curve Fitling with 31 ANPS
Curve ng with ANP
Curve Fill with NL-ANS
65
-04
-15
Figure 14: Examples of Curve Fitting in Periodic Kernel Cases.
2206
(MNIST)
(FMNIST)
2750
ICIFAR10)
000
NP (SWHN)
1300
300
500
200
000
110
400
200
420
620
202
Figure 15:
Evaluation KL Divergence Terms in Vanilla NPs. meta testing, we still vary the number of context points in image datasets. For NPs, the KL divergence value DEL [2)(2) |l (=()) is computed.
the ground truth due to partial observation. As for CIFARIO, it has more complicated semantics and challenging for SI-NPs with only MIPs in neural architectures.
For cases when SI-NPs are augmented by attention neural networks (Kim et al., 2019), we show more generated examples with learned Si-ANPs in Fig. (17)/(18). As can be seen, the completed results are not blurred and have high quality. Notably, the quantified variances are decreased with the increase in the context points, which shows excellent asymptotic behavior in this domain.
9
5
9
60
9
6 9
9
Figure 16: Examples of Image Completion Results using SI-NPs. From top to bottom in rows are original images, context points, means, and variances of completed images.
32

---

Published as a conference paper at ICLR 2023
Figure 17: Examples of CIFARIO Image Completion Results using SI-ANPs. From left to right and top to bottom are cases with 10, 100, 300, 500, and 800 randomly selected pixels as the context points.
Figure 18: Examples of FMNIST Image Completion Results using SI-ANPs. From left to right and top to bottom are cases with 10, 100, 300, 500, and 700 randomly selected pixels as the context points.
I COMPUTATION TOOLS
In this project, we use NVIDIA 1080-TiGPUs to finish experiments.
33


Reviews:
[{"review_id": "uEB_Ckond7h", "reviewer": "Reviewer_BUNX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies Neural Process (NP)-based meta regression. The work is motived by the well-known observation that the exact ELBO of NPs is intractable, which requires an approximation that destroys the guarantee that the resulting NP-objective is a lower bound to the log marginal predictive likelihood. The paper proposes a novel objective function for NPs, inspired by an Expectation Maximization (EM)-like approach, which is shown to offer an improvement guarantee. The approach is validated on several synthetic and image-completion experiments.\n", "summary_of_the_review": "The paper tackles an important question in NP-based research. Unfortunately, the presentation lacks clarity and the experimental results are inconclusive. In it's current form, I judge this paper not ready for publication, but encourage the authors to provide an improved version, incorporating my suggestions above.\n", "strength_and_weaknesses": "The paper tackles an important question in NP research, namely the validity of approximating what the paper calls the \"functional prior\" with the set encoder distribution. Unfortunately, the presentation of the approach lacks clarity, which makes it hard to judge its validity. Also, the experimental results are inconclusive. Please cf. my comments below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hD3YLRkfoXm", "reviewer": "Reviewer_sw5r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the inference sub-optimality issue of neural processes. To achieve competitive performance of NPs, the paper proposes a surrogate objective of the target log-likelihood in a meta learning setup, and introduces a tractable way to optimize the surrogate objective through variational expectation maximisation. The author(s) further validates the method with image completion tasks on MNIST, FashionMNIST, SVHN and CIFAR10.", "summary_of_the_review": "This paper is solid for publication in ICLR. ", "strength_and_weaknesses": "Strength:\n\nThe paper did thorough analysis on an important issue of inference sub-optimality of NPs, and introduced the novel self-normalized importance weighted neural processes model to address the issue. The paper is well written with comprehensive derivation in the appendix. \n\nWeaknesses:\n\nThis is a great paper, but the reviewer has some minor questions hoping that the author(s) could help to explain or address:\n\n1. Is there any proof of Proposition 2? It is unclear to the reviewer why it leads to an improvement *guarantee* w.r.t. the log-likelihood. \n\n2. In 4.2, it mentions that \"skip E-step #2 to avoid unstable optimisation in empirical results.\" Does it mean that using a learnable functional prior as the proposal is the key to avoid unstable optimisation? Is there any case that the author(s) would suggest to do E-step #2 to update proposal distribution in wake phase update? \n\n3. In 5.1 Table 1, for the functions sampled from GP with Matern - 5/2 kernel, the performances of SI-NP and ML-NP are of no statistically significant difference, but for ones with RBF or periodic kernels, SI-NP is much better than ML-NP. Is there any intuition or insights on this? \n\n4. In 5.2 Table 2, it seems that the improvements of SI-NP vs CNP or ML-NP is marginal on image complementation tasks. Would you expect a larger improvement with higher number of MC samples?\n\n5. The Sim2Real experiments in the appendix seems very interesting. It would be nice to have it in the main text and with some insightful discussion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ghva-dz7nB", "reviewer": "Reviewer_HNS9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study neural processes from an inference perspective, drilling into questions about how well the NP family really works as a function of the objective. In the course of this analysis, they propose a self normalized impottance sampling scheme to estimate the expectation over the posterior over the data latent variable z in p(z|D_t) and use that to obtain a  tighter estimate of the training objective. \nThey link this procedure to variational expectation maximization and show how iterative optimization of this objective leads to better NP models.\n\nAs an aside, the authors study the resulting \"prior collapse\" in their function prior p(z|D_c) as a function of training objective and link it to performance and diversity in function space.", "summary_of_the_review": "This work focuses on providing more depth into our understanding of neural processes from an inference point of view and suggest a modification which improves upon the commonly used strategies in a principled way.\nWhile a lot of issues remain with the model class, the clean approach used here and modest empirical gains suggest that this work should be interesting to the community working on neural processes and adds useful knowledge.\n\nGiven those I would lean towards recommending this paper for acceptance, even though I remain interested in unsolved topics surrounding modeling aspects of NPs which remain unadressed here but may also affect uncertainty and similar aspects the authors touch upon.", "strength_and_weaknesses": "Strengths:\n- studying the NP objective thoroughly and sidestepping some of the previous simplifying assumptions previously made is interesting and relevant here\n- the authors propose an interesting strategy using importance sampling to estimate expectations over p(z|D_t) and utilize that in a variational EM scheme\n- the results show quantitative evidence that this strategy outperforms the more commonly used training strategies\n\nWeakness:\n- While the authors bring up uncertainty at some point, the results shown here do not suggest that the ucnertainty estimates are dramatically improved\n- the function prior still basically collapses to a delta function, which feels like a pathological issue in the setup and is not quite prevented here merely by the inference scheme\n- inside the variational EM scheme there are still dramatically simplified assumptions present such as setting variational families to the prior etc., this is still relatively dissatisfactory when considering that this prior tends to collapse\n- in the GP example, the NP models proposed here outperform the baseline models, but are still dramatically far off from the GP.  This suggests that the NP model class does not really appear to be a suitable model as suggested here for this modeling task and surprises me. I would expect that a suitably large neural model with enough data should at least get predictive means similarly right as a GP, it is not clear to me this is the case here for the entire model class irrespective of inference strategy. If so, how useful is it to compare to GPs , really?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PjHaBzupno6", "reviewer": "Reviewer_ki6V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a new technique for inference in neural processes (NPs) based on expectation maximisation. The technique is related to what was proposed by Foong et al. (2020)—convolutional NPs—where multiple samples from the latent given task context are sampled to construct the likelihood objective. Unlike Foong et al., the authors here propose to use importance weighting to approximate expectation with respect to the true posterior. This is motivated by author's theoretical derivations which show that the consistency regulariser used in previous work on NPs causes suboptimal inference.", "summary_of_the_review": "I am not an expert on NPs. AFAICT, the paper is well written, and presents an advancement to the state of knowledge about NPs. If that is correct, I believe it justifies acceptance, even if some of the empirical gains are marginal, and the computational cost is somewhat high.", "strength_and_weaknesses": "### strengths \n* sensible theoretical motivation\n* better empirical results than baselines \n\n### weaknesses\n* the empirical improvement is somewhat marginal compared to the oracle (see Table 1)\n* increased computational complexity", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster