Decoupled Kernel Neural Processes: Neural Network-Parameterized Stochastic Processes using Explicit Data-driven Kernel
Authors: Daehoon Gwak
Gyubok Lee
Jaehoon Lee
Jaesik Choi
Jaegul Choo
Edward Choi
Venue: ICLR 2022 Submitted (2022)

Abstract: Neural Processes (NPs) are a class of stochastic processes parametrized by neural networks. Unlike traditional stochastic processes (e.g., Gaussian processes), which require specifying explicit kernel functions, NPs implicitly learn kernel functions appropriate for a given task through observed data. While this data-driven learning of stochastic processes has been shown to model various types of data, the current NPs' implicit treatment of the mean and the covariance of the output variables limits its full potential when the underlying distribution of the given data is highly complex. To address this, we introduce a new neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), which explicitly learn a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner. By estimating kernel functions with self- and mixed attentive neural networks, DKNPs demonstrate improved uncertainty estimation in terms of conditional likelihood and diversity in generated samples in 1-D and 2-D regression tasks, compared to other concurrent NP variants. Also, maintaining explicit kernel functions, a key component of stochastic processes, allows the model to reveal a deeper understanding of underlying distributions.

Under review as a conference paper at ICLR 2022
DECOUPLED KERNEL NEURAL PROCESSES: NEU- RAL PARAMETEREEE STOCHASTIC PRO- CESSES USING EXPLICIT DATA-DRIVEN KERNEL
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Processes (NPs) are class of stochastic processes parametrized by neural networks. Unlike traditional stochastic processes (e.g. Gaussian processes), which require specifying explicit kernel functions, NPs implicitly learn kernel functions appropriate for given task through observed data. While this data-driven learning of stochastic processes has been shown to model various types of data, the current NPs' implicit treatment of the mean and the covariance of the output variables limits its full potential when the underlying distribution of the given data is highly complex. To address this, we introduce new class of neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), which explicitly learn separate mean and kernel function to directly model the covariance between output vari- ables in data-driven manner. By estimating kernel functions with cross-attentive neural networks, DKNPs demonstrate improved uncertainty estimation in terms of conditional likelihood and diversity in generated samples in 1-D and 2-D regres- sion tasks, compared other concurrent NP variants. Also, maintaining explicit kernel functions, a key component of stochastic processes, allows the model to reveal deeper understanding of underlying distributions.
INTRODUCTION
Neural processes (NPs) (Grneloo et al. 2018 b) are a class of stochastic processes parametrized by neural networks. By embracing statistical properties stochastic processes, NPs can effectively esti- mate the uncertainty of underlying distributions of functions with a set of realizations and their data points. Different from traditional stochastic processes (e.g., Gaussian processes (GP) (Rasmussen) & Williams 2006)). NPs learn data-driven stochastic processes without a need to specify or keep an explicit form of kernel functions. As a result of their simplicity and flexibility, there have been numerous efforts to further develop improved variants of NPs (Kim et al. 2019 Lee et al.) 2020
Gordon et al. 2020) and apply them to various downstream tasks (Singh et al. 2019 Requeimal
al. 2019).
Though significant progress has been made in NPs, the current architectures of NPs either fails to capture output dependencies in Conditional NPs (Garnelo et al.) 2018a| Gordon al., 2020). or indirectly capture the full stochasticity present in the traditional stochastic processes. For instance, different from GPs, conventional NPs reserve stochasticity in global latent variable and output variables separately. The output variables estimate point-wise uncertainty, which corresponds to the diagonal elements of a kernel matrix. Similarly, the global latent variable takes charge of the fune- tional uncertainty and diversity, represented by the full covariance matrix in GPs. Due to this induc- tive bias of conventional NPs, the role of estimating functional stochasticity is mainly assigned to a fixed-length vector (i.e. the global latent variable), and consequently, capturing the underlying dis- tributions can be restricted in complex scenarios (e.g., variable relationships are periodic or abruptly changing at a certain point). Although several approaches attempt to alleviate the problem by intro- ducing attention (Kim et al. 2019) and bootstrapping (Lee et al. 2020) on top of conventional NPs, the problem still exists as the architectural limitation (i.e. implicit modeling of the mean and co- variance) has not been addressed directly. Besides this, as NPs implicitly learn the kernel functions inside the model, the interpretability of kernels such in GPs (Lloyd et al. 2014) is diminished.

---

Under review as a conference paper at ICLR 2022
ANP
Ours
Ground Truth
Ours
400
46
-45
42
P rediction
Covariance matrix
Figure I: (Left) Comparison of predictions given context points (black dot) by the Attentive NP (ANP) and DKNP (Ours) after learning samples generated from periodic kernel with fixed hyper- parameters. The mean (blue curve) and sigma (shaded blue area) predicted from the DKNP better represent the data, including target points (red dot) compared to the ANP. (Right) Visualization of kernel functions learned by the DKNP. As result of kernel learning in the DKNP, the underlying prior distribution of data can be inferred (periodic kernel in this case).
To address this concern, we propose Decoupled Kernel Neural Processes (DKNPS), a new class of neural stochastic processes that explicitly learn separate mean and kernel function to directly model the covariance between output variables in a data-driven manner. Our experiments in I-D and 2-D regression tasks reveal that the DKNP outperforms concurrent NP variants in terms of predictive likelihood, better global coherence of generated samples, and improved interpretability via explicitly learned kernels.
BACKGROUND
2.1 NEURAL PROCESS
Given a stochastic process sample consisting of points, let us denote the input and output as X [= and Y { /== respectively, where E dRdx and y; E Rdy, For set of target input XT = hee C X, NPs model the conditional distribution of target outputs Yr conditioned on the context set (Xc, Yc) {(x,, y.)lec using factorized Gaussian distribution:
Dog(Yr(Xr, = Σ logp(yi Xc, iET
(I)
For obtaining the predictive distribution p(y: x,,, Xc, Yc). NPs use an encoder-decoder architec- ture that ensures the permutation invariance of the predictions of the target points given the context set (Xc, Yc).
Kim et al. (2019) and Lee et al. 2020). we consider the NP encoder consisting of two separate paths, namely the deterministic path and the latent path. For the deterministic path, fo rep- resents each context points in {(a;; hiec r; E Rdd. i.e., fo( y:). Then, we aggregate the r;'s by averaging them across all context points, re Σiec F; where n. = |C'. This vector Te E Rd is the summarized representation of context points, and it is permutation invariant over the order of (æ,, y:) (Xc, Yc).
Following
The latent path of the NP encoder operates in similar fashion to the deterministic path, i.e., ec = 1 Eiece, e; where e; = [o([;;; Unlike the deterministic path, however, the la- tent path uses stochastic layers for obtaining distribution of the latent variable E IR:;; g(zlec) = A(z: o², where pe. o are the output of the additional fully-connected layer applied ec. Finally, by concatenating these aggregated vectors TC, z with the target inputs x; Xr, the decoder hip produces the predictive distribution p(y | TC, = =Nyii My of) where (,, (x,, rc, 2). Note that the oy is used for capturing the point-wise uncertainty. During the training phase, the parameters are learned by maximizing the evidence lower bound of log p( po(Y/|X, [x,, Xc. Yc) via the reparametrization trick (Kingma & Welling 2014: Rezende et al. 2014) as follows:
ne
2

---

Under review as a conference paper at ICLR 2022
Muitchead
MLPp
Attention
y4,
Query
Eay
M..P
MLP
yt.
MLP.
Esy
fp
P
C2
Xea
MLP3
ht
Multchead
Gross
Xrg
MLP
ht,
Quary
Kay
Vollan
MLP.
W:y
WW
Xr.
MLPS
hr
Res
MLP.
Rrg
X1g
MLP
Figure 2: Model architecture of DKNPs. DKNPs estimate the predictive distribution /V (y: My Ey) by using attention- based deterministic paths using Multihead Tross-Attention (MCA) to model the mean vector (upper path) and the full covariance matrix (lower path).
logp(Yr(Xx Xo, Yc) Yx) p(y z)] [I(((=)er))l(((eee))
Xr, Yr) is distribution of latent variable by encoding the set of target points (Xy, Yr) = (ax, yy)) her.. This objective function consists two parts: 1) the reconstruc- tion loss for the target points and 2) the KL divergence term, minimizing the divergence between two distributions ( Yr) and (=)|Xc, Yc). Note that, in practice, we assume that X C XT encourages two distributions inferred by the
where (2(er)
during the training phase. The KL divergence context sets and target sets to be similar, which is reasonable because two sets are generated from the same function. Therefore, during the inference phase, the distribution q( [c, Yc) captures the functional stochasticity which is demonstrated with the coherent sample generation. It can be thought that learns to capture the correlation output variables of the stochastic processes.
2.2 ATTENTIVE NEURAL PROCESSES
Multihead Attention Given key-value pairs of matrices K Raxdmode and V R Ruxdmode, and m queries Q E Rmxddmode the scaled dot product attention is formulated as:
QKT
Attention(C (Q, K, V) = Softmax
(Vdrnoted Vimodel
V
E
R R.Xddmddee
(3)
where K, V, and Q are projected by learnable linear maps WK WV. and W.O from the source and target T.
The attention mechanism can calculated from multiple subspaces, namely, multihead attention (MHA) (Vaswani al. 2017). Denoting single attention head as H Head, Altention(Qi, Ks, Vi), the aggregate attention from multiple subspaces can be expressed as:
MHA(Q, K, V) = Concat (Head,.... Head W E RmXdmoded
(4)
where WO is the learnable linear map for aggretating the subspaces.
Attentive NPs Attentive NPs (ANPs) leverage attention to resolve the underfitting issue in NPs. Unlike NPs that produce a single variable rO from the deterministic path, ANPs utilize a query- specific variable r by applying the attention score a, for each r; during the aggregation of the deterministic path, formulated as r Eiec a; Ts. The attention-based aggregation of ANPs resembles how the GPs utilize the correlation to estimate the predictive distribution of the context and the target points.
DECOUPLED KERNEL NEURAL PROCESSES
Decoupled Kernel Neural Processes use attention to explicitly learn a separate mean and kernel function to directly model covariances between output variables with related input variables in data-driven fashion, which is contrary to conventional NPs that implicitly model the mean and kernel function through the latent variable z. As shown in Figure 2 DKNPs estimate the predictive
3

---

Under review as a conference paper at ICLR 2022
distribution multivariate Gaussian NN (y: My: Ey) by using two attention-baseld deterministic paths using Multihead Cross-Attention (MCA) to model the mean vector (upper path) and the full covari- ance matrix (lower path). Here, attention modules are extensively utilized as Le et al. (2018) ; Kim et al. (2019) have demonstrated attention was helpful in achieving low predictive uncertainty near the context points. With the predictive distribution (y; My, Fny), DKNPs are trained and evaluated based on the function likelihood.
The core design of DKNPs is motivated by the predictive posterior distribution of GPs, where Xc, are used for deriving the posterior mean but only Xr,Xc for deriving the poste- rior covariance as follows:
GP:
P(Yr|Xr,X,, Yo)= =N YY; Yr; Xxaxo Exz.xr - Exo,x, Exorxe Excrx))
(5)
DKNP: NC, =
wo (V:; MCA(MLP.(XX MLP. (XT, Xc)),
MIP o MCA Xc)))
(6)
where, in contrast to GPs, DKNPs covariance is learned via attention This decoupled process allows DKNPs to explicitly learn the prior of the given dataset, and thus act as a true generative process, ensuring the global consistency of all points in the given stochastic process samples. Unlike DKNPs, NPs pack all information (Xc, Yc, Xy) into latent variables to derive both mean and co- variance, thus inherently becoming conditional process that requires sufficient amount of context points, unable to explicitly learn a prior.
Specifically. the DKNPs pass each context point { ys) hie,, the concatenation of x; and yi, to MIP, and represent it as e;. Similarly, we produce the representation vector of r;; h;, using another MLP a for all inputs {ar: JECUT. Then, e; and h; passed to the MCA module create the mean vector My. All heads in the MCA module perform cross-attention, Q = {he heT, K = {hr hec and V Hec We adopted the architecture of MCA used in image transformers (Parmar al., 2018). where the original query vectors are added to the output from the MCA through the residual path. This allows to do inference the output distributions without context points, which can be con- sidered as prior distributions of DKNPs the learned prior of DKNP. After the MCA, the last MLP, a generates the predictive mean vector My for each data point. Intuitively, this can be interpreted as predicting the target mean based on the context and the correlation, which resembles the estimation of the predictive distribution in GPs and ANPs
Different from the NPs. DKNPs explicitly capture the correlation between the output variables using another multihead cross-attention (MCA), where Q = {h Jiet K = (h. heec and V = Hec. Then MLP produces the representation vector w; E R for each position, which generate the covariance matrix Σ = WWT, where W..- w; and Egg = kernel (Z;, z;) w Wy. One might consider using self-attention to let the model learn the correlation between all data points. However, the self-attention module on only X as inputs receives no indication of context and target points and therefore fails to reduce the uncertainty near the points that have high confidence (e.g., context points). Also, the interaction between the target points through self-attente does not guar- antee consistency under the marginal distribution of target points when the context points are given. Lastly, it is also important to note that the representation h is shared when modeling both the mean and the covariance. This motivation drawn from Equation that the calculation of the mean is also based on the kernel matrices, Σ Xx.Xx and Σ>1 X
combined to
To train the DKNPs, the obtained mean vector μe and the covariance matrix E act as parameters of a predictive distribution N py, Σy), ). Instead of maximizing the lower bound of the likelliiddodd as in most NP models, the training objective of DKNPs is to maximize the tractable lig-likeihood of the Gaussian as follows:
log PYtppY///,,...... X Xc, log (YT;
where Ey WWT.
We follow the typical GP formulation where the mean function is set zero. NNote that DKNPs assume C' C ''' during training, which makes Eq. 5 and Eq. technically different. This difference comes from DKNPs using all data points for better learning the kemels, unlike -ernel GPs. DKNPss and GPs, however, are different methods and superior empirical performance led to the current design choice.
4

---

Under review as a conference paper at ICLR 2022
Table 1: Results (log-likelihood) of the NP variants and DKNP in multiple 1D regression tasks
Methods
RBF
Periodic
Matérn 5/2
CNP
0.695 (+0.010)
-0,328 (=0.032)
0.558 (#0.006)
NP
0.577 (=0.015)
-0,619 (=0.005)
0,417 (#0.009)
BNP
0.754 (+-0.004)
-0.018 (#:0.042)
0.617 (1:0.006)
ANP
1.086 (+:0.001)
0.831 (=0.011)
1.020 (+:0.000)
BANP
1.084
(+00001)
0.821 (+:0.018)
1.018 (4:0.001)
Ours
1.109 (++0.001)
0.941 (+00006)
1.039 (+0.004)
RBF
Periodic
Matér
ANN
me
23
Figure 3: Comparison of predictions by ANP and DKNP. The dark blue line indicates the mean, and the light blue shade indicates one standard deviation range.
Although the proposed objective function equivalent to CNP's (Garnelo et al. 2018a), model- ing the correlation between output variables for capturing functional stochasticity shares the same motivation of NPs, thus DKNP being one of NP variants.
EXPERIMENTS
We compare DKNP with diverse NP variants such as Conditional NP (Grnll et al. 201 8a), NP, Bootstrapping NP (BNP)Lee et al. 2020). ANP, and Bootstrapping ANP (BANP) on both and 2D regression tasks. For a fair comparison, all NP variants use two paths for context encoding. CNP, BNP, and BANP have two deterministic paths, and NP and ANP have one deterministic and stochastic path. Also, ANP and BANP have additional self-attention in both stochastic and deter- ministic paths. Following Lee et al. (2020), BNP and BANP were trained with 50 bootstrap context samples and tested with 4 samples. Note that mostly followed the same hyperparameter setup used in Lee et al. al. (2020) and the details are described in Appendix A Unlike the NP variants (except CNP) which are trained with the lower bound of conditional log- likelihood log p( ( Xc, Yc) or its slight modification, DKNP can directly evaluate as well as maximize it during training. All results are also reported with the conditional likelihoooo to evaluate the methods' ability to model the given stochastic processes. Following (Le et al.) 2018 Lee et al., 2020), we use impor- tance weighting estimation (Burda et al. 2016) with samples to evaluate the performance of NP variants that utilize 2.
4.1 ID REGRESSION WITH GAUSSIAN PROCESSES DATA
We first conduct basic evaluation of DKNP in comparison to NP variants by testing its ability to model ID stochastic process samples generated from Gaussian Processes of diverse kernels, namely RBF, Periodic and Matém 5/2. In this task, randomly sampled input from [-2, 2] and gen- crated y from GP kernels where the hyperparameters were also sampled both during training and testing. The number of context points n. is sampled from Uni 25) and the target points is
5

---

Under review as a conference paper at ICLR 2022
Periodic
Matérm
Ground Truth
Ours
Ground Truth
Ours
Ground Truth
Ours
Figure 4: Visualization of the correlation matrices learned from the DKNP compared to the ground truths. We take Monte Carlo estimate (averaging 1,000 kernel functions of the same type) visualize the ground truth kernels for multiple hyperparameters at once.
Perodd--
ANP
31
12
31
18
Outs
-18
35
Figure 5: Comparison of predictions by ANP and DKNP. The dark blue line indicates the mean, and the light blue shade indicates one standard deviation range.
ne + Unif(3,
n.c). Details about the data generation process (kernel hyperparameters), the training process, and the evaluation process are in Appendix C.I
The results Table show that DKNP can better model the given stochastic processes compared to various NP variants in terms of the log-likelihood. Note that the attention mechanism dramatically increases all methods' ability to model GP samples, as can be seen by the clear divide between CNP, NP, BNP and the rest. The true merit of DKNP, however, is its capability to better estimate the uncertainty of the stochastic processes thanks to its explicit modeling of the full covariance matrix.
Figure for example, clearly demonstrates the distinguishing feature of DKNP compared to previ- ous NP variants, in this case ANP, which recorded the second highest likelihood in Table [ In all kernel types, ANP fails to contain the target points within standard deviation range while DKNP successfully captures the uncertainty in all unobserved points. Specifically, ANP is over-confident when modeling kernel types RBF and Matérn 5/2, where only local structure exists. For periodic kernels, where both local and global structure exists, ANP fails to capture the underlying periodic- ity, due its architecture that tries estimate functional stochasticity with fixed-size global latent variable. Unlike ANP, however, DKNP can correctly model the whole stochastic process thanks to its ability explicitly model the correlations between all output variable.
Employing decoupled paths for modeling mean and covariance as in Figure 2 another advantage of DKNP is improved interpretability, as we can explicitly check learned prior by visualizing the covariance matrix as described in Section 3 Such example is depicted in Figure where we can readily compare the learned prior with the ground truth prior for all three kernel types. Note that DKNP not only model allows us visually check the learned prior, but also demonstrate its ability to accurately learn the ground truth kernels for all kernel types.
4.2 1D REGRESSION WITH CHANGE POINT
Next we test methods in a more challenging setup, where the underlying dynamics of the stochas- tie process changes at midpoint. Specifically, employ three different GP configurations: Periodic-

---

Under review as a conference paper at ICLR 2022
Kamel
Gonerated Samples
Kome!
Generatic Samples
Pertode Percok
RDF Perodc
Goussian Process
Dur
Figure 6: Visualization learned correlation matrices and the sampled data for Peridddd and
RBF-Periodic compared to the ground truths.
Table 2: Results (log- likelihood) of the NP variants and DKNP for three different change point configurations.
Methods
Periodic Periodic
Periodic RBF
RBF- RBF
CNP
-0.423 (#0.016)
0.051 (+0.028)
0.966 (+0.006)
NP
-0.531 (10.022)
-0.007 (10.011)
0.872 (10.017)
BNP
-0.358 (1:0.022)
0.105 (100.223)
0.984 (10.001)
ANP
0.602 (100.017)
0.883 (100.055)
1.142 (10.001)
BANP
0.605 (1.0.026)
0.877 (1.0.010)
1.145 (10.001)
Ours
0.892 (1:0.003)
0,955 (1.0.013)
1.260 (100.000)
Periodic, Periodic-RBE, and RBF-RBF. Each configuration uses one kernel type (with a fixed hy- perparamter setup) up to point 0, then another kernel type afterwards. As the correlations between output variables are more complex in this setup, we expect DKNP to demonstrate even more distin- guishing performance than NP variants. The training and evaluation processes are the same as the previous experiments in Section 4.1 See Appendix C. for data generation details.
The results in Table 2 as expected, show wider gaps between DKNP and NP variants for three configurations in terms of log-likelihood. Interestingly, unlike the the previous results (Table TT where DKNP marginally outperformed NP variants RBF kernels (but beyond one standard de- viation margin), the performance gap is more distinguished even for RBF-RBF configuration in this setup. This observation indicates DKNP's s ability to better capture complex correlations derived from combination of rather simpler kernels than NP variants.
Figure 5 demonstrates predictions by ANP and DKNP for all three change point configurations. Compared to ANP which fails to correctly capture the changing dynamics of the given processes (especially for configurations including periodic kernels), DKNP shows its ability to correctly learn the underlying dynamics before and after the change point. In Figure 6 compare the learned priors as well as the generated samples from them to the ground truths priors and samples for two change point configurations, learn two distinct kernels in both sides of the midpoint, (i.e. low and high frequency in the Periodic- Periodic case, smooth and periodic the HBF-PPrriode case) thus able to generate samples that are practically equivalent to the ground truth samples.
Periodic-Periode and RBF-Periodic. Note that DKNP can accurately
4.3 2D IMAGE COMPLETION
The purpose of the 2D image completion task is to test how well each model shows its functional flexibility to learn non-trivial kernels. Assuming that image data is generated by a stochastic process, the task is regress pixels that missing based the provided pixels (i.e. context pixels) in the image. Specifically, we use the CelebA dataset (Liu et al., (015), which consists of 202,599 number of facial images of celebrities of diverse ethnicities, gender, and age.

---

Under review as a conference paper at ICLR 2022
(Single) Context
BANP
DKNP Prior
Figure 7: (Left) The prior leamned by DKNP. (Right) Predicted means of various models given single context point shown in first column.
Sample
Original image
Our
2002
DSNP
Orginal Image
Onggel
Figure 8: Examples of the image completion task. Based on the original image, roughly 10% of pixels (100 pixels) were given the context. The filled images generated from DKNP (Ours) much more diverse and clearer, compared to other baselines.
For both training and evaluation, the number of context points n. is sampled from Unif(3, 200) and the target points is nee + Unif(3, 200 nc)) Details about the data generation process, the training process, and the evaluation process in Appendix C.2
In T consistent with the previous results, DKNP shows superior ability in modeling even two- dimensional stochastic processes compared to all NP variants. This is somewhat predictable based on two previous results (Table and Table 2 where the performance gap between DKNP and NP variants was more visible for stochastic processes with complex underlying dynamics (where both local and global structure exist), and the fact that the facial images are likely to follow non- non-trivial dynamics. For example, the dynamics to generate the hair or the background, which are typically low frequency signals, would be quite different from the dynamics generate the details of the face where eyes, nose, and mouth altogether compose higher frequency signals. And unlike the previous change point data where the boundary between two different dynamics (i.e. kernels) is point in ID line, the boundary between different kernels in facial images is (curved) line in a 2D plane (e.g. hairline). Therefore the correlations between output variables, as shown in Table 3
expect DKNP to significantly outperform all NP variants in estimating
Figure iillustrates the mean of the prior distribution learned by DKNP the CelebA dataset and the predictive mean of the NP variants and DKNP when single context point is given in the back- ground. The posterior mean of DKNP demonstrates distinct behavior for the face and background,
8

---

Under review as a conference paper at ICLR 2022
such that the given context uniformly affects the background while keeping the face practically intact. This shows that DKNP accurately captures the correlations between output variables by suc- cessfully learning kernel functions that behave differently for face and background. BANP, ANP, and NP, on the other hand, demonstrate unpredictable behaviors where the context point in the back- ground also affects the hair or the clothes, or the background is only partially affected by the given context point. This behavior reveals that NPs, which implicitly learn the kernel function, have a harder time modeling the accurate correlation between output variables.
We conclude this section with qualitative examples in Figure 8 that highlight the DKNP's capability to gener- ate diverse samples thanks to its explicit modeling of the covariance. Given 10% of the data points as context, ANP samples are generated by sampling the latent variable 2, BANP samples are generated via bootstrapping, and DKNP samples are generated with the predicted mean and covariance. While ANP, BANP and DKNP all pro- duce reasonable predictive means, ANP's generated sam- ples heavily resemble one another, whereas DKNP's gen- erated samples demonstrate considerable diversity such as different genders, facial expressions and hair styles. BANP also demonstrates more varied samples than ANP thanks to its bootstrapping process, but they sometimes contain noisy pixels.
Table 3: Results log-liiclihood) of NP variants and our model in the 2D image completion task.
Methods
Celeb A
CNP
2.280 (+00010)
NP
2.374 (+0.024)
BNP
2.876 (1:0.014)
ANP
3,471 (100.055
BANP
3.627 (1:0.001)
Ours
3,951 (1:0.011)
RELATED WORK
Many advances in NPs are made with the advent of neural network- based stochastic processes such as NPs (Garnelo et al. 2018b) and CNPs (Garnelo et al.] 2018a). ANP (Kim et al.) 2019) show a dra- matic performance gain by leveraging attention mechanisms in the aggregating operation. Also, Le let al. 2018) confirm through extensive empirical evaluation on the design of NPs such as architec- ture and objective functions that combining attention and NPs improve the predictive log-likelihood marginally. Lee et al.] (2020) extends NPs using the bootstrap technique for estimating functional uncertainty without maintaining a latent variable in the NP architecture. This notion of removing the latent variable part is inline with our motivation.
For modeling stationary stochastic processes, Gordon al. (020) introduced Convolutional CNP (ConvCNP), where translation cquivariance in the data is explicitly encoded in the model. Convo- lutional NPs (ConvNP) Foong al. (2020) are natural extension of ConyCNPs where global latent variable was introduced to model dependencies of the predictive distribution. Gaussian Neu- ral Processes (GNP) (Bruinsma et al. 2021 further generalize these classes of models with transla- tion equivariance by leveraging convolutional neural networks to capture the predictive correlations. While GNPs share a similar motivation to our work, the way covariance matrix is parameterized is quite different (c.g., 1-D, 2-D convolution versus attention) from DKNPs, as their focus is mainly on translation equivariance. Specifically, the covariance matrix of GNPs depends on Yr, while for DKNPs, motivated by posterior of GPs, the covariance only depends on the inputs Xc, Xx (see equation 6
CONCLUSION
We propose new neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), that learn an explicit kernel function to better capture the correlation between output variables. By lever- aging cross- and mixed attention mechanisms to model an explicit kernel function, DKNPs outper- form the concurrent NP variants in terms of predictive likelihood and better global coherence of generated samples. By the novel model architecture of DKNPs, the learned prior can be accessible, which provides deeper understanding of the underlying distributions of data. As future work, one could consider developing a method to manipulate a learned kernel or to impose a constraint on the kernel learning process of DKNPs with prior knowledge.

---

Under review as a conference paper at ICLR 2022
(Reproducibility Statement] We utilized the datasets and the baselines that are all available Github wtt.ghhhw..oouoooeeeeeeeoe As for our proposed model and code, we will make a comment directed to the reviewer and area chairs once the discussion forums open.
REFERENCES
Wessel Bruinsma, James Requeima, Andrew Y. K. Foong, Jonathan Gordon, and Richard E Turner. The gaussian neural process. In Symposium on Advances in Approximate Bayesian Inference (AABI), 2021.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference on Learning Representations (ICLR), 2016.
Andrew Y. Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and
Richard E. Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In International Conference on Machine Learning (ICML), 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXivv:887.001 201 8b.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In International Conference on Learning Representations (ICLR), 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. International Conference on Learning Representations (ICLR), 2019.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation of neural process objectives. In NeurlPS workshop on Bayesian Deep Learning. 2018.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Boot- strapping neural processes. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, 2015.
James Robert Lloyd, David Duvenaud, Roger Grosse, Joshua B. Tenenbaum, and Zoubin Ghahra- mani. Automatic construction and natural-language description of nonparametric regression mod- cls. Association for the Advancement of Artificial Intelligence (AAAI), 2014.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML), 2018.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. MIT Press, 2006.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learning (ICML), 2014.
10

---

Under review as a conference paper at ICLR 2022
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjn Ahn. Sequential neural processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.

---

Under review as a conference paper at ICLR 2022
A ARCHITECTURES DETAILS
We mostly followed the same model architectures the paper (Lee et al. 2020). num det path and nummee fet--path indicate the number of deterministic and stochastic paths.
A.1 ID REGRESSION TASK
CNP
NP
BNP
ANP
BANP
DKNP
dim_x
1
1
dim-y
1
hid
128
128
128
128
128
128
dim_lat
128
128
uum_dct_path
2
I
2
2
qum_stoch____
0
enc.y enc_v_des enc_ydeepth
4
enc Boo..k..eeph depth enc pre depth
2
2
2
4
4
4
4
4
post depth
2
2
dec deephh
3
Table 4: Hyperparameter setting for the ID regression task
A.2 IMAGE COMPLETION TASK
CNP
NP
BNP
ANP
BANP
DKNP
dim.x
2
2
2
dim-y
3
dim_hid
128
128
128
128
128
128
dim lat
128
128
num_de num_det_pane _pth num_stoch_pati enc v depth enc qk depth enc-pre.dephl
2
1
2
2
0
3
3
6
6
6
6
6
6
enc.post_depth
3
3
doc dde_hh
5
Table 5: Hyperparameter setting for the image completion task
B COMPUTATIONAL COMPLEXITY
GP
NP
ANP
Ours
Complexity
O((n + m)
O(n + m) O(n(n + m)) O(n(n + m))
Table 6: The complexity comparison between GP, NP, ANP, and DKNP
DKNP has attention mechanisms like ANP, therefore, it costs O(n(n+m)), which is still computa- tionally much more efficient than GP even with the full covariance matrix.
C EXPERIMENTAL DETAILS
C.I ID REGRESSION
Data generation from a single kernel We constructed training and test data generated from GPs with multiple kernels including RBF, periodic kemels, and Matérn 5/2, For each task, generated
12

---

Under review as a conference paper at ICLR 2022
2,, 2) and y from GP kernels where the GP hyperparameters sampled from uniform
distributions. For RBF kernel, k(z, x²) = 8² exp( (op--lla - we sampled s ~ 1nif(0.1, 1.0), I ~ Unif(0.1,, 0.6), and output additive noise (0, 10 100---2)))) 2). Given n, the size of the context C was drawn from Unif(3, 47) and the size of the target is sampled from Unif(3, 50 - ny). For Matérn 5/2 kernel k(z,, x²) (1 + v5d/l + 5d 5d2/(3/" /3(3)) exp(-v5d/t), (d || x') I'II), we sampled from 1.. and I ~ Unif(0.1,,00)
For periodic kernel, k(x, r'1l /p)/1), we sampled from ~ Unif(O.3 3,, 1.0), I ~ Unif(0.6, 1.0), and p Unif((.8, 1.0). We trained all models identically for 100,000 steps with training batch size of 100. We used Adam optimizer with initial learning rate of 5 10 and decayed using cosine annealing scheme.
s exp(-2sin) (T
Data generation from two kernels Similar single kernel case, data points are sampled from GP but with two types of kernels, RBF and periodic. The half of the points ranging from x = - and 0 were sampled from one kernel and the rest from x We generated samples from three scenarios: Periodic Periodic, RBF Periodic, and RBF RBF. Note that we the correlation between two kernels being zero. The training details are identical to the single kernel case.
2 were from another kernel.
C.2 IMAGE COMPLETION
CelebA32 Similar to 1-D regression tasks, we randomly sampled pixels given image at train- ing as targets, and treat subset of the points contexts. The size of the contexts and targets is sampled from Unif(3, 200) and Unni((0 200 - n..) For preprocessing, & is rescaled to [-1, 1] and y is rescaled [-0.5, 0.5]. We trained all models identically for 200 steps with training batch size of 100. We used Adam optimizer with initial learning rate of 5 10 with cosine annealing scheme.
13


Reviews:
[{"review_id": "cDhjuXBd676", "reviewer": "Reviewer_1GBU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) The motivation of this paper is clear and interesting, improving uncertainty estimation for the highly complex data distribution.\n\n(2) The model architecture looks straightforward. \n\nWeaknesses:\n\n(1) The novelty is limited. In the architecture, the path for generating the mean vector is the deterministic path of ANPs. Learning kernel function by the attention mechanism is a common strategy in many NPs-based papers.  \n\n(2) What does the “decoupled” mean in the paper? The separate mean and variance? Actually, they shouldn’t be decoupled since they share kernel matrices, e.g., K(Xc, Xc) and K(Xc, Xt).\n\n(3) The comparison between the proposed method and baselines is not fair. In the test stage, the proposed model takes all test samples as inputs simultaneously to predict the full covariance matrix. However, other models based on NPs task one batch of test samples as input. Could the author provide the results of the proposed method under the same test setting as other NPs-based methods? \n\n(4) Mean squared error (MSE) is also an important evaluation of regression tasks. Providing MSE in the experiments will make the results more convincing. \n\n(5) In Figure 1 (left), 4 and 6, the ground truth (the blue line for mean and the shade for variance) should be added. It will provide a straightforward comparison between each prediction and the ground truth.\n\n(6) I am confused with this statement: “Note that without any context, …, allowing us to view the prior learned by DKNPs, unlike conventional NPs”. What’s the prior of your model? More details are needed to make the paper self-contained.\n\n", "summary_of_the_paper": "This paper proposes Decoupled Kernel Neural Processes (DKNPs), a new neural stochastic process, which improves uncertainty estimation by learning a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner. It is demonstrated that DKNPs can improve uncertainty estimation in terms of conditional likelihood and diversity in generated samples in 1-D and 2-D regression tasks, compared to other concurrent NP variants.", "summary_of_the_review": "The motivation is interesting, but the model somewhat lacks novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b_bdPrddBs", "reviewer": "Reviewer_Xxs5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary\n\nNeural processes are a recent proposed type of flexible stochastic processes that models distributions of functions. The distribution of functions implicitly captured by a global latent variable Z that are fed into neural networks. Unlike NPs, the function-space view of Gaussian processes model the function distribution directly as a Gaussian using mean and covariance functions. This work proposes to combine the strength of the two by using attention-based neural networks to model the conditional mean and covariance functions, getting rid of the global latent variable Z. \n\n## Strengths\n\n* Fair amount of work seems to have been devoted to the design of the multi-headed attention architectures and they seem to work very well in the toy examples shown (better than architectures used in previous works including NP and Attentive NP).\n\n## Weaknesses\n\n* The main concern that I have after reading the text is the proposed model construction may not lead to a proper stochastic process. Specifically, the Kolmogorov extension theorem requires that all marginal distributions of the stochastic process to be consistent with each other. However, in the proposed model, I can't see how this could not be violated. How do you ensure that the marginal distributions will match for (f(x2), f(x3)) in the cases where X_T = (x1, x2, x3) and X_T = (x2, x3, x4)?\n\n* The description of Neural Processes in the paper is very confusing and gave me the feeling that the nature of the model is mis-interpreted. The whole background section on NPs are focused on the architecture, or what the authors referred as \"deterministic path\" and \"latent path\", without mentioning why these architecture emerges (they are there because we want to do amortized inference). The change of latent variable name $z_T$ to $z_C$ is also problematic as there is only one shared latent variable in the definition of NPs. If you take a look at Garnelo et al. (2018b), there is only $z$. It also seems to me that the evidence lower bound given in (2) is not well-understood that there is an approximation made to the prior so that p(z|context) can be replaced by q(z|context). I strongly encourage the authors to revisit the NP paper and revise the whole submission so that it does not contain misleading information to readers.\n\n* Besides, there seems to be confusion about NPs and conditional NPs. The proposed model are eventually trained in a way similar to conditional NPs (using the conditional distribution), which is different from other NPs trained by the conditional evidence lower bound with approximated priors.\n\n* The figures are generally not very descriptive. It is unclear from Figure 2 how the attention module works (which is the main contribution of the work). In Figure 3, it is confusing to concat $h_c$ and $h_t$ horizontally. Shouldn't it be concatenated in the first axis?\n\n* The empirical results are weak and are missing important baselines. I would expect ConvCNPs and GNPs to work equally well on the 1D regression examples. The proposed method also generates quite a few artifacts in the 2d image experiments. This might be related to the fact that the proposed model may not satisfy the consistency requirements of stochastic processes.\n\n## Overall suggestions\n\nI vote for rejection mainly due to the questionable validity of the proposed model as a stochastic processes, as the parameterization can violate conditions of the Kolmogorov extension theorem.\n\n## Questions and minor points\n\n* In section 5, you mentioned that the Gaussian Neural Processes share similar motivation to your work, but termed their seek of translation equivariance as weaknesses. However, is it also fair to say their construction ensures the model is a valid stochastic process but yours doesn't?\n* You may want to mention $X_C \\subset X_T$ at the beginning of section 2.1.\n* Using \\mathrm to wrap \"softmax\" and \"Concat\" makes it look better.\n* It is better to remove claims like \"we can predict DKNP will better adapt to even more change points composed of multiple kernel types than NP variants. ", "summary_of_the_paper": "Neural processes are a recent proposed type of flexible stochastic processes that models distributions of functions. The distribution of functions implicitly captured by a global latent variable Z that are fed into neural networks. Unlike NPs, the function-space view of Gaussian processes model the function distribution directly as a Gaussian using mean and covariance functions. This work proposes to combine the strength of the two by using attention-based neural networks to model the conditional mean and covariance functions, getting rid of the global latent variable Z. ", "summary_of_the_review": "I vote for rejection mainly due to the questionable validity of the proposed model as a stochastic processes, as the parameterization can violate marginal consistency required by the Kolmogorov extension theorem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AjmlmRMTSR-", "reviewer": "Reviewer_Ngox", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Perhaps it is because of my lack of background on neural processes, but from reading this paper and from a brief scan of the literature, it seems that the major leap forward for neural processes was made by [Kim et al., 2019] with the use of attention. The authors make heavy use of attention in this paper too, and although experiments show that improvements are made upon ANP, I think only time will tell whether this decoupling of mean and covariance functions will prove to be a major advancement in the long term. As I believe is typical with deep learning papers, theoretical results are non-existent, but empirical results are sufficiently extensive for this not to be a drawback. \nHowever, from my very limited judgment, the proposed method is well-motivated, and has intuitively appealing properties, such as those outlined around equations (5) and (6), and I am leaning towards acceptance. \n\nAs an aside, there are some English errors which I think should be corrected. For example, in the abstract, \"we introduce a new neural stochastic processes\" should be \"we introduce a new class of neural stochastic processes\", and just above the start of Section 2, it says \"a neural stochastic processes that explicitly learn...\" which should also be corrected similarly. ", "summary_of_the_paper": "The paper proposes Decoupled Kernel Neural Processes (DKNPs), which is a variant of neural processes first proposed a few years ago. The main contribution is to decouple the process of learning the mean and the covariance functions separately. Empirical evaluations are provided to demonstrate its usefulness. ", "summary_of_the_review": "Despite my lack of knowledge of the relevant literature, I still lean towards acceptance, because I like the idea of decoupling the mean and covariance function and believe this idea may lead to more advancements in the future. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-E2FGUzd6rN", "reviewer": "Reviewer_65De", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI think the paper is well-written with sufficient background introduction for people to follow and understand the difference between DKNP and ANP. The comparison between GP and DKNP is helpful for understanding. I think it is an acceptable paper to the community with a significant contribution.\n\nSome detailed comments:\n\n1. It would be helpful to add some computational analysis. One drawback of GP is its scalability to large-scale datasets. But the GP's benefit is its good uncertainly calibration with small data. Where does DKNP locate in this spectrum, also compared with NPs?\n\n2. One confusing point is in figure 1, the mean is achieved only given the three black context points, right? If not seeing the target points, how does DKNP manage to achieve a periodic mean estimate? There is no corresponding explanation of figure 1 in the main text. \n\n3. Is permutation invariance also guaranteed?\n\n4. In the original ANP paper, they showed examples of the image completion task with a reasonable diversity. How comes the diversity in figure 9 here is so bad for ANP?\n\n\n\n", "summary_of_the_paper": "The authors proposed a novel extension of the neural process by explicitly and separately modeling the mean and the covariance of output variables via multi-head attention. The general NP has the problem that it doesn't well calibrate the output uncertainty when the data sample size is small. However, DKNP seems to mimic the uncertainty calibration behavior similar to GP which is encouraging. The experimental results look convincing with an extensive comparison over state-of-the-art NP variants. \n", "summary_of_the_review": "I think the model is novel and has a significant contribution to the community. The connection between DKNP and GP is interesting. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Decoupled Kernel Neural Processes (DKNPs), a new neural stochastic process, which learns a separate mean and kernel function to directly model the covariance between output variables. Numerical experiments on 1-D regression and 2-D image completion are provided.

There  was a concern that the original version of the proposed model was not a valid stochastic process, since the consistency condition of Kolmogorov Extension Theorem might not be satisfied. The authors fixed this by replacing multihead mixed attention (MMA) in the covariance path with multihead cross-attention.
Overall, reviewers find the work interesting, but there remain concerns that the novelty is limited and that  the current work lacks sufficient experimental evaluation.
Decision: Reject