Title: Distance-informed Neural Processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural processes

Bi-Lipschitz regularization

Uncertainty estimation

uncertainty calibration

Out-of-distribution (OOD) detection

gaussian processes

stochastic processes
Abstract: We propose the Distance-informed Neural Process (DNP), a novel variant of Neural Processes that improves uncertainty estimation by combining global and distance-aware local latent structures. Standard Neural Processes (NPs) often rely on a global latent variable and struggle with uncertainty calibration and capturing local data dependencies. DNP addresses these limitations by introducing a global latent variable to model task-level variations and a local latent variable to capture input similarity within a distance-preserving latent space. This is achieved through bi-Lipschitz regularization, which bounds distortions in input relationships and encourages the preservation of relative distances in the latent space. This modeling approach allows DNP to produce better-calibrated uncertainty estimates and more effectively distinguish in- from out-of-distribution data.  Empirical results demonstrate that DNP achieves strong predictive performance and improved uncertainty calibration across regression and classification tasks.

Distance-informed Neural Processes

Abstract

We propose the Distance-informed Neural Process (DNP), a novel variant of Neural Processes that improves uncertainty estimation by combining global and distanceaware local latent structures. Standard Neural Processes (NPs) often rely on a global latent variable and struggle with uncertainty calibration and capturing local data dependencies. DNP addresses these limitations by introducing a global latent variable to model task-level variations and a local latent variable to capture input similarity within a distance-preserving latent space. This is achieved through bi-Lipschitz regularization, which bounds distortions in input relationships and encourages the preservation of relative distances in the latent space. This modeling approach allows DNP to produce better-calibrated uncertainty estimates and more effectively distinguish in- from out-of-distribution data. Empirical results demonstrate that DNP achieves strong predictive performance and improved uncertainty calibration across regression and classification tasks.

1 Introduction

Deep neural networks have achieved remarkable success in fields such as computer vision [5, 60], natural language processing [57, 42], and reinforcement learning [9]. However, they often produce overconfident and unreliable predictions, particularly when encountering data that fall outside the support of the training set [23, 58, 17, 1, 59]. Stochastic models such as Gaussian Processes (GPs) [66] address this limitation by defining a prior over functions using kernel functions, which encode the assumption that similar inputs produce similar outputs. Kernel functions measure similarity using a distance metric, assigning higher scores to inputs closer to the training data. Consequently, when the model encounters data points that are far from the training set, the predictive distribution reverts to the prior. In many cases, this prior has a high variance, indicating a high uncertainty about the output. However, the practicality of vanilla GPs is limited by their high computational cost, scaling cubically with the size of the training data and by their lack of flexibility in high-dimensional problems.

This paved way to the development of Neural Processes (NPs) [11, 12], that leverage meta-learning principles [20] to learn a distribution over functions from observed data. Unlike traditional kernelbased methods, NPs employ neural encoders to infer a global latent variable that summarizes the underlying function. The NP then conditions its predictions on the inputs and the global latent variable, facilitating rapid adaptation to new functions. However, the reliance on a single global latent variable limits representational expressiveness, and standard NPs often produce uncertainty estimates that are not well calibrated [14, 25, 63].

To address these limitations, we propose the Distance-informed Neural Process (DNP), which integrates both global and local priors for better uncertainty estimation and generalization. The global prior functions like standard NPs, capturing task-specific patterns. The local prior models distance relationships between observed data and test inputs through a latent space, introducing an inductive bias similar to GPs. However, neural networks used to parameterize these latent spaces can warp the input geometry causing similar points to be mapped far apart and unrelated points

(a) Input

(b) Vanilla MLP embeddings

(c) Bi-Lipschitz MLP embeddings

1

0

1

1.0

0.5

0.0

0.5

1.0

2.5

0.0

2.5

5.0

5

0

5

10

0

1

1.0

0.5

0.0

0.5

1.0

to appear close together in the learned representation [36, 55, 69]. To mitigate this, we add a biLipschitz regularization on the local latent network's weight matrices [68]. This constrains each layer's singular values within a fixed range, which minimizes distortions and encourages the latent space to approximately preserve input distances [36]. The distance-preserving latent space is then used to define a local prior that encodes data similarity. As shown in Fig. 1, the latent representations learned by a vanilla neural network distorts input geometry, while bi-Lipschitz regularization helps preserve the input structure by minimizing such distortions 1 . By conditioning on both global and distance-aware local latent variables, DNP yields better-calibrated uncertainty estimates and improved distinction between in- and out-of-distribution (OOD) data. We demonstrate its effectiveness in regression and classification through experiments on synthetic and real-world datasets. Code is available: https://github.com/cvjena/DNP.git .

2 Background

Let D = ( x i , y i ) N i =1 represent a dataset consisting of input-output pairs, where x i ∈ R d x is an input and y i ∈ R d y is the corresponding output. NPs define a family of conditional distributions over functions f ∈ F , mapping x to y . The dataset is split into a context set ( x C , y C ) = ( x i , y i ) i M =1 , where M ≤ N is an arbitrary subset of D , and a target set ( x T , y T ) = ( x i , y i ) N i =1 , which includes all data points. The model uses the context set to create a summary that conditions how it predicts the target outputs y T from the target inputs x T . Different variants of NPs differ in how this context-based conditioning is performed and how the predictive distribution is defined.

The Conditional Neural Process (CNP) [11] models the conditional distribution as p ( y T | x C , y C , x T ) = p ( y T | r C , x T ) , where r C = Agg(( x c , y c ) c ∈ C ) is a permutation-invariant summary (e.g. mean) of the context points. Each ( x c , y c ) is first encoded by a neural network, after which r C and x T parameterize a factorized Gaussian distribution over y T . The model is trained by maximizing the log-likelihood of the target set conditioned on the context set. In addition to the deterministic encoding of the context set, Neural Process (NP) [12] introduces a global latent variable z G ∈ R d z to capture task-level uncertainty. A separate neural network encodes ( x C , y C ) by first aggregating the context set into a global summary s C , which is then used to parameterize a Gaussian distribution over z G . The decoder models the conditional distribution over y T using x T , r C , and z G :

The Attentive Neural Process (AttnNP) [25] extends NP by replacing the shared context representation r C with a target-specific representation r t computed using self-attention over the context and cross-attention with each target input x t . The decoder then conditions on x t , r t and z G to model the target distribution.

GLOBAL LATENT PATH

X1

Y1

X2

Y2

X3

Y3

NN

NN

NN

r

1

r

2

r

3

X1

X2

X3

Cross-

Attention

NN

NN

NN

u1

u2

u3

NN

ut

Yt

m

Sc

~

ZG

Zt

𝜶𝒕

𝒄

Y1

Y2

Y3

ZG

Zt

ut

NN

LOCAL LATENT PATH

DECODER

regularization

Mean

Context set

XC

YC

h(.)

While AttnNP uses attention to capture input-specific patterns, it does not explicitly preserve distance relationships among inputs. Its attention weights, computed over learned embeddings, may distort input relations [69, 55], hindering the capture of local dependencies crucial for similarity-based predictions. In contrast, while DNP also employs cross-attention to obtain target-specific latent variables, its key contribution is the construction of a distance-aware latent space via bi-Lipschitz regularization, from which attention weights are computed.

3 Distance-informed Neural Process

The Distance-informed Neural Process (DNP) is illustrated in Fig. 2. The encoder consists of both global and local latent paths. The global path models a distribution over the global latent variable z G ∈ R d z , conditioned on the context data. To capture local dependencies, the distance-informed local path models a distribution over the target-specific latent variable z t ∈ R d z , conditioned on the target input x t and context data. The decoder then uses both the latent variables for the predictions.

3.1 Global latent path

The global latent variable z G follows the standard formulation in NP and AttnNP, where it serves as a function-level representation that captures global uncertainty. The context set ( x C , y C ) is first encoded into a summary representation s C using mean aggregation over the encoded context pairs. This is further used to define the prior distribution as:

where µ θ G ( · ) and Σ θ G ( · ) are neural networks that parameterize the mean and diagonal covariance of the Gaussian distribution.

3.2 Local latent path

To derive the local latent variable, we encode the input points from both the context and target sets into low-dimensional latent representations using a neural network h . Each context point x c ∈ x C is mapped to an embedding u c = h ( x c ) , and each target input x t ∈ x T is mapped to u t = h ( x t ) ,

where u c , u t ∈ R d u . These representations are then used to model similarity relationships among the inputs via cross-attention.

Accurately capturing these similarity relations require that the latent embeddings preserve the geometric structure of the original input space. However, neural networks can distort this structure during encoding [55, 19], which compromises meaningful distance relationships in the latent space (see Fig. 1). Such distortions can lead to two key issues: (1) over-sensitivity , where small perturbations in the input cause disproportionately large changes in the latent space; and (2) feature collapse , where distinct inputs are mapped to similar embeddings, thereby reducing the discriminability in the learned representations.

Definition 1 (Distortion bounded mapping) . A function h : X → U is said to be a distortion bounded mapping if there exist positive constants L 1 and L 2 such that, for all x 1 , x 2 ∈ X , the following bi-Lipschitz condition holds [45, 36]:

Here, d X ( x 1 , x 2 ) denotes the distance between x 1 and x 2 in the input space X , and d U ( h ( x 1 ) , h ( x 2 )) is the corresponding distance in the output (latent) space U .

The bi-Lipschitz mapping in Eq. 3, implemented by a neural network ensures that the latent space approximately preserves meaningful distances from the input manifold. The upper Lipschitz bound limits the sensitivity of the mapping, preventing small perturbations in the input from causing disproportionately large changes in the latent space. Conversely, the lower bound enforces sensitivity to meaningful variations by ensuring that distinct inputs remain distinguishable after mapping. Together, these constraints yield an approximately isometric transformation, promoting latent representations that remain faithful to the underlying geometric structure of the data.

Some well-known methods for enforcing bi-Lipschitz constraints in Euclidean space include twosided gradient penalties [16, 56], orthogonal normalization [4], spectral normalization in architectures with residual connections [36, 39], and reversible models [21, 2]. While these techniques are used in the context of GANs [39, 16] and deep kernel learning [55], applying bi-Lipschitz regularization to NPs is, to our knowledge, novel. Among the existing approaches, spectral normalization offers the best empirical trade-off between stability and computational cost [55]. For architectures with residual connections, bounding the largest singular values of the weight matrices via spectral normalization [39, 68], combined with Lipschitz-continuous activation functions is sufficient to achieve the bi-Lipschitz property of the network [2]. However, for more general architectures without residual links, we extend this approach by explicitly constraining both the smallest and largest singular values of each weight matrix to directly control the network's contraction and expansion behavior across layers.

Weight Regularization. A neural network typically consists of a series of L layers, where each layer applies a linear transformation followed by a non-linear activation function a as g l ( x ) = a ( W l x + b l ) . To encourage bi-Lipschitz behavior, both the linear transformation and the activation function must together form a bi-Lipschitz mapping. For the linear part, this is achieved by bounding the maximum and minimum singular values of the weight matrices W l . To ensure overall bi-Lipschitz continuity, this bounding is combined with bi-Lipschitz activation functions such as Leaky ReLU [38], Softplus, ELU [6], PReLU [18], etc which preserve both upper and lower Lipschitz bounds. Note that residual architectures using spectral normalization [2] have similar requirements, as they rely on Lipschitzcontinuous activations to ensure the bi-Lipschitz property. Let σ l min and σ l max denote the smallest and largest singular values of W l respectively. These values control how much the input can be contracted or stretched by the linear transformation. Accordingly, we define the bi-Lipschitz regularization loss:

where 0 < λ 1 ≤ λ 2 are hyperparameters that control the lower and upper bounds of σ l min and σ l max , respectively. These hyperparameters provide flexibility in managing the effective Lipschitz constants when combined with regularization techniques (e.g., Dropout, Batch Normalization). The overall Lipschitz constants of the function h = g L ◦ g L -1 ◦ · · · ◦ g 1 are governed by the product of the per-layer singular value bounds.

Note that computing the exact singular value decomposition (SVD) of each weight matrix can be computationally expensive, particularly for deep networks. For a matrix W ∈ R p × q , the computa-

tional cost of SVD is O (min( pq 2 , p 2 q )) , which can be prohibitive for large p and q . To obtain a fast approximation of σ l max and σ l min , we apply the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method [29, 51], an iterative algorithm originally designed to compute a few extremal eigenvalues and corresponding eigenvectors of large symmetric matrices. In our case, LOBPCG is used to approximate the largest and smallest eigenvalues of the symmetric positive semi-definite matrix W l W ⊤ l or W ⊤ l W l , depending on which of the two is smaller in dimension. For T iterations, the total computational cost per layer is O ( Tpq ) , where T ≈ 5 -10 . The approximate values of σ l max and σ l min are then obtained as the square roots of these eigenvalues respectively.

Similarity relations. Once the latent space embeddings are obtained, we use cross-attention [57] to model the relationships between the context and target points. The latent embeddings of the context data u c serve as the keys, while the embedding of a target point u t acts as the query. The cross-attention weights are computed using Laplace attention as follows:

The attention mechanism assigns a relevance score to each context point based on its similarity to the target point. Since the embeddings are constructed to preserve the geometric structure of the input space, the resulting attention weights more accurately reflect the true proximity between points.

Local latent variables. The target-specific prior distribution for the local latent variable z t is a Gaussian and follows a similar construction to [37]:

µ θ L ( · , · ) and Σ θ L ( · , · ) are neural networks used to parameterize the Gaussian distribution. The attention weight α c t control how much each context point contributes to the local latent distribution. When a target point lies in an OOD region and far from the context data, the attention weights tend toward zero. As a result, the local prior distribution over z t approaches a standard normal, functioning as a non-informative prior.

3.3 Generative Model

Once the global and local latent variables are defined, we construct the decoder (likelihood model) that generates outputs y 1: N from inputs x 1: N . This decoder network is conditioned on both the global latent z G and the local latent z 1: N corresponding to x 1: N , and is given by the conditional distribution p θ ( y 1: N | z G , z 1: N , x 1: N ) . The complete generative process is given by:

Note that we choose factorized Gaussian priors for z G and each z i for simplicity. This choice does not limit the generality of our approach: any permutation-invariant distribution over the global and local latents could be substituted in its place.

Proposition 1. Eq. 7 defines an exchangeable stochastic process by satisfying both exchangeability and marginal consistency, as required by the Kolmogorov Extension Theorem [41].

The proof is provided in Appendix C. Intuitively, exchangeability holds because the likelihood and local priors factorize across data points, so permuting the input-output pairs does not change the joint distribution. Marginal consistency holds because integrating out a subset of outputs and their corresponding latents simply removes those terms from the product, yielding a valid joint distribution over the remaining variables.

3.4 Training and Inference

Having defined the generative model, we aim to fit the model parameters to infer the latent variables, and to make predictions for new inputs x ∗ . However, exact inference using Eq. 7 is intractable.

To address this, we employ variational inference, which involves maximizing the Evidence Lower Bound (ELBO) on the log marginal likelihood [28]. We introduce the variational distributions q ϕ G ( z G | x T , y T ) and q ϕ L ( z t | x t , y t ) to approximate the true posterior distributions over the global and local latent variables, respectively. The global variational posterior is modeled as a Gaussian:

where s T is the mean-aggregated summary of the target set, obtained in the same manner as s C . The local variational posterior q ϕ L is also modeled as a Gaussian with diagonal covariance, parameterized by neural networks:

The resulting ELBO, derived in Appendix D is:

The final training objective of DNP is the sum of the ELBO loss and the bi-Lipschitz regularization loss:

where β is a hyperparameter that control the trade-off between data likelihood and weight regularization. For inference on unseen data points x ∗ , we use the posterior predictive distribution:

Note that at test time, the learned prior networks p θ G and p θ L are used to generate the latent variables and for the prediction. The algorithms and computational complexity for training and inference is provided in Appendix. A.1 and A.2.

4 Related Works

Stochastic Neural Networks (SNNs) have been widely explored for capturing uncertainty in neural networks. Bayesian Neural Networks (BNNs) [3] introduce a prior distribution over weights and use Bayesian inference to estimate posterior distributions, and estimate uncertainty in deep learning models. However, BNNs are computationally expensive, leading to the development of approximation techniques such as variational inference [15, 47], Monte Carlo dropout [10] and deep ensembles [32]. While BNNs approximate posterior distributions over weights, an alternative Bayesian approach is provided by Gaussian Processes (GPs)[66], which define distributions over functions directly. Unlike BNNs, GPs offer a non-parametric framework for capturing uncertainty, making them particularly appealing for small-data regimes. However, the cubic computational complexity of standard GPs limits their scalability, and they are less flexible in high-dimensional data, motivating various approximations such as sparse GPs [54, 49] and deep kernel learning [67].

Neural Processes (NPs) [11, 12, 25] have emerged as a powerful class of models that combine the flexibility of deep learning with the uncertainty quantification of Bayesian inference. In addition to the NPs discussed in Sec. 2, several other variants have been introduced to model different inductive biases. Bootstrapping Neural Process (BNP) [34] improves uncertainty estimation in NPs by incorporating a bootstrapping procedure to iteratively improve latent variable estimates. Convolutional CNP (ConvCNP) [14] and ConvNP [8] introduce translation equivariance by leveraging convolutional neural networks, which improves generalization in tasks involving spatial dependencies such as timeseries. Functional NP [37] leverages graph structures to model data dependencies, thereby obtaining more localized latent representations. In contrast, DSVNP [63] introduces a hierarchical structure of global and local latent variables to capture uncertainty at multiple levels of granularity. Transformer Neural Processes [40] replace the standard NP encoder-decoder with a transformer, improving expressiveness while ensuring context invariance and target equivariance. [64] extend NPs with

multiple expert latent variables to better capture multimodal distributions. [52] reformulate context aggregation as probabilistic graphical inference, yielding mixture and robust Bayesian aggregation variants that improve OOD robustness. [62] address NP underfitting via a surrogate objective in an expectation-maximization framework, leading to more accurate distribution learning. [13] propose a decoupled encoder trained with a contrastive objective, enhancing transferability and robustness across tasks. While these works focus on expressiveness and inference, DNP takes a complementary path by enforcing geometry-aware regularization through a bi-Lipschitz-constrained latent path. This preserves input structure and improves uncertainty estimation and OOD generalization. Following [63], a summary of selected NP variants is provided in Table 1.

5 Experiments

We evaluate DNP on both regression and classification tasks. For regression, we consider 1D synthetic examples, as well as multi-dimensional tasks on several real-world datasets. Performance is assessed in terms of predictive quality and uncertainty calibration [31]. For classification, we assess the model on CIFAR-10 and CIFAR-100 [30], evaluating classification accuracy, calibration performance, and OOD detection capabilities. The regression networks use a 3-layer MLP, while a VGG-16 architecture [30] is used for feature extraction in classification. The data generation process for the experiments, model architectures used and the hyperparameters are provided in Appendix A.

5.1 1D Synthetic Regression

In this experiment, we evaluate the methods on 1D functions generated from GP priors with three different kernels: (i) the Radial Basis Function (RBF) kernel, (ii) the Matérn-5/2 kernel, and (iii) the Periodic kernel. At each training iteration, kernel hyperparameters are randomly sampled to introduce functional diversity. During testing, the methods are evaluated on 2000 function realizations, using the log-likelihood (LL) and Expected Calibration Error (ECE) [31] of the target points.

Table 2 shows the results for evaluation on the target set, excluding the context. Across all three kernel types, DNP achieves a superior log-likelihood and a lower ECE scores in majority of the cases compared to the other baselines. Evaluation on the context set, inference latency and noisy data is provided in Appendix B.1. Figure 3 shows predictions from AttnNP and DNP on GPsampled functions, evaluated both in-distribution ( [ -2 , 2] ) and in OOD regions. While AttnNP often underestimates uncertainty, DNP is generally better calibrated. DNP can occasionally be overconfident due to the approximate nature of the bi-Lipschitz constraint. However, DNP achieves more reliable predictions, accurately modeling the target function in-distribution while expressing higher uncertainty in OOD regions.

5.2 Synthetic to Real-World Regression

Following [14], we consider the Lotka-Volterra model [65], which captures predator-prey population dynamics. Models are trained on simulated data generated from the Lotka-Volterra equations and

Method

RBF

5

2

Periodic

LL (

↑

)

ECE (

↓

)

LL (

↑

)

ECE (

↓

)

LL (

↑

)

ECE (

↓

)

CNP [11]

0.490

±

0.006

0.122

±

0.061

0.419

±

0.004

0.088

±

0.047

0.239

±

0.001

0.120

±

0.029

NP [12]

0.480

±

0.009

0.102

±

0.082

0.318

±

0.008

0.111

±

0.053

0.228

±

0.002

0.145

±

0.047

ConvCNP [14]

0.496

±

0.007

0.118

±

0.054

0.329

±

0.001

0.098

±

0.032

0.335

±

0.002

0.106

±

0.032

ConvNP [8]

0.609

±

0.005

0.190

±

0.074

0.313

±

0.002

0.137

±

0.095

0.431

±

0.003

0.117

±

0.011

AttnNP [25]

0.578

±

0.003

0.103

±

0.026

0.499

±

0.009

0.078

±

0.014

0.457

±

0.001

0.098

±

0.038

DSVNP [63]

0.582

±

0.002

0.101

±

0.032

0.512

±

0.006

0.101

±

0.012

0.412

±

0.002

0.092

±

0.042

NP-mBA [52]

0.521

±

0.005

0.108

±

0.012

0.324

±

0.007

0.091

±

0.021

0.375

±

0.005

0.091

±

0.010

Ours

0.696

±

0.003

0.093

±

0.054

0.620

±

0.002

0.083

±

0.010

0.635

±

0.001

0.074

±

0.022

(a)

(b)

(c)

(d)

AttnNP

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

Ours

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

4

2

0

2

4

3

2

1

0

1

2

3

evaluated on real-world data from Hudson's Bay hare-lynx data [35], presenting a distribution mismatch scenario. For each training batch, the number of context points is sampled uniformly from the interval [3 , 50] , while the number of target points is fixed at 50. During testing, 30 context points are randomly selected from the training set. In addition to NPs, we also include an exact GP with RBF kernel as a baseline. Table 3 reports log-likelihood scores on both simulated and real data. Results indicate that DNP achieves a better performance over the other baselines. Interestingly, the exact GP performs worse than the NP variants, which can be attributed to the limited context points. In contrast, NP-based methods are able to more efficiently incorporate context information, resulting in higher-quality predictions.

5.3 Multi-output Real-World Regression

To assess the performance of DNP on more complex and practical scenarios, we follow [63] and evaluate the models on the SARCOS [61], Water Quality (WQ) [7] and the SCM20D [50] datasets. The SARCOS dataset contains 48,933 samples, each with 21 input features and 7 output variables. The Water Quality (WQ) dataset includes 1,060 samples with 16 input features and 14 output variables. The SCM20D dataset comprises 8,966 samples, with 61 inputs and 16 outputs. Performance is measured using Mean Squared Error (MSE) and ECE scores. The results are provided in Table 4, which shows that DNP achieves superior predictive performance in most cases and provides consistently improved uncertainty calibration across all three datasets.

5.4 Image Classification

We evaluate the performance of DNP on image classification using the CIFAR-10 and CIFAR-100 datasets. The models are assessed in terms of classification accuracy, ECE, and the predictive entropy. Additionally, we evaluate OOD detection performance using the Area Under the Precision-Recall curve (AUPR) between the ID and the OOD probabilities. For classification using CIFAR10, we use the SVHN [46], CIFAR100 and the TinyImageNet [33] as the OOD datasets. For CIFAR100, the OOD datasets are SVHN, CIFAR10 and TinyImageNet. The VGG16 model [48] is used as the feature extractor for all the methods. In addition to the NP-based baselines, we also provide the results for a VGG-16 classifier trained using cross-entropy loss, referred to as "Deterministic".

Table 5 shows the results for classification uisng CIFAR10. From the results, DNP achieves the least ECE score, and closely follows Deterministic in terms of accuracy. DNP incurs higher inference latency per instance compared to CNP and vanilla NP. However, when compared to AttnNP and DSVNP, DNP achieves a faster inference by avoiding self-attention over the context set, instead relying on a bi-Lipschitz regularized latent space to preserve the relational structure. Interestingly, the in-distribution entropy of DNP is slightly higher than that of the vanilla NP. This could be attributed to DNP's more expressive uncertainty modeling, where the combination of global and local latent variables introduces additional variance to capture the predictive uncertainty. Consequently, DNP also shows increased entropy on the OOD datasets, allowing it to more effectively distinguish ID from OODsamples. This leads to improved performance in OOD detection, as reflected in its higher AUPR

scores compared to other methods. A similar trend is observed in the classification performance on CIFAR-100, as shown in Table 6. DNP achieves a lower ECE, while also outperforming all baselines in OOD detection.

To evaluate the contribution of different regularization methods aimed at controlling the Lipschitz constant of the network, we compare DNP variants using the 2-sided gradient penalty [16], orthogonal regularization [4], and spectral regularization [68] for CIFAR10 (ID) vs CIFAR100 (OOD) classification. The results in Table 7 show that while all regularization methods improve ECE and AUPR scores compared to the no-

regularization baseline, bi-Lipschitz weight regularization consistently outperforms the others. Notably, both spectral and bi-Lipschitz regularization yield a substantial reduction in ECE, with the latter achieving the lowest ECE and highest AUPR scores. In contrast, methods like orthogonal regularization are more restrictive, as they impose stronger constraints on the model's weight space, which may limit representational flexibility. Additional ablations on the influence of the number of context points, dimensions of d u and d z , λ 1 , λ 2 values for the bi-Lipschitz regularization, trade off parameter β and comparison with dot-product attention are provided in Appendix B.2.

6 Limitations and Future Work

The current bi-Lipschitz constraint is applied in a Euclidean space, which may not fully capture complex semantic relationships. Extending this constraint to incorporate more flexible distance measures or task-specific similarity metrics [69] could improve generalization across diverse domains. Moreover, the use of cross-attention introduces additional computational overhead, which may limit scalability to very large datasets. Exploring sparse attention mechanisms [22, 53] offers a promising path to reduce this cost while maintaining expressiveness.

7 Discussion and Conclusion

We introduced the Distance-informed Neural Process (DNP), a novel variant within the Neural Process family that integrates both global and local latent representations. DNP leverages a distanceaware local latent path, enforced via bi-Lipschitz regularization to approximately preserve distance relationships in the input space. This leads to a more faithful modeling of similarity relationships and improves both predictive accuracy and uncertainty calibration across regression and classification tasks. The broader implications of this work lie in its potential applications in safety-critical domains, where calibrated uncertainty estimates and generalization to unseen conditions are essential. Like most neural models, DNP can be sensitive to the quality and bias of its training data and dataset shifts. We encourage practitioners to validate the model carefully to ensure safe and reliable deployment.

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion , 76:243-297, 2021.

Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International conference on machine learning , pages 573-582. PMLR, 2019.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning , pages 1613-1622. PMLR, 2015.

Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093 , 2016.

Junyi Chai, Hao Zeng, Anming Li, and Eric WT Ngai. Deep learning in computer vision: A critical review of emerging techniques and application scenarios. Machine Learning with Applications , 6:100134, 2021.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Sašo Džeroski, Damjan Demšar, and Jasna Grbovi´ c. Predicting chemical parameters of river water quality from bioindicator data. Applied Intelligence , 13:7-17, 2000.

Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems , 33:8284-8295, 2020.

Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, et al. An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning , 11(3-4):219-354, 2018.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning , pages 1704-1713. PMLR, 2018.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

Muhammad Waleed Gondal, Shruti Joshi, Nasim Rahaman, Stefan Bauer, Manuel Wuthrich, and Bernhard Schölkopf. Function contrastive learning of transferable meta-representations. In International Conference on Machine Learning , pages 3755-3765. PMLR, 2021.

Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.13556 , 2019.

Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems , 24, 2011.

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems , 30, 2017.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv:1908.02729 , 2019.

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(9):5149-5169, 2021.

Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv:1802.07088 , 2018.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7482-7491, 2018.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05761 , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning , 12(4):307-392, 2019.

Andrew V Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM journal on scientific computing , 23(2):517541, 2001.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning , pages 2796-2804. PMLR, 2018.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. Advances in neural information processing systems , 33:66066615, 2020.

E Leigh. The ecological role of volterra's equations. Some mathematical problems in biology. , pages 1-61, 1968.

Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems , 33:7498-7512, 2020.

Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. Advances in Neural Information Processing Systems , 32, 2019.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , volume 30, page 3. Atlanta, GA, 2013.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018.

Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179 , 2022.

Bernt Øksendal and Bernt Øksendal. Stochastic differential equations . Springer, 2003.

Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems , 32(2):604-624, 2020.

A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

H A Priestley. Fubini's theorem and tonelli's theorem. In Introduction to Integration . Oxford University Press, 08 1997.

Mícheál Ó Searcóid. Metric spaces . Springer, 2007.

Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference on pattern recognition (ICPR2012) , pages 3288-3291. IEEE, 2012.

Kumar Shridhar, Felix Laumann, and Marcus Liwicki. A comprehensive guide to bayesian convolutional neural network with variational inference. arXiv preprint arXiv:1901.02731 , 2019.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems , 18, 2005.

Eleftherios Spyromitros-Xioufis, Grigorios Tsoumakas, William Groves, and Ioannis Vlahavas. Multi-target regression via input space expansion: treating targets as inputs. Machine Learning , 104:55-98, 2016.

Andreas Stathopoulos and Kesheng Wu. A block orthogonalization procedure with constant synchronization requirements. SIAM Journal on Scientific Computing , 23(6):2165-2182, 2002.

Dharmesh Tailor, Mohammad Emtiyaz Khan, and Eric Nalisnick. Exploiting inferential structure in neural processes. In Uncertainty in Artificial Intelligence , pages 2089-2098. PMLR, 2023.

Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International conference on machine learning , pages 9438-9447. PMLR, 2020.

Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics , pages 567-574. PMLR, 2009.

Joost Van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409 , 2021.

Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning , pages 9690-9700. PMLR, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Aishwarya Venkataramanan, Assia Benbihi, Martin Laviale, and Cédric Pradalier. Gaussian latent representations for uncertainty estimation using mahalanobis distance in deep classifiers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 44884497, 2023.

Aishwarya Venkataramanan, Paul Bodesheim, and Joachim Denzler. Probabilistic embeddings for frozen vision-language models: Uncertainty quantification with gaussian process latent variable models. arXiv preprint arXiv:2505.05163 , 2025.

Aishwarya Venkataramanan, Martin Laviale, and Cédric Pradalier. Integrating visual and semantic similarity using hierarchies for image retrieval. In International Conference on Computer Vision Systems , pages 422-431. Springer, 2023.

Sethu Vijayakumar and Stefan Schaal. Locally weighted projection regression: An o (n) algorithm for incremental real time learning in high dimensional space. In Proceedings of the seventeenth international conference on machine learning (ICML 2000) , volume 1, pages 288-293. Morgan Kaufmann Burlington, MA, 2000.

Qi Wang, Marco Federici, and Herke van Hoof. Bridge the inference gaps of neural processes via expectation maximization. arXiv preprint arXiv:2501.03264 , 2025.

Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning , pages 10018-10028. PMLR, 2020.

Qi Wang and Herke Van Hoof. Learning expressive meta-representations with mixture of expert neural processes. Advances in neural information processing systems , 35:26242-26255, 2022.

Darren J Wilkinson. Stochastic modelling for systems biology . Chapman and Hall/CRC, 2018.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics , pages 370-378. PMLR, 2016.

Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941 , 2017.

Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. Advances in Neural Information Processing Systems , 36:17882-17903, 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose integrating a bi-Lipschitz regularization to learn a distance-aware latent space that captures input similarity relations, aiming to improve uncertainty calibration and out-of-distribution (OOD) detection in Neural Processes.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and potential future extensions of our method in Section 6. A limitation is that the proposed bi-Lipschitz regularization is defined in Euclidean space and only approximates distance preservation. This may be suboptimal for tasks involving non-Euclidean input spaces, such as graph-structured data or data with underlying geodesic geometry. Additionally, the use of a cross-attention module introduces computational overhead.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state the assumptions i.e, conditional independence of targets given latents, permutation-invariant priors, immediately before Proposition 1. A formal proof for the proposition is provided in Appendix C. A short proof sketch is provided in the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The detailed implementation, including the dataset generation, model architecture, algorithm and hyperparameters is provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be made open-source. The code for synthetic data generation is included. The real-world datasets for the regression and classification experiments are already available publicly.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings and details are provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include the mean and standard deviations of all the experimental results. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources are mentioned in Appendix A. The inference latency is also provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The experiments are conducted in conformity with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts of the work is discussed in Section 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models with a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original papers of the used assets are cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Details about the model, training, limitations are discussed in Section 7 and Appendix A. The code is included in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Implementation Details

A.1 Algorithm

The training and inference procedures for DNP is outlined in Algorithms 1 and 2 respectively. The model combines a global latent variable that captures task-level patterns and a distance-aware local latent variable that models input similarity. During training, both the global and local paths are optimized jointly using a variational objective and a bi-Lipschitz regularization term. At test time, the model predicts target outputs by conditioning on both the global and local priors inferred from the context set.

Algorithm 1 Training DNP

A.2 Computational Complexity

CNPs and NPs aggregate the context set using a permutation-invariant operation (e.g., mean pooling), resulting in an O ( M ) operation. Each of the N target points is then processed independently, yielding a total prediction-time complexity of O ( N ) . AttnNPs and DSVNP uses self-attention over the M context points, which incurs a cost of O ( M 2 ) . Additionally, they compute cross-attention between each of the N targets and the M context points, resulting in a further O ( NM ) cost. Thus, the overall complexity during prediction becomes O ( M 2 + NM ) . Our proposed DNP eliminates the need for self-attention over the context set by constructing a distance-preserving latent space using a bi-Lipschitz-regularized encoder. At prediction time, each target point performs cross-attention over the context representations, resulting in a total complexity of O ( NM ) . During training, there is an added cost from the LOBPCG computation. The cost per layer is O ( Tpq ) where T is the number of iterations, p and q are the dimensions of the weight matrix.

A.3 Regression

A.3.1 Network Architecture

The overall architecture of regression using DNP is as follows.

Algorithm 2 Inference with DNP

Global latent encoder. The global latent encoder is a 3-layer MLP, constructed similarly to the structure in NP and AttnNP. The input to the MLP is the concatenation of the input variable x ∈ R d x and output variable y ∈ R d y .

Each context and target pair is first encoded independently, and then the resulting representations are aggregated via the mean function to produce a global summary.

The MLP architecture is as follows:

where µ l , Σ l ∈ R d z .

Next, to incorporate context information into each target, we apply a cross-attention between the target's embedding u t and all context embeddings u c to obtain the attention weights α c t . These attention weights then aggregate the per-context Gaussian parameters into a target-specific local prior.

Decoder. The decoder takes as input the concatenation of the sampled global latent z g ∈ R d z , the sampled local latent z l ∈ R d z , and the distortion-bounded embedding u ∈ R d u . This produces a vector of dimension 2 d z + d u , which is passed through a 3-layer MLP to output the parameters of the predictive Gaussian over y ∈ R d y :

Other baselines. The network architecture of CNP consists of a 3-layer MLP with hidden dimension d h and a latent representation of dimension d z . Each context pair ( x , y ) is passed through the encoder MLP, and the resulting representations are mean aggregated to form a global representation. The decoder then takes the concatenation of this global representation and the target input x t and passes it through another 3-layer MLP to predict the mean and log-variance of the output y t . Each layer in the MLPs except the output layer is followed by a Leaky ReLU activation.

For NP, we use the architecture proposed in [25]. The encoder is composed of two parallel paths: a deterministic path and a latent path. Both paths utilize a 3-layer MLP with hidden layers of dimension

Note that unless specified, each → in this text represents a linear layer followed by a Leaky ReLU activation, except for the final layer. The output of the final layer provides an embedding of each input pair ( x i , y i ) . These embeddings are then aggregated across the entire context set using a mean operation, resulting in a single summary vector. This summary vector is subsequently passed through a separate linear layer to produce the mean and log-variance of a Gaussian distribution over the global latent variable z g .

Local latent encoder. To obtain a distortion-bounded latent representation, we first construct a shared 2-layer MLP that maps each input x into an intermediate feature space of dimension d h . The MLP follows the structure:

The shared backbone is used to extract two components: (i) the distortion-bounded representation u , and (ii) the parameters of the amortized posterior distribution. The distortion-bounded latent representation u ∈ R d u is obtained by applying a linear projection to the backbone output:

To obtain the approximate posterior and prior distributions over the local latent variable, the hidden representation from the backbone is concatenated with the corresponding target output y and passed through an additional linear layer. This layer outputs the mean and log-variance of a Gaussian distribution:

d h and an output dimension of d z . Each input-output pair ( x , y ) is encoded independently, and the resulting representations are aggregated using a mean operation. In the latent path, the aggregated representation is used to produce the parameters of a multivariate Gaussian distribution, namely the mean µ g and log-variance log Σ g of the global latent variable. The decoder receives the sampled latent variable, the aggregated deterministic representation, and the target input x t as input, and outputs the prediction y t through a separate 3-layer MLP to predict the mean and log-variance of the output y t .

For AttnNP and DSVNP, we follow a similar architecture as NP, with the key difference that the mean aggregation in the encoder is replaced by multi-head self-attention and the deterministic path uses a multi-head cross-attention network, as proposed in [25]. The main distinction between AttnNP and DSVNP is that DSVNP models both global and local latent variables by predicting Gaussian distribution parameters for each, where the local latent is conditioned on the global latent. All other components, including the decoder, remain identical to those used in NP.

For both ConvCNP and ConvNP, we follow the architecture proposed in [14, 8], using 1D convolutional layers. The encoder consists of 3 convolutional layers, each with 64 channels and followed by Leaky ReLU activations. The input data is first interpolated onto a uniform grid using a Gaussian kernel before being passed through the encoder. The decoder also consists of 3 convolutional layers, each with 64 channels, and outputs the mean and log-variance of the predictive distribution for each target point.

The models were implemented using PyTorch [43] and trained on an Nvidia GeForce GTX 1080 with 12 GB of RAM. To obtain the singular values for DNP, we use PyTorch's implementation of the LOBPCG method, which employs orthogonal basis selection [51].

A.3.2 1D Regression

Inputs for both context and target points are drawn uniformly from the interval [ -2 , 2] . The number of context points is sampled uniformly from [3 , 50] , while the number of target points is fixed at 50. For all kernels, the lengthscale and the output scale values are sampled uniformly from [0 . 6 , 1 . 0) and [0 . 1 , 1 . 0) , respectively. Additionally, for the Periodic kernel, the period parameter is sampled uniformly from the interval [0 . 5 , 1 . 5) . A likelihood noise of 0 . 02 is added for all the cases. A batch size of 50 is used for training. During evaluation, the number of context and target points is sampled from the same distributions used during training. The models used the following hyperparameters: the encoder/decoder hidden dimension was set to d h = 64 , the latent dimension to d z = 64 , and the distance-aware latent dimension to d u = 64 . All the models are trained using the Adam optimizer [26] at a learning rate of 1 e -3 , for 200 epochs. For DNP, the bi-Lipschitz constraint is enforced with lower and upper singular value bounds λ 1 = 0 . 1 , λ 2 = 1 . 0 , and the balancing the ELBO and bi-Lipschitz regularization is set to β = 1 .

A.3.3 Synthetic to Real-World Regression

The data generation process follows [14]. For each training batch, the number of context points is sampled uniformly from the interval [3 , 50] , while the number of target points is fixed at 50. At test time, 30 context points are randomly selected from the train data. The following hyperparameters were used for the models: encoder/decoder hidden dimension d h = 64 , latent dimension d z = 64 , and distance-aware latent dimension d u = 64 . All networks were trained with the Adam optimizer [26] at a learning rate of 1 e -3 , using a batch size of 50 for 200 epochs. For DNP, the lower and upper bounds on the singular values were set to λ 1 = 0 . 1 , λ 2 = 1 . 0 , with a trade-off weight β = 1 balancing the ELBO and bi-Lipschitz regularization.

A.3.4 Multi-output Real-World Regression

We evaluate DNP on three benchmark datasets: SARCOS [61], which models the inverse dynamics of a seven-degree-of-freedom robot arm by predicting seven joint torques from 21 inputs (positions, velocities, accelerations); Water Quality (WQ) [7], which predicts species abundances in Slovenian rivers from 16 physical and chemical indicators; and SCM20D [50], a supply-chain time series for multi-item demand forecasting. All datasets are standardized to zero mean and unit variance per feature and split 80/20 into training and test sets.

For every training batch, the number of context points is randomly sampled from a uniform distribution in the range [3 , 100] , and the number of target points is 100. The network architecture uses hidden and latent dimensions d h = 128 , d z = 128 , and the distance-aware latent dimension d u = 128 . Models are trained using Adam optimizer [26] with a learning rate of 1 e -3 , and a batch size of 100, for 100 epochs. At test time, we randomly select 30 context points from the train data. For the bi-Lipschitz regularizer, we use λ 1 = 0 . 1 , λ 2 = 1 . 0 and set the ELBO/bi-Lipschitz trade-off weight to β = 1 .

A.4 Image Classification and OOD Detection

For classification tasks on CIFAR-10 and CIFAR-100, we use a VGG-16 network [48] with Leaky ReLU activation as the feature extractor.

Global latent encoder. In the global latent encoder, the feature embedding obtained from VGG-16 is concatenated with the one-hot encoded label y to form the input representation. The flow of operations is summarized as:

Local latent encoder. The local latent encoder also uses the feature embedding extracted by VGG-16. Each input x is passed through VGG-16 to obtain an embedding of size d h :

This backbone is used to produce two outputs: (1) A distortion-bounded latent embedding u ∈ R d u , obtained via a linear projection:

(2) The parameters (mean and variance) of the amortized posterior distribution over the local latent variable:

Cross-attention is then applied between the distortion-bounded target embedding u t and the set of context embeddings u c to obtain an attention-based aggregation of local latent parameters.

Decoder. The decoder takes as input the concatenation of the sampled global latent variable z g ∈ R d z , the sampled local latent variable z t ∈ R d z , and the distortion-bounded latent embedding u ∈ R d u . The resulting vector of dimension 2 d z + d u is passed through a 3-layer MLP to predict the parameters of a categorical distribution over the output classes.

The other baselines follow a similar architecture to that described in Section A.3.1, with the key difference that a VGG-16 network is used as a feature extractor for the input images. The extracted feature embedding is concatenated with the one-hot encoded label vector before being passed through the subsequent layers. Similar to DNP, the predictive distribution is modeled as a categorical distribution over the output classes.

For the experiments, the networks are trained on the training sets of CIFAR-10 and CIFAR-100, each consisting of 60,000 images categorized into 10 and 100 classes, respectively. Evaluation is performed on the corresponding test sets, each containing 10,000 images. The images have a dimension of 32 × 32 , and are normalized to zeros mean and unit variance. For OOD detection, images from the test set of the SVHN, CIFAR100/CIFAR10 and the TinyImageNet dataset is used. For a fair comparison, all the OOD images are resized to 32 × 32 and are normalized the same way as the CIFAR10/CIFAR100 images. For training, the number of context points is randomly chosen between from a uniform distribution between 16 and 128. A batch size of 128 is used. The network architecture has the dimesnions d h , d z = 512 and d u = 256 . The models were implemented using PyTorch [43] and trained on an NVIDIA A100 GPU with 40 GB of memory. Models are trained using the Adam optimizer [26] with a learning rate of 1 e -4 for 200 epochs. At test time, 100 context points are randomly selected from the train set. For the bi-Lipschitz regularizer, λ 1 = 0 . 1 and

λ 2 = 1 . 0 , with the ELBO/bi-Lipschitz trade-off weight β = 0 . 5 . To obtain the singular values, we use PyTorch's implementation of the LOBPCG method, which employs orthogonal basis selection [51].

B Additional Experiments

B.1 1D Synthetic Regression

Table 2 reports the log-likelihood (LL) and expected calibration error (ECE) for the target points, excluding the context data. In Table 8, following the evaluation protocol of [25], we provide the LL and ECE scores on the context points for the 1D synthetic regression task. Consistent with the results on the target set, DNP achieves higher LL and lower ECE scores, indicating better predictive performance and calibration. In terms of inference speed, CNP incurs the lowest computational cost due to its fully deterministic architecture. In contrast, ConvNP is more expensive because of its convolutional structure. Meanwhile, AttnNP, DSVNP, and DNP exhibit comparable inference times, as they share similar architectural components such as attention and latent variable sampling.

To assess the robustness of DNP to observation noise, we evaluate the performance of the NP models on 1D synthetic regression tasks. The functions are generated from the RBF kernel, with noise levels of 5%, 10%, and 15% on the observations. Evaluation on the target set excluding the context data is provided in Table 9. Results show that DNP consistently achieves the highest log-likelihood across all noise settings, while also maintaining strong uncertainty calibration with the lowest ECE in most cases. We also study the role of geometry-aware regularization. Table 10 reports the effect of bi-Lipschitz regularization on ConvCNP and ConvNP across different kernel families in the 1D synthetic regression task. Incorporating the bi-Lipschitz constraint consistently improves log-likelihood and reduces calibration error compared to the unregularized counterparts, with the strongest gains observed for ConvNP.

B.2 Ablation Study

Table 11 presents an ablation study evaluating the impact of the number of context points on CIFAR10 (ID) vs. CIFAR-100 (OOD) classification performance for both AttnNP and DNP. We observe that the accuracy and AUPR scores are robust to the number of context points for both models, remaining approximately constant across different context set sizes, with only minor fluctuations in ECE. Notably, DNP consistently outperforms AttnNP across all metrics and context sizes, achieving

superior calibration and predictive performance. In line with this, we use 100 context points for the main experiments, as it offers a good balance between performance and computational cost.

Table 12 reports the metrics on the effect of varying the distance-aware latent dimension d u and latent dimension d z on CIFAR-10 (ID) vs. CIFAR-100 (OOD) classification. We observe that increasing both d u and d z generally improves performance across all metrics, with the best results achieved when d u = 256 and d z = 512 .

Table 13 presents an ablation study examining the impact of varying the lower ( λ 1 ) and upper ( λ 2 ) bounds of the bi-Lipschitz constraint on model performance. The configuration λ 1 = 0 . 1 and λ 2 = 1 . 0 yields the best performance, balancing regularization strength and representational flexibility. As λ 2 increases while keeping λ 1 = 0 . 1 , we observe a modest trade-off: the accuracy remains competitive, but there is a slight degradation in calibration (ECE) and AUPR scores as λ 2 becomes larger. In contrast, increasing λ 1 (e.g., to 0.6 or higher) or setting λ 1 = λ 2 results in more substantial performance degradation. This suggests that overly tight constraints on the bi-Lipschitz regularizer hinder the model's ability to adapt to complex input-output relationships, limiting both its expressiveness and the quality of its uncertainty estimates.

Table 14 presents an ablation study on the effect of the trade-off parameter β which balances the ELBO and the bi-Lipschitz regularization loss, on CIFAR-10 (ID) vs. CIFAR-100 (OOD) classification. As β increases, calibration and uncertainty quantification improve significantly, peaking around β = 0 . 5 . Beyond this point, calibration remains stable but accuracy slightly drops, suggesting that excessive regularization limits predictive performance. While β was manually tuned in this work to balance predictive performance and uncertainty calibration, automating its selection remains an important direction for future work [24].

We also evaluate the performance when using Laplacian attention (Eq. 5) and dot-product attention, given by

The results are presented in Table 15. While both attention mechanisms perform similarly overall, Laplace attention yields improved classification accuracy, better uncertainty calibration, and OOD detection.

Table 16 presents an ablation analysis of the global and local latent paths in the DNP model. We observe that using only the local latent path yields the best performance in terms of classification accuracy and calibration (ECE), indicating its strong contribution to modeling instance-specific uncertainty. In contrast, the global latent path provides slightly better AUPR. Incorporating both paths yields the best overall performance (see main results in Table 5).

C Proof of Proposition 1

To show that the generative model defined in Eq. 7 is a stochastic process, it needs to satisfy two conditions of the Kolmogorov Extension Theorem [41]:

Permutation Invariance (Exchangeability): The joint distribution p x 1: N ( y 1: N ) must be invariant under any permutation π of the indices { 1 , . . . , N } ; that is,

Consistency (Marginalization): For any N and any M < N , the marginal distribution over any subset { y 1 , . . . , y M } of the outputs must be recoverable by integrating out the remaining variables from the joint distribution p x 1: N ( y 1: N ) .

We now prove each of these in turn. For the proof, we assume that the joint distribution over targets, latents, and inputs factorizes as

Exchangeability: Let π be any permutation of { 1 , 2 , · · · , N } . Since we use a permutation-invariant distribution over the global and local latent variables i.e., the prior p ( z G ) is shared across all inputs and each p ( z i | x i ) is independently defined-the joint distribution remains invariant under reordering of the inputs and outputs:

Thus, the model defines a permutation-invariant distribution over the input-output pairs, satisfying the exchangeability condition required by the Kolmogorov Extension Theorem.

Consistency: From Eq. (7), we have

By Fubini-Tonelli [44] (since all densities are nonnegative and integrable), we may interchange the integrals and the product over i , yielding

The inner bracket integrates to 1 (marginalizing out the last N -M points), leaving exactly

which matches the form of p x 1: M ( y 1: M ) . Hence, marginal consistency holds.

D Derivation of ELBO Loss

The generative model in Eq. 7 is intractable, we introduce a variational approximation q ϕ ( z t , z G ) to approximate the true posterior. Using the Evidence Lower Bound (ELBO) [27], we rewrite the log-marginal likelihood by multiplying and dividing by q ϕ ( z t , z G ) :

Applying Jensen's inequality, we obtain:

This expectation defines the ELBO.

To make inference tractable, we assume a structured variational approximation where the posterior factorizes as:

Expanding the ELBO by substituting the factorization, we obtain:

Rewriting the expectation terms in terms of Kullback-Leibler (KL) divergences, we obtain the final ELBO:

This ELBO consists of: 1. A reconstruction term that encourages accurate predictions of y t . 2. Two KL divergence regularization terms that enforce approximate posteriors q ϕ G and q ϕ L to be close to their respective priors.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces so called Distance-informed Neural Processes (DNP). DNPs build on Neural Processes but additionally include a local latent path and variable, with the main objective of an improved uncertainty calibration. To this end, bi-Lipschitz regularization is used to preserve distances from input to output. DNPs are evaluated on different rasks ranging from onedimensional regression to image classification, with the novel method typically showing strong accuracy and improved uncertainty calibration compared to competing methods such as Attentive NPs.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: I was not able to identify a technical issue in the paper. A proof for Proposition 1 is provided in Appendix C. A strong point is that a variety of applications/experiments from different contexts is shown, including an illustration of predictive distributions/confidence intervals. In addition, substantial additional material is contained in the Appendix, and the code is available in the supplementary material. Overall, I believe the paper to be a thoughtful and sound work.\n\nClarity: The paper was quite clear and relatively straightforward to read. The motivation for the research is presented well, as is the justification of the bi-Lipschitz regularization. Most of the technicals are also presented in a straightforward manner. Much detailed information is provided. I have only smaller suggestions for improvement. What was most confusing to me was the description of Table 4/5 and the corresponding results. The differences between using the two different datasets as ID should be discussed properly. Also, why is latency only shown for the CIFAR10-case? Moreover, I find it suboptimal that some quite closely related work is discussed in Section 2 “Background”, while some is discussed in Section 4 “Related Works”. For example, I was aware of the existence of ConvNP (Foong et al. 2020) and was thus surprised to not see it being mentioned in Section 2, but then later found it to be mentioned in Section 4. Another smaller point is that I did not get the rationale behind sampling the number of context points in Section 5.2.\n\nSignificance: Machine learning methods which provide well-calibrated uncertainty estimates are, without doubt, highly valuable. I thus believe that the paper aims at an important contribution and it improves upon existing NPs. That being said, Figure 3, especially the cases (c) and (d), show that the suggested method DNP still can suffer from substantially underestimating uncertainty and thus yield estimations that are way too confident, even slightly OOD. This would warrant further discussion.\n\nOriginality: To the best of my knowledge, while bi-Lipschitz regularization has been applied to many variants of neural networks, applying bi-Lipschitz regularization to NPs is novel and promising. I also found the reasoning for originality to be convincing overall. \n\nIn sum, I found the paper to be an interesting read and wish the authors all the best for their work on uncertainty calibration.\n\nReference: Foong, A., Bruinsma, W., Gordon, J., Dubois, Y., Requeima, J., & Turner, R. (2020). Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33, 8284-8295.\n\nQUESTIONS:\n1. The paper on ConvNP (Foong et al. 2020) states to propose a new maximum-likelihood objective replacing ELBO, how does the objective in the paper (cf. Appendix D) compare to it?\n2. It was surprising to see DNP being so overconfident in some settings (cf. Figure 3). What are the drivers behind this?\n3. Is there an alternative to the LOBPCG method applied?\nIf these questions are addressed satisfactorily, I may increase my evaluation score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Distance-informed Neural Processes (DNP), a novel variant of Neural Processes that improves uncertainty caliration and OOD detection by integrating both global and distance-aware local latent variables. DNP employs bi-Lipschitz regularization on its local latent space to preserve geometric structure, enabling better similarity modeling. Experiments show state-of-the-art results in regression and classification tasks with improved uncertainty calibration.\n\nSTRENGTHS AND WEAKNESSES:\nI summarize the strength of this work as follows:\n\n(1) This paper is easy to follow and the introduction of bi-Lipschitz constraints to preserve input space geometry in latent embeddings is well-motivated. (2) The developed model improved uncertainty quantification. The dual-pathway (global and local latent) design enhances the expressiveness and reliablity of uncertainty modeling. (3) It conducts extensive evaluation.The paper presents extensive empirical validation across synthetic and real-world tasks, including regression, classification, and OOD detection.\n\n---\n\nAs for the weaknesses, I find the following concerns are waiting to address:\n\n(1) Baselines are a bit outdated, I hardly see new after 2022. It would be necessary to include one of the following ones [1-5] published in NeurIPS/ICML/ICLR with others carefully discussed in the literature work.\n(2) As the distance regularizer is the primary contribution in this work, it is necessary to include one ablation experiment to show Lipshitz term’s influence under varying scales.\n(3) The limitation part should be added in the main paper.\n\n---\n\nReference\n\n[1] Nguyen, Tung, and Aditya Grover. \"Transformer neural processes: Uncertainty-aware meta learning via sequence modeling.\" arXiv preprint arXiv:2207.04179 (2022).\n\n[2] Wang, Qi, and Herke Van Hoof. \"Learning expressive meta-representations with mixture of expert neural processes.\" Advances in neural information processing systems 35 (2022): 26242-26255.\n\n[3] Tailor, Dharmesh, Mohammad Emtiyaz Khan, and Eric Nalisnick. \"Exploiting inferential structure in neural processes.\" Uncertainty in Artificial Intelligence. PMLR, 2023.\n\n[4] Wang, Qi, Marco Federici, and Herke van Hoof. \"Bridge the Inference Gaps of Neural Processes via Expectation Maximization.\" The Eleventh International Conference on Learning Representations.\n\n[5] Gondal, Muhammad Waleed, et al. \"Function contrastive learning of transferable meta-representations.\" International Conference on Machine Learning. PMLR, 2021.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nSee the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces *distance-informed neural processes* (DNP), which extends attentive NPs (AttnNPs) by encouraging the representations of the context and target points to maintain their relative distances from the input space. The authors achieve this by regularising the encoder through a bi-Lipschitz constraint, which encourages the singular values of the weight matrices to be within some predefined upper and lower bounds. The resulting model is trained through variational inference and evaluated on both regression and classification problems, outperforming baseline models on most tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed model appears original and generally outperforms the baselines significantly. \n2. The experimental section is extensive. In particular, the authors have performed several ablation studies to understand the effect of the introduced smoothness constraint.\n3. The paper is well-written and well-structured.\n\n**Weaknesses**\n1. While the model seems original, its novelty is limited. The paper seems quite incremental, and given that adding the bi-Lipschitz constraint to the objective function is a fairly small change, I would have expected a more thorough analysis of the effect of the constraint on, e.g., training stability, sample quality, and other NP architectures, say, the ConvNP.\n2. The description of the proposed model and its relation to other models from the literature could be improved, for instance with a side-by-side comparison of their (simplest possible) graphical models and/or key equations.\n3. While the reference list is extensive, there are surprisingly few papers on neural processes or neural network smoothness cited. And even among those that are cited, it is not entirely clear how the current model positions itself in the literature.\n4. I found the motivation for introducing the bi-Lipschitz constraint and Figure 1 confusing and a bit misleading. To me, it sounds like the constraint is imposed to try to enforce that the relative distances of the data are unchanged by the encoding, which Figure 1 seems to support, but that would essentially mean that the encoding is an identity map. Rather, the bi-Lipschitz constraint imposes smoothness in the encoding transformation, and I think this is a more helpful mental picture of what the proposed model does and why it should work.\n\nQUESTIONS:\n**Suggestions**\n1. As mentioned under weaknesses, I am not convinced that the motivation for introducing the bi-Lipschitz constraint (i.e., to maintain relative distances) is helpful. I would suggest focusing instead on the smoothness properties of the non-linear transformation, which, I realise, is to some extent a different way of saying the same, but it would be clear that you are not trying to learn an identity mapping (which Figure 1c seems to be and which would be counterproductive).\n\n**Questions**\n1. It seems to me that your approach to imposing the bi-Lipschitz constraint is readily applicable to other architectures, e.g., convolutions. Is this correctly understood? If so, did you test bi-Lipschitz ConvNP or similar?\n2. I’m curious to understand the effect of the smoothness constraint on the model performance. As another ablation study, did you try to train the same architecture as the DNP but with some standard regularisation (e.g. weight decay) instead of the bi-Lipschitz constraint?\n3. What do the errors in the tables refer to? They seem quite small.\n4. Did you try to visualise samples from your model, e.g., for the 1D synthetic experiment? AttnNP can suffer from weird kinks in the samples, and I am curious to see if the bi-Lipschitz constraint had an effect here.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Distance-informed Neural Process (DNP), a new member of the Neural Process family designed to improve uncertainty quantification and generalization. The authors identify a key weakness in standard NPs: their reliance on a single global latent variable often leads to poorly calibrated uncertainties and a failure to capture local data dependencies. To overcome this, DNP employs a dual-path architecture. A global path, similar to standard NPs, captures task-level uncertainty. The main contribution is a local latent path, which models input similarity explicitly. This is achieved by learning a distance-preserving latent space for the inputs, enforced through a novel bi-Lipschitz regularization on the encoder network. The similarity between points in this regularized latent space is then used to compute attention weights, allowing the model to make predictions that are informed by local context. Through extensive experiments on regression and classification tasks, the paper demonstrates that DNP achieves superior predictive performance, better-calibrated uncertainty estimates, and more effective out-of-distribution (OOD) detection compared to several state-of-the-art NP variants.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper proposes a well-motivated and technically sound solution to a known limitation in NPs by integrating a distance-aware local latent path.\n- The method's effectiveness is convincingly demonstrated through comprehensive experiments showing strong performance, especially in uncertainty calibration and OOD detection.\n- The paper is exceptionally clear and well-written, with intuitive figures and a thorough appendix that aids understanding and reproducibility.\n\nWeaknesses\n- The work is an incremental advancement over existing NP variants, combining known concepts in a novel way rather than introducing a completely new paradigm.\n- The model's O(NM) computational complexity from cross-attention presents a scalability challenge for tasks with large context sets.\n- The proposed bi-Lipschitz regularization is limited to Euclidean spaces, which may hinder performance on data with non-Euclidean geometries.\n\nQUESTIONS:\n1. **Connection to Classical Methods**: The core idea of using distance-based weights for local modeling is conceptually close to classical non-parametric methods like locally weighted regression (LWR) or kernel regression. Could you elaborate on the relationship between DNP and these methods? Specifically, what are the key advantages of your deep learning approach with bi-Lipschitz regularization compared to a modern variant of LWR or a deep kernel learning model that also aims to preserve local similarity?\n\n2. **Scalability of the Local Path**: The O(NM) complexity of the cross-attention mechanism is a practical limitation for large context sets. You mention exploring sparse attention as future work. Have you considered or run preliminary experiments on simpler, more scalable approximations? For instance, one could attend only to the k-nearest neighbors in the latent space. How does performance degrade under such approximations? A brief analysis would provide valuable insight into the performance-efficiency trade-off.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}