Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Authors: Huafeng Liu
Yiran Fu
Liping Jing
Hui Li
Shuyang Lin
Jingyue Shi
Deqiang Ouyang
Jian Yu
Venue: ICML 2025 poster (2025)
Keywords: neural process
robust learning
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Neural processes (NPs) are a promising paradigm to enable skill transfer learning across tasks with the aid of the distribution of functions. The previous NPs employ the empirical risk minimization principle in optimization. However, the fast adaption ability to different tasks can vary widely, and the worst fast adaptation can be catastrophic in risk-sensitive tasks. To achieve robust neural processes modeling, we consider the problem of training models in a risk-averse manner, which can control the worst fast adaption cases at a certain probabilistic level. By transferring the risk minimization problem to a two-level finite sum minimax optimization problem, we can easily solve it via a double-looped stochastic mirror prox algorithm with a task-aware variance reduction mechanism via sampling samples across all tasks. The mirror prox technique ensures better handling of complex constraint sets and non-Euclidean geometries, making the optimization adaptable to various tasks. The final solution, by aggregating prox points with the adaptive learning rates, enables a stable and high-quality output. The proposed learning strategy can work with various NPs flexibly and achieves less biased approximation with a theoretical guarantee. To illustrate the superiority of the proposed model, we perform experiments on both synthetic and real-world data, and the results demonstrate that our approach not only helps to achieve more accurate performance but also improves model robustness.

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Huafeng Liu Yiran Fu"
Liping Jing 123 Hui Li Shuyang Lin
Jingyue Shi
Deqiang Ouyang* Jian Yu" 12
Abstract
1. Introduction
Neural processes (NPs) (Gamelo al., 2018a;b) represent class of approximation models for Gaussian processes (GPs) (Rasmussen, 2003), offering promising attributes in terms of computational efficiency and uncertainty quantifi- cation. Unlike conventional statistical modeling, where a user typically needs to manually specify prior, such as the smoothness of functions characterized by Gaussian dis- tribution in Gaussian processes, NPs subtly define broad spectrum of stochastic processes using neural networks in data-driven fashion. When trained appropriately, NPs can delineate flexible range of stochastic processes that are particularly apt for representing complex functions that existing stochastic processes struggle to capture.
Neural processes (NPs) are a promising paradigm to enable skill transfer learning across tasks with the aid of the distribution of functions. The pre- vious NPs employ the empirical risk minimiza- tion principle in optimization. However, the fast adaption ability to different tasks can vary widely, and the worst fast adaptation can be catastrophic in risk-sensitive tasks. To achieve robust neural processes modeling. we consider the problem of training models in a risk-averse manner, which can control the worst fast adaption cases at tain probabilistic level. By transferring the risk minimization problem to two-level finite sum minimax optimization problem, we can easily solve it via a double-looped stochastic mirror prox algorithm with a task-aware variance reduction mechanism via sampling samples across all tasks. The mirror prox technique ensures better handling of complex constraint sets and mon-Euclidean ge- ometries, making the optimization adaptable to various tasks. The final solution, by aggregat- ing prox points with the adaptive learning rates, enables a stable and high-quality output. The pro- posed learning strategy can work with various NPs flexibly and achieves less biased approxima- tion with theoretical guarantee. To illustrate the superiority of the proposed model, we perform ex- periments on both synthetic and real-world data, and the results demonstrate that our approach not only helps to achieve more accurate performance but also improves model robustness.
NPs model the relationship between inputs and outputs in the form of probability distributions. This allows the model to generate predictions with associated uncertainty, which particularly beneficial when only a small amount of training data is available. By capturing the model's confidence, NPs. are more robust in few-shot scenarios. NPs can leverage meta-learning to extract shared knowledge from multiple related tasks. This enables the model to quickly adapt to new tasks by utilizing the information learned from previous tasks. Even with limited data in the current task, the model can make accurate predictions by drawing this shared knowledge. Unlike traditional Bayesian methods, NPs do not depend on explicit prior assumptions. Instead, they learn directly from the data, making them more flexible and effective in few-shot learning contexts. Thanks to the characteristics above of NPs, many researchers are focusing on the few-shot learning capabilities of NPs (Gordon et al., 2019; Liu al., 2024b; Bruinsma al., 2023)-
However, the optimization in most of the previous NPs optimizations has adopted the empirical risk minimization principle, overlooking the differences in rapid adaptation between tasks. Given a batch training tasks, NPs treat the task weights uniformly, which can result in insufficient training for the fast adaptation capability certain tasks. As shown in Figure 1, we present the original task risk frequency statistics. It can be observed that high-risk tasks (with risk values above 70%) account for more than 30% of the total tasks. Therefore, when a new task matches
'Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China "School Computer Science and Technology, Beijing Jiaotong University, Beijing. China 'State Key Laboratory of Advanced Rail Autonomous Operation, Bei- jing, China 'Collage of Computer Science, Chongqing University, Chongqing, China. Correspondence to: Deqiang Ouyang, Liping Jing <dcqiangouyangee @ cqu.edu.en. Ipjing @ bjtu.edu.en>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Frequency
and optimization algorithm can be flexible to work with vari- ous NPs and achieve less biased approximation. Extensive experiments conducted on both synthetic and real-world datasets demonstrate that the proposed solution can improve robustness.
#Onginne risk A Almproved
Higher frequency Lower frequency
Risk
2. Related Work
High risk
In this section, we briefly review three different areas that are highly relevant to the proposed method: neural processes and risk-sensitive optimization.
Figure 1. Illustration of the task risk distributions of original NPs and our robust NPs.
Neural Processes Neural Processes (NPs) are a class of models that aim to combine the flexibility of neural net- works with the probabilistic framework of Gaussian Pro- cesses (GPs). Introduced by (Garnelo et al., 2018a), NPs learn distribution over functions by conditioning on a set of context points. This allows them to perform tasks such as few-shot learning and function approximation with remarkable efficiency. Several mechanisms or techniques have been integrated into NPs to enhance their performance and applicability, such as attention mechanisms (Kim et al., 2019), hierarchical priors (Wang Van Hoof, 2020), transla- tion equivariance (Gordon et al., 2019; Kawano al., 2021), bootstraping (Lee et al., 2020). In recent years, cutting- edge technologies have been applied to NPs to enhance model performance, such as neural ODE (Norcliffe et al., 2021) 1), transformer (Nguyen & Grover, 2022; Maraval et al., 2024), diffusion model (Dutordoir et al., 2023), contrastive learn- ing (Ye & Yao, 2022), autoregressive mechanism (Bruinsma et al., 2023; Tai, 2023). Unlike making improvements to the model, many researchers have also explored the ap- plicability of NPs to different specific tasks, for example, recommender systems (Lin et al., 2021; Liu et al., 2022; 2024b), hyperparameter optimization (Wei et al., 2021), neuroscience (Cotton et al., 2020; Pakman et al., 2020), space science (Park & Choi, 2021), and physics-informed systems design (Vadeboncoeur et al., 2023).
these high-risk tail tasks, the prediction risk for the new task is similarly high, thereby affecting the performance of the new task. This issue is particularly critical in sensitive scenarios, such as spike sorting (Pakman et al., 2020), and robot navigation (Yildirim & Ugur, 2022).
In this paper, we foeus on the performance difference in fast adaptation various tasks an indispensable consid- eration. By investigating the learning paradigm from the view of risk distribution, we convert the optimization prin- ciples for NPs to stochastic optimization problems while minimizing the expected tail risk, i.e., conditional value- at-risk (CVaR) (Rockafellar et al., 2000). Specifically, we focus on expected tail risk minimization for NPs with CVaR, which can reshape the task risk distribution and control the task's fast adaptation ability at certain probabilistic level. When using variant of SGD for solving the tail risk, it is common to approximate the expectation through mini-batch sampling. However, if the batch is sampled uniformly at random, only a small fraction (c.g., a in CVaR) of the se- lected samples may carry meaningful gradient information. For the remaining points, their gradients are often truncated to zero due to the presence of the max- operation and the inherent non-linearity. On the other hand, for the points that do contribute, their gradients are amplified by a factor of a small number (e.g.. 1/0x,, which can result in exploding gradients. Meanwhile, only a small number of higi-loss samples contribute to the gradient, and the variance of the gradient estimates is significantly high, which makes the optimization trajectory highly unstable. These combined issues make stochastic optimization highly challenging.
Risk-sensitive Optimization Robust Optimization (RO) (Gabrel et al., 2014; Rahimian & Mehrotra, 2019; Yu et al., 2024) aims to improve model stability and reliability by designing algorithms capable of handling uncertainty and perturbations. The core idea of robust optimization is to consider the worst-case performance of a model with min-max formulation during the training process, rather than just its expected performance (Bertsimas & Sim, 2004) by introducing uncertainty sets or adversarial perturbations. Robust optimization strategies typically include two major categories: adversarial robustness and distributionally robust optimization. Adversarial examples are a specific form of perturbation in machine learning, where an attacker can deceive the model by making small perturbations to the input (adversarial attacks) (Gabrel et al., 2014; ?). Distributionally robust optimization is
Inspired by the previous work (Alacaoglu & Malitsky, 2022), we formulate the CVaR problem as distributionally robust optimization problem with stochastic minimax opti- mization and introduce a variance-reduced stochastic mirror prox algorithm. Specifically, we follow the double-loop structure of variance reduction and leverage the two-level structure compute the stochastic gradient based on a sam- pling strategy across all tasks, which can get a variance- reduced stochastic gradient estimator. The final solution, by aggregating prox points with the adaptive learning rates, enables stable and high-quality output. The proposed risk
2

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
method that focuses on ensuring model robustness when the training data distribution changes(Rahimian & Mehrotra, 2019). Among them, the risk principle Value-at-Risk (VaR), as representative method of robust optimization with probabilistic constraints, was first introduced in the financial sector (Rockafellar et al., 2000), gained significant attention (Quaranta & Zaffaroni, 2008), and has since been widely applied to other fields, such as healthcare (Chen et al., 2019), reinforcement learning (Pinto et al., 2017; Gagne & Dayan, 2021), smart grid (Ben-Tal -al & Nemirovski, 1999; Bertsimas et al., 2012), and so on. As for robust strategies for NPs, some researchers have explored approaches such as using bootstrapping (Lee et al., 2020) or ensemble methods (Liu et al., 2024a).
To achieve fast prediction on new context set at test time, NPs meta-leam distribution over predictors. Consider the distribution of tasks p(T) for NPs. Let the task space be S2 and be the task sampled from the distribution p(r). To perform meta-learning, we require meta-dataset (dataset of datasets) Pr,1:M = {D J m=1 with contains M datasets. In NPs, the task risk function E(DŞ, DT;; 0) evaluetes the model's adaption performance. Based on the commonly used risk minimization principle, the expected risk mini- mization of NPs is given as follows.
Definition 3.1. (Expected Risk Minimization for NPs) Given the task distribution p(T), the expected risk mini- mization for NPs is given as
min R, : (b))==)))(),, 8
(1)
3. Preliminaries
Let calligraphic letter (e.g.. A) indicate set, capital let- ter (e.g., A) for scalar, lower-case bold letter (e.g., a) for vector, and capital bold letter (e.g., A) for matrix. Sup- pose there is a dataset D = (X,y) = [x), with N data points X = [1, and corresponding labels y [V1, 1/...... : .NN] E BE Considering an arbitrary number of data points DC (Xe, ) {(x;, JiEc where C C
The task distribution determines the diversity and breadth of the meta-dataset, which plays crucial role in fast adaptation. By performing Monte Carlo estimation to (q.(1), we can get the empirical risk minimization as ming 1 EE_ (e(E,, (T: B
[N]] E RND
Next, we focus on defining the task risk functions as ran- dom variables. Given the task space Q, we can define probability measure p : ō, + [0, 1] over the task space as (S+, 5r.p), where 8, is a o-algebra on the subsets of task space. Let (R++, B) be a probability measure over the non-negative real domain defined by the task risk function {(DE, r. D 0) and B be a Borel o-algebra, the NP operator No: 2 + F is given as NP:: : T + DD;
is an index set defining context information, neural pro- cesses model the conditional distribution of the target values yT = hiteer at some target data points X = (xI based on the context DC, i.e. Eptcypo(y T |X" (T) = Excc) ΣiET po(y)(x,, DC), where p(C) is a context prior indicating selecting context information from the whole data set D, e.g., |C] ~ U[I,..., .N.. Usually, target set is defined as T = f1.2. ... NN. Only in CNP (Garneo et al., 2018a) T C (1, ... , and Tnc= 0.
In this case, the task risk function {(D,, ((DF,DT;)) can be viewed as a random variable, and we can induce distribu- tion p(E(DC, ;)) over the task risk function. The cumu- lative distribution function (CDF) of a real-valued random variable e(DC, DJ 6) can be formulated as Fr(E: = P(E(DC, :): S E), where & E R We cannot give an explicit form of this CDF since it implicitly depends on the loss function f(-) and model parameter A.
Usually, a latent variable z is introduced to capture model uncertainty and the NPs infer (o(2)DD) given context set using the reparameterizattion trick (Kingma & Welling, 2013) and models such a conditional distribution as po(ys|x; Xx; DC) = po (x;, De z)go (z 'DC) )D),, and it is trained by maximizing an ELBO: E(DE (T:
(erqq(<x(DT) [log [logge(y" Ix']] - K L ge (z) (z)D7 (lIpe(z|DD)
Based on the definition of VaR and CVar (Fq,( we can define the VaR and CVaR of the task risk function.
The problems are difficult non-convex optimization prob- lems, especially when the neural networks are intro- duced, and iterative methods, e.g., gradient- meth- ods, are adopted to find solutions converging local mini- mum (Wang& & Van Hoof, 2020).
Definition 3.2. Given the task risk function E(Dr, 0) with cumulative distribution function Fe(E; 0). the VaR of the task risk function at level a E (0, is given as Va (((,,..:M;)) 0)) = (nf 8) 2 a).
3.1. Risk Minimization of NPs
Definition 3.3. Given the task risk function E(D, ((D,,);;;)) 0) with cumulative distribution function Fe(E: and the VaR of the task risk function vo (E(D,, 0)). Assume the random variable is constrained by ( :); Z v, (E((D,,MM: 6)), the CVaR of the task risk function is defined as ca (e(((Dr,..v;) = foe EdFF 9), where (C; 0) is the normalized cumulative distribution, which
To achieve robust NPs and inspired by (Wang al., 2024), we can optimize CVaR of the task risk function for NPs with the adjustable confidence level a, and define controllable expected tail task risk based on the CVaR of the task risk function.
3

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
is
where ((DG, 7,13 D),50) 2 8, the gradient is scaled by 1/a. Since is usually small value (e.g.. 0.01 or 0.05), this seal- ing can cause the gradient magnitude to grow excessively large, leading to exploding gradients and destabilizing the optimization process. 3) High Gradient Variance: Because only a small number of high-loss samples contribute to the gradient, the variance of the gradient estimates is signif- icantly high. This high variance makes the optimization trajectory highly unstable and can cause the algorithm to converge slowly or get stuck in local extrema. In summary, the combination of vanishing gradients, exploding gradi- ents, and high gradient variance leads to inefficiency and instability in stochastic optimization algorithms for CVaR objectives.
<< (o(((rriaaar))) (E: = { F((E0))aa {2 Va (e(((Dr,:MM)))))
Thus, we can obtain the normalized probability measure (Ra.r: Pa)), where Sla,r = (EPP, (((P)..):))) [N Po I-- In this case, the task distribution constrained in Qa,r is denoted by (T)..
3.2. Learning with the CVaR
Given the constrained task distribution Pa: (T), the control- lable expected tail task risk is defined minimizing CVaR ca ca(C(Dr,1.MM) 0)). which can be rewritten follows:
min Ra.- == (p..> E
(3)
4. Algorithm for Tail Task Risk Optimization
The CVaR objective has a natural distributionally robust optimization (DRO) formulation. Given the DRO set Qa- = { E IR- S qm I 1/k; 4m = 1} with k = La the DRO problem can be formulated as the following min- max problem:
However, directly optimizing the above controllable ex- pected tail task risk is intractable since the constrained task distribution (T) implicitly depends on @ and and cannot access an explicit closed-form expression.
To avoid directly computing the constrained task distribu- tion Pa (T), by introducing slack variable / and auxiliary risk function [M(DC; DJ; 0) - I]+, the probability-constaal function in E3) can be converted into the following un- constrained form:
M
min max PEB O
F.(0.)) : Σ 4m Rm 112
(6)
The above problem is a finite-sum convex-concave saddle- point problem if (0) is a convex function. However, introducing neural networks often renders (n)()) non- convex, leading to variance in algorithms like stochastic mirror descent and stochastic mirror prox, designed for saddle-poin problems. This variance results in instability during the parameter learning process.
min R., (.,10))=11 12.10))=+++-0 6EBJERR
(4)
Let samples from Drm be {mm. where num is the number of samples in the dataset Dr,m and Z= (x, By replacing the expectation Ep(=) with the empirical tasks, the problem (4) yields the following learning problem:
finite-sum
Here, inspired by (Carmon & Hausler, 2022; Yu al., 2024), we incorporate variance reduction into the stochastic mirror prox algorithm. Specifically, we introduce a simple and effective task-sampling technique that selects M samples, one for each task, to form the stochastic gradients. Addi- tionally, we incorporate variance reduction strategies within each task to accelerate the convergence rate.
M Σ
min (l,( := 1+ (1 M DEOJER
Rm (0)
- ]+]
(5)
where (f) Enm Em (Zm...i))) This formulation focuses on the tail distribution of the loss by truncating smaller losses and optimizing only large ones.
4.1. Distance-generatinng Function and Bregman Setups
3.3. Challenges for Stochastic Optimization
Let ||. ||r denote a general norm on a finiteddimensional Banach space Eg, and its dual norm is defined as || 1|,,- = supyee, ((r, v) llyll. S 1). We define the set [S] {1,2, .S and its index set [S]0 =(0, (.......... S 1}. where S is positive integer. For (0,q) 0 x Qa where O. represents the simplex, we view (0.g) as the concate- nation of 0 and q. The gradient F.. (9, q) is expressed as (0, q) = (V&Fa (0, g); V Fa (0, q)). denoting the combined gradient with respect to 9 and g.
A variant of the SGD method can optimize the above prob- lem. However, there are several challenges in CVaR op- timization: 1) Vanishing Gradient Problem: In the CVaR optimization formula, gradients are non-zero only for sam- ples where the loss E(Df.,, DT.;; 8) exceeds the truncation threshold s. However, only a small fraction of samples (proportional to a) typically satisfy this condition. As a result, most samples contribute no gradient, leading to fewer effective updates and slowing or even halting the optimiza- tion process. 2) Exploding Gradient Problem: For samples
For primal-dual methods of the mirror descent type(Beck

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
& Teboulle, 2003), it is necessary to define distance- generating function and the corresponding Bregman diver- gence.
mini-batch gradient estimates of CVaR, particularly for non- convex problems like training deep neural networks. The double-loop structure of variance reduction is the commonly used strategy in optimization algorithms. It consists of an outer loop and an inner loop, which separate global updates from local refinements, providing a structured approach to achieving optimization goals. Inspired by the previous work (Carmon & Hausler, 2022; Alacaoglu & Malitsky, 2022), we adopt double-loop structure based on variance reduction. The outer loop computes snapshot points in both the primal and dual spaces, while the inner loop uses a modified mirror prox method. Instead of computing two stochastic gradients, we use a combination of a mini-bate gradient and a stochastic gradient, building upon classical mirror prox algorithms.
Definition 4.1. (Distance-generating function) A contin- uous function d/x : X - R is termed a distance- generating function with modulus 0g relative to the norm l1 if the following conditions hold: 1) The set X = {2r E X (x) * 0} is convex; 2) the is con- tinuously differentiable and ag -t-snnng convex with re- spect || 12. i.e., (m)) - V1lz(12), 22) 2 a=1|=1 - 22ll V2 E X
Definition 4.2. (Bregman divergence) The Bregman di-
vergence Br X x X + I associated with the distance-generating function dh is defined as: Bx (x,xx²)) = V2(x)) - (x²) - (V (x x xx).
Outer Loop The outer loop follows standard variance reduction procedure (Johnson & Zhang, 2013), periodically computing snapshot points based on the weighted average of previous iterates. Let the snapshot in the s-th outer loop be (6, q)", and Vib((,,) g)") be the corresponding mirror snapshot. Formally, we define (0, q)" as:
In this work, we equip @ with a distance- distance-generating function Vo()), which has modulus ag with respect to a norm || llo on E. Similarly, (+) is defined with modulus a, relative to the norm ll,- The choice of u/2, and || lla should align with the geometric structure of the domain. In this paper, we use (g) = E qi log qt, which corresponds to the entropy function.
(0.g)* = Ex=1 w w 4)
(8)
E w
The following standard assumptions are also widely used in Bregman setup analyses.
and the mirror snapshot is
Assumption 4.3. (Boundedness of the Domain) The diam- eter of the domain + under the distance-generating function vho(-) is bounded by constant Da, i.e., maxsee (0) mingge (0) < D2- Similarly, the simplex Qa is as- sumed to be bounded by Dy. Since the entropy function 1/4 is used, we have Dg M.
> km Wk-
Kh-t s-1 Vebi ()0,4)z²)
V ((0,g)))
Ek w
The full gradient (Fa((O, 9) is then computed as:
M
We define the Cartesian product space E x R-M with its associated norm and dual norm as follows. For any (0, 4) E θ x IR M and any (0,g)* 9)* = (0";4") ) E 8x x RM;
VoFa Σ d. R. (0") m=1
(10)
VOFa (0) = - (R)) ... "Rr((
II(0, q)ll =
V 2Dg
Cke
1011 +
2D3
0gg 12l.
Inner Loop In the k-th inner loop, we updates 2 using a modified mirror prox step:
2D
2D
[I(0.g) g) ||+ = V
0 01.13.
+
l1
13...
(0.9) = arg min ((FF(((0))))) (0. (0.9)EZ
ax
22
The corresponding distance- function is defined as ((00,)) 9)) 2ng (0) + 207 (g)
+ (1,
(0.) (11)
It is straightforward to verify that v²((0,g) g)) is 1-strongly convex with respect to the norm I-Il defined in Eq. (7). Using this, we define the Bregman divergence B (9 x 2.) x (O x 2x + R. as: B((0, (0, (0.9)) 9)9) == ((0,g)))
which utilizes the full gradient VF, (0','0))', The above update employs techniques such as "negative momentum" (Driggs et al., 2022) to enhance convergence speed. Note that the above prox point using full gradient differs from traditional mirror prox algorithms (Nemirovski, 2004), which use stochastic gradients. Although a full gradient can achieve more stable solutions, it is tonecconsuming. To alleviate computational complexity, we can replace the full gradient.
((e,, () (V)(()) (0).)))))))))))))))))))))) (),9)%).
4.2. Double-loop Structure of Variance Reduction
Unfortunately, this method proves ineffective when stochas- tically optimizing CVaR due to the high variance in the
gradient (Eq. (10)) with a
5

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
4.3. Variance-reduued Stochastic Gradient Estimator
Algorithm 1 Variance-Reducee Stochastic Mirror Prox Al- gorithm for Tail Task Risk Optimization
After (0, 4) is computed, we sample data points from each task (dataset) to construct the stochastic gradient. For the m-th task, we sample uniformly from the corresponding dataset Dr,m- The sampling process is denoted as: 1) for each dataset, uniformly sampling one sample: 2 ~V U UlZZm,s) 1. ), Vm E [M]]; 2) forming the task-specific samples as - == { 1M
Input: Risk functions (fm( (m) related to NPs, epoch number S, iteration numbers {, learning rates (ng J. and weights {wi}-
1: Initialize parameters arg min (((0, q)) as the starting point.
(0. = (8o, go) =
2: for 8 = S do
1-=1
3: Compute the snapshot (0, q)* and the mirror snapshot
The task sampling strategy ensures that stochastic gradients utilize information from all M datasets. In this case, we can get the task-specific stochastic gradient based the task- specciii samples {
Pv((0.q))) according to Eq. and Eq,( respec- tively.
4: Compute the full gradient F., ((0, g)) according to Eq. (10). (mini-batch is feasible)
k.m Im=1:
M Vol'aZk 0 ,0) Σ
5:
for k == 0 to K. - do
Am Ven (Z 0²))
6:
Compute (0, 9) according to Eq.(11).
m=1
7:
For each m E [M.) Zk.m. ~ Uf
V. Vo mm 0² ,0, - -00---((12 [Zi 0-), ess (k 0²)
8:
Compute the variance-reduced stochastic gradient estimator Se defined in (13).
(12)
9:
Compute (0, (0,4) according to (q.(14).
This formulation balances randomness across tasks, ef- fectively leveraging the two-level finite-sum structure of Eq. (6).
10: end for
11: Set (0, 1) ().
12:
end for
13: Return (0,4)* according to Eq. (15).
Leveraging variance reduction techniques from (Johnson & Zhang, 2013), we construct a robust and efficient approach to minimize variance in stochastic gradients while maintain- ing computational efficiency. Specifically, we define the variance- stochastic gradient estimator with the aid of the stochastic gradient at the s-th snapshot (0, q)* and the full gradient as follows:
which adopts adaptive learning rates vff as weighting coeffi- cients for the cumulated prox points. The whole variance- reduced stochastic mirror prox algorithm for tail task risk optimization is given in Algorithm 1.
g Fa (lZi JM (0, (0.9)%-111
4.4. Discussion
Fa (1Z k.m (0, )) + VFa((0,4))
The task sampling method uniformly selects data across all tasks, thus reducing the variance of stochastic gradients. This approach leverages the hierarchical structure of the objective function to achieve a Lipschitz constant reduction, ultimately lowering the computational complexity. In the gradient computation process of the outer loop, the weight- ing coefficient for the gradient from the previous step is typically determined using average weighting strategy, i.c., _ = k-- Ak = 1. K.-1. However, we can set it as a learning rate πy as shown in Eq. (15). In many al- gorithms, the learning rate is gradually decreased, meaning that earlier iterations may contribute larger weights. This op- eration can reduce fluctuations in each gradient update and improve the stability of the algorithm. Additionally, by com- prehensively leveraging historical solutions or gradients, the quality of the current solution can be enhanced. In summary, by using variance-reduced gradient estimator in the inner loop, our solution significantly reduces gradient variance, improving convergence rates. The mirror prox technique ensures better handling of complex constraint sets and non- Euclidean geometries, making the optimization adaptable to various tasks. The final solution (0, g aggregates the
(13)
M where the second term VF. (Z m= (0, (0,4)* is
given in Eq. (12), the third term (F.((0, 9)") is given in Eq. (10). The above gradient is an unbiased estimator since the samples across all tasks are used.
With the aid of the mirror prox method, the next step to compute (0,4)
(0,4)h+1 = arg min {niloE (0,
(0.g)E8x..
(6. (1.)))) (14)
+
In the above stochastic gradient, the variance-reduced stochastic gradient estimator of is used instead of the raw stochastic gradient at (H, g) to achieve variance reduction. The final solution is given as:
Σ5 = > k=0 The (0, 4)h
S-1 K,-1
(0,g)s
(15)
K,-1
E= Ek=0
1/
6

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Crggnal
Contest
CXP
Scip
SCNT
RCAP
ANF
BA
SANP
RAMP
Table Comparison of our robust NPs with the baselines log- likelihood of the target points on two real-world datasets: CelebA and EMNIST. We train each method with 5 different seeds and report the mean and standard deviation.
CelebA
EMNIST
Method
Seen (0-9)
Unseen (10-46)
8
CNP
2.0601
0.737322.004
0.485442.004
BCNP
2. 2.165210.006
0.7552.0.00s
0.498210.006
SCNP
2.1713
0.7631 1.00 20.006
0.512520.008
RCNP
2..896+0.005
0.7879 =0.009
0.5362+a..0n
(a) CelebA
NP
2.481620.015
0.7953=0.002
0.5847220000
BNP
2.7691
0.8706
0.716720.012
SNP
2.8714
0,8873
0.7021
3
3
3
5
3
3
3
3
5
RNP
2.8915
0.8911 10.005
0.0121
ANP
2.52214.0.004
0.9875 5.00000
0.884330.000
9
7
9
9
9
9
9
9
BANP
2.9411
20.008
0.98333a..0m
0.889 0.889120.008
SANP
2.9518 g0.000
0.9753g0.000
0.892420.005
RANP
2.9718 +0.006
0,9881
+0.004
0.8967
7
J
ConvNP
3.1241
1.1525 20.021
1.051
BConvNP
3. 3.1683g0.005
1.165 100.000
1.0511 20.004
SConvNP
3.2135
20.00
1. 1.0698
RCoavNP
3.2319 20.000
1.2348
g0.005
1.0721 1.072 100006
TNP
4.404220..co
1.5501
+0.004
1. 1.4196
(b) EMNIST
BINP
STNP
4.4151 4.4151 20.004
24.124.0.000
1,5561
200002
10,003
1. 1.4221 1.4221 1.,004
1.4236g0.008
Figure 2. Visualizations of CNP, ANP, and their bootstrapping ver- sions (BCNP, BANP), stable versions (SNP, SANP), and robust versions (RCNP, RANP) for image completion tasks.
RTNP
4.4226 20.00
1.5572 1.5572g0.00 20.00
1..013
Table 2. Bayesian optimization experiments on data generated by
different GP kernels
Method
RBF
Matérn 5/2
Periodic
5.1. Image Completion
ANP
0.124510.00 10.003
0.1518 10.003
0. 0.189210.00
BANP
0.134110.000
0.131610.00
0.1788
Following (Kim et al., 2019), we compared the models on image completion tasks on CelebA (Liu et al., 2015) and EMNIST (Cohen al., 2017), where each image is down- sampled to 32 32. More detailed experimental settings are given in the Appendix. Table lists the log-likelihood of the target point of our robust NPs over the baselines on two datasets. Notably, on the EMNIST dataset, in addi- tion to the standard experiments, we also conducted model- data mismatch experiments. Specifically, we trained on the first 10 classes and tested not only on these first 10 classes (seen classes) but also the remaining 37 classes (unseen classes). Similar to ID regression, our robust NPs still out- perform all baselines, especially in model-data mismatch settings (unseen classes). In the mismatch setting, all the models become less accurate, but NPs with our robust so- lutions are affected less, which is similar to the results in ID regression tasks. Furthermore, we visualize the gener- ated images and Figure shows that the proposed robust NPs produce noticeably better-completed images than the baselines.
SANP
0.1142 10.002
0.1201
10.002
0.1672 1..000 00.001
RANP
0.1025.0.002
0.1171
20.003
0.1779 =0 006
TNP
0.1125 10.03
0.1451
10.001
0,1715
L.. 003
BTNP
0.1037 10.006
10.000 0,1455
0.1691
1.0
STNP
0.0998
0.1351
0.1561
=0. 002
RTNP
0.0915.0.000
0.1289 20.003
0.1463 00.
updates through weighted averaging, enabling a stable and high-quality output.
5. Experiments
We started with learning predictive functions on synthetic datasets, and high- -igh-dimessioall tasks, e.g., image comple- tion, Bayesian optimization, and contextual bandits, were performed to evaluate the properties of the NP-related mod- els. We compare the basclines NP classes (CNP (Garnelo et al., 201 8a). NP (Garnelo et al., 2018b), ANP (Kim al., 2019), ConvNP (Foong et al., 2020), and TNP (Nguyen & Grover, 2022)), bootstrapping versions (Lee et al., 2020) (BCNP, BNP, BANP, BCOnvNP, and BTNP), stable ver- sions (Liu et al., 2024a) (SCNP, SNP, SANP, SConyNP, and STNP) to our robust versions (RCNP, RNP, RANP, RCon- vNP, and RTNP).
5.2. Bayesian Optimization
Following the setting in BANP (Lee et al., 2020; Nguyen & Grover, 2022), we conducted the Bayesian optimization experiment, see Appendix for details. Taking GP data with RBF, Matérn 5/2, and Periodic prior functions as examples, we gave the results of ANP, TNP, corresponding bootstrap- ping NPs (BANP, BTNP), stable NPs (SANP, STNP), and
7

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Table Comparison of our robust NPs with the baselines on cumulative regret on contextual handit problems with different values of ? We run each model 50 times for each value of y and report the mean and standard deviation.
Method
7 == 0.7
7 = 0.9
? = 0.95
? = 0.99
0.995
? = 0.999
CNP
4.081640.321
8.1512
2.012520.388
26.771+0.791 40.791
38.833
93.2252422.064
BCNP
3.8921+0.50
10.272661.168
10.01 1.07
29.6628
42.2516
95.562443.616
SCNP
3.526640.416
8.29152 525
7.2852=0.268
24.5264 40.619
37,4265
89.825643.669
RCNP
3.1876+0.304
7.3115
7.6796-0.647
23.4416-00647
35.5597
20.665
87.1514
NP
1.562610.125
2.9616110299
4.25131...25
18,2417
10.465
25,5646.00.
62.7342111.003
BNP
62.5115 11.076
57,4966
12.66112100 138
58.22 58.227612.248
58.9122 1:3778
62.5021
15:15
77.5255
SNP
1.4356 1..214
2.4451
10.582
3,4056L0..28
17.2551 10.357
23.5226
1.0 226
60.8862.10.952
RNP
10.133
2.5632 10.3000
3,2367 190195
15.6269 11.329
22.5154
20.61
58.26761111144
ANP
1. 16224610.172
4.0558
5.3983 100.00
19.5712:0..78
27.65
16
73,366 15.956
BANP
5.3.65
15,2415
34.1
1:2.1.77.224.1 1:2..107
08.23144129.06
62.3364=17.0m
69.3436118.300
SANP
1.8878
3.669
100004
4.8767 2:07704
18.25
20.552
25.1169 20.93
71.5169 2..5577
RANP
1.2555
2.6256 6336
3,8987 #:0663
16,5155 $0.641
24.1519 =0 988
70.14144:1.00
ConvNP
1.5035
2.4552:103177
3,1761 20.447
16.3585 +0.358550.08 888
25.235921.25)
72.3526:::.257
BC
2.2563-20225
3.6762:20.97
4.4626.20.551
18.2516::1112
26.3629
74.6324::42277
SConvNP
1.3846
2.008 1:000000 208
2.2662:20:228 226
15.2315 200.90
23.52666 =1446
70.125
10.11251::336554 #:3.54
RConvNP
1.15361:0010
1.7369 240.
1.9854 20.3411
13,2352
19.2516 352
21.11233:22.220
TNP
1.1895
1.7057 410.488
2.5562 200.438
3.5795 4: 1.222
4.6867
9.56944
BTNP
3.3686.0.356
2.7356 +000 022
4.637 #:0662
5.2526 1.048
7.3521 4:3347
11.2657 2:2277
STNP
1.1637 +0.128
1.43622 40.877
2.1526
-0.574
2.36
2 1.522
3.5267
+:1 2277
8.9824 00.678
RTNP
1.2155
1 1.5962 639
1.1371 1 +-0.48
2.08 13 400.89
3.1231
+0.957
7,2355
1.422
044
KGF maree
3.4
Hander SQ Kertel
WNP
1.5
ARP
CAN DRANP
3:40
ZANP SRANP
SAND
SANP
Ro00
RANP
0.30
5030
332
6.35
3.28
033
3.13
(a) CelebA
(b) EMNIST(Seen)
Figure 4. The histograms of task risks in image completion tasks.
Figure 3. Regret performance on 1D Bayesian optimization tasks. For each kernel, we generate 100 functions and report the mean and standard deviation.
problem involves dividing a fixed-sized circle into a low- reward region (the central circle) and four high-reward re- gions (the remaining outer ring divided into four equal parts), control variable y is introduced to manage the proportion between the high and low-reward regions. The higher the 7 value, the larger the central circle and the smaller the outer ring, and vice versa. We conducted experiments with different y, and Table 3 presents the experimental results of all methods. We can see that NPs with our robust solutions outperform all basclines by large margin in most settings, especially for harder settings (larger value of 7). We ob- tained similar experimental results as in (Nguyen & Grover, 2022), where the performance of models using the attention mechanism is generally lower than that of models without the attention mechanism, which is contrary to the results obtained in other experimental tasks.
robust NPs (RANP, RTNP). Note that we use NPs the surrogate functions and Upper Confidence Bound (UCB) as the acquisition function. To maintain consistent com- parison, we standardized the initializations and normalized the results. For each objective function, we run Bayesian optimization for 100 iterations, and simple regret is used as the evaluation metric. As shown in Table and Figure 3, we can see that our robust solutions consistently achieve lower regret than other NPs.
5.3. Contextual Bandits
Following (Riquelme et al., 2018; Nguyen & Grover, 2022), we focus the contextual bandit problem, which is a type of reinforcement learning problem where an agent makes a series of decisions, each with an associated context, to maximize cumulative reward. More detailed experimental settings, including problem definition, training, and testing procedure can be given in the Appendix. Considering this
5.4. Risk Distribution
As we stated previously, the proposed strategy can itera- tively reshape the task risk distribution to increase robust- ness, i.e., transport the probability mass in high-risk regions
8

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
to the lower-risk regions. To verify this point, we present the task risk distributions with and without using our strat- egy. Specifically, taking RANP as an example, Figure 4 shows the task risk distributions of ANP and RANP on im- age completion tasks with two different kernels. We select MSE as the risk measure. Overall, it can be seen that the robust solutions shift the original risk distribution to the left, reducing the proportion of high-risk tasks and effectively demonstrating the robustness of the proposed model.
Science Foundation of Chongqing (Grant CSTB2223NSSQ MSX1020).
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
6. Conclusion and Future Work
References
The paper introduces a novel approach to enhance the ro- bustness of Neural Processes (NPs) in skill transfer learning across different tasks. Traditional NPs focus on empirical risk minimization, which can lead to varying levels of fast adaptation, potentially resulting in catastrophic outcomes in risk-sensitive tasks. To address this problem, we propose incorporating controllable expected tail meta risk, allowing control over the worst-case fast adaptation scenarios at a given probabilistic level. By reformulating the risk mini- mization as a distributional robust formulation, we introduce variance-reduced stochastic mirror prox algorithm with double-loop structures and leverage the two-level structure to get stochastic gradient among samples across all tasks. The proposed risk and optimization strategy is versatile, working with various NPs while providing a less biased approximation.
Alacaoglu, A. and Malitsky, Y. Stochastic variance reduc- tion for variational inequality methods. In Conference on Learning Theory, pp- 778-816. PMLR, 2022.
Ba, J. L.,, Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv: 1eeel11077.00000 06450, 2016.
Beck, A. and Teboulle, M. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 3163):167-175, 2003.
Ben-Tal, A. and Nemirovski, A. Robust solutions of uncer- tain linear programs. Operations research letters, 25(1): 1-13, 1999.
Bertsimas, D. and Sim, M. The price of robustness. Opera- tions research, 52(1):35-53, 2004.
Bertsimas, D., Litvinov, E., Sun, X. A., Zhao, J., and Zheng, T. Adaptive robust optimization for the security con- strained unit commitment problem. IEEE transactions on power systems, 28( (1)::263333) 2012.
common optimization approach for the minimax problem of CVaR involves leveraging two-player game framework. Designing efficient methods to optimize the game between the two players is topic worth exploring. Although our variance-reduced stochastic mirror prox algorithm has been validated for its effectiveness both intuitively and experi- mentally, this kind of algorithm is primarily designed for convex optimization. Models with deep neural networks typically involve high-dimensional, non-convex objectives for which the method lacks general convergence guarantees. One future point is to investigate the algorithms' conver- gence guarantee. Additionally, improving the efficiency of the overall optimization process is another point of interest.
Bruinsma, W. P., Markou, S., Requiema, J., Foong, A. Y., Andersson, T. R., Vaughan, A., Buonomo, A., Hosking, J. S., and Turner, R. E. Autoregressive conditional neural processes. arXiv preprint arXiv:2303. /4468, 2023.
Carmon, Y. and Hausler, D. Distributionally robust opti- mization via ball oracle acceleration. Advances in Neural Information Processing Systems, 15:35866-35877 2022.
Chen, K., Yao. L., Zhang, D., Chang, X., Long, G., and Wang. S. Distributionally robust semi-supervised learning for people-centric sensing. In Proceedings of the AAAI conference on Artificial Intelligence, volume 33, pp. 3321- 3328, 2019.
Acknowledgements
This work was partly supported by the Talent Fund of Bei- jing Jiaotong University (2024XKRC075); Beijing Nat- ural Science Foundation (4244096); The National Natu- ral Science Foundation of China under Grant (62406019, 62436001, 621 The Joint Foundation of the Min- istry of Education for Innovation team 1B092234::3)): The National Key Research and Development Program of China (2024YFE0002900) The State Key Laboratory of Rail Traffic Control and Safety (RCS2023K006); Natural
Chow, Y., Ghavamzadeh, M., Janson, L.,, and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018
Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. Em nist: Extending mnist to handwritten letters. In 2017 Inter- national Joint Conference on Neural Networks (IJCNN), pp. 2921-2926. IEFE, 2017.

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Cotton, R. J., Sinz, F., and Tolias, A. Factorized neural pro- cesses for neural processes: K-shot prediction of neural responses. Advances in Neural Information Processing Systems, 33:1 1388-11379 2020.
Jones, D. R., Schonlau, M., and Welch, W. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455-492, 1998.
Kawano, M., Kumagai, W., Sannai, A., Iwasawa, Y., and Matsuo, Y. Group equivariant conditional neural pro- cesses. arXiv preprint arXiv:2102.08759) 2021.
Curi, S., Levy, K. Y., Jegelka, S., and Krause, A. Adaptive sampling for stochastic risk-averse learning, Advances in Neural Information Processing Systems, 33:1036-1047, 2020.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A.,
Rosenbaum, D., Vinyals, O., and Teh, Y. W. Attentive neural processes. arXiv preprint arXiv:100.05761, 2019.
Driggs, D., Ehrhardt, M. J., and Schönlieb, C.-B. Accelerat- ing variance-reduced stochastic gradient methods. Math- ematical Programming, pp. 1-45, 2022.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv: /3/2.11 14, 2013.
Dutordoir, V., Saul, A., Ghahramani, Z., and Simpson, F. Neural diffusion processes. In International Conference on Machine Learning, pp. 8990-9012. PMLR, 2023.
Lee, J., Lee, Y., Kim, J., Yang, E., Hwang. S. J., and Teh, Y. W. Bootstrapping neural processes. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F,, and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6606-6615, Curran Associates, Inc., 2020.
Fan, Y., Lyu, S., Ying, Y., and Hu, Learning with average top-k loss. Advances in neural information processing systems, 30, 2017.
Foong, A., Bruinsma, W., Gordon, J., Dubois, Y., Requeima, J., and Turner, R. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33: 8284-8299 8295, 2020.
Lin, X., Wu, J., Zhou, C., Pan, S., Cao, Y., and Wang, B. Task-adaptive neural process for user cold-start recom- mendation. In Proceedings of the Web Conference, pp. 1306-1316. 2021.
Liu, H., Jing, L., Yu, D., Zhou, M., and Ng, M. Learning intrinsic and extrinsic intentions for cold-start recommen- dation with neural stochastic processes. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 491 -500, 2022.
Gabrel, V., Murat, C., and Thiele, Recent advances in robust optimization: An overview. European journal of operational research, 235(3):471-483 2014.
Gagne, C. and Dayan, P. Two steps to risk sensitivity. Ad- vances in Neural Information Processing Systems, 34: 22209-22220, 2021.
Liu, H. Jing, L., and Yu, Neural processes with stability, Advances in Neural Information Processing Systems, 36, 2024a.
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, W., Rezende, D., and Eslami, S. A. Conditional neural processes. In Interna- tional Conference on Machine Learning, pp. 1704-1713. PMLR, 2018a.
Liu, H., Zhou, M., Song, M., Ouyang, D.,, Li, Y., Jing L., Yu, J., and Ng, M. K. Learning hierarchical preferences for recommendation with mixture intention neural stochastic processes. IEEE Transactions on Knowledge and Data Engineering, 2024b.
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint arXiv: 180.0000...122., 2018b.
Liu, Z., Luo. P., Wang, X., and Tang. X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Gordon, J., Bruinsma, W. P., Foong, A. Y., Requeima, J.,
Dubois, Y., and Turner, R. E. Convolutional conditional neural processes. arXiv preprint 1910. 2019.
Maraval, A., Zimmer, M., Grosnit, A., and Bou Ammar, H. End-to-end meta-bayesian optimisation with trans- former neural processes. Advances in Neural Information Processing Systems, 36, 2024.
Gotoh, J.-y. and Takeda, A. Cvar minimizations in sup- port vector machines. Financial Signal Processing and Machine Learning, pp. 233-265, 2016.
Nemirovski, A. Prox-method with rate of convergence (I/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1): 229-251, 2004.
Johnson, R. and Zhang, T. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
10

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Nguyen, T. and Grover, A. Transformer neural processes: uncertainty-aware meta learning via aequence modeling. In International Conference Machine Learning, pp. 16569-16594, PMLR, 2022.
Tai, J. Global perception based autoregressive neural pro- cesses. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10487-10497, 2023,
Vadeboncoeur, A., Kazlauskaite, I., Papandreou, Y., Cirak, F., Girolami, M., and Akyildiz, O. D. Random grid neural processes for parametric partial differential equations. In International Conference on Machine Learning, pp. 34759-34778 PMLR, 2023.
Norcliffe, A., Bodnar, C., Day, B., Moss, J., and Liò, Neural ode processes. arXiv preprint arXiv:2103.12413, 2021.
Ogryczak, W. and Tamir, A. Minimizing the sum of the k largest functions in linear time. Information Processing Letters, 85(3):11--122, 2003.
Vaswani, A., Shazeer, N.,, Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Atten- tion is all you need. In Advances in neural information processing systems, pp. 1998-60008 2017.
Pakman, A., Wang, Y., Mitelut, C.,, Lee, J., and Paninski, L. Neural clustering processes. In International Conference on Machine Learning, pp. 7455-7465, PMLR, 2020.
Wang, Q. and Van Hoof, H. Doubly stochastic variational in- ference for neural processes with hierarchical latent vari- ables. In International Conference Machine Learning, pp. 10018- 100088 PMLR, 2020.
Park, Y.-J. and Choi, H.-L. A neural process approach for probabilistic reconstruction of no-data gaps in lunar digi- tal elevation maps. Aerospace Science and Technology, 113:106672, 2021.
Wang. Q., Lv, Y.. Xie. Z., Huang, J., et al. simple yet effective strategy robustify the meta learning paradigm. Advances in Neural Information Processing Systems, 36, 2024.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforcement learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Wei, Y., Zhao, P., and Huang. J. Meta- Meta-learnig hyperpa- rameter performance prediction with neural processes. In International Conference on Machine Learning, pp. 11058-11067. PMLR, 2021.
Quaranta, A. G. and Zaffaroni, A. Robust optimization of conditional value at risk and portfolio selection. Journal of Bankiin & Finance. 32(10):0046-2555 2008
Williamson, R. and Menon, Fairness risk measures. In International conference on machine learning, pp. 6786- 6797. PMLR, 2019.
Rahimian, H. and Mehrotra, Distributionally robust op- timization: A review. arXiv preprint arXiv:1908.05630 2019.
Ye, Z. and Yao, L. Contrastive conditional neural processes. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 9687-9696. 2022.
Rasmussen, C. E. Gaussian processes in machine learn- ing. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Riquelme, C., Tucker, G., and Snoek, J. Deep bayesian bandits showdown: empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations, 2018.
Yildirim, Y. and Ugur, E. Learning social navigation from demonstrations with conditional neural processes. Inter- action Studies, 23(3):427-468, 2022.
Yu, D., Cai, Y., Jiang, W., and Zhang, L. Efficient algorithms for empirical group distributionally robust optimization and beyond. Proceedings of the 4 International Con- ference on Machine Learning, pp. 57384-57414, 2024.
Rockafellar, R. T., Uryasev, S., et al. Optimization of condi- tional value-at-risk. Journal of risk, 2:21-42, 2000.
Sani, A., Lazaric, A., and Munos, R. Risk-aversion in multi-armed bandits. Advances in neural information processing systems, 25, 2012.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R., and Smola, A. Deep sets. arXiv preprint arXiv: 14, 2017.
Schölkopf, B., Smola, J., Williamson, R. C., and Bartlett, P. L. New support vector algorithms. Neural computation, 1207 2000.
Shalev-Shwarte S. and Wexler, Y. Minimizing the maximal loss: How and why. In International Conference on Machine Learning, pp. 791-801. PMLR, 2016.

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
A. Basic Knowledge
A.1. CVaR
we recall standard definitions of Value-at-Risk and Conditional Value-at-Risk which are used in our solution. Definition A.1. (CConditional) Value-at-Risk) Given a random variable X with cumulative distribution function F(-), the Value-at-Risk VaR vo (X) and Conditional Value-at-Riss -iskk CVaR ca (X) at level a E (0, 1) are defined as:
Vol (X) inf : p(X S {) 2 a), and
(16)
co (x) v. (X)
E X U. (X)]+,
where the notation [[] max(0,-) and a is predefined confidence level or risk level.
In machine learning, the concept of Conditional Value at Risk (CVaR) has been widely applied. For instance, the v-SVM algorithm (Schölkopf et al., 2000), can be seen as addressing the minimization of the CVaR of the loss, as analyzed by (Gotoh Takeda, 2016). Additionally, (Shalev-Shwartz & Wexler, 2016) suggests focusing on minimizing the maximum loss across all samples, which is equivalent the limiting case of CVaR when + O. Building on this, (Fan al., 2017) extend the idea to optimize a top-k average loss. Although they do not explicitly connect their criterion to CVaR, their learning strategy aligns with the CVaR framework for empirical measures. For optimization tasks, (Ogryczak & Tamir, 2003) proposed method to minimize the maximum value of a summation over k functions. This approach is closely related to the "truncated" algorithm developed by (Rockafellar et al., 2000) for CVaR optimization. CVaR has since found applications in areas such as risk-averse reinforcement learning, including works by (Sani et al., 2012) and (Chow et al., 2018), and fairness machine learning, as discussed by (Williamson Menon, 2019). Despite its widespread use, the original CVaR optimization approach suffers from challenges such as the high variance of mini-batch gradient estimators (Rockafellar al., 2000). To overcome these limitations, adaptive sampling techniques (Shalev-Shwartz & Wexler, 2016) have been proposed, enabling the efficient handling of scage-ccal datasets and complex models, including deep neural networks (Curi et al., 2020; Wang et al., 2024).
A.2. Neural Processes
The neural process aims to learn stochastic process (random function) mapping target features x, to prediction y/s given the context set DC as training data (a realization from the stochastic process), i.e., learning
logp (y
(17)
Conditional neural process (CNP) (Garnelo et al., 2018a) describes p (yi|x;, DC) with deterministic neural network taking DE to output the parameters of p (ys Xx,, D9). CNP consists of an encoder fene(-), aggregator fagg()) and decoder sdec(-); the encoder summarizes DD and x; into latent representations fr..... Tejl R| via permutation-invarian neural network (Zaheer et al., 2017), where d is the number of latent dimensions, and aggregator summarizes the encoded context features a single representation r and decoder takes as input the aggregated representations re and x; and output the single output--pecific mean μ4 and variance of for the corresponding value of y/-
r fenc (Xi, iEC
c 1
Σ r
C
iEc
(18)
Φ fage (F²)
(Mi, σ2)
p (y: X,, DC) N (vEP,, iET
where Fene (.) and fa are feed-forward neural networks. The decoder output μi; and variance of are predicted mean and variance. We use Gaussian distribution N (y) 14, of) as predictive distribution. CNP is trained to maximize the expected likelihood EPCT)(P (yilx,, DC)).
12

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Neural process (Garnelo al., 2018b) further models functional uncertainty using a global latent variable. Unlike CNP, which maps context into deterministic representation F;, NP encoders context into Gaussian latent variable 2, giving additional stochasticity in function construction. Following (Kim et al., 2019), we consider NP with both deterministic path and latent path, where the deterministic path models the overall skeleton of the function F,, and the latent path models the functional uncertainty:
Send
(xi, Wi),
i iEC
re
ri
C
IEC
Sags (r)
(19)
(1², 0²)) Send (DC),
(z( (zi Mb 022)
(14, 0.) = Sdec(d, (0, 2, xi),
(v/|x, DC) = N (Ui iET
with (-) and (-) having the same structure feno (-) in (18) In this scenario, the conditional distribution is lower bounded as:
(x(e)) log ( z, DC) P(z[DC)
logp (y|X, DC)
E
(20)
yz)(X.y)
We further approximate q( D = p(z|D D and train the model by maximizing this expected lower bound over tasks. Furthermore, ANP introduces attention mechanisms into NP to resolve the issue of under-fitting.
B. Theoretical Investigations
Proposition B.1. (Curi al., 2020) Let h X + y be a finite function class (H]. Let E(h) H 10, 1] be random variable. Then, for any 0 < a < 1, with probability at least 1 - 5,
Ra(s, 0) - RR, 0 $ 1-a
1
log (2g(2
E
sup LEH
(21)
MN
To deeply analyze the Bregman setup, we give the following assumptions. Assumption B.2. (Boundedness of the Domain) The diameter of the domain θ under the distance-generating function pb(-)) is bounded by constant Dg, i.e.
max (0) min (0) s Dj. lEB
(22)
AEG
Similarly, the simplex Qa is assumed to be bounded by Dg. Since the entropy function viy is used, we have Dg / In M. Assumption B.3. (SSmoothness and Lipschitz Continuity)) For any sample from meta-dataset (dataset of datasets) D,,1:M (Drmm)Ma the loss function e.. (0) is assumed to satisfy L-smoothness and G-Lipschitz continuity.
For the task-specific stochastic gradient defined in Eq. (12), we have the following theorem to indicate its unbiasedness. Theorem B.4. The stochastic gradient
ΣM 9hh V. (2 (Zr 0*) [ 8²),,
ZZ 0" Fa
V F (Zi (Zi 0 9²)
(23)
Ze fm.m 8 )0)
lu kmm B8)]
defined in Eq. (12) is unbiased.
Proof. For the sampling strategy defined in Section 4.3: 1) for each dataset, uniformly sampling one sample: Z Ulttm (7mm))))) Vm [M]; 2) forming the task-specific samples as {Zg IM m== have
((0.g) q) 1 Qa Elem (Zim: m 0) (m),)), E[Vem (Z 0) Rm (0).
(24)
13

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Based the additivity of expectation, we can get
V(0. q) O Qa: [[
(25)
In this case, can be proved that the stochastic gradient is unbiased.
We assume that m mz 8) is L-smooth and G-Lipschitz contimuous for all m [M], k E [nm]] The following theorem analyzes the Lipschitz continuity of the stochastic gradient V (Z 0² g²).
Theorem B.5. For any 8 E [S]), k E [K,]% VZ 0+, ,0¹) is LL-Lipschitz continuous, where L. 2D, max G
Proof. Randomly selecting two parameters (0+, (7) and (07, ,q ,9)), we can bound the gradient of as follows:
[VaFa 0+ ,qt) - VaFa(Zz 0 .q 11ba,
M
Σ
Am
Vem m )*) - (Z 0 00
Σ
(9m 9m
V
Vem (Z Rmmm 8-)
m=1
m=1
1,,
<
Σ
4m
em ( 0+)
(Zk,mi vm 0 0-]
Vm
2
(gtte 4m Vo (2 0-)
m=1
0,+
m==
0,x
(26)
M
M
<
Im
Vem Ri 0+) - vem Z 0
+2
Im -mml Vem (Z 0 )1
m=1
m==1
Im
T.
1.2
-1 +
Σ
Im
Im |G
m=1
=2
0- 2G 22221la* - 11.
We also bound the gradient of q follows:
Vm [M]
( 0-) lm (i GilB 0 1l0-
(27)
Given 00 9²)) - ( .es (Zx we have
FagFalZE.m ,1,,tt)= -VgFalZa 0 00 11,
1m 7 k.mi 0++) em (7: 17 )] )]²
max
(28)
me MEIM
<G2 0 10.
Based on the above bound, we have
0+ .gt)= VF Zi,m 0 .9 ) -2D 0+ .4) FoF,(Z;, mi 0
112,
+ M Fa ++ (ttttt V VV kimi 0 00. 99)
1. + 20 In M) 0- 1/ 4D
14 + 1
a 1
[[AD)].2 + G2 In M) - 1
22
In -
2DDg
2 M
12
1
< 2D
110 - 1-
1a - 11
2ln M
x (0T .".9" .g))
(29)
14

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
C. Model Architecture
The architectural details of the CNP, NP, and ANP are the same as (Kim et al., 2019). Here we give the detailed architectures of the encoder and decoder of NPs.
C.1. Encoder without attention
Encoder focuses on learning embeddings for each data point in the context set, and the basic component is multi-layer perceptron, which defined by
MLP(I, din: ths doue) LINEAR (dh, dout) (RELU LINEAR dr) LINEAR den)
(30)
x(l-2)
where ( is the number of layers, din, dp and dout are dimensinalities of inputs, hidden unites and outputs. Here RELU(-) is adapted as activation function.
The encoder in Vanilla CNP uses a deterministic encoder that focuses on learning embeddings for each data point in the context set.
r. MLP((.1, d, + dy. dn. dn (1x;, vil),
Σn ri
Φ ML.((12. dn. dr) (rC)
(31)
IEC
where d, and dy are the dimensionalities of x; and yi-
To follow the encoder structure in NP, we introduce another encoder aligned with the original deterministic encoder permit the same number of parameters, i.e.,
r MLP(1e1, du + dy. dn, dn ((x,, 34l)
(1) Σ
Φ1 MLP(Ie2, dr dr) (r( (1)
IEC
r (2) MLP(Ie1, de dy dn, dn (n)(((x)))))))))))))1) Mil)
(32)
(2) (2)
Φ MLP(Ie2, dr, dr (1 (r(
IE
Φ (d), d2]
The encoder in NP contains deterministic path and latent path, i.e.,
r MLP(Hde1: de + dy. dn, dn (1x,, Hil)
(1)
Σ
(1) r o = MIP(Pde2 dp. dr (r(() (1))
IEC
(2)
MLP(Haa de + dy. dn, dn (x)(
(33)
r
(
Σ
(2)
[M2 MLP(H dn dn (r (+)
c
r
0, = 0.1 + 0.9 SIGMOID(O)), z ~~ A (1 diag ( ()))
In this case, the encoder outputs deterministic representation & and latent representation
15

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
C.2. Encoder with attention
The attention mechanism is widely used in NPs, Specifically, multi-head attention (Vaswani et al., 2017) adapted, which is defined by
Q' [LINEAR L dow (a)a)))))
{Q) 1 Phead
SPLIT(Q Q (Thead)
K' (LINEAR (dk,, dout ) KEK.
K
Thead [= f=
SPLIT(K' (heaad)
V {LINEAR(cL, du dowt) (v) Ivev,
(V/) Pheed ifi=l
= SPLIT(V' (Whead) )
(34)
H, SOFTMAX (((K)) /v dout) V
H = CONCAT ((H,) Pheadd Ji=l
H' = -LYERNORMM H)
MHA(dow:) (Q, V) = LAYERNORM(H'' RELU (dout dout)))
where dy. d,. dg are the dimensionalities of query Q, key K. and value V, respectively. Thead is the number of head. Here Layer normalization (Ba et al., 2016) LAYERNORM() is adapted. is easy to derive self-attention by setting Q = = V, i.e.
SA(dous)) = (InA(do (X,
(35)
For CNP, the encoder with attention still contains two deterministic paths,
fgx = MLP(Ly de, da, dn)
Q = fax(x), iET
(Sa((x)),, IEC
V (A(LL)) de day dr, (Ix wilx<,,i))1 vil))
Φ1 MHA(dh)(,,K,
(36)
H = SA(da) ((RELU MIP((le1, dx + dy, dn, du (xx;,
( h,
2 MLP(Ie, dh, dr)
(
SEC
Φ [01, d2]
Similarly, an encoder with attention NP contains a deterministic path and latent path, i.c.,
Sa MLP(IyRs da, dn, dr.
Q fax(xi), iET
K (fgM(xi)), iEC
(37)
V SA((d) de + dy. du, (x)), (.)ee)))eee)))
Φ (.A
and
H SA(d)) ((RELU MLP(le1, de + dy, dn dn) MEE)
1
1
Uu o'] MLP((ia, da dr.)
Σ
h;
C
(38)
a. = 0.1 + 0.9 SIGMOID(O').
z ~ N(,z diag(o?)).
16

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
C.3. Decoder
The decoder focuses on predicting output for target points based on the encoder's outputs Φ. For target point (XIJIET the decoder of CNP is defined by
[nc o.] = MLP(ddacs 2d, du du, 2d,] iET = 0.1 + 0.9 SOFTPLUS(O!)
(39)
N,)
Decoder NP defined by
[nc, .o. = MLP(daaes + d2 + das dn, 2d,)o. iET 0.1 0.9. SOFTPLUS(O.) y: O.),
(40)
D. Implementation Details and Experiments
D.I. Infrastructure
We implement our model with Pytorch, and conduct our experiments with:
CPU: Intel Xeon Silver 4316.
GPU: 8x GeForce RTX 4090.
RAM: DDR4 384GB.
ROM: 16TB 7.2K 6Gb SATA and 1x 960G SATA 6Gb R SSD.
Operating system: Ubuntu 18.04 LTS.
Environments: Python 3.7; NumPy 1.18.1; SciPy 1.2.1; scikit-learn 0.23.2; seabornn 0.1; torch geometric 1.6.1;
matplotlib 3.1.3; dgl 0.4.2; pytorch 1.6.
D.2. Experimental Settings
For CNP (Garnelo et al., 201 8a), BCNP (Lee et al., 2020), SCNP (Liu et al., 2024a), and our RCNP, we apply the encoder with attention described in Eq (36) and decoder described in Eq (39). For NP (Garnelo et al., 2018b), ANP (Kim et al., 2019), BNP (Lee et al., 2020), BANP (Lee et al., 2020) and our SNP and SANP models, we apply encoder with attention described in Eq (37) and (38), and decoder described in Eq (40).
D.2.1. 1D REGRESSION
For synthetic 1D) regression experiments, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and SCNP/SNP/SANP refer Appendix C. The number of hidden units is dr = 128 and latent representation d, 128. The number of layers are 4, 1.0- 4/a = Lyb = L, =
We generate datasets for synthetic 1D regression. Specifically, the stochastic process (SP) initializes with a mean Gaussian Process (GP) (0) ~ GP k(.,) ))) indexed in the interval æ E [-2.0, 2.0], where the radial basis function kernel k(x, x = o² exp(-11ar 2 /2(2) with ~ U0.1,, 1.0) and UO.. 0.6). Furthermore, GP with Matern Kernel adopted for model-data mismatch scenario, which is defined k(æ, x²) = o2 + v5d/l + 5d2 /(31)) exp(-v5d/1) and d = || z'| with 10..1, 1.0) and UO..1, 0.6.. For fair comparison, set the same data generation, training, and testing for all models.
We trained all models for 100, 000 steps with each step computing updates with batch containing 100 tasks. We used the Adam optimizer with an initial learning rate 5 10 and decayed the learning rate using Cosine annealing scheme for baselines. For SCN,, we set K 3. The size of the context was drawn as |C| U(3,200) Testings were done for 3, 000 batches with each batch containing 16 tasks (48, 000 tasks in total).
17

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
D.2.2. IMAGE COMPLETION
Analogous to the ID experiments, we take random pixels of given image at training as targets, and select a subset of this as contexts, again choosing the number of contexts and targets randomly (n U[3, 200], m + U(0, 200 nl). The x; rescaled to |-, and the y4 rescaled to [-0.5, 0.5]. We use batch size of 16 for both EMNIST and CelebA, i.c. use 16 randomly selected images for each batch.
For image completion experiments on EMNIST and CelebA dataset, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP, SNP, and SANP refer to Appendix C. The number of hidden unites is dn, = 128 and latent representation d= 128. The number of layers are le Ide 4, lta- = lyk = ly 5. hhead = 8
D.2.3. BAYESIAN OPTIMIZATION
We sampled 100 GP prior functions from mean and unit variance. After realizing them, the prior functions used to optimize via Bayesian optimization. We normalized these functions in order to fairly compare simple regrets and cumulative regrets across distinct sampled functions (Basically, since they are sampled from the same distributions, the scales of them are quite similar, but we used more precise evaluations).
As presented in the Bayesian optimization results, all the methods are started from the same initializations. We employed Gaussian process regression (Rasmussen, 2003) with squared exponential kernels as a surrogate model, and expected improvement (Jones al., 1998) as an acquisition function, which is optimized by the multi-started local optimization method, L-BFGS-B with 100 initial points.
D.2.4. CONXTUAL BANDITS
The whell bandit problem is introduced in (Riquelme et al., 2018) and can be illustrated in Figure 5. In this problem, a unit circle is divided into a low-reward region (blue area) and four high-reward regions (the other four colored areas). A scalar determines the size of the low-reward region, and other regions have equal sizes. The agent does not know the underlying Y and has choose among
arms given its coordinates X (X1, X2) within the circle. If || S 7, the agent falls within the low-reward region (blue). In this case, the optimal action is k = which provides reward N (1.0, 0.012). If the agent falls within any of the four high-reward regions (II > 7). the optimal arm will be one of the remaining four & = 2 - 5, depending on the specific area. Pulling the optimal arm here results in a high reward r ~ N (50.0, 0.012), and as before all other arms receive N (1.00, 0.012) except for arm which always returns (1.22, 0.012).
(1.22, 0.012),
while all other actions only return
We sample a dataset of different wheel problems K which are drawn from a uniform distribution y ~ U0, 1). For each problem, we sample N points evaluate and pick m points as context, in which each point is a tuple (X,r) of the coordinates X and the corresponding reward values of all 5 arms. The training objective is to regress the reward values from the coordinates. We set K 8, N 562, m 512 in experiments.
E. Additional Experiments
E.1. 1D Regression
Following (Kim et al., 2019; Lee et al., 2020; Liu et al., 2024a), we first conduct ID regression experiments. We generate training data from a GP with RBF kernels and test the trained model on the data generated from GPs with different kernels, including RBF, Matérn, and Periodic. Among them, data from Matérn and Periodic are used evaluate model performance on unseen functions. We give detailed description of experimental settings in the Appendix. Table 4 lists the average log- ligeliiel comparison in terms of different kernels. The best results are marked in bold. After several repetitive observations, we can observe that our robust version outperforms not only vanilla ones but also bootstrapping versions and stable versions. For model-data mismatch settings, all the models become less accurate, but our robust ones affect less. Visualizations for ID regression data are shown in Figure Each solid blue curve represents one sample function and the blue area around each curve represents the variance the predictive distribution. We can see that NPs with bootstraping, stable, and robust solutions generally achieve better fitting performance, especially data points where other methods fail to fit effectively.
18

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
0
0
-1
1
-1
1
-1
-1
-1
1
-1
Figure The wheel bandit problem with varying values y. (Riquelme al., 2018; Nguyen & Grover, 2022).
Table 4. Comparison of our robust NPs with the baselines on -glliielihood of the target points on various GP kernels. We train each method different seeds report the mean standard deviation.
Method
RBF
Matérn 5/2
Periodic
CNP
0.4334
0.2431
10.010 10.010
1.2521 10.008
BCNP
0.4589+0..0 006
0.2762+0.009
1.1915
SCNP
0.4733
0.2953
- -1.1779 -1.177940.000
RCNP
0.4798.1.003
0.3021 10.005
-1.1699
NP
0.385311.000
0.2041 10.015
-1.4255
BNP
0.4211
0.268940.000
-1.3988
SNP
0.435640.000
0.2844 10.005
-1.408510.008
RNP
0.4398 000002
0.2832+0.000
-1.3761
ANP
0.5763
0.636640.004
-1.1824400.00
BANP
0.5887 1.0008
0.65 4..000
-1.178110.006
SANP
0.5994 0044
0.665340.004
-1.1774
RANP
0.6025 0.00
0.6771
-1.177940.000
ConvNP
0.6503 0044
0.6557
-1.1761+0.000
BConyNP
0.6531 005
0.6787
-1.1675:40.00
SCOnvNP
0.6831
0.683640.004
-1.1681 000000
RConvNP
0.6898 0-0003
0.69164.0.044
-1.1547 10.004
TNP
0.8711 10.003
0.7151 10.001
-1.1625.1..00
BTNP
0.8981 +-0.006
0.725540.003
-1.159 140.00
STNP
0.8798 16799-10.000
0.7151 400006
-1.1552
RTNP
0.8815 .044
0.7389 +0.003
-1.1511+0 +0.004
19

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
CNP
BCNP
SCNP
RCNP
RBF
43
Matern 5/2
Periodic
ANP
BANP
SANP
RANP
RBF
4.00
+00
Matern 5/2
Periodic
Figure 6. Visualizations of CNP, ANP, and their bootstrap versions (BCNP, BANP), stable versions (SNP, SANP), and robust versions (RCNP, RANP) for ID regression data.
20

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
RBF Kamel
Matóm 5/2 Kamel
Periodic Kema!
0.42
0.43
0.40
CNP
CNP
CNP
0.33
BCNP
0.33
BCNP
0.39
BCNP
SCNP RCNP
SCNP RCNP
SCNP
0.30
0.30
0.39
0.20
0.23
0.20
0.10
0.13
0.10
0.00
0.00
0.00
ltarations
Barations
Earations
RBF Kernel
Matém Kernel
Periodic Kernel
0.35
ANP
0.35
ANP
0.35
ANP
BANP
BANP
BANP
SANP
SANP
SANP
RANP
0.23
0.23
0.29
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.03
0.05
0.92
0.93
0.00
Figure 7. Regret performance on ID Bayesian optimization tasks. For each kernel, we generate 100 functions and report the mean and standard deviation.
E.2. Bayesian Optimization
Taking GP data with RBF, Matérn 5/2, and periodic prior functions examples, gave the results of CNP, ANP, corresponding bootstrapping NPs (BCNP, BANP), stable NPs (SCNP, SANP), and robust NPs (RCNP, RANP). Note that use NPs as the surrogate functions and Upper Confidence Bound (UCB) as the acquisition function. To maintain consistent comparison, we standardized the initializations and normalized the results. For each objective function, we run Bayesian optimization for 100 iterations, and simple regret is used as the evaluation metric. As shown in Figure 7 and Table 5, we can see that our robust solutions consistently achieve lower regret than other NPs all three kernels.
E.3. The effect of confidence level a
The key parameter in our robust solution is the confidence level a, which controls the extent to which the learning model focuses on difficult tasks. Taking RANP as an example, we investigate the average log-likelihood terms of different on the ID regression, image completion, and Bayesian optimization, as shown in Figure 8. We can see that RANP obtains the best results (around 0.6), after that, the performance remained stable. This result is in line with our intuition. there too much focus difficult tasks, it may neglect the learning of general tasks. On the other hand, insufficient focus on difficult tasks can also affect the model's performance.
E.4. Risk Distribution
As stated previously, the proposed strategy iteratively reshape the task risk distribution increase robustness, i.e., transport the probability mass in high-risk regions to the lower-risk regions. To verify this point, we present the task risk distributions with and without using our strategy. Specifically, taking RANP as an example, Figure 9 shows the task risk distributions of ANP and RANP on, 1d regression, image completion, and Bayesian optimization tasks. We select MSE as the risk measure. Overall, can be seen that the robust solutions shift the original risk distribution to the left, reducing the proportion of high-risk tasks and effectively demonstrating the robustness the proposed model.
21

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Table 5. Bayesian optimization experiments data generated by different GP kernels
Method
RBF
Matérn 5/2
Periodic
CNP
0.1524
0.185210.004
0.2051 10.005
BCNP
0.1501
0.1815+0.000 0.03
0.201640.003
SCNP
0.1415
0.1752:+0.000
0. 0.1998
RCNP
0.1352 003
0.1711 +11111+0.00
0.1941+0.001
NP
0.1647=0.000
0. 0.1988+0.000
0. 0.1985+0.0077
BNP
0.161 +0.003
0.1901 100000
0.1962+u.003
SNP
0.1536 10.004
0.1871 100.003
1.191511.000
RNP
0.1415 10.003
0.1789 1:0.005
0.1873 0.18731..00 1.0.03
ANP
0.1245 00.03
0.1518 000000
0.1892
BANP
0.1341 10.003
0.1316
0.1788 1.0.005
SANP
0.1142 002
0.1201
0.16724+0001
RANP
0.1025 400002
0.1171 +00.003
0.177940.006
ConvNP
0.1211 10.004
0.1256.100003
0. 0.1854
BCONvNP
0.1187 +0.005 003
0. 0.1201 +0.004
0.1819+0.003
SConvNP
0.1176 006
0. 0.1179 11119990.003
0.1786:+0.004
RConvNP
0.1056 004
0.1156+ 0004
0.1773
TNP
0.1125 003
0.1451 +0.001
0.1715+0.000
BTNP
0.1037 +003700.000 006
0.1455
0.1691 +0.008
STNP
0.0998 +0.0040
0.1351 10.000
0.1561 1.0.002
RTNP
0.0915
0.1289 +0.000
0.1463
3.05
0.67
1.199
0.5
0.665
0.56
0.50
1-186
1:184
5566
3.68
0.845
0.64
1.10
0575
0.635
1.174
1.177
611
6.6
0.
Confidence
(a) RBF
(b) Matérn 5/2
(c) Periodic
0.858
2.98
0.988
2.94
2.90
2.52
0.566
0.564
0.895
0.896
2.88
0.893
2.34
3.86
0.976
0.574
0.892
2.82
0.572
0.4 Conficinnoo lavel
0.2
0.5
0.9
Confidence leval
Confidence level
(d) CelebA
(c) EMNIST(Seen)
(f) EMNIST)
0..1
0.124
0.184
0.115
0.122
0.182
0.105
0.118
0.18
0.116
0.178
0.095
0.176
0.09
0.112
0.174
0.2
0.6
0.9
0.1
0.2
0.4
0.6
0.8
0.9
0.2
0.8
Confidence lecel
Confistence lecwl
BO(RBF)
(h) BO 5/2)
BO(Periodic)
Figure 8. The performance evaluation when we set the confidence level a with different values.
22

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
ANP. & RRANP
0.6
BRANN
0.5
PANP RRAN
0.4
0.4
0.4
0.2
0.2
0.2
8
RL
1.0
1.0
2.0
1.0
(a) RBF
(b) Matérn 5/2
(c) Periodic
0.45
EANP GRANP
0.45
SANP RANN
RANP
0.30
0.15
0.15
& 1A.
30-
0.5
1.0
1.5
2.0
(d) CelehA
(e) (MNIST((Senn)
(I) EMNIST Unseen)
GANP RRNN
0.5
0.6
BANP 2 RANP
NRANP
0.4
0.2
(g) BO(RBF)
(h) BO 5/2)
(1) BOI Periodic)
Figure 9, The histograms of task risks in different tasks.
23

---

Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
Table Comparison of our robust NPs under different weight strategies on aggliikllhood of the target points on various GP kernels. We train each method with 5 different seeds and report the mean and standard deviation.
RBF
Matérn 5/2
Periodic
Method
average
adam
average
adam
average
adam
RCNP
0.4756_1.002
0.4798 10.003
0.2989 10.003
0.3021 1..005
-1.1751
10.005
-1.1699 10.006
RNP
0.4359: 1000033
0.4398 +0.000
0.2797
+0,003
0.283210.006
-1.3895
20.007
-1.376110.009
RANP
0.5989 10.004
1.6025-10.002
0.6688 10.005
0.6771 1:0.066
-1.1797
10.007
-1.1779
RConvNP
0.6857 10.004
0.6898
10.003
0.6871
1.0004
0.691611.000
-1.1699 10.008
-1.1547 1.0004
RTNP
0.8796::0.005
0.8815
0. 0.7276:0.004
0.7389
-1.1544
+1.1511+00000
Table 7. Comparison of our robust NPs under different weight strategies on log-likelihood of the target points on two real-world datasets: CelebA and EMNIST. We train each method with different seeds and report the mean and standard deviation. EMNIST(Seen (0-9))
CelebA
EMNIST(Unseen (10-46))
Method
average
adam
average
adam
average
adam
RCNP
2.1735 20.007
2. 2.1896_10.005
0.7845 1.0005
0.7879 10.005
0.5321 1.0005
0.5362.10.008
RNP
2.887 10.71100.00
2.891540.006
0.887640.00 0.0
1.8911+0000 +0.00
0.7101
0.7121
RANP
2.9651
20.003
2.9718 00000
0.9825
10.006
0.9881 1 100000
0.89154u.007
0.8967::0.005
RConvNP
3.2215
=0.015 20.004
3.231940.000
1.2251
10.006
1.23484100005
1.0663
1.0721 10.008
RTNP
4.4189 10.003
4.4226
10.005
1.5541
10.003
1.5572.0.002
1.4361 10.005
1.44131..000
E.5. The effect of wj
In the gradient computation process of the outer loop, the weighting coefficient for the gradient from the previous step typically determined using an average weighting strategy, i.e., w - -__. Vk 1,... K.- However, we can set it as a learning rate vig as shown in Eq. (15). We compared the performance of the robust model using the average strategy and adaptive learning rates, using three tasks as examples. The results are shown in Table 6, and 8. It can be observed that the experimental results with the adaptive learning rate (Adam) outperformed the average strategy, while the average strategy performed better than other bootstrap strategies or the stable strategy.
Table 8. Comparison of our robust NPs under different weight strategics on Bayesian optimization experiments with different GP kernels. We train each method with 5 different seeds and report the mean and standard deviation.
RBF
Matérn 5/2
Periodic
Method
average
adam
average
adam
average
adam
RCNP
0.1396440.003
0.1352
0.1753 L0.003
0.1711
0.1985 40.005
0.1941 10.004
RNP
0.148911. 10.004
0.1415 1..015 10.003
0. 081877 1..80000.00 10.003
0.
10.005
0.1908
10.004
0.1873
10.003
RANP
0.1097 +00003
0.1025 ..1025.0.000
0. 0.1204-0.00x
0.1171
00.00
0.1798
0.1779 +00006
RConvNP
0.1128 10.005
0.1056 0..004
0. L0.004
0.1156
00004
0.1793
10.006
0.1773 10.003
RTNP
0.0973.10.00
0.091510.004
0. 0.1327 0..027 10.005
0.1289
0.009
0.1517
0.1463
24


Reviews:
[{"review_id": "VvYPavbtbf", "reviewer": "Reviewer_x3vk", "summary": "This paper introduces a new framework for improving the robustness of Neural Processes. Traditional NPs optimize for average performance across tasks using empirical risk minimization, but this can lead to poor adaptation on difficult or high-risk tasks. \nThe authors propose a risk-averse optimization strategy based on Conditional Value-at-Risk (CVaR), which shifts focus toward minimizing the loss in the worst-performing fraction of tasks (top % highest-risk tasks).\n\nTo make CVaR optimization tractable for NPs, the authors reformulate the objective as a finite-sum minimax problem and solve it using a variance-reduced stochastic mirror prox algorithm. This method uses a double-loop structure, where an outer loop computes stable \"snapshot\" gradients and an inner loop performs refined updates using stochastic, variance-reduced gradients across tasks. \n\nThe authors evaluate their method across multiple domains—image completion (CelebA, EMNIST), Bayesian optimization, and contextual bandits; comparing robust NPs (RNPs) against standard, bootstrapped, and stabilized versions of NPs. Across all benchmarks, RNPs consistently demonstrate improved robustness, particularly under data distribution shifts, model-data mismatch, and adversarial task setups. The method effectively reshapes the task risk distribution, reducing the frequency of high-risk task failures.\n\nOverall, the paper contributes a theoretically grounded, practically effective approach for training Neural Processes to be more reliable under task uncertainty and variability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and valuable feedback. We have made efforts to address your concerns. If there are any further questions, please let us know, and we will reply promptly.\n\n**Q1.**   **Ablation experiments on CVaR α, inner/outer loop iteration lengths, and learning rate.** \n\n> **Reply**: We appreciate the need for more thorough ablations. Due to space limits, we included a subset of these in Appendix E.3, focusing on α.  Empirically, we have observed that α values in a moderate range (e.g., 0.4–0.6) often strike a good balance between focusing on worst-case tasks and not overly biasing the model toward rare outliers. We show an ablation (Figure 8) demonstrating that too small an α (e.g., 0.1) under-treats high-risk tasks, while an extreme α (e.g., >0.8) can degrade average performance. These observations align intuitively with the idea that CVaR reweights the “tail” portion of tasks. \n>\n> In the paper, $S$ denotes the number of outer loops (epochs), and $K_s$ represents the number of inner-loop iterations within each outer-loop $s$. In experiments, we set $K_s$ as a constant across all outer loops, i.e., $K_s = K$ for all $s$. $K$ is chosen to scale with the average number of samples per task. Empirically, we find that $K_s = 100-500$ often suffice for stable training in NP tasks; deeper loops help slightly but yield diminishing returns relative to the added cost. For outer loop epochs $S$, usually between 50 to 500 epochs. For simple tasks (e.g., low-dimensional regression, small datasets),  $S$ is 50–200 epochs.  For complex tasks (e.g., image completion),  $S$ often requires 200–500+ epochs. We will expand them to also show the effect of varying the number of inner/outer loop iterations in a revised version. \n>\n>We used Adam optimizer with an initial learning rate $5·10^{-4}$ and decayed the learning rate using a cosine annealing scheme.\n\n**Q2.**   **How does the double-loop optimization compare to simpler baselines in computational overhead?** \n\n> **Reply**: The additional overhead mainly comes from (i) maintaining snapshots of gradients and (ii) sampling from each task in each inner iteration. However, because we adopt a task-aware sampling scheme (i.e., one sample per task), the minibatch size remains comparable to that of standard NP training.\n> In practice, we find the extra overhead to be modest—roughly 1.1–1.3× the per-epoch training time compared to standard NPs, depending on the number of tasks and the length of inner loops.\n> We will add more quantitative details in the final version, including a time-per-epoch comparison and possible strategies (e.g., smaller inner loops) to reduce total training time.\n\n**Q3.**   **Why does improving tail performance not degrade average accuracy?** \n\n> **Reply**: Elevated attention to difficult tasks prevents overfitting to “easy” samples and encourages more robust parameter updates. This can reduce overall variance in predictions and in practice, often benefits average-case accuracy. Our experiments (Tables 1, 2, 3) demonstrate that while we significantly reduce high-risk failure rates, the average performance (e.g., log-likelihood) either remains on par or improves slightly compared to standard ERM-based NPs. CVaR reweights tasks in proportion to their risk. Importantly, for moderately chosen α, we are not only training on the worst tasks but rather adjusting the objective so that the upper tail influences the gradient more strongly. The result is a more balanced training signal across all tasks.\n\n**Q4.**   **The paper largely focuses on convex analysis, but deep NPs are inherently non-convex.** \n\n> **Reply**: We agree that in non-convex regimes, strong theoretical convergence guarantees are elusive (an issue for many deep learning algorithms). While mirror-prox methods do enjoy strong convergence rates for convex-concave saddle-point problems, we currently rely on their practical effectiveness for non-convex NPs.\n> We have followed prior work in robust optimization, which often leverages the same theoretical tools in non-convex settings with the understanding that local minima or stable solutions can often suffice in practice. In the final version, we will highlight this limitation more explicitly.\n\nLastly, thank you once again for your valuable comments.", "claims_and_evidence": "Most of the paper's key contributions — particularly the need for CVaR-based optimization, the effectiveness of the proposed algorithm, and performance improvements in multiple domains — are supported with clear and convincing evidence. \n\nHowever, a few theoretical claims would benefit from stronger empirical or quantitative backing:\n - generality: experiments only evaluate a fixed set of standard NP architectures (CNP, ANP, ConvNP, etc.). It’s unclear how well it transfers to significantly different variants or tasks not tested. This is potentially true. \n-  bias reduction: This is stated theoretically but not quantitatively evaluated in experiments (no analysis of bias vs. variance trade-off or formal bias reduction metrics).", "methods_and_evaluation_criteria": "Yes.the proposed methods and evaluation criteria are generally well-aligned with the problem of improving robustness in Neural Processes under task variability and risk", "theoretical_claims": "The paper does not contain unproven theoretical claims, and where limitations exist (e.g., convergence in non-convex regimes), they are appropriately acknowledged. No significant correctness issues were identified in the stated proofs or derivations.", "experimental_designs_or_analyses": "Minor issues:\n---\n- Hyperparameter sensitivity: no detailed discussion of sensitivity to CVaR level α, learning rates, or inner/outer loop lengths — which could affect robustness.\n\n- Runtime: no analysis of training time, convergence speed, or compute overhead from the double-loop structure, which would help assess trade-offs.", "relation_to_broader_scientific_literature": "- NPs and Meta-Learning: the paper extends NPs by addressing a critical limitation — their reliance on empirical risk minimization, which can lead to poor performance on outlier tasks. Unlike existing enhancements (e.g., bootstrapped or stabilized NPs), the paper introduces explicit risk-sensitive objectives.\n\n- Risk-averse and robust optimization: the paper is the first to apply CVaR-based risk minimization in NPs, treating task-level adaptation risk as a distribution and focusing on controlling the worst-performing fraction.\n\n- mirror prox and variance reduction: the paper Innovatively adapts these optimization methods to the meta-learning risk-averse process.", "essential_references_not_discussed": "Citations are well written.", "other_strengths_and_weaknesses": "Strengths:\n---\n- Originality: While the core components (CVaR, Mirror Prox, variance reduction) are established individually, their integration into the Neural Process framework is novel and well-motivated. \n\n- Significance: The proposed method addresses a real and underexplored challenge; robustness to high-risk task failure in few-shot and meta-learning settings. Given the increasing deployment of meta-learning models in uncertain environments (e.g recommendation or robotics), this contribution is both timely and impactful.\n\n- Clarity of technical sections: The paper does a solid job presenting technically dense ideas (e.g., minimax reformulation, Bregman setups) in a logically structured way. The supplementary material is especially helpful, providing detailed derivations and implementation notes.\n\n- Empirical breadth: The experiments span generative modeling, black-box optimization, and decision-making ; showcasing the generality of the approach. The consistent performance improvements across these settings strengthen the empirical case.\n\nWeaknesses:\n---\n- Clarity for broader audiences: While technically sound, some sections (e.g., optimization formulation and variance reduction) are dense and could benefit from more intuitive explanations or diagrams in the main text.\n\n- Ablation/Interpretability: There is limited ablation on the effect of CVaR level α, inner/outer loop iterations, or mirror prox updates. Understanding the sensitivity of the model to these choices would clarify its practical utility.\n\n- Efficiency analysis missing: The method introduces nontrivial computational overhead (double-loop structure, per-task sampling), but no runtime or convergence speed comparisons are provided. This could be critical for real-world deployment.\n\n- Limited theoretical guarantees: While the optimization method is principled, the paper lacks convergence guarantees in non-convex regimes — a common but notable limitation that could be acknowledged more explicitly.\n\n\nOverall:\n--- \nThe paper presents a significant and novel contribution by adapting risk-averse learning to NPs, backed by thoughtful algorithm design and broad empirical validation.", "other_comments_or_suggestions": "typos\n---\nsection 3.1: fundom -> random", "questions_for_authors": "Already addressed in the weaknesses section:\n\n- How sensitive is the performance to the CVaR level α?\n- What is the computational overhead of the double-loop optimization compared to standard ERM-trained NPs?\n\nAlso,\n- It’s not fully explained why improving tail performance doesn't degrade average-case accuracy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "78gTlpOgJh", "reviewer": "Reviewer_abBS", "summary": "This paper investigates the robust neural processes problem from a risk-averse perspective, aiming to control the expected tail risk at a given probabilistic level. The authors formulate the CVaR optimization as a distributionally robust optimization (DRO) problem and propose a double-loop stochastic mirror prox algorithm with variance reduction techniques. The outer loop follows a standard variance reduction process, while the inner loop incorporates momentum to accelerate convergence. Simulation results demonstrate the effectiveness of the proposed algorithm and show that it enhances model robustness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and valuable feedback. We have made efforts to address your concerns. If there are any further questions, please let us know and we will reply promptly.\n\n**Q1.**   **The contribution seems incremental.** \n\n> **Reply**: While CVaR optimization is well-established (e.g., in [Curi et al., 2020]), we believe our contribution lies in adapting it specifically to the Neural Process (NP) framework. This is non-trivial because tasks in meta-learning differ greatly in difficulty and distribution, making CVaR-based reweighting particularly relevant to “worst-case adaptation.”\n>\n> We provide a finite-sum DRO formulation for task-level NP optimization and find that standard SGD-based solutions for CVaR can suffer from exploding/vanishing gradients. The double-loop mirror prox with variance reduction addresses these problems, stabilizing updates while achieving robust generalization.\n>\n> As with many deep-learning contexts, providing rigorous global convergence guarantees in non-convex scenarios is challenging. We plan to investigate approximate convergence or local convergence guarantees in a future revision, building upon the convex analyses presented in Appendix B.\n>\n>*[Curi et al., 2020] Curi, S., Levy, K. Y., Jegelka, S., and Krause, A. Adaptive sampling for stochastic risk-averse learning. Advances in Neural Information Processing Systems, 33:1036–1047, 2020.*\n\n**Q2.**   **(a) What is the sample complexity and minimum number of iterations required to achieve a certain level of accuracy?(b) In Algorithm 1 (Line 5), how is the maximum number of inner loop iterations Ks determined?** \n\n> **Reply**: We follow standard distributionally robust optimization settings, where each task can be large, and the meta-dataset comprises many tasks. In each iteration, we typically sample at least one data point per task (one shot per task). Detailed formal sample complexity results exist for convex mirror-prox setups; translating them to non-convex NPs remains an open question.\n>\n> In the paper, $S$ denotes the number of outer loops (epochs), and $K_s$ represents the number of inner-loop iterations within each outer-loop $s$. In experiments, we set $K_s$ as a constant across all outer loops, i.e., $K_s = K$ for all $s$. $K$ is chosen to scale with the average number of samples per task. Empirically, we find that $K_s = 100-500$ often suffice for stable training in NP tasks; deeper loops help slightly but yield diminishing returns relative to the added cost. For outer loop epochs $S$, usually between 50 to 500 epochs. For simple tasks (e.g., low-dimensional regression, small datasets),  $S$ is 50–200 epochs.  For complex tasks (e.g., image completion),  $S$ often requires 200–500+ epochs. We will expand them to also show the effect of varying the number of inner/outer loop iterations in a revised version.\n\n**Q3.**   **Theoretical Questions (Appendix B)** \n\n> **Reply**: For Proposition B.1, we will clarify that H refers to a set of finite cardinality or finite VC dimension to avoid ambiguity.\n>\n> For Theorem B.5, the $a$ in the norm $|\\cdot|_{\\theta a,*} $  is a typo, it should be $|\\cdot|_{\\theta,*} $.\n>\n> For Line 740 (Eq.(26)), the second term should be $2(\\sum_{m=1}^M|q_m^+ - q_m^-|G)^2$. For the second term in line 737, we also omitted the square. Thank you again for pointing out the error.\n>\n> For the left-hand side of Equation (27), adding or omitting the absolute value or norm is acceptable. \n>\n> For Equation (28), we will correct $||\\theta^+ - \\theta^-||_{\\theta}$ to be squared, i.e., $||\\theta^+ - \\theta^-||_{\\theta}^2$\n\n**Q4.**   **Clarifications and Notation Issues** \n\n> **Reply**: Line 127 (Left Column): Garnelo et al. (2018a) assume the target set $T$ is disjoint from the context set $C$ in standard CNP. In our exposition, $T$ could overlap with $C$ or not. The choice is optional and widely used in BNP, DIVNP, TNP, SNP, and we will clarify that it differs slightly from the original CNP assumptions.\n> Lines 220–221: In the expression $max_{\\theta} \\psi_{\\theta}(\\theta)−min_{\\theta} \\psi_{\\theta}(\\theta) ≤ D_{\\theta}^2$, we will revise the notation to separate the parameters inside $\\psi(\\cdot)$ from the outer $\\theta$ symbol, ensuring clarity.\n\n**Q5.**   **Lines 436–437 suggest robust solutions shift the risk distribution left (Figure 4), but in Figure 4(b), RANP still has a rightmost bar. Could you clarify?** \n\n> **Reply**: RANP does not eliminate all high-risk tasks but significantly reduces their proportion. We will refine the text to specify “we shift the distribution, decreasing the fraction of tasks with high risk” rather than implying complete removal. We can also adjust Figure 4’s bin widths or annotate it more explicitly to highlight the reduction percentage.\n\n**Q6.**   **Typographical Errors** \n\n> **Reply**: We have thoroughly revised the spelling errors, typographical mistakes, and symbol ambiguities in the text.\n\nLastly, thank you once again for your valuable comments.", "claims_and_evidence": "Yes, the paper’s conclusions are well-supported by the provided evidence.", "methods_and_evaluation_criteria": "Yes, the simulation results verify the effictiveness of the proposed algorithm.", "theoretical_claims": "Yes, I check their Appendix B, see the following \"Other Strengths And Weaknesses\" part for details.", "experimental_designs_or_analyses": "Yes, the experimental results are reasonable.", "relation_to_broader_scientific_literature": "In theory part, i.e., Appendix B, the author cite the results in [Curi et al., 2020]. For Theorem B.4 and B.5, I think the proof techniques used are standard, I will add more comments in the following \"Other Strengths And Weaknesses\" part.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "See the following \"Other Strengths And Weaknesses\" part.", "other_comments_or_suggestions": "### 1. Main Contribution  \nThis paper is well-written and easy to follow. However, its contribution appears to be incremental. The Conditional Value at Risk (CVaR) optimization problem and the reformulation techniques in Equation (5) have been previously explored in the literature, such as [Curi et al., 2020]. The primary contribution of this work seems to be the Variance-Reduced Stochastic Mirror Prox Algorithm. However, the authors do not provide a theoretical convergence proof, which limits the impact of this contribution. While the experimental results demonstrate the algorithm’s effectiveness, a theoretical analysis would significantly strengthen the paper.\n\n### 2. Algorithmic Considerations  \n- What is the sample complexity and the minimum number of iterations required to achieve a certain level of accuracy?  \n- In Algorithm 1, Line 5, how is the maximum number of inner loop iterations, $K_s$, determined?  \n\n### 3. Theoretical Questions (Appendix B)  \n- **Proposition B.1:** The statement \"Let $h: \\mathcal{X} \\to \\mathcal{Y}$ be a finite function class $|\\mathcal{H}|$\" is unclear. Did you mean that $\\mathcal{H}$ is a function class with finite VC dimension, denoted as $|\\mathcal{H}|$?  \n- **Theorem B.5 (Proof, Line 728):** Regarding the norm $\\|\\cdot\\|_{\\theta a, \\star}$, how is the parameter $a$ defined?  \n- **Line 740 (Second-to-last inequality):** Should $q_m^+$ and $G$ be squared?  \n- **Equation (27):** Should the left-hand side be enclosed in absolute values or a norm?  \n- **Equation (28) (Last inequality):** Should $\\|\\theta^+ - \\theta^-\\|_{\\theta}$ be squared?  \n\n### 4. Clarifications and Notation Issues  \n- **Line 127 (Left Column):** In Section 2.3 (page 3) of Garnelo et al. (2018a), the authors describe training Conditional Neural Processes (CNPs) by predicting the full dataset conditioned on a random subset, without requiring the target set to be disjoint from the context set. However, in Section 3 of this paper, it is stated that $\\mathcal{T} \\subseteq \\{1,2,\\dots,N\\}$ with $\\mathcal{T} \\cap \\mathcal{C} = \\emptyset$ only in CNP. Could you clarify whether the target set in CNP is necessarily disjoint from the context set, or if this is simply an optional design choice?  \n- **Lines 220–221:** In the expression $\\max_{\\theta\\in \\Theta} \\psi_{\\theta}(\\theta) - \\min_{\\theta\\in \\Theta} \\psi_{\\theta}(\\theta) \\leq D_{\\theta}^2$, I suggest using a different variable to distinguish the parameter and argument of $\\psi(\\cdot)$ for clarity.  \n\n### 5. Simulation Results  \n- **Lines 436–437:** The authors state that the robust solutions shift the original risk distribution to the left in Figure 4. However, in Figure 4(b), RANP still has a portion in the rightmost bar. Could you adjust the probability level to make the shift more apparent?  \n\n### 6. Typographical Errors  \n- **Page 2 (Left Column, Lines 78–79):** \"higi-loss\" should be corrected to \"high-loss\".  \n- **Page 13 (Assumption B.3):** There is an extraneous bracket in the phrase \"Smoothness and Lipschitz Continuity\".  \n- **Page 14 (Theorem B.5):** The notation for the Lipschitz constant is inconsistent. The paper states that $\\nabla F_{\\alpha}$ is $L^\\star$-Lipschitz, but it should be $L_{\\star}$. Please unify the notation throughout the manuscript.", "questions_for_authors": "See the \"other comments or suggestions\" part for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R2SdkFirFo", "reviewer": "Reviewer_sKZn", "summary": "This paper proposes a training method for robust neural process based on risk-averse stochastic optimization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and valuable feedback. We have made efforts to address your concerns. If there are any further questions, please let us know and we will reply promptly.\n\n**Q1.**   **The paper uses simple regret as the primary performance measure, but a more robust metric like the CVaR of simple regret may be more suitable given the risk-averse theme.** \n\n> **Reply**: We agree that CVaR-based metrics align directly with our emphasis on robustness. We initially chose simple regret for consistency with prior NP-based Bayesian optimization work, which commonly measures average performance.\n> In the final version, we plan to include CVaR of simple regret in our Bayesian optimization experiments. Preliminary results suggest that our risk-averse training indeed lowers the worst-case tail of the regret distribution, indicative of stronger robustness.\n> We will highlight these findings to reinforce that the proposed method not only reduces average regret but also mitigates high-regret outliers.\n\n**Q2.**   “Perhaps give some reference on the equivalence between Eq. (3) and Eq. (4).” “… equivalence between Eq. (3) and Eq. (6).”** \n\n> **Reply**: Thank you for noting this. We do rely on standard transformations from the CVaR’s probability-constrained problem to its slack-variable-based and DRO-based forms. We will add more explicit citations (e.g., Rockafellar et al. (2000) for CVaR expansions and Shapiro et al. (2009) for standard references on DRO formulations) and a short annotated derivation in the supplement clarifying each step.\n>\n> *[Rockafellar et al. (2000)]: Rockafellar, R. T., Uryasev, S., et al. Optimization of conditional value-at-risk. Journal of risk, 2:21–42, 2000.*\n>\n> [*Shapiro et al. (2009)]: A. Shapiro, D. Dentcheva, and A. Ruszczy´ nski. Lectures on stochastic programming: modeling and theory. SIAM, 2009.*\n\n**Q3.**   **Typographical Error in “fundom” (Line 115, right column)There is a typo: “fundom.”** \n\n> **Reply**: We will correct “fundom” to “random” or “function domain” (whichever was intended in context). Thank you for pointing it out.\n\n\n**Q4.**   **Is there a theoretical way to show how our proposed approach improves the robustness of neural processes?** \n\n> **Reply**: As shown in Equations (3)–(6), our training reweights high-loss (or high-regret) tasks so they carry larger gradients, effectively controlling the model’s tail behavior. This is theoretically grounded in distributionally robust optimization.\n> A formal proof of how and by how much tail performance improves remains challenging, especially in a non-convex NP setting. Nonetheless, the link between CVaR and tail distribution improvements is well-known in risk-averse optimization theory (e.g., Rockafellar et al., 2000).\n> We will add references and clarifications that highlight how increasing the confidence level α in CVaR training can reduce the worst-case outcomes.\n\n**Q5.**   **The robust training process also improves average performance (Figure 3). Please discuss.** \n\n> **Reply**: Indeed, empirically, we observe that focusing on high-risk tasks can prevent overfitting to easy samples and help the model learn more stable, generalizable parameters. That can yield broader improvements in average performance as well.\n> A moderate amount of noise or risk weighting seems beneficial for both extremes (the tail and the average) by flattening observed losses. We will stress this empirical phenomenon more in Section 5.\n\nLastly, thank you once again for your valuable comments.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Mostly make sense, except the performance metrics for the experiment part.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes. I checked the Bayesian optimization part. The issue is that simple regret is used as the performance metric. However, since the paper is focusing on robust neural process, I think it is more appropriate to use robust metric such as the CVaR of simple regret.", "relation_to_broader_scientific_literature": "As far as I know, prior works of NP focus on average performance in prediction, which in many applications of NP, it is necessary to be risk-averse. This paper shows a method to train robust NP.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The considered problem is original and important for many practical problems. \n2. The proposed training method is novel based on distributionally robust optimization.\n\nWeaknesses:\n1. I think the experimental part should adopt metrics with more robustness consideration. For example, use CVaR of the simple regret for the Bayesian optimization part.", "other_comments_or_suggestions": "1. Line 115, right column, typo in \"fundom\"?\n2. Perhaps give some reference on the equivalence between Eq. (3) and Eq. (4).\n3. Also some reference on the equivalence between Eq. (3) and Eq. (6).", "questions_for_authors": "1. Is it possible to give a theoretical characterization on how the robustness of the neural process is improved with the proposed approach?\n2. Seems that the robust training process also improves the average performance (e.g., in Fig. 3). Can the authors discuss this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper is well written and  proposes a novel solution to an important problem.
Some reviewers were concerned about the lack of theoretical novelty compared to prior work on CVaR  [Curi et al., 2020], and the lack of  convergence proof for the neural processes setting.

It seems that the rebuttal has addressed most of the concerns raised by the reviewers.

In my opinion, while the theoretical contribution is indeed limited, the integration of the method to the neural processes setting and the empirical results are significant contributions to the community and merit publication in this venue.
Decision: Accept (poster)