Neural Conditional Probability for Uncertainty Quantification
Authors: Vladimir R Kostic
gregoire pacreau
Giacomo Turri
Pietro Novelli
Karim Lounici
Massimiliano Pontil
Venue: NeurIPS 2024 poster (2024)
Keywords: Operator learning
conditional density estimation
statistical inference
deep learning
theoretical guarantees
Primary area: learning_theory
TL;DR: A novel operator approach for learning conditional probability distributions with deep learning

Abstract: We introduce Neural Conditional Probability (NCP), an operator-theoretic approach to learning conditional distributions with 
a focus on statistical inference tasks. NCP can be used to build conditional confidence regions and extract key statistics such as 
conditional quantiles, mean, and covariance. It offers streamlined learning via a single unconditional training phase, allowing 
efficient inference without the need for retraining even when conditioning changes. By leveraging the approximation 
capabilities of neural networks, NCP efficiently handles a wide variety of complex probability distributions. 
We provide theoretical guarantees that ensure both optimization consistency and statistical accuracy. 
In experiments, we show that NCP with a 2-hidden-layer network matches or outperforms leading methods. 
This demonstrates that a a minimalistic architecture with a theoretically grounded loss can achieve 
competitive results,  even in the face of more complex architectures.

Neural Conditional Probability for Uncertainty Quantification
Vladimir R. Kostic
Karim Lounici Grégoire Pacreau 3
Giacomo Turri!
Pietro Novelli!
Massimiliano Pontil
'CMLL, Istituto Italiano di Tecnologia CCMAP-ccole Polytechnique
University of Novi Sad *AI Centre, University College London
Abstract
We introduce Neural Conditional Probability (NCP), an snerator-theoreti approach learning conditional distributions with focus on statistical inference tasks. NCP can be used to build conditional confidence regions and extract key statistics such as conditional quantiles, mean, and covariance. It offers streamlined learning via a single unconditional training phase, allowing efficient inference without the need for retraining even when conditioning changes. By leveraging the approximation capabilities of neural networks, NCP efficiently handles a wide variety of com- plex probability distributions. We provide theoretical guarantees that ensure both optimization consistency and statistical accuracy. In experiments, we show that NCP with a 2-hidden-layer network matches or outperforms leading methods. This demonstrates that minimalistic architecture with theoretically grounded loss can achieve competitive results, even in the face of more complex architectures.
Introduction
This paper studies the problem of estimating the conditional distribution associated with a pair of random variables, given finite sample from their joint distribution. This problem is fundamental in machine learning, and instrumental for various purposes such as building prediction intervals, performing downstream analysis, visualizing data, and interpreting outcomes. This entails predicting the probability of an event given certain conditions or variables, which is a crucial task across various domains, ranging from finance (Markowitz) 1958) to medicine (Ray et al. 2017). to climate modeling (Harrington) 2017) and beyond. For instance, in finance, is essential for risk assessment to estimate the probability of default given economic indicators. Similarly, in healthcare, predicting the likelihood of a disease, given patient symptoms, aids in diagnosis. climate modeling, estimating the conditional probability of extreme weather events such as hurricanes droughts, given specific climate indicators, helps in disaster preparedness and mitigation efforts.
According to Gao and Hastie 2022), there exist four main strategies to learn the conditional distribu- tion. The first one relies on the Bayes formula for densities and proposes to apply non- -ormmarmet statistics to learn the joint and marginal densities separately. However, most of non- parametric techniques face significant challenge known as the curse of dimensionality (Scott) 1991 Nagler and Czado, 2016). The second strategy, also known Localization method, involves training model unconditionally on reweighted samples, where weights are determined by their proximity to the desired conditioning point (Hall et al. 1999, Yu and Jones 1998). These methods require retraining the model whenever the conditioning changes and may also suffer from the curse of dimensionality the weighting strategy treats all covariates equally. The third strategy, known as Direct Learning of the conditional distribution involves finding the best linear approximation of the conditional density on dictionary of base functions kernel space (Sugiyama al. 2010 et al. 2007). The
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

performance of these methods relies crucially on the selection of bases and kernels. Again for high-ding nigh-dimensional settings, approaches that assign equal importance to all covariates may be less effective. Finally, the fourth strategy, known as Conditional Training, involves training models to estimate target variable conditioned on certain covariates. This typically based on partitioning the covariates space. X' into sets, followed by training models unconditionally within each partition (see Gao and Hastie 2022 Winkler et al. 2020; Lu and Huang 2020 Dhariwal and Nichol [202]] and references therein). However, this strategy requires large dataset to provide enough samples for each conditioning and is expensive as it requires training separate models for each conditioning input set, though they stem from the same underlying joint distribution.
The principal contribution of this work is a different conditional probability ap- proach that does not fall into any of the four aforementioned strategies. Rather than learning the conditional density directly, our method, called Neural Conditional Probabílity (NCP), aims to learn the conditional expectation operator Eylx associated to the random variables X E X and Y E y based on data from their joint distribution. The operator is defined, for every measurable function f: y R, as
Contributions
[y]x/ EI(M)
NCP is based on principled loss, leveraging the connection between conditional expectation operators and deepCCA (Andrew et al. 2013) established in (Kostic al. 2024), and can be used interchangeably to:
(a) retrieve the conditional density Pyjx with respect to marginal distributions of X and Y:
(b) compute conditional statistics E{J(Y) X] for arbitrary functions / y + R, including conditional mean, variance, moments, and the conditional cumulative distribution function, thereby providing access to all conditional quantiles simultaneously;
(c) estimate the conditional probabilities P|Y E B X A] for arbitrary sets B C y and A C X' with theoretical non-as non-asymptotic guarantees on accuracy, allowing us to easily construct conditional confidence regions.
Notably, our approach extracts statistics directly from the trained operator without retraining or resampling, and it is supported by both optimization consistency and statistical guarantees. In addition our experiments show that our approach matches or exceeds the performance of leading methods, even when using& basic a 2-hidden-layer network. This demonstrates the effectiveness of minimalistic architecture combined with a theoretically grounded loss function.
Paper organization In Section we review related work. Section 3 introduces the operator theoretic approach to model conditional expectation, while Section 4 discusses its training pipeline. In Section S we derive learning guarantees for NCP. Finally, Section 6 presents numerical experiments.
2 Related works
Non-parametric estimators are valuable for density and conditional density estimation as they don't t rely on specific assumptions about the density being estimated. Kernel estimators, pioneered by Parzen (1962) and Rosenblatt] (1956). are widely used non- con-paraeetric density estimation method. Much effort has been dedicated to enhancing kernel estimation, focusing on aspects like bandwidth selection (Goldenshluger and Lepski 2011), non-linear aggregation (Rigollet and Tsybakov 2007), and computational efficiency (Langrené and Warin 2020), as well as extending it to conditional densities (Bertin et al.] (014). A comprehensive review of kernel estimators and their variants is provided in (Silverman] (017). ) See also (Tsybakov] 2009) for a statistical analysis of their performance. However, most of non- parammertttic techniques face significant challenge known the curse of dimensionality (Scoti 1991; Nagler and Czado 20166), meaning that the required sample size for accurate estimation grows exponentially with the dimensionality of the data (Silverman 2017). Additionally, the computational complexity also increases exponentially with dimensionality [angrrnéé and Warin 2020
Examples of localization methods include the work by Hall et al. (9999) for conditional CDF estimation using local logistic regression and locally adjusted Nadaraya- Watson estimation, well as conditional quantiles estimation via local pinball loss minimization in YYu and Jones
1998). Examples
of direct learning of the conditional distribution include (Sugiyama et al. 2010) via decomposition on

---

a dictionary of base functions. Similarly, Li et al. (2007) explores quantile regression in reproducing Hilbert kernel spaces.
Conditional training is a popular approach which was adopted in numerous works, as in the recent work by Gao and Hastie (2022) where parametric exponential model for the conditional density (o(() is trained using the Lindsey method within each bin of partition of the space X. This
strategy has also been implemented in several prominent classes of generative models, including
Normalizing Flow (NF) and Diffusion Models (DM) (Tabak and Vanden-Eijnden, 2010 Dinh
ct al. 2014, Rezende and Mohamed| 2015a Sohl-Dickstein al., 2013). These models work by mapping a simple probability distribution into more complex one. Conditional training approaches
for NF and DM have been developed in many works including (e.g. |Winkler et al.) 2020; Lu
and Huang 2020 Dhariwal and Nichol, (02T). In efforts to lower the computational burden of conditional diffusion models, an alternative approach used heuristic approximations applied directly to unconditional diffusion models on computer vision related tasks (see e.g. [Song et al. [023] Zhang et al., 2023). However, the effectiveness of these heuristics accurately mimicking the true conditional distributions remains uncertain. Another crucial aspect of these classes of generative models is that while the probability distribution modelled explicitly, the computation of any relevant statistic, say [[||X] [X] is left
implicit problem usually solved by sampling from (v(y)>> and then approximating E [Y] [] via simple Monte-Carie integration. As expected, this approach quickly becomes problematic as the dimension of the output space y becomes large.
Conformal Prediction (CP) is a popular model-agnostic framework for uncertainty quantification (Vovk et al. 1999). Conditional Conformal Prediction (CCP) was later developed to handle condi- tional dependencies between variables, allowing in principle for more accurate and reliable predictions (see Lei and Wasserman] 2014 Romano et al., 2019 Chernozhukov et al., 2021; Gibbs et al. 2023 and the references cited therein). However, (CP) and (CCP) are not without limitations. The construc- tion of these guaranteed prediction regions need to be recomputed from scratch for each value of the confidence level parameter and of the conditioning for (CCP). addition, the produced confidence regions tend to be conservative.
3 Operator approach to probability modeling
Consider a pair of random variables X and Y taking values in probability spaces (X, Ex, nb) and (V, Σy, v), respectively, where X' and y are state spaces, Ex and Σy are sigma algebras, and j and are probability measures. Let p be the joint probability measure of (X, Y) from the product space X' x y. We assume that p is absolutely continuous w.r.t. to the product measure of its marginals, that p < μ x v, and denote the corresponding density by p = dp/d(f x v). also called point-wise dependency in Tsai et al. 2020), so that p(dxr, dy) p(ar, (dd(dr))(dd))
The principal goal of this paper is, given a dataset D, == (zi, ())m(n of observations of (X, to estimate the conditional probability measure
(BB|z) == P[Y E B|X= x]. x E Xx BE Our approach is based on the simple fact that p(B 2)) E[1
(I)
z]. where 1g denotes the characteristic function of set B. More broadly we address the above problem by studying the conditional expectation operator Eyix: 1200) + (X)), which is defined, for every f E (2) and x E X, as
[Exixf(<x) = =[Xx)())==E[((>))> =] S fly)(dy) ()) S. (())((,,))(()))
where (X') and 12(0") denotes the Hilbert spaces of functions that are square integrable w.r.t. to and v, respectively. One readily verifies that ||Ey|x|| = and E =1111111 xx-
A prominent feature of the above operator is that its rank can reveal the independence of the random variables. That is, X and Y are independent random variables if and only if Exix is a rank one operator, in which case we have that Eyyx 1x @ ly. It is thus useful to consider the deflated operator Dyjx Exix 1x & 1y: ()) (X)), for which we have that
f
+ +
f 12
(2)
For dependent random variables, the deflated operator nonzero. In many important situations, such when the conditional probability distribution is a.e. absolutely continuous w.r.t. to the target
3

---

measure, that is 1(112) & V for p.a.e. a E X, the operator Eyjx is compact, and, hence, we can write the SVD of Eyix and Dyjx respectively
Eyyx = Σ of u
and Dyjx Σ u 1
(3)
where the left (u,))e JiN and right (o) Die singular functions form complete orthonormal systems of T (2') and ())), respectively. Notice that the only difference in the SVD of Eyjx and Dy x is the extra leading singular triplet (oh: uh: vp) (1, 1 1x) of Eypx. In terms of densities, the SVD of | leads to the characterization
p(a, v) = Ei (T) (1) = 1+ Ef (2) (1)
The mild assumption that Eyjx is a compact operator allows one to approximate it arbitrarily well with (large enough) finite rank (empirical) operator. Choosing the operator norm as the measure of approximation error and appealing to the Eckart- Mowart-Youn--Mirkk Theorem (see Theorem 3 in Appendix B.I one concludes that the best approximation given by the truncated SVD, that is for every d E N,
Dylx [Drixla: >i of u @ uf: and [Dyxxla E arg min (aa - Al], (4)
where the minimum is given by o. and the minimizer unique whenever T241 < 0 This leads to the approximation of the joint density w.r.t. marginals p(a, v) 2 1 Σi of uz (xr) of (v), so that + of at (=))L[(())
E[S(Y) x]
(5)
which in particular, choosing f = 1g. gives
P|Y E B x] = P|Y E BJ Eim of ((
Moreover, we have that
Eyx11)) PPX E A]
(X) (Y)
PYEBI
& P|Y E B|+
Σ of P E A]
for which the approximation error is bounded in the following lemma.
Lemma (A (Approximation bound). For any A E Ex such that P[X E A] > and any B E Ey,
PY PY {1 (D
PLY E P[X E A]
I 02+1 V
(6)
P E A]
Neural network model Inspired by the above observations, to build the NCP model, we will parameterize the truncated SVD of the conditional expectation operator and then learn it. Specifically, we introduce two parameterized embeddings u² X' -> Rd and vº y - Rdd and the singular values parameterized by wd Rd, respectively given by
u ='(x):=(w((a ... (T v where the parameter & takes values in a prescribed set e.
(y)::-(x((()) ... ( and a. -v
(w2))
We then aim to learn the joint density function p(x, y) in the (separable) form
(= == 1 + Eieig (x) of (y) = + (o" @ u (x),, (v)),
where @ denotes element-wiss product. One of the prominent losses considered for the task of learning p E L2 -2 is the least squares density ratio loss E (p pa)2 p E 2E,Po, c.f. Tsai et al. (2020), also considered by HaoCCen et al. (2022) in the specific context of augmentation graph in self-supervised deep learning, linked to kernel embeddings (Wang et al. 2022 and rediscovered and tested on DeepCCA tasks by Wells et al. (2024). Here, following the operator perspective, use the characterization 44) of the optimal finite rank model propose new loss that: (1) excludes the known feature from the learning process, and (2) introduces penalty term to enforce orthonormality of the basis functions. More precisely, loss (0) = L(Ø) + R(( is composed of two terms. The first one
(()) EpxuPà 2E,po + [[ExxePo] [.][E [,,Po]] + 2E
(7)

---

is equivalent to solving with A = -E of [uf - =E, @ [u. - E and can be written in terms of correlations between features. Namely, denoting the covariance and variance matrices by
Var[z] == [[ [[=)) F(z))", (8) and abbreviating u² := u (X) and v. = v (Y) for simplicity, we can write
Cov(2,z'] := [[(= - [(z))(=' - =] and
[()) =- tr (Var[Vo o w²) VarlVo @ x 2 20ov o u", o
(9)
p=po for some A then the optimal loss is the and, as we show below, [(0) measures how well po y) - approximates Σis (z) However, in order to obtain a useful probability model, it is of paramount importance to align the metric in the latent spaces with the metrics in the data-spaces I,2, (x)) and (O). For different reasons, a similar phenomenon has been observed in Kostic et al. (2024) where dynamical systems are learned via transfer operators. In our setting, this leads to the second term of the loss that measures how well features u² and e span relevant subspaces in L (X) and (3)), respectively. Namely, aiming Eluç (X (Xu (X)] = Elu
= (1= i.j E {0, 1,.,,,,,,,, d} leads to == (X) u (x) 1- || (Y)
(Y))1. (10)
We now state our main result on the properties of the loss L... which extends the result in Wells et al. (024) to infinit-diimessonnl operators and guarantees the uniqueness of the optimum due to R.
Theorem 1. Let Eyix: : [2)())) - T. (2') be a compact operator and Dyix E @9 og be the SVD of its deflated version. If uf E (2) and vf E L)))), for all E and i E [d], then for every 0 E θ, (0) 2- Eield o²2. Moreover, if > 0 and 03 > o'41 then the equality holds if and only if (of, ul, u.)) equals (o, ut, vt) p-a.e., up to unitary transform of singular spaces.
We provide the proof in Appendix B.3 In the following section, we show how to learn these canonical features from data and construct approximations of the conditional probability measure.
Comparison to previous methods NCP does not fall into any of the four categories defined by Gao and Hastic (2022), as it does not aim to learn conditional density of Y directly. Instead, NCP focuses on learning the operator mapping I22 + (2)). from which all relevant task-specific statistics can be derived without requiring retraining. This approach effectively integrates with deep representation learning to create a latent space adapted to p(y[z)) As a result, NCP efficiently captures the intrinsic dimension of the data, which is supported by our theoretical guarantees that depend solely on the latent space dimension (Theorem 2). In contrast, strategies designed for learning density often encounter significant limitations, such as the curse of dimensionality, potential substantial misrepresentation errors when the pre-specified function dictionary misaligns with the true distribution p(yllz), and high computational complexity due to the need for retraining. Experiments confirm NCP's capability to learn representations tailored to a wide range of data types- appes-including manifolds, graphs, and high-dimensional distributions- wittht relying on predefined dictionaries. This flexibility allows NCP to outperform popular aforementioned methods.
Training the NCP inference method
In this section, we discuss how to train the model. Given training dataset D,, = (s,, Ms (e)ee( and
networks (u², o", o²), we consider the empirical loss (6) := +)()), where we replaced ( and (10) by their empirical versions. In order to guarantee the unbiased estimation.
show within
the proof of Thcorem [. two terms of our loss can be written using two independent samples (X.Y) and (X', Y') from p as L(0) = E[ (X) (X)-Eu((X), (x() (X))))x()()) v² (V)-Ex o (Y')- Eo"(Y"), (),a and R(0)= E R(u (X), ul (X'), (Y),, vd (Y")),, where the loss functionals L and R are defined for u, u', u,, v' E Rdd and 8 E [0, 1]d
L(u, u', u, s):=,, (w (iag(s))') + (uT diag(s)u') -uT diag(s)'' diag(s)u', (11)
R(u, u', U. (')==="(") v²) u'u'') (T)))(u-ull (u-w') v/) 2 (u-v') (u-
(12)
Therefore, at every epoch we take two independent batches DJ, and Då of equal size from Dn.. leading to Algorithm[ [ See Appendix T for the full discussion, and Appendix A.2 where we also provide in Figure 4 example of learning dynamics.
5

---

Algorithm Separable density learning procedure
Require:
training data train (rain) train u², o and v using the NCP loss
Center and scale Xuain and Yirai for each epoch do
From XmainnYaaii) pick two random batches (X mmii)) and (X
train Y trinn
Evaluate: U <- u (Xirain); U" <- u =x (xrnnn), V (Y main) ), V v.0 (Y train)
Compute [(0) as an unbiased estimate using or
Compute R() as an unbiased estimate using (10) or 12
Compute NCP loss (0) = (()) + RR(( and back-propagate
end for
Practical guidelines for training In the following, briefly report few aspects to be kept in mind when using the NCP in practice, referring the reader to Appendix for further details. The computational complexity of loss estimation presents three distinct methodological approaches. The first method utilizes unbiased estimation via covariance calculations in (9) and (IO), achieving a computational complexity of Ond?) for batch size An alternative approach employing U- statistics with (TT) and (T2) requires On"'d) operations per iteration, offering the estimation of the same precision. A third method involves batch averaging of (TT) and 112, reducing computational complexity to (nd), which enables seamless integration with contemporary deep learning frame- works, albeit potentially compromising training robustness through less accurate 4th-order moment estimations. Method selection remains contingent upon the specific problem's computational and statistical constraints. Further, the size of latent dimension d, indicated by Theorem[[rehe relates the problem's "difficulty" in the sense of smoothness of joint density w.r.t. its marginals. Lastly, after the training, an additional post-processing may be applied ensure the orthogonality of features v and d and improve statistical accuracy of the learned model.
Performing inference with the trained NCP model We now explain how to extract important statistical objects from the trained model (π") 70 a o²).
). To this end, define the empirical operator
Dyix: (0
of [+)x
E, fl, f E L20)) E X, (13)
where E == 4 (M) (1)))())) Then, without any retraining nor simulation. we can compute the following statistics:
Conditional Expectation:
E, fl(x), f x E
Conditional moments of order 2 1: apply previous formula to f(u) = u.
Conditional covariance: Cov (YIx)= = Ey ( Ey Ey
Y x [.
Conditional probabilities: apply the above conditional expectation formula with f() = 11()). that is, Pyy (B) = Egy [ss] and Po I)) Fyy Eiela of 7t 1 B E X. Then, integrating over an arbitrary set A E Σx we get
P A) = pa (B) + Endd 0f
È. E E.
E, 11..
(14)
Conditional quantiles: for scalar output Y, the conditional CDF Frixea(t) is obtained by taking B (-o0, (), and in Algorithm 3 in Appendix C
we show how to extract quantiles from it.
5 Statistical guarantees
We introduce some standard assumptions needed to state our theoretical learning guarantees. To that end, for any A E Ex and B Σy define important constants, followed by the main assumption,
1-P A P[X E A]
PLY B) e E
(x)(-)==1V
and (B):

---

Assumption 1. There exists finite absolute constants Cu. > 1 such that for any 0 E θ
ess supl u (x)]l(>> I Cu.
ess suppp |100 (y)lliz S Co.
yau
Next, we set (x)): Var =etro(():=Var((l(((() ((((()))))))) VM))= - -E(o and
<n(o)== (ca (caVec)) vd log(ed ( log(eo-1)
log 28
Fn (6)===2
n
12
n
(15)
for some large enough absolute constant C' 0.
Remark 1. It follows easily from Assumption 1 that (x) and ( Consequently, assuming that n 2 (cu, ce)d. then (6) X (c, v /ooloe/oo111111 //
Finally, for a given parameter BEA and E (0, 1), let us denote
Eg max [ | (6) == J241 + Eg + 2 + EE ()) + (8))
| 1 Vo Vo 1V and (16)
|1,
(17)
In the following result, we prove that NCP model approximates well the conditional probability distribution w.h.p. whenever the empirical loss (0) is well minimized.
Theorem 2. Let Assumption Tbe satisfied, and in addition assume that
P(XEA) A P(YEB)2E,,
and n2(cvVe,) )d V 8log(65-)) (log((o=))(xx(())xxx()) (18)
Then for every A Σx \ (x)) and B Ey {D}
P|YEB| XEA]
(B A) 4t/,(6/3) + [1+tn (2xx(4)) (19)
P[YEB]
(B)
VPIXEA|P[YEBB
and PIYEB - ( (A)A)
Spy (8)3, 2(1 2(1 (4)
(20)
P|
VP
hold with probability at least 1 - 5 w.r.. iid draw of the dataset D, = (I Vj Ji(()) from
Remark Appendix B.5 we prove a similar result under less restrictive sub-Gaussion assump- tion on the singular functions u (X) and (Y))
Discussion The rate (6) in
pivotal for the efficacy of our method. If we appropriately choose the latent space dimension d to ensure accurate approximation (a)+1 < 1), achieve successful
training (Eg & o241), and secure large enough sample size (6) << 1). Theorem 2 provides
assurance of accurate prediction of conditional probabilities. Indeed, 20 guarantees (up
logarithmic factor)
PIYEB] ( +V P[XEA]
P[YEB] [(t])EA)- (B) OP
Note the inclusion of the term pPxX E A] in the denominator of the last term on the right-hand side, along with x (A). This indicates a decrease in the accuracy of conditional probability estimates for rarely encountered event A, aligning with intuition and with a known finite-sample impossibility result Lei and Wasserman (2014 Lemma 1) for conditional confidence regions when A is reduced to
any nonatomic point of the distribution (i.e. A = {#} with P[X = z] = =0). For rare events, larger sample size and higher-dimensional latent space characterized by d are necessary for accurate estimation of conditional probabilities.
We propose next non-asymptotic estimation guarantee for the conditional CDF of Y|X when is a scalar output. This result ensures in particular that accurate estimation of the true quan- tiles is possible with our method. Fix E R and consider the set B, (-oo, f meaning that A
and P|Y E =F We define similarly for the NCP estimator of
the conditional CDF = B, A)). The result follows from applying 20 to the set B,
7

---

Table 1:
Mean and standard deviation of Kolmogorov- distance of estimated CDF from the truth averaged over 10 repetitions with n 10 (best method red, second best in bold black).
Model
LinearGaussian
EconDensity
ArmaJump
SkewNormal
GaussianMixture
LGGMD
NCP- W
0.010 0.000
0,005 0.001
0.010 1: 0.002
0.008 0.001
0.015
0.004
0.047 0.005
DDPM
0,410
0.236 0.217
0.338 0.317
0.250 + 0.224
0,404
0,405 0.218
NF
0.008 0.006
0.006 0.003
0.143 + 0.010
0.032 0.002
0.107 0.003
0,254 0.004
KMN
0.601 = 0.004
0,362 + 0.017
0.487 = 0.004
0.381 + 0.009
0.309 0.001
0.224 + 0.005
MDN
0.225 = 0.013
0.048 + 0.001
0.163 + 0.018
0.087 + 0.001
0.129 0.007
0.176 +: 0.013
LSCDE
0.420 = 0.001
0.118 0.002
0.247 0.001
0.107 + 0.001
0.202 0.001
0.268 0.024
CKDE
0.120 = 0.000
0.010 0.001
0.072 +
0.023
0.048 0.001
0,230
NNKCDE
0.047 + 0.003
0,036 0.003
0.030 1: 0.004
0,030 +: 0.002
0,035 0.002
0,183 0.006
RFCDE
0.128 + 0.007
0.141 1: 0.009
0.133 1 0.015
0.142 1: 0.012
0.130 0.012 0.012
0.121 1: 0.006
FC
0.095 0.005
0.011 0.001
0.033 1 0.002
0,035 0.007
0.016 0.001
0,047 0.003
LCDE
0.108 0.001
0.026 + 0.001
0.113 0.002
0,075 0.006
0.035 0.001
0.124 + 0.002
Corollary 1. Let the Assumptions Theorem 2 holds with probability at least 5 that
be satisfied. Then for any t E R and 5 E (0, 1),
Fyixea(t) - Fr|XEA(t)) SSVVVVA (t( +V PI (o 2V28. +2v22
- (x)t)cnn6)/3)
1 1)c, 4px(4)c,
(21)
An important application of Corollary lies in uncertainty quantification when output a scalar. Indeed, for any a E 10,), we can scan the empirical conditional CDF FYΙXEA for values la < l such that F (to) 1 a and ta2 to minimal. That way we define non-asymptotic conditional confidence interval Ba == (ta, ta with approximate coverage 1 - More precisely we deduce from Corollary that
|P[Y E Ba X E A] (1 a))
2 (6/66)
2
(sàt1 + 2/22, + (2V/ + (8/6( + (px(A))e((/66) (22)
P[X E A]
In App B.6 derive statistical guarantees for the conditional expectation and covariance of Y.
6 Experiments
Conditional density estimation We applied our NCP method to a benchmark of several conditional density models including those of Rothfuss et al. (2019) Gao and Hastie (2022). See Appendix C.1 for the complete description of the data models and the complete list of compared methods in
Tab. with references. We also plotted several conditional CDF along with our NCP estimators
in Fig. 6 To assess the performance of each method, we use Kolmogorov-Smme Sonnogorov-Smmrnov (KS) distance between the estimated and the true conditional CDFs. We test each method on nineteen different conditional values uniformly sampled between the 5% and -5%cpeecet of p(x)) and computed the averaged performance over all the used conditioning values. In Tab, D we report mean performance (KS distance + std) computed over 10 repetitions, each with different seed. NCP with whitening (NCP-W) outperforms all other methods on 4 datasets, tics with FlexCode (FC) on dataset, and ranks close second on another one behind NF. These experiments underscore NCP's consistent performance. We also refer to Tab. 3 in App C.I for an ablation study on post-treatments for NCP.
Confidence regions Our goal is to estimate conditional confidence intervals for two different data models (Laplace and Cauchy). investigate the performance our method in (22) and compare to the popular conditional conformal prediction approach. We refer to App C.2 for quick description
of the principle underlying CCP. We trained an NCP model combined with an MI.P architecture
followed by whitening prot-proc.s.ing See App C for the full description. We obtained that

---

NCF [mnae 0.0005)
NE (rese 0.035)
Conditional Confermal (mae 0.0036)
Tale
02
90
35
10
NCP
Conditional Conferma!
132
240
-21
Figure I: Conditional mean (top only) and 90% confidence interval for NCP, NFs and CCP. Top: Laplace distribution; Bottom: Cauchy distribution.
way the NCP conditional CDE model that we used according to (22) build the conditional 90% confidence intervals. We proceeded similarly to build another set of conditional confidence intervals based on NFs. Finally, we also implemented the CCP method Gf et al. (2023).
In Fig. 1 the marginal is X ~ ( 5l) and Y = x follows either a Laplace distribution (top) with location and scale parameters (p(xx), b(x)) = (x² .:) or Cauchy distribution (bottom) with location and scale parameters (w²,, + w) In this experiment, we considered favorable situation for the CCP method of Gibbs al. 2023) by assuming prior knowledge that the true conditional location is a polynomial function (the truth is actually the square function). Every other parameter of the method was set as prescribed in their paper.
In Fig. D observe first that the CCP regression achieves the best estimation of the conditional mean mse- 3.6 10-3 against msse 3.8 10-2 for NFs and mse 8.3 10-3 for NCP, expected since the CCP regression model is well-specified this example. However, the CCP confidence intervals are unreliable for most of the considered conditioning. We also notice instability for NF and CCP when conditioning in the neighborhood of x = O, with the NF confidence region exploding at O. We suspect this is due the fact that the conditional distribution violating the condition of existence of a diffeomorphism with the generating prior, a fundamental requirement for NFs models to work at all. Comparatively. NCP does not exhibit such instability around æ = 0; it only tends to overestimate the confidence region for conditioning close to x 0. The Cauchy distribution known to be more challenging due its heavy tail and undefined moments. In Fig (bottom), we notice that NF and CCP completely collapse. This is not a surprising outcome since CCP relies on estimation of the mean which is undefined in this case, creating instability in the constructed confidence regions, while NF attempts to build a diffeomorphism between a Gaussian prior and the final Cauchy distribution. We suspect the conservative confidence region produced by NF might originate from the successive Jacobians involved in the NE mapping taking large values. In comparison, our NCP method still returns some reasonable results. Although the NCP coverage might appear underestimated for larger v, actual mean coverages computed on test set of 200 samples are 88% for NCP, 99% for NF and 79% for CCP. Tab, 55 Appendix C.2 provides a comparison study on real data for learning confidence region with NCP, NF and split conformal predictor featuring Random Forest regressor (RFSCP).
0 is degenerate, hence
diien-dimesiional synthetic experiment We simulated the following d-distribution for different
values of E {100, 500, 1000). Let ē (ā1, Z2, O.....0) O)T E Rd where x' (Ø1, F2) admits uniform distribution on the 2-dimensional unit sphere. We pick random mapping A E O₂ and we set X = Aø and the angle 0(X) = arcsin(#2). Next we consider two conditional distribution models for Y|X (Gaussian and discrete) described in Figure 3 NCP performs similarly to NF in the Gaussian case and outperforms NF for discrete distribution. Figure 7 in Appendix C.3 demonstrates that NCP scales effectively with increasing dimensionality d. As the dimension rises from d 100

---

(A)
3.4
3.3
3.2
Polewase
3.1
data
comdisional expectation
catimated 92% quantiles
3.0
66.0
86.5
87.0
87.5
66.0
A8.5
80.0
80.5
(10)
Figure Protein folding dynamics. Pairwise Euclidean distances between Chignolin atoms exhibit increased variance during folded metastable states (between 87-88μs and around 89 39.S)s). Ground truth is depicted in blue, predicted mean in orange, and the grey lines indicate the estimated 10% lower and upper quantiles.
to d 1000, the computation time increases by only 20%, while maintaining strong statistical performance throughout.
Bcbs-1
Usta 1,57
thet
Betu 1.77
Figure 3: High-dimensional synthetic experiment. We consider two models for Y X|X with d 100. Left: Y N(A(X), sin(()))))) Right: Y E 21, 4, 5, admits discrete distribution depending on ((X): y|X P; ifx) E m22), Pa if E [f/2, R), Ps ( |, 3v /2), if ((X) [3,/2, 2x). We take = (1/5, 1/5, 1/5, 1/5), Pa 1/2, P3 (0, 0, = (0,0,0,1/2,14) 1/2).
High-dimensionas experiment in molecular dynamics We investigate protein folding dynamics and predict conditional transition probabilities between metastable states. Figure shows how, by integrating our NCP approach with a graph neural network (GNN), we achieve accurate state forecasting and strong uncertainty quantification, enabling efficient tracking of transitions. For further context and a full model description, see App
Conclusion
We introduced NCP, novel neural operator approach to learn the conditional probability distribution from complex and highly nonlinear data. NCP offers a number of benefits. Notably, it streamlines the training process by requiring just unconditional training phase to learn the joint distribution p(z, y). Subsequently, allows us to efficiently derive conditional probabilities and other relevant statisties from the trained model analytically, without any additional conditional training steps or Monte Carlo sampling. Additionally, our method backed by theoretical non-asymptotic guarantees ensuring the soundness of our training method and the accuracy of the obtained conditional statistics. Our experiments on learning conditional densities and confidence regions demonstrate our approach 's superiority or equivalence leading methods, even using a simple Multi-Layer Perceptron (MLP) with two hidden layers and GELU activations. This highlights the effectiveness of a minimalistic architecture coupled with theoretically grounded loss function. While complex architectures often dominate advanced machine learning, our results show that simplicity can achieve competitive results without compromising performance. Our numerical experiments suggest that, while our approach works well across different datasets and models, the price we pay for this generality appears to be the need for a relatively large sample size (n Z 104) to start outperforming other methods. Hence, future direction is to study how to incorporate prior knowledge into our method to make it more data-efficient. Future works will also investigate the performance of NCP for multi-dimensiooal time series, causality and more general sensitivity analysis in uncertainty quantification.
10

---

Acknowledgements
We acknowledge financial support from EU Project ELIAS under grant agreement No. 101227,7, by NexiGenerationEU and MUR PNRR project PE0000013 CUP J53C22003010008 "Future Artificial Intelligence Research (FAIR)" and by NextGenerationEU and MUR PNRR project RAISE "Robotics and Al for Socio-economic Empowerment" (ECS00000035),
References
Ambrogioni, L., Güçlüü U., van Gerven, M. A. J., and Maris, E. (2017). The kernel mixture network:
A nonparametric method for conditional density estimation of continuous random variables.
Andrew, G., Arora, R., Bilmes, J., and Liveseu, K. (2013). Deep canonical correlation analysis. In International Conference on Machine Learning, pages 1247-1255.
Bercu, B., Delyon, B., and Rio, E. (2015). Concentration Inequalities for Sums and Martingales. SpringerBriefss Mathematics. Springer.
Bertin, K., Lacour, C., and Rivoirard, V. (2014). Adaptive pointwise estimation of conditional density function. arXiv:
Bishop, C. M. (1994). Mixture density networks.
Caponnetto, A. and De Vito, (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368.
Chanussot, L.., Das, A., Goyal, S., Lavril, T., Shuaibi, M., Riviere, M., Tran, K., Heras Deraas-Domingo, J., Ho, C., Hu, W., Palizhati, A., Sriram, A., Wood, B., Yoon, J., Parikh, D., Zitnick, C. L., and Ulissi, Z. (2021). Open catalyst 2020 (OC20) dataset and community challenges. ACS Catalysis, 11(10):6059-072.
Chernozhukov, V., Wüthrich, K., and Zhu, Y. (2021). Distributional conformal prediction. Proceed- ings of the National Academy of Sciences, 11 18(43))21107794118.
Devroye, L., Györfi, L., and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer New York.
Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780-8794.
Dinh, L.., Krueger, D., and Bengio, Y. (2014). Nice: Non-linear independent components estimation. arXiv preprint arXiv: 1410.55/1
Durkan, C., Bekasov. A., Murray, I., and Papamakarios, G. (2019). Neural spline flows. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bue, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Freeman, P. E.. Izbicki, R., and Lee, A. B. (2017). unified framework for constructing, tuning and assessing photometric redshift density estimates in selection bias setting. Monthly Notices of the Royal Astronomical Society, 468(4))4556-4555
Gao, Z. and Hastie, T. (2022). Lincde: conditional density estimation via lindsey's method. Journal of Machine Learning Research, 23(52):1-55.
Gibbs, I., Cherian, J. J., and Candès, F. J. (2023). Conformal prediction with conditional guarantees. arXiv:2305./26116
Goldenshluger, A. and Lepski, O. (2011). Bandwidth selection kernel density estimation: Oracle incqualities and adaptive minimax optimality. Annals of Statistics, 39(3):1088-1622
Hall, P., Wolff, C., and Yao, Q. (1999). Methods for estimating a conditional distribution function. Journal of the American Statistical Association, 1444433.

---

HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, T. (2022). Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Advances in Neural Information Processing Systems, volume 34, pages 5000-5011.
Harrington, L. J. (2017). Investigating differences between event-as-class and probability density- based attribution statements with emerging climate change. Climatic Change, 141:641-654.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855
Izbicki, R. and Lee, A. B. (2017). Converting high-dimensional regression to high-dimensional conditional density estimation. Electronic Journal of Statistics, 11(2):2800 2831.
Izbicki, R., Lee. A. B., and Freeman, P. E. (2017). Photo-z estimation: An example of nonparametric conditional density estimation under selection bias. The Annals of Applied Statistics, 11(2):698 724.
Koltchinskii, V. and Lounici, K. (2017). Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, pages 110-133.
Kostic, V., Novelli, P., Grazzi, R., Lounici, K., and Pontil, M. (2024). Learning invariant repre-
sentations of time-homogeneous stochastic dynamical systems. In International Conference Learning Representations (ICLR).
Langrené, N. and Warin, X. (2020). Fast multivariate empirical cumulative distribution function with connection to kernel density estimation. 2rXiv:2000.03246.
Lei, and Wasserman, L. (2014). Distribution prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-96.
Li, Q. and Racine, J. S. (2006). Nonparametric Econometrics: Theory and Practice, volume I of Economics Books. Princeton University Press.
Li, Y., Liu, Y., and Zhu, (2007). Quantile regression reproducing kernel Hilbert spaces. Journal of the American Statistical Association, 102(477):255-268.
Lu, Y. and Huang, B. (2020). Structured output learning with conditional generative flows. Proceed- ings of the AAAI Conference on Artificial Intelligence, 54004):5005-502.
Markowitz, H. M. (1958). Portfolio selection: Efficient diversification of investments. Yale University Press, 23.
Mendil, M., Mossina, L., and Vigouroux, D. (2023). Punce: a python library for predictive uncertainty calibration and conformalization. In Conformal and Probabilistic Prediction with Applications, pages 582-601. PMLR.
Minsker, (2017). On some extensions of bernstein's inequality for self-adjoint operators. Statisties & Probability Letters, 127:111-119.
Nagler, T. and Czado, C. (2016). Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas. Journal of Multivariate Analysis, 151:69-89.
Papamakarios, G., Pavlakou, T., and Murray, (2017). Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30.
Parzen, E. (1962). On Estimation of Probability Density Function and Mode. The Annals of Mathematical Statisties, 33(3):
Pospisil, T. and Lee, A. B. (2018). Rfcde: Random forests for conditional density estimation.
Ray, E. L., Sakrejda, K., Lauer, S. A., Johansson, M. A., and Reich, N. G. (2017). Infectious disease prediction with kernel conditional density estimation. Statistical Medicine, 36(30):4908 4929.
Rezende, D. and Mohamed, (2015a). Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1530-1538.
12

---

Rezende, D. and Mohamed, S. (2015b). Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR.
Rigollet, and Tsybakov, A. (2007). Linear and convex aggregation of density estimators. Mathe- matical Methods od Statistics, 16:260-280.
Romano, Y., Patterson, E., and Candes, E. (2019). Conformalized quantile regression. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buu Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted assisted Citeveettoo-MMICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages
Springer.
Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics, 27(3):832-8377 837.
Rothfuss, J., Ferreira, F., Walther, S., and Ulrich, M. (2019). Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprini urXiv:1903. 00954.
Schütt, K. T., Hessmann, S. S. P., Gebauer, N. W. A., Lederer, J., and Gastegger, M. (2023). SchNetPack 2.0: A neural network toolbox for atomistic machine learning. The Journal of Chemical Physics, 158(14):14481.
Schütt, K. T., Kessel, P., Gastegger, M., Nicoli, K. A., Tkatchenko, A., and Müller, K.-R. (2019). SchNetPack: Deep Learning Toolbox For Atomistic Systems. Journal of Chemical Theory and Computation, 15(1):448-44 455.
Scott, D. W. (1991). Feasibility of multivariate density estimates. Biometrika, 78(1):197-205.
Silverman, B. W. (1986). Density estimation for statistics and data analysis, volume 26 of Mono- graphs on Statistics & Applied Probability. Chapman and Hall.
Silverman, B. W. (2017). Density Estimation for Statistics and Data Analysis. Routledge, New York.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference Machine Learning, pages 2256-2265 PMLR.
Song, J.. Vahdat, A., Mardani, M., and Kautz, J. (2023). Pseudoinverse- guideed diffusion models for inverse problems. In International Conference on Learning Representations.
Stimper, V., Liu, D., Campbell, A., Berenz, V., Ryll, L., Schölkopf, B., and Hernández-Lobato, J. M. (2023). normflows: A pytorch package for normalizing flows. Journal of Open Source Software, 8(86):5361.
Sugiyama, M., Takeuchi, I., Suzuki, T., Kanamori, T., Hachiya, H., and Okanohara, D. (2010). Conditional density estimation via least-squares density ratio estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 781-788.
Tabak, E. G. and Vanden-Eijnden, E. (2010). Density estimation by dual ascent of the log- likeehoood. Communications in Mathematical Sciences, 8(1):217-233.
Tsai, Y.-H. H., Zhao, H., Yamada, M., Morency, L.-P., and Salakhutdinov, R. R. (2020). Neural methods for point-wise dependency estimation. Advances in Neural Information Processing Systems, 33:62-72.
Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer Series in Statistics.
Vershynin, R. (2011). Introduction to the non-asymptotic analysis of random matrices. 101.07.7
Vovk, V., Gammerman, A., and Saunders, C. (1999). Machine-leaming applications algorithmic randomness. In International Conference Machine Learning, pages 444-453.
13

---

Wang, Z., Luo, Y., Li, Y., Zhu, J., and Schölkopf, B. (2022). Spectral representation learning for conditional moment models. arXiv preprint arXiv:2210.16525.
Wells, L., Thurimella, K., and Bacallado, S. (2024). Regularised canonical correlation analysis: graphical lasso, biplots and beyond. arXiv preprini arXiv:2403.02979.
Winkler, C., Worrall, D., Hoogeboom, E., and Welling, M. (2020). Learning likelihoods with conditional normalizing flows. arXiv: /911:0291..0004..
Yu, K. and Jones, M. (1998). Local linear quantile regression. Journal of the American Statistical Association, 93(441):228-233
Zhang. G., J., Zhang. Y.. Yu. M.. Jaakkola, T. S., and Chang, S. (2023). Towards coherent image inpainting using denoising diffusion implicit models. arXiv:2304.03322.
14

---

Supplemental material
The appendix is organized follows:
Appendix provides additional details on the post- post-processing for NCP.
Appendix contains the proof' of the theoretical results and additional statistical results.
In Appendix C
comprehensive details presented regarding the experiment benchmark utilized to evaluate the performances NCP.
A Details training and algorithms
A.1 Practical guidelines for training NCP
It is better to choose larger d rather than smaller one. Typically for the problems we considered in Section we used d {100, 500}.
The regularization parameter y was found to yield the best results for E (10-2,10-3)
To ensure the positivity of the singular values, transform the vector w" with the Gaussian function x + exp(-x²) to recover o during any call of the forward method. The vector w is initialized at random with parameters following a normal distribution of mean 0 and standard deviation 1/a.
With the ReLU function, we observe instabilities in the loss function during training, whereas Tanh struggles to converge. In contrast, the use of GELU solves both problems.
We can compute some statistical objects sanity check for the convergence of NCP training. For instance, we can ensure that the computed conditional CDFsatisfies all the conditions be valid CDF.
After training, an additional post-processing may applied ensure the orthogonality of operators v and v". This whitening step is described in Alg 2 in App A.3 A.3 It leads to an improvement of statistical accuracy of the trained NCP model. See the ablation study in Tab. 3
A.2 Learning dynamics with NCP
Training
Validation
2
Loss
2
1000
1500
Epochs
Figure 4: Learning dynamic for the Laplace experiment in Section b
A.3 Whitening post-processing
We describe in Algorithm 2 the whitening post-processing procedure that apply after training.
15

---

Algorithm 2 Whitening procedure
Require: new data (X now): trained u² o" and v
Evaluate ux = u (Xuain) and by = e. (Yrain)
Centering:
ux + ux - (Xmin)) and by t uy E(v (Y
ux (xxiiag(o") and by Uydiag(o")
Compute covariance matrices:
Cx + u. "z/n
Cy + o/ur/a
CxY + ukuv/m
U, V, + SVD Cxr
if (Xeew, Ynew) is different than (Xuinn Yumii) then ux t (Xnew) E(u² (Xuuin) diag(o
t 8 (Neow) - E((o"Yurii)) diag(o")
end
Final whitening:
x c uxxxx 12U
UY f- uy Cy V2V return new new new
B Proofs of theoretical results
B.1 A reminder on Hilbert spaces and compact operators
Definition Given vector space H. we say a Hilbert space if there exists an imner product (.)) such that:
H is complete with respect to the norm |a||| v (x,, for all x E H.
An important example of an infinite- infinite-dimentn dimensional Hilbert space is L (R). the space of square- integrable functions w.r.t probability measure μ on IR with the inner product defined as (f.g) JR J (x))g()))))) p(dx).
Definition 2 (Bounded Operators). Let Hi and H2 be Hilbert spaces. A linear operator T Hi H2 is called bounded if there exists a constant C 2 0 such that for all x E Hi, the following inequality holds:
||Tæ|lHa S C|
The smallest such constant C is called the operator norm Tf denoted by [TT], and is given by:
[T
|T = sup
[ H1
Bounded operators are continuous and play a key role in functional analysis.
Definition 3 (Compact Operators). Let Hi and H2 be Hilbert spaces. A bounded linear operator T: Hi + H2 called compact for any bounded sequence {zn} Hj, there exists a subsequence { such that T' converges in H2.
Compact operators can be viewed as infinite-dimensione analogues of matrices with finite rank in finie-dimensional spaces.
A key result in the theory of compact operators is the existence of a singular value decomposition (SVD) for compact operators. The following is the statement of the Eckart-Young Young-Mirsky theorem:
Theorem 3 (Eckart- Young- MMrngyy. Let H, - H2 be a compact operator between Hilbert spaces. Then T can be decomposed as:
50
Σ
a. (1s uu u,)o,,
16

---

where {ui} C H1 and { C H2 are orthonormal sets, and o, are the singular values of which satisfy 0 2 02 ... 20. 0.
Moreover, for any rank-k operator Tr =Ex= oil, u.)us we have:
IT Till min
IT S|1,
rank(S)
where ||
is the operator norm induced by the Hilbert spaces.
B.2
Proof of Lemma 1
Proof of Lemmal
7 It follows from 5 and 5 that
(1A, [Drxx
(1A, (Drlx [D [Drxxla)
P|Y T - PlY E BJ
P E A]
P E A]
Next, by definition of the operator norm, we have
KIA (Dyix [D>ixIa)) s |Dy |ynss/lllspooo-ggo/////////////
=
=|Dy [Dyxxldllagoo>-rrox vP[X Al AlVP(Y B),
where the operator norm ||Dy|x |Dvixluullago>-1220 is upper bounded by 0241 by definition of the SVD of Dylx-
B.3 Proof of Theorem
Proof of Theorem 7 In the following, to simplify notation, whenever dependency on the parameters not crucial, recalling that (X, Y) and (X'Y") are two iid samples from the joint distribution p. will denote the vector-valued random variables in the latent (embedding) space as u² (X), (Y) and v' := v (Y'), as well as
w' = == (X'), the training loss simply as E [L, - Eu, u Eu', v - Ev, v' -Ev', S)].
σ" and S := Sg. Then, can write
Let us further denote centered features as T7 = u Eu and : = Ev, and the operators based on them as Ug: Rd > 1, (2t and Vg: Rd + ()) by
Uoz = 2 [u E(u 1 and Vo = _ (r -lb for 2 E Rd
and prove that Co(0) = |U liss 2tr(S Dy x Vg). Since we have that Ue JuL' and V. e = Vo. where Jn = 111 1x and J, = I 1y @ 1y are orthogonal projectors in L2 (2)) and (2)),, respectively, as well as U Vo DyxxD
Uü Ey consequently
my) Elu (X)E (Y) X]] | (X
U, XV.
that is [ F[]] E(] space. Recalling that Ug Ejuu and Vg Vo E(ve'] J are covariance matrices in the embedding space, similarly we get that U F((u Fu)(u - Fu)T] and VVV, [F(v Fv)(U Fb)
is simply centered cross-covariance in the embedding
are centered covariances. Thus, we obtain
(o) - E[(S' (20) "/CT:///SS0TTTTT((((//////"""""" |tt ((( = -2E(T T ST] +
TT)
which, by taking (X, Y) and (X', Y') to be iid random variables drawn from p. gives that (0 can be written
USt N'ST' + σ. ST uSt - 1 S uu ST'T S1/2 + S'/2 Sou" S1/2)
= [ 2 (1' St)² - (u - 5')S(T - w')] = E[LLo u',
= EE(Lolu Eu (X') - Eu (X'), (Y) - Eu) - Ey'(
17

---

which implies that Lo(0) - V - 2 (r Vo). Moreover, to show that (0)
Lo(O) ((0). It suffices to note that
U| -11 ur((UU 1)2) ()P))tr((UU 2U -11
ir(E[uu | -[E[u'u"
E(tr(uu Tu' "uuu -uu" -u'u +1]] = E [[u Tu'²² - ||u|2 - || -0122 + d,
as well as that ||/ = ||Eu|| (Eu) = Eu u', and apply the analogous reasoning for random variable ~ V.
Now, given
d 1, let us denote D, = Eieir ojug & in and
0) = D,-U.S.VV 2 Hs |
(23)
Then, applying the Eckhart- Young-Mirsky theorem, we obtain that
()) 2 Ei
=
with equality holding whenever (a,, u., x = (a; ut, nf),, p-almost everywhere.
To prove that the same holds for (0), observe that after expanding the HS norm via trace in 23) we have that
()(0) = 2r tr UDD + |V llss
and, consequently.
(0) = [UoSs 2tr(S 2 /o (0x D,)Vo).
Thus, using Cauchy-Schwartz incquality, we obtain
(0) - (o) < ( ||Sa (550 [DyxI -DDy
S Md where the constant is given by M = maxie (Ilu Euf ( 10, Evf (<:0)) < cc. So, ()) Md < (0) S (0) + Md and, since > d + 1 was arbitrary, we can take arbitrary large to obtain of + 0 and conclude that Lo(() 2 Enetg of² with equality holding when (of ut of)
and, therefore, (0) (0) S o (
(at, ut, vf), p-almost everywhere, since then Dr UžDr U3UnSo = So
diag(a1,,. 11 od))
Finally, we prove that ? and 0j > oa-1 assure uniqueness of the global optimum. First, if the global minimum is achieved o3 > o241 allows one to use uniqueness result in the Eckhart- Sokhart-Young- Mirsky theorem that states that Eie(4 of ot o = -Eeedd of of @ T.. But since, 7 >> 0 implies that R0) = 0, i.e. (uf (efreta C (2) and (rf) C LZ are two orthonormal systems in the corresponding orthogonal complements of constant functions, using the uniqueness of SVD, the proof is completed.
B.4 Proof of Theorem
Proof of Theorem 2 Let us denote the operators arising from centered and empirically centered features as J. Ú:: Rd (x) and Vo. V Rd - (2) by
Uoz = Tuu (x) Voz zT T(o" -E (V)))) and Úoz 2 =Ta Voz : 2 TT0
respectively, for z e Rd.
We first bound the error of the conditional expectation model as |Dyrjx 5 as follows.
||y||x - -6p = ||Dr|x
U,,,, -Ü
S 5 + Eq + 1U U + UU,S, S,
18

---

Next, using that || l < 1 and that centered covariances are bounded by uncentered ones, i.c. U 3 UgUs. we have
S - Vol = | + - Vol
$ 110 - Vo11] + 1 1 - Voll
< 101 1.00v 1/22 1V s 2E + Eg.
In similar way, we obtain
S - U3S.Vol = 1 + UgS.V. Vol
| Volll 0ll + 10,11V, Voll
& U0 V
oll + IVa Velll|Uall + 110 - 1ol - Voll
V Eo (1),, [x)](X]]] |E, E()C(())
+ [uE (X)]II [[E (Y)IIIIII
22 (6) [0)
where ||E,(u²] of Lemma To summarize, it holds w.p.a.l. 1 -
en (6) and ||Ey[0"] Elv (Y)]l (6) hold w.p.a.l. 1 in view
||Dy|x - xll S ob41 + Eg + 2/11 Eo(Es + =6)) [Em]6]]] =: (6).
(24)
By definition in and 14]. we have
Do B A) = - + (1 Dy x1B)
(1A, x 1B)
PLY E BIX A
[[]]
[1(1
and
Drix1B) (A.((Dy) - De D
(1 1ix 1x) E,[MAl
[[]x(X)]
E[1a(X)]
ELA [[]]
Note also that
(E(L /P|X e AJ. (|B((\)l))0)) VE(LB(Y)
/PlY E BJ. for any A E Ex and B E Ey and
(1A, (Dyjx ) & ||Dy|x - fxxllll a
Combining the previous observations, we get
[E E(In(Y)]]
||y| 1
IP B B A (B(
E[1e(Y))
E(1s(Y)]
KIA, Drx 1B)1
[1Al - (x)I
(25)
[m[1A]
E[1a(X)]
and
1(1 x
E[1a(x)]
(KIA, E[]a(X))
KK,, - xxx))
[1A
Ex[1A]
E[(A(X))
E[1A(X)
E(1B(Y))
10DixII+|IDyis - D% V E[Ia(X)]
[x(1A]
E[]b(YY)
(1 (1+1IDyix -D21x1).
(26)
[_[1
F[La(X)
19

---

where we have used that |Drixll & 1.
Similarly, have
PIY E X A)
(B A)
PIY E B
PB(
(1 Dr
(1 v 1B)
[[EA
(IA. (Drix 1
Ex (,(
+ (1 1
E,,1|E,(11
(IA. (Dr1x (
E, [
(1 D2 x1B)
-ALLA--EDAI (EXX))E) + (E) - E((n(())) E., [1 a [1]
(1 (Drix - 1
E.[nn|Eg|1s]
(1., D
- + F[LA(X)) E[L (27) E [[AA]E,(1B))
[1A]
Next Lemmas 3 and combined with (18 and elementary algebra give w.p.a.l. 1 - 26 that
[1] (X)]
[1] - [[]s(Y)]
(A))
2B)r(B)
[1A]
[x[1
and
[[1 E,
E[] 2,
(X) (Y))(
<4pp
Eg[1a]
E,[nA|E [1B]
also holds on the same probability event as above that
(1A, (Dyjx - Dg )1))
|Dyix- i ([]a(X)] (Y)
| - D2
<4
E2(1A|E,
VE[LA(X)JE(AA(T)) É.[1A] E,(18]
Combining Lemma and 24). we get with probability at least 5 that |Dy|x
B - S (5).
By a union bound combining the last two displays with 24
25 and (26). we get with
probability least 1 38
P B E A) (B| A)
(1) + [1 + (5)) [2px (A) 4py (B) (B)) (8)
(28)
P[Y E B
(B)
(X)
and
P B B X E A]
(By( 2(1 + (6))px (A)E, + + (0)
(29)
P E B
(X)
Replacing 5 by 8/3, we get the result w.p.a.l. 1 8.
The following result will be useful investigate the theoretical properties of the NCP method in the iid setting.
20

---

Lemma 2. Let Assumption I be satisfied. Assume in addition that 2 c.dd. Then there exists an absolute constant C such that, for any 8 E (0, 1), holds w.p.a.l 1
log(ed- n
log(ed-)
E EX)II S Caaad
n
Similarly, in 2 ci,d, w.p.a.l. 1 -
log(ed-1)
log(ed
- - Elv' S Cc., vd
+
Proof of Lemma 2
We note that
Eg[u
Elu (X)]
1 71
12
Zi
with Z, = (X,,) Ex,,),,, Vi E [n]].
We note that ||
S 2c,, v
=;
U and Var(Z,) Var([lu((X)) - Eluu
for any
i [n.]. We apply Minsker 2017
Corollary 4.1) to get for any 12 {U +
(X)).
f/2
P
Z
$ 28 cxp
(30)
no (X) t1
Replacing L by nt and some elementary algebra give for any 12 w.p.a.l 1-28exp (-t),
U2
36 xx(x)
a
a
Z
4U L $ 3 n
200 200(X)
71 :=1
12
Replacing t by t + e we get for any t 2 0, w.p.a.l 1- - 28 cxp (-t+ 2)2
a
AU cc 2x)
ltc
Z
n
n
n
Up rescaling of the constants, there exists a numerical constant C' such that for any 8 E (0, 1).
w.p.a.l. 1- 1-8
Σ Z sc
U (x0
f + + + xx0
i
Elementary computations give the following bound, that is, there exists a numerical constant C > such that for any > 0, w.p.a.l. exp(-t)
Ca vd
2 d
V 0
3/2 3/2
xxx)
Vdt + oo (X) n2
2i1l <C
C
n²
3/1
Under Assumption and the condition <d < 1, it also holds that <2(x) < 1 since oz(X) < ci, Consequently, the bound simplifies and we obtain for any t 1, w.p.a.f. 1 exp( -t)
n Σ S
( Vi)
where C possibly a different absolute constant from the previous bound. Taking / log ed-1 for any 8 E (0, 1) gives the first result. We proceed similarly to get the second result.
21

---

Control on empirical probabilities
We derive now a concentration result for empirical probabilities.
Lemma 3. For any A E Ex and any ō E (0, 1), it holds w.p.a.l. 1 -
[L[LA] [[TA(X)]] $2 log 28-1
log 20-1
VP(X AJ(1 - P(X E 1)) EAI
Assume in addition that P(X E A) 2
logg2
Then it holds w.p.all 8
[E E[(AAX)]
log 28-1
P
2
E(1A(X))
P
n
Proof. We note that
- - Σ 2
with 2, 1x(x) E[1A(X,),, e [n].
i=1
We note that Z,| S 2 and Var(Z;) = P[X E A] - P[X E A)). Then Bercu et al. 2015 2.9) gives w.p.a.l. 28
Theorem
[1A] [[1)(X) < log 8-
log 8-1
VP[X AJ(1 - P[X ADM
n
Dividing by E[ (X)] gives w.p.a.l. 1- 25
E [J E()A
2 log 8-1
2 loog( 5 1/1)/n]]] V (1 P[X E A))
(X)
P[X E A]
Replacing i by 8/2 gives the result for X. The result for Y follows from similar reasoning.
The same proof argument gives an identical result for Y. Lemma 4. For any B E Eyy and any 8 E (0, 1), holds w.p.all 1
log 28-1 + VP B[[1- P(Y B)) E,[1a] [[]](Y)] <
log 26-1
n
1
Assume in addition that P(Y E BB) 2 2
lox 26-1
Then it holds w.p.a.l 5
Eg - E[1](Y]] [Y)
log 28-1
P[Y E B) P[Y B]
I
B.5 Sub-Gaussian case
Sub-Gaussian setting. We derive another concentration result under less restricted sub-Gaussian condition on functions uº and v This result relies on Pinclis and Sakhanenko's inequality for random variables in separable Hilbert space, (Caponnetto and De Vito 2007 Proposition 2).
Let (x) ex
1. 2 O. We define the o/2-Orlicz norm of random variable
llllly inf { {c C (C>0EE
d (w) $1}
We recall the definition of a sub-Gaussian random vector. Definition 4 Sub-Gaussian random vector). centered random vector X E Rd will be called Gau--aassiann iff. for all Rd.
(X,, u) liz & ((X,u) (L22())
22

---

Proposition 1. Caponnetto and De Vito 2007 Proposition 2) Let A,, i E [n] be i.i.d copies of random variable separable Hilbert space with norm ||-I]. If there exist constants L 0 and >> O such that for every m 2 2, ||| m mm! then with probability at least 1 -
1
4 Vn
L
2 8
A EA 71 iE|n]
log
(31)
n
random variable) Lemma 5.5. in Vershynin (2011) Let Z be random variable. Then, the following assertions are equivalent with parameters K; O differing from each other by at most an absolute constant factor.
Lemma 5
1. Tails: P[|Z) > 1 S exp(1 - (2) for all t 2 0:
2. Moments:
(E|ZP) < K2yD all p 2
3. Super-exponential moment: Eexp( (22 K33) 2
A random variable Z. satisfying any of the above assertions called sub-Gaussian random variable. We will denote by K3 the
norm.
Consequently, a sub- Gaussian random variable satisfies the following equivalence of moments property. There exists absolute constant e 0 such that for any m 2,
(E|
cK3s <cky/m
Lemma 6. Assume that (X) [ and -100 (Y) Elu (Y) are sub-Gaussian with mub-Gaussian norm K. We set (X) == Var(Ilu (X) EXu] (Y) = Var(Ilu (Y) (Y)]])) Then there exists an absolute constant C such that for any d E (0, 1), holds w.p.a.l 1 5
[E - Fluo(X)11
vn 03()
C + n
K2
1og(25-1).
Similarly, w.p.a.l. 5
Vn ) + K2
C
| - S
log(28
n
Proof. Set Z = (x) Eu (X) and we recall that (X) == Var(Ilu - Elu((X)]l) We check that the moment condition,
E2mh
< 5m!Lm~203(X))
Vm 2,
2
for some constant L > 0 to be specified.
The condition is obviously satisfied for = 2. Next for any 2 3, the Cauchy-Schward Schwarz inequality and the equivalence of moment property give
1/2 EZTh m < (EZ (EZ4) 1/2 < 4K)o)(X)² (EZZ(m-2)) 1/2
Next, by homogeneity, rescaling Z to Z/K₁ we can assume that K₁ = 1 in Lemma 5 We recall that Z addition non-negative random variable, then for every integer p 2 have
P(Z 2 t) p-1 dt < (
1-12 pt dt ) ((()
cP
EZ"
With 2(m 2), we get that EZ < e(m 2) (m - 2) = e(m - 2)! = cml/2. Using again
Lemma 5 take L = cK for some large enough absolute constant > 0. Then Proposition [
gives the result.
23

---

B.6 Estimation of conditional expectation and Conditional covariance
We now derive guarantees for the estimation of the conditional expectation and the conditional covariance for vector-valued output Y E Rdx.
We start with general result for arbitrary vector-valued functions of Y. We consider vector-valued function / = (/1,....,hd) where h3 E (2) for any E [d]. We introduce the space of square integrable vector-valued functions (V,, R equipped with the norm
|||||
Σ jE|d
1 II 1oor
Next we can define the conditional expectation of h(Y) (Y())))) ..)(.))))) conditionally on X E A as follows
EL A] [E])((Y)] (1A. Dr(xh))
....,, E[ha(Y)) (IA, Drixha) P(X A)
P(X E A)
E[[(Y]] (1A 11 & Dyjx(t)
P(X E A)
We define similarly its empirical version
Fl((Y) [Eylhrl + (1 Do h1) (1))
(1A. /2 ha) Eş[1a]
[1Al ,...,E,lha)
(1Ai [1g & DiIx
= EEyl4] +
Ea[1a]
Assuming that h(Y) is sub-Gaussian, set
K (Y) (E]((Y) o² o²(H(Y)) : Var(I[<(Y) E[A())II.
Define
1 o² (b(Y)) log(38-1) 16 (0)
K2
n
(5/33/3) + 2(1 1/. (5/3)))) (A))e)((((
VP(X E A)
Theorem 4. Let the assumptions of Theorem sub-Gaussian Then we have w.p.a.l. 8 that
be satisfied. Assume in addition that h(Y) is 2
[εO E A] E[K(Y) X E A]lI 55
(32)
Proof. We have
VELOIXX -ELL(Y) AJlI
|rix - 1
< [E[h(Y) EAIII
| @ x
1
VP(X A)
EL[1al
P(X e A)
||Drix 1
< [|E[[(Y)| - EA1l
VP(X A)
P(X A) -)
+ VP(X A)(I + ||Dy|x - D yix ||41
(33)
E. E A)
Recall that |Dyjx| || $ 24 and Lemma 3 Hence, union bound we get with w.p.a.l. 25 that
[E[h(Y) X E (E))
[
[E[h(Y)) [[((()))_E,( [+]]+
(34)
VI((EA)
24

---

We now handle the first term
- We recall that similar quantity was already studied in Lemma 6 We can just replace u (X) by h(Y) E R to get the result since we assumed that ((Y) sub-Gaussian, Hence there exists an absolute constant C' such that w.p.a.l. -
C Vn
K
[EEL(Y) - E S
(2()
+ 1og(26-1))
Actually, handle the conditional expectation E[Y X E A] in more direct way. Set
(6) : log(6 dy)
log(6-14)
Corollary 2. Let the Assumptions of Theorem be satisfied. Assume in addition that Y is Sab--Gassian vector. Then for any (0, 1), it holds with probability at least 8 that
[E[Y X A) - E"IYXX E All - ()r(Cov(Y)))s,
||A||
+
VP(XEA) (6/3) + 2(1 2 (x33) =: =0 (O). (35)
Proof. The proof of this result is identical to that of Theorem up to b(Y)= Y E Rdv, Then, applying Bernstein's 's inequality on each of the d, components of E[Y] Y and union bound, we get w.p.a.l.
34
Now if we specify
||[M]] F. v tr(Cov(Y))) log(0-1dg + max Y ( 11
log(d-;dg)
jE(dg)
Using again Definition
we obtain maxxe(d,))) (3
(| 1l 2 Vl[Cov(Y)I vtr(Cov(Y)).
It follows from the last two displays, w.p.a.l -
5 (r)(Cov(Y))e,()),
(36)
A union bound combining the previous display with 34 gives the first result.
O
We focus the conditional covariance estimation problem. We first define the conditional covariance as follows:
Cov(V A) = Cov + (1,,,(14, @ Da,)) s Drix s b) h/PPX A
- (1 (14, @ Dy @ (1 [1, & (PIx) E A)) (37)
is a dy x d, matrix. We Note that (1A, @ 14,)) @ Dylxlp @ h) = ((1A, 0 (Y A) of the conditional covariance obtain similar decomposition for the estimator Cov Cov(VIX A) by replacing Dyjx by De pxx:
Cov (X E A) = Cov(Y) + (1 [11,, & 1a,) @ D E)/Es[AAl
- A1 @ ( @ (1 [1,, @ (E (nA)) (38)
We define the effective of covariance matrix Cov(Y) as follows:
r(Cov(Y))) tr( tr(Cov(Y))
[|Cov(Y]]l
We set for any E (0, 1)
|r(Cov(Y))
r(Cov(Y))
log(d-1)
log o
c (2) = Cov(Y))l
(39)
7
and
[E |
(0) - 2(1
/
VP E A
+ (1/4)) X AIl (6/4))
25

---

Corollary 3. Let the assumptions of Corollary 2 be satisfied. Then any & E (0, 1). it holds with probability least 8 that
Cov (YIX E - E AIIl
(0).
(40)
Proof. We use again the function ((Y) Y. We note in view of 37 38 that
||Cov (rI A) Cov(YIX E A)lL S |Cov(()) Cov(Y)ll
D
1/ix
)
+ (1(1A, (1d, & 122)) @
h hohl
P A]
( & E(MYY X A]-E")n(Y)) XEA) & EYTA(Y XEA]II, (41)
Next, we note that
D
Dyix
AI(1A (1d, @ (11)) 8
L L0DII
E
Eg[1a]
Drix
1/
]
$ 1111A, (Id, @ (l1)) &
h hi) |
PIX E A]
[x(1
De D [TA]
DylX
$ VPX E All
Σ |1gVvelle30)
P(X A
J.EEl,
(PrAA- P|XEA]
(
)
SVP[XEA
1+111
(PLXEA] E,(nA))
Σ
[yg{&ellag0)
(.kE(ds)
Remind that Y is sub-Gaussian vector. Using the equivalence of moments property of sub-Gaussian vector, we get that
| EEY & E
Vj. E [dy].
By union bound combining the last two displays with 24 and Lemma 3 we get w.p.a.l. 1 - 25
De D (L(T
Drix F[X E A]
[I(1A (Id, @ 11,) @
h h)ll
(E[1)Y
S (6) + 2(1 + (0))
(42)
VP[X A]
Next, we set n -= == X E A] and E X E A). Then we have
||u u - 8 i|| < ||u & [|[[||=]| + | < Iu @|](2||u|| + l| - ull)
We apply next Corollary
2 to get w.p.a.l.
J.
uu 2 u à & all
< ( (2) X e A]l + ++
(43)
Next Koltchinskii and Lounici 2017 Theorem 4) guarantees that w.p.a.l 8
||Cov(YY Cov(Y)ll S (6),
(44)
where ( (2) (6) defined in
union bound combining
43 and 444 gives the result.
26

---

C Numerical Experiments
Experiments were conducted on a pege-performance computing cluster equipped with an Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz Sky Lake CPU, 377GB RAM, and an NVIDIA Tesla V100 16Gb GPU. Code is available at at/tttps:/ggichub.oom COMCMML IIT-UCL/NCP
C.1 Conditional Density Estimation
To evaluate our method's ability to estimate conditional densities, we tested NCP on six different data models (described in the following paragraph) and compared its performance with ten other methods (detailed in Tab. 2) We assessed the methods' performance using the KS distance between the estimated conditional CDF and the true CDF. Additionally, we explored how the performance of each method scales with the number of training samples, ranging from 10a 10 with validation set of 10 samples. We tested each method on nineteen different conditional values uniformly sampled between the 5% and 95%-percet of p(r)). Conditional CDFs were estimated on a grid of 1000 points uniformly distributed over the support of Y. The KS distance between each pair of CDFs was averaged over all the conditioning values. In Tab. 5 we present the mean performance (KS distance + standard deviation), computed over 10 repetitions, each with different random seed.
Synthetic data models. We included the following synthetic datasets from Rothfuss et al. 2019 and Gao and Hastie 2022 into our benchmark:
LinearGaussian, a simple univariate linear density model defined as Y = X + N0 0.1) where X Uni
EconDensity, an economically inspired heteroscedastic density model with a quadratic dependence the conditional variable defined Y X 2 cy. ~ (O, X) where X ~ (0, 1)
ArmaJump, a first-order autoregressive model with a jump component exhibiting negative skewness and excess kurtosis, defined as
2t [c(1 - a) + az:-1] + (1 24)c2 + 2, [-3c + 262),
where é, ~ (0, 0.05) and 2, ~ B(1,)) denote a Gaussian shock and a Bernoulli dis- tributed jump indicator with probability p, respectively. The parameters were left at their default value.
Gaussi MaussianMitture, a bivariate Gaussian mixture model with 5 kernels where the goal is to estimate the conditional density of one variable given the other. The mixture model is defined p(X,Y) Σ k= T (Mk: Σx) where The M4k and Σg the mixing coefficient, mean vector, and covariance matrix of the k-th distribution. All the parameters were randomly initialized.
SkewNormal, univariate skew normal distribution defined as Y (x (2)XX(((( (ax where o(.) and (-) are the standard normal probability and cumulative density functions, and is a parameter regulating the skewness. The parameters were left their default value.
Locally Gaussian or Gaussian mixture distribution (LGGMD) (Gao and Hastie 2022). regression dataset where the target y depends on the three first dimensions of x. with seventeen irrelevant features added to x. The features of x are all uniformly distributed between -1 and 1. The first dimension of gives the mean of Y|X, the second is whether the data is Gaussian or mixture of two Gaussians, and the third gives asymmetry. More specifically:
0. 0.5, 0.250..5 + 0.5)²)
Y|X
(0.5 + 0.5, 0.5. (.55( - 0.5) if) < 0.2
(45)
(0.25. - 0.5, 0.3) (()) 0.2
To sample data from EconDensity, ArmaJump, GaussianMixture and SkewNormal, we used the Ltw.eaatiooooaleenssssteeeeaaai (Rothff (Rothfuss al. 2019) available at atthtpp::// /tstps://gitaae thub
27

---

Training NCP. We trained an NCP model with u" and v" as multi-layer perceptrons (MLPs), each having two hidden layers of 64 units using GELU activation function in between. The vector o" has size of d = 100, and is set to 10-3, Optimization was performed over 10º epochs using the Adam optimizer with a learning rate of 10-3, Early stopping was applied based on the validation set with patience of 1000 epochs. To ensure the positiveness of the singular values, we transform the vector o" with the Gaussian function æ + exp(-2²) during any call of the forward method. Whitening was applied at the end of training.
Compared methods. We compared our NCP network with ten different CDE methods. See Tab. for the exhaustive list of models including a brief summary and key hyperparameters. In particular, the methods were set up as follows:
NF was characterized by a 1D Gaussian base distribution and two Masked Affine Autore- gressive flows (Papamakarios al., 2017) followed by a LU Linear permutation flow. To match the NCP architecture, each flow was defined by two hidden layers with 64 units each. The training procedure was the same as for the NCP model. The model was implemented using the library normflows (Stimper et al.) 2023).
DDPM was characterized by a U-Net (Ronneberger et al. 2015), noise schedule starting from 10 to 0.02 and 400 steps of diffusion as implemented httts:://itthul com/ Diffusion MNIST
CKDE's kernels bandwidth was estimated according to Silverman's rule (Silverman 1986).
MDN's architecture was defined by two hidden layers with 64 units each and 20 Gaussians kernels.
KMN's architecture defined by two hidden layers with 64 units each, 50 Gaussians kernels, and kernels bandwidth was estimated according to Silverman's rule Sillverman 1986).
LSCDE was defined by 500 components which bandwidths were set to 0.5 and kernels center found via a k-means procedure.
NNKDE's number of neighbors was set using the heuristics k = /n (Dvrroye et al. 1996). Kernels bandwidth was estimated according to Silverman's 's rule (Silverman 1986). We used the implementation available at https:///ittut: http///tthub.ccm/eeeeeeeeeee group- wwt.mmwwaa////
RFCDE was characterized by a Random Forest with 1000 trees and 31 cosine basis functions. The training was performed using the rfcde library available at htttpa://githhie thub. com/ lee- Cmarooupccm/RROOC
FC was trained using Random Forest with 1000 trees as a regression method and had 31 cosine basis functions. The training was performed using the flexcode library available at https://githuu
LinCDE was trained with 1000 LinCDE trees using the LinCDE. boost R function from https:///itthu: http:///tuuu..cmm///jj/u////////
CKDE, MDN, KMN, and LSCDE hyperparameters were set according to Rothfuss et al. (2019) and were trained using the library Conditional Density Estimation available at https: //gi thub com/ freel Sreelunchtheorent cettmm/oonniiiinii wtreiiinnnnnnntaaneeeeeee Saansitt__sttiatiit. of a neural network were assigned the same number of epochs given to NCP. All other method parameters were set as prescribed in their paper.
methods involving the training
Results. See Tab. for the comparison of performances for 10 See also Fig. carried out an ablation study centering and whitening post-treatment for NCP in Tab.
We also
C.2 Confidence Regions
The objective of this next experiment is to estimate confidence interval at coverage level 90% for two distribution models with different properties (Laplace and Cauchy) and one real dataset order to showcase the versatility of our NCP approach.
28

---

Table 2: Compared methods for the CDE problem.
Method
Summary
Main hyperparams
Generative models that transform
Normalizing Flows (NF)
simple distribution into a complex one through a series invertible and diflerentiable trans formations Generative models that learn to generate data by reversing a gradual noising process, modeling distributions through
Architecture
Rezende and Mohamed 2015b
Flow type
Denoising Diffusion
Number of difflusion steps Noise schedule
Probabilistic Model (DDPM) (Ho et 2020
iterative refinement
Nonparametric approach modeling the joint and marginal probabilities
Conditional KDE (CKDE) (Li and Racine 2006
KDE bandwidth
KDE and computes the conditional
density p(y (r)) = p(z, w)/p(x))
Uses NeuralNets which takes conditional x as input and governs all the weights of a GMM modeling p(y(
Mixture Density Network (MDN)
NeuralNet architecture
Bishop 1994
Number of kernels
NeuralNet architecture
Kernel Mixture Network (KM))
Similar to MDN with the difference that
Method for finding kernel eenters
Ambrogioni ct al. 2017
NN only controls the weights of the GMM.
Number of kernels
Method for finding kernel centers
Least-Squares CDE (LSCDE)
Computes the conditional density as lincar combination of Gaussian kemels
Number of kernels
(Sugiyama al. 2010
Kernels' bandwidth
Nearest Neighbor Kernel CDL (MKKDE)
Uses nearest neighbors of the evaluation point to compute
Number of neighbors
[lzbicki et 2017
Kemel bandwidth
Freeman 2017
KDE estimation of y.
Uses random forest to partition the feature space and constructs a weighted KDE of the outpul space, bused on the weights of the leaves in the forest.
Random forest hyperparams
Random Forest CDE (RFCDE)
Basis syslem
PPospisil and Loc 2018
Number of busis
Nonparametric approach which uses basis expansion of univariate y turn CDR into series of univariate regression problems.
Flexible CDE (FC) Izbicki and Lec 2017
Number of expansion cocff's Regression method hyperparams.
Conditional training of unconditional machine learning models to learn density
LinCDE (LCDE) Gan and Hastie 2022
Number of LinCDE trees
Table 3: Ablation study on post-treatment for NCP. We report the mean and std of KS distance of estimated CDF from the truth averaged over 10 repetitions with n = 10sh (best method in bold red). NCP-C and NCP-W refer to our method with centering and whitening post-treatment, respectively.
Model
LinearGaussian
EconDensity
ArmaJump
SkewNormal
GaussianMixuure
LGGMD
NCP
0.040 + 0.007
0.014 = 0.003
0.046 + 0.012
0.023 + 0.006
0.027 + 0.008
0.055 = 0.010
NCP-C
0.019 = 0.006
0.010 = 0.003
0.037 0.011
0.015 = 0.004
0.015 + 0.004
0.048 0.007
NCP-W
0.010 +: 0.000
0.005 0.001
0.010 0.002 0.008 0.001
0.015 0.004
0.047 0.005
Table 4: Mean and standard deviation of KS distance of estimated CDF from the truth averaged over 10 repetitions with sample size of 10t (best method in bold red, second best in bold black). NCP-C and NCP-W W refer to our method with centering and whitening post-treatment, respectively.
Model
LincarGaussian
EconDensity
ArmaJump
SkewNormal
Mixture
LGGMD
NCP
0,046 0.011
0.021 0.009
0.048 : 0.009
0.043 0.029
0.035
0.188 0.011
NCP-C
0.031 : 0.008
0.019 0.008
0.038 1 0.011
0.031 1 0.013
0.031 0.003
0.189 0.012
NCP-W W
0.026 = 0.002
0.016 + 0.003
0.020 + 0.002
0.024 - 0.011
0.030 = 0.002
0.176 + 0.014
DDPM
0,414 0.341
0.264 + 0.240
0.358 + 0.314
0.284 + 0.251
0.416 0.242
0.423 0.223
NF
0.011 0.002
0.015 0.003
0.141 = 0.005
0.039 0.005
0.113 0.006
0.288 0.010
KMN
0.599 1 0.003
0.349 0.019
0.490 1 0.007
0.380 0.009
0.306 0.003
0.225 0.008
MDN
0.245 + 0.011
0.051 1 0.002
0.164 1 0.005
0.089 =: 0.002
0.144 0.009
0.232 1: 0.008
LSCDE
0.418 + 0.003
0.119 +: 0.004
0.250 1 0.007
0.109 0.002
0,201 +: 0.005
0.295 4: 0.034
CKDE
0.187 = 0.001
0,023 + 0.003
0.125 + 0.002
0,046 + 0.001
0.085 0.003
0.241 + 0.021
NNKCDE
0.090
0.060
0.063 0.006
0.002
0,059 0.004
0.207 0.013
RFCDE
0.132 = 0.009
0.136
0.130 + 0.009
0.139 + 0.009
0.134 0.012
0.162 0.006
FC
0.090 = 0.004
0.030 + 0.006
0.042 + 0.003
0.033 + 0.002
0.033 + 0.003
0.065 + 0.008
LCDE
0.122 = 0.002
0.029 + 0.003
0.118 + 0.003
0.064 + 0.007
0.050 + 0.002
0.141 + 0.004
29

---

Lincar LincarGaussian
EconDensity
ArmaJump
10
KS
10-3-
10-2
2
10
10a
10
10
10a
10
10a
10a
10
10
SkewNormal
MixtsianMixture
LGGMD
10-1
KS
10
10
10a
10a
10a
10a
10a
10a
10a
10
10a
10
10
10
NCP-M
NNKCDE RECDE
NCP-C
MDN
CKDE
Figure 5: Performances for CDE on synthetic datasets w.r.t sample size n. Performance metric is Kolmogorov-Smingg Smirnov (KS) distance to truth.
Compared methods. We compared our NCP procedure for building conditional confidence inter- vals to the state-of-the conditional conformal prediction method in Gibbs et al. (2023). We also developed another method based on Normalizing Flows' estimation of the conditional CDE and we added it to the benchmark.
Experiment for Laplace and Cauchy distributions. We generate a dataset where the X variable follows uniform distribution on interval [0, 5] and YIX follows either Laplace distribution with location and scale parameters (f(x)), b(x:)) = (x² x) or a Cauchy distribution with location and scale parameters (f(z), b(x)) (x2.13 + z². We create train set of 50000 samples, validation set of 1000 samples and test set of 1000 samples.
For the Laplace distribution, we train an NCP where u and o are multi-layer perceptrons with two hidden layers of 128 cells, の a vector of size 500 and y = 10-2 Between each layer, we use the GELU activation function. We optimize over 5000 epochs using the Adam optimizer with learning rate of 10-3 We apply early stopping with regard the validation set with patience of 100 epochs. Whitening is applied at the end of training. To fit the Cauchy distribution, we increase the depth of the MLPs to and the width to 258.
We compare this NCP network with two state-of-the-arr methods. The first a normalizing flow with base distribution 1D Gaussian and Autoregressive Rational Quadratic spline flows (Durkan cet al., 2019) followed by a LU Linear permutation flow. All flows come from the library normfloo lows (Stimper al. 2023). The spline flows have each blocks of 128 hidden units match the NCP architecture. The normalizing flow is allowed the same number of epochs as ours with the same optimizer. The second model the conditional conformal predictor from Gibbs et al. (2023). This model needs a regressor as an input. We consider a situation favorable to Gibbs et al. (2023) as we assume as prior knowledge that the true conditional expectation a polynomial function (the truth is actually the quadratic function in this example). Therefore we chose a linear regression with polynomial features as in Gibbs al., (2023) as this regressor should fit the data without any problem. For all other choices of parameters, we follow the prescriptions of Gibbs et al. (023). For the sake of
30

---

0.031
1.00
LinearGaus
polyyl))
3-
0.75
2-
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
100
9
1.00
EconDens itns
0.75
pel (1))))
0.2-
0.50
0.1
0.25
2.3
0.0
2.5
5.0
7.5
10.0
2.55
0.0
2.5
5.0
7.5
10.0
y
-0.31
00,02
0.18
0.056 KS
KS
0,032
1.00
Armma ArmaJJmp Jump
Ps Ps bx)
0.75
4-
0.50
0.25
0.00
0..4
0.2
-0.64
01
0.63
K8
0.026
KS
0.007
KS
0.020
1.00
10
0.75
SkewNor
Pio (x)
F
0.50
0.25
0.00
-3.17
1.78
6.80
K8
0.045
KS
0. 0.057
KS 0.012
1.00
Gaus wwnsssiamituurr
0.3
0.75
pel (z)))z)
0.2-
0.50
0.25
0.00
X1
21
0066
22
0.0
0.6
0.167
KS
0.133
0.127
1.00
6-
LGGMD
Fie (o)
Truc Estimated
-0.5
0.0
0.5
00.5
0.0
0.5
y
Figure 6: Estimated conditional PDFs (left) and CDFs (right) for each synthetic dataset for 3 different conditioning points. Dotted lines represent the true distributions, while solid lines represent the estimates from NCP. The average KS distance over 5 repetitions is also reported on the right plots.
31

---

fairness, we note that the validation set used for early stopping in NF and NCP was also used as calibration set for the CCP method.
By design, the Conditional Conformal Predictor (CCP) gives the confidence interval directly. However NCP and NF output the conditional distribution. To find the smallest confidence interval with desired coverage, we apply the linear search algorithm described in Algorithm on the discretized conditional CDFs provided by NCP and NF. The results are provided in Fig. First, observe that although the linear regression achieves the best estimation of the conditional mean, as should be expected since the model well-speeified in this case, the confidence intervals, however, are unreliable for most of the considered conditioning. We also notice instability for NF and CCP for conditioning in the neighborhood of x 0 with NF confidence region exploding z O. We expect this behavior is due to the fact that the conditional distribution at z = is degenerate. Comparatively, NCP does not exhibit such instability around O. It only tends to overestimate the produced confidence region for conditioning close to = 0.
Algorithm 3 Confidence interval search given CDF
Require: Y vector of values, Fy vector of realisations of the CDF at points Y, [0, confidence level
Initialize hoow and hngh Initialize fiow = and high -1 Initialize s* = 0O
while Center and scale Xirinn and Yirain do if Fy [fnghh] - Fy [frm 2 a then
size = Y [tiggh)
Y [1ow)
if size < s+ then
fjow llow ''igh Unigh,
=size
end if
hno
flow +
else if Itigh (Y) - then
break
else
fngh Lnigh +1 1 end if
end while
Return [t Y [Fnghh]
Experiment on real data. We also evaluate the performance of NCP in estimating confidence in- tervals using the Student Performance dataset available at https:://wwww.t.. k/w.k///////w..agle..... kaggle. com/ datasets/ stta117200/sudent-o.
This dataset comprises 10000 records, each defined by five predictors: hours studied, previous scores, extracurric- ular activities, sleep hours, and sample question papers practiced, with a performance index as the target variable. In this experiment, the NCP's u and n are defined by MLPs with two hidden layers, each containing 32 units and using GELU activation functions, " is a vector of size d = 50 and 7 10 Optimization was performed over 50000 epochs using the Adam optimizer with learning rate of 10 10-3, We compare NCP with a normalizing flow defined as above in which spline flows have each two blocks of 32 hidden units to match NCP architecture. The normalizing flow is trained for the same number of epochs as our model, using the same optimizer. We further compare NCP with split conformal predictor featuring a Random Forest regressor (RFSCP) with 100 estimators. We used the implementation of the library punce (Mendil et al. 2023). For NCP and the normalizing flow, early stopping is based on the validation set, while for RFSCP, the validation calibration set. We performed 10 repetitions, randomly splitting the dataset into training set of 8000 samples and validation and test sets of 1000 samples each. We report the results of the estimated confidence interval at coverage level of 90% in Tab. 5 The methods provide fairly good coverage. NF did not respect the 90% coverage condition. Only NCP and RFSCP both respect the coverage condition but the width of the confidence intervals for RFSCP are larger than for NCP.
ormance wwwpaannemultilleeeaee -eeroooonne-rrrresssisooonooeeee
serves as the
Discussion on Conformal Prediction. Conformal prediction (CP) is popular -odel-agnostic framework for uncertainty quantification approach Vovk et al. (1999). CP assigns nonconformity
32

---

Table 5: Mean and standard deviation of 90% prediction interval (PI) coverages and interval widths, averaged over 10 repetitions for the Student Performance dataset from Kaggle.NCP-CC and NCP-W refer to our method with centering and whitening post-treatment respectively.
Model
Coverage 90% Pl
Width 90% PI
NCP-C
89.41 % i: 2.12%
0.39 i: 0.02
NCP-W
91.02% + 0.72%
0.38 = 0.01
NF
89.10% + 1.07%
0.35 0.00
RESCP
90.03% = 1.06%
0.41 = 0.01
scores to new data points. These scores reflect how well each point aligns with the model's predictions. CP then uses these scores construct a prediction region that guarantees the true outcome will fall within with a user-specified confidence parameter. However, CP not without limitations. The construction of these guaranteed prediction regions can be computationally expensive especially for large datasets, and need to be recomputed from scratch for each value of the confidence level parameter. In addition, the produced CP confidence regions tend be conservative. Another limitation of regular CP is that predictions are made based on the entire input space without considering potential dependencies between variables. Conditional conformal prediction (CCP) was later developed to handle conditional dependencies between variables, allowing in principle for more accurate and reliable predictions Gibbs et al. (2023). CCP suffers from the typical limitations of regular CP and the theoretical guarantees.
C.3 High-dimensional Experiments
3 trained NCP for d 100 using the same MLP architecture and the same NF with autoregressive flow as in our initial experiments based 10 samples {(x, Y?) ) with values in RR V. We plot the conditional CDF for several conditioning w.r.t. ((x) on 10 repetitions. NCP paired with small MLP architecture performs comparably to the NF model for Gaussian distributions. For discrete distributions, the NCP demonstrates superior performance compared to the NF model.
Experiment high-dimensional synthetic data. In Fig.
We repeated the experiment in Fig. for d E {100, 200, 500, 1000} and recorded the aver- age -ommogorov-Smirov (KS) distance the NCP conditional distribution the truth, computation time and their standard deviations over 10 repetitions.
100
500
1000
mcan
std
mean
std
mean
1.0
0.057
0.019
0.079
0.050
0.062
0.016
1.57
0.041
0.009
0.069
0.042
0.049
0.017
3.14
0.030
0.016
0.116
0.173
0.036
0.010
5.0
0.067
0.052
0.131
0.175
0.072
0.036
Figure 7: Left: we observe only R 20% increase in compute time going from d = 10" to d 10". Right:
average KS distance to the truth and standard deviation over 10 repetitions.
High-dimenniionl experiment in molecular dynamies: Chignolin folding. We investigated the dynamics of Chignolin folding, using a molecular dynamics simulation lasting 106us and sampled every 200ps, resulting in 524, 743 data points. Our analysis focuses on 39 heavy atoms (nodes) with a cutoff radius of 5 Angstroms. To predict the conditional transition probability between metastable states, we integrate our NCP approach with a graph neural network (GNN) model. GNNs, as demonstrated by Chanussot et al. (2021), represent the state-of -f------tthttrr--te art in modeling atomistic systems, adeptly incorporating the roto- tr--ssaanaaionll and permutational symmetries inherent in physical systems. In particular, we employed SchNet model SShütt et al. (2019 2023) with three interaction blocks. Each block features a 64-dimensional latent atomic environment, and inter-atomic distances for message passing are expanded over 20 radial basis functions. After the final interaction block, each latent atomic environment is processed through a linear layer and then aggregated by averaging. The model underwent training for 100 epochs using Adam optimizer
33

---

with learning rate of 10 We employed batch size of 256 and set y to 10 In Fig. 2 we show how our NCP approach enables the tracking transitions between metastable states, demonstrating accurate forecasting and strong uncertainty quantification.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: abstract, Introduction and section Related works.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
see discussion under main results Section Theoretical guarantees and conclusion.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Yes each statement clearly state all required assumptions and all proofs are provided Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper presents the pseudo-code our method, links to the datasets and methods used to reproduce our results
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
While the paper is predominantly theoretical, we have presented experiments which illustrate our theory. Data and code can be made available upon request during the rebuttal and will be made readily available should the paper be accepted.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/nnis.co cc/
public//guidee/CodeStniiii http:///ooddseuisssiooooooioooio ssiiiioicyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
haappa/.c//pbltccgiidde////e/e SonerddeeeeooSSSemiseseee onolcccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix C provides all details the architecture used for our method in order to reproduce the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Whenever appropriate, we provided standard deviations for the performance of the compared methods computed over several repetitions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the information at the beginning of Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eururp..pee http://rrippsccppulllleeeeeeeeee cctppubllcc/thhcccccc...eeellll
Answer: [Yes]
Justification: This is theoretical paper. Experiments were carried out on synthetic data or publicly available data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Paper of theoretical nature. There no particular concern..
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: there is no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all existing methods used in experimental study were properly cited in the main paper and/or Appendix C.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssiii wwwppppppttthhhooo thcode www.coo//aaaa/o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This is theoretical work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: see above.
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: see above.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
40

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "QNfYm1dlwt", "reviewer": "Reviewer_n6sh", "summary": "The paper proposes Neural Conditional Probability, a novel operator-theoretic approach to learning conditional probability distributions by learning parameters of the truncated SVD of the conditional expectation operator with a neural network. The authors provide a rigorous mathematical derivation and argue for statistical guarantees of their method. The empirical evaluations require major improvements to an otherwise solid paper.\n\n**As a general note:** I do not consider myself an expert on the theoretical aspects of learning theory. My background is in Bayesian deep learning and simulation-based Bayesian inference. As such, my confidence regarding sections 3 and 5 is rather low, and my review shall mainly consult on the remaining sections that focus on presentation, embedding into other literature, and empirical evaluations.", "questions": "- As per your answer to checklist item 5 (Open access to data and code), I would like to request access to the full and reproducible code of the empirical evaluations.\n- Since you want to compare your method with state-of-the-art conditional density estimation methods: Why don't you benchmark against conditional flow matching?", "limitations": "- The authors mention limitations throughout the manuscript, which I appreciate. However, I would recommend adding a dedicated **Limitations** section in the conclusion to give a compact overview for readers who don’t engage with the entire paper in detail.\n- The performance of NCP in high dimensions might be a limitation, but the authors do not study this crucial setting.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The introduction is excellent, with a high degree of accessibility for the broader NeurIPS community and sound motivation of the proposed method.\n- The method seems mathematically rigorous, well-motivated, and sound.\n- The authors compare their method against a high number of competing algorithms in the numerical experiments.", "weaknesses": "## Major\n- The Related Work section does a good job of acknowledging related works that aim to learn conditional distributions. However, it utterly fails to embed the current paper into this research landscape. I recommend the authors elaborate on the precise similarities and differences between the referenced papers and their methods in the rebuttal period.\n- The empirical evaluations are limited to low-dimensional toy problems. This is a stark contrast to the introduction of the method, where the authors repeatedly list the curse of dimensionality as a drawback of other methods. While I acknowledge that the paper is situated in the area of operator learning and ML theory, the quality standard of NeurIPS is not met by the authors’ experiments. This weak evaluation does not do the remainder of the paper justice and I strongly recommend the authors overhaul the experiments to feature high-dimensional tasks that cannot be solved with other state-of-the-art methods. This constitutes a major revision, and this is the main reason why I cannot recommend acceptance to NeurIPS 2024.\n\n\n## Minor\n\n- The empirical evaluation is missing some important information for real-world applications: What are the approximate wall-clock times for (1) training and (2) inference of the competing methods? Further, the authors mention the large required training set size, which might also influence the practically expected training duration in real-world tasks.\n- Please fix the citations throughout the manuscript: Most citations are ‘text citations’ even if their embedding in the sentence warrants parenthesized citations (Author, 1976).\n- This is just a personal preference, no need to address it: The ‘paper organization’ paragraph at the end of the introduction does not add value and the space could be used more efficiently elsewhere in the manuscript.\n- The first sentence in the conclusion is incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thoughtful feedback on our submission. We appreciate your recognition of our contribution to the field of operator learning and ML theory which was the primary objective of our work. We would like to address your concerns regarding the empirical evaluations and provide additional context.\n\n## On general note\nThank you very much for giving us the perspective to understand your review. While we followed all your suggestions for the revision, we would like to stress that theoretical guarantees developed in Sec. 3 and 5 are the integral part of this paper, and the heart of key contributions. Without being too technical, let us just briefly summarize what we feel as the most important aspects:\n- NCP is based on  __deep representation learning__ by training two DNNs (architecture depending on the data modalities) via a new loss so that the learned representations (u’s and v’s) can lead to __inference of diverse statistics derived from the conditional probability__.   \n- For each of the considered statistics we __prove finite sample generalisation bounds in high probability__ that rely on the __concentration inequalities__ and __optimality gap from DNN training__.\n- To the best of our knowledge, this is __the first work__ to show how to __provably infer__ conditional probability distribution __using finitely many deep features__, so that the error bounds depend only on the effective problem dimension and  __not the ambient dimensions__.\n\nWe hope that this summary brings more light to the quality and impact of our contributions.\n\n## Weaknesses\n\n### Major  \n1. We thank the reviewer for raising this point. __We added a discussion in the global response to situate our NCP approach in comparison to the four known strategies__. For the reviewer's convenience, we focus here on a detailed comparison with the best contenders to our NCP method, as shown in Tab. 1: Normalizing Flows (NF) and FlexCode (FC) from Izbicki and Lee (2017).\nFlexCode (FC) is an example of the direct learning strategy for conditional density estimation (CDE). Our results in Tab. 1 demonstrate that NCP, which learns an intrinsic representation adapted to the data, consistently outperforms FlexCode, which relies on a standard dictionary of functions, in all but one example. In the exception, NCP ties with FC. This also exemplifies that FC is not efficient for capturing the data's intrinsic structure when the FC model is overly misspecified. Regarding Normalizing Flows (NF), which is used in a conditional training strategy, a major limitation is the need to retrain an NF model for each conditioning. Additionally, NF builds a density model over the whole ambient space, which may not be efficient for capturing the data's intrinsic structure. Finally, to the best of our knowledge, NF lacks theoretical guarantees when applied to UQ tasks where strong guarantees of reliability are required. For more details on this aspect for the NCP, see our response to the general note above. We will incorporate this discussion in a revised version of our paper.\n\n2. Thank you for this remark. __We added high-dimensional experiments focused on UQ tasks. While most are synthetic in order to explore, as requested, the scalability of NCP, one is a complex real-world problem of protein folding__, see also our global response. For the latter, we used a dataset that is considered as challenging for the task of inferring dynamics of the stochastic process governed by the conditional probability kernel. Applying NCP to this setting, we were able to infer this conditional density from learned GNN representations of Chignolin mini-protein ensemble. This allowed us to predict the expected average pairwise atomic distances between heavy atoms and infer its 10th and 90th percentiles. As reported in Fig. 3, folded/unfolded protein corresponds to small/large coverage that originates from less/more variations of the ensemble. To the best of our knowledge, this is the first work that was able to perform UQ to infer metastable states of Chignolin, an important contribution in its own right.\n\n### Minor  \n- We added an experiment on how time complexity and inference quality scales w.r.t. dimensionality of $X$. In the attached pdf we included Fig. 2 presenting compute times for training and statistical accuracy for conditional density estimation (CDE) using the KS-distance. \n- Thank you a nice suggestion, we will follow it.\n\n## Questions\n1. As requested, __we provided the AC with a link to an anonymous repository__, as outlined in the NeurIPS guidelines, that includes notebooks for reproducing all our experiments.\n\n2. We thank the referee for bringing this method to our attention. From our understanding, conditional flow matching is designed to efficiently train continuous normalizing flows in high-dimensional settings, particularly for image generation tasks. However, based on our review of the relevant literature, this method has not been explored for the purpose of UQ. In our benchmark, we used a discrete-time masked autoregressive normalizing flow, which performs well on our synthetic benchmark due to its universal approximation property [Papamakarios et al. Normalizing Flows for Probabilistic Modeling and Inference. JMLR 2021, 22(57):1−64]. We did not encounter any issues with training convergence for this model. Additionally, our primary objective is to provide a simple yet effective approach for the UQ tasks we investigated. For these reasons, we believe that incorporating this more complex model in our benchmark lies outside the scope of our study.\n\n## Limitations\nAs elaborated above, we added additional experiments both on synthetic and real data to __show that the NCP operator approach doesn't suffer from this limitation__. Indeed it scales without any issue to high-dimensional and more complex data, without degradation of inference quality. Lastly, as requested, we will collect all discussions regarding limitations in one paragraph.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pygczyp9lB", "reviewer": "Reviewer_BaSA", "summary": "The authors propose a method (Neural Conditional Probability, NCP) for learning a conditional distribution P(Y | X) from a finite sample from a distribution. The method is based on following observations: (1) it is sufficient to learn the conditional expectation operator E_{Y | X}[f](x) = E[f(Y) | X = x]; (2) the conditional expectation operator can be written as (an infinite) SVD decomposition which could be truncated at some point, so the problem reduced to learning the finite number of functions in the SVD decomposition; (3) the joint distribution density can be written using the functions from the SVD decomposition of the conditional expectation operator, which gives an optimisation objective for fitting the functions from the SVD decomposition using the sample from a joint distribution. The authors provide an extensive theoretical analysis of the proposed method as well as a simulation study on a few synthetic datasets.", "questions": "- I am wondering about the choice of the specific loss function in Eq. (6). Could, for example, the log-likelihood potentially be used here? If so, what are the advantages of using the Eq. (6) instead of log-likelihood?\n- The loss function in Eq. (9) is roughly speaking a regularisation term enforcing that the singular functions are orthonormal, is it correct? Could it be possible to build the neural nets with such properties by construction rather than enforce it by regularisation?\n- What do you think about the scalability of the model to more complex datasets than in Section 6? For example, conditional image generation. Do you expect issues applying NCP in such cases?", "limitations": "The limitations are sufficiently addressed (NeurIPS Paper Checklist)", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ An interesting, novel and theoretically well-motivated method addressing an important problem of conditional distribution estimation\n+ The method uses a fairly simple neural network (MLP) but achieves the competitive to the methods using much more complex architectures\n+ Thorough theoretical analysis on statistical properties of the proposed estimator", "weaknesses": "- Limited experiments restricted to synthetic data making it difficult to judge the potential applicability of this method\n- It would be nice to have a short summary on the main properties of operators, their SVD decompositions, etc. I could generally follow the presentation without major problems, but having a such an operators summary would have made it easier to read the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful evaluation and valuable comments. In what follows, we aim to address the highlighted weaknesses and respond to the reviewer's questions.\n\n## Weaknesses \n\n- __W1.__ Thank you for this remark. __We added several high-dimensional experiments focused on UQ tasks, while most are synthetic to explore, as requested, scalability of NCP, one is a very complex real-world problem of protein folding__, see also our global response. For the latter, we used a dataset that is considered as challenging for the task of inferring dynamics of the stochastic process governed by the conditional probability kernel. Applying NCP to this setting, we were able to infer this conditional density from learned graph representations of Chignolin mini-protein ensemble. This allowed us to predict the expected average pairwise atomic distances between heavy atoms and infer its 10th and 90th percentiles. As reported in Fig. 3, folded/unfolded protein corresponds to small/large coverage that originates from less/more variations of the ensemble. To the best of our knowledge, this is the first work that was able to perform UQ to infer metastable states of Chignolin, an important contribution in its own right.\n\n- __W2.__ We thank the referee for the suggestion. We will add a section in Appendix summarizing the fundamental properties of operator theory that we used to develop the NCP approach and refer to it at the beginning of Section 3. This reminder will include in particular the definition of operators on infinite-dimensional Hilbert spaces and the definition of compact operators. Finally we will state the Erchart-Young-Mirsky theorem which guarantees the existence of the SVD for compact operators.\n\n## Questions\n\n- __Q1:__ We thank the referee for the interesting question. Maximizing the log-likelihood function to find model parameters is a central method in statistics, offering strong theoretical guarantees for models satisfying proper regularity conditions. However, this method becomes computationally complex for deep learning models, as the log-likelihood function can be non-concave with multiple local maxima, making it challenging to find the global maximum. In contrast, our approach represents the joint distribution through an operator and uses the regularized NCP loss designed according to the best low-rank rank approximation for operators. This method has a strong theoretical guarantee, as proven in our Theorem 1, that the unique global minimizer of this loss is the true target operator. Additionally, the empirical NCP loss is smooth and can be evaluated with linear complexity in batch size and latent space dimension, making it compatible with standard deep learning training methods.\n\n-  __Q2:__ Another great question. We've been investigating an alternative to regularization that leverages the Singular Value Decomposition (SVD) properties of compact operators to encode orthogonality directly into the model architecture, rather than enforcing it through regularization. This approach is still under investigation, and it is not yet clear if it will surpass our current regularization method. Indeed, our current regularization approach offers two main advantages: \n    -  __Ease of Computation.__ The regularization term is straightforward to compute and integrates seamlessly into the loss function, ensuring scalability and full compatibility with mini-batching and existing optimization methods.\n    - __Theoretical Guarantees.__ As established in Theorem 1, the regularized NCP loss uniquely identifies the true operator, a guarantee that an alternative method encoding orthogonality directly into the model might fail to have. Namely, the orthogonality is defined via the true distributions and not the observed empirical ones. Thus, currently it is not clear how to properly encode it in the architecture and obtain the same level of theoretical guarantees.\n\n- __Q3:__ We added additional experiments both on synthetic and real high-dimensional data to illustrate that the NCP operator approach scales without any issue to high-dimensional, more complex data. Please see our global response for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2UDhxFJeKK", "reviewer": "Reviewer_vE36", "summary": "I am not qualified to review this paper", "questions": "I am not qualified to review this paper", "limitations": "I am not qualified to review this paper", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am not qualified to review this paper", "weaknesses": "I am not qualified to review this paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We outline our contributions and offer further context in the global response, hoping these additions will better highlight the value of our work on the  inference of conditional probability and uncertainty quantification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlKYRZxN6s", "reviewer": "Reviewer_6sLX", "summary": "This paper proposes Neural Conditional Probability (NCP), a novel operator-theoretic approach for learning conditional probability distributions. Extensive theoretical results are provided to support the optimization consistency and statistical accuracy of NCP. NCP can be used to extract conditional density and compute statistical measures such as conditional mean, variance, moments and CDF once it is trained. Experiments on a collection of conditional density estimation datasets are conducted to highlight the efficacy of NCP.", "questions": "Please see Weaknesses.", "limitations": "The authors have discussed the limitations of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is mathematically solid and well-organized.\n- This paper focuses on a fundamental problem of learning conditional distribution in statistical learning and introduces an effective and simplistic approach that outperforms baselines with more complex architectures.", "weaknesses": "- The proposed NCP method is not clearly motivated or introduced. In Line 49-50, the authors mention that NCP does not belong to any of the four aforementioned approaches. But how is NCP in contrast with them and in what aspects does NCP make improvements? I believe adding some intuitive explanations accompanying theoretical analysis would help improve the readability.\n- Some key concepts or methods are not clearly explained, which makes it hard to understand the contributions of this work. For example, why is learning *conditional expectation operator* considered useful? Are there any baseline methods that also learn expectation operators?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful evaluation and valuable comments. In what follows, we aim to address the highlighted weaknesses and respond to the reviewer's questions.\n\n## Weaknesses\n\n- __W1:__ We thank the reviewer for this comment. __We have added a detailed comparison of the NCP method to the main existing strategies in the global response__. In brief, while the operator approach is fundamentally different, it can be viewed as a significant improvement over direct learning strategies that rely on a pre-specified dictionary of functions. The NCP method, in contrast, learns the latent space representation (dictionary) that is best adapted to the data under investigation.\n\n- __W2:__ The conditional operator approach has been the core idea in the ML theory of kernel mean embeddings and conditional mean embeddings [A], and, more recently, dynamical systems [B]. There, the operator is estimated on a universal (infinite-dimensional) reproducing kernel Hilbert space, in order to transfer probability distributions of one marginal to another by encoding them into the mean of the canonical feature map. While this theory is rich, it is __essentially limited to infinite-dimensional feature spaces__, and, hence, suffers from scalability limitations and the need for experts to design specific kernels for application at hand. __How to build finite-dimensional data representations and inference models__ based on them, to the best of our knowledge, was __an open problem__. So, we hope that our  work on the operator approach to inference of conditional probabilities opens a new line of research into this promising methodology that offers several significant benefits. From a probabilistic perspective, our approach allows us to learn the true joint distribution of $(X,Y)$ without needing to specify a model, even when the relationship between $X$ and $Y$ is complex. This capability provides valuable insights into $X$ and $Y$. For instance, the independence of $X$ and $Y$ is conveniently captured by the nullity of the conditional expectation operator. Moreover, the conditional expectation operator enables us to derive essential objects such as the conditional PDF, conditional CDF, conditional quantiles, and moments, even in complex nonlinear scenarios, without prior knowledge about $(X,Y)$. From a computational standpoint, we utilize the spectral properties of compact operators along with the recent NCP self-supervised loss function to create a theoretically sound training loss. This loss function guarantees that the true target operator is the unique global minimizer (see Theorem 1), while being inexpensive to evaluate, facilitating fast and stable training.\n\n### References:\n[A] Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur and Bernhard Schölkopf (2017), \"Kernel Mean Embedding of Distributions: A Review and Beyond\", Foundations and Trends® in Machine Learning: Vol. 10: No. 1-2, pp 1-141\n\n[B] Kostic, V., Novelli, P., Maurer, A., Ciliberto, C., Rosasco, L., and Pontil, M. (2022). Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces. In Advances in Neural Information Processing Systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful evaluation of our paper. We appreciate all their comments and remarks, which we will incorporate in our revision. Before addressing each review in detail, we would like to point out some general remarks that apply to all of them. 

## Positioning

Our main focus is on advancing Uncertainty Quantification (UQ) to handle nonlinear structures effectively. Several high-stakes nonlinear conditional probability estimation problems, such as CVAR for financial regulators (Since the 2008 financial crisis, regulators have required banks to limit their risk exposure, necessitating accurate computation of conditional value at risk, which relates to the conditional CDF), meaningful uncertainty quantification for engineering design (e.g., to drastically reduce the number of expensive real-world prototypes and tests in favour of a more computationally driven process), and rare-events prediction for predictive maintenance, are typically formulated in state spaces of moderate to high dimensions and often involve large datasets with complex, nonlinear structures, such as data residing on manifolds or graph data. Our goal is to showcase that the NCP method can effectively handle such structured data, offering a flexible and scalable modelling general solution. Importantly, we provide solid and provable statistical guarantees, aiming to advance safe and secure AI applications.

## Contributions

NCP leverages operator theory, a rich and powerful area of mathematics, allowing us to gain deep insights into the structure of functional spaces including probability distribution spaces. To the best of our knowledge, the NCP approach integrating operator theory in a self-supervised learning framework represents deeply innovative idea with significant potential for a range of ambitious applications. In the current paper, we focus on presenting NCP in the simplest possible way to illustrate its core benefits. 

**Strengths of NCP**

1. **Versatility:** NCP combined with a simple and compact MLP architecture, consistently outperforms more complex models and, when it does not, reliably matches their performance. 

2. **Adaptation to Intrinsic Dimension:** NCP effectively adapts to the intrinsic dimension of the data, constructing a well-suited low-dimensional representation space without requiring prior knowledge. This results in a more efficient handling of the data's inherent structure.

3. **Efficiency in Training:** NCP involves only a single training phase for the joint density. Once trained, we can easily derive all conditional probabilities without additional retraining, streamlining the process. In addition, NCP is easy and fast to train. Indeed we prove that it also scales linearly with latent space dimension and sample size. Our new experiments confirm that NCP scales with the ambient dimension, with compute time increasing at most sublinearly with the dimension (See Fig 2).

Additionally, our method also feature the following theoretical and conceptual strengths:

4. **Complete Theory:** Our approach is supported by a robust theoretical framework, providing strong guarantees for performance and reliability in both training and UQ accuracy.

5. **Assumption Comparison:** The density assumption with respect to marginals for NCP is significantly weaker compared to many other methods like Normalizing Flows (NF), which usually require density with respect to Lebesgue measure. This makes our approach broadly applicable, for instance for discrete distributions, or complex structured data like manifolds.


## Situating NCP method in the landscape of CDE methods

While NCP does not strictly fall into any of the four categories described by Gao and Hastie (2022), it is perhaps closest to the direct learning strategy but represents a significant improvement. Unlike the direct learning strategy, which uses a pre-specified kernel or a dictionary of functions to approximate the target conditional density, NCP directly learns the joint density by finding a low-dimensional latent space representation that efficiently adapts to the data. This is illustrated in our experiments where NCP demonstrated its versatility in handling diverse data including manifolds, graphs, continuous and discrete distributions (see Fig. 1, 2 and 3 in the attached pdf). Moreover, by learning proper representation adapted to the data, NCP can capture the intrinsic dimension of the data, which is reflected in our learning guarantees that depend on the latent space dimension rather than the ambient space dimension, even in the absence of prior knowledge. This theoretical foundation and added experiments (see our next point) demonstrate that NCP is capable of handling high-dimensional data. 

## Challenging Experiments 

As requested, we conducted additional experiments to further demonstrate the robustness and versatility of our approach across different dimensionalities and data structures. We have reported the results in the rebuttal pdf file and provided the AC with a link to an anonymous repository containing all code necessary to rerun our experiments.

We conducted high-dimensional experiments on synthetic data with discrete and continuous distributions scenarios, where the joint distribution is governed by a hidden low-dimensional manifold in a higher-dimensional space. Results in Fig. 1 and 2 in the attached pdf include KS-distance and computation times across various dimensions and conditional CDF plots for continuous and discrete distributions parameterised by a manifold. Our comparison of NCP and NF shows that NCP performs well and scales effectively in high-dimensional settings.

Additionally, an experiment on a real-world challenging application on folding dynamics of a protein is performed. There NCP is combined with a graph neural net architecture to forecast the quantiles of a physically relevant observable and infer the change from folded to unfolded state, see Fig. 3 for details.

Meta-review: Intuitively, the idea of this paper is very simple—essentially: Learn neural networks that transform x and y to some feature space, so that the joint density is the inner-product of the feature spaces. This has an obvious problem, namely that the result may not be normalized (or even positive). Seemingly for this reason (though the issue is never really acknowledged in the paper) a quadratic loss is used that essentially measures the squared difference between the learned "density" (not guaranteed to be normalized or positive) and the target. Then, an extra regularizer is introduced that encourages these feature spaces to be orthogonal. A high-probability bound result is given.

Reviewers were uniformly in favor of accepting this paper. The method appears to be novel and the results correct. But I do have some concerns that the presentation greatly obscures the simplicity of the underlying method. It seems perverse to not first describe the method in simple terms before giving the measure-theoretic motivation. (Which, after all, is just a motivation.) The theory makes quite strong assumptions (as is basically necessary given the presence of neural networks inside the method. As a service to the community, I urge the authors to be more clear about the simplicity of what is being proposed, and not to "hide" the simplicity of what is actually being proposed behind so much measure theory.
Decision: Accept (poster)