GDR-learners: Orthogonal Learning of Generative Models for Potential Outcomes
Authors: Valentyn Melnychuk
Stefan Feuerriegel
Venue: ICLR 2026 Poster (2026)
Keywords: causal machine learning
orthogonal learning
deep generative models
potential outcomes estimation
Primary area: causal reasoning
TL;DR: we introduce a general framework of generative Neyman-orthogonal (doubly-robust) meta-learners that target at the covariate-conditional distributions of potential outcomes (GDR-learners)

Abstract: Various deep generative models have been proposed to estimate potential outcomes distributions from observational data. However, none of them have the favorable theoretical property of general Neyman-orthogonality and, associated with it, quasi-oracle efficiency and double robustness. In this paper, we introduce a general suite of generative Neyman-orthogonal (doubly-robust) learners that estimate the conditional distributions of potential outcomes. Our proposed generative doubly-robust learners (GDR-learners) are flexible and can be instantiated with many state-of-the-art deep generative models. In particular, we develop GDR-learners based on (a) conditional normalizing flows (which we call GDR-CNFs), (b) conditional generative adversarial networks (GDR-CGANs), (c) conditional variational autoencoders (GDR-CVAEs), and (d) conditional diffusion models (GDR-CDMs). Unlike the existing methods, our GDR-learners possess the properties of quasi-oracle efficiency and rate double robustness, and are thus asymptotically optimal. In a series of (semi-)synthetic experiments, we demonstrate that our GDR-learners are very effective and outperform the existing methods in estimating the conditional distributions of potential outcomes.

Published as a conference paper at ICLR 2026
GDR-LEARNERS: ORTHOGONAL LEARNING OF GEN- ERATIVE MODELS FOR POTENTIAL OUTCOMES
Valentyn Melnychuk& & Stefan Feuerriegel LMU Munich & Munich Center for Machine Learning Munich, Germany me meInychuk@ 1mu.
ABSTRACT
Various deep generative models have been proposed to estimate potential outcomes distributions from observational data. However, none of them have the favorable theoretical property of general Neyman-orthogonality and, associated with it, quasi- oracle efficiency and double robustness. In this paper, we introduce general suite (doubly-robust) learners that estimate the condi-
of generative deyman-orthogome tional distributions of potential outcomes. Our proposed generative doubly-robust learners (GDR-learners) are flexible and can be instantiated with many state-of- the-art deep generative models. In particular, we develop GDR-learners based on (a) conditional normalizing flows (which we call GDR- (DFs), (b) conditional generative adversarial networks (GDR-CGANs), (c) conditional variational autoen- coders (GDR-CVAEL) and (d) conditional diffusion models
Unlike the existing methods, our GDR-learners possess the properties of quasi-oracle efficiency and rate double robustness, and are thus asymptotically optimal. In a series ssemi-)syntheite experiments, we demonstrate that our GDR-learners are very effective and outperform the existing methods in estimating the conditional distributions of potential outcomes.
INTRODUCTION
Causal machine learning (ML) is widely used to predict potential outcomes (POs), namely, the outcome after an intervention. medicine, for example, an accurate prediction of POs can guide the choice of the optimal treatment from several available treatment options (Feuerriegel et al. 2024)). The POs have a central role in the causal ML as they define various causal quantities, such as treatment effects (Curth & van der Schaar 2021) policy value (Qian & Murphy 2011 Frauen ct al. 2025b).
Recently, many works have suggested departing from estimating simple conditional averages of the POs (CAPOs) and rather aim at the whole conditional distributions (Louizos et al.) 2017] Yoon et al.,
2018 Zhang et al. 2021] Vanderschueren al. 2023 Ma al. 2024 Wu et al. 2025) In this way,
one can capture the inherent randomness of the PO (see Fig. I namely, its aleatoric uncertainty.
The aleatoric uncertainty is particularly important for reliable decision-making (Spiegelhalter 2017,
van der Bles et al., 2019): by knowing the whole distribution of the PO, decision- ceciiion-makers makers such as clinicians may be able evaluate the probabilities of unwanted outcomes. In this work, we thus focus on learning the conditional distributions of potential outcomes (CDPOs).
Various state-of-the-art generative models have been developed (or can be adapted) to model the Kocaoglu et al. [2018] Yoon et al.) 2018 Pawlowsk et al.) 2020) Sauer
CDPOs (Louizos et al. 2017:
Geiger 202 Zhang et al.
2021 Sanchez & Tsaftaris 2022; Ribeiro al. [0233] Vanderschueren Wu et al. 2025). These works mainly differ in (a) the underlying
et al. 2023 Ma et al.) 2024
probabilistic models (e. g., variational autoencoders (Louizos et al. 2017) Zhang et al. 2021 Ribeiro
et al. 2023),, , diffusion models (Sanchez & ssaataris 2022 Ma et al. [2024] Wu et al. 2025). etc.);
and in (b) their learning objectives (c. g.. plug-in in /Vanderschueren et al. 2023 and inverse propensity
of treatment weighted (IPTW) losses (Ma et al. 2024)).

---

Published as a conference paper at ICLR 2026
Conditional average potential outcome (CAPO)
Conditional distribution of potential outcomes (CDPO)
E
aatcome..
P( #)
B 2)
P x)
Perier IaaISIeES
Palent chamohorr,
Figure 1: Capturing the uncertainty in POs helps make reliable treatment decisions. Unlike conditional average potential outcomes (CAPOs, left), conditional distributions of potential outcomes (CDPOs, right) allow for quantifying the aleatoric uncertainty of the POs. As a result, they capture more information about the potential outcomes, such as heavy tails or multi-modalities.
However, these works rarely focus the optimality of the overall learning procedure. In fact, to the best of our knowledge, none of the methods fulfill general desirable asymptotic optimality of the
Foster &
Syrgkanis 2.23). Nevertheless, Neyman-orthoognality loss, and often comes with several favorable related properties such as quasi-oracle efficiency and rate double robustness.
In this paper, we introduce a novel suite of Neyman-orthogonal doubly-robust learners that learn the CDPOs. We refer to our learners as GDR-learners Our Reaa-eeassess proceed in two stages: in the first stage, the nuisance functions are estimated (i.e., conditional outcome distribution and the propensity score); and, in the second stage, the target generative models are fitted with a doubly-robust target objective. In this way. the target generative model class can be chosen arbitrarily. For example, one can even make restrictions due to the fairness or interpretability constraints.
Furthermore, we show that, due to the GDR-leamners also have favorable asymptotic properties: quasi-oracle efficiency and the rate double robustness, Here, quasi-oracle efficiency (Nie & Wager 202T) allows for learning of the target gener- ative model as if the ground-truth nuisance functions are known (even when the nuisance functions converging slowly). Further, rate double robustness (Kang & Schafer (2077) compensates for slow convergence speed of one of the nuisance functions with a fast convergence of another. Hence, GDR-learerrs are in (some sense) asymptotically optimal.
and under some mild conditions,
Our GDR-learners can be instantiated with many state-of-the-art generative models. In this work, we introduce four variants of our GDR GDR-learners: (a) conditional normalizing flows (GDR-CNFs); (b) con- ditional generative adversarial networks (GDR- CGRNs, (c) conditional variational autoencoders
CV)Es,,,,), and (d) conditional diffusion models (GDR-CDMA
Altogether, our contributions are as follows 2 (1) We introduce a novel framework of Neyman- orthogonal doubly-robust learners, called GDR-learners, that aim at estimating the conditional distributions of potential outcomes (CDPOs). Our GDR-learners are thus both quasi-oracle efficient and doubly-robust. (2) We instantiate our GDR-learners on top of four generative models, namely, CNFs, CGANs, CVAEs, and CDMs. (3) In several numerical experiments, we demonstrate the effectiveness of our Neyman-orthogonal learners in comparison with the existing methods.
RELATED WORK
Here, we discuss the most relevant existing works that deal with learning the CDPOs. We provide further relevant works in Appendix A
Meta-learners for CAPOs. wide range of meta-learners (Künzel al., 2019 were proposed
for predicting the CAPO (see the overview in Vansteelandi & Morzywotek (2025)). They include
different plug-in in approaches (Johansson et al. [016] Shalit et al. 2017 Johansson al. 2022). IPTW learners (Hassanpour & Greiner [209 Assaad et |l,] (021] 1). and Neyman-orthoognal learners (e. g.. DR-learner (Kennedy) (2023) and i-learner (Vansteelandt
Morzywoiek) 2025)). Yet, our paper
focuses different, much more challenging causal target: the whole conditional distribution of the POs (=CDPOs).
Ma et al. (202)) suggested a Neyman oorhoooal leamer for the CDPOs, yet under the condition that the class of generative models includes the ground-truth CDPOs.
*Code available at athttps / thub. com / ent 1997 /grr- 1eserr-eeareee learners
2

---

Published as a conference paper at ICLR 2026
Distributional treatment effects. Distributional treatment effects are related to the CDPOs: They also go "beyond the mean" and target either (i) some distributional aspects of the CDPOs or (ii) dis- tributional distances between the CDPOs. In (i). for example, plug-in in and doubly-robust learners
were proposed for point-wise cumulative distribution functions (CDFs) quantiles (Zhou et al. 2022
Kallus & Oprescu [023 Balakrishnan et al.,, 2025: Wu et al.
2023 Ham et al. 2024), and other
general statistics (Kallus & Oprescu, 2023) of the CDPOs In (ii) doubly-robust learning were
implemented in (Park et al., 2021; Chikahara et al. 2022 Naf et al. 2026). Yet, both (i) and (ii) are
not applicable in our setting. Specifically, methods in (i) learn the CDPOs only at specific grid points of the outcome space and, therefore, do not generalize well to the high- -igh-nimensinnal outcomes; and methods in (ii) target at the entirely different causal quantities (distributional distances between the CDPOs). Conversely, in our work, we aim to learn the best projection of the ground-truth CDPOs on some predefined generative model class without using any grid of points.
Learning the CDPOs with generative models. Methods that estimate the CDPOs from the obser- vational data can be split into two major categories based the main learning target. Methods in category (1) learn the whole causal data-generating process (DGP) (and the CDPOs along the way), while the methods in category (2) specifically target the CDPOs. We review both in the following. + Our contribution is located in category (2).
(1) Generative modeling of the whole causal DGP. Numerous methods were proposed to model the whole causal DGP (see the overview in Komanduri et al. (2024)) They differ in the generative models they employ (see Table IT) and in the underlying causal assumptions. For example, Louizos et al. (2017): Zhang et al. (2021) rely on the potential outcomes framework (= our setting); while Kocaoglu al. (2018): Pawlowski al. (2020); [Sauer& & Geiger (2021) |Sanchez Tsaftaris (2022); Chao et al. 2023); Ribeiro et al. (2023) assume a specific unconfounded casual diagram or structural causal model. principle, the latter methods can also be adapted to the potential outcomes framework (e. g.. by clustering variables together (Anand et al. 2023)), yet all of the methods in this category are in fact plug-in learners. More importantly, as we will show later, modeling the whole causal DGP is inefficient if we are only interested in learning the CDPOs.
(2) Generative modeling of the CDPOs. Works in this category are relevant to our method as they rely on the potential outcomes framework 3 Here, the existing methods (Yoon et al. 2018 Vanderschueren et al. 2023: Ma et al. 2024 Wu et al. 2025) vary in the underlying generative (probabilistic) models and in the type of meta- lecanees (see Table I. Ma et al. (2024) suggested an IPTW-learner based on conditional diffusion models that can be
but only under
special conditions that the target model class includes the ground-truth CDPOs, which we will clarify later (hence, we refer to it as "partial" Neyman-orrhoognnalyy
Research gap. To the best of our knowledge, there is no general Neyman-orthogonal (doubly- robust) learner that (i) targets the CDPOs and (ii) can be instantiated with any state-of-the-at generative modef") In particular, a method that has the favorable theoretical property of general Neyman-orthooonnlliy and, associated with it, quasi-oracle efficiency and rate double robustness is missing.
3 CONDITIONAL DISTRIBUTIONS OF POTENTIAL OUTCOMES (CDPOs)
Notation. Let capital letters X, Y denote random variables and lowercase letters x, a, y their realizations from the domains X, A. Let (Z) represent the distribution of random variable Z, and let P(Z = 2) denote its density or probability mass function. We denote the Ln-norm of a random variable ʃ(Z) as |I Ly = (E))( The empirical average of the random f(Z)
is written as F, F {z[) = n D [zz), where is the sample size. The propensity score is
defined as (2) = (PA = X = x). Conditional densities of the outcome are denoted by (y x) = P(Y = y X x, A a). For other conditional distributions and expectations, we often use shorter forms, for example. E( [x) = E(Y X = r). Throughout this work, we utilize the Neyman- Rubin potential outcomes framework (Rubin 1974). Specifically, Y[a) denotes the potential outcome under the intervention do( A = a).
'The potential outcomes framework makes minimal necessary assumptions for the CDPOs identification but not for the whole causal DGP (Barcinhoim & Plecko 2025). 'Here, we consider deep parametric models that model the density or the generative process of the CDPOs.
3

---

Published as a conference paper at ICLR 2026
Table I: Overview of existing generative methods for the estimation of the CDPOs.
Target
Work
Generative model
Leurner
Neyman orthogonality
type
Panial
General
CDPOs x DGP
CausalGAN bocaogiu ot 2018 CGN Sauer & Goiger (021)
CEVAE Louizos 2017 TEDVAE Zhana et 2021
GAN
VAE
Plug-in
Plug-in
x
X
X
DSCM Paw
VAE/NF
Plug-in
Dift (aachee sallers 2023
DCM Chao al. 2023
DM
Plug in
X
X
CausalHVAE [Cibearo
VAE
Plug-in
GANITE Yoon et 2018
GAN
RA
X
NOFLITE Cameenehieeee at
2023
NF
Plug-in
X
DifiPO (Ma ar 2024
IPTW
X
PO-Flow Wa 2025
NRDM
Plug-in
GDR-learners (our paper)
any
DR
Generative model: VATE variational autoenodder; GAN: generative adversania network; NI; normalizing flo;; diffuston model.
Leamer rype: RA: regression-adjustee;; IPTW: inverse propensity of treatment weighted; DR: roubly.
Problem setup and causal estimand. To estimate the CDPOs, we use an observational sample D D= [(2,, 0z, y,));= sampled i.i.d. from P(X,A,Y) Here, X X' C Rd are high-dimensional pre-treatment covariates,
E 1} is a binary treatment, and Y E E C TRd, is a continuous outcome. For example, in cancer therapies, Y can be tumor size, A indicates whether radiotherapy administered, and X contains patient covariates such as age and sex. We are then interested in estimating the conditional distribution of potential outcomes CDPOs, namely P(Y(a) | v)
(here V C X), with any generative model of choice. Sub-setting the covariates (V C X) allows for imposing fairness interpretability constraints.
Causal assumptions and identification. To consistently estimate the CDPOs, we rely on assumptions
of the potential outcomes framework (Rubin, 1974). that standard for causal MI (Curth & van der Schaar 2021 Kennedy [2023] Morzywolek et al. 2023 Vansteelandt & Morzywolek] 2025
Melnychuk et al. ).23). Specifically, we assume (i) consistency: Y[A] Y: (ii) strong overlap:
P(E < (XX) < $) = 1 for some E > O, and (iii) unconfoundedness: [O] Y [ LL A X.
Then, under the assumptions (i)-(ii), the density of the CDPOs is identifiable follows:
[aY[a] = y =) [P = y X, A a) V o| (l( V == v).
(1)
and, in the case of V X, we have PY [a] = y X = x) = E. (y .r)
In this work, we aim to learn the CDPOs with some generative model that takes the covariates V C X as an input and models the distribution of Y [a] with either (i) an explicit density or (ii) implicitly, as a data-generating process.
Outlook. In the following, we develop a theory of Neyman-orthhognnl learning for the CDPOs. More, specifically, we aim to derive a larget risk L for a generative model of choice Ia with the following key property of a
D,,,, 1)19. - 9h. - n| = for all Ja G and f E H.
(2)
where D. are path- path-wise derivatives, 9h the best approximating model in a model class G. and n, i E H are the ground-truth and estimated nuisance functions, respectively. Thus, informally, Neyman-orrthognnalty means first-order insensitivity of the gradient of the target risk wrt. the misspecification of the nuisance functions. We will further show that for Neyman-orthogonal risks of the CDPOs, two important properties hold: (a) quasi-oracle efficiency and (b) double robustness. Specifically, (a) the L2 error of the fitted model, l| فوllg. only depends the higher-order errors of the nuisance functions, |/v f and (b) this higher-order error consists of both the errors of the propensity score and the conditional densities of the outcome: || myy-mi -ill
IlEa Ealles I| - Rall,,- We refer to Appendix B.2 for exact definitions and further details.
4 NAÎVE LEARNING OF CDPOS
Target risk. To learn the CDPOs for the treatment a with a generative model of choice, we want to find the best projection (wrt. distributional distance) of the ground-truth CDPOs on the predefined generative model class G = (g. 2 v) : y Z x V + R.). Here, (-)-) explicitly or implicitly contains estimated conditional density of Y[a), and Z is auxiliary latent variable with assumed
4

---

Published as a conference paper at ICLR 2026
Table 2: Base generative models for our GDR-learners. Here, we consider the following e- conditional generative models & for estimating CDPOs, i.e. F(Y[a] v). We highlight the parts of the models Je that generate the estimated potential outcomes with orange. We refer to Appendix B.] for the details regarding the correspondence of different target risks to the minimization of certain distributional distances (last column).
Generative model
Optimization of C +> Projection of P(Y v) wit.
9a(8. #
Z
Ex
(a) CNF
Ba(y m]
Z B
maxp, KLD min,, maxd, L + JSD
(b) CGAN
da(y u) (1 da x))
2 E Rg,,
(a(2)
(c) CVAE
Pa(U22 /q.(2 v. v) (r. /9.(=
Z E Igd,
(a(2 v. (a (0)
m.. L + KI.+II IG maxp. L. + KLD IG
(d) CDM
Zi.7 E Ig
Purgu: Gu: conditional densities; da: conditional discriminator; fa: conditional generator
KLD: Kullback Leibler divergence; JSD: Jensen Shannon divergence: IG: inference gap
density P(Z = 2) = (z). Then, the best projection 9. can be found by minimizinggmaxxiiiznng the following target generative risk wrt. ga:
gir arg min / aarg max (gaa)
=(ga)=
E logga V)], 2~=,,
9.E9
S.E9
where E is some sampling distribution defined on (can be ha(2) or other part of ga). This target risk, due to the identifiability assumptions (i)-(ii), is equivalent to the following:
((g.) = =E = [
L logg. LzNe,
v)
E. (vlx)) dy = [(A=a] a}
E log (Y, 2 [v)
(4)
(X) Zmee
We refer to Lemma in Appendix Seanndixfforr for a proof of the equality above. The target risks in Eq. 3
general in the sense that they characterize many state-of-the generative models. For example.
by varying fa. Z, and 6g. we can define (a) conditional normalizing flows (CNFs) (Rezende &
Mohamed [015] [Trippe & Turmer 2018); (b) conditional generative adversarial networks (CGANs)
[Goodfellow et al. 2014: Laria et al.) 2022); (c) conditional variational autoencoders (CVAEs)
(Kingma & Welling [2014] Oh & Peng 2022); ; and (d) conditional diffusion models (CDMs) (Ho)
et al. 2020 Lutati & Wolf, 2023). We provide short descriptions of all the models and their risks in
Table and full definitions in Appendix B.I
Plug-in learners. When V = X, our causal estimand coincides with the conditional outcome
distribution (y xr), see Eq. [ Hence, one might be tempted to naively learn (y x) with a following plug-in loss:
(o((gw) (1,( a} E log go (Y, l
Z=c;
In treatment effect estimation literature, this approach is known as a plug-in learner (Künzel et al. 2019; Morzywolek et al. 2023, Vansteelandt & Morzywolek 2025). Yet, the plug-in risk differs from our main target risk from Eq. (3). Although the minimization of two risks is asymptotically equivalent when the model class G includes the ground-truth & they might differ drastically when the risks are regularized or model class G is constrained. In this case, the plug-in learner yields the best projection of Ea only treated/untreated sub-populations (i. when A = a) (Vansteelandt & Morzywolek] 2025); while our main target risk aims to project it well for the whole population. Thus, the main target risk from Eq. (3) is better learning objective learn P(Y [a] xz).
RA- and IPTW-learners. Alternatively, we can employ the identification formulas in Eq. 4 which yield well-known two-stage regression-adjusted (RA) and inverse propensity of treatment weighted (IPTW) learners with the following losses:
Eur(ga. (x) FA
1 1(A
a)
IE log go (Y. V) + 1(A a} /, 1, Z~Ea
log.ga(v. (v. Z n) Ea(o)x)dy
(6)
Evew (go, =ta)= Pn
[[A=a] a} (X)
log ()))))
(7)
z~c;
where Éa and R. are at the first stage nuisance functions fi: the conditional distribution of the outcome and the propensity score, respectively. Both the RA- and IPTW-learners offer better estimates of the target risk than the plug-in in learner. However, they both depend on the nuisance functions, and the error of those propagates with the same order to the final target risk (Vansteclandt & Morzywotek 2025). This motivates our main method, which estimates the target risk so that it is first-order insensitive to the nuisance functions errors (= Neyman-orthoognaaliy).
5

---

Published as a conference paper at ICLR 2026
5 GENERATIVE DOUBLY-ROBUST LEARNERS (GDR-LEARNERS)
GDR-learners. Here, we present novel class Neyman-orthogone learners, namely generative doubly-robust learners (GDR-learners) that are given by the following loss
LODR (9h. f) (E., Ra))
(8)
1 1(A=a) =a a} (X)
I logg. (Y, 2 + (1 1 ))) / Lamas log logga z [1] Ea (xlxx) dy (X)
PM
We derived Eq. by using one-step bias correction of the RA-learner (i. e., by following (Kennedy 2024 Kennedy et al. 2023 Melnychuk et al.) 2023)); see Lemma 2 in Appendix
Our GDR-learners then proceed in two-stages: first, the nuisance functions 1 (Ea: To) H. are estimated (c. g., with generative model of choice); and second, the target generative model is fitted with the loss in Eq. (8.
5.1 THEORETICAL PROPERTIES
Our GDR- EDR-learners, unlike the plug-in learner, estimate the desired target risk from Eq. (3). which is easy to see by putting the ground-truth nuisance functions. Unlike existing RA- and IPaW-learness our GDR-learners have several favorable asymptotical theoretical properties. (1) First, they are first-order insensitive to the errors in the nuisance functions, namely, Neyman-orthogonae This can be formalized with the following theorem.
Theorem 1 (Neyman-ortin orthogonality). The risk given by our GDR-leamerss is Neyman-orthooonal (i. e. first-order insensitive to the muisance function errors), namely:
Dr,2, B.PaCam LaDa (9a, 1)(ga- Ia 9
= for all ga E and á E H,
where D.I are path-wise derivatives (see Appendix Proof. See Appendix C
B.2 for definitions).
Furthermore, as a consequence of the Neyman-orthogonaliyy we can show that: (2) our GDR-learners offer important properties of (a) quasi-oracle efficiency and (b) double robustness (Chernozuukoy
et al. 2018 Nie & Wager 2021 Foster & Syrgkanis, 2023) Morzywolek et al. 2023) (see definitions Appendix B.2p. That is, the following holds (see Appendix C for the full version).
(Quasi-oracle efficiency and double robustness). Under mild model-dependen convexity conditions, the squared distance between the opti- arg ming.eg arg max LEDR (ga,") and the optimizer ga arg min eg / arg max...e Eg LGDR ()) can be upper-bounded by the following:
Theorem 2
mizer 92
1100 Ball & LaDR( 1) - LLDR
I5. - Eall 1|TT Kall
(10)
(I)
(II)
where [|ga|] is a model-specific norm, (I) is standard optimization error term, and (II) is a higher- order nuisance error term. This inequality implies that our GDR-leamerrs are (a) quasi-oracle efficient and (b) doubly-robust.
Proof. See Appendix C
Note on the mildness of assumptions. Theorem relies on the mild model-dependent convexity conditions (see a more detailed formulation of Theorem in Appendix C For those to hold, we require (i) the densities of target models in G and the conditional densities of the outcome &a to be finite. Then, we assume that (ii) the nuisance functions are Hölder smooth (standard estimability
'The idea of the one-step bias correction is the efficient influence function of the target risk correct any plug-in in estimator (RA- IRA-leareer can seen as a plug-in estimator the target risk). The efficient influence function describes how the infinitesimal perturbation of the data generation mechanism changes the target parameter. Thus, by using it in the bias correction, we yield efficient estimator as we use the information from all the musance functions, also those that overlooked in the plug-in estimator (c. g., the propensity score is not used in the RA-leamer).

---

Published as a conference paper at ICLR 2026
g
9a
9u
9u
Sm
g
9
9
Scenario 1.
Scenario 2.
Sconario 3.
Population Ercor(sh *)- ErsoelE, *) U
Populaaton: Bricelg, *) Exvor(Ea *) Sampe:
Population: Eecorish *) Errorlaa *)
Sampo: Ervoti *) Recor(fa, *]
Errorlaa: *) 2 Errcetés. *)
Sampe: Broeffas *) 2 Extor(fa, *)
Figure 2: Difference to the IPTW-learner. Comparison of the IPTW-learners (= variant of our GDR-leameers with the target model class C E 5) and our original GDR-leamerrs (with the target model class Dh E () when V X and G C E. Here, we show three scenarios depending on whether the ground-truth CDPOs, * = P(Y [a] z), belong to both G and 5, only E, or neither of both. In Scenarios 1 and 2, both the IPTW-learner and our GDR-learners are Neyman-orthogonal and, therefore, quasi-oracle efficient. In Scenarios 3, on the other hand, none of the learners are Neyman- orthogonal, * = Ea & S. However, in Scenario our GDR-learners have a smaller empirical error as G C E. Thus, we should prefer the IPTW-learners in Scenario 2 and our GDR-learners in Scenario Note that the real scenario is unknown in practice.
assumption (Kennedy 2023)). Given (i) and (ii). we then show in Remark of Appendix C Cthat the model model-dependent convexity conditions hold asymptotically. Thus, they can be considered mild.
Interpretation. Theorem provides a foundation for the asymptotic optimality of our GDR-learners:
Without assuming any additional structure in the data-generating process (DGP) P(X. Y. A), our GDR-learners are optimal in a min-max sense (Balakrishnan al. 2023 Jin & Syrgkanis, 2025). Therefore, for example, it is not necessary to model parts of the DGP other than the nuisance functions Ea and T.. In practice, both properties (a) and (b) lend to the following interpretation. (a) Quasi- oracle efficiency ensures that, even when slow nuisance functions at least Op (n)4)),)), the minimization of the GDR-learnen loss is asymptotically equivalent to minimization of the target risks with the ground-truti nuisance functions (i. c., from Eq. (4). On top of that, (b) double robustness allows for compensating for slow convergence of one of the nuisance components with a faster convergence of another.
converging slowly (namely,
Are the or (n 1 1/4) rates achievable? An important practical question arises on whether the rates required by Theorem 2 can be achieved by state-of-the-art generative models. As recently discovered by [Schulte et al. (2025), deep neural networks can substantially improve the estimation of the nuisance functions for the ATE. This happens, the neural networks can effectively address low- dimensional manifold hypothesis and, thus, migns,,igiiaaam significantly improve the non- non-parametric convergence rates of Stone (982). In our case of the CDPOs, need to learn the conditional distributions of the outcomes as one of the nuisance functions, which is arguably more complicated than learning the conditional expectations. Specifically, the conditional densities
be seen functions from X' x ) to R- (Efromovich) 2007), and the non-parametric convergence rates of |Stone (1982) are now Op(n ((2)), where is the smoothness of the conditional densities of the outcome. However, again, we again might use low-dimensional manifold hypothesis and deep generative models to improve these rates, analogously to tohhuueete et al. (2025). The convergence rates of the state- of-the-art generative models for estimating conditional distributions are an active area of research Kumar et al. 2025
5.2 PARTIAL AND GENERAL NEYMAN-ORTTIOGONALIT
Remark on IPTW-learner. As noted in (Vansteelandt & Morzywolek 2025 Ma et al. 2024). when V = X, the IPTW-learners for the CDPOs can also become Neyman-orthogonal if the model class G includes the ground-truth &a. Yet, in this case, the IPTW-leamers can be shown to be a special case of our GDR-learners, where we set target model equal to one of the nuisance functions
Ôa = = E 5, where 5 is a nuisance model class (see Remark 2 in Appendix Ch. To see when this variant of our GDR-learners (= IPTW-Iearners) outperform the original GDR-learners, we refer to Fig. 2 Therein, we compare the IPTW-learners (= GDR-learners with a target model class 5) and comparable original GDR-learners (= with a separate nuisance model class 5 for & and a target model class G C 3). Then, when G = 5, both learners are asymptotically equivalent. Also, we
7

---

Published as a conference paper at ICLR 2026
Nuisanoe condilonn gonerative model
Target dional generative
NR
NAGANWMECM
E
Putz)
PO V)
Ep1
FC
(
FC2
Nilsonce
corellann
Ra(2X)
FC
FC1
1. Nuisance generative model
2. Target generative model
Ouipus with comesponcing disributions
Legend
Nolte rnguartzztion
Gencrative model parametrized by
Inputs connections
FC
Pullycoomeeteed subrenecck
Gradient
proceed in two stages. In the first stage, nuisance conditional generative models are trained to estimate the nuisance functions v = (Sa aa) jointly for E {0, 1]. In the second stage, target conditional generative models use the outputs of the nuisance conditional generative models to optimize the LODR wrt. go and 91, see Eq. (
Figure Overview of our GDR-learners. Our
that our original GDR-leamers are guaranteed to have lower error a scenario when both 9 and E contain the ground-truth.
Restrictions of G. When V X, our original GDR-learners should also be preferred when the model class G has to be specifically restricted (e.g.. for interpretability faireess reasons). Then, the IPTW-learners might lose their Neyman-ortin orthogonaaltyy but our GDR-leamnrss always preserve it. That is, our GDR-learners use more expressive E as a nuisance model class for Éa while the IPTW-learners have to restrict themselves only ga G. We later validate this in semi-synthetic experiments.
5.3 INSTANTIATIONS OF GDR-LEARNERS
Architecture. We instantiate GDR-learners with four different state-of-the-art deep conditional
generative models. That is, both the nuisance (n) (Sa:fa)) ft)) and the target models (oo. .1)
implemented with one of four deep generative models: (a) conditional normalizing flows (Rezende)
& Mohamed| 20155 Trippe & Turner 2018) (GDR-CNFs): (b) conditional generative adversarial
networks (Goodfellow et 2014 Laria et al. 2022) GDDR-GGANs). (c) conditional variational
autoencoders (Kingma & Welling 2014. Oh & Peng 2022 (GDR-CVAEs), and (d) conditional
diffusion models (Ho et al. 2020 Lutati & Wolf 2023) (DDR-DMMs). To implement conditioning on X and V for all four generative models in a comparable way, we employed hypernetworks (Ha) et al. (017) or feature-wiss linear modulations (Perez et al. (2018). We refer to Fig. 3 for an overview of general architecture of our GDR-learners instantiations.
Training. Our GDR-learners are trained in two stages. (1) In the first stage, nuisance conditional generative model aims to optimize the plug-in loss LpI together with minimizing a binary cross- entropy loss LECE for both treatments conditional generative model is frozen, and its outputs are used to train a target conditional generative model ÿa wrt. GDRRlearners loss from Eq. (8) The target conditional generative model is also trained jointly for both a = 0,
0 and 1. (2) Then, in the second stage, the nuisance
Implementation details. To evaluate second term in the Eq. (8) (which requires integration wrt. C. X)). employed MC-sampling with 7MC 1. In this way, estimator & needs only provide a direct conditional sampling mechanism and not necessarily an explicit conditional density (this is the case for CGANs, CVAEs, and CDMs). To additionally stabilize the second-stage stage model training, we employed the exponential moving average (EMA) of model weights (Polyak & Juditsky 1992) with a hyperparameter À 0.995. This also helped heuristically ensure that G S E. We refer to Appendix D for other implementation details and the details hyperparameter tuning.
EXPERIMENTS
Setup. We now evaluate our GDR-learners. For this, we used several Jyynhe)yy causal MI. benchmarks with varying n, dy, and d,. In this way, have access the ground-truth
8

---

Published as a conference paper at ICLR 2026
Chis
e- 10 CIBAN
3-m 0- ENARS
0.00
0.65
3.00
1.3
8 0.00
0.0
340
0.01
0.80
031
0.10
Teurse
Pagto
Chis
CIAAN
3-2 LNARS
IPTW
0.10
0.65
0.06
3.00
3.00
603
10000
20000
1000
2000
30.00
1000
1000
10300
2000
1000
3000
3000
Figure Results for the synthetic experiments with varying size of training data (Nugain). Reported: mean out-sample W2 + se over 20 runs (lower is better).
counterfactuals: either as the target CDPOs, P(Y X x)) or, alternatively, as a joint POs sample v(2,,/00,, )=
Table 3: Aggregated results for 77 semi- synthetic ACIC 2016 experiments in (a) full and (b) linear settings. Reported: the % of runs, where our GDR GDDR-leamers improve over other learners wrt. the out-sample log-prob. Detailed results are in Fig. 6 in Appendix E
Evaluation metries. Whenever the ground-truth CD- POs are available, we report the empirical Wasser- stein distance (W2) based on p = 200 samples of the ground-truth (P X = I) and the estimated gen- erative model averaged over {z, {==), On the other hand, when only the joint potential outcomes sample available (e. g., for the ACIC 2016 datasets), we use the average prog-abbbbiltyy of the estimated POs conditional density (log-prob).
(a) full
(b) linear
a= 0
a= CNFs
a=
a
Learner
CNFss
CNFs
CNFs
Plug-in IPTW
45.97%
44.42%
51.43%
54.81%
Baselines. We compare four instantiations of our GDR-learners (namely, GDR-CNFs, GDR-CGANs, GDR-CVAEs, and GDR-CDMs) with the correspond- ing instantiations other learners (i. e., plug-in, RA-,
47.27%
50.65%
61.82%
60.26%
RA
8.05%
10.13%
22.34%
25.45%
Higher
better (improvement over the baseline more than 50% of runs in green)
and IPTW-learnes)). Here, we set V X for all the experiments, and, for fair comparison, we implemented plug-in and leaanerss as single-stage models and RA- and GDR-learner as two-stage models (see Appendix for details). This results in comparing 16 different models (four instantiations times four meta-learners). In this way, we cover all the relevant existing baselines described in Table For example, RA-CGANs correspond to GANITE (Yoon et al. 2018). Plug-in CCNFs to NOFLITE (Vanderschueren et al. 2023), IPTW-CDMs to DinPO (Ma et al. 2024), etc.
Synthetic data. We adopt synthetic data generator from MMelnychuk al., 2023) with dy 2 and d, = 2. In this dataset, the covariates are sampled from the noisy moons data generator and then rotated by a random angle depending on the treatment. Thus, the ground-truth CDPOs are available as the sampling processes. We then sample Rarin E {500; 2000; 4000} training and Reest 1000 E we additionally report results for another popular semi-synthetic benchmark dataset, namely, the IHDP100 dataset (Hill 2011). Results. Results are shown in Fig. 4 Therein, our GDR-learners achieve the best performance when the dataset size grows (this is expected due the asymptotic optimality properties). Furthermore, our GDR-CDMs achieved the best performance overall, when Aerain E {2000; 4000).
test datapoints. In Appendix
ACIC 2016 datasets. The ACIC 2016 dataset collection (Dorie et al.) 2019) consists of 77 semi- synthetic datasets (n = 4802, d, = 82, d = 82) with varying degree of overlap and CDPOs entropy, Here, we do not have access the ground truth CDPOs but only a sample from the POs joint distribution. Thus, the only viable evaluation metric is log- log.prob, and we can only evaluate models that provide explicit conditional density (namely, only CNFs). Setup. As noted in the last remarks of Sec. 5] when V = X and the model classes for the nuisance functions and target models coincide, the IeTa-leareers and our GDR-learners are equivalent. Thus, our GDR-learners are only guaranteed to outperform IPTW-learners when the target model class is restricted (see Sec. 5 To simulate this, we conducted experiments in two settings: (a) full, where the target model is only smoothened with EMA of model weights (as in all the other experiments), and (b) linear, where the target model is artificially restricted to one linear layer in the conditioning hypernetwork. Results. We demonstrate aggregated results in Table 3 and full results in Fig. 6 Appendix E Here, in the (a) full setting,

---

Published as a conference paper at ICLR 2026
Table 4: Results for the HC-MNIST dataset. Reported: median out-sample W2 + std over 20 runs.
0
Learner
CNFs
CGANs
CVAEs
CDMs
CNFs
CGANs
CVAE:
CDMs
Plug-in
0.665 + 0.018
1,511 + 0.243
1,333 = 0.013
0683 11.272
0.653
0.010
1,361 0.179
1,305
0.006
0.601 # 86.159
IPTW
0.702 + 0.013
1,45 + 0.254
1,327 + 0.013
0.678 + 0.045
0.635 0.011
1 1506 0,486
1,295 + 0.010
0.595 0.022
0.169
1.665
1.35
0.688 1: 0.052
0.593
1.562 0:177
0.013
0.57
GDR
0.513 +: 0.205
1,909 4: 0.198
1,306 :: 0.107
0.660 :: 0.227
0.572 0.212
1,715 4: 0.259
1.275 +: 0.155
0.572 1.154
Lower
better (best in bold. second best underlined)
Table 5: Qualitative results for the colored MNIST dataset. Shown: samples from fitted generative models for a digit intervention a
Leamer
CNFs
CGANs
CVAEs
CDMs
O
A
Plug-in
O
C
Ø
0
0
O
0
O
O
a
0 0
0 + 0
0
0 0
0
0
IPTW
C
0
O
0
5 5
0 i 4
0
0
RA
0
0
0
0
0
0
5
0
0 0
GDR
0
0
Ground-truth
our GDR-learners perform similarly to the IPTW-learners as they both are Neyman-orthogonall However, in the (b) linear setling, only our GDR-learners are Neyman-orthogonal (as the target model is restricted), and they outperform both the plug-in and the IPTW-learners in the majority of the runs. Notably, our GDR- rarely outperform RA-learners (even though RA-Icarners are not Neyman- -othoogooh)))) which can be expected. The main reason here is that our GDR-learners learners are only guaranteed to be quasi-oracle efficient wrt. L2-norm and not wrt. the nog-prob, which is used in this benchmark (log-prob tends to overestimate the support of the distribution and may assign large negative values due to outliers). Still, our GDR-learners are highly effective against plug-in and IPTW-learners in the restricted target model setting.
HC-MNIST dataset. To showcase our GDR-learners on the high-dimensional data, we used the HC-MNIST dataset with dy =1, d2 784+1,, 1,
70, 000 (Jesson al. 2021) (= high-dimensional confounders setting). Here, the ground truth CDPOs are also available as the conditional normal distributions. Results. Results are in Table Here, our GDR- consistently outperform other learners for the majority of the generative models and treatment arms. This supports the effectiveness of our GDR-learners even in the nigh-dimensional confounding setting.
Colored MNIST dataset. To further evaluate our GDR-learners on the high-dimensional data, we adapted a colored MNIST dataset (Xia & Bareinboim 2024) (= high-dimensional out- comes setting). Namely, we considered A = {0, 3,2,3,4 4} digits as treatments and X' = {red, orange, yellow, lightgreen, green] colors as covariates. We also down-scaled the images to 10 x 10 pixels. Thus, d, 3; dg = 5. and dy = 10 x 10 x 3 (= width x height x channels). Then, we used Perain 30, 000 and flest = 5, 000 images. Results. The qualitative results are in Table 5 where we show potential outcome samples (a 0) from different generative models and learners. Here, our GeR-learners (in comparison to other learners) specifically allow to better preserve the shapes of digits. This was expected, as the digits are chosen as the treatment variables. We also provide detailed quantitative results for the same experiments Appendix E
Conclusion. In our work, we introduced GDR-learners a novel general class of Neyman-orthogonal meta-learners for estimating the conditional distribution of potential outcomes. Unlike the existing methods, our GDR-learners possess general theoretical properties of quasi-oracle efficiency and double robustness. Furthermore, we introduced several variants of our GDR-leareed on top of state-of-ther thr deep generative models, namely, normalizing flows, generative adversarial networks, variational autoencoders, and diffusion models.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work is primarily theoretical and does not involve sensitive data, human subjects, or system deployment. As such, do not anticipate direct ethical risks.
However, since the research relates to learning algorithms and generalization, possible downstream applications could include decision-making systems in sensitive domains (e.g., education, healtheare, or finance). We emphasize that such uses must consider fairness, interpretability, and safety before deployment. Our work does not include experiments on personal data and does attempt to optimize or deploy models in
sensitive contexts.
REPRODUCIBILITY STATEMENT
We follow ICLR's reproducibility guidelines. In particular:
Theoretical results. All theorems, lemmas, and propositions in this paper are presented with complete proofs (see Appendix []. Proofs are written to be self-contained, without reliance on unpublished materials.
Empirical evaluation. We provide details of datasets, hyperparameters, and training setups in the Appendix D ensuring reproducibility.
Resources. We released all the code, along with a README describing how to reproduce the results.
LLM USAGE STATEMENT
We used ChatGPT during the research and writing process in limited ways. Specifically, it was employed to brainstorm alternative proof strategies, clarify technical arguments during early drafts, and to improve the clarity of explanations in some sections of the paper. All mathematical results, definitions, and proofs were verified independently by the authors. The LLM was not used to generate novel research contributions, datasets, experiments.

---

Published as a conference paper at ICLR 2026
Acknowledgments. This paper is supported by the DAAD program "Konrad Zuse Schools of Excellence Artificial Intelligence". sponsored by the Federal Ministry of Education and Research. S.F. acknowledges funding via Swiss National Science Foundation Grant 186932. This work has been supported by the German Federal Ministry of Education and Research (Grant: 01IS24082). Additionally, the authors would like to thank Dennis Frauen for his helpful remarks and comments on the content of this paper.
REFERENCES
Ahmed Alaa and Mihaela van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In International Conference on Machine Learning, 2018.
Ahmed M. Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task Gaussian processes. In Advances in Neural Information Processing Systems, 2017.
Tara Anand, Adele H. Ribeiro, Jin Tian, and Elias Bareinboim. Causal effect identification in cluster DAGs. In AAAI Conference on Artificial Intelligence, 2023.
Serge Assaad, Shuxi Zeng, Chenyang Tao, Shounak Datta, Nikhil Mehta, Ricardo Henao, Fan Li, and Lawrence Carin. Counterfactual representation learning with balancing weights. In International Conference on Artificial Intelligence and Statistics, 2021.
Sivaraman Balakrishnan, Edward H. Kennedy, and Larry Wasserman. The fundamental limits of structure-agnostic functional estimation. In IMS International Conference on Statistics and Data Science, 2023.
Sivaraman Balakrishnan, Edward H. Kennedy, and Larry Wasserman. Conservative inference for counterfactuals. Journal of Causal Inference, 13(1):2023007, 2025.
Elias Bareinboim and Drago Plecko. On the structural basis of conditional ignorability, 2025. URL https::/causaal hat.u/aalaaiee///22200 pd
net/r120
Peter J. Bickel, Chris A.J. Klaassen, Ya'aco ac'acov Ritov, and Jon A. Wellner. Efficient and adaptive estimation for semiparametrie models. Springer New York, 1993.
Undral Byambadalai, Tomu Hirata, Tatsushi Oka, and Shota Yasui. On efficient estimation of distri- butional treatment effects under covariate-adaptive randomization. In International Conference on Machine Learning, 2025.
Patrick Chao, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan, Interventional and counterfactual inference with diffusion models. arXiv preprint arXiv:2332..0086) 2023.
Victor Chernozhukov, Iván Fernández- Val, and Blaise Melly. Inference on counterfactual distributions. Econometrica, 2166)22205- 2013.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1)CC1-C63 2018.
Yoichi Chikahara, Makoto Yamada, and Hisashi Kashima. Feature selection for discovering distribu- tional treatment effect modifiers. In Uncertainty in Artificial Intelligence. 2022.
Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms. In International Conference Artificial Intelligence and Statistics, 2021.
Alicia Curth, David Svensson, Jim Weatherall, and Mihaela van der Schaar. Really doing great at estimating CATE? A critical look at ML benchmarking practices in treatment effect estimation. In Advances in Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2026
Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it- yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1):43 68, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
Sam Efromovich. Conditional density estimation in a regression setting. The Annals of Statisties, 35 (6):2504-255 2535, 2007.
Jake Fawkes, Robert Hu, Robin Evans, and Dino Sejdinovic. Doubly robust kernel statistics for testing distributional treatment effects. Transactions on Machine Learning Research, 2024.
Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Alicia Curth, Stefan Bauer, Niki Kilbertus, Isaac S. Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958-968 2024.
Sergio Firpo. Efficient semiparametric estimation quantile treatment effects. Econometrica, 75(1):
259-276, 2007.
Dylan J. Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3): 879-908, 2023.
Dennis Frauen, Konstantin Hess, and Stefan Feuerriegel. Model-agnostic meta-learners for es- timating heterogeneous treatment effects over time. In International Conference on Learning Representations, 2025a.
Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Mihaela van der Schaar, and Stefan Feuer- riegel. Treatment effect estimation for optimal decision-making. In Advances in Neural Information Processing Systems, 2025b.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked autoencoder for distribution estimation. International Conference on Machine Learning, 2015.
Ian J. Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
David Ha, Andrew M. Dai, and Quoe V. Le. HyperNetworks. In International Conference on Learning Representations, 2017.
Dacyoung Ham, Ted Westling, and Charles R. Doss. Doubly robust estimation and inference for log-concave counterfactual density. arXiv preprint arXiv:2403. 19917, 2024.
Negar Hassanpour and Russell Greiner. Counterfactual regression with importance sampling weights. In International Joint Conference on Artificial Intelligence, 2019.
Konstantin Hess, Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Bayesian neural controlled differential equations for treatment effect estimation. In International Conference on Learning Representations, 2024.
Jennifer I.. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, In Advances in Neural Information Processing Systems, 2020.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, 2018.
Andrew Jesson, Sören Mindermann, Yarin Gal, and Uri Shalit. Quantifying ignorance in individual- level causal-effed estimates under hidden confounding. In International Conference on Machine Learning, 2021.
13

---

Published as a conference paper at ICLR 2026
Jikai Jin and Vasilis Syrgkanis. Structure-agnoste optimality of doubly robust learning for treatment effect estimation. In Anmual Conference on Learning Theory, 2025.
Fredrik D. Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International Conference on Machine Learning, 2016.
Fredrik D. Johansson, Uri Shalit, Nathan Kallus, and David Sontag. Generalization bounds and representation learning for estimation of potential outcomes and causal effects. Journal of Machine Learning Research, 23:7489-7538, 2022.
Nathan Kallus and Miruna Oprescu. Robust and agnostic learning of conditional distributional treatment effects. In International Conference Artificial Intelligence and Statistics, 2023.
Joseph DY Kang and Joseph L. Schafer. Demystifying double robustness: comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22(4):523- 539, 2007.
Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics,, (11)):008-3044)) 2023.
Edward H. Kennedy. Semiparametric doubly robust targeted double machine learning: review. Handbook of statistical methods for precision medicine, pp. 207-236, 2024.
Edward H. Kennedy, Sivaraman Balakrishnan, and L.A. Wasserman. Semiparametric counterfactual density estimation. Biometrika, 10(4))875-896, 2023.
Kwangho Kim, Jisu Kim, and Edward H. Kennedy. Causal effects based on distributional distances. arXiv preprint arXiv: 02935, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Diederik P. Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends in Machine Learning, 12(4):307-392, 2019.
Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. CausalGAN: Learning causal implicit generative models with adversarial training. In International Conference on Learning Representations, 2018.
Aneesh Komanduri, Xintao Wu, Yongkai Wu, and Feng Chen. From identifiable causal representations to controllable counterfactual generation: A survey on causal generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Shivam Kumar, Yun Yang, and Lizhen Lin. A likelihood based approach to distribution regression using conditional deep generative models. In International Conference on Machine Learning. 2025.
Sören R. Künzel, Jasject S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10):4156 4165, 2019.
Héctor Laria, Yaxing Wang, Joost van de Weijer, and Bogdan Raducanu. Transferring unconditional to conditional GANs with hyper-modulation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Christos Louizos, Uri Shalit, Joris M. Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effecc inference with deep latent-variable models. In Advances in Neural Information Processing Systems, 2017.
Alex Luedtke and Incheoul Chung. One-step estimation of differentiable Hilbert-valued parameters. The Annals of Statistics, 5244):1534 2024.
Shahar Lutati and Lior Wolf. OCD: Learning to overfit with conditional diffusion models. In International Conference Machine Learning, 2023.
14

---

Published as a conference paper at ICLR 2026
Yuchen Ma, Valentyn Melnychuk, Jonas Schweisthal, and Stefan Feuerriegel. DiffPO: A causal diffusion model for learning distributions of potential outcomes. In Advances in Neural Information Processing Systems, 2024.
Diego Martinez-Tabodia and Edward H. Kennedy, Counterfactual density estimation using kernel Stein discrepancies. In International Conference on Learning Representations, 2024.
Diego Taboaaa Aaditya Ramdas, and Edward H. Kennedy. An efficient doubly-robust test for the kernel treatment effect. In Advances in Neural Information Processing Systems, 2023.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Normalizing flows for interventional density estimation. International Conference Machine Learning, 2023.
Pawel Morzywolek, Johan Decruyenaere, and Stijn Vanstcelandt. On a general class of orthogonal learners for the estimation of heterogeneous treatment effects. arXiv preprint arXiv:2303.1266. 2023.
Wenhao Mu, Zhi Cao, Mehmed Uludag, and Alexander Rodríguez. Counterfactual probabilistic diffusion with expert models. arXiv preprint arXiv:2508.13:: 13355, 2025.
Krikamol Muandet, Motonobu Kanagawa, Sorawit Saengkyongam, and Sanparith Marukatat. Coun- terfactual mean embeddings. Journal of Machine Learning Research, 22:1-71, 2021.
Jeffrey Năf, Junhyung Park, and Herbert Susmann. Causal-DRF: Conditional kernel treatment effect estimation using distributional random forest. In International Conference on Artificial Intelligence and Statistics, 2026.
Whitney K. Newey. The asymptotic variance semiparametric estimators. Econometrica: Journal of the Econometric Sociery, pp. 1349-1382, 1994.
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108:299-319, 2021.
Geunseob Oh and Huei Peng. CVAE-H: Conditionalizing variational autoencoders via hypernetworks
and trajectory forecasting for autonomous driving. arXiv preprint arXiv:2201.09877 2022.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2017.
Junhyung Park, Uri Shalit, Bernhard Schölkopf, and Krikamol Muandet. Conditional distribu- tional treatment effect with kernel conditional mean embeddings and U-statistic regression. In International Conference on Machine Learning, 2021.
Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In Advances in Neural Information Processing Systems, 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018.
Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averaging.
SIAM Journal on Control and Optimization,, 30(4):838-855, 1992.
Min Qian and Susan A. Murphy. Performance guarantees for individualized treatment rules. Annals of Statistics, 39(2):1180, 2011.
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In International Conference Machine Learning, 2015.
Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker, High fidelity image counterfactuals with probabilistic causal models. In International Conference on Machine Learning, 2023.
James M. Robins. Robust estimation in sequentially ignorable missing data and causal inference models. In Proceedings of the American Statistical Association, 2000.
15

---

Published as a conference paper at ICLR 2026
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, 2015.
Jonas Rothfuss, Fabio Ferreira, Simon Boehm, Simon Walther, Maxim Ulrich, Tamim Asfour, and Andreas Krause. Noise regularization for conditional density estimation. arXiv preprint arXiv:1907.08982 2019.
Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5);688, 1974.
Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation. In Conference on Causal Learning and Reasoning, 2022.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Conference Learning Representations, 2021.
Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems, 2017.
Rickmer Schulte, David Rügamer, and Thomas Nagler. Adjustment for confounding using pre- pre-trained representations. International Conference Machine Learning, 2025.
Vira Semenova and Victor Chernozhukov. Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal, 24(2):264 289, 2021.
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: General- ization bounds and algorithms. In International Conference on Machine Learning, 2017.
David Spiegelhalter. Risk and uncertainty communication. Annual Review of Statistics and Its Application, 4(1):31-60, 2017.
Charles J. Stone. Optimal global rates of convergence for nonparametric regression. The Annals of Statistics, pp. 1040-1053, 1982.
Esteban G. Tabak and Eric MijdeenEEinden. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1):217-233, 2010.
Brian L. Trippe and Richard E. Turner. Conditional density estimation with Bayesian normalising flows. arXiv preprint arXiv: TEt::/802.0490 2018.
Anastasios A. Tsiatis. Semiparametric theory and missing data. Springer New York, 2006.
Anne Marthe van der Bles, Sander van der Linden, Alexandra I.J.. Freeman, James Mitchell, Ana B. Galvao, Lisa Zaval, and David Spiegelhalter. Communicating uncertainty about facts, numbers and science. Royal Society Open Science, 6(5):181770 2019.
Mark van der Laan. Statistical inference for variable importance. International Journal of Biostatistics, 2(1), 2006.
Mark van der Laan and James M. Robins. Unified methods for censored longitudinal data and causality. Springer New York, 2003.
Aad W. van der Vaart. Asymptotic statistics, volume Cambridge University Press, 2000.
Toon Vanderschueren, Jeroen Berrevoets, and Wouter Verbeke. NOFLITE: Learning to predict individual treatment effect distributions. Transactions on Machine Learning Research, 2023.
Stijn Vansteelandt and Pawel Morzywolek. Orthogonal prediction of counterfactual outcomes. Journal of Causal Inference, 13(1):20240511, 2025.
Dongze Wu, David I. Inouye, and Yao Xie. PO-Flow: Flow-based generative models for sampling potential outcomes and counterfactuals, arXiv preprint arXiv:2505.1605 2025.
16

---

Published as a conference paper at ICLR 2026
Guojun Wu, Ge Song, Xiaoxiang Lv, Shikai Luo, Chengehun Shi, and Hongtu Zhu. DNet: Distribu- tional network for distributional individualized treatment effects. Conference on Knowledge Discovery and Data Mining, 2023.
Shenghao Wu, Wenbin Zhou, Minshuo Chen, and Shixiang Zhu. Counterfactual generative models for time-varying treatments. In Conference on Knowledge Discovery and Data Mining, 2024.
Kevin Xia and Elias Bareinboim. Neural causal abstractions. In AAAI Conference on Artificial Intelligence, 2024.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GANITE: Estimation of individualized treatment effects using generative adversarial nets. In International Conference Learning Representations, 2018.
Weijia Zhang, Lin Liu, and Jiuyong Li. Treatment effect estimation with disentangled latent factors. In AAAI Conference on Artificial Intelligence, 2021.
Tianhui Zhou, William E. Carson IV, and David Carlson. Estimating potential outcome distributions with collaborating causal networks. Transactions Muchine Learning Research, 2022.
17

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Semi-parametric efficiency & Orthogonal statistical learning. the context of the treatment effect estimation, augmented inverse propensity of treatment weighting (A-IPTW) methods were developed to enable efficient, semi-parametric estimation of
target parameters (Newey 1994 Robins 2000). Conceptually, the A-IPTW estimators apply first-order bias correction to plug in estimators (Bickel et al. 1993] Tsiatis (006). However, in our case, we aim at estimating the CDPOs, which are functional, infiniteddimnssioaal targel parameters; and, thus, the standard semi semi-parametric efficiency theory does not apply here. As a remedy, a so-called Neyman-orthogonal
learning was proposed (van der Laan & Robins 2003 van der Laan 2006) Chernozhukov et al. 2018)
Semenova & Chernozhukov [202]] Kennedy 2023 Foster & Syrgkanis [023 Morzywolek et al.
2023] Vansteelandt & Morzywotek 2025), also known as double machine learning (ML). It resolves
the anhiite-dimensionallle of the target parameter by focusing debiasing target risk /target score quantities). In our paper, we specifically tailor a general
functionals (those are umite-dimension
idea of the Neyman-orrthogonlliy to learning the CDPOs.
Orthogonal learning of marginal POs distributions. Multiple works suggested efficient estima- tors/learners for the marginal distributions of the POs. They target either (i) at some distributional aspects of the marginal POs distributions (c. g., point-wise CDFs/quantiles) (Chernozhukov et al.
2013 Firpo 2007: BBaamadalli al. 2025);; (ii) at the distributional distances between the former (Kim et al.] 2018) 8j Martinez-Tabbon Taboada et al. 2023 Fawkes et al. 2024); or (iii) at the whole distribu- tions of the former (Kim et [0118] Kennedy al. [2023] Melnychuk et al. 2023 Martinez-Taboadaa
& Kennedy [2024] Luedtke & Chung] [024). Out of those, the works of Kennedy et al. (2023) and
most similar to our work, as they also aim to find the best approximation of the POs distributions in the class of some parametric probabilistic model. Our work,
Meinychuk et al. (2023) are probably
thus, may be interpreted as a non-trioo trivial extension of (Kennedy et al. 2023 Melnychuk et al. 2023) to the covariate-condition
POs distributions.
Non-pa con-parametric estimators of CDPOs. Several non-parametric estimators were proposed to estimate the CDPOs. For example. Muandet et al. (02T) used distributional kernel mean embeddings, and Alaa & van der Schaar (2017: 2018) employed Bayesian non non-parametrics (Gaussian processes). However, both approaches are in fact plug-in in learners and scale badly to high dimensions (due to inability to learn low-dimensional manifolds of high-dimensional data). Therefore, we excluded those methods from the empirical comparison.
Time-varying probabilistic models for POs. Some works suggested generative models for time- varying potential outcomes distributions. They relied on different probabilistic models: Gaussian processes (Schulam & Sarial 2017), Bayesian neural networks (Hess et al. 2024), and diffusion models/normalizing flows Wu al. 2024; Mu et al. (025). However, they only restricted on plug-in losses (Schulam & Sarial [017]] Hess et al. |024|1aa et al. 2025) and IPTW losses (Wu et al. (024)).
Our paper operates only in the cross-sectional setting, and, thus, we also establish an important foundation for future work on the Neyman-orthogon generative learners in the time-varying potential outcomes setting (e. g., analogously (Frauen al., 2025a)).
18

---

Published as a conference paper at ICLR 2026
B BACKGROUND MATERIALS
B.1 GENERATIVE MODELS
In the following, we formalize four popular deep generative models that are later used as instantiations
to GDR-learners: (a) conditional normalizing flows (Rezende & Mohamed 2015
Trippe &
Turner 2018); (b) conditional generative adversarial networks (Goodfellow et al. 2014
Laria et al.
2022), (C) conditional variational autoencoders (Kingma & Welling 2014 4 Oh & Peng
2022). and
(d) conditional diffusion models (Ho et al. 2020 Lutati & Wolf 2023).
B.1.1 (A) CONDITIONAL NORMALIZING FLOWS (CNFS)
Probabilistic model. Conditional normalizing flows (CNFs) (Tabak & Vanden- Majden-Eijndenj 2010 Rezende Mohamed 2013 Trippe & Turner 2018) are flexible probabilistic models with an explicit tractable density. They employ invertible and differentiable transformations fa(= v) Z + of some latent variable E Rd with a known density (z). Then, the conditional density of the
outcome is given by change of variables formula:
dff de
(J. (1.'
(v v) (= = fz
det
(11)
Then, fit the CNFs by parameterizing fe and directly maximizing the log- lillikelihood of the data. Notably, the marginalization wrt. & is not needed for the CNFs.
Learning distributional distance minimization. CNFs aim maximize the log-likelihood wrt. Pa, namely
max ((ga) P.E9
max log Pa [a[a) [a] V)], Ps P9
(12)
where L is defined in Eq. 4 and ga v) = P. (y v). As noted in (Kennedy et al. 2023 Melnychuk al. 2023), the maximization of the likelikeld is equivalent the Kulbbck-Leibler (KL) divergence projection of the ground-truth on the chosen model class G. In our context, this is equivalent the following:
Pa arg max C(pa) < P = arg min By KTD((P(V(A) V) (V), P.
(13)
P.Eg
where (-D(-II-) is the KL divergence.
B.1.2 (B) CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS (CGANs)
Probabilisttc model. Conditional generative adversarial networks (CGANs) Goodffllow et 2014 Laria et al. 2022) also apply a differentiable transformation fa v) Z - y to the predefined latent variable E dRd. with a known density (=). Here, Ja called conditional generator, and it induces an implicit conditional distribution of the POs, pa (y v). Also, CGANs define an auxiliary model, a conditional discriminator (y v) : y - [0, 1], that is used to train the whole model.
Learning distributional distance minimization. CGANs are trained in the adversarial manner: While the conditional discriminator tries to distinguish the real samples from CDPOs, conditional generator tries to trick the conditional discriminator by generating realistic samples. Thus, the final objective wrt. fa and da as follows:
min max C(ga) f.Eg d.Eg
= min max faeg d.
E log du (Y V) Z~ee
-. (Ja (d
(14)
where defined in Eq. (4) Eg hrg (2). and go (o z
v) da v) (1 da(f(( u) v)).
Furthermore, one can show (Goodfellow et al. 2014) that the conditional generator implicitly minimizes the Jensen-Shannon (JS) divergence:
f = arg min max L(ga) 0 ph = arg min Ey JSD(P(Y[s)) V) PY V)),
(15)
f.eg d.Eg
P.EG
where (.D(- is the IS divergence.
19

---

Published as a conference paper at ICLR 2026
B.1.3 (D) CONDITIONAL VARIATIONAL AUTOENCODERS (CVAES)
Probabilistic model. Conditional variational autoencoders (CVAEs) (Kingma & Welling 2014
Oh & Peng 2022) also use latent variable Z E Rd with the density ha (z), However, instead of defining transformation between and Y Y[a], they introduce conditional distribution Py 2, v) (a conditional decoder). To sample from ( u). one can first sample from Z ~ (Z) and then from [[a] ~ Pa Z, v). Furthermore, to fit the conditional decoder, we employ an auxiliary model (= y, u) (a conditional encoder) that serves variational approximation of ground-truth intractable posterior (= y, = Pa (y 2, v) (=)) [a] = y v)-
Learning as distributional distance minimization. To fit CVAEs, we aim to maximize an evidence lower bound (ELBO) wrt. Pa and qa:
Pa Pa 2 V)
max L(ga) = max E Pu
E log ZIcza
(16)
ParRa EG
ga ( Z
where defined Eq. Pa (a(((a) [a], Z V) PY Z, V) (Z), & ga Z Y [a] V).
Thus, Ia (v. = v) equals Pa (y. = ga/ga(z y, v). Furthermore, as demonstrated by Kingma et al.
(2019). by maximizing the ELBO, we minimize the following distributional distance:
P arg max max C(ga)
- P arg min Ry
(T.D(P(V(a) V) (l
(17)
EGEG 9gE9
P.EG
+ E [4 KLD (2 lI V)).
inference gap (IG)
where Pa(y u) = Se Pa 2,,)aa de, KLD(-II-) is the KL divergence, and the inference gap (IG) shrinks as we make a variational family of qa more expressive.
B.1.4 (D) CONDITIONAL DIFFUSION MODELS (CDMs)
Probabilistic model. Conditional diffusion models (CDMs) (Ho et al. 2020 Lutati & Wolf 2023) extend the idea of the CVAEs and construct the whole sequence of the latent variables Zi:r where T> > 1 is a number of diffusion steps and each Ze E Rd,, The CDMs gradually add noise to Y|a) by defining fixed diffusion process (2 noise distribution ha (2T). Then, the probabilistic model Pay 21.7, 0) is defined as a conditional denoising process that restores the value of Y(a). We refer to Ma et al. (024) for further details of the model definition.
that the final Zr has the predefined latent
Learning as distributional distance minimization. Similarly CVAEs, CDMs aim maximize the ELBO, yet only wrt. to Pa:
(Y ZTT [V) Qa (2
max C(g.) = max E P.G
log
(18)
P.EG
Z~EE
Y ((a)
where defined in Eq. Pa (Y [a], Z.. V) = Mo(2r I
1= Pa (Z-1
Ze, V) with Zo=
[[a], c2 = 2a (Zi.T (aa)). Hence, 9a (y, = v) = (v,,, 21:T 10/ga(2177
y). The denoising
process of the CDMs then has similar interpretation as CVAEs: it minimizes the KL divergence plus the inference gap (see Eq. (I7).
20

---

Published as a conference paper at ICLR 2026
B.2 ORTHOGONAL STATISTICAL LEARNING
In the following, we provide some key definitions of orthogonal statistical learning (Chernozuukoy
al. 2018 Foster & Syrgkanis 2023 Morzywolek et al. 2023). For that, we use some additional
notation: Ill1r,, denotes the bp norm with Il - E(I (2) /p S b implies that there exists
C2 0 such that I C. b, and X, = op(rn) means X,. 4 0.
Definition 1 Seeyman-orthogonally (Foster & Syrgkanis 2023; Morzywolek et al. 2023 A risk L called -ermman-orthooonll if its pathwise cross-derivatives equal zero, namely,
g". 7n 0 for all g 9 and nj H. Dg,gg .7)g 1019-99'110
(19)
where D)F()) = a F(S + th)]=o and F() hg]
0
F(f + 411 + +...
at
tyhh))e )e, = are pathwise derivatives Foster & Syrgkanis 2023 g and 17 is the ground-trute trrtth muisance function.
arg min L(g. n):
Definition 2 (Quasi-oracle efficiency). An estimator 9 = arg ming Lig, i) of g
arg min geg L(g.") 1)) is said to be quasi-oracle efficient if the estimator in of the nuisance function n is allowed to have slow rates of convergence, og(n og(n-1/4), and the following still holds asymptotically:
1 -0'''1 ((G,9) L(g", () + op(nn112)
(20)
where L(g, fi)
E(g", fi) an optimization error term: the difference between the risks of the estimated target model and the optimal target model where the estimated nuisance functions used.
Definition 3 (Star hull). A star hull (Foster & Syrgkanis 2023) for elements a and x' of vector
space X is defined as star (X, x, x') = ftx + (1 f)z E 00 1]}.
21

---

Published as a conference paper at ICLR 2026
C THEORETICAL RESULTS
Lemma (Identification of the target risks). Under the identifability assumptions (i)-(iii), the target risks (g9)) in Eq. (3) can be identified as follows:
((ga)= = S E log (y. Z [V] Ea(v/X) dy = [1AA= a} LZmex
log 9a (Y, VV) V)
Ta (X)
Zine=
(21)
Proof. The first equality holds due the law of total expectation, and the application of (i) consistency and (iii) unconfoundedness:
((ga)
E
E ZINE
log 9n (Y z V) Ex E ZINE=
log 9a (Y[a), z 1 (V) x]
(22)
Ex
E
log D. (Y (Y(a) [a] Z V) x, x,A= a]
(23)
ZINz=
Ex E Zine,
lo (Y, Z V) X, A=
(24)
[
E I. z vi] Ea Ea(y x) dy) LZne,
= Ex
(25)
The second equality is then easy to obtain from the first one, by using the laws of conditional probability:
|
((ga) = Exx
log ga (v, 2 | v3] E. (a(l()) dy
(26)
Lines
=Ex
1/1
E
log 9a (v, 2 v)]
P(Y - y, A = lX)
dy
(27)
Limez
Tx(X)
[(A
= a}
E log 9a (Y, Z m/]
E
(28)
T. (X)
Zinez
where the last equality holds due to the laws of total probability.
Lemma (One-step bias correction of the RA-learner). To construct a Neyman-orthogonal learner, perform ste-step bias correction of the RA-learner, namely:
LODR(Ba> i) = LRa 7) + PP,, (TC(
(29)
where [E: (X, n) an efficient functionn (EIF) and is given by the following:
Kennedy 2024 of the target risk
[F(:: (X, A,Y): = 1(A= aa log (Y, V)
(30)
(x(X) ZNN.
(1- 1(A = a}
1{A=a)
(X) /
logga(v) (v. Z 1y Ea(y) X)dy L.
E Lanss
Proof. To derive the EIF for our target risk, we use multiple building blocks from Kennedy (2024). namely, the chain rule and the EIF of the conditional densities:
USLLE
[LE
log Z v)] Ea (x(v dg) Lame.
[F (X,AA))))) IF Ex
E
(31)
IF (1B
log ga (v, Z v)] (a(w))) dw / /.
-Ex
E Lames
log go (v, Z [1
(a( dy L.
Liwes
(32)
22

---

Published as a conference paper at ICLR 2026
Then, unroll the inner part of the first term further:
(1
nF
E Lnme
log 9u (v, Z v da(vlx)
(33)
= =f8 IIF
[
E U~NE=
log Da (v, Z ml]) Ea (x) dy /
E log log ZIV) v)] IF (F(Ex(U)X)) dy
(34)
=0 + / L log 9a (y. Z v)]
- P(X =(X=x,A= A= (o1Y -
-Ea -
Xx) dy,
(35)
Lines
where appears since go is not an attribute of the ground-truth DGP, and o{-) is Dirac delta function. Then, by applying the outer expectation wrt. to the X, we can "smooth out" the Dirac delta functions:
o{x x =}{{A= a} (ofY 4) - Ev X)) du P(X x, A
Ex
E log (v, 2 [V]
(36)
LINEz
1 [{A
log 9a (Y, Z V) /, Lune.
log 9u Z 11 x))
E
(37)
No (X)
ZiNEe
Finally, by regrouping the terms, we obtain the final formula:
={A=a} E
[F(C;
log aa 2Y,
(38)
(X(X)
ZINE.
+ (1 - /
a} Ta (X)
[LE Lanez
E
logga Z (V) Ea(y x) L.
Theorem Neyman--rthhoonnaliyy The risk given by our GDR-learners is Neyman-orthogond (i. e., first-order insensitive to the nuisance function errors), namely:
Dr,Dy (ga, 7)(9. Da 9 - n) = 0 for all 9a EG,
(39)
where D.I(-) are path-wise derivatives (see Appendix B.2 for definitions).
[(A=a} wa X)). Also, define DR pseudo- Proof. Let us denote the IPTW weights xx(X)
distribution (2, A) (A, X) X) (Y X)) (Y X). Then, given the DR
dseudo-distribution, our GDR-learners risk simplifies to
LoDR(g". n) = E / (X, A,y) _E logg% Z V) dy
(40)
ZINEs
We verify Neyman-orthogonallty by deriving the pathwise derivatives of LODR- First, we derive derivative wrt. 9a:
d LanR(sa + -(gg ga). 1==
D, DyCCDR (l). --al= dt
(41)
As demonstrate in the following, it depends generative target model. (a) For CNFs, the first-order pathwise derivative is
De D&Lana(g' 1)lga s.ll [
EN (X, log (p2l/V) (lpalllpllli (V) p4ly V2) dy
(42)
1-0
E =E( =IS, (x) Pa(v V) pl (V) dy (5 V)
(43)
Then, we take a cross-derivative wrt. the nuisance functions:
D=. Dy Lape(( 1)190 - 9e. Aa =a]
(44)
1 (x)) + ((F (x))) (Ea(v) 1X) En(vl x)) Ea(21) (x)
a{A=a}
Pa(v V) [V- p
dt
p5lv V)
1-0
S (x(X)) (X)
Pa(v I V) pā(vl V) dy =
-B
(5a(3) | X) Ea(v x))
(45)
=x(X)
Pā(v V)
23

---

Published as a conference paper at ICLR 2026
Dea Dg DoCoom(ds:1) n)'ga - Eo - -ca]= = d F.
I ( (1(d=a) (soly x) - Co(v ( - ((Ea(v) x) - Cv(( xi))
di
xx(
(46)
+ Evl (X) + (f (x) sa(v1x))) pa(v V) - -v v dy
plly V)
1--0
=B S
Ta(X) (Eoly) (x) sa(o) I X)) 1 (E.(v) | x) E.(M s.(w1 x)) Pa(y V) P.(y V = 0. (47)
(X)
pi
V)
(b) For CGANs, we first note that the optimal discriminator du has the following form:
(3, v)
dy u)) (x, a. v) + pz(y
(48)
where [v u) is an implicit conditional density of the conditional generator (see Appendix B. for details). Therefore, the first-order pathwise derivative has the following form:
Dy PoCoouisa Wlaa - sāl== = a. E [1 (xX A.v) log A2, + P2(M V) (xpa V) pt vi) dy
(x,
(49)
+ $/ (1560 V) v(pa(p v) - Pp V))) log A, v) + pb(yy) V) (pa - p5y V))
ro V) + ( v) - pb v)
dy
r=
paly V) - (3 v) -,xe (X, A) + Pb( v)
dy + / (poly - - v3) log A.) pá (v
13 V)
dy
V)
(50)
+ 1 palu 1V) Paly V) - p v)
Pa V) - V)
(X, A, + 12( dy
1y V)
y) + Pz(M v) du 5 1 (paCulV) V) pllu log A),
Pa V)
(51)
Then, we take a cross-derivative wrt. the nuisance functions:
Dx, Dr CaDoRss
1)lsa
9 tu
ma]
(52)
d
E J log
1{A o} (x)) + (X) - (X)) (Saly) x) Co(vv I xx) + Ea((v x) + v2) (pa v) v) VJ) dy
dt
1=0
(X) (5alv) I X) - Ea(y) I x))
-EB
(E (X (x (x) + Ev(M I X) + p (po(y) pM v) dy =0.
1(A=o) [x(X)
(53)
Deo Dy PyConelsh Ea {a]= =
(54)
dg Eg
/ log
[[AA=a] xx(X)
(ea(s) | x) to(y x) - r(Ealv) x) Ex (x)))
= dt
+ Ea(y) | (((((v)) x) - ca(v| x)) + pv v) (pr V) - pr / v)) dy
1=0
(E. x) Ex 1 x)) + (v x) Eo x))
B
(pa(v VV) (v) -0 (V) = =0. (55)
((Ama) [x)
(c.(v) | XX) xx) xx) + E. (v +
(c)&(d) Now, we derive the first-order pathwise derivative for CVAES and CDMs together, as they are both based on the maximization of the ELBO. In the case of the CVAEs, we need to take a derivative wrt Pa and ga simultaneously:
24

---

Published as a conference paper at ICLR 2026
Dy CCDR( nalaa 9āl
(56)
- [/ (,(x, log Pa(y.= V) + (paay. =1 V) pa V))
(7 v. V) ((ga(2 y. V. C2 (z v. VJ)
(1al2 (Mdl=l..V v. 1(aa(= (0a(=110) v. V. (.(= v. v)) dz
1--0
= =E (2 A. / (pa(u: v) p5 (v. 1))
22 v. V)
P. Pul (ga(2 (v.) (=(= V)) log (1,
de dy
V) (57)
Then, take the cross-derivative wrt. the nuisance functions is also zero (analogously to (=lu.
(a) CNFs, namely with S2 (pa(y, = V) - pā,,; V)) P.aaaV (qa(= v, V) do (=
v)) log Pala.aVV) dz instead of (e(vlV)-E5(01)) The formulas for the CDMs are analogous, i. (=llv.V)
=2(1lV)
with qa (2 [2] y. V) qa(= V).
Theorem 2 (Quasi-oracle efficiency and double robustness). Let us denote the IPTW weights as (A=a)
xx(X) =W(A,, X). Also, we define DR pseudo-distribation (X, A, Y) (A, (YaE(Y
X) - Ea x (Y X). Assume that for some 0 the following convexity conditions hold for different target generative models (a)-(d):
(Y (V) - P.(Y (V)?
AXX)
2 a callp'
Palli
for (a) CNFs.
(58)
Pa(Y (V
EY [x)
(Y V) PEY V))
(X))
E
2 2al1p - Pa 122,2
for (b) CGANs,
(59)
(Y (V
Ea (X)
(b, Z V)- -, VI>
(,
(QalZ Y,V) 2. V,V
(X)))
E
(60)
Zoes
Pa(Y, V)²
Ea(Y (X]
qn Y,, V)2
E |X)
2 2a( palli., + l1a2 dh ll,)))
for CVAEVCDMs.
Then, the squared distance between Th = arg migming.es ga eg / axg max eg LODR (9a: 1) and ga = arg ming.eg/ / arg m.xx9.59 CGDR(ga.>) 1)) can be
by the following:
1l Da 1/3 S LADR (āa 1) - LADR (...1) + La - Ea
ITTT t La
(61)
(
(1)
where llgalle = V EYpa(Y (VP) for (a)&(b) CNFs and CGANs, I1aall
/ /E(EZ~e, (Pa(Y, Z VУ + qu (Z Y, VYJ) where E. (Z Y,V) star (G, gas Ia) for (a)&(b) CVAEs and CDMs, (I) is an optimization error term, and (II) is a higher-order nuisance
error term. This inequality implies that our GDR-learners are (a) quasi-oracle efficient and (b) doubly-robust.
Proof. Given the DR
our GDR-learners risk simplifies to LODa Caa +) [/, (x, A. E. log (v, V) dy
(62)
To prove the quasi-oracle efficiency, we apply a functional Taylor expansion to the following target risk:
LaDa (g.. i) = Lona (9h. ii) + D, Cena (gi. 1). - 9.l + / Cano Cene (ga. 1) g.. åo - -g]s
(63)
where Ha E star(g, g..g.. G.). Hence, we need to derive the second-order pathwise derivative of LGDR wrt. Ja- As demonstrate in the following, it depends generative target model.
25

---

Published as a conference paper at ICLR 2026
(a) For CNFs, the second-order pathwise derivative is
(pa (Y (V) PTLY Vi)
LoDo M)
Is: Go
dol =
(64)
Ea (X)
X)
PY
V)a
(b) For CGANs, first note that the optimal discriminator du has the following form:
(2 a. y)
da 0) (2, a, Pa(y
(65)
where (y v) is an implicit conditional density of the conditional generator (see Appendix B.1 for details). Then, the GDR-learnen risk for CGANs is equivalent
(X, A. g) CODR 1)) = =B| [ (,y) (X, A, + Pa V)
dy + Ba V) (x, A. + Pa V)
Paly V)
dy
(66)
Finally, we can derive the second-order pathwise derivative:
CiDe (d: 01aa - ga. 9a - 9.. = (fu (Y VV) -V Ea (X)
CX, A,Y PolY () (V) (X
(67)
(c)&(d) Now, we derive the second-ordes pathwise derivative for CVAEs and CDMs together, as they both based on the maximization of the ELBO. In the case of the CVAEs, we need to take derivative wrt. Pa and Ga simultaneously:
CenR lāa
9a Au sal
(68)
(ba Z V) P.Y. Z V)D
(X,,Y (d.(Z (2(Z (Y,))) PY VV2
(X, Y.
E
2~Ix
(Y X)
(
Ta (Z V1
where & Ta (Z Y, V). The formula for the CDMs is analogous, but with qu (Z Y,, V, (2 V).
Now, under convexity conditions, the following holds for all the models:
(ā M) gà. âa gāl llg. 9a l[g
(69)
where a >
Therefore, from Eq. (63) we get the following:
|1 Bulls S Louelda, ?) Leaw (ga n1) D Lean 11.0. - 9al
(70)
Rg CoUk (gn, 1)(ga g%).
(71)
(a) For CNFs, the last term, D.LCODR(E 1)laa gal can be expanded further as follows:
Ed (X, A, picle V) fa V)
LyDoa n)lg. shll 12
dy
(72)
( V)
= = S (X, A,y colv x)) (u V) - Pa V) dy
+E Calv) xy P% V)
PM(M V)
dy
pa(y V)
pz(v
V)
R2(n.6)
(73)
(b) For CGANs the last term, - DD LGDR 1)(9a - ghl is:
D9 DyLan (o%..1).. 1). s.ll
(74)
S
Pl V) log el(X. A. + PAlV EN (X, A, v)
P (v1 V)
= E
log
(pilv (V) Pa
palv VV)
Ra(m:9)
pcly (X( A) + rā(y
+B
log
(pi V) Paly V)) dy
26

---

Published as a conference paper at ICLR 2026
(c) For CVAEs and CDMs, analogously (a) CNFs, the last term, is:
CDyC (9a n)19. gāl
DDCCan (9a v)( sāl
(75)
(E (X,, A. Ealv I x)) / (Palv.= p. V))
(a v. V)
p.. 2 V)
(fa(2 1 y. V) 2. vs v) log
dy
Pa (a v)
Th y. V)
(a(v))))
Eoly x) (Dal0.=1V) ( V)
Pá Pa IV) (fa(=
Ca(= y. V)
v. V)) log
Pa (V) dy
25 y. v.
(+)
Here, in all cases (a)-(d), the first term is called second-order remainder (2 fi) and the sec- ond term (*) equals -Dy CGDR (g% n))laa - g.l (it always stays negative due the virtue of minimization).
(a) For CNFs, the second-order remainder (2(1) f) is then:
Ra(n, 6) = [, (A))()))) x) Ev(y x)) + En(y I x) - Ev(11 (c(vl))) pilu V) Paly V
dy
pC V)
(76)
= =x(5, (10a(A,))- ()(Ea(v) x) Ev (x))) plly v) Paly IV) dy
(77)
(a V)
Hence, by denoting B(ʃsy My. dy) = Cr for any function h(y, x) and by applying the Cauchy- Schwarz inequality, we obtain the following:
Cp:
||m tu 1la [l.. Eal1e, Ilp2 palla:
R2(y, iil
(78)
where is a margin of the strong overlap assumption.
(b) For CGANs, the second-order remainder R2(n. f) is:
R2(n, A) = -B. [ log (E)( A. (E) rilu v)) log (Sa(v) x) + pils v\) (nttu V) - Poly V)) dy
(79)
E (X, A,y) Ealy X)) (pā(v Pa(u V)) dy
(80)
A,
= (X)
I) (x(
(Mi v) - Pa V)) dy
(81)
where holds by mean value theorem for some ij E star (H, û 11) This can be upper-bounded as:
C2,
|m. - Rulle, k.. Eall., Ilp. pallg
Ra(n i)
(82)
where is a margin of the strong overlap assumption.
(c)&(d) Analogously to CNFs, for CVAES and CGANs, the second-order remainder R2(n, n is:
R2(n, n) = E [1, (I 1)(5a(v X) (a(y1
(83)
f (po(v.> ph(u. V)
(x 1..1.. V) ( V)
PC (y, VV 0= v. V) de dy
(ğa(æ v. V) - q v. V)) log
Let us now denote E(fyxz N....x upper-bounded by
dz dy) as Cn for any function h(y, Then, [R2 1) can
R2(1, ri) < Cpc Cse
|| ta He [lc. - EEll (Coz Ip Pally + C zá 1142 4. anlls), (84)
qu log
75
where E > is a margin of the strong overlap assumption, and l| =
/ E((EZz,, (ga(Y, 7 V)]).
27

---

Published as a conference paper at ICLR 2026
Therefore, Eq. (70) can be upper-bounde bounded for all models (a)-(d):
l9g. 3.1l3 = n,
1)) 1) gal + C. I|
Ra Il,, [..
E0
1192
Ia lellg. (85)
where C. is model dependent constant, see Eq. 78). and
84).
Finally, by using an AM-GM inequality and noting that the term (*) - Dy CODR (ga n) gal always stays negative (similarly to (Morzywolek et al., 2023)), we recover the final inequality in Eq. (61
Remark 1 (Mildness the convexity assumption). The convexity assumptions in Eg. 58 (59) are relatively mild. Specifically, first, the densities Par Js and &. have all to be funite that their inverse is always greater than or equal to some ô > 0). Then, we need to ensure that:
59 and
(X, A,Y) ())(A, (Y |X) - EY X)) +128>0 Eo(Y X)
(86)
The latter holds asymptotically, assuming that (i) both l and f are Hölder smooth and (ii) Eu Ea
Eall I| Ralli. O. Here, (i) regular assumption for the estimability of g and n, and (ii) is required for quasi-oracle efficiency to hold.
Remark (IPTW-learner). When V X and the target model class G is set be the same as the model cluss for & E E, our GDR- learnerss simplify to the IPTW-learner. This can be seen from the definition of the GDR-learners risk (Eq. (8):
1 { a}
LavR(ga EGa =
aa)=
E log ( 2-1e
(87)
(X(
(1 1(A=a)
/
L E logEa Z xi] Ed(vlx) 2INE=
(x(
Here, the term (x) contains the gradient blocking of Ealy | 2) (otherwise, the risk stops being sayman-orthogonan Also, because of that, the optimization of the term (+) does not change the Êa- as the optimum of (+) is achieved when & (y x) matches with &a (y, x)) (see the interpretation of the target risks as the minimization of the distributional distances in Appendix
Therefore, the optimization of the risk in Eq. (87 is equivalent to the following:
Cirrwlga E no) ((A=) a]
E EogEa Z 11x)
(88)
(X)
which recovers the original IPTW-learner with =. Furthermore, if the ground-truth CDPOs &a belong to E. this IPTW-learner becomes Neyman-orthogonde and quasi-oracle efficient. D Lirrw(Ea, n), Eal = O.
To see that, one can easily verify that
28

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATION DETAILS AND HYPERPARAMETERS
Architecture. The architecture of neural instantiations for our GDR-learners is given in Fig. 3 Depending on the structure of the potential outcome (()) tabular or (ii) image), we implemented conditioning for both the nuisance and target conditional generative models differently. Specifically, (i) for all but the colored MNIST experiments, we used hypernetworks (Ha et al., 2017) with two fully-connected sub-networks FC1 and FC2. Both sub-networks had L hidden layers with dj hidden units and ELU activation function. The second subnetwork FC2 outputs the parameters of a generative model 0(X A) (nuisance generative model) or ((V, a) (target generative model). On the other hand, (ii) for the colored MNIST experiments, we used the first fully-connected sub-network FC₁ from (i) together with feature-wist linear modulations (Perez et al.) 2018). We then implement the generative models as follows:
(a) CNFs. (i) For all but the colored MNIST experiments, we used neural spline flows (Durkan et al. 2019) with additional auto-regressive transformation (Huang et al. 2018) when dy I. On the other hand, (ii) for the colored MNIST experiments, we employed conditional masked autoregressive flows (MAFs) (Germain et al. 2015 Papamakarios et al. 2017). As base distribution, we use a normal distribution Z (0; l2,) We further tuned the main flexibility hyperparameter: (i) the number of knots Tiknots for the neural spline flows, and (ii) number of autoregressive layers ny for the MAFs, respectively, Additionally, to regularize the log-liiklhhodd, we add noise regularization (Rothfuss et al. 2019) to the outcomes Ey ~ [0; ag)
(b) CGANs. (i) For all but the colored MNIST experiments, the generators and discriminators of our CGANs were both fully-connected networks with one hidden layer with dga hidden units and an ELU activation. (ii) For the colored MNIST experiments, we used two (de)convolutional layers with dyu output channels preceded or followed by fully-connected layer (for generators and discriminators, respectively). all with the ELU activations. For the generator, sampled Z N NO; Ia where d_ dy in (i), and d₂ is tunable in (ii).
(c) CVAEs. Similarly to CGANs, the encoder and decoder of the CVAEs were the networks with either (i) fully-connected or (ii) convolutional layers. For our CVAEs, we set a prior as Z N (0;;
(d) CDMs. In our experiments, we closely followed an implementation of DifiP (Ma et al. 2024b. Thus, the diffusion process uses conditional normal distributions and ZT ~ N(O la, (here is the number of diffusion steps). As the e-net of the CDMs (s-net aims to reconstruct the noise added during the diffusion process), we used (i) a fully- cully-connected network with one hidden layer de hidden units and the ELU activations in all but the colored MNIST experiments. At the same time, (ii) for the colored MNIST experiments, we used a U-Net (Ronneberger al. 2015) as the e-net with four convolutional layers (each with de output channels) and the ELU activations. The middle layer of the U-Net had tunable number of units Thg. The dimensionality of time embedding was set d, 20.
Implementation. We implemented all the baselines with PyTorch and Pyro. To regularize the conditional generative models, we employed noise regularization (Rothfuss et al. 2019) after the first fully- fully-connected sub-network FC1 (Ex N (0; of)). Also, for IPTW- and GDR- too low propensity scores (lower than (i) 0. or (ii) 0.05), which helped to stabilize the training. For target conditional generative models, Juditsky| 1992) with a hyperparameter X = 0.995. We use the same training data 'D for two stages of learning: This does not harm the theoretical properties of our GDR-learners as the (regularized) NNs belong to the Donsker class of estimators (van der Vaart 2000 Kennedy 2024).
clipped
additionally use EMA of the model weights (Polyak &
Hyperparameters. For all the baselines, we performed an extensive hyperparameter tuning. Therein, we tuned the flcxibility and regularization hyperparameters of different conditional generative models. Further details on the hyperparameter tuning are Tables ( [ Also, ensure reproducibility. we report the tuned hyperparameters in our GitHut as YAML files.
"Code available at httppp/////gttttb.coo///aaa/
29

---

Published as a conference paper at ICLR 2026
Table Hyperparameter tuning for different meta-learners and generative models for (i) synthetic, IHDP, ACIC 2016, and HC-MNIST experiments.
Model
Suh model
Hyperparameter
Range Value
Number knots (Naman))
5. 10, 20
Intensity of noise regularization (o2 Inteusity of noise regularization (o
0.0, B.013, 0,052, 0.12
0.0, 0.01", 0.05". 0.1²
nuisance CNFs Plug-in CNFF ( IPTW-CNEs)
Learning rate (124)
0.001, 0.005
Minibatch size (lx)
32, 64
Tuning strategy
random grid search with nuns Lxtr
Tuning uriterion
Number epochs ((l,))
RA-CNFS, GDR-CNPs
Optimizer
SGD (momentum 0.9)
Number knots (Tieweec) Intensity of noise regularization (o Intensity of noise regularization (o,,)
Oloon 02
0.01
Larget CINEs
Learning rale (177)
Minibatch size (bx)
64
Tuning strategy
wlo tuning
Number of epochs (ne,7)
E
Optimizer
Adam
Generalarliisrrininnn hiddee dimnension (dgux)
5. 10, 15, 20, 25, 30, 50 0.0. 0.01 0.05*,
Intensity of noise regularization (x)
nuisance CGANs
Leatning rale (1N)
0.005, 0.001, 0.0001, 0.0005
Plug-in OGANs) e @ITTW-CANS)
Minibatch size (bx)
32, 64
Toning strategy
random grid search with 50 runs Lusa: of generated sample
Tuning criterion
Number of epochs (Ma,))
RA-CGANs, GDR CGANs
Optimizer
Adam
Generatardisciminntoor hidden dimension (dgar)
degx o
Intensity ol moise regularization (of.r)
Learning rate (Wt)
0.00005
target CGANs
Minibatch size (by)
64
Tuning strategy
we tuning
Number of epochs (Me,T)
Optimizer
AdamW
Lateut variable dimension (dz,,)
3,5.7 7
Encoderidecnder hidden dimension (daux) Intensity of noise regularization (x)
3.5,
0.0, 0.012, 0.05², 0.1
muisance CVAEs
Learning rate (184) Minibatch size (Ir) Tuning strategy
0.01, 0.001, 0.005, 0.0001, 0.0005
Plug.in in CVAEs)
32, 64
IPTW-CVAEs)
random grid search with
Tuning uriterion
Number epochs (11,)) Optimizer
RA-CVAES, GDR
AdamW
Lateni variable dimnension (d&,T)
Fncnderkdecoder hidden dimension (dlar) Inteusity of noise regularization (22.))
10
Larget CVAEs
Learning rate (yr)
0.001
Minibatch size (br)
64
Toning stratogy
weo tuning
Number epochs (Me,))
Optimizer
AdamW
Number of diffusion steps (Tx)
50, 100
e-net hidden dimension (d.x)
10, 15, 20
Intensity of noise regularization (o
0.0, 0.012, 0.05², 0.1
muisance CDMs Plug-in CDMs) C IPTW-CDMs)
Learning rate (1p4)
0.001, 0.005, 0.0001, 0.0005
Minibatch size (bx)
32, 64
Tuning strategy Tuning critcrion
random grid search with runs Lara: of generated sample
Number epochs (M,,))
20
RA-CDMs, GDR CDMs
Optimizer
AdamW
Number diffusion steps (T2)
100
E-net outpur channels (d_.1)
128
Intensity of noise regularization (of,z)
target CDMs
I learning rate (vr)
0.0001
Minibateh size (b2)
64
Tuning strategy
w/o tuning
Number epochs (P,,T)
20
Optimizer
AdamW
E. 100 (synthetic data), 200 (IHDP dataset), = 50 (ACIC 2016 datasets), = 20 (HC-MNIST datnset)
4, (syathetic data), 10 (IHDP dataset), 15 (ACIC datasets), 64 (HC. MNIST dataset)
30

---

Published as a conference paper at ICLR 2026
Table Hyperparameter tuning for different meta-learners and generative models for (ii) colored MNIST experiments.
Model
model
Hyperpaaaaeterr
Range Value
Number of autorogressive layers (RU))
1. 1,3,4,0 4,6, 6,
Intensity of noise regularization (n(x)
0.0
Intensity of noise regularization (o))))
0.0, 0.01 0.05", 0.1
muisance CNFs
Learning rate (rpy)
0.001. 0.005
Plug-in CNF) IPTW-CNFs)
Minibatch size
32, 64
Tuning strulegy
random grid search with 50 runs
Tuning criterion
CNLL
Number epochs (f2,N)
10
CNTS.
Optimizer
Adam W
Number autoregressive layers (RLT)
Intensity of noise regularization (.))
0.0
Intensity of noise regularization (o))z)
0.01
target
Learning rale (1p1)
0.0001
Minibatch size (fr))
64
Tuning stralegy
w/o tuning 10
Number of epochs (f,,,)
Optiizzrr
Adam
Latent variable dimension (d,x)
50, 100, 200
Gemeratorffiscrimnnnttr oulpui channels (dguN)
64, 128
Intensity of noise regularization (dāx)
0.0
maisance CGANs
Learning rate (rnc) Minibatch (bx))
0.001, 0.0001, 0.0005
Plug-in in CGANs) ( IITWW-CANSs
32,
Tuning stratogy
random grid scarch with 50 runs
Tuning criterion
Lusr generated sample
Number of epochs (N,,,)
10
RA-CGANs, GDR-CGANs
Optimizer
Adam W
Latent variable dimension (ds,7)
150
Gencratorrdiscrmiiin output chanuels (dgur)
64
Intensity of noise regularization (02,)
0.0
target CGANs
Learning rate (n))
0.00005
Minibatch size (ir)
64
Tuning strategy
toning
Number of epochs (nt,1)
10
Optimizer
Latent variable dimension (d.,))
50, 100, 150
rncoder/dccodes output channels (duas)
64, 128
Intensity of noise regularization (or
0.0
muisance
Learning rate (rpe)
0.01, 0.001, 0.005. 0.0001, 0.0005
( Plug-in CVAEs)
Minibatch size (0x) Tuning strutegy
32, 64
IPTW-CVARS
random grid search with 50 runs
Tuning criterion
Cm
Number epochs (N.,n)
RA CVAES, GDR CVAES
Optimizer
Adam W
Lateat variable dimension (d.,1
50
Incoderfdecnder output channels (deag)
64
Intensity of noise regularization (d..)
0.0
Learning rate (177)
Larget
Minibaich size (br)
64
Tuning strategy Number of epuchs (H,,T)
tuning
10
Optimizer
AdamW W
Number of diffassio steps (Ty)
50, 100, 150
Number units middle layer U-Net (de,))
200,
c-net oulput channels (d, Intensity noise regularization (aā.)
32, 64, 128
0.0
muisance CDMs Plug-in CDMs)
Leaminng rule (1N)
0.01, 0.001, 0.005, 0.0001, 0.0005
IPTW-CDMs)
Minibatch size (bx) Tuning stratogy Tuning criterion Number of epochs (f4,,)
32, 64
random grid search with 50 runs
Cusn of generated sample
20
RA-CDMs, GDR-CDMC
Optimizer
Number of diffusion steps (Tr)
300
Number units in middle layer the U-Net (dg,)) E-net output channels (d_,1)
128
Intensity of noise regularization (a3,))
0.0
target CDMs
Learinng rate (17c)
0.0001
Minibatch size (by)
64
Tuning strategy
w/n tuning
Number epachs (NG,T)
20
Optimizer
Adam W
31

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENTS
E. SYNTHETIC DATASET
Results. In Fig. 5 we provide additional results for our GDR-learners based on the synthetic data, where we vary the strength of EMA smoothing for the target models, À E {0.5, 0.9, 0.995) (see Appendix for details). We see that the performance of different generative models remains relatively similar, with CDMs profiting from larger EMA smoothing values the most. Therefore, we set 0.995 for all the other experiments.
CNE
DEAN
CYARS
A-
0.123
0.100
1.5
F 9.05
0.31
0.00
0.5
0.00
0022
(+)
CNEs
0GAN
CYAB
CDMa
0.10
0.30
0108
01:55
Figure 5: Results for our GDR-learners on the synthetic experiments with varying size of training data (Teain) and varying strength of EMA smoothing (A). Reported: mean out-sample W2 se over 20 runs (lower is better).
E.2 IHDP100 DATASET
Results. We report the out-sample results for the IHDP100 dataset (Hill 2011 Shalit et al., 2017) in Table g
As expected, our RA-learners achieve the best performance. This happens due to the severe overlap violations in the IHDPI00 dataset (Curth & van der Schaar 2021 Curth et al. 202 and learners that use IPTW weights are theoretically expected to perform worse
0
Leamer
CGANs
CDMs
Plug-in
0,046
0,109
0.314
0.274
0.061
0.039
0.042 = 0.021
IPTW
0.059
0.130
0.681
0.385
0.132
0.164
0.050 0.029
RA
0.034
0.049
0.870
0.763
0,076
0.059
0.040
0,035
GDR
0.082 + 23.402
0.876
0.869
0.088
0.077
0.047 = 0.032
a=1
Learner
CNFs
CGANs
CVAEs
CDMs
Plug-in
0.040 :: 0.095
0.147 :: 0,386
0.089 :: 0.092
0.025 :: 0.031
IPTW
0.085 : 0.103
0.164 0.240
0.062
0.046
0.033 0.054
RA
0.028 = 0.065
0.390 + 1.152
0.7
= 0.100
0.024 = 0.052
GDR
23.829
0.519 0.976
0.110 0.115
0,021 0.047
Lower (best bold, second best undertined)
Table 8: Results for the IHDP100 dataset. Reported: median out-sample W2 + std over 100 train/test
splits.
E.3 ACIC 2026 DATASET COLLECTION
display the result for every of the 77 datasets in the ACIC 2016 collection. There, we see that our GDR-learners outperform the majority of other learners when the target model class restricted linear (= (b) linear setting).
Results. In Fig. 6
32

---

Published as a conference paper at ICLR 2026
10
-05
122
-2.9
Leaner
Plgi
RA
GER
-05
becena
100
4
CLECT on
HENGSAA
123
(7
FOR
(b) TrAe
X:
31
-11
(1) Tane
Lanner
Plagi
-0.5
RA
-14
301
GDS
6:0
200
EnVeOE on
1010011 oo-
WORSSTI
Figure 6: Full results for 77 semi-synthetic ACIC 2016 experiments in (a) full and (b) linear settings.
Reportdd Sespordf-MMiiin MMNSTppetteettt log-prob over 5 runs.
Results. We show the detailed results of Table 4 in Fig. Therein, our GDR-learners have slightly higher variance than the other meta-learners, mainly due to several outlier runs. This can be expected, the GDR-learners employ the inverse propensity scores and, thus, guaranteed to have good finite-sample performance. Yet, the GDR-learners outperform other meta-learners in terms of the median performance, especially for CNFs and CVAEs.
E.5 COLORED MNIST DATASET
Results. We report the quantitative results of the colored MNIST experiments in Table 9 Therein, GDR-learners almost always achieve best / becond-best performance in terms of W distance among other learners for all the generative models. This demonstrates good scalability of our GDR- learners the high-dimensional outcomes setting. Furthermore, report detailed results in
33

---

Published as a conference paper at ICLR 2026
1.3
8
68
0.-
00
ONE
CVAE
OGAN
CD
1.3
7244"
1.4
1.2
1.0
4,22
68
0.5
Figure 7: Detailed results for the HC-MNIST dataset. Reported: sample W2 for 20 runs. Note that the y-axis for CDMS is truncated and, thus, it omits some outliers for the plug-in learner.
Fig. 8 Here, interestingly, our -earnerss have relatively low variance, unlike in the HC-MNIST experiments (see Fig 7). This result could be expected, the colored MNIST dataset possesses relatively good overlap, given the one-dimensional confounder.
Learsr
CNFs
CGANs
CVAESS
CDMs
CNFs
CGANs
CVAEs
CDMs
Plug
46.15
7.88
29.98
7.56
53.59
32.57
20,80
49.15
12.04
19.65
5.58
41.65
36.65
24.95 6.68
IPTW
64.07 + 26.39
29.98
+ 6.04
32,90
13.87
25.48 + 7.04
73.84
+ 29.86 86
22.53
4.28
25.79 + 11.08
37.43 + 11.45
RA
46,02 10.53
31.84
4,58
38.26
9.16
20.29
2.38
37.99
16.46
17.62
4.49
22.00
9.72
26.51 J. 5.53
GDR
39.69 1 5.77
29.55
3.57
32.98
8.20
20.94 1: 1.71
29.03
9.10
17.94
5.47
18.89
3.51
24.73 6.10
a=2
6=
CNES
CGANs
CGANs
CVAES
Plug-in
42.69 + 9.90
24.08
7.57
49.02
32.24
18.44 + 2.97
39.51 9.40
22.67 7.75
47.01 + 12,85
19.21 + 3.27
5.90
25.88
RA
39.93 + 11.49
24.56 :: 5.68
33.24 = 9.43
18.92 1: 2.58
38.11 + 12:48
21.99 1 5.21
30.78 8.91
19.59 1: 3.03
GDR
2.72
Learner
CNFs
CGANs
CVABs
CDMs
Plug in
45.53
10.10
21.86 1 6.90
45.72
33.65
20.70 1 4.17
IPTW
67.06 +: 29.25
23.26
1 4.70
27.43 10.77
29.48 + 9.27
40.16 13.32
21.35
+ 4.25
28.20 =
20.44
GDR
32,55 - 7.89
20.55
1: 3.50
24.21 =: 4.43
20.60 2.71
Lower
better (best i bokd, second underlined)
Table 9: Quantitative results for the colored MNIST dataset. Reported: mean out-sample W₂ = std
over 10 runs.
E.6 RUNTIME COMPARISON
Table 1O provides the runtime comparison of different neta-learners and generative models. Here, the runtime of the RA- and EDR-llerrnes is roughly twice as long as that of the plug-in in and IPTW- learners. This expected, as both RA- and GDR-learners have two learning stages. Still, our GDR-learners are well scalable.
Learner
CNFs
CGANs
CVAEs
CDMs
Plug-in in
1.25 = 0.02
0.99 0.01
0.71 1 0.01
0.35 + 0.02
IPTW
1.26 = 0.01
1.10
0.001
0.35 1 0.01
+ 0.02
1.89
189+000
1.25
0.69 + 0.01
GDR
2.40 = 0.04
2.01 -: 0.02
1.25 0.00
0.68 1 0.01
Table 10: Total runtime (in minutes) for different meta-leamers and generative models based on
the synthetic experiments with Prain = 500. Reported: mean duration + std (lower is better). Experiments were carried 2 GPUs (NVIDIA A A100-PCIE-40GBB with IntelXeon Silver 4316 CPUs @ 2.30GHz.
34

---

Published as a conference paper at ICLR 2026
100
130
100
80
80
00
60
OGAN-
CDMs
120
80-
100
25
80
80
40
15
20-
100
35
80
30-
20
20
10
OGAN-
CDM
30
100
40-
35-
80-
20
30-
80
00
20-
3
20-
20
10
15
100
100
80
25
20
60
30
20
Plagein
JPTW
R.A
GDR
Piugei
IPTW
R.A
GDR
Plagsin JPTW
RA
GDR
Plagsin JPTW
RA
GDR
Figure 8: Detailed results for the colored MNIST dataset. Reported: out-sample W2 for 10 runs.
35


Reviews:
[{"review_id": "d3lpeswYox", "reviewer": "Reviewer_UrWJ", "summary": "The authors propose an approach to generate/model potential outcomes using deep generative models such as diffusions, GANs etc in a wya that preseves the neyman orthogonality property, which allows for double robustness and efficient semiparametric type guarantees.", "questions": "1. Typically, one thinks of these DR/Neyman orthogonal approaches are using ML to nonparametrically handle a nuisance with the bigger goal to infer a finite dimensional parameter of interest, in a way that the slow convergence of the nonparametric part doesn't affect the nice properties of the finite dimensional inference on the parameter.  In your case, the parameter of interest appears to be the CDPOs, which themselves are nonparametrically generated. In this case, one can reasonably think that learning the CDPOs themselves might be a more difficult problem than learning the nuisance, which calls into question the value of double robustness/quasi oracle efficiency in this setting. For example ,what if the rate of learning the CDPOs is even slower than estimating the nuisance? \n\n\nMinor points: the abbreviation GDR is used in the abstract before definition", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Originality: The ideas in the paper are original and non-trivial in my opinion. \nSignificance: Causal ML and Neyman Orthogonality are certainly \"hot\" topics as of late, and bringing that together with deep generative models is a timely contribution that I find significant enough to meet the bar for ICLR. \nClarity: The clarity/presentation style of the paper is something that needs some minor improvement. I think the expert will have no trouble understanding the paper, but for a general ML/AI audience this paper will be a rather difficult read. \nQuality: the quality and soundness of the math/theory portions of the paper is high. I find the arguments to be well reasoned and to the best of my knowledge, sound. The authors also performed some experiments which I find sufficient for this paper.", "weaknesses": "Clarity of presentation and accessibility of writing are the biggest areas that this paper can improve on. I highlight some aspects of this below. \n\n1. GDR, Neyman Orthogonality, Double robustness etc are the core concepts that underlie and motivate this paper. Yet, neyman orthogonality was not formally defined until page 6 of the paper (inside a theorem, and then more formally in the appendix), and double robustness, quasi-oracle efficiency etc are nowhere formally defined in the main text (high level explanations in 2nd page and formal definitions only given in the appendix). this is understandable given space constraints, but I believe some more formal/rigorous explanation of these terms in the beginning of the main text is *absolutely required*. These are not part of the standard training/vocabulary of a typical ICLR or general AI/ML reader, and clarifying these terms will help broaden the reach of the paper to a more general audience. \n\n2. There are important references (canonical, even) that this paper seems to have missed. Many examples in econometrics come to mind, with the double machine learning paper by Chernozhukov et al and related semiparametric efficiency papers (e.g. Whitney Newey) not mentioned. There are too many references to be listed here. I suggest giving more proper reference to the econometrics literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BE6ZrgxjYP", "reviewer": "Reviewer_ebKH", "summary": "This paper proposes GDR-learners, a general two-stage framework for estimating the conditional distributions of potential outcomes from observational data using deep generative models. The key idea is to combine estimates of conditional distributions of potential outcomes with a doubly robust, Neyman-orthogonal target loss so that errors in estimation do not affect the second-stage objective to first order. The framework is model-agnostic and can be instantiated with conditional normalizing flows, GANs, VAEs, and diffusion models. The authors claim quasi-oracle efficiency and rate double robustness in theory, and report empirical gains over plug-in, RA, and IPTW alternatives on synthetic and semi-synthetic benchmarks.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method and objective is model-agnostic and applicable to multiple generative models, including CNF, CGAN, CVAE.\n2. The proposed framework is able to computer conditional distribution of potential outcomes.\n3. Empirical results show improved performance compared with other models.", "weaknesses": "1. Some theoretical conditions (“mild model-dependent convexity and positivity”) are not fully unpacked. The identifiability condtion i - iii are not clear.\n2. The proposed method show show very large reported standard deviations in some datasets, such as HC-MNIST. Is this considered as instability. Or some other reason?\n3. In additional to the median out-sample w2+std results, it is necessary to report the worst case (e.g., max difference between the estimated distribution and the truth) estimation results. This result will show the performance under failure modes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3JTs3XBEj", "reviewer": "Reviewer_fv3L", "summary": "This proposes to employ conditional deep generative models for estimating the conditional distribution of a potential outcomes in a causal framework for conditional outcomes.  The key contributes lies in an estimation procedure (with appropriate objective functions) that guarantees  the general Newman-orthogonality constraints.  They also provide some theoretical guarantees by establishing the Quasi-oracle efficiency and double robustness.", "questions": "Q1: can you provide more intuition and explanation of the bias correction of RA learner behind equation (7)? First glance, it weights the RA and IPTW learners. \n\nand a quick comment: GDR. should be defined the first time it was introduced ( in the abstract)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+propose a class of deep generative models for modeling the conditional distribution of the potential outcomes that satisfy the Neyman-orthogonality and establish quasi-oracle efficiency and double robustness of the learners \n\n+ the paper is well-written with very clear presentations (e..g, the graphical presentation in Figure 3 is extremely helpful) \n\n+a thorough numerical study is included", "weaknesses": "-in my opinion, the key contribution lies in the formulation of the objective functions that ensures the Neyman-orthogonality (as given by equation (7)). This is not a strong weakness per say, but the degree of novelty is somewhat in question \n\n-given equation (9), one should further discuss the rates of the error based on the various conditional deep generative estimators, utilizing an existing  literature on theory and convergence rates of conditional VAE, conditional discussion models and. so on. \n\n\n- \n\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ak0w9gF1XN", "reviewer": "Reviewer_VATR", "summary": "This paper introduces GDR-Learners (Gradient Disentangled Regularization Learners), a framework for causal representation learning that enforces gradient-based orthogonality among latent variables. The key idea is to regularize the model such that the gradients of reconstructed inputs with respect to different latent dimensions are orthogonal, promoting disentanglement between the causal mechanisms encoded in each latent. Theoretical analyses suggest that under mild assumptions, this orthogonality constraint helps achieve identifiable and disentangled representations up to monotone transformations. Empirical results on standard disentanglement benchmarks show consistent improvements over state-of-the-art baselines.", "questions": "See Weaknesses Part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important challenge in causal representation learning: achieving identifiable and interpretable factors without strong structural assumptions.\n2. The proposed GDR term is simple to compute, model-agnostic, and can be seamlessly added to a wide range of encoder–decoder architectures.\n3. Experimental results show that the method improves disentanglement scores across several datasets and remains robust when combined with other regularization terms.", "weaknesses": "1. While the paper provides some intuitive propositions, it does not establish a formal link between gradient orthogonality and causal identifiability. The theoretical results remain qualitative and lack proofs that guarantee uniqueness or sufficiency for disentanglement.\n2. The paper mainly compares against unsupervised disentanglement models (β-VAE, FactorVAE, etc.) but omits direct comparison with causal representation learners such as iVAE or CausalVAE, which share similar objectives and stronger theoretical grounding.\n3. The evaluation focuses entirely on disentanglement metrics. There is no demonstration that GDR-learners improve causal inference capabilities.\n4. The gradient regularization term introduces additional computation and hyperparameters (e.g., λ). The paper does not provide runtime analysis or ablations to assess stability and sensitivity across hyperparameter choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)