Marginal Causal Flows for Validation and Inference
Authors: Daniel de Vassimon Manela
Laura Battaglia
Robin J. Evans
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal Inference
Normalising Flows
Synthetic Data
Marginal Structural Models
Primary area: causal_inference
TL;DR: We show how Normalising Flows can be used to explicitly parameterise marginal causal distributions, and illustrate its utility for both inference and synthetic data generation/benchmarking.

Abstract: Investigating the marginal causal effect of an intervention on an outcome from complex data remains challenging due to the inflexibility of employed models and the lack of complexity in causal benchmark datasets, which often fail to reproduce intricate real-world data patterns. In this paper we introduce Frugal Flows, a likelihood-based machine learning model that uses normalising flows to flexibly learn the data-generating process, while also directly targeting the marginal causal quantities inferred from observational data. We provide a novel algorithm for fitting a model to observational data with a parametrically specified causal distribution, and propose that these models are exceptionally well suited for synthetic data generation to validate causal methods. Unlike existing data generation methods, Frugal Flows generate synthetic data that closely resembles the empirical dataset, while also automatically and exactly satisfying a user-defined average treatment effect. To our knowledge, Frugal Flows are the first generative model to both learn flexible data representations and also \textit{exactly} parameterise quantities such as the average treatment effect and the degree of unobserved confounding. We demonstrate the above with experiments on  both simulated and real-world datasets.

Marginal Causal Flows for Validation and Inference
Daniel de Vassimon Manela' University Oxford manela@stats. ox. ww.oo...cc..... ac.uk .uk
Laura Battaglia" University of Oxford
Robin J. Evans
University of Oxford
battaglia@stats ox ac. uk
evans@stats ac uk
Abstract
Investigating the marginal causal effect of intervention complex data remains challenging due to the inflexibility of employed models and the lack of complexity in causal benchmark datasets, which often fail reproduce intricate wallworld data patterns. this paper we introduce Frugal machine learning model that normalising
outcome from
Flows, novel likelihood- flows to flexibly learn the data-generating process, while also directly inferring the marginal causal quantities from observational data. We propose that these models are exceptionally well suited for generating synthetic data to validate causal methods. They can create synthetic datasets that closely resemble the empirical dataset, while automatically and exactly satisfying a user-defined average treatment effect. To our knowledge, Frugal Flows are the first generative model both learn flexible data representations and also exactly parameterise quantities such the average treatment effect and the degree of unobserved confounding. We demonstrate the above with experiments on both simulated and real-world datasets.
Introduction
Simulating realistic datasets such that the marginal causal effect is constrained to take specific form significant challenge in causal inference. Many methods for inferring these effects exist, but simulating from them is a significant challenge (Young et al.) 2008 Havercrolt and Didelez 2012 Keogh et al. 202Tp. particular, it is difficult to simulate complex benchmarks from generative models in such way that custom marginal effect exactly holds.
The frugal parameterisation (Evans and Didelez 2024) provides solution this problem by constructing a joint distribution that explicitly parameterises the marginal causal effect and builds the rest of the model around it. Frugal models typically represent the dependency between outcome and pretreatment covariates using copulae. Standard multivariate copulae are parametric, leading to potential model misspecification.
In this paper we show how one can construct frugally parameterised marginal causal models using normalising flows (NFs, Rezende and Mohamed 2015 Dinh et al. (016)) target the causal margin of the distribution (a conditional univariate marginal density of an outcome conditioned on a treatment). We name the resulting model a Frugal Flow (FF). To the best of our knowledge, FFs offer the first likelihood-based framework for learning marginal causal effect while modelling the outcome and propensity nuisance parameters using flexible generative models.
FFs are exceptionally well suited for generating benchmark datasets for causal method validation. Since FFs enable direct parameterisation of the causal margin, they provide framework for gener- ating causal benchmark datasets which resemble real-world datasets, but which also allow users to encode causal properties in order to validate novel inference models. FFs can be used to generate benchmarks with customisable degrees of unobserved confounding. This can aid the validation
Equual Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of model robustness under conditions where the assumption of conditional ignorability does not hold. Here, conditional ignorability (or conditional exchangeability) means that marginal distribution of the potential outcomes is independent of the value of treatment, conditional on the observed covariates (Pearl 2009).
FFs offer marked improvements over current benchmarking generation methods, which use soft constraint optimisation to enforce the desired causal restrictions (Kendall 1975 Parikh et al. 2022). As result, post hoc checks are required to see whether these conditions are present in the synthetic data. FFs do not require this second step, as relevant conditions are explicitly encoded in the underlying likelihood. Finally, FFs allow for outcomes to be sampled from marginal logistic and probit models, making them the first generative benchmarking model facilitate the simulation of binary outcomes with a choice of user specified risk differences, risk ratios, or odds ratios.
2 Background
In this paper we consider a static treatment model with an outcome Y E y C R and T' binary treatment T {0, Let the set of measured pretreatment covariates be Z Z C R. Additionally, we will use the notation of Pearl (2009) where intervened distributions are indicated by the presence of "'do-)" operator, with absence indicating that the distribution is from the observational regime.
2.1 Marginal Causal Models
Causal inference methods are generally developed to estimate the average effect of treatment (T) on an outcome (Y) for population defined by set of pretreatment covariates (Z) (Hernán and Robins 2020). Let the variables be distributed according to T, Y) ~ PZTY with density pzry. We make the standard assumptions of a stable unit treatment value (commonly referred as SUTVA), positivity, and conditional ignorability (equivalent to conditional exchangability) outlined in Pearl (2009). Additionally, the covariate set Z must only include pretreatment covariates. The conditional distribution of Y and Z after an intervention on T is equal to
doy)) (z, y t) = pz(=) Py Py] (y/ 2, t).
Causal practitioners are often interested in the marginal effect of T on on the intervened system, sometimes referred to as the marginal outcome distribution (MOD), py do(T):
PY]do(T) (y 1) = / dz PYiZ,OoTT)(U 2, pz()).
The difference between the means of Y under this margin between different values of T is called the average treatment effect (ATE), T where, т = IE[Y do(T = 1)] EEY do = 0)]. Models which target this marginal quantity are known as marginal structural models (MSMs, Robins 1998) and are frequently used in cpidemiological and medical domains to account for time-varying confounding, In particular, they effective at quantifying the effect of intervention over a population, where the specific relationships between the outcome and (possibly high dimensional) pretreatment covariates not relevant, and modelled as nuisance parameters. The semiparametrie question of estimating finite dimensional quantities in the presence of high dimensional nuisance parameters has a long history (Robins et al. 1995 Robins and Rotnitzky 1995), but has undergone a renaissance since the development of methods such as targeted maximum likelihood estimation (van der Laan and Rose| 2011) and double machine learning (Chernozhukov et al. 2018), which allow for general machine learning algorithms to flexibly describe the nuisance models and still have valid inference on dowddiinnli treatment effect.
2.2 Frugal Parameterisations
Frugally parameterised distributions consist of three distinct components: the distribution of the 'past,' OZT: the intervened causal quantity of interest, 0y do());; and an intervened dependency measure between Y and Z conditional on T, (bzY)do(T)) The key idea is to explicitly parameterise the marginal causal effect, and build the rest of the model around In this paper we encode all the dependence among covariates in the copula, so 'the past" is really just the propensity for treatment

---

(also called the propensity score) and the product of the univariate margins (Evans and Didelez 2024). Figure provides an illustrative summary of our framework, and outline which models are used to parameterise each component of frugal model.
0z: {Fzs = Fz d=1
Z
do()) : CZY]do(T) ( Yydoo(yy Fz1 ...., F2D
Onz Priz = CTZ (FT+)
1Fz
F
Zo
T
Y
0yldo(r)= Py do(T) Yido(T)
Figure A visual abstract outlining the different components of frugal model, and how each specific component is parameterised. Univariate CDFs are denoted by F, and copula distribution functions are denoted by C. The marginal causal effect, dy]d)(T)> is modelled with univariate normalising flow, which we denote by F (see Section 2.4) The intervened dependency measure, dzy])o(T)) is modelled with copula flow which we denote by C (see Section 2.5 The past, 8zr, is modelled by the combination of univariate normalising flows (for the univariate pretreatment covariate distributions) and a copula flow (for the propensity of treatment).
Variation Independence Any smooth and regular 'dependency measure' can be chosen for pa- rameterising @zY]do(T); this is defined as quantity which, when combined with the marginal distributions, smoothly parameterises the joint distribution. It is desirable that the three parameter sets (0zT, do)d))()) ) do(T)) are variation independent BBarnorrf--Nelson 2014) of each other; such parameterisations have the benefit of allowing the measure Øzy do(T)) to be Ireely specified without restricting the rest of the model. Copulae are an example of such a dependency measure, and are natural choice for frugally modelling dependencies in continuous and mixed datasets. For further detail we refer the reader to Appendix AI
2.3 Copulae
multivariate copula, denoted by [0, 1J [0, 1] is a multivariate cumulative distribution function (CDF) defined over set of d uniform margins, with associated density c(-) if it is continuous with respect to its arguments (Sklar 1959; Joe 2014). Copulae are often used to parameterise the dependency structure of joint distribution independent of its univariate margins. Large, complex dependency structures are often modelled by pair-copula constructions (PCCs) or vine copulae (Czado and Nagler 2022; Joe and Kurowickal 2011). These methods factorise the dependency structure into a set of non-overlapping bivariate copulae. However, these approaches typically impose the constraints of finite dimensional parameterisation the dependency structure in the bivariate copulae used. A more comprehensive introduction to copulae can be found in Appendix B
Copulae in Machine Learning More complex ML models have been developed to more flexibly learn copula distributions. Several alternatives have been proposed, some targeting specific copula classes (Ling et al.] 2020 Wilson and Ghahramani 2010), and others constraining neural network- based architecture to estimate valid copulae, though often with limited scalability (Zeng and Wang 2022; Chilinski and Silva, 2020) using variational approximations (Letizia and Tonello (2022). However, the most active research area in this field makes use of normalising flows, leveraging their likelihood-based, composable and invertible nature to chain transformations of marginal quantities to the fitting of the copula density.
Paper Motivation the copula
A key motivation for this paper is the search for a flexible parameterisation of
(ZY]do(T)(Z (2. t) =>)==<FFFramae c(Fr)ao(TT(s) 1t), F2, ..... Fz. (2D))
(2)
3

---

between the probability integral transforms of the univariate pretreatment covariates and conditional univariate quantity which parameterises the causal margin. Evann and Didelez (2024) show that this can be done using parametric copulae, and also prove that it targets the marginal causal rather than the conditional distribution when ( is parameterised by a multivariate copula. Consider the multivariate copula for the distribution of Z and Y conditional on T:
al d(Frr ..., F2.1T).
For an intervened distribution, all pretreatment covariates are marginally independent of of and so the intervened joint density becomes
Z = Pr]da(T) c(Fr)do(T) Fz....., Fzo).
where do() is the marginal causal effect of on Y. The final propensity score density Pxiz does not affect the marginal densities in the observational model as there is a parameter cut between 2014). However, Oy]do(T) and do()) are functions of
Priz and Py ( (Barndorff-Niell Pr]Z,d((T) and thus should be estimated jointly. py]do(2) estimated separately from the copula, the marginal conditional effect will be inferred rather than the marginal causal effect.
Generative ML methods allow for estimating more flexible and general copulae, but have struggled so far to learn copulae together with conditional univariate quantities. We resolve this problem and design NF-based copula inference method that allows for these quantities to be estimated jointly as required by the frugal parametrisation (see Section 2.5). The model is then trained on real-world data and used for generating customised causal benchmarks which closely resemble the original dataset.
2.4 Normalising Flows
2015 Dinh et al. 2016) allow for density estimation via learning diffcomorphic transtormation F that maps the unknown target distribution px (æ:). x E R.D to a simple and known hase distribution pv (u). u E TR D so that when X px and U pu then U = FF=-((XX (X).
Normalising flows (NFs) (Tabak and Turner, 2013 Rezende and Mohamed
F is usually composition of invertible and differentiable transformations F, parametrised by neural networks, and is often trained by maximising the log-likelihood of observed {a:) This can be conveniently done in closed form exploiting the change of variable formula
[O(F ( (((5-((1)))
Px (x) = pu (F- det
(3)
dx
provided that the chosen model for F allows for efficient computation of the Jacobian determinant det()((=1(x)) ((x)))rrr), The implementation of F-1 then allows for density evaluation, whereas J can be used for sampling from the joint.
As for the choice of F, the literature has explored a number of implementations that retain invertibility while allowing for computational tractability of the determinant. See Paaamakaaios al. (202I) for an introduction and overview. Our implementation relies on neural spline flows (NSF, Durkan et al.) 2019).
particular type of autoregressive flows that will be further illustrated in Section 2.5
2.5 Copula Flows
Our Frugal Flow approach builds upon the copula-based flow model proposed by Kamthe et al. (2021) for synthetic data generation. The authors start by considering copula C(Fx Fxp) defined over the marginal probability integral transforms Fx,..... Fx, of a random vector X "XI....,pp) Assuming the copula density exists, the joint density of X can be written as
D [Ixx.
( xa) = cx (Fx, (21), Fx,()D))
where Px, is the marginal density of X2- This factorisation of the density can similarly induced by a NF that composes D flows Fi, ......,2 FD for the marginal quantities and flow Cx for the copula.
For the rest of this paper, we will let U Uniform (0, 1) represent a vector independent uniforms, and let V C' represent vector of dependent uniforms a multivariate copula C.. The generative

---

procedure for this NF takes samples U from base distribution of independent uniforms and first pushes them through the copula flow Cx. obtaining correlated uniform samples V Cx (U). Then V is mapped through the marginal flows Fx Fx.....,Fxx to obtain the random vector
X = Fx
The composed flow X = as in eq. (3) it induces specific factorisation of the density of X Here, we quote the result from Kamthe et al. 202TT:
is also a valid flow, and via the change of variable formula
10Fx (X)
(x)
px - pv (X)) det
0x
aC ( (x)) OJ x (X)
D
OJ OF xa
= der
(5)
0Xa
As the univariate mapping from a uniform to random variable is uniquely defined by the CDF, the flows Fx [Fx.....; Fxp target the marginal CDFs Fx,..... .F,, Note how eq. the density of X into copula density and product of marginal densities as in eq.
5 factorises
Cx is estimated with a NSF, a of the autoregressive flow class, Autoregressive flows (Papamakar
ios et al. 2017 Huang et al. 2018) factorise Cx as a recursive sequence of univariate conditional
flows:
1::0C(U1)
Va == Cap...-- (Us) Vi...., Va-1)
2
(6)
In principle, since the input U is a vector of independent uniforms, the conditional flows would approximate the inverse of the Rosenblatt transform (Rosenblatt) 1952) and thus be universal approxi- mators if the flows were sufficiently expressive (Papamakarios et al., (021). ) The Rosenblatt transform sequentially maps each component S₂ of any random vector S with strictly positive density through its corresponding conditional CDF Fss, Sa.....a-11' obtaining a vector U of independent uniforms. known to be a diffeomorphism, so inverse bears the same structure as eq. (6). but uses inverse conditional CDFs C dd-
for each Va. We use the notation C-1 to emphasise that in the copula flow case we are dealing with inverse copula CDFs, whose codomain also uniform.
Autoregressive flows estimate each univariate conditional flow Can...--1 with a strictly monotone function whose parameters are only allowed to depend dimensions 1,..., d 1. The monotonicity of the function ensures invertibility, while the autoregressive structure in the function parameter dependence gives a triangular Jacobian whose determinant is tractable. Kamthe et al. (202T) use NSF, where the monotone function is given by a monotone rational quadratic spline, whose knot parameters are provided by a neural network where weights autoregressive structure. The univariate marginal flows Fx are estimated with separate NSFs before training the copula flow using the transformed data V.
appropriately masked to ensure the
While a NSF can constrain the support of both the base and target distributions, it cannot control the form of the marginal distribution. If marginal and copula flows are learned simultaneously, neither will be correctly inferred due to the infinite possible combinations of (Fx. : Cx which yield the same composite flow W Fx Cx. These flows must be learned sequentially if C is model a copula.
In our application, we wish to infer multivariate copula which models the joint dependence between univariate pretreatment covariates and conditional univariate quantities such that the latter parameterises the causal margin. Inferring the MOD separately from the copula, as copula-based flows do, will target the conditional causal effect rather than the marginal causal effect. We propose solution in the form of Frugal Flows, which we introduce in Section [3..TT Moreover, for discrete variables use a dequantised form of the empirical CDF rather than NSF adaptation (see Appendix B.2 for further details).
2.6 Validating and Benchmarking Causal Methods
Methods for validating causal models can be broadly categorised into two groups. The first comprises auxiliary analyses conducted after fitting causal model and estimating treatment effect. These include but are not limited to sensitivity analyses (Imai et al. 2010), subgroup analyses (Cochran and Chambers 1965), placebo tests (Eggers al.) 2023). and negative controls (Shi et al. 2020).
5

---

The second set of validation methods is where we see FFs having significant impact. These methods are used construct synthetic datasets while allowing the causal practitioner customise specific features of the data-generating process. For example, when validating an inference method which estimates an ATE under certain confounding assumptions, is crucial that generated data follow the "ground truth" ATE and confounding assumptions one wishes to measure. However, synthetic data risk being oversimplified and contrived, failing to reflect the complexity of real world datasets.
To mitigate this, generative models are trained on real-world data and calibrated to generate samples with modifiable causal constraints. Such constraints include the average causal treatment effect, unobserved confounding, and positivity. To our knowledge, the FF framework proposed in this paper the first method to allow all of these conditions adjusted by the user. Existing methods (Neal) et al., 2020; Athey et al., (02T]] Parikh et al. 2022) encode these effects through soft optimisation constraints, hence there no guarantee that the constraints are satisfied, Enforeing these constraints too strongly may negatively impact model optimisation, and may affect the reconstructive ability of the underlying model. Furthermore, since these approaches do not explicitly parameterise the causal effect, samples from trained models must be tested post hoc ensure the desired constraints are present in the sampled data. A key benefit of frugal models is that the marginal causal effect is directly parameterised by the user through the likelihood. As result, synthetic data samples will exactly satisfy these constraints.
3 Method
3.1
Building the Joint Distribution
In this section we parameterise the full observational joint using FFs. Section 3.1.1 outlines how the FF is constructed; we first learn the probability integral transforms of the pretreatment covariates, and then infer the causal margin jointly with an extended copula flow, the Frugal Flow. To infer the causal margin, this is sufficient. Nevertheless, the propensity score is needed to complete the joint in order to generate benchmarks which are confounded in a similar fashion to the original real-world dataset. We describe the fitting of the propensity score in Section 3.1.2
3.1.1 Constructing Frugal Flows
The first step involves learning the margins for the pretreatment covariates Z. This is done in similar fashion to that of Kamthe et al. (202T)'s 's copula-based flows, as described in Section 2.5 The outcome, treatment, and the inferred ranks Vz of the pretreatment covariates are then used to train
the Frugal Flow (see bottom part of Figure 2 that models do())) together with the copula flow. This is required in order learn the causal marginal PY] do(T) rather than the conditional pyJr-
The Frugal Flow of dimension D+1 1 transforms the joint input of (Y, Vz do(T)) into a random vector U which we set to be distributed according an independent uniform base distribution. In the first subflow of the composition, Y is pushed through univariate flow Fy)do(T) conditioned T to obtain VylaotT), while the Vz remain untransformed. Subsequently, Vyj kept fixed, while a copula is learnt over Vz conditional on Vy|do(7)) via an NSF Importantly. specific ordering of the variables is imposed, such that the causal margin is ranked first. In this way, we ensure that Un and (T) have the same distribution, and do]))(T) is therefore constrained to be uniform. The marginal flow F d|do(T) thus targets the CDF of the marginal causal effect, [do())-
In summary, we construct a flow Q-1 : VY.
T) - V as a composition of a marginal flow Fr|do(T) and conditional copula distribution C C- (DV]joyo)) U2, .22)
C(uz,...., Uy]do(T)). More on the implementation details can be found in Appendix C
3.1.2 Learning the Propensity Flow
We constructed the conditional distribution of Y and Z after an intervention on T in Section 3.1.1
P2Y]do(T)(z)) (2 = II
pz, (zi)
-Pridorr (2, 1) (oy) ......, U7p))

---

11
Fz (1)
Vz1
:
Vz
[D]
Fi
VZo.
Y do(T) Vz
F (1) (F Fiidorm (Vido(TT Vz
Un c(uz Dy(do(T))) U2(D+1)
c(Uyjao(T))
=1 1)
NSF
Figure 2: Structure for learning a Frugal Flow. The top line outlines the process for learning the univariate marginal flows the pretreatment covariates Z. The bottom transform illustrates the Frugal Flow, which learns the conditional copula c(Uz Uy]do(T)) jointly with the causal marginal flow doT)) by enforcing (o())) to be marginally uniform.
Inferring the above is sufficient for identifying the causal margin. However, to generate realistic samples for causal method validation, also needs learn the propensity score, Pr z Pr cTiz. By decoupling the marginal treatment density pr from the conditional copula cT one can can modify the marginal treatments while retaining the dependence of the original data. We therefore learn an approximate probability integral transform the discrete treatment T (see Appendix B.2. for further details), followed by the conditional copula flow of I on Z, Cr VT + Vriz Z.
One could directly model Priz using normalising flow, which would also constitute a valid frugal model. We instead choose to model the conditional copula using flow, Crz CrJZ allowing users to encode degree of unobserved confounding in the generated data by sampling the ranks Vry and Vy)do(T) from non-independence copula. Assuming ignorability, these ranks would independent. However, unobserved confounders imply dependence between these ranks. Sampling them from copula can replicate this effect, as demonstrated in the far-right plots in Figures Bandr
The above section describes how one can estimate the propensity of treatment from a real-world dataset. However, we remark that one can choose any custom propensity score function generate treatments conditional on the pretreatment covariates via inverse probability integral transforms on Vriz. Hence, one can fully control the overlap/positivity of FF generated benchmark datasets.
3.2 Generating Synthetic Benchmarks
Data generated from a fitted FF can be customised with a range of properties, allowing for model validation against range of customisable causal assumptions. We describe these below.
Modifying the Causal Margin The central output of the Frugal Flow is method for sampling ranks for each of the margins in PyJa Pz........ .z.- Any causal marginal density ạyl do(r) can be used to generate samples of Y via inverse probability integral transforms. Since the Frugal Flow returns ranks for the intervened causal effect, these can be inverse transformed by any valid CDR Unlike other methods, this constraint is strictly enforced by the the frugal likelihood.
Simulating from Discrete Outcomes Since FFs return Vy|do(T) ranks, one can sample from any custom causal margin. This extends both continuous and discrete causal margins. One can simulate from a logistic marginal effect Y do(T) ~ Bernoullipp model Y do"T")
expit(37' + c)) or probit Bernoullipp ( c)) where 4(-) is a univariate standard Gaussian CDF. This is non-trivial, because logistic regression is not collapsible, meaning that if (for example) Y Y t, Z z is a logistic regression, then Y | do(T = t) generally will not be. Hence is infeasible for fully conditional method of simulation to produce outcomes where the causal margin uses logistic link. For experimental results see Appendix D.2.1
Modifying the Degree of Unobserved Confounding One can sample data from FFs as if the outcome is affected by unobserved confounding. The variables Vy and V/1z are independent of each other if no unobserved confounding assumed. Introducing dependence between these ranks replicates the effect of unobserved confounding. This can be achieved by sampling (Vrjao(T)) Vriz) from a Gaussian bivariate copula, e(Uy]do(T)) UT|Zi P), where p quantifies the degree of unobserved confounding in the sampled data.
7

---

Customising Treatment Effect Heterogeneity Consider stationary treatment with pretreament covariate set = (W, W) where W C with Z D. W d, and W| = - d. We
proceed considering the case where d < D. Interest may lie in the causal treatment margin conditional on the subset of variables W:
AY (y w,t) Jw dw PriZab(T)) (v w. w. PWIW W W w)
(7)
We propose method to exactly parameterise heterogeneous treatment effects using subset of pretreatment covariates, W C Z. FFs offer exact parameterisation of Wlo))))(T) allowing for customisation of heterogeneity while capturing complex dependencies between other covariates. Specifically, the model infers the conditional treatment margin, PY]W, do(r) (1 wo D), ensuring proper inference the joint pretreatment covariate distribution, (-))). Thus, one may simulate data where causal effects conditional selected subset of variables, offering flexible and precise control over treatment heterogeneity. Further details may be found in Appendix D.2.2
Customising the Propensity Score Since the propensity score variation independent from the rest of the model, one has complete flexibility on how to parameterise the propensity score. Any distribution Priz can used generate treatments with varying degrees of overlap in a manner that is completely customisable by the user.
Experiments
The following section discusses our experiments and results, which aim to i) demonstrate that FFs accurately infer the true MOD for confounded data, and ii) show how a trained FF can generate synthetic datasets that meet specified causal margins and unobserved confounding.
4.1 Inference
We generate simulated data from three models. The first two parameterised by four pretreatment covariates Z [Z,..... Z1} with a binary treatment T, a linear Gaussian causal margin Y dT) Niu T + 1,0 1). and copula dependence measure c(Uy Uz,..... Uz.). In the first model M1, all four covariates follow a gamma distribution. In the second M2, the data is generated from model M3 with ten pretreatment covariates comprising five gamma and five binary variables. A more quantitative description of the simulated data generating process and hyperparameter values presented in Appendix D.
split of gamma and binary covariates. Additionally,
generate data from
Table 1: Mean and 2o confidence interval of the inferred ATE, bootstrapped over 25 different runs and with a data size of N 25,000. The number of pretreatment covariates in each model is denoted by Bold confidence intervals contain the true ATE. OR quotes the results obtained by linear outcome regression, and CNF reports the ATE estimted by causal normalising flows.
Model
True ATE
D
Frugal Flow
OR
Matching
CNF
Mi
1
4
0.98 + 0.12
1.28 + 0.06
0.78 + 1.06
0.73 + 0.16
Mi
5
5.00 + 0.24
5.29 - 0.04
4.68 + 1.06
4.23 + 0.20
M2
1
4
1.01 + 0.10
1.46
1.36 0.72
1.01 - 0.20
M₂
5
5.01 0.18
5.44 0.05
5.55 - 0.88
5.03 + 0.44
Ms
1
10
1.00
+ 0.09
1.13 0.06
0.90 0.48
0.87 + 0.15
M3
5
10
5.18 + 0.30
5.13 + 0.26
4.90 + 0.47
4.73 + 0.28
We generated datasets with sample size of N 25,000 across B 25 different runs. Frugal Flows (FFs) were compared against outcome regression (OR), traditional causal propensity score matching (Stuart) 2010), and state-of-the-arr causal normalising flows (CNFs) (Javaloy et al. 2024). Further details on the methods can be found in Appendix D.3.3 A default set of hyperparameters was used for all models. The estimated ATEs are shown Table OR models, which estimate the conditional rather than the marginal effect of I on Y. consistently exhibited bias, pulling the

---

estimates away from the true value. In contrast, FFs achieved the lowest error in identifying the true ATE, outperforming both statistical matching and CNFs.
Our results demonstrate that Frugal Flows can correctly identify causal relationships under ideal conditions, confirming that they are valid, efficient way parameterise causal model using deep learning architectures. A drawback of FFs is that they need large datasets to accurately infer causal margins. Additionally, the complexity of data dependencies might require careful hyperparameter tuning to prevent the copula from overfitting, which could bias the inference of the causal relationships. Because of these challenges, we do not recommend using FFs on real-world datasets for statistically inferring treatment effect sizes, as causal benchmark datasets are usually small.
4.2 Benchmarking and Validation
In this section we present the results of multiple causal inference methods on data generated from FFs trained two real-world datasets. The first is the Lalonde data, taken from randomised control trial to study the effect of temporary employment program in the US on post intervention income level (LaLonde 1986). The second is observational dataset used to quantify the effect of individuals' (k) eligibility on their accumulated net assets, in the presence of several relevant covariates (Abadie 2003). Both datasets have binary treatment and continuous outcome. Appendix [D.3 can be referred to for a more comprehensive description of the data. In addition, we present diagnostics on the quality of the model fit in Appendix D.3.6 and the loss optimization for both datasets presented in Appendix D.3.7
No Confounding
With Warld Confounding
With Hidden Confounding 0.5
Prop. Score Matching
Linear X Leannr
Linear Learner
Linear Learner
Linear DML
GBT X Learner
Leanne
GBT Learner
Diff. of Moan-
Causal Forest-
1000
2000
3000
-5000
5000
5000
10000
15000
2000
ATE
ATE
ATE
Figure 3: Boxplot of ATE estimates from 10 inference methods, estimated across 50 different samples from FF trained on the Lalonde dataset. The dotted red line represents the customized ATE of samples generated from the trained Frugal Flow.
FFs were fitted to both datasets and used to simulate data with an ATE of 1000. We simulate 50 datasets of size N = 1000 from three different cases each: i) no confounding, ii) with confounding according to the propensity flow inferred the model fitting, and iii) with propensity flow confound- ing and unobserved confounding introduced via Gaussian copula. A variety of causal inference methods (see Appendix D.3.4 for a more detailed description) were fit to the data sets, including difference of means (DoMs) estimate which is unbiased estimator of the treatment effect for randomised data. The inferred ATES across all runs are presented in Figure and Figure
In both cases, all inference methods demonstrate no bias when fitted to unconfounded data. With real-world confounding, most methods estimate the correct ATE in Figure 4 whereas the DoMs shows a substantial bias from the ground truth. In Figure 3 however, all methods infer the correct ATE including DoMs. This is not surprising as the original data was randomised; the propensity flow here appears to simply add more noise to the outcomes. Finally, we note that all causal inference methods show confounding bias the far right hand plots, demonstrating that FF' can simulate data with replicate the effects of hidden confounding.

---

No Confounding
With Real World Confounding
With Hidden Confounding 0.2
TMLE
Prop. Score Matching
Linear Learner
Learsr
Lincar Learner
Linear DML
GBT Learnr
GBT Learner
Dift. of Mean
Causal
Causal BART
6000
5000
5000
5000
5000
10000
ATE
ATE
ATE
Figure Boxplot of ATE estimates from 10 inference methods, estimated across 50 different samples from FF trained on the (k) dataset. The dotted red line represents the customized ATE of samples generated from the trained Frugal Flow.
Conclusions
We introduce Frugal Flows, novel likelihood-bari model that leverages NFs flexibly learn the data-generating process while directly targeting the marginal causal quantities inferred from observational data. Our proposed model addresses the limitations of existing methods by expliclitly parameterising the causal margin. FF's offer significant improvements in generating benchmark datasets for validating causal methods, particularly in scenarios with customizable degrees of un- observed confounding. To our knowledge, FFs are the first generative model that allows for exact parameterisation of causal margins, including binary outcomes from logistic and probit margins.
5.1 Limitations and Future Work
Our experiments validated the empirical effectiveness of FFs, showing that they can infer the correct form of causal margins on confounded data simulations. Despite these promising results, FFs come with certain limitations that need to be addressed in future research. NFs require extensive hyperparameter tuning, which can be computationally intensive and time-consuming. Moreover, see that FFs perform better in inference tasks with larger datasets. Future work could explore alternative ML copula methods and architectures that may be more effective for smaller datasets. Fortunately, this is less problematic for simulation specification of the exact causal margin is left to the user. Additionally, the dequantising mechanism used by FFs implicitly shuffles the order of discrete samples, potentially losing some inherent structure in the data, making FFs less suitable for categorical datasets without implicit ordering.
In summary, Frugal Flows offer a novel approach to causal inference and model validation that combines flexibility with exact parameterisation of causal effects. Future work will refine the inference capabilities and extend the applicability of FFs wider range of data types and sizes.
10

---

Acknowledgements
The authors express their deep gratitude to Stefano Cortinovis and Silvia Sapora for their valuable suggestions regarding the development of the software and experiments. We also thank Christopher Williams for his insightful advice on the framing of the paper. We thank Geoff Nicholls for his suggestions and fruitful discussions the paper and opportunities for future development. Last but certainly not least, special thanks must go to Shahine Bouabid for his invaluable assistance with both the coding aspects of this paper and his recommendations on clarity.
DdVM supported by studentship from the UK's Engineering and Physical Sciences Research Council's Doctoral Training Partnership (EP/T317811/1), LB is supported by Clarendon Scholar- ship.
References
Alberto Abadie. Semiparametric instrumental variable estimation treatment response models. Journal of Econometrics, 2003.
Susan Athey, Guido W. Imbens, Jonas Metzger, and Evan Munro. Using Wasserstein generative adversarial networks for the design of Monte Carlo simulations. Journal of Econometrics, 240(2), 2021.
Ole E. Barndorfi-Nielse. Information and exponential families: families:: in statistical Sons, 2014.
theory. John Wiley &
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake Vanderplas, Skye Wanderman- and Qiao Zhang JAX: composable transformations of Python+NumPy programs, 2018.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), 2018
Pawel Chilinski and Ricardo Silva. Neural likelihoods via cumulative distribution functions. In Conference on Uncertainty Artificial Intelligence. PMLR, 2020.
William G. Cochran and S. Paul Chambers. The planning of observational studies of human populations. Journal of the Royal Statistical Society, Series A, 128(2), 1965.
Claudia Czado. Analyzing dependent data with vine copulas. Lecture Notes in Statistics, Springer, 2019.
Claudia Czado and Thomas Nagler. Vine copula based modeling. Annual Review of Statistics and Its Application, 9, 2022.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In Proceedings of the 5th International Conference on Learning Representations, 2016.
Vincent Dorie, Hugh Chipman, and Robert McCulloch. dbarts: Discrete Bayesian Additive Regres- sion Trees Sampler, 2024. URL httpt://css/R -R- http///et...r...pccceeeaaaarreee R package
version 0.9-28.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural Spline Flows. In Proceedings of the 33rd Conference on Neural Information Processing Systems, NeurIPS, 2019.
Andrew C. Eggers, Guadalupe Tuñón, and Allan Dafoe. Placebo tests for causal inference. American Journal of Political Science, 68(3), 2023.
Robin J. Evans. causl, 2021. URL https://gitthuub.c///////2//2
Robin J. Evans and Vanessa Didelez. Parameterizing and simulating from causal models. Journal of the Royal Statistical Society, Series B, 86(3), 2024.

---

Christian Genest and Johanna Nevslehova. A primer on copulas for count data. ASTIN Bulletin: The Journal of the IAA, 37(2). 2007.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked Autoencoder for Distribution Estimation. In Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, 2015.
W. G. Havercroft and Vanessa Didelez. Simulating from marginal structural models with time- dependent confounding. Statistics in Medicine, (10)), 2012.
Miguel A. Hernán and James M. Robins. Causal Inference: What If. Chapman Hall/CRC, 2020.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural Autoregressive Flows. 2018.
Kosuke Imai, Luke Keele, and Teppei Yamamoto. Identification, Inference and Sensitivity Analysis for Causal Mediation Effects. Statistical Science, 2010.
Adrián Javaloy, Pablo Sánchez-Martin, and Isabel Valera. Causal normalizing flows: from theory to practice. Advances in Neural Information Processing Systems, 36, 2024.
Harry Joe. Dependence modeling with copulas. CRC Press, 2014.
Harry Joe and Dorota Kurowicka. Dependence modeling: Vine copula handbook. World Scientific, 2011.
Sanket Kamthe, Samuel Assefa, and Mare Deisenroth. Copula flows for synthetic data generation. arXiv preprint (tt:::2011.000000 2021.
J.W. Kendall. Hard and soft constraints in linear programming. Omega, 3(6), 1975.
Ruth H. Keogh, Shaun R. Seaman, Jon Michael Gran, and Stijn Vansteelandt. Simulating longitudinal data from marginal structural models using the additive hazard model. Biometrical Journal, 63(7), 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Robert J. LaLonde. Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 1986.
Nunzio A. Letizia and Andrea M. Tonello. Copula density neural estimation. arXiv preprint arXiv:2211.15353 2022.
Chun Kai Ling, Fei Fang, and J. Zico Kolter. Deep archimedean copulas, Advances in Neural Information Processing Systems, 33, 2020.
Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. benchmarking. arXiv preprint arXiv:2011.15000 2020.
RealCause: Realistic causal inference
Anastasios Panagiotelis, Claudia Czado. Harry Joe, and Jakob Stöber. Model selection for discrete regular vine copulas. Computational Statistics & Data Analysis, 106, 2017.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research, 22(57), 2021.
Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating Causal Inference Methods, In Proceedings of the 39th International Conference on Machine Learning, volume 162, 2022.
Judea Pearl. Causality. Cambridge University Press, 2009.
12

---

F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and H. Duchesnay. Scikit-learn; Machine learning in Python. Journal of Machine Learning Research, 12, 2011.
Microsoft Research. EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation. https///tthhb..ooom/cccrssffff/t
2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference on Machine Learning, 2015.
James M. Robins. Marginal structural models. In Proceedings of the American Statistical Association, section on Bayesian Statistical Science, 1998.
James M. Robins and Andrea Rotnitzky. Semiparametric efficiency multivariate regression models with missing data. Journal of the American Statistical Association, 90(429), 1995.
James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association, 90(429), 1995.
Murray Rosenblatt. Remarks on a multivariate transformation. Annals of Mathematical Statistics, 23 (3), 1952.
Ludger Rüschendorf. On the distributional transform, Sklar's theorem, and the empirical copula process. Journal of Statistical Planning and Inference, 2009.
Xu Shi, Wang Miao, and Eric Tchetgen Tchetgen. selective review of negative control methods in epidemiology. Current Epidemiology Reports, 7(4), 2020.
M. Sklar. Fonctions de répartition à dimensions et leurs marges. In Annales de I'ISUP, volume 8, 1959.
Elizabeth A. Staart. Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25(1: 2010.
Elizabeth A. Stuart, Gary King, Kosuke Imai, and Daniel Ho. MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, 42(8), 2011.
Esteban G. Tabak and Cristina V. Turner. A Family of Nonparametric Density Estimation Algorithms.
Communications on Pure and Applied Mathematics, 66(2), 2013.
Mark J. van der Laan and Sherri Rose. Targeted Learning. Springer, 2011.
Daniel Ward. FlowJax: Distributions and Normalizing Flows in Jax, 2024. URL https/ts:///gitm fonm/daniielward27/f1ooooo
Andrew G. Wilson and Zoubin Ghahramani. Copula processes. In Advances in Neural Information
Processing Systems, volume 23, 2010.
Jessica G. Young, Miguel A. Hernán, Sally Picciotto, and James M. Robins. Simulation from structural survival models under complex varying data structures. JSM Proceedings, Section on Statistics in Epidemiology, Denver, CO: American Statistical Association, 2008
Zhi Zeng and Ting Wang. Neural Copula: A unified framework for estimating generic high- dimensional Copula functions. arXiv preprint arXiv:2205.15031 2022.
Aurelius A. Zilko and Dorota Kurowicka. Copula in multivariate mixed discrete-continuous model. Computational Statisties & Data Analysis, 103, 2016.
Paul Zivich. Zepid, 2020. URL http::t/tts/ettt http::::/ggittubb.oo////// http:///ggttttaoocooppaaavii///t
13

---

A The Frugal Parameterisation
The frugal parameterisation proposed by Evans and Didelez (2024) provides method for simulating from a parametric marginal causal model, by starting with this distribution and building the rest of the model around it.
T
Figure 5: A generalised example of static causal treatment model. The past P(T, Z) (black) be freely specified separately from the causal effect (blue). However, the dependency measure between Z and Y (red), & should be parameterised in such way that the margins P Z)) and d(Y))o(T)) are invariant to changes in øb
We specify the notation used in this appendix. Functions labelled Fr(-) are CDFs for the variable i. Apart from this, in general density functions will be labelled with lower case letter, whereas CDFs will named with the upper case (e.g. we contrast the copula density c(u1, u2) with the distribution function C()),
Consider firstly the case of a static treatment model with single outcome Y, single treatment T and an effective pretreatment covariate set Z. Assume that any of these covariates occur prior treatment even they do not causally affect the treatment directly. [vvan and Didelezi (2024) construct frugal models in three parts:
The causal distribution of interest P(Y do(T))
The past P(Z,T)
The intervened variation independent dependency measure o(Y, do(T)).
The three frugal components are variation independent in the sense that they characterise non- overlapping components of the full observational joint. We quote the following definition from Evans and (202))
Definition 1
Take set θ and two functions defined on it d, v. We say that é and 1 are variation independent if (ø x (e) = ((O) x (O)); i.e. the range of the pair of functions together is equal the Cartesian product of the range of them individually.
Variation independence (VT) is a highly desirable property for a parameterization, since it allows different components to be specified entirely separately. This is extremely useful if one is trying to use link function GLM, or to specify independent priors for Bayesian analysis. In addition, VI is important in semiparametric statistics. The definition simply states that the Cartesian product of the images is the same as the image of the joint map. For example, a bivariate gamma-distribution with positive responses, then μl E IR+ and μ12 since
R+ is variation independent parameterization,
(N1 p2) (2) R$ R (e) (2)
However, if we replace μ1 with /1₂ = μ2 - μ1 (for example), then although the range of this parameter is R.
(m) x (b) = (2, y)
0, -
= fn (0) x (6()).
Central to this is the choice of 6* This dependency measure should encode dependencies between Z and Y dooT"), but not provide information about their marginal distributions.
Discrete frugal models can parameterised by conditional odds ratios, while continuous variables typically use copulae. Both allow for variation independent parameterisation of the full joint dis- tribution. The methodology facilitates the creation and simulation of models with parametrically determined causal distributions, enabling fitting using likelihood based techniques, including fully Bayesian methods. Furthermore, this parameterisation covers range of causal quantities, such as the average causal effect and the effect of treatment the treated.
14

---

B
Copula Theory
Copulae present a powerful tool model joint dependencies independent the univariate margins. This aligns well with the requirements of the frugal parameterisation, where dependencies need be varied without altering specified margins (the most critical being the specified causal effect). Understanding the constraints and limitations of copula models ensures that causal models remain accurate and consistent with the intended parameterisation.
B.1 Sklar's Theorem
Sklar's theorem Sklar 1959 Czado 2019) is the fundamental foundation for copula modelling, as provides a bridge between multivariate joint distributions and their univariate margins. It allows one to separate the marginal behaviour of each variable from their joint dependence structure, with the latter being represented by the copula itself.
Theorem For d-variate distribution function Fita FFFi,.... Fa), with jth univariate margin Fj, the copula associated with F is distribution function C: [0, 1] - [0, 1] with uniform margins on (0, 1) that satisfies
Fi.(
E R
1. If F is continuous d-variate distribution function with univariate margins F1,..., Fa and rank functions FT ,...,Fr. Fr 'then
C(u)
(wa)) [0,
2. If Fi.d is d-variate distribution function of discrete random variables (more generally, partly continuous and partly discrete). then the copula is unique only the
(oange(Fi) x ... (tange(Fa),
The copula distribution is associated with density c(-)
fly) c(Fi F(f((u))....FF
where Ji(-) is the univariate density function of the 1th variable.
Note that Sklar's theorem explicitly refers to the univariate marginals of the variable set {Yi,..., Ya) to convert between the joint of univariate margins C(u) and the original distribu- tion Fly). For absolutely continuous random variables, the copula function C' is unique. This uniqueness longer holds for discrete variables, but this does severely limit the applicability of copulae to simulating from discrete distributions. The non-uniqueness does play a more problematic role in copula inference, however (Genest and Nevslehoval 2007).
An equivalent definition (from an analytical purview) is C:: |0, 1]d + [0, 1] is a d'dimensiioal copula if it has the following properties:
1. C(ui...., O,..,, Ud) =
2. C(I...., u4 1,....111.. 1) = d4-
3. C is d-non-decreasing.
Definition A copula C is d-non-decreasing if. for any hyperrectangle I[ [u,, vil S [0, 1]4. the C-volume of H is non-negative.
| C(u) du 2 0
B.2 Copulae for Discrete Variables
Accurately modelling the univariate marginal CDFs of pretreatment covariates is crucial step in training Frugal Flows, particularly when the dataset includes discrete variables. For continuous covariates, the mapping between observations and ranks is unique, allowing for straightforward
15

---

estimation of the marginal distribution. However, with discrete covariates, this mapping becomes one-to-many. as the same observation can be transformed from different ranks. This con-unqquesess introduces significant challenges when modelling the joint distribution via copulae, as the joint set of ranks for discrete covariates lacks a unique representation. As a result, estimating the dependency structure between variables becomes more complex and less reliable.
To extend Frugal Flows accommodate mixed data types, it is essential generate empirical ranks for discrete covariates in a way that allows the model to capture their dependencies, Our goal is obtain valid rank samples that can be used to train a Frugal Flow without introducing distortions in the copula structure. While this issue has been widely explored in the copula literature for parametric models, there remains a gap effectively addressing within more flexible, non- parametric frameworks. In this work, we implement a generalised distributional transform for discrete covariates (presented in Appendix B.2), which allows Frugal Flows to be trained effectively, maintaining the flexibility of the model while accurately capturing the relationships between variables.
B.2.1 Challenges and Motivation
In addition to the above, copulae encode degree of ordering in the joint probability integral transforms are inherently ranked, and hence should only be used for variables that have an inherent ordering of their own (e.g. count or ordinal data models). While approaches model discrete variables exist in parametric copulae models (Zilko and Kurowicka 2016 Panagiotelis et al. (017)) more flexible non- non-poramee con-parametric copulae struggle to capture the dependencies of empirical copulae. Similar to Kamthe et al. [202]) we use the approach suggested by Rüschendorf (2009). An outline of this method is presented Appendix B.2.2 However, unlike Kamthe et al. we use the empirical CDF inferred from the discrete data as opposed to modelling the CDF with marginal flow.
B.2.2 Empirical Copula Processes for Discrete Variables
In order to deal with discrete variables, we use a similar approach as taken by Kamthe al. (202T). who quote the generalised distributional transform of a random variable found originally proposed by Rüschendorf 2009). We quote the main result from Rüschendorf (2009) below.
On probability space (Q,A, P) let X be real random variable with distribution function F and let ~ (0, 1) be uniformly distributed on (0, 1) and independent of X. The modified distribution functtion F(I, X) is defined by
Theorem 2
A( P(X x) + XP = xx).
We define the (generalised) distributional transform of X by
U= VX.
An equivalent representation of the distributional transform is
U = =(X-)++(((X)) F(X-))
Rüschendorf (2009) makes key remark about the generalised transform's 's lack of uniqueness for discrete variables. Such a dequantisation step may introduce artificial local dependence which may lead to incorrect flow being inferred, and therefore hinder the inference of the causal margin.
C Frugal Flow Implementation Details
The FF software used for this paper can be found the GitHub repository https://github.. https://giiuub..om/llaaara www..///aaaaaaaattttuaaaaaaaaaaa
FF software builds upon FlowJax (Ward 2024). a Python package implementing normalising flows in JAX (Bradbury et al. 2018). JAX is an open-souree source numerical computing library that extends NumPy functionality with automatic differentiation and GPU/TPU support, designed for nigh-performance machine learning research.
Frugal Flow architecture.
The Frugal Flow component builds a flow of the form the bottom part of Figure 2 allows us to implement Fylz with either (i) a customised CDF conditioned T,, of a known parametric family;
16

---

(ii) a univariate NSF on the [-1, 1] interval modified to allow a location translation parameter that represents the ATE for T. where the imput is mapped from the real line via tanh transform; or finally (iii) a univariate NSF on the [-1,1] interval that is not conditional on T, where the input is mapped from [0, 1] via an affine transform. As for known parametric families, only the Gaussian CDF is currently implemented, but the architecture allows us to define any different parametric class provided that it constitutes a diffeomorphism. As for the univariate NSF in (ii), it does not explicitly learn an ATE in the training phase, but can be used for simulation of e.g. binary outcomes by applying subsequent logistic transformation to its outcome.
The multivariate NSF element is composition of multiple modified NSF subflows. In each subflow the first transform is fixed to be an identity. while the other dimensions are transformed with a monotone rational quadratic spline whose knot parameters are produced by masked multilayer perceptrons (MLPs) implemented as in Germain et al. (2015), conditional on the first dimension. In order to increase expressivity, dimension permutation is usually applied between the different subflows in the NSF composition. We still allow this permutation but excluding the first dimension, that is fixed to be at the top in each subllow. The NSF acts on the on the [-1,1] interval and is mapped from and back to the quantile space with affine transforms.
Tunable hyperparameters to the Frugal Flow component are the number of subflows of the multivariate NSF, the width and depth of the MLPs and the number of spline knots, together with the specific hyperparameterrs of the chosen FyJr-
Marginal flows architecture for continuous variables.
Each marginal flow for the continuous covariates maps each variable from the real line to the [-1, 1]
interval with tanh transform, then applies univariate NSF on the [-1,1] interval, and maps back to the standard uniform base distribution via an affine transform. Tunable hyperparameters are the number of subflows of the NSF, the width and depth of the MLPs and the number of spline knots.
Marginal transform architecture for discrete variables.
To map a discrete Z to the ranks Vz, we compute its empirical CDF and then apply the procedure outlined in Appendix B.2.2 We use the inverse of the same empirical CDF to map ranks back to the Z for sampling.
Propensity score model architecture.
To map a discrete T to the ranks Vr, we compute its empirical CDF and then apply the procedure outlined in Appendix B.2.2 A univariate NSF flow with a uniform base distribution is then applied to learn the copula CDF of on the [0, 1] support, conditioned on Z. The Z conditioning is obtained by adding Z as an (unmasked) input to the MLP that produces the knot parameters for the rational quadratic spline. This is standard in NF literature. Tunable hyperparameters are the number of subflows of the NSF, the width and depth of the MLPs and the number of spline knots.
The propensity score model can be inverted to generate T samples conditioned on given Z. Uniform samples are pushed through the trained univariate propensity score flow to obtain ranks, that are then mapped to the discrete space via the inverse of the empirical CDF of T.
Training the Frugal Flow and the propensity score flow.
In order to train a FF, one must fit the marginal flows first. The marginal flows are trained (for continuous variables) via maximising the likeliiiddood with stochastic gradient descent, and/or the discrete Z are mapped to the rank space via the procedure in Appendix B.2.2 Next, the FF is trained via maximum likelihood estimation (MLE), taking as input the outcome Y together with the ranks Vz. and conditioning the flow for the causal margin on treatment T where required by the chosen method. For MIE optimisation, we take advantage of JAX automatic differentiation capabilities and use the Adam optimiser (Kingma and Bal 2015), whose hyperparameters can also be tuned. If required, the propensity flow is likewise trained on VT conditioning on Z via MLE with an Adam optimiser.
Simulating benchmarks.
One can use a trained FF for simulation of causal benchmarks. The general data simulation pipeline is:
17

---

1. Generate sample of Unz, Vy)) from a bivariate Gaussian copula, parameterised by correlation p. which quantifies the degree of unobserved confounding one wishes encode in the benchmark. If no unobserved confounding is desired, set p = 0.
2. Generate sample of D independent uniforms, Uz
3. Push the sample do(T) Uz) through the trained FF and save the resulting correlated Vz samples associated with dy])
4. Generate Z from uniform samples via inverting the univariate marginal flows (continuous variables) and/or using the learnt inverse empirical CDF (discrete variables)
5. Generate 7 as a function Z by pushing Uriz through the inverse of the trained propensity score flow and mapping ranks Vr back to the discrete space via the learnt inverse empirical CDF
6. Push dy)) through the desired causal margin transform to obtain outcome samples conditioned Currently, the package supports:
(a) Sampling from an inverse CDF provided by the user, taking Vy as input and con- ditioning the given univariate T. Currently Gaussian inverse CDF is implemented, where the coefficient on 7 can be chosen to impose the desired ATE. The user is free define different inverse CDFs.
(b) Sampling a binary outcome with probabilities produced from logistic function taking Vy]do(T) as input and conditioning the given univariate T.. The coefficient T can be chosen to impose the desired odds ratio.
(c) Sampling from the univariate NSF learnt during the FF training, but with user-defined location translation parameter representing the ATE and conditioning on the given treatment T. This method exploits the flexible margin distribution learnt for 7 0 during FF training, but allows to choose a different ATE for the location-translation effect produced by T = 1.
D Experimental Details
All experiments were run MacBook (16-inch, 2021) with an MI Max chip and 32GB memory using the CPU.
D.1 Simulated Data Experiments
The simulated data generated for the inference experiments was generated using the causl package written in R, which was called in Python via the rpy2 package (Evans 2021 Evans and Didelez! 2024).
The covariates were either selected to be binary (marginally distributed according to Bernoulli(p 0.5) or continuous (marginally distributed according to Gamma = 1, Φ = 1). The marginal causal effect was chosen to be a linear Gaussian; Y do(7)) (T,
The underlying data generating process uses multivariate Gaussian copula to generate dependencies between the marginal covariates and the causal effect. The Spearman correlation matrix used to generate the data for models Mj and M2 is
1.0
0.5
0.3
0.1
0.8
0.5
1.0
0.4
0.1
0.8
R
0.3
0.4
1.0
0.1
0.8
0.1
0.1
0.1
1.0
0.8
0.8
0.8
0.8
0.8
1.0
18

---

and the correlation matrix used to generate data for model M3 is
1.0
0.3
0.4
0.5
0.1
0.2
0.7
0.5
0.4
0.5
0.5
0.3
1.0
0.3
0.6
0.3
0.4
0.4
0.6
0.3
0.2
0.5
0.4
0.3
1.0
0.5
0.2
0.1
0.1
0.0
0.4
0.4
0.5
0.5
0.6
0.5
1.0
0.2
0.2
0.5
0.5
0.3
0.4
0.5
0.1
0.3
0.2
0.2
1.0
0.1
0.5
0.6
0.2
0.4
0.5
R10
0.2
0.4
0.1
0.2
0.1
1.0
0.0
0.4
0.2
0.5
0.5
0.7
0.4
0.1
0.5
0.5
0.0
1.0
0.4
0.4
0.4
0.5
0.5
0.6
0.0
0.5
0.6
0.4
0.4
1.0
0.4
0.4
0.5
0.4
0.3
0.4
0.3
0.2
0.2
0.4
0.4
1.0
0.4
0.5
0.5
0.2
0.4
0.4
0.2
0.5
0.4
0.4
0.4
1.0
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
1.0
where the later rows/columns are indexed by the causal effect ranks, VyJao(T)> and the earlier rows/columns correspond the Spearman correlation matrix between the ranks of the covariates,
Vz.
The propensity model for M1 and M2 sigmoid
P (t 2) Sigmoid(-O.3 + 0.12; + 0.22z + 0.52 - 0.2zg + 24),
and M3 was parameterised by
Ptz 2) Sigmoid(-Oi) - 0. + 0.22 0.523
0.2za + 25
0.33z 0.427 0.7 0.1 0.9
D.1.1 Hyperparameters and Runtime
The hyperparameters and runtime for the simulated inference datasets are presented in Table 4 In these cases, the models were trained with a default set of hyperparameters.
Table 2: Runtime and hyperparameters for fitting 25 different runs each model, with a datasize of 15,000.
Model
Total Runtime
RQS Knots
Flow Layers
Learning Rate
NN Width
NN Depth
Mj
45,4 mins
5e-3
50
M2
44.1 mins
8
5e-3
50
M3
66.3 mins
5e-3
50
D.2
Additional Results
D.2.1
Logistic Benchmark Simulation
To demonstrate the FF ability to generate discrete outcomes from known marginal logistic models. we ran the following experiment. First, data of the following form
Z ~ N~ = 0, = 2)
Vz, Vylao(T) Caussian (p 0.8)
Y do(T") ~ 2X,
was generated from the causl package. It was then fitted with FF using the same hyperparameters as in Appendix D.I.I We then generated samples from custom logistic CDF such that
Y do(T) Bernoullip S2g 1))-
A dataset size of N 1000 was generated from the model, and fit using two models. The first is
a Bernoulli outcome regression model, and the second uses inverse propensity weighting (IPW) to estimate the logistic parameters instead. The outcome regression (OR) estimates are biased indicating a clear confounding effect, whereas the IPW estimates comfortably contain the true parameters within their 20 bounds. These results presented in Table 3
19

---

Table 3: Mean and 2-sigma confidence interval of the logistic parameter estimates. The "ground- truth" estimates are contrasted alongside the IPW estimates and OR methods, the latter of which demonstrates clear data confounding.
Model
Parameter
Parameter 2
Ground Truth
+2 1.6 + 0.48 3.16 0.34
Robust IPW
-0.88 + 0.38
Outcome Regression
-1.59 + 0.24
Vw W, .o(()) V w
II()) F Friwzagor, I(())
Vw VrIWadoTT) V w
Uvad Ua+i U4+2:(D42)
c(ow) d(yyW.do(T)) =1 <@uw (Y]W,do(T): vw)
NSF
Figure 6: Structure for learning Frugal Flow with a heterogeneous treatment effect. This enforces that ow do()) and that the copula density of ow can be inferred via the factor c(tw do(r))(r))
D.2.2 Heterogeneous Treatment Effects
In the main paper, we comment on the ability of the model to simulate data from distributions where the causal effect is marginal quantity taken over the entire covariate set
PY]do(T) (y t)
dz PriZAo(TT(U (v 2,t) pz(z).
However, one may wish to simulate from more complex heterogeneous treatment effect models. Consider a stationary treatment with pretreament covariate set Z [W,W} where W C Z and |Z| = D, W = d, and [W|- = D We proceed considering the case where 0 < d < D.
Interest may lie in the causal treatment margin conditional on the subset of variables W:
//x dw PriZ,d(T) (3) w, w. Pw w
W.oerr (y w.
W
w)
which we call the conditional treatment margin. We infer this effect by constructing a Frugal Flow which ensures that the pretreatment covariate joint pz(-) is correctly inferred and that the conditional treatment margin ranks are uniformly distributed. A modified version of the Frugal Flow illustrated Figure used to account for this change. The choice of Fy do()) (+) can left to the user for inferring the Frugal Flow. For simulating benchmarks, the conditional treatment margin can be free set to any valid CDF, for example:
Y W, do(T)) NNu g(W, T), 1)
where : Rd T
IR can be chosen encode arbitrary heterogeneity in treatment effects.
D.3
Real-World Data Benchmarks
D.3.1 Lalonde Temporary Employment Program
The Jobs dataset by LaLonde a benchmark in causal inference studies, where job training serves as the treatment and the outcomes are post-training income and employment status. Originating from National Support Work Demonstration (NSW), this randomized controlled trial (RCT) examines the impact of temporary employment program (i.e. the treatment) in the US on participants' income levels Londe 1986). Due its design the treatment assignment is random, eliminating unobserved confounding, The measured features, all recorded in 1975, are:
20

---

an individual's 's age in years;
the number of years an individual spent in education;
whether an individual is black;
whether an individual is hispanic;
an individual's marital status (1 if married, otherwise);
whether an individual has high school degree.
The outcome is the individual's 's real earnings in 1978.
D.3.2 401(k) Eligibility
The (k) savings plans dataset has been analysed in variety of studies. We use investigate the impact of eligibility to enroll on the increase in net assets.
The dataset includes 9,915 individuals with the following variables measured:
age of the individual in years;
income of the individual;
years of education the individual has completed;
size of the individual's family;
indicator variable of whether the individual is married (1 for married, O otherwise);
indicator of whether there are two earners in the household (1 if two earners, O otherwise);
membership of a defined benefit pension scheme (1 if true, O otherwise);
eligibility for Individual Retirement Allowance (IRA) (1 if true, O otherwise);
homeownership status of the individual (1 if true, 0 otherwise).
D.3.3 Causal methods used for benchmarking FF inference
Section 4.1 reports FF ATE inference performance in simulated setting comparing to number of methods.
Outcome regression (OR) ATE results are obtained by regressing Y on the treatment T together with the covariates in the different scenarios and reporting the coefficient and confidence interval on T.
Propensity score matching is implemented using the R package MatchIt for estimating the ATE (Stuart al. 201T).
Causal normalising flow (CNF) (Javaloy al., 2024 trained using the causal abductive model with one layer, that the paper reports to be the best rest-performing model variation (Paragraph 1). We use the hyperparameter settings recommended in package and do not perform hyperparameter tuning. For the flow architecture, we use neural spline flows, as the paper reports Appendix D.3 that they yield better performance than simple masked autoregressive flows, plus this resembles our architecture choice in Frugal Flows. We do not add uniform independent noise the binary inputs as recommended in paragraph 3.1 as we find this worsens the ATE estimates for the model.
D.3.4 Causal methods used for validating FF as a benchmark
Similar to Parikh et al. (202)) we use a variety of different causal inference methods to validate the generated benchmark samples our model.
Propensity score matching is implemented using the R package MatchIt for estimating the ATE (Stuart et al. 201T).
Causal BART is implemented via the R package dbarts using default hyperparame- ters (Dorie et al. 2024).
21

---

The double machine learning methods are implemeneted using the Python package EconML (Research) 2019) using the scikit-learn' machine learning API for the same. For GBT DML. we used the method with 100 trees, and the linear DML used ridge regression (Pedregosal et al. 2011).
EconML also used for implementing the S-, T- and X-learners also using scikit-learn's MI. API to for gradient boosting trees and ridge regression.
TMLE was implemented using the zepid Python package Ziiiich 2020).
D.3.5 Hyperparameters and Runtime
For both datasets, a random hyperparameter search was conducted by choosing the hyperparameter which minimised the validation loss, given train/test data split of 9/1. The total number of neural network of the hyperparameter tuned Frugal Flow for the Lalonde and c401(k) datasets are 485243 and 106969 respectively.
Table Runtime and hyperparameters for fitting both Frugal and Propensity Flow to the Lalonde and e401(k) data.
Benchmark
Runtime
Knots
Flow Layers
Learning Rate
NN Width
NN Depth
Lalonde
1.2 mins
9
6.3e-3
50
10
c401(k)
4.9 mins
2.6e-3
34
3
D.3.6 Realism of Datasets
We conducted additional validation of the proposed Frugal Flows method to enhance its robustmess in comparison to current state-of-the the-art methods such as Credence (Parikh al. 2022) and RealCause. Credence allows for the exact specification of conditional average treatment effect (CATE) in genera- tive samples, whereas RealCause adjusts the causal effect post hoc, preventing from realistically modelling null hypothesis where the average treatment effect (ATE) is zero. Additionally, Frugal Flows and Credence both model unobserved confounding. feature absent RealCause, making Credence the more suitable method for direct comparison.
We ran the benchmarking simulations Section 4.2 with Credence, using its default parameters, evaluating model performance on Lalonde and the e401(k) dataset. We compared the correlation matrices of the pretreatment covariates and outcomes for the original data, Frugal Flows- genn--geeerated data, and two peddence-generatee datasets with different causal constraints. The results, illustrated in Figure 7 and Figure X show that Frugal Flows produce samples closely resembling the original data, especially for the larger e401 (k) dataset.
While Credence also performs well the ek) dataset, altering its causal constraints significantly affects the covariate dependencies. In contrast, Frugal Flows optimise the model once, allowing for causal constraint modifications without altering the covariate joint distribution or propensity score.
Dieened
Tragel dalande
COEDONOE bus- 2000 Debattllandd))
takerde
-05
100gp0
1319
1000
Correlation matrices across covariates and the outcome (re78), comparing the second moments of distributions for the Lalonde observed real data, well synthetic data generated by a trained Frugal Flow (2 column) and Credence (3rd column) models. The comparison further extended to models with default settings and those with modified bias rigidity 44 column).
Figure Lalonde:
22

---

Obeanee adoios Data
Trigal Floes 14401900
GREDENCEE has + 5000 Dehsul od0id
GOERENGE bas 1000 jat21pc
1.0
Downr
Figure 8: ek)] Correlation matrices across covariates and the outcome (net_ comparing the second moments of distributions for the e401(k) observed real data and synthetic data generated by trained Frugal Flow (2nd column) and Credence (3th column) models. The comparison is further extended to models with default settings and those with modified bias rigidity (4th column).
D.3.7 Loss Optimisation
In training, we perform train-val split and use "patience" criterion the validation loss criterion to stop the training. Namely, we monitor validation loss and stop training the validation loss does not improve for specified number of epochs (we set the patience value to 100). This aims to prevent overfitting and saves computational resources by not continuing training unnecessarily. standard in machine learning model training and was implemented in the FlowJax package that use as code-base to build the Frugal Flow package from.
The observational likelihood losses during model training for both real-world datasets are presented
in Figures and 10
Lalonde
e401(k)
2.0
0.5
Training Validation
Volidation
1.5
0.0
1.0
Loss
Loss
0.0
-1.0
-0.5
-1.
11.0
-1.5
100
lteration
lteration
Figure Training and validation losses when fitting a Frugal Flow to the Lalonde dataset using optimal hyperparameters and "pa- tience" setting of 200 iterations for illustrative purposes.
Figure 10: Training and validation losses when fitting a Frugal Flow to the (401(k) dataset using optimal hyperparameters and "patience" setting of 200 iterations for illus- trative purposes.
23

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract outlines the a key problem in generating causal benchmarks for method validation, in particular that it is hard to train and sample from generative models which explicitly target the marginal causal effect. Our method (Frugal Flows) builds on statistical work by Evans and Didelez (2024), and we develop a normalising flow framework which allows us to exactly specity the marginal causal effect and simulate from it exactly. We show that it does indeed target the causal effect by fitting FFs to simulated data with known causal margin, and demonstrates that it infers the correct parameters despite the data being demonstrably confounded. Additionally, fitting popular causal inference models to synthetic data generated from trained Frugal Flows verifies that the samples are follow the desired causal margin when the data is confounded and unconfounded.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Section 5.1 elaborating in particular on the drawbacks of Frugal Flows for inference purposes. In particular, while we believe Frugal Flows are powerful generative models for flexibly simulating causal benchmarks, their inference capabilities are fairly limited, and require large datasets to learn the correct causal margin.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not derive any theoretical results, but quotes the relevant theorems if necessary, and the papers from which they are derived.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: Information about the datasets used can be found in Appendix D.3 Appendix elaborates on the model infrastructure. The code required to reproduce the results is attached to this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
25

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the codebase as ZIP attachment (and also as a aanonymised link for implementing our methods and experiments along with the required external packages. The process for generating the data is recorded in the appendix of the paper, and the code for doing so is also provided. We provide the code for processing the real-world data for the benchmarking experiments, and ensure that it is documented and referenced in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
cublic/guidess httaai//guiias//oeeuuuniiissss CoodeSumiissionooo Peiiooicylc) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 ctcips.cc/publlc. guwdes/ httppp/iii//.iiiieeeeeeteeaaiiaa Cooe/oooeeeeeeeeeeeeeeee ssiiiiiiccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the hyperparameter set and the tuning process for both the simulated and real-world examples in Appendix D.I.T and Appendix DD3.5 respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
26

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results presented in tables record the mean and 2a error with a description on how these were estimated (either multiple runs, standard deviations on estimated parameters inferred from robust linear regression). Other results are presented using boxplots which present the relevant quantile ranges, median, and range. The methods for generating these uncertainty bounds (multiple runs) is documented in the main body of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We name the compuational resources and runtime used Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///.aaaaeeccppppp////aaaa
Answer: [Yes]
27

---

Justification:
We read and verified that our research conforms to the NEURIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss how method be used to flexibly generate causal bench- marks, which we hope will encourage the further methodological development in the causal inference space.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that the models we develop have a low risk misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
28

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For the relevant licenced assets and packages, we cite the paper corresponding to the package if possible, and if not, reference the URL and the year the package has been developed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswithhaaeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code with README which points the user to the relevant notebooks/scripts used generate particular experiment in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human or animal subjects were used in the development of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "MNRE6lf7KL", "reviewer": "Reviewer_PATR", "summary": "This work proposes to leverage existing neural density estimators (specifically, normalizing flows) to exploit a newly-proposed \"frugal parametrization\" that can capture the causal marginal distribution of an underlying causal model. Under this parametrization, the authors show how to specify and train each component of the model, and thus train the proposed Frugal-Flows to match the observational distribution as closely as possible, while being able to tune the marginal causal effect present in the generated data.\n\nThis way, frugal flows can be used to generate synthetic causal benchmarks that closely represent the _observational_ data while having more difficult-to-estimate causal effects, putting existing approaches to the test.", "questions": "- **Q1.** I am not sure that I understand what does the dotted red line represent in the boxplots.\n- **Q2.** Doesn't the statement in lines 291-294 directly contradict what you say later in lines 296-297?\n- **Q3.** Why is Figure 1 placed there?", "limitations": "I think limitations are properly discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **S1.** The proposed frugal flows provide a way of generating new datasets that can be challenging from a causal-inference point of view, which I believe _important_ to test new and existing methods.\n- **S2.** The construction of the proposed architecture is quite rich in details.\n- **S3.** I find the frugal parametrization conceptually quite interesting.\n- **S4.** The authors motivate different scenarios for frugal flows in Sec. 3.2, as well as empirically show positive results on some synthetic and real-world scenarios.", "weaknesses": "- **W1.** I find the frugal parametrization to be extremely under-explained, relying too much on the reader having full knowledge of the referenced work. Similarly, there is little to no explanation/intuition on why the frugal parametrization would properly capture the marginal causal distributions.\n- **W2.** The lack of explanations also applies to other concepts, e.g., \"conditional ignorability\" (line 39) \"variation independence\" (line 82, and I know the definition is later in App. A), or why copula-based flows would target conditional causal effects instead of marginal causal ones (line 182). (similar with lines 221 and 229)\n- **W3.** There are no mention to related works that propose similar ways of constructing causal benchmarks. From a 1-min search in google scholar, I already found some likely relevant works: [Work 1](https://arxiv.org/abs/2406.08311), [Work 2](https://arxiv.org/abs/2011.15007).\n- **W4.** I find the experiments a bit underwhelming, specially those from Section 4.1. The authors should at least show how is the fitting of the observational likelihood, and if they want to show the capabilities of frugal flows for causal inference (and not only causal-benchmark generation), they should compare with other methods like [Causal Normalizing Flows](https://arxiv.org/abs/2306.05415).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s feedback and suggestions for our submission. Below are our responses to the noted weaknesses and questions:\n# Weakness 1\nWe acknowledge that the frugal parameterization was briefly introduced. Due to page constraints, we provided a brief overview in the main text and a more detailed discussion in the appendix, drawing largely from Evans and Didilez (2024). In response to the reviewer’s comments, we will reorganize section 2.2 to better explain frugal models, the relevance of copulas, and their focus on the causal margin. We will also enhance the appendix with a comprehensive introduction for those interested in technical details.\nRegarding why the frugal parameterization captures the marginal causal distribution when the dependency measure is parameterised by a multivariate copula, consider the copula for the distribution of $\\mathbf{Z}$ and $Y$ conditional on $X$: \n$$c(F_{Y|do(X)}, F_{Z_1|X},\\dots, F_{Z_{D}|X}).$$\nFor an intervened distribution, all pretreatment covariates $\\mathbf{Z}$ are marginally independent of $X$, simplifying the copula to\n$$ c(F_{Y|do(X)}, F_{Z_1},\\dots, F_{Z_{D}}),$$\nand so the intervened joint density becomes\n$$p(\\mathbf{Z},~Y \\mid do(X)) = p_{Y|do(X)} \\cdot \\prod_{d=1}^{D} p_{Z_d} \\cdot c(F_{Y|do(X)}, F_{Z_1},\\dots, F_{Z_{D}}),$$\nwhere $p_{Y|do(X)}$ is the marginal causal effect of $X$ on $Y$. The final density required to parameterise the observational distribution is the propensity score $p_{X|\\mathbf{Z}}$, which does not affect the aforementioned marginal densities in the observational model (see Chapter 10 in \"Information and Exponential Families\"(Barndorff Nielsen, 1978)). We hope this clarifies why frugal parameterizations target marginal causal quantities and will add these clarifications to the manuscript.\n# Weakness 2\nWe agree that improved clarity in definitions and terminology would enhance readability and are happy to make these additions to the manuscript. We use the definition of conditional ignorability (equivalent to conditional exchangeability) stated in “Causality: Models, Reasoning and Inference” (Pearl, 2009), where the marginal potential outcomes are independent of the treatment, conditional on the observed covariates.\n\nRegarding why copula-based flows target the conditional effect, copula flows require all marginal flows to be trained independently of the copula likelihood. This is because copula flows infer a joint density within a unit hypercube and do not constrain marginal densities to be uniformly distributed. Thus, empirically uniform data must be provided to generate samples from a copula. Since one of the marginal densities being trained is a conditional quantity ($\\mathcal{F}_{Y|X}$), training this flow independently targets the conditional of $Y$ on $X$. Frugal flows target the marginal causal effect as they are trained jointly with the copula flow, with the causal effect density constrained to be uniformly distributed, as outlined in Figure 2 of the original submission.\n# Weakness 3\nOur aim was to contrast our contribution to generating synthetic causal benchmarks against other methods in the literature, as discussed in section 2.6. We reference three key papers on line 200, including the second work the reviewer mentioned. We appreciate the reviewer sharing Work 1, which we were not aware of and was submitted to arXiv after our paper submission! However, Work 1 focuses more on validating structural causal benchmarks, whereas Work 2 (and the other two papers we reference) focus on generating observational data with customizable causal effects.\n\nCredence allows exact specification of the CATE in generative samples. In contrast, RealCause only allows users to scale the causal effect post-hoc, which hinders its ability to model the null hypothesis of the ATE=0, because every outcome is scaled to zero. Additionally, frugal flows’ ability to model unobserved confounding is shared with Credence but not with RealCause. Therefore, we chose Credence as the more appropriate method for comparison.\n\nWe assessed the realism of samples from frugal flows for the Lalonde and e401k datasets, comparing them to those from Credence. These results are presented in Figures 1 & 2 (in the attached material) showing the correlation matrices of covariates and outcomes for frugal flow and Credence samples against the original data. We find that Credence's fitting process is sensitive to the causal effect one wishes to constrain, requiring hyperparameter optimization for every setting, while frugal flows need optimization only once, allowing for sample generation from different causal models without re-training.\n# Weakness 4\nFollowing the reviewer’s recommendations, we used the CNF package to estimate ATEs in our synthetic data experiments with different causal effects, comparing the results against frugal flows and a causal statistical matching algorithm. These results are presented in Table 1 in the attached document. CNFs showed higher variance in ATE estimates than frugal flows. In addition, we also provided figures showing the observational likelihood loss during model training for both real-world datasets in Figures 4 & 5.\n# Additional Questions:\n* The dotted red line represents the customized ATE parameter of the frugal flow, indicating that causal inference algorithms infer the specified ATE. We will clarify this in the updated version of the paper.\n* Lines 291-294 indicate that frugal flows are not recommended for inferring ATEs in small to medium-sized datasets. Figure 3 in the attached experimental results showing that frugal flows struggle to converge to the true ATE of synthetic data when $N < 10,000$. However, for benchmarking, frugal flows can customize the causal properties of generated data and generate realistic samples with specified ATEs even for smaller datasets like Lalonde.\n* We agree that Figure 1 should be placed closer to section 2.2 and will make this change in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EkGmnslyYW", "reviewer": "Reviewer_SZi8", "summary": "This paper proposed a generative model called Frugal Flows making use of copula flows to infer about marginal causal effects by simulating the data generating process.", "questions": "I have two question --\n\n1. A highly related (and I suspect might be viewed as a \"dual\" appproach to your Frugal Flow) is the statistical matching (e.g., bipartite matching) to estimate the average treatment effect. It would be very informative to compare this as one of the baselines in your benchmarking and validation, as this also shed lights on how these two different schools of causal inference may (or may not) converge on ATE estimation.\n\n2. I think it is good to use real data for benchmarking/validation (but perhaps benchmarking is a bit strong here since only two datasets were used), but usually it is unclear how to interpret the results since the ground truth is unknown. Can you design and run some controlled synthetic experiment to verify the model?", "limitations": "See questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of inferencing marginal causal effects is an interesting and important problem\n- The idea of using generative models to estimate the marginal effects in the paper is interesting", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments and suggestions for our work. Indeed, we agree that frugal flows are an interesting addition to inference algorithms for estimating marginal causal densities in large datasets using Normalizing Flow models.\n\nIn addition, we believe a key contribution of frugal flows lies in their ability to enable users to learn representations of realistic datasets and precisely customize causal features such as the ATE, degree of unobserved confounding, propensity score, and simulating from discrete outcomes. When referring to model benchmarking, we propose that frugal flows can generate realistic datasets with customizable causal features for benchmarking purposes; being able to simulate realistic data with a ground truth is clearly much more useful for benchmarking novel causal methods, than using real datasets with unknown underlying causal effects (see Section 2.6 in the original submission).\n\nIn response to the reviewers questions, we sincerely appreciate your suggestions for improving the scope of our inference experiments. We have made a couple of modifications and additional experiments based on your suggestions:\n1) We agree that Section 4.1 would be more compelling if frugal flows were contrasted with other causal inference methods, rather than linear regression, which was initially included to demonstrate the complexity of the confounding. Consequently, we have added results from a statistical matching algorithm as suggested. Additionally, we present ATE estimates calculated using an alternative method, **Causal Normalizing Flows** (Javaloy et al, 2023), recommended by other reviewers. The matching algorithms yield results consistent with frugal flows, whereas the CNFs produce ATE estimates with higher variance, and in some cases a bias, compared to both statistical matching and frugal flows.\n2) In response to the desire for more comprehensive synthetic experiments, we demonstrate that with sufficiently large data sizes, they can accurately identify the true ATE across a range of simulated data. These results are presented in Table 1 (in the document attached to the rebuttal) for models with an ATE of 1. We have also rerun similar experiments with a true underlying ATE of 5, incorporating each of the algorithms suggested by the reviewers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dRPnh1a0br", "reviewer": "Reviewer_Cwuj", "summary": "The paper introduces a generative modeling approach called Frugal Flows, designed to learn the data generation process with an explicit parametrization for the marginal causal effect of treatment on outcomes. Inspired by the frugal parametrization of marginal structural models, this approach models the marginal intervention distribution $p(Y|do(X))$ directly, rather than the joint distribution $p(Y|Z, do(X))$. This helps in preserving any constraints on the average treatment effect while flexibly modeling the data generation process. Frugal Flows employs copula flows to parameterize the model, accommodating constraints on the average causal effect and handling unobserved confounding during data generation. The authors validate the proposed method through experiments on both synthetic and real-world datasets, demonstrating its ability to generate realistic datasets with user-specified constraints.", "questions": "* A suggestion for the notation is that the authors could use $T$ instead of $X$ to denote the treatment variables in the paper. This way the notation would be less confusing, as $X$ represents a general random variable in Section 2.5 and Section 2.6\n\n* Maybe there is a typo in Figure 2? The top row should be $\\mathcal{F_{Z_i}}^{-1}(.)$   as we transforming the covariate $\\{ Z_i \\}$ to the correlated uniform variables $\\{ V_{i} \\}$\n\n* I don't understand the Section 3.1.1 on copula flow for $X$ on $Z$. Why don't we directly model $p(X|Z)$ using a normalizing flow and why do we need to parametrize using copula flow as $p(X|Z)= p(X).c(X|Z)$?", "limitations": "Yes, the authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper's approach to validating causal models using simulated datasets is indeed impactful and relevant. It addresses a significant gap by allowing for general constraints on quantities of interest, such as average causal effect and unobserved confounding, during data generation. This capability is crucial because many prior generative modeling approaches for causal datasets either do not offer such flexibility or cannot ensure the preservation of these constraints, thus making this work a notable advancement in the field.\n\n* The paper is well-written, with clear explanations in the background sections on frugal parametrization and flows, which help the reader grasp the proposed approach. The details of the approach are well-articulated, and the experimental results are presented effectively.\n\n* The proposed approach is indeed novel. While it builds on established concepts like frugal parametrization, the specific application of normalizing flows for parametrization and its focus on average causal effect estimation represent a significant and innovative contribution.", "weaknesses": "My main concern with the work is the limited empirical validation of the proposed approach. Given that the primary contribution is the learning methodology rather than theoretical analysis, I would expect a more extensive set of experiments to validate its effectiveness. For example, prior research on generative modeling for causal inference, such as the work by [1], includes comprehensive experiments with various statistical tests to assess the realism of generated samples and a broader benchmarking of causal estimators. This paper would benefit from similar depth in its empirical evaluation.\n\nIt would nice if the authors can conduct similar experiments to asses whether learned generative model generates realistic samples and evaluate it on more datasets. Also, the authors should compare with the prior works [1, 2] as baselines to establish which approach is the best at capturing the underlying data generation process, and empirically validate their claim (Section 2.6) that the proposed approach would be better than prior works at capturing used-specified constraints on the average causal effect.\n\nReferences\n\n[1] Neal, Brady, Chin-Wei Huang, and Sunand Raghupathi. \"Realcause: Realistic causal inference benchmarking.\" arXiv preprint arXiv:2011.15007 (2020).\n\n[2] Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating causal inference methods. In International conference on machine learning, pages 17346–17358. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed analysis of our paper. Before addressing the specific questions, we would like to comment on the important points raised in the weaknesses section.\n\n## Weaknesses \nWe agree that a more comprehensive validation of our proposed approach would strengthen the narrative and enhance the perception of frugal flows as a reliable method for improving synthetic data generation. The two papers referenced by the reviewer, which we also mention in our paper, provide algorithms most comparable to our proposal. Credence (Parikh et al, 2023) allows exact specification of the CATE in generative samples. In contrast, RealCause (Neal et al, 2021) only allows users to scale the causal effect post-hoc, which hinders its ability to model a null hypothesis of ATE=0, where all outcomes are scaled to zero. Additionally, frugal flows’ ability to model unobserved confounding is shared with Credence but not with RealCause. Therefore, we chose Credence as the more appropriate method for comparison.\n\nFollowing the reviewer’s suggestions, we reran the benchmarking simulations in section 4.2 with Credence, using the default parameters specified in their package. We present an illustration of the correlation matrix of the pretreatment covariates and the outcome for the original data, data samples from frugal flows, and two distinct data samples from Credence trained with different causal constraints. We refer the reviewer to Figure 1 and Figure 2 in the document attached to the rebuttal. Frugal flows generate samples closely resembling the original data, particularly for the larger e401k dataset.\n\nThe samples generated by Credence also perform well in generating realistic samples for the e401k dataset. However, we note that altering the causal effect rigidity significantly impacts the covariate joint and introduces dependencies not present in the original data. A key advantage of frugal flows is the need to optimize the model fitting only once, allowing for direct modification of causal constraints without affecting the covariate joint and propensity score.\n\nWe also conducted the same multivariate statistical tests used by Neal et al. (2021) to validate RealCause. These tests include the Maximum Mean Discrepancy test (Gretton et al., 2012), the energy test (Szekely & Rizzo, 2013), the Friedman-Rafsky test (Friedman & Rafsky, 1979), and the k-nearest neighbor (kNN) test (Friedman & Rafsky, 1983), all implemented in the $\\texttt{torch-two-sample}$ Python library. Across both datasets, the tests suggest that frugal flows generate samples that are more comparable to the original data than those generated by Credence. In most tests, there is not sufficient evidence to claim that samples from frugal flows are distinguishable from the original data.\n\nRegarding benchmarks, we conducted experiments on two real-world datasets (Lalonde, e401k), which is the same number as the Credence paper (Lalonde, Project STAR). Due to time constraints, we could not obtain results by the end of the rebuttal period but are happy to run additional experiments to validate and contrast frugal flows against Credence using the Project STAR dataset, which was also explored in the Credence paper. This would put us on par with the RealCause paper, which validates their method using three real world datasets.\n\n## Questions\nIn response to the reviewer’s questions, we provide the following answers:\n\n1) We fully agree with your suggestion to change X to T when referring to treatments in the paper. Although we have not made this change in the current rebuttal to avoid confusion among other reviewers, we have incorporated this change in the manuscript.\n2) Thank you for spotting the typo! You are absolutely correct, and we have corrected the error in the manuscript.\n3) You’re right here—one could directly model $p_{X|\\mathbf{Z}}$ using a normalizing flow, which would be a valid frugal model. We model $c_{X|\\mathbf{Z}}$ to encode a degree of unobserved confounding in the generated data by sampling the ranks $U_{X|\\mathbf{Z}}$ and $U_{Y|\\text{do}(X)}$ from a non-independence copula. Assuming ignorability, these ranks would be independent. However, unobserved confounders imply marginal dependency between these ranks. Sampling from a copula replicates this effect, as demonstrated in the far-right plots in Figures 3 and 4 in the original submission. We discuss this in lines 260-265, explaining how to simulate unobserved confounding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wSUiyYafZQ", "reviewer": "Reviewer_J42v", "summary": "This paper introduces _Frugal Flows_ a method that learns the data distribution of data for causal effect estimation; namely outcome $Y$, binary treatment $X$ and pretreatment covariates $\\mathbf{Z}$. \nThrough a combination of frugal parametrisation, normalizing flows and copulas, separate components for the marginal causal effect $p_{Y| do(X)}$, the probability integral transforms of $\\mathbf{Z}$ and the propensity score are leaned.\n(The components of) the learned model, can be used for (i) estimating the marginal effect and (ii) to generate synthetic data with a fixed marginal effect for benchmarking other causal inference methods.\nIn the second application the component for the marginal effect is switched out for another with desired properties.\n(i) is demonstrated on small synthetic datasets.\n(ii) is demonstrated by fitting FFs to two real-world datasets and generating synthetic data with adjusted properties.", "questions": "- Fig. 1: What's the meaning of the red undirected edges. Does it mean they could go either way and/or they could be confounded? Please specify this somewhere in the paper or appepndix.\n- Def. 1, App. A: I struggle to understand this definition. The cartesian product on the RHS makes sense to me, but what does the LHS of the equation mean? What is the \"x\" operation between functions? Do the two functions map to the same space?\n- Fig. 2: Why is the first line needed? If I understand correctly, this just makes all pretreatment variables uniform. Couldn't you put $\\mathbf{Z}$ directly in the second line?\n- You show synthetic data generation based on two datasets in Sec. 4.2. Why couldn't you use the same datasets to test causal effect estimation in Sec. 4.1?\n- How many datapoints are in the Lalonde dataset?\n- In training, how did you check whether the training has succeeded? I suppose you minimize the log-likelihood, how did you define \"good enough\"? I'm asking because the training seems pretty fast (App. D2.4). What's the total number of parameters for each of the datasets?", "limitations": "- The biggest limitation of the work seems to be the requirements for the dataset size, needed to train normalizing flows. The authors mention this in Sec. 4.1. There could be some applications that have enough data for using Frugal Flows for causal effect estimation (e.g. online businesses with many customers, or well-curated medical datasets like the ones from healthcare providers in Isreal). However, for most applications the data won't be enough.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written.\n- It tackles an important problem in causality research. Since, randomized data is hard and expensive to get, many causal methods are only evaluated on synthetic data and generating realistic/semi-synthetic data is hard. This paper makes a great contribution towards improving synthetic data generation. If the code for the method is provided in a user-friendly manner, I could see this having a big impact on the causality community.", "weaknesses": "- Normalizing Flows have been used in the causal modelling context before (see [1, 2]). While prior works solve different problems (the inferred latents correspond to exogenous variables of an SCM, not directly applied to causal effect estimation), I think it would still be valuable to contrast this work to what has been done before for future reference in the literature.\n- L59: The basic causal assumptions aren't explicitly stated. What are the causal assumptions on $X$, $Y$ and $\\mathbf{Z}$? It seems like the method wouldn't hold if $\\mathbf{Z}$ was a mediator (I suppose the equation after L60 wouldn't hold). A reference to a 500+ page book is given for the assumptions, which feels like a slap in the face for the reader.\n- The notation for interventional distributions is confusing: what's the difference between using an asterisk and explicitly using the do-notation? In the equation after L60, the LHS seems to be an interventional quantitiy (asterisk, but no do-notation), whereas Equation (1) has the do-notation, but no asterisk. Do the two notation elements mean different things?\n- I think this paper would greatly benefit from a visual abstract showing how the different flows and distributions come together. Maybe this is something that could be added for the camera-ready.\n\n\nMinor:\n\n- L201: typo\n\n[1] Javaloy et al. \"Causal normalizing flows: from theory to practice.\" NeurIPS 2023\n\n[2] Wendong et al. \"Causal component analysis\" NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive assessment, thoughtful commentary, and suggestions. We share your aspiration that frugal flows can be engineered and presented in a user-friendly manner for the causality community.\n# Weaknesses\nWe would like to address the comments under the Weaknesses section, responding in the order presented:\n1) We appreciate the references shared by the reviewer. The Causal Normalizing Flows (CNFs) paper, also referenced by another reviewer, provides an interesting contrast to our method. While these papers tackle slightly different problems in inferring marginal effects, we are happy to include and discuss them in our literature review. Additionally, we implemented a variation of CNFs using default hyperparameters to estimate ATEs within our synthetic experiments in section 4.1. These are presented in Figure 1 of the attached document to the rebuttal. CNFs showed higher variance in ATE estimates than frugal flows, which is expected as CNFs are not designed for causal effect estimation. We also provided figures showing the observational likelihood loss for real-world benchmarks and synthetic experiments\n2) A key assumption for our model is that the covariate set $\\mathbf{Z}$ must be pretreatment covariates. You are correct that the parameterization would not hold if a $Z_{d} \\in \\mathbf{Z}$ were a mediator. We will explicitly summarize the required assumptions for our method in our paper.\n3) Thank you for the suggestion regarding the labeling of interventional distributions! The asterisk notation, taken from Evans and Didelez (2024), is used to contrast a wider range of distributions. However, in this paper, we consider a single static treatment model and are happy to use a simpler notation, specifically $do(X)$ operators, in an updated submission.\n4) We appreciate the suggestion of a visual abstract and believe it would significantly improve the presentation and readability of our method. We are delighted to incorporate this addition into our paper.\n\nWe also thank the reviewer for flagging up a typo, which we have corrected! \n# Questions\nIn response to the reviewer’s specific questions:\n\n* Fig 1: There is an error in Figure 1. There should be directed edges pointing from each $Z$ to $Y$ ($Z \\to Y$) and undirected edges between each of the $\\mathbf{Z}$ variables. The undirected edges indicate that the edges could point either way. We will clarify this in the paper.\n* Variation independence (VI) is a highly desirable property for a parameterization, since it allows different components to be specified entirely separately.  This is extremely useful if one is trying to use a link function in a GLM, or to specify independent priors for a Bayesian analysis.  In addition, VI is important in semi-parametric statistics.  The definition simply states that the Cartesian product of the images is the same as the image of the joint map.  For example, in a bivariate gamma-distribution with positive responses, then $\\mu_1 \\in \\mathbb{R}^+$ and $\\mu_2 \\in \\mathbb{R}^+$ is a variation independent parameterization, since \n$$\n(\\mu_1 \\times \\mu_2)(\\Theta) = \\mathbb{R}^+ \\times \\mathbb{R}^+ = \\mu_1(\\Theta) \\times \\mu_2(\\Theta).\n$$\nHowever, if we replace $\\mu_2$ with $\\mu_2’ = \\mu_2-\\mu_1$ (for example), then although the range of this parameter is $\\mathbb{R}$,\n$$\n(\\mu_1 \\times \\mu’_2)(\\Theta) = \\{(x,y) : x > 0, y > -x\\}  \\neq \\mathbb{R}^+ \\times \\mathbb{R} = \\mu_1(\\Theta) \\times \\mu’_2(\\Theta).\n$$\n* Fig 2: We separate the training of the marginal densities of the pretreatment covariates to ensure that the normalizing flow on the second line represents a copula flow. Training in a single line would not enforce that the marginal densities of the copula flow are uniform (except $V_{Y|do(X)}$). Thus, the marginal flows and copula flow must be trained separately (see lines 175-178 in original submission). Training the multivariate flow on the empirical CDF of the covariates also provides flexibility to generate realistic synthetic data from populations with marginal densities differing from the training data while maintaining the underlying dependencies of the original dataset.\n* In their current form, we do not believe frugal flows can reliably infer ATEs from small to medium-sized datasets. Figure 3 in the attached material illustrates the sensitivity of the ATE estimate as a function of data size, using the synthetic data models $M_{1}$ and $M_{2}$ in section 4.1 (original paper). Our experiments show that ATE estimates are reliably inferred only at data sizes greater than $N \\approx 10,000$. For this reason, we did not present ATE estimates for the real-world data, as they are both smaller than $10,000$. The e401(k) dataset is of similar size but with more than double the number of dimensions. This however does not impede us from generating realistic synthetic benchmarks as validated by Figure 2, Figure 3, and Table 2 in the document attached to the rebuttal. These provide empirical evidence that generated data is statistically similar to the original dataset.\n* The processed Lalonde dataset is of size $N = 614, D = 6$. We will add this information to the paper.\n* The parameter number of the frugal flow trained on the Lalonde and e401(k) data are 485243 and 106969 respectively. We will add this information to the paper.\n* In training, we perform a train-val split and use a “patience” criterion on the validation loss as a criterion to stop the training. Namely, we monitor the validation loss and stop training if the validation loss does not improve for a specified number of epochs (Chapter 5.5.2 in “Pattern recognition and machine learning” (Bishop and Nasrabadi, 2006)). We set the patience value to 100. This aims to prevent overfitting and saves computational resources by not continuing training unnecessarily. It is standard in machine learning model training and was implemented in the FlowJax package that we use as code-base to build the Frugal Flow package from.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their comprehensive commentary and very helpful suggestions for our paper. In addition to the individual responses to each reviewer, we provide a more global summary of what we believe were the core themes across all four reviews. These centre around 1) clarity and terminology, 2) further inference experiments (following Section 4.1 in the original submission) and 3) a more comprehensive validation of the synthetic data generation in Section 4.2, showing that frugal flows indeed generate realistic data samples which resemble the original dataset.
# Clarity and Terminology
Some reviewers suggested to increase clarity and precision in our definitions and terminology. In response, we have made the following changes:
1) **Definitions and Assumptions**: We are happy to add more detailed explanations and explicitly define terms such as conditional ignorability and variation independence. In addition, we wish to clarify the assumptions required for frugal flows; in particular, the covariate set must only include pretreatment covariates
2) **Clarification on Frugal Parameterization**: We will elaborate on our explanation of the frugal parameterization and more acutely describe how targets the causal margin rather than the conditional effect. This includes an expanded discussion on the relevance of copulas and how they enable frugal parameterization to capture the causal margin effectively.
# Further Inference Experiments
Reviewers expressed a desire for more extensive inference experiments, expanding on the contents in Section 4.1 in the original submission. We have addressed this by:
1) **Benchmarking Against Comparable Methods:** We are grateful for the additional references provided by the reviewers, in particular the Causal Normalizing Flows (CNFs) paper, which we are happy to add to the paper. We reran the experiments in Section 4.1 and use CNFs to estimate the ATEs in simulated datasets. We use the hyperparameter settings recommended in the package and do not perform hyperparameter tuning. For the flow architecture, we use Neural Spline Flows, as the paper reports in Appendix D.3 that they yield a better performance than simple Masked Autoregressive Flows. In addition to a (misspecified) linear outcome regression in the original submission, we use an implementation of a causal statistical matching algorithm to add further variety to the inference algorithms
2) **Expanded Experiments**: We conducted experiments using simulated data generated from two true ATEs (1 and 5) using models $M_{1}$ and $M_{2}$ described in Section 4.1 in the original submission ($M_{3}$ omitted for now due to time constraints, but we will update the paper with the results of all three models). The results, which are presented in Table 1 in the attached material, show that frugal flows consistently identified the true ATE with the lowest standard error. CNFs performed worse for data generated in $M_{1}$, demonstrating a clear bias, but correctly estimated the true ATE in data generated from $M_{2}$, with a lower standard deviation than statistical matching but higher than frugal flows.
# Validation of Synthetic Data Generation
We agree with the reviewers that comprehensive validation of our approach is crucial. To this end, we make the following comments:
1) **Contrasting against Credence and RealCause**: We have enhanced the validation section by adding a discussion contrasting Credence (Parikh et al, 2023) and RealCause (Neal et al, 2021), the two most similar existing methods for generating realistic synthetic data for validating causal inference algorithms. Credence allows the user to exactly specify what the conditional ATE is in the underlying data, much in the same way as frugal flows allow the user to exactly specify the ATE. RealCause only allows the user to scale the causal effect posthoc, which in particular hinders its ability to model a null hypothesis (i.e all observations will be set to zero); this is straightforward for frugal models (see Section 3 in Evans and Didelez, 2024). Furthermore, frugal flows’ ability to model unobserved confounding is shared with Credence and not with RealCause. For these reasons we chose Credence as a more appropriate method to compare experimentally against frugal flows, and discuss these experiments in the next paragraph.
2) **How Realistic are the Synthetic Data?:** We compared frugal flows against Credence using both the Lalonde and e401(k) datasets from our original submission. We present correlation matrices of pretreatment covariates and outcomes for the original data against samples generated from frugal flows and Credence trained with different causal constraints.These results are shown in Figures 1 and 2 in the attached document. Frugal flows generated samples closely resembling the original data, particularly for the larger e401(k) dataset. While Credence performed well in generating realistic samples, altering the causal effect rigidity impacted the covariate joint and introduced dependencies not present in the original data. A key advantage of frugal flows is the need to optimize the model fitting only once, allowing for direct modification of causal constraints without affecting the covariate joint and propensity score.
3) **Multivariate Statistical Tests:** We conducted a variety of multivariate statistical tests (Maximum Mean Discrepancy test, energy test, Friedman-Rafsky test, and k-nearest neighbor test) to assess whether the synthetic data from the generative models is statistically similar to the original training data. These results are presented in Table 2 of the attached document, and suggest that frugal flows do indeed generate realistic looking datasets.

Once again, we thank the reviewers and the area chair for their consideration of our paper, and we are happy to answer any follow up questions you may have.

Meta-review: Summary:

The paper introduces Frugal Flows, a generative modeling approach that learns the data generation process with explicit parametrization for the marginal causal effect of treatment on outcomes. 

Pros:

+ Addresses a significant gap by allowing for general constraints on quantities of interest during data generation for causal inference.

+ Novel approach combining frugal parametrization, normalizing flows, and copulas for causal effect estimation and synthetic data generation.

+ Well-written with clear explanations of complex concepts and potential for high impact in the causality research community.

Cons:

+ Limited empirical validation compared to prior work in generative modeling for causal inference.
Decision: Accept (poster)