SpaCE: The Spatial Confounding Environment
Authors: Mauricio Tec
Ana Trisovic
Michelle Audirac
Sophie Mirabai Woodward
Jie Kate Hu
Naeem Khoshnevis
Francesca Dominici
Venue: ICLR 2024 poster (2024)
Keywords: benchmarks
spatial confounding
graph neural networks
semi-synthetic data
causal inference
Primary area: datasets and benchmarks
TL;DR: We introduce SpaCE, the first comprehensive toolkit providing realistic benchmark datasets and tools for evaluating causal inference methods in spatial confounding scenarios.

Abstract: Spatial confounding poses a significant challenge in scientific studies involving spatial data, where unobserved spatial variables can influence both treatment and outcome, possibly leading to spurious associations. To address this problem, we introduce SpaCE: The Spatial Confounding Environment, the first toolkit to provide realistic benchmark datasets and tools for systematically evaluating causal inference methods designed to alleviate spatial confounding. Each dataset includes training data, true counterfactuals, a spatial graph with coordinates, and smoothness and confounding scores characterizing the effect of a missing spatial confounder. It also includes realistic semi-synthetic outcomes and counterfactuals, generated using state-of-the-art machine learning ensembles, following best practices for causal inference benchmarks. The datasets cover real treatment and covariates from diverse domains, including climate, health and social sciences. SpaCE facilitates an automated end-to-end pipeline, simplifying data loading, experimental setup, and evaluating machine learning and causal inference models. The SpaCE project provides several dozens of datasets of diverse sizes and spatial complexity. It is publicly available as a Python package, encouraging community feedback and contributions.

Published as a conference paper at ICLR 2024
SpaCE
THE SPATIAL CONFOUNDING ENVIRONMENT
Mauricio Tec, Ana Trisovic, Michelle Audirac
[mauriciog] ciogtec, trisoriiiie maudirac, khu) @haph harvard. Department of Biostatistics Harvard University
Sophie Woodward, Naeem Khoshnevis
Francesca Dominici
(swoodward, nkhoshnevis (efas hwwvard. edu Department of Biostatistics Harvard University
Edominic harvard. edu Department of Biostatistics Harvard University
ABSTRACT
Spatial confounding poses significant challenge in scientific studies involving spatial data, where unobserved spatial variables can influence both treatment and outcome, possibly leading spurious associations. To address this problem, we introduce SpaCE: The Spatial Confounding Enviromment, the first toolkit to provide realistic benchmark datasets and tools for systematically evaluating causal inference methods designed to alleviate spatial confounding, Each dataset includes training data, true counterfactuals, spatial graph with coordinates, and smoothness and confounding scores characterizing the effect of a missing spatial confounder. It also includes realistic semi-synthetic outcomes and counterfactuals, generated using state-of-the-aat machine learning ensembles, following best practices for causal inference benchmarks. The datasets cover real treatment and covariates from diverse domains, including climate, health and social sciences. SpaCE facilitates an automated end-to-end pipeline, simplifying data loading, experimental setup, and evaluating machine learning and causal inference models. The SpaCE project provides several dozens of datasets of diverse sizes and spatial complexity. It is publicly available as Python package, encouraging community feedback and contributions.
INTRODUCTION
Spatial data is paramount in several scientific domains, such public health, social science, eco- nomics, climate science, and epidemiology. Spatial patterns may take various forms, such as geographical coordinates network-defined adjacencies. These spatial patterns add complexity statistical inference problems and constrain our ability to answer important questions surrounding the causal effects that treatment variable has on an outcome of interest. fundamental concern is spatial confounding, which occurs when an unobserved spatial variable influences the outcome and treatment simultaneously, which may lead spurious associations between them, potentially resulting in biased estimates of causal effects (Gibbrrt et al. 2021b).
Consider, for example, a nationwide study estimating the effect of air pollution on cardiovascular hospitalizations. Urbanization is confounder since it is correlated with air pollution exposure and associated with other risks leading to higher hospitalization rates (Wu et al., 2020). If the study does not control for urban development, it may lead to incorrect estimates of the health effects of air pollution exposure. Since urbanization also varies smoothly in space (an urban area is likely to be located next to urban area), it is spatial confounder. In this specific scenario, one could account for urbanization using population density. However, confounders may be unknown, or measurements of known confounder might not exist. The field of spatial confounding in causal inference aims to control for potential unobserved confounders by exploiting the spatial structure of the data, under the hypothesis that the confounding mechanism varies smoothly in space.

---

Published as a conference paper at ICLR 2024
Spatial confounding methods seek to leverage spatial patterns to account for unobserved variables that vary smoothly in space (Gilbert et al. |202a Papadogeorgou et al. 2019 Tec et al. (023). While
there is an increasing interest in developing flexible machine-learming methods specifically tailored to alleviate spatial confounding (Gilbert et al., 2021b Veitch et al. 2019). realistic benchmark datasets to measure method's ability to reduce confounding are currently lacking. A likely reason for this absence is that counterfactuals and missing confounders cannot, by definition, be observed in real data, a challenge often known as the "fundamental problem of causal inference" (Holland 1986).
To address this challenge, we present SpaCE: The Spatial Confounding Environment, com- prehensive set of software tools and datasets for evaluating machine-leamning methods to ad- dress spatial confounding in causal inference. SpaCE datasets comprise real treatment and confounder data from publicly available sources commonly used in environmental health, social science, economics, and climate science studies, among other domains. Core to our approach is generating realistic semi-synthetic outcomes approximating true outcomes of interest using state-of-art machine-leaming ensembles (Erick- son et al. 2020) paired with cross-validation procedures for spatial data (Roberts et al., 2017) and spatial correlation modeling. In each dataset, there is a set of masked spatially-varying covariates inducing spatial confounding, and categorized by degree of spatial smoothness. SpaCE datasets contain spatial information in an adjacency graph and/or geographical coordinates that methods can utilize. The diversity of SpaCE datasets, illustrated by Fig. [ spans multiple treatment types (binary, continuous), spatial formats (graph, coordinates), and evaluation metries (average causal effects, dose- response curves, counterfactual prediction at the unit level). In addition to our SpaCE Python package, we provide tools to reproduce existing datasets and generate new ones from a user's data
Treatment type Binary Continuuus
Misting confounding High
Confounder smoottrness High Lou
Lou
Spalial Format
Csural estimand
Graph
Plu. treatmentt
Coordinates
Spa Environments
Effect Heterogenclly
Grid
Expopur e-nesponse
Figure I: SpaCE encapsulates essential compo- nents necessary for causal effect estimation algo- rithms using spatial data, including ground-truth counterfactuals.
Insufficiency of current solutions Benchmark datasets that are currently used in causal inference lack a spatial structure and thus cannot be used for evaluating spatial confounding methods. Nonethe- less, there has been an increasing interest in establishing best practices for the evaluation of causal inference methods outside of the spatial confounding literature (Cheng et al. 2022 Curth et al.] al.] 2021). There are three predominant approaches. The first one uses fully simulated treatment, confounders and outcome data by sampling from known probability distributions to verify the theoretical properties of method in controlled settings (Morris al., 2019). While simulation studies play an important role in developing new methods, they are insufficient for evaluating a method's 's performance in practice wide range of settings, particularly the context of machine learning (Breiman 200T). A second approach is to use only real data and create benchmark datasets using biased sampling from randomized controlled trials (RCTs) (Gentzel et al. 202T). However, RCTs are not available for spatial data. Additionally, RCT methods do not support the generation of counterfactuals. The third approach combines synthetic outcomes with real covariate and treatment data. These methods can be categorized as those using data-driven methods for generating synthetic outcome (Neal et al. 2020) and those using random user-fpe probability distributions without referencing actual outcome data. The latter category has the advantage of generating a large number of diverse benchmark datasets (e.g., Dorie et al. (2019)), but has recently received criticism since it lacks representativity of real data and can artificially favor arbitrary algorithms (Curth et al. (021). The Infant Health Development Program (IHDP) dataset is the best-known example (Hill 2011). By contrast, SpaCE belongs to the former category by using ensemble methods approximate real outcomes.
2 BACKGROUND ON SPATIAL CONFOUNDING
We introduce some notation using the potential outcomes commonly used in the causal inference literature (Rubin 2005). Let A, and Y. be the treatment and outcome of interest at each location E S. X, indicates the vector of confounders. Boldface notation indexes spatial processes, for example, X (X.)ses. The spatial structure in S is determined by discrete graph or by a continuous
'The spack source code is available hnttps://anonyyouus.aae 4opan sci httpppen.scieennee///aaa.cc.. RC93
2

---

Published as a conference paper at ICLR 2024
Syrtweic porterfecial Syntwec (ardg)
3100
000
310
(a) True outcome Y (b) Synthetic outcome Y, (c) True residuals ft, and (d) Counterfactuals (po- synthetic residuals R. tential outcomes)
Figure Example synthetic outcome and residuals from the synthetic data generation in the healthd pollutn mortalitytttto...m... cont environment.
coordinate system. Y represent the potential outcomes under a potential treatment assignment a at location We also refer to potential outcomes counterfactuals.
The goal of causal inference is to estimate some function of the counterfactuals that we commonly
refer to as causal effects. Examples include the average treatment effect Tate = [S] E.es(Y. (Y:
Y for binary treatment, the exposure-response function Tar(a) - 2.ER Yo, and the counterfactuals themselves. when there E|Y,|A, = a, X,]. The left-hand side in this last expression involves counterfactuals, whereas the right-hand side can be estimated via regression. This result is known identification. When confounder is missing, identification is not guaranteed, that is, E # TE X,,
well known that when all confounders are observed -tht is, E|Y;|X,]
unobserved variables affecting Ag and Y simultaneously-
X,, Ag = a)
(Rubin 2005 Pearl 2009).
Spatial confounding results from the co-occurrence of two conditions: there is an unobserved
confounder (say, X, (X X )), and the process X miss shows strong spatial auto-correlation. The closer locations are s are, the more correlated X and X become (as illustrated in Fig. 2 pronounced spatial distribution of missing confounder suggests a similar confounding nature between nearby locations and s'. This lack of identification is what methods for spatial confounding aim address, exploiting knowledge of the spatial structure. While this goal may be achieved in different ways in different methods, the general strategy can intuitively be described as learning some function of space Z. so that identification holds, that is, E| 2.] ~
E|Y, A = a, X Zz).
As technical remark, the spatial confounding literature typically assumes that the treatments of one unit not affect the outcomes of other units. condition known interference (Forastiere al. 202T). Much like spatial confounding, interference can bias causal estimates. However, their underlying mechanisms and solutions are distinct (Ogburn & Vander Weele 2014; Gilbert al. 2021b Paaaddgeergouu
Xobs
xmiss
Xg
miss
Ag
Ag
Samanta 2023). It is worth noting that failing to address spatial confounding may give the false impression of interference even if the latter is present in the data (Papadogeorgou & Samanta 2023). Here, we focus on spatial confounding, deferring interference future work (we provide additional discussion in Section (6).
Ysr
Figure 2: Causal diagram spa- tial confounding with neighbors s and s'. Arrows represent causal relations; undirected dotted lines represent non-necessarily causal associations. The correlations in- crease as the distance between s and s' decreases.
Lastly, notice that the term spatial confounding has been used inconsistently across research, sometimes without clear causal inference interpretation (Gilbert et al., [0211b] Khan & Berrett 2023). For instance, spatial confounding has sometimes been characterized as a property of statistical models (Dupont et al.) 2022), specifically, the correlation between the spatial random effects of model and the treatment variable. Another definition has been concerned primarily with variance adjustments for spatially autocorrelated data (Khan & Calder 2022 Reich et al. 2006).
3

---

Published as a conference paper at ICLR 2024
Environment and Dataset Generation
AutoMil counterfactunts with spatsppll--aaaeee sawars tuning
Dataverse g
Seport spacebench as space space dataset env aske((
(naee)
Project
Real datasets of treatment, canfounders, and putcomes, based on existing studies.
indures model asgrosimabig the culcome. original data are incalid d
Environments with known counterfactuals, confounding scores, and spatial smoothness scores per confounder. They are hasted on Dataverse server for accessibility and version control.
Easily retrieve environments and create realistic causal datasets with controlled spatial confounding
DataCollection Initiaa collections
SpEcify creadment, gasorate Cuntertactssis
SpaceEnv
mask group spacially smooth confounders
SpaceDataset 224 initial dutaseta
11 Antcial envirorments PRIDICE san0el assirsiion
Terminology in the SpacE API
Figure 4: (Top): The semi-synthetic data generation pipeline and acquisition. (Bottom); Summary of
the key terms used in SpaCE.
3 SpacE: THE SPATIAL CONFOUNDING ENVIRONMENT
We first explain the design goals and overview of the SpaCE pipeline and its components. Next, we provide details about the various implementations of these components.
Design goals The first requirement is that the generated datasets are representative of real data. We will achieve this by using real covariate and treatment data with realistic, semi-synthetic outcomes and counterfactuals. We will provide details later in this section. Data generating mechanisms producing datasets that could artificially favor specific model type during causal evaluation should be idcally avoided (Curth et al. 2021). The spatial autocorrelation of the residuals (the unexplained variation in the outcome) should also be represented accurately since various spatial confounding adjustment methods are based on adding "spatial effects" to a predictive model (Dupont et al.) 2022), and their performance in resolving confounding is affected by the spatial autocorrelation pattern in the data.
Overview of the pipeline The SpaCE pipeline consists of two stages: (1) generate a synthetic data environment by selecting a treatment, outcome, and covariates from a publicly available data collection based on realistic scientific question; and (2) obtain a benchmark dataset with controlled spatial confounding by masking group of spatial covariates in the environment. To illustrate the idea with an example, let us consider the treatment is air pollution exposure and the outcome hospitalizations. First, we will fit a predictive model using air pollution and set of observed confounders, such as socio-demographic characteristics, including urbanization. For the residuals, we will then use a probabilistic model to obtain an independent sample of spatial process approximating the distribution of this model's 's errors. Finally, we would create benchmark dataset by masking group of confounders, such as urbanization variables. This example is present in SpaCE, with the generated data displayed in Fig. 3, showing the counterfactuals, outcomes, and residuals alongside the real data, highlighting their striking resemblance.
Key terminology We introduce three key terms that will facilitate the presentation of our data generation pipeline and are used throughout the Sp&CE CE API. These terms are illustrated in Fig. 4 First, DataColl lect in comprises variables from publicly available data surrounding a theme or topic of interest. From each of these collections, we can obtain various causal inference environments, or SpaceEnvs, after specifying a treatment, outcome, and set of confounders-emmen emulaatig a realistic study-and implementing our data fitting and generation method. Each environment contains semi- synthetic counterfactuals with important metadata, such as a list of edges or geographic coordinates. Environments are stored on a server where it is publicly accessible, versioned, and documented. Lastly, each environment yields various benchmarking datasets, or SpaceDatasets, obtained by masking group of related confounders varying smoothly in space. Every SpaceDatases contains metadata about the degree of spatial smoothness and a "confounding score" determined by their respective omitted group of confounders.
Data Collections Dat cerooleeciion ions are the entry-point to the semi-synthetic gettrggnne six DataColl 1 deectionnn from various domains and sizes,
algorithm. Currently, SpaCE offers summarized below and in Table Appendix A provides additional details about the public data sources composing each collection.
The smallest dataset represents a spatial graph of 3, 109 nodes, while the largest has nearly million nodes. Notice, however, that the size of the smaller datasets may already be computationally
4

---

Published as a conference paper at ICLR 2024
Table I: List of Dat a DataCollection and SpaceEnvs A SpaceEny name has four components: prefix is the code for the source DataColl Setaaoollectt leelleciiio ion (cdesvi, el imate, healthé, county), it is followed by the treatment and outcome codes, and ends with the code for the treatment type (cont, disc).
Lacllll lection
Trestmntt
Spatial Unit
Regioa
#Conssiates
#Nodes
#Edges
SpaceEny
Type
#SpaseDacasets
Social Vulnerailitt and Welfare
comsus tract
Texas
16
6,828
21,585
odeavi_ enn aburdie_com
consinauus
12
soverty_comt
comtinuuus
11
cdleevi Mellevi__riieee Liiiddd powerr by_.
bisury
11
Expusure and Willlires
22
8,616
26,093
clinate_relhun. (artuuun_kktkk_eooo
contimous
clinatc, wtonoce minrtyydiissi. .i02
Nearry
Air Pollitim and Morially
county
USA
34
3,100
9,237
hea lthd
nortality wiitt by. diac
hinury
healtha www.pott_norraa..... ttyrsality_.ont
consimouus
Welfare Flections
county
USA
45
3,109
9:237
county. y. rddducatnn n_.iiattn_=lettii elett ion_oont
continecs
county county_physetiet physetiv_ wttps___wwwooep_o_oo___
contimouus
16
county dngros elect diec
bisary
14
PM, Components
USA
3,882956
7,697552
pm25 unis__hiess no pon_o po25 mnie__ires___ 2022 cont Socioeconmii Stans and Broadband Usage zot Lnnoee, besote___reeeee breadlband diac 2ota_age_ wrw.aaaggdoodaaddd.ooo.
contimous
consinauus
code
USA
16
30,33
174,35
discrete
demanding for some spatial confounding algorithms with polynomial space complexity. In our experiments, we found this be true using many practical implementations of spatial regression methods, such as those based on the pysal library (Rey & Anselin 2007). Further, the literature on spatial confounding has focused strongly on developing /n--consistent estimators that have quick convergence with high performance on similarly sized or smaller datasets (Gilbert et al.) 2023). Notably, our benchmark dataset sample sizes surpass those in many simulation studies of these methods. On the other hand, the largest datasets can be interesting for the performance of scalable methods, such as neural networks.
4 DETAILS ABOUT THE DATA GENERATION PIPELINE
We now provide technical details about the data-generating mechanism in SpaCE.
Spatial Semi-synthetic Data Generation: From MataCollection to SpaceEnv Given DataCollection. we select a set of treatment, outcome, and covariates to represent realistic scientific question. Each combination produces a semi-synthetic outcome and counterfactuals, resulting in unique SpaceEnv. Let X, A, and Y denote the confounders, treatment, and outcome. We approximate the distribution of the outcome as Y, = =(Xx, A,) + R, such that Ar X) x Pr(Y A, X). We learn / and R for each SpaceEnv. We can then generate counterfactuals by evaluating on any treatment value:
y = f(x,, R,,
(1)
The strategy to generate counterfactuals has four steps. First, we learn f that best predicts Y, from (x,,A)) using AutoML. Second, we estimate the empirical additive errors f. = Y,, 105-(Aa -(X,, Ag) and their joint (spatial) distribution R ~ PR- Third, we replace these endogenous residuals with an independent similarly distributed exogenous noise R ~ PR. Finally, we obtain counterfactuals with Eq. by varying the treatment while holding constant the confounders and the exogenous noise. The methodology implementing these steps pursues the following desirable properties: (a) f captures non- non-linear relations and interactions; (b) R exhibits a similar spatial autocorrelation the one observed in the data when f captures all the causal relations from (X, A) to Y: (c) the resulting dataset does not exhibit additional unobserved confounding. Sampling R independently ensures that the synthetic residuals are independent of the treatment, and therefore, the only possible confounders are those in X by the backdoor criterion (Pearl 2009). Below, we provide more details about the implementation.
Predictive model. For (a), we learn f using ensembles of muahine-learning models where the ensemble weights are determined by their predictive ability validation data. We do this to
5

---

Published as a conference paper at ICLR 2024
avoid favoring causal algorithms based on a specific type of model, recommended by Curth let al. (2021). who discuss best practices for causal inference benchmarks. We fit the ensemble using the Aut Python package (Erickson et al., 2020) to reduce human intervention, automatically selecting the best hyperparameters and performing various overfitting reduction techniques. The package options and configuration are described in Table in the supplement. We found it critical to implement a spatially-awar train-validation data split (Roberts et al. 2017), resulting in extreme overfitting without it, caused by the duplicity of training data in the validation set due to spatial correlations. The spatially-awart splitting scheme selects small subset of validation nodes and uses breadth-firs search remove their surrounding neighbors from the training subset. This algorithm is described in Algorithm in the supplement.
Residual sampling. For (b) and (c), residual sampling, we
Gaussian Markov Ran- dom Field (GMRF) from a spatial graph, a convenient and scalable spatial process (Rue Tee et al., 2019). More precisely, we sample the synthetic residuals as R ~ia Mulmial 0. X(D PAD) where & the adjacency matrix of the spatial graph; ID is a diagonal matrix with the number of neighbors of each location; و determines the correlation between an observation and its neighbors, learned from the true residuals (the errors of the predictive model in Step 1); and À is chosen to match the variance of the true residuals exactly, A key technical element in our implementation was sampling from large Gaussian random field using matrix factorizations for sparse matrices (Chen et al. 2008), enabling scalability to massive spatial graphs (such as in the PM2.5 Components Dat www.aaoolcciiiiooo
& Held 2005;
Panel (c) in Fig. 3 iillstrated the marginal distri- bution of the fitted residuals compared with the original ones. Further, panel (d) showed the fit- ted counterfactuals, which are visualized as fune- passing through
Table 2: Contents of SpaceDataset.
Name
Description
Training data Counterfactuals
X A and Y
tions of the treatment/exposiee the observed data points. Fig. F the appendix provides additional visualizations comparing the synthetic and real residuals. In particular, panel (b) compares the Moran I measure for spatial au- tocorrelation across all of the SpaCE datasets, indicating strong correspondence validating the learning procedure.
Y for each treatmntt value A. cumputed with Eq. (T For contin- treatments, discretization of [A] = 100 values is used
Graph and coordinates Treatment type
A of odges coordinate matrix
An indicator if [A] = 2 (binary) or
[&] (continunus)
Spatial smoothness
Auto-correlatoon of X
Confounding score
metric of the degree of confound-
ing induced by X
Inducing Spatial Confounding: From SpaceEnv to SpaceDataset A benchmark dataset is obtained by masking a group related confounders a SpaceEnv. Masking entails removing a group of related covariates from the training data. A dataset object encapsulates the essential compo- nents required for estimating causal effects and comparing the performance of spatial confounding methods. The contents of SpaceDataset are summarized in Table 2 The variable names in the space datasets are anonymized to discourage inappropriate use of our benchmark datasets to infer effects in the original data collections. We provide additional discussion in Section 6
Eq. T specifies an additive noise model (ANM) used for counterfactual generation. While this model is additive, SpaceDatt tasets can exhibit more complex forms of non-additive noise due to the interactions with the missing confounder. Nonetheless, it is worth noticing that ANMs remain the focus and leading open challenge in the spatial confounding literature (c.f., Akbari al. )023)). Even when outcomes are count-based or binary, which could indicate non-additive noise, simple techniques from generalized línear models (e.g., Poisson regression) or outcome transformations (e.g., logarithms) have been adapted to complement spatial confounding methods (c.f., Urdangarin al. (2023). Thus, SpaceDat SaceeDattseett asets s are highly relevant to the current state of the art in spatial confounding adjustment. The ANM will be sensible model when the empirical residuals are approximately symmetric and continuously supported, as exemplified in Fig. 3p. Documentation of the outcome transformations and histograms of the empirical and synthetic residuals are stored with each SpaceEnv in the Harvard Dataverse platform.
Tasks and Metries Tasks in SpaCE are determined by causal effect target which, in turn, de- termines the metrics used to evaluate performance. Not all algorithms can estimate all types of causal effects, and the relevant effects also depend whether the treatment is binary or continuous. Section 4 summarizes the causal effect targets currently supported in SpaCE and the associated metrics which can be computed from the counterfactuals. SpaCE provides an auxiliary class called

---

Published as a conference paper at ICLR 2024
Table Supported causal estimation tasks and metrics. r. denotes causal effect and f. is an estimate, where is the name of the task. For the last column, Y: is estimate of the counterfactual Y, for each location s and treatment a, continuous treatment types, we assume | = 100. The metrics are standardized by the standard deviation of Y., denoted dy.
Causal Rffect
Treatment type
Metric
average treatment effect (ATE)
binary
absolute bias (BIAS)
Tax [8] EV,' Y Y,)
Fate
Tate
exposure-response function (ERF)
continuous
root mean integrated squared error (v Mv/I)E)
A me((a) = |8| Exes Y
"y VIA - (f (a))))
individualized treatment effects (ITH) Va A 8::
binary and continuous
precision estimating heterogeneous effects (PEHE) || 1 Y2y2
DatasetEva Luator to estimate these quantities. The performance metrics are standardized by the standard deviation of the synthetic outcome that they are comparable across datasets and environments.
The three causal estimands currently supported in SpaCE are the average treatment effect (ATE) (for binary treatments), the exposure-responss function (ERF) (also known as the average dose-response curve), and the individualized treatment effects (ITE). The value of these metrics is defined in terms of potential outcomes in the first column of Section 4 The ATE and ERF are averaged across the population, while the ITE is simply the counterfactual value for each unit. We will measure performance based on metrics commonly used in the machine learning literature for these estimands (c.f., (Cheng al. 2022 ): the absolute bias BIAS for the ATE (Hill 2011 Shi et al.) 2009);; the mean integrated squared eтror MISE for the ERF (Schwab et al. 2020 Nie et al. 2020); and the precision in estimating heterogeneous effects РEНE for the ITE (Hill 2011 Shi et al. 2019). The names of
these metrics vary in the literature, but the definitions are consistent or present small variations. The standard deviation of the outcome normalizes all effects so they are comparable across datasets.
Confounding score and smoothness scores These scores characterize the properties of the masked confounders of SpaceDat aset.. Smoothness scores are calculated using Moran's 's statistic, which is approximate form of spatial correlation and takes values in [- (Moran 1950 The confounding score is computed as a baseline model's change causal estimates when masking set of confounders. We use the same AutoML configuration of the predictive model as the baseline. Each causal estimand defined in Section 4 has an associated confounding score for each benchmark dataset. For ease of presentation, we the confounding score of the exposure-response function as the single reference. We will also classify datasets in low or high confounding/smoottness, using the median across all datasets as the threshold.
5 EXAMPLES AND EXPERIMENTS
Our first experiment investigates whether or not the generated SpaceDatasets have the property that (1) the causal effects can be learned from the generated semi-synthetic outcomes and (2) masking confounders reduces the estimation performance. To assess this, we collected the estimation errors of three baselines when using the unmasked full covariates and when learning with masked spatial confounders based on the sidered are ordinary least-squares (OLS), gradient boosting (XGBOOST), and a multi-layer perception (MLP). We did not tune for the best architecture in this experiment since the purpose was not to discover the best method but to evaluate points (1) and (2) just mentioned.
environment. The baselines con-
The results of the experiment are shown in Fig. 5 In all cases, training with the masked SpaceDatasets decreases performance compared to using all the confounders. The MLP is the best performer, also corroborated by the examples of fitted curves in panel (a), which reason- ably acurate. Another takeaway is that a method cannot perfectly estimate the true causal effects due to model limitations and finite sample error, even when using all the confounders.
Our second experiment illustrates the evaluation of spatial confounding methods. It is worth noting that linear models have dominated this literature without strict attention scalability. Thus, due to
7

---

Published as a conference paper at ICLR 2024
05
50
50
40
20
rplason
40
80
22
80
100
-0
120
enpoe
Thelmment
Theefment
Twetment
(a) ERF estimation in SpaceDataset.
(b)
Errors
SpaceDatasets.
Figure 5: Examples using the cdosvi_ wwwp:/ohhhsppppporoooooo centererneycoootteeeeeeeeeet environment.
limitations in the scalability of some of the baselines taken into account below, we only conducted this experiment on the first four data collections in Section 4
We first include two baseline models from the PySAL Python library (Rey & Anselin 2007). These models are the Spatial Two Stage Least Squares (Anselin 1988) (s2SLS), implementing an outcome auto-regressive model, and the General Methods of Moments Estimation of the Spatial Error (Kelejian & Prucha 1999, 1998) (GLMERR), implementing a residual auto-regressive model. Next, we consider the SPATIAL and SPATIAL+ models, which use spatial splines and a two-stage regression procedure to account for the spatial portion of the treatment (Dupont al. 2022). We also consider a Graph Convolutional Neural Network (Kipf & Welling 2016) (GCNN), which uses neighboring information similarly to S2SLS but can also model non-linear relations from potentially longer range of spatial dependencies. Thus, while not typically considered a spatial confounding method, it worth evaluating its performance. Next, we consider the distance-ate adjjusee propensity score method (DAPSM) (Papadogeorgou et al., (2019), which uses a matching estimator (Rubin 1973) that prioritizes nearby matches. This method finds pairs of treated and untreated units to compare their outcomes as an estimate of the causal effect (only applicable in the binary case).
To ensure a fair comparison, we use the RAY TUNE (Liaw et al. 2018) framework for hyperparameter tuning. In all cases but DAPSM, we use the out-of-sample prediction as the tuning criterion. For DAPSM we use the covariate balance criterion following laaaadggeorgou et al. (2019). Appendix D provides additional details about the tuning hyperparameters of each method. Importantly, we use the spatially-awar aware vain-validation splitting for computing the tuning metric since random splitting would result in extreme overfitting (Roberts et al. 2017).
Table shows the results, which vary strongly across settings. Notably, the SPATIAL+ algorithm excels in estimating binary average treatment effects yet underperforms in continuous treatment scenarios. In the latter case, the GMERROR seems to yield the best performance, but without statistical significance. Another interesting result is that linear models consistently outperform GCNN, even when they cannot inherently account for effect heterogeneity as the neural network. Although the GCNN could perform well in various experiments, it would sometimes lead to large errors, affecting the final metric. In general, models highly sensitive to overfitting were more challenged by the challenges of hyper-parameter tuning due to the spatial correlations.
Fig. 10 in the appendix summarizes the experiment results grouped by SpaceEnv. One takeaway is that the linear spatial regression baselines GMERR and SLSSL offer only very small improvement over baseline OLS in all environments. This result could indicate the need to focus on expanding the caatee-eernnnng--ee
models. We can see that the GCNN had better performance than the other models in the two environments from the Heat Exposure and Wildfires collection (see Table M despite its overall worst performance, warranting future investigation of what properties of this collection allowed for better performance of the GCNN.
literature on
Lastly, we analyzed the baselines' performance by smoothness and confounding score. For this purpose, we use a linear mixed effects model (Galecki et 2013). See Appendix for additional details of the model specification. The results of this analysis, shown in Table 9 of the appendix, indicate that the errors were lower the smoothness score was higher for all baselines. DAPSM benefited the most from higher smoothness. Conversely, a higher confounding score is associated with higher errors in all but case. Many of the effects are moderately statistically significant. This analysis provides additional evidence that the smoothness and confounding scores are meaningful in describing the complexity of benchmark dataset, although with high variance.
8

---

Published as a conference paper at ICLR 2024
Table Benchmarks are collected across all datasets. Table entries show the mean with 95% confi- dence intervals using the asymptotic normal formula. "High smoothness" and "high confounding" denote datasets in the top 50% based on spatial smoothness and confounding score. Sample sizes, depicted in Fig. 8 reflect dataset combinations of smoothness, confounding, and treatment type.
BINARY TREATMENT
CONTINUOUS TREATMENT
SMOOTHNESS
CONFOUNDING
METHOD
ATE
ITE
ERF
ITF
HIGH
HIGH
DAPSM
0.29 0.20
0.25
N/A
GCNN
0.16 + 0.07
0.25
0.08
0.49 +
0.72+00.16
OMERROR
0.07 = 0.04
0.05
0.26 0.07
0.45
S2SLS
0.06
0.007
0,26 0.07
0.45
SPATIAL+
0.03
0.13
0.42
0.56
SPATIAL.
0.05
0.006
0.26 0.08
0.44 0.11
LOW
DAPSM
0.396
0.37 0.00
N/A
NIA
0.14
0.28 0.004
0.44
0.12
GMERROR
0.05 0.02
0.18 a: 0.04
0.37 + 0.19
0.19
$2SLS
0.02
0.18 0.04
L.01
0,94 1.06
SPATIAL+
0.03
0.18 + 0.04
0.41
0,47
SPATIAL
0.01
0.18 0.04
0.37
0.42
LOW
HIGH
DAPSM
0.18
0.10
N/A
N/A
GONN
0.14
0.18 = 0.08
0.31 0.08
0.57 0.07
GMERROR
0.09
0.006
0,31 0.07
0.48 = 0.10
$2SLS
0.09 1: 0.01
0.16 0.06
0.01
0.48 0,
SPATIAL
0.00
0.47 0.06
0.60
SPATIAL
0.06
0.15 =
0
0.48 0.10
LOW
DAPSM
0.51
0.45
N/A
GCNN
0.17
0.34 0.003
0.36 0.09
0.50 0.09
OMERROR
0.02
0.32 0
0.37 = 0.19
$2SLS
0.08 + 0.05
0.19
0.77 0.57
SPATIAL
0.05 0.02
0.00
0 0.24
SPATIAL+
0.03 0.01
0,18 0.004
0.19
0,40
6 CONCLUSION AND DISCUSSION
By enabling a systematic evaluation and comparison of spatial confounding in realistic scenarios, SpaCE represents a significant step towards addressing spatial confounding, a fundamental issue in many scientific problems with spatial data. SpaCE advances causal inference in spatial data by providing a stll-structured, adaptable environment for benchmarking and improving existing and future methods. We will continuously grow the available environments in the SpaCE Python package to include additional DataCol 1 rection from various domains.
There are exciting opportunities for future work. One possible extension is temporal confounding, which shares mechanisms with spatial confounding when a confounder has lagged effects and varies smoothly in time (Imai et al. 2021 Papadogeorgou et al. 2022). Our methods to generate semi- synthetic data may be reused. In particular, our approach to function estimation could use time series methods and residual generation could be generalized to spatio- teatorll graph (Tec al.) 2019). However, the methods to resolve temporal and spatial confounding can differ significantly. For instance, iifferenceeiindiffeeenee methods are only used with longitudinal data. Similarly, baselines will be different, as well as residual sampling strategies.
Another possible extension is interference, also known as spill-overs, in which the treatment of neighbors affects the outcomes of other units. However, significant additional work is needed for this extension. First, ensembles of mainstream machine learning models (e.g. boosting, random forests, MLPs) are not suitable for data with unknown interference (Bhattacharya et al. (020). GNNs are promising tools, but much evaluation is still needed (Ma & Tresp 2021), and, for best practices, ensembling methods should not rely on single type of generative model (Curth et al., (2021) Second, the interference literature focuses on estimating spillover effects (Hudgens & Halloran 2008). Metrics and evaluation tools for this task need careful design. Third, the data domains of interest in the spill- overs literature may vary. For instance, it has concentrated more often on social networks (Forastiere) et al. 202T), while they have been of less interest in spatial confounding.
Addressing these future work opportunities here would have required substantial developments that some solutions for these extensions already exist (Cheng et al. 2022). while SpaCE addresses a fundamental gap in spatial confounding, for which no solution with realistic data currently exists.
beyond the scope of spatial confounding. It must be highlighted that

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT AND BROADER IMPACT
We consider several ethical aspects while designing and deploying SpaCE. First, we take several actions to minimize potential harm due to misusing semi-synthetic datasets to derive conclusions and inferences about causal effects in real data: we (1) anonymize variable names SpaceDatasef (2) issue warning statements clarifying limitations when users import the Python package, and (3) emphasize the synthetic nature the data in the description of each environment where they available publicly. Second, to increase transparency and reproducibility, we provide detailed instructions for full reproducibility, including publicly available and accessible code. Third, while our work does not directly address concerns about fairness and societal biases, data collections integrate sources commonly used in studies of social vulnerability and health equity, driving methodology improvement for social good.
data repository
ACKNOWLEDGMENTS
This work was supported by the NIH awards RFIAG0S0948-S) (MT. NK). RO1ES029950 (MT), IR01MD016054-01A (FD), 5R01ES030616-D (MT), AG0743307472- (MT), P30ES000002 (MT), and R01MD01 S0244-2 (MT); NIEHS training grants T32ES007142 (SW) and T32ES007069 (KH); Alfred P. Sloan Foundation grant G-2020-13946 (FD, KH); and the Fern- holz Foundation (MT). The computations in this paper were performed on the FASRC Cannon clusters supported by the FAS Division of Science Research Computing Group at Harvard University.
REFERENCES
Kamal Akbari, Stephan Winter, and Martin Tomko. Spatial causality: A systematic review on spatial causal inference. Geographical Analysis, 55(1):56-88) 2023.
Heresh Amini, Mahdieh Danesh- Danesh-Yazdi, Qian Di, Weeberb Requia, Yaguang Wei, Yara Abu-Awad, Liuhua Shi, Meredith Franklin, Choong-Min Kang, Jack Wolfson, et al. Hyperlocal super-learned pm2. 5 components across the contiguous us. 2022.
Luc Anselin. Spatial econometries: methods and models, volume 4. Springer Science & Business Media, 1988.
Rohit Bhattacharya, Daniel Malinsky, and Ilya Shpitser. Causal inference under interference and network uncertainty. In Uncertainty in Artificial Intelligence, pp. 1028-1038, 2020.
Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3):199-231, 2001.
Bureau of Labor Statistics. Local area unemployment statistics. http:://wwww.bb..govww/wttttaaa 2019.
CDC. Behavioral risk factor surveillance system (brfss). http://wwww.dd...wwnrr/s/nddd... 2010.
Centers for Disease Control and Prevention. http:///wwwwwddddd.ooo//////dees
2010
census.
2020.
Yanging Chen, Timothy A Davis, William W Hager, and Sivasankaran Rajamanickam. Algorithm 887: Cholmod, supernodal sparse cholesky factorization and update/downdate. ACM Transactions on Mathematical Software (TOMS), 35(3), 2008.
Lu Cheng, Ruocheng Guo, Raha Moraffah, Paras Sheth, K Selçuk Candan, and Huan Liu. Evaluation methods and measures for causal learning algorithms. IEEE Transactions on Artificial Intelligence, 3(6):224 943, 2022.
Marissa L Childs, Jessica Li, Jeffrey Wen, Sam Heft-Neal, Anne Driscoll, Sherrie Wang, Carlos Gould, Minghao Qiu, Jennifer Burney, and Marshall Burke. Daily local-level estimates of ambient wildfire smoke pm2. 5 for the contiguous us. Environmental Science & Technology, 56(19): 13607-13621, 2022.
10

---

Published as a conference paper at ICLR 2024
Alicia Curth, David Svensson, Jim Weatherall, and Mihaela van der Schaar. Really doing great at estimating cate? a critical look at ml benchmarking practices in treatment effect estimation. In Tiirty-fifth conference neural information processing systems datasets and benchmarks track (round 2), 2021.
Qian Di, Heresh Amini, Liuhua Shi, Itai Kloog, Rachel Silvern, James Kelly, M Benjamin Sabath, Christine Choirat, Petros Koutrakis, Alexei Lyapustin, et al. An mnsemleebbsse model of pm2. concentration across the contiguous united states with high spatiotemporal resolution. Environment international, 130:104909, 2019.
Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, and Dan Cervone. Automated versus do-it- yourself methods for causal inference. Statistical Science, 34(1):43-68 2019.
Emiko Dupont, Simon N Wood, and Nicole H Augustin. Spatial+: a novel approach spatial confounding. Biometrics, 78(4):1279-12999) 1290, 2022.
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020.
Laura Forastiere, Edoardo M Airoldi, and Fabrizia Mealli. Identification and estimation of treatment and interference effects in observational studies on networks. Journal of the American Statistical Association, 1166534):001-918, 2021.
Andrzej Galecki, Tomasz Burzykowski, Andrzej Galecki, and Tomasz Burzykowski. Linear mixed- effects model. Springer, 2013.
Amanda M Gentzel, Purva Pruthi, and David Jensen. How and why to experimental data to evaluate methods for observational causal inference. In International Conference on Machine Learning, pp. 3660-3671, 2021.
Brian Gilbert, Abhirup Datta, Joan Casey, and Elizabeth L Ogburn. A causal inference framework for spatial confounding, arXiv preprint a1Xiv:2112.14946. 202
Brian Gilbert, Abhirup Datta, and Elizabeth Ogburn. Approaches spatial confounding in geostatis- tics. arXiv preprint urXiv:21 14946, 202
Brian Gilbert, Elizabeth Ogburn, and Abhirup Datta. Consistency of common spatial estimators under spatial confounding. arXiv preprint arXiv:2308.1218 12181, 2023.
Glasmeier, Amy K. Living Wage Calculator. County health rankings & roadmaps. lwingwage.mit.cdh 2020.
Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Paul W Holland. Statistics and causal inference. Journal of the American statistical Association, 81 9396):945-960, 1986.
Michael G Hudgens and M Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):832 842, 2008.
Kosuke Imai, In Song Kim, and Erik H Wang. Matching methods for causal inference with time-series cross-sectional data. American Journal of Political Science, 2021.
Institute for Health Metrics and Evaluation. United states life expectancy and age-specific mortality 2014.
risk by county. http:/ghhw...aatthhaaat...//////
Harry H Kelejian and Ingmar R Prucha. A generalized spatial two-stage least squares procedure for estimating a spatial autoregressive model with autoregressive disturbances. The Journal of Real Estate Finance and Economics, 17:99-121, 1998.
Harry H Kelejian and Ingmar R Prucha. A generalized moments estimator for the autoregressive parameter in spatial model. International economic review, 40(2):509-533, 1999.

---

Published as a conference paper at ICLR 2024
Kori Khan and Candace Berrett. Re-thinking spatial confounding in spatial linear mixed models. arXiv preprint arXiv:2301.05745 2023.
Kori Khan and Catherine A Calder. Restricted spatial regression methods: Implications for inference.
Journal of the American Statistical Association, 117(537):482-444, 2022.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
arXiv preprint (ttiv:6609.0299000 2016.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: research platform for distributed model selection and training. arXiv preprint arXiv: /00.:0/800.5/// 18, 2018.
Yunpu Ma and Volker Tresp. Causal inference under networked interference and intervention policy enhancement. In International Conference on Artificial Intelligence and Statistics, pp. 3700-3708, 2021.
Anthony McGovern and Bill Morris. SttpggiiwwboooooooongnnsssSss_o
Us county level presidential results. 16-11, 2016.
Patrick A. P. Moran. Notes on continuous stochastic phenomena. Biometrika, 37(1/2):17-23, 1950.
Tim P Morris, Ian R White, and Michael J Crowther. Using simulation studies to evaluate statistical methods. Statistics in medicine, 38(1 1):2074-2102, 2019.
Brady Neal, Chin Chin-Wei Huang, and Sunand Raghupathi. Realcause: Realistic causal inference benchmarking. arXiv preprint (0ti::0011.50000 2020.
Lizhen Nie, Mao Ye, Dan Nicolae, et al. Venet and functional targeted regularization for learning causal effects of continuous treatments. In International Conference on Learning Representations, 2020.
Elizabeth L Ogburn and Tyler J VanderWeele. Causal diagrams for interference. Statistical Science, pp. 559-578, 2014.
Georgia Papadogeorgou and Srijata Samanta. Spatial causal inference in the presence of unmeasured confounding and interference. arXiv preprint vrXiv:2303.08211 2023.
Georgia Papadogeorgou, Christine Choirat, and Corwin M Zigler. Adjusting for unmeasured spatial confounding with distance adjusted propensity score matching. Biostatistics, 20(2):256-25 272, 2019.
Georgia Papadogeorgou, Kosuke Imai, Jason Lyall, and Fan Li. Causal inference with spatio- temporal data: estimating the effects airstrikes on insurgent violence in iraq. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(5), 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
Mayana Percira, Allen Kim, Joshua Allen, Kevin White, Juan Lavista Ferres, and Rahul Dodhia. U.S. broadband coverage data set: A differentially private data release. CoRR, abs/2103.14033 2021.
Oregon State University PRISM Climate Group. URR.httpss//rrimmmorrponos Created: 2014-02-04; accessed Jan 2022.
Brian J Reich, James S Hodges, and Vesna Zadnik. Effects of residual smoothing on the posterior of the fixed effects in disease-mapaa mapping models. Biometrics, 62(4): 2006.
Sergio J. Rey and Luc Anselin. PySAL: A Python Library of Spatial Analytical Methods. The Review of Regional Studies, 37(1):5-27, 2007.
David R Roberts, Volker Bahn, Simone Ciuti, Mark S Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, José J Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, et al. Cross- validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecog- raphy, 40(8):913-929, 2017.
Donald B Rubin. Matching to remove bias in observational studies. Biometrics, pp. 183, 1973.
12

---

Published as a conference paper at ICLR 2024
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, (00(469))322-3311 2005.
Havard Rue and Leonhard Held. Gaussian Markov random fields: theory and applications. CRC press, 2005.
Patrick Schwab, Lorenz Linhardt, Stefan Bauer, Joachim M Buhmann, and Walter Karlen. Learning counterfactual representations for estimating individual dose-response curves. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
Julia Stoyanovich and Bill Howe. Nutritional labels for data and models. Quarterly bulletin the Computer Society of the IEEE Technical Committee on Data Engineering, 42(3), 2019.
Mauricio Tec, Natalia Saniga-Garcia Randy B Machemehl, and James G Scott. How likely are ride-share drivers to earn living wage? large-scale spatio-temporal density smoothing with the graph-fused elastic net. arXiv preprint arXiv: 191 1011.000 1.08 106, 2019.
Mauricio Tec, James Scott, and Corwin Zigler. Weather2vec: Representation learning for causal inference with non-local confounding air pollution and climate studies. Proceedings of the 37th AAAI Conference in Artificial Intelligence, 2023.
University of Wisconsin Population Health Institute. County health rankings & roadmaps. 2021.
overcome spatial confounding. Revista Matemática Complutense, 36(2):333-360, 2023. Arantxa Urdangarin, Tomás Goicoa, and María Dolores Ugarte. Evaluating recent methods to
US Census Burcau. 2010 census.
census/decade:2210..m.. 2010.
Victor Veitch, Yixin Wang, and David Blei. Using embeddings correct for unobserved confounding in networks. Advances in Neural Information Processing Systems, 32, 2019.
Washington Post. Police shooting database. http:///wwwwwwwwgooooooooooooonn shootings-database/,, 2020.
Wu, Braun, J Schwartz, and MA Kioumourtzoglou. Evaluating the impact of long-term exposure to fine particulate matter on mortality among the elderly. Science advances, ;(29):eaba5692, 2020.
Omry Yadan. Hydra framework for elegantly configuring complex applications. Github, 2019.
URL https: h/tpt/ggttttttt/// http:///..../ccccooooeeerrr/r//a
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A DATA COLLECTIONS
We provide additional details of each DataCollection See also the Code and API paragraph in the next section describing their documentation in the package repository.
The Social Vulnerability and Welfare collection covers data related to the Social Vulnerability Index (SVI) provided by the Agency for Toxic Substances and Disease Registry (ATSDR) (Centers for Disease Control and Prevention 2020). The SVI is a widely used index provided by the Centers for Disease Control (CDC). This collection is built with census tract-level variables for the state of Texas, USA, and incorporates data on unemployment, racial and ethnic minority status, and disability (US Census Bureau 2010).
The Heat Exposure and Wildfires collection includes information about the monthly weather and climate conditions during the summer of 2020 in California. It covers factors like tempera- ture, wildfire smoke, wind speed, and relative humidity and population density aggregated in Regressions Inde- pendent Slopes Model (PRISM) (PRISM Climate Group), the ambient wildfire smoke PM2.5 model (Childs et al. 2022), and National Oceanic and Atmospheric Administration (NOAA).
the U.S. Census tract level. These are sourced from Parameter-elevatiie
The Air Pollution and Mortality collection incorporates mortality data from respiratory and cardio- vascular diseases among the elderly sourced from the CDC, along with air pollution exposure variables (Di et al., 2019), risk factors from BRESS (CDC) 2010), and Census data (US Census Bureau 2010) covering the mainland US at the county level for 2010.
The Welfare and Elections collection
subset of the US County data on election outcomes, specifically the percentage of the population in each county that voted for the Democratic Party ex expanded with demographics, such as educational attainment and health (University of Wisconsin Population Health Institute] [021 Glasmeier Amy K. Living Wage Calculator 2020 Institute for Health Metrics and Evaluation 2014).
in 2020 (McGovern & Morris 2016).
crime (Washington Post, 2020), and employment statistics (Bureau of Labor Statistics) 2019)
from 2019 and 2020.
The PM2.5 Components collection combines the high resolution total PM2.5 dataset of Di et al. 2019) with the PM2.5 composition data from Amini et al. (2022). both available at high spatial resolution of x 1 km grid. We use the datasets corresponding to the annual average for 2000. The Socioeconomic Status and Broadband Usage collection obtains 2010 zip-code level broadband usage from Pereira et al. (2021) and combines it with socioeconomic status variables from the census (US Census Bureau 2010
B SpaceEnv GENERATION DETAILS
This section explains the training procedure for obtaining the SpaceEnv from DataCollection.
Recall from Section 3 that we learn generative model of outcome and obtain semi-synthetic counterfactuals as e f(X,,a) + R. where / is obtained using AutoMI. and R )sE8 is learned as Gaussian Markov random field matching the spatial distribution the observed residuals in the training data. Below, we describe each step of the environment generation proceedure and the API that a user can use to generate new SpaceEnvs.
Training using AutoMI. The predictor f obtained using auto machine-learning (AutoMI.) techniques implemented with the AutoGluon package in Python (Erickson et al.) 2020). This package trains an ensemble of models and computes a weighted ensemble where the weights are based on the cross-validation performance. Table 5 describes the default settings used for Autogluon. It is important to note that our models have minimal manual tuning, since we aim that training with real data drives the generated synthetic outcomes.
We used special train- validatiinii split since the default random split led to extreme overfitting caused by spatial correlations. Intuitively, spatial correlations create duplicates in the data. As
14

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameters used in AutoML
Parameter
Value
package
AutoGluon v0.7.0
fit.preset.
good_quality
fit.tuning_data
custom with Algorithm false
fit.use_ c.t.use__baghholdotte holduut
fit. fit.time_limit
3600
feature_importance time_ limit
3600
result, random split creates almost identical copies in the train and validation sets. This means the cross-validation procedures will always choose algorithms that overfit more since they would minimize the error on the identical copies in the validations set.
To solve over-fitting due to spatial correlations, our spatially-awar validation splitting algorithm is explained in Algorithm[] using breadth-first search (BFS) obtain a spatially contiguous validation set with buffer. The algorithm relies on specifying number of initial seeds for the validation set obtained with random sampling. expands the validation set with a specified number of BFS neighbors. It removes additional BFS levels from training and validation. Using the default parameters specified in Algorithm we consistently obtain training splits of size 50% 70% and validations splits of size 10% 20%. We recommend checking if adjusting these values is required when using new data collections.
Residuals
Synthetic Residuals
50
50
5
40
40
30
30
-5
-120
-100
-80
120
-100
(a) A comparison of the true and syathetic residuals on the healthe hhinco watnnccoorrraileeeeeee www.mtttaittt. airtality. cont envi- ronment. Coordinates represent latitude and longitude. Missing data is shown in white in the left figure. The synthetic residuals be sampled in missing are regions because the treatment and confounders are available.
Serpcihmesa store of syrthali rasiuulla
xx10
2x 10
10"
10"1
2* 10" 3x 10' to" rasidiils
Smoothness
(b) comparison of spatial smoothness (measured by Moran's D) between the true residuals from AutoML. predictive model and the synthetic residuals from the Gaussian Markov Random Field.
Figure Additional visualizations of the synthetic residuals.
Sampling R using a Gaussian Markov Random Field We sample synthetic residuals as
R ceii MultivariateNormaa00.. \(D pAD))-),
15

---

Published as a conference paper at ICLR 2024
Algorithm 1 Spatially- validation split selection
Input: Graph as map of neighbors - N. where N2 C S is the set of neighbors of 8.
Params: Fraction of seed validation points (default 0.02); number of BFS levels L to include in the validation set (default
1); buffer size B indicating the number of BFS levels leave outside training 1).
validation (default
Output: Set of training nodes T and validation nodes
C
1: # Initialize validation set with seed nodes
2: V WimmpleeWthookkepaeeeeentt
a))
3: # Expand validation with neighhors
for E (0.....
do
5: tmp =
6:
for tmp do V.VUN.
7:
8: end for
9: end for
10: # Compute buffer
11: TB -V
12: for E {0, ..... B 1} do
13: tmp
14: for E tmp do
15: B = BUN,
16: end for
17: end for
18: Exclude buffer for training set
19: T=S
20:
return T.V
where A is the adjacency matrix of the spatial graph; IB is diagonal matrix with the number of neighbors of each location; determines the correlation between an observation and its neighbors, learned from the true residuals (the errors of the predictive model Step I); and is chosen to match the variance of the true residuals exactly. We sample R independently ensure that the synthetic residuals are independent of the treatment, and therefore, by the backdoor criterion (Pearl) 2009). the only possible confounders are those in X. The parameter ý by first computing the vector of the neighhors' mean for each node in the graph and then taking the correlation between the nodes and the neighbor's means. Fig- contains visual example in spatial map of true and generated residuals and a comparison the smoothness of the true and synthetic residuals across all SpaceEnvs, entailing an almost exact match.
Code and API package at/ttts:: anonymous wwtpeennscinnce/eeeeeeee this codebase to generate new SpaceEnvs from existing or new data collections using the steps outlined below. See the repository's documentation for details.
The codebase for generating an enviromment is provided separately from the main
-ateeeedaa--77--- Users can
Each environment is specified by a config file using the Hydra framework (Yadan 2019) and located in the code repository under conf / spaceenv/ as yaml file. The generation process starts with simple command:
python traii_ py spaceen-=<confii_raaee name>
For example, confic figgigg.. Sariiignnmmmmmen name=hea httpeeehhhaaahd_ddnaee ttr_ddnrrrs_mortaaaaiieo
sc corresponds to the con-
fig file illustrated in Fig. 7
is possible to use the config file system to generate transforms of variables (for example, to binarize or take logarithms), change the AutoML engine defaults (see Table 5). etc. The training script will try to download the contents of aatt_pathh (which points to data collection) and graph_ bathhhhhhhhhhh from Harvard Dataverse repository when not found locally. It also allows users to use their own Datavererse repositories. For instance, larger datasets may require extending the t time cime_line limit parameter in Table 3 See the repository documentation for details.
16

---

Published as a conference paper at ICLR 2024
data_collecticn, aar.polllutionnmmertall......
treatment: qd mean pm25
outcome: cdc add__oottaliy__oct
covariate_ novariate_groups:
cs_poverty
racc:
ca hispanic
cs cs_white
[...]
Figure Config file example: cott://aacee//eelaaaaa..pgsssspp disc, yaml.
Proposed easy/hard threshold
Praposed easyhard threshold
40
15-
10
harder
caster
0ostrr
5-
10
spattill soores
confounding scoros
Smoothness Confounding
Binary
Continuous
high high
7
22
low
15
17
low high
7
27
low
15
14
Figure 8: (Top left): Spatial smoothness scores for all covariates in all environments; (Top right): confounding scores for all covariates in all environments; (Bottom): number of datasets in each combination of low and high smoothness and confounding.
Sharing and documentation of collections and environments The SpaceEnvs supported in our Python package are publicly available through our Harvard Dataverse research repository. The metadata for each generated environment contained in their corresponding entry our Harvard Dataverse repository, including the full config file for reproducibility with API explained above. It is also saved property of SpaceEnv object when using the Python package.
Each DataCo 1 lection on documented the package repository and documentation. We include description of the data sources, purpose, exploratory data analysis, and other relevant information. The contents of the documentation is inspired by data nutrition labels (Stoyanovich & Howe (001)). This documentation be accessed (anonymized) at wttttttp///. /eteps:/aanonymous 4open. sci science/r/ space- data-0073
Distribution of spatial and confounding scores Fig. shows the distribution of the spatial smoothness and confounding scores (from the aeposure-respons. function) among all covariates of all environments. The median across all datasets is indicated as vertical red line. The smoothness scores appears to be distributed uniformly across environments, while the confounding scores are heavily skewed.
17

---

Published as a conference paper at ICLR 2024
from spacebench import SpaceEnv
SpaceEnv( "tha wwwgccs_mortalatt..duu
dataset env.make()
print(dataset)
SpaceOataset with a missing spatial confounder:
treatment: (3109,) (binary)
confounders: (3109,
outcome: (3189,)
counterfactuals: (3109, 2)
confounding score of missing: 0.02
spatial smoothness score of missing: 0.11
graph edge
gr graph node coordinales: (3189, 2)
parent SpaceEnv:
WARNING this dataset contains (realistic) synthetic outcome! By using it, you agree understand its Linitations. The variable nanes have about the data.
inferences can made emphasize
(a) Creating benchmark environment.
from spacebench import DatasetEvaluator
results cour_methodddattsett
metrics DataseeEvalaatoo((tttsttt.. ate=results['ae (ate''
erf=results( 'err'), # exposure response curve
counterfactunls=reuutt cconteraccttt J,
keys() contains netrics
dict (lw.eteyyseeeaaaeeeelleeeaaeeeee 'erf_ase", 'ate_se']]
(b) Evaluating the performance of method dataset.
Figure 9: Examples of SpaCE Python package usage.
18

---

Published as a conference paper at ICLR 2024
C SpaCE API: ACCESSING ENVIRONMENTS, MAKING BENCHMARK DATASETS, AND EVALUATING THEM
Recall that to obtain a benchmark dataset for spatial confounding, must 1) create SpaceEny which contains real treatment and confounder data, and a realistic semi-synthetic outcome, 2) create SpaceDataset which masks spatially-varying confounder and facilitates the data loading pipeline for causal inference. Fig. fillustrates the use of the SpacE package. The top panel illustrates how to generate SpaceEnv simply by calling the constructor from the environment name (which takes care of downloading the necessary data) and using the make () method to obtain the SpaceDat D aset (which masks the missing confounder and packages the dataset elements facilitate data loading for causal inference methods). As with any software tool, the API is subject to change based on feedback from users.
D
TUNING AND ADDITIONAL DETAILS OF BENCHMARKS
Implementations of the baselines discussed in Section given in the SpaCE package under the spacebench. algorithma submodule the SpaCE python package. They usage is doc- umented in the package documentation. The benchma r folder in the implementation code contains all the scripts to replicate the reported baselines.
For the experiments, we looped SpaceDatasets, s, 124 in total. We implemented automatic hyper-parameters for the relevant baseline models using the library Ray Tune, thereby minimizing human-induced biases. For all but DAPSm, the tuning metric is implemented as the out-of-sample mean-squared error (MSE) from a validation set obtained with the spatially-conscione splitting method of Algorithm ! After selecting the best hyperparameters, the method was retrained in the full data. One exception is DAPSm, for which the tuning metric was "covariate balance," " as originally proposed by the authors. Notice that test MSE is not metric tailored to causal inference. However, it is reasonable to expect the method that performs less overfitting would also perform better at estimating causal effects in the training data, but this need not always be the case. Unfortunately, no obvious widely applicable hyperparameter selection criteria exist for causal inference. Table summarizes our hyperparameter search space for different baseline models.
all possible SpaceEnvs and all
possible
Model
Iterations
Tuning Metric
Value
GCNN
2,500
Dimension of the hidden layers (hid-
16 or 64
den den_dim)
Number of hidden layers (hidden layers)
or
Weight decay (weight_decay)
le-6 to le-1
2,500
lam_ lam_y lam
loguniform between le-5 and
loguniform between le-5 and 1.0
2,500
loguniform between Ic-5 and
DAPSM
N/A
propensity score lecle_eenaatlyylaeeeeeeeeee value propensity score_penalleeeeeeeeeeeeeeeeeese penalty_tyypp spatial_ weight
Choice among [0.001, 0.01, 0.1. 1.0] Choice between 11 and 12 Uniform between 0.0 and 0.
caliper_tye type
daps
matching algorithm
optimal
Table 6: Hyperparameters tuning for different baseline models. The models are tested with validation set from spatially aware folds as a breadth-first search with 2% of points and their neighbors, resulting in about 10% of the data.
Computing and hardware resources The computations this paper were run on a Mac OS M1 with ten cores in approximately 24 hours. most of the computation driven by training the graph neural network benchmarks.
E LICENSES OF NEW AND REUSED ASSETS
We provide the following free and open source resources:
19

---

Published as a conference paper at ICLR 2024
Resource name
License
Python package Space
MIT
Medel-training software repository space-cata
MIT
Benchmark data collection hosted Harvard Dataverse
CC0 1.0
Table Provided new resources
The DataCol 1 lect i used for generating SpaceEnvs aggregate various data sources. The licenses of these data sources are summarized in Table which allow sharing and reuse for non- commercial purposes.
Data source
Reference
License
Syuthetic Medicare Data for Environmental
httpa:/ /do1 org/10.
CC0 1.0
79 7910/ 11/1//0/
Health Studies
https:
7 WWW nettiieett charts.
NOAA
CCO 1.0
noBa. nosa.gov/Gatc
CDC
RE B.tps::. wwrder. edl yuv/
Public domain
Pattasse.htm
BLtps:77
guv/
Bexxeb/PnServletC...e
Census
Pyλct ivi
Public domain
pyMob: 1oSnapStarts
SeticleID-KCB 4928
kttps:
The Bureau of Labor Statistics (BLS)
7/WNM. bws.gov. oppb/
Puhlic domain
copyright ght Into-nanormmaoo
Open for
https://priem.
PRISM
commercial purposes CC
oregonatate,com www.:aatteeeeeeteeeeeeeeeee /erma///
https: / tithh com/
Police shooting by the Washington Post
washingg.ompest
BY-NC-SA
jata-pollce-shooting
Open Use of Data greement
attps:
US Broadhand Data
7/qithub.com com/ crosoft/
JsBroscband(sagee urcent ages
v1.0
https: http:///edaccciciiiaae ciesin.
Total PM2.5
Public domain
Sich-mm---conccntte lons- contt-coottttooooooo quous-us- 2000--0000022200002-000000-00000
https: http://wwwwoo http:/www.ee..n.mmmmmooo
PM2.5 Components
cclumbas.ddudaaac.
Public domain
pm2 Bc NH1 4 ND3 -0-000----00 804- 804-2030-2209)
Table 8: Major existing reused data resources
F ADDITIONAL EXPERIMENT ANALYSIS
This section contains supporting figures and tables for the experiment results presented in Section
Fig. 10 presents visual outcomes (means and error bars) for various environments. Error bars represent 95% asymptotic normal confidence intervals.
contaiss the results of mixed-effects model (Gatecki et al. (0133) evaluate the interaction between the methods and the confounding and smoothness scores. The dependent variable is the eтrоr in causal effect estimation. Specifically, it the RMISE when estimating the ERF. The independent variables are indicator variables of the spatial confounding methods and their interactions with the smoothness and confounding scores. positive coefficient means more error, and a negative coefficient means less error. The model controls for random effects by SpaceDataset. The I-value statistic indicates statistical significance.
See Section 5 for discussion of the results.
20

---

Published as a conference paper at ICLR 2024
Causal estimate (ERF)
county_dmgrcs cttpiicseeeeeonnnneee disc healthd hihinc_mmonane wwo_mmorrllyyoooo cont county_ _etrity_edcctnn._oo eleeciinn contt dlimate relhuureeemmmmm _ellhum_wttmkkkee cont
0.20
0.15
0.4
0.8
0.3
0.10
0.4
0.2
0.05
0.1
0.00
0.0
dap.sm
genn
gmerris
a2ats Sttiial
spatlar
dapsm
gamn
gis
s2ats lay
spatial+
dapsm
gann
gmerr
spinial
spatiale
dapsm
gemll
gmeeoas
82sts lan
spatiars
22ts=
a mortality disc
cdcavi nohaop poverty_ddie
healthd polluth mortality_com contt diimate w/smoke miirtt_ddiie
healthd dmgrrs
estimao error (ERF)
0.4
0.15
0.05
0.04
0.04
0.02
0.0
0.00
0.00
0.00
dapsm
gom
gmerg de
alss Iag
speuial
Spatial
dapam
gam
gmerrig g
2a statial
spatial
dapam
gam
gmerrgg gi
sall tag
sprual
spatial
dapsm gem
gmerrgi
s2sle-lag lag
spattal
county_phyaane phyacay lifexpey cont
county_
edcsvi nohsdp powertyoeoom ccont
cdcsvi limleng hburdic cont
0.8
Causal seamate
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
dapam
genn
gieergie
spattal+
dapam
genn
gmeergga
s2als-lag]
spatial
spalial+
dapsm
genn
gmerrgga
spatial
spatial+
s2ats-la
s2ets-
Figure 10: Baseline model results by SpaceEnv. Models with lower bars (errors) perform best. The model can be described as
TERM
COEFFICIENT
STD. ERROR
T-VALUE
(INTERCEPT)
0.15
0.07
2.22
SPATIAL
0.02
0.08
0.26
SPATIAL
0.00
0.08
0.06
GONN
0.10
0.09
1.17
DAPSM
0.78
0.19
4.13
GMERROR
0.00
0.08
0.01
00.01
0.08
+0.08
OLS:SMOOHENESS
40.15
0.10
-1.51
SPATIAL:SMOOTHNNES
-0.19
0.10
-1.88
SPATIAL #SMOOTTNESS
-0.09
0.10
0.94
GONN:SMOOTHNESS
-0.21
0.13
-166
DAPSMESMOOTHNESS
-0.80
0.15
-5.9
SMMFRROR:SMOOTHNESSS
-0.15
0.10
-1.33
$2OLSESMO
-0.14
0.10
-1.66
OLS:CONFOUNDING
0.61
0.34
1.77
OPATIAL:CONFOUNNIIII
0.98
0.34
2,86
SPATIAL+:COD CONFOUNDING
1.09
0.34
3.17
GONN:CGNFOUNDING
0.73
0.37
1.96
-2,94
2.38
-1.23
COMFNNINCCNNOONNNIIGG
0.61
0.34
1.78
C2NLO:CONNOUNDING
0.63
0.34
1.83
Table 9: Coefficients of statistical linear model for the error in ERF estimation with random effeets
per dataset and environment and fixed effects by algorithm and environment complexity (smoothness and confounding).
21

---

Published as a conference paper at ICLR 2024
G Q&A
Q: Why use AutoMI. for synthetic data generation instead of causal alternatives such as Meta- learners? Generating valid SpaceEnv does not require that the first step of the AutoML is causal. On the one hand, we need not claim that the counterfactuals of a SpaceEnv are causal in the sense of the original data collection. On the other hand, using existing causal tools (like meta-learners) was challenged by the availability of implementations for continuous treatments and individualized effects that do not rely on specific models. AutoMI. offered several advantages for automatic calibration, ensembling from a large model, etc. Putting everything in the balance, relying on AutoMI. without Meta-learners seemed to achieve better the goals of reducing human intervention in the data generation and avoiding favoring specific models other than others by using large ensembles.
Q: Why inferences about the original data sources not possible if the data is realistic? There are three main reasons that result from the fundamental problem causal inference of not observing counterfactuals in real data. The first one is because do not know if the original data collection includes all relevant confounders; second, as highlighted in the previous answer, the synthetic data generation step need not be causal; third, even if we were using causal methods in the synthetic data generation step, estimation errors and model limitations would mean that we could still not consider the results as reference for the true causal effect, but simply an estimate. For these reasons, we prefer to strongly advice against such interpretation.
Q: What measures are you using to avoid data misuse? Warnings are issued when loading the package and loading environments and datasets in the Python package, shown in Fig. 9 We also include warning statements in the package documentation and Github repositories.
Q: Why not learning the joint distribution of A,Y)?
Sampling from the joint distribution is attractive because it could have the advantage of generating an infinite stream of benchmark datasets from a single data source. However, there could be some limitations with respect to the specific aims of this work. First, we seek maximize the use of real data because we believe that the complexity of real data is hard to capture in model. Sampling from the full joint distribution approach would mean using only simulated data, even if grounded on real data. Second, the single 1-dimensional conditional distribution of (Y X .A) is simpler to model than a complex multivariate distribution. Some flexible models allow us to learn complex multivariate distributions, for example. However, relying single a few generative models could artificially favor some causal inference methods benchmark tasks. Instead, we use an ensemble of diverse predictive models for the 1-dimensional conditional distribution. Last, notice that standard implementations of flexible generative models need not preserve the spatial correlation structure and isolate unobserved confounding (Neal al. 2020). Therefore, while joint modeling approach is promising and warrants future investigation, we feel that using the real observed data for the covariates and treatment aligns better with this project's scope.
22


Reviews:
[{"review_id": "ifhqyEq2Pw", "reviewer": "Reviewer_ykyy", "summary": "The authors describe a framework for evaluating methods for causal inference in spatial data.", "questions": "Why not include various types of effects (see above) besides confounding?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The authors propose an environment for producing realistic benchmark data sets. Opinions may differ about whether this is a significant contribution, but it is certainly an important step toward verifiable research progress in causal inference.\n\nDespite the problems outlined below, this paper is still worth accepting. The paper itself is well-written and detailed, and the work itself provides valuable infrastructure for later research.", "weaknesses": "The decision to defer work on interference is dubious. There are many ways in which the treatments, outcomes, and covariates of neighboring units can interact, including treatment of one unit causing outcome in neighboring units, treatments causing treatments, and outcomes causing outcomes (and even outcomes causing treatments). It would seem reasonable to construct semi-synthetic data generation methods that could, in principle, produce all of these effects.\n\nThe authors produce a variety of data sets that use real treatments and covariates (and synthetic outcomes). Given that real treatments are used, there clearly could be spatial correlations among those variables. Thus, generated data sets may already have some degree of treatment-to-treatment spillover or covariate to (multiple) treatment dependence that produces that spatial dependence among treatment values.\n\nThe authors do not cite a large and recent literature on evaluation methods for causal inference. While this literature is not about spatial confounding specifically, it provides a large number of analogs in the context of non-spatial confounding and it is worth citing. These include:\n\nDorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019). Automated versus Do-It-Yourself Methods for Causal Inference. Statistical Science, 34(1), 43-68.\n\nGentzel, A. M., Pruthi, P., & Jensen, D. (2021, July). How and why to use experimental data to evaluate methods for observational causal inference. In International Conference on Machine Learning (pp. 3660-3671). PMLR.\n\nCheng, L., Guo, R., Moraffah, R., Sheth, P., Candan, K. S., & Liu, H. (2022). Evaluation methods and measures for causal learning algorithms. IEEE Transactions on Artificial Intelligence, 3(6), 924-943.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzD2OfIzdN", "reviewer": "Reviewer_189J", "summary": "The authors introduce SpaCE, to provide benchmark datasets and tools for systematically evaluating causal inference methods designed to alleviate spatial confounding. In SpaCE, each dataset includes training data, true counterfactuals, a spatial graph with coordinates, and smoothness and confounding scores. It also includes realistic semi-synthetic outcomes and counterfactuals, generated using state-of-the-art machine learning ensembles, following best practices for causal inference benchmarks.", "questions": "See the weaknesses.\nIf the authors address my concerns, I will increase the score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper targets a meaningful problem and tries to provide a solution. The structure of the paper is good, and code is provided. The authors provide examples and experiment results to show the effectiveness of the dataset.", "weaknesses": "The reviewer has several concerns that need the authors to address.\n\n1. In equation 1. if R_s is the autocorrelation with neighbors, why do the authors consider only an additive model? What if Y = f(X, a, R)? Does the method fail in the cases of Y = f(X, a, R)? \n\n2. Counterfactual inference/generation typically starts from the abduction step that infers the value or distribution of exogenous variables. However, in causal model (1), it is not clear which letter(s) denotes the exogenous variable(s). \n\n3. What is the relationship between the residual R and the exogenous variable in the generation of Y? If R is the exogenous variable for Y, the authors need to consider a more general causal mechanism, rather than just the additive noise model (ANM). If R is not the exogenous variable, why the authors can use equation 1 for counterfactual inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ul1nE7W8Nf", "reviewer": "Reviewer_oEAN", "summary": "The authors focus on the problem of latent spatial confounding, where a number of latent variables that vary smoothly across space causally affect both treatment and outcome.  Some algorithms have been proposed to account for this.  However, evaluation of these algorithms is hindered by a lack of empirical datasets with known ground truth.  To fill this void, the authors propose a method for generating semi-synthetic data with realistic dynamics and spatial confounding.  The spatial variables can then be removed from the training data to create a dataset with latent spatial confounding.  The authors create sets of empirical data in 6 domains, describe the process of generating the semi-synthetic data, and evaluate the performance of multiple spatially-aware causal discovery algorithms across various data settings.  The entire processing framework is encapsulated in a Python package.", "questions": "No questions", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a well-written paper!  As someone familiar with causal modeling but largely unaware of advances in spatial modeling, this paper does a great job of motivating the problem.  While the contribution (generating semi-synthetic data from realistic treatment, outcome and confounders) isn't novel in and of itself, the combination of the extension to a spatial setting, the curation of six data collections, and the creation of a Python package to support the full pipeline are important enough contributions on their own.  I think this paper has a logical flow and explains each piece well, making it a smooth read.  While I can not speak to the algorithms compared, the hyper-parameters varied (smoothness, level of confounding, and binary vs continuous treatment) provide a nice coverage.  I also appreciate the authors' treatment of future work and ethical considerations.", "weaknesses": "In Section 2, Figure 2 is described as illustrating that \"The closer locations s and s' are, the more correlated X_sMiss and X_s'Miss become.  While Figure 2 does show correlation between two locations s and s', the graphic certainly doesn't show that the correlation increases as s and s' get closer...\n\nGiven how information-dense Figure 4 is, I wish more analysis were provided of the results.  As it stands, there is only a single paragraph in Section 5 that discusses findings in Table 4.  In addition, With table 4 being aggregated across all 6 datasets, we're unable to assess how the performance differs across them.  While Figure 10 in the Appendix does show the breakdown by dataset, there is no discussion of the results.  I understand that space is tight, but I think including a bit more analysis in the paper, or at the very least in the appendix, would help the experimental results feel more impactful.\n\nAt the end of the second paragraph of Section 2, it says \"When a confounder is not unobserved, identification is not guaranteed\" - did you mean to say 'When a confounder is not observed\" or \"unobserved\"?\n\nMinor error: in the second to last paragraph of the Introduction, there's the line \"SpaCE offers Each dataset has a set of known missing confounders [...]\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)