Do causal predictors generalize better to new domains?
Authors: Vivian Yvonne Nastl
Moritz Hardt
Venue: NeurIPS 2024 spotlight (2024)
Keywords: causality
domain generalization
tabular data
Primary area: causal_inference
TL;DR: Predictors using all available features, regardless of causality, have better in-domain and out-of-domain accuracy than predictors using causal features.

Abstract: We study how well machine learning models trained on causal features generalize across domains. We consider 16 prediction tasks on tabular datasets covering applications in health, employment, education, social benefits, and politics. Each dataset comes with multiple domains, allowing us to test how well a model trained in one domain performs in another. For each prediction task, we select features that have a causal influence on the target of prediction. Our goal is to test the hypothesis that models trained on causal features generalize better across domains. Without exception, we find that predictors using all available features, regardless of causality, have better in-domain and out-of-domain accuracy than predictors using causal features. Moreover, even the absolute drop in accuracy from one domain to the other is no better for causal predictors than for models that use all features.  In addition, we show that recent causal machine learning methods for domain generalization do not perform better in our evaluation than standard predictors trained on the set of causal features. Likewise, causal discovery algorithms either fail to run or select causal variables that perform no better than our selection. Extensive robustness checks confirm that our findings are stable under variable misclassification.

Do causal predictors generalize better to new domains?
Vivian Y. Nastl
Max Planck Institute for Intelligent Systems, Tübingen, Germany and Tubingen Al Center Max Planck ETH Center for Learning Systems
vivian nawt weweett@tubbinnn... mpg.ee..pp.eeee...
Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen, Germany and Tübingen Al Center hardt@is mmg..ppg..... de
Abstract
We study how well machine learning models trained on causal features generalize across domains. We consider 16 prediction tasks on tabular datasets covering applications in health, employment, education, social benefits, and politics. Each dataset comes with multiple domains, allowing us test how well model trained in one domain performs in another. For each prediction task, we select features that have causal influence the target of prediction. Our goal is to test the hypothesis that models trained on causal features generalize better across domains. Without exception, find that predictors using all available features, regardless of causality, have better in-domain and out-of-domain accuracy than predictors using causal features. Moreover, even the absolute drop accuracy from one domain to the other is no better for causal predictors than for models that use all features. In addition, we show that recent causal machine learning methods for domain generalization do not perform better in our evaluation than standard predictors trained on the of causal features. Likewise, causal discovery algorithms either fail to run or select causal variables that perform no better than our selection. Extensive robustness checks confirm that our findings are stable under variable misclassification.
Introduction
The accuracy of machine learning models typically drops significantly when model trained in one domain is evaluated in another. This empirical fact is the fruit of numerous studies [Torralba and Efros 2011 Gulrajani and Lopez-Pare 20200 Miller et al. 2021]. But it's less clear what to do about it. Many machine learning researchers see hope causal modeling. Causal relationships, the story goes, reflect stable mechanisms invariant to changes in an environment. Models that utilize these invariant mechanisms should therefore generalize well to domains (Peters al. (017). The idea may be sound in theory. Intriguing theoretical results carve out assumptions under which causal machine learning methods generalize gracefully from one domain to the other (Heinze-Deml et al.)
2018
Meinshausen 2018, Schölkopf et al. 2021 Pearl and Bareinboim, 2022 Subbaswamy et al.
2022
Wang et al. 2022b
These theoretical developments have fueled optimism about the out-of-domain generalization abilities of causal machine learning. The general sentiment is that causal methods enjoy greater external validity than kitchen-sink model fitting. In this work, we put the theorized external validity of causal machine learning to an empirical test in a wide range of concrete datasets.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

All
Arguably causal
Causal
Constant
Same performance
Out-of-domainn accuracy
Shift gap (higher batter)
1.0
0.0
0.8
0.2
0.6
-0.4
ASSISTments Hospital Readmission
Hypertension ICU Length of Stay Utilization
Public AibliCCvveaaee Poverty
Income
Food Sta mps
Diabetes
Vating
College Scorecard
Hospital Mortality
Childhood Lead
Sepsis
Unemployment
ASSISTments
Hospital Readmission
Hypertension
Length of Stay
Utilizaiio
PubllcCoverage
Poverly
Income
Food mps
Diabetes
Vating
College Seorecard
Hospital Mortaliy
Childhood Lead
Sepsis
Unemployment
Tasks
Tasks
Figure I: Best out-of-domain accuracy (left) and corresponding shift gap (right) by feature selection. Predictors based on all features have better out-of-domain accuracy than predictors using causal feature selections. Their shift gap smaller too, up to error bars.
Our results. We consider 16 prediction tasks tabular datasets from prior work [Ding al. 2021 Hardt and Kim 2023 Gardner et al. 2023] covering application settings including health, employment, education, social benefits, and polities. Each datasets comes with different domains intended for research on domain generalization. For each task we conservatively select a set of causal features. Causal features those that most strongly believe have causal influence the target of prediction. We also select more inclusive set of arguably causal variables that include variables that may be considered causal depending performance of machine learning methods trained on causal variables and arguably causal variables with those trained all available features. In all 16 tasks, our primary finding can be summarized as:
modeling choices. For each task, we compare the
Predictors using all available features, regardless of causality, have belter in-domain and out-of-domain accuracy than predictors using causal features.
Across 16 datasets, we were unable to find a single example where causal predictors generalize better to domains than standard machine learning model trained all available features. Figure summarizes the situation. In greater detail, our empirical results are:
Using all features Pareto-dominates both causal selections, with respect in-domain and oft-of-domain accuracy (up to error bars). We provide a closer look at the Pareto-frome frontiers of four representative tasks in Figure
The inclusive selection of arguably causal features Pareto-dominates the conservative selec- tion of causal features, with respect to in-domain and out-of-domain accuracy (up to error bars).
The absolute drop in accuracy from one domain to the other is smaller for all features than for causal features.
Adding anti-causal features-i.e. i... features caused by the target variable-to the set of causal features improves out-of-domain performance.
Special-purpose causal machine learning methods, such as IRM and REx, typically perform within the range of standard models trained on the conservative and inclusive selection of causal features.
Classic causal discovery algorithms, like PC and ICP, do not provide causal parent estimates that improve upon the inclusive selection of causal features.
Extensive robustness checks confirm that our findings are stable under misclassifications of single features.

---

Diabetes
Out-of-domainn accuracy
0.830
0.830
0.828
0.828
0.825
0.825
0.872
0.874
0.876
0.878
-0.050
-0.048
-0.046
-0.044
-0.042
In-domain accuracy
Shift gap
Unemployment
Out-of-domain accuracy
0.960
0.960
0.955
0.955
0.950
0.950
0.966
0.968
0.970
0.972
0.974
-0.018
-0.016
-0.014
-0.012
-0.010
In-domain accuracy
Shift gap
Income
0.800
0.800
Out-of-doman
accuracy
0.750
0.750
0.700
0.700
0.850
0.650
0.600
0.600
0.700
0.750
0.800
-0,800
-0.060
-0.040
-0.020
In-domain accuracy
Shift gap
Hospital Mortality
Out-of-domin do-oomainn
accuracy
0.890
0.890
0.880
0.880
0.870
0.870
0.900
0.920
0.940
0.960
-0.060
-0.050
-0.040
-0.030
-0.020
In-domain accuracy
Shift gap
Arguably causal
Causal
Constant
Diagonal
Figure 2: (Left) Pareto-frontiers of in-domain and out-of-domain accuracy by feature selection. (Right) Pareto-frontiers of shift gap and out-of-domain accuracy by feature selection. Predictors using all features Pareto-dominate predictors using causal features, with respect to in-domain and out-of-domain accuracy. Other tasks are in Appendix
To be sure, our findings don't 't contradict the theory. Rather, they point the fact that the assumptions of existing theoretical work are unlikely to be met in the tabular data settings we study. It is, of course, always possible that those causal prediction techniques yield better results on other datasets. From this perspective, our study suggests that the burden of proof is on proponents of causal techniques to provide real benchmark datasets where these methods succeed. On the many datasets we investigated, it proved infeasible to make use of causal techniques for better out-of-domain generalization.

---

1.1 Related work
Existing work in causal machine learning relies on the assumption of the invariance of causal
mechanisms [Haavelmo] 1944 Aldrich 1989, [oover 1990 Pearl 2009 |Schölkopf et al. [012]. The conditional distribution of the target, given the complete set of its direct causal parents, shall remain identical under interventions on variables other than the target itself. In their influential work, Peterrs al. (2016) utilize this invariance property for causal discovery. further works, is extended to non-linear models Heinze-Demi et al.] 2018), and discovery of invariant features [Rojas Carulla et al. 2018]. To overcome the computational burden in high-dimensional settings, Arjovsky al. [2119] propose Invariant Risk Minimization (IRM), which learns invariant representation of the features instead of selecting individual features. Rosenfeld et al. bb2 however identify major
failure cases of IRM. In response, multiple extensions of IRM have been proposed [Krueger et al.
2021 Wang et al. 2022a Ahuja et al. 2022 Jiang and Veitch 2022 Chen et al. 2023 Another line
of research assumes graphical knowledge remove variables apply independence constraints for
regularization SSubbasammy and Saria 2018 Subbaswamy et al.) [019 Kaur et al. 20022 Salaudeen
and Koyejo 2024). We refer the reader to Kaddour et al.
[022 for overview. Aside from causal
learning approaches, various domain generalization algorithms and distributional robustness methods
have been developed [Ajakan et al. 2015 Sun al.
2013 (Sun and Saenko 2016 Li al. 2018
Levy et al. 2020 Sagawa et al. 2020 Xu et al. 2020 Zhang et al. 202
Each method assumes
unique type of (untestable) invariance across domains.
Gulrajani and Lopez-Paz 2020 conduct extensive experiments on image datasets to compare the
performance of domain generalization algorithms, including the causal methods IRM and Risk Ex-
trapolation (REx) [Kueeerr et al. 2021]. in realistic settings. They find that no domain generalization methods systematically outperforms empirical risk minimization. Recently, Gardner al. 2023 demonstrate a similar behavior for tabular data.
In work, we shift the focus from the out-of-domain performance of specific causal machine learning algorithms to the performance of causal feature feature sets.
1.2 Theoretical background and motivation
To frame our empirical study, we recall some relevant theoretical background first. A domain 'D is composed of samples (Iz, ys) ~ P, where 2, E X c Rº are the features and y E y C R is the target [Wang et al. 2022b]. Let X and Y denote the random variables corresponding the features
and the target.
We are given training domains Dirmin = [Dd 1,. mm). The joint distributions of features and target differ across domains, i.e. pd + pe for d j e. Our goal is to learn a prediction So from the training domains Dirran that achieves minimum prediction error on unseen test domain Diest
0 = arg min E [((Y, fo()))
where ((5-) is some loss function. We can compose the objective into two parts
E Epaio fo())) A,
where A = pmax [e(Y, [o(]] E [e(Y, fo is the shift gap. Hence, we aim to learn classifier with the best trade-off between predicting accurately and having a low shift gap. In our empirical work, we measure the shift gap as the difference in accuracy.
Aace acc( fe, Diest) acc(fo, Ditain).
(3)
Distributional robustness of causal mechanisms. Suppose we have a directed acyclic graph G = (V, E) with nodes V (1,.....) g). random variable (Z, Y) and noise variables E R". A common assumption is that the target described by the prediction so via the coefficient genusal
Y faama (2) + Eq.
The invariance of the causal mechanism implies that these causal coefficients provide the robust estimator for the set of do-interventional distributions on the features [Meinshausen] 2018).
gemual = arg min sup Eq (Y, (Z)), ((db) = =prSh) (a,V\( C (
R-
(5)
9
QeQia)

---

To link to domain generalization, we need to assume that all causal parents of Y are included in the feature set X. We set X Z w.l.o.g. We also presume that the distribution of the testing domain do-intervention on the features, i.c., pres Q(de), Intuitively, this postulates that the causal mechanism generating Y stays the same across domains, while features may encounter arbitrarily large interventions.
Then, the prediction error of the causal coefficients in the test domain is minimax optimal bounded, p |((Y Socna (X)) S min sup Eq le (Y, So (X))]
(6)
QEQ(+)
Recent work in causal machine learning already pointed out that the minimum prediction error on test domains with mild interventions can be much smaller that the prediction error achieved by the causal coefficients [Rothenhäusler et al., 2020 Subbaswamy et al.) 2022]. We conduct synthetic experiments similar to Rothenhääslerr et al. [2220] further supporting the insights that that strong
shift is needed before causal features achieve best out-of-domain accuracy. The details on the setup
and results of the synthetic experiments are provided in Appendix D
Our empirical study complements these theoretical developments, as we evaluate domain generaliza- tion abilities of causal features in typical tabular datasets. We emphasize that we do not challenge the validity of causal theory like (6). but rather challenge how realistic their assumptions are.
Methodology
We conduct experiments on 16 classification tasks with natural domain shifts. They cover applications in multiple application areas, e-g., health, employment, education, social benefits, and politics. Most tasks are derived from the distribution shift benchmark for tabular data TableShift (Gardner et al., 2023]. Some others are from prior work (Hardt and Kim, 2023 All tabular datasets contain interpretable personal information, e.g., age, education status, individual's 's habits. Therefore, we can consult social and biomedical research on the causal relationships between features and target. To reflect existing epistemic uncertainty, we propose pragmatic scheme to classify the relationship between features and target.
We term features that clearly have causal influence on the target causal. We are conservative and only label features as causal when: (1) The feature has almost certainly causal effect on the target, and (2) reverse causation from target to feature is hard to argue. We sort out any spuriously related or possibly anti-causal feature (Schölkopf et al. 2012]. However, we risk excluding relevant causal parents of the target.
The
For this reason, we propose the concept of arguably causal features. is epistemically uncertain how these features are causally linked to the target. To be specific, we term feature arguably causal when it suffices one of the following criteria: (1) The feature is causal feature, or (2) the feature has a causal effect on the target and reverse causation is possible. or (3) it is plausible but not certain that the feature has a causal effect on the target. We exclude variables where is implausible that they affect the target. Ideally, the arguably causal features cover all causal parents present in the dataset. We emphasize that both, causal features and arguably causal features, are merely approximations of the true causal parents based current expert knowledge and restricted to available features. We further note that relationships between causal features and target might be confounded.
In some datasets and tasks we are also confronted with features that are plausibly anti-causal, that is: (1) The target has almost certainly a causal effect on the feature, and (2) a reverse causation from feature to target is hard to argue.
We apply this scheme the features of every task, after seeking advice from current research, governmental institutions and a medical practitioner. We describe the selection procedure for diagnosing diabetes in the following, and give more examples in Appendix A Details the feature selections of all tasks are provided in Appendix D
2.1 Example: Variables in diabetes classification
The task is to classify whether person is diagnosed with diabetes Gardner et al. 2023 The domains are defined by the preferred race of the individuals. We illustrate the feature grouping in Figure 3
5

---

Domain variable: race
Former smoker
Sex
Education
Causal features
Healthy food
Arguably causal features
Obesity
Diabetes
Hyper- tension
Heart disease
Anti-causal features
Figure 3: An example grouping for the task 'Diabetes..
Causal features. Socio-economic status, in particular education level and former smoking, are widely acknowledged risk factors for diabetes [Brown et al.] 20044 Agardh et al., 2011 Madduta (et al. 2017 Centers for Disease Control and Prevention| 2024b). Recent research health care found evidence that individual's sex impacts their diaberes diagnosis, e.g., pregnancies unmask pre-existing metabolic abnormalities in female individuals (Kautzky-Will)
al. 2023 We also
include marital status causal feature, recent research showed that marital stress adversely affects the risk developing diabetes [Whisman et al. 2014].
Arguably causal features. The individual's lifestyle, health and socio-economin status impacts their risk to develop diabetes, i.e., obesity, current smoking, healthy food, alcohol consumption,
physical activities, mental health and utilization of health
services [Lindstrom et al. 2003
Brown et al. 2004 Engum 2007 Baliunas et al.) 2009 Agardh et al. 2011 Madduta et al. 2017
Centers for Disease Control and Prevention| 2024bb Klein al. 2022 Centers for Disease Control
and Prevention 2024a]. At the same time, a person with diabetes is incentivized to improve
their behavior control their blood sugar and improve insulin sensitivity (Klein al., 2004
They are also more at risk to increase their weight due to the insulin therapy [McFarlane 2009
experience distress [Centers for Disease Control and Prevention 2024cc.. and have limited economic opportunities [American Diabetes Association [001]. Because of these bidirectional relationships, we regard features encoding these behaviors arguably causal.
Anti-causal features. Researchers found evidence that diabetes increases the risk of hypertension, high blood cholesterol, coronary heart disease, myocardial infaretion and strokes [Petrie et al.) 2018 Schofield et al. 2016). Due to treatment costs of diabetes, affected individuals are encouraged to obtain a health insurance [National Institute of Diabetes and Digestive and Kidney Diseases 2019 Therefore, we regard the current health care coverage anti-causal to diabetes.
2.2 Tasks and datasets
We consider 16 classification tasks, listed in Table The data is collected from a multitude of sources. We build on 14 classification tasks with natural domain shifts proposed ThifeShift. We use the TableShift Python API to preprocess and transform raw public forms of the data In addition, we conduct experiments on two established classification tasks (MEPS, SIPP). Data preprocessing is adapted from Hardt and Kim [2023). Further details on the tasks and their distribution shifts are in Appendix
2.3 Machine learning algorithms
In our experiments, we evaluate multiple machine learning algorithms. We list them in the following.
'here an active debate causal research whether non- ma--maiipulbble variables like sex are proper causes (Holland 2001 Pearl 2018). acknowledge them as causes in our work. httpss:/tableehifft.om

---

Table 1: Description of tasks, data sources and number of features in each selection. Details and licenses are provided in Appendix E
Task
Data Source
#Features
#Arg, causal
#Causal
#Anti-causal
Food Stamps
ACS
25
12
Income
ACS
23
15
4
3
Public Coverage
ACS
19
16
8
Unemployment
ACS
26
21
11
3
Voting
ANES
36
8
Diabetes
BRESS
25
17
4
Hypertension
BRESS
18
14
5
College Scorecard
ED
118
34
11
ASSISTments
Kaggle
15
13
9
Stay in ICU
MIMIC-ii
7491
1445
5
Hospital Mortality
MIMIC-ii
7491
1445
5
Hospital Readmission
UCI
46
42
5
Childhood Lead
NHANES
6
5
Sepsis
PhysioNet
39
5
Utilization
MEPS
218
129
20
Poverty
SIPP
43
15
Baseline and tabular methods. We include tree ensemble methods: XGBoost (Chen and Guestrin 2016). LightGBM [e et al. [017] and histogram-base GBM. We also evaluate multilayer percep- (MLP) and state-of-the-art deep learning methods for tabular data: SAINT (Somepalli al.
2021
TabTransformer (Huang et al. 2020). NODE [Popov el al. 2019 FT Transformer Gorishniy
al.
2021
and tabular ResNet [Gorishniy et al. 2021].
Domain robustness and generalization methods. We consider distributionally robust optimization (DRO) (Levy et al., [020]. Group DRO (Sagawa et al.] 2020) using domains and labels as groups,
respectively, and the adversarial label robustness method by Zhang al. [2021). We also include
Domain-Adversarial Neural Networks (DANN) [Ajakan et al.
2015]. Deep CORAL [Sun and Saenko
2016), Domain MixUp [u et al. 2020 and MMD [L et
2018
Causal methods. We assess Invariant Risk Minimization (IRM) [Arjovsky al. 2019 Risk
Extrapolation (REx) [Krueger et al.] 2021]. Information Bottleneck IRM (IB (RM) AAhuja et al.
2022 AND-Mask [Parascandolo al. 2021] and CausIRL [Chevalley al. 2022
Domain generalization and causal methods require least two training domains with sufficient number of data points. This is provided in eight of our tasks. Detailed descriptions of the machine learning algorithms and hyperparameter choices are given in Appendix and Gardner et al. 2023
2.4 Experimental procedure
We conduct the following procedure for each task. First, we define up four sets of features based on expert knowledge: all features. causal features, arguably causal features and anti-causal features. Second, we split the full dataset into in-domain set and out-of-domain set. We adopt the choice of domains from Gardner et al. (2023). We have train/testvalliaation split within the in-domain set, and test/validation split within the out-of-domain set. For each feature set:
1. We apply the machine learning methods listed in Section 2.3 For each method:
(a) We conduct hyperparameter sweep using HyperOpt [Bergstra al. 2013 on the in-domain validation data. A method is tuned for 50 trials. We exclusively train on the training set.
(b) The trained classifiers are evaluated on in-domain and out-of-domain test set.
(c) We select the best model according their in-domain validation accuracy. This follows the selection procedure in previous work (e.g., [Gulrajani Lopez-Pat 22220 Gardner To ensure compatibility with TableShift, we add the best in in-domain and
al.
2023

---

Unemployment
-ul-of-domain accuracy
0.960
0.960
0.955
0.955
0.950
0.950
0.966
0.968
0.970
0.972
0.974
-0.018
-0.016
-0.014
-0.012
-0.010
In-domain accuracy
Shift gap
All
Causal
Causal & Anti-causal
Arguably causal
Anti-causal
Constant
Figure 4: (Left) Pareto-frontiers of in-domain and out-of-domain accuracy by feature selection. (Right) Pareto-frontier of shift gap and out-of-domain accuracy accomplished. Adding anti-causal features improves out-of-domail accuracy. Results of other tasks in Appendix C
out-of-domain accuracy pair observed by [Gardner al. analysis to this selection.
2023 We restrict our further
2. We find the Pareto-set P of in-domain and out-of- accuracy pairs. We compute the shift gaps, and find the Pareto-set of shift gap and out-of-domain accuracy of the set P.
provide further details and illustrations of the individual steps Appendix
3 Empirical results
In this section, we present and discuss the results of the experiments on all 16 tasks. A total of 42K
models were trained for the main results and an additional 468K models for robustness tests. Our
code is based on Gardner et al. [2023], Hardt and Kim [023] and Gulrajani and Lopez- Paz (2020]
It is available at httph:: thub. com/ com/social soccssllloooooooooooo www.fouuniiiinnn
causallfeaturr
In experiments, analyze the performance of feature selections based on domain-knows knowledge causal relations. A summary of the results shown in Figure Details on four representative tasks given in Figure The other tasks are in Appendix C The accuracy results are presented along with 95% Clopper-Pearson intervals. They are the baseline for the approximate 95% confidence intervals of the shift gap. See Appendix for the exact computation and justification of the confidence intervals.
In-domain and out-of-domain accuracy. Models trained the whole feature set accomplish the highest in-domain and out-of-domain accuracy, up to error bars (16/16 tasks). The arguably causal features Pareto-dominate the causal features, up error bars (16/16 tasks). Recall that arguably causal features are a superset of the causal features, and have considerably more features (Table Models based causal features often essentially predict the majority label (7/16 tasks).
Shift gap. The shift gap measures the absolute performance drop of the feature sets when employed out-of-domain. All features often experience a significantly smaller shift gap than causal features (7/16 tasks). The causal features solely surpass all features (within the error bounds) for the task 'Hospital Mortality' by predicting the majority label. In most cases, the shift gaps of all features and arguably causal features are indistinguishable (15/16 tasks).
Anti-causal features. In five tasks, we have features that we regard as anti-causal. Results are shown in Figure and Appendix C.2 The anti-causal features not perform significantly different from the constant predictor in-domain (5/5 tasks). However, they sometimes perform extremely poor oft-oa-dommin (2/5 tasks). therefore astounding that the out-of-domain performance of the (arguably) causal features is improved by adding anti-causal features (5/5 tasks).

---

Unemployment
Out-of-domain accuracy
0.960
0.960
0.955
0.955
0.950
0.950
0.966
0.968
0.970
0.972
0.974
-0.018
-0.016
-0.014
-0.012
-0.010
In-domain accuracy
Shift gap
All
Causal
IRM
IB-IRM
AND-Mask
Arguably causal
Constant
REx
Causal IRL
Figure 5: (Left) Pareto-frontiers of in-domain and out-of-domain accuracy causal methods and features selection. (Right) Pareto-frontiers of shift gap and out-of-domain accu- racy attained. The performance the causal methods interpolates between the performance of the causal and/or arguably causal features. Results of remaining tasks are in Appendix C
Causal machine learning methods. We restrict ourselves to the Pareto-set of the standard models for each feature set and compare them to causal methods 3 We showcase a representative performance in Figure 5 Details are in Appendix C.3 The causal methods do not improve upon the arguably causal features trained on standard models (8/8 tasks). In fact, their performance typically spans between the causal features and arguably causal methods trained on standard models. The in-domain and out-of-domain accuracy even indistinguishable from the causal selections multiple cases (IRM: 3/8, REx: 4/8, IB-IRM: 2/8, CausIRL: 5/8, AND-Mask: 5/8 tasks). Possible explanations (1) the causal methods manage to extract a causal representation the features similar to our selection of causal and/or arguably causal features; or (2) it is artifact of having low predictive power.
Causal discovery algorithms. We apply invariant causal prediction (ICP) (Peters et al. 2016 and classic causal discovery algorithm, Peter-Clark (PC) algorithm (Spirtes et al.) 2000] and Fast causal inference (FCI) algorithm (Spirtes et al.] 1995). to tasks. See Appendix C.4 for the results. The algorithms rarely outputs any causal parents (ICP: 1/6, PC: 4/14, FCI: 1/14 tasks). When they do, they select very few features causal parents. Some of them are features we also regard as causal based on domain knowledge, others anti-causal or without causal relations to the target. For example, PC outputs individual's occupation and the number of weeks worked in the last 12 months as causal parents of unemployment. While we agree that the occupation has a causal influence unemployment,
view the amount of weeks an individual worked as result of their unemployment rather than the reason. The causal parents estimated by the causal discovery algorithms often perform similar the causal features though (ICP: 1/1, PC: 1/4, FCI: 1/1 tasks). Note that their performance always bareto-dominated by our arguably causal features (ICP: 1/1, PC: 4/4, FCI: 1/1 tasks). Therefore, whichever feature selection one choose to believe, ours or causal discovery algorithms', one never improves upon the whole feature set.
Robustness tests. Results are in Appendix sensitive misclassifying by removing one feature at a time. The test subsets do not achieve higher out-of-domain accuracy than using all features, with one exception in the task 'ASSISTments", We find that the supersets of arguably causal features with one additional features obtain similar or better out-of-domain accuracy. We randomly sample 500 feature subsets for each task and check whether any subset significantly outperforms the whole feature set. None of the sampled subset does, with few outliers in the task 'ASSISTments'. We consider the divergent task in detail. The task is about predicting whether student answers a question correct. Surprisingly, all outperforming random subsets and the subset from the misclassification tests coincide one regard: missing the feature encoding the tested skill, e.g., rounding. We encourage further work to explain this oddity, as the tested skill of task
We test whether our conclusions feature. Therefore, we form subsets of the set of causal features
and C.6
We refer machine learning methods that are not explicitly causally motivated as standard: baseline methods. tabular methods, domain robustness methods and non-causal domain generalization methods.

---

clearly has a causal influence. We also provide insights into which non-causal features improve the out-of-domain performance, and discuss potential explanations. Our findings remain valid when using balanced accuracy as metric.
4 Discussion and limitations
Our findings may not come as a surprise to everyone. Unlike causal machine learning researchers, social scientists generally see reason to believe in the universality of causal relationships. For example, smaller classroom sizes may cause better teaching outcomes in Tennessce [Mosteller] [195]] but much less in California (Jepsen and Rivkin 2009). Such variation is the rule rather than the exception. Indeed, philosopher of science Cartwright 1199 2007) argued that causal regularities are often more narrowly scoped than commonly held.
Our study mirrors these robust facts in a machine learning context. the many common tabular datasets we consider, we find no evidence that causal predictors have greater external validity than their conventional counterparts. If the goal is to generalize to new domains in these datasets, our findings suggest might as well train the best possible model on all available features. The one exception to the rule we found is the case of the skill variable in the Kaggle ASSISTments task. It appears though removing this variable increases out-of-domain generalization. Curiously, the variable is also almost certainly one of the better examples of a causal variable in our study. Removing it therefore gives advantage causal predictors. For all tasks, we used available research and our own knowledge to classify variables as causal and arguably causal. We likely made some mistakes in this classification. This is why we extended our study with extensive robustness checks that confirm our findings. In addition, we did not find any relief in state-of-the-art causal methods, or causal discovery algorithms.
Demonstrating the utility of causal methods therefore likely requires other benchmark datasets than the ones currently available. We consider this promising avenue for future work that derives further motivation from our work. We point to two classification tasks, where recent research suggests that causal prediction methods have utility for better domain generalization (Schulam and Saria 2017, (Subbaswamy and Saria 2019]: predicting the probability pneumonia mortality outside the
hospitals [Cooper et al.] 1997 Caruana et al. 2015]. and hospital mortality across changes in the
clinical information system [Nestor et al.) 2019). We refer to Appendix A for details. Another direction for future research is evaluate which extent our findings generalize to other applications and data modalities. Recent advances in causal machine learning suggest, for example, promising results in real-world image datasets for classifying wild animals (Terra Incognita) and urban vs. non-urban examples (Spurious PACS), see (Salaudeen and Koyejo 2024].
In light of our results, it's worth finding theoretical explanations for why using all features, regardless of causality, has the best performance in typical tabular datasets. In this vein, Rosenfeld et al. [2021a] point to settings where risk minimization is the right thing to do in theory. We seed the search for additional theoretical explanations with a simple observation: If all domains are positive reweightings of one another, then the Bayes optimal predictor with respect classification error in one domain is also Bayes optimal in any other domain. Standard models, such as gradient boosting or random forests, often achieve near optimal performance on tabular data with a relatively small number of features. In such cases, our simple observation applies and motivates a common sense heuristic: Do the best you can to approximate the optimal predictor on all available features.
Acknowledgments and Disclosure of Funding
We're indebted to Sabrina Doleschel for her help in classifying the arguably causal features in task 'Length of Stay' and 'Hospital Mortality' based on MIMIC-III. We thank André Cruz, Ricardo Bominguez-Olmedee Florian Dorner, Mila Gorecki, Markus Kalisch, Nicolai Meinshausen, and Olawale Salaudeen for invaluable feedback on an earlier version of this paper. In addition, we thank Josh Gardner, Francesco Montagna and Ricardo J. Sandoval for sharing their code with us and being open for follow-up discussions.
This project has received funding from the Max Planck ETH Center for Learning Systems (CLS).
10

---

References
E. Agardh, Allebeck, Hallqvist, T. Moradi, and A. Sidorchuk. Type 2 diabetes incidence and socio-economic position: systematic review and meta-analysis. International Journal of Epidemiology, 40(3):804- 818, 02 2011.
Agency for Healthcare Research and Quality. Medical expenditure panel survey (MEPS), 2019.
K. Ahuja, E. Caballero, D. Zhang, J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish. Invariance principle meets information bottleneck for cutt-f-distributton generalization. 2022. URL http://doi. htt.://....orr//// 484504555 arrvvv/rrvvv... 210.. 06607
H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand. Domain-adversane neural networks. 2015. URL /tttp://doi.ooo org/ 48550/ arXiv 1412. 4446
R. Akee, M. R. Jones, and S. R. Porter. Race matters: Income shares, income inequality, and income mobility for all U.S. races. Working Paper 23733, National Bureau of Economic Research, 2017.
URL http://doi org/10. httg:/...386//100 #386/w23733
J. Aldrich. Autonomy. Oxford Economic Papers, 41(1):15-34 1989.
M. Ameri, L. Schur, M. Adya, S. Bentley, P. McKay, and D. Kruse. The disability employment puzzle: A field experiment on employer hiring behavior. Working Paper 21560, National Bureau of Economic Research, 2015. URL htt.httpp///i..rrg/// wwt.:////.......1/1eeeee....
3386/w21560
American Diabetes Association. Diabetes and employment. Diabetes Care, 34:882-S86, 2011. URL http://doi. otg/ 2001118/111712
American National Election Studies. ANES time series cumulative data file, 1948-2020, 2020.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 2019.
T. Averbuch, M. Mohamed, S. Islam, E. Defilippis, K. Breathett, M. Alkhouli, E. Michos, G. Martin, E. Kontopantelis, M. Mamas, and H. Van Spall. The association between socioeconomic status, sex, race / ethnicity and in-hospital mortality among patients hospitalized for heart failure, Journal of Cardiac Failure, 28(5):697-709, 2022. Uttp://///doi org 10. 1016/j cardfail 2021 09..012
R. S. Baker, S. K. D' D'Mello, M. T. Rodrigo, and A. C. Graesser. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Hluman-Compute Studies, 68(4):223-241, 2010. ISSN http://doi..rg/100.1111 1-5819. URL http:://doo org/10.1016/ j.ijhcs 2009. 003
D. O. Baliunas, B. J. Taylor, H. Irving, M. Roerecke, J. Patra, S. Mohapatra, and Rehm. Alcohol as risk factor for type 2 diabetes: A systematic review and meta-analysis. Diabetes Care, 32(11): 2123-2132, 11 2009, URL httt::///oo..ogg/1022/////60-
C. N. Bell, Thorpe, and T. A. La Race/ethnicity and hypertension: The role of social support. American Journal of Hypertension, 23(5):534 540, 2010. URL http::tttp:/oorrgg//e// 1038/ajh.20 2016//.h.201......
2010.
Bendor, D. Diermeier, and M. Ting. A behavioral model of turnout. The American Political Science Review, 97(2):261-28) 2003. http: org/ stable/31eee310
J. Bergstra, D. Yamins, and D. Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 115-123, Atlanta, Georgia, USA, 2013, PMLR. URL
wtwp:iiinnnnn....ees//e/eee mlr www.ppresssseee v28 wttees//v2bbbeggtraall. www.bbrrgggg..eeeeee.ee httl
D. Blau and I.. M. Kahn. The gender wage gap: Extent, trends, and explanations. Working Paper 21913, National Bureau of Economic Research, 2016. htt:htt::/ //doi. org/10. htrg/1..388///2/ 3386/w21913

---

S. Bonaccio, C. Connelly, I. Gellatly A. Jetha, A. Jetha, and K. A. M. Ginis. The participation of people with disabilities in the workplace across the employment cycle: Employer concerns and research evidence. Journal of Business and Psychology, 35:135-158 2019. URL / //doi.org/10 1007 0ttp:/0011000/100080-0------0000
C. Bosquet and H. G. Overman. Why does birthplace matter so much? Journal of Urban Economics, 110:26-34, 2019, URL
http:////....0g1.111111/1/////// juee 2019.022222 01. 003
A. F. Brown, S. L. Ettmer, J. Piette. M. Weinberger, E. Gregg, M. F. Shapiro, A. J. Karter, M. Safford, B. Waitzfelder, P. A. Prata, and G. L. Beckles. Socioeconomic position and health among persons with diabetes mellitus: A conceptual framework and review of the literature. Epidemiologic 109./ www.ep/mxhhh0000
Reviews, 26(1):633 ::6-77,, 07 2004. httphttt:///oi.oor 101///111111999/
R. E. Bulanda and R. Bulanda. Six: The Problem of Unpaid Parental Leuve, pages 53-62. Policy Press, 2020. htLhttp::/ddet
ch006
Bureau of Economic Analysis. Personal income by state, 2024. URL https:/ /wwp www. bea.gov eata/innome-ee wwwins/eessiiinnnn//li.eeeaeaeel htt.:ppersonaliincoeee. www.-iiiioem---eeee by-state
D. Card. The causal effect of education on earnings. volume of Handbbook of Labor Eco- nomics, pages 1801-1863. Elsevier, 1999. URL http:http//oiii httt:://doi.org/1u/ wttg//..1/////////. 1011111131114144360990 (99) 03011-4
N. Cartwright. The dappled world: study of the boundaries of science. Cambridge University Press, 1999.
N. Cartwright. Hunting causes and using them: Approaches philosophy and economics. Cambridge University Press, 2007.
R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pages 1721-1730. Association for Computing Machinery, 2015. URL http: http:://do... org/ /00 1145/ 2783258 2788613
Centers for Disease Control and Prevention. National health and nutrition examination survey questionnaire, examination protocol, and laboratory protocol (1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, 2017). 2017.
Centers for Disease Control and Prevention. BRFSS Survey Data (2015, 2017, 2019, 2021.
Centers for Disease Control and Prevention. Diabetes risk factors, 2024a. URL httlhtttt//ww..co. http//daabtessriskkkkk
Centers for Disease Control and Prevention. Smoking and diabetes, 2024b. URL httph:://////
cdc. http://attttttiiiaaaaaarrrrrraaa -nd- .html
Centers for Disease Control and Prevention. Diabetes and mental health, 2024c. URL https html
/ //www.cde www.cccc.cc//....... wwtp//iaabetee/lviiin. htop://abttte/iiiiigattttttaaaaa
T. Chen and C. Guestrin. Xghoost: A scalable tree boosting system. In 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data mining, pages 785-794, 2016
Y. Chen, K. Zhou, Y. Bian, B. Xie, B. Wu, Y. Zhang, M. KAILI, H. Yang, P. 7hao, B. Han, and Cheng. Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distributio generalization. In The Eleventh International Conference on Learning Representations, 2023.
Fitp OpSL.
M. Chevalley, C. Bunne, A. Krause, and S. Bauer. Invariant causal mechanisms through distribution matching. 2022. URL http:://oi..or: orgg 411.0/.855///80 arXiv 2206 1.....16660 11646.
Choi and A. Valladares-Esteban. The marriage unemployment gap. The E. Journal of Macroeco- nomics, 18(1 (8())200160000 2018. http://// org/ 1ttp://...1111////1111 1 be/5///ejii jm 2016-0060
12

---

J. Clore, K. Cios, J. DeShazo, and B. Strack. Diabetes 130-US hospitals for years 1999-2008. UCI Machine Learning Repository, 2014. / 24442//552443
G. F. Cooper, C. F. Aliferis, R. Ambrosino, J. Aronis, B. G. Buchanan, R. Caruana, M. J. Fine, C. Gly- mour, G. Gordon, B. H. Hanusa, E. Janosky, C. Meek, T. Mitchell, T. Richardson, and P. Spirtes. An evaluation of Baachine-lcarningg methods for predicting pneumonia mortality, Artificial Intel- ligence in Medicine, 9(2):107-138, 1997. URL 00367-3
(96)
M. M. L. Crepaz. The impact of party polarization and postmaterialism on voter turoout. Euro- pean Journal of Political Research, 18(2):183-200 1990. |ttttt::///oo..orr 10.10..111000111111/111....111
1475-6765 1990. tb00228.x X
G. De Jong and A. B. Madamba. A double disadvantage? Minority group, immigrant status, and underemployment in the united states. Social Science Quarterly, 82(1):117-130, 2001. URL http://doi. org/ 10. 1111/100111111 4941.00011
Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. In Proceedings of the 35th Conference on Neural Information Processing Systems, NeurlPS 2021.
pages 6478 6490. 2021. URL pttps::t///p .nwrrps........ cctppper..ppppepeeeeeeeop fttpppper_flleepppeprrrr Paper pdf
K. S. Dorans, Mills, Y. Liu, and J. He. Trends in prevalence and control of hypertension according to the 2017 American college of cardiology/Ameeriann heart association (acc/aha) guideline. Journal of the American Heart Association, 7(11):c008888 2018. URL http: org/10. 1161/ JAHA.
A. Engum. The role of depression and anxiety in onset of diabetes in a large popsea study, Journal of Psychosomatic Research, 62(1):31-38, 2007. URL http:///6o..org/10..10
jpsychores. 2006. 00006.0...0 07.009
Ermisch and M. Francesconi. The effect of parental employment on child schooling. Journal of Applied Econometrics, 28(5):79--82,, 2013. URL http:ttt///oooorg///11 1002//ae..........e.
M. Feng, N. Heffernan, and K. Koedinger. Addressing the assessment challenge in an intelligent tutoring system that tutors it assesses. The Journal of User Modeling and User-Adapted Interaction, 19:243-266, 2009.
Gardner, Z. Popovic, and L. Schmidt. Benchmarking distribution shift in tabular data with tableshift. In Proceedings of the 37th Conference on Neural Information Processing Syy- tems, NeurIPS 2023, 2023. URL URI httt.httts:/// hattps://prooeedingg. new..ns.r.pp... Paper .pfp
T. A. Gerace, Hollis, J. K. Ockene, and K. Svendsen. Smoking cessation and change in diastolic blood pressure, body weight, and plasma lipids, 1991.
A. L. Goldberger, L. Amaral, L. Glass, J. Hausdorff, P. C. Ivanov, R. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: Components of new research resource for complex physiologic signals. Circulation, 101(23):e215-222) e220, 2000. URL http://doi.org org/ 10. 1161/1111111. cir. 101 e21.
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances Curran Associates,
in Neural Information Processing Systems, volume 34, pages Inc., 2021.
I. Gulrajani and Lopez-Pazz In search of lost domain generalization. 2020. URL http://doi 2rg/10.ar 48550/ 38550//rXiv.
arXiv 20.7.004033 01434
Haavelmo. The probability approach in econometries. Econometrica: Journal of the Econometric Society, pages ii-1115, 1944.
13

---

D. Halvarsson, M. Korpi, and K. Wennberg. Entrepreneurship and income inequality. Journal of Economic Behavior Organization. 145:275-293, 2018. URL http:///// pttg/1..116/// jebo 2017. 11.003
M. Hardt and M. P. Kim. Is your model predicting the past? In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '23, New York. NY, USA, 2023, Association for Computing Machinery. URL http:///oil org/10
1145/3617694.36233
C. Heinze- J. Peters, and N. Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2):20170016 2018.
P. W. Holland. The false linking of race and causality: lessons from standardized testing Race and Society, 2(2):219-233, 2001. URL
org/ 10. 00000666651109965522310000011111
K. D. Hoover. The logic of causal inference: Econometrics and the conditional analysis of causation. Economics & Philosophy, 6(2):207-2344 1990.
X. Huang, A. Khetan, M. Cvitkovic, and Z. Kamin. Tabtransformer: Tabular data modeling using contextual embeddings. 2020. 1
C. Jepsen and Rivkin. Class size reduction and student achievement: The potential tradeoff between teacher quality and class size. Journal of human resources, 44(1):223-250, 2009.
Y. Jiang and V. Veitch. Invariant and transportablle represttttosss for anti-causal domain shifts. 2022. URL attp:://di..rr//
A. E. Johnson, T. J. Pollard, L. Shen, L.-W. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. MIMIC-III, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kaddour, A. Lynch, Q. Liu, M. J. Kusner, and R. Silva. Causal machine learning: A survey and open problems. 2022. URL http::/tto::.o./ org/ 48550/855////8 arXiv 2206 15475
M. Kalisch, Mächler, D. Colombo, M. H. Maathuis, and P. Bühlmann. Causal inference using graphical models with the package pealg. Journal of Statistical Software, 47(11):1-26 2012. URL http::tpp///...o http:////oo....g/1// 10. 11637/ jss. v047. i11
J. S. Kaufmman L. Dolman, D. Rushani, and R. S. Cooper. The contribution of genomic research to explaining racial disparities in cardiovascular disease: A systematic review. American Journal of Epidemiology, 18 4144446644477, 03 2015. URL httpp://d/oo org/ 1093/ 1093//j/e//u33/
J. N. Kaur, E. Kiciman, and A. Sharma. Modeling the data-generating process is necessary for out-of- distribution generalization. 2022. URL http:ttt:://o..rrrg// http://..0.r..100..///0/////.//0
48850/445551111 arxiv. wwt.:xii..22.6000 2783.07837
A. Kautzky-Willer M. Leutner, and J. Harreiter. Sex differennces in type 2 diabetes. Diabetologia, 66 (0),98611002, 2023. URL http::///aa...rgg000.00//////0/0 100.000010000000000000500
023 05891-x
G. Ke. Q. Meng. T. Finley, T. Wang, W. Chen, W. Ma. Q. Ye, and T.-Y. Liu. Lightgbm: highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages URL http::t/tt..oorreee////
Curran Associates Inc., 2017.
50555555553355 32950744
S. Klein, N. Sheard, X. Pi-Sunyer, A. Daly, J. Wylie-Rosett, K. Kulkarni, and N. G. Clark. Weight management through lifestyle moditication for the prevention and management of type 2 diabetes: rationale and strategies. The American journal of clinical nutrition, 80(2):257-263, 2004.
S. Klein, A. Gastaldelli, H. Yki-Järvinen, and P. E. Scherer. Why does obesity cause diabetes? Cell Metabolism, 34(1) (1):1111111)))))11 1000, 2022. http: /ttot:::/ddo.oooo 10. 1016 cmet. 2021. 012
D. Krueger, F. Caballero, J. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the 38th Interna- tional Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
14

---

J. E. Leighley and J. Nagler. Who Votes Now?: Demographics,, Issues, Inequality, and Turnout in the United States. 2014.
B. LLng, Y. Jin, G. Li, Chen, and N. Jin. Socioeconomic status and hypertension: a meta-analysis. Journal of Hypertension, 33(2):21-222,, 2015.
D. Levy, Y. Carmon, J. C. Duchi, and A. Sidford. Large-scale methods for distributionally robust optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8847-8860 Curran Associates, Inc., 2020.
H. Li, S. J. Pan. S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018. URL http://////o org/10 10. 1109/CVPR. 2018. 00566
J. Lindstrom, A. Louheranta, M. Mannelin, M. Rastas, V. Salminen, Eriksson, M. Uusitupa, J. Tuomilehto, and F. D. P. S. Group. The finnish diabetes prevention study (dps) lifestyle intervention and 3-year results on diet and physical activity. Diabetes care, 26(1))3230-3236 2003.
D. Loughran. Why Is Veteran Unemployment So High? RAND Corporation, Santa Monica, CA, 2014.
ort/ wtt.g//....///...../
Madduta, E. -arrro,,, and Evans-Moling Smoking and the risk of type 2 diabetes. Translational research, 184:101-107, 2017. URL http::///o...rr//...ooo. 02.004
trsl 2017
C. M. McEniery, I. B. Wilkinson, and A. P. Avolio. Age, hypertension and arterial function. Clinical and Experimental Pharmacology and Physiology, 34(7):665 671, 2007. URL http://d// org/
10. 1011//1 144.14101000 1681 2007 04657.46. .x
McFarlane. Insulin therapy and type diabetes: management weight gain. Journal of clinical hypertension, 11(10):001-677, 2009. URL http:://oii.orgg//// 10. 1111/j 1751-7176..1. 2009 00063.x
N. Meinshausen. Causality from a distributional robustmess Workshop (DSW), pages 6-10, 2018. URL httt::////// org.0/........... 10.................... 2018. 8439889
robuutttessrrrrrssseseeeoo point of view. In 2018 JEEE Data Science
J. P. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between -istt--iistiiiution and in-distribution generalization. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pruceedings of Machine Learning Research, pages 7721-7735. PMLR, 18-24 Jul 2021. URL /tttps::/ppoceeee v139/miller21b.hr
mlr, mlr.press/
K. T. Mills, A. Stefanescu, and J. He. The global epidemiology of hypertension. Nature reviews, 16 (4):223-237, 2020. URL http: //doi.
F. Montagna, N. Noceti, L. Rosasco, K. Zhang, and F. Locatello. Scalable causal discovery with score matching. van der Schaar, C. Zhang, and D. Janzing, editors, Proceedings of the Second Conference Causal Learning and Reasoning, volume 213 of Proceedings of Machine Learning Research, pages 752-771. PMLR, 2023.
F. Montagna, A. Mastakouri, F. Bulig, N. Noceti, L. Rosasco, D. Janzing, B. Aragam, and F. Locatello. Assumption violations causal discovery and the robustness score matching. Advances in Neural Information Processing Systems, 36, 2024.
Mosteller. The tennessee study class size in the early school grades. The future of children, pages 113-127, 1995.
National Centers for Environmental Information. U.S. Census Divisions, 2024. URI. https // /www.oc ncwi. noaa. httpa//ww.w....oooooooo//////ea/
15

---

National Institute of Diabetes and Digestive and Kidney Diseases, Financial help for dia- betes care, 2019. URL https://www niddk wiw.goo/healle.e http://haatti-iffomaattttiiaat information/dieo www.oatiioniaaaeteetttt.taa
hhtpraallleee---aaeeeeeeeeeeeeee care
B. Nestor, M. B. A. McDermott, W. Boag, G. Berner, T. Naumann, M. C. Hughes, A. Goldenberg, and M. Ghassemi. Feature robustness in non-stationary health records: Caveats to deployable model performance in common clinical machine learning tasks. In F. Doshi-Velle Velez, J. Fackler, K. Jung, D. Kale, R. Ranganath, B. Wallace, and J. Wiens, editors, Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106 of Proceedings of Machine Learning Research, pages 405. PMLR, 2019. URL htttp:://proceeeiiiiiee
mlr. wwwps/ewee/ wwwov//eeeeeeooree htm.
S. Pandey, A. Kalaria, K. D. Jhaveri, S. M. Hermaann and A. S. Kim. Management of hypertension in patients with cancer: challenges and considerations. Clinical Kidney Journal, 16( 16(12)2233-2344) 2023. URL.ttp://dddl
G. Parascandolo, A. Neitz, A. ORVIETO, L. Gresele, and B. Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. URL
https::/openreviier
net/fooum?id hbbsDDDDSL
J. H. Park. The earnings of immigrants in the United States: the effect of English-speaking ability. American Journal of Economics and Sociology, pages 43-56, 1999.
Pearl. Causality. Cambridge University Press, 2009.
Pearl. Does obesity shorten life? is it the soda? On non-manipulable causes. Journal of Causal Inference, 6(2):201 82001, 2018. /ttp:///dd org/10. 1515/jci-2018..1.. 201---2---1120000
J. Pearl and E. Bareinboim. External validity: From do-calculus to transportability across populations. In Probabilistic and causal inference: The works of Judea Pearl, pages 451-482. 2022.
J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: iden- tification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947-0012, 2016.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2017. URL https: htttps://iippress..o tpress mit www/books/////eeeoooooooooo.o Leeeentttccusasee inference
J. R. Petrie, T. J. Guzik, and R. M. Touyz. Diabetes, hypertension, and cardiovascular disease: Clinical insights and vascular mechanisms. The Canadian journal of cardiology, 34(5):575-584, 2018. URL httptttp:
org/ 10. 1016/0 2017. 12. 005
S. Popov, S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. 2019. URL httt::://doi.oooooeoo 100 48850//aXivv.
/ttp://d....0//00.0../686///// 1909 06312
R. Putnam. Bowling Alone: The Collapse and Revival of American Community. Touchstone book.
Simon & Schuster, 2000. httlhhtps////
google.
L. A. Ramirez and J. C. Sullivan. Sex differences in hypertension: Where we have been and where we are going. American Journal of Hypertension, 31(12):1247-1254 2018. hRLht 7/doi.org//0. 1ttp://00.099/////////p/
M. A. Reyna, C. Josef, R. Jeter, S. Shashikumar, B. Moody, M. B. Westover, A. Sharma, S. Nemati, and G. D. Clifford. Early prediction of sepsis from clinical data: The physionet/compputng in cardiology challenge 2019 (version 1.0.0). PhysioNet, 2019. URL htttp://doi.org//t 13026/ v64v-d8577
M. A. Reyna, C. S. Josef, R. Jeter, S. P. Shashikumar, M. B. Westover, S. Nemati, G. D. Clifford, and A. Sharma. Early prediction of sepsis from clinical data: The physionet/computing in cardiology challenge 2019. Critical Care Medicine, 48(2):210-217, 2020. URL http:///oi org/10 1097/CCM. 00000000000001445
C. Rogowski. Flectoral choice, ideological conflict, and political participation. American Journal of Political Science, 58(2):479-494) 2014, URL UtLLhtp:://oooorr//// 1111/ajpp 12059
16

---

M. Rojas-Carulla, Carulla, B. Schölkopf, R. Turner, and J. Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(36):1334 2018. URL http::/jjll org/ papers////// v19/16 16 432. html
Rolland, V. Cevher, M. Kleindessner, Russell, D. Janzing, B. Schölkopf, and Locatello, Score matching enables causal discovery of nonlinear additive noise models. In K. Chaudhuri, S. Jegelka, I.. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18741-18753, PMLR, 2022.
E. Rosenfeld, P. Ravikumar, and A. Risteski. An online learning approach to interpolation and extrapolation in domain generalization. 2021a. URL/http:://oi.orrg// 48550/ 2t..550/aXXiv..110 13128
E. Rosenfeld, P. Ravikumar, and Risteski. The risks of invariant risk minimization. 2021b. URL http://doi.org org/ 10. arwiv. 2010. 2010.05761111111
S. Rosenstone. Economic adversity and voter turmout. American Journal of Political Science, 26 (1):25 46, 1982. http:://w// http://sstoooor//saaee////// brgg/stable/211066
D. Rothenhäusler, N. Meinshausen, Bühlmann, and J. Peters. Anchor regression: heterogeneous data meets causality. 2020. htt. http:///w...55//......... 1801000011.00000909999
Sagawa, P. W. Koh, T. Hashimoto, and Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. 2020. URL
http://doi.cco orgo 10. 48550/4550000 arwiv...iivv.... 1911.
O. Salaudeen and S. Koyejo. Causally inspired regularization enables domain general representations. In International Conference on Artificial Intelligence and Statistics, pages 3132.. PMLR, 2024.
E. Salter. Deciding for a child: comprehensive analysis of the best interest standard. Theor Med Bioeth, 33:179-199 198, 2012. URL http: org/ ftrg/10.11 10.
C. Schoen, K. Davis, DesRoches, K. Donelan, and Blendon. Health insurance markets and income inequality: findings from international health policy survey. Health Policy, 51(2))677885 2000. ISSN 0168-8510. URL CRt http::t/ti..orrg/////
000844-
J. D. Schofield, Y. Liu, P. Rao- Sao-Balakrishna, R. A. Malik, and H. Soran. Diabetes dyslipidemia. Dia- betes therapy, 7(2):003 219, 2016. /tt.a//p///oalororo...0./////000
org/ 10. 10700000301000010000000000
B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on International Conference on Machine Learning, ICML 2012, pages 459 466, 2012. URL http:://oi. org /011000. 5555/
3042573. 3042635
P. Schulam and S. Saria. Reliable decision support using counterfactual models. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: htap::://proceeedngs.eeeee www.nnnuuiiccmm http:/ppprr.lleee/pee/////////// 39922a220.22060011544440000e2223
pdf
B. Schölkopf, F. Locatello, Bauer, N. R. Ke, Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, (ttp://oo...rr//....6)1611221- 2021. URL
http://doi. 111.1.....1111111 11R0//PRRC... 2021. 3058954
A. Sionakidis, I.. McCallum, and S. Padmanabhan. Unravelling the tangled web of hypertension and cancer. Clinical Science, (35(13):1609-15) 2021. URL. http:://di..rgg/ 1042/ CS20200307
Skirbekk, Age and individual productivity: literature survey. Vienna Yearbook of Population Research, 2:133-153, 2004.
17

---

S. Soffer, E. Zimlichman, B. S. Glicksberg, O. Efros, M. A. Levin, R. Freeman, D. L. Reich, and E. Klang. Obesity as a mortality risk factor the medical ward: a case control study, BMC Endocrine Disorders, 22(1):1-8, 2022.
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. 2021. URL
http://doi.. org/ 10. 48550/ 48550/arXiv arwiv. 2106. 01342
R. M. Sondheimer and D. P. Green. Using experiments to estimate the effects of education on voter turnout. American Journal of Political Science, 54(1):174- 189, 2010. HRRLhttp::/// org/
10. 1011/1 111111/.15400-901 1540-59077 2009 004.000222....
P. Spirtes, C.
Meek, and T. Richardson. Causal inference in the presence of latent variables and selection bias. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, UAT955, pages 499-506, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. URL http://doi. org/ 10.5555/2074158.2000 2074215
Spirtes, C. Glymour, and Scheines. Causation, Prediction, and Search. MIT press, 2nd edition, 2000.
B. Strack, J. P. DeShazo, C. Gennings, J. L. Olmo, S. Ventura, K. J. Cios, and J. N. Clore. Impact of hbalc measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed Research International, 2014, 2014.
A. Subbaswamy and S. Saria. Counterfactual normalization: Proactively addressing dataset shift using causal mechanisms. R. Silva, A. Globerson, and A. Globerson, editors, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, volume 2, pages 947-957. Association For Uncertainty in Artificial Intelligence (AUAI), Jan. 2018.
A. Subbaswamy and S. Saria. From development to deployment: dataset shift, causality, and shift- org/10
stable models in health AI. Biostatistics, 21(2):345-352, 11 2019, URL 1.99/bblstaatisticc//eese kxx204
A. Subbaswamy, P. Schulam, and S. Saria. Preventing failures due to dataset shift: Learning predictive httptpp:///ooo.or1110...40///
models that transport. 2019.
1812.04597
A. Subbaswamy, B. Chen, and S. Saria. A unifying causal framework for analyzing dataset shift- stable learning algorithms. Journal of Causal Inference, 10(1):64-88)) 89, 2022. URL http://doi org/10.1 1515/jci 2021-0042
B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. 2016. URL http://doi: org/ 10. (13550/arXiv 1607. 066070077 01719
B. Sun, J. Feng, and K. Saenko. Return of frustratingly easy domain adaptation. 2015. URL http://doi... org/ 10. 18550/arXiss 1511 1511.05547
P. Taubman. Role of parental income in educational attainment. The American Economic Review, 79 (2):57-61, 1989. ISSN 00028282. URL http::/tw.wwjjstttee peatoor.og//see httg//taaee/1880
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEBE, 2011.
U.S. Army. Take charge of your present and future, 2024. URL https://www.gon www.wy.ar...om com/ benefits. html
U.S. Bureau of Labor Statistics. Average weeks unemployed [lnu03008275], 2024. URL https //fred.stlouiste. wttp///eeee//LLL/000222222
U.S. Census Bureau. Survey of income and program participation (SIPP), 2014.
U.S. Census Bureau. American community survey (ACS), 2018.
U.S. Census Bureau. Medical expenditure panel survey insurance component shows 86% of private-sector employees worked for establishments that offered health insurance, 2024. URL.
html costs html
18

---

U.S. Centers for Medicare and Medicaid Services. Medicaid, 2024a. URL www.meddidddd gov/
U.S. Centers for Medicare and Medicaid Services, Medicare, 2024b. URL gov/
U.S. Citizenship and Immigration Services. Should consider U.S. citizenship?
2024. URL https://uww. uscis. http:////eeatiia//.eoooooooooaaa
should- 1- www.oooo-------iiiieeee--
U.S. Department of Education. College scorecard, 2023.
U.S. Department of Labor. Employment and earnings by occupation, 2024. URL https://www dol htt.l/////gggccccee//////aaaaaaa
U.S. General Services Administration. Register to vote, 2024. URL /ttp::tt::/voo
U.S. Office of Foreign Labor Certification. Scasonal jobs, 2024. URL https://seasonalljo
dol. gov/
A. Virdis, C. Giannarelli, M. F. Neves, S. Taddei, and L., Ghiadoni. Cigarette smoking and hyper- tension. Current pharmaceutical design, 16(3):2518-2525 2010. URL http:://////
2174/118161221099262222
M. Walicka, M. Kuzianowska-Kuzzick,,
Kuzmicka, M. Chlebus, A. Sliwczyúski, M. Brzozowska, D. Rutkowski, L. Kania, M. Czech, A. Jacyna, and E. Franek. Relationship between age and in-hospital mortality during 15,345,025 non-surgical hospitalizations. Archives of Medical Science, 17(1):40-66, 2021. URL http:://tt...../ org 10. 01100/011111111 a896/8978
H. Wang, H. Si, B. Li, and H. Zhao. Provable domain generalization via invariant-feature subspace recovery. 2022a. URL http: org 48550/ htt///rXiiiv....vvv 2201 12919
J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng. and P. S. Yu. Generalizing to unseen domains: A survey on domain generalization. 2022b. URL http://doo htattp://do..org/1//.
arXiv. 2103. 03097
S. Wang, M. B. A. McDermott, G. Chauhan, M. Ghassemi, M. Hughes, and T. Naumann. ExtMCcExtract: a data extraction, preprocessing, and representation pipeline for MIMIC-III In Proceedings of the ACM Conference on Health, Inference, and Learning, CHIL '20, pages 222- 235. Association for Computing Machinery, 2020a. URL http:://oo/ org/ 10. 1145/3368555
3384469
S. Wang, M. B. A. McDermott, G. Chauhan, M. Ghassemi, M. C. Hughes, and T. Naumann. MIMIC-Extract: data extraction, preprocessing, and representation pipeline for MIMIC-III In Proceedings of the ACM Conference on Health, Inference, and Learning, ACM CHIL '20. ACM, 2020b. URL http: http:///oo.or///////// 1 3368555. 3384469
M. A. Whisman, A. Li, D. A. Sbarra, and C. L. Raison. Marital quality and diabetes: results from the health and retirement study. Health Psychology, 33(8):832, 2014.
M. Xu, J. Zhang, B. Ni, T. Li, C. Wang, Q. Tian, and W. Zhang. Adversarial domain adaptation with domain mixup. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))650--6509 2020. http:htt://aoo org/ 10. 1609/ 1609/aaai. aaai. v34104. 6123
J. Zhang, A. Menon, A. Veit, S. Bhojanapalli, S. Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. 2021. URL httt:::/aol.org//0..4653. 12230
orw./o. arviv 2010
19

---

Supplemental material
We provide detailed descriptions and results supplement our main paper, as listed in the following.
List of Appendices
Examples
Selection into causal, arguably causal anti cati-causal features
20
2
Outlook: Classification tasks that motivate causal modeling
24
B
Details on experimental procedure, algorithms and compute resources
24
B.
Experimental procedure
24
B.2
Machine learning al algorithms and hyperparameter.
27
B.3
Experiment run details
28
Details on empirical results
29
Main empirical results
29
Anti-causal features)
43
C.3
Causal machine learning methods
45
C.4
Causal discovery algorithms)
................................
48
C.5
Random subsets
53
Ablation of anti-causal and non-causal features
58
Empirical results across machine learning models
64
D
Synthetic experiments
68
E
lasks and data sources
68
E.1
TableShift: ACS
69
TableShift: ANES
71
TableShift: BRESS
73
F.4
TableShifi: HD
74
E.5
TableShift: Kaggle
77
E.6
TableShift: MIMIC
78
E.7
TableShift: NHANES
93
E,8
TableShift: Physionet
.y..............................
93
E.9
TableShift: UCI
................................
94
MEPS]
................................
E.11
SIPP
97
2
Details distribution shifts
99
A
Examples
In this section, explain our feature selection for multiple examples. We also discuss examples of datasets that motivate incorporating causal theory to enhance predictions.
A.1 Selection into causal, arguably causal and anti-causal features
We choose seven representatives tasks to illustrate how we select causal, arguably causal and anti- causal features. These tasks include predicting: economic outcomes (income level and unemploy- ment), political decisions (whether an individual vote in the presidential election), health diagnoses (diabetes, hypertension and hospital mortality) and educational achievements (correctly solved problems in an online learning program). We refer to Section 2.1 for the task 'Diabetes'.
20

---

Werl North cereal
England
Mentae Teame
Last Saur
West South
X0 unit
Figure 6: US Census Divisions. Source: National Centers for Environmental Information 2024
A.1.1 Income
The task to classify whether person has low or high income level [Gardner et al. 2023 The domains defined by the U.S. Census Divisions, with New England being set out-of- We provide a list all features in Appendix(E.1) E.112
|
See Figure 6
Causal features. [Skirbekk 2004) showed that individual's age affects their productivity, that is, their job performance, and therefore, their income. Marginalized groups, such as women and people of color, face discrimination in pursuing higher income, apparent in gender and race wage gaps (Blau and Kahn 2016 AAee et al. 2017]. Therefore, we include the self-reported age, gender and race as causal features. Bosquet and Overman 2019 also found evidence that the place of birth influences an individual's later income.
Arguably causal features. Personal income varies across U.S. states due to different economic situations [Bureau of Economic Analysis 2024). These diverse economic opportunities, on the other hand, lead to internal migration across states. Hence, we regard the individual's current state of residence merely as arguably causal. While educational attainment and the ability to speak English enable nigher-paying jobs [Card 1999 Park 1999]. certain level of income is needed to pay for their annual earnings [U.S. Department of Labor 2024]. The specific work habits, e.g. number of weeks worked and usual hours worked per week, naturally affect the individual's 's carnings and thus income. Then again, the choice of occupation and work habits may stem from certain level of income from other sources, e.g. investments or lack thereof (Halvarsson et al. 2018). Therefore, all work-related features are regarded as arguably causal. Giving birth to child usually leads to short-timed drop in income due to (unpaid) parental leave or child care Bulanda and Bulanda] [2020]. A certain level of income may, however, be a consideration for some people when deciding child [Salter 2012). Citizenship enables individuals for governmental jobs and certain social benefits, alluring people to obtain U.S. citizenship UUS.. Citizenship and Immigration Services) 2024
college tuition or language courses Taubmann 1989 Different occupations differ drastically
Anti-causal features. Insurance purchased directly from an insurance company requires a certain level of income to the regular payments [Schoen et al. 2000). On the other hand, low income is a requirement to be able to apply for Medicaid and other government assistance plans (U.S. Centers for Medicare and Medicaid Services 2024a Hence, view these insurance types features that are anti-causally related to income. A low level of income may necessitate person to look for work, if they haven't 't any yet.
Other features. We don't 't see any obvious direct causal link between a person's income and their marital status. Insurance through an employer or union and Medicare are benefits not tied to income. rather the person's employer or age and medical condition (U.S. Census Bureau, 2024 U.S
Centers for Medicare and Medicaid Services) 2024b).
21

---

A.1.2 Unemployment
The task is to classify whether a person is unemployed [Gardner et al. 2023]. The domains defined by the education level, with no high school diploma being of all features in Appendix E.1.4
We provide a list
Causal features. Some people are unable to work due to their disability. Even if they are capable to work, many employers are still unwilling to (continue to) employ them [Ameri et al. [013 Bonaccio al. 2019]. Hence, we regard features noting the self-reported disabilities as causal. Moreover, immigrant workers also face initial disadvantages in labor force assimilation (De Jong and Madamba 2001]. The immigrant status is encoded as the place of birth in our features. We also view age, sex, race and ancestry as causal. The same arguments apply as in Section A.I.T.
Arguably causal features. Some occupations are mainly seasonal, c.g., working on farms, in landscape or construction (U.S. Office of Foreign Labor Certification 2024], and therefore, may lead to regular short-term unemployment. On the other hand, being unemployed may necessitate an individual to take on training and change their chosen occupation. Being unemployed and looking for work may motivate an individual consider joining the armed forces (U.S. Army 2024]. Later on however, veterans are less likely to be employed due to poor health, employer discrimination, or skill mismatch [Loughran] 2014]. Hence, unemployment and occupation/military service are tangled together in complex way, which is why we regard the feature encoding them as arguably causal. The ability to speak English a requirement some jobs. Conversely, an individual may learn English naturally by interacting with their co- workers. The family situation, described by the marital status and the employment status of the parents, may (indirectly) impact the individual's unemployment, Choi and Setladdrre--sstebaa [2018) show that single workers face higher job losing probabilities than married ones, and multiple studies establish that the employment status of an individual's parents impacts the child's attainments [Taabbman] 1989 Ermisch and Francesconi] 2013). Similar arguments as in Section A.I.I apply for citizenship, current state, mobility status and giving birth to child.
Anti-causal features. As people are unemployed on average for around 15 to 25 weeks in the U.S. [U.S. Bureau of Labor Statistics) 2024), unemployment directly impacts the number of weeks worked during the past 12 months, the usual hours worked per week and whether a person worked last week.
A.1.3 Voting
The task is to classify whether a person voted the U.S presidential election [Gardner et al. 2023 The domains are defined by U.S. Census Regions, with South set as out-of-domane South consists of the U.S. Census divisions West South Central, East South Central and South Atlantic. See Figure We provide list of all features in Appendix E.2.1
Causal features. All states except North Dakota require that a person register before voting in an election U.S. General Services Administration 2024], which is one of our features. Leighley and Nagler 2014] discuss in detail the difference voting behavior between demographic groups, e.g.. defined by age, gender, race/ethnicity, and state. Hence, we again view the demographic features as causal. There is also evidence that education and occupation influence the decision to vote [Sondheimer and Green 2010 Rosenstonel 1982). The current social climate and ideological conflict between competing electoral options also affects voter turnout, encoded by the election year [Rogowski] 2014 Putnam 2000].
Arguably causal features. Participating in politics, being interested in the election, or least being confronted with the election via media may strengthen a person's 's resolve to vote. As deciding to vote can not be explained purely rationally [Bendor et al. 2003). person's 's view on how much influence their vote has naturally impacts their decision to vote. The individual's view is measured by multiple features in our dataset. [repazz [1990) found that polarization of political parties lead to higher voting turnouts. Therefore, person may be more inclined to vote when they like/identify with one party but not the other, or when they have clear preference for one candidate. Diverse features aim to measure these inclinations. [Rosenstone| 1982 showed that economic adversity impacts the voting turnout. Economic problems, e.g., measured by rating of governmental economic policy or current economy, reduce person's 's capacity to attend to politics and hence, participate in the elections.
22

---

Other features. We don't find any obvious causal link between voting and specific party preference, especially within political topics. For example, it unclear in which way preferring the Democrats on the topic of pollution and the environment impacts the decision to vote. Similarly, we don't see any direct causal link between voting and a person's 's political opinion on specific topics, e.g., the importance of gun control, allowing abortion, or defense spending.
A.1.4 Hypertension
The task is to diagnose whether person has hypertension (high blood pressure) [Gardner al. 2023). The domains are defined by the BMI category, with people classified as overweight or obese as out-of-domain P
We provide a list of all features in Appendix F.3.2
Causal Causal features. Aging has a marked effect on the cardiovascular system and hence, increases the risk of hypertension McEnnery et al. 2007]. is also well established that men have higher prevalence of hypertension compared with women (prior to the onset of menopause) [Ramirez and Sullivan 2018 Some researchers attribute this difference women having contact with healthcare systems more freguently [Leng et al., 2015). Prevalence of hypertension also differs across racial/ethnic groups [Bell et al., 2010 Dorans et al. 2018 We thus regard the demographic features age. sex and race as causal. Cigarette smoking is associated with an acute increase in blood pressure, mainly through stimulation of the sympathetic nervous system. Several research studies suggest that increased the risk of hypertension [Mills et al., [2200] Virdis et al.) 2010). and even cessation of chronic smoking does not lower blood pressure [Gerace et al. 1991]. Therefore, we classify former smoking as causal feature. Moreover, patients diagnosed with diabetes are at higher risk to also develop hypertension [Petrie et al. 2018).
Arguably causal features. The individual's current lifestyle affects their risk for hypertension, e.g., obesity, alcohol consumption, smoking, physical inactivity and unhealthy diet [Virdis al. 2010 Mills et al. [020]. At the same time, patients diagnosed with hypertension might cease the harmful behaviors to improve their health. Therefore, we view features encoding these behaviors as arguably causal. Researchers have long established that low socio-economic status, c.g., poverty and employment, increases the risk of hypertension [Leng et al. 2015). When a person indeed develops hypertension, their situation may even worsen.
Anti-causal features. Some researchers regard hypertension as risk factor for the development of certain types of cancer (Sionakidis et al. 2021 Pandey et al. 2023
A.1.5 Hospital mortality
The task is to classify whether ICU patient expires in the hospital during their current visit Gardner eet al. 2023). The domains are defined by the insurance type, with being insures by Medicare being out-of-domain We provide list of all features in Appendix E.6.2
Causal features. Walicka et al. [202]] showed that the in-hospital ron-eud mortality rate significantly increased with age. Averbuch et al. [2022] found evidence that sex and ethnicity are independently associated with the risk of inpatient mortality. They argue that the finding are possibly results from differences in care received in the hospital, e.g., role of bias in assessing medical risk. Therefore, we regard a person's 's age, sex and ethnicity as causal. [Soffer et al.] [2022] discuss the 'obesity paradox', i.e. medical ward patients with severe obesity have a lower risk for mortality compared to patients with normal BMI, measured by a person's 's height and weight upon entering the ICU.
Arguably causal features. We ask a medical practitioner working in an ICU unit for help in selecting the most important vitals that are routinely checked. While they are proxies of the patient's '' health, the selected vitals are also paramount in deciding the medical treatment received and therefore, the risk of in-hospital mortality.
'BMI measures nutritional status adults. is defined as a person's weight in kilograms divided by the square of the person's 's height in meters. Check out the WHO recommendations for more details.
'There is no evidence that racial and ethnic disparities in risk of hypertension explained by genetic factors
though [Kaufman et al. 2015].
'The causal relationship between hypertension and cancer is a prime example for major epistemic uncertainty. They are intricately linked that they inspired their own research field (Pandey et al. 2023
23

---

A.1.6 ASSISTments
The task is to predict whether student solves problem correctly on first attempt in an online learning tool [Gardner et al., 2023]. The domains are different schools. We provide list of all features in Appendix E.5.1
Causal features. We regard all features encoding information of the problem causal. For example, the skill associated with the problem, the type of problem, the number of hints, and how it is framed in the online learning tool. When student asked for a hint or solves a problem in tutor mode, the system automatically marks it as incorrect. Therefore, the target directly depends on the first action of the student, that is, whether a student asks for a hint or an explanation.
Arguably causal features. The system predicts the student's concentration, boredom, confusion and frustration. While research established that learners' aegnitve--fffectvv state influences their performance [Baker et 2010), the system's 's predictions are at best proxies of their true state of
mind.
Other features.
We don't see a clear link the time in milliseconds for the student's responses.
A.2
Outlook: Classification tasks that motivate causal modeling
We highlight two prediction tasks where recent research suggests that they benefit from causal
theory (Subbaswamy and Saria 2019. Schulam and Saria 2017
Cooper et al. 1997] built a predictor for the probability of death for patients with pneumonia. The goal was to identify patients at low risk that can be treated safely at home for pneumonia. Their dataset contains inpatient information from 78 hospitals in 23 U.S. States. Cooper et al. [1997] assumed that hospital- -treated pncumonia patients with very low probability of death would also have a very low probability of death if treated at home. Caruana et al. (2015) pointed out that this assumption may not hold, for example, in patients with a history of asthma. Due to the existing policy across hospitals to admit asthmatic pneumonia patients to the ICU, the aggressive treatment actually lowered their mortality risk from pneumonia compared to the general population. While Caruanal jet al.] 2013) use this observation to argue for interpretable models, (Schulmm and Saria [017]] take it as motivation for causal models ensure generalization.
In another example, Nestor et al. [2099] trained predictive models on records from the MIMIC-III database between 2001 and 2002, and tested on data of subsequent years. When the underlying clinical information system changed in 2008, this caused fundamental changes in the recorded measurements and significant drop in prediction quality of machine learning models trained raw data. The predictive performance, however, remained surprisingly robust after aggregating the raw features into expert-defined clinical concepts If these clinical concepts reflect causal relationships, this example may be viewed as empirical support for causal modeling.
B Details on experimental procedure, algorithms and compute resources
We go into the details of the experimental procedure, including information on the confidence intervals, in Appendix [..]] We describe the machine learning methods and their hyperparameter in Appendix B.2. Experiment run details and compute resources are provided in Appendix B.3
B.1
Experimental procedure
In this section, we go into the details of the experimental procedure and visualize the steps for three selected tasks.
First, we define up to four sets of features based on domain knowledge and common sense: all features, causal features, arguably causal features and anti-causal features. The sets of features are provided for each task in Appendix E Exemplary explanations are provided for seven tasks. See Section 2.1 and Appendix A
Second, we split the full dataset into in-domain set and out-of-domain set. We have a train/test/val- idation split within the in in-domiin set, and a test/validation split within the oft-o--domain set. We
Various measurements of the same biophysical quantity grouped together.
24

---

Income
PublicCoverage
Hypertension
0,800
Out-of-domain accuracy
Out-of-domain
Out-of-domain accuracy
0.600
0.600
0.600
0.500
0.400
0.400
0,400
0.600
0.800
0.600
0.700
0.800
0,400
0.600
In-domain accuracy
In-domain accuracy
In-domain accuracy
All
Arguably causal
Causal
Constant
Diagonal
Figure 7: In-domain and out-of-domain performances of all trained classifier.
adapt the split choices from TableShift, that 80%/10%//10% split in-domain and 90%/10% split out-of-domain. For each feature set:
1. We apply the machine learning methods listed in Section 2.3 For each method:
(a) We conduct a hyperparameter sweep using HyperOpt [Bersttra et al., 2013). We exclusively train on the training set, and use the in-domain validation accuracy for hyperparameter tuning. A method is tuned for trials. Note that we withhold the out-domain validation set from training, i.e., domain adaption is not possible.
(b) We evaluate the trained classifiers using accuracy on in-domain and out-of- test set. In total, we train 3- 50 1, 800 classifier for tasks with one training domain, and 3 23 50 = 3, 450 classifiers for tasks with at least two training domains. See Figure
(c) We select the best model according their in in-domain validation accuracy. This follows the selection procedure in Gulrajani and Lopez- (2020) and Gardner et al. (2023). See Figure To ensure compatibility, we add the best in-domain and out-of-domain accuracy pair observed by Gardner et al. 2023). See Figure 9 We restrict our further analysis to this selection.
2. We find the Pareto-set of in-domain and out-of-domain accuracy pairs. See Figure [0 We
dismiss Pareto dominant classifiers whose in-domain accuracy is smaller than the constant prediction, i.e., predict worse than the majority prediction in-domain. This is the final selection of classifiers. We also add the Pareto-dominated region. See Figure IT We compute the shift gaps, and find the Pareto-set of shift gap and out-of-domain accuracy of the set P.
Note that do not use the out-of-domain validation set in our experiments. It is left for further analysis, e.g., measuring the shift between in-domain and out-of-domain [Gardner et al. 2023
Error bounds. We use 95% Clopper-Pearson confidence intervals for accuracy. They attain the nominal coverage level in an exact sense. We approximate 95% confidence intervals for the shift gap. the difference between in-domain and out-of-domain accuracy. The Clopper-Pearee Pearson confidence intervals for the accuracy approximately equal the normal version for large sample sizes n,
[lCP, uCP] ~ Ip 20.9 20.9750, P+ 20.9750]
with 2a the a-quantile the standard normal distribution and a ((1D) the standard error.
We can immediately infer confidence intervals for the shift gap, as the in-domain and out-of-domain test sets are independent. The variance of the shift gap is the sum of the variance of in-domain and
25

---

Income
PublicCoverage
Hypertension
0,800
0.600
Out-of-domain accuracy
Out-of-domain accuracy
Out-af-domain accuracy
0.700
0.500
0.600
0.400
0.600
0.700
0.800
0.600
0.700
0.800
0.600
0.650
In-domain accuracy
In-domain accuracy
In-domain accuracy
All
Arguably causal
Causal
Constant
Diagonal
Figure 8: In-domain and out-of-domain performances of trained classifier with best in in-validation accuracy within model class.
Income
PublicCoverage
Hypertension
Out-of-dommin
0.800
Out-of-domain accuracy
Out-of-domain accuracy
0.600
accuracy
0.600
0.700
0.500
0.600
0.400
0.600 0,700
0.800
0.600
0.700
0.800
0.600
0.650
In-domain accuracy
In-domain accuracy
In-domain accuracy
All
Arguably causal
Causal
Constant
Diagonal
Figure 9: In-domain and out-of-domain performances of trained classifier with best in-validation accuracy within a model class, as well as results obtained by Gardner et al. (2023
out-of-domain accuracy. We therefore compute approximate confidence intervals as
(Icpiindomaia
Pin-domain) ) (ICP, (lcP.oot-of-dommill offoooott----mmin Pout-of-domain )2
UCCP in-dmmiinn
Pin domain))) )2 + (UCP,out-on WWCCC.uttooff..cccoc out-of-domain
Pout of drmain )
26

---

Income
PublicCoverage
Hypertension
Out-of-domain accuracy
0,800
Out-of-domain accuracy
Out-af-domain accuracy
0.622
0.700
0.600
HIN
0.400
0.600
0.700
0.800
0.600
0.700
0.800
0.600
0.650
In-domain accuracy
In-domain accuracy
In-domain accuracy
All
Arguably causal
Causal
Constant
Diagonal
Figure 10: In-domain and out-of-domain performances of Pareto dominant classifier.
Income
PublicCoverage
Hypertension
0.700
Out-of-domain
0.800
Out-of-domain accuracy
Out-of-domaiin accuracy
accuracy
0.600
0.620
0.700
0.500
0.600
0.400
HBH
0,700
0.800
0.780
0,800
0.600
0.650
In-domain acouracy
In-domain accuracy
In-domain accuracy
All
Arguably causal
Causal
Constant
Diagonal
Figure 11: In-domain and out-of-domain performances of the final selection classifiers, i.e. Pareto dominant classifier whose in-domain accuracy is better than the constant prediction. Dotted lines indicate Pareto frontiers and shaded areas the Pareto-dominated sets.
B.2 Machine learning algorithms and hyperparameters
We describe the causal methods, and refer the reader to Gardner et al. 2023] for the other machine learning algorithms. All the causal methods are motivated by causal theory and seek to find some form of invariance across multiple training domains.
Invariant Risk Minimization (IRM). IRM [Ahuja et al. 2022 modifies the training objective learn feature representation such that the optimal linear classifier that maps the representation to the target the same across domains.
Risk Extrapolation (REx). REx [Krueger et al. 202 seeks to reduce variances in risk across training domains, in order to gain robustness to distributional shifts.
Information Bottleneck IRM (IB-IRM). IB-IRM Ahuja et al. 2022 augments IRM with an information bottleneck constraint. The constraint resolves some issues of IRM.
Causal Invariant Representation Learning (Causal IRL). Causal IRL [Cheaalley et al. 2022 proposes regularizer that enforces invariance through distribution matching. We train both versions of the algorithm, i.e. with CORAL and MMD.
ANDMask. ANDMask Parascandolo et 2021 an algorithm based on the logical AND.
aims to focus on invariances and prevents memorization.
27

---

Table 2: Hyperparameter grids of causal methods
Model
Hyperparameter
Values
IRM
IRM À
LogUniform(1e 1, 1e5)
IRM Penalty Anneal Iters
LogUniform(1, le4)
REx
REx À
LogUniform(1e: 1e5)
REx Penalty Anneal Iters
LogUniformm1,,1 1e4)
IB-IRM
IRM À
LogUniform((e 1c5)
IRM Penalty Anneal Iters
Log 1e4)
IB À
Log LogUniforrm(1e 1e5)
IB Penalty Anneal Iters
Log Uniform((1 1e4)
CausIRL ANDMask
MMD 7
Uniform(1e, le1)
ANDMask r
Uniform(0.5, 1)
Table 3: Summary trained and evaluated models. We train 23 models for tasks with least training domains, and 12 models for tasks with one training domain. Main results include the models trained on causal, arguably causal and all features.
# #Task
#Models
# #Trials
Total
Main results
3x8
23
50
27,600
3x8
12
50
14,400
Anti-causal
2
23
50
2,300
3
12
50
1,800
Causal discovery
4
23
50
4,600
12
600
Misclassification tests
100
23
50
115,000
92
12
50
55,200
Random subset tests
14x500
4
10
280,000
Ablation study
234
10
9,360
Total
510,860
The hyperparameters are chosen with HyperOpt [Bergstra al. 2013). ter grid for the causal methods in Table The hyperparameter grids are adapted from Gardner et al. 2023 and Gulrajani and Lopez-Paz 2020
We provide the hyperparame-
B.3
Experiment run details
All experiments were run as jobs submitted to a centralized cluster, running the open-source HT- Condor scheduler. Each job was given the same computing resources: CPU. Compute nodes use AMD EPYC 7662 64-core CPUs. Memory was allocated required for each task: all jobs were allocated least 128GB of RAM; for the tasks 'Public Coverage' jobs were allocated 384GB of RAM. An experiment job accounts for training and evaluating single model for a given tasks and feature selection. Jobs were terminated when runtime exceeded hours.
We typically train and evaluate 600 models per task with single training domain and 1,150 models per task with at least training domains, for each feature selection. A total of 42K models were trained for the main results, and 468K models for additional results and robustness tests. We detail the number trained models in Table Preliminary experiments merely required negligible amount of compute, and were run on a local computer.
We the implementation of HyperOpt [Bergstra al. 2013 in the TableShifi API sample from the hyperparameter space of the model. Detailed descriptions and hyperparameter choices are found in Appendix B.2 and Gardner al. (2023).
provide the complete code base to replicate our experiments under https: /tte:://gtttttt com/ Dociaalfonmdtiionnccuuull----eee
28

---

C Details on empirical results
In this section, we provide figures and details our empirical results. First, we provide the main empirical results in Appendix C.1 and results on anti-causal features in Appendix C.2 Then, we describe and show results from the causal machine learning algorithms and causal discovery methods in Appendix CB3 and Appendix C.4 respectively. We give details and results on the robustness test involving random subsets in Appendix C.3 We follow with an ablation study in Appendix C.6 Last, we add details on the different machine learning models to the main results in Appendix C.7
C.1 Main empirical results
We show main results for all tasks Figures 12 27
More precisely, we provide: (1) the Pareto- frontiers of in on-domain and ofttof--omain accuracy by feature selection; (2) the Pareto-frontier of shift gap and out-of-domain accuracy by feature selection; (3) Pareto-fronticn of in-domain and out-of-dor of-domain balanced accuracy by feature selection; and (4) out-of- accuracies and shift gaps obtained by robustness tests for causal and arguably causal features.
All
Arguably causal
Causal
Constant
Diagonal
Belanced out aul-of-domain accuracy
Out-of-domain accuracy
0.820
Out-of-domminn accuracy
0.700
0.800
0.800
0.780
0.780
0.500
0.820
0.840
0.030
0,500
0.600 0.700
In-domain accuracy
gap
Balanced accuracy
(a) Mareto-frontien, by feature selection.
0.000
Out-of-domain
-0.010
0.800
Shift
H-
0.780
-0.030
All
Causal
Test
Test
Test
Test
Teat
Test
Test 8
Test 10
Conslant
All
Causal
Test
Text
Teat
Text3
Teat4
Teats
Teat6
Test
Teat
Test
Teat 11
Consiant
Test
0.000
Out-of-domain
accuracy
-0.010
0.800
Shift
-0.020
-0.030
All
Arg. causal
Test
Test
Test
Constent
Arg. causal
Text
Test
Teet
Constant
(b) Robustness tests for causal features (upper) and arguably causal features (lower).
Figure 12: Food Stamps
29

---

All
Arguably causal
Causal
Constant
Diagonal
Balenced out- of--mf--omain accuracy
0,800
0.800
0.800
Out-of-omminn accuracy
Out-of-domain accurace accuracy
0,700
0.600
0.650
0.600
0.500
0.700
0.800
-0.075-0.650-0.22
0.600
0.800
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto prccto-frontirs by feature selection.
0,000
Out-of-domain
-0.025
Shift
-0.050
-0.075
Caussl
Test
Test 2
Test
Consiant
All
Causal
Test
Test
Test
Test
0.000
Out-of-domain
0.800
H
-0.025
0.700
Shift
-0.050
.0.075
0.600
Arg. csusal
Test
Test
Test2
Test
Test5
Test
Test
Constent
Test
Test
Test
Test
Test4
Test 5
Test6
Test
Conslant
Arg.
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 13: Income
30

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain accuracy
0.700
Out-of-domaiin accuracy
But-of-domain accuracy
0.700
0.600
0.700
0.500
0.600
0,600
0.400
0.500
0.775
0,800
-0.100
0,000
0,500
0,600
0.700
In-domain accuracy
Shift gap
Balanced In-domain accur
(a) preett-foonterss by feature selection.
0.100
Out-of-domann
accuracy
Shift gap
0.600
0.000
0.400
-0.100
Causal
Test
Test
Test 2
Test
Test
Test
Test 7
Constant
Causal
Test
Test
Test
Teat
Test
Teat
Test 7
Consiant
Out-of-domainn
accuracy
Shift gap
0.000
0.600
-0.100
0.400
All
Arg. causal
Test
Tost
Constant
All
Arg. causal
Test
Test
Test
Constant
(b) Robustness tests for causal features (upper) and arguably causal features (lower).
Figure 14: Public Coverage
31

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced oft-oaf-dmain
Out-of-domain accuracy
0.960
Out-of-domain accurag acy
0.960
accuracy
0.800
0.955
0.955
0.950
0.600
0.965
0.970
-0.020
-0.015
-0.010
0.600
0.800
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto Prccto-frontirs by feature selection.
0.000
Out-of-domain
0.960
Shift gap
-0.010
1
0.940
-0.020
All
Caussi
Test
Test
Test
Test
Test
Test
Test 8
Test
Test 8
Teal 10
Conslant
All
Causol
Test
Test
Test2
Test
Testd
Test5
Test6
Test
Test
Test
Test 11
Constent
leat
0.000
Out-of-domain
0.960
0.950
Shift
0.940
-0.020
All
Ar. coussa
Test
Test
Test
Test4
Conslant
Test
Test
Test
Test
Test
Constant
Arg.
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 15; Unemployment
32

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain
Out-of-domain accuracy
0,800
Out-of-domain accuracy
0.800
accuracy
0.700
0.700
0.600
0.600
0.500
0.700
0.800
0.900
-0.100
-0055
0.000
0.600
0,800
In-domain accuracy
Shift gap
Balanced accuracy
(a) Prccto-rronters by feature selection.
0.000
Out-of-domain
0.800
-0.050
Shift
0,700
-.100
0.800
Caussl
Test
Test
Test2
Teat
Test
Test
Test
Test
Constent
All
Test
Test
Test
Test
Test
Test 5
Test
Test
Constant
0.000
Out-of-domain
0.800
Shift
-0.100
0.600
Arg cause
Test
1851
lest
Test
Test 10
195t
1881 ?
Test
Test
T861
Constant
Test D
Test
1651
18515
16519
16518
Test
Teat
[esl
lest
881
lest
Constant
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 16: Voting
33

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain accuracy
0.650
0.600
Out-of-domain accuracy
Out-of-domaiin accuracy
0.600
0.500
0.500
0.550
0.500
0,700
0.800
0.900
-0.500
-0.250
0.500
0.750
In-domain accuracy
Shift gap
Balanced In-domain accur
(a) preett-foonterss by feature selection.
Out-of-domain
accuracy
0.750
Shift gap
0.000
0.500
-0.250
-0.500
Test
Test
Test
Test 3
Test T
Test 6
Teat 7
Test
Constant
Causal
Test
Test
Test 2
Test
Test
Test
Test 6
Test
Test
Constant
0.000
Out-of-domain accuracy
0.600
Shift gap
-0.200
0.400
All
Arg. causal
Test
Test
All
Arg. causal
Test
Consfant
(b) Robustness tests for causal features (upper) and arguably causal features (lower).
Figure 17: ASSISTments
34

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain acturacy
Rut-of-domain accuracy
Out-of-domainn
D 0.620
accuracy
0.620
0.600
0,550
0.580
0.500
0.600
0 850
050
-0
025
0.500
0.600
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto-frontiers by feature selection.
0.200
Out-of-domain
0.620
Shift gap
0.100
0.600
0.000
All
Causal
Text
Test
Test 2
Teat
Test
Condant
All
Tost
Test
Toat
Test 3
Test 4
Conslant
Out-of-domain
0.050
0.620
Shift
0.600
0.000
.0.050
Arg. causal
Test 2
Arg. causal
Test 0
Test
Consant
Test
(b) Robustness for causal features (upper) and arguably causal features (lower).
Figure 18: Hypertension
35

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out- ouf-af-domain accuracy
0.835
0.835
0.580
Out-af-domiin accuracy
Out-af-domain accuracy
0.833
0.540
0. 830
0.830
828
0.828
0.520
0.825
0.825
0.500
0.872 0.875.. 877
-0.050
-0.045
0.500
0.525
0.560
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto prccto-roonters by feature selection.
0.835
0,000
Out-of-domain
0.830
-0.020
0.825
Shift
-0 040
Caussl
Test
Tesl 2
Test
Consiant
All
Test
Test
Test
Test
0.000
Out-of-domain
0.830
0.825
Shift
0.820
-0.044 0..000
Test
Test
Test
T
Test Test
Ng. cousal
Test
Test
Test
Test4
Test
Constant
Arg.
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 19: Diabetes
36

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain accuracy
0.900
0.900
Out-of-domainn accuracy
Out-of-domainn accuracy
0.800
0,800
0.800
0,600
0.700
0.700
0.900
0.950
-0.200
-0.100
0.500
0.750
In-domain accuracy
Shift gap
Balanced In-domain accur
(a) prccto-frontirs by feature selection.
Out-of-domain
0.900
0.000
accuracy
Shift gap
0.800
-0.100
0.700
-0.200
All
Causal
Test
Test
Test
Test
Test4
Test
Test
Test
Test
Test
Test 10 Constant
All
Causal
Test
Test
Test
Test
Test4
Test
Test
Test
Test
Test
Test 10
Consiant
(b) Robustness tests for causal features.
Figure 20: College Scorecard
37

---

All
Arguably causal
Causal
Constant
Diagonal
Belanced out-of-domain accuracy
Out-of-domaii accuracy
Out-of-domainn accuraac
0.600
0.550
0.500
0.600
0.650
-0.100
0.055
0.500
0.600
accuracy
gap
Balanced accuracy
aa) Pareto-frontien. by feature selection.
Out-of-domain
0.000
0.550
Shift
-0.050
0.500
All
Causal
Test
Teat
Teat.
Test
Constant
All
Causal
Teat
Test
Test
Teat
Test
Constant
0.000
Out-of-domain
0.600
-0.025
Shift
-0.050
0.550
0.500
Arg. causal
Teat
Test
Test
Constant
AlI
Test
Teat
Test
Test
Constant
(b) Robustness tests for causal features (upper) and arguably causal features (lower).
Figure 21: Hospital Readmission
38

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out of co--domann accuracy
Out-of-domainn accuracy
0.650
Out-of-domain accuracy
0,650
0.650
0.600
0.600
0.550
0.550
0.550
0.500
0.600
-0.07 220.0.500.022
0.500
0.600
0.700
In-domain accuracy
Shift gap
Balanced In-domain accurar
(a) Pareto-frontiers by feature selection.
0.000
Out-of-domain accuracy
Shift gap
0.600
-0.050
0.550
All Causal
Test 0
Tost 1
Test 2
Test
Trest 4
Constant
All
Causal
Test
Test 1
Test 2
Test
Tost 4
Constant
(b) Robustness tests for causal features.
Figure 22: Stay in ICU
All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of- domaim accuracy
0.600
Out-of-domain acciracy
Out-of-domain accuracy
0.890
0.890
0.880
0.880
0.550
0.870
0.870
0.500
0.900
0.950
-0.050
-0.025
0.500
0.600
0.700
In-domain accuracy
Shift gap
Balanced In-domain accurac
(a) Proettefsoosierss by feature selection.
0.000
Out-of-domainn
accuracy
0.890
Shift gap
-0.025
0.880
-0.050
0.870
All
Causal
Test 0
Test 1
Test
3
Test 4
Constant
All
Causal
T
Tost
Test
3
Tost
Constant
(b) Robustness tests for causal features.
Figure 23: Hospital Mortality
39

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out- off-maf-omainn accuracy
0.925
Out-of-domain accuracy
Out-af-domainn accuracy
0.520
0.920
0.920
0.500
0.915
0.915
0
0.960
0.970
0.960
-0.060
-0.060
-0.040
0.475
0,500
0.525
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto prccto-frontirs by feature selection.
0,000
Out-of-domain
0.925
0.920
Shift
-0.040
-0.060
Causal
Test
Tesl 2
Test
Consiant
All
Text
Test
Test
Test
0.000
Out-of-domain
-0.020
Shift
0.915
-0.040
0.910
0.0000
All
Arg. causell
Conslant
Test
Constant
Arg.
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 24: Childhood lead
40

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domain accuracy
0.600
Out-of-domainn accuracy
0,928
Out-of-domainn accuracy
0.928
0.926
0.926
0.550
0.924
0.924
0.500
0.987
0.988
0.988
-0.065
-0.060
0.500
0.600
In-domain accuracy
Shift gap
Balanced In-domain accur
(a) Pareto- prectt-frontiers by feature selection.
0.000
Out-of-domaiin accuracy
0.925
Shift gap
-0.025
0.920
-0.050
0.915
All
Causal
Test 0
Tost 1
Test 2
Tost 3
Test 4
Constant
All
Gausal
Test 1
Test 2
Test 3
Tost 4
Constant
Test
(b) Robustness tests for causal features. No robustness test for arguably causal features there is only one additional features.
Figure 25: Sepsis
Arguably causal
Causal
Constant
Diagonal
Balanced out-of of-domaain accuracy
Out-of-domaiin accuracy
0.700
Out-of-domaan accuracy
0.700
0,700
0.600
0.600
0.600
0.500
0.600
0.800
-0.100
0.000
0.600
0,800
In-domain accuracy
Shift gap
Balanced In-domain accura:
(a) froett-fso by feature selection.
0.000
Out-of-domain accuracy
0,700
Shift gap
-0.550
0.600
-0.100
Constant
(b) Robustness tests for causal features.
Figure 26: Utilization
41

---

All
Arguably causal
Causal
Constant
Diagonal
Balanced out-of-domainn accuracy
0,800
Oul-af-domain accuracy
Out-af-domainn
0. 0,700
accuracy
0.700
0.800
0.600
0.500
0.700
0.800
0,900
-0.200
-0.100
0.600
0.800
In-domain accuracy
Shift gap
Balanced accuracy
(a) Pareto-frontiers by feature selection.
0.800
Out-of-dmmain
Shift gap
0.700
0.000
0.600
-0.200
All
Causall
Test
Test 1
Tesl
Test
Test
Test
Test
Test
Test 10
lest 11
Teet 12
Test 13
Test 14
Condant
Causal
Test
Test 1
Tesl
16813
1est
1661
Test
Te8l
Test
Test
Teat
Test
Teal 14
Consiant
0.800
0.000
Out-of-domain
0.700
Shift gap
-0.100
0.600
-0.200
All
Arg. causal
Test 0
Test
Test 3
Test 4
Test 5
Conslart
Arg. cansal
Test
Test
Test
Test 5
Constant
(b) Robustness tests causal features (upper) and arguably causal features (lower).
Figure 27: Poverty
42

---

C.2 Anti-causal features
We have five tasks in which some features are plausibly anti-causal: 'Income', 'Unemployment" 'Diabetes', "Hypertension' and 'Poverty'. Figures and and show the Pareto frontiers of anti-causal feature in comparison the causal selections and all features. We also show the Pareto frontiers we achieve by training on arguably causal and anti-causal features.
Income
Out-ol-domain aocuracy
0.800
Io-of-domminn accuracy
0.800
0.750
0.700
0.700
0.650
0.600
0.800
0.700
0.750
0.800
-0.080
-0.060
-0.040
-0.020
In-domain accuracy
Shift gap
Unemployment
Out-of-domain
0,960
Out-of-domain
0.980
accuracy
accuracy
0.955
0.955
0.950
0.960
0.966
0.966
0.970
0.972
0.974
-0.020
-0.017
-0.015
-0.012
-0.010
In-domain accuracy
Shift gap
Diabetes
0.835
0.835
Out-ol-domainn
Dut-of-domain
0.833
Bocuracy
accuracy
0.830
0.830
0.828
0.828
0.825
0.825
0.872
0.874
0.876
0.878
-0,550
-0.048
-0.048
-0.044
-0.042
In-domain accuracy
Shift gap
Arguably causal
anti-causal
Constant
Figure 28: (Left) Pareto-frontiers of in-domain and out-of-domain accuracy of anti-causal features in comparison to causal features sets and all features. (Right) Pareto-frontiers of shift gap and out-of- domain accuracy attained.
43

---

Hypertension
Out-of-domain
0.600
-of-of-domain
0.600
accuracy
accuracy
0.550
0.550
0.500
0.450
0.450
0.600
0.625
0.850
0.675
-0.150
-0.100
-0.050
0.000
In-domain accuracy
Shift gap
Poverty
0.800
0.800
Du-af--omainn
accuracy
0.700
-l-oa--dmmain accuracy
0.700
0.600
0.600
0.500
0.500
0.700
0.750
0.800
0.850
0,900
-0.300
-0.200
-0.100
In-domain accuracy
Shift gap
Arguably causal
Causal
Anti-causal
Causal Anti-causel
Constant
Figure 29: (Left) Pareto-fromd frontiers of in in-domain and out-of-domain accuracy anti-causal features in comparison to causal features sets and all features. (Right) Pareto prectt-fronsiers of shift gap and out-of- domain accuracy attained. (Continued)
44

---

C.3
Causal machine learning methods
We evaluate five causal methods: Invariant Risk Minimization (IRM) [Arjovsky al. 2019 Risk
Extrapolation (REx) Krueger et al. [202T]. Information Bottelneck IRM (IB-IRM) Ahuja et al. 2022] Causal IRL based CORAL and MMD [Chevalley et al. 2022] and AND-Mask Parascandolo et al.
2021 A description and the hyperparameter grids are given in Appendix B
The causal methods require least testing domains with each sufficient amount of data. Eight of our task satisfy these requirements: 'Food Stamps', 'Income', 'Unemployment', 'Voting', 'College Scorecard', 'Hospital Readmission', 'Hospital Mortality' and 'Length of Stay'. Note that the task 'ASSISTment is not included. It has multiple training domains but very few data point in some of them.
We provide results in Figure 31 and 32 The bar plot in Figure 30 summarized how often the performance is: (i) smaller than the performance of the causal features, (ii) similar to the performance of causal features, (iii) between the performances the causal features and arguably causal features, and (iv) similar to the performance of the arguably causal features. Note that the causal machine learning algorithm never outperform the arguably causal features.
causal features
causal
causal fealures and arguably causal features
arguably
REX
IB-IRM
Causal IRL
AND-Mask
Figure 30: Performance causal methods in comparison to causal and arguably causal features. Summary for the 8 tasks with multiple training domains.
45

---

Food Stamps
Ood accuracy
0.820
Ood accuracy
0.820
0.800
0.800
0.780
0.780
0.810
0.820
0.830
0.840
0.850
-0.055
-0.030
-0.025
-0.020 -0.015
accuracy
Shift gap
Income
0.800
Ood accuracy
Ood accuracy
0.700
0.600
0.600
0.700
0.750
0.800
-0.880
-0.060
-0.040
-0.020
0.000
accuracy
Shift gap
Unemployment
Ood accuracy
0.960
Ood accuracy
0.960
0.955
0.955
0.950
0.950
0.968
0.970
0.972
0.974
-0.020
-0.018
-0.016
-0.012 -0.010
Id accuracy
Shift gap
Voting
0.900
Ood accuracy
0.800
Qod accuracy
0.800
0.700
0.700
0.600
0.600
0.700
0.750
0.800
0.850
0.900
-0.125 -0.100 -0.075 -0.050 -0.025 0.000
accuracy
Shift gap
Causal
IB-RMM
AND-Mask
Arguably causal
Constant
REx
Causal IRL
Figure 31: (Left) Pareto-frontiers of in-domain and out-on- accuracy of causal methods and knwwlldge features selection. (Right) Pareto-frontiers shift gap and out-of-domain accuracy attained.
46

---

College Scorecard
0.900
0.900
Ood accuracy
accuracy
0.800
0.700
0.700
0.860
0.880
0.900
0.920
0.940
0.960
-0.250
-0.200
-0.150
-0.100
-0.050
accuracy
Shift gap
Hospital Readmission
Ood accuracy
Ood accuracy
0.600
0.550
0.550
0.500
0.500
0.575
0.600
0.625
0.650
0.675
-0.100
-0.080
-0.060
-0.040
-0.020
accuracy
Shift gap
Hospital Mortality
0.900
Ood accuracy
accuracy
0.890
0.880
0.870
0.900
0.920
0.940
0.960
-0,060
-0.050
-0.040
-0.030
-0.020
accuracy
Shift gap
ICU Length of Stay
Ood accuracy
0.650
accuracy
0.650
0.600
0.600
0.550
0.600
0.650
0.700
-0.080
-0.060
-0.040
-0.020
accuracy
Shift gap
Causal Constant
IRM
IB-IRM Causal
AND-Mask
Arguably causal
Figure 32: (Left) Pareto-frontiers of in-domain and out-of-domain accuracy of causal methods and Nooainnkkowweede features selection. (Right) Pareto-frontiers of shift gap and out-of-domain accuracy attained. (Continued)
47

---

C.4 Causal discovery algorithms
We consider invariant causal prediction (ICP) |Peters et al. 2016] and classic causal discovery methods. In addition, we contemplated score matching methods. When benchmarked to other causal discovery methods, they showed surprising robustness settings where assumptions on the data may be violated [Montagna et al., [024]. We first describe results of causal discovery methods we analyzed in our experiments. Then, we explain the challenges we encountered with the score matching methods and the reason we didn't 't include them in our final analysis.
We list the evaluated causal discovery methods and describe their results.
Invariant Causal Prediction (ICP). ICP [Peters et al. 2016) collects all subset of features that show invariance in their predictive accuracy across domains, and outputs valid confidence intervals for the causal relationships. The variables with effect significantly different from zero are the causal predictors under sufficient assumptions. ICP requires at least two training domains, each needs sufficient amount of data. In addition, the number of features needs to be of reasonable size. This is provided in tasks: 'Food Stamps', 'Income', 'Unemployment', 'Voting', 'College Scorecard' and 'Hospital Readmission'. We use the boosting implementation from the R package InvariantCausalPrediiee by Peters et al. (0016) We choose a confidence level of a = 0.05; is the default setting.
Peter-Clatt Cherk algorithm (PC). The PC algorithm [Spirtes al. 2000 is classical causal discovery method. It is based on conditional independence testing and estimates a completed partially directed acyclic graph (CPDAG), We use the implementation from the R package 'pcalg' by Kalisch et al. [2012) with the default confidence level of = 0.01. We do not consider the tasks 'Hospital Mortality' and 'Stay in ICU" due to computational costs.
Fast causal inference algorithm (FCI). The FCI algorithm (Spirtes et al. 1995 is a generalization of the PC algorithm. It allows arbitrarily many latent and selection variables. It outputs partial ancestral graph (PAG). We use the implementation from the R package 'pcalg' by |Kllishh et al. [2012) with confidence level of 0.01. We do not consider the tasks 'Hospital Mortality' and "Stay in ICU' due to computational costs.
Standard conditional independence tests assume common data type, that is, either binary, discrete or Gaussian. Our datasets are however a mix of binary, categorical and continuous variables. We decide to bin the continuous variables into five categories, and then discrete independence test in the PC and FCI algorithm. Note that ICP is applied to standard preprocessed data, that is, binary, one-hot encoded categorical and continuous data.
We run the causal discovery algorithms on the in-domain validation set. If the algorithm outputs any causal parents, train the machine learning methods listed in Section 2.3 the training set. We tuned each method for 50 trials. We provide the results in Table 4 and [ Estimated causal parents are denoted in square brackets. A task is rejected from ICP in Table 4 no subset of variables leads to invariance across the domains. We showcase an example of a CPADAG from the PC algorithm in Figure 33 An example of PAG from the FCI algorithm is given in Figure 34
We provide the performance of all estimated causal parents Figure 35 and 36
Score matching methods and compute memory. We considered the implemen- tation of the score matching methods provided by |Montagna et al.) (2024]. See https://github. http:///ggtuuu..opy..oo..
The algorithms compute matrix of size [sample size x sample size x features]. This is computationally infeasible when using the whole in-domain validation set. For example, this requires 2,88 TiB of memory for 'Income' and 3.09 TiB of memory for "Unemployment" Note that the validation sample sizes are merely 121,154 and 158,015, respectively.
A solution is to randomly sample, say, 1,000 data points from the validation set We perform preliminary experiments to assess this approach. We sample 1,000 data points, and run the SCORE Rolland et al. 2022) and Discovery At Scale (DAS) algorithm by [Montagna et al. 2023 for the main tasks "Diabetes', 'Income" and 'Unemploymen'.
'This is the largest number of sample size in the analysis of Montagna al. 2024
48

---

Table 4: Summary of empirical results for invariant causal prediction (ICP) with 0.05. The descriptions of features are given in Appendix E
Task
#Features
Has 2 2 training domains
ICP
with sufficient sample size
Food Stamps
28
rejected
Income
23
rejected
Public Coverage
19
applicable
Unemployment
16
[RELP, JRKK
Voting
54
no causal predictors not applicable applicable rejected not applicable not applicable not applicable rejected not applicable not applicable not applicable not applicable
Diabetes
25
Hypertension
18
College Scorecard
118
ASSISTments
15
Stay in ICU
7491
Hospital Mortality
7491
Hospital Readmission
46
Childhood Lead
Sepsis
40
Utilization
218
Poverty
54
ST
DIVISONN
MIG
NATIVTY
ANC
DEAR
DEVE
DREM
DPWY
OCCP
WRk
FER
MAR
POBP
RELP
ENG
Figure 33: CPDAG estimated by the PC algorithm for the task 'Unemployment'. The target denotes the employment states. The descriptions features are given in Appendix E
The computation of the SCORE algorithm fails for all tasks. One computation step during the pruning does not converge after a few pruning steps.
We obtain results from the DAS algorithm. We provide the estimated DAG from DAS under the filenames das diabctes.svg. das income.svg and daa_ unemployment..svg
https://github. ceperiments_caile wtt__aaasalaadddoooaaaa htt.oooorreuuuceeuuaaaaaaaaaeeee diwcoddcsscccrcy//ooo
The algorithm doesn't i estimate any causal parents for being diagnosed with diabetes ('Diabetes') having a certain income level ('Income"). The only causal parent output for being unemployed is being born in South Dakota ("Unemployment') While the results are intriguing, they hardly promise supreme prediction outcomes. Therefore, we didn't 't pursue the score matching methods further. We however encourage future research checking their performance future research is to find more sophisticated solutions to reduce the required memory amount.
all tasks. Another path for
49

---

Table 5: Summary of empirical results for the PC and FCI algorithm (Spirtes et al.
1995 2000
with
0.01. The descriptions of features are given in Appendix
Task
#Features
PC
FCI
Food Stamps
28
[HUPAC.PUBCOV]
no causal parents
Income
23
[WKHPAAGEPHINSI OCCCP)
causal parents
Public Coverage
19
no causal parents
no causal parents
Unemployment
26
[OCCP,WRK] по causal parents
no causal parents
Voting
54
causal parents
Diabetes
25
[HIGH BLOOD PRESS] no causal parents
HIGH BLOOD BLOOD_PRESS]
Hypertension
18
causal parents
College Scorecard
118
no causal parents no causal parents
no causal parents
ASSISTment
15
causal parents
Stay in ICU
7491
not applicable not applicable no causal parents no causal parents no causal parents no causal parents no causal parents
not applicable not applicable no causal parents causal parents no causal parents causal parents no causal parents
Hospital Mortality
7491
Hospital Readmission
46
Childhood Lead
7
Sepsis
40
Utilization
218
Poverty
54
WKCHPP
WKW
ST
MAR
DIVISION
POBP
ANC
NATNITY
DEAR
DEVE
DREM
DPHY
CIT
Figure 34: PAG estimated by the FCI algorithm for the task 'Unemployment'. The target denotes the employment states. The descriptions of features are given Appendix E
50

---

Unemplayment
Out-of-domain accuracy
0.960
0.955
0.950
0,966
0.968
0.970
0.972
0.974
In-domain accuracy
All
Arguably causal
Causal
Constant
PC
Diagonal
(a) PC algorithm the task 'Unemployment',
Unemplayment
Out-of-domain Rocuracy
0.960
0.955
0.950
0.966
0.968
0,970
0.972
0.974
In-domain accuracy
Arguably causal
Causal
Constant
Diagonal
(b) ICP algorithm the task 'Unemployment'
Food Stamps
0.820
Out-of-domainn
accuracy
0.800
0.780
0.810
0.820
0.830
0.840
0.850
In-domain accuracy
All
Arguably causal
Causal
Constant
PC
Diagonal
(c) PC algorithm on the task 'Food Stamps'.
Figure 35: Performance of causal parents selected by causal discovery algorithm, in comparison to domain knowledge selected causal features and whole feature set.
51

---

Income
0,800
Outof-domain accuracy
0.700
0.650
0.680
0.700
0.720
0.740
0.760
0.780
0.800
0.820
0.840
In-domain accuracy
Arguably causal
Causal
Constant
Diagonal
(a) PC algorithm the task 'Income',
Diabetes
0.835
Out-of-domain accuracy
0.833
0.830
0.828
0.825
0.872
0.874
0.876
0.878
In-domain accuracy
All
Arguably causal
Causal
Constant
PC
Diagonal
(b) PC algorithm on the task 'Diabetes'.
Diabetes
0.835
Out-of-domain
0.833
aocuracy
0.830
0.828
0.872
0.874
0.876
0.878
In-domain accuracy
All
Arguably causal
Causal
Constant
FCI
Diagonal
(c) FCI algorithm on the task 'Diabetes'.
Figure 36: Performance of causal parents selected by causal discovery algorithm, in comparison to domain knowledge selected causal features and whole feature set. (Continued)
52

---

C.5
Random subsets
We test whether there exists subset of features that achieve significantly higher out-of-domain accuracy than the whole feature set. It is however computational infeasible to evaluate all possible subsets of the features for our tasks. For example, the task 'Income' with 23 features has already million subsets.
We randomly sample 500 subsets for each task, with exception to 'Hospital Mortality' and 'Stay in ICU". We don't t think that 500 random sample are informative for 'Hospital Mortality' and 'Stay in ICU", as it is just teeny fraction of the power set of features (27491 subsets).
Due to computational cost, we further restrict our analysis to the models XGBoost, LightGBM, FT Transformer and SAINT. These models achieve the highest average out-of-domait accuracy across tasks. See Appendix C.7 and Gardner et al. 2023]. The methods are also tuned for 10 trials, instead of 50.
We provide the results in Figure 37 38 and 40 Except for some subsets in the task 'ASSISTment ments',
none of our random subsets outperforms the full feature set, not in in-domain accuracy nor in out-of- domain accuracy. In the task 'ASSISTments' we predict whether question is correct answered by student in online learning tool. The exception occurs when removing feature 'skill id', encoding the type of skill required. The distribution of the feature 'skill shifts significantly across schools, that is, from training schools to out-of-domain testing schools.
53

---

Food Stamps
Out-of-domail accuracy
0.820
0.800
0.780
0.810
0.820
0.830
0.840
0.850
In-demain accuracy
Income
Out-of Out-of-domainn domaann accuracy
0.700
0.600
0.880
0,700
0.720
0.740
0.760
0.780
0.800
0.820
0.840
In-domain accuracy
PublicCoverage
0.700
Out-af- Out-of-domaln fo--dmminn
0.600
0.400
0.775
0.780
0.785
0.790
0.795
0.800
0.805
0.810
0.815
In-domain accuracy
Unemployment
Out-of-domain
0.960
0.955
0.950
0.966
In-domain accuracy
All
Arguably causal
Causal
Constant
Random
Diagonal
Figure 37: Performance of random subsets in comparison to causal feature selection and the whole feature set.
54

---

Voting
Out-of-domainn
0.800
0.800
0.700
0.750
0.800
0.850
0.900
In-domain accuracy
ASSISTments
Out-af-domaa
accuracy
0.800
0.600
0.400
0,700
0.750
0,800
0.850
0.900
0,950
In-domain accuracy
Hypertension
Out-of-domainn
0.800
0.500
0.600
0.620
0. 3.640
0.660
0.680
In-domain accuracy
Diabetes
0.835
Out-of-domain
accuracy
0.830
0.825
0.872
0.874
0.876
0.878
0.880
In-demain accuracy
Arguably causal
Causal
Constant
Random
Diagonal
Figure 38: Performance of random subsets in comparison to causal feature selection and the whole feature set. (Continued)
55

---

College Scorecard
0.900
Out-of-domaln
0.800
0.700
0.880
0.900
0.920
0.940
0.960
In-domain in accuracy
Hospital Readmission
Out-of-domain
0.800
0.500
0.400
0.400
0.450
0.500
0,550
0.600
0.650
0.700
In-domain accuracy
Childhood Lead
0.925
Out-of-domaln
0.920
0.915
0.960
0.965
0.970 In-domain accuracy
0.975
0.980
Sepsis
Out-of-domain
0.928
0.926
0.924
0.987
0.988
0.988
0.988
0.988
0.988
0.989
In-domain accuracy
Arguably bly causal
Causal
Constant
Random
Diagonal
Figure 39: Performance of random subsets in comparison to causal feature selection and the whole feature set. (Continued)
56

---

Utilization
Out-of-domalnn accuracy
0.700
0.800
0.550
0.600
0.650
0.700
0.750
0.800
In-domain accuracy
Poverty
0.800
Out-of-domain
0.700
0.500
0.700
0.750
0.800
0.850
0.900
In-domain accuracy
Arguably causal
Causal
Constant
Random
Diagonal
Figure 40: Performance random subsets in comparison to causal feature selection and the whole feature set. (Continued)
57

---

C.6 Ablation of anti-causal and non-causal features
We conducted an ablation study and provided the results in Figure 41 to 43 We remove anti-causal and non-causal features one at a time and measure the corresponding out-of oft-of-oo oomain accuracy. In the following, we discuss in detail the non-causal features whose removal significantly dropped the out-of-domain performance and try to give explanations. We split by task.
Food Stamps Target is food stamp recipiency in past year for households with child across geographic region.
Relationship to reference person: There could stable and informative correlation within the survey of US Census between kind of household members (encoded in relationship to the reference person/head of the household, e.g. multiple generation household vs roommates) and food stamp recipiency. We didn't 't classify this variable causal, it is survey related.
Income
Target is income level across geographic regions.
Relationship reference person: Same argument as in the task 'Food Stamps' applies.
Marital status: Marital status and personal income are both intricately linked with socio- economic status, although we haven't 't found any research causally linking them together.
Insurance through current or former employer union Medicare for people 65 older, or people with certain disabilities: These insurances are benefits not tied to income, but rather the person's s employer or age and medical condition. They are however indicative of the economic and social environment of the individual, which is informative of the income level.
Year: The year, e.g., 2018, encodes information about the economic status, which may be predictive across geographic regions.
Public Coverage disability status.
Target public coverage of non-Medicare eligible low-income individuals across
State / Year: The current state of living and year encode information about the economic status.
Target is whether an individual voted in the US presidential elections across geographic regions.
Voting
Party preference on specific topics, e.g. pollution / Opinion on party inclinations, e.g. which party favors stronger government / Opinion on sensitive topics, e.g., abortion, religion, gun control: The opinions/preferencee of an individual may sort them to specific sub-groups of the populations, wherein civil duty is or is not prominent. It is fathomable that similar sub-groups form across geographic regions.
Hypertension
Target is high blood pressure across BMI categories.
The current state of living encodes information about the socio-economic status, which research linked to hypertension in several studies [Leng et al. 2015).
State:
Sepsis
Target is sepsis across length of stay in ICU.
Hospital: Hospitals serve different groups of the populations which differ in their risks of attaining sepsis.
58

---

Food Stamps
0.830
Out-of-domain accuracy
0.820
0.810
0.790
0.780
ACS YEAR
RELP
PUBCOV
Constant
Out-of-domminn accuracy
0.750
0.850
HINS1
NWLK
HINS3
HINS4
MAR
RELP
HINS2
Constant
ACS
PublicCoverage
0.690
Out-of-domailn accuracy
0.670
0.660
0.850
0.640
AlI
ACS, YEAR
ST
DIVISION
Constank
Figure 41: Removing one feature at time. Anti-causal features are colored in orange, non-causal in grey.
59

---

Unemployment
Out-of-domain accuracy
0.960
0.955
0.950
0.945
0.940
All
WRK
RELP
WKHP
WKW
Constant
Voting
Out-of-domainn accuracy
0.850
0.800
0.750
0.700
0.850
0.600
VCF8011
VCF9206
VCF9010
VCF0875
VCF0846
VCF0806
VCF9045
VCF9238
VCF0638
VCF9008
VCF9052
VCF0301
VCF9204
VCF0809
VCF0623
VCF0521
VCF0898
VCF0843
Consient
ASSISTments
Out-of-domain accurscy
0.600
0.580
0.560
0.540
ma first, resporse
overlap time
Figure 42: Removing one feature at time. Anti-causal features are colored in orange, non-causal in grey. (Continued)
60

---

Hypertension
Out-of-domainn accuracy
0.630
0.620
0.590
0.580
All
STATE
CHCOCNOR
CHOSONOR
Constant
Diabetes
0.835
Out-of-domain accuracy
0.825
0.820
All
TOLDHI
MICHD
STATE
HEALTH. COVV
CVDSTRK3
HIGH BL OLOOD PRESS
Constart
PAST_
College Scorecard
0.900
Out-of-domain acouracy
0.850
0.800
0.750
0.700
UG UNKS
F38
pet GosTe
pet Berregi6
UGDS 290
UGDS (RA
Coofiiiion
U6
UGDS
Ucds
NUMB
AVGF
UGB
age
UG
UCDS
UG
UGD
UGDS
Figure 43: Removing one feature at time. Anti-causal features are colored in orange, non-causal in
grey. (Continued)
61

---

Hospital Readmission
0.625
Out-of-domain accuracy
0.600
0.575
0.525
Ai
num, medications
mum procedures
admission typa Id
mum Jab propccdeuoosss
Constant
Utilization
Out-of-domain accuracy
0.700
0.850
0.600
0.550
Childhood Lead
0.925
Out-of-domain accurscy
0.922
0.920
0.917
0.915
0.912
0.910
All
DMDMARTL
Constant
Figure 44: Removing one feature at time, Anti-causal features are colored in orange, non-causal in grey. (Continued)
62

---

Poverty
0.800
Out-of-domain ancuracy
0.750
0.700
0.650
0.800
0.550
FOOD ASSISTANCE
ASSISTANCE
LIVING QUARTERS TYPE
WIC, ASSISTANCE
MEDICAID ASSISTANCE
TANF ASSISTANCE
Constant
Sepsis
Out-of-domain accuracy
0.927
0.925
0.922
0.920
0.917
0.915
AlI
set
Constant
Figure 45: Removing feature time. Anti-causal features colored in orange, non-causal in grey. (Continued)
63

---

C.7 Empirical results across machine learning models
We show the dareto-dominate performances for each machine learning model in Figure 46 49 The detailed results are provided at htthtts::/ggthhb com/socian oundati ons causal features tree/ade brdeeaad-aa mtts///rpprimmoots_aaaaaaaereeee for each task.
We have a summary table saved csv file
Food Stamps
Out-of-domain accuracy
Out-of-domain accuracy
0.820
0.800
0,800
0.780
0.780
0.800
0.820
0.840
-0,040
-0.030
-0,020
In-domain acouracy
Shift gap
Income
Out-af-domain accural aayy
0,800
Aut-af-domainn acourapy
0.800
0.700
0.700
0.800
0.800
0.700
0.750
0.800
-0.075
-0.050
-0.025
0.000
In-domain acouracy
Shift gap
PublicCoverage
Out-of-domeinn
Out-of-domainn
arcuracy
0.600
0,800
0.500
0.500
0.400
-0.200
In-domain accuracy
Shift gap
Unemployment
Out-of-domainn domfnn accuracy
Out-of-domain
0.960
accuracy
0.955
0.955
0.950
0.950
0.962
0.965
0.967
0.970
0.972
-0.020
-0.017
-0.015
-0.012
-0,010
In-domain accuracy
Shift gap
Adv. DRO
DeepCORAL
IB-IRM
MLP
SAINT
AND Mask
DRO
IRM
MMD
TabTranstormer
CausiRL Coral
FT-Transfomer
Label Group DRC
NODE
ReX
CausiRL MMD
Group DRO
LightGBM
ResNet
XGB
DANN
HislGBM
MixUp
Figure 46: (Left) Partta-dominate performance of in-domain and out-of-domain accuracy by feature selection and machine learning model. (Right) domain accuracy accomplished by feature selection and machine learning model. The feature sets are color-coded. Red indicates all features. The causal features shown in blue, the arguably causal features in grey.
performance of shift gap and out-of-
64

---

Voting
Out-of-dommain
Out-of-domainn accuracy
accuracy
0.800
0,800
0.700
0.800
0.800
0.700
-0,100
In-domain accuracy
Shift gap
ASSISTments
0.600
do--a--domain accuracy
Out-of-domain
accuracy
0.500
0.500
0.700
0.800
0.900
-0.500
-0.400
-0.300
-0.200
-0.100
In-domain accuracy
Shift gap
Hypertension
Out-of-domainn
0.800
Out-of-domain aocuracy
0.800
accuracy
0.500
0.500
0,400
0.600
0.825
0.650
0.675
-0.200
-0.150
-0.100
-0.050
0.000
In-domain accuracy
Shift gap
Diabetes
0.835
0.835
Out-of-domainn
Out-of-tomain
accuracy
sccuracy
0.830
0.825
0.825
0.872
0.874
0.876
0.878
-0.050
-0.055
-0.040
In-domain acouracy
Shift gap
DeepCORAL
IRM
TabTranslormet
CausiRL Caral
FT-Transfomme
Label Group DRO
NODE
ReX
CausiRL MMD
Group DRO
LightGBM
ResNet
XGB
DANN
HistGBM
MixUp
Figure 47: (Left) Pareto-dominate performance of in-domain and out-of-domain accuracy by feature selection and machine learning model. (Right) Pareto-dominate performance of shift gap and out-of- domain accuracy accomplished by feature selection and machine learning model. The feature sets are color-coded. Red indicates all features. The causal features are shown in blue, the arguably causal features in gray. (Continued)
65

---

College Scorecard
0.900
Out-of-domain accuracy
Out-of-domainn accuracy
0.800
0.800
0.700
0.900 In-domaiin accuracy
-0.200
-0.100
Shift gap
Hospital Readmission
-ul-of-domain accuracy
Out-of-domain accuracy
0.600
0.550
0.550
0,500
0.500
0.600
0.650
-0.100
-0.075
-0.050
-0.025
In-domain accuracy
Shift gap
Hospital Mortality
Out-of-domain
Out-of-domain
accuracy
0.880
Bocuracy
0.880
0.860
0.840
0.900
0.920
0.940
0.960
-0.080
-0.060
-0.040
-0.020
In-domain accuracy
Shift gap
ICU Length of Stay
Out-of-domman accuracy
Ouf-of-domain sccuracy
0.800
0.500
0.500
0.600
0.650
0,700
-0.100
-0.060
0.000
In-domain acouracy
Shift gap
DeepCORAL
IRM
TabTranslormet
CausiRL Caral
FT-Transfommr
Label Group DRO
NODE
ReX
CausiRL MMD
Group DRO
LightGBM
ResNet
XGB
DANN
HistGBM
MixUp
Figure 48: (Left) Pareto-dominate performance of in-domain and out-of-domain accuracy by feature selection and machine learning model. (Right) Pareto-dominate performance of shift gap and out-of- domain accuracy accomplished by feature selection and machine learning model. The feature sets are color-coded. Red indicates all features. The causal features are shown in blue, the arguably causal features in gray. (Continued)
66

---

Childhood Lead
0.925
0.925
Out-of-domain arcuracy
Out-of-domainn accuracy
0.920
0.920
0.915
-0.060
-0.050
In-domain accuracy
Shift gap
Sepsis
Out-of-domain
0.928
Out-of-domain accuracy
0.928
accuracy
0.926
0.928
0.924
0.924
0.987
0.988
0.988
-0.086
-0.064
-0.062
-0.060
In-domain accuracy
Shift gap
Utilization
Out-of-domain
Out-of-domain Bocuracy
0.700
accuracy
0.600
0.600
0.800
0.700
0,800
-0.150
-0.100
-0.550
0.000
In-domain accuracy
Shift gap
Poverty
0.800
0.800
Out-of-dommain
Out-of-domainn off-ddaaainn
accuracy
Sccuracy
0.700
0.800
0.600
0.500
0.500
0.700
0.750
0.800
0.850
0.900
-0.300
-0.200
-0.100
In-domain acouracy
Shift gap
DeepCORAL
IRM
TabTranslormet
CausiRL Caral
FT-Transfomme
Label Group DRO
NODE
ReX
CausiRL MMD
Group DRO
LightGBM
ResNet
XGB
DANN
HistGBM
MixUp
Figure 49: (Left) Pareto-dominate performance of in-domain and out-of-domain accuracy by feature selection and machine learning model. (Right) Pareto-dominate performance of shift gap and out-of- domain accuracy accomplished by feature selection and machine learning model. The feature sets are color-coded. Red indicates all features. The causal features are shown in blue, the arguably causal features in gray. (Continued)
67

---

D Synthetic experiments
We conducted synthetic experiments. The setup is depicted in Figure 50 modeled as (i) linear with weights randomly drawn in (-1,1) and (ii) based on a neural network with random instantiation. The noise variables are drawn from standard normal distribution. The task is to classify whether the target is larger than O.
The causal mechanisms
confounder
confounder
confounder
spurios
causal
target
anticausal
Figure 50: Causal graph to generate samples.
Similar to Rothenhäusler et al. 2020 vary the degree of domain shift using shift intervention target, features and confounders. We draw 1,000 training samples from the causal mechanism, and evaluate the performance 1,000 testing samples from the intervented causal mechanism with shift interventions varying from to 10; step size is 0.1. We provide example performances in Figure 51 Our code is based the synthetic study conducted by Montagna al. 2024
(a) Linear mechanism and normal noise, estimated by logistic regression. Dotted lines indicate in-domain testing accuracy.
Neural mechanism and normal noise, estimated by logistic regression.
(c) Neural net mechanism and normal noise, estimated by neural net.
Figure 51: Synthetic experiments. Mechanisms are randomly instantiated. Task classify target 0.
The synthetic experiments confirm our empirical findings. Using all features achieves best out- of-domain prediction accuracy. The one exception is if the distribution shift exclusively on the anti-causal features and even in this case, strong shift is needed before causal features achieve best out-of-domain accuracy.
E
Tasks and data sources
In this section give details shift. We refer the reader to Gardner et al. (2023) for more details on the Tableshift tasks, their data sources and the reasoning behind the proposed domain shifts. We provide links to the datasets, their access and licenses in Table 7
tasks. We briefly describe the data source, target and distribution
We list the features. and sort them into causal, arguably causal and anti-causal. In Section 2. and Appendix A we justify the sorting for seven examples: 'Diabetes'. 'Income", "Unemployment" 'Hospital mortality', 'Hypertension', 'Voting' and 'ASSISTments'. While in good faith, we do the selection under epistemic uncertainty. Future research health care social science may rebut our sorting. Therefore, advise caution when using our classification for follow-up research.
68

---

Table 8 provides an overview of all tasks, their training and testing domains, shift gap of the constant predictor and number observation in the dataset. We provide additional insights into the distribution shift in Appendix E.12
E.1
TableShift: ACS
We have multiple tasks based on American Community Survey (ACS) (U.S. Census Bureau, 2018 derived from Folktables [Ding et al. 2021). The encoding is found in the ACS documentation
E.1.1
Foodstamps
Target:
Food stamp recipiency past year for households with child [FS]
Shift: Geographic region (U.S, divisions) [DIVISION]
List of causal features:
Age in years [AGEP]
Sex [SEX]
Race [RACIP]
Place of birth [POBP]
Disability [DIS]
Hearing difficulty [DEAR]
Vision difficulty [DEYE]
Cognitive difficulty [DREM]
Ancestry [ANC]
Nativity [NATIVITY]
Marital status [MAR]
State [ST]
List of arguably causal features: Ability to speak English [ENG]
Gave birth to child within the past 12 months [FER]
Citizenship status [CTT]
Educational attainment [SCHL]
Households presence and age of children [HUPAC]
Occupation [OCCP]
Military service [MIL]
Workers in family during the past 12 months [WIF]
Usual hours worked per week past months [WKHP]
Weeks worked during past 12 months [WKW]
Worked last week [WRK]
On layoff from work [NWLA]
Looking for work [NWLK]
List of other features:
Year of survey YEAR)
Relationship to reference person [RELP]
Public health coverage [PUBCOV]
E.1.2 Income
The selection procedure for the task 'Income" is discussed in detail in Appendix A
Target:
Total person's income 2 56k for employed adults [PINCP]
Shift: Geographic region (U.S. divisions) [DIVISION]
List of causal features:
Age in years [AGEP]
census http://wwwww.pt/ooocoooocooctttt
69

---

Sex [SEX]
Race [RACIP]
Place of birth [POBP]
List of arguably causal features:
State [ST]
Ability to speak English [ENG]
Gave birth to child within the past 12 months [FER]
Citizenship status [CTT]
Educational attainment (SCHL]
Occupation [OCCP]
Class of worker [COW]
Usual hours worked per week past 12 months [WKHP]
Weeks worked during past 12 months [WKW]
Worked last week [WRK]
On layoff from work [NWLA]
List of anti-causal features: Insurance purchased directly from an insurance company [HINS2]
Medicaid, Medical Assistance, or any kind of government-assistane plan for those with low incomes or a disability [HINS4]
Looking for work [NWLK]
List of features:: Year of survey [ACS YEAR]
Marital status [MAR]
Insurance through current or former employer or union [HINS1]
Medicare, for people 65 and older, or people with certain disabilities [HINS3]
Relationship to reference person [RELP]
E.1.3 Public Coverage
Target:
Public health coverage [PUBCOV]
Shift: Disability status [DIS]
List of causal features:
Age in years [AGEP]
Sex [SEX]
Race [RACIP]
Hearing difficulty [DEAR]
Vision difficulty [DEYE]
Cognitive difficulty [DREM]
Ancestry [ANC]
Nativity [NATIVITY]
List of arguably causal features:
Employment status of parents [ESP]
Total person's income in dollars [PINCP]
Employment status [ESR]
Gave birth to child within the past 12 months [FER]
Marital status [MAR]
Citizenship status [CIT]
Educational attainment [SCHL]
Mobility status [MIG]
List of other features: Year of survey [ACS_ YEER]
State [ST]
Geographic region [DIVISION]
70

---

E.1.4 Unemployment
The selection procedure for the task "Unemployment' is discussed in detail in Appendix A
Target: Employment status unemployed) [ESR]
Shift: Educational attainment [SCHL]
List of causal features:
Agge in years [AGEP]
Sex [SEX]
Race [RACIP]
Place of birth [POBP]
Disability status [DIS]
Ancestry [ANC]
Nativity [NATIVITY]
Hearing difficulty [DEAR]
Vision difficulty [DEYE]
Cognitive difficulty [DREM]
Ambulatory difficulty [DPHY]
List of arguably featureee
Ability to speak English [ENG]
Occupation [OCCP]
Employment status of parents [ESP]
Military service [MIL]
Gave birth to child within the past 12 months [FER]
Marital status [MAR]
Citizenship status [CIT]
Mobility status [MIG]
State [ST]
Geographic region [DIVISION]
List of anti-causal features:
Usual hours worked per week past 12 months [WKHP)
Weeks worked during past 12 months [WKW]
Worked last week [WRK]
List of other features:
Year of survey [ACS YEAR
Relationship to reference person [RELP]
E.2
TableShift: ANES
We have task based on American National Election Studies (ANES) (American National Election Studies 2020
E.2.1
Voting
The selection procedure for the task 'Voting' is discussed in detail in Appendix A
Target:
Voted national election [VCF0702]
Shift: Us census region [VCF0112]
List of causal features:
Election year [VCF0004]
State [VCF0901b]
Registered to vote pre-election [VCF0701]
Age [VCF0101]
Gender [VCF0104]
http:://elleeiiionuuudeee..rr
71

---

Race/ethnicity [VCF0105a]
Occupation group [VCF0115]
Education level [VCF0140a]
List of arguably causa
Democratic party feeling thermometer [VCF0218)
Republican party feeling thermometer [VCF0224]
Party identification [VCF0302]
Like-dislike scale placement for democratic party (0-10) [VCF9201]
Like-dislike scale placement for republican party (0-10) [VCF9202]
Do any of the parties in the U.S. represent views reasonably well [VCF9203]
Better when one party controls both presidency and congress or when control is split
[VCF9206]
President thermometer [VCF0428]
Vice- iice-president thermometer [VCF0429]
Rating of government economic policy [VCF0822]
Better economy in past year [VCF0870]
L -iberrl-conserraative scale [VCF0803]
Approve participation in protests [VCF0601]
Voting is the only way to have a say in government [VCF0612]
matters whether vote [VCF0615]
Those who don't care about election outcome should vote [VCF0616]
Somcone should vote if their party can't win [VCF0617]
Interest in the elections [VCF0310]
Belongs to political organization or club [VCF0743]
Tried to influence others during campaign [VCF0717]
Attended political meetings/rallies during campaign [VCF0718]
Displayed candidate button/sticker during campaign [VCF0720]
Donated money to party candidate during campaign [VCF0721]
How much of the time can you trust the media report the news fairly [VCF0675]
Watched programs about the election campaigns [VCF0724]
Heard radio programs about the election campaigns [VCF0725]
Read about the election campaigns in magazines [VCF0726]
Saw election campaign information on the internet [VCF0745]
List of other features:
Think of yourself as closer to the republican democratic party
[VCF0301]
Party preference on pollution and enviromment [VCF9008]
Party preference on inflation [VCF9010]
Party preference on unemployment [VCF9011]
Party in U.S. that represents views best [VCF9204]
Which political party represents views best [VCF9205]
Which party favors stronger government [VCF0521]
Which party favors military spending cut [VCF0523]
Most important national problem [VCF0875]
Are things in U.S. going well or not [VCF9052]
Guaranteed jobs and income scale (support/don" it support) [VCF0809]
Government services and spending scale (fewer/more services) [VCF0839]
Defense spending scale (decrease/inereasee) [VCF0843]
Position of the U.S. in past year [VCF9045]
When should abortion be allowed by law [VCF0838]
Importance of gun control [VCF9239]
Importance of religion [VCF0846]
How much does federal government waste tax money [VCF0606]
72

---

E.3
TableShift: BRESS
We have two tasks based on Behavioral Risk Factor Surveillance System (BRFSS) [Centers for Disease Control and Prevention, 2021]. The encoding is found in BRESS data dictionary
E.3.1 Diabetes
The selection procedure for the task 'Diabetes' is discussed in detail in Section 2.1
Target: Diagnosed with diabetes [DIABETES]
Shift: Preferred race category [PRACEI]
List of causal features:
Highest grade or year of school completed [EDUCA]
Answer the question 'Have you smoked at least 100 cigarettes in your entire life?" [SMOKE100]
Sex of respondent [SEX]
Marital status [MARITAL]
List of arguably causal features: Annual household income from all sources [INCOME]
Number of days during the past 30 days where physical health was not good
[PHYSHLTH]
Body Mass Index (BMI) [BM15]
Body Mass Index (BMI) category [BMISCAT]
Answer to the question 'Do you now smoke cigarettes every day, some days, or not at all?' [SMOKDAY2]
Consume Fruit or more times per day [FRUIT ONCE PER PER_DAY]
Consume vegetables or more times per day [VEG_ONCE ONCE PERC__EERRREEEEEEE PER_DAY
Total number of alcoholic beverages consumed per week [DRNK PER [ER__WEK]
Binge drinkers (males having five more drinks or more drinks on one occasion) [RFBING5]
occasion, females having four
Physical activity or exercise during the past 30 days other than their regular job [TOTINDA]
Time since last visit to the doctor for a checkup [CHECKUPI]
Answer to the question "Was there time in the past 12 months when you needed to see a doctor but could because of cost?' [MEDCOST]
Answer to the question 'for how many days during the past 30 days was your mental health not good?' [MENTHLTH]
List of anti-causal features:
Diagnosed with high blood pressure [HIGH BLOOD PRESS]
Time since last blooc cholesterol check [CHOL_ CHK BOIK_AAST_S__SSS SEERS]
Diagnosed with high blood cholesterol [TOLDHI]
Diagnosed past stroke [CVDSTRK3]
Reports of coronary heart disease (CHD) myocardial infarction (MI) [MICHD]
Current health care coverage [HEALTH_COM)
List of other features:
State [STATE]
Year of BRESS dataset [[YEAR]
E.3.2 Hypertension
The selection procedure for the task 'Hypertension' is discussed in detail in Appendix
Target: Diagnosed with high blood pressure [HIGHLBLLOOD_PRESSS]
Shift:
Body Mass Index (BMI) category [BMISCAT]
pdf
73

---

List of causal features:
Age group [AGEGSYR]
Preferred race category [PRACEI]
Sex of respondent [SEX]
Answer the question 'Have you smoked at least 100 cigarettes in your entire life?"
[SMOKE100]
Diagnosed with diabetes [DIABETES]
List of arguably causal features: Binary indicator for whether individuals' income falls below the 2021 poverty guideline for family four [POVERTY]
Current employment status [EMPLOY1]
Consume Fruit or more times per day [FRUIT ONCE PER [AY]
Consume vegetables more times per day IVEG _NNCE PER DAY]
Total number of alcoholic beverages consumed per week [DRNK_ PER [RR_WEK]]
Binge drinkers (males having five or more drinks on one occasion, females having four or more drinks on one occasion) [RFBING5]
Physical activity or exercise during the past 30 days other than their regular job [TOTINDA]
Answer to the question 'Do you now smoke cigarettes every day, some days, or not at all?' [SMOKDAY2]
Answer to the question "Was there a time the past 12 months when you needed to see a doctor but could not because of cost?" [MEDCOST]
List of anti-causal features:
Diagnosed with skin cancer [CHCSCNCR] Diagnosed with any other types of cancer [CHCOCNCR]
List of other features:
State [STATE]
Year of BRESS dataset [TYEAR]
E.4
TableShift: ED
We have one task based on the data that appear on the college scorecard by the U.S. Department of Education (ED) U.S. Department of Education 2023 12
E.4.1
College Scorecard
Target: Completion rate for first-time, full-time students at four-year institutions (150% of expected time to completion/6 years) [C150 4]
Shift: Carnegie Classification basic [CCBASIC]
List of causal features: Accreditor for institution [AccredAgency]
Highest degree awarded [HIGHDEG]
Control of institution [CONTROL.]
Region (IPEDS) [region]
Locale of institution [LOCALE]
Degree of urbanization of institution [locale2]
Flag for Historically Black College and University (HBCU)
Flag for
education [DISTANCEONLY]
Poverty rate, via Census data [poverty_rane
Unemployment rate, via Census data [unemp_ram
Carnegie Classification size and setting [CCSIZSET]
List of arguably causal features:
In-state tuition and fees [TUITIONFEE
Out-of-state tuition and fees (TUITIONFEF_OUT]
Tuition and fees for program-year institutions [TUITIONFEE_ PROG]
74

---

Admission rate [ADM RATE]
Admission rate for all campuses rolled up to the 6-digit OPE ID [ADM RATE_ALL,
Midpoint of SAT scores the institution (critical reading) [SATVRMID]
Midpoint of SAT scores at the institution (math) [SATMTMID]
Midpoint of SAT scores at the institution (writing) [SATWRMID]
Midpoint of the ACT cumulative score [ACTCMMID]
Midpoint of the ACT English score [ACTENMID]
Midpoint of the ACT math score [ACTMTMID]
Midpoint of the ACT writing score [ACTWRMID]
Average net price for the largest program at the institution for program-year institutions [NPT4 PROG]
Average cost of attendance (academic year institutions) [COSTT4 A]
Average cost of attendance (program-yee institutions) [COSTT4_PP P]
Share of students who received a federal loan while in school (oan__evey
Share of students who received a Pell Grant while in school [pell_ever]
Percentage of undergraduates who receive Pell Grant [PCTPELL]
Median household income [median_hi, hh__inc]
Average family income [famine]
Median family income ind_famine)
Enrollment of undergraduate legree-seeking students [UGDS]
Enrollment of all undergraduate students [UG]
List of other features: State postcode [STABBR]
Predominant degree awarded (recoded Os and 4s) [sch deg]
Flag for main campus [main]
Number of branch campuses INUMBRANCH]
Percentage of degrees awarded in Agriculture, Agriculture Operations, And Related
Sciences [PCIPO1]
Percentage of degrees awarded in Natural Resources And Conservation [PCIPO3]
Percentage of degrees awarded in Architecture And Related Services [PCIP04]
Percentage of degrees awarded in Area, Ethnic, Cultural, Gender, And Group Studies [PCIPO5]
Percentage of degrees awarded in Communication, Journalism, Jouruiiai,,,,,, And Related Programs [PCIP09]
Percentage of degrees awarded in Communications Techmologie/Teccmiiiaas And
Support Services [PCIP10]
Percentage of degrees awarded in Computer And Information Sciences And Support
Services [PCIP1I]
Percentage of degrees awarded in Personal And Culinary Services [PCIP12]
Percentage of degrees awarded in Education [PCIP13]
Percentage of degrees awarded in Engineering [PCIP14]
Percentage of degrees awarded in Engineering Technologies And Engineering-Relate Related
Fields [PCIP15]
Percentage of degrees awarded in Foreign Languages, Literatures, And Linguistics [PCIP16]
Percentage of degrees awarded in Family And Consumer Sciences/Human Sciences [PCIP19]
Percentage of degrees awarded in Legal Professions And Studies [PCIP22]
Percentage of degrees awarded in English Language And Literature/Letters [PCIP23]
Percentage of degrees awarded in Liberal Arts And Sciences, General Studies And Humanities [PCIP24]
Percentage of degrees awarded in Library Science [PCIP25]
Percentage of degrees awarded in Biological And Biomedical Sciences [PCIP26]
75

---

Percentage of degrees awarded in Mathematics And Statistics [PCIP27]
Percentage of degrees awarded in Military Technologies And Applied Sciences [PCIP29]
Percentage of degrees awarded in Mullt/hteeriiccilinaay Studies [PCIP30]
Percentage of degrees awarded in Parks, Recreation, Leisure, And Fitness Studies
[PCIP3I]
Percentage of degrees awarded in Philosophy And Religious Studies [PCIP38]
Percentage of degrees awarded in Theology And Religious Vocations [PCIP39]
Percentage of degrees awarded in Physical Sciences [PCIP40]
Percentage of degrees awarded in Science Cechnologies/Techniiinn [PCIP41]
Percentage of degrees awarded in Psychology [PCIP42]
Percentage of degrees awarded in Homeland Security, Law Enforcement, Firefighting
And Related Protective Services [PCIP43]
Percentage of degrees awarded Public Administration And Social Service
Professions [PCIP44]
Percentage of degrees awarded in Social Sciences [PCIP45]
Percentage of degrees awarded in Construction Trades [PCIP46]
Percentage of degrees awarded in Mechanic And Repair Technologies//echmiicans
[PCIP47]
Percentage of degrees awarded in Precision Production [PCIP48]
Percentage of degrees awarded in Transportation And Materials Moving [PCIP49]
Percentage of degrees awarded in Visual And Performing Arts [PCIP50]
Percentage of degrees awarded in Health Professions And Related Programs [PCIP51]
Percentage of degrees awarded in Business, Management, Marketing, And Related
Support Services [PCIP52]
Percentage of degrees awarded in History [PCIP54]
Total share of enrollment of undergraduate degree-seeking students who are white [UGDS_WHITE]
Total share of enrollment of undergraduate degree-seeking students who are black
[UGDS BLACK]
Total share of enrollment of undergraduate degree-seeking students who are Hispanic
[UGDS_HISS)
Total share of enrollment of undergraduate degree-seeking students who are Asian
Total share of enrollment of undergraduate degree-seeking students who are American
Indian/ Alaska Native (UGDS [GGDS_AIAN]
Total share of enrollment of undergraduate degree-seekingg students who are Native HawaiianPaccfic Islander [UGDS [GGDSNNHPI]
Total share of enrollment of undergraduate degree-seeking students who are two
more races [UGDS [MOR]
Total share of enrollment of undergraduate degree-seeken seeking students who are
non-resident aliens BUGDS_NRRY
Total share of enrollment of undergraduate degree-secking students whose race is
unknown BUGDS_UNKKR
Total share of enrollment of undergraduate degree-seeking students who are white
non-Hispanic [OIDSS_WHTTNH]]
Total share of enrollment of undergraduate degree-secking students who are black
non-Hispanic [UGDS BLACKNH]
Total share of enrollment of undergraduate degree-seeking students who are
Asian/Pacific Islander [UGDS_AL API]
Total share of enrollment of undergraduate degree-seeking students who are American Indian/Alaska Native (UGDS ALANOId]
Total share of enrollment of undergraduate degree- seeking students who are Hispanic
[UGDS HISPOld]
76

---

Total share of enrollment of undergraduate students who are non-resident aliens
[UG NRA]
Total share of enrollment of undergraduate students whose race is unknown
[UG [UN_]
Total share of enrollment of undergraduate students who are white non-Hispanic
[UG WHITENH]
Total share of enrollment of undergraduate students who are black non-Hispanic
[UG]BLACHE]] BLACKNH]
Total share of enrollment of undergraduate students who are Asian/Pacific Islander
[U]_API]
Total share of enrollment of undergraduate students who are American Indian/Alaska
Native [UG AIANOId]
Total share of enrollment of undergraduate students who are Hispanic [UGHISPPOI]]
Share of undergraduate, degrec-/certfficatt-seccinn students who are part-time
[PPTUG [EF]
Share of undergraduate, degree-/cerrtfficatt-eeei students who are part-time
[PPTUG EE2]
Net tuition revenue per full-time equivalent student [TUITFTE]
Instructional expenditures per full-time equivalent student [INEXPFTE]
Average faculty salary [AVGFACSAL]
Proportion of faculty that is full-time [PFTFAC]
Average age entry, via SSA data [age_entry]
Average of the age of entry squared [age_entry_sq]
Percent of students over 23 at entry [agege24]
Share of female students, via SSA data [female]
Share of married students [married]
Share of dependent students [dependent]
Share of veteran students [veteran]
Share of first-gent seateeeneration students [first_ gen]
Percent of the population from students' zip codes that is White, via Census data
soct_white]
Percent of the population from students' zip codes that is Black, via Census data [pct_black]
Percent of the population from students' zip codes that is Asian, via Census data [pet_asian]
Percent of the population from students' zip codes that is Hispanic, via Census data [pet_ [ic__hisa
Percent of the population from students' zip codes with bachelor's degree over the
age 25, via Census data [pet ba]
Percent of the population from students' zip codes over 25 with professional degree, via Census data [pet_ grad_ppoff_
Percent of the population from students' zip codes that was born in the US, via Census data [o]
E.5
TableShift; Kaggle
We have task based the data collected from online learning too and released Kaggle [Feng et al. 2009
E.5.1
ASSISTments
The selection procedure for the task 'ASSISTments' discussed in detail in Appendix A
http:://ee..asisstmmttss
77

---

Target: Correct on first attempt [correct]
Shift: School Ischool_ide _
List of causal features:
Number of hints this problem. thint_ccuntl Number of student attempts on this problem. [attempt count]
ID of the skill associated with the problem [skill_id] _id]
Problem type [problem_tynee __tpe]]
Whether not the student asks for all hints pootoomhinnt
Tutor/Test mode [tutor_ [otlor_mde]]
Assignment position on the class assignments page [position]
Type of the head section of the problem set[type]
Type of first action: attempt or ask for a hint [first_action]
List of arguably causal features::
Predicted Boredom of student for the problem [Average_ [Average_confidenceee confidence(BOREDD)
Predicted Engaged Concentration of student for the problem
[Average_ AAverag__confidencel/ CCnfileneeCOONEETTRAAA
Predicted Confusion of student for the problem [Average_ (NveragecoffieeeeCCONNTUUDD))
Predicted Frustration of student for the problem
[Average_cone cowniiieeeeeeRRUSSRRRR
List of other features: Time in milliseconds for the student's first response lins_fisttrreee firso_esssoosee
Time in milliseconds for the student's overlap time [overlap_ting
E.6
TableShift: MIMIC
We have two tasks based on Medical Information Mart for Intensive Care (MIMIC-III), derived from MIMIC-Extract [Johnson et al., 2016 Wang et al. 2020a b).
E.6.1 Stay in ICU
Target: Stay in ICU for longer than 3 days [los []
Shift: Insurance type (Medicare, Private, Medicaid, Government, Self Pay) [insurance]
List of causal features: Age in years [age]
Gender [gender]
Ethnicity [ethnicity]
Height [height_mean_0)
Weight [weight__men_0]
List of arguably causal features: Bicarbonate [bicarbonate mask O,
bicarbonate_mms laa__aask2222 23, bicarbonate_i _ 0,a O..., bicarbonate_no _ea___eaa_______ _3,
Bicarbonate_mcca____o
bicarbonate satpatttettmeeeteeeeeee measured 0, ... bicarbonate time_sinccemmasuuredd ____eereuddd_ 23]
Co2 [co2_mask_0, ...,, co2_mask, 23, co2_mean_ me. 200__mean_23,
co2 _02_time_since___ Lttp__iinncemmmsuuee______ O, ... co2 fe__tim__ssee aatee_iicc_mmasurreee
Partial pressure carbon dioxide (pCO2) and end_tidal CO2 (ETCO2)
0.
(etco2_pco2_etc)
23,
(etco2 ______co2______ etc) _eea 0, .... co2 (etco2 2_c222_pp22___21 _tc) mean 23, co2 (etco2 Sereco__ppo_____111111 etc)_timeee_____teeeeiiiitieee since measured 0. [ictco2__co___________ time__ttm____eeeee_em_ete since pecseeemaeeeeeeeeeeeeee 23]
Partial pressure of oxygen [partial pressure of mask .....
ppassurl_peessureeeeeeee
patiaalpesuureoofoxxgeeeeeeee
Bitiia__pessure_o_ooogeen wts.cceemmeuureedd .....
sinygen_time_sicce
78

---

Fraction inspired oxygen [fraction inspired oxygen mask .....
fraction inspired_ _aatreed___yenn__e
oxygen mask 23, fraction inspired oxygen mean O.....
@aygen_mean__
fraction_inspire
_ oxygen_ttii
siise_measured O..... Seapireed_x_geneeeee oxygee__ting tttee_eineeeenieeeeeeeeeeee
"raction_inspired
__,
fraction_inspire Lanspired_oxyeen..
soygen_set_maae
fraction fraction_inspiree Learirrddoxygenneeeee [as_mmas________
pmactoon_inspire maispired__xyyenn.ooo pxygennset__meeee
.... fraction_ iirioo__nsiireddd_e oeaoolooxxyeeeeee meem
fraction Lacpoa__insipi___________ _xygen sereee__st__ttte set_time since_sccee___e paseremmeaureddee 0, .....
fraction inspireedooxyeeenn oxygen_ set time_simmee___________ioe_tt since measured 23]
Glascow coma score [glascow_ comaaooomceeeee saanaasssll_aaee sotl__ttal_aa Bacee_tttllaa______ee_e .....
glascow_coma sacilasslle__ooeeeee saall_tttalmmakkk 23, glascow_cool http:w_wmaa_eeeeeeee saap__call_____e__ teeatlmaaaaaan_n___mm_m mean O....
glascow_coma hacocoooaaaaasseeeeeeeeeeeee scale_total _____man__2___ 23,
glascow_come scale total _ti___ti_________ since measured 0, .....
glascow http://womaaaaee set___salll____eeeee time since measured 23]
Lactate |lactate_mask_____ 0, ..... lactate maceee_aaa______ 23, lactate mean S,aa__0... O..., lactate maaeee__mea_____ 23, since measured O,
pactate_timme _ttee_simmeeee____eeeeeeeeteeeee
poctate_tima wewcssiiinssueeeeeeeee measured_ __]
Lactic acid [lactic_ acid mask 0, ... lactic lactic_acid mask 23, lactic_acis mean_2an____ lactic acid time clse__sncceeeeeee ..... lactic acid time_ time_since, measured_22
mean 0,
lactic_acid
Sodium [sodium mask 0, .... sodium mask 23, sodium mean O...., sodium_ Seatummmmmaa___-_ 23, sodium time mance_measured__0 0,.
cicttum_timessiie Berteee_mauuue_____ 23]
Hemoglobin [hemoglobin mask 0, .... hemoglobin mask 23, hemoglobin mean 0,
Beemoglobin_mean221
O....
segllobin_time_sincee httpie__iiiceemmsuredd____
Mean blood pressure [mean aemea__loood__e filooddpreesuue_mmssee
mean mean_blood _at___rrssrr_maasee _3, meen__loodd
mean mean_blood bllood_pressure_nnge mean_ 2aneen__,,nan_ mean_mlemnee frrooddpresuure_timeee sinnce_maasured___
mean_ wwwloooppoesuuuuleeeeeeeeeeeeeee saatiietttm_ssee__meeeeee Siince_meeauuee__21
Oxygen saturation longen_ssturaaten Littrrattionmsss___00 oxygen saturation_mast 23,
oxygen_saturation Bacaanmmmmaa____ ... oxygen_saturaatonn Leatriioonmma_____1 23,
oxygen_saturaation_tnn http_uuiinnnsiicccmcuuueeee____e
Ph ph. ph__msh_223_333301 ph_mean_0_ ph_mean_22
ph ph_time__iince_mmasureed
siime_since_mmasureeee
Respiratory rate [respraator__rat__mask
respirator__rte__mas___ respiratory_rate_mmann ... despiratory_rate__menn
respiratory titeettettttttteeee_eeee since measured 0.
respiratory_ratt ratee time since measured 23, respiratory pattoorryyeteeeeo set mask
rate
respiratory_rate_se htt__ratt_sst__akkkkkkk 23, respiratory rate_ set_mean_00 O....
respiratory_rate_ct _erattessee_eae_______e mean 23, iispiratory_rate rate__reee____e site__st__tee_e time_timmeeee maanucee
respiratory_rate_set lette_sstttttte___eeeeeee saseetttteesieee____eeeeeeee sicicc_mmasured_222 23]
Systolic blood pressure [systolic blood prrllprresseeeee mask_ O.....
systolic_ _rrod_presuure_ecoo mask 23, systolic blood _oroodupeeuureeeeee mean_00 O....
systolic_blowe blood Ball___resue__emeeee __an 23, systolic jystolic_blood__ pressure_tine since_measured,
sssttlic_bllsse Saaeeeooooooooouuuoreeee pressure teesssrr__tme__ttt
Heart rate [heart_rate rrte___rte_______ mask 0, ... heart_rate_mann _____aas______ 23, heart_rate,_ mean O....
neart_rate_mmen.
siiee_tte_ssseeee saate_time_since_meeeeeee since_meesuree 0,,
Temperature [temperature wtwpreerattrreaaaaaaaa
comperature_mask 23, temperature_mean,
.... t.teppraauue_mmman_222222 temperature sigeraeeeeeeeeeprrruaueeenineeie http://iee_siieemesseeeeeee
setteratur__time__sieeeeeee
__]]
White blood cell count|white blood_ cewpooo__el_oott_msseee
white_ leoodddell_coun_____ white blood leaeddeeeel____neneeeeeee 23,,__2,,____ white seee_blood_eee_eeeee
mask_23, siitte_blood_eeee
count_mean__n__
beell_count_tiiee since_ measured_
white wlood cell_ count couuttttmeee__toooottettteeeeeee sinne__mcsured 23]
79

---

List of otherrfeatuees
Height [height mase__aask____ ...., height mask 23, height mean I....
height mean 23, height time since measured_ O,.... height_time, sinesseee____eeeeeeeeeeeeeeee__e measured_23]
Weight [weight_mms Saatttmmsskk__ .... weight_mas caant_mask_2200 23, weight mean mean__,....
weight_mean___ weight_mean_ 23,,,,2,,, weight_ time Since__easuure__.....
weight_tingg seacht__tim_sinceeee sice_ssnce____eeee_eeeeeeeee pincemmasurrd 23]
Alanine aminotransferase
23, alantie-aminooraasfersss__maa
aaaniiee_amintttasffeeaaa mean_ 22.
caaniie_amminttaassfeenne time time_since _ince_meesureee .....
alanine satiine_aminttaanfffaaae sactcttttteeeeeeeeeeeeeeeeeeeeee saaice_eeauurdd 23]
Albumin [albumin Batpl_maskk..00 0, .... albumin_mask albumin_mean_0...
albumin_mean 23, albumin_tin Limetntt_t____________e tiie__iice_mmmssuueeee ___icc__meeuuree_..
albumin_time_ since saaiice_mmauuue___22
Alanine aminotransferase [albumin ascites mask 0, .... albumin ascites mask 23, catasssmemma____ 0, ... albumin ascites_ meaesss_m_______ 23,
albumin_ascites_tie lac___tte_ssiieeee sisice_meauured_. O....
albumin_ascinn time since siccce_maauredd_ 23]
Albumin pleural [albumin _ecurralppraallleeeeeee mask O.... albumin pleural mask 23,
albumin palulalpleraalle mean O..... albumin pleural mean 23,
albumin Belumin_pleurall. time sincessssee_eeeeeeee_eee measured O.....
albumin_pleural_ting httplpurra__me_siic______eeee___ mti__mmasuud____2
Albumin in urine [albumin_uu Laaarii__uiiiemaaaa___ wwa.maask___ ..... albuminn urine saaee_mask_222
albumin_urine_nnn cicee_mean_te_ 0, ... albumin urine_mean 23,
albumin_urine saaeee_ttmeeeeeeeee tiiee_iince______eeeeeeeeiiseeet
O. .....
albumin_uuie _urieerrrr______eeeeeee___e time since measured 23]
Alkaline phosphate
mask O.... alkaline Bacihie__phhhhhh___maa_e Bachhiii___hpphahheeeeeee cacps__ean_____ __3
Baapliie_phhpphate__ggee Sattiaateeeee__,,___
alkaline_phosste ltt_ttttieeeeeteeeeeeeeeeeee since__masuueed_
Sosaline_phosphate teme_tmm_msst_e since_measured __]]___________
Anion gap [anion Baaiinnaaa____ gap_ma__m______ gap_mask 0, .... anion gap_mast_ mask 23, anion gap mean 0,
anion_gan Latoo_gap_mean___ 23, htt_______iessine__neeeee
hasp__tmm__sicc_meeeeee
Asparate aminotransferasee [asparate aminntraansferaseeneee smciseaasssmmsss____ O....
aanaraae_aminottassffrreess _aaas/ssersssssssaaaas 23, asparate_ aattiaaaannnnseereeeeeeeee _eaaeasseeee_ee_____eee
apparaae_ammnnotaasfeeeeaa mean 22ean_23.
aaaarate_ammnootrasffreeee siteerassseett_meeeeeeeeeee since_measured measured .....
asparate_aminnttrasfeere aminotransserasse time since picce_ueeeuredd 23]
Basophils [basophils mask O, ... basophils mask 23, basophils mean O....
basophils mean_ 23, basophils time_since basophils time since, measured 23]
.....
Bilirubin [bilirubin, mask_ O, ... bilirubin satil__ass__2__0 23, bilirubin mean m O....
bilirubin_mean_222 bilirubin sictiit_ttte_sssie siaeeeseeieeeeuuued_______
fitrubinntiieessine htt__siicc_mmasured___
Blood urea nitrogen [blood urea_ iirea__itrogen_nnee Lartggnn_mas____
blood_ume ____uurraattteeeen Baal___ass_____ blood _al___ua____gg tii__nntrogeneeeeee mamen_0.... 0.....
soedd_rra_nitrrgge _eaean__m_______ 23, blood urea hitrroen_timme since_ meauuueeeeerdddd..ee
blood_ blood_urea_nine gaarea_nntrogen_iiine
Calcium [calcium mask 0, ..... calcium mask _2, calcium mean 0, O.... calcium mean 23, calcium time since measured O,....
calcium time since measured 23]
Calcium ionized [calcium ionized mask O...., calcium ionized mask 23, (ilcium_ionizer _eeddd_eea_____ ... calcium ionized mean 23,
calcium_ionzzed
calcium_ionized cenniieed_tme__aee sime__since liince_measured O....
calcium jonized time sime_since_____t since measured, 23]
Calcium in urine [calcium_urine) Bacttuu__rine_mmakk bai__mask_________ ..... calcium_urine_mas
calcium calcium_urine___ O, ... calcium urine _urine_mean___222 23,
calcium urine sime_sinee _aauure_mauurdd___ .... calcium, urine time since, m 23]
80

---

Cardiac index [cardiac index mask O..., cardiac index mask_makkk___ 23,
cardiac index mean O, .... cardiac index mean 23, cardiac_indes beressiindx__iie siilt_tte___see__ btaasstttmeeseee___ee_ee since_measured .....
23]
Cardiac output by Fick principle [cardiac_ cutrc__uppp_tttt tarippttfic_mssk_k... mwskmaakk___ O.... cardiac _accs__uuuppuutteeeeeeeee fick mask 23, cardiac output
aardiac_output serput_fick_.__ fick mean_ O......
cardiac cuttcctt_uuuuttttt fick mean 23, cardiac cardiac_output, fick time_since since measured 0,
cardiac_ood Bactci__uuuuttttttee fick time since measured _3]
Cardiac output by thermodilution [cardiac ccarhiccooutuut wht.ttthemmmiltionnn..
.....
cardiac cattss__uuppputteeeee wwwppuutthemmooiiieee thermooilltiion mas___aas_______ 2_,,,,2,,,___ cardiac_outpul Battlcu__uuuuu_uhhheeeeee twwrmodilttiinnnnnoooooooooootte
.... cardiac output thermodilution Baann__ea______ 23,
cardiac output thermodilution time _ine_ssnce_eeee _eeeeuree
cardiac_outee _uttcctt__uuttttt ttermodilutionn time time_timmeeeeeee _es_sincee_______ee piccsuueddered 23]
Central venous pressure [central venous prrpsuureppuueeeeeeeeee mastere___ss_____ 0,
central_venous, Brrnouuspreeuue__mme mase___aak____ 23, central pecnoouspressure_meeeee ....
central Sontral_venous_paa wtw__eeouuuuuusuuuueeeeeeeeee _eierr___ee______ 23,
time_ siine_ssnce_ae_e siace__masured_cmm O....,
central Sereenousppeesuueeeee time since measured 23]
Chloride [chloride mask O, ... chloride mask 23, chloride meaeee_mma_____ 0, ....
chloridc_mean 22, chloride_tinme siceeessneee_eeeeee _atice_mmaurrd____ 0,
chloride_time since_ measurdd__2 23]
Chloride in urine [chloride _aallle_uuee_____ee maa__aask__pp ..., chloride urine_mraink mask_2ss chloride_urine__mest
Bitpridd_urie_mmss____
chloride_ chloride_urine_mennt _urine_mean______ 23,
caloordd_uuriee_ee chloride urine time since lacceeemasuueeeeeeeeeee O....
www.uurnnnneeeee__eeeeeeee time since_ @icce_mcauured_c. 23]
Cholesterol [cholesterol_nge _aa____ask____ .... cholesterol_man _mateerro_mmss___1 23, cholesterol_mean_0..
cholesterol_mean. cholesterol_time caaasstro__iiieeeseeeeeeesseeeee laaeeesiiic__msuure___00
cholesterol_timee_siine wtt.siiceemmauure__22222
HDL cholesterol [cholesterol hdl mask O,... cholesterol hdl mask_ 22,
cholesterol mean_o, cholesterol Laa____addd______11_1
mean_ 23,
cholesterol hdl Lodl__time_sinee siince_mesuuree .....
_3]
LDI cholestero [cholesterol_ _allk_as________ O...., cholesterol Idl mask_aaa_______a__ 23,
cholesterol @idl_mean_0,
cholesterol Idl time_simmee time_since_mmeasured__ O....,
cholesterol_ide _dl__iime_sineeeeee Sisnncemeeauued_221
Creatinine [creatinine Jrreatiniee_mask_AAA O, .... creatinine mask 23, creatinine mean ..... creatinine _ateee_mmmaa___eee 23, creatinine time Since_measured O....
creatinine_time. since_measured __3]
Creatinine ascites [creatinine ascites mask 0, ..., creatinine ascites mask 23, creatinine creatinine_ascites mean_ ..., creatinine ascites Baaili_eeaa_____ 23,
creatinine creatinine_ascites ascites time since_ncee_e_ee measured O,...
creatinine jaciiiieneessitessseeeeeeeee time since sancc___esureddeee 23]
Creatinine body fluid [creatinine Seeratinineebdyy__oa f_liifluid_ddd __s_mmask...
greatinine_body
body_fluidddddoo uodd_ffuid_nn fhiid_mak.._ 23, creatinine Secatinineeboyy____ee [t__ffuddmeea__00_ O....
creatinine body__odddoalle mean 23, creatinine body fluid Lime_since
leleo_sicceeeeeeeeeeeee cecteeemeeeeedd.eeeeeee 0,
creatinine __dd__ooo______e_ fluid time since measured 23]
Creatinine pleural [creatinine Secpatinineppeeuaallll mask __, O.... creatinine Lacll_pleeeaaaaeee maal__aas_____11 23, ceattinnne_peuura_nng m_a__mea_____ ... creatinine Lllllppleee_eeeee mean 23,
creatinine_pleural
httpppleura_ttmmme___eee t__t_tim__iiig since_ssnee____eeeeeeeee caaiiee_meaureddeee O.....
greatinine_pleurae time
time_since_______tttiiiittii since measured 23]
Creatinine in urine [creatinine_ _urininneeeueeeeeeeeeeee mask_masa.____aaa ..... creatinine Seatinine_urine_nne arrne__mak____ __,,
creatinine wtw.uuuuuiunnnn____eee
mean_ 0,
creatinine
m 23,
creatinine_urine uurnee time semme_sine___eeeeeeeee__ maapuee_meaured___m O....
creatinine_urin jurieeeeeuru__eeeeeeeeeee time tiieeteineemnnneeeeeeeeee Saince_meaaured_22
Diastolic blood pressure [diastolic_ hat__booodppssuureeeeee Lal_sppreeeaasseeee mas__aask______ppp 0....
diastolic diastolic_blood pall__prrreeeemaeee
citastoiic_bloodpresstee mean
castolic_blocke feeeddpppeeeemeaaa
81

---

diastolic blood pressure_ time since measured 0,
diastolic blood pressureeeeeeeee time siioe__eesureddc 23)
Eosinophils [eosinophils mask O, ..., eosinophils_mne mask_ 23, eosinophils_meet Seapiilssmmee____e
.... eosinophils mee__maan______ _3, eosinophils time seee_since_mmaareee 0,
eosinophils_ time since_ lel_s_sscceemmsseee measured _3]
Fibrinogen [fibrinogen _as_maask____ O, ..... fibrinogen mask 23, fibrinogen mean O.....
fibrinogen _eaa_mmaa_________ @aean_23, fibrinogen timet Lince_measured_]
fibrinogen_iie sarreeenttmme_e_eeeee__ee sime_since,_ _ee__maasurd_____ 2_]
Glucose (glucose mask 0, ... glucose mask 23, glucose mean O....
glucose mean 23, glucose time since mnuee_measured11 0, O....
bllcose_time__e
23)
Hematocrit [hematocrit mask .... hematocrit mask 23, hematocrit mean 0,
hematocrit mean 23, hematocrit time_since mwwsured_ddd___
hematocrit time since measured 23]
Lymphocytes [lymphocytes. sarooocyttssssss____ .... lymphocytes_mant sacis__ass____ 23,
lymphocytes mean 0, .... lymphocytes lal_a__eaa_________ 23,
lymphocytes_timee
ttteetttieeeeeeeee_eeeeeeeee since measured ... lymphocytes_ting tettisstttmeseeeeeeeeeeee __]]
Lymphocytes ascites [lymphocytes. ascitessttsssss sasss__aas__________ O.....
lymphocytes_aae _scistsssssssssssseeeessssseeees Docites_mask___ mask 23, lymphocytes_eee ascites mean O.....
lymphocytes_acciooe sl__aasccitss__eeeeeee mean 23, lymphocytes
siaiiessstteesseee___eeeeeeeeeee since wit_:ssmannneeeeeeeeee mea.mreeadddeeee
lymphocytesaaccioo tttis_tttm___e_ee since measured_ 23]
Atypical lymphocytes [lymphocytes atypacall mask O.....
bttpbboottts__ttyipieeee mask 23, lymphocytes _atspicalllllll mean O....
Batpooyytes_ttticcll__ge 23, lymphoeytes_atypeee
wtapa_imieeeneeesssueeeeee httpphooooteesstpiiccal time since measured__dddd_
....
cupphocytes_ttpicall__oo __l___mas_____ 0. .... hrnnakoo.....yyyyooctteeeeeeee
tetpooyyyttstttppitaee cropiccl_ess_maaa _sl__mak____ _3,
lymphocyyts_atypicall _aass__________________ ... stathooctessattpiccall Adyypica__ss__men___0 maaa__ma____
yymphocytes_atyiica_sssl lal_l_sssstimmm_meeeeeee Sassiineemmmuurd______
Lymphocytes in body fluid [lymphocytes_hon saceees_ood__ffueee mask O, O....
lymphocytes_bon body_fluiii Liaidd_ass____ 23, lymphocytes bbdy fluid mean O.....
lymphocytes_bee becisss_oooo____eeeeee___eeeee cady_ffudddmean_222
lymphocytes_body body__ooiiiiii (eluud_time_stt wwt_s_siiceeeurreeee 0,
lymphocytes_bood S_dds_boo_______l_ body_fluid time wwwsssnnnnssseeeee __3]
Lymphocytes percentage
caaphooyyee_ppereeeee peercent_maskk__ 0,
lymphocytes_peree petrlsss__eeee___eee mask 23, lymphocytes_pereent mcan ....
Setpeyytttssppeenttte _eatttt__aa_____a 23, lymphocytes_peree Setpittesseeeeetteeeee time since meceeeeeeaasd..._eea
lymphocytes_perceee percent time since measured __]
Lymphocytes pleural [lymphocytes_p pleural mask O....,
Sacprccrytesppepaalale lymphocytes pleural mask 23, lymphocytes _etilspppeaaa__e mean O....
lymphocytes plcural mean_ 23, lymphocytes. lllllpplcpllll time since pecpsrmeeaeeeeeeeeee 0, O,...,
smphocctesspleural time__sice,______t wwwcssnnensureeeeee mw__sueaadddd..
Magnesium [magnesium mask ... magnesium mask 23, magnesium mean .... magnesium mean 23, magnesium nime_since
since measured O....
magnesium time since measured 23]
Mean corpuscular hemoglobin cecan__crruuuular suapssulla_hhemooloinn
....
hemoglobin mean mean corpuscular hemoglobin mask 23, mean corppusccouuularrcccaaa corpusculaarbemmgllohi
.... mean
www.umaaeaancellllleaa corpuscular hemoglobin mean_
mean mean__cappaaaaaa cectuucoopuuuuarr hemoglobin Limeeeeiiiiimmmmeeeeeeeeeeeee since measured 0. ....
mean_com hat___cpouuuuarr_heeeeeeee Baaioglloin_time____ wwwssiinnnnnceeeeeee 23]
Mean corpuscular hemoglobin concentration
[mean, [mean_ corpuscular hemoglobin hemnollobincconcenrttonn mask 0,
mean_ hhwps/scclaaeeeooooooooooooooooo
mask 23,
mean_corpin oegaauuocopuuuualllleeegooooooon
manieeeeeauured..cm 0, ....
Lictee_mmauured__21 _2]
82

---

Mean corpuscular volume [mean corpuscular, volume mask .....
sopuseuula_volummee
mean__crrpueeee corpuscular volume mask 23, mean corpuseull volume mean 0, .....
mean
mean_ arran_ccrpuucullr_vollre sariarr__vlmmemeeeeee Laite_mean_220 mean__3,
mean_corrpuscular__ee volumarreoveeeeeee time_ttmeeeee since_iennseeee
....
mean_ httpiiulrrr_olummetttetteeeeeeee meaiieeeeeaered___ [aassureed_3]]___________
Monocytes [monocytes mask .... monocytes mask 23, monocytes mean ...,
monocytes mean _ain__,,____ monocytes time since peeesmeeeeeeeeeeeeeeeee O.....
monocytes time since measured 23, monocytes esl mask 0. ....
monocytes esl mask_ 23, monocytes csl mean O, O..... monocytes sectiytessssssseeeeee mean 23, monocytes_ esl time seceesssne____eeeeeeeeeeeee monocytes_ esl leisssinnnnuueeeeee measured O.....
Neutrophils [neutrophils_ ____mask_____ .... neutrophils_mant masss_mas___ 23, neutrophils_mean neutrophils mean 23, neutrophils _eteeiiisssm__m___________
saccee__eauredee. 0, .....
neutrophils time since cerrreemeauredeeeee 23]
Partial pressure carbon dioxide [partial pressure of car__oardooooideeeee dioxide mask O....
partial_pressuure pwwsspureeeeppe of carbon dioxide mask 23,
Lecaa___.....
partial_pressuure_off wttpaaeeeeeeeeoaeeaeeaeaeeeeeeee
lii_dddxxxxieeaeleee Laclle__me_____ 23,
0, ...,
partial_reesuure_o__aabbee hatee_aabbooooooiieeeeeeeeeeeeee dioxide wwt_ttimmmeeeeeeeeee
23]
Partial thromboplastin Ipartial thromboplastin mask 0. O....
partial thromboplastin mask 23, threribbboooomlliiin... mean O....
partial thromboplastin mean 23, partial thromboplastin time since measured
partial_ thromboplastinn srce__sneee_eeeeeeeeee
23)
Peak inspiratory pressure [peak_ inspiratory_pressuree piteessure_mas_O.......
peak jeaa__inspratorr__eeesuue mask__a. peak_inspirator__presueee,
pelk_iispirtaryypressuuemmmaaa 23,
peak wwtpiiiatttr_ppeeseeeeee jetessuurettte___eeeee siie_ssinee______i_i_iiiiii sanec_reesuredd 0, .....
witsiiattoryppressuueee time sinne_sssee_ee_eeeeeee sictceemeaurreddeeee 23]
Ph in urine Iph _iiuurine__ mask O,... ph urine mask 23, ph urine mean O....
ph uriuriee mean 23, ph uhine time since measured .....
wtt__tttme_ssieeemmssseeee _3]]
Phosphate [phosphate_nn) saaeee__mss____ 0, .... phosphate_man Beaphate_mask____1 23, phosphate_mest phosphate_mean_225 phosphate_tine tet___ttt_t_iiieeee
Phosphorous [phosphorous mask__akk______ 0, ...., phosphorous_ mask_aakk___
phosphorous _aclasss_maa_____ 0, ... phosphorous mean 23,
phosphorous timntttime__ee- since measured 0, ..... phosphorous _ime sincs_sineeeeeeeee measured 23]
Plateau pressure [plateau_pressuree___ pressuree pressure_mask___ O, ... plateau plateau_pressurree 23,
plateau_pressure prressure_mean____
mean_ O, ... plateau platcau_pressurre pressure_mean_22 mean_23,
plateaug_reessure_iinn siaptureettte___eeeeeeee lrt___icceemmuuured_ee
Platelets (platelets_msateteeeeeeeesessee 0, platelets platelets_mmak_ mask 23, platelets_mean mean O....
platelets_mean_22 platelets_tte time since since__massured_
platelets_time_sineee httpg_uimee_siccccmmuuurrdd____
Positive end expiratory pressure [positive anppeppirttoryyppssuueeemaaaa
positive_eent end l_t__eccrttt_ppreeeee pressure_mmen 0, ....
lit___eieeemmuureedd 0, ....
positive_end letpeeeepptt_ppesseeeeee
time_tteeeeee since measured 23, positive_ end wwwprettrryppeesseeeeee mask 0. ....
positive_ent exprraaryyypreesureeeetnemm 23,
hatprveeedd_ppiiiooooeeeeseeeeee positive_end_em
cend_exprraooy__presuueeeee positive_end
htt_ttte__sineemmsesrreeeee .....
83

---

Post void residual [post_voide void residual mask ..... post void residual mask 23, post_ void residual mean 0, .... post_ post_void void residual mean 23,
post http:/eeduuaa_tmmme_eeeeeeeeeeee _tt___siinneemmssueeeee.e.. mts_mmmasudde__ee O.....
post_void_residua__oe
Potassium bannassiummmaske
potassium_mask potassium_mean O.....
potassium_ment meaallmmmmmmmmm__eee 23, potassium _itit_ttt___i____
Potassium serum (potassium_serum mask_o,......... potassium serum serum_mask___ 23, potasium_serrm__nen
potassium_senve seruuuuuusssmmmeeeeeeeeee serum__ean_233 23,
potassium_serum salimesiine___ee_eeeee_ie___ siame_siine_mmauureedd_e measured .....
potassium_ serum ceerum_timeeet since measured 23]
Prothrombin time tested with INR [prothrombin simbbiiin_tmmmmmeeeeeeeeeee inr par_mask_00 maoss_0.... O, prothrombin time inr mask_ 23, prothrombin time inr mean O........
prothrombin time inr mean 23, prothrombin time inr time_ssiceee_____eeeieiieettteet since measured 0,
.... prothrombin time inr time sesssssnee____eee sitice_meaured__ee _3]]
Prothrombin time using PT [prothrombin_ting time_pt_mask_0, prothrombin prothrombin Brcpbbiinttmmm____ _ime_pt_mean______tttt prothrombin _immme____tm____________________ since seccce_mmauurdd__eee 23]
23, prothrombin_ _imm__tmm________ hain__t_mean_0, .....
mean 23, prothrombin tearbbiinttmmm____e time_pt_time pet_ttim_ssineeee____eeeeeeeee_e cateeeemaaa______
Pulmonary artery pressure ppulmonary_artest Beceery_presue___ig wressure_mask_00..... mask_0.....
pulmonary_arr arttarrrttesssseeee mask 23, pulmonary_maiaa aataarrttteessseeee macessurreema____eeeee mean 0, ....
gulmonary_arterg presssurerruueeeee__er seaerre_eea_____ 23,
Satponaary__rrtt___eeeseee time__ti_________ se__ssscee____e__e manue_meesured.mm O.....
mulmonaryyartery__rrsseee time time__sicee_____ttttititttttee e since measured 23]
Systolic pulmonary artery pressure(pulmonary pressuuee sostuuuessstttciceeeeeee mask O.....
pulmonary aattrrrryyaatayaaaaaaee wttplypppesseuueseeeeee systsssssssccccc mask 23,
siiie_since_mmaeeeee 1. O.....
_2]
Pulmonary capillary wedge pressure
.... http:oonnrryccccllarry.am wedge_ pressure mask_23,
wedge_pressurre
O.....
pulnonayy_coplllr_ccddeerrrsssse
O....
p time ceneesssseee__eeeeee monosuure 23]
Red blood cell count [red blood cell_ccuunt meask_,.....
red blood cecl_ceelllleeeeeeeee collt__ott____ mask_ 23, red blood cell coluttuonaannn Sount_mean__0 0,
red www.o_looollll_eeee_eeeeeeeeeele cecl__ccutl______ coonttmman______tee 23, red_reeooeeeeee
red blood www.o_oooolllleeeeeeeeeeeeleeele celll_count time_since. Sotee_mmaa_______
Red blood cell count ascites [red_blood blood coll__oottaastts_______ _.... red blood_celllllloeoee__eeeeeeeeeee
@oount_scitte_mnee _aa__aask2222 23, red bed_blood whl.h_ooelllloounneneee count ascittes_sssssssssseeeese___ee mean 0,
red blood aalood_cell__e lll___cel____u___ee calll_count_ee count_ascites__________ Lascitts_mean____ 23,
red wwwooloooollll_oeelleeeeeeeeeele Beroddcell_eeee hat___clllll__eeeeeeeeeeee aww.ttessitttttee....
_0,
red haaoo_ecee__oonttttttsssseiiiii siime__iicc__mmsure__2333
Red blood cell count csf [red blood_ _ell call__oont_ceeeel couuttcsff_aa mask O....
red blood_ cell counttcouccoceeeee esf mask 23, red blood cell cell_count esf mean
red blooolloooohceeeeeooooooooelebee _clll_eelee_aeeeeeeeeeeeee cal___ooou_____eeeee 1aa__maa__221 23.
blooo_ccollllleeed_ebbeeeeeeeeee red blood cell coluttouuuccceeeeeeeeeeeee esf time time_since, aance_measured 0, ....
red blood cell_cccunt____ count esf time_since m___uueedddd__.e 23]
Red blood cell count pleural [red blood cell count pleural mask O....
red red blooo_ooocclll___
__,, red__lood____eeeeeeeeeeeeeeeeeee blood_ooollllleoeeeleee cered__eel_lcuntt__eeeeee_eeseee coouuttpleural_mmtt mean__
Becll_count_plerre _ectural_mma____ 23,
red bllood_cellll ceell_count_peuuee fiteural_timee___ sineeessseeee_eeeeeee
manne__mesuured O.....
red blood_oooooooooooooooooooeeeee_e cet__cell_o__eeeee cell_ccuunt_pleraal _____tmm__iiiet Bttpeesccicccsuuue_____
Red blood cell count in urine fred_blodd _ellddeeel____ee cell_eell___eeeeeee coall__cut_______eeeeeeeeee cotttttuuin____e _ww.uuumais____________
red blood_coo count urine mask 23, red blood cell count urine mean 0,
84

---

red blood cell count urine_ mean 23,
red blood cell count urine time sime_since
since measured 0, .....
red sdd_lood_cces ceaeod__ellccuut_uuul saint_urtnttttsiineee saaiee_mmasured_222
Systemic vascular resistance [systemic] lattinnicvvssuaarreeeeeeee
vascular_reeistanee__ssss
Baalemic_vascular_cce saap___rssisstee_sss... saatiiee_mss__3__3
systemic sestemic_vascularr resistance mean 23,
systemic_vasenn wtwp:aaaauuuussitiiie resistance time since_ seeeemmeaeredd O.....
musscularrresistnnee time toteetiimeeeeeeeeee sincessnee___eeee_ measured
_ince__mesuureddd 23]
Tidal_voluma dital_volumeeobservedl [tidal_ volume oobserve__aase
tidal uol__oolluee_bbbereeddd mmask_23 tidal_
tidal aoda__olumee_obsereeee Laateerrdddmeee____eee 22,
sinnce_messured_22)
Tidal volume Loidalvoluueee set___e____________ matt__mas_O.....
volume_set_mase set_mask_22
haal__volume_see set mean 1,0aaa_______ .... tidal. volume set mean 23,
didal_volume, poolume_set_timee seceeesssee_____eeee mariceueeeeueddeee O.....
tidal_ adda_volumee_see set time since measured 23]
Tidal volume spontaneous [tidal_ Laliall_vlummeeeee Ltt__vvlllsoooaeeeeeeee sponttanoonsssssssee satiss_mass____0 O....,
rida_woluume_ssontaeeuuu _aataaannuusssss____e 23, tidal meanaasssmmm_____
vida__volumeessonntaeeusssss _e
tidal_volume__sponaaneeee http:/uusssiieeesseeeeeeeeeeeeee
tidal_voliun susiuume_spottncouuss _attaaecouuutmmeeeeeeeee since mecsurede 23]
Total protein [total _artall_r_eiin____ mask 0, .... total _acall_roteei__eee
1aap__mas_____ 23,
total Bacl__pooein_mmee potal_protein_tine sirreiit_tteesiiceeeeeeee since measured 0, total protein pootein_time__ee since_measured 23]
.... total pacatllprttein__ee mean 23,
Total protein in urine [total Lonal_protein laanen__skk___ 0, ... total protein _ai__uuii______ mask__ask___ 23, sati__urnee____ee urine_mean__2
total _tt__poottnuuune_________ Baaele_mea_____
total_protein_urinee Baaeeeet_tttteseeeeeee siinseneeeatteese
....
total_protcin_urine_iiinn saatn__sincemmmsureee
Troponin [troponin i_mask O, ... troponin_ mask 23, troponin mean roponin_i_mean 23, troponin_ wetttttmeeeeeee_eeeeee sea_snce_mmauure__....
Biptomin_ittme__eee httpssicceemmmuueedde_e_e 23]
Troponin_t [troponin_t_makk,
.... troponin_t_men __,, troponin_t_menn
troponin mean 23, troponin t time sincsssnce___eee measuredd O.....
troponin_Le time since measured 23]
Venous pvo2 [venous __tps__v22____ mask O,.... venous pvo2 mask 23,
venous _vo2 mean wwwa_meen....... venous pvo2 mean 23,
Satpll_oooooomo___11111 Litpt_pvo_tttm_siiii ditiieiincemmmaaureeedooooo measured .... Senous_pvo2_nnee
foo___tmm_ssiee_neeee [attnc__mmaured___1 __3]
White blood cell count in urine [white blood ceul_cccunt_____eeeeeeeee_ @trine_mask___ 0......... .... white_blood
white_blood. ceell_couttuurie__nnee
white_blood
wwwut__uuin__eeeataaa arrie__een__23.
white blood cell count urine time since measured O,... white blood cell count urine time since measured 23)
E.6.2 Hospital Mortality
The selection procedure for the task 'Hospital Mortality' is discussed detail in Appendix
Target: Hospital morality (that the patient dies at any point during this visit, even if they are discharged from the ICU to another unit the hospital). [mort_hosp]
Shift: Insurance type (Medicare, Private, Medicaid, Government, Self Pay) [insurance]
List of causal features:
Age in years [age]
Gender [gender]
Ethnicity [ethnicity]
Height [height_mean
Weight [weight Brigght_mean_00 mean_
85

---

List of arguably causal features: Bicarbonate [bicarbonate mask O.... bicarbonate mask 23, bicarbonate mean O,. bicarbonate mean 23,
bicarbonate_ tittintttttttt_ttiiee sinesssce_____eeeeeeeeee mww.umedaaeddep.mmmm
bicarbonate_ tipe__tte____e_ee ttttttsiineeeeaaurrreeeeeee meseereeeadddddd_e __
Co2 [co2_ ____mask_____ O...., _o2_mask__ 23, co2 112___man___ me__men__o.... .a.am__....... ,o._ mean 23,
co2 _ime_ime___ since__massured Sartsuured__...... O,... co2_time____ sinesiineeea measured 23]
Partial pressure of carbon dioxide (pCO2) and end_tidal CO2 (ETC02)
(co2_(etco2__pg) pccp_ptotooopeppp
(10____s______ O..... ce2_(etco2___ 2ec22__po2_____ peo2_etc) eeee_etccsss 23,
co2_(etco2_peot 1acooo__co2__et____
O..... co2 (etcco___po2222 eacc_ett____eeie _ee____tt__mm__________ 1etl__eaa____0 _3, _etc__tmee_________________e____ te__ttmm__iiig senn__ince__a siice_messured O....,
co2_(etco2_paet 110___po2_eec___________________ (rcoo__tc)_ttte
co2_(etco2_pco22 _eooo__ec___________ _e_c_tt___nnn sttc)_time_____eeeeee_te_ Stttl_ttesseeeeeeeeeeeeeee since_measured, 23]
Partial pressure of oxygen [partial_pressuree cotrssurr_oooogeeeen
O....
partial_presuure_o__ooy
partial_pressure_of_ooom cacto__ean___0 0,. partial watpartillpppsuureeeeeeeeee prwssureeeeoo of ooaesureeooooooooene_eeeee Leegooxyggnnnnee mean _3,
O....
Fraction oxsprredxeesireeoxxxeoeee oxygen ffractioniinspieed anspprred_oxygennnee mactse___ss____ _.............
"raction_inspiredd www.oooggggssa...e 23,
oxygen
oxygen__ean mean O.... fractionn dispired_oxygent beaayygeeemannn___eeeee mean_ 23, lace__maauure_____ O....
(ractionjisspicee oxygen__tinggg
fraction_inspired_ie _oygeen_____e_e owwgen_ttieeeaeeeeeeeeeeeee seseee_ttt______eeeeeeeeeee time_since, fine_since_measured 23,
pration_inspired Sxappieedoxxyen___oo saci__sst_sssss_s_eee_e maask0..... fraction_ inspired oxygen_geeeeeee set_ssss________________________
fraction traction_inspired oxygen set meammmman__. O... fraction inspiredppdddde oxygen_seee mean _2, fraction inspired oxygen set time since measured O....,
sime_since
fraction inspired eeppired_xyygenn. oxygen set set_time_____eeeeeeeett_ssttettt time_since measured 23]
Glascow coma score [glascow_come cwwaa/scaleeooooeeooooooooooosoo Litial_mask0... O....
glascow_coma, saaaaassalee_eee total mask 23,
glascow_comat scale_totat mean ....................... www..ggllawwwwwooo Batpe_eooma__aaeeeeeee comaassale_totae mean 23,
glascow http://oommaaaae scale total time since measured O.....
aaaeoo_cooma_saleffotaala Ltt_t_ieesicceeeaaiiieee __3]
lactttt Lactate [lactate_ mask O.... lactate mask_ lactate mean 0,... lactate mean_mean__ 23, lactate_timeg sictiie_ttme_siiii since_ measured 0............... lactate time_since, Laanccemeesured___ _33
Lactic acid [lactic acid mask 0.... lactic acid mask 23,
lactic_aae acid mean 0.. .lactic acid mean 23,
lactic_act hime_since, sactce__eauured__e O, lactic lactic_acid_ acii_ttiiee__de_eeeeeeeeeeeeeeee time__since_messureed__
Sodium [sodium mask 0.... ... sodium, mask 23, sodium mean O.... ,sodium mean 23, sodium time since measured O....,
sodium time since measured 23]
Hemoglobin [hemoglobin mask 0.... hemoglobin mact__aass_____ 23,
hemoglobin _eaata_mmaa_____ 0,. .hemolloooll mean 23,
hemoglobin Limebiii___tmmme_eee time_since since measured __.._.._. hemoglobin time time_since_mmauured 23]
Mean blood pressure [mean blood_pressurre
pressure mask O....,
mean mean_blood Lal___prrsee_maaae
23,
mean_blood Bectldd_ressu_eeeeeeeee
paa__meaaa________ Srrean_O....,me mean blood blood_pressuree pressure_mean
pressure mean_ _3a__
mean blood prrrssur_eeuureeee__ time
tite_eiiceee_____eaeeeiooititee since measured, 0.....
mean Licia_blood__pae gelloddppeesurr_tiinn simm_ssinee_mmsuuredd _3]
Oxygen saturation (oxygen_saturr)
_as__mask______ 0.... soygen_saturation _aaooom_maaa_____
oxygen_saturation Sali___ea______ ..................... mean_0.....oxgen. saangen_tturattionnno maaioon__mea_____1
paygen_saturation_tte dime_since, lere_sncceemmuuueeddeeee O....
oxygen_saturea simeiioonttit__ee_ee since saacce_meaurred_ee _3]
Ph [ph_mask_0.... aah_mask__00 23, ph_ phlpppppaee____eppppp mean mean_0........ O.. Lahh_mea___ 23,
0,..., th__tmme_sine maaiieeeeaaured____e 23]
Respiratory rate [respiratory_ert mask O.... respiratory_ree pate La___rrt__a_________________ mask 23,
respiratory_rate mean 1aiim________ ....iirar rateyyyrateeeeeeeeeeee vate_mman___ _3,
respiratory_rateettnnn_-jii meanueeeeeuured___ O....
Seppirator__rae_ttmme. maaeee_eeauredd__m 23, respiratory_rate_ set_assssssssk-ee--__eaa___ O.....
respiratory_rate_set mask_aa, 23,
respiratory_rate_set__no (.... seee_meen_......pprrttoot.tttt
86

---

respiratory_ rate_aeeetttt_eeeeee time_timmiiee since_iiecceese measured O....,
respiratory_raa rate haarl__rrr_eeeeeeeeeeeeeeeeeeeee set time since measured 23]
Systolic blood pressure [systolic blood bloood_rresuur_mmeee mask O.....
pystolicbblood.1r @lood_pressure _aatrre___ss______ 23,
systolic blood prroddpresuureeee mean 0,. syytolli blood porisdppreeeeeeee Saaeee_meaaa_2__ 23,
systolic_blood.oo
satrte_ttte_sieee_eeeee maanee_mmauure___...
Heart rate [heart_rate_mant O.... heart_ rattrrattt_____eeeeee
heart_rate sa___mea__0__ 0............... _heart rate mean 23, heart preart_rate____ simetttme____eeee time_tiieeeeeeee since_ martce_masurrd_O.. .....
heart_rate_timme
lttarrtttteeeieeeee__eeeeee cineesssnee___eeeeeeeeee macoce__eauurddccmmo 23]
Temperature [temperature_i] sas___aass_____ O.... camperature_maske
Cemperature_mean__1 mer.mmannn.....mpeee.. hemperattreeraeeeeee_eemmmee
temperature_tinn fttttttureettttteeeeee _iicscsscee______ee measured O.... temperature time sew_sssnee_____eeeeeeeeeeeeeeeee mwcpsreeeaeeedeeeee _33]
White blood cell count[whin white
crod_ccll_counle maattt_mask__ O....
white_ bts__bllod_ee__e__ peoo_ccel_ccontt, Soantttmss____ee 23,
white blood _ae___ll______ cell__cuutt__ Lall_coottmmaalt mean white _al__lllodddeeeeeee ceult_couttll____eeeeeeeee_eeeee count mean 23,
white blood cell count time since_ss_________e_e_eee measured O....,
white blood cell_countt time wwwsssnnnncsiieeeeee _3]
List of other features: Height [height_maal mask_0.....
height_meot feahht_eea______ .height mean 23, height_ timm_sinccemmassrreee
Beright__een____
Beight_time___ since_sisee____e_e__eeee_eeeeee_ cectseemeauuredeeee __]]
Weight Brright_mask___
weight__men__.. hranaan_....weeiht_nnneee wwightt_men__22
weight_ time since measured O.... weight time since_ measured 23]
Alanine aminotransferase [alanine_aminoteeneeeeeeeeeeeeet amminotransffrrss_mmsss. om.eee_..s.........
Laanine_aminotraasffraass Lactrraassemeee_____ mean_0..... wwl....lanneeeeeee amminotransserrse_mmma 23,
alanine saniiiaammmtttsffrrsseeee Lime_siicee peice__ederredd O.....
alanine aminotransferase time since mecaurrd 23]
Albumin [albumin, mask O.... albumin mask 23,
albumin Seamiinmean_0ee a maa__mean_____ albumin, wttpttttieeeeeeeeeeeeee O.....,
albumin_time sineeesssseeeeeeeeeeeeee saciioe_meauuredd. _3]
Alanine aminotransferase [albumin ascites mask O..., albumin ascites mask 23, albumin aabumin_ascites_. ascites_mmaan______e_ _ ......,,lbumiin ascites mean 23,
albumin_ascites http_asccitessttm______eee since_mmeaaceess O....
Albumin pleural halbumin_pleuree www._ma.s........_. O.... bobumin_ppeurge pleural_mask mask _2,
albumin_pleural_mee www_mmaann_.... albummin wttptiiiiineeeaammmeeeeeae
Bittrral_ttmmsieeee_eeee 00 0....
albumin_pleural http_pppeerra__ime__sii_e_______
Albumin in urine [albumin urine mask O.... albumin urine mask 23,
caabumin_urine_eee
@oumin_urinee_o Bacee_eea___u_ ..... albummin urwne_rriiaaa Serine_mea__21 23,
albumin_urin urine temeeteeeeeeee faane__since_mmeeee wwwsured O.. albumin urine time since_seeee_e Lance_measured_.m
Alkaline phosphate [alkaline_phos/i] phwppphhhhspppppppeeeeeeaeeeeeee
O.... alkaline_phee alkaline_phosphate_eeeeeeeeeeee __pppphottomssskk
alkaline phhssphtee alkalinnepphopphaee mean 0,... alkaline wwwpppppphpppppheeeeeeaeaaaaeeee meanemmmaa___ 23,
alkaline Bpcpllehhhpppppeee time since measured O.....
alkaline
timee_inceeeeee_etttttteettttee alkaline_phosphate -hhosphatee time since mewssredaaeeeeea 23]
Anion gap [anion_gaj] lai__aaapass___ O.... anion Lisoo_gap_aate
haap_ask__22
anionn anion Baroongap__o gap_mean O, ,anion gappionnnapp_aa gap_gaapppgaae mean 23.
anion gap_ggppt_____ time_siice since measured O..., anion_gap time since mewsuredeeeeeeeeeee 23]
Asparate aminotransferase
aminootaaasneeeeeaseeeeee Sacpseerreee_ssssaae_ee mask O.....
asparate_ane www.notraanffeaaeeeeeee saaelee_ass_______ 23,
shparate_aminottaafferrsse___ mwwn_m.........a .ttpp_.......... asparate awinotransserase_meee _ean_____ee_____ meaa__23,
asparate_aminotraasferass time seree_ssceeeee_eeeeeeee
sime_since_measuree micieer_eauurdddcc O....
Basophils Stasophils_maske O.... aasophils_maske 23,
pasophils_mean aete__.....k Serphhilsmmea__2111 mean_23, basophils simeeeissstte___eeeeeeeeee time time_-iieeeeeeeeeeeeeeeeetieeeee moreeeeeedeedd_00
basophils time simee measured 23]
87

---

Bilirubin [bilirubin _elirubin_mask_____ O...., bilirubin mask 23, bilirubin, mean O,.. billirbbn mean 23, bilirubin time since measured O.....
bilirubin_time caarbbintttmm___ee _ttpttttteeeeee____e_eee Ltt___siceemmameed_______ mwwsured___]]
Blood urea nitrogen [blood urea marannittogennnn mask O....
blood wrea_rrelli
arra__itrogenn.mm mask 23,
blood wrea srea_nitrogen mean O,......oo. urea mitrogen_mean mean 23,
abbod__rea_nniroggeer ciiimeesinee_iiee send__urraniiroge_iimeeeee
lrre__innceemmsurreee ()...,..
Calcium [calcium mask O.... calcium mask_ 2_,
calcium mean 0.. calcium calciuum mean 23, calcium time since senee__easuredd O.....
calcium time since measured 233
Calcium ionized galciuum_ionzzdd Baateedd_mss_0... calcium calcium_ionized 23,
calcium ciciiijoooiidd_ine _a___mee_______ calcium, cateinjooiiiieeeee mean___ea______
calcium ionized time since measured calcium ionized time_ sime_since
time_since
O....,
since measured_22 23]
Calcium in urine [calcium_ urine mask O..... calcium__rinn
palcium_urine_0 caanne_mean_O.. Ineea_o.....clciium cacaaciuuuuurinemmeaaa
calcium urine s_me_ttmmm___e___ since measured O.... calcium urine time since measured 23]
Cardiac index [cardiac_inge cinitacc_iie_____ mask Booiok_....._ cardiac index_mask_22 23, cardiac_inde
cardiac wttpaaaadacccce___nneeee indeinnddeaaaa 23,
Bardiac_index_tinee Siince_mesuuued__eeeeee O.... cardiac cardiac_index_iee feies__tmm_ssiieee time_sincc_mmasurred
Cardiac output by Fick principle [cardiac_outpe] Baciiaace_uuuuu_eeeeeee fick mask O..... cardiac_ond Lactlac__uuuuffftee __i___cc______________ lock_mask__. 23,
cardiac cardiac_output fick mean carddaac _wt.diaccouupuuutttttoaaaapppeee fick Lorkk__ean___ mean__en, 23,
Cardiac output by thermodiluttin [cardiac_ooe Bacclllc___uuutttteeee
tatptutttttmmiittionnmnnm ma.k_O......
cardiac cattrr__utuut_tte output_thermooleeeto thermodilution mas__aak____ 23,
diac_ haau__otput___e wwtttottttertruuuootee thermodilution
Satttttiioonnnnnnaeeee mean _aadiiacooutput thruttttthomoiuttionnn mean 23,
cacpc__oouuuuuu_hhhhotoee siatriionttmeeeeeee time_timeee laanee_sice_mmessrrreee
O.....
Central venous pressure [central venous parnoousppeesureeeeomm mask O....
mask 23,
centralLyaouus.peessttteeennnnne
orw. venouss_rresuure_mman O.....
central__enoos__nessuureeiiiiiie
Chloride (cclorrdeen mask 0.... cwloride_mmeeee sas___mas_____ 23,
chloride_meant 0........... Baatooridd_eee_______1 mean chloride ttmm__innc__mmssuree___e
measured_0. O......
chloride_time_since- lrlee__iiicenuuuud____e
Chloride in urine [chloride cthloride_urine..o maskes_maashh_ O.... chloride_ urine mask_aaskk.. 23,
chloride_urine mean 1acaaa_______
wurine_meannnnneeeeeeee m _3,
chloride_ juinee_rrr_____ wwwpuuueeineeeee_eeeeee httpetttteeeeeeeeeeeeeeeeeeeeeee since measured 0............. chloride urine time since_mmeaseeeee 23)
Cholesterol [cholesterol mask ..... cholesterol _ 23,
cholesterol www.nneaaanccc 0,.aa_00.___ ccw.lettrrolllo mean 23,
cholesterol Birooott_tti___eeeeeeee Liine_since___ paciieremeaaruedeee __......_______._______. cholesterol witeooolttteeeeeeeeeeeeeee t__e__imem__ee___
lelpe_ssiccceerreeeeee measured_ 23]
HDL cholesterol [cholesterol hdl__mask 0.... .,, cholesterol aadl_mask_221
cholesterol hdl mcan .............sss.. cholesteerol hdl mcan 23,
cholesterol hdl time simme_since____e maauuured O....
cholesterol lidl__tmmesstee sime_since____ caaincc_mmauurdddeeee 23]
LDI. cholesterol [cholesterol_id] latteerroodddd_mmskk _al__mask______ _anask......... cholesterol_ld__mee @idd__mskk22_ cholesterol adl_mean_00,
cocnoleetero__dd_meeeeee 23,
cholesterol ldl time since__ssee____eeeeeeeee measured 1.reeedd__....... O....
cholesterol _dl__iittt time Lime_since sacicceeeauurdddcc _2]
Creatinine Creatinine_mmak_1 creatinine Lacee__eaa____
greatinine_mase
0... .............. ,reatinnen Laaeoiiiin_eeee_____ mean_23. creatinine time_ since measured O....
creatinine Limtinine__imee_____ since sincee_easured 23]
88

---

Creatinine ascites [creatinine _sccitsssssesses mask o.... creatinine ascitessssssaa mask 23, creatinine _asiiittieeesseesseeeeeee mean .. creatimine ascites mean 23,
ereatinine_ascites. tatrssstttte_iiieee siimessice_mmaauueedd mes__meaauredd_. __.......
Creatinine body fluid [creatinine body mask O....
creatinine_boke bodee_boo______ee fluid _al__mask_____ _3,
_acpiliiineeo______ ff mean O.... creatinnee body fodyyfluud__ fluid mean 23,
1_o.ddd______. O.....
creatinine_bodd_fuuicc time since_ httd__iii__inee_mmsueee_e_e_
Creatinine pleural (creatinine pleuraallpppaallee Laneuuralmmass_._ O...., creatinine peeainine_pleurall _aaaal___sk______ 23,
creatinine_pleural_ Baaiaal__mam___a -reato.....creatiiinn batptttiiiiieepuuaalleeeee plwpralleeeeeeeeeeaeae mean_23.
Segattine__peuura_timee_eeeeeeee wasainccemeauureddcm O....,
Creatinine in urine
O.... creatinine_uring 23,
vreatinine_urine_. out.:rnneeaan_.....rrraaaaiiiiia ceneeatiinee_rine_neee aarine_men__22 23,
creatinine_urrie_tiine sitme_since_mmmsuree_/l/
__]
Diastolic blood pressure [diastolic blood proosuur mask O....,
diastolic blood Brreodd_pesureeeeeeee _acrere__ss______ 23,
diastolic_blood paressure_mee____
diasolllii
satassurrdddd........... O....
Eosinophils [eosinophils_manl 0 eosinophil saa___mask_____ 23,
eosinophils_mee _eclll_s_mmme____ O www.ippoi.iippiiiissssssseee Sattiiopphhls_mmen__1_ 23,
cosinophils_timg Stilllssttteee_eeeeeee ciaeeessnce___eeee wwwne__measurredeee
eosinophils_tiee sttisstttte___eeee siine_since_mentt Liince_maasure__22
Fibrinogen (firrinoge__mall
0.... fibrinogen_man __k
fibrinogen_men mea__mean_a_ 0, fibrinooe, mai__meaa____1 fibriogen_timee aame__sine_mmauureee
fibrinogen_time. caanee_sinee_mee Laaeee_sinceemssured______
Glucose [glucose_man _aaeee___m_____ O.... glucose glucose_mask 23,
glucose_mess mean 0,. ......luoooe mean 23, glucose time since measured O....,
blucose_time__ siaeeessssee__eeeeeee sinne_measurrd _3]]______________
Hematocrit [hematocrit mask 0.... hematocrit mask 23.
hematocrit_meen Bacii__eaan_g__ .hemattrettt mean 23,
hematocrit, hematocrit_time_
laaiicrrttttimmsiieeeeeee ciine__since_e_e Saaince_maauuedd__c 0.... hematocrit time since maaeeereeaeree____e
Lymphocytes [lymphocytes. mask O.... lymphocytes mask 23, Sacatttteee_____eeeeeee
lymphocytes mean 0,. Jymphocytes mean 23,
lymphocytes time since measured O,... lymphocytes time since measured. 23]
Lymphocytes ascites syyppoocyts_ascciie ascites mask O.....
symphocytesaasciine asciitsssssssees ascitessmask_2210
secahoocteessscitteee _aaissss_mma___eeeee O.... .h....lympocytts.... saaphicytes_asciees.ee caaess_eeaa______
lymphocytes saapcittttssasciiessss timessst_tm__eeeemmmeeeeee sel__siceeeeee_eeee peceremmeaureddee O....
lymphocytes assitttttsssssstssseeeeeeeeeeee time since measured 23]
Atypical lymphocytes [lymphocytes atypical mask O....
lymphocytes stttopyyyytsttttclllllleal mask 23,
Saaphocyts__ttyical_neo Sacaall_em_______ "...hh......mpotttte httpllsssttttttttttaee Lacaall_________ _3,,
yymphocytes_ayypicll http://sssstttttllt_eeeee tit_tttmmeiee_eeeeee measured 0....
yymphocytes_atypiall time ssnne measured 23,
ymmphocytes_ayyical_ss____
0 lym.locyphhhh.... yympphoyyts__typiccllooo _as___msk22___
ymmphocytes_atypiillcc..... hltp.o.....ymmmoceeeeeeaaaoo
yymphocytes_ayppial_cs// Ltt__iim__iimeemmmsuee_______
ymaphocyee__ayyicalccss... siree_sinee___ee
wwwsured_ea.a..mmmm 0....
since wewnssneeesuueeeeeeeeeee mww.umeed__me
Lymphocytes in body fluid [lymphocytes_hnn _odyy fluid mask O....
stathhocttt__ody__flle _tluuddduuadd____e_e_e 23,
lymphocytes_booke httpr//eeebbboooiiiiiimaaa...... Sordddmea_O...
.ymphhotyesttttsssssss.ooo.o.... body boddy_flid_meaal 2iiid__men_2....
lymphocytes_booke body_fluid
lymphocytes_bodde bod___ooy___f_ bddy_fluidttim fitiid_timm___eee sinee_iice___ seatreemmasu_e_____
Lymphocytes percentage [lymphocytes_percee mask O....
lymphocytes_pereee
porcent_maskk_.
89

---

lymphocytes pereeett mean O,... .y..h.yymhoestts _etrss_eeeeetteeeeeeee mean 23,
yymphocytes_percene time since_
meassureeeeuredd O....,
ympbhccyees_erreen__ineeee.e
Lymphocytes pleural Jlympphocytes_peeraal mask O....,
yymphocyyes_pleural_maaaa Lacteuural_msk223.33
yynphocytes_pleuua__maan ottp:/arraaaaaa.........aaa hww.aa/aa........yyeeeeeeeeee.rr mean_23,
http://e__eeeuuaiiiisiisieeeen_e .....
Seaphocytes_pleurree time Lime_since_____ pateeeemeeuredd_eeee 23]
Magnesium [magnesium_mant O.... magnesium_mast
magnesium_men me___mean_______ .aa.nessnnnnnummmmmmmmm.. Bactlluummeeaea___
magnesium time since_ pacteemmeeueeeeeeeee O,.... magnesium time sinessiinnsseeeeeeeeeeeeeeeessee measured 23]
Mean corpuscular hemoglobin [mean_corpuuell http:usccuuurhhmmoooooooo mask_O...... O....
mean_ mean_ corpuscular hemoglobin Batgglobin_mass____ 23,
mean_ corpunccuueellar
satralaleemooooooneeeeee _ea__meaaa___ 0.... me.n corpusuuuruulllr helilllliooooooomnoe Leregobinn_men____ mean 23,
mean_corpuscule suapuuscclar_hhmmlloiinn time time_sinne measueee O....,
mean mean_corpussculrr_heee cogooppuseuarrhhmmooooooinggnnnn httpelobbiiniieeiinnneeeeeeeee
23]
Mean corpuscular hemoglobin concentration
macis__ma______ mask_O.....
mean_corpuscularr
23,
mean ccrppsocuuuuulrr hemnogoobn
time since saceeeemeurreeee 23]
Mean corpuscular volume crreanncorpuucularraa volumeerarvoooeeeemeemmm hatpercullr_ollummeemaok...... maa._mass........a
mean cetta_ooouucuulrwveeee Soluurrrooomeeeeee Dollme_mask__15 mask_23.
mean httpe_oooouuuuurweeeee volume
mean a voluieooooeeeeeeeeeeeeeeeeeeeeee mean 23,
caauu_coppucuulrrrvooloooo volume time cccee_sncee___eeeeeeeee penoe__meauued_cmm O....,
mean mean_corpuseeee wwtpppccoouuuuurcccoo
soouuuuullllooleeeeeeee volume time since measuree 23]
Monocytes [monocytes_mie Sactyyttesmss_______ O ... monocytes_man maak_233
monocytes_ Baciiiesseeeeee___ monoyytts Letooooctte__mman____ _3,, monocytes sennncyees_timeee_e_ since_measured O.....
monocytes_timg sacessstttmmstseeee cive__snce_mnee maruce__mauurd___.e __, monocytes monocytes_csl_mant sastoessss_msssO......
Betecytee_ss__mmasee
monocytes_sceeoco sectyytessss___meeee oet._eee__.......oo... .om 2etprteee___s___e______
monocytes esl time since, measured 23]
Neutrophils [neutrophils_mant masiilss___s______
neutrophils_mmnn 2aathill_mss______311
neutrophils_ment sas___eea______ .n..rroout Saappppiilsmma____ee mean__3.
neutrophils time since measured O.... neutrophils time since measured 23]
Partial pressure carbon dioxide [partial pressure ca__ooooonnccoooooaeooo_ooooeeee dioxide mask O.... partial Biceiaal_ppssueee__ _f carbon, dioxide dioxide_mask__ 23, par-
jial_pressure_of__e_ of of__caroo_diixiieeeeeee.eeoa @roxide_meen_____ mean 0 Op.rtal partiial_presuureeeeeepppeeeee__ ofwpoooooonddoiiiiiieeooo 2eaaod___exxid_mm______._
partial partial_pressurr_oo__ cacoleaaaaoo__eeeeeeeeeeeeeeeeee dioodddiiiieeeeeeeeeeee Liexxidd__ime______ since_ measured 0....,
parrial_presuure_o__carrte wawrrranondoxxdeetieeeeeee haapllettmm__eeeeeeeeeee Seiicc_measured__a
Partial thromboplastin [partial
Staabboplaati_mmmss___ porrial_thromboplastte mask_ 23,
thriiialthhmbollasttnn sacttt_meaa__l O.... .a....llaatall,, thromboplastin Saapppaasttt_maa_____
partial thriioopttoo tetttttm_siiiee
1.reed____...... O....
trrtial_troombollastin.
Peak inspiratory pressure [peak inspiratory pressus mask O....
Laessurremakk___ mask 23,
peak taak_inspirator_peeeee _prrssureeruueeeeeee mean 0...__....... O....ppaa__iippprtrrroppppp perrrtyprresseeeeeeeeeee _eauu 23,
peak Laai__iipiiitty_preeeeeee prepsuurreppeeeeeeeeeee ttme_tteeee__e _ttte_siieeee_____e_ee_ee sincee_mesuured measured O....
peak maak__inpiratory_pressiut pttptiooyy_presuueetteeeeee ceteessureettmm__eeeeeeee time__immeeeeeeeeeee measuueeeeeeedeeeeeeeeee 23)
Ph in urine iph_urine_manee www._a........... 0.... phlprine_maaeeeeeeeeeeeepppppppp mase__mask______ _2as
ph_urine_mean 00....ph.pppp urine mean 23, ph _uineriee__ www.uurneeeeeeee__eeeeee meanuce
ph_urine_time since 2atnnc__masurrd_2222
Phosphate [phosphate_mas? 1..sk O.... phosphate_mast saattte_mmsk_222 _aak
phosphate mea__ 0.. pheosphaee _al____aa______ 23, phosphate time cecteeemeaaeeddeeee
Saapphatt_mea_______
phosphate time since. measured 23]
90

---

Phosphorous Iphosphorous mask O.... phosphorous maassssssss_____ 23,
phosphorous mean O.. .hossphoospso mean 23,
phosphonous_time sismm_sinneemmmssrreeee 0,... phosphorous_timee htapriousssieeesseeeeeeeeeeee meretee_mauurdd___e
Plateau pressure
watpssuuuremas____ee p prressure_mask222
ptt__pprrsseeeeaeeee ......... plateau paateauupressurre_mman
plateau pressure time since measured O.....
plateau_preesteee pwwssureeeeeppeee time since Letsssiiinaaaeeeeeeeeeeee ceree_mmaureddeee 23]
Platelets [platelets Saaateletssmask_ae O.... platelets mask 23,
platelets_men meaeoess_mee_____ O,. platelets plateeees mean_ 2asan__,,________ platelets time_since wetesssssnseeeeeeeeeeee measured O....,
Positive end expiratory pressure [positive_ents expiratory_pressure.
O....
positive_ end _ _eessory_peeureeeeeeeee cressure_mask_222 23, posi-
tive live_end_eem Lar___epptttr_ppeeeeee pressure mean_
mrean_O...... 0........ positivve secplivv_enddeeee _ewpprattory_prresureeeeeeeeee
positive end expiratory pressure _ime since measured O....,
pepesutory_pressure
_imeee__t_m___ee___ee_eeeeeee since maaseermeeuredd__ 23, positive_ end wetpeeeetpppppeeeeeeee _et mask O....,
positive_ennd tee_eeeetttr_peeeeee
Sesiiie_eed__xppiattrr_____ tive pr___edd_eppprrorr__ppsssuu mettsmrrrresteeenn___eeeee set_mean__ (rrean_0...-0 p....i.ossiiiveeeeeee.
mask 23, posi-
wrwressure_se_tii__sse_ee____e Leiiicc_aaauurd____ O....,
caanee_iincemmssureeee 23]
Post void residual [post_voole void residuee mask O...., post_ post_void void residual mask 23, post_void iedddrreeuuall mean post void residual mean 23,
post post_voidd__ bociddresiduall_ time since measured O....
jost_void_residual time since measured 23]
Potassium [potassium_mask 0 ... ttrassiummmash 2__
Bectiim_mmm_a___e 23, potassium wtwp:/__emmmeeeeeemmsereeeeeeeee
Potassium serum [potassium_serum_neee saatpm_mass__... polassium_seru__mass
potassium_setee serrssiuummrmmmmmma mean .... potasssum serum mean 23,
potassium serum time_ttme____eeeee since measured O....,
potassium serum time since measured 23]
Prothrombin time tested with INR [prothrombin_ting timettimmiene.eee iir_mass_00_... prothrombin time_inr fiin__ir__mas_233 Battroobbintttteii________ hht.annn.....pootoonnnno.ooooon
prothrombin_timee tatiiitttmttitteee _t_tt___in__tmm__iiceeee_ee_____ wwwphpoomointiiiieen bittonnbiitttmeeiiee _irae_ii_eeeeaa_______ee_
O.....
prothrombin time inr time since measured 23]
Prothrombin time using PT [prothrombin time __t___________ee_ mask O.... prothrombin time_ppp_pp_ ________________ mask 23,
prothrombin time _p__p_________p_ mean O.... prothrromtnninnn time pea.aan_______eeeeeeeaeeee_ httpaii__nnin____ii_eeeeeeeeeeee
mean 23,
Pulmonary artery pressure
O....
pulmonary_ane htt_aarrrrrre_seee maak 23,
pulmonary_arm artery _eccrssr_psuueeeee_e mean 0... D0.....pummonar artery pressure P
_rterry_pressueeeeeee pulmonary_arte aaterrrrttrr_rrrrrerr pressure time_since since measured O....
p pulmonary_artes artery pressure time since measured 23]
Systolic pulmonary artery pressure/pulmonary. cartoonarrryrrrtryaaeeee ptt_epsrrrrssstoliiieee O....
aaanooaary_aater_ppresstt presssuressytoolicccc wwwssoliccmmss__22.. pul-
pulmonary_artee arterrrrtrrr_rreeeeeeee _ttssrrrrppsssuuueeeeeee systrllsssssccsssso time since mecesureedeeeeeee O.....
pulmonary_arte aaterrryyyyayyyeeeoeee peressurppeuueeeeee presssueessstoliic systolic time since measured 23]
Pulmonary capillary wedge pressure
wedge_pressuure pressure mask O.... pul- catpnnaaecaplllr__weeeeeeee pulmonary wwwprrraaapllllraaeeeeea wedge_presssuee sime_since___ measured_ _0..d____.... O....,
ppulmonary_capillaee wwwpppplppppeeeeeecc
monary_capillary
watpppppppllllaaaeeee wvedge_pressure
sacee__eaa_00 1ama__0,...
pulmonary_capille srcpot_wwdee__eeee_eeee_ time sinee_sice_____ sinne__meastee _2]
91

---

Red blood cell count [red_ blood cell count mask O....,
red blood cell count mask 23,
red bi__blood__eee Las___eel____ggg [ell__cont__m___ Sattt_mea____.0 .eean_ red_blood____eeeeeeeeeorreeeeedd _ell__cunt mean_ 23,
[e__blood_celll lal___ccel___leee cell__cont_tim_t tettttiiieeeee_eeeeeeeeeeeeeeeee
cime__iinee_mmauured measured O.....
red b___looo__eell lacll_eeel__cu__eeee
Red blood cell count ascites [red_
_es__llod___eee cell coontt count__scites_mase
red
red blood cell count ascites mean O.... .red blood ceul_ ascites mean 23,
red blood_ _ccll_eell________eeeee coll_c ascites satsstttt___________eeeee time_since since measured O....,
red red_blood blooblooddecooo _eloddeelll_aeeee count ascites time since measured 23]
Red blood cell count esf [red blood cell count esf mask O.... red blood cell count esf mask 23,
red blood_coos Lacii_eel_____eee _ol__coont____ esf mean O... ,ed blood cell call_oout_cctt csf mean_menn___ 23,
red hid__lood_ccee ioMood_cell_coutte count_ccucstsreeee cacpuntttss_tt_____________ teme_tmmmingg saance_masuueed..m 0....,
red
Limssf_ttmm________ sinceessseee__eeee Latiice_mmsured___ee 23]
Red blood cell count pleural reo__loodd___eeeeeeeeeeoood_eedd
cell__count_pleura__nee 0...__............
red__loode red htt__ell_oooot_llerr__aaaaae
_e. ree_blood www.oooooolooooooeeeeeeeeeeeeeee cell__count_peeurl__ngge
O....,
red bloob llr_cl_ccoot__perrraal Limet_ttt________e Liinee_ssec______e sinne__mcsuuee, 23]
Red blood cell count in urine [red ht___blod__eel_coo cactuntt_rine_maa___ O....
red blood cell _ount urine mask 23,
red blood cell count urine mean .rd blood cell count urine mean 23, red blood _clll_eellleellleeeeeeeeee cell_count urine time time_since measured O....,
red hww__bloo__eelloooe _ocll__coun_____ beaadd__ll_ccuut_uuieee_eeeeee
__]
Systemic vascular resistance [systemic vascular cetplurrresissteseeeeeeee ma.k_O......
castemic_vasclarrgr resissance mask 23,
systemiiyvscelarreesstttteee.. srreaan_o........
systemic_vaccular.essssannee weicccnmeesurredd O....,
Tidai_volumeeobservee
@ollume_observedd mask_O......
tidal_volume_obbereee m____mas____________
www.wweolwseeeeeeeeeeeeeooo mean "h....tidal volume obberved mean 23,
tidal_volume volume observed
volumee_bbsereed time s_ncccsseeeeeeeeeeeeeeeeeeeee measured O.....
@aaa__volume_obbl .observedd
measured 23]
Tidal volume [tidal volume set mask O.... tidal volume set set__set_________ee_____________ mask 23,
set mean ,idall volume 1at__mean2220 23,
tidal vodal_volume_se set time since meweeureeeeeeeeeee O.....
Bala__wolume_seettine___oo sectseett_ttmeeiiie____eeeeee httptiiie_iiiieemmsereeeeeeeeed
Tidal volume sponuanness wawpllovouee_ppeeeeeeee soluume_soontaneous. mask
uoddl_volumeesspptannousssooe
tudal_vvlummeesspnttnnooussssss
O_...,
tidal hidh__volume_sppottnnoous__iiii
Total protein
masiiii__aaa_____ total Litiallprottin____ 2rat_poortti_ma_____0 23,
dotal_procein_mman
cotaal_rotein_niee sereein_meaa___
ootal_protein_tingg since measured O...., total protein time since meccsuree __3]
Total protein in urine [total_protein pantein_urinn__ee urine_mask_aa O.... total protein trrine_maskk 23, total Lll__pprooouuiiie _urierrineeeeee _arine_mean_________ mean O.... total protein urine mean 233
total Lotal_protein_uring Bictiineettmeeeeeeeeeeeee sewcssiinnaaeeeee
O....
total total_prottin_urring time__simee mecsuureeeeureddeeeee 23]
(....,
23,
mroponi_i_meete i____i_mmma____ __t_ea___.....rrponniin
troponin_i since Sammeeiiine_mmsuuredd000m
troponin_ii
Troponin _ [troponin_t_mase O..... troponin_t_mask
troponin_t_mm Saciiitt__eann Livean_,..... 1tw....rroooinn___meeee ________________
troponin ttime_iince________iiiiiiiiieeee socireremmaureeeeeee O..... troponin t time since, measured 23]
92

---

Venous pvo2 [venous pennous_poo____ mask O.... venous pvo2 mask 23,
venous_pvo2 mean ,enouss _ocuus_pp_o222_ mean 23,
0 venous venous_pvo2_ine Set___teessieeeeeeeeee measured_ crcpuutt_uinnenmeee macke__mas_____ O....
White blood cell count in urine [white_bir white_blood__e cecood__el_ccuttt_eee
mas__mas__2__ 23,
white blood cell count urine mean mean__..... white. blood _ell__ell_e_e couut urine mean 23, white_ blood cell cell_count___ urine time siceesssce____eeeeeeeeeeeeeeeeee sictc__meauurdd_. O,...,
white blood cell count urine time since mcasured 23)
E.7
TableShift: NHANES
We have one task based on the National Health and Nutrition Examination Survey
(NHANES) [Centers for Disease Control and Prevention 2017 14
E.7.1
Childhood Lead
Target: Blood lead (ug/dL) [LBXBPB]
Shift: Binary indicator for whether family PIR (poverty-income ratio) S 1.3. [NNDFMPIRBlloCuuoi]
List of causal features:
Country of birth [DMDBORN4]
Age in years [RIDAGEYR]
Gender [RIAGENDR]
Race and hispanic origin [RIDRETH merged]
Year of survey [nhanes_
List of arguably causal features: Highest grade level of school completed highest degree received [DMDEDUC2]
List of other features:
Marital status [DMDMARTL]
E,8
TableShift: Physionet
We have one task based on the 2019 PhysioNet Challenge Reyna et al. 2020 2019 15 released by PhysioNet [Goldberger al. 2000).
The data is
E.8.1
Sepsis
Target: For septic patients, SepsisLabel is if 2 f_sepsis - and if < f_sepsis - 6. For non-septic patients, SepsisLabel is O. [SepsisLabel]
Shift: ICU length of stay (hours since ICU admission) [ICULOS]
List of causal features:
Age (years) [Age]
Gender [Gender]
Administrative identifier for ICU unit (MICU); false (0) true (1) [Unit]]
Administrative identifier for ICU unit (SICU); false (0) true (1) [Unit2]
Time between hospital and ICU admission (hours since ICU admission) [HospAdmTimme]
List of arguably causal feature:
Temperature (deg C) [TTmp]]
Leukocyte count (count/L. [WBC]
Fibrinogen concentration (mg/dl.) [Fibrinogen]
Platelet count (count/mL) [Platelets]
Heart rate (in beats per minute) [HR]
Pulse oximetry (%) [O2Sat]
hhtpp///www.dddeeeyy///ahhhaaane htm
http:///byysssoerrrrrrr//eeetetr 2019/1.0.0/
93

---

Systolic BP (mm Hg) [SBP]
Mean arterial pressure (mm Hg) [MAP]
Diastolic BP (mm Hg) [DBP]
Respiration rate (breaths per minute) [Resp]
End tidal carbon dioxide (mm Hg) [EiCO2]
Excess bicarbonate (mmol/L) [BaseExcess]
Bicarbonate (mmol/L) [HCO3]
Fraction of inspired oxygen (%) [FiO2]
pH [pH]
Partial pressure carbon dioxide from arterial blood (mm Hg) [PaCO2]
Oxygen saturation from arterial blood (%) [SaO2]
Aspartate transaminase (IU/L) [AST]
Blood urea nitrogen (mg/dL) [BUN]
Alkaline phosphatase (IU/L) [Alkalinephos]
Calcium (mg/dL) [Calcium]
Chloride (mmol/I.) [Chloride]
Creatinine (mg/dL) [Creatinine]
Direct bilirubin (mg/dI.) [Bilirubin_dinn _irect]
Serum glucose (mg/dL) [Glucose]
Lactic acid (mg/dL) [Lactate]
Magnesium (mmol/dL) [Magnesium]
Phosphate (mg/dL.) [Phosphate]
Potassium (mmol/L) [Potassium]
Total bilirubin (mg/dl.) [Bilirubin_tove [otal]]
Troponin (ng/mL) [Trooponin]]
Hematocrit
Hemoglobin (g/dL) [Hgb]
Partial thromboplastin time (seconds) [PTT]
List of other features:
The training set (i.e. hospital) from which example drawn [set]
E.9
TableShift: UCI
We have one task based on a dataset by Strack et al. [2014) from the UCI Machine Learning Repository CClore al. 2014
E.9.1
Hospital Readmission
Target: No record readmission [readmitted]
Shift:
Admission source [admission_soee sourooe_ssuuree_eee _i]
List of causal features: Race [race]
Gender [gender]
Age [age]
Payer code [payer_codee code]
Medical specialty of the admitting physician [pecical_specialty]
List of arguably causal features:
Weight in pounds [weight]
Primary diagnosis [diag_1]
Secondary diagnosis [diag_2]
Additional secondary diagnosis [diag_3]
1999-2008
94

---

Total number of diagnoses [number_ diagnoses]
Discharge type [discharge dispsssdspssiiinnii _i]]
Count of days between admission and discharge [time in hospital]
Number of outpatient visits of the patient in the year preceding the encounter
Inumber_outpatieen
Number of emergency visits of the patient in the year preceding the encounter
[nunber_emergenny]]
Number of inpatient visits of the patient in the year preceding the encounter coumberjinpatien
Max glucose serum Beaxx_glusseeee
Hemoglobin Alc test result [A [AICresult]
Change in metformin medication [metformin]
Change in repaglinide medication [repaglinide]
Change in nateglinide medication [nateglinide]
Change in chlorpropamide medication [chlorpropamide]
Change in glimepiride medication [glimepiride]
Change in acetohexamide medication [acetohexamide]
Change in glipizide medication [glipizide]
Change in glyburide medication [glyburide]
Change in tolbutamide medication [tolbutamide]
Change in pioglitazone medication [pioglitazone]
Change in rosiglitazone medication [rosiglitazone]
Change in acarbose medication [acarbose]
Change in miglitol medication [miglitol]
Change in troglitazone medication [troglitazone]
Change in tolazamide medication [tolazamide]
Change in examide medication [examide]
Change in citoglipton medication [citoglipton]
Change in insulin medication [insulin]
Change in glyburide_metie m_tfo_mmeeffmnnneoo medication gelyburide_metforminn
Change in
medication (glipizide_metformiii
Change in gimmepinde_pioollttzzoee medication [glimepirid_ppogglitzoeee]]]]]]]
]ioglitazone]
Change in aenfoominnrrsiglitaaana medication mnetformin_rooigiitan
Change in metformin caaanniinpolliizzeeeeeee medication [metformin piritiii_oilllzzeee
Change in any medication [change]
Diabetes medication prescribed [diabetesMed]
List of other features:
Admission type [admission_typer
Number of lab tests performed during the encounter [num lab procedures]
Number of procedures (other than lab tests) performed during the encounter @num_procedure
Number of distinct generic drugs administered during the encounter
[num] moum_mdications
E.10
MEPS
We have one task based on the Medical Expenditure Panel Survey (MEPS) (Agency for Healthcare Research and Quality 2019.
E.10.1 Utilization
Dataset. We consider the MEPS 2019 Full Year Consolidated Data File. The dataset contains
information on individuals taking part one of the two MEPS panels in 2019. In particular, these individuals belong either to Panel 23 in its 3-5 round, to Panel 24 in its 1-3 round. We train on the
95

---

first round in 2019 for each panel, that is, Round 3 of Panel 23 and Round of Panel 24, and predict the total health care utilization across the year 2019. We adapt the target definition by Hardt and Kim 2023
Distribution shift. We split the domains by health insurance type, analogous to Taife in the task 'Stay in ICU" and 'Hospital Mortality'. We train on individuals with public health insurance, and use individuals with private health insurance as testing domain.
Target: Measure of health care utilization > 3 [TOTEXP19]
Shift: Insurance type [INSCOV19]
List of causal features:
Sex [SEX]
Race IRACEVIX, RACEV2X, RACEAX, RACEBX, RACEWX, RACETHX]
Hispanic ethnicity [HISPANX, HISPNCAT]
Years of education [EDUCYR]
Educational attainment [HIDEG]
Paid sick leaves [SICPAY31]
Paid leave to visit doctor [PAYDR31]
Person is born in U.S. [BORNUSA]
Years person lived in the U.S. [YRSINUS]
How well person speaks English [HWELLSPK]
Speak other language at home [OTHLGSPK]
What language spoken other than English [WHTLGSPK]
Region [REGION31]
Age [AGE31X]
List of arguably causal features: Family size [FCSZI23I,FAMSZE311
Martial status [MARRY3IX]
Flexible Spending Accounts [FSAGT31, HASFSA31, PFSAMT3I]
Employer offers health insurance [OFREMP31, OFFER3IX]
Insurance coverage from current main job [CMJHLD31]
Covered by Medicare [MCARE31, MCRPD31, MCRPB31, MCRPHO31, MCARE31X, MCRPD31X]
Covered by Medicaid [MCAID31, MCDHMO31, MCDMC31, MCAID3IX.
MCDAT3iX]
Covered by TRICARE/CHAMPVA [TRIAT3IX, TRICR3IX, TRILI3IX, TRIST3IX, T TRIPR3IX, TRIEX3IX, TRICH3IX]
Detailed type of covering entity [PRVHM031, GOVTA31, GOVAAT3I, GOVTB3I,
GOVBAT31. GOVTC31, GOVCAT31, VAPROG3I, VAPRAT3I, IHS31, IHSAT31, PRIDK31, PRING31, PUB3IX, PUBAT3IX, PRIEU31, PRIOG31, PRSTX31, PRINEO31, PRIEUO31, PRIV31, PRIVAT31, DISVW31X.)
Health insurance held from current main job [HELD31X]
Insured [INS3IX, INSAT31X]
Dental insurance [DENTIN31, DENTIN31, DNTINS31]
Prescription drug private insurance [PMEDIN31, PMDINS31, PMEDUP31, PMEDPY31]
Pension Plan [RETPLN31]
Employment status [EMPST31]
Student status [FTSTU31X]
Has more than one job [MORJOB3I]
Difference in wage by round [DIFFWG31]
Updated hourly wage [NHRWG31]
Hourly wage of current main job [HRWG3IX]
Hours per week [HOUR31]
96

---

Temporary current main job [TEMPJB31]
Seasonal current main job [SSNLJB31]
Self-employed [SELFCM31]
Choice of health plans [CHOIC31]
Industry group [INDCAT31]
Occupation group [OCCCAT31]
Union status [UNION31]
Reason for not working [NWK31]
Paid vacation [PAYVAC3]]
Instrumental Activities of Daily Living (IADL) help [IADLHP31]
Activities of Daily Living (ADL) help [ADLHLP31]
Use of assistive technology [AIDHLP31]
Limitations in physical functioning (WLKLIM31, LFTDIF31, STPDIE31,
WLKDIF31, MILDIF31, BENDIF31, RCHDIF31, ENGRDF31, ACTLIM31]
Social limitations [SOCLIM31]
Work, housework, and school limitations [WRKLIM31, WRKLIM31, HSELIM31,
SCHLIM31, UNABLEF1]
Cognitive limitations [COGLIM31]
Priority condition variables [ASTHEP31, ASSTIL31, ASATAK31, CHBRON31]
Asthma medications [ASMRCN31, ASPREV31, ASDALY31, ASPKEL31,
ASEVFL31, ASWNFL31, ASACUT3I]
Active duty in military [ACTDTY31]
Perceived health status [RTHLTH31]
Perceived mental health status [MNHLTH31]
List of other features: Current main job private for-profit, nonprofit, or government entity
[JBBORG3]]
Self-employed business incorporated, a proprietorship, or a partnership [BSNTY31]
Number of employees [NUMEMP31]
Firm has more than one location [MORE31]
Month started current main job [STJBMM3I]
Year started current main job [STJBYY31]
Veterans Specific Activity Questionnaire (VASQ) [VACMPY31, VAPROX31,
VASPUN31, VACMPM31, VASPMH31, VASPOU31, VAPRHT31, VAWAIT31, VAWAIT31, VALOCT31, VANTWK31, VANEED31, VAOUT31, VAPAST31, VACOMP31, VAMREC31, VAGTRC31, VACARC31, VAPROB31, VAREP31,
VACARE31, VAPCPR31, VAPROV3I, VAPCOT3I, VAPCCO31, VAPCRC31,
VAPCSN31, VAPCRF31, VAPCSO31, VAPCOU31, VAPCUN31, VASPCL31, VAPACT31, VACTDY31, VARECM31, VAMOBL31, VACOPD31, VADERM31, VAGERD31, VAHRLS31, VABACK31, VAJTPN31, VARTHR3I, VAGOUT31, VANECK31, VAFIBR3I, VATMD31, VACOST3I, VAPTSD3I, VABIPL3I, VADEPR31, VAMOOD31, VAPROS31, VARHAB31, VAMNHC3I, VAGCNS31, VARXMD31, VACRGV31, VALCOH31]
Data collection round [RNDFLG31]
Imputation flag [HRWGIM31]
How hourly wage was calculated [HRHOW31]
Verification [VERFLG31]
Survey related information [REFPRS31, REFRL3IX. FCRP1231, FMRS FAMS RESP31, PROXY31, BEGRFMBI, BEGRFY31, ENDRFM31, ENDRFY31, INSCOP31, INSC1231, ELGRND31, MOPID31X, DAPID31X]
Round [RUSIZE31, RUSIZE31, RUCLAS31, PSTATS31, SPOUID31, SPOUIN31]
E.11 SIPP
have one task based on the Survey of Income and Program Participation (SIPP) U.S. Census
Bureau 2014
97

---

E.11.1
Poverty
Dataset. We work with Wave and Wave 2 of the SIPP 2014 panel data. We train on Wave 1 and want predict whether an individual has an official poverty measure larger than the median in Wave 2 Hardt and Kim 2023
Distribution shift. We use individuals with U.S. citizenship as the training domain, and individuals without U.S. citizenship as testing domain. This simulates a survey collection with a biased sample, e.g. individuals without U.S. citizenship are systematically excluded.
Target: Household income-to-povent ratio 2 [
Shift:
Citizenship status [CITIZENSHIP_SS JSSATUUUSSSSSTU
List of causal features: Marital status [MARITAL_STATTU]
Educational attainment [EDUCATION]
Race|RACE]
Gender [GENDER]
Age [AGE]
Spanish, Hispanic, Latino[ORIGIN]
Disability status [HEALTHDISAB]]
Hearing difficulties [HEALTH HEARING]
Vision difficulties [HEALTH SEEING]
Cognitive difficulties [HEALTH [HEALTH_COGNITVVV] COGNITIVE]
Ambulatory difficulties [HEALTH
[HEALTHAAMBULADOON AMBULATORY]
Difficulties in self-care [HEALTH_SELL] [CC__
Difficulties in doing errands
Core disability [HEALTH CORE DISABILITY]
Supplemental disability
List of arguubly causal features: Household income [HOUSEHOLD _
Family size [FAMILY SIZE_ _VG]
Received worker's compensation [RECEIVED WORK COMP)
Unemployment compensation [UNEMPLOYMENT COMP)
Amount of unemployment compensation WANEMPPLOMENT_COMEE WOMP_
Severance pay and pension(SEVERANCE PAY PENSION]
Amount for forster child care (TOSTER_CHIII CCHLD_CARE__ CARE_AMTI
Amount for child support [CHILD (CHILD_SUPPORT SUPPORT_
Alimony amount [ALIMONY_AMA]
Income [INCOME]
Income from assistance (NNCME_FROM CROM_ASSISTANCE
Amount of savings and investments [SAVINGS ANV_AMOUNTY
Amount of veteran benefits WV_BENEFIIT ]
Amount of retirement income [RETIREMENT INCOME, AMOUNT)
Amount of survivor income [SURVIVOR, INCOME_ (MOMU])
CERVIIVORINCOMEEBCO
Amount of disability benefits [DISABILITY BIEILLITTYBEEETTTSSLLS AMMUNT]
Percentage of year which individual received assistance from MEDICARE
AASDDCCARRASSISTANNCEE
Number of sick days [DAYS_SICK,
Number of hospital nights HHOSPITAL_NIGHT
Number of presciptions for medicaments [PRESCRIPTION_MEER)
Number of dentist visits[VISIT DENTIST NUM]
Number of doctor visits [VISIT] DOCTOR NUM
Amount paid for non-premium medical out-of out-of-pocket expenditures
SRELLH_OVEES TOOUUTTEECC COOOCTTTEROOOOCCTSSPPAAC
98

---

Amount paid medical care [HEALTH MEDICAL SERDEAL_CAREEE PAY]
Amount paid for health insurance premiums [HEALTH INSURANCE PREMIUMS)
Amount of social security benefits [SOCIAL_SEC, SOCCAAL_SECBENNFFTTS)
Transportation assistance [TRANSPORTATION, IANSPORRTATIONAASSSTANCGCC
Own living quarters [LIVING OWNERSHIP]
List of anti- features:
Type living quarters [LIVING SENGG_QAATRRSS [ Percentage of year which individual received assistance from TANF
[TANF_ [ANNFAASSSTANCE]
Percentage of year which individual received food
assistance(FOOOD SAAAAICCCOOOOOOASSSTTTAAAAAAA [ASSISSAACCE
Percentage of year in which individual received assistance from SNAP
[SNAP_ASE JSSISTACNIT
Percentage of year in which individual received assistance from WIC
[WIC_ASSISTANCE)
Percentage of year which individual received assistance from MEDICAID
SOBICCAIDAASSSTAANC
E.12 Details on distribution shifts
We provide T with details on the observed distribution shifts. We adapt the metrics for target shift, concept shift and covariate shift from Gardner et al. [2023). See Appendix E.2 of their paper for the detailed definitions. For selected tasks, give additional insights into the concept shift by detailing it on the variable level See Figure 52 to Figure 55 We note that we conducted the in-depth analysis of the distribution shift post selecting the causal features and running our unpublished script
experiments described in Section 3 and Appendix C Our code is based
by Gardner et al. 2023].
Table 6: Summary of tasks and their associated distribution shifts.
Task
Covariate shift (OTDD)
Concept shift
Label shift
(FDD)
(L2 distance)
Food Stamps
14.20
640.82
0.0008
Income
30.60
1.40
0.0060
Public Coverage
5.79
4.06
0.1701
Unemployment
75.47
13.389.512.51
0.0003
ANES
13.60
2.23
0.0025
Diabetes
12.28
0.10
0.0332
Hypertension
4.69
0.04
0.0022
Hospital Readmission
42.37
1.30
0.0060
Childhood Lead
1.30
0,01
0.0026
Sepsis
6609.73
8.44
0.0040
ICU Length of Stay
56,439.324.672.00
47,042,729.585.11
0.0033
ICU Hospital Mortality
64.479,002,,36..1
42,639,188
487.47747
0.0015
ASSISTments
24.054.59
1137.42
0.0670
College Scorecard
43,566.39
2116.63
0.0337
SIPP
6,344,306.0
5,752,406.89
0.0751
MEPS
66.28
4.01
0.0013
We perform the analysis for tasks with less than 100 features due computational costs
99

---

Food Stamps
120.000
Concept shift (FDD)
100.000
80.000
20.000
0.000
ENG
FER
HUPXC
WIF
NWLL
NWLK
OCCP
POBP
RELP
WKHP
WKW
WRK
DIS
MIL
ANG
NATIVITY
DEAR
DEYE
DREM
PUBCOV
AGEP
SEX
ST
MAR
CIT
ACS. YEAR
20.000
Concept shift (FDD)
10.000
5.000
0.000
Cow
ENG
FER
HINS1
HINS2
HINS
HIN84
NWLA
NWLK
OCCP
POBP
RELP
WKHH
WKW
WRK
AGEP
SEX
ST
MAR
ACS
Unemployment
100.000
Concept shift (FDD)
80.000
60.000
40.000
20.000
0.000
ENG
POBP
RELP
WKHP
WKY
WRK
OCCP
DIS
ESP
MIG
MIL
ANC
NATIVITY
DEYE
DREM
DPHY
FER
AGEP
SEX
MAR
CIT
RAC1P
DIVISION
ACS YEAR
Figure 52: Concept shift on variable level. Measured in FDD distance (Gardner al. 2023
100

---

PublicCoverage
40.000
Concept shift (FDD)
20.000
10.000
0.000
ESP
MIG
ANC
NATIVITY
DEYE
DREM
PINCP
ESR
FER
AGEP
SEX
ST
MAR
CIT
RAC1P
DIVISION
ACS, YEAR
Voting
Concept shift (FDD)
10.000
5.000
0.000
ASSISTments
150000.000
Concept shift (FDD)
125000.000
100000.080
75000.000
50000.000
25000.000
0.000
Average corfidene(BORED)
Average confiderce CONFUSED)
Average, confidercee FRUSTT TESTTRATEE ED)
attempt pount
hint count
botto hint
ma first, response
butor mode
overlap time
first action
Avergee Carfidene((ONCCNTTRA
Figure 53: Concept shift on a variable level. Measured in FDD distance Gardner et al. 2023 (Continued)
101

---

Hypertension
80000.000
Concept shift (FDD)
40000 000
20000.000
0.000
AGEGBYR
FRUIT ONGE PER DAY
VEO ONCE PER DAY
DRIK PER,PERR WEEK
REBINGS
TOTINDA
SMOKET00
SMOKDAY2
CHCSONCR
CICOONCR
DIABETES
POVERTY
EMPLOY1
IYEAR
MEDCOST
PRACET
SEX
Diabetes
4000.000
Concept shift (FDD)
3000.000
1000.000
0.000
PHYSHLTH
PRESS
CHOL CHK PAST 5, YEERS
TOLDHI
BM15
BMISCAT
SMOKE100
SMOKDAY2
CVDSTRK3
MICHD
FRUIT ONCE PER DAY
VEG ONCE PER DAY
DRIK PER WEEK
RFBINGS
TOTINDA
INCOME
MARITAL
CHECKUP1
EDUCA
HEALTH COV
MENTHLTH
IYEAR
STATE
MEDCOST
SEX
Hospital Readmission
250.000
Concept shift (FDD)
200.000
150.000
100.000
50.000
0.000
weighh
admiccionn on type discharge risposition id fime hospiial payer coce
medical caal spicialty num lnb procedures
fum procedures run med caliaaioa number oulpaientt
number emerggency
inpsttent
2
dieg 3
number ciagroses
max gM senum
AT Cressull
melfommin
repaginnde
nateglinide
chlorpropane eneropameee
glimapinee
scetometeimin moveeemmmm
gllpizite
rosiigl rosiilllzzoee
sicerboeea rereece
mig tol
rrogliaazane
oxnniee
citippiie pptioon
Inssllin
pipppeeeeeee ne--mmttoo formnnn
alimapitite ridiiiideee wititee-ppgllleeeeeoooo Maazare
medormin-ple ailllllzoo
MabbleesMMd
Toldz
Figure 54: Concept shift on a variable level. Measured in FDD distance Gardner et al.
2023
(Continued)
102

---

Childhood Lesd
3,000
Concapt shift (FDD)
2.500
2.000
1.500
1.000
0.500
0.000
nhanes year
DMDBORNA
DNDEDUC2
RIDAGEYR
RIAGENDR
DMDMARTL
RIDRETH mmrged
Sepsis
800,000
Concept shift (FDD)
600.000
400.000
200.000
025al
Tenip
SBP
DBP
Resp
ETG02
FID2
pH
PaC02
AST
BUN
Alkalinaphos
Caldum
Chiorio
Creainime
Biirubin direct
Clucoss
Magnesium
Phosphale
Porsssum
Tropromini
Hel
Hib
PIT
WBC
Fibringgen
Plateiets
Age
Gender
Uhil2
HospAdmTiino
Batee
Biliubinn
Poverty
200000000.000 000
Concept sh (FDD)
150000000.000 OCO
100000000.000
50000000.000
0.300
LIVING QUARTER PATIO
STAT STATUS
FAMILY ORIE
AV
HOUSE LOUSEHOLE TANCE
ASSISTANCE SEVERS POLLOEENT VENNT COMP FOSTETTED
BRR... PEOO SAS AVT
INCOME FROM ASSISICON
RICOME
UNEMPLO OY OYYMETT COME COME AMOUNT
BEREE DE AMOU
COME AMOUN
DISABILITY. BE BENEFIT SASSTAA ANTAN
EDUCAT
BA
GEND
LIVING OWNERS SHI
MED CADE ASSISTA MEDICA AS STANNN
ISSTIDIS DAY'S SICK
HOSP TAAL NIGHTS
VISIT SE NUM
TRANSPOR ON ASSISTAL FANCA
HEALTH OVER THE COUNTES PRODUCT TORR EALY
HEALTH EDICALL CARE PAY
TH SEE
COUNITT POSNITTVE
HEALTH ARBIL ATTOR
HEALTH EARANDS CARE
HE SU INSURND PREMUR BREMIUMS SOCIAL SEC BENERTS
MAR MARTERRS
TANE ASSIST INEERLLYNE
SU MON
ASS
HEA TEATHH
CORE
RECEIVER
SAVINGS
RETIREMENT ING
FOOD
SNAP
LEAL
USI
Figure 55: Concept shift on a variable level. Measured in FDD distance Gardner et al.
2023
(Continued)
103

---

Table 7: Details to data sources, access and licenses.
Task
Data Source
Data access
License
Food Stamps
American Community Survey
Public
CCO
Income
American Community Survey
Public
CCO
Public Coverage
American Community Survey
Public
CCU
Unemployment
American Community Survey
Public
CCO
Voting
American National Election Studies
Restricted-use
Unknown
Diabetes
Behavioral Risk Factor Surverllance System
Public
Open Data Commons Open Database License
Hypertension
Behavioral Risk Factor Surveillance System
Public
Open Data Commons Open Database License
College Scorecard
U.S. Department of Education
Public
Creative Commons Attribution License
ASSISTments
Kaggle
Public
Unknown
Stay in ICU
Medical Information Mart for Intensive Care
Restricted-use
PhysioNet Credentialed Health Data License
Hospital Mortality
Medical Information Mart for Intensive Care
Restricted-use
PhysioNet Credentialed Health Data License
Hospital Readmission
UCI Machine Learning Repository
Public
Creative Commons Attribution License
Childhood Lead
National Health and Nutrition Examination Survey
Restricted public
Open Database License
Sepsis
PhysioNet
Public
Creative Commons Attribution License
10L
Utilization
Medical Expenditure Panel Survey
Restricted public
Open Data Commons Open Database License
Poverty
Survey of Income and Program Participation
Public
Unknown

---

Table Description of tasks.
Task
Target
Shift
In-domain
Oft-of-domain
Shift Gap
Obs.
Food Stamps
Food stamp recipiency in past year for households with child
Geographic region (U.S. divisions)
New England, Middle Atlantic, East North
East South Central
2.90%
840,582
Central, West North
Central, South Atlantic,
West South Central, Mountain, Pacific
Income
Income 2 56k for employed adults
Geographic region (U.S. Divisions)
Middle Atlantic, East
New England
7.71%
1,664,500
North Central, West North Central, South Atlantic, East South Central, West South
Central, Mountain, Pacific
Public Health Insurance
Coverage of non-Medicare eligible low-income individuals
Disability status
Without a disability
With disability
14.00%
5,916,565
10L
Unemployment
Unemployment for non-social
Education level
High school diploma higher
No high school diploma
17.69%
security-eligible adults
Voting
Voted in U.S. presidential election
Geographic region (U.S. regions)
Northeast, North Central, West
South
11.11 1%
8,280
Diabetes
Diabetes diagnosis
Race
White
Black African
4.70%
1,444,176
American, American
Indian or Alaskan
Native, Asian, Native
Hawaiian or other Pacific Islander, Other
race
Continued on next page

---

Table 8: Description of tasks, (Continued)
Task
Target
Shift
In-domain
Out-of-domain
Shift Gap
Obs.
Hypertension
Hypertension diagnosis for high-risk age (50+)
BMI category
Underweight, normal weight
Overweight, obese
1.37%
846,761
College Scorecard
Low degree completion rate
Carnegie classification
Different institution, e.g., special focus institutions (health professions). Master's 's colleges & universities (medium programs),
Special focus institutions (schools of art, music, and design, theological seminaries, bible colleges, and other faith-related institutions, others), Baccalaure- ate/Associate 's colleges, Master's colleges & universities (larger programs)
18.36%
124,699
ASSISTments Stay in ICU
Next answer correct
School
700 schools
10 schools
13.18%
2,667,776
106
Length of stay 2 3 hrs in ICU
Insurance type
Private, Medicaid,
Medicare
5.78%
23,944
Government, Self Pay
Hospital Mortality
ICU patient expires in hospital during current visit
Insurance type
Private, Medicaid, Government, Self Pay
Medicare
3.85%
23,944
Hospital Readmission
30-day readmission of diabetic hospital patients
Admission source
Different admission sources, e.g., physician referral, clinic referral, transfer from hospital, court/law enforcement, transfer from hospice
Emergency room
7.77%
99,493
Childhood Lead
Blood lead levels above CDC blood level reference value
Poverty level
Poverty-income ratio
Poverty-income ratio <
4.82%
27,499
1.3
1.3
Continued page

---

Table 8: Description of tasks, (Continued)
Task
Target
Shift
In-domain
off-oo-dommin
Shift Gap
Obs.
Sepsis
Sepsis onset within next 6hrs for hospital patients
Length of stay
Having been ICU for 47 hours
Having been in ICU for 47 hours
6.40%
1,552,210
Utilization
Measure of health care utilization 3
Insurance type
Any public
Private only
-4.01%
28,512
Poverty
Household nucome-to-povert ratio 2
Citizenship status
Citizen of the U.S.
Not citizen of the U.S.
21.59%
39,720
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide details Section to the claims made in abstract and introduction. We point out in the introduction that our study is based on monaiin-knowlddee selection of causal features, and empirical observations might not hold for other datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our study in the Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
108

---

Answer:
[NA]
Justification:
We do not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explain the experimental procedure in Section 2.4 and in more detail in Appendix [ We give list of our feature selections along with the datasets and distribution shifts in Appendix The code to replicate the experiments is provided at at/ttps: //github We link the reader to the datasets used in most of them have public access.
com/ wwwialfoundatiinoooooooooomomma. inneaaatiisss/ieoooo caaettiiiinncuusaaaaaaaaa aesusal-feature
our study in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
109

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access the code to reproduce our empirical results and in- structions at https://githu:
fow.://oiaaounnaaiiiiiioi com aocialfoooooooooooo.cooo.ooooooo foonatiiion cottions/caa watpndatioosccuual----a1
We link
the reader the raw datasets in Appendix E Most of them public accessible (1 1116 tasks), some allow only restricted use (5/11 tasks). We give details on steps needed for the preprocessing at https: bictps://githute com/ socialf cundations/callt fus/causal-eeatueee features/ blob/ DaEaainnRRRDE..... md Experimental run details are provided in Appendix B.3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
gublic/guides httgg//dd///eeuummiiisec
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https wwwoodeummisssoooo...o
7 //nips wttpcccccppppcccccccccccoooc httplllccgguideeeeeeeeeoe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 2.4 and more details in Appendix B The code is provided at bttts://ggtthe htt.a///..cmooocaaaaaaaaatttiiii oon.aaaooooaatiiicaaaaaaa causal features
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report all results accompanied by (approximate) confidence intervals. We describe the confidence intervals briefly in Section and provide details Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
110

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research does not involve human subjects or participants. We do not create dataset, but respect the terms of datasets used and link to their licenses.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

Justification: Causal inference is an important tool in policymaking and as such it has large potential societal impact, both positive and negative, Our work helps inform the use of some causal methods by showing that these methods may not achieve better performance than standard predictors. At the same time, we emphasize in Section that our analysis applies to questions of domain generalization for particular range of settings we considered. We caution that our findings should not be applied too broadly to causal inference and should not deter the practitioner from engaging rigorous causal inference.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is foundational research and therefore, poses high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
112

---

Justification: We cite the creators of models, evaluated in our paper. in Section 2.3 and Creators of code, used in our paper, are credited in Section 3
We provide details and
licenses of the code httttss::// http:::://ttttte
causal features
The original owners the data are listed in Section 2.2 and Appendix H The licenses,
available, are provided in Table
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide new code along with sufficient documentation at https:/ github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
113

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
114


Reviews:
[{"review_id": "YXev77GXMH", "reviewer": "Reviewer_yvyj", "summary": "The authors provide a thorough benchmark and analysis of machine learning models trained on different features to generalize to unseen domains. They use tabular datasets from various fields, including health, employment, education, etc., and categorize features into groups ranging from causal to anti-causal to test their influence on the model performances. Their key finding is that models utilizing all available features, irrespective of their causal nature, achieve better in-domain and out-of-domain accuracy than those relying solely on causal features across a battery of methods. This work challenges the practical applicability of theoretical causal advantages in real-world tabular data scenarios.", "questions": "- I would be curious to see if these findings also translate to genomics, where the feature size is huge, and a subset of genes has been discovered as causal for certain phenotypes. Have you considered this?", "limitations": "The authors adequately addressed the limitations!", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- very clear and accurate writing style, intuitive presentation of results in form of the figures, clear summary statements\n- the authors use a variety of datasets from different areas like health, employment, and politics, which makes the findings more applicable to real-world scenarios\n- the separation of features into the four categories, seems reasonable and human-like (i.e. that's how everyone should cluster features before training a predictive model). \n- the large battery of models is great\n- the authors conducted thorough checks to ensure their results are robust, making their conclusions seem trustworthy, and try to give explanations why certain results are observed", "weaknesses": "- Could you add the sample size to Table 1?\n- The explanations about the results seem to be a bit short. It would be great to elaborate a bit more and interpret the results wrt to the strength of the domain shift (e.g. it's obvious that anti-causal features are predictive as long as the shift is small)\n\nTypos:\n- Figure 1: Predictors based ON all features\n- L63: theortical -> theoretical\n- L108: that THE distribution\n- L172: Remove \"a\"\n- L214: considerably\n- L275: knowlege -> knowledge", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and positive feedback! We will fix the typos, thanks for pointing them out to us. To answer the reviewer’s questions:\n\n\n> add the sample size to Table 1?\n\n\nYes, we will add the sample sizes to Table 1. Note that they are currently provided in Table 6 in the Appendix.\n\n\n> elaborate a bit more and interpret the results wrt to the strength of the domain shift\n\n\nWe added a table with details on the strength of the domain shifts in the PDF of the author’s rebuttal. We adapted the metrics for target shift, conditional shift and covariate shift from Gardener, et al, 2023.\n\n\nWe will explain and interpret the results in more detail, taking into account the strengths of the domain shifts.\n\n\n> these findings also translate to genomics [...] Have you considered this?\n\n\nNo, we have focused our attention on datasets with easily interpretable features so far. If the reviewer has a suggestion on a plausible task based on genomics data and an appropriate natural distribution shift, we’d be happy to include that in our analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jogO9qfELI", "reviewer": "Reviewer_MY3d", "summary": "In this paper, an extensive evaluation of Ml methods with different feature sets is performed. \nOnly tabular data is considered and different feature set are considered: all, arguably causal and causal. \nIn the experiments no advantage of using causal features is shown for the domain generalization task.", "questions": "- It would be interesting to add an experiment with simulated data, with varying degrees of difference in the out-of-distribution data, so as to probe the theoretical ideas behind the claim that causal features should generalize better to unseen domain (something similar to what is done in anchor-regression literature).", "limitations": "yes", "rating": 9, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- very useful, inspiring results which will probably spark new research directions and/or rebuttals. \n- very well-written, clear and well-structured paper \n- code available and well-described experiment\n- interesting \"negative\" results for an interesting question which is often stated as a \"strength\" of causal machine learning models.", "weaknesses": "- maybe the number of different benchmarks/tasks could be higher, but I think that it is sufficient as it is\n- maybe simulation experiments? (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and amazing feedback! To answer the reviewer’s question:\n\n\n> add an experiment with simulated data, with varying degrees of difference in the out-of-distribution data [...] (something similar to what is done in anchor-regression literature)\n\n\nFollowing your suggestion, we conducted synthetic experiments. The setup is depicted in the PDF of the author’s rebuttal. Similar to Rothenhäusler, et al., 2021, we vary the degree of difference in the out-of-domain data using shift intervention on target, features and confounders.\n\n\nThe synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.\n\n\n*References:*\\\nRothenhäusler, D., Meinshausen, N., Bühlmann, P., & Peters, J. (2021). Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2), 215-246.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2joUUXUYdg", "reviewer": "Reviewer_bBCb", "summary": "The paper empirically investigates the hypothesis that machine learning models trained on causal features generalize better across domains. Analyzing 16 prediction tasks across various datasets, the study finds that models using all available features outperform those using only causal features in both in-domain and out-of-domain accuracy, challenging the assumed external validity of causal modeling techniques. Extensive robustness checks support these findings, suggesting that current causal methods do not necessarily provide better generalization in practical settings.", "questions": "- Anti-causal improves out-of-domain performance of arguably causal features (ll 221): Do you have an explanation/hypothesis?", "limitations": "- Feature classification into causal/non-causal is somewhat subjective (the authors do acknowledge this fact).\n- The study only provides empirical results. It does so very carefully, but still, this makes it hard to make any more general claims.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive empirical analysis with 16 diverse tasks across domains\n- Robustness checks confirm the stability of results\n- Challenging an existing assumption contributes valuable insights for the field\n- Great detail in the appendix.", "weaknesses": "- It may have been my mistake, but it took me a while to grasp how the domain split is done. Maybe this can be described more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and positive feedback! To answer the reviewer’s comments:\n\n\n> how the domain split is done [...] can be described more clearly\n\n\nThank you for drawing our attention to that. We will state how we obtain the domain splits more clearly.\n\n\n> Anti-causal improves out-of-domain performance of arguably causal features (ll 221): Do you have an explanation/hypothesis?\n\n\nA reasonable explanation is that the relationship of the anti-causal feature and target does not change strongly between domains. For example, patients with diabetes have a considerably increased risk of cardiovascular disease. Their relationship is partly explained by biomedical mechanisms, e.g., dyslipidemia (abnormal levels of lipids in the bloodstream) is a common feature of diabetes and poses a significant risk factor for cardiovascular diseases (Schofield, et al., 2016). It is conceivable that these mechanisms are stable across domains (population groups with different racial identity).\n\n\n> The study only provides empirical results [...] hard to make any more general claims.\n\n\nTo address your concerns, we conducted synthetic experiments to test how far our claims generalize. The setup is depicted in the PDF of the author’s rebuttal. Our code is based on the synthetic study conducted by Montagna, et al., 2023.\n\n\nThe synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.\n\n\n*References:*\\\nSchofield, J. D., Liu, Y., Rao-Balakrishna, P., Malik, R. A., & Soran, H. (2016). Diabetes dyslipidemia. Diabetes therapy, 7, 203-219.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DAwjkFS4cX", "reviewer": "Reviewer_QBUN", "summary": "This paper attempts to test the hypothesis of whether models trained on causal features generalize better across domains.\nThe authors found that predictors using all available features both causal and non-causal, have better in-domain and out-of-domain accuracy compared to causal features-based predictors. The authors also discuss that causal discovery algorithms perform poor in selecting causal variables. The authors provide empirical  analysis to support their claims.", "questions": "Below I share my questions:\n* Why are the non-causal features helping the model to perform well?  Are completely irrelevant features helping as well?\n* To my knowledge, the PC algorithm performs conditional independence tests to discover the undirected skeleton. How do the authors obtain causal parents from PC (Line 240)?\n* Please answer and discuss the questions in the weakness sections.", "limitations": "The authors clearly discussed their limitations.", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The paper is written in a well-presented manner and is easy to follow. The authors provided an extensive experimental analysis considering different real-world scenarios.", "weaknesses": "Here I discuss my major concerns about the paper:\n\n* It is unclear what type of distribution shift the authors are considering. For example, for a pair of variables X, Y, is it target shift: P(Y ) changes while P(Y |X) stays fixed, ii)conditional shift: P(Y |X) changes while P(Y ) stays fixed or, iii)covariate shift: only P(X) changes (see [1],[2] for details).\n* It is unclear if the test datasets the authors considered properly represent enough domain shifts. The author should more precisely numerically mention (like above) how much the test distribution changed compared to the train distributions.\n* One possible reason for causal features not performing well might be that the test distribution is not different enough compared to the training distribution. For example, if we consider a dataset of birds (Caltech-UCSD Birds-200-2011 (CUB) dataset) and we consider all available features including the background (water, land, forest, etc) for training, we will probably achieve better performance (due to background shortcut) than only causal features. However, if we consider a test dataset with higher domain shift such as Waterbirds [3], a model trained on all features will perform poor since the background feature will affect its prediction. In this scenario, causal predictors might perform well [4]. Maybe the irrelevant/non-causal features are helping because the test distributions are not significantly different. I would request the authors to share their perspectives about this possibility.\n* The authors should perform an ablation study by removing one non-causal feature at a time and measure the corresponding performance. This should show which non-causal features are improving the accuracy and how much. If they are not causal features, then why such features are improving the model accuracy. These should be discussed in detail.\n* If the conclusion drawn by the authors is to utilize all features for prediction, then the prediction would highly depend on features whose distribution changes with the domain. As a result, the prediction would be highly dependent on the training domain and perform poor in the test domain.  This would prevent the model from generalization. Please read the introduction section of [2], [5] for details. How do the authors plan to deal with such domain dependence and achieve generalization if they use all features.\n\n* Are the authors considering the possibility of confounders (shared unobserved parent of observed variables)? Due to the presence of confounders, we might not observe all the causal parents. For example, [5] discusses a medical scenario, where the goal is to diagnose a target condition T, say lung cancer using information about patient chest pain symptoms C and whether or not they take aspirin A. Lung cancer leads to chest pain (L->C) and that aspirin can relieve chest pain (A->C).  Smoking K (unobserved confounder) is a risk factor for both lung cancer (T), and aspirin (A) is prescribed to smokers as a result: T<—[K]—>A. In such scenarios, different causal methods such as [2,5] utilize non-causal features such as children (C) or neighbor (A) for prediction (T) as causal parent smoking (K) is unobserved. Thus, if we are only using causal parents, it is necessary to make sure that there are no unobserved causal parents.\n\n* The authors performed their experimental analysis on their selected datasets. They should also show their results on the datasets where causal predictors claim to perform well. Did the authors check any such datasets?\n\n[1] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference on machine learning, pages 819–827. PMLR, 2013.\\\n[2] Lee, Kenneth, Md Musfiqur Rahman, and Murat Kocaoglu. \"Finding invariant predictors efficiently via causal structure.\" Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence. 2023.\\\n[3] Sagawa, Shiori, et al. \"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.\" arXiv preprint arXiv:1911.08731 (2019).\\\n[4] Kaur, Jivat Neet, Emre Kiciman, and Amit Sharma. \"Modeling the data-generating process is necessary for out-of-distribution generalization.\" arXiv preprint arXiv:2206.07837 (2022).\\\n[5] Adarsh Subbaswamy, Peter Schulam, and Suchi Saria. Preventing failures due to dataset shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3118–3127. PMLR, 2019s", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. To answer the reviewer’s concerns and questions:\n\n(*Distribution Shift*) \n\n> unclear what type of distribution shift\n\nWe are considering natural distribution shifts. For example, the distribution shift induced by switching between geographic regions or demographic groups. Therefore, we are likely to suffer from all three forms of distribution shift at once. A recent study by Liu, et al., 2023, shows that conditional shifts are prevalent in tabular settings.\n\n> precisely numerically mention how much the test distribution changed\n\nFollowing your suggestion, we added a table with details on the observed distribution shifts in the PDF of the author’s rebuttal. We adapted the metrics for target shift, conditional shift and covariate shift from Gardener, et al, 2023. See Appendix E.2 of their paper for the detailed definitions.\n\n> Maybe the irrelevant/non-causal features are helping because the test distributions are not significantly different\n\nWe agree with the reviewer that one explanation of the empirical results is that the test distribution may not differ from the training distribution in a substantial way. In support of this argument is the fact that the training and testing domains are, for instance, different geographic regions within the U.S. bordering each other, or population groups with different education levels or racial identity.\n\n(*Datasets*)\n> Please read the introduction section of [2], [5] for details. How do the authors plan to deal with such domain dependence and achieve generalization if they use all features\n\nLee, et al, 2023, and Subbaswamy, et al, 2019 propose proactive algorithms, anticipating potential sources of unreliability in the data. In that context, our study should be interpreted as testing how unreliable variables in common tabular data are. In other words, do we need to act proactive when dealing with tabular data encountering natural distribution shifts? Are there domain dependences that limit the generalization?\n\nWe didn’t find empirical evidence for that within our datasets. All datasets we studied are commonly used in empirical evaluations, and most of them are currently part of a benchmark for out-of-domain generalization. Hence, we conclude to utilize all features for prediction in common tabular datasets. Note that our conclusion does not extend to, in that sense, uncommon datasets with known high domain dependence.\n\n> They should also show their results on the datasets where causal predictors claim to perform well\n\nWe looked for tabular datasets with (i) public or open credentialized access and (ii) interpretable features that can be classified in ‘causal’ and ‘non-causal’. We haven’t found any dataset matching these criteria where the causal predictors performed well. We’d welcome any suggestions from the reviewer, and are happy to include these datasets in our study.\n\n(*Ablation Study*)\n>  perform an ablation study by removing one non-causal feature at a time [...] Why are the non-causal features helping the model to perform well?\n\nWe conducted an ablation study and provided the results in the PDF of the author’s rebuttal. We remove anti-causal and non-causal features one at a time and measure the corresponding out-of-domain accuracy.  In a later comment, we discuss in detail the non-causal features whose removal significantly dropped the out-of-domain performance.\n\n>  Are completely irrelevant features helping as well?\n\nOur datasets are carefully curated. They are based on surveys that collect information experts in that field deem relevant. In some cases, features are selected from a multitude of variables, e.g., 23 features from the US Census data for predicting income. In addition, our datasets cover applications dealing with complex social dynamics (health, employment, politics,...). Therefore, it is hard to declare any feature as completely irrelevant.\n\n(*Confounders*)\n> Are the authors considering the possibility of confounders?\n\nWe strongly believe that there are confounders within our datasets. Their existence is another plausible explanation for the inferior performance of the causal features. We do not model the data generating processes precisely enough to employ methods proposed by  Lee, et al, 2023, and Subbaswamy, et al, 2019.\n\n(*PC Algorithm*)\n> How do the authors obtain causal parents from PC (Line 240)?\n\nThe PC algorithm estimates a completed partially directed acyclic graph (CPDAG), see Figure 33 in Appendix C.4 as an example. We use the implementation from the R package ‘pcalg’.\nThe edges connecting the target with other features are directed for the tasks ‘Food Stamps’, ‘Income’ and ‘Unemployment’. Therefore, we can readily identify these features as estimated causal parents.\nIn the task ‘Diabetes’, the target is only connected to high blood pressure via an undirected edge. We treat high blood pressure as potential causal parent. If deemed necessary, we can also include all skeletons as figures in our appendix.\n\n*References:*\\\nLiu, J., Wang, T., Cui, P., & Namkoong, H. (2024). On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances in Neural Information Processing Systems, 36.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQDEF8tQIw", "reviewer": "Reviewer_6nV6", "summary": "This work aims to provide an empirical study of how well models trained on causal features generalize across domains compared to models trained on all features. The major result from this study is that, contrary to the existing understanding that using causal features can generalize well in different environments, across 16 prediction tasks, models trained on all features generally perform better than those using only causal features. A similar trend is observed when using arguably causal features. Other experimental results, such as the inconsistency in the identification of causal features using classical discovery algorithms and the accuracy performance comparison between standard ML models and causal ML models, are also provided.", "questions": "1. On the choice of causal features: While I understand that the authors have tried their best of knowledge to select the features that are causal, it is also very likely the case that these chosen features are not actually causal. Under this mis-specificity, it is not surprised that the accuracies, irregardless of in-domain or out-domain accuracies, will suffer. Think about the case where some of the real causal features are selected and some of the spurious features are selected. Using these features to train the model will result in expected sub-optimal prediction performance. In your experiment, it seems it might be the case the above issue is true, as when you include the arguably causal features, the prediction errors significantly drops. \n\n2. Similar to the above issue, if the domain knowledge used to select causal features was incorrect, it is expected that the features identified by the causal discovery algorithm would not significantly overlap with the hand-selected causal features. While it is understood that causal discovery algorithms are not perfect, assuming these algorithms can accurately identify causal features, any inconsistency would likely be due to incorrect hand-selected features. Based on this reasoning, I suggest that the authors create a synthetic dataset where the ground truth is known to strengthen their claims.\n\n\n3. Similar to the above issue, the robustness tests seem unconvincing. Specifically, the robustness results are based on selecting a subset of features from the hand-selected causal features. However, if the original causal features are incorrect, any subset chosen from these features is unlikely to improve generalization. \n\n\n4. It is uncertain whether the comparison of accuracies between the use of machine learning models, causal methods, and methods tailored to domain robustness is a fair comparison. Specifically, for instance, it is unfair to compare XGBoost, a non-linear model, with IRM, which is a linear model. Such differences in prediction performance might come not from the methods themselves but from the different hypothesis classes and model complexities. Similarly, including methods tailored to domain robustness in the comparison can be confusing. For example, one could use the same causal features but achieve better domain generalization by employing domain robustness methods, such as sharpness-aware minimization (which is similar to the flavor or DRO), which finds flatter minima in the loss landscape to enhance generalization, which is entirely unrelated to the choice of causal features.", "limitations": "Although the authors include a limitations section, the authors does not directly address how their work might have potential limitations or potential negative societal impacts.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(High clarity in terms of experimental Setup and Results) This work provides a detailed exposition of how causal features were chosen for each task. The authors include a diverse set of tasks from different domains for the experiment, and the experimental procedures are clearly described. This ensures the validity of their results and is crucial for reproducibility.", "weaknesses": "1. (Quality in terms of writing): \n\n\tThe style and structure of this work require significant improvement. Specifically, the authors should adopt a more academic writing style and strive for greater specificity in each sentence. For instance, the sentences on lines 20, 24, 62, and 70—\"But it’s less clear what to do about it,\" \"The idea may be sound in theory,\" \"To be sure, our findings don’t contradict the theory,\" and \"... also discussed under the terms ‘autonomy’, ‘modularity’ or ‘stability’\"—are ambiguous. The authors should clearly describe the specific theories they are referring to in these instances and precisely define what 'autonomy', 'modularity', or 'stability' mean in the context of causal machine learning.\n\nThere are many other places with similar writing issues, and I leave it to the authors to identify these and make further improvements.\n\n\n2. (Quality in terms of validity of the experiment): There are several aspects of the experimental setup that are not entirely convincing or fair, which raises concerns about the reasonableness of the claims made in this work. I will raise these specific issues in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Below, we aim to answer the reviewer’s concerns:\n\n(*Quality in terms of writing*)\n\nWe will revise our work and adopt a more precise writing style, as well as reference in greater detail to what we are referring to.\n\n(*Hand-selected causal features*)\n\n> it is also very likely the case that these chosen features are not actually causal\n\nWe agree with the reviewer that the chosen features are likely misspecified. We even state a similar concern in the discussion section (“We likely made some mistakes in this classification\", line 271f.).\nNevertheless, we tried out every conceivable way to obtain causal features (domain knowledge, remove single features from the ones obtained by domain knowledge, causal discovery algorithms, causal machine learning methods). If the reviewer knows another way to obtain the causal features in an empirical study, we would be happy to include it in our analysis.\n\n> assuming these algorithms can accurately identify causal features, any inconsistency would likely be due to incorrect hand-selected features\n\nThis is true. That’s why we tried out both approaches in our study: hand-selecting based on domain knowledge and applying causal discovery algorithms. The results don’t change though.\n\n(*Synthetic dataset*)\n\n> I suggest that the authors create a synthetic dataset [...] to strengthen their claims.\n\nFollowing your suggestion, we conducted synthetic experiments. The setup is depicted in the PDF of the author’s rebuttal. Our code is based on the synthetic study conducted by Montagna, et al., 2023. We refer the reviewer to their results for a detailed performance analysis of the causal discovery methods.\n\nThe synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.\n\n(*Robustness tests*)\n> if the original causal features are incorrect, any subset chosen from these features is unlikely to improve generalization\n\nYes, the robustness test on the causal features merely tests for misclassifying one feature as causal although it is not. To enhance our robustness test, we also tested 500 random subsets for each task. The subsets are randomly drawn from all features. See Appendix C.5. for details. We welcome any suggestions to further test the robustness of our results.\n\n(*Empirical evaluation*)\n> It is uncertain whether the comparison of accuracies [..] is a fair comparison. [...] Such differences in prediction performance might come [...] from the different hypothesis classes and model complexities.\n\nWe follow the example of preceding empirical studies when comparing standard machine learning methods, causal machine learning methods and domain robustness methods via accuracy (Gulrajani and Lopez-Paz, 2020, Miller, et al., 2021, Gardener, et al., 2023). The objective is the applicability in practice. It is of separate interest to explore and disentangle the reasons for the inferior performance of certain methods, e.g., different hypothesis classes and model complexities.\n\n> For example, one could use the same causal features but achieve better domain generalization by employing domain robustness methods\n\nWe train all feature sets in the main analysis with the same methods. In particular, we also train on the causal features on domain robustness methods (DRO, GroupDRO, an adversarial label robustness method).\n\n>  address how their work might have potential limitations or potential negative societal impacts.\n\nWe will improve our limitation section and explicitly state our limitations and potential negative societal impacts.\n\n*References:*\\\nMontagna, F., Mastakouri, A., Eulig, E., Noceti, N., Rosasco, L., Janzing, D., ... & Locatello, F. (2024). Assumption violations in causal discovery and the robustness of score matching. Advances in Neural Information Processing Systems, 36.\\\nGulrajani, I., & Lopez-Paz, D. (2020). In search of lost domain generalization. arXiv preprint arXiv:2007.01434.\\\nMiller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., ... & Schmidt, L. (2021, July). Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning (pp. 7721-7735). PMLR.\\\nGardner, J., Popovic, Z., & Schmidt, L. (2024). Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,


Thank you for your thoughtful comments and suggestions!


(*Contribution*) We are encouraged that you found our work has "very useful, inspiring results which will probably spark new research directions and/or rebuttals" (MY3d) and "contributes valuable insights for the field" (bBCb). Our separation of features into causal, arguably causal, anti-causal and non-causal is praised as "reasonable and human-like (i.e. that's how everyone should cluster features before training a predictive model)" (yvyj).


(*Soundness*) We are particularly pleased that our experiments are described as "extensive" (QBUN) and "comprehensive"(bBCb). The reviewers appreciated that we used a "diverse set of tasks from different domains" (6nV6), and concluded that this makes our findings "more applicable to real-world scenarios" (yvyj). Our robustness checks are perceived to "confirm the stability of results" (bBCb) and make our conclusions "seem trustworthy" (yvyj).


(*Presentation*) Our work is praised to have "high clarity in terms of experimental setup and results" (6nV6), be a "well-described experiment" (MY3d) and have "great detail in the appendix" (bBCb). The reviewers highlighted that this "ensures the validity of their results and is crucial for reproducibility" (6nV6).


We have also taken your feedback into account and made the following key changes to improve our paper:

1. (*Synthetic experiments*)\
Following your suggestion (6nV6, bBCb, MY3d), we conducted synthetic experiments. The setup is depicted in the PDF of the author’s rebuttal. The causal mechanisms are modeled as (i) linear with weights randomly drawn in (-1,1) and (ii) based on a neural network with random instantiation. The noise variables are drawn from a standard normal distribution. The task is to classify whether the target is larger than 0.\
\
Similar to Rothenhäusler, et al., 2021, we vary the degree of domain shift using shift intervention on target, features and confounders, as proposed by (MY3d). We draw 1,000 training samples from the causal mechanism, and evaluate the performance on 1,000 testing samples from the intervented causal mechanism with shift interventions varying from 0 to 10; step size is 0.1. Our code is based on the synthetic study conducted by Montagna, et al., 2023.\
\
The synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy. We highlighted the plots where the exception occurs.

2. (*Strength of distribution shift*)\
To meet your requests (QBUN, yvyj), we added a table with details on the observed distribution shifts in the PDF of the author’s rebuttal. We adapted the metrics for target shift, conditional shift and covariate shift from Gardener, et al, 2023. See Appendix E.2 of their paper for the detailed definitions.

3. (*Ablation study*)\
As you suggested (QBUN), we conducted an ablation study and provided the results in the PDF of the author’s rebuttal. We remove anti-causal and non-causal features one at a time and measure the corresponding out-of-domain accuracy.  We will discuss in detail the non-causal features whose removal significantly dropped the out-of-domain performance and try to give explanations.


We hope these updates address any concerns expressed by the reviewers — we are happy to respond to any additional concerns that might arise during the Author Reviewer Discussion period!


Best regards,\
The authors of #16003


*References:*\
Gardner, J., Popovic, Z., & Schmidt, L. (2024). Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36.\
Montagna, F., Mastakouri, A., Eulig, E., Noceti, N., Rosasco, L., Janzing, D., ... & Locatello, F. (2024). Assumption violations in causal discovery and the robustness of score matching. Advances in Neural Information Processing Systems, 36.\
Rothenhäusler, D., Meinshausen, N., Bühlmann, P., & Peters, J. (2021). Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2), 215-246.

Meta-review: This paper performs empirical comparison on 16 datasets and show that predictions using all features generalize (both in-domain and out-of-domain) better than when we use only causal features.

The study confirms the reported struggles in finding improvement in the predictive performance of ML algorithms by infusing the causal principle of invariance/stability/autonomy. It attempts to be as rigorous as possible, however, as Reviewer 6nV6 points our and the authors agree, categorization of features into causal/non-causal is quite difficult. I trust the authors have done their best in this aspect.

This is a timely paper for the ML community. I hope the authors continue this line of work by providing a theoretical understanding of why predictors on only causal features do not generalize better. The authors need to emphasize that they do not challenge the validity of causality theorems, but rather challenge how realistic the assumptions are.
Decision: Accept (spotlight)