Smoke and Mirrors in Causal Downstream Tasks

Authors: riccardo  cadei
lukas  lindorfer
sylvia  cremer
cordelia  schmid
francesco  locatello
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:16:13

Abstract: Machine Learning and AI have the potential to transform data-driven scientific discovery, enabling accurate predictions for several scientific phenomena. As many scientific questions are inherently causal, this paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations in a Randomized Controlled Trial (RCT). Despite being the simplest possible causal setting and a perfect fit for deep learning, we theoretically find that many common choices in the literature may lead to biased estimates. To test the practical impact of these considerations, we recorded ISTAnt, the first real-world benchmark for causal inference downstream tasks on high-dimensional observations as an RCT studying how garden ants (Lasius neglectus) respond to microparticles applied onto their colony members by hygienic grooming. Comparing 6 480 models fine-tuned from state-of-the-art visual backbones, we find that the sampling and modeling choices significantly affect the accuracy of the causal estimate, and that classification accuracy is not a proxy thereof. We further validated the analysis, repeating it on a synthetically generated visual data set controlling the causal model. Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones. Further, we highlight guidelines for representation learning methods to help answer causal questions in the sciences.

Main paper: 
 introduction: 
 Uncovering the answer to many scientific questions requires analyzing massive amounts of data that humans simply cannot process on their own. For this reason, leveraging machine learning and AI to help answer scientific questions is one of the most promising frontiers for AI research. As a result, AI is now predicting how proteins fold [Jumper et al., 2021] , new materials [Merchant et al., 2023] , precipitation forecasts [Espeholt et al., 2022] , and animal behaviors [Sun et al., 2023] . Even predicting counterfactual outcomes for treatment effect estimation appears to be possible [Feuerriegel et al., 2024] . In scientific applications, these predictions are often incorporated into broader analyses to draw new physical insights. In this paper, we focus on the problem of estimating the strength of the causal effect of some variable on another, which is a common type of question across disciplines [Robins et al., 2000 , Samet et al., 2000 , Van Nes et al., 2015 , Runge, 2023] .While our discussion and conclusions are general, we follow a simple real-world example throughout the paper: behavioral ecologists want to study the social hygienic behavior in ants and, thereby, the ability of the insects to remove small particles from the body surface of exposed colony members. Such grooming behavior performed by nestmates plays an important role in restoring a clean body surface of the contaminated individual, which, in case of infectious particles being groomed off, assures the health of the individual and prevents disease spread through the colony's [Rosengaus et al., 1998 , Hughes et al., 2002 , Konrad et al., 2012] . To study whether different microparticles differ systematically in their induction of grooming behavior, the biologists thus perform an experiment under controlled conditions, where a focal worker ant is treated randomly with either of two microparticle types, and the behavior of two untreated colony members towards the treated ant is filmed in multiple replicates. This is followed by detailed behavioral observation to quantify ant activity, as well as statistical data analysis to determine if treatment has a significant effect. This step could obviously be entirely replaced with deep learning, dramatically accelerating the workflow. In fact, many data sets and benchmarks have been proposed with the specific reason of supporting downstream science in behavioral ecology and biology [Sun et al., 2023 , Beery et al., 2018 , Kay et al., 2022 , Chen et al., 2023] and other scientific disciplines [Beery et al., 2022 , Lin et al., 2023 , Moen et al., 2019] .Our paper questions the simplicity of this narrative in both theory and practice. While we take experimental behavioral ecology as an example for our motivation and experiments, our theoretical results and experimental conclusions are general, and we expect them to be applicable across disciplines. Our key contributions can be summarized as follows:• We theoretically show how many design choices can affect the answer to a causal question, from the data used for training, the architecture choices, and even seemingly innocuous standard practices like thresholding the predictions into hard labels, or using held out accuracy for model selection (a common practice in many AI for science benchmarks, e.g., [Sun et al., 2023] ). To facilitate future research on representation learning for causal downstream tasks, we formulate the representation desiderata to obtain accurate estimates for downstream causal queries together with best practices.• To showcase the practical impact of these design choices, we design and collect a new dataset, ISTAnt, from a real randomized controlled trial, reflecting a real-world pipeline in experimental behavioral ecology, which we will release to accelerate research on representation learning for causal downstream tasks. To the best of our knowledge, this is the first real-world data set specifically designed for causal inference downstream tasks from high-dimensional observations.• On our dataset, we fine-tune 6 480 state-of-the-art methods [Dosovitskiy et al., 2020 , Zhai et al., 2023 , Radford et al., 2021 , He et al., 2022 , Oquab et al., 2023] in the few-and many-shot settings.Empirically, we confirm that the seemingly innocuous design choices like which samples to annotate, which model to use, whether or not to threshold the labels, and how to do model selection have a major impact on the accuracy of the causal estimate. Since our ground-truth estimate of the causal effect depends on the trial's design, we propose a new synthetic benchmark based on MNIST [LeCun, 1998] controlling for the causal model, and we replicated the analysis.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the abstract, we claim that we theoretically discuss potential sources of biases in downstream treatment effect estimation from Machine Learning pipelines in randomized controlled trials. In the paper, we discuss it in Section 3 (proofs in Appendix A). In the abstract, we claim that we validate our results experimentally, introducing a new real-world dataset and a new synthetic one. In the paper, we introduce the new benchmarks in Section 2, Appendix E and C, and we report the results in Section 6 and Appendix E.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
setting: 
 We consider the RCT setting, where binary treatments T are randomly assigned within an experiment with controlled settings W . In many applications, the outcome of interest Y is not measured directly. Instead, it is collected in high-dimensional observations Xe.g., frames from a video of the experiment. Our goal is to estimate the causal effect of T on Y , which is quantified by the estimation of the Average Treatment Effect (ATE), i.e.:EQUATIONAssuming an RCT (i.e., Ignorability Assumption [Rubin, 1978] ) is the ideal setting for causal inference because the ATE directly identifies in the Associational Difference (AD), i.e.,EQUATIONHowever, annotating Y from the high-dimensional recordings X requires costly manual annotations from domain experts. Leveraging state-of-the-art deep learning models, we can hope to alleviate this effort. Instead of labeling all the data, we only partially annotate it. We introduce a binary variable S, indexing whether a frame is annotated by a human observer or not. We denote the annotated samples withD s = {(W i , T i , X i , Y i ) : S i = 1} ns i=1and the not annotated ones withD u = {(W i , T i , X i ) : S i = 0} nu i=1. We use D s to train or fine-tune a deep learning model to estimate the labels on D u . Next, we leverage the Ignorability Assumption on the full RCT to identify the ATE in the AD and consistently estimate it. Ideally, it would be most useful if D s = ∅, but for the purpose of this paper, we assume that at least some samples are annotated, for example, during quality controls.Besides the clear statistical power considerations, recovering the full RCT enables the identification of the causal estimands. Estimating the ATE only on D s may not be feasible even if one aims to adjust for W due to possible violations of the Positivity Assumption (i.e., 0 < P (T = 1|W = w) < 1 ∀w : P(W = w) > 0). In principle, S should be assigned randomly (independent from any other variable), but for practical reasons, it is often a function of the experiment settings W . For example, when annotating grooming in behavioral experiments, experts annotate experiment by experiment, marking the beginning and end of each behavior event, allowing for some selection bias. The experimental setup can be described with the causal model in Figure 1 , where we omit the corresponding exogenous random noises for simplicity. For simplicity of exposition, we will now assume a binary outcome, but all the following results naturally generalize to the continuous case.Motivating Application and the ISTAnt Dataset. Ants show strong hygiene behaviors and remove any particles that attach to their body surface, including dust, dirt, and infectious particles.In a process termed "grooming", they use their mouthparts to pluck off adhering particles, collect and compact them in a pouch in their mouth, and later expulse them as pellets. As a social behavior, ants groom one another to keep all colony members clean and healthy. To understand how social insects like ants may react to changes in their ecosystem, it is of great interest to research in collective hygiene how different particles differentially affect the intensity of grooming by colony members.For this purpose, we recorded groups of three Lasius neglectus worker ants interacting in a controlled environment, where we treated one focal ant by applying either of two microparticle types to its body surface and observed the grooming activity of the other two towards it. Our exemplary task is to estimate the causal effect of the microparticle type on ant behavior. Sample frames of these recordings used to build our new benchmark are reported in Figure 2 .Predicting animal behavior is a standard machine learning and computer vision task [Sun et al., 2023 , Chen et al., 2023] . At the same time, we hope to use these predictions within the context of a causal downstream task. In this paper, we question whether the naive application of deep learning methods leads to consistent estimates that can be used to draw scientific insights, even if the data we collect is ideal, i.e., a randomized controlled trial. Likewise, in causal inference, the factual effects are always assumed to be given, and the statistical consideration of using machine learning to estimate them is missing.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are discussed in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
biases in downstream ate estimation from ml pipelines: 
 In this section, we formalize a model's bias for a downstream Treatment Effect Estimation and its relationship with (vanilla) prediction accuracy measures. We then highlight possible sources of biases from both the data and the model.Definition 3.1 (Treatment Effect Bias). Let f : X → Y a model for E Y [Y |X = x].We define the treatment effect bias of f w.r.t. a treatment T on an outcome Y and a signal X as:TEB :=   E X|do(T =1) [f (X)] -E Y |do(T =1) [Y ] Interventional Bias under Treatment   -   E X|do(T =0) [f (X)] -E Y |do(T =0) [Y ] Interventional Bias under Control   (3)f is treatment effect unbiased if T EB = 0, i.e., the difference among the systematic errors per intervention (over/under estimating) compensates, or directly, the ATE on the predicted outcomes equals the true ATE (despite possible misclassification).Lemma 3.1 (Informal). Assuming the setting described in Section 2. A predictive model f for the factual outcomes with accuracy 1-ϵ can lead to |TEB(f)| = ϵ mint P (T =t) ≥ 2ϵ, which invalidates any causal conclusion when the ATE is comparable with ϵ and/or the dataset is unbalanced in T.A formal statement and proof for Lemma 3.1 is reported in Appendix A.1. Lemma 3.1 explicits that misclassification can lead to biased causal conclusion, but not necessarily. Clearly, if the prediction accuracy is perfect (i.e., ϵ = 0), also the objective of treatment effect estimation is perfect. However, for each error rate ϵ > 0, several predictions with different treatment effect biases are possible, from 0 to the worst-case scenario ϵ mint P (T =t) , which drastically invalidates any causal conclusion for ϵ ≫ 0 or strongly unbalanced dataset with respect to the treatment assignment. Accuracy and similar metrics do not provide a full picture of the goodness of a model for such a downstream task.Due to the Fundamental Problem in Casual Inference [Holland, 1986] , we cannot estimate the treatment effect bias directly. By design (i.e., Ignorability Assumption), the interventional expectations are identified in the conditional ones on the whole population, but not on D s individually due to the effect modifications activated by conditioning on S. Still, in practice, a validation set, ideally Out-of-Distribution from the training sample in D s , can be considered to approximate the TEB.Links to Fairness This idea of enforcing similar performances (or at least similar systematic errors) among the treated and controlled groups can be revisited in terms of fairness requirements [Verma and Rubin, 2018] . In particular, it strictly relates to Treatment Equality [Berk et al., 2021] , where the ratio of false negatives and false positives for both treated and control groups is enforced to be the same, while in TEB we measure the difference, but in a similar spirit. In our setting, the difference is actually a more stable measure since the ratio can be ill-defined when the number of false positive predictions approaches 0. This discussion leaves open where the bias originates and, in the fairness literature, this is reflected in the distinction between bias preserving and bias transforming metrics [Wachter et al., 2021] . For our purposes, the data as a whole is assumed unbiased in principle since we assume an RCT, but the sampling scheme S could introduce bias in the training data. Orthogonally, the model choices can amplify existing data biases differently or even introduce new ones.Data bias from sampling choice From the assumed causal model illustrated in Figure 1 , we have that P (X,Y )|S=0 generally differs from P (X,Y )|S=1 . Indeed, conditioning on S acts as an effect modification on X and Y . It follows that the risk in predicting Y over the annotated population can differ from the expected risk over the whole population, i.e.:EQUATIONDue to this distribution shift, we should expect some generalization errors at test time through empirical risk minimization even if n s → ∞. It follows that the Conditional Average Treatment Effect (CATE) estimate for the experimental settings poorly represented in D s can introduce bias in D u .Mitigation: Randomly assigning S is crucial to suppress any backdoor path and avoid generalization errors. Model selection should also take into account the TEB. Although we cannot estimate it directly, a validation set, ideally Out-of-Distribution in W , should be considered to bound the TEB, replacing the interventional distributions with the corresponding conditionals.Model bias from the encoder choice Since X is high-dimensional, we decompose the model f in h • e, where e is an encoder potentially pre-trained on a much larger corpus through a representation learning algorithm and h is a simple decoder (e.g., multi-layer perceptron) for classification. A good representation should be both sufficient and minimal [Achille and Soatto, 2018] . If a representation is only sufficient, redundant information from W or S could be preserved, potentially leading to systematic errors on D u due to spurious correlations and the abovementioned covariates shift. Frozen state-of-the-art models are most likely not minimal for our task, making the sampling choices even more relevant. If the representation is not sufficient, then it is biased by definition.Mitigation: Before deploying a new backbone, one should attempt to quantify its biases. If needed, new methodologies to mitigate this bias during adaptation should be investigated. Overall, models with lower bias may be preferable even if they have lower accuracy.We can encounter a final source of bias in post-processing the predictions. Indeed, despite the majority of the classification methods directly modeling the conditional expectation E[Y |X = x], we could naively be tempted to binarize this estimate to the most probable prediction or setting a fixed threshold. See indeed how the default choices for the predict module, even in established libraries, e.g., Logistic Regression implementation in sklearn.linear_model [Pedregosa et al., 2011] , is to output the most probable prediction directly. Similarly, even econML, the most popular library for causal machine learning [Battocchi et al., 2019] , allows for binary outcome prediction methods. Despite being apparently innocent and common practice in classification, discretizing the conditional expectation is biased for downstream treatment effect estimation.Theorem 3.1.[Informal] Let a binary classification model converge to the true probability of the outcome given its (high-dimensional) signal. Then, its discretization (i.e., rounding the prediction to {0,1} with a fixed threshold) also converges, but to a different quantity with a different expectation. It follows that, for causal downstream tasks from ML pipelines, discretizing the predictions biases the ATE estimation.A formal statement and proof of Theorem 3.1 is reported in Appendix A.2. It shows that even if we rely on a consistent estimator of the factual outcome for each subgroup, its discretization would still converge but on a different quantity, i.e., it is biased. There is then no reason to discretize a model for E X [Y |X] if we can model it directly, e.g., using sigmoid or softmax activation [Senn and Julious, 2009, Fedorov et al., 2009] . Likewise, if there is uncertainty over human annotations (e.g., because multiple raters disagree), the soft label should be used and not the majority one.Example 1. To intuitively visualize this result, consider a generative process following the causal model introduced in Mitigation: Never arbitrarily discretize predictions for downstream treatment effect estimation. 
related works: 
 Representation learning for scientific applications The setting we consider is very common, and there are several benchmarks that have studied representation learning as a means to help domain experts in sciences, for example, [Sun et al., 2023 , Beery et al., 2018 , Kay et al., 2022 , Chen et al., 2023] in ecology alone. However, these works focus on downstream prediction accuracy following standard machine learning evaluation practices, which do not necessarily indicate good downstream causal predictions. One notable positive example is Beery et al. [2018] , as computing the prediction accuracy separately for different locations allows us to estimate the bias of the model. Overall, we argue that when the ultimate purpose of training a machine learning model is to support scientists in answering some research question that is causal in nature, the specific question should be part of the design and evaluation of the benchmark. For this reason, our paper uniquely starts from the causal downstream task. Only then can we formalize the properties that methods should have in order to do well and be useful in answering the overarching scientific question. To the best of our knowledge, ours is the first real-world computer vision data set with an associated well-defined and real causal downstream task.Causal representation learning In our analysis (both theoretically and experimentally), we focused on traditional representation learning algorithms, but there is a whole community interested in identifying causal variables from high-dimensional observations [Schölkopf et al., 2021] . Superficially, identifying Y may be useful to estimate the ATE. However, all existing methods seem to cover two main classes of assumptions that are unfortunately inapplicable to our setting. Interventional methods [Ahuja et al., 2023 , Buchholz et al., 2023 , Squires et al., 2023 , Varici et al., 2023 , Zhang et al., 2024] require intervening on the behavior Y , which is practically impossible and, even if we could, then we would not need to identify it. Multi-view approaches [Ahuja et al., 2022 , Brehmer et al., 2022 , Locatello et al., 2020 , von Kügelgen et al., 2021 , Yao et al., 2024] would require access to positive and negative pairs of samples with respect to Y . However, it is not clear how to construct such pairs in our setting without knowing Y already. Further, all these approaches only cover continuous variables. A notable exception is Kivva et al. [2021] , which covers discrete variables but has a non-degeneracy assumption (Assumption 2.4) that is severely violated in our case (i.e., most pixels are not affected by the behavior variable because ants are small). For these reasons, despite having a very clear causal downstream task, we had to resort to classical representation learning algorithms that are not identifiable. We hope that our data set can serve as a new real-world benchmark for developing algorithms with realistic assumptions that can be applied in practice.Other Related Works In causal inference, only Chakrabortty et al. [2022] shows how to use semi-supervised learning to perform imputation on missing effect annotations. Unfortunately, their setting is comparatively very low-dimensional (observations are 200 binary variables). Instead, we consider high-dimensional real-world images in a representation learning setting, which introduces additional new challenges as described in Section 3. Remarkably, they do not discute discretization bias. Curth et al. [2024] already mentioned that the Positivity and Ignorability/Unconfoundness Assumptions are critical for using machine learning in the context of ATE estimation. However, their work does not explain precisely how confounding effects can arise in the representation learning setting, which we thoroughly addressed. Close in spirit to our discussion are [Angelopoulos et al., 2023, Zrnic and Candès, 2024] , considering the role of predictions in statistical estimates. Our setting is related but additionally motivated by the hope of leveraging causal identification properties on the prediction-powered dataset. 
experimental setup: 
 We validate the theoretical results from Section 3 on our new real-world dataset. We assume D s ∪ D u being a full RCT, and we compare the treatment effect biases among several design choices in annotating and modeling. Overall, we fine-tuned 6 480 different models and tested all the mitigations proposed. We then replicate the experiments on CausalMNIST, a new synthetic benchmark we propose that allows controlling for the causal effect.We applied microparticles to the body surface of a (focal) Lasius neglectus worker ant and recorded the behavioral reaction this treatment elicits in two other worker ants from the same colony. To distinguish between the treated individual and the untreated two nestmates, the latter had been colorcoded by a dot of blue or orange paint, respectively, before the experiment. We used two different microparticle treatments to compare grooming responses by the nestmates between treatment types, assigning them at random (i.e., RCT). For five batches, we simultaneously filmed nine ant groups of three ants each under a single camera setup in a custom-made box with controlled lighting and ventilation. In total, we collected 44 videos1 of 10 minutes at 30fps each for a total of 792 000 frames annotated following a blind procedure, and we run the analysis at 2fps for a total of 52 800 frames. More details about the experiment design are reported in Appendix C. We remark that this is the first real-world data set for treatment effect estimation from high-dimensional observations, which we will release to accelerate future research. Since it encompasses a real-world scientific question, we can, at best, enforce the Ignorability Assumption by design in the trial. We do not have actual control over the underlying causal model and the causal effect. We take the treatment effect estimation computed with the expert annotations as ground truth.Annotation Sampling Annotating frames individually is significantly more expensive in terms of time and not adopted in practice. The practical gold standard through current software for human annotation is per-video random annotation, where only a few videos taken at random are fully annotated. We compared this criterion with per-video batch (W 1 ) and per-video position (W 2 ) annotation criteria, where only the videos in certain batches or positions were considered in D s . For each of the three criteria, we further considered a many-shots (D s ≫ D u ) and a few-shots setting (D s ≪ D u ). Details about the dataset splitting per annotation criteria are in Appendix D.1.Modeling We modeled f as a composition of a freezed pre-trained encoder e and a multi-layers perceptron h fine-tuned on D s . For the encoder, we compared six different established Vision Transformers (ViT), mainly varying the training procedure: ViT-B [Dosovitskiy et al., 2020] , ViT-L [Zhai et al., 2023] , CLIP-ViT-B,-L [Radford et al., 2021] , MAE [He et al., 2022] , DINOv2 [Oquab et al., 2023] . For each encoder, we considered the representation extracted (i) by the class encoder (class), (ii) by the average of all the other tokens (mean), or (iii) both concatenated (all). For each representation extracted we trained different heads, varying the number of hidden layers (1 or 2 layers with 256 nodes each with ReLU activation), learning rates (0.05, 0.005, 0.0005) for Adam optimizer [Kingma and Ba, 2014] (10 epochs) and target (independent double prediction of 'blue to focal' and 'orange to focal' grooming, or unique prediction of grooming either 'blue to focal' or 'orange to focal') via (binary) cross-entropy loss. We either discretized or not the output of the model, already in [0, 1] due to the sigmoid final activation. For each configuration, we repeated the training with five different random seeds. A summary of the architectures and training description is in Appendix D.2.Evaluating For each trained model, we computed the binary cross-entropy loss, accuracy, balanced accuracy, and TEB on validation; and accuracy, balanced accuracy, TEB, and TEB using discretization on the full dataset D = D s ∪ D u (where the average potential outcomes in the TEB are estimated by the sample mean). Since the ATE does not have a reference scale, for interpretation purposes, we replaced the TEB with Treatment Effect Relative Bias (TERB = TEB/ATE) in the visualizations.CausalMNIST is a new synthetically generated visual dataset we designed for downstream treatment effect estimation. It is a colored manipulation of the MNIST dataset [LeCun, 1998] , following an underlying generative process in agreement with the causal model assumed in our framework (see Figure 1 ). We explicitly controlled the ATE and generated 400 different samples from such a population (each one as large as the MNIST dataset, i.e., 60k images), allowing for Monte-Carlo simulations to accurately provide confidence intervals of our estimations. We omitted a comparison among pre-trained encoders since the visual task is relatively simple and can be solved directly by a simple convolutional neural network in a supervised fashion. A full description of the dataset is in Appendix E, together with its experiments, which align with our conclusions from ISTAnt. 
results: 
 Annotating criteria matter Theory suggests that biased annotating criteria (i.e., depending on the experimental settings) can lead to biased treatment effect estimation, wrongly retrieving the conditional treatment effect on unseen experimental settings. Figure 4 validates this observation, particularly in the few-shots regime. Despite the average estimation of the TEB is (almost) always biased, as illustrated in Table 1 , the distribution for (per-video) random annotation is more centered towards 0. The benefits of random sampling are less obvious in the many-shots regime since D u becomes less and less Out-of-Distribution. Still, this setting is rarely the case in practice since scientist hope to label |D s | ≪ |D u | frames to have a concrete advantage in their workflow.Encoder Bias Vanilla classification evaluation (e.g., accuracy, F1-score, etc.) well describes the goodness of a representation for a predictive downstream task. However, it is still unclear how to measure the goodness of a representation for a causal downstream task since we do not directly observe the ground truth (fundamental problem of Causal Inference). Even in our simple setting where we can easily identify the treatment effect over the whole population, it is not possible to condition just on a biased subsample (e.g., the validation set). Figure 5 shows clearly how the TERB doesn't correlate with balanced accuracy on the whole sample once it is sufficiently good (i.e., > 0.9). Even among models with balanced accuracy > 0.95 we estimated TERBs up to ± 50%, which can drastically lead to wrong causal conclusions. Among the different encoders, MAE is significantly underperforming all the others. We postulate the reason for this gap is that the masked reconstruction training leads to overly focus on background conditions instead of the comparatively small ants.Evidence for this hypothesis is reported in Table 2 where we observe that for 'position' splitting criteria, the Fréchet Distance between the extracted embeddings by MAE in D s and D u is maxima and significantly higher than for the other splittings, probably due to spurious correlation with the background which is indeed non changing as much for "random" and "experiment" splitting. Despite some (e.g., DINOv2 and CLIP-ViT-L) having better downstream predictive performances, the other encoders all have similar TERB ranges. New criteria to better estimate and bound the treatment effect bias already on validation and methodologies to unbias these models during training are required.Discretization Bias We considered the absolute value of the TEB over all the 6 480 fine-tuned models, evaluating independently the models predicting both 'Blue to Focal' and 'Orange to Focal' grooming for a total of 9 720 evaluations. We tested (t-test):H 0 : E[|TEB(f )|] = E[|TEB(1 [0.5,1] (f ))|] vs H 1 : E[|TEB(f )|] < E[|TEB(1 [0.5,1] (f ))|] (5)We found strong statistical evidence to confirm that discretizing the model outcome worsens treatment effect estimation (t statistic=-10.42, p-value=1.07 • 10 -25 ), confirming Theorem 3.1 .Prediction is not Causal Estimation Distinct statistical and causal objectives cannot be used as a proxy for one another. We already formalized this in Lemma 3.1 and partially observed it in Figure 5 .In Figure 6 , we systematically show it by comparing the rank-correlation among 1 620 models. We further observed that simply computing the TEB on a small validation is a better predictor of the TEB over the full dataset than the metrics focused on prediction accuracy (even on the full dataset). For the few-shot and experiment sampling (the most realistic), if we select the single best model on validation based on the TEB versus the accuracy, we underestimate the effect by 11% and 18%, respectively. While this is not perfect, is a significant improvement. We encourage future research to investigate theoretical generalization guarantees and new techniques to approximate the TEB on validation data. Discussion Overall, our results clearly show that it is possible to leverage pre-trained deep learning models to accelerate the annotation of experimental data and obtain downstream causal estimates that are consistent with those from domain experts. At the same time, we find that experimental practices need to incorporate the specific needs of these causal downstream tasks. While our theoretical statements are "worst case scenarios" and only indicate that bias can arise (but does not always have to), we find empirical validation that it unfortunately easily manifests in practice. Remarkably, the fact that we performed and collected data within a randomized controlled trial, which is the best-case scenario of causal inference, did not alleviate the issue. Therefore, we can expect that the opportunities for bias can be even greater in observational settings, and even greater care is needed in model selection with the TEB and adaptation-time debiasing techniques. 
conclusions and limitations: 
 As AI models are increasingly used to answer scientific questions and support human decisionmaking, it is important to understand how design choices in machine learning pipelines affect the final results. In this paper, we took a closer look at the impact of pre-trained deep learning models in answering downstream causal treatment effect questions. We presented a real-world example in experimental behavioral ecology, creating the first-ever data set for treatment effect estimation from high-dimensional observations. Both theoretically and empirically, we observed that common choices, most notably discretizing the predictions and using in-distribution accuracy for model selection, can significantly affect the downstream conclusions. Two clear limitations of this work are that we did not do anything to the training to mitigate the bias, we kept the backbones frozen, and we did not incorporate the unlabelled data for semi-supervised training. Here, it would be very interesting to study how tools developed in the fairness literature can be extended to causal questions. For future benchmarks targeting scientific applications, we remark that it is vital to include the actual downstream question in the design of the data set. Otherwise, there is a risk that any model produced on that data may be unusable in practice, as it can bias the answer on an otherwise perfectly designed experiment. Finally, we would recommend that future work in causal representation learning starts from a clear downstream task like the one presented in this paper and works backward to reasonable assumptions. To facilitate this process, we will release our data set including all the experimental variables, so that relevant future work on e.g., discovering confounding or semi-supervised effect discovery, can take place on a real problem. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Computer resource requirements are discussed in Appendix D and E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Data collection and all experimental work were performed in compliance with international, national and institutional regulations and ethical guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: We do not work with human subjects or crowdsourcing. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Lemma 3.1Lemma. Let T ∼ Be(p T ), Y ∼ Be(p Y ), X ∼ P X and let f :X → [0, 1] a model for E Y [Y |X = x]. Assuming (i) Ignorability (i.e., T ⊥ Y |do(T = 1), Y |do(T = 0)) , (ii) E X [|1 [k,1] (f (X)) = f (X)|] = 0where k ∈ [0, 1], and (iii) f with accuracy 1 -ϵ, i.e., : Proof. Starting from the definition of Treatment Effect Bias and using assumption (i):EQUATION|TEB(f )| = |(E X|do(T =1) [f (X)] -E Y |do(T =1) [Y ] Interventional Bias under Treatment ) -(E X|do(T =0) [f (X)] -E Y |do(T =0) [Y ] Interventional Bias under Control )| = = |(E X|T =1 [f (X)] -E Y |T =1 [Y ] ϵ1 ) -(E X|T =0 [f (X)] -E Y |T =0 [Y ] ϵ0 )| = = |ϵ 1 -ϵ 0 | ≤ |ϵ 1 | + |ϵ 0 | (6)whereEQUATION) represent the overestimation of each conditional outcome expectation (i.e., conditional bias under treatment/control).By assumption (iii) and the law of total probability:ϵ = P 1 [k,1] (f (X)) ̸ = Y = = P 1 [k,1] (f (X)) ̸ = Y |T = 0 • P(T = 0) + P 1 [k,1] (f (X)) ̸ = Y |T = 1 • P(T = 1). (8)By Jensen's inequality and linearity of the expected value:EQUATIONCombining Equation 8 and 9 we using the assumption (ii), for all t ∈ {0, 1} we have:EQUATIONAnd finally, combining this with Equation 6, we get:EQUATION)The bound we found corresponds to the worst-case scenario where we misclassify, only overestimating or only underestimating, always in the least probable treated group. Since T is binary, then (min t P (T = t)) > 0.5, and so the thesis.Comment: Assumption (ii) is only used to find the worst-case scenario explicitly. Similar results can be stated bounding this discretization error.A.2 Proof of Theorem 3.1Theorem. Let T ∼ Be(p T ), Y ∼ Be(p Y ) and X ∼ P X . For all t ∈ {0, 1}, let τn (X, t) a succession converging in mean L 1 to τ (X, t):= E Y [Y |X, T = t], i.e., E X [|τ n (X, t) -τ (X, t)|] n -→ 0 (12) Let τ * n (X, t) = 1 [k,1] (τ n (X, t)) for all n and τ * (X, t) = 1 [k,1] (τ (X, t)), where 1 [k,1] : R → {0, 1} is the indicator function with threshold k ∈ [0, 1]. Assuming τ (X, t) having continuos CDF (i.e., F τ (X,t) ∈ C 0 ), then:EQUATIONi.e., they are generally different unless for a value k ∈ [0, 1] depending on the distribution of τ (X, t) (not observed in practice).Proof. Convergence in mean L 1 of that binarized estimator (Equation 13) follows directly from the fact the L 1 convergence implies convergence in distribution and Portmanteau Thereom (using the continuity assumption of τ (X, t) CDF).It only remains to show that the expectations of their limits generally differ. By developing the expected value of the τ * (X, t) we have:EQUATION)where, by definition, k is the α-quantile for τ (X, t), with α = 1 -E X [τ (X, t)] (uniqueness due to the continuity of τ (X, t) CDF).Let's consider the following structural causal model in alignment with the generative process in Figure 1 . Noises:EQUATIONwhere p T ∈ (0, 1) and σ 2 Y > 0. and structural equations:T := n T (20) W := n W (21) X := T + W + n X (22) Y := 1 [0,+∞) (X + n Y )(23) By the Law of Total Probability and additivity of Gaussian distributions, it follows:EQUATION)Y |T = 0 ∼ Be(0.5) (28)Y * := 1 if E Y [Y |X] > 0.5 0 if otherwise (29)Then:EQUATIONAnd:EQUATIONLet f (x) a logistic regression estimator for E[Y |X = x] and:EQUATION)Setting p T = 0.5 and σ 2 Y = 1, we run a Monte-Carlo simulation with 50 different random seeds per sample size n, estimating the associational difference by the empirical associational difference (EAD), i.e., using the sample mean. The results are reported in Figure 3 . We observe that f leads to a consistent estimate of the true associational difference, which corresponds to the ATE due to the Ignorability Assumption encoded in the causal model:EQUATIONand so:EQUATIONBut, according to Theorem 3.1, its discretization is biased:AD Y * ,T -AD Y,T = ϕ(1/ √ 2) -ϕ(1/ √ 3) ≈ 0.042 > 0 (37)and more generally, the stronger is the variance in the effect random noise n Y , the bigger is the bias.In our study, we analyzed grooming behavior in the ant Lasius neglectus in groups of three worker ants. The workers for the experiment were obtained from their laboratory stock colony, which had been collected from the field in 2022 in the Botanical Garden Jena, Germany. Ant collection and all experimental work were performed in compliance with international, national and institutional regulations and ethical guidelines. For the experiment, the body surface of one of the three ants was treated with a suspension of either of two microparticle types (diameter 5 µm) to induce grooming by the two nestmates, which were individually color-coded by application of a dot of blue or orange paint, respectively. The three ants were housed in small plastic containers (diameter 28mm, height 30mm) with moistened, plastered ground and the interior walls covered with PTFE (polytetrafluoroethane) to hamper climbing by the ants. Filming occurred in a temperature-and humidity-controlled room at 23°C within a custom-made filming box with controlled lighting and ventilation conditions. We set up nine ant groups at a time (always containing both treatments) and placed them randomly on positions 1-9 marked on the floor in a 3x3 grid with a distance of about 3mm from each other. Figure 7 illustrates the filming box and the displaying of the containers in each batch. The experiment was performed on two consecutive days. Videos were acquired using a USB camera (FLIR blackfly S BFS-U3-120S4C, Teledyne FLIR) with a high-performance lens (HP Series 25mm Focal Length, Edmund optics 86-572) in OBS studio 29.0.0 [Bailey, 2017] at a framerate of 30 FPS and a resolution of 2500x2500 pixels. From each original video (105x105 mm), we generated 9 individual videos .mkv (each 32x32 mm, 770x770 pixels) by determining exact coordinates per container from one frame in GIMP 2.10.36 [Kimball and Mattis, 2023] and cropping of the videos with FFmpeg 6.1.1 [Tomar, 2006] . Annotation was performed over two consecutive days by three observers who had not been involved in the experimental setup or recording and were unaware of the treatment assignments to ensure bias-free behavioral annotation. They annotated the behavior of the ants during video observations, using custom-made software that saves the start and end frames of behaviors marked in a .csv file. In one of the videos, one of the nestmates' legs got inadvertently stuck to its body surface during the color-coding, interfering with its behavior, so the video was discarded. This left 44 videos from 5 independent setups (n=24 of treatment 1 and n=20 of treatment 2) of 10 minutes each for a total of 792 000 annotated frames. For each video, we provide the following information: the number of the set to which it belongs (1-5); the number of the position within the set reflecting the position of the ant group under the camera (1-9), for which we also provide 'coordinates' in the 3x3 grid (taking values -1/0/1 for both X and Y axis); treatment In this section, we provide additional information on the experimental settings for the main experiments (on ISTAnt dataset). In particular, we describe the annotation splitting criteria selected, then the modeling choices and the training details. We run all the analyses using 48GB of RAM, 20 CPU cores, and a single node GPU (NVIDIA GeForce RTX2080Ti). The main bottleneck in the analysis is the feature extraction from the pre-trained Vision Transformers. We estimate 96 GPU hours to fully reproduce all the experiments described in the main paper.Let W 1 ∈ {1, ..., 5} representing the number of batch experiment and W 2 ∈ {1, ..., 9} the relative position of a video inside its batch. We defined the annotation splitting criteria based on the value of the experiment settings W 1 and W 2 , in agreement with Table 3 .where Ω = {(1, 2), (1, 3), (2, 4), (2, 5), (3, 1), (3, 2), (4, 3), (4, 4), (5, 9)}. For validation (used to generate the Figure 6 ) we consider 1 000 random frames from D u .We extracted once the embedding of each frame in the dataset using a pre-trained encoder, and we fine-tuned multi-layer perceptron (MLP) heads for classification according to the training details reported in Table 4 . We considered the following encoders for feature extraction, also report the corresponding Hugging Face code ID for reference:Annotations Criteria D s n s D u n u Many Random (W 1 , W 2 ) / ∈ Ω 42 000 (W 1 , W 2 ) ∈ Ω 10 800 Experiment W 1 ̸ = 5 42 000 W 1 = 5 10 800 Position W 2 ̸ = 8 46 800 W 2 = 8 6 000 Few Random (W 1 , W 2 ) ∈ Ω 10 800 (W 1 , W 2 ) / ∈ Ω 42 000 Experiment W 1 = 1 10 800 W 1 ̸ = 1 42 000 Position W 2 = 1 6 000 W 2 ̸ = 1 46 800Table 3 : Annotation splitting criteria details for the extensive experiments on ISTAnt described in Section 5 and 6.• ViT-B [Dosovitskiy et al., 2020] : google/vit-base-patch16-224• ViT-L [Zhai et al., 2023] : google/siglip-base-patch16-512• CLIP-ViT-B [Radford et al., 2021] : openai/clip-vit-base-patch32• CLIP-ViT-L [Radford et al., 2021] : openai/clip-vit-large-patch14-336• MAE [He et al., 2022] ,: facebook/vit-mae-large• DINOv2 [Oquab et al., 2023] Encoder (token) refers to which embedded tokens were considered for representation from each ViT. 'class' stands for the class taken, 'mean' for the mean of all the other tokens and 'all' for their concatenation. Task refers to which outcome we aimed to model directly: either the two independent grooming events ('Blue to Focal' and 'Orange to Focal') or the single grooming event ('Blue or Orange to Focal'). Overall, we finetuned:EQUATIONheads.To replicate the results on ISTAnt controlling for the causal model, we proposed CausalMNIST: a colored manipulated version of MNIST [LeCun, 1998] , defining a simple causal downstream task (treatment effect estimation). Starting from MNIST dataset, we manipulated the background color B of each image (1: green, 0: red), and the pen color P (1: white, 0: black) to enforce the following Conditional Average Treatment Effect:EQUATIONand Average Treatment Effect:E[Y |do(B = 1)] -E[Y |do(B = 0)] = 0.3 (41)where Y is a binary variable equal to 1 if the represented digit is strictly greater than d ∈ R, 0 otherwise. Arjovsky et al. [2019] already proposed a colored variant of MNIST as a benchmark for robustness in a multi-environment setting, but without controlling for any causal model and presenting a causal downstream task. A simple interpretation of this new task is estimating the effect of the background on the chances of writing a big digit (i.e., greater than d).To obtain a sample from such a population manipulating MNIST dataset, we converted each gray image into a RGB, coloring the background B and the pen P according to Bayes' rule:EQUATIONSince the digits in MNIST dataset are uniformly distributed:EQUATIONwhere p Y = (9 -d)/10.We then set: B, P i.i.d.∼ Be(0.5) (44) and: Overall, the final structural causal model can be summarized as follows:EQUATION• Noises (independent):n B ∼ Be(0.5) (49) n P ∼ Be(0.5) (50) n X ∼ P n X (51) n Y ∼ P n Y (52)Figure 8 : Random samples from CausalMNIST dataset.• Structural equations:EQUATIONwhere P n X , P n Y , f 1 and f 2 are unknown and characteristic of MNIST dataset. The corresponding causal model matches the setting described in Section 2 where B represents the treatment T and P the experiment settings W . In this analysis, we set d = 3. Six examples of colored handwritten digits from CausalMNIST are reported in Figure 8 .Annotation Sampling Similarly to ISTAnt experiments, we compare the random annotation, where S is assigned independently from P , and a biased annotation, where only the images with a black pen (P = 0) are annotated in both few and many annotation setting. The biased annotation criteria don't provide any information in D s about the white pen (P = 1) CATE, and retrieving the annotations on D u becomes mandatory. Unfortunately, a model could misclassify the new images under this covariate shift or hallucinate just for a specific treatment group (e.g., green background and white pen), leading to a biased estimate of the ATE. In Table 5 , we summarize the 4 different annotation sampling proposed. For validation, we consider a random subsample of D u as large as D s . Please observe that for the biased subsampling, not all the images with black pen (P=0) are allocated D s . Indeed, since P(P = 0) = 0.5 > ns ns+nu (in both few and many annotations regime), then D u contains both images of hand-written digits in white and black. NeurIPS Paper ChecklistOne video was discarded for analysis since a leg of one of the two nestmates got stuck in the dot of the color code, impairing its behavior. 


Decision: This paper investigates the bias when estimating the causal quantities in high dimensional RCTs. The author identifies the bias led by the common design choice. More importantly, the author provide a high-dimensional benchmark dataset for causal inference problem, which is a valid contribution to the causal community.

Reviews: review: paper_summary: paper_summary: The paper explores the challenges associated with using machine learning, particularly deep learning, to estimate causal treatment effects from high-dimensional data, such as images, in Randomized Controlled Trials (RCTs). The authors point out that standard practices in machine learning, such as selecting models based on accuracy and discretizing predictions, can lead to biases in causal estimates. To substantiate their claims, they provide both theoretical analyses and empirical results. They introduce a novel real-world benchmark dataset that involves the grooming behavior of ants, which they assert is the first benchmark designed for causal inference downstream tasks. The authors stress the necessity of incorporating causal downstream tasks into benchmark design and offer guidelines to enhance causal inference in scientific applications through machine learning.
strength_weakness: strength_weakness: I am not very familiar with this topic, so I may adjust my rating based on further discussions with the authors and reviewers.

1. The main theoretical analysis focuses on binary classification, which may not generalize to many real-world settings. On line 251, the authors claim that Theorem 1 is validated on their dataset. Could you provide more details on how Theorem 1 is validated? I found it difficult to follow.

2. The setting of this paper differs from many causal representation learning algorithms, making it challenging to directly evaluate those algorithms on this benchmark. This may limit the applicability of the proposed benchmarks. Could you apply some existing causal representation learning methods to the proposed datasets and compare the conclusions? The experiments only test different pre-trained models, but it would be more compelling to demonstrate that this new benchmark can help answer causal questions with training in a causal manner.

3. The authors frequently mention causal inference downstream tasks. Can you further elaborate on why the proposed dataset is superior to other benchmarks for answering causal questions? The experiments primarily show that bias can arise, but I am not fully convinced that the proposed dataset is better than the existing ones.

4. While the authors acknowledge limitations such as keeping the backbones of models frozen and not incorporating semi-supervised learning, these constraints might affect the generalizability of their findings.
questions: questions: Please see the weaknesses below. I may reconsider my score after further discussion with the authors and reviewers. My main concern is the significance of the proposed benchmark and its utility in evaluating different causal representation algorithms. Why is this dataset better suited for answering causal questions and reducing bias compared to existing datasets?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper considers the task of causal effect estimation $P(Y|do(T))$, where the treatment is mediated through a (potentially high-dimensional) observation $X$.
Additionally, a semi-supervised setting is studied, where labels $Y$ are only available in a subset of the data.
A set of possible biases affecting the treatment effect estimation that can arise in this setting are discussed together with possible mitigation strategies.
Two datasets are introduced as new benchmarks:
(i) An Ant Dataset, where groups of ants are filmed under the treatment of putting microparticles on the body surface. Grooming behaviour is observed through video recordings.
(ii) CausalMNIST: A dataset that adds colour coding to MNIST data and adds a downstream causal task.
The biases discussed above are empirically evaluated on a large battery of models that are fine-tuned in few- and many-shot learning.
strength_weakness: strength_weakness: The biggest weakness of this paper is that it tries to do a bit too much.
Given the NeurIPS page limit, this leads to things being condensed to a point that makes it difficult to follow at times.
For example, Theorem 3.1 tries to convey an idea without properly defining most of its elements, and IMHO it fails to convey much at all.
Without looking up the formal definition in the appendix, it's difficult to understand.
questions: questions: - Sec. 5.1: What exactly are $X$ and $Y$ for this dataset?
limitations: limitations: - "Model bias from the encoder choice" (L146): This point is a bit wishy-washy and, as it is, it has limited use to the practitioner. It is too generic and doesn't offer much practical guidance on how to e.g. "attempt to quantify its biases". Maybe this space would be better spent on expanding on other parts of the paper.
paper_summary: paper_summary: This paper theoretically reveals that many common choices in the literature may lead to biased estimates. To test the practical implications of these considerations, this paper recorded the first real-world benchmark for causal inference downstream tasks on high-dimensional observations through an RCT studying how garden ants (Lasius neglectus) respond to microparticles applied to their colony members by hygienic grooming. By comparing 6,480 models fine-tuned from state-of-the-art visual backbones, they found that the sampling and modeling choices significantly affect the accuracy of the causal estimates, and that classification accuracy is not a reliable proxy for this accuracy.
strength_weakness: strength_weakness: 1. The link to the data set provided by the author cannot be accessed.
2. The author's presentation of the provided data set is a little bit limited, especially in terms of visualizations. It would be helpful to include some actual images of the physical data.
questions: questions: 1. Could authors provide more details on the dataset collection process and explain why ants were used as an example?
2. Could the method proposed in this paper be generalized to other fields, beyond the ecological example mentioned?
limitations: limitations: The authors thoroughly discussed the limitations of this paper.
paper_summary: paper_summary: the paper makes a deep dive into the various types of biases that can arise in RCTs that would invalidate causal estimants
strength_weakness: strength_weakness: - The literature review only focuses on the last 2 years and ignores prior works that are still relevant
- there is little theoretical argumentation in the proposed solutions 
- there is little experimental evidence on ways to overcome the identified problems
questions: questions: - not much to be honest, the identification and codification of the RCT biases and how these effect causal estimates makes the paper pass the bar for me
limitations: limitations: - adequately addressed

Comments: title: Acknowledgement
comment: Sorry for the late reply. Thank you so much for the detailed rebuttal. Most of my concerns are addressed. Regarding my first question, I meant Theorem 3.1—sorry for the typo. In lines 301-302, I saw that Theorem 3.1 is empirically validated, but adding more analysis could be beneficial. After reading other reviews, I cannot find significant issues, and I am willing to increase my score to 6 while keeping the confidence level at 2.
I acknowledge that I have read the authors rebuttal, and I maintain my score of acceptance
Thanks for the author's rebuttal. I will keep my score.
Thank you for the feedback. We are also happy either way and will decide based on the amount of space we have available as we prepare the next revision.
Thank you for the clarifications.

On the limitation: I'm happy with either solution (if you expect me to weigh in on which option to choose).