X-Hacking: The Threat of Misguided AutoML
Authors: Rahul Sharma
Sumantrak Mukherjee
Andrea Sipka
Eyke Hüllermeier
Sebastian Josef Vollmer
Sergey Redyuk
David Antony Selby
Venue: ICML 2025 poster (2025)
Keywords: explainable AI
SHAP
AutoML
Model Multiplicity
Rashomon sets
Ethics
Reproducibility
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: Deliberate manipulation of SHAP explanations at scale using AutoML

Abstract: Explainable AI (XAI) and interpretable machine learning methods help to build trust in model predictions and derived insights, yet also present a perverse incentive for analysts to manipulate XAI metrics to support pre-specified conclusions. This paper introduces the concept of X-hacking, a form of p-hacking applied to XAI metrics such as Shap values. We show how easily an automated machine learning pipeline can be adapted to exploit model multiplicity at scale: searching a set of ‘defensible’ models with similar predictive performance to find a desired explanation. We formulate the trade-off between explanation and accuracy as a multi-objective optimisation problem, and illustrate empirically on familiar real-world datasets that, on average, Bayesian optimisation accelerates X-hacking 3-fold for features susceptible to it, versus random sampling. We show the vulnerability of a dataset to X-hacking can be determined by information redundancy among features. Finally, we suggest possible methods for detection and prevention, and discuss ethical implications for the credibility and reproducibility of XAI.

X-Hacking: The Threat of Misguided AutoML
Rahul Sharma
Sumantrak Mukherjee' Andrea Sipka' Eyke Hüllermeier? Sebastian Vollmer'
Sergey Redyuk"
David Antony Selby!
Abstract
However, the growing demand for interpretable models and 'data-driven' decisions creates an incentive for actors, unscrupulously- or through lack of time or experience to seek out model explanations that support pre-specified con- clusions, conceal hidden agendas or evade ethical scrutiny.
Explainable AI (XAI) and interpretable machine learning methods help to build trust in model pre- dictions and derived insights, yet also present a perverse incentive for analysts to manipulate XAI metrics to support pre-specified conclusions. This paper introduces the concept of X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values. We show how easily automated ma- chine learning pipeline can be adapted to exploit model multiplicity at scale: searching a Rashomon set of 'defensible' models with similar predictive performance to find a desired explanation. We for- mulate the trade-off between explanation and ac- curacy multi-objective optimisation problem, and illustrate empirically on familiar real- wooldd datasets that, on average, Bayesian optimisation accelerates X-hacking 3-fold for features suscep- tible to it, versus random sampling. We show the vulnerability of dataset to X-hacking can be determined by information redundancy among features. Finally, we suggest possible methods for detection and prevention, and discuss ethical implications for the credibility and reproducibility of XAI.
In this paper, we introduce the concept of explanation hack- ing, or X-hacking form of p-hacking (Head et al. 2015) applied XAI metrics. X- X-hacking refers to the practice of deliberately searching for and selecting models that produce desired explanation while maintaining 'acceptable" accu- racy. Unlike other adversarial XAI attacks, X- X-hacking ex- plores plausible combinations of analysis decisions to build a pipeline that might otherwise have been found innocu- ously. is a strategy of lying through omission, exploiting phenomenon known as model multiplicity (Black et al.) 2022 where different models offer equivalent predictive performance (Brunet et al. 2022).
We extend existing work on model multiplicity to incorpo- rate the full analysis pipeline, including data preprocessing, feature extraction, choice of model class and hyperparame- ter tuning, steps which are often poorly reported in scientific publications.
Though the search for such decision sets by hand is long and laborious, it can be accelerated through the exploita- tion of automated machine learning (AutoML) solutions, or any automated model selection tools that facilitate dis- covery of model that may also be found-aa or presented as found-via human decision-making Recently, a number of AutoML solutions have been introduced that integrate data preprocessing into the pocision-making pipeline, opening up additional avenues for explanation attacks (Salchin et al., 2024
1. Introduction
Machine learning (ML) models are increasingly integral to eecision-making in critical sectors such as healthcare, crim- inal justice and public policy. As these models grow more complex, does the challenge of interpreting the rationale behind their predictions. This has given rise to explainable AI (XAI) methods, which aim to make model reasoning more transparent and globally maintain trust in MI. systems.
We demonstrate empirically in a post-hoc manner how off- the-shelf AutoML pipelines can be used, even with a lim- ited computational budget, to perform X-hacking on SHAP values for familiar real-world datasets, by 'cherry picking' those models that support desired narrative. Secondly, we develop a custom AutoML solution to facilitate ad-hoc X X-hacking at scale, formulating the task as a multi-objective optimisation (MOO) problem, selecting from the Pareto frontier of predictive performance and model explanations to find a 'defensible' model in the Rashomon set. We com-
'Equal contribution 'Deutsches Forschungszentrum für Künstliche Intelligenz GmblI IIstittte of Informatics, Ludwig- Maximilians- Universität München. Correspondence to: David Antony Selby <david antony selby @ dfki.de>>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

X-Hacking: The Threat of Misguided AutoML
AutoML
Preprocessing
Performance
Data
Feature extraction
Explanations
Model fitting
Framework for 'X-hacking' using AutoML. For a given dataset, an automated pipeline evaluates different combinations of modelling decisions, simultaneously optimising performance and XAI metrics
Figure 1:
pare random and Bayesian hyperparameter optimisation to maximise accuracy and minimise SHAA explanations of top features from common baseline and show that Bayesian hyperparameter optimisation is, on average, three times faster in finding defensible models. We compare the time taken in post-hoc and ad-hoc settings to obtain defensible model for susceptible datasets: the ad-hoc framework took an average of minutes- 10 times faster than post-hoc.
to examine the effect of different inputs and investigate po- tential biases (Gunning et al.) 2019). These methods can be model-specific, such as feature importance for random forests and DeepL.FFT for deep learning (Shrikumar et al. 2019) or model-agnostic, such LIME (Ribeiro et al. 2016 and SHAP values (Lundberg & Lee 2017). While
not without criticisms (Gosiewska & Biecek 2020)) Shap- ley value-derived explanations enjoy widespread adoption
Hickling et al., 2023). Increasing interest XAI and algo- rithmic fairness (Garg et al. 2020). and their inclusion in future legislative frameworks regulating the use of AI (Bordt et al. 2022). brings incentives for manipulation. Adversarial attacks against XAI or fairness metrics are emerging area of research. Attacks can involve manipulating model's architecture or its explanations or 'poisoning' training data Baniecki & Biecek (024). In particular, fairwsshig de- scribes the practice of attempting to deceive measures of algorithmic fairness, Approaches include approximating complex. biased model with ostensibly fair, interpretable one (Aivodji et al. (019)), designing an unfair model that switches to generating "fair' predictions when being au- dited, akin to automotive manufacturers cheating emissions tests (Slack et al. 2020), or performing biased sampling of the data points used compute the SHAP values (Laberge) et al. 2023a). Recently, authors have proposed methods to detect and thwart such attacks (Shamsabadi al. 2022 Carmichacl & Scheirer 2023).
Our contributions are as follows: (I) we show the ease with which X-hacking can be performed using off-the-shelf soft- ware with attacks different types of SHAP summaries in a post-hoc manner; (2) we perform directed (ad-hoc) X-hacking at scale, using Bayesian multi-objective optimi- sation, comparing it with random sampling: (3) we demon- strate how some datasets are more vulnerable than others to confirmatory explanations unrepresentative of the ground truth; (4) we compare post-hoc and ad-hoc X-hacking by calculating the time required to find the first defensible model; (5) discuss methods for prevention and detection of X X-hacking. Code to replicate our experiments is availlale on GitHub
2. Background
In this section, introduce the key concepts of XAI, multi- plicity and questionable research practices, whose intersec- tion gives rise to the threat of X-hacking.
Multiplicity Models and their explanations are
Explainable AI Traditional statistical models, such as generalized linear regression models or decision trees, are inherently or ante-hoc explainable and easily interpreted by humans. Algorithmic ML models (see Breiman 200IT) tend be more opaque, requiring post-hoc explanation methods
unique; multiple alternative models can deliver roughly equivalent predictive performance (Pawelczyk et al.) 2020 Brunet et al.) 2022 Rudin et al., 2022)). This phenomenon
known variously as model multiplicity (Marx et al. 2020).
underspecification (D' Amour et al., 2022) or the Rashomon
2

---

X-Hacking: The Threat of Misguided AutoML
effect (Breiman 2001 D' A'mour 2021 Procedural mul-
tools (including AutoML) make it easy for people obtain models with desirable Existing studies of multiplicity, however, are restricted to specific model families and do not cover modern end-to-end data science pipelines.
tiplicity describes models that have identical predictive ac- curacy but differ in their internal structures (Mehrer et al.] 2021 concept also known as a Rashomon (ishher et al. 2019) or the 'set of good models' (Gaanehh et al. 2025). A special case of procedural multiplicity is
not,
2020 Black E.
set
Consider a set of models M designed to perform a task (e.g. classification) on dataset D. We can quantitatively evaluate a model m E .M via a procedure eval (m, D, Q), where Q is quality measure, either a predictive perfor- mance metric (e.g. accuracy), denoted Qn(m) = (m)r(mm) or an inferential summary for a feature of interest (D(m) = I[m, x) for x E D. Classical inference is typically tied to measures such as model coefficients, p- p-values or effect sizes (e.g. Cohen's 's d), which have capacity to mislead (Pogrow) 2019), but also limit the family of models and data science pipelines at the (unscrupulous) analyst's ss disposal. Model- agnostic XAI metrics, such as SHAP values, open up a wider range of algorithms and possible combinations of hyperpa- rameters that may yield varied interpretations.
predictive multiplicity: where models achieve the same ac-
curacy but produce different predictions (Black et al.) 22022) The effects of multiplicity can be felt at the level of individ- ual predictions (Marx et al. 2020)) and on global properties such as model fairness and robustness (Rodolfa et al.) 2021 D' A'mour et al. 2022). Recent work has studied multiplicity in specific classes, including linear and generalized addi- tive models (Dong& & Rudin 2020; Zhhon et al.), neural networks (Mehrer et al., 2020 Black E., 2021 Laberge
et al. 2023b), sparse decision trees (Wang al. 2022) and
random forests (Smith et al.) 2020).
p-hacking In null hypothesis significance testing, p- hacking is questionable research practice whereby re- searchers, equipped with many possible data analysis choices, only report those yielding a 'significant" result (Wasserstein & Lazar 2016) while others languish unpub- lished (Scargle, 1999). Strategies for p-hacking include multiple testing, selective reporting and favourable impu- tation (Stefan & Schönbrodt 2023). Any choices that are contingent on data, rather than pre-specified study proto- col, are vulnerable to these 'researcher degrees of freedom', whether or not there is a conscious desire to mislead (Gel- man & Loken 2013). This phenomenon is considered to have contributed to the reproducibility crisis in scientific research (Wicherts et al. 2016)) Even in systematic reviews of literature, p-hacking may not be easy to detect (Rooprai ct al. 2022), though pre-registrations and replication studies aim to discourage the practice (Hussey 2021). In quanti- tative sciences, details of data preparation are often poorly reported, yet can have a significant impact on results (Jani cet al. 2023). Some authors (Heyard & Held 2022) suggest data and code should be made available at the preprint stage rather than at final publication time, however the willingness of researchers to do varies by discipline (Hussey 2023
Ordinarily, model search is an optimisation problem.
arg max Qn(m), MEM
where Q= = pe)((m)) X-hacking extends this with com- peting objective, Z(m,
A post-hoc strategy retrieves a (Rashomon) set of 'good' models exceeding a pre-specified baseline, e-g. M, (m : perf(m) 2 b). then optimises the other metric, e.g. arg maxmeM, [n(m, x). Alternatively, an ad-hoc strat- egy combines these metrics. inter alia (e.g. risk of get- ting caught, 2) into multi-objective optimisation, where QD = ('1...., .n. is a vector.
Here, we intentionally do not specify the expressiveness of conditions that encode model interpretations or the process of generating the data science pipeline.
4. Explanation Hacking
In this section, we outline framework, illustrated in Fig ure I that unscrupulous or inexperienced actors might em- ploy to find publishable ML models with predetermined, possibly contrarian explanations. We consider three quan- tifiable metrics that an actor might consider when trying to hack XAI metrics.
Goldacre et al. 2019).
3. Problem statement
The goal of explanation hacking is to find model that cor- roborates a predetermined view of the world. For example, pharmaceutical company may wish to show that drug is not associated with adverse outcomes, an organisation may wish to claim decisions were not biased against protected group, or lobbyist may wish to show that smoking protects against cancer.
Explanations SHAP values are typically presented visu- ally in the form of force plots, SHAP summary plots, partial dependence graphs or feature importance (see, e.g. Lund berg et al., 2018 Stenwig et al. 2022). If a specific data point, xG, such as an individual patient or credit card appli- cant (the 'suing set'; "o';:SShan (Shamsabadi et al., 2022), is of interest, then the adversary can 'fish' for a trained model that gives
Thanks to predictive multiplicity, automated model selection
3

---

X-Hacking: The Threat of Misguided AutoML
SHAP value in the desired direction, e.g. I(m, v.) O, allowing them claim that Al-assisted decision was not based on protected characteristic. Model- or large group- level explanations require aggregated SHAP statistics. If in- terpreting a model based on ranked feature importance, then
that confirm the desired hypothesis and the Rashomon set: the confirmatory Rashomon set.
4.2. Directed Search
one could set [m,, x) = rank (mean | [SHAP(x)] )) > k,
A more efficient, but higher-effort, ad-hoc approach involves custom multi-objective optimisation.
to seek models where the feature of interest is not in the "top k' most important features. Partial dependence plots, despite being visual summary, may also be 'hacked": adversary can examine the relationship between SIIAP and specific feature approximately with the slope of simple lin- car model (Figure 4), or even I = Corr(x, SHAP(x)) and search for models with positive, negative or no correlation.
Let M be the population of models in the AutoML search space and (m(m, x) = {q1, 92) be vector of objectives, with ạ = perf(m, D) and (2 = I& (m, x) for m E M. Then, for each feature of interest x, we select the optimal model using ( either from a random subset of models [my} from the suggestions of Bayesian optimiser.
We say model m (Pareto) for a feature x if and only if there is no criterion in which m' > m (m' is superior to m), and at least one criterion qj in which m > m'. An analyst may select a trained model from the Pareto front of feasible solutions.
another model m' E A
Defensible models The premise of an X-hacking frame- work is for agent to select combinations of analysis decisions-such the model family, architecture, data transformations or imputation methods-cach of which is individually defensible (i.e. could be justified in peer re- view), but whose combination leads to a confirmatory result. decision defensible if a plausible justification can be proffered the report accompanying the analysis. For this we consider two alternative appeals to authority: the model has superior predictive performance to baseline (i.e. the
A weighted-sum scalarisation approach multi- objective optimisation can also be used. Details are in the Appendix C
Remark
5. Experiments
Rashomon set of 'good models" Ganesh et al.) (025); y or the choice is considered 'standard practice" by peers in the target domain. As the former is more readily quantified (i.e. (m.) in an automated search, for the remainder of this paper we use the term "defensible" models simply to refer to the Rashomon set of models with good predictive performance.
In this section, we explain our experimental setup and present empirical results of X-hacking as described in $ $ and & 4.2 Next, we demonstrate how certain features can be robust to X-hacking. Finally, we compare X-hacking methods to see how quickly they are each able to find a model that is in the confirmatory Rashomon set.
Hence, we treat X-hacking as balancing competing objec- tives of predetermined explanations, measured via XAI met- rics, and predictive performance.
5.1. Cherry Picking
To demonstrate the feasibility of "easy' X-hacking ap- proaches, we present an empirical evaluation using off- the-shelf XAI and AutoML software packages and several familiar real-world datasets.
4.1. Cherry Picking
Let my be a suitable baseline model and let b = perf, (m+) be its performance, and I& (x) = (S (x))), (x)),, the mean absolute SHAP of feature x for model m. Let x be the top ranking feature by importance (rankm, (xmn)) = 1) for the baseline model, where X == arg maxxeD Im, (x). Let M be the search space of mod- els in the AutoMI. solution and St,D C M the set of models returned by AutoML after running for duration t. The Rashomon set of models having comparative or su- perior performance the baseline is given by RTDD := {m m St perf
Data We selected 23 datasets from the OpenML-CC18 classification benchmark (Bischl et al. 2021). where the task was binary classification from tabular features. For all datasets, categorical features were one-hot encoded and samples with missing values omitted.
Tools Python packages scikit-lears Pedrreosaa et al. (01TT) and auto-sklearn (Feurer al.) 2019) built and trained the MI models, and package shap (Lundberg) et al. 2018) estimated SHAAP values from the successfully evaluated models. Models were evaluated on an internal cluster using 192 CPUs with 300GB RAM.
2 b). Let C.D be the set of models that confirm the desired hypothesis, e.g. CrDD {m rankm (xin)) ) > k). the set of models where most important features.
X is not in the top k
A human or machine can then simply 'cherry-pick' any model from D, the intersection of the set of models
The baseline model was random forest classi- fier trained with sci k weiit-een en with all parameters set to
X-hacking

---

X-Hacking: The Threat of Misguided Auto.L
Bioresponse
pc4
wdbc
500 85
blood-tran trand-transfssion
0,000
Dest
climate model
diabetes
49
-0.001
madelon
0.002
wilt
415
qpar-biodag
0.003
pc3
numera128.6
credit-g
0.%
Jm1
17.0
bank-marketing
kc2
68.%
Upd
Figure 4: Lines of best fit of SHAP against feature he ght, for different models evaluated by AutoMI. indicating avail- ability of several choices for models (Other) that 'flip" the SHAP value
pc1
65.99
predit-approval
kc1
35.7%
cardiac-disease
Figure 2: Proportion of models with superior predictive performance where 'top" feature from baseline is no longer ranked in the "top 3", Each bar represents one dataset
slope indicating the sign of any (approximate) linear rela- tionship between the feature and the outcome. Predictive performance measured by accuracy score; the AutoML. pipeline was optimized for this metric as single objective (the default) with no consideration given to SHAP during the initial model search; model explanations were evaluated post-hoc.
default. To get different candidate models against the base- for one hour on each dataset,
line, we ran auto-skles with runtime limit of 100 seconds for each model configu- ration. For the baseline and all models evaluated in AutoML, the shap's model-agnostic Kerne e Expla nne routine was employed to compute SHAP values, using background sample size 50 and test sample size 100.
Results: feature importance With no agenda of our own, we arbitrarily chose the most important feature at baseline as the target for explanation manipulation. FFigure shows, for each dataset, the proportion of models returned by the AutoML pipeline where this feature is no longer ranked in the 'top 3' for feature importance, conditional the model's ' predictive accuracy being higher than at baseline.
Manipulation of two different SHAP-based explanations was attempted: feature importance ranking and partial depen- dence plots. Feature importance was determined as either the ranking magnitude of the mean absolute SIIAP val- of the features. The visual appearance of the feature dependence plot quantified by simple linear regres- sion model of SHAP against respective feature values, the
For about a third of the datasets, it was not possible the time available to find a more accurate ML model that 'top- pled' the baseline most important feature. For similar proportion of datasets, almost every more performant model had different top features. Figure shows the distribution of change in mean absolute SHAP values, compared to the base- line random forest classifier, for the cardiac- di Riseese dataset. The distribution of differences close to zero for the top three features, suggesting that this dataset, may be difficult find a modelling pipeline that renders the 'top' feature at baseline unimportant in reported model.
0809
0,02
0000
-2002
-0.004
Results: feature dependence As well as the importance of feature unsigned quantity misguided analyst may wish to manipulate the shape of the effect. The lines of best fit between SHAP and feature height (which has high importance at baseline; feature in Figure 3) for the cardiac-di sease dataset are plotted in miFiurre 4 Sev- eral modelling choices are available that 'flip' the SHAP
Figure 3: Changes in feature importance relative to baseline for cardiac-disse sease data. Features are ordered by baseline importance.
5

---

X-Hacking: The Threat of Misguided AutoML
value for this feature while being more accurate than the baseline. Hence, an analyst might present the underlying partial dependence plot evidence of the negative effect of height on cardiac disease-omiln alternative models, including the baseline, show either pos- itive or no linear effect. Further results are given in Ap- pendix D
Bayesan ond hewwy
the information that
Whilst "cherry picking' desired model from an undirected set of candidates may seem easier approach implement, such a 'cherry' may not be found in a reasonable time using standard AutoML tools, or it may be difficult to define a space of uniformly defensible models.
suraton
10
12
Time (h)
5.2. Directed Search
Cumulative minimum of mean absolute SHAP for top features of dataset credit pedit-gg t- for Bayesian optimisa- tion and random sampling
Figure 5:
To deliberately hack explanations, we created a custom Au- toML solution that allows us to perform multi-objective op- timisation performance and explanations in a distributed manner on familiar real-world datasets.
and mean absolute SHAP was optimised to be minimum. To demonstrate how fast/slow Bayesian multi-objectivve opti- misation is, we run the setup with same time and resource limits but with a random optimiser.
Data as given in $ 5.1
Tools For this experiment, the AutoML solution should support multi-objective optimisation over many model classes and their hyperparameters in distributed manner. Most AutoML solutions do not support these requirements seamlessly (see Appendix A). Therefore, we create our custom AutoMI. solution by using the search space from auto-sklearn, enabling Bayesian multi-objective op- timisation using untuna and running of pipelines in a distributed manner using Ray-tune. More details are in Appendix B
Results: directed search The cumulative minimum mean absolute SHAP of four 'top' (at bascline) features for defensible models was recorded for random sampling and Bayesian optimisation over 12 hours of runtime and plotted in Figure for the credi t-gg dataset. To sum- marise this information for all datasets, Figure 6 shows the ratio of duration of time when cum min (nean[[SHA((CC for Bayesian optimisation was always lower than random sampling over the top 4 features from baseline across all datasets. For these features, on average, Bayesian opti- misation reached the overall minimum of mean absolute SHAP obtained by random sampling 3 hours earlier. For the remaining features where the overall minimum mean absolute SHAP was lower for random sampling, on average, random sampling reached the overall minimum obtained by Bayesian optimisation nearly 55 minutes earlier. This indi- cates, on average, Bayesian optimisation
Directed X-hacking To demonstrate intentional manipu- lation of explanations, the top features from each dataset were taken, which correspond to the top 4 most impor- tant features in the random forest baseline. In total, we perform this experiment for 92 features. To get differ- ent candidate models against the baseline, the custom Au- toML setup was run for 12 hours for the top features indicated by the baseline, with runtime limit of 1 hour for each model configuration suggested by the Bayesian optimiser. For all models evaluated in our custom Au- toML setup, shap's 's model-agnostic explainer. Kernel Explainer routine was employed to compute SHAP val- ues, using a background sample size 50 and test sample size 100. Manipulation of SHAP-bases feature importance was attempted. MOTPESampler (Ozaki et al.) (2022) was used as the Bayesian optimiser. Feature importance was determined as the magnitude of the mean absolute SHAP values of the features. Predictive performance was measured by accu- racy score. The AutoML pipeline was optimised for these two objectives simultaneously as a multi-objective optimi- sation where accuracy score was optimised be maximum
times faster than random sampling for a majority of features defen- sibly hack explanations. For some features (20/92), no defensible models were found by both random sampling and Bayesian optimisation in the current setting, indicating their robustness to X-hacking.
5.3. Vulnerability to X-hacking
Quantifying the vulnerability of dataset to X-hacking is equivalent to measuring predictive multiplicity: exist- ing metrics include Rashomon Capacity, based on the KL- divergence for a set of probabilistic classifiers (Hsu & Cal mon 2022). practice, predictive multiplicity may only be
6

---

X-Haccing: The Threat of Misguided AutoML
Simulation results Figure shows three simulated datasets containing two collinear features measured with varying levels of noise. Additivity of SHIAP values allows attribution to be transferred from one feature to the other without loss of accuracy, but this effect decreases with the signal-to-noiss ratio. This shows the ability to manipulate XAI metrics for a given feature depends on the level of information shared with others.
46.
30
#Raturees
20
15
We demonstrate in Figure 8tthtt is possible flip SHAP values of features with redundant information present in other variables, without loss of accuracy, but not for vari- ables without such information redundancy. Having access to the data genteranng process allows us to be aware of the relations between variables and the redundancy of informa- tion present within them. We select two variables (one with redundant information captured by dependent nodes and one without) to demonstrate which variables can and cannot be flipped for particular dataset. Therefore, how easy it is to manipulate the SIIAP values of a particular feature- and hence the need for a directed search or a more tightly curated search space- is determined by redundancy among the features. If all input features are completely independent of other, manipulation of SHAP values values-without without change of model predictive accuracy is not possible.
5
0.0 0.1
0.2
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Ratto
Figure 6: Ratio of time duration (over 12 hours) Bayesian MOO had better optimisation for mean absolute SHAP than random MOO for top 4 features from all datasets
0.92
0.91
0.50
No noise
0.89
High noise
Shaps/festure 1)
5.4. Cherry-picking vs directed search
Figure 7: Trade-off between accuracy and (approximate) SHAP dependence slope
model from Rashomon set obtained after passing dataset through an off-the-shelf AutoML. solution, using directed search, one can continue performing X-hacking until a confirmatory model is found. Using the results obtained in the experiments $ and $ 5.2 we see the time it takes for both X- X-hacking techniques to find the first confirmatory defensible model in one hour of runtime. In this experiment, we see those defensible models where the rank of the top feature from bascline is no longer in the 'top 3'.
As opposed
approximated, as is typically computationally infeasible to evaluate every possible analysis pipeline. To determine when a feature is robust X-hacking, we perform simula- tion.
Simulation Using the multi-objective optimisation pack- age Optuna (Akiba al. 2019). evaluate 50 models simulated dataset. SHAP values are caleulated using the shap package's 's mgdel-agnostic explainer. Sample sizes are the same as in $ 4.1
Data and Tools as given in $ 5.1 and $ 5.2
11V
Results: First confirmatory defensible model For 15 out of 23 datasets, using directed search, defensible model could be found without waiting for an hour, as opposed to the case of post-hoc setting. For 14 such datasets, a defensible model could be found in less than 6 minutes of directed X- X-hacking- Table Tssows the time it took (in seconds) for directed X-hacking to find the first defensible model. This demonstrates on average 10x faster finding of defensible models for datasets and their features vulnerable to X-hacking.
04
(a) Redundant
(b) No redundancy
Figure& 8: Predictive accuracy versus SHAP values for simu- lated features (each point corresponds a trained pipeline).

---

X-Hacking: The Threat of Misguided AutoML
Table I:
Time taken (in minutes and seconds) by directed search to find the first confirmatory defensible model for the top feature of each dataset according to the baseline random forest in an ad-hoc X-hacking setting.
computing power and well-specified search space, the au- ditor may explore all possible pipelines, but in practice the explored pipelines may not be representative, so an effec- tive search would exploit domain knowledge or additional information. The goal is not necessarily to reverse-engineer the exact pipeline used, rather to investigate the sensitivity of reported explanations. An example given in FFiuure
Dataset Name
Time taken
breast-w
3 s
ilpd
5 s
diabetes
7
Pipeline analysis How close is a coded pipeline to what is reported in the paper, or to accepted practices in the field? Though labour-intensive task for human domain expert, pipeline-pipeline similarity can be converted into graph isomorphism problem, if data analysis pipelines are repre- sented as directed acyclic graphs and the similarity quanti- fied as graph edit distance (Ono et al., 2021). A challenge of this approach is extracting the pipeline correctly (Redyuk) et al. 2022) from code or text. Moreover, parsable, discov- erable pipelines in the literature may not be representative of best practice.
blood-transfusion
10s
cardiac dissc--iieaas sease
24
pel
37
kc2
428
wilt
47
climate-model
1 m4s
wabe
1 m 16s
numerai28.
m 45
credit-approva
2 m 17s
pe3
4 m 39
kcl
5 m 20
credit-g
55 m 18
7. Discussion and limitations
2000
The experiments presented here exploit SHAP values, which have their own limitations (Huang Marques-Silva 2023). but the general framework may be adapted to p-values or other XAI metric. In principle, any post-hoc explanation method susceptible to model multiplicity faces similar risks. A systematic exploration of different XAI metrics is planned as future work.
baseline best
1500
Density
1000
500
0.00-00.000-00000--000000 0.0000 0.0002 0.0004 0.0006
Figure 9: Marginal distribution of SHAP slopes over AuvaML--eallaaed models for the feature height
The unethical use of statistical tools has a long history in science (Huff) [993). exacerbated by a 'publish or perish" culture. While we do not present any evidence of X-hacking 'in the wild', we argue the latest developments in AutoMI. and XAI provide means, motive and opportunity. A guided search over an ever- ever-growing field of data science operators, including X-hacking, presents a more dangerous type of scientific 'fake news' as can be done at scale, with low additional effort by using readily available tools, and is much more difficult to detect it does not take much effort to disguise its use even from an educated, expert mind.
6. Detection and Prevention
Understanding the scale of the problem- as well as detec- tion and prevention-requinn systematic ability to identify potentially selectively reported MI. pipelines. In this sec- tion, we outline several approaches that might be used to highlight inconsistencies indicative of explanation hacking. We consider these approaches to complement existing pro- posals for detection of fairwashing (c.g. [Shamsabadi et al. 2022). Feasibility of these methods depends on the level of access to raw data, code and detailed analysis protocols.
With small sample sizes, the effort and statistical expertise needed to 'squeeze' insights out of the data was much higher. Al methods enable such dishonest practices to be (semi-) automated, scalable and accessible to researchers and practi- tioners at lower effort and cost. The deception need not even involve all authors of publication, as one author might be tempted to use X-hacking in isolation, without knowledge of their co-authors. Pre-registered analysis plans could prove valuable, if comprehensive enough to cover the range of models and hyperparameters in an AutoML search space.
Explanation histograms With access to raw data, a re- viewer may set up their own AutoML pipeline to explore the space of possible pipelines, performing the same process as in 4.1I but with the goal of visualising the marginal distri- bution of XAI metrics over likely pipelines and locating the reported metric within this distribution. A reported value lying in the extreme tails (or outside the distribution com- pletely) would be consistent with explanation hacking, and may be quantified via hypothesis testing. With infinite time,
Aware of the ease of lying through X-hacking, we do not present comprehensive how-to guide or tool that
8

---

X-Hacking: The Threat of Misguided AutoML
could be used by an adversarial actor, demonstrating only proof of concept. However, our code and exper- iments are open source (GitHub: lhttps://githti com/
References
Agrawal, A., Pfisterer, F., Bischl, B., Buet-Golfouse, F., Sood, S., Chen, J., Shah, S., and Vollmer, S. Debias- ing classifiers: is reality at variance with expectation?. May 2021. /tthtt::////// doi. dow...rg/....13333333
and we encourage others to adapt the experimental setting to their own view on what plausible effort might be on the part of an adversary.
ssrn 3711681
In this paper, we have considered the case of inexperienced scientists or adversarial analysts, however the potential for perverse incentives ranges much further, due to additional financial motivations for reporting misleading explanations. Existing and future legislative frameworks regulating the use of AI, particularly in high-stakes scenarios (e.g. EU Al act; GDPR) increasingly mandate or encourage explainability (Panigutti et al. 2023) including in industrial contexts.
Ahmed, Z., Amizadeh, S., Bilenko, M., Carr, R., Chin, W.-S., Dekel, Y., Dupre, X., Eksarevskiy, V., Filipi,, S., Finley, T., Goswami, A.. Hoover, M. Inglis, S., Inter- landi, M., Kazmi, N., Krivosheev, G., Luferenko, P., Matantsev, I., Matusevych, S., Moradi, S., Nazirov, G., Ormont, J., Oshri, G., Pagnoni, A., Parmar, J., Roy. P., Siddiqui, M. Z., Weimer, M. Zahirazami, S., and Zhu, Y. Machine learning at microsoft with ml.net. In Pro- ceedings of the 25th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, KDD 119 pp. 2448-2458, New York, NY, USA, 2019. Associa- tion for Computing Muchinery. ISBN 9781 9781450362016. /doi doi: 10. 10.11455322250003333666 URL, http:::/t///t...... /doi
We view our demonstration of the plausibility of this new modality of lying necessary step for developing effec- tive countermeasures.
Impact Statement
3330667
This paper introduces the concept of 'X-hacking,. a misuse of XAI and AutoML tools produce defensible but mis- leading model explanations. The study highlights a potential avenue for adversarial misuse that could undermine trust in machine learning systems and exacerbate the reproducibility crisis in scientific research.
Aivodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., and Tapp. A. Fairwashing: the risk of rationaliza- tion. In Proceedings the 36th International Conference on Machine Learning, pp. 161-170. PMLR, May 2019. URL http:: ltttps://proceed.nnng . www.m.......////ee a aivod]il9a.
ISSN: 2640-3498.
While the goal of this work is to inform the community of the risks associated with X-hacking and propose coun- termeasures, there is a potential for these findings be misused by actors seeking to exploit these vulnerabilities. Technical details that would lower the barrier to adversarial use have been omitted. Our open source code is intended to facilitate reproducibility and to encourage further research into detecting and preventing X- X-hacking. We emphasize the importance of responsible dissemination of related tools and frameworks and advocate for the inclusion of detection strategies as part of routine XAI evaluations.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A Next-generation Hyperparameter Opti- mization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, KDD) '19, pp. 2623-263311111 2631, New York, NY, USA, July 2019. Association for Comput- ing Machinery. ISBN 978-1-4503-6201-6, 11 1145/3292500.3330701 URL /tt.httpss///o..org// 11.145/33992500333000
doi: 10.
Amazon. Amazon SageManere 2017. URL https
The societal implications of X- X-hacking are potentially broad, particularly in high-stakes domains such as healthcare, fi- nance and criminal justice, where biassed or misleading explanations could have significant ethical and legal ramifi- cations. Our aim is to mitigate these risks through awareness and contribute to the development of more robust and trust- worthy AI systems.
amazon wwwp://......cmmaaaaaaaaaeee
Autonomio. Automonio Talos. Computer software. URL
httt:s://giChbb..com//u http://////.ooooooo/oaalee.o//e. Ac- cessed April 29, 2025.
Bakshy, E., Dworkin, L., Karrer, B.,, Kashin, K., Letham, B., Murthy, A., and Singh, S. AE: A domain-agnostic plat- form for adaptive experimentation. In Proceedings of the International Conference on Neural Information Process- ing Systems, 2018. wo/papers//aeeeeooooo workshop.pdi
Acknowledgements
The project on which this report is based was funded by the Federal Ministry of Education and Research under the fund- ing code 03ZU1202IA. The responsibility for the content of this publication lies with the author.
github
Baniecki, H. and Biecek, P. Adversarial attacks and defenses in explainable artificial intelli-
gence: survey. volume 107, pp. 102303, 2024.
9

---

X-Hacckng: The Threat of Misguided AutoML
doi: URL
Brunet, M.-E.,, Anderson, A., and Zemel, R. Implications of Model Indeterminacy for Explanations of Automated Decisions. In Advances in Neural Information Processing Systems, volume 35, pp. 7810-7823, December 2022.
com/
science science/articleeee
Bennet, P., Doerr, C., Moreau, A., Rapin, J., Teytaud, E., and Teytaud, O. Nevergrad: black-box optimiza- tion platform. SIGEVOlution, 14(1):8-15, April 2021. doi: 10. 10.114573460330334660312. httt:stt:://// org 10. 1ww.1114/3343330 0:r/:00.1111/36001133363622
htt.htttts:iiii y/ 3/33blyo]]
Carmichael, Z. and Scheirer, W. J. Unfooling Perturbation-
Based Post Hoc Explainers. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6925-65 6934, June 2023. ISSN 2374-3468, 2159-5399. doi: 10.1609/ aaai vaai.v37i6.258477 wttppttpsss///os.aaaa..oo
Bergstra, J., Yamins, D., and Cox, D. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th Inter- national Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 115- 123, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL RRLLtttss:///rrceeeee bergstral3. html
Claesen, M., Simm, J., Popovic, D., Moreau, Y., and De Moor, B. Easy hyperparameter search using optu- nity, 2014. URL htttttt//aaxx..coo org/ /atgg//b////e 1412 1114
mlr. wwwpsrrsssss.
Cowen-Rine Rivver,, A. I., Lyu, W., Tutunov, R., Wang, Z., Gros- nit, A., Griffiths, R. R., Maraval, A. M., Jianye, H., Wang, J., Peters, J., and Bou-Ammar, H. Hebo: Push- ing the limits of sample-effciiet hyper-parameter op- timisation, September 2022. ISSN 1076-9757. URL http:///010....11111111111/11//1
Bischl, B., Bischl, B., Casalicchio, G., Feurer, M., Gijs- bers, P., Hutter, F., Lang, M., Gomes Mantovani, R., van Rijn, J., and Vanschoren, J. Openml benchmarking suites. In Vanschoren, J. and Yeung, S. (eds.), Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL
D' A'mor, A. Revisiting rashomon: A comment on "the two cultures". Observational Studies, 7(1):59-63 2021. doi: 10. 302.3/3/bb22222200000 URI /ttpt::/tt//..../// Scg/10.1153//pp 2021. 0022
https: /ttp://bit..../ 4ipxYLL
Black, E., Raghavan, M., and Barocas, S. Model Mul- tiplicity: Opportunities, Concerns, and Solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, pp. 850- 863, New York, NY, USA. June 2022. Association for Computing Machinery. ISBN 978- 11881--4503-9352-22 doi: 10.1 00.1145//5331446333334444, URL https://dl acm
D' A'mour, A., Heller, K., Moldovan, D., Adlam, B., Ali- panahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y.. McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T. F., Raman, R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne,
Black F.,, Leino K., F. M. Selective ensembles for consistent predictions, 2021. URL
M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M.. Wang. X., Webster, K., Yadlowsky, S., Yun, T., Zhai,
biblic/10348722
X., and Sculley, D. Underspecification presents chal- lenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23(1)222::1023- 10297, January 2022. ISSN 1532-4435.
Bordt, S., Finck, M., Raidl, E., and von Luxburg, U. Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Trans- parency, FAceT '22, pp. 891-905, New York, NY, USA, June 2022. Association for Computing Machin- doi: 10.1145/3531146 3533153. URL http:::://dllaa.......... 1145/3531145 3533155
Dong, J. and Rudin, C. Exploring the cloud of variable importance for the set of all good mod- els. Nature Machine Intelligence, 2(12):810-8) 824, De- cember 2020. ISSN 2522-5839. doi: 10.1038/ com/ +42256--20000224--0 URL httt::ttt///w/t www.we.www..eeemmmmm..wa.a.w Num-
ery. ISBN 978- 978-1-4503-9352-2.
Breiman, L. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statisti- cal Science, 1993)1199-231, August 2001. ISSN 0883- 4237, 2168 8745. doi: 10. 00.111/////1002211777
ber: 12 Publisher: Nature Publishing Group
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. AutoGluon -abuulaar Robust and accurate AutoML for structured data, 2020. URL http: wwwps/wwwwwwggaa////.oesa
URL bit bit.ly/ 1tt..ly/43yRR Publisher: Institute of Mathemati- Statisties.
10

---

X-Hacking: The Threat of Misguided AutoML
Falkner, S., Klein, A., and Hutter, F. BOHB: Robust and efficient hyperparameter optimization scale, 10-15 Jul 2018. URL URLattps::/prrceeinngg
Gosiewska, A. and Biecek, P. Do Not Trust Additive Expla-
nations, May 2020. URL hhttt:://aaiiiii www.riiiivoorooooeeeeeoeree /abs/
press/
[cs, stat].
w80/falkner18a.c
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., and Yang. G.-Z. KAI-Explainable artificial intelligence. Science Robotics, 4(37)xeay77120, 2019. doi: 10.1126/sciroootiiss.aay 1200 URL
Feurer, M., Klein, A.. Eggensperger, K., Springenberg, J. T., Blum, M., and Hutter, F. Auto-skleam: Efficient and Robust Automated Machine Learning. In Hutter, F. Kotthoff, L., and Vanschoren, J. (eds.), Automated Machine Learning, pp. 113-134. Springer International Publishing. Cham, 2019. ISBN 978-3-030-05331111 978- 3-030-05318-5. doi:
https://wws http://www.eeeeee..eee.........a
1 1126/ htt//scirroooioco aay.ll
Head, M. L.,, Holman, I.,, Lanfear, R., Kahn, A. T., and Jennions, M. D. The Extent and Consequences of P- Hacking in Science. PLoS Biology, 13())e1002106 106, March 2015. ISSN 1544-9173, doi: 10.1571/joumll.pboo 1002106. URL https::/www. ncbi mwm.nnmmn.. mih gov httpp////rrtticlee////433333333/
URL
spr iiger. com/10. 978 3- 030-033011100 - __6 Series Title: The Springer Series on Challenges in Machine Learning.
Fisher, A., Rudin, C., and Dominici, F. All Models are Wrong, but Many are Useful: Learning a Vari- able's Importance by Studying an Entire Class of Pre- diction Models Simultaneously. Journal of machine learning research JMLR, 20:177, 2019. ISSN 1532- 4435. URL htw.t:ttps..cc www. ncbi n mlm. n nih. gov/
Head, T. Kumar, M., Nahrstaedt, H., Louppe, G., and Shcherbatyi, I. sclki-opiimizescikitoppiiiiiii 2021. URL hittss://zenooe , org/ www./r/gceeeeoo.o.. records 5565057
pmc/ arti Setticcles/MMc8322
Heyard, R. and Held, L. When Should Data and Code be Made Available? Significaance, 19(2):4-5, 03 2022. ISSN 1740-9705. doi: 10.1111/174099713300622,
Ganesh, P., Taik, A., and Farnadi, G. The curious case of arbitrariness in machine learning, 2025, URL https:
//arxiv. wwwpiioorr//bba. ab5/ 14959
URL 21t1:///00...o0//111111777111131 61623
111111140111111111
Garg, P. Villasenor, J., and Foggo, V. Fairness Met- rics: A Comparative Analysis. In 2020 IEEE Interna- tional Conference on Big Data (Big Data), pp. 3662- 3666. December 2020. doi: 10.1109/Bieatte5002.. 2020.9378025. URL http:::://eeeppplooo/// 1eee
Hickling, T., Zenati, A., Aouf, N., and Spencer, P. Explain- ability in deep reinforcement learning: A review into cur- rent methods and applications. ACM Comput. Surv, 56(5), December 2023. ISSN 0360-0300, 10.11553623377 URL
hrg/abstrace http://dmmmmeeee//eee///e
Gelman, A. and Loken, E. The garden of forking paths: Why multiple comparisons can be a prob- lem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time, November 2013. URL http:://ww. httt:://ww.stat...o umbia -gelua/geelmma pdf
Hsu, H. and Calmon, F. Rashomon Capacity: Metric for Predictive Multiplicity in Classification. Advances in Neu- ral Information Processing Systems, 35:28988-29000 December 2022. URI. /ttttp:///ii... http://.tt..t.//fl.....
Huang, X. and Marques-Silva, The Inadequacy of Shapley
Values for Explainabiliity, February 2023. /tt/ttp::// arxiv. strlv.org/abs/2302.00 08160 arXiv:2302.08160 [cs].
Gijsbers, P., Bueno, M. L. P., Coors, S., LeDell, E., Poirier, S., Thomas, J., Bischl, B., and Vanschoren, J. AMLB: an automl benchmark, 2024. /ttL:ttt:///mm..
Huff, D. How to Lie With Statistics. W W Norton & Co Ltd, New York, reissue edition edition, September 1993. ISBN 978-0-393-31072-6.
papers/v25/22-1 0493. html
Goldacre, B., Morton, C. E., and DeVito, N. Why researchers should share their analytic code. BMJ, 367:16365, November 2019. ISSN 1756- 1833. doi: 10. 10.11366bmj.1635. URL https:/ www. .mjj com/ content/367/bnm 16365 Publisher: British Med- ical Journal Publishing Group Section: Editorial.
Hussey, I. A method to streamline p-hacking. Meta- Psychology, 5, April 2021. doi: 10.15626/mp.2020. 2529. URL https: :tttpa///oi... www.oogg....... 15626......m.m.
2020 2529
Google Cloud. API usage overview Vertex AL URL
Hussey, I. Data is not available upon request, May 2023.
https:// https////looudggoooe..oooo google. wtt.eeeeeeeeeeeeaaeeleaaaaa docs reeeeence
URL
burr/

---

X-Hacking: The Threat of Misguided AutoML
Jani, M., Yimer, B. B., Selby, D., Lunt, M., Nenadic, G., and Dixon, W. G. "Take up eight tablets per day": Incorporating free-text medication instructions into transparent and reproducible process for preparing drug exposure data for pharmacoepidemmiology Phar- macoepidemiology and Drug Safety, 32(6))651-660, 2023. ISSN 1099-1557. doi: 10. 1002/p/dss.555 URL https://onlinelibn
Lindauer, M., Eggensperger, K.,, Feurer, M., Biedenkapp. A., Marben, J., Müller, P., and Hutter, F. BOAH: A tool suite for multi-fidelity bayesian optimization & analysis of hyperparameters. 2019. URL http:://rxxx...oo a abs/ 0011990807555
Liu, Y.R.,, Hu, Y.-Q., Qian, H.,, Qian, C.,, and Yu, Y. ZOOpt: a toolbox for derivative-free optimization. Science China Information Sciences, 65(10))207101, 2022. ISSN 1869- 1919. doi: 10. 10.10/0/ 0100000//1111422221--22--- URL https -221--1----22----------
wiley _eprint:
com/ / /bbs/////////aa httpp///1111000pp.......
7/doi. //6bi..rg/10.1007///11111 1ttp:/1....1000011111
Jin, H., Song, Q., and Hu, X. Auto-Keras: An Efficient Neu- ral Architecture Search System. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD '19, pp. 1946- 1956, New York, NY, USA, July 2019, Association for Computing Machinery. ISBN 978-1-4503-6201-6. doi: 10.1155/29255003336622 URL (Rtttt/t/:///d acm /211111///22222 33302255003333404 3380648
Lundberg, S. M. and Lee, S.-I. A unified approach to in- terpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Infor- mation Processing Systems 30, pp. 4765 4765-4774. 4774. Cur- ran Associates, Inc., 2017. URL httth:t//itt..... 4GD4CPP
Komer, B., Bergstra, J., and Eliasmith, C. Hyperopt- sklearn. In Hutter, F., Kotthoff, L.,, and Vanschoren, J. (eds.), Automated Machine Learning: Methods, Sys- tems, Challenges, pp. 97-111. Springer International Publishing, 2019. ISBN 978-3-030-05318-5.
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F, Kim, J., and Lee, S.- I. Explainable machine- predictions for the prevention of hypoxaemia during surgery. Na- ture Biomedical Engineering, 2(10):749-760, Octo-
doi:
10. 10.10077977--3-000005531188 5. URL https: //do1
030 0000005588-0 5_5
ber 2018. ISSN 2157-846X. $41551-018-0304-0, URL https://www.w.... nature com/ conn/articlee/s455551 11////41111100000--000000 Number:
doi: 10.1038/
Laberge, G., Aivodji, U., Hara, S., Marchand., M., and Khomh, F. Fool SHAP with Stealthily Biased Sam- pling. March 2023a. URL
10 Publisher: Nature Publishing Group.
abs 15419
ar arXiv:2205.15419 [es].
Martinez- Caattin, R. BayesOpt: A Bayesian optimization library for nonlinear optimization, experimental design and bandits. Journal of Machine Learning Research, 15 (115):3915-3919, 2014. URL httt:://mml....m
Laberge, G., Pequignot, Y., Mathieu, A., Khomh, F., and Marchand, M. Partial order in chaos: Consensus on fea- ture attributions in the rashomon set. Journal of Machine Learning Research, 24(364):1-50, 2023b. URLhttp. 0149. html
7/jmlr. wwwp:oopppppp///////////
Marx, C., Calmon, F., and Ustun, B. Predic- tive Multiplicity in Classification. In Proceed- ings of the 37th International Conference on Ma- chine Learning, pp. 6765-6774. PMLR, November 2020. v 1119/ma 1199mara200ammm rx20a. html
Le, T., Fu, W., and Moore, H. Scaling tree-based auto- mated machine learning to biomedical big data with fea- ture set selector. 36(1):250-22 256500256,,000 2020. ISSN 1367-4803. doi: 10. 10.10033/boinffrmmatissszz777/ URL https:/7
mlr. .prss//
ISSN: 2640-3498.
McGushion, H. HutterMoGGnshoollyyppppaanneeeee hunter, 2018.
LeDell, E. and Poirier, S. H20 AutoML: Scalable automatic
URL https:///ittub.co
machine learning. 7th ICML Workshop on Automated
MuntterMooushioon//vyeeeaaaa original-date: 2018-96-011123:11.000)
_unterr
Machine Learning (AutoMI), July 2020. URL
7/www. automl. 2020/07 201_:l0___2022000___0000
Mehrer, J., Spoerer, C. J., Kriegeskorte, N., and Kietz- mann, T. C. Individual differences among deep neu- ral network models. Nature Communications, 11(1): 5725, November 2020. ISSN 2041-1723 doi: 1038/ 11447--00-19632--1 URL htttts:////// nature
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I. Tune: A research platform for distributed model selection and training, 2018. URL http:://rxix
httpp//aaxxi..oggabb/////// wwwpoorgg//////ooooeoe 05118
com/articles//ee 44
12

---

X-Hacking: The Threat of Misguided AutoML
Microsoft.
Automated machine learning
(Au- URL
Artificial Intelligence (UAI), pp. 809-818. PMLR, August
toML)
Microsoft Azure, 2018.
2020. URL.htttss:/prroeeee ngg..n....... ml r. prrsss/ v124 #124/pawelczyk20..uoo
https: https://azure..iic http:/aarrr.iiccooooree com/ au/
ISSN: 2640-3498.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Microsoft. Neural network intelligence, 2021.
URL
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P.. Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., and Cournapeau, D. Scikit-learn: Machine Learning in Python. MACHINE LEARNING IN PYTHON, 2011.
original-date: wwt wrwsoft/nsttoooooooo
https://git twagithub.....
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang. E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., and Stoica, I. Ray: a distributed framework for emerg- ing ai upplications. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implemen- tation, OSDI' pp. 561-577, USA, 2018. USENIX Association, ISBN 9781 1771999997778.
Ploíska, A. and Ploński, P. Mijar: State-of-the-art au- tomated machine learning framework for tabular data. version 0. 10.3, 2021. URL http:::/tttttthhtt com/
m_ mijar/mn -
Pogrow, S. How Effect Size (Practical Significance) Mis- leads Clinical Practice: The Case for Switching to Practi- cal Benefit to Assess Applied Research Findings. The
O' O'Malley, T., Bursztein, E., Long. J.. Chollet, F., Jin, H., Invernizzi, I.., et al. Keras Tuner. https:::t//..... //githutl com/
American Statistician, 73(sup (::15up)):22222344, March 2019.
2019.
ISSN 0003-1305, doi: 10.1188000033330522118888888 URL https:
Ono, P., Castelo, S., Lopez, R., Bertini, E., Freire, J., and Silva, C. PipelineProfflle:: A Visual Analytics Tool for the Exploration of AutoML Pipelines. IEEE Transactions on Visualization and Computer Graphics, 27( February 2021. ISSN 1077-2626, 1941-0506, 2160-9306. doi: 10. C0.109//VVCG.22000300000 1eeexp ore. wtt.:eeeeeeeeeeeeeeeeeeeeeeeeeee wttp:ooo////ooooo/o 9222086/
/tttps://do..oo /tt::///oii.o oro/ 000.:..000000001100000
2018. 2018.1549101 Publisher: Taylor Francis .eprint: https://doi.org//1..
http:////////000000000000 100000003000000 20...000....9911111
Redyuk, S., Kaoudi, Z., Schelter, S., and Markl, V. DO-
URL https://
RIAN in action: assisted design of data science pipelines. Proceedings of the VLDB Endowment, 15(12)3374-3371 August 2022. ISSN 2150-8097 doi: 10.14778/3554821
Ozaki, Y., Tanigaki, Y., Watanabe, S., Nomura, M., and On- ishi, M. Multiobjective Tree-Structured Parzen Estima- tor. Journal of Artificial Intelligence Research, 73:1209- 1250, April 2022. ISSN 1076-9757. 1.13188. URL https: http://:wwwwwwww...ooorrooo/ wair.. hhp//aarr/rticleee/eoo 13188
3554882. URL https://al........e waw. wrg.aaammm.oo......m /oo//1
14778/355492 3554882
doi: 10.16r3/jair
Ribeiro, M. T., Singh, S., and Guestrin, C. "Why should I trust you?": Explaining the predictions of any clas- sifier. In Proceedings of the 22nd ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Mining, ACM, August 2016. doi: 10.1145/ 2939672.2939778. URL https::/ org/10
Panigutti,, C., Hamon, R., Hupont, I., Fernandez Llorca, D., Fano Yela, D., Junklewitz, H., Scalzo, S., Mazz- ini, G., Sanchez, I., Soler Garrido, J., and Gomez, E. The role of explainable AI in the context of the AI Act. In 2023 ACM Conference on Fairness, Account- ability, and Transparency, pp. 1139-1150, Chicago II. USA, June 2023. ACM. ISBN 9798400701924. doi: 10.11445/59900133359441 URL httph:/tt/:/ll. acm acm 011111455/553113333 3594069
Rodolfa, K. T., Lamba, H.. and Ghani, R. Em pirical observation of negligible fairness trade-offs in machine learning for public policy. Nature Machine Intelligence, 3(10):896 904. Octo- ber 2021. ISSN 2522-5839.
doi: 10.1038/
Parmentier, L., Nicol, O.,, Jourdan, L., and Kessaci, M.-E. TPOT-SH: A faster optimization algorithm to solve the AutoML problem on large datasets. In 2019 JEEE 31st International Conference on Tools with Artificial Intelli- gence (ICTAI), pp. 471 478, 2019. doi: 10. 109/ICAA.
s42256-021-003966xx URL http::ttp:////c... nawure com com/articles///ee
021 00396-x
Rooprai, P., Islam, N., Salameh, J.-P., Ebrahimzadch. S., Kazi, A., Frank, R., Ramsay, T., Mathur, M. B., Absi, M., Khalil, A., Kazi, S., Dawit, H., Lam, E., Fabiano, N., and McInnes, M. D. F. Is there evidence of p-hacking in imaging research? Canadian Association of Radiologists Journal, pp. 0846537 2213372221111 139418, 2022. doi: 10.1177/ 084653712211139418.. URL http:: //ooi http::://oooooog10//0 org/10
2019.00072. URL https: /wteeeeeeeeeeeeeeeeeeeeeeeeeeeee 1eee orq/document/ 8 8995201
ISSN: 2375-0197.
Pawelczyk, M., Broelemann, K., and Kasneci, G. On Coun-
terfactual Explanations under Predictive Multiplicity, In Proceedings of the 36th Conference on Uncertainty in
1177/08465371221
PMID: 36412994.
13

---

X-Hacking: The Threat of Misguided AutoML
Rudin, C., Zhong, C., Semenova, L.,, Seltzer, M., Parr, R., Liu, J., Katta, S., Donnelly, J., Chen, H., and Boner, Z. Position: amazing things come from having many good models. In Proceedings of the 41st International Conference on Machine Learning, ICML/24, JMLR.org, 2024.
Stefan, A. M. and Schönbrodt, F. D. Big little lies: compendium and simulation of p -hacking strategies. Royal Society Open Science, 10(2):220346, February 2023. ISSN 2054-5703. doi: 10.1099// URL
/top:://rooyalsscceeppppbb wiw.gg.oriiiiiggg../..
Salchin, I., Islam, M. S., Saha, P., Noman, S., Tuni, A., Hasan, M. M., and Baten, M. A. AutoML: A systematic review on automated machine learn- ing with neural architecture search. Journal of Information and Intelligence, 2(1):52-81, January 2024. ISSN 29497159, doi: 10.1016/..ixxx.2223.. 10.002. URL
Stenwig, E., Salvi, G., Rossi, P. S., and Skjærvold, N. K. Comparative analysis of explainable machine learning prediction models for hospital mortal- ity. BMC Medical Research Methodology, 22
(1):53, December 2022. ISSN 1471-2288. doi:
bncmmeddeemmettoooooooooooeeeeee 10.1886612274402--005540w.
URL https: /
com/ wettrieve/////2244715533332
Salinas, D., Seeger, M., Klein, A., Perrone, V., Wistuba, M., and Archambeau, C. Syne tune: library for large scale hyperparameter tuning and reproducible research. In Proceedings of the First International Conference on Automated Machine Learning, pp. 16/1-23. PMLR,
Tang. Z., Fang, H., Zhou, S., Yang, T., Zhong, Z., Hu, C., Kirchhoff, K., and Karypis, G. inttoluoonmultimode (automm): Supercharging multimodal automl with foun- dation models. In Eggensperger, K., Garnett, R., Van- schoren, J., Lindauer, M., and Gardner, J. R. (eds.), Pro- ceedings of the Third International Conference on Auto- mated Machine Learning, volume 256 of Proceedings of Machine Learning Research, pp. 15/1-35, PMLR, 09- 12 Sep 2024. URL pttp://st///roocdnnggg .m1r press / /tt/////ttgn...ammammm
2022. URL https: htt.tpss//proceed.nng www.inggmllll.. .ml..press/ #188/sa inas2na.
ISSN: 2640-3498.
Scargle, J. D. Publication bias (the "file-drawer problem") in scientific inference, 1999. URL htttts://aaaatt org/abs htwaaa/shyysccss909000
Shamsabadi, A. S., Yaghini, M., Dullerud, N., Wyllie, s., Alvodji, U., Mkean, A. A. A., Gambs, S., and Papernot, N. Washing The Unwashable: On The (Im)possibility of Fairwashing Detection. In Advances in Neural Infor- mation Processing Systems, May 2022. URL https: //openreview.o wwwprrewiwwwwwweeeeeee.eee famet/forumte 3= Vy
Thortoo,, C., Hutter, F., Hoos, H. H., and Leyton- Leyton-Brown, K. Auto-WEKA: combined selection and hyperpa- rameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '13, pp. 847-855. Association for Computing Ma- chinery, 2013, ISBN 978- 778-1-4503-2174-7, doi: 10. 14452487755.2487666 URL https::/doi..oo. 10.1145/24875 24877777..2477
Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differ- ences, 2019.
Simon Blanke. Hyperactive: An optimization and data collection toolbox for convenient and fast prototyp- ing of computationally expensive models. https: //github. cwwm www.//////omnmoecc http//iimonnlaaeeee since 2019.
Vasile, M.-A., Pop. F, Nijă, M.-C,, and Cristea, V. MLBox: Machine learning box for asymptotic scheduling. Information Sciences, 433-434:401-416.
2018. ISSN 0020-0255. doi: 10.10166f.ins.201..00
Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Soci- ery, pp. 180-186, New York NY USA, February 2020. ACM. ISBN 978- 178-1-4503-7111-00 doi: 10.3145/3375627. 3375830. URL 1145/3375621 0111//337677733535333 3375830
005. http:: http:///wwwwwwieeeeeerreee.....e science/ /iieoce/arttaae///////0222222232
Wang, C., Wu, Q. Huang, S., and Saied, A. Econom- ical hyperparameter optimization with blended search strategy. In The Ninth International Conference on Learning Representations (ICLR 2021), 2021a. URL https://biis http///tt...//uuweee...ee
Smith, G., Mansilla, R., and Goulding, J. Model Class Reliance for Random Forests. In Advances in Neural Information Processing Systems, volume 33, pp. 22305- 22315. Curran Associates, Inc., 2020. URL https: //bit. 10..llyy33251000
Wang, C., Wu, Q. Weimer, M., and Zhu, E. Flaml: A fast and lightweight automl library. In Smola, A., Dimakis, A., and Stoica, I. (eds.), MLSys. mlsys.org, 2021b. URL ht /bllp. uni-trier de/ db/ conf f/lf/mllee sys
honff/mlyy//mmyyvee httpmmmyys222222 htmll 20001ww22111
14

---

X-Hacking: The Threat of Misguided Auto
Wang, Z. J., Zhong, C., Xin, R., Takagi, T., Chen, Z.,, Chau, D. H., Rudin, C., and Seltzer, M. TimberTrek: Exploring and Curating Trustworthy Decision Trees with Interactive Visualization. In 2022 IEEE Visualization Conference (VIS), 2022.
Wasserstein, R. L., and Lazar, N. A. The ASA Statement
-Values: Context, Process, and Purpose. The American
Statistician, 70(2): 10(2):129-1133) - April 2016. ISSN 0003- 1305, 1537-2731. doi: 10. 10.108000000110000000 305 1144100 15.
URL https://www.tanffoo.nnee.ccm hwwp://wwaaaaannnnnnooooooooaaaa
/oi/
Eull/I0 11111000000000011100000111111111
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., and van Assen, M. A. L. M. Degrees of freedom in planning, running, ana- lyzing, and reporting psychological studies: check- list avoid p-hacking. Frontiers in Psychology, 7, 2016. ISSN 1664-1078. doi: 10.3389/fssy
01832. URL httpp::/wwww w http://wwwww.otiii.....o.m. articles/10, 3389/ 2016. 01832
Wu, Q., Wang, C., and Huang, S. Frugal optimization for cost-related hyperparameters. Proceedings of the AAAI Conference Artificial Intelligence. 35(12):10347-
10354, 2021. ISSN 2374-3468, 2159-5399. doi: 10.1609) /rg/
aaai. ami.v3351...1239, URL http:://j...a.... Tndex..pp//AAAA//tttille///////
Zhong, C., Chen, Z., Liu, J., Seltzer, M., and Rudin, C. Exploring and interacting with the set of good sparse gen- eralized additive models. 36:56633-5666999 ISSN 1049- 5258. URL / pmc/artic /ttpaarrreeee/MM111132233
15

---

X-Hacking: The Threat of Misguided AutoML
A. AutoML and multi-objective optimisation
In order to conduct experiments to answer our research questions, an off-the-shelf AutoML solution should be able to perform multi-objective optimisation with custom objective functions over many model classes. However, to our knowledge, an off-the-shelf solution that incorporates all the required capabilities in single package does not exist, Table 2 shows different AutoML and hyperparameter optimisation libraries and their capabilities: support for multi-objectivve optimisation (MO0), support for many model classes (MMC) and support for custom objective functions (CO).
Table 2: Off-the-shed capabilities of different AutoML and hyperparameter optimisation solutions. Our solution (36) enables multi-objective optimisation in an existing AutoML library
Library
Type
MOO
MMC
CO
1. Ax (Bakshy et al. 2018)
hyperopt
Yes
No
No
2. Ba Ppt (Martinez- Cantin
2014)
hyperopt
No
No
No
3. BOHB (Falkner et al. 2018
hyperopt
No
No
No
4. CFO (Wu et al. 2021
hyperopt
No
Yes
No
5. BlendSearch (Wang et al.
202
hyperopt
No
Yes
No
6. HEBO (Cowen-Rivers et
2022
hyperopt
yes
no
yes
7. HyperOpt (Bergstra et al. 2013)
hyperopt
No
No
Yes
8. SkOpt (Head et al. 2021
hyperopt
No
No
Yes
9. Nevergrad (Bennet et al.
2021)
hyperopt
No
No
Yes
10. Optuna (Akiba et al., 2019)
hyperopt
Yes
No
Yes
11.
ZooOpt (iu et al. 2022)
hyperopt
No
No
Yes
12.
TPOT (Parmentier et al.
2019
AutoML
limited
Yes
Yes
13.
TPOT2 Alpha (Le et al.
2020
AutoML
limited
Yes
Yes
14.
Auto-Sklearn (Feurer et al.
2019)
AutoML
No
Yes
Yes
15.
Hyperopt-sklearn (Komer al.
2019
AutoMI
No
Yes
No
16.
FLAML (Wang al. (022bb
AutoML Yes
Yes
Yes
17.
H20 AutoML (LeDell & Poirier
2020
AutoML
No
Yes
No
18.
AutoGluon (Tang et al., 2024)
AutoMI
No
Yes
Yes
19.
MLBox (Vasile et al.
2018)
AutoML No
Yes
No
20.
Auto-Keras (Jin et al.
2019)
AutoML Yes
No
Yes
21.
AutoGluon-Tabular (Erickson et al. 2020
AutoML No
Yes
Yes
22.
AutoWEKA (Thornton et al. 2013)
AutoML No
Yes
No
23.
AutoML mijar-supervised Ploonkka & Ploński 2021
AutoML No
Yes
No
24.
Hyperactive (Simon Blanke since since
2019)
hyperopt
Yes
No
Yes
25.
Optunity (Claesen et al. 2014)
hyperopt
No
No
Yes
26.
HyperparneterHunter (McGushion
2018
hyperopt
No
No
No
27.
KerasTuner (O Malley al. 2019
hyperopt
No
No
Yes
28. Talos (Autonomio)
hyperopt
No
No
No
29. ML.Net (Ahmed et al. 2019
AutoML
No
Yes
No
30. NNI (Microsofi 2021
AutoML
No
No
No
31.
Azure AutoMI. (Microsoft 2018)
AutoMI.
No
Yes
No
32. Amazon SageMaker (Amazon 2017
AutoML
No
Yes
No
33. Google Vertex Al (Google Cloud)
AutoML
Yes
No
No
34. Ray Tune (Liaw et al. 2018)
hyperopt
Yes
No
Yes
35.
syne-tune Salinas et al. 2022
hyperopt
Yes
no
yes
36. Ray tune optuna autoskleamm
Automl
Yes
Yes
Yes
16

---

X-Hacking:
The Threat of Misguided AutoML
B. Custom AutoML solution
To enable distributed multi-objective optimisation (MOO) over the search space of classification models, data pre-processors, and pre-processing steps provided by aut autosk osklearn we incorporated optuna for MOO and Ray Tune for distributed computing. We repurposed the search space of autoo-kkearrs and created our own solution to enable multi-objectivve optimisation over the search space of the models and their hyperparameters provided by auto-sklearde optuna(Akiba et al. 2019), we added multi-objective optimiser on top of auto-skl sklearn.. To enable distributed com- putation of models and optimisation of their hyperparameters, we used ray tune along with optuna. The combination of these three already available solutions allowed us to perform multi-objective optimisation at scale, which was not possible in standalone aut auto-sklease multi-objective optimisation is not an official feature in the package. Another choice is FLAME Wang et al., 202 however, its documented moderate reliability (Gijsbers et al. 2024) finalised the choice of auto sklearn. The OpenML CC-18 dataset and its features are fed into the Bayesian or random optimiser, or both (depending on the experiment conducted). These optimisers are provided by Optuna. Depending the experiment, also feed the random forest baseline metrics. Optuna generates hyperparameter configurations and Ray Tune runs them in parallel for each feature. An illustration of the architecture is given in Figure 10
By using
Arguably, autosklee earn 2. O fixes these issues, however, at the time of this writing,
Feeturs
tune Paruded rora
Feature...
macagement
Ganannte
HP configuration
Trial 1
Optimiser
Catsset
Optimsser
Random
confiuration
festun
HP configuraton....
...
Band ne matris
HP configuratinn n
Trial n
Baod no malr
autoskleam contigs
autosMaam contg space
sOore
Figure 10: Architecture of our custom AutoML solution
Extracting the search space To create our custom AutoML solution, we extracted the full search space of autoskleat library. The classification and pre-processing algorithms are given in Table and Table 4] respectively. The library is built using Con figpace (Lindauer et al. 2019), which provides a simple function to extract the entire search space including classification algorithms, their hyperparameters, type of hyperparameter: categorical or continuous, and the conditions on them, any.
Optuna for multi-objective optimisation Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Optuna features multi-objective optimisation and defines the following basic concepts.
Study: optimization based on an objective function
Trial: single execution of the objective function
The goal of study is to find an optimal set of hyperparameter values through multiple trials.
We leverage the define-by-run style user API and recreate the extracted search space from auto-sklearn. This recreated search space can be used with optuna and the optimisation algorithms in the package, which support multiple objectives.
Ray Tune for distributed computing of trials Ray MMoriz et al. 2018) is an open source framework to build and scale machine learning applications. Tune (also called Ray Tune), a sub-package of Ray, is a Python library for experimental
17

---

X-Hacking: The Threat of Misguided AutoML
Table 3: Classification algorithms provided in auto- sklearn. The number of hyperparameters associated with the algorithm (A) are separated into categorical (cat) and continuous (cont). The numbers in the parenthesis are the conditional hyperparameters, which are relevant when other parameters are active.
Name
#\
cat (cond)
cont (cond)
AdaBoost (AB)
1
(-)
(-)
Bernoulli naive Bayes
2
1
(-)
1-)
decision tree (DT)
1 (()
3(-)
extreml. rand. trees Gaussian naive Bayes gradient boosting (GB)
5
2()
3()
6
6()
KNN
3
2(-)
5()
LDA
4
16)
10)
linear SVM
4
2(-)
3()
kernel SVM
7
2(
5(2)
multinomial naive Bayes
2
1(-
passive aggressive
3
16+
2()
QDA
4
1(4
1()
random forest (RF)
5
2(-)
3()
Linear Class. (SGD)
10
4()
63)
execution and hyperparameter tuning at any scale. We use Ray Tune to enable distributed execution of Optuna trials, saving trials, metadata and maintaining trial states during and after the end of trial run.
C. Weighted-sum scalarisation approach to multi-objective optimisation for X-hacking
The typical target of an automated machine learning pipeline is model classification performance metric such as confusion matrix, derived statistics such accuracy, Fi score and area under the receiver operating characteristic (ROC) curve. At the same time, we wish to optimize our desired model explanation, which could be described through p-value or coefficient
value, feature importance, effect size (e.g. Cohen's d), Shapley value or fairness metric (Agrawal et al. 2021 $ 4). The task is therefore a multi-objective optimization problem. How much predictive performance must be sacrificed, on average, to get the explanation desired? To explore the tradc-off between accuracy (or AUC, Fi, Brier score etc.) against the chosen XAI metric (SHAP, p-value, Cohen's 8), we propose scalarized scoring function, Q, based on weighted sum of lying (2). performance and obviousness
Let (m) denote the predictive performance of model pipeline and let obv(m) denote some quantified measure of obviousness, audacity or inadmissibility. Let m4 arg max,, {perf(m)) be the best best-performing pipeline and m denote an acceptable baseline mode Let v denote the feature explained by an XAI metric (m))), such as SHAP. Then we maximize the scalar objective function
Q = - sgn (Xmt(v)) Xm)
(2)
+ (m)f(m) p oby(m))
(3)
where and j are parameters that define a Pareto front possible solutions. Such an objective may be passed to any AutoML framework that accepts custom scoring metrics.
As XAI and predictive performance metrics lie on different scales, efficient selection of À is crucial. Suppose that the "best' model is anticipated to have accuracy greater than 0.9, while baseline model achieves accuracy of 0.7. If baseline or prior SHAP is +1, and the aim is to find model that 'flips' or nullifies this, then pragmatic starting point is < (perf(m+) (erf(mm))) X) (0.9 - 0.7)/1 = 0.2, so that larger-scale changes in SHAP 'drive" the optimizer.
"mi= : arg min,, [perf(m)) for acceptable models m E M
18

---

X-Hacking: The Threat of Misguided AutoML
Table 4: Pre-processing algorithm provided in auto- sklearn. The number of hyperparameters associated with the algorithm (A) are separated into categorical (cat) and continuous(cont), The numbers the parenthesis are the conditional hyperparameters, which are relevant when other parameters are active.
Name
#\
cat (cond)
cont (cond)
extreml. rand. trees prepr.
2 (-)
(-)
fast ICA
(-)
1(1
feature agglomeration
4
(-)
1(-
kernel PCA
5
1()
4
rand. kitchen sinks
2
4
linear SVM prepr.
3
14)
2()
no preprocessing nystroem sampler
4
5
10
4(3)
PCA
2
10
10
polynomial
3
20
1()
random trees embed.
4
2(
10
select percentile
2
10
select rates
3
2
one-hot encoding
2
10)
1
imputation
1
14
balancing
1
1()
rescaling
I
1()
D. Details on experimental setup and results
This section covers the implementation details and results from post-hoc and ad-hoc strategies for X- X-hacking. For each strategy used to perform X-hacking we first list the datasets used, give details on the resource and implementation and finally give the results for all the datasets used to perform X-hacking.
D.1. Experimental setup and results for post-hoc X-hacking (Cherry-picking.
This section covers the implementation details and results from the experiments for all the datasets for post-hoc X-hacking. Following listing the datasets used and implementations details we show the graphs for changes in feature importance relative to baseline for all the datasets, lines best fit SIAAP for all the features for the datasets, and marginal distribution of SHAP slopes over AuvaMt-evaluated models for all the features of all the datasets.
D.1.1. DATASETS
For the current experiment, have 23 datasets from OpenML Benchmark CC-18 (Bischl al., (021). The current experiment for explainability of the algorithms are carried out on datasets with a binary target. The datasets are listed in T
D.1.2. RESOURCE AND IMPLEMENTATION DETAILS
For of the calculations, we stick to parallel computation using CPUs. For our experiments we have used at most 192 CPUs in parallel on an institutional computing cluster. Since the datasets are structured, maximum of 300 GB of RAM used for the experiments. Since computation time for SHAP calculations increase samples, parallelism become imperative. We restricted the number of test samples to 100 to calculate SHAP for all the datasets for resource management reasons.
increases the number of test
The implementation done in Python programming language. We used pandas and numpy libraries for data wrangling, Seikit-leati as our base ML library, sklearn for automated finding models, opt una for multi-objective optimisation, and shap for calculating shap values.
19

---

X-Hacking:
The Threat of Misguided AutoML
Table 5: Binary classification datasets from OpenML CC-18 Benchmark suite.
OpenML ID
Dataset
#Features
breast-w
29
credit-approval
15
31
credit-g
20
37
diabetes
8
1049
pe4
37
1050
pe3
37
1053
jml
21
1063
kc2
21
1067
kcl
21
1068
pel
21
1461
bank-marketing
16
1464
blood-transfusion
4
1480
ilpd
10
1485
madelon
500
1494
qsar-biodeg
41
1510
wdbc
30
1590
adult
14
4134
Bioresponse
1776
23517
numerai28.6
21
40701
churn
20
40983
wilt
5
40994
climate-model
45547
cardiac-disease
data split. for training all the models, baseline and AutoML, we used 20% of the samples as test dataset for all of the datasets mentioned in 5
preprocessing for all of the datasets, the samples where any feature had missing (NaN) values were removed. The indices of the omitted data are saved for later results.
random seed for reproducibility of the results, a random seed of 42 is used everywhere.
baseline the baseline model is the default sklearn. ensemble Rew.dooooeeeeeeeesssssssiieeeee. with the mentioned random state.
AutoML
for each dataset, we run auto-skleas for 3600 seconds in total and run time limit of 100 seconds for each candidate model with the mentioned random seed. An ensemble size of 1 is used since ensemble models are out of the scope for the current experiment. A model that takes more than 100 seconds to train is omitted.
explainer due to its model agnostic behaviour, we use shap Kerne ainelainer for calculating the SHAP values. A background sample of 50 and test sample of 100 samples from the test split used. The respective indices of the 100 samples are saved for regression analysis discussed later.
1.1.3. RESULTS
In this sub-section show all the graphs for the experiment conducted all of the datasets. Due to large number of features in the dataset Bi Bioresponse we do not include the graphs for them here. However, they are available in the GitHub repository Similarly, since there are too many visualizations for lines of best fits and kernel density graphs for fitted slopes for SHAP values, we include few examples here and the rest are available on GitHub
Changes in feature importance relative to baseline feature e for an AutoML. model is calculated as:
The changes in feature importance relative to the baseline for a
20

---

X-Hakking: The Threat of Misguided AutoML
SHAP,
haseline
SHAP,
antoml
y MV
(5)
M
Σ
SHIAP
leaseline
Σ
SHAP,
lautoml
where V is the set of all features for a dataset. This difference gives us the relative change in explainability from the baseline model for each feature in all the datasets. This can visualised through violin plots for all the datasets. The violin plots are given below.
0925
0.0023
Nuntabed moant3se)))
0015
0030
0.005
-00250
-0.020
-00075
-0015
Teaturn
Teeture
Figure 11: Relative change for dataset breast -w
Figure 12: Relative change for dataset -redittapproaal
Nurnalee moonn(3ree)) (r60))
Normltte SHOR) (H400)
0000
6.00
-0005
-.02
-0.04
-0015
-0.00
Figure 13: Relative change for dataset credit
Figure 14: Relative change for dataset diabetes
21

---

X-Hacking: The Threat of Misguided AutoML
Rirratzdd moantsnatt
0.055
3.005
-0.030-
2 45
5250500 300438888
Feature
Festure
Figure 15: Relative change for dataset pc4
Figure 16: Relative change for dataset pc
0.0100
0.000
Rormatve (H04)
0.0020
0.005
0.0925
0.000
-200035
-0.0000
Figure 17: Relative change for dataset jml
Figure 18: Relative change for dataset kc2
0010
konmattzed masn((ssaa))
0025
6.0
0000
0.055
-0.00
-0.56
12
Peature
hadure
Figure 19: Relative change for dataset kel
Figure 20: Relative change for dataset pc1
22

---

X-Hacking: The Threat of Misguided AutoML
0.023
Nerma miwnnj/soo)
0.0019
0.0050
000
-050
0,920
-20025
hasture
reature
Figure 22:
Relative change for dataset
Figure 21: Relative change for dataset bank-marketing blood-trans
110075
0.92
0.004
Normatso (oant)5647)
0.002
0,000
0.00
0.01
Figure 23: Relative change for dataset ilpd
Figure 24: Relative change for dataset madel on
0.006
000100
0.0055
0.044
Nomalied PHANCWHA)
0.002
0.0025
0.000
0.0000
-3.0075
-3.0100
123
1011 1214 19202) 22200222223322227272000
Feahura
Teeture
Figure 25: Relative change for dataset gsar- bbodeg
Figure 26: Relative change for dataset wabcc
23

---

X-Hacking: The Threat of Misguided AutoML
0.100
Netratzdd monntsmart
0.004
0.002
0.000
0000
-0.002
14
10
Feature
Festure
Figure 27: Relative change for dataset adult
Figure 28: Relative change for dataset cardiac-disease
0.015
0.0120
0.0075
0.000
Rormalve (H004)
0.0010
0.005
Rommittdd
0.0020
0.000
-20025
-20030
-0.0035
Figure 29; Relative change for dataset numerai28. 6
Figure 30: Relative change for dataset churn
0.0
0.0033
hormartee menn(p2oon)
0.54
0.050
020
0.00
-0.0005
-0.0500
Pealure
Trature
Figure 31: Relative change for dataset wi. 1.t
Figure 32: Relative change for dataset cl imate-model
24

---

for the baseline as well X-Hacking: The Threat of Misguided AutoML
Lines of best fit of SHAP To calculate the lines of best fit for SHAP for all features of dataset, the subset of the test split which the SHAP values regression model is fitted on this data using sklearn. wwtneerrroodeee Beerodee.Linerafee LinearRegressii as all the models that have accuracy at least as good as baseline found by running AutoML on the dataset. Since displaying all the graphs not feasible, we provide with a few examples here. The lines of best for the rest of the features for all of the datasets can be seen in the GitHub repository
calculated taken the data with the corresponding SHAP values as the target. A linear
0.94
Baseline
Best medel cither
Best model other
0.02
0.006
-0.02
0.002
-0.08
0.08
0.000
1.5
1.0
1.5
Clump_Thiee Thickness
A1l
Figure Lines of best fit of SHAP for feature Clump Figure 34: Lines of best fit SHAP for feature A1 in dataset Thickness in dataset breast-w
credit-approne
Beseline
Baseline
0.04
Rest model Other
Bect model
0.000
0.00
-0.02
0.002
0.003
-2.0
-15
-1.0
3.0
prcs
DESIGN DENSITY
Figure 35: Lines of best fit of SHAP for feature pres in Figure 36: Lines best fit of SHAP for feature Design dataset diabetes
Density in dataset pc4
25

---

X-Hacking: The Threat of Misguided Auto.L
0.004
Bascline
0.014
Baseline
Best medel
Best model
0.005
0.012
0.020
0.008
0.000
0.90
-0.901
0
-1.0
-0.5
3.0
0.5
1.0
1.5
2.0
2.5
3.3
ESSENTIAL ESSENTAI_CCMPLERR COMPLEXITY
Figure 37: Lines of best fit SHAP for feature Essentia a Figure 38: Lines of best fit SHAP for feature in dataset Complexity in dataset pc3
jm1
Baseline
0.06
0.008
Best medal
Bost model
Ltthe
other
0.04
0.002
-0.94
-0.002
0.06
InsoodeAndCommmer
Figure Lines of best fit of SHAP for feature Figure 40: Lines of best fit SHAP for feature V1 in dataset LocCodeandComment in dataset kel
blood-traassusiin
Raseline:
0.010
Baseline model
0.02
0.004
0.01
0.000
0.0
-0 00.002
-001
-0.00 -0.004
V25
Figure 41: Lines of best fit of SHAP for feature V6 in dataset Figure 42: Lines of best fit of SHAP for feature V155 in dataset ilpd
wdbc
26

---

X-Hacking: The Threat of Misguided Auto.L
Best model Other
0.02
Best model
Other
0.01
0.002
0.00
-0,002
-0.44
tightt minutes
Meam
Figure 43: Lines of best fit of SHAP for feature Figure 44: Lines best fit of SHAP for feature Mean NIR cotal_night mrchhttmmmmin minutes in dataset churn
in dataset wi 1t
Usseline
0.000
Other
Other
-0.002
-0.003
0.004
0.003
0.005
0.004
0.006
005
Iidal mix__maa max
hoight
Figure 45: Lines of best fit of SHAP for feature Figure 46: Lines of best fit of SHAP for feature height in tidal mi x max in dataset climate-mode dataset cardiac-disean
27

---

X-Hacking: The Threat of Misguided AutoML
D. 1.4. MARGINAL DISTRIBUTION OF SHAP SLOPES OVER AUTOML-EVALUATED MODELS
Here show the marginal distribution of slopes over all Auto-ML models that have the accuracy least good the baseline model. The graphs correspond to the ones displayed in the previous sub-section.
baseline
baseline best
300
best
Densiy
400
700
100
200
-0.04
-0.03
-0.02
-0.01
0.00
-0.002
-0.001
0,000
0.001
0.002
0.003
Figure 47: Marginal distribution of SHAP slopes for feature Figure 48: Marginal distribution of SHAP slopes for feature Cl Clump_Thiee Tww.Thiicknnn ckness dataset breast-w
A11 in dataset credi Meedit-ae appproval
baseline
baseline
Density
1000
best
0. 3.0050-0.0002 0.0000 0.0025 0.0050
0.0075
0.0100
0000000000001 0.8015-0.0010-00000 0.0000 0.0005 0.0010 0.0015
Figure 49: Marginal distribution of SHAP slopes for feature Figure 50: Marginal distribution of SHAP slopes for feature pres in dataset di abetes
Design Density in dataset pc4
2000
baseline
400
beseline
Density
Demsity
1000
200
500
0
-0.0008-0.000 0.000--0.00--0000 00000000000000000000000000000000 10000-0.00020.000
0.0002 0.0004 0.0006 0.0008
-0.002
0.000
0.002
0.004
0.006
Figure 51: Marginal distribution of SHAP slopes for feature Figure 52: Marginal distribution of SHAP slopes for feature Essential Complexity in dataset pc3
in dataset jm1
28

---

X-Hacking: The Threat of Misguided Auto
1000
baseline best
best
Denaiy
750
Density
100
500
10.0010-0.000
0.0000 0.0005 0.0010 0.0015
0.0020
0.01
0.02
0.03
0.04
Figure 53: Marginal distribution of SHAP slopes for feature Figure 54: Marginal distribution of SHAP slopes for feature LocCodeandConment in dataset kcl
v1 in dataset blood-trans
baseling
200
best
best
Densiyy
150
1000
100
500
50
-0.002
0.000
0.002
0.004
0.006
0.008
-0.0005
0.0000
0.0005
0.0010
0.0015
0.0020
0.0025
Figure 55: Marginal distribution of SIIAP slopes for feature Figure 56: Marginal distribution of SIAAP slopes for feature V6 in dataset ilpd
V15 in dataset wdbc
baseline
1000
best
best
Denaily
500
0.0000
0.0005
0.000
0.002
0.006
0.005
0.012
0.014
Figure 57: Marginal distribution of SHAP slopes for feature Figure 58: Marginal distribution of SHAP slopes for feature cotal-night mi nutes in dataset churn
Mean_NIR in dataset wilt
2000
baseline
400
best
1500
best
Density
Density
1000
200
0.002
-0.001
0.000
0.001
0.002
0.003
10..00000..000-000000-000000 0.0000 0.0002 0.0004
Figure 59: Marginal distribution of SHAP slopes for feature Figure 60: Marginal distribution of SHAP slopes for feature ridal mix max in dataset climate e-mdmeeeemddll
height in dataset cardiac-dises
29

---

X-Hacking: The Threat of Misguided AutoML
D.2. Experimental setup and results for ad-hoc X-hacking (Directed search)
This section covers the implementation details and results from the experiments for all the datasets for ad-hoc X-hacking. Following listing the datasets used and implementations details, show the graphs for cumulative minimum of mean absolute SHAP for Bayesian optimisation and random sampling. These graphs shown for top-4 features from each dataset according to the random forest baseline.
D.3. Dataset
The datasets are the same given in D.1.1
D.4. Resource and implementation details
For all of the calculations, stick to parallel computation using CPUs. For ad-hoc X-hacking have used most 292 CPUs in parallel on institutional computing cluster. To many candidate models for each dataset, maximum of 1 TB of RAM was used for the experiments. Since computation time for SHAP calculations increase as one increases the number of test samples, parallelism become imperative. We restricted the number of test samples to 100 to calculate SHAP for all the datasets, for resource management reasons.
The implementation done Python programming language. We used pandas and numpy libraries for data wrangling, scikit -earn as our base ML library, auto sklearn for its search space including models and its hyperparameters. ConfigSpace to extract the search space from auto-sklearn, optuna for enabling multi-objective optimisation, ray tune for parallel running of different models and intermediate storage of experiment related metrics, and shap for calculating shap values.
data split for training all the models, baseline and models from our custom AutoMI inlummmnimllm we used 20% the samples as test dataset all the datasets mentioned Table
preprocessing for all the datasets, the samples where any feature had missing (NaN) values were removed. The indices of the omitted data are saved for later results.
random seed for reproducibility of the results, random seed of 42 is used everywhere.
baseline the baseline model is the default sklearn. ensemble Raw.ooooooeetcaassiiieeeeeeeeeee aaw.eaaeeeeeeeoadoorroereeetttee random state.
with the mentioned
AutoML for each dataset, we run our custom AutoML solution described in $ B 12 hours (43200 seconds) in total and run time limit of 1 hour (3600 seconds) for each candidate model with the mentioned random seed. We the custom AutoML solution with both Bayesian optimisation and random sampling
explainer due its model agnostic behaviour, background sample of 50 and test sample of 100 samples from the test split is used. The respective indices of the 100 samples are saved for regression analysis discussed later.
shap Kerne 1 1Exp Explainer ainer for calculating the SHAP values. A
D.4.1. RESULTS
In this sub-section we show using graphs for cumulative minimum of mean absolute SHAP Bayesian optimisation and random sampling for all the datasets. It is expected that some features are vulnerable to X-hacking, and thus for those features do not see the data in the p
Fmmpty graphs represent features being robust X-hacking. Marked with
30

---

X-Hacking: The Threat of Misguided Auto.L
Bayasien Rendom
Bayesien Random
Clump_fhisknng (hickness
Chromatin
Uniformity
Mitoses
Time (h)
Time (h)
Figure 61: Cumulative minimum of mean absolute SHAP for Figure 62: Cumulative minimum of mean absolute SHAP for top features dataset breast- -w for Bayesian optimisa- top 4 features of dataset credit aredit-appros -aprroval for Bayesian tion and random sampling
optimisation and random sampling
Bayasian
Bayesien Rtaddm
Random
purpose
Insu
credit_history
10
12
Time (h)
Time (h)
Figure 63: Cumulative minimum of mean absolute SHAP for Figure 64: Cumulative minimum of mean absolute SHAP for top features dataset credit-g for Bayesian optimisa-top 4 features of dataset diabetes for Bayesian optimisa- tion and random sampling
tion and random sampling
31

---

X-Hacking: The Threat of Misguided Auto.L
Bayesian Rendom
Bayesian Random
PARAMETER, COUNT
NUM OPERATORS
AND
ESSENTIAL
HALSTEAD_EEROOK EEROO_ESTR
HALSTEAD
Time (h)
Time (h)
Figure 65: Cumulative minimum of mean absolute SHAP for Figure 66: Cumulative minimum of mean absolute SHAP for top features dataset po4 for Bayesian optimisation and top 4 features of dataset po3 for Bayesian optimisation and random sampling
random sampling
Bayas Random
Bayesien Raadom
10Blakk
branchCount
Rotal_Opnd
6.
Time (h)
Time (h)
Figure 67: Cumulative minimum of mean absolute SHAP for Figure 68: Cumulative minimum of mean absolute SHAP for top features dataset jm1 for Bayesian optimisation and top 4 features of dataset kc2 for Bayesian optimisation random sampling*
random sampling
32

---

X-Hacking: The Threat of Misguided Auto.L
Bayesien Random
Bayeslen Random
10Blonk
totalOpp
Ioc
108lank
0.
Time (h)
Time (h)
Figure 69: Cumulative minimum of mean absolute SHAP for Figure 70: Cumulative minimum of mean absolute SHAP for top features dataset kel for Bayesian optimisation and top 4 features of dataset pol for Bayesian optimisation and random sampling
random sampling
Bayesien
Bayesien Random
Time (h)
Time (h)
Cumulative minimum of mean absolute SHAP top features of dataset bank ma markeiiing for Bayesian for top features of dataset blood- trans fusion for optimisation and random sampling*
Figure 71: Cumulative minimum of mean absolute SHAP for Figure 72:
Bayesian optimisation and random sampling
33

---

X-Hacking: The Threat of Misguided Auto.L
Bayasien Random
Bayesian Ranzor
VS
V389
V243
V401
Time (h)
Time (h)
Figure 73: Cumulative minimum of mean absolute SHAP for Figure 74: Cumulative minimum of mean absolute SHAP for top 4 features of dataset iipd for Bayesian optimisation and top features dataset madelon on for Bayesian optimisation and random sampling
random sampling
Bayaslan Random
Bayesien Raadom
VM4
V20
10
Time (h)
Time (h)
Figure 75: Cumulative minimum of mean absolute SHAP Figure 76: Cumulative minimum of mean absolute SHAP for for top features of dataset qsar beaarbbiodeg for Bayesian top features dataset wdbo for Bayesian optimisation and optimisation and random sampling
random sampling
34

---

X-Hacking: The Threat of Misguided Auto.L
Bayesien Random
Bayesien Random
ducation num
4p
Inlwgt
gender
Time (h)
Time (h)
Figure 77: Cumulative minimum of mean absolute SHAP for Figure 78: Cumulative minimum of mean absolute SHAP for top features of dataset adult for Bayesian optimisation top 4 features of dataset cardiac-diseast dessese for Bayesian and random sampling"
optimisation and random sampling
Bayesian
Bayesian Random
attribute__
phone_ number phane_number
Stribute_
Accoun_leengt
attribute_15
10
Time (h)
Time (h)
Figure 79: Cumulative minimum of mean absolute SHAP Figure 80: Cumulative minimum of mean absolute SHAP for for top features of dataset numerai28.. for Bayesian top 4 features of dataset churn for Bayesian optimisation optimisation and random sampling
and random sampling
35

---

X-Hacking: The Threat of Misguided Auto.L
Bayasien Random
Dayasien Randem
Mean_R
tidal_mi__ma
Plan
officiency,
10
Time (h)
Time (h)
Figure 81: Cumulative minimum of mean absolute SHAP for Figure 82: Cumulative minimum of mean absolute SHAP top 4 features of dataset wilt for Bayesian optimisation and for top 4 features of dataset cl Elimate e-model for Bayesian random sampling
optimisation and random sampling
Bayeslan Random
D99
D1324
D1091
0
2
10
12
Time (h)
Figure 83: Cumulative minimum of mean absolute SHAP for top features of dataset Rioresponse for Bayesian optimisation and random sampling*
36

---

X-Hacking: The Threat of Misguided AutoML
E. Exploring Shap Value Sensitivity to Noise and Information Sharing
In the first scenario, we examine a case featuring two identical features with no inherent independent noise. In this context, we observe no discernible shift in accuracy across different variables. This outcome can be attributed to the intrinsic additive nature of SHAP values, whereby the transfer of SHAP values from one feature to another occurs seamlessly without any resultant loss of accuracy. This is principally due the absence of information loss.
The second experimental model introduces lower noise-to-signal ratio, distinguishing between noise (independent information associated solely with a specific feature) and signal (information shared between both variables). When we transfer SHAP values from one variable to another within this context, we encounter a scenario where the transference fails to capture identical information. Consequently, accuracy experiences a decline, albeit not of significant magnitude.
In the third experiment, we explore high noise-to-signal ratio, leading noteworthy decrease in accuracy when manipulating SHAP values across different features. This emphasizes the sensitivity of SHAP values to the presence of substantial noise within the data.
These experiments are strategically designed to demonstrate the circumstances under which it is feasible to manipulate SHAP values for a given dataset without incurring substantial accuracy loss. It becomes easier to reduce or alter SHAP values for a specific feature when the information associated with that feature is shared among other variables.
In the context of an automated machine learning (AutoML) pipeline, different preprocessing steps and algorithms may yield models with comparable accuracy but varying feature importance, as indicated by SHAP values. The likelihood of encountering such models is contingent upon the search space of available models and the non-linearity inherent in the relationships among features or the underlying data generation process. For instance, when the relationship is profoundly non-linear, and the model's search space predominantly consists of linear models, these models are more prone to assigning higher SHAP importance to linear relationships within the data generation process.
the equations are
fo Uniform (0,5) 5)
fi 10 fo + +N(0, 11)
(6)
f2 20 fo +N 02)
fs ~ 3 fi + 4- f2 + N
The variation in the normal distribution's standard deviation can be harnessed as a mechanism for adjusting the level of independent information within dependent variables. By employing this approach, we can simulate diverse scenarios representing distinct degrees of independent information.
Our empirical findings clearly illustrate that as the magnitude of independent information in the dependent variables is augmented, the corresponding reduction in predictive accuracy resulting from changes in SHAP (SHapley Additive exPlanations) values also experiences a corresponding increase. This relationship serves as compelling evidence of the critical role played by independent information in shaping the accuracy of predictive models, particularly in the context of SHAP value manipulation.
F. Leveraging Redundancy for Shapley Value Manipulation
Incorporating surplus of redundant variables into model can be a strategically advantageous approach, particularly when the objective is to reverse the prevailing trend exhibited by Shapley values associated with a specific feature. This concept hinges on the premise that each child variable linked to the feature of interest encapsulates not only the information inherent to that feature but also additional information contributed by the respective child variable. For the sake of simplification and rigorous analysis, one can consider the information stemming from child nodes as form of 'noise", while the data originating directly from the feature of interest, acting as the 'parent' variable, is regarded as the 'signal'. The overarching goal is to amplify the signal-to-noise ratio for the feature by strategically configuring different combinations of child variables.
In the ensuing illustrative examples, we expound upon this concept by showcasing the characteristic trends observed in Shapley values for distinct input values of specific features, achieved by fitting approximate linear regression models. A notable scenario arises when one child variable introduces an additional layer of information that follows a disparate trend
37

---

X-Hacking: The Threat of Misguided AutoML
compared another child variable linked to the same parent. In such instances, linear amalgamation of these variables serves the purpose of disentangling the signal from the noise. This amalgamation strategically leverages the opposing noise effects exhibited in different models, thereby neutralizing their individual impact. If the noise effects happen to be multiplicative in nature, log transforms may be judiciously employed before engaging in linear combinations to attain similar results.
foundational principle emerges from this exploration: the greater the number of independent child nodes associated with specific feature, the more susceptible that feature becomes to a phenomenon referred to as "flipping", where its trend contrasts with the true underlying trend. Redundancy in the information domain empowers to reconstruct the feature and construct a model that effectively showcases inverse trend for the given feature values. Notably, this transformation can be accomplished while maintaining equivalent level of predictive accuracy, as exemplified in the earlier instances illustrating the Shapley value versus accuracy Pareto front.
38


Reviews:
[{"review_id": "MokTVI20hy", "reviewer": "Reviewer_9ZtB", "summary": "The paper introduces X-hacking that manipulates XAI metrics by exploiting model multiplicity to find explanations supporting desired conclusions. Bayesian optimization helps to find models that support a desired explanation while maintaining acceptable accuracy, allowing for manipulation of SHAP values.  Datasets with high feature redundancy are particularly vulnerable to X-hacking. The paper also discusses detection and prevention strategies, highlighting ethical concerns and the risks X-hacking poses.\n\n## update after rebuttal\nThank you to the authors for their reply and willingness to further improve the manuscript. I would like to encourage the authors to incorporate a discussion on ethical issues related to the intent of model selection under multiplicity. While I still believe that the paper would greatly benefit from additional results based on non-feature-attribution explainability methods, I acknowledge that the empirical results for AutoML pipelines are extensive. I have re-read the paper and the authors’ reply and have adjusted my scores accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and valuable references.  \n\nWe agree that the phenomenon of having many equally accurate, yet interpretively distinct, models is well established, and we cite several related papers (e.g., Fisher et al., 2019; Brunet et al., 2022) to acknowledge this. However, our paper contributes a new perspective by explicitly demonstrating how modern AutoML pipelines can be adapted (or misused) to exploit those known multiplicities at scale to manipulate XAI metrics (referred to as “X-hacking”). While multiplicity has certainly been studied, we highlight in the related work section how this is limited to certain model families. Our approach highlights automation and practicality—showing that even modest budgets given to AutoML systems can allow for systematically “cherry-picking” explanations to support a predefined narrative, all while preserving acceptable predictive performance. This practical demonstration of how easily multiplicity can be exploited via off-the-shelf or lightly customized AutoML solutions distinguishes our work from purely theoretical or smaller-scale explorations of Rashomon sets. We acknowledge the recent preprint, Ganesh (2025) and add a reference to this in Section 2 (Background).  \n\n\n“**More nuanced view on X-hacking is needed, as it can be used as an assistive tool to align the model with domain experts’ intuition.**”: We acknowledge that choosing models that align with intuitions of domain experts may not always raise ethical concerns. For example, when domain experts prefer interpretable patterns that reflect established science or policy. However, the distinction lies in intent and transparency. We appreciate this insight and intend to add in Section 7 (Discussion) that model selection for domain alignment may not always attribute to malpractice provided it is transparent and reported in good faith. While our paper emphasizes the opposite, an expanded discussion of both benign and malicious scenarios will enhance the paper’s balance.  \n\n\n“**Only one XAI method is used … it is not clear how X-hacking behaves for other XAI methods.**”: We worked with SHAP for two reasons: it is a model agnostic approach, and it is a popular metric, making it a strong representative for our demonstration. We show that quantitative XAI metrics like SHAP can be easily manipulated using off-the-shelf AutoML solutions, thus sufficiently establishing the core risk of X-hacking at scale. We believe that any post-hoc explanation method -- especially those susceptible to model multiplicity -- will face similar risks, however, systematic exploration of other XAI metrics is planned as future work. \n\n\n“**The methods can be time-consuming and might be hard to generalize to more complex models.**”: We share the concern that scanning a massive analysis space can be computationally expensive, however, we show that current off-the-shelf AutoML solutions and their search spaces can be used to find “defensible” models even under modest resource budgets. For extremely large or complex models, the cost (computational resources and time) of repeated training might be higher, however, it is the choice of the researcher to allot higher budgets in the context of their research. Our current intention is to show that X-hacking can be easily and effectively done at scale, even with small budgets.", "claims_and_evidence": "Claim 1: Automated machine learning (AutoML) pipelines can be easily adapted to exploit model multiplicity at scale, making them vulnerable to X-hacking. \n\nClaim 2: Bayesian optimization accelerates X-hacking for features susceptible to it, compared to random sampling.\n\nClaim 3: The paper shows that some datasets are more vulnerable than the others to X-hacking (probably due to different level of multiplicity).", "methods_and_evaluation_criteria": "The methods presented in the paper effectively illustrate the concept of X-hacking and the role of AutoML. However, they can be time-consuming and likely will be hard to generalize to more complex models.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The paper's experimental designs is generally sound for demonstrating X-hacking.", "relation_to_broader_scientific_literature": "The core idea that model multiplicity allows selecting models with different feature importances (Shapley values) is not new. It's well known that different models in the Rashomon set can emphasize different features while achieving similar predictive accuracy (see Rudin Amazing Things Come From Having Many Good Models, 24). Therefore one can choose models based on these features. There are tools that visualize these models as well (Timbertrek, 22).", "essential_references_not_discussed": "The field has a lot of recent papers, and the authors did a great job referencing important papers. However, some literature on the Rashomon set is missing. See [Ganesh, The Curious Case of Arbitrariness in Machine Learning, 25] for reference.", "other_strengths_and_weaknesses": "**Strength**\nThe analysis of different features to understand their susceptibility to X-hacking is particularly interesting\n\n**Weaknesses**\n\nMore nuanced view on X-hacking is needed, as it can easily be used as an assistive tool to align model with the domain experts intuition. \n\nOnly one XAI methos is used, which is feature importance-based metric, so it is not clean how X-hacking behaves for other XAI methods.", "other_comments_or_suggestions": "NA", "questions_for_authors": "How does the paper results generalize to other explainability methods that are not feature importance based?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5TkhmyRk2p", "reviewer": "Reviewer_Se87", "summary": "The paper notes that AutoML pipelines allow training of multiple ML models, including sets of models with 'defensible' performance.\n\nThus, AutoML tools make it easier for authors to cherry-pick models to fit preconceived notions, as embodied by explainability metrics.  The paper uses the Shapley value as its representative of these.\n\nExamples show that the possibilities for doing so increase in the redundancy of the feature set: when e.g. $x_1$ and $x_2$ are highly correlated, models may give weight to either without compromising their predictive power, allowing either to have a large Shapley value (the standard collinearity problem of econometrics/statistics).\n\n# update after rebuttal\n\nStands.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. The following addresses the reviewer’s questions and concerns. \n\n**p-hacking and X-hacking**: Both p-hacking and X-hacking can arise from scale, chance and collinearity, which are interrelated rather than distinct. For instance, collinearity can cause regression estimates to become unstable, thus exacerbating p-value non-robustness. X-hacking, facilitated by AutoML, leverages model multiplicity and feature redundancy to align with desired explanations, also affected by these factors. X-hacking can be viewed as a generalization of p-hacking, extending the manipulation of statistical significance to the broader manipulation of model explanations; however the focus in our paper is on model-agnostic explanations, not including p-values, which tend to be considered only for a subset of models such as GLMs. Both phenomena can also occur unintentionally, highlighting the need for robust statistical practices and transparent reporting to mitigate these risks. \n\n**TimberTrek**: As pointed out by Reviewer 9ZtB also, in a revision we will cite TimberTrek in Section 2 as an important work on visualising and understanding Rashomon sets.  \n\n**Rashomon set and defensible models**:  The Rashomon set R is defined as the set of models that exhibit nearly equivalent predictive performance within an acceptable threshold $\\epsilon$. \n\nR = {$\\{ m \\in M | perf(m) \\geq perf(m^*) - \\epsilon \\}$}\n\nGiven M as the set of all possible models, perf(m) as the predictive performance of model m, and m∗ as the best-performing model, the set of \"defensible\" models is equivalent to the Rashomon set R. This set allows for a performance decrease ϵ(m) that may vary by model, accommodating those considered 'acceptable' due to their adherence to standard practices in the domain of application/publication. Therefore, the set of defensible models is a Rashomon set: R(t, D) in the paper (section 4.1). Additionally, the set C(t, D) ∩ R(t, D) is the set of those defensible models that changed the explanations. We will make it clearer and keep notation for the set C(t, D) ∩ R(t, D) (Pg. 2 para. 1) where we mention finding defensible models which are the models from the set C(t, D) ∩ R(t, D), i.e., “defensible models that changed explanations”, to avoid confusion. \n\n**Figure 2 and Table 1**: Figure 2 and Table 1 measure different aspects of the same set, C(t, D) ∩ R(t, D) defined in section 4.1. In figure 2, we see how many (the size) models in this set are found in a post-hoc manner by an off-the-shelf AutoML solution demonstrating that—with no special adjustments—AutoML is susceptible to X-hacking. On the other hand, in Table 1 we see how quickly the first member of this set is found by ad-hoc X-hacking. For a dataset the set C(t, D) ∩ R(t, D) may be large, yet a member of this set appears relatively late in the search process (longer time in Table 1). For another dataset C(t, D) ∩ R(t, D) may be small, but yields one member quickly during directed X-hacking (shorter time in Table 1).\n\n**A motivating example**:  Our following empirical analysis can be added as motivation after background section. \nStudies have shown a strong relationship between gender and cardiac disease [[98](https://shorturl.at/qn04f), [99](https://shorturl.at/fTX5J), [100](https://shorturl.at/vP9fo), [101](https://shorturl.at/Y8a0Z)], but in an empirical experiment with our ad-hoc X-hacking, the importance of the feature gender was manipulated to drop to Rank 6 from a Rank of 1 in a baseline. The set C(t, D) ∩ R(t, D) had many candidates. First such candidate was found only in 24 seconds. \n[This figure shows the results](https://imgur.com/TcEvbK9).\n\n**Pre-registration and X-hacking**:  We mention study pre-registration in Section 2 under p-hacking. Pre-registration is a valuable safeguard against X-hacking but might require more rigorous methodological detail. We will add this in Discussion.\n\n**Concern regarding notation**: Obviousness and risk of getting caught are both represented by O. It is an arbitrary function in the current context. To avoid confusion with complexity notation, we will change O to another letter, say Z.  \n\nIn Section 3 para. 3, we mention Q as any quality measure, which can be performance of a model perf(m) or an inferential summary of feature of interest I(m, x). It is only used to define a quality measure and later says (Eq. 1) that generally one optimises for a quality measure of performance during a model search: Q = perf(m).  \n\n**Balanced view of X-hacking**: In Section 3 para. 2 we mention that such hacking can be “deliberate or not”. However, explicitly mentioning it earlier in Section 1 para. 2 can set a broader view at possible reasons behind X-hacking. Specifically, we will add the phrase “unscrupulously, or through lack of time or experience”.  \n\n**Editing “audacity” section**: To maintain the focus of the reader, the subsection 3.3 will be removed to supplementary materials.", "claims_and_evidence": "The claims are assessed on 23 datasets, as well as some simulated data with known ground truth.  This seems fine.", "methods_and_evaluation_criteria": "I have no concerns with the methods or evaluation criteria.", "theoretical_claims": "The paper presents no theoretical results, as such.", "experimental_designs_or_analyses": "I have not checked any code.", "relation_to_broader_scientific_literature": "The paper is motivated by reference to $p$-hacking, from the statistics literature.\n\nIn the light of that literature, the present results are unsurprising: larger models offer more possibilities for misrepresentation - whether intentional or otherwise.  (My favourite example remains Bennett et. al.'s fMRI analysis of Atlantic salmon, winning him an IgNobel.)\n\nI _think_ that slightly different issues arise: in the case of $p$-hacking, the problems arise from scale and 'chance'; in $X$-hacking, they arise from collinearity.  Is this correct?  If not, what is a more careful intuition?\n\nIt could also be useful to connect this to Wang, Rudin et al.'s 2022 TimberTrek, a visualisation tool for exploring Rashomon sets.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Overall, I found the paper's results unsurprising: more dimensions allow more misrepresentation.  This said, there needs to be a paper of record clearly establishing this.\n\nI would _like_ this to be that paper.  To do so, though, the story and exposition need to be much tighter.  For example, I would like to see:\n1. the set of 'defensible' models more tightly related to the Rashomon set: if they're the same, let's not duplicate language.\n1. a more careful consideration of what datasets are more/less vulnerable to $X$-hacking: how do you explain the ranking of datasets in Figure 2?  Why does the ranking differ from that in Table 1?\n1. it would be good to start the article with strong motivating examples 'in the wild' (p.8): at present, the concern is only plausible.", "other_comments_or_suggestions": "1. in econometrics, an increasingly common technique for addressing $p$-hacking is advance filing of research plans.  How well does this handle $X$-hacking concerns?\n1. the $\\mathcal{O}$ notation is used twice, once for 'obviousness' (p.3) and once for the 'risk of getting caught' (p.4).  Are these the same?  Overall, this does not make sense, as we do not have a model of cheating/getting caught.  The notation tends to be reserved for complexity measures, so is misleading.\n1. other notation also seems inconsistent/confusing: e.g. $Q_D(m) = perf(m)$ and $Q_D(m) = \\mathcal{I}(m)$.  I recommend picking one or the other convention.\n1. I don't think that the paper needs to be cast in terms of unscrupulous authors: authors under time pressure, inexperienced authors, etc. will all cherry pick, wittingly or otherwise.\n1. cut the 'audacity' section: you don't develop the idea; it feels like a distraction.\n1. a fun experiment to establish $X$-hacking risks could be to: generate a defensible set with e.g. each feature given the largest Shapley value; ask an LLM to write an abstract for a paper corresponding to each.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTTWTTPHA2", "reviewer": "Reviewer_2gSL", "summary": "In this work the authors propose the concept of “X-hacking,” where scientists or ML service providers leverage the multiplicity of ML systems to provide misleading explanations despite maintaining performance.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and supportive comments. Furthermore, we will incorporate your suggestions to further strengthen the exposition and clarify details around our adversarial concepts, notation, and experimental setups. Specifically:  \n\nWe agree that our discussions of adversarial objectives, risk tolerance, and detection were presented at a more conceptual level compared to the more rigorous empirical work. Our focus was to demonstrate and quantify the practicality of manipulating ML explanations, not to propose or experimentally validate a complete “risk model” of adversarial behaviour. In a revised version, we intend to shift parts of Section 6 (Detection and Prevention) to Section 7 (Discussion) to improve clarity.  \n\n“**… not sure why the authors heavily explored the benefits of Bayesian optimization over random optimization. I am more interested in the gains an adversary would get over the cherry-picking method — i.e., fewer runs to achieve a desired X-hacking effect.**”: We wanted to show that a malicious or benign user targeting explanations can exploit a more systematic search. We also wanted to show that given a greater time budget, random sampling will not on average perform better than Bayesian optimization in finding a defensible model. We explore this in an empirical manner and conclude that in our experiments, Bayesian optimization was 3 times faster at random sampling in the ad-hoc X-hacking setting (Section 5.4 (“Time to First Defensible Model”)).  \n\n“**… you describe the data as collinear. Was it perfectly dependent or just correlated? Also, in Figure 8, is each point a different model? It would help to clarify that each point is a distinct pipeline choice and that for a fixed performance we can shift feature dependencies as we like. In Fig. 3, are the features ordered by baseline importance? It should be mentioned explicitly.**”:  \n\n**Collinearity**: We used both moderate and high correlation scenarios, but not perfectly dependent features. Here we use “collinearity” referring to imperfect collinearity rather than perfect collinearity: i.e. the relationship is nearly but not exactly linear. We will clarify that the data is not perfectly dependent, but “redundant enough” to shift how SHAP values are allocated among correlated predictors without significantly affecting accuracy. \n\n**Figure 8**: Yes, each dot on the plot corresponds to a different trained model/pipeline. In the revision, we will explicitly state that “each point represents one pipeline configuration with its resulting performance and SHAP score for that feature.”  \n\n**Figure 3**: Yes, the features in Figure 3 are shown in descending order of mean absolute SHAP under our baseline model. We will add a sentence in the caption or text noting that “features are ordered based on their baseline SHAP importance,”", "claims_and_evidence": "The authors claim the potential for X-hacking in ML systems, especially AutoML pipelines where one can scalably provide candidate models with different explanation behavior. They provide thorough evidence for this case both in a post-hoc setting (generating many models with good accuracy, finding one with a desired explanation) and ad-hoc setting (jointly optimizing performance and explanation goals in an AutoML pipeline). Their empirical results tell a compelling narrative of the existence of manipulable explanations, potential ways they can manifest, and how this manipulation can be optimized for.\n\nHowever, multiple times throughout the paper the authors hypothesize about adversaries and countermeasures in an abstract manner. They say (paraphrasing) an adversary “could also consider risk of getting caught as an objective,” or that detection could occur through pipeline analysis without any experimental exploration of these ideas. I appreciate the discussion and understand the need to discuss all potential risks and countermeasures, but these claims felt very abstract and took away from the rigor found elsewhere in the paper. I might consider moving some of this discussion to the appendix to improve the clarity and focus of your work, but this is a minor comment and I still found their discussion interesting.", "methods_and_evaluation_criteria": "The authors use a Bayesian optimization and random sampler for the ad-hoc setting of X-hacking. The method makes sense for their joint prediction and X-hacking objective formulation.", "theoretical_claims": "The authors make no theoretical claims in the main body of the paper.", "experimental_designs_or_analyses": "The author’s experiments are sound, specifically both their cherry picking/post-hoc setups (5.1) and directed search setups (5.2) although I am not very familiar with the optimization method used. I thought the analysis was good although I am not sure why the authors so heavily explored the benefits of Bayesian optimization over random optimizing in 5.2. Intuitively, optimizing for these objectives should be better/faster . However, I am more interested in knowing the gains an adversary would get over the cherry-picking method – could they get a desired X-hacking effect with fewer model training runs?\n\nI had a question about the setup for 5.3 using simulated data. You include a dataset with correlated data so that you can shift SHAP weight around at will to manipulate explanations, correct? Why did you describe the features as colinear? Was the data perfectly dependent or just correlated? Also, in Figure 8, are each of the points in the graph a different model and the same feature? It should be clear that each point represents a choice of model, and so selecting points along a fixed horizontal line allows us to change feature dependencies in our explanation while maintaining performance.", "relation_to_broader_scientific_literature": "This work provides a great complement to literature on the Rashomon set and multiplicity by asking the opposite question of picking a model with a “bad” explanation. Additionally it builds nicely upon past work on the brittleness of explanations.", "essential_references_not_discussed": "The authors covered most important references in this work.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "In Fig. 3, are the features ordered by baseline importance? It might be good to mention that somewhere.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a2w6mcHBJ7", "reviewer": "Reviewer_vv4C", "summary": "The paper introduces the concept of X-Hacking, which refers to the practice of deliberately exploiting model multiplicity, where different ML models can have comparable performance, to select a model that has certain desirable explainability characteristics (like SHAP). The paper demonstrates the ability to select models based on explainable AI (XAI), discusses when this ability to select based on XAI is more feasible, and suggests some ways to make this selection more transparent.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their strong approval of the paper and the valuable feedback. Below we address the questions raised by the reviewer. \n\n“**Selecting a model because it uses some characteristics rather than others (as accurately represented by SHAP) is not inherently problematic...**”: \n\nChoosing a model x ∈ R(t, D) where R(t, D) is the Rashomon set because it meets certain ethical, policy or domain-driven requirements is not problematic. Even if x ∈ C(t, D) (models that change explanations), a confirmatory analysis is legitimate if it is reported transparently. By contrast, malicious X-hacking occurs when an actor withholds other valid models from R(t, D) and only presents a model from the set C(t, D) ∩ R(t, D) that supports a desired narrative, thereby concealing alternative explanations that might contradict the narrative.  Also pointed out by reviewer **Se87**, we appreciate this distinction and intend to put it earlier in the paper (perhaps Section 1) and then later discuss more in Section 7 (Discussion) \n\n“**Could you clarify what criteria or guidelines you envision for distinguishing harmful X-hacking from legitimate and beneficial model selection based on SHAP values?**”: Our vision is that effective countermeasures or additional requirements from a publication venue to make reporting more transparent can help identify or raise flags for X-hacking which may require further scrutiny. Specifically,  \n\n1. Transparent reporting of model choice, clearly mentioning that different models yield different explanations and clarification of why the chosen pipeline/model’s explanation is more appropriate.  \n\n2. Pre-specifying a research plan and adherence to it which can be later verified will limit the degrees of freedom that produced the reported model.  \n\n3. Justifying the choices of explanations in a selected model may help give valuable insights. \n\n4. A well documented research journey which is open, reproducible, and consistent with methodological standards will help identify X-hacking.  \n\n5. Awareness about model multiplicity and X-hacking and the need for effective countermeasures in the research community.  \n\nWhile the above points focus more on what researchers reviewers and peers can look for and raise concerns based on the non-availability of information that threatens transparency, a full-fledged automatic detector for X-hacking is an ideal vision for further development.", "claims_and_evidence": "The authors convincingly demonstrate that varying SHAP explanations can be obtained through either “cherry-picking” from multiple similarly performing models or via a targeted optimization approach (“directed-search”). They effectively show that this flexibility arises due to correlated or redundant features.\n \nHowever, the paper's analogy to p-hacking raises conceptual difficulties. Unlike p-values, which are fundamentally about statistical inference, SHAP explanations are inherently model-specific reflections of what each model genuinely uses in prediction. Selecting a model because it uses some characteristics rather than others (as accurately represented by SHAP) is not inherently problematic. For instance, if policy explicitly mandates avoiding certain protected characteristics (like criminal record), choosing an alternative model that does precisely that is compliance—not manipulation. SHAP explanations truthfully reflect model behavior and do not inherently mislead; thus, selecting among models based on desirable explanations isn't automatically problematic.\n\nA useful analogy is found in recent fairness literature, where “model multiplicity” is positively leveraged to select models with similar predictive power but improved fairness metrics. There, selecting among multiple models is explicitly beneficial, as it enables analysts to choose ethically or legally preferable options. This illustrates clearly that model selection itself—even based on explanation—can legitimately fulfill regulatory or ethical goals, provided it substantively addresses regulatory concerns.\n\nThus, the concern raised by the authors about “X-hacking” needs careful delineation: it is not simply choosing among multiple valid models that is problematic, but specifically doing so to create a misleading impression of objectivity or robustness when reporting explanations. This distinction warrants greater emphasis and clarification by the authors.", "methods_and_evaluation_criteria": "The proposed evaluation approach generally makes sense for illustrating the feasibility of “X-hacking.” The authors' choice of real-world datasets is appropriate, and their use of both off-the-shelf AutoML and a custom Bayesian optimization approach to systematically explore model multiplicity is reasonable and insightful.\n\nHowever, (to somewhat repeat the point made above) the evaluation criteria focus primarily on the speed and ease of finding models that yield desired SHAP explanations relative to predictive performance. A critical limitation is that the authors do not clearly justify why the selective search for particular SHAP values constitutes harmful “hacking,” rather than legitimate model selection. The evaluation would benefit from explicitly distinguishing harmful selective practices from legitimate uses of model multiplicity, particularly by providing clear normative criteria or guidelines for differentiating the two scenarios.", "theoretical_claims": "The paper is primarily an experimental paper and does not cover theoretical claims.", "experimental_designs_or_analyses": "The experiment design as reported in the paper is robust and convincing. I particularly appreciated the development of a custom AutoML solution for multi-objective optimization. The cherry-picking examples would be sufficient to conceptually demonstrate x-hacking but the directed-search example shows that this potentially can be done efficiently and at scale.", "relation_to_broader_scientific_literature": "The paper correctly situates itself within the model multiplicity and explainability literature. Although these themes have been explored extensively in the literature, the paper presents a novel and important approach and framework for considering the robustness of XAI in the context of searching for desirable explanations.", "essential_references_not_discussed": "I cannot think of specific citations that I would deem essential to have discussed. \n\nHowever, I believe that the paper would benefit from considering some of the recent literature on model multiplicity and fairness. While not requiring a specific citation, this literature would add some richness to the paper. Here are some examples- \n\nD-Hacking (https://dl.acm.org/doi/abs/10.1145/3630106.3658928)\nThe Legal Duty to Search for Less Discriminatory Algorithms (https://arxiv.org/abs/2406.06817)\nFundamental Limits in the Search for Less Discriminatory Algorithms—and How to Avoid Them (https://arxiv.org/abs/2412.18138) \nOperationalizing the Search for Less Discriminatory Alternatives in Fair Lending (https://dl.acm.org/doi/abs/10.1145/3630106.3658912)", "other_strengths_and_weaknesses": "Overall, I thought this was a great paper making a very important contribution to the literature. I can see this paper being very influential and opening up a future line of research.", "other_comments_or_suggestions": "see above", "questions_for_authors": "You convincingly demonstrate how easily AutoML tools can find models with desired SHAP explanations, but it's unclear precisely when selecting such a model becomes “manipulative” rather than beneficial or compliant (e.g., choosing models that intentionally exclude sensitive characteristics). Could you clarify what criteria or guidelines you envision for distinguishing harmful X-hacking from legitimate and beneficial model selection based on SHAP values?  A clear response would strengthen the practical relevance and conceptual precision of your paper, particularly for readers concerned with regulatory or ethical compliance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies the manipulation of XAI metrics by exploiting model multiplicity to find explanations that support specific conclusions.

The reviewers generally liked the setting of the paper and found the experimental design to be convincing and robust. In particular, providing cherry-picking examples is by itself sufficient to demonstrate x-hacking. However, the authors take the additional to show that by directed-search, the x-hacking can be done efficiently at scale. 

There were minor concerns about missed references and a focus only on SHAP values/feature importance metrics. I encourage the authors to cite the provided references and add a discussion about the generalizability of their results to other explainability techniques in the limitation/future work section.
Decision: Accept (poster)