TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Authors: Zi-Jian Cheng
Ziyi Jia
Zhi Zhou
Yu-Feng Li
Lan-Zhe Guo
Venue: ICML 2025 poster (2025)
Keywords: tabular data
feature shift
open environment
Primary area: deep_learning->robustness

Abstract: Tabular data is widely utilized in various machine learning tasks. Current tabular learning research predominantly focuses on closed environments, while in real-world applications, open environments are often encountered, where distribution and feature shifts occur, leading to significant degradation in model performance. Previous research has primarily concentrated on mitigating distribution shifts, whereas feature shifts, a distinctive and unexplored challenge of tabular data, have garnered limited attention. To this end, this paper conducts the first comprehensive study on feature shifts in tabular data and introduces the first **tab**ular **f**eature-**s**hift **bench**mark (TabFSBench). TabFSBench evaluates impacts of four distinct feature-shift scenarios on four tabular model categories across various datasets and assesses the performance of large language models (LLMs) and tabular LLMs in the tabular benchmark for the first time. Our study demonstrates three main observations: (1) most tabular models have the limited applicability in feature-shift scenarios; (2) the shifted feature set importance has a linear relationship with model performance degradation; (3) model performance in closed environments correlates with feature-shift performance.  Future research direction is also explored for each observation.

Benchmark: [LAMDASZ-ML/TabFSBench](https://github.com/LAMDASZ-ML/TabFSBench).

Tabular data (Altman & Krzywinski, 2017) represents a meticulously structured format of data, systematically ar- ranged in rows and columns, where each row signifies a Tabular Benchmark for Feature Shifts in Open Environments
TabFSBench:
Zi-Jian Cheng" 12
Zi-Yi Jia 12 Zhi Zhou
Yu-Feng Li Lan-Zhe Guo
Abstract
record and each column corresponds a feature (Sahakyan et al., 2021). The applications of tabular data in worldworld scenarios are extensive. For instance, tabular data is uti- lized in financial analyses, including credit scoring (West, 2000) and stock market predictions (Zhu et al., 2021). Fur- thermore, tabular data forms the foundation for various applications in the medical sector, encompassing disease di- agnosis (Yildız & Kalayci, 2024) and pharmaceutical devel- opment (Meijerink et al., 2020). Existing machine learning models applicable to tabular data can be divided into four categories, containing tree- based models (Prokhorenkova et al., 2018; ; Ke et al., 2017), deep-learning models (Holl- mann et al., 2023a; Ye et al., 2024), LLMs (Touvron et al., 2023; OpenAI, 2024), and specialized LLMs designed for tabular data (Hegselmann al., 2023; Wang al., 2023), namely tabular LLMs. These models have exhibited remark- able effectiveness across diverse tabular tasks.
Tabular data is widely utilized in various ma- chine learning tasks. Current tabular learning research predominantly focuses closed envi- ronments, while in real-world applications, open environments are often encountered, where distri- bution and feature shifts occur, leading to signifi- cant degradation in model performance. Previous research has primarily concentrated on mitigat- ing distribution shifts, whereas feature shifts, a distinctive and unexplored challenge of tabular data, have garnered limited attention. To this end, this paper conducts the first comprehensive study on feature shifts tabular data and intro- duces the first tabular feature-shiff benehmark (TabFSBench). TabFSBench evaluates impacts of four distinet feature-shift scenarios on four tabular model categories across various datasets and as- sesses the performance of large language models (LLMs) and tabular LLMs in the tabular bench- mark for the first time. Our study demonstrates three main observations: (1) most tabular models have the limited applicability in feature-shift sce- narios; (2) the shifted feature set importance has a linear relationship with model performance degra- dation; (3) model performance in closed environ- ments correlates with feature-shift performance. Future research directions and recommendations are also explored for each observation.
Most tabular models of the aforementioned four cate- gories are typically trained and tested in closed environ- ments (Zhou, 2022), where both the distribution of data and the space of features are maintained consistently. However, real-world tabular tasks frequently occur in open environ- ments (Parmar al., 2023), where significant challenges such as distribution and feature shift occur. For example, in traffic management system, alterations in traffic flow due to unforeseen incidents (e.g., extreme meteorological events) illustrate modifications in data distribution. Simultancously, the failure or substitution of specific traffic monitoring equip- ment can induce feature shifts. These dynamic alterations underscore the complexity and unpredictability associated with machine learning tasks in open environments, necessi- tating the development of models characterized by enhanced adaptability and robustness. Therefore, recent research has progressively concentrated on the advancement of tabular machine learning models designed to acclimate effectively in open environments.
Benchmark: LAMDASZ-MI/TABFSBan
1. Introduction
Distribution shift (Zhou et al., 2025) represents chal- lenge in open environments that has attracted substantial research attention. It refers to the phenomenon where tabu- lar data processed by a model during the training and testing phases exhibit significant differences in distribution (Wang et al., 2021a). Numerous research have proposed solu- tions to address distribution shifts, including domain adapta- tion (Ajakan et al., 2014; Arjovsky et al., 2019) and domain
'School of Intelligence Science and Technology, Nanjing Uni- versity, China "National Key Laboratory for Novel Software Tech- nology. Nanjing University, China "School of Artificial Intelli- gence, Nanjing University, China. Correspondence to: Yu-Feng Li niyf@@nj.eduu.cm Lan-Zhe Guo <guolz@nju.cduu.cm
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
a) Distribution Shifts
b) Feature Shifts
Training phase
Training phase
Age
Education
Occupation
Torget
Survool
Increment
52
HSS grad
Exee-managerial
50K
49
Rachelors
Craft-rqpair
50K
Mostes' samples training
Testing phase
Testing phase
b.l) Feature Increment
Ago
Education
Occupaion
Target
37
Masters
Exec manaaaraaaa
50K
42
Masters
Prof-specially
30K
Feature
samples
Figure 1. Open environment challenges: a) Distribution Shifts. Change in data distribution between training and testing while keeping features unchanged. b) Feature Shifts, Change in the feature set between training and testing, either by addition removal of features.
generalization (Zhou et al., 2022; Zhao et al., 2024). Vari- ous benchmarks on distribution shifts in tabular data have been established. For example, TableShift (Gardner et al., 2024b) focuses on distribution shifts in tabular classification tasks, while Wild-Tab (Kolesnikov, 2023) concentrates on distribution shifts in tabular regression tasks.
shift, and random shift, to evaluate models.
Implementations of Various Models and APIs. We evaluate four categories of tabular models and conduct the first assessment on LLMs and tabular LLMs in the tabular benchmark. We provide callable APIs to facilitate future research on the feature-shift challenge.
Feature shift, another prevalent and substantial challenge in open environments, denotes the phenomenon in which the set of features available to a model for the same task dynamically changes due to temporal evolution or spatial variations. For instance, in the weather forecasting task, critical sensors may cease functioning due malfunctions aging, they may be replaced by sensors due to technological upgrades, leading to significant alterations in accessible features. This shift not only disrupts the consis- tency of the model's input features but may also significantly degrade the model performance and robustness. However, research on feature shifts is relatively limited, and there is lack of high-quality benchmarks for feature-shift scenarios. Therefore, comprehensively investigating the challenge of feature shifts in tabular data is important for enhancing the performance and robustness of models on feature shifts in open environments scenarios.
Empirical Research and Analysis. We conduct ex- tensive experiments feature shifts and identify three observations alongside associated future work:
Most models have the limited applicability in fea- ture shifts, where tabular L.LMs demonstrate po- tential. Future work can design multi-level fine- tuning frameworks enhance tabular LLMs' rea- soning capabilities.
Shifted features' importance has linear trend with model performance degradation. Future work propose feature importance-drive -nrivnn op- timization algorithms to emphasize and protect strong-correlated features.
Model closed environment performance corre- lates with feature-shift performance. Solid theo- retical analysis of the relationship between close and open environments performance should be studied and how to further improve the robustness of these models open environments also an important research direction.
To this end, this paper presents the first systematic study on the challenge of feature shifts in open environments scenarios and proposes the first feature-shift benchmark for tabular data (TabFSBench). The contributions of this paper are summarized as follows:
2. Task and Notation
Tabular Benchmark for Feature Shifts. We provide publicly available datasets from classification and re- gression tasks across various domains. We design four feature-shift scenarios: single shift, rosllan
2.1. Task Setting
Formally, the goal tabular prediction tasks to train a machine-learning model f X' -> V. where X' is the input
2

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table I. Datasets TabFSBench. #Numerical means numerical features, #Categorical means categorical features.
Tasks
Dataset credit
#Samples
#Numerical
#Categorical
#Labels
1,000
7
13
2
electricity
45,312
8
0
Binary Classification
heart
918
6
5
2
MiniBooNE
72,998
50
0
2
Iris
150
4
0
3
penguins
345
4
2
3
Multi-Class Classification
eye movements
10,936
27
0
4
jannis
83,733
54
0
abalone
4,178
7
1
bike
10,886
6
3
Regression
concrete 1,031
8
0
laptop
1,275
8
14
space and y the output space. We define the set of features in X C.
Cteat and retains features from Cteat corresponding to the features in Otrai
2.2. Feature Shifts Notation
Feature Decrement. Certain existing sensors cease function and no new sensors added, leading reduction in the feature set (see Figure 1(b.2)), i.e., Crest 2 Ctrain In this case, to maintain consis- tency in input dimensions between training and testing phases and allow the model to predict properly, shifted features Crest need to be imputed.
partition the feature set C' from training and testing phases into Ctrain and Ctest In closed environments, the feature set from the training phase is identical to that from the testing phase, i.e., Ctrain Ctest While in open environments scenarios, although Ctrain remains invariant, Creest may encounter two challenges, namely, distribution shift and feature shift. We illustrate these challenges by using forest disease monitoring as an example, where a sensor can regarded as a feature. Figure depicts these challenges of open environments.
Given the limited research addressing feature shift and the established observation that feature increment generally does not result in model performance degradation, we con- duct an empirical experimental analysis focusing on feature- decrement scenarios.
Distribution Shift. When certain raw sensors fail, new sensors are installed and commence monitoring. The num- ber of features remains unchanged, i.e., Crain = Cusa, but the monitoring precision (data distribution) shifts. There- fore, distribution shift refers the phenomenon where data distribution varies between training and testing phases (see Figure (a)). Predictions can be generated without any data processing, although model performance may deteriorate due to covariate shifts or concept shifts (Shao et al., 2024).
3. TabFSBench: A Feature-Shift Benchmark for Tabular Data
TabFSBench is benchmark for evaluating feature shifts in tabular data, comprising twelve tabular tasks and assess- ing four categories of tabular models. We compare model performance across four feature-shift scenarios and closed environments. Additionally, provide callable Python APIs! to facilitate future research on the feature-shift chal- lenge in open environments.
Feature Shift. Feature shift is another open environments challenge, wherein the feature set previously utilized as inputs is either removed partially or added by new features. It contains two scenarios:
3.1. Datasets
To effectively reproduce feature-shift scenarios, we select open-source and reliable datasets from OpenML and Kag- gle's 'e's extensive dataset library, including three curated tasks of binary classification, multi-class classification, and regres- sion, covering various domains such as finance, healthcare and geology. The primary attributes of the datasets used in
Feature Increment. Additional sensors deployed and no existing sensor fails, resulting in expansion of the feature set (see Figure 1)b. i.c., Otrain Ç Ctest In this case, to maintain consistency in input dimensions between training and testing phases, the model typically truncates the newly added features in
http://githb..omm//MMAAAAAAAZ
3

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Random Shift. To systematically evaluate model robust- under feature shifts, we design controlled experiment where randomly sample feature subsets from the train- ing feature space Ctroin to create shifted test scenarios. By progressively removing features during testing (where re- moving one feature represents shift ratio of 1/n, with n being the total feature count), we quantify performance degradation across different shift magnitudes. For statistical reliability, evaluate up min 000, (*)) distinct fea- ture combinations per shift ratio n/N and report the mean performance across all combinations.
Table Kendall's r coefficients for four feature importance met-
rics consistency.
Metric
T
Pearson
0.60
Spearman 0.61
SHAP
0.49
Mutual Information
0.53
TabFSBench are presented in Table Detailed information on the datasets can be found in Appendix D.
3.3. Impute Strategy.
3.2. Feature-shift Scenarios
We compare the performance of various models by using their respective imputation methods, random imputation, and mean imputation. The results (see Appendix F) demon- strate that mean imputation better simulates scenarios with shifted features. Notably, benchmarks such as LAMDA- Talent (Liu et al., 2024) also adopt mean imputation for handling missing values, underscoring its widespread ap- plicability. Hence, opt for uniform mean-value imputa- tion of shifted features to ensure that predictions intuitively reflect model robustness in feature-shift scenarios. Specifi- cally, for numerical features, we employ the mean value of the feature within the training set as the imputed value. For categorical features, we utilize the value that occurs most frequently the imputed value. We do not select zero or other arbitrary values for imputation, as this would introduce artificial shifts in the data distribution, which contradicts the objective of evaluating model robustness in the context of feature shifts.
To evaluate the consistency of feature importance rankings, compute Kendall's 7 correlation coefficients among four metrics: Pearson correlation coefficient, Spearman's 's rank correlation, SHAP values, and mutual information. Table 2 reveals high concordance across these measures, with par- ticularly strong agreement between PCC and Spearman (r 0.61). While both demonstrate comparable performance, we ultimately select PCC for its widespread adoption and intuitive interpretability. The marginal differences between these metrics' rankings are found to be statistically insignif- icant and not affect our analytical conclusions.
Hence, to effectively assess the impact of feature shifts on model performance and analyze whether there a relation- ship between the importance of shifted features and model performance degradation, we employ pearson correlation p to rank features of the given dataset, thereby indicating the importance of each feature for the task. |pl greater than 0.7 can be considered as a moderate linear corvelation(Iversen & Gergen, 2012). Details regarding experiments on four feature importance metrics and pearson correlation are pro- vided Appendix G.
3.4. TabFSBench API
To facilitate the use of TabFSBench for experimen- tal setups in feature-shift scenarios, we have designed the TabFSBench APIs. More details are available at https://gtthub.com// COm/LAMDASZ/MM//AASSSS are divided into five parameters: dataset, model, task, de- gree, and e dataset.
Single Shift. To evaluate the impact of the absence of features with different importance levels on model perfor- mance, we design the single shift experiment. For a given dataset, we first compute correlations of features. Then, we sequentially remove one feature by employing a sampling- with-replacement approach in the ascending order. Finally, compare the gap in model performance among shifted features with different correlations.
The APIs
The dataset parameter specifies the dataset to be used and requires the full name of the dataset. TabFSBench sup- ports datasets from OpenML, Kaggle, and local directories.
The model parameter defines the model to be evaluated and can be selected from tree-based models, deep-learning models, LLMs, and tabular LLMs. New models can be added by following the instructions in the "How to Add New Models" section.
Most/LeasttReelvann Shift. To evaluate how the model performance changes when the feature set with different importance is shifted, we design the most/least-relevant shift experiment. For a given dataset, we first compute correlations of features. Then, we remove features in the ascending (least relevant) or descending (most relevant) order. Finally, we compare the gap in model performance among shifted feature sets with different importance.
The task parameter determines the type of feature-shift experiment to be conducted. The available options include single, least, most, and random.
The degree parameter indicates the proportion of features

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
0.30
0.8
0.033
0:40
0.012
2:111
0.34
-0.04
D:008
0.0200
5009
TabPFN (Hollmann et al., 2023a), and other tabular deep- learning models provided by LAMDA-TALENT', including AutoInt (Song et al., 2019), TabNet (Arik & Pfister, 2021), Tabular ResNet (Gorishmiy et al., 3021), DCN2 (Wang et al., 2021b), NODE (Popov et al., 2020), GrowNet (Badirli et al.,
0.000
0.7
01014
0.012
2020). DANets (Chen et al., 2022), Saint (Somepalli et al.,
0.6
D011
2021). Snn (Klambauer et al., 2017). Switchtab (Wu et al.,
0.
2024), Tabcaps (Chen et al., 2023). Tabr (Gorishniy et al.,
Ligee500
Stilked
Taiph
MLP
No2z
Raster
Tale
Tablby
TAINE
Aurali
F7. Trunaffoory
DENV
Kay
Ster
Tah Tale
Suyy
Medumaci Lians op T Unspreded
2024), Tranfffffrrmer (Huang al., 2020), Tangos (Jef-
fares et al., 2023) and ModernNCA (Ye et al., 2024).
Ascrage Original Performance
Averaee Upper- manormance
LLMs. We select Llama3-8B, LLM released by Meta AI in April 2024, as the representative LLM for our evaluation. To construct the input text for Llama3-8B, we employ the List Template format, leveraging the demonstrated profi- ciency of LLMs in reading and parsing structured list-based inputs, as evidenced by prior research (Hegselmann et al., 2023). A comprehensive explanation of the List Template is provided in Appendix E.3.
Figure 2. Upper bound and original model performance.
to be shifted. The valid range for this parameter from O to where 0 signifies no features are removed, and I signifies all features removed.
The export_datase
parameter controls whether the modified dataset (after removing shifted features) is ex- ported as a CSV file for further use.
Tabular LLMs. LLMs have demonstrated remarkable per- formance zero-shot and few-shot tabular tasks leading to the development of tabular L.MMs which are specifically designed based on LLMs for tabular tasks. We evaluate TabLLM (Hegselmann al., 2023) and Unipredict (Wang et al., 2023). About Unipredict, we choose the light ver- sion instead of the heavy version, because we observe that Unipredict-ight can achieve better performance in the orig- inal paper and our own evaluations. The rest of model settings follow the original paper.
An example command for running a feature-shift experiment in TabFSBench is as follows:
Example Command
python run_ wun_xpprrmeet.... -yy
--dataset Adult --model LightGeM
task random --degree 0.3
-eepoet_ppttaeeee _atsse True
4.2. Pipelines
4. Experiment Setup
Data Set Segmentation. We begin by partitioning the dataset into a train& validation set and a set of test sets. Ap- pendix D shows the segmentation details of each dataset, including Pearson correlation heat maps of datasets. We uniformly use the same feature segmentation and data pre- processing methods.
4.1. Models Benchmarked
We evaluate a suite of models designed for tabular data, drawing from four categories: tree- models, deep- learning models, LLMs and tabular LLMs. Appendix E provides details of all models and their hyperparameters.
Hyperparameter Optimization. We use hyperparameter optimization to help models achieve optimal performance in different datasets. In Appendix E, we provide full hyper- parameter grids for each model.
Tree-Based Models. Gradient Boosting Decision Tree (GBDT) is traditional type of tree-based machine learning models that incrementally improve the model by sequen- tially adding decision trees, optimizes the loss function using gradient descent, and prevents overfitting through regularization and tuning parameters. GBDTs are consid- ered to be state-of-the-ar models on tabular tasks (Grin- sztajn et al., 2022). Hence, we evaluate LightGBM (Ke al., 2017), XGBoost (Chen Guestrin, 2016) and Cat- Boost (Prokhorenkova al., 2018) of GBDTs.
Evaluation Metries. For classification tasks, we utilize ac- curacy and ROC-AUC as model performance, where higher values denote superior model performance. For regression tasks, we utilize Root Mean Square Error (RMSE) as model performance, where lower values denote superior model performance. We also consider the percentage of model per- formance gap A as the robustness of model on feature-shift
Deep-Learning Models. Deep-learning models we evalu- ate consist of MLP, FT-Traasformer (Gorishniy et al., 2021),
hhtps:/gethaa.coomqli22000000
5

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table Average performance gap in different tasks. We attribute the different feature-shift degree for each dataset 20% 40%, 60%, 80% and 100%. Then compute the model's performance gap for each degree of feature shifts, task by task. "\" means this model can't 't handle the regression task. For classification tasks, we choose accuracy. For regression tasks, we choose RMSE. The best in bold and second best is underlined. Model abbreviations are in Appendix E.
Skin
1.00
XGB
CATB
DAN
MILF
NODE:
SW1
CAP
GDS
DCN
FTT
GRO
SKN
TTF
NCA
1.4
T1.1M
4.325
425
<1.992
<0.218
333
-U.11
-3885
40962
-0231
-4.81
40831
-6351
4.355
-0160
-3.33
-0.843
-0042
-3.46
40344
-8.13
41.71
-8.11
-5042
-3.33
Heary Clasilontios
62%
48:389
.0.61
155
0U165
4000
J1881
115
8180
699
.113
11155
8.1
-6807
4U072
4.88
4007
48214
11.01
8177
JUUSS
11075
-0:353
0.377
0.332
0.30
03234
0.271
4.379
4.315
41.35
0.30
4.200
4.386
4.389
0.385
0.305
-0.06
40.47
-0.26
4.303
4:31
40357
4051
4./00
-0244
70%
11817
01.01
007%
1090
4.01
0:075
0.:65
5106
019
6/66
0:115
42%
-4.644
-0122
0.123
-U00
03.83
4UM6
30123
2.005
4U58)
-9444
4U820
-UUR2
-0871
OUMI
03.83
-SAC6
JU3S
4.003
4U032
-6.12
41335
-000
-001
.0.13
85%
940
08383
0177%
178
00.00
4020
236
0.166
110
8.736
0.363
8.716
03
TERTT
4 164
40535
8.505
0.560
8.70
6.314
1107
41.522
-0406
4.31
40308
0.287
0.590
40545
2.48
4.347
0.23
41.20
-0-22
4.793
40362
-3464
4.105
4.21
41..00
40222
41320
-0420
4.346
-4.76
30%
03317
0.333
0.553
0.311
00233
0.31
0.61
0011
0.351
0.03
3.60
D/06
0017
0.033
01311
0163
0.55
6.23
00333
00266
0.855
3523
Regrowker
629
0.733
0360
0.933
0033
0.06
0.883
01090
0380
01812
0.33
85%
1.394
01186
0.000
0.807
0.381
03-44
3008
0606
611
0186
0.45
0.96
0.00
140
0.560
scenarios in open environments,
resentation, the inherent distributional shift substantially diminishes model robustness.
(metric, - metrico)
Most models can't handle feature shifts well. Table 3 reveals that the majority of models encounter significant difficulties in effectively addressing the challenge posed by feature shifts. As the magnitude of feature shift intensifies, the disparity in performance between models widens cor- respondingly. Notably, tree- based models are particularly vulnerable to the adverse effects of feature shifts. Within the context of classification tasks, NODE and tabular LLMs demonstrate relatively lower levels of performance degra- dation. Conversely, in regression tasks, DANets and Tab- Transformer exhibit greater resilience. Despite the current limitation of tabular LLMs to classification tasks, they ex- hibit robust adaptability to feature shifts, suggesting their potential for broader application in the future.
metrico
metric; denotes the model performance where i features shift. subsequent sections, we use metric to refer to performance, and A to refer to robustness.
5. Experiment Results
By conducting four types of experiments to compare the performance and robustness of models in feature shifts, we demonstrate three main observations and corresponding fu- ture work below, which are detailed below. Comprehensive experimental details are provided in Appendix H.7.
Observation 1
No Model Consistently Outperforms. Table reveals that no model can consistently outperform in feature- shift see- narios. Although tabular LLMs display notable robustness, their performance in closed environments and low-degree feature-shift scenarios is not superior. CatBoost contin- ues to highlight its effectiveness in solving tabular tasks. Moreover, the limited performance of NODE, DANets, and Tab Transformer, which demonstrate outstanding robustness in Table 3, confirm that some models' robustness is achieved at the expense of closed environment performance.
Most models have the limited applicability in feature-shift scenarios.
To systematically investigate model robustness under vary- ing degrees of distributional shift, we design controlled experiments by simulating four types of feature-shift scenar- ios. We employ A quantitative metric to assess model performance degradation across different shift magnitudes. Our empirical analysis yields three key observations.
The potential of tabular LLMs suggests that future work should focus on developing fine-tuning frameworks to en- hance semantic parsing capabilities of tabular LLMs for features and improve their reasoning abilities in feature- shift scenarios. Additionally, integrating adaptive prompt- ing mechanisms that dynamically adjust to the presence of feature shifts could further strengthen model robustness. By embedding domain knowledge and causal reasoning into the prompting strategy, LLMs are able to achieve deeper understanding of feature shifts, thereby generating more accurate and reliable predictions.
Feature shift impairs model robustness. To rigorously assess the theoretical performance limits of the models un- der feature shift conditions, we conduct an empirical eval- uation by training the models directly on the shifted data distribution and subsequently evaluating their generalization capability a held-out test set. As illustrated in Figure 2, our experimental results reveal consistent performance gap between tabular models trained on the original dataset versus those trained on the feature-shift dataset. The com- parative analysis demonstrates that while training on the original dataset offers the advantage of broader feature rep-
6

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Credit
Eleetreety
Eyrmovemnnnd
Penguins
Janib
Abslun
Paak
Eyen Sept
Laptop
AscincyyRMSF
Figure Results for single shift experiments with correlation analysis experiments across 12 datasets. The æ-axis represents features ordered in ascending correlation with the target, the left y-axis represents the average performance (accuracy or RMSE) of all models and the right y-axis shows the absolute value of correlation. Features removed in ascending order of their correlation values observe the impact model performance.

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table Average rank in different tasks. We attribute the different feature-shift degree for each datasct 20%, 40%, 60%, and 80%. Then compute the model's performance rank for each degree of feature shifts, task by task. 0% means the model ID performance. "\" means this model can't It handle regression task. classification tasks, we choose accuracy. For regression tasks, we choose RMSE. The best is in bold and second best is underlined. Superior accuracy in classification tasks is in itaiics Model abbreviations are in Appendix E.
Task
Shult
LGB
XGB
CATB
PFN
DAN
MLP
NODE
RES
SW3
CAP
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TIF
YASK
NCA
LMA
TLLM
UNI
15
Blaney Clasification
60%
40%
22
21
25
23
5
4
12
13
15
10
20
16
14
9
5
4
17
19
18
14
1.
18
17
20
22
21
16
24
24
12
13
3
80%
26
17
16
16.
11
15
71
11
2
20
15
22
12
19
10
13
17
20%
10
8
4
16
70
14
15
13
19
21
13
18
12
1.
74
17
21
9
Clandiication
01%
13
70
11
25
21
10
13
16
18
LO
100%
24
18
17
11
19
10
22
1.5
21
23
16
06
17
20
16
10
13
15
Regression
10%
6
10
87
20
15
12
13
107
70
3
17
14
18
17
13
10
15
13
16
9
100%
19
11
10
14
17
16
70
Top
Average Bank
13.1
13.6
4.8
10.4
15.5
98
16.5
9/6
10.9
12.3
13.0
3.2
15.0
8.4
14.2
18.3
9.2
14.2
18.9
10.6
12.1
12.
2.9
Observation 2
importance of strongly correlated features. Therefore, is essential to develop feature importance-driven optimiza- tion algorithms that incorporate adaptive mechanisms, such as dynamic feature weighting hierarchical feature selec- tion, to emphasize tnong-correlate features. Additionally, this highlights the necessity of mitigating shifts in strong- correlated features in open environments.
Shifted features' importance has a linear trend with model performance degradation.
Considering that feature shifts exert negative impacts on models from Observation 1, it is necessary to determine whether there is a relationship between the importance of shifted features and model performance degradation. To this end, we conduct single shift experiments by analyz- ing the average performance of all models for each dataset, and
Observation 3
Model closed environment performance corre- lates with feature-shift performance.
shift experiments by comparing the correlation sum of shifted feature sets with model perfor- mance degradation in feature-shift scenarios. We obtain two observations from these experiments.
In Figure 5, a comparative analysis of model performance presented, contrasting the outcomes in closed environments with those in feature-shift scenarios within the context of random shift experiments. The analysis reveals a notable trend: models that exhibit superior performance in closed environments tend to demonstrate relatively enhanced per- formance when subjected to feature-shift scenarios. This observation suggests the existence of a positive correlation between model performance in closed environments and their robustness in feature-shift scenarios. Specifically, the enhanced performance in closed environments appears to confer degree of resilience to the models when confronted with the challenges posed by feature-shift scenarios, thereby highlighting the potential interdependence between these two distinct operational contexts.
Single strong correlated feature shifted causes greater model performance degradation. Figure 3 illustrates that model performance decreases more significantly as the shifted single feature becomes more relevant to the target. The extent of model degradation caused by a strongly cor- related feature shift is larger than that caused by a weakly correlated feature shift. We also observe that model per- formance may potentially improve if features that are less relevant to the target are removed, whose detailed analysis can be found in Appendix H.3.
Shifted feature set's correlations have a relationship with model performance degradation linearly. Figure shows that as the correlation sum of the feature set gets larger, the model performance degradation gets larger. We observe linear trend between the correlations of the shifted feature set and model performance degradation (see the blue line in Figure 4; p 0.74).
This observation suggests that improving model perfor- mance in closed environments may serve foundational step to enhance their adaptability and robustness in feature- shift scenarios. A rigorous theoretical investigation into the relationship between closed environment performance and open environments adaptability is warranted, with particular emphasis on elucidating the underlying mechanisms that govern model robustness. Further exploration into method-
The linear trend between the correlation sum of shifted fea- tures and model performance degradation underscores the
8

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
0.3
Model Frottue- Stufh Porformance
Avanng: Mokel Do Deggraaiiiiiiiii Degrsdatins
Model Performance Gap
04
0.3
031
0.2
0.1
Dath point
0.0
55% Confidence Interval
Model Closed-Envireemee Performance
0.1
Rache:
F1-Iaadeane
Libra--53
Sum of Shifted Feature Set's Correlations
Millow
SwashTaa
Taa11 M
Calhoos
Cpp
Sekor
LACEN
tahhe
Linor Fa
DANes
DANGUS
Taki
Cintfiiaa Ieirral
NCDE
ModeniNCA
Figure use A (described in equation 1) to measure the per- formance decrease. Sum of shifted feature set's correlations refers to the sum of Pearson correlation coefficients of shifted features. Notably, performance decrease and sum of shifted feature set's correlations demonstrate strong correlation, with Pearson cor- relation coefficient of p 0.7405.
Figure 5, Model performance in closed environments model fcature-shift performance. Closed environment means that the dataset does not have any degree of feature shift. Feature-shift means average model performance in all degrees of feature shifts.
ologies for enhancing model resilience in open environ- ments, such as through domain adaptation, transfer learning, or robustness-awe aware training paradigms.
remains limited by the current absence of large-scale tab- ular LLM evaluations and comparative studies with other shift types. These aspects represent important directions for future empirical validation and benchmark expansion feature-shift challenges in open environments.
6. Conclusion
We introduce TabFSBench, comprehensive and rigorously designed benchmark specifically tailored to systematically investigate feature shifts in tabular data. TabFSBench en- compasses diverse tasks, enabling the evaluation of model performance and robustness, and benchmarking of tabular models in feature-shil scenarios. To enhance accessibility and ensure reproducibility, we provide intuitive and user- friendly Python APIs, facilitating seamless dataset retrieval and integration into experimental workflows, Additionally, we conduct extensive empirical evaluations across four dis- tinct feature-shift scenarios. Our three observations not only underscore the significant challenges posed by feature shifts but also offer insights for the future development of fcature-shift research.
Benchmark Availability Statement
The benchmark code for this paper is available at www.../.AAAAAAAAA/AA....
The project page TabFSBench which contains the leaderboard is available Tanee-TabBBBech..
A Acknowledgements
This research is supported by the National Science Founda- tion of China (62306133, 624B2068), and the Key Program of Jiangsu Science Foundation (BK20243012), We would like thank reviewers for their constructive suggestions.
While this paper provides comprehensive evaluation of fea- ture shift impacts on tabular models, several limitations warrant discussion. First, the current framework focuses primarily on feature decrement scenarios, as conventional tabular models inherently require fixed input dimensions and thus automatically ignore newly added features in increment scenarios. Second, our evaluation excludes specialized ar- chitectures designed for feature-in increment issues. Third, the analysis does not examine how models respond to shifted features with identical correlation structures. Additionally, while we have evaluated diverse tabular tasks, scope
Impact Statement
The paper introduces novel contribution aimed at advanc- ing the burgeoning field of open environments. The ob- servations elucidated within this work possess broad and multifaceted societal implications. However, due to the limitations of this paper, detailed elaboration of these im- plications is reserved for future discourse. These discussions are expected to center best practices and the development of regulatory frameworks that can effectively harness the benefits of open environments machine learning.

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
References
Fang, X., Xu, W., Tan, F. A., Zhang, J., Hu, Z., Qi, Y. J., Nickleach, S., Socolinsky, D., Sengamedu, S., and Falout- sos, C. Large language models tabular data: Prediction, generation, and understanding-a survey. arXiv preprint arXiv:2402. 2024.
Ajakan, H., Germain, P., Larochelle, H., Laviolette, F,, and Marchand, M. Domain-adversariaa neural networks. arXiv preprint arw.v:1412.444444 2014.
Altman, N. and Krzywinski, M. Tabular data. Nature Methods, 14(4):329-331,2)) 2017.
Gardner, J., Perdomo, J. C., and Schmidt, L. Large Scale Transfer Learning for Tabular Data via Language Model- ing. arXiv preprint arXiv:2406. /2031, 2024a.
Arik, Ô. and Pfister, T. Tabnet: Attentive interpretable tabular learning. In Proceedings of the 35th AAAI Con- ference Artificial Intelligence, pp. 6679-6687, 2021.
Gardner, J., Popovic, Z., and Schmidt, L. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, pp. 53385- 53432, 2024b.
Arjovsky, M. Bottou, L., Gulrajani, I,, and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: wwt.i19070..999 2019.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and numan-labeles dataset for audio events. In Proceedings of the IEFE International Conference on Acoustics, Speech and Signal Processing, pp. 776-780, 2017.
Badirli, S., Liu, X., Xing, 7., Bhowmik, A., Doan, K., and Keerthi, S. K. Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07977, 2020.
Bao, M., Zhou, A., Zottola, S., Brubach, B., Desmarais, S., Horowitz, A., Lum, K., and Venkatasuuraamanian. It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks. arXiv preprint arXiv:2 106.09290 2021.
Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. Ad- vances in Neural Information Processing Systems, pp. 18932-18943, 2021.
Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. TabR: Tabular Deep Learning Meets Nearest Neighbors. In Proceedings of the 12th International Conference on Learning Represen- tations, 2024.
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M. and Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, ((16):7499-5199 2022.
Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree- based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, pp- 507-520, 2022.
Chen, J., Liao, K., Wan, Y., Chen, D. 7., and Wu, J. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, pp. 3930-3938, 2022.
Guo, L.-Z., Jia, L.-H., Shao, J.-J., and Li, Y.FF Robust semi- supervised learning in open environments. Frontiers of Computer Science, 19(8): 198345, 2025.
Chen, J., Liao, K., Fang, Y., Chen, D., and Wu, J. Tabcaps: A capsule neural network for tabular data classification with bow routing. In Proceedings of the 11th Interna- tional Conference on Learning Representations, 2023.
He, H., Queen, O., Koker, T., Cuevas, C., Tsiligkaridis, T., and Zitnik, M. Domain adaptation for time series under feature and label shifts. In Proceedings of the 40th International Conference on Machine Learning, pp. 12746-12774, 2023.
Chen, T. and Guestrin, C. XGBoost: A Scalable Tree Boost- ing System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-788777 1235794,777 2016.
Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In Proceed- ings of the 26th International Conference on Artificial Intelligence and Statistics, pp. 5549-5581, 2023.
Chizat, I., Roussillon, P., Léger, F., Vialard, F.-X., and Peyré, G. Faster Wasserstein distance estimation with the Sinkhorn divergence. Advances in Neural Information Processing Systems, pp. 2257-2269, 2020.
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. In Proceedings of the 11th International Conference on Learning Represen- tations, 2023a.
Deng. J., Dong, W., Socher, R., Li, L.-J., , Li, K., and Fei-Fei, L. ImageNet: A large- scale hierarchical image database. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 248 225,2 2009.
10

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Hollmann, N., Müller, S., and Hutter, F. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering, Advances in Neural Information Processing Systems, pp. 44753- 44775, 2023b.
Malinin, A., Band, N., Gal, Y., Gales, M., Ganshin, A., Ches-
nokov, G., Noskov, A., Ploskonosov, A., Prokhorenkova, L., Provilkov, I., Raina, V., Raina, V., Roginskiy, D., Shmatova, M., Tigas, P., and Yangel, B. Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks, Advances in Neural Information Processing Sys- tems, Datasets and Benchmarks Track, 2021.
Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tab- transformer: Tabular data modeling using contextual em- beddings. arXiv preprint W0Xiv2002.006678 2020.
Malinin, A., Athanasopoulos, A., Barakovic, M., Cuadra, M. B., Gales, M. J. F., Granziera, C., Graziani, M., Karta- shev, N., Kyriakopoulos, K., Lu, P.-J., Molchanova, N., Nikitakis, A., Raina, V., Rosa, F. L., Sivena, E., Tsarsi- talidis, V., Tsompopoulou, E., and Volf, E. Shifts 2.0: Extending the dataset of real distributional shifts. arXiv preprint arXiv:2206. 10000.15400 15407, 2022.
Iversen, G. R. and Gergen, M. Statistics: The conceptual approach. Springer Science & Business Media, 2012.
Jeffares, A., Liu, T., Crabbé, J., Imrie, F., and van der Schaar, M. TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization. In Proceedings of the 11th International Conference on Learning Representations, 2023.
McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? Advances Neural Information Processing Systems, pp. 76336-76369, 2023.
Jia, I..H., Guo, L.-7., Zhou, 7., and Li, Y.-F. LAMDA-SSL: comprehensive semi-supervised learning toolkit. Sci- ence China. Information Sciences, 67(1):117101, 2024a.
Jia, L.-H., Guo, L.-Z., Zhou, Z., and Li, Y.-F. Realistic evaluation of semi- semi-supervised learning algorithms in open environments. In Proceedings of the 12th International Conference on Learning Representations, 2024b.
Meijerink, L.,, Cinà, G., and Tonutti, M. Uncertainty esti- mation for classification and risk prediction medical tabular data. arXiv preprint Stfi::2004.058222 2020.
Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effect of natural distribution shift on question answering models. In Proceedings of the 37th International Conference on Machine Learning, pp. 8905-6916 2020.
Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well- tuned simple nets excel on tabular datasets. Advances in Neural Information Processing Systems, pp. 23928- 23941, 2021.
Nam, J., Kim, K., Oh, S., Tack, J., Kim, J., and Shin, J. Op- timized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning. Advances in Neural Infor- mation Processing Systems, pp. 92352 29350, 2024.
Ke. G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. LightGBM: Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems, pp. 31 3149-3157, 2017.
OpenAL. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2024.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. Advances in Neural Information Processing Systems, pp. 972 -911, 2017.
Parmar, J., Chouhan, S., Raychoudhury, V., and Rathore, S. Open-world machine learning: applications, challenges, and opportunities. ACM Computing Surveys, 15(10) 2023.
Kolesnikov, S. Wild-Tab: A Benchmark For Out-Of- Distribution Generalization In Tabular Regression. arXiv preprint arXiv:2312.01792 2023.
Popov, S.,, Morozov, S., and Babenko, A. Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Liang, J., He, R., and Tan, T. A comprehensive survey on test-time adaptation under distribution shifts. Interna- lional Journal of Computer Vision, pp. 1-34, 2024.
Liu, J., Wang, T., Cui, P., and Namkoong, H. On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances Neural Information Processing Systems, pp. 51371-51408, 2023.
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush. A. V., and Gulin, A. CatBoost: unbiased boosting with categori- cal features. Advances in Neural Information Processing Systems, pp. 6639-66699, 2018
Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Ye, H.-J TAL- ENT: A Tabular Analytics and Learning Toolbox. arXiv preprint ar arXiv:2407.0400 2024.
Ruder, S. An overview of gradient descent optimization algorithms. arXiv preprint 1609::6000.000 2016.

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Sahakyan, M., Aung, Z., and Rahwan, T. Explainable artifi- cial intelligence for tabular data: A survey. IEFE access, 9:135392-135422, 2021.
West, D. Neural network credit scoring models. Computers & operations research, 27(11-12:1111-1152, 2000.
Wu, J., Chen, S., Zhao, Q. Sergazinov, R., Li, C., Liu, S., Zhao, C., Xie, T., Guo, H., and Ji, C. Switchtab: Switched autoencoders are effective tabular learners. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pp. 15924-15933, 2024.
Shao, J.JJ., Yang, X.-W., and Guo, L.-Z. Open-set leamnng under covariate shift. Machine Learning, 1 13(4))1643- 1659, 2024.
Shen, M., Bu, Y., and Wornell, G. W. On balancing bias and variance in unsupervised multi-source-free domain adaptation. In Proceedings of the 40th International Con- ference on Machine Learning, pp. 30976-30991, 2023.
Ye, H.-J., Yin, H.-H., and Zhan, D.-C. Modern Neighbor- hood Components Analysis: A Deep Tabular Baseline Two Decades Later. arXiv preprint arXiv:2407.03257, 2024.
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre- training, arXiv preprint 1xXiv:2106.01:44 2021.
Yıldız, A. Y. and Kalayci, A. Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data. arXiv preprint arXiv:2410.03705, 2024.
Zhang, T., Zhang, Z. A., Fan, Z., Luo. H., Liu, F., Liu, Q.. Cao, W., and Jian, L. OpenFE: automated feature gener- ation with expert-level performance. In Proceedings of the 40th International Conference on Machine Learning, pp. 41880-41901, 2023.
Song, W., Shi, C., Xiao, Z., Duan, Z., Xu, Y.. Zhang, M., and Tang, J. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1 1161-1170, 2019.
Zhao, C., Zio, E., and Shen, W. Domain generalization for cross-domain fault diagnosis: An application-oriente perspective and a benchmark study. Reliability Engineer- ing & System Safety, pp. 109964, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13977, 2023.
Zhou, K., Yang. Y., Qiao, Y., and Xiang, T. Domain Gen- eralization with MixStyle. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Wang, A., Singh, A., Michael, J., Hill, F,, Levy, O., and Bow-
man, S. GLUE: A Multi- Task Benchmark and Analysis Platform for Natural Language Understanding. In Pro-
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. Do- main generalization: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396- 4415, 2022.
ceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353 353-355,220 2018.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Dar- rell, T. Tent: Fully Test-Time Adaptation by Entropy Minimization. In Proceedings of the 9th International Conference on Learning Representations, 2021a.
Zhou, Z., Guo, L.-Z.,, Jia, L.-H., Zhang, D., and Li, Y.-F. Ods: Test-time adaptation in the presence of open-world data shift. In Proceedings of the 40th International Con- ference on Machine Learning, pp.
2023.
Wang, R., Shivanna, R., Cheng, D., Jain, S., Lin, D., Hong, L., and Chi, E. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank sys- tems. In Proceedings of the Web Conference 2021, pp- 1785-1797, 2021b.
Zhou, Z., Yang, Y.-K., Guo, L.-Z., and Li, Y.-F. Fully Test- time Adaptation for Tabular Data. In Proceedings of the 39th AAAI conference on Artificial Intelligence, 2025.
Zhou, Z.-H. Onvneenvironmenn machine learning. National Science Review, 98):nwac 123, 2022.
Wang, R., Wang, 7.., and Sun, J. Unipredict: Large language models are universal tabular predictors. arXiv preprint arXiv:2310.03266. 2023.
Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F,, and Chua, T.-S. TAT-QA: A question answering benchmark on hybrid of tabular and textual content in finance. arXiv preprint arXiv: 105.0:2205.0224,,, 2021.
Wang, Y., Chen, H., Fan, Y., Sun, W., Tao, R., Hou, W., Wang, R., Yang, L., Zhou, Z., Guo, L.-Z., et al. Usb: A unified semi-supervised learning benchmark for clas- sification. Advances in Neural Information Processing Systems, pp. 31333333399 199381361,,22 2022.
Zhu, X., Hu, H., Lin, S., and Dai, J. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 9308-933 9316, 2019.
12

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
A. Related work
A.1. Open Environments Challenges
Traditional machine learning research predominantly assumes closed environment scenarios, where key factors in the learning process remain stable and predictable (Guo al., 2025; Jia al., 2024b). Although machine learning has achieved remarkable success across various applications, an increasing number of real-world tasks partielllly those situated in open environments- face dynamic changes in critical factors. These challenges have given rise to the field of open environments Machine Learning (Open ML). Zhou (2022) identified four core challenges in Open ML: the emergence of new classes, data distribution shifts, evolving learning objectives, and feature space shifts.
The emergence of new classes refers to the occurrence of previously unseen categories during the testing phase that were entirely absent during training. Data distribution shift pertains to changes in the distribution of test data, violating the traditional assumption that data are independent and identically distributed (i.i.d.). Evolving learning objectives arise as data volume increases and model accuracy improves, prompting a shift in focus from solely maximizing accuracy to addressing additional practical priorities, such minimizing energy consumption. Feature space shift is characterized by the introduction of new features or the removal of existing ones. Conventional machine learning models, which typically rely on the assumption that training and testing data share the same feature space, often struggle to adapt to such changes, leading to substantial performance degradation in open-environment deployment scenarios. To address these emerging challenges, recent research has primarily centered on domain adaptation and domain generalization. Domain adaptation methods utilize a limited amount of labeled or unlabeled target-domain data to transfer knowledge effectively (Shen et al., 2023). In contrast, domain generalization focuses on training models with robust generalization capabilities to maintain high performance across diverse target-domain distributions. For instance, the MixStyle method (Zhou al., 2021) improves generalization by blending statistical properties, such as mean and standard deviation, from different domains. More recently, Test-Time Adaptation (TTA) has been proposed as a promising and flexible approach to address data distribution shifts. TTA enables models to adapt dynamically during inference by leveraging test data batches for continual learning (Liang et al., 2024). However, much of the research on these approaches has primarily targeted non-tabular domains, such as computer vision and natural language processing (Miller et al., 2020). Moreover, these methods often fail to surpass the performance of traditional optimization algorithms, such as Stochastic Gradient Descent (SGD) (Ruder, 2016), highlighting the need for further advancements tailored to open-environment challenges in tabular data.
Although existing Heterogeneous Domain Adaptation (HeDA) methods have achieved significant progress on feature shift of images, tabular data presents fundamentally different pattern. The inherent structure of it presents challenges when attempting to directly implement HeDA on such datasets. Moreover, our review of the literature reveals that what is often referred to as "feature shift" in many papers is essentially a form of distribution shift. For example, He et al. (2023) regards covariate shift as feature shift.
A.2. Tabular Data in Machine Learning
Tabular data, characterized by its structured and heterogeneous features, is extensively used across various fields, including medical diagnostics, financial analysis, recommendation systems, and social sciences (Borisov et al., 2022; Kadra et al., 2021). Unlike domains such as computer vision or natural language processing, tabular data poses unique challenges to machine learning models due to its high dimensionality, heterogeneity, and the complex interdependencies among features (Fang et al., 2024). Current approaches for modeling tabular data can be broadly classified into two main categories: tree-based ensemble models and deep learning models.
Tree-based ensemble models, such as XGBoost (Chizat et al., 2020), LightGBM (Badirli et al., 2020), and CatBoost (Prokhorenkova et al., 2018), have long been regarded as the state-of-the-ad the-art for tabular data modeling. These models excel in handling irregular patterns and non-informative features within the objective function and are well-suited for addressing the non-rotation invariance of tabular data (Grinsztajn et al., 2022). Their robustness and interpretability further contribute to their widespread adoption. In contrast, the rise of deep learning has spurred the development of numerous deep learning-based models tailored for tabular data. Notable examples include DCN V2 (Wang et al., 2021b), which integrates multi-layer perceptron (MLP) modules and feature crossover modules; AutoInt (Song et al., 2019) and FT-Transformer (Gorishniy et al., 2021), both of which leverage Transformer architectures; Res-eet- tabular variants (Gorishniy et al., 2021); and differentiable tree-based heuristics, such as Neural Oblivious Decision Ensembles (NODE) (Popov et al., 2020). These models aim to exploit the complex inter-feature dependencies inherent in tabular data to improve predictive performance. In eosseeenvvronment scenarios, the performance of deep learning models on tabular data has not yet surpassed
13

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
that of tree-based models. Furthermore, the generalization capabilities of both types of models in open environments. particularly when facing feature distribution shifts, have not been assessed.
A.3. Benchmark in Tabular Data
In machine learning, establishing effective benchmarks is essential for evaluating and comparing the performance of various algorithms. An ideal benchmark provides standardized datasets and evaluation criteria ensure the effecttivnness, reliability, and robustness of algorithms in practical applications (Wang et al., 2022; Jia et al., 2024a). While benchmarks in domains such as computer vision (e.g., ImageNet (Deng et al., 2009)), natural language processing (e.g., GLUE (Wang et al., 2018)), and audio classification (e.g.. AudioSet (Gemmeke et al., 2017)) are relatively mature, benchmarks for tabular data remain underdeveloped. Many existing tabular data benchmarks suffer from significant quality issues. For instance, the German Credit dataset is small in scale, and the Adult dataset contains inherent biases and data quality problems (Bao et al., 2021). These limitations constrain their utility for conducting in in-depth research and hinder the development of robust machine learning methods for tabular data.
Furthermore, benchmarks capable of systematically evaluating distribution shifts are crucial for assessing the robustness and adaptability of models to real-world data variations. Distribution shifts refer to the discrepancy between the data distribution during inference and that during training, which can severely impact model performance (Zhou et al., 2023). To address this, several benchmarks have been proposed to evaluate the handling of distribution shifts in tabular data. For example, Shifts and Shifts 2.0 (Malinin et al., 2021; 2022) focus on uncertainty estimation and include tasks with temporal and spatio-temporal variations in tabular data. WhyShift (Liu et al.. 2023) specializes in evaluating spatio-temporal shifts using five real-world tabular datasets. TableShift (Gardner et al., 2024b) provides comprehensive evaluation of 19 model types across 15 binary classification tasks, with 10 tasks explicitly related to domain generalization. Similarly, Wald-Tab (Kolesnikov, 2023) emphasizes domain generalization in tabular regression tasks, comparing 10 domain generalization methods against standard Empirical Risk Minimization (ERM) applied to Multilayer Perceptrons (MLPs). However, despite these benchmarks offering comprehensive scenarios for addressing distribution shifts, their focus remains primarily on changes in data distributions, with limited attention to feature space shifts, such as the addition or removal of entire features in dataset. Although Gardner et al. (2024a) explored feature drop experiments, their evaluation was confined XGBoost and their proposed model, lacking comprehensiveness. The absence of benchmarks that adequately account for feature space shifts hinders researchers from effectively evaluating and optimizing algorithms to address such challenges, which remains a critical gap in the field of tabular data processing.
B. TabFSBench Components
To promote systematic evaluation and community collaboration in the context of feature-shifee shifted tabular data learning, we present TabFSBench, a modular benchmark framework. It features reproducible evaluation protocols, an extensible API interface, and continuously updated public leaderboard.
B.1. Benchmark Composition and Datasets
TabFSBench currently comprises 12 datasets selected from Grinsztajn al. (Grinsztajn et al., 2022) and TabZilla (McElfresh al., 2023). These datasets exhibit significant heterogeneity in terms of scale, domain, and task structure, covering wide range of potential feature shift scenarios such as covariate shift and missing values. To comprchensively assess model robustness under real-world distributional changes, we design four distinct experimental configurations that systematically evaluate model generalization across various perturbations. All experiments are repeated with multiple random seeds to mitigate stochastic variance. We further aim to construct practically meaningful feature-shift datasets in future versions, thereby encouraging deeper investigation into robustness under feature shift.
B.2. Public Leaderboard and Community Contribution
We have established and actively maintain a project homepage and a complete ranking system. Evaluation results of newly released models or datasets are updated regularly to ensure that the research community can access the latest benchmark performance. For instance, TabFSBench already includes evaluation results for representative models such as TabPFN v2. We encourage researchers submit evaluation results for their custom models datasets. In future updates, we plan to support both public and private leaderboard modes to meet broader range of user requirements.
14

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
B.3. API Design
To enable reproducibility and flexible evaluation, TabFSBench offers command-line option allows users to export dataset variants under different feature-shiff settings (e.g., single-column missingness, controlled levels of missingness, and complete enumeration of missing scenarios) by setting the flag True.
which
We also provide a README... md file, which describes how users can add new datasets and models. Further details on code functionality will be elaborated in the final version.
C. Real-Worke Feature-Shift Challenges
C.1. Werl-World Feature-Shift Datasets
There currently exists no dataset specifically designed for feature shift, unlike Tableshift (Gardner al., 2024b) which was developed for distribution shifts. However, we have preliminarily constructed feature-shifted dataset based on the Heart dataset. Given that different features in the original dataset require distinct measurement instruments, we categorized the features into three groups: basic features, electrocardiogram (ECG) features, and exercise stress test features.
In the constructed ceature-shifted Heart dataset, both the training set and the test set step 0 contain all features. However, patients in the test set step lack ECG measurements, resulting in the absence of RestingECG and ST r_SSlope features. Similarly, patients in the test set step did not undergo an exercise stress test, leading to the absence of ExerciseAngina and httph/hhh.oo//AAAA////ii//iaaaaa
Oldpeak. A subset of examples is provided at
Note that for meaningful evaluation of feature-shifted datasets, models must be assessed under specific partitioning schemes. Applying the four experimental settings proposed in our paper would undermine the unique characteristics and practical relevance of such datasets.
C.2. Real-World World Feature-Shift Issues
Section 2.2 of this paper employs forest disease monitoring as a case study to demonstrate how sensor degradation leads to a reduction in available features. As further evidenced by the designed heart dataset in above, incomplete medical examinations may result in missing diagnostic indicators (features) due to the absence of specific equipment.
The feature shift phenomenon also manifests prominently in financial and transportation domains:
Stock prediction models trained on comprehensive features (e.g.. financial ratios, macroeconomic indicators) may encounter missing features (c.g., sentiment indices) during real-world deployment due to unforeseen events.
Finance:
Transportation: Accident prediction models relying features like road conditions and weather data may experience partial feature absence caused by sensor failures or insufficient data collection.
D. Benchmark Datasets
This section provides background information and the sources of each dataset in TabFSBench. Pearson correlation analyses of the datasets are also provided in Figure 6.
D.1. Binary Classification
Credit The original dataset contains 1,000 entries with 20 categorical/ymbolic attributes prepared by Prof. Hofmann. In this dataset, each entry represents a person who takes a credit from bank. Each person is classified having good or bad credit risk according the set of attributes. The target is to determine whether the customer's 's credit is good or bad. This dataset is available at https://www..ppstt www.wppww.....arrrttooaaaaaaoooe
Electricity The Electricity dataset, collected from the Australian New South Wales Electricity Market, contains 45,312 instances from May 1996 to December 1998. Each instance represents 30-minute period and includes fields for the day, timestamp, electricity demand in New South Wales and Victoria, scheduled electricity transfer, and a class label. The target to predict whether the price in New South Wales is up or down relative to a 24-hour moving average. based on market demand and supply fluctuations. This dataset available on
15

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Heart Cardiovascular diseases (CVDs) are the leading cause of death globally, responsible for 17.9 million deaths annually. Heart failure is a common event caused by CVDs and this dataset contains 11 features that be used to predict possible heart disease. The target is determine whether the patient's s heart disease is present or absent. This dataset is available on falurtf--truueee-tttitnn.. prediction.
This dataset aims to construct a predictive model using various machine learning algorithms and document the end-to-end steps using a template. The MiniBooNE Particle Identification dataset is binary classification task where we attempt to predict one of two possible outcomes. The target determine whether the neutrino is an electron or muon. This dataset is available at
Miniboone
D.2. Multi-class Classification
Iris The Iris flower dataset, introduced by Ronald Fisher in 1936, contains 150 samples from three Iris species: Iris setosa, Iris virginica, and Iris versicolor. Each sample has four features: sepal length, sepal width, petal length, and petal width, measured in centimeters. The target is to classify the Iris species as setosa, versicolor, or virginica. This dataset is available on
Jannis This dataset used in the tabular benchmark from (Grinsztajn et al., 2022). It belongs to the 'classification on numerical features' benchmark. The dataset is designed to test classification performance using numerical features, and presents challenges such as varying data distributions, class imbalances, and potential missing values. serves as a critical evaluation tool for machine learning models in real-world scenarios, including medical diagnosis, credit rating, and object recognition tasks. This dataset is available on
Penguins Data were collected and made available by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER, a member of the Long Term Ecological Research Network. The goal of the Palmer Penguins dataset to offer a comprehensive resource for data exploration and visualization, serving as an alternative to the Iris dataset. The target is to classify the penguin species as Adelie, Chinstrap, or Gentoo. This dataset is available at htl.:aaaw..ooo/uuusssssssssaaaaa owg.r//ooowwwllcuuccceeieeeenipe penguins-species.
Eye Movements This dataset is designed to predict the relevance of sentences in relation to given question based on eye movement data. The target is to classify sentences as irrelevant, relevant, or correct, using 27 features, in- cluding landing position (landingPos), first fixation duration (P stittiitaion), next fixation duration (nextFixDur). time spent on the predicted region (timePrtctg), and other relevant eye movement metrics. This dataset is available at http://wwwwkagglle.
D.3. Regression
Abalone The age of abalone is traditionally determined by cutting the shell, staining it, and counting the rings under a microscope, a process that both tedious and cine-consuming. This dataset uses casier-to-obtain physical measurements, such as length, diameter, and weight, to predict the abalone's 's age. The target is to predict the age, providing more efficient approach. This dataset is available on https:/wwww.aggll.coo wwtp////wweeeeeeoooooooeee/neeee
Bike The dataset records the rental of shared bikes in the Washington area from 201 1-01-01 to 2012- 12-31, contain- ing 11 features such as season, holiday, working day, and weather conditions. The goal is to predict the total count of bikes rented each hour, with the target being to forecast the number of bicycles available for rent today based on his- torical rental patterns and external factors like temperature, humidity, and scasonal trends. This dataset available on
Concrete Concrete is the most important material in civil engineering, and its compressive strength is influenced by highly nonlinear relationship with its ingredients and age. The dataset contains 9 attributes, including variables such as cement, water, and age. The target is to predict the concrete compressive strength (measured in MPa) using these input variables. This dataset is available on
Laptop
The original dataset was pretty compact with a lot of details in each column. The columns mostly consisted of long strings of data, which was pretty human-readable and concise but for Machine Learning algorithms to work more
16

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 5. Hyperparameter Grids of Tree-hased Models.
Model
Hyperparameter
Values
Num Leaves
{31, 127)
Learning Rate
{0.01,0.1
LightGBM
Min Data In Leaf
(20,50, 100}
Min Sum Hessian Leaf
{1c
3, le 2,
1}
Learning Rate
{0.01, 0.13
Max. Depth
{1,5,9}
N Estimators
(10000, 20000, 30000}
XGBoost
Subsample
(0.5, 0.8, 1.0)
Colsample Bytree
{0.5, 0.8, 1.0)
Min Child Weight
{1,3,5}
Learning Rate
{0.01, 0.05,
CatBoost
Depth
{4 6,
Iterations
(500, 1000, 2000}
efficiently it's better to separate the different details into their own columns. After doing so, 28 duplicate rows were exposed and removed with this dataset being the final result. The target is to predict the price of this laptop. This dataset is available on www.:/awgge..oo///aastttt//oeoee
E. Benchmark Models
In this section, we provide introductions tree-based and deep learning models and hyperparameter grids in Table 6. Among them, TabPFN and tabcaps only applicable to the classification task, not to the regression task. We implement adaptive hyperparameter optimization based on the Optuna framework and following previous studies (Liu et al., 2024), fixing the batch size at 1024 and conducting 100 independent trials through train-validation splits to prevent test set leakage, with the best pest-perfommed hyperparameters fixed during the final 15 seeds. For LLMs, we provide their prompts in Figure 7.
E.1. Tree-Basee Based Models
LightGBM LightGBM(aadirli et al., 2020) is a machine learning model based on the Boosting algorithm. Its core algorithms are Gradient- maadient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). GOSS calculates the importance of each sample through gradients, discards unimportant samples, and selects subset of important samples for training. EFB is used to reduce the dimensionality of the feature space."
XGBoost XGBoost(Chizat et al., 2020) is an efficient and flexible machine learning model that incrementally builds multiple decision trees by optimizing the loss function, with each tree correcting the errors of the previous one to continuously improve the model's predictive performance. XGBoost also incorporates the gradient boosting algorithm, iteratively training decision rre-based models with the goal of minimizing residuals and enhancing predictive accuracy.
CatBoost CatBoost (Prokhorenkova et al., 2018) is a powerful boosting- based model designed for efficient handling of categorical features. It uses the "Ordered Boosting" technique, which calculates gradients sequentially prevent target leakage and maintain the independence of each training instance. At the same time, CatBoost employs "Target-based Categorical Encoding," converting categorical variables into numerical representations based on target statistics, thereby reducing the need for extensive preprocessing and improving model performance.
E.2. Deep Learning Models
AutoInt Autolnt(Song al., 2019) efficiently handles large-scale data by mapping numerical and categorical features into the same low-dimensional space and leveraging multi-head self-attentive neural networks to model feature interactions.
DANets DANets(Chen et al., 2022) enhance the feature representation capacity of tabular data by introducing Abstract Layers and shorteut paths, and employ structure re- pe-araameterization reduce computational complexity, demonstrating
17

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Jannis
640
Cywrovementt
Abalonn
8.00
0.2
149
A/14
6.66
Bike
Laptop
480
0.1
046
0.1
2:40
4:31
8.
JaC7
0:01
4:30
1.59
Figure 6. Pearson Correlation Analysis of the Datasets. Images sorted in binary classification,, multi-class classification, and regression order. The last column of each image is chosen as the input column with the target Pearson correlation coefficient.
18

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
effectiveness and extendibility in tabular data tasks.
DCN2 Deformable ConvNets v2 (DCN2) (Zhu et al., 2019) enhances object detection and instance segmentation by adapting to the geometric variations of objects. Its reformulation improves the network's 's focus on relevant image regions through advanced modeling and modulation techniques. Additionally, feature mimicking scheme is introduced to guide network training, leading to significant performance gains on the COCO benchmark.
FT-Traasformer FT- TT-Transformer (Gorishniy et al., 2021) is a Transformer-baur based model specifically designed to handle tabular data. It employs separate feature tokenizers for numerical and categorical data, enabling the Transformer to effectively capture complex relationships between features. This adaptation improves performance in structured data tasks.
GrowNet GrowNet (Badirli al., 2020) is a gradient boosting model that uses shallow neural networks weak learners. It incorporates fully corrective step to address the greedy function approximation issue, enhancing performance in classification, regression, and ranking tasks.
MLP A Multi-Layer Perceptron (MLP) comprises multiple layers of fully connected neurons, typically including an input layer, more hidden layers, and output layer. During training, the MLP iteratively updates the connection weights between neurons using optimization techniques such backpropagation and gradient descent, aiming to minimize the prediction error and improve model generalization.
ModernNCA ModemnNCA (Ye al., 2024) is an enhanced Neighborhood Component Analysis (NCA) model that improves tabular data processing by adjusting learning objectives, integrating deep learning architectures, and using stochastic neighbor sampling for better efficiency and accuracy.
NODE Neural Oblivious Decision Ensembles (NODE) (Popov et al., 2020) is a deep learning method that combines the Oblivious Ensembles algorithm with neural networks, enabling end-to-end gradient-basee based optimization and multi-layer hierarchical representation learning for tabular data tasks.
Saint Based on the Transformer architecture, SAINT (Somepalli et al., 2021) employs an enhanced embedding method to classify features better while performing attention over both rows and columns to improve the model's focus on relevant data for tabular problems.
SNN SNN (Klambauer et al., 2017) is neural network designed to improve tabular data processing by using self- normalizing properties and the Scaled Exponential Linear Units (SELUs) activation function, enabling robust training of deep networks with layers and enhancing performance across various tasks.
SwitchTab SwitchTab Wu al., 2024) is self-supervised method that asymmetric encoder-decoder framework to separate mutual and salient features in tabular data, generating more representative embeddings for improved prediction and classification performance.
TabCaps TabCaps (Chen et al., 2023) is capsule-base based neural network architecture that encapsulates all feature values of a record into structured vectorial representations, thereby enabling collective feature processing and simplifying the treatment of heterogeneous and nigh-dimensional tabular data for enhanced classification performance.
Tabnet Tabikt(Arik & Pfister, 2021) enhances tabular data modeling by using sequential attention to select salient features at each decision step, enabling efficient learning and interpretability. It utilizes sparse attention mechanisms within decision steps to prioritize the most relevant features.
TabPFN TabPFN(Hollmann et al., 2023a) is Transformer-be based model that approximates the posterior predictive distribution for tabular data, enabling fast supervised classification with no hyperparameter tuning. It performs in-context learning. making predictions with labeled sequences without further parameter updates, and can be reused for downstream tasks without retraining.
19

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
TabR TabR(Gorishniy et al., 2024) is neural network that enhances tabular data processing by integrating k-Nearest- Neighbors-like component. It uses attention mechanism to efficiently retrieve neighbors and extract valuable information, boosting predictive performance.
Tab Transformer YabTTansformerrHuung et al., 2020) is deep learning model that leverages Transformer layers to learn contextual embeddings for categorical features and normalize continuous features, achieving higher prediction accuracy and robust performance against noisy or missing data.
tabular ResNet Tabular ResNet(Gorishni) et al., 2021) introduces residual connections by combining the original input x with its transformed version (a); meaning the output of each block is x + [(x),, which improves gradient flow and captures complex feature interactions in tabular data.
TANGOS
A regularization framework that enhances tabular data modeling by encouraging orthogonalization and special- ization of neuron attributions. TeeaaSsJeffares et al., 2023) leverages gradient attributions of neurons input features to promote focus sparse and non-overlapping features, thereby achieving diverse and specialized latent unit allocation and improving generalization performance.
Table 6: Hyperparameter Grids of Deep Learning Models.
Model
Hyperparameter
Values
N Layers
[n {8, 16, 32, 64, 128]
D Token
Residual Dropout
Uniform (0.0,
AutoInt
Attention Dropout
Uniform (..).
Learning Rate
Le
Weight Decay
Lo
N LaLeess
Int(6,32)
Dropout
Uniform {0.0, 0.2]
DANets
Base Outdim
Int(64, 128}
Learning Rate
Loguniform {e 0.1)
Weight Decay
Loguniform [c 0.001]
Num. Blocks
2, 4}
Residual Dropout
Uniform 0.2)
FT-transformer
Attention Dropout
Uniform {0.0,
FFN Dropout
Uniform {0.0, 0.5]
FFN Factor
{64, 128, 256,512)
D Embedding
Int(32,512]
Hidden D
Int(32,512)
Learning Rate
Loguniform {
GrowNet
Weight Decay
Loguniform 6 0.0.001]
Epochs Per Stage
1,
Correct Epoch
11.2
D D_layers
{1,8, 64, 512}
Dropout
Uniform (0.0, 0.5)
MLP
Learning Rate
(eguniform(e-33 0.011 0.000)
Weight Decay
Loguniform (e (e-9,0.001)
Dropout
Uniform (0.0, 0.5)
D_block
Int(64, 1024)
ModernNCA
N. blocks
(0t 2}
N frequencies
Int Int(16, (16, 96}
Continued on next page
20

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 6: Hyperparameter Grids of Deep Learning Models.
Model
Hyperparameter
Values
Frequency Scale
Loguniform 103
D embedding
Int(16, 64}
Sample Rate
Uniform (0.05, 0.6}
Learning Rate
Loguniform 0.1}
Weight Decay
Loguniform e 0.0013
Num Layers
(1,4
Depth
[nt
Tree Dim
[2,
NODE
Layer Dim
(512, 1024)
Learning Rate
Loguniform 0.13
Weight Decay
Loguniform 0.001
Depth
(4.6)
Heads
{4. 8}
Dim
{16, 32, 64)
Attn_ ttt_ourpout
Uniform (0.0, 0.5}
Saint
FF Dropout
Uniform 0.5)
Attentiontype
("colrow", "row" " col")
Learning Rate
Loguniform (3e-6.0.001
Weight Decay
Loguniform (e e-9,0.00011
D D_layers
{2, 16, 512)
Dropout
Uniform (0.0, 0.13
SNN
D_ embeedding
(nt(64,512)
Learning Rate
Leguniform(e-5 9.001 Loguniform(c- 900000)) 0.0001
Weight Decay
Loguniform00.01.1000
SwitchTai
Learning Rate
(eguniforme-9.0000
Weight Decay
Loguniform 6- 0.0011
Learning Rate
Loguniform 0.1)
Weight Decay
Loguniform (e
Sub Class
Int[1,5)
TabCaps
Init Dim
Int(32, 128}
Primary Capsule Size
Int (4 32}
Digit Capsule Size
Int [, 32}
Leaves
Int(16, 64]
Learning Rate
uniform (0.001, 0.013
?
uniform(1,
N Steps
Int(3, 10}
Tabnet
N Independent
Int [1,
N Shared Momentum
Int [1,
(iiform(0.01,,
D_main
Int, 384)
Context Dropout
uniform(0.0) 0.6}
TabR
Encoder N Blocks
Int(O, 1}
Predictor Blocks
Int 1, 2]
DropoutO
uniform (0.0, 0.6)
N Frequencies
Int 16, 96)
Continued on next page
21

---

TabFSBench:
Tabular Benchmark for Feature Shifts in Open Environments
Table 6: Hyperparameter Grids of Deep Learning Models.
Model
Hyperparameter
Values
Frequency Scale
Loguniform (0.01, (0.01, 1003
D Embedding
Int 16, 64)
Learning Rate
Loguniform (3e- 0.001]
Weight Decay
Loguniform (e 0.001]
Depth
(1, 2, 3, 6, 12)
Heads
{2, 4, 8)
Dim
{32, 64, 128, 256}
TabTransformer
Attn Att__ropout
Uniform {0.0, 0.5]
FF Drrrooutt
Uniform (0.0,0.5)
Learning Rate
Loguniform (013)
Weight Decay
(e 0.0001
N Layers
Int[1,8)
D
Int(64,512)
Hidden Factor
Uniform {1.0, 4.03
tabular ResNet
Hidden Dropout
Uniform (0.0, 0.5]
Residual Dropout
Uniform 0.5)
Learning Rate
Le 900000
Weight Decay
Loguniform (e 0.0011
layers
{1,8, 64, 512)
Dropout
uniform (0.0, 0.5)
A
Loguniform (0.001,
TANGOS
X2 Subsample Learning Rate
Loguniform (0.0001,
Int 100}
Loguniform (offfformm..0000.0..00 0.001)
Weight Decay
(e (e-,0if 0.0001
E.3. Large Language Models
Llama3-8B Llama3-8B, released by Meta AI in April 2024, a high-performance Transformer- based model. utilizes open source datasets and multiple optimization techniques for efficient training. The model integrates reinforcement learning from human feedback address multi-turn consistency, while dynamic position encoding overcomes the limitations of traditional positional encoding in processing long sequences. The development process ensures data legality and fairness.
We convert every row from tabular data into List Template for input into the LLM. The List Template is a list of column names and feature values, achieving the same outstanding performance as the Text Template according to (Hegselmann et al., 2023). We fix an arbitrary ordering of the columns. Figure uses the Iris Dataset as an example to show how we use the LLM to handle tabular tasks.
E.4. Tabular LLMs
TabLLM TabLLM (Hegselmann et al., 2023) is a framework that leverages LLMs for efficient tabular data classification. converts tabular data into natural language strings and uses few labeled examples for fine-tuning. This approach enables high performance in both zero-shot and few-shot settings, demonstrating its ability exploit the prior knowledge encoded in LLMs tabular data.
UniPredict UniPredict (Wang et al., 2023) is framework that leverages LLMs for data-efficient tabular classification. Unlike TabLLM, which uses a generic LLM directly for the task, UniPredict is the first model which is trained on multiple datasets to acquire a rich repository of prior knowledge. This approach allows UniPredict to efficiently handle diverse tabular prediction tasks, achieving strong performance both few-shot and full-data scenarios while offering scalability
22

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
island
culmen length
body mass
Species (Target)
List Template
Serialization
Torgersen
39.1
3750
Adelic
island Biscoe
Dream
46.4
3700
Chinstrap
culmen lengthh 45.1
Biscoe
45.1
5000
body mass 5000
Dataset Informatioo:
Prompt
colleted prorppegaais
Enrigiial
sincaliation
Generation
Features and their explanations
"east "his cilerreen coner the wores or doreifoes nee for the idbeds when dii wings thes
caloee kegt Thiccolane oppranse the legg cftte cateen you uppor parpuiis healy odtinnens
Predict
Train
pongaiie
ceph
"ayssr lengh The pingitis, the zttem soold repoouer the linght oritic gran' figgprr eiltramees body roc""
carota the hody ras ofti pingite grana..
LLM
benscs pengais.
Here's specifies about kind penguin:
idlan: Bhace culnee
ceclinne dipth: 14.5 Sippor leaph: 215 budy 5000, Feerada
Queation:
Ia to cac, the kad th peazan is liely to (iieeic
Figure LLM Prompt the experiments. Take the Penguins dataset as an example.
across a wide range of datasets.
F. Feature Imputation Methods
We compared the performance of various models using their own imputation methods, random imputation, and mean imputation. Table 7 provides comparison of representative models employing these three missing-featurr imputation methods. We present the average performance of several representative models under different imputation methods. Table 8 shows that the model performs best under mean imputation, and its performance declines under all three types of imputation methods, indicating that the model still faces challenges from feature shift.
G. Feature Importance Analysis
G.I. Importance Metries
To evaluate the consistency of feature importance rankings, we compute Kendall's T correlation coefficients among four metrics: Pearson correlation coefficient (PCC), Spearman's 's rank correlation, SHAP values, and mutual information. Figure 8 reveal high concordance across these measures, with particularly strong agreement between PCC and Spearman (r = 0.61). While both demonstrate comparable performance, we ultimately select PCC for its widespread adoption and intuitive interpretability in the research community. The marginal differences between these metrics' rankings were found to be statistically insignificant and did not affect our analytical conclusions.
Notably, the feature importance derived from PCC exhibits strong negative correlation with model performance degradation upon feature removal (p -0.78, p 0.01). This inverse relationship indicates that features assigned higher importance scores by PCC corresponded to greater performance declines when omitted, suggesting consistent feature dependencies across both closed-environmen training and feature shift scenarios. This observed stability likely stems from our model's robust feature selection and dynamic weighting mechanisms.
G.2. Pearson Correlation
To effectively evaluate the impact of feature shifts on model performance, we employ Pearson correlation to rank the features within a given dataset. The Pearson correlation explicitly provides the correlation coefficient between an input feature and the target variable. By applying Pearson correlation analysis to datasets with feature shifts, we can observe how the absence of features with varying degrees of correlation affects model performance. This analysis facilitates the identification of patterns and provides insights into feature shifts from multiple perspectives. Pearson correlation serves an enhancement
23

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
15
Figure 8. Visual comparison of feature importance rankings across Pearson correlation, Spearman correlation, mutual information, and SHAP values on twelve datasets.
24

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 7. Representative self-imputation methods.
Method Type
Method
Representative Models
Treat missing-fcature values as feature minima
CatBoost
Left subtree split
XGBoost, LightGBM TabTransformer Llam3-8B
Has Internal Module
dussing-fcature values as a separate category No dimension consistency requirement
No Internal Module
Imput missing features by
Most doep-lcarning models
Table 8. Methods performance comparison: model imputation methods vs. mean-value imputation vs, random-value imputation.
Shift
CaiBoost
XGBoost
TabPFN
Tab Transformeer
Llam3-8B
NAN
Mcan
Random
NAN
Mcan
Random
NAN
Mean
Random
NAN
Mcan
Random
NAN
Moan
Random
NAN
Moan
Randem
10%
0.851
0.823
0.800
0.747
0.724
0.858
0.829
0.513
0.772
20%
0.820
0.826
0.807
0.784
0.798
0.796
0.7034
0.833
0.665
0.828
0.848
0.797
0.480
0.516
0.470
0.750
0.752
0.526
30%
0.792
0.801
0.767
0.746
0.773
0,756
0.686
0.821
0.624
0.798
0.835
0.762
0.486
0.502
0.476
0.736
0.721
0.520
0.775
0.247
0.656
0.598
0.724
0.511
0.470
0.708
50%
0.733
0.747
0,7002
0.673
0.720
0.681
0.639
0.803
0.385
0.744
0.805
0.698
0.466
0.513
0.459
0.720
0.677
0.502
60%
0.702
0.716
0.665
0.639
0.692
0.655
0.616
0.792
0.580
0.694
0.784
0.656
0,482
0.510
0.472
0.704
0.683
0.546
70%
0.670
0.682
0.630
0.606
0.662
0.625
0.584
0.779
0,580
0.657
0.762
0.632
0.477
0.492
0.467
0.685
0.672
0.539
80%
0.637
0.643
0.608
0.577
0.628
0.596
0.572
0.725
0.582
0.610
0.734
0.600
0.467
0.483
0.457
0.659
0.655
0.555
90%
0.600
0.596
0.574
0.559
0.589
0.571
0.546
0.718
0.578
0.593
0.695
0.575
0.467
0.494
0.455
0.625
0.651
0.529
100%
0.538
0.541
0.502
0.561
0.545
0.548
0.559
0.621
0.551
0.557
0.638
0.356
0,467
0.482
0.454
0.575
0.622
0.177
over both Euclidean distance and cosine similarity when handling datasets with missing dimensions. The Pearson correlation coefficient quantifies the linear relationship between two variables by computing the ratio of their covariance to the product of their standard deviations. It is defined within the range [1, and is calculated as follows:
Px,y = coo(X, (X EI(X - PY - pY)]
(2)
oxOY
σxay
Here, X and Y represent two variables, cou(X, Y) denotes their covariance, and ax and ay are their respective standard deviations. If the Pearson correlation coefficient (PCC) px,y equals O, it indicates no linear relationship between X and Y. A positive correlation (0- < Px,y I 1) implies that as X increases, also increases. Conversely, a negative correlation (-1 S Px.y < 0) signifies that as X increases, decreases. The closer | is to 1, the stronger the linear relationship between the two variables.
G.3. Automated Feature Engineering Analysis
We posit that Automated Feature Engineering (AutoFE) can address feature shift scenarios through two approaches.
Imputation: CAAFE (Hollmann et al., 2023b) and OcTree (Nam et al., 2024) generate rules for imputing specific features, while OpenFE's generated features do not match the originals (Zhang et al., 2023). We test single shifts using CAAFE and OctTree for imputation. Table shows that LLM-bbsed AutoFE can effectively generate matching features, enhancing model robustness compared to mean imputation.
Generation: We use AutoFE generate new features to offset the impact of missing original features on model performance. Table 10 shows that LLM-based AutoFE has significant potential in feature shift scenarios.
As the importance of features increases, both model performance declines more significantly, further corroborating Observation 2 of TabFSBench.
H. Experiment Details of Observations
H.1. Training Details
The deep learning models, LLMs, and Tabular LLMs were trained on an NVIDIA A800 GPU. Gradient-boosted tree models, where applicable, were trained on a CPU rather than a GPU, using an AMD Ryzen 5 7500F 6-Core Processor. All experimental results are reported as the average of three different random seeds to ensure robustness.
25

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table Results from AutoFF imputation strategy. We evaluate CAAFF and OcTree the downstream model of CatBoost and
TabPFN under the Single Shift experiment.
CatBoost
TabPFN
Shift
Baseline
CAAFE
OcTreee
Baseline
CAAFE
OcTrree
1
0.870
0.869
0.870
0.866
0.875
0.873
2
0.871
0.880
0.872
0.862
0.877
0.872
3
0.864
0.874
0.872
0.859
0.876
0.877
4
0.861
0.865
0.864
0.861
0.868
0.874
5 0.875
0.882
0.885
0.866
0.861
0.868
0.859
0.859
0.863
0.857
0.883
0.875
7 0.844
0.854
0.853
0.853
0.865
0.868
8
0.866
0.871
0.868
0.873
0.845
0.852
9
0.871
0.875
0.871
0.868
0.871
0.866
10
0.864
0.874
0.864
0.871
0.875
0.874
11
0.663
0.664
0.666
0,817
0.868
0,872
100%
0.464
0.832
0.830
0.518
0.865
0.857
H.2. Different Types of Feature Shift
We address the analysis that compare model performance on different types feature shifts by clarifying two distinct analytical perspectives on feature shifts. First, regarding different types of feature shifts, we direct attention to our comprehensive robustness evaluation across multiple shift categories presented in the leaderboard analysis. Second, examining shifts by feature type, Table II reveals differential sensitivity patterns: model performance demonstrates greatest vulnerability to categorical feature perturbations, followed by boolean features, with numerical features exhibiting the most stable behavior. This hierarchy of susceptibility persists across all tested datasets, suggesting inherent algorithmic dependencies on feature types.
H.3. Removing Most/Least Relevant Features
As shown in the experimental results in Figure H.3, under the least relevant feature removal setting, the model's performance remains largely unchanged when removing 0 ~ 5% of the least relevant columns. Interestingly, we observe that removing a certain number of the most irrelevant columns improves model performance. This observation aligns with the insight from (Grinsztajn et al., 2022) that uninformative features can negatively impact model performance.
H.4. Detailed Explanations of Figure 4
The reason for the presence of multiple trajectories in Figure is that each trajectory comprises points derived from a single dataset. During the fitting process, the results from all datasets were aggregated, leading to the emergence of distinct trajectories corresponding to individual datasets. Notably, these trajectories capture the relationship between feature importance and model performance across different datasets. Specifically, each dataset's trajectory illustrates how its particular pattern of feature importance influences the resulting model performance.
To better support our conclusions, Figure H.4 provides correlation and accuracy plots for twelve different datasets. It indicates that although levels of feature importance vary across different datasets, they all support our research conclusion, namely, that there is a significant linear correlation between feature importance and model performance. These trajectories further confirm the generality and reliability of Observation 2.
H.S. Results on High-Dimensional Datasets
Following OcTree (Nam et al., 2024), we conducted random shift experiments on two high-dimensional datasets, madelon and nomao, and present model performance under 10%, 20%, and 100% feature shift degrees. Table 12 and 13 demonstrate that as the degree of feature shift increases, the model performance decreases significantly.
26

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 10. Results from AutoFE. feature generator. We evaluate OpenFF, CAAFF and OcTree on the downstream model of CatRoost and TabPFN under the Random Shift experiment.
CatBoost
TabPFN Baseline
Shift
Baseline
OpenFE:
CAAFE
OcTrree
Baseline
OpenFR
CAAFE
OcTree
0.879
0.886
0.887
0.888
0.862
0.853
0.863
0.860
9%
0.851
0.859
0.861
0.859
0.859
0.871
0.874
0.876
18%
0.826
0,857
0.859
0.857
0.848
0.866
0.876
0.872
27%
0.801
0.855
0.862
0.855
0.835
0.866
0.872
0.870
36%
0.775
0.853
0.860
0.853
0.820
0.864
0.874
0.870
45%
0.747
0.841
0.849
0.841
0.803
0.868
0.878
0.862
54%
0.716
0.842
0.845
0.842
0.784
0.851
0.861
0.861
63%
0.682
0.835
0.842
0.835
0.762
0.853
0.860
0.869
72%
0.643
0.842
0.844
0.842
0.734
0.846
0.857
0.857
81%
0.596
0.850
0.853
0.850
0.695
0.857
0.860
0.867
90%
0,541
0.841
0,850
0,841
0.638
0,856
0,860
0.863
100%
0.464
0.830
0.832
0.830
0.518
0.853
0.865
0.857
The madelon dataset originally designed for the NIPS 2003 feature selection challenge. It artificial binary classification task in which instances generated from clusters positioned the vertices of a five-dimensional hypercube. The dataset contains 500 features, including 20 informative (or redundant) features and 480 noise features, making it prototypical benchmark for evaluating robustness and feature selection under high digenniinaii settings. The dataset is available from the UCI Machine Learning Repository: https://archive.ccsuuii..k wttp://....eeddadaedeeee/eee/ /maaeeon..
Madelon
The nomao dataset originates from real-world deduplication task involving geolocated business entries. Each sample represents a comparison between two records, described by both numerical and categorical similarity features such as name, address, and geocoordinates. The dataset includes 34,465 samples and 118 features (89 numeric and 29 categorical), with notable proportion of missing values. Its mixed-type, noisy, and partially incomplete nature makes valuable resource for studying model robustness in practical high-dimensional scenarios. The dataset is publicly accessible at http:/cchiiiii.......ddtttaaaeea
Nomao
H.6. Feature-Shift Performance Runtime
In Figure H.6, plot feature-shiff performance versus runtime for all models, averaged across datasets. LLMs and Tabular LLMS exhibit the longest runtime, whereas tree-based models achieve excellent performance with minimal runtime requirements. However, we observe that models such as NODE and LLMs consume significantly more memory compared to tree-based approaches.
H.7. Dataset-Specific Experiment Details
This section provides detailed dataset-specific experimental results. For each dataset, we list all models and their corre- sponding performance metrics. The experiments are categorized into four scenarios: SC (single-column missing). MC-D (multi-columm missing in Pearson correlation descending order), MC-A (multi-column missing in Pearson correlation ascending order), and MC-R (multi-column missing randomly). For classification tasks, we report accuracy and ROC-AUC scores, while for regression tasks, we provide RMSE results.
27

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 11. Results on different types of features shifted. We divide feature types into Categorical, Numerical and Boolean.
Model
Baseline
Categorical
Numerical
Boolean
LightGBM
0.833
0.786
0.83
0.821
XGBoost
0.855
0.804
0.84
0.81
CatBoost
0.879
0.792
0.868
0.861
TabPFN
0.862
0.845
0.865
0.864
DANets
0.674
0.648
0.667
0.661
MLP
0.848
0.837
0.848
0.838
NODE
0.663
0.651
0.641
0.637
ResNet
0.857
0.845
0.86
0.851
SwitchTab
0.871
0.858
0.867
0.859
TabCaps
0.775
0.76
0.772
0.763
TabNet
0.681
0.659
0.678
0.654
TANGOS
0.859
0.853
0.863
0.86
AutoInt
0.683
0.678
0.673
0.696
DCNv2 0.868
0.854
0.861
0.841
FI-Transformer
0.774
0.734
0.773
0.768
GrowNet
0.589
0.586
0.598
0.591
Saint
0.875
0.861
0.866
0.861
SNN
0.788
0.775
0.78
0.776
TabTransformer
0.522
0.504
0.523
0.508
TabR 0.893
0.877
0.893
0.896
ModernNCA
0.88
0.849
0.881
0.889
Llama3-8B
0.848
0.76
0.805
0.765
TabLLM 0.783
0.786
0.795
0.766
UniPredict 0.853
0.861
0.849
0.84
Table 12, Results nomao dataset. We don't don't evaluate TahPFN(due to the limitation of TabPFN high-dimension datasets which have more 100 features), Jlama3-8B Unipredict (due the limitations of input).
ACC
ID
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LightGBM
0.970
0.969
0.968
0.969
0.967
0,964
0.958
0.961
0.954
0.926
0.890
XGBoost
0.974
0.974
0.973
0.973
0.972
0.967
0.971
0.964
0.933
0.939
0.874
CatBoost
0.970
0.969
0.969 \
0.969
0.969
0.960
0.960
0.960 \
0.955
0.945
0.892
TabPFN
\
DANets
0,933
0.930
0.921
0.908
0.888
0.872
0.845
0.816
0.808
0.719
0.713
MLP
0.949
0.946
0.938
0.931
0.914
0.894
0.884
0.847
0.874
0.762
0.756
NODE
0.778
0.768
0.760
0.750
0.723
0.716
0.713
0.713
0.713
0.713
0.713
ResNet
0.953
0.946
0.936
0,929
0.882
0.852
0.815
0.777
0.824
0.719
0.714
SwitchTab
0,936
0.941
0.935
0.931
0.911
0,886
0.874
0.847
0.867
0.767
0.753
TabCaps
0.920
0,916
0.905
0.897
0.899
0.882
0.876
0.852
0.864
0.832
0.804
TabNet
0.921
0.899
0.907
0.900
0.822
0.878
0.875
0.806
0.876
0.715
0.756
TANGOS
0.955
0.950
0.937
0.930
0.904
0.874
0.869
0.816
0.881
0.760
0.736
AutoInt
0.939
0.934
0.912
0.901
0.862
0.823
0.759
0.757
0.771
0.718
0.714
DCNv2
0,949
0.946
0.938
0.930
0.916
0,903
0.899
0.859
0.881
0.772
0,806
FT- Transformer
0.934
0,933
0.925
0.909
0.860
0.859
0.801
0.772
0.790
0.714
0.713
GrowNet
0.832
0.818
0.812
0.797
0.779
0.785
0.759
0.741
0.759
0.715
0.729
Saint
0.952
0.943
0.928
0.915
0.852
0.842
0.801
0.762
0.792
0.718
0.713
SNN
0.910
0.909
0.904
0.890
0.889
0.880
0.885
0.859
0.857
0.837
0.798
TabTransformer
0.928
0.925
0.915
0.905
0.884
0.879
0.871
0.852
0.877
0.802
0.779
TabR
0.915
0.913
0.904
0.894
0.894
0.881
0.881
0.855
0.860
0.834
0.801
ModernNCA
0.964
0,952
0.945
0.928
0.842
0.861
0.842
0.768
0.812
0.719
0.713
Llama3-8B
\
\
TabLLM
0.959
0,957
0.938
0.893
0.856
0,842
0.616
0.676
0.506
0.432
0.329
UniPrediet
\
28

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Mitbbeas
Kycnsomeess
Digee
Progdes
Nibe
Cauntr
Lusto
100
Teatre 1
Tsstes Do.ges
Mari Ravelesz
Rassoro
Razarre
Figure Results for most/least shift experiments correlation analysis across datasets. The r-axis represents percentage features removed. while the y-axis indicates the average performance (accuracy or RMSE) of all models. Three feature removal strategies are compared: Remove Most Relevant (bluc), which removes & most correlated columns; Remove Least Relevant (green), which removes & least correlaeed columns; and Remove Random (yellow), which removes & randomly selected features.
29

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Credr
Duo point
Data paint
Uiear T3
Elbwu Fi p- E3300
p-000007
p- 6.604
04-
Coefaree Itaenal
04
19% Crefidene Itann
Modl Perfmmmee Cupp
04-
95% Lontidence heaval
Model Peromazee
Mode
62
0.3
-0.1
25
16
-03
05
-01
02
25
15
Sum of Shiflel Feature Set's Corelations
Sum of Shified Fostare Sets Correlatinss
Sam of Shiled Forure Sels Corrdssione
Minibann
Irie
00
palz: Liaa
2-10962
Modd Porfomman Gap
Mode Performnne
0.3
0.3
0.2
-0.1
0.0
65
1a
20
-0.
0.9
85
15
24
-0.1
0.
35
18
14
20
3.0
Sum of Shifled Feature Set's Comelations
Sum of Shifial Foriie Sefs Comclations
Sum of Shiflel Fezuro Se's Coerdbtions
Penguits
Jannis
Abalone
0.5
0.5
85
Mocel Porfocmencc
Petoemunce
0.1
Sum of Shifol Fenture Sel Correlatints
Sum of Shilaa Feature Sef' Creneiatinss
Sun of Shifle Festure Sala Coneluions
Dea
Dkis
Liven Fi
Line p-83500
Liaari
04-
aorcooe Itannl
W9CCCefiiee heaval
Modil Pafferome Gop
044
ot's Cieidtence beaval
Model Perfeemace
60
-01
-01
30m of Shitled Feature Set's Comvlations
Sum of Shifted Festure Sets Conelation
Sum of Shitted Festane Set's Cocrestions
Figure 10. Trajectories between correlation and accuracy for each of twelve datasets.
30

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 13. Results madelon dataset. We don't evaluate (du to the limitation of TabPFN on high-dimension datasets which have more 100 features), Llama3-8B, M and Unipredict (due to the limitations input).
ACC
ID
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LightGBM
0.819
0.813
0.815
0.819
0.825
0.796
0.737
0.723
0.635
0.574
0.562
XGBoost
0.828
0.818
0.827
0.804
0.829
0.769
0.772
0.693
0.614
0.560
0,554
CatBoost
0.865
0.859
0.797
0.862
0.831
0.836
0.770
0.786
0.630
0.547
0.841
DANets
0.523
0.523
0.524
0.514
0.513
0.510
0.509
0.502
0.503
0.496
0.495
MLP
0.518
0,521
0.521
0.515
0.528
0.519
0.511
0.506
0.507
0.503
0,498
NODE
0.507
0.512
0.515
0.500
0.526
0.510
0.505
0.494
0.501
0.499
0.495
ResNet
0.555
0.553
0.553
0.535
0.537
0.547
0.536
0.516
0.529
0.507
0.501
SwitchTab
0.483
0.483
0.483
0.483
0.483
0,483
0.483
0.483
0.483
0.483
0.479
TabCaps
0.516
0,509
0.515
0.508
0.516
0.514
0.490
0.504
0.503
0.502
0,498
TabNet
0.512
0.504
0.504
0.507
0.505
0.502
0.496
0.500
0.497
0.496
0.490
TANGOS
0.571
0.560
0.562
0.551
0.566
0.545
0.550
0.523
0.525
0.511
0.504
AutoInt
0.592
0.558
0.543
0.543
0.532
0.530
0.526
0.519
0.511
0.503
0.501
DCNv2
0.528
0.522
0.523
0,509
0.519
0,512
0.528
0.512
0,508
0.513
0.507
FT- Transfffomer
0.640
0.579
0.553
0.554
0.546
0.541
0.540
0.536
0.522
0.521
0.517
GrowNet
0.506
0.503
0.494
0.496
0.499
0.508
0.509
0.502
0.517
0.509
0.506
Saint
0.544
0.543
0.541
0.539
0.536
0.528
0.524
0.523
0.506
0.503
0.501
SNN
0,508
0.507
0.503
0.498
0.508
0,506
0,513
0.501
0.493
0.505
0,502
TabTransformer
0.543
0,544
0.534
0.532
0.548
0,540
0.534
0,513
0.523
0.520
0.514
TabR
0.640
0.620
0.556
0.538
0.534
0.525
0.520
0.511
0.505
0.506
0.503
ModernNCA
0.593
0.587
0.573
0.549
0.588
0.553
0.539
0.519
0.516
0.505
0.502
Llama3-8B
TabLLM
\
V
UniPredict
\
10'
10'
Runtime
10'
0.600
0.635
0.650
0.675
0.750
Model Performance in Feature Shift
LightGBM
NODE
-X-
Autolat
Tab Transformer
XGBooot
ResNet
DCN22
TabR
CatBooot
SwitchTab
FT- Traneffomer
ModeraNCA
TabCaps
GrowNet
Llama3-8B
TabNet
MLP
TANGOS
SNN
UniPredict
Figure 11. Average feature-shift performance average runtime for each model.
31

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 14, Model performance on Credit datasct.
(a) Model performance of SC experiment under accuracy on Credit dataset.
Coluras
PCC
1.G8
XGB
CATR
PFN
DAN
MLP
No06
R15
CAP
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTP
TABR
NCA
TMA
T1L.M
UNI
AVE
-0.00
0.755
0.773
0.753
0,685
0.92
0.693
0.085
0,675
0.540
0.695
0.687
0.695
0.007
0.655
0.543
0.537
0.697
0.713
0.645
0.500
0615
0.61
0.06
0.233
0050
0.772
0235
0.85
0.622
0.615
11990
0660
01525
0.80
00.8
0895
0.60
11:541
0.345
0880
0.215
01800
0M6
06:11
3
-0.000
0.733
0.718
0.80
0.763
0.085
0.87
0.685
0.682
0.08
0.67
0.583
0.88
0.683
0.85
0.655
0.507
0.685
0.533
0.578
0.63
0.713
0.645
0.595
0,655
0.63
-0019
0.734
0.757
0.772
0.233
0.955
0,991
0.92
0.680
11993
0450
03542
0,805
05588
0.682
0666
0.61
0.655
11533
0.527
0.703
0665
0.50
0634
0.028
0.750
0.750
0.780
0.237
0.655
0.95
0.633
0.600
0.005
0.688
0,522
0.895
06500
0.67
0656
0.65
0.665
01580
0.537
1660
0.713
0.700
0555
0600
0.7
0.055
0.78
0.21%
0.99
0983
0981
0ATS
0004
0814
0557
0.88
0167
0661
088
0530
0.85
0111
0.112
0668
0221
0981
0:99
0410
067
0036
0.730
0.743
0.722
0.757
0,65
0.97
0.68
0,05
0.682
0.577
0.655
0578
0,82
0695
0.22
0.685
0.528
0685
0.003
0.60
0.75
0,595
0.09
6.544
0.755
0.757
0.05
0.97
0.695
0,055
0,675
0.522
0,655
0.82
0,695
0617
0.655
0.545
0.68
0.71
0.655
0.500
0.595
0.71
10 0026
0.42
1.365
0757
0/92
04633
0A42
1190
0436
01525
0825
0415
080
0.65
11.44
0.44
0.22
0525
0840
0872
11 400/
0.232
0.65
0.39
0.31
0/93
0.63
0.65
11915
0641
0.52
0,95
0.581
0.85
0615
0,65
01552
0.325
06800
0.15
0675
0530
0621
06/0
12
.0083
0.75
0.747
0.75
0742
0:91
0681
0.611
0.485
0.61
0473
0.55
0.01
0.68
0681
0494
0517
0.855
0.515
0.76
0711
0.600
0.585
0405
060
13 6.000
0.745
0.722
0.725
0.33
0/03
0.63
0.65
1.005
0453
0543
0,95
1677
0.85
0416
0665
0.65
11542
0.42
1660
0.715
0.725
0.595
0610
0672
14
0.911
0751
0.71
8171
0210
0982
0.81
0.85
0413
8521
0881
0181
0009
0M0
0.61
0.78
870
0221
6715
0190
046
R67
15 .0097
0.73
6.42
6777
0.58
0666
0.67
0692
0.67
0,95
0.68
0585
0.05
0682
0.03
0696
0613
0.65
0.520
0,535
0.88
0.11
0.50
0.59
8625
0.71
0.808
0.73
6.733
0.07
0,770
0,65
0693
0,05
0,675
0.53
0.685
0.68
0,65
0.22
0.655
0.515
0,68
0.223
0.65
0.505
0610
0.67
0.155
0.745
0.745
0.56
0.752
0:08
0,99
0.695
0.085
0.633
0.513
0.695
0.77
0,695
0.30
0.685
0.540
0.000
0.725
0.650
0,500
0.67
19
40215
0.233
0.16
0.233
0.233
(186
0.61
0.62
0.62
1196
0672
0.52
0,95
0.63
0.65
0815
0.22
0.69
01558
0.347
06/75
0.231
0.595
0.540
0860
30
0.300
0.727
0.645
0.35
0.718
0.095
0.881
0.691
0.678
0.085
0.675
0.50
0.685
0.683
0.81
0.695
0,07
0.685
0.50
0.532
0.66
0.683
0.695
0.485
0.615
0.603
(b) Model performance of SC experiment under ROC-AUC on Credit datasct.
Colaass
PCC
LGB
CATB
PWW
DAN
NO0E
RE5
5W]
CAP
NET
005
DCN
5.11
TTF
TABR
NC5
TLLM
AVE
0.761
0.755
0.799
0.781
0.486
0.504
0/644
0.697
0.099
0.650
0.69
0.592
0.538
0.391
0.576
0.470
0.733
0.551
0.525
0.691
0.755
0.701
0.590
0.642
0.634
-0.006
0.799
0.182
11304
0.782
[1483
0.504
0.441
0.666
11,93
0.633
(.69)
0.395
[1345
0.394
0.368
0.670
6.266
[1.663
0.330
[1897
0.230
06615
0.318
06611
0.629
0.003
0.703
0.715
8.7
0.71
0.487
0.510
0.641
0.488
0.000
0.655
0.881
0.989
0.959
0.61
0.563
0.220
6.768
0.166
0.51
0.607
0.51
0.569
0.168
0.64
0.67
-0.009
0.754
6.346
0,305
0.784
0.471
0.433
0.45
0.62
0.70
0.60
0.555
0.518
0.532
0.587
0.578
0.478
0.333
0.564
0.527
0.501
0.357
0.536
0.527
0.677
0.625
-0019
0.767
0:995
0.531
0101
0.67
0.00
0.166
0.766
0.909
0.000
0715
0.67
6,033
0,702
0.756
0.500
0.781
0.45
0.492
0/40
0.698
0.704
0.653
0.683
0.576
0.585
0.566
0.46
0.733
0.58
0,515
0.115
0.756
0.50
0.42
0.627
0.066
0.000
0355
0.75
11486
0,211
0AM
0.00
1198
0450
01482
0.316
13547
01591
0.344
0.484
0.729
11554
0.541
0250
0.361
0633
8
0.966
0.766
0.700
0503
0.782
0.60
0,535
0.655
0.689
0.0
0.551
0.53
0.601
0.557
0.585
0665
0.66
0.759
0,50
0.55
0,00
0.761
0.553
0.500
0/631
0.63
9 0024
0.281
0.726
0.782
(1477
0,500
0635
0.911
11995
0450
01501
0.39
11565
0.85
0312
04472
6:229
11555
0.449
0.249
0336
0351
0815
10
0.06
0.75
0.717
0.768
0.782
0.889
0.881
0.427
0.689
0.001
0667
0.663
0.511
0.55
0.00
0.513
0.54
0.31
0.611
0.544
0.03
0.51
0.558
0467
0.6
11 -0073
077
0774
6.M
0.267
0.779
0.11
0A4
0.81
01691
0615
0.191
0.566
0131
030
0.177
0:71
6.75
0.61
0.53
026
0:997
0157
0660
0.%
12 .0662
0.756
6.751
0.299
0.776
0.79
0,985
0640
0.655
0,696
0,640
0.490
0.544
0,83
0.57
0.444
0.721
0.525
0.66
0.553
0.56
0527
0652
0.24
6,011
0.714
0.59
0.766
0.47
0.501
0.705
0,650
0.699
0.991
0.555
0.75
0.5%
0.746
0,532
0317
0.755
0318
0.25
15 40097
0.739
07714
11584
0369
1487
0.70
0.41
0.60
1007
0411
0.49)
0.54
0.565
0.54
0.88
0.59
6:20
1.54
0.31
1196
0262
0.55
0347
0M6
062
16 0.86
0.345
0:40
0.39
0751
0483
0,01
0.46
0061
0198
0432
0.28
0.91
0.23
0.8
0376
0.77
6212
12577
0.21
11977
0269
0.51
0357
0842
0825
17
0.138
0.78
0.744
020
0755
0.570
0.433
0/67
0,677
0.79
0413
0.677
0.359
0.54
0.75
0.565
0.96
6.70
0.53
0.585
0662
0.30
0.500
0,500
0.50
064
18
-0155
0.754
6:757
0.008
0.699
0.45
0.515
0642
0.67
0.63
0450
0.88
0.996
01588
0.79
0.71
0.41
6.211
01563
0.325
0.83
0.239
0600
0306
0624
0.6)
-0315
075
0.76
0707
0.766
0.685
0.081
0.63
066
0.13
0531
0.516
0.9
0477
0.79
0.510
0.778
0.51
0550
8168
0427
0.00
20
0.302
0.680
0.679
0.712
0.692
0.77
0.414
0.639
0.650
0.64
0.5%6
0.500
0.546
0.537
0.568
0.550
0.72
0.688
0.538
0,519
0.63
0.090
0.567
0.548
0.60
0,515
(c) Model performance of MC-M experiment under accuracy on Credit dataset.
Ftegree
14GB
XC3I
[CATII
PIN
BAN
MIP
NOII:
SW1
CAP
NET
.G35
INT
1XCN
FTT
(RRD)
SAT
SNN
TT"'
TABR
NEA
1.M
TLLM
UN
AVI:
0
0.257
0.250
0.775
0.262
0,695
0.693
0.693
0.690
0.995
0.688
0.615
0.585
01.65
0.505
0.608
0.605
01.37
0.328
0.087
0.217
0.250
0.600
0.620
0.676
10%
0.712
0,700
0.723
0,997
0.692
0,675
0.055
0,653
0.562
0.585
0.655
0.585
0.645
0.605
0.675
0,702
0.645
0.585
0.000
0.657
15%
0.727
0.45
0.705
0.230
0.695
0.622
0.695
0.665
0.85
0,686
0.57
0.69
0.887
11675
0.691
0.691
01.07
0.353
0.67%
0.683
0.505
01585
01500
0A51
20%
0.725
0.537
0.692
0.728
0.615
0.92
0.693
0.533
0.055
0.633
0.570
0.655
0.682
0.675
0.595
0.65
0.605
0.477
0.553
0.075
0,697
0.585
0,500
0,500
0.648
25%
0.712
0.475
0.665
0.730
0.695
0.655
0.695
0.655
0.005
0.638
0.570
0.605
0.687
0.672
0.695
0.63
0.605
0.667
0.553
0.600
0.712
0,590
0.505
0.525
0.643
30%
0.466
0,50
02215
0.95
0.69
0.05
0.16
0376
0660
0.811
01653
0.00
0.60
0,60
0.463
0.547
08871
0.666
0350
0183
(1330
35%
0.73
0.518
0.55
0.788
0.685
0.97
0.695
0.65
0.65
0.67
0.57
0.65
0.683
0.63
0.695
0.60
0.695
0.480
0.50
0.61
0.688
0,00
0.500
0,40
0665
40%
0.723
0,450
0567
0.718
0.95
0,63
0.69
0.26
0.95
0.597
0.91
0.66
0.755
0.577
0.505
0.67
0,666
0.480
0.62
0.05
0.68
8405
0.29
01565
0.44
0.500
50%
0.20
0.05
0.562
0.585
0.690
0.605
0.66
0.07
0,500
0.51
55%
0.720
0.643
0.722
0,95
0.693
0,617
0.055
0.543
0.15
0,595
0.650
0.585
0.62
0.693
0.600
0.555
0.673
0,693
0,000
0.000
0.55
0.60)
AUS
0.717
0.77
0.70
0.25
0.695
0.695
0612
0895
0.540
066
0.81
0182
0.6
UM6
0,69
(1:57
0.51
0.61
0.695
1,80
0155
0810
0441
65%
0.730
0.528
0.682
0.73
0.695
0.97
0.695
0.60
0.655
0,540
0,00
0.66
0.685
0.62
0.685
0.65
0,605
0.687
0.557
0.645
0,695
0,005
0.585
0555
0.64
30%
0.721
0,82
0875
0.255
0.95
0.69
0.90
0.05
0.25
086
0,90
0.00
0.833
0,00
(1475
0.322
0.09
01200
01300
BAW
75%
0.723
0.487
0.585
0.25
0.685
0.85
0.695
0418
0.65
0.533
0.008
0,95
0.692
0.68
0.595
0.66
0.695
0.687
0.535
0.07
0.695
0,10
0.500
0.50
0641
105
0.731
0.68
0.687
0735
0.91
0,85
0.895
0453
0685
0.533
0.08
0A9
0.591
067h
0.591
0663
0.491
0:481
0.538
0.76
0.991
0,00
0.50
0625
0.55
85%
0.723
0.713
0.68
0.733
0.995
0,95
0.695
0,62
0,95
0.523
0,82
0.66
0.687
0.622
0.505
0.65
0.605
0.507
0.518
0.05
0,695
0,000
0.595
0.500
0652
95%
0.695
0,61
0.055
0,512
0,505
0.637
0.685
0.95
0.605
0.500
0.05
0,695
0.500
0.585
0515
0.52
100%
0.725
0.725
0.721
0.25
0.695
0.685
0.695
0.565
0.85
0.368
0.595
0.60
0.595
01685
0.605
0185
0.688
01565
0.365
0.895
0.688
01.05
0.575
(1568)
0A51
(d) Model performance of MC-M experiment under ROC-AUC on Credit dataset.
Degioe
XGB
CATB
MLP
NODE
RES
SW1
CAP
GO5
INT
DCN
FTT
GRO
TABR
NCA
TLLM
UNI
AVE
0
0.761
0.755
0.790
0.751
0.196
0.501
01611
0.697
0.689
0.6%)
0.195
0.590
0.554
11.011
0.576
01470
0.313
(1:19
0.573
0.601
0.355
0.701
(1500
01642
0.31
5%
0.680
0.670
0.712
0.692
0.47
0.494
0.609
0.500
0.640
0.5%
0.530
0.546
0.537
0.558
0.550
0.422
0.688
0.535
0.519
0.630
0.600
0.565
0.536
0.635
0,593
10%
0.665
0,686
0.677
0.72
0,578
0.775
0.351
0.61
0.31
0.38
0831
0.325
0.52
11511
0.35
1.277
0.660
(1829
0.524
0.663
(1566
01114
11685
0.384
15%
0.662
0.668
0.641
0.642
0.886
0.091
0.579
0.611
0.602
0.587
0.522
0.522
0.50
0.556
0.527
0453
0.651
0.524
0.515
0.601
0.655
0.556
0.510
0,650
0.574
2076
8181
0821
0618
0.466
1.415
0.46
0.596
0.665
0.364
8319
0,487
0.51
0550
0.114
0.822
0.44
0.81
01855
0.338
15%
0.600
0.00
0.14
0.609
0.516
0.581
0.555
0.568
0.114
0.31
0.534
0.511
0.74
0.591
0:58
0.597
0.00
0.50
0.11
0,661
0.509
0.559
0.90
35%
0.589
0.564
0.562
1.444
0.348
0,500
0.590
0,033
0.55
0.47
0.515
0.493
0.5%6
0.556
0.003
0.362
0.56
0.522
0.540
40%
0.51
0.75
0368
01550
0.11
0.19
0.5%%
0.561
0511
0.5%
0.409
0554
11409
0.50
0.500
11477
0361
03966
8766
0351
01500
0.500
45%
0.562
0.557
0.563
0.5%%
0439
0.453
0.550
0.50
0.15
0.550
D465
0.49
0.25
0456
0,511
0.505
0.556
0.47]
0.338
0,533
0.38
0.40
0.506
071
0.524
30%
0.40
0.114
0.21
0.365
0,500
B-755
0.59
0.52
DSES
0.544
0.722
0.46
0550
0,112
1112
0.587
0.024
0.566
0.576
8511
0150
11704
0.32
25%
8519
0311
0.22
0.00
0439
1.255
0.333
0353
0.311
DA2
0359
11611
0.30
11903
0.372
0:35
0326
0.30
1.995
(1221
11097
0:15
40%
0.16
0.539
0.117
0.500
0.51
0.419
0.558
0.500
0.518
0.550
DJ81
0.466
0.65
0.55
0.597
0434
0.588
0.663
0.546
0.503
0.553
0.99
0.50
0.76
0.14
65%
0.535
0.31
0.34
0.544
6457
0.482
0.347
0.524
0.525
0.544
8.665
0.442
0545
0.64
0.534
0.70
0.540
0.47
0.341
0.95
0.566
0.433
0500
0718
0.520
0.976
0516
0353
0.91
07M
15%
0.350
0.34
0.560
0.511
80%
0.119
0.58
0.53
0.557
6356
0.523
0.533
8425
0.315
0.47
0436
0.565
0.07
0.530
0.548
0.00
0.364
8.490
0.566
07354
0.500
35%
0.50
0.56
0351
0.515
0.651
0,322
0351
0197
0.515
8/26
0,000
0515
01415
0.52
11:4
0.462
(1884
0361
0.4M
01300
01304
0.58
50%
0,665
0.466
0,422
0.529
0,62
0.475
0.4%
0.60
0.67
0,517
0.53
0.455
0.52
0.47
0.539
0.115
0.400
0.680
0.12
0.69
0.453
0.49
0.507
0.68
0.44
95%
0462
0.822
04524
11526
0.995
0.422
0469
0.000
0.485
0.889
0.40)
0.490
01214
0.122
[1988
0.430
0.800
0.495
0.4M0
0.00
U111
0.495
100%
0.503
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0,500
0.500
0,500
0.500
0.500
0.500
0,500
0.564
0,53
0.502
32

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(c)
Model performance of MC-L experiment under accuracy on Credit datasct.
Degoes
LGB
XGB
CABB
PPN
DAN
MLP
NO0E
RES
SWI
CAP
NET
GUS
INT
FIT
CRR
5/41
SNN
TFF
TABR
NCA
LMA
VLLM
UNI
AVE
0.750
0.62
0.680
0.005
0.685
0.585
0.658
0.71
0.750
0.630
0.630
0.676
10%
0.755
0.755
0.775
0.768
0.695
0.992
0.690
0.668
0.685
0.678
0.537
0.685
0.681
0.678
0.591
0615
0.691
0.551
0.138
0.015
0.710
0.690
0.500
0615
0.600
15%
0.743
0.250
0.772
0.760
0,695
0.65
0,655
0.77
0.95
0,672
0.576
0.65
0.590
0.590
0.605
0.007
0,605
0.560
0.543
0.681
0,310
0,535
0.515
0.955
0,66
705
0.717
0.55
0.787
0.755
0.095
07200
0.691
0475
0.085
0.667
0.565
0.69
0.577
11200
0.591
0.60
0.691
0.577
0.538
0.601
0,30
0.530
0.51
0550
25%
0.725
0.35
0.781
6.70
0.095
0.66
0.62
0481
0.08%
0.62
0.777
0.69
0.881
11.08
0.611
0.00
0.61
01508
0:532
0.80
0.76
0.88
11600
308
0.735
0.732
0,700
0.735
0,655
0.97
0.683
0.685
0.955
0.633
0.565
0.65
0.585
0.68
0.685
0.005
0.695
0.565
0.535
0.703
0.706
0.555
0,000
0.225
0.60
35%
0227
0.91
0.286
0.95
1868
046
0A8)
0.84
1157
0.61
0,6
01565
0-22
0999
0250
40%
0.78
0.70
0.697
0.085
0.680
0.677
0.658
0.585
0.25
0.57
0.65
0.70
0.50
0.500
0610
0.000
0852
0.00
50%
0.60
0.61
0.50
0,000
0615
0.6
59%
0.713
0.30
0.774
0.56
0,87
0.691
0.600
0.65
0.668
0.007
0.699
0.581
0617
0.591
0993
0.691
0.517
0.118
0.64
0.705
0.514
0.50
0670
0.66
60%
0.733
0.735
0.775
0.343
0,695
0,702
0,655
0.66
0,655
0,670
0,915
0.65
0.685
0.500
0.605
0.15
0,605
0.585
0,515
0.670
0.701
0.575
0.905
0.25
0.677
65%
0.71T
0.30
0.761
6.78
0.095
0.695
0.60
0.68%
0.668
0,20
0.69
0.74
11677
0.411
11881
0.91
01565
0:522
0.67
0.600
1.595
0610
0.500
6A6
708
0.747
0.728
0.720
0.753
0,695
0,65
0.695
0.64
0.995
0.677
0.627
0650
0580
0.00
0.685
0.587
0,695
0.590
0.523
0568
0,700
0.665
0,590
0.000
0.72
75%
11733
0215
0,732
0.232
0.99
0464
0.85
0.68
0600
0861
166
0.61
0.6
0.49
(1883
0:355
0.83
02995
0A72
40%
B.729
0220
0/721
0:40
0.95
0A6
0862
8895
0A81
8818
0.91
186
0.66
MB11
0,66
(1991
0-388
0881
0A00
0.75
0813
85%
0.63
0,055
0/67
0.991
0.93
0.595
0.20
0.600
0.600
0.58
0615
0/6
59%
0.731
0.225
0.671
0.235
0.685
0.699
0.607
0.655
0.607
0.69
0.691
0.68
0.691
0.00
0.691
0.565
0.168
0.66
0.691
0.244
0.000
0.540
0447
100%
0.723
0.723
0.723
0.723
0,615
0.695
0.695
0.565
0.695
0,565
0.095
0.65
0.505
0.695
0.605
0.695
0,605
0.565
0.565
0,015
0,685
0.295
0.515
0.560
0.651
(I) Model performance of MC-L experiment under ROC-AUC on Credit dataset.
EJegrne
NO1I:
RUBS
SW1
DCNN
FTT
TLL.M
UNI
AVII:
0
0.961
0.755
0.799
0.261
0,485
0.504
0.644
0,617
0,699
0.650
0,4%5
0,592
0.538
0.591
0,596
0.470
0.732
0..59
0.328
0.991
0.355
0.70)
01590
01642
0.634
15
0.769
0.762
0.801
0.787
0.181
0.904
0.611
0.666
0.688
0.653
0.090
0.995
0.545
11.99
0.176
0:070
0.73
0.961
0.536
0.607
0.770
0.61%
0.908
0.61
0.429
10%
0.770
0.752
0.799
0.270
0,481
0.511
0.637
0.892
0,700
0.640
0,490
0.638
0.554
0.612
0.562
0.473
0.342
0.573
0.528
0.201
0.349
0.619
0.510
0.627
0.630
19%
0:763
0.75
0.809
0.862
0.128
0.039
0.91
0.38
0,67
0.51
0,001
0.561
0615
0.561
8-877
0341
0.579
0:570
0.945
0.750
0.485
0.90
0612
0.69
20%
0.763
0.756
0.813
0.799
0.489
0.510
0.632
0.995
0.099
0.64
0.541
0.635
0.511
0.627
0.541
0.41
0.738
0.001
0,500
0.003
0.738
0.575
0.11
0,500
0.630
25%
0.751
0.913
0.775
0.491
0.538
0.639
0.088
0.007
0.657
0.555
0.44
0.51
0.629
0.543
0.71
0.738
0.003
0.457
0,00
0.766
0.9%
0.53
0,69
0/634
15%
0.76
0.750
0,816
0.71
0.501
0.537
0.600
6314
0.507
0.606
0.565
0.69
0.581
0.69
0.554
0.73
0.754
0.003
0.405
0.080
0.75
0.00
0.531
0615
0.55
40%
0.765
0.257
0,81
0.256
0,585
02552
0.456
6308
0,000
0,676
0.545
0.41
0.555
0513
0.595
0.43
0.331
0.61
0.79
0.687
0,342
0,50
0514
0634
45%
0.769
0.57
0.11
6.376
0.909
0.55
0.63
0.706
0.661
0.922
0.50
0567
0613
0.591
0.778
0407
0.50
0.61
0.7M
0.50
0913
067
50%
0.764
0.255
0,897
0.729
0,510
0,555
0.657
0.96
0.11
0.699
0.543
0.63
0545
0.59
0.595
0.463
0.727
0.099
0,506
0.64)
0.757
0.572
0.524
0.00
0.36
55%
0.761
0.752
0,505
0.768
0.511
0.549
0.600
0.84
0.706
0,662
0.527
0.638
0525
0.54
0.595
0.45
0.721
0610
0.11
0.675
0.752
0.51
0.565
0632
0.33
40%
8.46
0.286
6-001
BAN2
0,20
0.57
8.09
DAS
0225
1379
0723
0211
0870
0.728
8.90
0530
0M9
65%
0.753
0.605
0.55
0.508
0.575
0.666
0.64
0,090
0.600
0.41
0507
0,677
0.594
0:44
0.704
0.044
0.506
0.661
0.713
0,500
0537
0,27
0/38
0.46
15%
0.751
0.756
0.76
0.902
0.67
0.705
0.707
0,679
0.516
0.50
0.50
0.541
0.627
0.718
0.00
0.458
0.707
0.741
0:56
0.53
0672
0.66
10%
0.734
0.366
0.79
0.516
0.96
0465
0.16
0.12
0.674
0.5%
0A7
0.09
0.51
0.64
0.31
0.114
0,06
0.467
0711
0.31
0:01
0.61
0662
06M
55%
0.715
0.704
0.765
0.736
0,531
0.534
0.54
0.64
0.679
0,652
0.221
0.644
0.505
0.511
0.66
0.45
0,699
0,560
0.525
0,093
0,366
0,491
0.11
0662
6619
0.71K
0.27
0.745
6.40
0.975
B.7%%
0480
0.685
0.68
0.661
0.69
0.51
0411
0.61
0.90
0:137
0.64
0.60
0.900
0513
010
558
0.714
0.714
0.714
0,690
0.565
0615
0.629
0.629
0.077
0.627
0.525
0.69
0.75
0.451
0.627
0.99
0.62
0.434
0.586
0.629
0,653
0,500
0.524
0.689
0.601
100%
01500
0.500
0.500
0.300
0.500
0.500
0.300
0.500
0.500
0.500
0,00
0.500
0.50
0.000
0.500
11500
0.300
0.000
0.310)
0.000
0.300)
0,500
0.304
0.538
0.310
Model performance of RD experiment under accuracy on Credit dataset.
Degroe
1GGB
XGB
CATB
PFN
DAN
MLP
NO0E
RES
SW1
CAP
NET
GO5
INT
DCN
FTT
GRO
5XT
SNN
TTF
TABR
NCA
L.M
TLLM
UNI
AVE
0
0.757
0.250
0.775
0.262
0.68%
0.695
0.683
0.605
0.668
0.535
0.695
0.584
01.85
0.591
0.98
0.991
0.517
0.538
0.617
0.717
0.750
0.630
0.670
0.67
516
0.745
0.240
0.770
0.750
0.095
0.693
0:681
0.095
0.662
0.531
0.683
0.655
0.505
0.620
0.695
0.531
0.533
0.686
0.716
0.733
0.507
0.006
0,672
15%
0.736
0.719
0.764
0.745
0.685
0,697
0.695
0,676
0.095
0.657
0.561
0,695
0.600
0.654
0.685
0.629
0.605
0.534
0.355
0.683
0.711
0.731
0.586
0.599
0,671
20%
0.722
0.6m
0.750
0:34
0,595
1.655
0.655
06/0
0.95
0.522
0,57
0,60
0.81
11855
0,00
0827
0,685
0.505
0.354
0.811
0.206
0.725
0091
25%
0.731
0457
0.351
0.23
0.555
0.697
0.94
0567
0.95
0,500
0.562
0.65
0.82
01651
0,055
0637
0,695
0.531
0.540
0.881
0.204
0.721
01855
US87
0A67
105
0.719
0.81
0.110
6.77
0.695
0.66
0.695
0478
0.655
0.60
0.571
0.05
0.681
0665
0.691
0.609
0.491
0.540
0.500
0.011
0.713
0.718
0.511
n611
35%
0.713
0.600
0.744
0.231
0,955
0,69
0.694
0.60
0,95
0.646
0.72
0.65
0.644
0.61
0.685
0.33
0,695
0.537
0.532
0060
0.705
0.713
0.883
0.600
0.65
405
0.717
07111
0.76
0.91
0887
0.91
0.67
0885
0.6
0575
0.6
0.61
0647
0.991
0616
0.991
0.511
0.16
0.01
0.00
0.13
0.89
0A61
45%
0.201
0:35
0.7%
0.995
0.695
0.655
0.05
0.656
0.685
0.650
0.505
0.44
0,695
0.526
0.355
0.67
0.70
0.17
0.56
0.500
0.66
40%
0.60
0.89
0142
0.61
0AM
0108
0AM
60%
0700
0.911
0718
0.76
0,615
0,91
0.695
0.63
0.055
0,617
0.599
0.68
0.001
0.68
0.685
0.39
0.691
0.536
0.333
0.05
0.691
5,099
0.58
0.535
0.61
65%
0.10
0665
0.221
0.224
0,95
065
0.64
0456
0.95
0.619
0.15
0666
0583
11855
0.055
0660
0,685
0.596
0.35
0.06
0,666
0.711
0.590
DAK
705
8170
0367
0T15
6.766
0.695
B682
0.695
0467
0.65
0.10
5,61
0665
0567
0667
0.911
0.656
0/691
0.537
0.528
0.81
0491
0,00
0.51
0353
066
75%
0.717
0.655
0.715
0.225
0,95
0.95
0.65
0.45
0.95
0,5%4
0629
0665
0581
0.55
0.695
0.6
0,695
0.517
0.536
0085
0.68
1.000
0.886
0.56
0.60
50%
0.701
0,508
0.709
0.723
0.895
0.95
0.655
0.54
0.955
0,632
0.85
0.60
0587
0.61
0.685
0652
0,695
0.525
0.526
0183
0.65
0.705
0.544
0.9]
0.60
85%
81722
8A10
6:110
0.25
0.95
0.6
0,69
0.63
0.65
0411
836
0Am
0.61
8669
0.911
8M8
0.19
0433
0/64
0AM
0.00
0887
088
90%
0.718
0,61
07155
0.73
0.995
0.695
0.627
0.095
0.60
0.687
0.690
0.595
0.680
0,695
0.554
0.533
0.68
0,65
0.04
0,587
0.556
0.663
95%
0.733
0,735
0.733
0.735
0.695
0.565
0.095
0.56
0.695
0.695
0.055
0.691
0.565
0,565
0.065
0.691
0.285
0.575
0.560
0.651
(h) Model performance of RD experiment under ROC-AUC on Credit dataset.
Degnoe
LGB
XGB
CAUB
PPN
DAN
MLP
NODE:
RB5
SW1
CAP
NET
GUS
INT
DCN
FIT
CRR
5XT
SNN
TIP
TABR
NCA
LMA
VLLM
UNI
AVE
0
0.799
0.781
0.486
0.504
0.644
0.697
0.099
0.650
0,496
0.592
0.538
0.591
0.576
0.470
0.732
0.559
0.528
0.091
0.755
0.701
0.580
0.642
0:634
5%
0./34
01246
0./95
0.201
0.651
11302
01641
0.677
0.80
0,862
0.090
0.313
0.540
11.355
0.571
(1:91
0./26
(1.555
0.530
01.84
0.345
11.00
01.75
11825
0.666
10%
0.745
0.742
0.785
0.763
0,455
0.555
0.640
0.688
0.689
0.643
0.195
0.550
0.547
0.555
0.564
0.674
0.722
0.555
0.536
0.687
0.740
0.643
0.535
0.65
0.633
15%
01241
0.235
0.779
0.255
0479
0.506
0.831
0674
0.76
0.644
0.824
0.576
0.52
01377
0.574
0.475
0230
01550
0.523
0.875
0.727
0.51
01655
0.199
30%
0.733
0.718
0.771
0.755
0.481
0.620
0.677
0.67
0.623
0.513
0.578
0.551
0.508
0.558
0.71
0.705
0.555
0.519
0.665
0.724
0,03
0.535
0.75
0617
10%
0.730
0.727
0.767
0241
0:161
0.17
0.68
8658
0.620
0.571
0.511
0.151
0.601
0554
0.534
0.6T
0.300
0511
0674
0
30%
0.719
0.717
0.74
0.330
0478
0.4%
0.635
0.611
0.685
0.634
0.10
0.550
0.53
0.500
0.565
0.48
0.715
0.557
0.529
0.697
0.331
0.58
0511
0,622
0616
40%
0.705
0.552
0.55
0.50
0.61
0.511
0.60
0,614
0.50
0,577
0.551
0.565
0.564
0.55
0.682
0.355
0.10
8.60
0.604
0:385
0.51
0647
0.66
45%
1222
0.718
0341
0.68
0.40
0.35
0A66
0.495
0.58
0.555
11345
0.366
1.26
0A8I
(1540
0.15
01692
(1500
UARS
50%
0,693
0.622
0.695
0.658
0,92
0.500
0.631
0.71
0.67
0.635
0.536
0.591
0.53
0.570
0.533
0:45
0.702
0,559
0.507
0.600
0.719
0,503
0.114
0,75
0.60
15%
0,633
0.69
0.697
0.63
0.185
0356
0.603
0434
0.42
0.567
0.905
0.564
0.59
0.555
0.55
0.856
0.698
0,599
0314
0,300
0.678
0.50
0.11
0669
0.534
50%
81874
0.688
0577
0,690
0482
0.491
0,610
0.24
0.32
0.585
0.494
0.541
0.54]
0.509
0.545
0.73
0.51
01523
0.525
0000
0.664
11525
0.513
01663
0.532
60%
BAT7
0.80
0688
0:881
0.587
0.60
8615
0.500
0.51
0.34
055
0.47
0888
0,65
0.500
0:500
0/511
0516
0.03
0:77
70%
0.683
0.684
0.635
0.600
0.94
0,505
0.587
0,0
0610
0.572
0.21
0.542
0.54)
0.97
0.62
0.527
0.317
0612
0.647
0.500
0.50
0.622
0.5%
75%
0610
0.311
0575
0.89
0317
0.566
0489
0A0
0511
85%
0.627
0,627
0.511
0.36
0.577
0350
0.56
0:56
0.564
0.09
0.509
0.60
0.535
0.556
0.73
0.594
0.56
0351
0.38
0:503
0.11
0732
0.355
90%
0.573
03314
0.565
01354
0,11
11324
(.361
0.50
01565
0.364
0479
0.39
0485
01338
0.599
01410
0,375
0151
0.16
0386
0.380
01811
0111
0/12
0.44
95%
0.541
0.512
0,536
0.532
0.500
0356
0.533
0.33
03536
0.534
0.475
0.510
0.144
0.518
0.533
0.500
0.551
0.691
0350
0.500
0.31
0,500
0311
0707
0:338
100%
0.500
0.500
0,500
0.500
0.500
0,500
0.500
0.500
0
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0,500
0.500
0.500
0.500
0.500
05500
0.504
0,53
0.302
33

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(i) Model performance gap A of RD experiment under accuracy on Credit dataset. Partial model performance degradation of 0.000 as the table retains three decimal places.
Degice
LGB
XGB
CATB
PPN
DAN
MLP
MCDD
RES
SWI
CAP
NET
005
1ST
DCN
FTT
GRD
SAT
SNN
TTF
TABR
NCA
LMA
TLLM
UNI
AVE
5%
0.012
-2014
.3.005
40015
0..00
0,922
0.030
0.001
0.000
-0010
0611
0.000
-0,935
0.000
0.000
0.019
0.000
-0.011
0.009
0.000
-0.002
.0.04
-0.63
-0023
-0,005
10%
01.22
-0028
0.011
0.200
0.000
01014
0.000
-0033
0.300
4,011
0.02
0.000
-0011
-0007
0.000
0.021
0.000
.0.03
0611
0
4..00
0..21
00064
00031
00003
15%
01827
0044
0.015
0.24
0.000
01055
0.002
0996
0.000
0.017
0.688
0.000
00012
00001
0.000
0.31
0.000
.0005
0613
0.005
0.008
0.925
0.070
0.034
0,05
202
00835
0.059
0.000
0.07
0.000
0103
0.001
0014
0.00
0.025
0.92
0.000
0001
.0004
0.000
0.000
0.000
0003
0611
0.005
0.005
0.000
0.09
0.056
0.044
255
.0853
0008
01029
0.01
01000
0105
0.001
0019
0.00
40007
0.011
0.000
4003
-0006
01000
0.96
0.000
0001
0621
0007
0.018
0.001
0.071
0.253
40014
975-
-0850
00590
41054
0.06
01000
01214
0.93
0903
0.000
40004
0.087
0.000
-003
0.500
0100
0011
0.500
0.07
0,00
46006
.0.47
.607
.0015
.0013
515-
-0857
-0090
40000
2.00
0.000
000
0.001
4009
0.000
40044
0.000
-000
-0.06
0.000
0.49
0.500
0.00
0.088
-0/09
-00166
-0.48
.6675
JA255
40016
40%
-0053
40044
20044
0.01
0.001
4005
0.000
.0.49
0.000
-0000
6,00
0.000
0946
0.000
-007
0.01
0.001
.0019
.0.61
.0.09
.U250
.0011
45%
.0038
40051
2006
0.02
0.001
4001
0:000
-0044
0.01
0.000
4000
-0007
0.000
0.044
0.00
.0017
0.11
0.000
40000
-0.45
.00m
.0281
40016
50%
.0083
0.00
2047
0.02
4017
0.000
4.000
0.000
40005
-0004
0.000
0.00
0.013
0,10
0,002
40025
-0.02
.0835
40013
55%
0.063
2046
4046
0.04
0,000
.0005
0.000
0.000
0.01
0.000
0.003
0011
00007
.0.19
.0073
.0574
.0020
606
0007
0.074
2046
0.01
4054
0,00
0,000
0.003
-0011
0.000
0.000
.0023
0.066
.0012
40032
.0.76
0.021
75%
80%
85%
-0022
8/6
00017
056
0,014
0.000
00020
1008
0,000
0.000
0,155
0.000
00085
i Model performance gap A RD experiment under ROC-AUC on Credit dataset.
Dagree
LGB
XGB
CATE
IFN
DAN
ML.P
NODE
1:35
SWI
SET
006
INT
DCC
FTT
CBO
5XT
5MN
FTF
TAABR
NCA
L.A
11.LM
UNI
AVE
194
.0.52
-0.022
-0.002
-3015
40005
4.085
.0061
-.507
4..50
-3005
-.007
-0035
254
0.051
.0001
0.015
-0050
-.141
0005
4004
0.03
0.66
B020
0/029
0.009
0660
0.16
4.11
0.30
356
0.000
0.844
0/025
0.005
4,020
4.03
..175
4.121
0010
0046
4003
0006
0/06
41152
0,2
4005
0,055
01044
01001
1634
4,034
4.11
0046
45%
-0136
40154
01511
0006
D625
0100
41056
01000
4007
41017
0614
0096
-054
4,024
-0152
40084
4.11
0011
0069
105
40083
11127
0110
0012
D651
40000
0.011
4,111
0181
40077
4001
4005
0.000
1004
40011
01300
4.945
-0011
40169
-0217
-0114
0081
400
59%
0.16)
4.167
0.134
0.182
0,501
D629
0.064
0.291
3051
4,095
0,021
0.045
0.001
0.06
0.000
0655
0099
-0099
4,027
-0883
4.100
40211
4111
0941
4079
0.114
4.115
0.153
0.115
0.00
0636
0103
0.304
4,055
0015
0.855
0,055
0.105
0.07
0606
0.10
4,90
..889
4.121
.0.47
4.11
0233
4083
65%
0.105
0.103
(.14)
0.118
00C
D606
0188
0.129
2.100
4.167
0015
0188
0105
0002
0180
0661
0.16
2634
4,054
-0137
4.163
40.5%
4111
0245
0.90
TUS
0.103
41M3
11266
01143
B01T
b600
0189
0.135
21177
4.120
01812
0184
4181
4063
b659
0,10
-005
4,021
-0.15
4114
.1.2)
-111
0946
0092
75%
.0.15
4.139
41222
0.195
0000
0033
4.108
0.778
-0155
4.153
-0115
4.137
418
41134
4.735
-0033
4.837
-0154
4130
-1271
-115
0,70
411
MUS
.024
.1.28
41341
4032
4000
8000
4.00
4.363
-0156
4,135
-00M
4087
4007
4106
4007
hoss
4017
-001
4.m
-0160
4130
4176
-0171
0/ST
-1111
-0.171
4..60
41.44
41350
0045
D646
4.177
4.144
-0183
4.173
0,87
40071
41 MM
40094
40m
-118
-3011
4,97
-0.70
4122
41761
4018
0.41
-0134
-0337
4.299
41.0
40391
D641
4.124
4.311
-0191
4.161
2083
40885
4002
40069
400
D000
4015
-0071
-0204
40347
40285
2,11
0
-0.41
.0.28
4.282
4
4.11
0,08
0006
4.175
4.343
-0313
4.19
40HL
4.138
4004
40124
4.044
D0SS
4047
-0.11
.0254
40225
40287
2,13
0
-0.67
100%
0.343
0.338
4.594
4.35
0.08
-0066
4.224
4.392
4244
4.230
0.055
4.155
4.01
4.184
4.151
0665
43:7
-0105
4,03
.0277
4,33
4028
2.131
-0156
-0359
34

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 15. Model performance on Electricity dataset.
(a) Model performance of SC experiment under accuracy on Electricity dataset.
Calzrux
PCC
LGB
XGB
CATB
PIN
DAN
MLP
N4006
SW]
CAP
NET
GUS
IN1
DCN
FTI
GBO
5.T
53NN
T1F
TABR
NCA
LMA
TLLM
UNI
AVE
1.
0.913
0.913
0.953
0.961
0.547
0.863
0.739
0.373
0.829
0.845
0.857
0.871
0.861
0.962
0,855
0.794
0,878
0.544
0.730
0.540
0.955
0.504
0.921
0,858
0.86
0.006
0.813
0,12
0.52
0.199
0.44
0.813
0.845
0.215
0839
0.761
0.305
-0.48
0.905
0.906
0.925
0.851
0.899
0.52
0.227
0.859
0.828
0.835
0.807
0.57
0.850
0.89
0,55
0.798
0.857
0.33
0.751
0.52
0.811
0.501
0.888
0.864
0.34
0.065
11998
0:955
IU:45
0.849
0.53
0363
0:728
0.515
0333
0.813
0.856
0371
0.500
0362
0365
0.794
0.576
11834
0.222
14977
0.76
0.501
0:521
0.558
0S477
0.122
0.985
0.910
0.96
0.548
0.39
0.57
0.734
0.862
0.87
0.837
0,566
0.852
0.55
0.57
0.859
0.788
0.870
0.25
0.721
0.595
0.722
0:921
0.57
0.412
0.155
0.98
0.318
0.42
0.82
0.53
0.856
0.715
0.809
0.63
0.87
6
0.234
0.908
0:908
0.967
0.833
01844
0.55
0.740
0.855
00530
0:841
0,544
0.622
0.855
0.37
0.60
0.781
0.370
01851
0.715
0.874
0.775
0.316
0.521
0.864
0341
7
0.314
0.911
0.911
0.547
0.822
0.534
0.334
0.726
0.540
0.817
0.825
0.82
0.812
0.500
0.37
0.854
0.768
0.854
0.814
0.701
0.561
0.982
0.502
0.920
0.861
0.830
0,319
0,485
0.14
M.67
0.691
0.65
0.48
0.587
14552
0.689
(b) Model performance of SC experiment under ROC-AUC on Electricity dataset.
Colorn
PCC
1.38
XGB
CATB
PPN
13N
MIP
M403F:
RE5
.WW7
CAP
NFT
GGS
INT
DCN
FTT
(RRO
SAT
SNN
TTF
TABR
NCA
1.A
T1.1M
UNI
AVF
1.
0.913
0,915
0.953
0.961
11347
0.963
0.729
0.529
0.813
0.857
0.811
0.362
0.853
0.794
0.518
01534
0.729
0.940
0.985
0.921
0
-0506
0.17
0.318
0,540
0.842
01522
0.344
6.731
0.547
01819
0.830
0.534
0344
0.443
0.838
0.345
0.716
0.339
0.764
0.510
0.587
9.77%
01306
-0.048
01905
0:906
11525
0:51
0139
0352
0.727
0,859
11325
03855
01567
0357
0.55
0330
0.853
0.79
0.57
11:25
0.251
10852
0X81
0.501
0.588
0.864
0.065
0.905
0.906
0.445
0.549
0.48
0.853
0.738
0.173
0186
0.845
0.86
6.71
0.560
0.62
0,86
0.794
0.876
0.834
0.732
D.987
0.776
0.501
0:920
0.858
0847
-0122
03003
013))
0127
0362
0357
0.570
0.721
01995
0142
0.155
0.008
0.909
0.45
0.848
0142
0.51
6.731
0.358
0822
0.855
0.63
0.38
0.81
0.83
0.58
0.73
0.356
0.715
0.80
0.63
0.500
0.916
0.855
0.87
6
0.234
0.908
0.008
0.54
0.853
01544
0.85
0.740
0.865
0.841
0.544
0.833
0.855
0.87
0.882
0.781
0.370
0.631
0.715
0.874
0.75
0.508
0.921
0.864
0184
0.334
0911
0:911
11567
0.822
01344
03344
0.736
0.540
11817
0.833
0.22
0.442
0.541
0337
0.854
0.764
0.854
11514
0.711
11664
0.82
0,512
0:921
0.861
03330
0,379
0.485
0.64
0.677
0.658
0.646
0.587
0552
0.888
0.679
(c) Model performance of MC-M experiment under accuracy on Electricity dataset.
Degree
LGB
XGD
CATD
PTN
DAN
MLP
NN0DC
RDS
SW]
CAP
NCT
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTT'
TABR
NCA
LMA
TLLM
UNI
AVE
0.766
0.798
0.763
0.785
0.756
0.761
0.729
0.941
0.925
0.740
0.794
12.5%
0.680
0.477
0,532
0,580
0.67
0.600
0.606
0.634
0614
0.644
0.625
0.639
0.632
0.622
0.625
0.255
0.623
0.623
0.578
0.554
0.506
0.427
0.336
0.669
0.601
35%
0.482
0.450
0.518
0.492
0,552
0.514
0.500
0.510
0.887
0.621
0.571
0.542
0.562
0.517
0.597
0.581
0.583
0,522
0.555
0.527
0.508
0.47
0:35
0667
0.555
37.5%
0:474
0,173
0.518
0.391
0.541
0,530
0.572
0.531
0.567
0.601
0.559
0.559
0.570
0.525
0.576
0.499
0.576
0.519
0.545
0.505
0.699
0:425
0.34
0.668
0.351
50%
0.572
0.403
0.997
0.493
0.909
0.32
0.670
0.515
62.5%
0.682
0,445
0.495
0,463
0.316
0
0,372
0.516
0.265
0.508
0.547
0.513
0.53
0
0.508
0.509
0.518
0.525
0.517
0.499
0,491
0.427
0.3
0.668
0.320
15%
0.579
0.474
0.191
0.318
0.190
0.505
0,572
0.510
0.536
0.465
0.550
0.510
0.504
0.505
0.504
0.505
0.519
0.525
0.538
0.515
0.491
0.125
0.31
0.600
0.327
87.5%
01526
0,572
0.49
0.539
0.335
0.00
01706
0.547
100%
0.425
0.425
0,572
0.572
0.72
0,476
0.476
0.428
0.428
0.476
0.334
0,438
0.83
0.618
0.509
(d) Model performance of MC-M experiment under ROC-AUC on Electricity dataset.
Deggoe
CAUB
DAN
N0DDE
RES
SW1
CAP
INT
DCN
FTT
GRR
5NNN
TABB
NCA
YLLM
UNI
AVE
0.913
0,913
0.953
013611
0.547
0.365
0.739
0.873
0,29
0.343
0.561
11862
0.563
0.734
0.578
0.965
0.514
01921
01355
0.361
12.51
0.485
0,450
0.571
0.013
0.667
0.665
0.687
0.699
0.675
0.700
0.560
0.691
0.569
0.657
0.655
0.665
0.647
01675
0.587
01553
0.504
01500
0183
0.745
0.639
255
0.487
0.454
0.592
0.486
0.563
0.527
0.652
0.535
0.583
0.605
0.531
0.558
0.545
01523
0.573
0.688
0.546
0.908
0.558
0.501
0.479
0.500
0131
0.782
0.564
37.5%
0.435
0.54
0.335
0.581
0.500
01521
0.545
11511
0.545
0.434
0.33
01781
0.552
50%
0,479
0.498
0.450
0.516
0.900
0.500
0:511
0:46
0.544
0.488
0.514
0.501
0.499
0.490
0.533
0.488
0.535
0.460
0.468
0,500
0.300
0.783
0.531
62.5%
0.437
0.434
0.495
0.480
0.540
0.500
0.550
0.509
0413
0.581
0.493
0.517
0.517
0.479
0.501
01535
0.492
0.523
0.489
0.446
0.467
0,500
0.22
0.779
0.522
75%
0.479
0,479
0.685
11582
0.505
1,502
0,527
0.514
0.518
0.504
0.497
0.506
0,415
11499
0.494
01519
0.417
0.519
0.526
0.09
0,488
0,500
01330
01908
0.528
87.5%
0.484
0.485
0.904
0.485
0:400
0819
0.516
100%
0,500
0.500
0.500
0.500
0.500
0.500
0.501
0.439
0.500
0,500
0.500
0.500
01501
0.500
0.439
0.499
0.500
0.500
0.495
0.497
0,500
0.536
01802
0.536
(e) Model performance of MC-L experiment under accuracy on Electricity dataset.
Degrne
NOTE
RBT
5W1
DCN
FTT
CRO
5NN
TABR
NCA
TI..M
UNI
AVE
0.838
0.819
0.880
0.764
0.772
0.287
0.633
0.798
0.763
0.771
0.780
0.795
0.785
0.756
0.784
0.729
0.799
0.763
0.660
0.860
0.941
0.833
0.925
0.740
0.704
12.5%
0.683
0.636
0.685
0.770
0.762
0.766
0.633
0.768
0.754
0.760
0.762
0.759
0.767
0.763
0.716
0.705
0.790
0.759
0.651
0.689
0.705
0.427
0.589
0.675
0.714
35%
0.617
0.653
0.687
0.265
0.762
0.758
0.631
0,788
0.758
0.357
0.762
0.759
0.765
0.757
0.139
0.714
0.390
0.756
0.675
0.645
0.710
0:63]
0.686
0.706
0.720
37.5%
0.760
0.630
0.788
0.762
0.762
0.766
0.758
0714
0.756
0.710
0.885
0.703
0.721
50%
[.54]
0,612
0.674
0.256
0.761
0.765
0.627
0.76
0.764
0,355
0.260
0.756
0.765
0.755
0.726
0.720
0.740
0.751
0.691
0.682
0,708
0.632
0181
0.710
0.718
62.5%
0.605
0.637
0.605
0.768
0.756
0.756
0.624
0.755
0.755
0.740
0.752
0.757
0.753
0.755
0.677
0.73
0.312
0.755
0.622
0.605
0,700
0.128
0.886
0.712
0.700
75%
1,500
0.245
0.000
0.755
0.755
0.741
0.750
0.754
0.066
0.689
0.712
0186
1.720
0.697
87.5%
0.802
0.579
0.736
0.44
0.577
0.750
0.349
0.721
0.738
0.742
0.689
0.624
0.723
0,700
0.81
0.724
0.686
100%
0.425
0.426
0.425
0.475
0.524
0.572
0.572
0.572
0.569
0.476
0.524
0.572
0.475
0.476
0.425
0.476
0.324
0.572
0.428
0.628
0.428
0.427
0.63
0.613
0.511
(f) Model performance of MC-L experiment under ROC-AUC on Electricity datasct.
Deegoee
XCBB
CATR
DAN
MI.
NOTE:
RPS
SW1
CAP
G05
INT
DCN
FTT
CIRO
5AT
SWN
TARR
NCA
1.A
T1..M
1/N1
AVIE
0
0.913
0.913
0.953
0.861
0.547
0.963
0.729
0.873
0.529
0.343
0.857
0.371
0.561
0.562
0.583
0.794
0.378
11.34
0.720
01940
0.965
11804
(1922
01858
0.351
12.5%
0.517
0.318
0.540
01842
0.852
01344
0:331
0.847
0819
0.350
0.634
0.347
0.543
11843
0.534
0.772
0.345
0.821
0.716
0.539
0.764
0.500
(1887
11775
0.306
25%
01332
0.726
0.221
0.826
0.535
01332
0.525
0.775
0.524
0,773
01777
1111
0.812
37.5%
0.501
0.11
0.33
0.224
0331
0.738
0.536
0824
0.536
0.535
0331
0.526
0.715
0.528
0.16
0,771
0.503
0.775
0817
0.800
50%
0.795
0.736
0.804
0885
0.112
0319
0.752
0.119
0816
0.314
8811
0.319
0.525
0317
0.516
(1772
0.515
01806
0.700
0.775
0.769
0.503
0.576
0.19
0.702
62.5%
0L784
0.754
0.714
0382
0.305
0311
0.337
0.110
0504
0.300
0.795
0.111
0.51
0830
0.505
01749
0.788
01303
0.766
0.754
0.77)
0.500
0176
US15
0.733
75%
0.285
0.505
0.788
0.844
0.505
0.72
0.700
0.400
0.875
0883
0.562
87.5%
0.781
0.751
0.780
0.700
0.778
0.778
0.77%
0.778
0.756
0.754
0.778
0.778
0.778
0.778
0.728
0.778
0.778
0.727
0.749
0.767
0,500
0.571
0827
0.766
100%
0.500
0.500
0,500
0.500
0.500
0.500
0.500
0.500
0.686
0.500
0.500
0.496
0.500
0.500
0.500
0.500
0.502
0.500
0.500
0.498
0.501
0.500
0.636
0.801
0.526
35

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
g
Model performance RD experiment under accuracy on Electricity dataset.
Degiee
LGB
XGB
CATE
PPN
DAN
MLP
NODE:
RES
SW1
CAP
NET
GOS
INT
DCN
FTT
CRR
SAT
SNN
TTF
TASR
NCA
LMA
TLLM
UNI
AVIE
0.798
0.763
0.785
0.756
0.784
0.729
01925
0.740
0.794
12.5%
11747
01621
0761
0,342
0.51
01755
0.750
0777
0.762
0.723
0.552
25%
0.893
0.698
0.715
0.714
0.727
0.776
0.600
0.72
0.721
0.729
0.734
0.731
0.727
0.734
0.715
0.62
0.729
0.721
0.68
0.663
0.664
0.764
0.57
0.714
0.713
37.5%
0.633
0.635
0.651
0.622
0.702
0.644
0.599
0.94
0.677
0.706
0.00
0.701
0.085
0.92
0.676
0.57
0.688
0.007
0625
0.611
0.564
1.649
0.81
01703
0.677
50%
0,900
0.699
01686
62.5%
0615
0.67
0.668
0.673
0.56
0.654
0.617
0669
0.400
0.500
0.001
0.61
0.73
0.635
75%
0.491
0.991
0.495
0.578
0.614
0.595
0.577
0.596
0611
0.602
0.655
0.666
0.991
0.556
0.51
01573
0.599
0.614
0.575
0.518
0.480
0.538
0.57
0.655
0.534
87.5%
0.655
0.456
0.657
0.541
0.610
0.586
0.575
0.596
0.994
0.563
0.613
0.599
0.571
0.551
0.496
0.543
0.583
0.573
0.492
0.454
0.427
0.499
0.51
0655
0.554
0618
0.511
(h) Model performance of RD experiment under ROC-AUC on Electricity dataset.
Degree
LGB
XGB
CATD
PFN
DAN
MLP
NODE
RES
SWT
CAP
NCT
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
TLLM
UNI
AVE
12.5%
0.844
0.876
0.816
0.821
0.878
0.736
0.835
0.807
0.818
0.821
0.856
0.825
0.827
0.831
0.765
0.835
0.305
0.835
0.807
0.814
0.90)
0.843
0.870
25%
0.780
0.779
0.814
0,774
0.791
0.791
0.721
0.795
0.781
0.790
0.782
0.799
0.791
0.759
0.796
0.729
0.793
0.777
0.688
0.720
0,711
0.761
0.673
0.827
0,277
37.5%
0,722
0.720
0.755
0.735
0,758
0.752
0.714
0.754
0.749
0,799
0.742
0.751
0.75]
0.749
0.761
0.710
0,782
0.743
0.688
0.665
0,658
0.677
0.576
0.87
0.739
50%
0.13
0.34
0.736
0.752
0.740
0.762
0.09%
0.750
0335
0813
0.724
62.5%
0.623
0.621
0.518
0.619
0.721
0.710
0.704
0.705
0.693
0.723
0.702
0.716
0.704
0.704
0.339
0.77
0.318
0.895
0.641
0.518
0,370
0.65
0.366
0
0.690
75%
0.582
0,579
0.599
0.633
0.639
0.69
0,657
0.117
0.609
0.649
3.611
0.630
0.509
11610
0.649
0.666
0.330
0.624
0.581
0.001
0,530
0.569
0147
0810
0A28
87.5%
0.555
0.648
0.699
0.567
0.635
0.552
0.655
0.699
0.578
0.530
0.33
0811
0.601
1005
0.095
0.35
0.327
(i) Model performance gap A of RD experiment under accuracy on Electricity dataset.
Degree
1.GH
CATR
PFN
MIP
MODE
SW1
GUS
IX7N
FTT
(BR
111.M
UNI
AVE.
12.5%
-01912
.1183
.1.105
-1047
.U108
-0638
-1.119
-0:345
-1011
4.175
-U1136
-11142
.11036
-1.099
41.44%
-0651
-1046
-2615
-2.11
-0.123
41.195
-1.001
-1/111
-1611
-0005%
25%
-0.173
4..144
.0.87
.0.58
-0:07
.0089
-3055
.0003
-0.079
-0.68
-0065
.0088
-3055
4,034
-0.229
.0.244
.0082
-0/49
-0.101
37.5%
-0.245
4.245
40.261
0.130
0..90
-0.17
4.063
0
-3056
4.081
-0.108
4..18
-0.13
-0.120
4.117
00.10
-0.17
-0033
-4083
-0.29
40000
-0.221
-0,917
-0650
-0.44
50%
0,309
..065
.0.14
4.110
0.141
.3093
4,088
.0,33
4.428
4
-0.6T
62.5%
0.3365
0.365
.0.885
0.216
0.115
00.50
0.000
0.365
-0.25
4,124
00.39
0.149
4.148
0.156
0.213
0.151
0.113
-0.25
4,077
-0334
40,59
-0.278
-0,067
-0891
-0200
79%
41457
-0244
4.297
0.214
4.250
-3196
-0116
-0.34
37.5%
00.85
0.457
0..00
0.310
0.209
0.253
0
0.254
0,213
4.269
0.215
0.247
4. 266
0.299
0.367
0.255
0.39
-0219
4.254
0.472
0,596
-0.155
-00080
-0.15
-0300
1005
-1516
4.321
0,373
0,311
0,502
40487
40165
-0335
(i) Model performance gap A of RD experiment under ROC-AUC on Electricity dataset.
Degree
CATR
MPN
MIPP
NODE
SW1
G06
DXN
111.
UNI
AVE.
13.5%
.0175
-1.076
.1100)
.1062
.U.511
-0004
-11035
-0:343
-10136
-0113
.0/42
-2114)
.11038
-0.41
41057
-0634
40049
-2/33
-.1131
-0.111
40.181
01013
-0/122
-U618
-0/94
25%
-0.145
-0.147
-1.146
-1101
-L066
-3084
-0.011
-0/899
-3055
-0/63
-91335
-0182
41182
-0085
4L17%
-U0TT
-0097
-3069
-00944
-0234
41227
4105
-0053
-0003
-009T
31.5%
-0,200
-0.211
.1.28
JL.48
JU116
-0:29
-U101
40.35
-U091
40.00
.0.134
42.225
JL123
4L151
AL119
-0.07
J0143
-0.09
.U101
-0.292
4U.32
JL182
.0044
JU648
.0142
0.269
.0.33
0.141
0.117
.0.21
0.145
-0.21
4,073
4.354
4.19%
4,068
.0052
40.60
63.5%
.0,113
.0,300
.1.331
41.46
..149
.0:78
40104
4.185
-0154
4,142
.0181
4.177
JL183
4U183
JL.44
-0147
4,142
-0166
4,110
-03343
41421
.1234
JUm71
JU863
.0.99
79%
4.371
0.345
.0282
0,099
-3265
0.287
0.292
0.348
0,233
-0251
4,193
0,357
4,461
4.292
4,080
.0057
4.271
87.5%
-0:01
0,405
0.114
0.355
0.216
0.316
0.111
0.280
0.317
4,246
0.253
0.233
0.315
0.360
0,241
0.220
00312
00331
4.293
-0.35
40,800
-0,341
-0094
-0055
-0300
100%
4L410
94411
4,417
4L421
0U370
4448
-3400
4,315
-0,470
+008
40064
4.319
36

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 16. Model performance on Heart dataset.
(a) Model performance of SC experiment under accuracy on Heart datasct.
Cobrrn
PCC
PIN
SAT
111-
TABR
NCA
0.833
0.962
0.674
0.671
0.215
0.581
0.859
0.965
0.774
0.589
0.855
0.322
0.893
0.380
0.792
0.366
11871
0.870
0.850
0.533
01339
0.883
0.746
0379
0.766
0.233
0.817
0.324
0.859
0.74
0/4P
03622
0761
0,683
0359
0.51
0372
0,592
0.51
0.515
0379
0.77
0.53
9.857
0.735
0.257
0322
0861
0,61
0683
0383
0750
0371
0364
0797
0.591
0.88
0.522
0391
0385
9,55
0.790
0.352
0.822
0.333
0575
0866
0676
0,55
0:611
0.83
0.66
0.656
0.75
0.75
0752
0.007
0.366
0.334
0.995
0.800
0.211
0.733
0.664
0.79
0.306
0821
0.79
089
0317
0664
03T1
0941
0.455
1188
0.84
8.66
0361
0597
086
0372
0.5
0.70
0.38
0.87
0899
0389
0880
0.66
0837
0.29
4.367
0.744
6.805
084
0.5
062
0.46
0.44
0.851
086
0.72
0.54
0.61
0500
0.61
0.30
0.56
0.30
0.904
088
0.77
0.35
0510
0.6
0.776
8.400
0.888
0.450
0366
0.73
0.66
0.39
0.54
0.857
0885
0.77
0.679
0.59
0.66
034
0.761
0,591
0.811
0.755
0.24
0.855
0.77
0.728
0.788
0359
0.2
04444
0330
0.435
0871
0.88
0167
0357
0641
0.859
1180
0.775
0.574
0350
0.572
0.89
0.774
0.005
0.801
11363
0.22
0.50
0380
0.734
0.79
0/226
0.26
10
0.694
0.211
0.828
0.564
0.871
0.688
0.399
0.630
0.348
0.530
0.241
0.64
0.89
0,590
0.33
0.764
0.596
0.851
0.764
0.319
0.599
0.897
0.723
0.772
0,342
0.77
11
0.559
0.736
0.750
0
0.817
0.582
0.808
0.639
0.824
0.53
0.721
0.656
0.835
0,689
0.332
0/694
0.596
0.822
0.770
0.487
0.553
0.788
0.674
0.761
0,848
0.736
(h) Model performance of SC experiment under ROC-AUC Heart dataset.
PCC
LGGB
PPN
SW6
005
DCN
SAT
TTF
TABR
NCA
AVF
0.902
0.939
0,949
0.947
0.935
0.955
0.445
0.961
0.960
0.849
0.087
0.3413
0.934
0,951
0.945
0.442
0.878
0.954
0.465
[1958
01850
0.806
0.036
0.734
0.030
0.449
0.938
0,872
0.957
0.447
[U690
0.856
-0.353
0.689
0.937
0.34
0.40
0.925
0.847
0.457
0.955
0.500
0.849
0.11
0.81
0,011
0.415
0.86
0,306
0338
0829
0681
0425
0,84
0944
0953
0,007
0.51
0,422
050
0950
8.53
0,701
050]
0337
-0337
0.66
0.376
0.20
0919
0.35
0013
0.99
0.46
0.60
0.928
0.92
0814
0.90
0.976
0,433
0,939
0.038
0.584
8.76
0.26
-0400
0.920
0936
0737
0919
0.63
0.937
0.15
0.45
0.949
0.466
0.588
0.958
0.679
084
0.404
0.896
899
0917
0363
091
098
0.15
0.144
0948
093
0.89
0.61
0.95
08%
0.83
0.064
0.16
0.67
0.70
0890
0444
0.891
6.900
0.93
0918
0.725
0.20
0.45
0.10
0.88
0.16
0.21
0.63
0.22
0.911
0851
0.411
0.98
0.69
0.72
0.764
0.501
0841
11
0.559
0.648
0.851
0336
0.98
0.611
0.80
0.843
0:918
00500
0.826
0
0935
0.720
0.000
0.786
0.640
0.966
0151
0.45
099
0919
0.500
0.750
0.896
0881
(c) Model performance of MC-M experiment under accuracy on Heart dataset.
Degree
X5.0
CAII
PVN
DAN
MI.P
NOO0X
RIUS
SW1
CAP
NET
GUS
INT
1X.N
FTT
(RRU)
SAT
SMN
TAIBR
NCA
1.MA
YI.LM
UNI
AVI
0
0.833
0.855
0.879
0.862
0.674
0.846
0.663
0.857
0.871
0.775
0.681
0.359
0.583
0.868
0.774
0.580
0.375
0.785
0.322
0.893
0.350
0.845
0.783
0.833
0.792
9%
0.7336
0,817
04639
0.824
0,355
0,721
0.655
0.859
01812
0.044
0155
0,833
01770
01883
0.753
01777
01542
0.736
18%
0.565
0.737
0.561
0.297
0.521
0.285
0.16
0.781
0831
0,668
0.61%
0.795
0.561
0.777
0.681
0.%6
0.793
01712
0.518
01843
0.334
01781
01889
0.716
27%
0.779
0.600
0.766
0,750
0.645
0.763
0.678
0.5%
0.714
0.793
0.810
0,702
168
0,737
0.571
0.745
0.752
0.650
0.575
0.63%
0.625
0385
0.714
0.712
0.707
0.866
0.600
45%
0
0.799
0.553
0.651
0.520
0.644
0.80
0.631
54%
0.66
0.543
0.992
0.505
0.557
0.558
0.22
0.596
0.50
0.722
0.811
0.601
69%
0.78
0.554
72%
0,003
0.566
0.545
0.45
0.555
0.558
0.52
0.543
0.114
0547
0.565
0.572
0.565
0.76
0.79
0.785
0.501
81%
0515
0.534
0.544
0.38
0.53
0.25
0.505
0.515
0.53
0.33
0.55
0.750
0.54
50%
0.536
0.482
0.354
0.524
0.502
0.547
0.520
0.538
0.533
0.507
0.411
0.502
0.560
0.493
0.534
0.496
0.511
0.511
0.799
0.733
0.534
100%
0.535
0.452
0,464
0,318
0.533
0.533
0.533
0.677
0.511
0.489
0.511
0.533
0.467
0.533
0.533
0.489
0.533
0.467
0.511
0.511
0.13
0.712
0.527
(d) Model performance of MC-M experiment under ROC-AUC on Heart datasct.
Degres
1GBB
SCBB
(CATB
PRN
DAN
MIP
NOOE
SW1
CAP
NET
GOS
INT
DCN
FTT
CRRD
SAT
SWN
TTP
TARR
NCA
1.MA
11..M
1/NI
AVIE
81
01.017
11:20
0.915
0.93M
0.377
0.911
0.351
0.917
0.949
0.867
0.715
0.947
0.764
11.1%
0.565
11676
0.954
(18955
0.448
01961
0.980
01661
(1771
(1911
0.349
9%
0.545
0.351
0.555
0.908
0.651
0.510
0.843
0.918
0.530
0.836
0.691
0.935
0.720
0.920
0,794
0.646
0.936
0.851
0.457
0.942
0.919
0,630
0.762
0.585
0.318
18%
0.817
0.832
0.811
0.897
0.613
0.887
0.319
0.892
0.911
0.337
0.611
0.917
0.734
0.877
0.783
0.61
0.923
0.905
0.442
0.925
0.904
0.640
(177)
01894
0.756
27%
0.789
0.310
0.834
0.862
0.592
0.858
0.781
0.611
0.611
0.655
0.640
0.350
0.715
0.455
0.505
0.61
0.593
0.737
0.406
0.693
0.861
0.680
0.782
0.896
0.751
16%
0881
0,866
0.35
0.588
0826
0.367
0361
0859
0.716
0.599
0.800
0.71
01711
0.766
0615
0.570
01711
0.510
0175
0.867
01762
01855
0.767
45%
0.727
0,237
0.741
0.560
0.18
0.219
0.77
0.966
0.77
0.600
0.522
0.88
0.591
0.76
0.64
0S51
0.798
0.66
0.471
0.79
0.750
0.155
01781
0189
0.300
54%
0.708
0.724
0.750
0.701
0.589
0.700
0577
0.608
65%
0.215
0,099
0.712
0.597
0663
0.557
0.68
01855
0645
72%
0.15
0.562
0.56
0.701
0.858
0.623
50%
0.530
100%
0,500
0.500
0.500
0.500
0.500
0,500
0.500
0.500
0,500
0,500
0,500
0.500
0.500
0.500
0.500
0.500
0.500
0,500
0.500
3,500
0,500
0,500
0.732
0355
0.525
(e) Model performance of MC-L experiment under accuracy on Heart dataset.
Degree
XLII
CATH
PIN
DAN
MLP
NODE
RBS
SW1
CAP
NET
GUS
INT
IX.N
FTT
(RR)
5AT
SMN
TT1'
TABRR
NUA
1.M
TLLM
UNI
AVE
0
0.833
0.855
0.870
0.862
0.624
0.548
0.663
0.557
0.871
0.775
0.681
0.850
0.583
01858
0.774
(1559
0.575
01758
0.532
0.803
0.360
0.865
0.783
01833
0.702
0%
0.637
0.859
0.870
0.866
0.683
0.857
0.657
0.861
0.871
0.768
0.687
0.854
0.585
0.870
0.777
0.594
0.580
0.788
0.522
0.889
0.362
0.653
0.755
0.670
0.706
18%
(15432
03357
0.571
03551
0.68
03322
01822
0.564
03775
0.795
0.685
0.871
0.712
01871
0.701
11000
0.874
(1781
0.516
01889
0,879
01894
(1115
0187
0.795
27%
0.517
0.330
0.544
0.355
0,990
0.20
0.617
0.50
0.66
0,310
0.69
0.852
0.585
0.851
0.781
0.515
0.561
0.793
0.516
0.575
0.875
0.793
0.788
0.853
0.757
36%
0.524
0.306
0,532
0355
0.330
0357
0.634
0.511
0546
0.700
0,09
0.361
0.585
0.87
0.797
0.603
0.355
0.785
0.133
0.565
0.862
D.10
0.799
01332
0.755
45%
0.813
0.758
0.530
0.53
0.32
0371
0611
0.50
0.61
0.763
0.50
0.852
0.585
0.800
0.770
0.65
0.353
0.71
0.525
0.662
0.366
0.765
0.76
0.699
0.783
54%
0.795
0.674
0.825
0.24
0.221
0.46
0.62
0.66
0.41
0.317
0.644
0.345
0.567
0.811
0.56
0.52
0.346
0.781
0.460
0.673
0.862
0.793
0.739
0.62
0.766
69%
(1715
0.60
0.044
038
0.150
0361
0660
0.85
0812
0.806
0612
0.3)
0.10
0856
0.21
099
0.331
(1541
0.4%
01890
0,83
0.701
01112
01777
0:447
12%
0.663
0.67
0.76
038
0.211
0817
0.6
6312
0.70
0.32%
0.540
0.10
0.561
084
0.15
0151
0.336
(1341
0.508
0.815
0,83
0.701
0.7%
0142
0.332
81%
0.750
04338
0.76
0.53
0.634
0.56
0.536
01785
0.734
90%
0.790
0.645
0.533
0.46
0.743
0.707
0.766
0.503
0.11
0.533
0.499
0717
0.712
0.327
37

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(f) Model performance of MC-L experiment under ROC-AUC on Heart datasct.
Degree
RPS
FTT
(RR0
VIL.
UNI
AVIE
0.957
0.940
0.705
0.784
0.935
0.965
0.626
0.955
0.773
0.911
0.349
0.990
0.496
0.808
0.362
27%
0.76
0.28
0346
0.777
0338
0.572
0603
0552
0.54
0.914
0.744
0.914
0:857
10%
0.29
0131
0.584
0.09
0.500
01905
0.257
45%
0,922
0.20
0.751
0.52
0.945
0.864
0.977
0.779
0.35
0.577
0.42
094
0.947
0.511
0.766
0.767
0.1
0.356
54%
01877
0.893
0.26
0977
0.06
0.944
0.36
0.76
0.000
0929
0.689
0.99
0941
0.81
04488
0.941
0.94
0R15
0.755
0.92
0.443
69%
0.820
0.37
0,800
0910
0.911
0:11
0.909
0.874
0.909
0.591
0936
0.555
0.902
0.19
0.318
0.915
0.944
0.795
0.73
0.88
0.112
72%
0.851
0.866
0.866
0,710
0.718
0.86
0.88
0.888
0.87
0.871
0.564
0.866
0.643
039
0.776
0.11
0.906
0.751
0.350
0,95
0.916
0.36
0.744
0911
0.802
81%
01844
0.843
0.83
03877
0.823
0369
03883
0.833
8889
0.891
0.009
0.814
0.70
0877
0.30
0621
0.892
0661
0.564
0.894
0.896
0
(.73)
0876
0.995
90%
0
0.831
0.501
0.798
0.836
0.836
0.836
0.336
0.36
0.342
0.835
0.836
0.512
0.836
0.612
0.400
0.849
0.612
0.186
0.542
0.842
0.773
0.723
0.847
0.753
100%
0,500
0.500
0,500
0,500
0.500
0.500
0,500
0.500
0,500
0,500
0,500
0.500
0.500
0.500
0.500
0.500
0.500
0,500
0.500
0,500
0,500
0,500
0.732
0.856
0.325
(g) Model performance of RD experiment under accuracy Heart dataset.
Dagroc
N00E
RES
SWT
DCN
FTT
CRR
TABR
NCA
TL1M
USI
AVE
0.653
0.857
0.571
0,775
0.683
0.858
0.774
0.569
0.875
0.783
0.853
0.292
916
0,359
0.845
06400
0.356
0.564
0.574
0.357
0.760
01593
0.564
0.589
0.872
0.791
01847
0.781
18%
0.549
0.858
0.772
0.656
0.846
0.742
0.999
0.549
0.792
0.835
0.369
27%
0.600
0.856
0.44
0.555
0.827
0.715
0.555
0.758
0.72]
0.780
0,83]
0.753
0.77
0.750
49%
0583
0821
0.335
14%
6716
0.68
0617
0315
0325
0.55
0.29
0753
0.67
0.73
0.76
0717
0.777
0.77
0827
07731
63%
0.662
0,622
0.62
0.592
0.801
0813
0.333
0.501
0.56
0.633
0.83
0.74
0.999
0,500
0.756
0.67
0.773
0.804
0.702
72%
0.629
0.64
0.34
0.557
0.722
0.77
0.725
0.572
0.504
0.893
0.578
0.63
0.721
0.761
0.54
0.790
0.622
91%
0.65
0.585
0.28
0:591
0.340
0.764
0.118
0.2
0.577
0.61
0.611
0.665
0.691
0.61
0.41
0.84
0.656
50%
0.568
0.541
0.638
0.61
0.539
0.633
0.74
0.640
0.627
0.564
0617
0.540
0.35
0.564
0.67
0.506
0.22
0.74
0.783
0.601
1009
0.535
0.482
0.464
0.518
0.53
0.335
0,533
0.467
0.511
0,489
0311
0.333
9.467
0333
0.333
0.889
0.333
01.88
0.467
01511
0.511
0.583
0.717
07713
0.522
(h) Model performance of RD experiment under ROC-AUC on Heart datasct.
Degrne
1.688
XCBB
CATR
PIN
DAN
MI.
NO13F
RB1
SWI
CAP
NE:1
G0S
INT
DCN
FTT
(RR
SAT
SWN
TTP
TARR
NCA
1.MA
VI.IM
UNI
AVIF
0
0.902
0.939
0.938
0.939
0.727
0.931
0.361
0:937
0.940
0.867
0.705
0.947
0.784
0.938
0.563
0.626
0.955
0.865
0.448
0.961
0.960
0.683
0.773
0.911
0.349
9%
01891
0.900
0,995
0.311
0,321
0.97
0,874
0.951
0.445
0.358
0.718
0.942
0.755
0.930
0.35)
06635
0.54
(1815
0.44)
01915
0.951
0.771
01787
(1903
0.348
18%
0.881
0,849
0.917
0.922
0,712
0.911
0.872
0.926
0.99
0.88
0.716
0.917
0.72%
0.11
0,339
0615
0.944
01817
0.456
01941
0.943
0.751
0.793
01877
0.340
27%
0.831
0.917
0,50
0.11
0.620
0.934
0.525
0.949
0.931
0.776
0.596
0:426
16%
0.88
0994
0.370
0.910
0350
0.679
01955
0,322
U641
0.534
0.93
0,701
01855
0:325
45%
0.599
0.856
0.784
0.67
0.776
0.835
0.310
54%
0.001
0.856
0.789
0.677
0.772
0.992
0.756
0.769
72%
0.767
0.28
0347
0.777
0.623
0.54
0.712
0.06
0.555
0,85
0.755
01355
0.74
81%
0.789
0308
0.76
0.734
0.66
0.82
0.796
0.74
0603
0172
0.222
90%
0.537
0.708
0.603
0.686
0.682
0.678
0.663
0.613
0.559
0,695
0.623
0.680
0.34
0,579
0.642
100%
0.500
0.500
0,500
0.500
0.500
0.500
0.500
0,500
0.500
0.500
0.500
0.500
0.500
0,500
0,500
0.732
0.556
0.525
(i) Model performance gap A of RD experiment under accuracy on Hcart dataset.
Degree
LGB
XGB
CATB
MPN
DAN
MLP
NODE
RIS
SW]
CAP
SET
GO6
INT
DCN
FTT
CRO
S.T
SNN
TTF
TABR
NCA
LMA
TLLM
UNI
AVE
-10022
-1006
-0.001
-0/006
-0:001
0/015
0.000
-1003
-40022
-0007
16007
-0013
-3001
-.001
-3.155
41010
0.014
-00007
-0/05
18%
-0147
-0.077
-1.000
-1.117
-U097
-0182
-0:359
-30115
-0004
-0103
-01877
-0113)
-1.25
-U.41
DG11
-0029
-3026
.0001
-0103
41185
.1113
0.014
-0021
-0.029
27%
0.774
0,195
0.089
0.052
0.061
BU032
0,090
0.024
20031
6.011
0.774
0.024
0,00)
0.047
0,991
0.007
0049
20099
-0037
00017
0.083
0.150
00003
0026
0.049
16/
0.103
0.127
0.118
0.500
0.027
0632
0/02
0:030
3621
0/224
0/889
0.019
0.02
0.009
0.085
D627
0/47
3642
0/020
0.855
0.043
0.166
0.9%5
0626
0:53
45%
.0.134
4.155
41159
JU361
.0C53
400K3
.0042
-2/47
4.01
-0194
40188
4U10)
41068
4L115
40033
-3677
-2817
-0171
-1104
.1211
-0077
J0638
-0075
54%
0.168
4.191
4.185
41091
4.103
40052
0198
0629
.0090
4,023
.0183
41306
0.091
65%
0.2M
0.316
0,622
D082
0.107
0:055
5051
0.134
0.044
0.119
0,182
0010
0,345
3.14
0,87
0.004
0,141
0.37
0,012
W05T
0.114
72%
0.245
0.265
0.299
0.148
0.173
0.147
0.139
0.110
3.159
4.065
0.161
0.112
4.115
0.201
0.255
0046
0.156
-0117
4,014
0.157
0.135
0,228
0,006
0085
0.151
81%
-1
40311
-1.21
-0.10
196
133
-0.
54
4.109
4.116
-0123
4,074
-0.09
4.155
129
40 168
4.354
-044
4.345
-0155
4.054
-0.17
40213
-0223
-0,53
-0063
-0.12
50%
41318
4.362
.1.884
40.600
150
-0366
4.303
4.338
-0326
4.174
4.22
4.20
41
40.29
41 31
40312
-0217
2,05
0.287
4.323
41.66
-0004
-0683
-0:32
1006
-0337
41,36
41472
4.210
0.312
4.197
4.351
-0.37
4.104
-0336
-0.35
(i) Model performance gap A of RD experiment under ROC-AUC on Heart dataset.
Degree
LGB
XGB
CATB
IEN
DAN
MLP
NODE
R135
SW]
CAP
NET
GDS
INT
DCN
FTT
GBO
5AT
SNN
ITF
TIBB
NCA
LMA
TLLM
UNI
AVE
9%
-0.011
400000
.0.08
00008
.0.06
-0005
-0.007
-0/05
-5005
0.001
0.003
-00055
-0.057
-0.08
-0.018
-0601
-0066
-3013
0.005
-0.005
-0.00
0.164
0.019
-0606
-0001
18%
-1122
-1001
-1002
-0.08
-0.011
-0:01
-1.00
-0011
0.001
-4.00
-10071
-0.021
-0.052
n612
-0012
-3613
0.015
-00015
0.154
0,015
-0015
-0/10
27%
.0113
-0.155
.0167
.0.50
.U.61
-0020
-01147
.0:211
-2021
4/01
-0133
-01121
-1.08
-0.029
-U.52
.B010
-0023
-3044
-.000
-01113
.0.02)
0.104
0,005
-0016
-002T
3//
.01144
.1102
.1154
.1042
.L057
-B015
40002
40029
.0621
4.014
.0157
.0123
.0186
.1087
4LDEU
1/025
.0023
-3049
.CJ125
.0164
+1105
0/012
-0014
-0025
49%
.01074
-0.072
.0.05
.0061
-U.00
-0.05
.0044
40041
-0636
4,000
.0844
-0.44
-0.002
-0.055
-0.094
-0014
.0019
-0/80
0.06
.0853
40.45
0.004
0.044
.0834
-0/145
54%
.U1B
-1007
-1.00
40162
-U067
.0044
401155
40052
-2051
4/037
-0043
-01551
41140
-00677
4U068
-0614
40650
-0.05
0.667
-0175
-1.06
0.002
-000
.0621
-0060
63%
-0.124
4.129
41131
AU057
.0064
40135
-0167
4L185
410801
4L105
16026
-0.30
01225
-0190
-U032
-0085
125
0.169
0.171
4.172
0.155
0
0.104
0.129
0.116
6,92
0.102
0.105
0.23
0.164
0.17%
0032
0,305
3141
0.61
0.32
0,108
0,015
0/12
0023
0.13
815
0.221
0.229
0.211
0.13
0.17
0:21
0,192
0.158
3.119
0,009
0.151
0.130
0.211
0.152
0.147
0063
0.133
9.29
0,25
0.12
0.194
0.003
0/011
0043
0.149
90%
0.315
0,318
4.330
0.307
0.262
0240
0,316
4.366
0.24
4,218
0.182
0.256
4,312
0.295
4.296
0. 0.16
4.215
-0261
0.22
-0277
4.292
1.155
-0090
0005
4.244
1096
4..46
4.440
4.467
4.46
43313
-0463
4.430
40446
-0473
4.43
4.200
4.473
41 3/9
40467
4L422
-0301
40477
-0413
0.115
20490
.0.79
-0.54
4.053
-0660
4.343
38

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 17. Model performance on MiniBooNE, dataset. 
(a) Model performance of SC experiment under accuracy on MiniBooNE dataset.
Colazas
PCC
LOB
XOB
CXTB
PPN
DAN
MLP
N40E
R135
SW]
CAP
NET
005
IN1
DCC
FIT
(180
5.41
55.34
T1P
TABR
NCA
LMA
'LLM
UNI
AVE
-0.04
0.925
0.935
0:925
0:925
0.54
0.543
0.792
0.11
0.573
0.53
0.893
0.883
0.772
0.772
0.906
0.967
0.873
0.58
0.870
0.869
0.570
0.570
0.902
0.902
0.504
0.504
0.888
0.885
0,858
0.858
0.300
0.532
0.914
0914
0.634
0.834
0.862
0.63
0.520
0.918
0934
0.540
0.567
0.923
0.915
0.914
0.915
0:546
0872 0,822
0,891
-0018
0.923
6925
1142
0387
4353
0380
0:758
0.890
11:63
0336
03867
0390
0377
0.811
01211
0:909
01811
0.4hh
1991
0928
0.371
08%
0.04
0.51
6.933
096
0.17
0.71
0.81
0.772
0.906
0855
0.850
0.50
0.88
0.500
0.87
0.856
0.59
0.909
0634
0.861
0917
0536
0.00
0315
0816
5
6.026
0.043
0,915
0,899
0.710
0.892
0.851
0.568
0,890
0.855
0.877
0.525
00111000
0.79
0.27
0,59
03315
0372
6-
7
0.077
0.088
0800
0911
0919
0.611
1871
199
0911
0853
1100
0.81
0981
0.71
0.73
0.661
0.67
0858
1896
0.771
0.859
0351
0.59
0.90
0.899
01977
0:50
0887
0888
0.868
0-887
0.50
0.901
0911
0362
0861
11000
0980
0900
0.99
0.001
0491
6515
0517
1880
10
11 0:85
8-
-0095
0000
0M21
0.917
0.925
6000
0917
0450
0035.00
2010
0001
05:5
0.572
0.003
0.80
0292
0.772
6.712
0.710
0.977
0.662
0.906
0558
0182
0.56
0.870
0.864
0.870
0,60
0.66
0.71
0.500
0.90
03900
1300000000
0.561
0396
0.07
0984
0351
0.014
0007
6.04
0.25
0.655
0.54
0361
0361
0920
0.005
0:335
0.065
0.59
0653
0216
0365
0535
08%
0091
12
6037
0.923
0.26
2004
0.522
0522-
0.90
0.772
0.967
0.58
0,870
0.70
0.002
0.500
088
0:855
0525
1 0862
0:30
0:06
0518
051e
13
40097
0060
0274
0.71
0.800
1199
0:663
0:50
0:911
0:00
1922
000
0516
1887
16 6.241
14
17]
0.017
0040
0994
0915
0910
0015
0-15
00100000000000000000000000000000
1966
2011
020
0581
0.77
- 03711
0.712
6.772
0.90
0.900
0.966
0.907
0857
0508
0858
0:300
0,510
0.560
0000
0000
0820
0800
0981
0000
1000000000
15561
088
0955
0:388
0350
0800
0:011
0:011
6334
1855
0961
1081
0263
11000
0000
1050
0220
0.30
03311
0.00
0310
0515
0314
0315
0515
0344
000
197
18 0.17
0917
1971
6.77
0
0858
0:000-100
0820
0.00
180 0:966
6:311
1322
1000
0.325
85377
0-10
0515
188
19
0042
0.83
0.86
012
0
6.72
0.96
0.57
0000
0290
0:000
8:00
0:18
0223
0:04
0220
0.80
020
21
22 6042
0.442
0925
0001
0000
0155
0.712
0.11
0,772 0.967
0.400
0857
1858
0.10
03001
0991
098
0315
0383
0.34
0965
0950
1000
0.04
0.200
6315
0514
0510
0874
21 00042
0004100
-0772 0.712 0.90
1887
1:000
1
0:888
0
0000
6301
14224
1082
1100
1050
0381
0:16
0597
24
0945
0863
0292
0
6:977
1888
0.00
1800
1
1885
1000
0.221
1517
0511
25
0044
0847
102220000
001110000
0200
0922
0200
0.77
6.72
0,11
0.90
0.000
0.906
0857
0557
0.070
0.810
0,510
000000000
0800
0800
0.000
0209
0:48
6324
0001
0263
0000
0000
0.30
0.42
0.211
0:81
8356
8315
0314
0342
080
096
24
0:011
0015000
699
0954
1061
0581
0.713
0.907
8847
0800 0000
1
0350
0:01
0:01 1845
100000000
000
6.00%
0315
0517
74
0915
8.7
9611
1611
1887
0130
0:406
8857
0.000
0:00
1
6:31
0
1000
1000
0.23
0
0-155
0516
1884
30
0044
0.225
6026
0122
0000 6.71
0:966
0.57
0:200
0200
0:00
0:18
0:04
01244
0363
0.04
0821
51
0666
0200 0277
0311
0
1187
1020
1084
0003
1100
031)
33
0045000
0
0346
1924
0155
0.772 0.907
0.11
0300
0991
0388
0351
6304
0354
1982
0.76
0.59
0315
0541
0865
54
0:044
0915
0004
0
0.712 0.900
1887
0901
0
1822
1
11824
1982
10000000
0912
33
0044
1
0291
0.71 0.60
1187
0820
0:000 1099
1885
1953
0:877
1922
1281
0263
4529
36
26 0,46
0.06
0.45
0914
0.62
099
0015
061001000
1951-
100000000
0866
0985
0.212
0.733
0.112
6.96
0.00
0.055
0857
0157000
8847
0.500
0364
1963
0.00
000000000000
0:00
6561
000
0311
0004
0:01
0001100
1851
0361
0
6.76
0.200
0920
0.00
0.50
0716
(315
0315
0516
087
088
030
4) 0,968
0.015
0,12
0000
0015
0293
0.067
0557
0,800
0550 | 0:501
0:500
0000 0885
0:657
0233
0:04
16824
0 1936
0222
03550
0618
0
41 0948
0.776
6.724
1:200
0201
0523
6.712
6.906
0.58
0,710
0:001
0:18
082)
0.04
11544
052)
0.00
0017
0212
0.00 0
44 6,000
0.50
0915
0.914
0014
6472
6418
0010000
0.52
0581
0991
0.77
0.51
0,722
0.902
846
1887
085600
0.871
0.869
0507
0364
0567
0000
(
0.58
0
036
0351
6311
0354
0361
150
1509
0.55%
016
0314
0917
0842
0571
0875
45
0.01
0925
0001
0911
0381
0271
02712
0.906
0185
0.50
0300
0808
0387
082
5954
1622
0361
1652
0931
0912
0.50
085
M
0004
0300
6.89
1196
0.8%
1065
0280
6700
6.111
0XIT
0922
0.20
0278
1921
0:599
1919
0811
0.80
896
47
0:350
0.92
0974
0.62
0916
0.5
6.665
085
0.67
0.67
0.87
0.01
0.81
0961
0827
0912
0519
0858
0919
0.37
0.593
8947
0.75
0.910
0000000000000000
0.49
0925
0.91
0,900
0.717
0.887
01345
0.865
0.867
0,995
0.87
0
0316
0.01
0551
0.82
0995
0.330
0.559
0,914
0,944
0883
90
0.177
0917
0.417
0.41
0.907
085
0351
0.76
0.661
050
0.315
0.66
0,500
0.576
0.33
0.800
0773
6.866
0781
0.83
0.00
0476
0.566
0.14
0441
0887
(b) Model performance of SC experiment
SCexperiment under ROC-AUC on MiniBooNE dataset.
PCC
LGB
XGB
CXTB
PPN
DAN
MLP
.4006
REES
SW]
CCP
NETT
.005
INT
DCN
FTT
0180
5AT
55NN
TIF
TABB
NCA
LMA
TL.M
UNI
AVE
0.975
0:975
0.985
0.973
0.563
0:951
0.875
0:566
119335
0.539
0.564
0.963
0.564
0.955
0855
0.911
0:970
0.910
0.534
0.973
0:583
0.550
0.914
0:590
0.95
.0004
0.975
0.955
0.964
0.947
0.44
0.955
0.875
0.45
0.010
0.944
0.964
0.964
0.955
0.955
0.903
0.970
0.10
0.534
0.973
0.981
0.561
0915
0.935
2
0.033
0.955
0.975
04075
6.744
0475
0.5
0985
1961
0967
0.75
8947
6455
0.456
03875
6.966
0.465
0991-000
0537
0.60
0.64
0963
0963
1.99
0.61
0.955
0951000
0.655
0.90
0910
0.99
0.969
6.97
0.953
0991
0492
0981
0933
0.596
6315
0.16
8915|
0914-0
0:500
0:580
5
0.026
0.935
0.275
0,72
0993
0951
0.510
0.059
0.915
0.92
0957
0,500
0947
0:44
0.969
0.503
0.044
0.973
0.51
0.376
0,915
5
0.227
0.975
69075
0.211
0.63
0,815
0:966
0938
0263
0.663
8.66
0.855
0
0:970
0.90
0.934
0573
0.983
0.99
0.991
7
9 0
0.000
6971
0055
0915
10000
0011
0651
0651
0315
0515
0496
0.66
0966600
0555
0054
0.11
0363
0363
0362
8564
0.00
0555
0555
0955
0505
0.00
6300
00000000
0316
0N71
0
0053
0.011
0061000
0.555
000000
0316
0495
0550
0.05
10
40005
0994
1899
1991
1092
0308
0001
1990
0835
1992
15500
0:99
1M0
0966
16566
0931
1
0915
0599
11
0085
0974
0976
0694
1001
0.63
1963-000
0557
0:566
0985
0019
0683
0:663
0566
0.855
0555
0510
6900
0910
0.34
0.833
0.00
0650
00000000
0.95
0.93
12
0087
0.975
6975
0000
0021
0963
0957
0.96
0985
0.661
0.055
0955
0955
0000
0.200
0000
0.73
0:00
0:04
0312
0550
0.00
13
40007
0070
0000
0000
0000
15 6011
0951
0012
0
0857
0815 0006
0966
00066000
0411
0411
0664
0963
1 0455
000
6.30
B910
0011
1053
0453
0450
0310
16]
00411
0015
0000
0001
1065
0553 0515
00066100
1055
08229-100
0003
0365
0005
0955
0508
16910
19100000000 0004
095)
0599
17
0017
0924
6511
1891
1901
0411
1811
01555
6300
8996
89m
891
19
6042
6042
0
0.574
0.71
001
0000
646
0965100
0005
0016
0526
0000
0055
0000
6380
[600
(310
0004
0000
0.01
0.00
0.30
00000000000
031
0310
0550
21
0042
0015
1
1894
0551
0515
1003
1063
0:055
180
001000000000
0054
1
24
26
23
0043
0042
0975
0.755
0.975
007
0975
0000
0
0681
0000
0367
0960
1963-100
0557
0875
0
0:66
0366
8066
09966100
1000
0985
0010
0419
0625
06214
0.663
0.64
8561
0.005
0005
0655
03555
0555
0965
0000
101
0610
000
0000
0.000
5300
6:00
0000
0910
8910
19100
0010000000000
00000000
0011
0000
0034
0074
1053
0.001
0.66
0911
0.00
8559
0570 0000
0316
0550
0580
0000
099
0
0
27
00045
0015
13000
0007
0553
0515
0:0561000
1000
0.229
0365
0205
0935
0508
0000
16910
0004
0304
0599
28
0011
0975
0891
- 0
0115
1901
01555
6330
0910
11073
0380
0118
30
6045
6045
0.71
005
0000
0055
1000
0000
0000
000
0819
0563
1064
0055
1000
0000
0.500
0950
0010
0910
1000
0060
0.00
031
0490
0500
33
33
0043
0005500
0.75
109
6975
1
1004
0867
0963
0557
0675
0
0066
0.66
100
0985
0019
0894
0800
0:03
8566
0005
1063
0555
0610
100000000000
0300
6300
16000
8910
0004
1
0:27
8558
1616
1312
0550
0.54
0044
0.933
0007
1900
1903
1981
100
6:20
1696
1983
0:01
6590
3
00440000
1000
0000
0300
1010
1000
1000
09400
6045
6246000
0975
0015
6075
0
0.50
0365
0905
0010
0369
0010
0034
0901
0
0
0.066
1054
0
0515
0.066
1000
0365
0065
00955100
0005
0000000000
11100
00000000
0014
19972
0355
0912
0000
41
0010
6046
0016
0050
0995
0991
893
0415
6916
0000
0415
1864
1890
0577
0501
084
8967
0951
0315
0315
0405
0.65
0365
0065
166
1192
199
0411
0410
0419-
0
0
08:1
0361
1365
0366
0.95
0955
64555
0955100
0.70
0300
6370
6300
000000000
0000
[1000
0310
16100
0811
0004
10%
192
057
0960
0618
0318
0310
0313
0311
110000000
0590
0499
000
099
44
0050
0934
0.074
0.74
0880
0
0557
0.964
0.34
0039
0962
6963
0.835
050
6969
8509
00000
0004
0.972
0983
0.53
1916
0.96
47
0.250
0281
095
0.93
6487
0/14
1966
0300
(140
1983
0
0061
0996
4954
0333
0919
096
0992
0.60
0961
036
0.66
0.95
0.85
0981
0951
0195
096
000
6370
036
0.500
0934
1009
0075
0962
0.90
1310
0350
002
6409
0.91
0055
0
0350
0/244
1093
03915
(199)
0.35
(1599
0.084
0002
0
0335
0003
91
6:77
0990
0:410
00401000000
8
09%
0361
6:41
0999
0419
0921
0.9%%
0.909
0419
05/
649
BMA
0.80
0:89
0475
0915
0:50
0916
39

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments 
Tee-MMxxprimmne experiment under accuracy on MiniBooNE, dataset.
(c) Model performacee of MC-M
LG0
NG10
PIN
BAN
MLP
N000::
SW1
CAP
NE:
GUU
INT
DCNN
FTV
(RR0
5A1
SNN
T-V
TABR
NCA
LMA
VILMM
UNI
AVI
08
0.75
0.955
0917
0.873
0.885
0.777
0.917
0858
0.870
0.70
0,900
0.90
0.901
0.848
0.488
0837
0.14
08844
0.867
0.9%
0.400
0,933
0914
0.16
0:807
0.61
0017
0.441
0.58
0.836
0.51
0.765
0:861
0.442
0.835
0.565
0.350
0.575
0.866
0.540
0.73
0.896
0.784
0.13
0.004
0.25
0.914
0.944
0.945
0.866
866
0
0.687
0.44
0.05
0
0.991
0.27%
0.87%
6.799
0.738
0:599
0.763
0.566
0.653
1.568000
6.79
0.26
0.778
0.355000
0.762-00
0.761
0.763
0.00
0.700
0.740
03377
0.710
0.545
0.590
1964
0.801
0.358
0.341
0.858
0.671
0.59
0.565
00.19
0.916
0.00
0.942
094
0.511
0.792
105
0.75
0.756
0.822
0.011
0.572
0.550
0.719
0451
0.095
0.550
11:41
0.530
0.50
0.508
0643
0.711 03533
CANI
12%
14%
0.6
6267
0.77
0.165
14110-100
0.00
0.776
0665
6978
0.5
1650
0350
0.500
0.550
1.500000
0:587
0:544
0.09
0.01
0.64
0,63
0.995
0.96
0.347
0.513
0.50
0.590
0.534
1158
0.523
0.127
0.33
0.505
0.00
0.00
0.513
0A37
0.645
0.725
0.722
0.820
0.6) 0681
0,00
0.648
0.361
0.55
0679
0.22
0.49
0.34)
01.31
0.319
0.50
0.79
(.992
0A54
0.550
0,610
0.510
0.000
0411
0515
0.55
6.534
0554
0.67
06610000
0.09
0.51
0.50
0.505
0153
0351100
0.110
0511
0.791
0.51 00010061000
0619
0.:00
0.539
0.551
0:51
0.907
1-907-00 1621
155:-00
0547
0.50
0.53
0665
00650000
0418
0418-000
0.00
0554
0.56
0.51
0.11
0.11 0.53
0.07 0.%1
00010611
0617-000
0.96
1887
24%
0.001
0220
6:25
0250
6500
0900
0.38
0:00
0.002
0.55
0.233
0.500
0537
05:77
0.09
11597-00 0558
0.62
0.50 0123
0226
0.578
30%
0.30
0100
0.509
1550
0.00
0.00
0.50
0.302
000
0
18
1581:00
0:355
635
0.526
0.630
0.65
0.99
0.00
0.107
0.07
0.59900 0.55
0.594
0.525
0.514
0.765
0.339
0.30
0.34
000
0.00
000100::00
0/000
0.451
0315
01966
0093
0:535
C361
0.552
0241
0.05
0.000
15111000
0.550
8515
0
01567
1901
28%
0.000
0.000
1220
1800
0.70
0.00
0714
0.72
1200
0.50 0333
0
0.50
0.50
0:501
0228
1000000000
100
0541
0291
1231
0.00
18200
0.520
0324
0.22%
0515
1354-0000
0017
0
0:00
0.30 0368
0200
00000000000
0.00
0000
0.30
0:500
6:511
0.66
0189
1007
0000
1907
0.900
1550
6711
00000:000
6:55
6301
0.400
05.00
0.11
03318
0.00000000000
1:000000
0587
0.967
0
0:568
0.785
0:31
1956
45%
06300
1520000
0:33
0.52-
122-000
890
1000-4400
0/06
0599
0.577
0.225
0.25-000
15:3 0
0.995
0:556
0.721
1007
1220
00:00
0:201
0220
0:229
0:200
0.225
1225-000
0420
0560
0:27 0.223
0518
0:87
0.29
0-500
0.778
0002
0.599
0.00
03:00
0
0:30
000
0:51
0
0000000000
0008
0
0350
1226 0550
0
0.00
0.50
1003
52%
0.30
1200
0:561
000000000
000
000
0:34
636
0303
0631
0000
0550
0351
0.30
0.00
050
055
0.00
0.961
0
0.300
01811
0896
13000
1820
1600
0 000100600
1
040
0/00
0580
1151
11541 0328
0598
0.500
1086
08:0
(1500
(1591 6/00
8908
0:000
0360
1655
0222
0.28
13:1
0:11-
0.00
0.70
58%
1.00000
0200
0.60
0:301 9407
0.54
0.511
0100
0.55
0.520
0.20
1.01 0310
0.50
0.00
0.905
0.508
0.903
0000
00000
000
03345
0.550
0400
0359
0223
0351
01300
0
0230
0330
0.30
68%
66%
64%
0.000
0.50
0.00
0.900
0.500
1550
1220
0550
00000
0.905
01202 000
0:50
0000000000000
040
00006000
0630-0000
0:44
6:5
0:5m
6367
0:00
0:516
06335
8813
0/06
0400
0.000
0.59
0539
0.500
0.50
(157)
15511 0.22
0.500
0.550
0327
05:1
0.11
0.51
05:1 0.119-00
0511-00
0:31 0.000
0:11
0.29
1.829000
1:500000
0.57 00000
0.587
0.00
0.981
0-583
0.577
0:570
0.98
0.789
0.725
0.781 0586
0220
088
088500
0.397
14%
700
0.000
0.500
0.00
0.00
1500
0.000
0500
0301
0.00
-000
0000000
0.500
0.77
0356
0,634
0.69
0
0550
0.50
0555
0.5%
0.57
0.500
0.500
0.11
0:11
0.11
0.000
0.500
0
0.587
0
0 00388
0.00
0.309
0555
0.400
0.00
0:591
0
0900
0.516
0.803
5615
11907
0:774
0365
0431
8.715
0.59
78%
0.500
1.900-00
0.20
0:901
0.50
0.501
6467
0029-00
0.50
0.57
0.11
0513
0.570 0.544
0.307
0.724
1.5000000
0.98
0.785
0.84
0586
0.592
50%
120
50%
0.500
0.500
0.00
0550
0.000
0.17
6505
0
0.50
12200
0.501
0:500
0000
0.007
0.00
6407
6407
0.97
0.50
0.51
0.50
01310
0.517
0.57
6330
6322
(330000
0.561
0.00
0:501
0.61
0.55
0.5m
0.56
0551
0.05500
0514
0.500
0200
0.5%
0.75
0.533
0.51
0.51
0
0.066
0.20
0700
0.30
0.75
0.50
0.906
1.30-0000
0.60
0.588
0551
0
0.50
0.90
0.558
0.11
0/36
06639
0963
100 0943
0858
0.377
0.567
0
88%
0.500
0
0,500
0.94
0.500
0.500
6497
0.57
0.514
0,500
0,500
0.55
6.55
0,50
0329
0312
0619-00
06305
0551
0315
0851
0.350
90%
0.500
0.30
133000000
6-61
0.50
125000000
6.99
0.511
0.118
0.500
0.34
0.344
0.560
0.50
0400
0,50
0.28
0:311
0.22
0.900
0.903
8.44
1881000 0556
0.348
94%
96%
0,500
0.500
0.500
0,500
0.30
0.501
0.00
0.500
0.500
0.99
0405
0496
0550
0,530
0
0.36
0,50
631
0.5%
0.406
0.54
0.30
0.97
0447
0.415
0.500
0
0.501
0.5%
0.59
0.005
0.35
0335
0,50
0.53
0
100%
9RT.
0,50
0,500
1,50000500
0.500
0.500
0.00
0.500
0.500
01500
1.5000000
0.007
0.97
0.501
0.40
0.407
0
0.407
0.947
0.527
0,497
0.477
0.501
0.02 0.497
0.497
0.4970000
0.491
0.497
11/66
0.497
0.407
0496
0.87
0.497
0.511
0.501
0.49
0000000000000000
0.50
0.505
0
0.94
0.303
0.500
0.50
0
1
A515
2515
0.66
0.874
0889
0.53
0.531
(d) Model M performance of MC-M experiment under ROC-AUC on MiniBooNE dataset.
Diegroe
1.3B
XG31
PIN
DAN
MLP'
NODE:
RUS
SW11
CAP
NETT
GUS
INT
IXNN
FTT
(RRL)
SAT
SNNN
111'
TAAR
NCAA
LMA.
VLLMM
UNI
0:905
0.013
0997
86
65
4%
0,570
0.96
000000000
0.570
0,570
0550
0.90
0.97
0.993
0.983
0.979
0.000
0155
0
0.854
0.899
0858
0.30
0.881
0351 -6.7
0.410
0.766
0.851
0.419
0.034
0.408
0377
0.556
03655
0.927
0.800
0.310
0.919
0.864
0.857
0.861
0,023
0.665
11892
0.239
0.898
0.938
0.852
0.910
0.949
0.575
0.881
0.909
11874
0.17
11842
0,919
0,874
0.554
0.455
1.718
11991
0.840
0.635
0.911
08554
0.958
0.916
0.869
0.629
11700
0.748
0795 0.915
0.951
0.964
1.936
0
0.975
0.966
0,570
1101
1.13300
- 11900
0.514
0.000
0.955
11965
0.59
0.931
0.854
12%
14%
0.94
0244
6.60
0.90
6.00
0.710
0.307
0:700
0.801
0.751
0
0.793
0.763
0.67
0.654
0,677
0.736
0.755
0.674
0.110
0.516
0363
0.533
0.57
0.375
0,573
0.033
0.003
0667
0.000
0341
09933
0.004
0
0.900
0,000
0
0
0.705
0.705
0.570
0659-000
0.777
01777000
0173000000
0.765
0.675
0.678
0.731
0.65
0.505
0.30000000
0.55
0.53-000
0.574
0.24
0464
0.000
0.03
0.05
0.823
0.80
0056000
18%
0.44
0054
0.705
0676
0.765
6765500
0.715
0.715
0.766
0.677
0.722
0.727
0.655
0.655
0.700
0.564
(6623
0462
1
1196
24%
305
0815
0821
00000
0300
0:581
000
0150
020
0220
0226
0350
0.70
6000
0.22
0571
0.000
0.55
0759
0.44
0.000
0.00
0.76
0.709
6.777
0
0.555
0.64
0.70
0.00
0.60
0.590
0:567
0-861
0.1
0.561
00030040
0610
0679
00000000000000
000000000000000
0607
0,600
0.665
0150
0.21
1520
000000000
0000
0.266
0
1206000000000
0361
1.705
17000000
0.34
0
0077
0222000000
01400
1108
-
00200
0 1911
0.727
0000
0063
0.785
0.704
1.756.00
0676
0.00
0.000
0.000
0.60100000
0005000
0.548
0 0.500 0619
0.462
8881
0166
0865
0.757
12%
1000
0200
1200
0.00
1.715
0767
1125
0000
(1:01
0010
0019
(02)
1983
1150
0.75
0.044
05558
0.005
0558
0
0.71
0.561
0700
0.583
0.50
0010
000
0.500
1094
0.76
-
0022
1
0009
0909
0244
0.736
0.65
1065-00
0
0.255
0377
0.777
02217
0.000
000000000000000
0.705
17000000
8539
0617
0.561
(183
1104
0.200
47%
42%
1655
1821
1255
0.20
0740
07%
10000000
0020
0111
0
1230
0250
0250
0
0.728
000
0.65
0.60
0.30
0.360
0
0.704
0.700
0.005
0266
0266
0.744
0.56
0.717
0216 0:50
0
0.50
0,561
0.06
0.00
020
1705--0 0.564
0
0.354
0.599
0.577
0.00
0617
0418
00000000000000000000
0.00
0000
0
0.44
0.7
0.701
0.
0.90
0.773
0.700
0.50
0001
1611
1524
0
1250
0094
0.60
10000000
0.734
0260
0:311
0
0:716
1309
1094
525
10023
1022
00000
0000
027280000
01385
0250
0685
0.000
1660000
0.727
0.705
0.277
0.755000 0215
0.601
10055000
0:564
0.500 0418
0.746
18815
0.760
25%
1822
0220
0.228
0:220
0.708
0256
0675
11206
0209
0-364
(1559 0018
0199
1.70
1100
000
0.00
0.22
0.000
0.20
0.009
0-444
0000000110
0
0.00
0.700
500
0.650
0.70
0552
0.7%
0.000
0.00
0.564
0.56
0618
00000000000
0004
0.750
00001
0186
1090
0.66
62%
1655000
0.705
0.264
0573
11:00
0200
0000000000
0700
1705-000
[1561
0618
0:745
1L704
(1866
1950
6.61
66%
65%
1624
0000
1616
00000000000
0302
0.
0.200
1700.0000
0251
0
0:20
060
0607
0
8.000
0
0.005
-
0201
0201
0201
0,50
0659
0700
070
0,344
0.60
0.61
0618
0418
0:91
0:95
0.17
0:745
0.720
0.78 0983
015
(961
1969
0000
000
0000
0.044
0.700
0.20
0.600
0.50
0.354
03663
0618
0.06
0221
0350
52%
12500
0.53
0.722
0.00
0
0304
0.555
0.306
000600330
0:384
0.363
0
0165
0:300
0183
1064
165
0
02500
0095
0
0.705
0.568
0.00
0.044000
0.690
00690000
0.705
1705-000
0.833
0336
0
0:514
01855
0665000
50%
200
0.27
(2:5
0557
0.96
1220
1220
0.67
0021
0
05565
0612
0614
0.00
0671
0.46
0.66
0.665
0566
0655
0.300
0.55
06644000
0629
1884-
0.348
0354
0.50
1308
0.35
0395
0290
0000
0.009
0601
03364
0.303
0355
0.023
11731
0.5
(1945
0255
UAM
0
0376
0.565
0A29
2595
11:47
0.579
1617000
0.33
0354
01300
0:344
1147
0843
0843
0119
001
0401
0
0557
0420
0613
0.91
0591
0.66
0616
0.573
0600
06000000 0357
0514
0.51
05766
0347
054)
0946
88%
94%
90%
0.53)
(1.50)
0.585
0.55
0.70
15552-0
0340
0324
0357
055
135%
1
0.58
0:300
0.300
0342
0,529
0.542
0.58
0.3%
0.50
0.50
0338
0.44
0360
0560
0.5%
0.75
0.55
0.50
0.55900
0.50000
0.58
0.29
0700
1355000
0.554
0.90
01905
0300
0.34
0.48
0336
0:46
0.566
0361
0.86
03.50.33
0/06
0.91
0.334
0.00
083
0:31
004
0049
0.573 0
0401
0.60
0.555
0
0:23
0.57
0.57
0.57
030
0.522
8
0.500
0532
0.40
0.50
0550
0
0.500
0.290000
0.00
0300
0:29
0511
0.44
03000
0045
0.561
98%
0.49
0111
0111
0467
0.524
0
0
0672
0.591
0.54
0:125
0.50
0672
0.557
0:41
- 0913
0611
100%
|
0.500
0.500
0030000000
0.500
0.50
1130000000
0.500
0030000000
0.500
1150000000
0.50
0.500
0.500
0.500
0.50
0.500
0
0.500
035000200
0.500
0.500
0.500
0.50
1030000000
0.30
11300000000
0.000
023000000
0.30 0
000000000000 0.900
0.60
0.533
40

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Enviromments 
(e) Model performance of MC-L. experiment under accuracy on MiniBooNE dataset.
Deggee
LGB
PIN
DAN
MLF'
NO00::
SW1
CCPP
INT
FTV
(ORL
5,41
SNN
TABR
NCA
LMA
VILMM
UNI
AVI:
26
08
0000
0
0.25
6.922
1.292-00
0.873
0.873
0.85
0.93
0.777
0.272-0
0:505
0.917
0.875
0.85%
0.350
0.870
0,59
0870
0.900
0.922
0.904
0.901
0.858
0.848
0.583
0.484
0837
0.20
0911
0914
0.874
0.624
0.867
0.62
0.915
0,9%
0.944
0.410
1.973
1.567
0914
0.915
0.16
0.947
0.872
0.897
6%
866
0.022
0.023
091
0.722
0.722
0.43
0.275
0.8%
06.70
0.768
0.768
0.588
0
0
0.875
0.55
0.851
0,55
0.564
0.94
0,564
0.25
0.447
0.547
0.854
0.
0.587
01875
0.573
0,577
0.51
0.906
0.810
0.342
0.441
0.014
0.022
0012
0014
0.11
0914
0.874 0
1
0000
6.76
0.803
0.873
0.533
0353
037 0.50
(1871
11871
0.876
03455
0.913
1000
1996-
6.073
14%
128
0.77
0.14
0.76
0.26
0357
0260
0177
0360
0.76
0.766
0.520
0
0.63
0,550
0.511
0150
0555
0.80
0.700
11872
0.574
0.876
0.574-000
0819
0.000
0944
0.455
0.913
0.31
0.24
02300
100000000
0.65
0.89)
10022
0.272
0
0264
0:71
0539
0.315
0.671
0350-0
0854
0,316
0781
00000.33
0.3
0245
0911
0:33
00010000
0:81
375
24%
0.777
0.712
1222
1206
6.00
6:05
000
000
0266
0202
0:59
0:51
0539
1
0.346
8520
0240
0855 0361
0.544
0854
0:817
0516
0.221
0221
0.66
0837
0.93
0716
0000
0.89
0A2
0.224
0.24
0975
0710
0
030
0100 0
0.111
0.574
05700
0.26
1558
0
0:354
0000
0:711
635
0945
0.34
0244
0302
0356
0,318
0.204
0.815
0350
0.558
0200
01594
0.16
0352
12%
1
1
0:550
1000
0311
1
03315
1885000
0335
11555000
0:374
0.744
0838
0356
1811
1919
25%
0:11
0:999
1189
0:118
0228
0:51
6277
1222
0.31
0.20
0000
0220
0.10
6322
0.67
0221
0304
0264
0556
08:7
0370
0.744
0220
0533
0-50
0503
0.54
000
0.772
1230
0212
1986
0.356
0.379
0220
10000
10611
45%
00000000
1855
0333
8955
03355
0811-
1851
0355
0919
0:375
0.74
0.437
0:551
13411
01511
45%
00500000000000000
0.70
0:32
03R4
0111
0339
1985
0.556
0.55
0:315
06704
02:0
0:31
0:50
1805
3183
0001
6.708
0211
0:356
0:11
0.92
0220
1200
0
0:300
0334
0 0000
0.61
02200
00000
0.00
52%
0:547
0350
0:34
03555
0336
0.03
0305
0:30
0003
0866
0096
0300
1822
1
8812
1885000
0:334
0
1801
203
2312
0:112
0891 0812 2012
1885
0:110
0828
0-96
2113
1881
588
0267
0:500
0312
0.708
0.000
1860
0272
2307
0864
0.54
0865
0:73
0.000
0,72
0228
0533
00:00:00000
0.500
0.60
0.00
0300
(1000)
1000
0212
0328
0004
02866
10855
625
0960
0
0045 0812-00
0399
0865
0351
0955
0:344
0.733
0
0.537
6:11
0955
1887
0306
56%
0162
6115
03R4
1
0:541
00010027
0
0295
1885
0.553
0:314
0438
0335
0:56
1815 (1eh
1089
68%
0
0217
0265
0:553
0:144
0.92
0228
088
0:001
0.00
02300
0.80
0.00
0300
54%
0758
0960
0344
0354
0.553
0.54
0.500
0.00
035
0003
0355
0136
0003
0302
50%
199.
188
1
0.666
0.700
1020
0.000
6.0
0.755
0.700
0.70
0.70
0.700
0358
0
08600000
000000000
0.342
0:312
0.990
0271 1701
0
1
0000
0863
0.61
0.500
0383
0.551
0.22
0881
0.700
0.568
0:170
0.20
0.700
0
0226
0377
0433
0.00
056
0.007
0,47
0.564
0.500
0.700
0.45
0.785
01335
1195
0007
0004
0.50
0.530
13340
6.67
(36
0.345
0.343
0.59
0
1361
03345
u850
1999
10:0000:440
0:366
07700
0.50
0325
0
6.97
0:57
1.771
0.64
0539
1067
0953
88%
(..00)
0.509
0:260
0325
0339
0250
1989
0:558
16785
0231
0556
0353
10000
6.77
90%
0..00
0.505
0222
6.00
0.787
0:66
220
2200
085
0.55
0:20
96%
0.50
0.00
0.303
0,500
0.602
0.300
0.34
036
0.76
0.77
0.79
0200
0.20
0.705
020
0.99
0,65
0.001
0.63
1700
0.530
0.54
0
0.776
0.775
0
0,000
0
0,501
0714
1964
0.60
0.500
(.500
0.734
0.07
0.72
0.987
0.62
0.27
0.57
0.671
0643
0431
0
0713
0733
0.747
0.01
0741
0
0536
0.400
0875
0.00
100% 100%
0,500
01:000.30
0.500
0.500|
00500
0.500
0.50
0.99
0.50
0.501
0.44%
0.497
0,501
0
0.30
0
0.407
0,497
0.407
0.407.00
0.497
0
0,477000
0.49
044700000 0.501
0.99
0
0.503
| 0.500
0.503
0,508
0,865
0.844
0,531
0.531
( Model M M of MC-L www.eeeppemmnttcm unperrr under ManterroC-AUe ROC-AUC COOC-AUCOnMMMM1 on MiniBooNE
M dataset.
Dlegres
1.3B
X1:0
PIN
MLP'
NODE:
RUS
SW110
CAP. NETT
GUS
INT
IXN
FTT
SKNN
111'
TLLMM
UNN
4%
0.975
0.975
0.05
0.975
0.975
0.994
0
0.94)
0.543
0.55
0997
0..77
0.571
1.875
0.96
0.965
0.955
0.530
0.11
1911
8,044
0.4%
0.653
0.956
1.354000
0.950
0.961
0947
0.955
0.355
0.963
0.955
0.96
1901
0.570
0.567
0.570
1189 00000
0.100
6454
0434
0.073
0991
0.981 0.99 00000
0.474
01561
1.990-10 0914
0.9.5
[.989
86
65
0.92
0.000
0.24
0.994
0.944
0
0:225
0048
0.371
0.870
1 0.955
0954
0.46
0945
0.927
0.921
00000055555 0.95
0.94
0.946
11966
0.991
0.500
0.91100
0.888
1888 0.996
0,89
0.633
0
01887
1.977000 0921
04333
1.090000
0.13
0913
1911-0
(1
11900
0.099
0
124
10%
0.97
0.74
0.76
0.98
0.93
0453
6931
0047
0.86
0.365
0.962
634
0:902
0.95
0.55
0.943
0.055
0.530
0.001
0.90
0.953
0.954
0.945
0.945
0.949
0.001
1886000
0.500
0.000
0.00
0921
0410
0921
0
0.69
0.51
00000000000000000000000000000000
0.577
1014
0.95
0.58
01860
0911
0.999
0
18%
0.009
00000000000000
0,00
0.356
0.034
0.230
20033
05477
11945000
0,00000
1
0951
0.25
0,915
19944
11955000
0933
705
0.96
1858
0920
0857
0417
6933
0911
09%
0918
0:41
0913
0018
0925
0915
0855
0981
15%
1667
040
0917
0911
0905 0876
0,00
0913
0917
000000000000
0910
0.90
0.2%
14%
0006
0344
0.60
0.550
8004
0341
0,500
0040
0210
0000
0.22
00000000
|
001100001
1966
0551
1617
030
00041
0126
1017
205
8557
00200
0321 0351
0955
6933
0.06
0.935
0.6 0020
0582
0937
0.00
0.87
0945
0012
0042
0006-00000
0.000
0077
12%.
1997
0000
0394
0
0336
0942
2851
1001
1913
0000
0000
1000
1000
0244
0013
0000
0:106
1034
100
1122
0200
0.53
0355
01055
0303
1000
0344
0010
0:389
1067
0910
1561
0
0.025
10055000
0315
2025
2857
11015
0957
(100
00000004
0:44
0514
0:117
1055
1819
0
0:501
0.25
0312
0326
1811
0903
000000000
0965
0510
0551
1975
0.00
0200
0935
0801
0.00
0.00
0980
0810
044 0230
0,000
0.5
000
1001
0955
0.15
0000
0005
0000
0000
0014
1611
0915
051 0531
1
B951
0511
1811
0007
[100
00000020
00441
(3440)
0:71
1968
525
0517
0:561
0365
0.024
0314
0060
0534
0035
0:53
0000
000000000
0.45
0016
0000
0:114
1852
0166
1067
0917
22%
1840
1012
(200
0:50
096
0364
192h 0962
000
1913
0900
0000
1000
0300
0010
0011
02000
0:53)
1965
500
000
0200
035
0313
11014
0303
0000 00361
0000000000000000000000
0014
0.400
0063
62%
0110
1 0:50
00044
0912
0861
11014
0057
(100
0:244
0515
2009
0:340 18999
1133
1094
66%
66%
0240
0
0316
0363
0365
0.344
0931
0241
0011
0330
0980
1914
0933
0
0361
0518
0010
0.00
0929
0:155
0960
(961
0917
680
1001
0311
0004
0000
0000
0015
0000
0.794
1
0919
52%
0
0083
0.111
0525
0004
0951
000
000000500 (00111030
0.442
0
0:557
0185
0002
145:
8045
1521
0
10955000
1.911000
6.25
0526
0955
0951
000
00000001
0341
0015
0004
0.892
0133
0.70
60%
200.
0550
0310
0518
1
0.64
0
BM5
0533
0311
00911000
8,055
054)
0261
0525
0955
1905
1975
0000
0000
(000
1107 03400
0061
00000000000000000000000
0341
0:41
0.86
0915
0315
0514
0:409
0,597
0.740
01000
0920
55%
11914
1912
03500
6943
01012
1
0.912
2022
0994
1005
0006
1100
000000250
03557
0915
0921
0:544
(1911
(191
88%
0.91
8915
6029
0517
0.80
0315
0413
0307
0570
1025
0300
0361
0.40
0040
0.92
0.928
0.02
0871
0914
0996
0.966
0.99
1000
090
0937
0986
0014
0515
0920
0915
0,10
0-468
094)
0909
0900
090
94%
90%
0.59
(..00
0612
0.25
0.50
0.66
0881
0.361
0.350
0.800
0.44
0550
0.53
0930
0.46
0.80
0012
2016
0.665
0,30
0:345
0110
0070
0.56
-
0..75
0.765
0.753
0.704
0.70
0.755
0.000
0400
0.20
0259
00000
0.00
0382
0.240
0.20
0.00
0.255
0.79
0.599
0.79
0
0.704
0.767
013
100%
|
0.500
0.799
0.76
0.500
0.799
0
0.50
0.797
01997
0
0.30
0.790
0.700
0.300
0.500
6.79
0000000000
0.500
0
0
0.50
0.79
0.50
0
0
0.500
1
0.500
0
1.50000:0
0.00
115000055
0.500
0.76
0.50
0.774
1.5000250
0.500
0.50000
0.500
0,500000
0.000
03500000000
0.500
0.76
0.300
0/766
0.300
0.500
0350
0.000
0.90
0.500
0
0,500
0.761
0
0.300
1550
0.33
00000
091K
0
0.53
41

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments 
( Model performance of RD experiment under accuracy
p
accuracy on MiniBooNE dataset. dattasst
Deggne
LG0
C'AUB
PIN
DAN
MLP'
NO00::
SW1
CAP
GUS
INT
D4.N
FTV
5.41
SNN
TABR
NCA
LMA
1.MA
VLL.M
UNI
AVI:
08
0.775
0.255
0.87
0.885
0.773
0.917
0.58
0.870
0870
0.907
0.501
0.848
0.484
0833
0914
0.874
0.867
0.9%
0.400
0914
0.96
0.897
-1
0.05
0.915
0.99
0.872
0892
0.772
0:504
0.35
0.350
0,569
0.911
0.902
0.858
0.587
0.229
0014
0.624
0.611
0.015
0.714
0.944
0.945
866
65
0.850
0
0.40
0.925000
0.85500
0361
0.599
01832
0.330
0.768
0.300
0.581
0
0,55
0855
0.899
0.856
0.89
0.611
0.445
0.877
0.877
0.585
0.50
0.94
0.585.00
01875
0.62200
0.562
0,455
0.562
0.823
0.50
0.905
0.815
0.41
0.340
00010
0.012
00.77
0.771
0,692
0,557
0913
0001
0.029000
0:35%
0306
0911
0.59
0.70
0.80
0386
0.544
11872
0.877
03447
11900
63577
14%
128
0.013
0811
03400
0.77
0.277
0.267
635
0.57
0
BKin 0.826
0850
0.00
0.829
0.63
0.29
0.50
0.559
0557
0.55
0,70
11825
0.320
0.819
0.571
4830
08:0
0.000
0.381
0.58
0.00
11799-000
0245
03335
0.00
0.51
0.800
0.02
0.764
0310 0555
0 0.766
0800
1
0.55
60000
00000000
0.000
0.008
0.505
0.591
0
0.755
0:m0
0.577
0991
0381
0.75
6:300
378
0.22
0
0520
0.761
8835
0835-00
0.799
0.00 0520
02299
0.79
0.79500
0213
0.723
0.50
0911
24%
0.771
0.76
0.44
0:01
0220
0:000
0.385
0212
0825
0.29
0.500
0995
0.350
0277
0.006
0,055
0.295
0.60
0.501
0240
0.353
0150
1101
10040622
0.700
0539
0
0.766
6336
0,40
0.336
0344.00
0351
0234
0.356
0.903
0.300
0.000...0
0.703
0923
0.756
0.388
0365
0,396
0024
0.02
12%
1918
0
0000
0:76
0.724
02200
1121
0371
1
0:174
0241
0.72
0.564
1997
25%.
0126
1522
0.67
0000
0.46
0
0.8)
0:310
0.88
0.782
0212
0291
0523
0.20
1124
0.001
0034
0.73
0550
0:871
0.777
0.70
0.771
0211
0280
0.000
0-50)
0.700
0.855
0.764
022
-
0.300
0:340
1872
0:547
633
0953
0:32
0.700
0.755
0735
0361
0
1700-000
03561
0.77
0800
6TT
0:55
0156
45%
1
6.70
0
0.50
0:700
0.338
630)
0.52
0.729
0:42
0.701
0.784
0.237
0.78
0759 0247700
0224
0229
0:301
0020
0.22
17500000
0.55
0.354
0.91
08783
0218 01e:
0.21
0514
0:55
0:1
8:725
1963
11800
(88)
0 0
0229
1200
623
0.70
0.21
0.5
0200
0220
070
071
0240
52%
1545
000
6914
0.39
0
0755
0356
0001
0.46
0318
0
0041
0311
0.911
0891
1723
17255000
0 0.73
0240
1185
588
203
0000
0.30
0x00
0260
0.500
0:00
0.26
0.70
0338
0:21
0201
0227
112]
0.788
0272
0.205
0-134
0.701
15577 0911
0238
0:50
0:81 0:50
0:50
1711
0.50
0.75
11925
0.73
0
0201
0.500
0344
0000
0223
0122
0021
1000
0.96
1000 0000
0250
0200
1000
01999
645
0.967
15511
09600
008000000
0:555
0955-00
0853
0961
0.66
0.356
0.666
0750
00000000
0.6
0:511 180k
0001
0151
0855
56%
0:300
10000000
0391
07255000
0.00
0352 0:50 0367
1325
0355
6290
0009
0356 0:1
0:508
0135
0.85
0
68%
1220
1991
04:0
053
1.231
6:13
0950
0.69
0.60
0.41
0.66
0900
0
0
0.00
0601
0200
2013
0.60
0.5
0601 0650
0330
1000000000
030
1
54%
0000
0.345 000
0004
02200
0,00
0.50
0210
0.51
0601
0.600
0000
0:304
0.9
169.
000000000
0
05500
0361
0646
0.75
0.555
1180
0357
07611
0445
0:333
01350
1876-00 8
50%
185
-.50
0.177
0.499
0.500
0243
0.000
0000
05455
0661-00
0.593 065
0665
0.00
0718
0550
0.51
0560
0
0.6%
0.556
0:50
0313
632
0
0.00
0661
0555
0200
000002233 0.00
0222
0.565
0.94
0511
0.540
0344
0.441
0555-00
0551
0587
0,28
1556
1836
1545
0.586
0323
0350
0:21
0.56
0.564
0.738
0310
0.000
0.633
0.90
0.113
0.999
0.547
0416
0714
0017
0366
0.500
0.95
0.90
0220
0-497
0-077
0.054
0.38
0.99
0.65
0636
0.41
-00
0872
000000000
0.39
6.57
0
0
88%
0.49
0409
0.395
0.350
00000000000
0261
0014
0.45
6350
0.64
0.555
0525
0617
0379
0630-0
16677 0626
0611
0410
B519
0880
0.99
0.994
0.37
0.329
6919
0,62
0:566
6611
8.65
0010
0.57
0.566
0.21
11595-000
0926
0557
0.59
0893
62%
96%
0400
0.499
0400
0499
0.48
0,49
0.599
0.56
0.501
0
00066000
0644
0645
0.504
0655
0.50
0411
635
0.44
0.71
0.50
0.39
0.49
0.50
0.5%
0.49
0.54
0.11
0.01
0.55
0.504
0.555
0.592
0.491
0.59
0653
0.50
050
0,60
0517
0663
0000
0.00
0.505
0
0.521
0
0.50
0851
0.546
0.60
0
100%
- 0.190
0,500
0.5000.00
0.090
0.500
0.500
0.511
0.500
01500
110000000
0.500
0.97
0.07
0.501
0.501
0.525
0.497
0,501
0.197
0.40
1.070000
0.497
0.530
0.501
0.49
0.697
0.497
0.49
0.993
0.497
0.497
0.407
0.497
0.497000
0.50)
0.561
0.99
0
0.503
0 0..00
0516
0300
0,508
0/496
0
A515
0.865
0840
0.874
6.51
0.531
(h) Model M performance RD experiment i under ROC-AUC on MiniBooNE dataset.
Diegroe
1.3B
XL.1
PIN
JANN
MLP'
NODE:
RUS
SW11
CAP
NETT
GUS
INT
IXN
FTT
(RR))
5AA
5NNN
111'
1.M.A
TLLMM
UNN
4%
0.974
0.924
0.97
0.975
0.575
0
0
0.560
0.44
0.01
0.57
0997
0.57)
0.576
1 0.661
0.964
0.535
1091-000
0.538
0.043
1011-000
0.96
0.963
0.663
0.553
0.964
0.611
11955
0.355
0.951
0,955
09:1
1196
0.570
0.570
01900 (1905
1.09
6454
0.093
0.953
0.418
0.933
0.012
0.914
0.915 0
86
65
00.00
0.990
0.99
0.98
0
0553
0934
0.811
0.96
0.65
B912
0.530
0.932
0,933
0.07
0.955
0.055
0.90
0.955
0,90
0.944
1994
0.945
0.945
0.003
1.805
0.567
0.967
00004
1004-000
04225
- 0.55
0.991
8 0.66
0 0.05
3.911
0.807
11977
0.04
0
10%
0.90
0.96
0066
0.78
0961
0934
0.874
0:953
0.935
0.511
0.997
018370000
0.965
0.941
0:945
0.891
0.954
0.90
0
0.988
124
0
0.025
0.56
0.86
0.50
0.31
00050000
0.523
0,953
0510
00130000
0.66
0557
0.50
0423
0051
00010..00 1021-119)
0.501 1995
0361
01899
18%
1
0.225
000000222
0305 1015
00022
1859-000
03932
8826 0
0001
000200221
0:221
01857
305
1850
0412
0353
0919
09190000
0829
0559
0.59
(97) 0000
0978
0:119
0
17%
0910
040
0357
0900
0911
0011
0571 0.91
0.911
0947
0500
0570
14%
000
000
0911
0,333
0005
0500
2013
0354
0.50%
0.00
0810
0000
02120
0.804
0956
0
1555
0661
11855
0951
1151
(3440)
1300%
01550
1195
205
0911
0:55m
0996
0.02
10200000
0915
0.024
00555
[38)
0.944
0.81
0.41
8829
0006
00955
0.506
1811
0.886
0.88
0915
12%.
1915
0511
02:0
caR
0:862
0251
0.000
0244
11720
0:400
0330
0000
0200
1000
0.001
0200
0:39
0.00
000000000
000
0300
1055
0500
0136
0000
0,00
0.00
11964
0336
1000
0:500
00000000
11891
0359
(1336
0:000
0500
000000000
0:354
(1963
1963
1928
0
0000
0000
0
0:115
5:00
0388
0000
0.000
1000
0
0
0500
0300
030
0.03
0.003
0131
052
0855
0360
0240
0.00
0.30
0:267
0.72
0.90
0155
0967
0361
400
0000
0000
0200
000
0330
0000
0000
0210
0000
052
0000
0,000
1108
0.897
(0000000
B1ST
1083
00.9
0151
0:001
1721-0000
0006
800
1001
525.
18800
1021000
0
[ 0706
0553
0879
0:22
5330
0300
0.996
0:501
0:500
0206
0912
0307
0586
118t2
1356
0.00
0.533
(221)00:00
0:10
0.500
18818 0
0241
0:700
0:318
0.729
1163
10056
0355
030
100
0800
0220
0333
0.00
0220
500
0300
0330
8024
0200
0,503
0374
1005
0330
62%
1872
1812
6327
0:541
2016
1181
0357
11235
0:354
0332
0512
1001
1070
66%
66%
0262
0
0
0200
0100
0.103
0.70
c:0
0981
0200
0:163
0500
0212
0331
122
0274
0323
0:10
0.138
0:35
0433
0.75
0.700
013
0551
(1964
1999
0334
0300
0240
0009
0.000
0.366
0.220
0.000
52%
0455
8336
0255.00
0275
0.305
0:30
000000744
0:743
8043
0356
00000
0300
0319
0135
1064
0.77
0.700
0390
072+
0239
0.237
0:700
0
0550
07241
0359
0973
0.335
0266
0:725
05555
01986
1069
200.
0852
1187
002
0461
0.99
0812
1966
0689
0.341
119t
0.77
0000
120
1
0319
0.01
0:11
0775
1990
602
0863
055
1256
1
636
0.55
0.501
0.70
0.764
0.75
6756
0.555
1124
0.70
0.77
0.04
0.70
0.761
0.500
0:387
1105T
0.767
85%
IL785
0170
1
0200
0201
0000000000
0,00
0000
0570
0.335
0.730
11957
80%
0745
0245
0.76
0760
0615
0655
064
0.00
0:00
0012
0610
0626
0573
0/66
002
0478
0601
0.722
0634
0959
30%
0.715
0.35
0220
0
0740
0.00
0.00
0.75
0.71
47190000
0.112
0266
0.55
0.700
0.000
0.901
0620
067
0753
0.9
0250
0:30
0.90
0,20
0.76
as1
0.64
0985
94%
0.54
0.699
1
0200
0656
031
0.59
6364
0:347
0,345
0.735
0
0315
134
0.42
0533
0150
0:43
0.71
0,35
0052
06625
0.000
0.555
0440
0670
0.00
-
0.36
0.663
0,00
0.00
0.58
0
0.64
1 0.500
00000
0629
0000000
8
01306
0
000440000
0
100%
99%
|
0.500
01584
0.500
0.500
0.5%
0
0.500
0558
0551
0.000
0.57
0.30
0.000
0691
0000000000
0225
0.000
0
0725
0.760
0.30
0.765
733000000
0.500
0
0.500
8817
00000000000
0.500
0.727
1.5000000
0.000
8:67
115000220
0.500
0.50
0.52
0.500
0.500
0.50000
0.500
0.30
0.500
0.30
3.3000000
0.30
0.000
0600
0
0.500
0
0.7
0,500
0.00
0.900
1:95
083
1911
0000000000000
0.53
0
42 42

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Model performance gap A of RD experiment under accuracy on MiniBooNN dataset.
Degice
LGB
XGB
CATS
PFN
DAN
MLP
N006
R35
SW]
CAP
NET
606
INT
DCN
FTT
-80
5.T
5NN
TTF
TABBR
NCA
1Mhh
TULM
UNI
AVE
2%
-0.035
-1.00
-0.00
-4.01
-9:01
0.000
4.303
-0032
0.000
-0191
-0.001
-11022
0.000
-4.001
-9.044
-4081
-2001
-2,001
-0393
40.15
-0.01
0,000
-4001
-0000
45
-0171
401a
81066
-8:04
-1801
-0006
-3035
-6001
-0197
-0:91
-1185
-1001
-1.071
-8:05
-6268
-2.09
-081
4.057
-165
0107
-0008
-0:11
6%
-0039
0.025
40066
4.011
-8593
-.003
40216
-0036
4001
-0.03
4.015
4.000
0.015
4.015
-9515
4007
.0615
4.014
00007
0.100
-0.00
0,00
-0310
40035
105
0.117
4.128
4085
0.016
40002
0.023
-001
4,013
.0010
40000
40011
0..18
0.017
0663
0005
-006
4,017
.0.07
.0.83
4.057
2,00
0006
4:63
129
4013
411886
4185
41007
41054
01033
40000
-2016
-0144
-01773
-01779
41071
41577
-1026
405TT
-31231
40345
-015
41.91
4108
4008
165
-0884
0.008
-0043
00053
00050
-6656
0.001
0.033
00099
4,023
.081
-0023
.0029
.0.03
.0.18
400.9
.0033
4,021
.0.15
4.30
.0051
-0017
.0031
4605
4.133
4.171
41186
-0.14
40.08
-8649
46066
10010
40131
-0154
-0185
41182
40100
41184
-8630
-0037
-2023
.0382
41545
-0817
-4516
-0085
183
-12
4130
-8.61
-0:66
4.0%
0.019
2016
6,946
.0059
4.09
4000
4.500
-0:49
.2611
4:015
0.00
118
-6866
309
-119
4,190
40.04
-1265
4.250
-0581
-0/24
4.296
-0025
4,09
-0071
40085
4.00
-0.79
4.889
-0571
4026
.0047
4,099
0.197
0,50
4.000
-0219
4009
4005
4/01
4116
4.001
349
4.160
0214
40071
0.17%
0,995
-0095
0.11
0044
4006
00101
4,50
.0087
.0000
.009
4,095
2.16
4,59
0.151
20000
-044
4.06
4024
41.15
4008
41111
40106
4.301
-0020
4/M1
-01M0
40155
4007
-107
-031T
-3idf
-0183
41:90
-0815
-0013
4.100
4.175
400M
4..19
40066
-B055
4189
40011
4J111
-0155
40199
41883
40007
40056
-2013
4JHI
-0211
41851
4112
-0014
-0089
305
0.335
0.27
0.169
0.000
06556
4.078
0011
4,099
0.06
.000
-6655
0049
.003
4,044
0.200
0.374
0.16
0,000
.003
4:01
325
4211
41118
41286
16233
-6:13
41133
4.170
20081
4.11
-111
41121
41110
-1119
46127
-8263
4024
-2144
4-125
-211
4.39
412
2080
46811
-.14
0.30
6
4.99
4,00
0/07
0:45
0000
4/0
013
4123
-117
4017
2001
4.05
0395
0,07
.0111
4137
-244
WR
40%
1356
155
00.56
00000
4.355
-001
-0.008
-0105
.0081
-3011
4:081
-0187
40.54
-015
-0051
-0058
-111
40%
40.300
4.365
4.126
0.129
4.105
-0.36
00124
0.159
-050
4081
-0.13
-0.14
4.143
.0.44
4..40
.0692
40895
-5665
4,890
-0,45
-0.25
-088
.0051
40155
175
-2277
4319
-1124
-8.22
-0.05
-B00
-0000
4.191
-0012
-0151
-6.90
-0.15
-0.79
-008T
-0011
-009A
-0MM
-0194
41.05
-08M
-006
-1131
46%
0.905
1.295
4.126
4..7%
0.044
00.46
0003
4.225
0080
4,090
0.198
4.183
4.190
-0.00
4.188
06555
4006
.0050
4.882
.0.33
0.45
.0213
2,15
.000
-.163
-1199
-1287
46264
-0014
4/111
-0104
-118
-8089
-6000
40244
-2090
-22N
4..05
41210
-01M
46269
4:41
48%
0.354
0.553
4113
-336
0.17
-.143
4,77
0.100
20044
4124
0.10
4.155
416
0.16
4.168
-.166
4009
-1177
4.1%
-0314
0.44
.1133
0,07
.054
4.69
41175
-221
0.41
4.411
00330
0.153
01156
4051
4313
5074
4.104
-0185
0.173
0.109
00100
.0066
4.117
00455
-030
000
0057
54%
4.399
41,599
40.1
41.99
41.34
-0.32
4000
4.392
-5612
4.102
-0.14
40.14
4.13
41159
4.157
-0.30
-8121
4.116
.0.65
4.463
41187
-0881
40555
41135
-3359
0.355
-0354
-1.7%
-0315
-0744
0.190
-0116
4.11
-034
-0.55
-0.55
-0.76
-004
4,35
-0a0
-000
-001
0014
58%
4.382
4.555
4.1%
-0240
-0145
-0.30
4112
4.200
-001
4.123
4.153
4,100
4.165
-..65
4.168
-0.55
-0149
4.144
-0,00
4,335
4.22
4016
MTS
-1 199
-6281
9-21
-111
4.100
4 1919
-1 119
-0 1.99
-211
-1.91
-11K
-0191
4:141
4.00
41322
2004
-0011
-.21
626
-0428
-119
6.32
.1166
4.44
4
0230
0.155
4164
A115
4:137
0.42
0.477
0313
0,01
0.359
0,307
0.3
0.318
0666
4.230
-1226
41345
41.35
41.10
-1394
4.11
-3151
4.195
-0:54
41.50
41.94
41338
-0.13
40.19
-0350
41.59
41200
10061
-0255
689
0.449
0.338
0.254
00307
00319
00330
0.166
0,303
00140
4.199
0.253
0.335
0
0.254
0.267
0.335
0.138
-0175
4.190
.0,65
-0.54
.034
0.00
00055
4,345
-0435
-3424
-1251
-1.00
41215
-8225
4.124
4.330
-0150
4.190
-0.41
42283
41226
-1225
4..73
-0.95
-0397
-2114
4.193
-2419
41453
-6090
-0235
TNS
- 39
6 1.50
0.
-021
-2205
4.5%
0.259
4,24
018
4.11
-
0215
-255
-224
.0271
4.315
-337
0.805
2003
0.49
14%
44489
-.299
-0251
-8216
4.158
4.314
-0234
4:240
4.355
4.44
41255
4.257
-8265
-6191
-0.21
4:235
0,39
0.464
41.43
40053
4.255
-0-11
4.711
-000
4.871
0.400
40316
.0.55
-0.34
00373
4.342
00229
4.294
00357
4.300
0.390
.0413
03511
0.353
-0233
4,207
-0445
4.314
MFR
-0.41
-3413
-1.315
-1.14
-1.35
-0.33
4.101
4.355
-2191
4.173
-0.11
4.314
41287
41.95
.1.39
-1.35
40313
-3.15
40.13
-0%6
41451
-1361
-0176
.0591
.0.30
628
0.400
0.460
-..4g
.0.39
.0.29
.0.39
0.345
0415
-0314
4.244
0.390
4,90
4.34
0.500
0.40
.0315
.0214
.3214
4.285
03365
4.46
.0.31
0.0
.00%
4,315
MRR
-0:90
-3.00
41325
-8.366
-4.19
-13322
4227
4413
-2250
4.21
-03773
4.39
41345
41354
-1.06
-129
-0194
-2254
4:21
-0.8
41.90
-6033
-0113
504
0,000
0,40
0.2%%
0242
0,35
0.9%
0.84
033
4,300
0-10
4,40
0-00
0.40
-213
-220
.331
434
033
0.m
A361
-0,00
-3440
40390
-1.94
41240
-2213
4:11
-224
4234
4,31
4,00
4216
4644
-6249
-221
-2201
4:21
41457
40210
4.391
0.00
4.00
1.00
-
0.13
.0291
0.00
4.81
4335
-330
4.71
0.94
4031
4.35
4.477
4,396
-0099
4.311
99 -9:90
4.400
-1:51
0.490
4. 1116
0.19
-375
4.M1
4.91
41331
433
0:44
-3356
4.310
-000
-0415
-1151
.0.99
0.378
WA
0.490
4.440
-0448
0045
4.31
-0455
4.34
0455
-3366
4,31
4449
4,44
4.42
4443
0.440
-444
4237
.3313
4,30
-9450
4.445
40453
4,00
.06%
40315
MFL
-0:90
-0:60
41.53
-4455
-1.5%
-840
4:33
4.12
-140
4.81
4.89
41555
40400
4.:00
-8386
4.36
-0.50
4.77
-0.99
41:71
41151
-11001
-6.55
1006
0.449
0.459
.0.70
0.455
0.40
-449
4.355
0.44
0440
4,425
0.424
4.49
0.400
0.400
0.440
00400
4442
-036
4
0.45
0,45
0.44
0,054
.0077
-0.405
(i) Model performance gap A of RD experiment under ROC-AUC on MiniBooNE dataset.
Degice
LGB
XGB
CATS
PFN
DAN
MLP
N0D6
RES
SW]
CAP
NET
606
INT
DCN
FTT
CRO
5AT
5NN
1TF
TABR
NCA
1MA
TLLMM.
UN1
AVE
2%
-0/02
-0.02
-00002
-0.00
0.300
0.50
0.001
-0:02
0.00
0.003
-0/022
-0.001
-0.001
0,500
0,500
-0:58
0.000
-3001
-4001
0.003
4.001
.0.029
0.000
0.000
-0083
6%
-0001
-0001
0.004
0.006
.0007
-0:00
-0.05
4.300
-0008
-6007
-0005
-6.00
.0.05
.0.12
0,000
-0.17
-0/03
-0013
-50100
.0.03
40.45
.0.57
4000
.0005
-001
5%
-0139
-2100
-108
-1.00
-0:01
-2002e
.0/355
-01010
-2101
-1002
415065
-1:00
-0005
-31556
-5/935
-0101
40085
-000
-0091
105
-0007
00007
-0007
.001
00000
.0G12
.0002
0.003
0.000
4,008
.0.0
.0013
.0.00
.0.15
..011
.0632
40066
-0616
4,012
-002
4.064
.0.77
0,003
.0000
4614
125
-080
411455
-8086
-BOW
46029
20015
-01223
-0109
40151
-8082
-8621
-4008
-26135
-0H12
41.08
-0814
-0007
-0031
145
0011
4011
0.000
0005
-0018
-8516
0.01
-004
4013
0001
0,07
0001
-006
0.012
-6624
-0/10
-2015
4,013
.0m
0.154
.006
0,07
0007
-009
0109
4013
-8023
-0015
40025
-0010
5005
-0124
4009
-8504
4.918
-8535
-0021
-0011
4,09
4.185
0,01
-0009
-023
4018
4005
4003
4.77
-00
-007
-000
-004
.0075
.00
304
40100
4004
5011
.0.81
-0060
40055
-0043
4,09
4047
41167
-000
4009
4000
.0044
-8091
.0im5
40097
4/11
-0015
40151
-0084
.uom
.0550
-Sid11
-0015
41.99
-0077
-001
-0004
349
.081
4002
4014
-05%
-9265
4005
0.044
5415
4059
.094
.0044
0.044
.026
-0055
.0103
4,08
.084
..170
.0147
-003
-0015
4069
204
1011
0012
0000
-0885
-099
-8541
0245
0012
4,25
0002
0.945
0882
0055
-.04
-959
.0044
.0042
4,03
00005
4.153
00B15
0007
4039
181
4009
1629
16894
4009
2001
4011
-0011
40000
-8991
-001
-2010
-0154
4.153
.1341
-6814
-001
30%
0027
4884
2013
2006
4026
4,00
4.05
0.05
4002
4020
-0033
4001
0000
0.178
0009
0099
41145
0267
1926
6636
5051
4011
600
4957
400
0007
0.00
.0645
0650
-566
4,035
.5083
.11
-007
.015
4651
4012
1829
10544
18285
4011
5016
4044
-0144
40155
4002
40667
41945
40540
-5813
40857
41115
41150
-0843
40612
4055
-001
0001
4055
1009
-020
-BCAA
6004
6051
-00M
4009
4000
-057
-0073
-007h
-0075
41180
-1.WW
-005
-6636
40%
0002
00125
4085
-0555
-006
-9587
0081
4002
0031
4051
40073
4002
..677
0.00
-6082
40066
-0055
4,055
.0084
4.19L
-0,15
.0155
-0877
4005
-004
-0046
4002
1629
1921
-8095
41m
4004
-0014
4/176
-004
41M1
400
-1000
4184
40011
-2099
4JH4
41199
-009
-6015
46%
4.00
.059
0,00
4004
2007
4052
0,09
0,000
4.075
0.072
400
80044
4043
.069
4,054
-0114
4.16
.0227
0,075
-007
4071
0005
0.119
.0.06
4007
0.10
-6550
4.101
.009
4.105
4.100
.0.16
0.000
.0.41
0003
-1137
4.122
4.555
4054
.0.00
40071
.002
A111
-8166
40184
-1112
-0136
41124
4.154
-5158
4.11
-0131
4.185
41135
41159
41125
-1.75
40099
-5150
45137
-0154
41112
41215
-0844
.6515
4012
0055
4057
-0688
.0055
6055
4.101
-0199
4,155
.001
4001
4009
.856
.0.99
-5101
4.007
-010
.1344
.00
4006
2112
4114
4000
-0125
4015
-8:05
4,101
40294
4,093
0087
0.09
4.105
4..07
4.000
-0125
40200
-0120
4,100
-0184
40222
4.19
-6066
4001
-100
-0716
4177
0.345
-1100
4187
-191
4,50
4123
-8766
-017
-3191
4177
0.70
4116
-000
.n01
4110
400
4131
-6114
4139
4.114
.0000
4.115
2,073
4.108
4.124
4.116
4.106
-8154
4162
4.121
0.17
4.10
..34
0,0%
.0033
4.11
-3106
-0:35
4.346
4.104
-0.16
40.15
41151
41.08
-0.99
4..76
-5123
45.10
-015
41316
4133
Ju830
-0.30
0084
4115
-0.3T
0000
0.163
4.162
.0134
4.164
0.145
0.154
4.159
00.55
0114
.0091
4,110
.0.93
4.278
0.965
.0652
4.134
-013
41116
41185
11.10%
4.123
4.206
4.153
-0152
40.95
41175
4..9
4.194
-1329
40305
-0.29
4.161
-0119
41289
-6815
-0139
68%
0009
4
4111
0.197
00179
4
4,300
-110
4.53
4.170
4.193
115
0204
4.13
0.30
-1166
4.62
0.187
0.25
0.85
-686
A159
10%
4123
41120
-1124
16240
-6:90
41144
4.184
-0119
4.169
-0187
4.19
4122
41150
4L164
-1.19
-1133
.8181
4.135
0.15
4.237
41300
4004
-0041
4113
4115
-117
4.75
4:11
4173
4151
4175
4
.119
4.13
0.30
0004
14%
10111
-8214
-0190
.0125
-1261
-.147
-0.59
.025
4330
-081
-6266
TW4
-1116
4111
-1151
4116
.0395
.0391
43545
4.304
-5311
4.311
-3.55
45
-1.00
.0.79
.0.54
-5.301
4:36
-051
40.50
-009
.0514
40315
188
-0.34
4,209
..18
0.39
.0254
4.188
4.315
-216
4.23
-215
4.21
0.36
4175
4.36
.0.07
4.156
-5247
4.22
-0247
4.29
4034
40079
.0620
4226
MFR
-0125
40.23
41
18.95
-0154
4113
4.111
4.14
4.19
4.181
4317
41168
41.00
-0.15
-0131
4.17
-0215
41202
41375
-601K
-0171
858
02214
4218
0023
0274
4,190
.0200
4.1
4342
4,30
0.299
4.59
0255
-330
0.237
0315
A191
-119
4
.020
0337
.003
4210
49195
41194
41212
-1222
4129%
-1263
422
4.183
4:21
-0290
4129
-1200
41211
-129
4449
-2200
4:20
4220
41251
41499
-6015
4.344
4.37
41
194
.031
8836
4352
4367
4313
4,31
0.36
4.30
0
1365
0331
.0345
4315
-331
4.31
0301
4.05
..00
.0311
4,30
4,30
.050
4.551
.0351
.0350
8:66
4.15
-19
4314
11195
.515
-119
4158
.656
4:11
928
-0233
4.222
4.306
01200
.8444
4115
0,31
-233
4.36
4310
4
4.594
4129
4.155
011
-52%
4.2
4440
4.31
41477
40634
4:35
-0:11
41
4355
-031
4114
0,00
4306
4
4.31
4131
4333
A311
-2181
4000
-927
4211
-0016
-.711
0039
4354
4.303
0333
0221
0250
4230
8:11
0378
4,24
0.404
4.577
4 0.44
0335
4,27
8459
-330
4,30
0,30
4,39
4.46
2,000
4315
99%
0.432
0,43
4.3
0.446
-8345
4122
4353
4340
4:126
4,30
4,43
434
0139
4136
0350
4318
-411
4:39
0.71
4.30
.0433
0.0%
4039
4316
100%
40,87
41487
4.442
44486
44400
4,429
46492
4,47
4,771
4.851
41:81
41477
41478
-441
-9490
4:05
41391
41405
4008
46000
43

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 18. Model performance on Eyemovements dataset. (a) Model performance of SC experiment under accuracy on Fyemovements dataset.
Coluna
PCC
LGB
XGB
CNTB
PFN
DAN
MLP
N006
RE5
SW]
CXP
NET
GOS
INT
DCN
FIT
GRD
5.T
5NN
TIP
TABR
NCA
LMA
TL.M
AVE
0.594
0.684
0-405
0.471
11553
0.417
0.414
0:555
0.510
0.561
0.913
11195
0:50
-0:001
0.501
0-005
01531
0.442
0.466
0.543
0.59%
0.56%
0.306
01596
0.884
0.925
-0:011
0366
0.515
04055
0.525
0552
0.440
0.465
0.555
0.505
0.545
0.500
0.572
0.305
0.585
0.921
0.379
6.700
-0.005
0.514
0.591
0.512
0.405
0.525
0412
0.500
0.437
0.443
0.553
0.507
0.542
0.590
0.358
0.573
0.507
0.395
0.58
0.900
0918
6.006
0.596
0,592
0.881
0.536
0.404
0.524
6411
0.521
0.442
0.446
0.552
0.502
0.541
0.588
0.359
0.571
0.590
0.396
0.596
0.855
0.290
0,545
5
00015
0.914
0.588
0.528
0.406
0.523
0412
0.520
0.436
0.443
0.551
0,502
0.545
0.507
0.355
0.566
0.505
0.395
0,500
0.89
0.250
0,542
6
6017
0.503
0,512
0.405
0,334
0413
0,59
0.49
0.445
0.555
0510
0.599
0497
0.356
0.570
0.506
0.399
0,59
0.695
0300
0,503
3
0017
0.544
0.515
0,83
B517
0437
0.500
0.356
0.972
0.00
0,525
6017
0.586
0.406
6414
0516
0.415
0.499
0.545
0,502
0,509
0449
0.354
0.572
0.505
0.992
0.582
0.800
0330
0318
0,541
9
-0020
0.538
0671
0,521
0.403
0,519
0410
0.542
04455
0441
0.541
0498
0.527
0416
0.357
0.547
0.504
0.395
0563
0.52
0,515
10
6.211
0.591
0.558
0,534
0.407
0,500
0413
0.557
0,410
0458
0.553
0.504
0,543
0.504
0.354
0.572
0.50
0.39
0.00
0000 0.00
0,50
6.94
0.544
0.555
05574
0,512
0.405
0.26
0411
0.554
0.57
0.417
0.533
0.00
0.507
0.501
0,353
0.572
0.505
0.39T
0,587
0000
0100
0,535
12 6082
14 0.811
0055
-0063
0.54
0.55
0.580
0.54
0.573
0.590
0.566
0.518
0.510
0.519
0.525
0,40
0.00
0.455
0.406
0515
0534
0525
0.59
6412
0,122
0413
0413
0.50
0.554
0.554
0.545
0,507
05533
0.435
0.436
0,437
0.438
0.433
0.441
0.544
0.53
0352
0.0
0.56
0500
0.543
0.512
0.51
0.516
0.510
0.566
0,45
0.50
0550
0.50
0.570
0.513
0.550
0.556
11:11
11:00
0.508
01502-000
11:11000
0.399
0.355
0.3%
0.59
000
0300
0000
0000
0.66
0000
0320
0655
000
8319
0.53
0.535
0.54
0.544
16 0.066
0.51
0.568
1660
0,534
0.404
052
6416
0.554
0.51
0.437
0,55
0,544
0.516
0355
0.357
12:0000
0,57
0.001
0325
0,544
17
0.877
0.584
0.555
0.55
0.40
0534
0.554
0.414
0.518
0.56
0.570
11501-0000
0.56
0,587
0.540
0.088
0.574
0.404
0.19
0513
0,400
0.33
0.500
0.512
0.4%
0.552
0,51
0.56
0.85
1910-000
0.535
19
0.092
0.567
0.565
0.578
0.00
0534
0,415
0.548
01555
0.417
0.553
0550
0.507
0,301
0.588
11:05
11570
0.859
0.76
0,500
20
0.805
0.57
0.16
0.509
0400
0.99
0,400
0.178
0101
0.417
0.557
0.566
0,005
0.514
0.605
0.300
0,500
11:99
0.590
11550
0.551
0.313
0.66
0.500
21
0.133
0.574
0.114
0.5%
0400
0.311
0,412
0.111
01514
0.416
0.465
0.54
0468
0.555
0,500
0.59
0.565
0-901
0.905
01570
0,880
0.377
0.055
0.76
0.59
22
0.125
0.500
0.117
0.510
1404
0.536
0411
0.154
0151
0.11
0.55
0.550
0.59
0.5
0,156
0.554
0.556
0:511
0.507
01587
0.902
0.391
0.052
0.709
0.514
0.155
0.587
0.559
0677
0.527
0.40
0518
0411
0.547
0158
0.444
0.4%
0.56
0.50
0.525
0.5%
0.34
0.567
11504
0.400
0.53
0.856
0.250
0.920
0.730
0.54
24
0.168
0.551
0.510
0.512
1:001
0.533
0.410
0.578
0150
0.433
0.557
0.526
0,495
0.525
0.695
0,361
0.540
1.686
0.355
01552
0629
0.26
0.921
0.715
0.517
25
0.213
0.51
0.118
0.774
0.578
0400
0.514
0,414
0.544
0552
0.40
0.655
0,500
0.50
0.533
0.50
0,355
0.559
1150
0.933
0.899
0.275
0.920
0.71
0,500
26
0.215
0.535
0.115
0M4
0.964
0400
0.533
0,411
0.545
0.476
0.434
0.53
0.98
0.524
0.444
0.50
0.511
0.995
0.912
01555
0.854
0.761
0.994
0.991
0.533
27
0.718
0.500
0.555
0.00
0.325
04406
0.511
0.450
0.545
051
0.427
0.557
0,51
0449
0.53
0.497
0.589
0.565
0.99
0.944
0.581
0.886
0.257
0.914
0.706
0.533
(b) Model performance of SC experiment under ROC-AUC on Eyemovements dataset.
Oukern
PCC
LGBB
XGB
CATB
PPN
DAN
MLP
NO0E
0.5
SW]
CAP
NET
GOS
INT
DCN
FIT
(RR
5XT
SNNN
TTF
TABS
NCA.
LMA
T1..MM
UNI
AVE
0.885
0.718
0661
0.231
11711
0.699
0.888
0.349
0.31
0.69
0.510
0.713
0.571
11780
0.919
-0.001
0.781
0.717
0.71
0.31
0.245
0.300
0.73
0.69
0.511
0.751
0.303
0.514
01782
0.976
6.810
0.723
2-
-0.001
0.783
0.718
0682
0.221
0.71
0.31
0.89
0.249
0/302
0.731
0.69
0.752
0.703
0.512
00782
0.970
0.500
0.60
0.721
3
-0.05
0.784
0.556
0.218
0631
0.223
0641
0.21
0.666
0.58
0.749
0.751
0.99
0.508
0.772
0.703
0.531
0.979
0.500
0318
0.722
4
0.066
0.731
6750
035
0.71
0723
0642
0771
0630
0.60
0.47
0.70
0.505
0.70
0.572
0.7%%
0,517
0.233
5
-0015
0.78
0.89
0.712
0641
0.46
0.11
0.572
0.97%
0.21
6
0857
0.717
0651
0.22
0750
0,12
0.53
0.751
0.500
0.15
7
6017
0.775
0.221
0.200
0.699
0.245
0.751
0.751
0.515
0.882
0.500
0.000
8
6017
0.773
0558
0.733
0682
0.221
0.720
0.51
0.66
0.247
0.729
0.700
0.509
0.750
0.703
0.571
0.780
0.979
0.500
0.52
9
-0020
0.757
0628
0211
0.12
0.611
0.66
0.735
0,992
0.717
0,692
0.505
0.751
0.573
D.743
0.732
0.506
0.50
10
60211
0.792
0.778
1550
0.735
0.35
0.21
0642
0.720
0.611
0.657
0.749
0.732
0.699
0.507
0.772
0.572
0.792
0.976
0.506
0.244
11 6.94
0.775
0.255
0.631
0.22
0638
0.25
0.748
0.730
0.699
0,513
0.712
0.575
0.579
0.507
0955
0.719
12 0,82
0.783
6.79
0682
0.231
0641
0.699
0.69
0.245
0.735
0.507
0.910
0.50
0000
6,055
6.770
0.21
0.221
0.200
0.629
0.53
0.249
0.708
0.735
0.701
051
0.752
0.513
0.792
0.970
0.504
0.504
0.223
14 6,081
15
0.083
0.772
0.78
0.770
0.760
0.230
0.215
0634
0.07
0.223
0643
0.115
0.629
0.630
0.54
0.40
0.745
0.741
0.702
0.731
0.725
0.695
0.700
0.50
051
0.763
0.767
0.98
0.704
0.565
0.568
0.776
0.700
0.579
0.911
0.50
0.507
0.91
040
040
0.57
0.72
17 0,87
16 0,066
0.790
0.736
0.735
0.35
M627
0.222
0.221
0.699
0642
0.749
0.21
0.71
0629
0.33
0.39
0556
0.746
0.747
0.72
0.531
0.701
0.697
0511
0.731
0.759
0.76
11712
0L715
0.514
0.81
1.779
0550
0,50
0,491
0400
0410
0.522
18
0,88
0.773
0.768
1850
0.720
M62
0711
0634
0.735
0112
0631
0.734
0.09
0695
115
0.754
0.56
10667
0.55
0.315
0.76
0715
0.992
0.77
0.715
0.76
01000
0511
0/43
0.745
0771
0678
0.68
0.54
0.724
0.701
1555
0.711
11714
0.72
1077
0559
0.518
0955
077
019
20 0,108
0.76
0559
0.719
0621
0201
0/699
0.005
0.33
0.713
0.688
0.00
0.750
0.000
0.71
0.741
0.65
8075
0.94
22 0.125
21 0.112
0.77
0,773
0.76
1354
0.75
0.23
0654
0.33
1
0723
0644
0636
0.745
0.748
0.17
0722
0.24
0652
0.66
0.742
0.74
0.725
0.50
0.701
0.695
0.50
050
0.66
0.767
0.00
0.65
0511
(1500
03755
0534
0.576
0.11
0,500
0.19
0.15
23
0.155
0.78
1555
0636
0.745
0118
0427
0.44
0.08
0.22
0.59
0.66
1079
0507
0719
0.168
0.5
0.739
M63
0698
0612
0.731
0623
0.62
6711
0.06
0.14
0657
0512
0.76
166
0.566
1176
0.833
0.96
0,30
0.99
25
0.115
0.787
0.718
088
0.11
0621
0.14
0.69
0.740
0770
0A18
0.666
0.500
0.66
0.235
0.700
0.99
0.766
0.560
0178
0507
0.58
0.90
0311
0317
26
0.315
0.24
0.741
0834
0.865
0650
0.213
0.60
0.717
0000
0.20
0.50
0.211
046
0775
0.65
0.51
0.77
0161
0.461
0735
0.964
0.467
0.16
0.90)
0.3
27
0.218
0.77
0.714
0.36
0.710
0669
0211
0622
0,341
0713
0.699
0.636
0.240
0.68
0.2
0669
0.515
0.354
068
0.365
0777
0.976
0.488
0.922
0.80
0.12
(c) Model performance of MC-M experiment under accuracy on Eyemovements dataset.
Degres
LGB
XGB
CATB
PRN
DAN
MLP
N00DE
SW1
CAP
NET
0OS
INT
DCN
FIT
CRRO
5ATT
5NN
TTF
TABR
NCA
LMA
TLLM
UNI
AVE
0:904
0.580
0.684
0.555
0:405
0.537
0:411
0.555
0.533
0:477
0.464
0.555
0.541
0.902
0:161
0.572
0.597
0.566
0.900
0.277
0.11
01:5
0:541
1.%
0.569
0.583
0.020
0.525
0.415
0.517
0.409
0.45
0.512
0.47
0.477
1,539
0.493
0.532
0.457
0.357
0.565
0.490
0.314
01551
0.315
0.256
0916
0.723
0:533
7.41%
0.475
0:113
0.574
0.432
0.012
01502
0.400
0.56
0
0.422
0.155
0.51
0.476
0.09
0.474
0.358
0.14
0.474
0.390
0.545
0545
01255
0396
1.064
0:500
11.11
0.074
0:100
0.583
0.465
0.011
0.892
0.100
0.16
0.090
0.31
0.10
0.50
0.466
0.599
0.463
0.389
0.504
0,433
0.389
0.335
0.75
01365
1.07%
0:501
14.15
0446
0:467
0.56
0.415
0.98
0.450
0.400
0491
0.668
0.19
0.0N
0.88
0.551
0.348
0.663
0.356
0.576
0336
0.071
0:471
18.525
1.91
0.301
0.409
0470
0.681
0479
0.419
0.400
0.265
0.84
1.0%
0.466
0.406
0.88
0.307
0.400
0.54
0.411
0468
0.442
0425
0.400
0.55
0.77
0.444
0:345
0.429
0.385
0.479
0.500
0.265
0.467
25.07%
0.461
0.400
0.402
0.35
0.455
0,422
0.387
0.265
0.600
0.458
29.69%
0.355
0.456
0.500
0.364
0.501
0.424
0.396
0427
0.424
0.406
0.303
0.423
0.435
0415
0.399
0.44
0.466
0.399
0.392
0.372
0.255
0885
0.452
$1.19 13.399
0.794
0.57
0.356
0.500
0.476
0:401
0.456
0.670
0.005
0.161
0476
0.466
0.33
0.47
0.W1
0.394
0.324
0336
0814
1.666
13.04%
0-111
0.46
0.34
0.406
0.389
0.419
0.407
0.400
0416
0.370
0421
0.48
0.151
0417
0:390
0.90
0.975
0,525
0.368
0.844
1675
0412
1.20
0.408
0.511
0.500
0.806
0.391
0.18
0.405
0.358
0420
0.397
0.993
0.350
0.265
0813
0675
44 41,4%%
48. 4
0.57
1.20
0.373
0.400
0.594
0.344
0.353
0.502
0.500
0.366
0.387
0.399
0.395
0.401
0.375
0.372
0.396
0412
0416
0.397
0.453
0.351
0.340
0.400
0.366
0.393
0.350
0.361
0.360
0.265
1.265
1:265
0.824
0.81
0646
0.402
0414
5 51.8%
0.26
0.385
0.35
0.300
0400
0.997
0.3%
0.371
0.397
0.344
0.403
0.397
0335
0.811
0.382
0.390
0.355
0.355
0.265
0.813
0.402
55.5
0.24
03584
0.345
0.564
0.392
0.344
0.350
0.3%6
0.999
0.400
0.314
0.397
0.344
0.402
0.397
0.358
0.395
0.367
0.385
0.377
0.355
0.255
0
99 99.104
0.55
0.20
0.374
0.555
0.384
0.392
0415
0.999
0.39
0.3%
0.396
0.340
0.402
0.397
0.3%6
0.385
0.392
0.341
0.355
0.355
0.344
0.622
0.09
62. 62.00%
0.557
0.250
0.36
0.500
0.350
0.365
0.503
0.364
0.397
0.306
0.392
0.392
0410
0.305
0.401
0.396
0.395
0.362
0.360
0.300
0.364
0.115
0.509
0.395
0.39
0335
0.384
0.300
0.393
0.387
0.383
0.377
0.355
0.265
0881
0815
0614
0.402
0.402
0.362
0.365
0.562
0.354
0.505
0.401
0.363
0.396
0.395
0.357
0.382
0.393
0.401
0.395
0.326
0.33
0.383
0.342
0.365
0.265
0.677
0.3%6
74.0%%
0.264
0.360
0.375
0.364
0.33
0.355
0.399
0.375
0.355
0.3%
0.395
0,395
0.324
0.67
0.371
0.375
0.347
0.367
0.255
0611
0.3%6
0.261
0.326
0.554
0.503
0.391
0.384
0.385
0.575
0.370
0.359
0.397
0.39
0.355
0.352
0.73
0.3%
0.390
0.330
0,375
0.355
0826
0.56
0.300
41.40%
0.340
0.336
0.37%
0.395
0.397
0.380
0.300
0.565
0.351
0.355
0.395
0.308
0.395
0.344
0.33
0.60
0.385
0.342
0.324
0.35
0545
0.366
92.59%
65.19%
89.80 19.999
0.361
0.361
0.251
0.351
0.35
0.35
0.353
0.588
0.394
0.392
0.364
0.364
0.38
0.391
0.388
0.380
0.394
0.354
0.355
0.566
0.500
0.370
0.359
0.511
0.542
0.37
0.366
0.322
0.384
0.314
0.362
11395
0.394
0.364
0.311
0.300
0.321
0.3%
0.396
0.394
0.35
0.994
0.30
0.338
0.34
0335
0333
0.55
0.367
0.387
0.388
0.36
0.379
0.375
0
0315
0.334
0.33
0.3
0.36
0366
0326
0.27
0337
1156
0335
0.377
0.36
96.3%
0.314
0.2%
0.392
0.394
0.319
0.384
0.59
0.357
0.386
0.385
0.394
0.311
0.394
0.394
0.357
0.31
0.367
0.384
0.352
1006 1006000
0.200
0.306
0.300
0.393
0.394
0.379
0.394
0,348
0.349
0.394
0.314
0.394
0,348
0.394
0.394
0.349
0.32
0,394
0,37
0.35
0
0
0325
0.333
44

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(d) Model performance of MC-M experiment under ROC-AUC on Eyemovements dataset.
Degree
LGB
XGB
CATB
PPN
DVN
MtP
.00DE
KES
SW]
CAP
NET
605
INT
DCN
PTT
CRR
.AT
5NN
TTP
TABR
NCN
LMA
7LLM
UNI
AVE
0%
0.765
0.779
0.855
0.338
0.631
0.725
0.640
0.750
0.721
0.629
0.538
0.549
0.202
0.731
0.699
0.510
0.772
0..10
0.571
0.762
0.919
0.495
0.927
0.905
0.722
3.74
0.777
0.774
0.22
0.330
0.619
0.211
0.622
0.741
0.713
0.699
0.625
0.540
0.720
0.689
0.515
0.764
0.688
0.565
0.277
0.975
0.438
0931
0.909
0.712
7.41%
0.221
0.725
0.794
0.666
0.605
0,693
0.609
0.721
0.890
0.506
0.597
0.713
0.703
0.664
0.516
0.70
0.671
0.583
0.240
0.485
0.20
0.791
0.689
11.1%%
0.715
0.718
0.29
0.666
0.589
0.675
0.591
0.69
0.675
0.579
0.567
0,67
0.685
0.544
0.503
0,900
0.654
0.535
0.216
0.932
0.501
0.20
0.777
0.675
14.1%%
0.672
0.676
0.882
0.620
0.592
0.668
0.581
0662
0.657
0.572
0.558
0.64
0.565
0.44
0.507
0.64
0.634
0.531
0.671
0.755
0.501
0.923
0.780
0.648
18.52%
0.659
0,666
0.73
0.5%
0.51
0.565
0.632
0.570
0.541
0.65
0.54
0.26
0.506
0.636
0.613
0.525
0.647
0.743
0915
0.784
0.634
22.229
0.600
0.5M4
0.780
0.607
0.522
0.03
0.562
0654
0.623
0.568
0.590
0.590
0626
0.25
0.11
0.625
0.66
0.65
0.714
0920
0.784
0.6%
25,954
0.69
0,687
0.765
0.592
0.511
0.5%
0.555
0614
0.004
0.561
0.527
1000
0.555
0621
0616
0.507
0.45
0.591
0.113
0645
0.763
0.501
0917
0.741
0627
29459
06666
0/60
0.77
0.562
0.558
0.57%
0.563
0.54
0.585
0.550
0.516
01541
0.568
01013
0008
0.011
0.27
0.576
0.518
01551
0.5%
11:01
0361
U310
0.00
31.199
0629
0.71
0.568
0.51
0.576
0.545
0.94
0.583
0.544
0,517
0.50
0.557
(191
0005
0.494
0.65
0.574
0.519
0.585
0.56
0.501
0855
0758
0.596
0.544
0.755
0.550
0.545
0.561
0.526
0.53
0.572
0.546
0.508
0.500
0.548
0.81
0,588
0485
0.601
0.500
0.501
0.62
0.77
0.558
40.74%
0.609
0,615
0.746
0.562
0.535
0.358
0.519
0.71
0.564
0.542
0.40
0,509
0.539
0.75
0.78
0.4%
0.59
0.559
0515
0.550
0.522
0.500
0355
0.784
0.553
61.42%
0600
0.20
0.745
0.564
0.5%
0.550
0.514
0.54
0.563
0.53
0,494
0.572
0.500
0573
0.414
0.533
0.550
0.505
01.61
0.53
01:00
0364
0.71
0.533
0.565
0.591
0.733
0.534
0.511
0.545
0.500
0.51
0.511
0.536
0,093
01553
0.330
0561
0.51
0.4%
0.56
0.53
01:64
0,500
11:01
0856
0777
0.56
51.899 899
0.522
0.385
0.716
0.540
0,555
0.541
0.501
0.61
0.544
0.535
0.691
0.563
0.558
0560
0.50
0.494
0.68
0.55
0:577
0,517
01:01
0855
0775
0.568
99. 199
0.531
0.53
0.588
0.580
0.716
0.716
0.318
0.5%
0,595
0.525
0.546
0.545
0.500
0.500
0.561
0.5%
0.545
0.545
0.33
0.335
0.491
0.491
0.563
0.561
0.556
0.556
0.55
0.51
0.566
0.566
0.491
0.489
0.50
0.80
0.537
0.534
0.50
0.401
0:567
0.571
11511-0
1:567
0.517
0.516
0.501
0.501
0855
0856
0.24
0773
0.570
0.504
62.909
0.561
0.575
0.500
0.525
0.522
0.542
0.496
0.500
0.539
0.531
0.491
0.556
0.519
0.552
0.555
0.491
0.55
0.529
0.513
0.585
0.515
0.901
0884
0.89
0.551
66.77
0.561
0.571
0.55
0.525
0.534
0.555
0.494
0.63
0.515
0.529
0.094
01561
0.521
0550
0.55
0.491
0.544
0.535
01566
0,115
11:00
0356
0.744
0.56
70.17%
0.535
0.525
0.98
0.511
0.16
0.514
0.492
0353
0.527
0.511
0,091
0.512
0.506
0.59
0.59
0.88
0.56
0.525
0.507
0.330
0.512
01:01
0963
0813
0.550
74.0%%
0.53
0.576
0.98
0.519
0.905
0.537
0.495
0.22
0.501
0.53
0.091
0S5
0.505
0.54
0.57
0.687
0546
0.520
0.09
0.51
0,510
0.901
0.81
0537
0.366
71.78%
0.533
0.512
0.540
0.510
0.497
0.531
0.500
0.552
0.51
0.50
0.489
0.11
0.504
0.555
0.535
0.483
0.55
0.514
0.001
0.335
0.508
0.901
0884
0845
0.541
81.88
0.312
0.523
0.37
0.510
0.501
0.325
0.504
0.550
0.51
0.499
0.422
0.515
0.490
0.524
0.566
0.491
0.534
0.511
0.001
0.330
0,513
0.501
0884
0842
0,542
85.1P2
0.323
0.521
0.534
0.521
0.506
0.518
0.510
0.524
0.518
0.525
0.467
0.827
0.502
0.22
0.519
0.494
0.526
0.510
0,485
0.320
0.514
0.901
0852
0856
0,342
89.89%
0.518
0,518
0.534
0.325
0.505
0.520
0.515
0.519
0.517
0,502
0.463
0.523
0,512
0.520
0.520
0.49
0.527
0.509
0,490
0.321
0.516
0.901
0.54
0.49
0.541
92.59%
0.521
0.520
0,533
0.515
0.502
0,512
0.514
0.514
0,511
0.499
0.489
0.515
0,502
0.519
0,512
0.488
0,519
0,504
0.485
0.313
0,514
0.501
0.84
0.35
0.537
96. 1%
11502
0.500
0.508
0.506
0.488
0.506
0.50
0.500
0.995
0.505
0.491
0.992
0.496
0.50
0.901
0.496
0.13
0.500
0.490
0.902
0.507
0.901
0854
0878
0.310
100%
0.500
0.500
0.500
0.500
0.500
0.501
0.500
0.500
0.997
0.500
0.500
0.901
0.499
0.500
0.500
0.499
0.502
0.500
0.500
0.497
0.503
0.501
0852
0830
0.328
(e) Model performance of MC-L experiment under accuracy on Eyemovements dataset.
Degree
LGB
XGB
CATB
PPN
DAN
MLP
NO0E
RB5
SWI
CAP
NET
GOS
INT
DCN
PTT
GRR
5AT
SNN
TTP
TABIR
NCA
LMA
TLLM
UNI
AVE
0.684
0.355
0.405
0:411
0.585
0.522
0:437
0.464
0.555
0.506
0.541
0:902
0:161
0,572
0.597
0.991
0.556
0.900
0.277
0916
07:55
0:545
3.7%
0.903
0.588
0.680
0.331
0.405
0.536
0.414
0.557
0.521
0.442
0.466
0.565
0.540
0.503
0:3w3
0.566
0.510
0.546
0.894
0.250
0.23
0.15
0.544
7.41%
0.585
0.660
0.333
0.405
0.325
0.413
0.556
0.522
0.444
0.447
0.533
0.506
0.541
0.500
0.365
0.569
0.500
0.403
0.565
0.894
0.350
0.543
11.11%
0.992
0.587
0.657
0.531
0.404
0.525
0.414
0.557
0,515
0.443
0.445
0.500
0.565
0.54]
0.499
0.360
0.570
0.506
0.399
0.564
0.805
0.256
0.701
0.541
14.31%
0.593
0.588
0.654
0.552
0.404
0.524
0.14
0.549
0,515
0,45
0.447
0.545
0.539
0,500
0,30
0.569
0.507
0.399
0.592
0,890
0.292
0.20
0.541
18.52%
0.590
0.587
0.654
0.530
0.407
0.522
0.14
0.547
0.525
0.441
0.444
0.546
0.506
0.58
0,500
0.364
0.565
0.508
0,403
0.588
0.887
0.297
0317
0.51
0.541
22.229
0.588
0.574
0.654
0.529
0.405
0.520
0,13
0.54
0.515
0.442
0.439
01545
0.515
0.532
0.490
0,363
0.565
0.5%
0.414
0.557
0.045
0.348
0057
0.530
25.88%
0.569
0.563
0555
0.528
0.404
0.331
0.400
0.545
0,514
0.444
0.535
0.499
0.564
0.506
0.566
0855
0000
0.521
29.63%
0.575
0.564
0.53
0.521
0.520
0.46
0,543
0.46
0.439
0.316
0,68
0.33
0.488
0.361
0.565
0,516
0.395
0.615
0.315
0551
0.00
0,518
33.35%
0.522
0.553
0.551
0.513
0.402
0.508
0,15
0,519
0,505
0.433
0461
0513
0.991
0.361
0.562
0,497
0.395
0.525
0.447
0.255
0.253
0531
0.01
0.406
37.04%
0.551
0.544
0.513
0.405
0.509
0.16
0.509
0.46
0.43
0:503
0465
0515
0.655
0,367
0.531
0,500
0,400
0.330
0.442
00:00
0650
0.48%
40.74%
0.523
0.543
0.509
0.512
0.402
0.11
0.14
0.524
0.504
0.29
0.439
0512
0.511
0.300
0.535
0.500
0.401
0.525
0,442
0.258
0335
0.495
44.4%%
0.525
0.543
0.509
0.510
0,000
0.11
0.418
0.21
0.505
0.411
0.634
0:50
0.511
0.362
0.533
0.500
0.399
0.508
0,442
01351
0337
0654
0.497
0.590
0.508
0.511
0.003
0.500
0.415
0.18
0.504
0.411
0.635
0.504
0862
0.39
0.533
0,496
0.005
0.537
0,443
0.315
0856
0057
0.495
11.899
0.532
0.540
0.504
0.512
0.001
0.500
0.415
0.54
0.505
0.455
0.641
0353
1513
0865
0.661
0.535
0.501
0.390
0.534
0.630
0.746
0835
066
0.494
55.564
0.516
0.527
0.4%
0.513
0.403
0,415
0.515
0,500
0.49
0.438
0315
1400
1500
0,15
0.364
0.19
0,491
0.395
0.35
0.277
0535
0.59
0.400
99.20AL
0.515
0.538
0,003
0.502
0.295
0.688
0,416
0516
0.49
0.48
0.433
0331
1500
0.52
0.19
0,492
0.40]
0.318
0.451
0.256
0.400
61.50%
0.522
0.517
0.504
0.395
0.6%
0.416
0,500
0.441
0,434
0315
1500
0488
0.557
0518
0,490
0.399
0.310
0.271
11:71
0.39
0155
0.48
66.6%
0.501
0.530
0.27%
0.493
0.995
0.880
0.416
0.69
0.985
0.47
0.634
11888
0.600
0.487
0,095
0.343
0.694
0,436
0.260
11260
0851
0659
0.451
76.17%
0.467
0.501
0.67
0.497
0.96
0.881
0.414
04000
0.985
0.48
0.411
0.886
0461
0360
0.87
0.401
0.395
0.895
0,636
0.345
0855
0644
0.000
24.07%
0.414
0.461
0.47%
0.476
0.996
0.470
0.304
0471
0.425
0.444
0.441
0.45
0.45
0461
0464
6361
0.42
0.467
0.001
0.468
0.42
0.336
0886
0653
0.16
71.784
0.965
0.419
0:663
0.422
0.99
0.465
0:304
C471
0.465
0.235
0.441
0447
0.18
(46)
0462
6332
0.465
0.45%
0.403
0.465
0.42
0.364
0837
0.460
8L484
0.426
0.460
0.467
0.471
0.396
0.466
0.396
C465
0.467
0,422
0.42
0.66
0.440
046)
0461
0373
0.45
0.462
0.403
0.466
0.490
0.266
0836
0831
0.460
85.199
0.356
0.46
0.667
0.46
0.997
0.462
0.394
0465
0.40
0.421
0.433
01270
04411
0.66
0446
0.80
0.53
0,664
0.412
0,770
0.429
0.331
0835
U644
0.466
89.999
0.318
0.461
01458
0.40
0.985
0.414
0.394
0440
0.635
0.455
0.431
0457
0443
01445
0.46
0.801
0.43
0,435
0.415
0.417
0.410
0.285
0856
0613
0.455
91.595
0331
0.440
0.41
0.41
0.91
0.414
0.704
0.47
0.431
0.455
0.407
0436
0.4%)
0420
0412
0.17
0444
0.418
0.404
0.417
0.411
0.331
0837
0635
0.40
96. 1%.3%%
0.350
0.311
0445
0.422
0.561
0.405
0.304
0413
0.405
0.415
0.
0.19)
0416
0.361
0489
0.499
0.160
0.56
0.406
0.993
0.310
0,353
0.265
0836
0623
0.465
100%
0.250
0.306
0.360
0.383
0.591
0.379
0.394
0.348
0.349
0.394
0.591
0.594
0.348
0.394
0.394
0.349
0.352
0.394
0.379
0.335
0,33
0.265
0.825
0.511
0.333
(f)
Model performance of MC-L experiment under ROC-AUC on Eyemovements dataset.
Degree
LGB
XGB
CATB
PFN
DAN
MLP
NODDE
RES
SW1
CAP
NET
GOS
INT
DCN
FIT
GBO
5AT
5NN
TTF
TABR
NCA
LMA
TLLM
UN1
AVE
17%
0.265
0.579
0.858
0.538
0.631
0.235
0.640
0.350
0.721
0.629
0.538
0.349
0.731
0.609
0.510
0.772
0.790
0.762
0.929
0.405
0.927
0.806
0.332
3.7%
0.763
0.780
0.857
0.532
0.611
0.232
0.349
0.721
0.631
0.543
0.246
0.330
0.609
0.511
0.721
0.708
0.524
0.782
0.925
0.501
0.955
0.922
7.41%
0.753
0.779
0.644
0.557
0.62
0.222
0.640
0.349
0.721
0.632
0.544
0.246
0.330
0.699
0.515
0.721
0.700
0.575
0.781
0.975
0.514
0.930
0.954
0.322
11.1%%
0.753
0.779
0.542
0.357
0.622
0.222
0.641
0.749
0.221
0.632
0.543
0.347
0.30
0.699
0.513
0.71
0.700
0.575
0.751
0.97
0.511
0.500
0.57
0.022
14.1%
0764
0.40
0.7
0.34
0.23
0.62
0.366
0.72
0635
0.245
0.609
0.111
0.79
0.700
0.500
0.566
0911
03007
0.77
0.645
0.245
0.700
0.11
0.97
0.252
0.773
0.809
0.531
0.635
0.266
0.647
0.340
0.725
0.644
0.534
0.242
0724
0.609
0.514
0.67
0.701
0.251
0,50
0.345
0.855
0756
0.913
25.88%
0.774
0.759
0.350
0.217
0.652
0.715
0.655
0.599
0.341
0724
0.688
0.513
0.76
0.700
0.550
0.445
0.800
0756
0.506
29.63%
0.755
0.770
0.757
0.726
0.635
0.655
0.740
0.637
0.538
0.340
0.66
0724
0.688
0.515
0.764
0.700
0.574
0.77
0,500
0.325
0.50
0331
0.307
33.354
0.752
0.740
0.748
0.745
0.752
0.319
0.721
0.632
0.26
0.26
0.654
0.653
0.511
0.921
0.507
0.507
0.655
0.629
0.537
0.538
0.226
0.226
0464
0666
0510
0.099
0.69
0.600
0.513
0.511
0.440
0.763
0.6%
0.695
0.575
0.58
0.736
0.752
0.645
0.645
0.30
0318
0.866
1760
0.69
40.74
0.751
0.746
0.720
0.722
0.636
0.60
0.651
0.552
0.505
0.61
0.639
0.225
0.508
0668
0.113
0.400
0.955
0.589
0.355
0.635
0.500
0772
0657
48.15%
44.44%
0.750
0.746
0.745
0.719
0.199
0.720
0.520
0.645
0.63
0.26
0.264
0,652
0.652
0.555
0.525
0.503
0.505
0.630
0.631
0.543
0.500
0.725
0.725
0684
0506
0.508
0.91
0.600
0.505
0.505
0.33
0.73
0,690
0.694
0.599
0.900
0.647
0.645
0.336
0.546
0.866
0756
0762
0.67
0.588
55. 1
51.8%
0.339
0.746
0.702
0.114
0.330
0.64
0.649
0.60
0.204
0,55
03555
0.503
0.632
0.545
0.545
0.225
0.718
0661
450
0500
0238
030
0.50
0.335
0.27
0.600
0.694
0.38
0.59
0.255
0.333
0.649
0,600
0.300
01388
0.82
0.61
0.59
0.52
0.677
56
0.730
0.338
0.700
0.30
0.655
0.80
0.000
0.915
0.63
0.641
07117
0260
0,403
0.755
0,611
0.384
0.255
0318
0.866
07555
0.736
0.06
0.30
0162
0.69
09.9
051
0.695
0637
0.642
0.766
0260
050)
0687
0.085
0.74
0.600
0.20
0.60
11:05
0.85
07545
0681
0.728
0.66
066
0167
0657
0963
UM4
0.688
0710
0850
0661
0685
0.84
0.766
0.60
0.51
0356
0.69
01511
0965
07555
0677
0.50
0311
0A68
0.60
0666
0661
650
0.42
0.640
0668
0657
0.900
0.85
0601
0.544
0.36
0,690
0.545
0987
0966
067
74.0%
0.692
0.668
0.002
0.644
0.63
0,75
0.67
0684
0.675
0.66
0.642
0.657
0675
0670
0511
0.60
066
0.94
0.65
0.618
0.333
0351
0.500
0.64
0682
0.67
0.67
0.40
0675
0.65
0615
0.641
0.79
0661
066
0.00
0.73
0663
0.966
0.75
0.61
01524
0.53
0.74
0659
0.000
0.600
0672
0.600
0.67
0645
0672
0.65
0620
0.600
00072
0656
0668
066
0310
0663
0667
0.59
0668
0.14
11:02
0155
0775
065
0.60
0.066
067
0AM
0.65
0.67
0A4
0864
0.67
0621
0.66
1M6
0AN
0668
063
0354
0169
0666
0.00
0167
0613
01:91
0886
0772
000
89305
0666
0.611
0160
0A4)
0.66
062
0657
064
0644
0.995
0.654
0.6
BAM
064
0645
0.53
0664
0661
0.595
06600
0.61
0.5%
0153
0755
0641
91.595
0.445
0.641
0.66
0AM4
0.6)
0617
0,32
0.69
0667
0.64
0.63
061
0A51
0651
0619
0,53
0660
066
0.547
0.6
0614
0.579
0884
0183
96.33 5%.3%
0.566
0.566
0636
0.556
0.551
0.569
0.500
0.588
0.565
0.552
0.564
0.500
0.560
0.567
0.58
0477
0553
0.588
0.590
0.565
0.540
0.501
0,855
0878
0.5K2
100%
0.500
0.500
0.500
0.500
0.500
0.301
0.500
0.500
0.497
0.500
0.500
0.501
0.499
0.500
0.500
0.499
0,502
0,500
0,503
0.497
0.503
0.501
0,52
0830
0.328
45

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(g) Model performance of RD experiment under accuracy on Eyemovements dataset.
Degree
LGB
XGB
CATB
PPN
DWN
MLP
.00DE
KES
SW]
CAP
NET
GOS
INT
DCN
FTTT
(RRO
5AT
5NNN
TTP
TABR
NCN
LMA
TLLM
UNI
AVE
0%
0.594
0.599
0.684
0.553
0.405
0.537
0:411
0.555
0.522
0.437
0.464
0.555
0.506
0.541
0.502
0.361
0.572
0.597
0.397
0.586
0.900
0.677
0.915
0.715
0.500
0.554
0.580
0.66
0.527
0.404
0.530
0:411
0.548
0,510
0.466
0.436
0.546
0.556
0.502
0:357
0.561
0.856
0.659
0.70%
0.552
7.41%
0.550
0.576
0.56
0.525
0.404
0.523
0.415
0.546
0,510
0.46
0.434
0.544
0.535
0.502
0:354
0.556
0.505
0.570
0.500
0.65
0.806
0.704
0.545
11.11%
0.556
0.556
0.43
0.513
0.403
0.511
0.400
0.536
0.505
0.31
0.425
0.331
0.523
0.497
0.357
0.550
0,495
0.395
0.550
0.735
0.570
0.595
0.997
0.533
14.1%%
0.554
0.555
0.22
0.511
0.403
0.511
0.44
0.531
0.503
0.45
0.429
0.527
0.524
0.458
0.351
0.545
0.493
0.554
0.676
0.993
0.901
0.701
0.530
18.52%
0.542
0.527
0.00
0.510
0.403
0.510
0411
0.529
0,501
0.34
0.425
0.529
0516
0.4%8
0.55
0.53
0.494
0.395
0.550
0.63
0.529
0.682
0.523
22.22%
0.515
0.52
0.57
0.491
0.401
0.650
0.49
0.514
0.491
0.27
0.423
0.514
0.505
0.47
0.348
0.25
0.4%
0.315
0.535
0.544
0.53
0.55
0.510
25.959
0.515
0.513
0.493
0.401
0.659
0.40
0.51
0.493
0.477
0.425
0.56
0.474
0511
0.466
0.353
0531
0.484
0.344
0.5%
0.001
01.0%
0.06
0105
0.509
0.852
0.514
0.557
0.479
0.395
0.855
0.407
0.111
0.442
0.455
0.415
0111
0,73
01014
0,455
0.351
0521
0,402
0.390
0.515
0.58
01512
035
0106
0.503
33.35%
0.509
0.546
0.61
0.401
0.496
0411
0,512
0.487
0.31
0.422
0.507
0507
0.354
0.16
0.599
0.504
0.501
37,04%
0.46
0.455
0.539
0.46
0.401
0.455
0412
0,501
0.479
0.455
0.416
0.9%
0464
0401
0,04
0.353
0.504
0,422
0.486
0.463
0.65
0.63
0.67
40.74%
0.40
0.488
0.535
0.466
0.2%
0.471
0.007
0478
0.465
0412
0.405
0.775
0414
0,50
0.348
0.474
0.499
0.300
0.496
0,417
0.433
0.66
0.666
0.77
44.42%
0.48
0.47%
0.059
0.499
0.400
0.476
0.40
0.487
0.473
0.429
0.417
0.286
0451
0.552
0.494
0,46
0.344
0.412
0.481
0.435
0359
0663
0.477
199
0.466
0.480
0.503
0.402
0.216
0.479
0.407
0,466
0.676
0.419
0.414
0.56
0.884
0.385
0.094
0.400
0.315
0.473
0.662
01413
0833
0666
0.455
51.859 899
0.554
0.425
0.883
0.4%6
0.395
0.462
0.406
0466
0.965
0.414
0.005
0.570
(451
0.65
03345
0.470
0,45
0.344
0.475
0,639
0.47%
0876
U012
0.461
0.911
0.4%
0.454
0.441
0.966
0.460
0.400
0467
0.690
0,422
0.405
0.668
0415
0044
6318
0.668
0.450
0.991
0.457
0.427
0.988
0.50
0850
0645
0.452
96. 169
0.937
0.426
0.420
0.488
0.99
0.465
0.406
0468
0.962
0.47
0.11
0.534
0.40
0466
0446
0.14
0.681
0.45
0.594
0.435
0.694
0.455
0.50
0.49
0.49
62.964
0.397
0.462
0418
0.298
0.454
0.404
0454
0.443
0,12
0,402
0.544
0456
0.466
0.45
0,41
0,300
0.461
0.436
0.312
0833
UG34
0.477
6..7%
0:310
0.390
0.43
0.416
0.997
0.418
0.415
0,455
0.643
0.400
0.004
11449
0.46
0655
0.41
0.50
0.440
0,49
0.341
0.412
0.424
0.361
0835
U636
0.411
70.17%
0.312
0.412
0.433
0.43
0.397
0.435
0.394
0451
0.634
0.400
0.401
01432
0,418
0.56
0478
0351
0.42
0,42%
0.391
0.407
0.384
0.361
0673
0443
94.0%%
0.336
0.481
0.19
0.455
0.986
0.416
0.400
0454
0.438
0.466
0.401
0.400
0.40
0451
0439
6341
0.12
0.45
0.593
0.435
0.184
0.974
0830
0678
0.488
71.78%
0:310
0.411
(1401
0.42
0.93
0.425
0.704
0423
0.419
0.400
0.194
0.344
0.162
04256
0418
0316
0.401
0.431
0.193
0.36
0.966
0:947
0886
0610
0.42
81.88
0:233
0.376
0.386
0.395
0.361
0.414
0.397
0469
0.398
0.386
0.391
0411
0.350
0423
0412
0347
0.402
0.406
0.384
0.305
0,373
0.334
0826
0822
0.415
85.102
0.256
0.354
0.379
0.393
0.365
0.405
0.396
0.394
0.394
0.386
0.396
0.002
0.364
0412
0411
0336
0.386
0.302
0.387
0.330
0.361
0.341
0.26
0607
0.407
83.89%
0.274
0.411
0,382
0.406
0.365
0.468
0.397
0.393
0.392
0.386
0.398
0.407
0.355
0414
0.466
0.345
0,386
0,392
0.385
0.359
0.362
0.365
0.23
0.37
0.42
91.195
0.35
0:124
0.561
0.3M0
0.81
0.985
0.704
0.124
0.379
0.349
0.191
0.97
0.315
0400
0.99
0.140
0.66
0.396
0.384
0.317
0.341
0.30
0835
0610
0.796
96.12 1%
0.361
0.131
01331
0.392
0.98
0.911
0.395
0.168
0.535
0.364
0
0.196
1.994
0.135
0
0.566
0.146
0.965
0.377
0.581
0.312
0,340
0.37%
083
0601
0.794
100%
0.250
0.106
0.260
0.393
0.594
0.379
0.394
0.348
0.549
0.394
0.191
0.994
0.348
0.594
0.384
0.349
0.352
0.394
0.379
0.335
0,333
0.265
0.855
0.531
0.383
(h) Model performance of RD experiment under ROC-AUC on Byemovements
dateneetttsttttt.t.t.teeee dataset.
Degree
LGB
XGB
CNTB
PHN
DAN
MLP
MODE
RB5
SWI
CAP
NET
60S
INT
DCN
PTT
GRO
5AT
SNN
TTP
TABRR
NCA
LMA
TLLM
UNI
AVE
0.265
0.979
0.858
0.3%
0.631
0.640
0.350
0.721
0.639
0.684
0.349
0:751
0.609
0.310
0.772
0.700
0.262
6.929
0.295
0.977
0.908
0.334
0.259
0.576
0.647
0.335
0.629
0.216
0.6%
0.345
0.713
0.626
0.525
0.342
0.725
0.607
0.509
0.763
0.699
0.530
0.276
0.246
0.924
0.797
0.326
7.41%
0.257
0.573
0.640
0.331
0.62
0.216
0.641
0.342
0.712
0.629
0.525
0.341
0.68
0.688
0.506
0.769
0.700
0.571
0.259
0.925
0.265
0912
0.322
11.11%
0.760
0.757
0.830
0.318
0.622
0.255
0.628
0.728
0.705
0.655
0.514
0.227
0.114
0.691
0,510
0.749
0.688
0.565
0.759
0.672
0.676
0912
0.79
0,309
14.31%
0.759
0.755
0.11
0.316
0.627
0.205
0.636
0.726
0,705
0.62
0.519
0715
0.92
0.505
0.743
0.690
0.564
0.752
0.535
0.687
0.708
18.52%
0.745
0.740
0.797
0.712
0.624
0.706
0.634
0.726
0.704
0.69
0.519
0.726
0713
0.92
0.506
0.742
0.691
0,570
0,247
0,525
0.641
0.907
0.587
0,700
22.229
0.737
0.733
0.777
0.697
0.65
0.25
0.709
0.013
0.12
0.614
0.10
0676
0,700
0.85
0.504
0.729
0.600
0.563
0.736
0,753
0.63
0.50
0.691
25.88%
0.739
0.621
0,62
0.627
6.111
0.895
0.514
0.01
0.644
0.509
0.33
0.678
0.25
0.800
0.83
0.600
29.63%
0.724
0.730
0.760
0.685
0.621
0.61
0.63
6507
0.992
0.636
0.513
0.207
0.96
0.62
0.501
0.20
0.678
0.210
0,67
0.71
0.644
33.35%
0.713
0.752
0.655
0.624
0.81
0.06
0.990
0.688
0.514
0.004
0.007
0.82
0,510
0.16
0,617
0.569
0.713
0.737
0.625
0.866
0.744
0.654
37.00%
0.700
0.701
0.740
0.667
0.619
0.78
0.34
0.080
0.629
0.505
0,90
0.687
0.63
0.506
0.703
0.666
0.561
0.655
0.594
0.776
0.71
40.74%
0.707
0.35
0.74
0.000
0.65
0.604
0665
0.059
0.588
0.589
1.66
0639
0.64
0.51
0.505
0.664
0.47
0.550
0.698
0.069
0.5%
0.91
0.785
0.55
44.4%%
0.65
0,000
0.709
0.661
0611
066
0611
0676
0.69
0.599
0.604
0.075
0647
0.771
0.66
0.007
0.653
0.55
0.50
0.622
0.660
0.550
0385
07766
0.000
199
0.63
0.601
0.15
0.62
0.621
0.71
0622
0481
0.67
0.05
0.611
064
0677
0674
0.495
0.665
0466
0.564
0672
0.643
01563
0863
07565
0462
$1.899 1.844
0.46
0.647
0.02
0/00
066
0.01
0660
0.65
0.5%
0.584
0M1
0A5
0161
0.66
0.101
0.69
0.41
0.545
0671
0,677
0.575
0870
0774
0.41
55.566
0.667
0.666
0.683
0.655
0.002
0.647
0.606
0655
0.652
0.591
0.592
0.644
0663
0.54
0.500
0.662
0.41
0.555
0.655
0.605
0.352
0.85
0.50
0.42
99.204
0.441
0.686
0.62
0616
0.51
0613
0659
0.659
0.5%4
0.505
1.62
0.40
0161
0.62
0.504
0.677
0647
0.558
0.69
0.615
0.358
0.56
0.53
0.455
61.50%
0617
0.615
0.766
0.006
0.5%
0.226
0.591
0651
0.625
0.576
0,573
16300
0.08
0632
0636
0.007
0145
0618
0.534
0.62
0.615
01:37
0.84
0S5
0625
66.7%
0624
0.25
0647
0,66
0.541
0.26
0.395
0651
0.629
0.514
0.575
0611
0652
0635
0.105
0.61
0621
0.543
0611
0.042
01508
0984
0951
0623
10.17%
0612
0.500
0A25
0.58
0665
0.584
0418
0619
0.571
0.574
0616
0600
0671
0.00
0.61
0611
0.543
0.599
0.557
0.535
0835
0750
0A15
24.07%
0.44
0.688
0611
0A12
0.96
0.700
0.595
0.52
0.24
0.575
0.581
0622
062
0610
0.49
0.35
0618
0.544
0666
0.520
0.540
0856
0903
0631
71.784
0612
0.615
0610
0,600
0.53
0.884
0.571
0.595
0.995
0.356
0.561
0.864
0.553
0600
0629
0.501
0.694
0.501
0.5%4
0.818
0.544
0.535
0855
0819
0.60
0.500
0.590
0.555
0.570
0.560
0.575
0.550
0.79
0.575
0.334
0.574
0.564
0.51
0.586
0.407
0.500
0.517
0.527
0.366
0,550
01:30
0.5
1924
0.557
85.194
0.533
0.575
0.564
0.56
0.558
01562
0.351
0.5%
0.564
0.50
0.59
01555
0.555
0566
0.57
0.044
0.561
0.566
0.501
01565
0,545
0.516
0385
1824
0.57
85.80 899
0.586
0.595
0.85
0.553
0.51
0.567
0.563
0.55
0.565
0.536
0.511
0.544
0.15
0566
0.574
0.96
0.60
0.568
0.501
0.50
0,540
0.518
0853
07565
0.37
91.195
0.576
0.129
0.56
0.500
01540
0.515
0.529
0.912
0.59
0.526
0.527
0516
0.1%0
0.5
0.500
0.88
0536
0.3%
0.513
0.54
0.50
0.51
0883
0831
0.159
96. 1%.3%%
0.517
0.593
0.530
0.335
0.519
0.520
0.517
0.514
0.525
0.516
0.517
0555
0,512
0.51
0.566
0.504
0.35
0.520
0.513
0.53
0.518
0.512
0881
0819
0.547
100%
0.500
0.500
0.500
0.500
0.500
0.501
0.500
0.500
0.997
0.500
0.500
0.501
0.499
0.500
0.500
0.49
0.502
0.500
0.503
0.497
0.50
0.501
0.822
0830
0.528
(i) Model performance gap A of RD experiment under accuracy on Eyemovements dataset.
Degro
LGBB
NC18
CNEB
IMN
MLPP
NODE:
R3S
SW]
CAP
N1:1
1308
INT
DUUN
PT1
(380
5.A'
5NNN
1T1.
'AA8K
NCA
1.MA
11.1M
UNI
AVIS
3.7%
4.018
-0.013
-0.034
-1.10
-0.000
-0013
0.001
4.214
-0002
-0.00
.0.00
-0.17
-0:055
-0010
-./01
4.010
-0.015
-4.005
.0.53
-0005
4:044
.0.02
-3006
.0.10
-0003
741%
-.024
-0.02
-0.15
1.915
-0.00
-0:086
0.009
-.2.6
-2022
0.000
-0.02
-0.021
-0:306
-5011
-5001
-0.019
-0.023
-0.00
-0:501
-0027
-0.10
.0.09
-2024
-0016
-0036
11.119
-0004
-0000-10
-0070
-0003
4.015
-0031
-0012
-089
-0044
-0001
-0614
-0111
-00011
-008
-0.05
-6650
-0/45
.0183
-0.40
-2075
-6.08
-0/41
34.5%
0.069
0.000
0.006
0020
0.005
0.443
0/136
0,003
0.644
0.51
0.228
D031
0,009
0,024
0.06
0,009
0.006
0:055
-0.249
0.124
2005
0.50
0.054
1..32%
0.089
0.005
0.122
0.00
0021
0.000
0/447
0,007
0157
0047
00228
6,079
0,023
0.50
0.027
0.003
0:061
0.252
0.219
0033
0.566
23.225
4.134
-2.044
-0.13
4.005
4011
40055
4.004
4.035
-2050
4.003
4.048
40073
4.350
-0053
4,031
40000
-0.82
4..42
-0.005
-0/05
.0,351
-02200
-0023
-0651
-0/06
25.91%
-1134
-0128
-0
41075
-0000
4005
41005
4:098
-0856
-0102
41062
40071
40096
-0057
-0113
-01773
-0007
41.46
-0.00
-0/03
-0:311
-0334
-0015
-0042
-0/91
29.63%
0.172
0.128
0.195
0.101
0.015
0.500
0.005
00000
0.058
0.025
0.088
0,079
0:66
5059
01121
.0.89
0.005
0.125
.0.33
.0243
-0866
0641
0.10
33.33%
0.170
0.136
0.199
01096
0011
0029
0101
0:09
0.057
0.012
0.000
1006
0055
5053
61134
0.019
01256
0600
0.12
00378
-0255
0666
0.005
33.04%
4.264
-01199
40212
4.124
.0011
-0060
0.004
4:057
-2.03
4.031
4.065
40:00
40384
5055
40557
-0023
-0.19
4.570
.0600
.0.71
-3.43
-0.31
-5880
.0673
4.130
40:34%
4.275
40217
41
41100
40011
40106
40011
4,119
-0.10
40857
410001
41144
4112
-119
-0185
40007
-1.11
41.65
-1016
-0153
-0.44
-1%50
-0054
4014K
44.44%
4.244
40.55
41.19
-0012
40096
40003
4,132
-0004
4000
40000
40128
-5191
40061
-0.55
-1168
-0.95
-5.455
-5054
J1614
- 4k
45.15%
0.267
0.264
0018
0.125
0.088
0,440
0.000
0.19
0259
2101
6055
0162
0.135
0.506
00006
0.195
-0447
0.540
5093
0083
0.151
51.85%
4.35
4.274
41182
40015
-0.25
4.013
4.63
-3.00
40053
40.06
-0.54
4033
-0130
40079
40044
-017%
4.110
-0606
-0.89
4.512
-1.567
-0057
-0069
-0.76
55.500
4.11
40 12:315
41172
4063
41176
41102
4.100
-0121
40185
41185
0138
41345
-013
-0197
41105
-1187
41110
-4011
00210
-0536
-0811
-0133
59.00
40:11
41178
4009
4031
41005
-0.15
4107
41311
-014
-0173
41007
-1199
41.005
-6006
-6.63
-0517
-0.004
-3086
-0100
0434
0.326
0.325
0.14
0017
0.33
0.016
0.133
0.15
0.035
0.996
0.184
0158
6.111
0107
021
0.130
0017
0.214
0.15
0.451
3092
0.14
0,302
66.67%
4.476
4.317
-0.50 3,561
41219
4002
-0.49
4.014
4.11
-0151
4006
40000
-0:00
AL178
-0199
4.122
4,00
-0250
4.136
-0014
.0:95
4.313
.0.33
.5092
A.11
4211
70.37%
-6411
-.300
40367
4.189
.0051
.U.77
40040
4219
-0166
-0.00
4.00
.0322
40300
-0.95
.6147
4,025
-0.61
4.156
-0814
4.305
4.373
0.465
AU115
4:26
34.0%
-.451
-0.58
4L315
40022
4:16
-0161
40.00
40004
40308
4U195
-5140
-0175
-008
-1255
4L14%
-0016
4.719
4.99
.0,67
-0091
-6323
4:219
77.75%
0.455
0.29
0,267
0.94
0.44
0:219
2.198
0,084
0.
0.105
0.336
0,44
5213
4.167
0.29
0.171
0017
0,317
0.593
0,487
0.000
0.145
0216
85.19
$1.455
..540
4.532
-2.3%
23453
23447
4.360
40055
40000
4330
4214
4.88
4103
4:51
4364
.2225
-2245
4.116
4.116
J1
4
185
41356
4392
40311
4.181
4.190
40142
-0336
-0307
4.300
-0227
-6651
.0023
4336
-0315
4,39
-
-0,86
-0447
-0100
-0100
-0152
-0130
-0274
4.85%
4.319
4:911
41235
41336
4164
4N1
-0345
4.117
41
101
41367
4.359
4,191
-1.25
4123%
46631
0.395
-0544
-1861
-0194
-1109
-0365
92.5%
0.555
0.448
0.472
0.357
00000
0255
0336
0.275
4,108
4.117
0.344
0.33
0253
0.305
0101
0.367
0.33
0621
0415
0101
0.147
0.99
9.39
0.50
0.400
0.434
01364
025T
0.00
0311
0.29
0.120
0.105
0230
0.358
0.59
6211
0143
0362
0.277
0634
0415
0811
.0442
0102
0.19
0.29
107%
4.562
44,90
40619
41263
4007
4330
4.0
4373
-.31
4.0%
4.113
41200
4312
-0222
4221
4,33
4
4.23
4664
4444
2,630
.0100
4.15
.311
46

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Model performance gap A of RD experiment under ROC-AUC on Byemovements dataset.
Degrae
INT
DCN
5AT
SNN
TTF
TABB
NCA.
1M4x
TLLM
UNI
AVE
3.7%
01011
0001
01131
0,002
3033
0011
741%
0.001
-0011
-0305
-0037
40002
4.003
.0.066
-.50-
-0501
-0817
-2088
-0.115
-3013
-0814
-001T
11.01%
-013
4083
4107
4004
-0025
41103
4/010
-0811
-0102
-1057
-0009
40000
-0011
-0113
01803
-1191
41002
-0559
-0/33
-0.19
-2016
-6019
-0/34
14.819
-081
4085
4.009
3000
-0035
40008
40011
4000
.003
40019
-0031
-0111
41093
4005
40.001
-0511
-0010
-.147
-0.13
-5011
-6817
-0/16
18.52%
0.048
0380
0.01
0.104
0011
0024
0.010
0.03
0/033
0.015
0.000
0.51
0.200
D025
0011
0.877
0.099
0.017
0.002
0.04
0.154
0.190
0021
0666
0.43
23.225
4,09
.0.05
4.065
2016
-0022
4.004
4.055
-0035
2,025
2.038
4002
40397
.0042
40023
-0002
-0.056
-0.033
-0014
-0059
.0.21
-0.205
-0013
-0022
-0059
25.91%
0004
-0005
41085
41001
4.053
-01335
41057
4001
-0043
-0H123
401812
-0053
41.06
-0006
-0/83
-02217
-02144
-01900
-8011
-0/89
36A75
-0077
114
41002
40016
40046
41007
4.051
-5040
41003
4109
41066
40042
-0648
-0H75
-0017
-0107
-0.08
-0005
-0/983
-0.235
-0:23
-0890
-0014
-0/86
33.32%
0,055
0,835
0.124
0.002
0012
D.045
0.015
0:055
0.442
0.015
0.033
0.06
0.355
D/46
6/125
01001
0.050
1.357
0603
0.085
.0.447
-0215
2644
0629
0.06
37.04%
4.10
-0.00
-0.137
4..9%
..019
.0062
4.005
4,075
-3457
4.082
4.08
40079
40057
-0059
.0035
-0000
-0.00
-0.52
.0615
-0.14
.0.291
-0.25
-5041
-0659
.0085
40.74%
4,09T
-0,09
.0.135
4.067
.0049
.000%
4.036
4.114
-0085
4.065
4.077
..118
.0.89
-0091
..069
-0,099
-0.140
.0.500
-0636
-0.05
-0317
-0.25
-0035
-0625
40.04
44.44%
-.115
40.13
4.009
-0071
-3043
-0147
-019%
40.01
10600
-0.40
-0.57
-07594
-3045
40688
-0101
0.24
51.85%
0.143
0.355
3051
55.56%
-0.63
4.352
3009
-0023
4.125
62,951
0.20
0001
0091
0J015
0.165
0,121
0058
0.66
0.371
0.225
2004
0014
0,145
$6,77E
GNIL
0.21
0,355
0312
2004
0.51
70.37%
4.193
-2192
-025
4.153
9.076
-0087
4.176
-0141
10000
-.176
4.140
-0133
4.112
4,015
-0.95
4,122
.051
4:234
4:43
-022
-007
4.62
34,0%%
4.171
0.00
0.30
4L170
4006
0114
4:111
2111
4088
4100
1.316
-0145
4007
40500
4:213
-3311
-0014
-114
71.799
4.218
4.216
-228
4L172
-000
-1119
4 1.108
4:00
-2121
40
4111
-1700
-017%
4.190
4003
42118
4115
4.365
2:445
0.59
.007)
0.144
$1.4%%
0.246
0:228
0.307
0.227
0.122
0.204
0.128
0:229
0.202
0.153
0.16
0225
0.197
0.24
6.161
0J06
0243
0,179
0,777
0.251
0,439
0,33
0075
0006
0,201
$5.19%
4.24
.2261
40.19
4.234
1.116
A0222
4:359
-2213
4.142
1.155
41390
4.359
-0215
4.176
4,02
4.235
4.196
-0066
-0277
.0,43
-0,32
-0075
0.000
-0212
5..5%
4.119
4:211
0:331
41310
11125
01316
4183
4:265
-0211
4.151
41105
41260
41311
41315
4,174
4107
41724
4.196
4.79%
+0445
-0315
-0075
46638
4.214
97.49
4.114
4:131
4.300
41144
4L360
41174
4.11
.0.60
4.164
41175
1111
41345
40366
4.214
4099
41
41.3%
-0100
-0.09
+0,49
-0.35
-0080
-034
0.314
0.31
0.38
0.278
0.178
0,200
0.195
0335
0,278
0.190
0.197
0312
0,711
0257
0.245
0M11
0.333
0.200
0.101
0.330
0.471
-0357
2082
0.329
0.255
100%
4,361
4,339
40.418
41.02
4031
40367
41219
4,31
-03311
43335
40.216
40352
40358
-0315
4.255
4.001
40.50
4.259
-0114
40364
-0,49
-0.371
-0651
0.827
-0.30
47

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 19. Model performance Iris dataset.
(a) Model performance of SC experiment under accuracy Iris dataset.
Columa
PCC
LGB
XGB
CNTB
PFN
DAN
MLP
NC0E
5W]
CAP
NET
G05
DCN
FTT
CRD
SAT
5NN4
TTF
TABR
NCN
LMA
TLJM
ANE
0.94
0.999
0.600
0211
0.322
01822
0.63
0.889
0.855
0,81
0.478
0.44
0.333
0.867
1,000
0.905
-0427
0.989
0.556
0.667
0.967
0.875
0.999
0.344
0911
1.00
0.819
0.755
0.918
0174
0.799
0.367
0.243
0.789
0.756
0856
0.56
0.867
0.335
0.78
0.755
(b) Model performance of SC experiment under ROC-AUC on Iris dataset.
Colama
PCC
LGB
XGB
CATB
PFN
DAN
MLP
.NODE
RE5
.W]
CAP
NET
005
INT
DCN
FTT
CRO
5AT
SNNN
1TF
TABR
NCA
LMA
71.1M
UNI
AVE
0.993
1,000
0.968
0.963
0.96
0.918
0.997
0.816
0.966
0.433
0.994
1.000
0.907
0.925
-0442
0.997
0.999
0.914
0.955
0.956
0.18
0.916
1.00
0958
0.988
0.337
0.998
1.800
1.000
0538
0.925
6.783
0.964
1,000
0.73
0.73
0,007
0963
0.950
0.73
0921
0.535
0.911
0.334
0.972
0.999
0.977
1,500
0982
0.000
0:49
0.523
0.594
0997
0.000
0.90
0.823
0,99
0912
0.13
0.974
0.93
0929
0.994
0.514
0,993
1,000
0.981
0.978
9:925
0.001
6.57
0.956
0.917
0.965
0.985
0.999
0.981
0.317
0.019
0.94
0.911
0.661
0.94
0.993
0.989
0.966
0.908
0.418
0.990
1.000
0.811
0.957
0.891
0.903
(c) Model performance of MC-M experiment under accuracy on Iris dataset.
Degree
LGB
XGB
CATB
PPN
DAN
MLP
NODE
RES
SWT
CAP
NET
GOS
INT
DCN
FTT
GRD
SAT
SMN
TTF
TABK
NCA
LMA
TLLM
UN]
AVE
0%
0.944
0.957
0.089
0.939
0,000
0.811
0.522
0.864
0.822
0.911
0.61
0.859
0.623
0.856
0.875
01-78
0.844
01800
0:355
0.861
1.000
0.913
1.000
0.967
0.808
35%
0.789
0.760
0.967
0.756
0.400
0.789
0.338
0.811
0.22
0.556
0.522
0.822
0.45
01769
0.875
0:435
0.678
0.811
0:356
0.844
0.556
0.733
0.900
0.867
0.695
50%
0.344
0.344
0.356
0.400
0.344
0.788
0.333
0.567
0.767
0.456
0.544
0.678
0.467
0.522
0.611
0.356
0.456
01667
0.267
0.556
0.333
0.267
0.600
0.800
0.496
75%
0.344
0.344
0.344
0.367
0.389
0.481
0.344
0.533
0.544
0.456
0.333
0.389
0.323
0.400
0.466
0.400
0.333
0.489
0:333
0.367
0.333
0.400
0.800
0.735
0.424
100%
0.344
0.344
0.344
0.344
0.333
0.335
0.333
0,333
0.333
0,333
0,383
0.338
0.333
0.333
0.333
0.333
0.333
0.833
0.333
0.333
0.333
0.367
0,600
0.667
0,370
(d) Model performance of MC-M experiment under ROC-AUC on Iris dataset.
Deguee
MLP
NODE
RES
SW]
INT
DCN
FTT
GRO
SNN
TABR
NCA
TLLM
UNI
AVE
0%
0.944
0.967
0.989
0.939
0,500
0.811
0.522
0.944
0,822
0.911
0.633
0.359
0,622
0.856
0.578
0.478
0,944
0.800
0.333
0.567
1,000
0.033
1,000
0.767
0.306
25%
0.789
0.759
0.967
0.756
0,400
0.789
0.333
0.311
0.22
0,556
0.523
0.822
0.475
0.759
0.578
0.433
0,678
0.811
0.356
0.544
0,556
0.733
0.900
0.867
0.695
50%
0.344
0.346
0.355
0.450
0.344
0.78
0.335
0.567
0.767
0.456
0.544
0.78
0.61
0.822
0,11
0.356
0.456
0.667
0.267
0.556
0.333
0.267
0.800
0.800
0.496
75%
0.344
0.344
0.344
0.367
0.389
0.411
0.344
0,533
0.544
0.456
0.333
0.369
0.322
0.400
0.456
0.400
0.333
0.689
0.333
0.567
0,333
0.400
0.800
0.733
0.424
100%
0.344
0.344
0.344
0.344
0.333
0.335
0.333
0.33
0.33
0.333
0.333
0.333
0.333
0.33
0.333
0.33
0.333
0.333
0.333
0,333
0.333
0.567
0.800
0.667
0.370
(e) Model performance of MC-L experiment under accuracy on Iris dataset.
Degsce
XGB
CATB
MLP
.N0DE
RBB
SW1
CAP
GO5
1NT
DCN
FTT
CRR
5XTT
5NN
T1F
TABR
NCA
LMA
TLLM
UN1
AVIE
0%
0.864
0.622
0.522
0.856
0.575
0:47
0.944
1,000
1.000
0.767
0.308
25%
0.867
0.867
0.533
0.878
0.575
0.489
1,000
0.033
0.319
50%
0.055
0,444
0.744
0.656
0.522
0.844
0.511
0.367
1,000
0.967
0.767
0.340
75%
0.278
0,444
0,744
0.711
0.767
0.644
0.33
0.444
0.767
0.596
100%
0,342
0,333
0.335
0.33
0.333
0.33
0,500
0.667
0,370
(f) Model performance of MC-L experiment under ROC-AUC on Iris dataset.
Dognor
XGB
CATB
PRN
DAN
N000E
RES
SW1
CAP
008
INT
DCN
FTT
CRO
3XT
TASK
NCA
TLLM
UNI
AVI
0%
0.997
1.000
1000
0.943
0.991
0.35
0:997
0.953
0.966
0.500
0.955
0.621
0.997
0.999
0.66
0.96
0.917
0.483
0.991
1.000
0.951
1.000
0.977
0:979
x0%
0.908
1.00
0.909
0.911
0.859
0.498
0955
1698
7666
0.51
1.000
0.954
0.94
0:990
0991
0:532
0.ws
1,000
0.951
1.000
0.915
0.955
14%
0.681
0.940
01997
0.740
0911
0.818
0.911
0.98
0.831
0.937
0.431
0:51
0.87
0.789
0.900
0.988
0.95%
0.91
0,335
100%
0.500
01 0:500
0.500
0 1:0
0.500
0.500
0 500
0.500
0.500
0 6.500
0.400
0.500
0.593
0.500
0.509
0.500
0.500
0.400
0.500
0.900
0.500
0.500
0.500
0.938
0.534
(g) Model performance of RD experiment under accuracy on Iris dataset.
Degpoz
LGB
NGB
CABB
19N
DAN
MLP
N00E
8B5
SW1
CAP
NET
005
INT
DKN
FTT
CRR
SAT
SNN
TTP
TASR
NCA
LMA
V...M
UNI
AVI
0.822
0.47
1,000
25%
0.783
0.534
0.817
0.899
0.940
0.823
0,739
50%
0.641
0.600
0,836
0.750
0.426
0.292
0.37%
6.703
0.728
0.693
0.531
0.806
0.515
0.260
0.31
0.220
0.600
0.680
0.300
0.777
0.543
0.533
0.887
0813
0.657
75%
0.385
0.450
0.356
0:572
0.831
8.5%
0.367
0.593
0.09
0.611
0.907
0.611
0.402
0.591
0.511
0.363
0.373
0.551
0:327
0.500
0.356
0.583
0.53
0.733
0.538
1009
0.383
0.333
0.333
(h) Model performance RD experiment under ROC-AUC on Iris dataset.
Degsee
LGB
XGB
CATB
PFN
DAN
MLP
.N0DE
RES
.W1
CAP
NET
GO5
INT
DCN
FTT
GRO
5.NT
SNNN
TTF
TABR
NCA
LMA
TL.M
UN1
AVIE
0%
0.992
0.997
1,000
1.000
0.356
0:997
0.953
0.986
0,899
0.821
0.997
0.959
0.66
0.937
0.994
1,000
0.054
1.000
0.907
0.028
25%
0.941
0.963
0.995
0.958
0.979
0.778
0:974
0.889
0.948
0.789
0.974
0.932
0.997
0.977
0.586
0.979
0.998
0.902
0.970
0.945
0.397
50%
0.872
0.955
0.738
0.934
0.444
0.907
0.769
0.926
0.905
0.75
0.999
0.525
0.966
0.967
0.574
0.927
0.945
0.350
75%
0.764
0.710
0.832
0,119
0.758
0.674
0.831
0.778
0.836
0.705
0.729
0.14
0.554
0.335
0,892
0.913
0,878
0.702
0.009
0.927
0.779
100%
0.500
0.500
0.500
0.300
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.900
0.800
0.918
0.534
(i) Model performance gap A of RD experiment under accuracy on Iris dataset.
Degree
LGB
XGB
CAD5
IVN
DAN
MLP
MDDE
R15
SW1
CAP
SET
GUS
IN1
DCN
FTT
GBU
S/1
SNN
11F
'IABK
NCA
LMA
11.M
UNI
AWE
25%
.0.12
4..75
.1100)
41.10
41131
-0030
40.15
002012
200366
4.130
0.142
4.000
41141
41000
4151%
-0067
40125
-0101
40003
.0000
41.228
.0127
40000
0004
-0086
10%
.0.318
.0.500
163
.0.343
403077
-0.997
4.227
4.167
-0.15
4.239
.0158
40004
4.171
-0.12
4.181
-01211
-0330
-0.300
4,000
-0.104
40457
.0000
.0.13
0061
-0.17
79%
-0.55
40.34
0.438
0.421
0.281
0.390
0.336
4.297
02235
4,339
4.300
4.20
4.354
0.309
0.418
00242
0.655
-0311
4,022
-0333
0.641
-0375
0.147
0/09
-0317
100%
0,025
0.641
0.062
0.662
0.414
0.559
0,362
0.86
0.95
4634
0.14
0.025
0.561
0.500
0.500
0.300
0647
-3583
0.000
0.15
0.607
0.007
0.200
0.120
00542
(i) Model performance gap A of RD experiment under ROC-AUC on Iris dataset.
Degree
1GB
XGB
CXTD
PFN
DAN
MLP
NODE
TABR
NCA
LMA.
UNI
AVE
25%
00051
0,031
00004
00002
-0004
-00016
-0091
0,004
-0,067
4,039
-0.13)
00.03
-0.03
4.003
00.033
4..00
00.018
-0.64
-0.000
-0.016
00.00
-0.033
-0,000
6/042
-0033
50%
0.121
0.128
00149
-1145
00.15
00143
01150
00644
0.115
0,079
4.194
(1067
.0063
4.171
01186
41062
41167
41116
41105
(U129
41163
11084
.0163
0/922
-0475
79%
-0.300
-0.38
-0.16
-0181
-0.300
-0199
41312
41
41233
-1364
-0.16
1006
-0.95
4,499
4,300
00.90
-0.43
-0.97
-0416
4,889
-0,45
4,403
-0.44
40.94
-0.391
4,499
-0.47
-0 -0.18
-0.98
-0.66
0.96
-0.97
4.500
-0476
-0.10
0004
-0.45
48

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 20. Model performance on Jannis dataset.
(a) Model performance of SC experiment under accuracy on Jannis dataset.
Cobern
PCT:
1.G8
CATR
PNN
DAN
SW6
GDS
DCN
SAT
T1P
TABR
NCA
AVR:
0.690
0.715
0.629
0.657
0.550
0.665
0.633
0.694
0.675
0.655
0.755
0.616
0.751
0.725
0.662
0.619
11966
0.366
0.616
17700
0.25
0.000
0.882
0.715
0.629
0.67
0.69
0:511
1,005
0.33
0.994
0.679
0835
0.531
0.705
0.637
0331
0.725
0.50
0.581
0.663
0.000
0715
0.639
0.57
0,60
0.665
0.626
0.644
0.69
0.64
0.500
0.355
0.38
0.666
0.700
0.225
0510
0.579
0.64
0:000
0.600
0.881
0775
0.630
0.57
0.665
0.627
0.855
0.65
0.65
0.511
0.307
0.26
0731
0.225
0450
0.584
0.548
0.67
<0001
0.66
0.10
0.12
0.699
0.56
0A6
0.550
0A87
1000
0611
0.50
0.90
0.55
0.25
0678
0551
0.66
0557
0005
0224
0510
8363
0.001
0.600
0.681
0.715
0.630
0.677
0.699
0.500
0.695
11665
0.83
0.626
0.944
0.561
0.79
0.855
0.28
0.706
0.63
1020
0.225
0.570
0.590
0450
0.662
7 0.001
0.600
0.611
0.715
0.600
0667
0.60
0.550
0.693
0.665
0.633
0.536
0.694
0.562
0.69
0.644
0.511
0.706
0660
0055
0.735
0.500
0.565
0.550
0.63
0:011
0.000
0481
0.715
0.600
0658
0.69
0.550
0.655
0.666
0433
01627
03633
0,02
0.679
0085
0150
6206
1005
1031
0.725
0,000
0.594
11664
9
0:001
0.681
0.681
0.715
0.600
0658
0.659
0.550
0.692
0.665
0.633
0.525
0,944
0.561
0.75
0/654
0,500
6.706
0.638
0236
0730
0.255
0,531
0,552
0.847
0.60
10
0601
0.681
0.682
0.715
0.600
0.58
0.70
0.550
0.695
0.665
0.633
0.527
0,94
0.662
0.79
0.655
0.332
6,307
0.677
0237
0731
0.725
0.310
0.594
0.52
0.600
11
20601
0.690
0,62
0.715
0.11
0667
0/669
0,550
0.663
0.665
0.533
0.636
0.655
0.661
0675
0.655
0.511
6.306
0668
0881
0751
0.335
0.350
0.556
0,50
0.66
000
0.000
0.60
1006
0433
0.90
000
0054
055
0.06
0056
1051
0.725
00002
0.00
0.60
087
0.685
0550
0.00
0.225
0310
0550
0.665
0,00
0.610
01665
0.00
0000
0964
0.20
0000
0.22
(650
0260
0,665
0.00
0.001
6681
0.15
0,30
0668
0.69
0550
0.93
10000
0413
0020
0.9
0500
967
9684
1520
0200
1987
9026
0222
0216
0,609
0.003
0.681
0.682
6715
0.630
0.57
0.69
0.560
0,66
0.624
0525
02811
0590
0.0
0685
050
0.007
068
1200
0.22
0440
0
0,665
17 .0003
0.600
0.622
6715
0,611
0.67
0.69
0.560
0.62
0,66
0.07
0.66
0,73
0500
0.00
0667
K229
0.225
0,663
18 0,044
0.6
0A62
0215
0,10
0688
0.60
0.960
6.6%
1906
0013
0822
0204
002
0965
055
0.00
863
1
0.225
0100
1000
19
0.004
0.600
0.811
0775
0,30
0.677
0.69
0.500
0.993
1006
0433
0020
0.94
0101
067
0665
0500
0707
0667
0020
0.225
0,59
0363
1663
20
-0.04
0.690
0.681
0715
0.629
0.657
0.70
0.560
0.683
0.665
0/433
0555
0.994
06561
0073
8581
0.706
0669
0730
0.725
0.60
0147
01855
21
-0008
0.000
0A42
0315
0610
0167
0.700
0,550
0.695
1166
0413
1000
17941
11:00
1051
0988
0551
6206
1000
0.2
0A0
0.3
0151
106
22
06605
0.600
0.681
0.755
0831
0.657
0.699
0.550
0.693
0.665
0/633
0223
0994
0,50
0075
0655
0332
6307
0.638
0235
0030
6.25
0.10
0.591
0350
0664
29
-0005
0.690
0661
0315
0610
0657
0.656
0.550
0.683
0667
0.534
0821
0.661
063
0/644
039)
6306
0668
0986
0331
6335
0.550
0,500
0343
0.67
24
-0605
0.681
0.61
0715
0.10
0657
0.50
0,550
0,683
0,665
0/433
0856
0.661
083
0665
0555
6.366
0667
188
0131
0.355
0.50
0.584
0344
0665
40006
0115
0629
0.99
1066
0633
1004
0006
1851
0776
0570
0.006
0,665
0,95
0.661
0579
0.65
0.51
0.206
0331
0.77
0.66
40007
0.610
0,66
0.00
0.00
0965
0.00
0668
0000
0.25
0660
0.561
0,66
0006
6681
0.75
0,600
I667
0.69
0.50
1000
0013
0627
0.90
9945
(22)
0206
103
9026
1221
0.25
0640
0367
0.600
29 .0009
0.673
0662
0.705
0,627
0.64
0.61
0.560
0,683
0,0
0628
0623
0:011
0653
0,73
0678
058)
0.677
0.64
1200
0,590
0.553
0.44
0.63
.0546
0.673
6677
0.10
0.629
0658
0.67
03500
0.691
0,65
0.22
0.07
0500
0500
0.75
0683
0529
0.00
0698
0625
0229
0720
8510
0.50
0.46
0.662
M
0,06
0.75
6617
6110
0.05
0667
0.61
0.960
0,622
0663
0/623
(.2)
0.51
0569
0,64
0440
0521
0.60
0.215
0.93
0,65
32
0048
0.674
9070
1110
9625
1105
0.688
0.50
0.099
1066
0629
0020
0.911
11000
0100
0675
0084
0520
0.00
1011
1025
0225
0727
0,00
0.555
0.447
0,000
33
-0065
0.677
0.688
0.699
0.666
0.65
0.560
0.691
0/622
0.25
0.994
05588
0.77
0/644
0520
0.00
0666
0,35
1027
0.725
0.620
0.562
0.547
0,63
S4
1010
0.64
0968
0601
[1664
0.65
0.551
0,12
0667
0.656
11544
1065
09666
0320
0356
1005
1009
0.09
0.501
015
11618
35
0.106
0.676
0:17
1111
0824
0.644
0.64
0.559
0.689
10622
0662
0325
0390
0.57
0075
0653
0.23
0896
0.635
0231
0769
0.000
0,579
0.819
0.60
36
2016
0.67
0,58
0315
0829
0667
0/65
0.560
0,601
06665
09612
0.27
0981
0.600
0675
0d65
0.590
6300
0638
0835
0.31
0,550
0.581
0.340
0.6
37
0.116
0.644
0.56
0314
0829
0653
0/65
0.561
0.689
0,665
0:531
0.366
0982
0.688
0875
0:65
0.539
6.30
0.65
0235
0155
0.724
0.10
0.553
0347
0662
4015
0.77
0618
11114
0669
0187
0MA
0.550
1966
0613
01856
0994
0661
1064
0665
0.50
6306
068
0995
1959
0935
0400
0.165
0.674
0629
10666
0,90
0677
0.644
0.712
0,00
0,662
0.67
0655
0,67
0/610
0,00
0.50
0.6%
0,65
0.20
0.00
000
0.11
0,00
0,62
401
0.073
0629
0056
0.00
0361
1661
0013
1604
100
0964
100
9070
0065
1927
900
0000
0260
0,00
0.168
0.677
0677
6.00
0.255
0.67
0.67
0362
0668
0,665
0/12
6623
0.88
8659
0.83
0660
0.89
0626
0200
0715
0.000
0356
0248
0,60
6
0.7)
6A79
6314
0.69
0688
0.66
0.519
0.69
1,66
6622
0.09
848
002
0663
1520
0.34
200
1010
0229
0.24
060
0.99
1551
0000
0:210
0.600
0660
0715
0.69
0.56
0.66
0.563
0.61
0,66
066
0.844
8699
0.77
0664
053
0.76
8685
0,05
1722
0.255
0.500
0.581
0347
0,62
0218
6.0%
0622
0646
0.60
0350
0.044
1660
0429
1005
060
0619
0224
0.09
1109
0685
0.300
0.64
032
1600
46
6.233
0.673
0,675
0315
0.629
0665
0.655
0.562
0,690
16665
0/632
0523
0.92
0:61
0.75
0/633
0523
0.706
0636
0,33
1722
0.725
0.610
0.572
0148
0,922
47
0.342
0.66
0,12
B311
0,69
0167
0.65
0.651
0.666
11665
0.61
01677
0311
0.50
0665
0661
8322
6.20
0.944
1179
0.723
06600
0.564
0157
116
48
0.29
0.54
0583
0.61
063
0645
0,545
0.668
11645
0.612
0.73
0665
0.57
0001
0650
0.30
0/37
16615
0615
1.096
063T
0,570
0,77
0341
0.3
40257
063
0.644
0704
0.25
0648
0.555
0.667
0664
0/627
0.15
0.688
0653
0657
0.676
0.23
0.69
0,699
0629
0.094
0.754
0.600
0.550
0334
0630
50
0.306
0.66
0,570
0314
0.29
0.66
0:655
0.555
0,500
0.63
0.530
0.524
0.94
0.666
0677
0/644
0.21
0.766
0.688
0833
0720
0.733
0.510
0.500
0.25
066
S1
2010
0666
0666
0111
0611
0668
0/665
0.158
0.67
0667
0.11
0162
0.600
0668
0665
066
0.525
0.609
0624
0611
0724
0.10
0.00
0.63
03477
0.60
0.317
0629
0666
11667
0.993
0.705
0635
0/27
0.719
0.622
0.310
0.612
0651
0.660
0,625
0.665
0.688
0.12
0.207
0.33
0.332
0,023
6A17
0.64
0M53
0.603
0.355
0060
(.53)
0,68
0.624
0678
0.59
0.45
0.31
WT17
0.115
0,62
0.651
49

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(b) Model performance of SC experiment under ROC-AUC on Jannis dataset.
Colama
PCC
LGB
XGB
CATB
PTN
DAN
MLP
NO06
RES
5W1
CAP
NET
GO5
DCN
GRD
SAT
5NNN
TTF
TABR
NCA
TLLM
AVE
0.545
0.777
0.751
0.517
0.33
0.747
0.835
0.266
0.792
0.855
0.839
0.854
0.674
0.364
0.791
0.587
0.883
0.804
0.000
0.545
0.751
0.517
0333
0.33
0.266
0.855
0.839
0.854
0.855
0.751
0.557
0.883
0.000
0.84
6340
0.791
0517
0831
0835
0.766
0.855
0.889
0,854
0.864
0.791
0.586
0.883
0.50
0530
0,80
0.001
0389
022
0817
0747
0,55
0825
0766
0.71
0355
0309
0854
0.70
0.844
022
13356
0381
6700
0526
0:000
0.329
1877
0.23
0817-000
628
6356
083
0766
0.793
0.35
0.529
0522
0354
0.7
0.664
0.78
0.00
0.877
0.83
0.00
0800
-0001
084
0.44
0823
0260
0816
0811
6261
6.500
0181
0.66
0.99
0.313
0326
088
0244
0:55
0.081
1
0.80
0:87
0.505
0.99
0528
6
7
0.001
00001
0.899
0528
0.540
0,540
0.349
087
0221
1
1317700000
18177000
0381
0.30
0247
6241
0F59
0.55
1630
0.766
0.70
0.766-00
0293
0.99
0355
0888
0357
0:55
0.56
0675
0625
0384
0384
01.0
0.781
0222
0221
1
1
0
0500
0:300
0290
0520
05200
019
8
0:001
0.449
0.54
0.817
111770000
024T
1035
0.766
0.93
0355
0338
0.554
0574
0364
0
0.781
0
0500
4559
0520
9
00000000
058
0.50
6.449
1
11517
11877000
0.41
0.154
1085
0.766
03355
11:29
1880
0505
11:00
0
1
0.38
0520
10
0001
0.44
0.349
0377
0791
0791-000
0.917
0303
0.41
0,554
0335-00
0.766
0355
0,529
0339
0.854
0.71
0355
0.731
0791
1037
03833
0,501
0.56
0551
0104
11 .0001
0.49
0.349
0.73
0.32
0.117
0.33
0.347
0.354
0335
0.766
0.92
0355
0.339
0339
03854
0.674
0.344
0.781
0331
0.333
0.361
0399
0629
0.04
4000
088
0.449
0743
1166
0351
0231
0.55
0055
0.56
03355
19859
0,55
0.04
0.5
09511
10827
0800 0
0,00
6,088
0.000
0330
0030
0.201
0817
0800
0301
0201
0.09
6388
1200
1200
0.533
0333
13500
0080
1000
055
0.30
0.084
0365
0020
0234
0.000
0.081
0.881
0310
8356
0,00
0.00
0200
0006
1877
0223
0117
1280
1880
0709
0200
0.80
1938
0000
17 .0000
0.39
6340
1
0517
0.54
0766
0700
0333
0550
0
0080
0350
0364-000
0220
1989
0
0383
030
18 0.004
19
0000
0848
0889
0344
0.49
0872
0.21
1221
11887
011
0800
1300
6241
0261
0159
088
1630
0.70
0.66
13:00
1881
1332
0250
0000
0604
0.3%
0900
0..00
0220
9220
1983
000
188)
20 -0.04
0.449
0.54
0221
0.887
1330
0359
0.760
0270
1229-00
0/500
0973
0365
0220
22
21
-0005
00.04
0.549
088
0,349
6.549-000
1877
02211
0221
11317
13887
1330
0311
0.47
6247
0.55
0:154
1088
0035
0.76
0.766
0170
0355
11355
11530
01:00
13200000
1080
0337
0/554
0:554
0571
0624
0356
03655
1
0.78
11:81
1024
0220
0221
1139
0:883
033
05500
11894
2F 0,005
0.149
0377
02511000
01517
0.41
0:411
0.550
1035
0.766
0.7661100
0355
13330
1030
03554
0074
635
11781
0221
10377
0381
0201
0266
11004
24
-0.00
0.549
0.349
0351
0391
0,817
0333
0:41
0.354
0.354
0335
0.766
0393
6355
0.839
0339
03854
0.74
0354
0.781
0391
1036
0:333
0.000
0390
0804
24
40005
088
0.540
0851
0941
0.44
0885
0.56
1385
08200
10845
0,34
0001
635
01181
09511
1157
0,500
0351
0551
1033
0200
0330
0080
0.655
0300
27 40007
40006
0.849
0.84
0.00
0301
0201
0241
1200
0.00
0.338
0000
0380
0.86
0.355
0365
0204
0220
0.081
03881
0316
30 0.06
40000
0.4
0.90
0.49
0225
0220
010
1221
6297
0244
0359
10000
0.00
0351
0330
055
0.08
0363
0220
0220
1889
1932
0280
02200
1000
1900
S1
0.45
0800
0220
011
0246
6346
0885
0.65
1
0.44
0881
0823
0.3
0.37
0226
0.30
32
0440
1828
1228
11100
1380
0281
1039
0.70
1922
1880
02535
1972
038
9229
1989
1380
1880
33
$4
A018
-0083
0.53
0:41
1890
0575
1891
0230
1100
0116
1330
0247
0247
0
0:52
1092
0.765
0.76
0770
11381
18319
0327008
0830
1033
00540
0670
0823
0355
0356
0.700
0239
0255
1153
1987
0:375
0.389
1024
0866
1804
0258
11315
0.746
1083
0.764
13333
1185
0358
0.777
0237
0 1052
0:33
(10)
36
4.116
0:488
0730
0740
11917
0303
07488
0,55
1005
0.766
0334
1030
0035
0825
033
0750
1030
0381
000500000
11011
37
0.116
0.445
0:41
0556
0350
0.114
0330
0.745
0,350
0034
03764
0293
0334
0 0327.
033
0355
067
6383
0.778
0239
1053
080
2019
0.165
033
0.30
0340
0859
0750
0941
11317
01117
0811
0:347
0.344
0885
1053
0.66
0334
6335
1823
0800
1891
0355
0661
03651
6385
01080
11700000
1150
1156
0381
0530
0000
40
0.477
0.200
0.16
0.553
0300
0000
0.50
0.20
0,00
0.184
0.43
0340
0.20
0.15
022)
0.44
0200
0000
0.40
0220
0.881
0.00
4188
0141
0.49
0516
0820
0240
0700
0.211
0200
0.531
0.778
0220
1122
0800
6.18
0.445
0.40
1826
0.50
0517
1380
6747
0,51
0,551
0766
1128
0225
0363
0220
0220
0300
0:200
0.845
6349
6:49
0877
0.70
0117
081
6748
0.522
0885
0.66
0
0.33
0326
0.83
085
0670
0.6
0.777
0.20
0.85
0.331
0.00
0800
UK9
0.15
1891
0284
1111
0246
0246
0.44
0.44
1080
0.063
0/800
0211
1000
0344
1.771
0229
11800
0330
0520
46
6,233
0.566
0,347
0577
0.50
0.50
0.815
0331
0.746
0.355
0.000
02902
0.554
0335
0.53
0667
0365
0.700
0.75
EV5R6
0:881
0:10
05503
0529
0302
47
0.342
0829
0.542
1806
0.40
0.200
0331
0.400
0.548
0.548
0083
0.765
01811
11
088
0.55
0350
1L782
0.050
0269
0
11100
0522
0000 0
48
0.34
0.535
0320
0.770:00
0.62
0x15
0.737
6.737
0.540
0.750
0757
0.330
0823
0.540
0563
0.54
(1760
0.774:00
0.55
0.500
01588
0523
0.78
40267
0335
0.117
0776
0112
0313
0.47
0,540
1035
0.763
0322
0389
6355
1L771
0.785
1100
0370
50
00306
0.54
0.345
0.77
0.70
0.17
0331
0.747
0,352
0837
0.766
0.791
0.54
0.577
0335
0,53
0.67
0.53
0.790
0.785
0.586
0.812
0.506
0.5%
0.64
8314
0.02 0902
SI
AL310
0:47
0.347
0.56
0.799
0117
0.30
0.341
0.550
08%6
0.764
0.788
0,33
0,22
0837
0,55
0.65
6.3%
0.784
0.749
01581
0.834
0.506
0359
0604
0.00
0.317
055
0.700
1117
0036
0354
0335
0.66
0.851
11781
0.750
11382
0.75
0.508
0.319
0813
0.784
0511
0323
0.742
0.22
0.759
0.443
0.800
0.44
0.66
0.347
0.786
0.875
0.869
0:500
0.55
0,332
0814
0.766
0813
083
0.350
083
0.546
0.768
01775
0.375
0.59
0.201
50

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(c) Model performance of MC-M experiment under accuracy on Jannis dataset.
Dearee
LGB
XGB
CATB
PRS
DAN
MLP
NODE
RES
SWI
CAP
NET
GO5
DCN
FTT
GBO
5/T
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
1.85%
0%
0635
0.62
0.680
0.636
0.57
07705
0.644
00000
0100
0644
0.550
0.355
0.560
0.59
0.87
0.693
0.665
0.649
0.662
0.15
0.628
0.635
0.506
0.500
0.535
0.994
0655
0644
0.674
0.678
0.622
0.678
0.685
0.515
0.514
0.530
0.705
0.685
0.57
0.620
0.633
0.638
0.60
0.636
0.217
0.66
0.725
0.716
0.560
0550
0.582
0.50
0.680
0.662
0.677
5501
0000000000000000000000000000000
0000
100
00000
0:52
0657
0,009
0617
0.M
0064
0064
0646
0.02
0510
0.63
0.20
0.05
0001
0622
1622-0000
063
0000
063
060
0.568
0647
0,649
0.13
0.566
0.66
0642
0639
0.52
0.002
0.25
0.18
0915
0.600
0663
0.00
1200
0550
066 DA0
0.10
0.644
0.00
0624
110440000
0.400
0615
0615
0.05
06
0.99
D6W
13,963 0.968-000
11.118
0617
1374-000
0:524
0007
0000
0000
0.933
0
0329
0
0580
0.504
0.815
0,2
10055000
0:566
1590-000
0.50
0.481
000
0.93
0000
1981
0:90
9621
0642
0883
0476
0.64-000
0.571
0.607
0.581
0,000
0.599
0.587
0:66
0.099000
0.700
1.700-0000
0.55
1005
1680
0.584
0615
1 14.91%
1489-000
0:00
0E33
0.500
0.57
0.506
0.59
0394-000
0:557
0.456
0:545
6377
0555
0331
0453
0.541
0.587
0.555
0.555
0,900
0.30
1555
0677
0.564
16.79
0:54
0.568
0.506
0550
0.86
0.568
0.455
0.543
0.558
0550
1130
0488
0.534
0.579
0.555
0.54
0.589
1607
0.561
1 18.52%
0:330
12499-00
0.061
0:455
0.4%6
0474
0.533
0.500
0,343
0.476
0,73
0258
0258
0.26
0.44
0.530
0.191
0.435
0.467
0300
0.79
0.49
20.37%
0.455
0:43
0:136
0:45
0,254
0:445
0.479
0.400
C466
0.523
0.324
0.300
0.49
0461
0455
0442
0.43
0.47%
04469
0.47%
0.465
0800
03544
0677
0.499
21.229 00022440000
0.12
0.256
0.419
0A5
0.508
0.515
0.94
0.006
0442
0.636
0,451
0.888
0.200
04488
24.07%
00.0%
0,41
0.55
0400
0.60
0,99
0050
0.49
0.43
0479
27.70% 0
25.099
0.400
0.300
0.000
0410
0.500
0.000
0.50
0.50
1110
0-16
0.15
0419
0.40
0,407
0.15
0350
0676
0677
0.422
0.460
25
1.200
011
0/00
1900
0010
1111
0.595
0/00%
6.088
11007
101
04W
31 31,48
0.400
1410
0.60
0553
0:553
1-110
B410
0280
0880
0390
0.405
0,40
0.425
0.000
0.40
0550
0.73
0438
38,590 38,599
20, 198
37,040
0
356
335
0:337
04100
0:302
0
1361
0300
0360
0.17)
0:555
0.291
0-12
0:100
0:355
0:463
0:061
0:867
0:467
0357
C300
0.295
0.002
0.25500
0.900
0.80
0.358
0.
0223
0A01
0
0410
0288
0381
1611
4412
0880
1181
0257
1221
0390
0394
0396
039
0.580
0.483
0.563
01386
0-431
0,438
1.400.00
0
0.90
0,45
0,993
0.427
0,404
01.90
0.347
0.375
0-400
0:375
0.347
0.999
0.20
0:578
0.342
0.996
0.760
06600
0550
0555
035
1001
0.79
0.76
0441
0.41
0.424
0.404
40.74
8.855
0.341
0:53
0:002
0
1.400
00400000
0:400
0.385
0.:23
03:61
11:00
1
01562
0:418
0.10
0.
1.555
0,347
0.60
0.500
15500
1167
1679
0.113
41.59% 00.9990000
0.347
0.349
0.354
0359
0.34
03667
03378
0.402
0.335
0.300
0.313
0330
1350
0349
0397
0.355
0,415
0.110
11355
0.241
0.630
0.555
1.76
0.455
44.44%00
0.342
0.342
0.31
03333000
0.360
0.349
0.365
0.467
036
0.399
0.456
0.300
03365
0
0381000
0312
0.396
0.353
0.428
0.411
1.336
0.317
0.650
0.575
0675
0.404
48.15%
46.33 1..3%-0000
0:377
0.322
0.319
0.356
0.342
0.51
1254-000
0335
0.377
0.30
03385
0:361
0,607
0.462
0.53
0,345
0150
0.1
0.349
0.88
0.
0200
0.140
020
1950
0.196
0.344
0.540
0.424
0.409
0.550
0.313
0.299
0.260
0.570
0.5%
03550
1683
0.392
0.377
51. $1,859
0.314
0:315
0310
0,30
0.500
0:34
0:34
0230
0200
0200
022
0300
0.343
6009
0.30
0.361
1:300
0350
0221
0981
0.379
0
35 35.506
0:317
0:353
0320
0.50
0.00
0:00
0:000
1
0:44
0200
028
1200
0210
0213
0247
0326
03433
0.543
0428
0111
641
0228
0,60
0.701
0
0.40
1000
1000
0:373
0
61.118
0.335
0.335
0.366
0
0
0:115
0300
0.347
0.396
0.50
0.0
0300
0.30
0
0.467
C313
1
1
0800
0:445
0:444
0
0.444
0.745
0.34
0:44
0256
0297
0266
0240
0357
0358
1
0221
1229
0286
039
0390
0.44
0.344
0/4
0432
0.458
0:438
0/11
0,41
0:10
0.20
0.23
(120)
0.20
0.362
0.363
0.39
0229
0
0.20
0.350
0.40
0550
11855
097
1000
1690
1186
0.371
0.374
0:17
03772
0
13222000
0.317
0:347
0.333
0:000
2200
0
0.744
0.345
0555
1
0390
0.440
0.10
0410
[1205
0.361
0.350
1.550
1187
0.374
66677
03331
0:315
0337
0:35
03007
0039
03315
0.364
0.342
0284
0355
5
0399
11345
0.411
0.112
11:91
0
0.200
U1511
10600
0.310
68.53 68.53%
0.320
0.315
0.343
0.346
0.337
0.355
6367
0305
0379
0.445
0.341
0.343
0.04
0357
0245
0397
0,344
0.443
0.412
0.291
0.262
0.300
0.500
0677
0.311
N0.176
0.318
0.119
0:51
0:345
0.900
0366
6364
0999
0.445
0.141
0.541
0358
0361
0054
6399
0.544
0.449
0.15
0:361
0.330
0.5%
067
0,37
0.312
0.345
0:345
0300
C30
0,57
0.453
0.342
0114
0390
0.345
0.416
0.551
0.263
0.530
0.500
0.600
0.380
74.07%
0
0.921
0:315
0.345
0333
0.46
0.55
0.340
0
0.448
0415
0.25
0.365
0.76
0.60
0.350
75.939
0
0.315
0.310
0.345
0330
0.300
0360
0.60
0.57
0.500
0:30
0200
0211
0.344
0,400
0415
0416
(1:00
0.364
0.500
0.60
0.381
77.709 0
0.955
0:112
0:345
0.50
0.00
0:000
6200
0.57
0.200
0.70
02411
0.10
0.545
0,447
0014
0414
0.24
6.76)
0.600
0.63
0.162
79,639L 00
0.366
0:112
03344
033
0337
0200
0228
0.00
0.2%
0,275
0.200
0328
0200
1220
021
0244
0338
0.44
0,455
0.47
0.200
0,366
0G10
0582
0.67
0.54
0.155
0:111
0.44
03344
0337
03358
0.20
0.733
0/466
0.30
0:37
0291
135)
0
0.344
0.456
0,430
1
0.36
0.460
0535
0.79
0.377
0:111
0.50
0.300
0290
01373
0.44
0287
11500
0245
(d00
0.945
0.458
0.429
01297
0.261
1.300
11079
0.310
35.199
0.319
0.314
0.41
0.363
0333
0.00
0:50
0.667
0.290
0.371
03711000
0:577
0.34
0.00
0:02
0214
0.444
0,400
0.419
0.299
0.366
0.30
0.32
0,600
0.372
83.04/5
0:319
0:315
0.35
1.399-00
0.299
0.300
0:456
0
0.74
0.341
0:50
0:506
0150
1245
0398
01542
1-461
1.414-00
0.768
0.270
DS82
0.60
0.370
$5,995
0.322
0.316
0.343
0.360
0.360
0.467
0.290
0.365.00
0.50000
0.340
0.325
0.301
0.301
0243
0.482
0.414
14444
|
0.367
1.320
0.500
0.679
0.372
0.335
0.318
0.545
03365
0:39
0.867
0.281
0.356
0.341
0.296
0151
0244
0394
0.345
0,033
0.115
11336
0.266
0.300
11775
01600
0371
92.59%
0.255
0.251
0.343
0.343
0337700
0.44
0.677
0.822
0.392
0.459
0,340
0.33
0.29
0.37
0.16
0395
0.337
0,63
0,40
0.236
0,181
0,300
0.512
0.69
0.36
94.445
0.346
0:347
0:343
0:357
0.3%
0.467
0.180
0.324
0.4%%
0.357
0211
0.350
0357
0.16
0.99
0,37
0.464
0.47
0.236
0.177
0.200
0542
0.79
0.341
0.345
0:343
0.467
0.177
0.575
0.465
0.31
0.176
0390
0.337
0.238
0.250
0.50
0675
0.342
98.15%
0.340
0.256
0.343
0.343
0.337
0.467
0.177
0.359
0.477
0.357
0.88
0.37
0.176
0.370
0.37
0.250
0.176
0.250
0.51
0.69
0.340
100%
0211
0.44
0.345
0.462
0.176
0.176
0.667
0.76
0:118
0.37
0563
0.79
0.124
51

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(d) Model performance of MC-M experiment under ROC-AUC on Jannis dataset.
Dearee
LGB
XGB
CATB
PFS
DAN
MLP
NNODE
RES
SWI
CAP
NET
GO5
DCN
FTT
GBO
5/T
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
0%
0.549
0.549
0.577
0.791
0.617
0.747
0.854
0.835
0.756
0.792
0.855
0.839
0.854
0.674
0.864
0.781
0.857
0.883
0.504
0.595
0.929
0.804
1885%
0.818
0.522
0.869
0.756
0.513
0.349
0.531
0.764
0,850
0.34
0.869
0.663
0.545
0.776
0.875
0.876
0.50
05500
1000
0.80
0.122
0.54
0.700
0.007
0.89
0.355
0.526
0.17
0.749
0.770
0513
0.27
0653
0815
0.761
0.774
0.52
1990
1911
0.700
5.08
000000000000
0100
07000
1200
0,23
0.19
0.44
0.751
4811
0220
0645
0N10
0.70
0.774
03845
0.47
0270
7.41%
0.802
0700
0700
0.90
0:53
0814
0815
0.766
0.40
0.17
0200
0823
0.35
0.777
0.764
0.77
0.836
0.24
0315
0550
0223
0.299
0.78
0.023
0881
12411-00
0.00
0.56
6.70
080
0684
0.780
0.76
0.
0.001
0.89
0.99
190
11.118
0.781
1
0.70
0.78
0.002
0.2
0.224
0734-000
0.723
0,90
0786
0786
0802
0005500000
0.775
02800
0.24
1000
1851
00000000
1 13,963
1 14.91%
0.762
0.727
0,72
0.724
0.04
0724
0.700
0000
0.703
0.735
0.260
0.752
0.672
0.679
0329
0.77
0288
02810000000
0.606
0.998-00
0.536
0.535
0.722
0.700
0260
0753
0700
0205
120
0277-00
02010000
0.578
0.592
0.732
0.784
0.697
0.703
0.712
0.700
0.781
0.771
0.789
0.511
0.29
0396
0035
0.33
0.720
16 16.79
0.739
17700000
0:344
0:335
0.00
0355
0.089
0.67
0.694
0.746
1756
0364
0.725
03683
0.881
0.760
0.770
0:422
1915
0,214
18.52%
10655000
0700
0224
0.733
0662
0200
1881
0.75
0.62
0.574
0.000
0578
0573
0.75
0.547
0.656
0.619
0.641
0.712
0,081
0,477
0.592
1107
0662
20.37%
0650
0.001
0680
0630-00
0457
0.644
(960
0.725
0.647
0.570
066
0664
0660
0673
0.590
0.64
0.579
0.622
0.705
0.675
0:466
0.596
0.3
0.53
21.224 11.229100
0000
1000
0006
061
0,90
0248
0.002
0A26
0,50
0064
0340
0035
0.45
0.57
0.551
0.695
0.069
0.475
0.50
0957
0040
24.07% 0000%
0,606
0000
0000
0.26
0500
0.600
0.50
0.56
0.539
0.63
0.656
0.000
0.936
0634
27.70% 01.78990000
0.300
0.55
0/00
060
0.601
0600
0.35
0.555
0.550
0.50
0010
1200
0.688
0.688
0,500
0.528
0612
0.50
0.50
0550
0,652
0.619
0,619
0.900
0935
0.05
0.602
31 31,48%
0
0620
0000
0630
0.20
0550
0200
0400
0022
0550
0:200
0
0.0
0508
0250
0389
0500
0.15
0.655
0524
0523
0.63
0.997
0.20
0.00
0.01
0551
11.70
0620
0011
060
0520
0.35
0,50
0.997
38,90 38,59%3
81.309
20, 199
37,040
0
0623
039
0023
00000000000
0.000
02000
0:583
0:000
000
100
0130
0300
0110
0335
0:553
0:55
0.93
0.55
0395
0500
0357
0590
0923
0001
0.554
1187
0.304
0:336
0.90
00000058 0/530
0.000
0,53
0.900
0350
0570
0258
0390
0565
0559
0.56
0.07
0:57
0617
0:520
0.315
03115
0.500000
0.565
0.94
(1585
0.553
0.53
0.50
0.531
0.541
0.537
0.505
06529
01554
0.60
0002
0/611
03555
0.00
0661
0.90
0:583
1.90
035
09th
1191
0035
0.034
0.377
0.59
0.509
0.508
40. 40.748
887
0 0:988
0:330
0.965
0:559
03566
03566
1150
0.534
0.554
1::01
1180
1
03115
0.550
0.560
03544
(1:66
0:981
1915
0.574
41.59% 1..599000
44.1-%
46.99 1...0.00000
0.578
0857
13563-00
0.564
0.500
000000000
0550
1
03300
0.540
0:555
0.500
0355
63566
0332
0354
(356
13533
0333
11599
03309
0.496
0.495
0.536
0.51
0.517
0.508
0:338
0351
0550
0268
0.356
1150
0.50
19565
1150000
0.504
0.504
0313
0.545
0.566
0.541
0.554
0.533
0332
0.539
0.313
0.15
0.555
0.356
0.555
0.557
0,50
0359
01:00
0:306
0.885
0.500
0.000
10000
1350
1995
1917
0.935
11534
0.533
0.560
0.554
48.15%
0.500
0.50
0 0:56
00000
0.000
0.80
0000000000000000000000
0/000
0000
0
0.00
0:00
0200
0220
0.000
0.00
120
0.30
0.58
0.89
0.00
0.500
0.50
0:00
0-00
0-00
0.00
0400
0000
05000
03500000
01111
0111
0211
0211
0231
0300
050
0200
0.00
0.00
0.500
0.000
0.000
0400
0,00
0000
0/0
05:
0228
0.20
0.90
0.00
0:00
0
0.30
0500
1000
0935
1006
1006
0432
0.322
0.523
0.523
0523
87-480
0300
640
0311
0500
0900
00000000000
0
0:90
0:000000000
0550
0.522
1696
1111
(00)
(189
0.49
0
11:00
01:00
0.90
1990
0.22
61.11
03303
0
0:00
0501
2692
0.296
0.50
1007
1150
0300
0.696
0.487
0
01308
0.507
0.855
0.333
900
0:500
0:504
0:564
0
0:400
0.00
0.00
0000
0
0:499
3500
1501
1
0:486
0.90
1000
050
1091
14495
11510
110
1
(100
(300
0
0.499
0-497
1.499-00 0.495
0.995
(1
(1:06
0.907
0.907
0
1100
1014
1004400
0.322
0,322
0.500
0:308
0.900
01006
11:00
6:000
(350
01291
0.386
0.501
10000
0300
0291
18855
1111
0300
0.086
0.498
0.191
0 11508
11:00
0395
1107
0.322
68.33%
0.304
0.484
0.401
000050000
0
6:300
0.090
0.499
0.501
0:004
0.59
0403
0511
0.498
0.493
0,499
0.497
1
0.508
0.30
0:301
0390
12900000
0.336
0.522
0.99
0
6:500
0399
01005
0.496
0.50
0500
05000000000000
(1001
0511
0.503
0.441
0.407
0.491
01906
0.50
035
1914
0.522
71.224
0.555
0:000
0.000
02200
0.490
0.500
0.500
0300
0.544
0512
0.500
0.592
0,497
01.08
0.505
01:00
0000
1000
0934
0.00
0.500
0.000
0.689
0.00
0.499
0.405
01:07
0.503
0:308
0.066
0.522
0.500
0
0.000
0.000
C.99
0,900
0.497
0.50
0405
0.510
0.499
0.491
0.497
0.495
01:07
0.501
0:10
0.37
0.522
27.
0000
0.000
0200
0.899
0.00
0.11
0:301
0.941
0,40
0/401
0.495
0.500
0917
0:325
0.507
0.00
0.00
0.37
0.28
0.00
0.00
0.11
0500
0.335
0,498
0,495
0.505
0.23
01000
0522
81,403 31,4803
0.307
0:008
04400
0.90
0.609
0.500
0500
0687
0.500
0.00
0.00
0000000000
6407
0,994
0.11
0.500
0.488
0.499
0.005-000
11307-000
0.505
0.506
0.27
0917
0.522
8:00
0.10
0:006
0300
(188%
0.00
170
0/47
16493
01111
0000
[168
0.49%
0.995
11:00
0/305
11:00
11957
0.222
85, 85.19%
0.307
0:508
0.05
0.000000
0:00
0:299
0.00
0,685
0.500
0.500
0
0
0.495-0
0311
0311:
0,303
[1485
0,499
1.495-00
1.506-00
0.505
0.506
01933
0.935
0.523
83.041
0:100
0,502
0.684
0.50
0-406
01496
0512
0.00
01487
0.500
0.494
1.506
0.505
(1935
11985
0915
0.522
35,99%
0:500
0:355
04007
0.096
0.50
0:301
0
0.490
0.683
0.307
0.501
0.403
0.407
0.494
0.511
0,502
0.455
0.499
0.494
6.994-
1.505-00
6.507-00
[.402
1.995
0.035
0.521
0:300
0.965
03406
0.499
0.496
0.584
0.50
0.501
0.496
01495
U112
0300
01555
0.502
0.095
1
0.377
11:97
11566
1196
0.72
92.59% 0..59%000
0.305
0.306
0:004
0.497
0.498
0.499
0.500
0.499
0.503
0.503
0
0,498
0.4%
0.69
0.500
0.501
0.500
0,503
11500
0.405
0.491
0,590
0.936
0.522
94.40%
0.502
0.560
0.500
0.499
0.502
0.490
0.508
0.501
0.900
0.499
0497
0.499
0.4%8
0.50
0.501
0.501
0.982
0.495
0.982
0.566
0917
0.322
0.502
0.500
0.506
0.50
1
0.500
0.500
0.501
0.889
0501
0.501
0.000
0.501
0.500
01506
0.595
09955
0.233
98.15%
0.505
0.508
0,305
0.501
0.456
0.500
0.499
0.499
0.532
0.501
0.300
0.500
0.907
0.500
0.405
0.302
0.995
0.937
0.522
100%
0.500
0.500
0.500
0.500
0.503
0.500
0.500
0.503
0.499
0.500
0,500
0915
0.422
52

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(e)
Model performance of MC-L experiment under accuracy on Jannis dataset.
Dearee
LGB
XGB
CATB
PRS
DAN
MLP
NODD
RES
SWI
CAP
NET
GO5
DCN
FTT
GBO
SAT
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
0.680
0.682
0.755
0.629
0.657
0.560
0.693
0.665
0.635
0.635
0.994
0.678
0.685
0.530
0.705
0.638
0.636
0.731
0.725
0.560
0.582
0.850
0.662
1.85%
0.681
0,80
0.657
04400
0.500
0.665
0.635
0.655
0.529
0.706
0.638
0.730
0734
672
0.57
0851
0.61
5.08
7.41%
0681
0661
068
0715
4713
0.001
0620
1000
966
9002
0.00
000
0.561
0.001
0:561
0694
0.94
06M
0,665
0,005
0.565
0.63
0634
0.644
0.63
9.27
0.627
0.85
065
0040
0.60
0.80
0.084
0.644
0.655
0.33
0531
0.590
0.00
0.788
0.706
00.88
0.638
0687
0.67
0627
0.81
0.70
0.730
0.221
0.72
0:40
0550
0676
0.7
0653
0.61
0.64
9.369
066
0/00
0.66
0620
0:501
0.60
0.2
0.6
0600
11000-00
065
0677
0078
0.531
0.200
06
122000000
0.90
1880
067
0.49
11.118
0600
0:00
0002
0.06
0.20
050
0487
0.055
100550000
0.622
0.001
0.60
0/51
0025
0.78
0326
(.70)
0.637
0.59
0711
0.00
1339
0.75
0650
13.969
0.670
06666
0620
0:561
0.88
0965
066550000
0:633
0002
0400
0550
0,78
0338
0.701
0.637
0,666
0.720
0.714
0-400
0.79
0.649
14.91%
0.670
0671
0.712
0.622
0.656
0671
0:560
0687
0965
06655000
0:633
0.007
0/000
11000000
0552
0678
0.75
0.537
0.701
0.637
0.636
0.720
0.715
0:400-00
0.500
0.600
0.649
16.79
0601
0,722
0.13
01655
0621
0:500
0668
0965
0.6%4
0.00
1900
0675
0.58
01701
066.6
0.895
0.720
0.715
0.577
0.79
0644
18.528
0.72
0.672
0711
0633
0.606
0.71
0361
0696
0.665
0.344
0.034
0.90
0655
0579
0.78
0.54
0.701
0.636
0.635
0.720
0,715
0,380
0.552
0.75
0644
20.37%
0671
0.671
0771
0655
0.666
0671
0.560
0.87
0.665
0.63
0.037
0,00
0.54
0679
0.78
0.543
0.701
0.666
0.636
0.73m
0,714
0.360
0.584
0677
0.44
21.229
0672
0.611
0A55
0.65
0.500
0687
01665
0A34
0678
0678
0.43
0.201
0.115
0.300
0353
U679
0A4
24.0%%
0.62
0.622
0.644
0.685
000
0.561
0658
0,665
0.78
0.545
0.701
0.697
0.115
0.50
0.55
0642
0713
0655
0.65
000
0.500
0,665
0.688
0.547
0.70
27.70%
0.72
0712
0.65
000
0:500
0688
0,965
0.633
0676
0.688
0.548
0.702
0.86
0550
0679
0644
0.02
0001
0713
0.20
0.05
002
0:000
0288
0,005
0034
0.00
0941
0678
0.78
0588
0.203
0221
0.7
0647
31,4%%
0.77
1230
0712
0.657
0.655
061
0:501
0688
0.665
0.634
0,003
0600
0679
0.678
0.50
0.703
0.37
0.66
0.722
0.22
0710
0-80
0.67
0650
0.000
0600
0711
0.657
0655
062
0:501
0
0
0,665
0.64
0.60
000
0009
0629
0.78
0.553
0.705
0.63
0.221
0:111
0-200
065
0.50
20, 1
0.000
0000000000
0711
0028
0665
0002
0300
0689
0.060
0664
0.004
0/6
0080
0.69
0353
01700
0.37
0535
0.720
0/11
010
0651
37.04%
38,59%
0:67
0.671
0671
06711000
0.712
0.713
0658-000
0:656
06666000
0403
0022
0350
0:560
0
0.665
0.665
0.635
0.636
0.033
0.633
0003
0000
0948
0647
0681
0.60
0.679
0.679
0.554
0355500
0.707
0.707
0.637
0.657
0181
0687
0.720
1.720-00
0:715
0.500
0.700
0550
0.67
0.67
0.511
0.63
40.748
0.61
0713
0A68
06655
0:561
0688
0.665
0.676
0.60
0946
0681
0.699
0.556
01707
0.616
0.66
0.720
0.716
0.510
0.500
0181
0.51
12.59%
0.001
0.671
00610000
0711
0638
0.665
0,033
636
0.665
0.636
0.034
0.00
0366
0602
0.69
0351
0.706
0.636
0.635
0.721
0:115
11250
11250
0.550
0.600
0643
44,44%
0.671
06600000
0.712
0.637
0.635
0.544
03561
0690
0.665
0.636
0.629
0.03
0,47
0681
0.699
0.558
0.708
0.636
0.636
0.721
0,715
0:400
0.571
0.79
0,47
46.33
0.611
0.611
0712
0A78
01655
0667
0.561
048
0.665
0.35
0.6
0005
0343
0181
0679
0.59
0.007
0668
0.64
0:31
0.116
0:450
0.53
0669
0448
48.15%
0.637
0669
0.665
0245
0643
0.79
0.709
0.50
$1,89%
0.700
0.622
0.660
0701
0.68
0.657
0,685
0.656
0.350
0.300
0,665
0.665
0.637
0/06
0245
0640
0662
0,70
0.70
0.56)
0.562
0.10
0.10
0,634
0,33
0684
0240
0.55
0676
0.69
8.066
0/00
0000
0025
0.00
1000
0:00
01561
0612
0.00
0,00
0001
0367
0.200
00:00
0600
0000
1290
107
000
35.5
0.600
0000
0.35
0.00
0500
0.661
0.632
0/00
1300
0.688
056
0.05
0620
0.70
00000000000 0000
067
0
$7.44
0654
018)
0.6
0350
0.661
0.628
0.60
0,60
000
0110
0657
0.660
0.551
0
0.67
0623
052
0530
067
0.33
61.18
29,260 2622
1.00
1:40
0.73
1000
0:635
1007
000
0000
000
0.558
0
1000
0.000
0.634
0.0000000
0,000
0.566
1000
0000
0000
0647
1089
0,455
0.45500
0.550
0388
0.622
11000-000
0.18
0.00
0.647
1000000000
0372
0.75
101
0632
0618
0001
11500-0
0.001
1164
0342
0639
0.69
0004
11:00
1699
0.59
60075 0
0.07
0:606
0.400
0.00
0130
0.25
1000000000
0391
03000
0.005
060
0
13595
0559
0663
11034
06855000
0.599
0.96
0.602
0:013
1666
U615
0
05574
0572
0515
11599
0.01
0.538
0338
0.24
0615
0.66
0A00
0.595
(1613
0.966
1.177
(1:40
11177
0550
0875
1015
0.75
0350
0.598
68 68.33%
0604
00000-000
0:02
6357
0.001
0664
03359
0354
0.63
0:395
0.613
0365
0613
0.997
0.536
0612
0.500
0.994
0.596
0,592
0.400
0.555
0675
0.591
N0.1%
0000
0m0
0350
0000
0664
0.350
0581
0.844
0.594
0.511
0.50
0661
0,59
0612
0.507
0.591
0.902
0.591
0.100
0.544
0679
0.579
0612
0000
01001
0615
0.500
0382
0.634
0.5c
0.513
0200
0613
0.555
0353
0.604
0.595
0.599
0.56
0.79
0.500
740000
0.600
0.50
0.00
0/60
0.55%
0.57
0.62
0.500
0.514
0351
1500
0.520
0.600
0.501
0.574
0,559
0.57
0.572
0.500
0.500
0.90
0.50
0613
0.358
0.513
0500
0.564
0,534
0.587
0.363
0.50
01365
0.542
0.500
0.76
0.80
0.500
0.00
0:500
0.55
0.805
0.566
0.50
0.561
0.32
0.57
0.566
0.59
0970
0.69
0:595
81, 81,483
79.000 0
0
0.563
0:580
0:58
0.59
0.9
0:25
0:306
0.50
0.5%6
0.233
0.77
0.558
0.50
0.477
0549
0.601
0.999
0.7%
0.578
0.00
0.09
0.66
0.50
0.500
0.03
0240
0343
0335
0576
0.64
0,555
0524
0
0573
0.73
0.7
0377
0.576
0.75
0.511
11560-00
0.59
0.538
0.174
0.174
0573
0.75
0.75
0.60
0.59
0.559
81. 81.309
01112
0.300
0315
01285
0.99
0558
1200
1000100
0329
0139
1.000-00
0.450
01142
U.79
1.099
0332
85.19%
0.222
0.912
0.500
0.555
0.53
0.537
0.559
0310
03100000
0,587
0.570
0.585
0
0
0555
0.01
0:010000
0321
0.557
0.564
0.565
1
0.456
0.300
0.77
0.8]
0.37
37.04/5
0-450
0.500
01570
0:547
0.552
0.488
0.575
0.564
0.583
0-644
1554
0157
0.34
0.44
0.561
0.60
0.44
0.250
D.885
0682
0.526
85.999
0-227
0.436
0:434
0.497
0.509
1.400-0
0.530
0.445
0,51
0.524
0.515
0.466
0/10
0485
0412
0.500
0.407
0.522
0.11
0.425
0.364
0.270
0.579
0.69
0,47
MOTOR
0.392
0.393
01424
0.444
0.499
01455
0:51
0410
0.994
0.500
0.511
0.347
0450
1339
0305
01556
0.500
0.01
11399
0,345
02500
0156
0.456
92.59%
0.379
0.378
0.414
0.476
0.488
0.411
0.541
0405
0.477
0.511
0,502
0.426
0.353
0457
0361
0.497
0.46
0.444
0,990
0.39
0,341
0.250
0.579
0.677
0.47
94.445
0.314
0.377
0.356
0.443
0462
0.46
0.530
0.393
0.452
0,42
0.463
0.499
0.117
0423
0.311
0462
0.26
0.455
0.455
0.285
0.341
0.250
0.54
067%
0.430
0.374
0.361
0.259
0.44
0.64
0.326
0.36
0.414
0.455
0419
0.266
0.37
0.385
0.446
0.266
0.250
01600
0.400
98.15%
0.460
0.464
0.220
0.343
0.337
0.444
0.467
0
0.447
0.4%
0.392
0.299
0.193
0.387
0.33
0.313
0.218
0.176
0.250
0.500
0.67
0.353
0211
0.443
0.345
0.462
0.176
0.176
0.677
0.166
0:118
0.317
0563
0.79
0.124
53

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(I) Model performance of MC-L experiment under ROC-AUC on Jannis dataset.
Dearee
LGB
XGB
CATB
PFS
DAN
MLP
NODE
RES
SWI
CAP
NET
GO5
DCN
FTT
GBO
5/T
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
1.85%
1.7%
0%
-
0.540
0.349
0.849
0.549
0.877
0.390
0.517 0.617-00
0.615
0,233
0333
0.747
0355
0.5%1
0.854
0585
0,811
0.835
1335
0.556
0.756
0.756
0.701
0.792
0.855
0.520
0.540
0.839
0.540
0.84
0.854
0.675
0.674
0.865
0.865
0.864
0.781
0.781
1.761-00
0.70
0.857
0.857
11887-00
0.883
0.583
0.881
0.488
0.59
0.923 0931
0.929
0
0.800
0,804
5500
0940
082
0.200
0396
0200
0677
0.761
0880
1.4 7.41%
0388
0844
0300
0886
0802
0999
0388
085
0.78
0.00
6700
0.30
083
1236--000
0.20
0.22
088
0823
0678
066
0.665
0.60
0.90
0.781
0.877
0880
0.88
1500
0
0300
0300
13,963
11.118
0.443
0.443
0.704
0225
0,810
0.936
0832
0788
0.768
0351
0351
0720
0,703
0325
0525
0888
0683
0681
0160
0960
0.70
0.761
02800
0337
0678-000
03300
0900
185)
100
0380
1 14.31%
0.544
0.444
1633
0.706
0.996
0:32
0.747
0351
0335
0.70
6.25
0348
0681
0.60
0.783
0.703
0:511
01578
0035
0
1 16.699
0.544
6.90
01356
1351100
10855
0.70
0335
11815
0860
0:763
0.714
0035
11:00
1015
0.301
20.37%
1 18.52%
36.37
0.544
0.344
138440000
0.040
0:041
0:341
0053
6796
0000
03516
0.166
0.33
03135
6347
6:44
G351
0335
0335
0335
0835
0.768
6.7
023
0:35
0525
0339
0359
0348
0635
0.666
0961
0.622
0.762
0.762
0.794
0.711
0331
0:31
0375
0:405
0:311
0.5
0395
0.34
11555
0
0
24.07 00008000000
22 21.129
0.555
0.16
03335
07766
2006
1084-000
0x35
0
0
0,68
0466
0.862
0.62
0.763
0.74
0.59
1945
0035000
27.78'
25.099
0.25
1000
0.000
2000
0340
0.000
0.50
0600
0.661
0.861
1.63.
0.703
02411
100
0.804
31
25 29.000
0
0:000
0:200
0383
018
11200
0706
0.00
1
0.00
0.66
1855
0224
0220
1290
0280
0880
0690
0600
0963
0963
0.663
1291
0281
02000
0190
0000
0.00
0
0300
20.10 1919
0300
0.768
07789
0350
0353
0135
02200
0.200
0.70
0.70
8:56
1340
0:23
(199
0285
060
0.99
0.66
0160
0.78
0.764
0.805
0300
40.740
38,498
0.346
0:966
0:545-
6.748
0.768
C355
0335
0085
0.271
0.770
0.70
6.704
18556
8336
0:35
0
11815
0385
0491
0.93
0187
0.867
1
0.764
0.764
0.700
11:00
11000
11015
0.300
0.304
41.59% 42,599
44.10%
0.566
0.366
0.046
0.046
0053
0.617
03517
0:36
6708
6.46
0355
0853
0333
0335
0335500
0.312
0
6.7M
0.708
0335
0237
0
00000003 0.35
0325
0344
0848
0655
0.966
0167
0.666
0:764
0:764
0.704
0.997
0:81
11311
01375
11:09
0:40
10355000
0300
0.300
46.93
3094
18.15%
0.546
0.466
0.940
0.900
0:946
0
0.500
0.500
6748
0.888
0:22
0.222
0.991
0.00
0889
0885-00
0
0335
0335
0,35
0.25
1
0700
0200
0.88
0.801
0.88
0.665
0.700
0:364
0.704
0.744
0281
055
0.00
0.00
0.600
61.18
355
0:800
0
0:50
0
0990
1130
0220
0000
00300
1000
0.006
0.008
2000
6.740
03801
1282
C323
1300
0523
0877
(1815
01815
1001
01200
0.700
0.76
0.777
0.753
0.00
0.700
0.904
0,707
0.061
18:00
18:14
1246
0.77
0.784
02700
8308
1800
1199
0110
(1501
0280
1800
0256
0825
11003
0690
0600
0663
000
0687
BB0
0.704
0.277
0.70
1220
0.755
0.766
01761
0.770
0.77
0.10
0.7
0:38
(1810
(1821
0,53
0500
0,70
0.00
0109
0.000:000
1000
1500
0990
1000
1100
1000
190
1000
1001
1001
0200
0300
0776
0700
0.369
0774
68.33%
0.300
0.000
0.90
0.00
0300
0:301
0330
0:30
0300
0356
0365
0707
0.00
0
0.70
0.700
0.730
6700
6:440
6340
0.790
0.75
0.20
0
00014
0.785
00033
10014
0.220
0751
0.748
0.44
0.30
0.500
0.000
0.000
0.000
0.701
0.762
05004
19997
0300
0356
0296
177
0.5
055
0.66
0.55
1794
0.51
0
0.60
0487
0.655
0685
0.72
0.792
0.787
0.756
0.755
0.752
0:35%
0,740
0:755
1.753-00
0.
0.761
0.756
0,755
0.969
0,760
0.701
0.779
0.764
0.752
01766
0.792
11779
0.75
0.780
0.787
0.400
01500
0:302
0.000
0:455
0.50
0396
07655
1015
1161
1055
0
00150000000
0.56
0760
0.00
0.40
0.25
0.781
0.00
0.60
0481
0.75
0.746
0.71
0.700
0,799
0.50
0740
0.20
07200
0243
0.775
0.755
0.70
0726
0745
0.59
0681
0.75
0,745
0.700
0.748
0.500
0741
0.722
0.00000000
0.200
0763
0.40
0.400
0.777
0.76
0.777
0:51
0742
0760
0.257
0683
0.754
0.47
6.449
0.75
0.758
0.748
0.506
06th
0
0.47
85,198 15,198
01255
0.000
0.755
0.755
0755
0:20
0221
0.700
0.751
0
-
02115
0.11
0.755
0237
0.700
0.001
0
0.727
0218
1711
0169
0.20
0228
02200000
0677
0.680
(1235
0.119
0.734
0.744
0.7.9
0.54]
0.716
0.714
0:111
0.711
0-88
0835500
0727
0.76
0
07400
1714
0733
0.766
0.788
0
0
0.743
11743-00
0.758
0665
0137
0.05
0205500
0.70
1.707-00
0.736
0.73.
0.736
0.122
0.7113
11981
1916
0.18
88.55 85.995
0.655
0.606
0.000
11000-00000
0:686
0,996
0668
0.722
0
0.654
0.695
0.711
0.76
0.55
0603
0.61
166110
0.59
1.696-0
0.69
0.999
0.67
0.671
0.664
01:05
(1599
1035
0.63
MOTOR
0.05
0107
0668
0.64
0665
0724
0665
0.683
BANS
0.701
0.597
0169
0620
0645
0.49
0A68
0.09
0.11
11502
1
11955
DANI
92.59%
0.619
0.629
0.63
0.609
0.662
0.655
11645000
6.21
0.64
0.677
0.677
0,692
0.599
0663
0610
0.32
0.645
0.644
0.675
0667
0.616
061650
0.50
0.302
1:544
0.925
0651
94,445 94.445
0.607
0.4
0.48
0.67
0.300
0,57
0.671
0.609
0.56
0600
0.577
0625
0.977
0643
0636
0.644
0.61
0.907
0.581
0.9
0914
0635
100900000000
0622
1
0.59
0.634
0.35
0.56
0406
0.623
0668
0.64
0.66
11:02
0.596
0995
0625
98.15% 00
0.596
0.607
0.635
0.630
0619000
0.657
0.569
0.63
0.646
0.530
0.565
0.635
0.356
0.540
1:302-00
0.995
0.934
0.15
100%
0:000
0.500
0.495
0.500
0.500
0.500
0.503
0.500
0.590
0.50
0.500
0.003
0.49
1.503-00
0.499
0.500
1.402-10
0.50
0915
0.522
54

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(g) Model performance of RD experiment under accuracy on Jannis dataset.
Dearee
LGB
XGB
CATB
PRN
DAN
MLP
NODE
RES
SW1
CAP
NET
GO5
DCN
FTT
GBO
S/T
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
1.85%
060
0.680
0.600
0.681
0.66
0700
0.755
0630
0.629
0.655
0.560
0.500
0.560
0.677
0691
0.693
0.665
0.665
0.665
0.631
0.635
0.61
0.534
0.623
0.625
0.994
0.66
0.678
0.83
0.655
0.611
0.528
0.526
0.530
0.704
0.702
0.705
0.637
0.638
0.64
0.636
0.76
0.731
0.720
0.560
0.582
0679
0.675
0.500
0.662
0.677
0.566
5.08
7.41%
000000000000
0001
0657-0000
0000
0700
0620
0000
900
0000
0.55
0:556
0:00
0674
0679
0.689
0,063
0629
0.63
0.20
0.509
0643
0648
0654
0670
0663
0.7
0.70
0526
0.21
0.0%
0.666
0.655
0770
000
060
192
0581
0.2
0651
0644
11.118
0663
0647
0000
0000
100
0000
0000
0000
0.600
0500
0:501
0680
0.600:000
0.664
0000
0.627
0.60
0.67
0.619
0.60
0.64
0670
0670
0065
0078
0671
0.24
0325
0682
[168)
0633
0.30
0000
15000000
0.70
0.001
0.88
061
0.554
1554
062
0.75
D/48
0645
13,963
0:635
0005
0:02
069
0:585
03665
06655000
0.621
0.507
0.67
0.42
0666
0664
0.50
0.77
0.62
0623
0:901
0.79
0.640
14.11%
0.688
0660
0076
000
0644
0350
0:558
06655000
0.622
0.513
0.667
0.69
0.68
0.17
0.685
0.636
0.22
01681
0.38
1550
0.600
0.636
16667
0.666
0608
0.0
1
0
01628
0:555
06665
0.66
0.50)
DA17
0660
1617
0.11
0.644
0669
0.50
0.590
D.8S
067%
0.76
18.52%
0.66
0:634
0.07
0608
0.009
0.68
0.55
0856
0.055
0.6
0,510
0.655
0429
0662
0653
0.15
0.662
0.27
0,025
0665
0,062
0.586
0.552
0
0630
0623
06638
0601
0,807
0.225
00255000
0.34
0.553
0643
0.643
0.13
0.595
0.40
0,17
0.41
0641
0.513
0.664
0612
0.614
0.666
0.642
0.374
0.592
0679
0
24.00%
21.229
0612
0,62
0.00
0.00
0936
0.87
11677000
0000
0006
0666
0.356
0.46
0.644
01699
0,60
0615
0.595
0661
006
0.509
0.14
0.65
0.662
0A14
0.64
0,615
0.6
0.542
0.559
0.54
0.52
0.79
U679
0616
0A25
0.000
0966
0.55%
0.66
0.600
0.638
0,510
0.033
0.512
0670
37.78%
0.503
0.000
0.000
0:530
066
0.644
0.66
0.666
0.521
0.51
0.647
0.330
0.50
0676
0614
1000
00000
0000
6007
0:200
064
0,01
0010
000
0626
0.016
0510
0010
0.64
0.7
0612
38,59% 0
31, 31,48%
81.300
30, 198
37,040
0.
065
0400
0:567
0003
100
0000
100
0200
00000
0000
0.50
1000
0601
0.18
0:556
0.506
0:553
0:344
0:51
0
0523
0
0010
0813
0.62
0.637
00200000
0.002
0.557
0.34
0:620
0.500
0.50
0.899
0.537-00
0
0
000
0,06
0.99
0620
1003-00
0010
0.47
0.50
0577
0396
0624
6624
0505
0614
1
0.59
0,500
0,70
0.645
0631
05500000
0.111
0516
0308
0489
03302
0617
0.25
[100)
0.665
0.41
0.533
0.00
0.001
0A10
6917
6:01
0,585
0099
0.985
0.60
0612
(1597
1.555-00
0:575
0645
0.552
0
0.441
0-174
055
0550
0
0
11000
0678
0.67
0.69
0.000
0.699
0613
0.576
0.00
0.00
0.594
40.70 40.748
0:966
01600
00000-00
0666
0.344
0.11
0.699
0.505
0.531
0..00
0614
0618
0498
0.00
0.507
0,00
0/613
10:00:00
0.500
15500
0.592
41.59% 1..59%
0.509
0:334
0.503
0.008
0.644
0.550
0812
0.632
0.00
0.576
0.020
0
0668
0.49
0.555
0.00
0.001
1.606
0.595
0.519
0.511
1.79
0.56
44.44%
0.540
0.590
0.369
0.554
0.369
0.579
0.335
0.573
0.607
0.378
0.554
0.513
0.342
0.585
0513
0.80
0.581
0,513
0.581
0.344
0.547
0.454
0352
0.79
0.565
46.93
0.515
0.900
0.555
0.886
0.342
0.778
01617
0.580
0.58
0.580
0.55
(191
0554
0.491
0.677
0.588
0.58
0.560
0,53
0.45%
0.43
1669
0.504
51 11,850
48.15%
0.546
0.500
0.566
0:359
0.553
0.530
0.577
0.599
0,500
0550
0.55
0.505
0:565
0.529
0.545
0.348
0,58
052
0.55
0.627
0.009
0.5%4
0.354
0.556
0.000
0.501
0.500
15300
0.255
0,546
0480
0,400
0.900
0.40
0.500
0.50
0.901
0.356
0.006
0.581
0504
0555
0555
007
0675
0.345
0.59
0.569
1:200
0.232
0344
01629
0.577
0.00
0.00
0217
0227
0.90
0.00
0.10
101
0455
35,716
0236
0550
0559
6527
0.005
0554
1150
9598
0250
0486
0553
0271
0505
0233
0.00
0
0.722
0.59
0,804
0.581
0.500
035
0:333
0.98
0.90
8-700
0336
0850
0.547
222
1:200
0.22
0
0331
0.59
0.55
1202
1350
0d89
0.967
0353
03200
0:48
118
0354
61.11
0-822
0:500
0.00
0330
0.544
0
0:529
0340
0.587
0.568
03:33
1500
0306
0463
0.455
0.70
0552
0
0.211
0:441
0-50
0
0.49
0.327
055
0555
0:500
0582
1001
03377
0.156
0118
1552
1872
0472
1667-00
(1:77
01899
0.33
0:46
1981
0.500
0.566
0:531
0.8K
0:557
0.502
0.50
0.536
0456
05537
1186
0470
0.902
0.564
05587
(1:45
0.400
0:448
0.515
0.699
0.502
68:33 68.33
0.466
0.070
0:403
635
6500
0241
0474
0:545
0:57
0372
0:329
1351
0342
0.554
01514
03554
0.318
035
0.479
03311
0:482
0111
1
0411
0258
0456
0474
0.561
0.499
0.515
0.556
0.547
0.569
0:455
11:155
0.386
0.655
0:428
0.315
0.316
0.681
0
0.479
0511
10.15
0.477
0.46
0:511
0338
0332
0467
0.523
0.556
0.55
01514
0458
0551
0451
0.470
0.440
0.550
0.549
0:471
0.464
0411
0355
067%
0.492
0.465
0.48
0.888
0.055
0:31
0.57
0461
0.573
0.55%
13506
0,52
0.4%
0.462
0.540
0.441
0.414
0.254
0.315
0.600
0.490
0.220
0.000
0415
0.602
0314
0.475
0.00
0.400
0.50
0475
0,562
0.576
0.559
0.356
0.00
0:00
0521
0.70
0.450
0,473
0.44
0.456
0.341
0.534
0.599
0.538
0.353
0.437
0.451
0.448
0.445
0316
0.16
0.65
0.478
0.401
1000
0.10
0.43
0:300
0
0815
0.500
0410
0.64
0200
0345
0.4%
0.403
0.400
01462
0:117
0.77
0444
79,000
0445
0.50
0453
0412
0.537
0.522
0.50
0505
1467
0401
0205
0.56
0.46
0.405
0317
0.525
0.315
0.399
0/422
0315
0.675
0454
81 81,483
0.379
0.36
0:63
0413-000
0464
13100000000
0.49
0.44
0.552
0.541
0444
0.455
0400
0517
0.809
0450
0.384
0.599
1.542-00
0.383
0.395
1.352-00
0316
0.69
0455
N1. 81.308
033
0422
1100000000
0.425
0.30
[1561
0.26
0.34
0.449
66626
0336
0400
0112
0300
1.999000
0.337
0.30
1.905-00
U.16
1183
0423
85.19%
0.405
0,465
0374
0448-000
0.997
0.499
0.350
0,505
0.144
0,54
0.391
0.991
0.329
0433
0250
0250-00
0432
0.429
0,491
1.509-00
0.316
0.200
1
D336
0,600
0420
83.04/ 00004/000000
0.500
0.415
0.469
0.354
0.65k
0-81
0.304
0.115
8424
0203
0417
03885
0.482
0,221
1.854
D536
0682
0.396
85.999
0.366
0:400
0.502
0.451
0.486
0.358
0.483
0.539
0.430
0.405
0.337
04445
0.03
0419
0.562
0.408
0.501
(.501100
0.305
0.324
0.337
0.515
0.87
0411
MOTOR
0.542
0.741
0355
03M
0.377
01456
0.484
0.294
0.883
0.506
0.600
0.373
0.315
04499
0/21
0433
0.315
0.500
0.00
01255
0.264
1:33
0.516
0.99
0396
92.59%
0.310
0,346
0.37
0.341
0.313
0.677
0.239
0.417
0.452
0.376
0.291
0.244
0.30
0218
0.997
0,345
0.400
0,400
0.265
0.256
0.347
0.315
0.69
0361
94.445
0.903
0:391
01500
0.355
0.340
0:311
0.469
0.205
0.417
0.6
0.383
0.345
0.314
0.311
0191
0.374
0.358
0.447
0421
0.271
0.211
0.353
0315
0678
0.353
0.254
0.2M4
0.500
0.355
0.897
0.410
0.477
0U537
0
0.416
0211
0.388
0.365
0,466
0,312
0.344
0.370
98.15%
0.257
0.258
0.33
0.351
0.357
0,472
0.204
0.429
0.42
0.406
0.339
0.62
0.361
0.41
0.252
0.276
0.431
0.91
0.60
0.364
0211
0.44
0.343
0/462
0.176
0.176
0.677
0.76
0:118
0.37
0563
0.79
0.124
55

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(h) Model performance of RD experiment under ROC-AUC on Jannis dataset.
Dearee
LGB
XGB
CATB
PFN
DAN
MLP
NODD
RES
SW1
CAP
NET
GO5
DCN
FTT
GBO
5/T
SNN
TABR
NCA
LMA
TLLM
UNI
AVE
0%
0.849
0.549
0.877
0.791
0.817-0
0.747
0.854
0.835
0.756
0.792
0.855
0.839
0.854
0.674
0.864
0.781
0.857
0.583
0.504
0.595
0.929
0.804
1.7
1885%
-
0.546
0.540
0.50
0.900
0.000
0.55
0.56-00
0330
0.447
088
0,835
0.755
0.755
1.555-00
0.790
0.82
0.833
0.36
0.51
0.53
0.633
0.69
0.863
0.863
0.779
0.760
0.70
0.855
0.878
0,85
0.883
0005
1915
0 0
0.800
7.4 7.41%
13,963 2.968-000
11.118
0:35
0877
0.87
0.21
0.030
0122
0999
0380
0.70
0.881
0.200
0770
0.704
03853
C383
0822
083
0837
0200
0778
0.72
0:735
0.10
0.796
6.000
0.706
0.727
6761
0.54
0.96
0346
0841
0.22
0.17
6319
05118-000
0529
0221
084
08t
0341
0347
0339
0664
0461
0.63
0668
0661
0359
01455
0.845
0846
0.48
0.72
0.7
0.760
0.76
0.73
0.778
0.78
0.
0.79
0.
0.781
0.785
0.784
0,86
0.873
11877
0.83
0213
0520
0.861
0.55
0.66
0.801
192
0611
1911
183)
1035
0.70
0:000
0300
0
1 14.91%
0810000
0.72
0,086
0.785
C333
01225
0.729
0,700
0336
0215
0521
0339
0.58
0.845
0.770
0.781
1.859
0.59
0.38
1035
0.702
1 16.67
0.855
6:241
0187
0
0.358
18804
0.506
0814
0.46
0.76
0.761
0.775
0.856
0.815
1915
0.764
1 18.52%
26.37%
0.907
0311
0:31
0:119
0350
6994
0755
0301
0317
0.055
6704
6:44
G310
0
0815
0325
0355
0.751
0.77
0.766
0X33
0223
0.000
0,300
0812
0322
0812
0821
0654
0.53
0.33
0.82
0.753
0.771
0.775
0.783
0.855
0.812
0,411
0,626
0.355
0.3
0.996
0395
1396
1
0936
1924
0.792
0.799
21.229 00022900000
0359
0312
0300
0330
0.50
0815
0.767
03417
0.58
0396
0915
0785
25 25.029
27.780
24,07%
0
0
0.255
02530
0.000
0000
0.20
1200
0811
0.200
0.000
0200
0810
1610000
0,50
0.60
0.20
0.61
0657
0.642
0.795
0.800
0.60
0.70
0.340
0:781
0.83
0,14
0
0.50
0.90
0015
0015
0.375
0.66
0
25 25.00
0.200
1200
0214
(280
0220
0669
07003
0.723
31
0200
0.30
0.00
0200
0.723
0306
0823
0699
0.80
0.76
0.775
0220
0530
0241
0050
0.76
81.80 11.89
013
0.70
0300
0.80
080
086
0.66
0.8
0.473
0.99
0.76
0.795
0.368
20, 198
02000
07000
0.791
07288
0295
1881
0722
0.223
00080000
0281
0113
0643
(1776
0.768
0.90
0.00
0.807
0
3 38,99%
37,040 -
0.761
07769
0.70
0.766
0
0
0100
0760
0.733
0.700
0.770
0.733
0.785
0
0
0377
01815
0293
0.745
0.721
0.711
0.70
0.700
0.785
0.785
0.700
0.67
0.245-00
0790
0788
0.25
0293
0.61
0.65
0.782
0.80
0.751
0.751
1.732-00
0.765
0.746
0.922
0.750
0:778
0.702
0:339
0390
1036
1035
0.74
0.761
0.444
40.105 0
0:771
01077
0.988
0.700
0.766
0.300
0
0.736
1.704
0.688
0.29
0.600
0.767:00
0.746
0.000
00:00
11:06
11:00
0.355
44.4-%
41.59%
0.751
0.738
03745
0:790
0:590
02745
0
0700
0.00
6:355
0.750
0
0751
0:557
0:788
6736
6334
0351
0399
0.785
01905
0,244
0.725
0.769
0.727
0.788
0.766
0.336
0349
0.91
0766
0.666
0794
0630
0.15
0.751
0.768
0.788
0.728
0748 0.705
0.742
0.70500
0.762
0.787
0.733-0
0.519
0.454
1
11330
1936
0935
0754
6.356
46.33 10
0:947
0.766
0766
01590
0.2
0.751
0.711
0.44
0191
0735
0421
0.744
0.727
0.71
0.758
0.755
0.45%
1091
1015
48.15%
0.766
0:751
0.025
0.75%
0.760
0.769
0.700
0.783
0.708
0.729
0.734
0.740
0.731
0.770
0225
0230
0200
021
0763
0614
0.53
0.755
0.75
0,001
0.252
0.720
0.62
0.706
0200
0231
0.74
0611
0.768
0.00
100
35,116
0.221
0.20
0.007
0220
0220
0752
0.001
0770
0220
0220
0234
0.786
0.225
0.000
0,007
0731
0201
4223
0244
0240
0616
060
0.23
0.209
0710
0720
0.44
0.70
0.70
00000
0500
0718
0722
0.725
2222
0250
0220
0.708
0355
0220
023
0
0.77
0.29
0227
0.79
0700
0220
1227
0
0621
0600
0.64
0040
0.001
0.700
0333
0-70
0600
100
00000000
0.23
0213
61.11
0.709
0.717-100
0.228
0.740
6724
0334
0025
0.736
0
0.700
0.700..8
0.725
0960
0700
0229
0.592
0.001
0.337
0.726
0.685
0
0830
0.707
0:00
0.000000
02222
60200
00111000
1025
0.220
12200000
0.72
0000
1122
0395
16655-00
0.720
11766
6.707
199
0.305
0.00
00000
0.715
6:733
0.500
0.00
0.706
0 0.687
0.722
0000000000
0119-000
0213
0.595
0.95-00
1706-000
0663
0.717
0.177
0.677
00:40
0:446
0.63
0.001
03112
0330
0
01331
0
6006
1200
1733
0.315
0:00
0.720
006
0716
0405
0.05
0.701
0.729
(1655
0,000
01066
0995
0497
68:33%
0882
0:00
0723
6334
01140000
0.705
0.64
6304
0650
0734
0,622
0.086
0668
0341
0665
0660
0.53
0.64
0.679
0.712
0.658
0,648
0:428
0.496
0.936
0.675
10.15
0691
0960
6765
6665
0.3
6315
0291
0751
0.31
0.901
0456
0111
0315
0.586
0.655
0.600
0.333
01694
0.884
0418
0.406
0915
0A65
0681
0/001
0714
0/66
6338
0.54
0.501
0,00
0006
0.96
0.500
0.655
0.60
0.715
0.691
0.050
0.451
0.296
0.936
0,60
0.60
0,000
0650
0.00
0.66
0005-000
0200
0.60
0.60
0581
0.757
0.745
0.644
0.506
0.50
0,007
0.00
0650
0655
0.099
0.09
0.666
0.67
0.588
0.567
0.645
0.665
0.699
0,670
0710
0,680
0.655
0.670
0,682
0.44
0.448
0.445
0.26
0.935
0.935
0.699
0,677
0.00
0.20
0.21
0016
0406
0419
0.66%
0678
0.50
0619
0627
0.600
0.540
0.63
0.699
0.595
0-11
0400
0915
0644
79.000
0634
0600
0671
0.63
0.50
0666
0614
0,G1T
0,77
0.778
0620
0.47
0.645
0564
0.611
0.544
0,044
0.601
0.005
0/422
0.26
0.955
0643
81, 81,483
0
0.656
0.689
0.664
0.600
0.722
0.688
0.683
0.3
0600
0.69
0.670
0.569
0.608
0.670
11996-000
1
0.44
0.35
0.352
0497
0.955
0.51
81. 81.309
0024
0431
D.67
0641
1
[1061
0025
0.89
0.99
06696
0.26
0.24
0.72
1000
0.99
8887
01900
DA22
85.19%
0.627
0647
0.673-000
0.660
0,24
0.677
0.598
0.669
0.64P
0.615
0.007
0.59
0.594
0.221
0.633
0.552
0.658
0.635
1.567-00
0.567
0.391
0.696
0.02
0.622
87.04/ 000048000000
0.997
0635
0.625
0.33
0,600
[.611
0.589
0.563
0.566
0621
0.60
0.500
(1963
0.592
0600
0.601
1.854
0.99
0.00
8..99
0.606
0.6000000
0.619
0.645
0.663
0.603
0.677
0.638
0.500
0662
0.566
0.566
0617
0661
0.535
0.574
0.614
1.599-0
0.500
1.337
0.455
0.009
0.609
0.566
0.006
0
0.65
11600
0A62
0.677
[16bb
0.66
0.514
0.99
0.55
0355
0345
0.79
0131
0.64
11516
0,23
1.33
0.996
01955
0.39
92.59%
0.531
0.584
0.606
0.545
0.557
0.553
0.569
0.523
0,511
0.531
0.522
0.59
0.51
0.509
0.551
0.535
0,549
1.557
0.533
0.347
0.496
0.936
0.556
94.40%
0.562
1.564-000
0.54
0.500
0.50
1
0.554
0.111
0.565
0.521
0.523
0.955
0,518
0.19
0.37
0.51
0512
0.409
0.500
0.514
0.515
0.353
0.466
0915
0.542
0.554
0.52
0.590
0.00
0
0638
0661
0.19
0.50
0.59
0.544
0.57
0.513
0,53
0.094
09955
0.575
98.15%
0.530
0.530
0.41
0.337
0.501
0.394
0,612
0.574
0.604
0.355
0.593
0.582
0.583
0.523
0.525
0.545
0.665
0.350
0.536
0.494
0.935
0.571
100%
0.500
0.500
0.500
0.500
0.495
0.500
0.500
0.503
0.500
0.500
0.000
0.499
0.500
0.500
0915
0.522
56

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Model performance gap A of RD experiment under accuracy on Jannis dataset.
Begro
XGB
CATE
IPN
DAN
NODE
CAP
NET
1208
INT
DCN
PTT
(1R0
SAT
SNN
TIF
TABR
NCA
1.MA
11
UNI
AVE
188%
-0001
-3003
-4/013
-1094
-4.002
-0.50
-0/003
-00035
-3033
-4:30
40009
3.%
0.08
0,008
0011
6201
0:055
0.008
1/036
0/915
0.006
0.005
0,005
0.538
0,000
0013
0.10
3031
0.201
0,09
5569
6044
-2035
-001
4.05
2.008
-0012
-0.001
4.030
-2005
-0.005
-0000
-0017
40301
-0013
-0010
40007
-0011
-0.00
-0556
-0033
20031
0.214
-2026
.0222
-001T
7414
0/224
0.18
0,000
4009
-00035
0/025
-0011
4.015
4.007
40000
-0023
-0121
40006
-108
40:00
-0018
-0028
-003
012
2002
-6201
-0/25
936
0,025
0,03
0.011
0.006
0000
0.002
0.007
20003
0.001
0,014
0.055
0.022
2011
4,10
0,003
0.033
0.007
0.506
00035
22050
0.121
0002
11.115
20046
-2041
23082
41.04
-1015
40019
-1.011
4.026
-30000
-2.005
-1012
-1025
40005
.0019
-5/20
-01008
-0.85
-0.012
-0010
-0015
0.563
000
-0210
-0/05
12.96%
4.06T
41023
4.012
.0005
-0030
40007
4,039
-3.07
41.01
4.062
-0.39
-0025
-2033
.0/29
-0017
-0.400
-0.00
.0621
.0066
.2.947
0.171
-3655
-0201
-0634
14.1%
-0070
-0051
41000
40611
-0037
4.001
40018
-0011
4,006
41001
-0088
40014
-3021
-0114
-0/033
-0103
40.00
40018
40052
-00051
0.01
-004
4020
40019
36.57%
03005
0.073
0.125
0.090
0001
0,200
0.054
15.925
4005
4091
4008
-0051
41.05
-0036
4001
-006
-1.17
-0018
-.111
-0057
0.06
-5001
4.300
-0046
(100
0155
0000
0051
0.12
-0171
9161
0181
0.06
0087
60064
0101
0.899
0,000
4604
0,162
-0114
01025
0000
01201
4069
21.22%
4,071
-2966
.0047
4.008
4066
00026
0,030
4.04
0.067
00068
2002
40054
0.129
-0.80
-.397
.0955
0006
-0.03
-00022
3.04
0.300
00059
24.0%
0.115
0.006
0.682
0.046
B056
0.013
0.089
2/14
0,035
0.853
0.066
0.64
D056
0075
0,000
0.002
0,042
0633
8089
0,119
0,022
00000
0.301
0,700
35.0P5
A.145
22133
-2 182
-8068
-0081
41004
20066
.0016
2,040
21106
-1
40368
-0056
.0081
-0088
-0.16
-0.06
-6650
-0/066
-0.138
-0.195
0.011
-0.01
-0/06
11.78%
0.128
4X118
4000
40066
-0034
40066
6115
4001
-0.16
4.057
00038
0.201
-0,107
-0403
-0034
-0.00
-0072
29
0.113
4.108
0.105
0.046
0052
4.003
4001
-2227
4,835
0.000
0.382
D052
50055
-0136
.0.091
00.45
00641
00113
-0.107
-0.079
-0036
00252
0.076
11.49%
4.14
47 101
40007
-0.44
-0053
4.08
4000
-0021
4,079
40099
-0063
40.76
-0056
40157
-002
-0.95
40.40
-0650
40.11
20.11
-0,094
0.0%0
40301
40014
33.33%
4110
0101
0.105
0,002
0.199
0.168
0464
0.001
0/91
15.19%
4.41
9137
4016
4.19
4073
4098
-0
-0175
-2071
-119
4094
37.04%
4.73
0.00
0.91
0137
2041
0,82
0.055
0114
2024
0351
0.09
0,357
0053
0362
0.207
0,057
3072
0.201
0,083
4.15
-
4000
1000-1110
- 10C
-1106
40946
4140
-2070
4.022
4114
-6173
4007
-118
-1.66
-0560
.6239
-0154
-207
-1201
-0130
40.74%
0.00
00000
0002
0009
0.09
0115
0336
0.900
0.085
0.115
0.008
0033
6081
0,00
0113
0.06
0057
0.169
0115
1110
2004
0200
0.008
42.9%%
0.235
42229
-1177
10055
4.013
4.16
.2050
4,052
2.100
11.06
4.124
4.111
4.058
4.199
4.006
0005
-0170
.0.175
.0.73
-0003
-0201
-0.15
44,445
41144
4119
41001
1115
41046
4.131
-0054
43005
41116
41128
41190
4.161
-0094
-0777
40.199
-0067
-0.25
20:245
-0.181
-009
-6200
-0141
46.9%
4:243
-52233
40
47174
41.00
-0111
-0.06
41063
4.165
-0071
4.09
40.00
40152
40361
-0128
40162
-0000
-0197
40562
.0675
-0231
-0223
-0.155
0.001
.0201
.0.41
48.15%
0. 0.163
0.302
-100
0.310
0.002
0.259
0,511
-0.14
5003
0.141
59%
-0244
4.135
00174
0.060
00611
0.227
.0.198
-0.11
5001
0.206
-0140
$1,85%
4.174
4008
-0198
4.251
-1000
-1.1%
-0.25
2.316
-0.15
0000
-0301
-0176
53.74
0:218
0:22
0.256
0.155
0.17
0100
0219
0113
0.300
0217
0145
01777
0.53
0,067
0055
0.294
0,221
0,011
1000
0200
0
$5,564
4-345
1::45--0
4.148
-0156
-0150
4.94
4.40
-0090
4.077
4.171
-217
4:237
-009
4,259
4002
-0216
40100
.0087
..250
4.339
-0.93
.0300
.0333
$7.41%
0:255
0.244
0.127
0.11
0.34
01000
0:20
20093
0,081
0. 1.106
0.196
0.224
0111
42077
0.064
0.256
0.067
0.084
0.255
.03300
-9161
-0451
0/000
0.174
99,009
4:14
42.35
41150
-1
41:49
41190
4:211
-0.13
-0 101
11146
40310
41336
4177
4.361
-1164
-0.381
41.17
-0.44
20334
-0130
-0451
.4000
-0195
51.11%
0.275
0.266
0.399
0L145
0.175
0.001
01085
0.230
0.115
0,002
0.173
0.36
0.251
0131
0.261
0.125
0.302
0.106
0699
0.325
0.392
0.195
3455
0601
0.354
63.969
4.326
4331
40.305
41130
4.155
.0.48
41136
4218
-0005
4.088
41138
4.316
4.238
4.107
40358
4.105
-0692
-0344
.0311
-0.154
-3459
-0601
$4.11
4.115
45333
41
1.198
40316
40:50
4006
4:24
-1118
4..19
41
1989
40350
40310
-008
4.293
40113
4.148
-0114
40.94
4.994
-0.533
-3499
-034
56.67%
0,007
0,104
0.36
0.106
0.394
0.358
0.152
3451
0.225
08.52%
0.359
0:386
0:344
0225
0106
9.168
0.34
033
0.30
0.718
0226
6,799
0,139
0,146
01140
0,378
0,008
0220
4199
0.27
70.37M
-411
0220
400
4:57
-014
4.105
-217
4.99
4.12
4330
0.137
-1121
4.35
4.360
.225
-045
02000
0.25
93,21
4.119
4:00
4236
4007
4-124
-2140
4.11
4030
4.11%
-0014
-1143
40 165
4.15
-0121
-0451
-020
.6259
74.0%%
0.369
0:344
0.915
0.355
02117
0:99
0.104
0:243
0.135
0,120
0.162
0255
0.327
0.29
0313
0,109
0.533
0.151
0121
6425
0.93
0200
0200
0.25
75.0P%
A.363
.2334
42410
43365
4211
41100
4314
-2.156
4.148
40.30
-1390
4.339
-0226
4.293
4.130
40
-..622
-0154
-0445
-0375
-0.205
-0451
-0300
-027%
71.74%
-0421
41421
0.30
4:251
41387
4.373
4.00
4.18
-0223
4.198
41392
41354
4.419
-0391
00885
-0.173
-1451
41.87
-0316
-0.98
-0.55
-0395
-0459
40330
79,679
-0411
-3.401
40.41
41228
40399
40246
41197
4.404
-3.14
40.190
41
40128
4.395
-0356
-0423
-0.55
-1425
41.89
-U.14
40,55
-0.44
-0.345
-0496
-0.14
$1.44%
0.149
0451
0.571
D455
0.200
0,313
$3.31%
41371
4,509
-0415
-0211
-0216
-0,55
-0.200
-0457
0.197
-0360
65.99%
4.311
-0351
4.57)
-0,15
-0.200
-0.55
-0.30)
-0451
-020
-0,85
57.04%
0,335
0,162
0.419
03313
0.323
2.37
4.72
0.212
0.455
0.345
0.243
0,599
0.64
0,363
3.457
0.197
0,401
$1.8%5
-2.45
-0.481
41540
J0408
.0.16
4.130
4414
-0275
4.195
-1299
.046
4.400
-0.44
4,517
4.209
-0467
-0.26
.0312
.0.95
J.354
40.99
.0.99
J.12
4.310
90.745
0,457
.0,301
.0,500
...500
J0426
A.311
.1.13
4:535
-2274
.0200
.0.288
4.462
0.539
0.335
4,650
4.183
-0.58
4.309
-0212
-0652
.0,63
.0,395
-0451
.0201
-0402
97.9%%
4:545
-3,55
43.16
41400
-1481
4.445
41
166
4055
-0.34
41.70
41
41.95
41366
-645
4.68
41251
40511
41.19
-1.16
-0613
-0.64
-1390
-0455
-170
-0454
94.44%
0,555
0.5%
0,510
0.436
0.415
0.164
0.314
0.399
0.390
0.647
0.566
0443
0.71
0.24
0.492
0.29
0.355
0,39
0661
00370
0451
0202
0.65
96.3%
4.58
-1.56
41.525
41435
4.46
4416
41162
-3334
41226
41357
40572
4.578
-0314
4.62
4.243
-1488
40238
-0334
4.51
25843
-0.44
-3461
40201
-0441
98.1%
0.505
0.391
0.355
4.391
4A18
40318
40517
41211
-0619
-0250
-0031
41200
-011
1003
4.743
4,399
.0.23
0.267
0440
0.585
4.757
-0.554
-000
0.201
0.511
57

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Model performance gap A of RD experiment under ROC-AUC on Jannis dataset.
Begro
XGB
CATE
IFN
DAN
NODE.
CAP
NET
1308
INT
DCN
PTT
(1R0
SAT
SNN
TTF
TABR
NCA
1.MA
11
UNI
AVR
188%
-0:001
-0001
-0001
-3003
-2/01
-1001
-4.501
-0/001
0.954
-3034
0.007
5.0%
37%
0.04
0201
0.004
0.00
0.333
0/033
0,877
0.002
0,002
0.502
0,10
0.938
0.255
3021
0.506
3,066
5569
4.035
-2030
-0.005
4.01
-0.06
-0009
-0.001
4.013
-2.05
-0.034
-008
-0.10
40306
-0039
-2007
4,003
-0.00
-0.00
-0656
-0001
2001
0.32
-2000
3000
7414
0005
0013
0,00
0.006
2.000
-0011
-0000
4.055
-09331
-0.035
40014
00016
40000
-0012
-0011
00013
4008
4.310
-0559
-0005
-0000
028
2002
1080
-9204
9.6
2004
0,12
0,01
0.000
0.004
0.006
0.002
0.000
20002
0.001
0,011
0.006
0,022
2007
4,007
0,013
0.001
0,505
0.008
0.023
-2025
0.255
0001
00007
80002
11.115
4003
-2825
-2016
41.00
-0000
-0015
-1002
4.099
-3003
-2.005
-1008
-1016
40015
.0013
-0014
-0109
-0102
-0.009
-0006
-0/05
0.201
000
-0005
12.96%
4003T
-0032
-0015
41.00
.0.14
-0018
40004
4,022
-3011
400011
4.101
-0.21
-0014
-2018
-0017
40009
-0122
40.155
.0612
40635
-2,05
0.301
0000
0.007
.0.009
14.1%
4/04
-0004
4.025
-0015
40019
4.001
40015
-0011
400079
41015
40633
40018
-0019
-0017
40104
40102
40.00
-0012
-0031
-0002
0.15
-0035
00066
-0014
36.57%
-J441
0.011
0.50
0.065
.0074
0.500
0001
0.006
15.929
4000
41000
40017
-0030
41004
-0006
-0.05
-0010
20046
-0033
0,006
-001
30.1%
6/4R
9161
0/11
0023
6941
00130
0057
0029
20032
0101
0.01
0,006
0059
-0055
0112
000
0011
21.22%
6,76
0,07
8005
0.005
4035
-0014
4,016
4.61
00334
-0036
40025
0.125
-0055
-0015
20056
0.000
2002
0:566
-003
24.0%
0.064
0.41
0.106
0.67
0044
0.91
0.061
2/12
0,025
0.063
0.054
0.349
0047
0051
0,447
0.068
0,355
0529
0051
01033
2002
0.006
25.0P5
4,075
-0067
-0.40
4049
41014
2006
-0025
2,029
-0.59
40055
-0651
-0052
41044
-0004
-0.96
-0008
-0062
-20855
-000
0.007
-0.045
31.78%
0.711
41063
0009
0033
4000
40041
-0125
0,07
41000
46311
-0014
-1139
41034
-0100)
40005
10030
-0120
-0,956
0009
-003)
00006
-0031
29.67%
4,065
0.054
01106
0.004
0035
4.001
4:947
-2015
4,020
4.109
4.039
0.346
4033
.0122
.0002
0.323
00622
0.106
.2,955
0.023
-0055
0.307
11.45%
40000
-1077
4008
40061
40005
.0014
40007
40051
-0015
4.018
4008
4004%
-0347
-001
40135
-0035
40064
40.02
.0630
40965
21063
0.071
0,0%0
01007
.0095
33.33%
0.099
0,04
0,95
0.033
5071
0,066
0045
15
4000
0020
4001
4101
0000
0,096
4.002
4,042
4013
-00M
000
-3071
0,26
4049
37.04M
0,094
0.04
0.166
0044
0000
0085
2002
0,27
0199
0.76
2051
5071
01844
0.02
0,88
0631
0096
0111
0067
300
0.506
0053
4.144
-208
-9077
4067
-008
4002
4000
4085
4004
-007)
-008
4081
-0095
-0556
-1154
-013
.009
0.%0
0.007
-0074
40.74%
0.001
0160)
0000
0026
0100
0/28
0/239
0.06
0167
0002
0005
0034
0.05
0.12
0.04
4540
0003
00012
0001
0.006
0081
42.9%%
A.130
.1133
0000
10001-00
.002
0055
4011
40076
.2136
4,025
2.14
0056
4000
0.065
01111
4.042
-0094
.0,008
0.02
-0003
0.006
-006
44,44
4.115
J0.008
11000
40990
4101
4.11
-0856
41051
-1082
41 10S
41112
4.102
40187
-0111
-0.08
-6061
-0.40
.0.14
-0.100
-003)
0.066
-0084
46.9%
4.134
-0121
40.67
41165
-0063
-0075
41006
4
-0054
41045
40006
-0.9
40.59
-0011
-0/95
-0.75
-0.50
-0.69
.0650
-0.45
-0,145
-0.02
-0031
0.066
4001
48.1%%
0.009
0,105
01884
0.143
0.500
0053
0.145
-0.7%
-0.16
0,000
0000
0082
59%
-111
0,009
-0122
0.57%
00045
00133
.0.134
-0.012
-0001
0.006
0003
$1,85%
40 10100
40084
40051
-012
4000
-0111
-1.15
-0064
-0169
-0.00
.0.199
0.007
4.105
53.7%
0122
00000
0.068
D088
0.015
0141
0055
0121
0.128
0.194
6.125
0.85
0.179
0,576
0044
0.157
0.192
0.113
0,000
0,006
0099
55,564
4.17
11166
11124
4.006
30060
00096
4.006
41140
-005
4.054
111
-112
-.143
-0114
.6131
4,099
-0144
.0.69
.0560
-0.16
.0.19
0.022
-3166
0.007
0.107
$7.41%
0.145
0.120
0.033
0.99
0.017
0.130
20058
0,048
0.072
a.121
0.132
0099
4.112
0.075
0.39
0.261
0.054
00154
-0.189
.0,00
-0165
0.000
0.08
99,00-9
125
41009
100
40
A.137
.008
-1064
-1 11
-1
41.49
-0113
-0.17
-0005
-0 167
40.00
.0055
-0.40
.0.189
-01944
-0163
-4.01
-0.13
0.165
0.135
0.148
0.079
0.000
0.100
0161
0.140
04044
0.05
0.116
0,157
0.157
0.145
6.121
0.188
0.011
0667
0.1N2
0.225
00.05
0,000
0.12
63.969
A.13
41135
0.096
-0.15
40023
A115
-0033
4,059
4.000
40139
4.157
-0117
4.154
40116
-0.30
4057%
.0674
-0254
.0,550
.0.00
-0116
.0601
-0123
S4.11
40 16
41
41.00
40121
-0:41
40009
4.191
-0055
4,079
41133
40155
40.95
-0141
-0165
40116
41183
.0699
-0211
40234
-0.111
-3156
(1006
-0
56.67
0,109
0.19
0.21
0.058
0.165
0.006
0.133
BUST
9175
0137
0.00
0167
0215
9121
0.122
0.196
0.237
0180
6,204
0130
0200
4131
0100
0.214
0.296
0.11
3116
0,60
70.37M
4.233
2223
4.147
-.104
40:42
4096
4119
2010
4.02
11121
4.185
5.162
4130
9.195
4117
0081
-211
4.236
.0177
-3166
0.006
-0149
4145
1.191
-1126
10126-411
4009
4218
-0110
4.085
4010
-019
-5185
-1.99
-222
4.11
-0566
4:211
4:264
2.00
-0196
-6151
74.0%
0.221
0216
0.3%
0.144
0.136
0.150
0.000
0.359
0.008
0.090
0.120
0,10
0155
0.194
0,127
0.20
0112
0100
0.259
0221
0.11
0114
D0206
0.15
75.025
A.443
.2233
42210
411
-.149
-1157
40001
4.02
22.15
J.
1.105
-1170
41178
46356
-115
4.184
4.138
-0254
4.117
-.129
-0255
.0242
.0.17
-0116
0.006
-0.16
71.74%
4.230
40233
4.191
41110
40237
4.235
4106
4,215
-0.300
40.157
41264
41383
4.277
-2253
4,214
40.154
40
41257
-0185
-0211
-0336
4.185
-2165
0,06
-0211
4:251
4.244
41.255
41.52
-0.63
40219
41 11
4:311
-3165
40.166
41.14
4035%
40355
4.245
41160
11366
41162
-0.14
.0.13
2.311
161
-B116
-0000
$1.44%
0.156
0.142
0.270
0.302
0/00T
0.10
$3.31%
41261
4.365
40222
-0200
-0.155
-0.35
4,312
-0.50
-0154
0,000
-0221
65.99%
-0.19
4.386
40150
4.29
4,198
-0.58
-1187
-0119
-0.00
-0339
-0.244
-9166
-0226
57.04%
0.256
0.233
0.276
0131
0.253
0.167
0:235
0224
0.315
0174
0.394
0257
0.733
0213
0.314
0.341
0.216
0,353
0,31
0.29
3115
0501
0,25
$1.8%5
4.281
-0.38
41218
.0356
4.113
4:244
-0219
4.
40.66
41343
-0384
A.354
-035
4:344
4.206
-0335
40314
.0176
.0315
4,34
.030
.2161
.0811
.0343
90.745
4.359
.0.305
00,00
4.215
..186
4.379
.1.14
4:316
-2223
.0.15
.0.25
A.307
4.333
4317
4,186
-0.50
-0.366
-0116
-0314
.0,408
-0,35
-0167
0.006
-0255
97.5%%
4.716
4.554
41235
JL.45
4:285
4:11
-0.11
4.318
41.39
-1155
4L111
40366
4.54
41241
41
41.15
-6.06
-0361
-0891
-0512
-0156
(UX0
-0.99
94.44%
0.337
0.338
0,345
0.254
0.364
0.350
0.299
0:00
0.323
0,330
0.341
0.386
0,315
0381
0383
0233
0.358
0.361
0.31
0415
-3417
00300
3156
0.066
0.315
96.3%
4.44
-0332
41.300
41.54
40362
4.316
41.14
4:335
-3266
4.192
41.36
40356
4.357
43445
4.38
-1324
40363
40315
4035
-3448
-0.31
.0165
0.007
.0365
98.1%
41 1.364
01301
01.00
3096
4,11
48
40
4LMC
41220
-094
-3116
(1066
00.90
100
0.399
6414
0443
4.15
4
0.422
0.360
0.366
0.451
-0434
-0005
-3029
0.007
00351
58

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 21. Model performance on Penguins dataset.
(a) Model performance of SC experiment under accuracy on Penguins dataset.
Colrmn
PCC
PPN
DAN
GO5
SAT
TTP
TABR
NCA
0.561
0:311
1.661
0,950
0.473
03384
0.976
0.792
0.541
0.754
0.225
0.749
0.344
0.754
[1541
0.256
0.50)
1,00
0.771
6015
0.971
0.966
0.965
0.985
0.473
0.821
0.629
0.522
0.981
0.797
0.464
0.254
0.377
0.92
0.49
0.256
0.254
0.641
0.261
0.860
1,80
0.855
0.825
0.966
0.265
-0656
0.865
0.57
0.522
0.985
0.463
0.865
0.628
0.611
0.936
0.775
0.556
0.254
0.353
0.93
0.399
0.246
0.254
0.550
0.242
0.555
0.985
0.754
0.826
0.55
0.756
6.731
0.715
0.715
0.758
0.734
0.483
0.754
0.629
0.710
0.75
0.611
0.512
0.254
0.300
0.72
0039
0.304
0.340
0.996
0.25
0.349
0.234
0.725
0.541
0.57
0.663
-0.44
0.971
0.52
0.66
0.911
0440
0.825
0.634
0.928
0.91
0.768
0.517
0.734
0.257
0.90
0.325
0.251
0.739
0.531
0.211
0.545
0.971
0.826
0.541
0:571
0.756
6,750
0.661
0.557
0.066
0.61
0.430
0.874
0.955
0.859
08502
0.739
0.473
0.758
0.406
0.32
0.344
0.271
0.754
0.536
0.275
0.570
1,000
0.841
0,512
0:986
0.76
0.854
0.066
0.966
0061
0.90
0.430
0.884
0.688
0.957
0.91
0.807
0.27
0.754
0.34
0.25
0.34
0,304
0.749
0.516
0.213
0.574
0.955
0.797
0.541
0,986
0.799
(b) Model performance of SC experiment under ROC-AUC Penguins dataset.
PCC
LGB
DAN
5W]
DCN
TABR
NCA
AVE
0.990
0.950
0.993
1,000
01856
0.989
0.911
0.997
01.99
0.919
0.590)
0.938
0.556
0.99
0,991
0.228
0.982
0.047
0.298
0.999
1.000
0.823
0.857
0.986
0.887
6.015
0.989
0.959
0.990
1,50
(.689
0.915
0.915
0.997
0.999
0,911
0.821
0.918
0.530
0.99
0.994
0.219
0.963
0.650
0.388
1.00
1,000
0.856
0.543
0.986
0.882
-0.55
0,575
0.987
0.663
1,000
0.501
0987
0.892
0.998
1199
0.925
0.775
0.956
0.541
0.99
0.855
0.230
0.993
11564
0.492
0.566
1,00
0.759
0.574
0.8%
01875
0.731
01374
0.855
0198
0.819
0.538
0919
0.854
0.914
11877
0.700
0.17
0996
0.538
0.77
0.500
0.395
6993
01324
0.374
0.888
0.776
0.561
0:584
01901
-01764
01911
0:952
0.677
1900
0.668
0930
0910
0.951
1197
0385
0.67
0991
0,522
0199
0.988
0.38
6.949
0.822
0.99
1050
0.823
0.571
0:971
0188
0.750
0.989
0:988
0.97
1900
01329
0.94
0.877
0.989
1197
0.800
0.84
0.999
0.558
0199
0:881
0.266
0.900
0.500
0.805
0.98
1950
0.841
0.857
0.986
01879
0.854
0.901
0.959
0.994
1,00
0.705
0.75
0.886
0.906
01996
0.800
0.782
0.96
0.50
0.99
0.991
0.200
0,966
0.955
0.365
1.000
1.000
0.199
0.671
0.986
0.876
(c) Model performance of MC-M experiment under accuracy on Penguins dataset.
Degree
LG3B
XGB
CATB
PPN
DAN
MLP
.M0DE
RB5
.WW
CAP
NET
603
DCN
PTT
(RR
SAT
SNN
TABR
NCA
L.A
7LLM
UNI
AVE
0%
0.981
0.971
0.961
0.990
0.478
0.284
0.614
0,966
0.575
0.792
0.541
0.754
0.319
0.925
0.749
0.245
0.754
0.841
0.255
01850
1,000
0.826
0.526
0.586
0.771
16.67%
0.176
0.966
0.961
0:900
0.430
0.284
0.618
0:557
0.981
0.807
0.527
0.754
0.348
0.928
0.744
0.304
0.749
0.816
0.213
0.874
0,915
0.797
0.541
0.986
0.700
31.35%
0957
0.953
0.961
0:961
0.343
0.800
0.546
0.830
0
0.688
0.004
0.754
0/421
0.584
0.729
0.338
0.731
0.836
0.233
0.870
1,000
0.113
0.41
11966
0.753
506
8.656
0.560
0.633
0.845
0.387
8.071
0.478
0.855
0.97%
0.517
0.55
0.715
0.455
0.889
0.33
0.80
0.61
0.768
0.265
0.765
0.336
0.899
(1971
0485
66.67%
0.362
0.40
0.483
0.488
0.382
0.367
0.433
0.512
0.55
0.319
0.444
0.527
0.401
0.512
0.433
0343
0.430
0.643
0.227
0.517
0.556
0.551
0.841
0.057
0.495
83.33%
0.329
0.353
0.314
0.309
0.382
0.391
0.367
0.362
0.353
0.362
0.377
0.577
0.372
0.372
0.222
0.34
0,391
0,314
0.346
0.329
0.246
0.440
0.54]
0.297
0.390
1006
0.271
0.357
0.271
0.232
0.382
0.391
0.382
0.302
0,382
0.391
0.382
0.342
0.391
0.392
0.391
0.391
0.59]
0,391
0.346
0
0.348
0.493
0.44]
0.797
0,397
(d) Model performance of MC-M experiment under ROC-AUC on Penguins dataset.
Degres
LGB
XGB
CATB
PFS
DAN
MLP
S000E
RBS
SWT
CAP
NET
GOS
INT
DXN
FTT
GBO
SAT
SNN
TTF
TABl
NCA
L.M
111.M
USS
AVI
1rs
0.900
0.991
0.987
1.000
0.856
0.989
0.911
0.897
0.999
0.919
0.59)
0.988
0.566
0.990
0.91
0.378
0.912
0.947
0.494
0.939
1.000
0.835
01837
01986
0.3K1
16.77
0.981
0:589
0.994
1.000
0.705
0.977
0.386
0.996
0.995
0.890
0.763
0.998
0.601
0.900
0.91
0.29
0.985
0.935
0.365
1,030
1.000
0.759
0157
0.986
0.875
$3.38%
0.991
0:979
0.988
1.000
0.66
0.911
0.332
0.987
0.990
0.86
0.725
0.996
0.666
0.89
0.72
6328
0.981
0.948
0,432
0.996
1,000
0.825
8857
0.986
0.570
50%
0.955
0:924
0.981
0.993
0.655
0.657
0.803
0.922
0.985
0.756
0.712
0.994
0.567
0.63
0962
0.49
0.979
0.885
0.473
0.991
0.999
0.908
0.85
0.986
0.869
66.67%
0.725
0.501
0.811
0.760
0.648
0.250
0.699
0.349
0.725
0.515
0.531
0833
0.044
0.799
0.73
0.584
0.21
0.706
0.440
0.834
0.845
0.613
0.847
0.984
0.727
83.33%
0.492
0.486
0.498
0.491
0.489
0,692
0.501
0.510
0.470
0.464
0.505
0.676
0.499
0.494
0.422
0.433
0.504
0
0.500
0.463
0.455
0
0.547
0.987
0.525
100%
0.500
0.500
0.500
0.500
0.500
0.900
0.500
0.500
0.503
0.500
0.500
0.900
0.500
0.400
0.500
0.100
0.500
0.500
0.503
0.900
0.500
0.537
0.847
0.987
0.316
(c) Model performance of MC-I. www.eeeppeeeettm under accuracy on Penguins dataset.
Dogres
LGB
XGB
CATTB
PFB
DDA
MLP
S0006
RBB
SWI
CAP
NET
GOS
INT
IXN
FTT
GBB
SAT
SNN
TTF
TABlk
NCA
L.MA
T1..M
UNI
AVI
0.981
0:971
0.961
0.990
0.476
0.884
0.614
0.966
0.975
0.792
0.541
11754
0.119
0.99%
01349
0.366
0.754
0.341
0.246
0.889
1.000
0.875
0.876
01986
0.771
16.67%
0971
0.966
0.965
0.986
0.473
0.821
0.609
0.952
0.981
0.797
0,444
0.754
0.377
0.942
0.249
0.256
0.754
0.841
0.361
0.850
1,000
0.855
0.526
0.985
0.764
33.35%
0.792
0:952
0.957
0:976
0.483
0,82
0.609
0.94
0.965
0.768
0.485
0.758
0.401
0.953
0.249
0,295
0.754
0.851
0.290
0.850
0.985
0.730
0355
0.70
0.753
506
8.609
0.091
0:686
0.686
0.683
0.720
0.009
0.744
0.744
0.688
0.664
0.754
0.391
0.755
0.754
0.396
0.754
0.667
0.285
0.77%
0.633
0.652
0.41
0885
06688
66.67%
0.377
0.985
0:641
0.500
0:411
0.071
0.385
0.68
0.715
0.688
0.513
0.734
0.420
0.61
0.71
0.391
0.700
0.687
0.245
0.766
0.523
0.687
0836
01884
0.594
81.199
137
0.185
0.221
0.488
0.991
0:475
3.50
0.531
0.544
0.575
0.409
0.730
0.4%9
0.55
0.33
0.387
0:449
0.454
0.366
0.715
0.181
0.667
0.41
0.81
0.134
100%
0.231
0.180
0.221
0.333
0.183
0.381
0.982
0.767
0.583
0.991
0.181
0.883
0.991
0.311
0.191
0.991
0.391
0.366
0.991
0.336
0.41
0.747
0.197
(f) Model performance of MC-L experiment under ROC-AUC on Penguins dataset.
Degree
LGB
XGB
CATB
PFN
DAN
MLP
.NODE
RES
SW]
CAP
NNTT
605
INT
DCN
FTT
GBO
5.T
SNN
TTF
TABBR
NCA
L.M
TLLM
UN]
AVE
0%
0.950
0.500
0.997
1.000
0.856
0.559
0.911
0.397
0.599
0.91
0,500
0.995
0.546
0.599
0.991
0.228
0.992
0.947
0.495
0.930
1,000
0.823
0.857
0.986
0.853
16.67%
0.939
0.980
0.997
1.000
0.559
0.955
0.925
0.997
0.599
0911
0.521
0.995
0.530
0.999
0.94
0.119
0.993
0.950
0.505
1,030
1,000
0.856
0.543
0.986
0.832
$1.199
0.992
0:581
0.981
0.999
0.580
0.942
0:902
0.992
1,009
0.938
0.566
0.944
0.96
0.66
0:306
0.984
0.935
0.997
1.000
0.245
0835
0.887
0.376
50%
0.904
0.549
0.707
0.537
0.554
0.884
0.334
0.875
0.818
0.282
0.599
0.08
0818
0:387
0.882
0.791
0.384
0.988
0.826
0.668
0.847
0.987
0.978
66.67%
0.945
0.543
0.833
0:914
0.864
0.82
0.110
0.892
0.657
0.708
0.679
0.004
0.65
0.48
0.492
0.881
0.747
0.907
0.785
0.611
0.832
0.985
0:074
83.35%
0.820
0.566
0.677
0.781
0.54
0.022
0.606
0.807
0.791
0.657
0.508
0.998
0.73
0.82
0482
0.855
0.655
0.484
0.888
0.37
0.687
0.847
0.987
0.741
100%
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.527
0.847
0.987
0.536
(g) Model performance of RD experiment under accuracy on Penguins dataset.
Degres
XGBB
(AATH
PIN
DAN
MI.
NO00
RBB
SW1
CAP
NET
608
INT
DXN
P/TT
GRO
SAT
SNN
TTP
TABIR
NCA
1.MA
11.1M
1/N1
AVIE:
0S:
09911
0:971
0.961
0.500
0:478
0.884
0.614
0.966
0.975
0.750
0.561
0.754
0.319
0.978
0.249
0.346
0.754
0.341
0.256
0.850
1.000
0.876
0.876
01588
0.771
16.679
09900
0.919
0.976
0.940
0.485
0.877
0.609
0.399
0.979
0.761
0.50
0.751
0.894
0/240
0.327
0.750
0.810
0.281
0.842
0.948
0.915
0.811
0.919
0.750
$1.19%
0.267
0.367
0.811
0.391
0.435
0.785
0.592
0.871
0
0.218
0.476
0.548
0.24
0.734
0:380
0.341
0.77%
0.261
0.540
0.916
0.800
0.641
0.14
0.734
59%
0662
0.798
0.231
0.317
0.405
0.205
0.541
0.796
0.815
0.699
0.461
0.726
0.75
0.688
0:320
0.705
0.713
0.763
0.824
0.23
0.845
0.901
0.661
66.67%
0.589
0.676
0.301
0.17
0.514
0.678
0.715
0.556
0.469
0.081
0.311
0.360
0.621
0.644
0.677
U339
03833
0.598
X1.358
0.451
0/417
0.532
0.577
0.518
0.387
0.433
0836
0367
0.497
0.352
0.367
(h) Model performance of RD experiment under ROC-AUC on Penguins datasct.
Degree
LGB
XGB
CATB
PFN
DWNN
MLP
.NODE
RES
SW1
CAP
NET
0005
1NT
DCN
FTI
GRO
5AT
SNN
TIF
TABB
NCA
LMMA
TLLM
UNI
AVI
0.990
0.990
0.997
1.000
0.855
0.911
0.997
0.999
0.919
0,590
0.995
0.991
0.128
0.992
1,000
0.823
0.857
0.986
0.383
16.67%
0.970
0.971
0.978
0.981
0.525
0.974
0.892
0.981
0.967
0.367
0.679
0.990
0.978
0.68
0.276
0.979
0.933
0.455
0.950
0.983
0.912
0.861
0.969
0:372
33.35%
0.926
0.946
0.951
0.955
0.32
0.63
0.378
0.970
0.969
0.55
0.682
0.982
0.60
0.300
0.773
0.988
0.733
0.862
0.566
0.947
0:861
50%
0.858
0.917
0.91
0.922
0.444
0.025
0.832
0.18
0
0.411
0.671
0911
0.375
0.400
0.44
0.712
0.61
0.570
0:822
66.67%
0307
0.565
0836
0.854
0.757
0370
0.768
0.8%6
0.53
0.71
0.645
0355
0410
0383
0.792
0.685
0S46
0967
0.711
X3.35%
0.701
0.500
0.500
0.500
0.500
59

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Model performance gap A of RD experiment under accuracy on Penguins dataset.
Degme
16.575
33.32%
0.368
0.107
0.115
0.200
0.50
0.106
0/33
0,088
0,000
0.063
0.130
0.008
0.391
0657
0.055
0.137
0017
0675
0.025
0023
0084
0053
0.008
0002
0061
596
J.336
-0.78
-0.40
AL175
-0145
4L208
40.15
-0177
4.164
-0118
40142
41087
0.317
...61
4.061
11388
40063
-0112
40058
-0.13
4.176
4.34
0.01
4005
-0.14
$6.57%
0.496
AL258
.0416
0.318
4.183
4.30
40163
41299
4126
4130
40154
4000
0.311
3.355
4.157
B461
40.16
-U233
0.23
-0174
-0313
40331
0.06
4.054
-0,233
(i) Model performance gap
of RD experiment under ROC-AUC on Penguins dataset.
Degre
1.68
XGB
CATB
PTN
DAN
MLP
N0066
RES
CAP
NET
008
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
T1..M
UNI
AVE
36.57%
-1.005
-0X12
33.33%
30%
0.303
0001
0.085
0,079
0.131
0.064
0.67
0.09
0.083
0.191
0.029
0.000
0,085
0.78
0.81
0641
0.83
0.102
0,147
0.089
0.086
0.11
0.005
0016
0069
56.5%
0.175
0.126
0.162
0.145
0,116
0.120
0.15
0.141
0.163
0.236
0.06
0.065
0.011
0.40
0.137
0.79
00899
-0.63
-0.132
4.091
0.185
168
00011
460/
0.115
$1.37%
4,233
-0:218
4.203
-0.77
4.190
-02233
-0343
4.343
0.265
302
-0311
4.007
40359
40314
0911
-0.00
-0.20
0.001
-0.23
-0.23
0.356
-0003
40801
-0.36
60

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 22. Model performance Abalone dataset. (a) Model performance of SC. experiment under RMSE on Ahalone dataset.
Culunn
PCC
LGB
XGB
CATB
DAN
MLP
NODE
RES
SW]
NET
GO5
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
AVE
0
1.
2.184
2.183
2.122
3.103
2.367
3.076
2.233
3.124
2.639
2.237
2,674
2.509
2,606
3,048
2.487
3,003
3.172
2.174
2.194
2.422
2,603
-0:035
2,235
2.237
2.169
5.102
3.076
2.318
3.123
2.311
2.234
2.831
2.616
5.655
3,009
2.957
3.185
6.674
2
0.441
4.806
4,846
3.640
3,009
2,534
3.078
2,699
3.119
2.701
2,931
2,704
2.545
2,939
3,015
2.526
3.968
3.201
2.782
3.377
4.411
3.248
0.504
2.411
2,429
2.765
3.103
2,374
3
2.341
2.755
2.706
2.574
3.65
3,023
2.487
3,099
3.175
4.199
4
0.540
4.931
4.996
2.535
3.104
2.577
3.082
2
3.123
2.783
2.246
2.639
2.562
5.109
3,026
2,492
2.715
3.151
2.215
2.244
5.988
3.087
0.557
2.260
2.267
2.188
3.110
2.388
3.085
2.239
3.123
2.621
2254
2.661
2.574
3,022
3,026
2.497
2.995
3.148
2.236
2.217
9.585
2,977
6
0.557
2,245
2,234
1.231
3.103
2,443
3.082
2.530
3.123
2824
2.300
2,675
2.602
3,016
3,026
2.537
2.891
3.153
2.240
2,270
5.166
2.791
7
0,575
2.277
2.287
2.198
3,107
2.416
3.084
2.326
3,124
2,805
2.251
2,668
2.577
3.114
3.024
2.497
2.949
3.206
2.207
2.210
3.503
2,501
0,628
3.107
3,079
2.827
3.106
2,578
3.084
2,8
3.125
2.007
2,695
26615
3,037
3,031
2.833
8.728
(b) Model performance of MC-M experiment under RMSE on Abalone dataset.
Degres
XCIB
CATB
DAN
MI.P
NODE
SWI
NET
DCN
GRO
SAT
SNN
TTF
TARR
NCA
1.MA
0%
2.184
2.183
2.122
3.103
2.367
3.076
2.233
3.124
2.639
2.237
2.674
2.509
2.606
3.048
2.487
3.003
3.172
2.174
2.194
2.922
2.603
12.5%
3,079
3:106
2.550
3.085
2,836
3.126
2.691
2,616
3.038
2.617
2.836
3.176
2.620
25%
3.452
3,431
3.647
3.110
2.743
3,092
3.138
3.126
3.136
3.126
2.727
2,859
2,644
3.078
2.712
2.912
3.198
2.755
3.007
4,315
5.112
37.5%
3,868
3,846
4.099
3.112
3,024
3.097
3,477
3.130
3.045
3,590
2,895
2.754
3.027
2.914
3.162
3.170
3.144
3.332
3.819
3.256
50%
3,461
3,415
2,809
3.120
3.151
3.105
3.326
3.129
2,906
3,732
2,888
3,039
2.865
3.015
3,000
3.661
3.121
3.237
3.615
4,014
3,230
62.5%
6,941
6.997
2,879
3.121
3.267
3.110
3,750
3,151
2,988
3.956
3.137
3.153
3,006
3.089
3.119
3.994
3.119
3.523
4.297
5.614
3,810
75%
2,783
3.123
3,353
3.115
3.869
3.125
4.021
3.190
3.083
3.135
3.180
4.237
3.136
3.579
87.5%
3,309
3,309
2,825
3.119
3.067
3.116
3.025
3.125
3.097
3.098
3,259
3.075
3.110
3.079
3.111
3.332
3.192
3.047
3.025
5,989
3.266
1008
3.118
3.121
3,116
3.125
3.136
3.118
3.118
3.142
3.123
3.197
3.134
(c) Model performance of MC-L experiment under RMSE on Abalone datasct.
Degres
XGB
NODE
RES
SAT
SNN
TIF
TABR
0%
2.184
2.183
2.122
3.103
2.367
3.076
2.233
5.124
2.639
2.237
2.674
2,509
2,606
3.048
2.487
3.003
3.172
2.174
2.194
2.922
2.603
12.5%
2.233
2.227
2.169
3.102
2.391
3.076
2.318
3.123
2.731
2.274
2.831
2.616
3.065
3.009
2.510
2.957
3.185
2.357
2.238
6.674
2.854
25%
4.709
4.755
3.110
3,098
2.541
3.078
2.725
3.119
2.749
2.938
2.822
2.604
2.981
3.016
2.548
4.019
3,213
2.894
3.344
4.561
3.241
37.5%
5,394
5,455
3.726
3.099
2.521
3.083
2,714
3,116
2.789
2,983
2,833
2,578
2.987
3.029
2,553
3,802
3.233
2.962
3.533
4,644
3,351
50%
2,535
3,843
3:101
2.488
3.089
2.502
3.117
2,600
2.794
2.722
2.564
3,088
3.048
2.595
3.182
3.196
2.780
3.04
2.951
62.5%
2,713
2.702
5,306
3.108
2.514
3.097
2,585
3.118
2,683
2.821
2,724
2,608
3.160
3.061
2.658
2.906
3.146
2.852
3.269
4,283
3,066
3.110
3.081
3.176
5.461
87.5%
2,501
2,503
3.245
3.116
2.709
3.108
2,612
3.120
2.684
2.708
3.054
2,829
3.196
3.094
2.878
2.628
3.147
2.804
3.328
5,550
3.039
100%
3.314
3,315
3,277
3.118
3.121
3,116
3.123
3.125
3.128
3.124
3.809
3.136
3.118
3.118
3,142
3.123
3.197
3.134
3.123
6.292
3,348
(d) Model performance of RD experiment under RMSE on Abalone dataset.
Degres
LGB
XGB
CATB
DAN
ML.P
NODE
RES
SWI
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
AVE
0%
2.184
2.183
2.122
3.103
2.367
3.076
2.233
3.124
2.639
2.237
2.674
2,509
2.606
3.048
2.487
3.003
3.172
2.174
2.194
2.922
2.603
12.5%
3.034
3,046
2,510
3.104
2.438
3.081
2.419
3.122
2.692
2.424
2.700
2.564
2.590
3.057
2.520
3.048
3.175
2.357
2.469
3.030
2.769
25%
3,603
3,624
2,868
3.106
2.517
3.087
2.584
3.122
2.495
2.733
2.631
2.598
3.069
2.587
3.100
3.176
2.463
2.713
3.189
2.905
37.5%
3.955
3,979
3.166
3.109
2.602
3.092
2.725
3.123
2.844
2.755
2.777
2.705
2.629
3.077
2.678
3.158
3.175
2.651
2.931
3.394
3.026
50%
4,150
3.111
2.692
3.097
2.684
3.086
2.777
3.199
3.173
62.5%
4,132
4.157
3.50K
3.112
2.788
3.102
2,940
3.124
2.925
3,310
2,890
2,864
2.763
3.096
2,877
3,233
3.171
2,873
3.238
3,761
3.195
75%
3.993
4.015
3.52
3.114
2,886
3.107
3,019
3.124
2.954
2,859
3.107
2.964
3.340
3.167
2.912
3.185
87.5%
3.718
3,732
34588
3.116
2.996
3.112
3,079
3.127
3.029
3.084
3,089
3,031
2.988
3.108
3.056
3,216
3.167
3.022
3.272
3.469
3.194
100%
3,314
3,315
3,277
5.118
3.121
3.116
3,123
3,125
3.128
3.134
3.809
3.136
3.118
3.118
3.142
3.123
3.190
3.134
3.123
6.292
3,348
(c) Model performance gap of RD experiment under RMSF on Abalone dataset.
Degree
LGB
XGB
CATB
DAN
MLP
NODE
RES
SW1
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
AVE
12.5%
0.389
0.396
0.183
0.001
0.030
0.002
0.083
0.000
0.020
0.084
0.010
0.022
.0.005
0.003
0.013
0.015
0.001
0.084
0.126
0.037
0.064
25%
0.650
0.660
0.351
0.001
0.063
0.003
0.157
0.000
0.057
0.116
0.022
0.348
-0.005
0.007
0.040
0.032
0.001
0.133
0.237
0.092
0.115
37.5%
0.810
0.823
0.492
0.002
0.090
0.005
0.220
0.000
0.078
0.252
0.038
0.078
0.009
0.010
0.077
0.051
0
0.219
0.336
0.161
0.163
50%
0.888
0,901
0.595
0.003
0,137
0.007
0.273
0.000
0.091
0.263
0,058
0.109
0.030
0.012
0.117
0.065
0.000
0.269
0.418
0.221
0.195
62.5%
0
0.904
0.653
0.003
0.178
0.008
0.316
0.000
0.108
0.480
0.081
0.141
0.060
0.016
0.157
0.076
0.000
0.321
0.476
0.287
0.227
75%
0.828
0.839
0.665
0.011
0,219
0.010
0.352
0.000
0.119
0.318
0,106
0.176
0.097
0.019
0.192
0.079
-0.02
0.339
0.511
0.186
0,224
87.5%
0.700
0.710
0.630
0.04
0.265
0.012
0.379
0.001
0.148
0.379
0.155
0.208
0.147
0.020
0.229
0.071
-0.02
0.390
0,492
0.887
0.227
100%
0.517
0,519
0.54
0.005
0,318
0.013
0.398
0.001
0.185
0.397
0,121
0.250
0.196
0.023
0.263
0.010
0.008
0.111
0,124
1.153
0,286
61

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 23. Model performance on Bike dataset.
(a) Model performance of SC experiment under RMSE on Bike dataset.
Colorn
PCC
LGBB
XGGB
CXTB
DAN
ML.
NODE
X125
5961
NET
065
IST
DCN
FCT
GRR
5XT
SNN
11F
YABK
NCA
UMA
AVE
144.013
141.99
115.00
179.90
145.206
115.332
975
145,091
148.99
179.343
180.134
142.64
13/90
111.75)
152.19
0.035
148.954
189101
140.7%
172-50
150,95
175.880
190.253
183:25
155.370
150.30
152,503
150.933
153.77
177.833
151.77
191.596
119992
199755
102:29
115.245
158.271
2.
-.005
144.150
143.91
135,075
179.522
145.788
175.828
144.559
180.581
151.71
143.099
149.77
144.500
148.78
137.324
145.186
179.50
150.149
142.566
131.547
155.208
153.
D111
144.915
141961
19.364
179.90
145 291
115.95
168 957
193-133
155449
146.77
155.427
145.918
145
175.00
146 112
151 985
1800330
148.06
144.59
147.514
156.000
0.688
144.349
143.9
136,655
179:50
145.38
175.878
145085
192
155.55
145.89
150.317
145,80
149
176955
146.299
153.28
179-9%6
143317
134.50
148.900
155.5%
5
0.101
144,332
144234
136,355
179
145.215
175.578
144.072
193.06
153.004
155
149.735
145.355
141550
176.22
145.455
191919
179.815
143547
131956
157.222
125.614
6
0.311
157.476
173
142.62
179.897
152.234
176.722
154.015
193338
157.63
152
25
154.775
152.375
154
177,00
152.47
175.218
150.148
152.523
163.43
208.215
164D10
155.10
6M0
174957
15:
116
41V
193.85
155
461
K27
175.132
156.15
175
150,250
153
152-415
104 138
149.354
149.46
522
149
145.553
154
150.715
153/02
153
190,00
(b) Model performance of MC-M experiment under RMSE on Bike dataset.
Degree
11:88
ML38
CATTB
DAN
N4038
K1.5
SW1
NET
CU35
IN1
DUN
FTV
GRU
5.4"
SNN
T1F
TABR
NCA
LMA
AVE
0%
144.01
1-35:339
155:000
175:802
145.765
175.832
144.387
180.375
1SL.725
145:051
148.599
144.767
145.750
177.111
145.135
179:345
190.104
142.864
130,001
111.751
152.775
11%
342
104.13%
140.775
179:716
1901812
137.315
151 397
180.355
156.311
150.132
154.115
151.396
153.99
175.954
157.551
175.835
190.194
150.017
170,913
125.
100.3113
22%
182
556
184.250
140.775
179:956
163,099
178.474
162.96
180.356
166.727
154.767
167.489
162.138
163.993
150.110
167.43
185.835
192.743
156.408
172.426
120.357
163.495
S99
191
677
143.101
143925
150044
173435
198.945
124.119
180.331
175.26
135.915
174
135.515
174
151.8%
178.185
59.599
181.244
190.575
199,62
117.007
134.900
449
184
807
154:319
145 305
150.000
194.30
170.540
184
180.307
18
156.475
154
182.692
183
182.199
184
195 196
181.30 06.362
101 540
183.502
200.00
183
35R
185.725
196
150000
154
12
185
183.254
154.000
013
193.716
188
19411994
66R
185.550
157.21
157.5%6
182.985
186
195.900
199.614
195.764
170
8PR
185.566
180023
150,000
304
188
214
147.009
182.706
.44
176.574
200.588
197.90
291.216
185.435
1006
151
183.135
182
180020
190
180.345
181 009
185
181 311
180.344
180.153
131.619
180.036
190
158.463
181.918
188.5
182.643
(c) Model performance of MC-L experiment under RMSE on Bike dataset.
Degres
LGB
XGB
CATB
DAN
MLP
NC00
RES
SW1
NET
GO5
INT
DCN
FTT
GBO
5AT
SNN
TTF
TABB.
NCA
L.A
AVV
0%
144.013
143.939
135 000
178.892
145.766
175.832
144.337
180.375
15L.72
145.331
143.039
144.767
143.750
177.111
145.135
179.345
190.104
142.864
130,901
111.751
152.779
119
96-
149//00
140
179:000
190.01
175 680
191.351
180.165
155.500
150.50
157 545
150.505
151.244
197.35
151
193.994
164
149.355
160
113.445
156-271
229
145.973
172
179,895
183.395
154658
150.465
150085
153.742
178.262
195758
179.993
154.350
135.206
33R
151
183444
151575
150739
153
138.266
153.99
396,9%
179.775
154.235
199.176
147.334
163.216
449
635
149.145
185
170:006
175.999
153.167
180423
152.566
151.712
250.771
179.23
156.412
199.96
148. 48.500
161
148-100
179
306
199
157.952
164.093
60%
163.622
912
179.867
154977
177.24
177
188.406
161141
155.377
156.344
156.73
177.114
179
355.292
152.207
167.485
203.315
109.995
77%
171.378
185944
170000
164923
137.540
635
180.370
170.03
166312
176
165.594
179.966
737
179.000
175.476
173.204
158.635
174.414
899
168
600
105.03
188
180113
191631
198 796
125
250
170 224
108054
160
K25
10:317
17/95
175
177 866
575
193664
1M6
134
196731
170 129
1006
182.10
151
183.123
182
180029
190
150.345
181 909
180
181
150.544
140
180.553
181 18L.940
151.519
186534
380.066
180
158.463
181.918
188.774
182.643
(d) Model performance of RD experiment under RMSE on Bike dataset.
Degree
CATU
DAN
MLP
NODE
RB5
SW/
DCN
GBO
5AT
SNN
1.66%
0%
144.015
135.000
145.766
144.357
190.315
144.767
143.750
177.311
145.135
179.243
150.104
130.901
113.751
152.779
119
1.0000
141.994
176
148
155.500
195.63
171.671
190,775
150111
147.550
103.170
156.193
234
133.960
F40.359
151.223
176.85
153
156.292
154611
171.441
190,879
150.117
151.44
5587.00
203,513
33R
163679
136-133
12
157
177.200
15
190155
157.28
100.47
157
158.18
175.481
191,000
151127
151.24
105.211
145.540
104.75
44%
165.00
026
165-29
16.466
120
161118
137.347
162
188.144
165.471
161.999
164.123
162152
177.850
153,464
150.114
15.949
174.111
305.242
176.667
699
19
1n0.54
546
180 004
165.90 665
157.900
166 W7
190
170418
165.29
166.48
189
166.18
170
170
154.350
180348
182884
175933
812.100
124360
66
172963
171-33
172.446
178
120057
137.870
172 212
190.00
377134
174.075
170
170614
182.367
194 7G
185945
180166
156001
179.115
177.804
1250134
77%
176.42
176.966
176.558
170
172.06
179.55
173
196.11
776691
172.503
176.668
172.90
172.750
191.57
177.5 530
153.319
180.17
16:993
190.955
185.871
127.335
119
410
190210
140
179
99
120 293
199
175 411
190
21.
170
931
141.63
176
664
17 376
181
182 427
187.376
177476
171
211
153.739
195.54
180054
1006
182
151
123
143.730
180
020
190
1800055
181 900
190.195
181.11
190 594
190.00 541
180 155
181 940
181
196 534
180.05
180658
188
483
151.11
155.754
187.643
(e) Model performance gap A of RD experiment under RMSE on Bike dataset.
Degree
LGB
XGB
CATB
DAN
MLP
NODE
RES
SWI
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
AVE
11s
0.042
0.042
0.858
0.000
0.825
0.003
0.032
-0.301
0.027
0.027
0.024
0.028
0.015
0.003
0.032
0.005
0.000
0.048
0.130
-0.077
01.02
22%
0.060
0.081
0.111
0.000
0.051
0.005
0.064
-0.001
0.055
0.056
0.049
0.058
0.039
0.002
0.065
0.009
0.000
0.048
0.211
24:834
0.954
33%
0.114
0.116
0.160
0.078
0.095
0.084
0.075
0.077
0.064
0.000
0.271
0.332
44%
0.146
0.148
0.203
0.000
0.105
0.011
0.128
-0.001
0.091
0.114
0.102
0.116
0.090
0.001
0.130
0.024
0.000
0.063
0.330
1.917
0.155
55%
0.175
0.178
0.343
0.001
0.134
0.011
0.153
-0.001
0.123
0.143
0.133
0.141
0.117
0.012
0.172
0.028
0.001
0.069
0.344
1.973
0.174
66%
0.201
0.205
0.277
0.000
0.167
0.012
0.19:
-0.001
0.167
0.180
0.173
0.178
0.147
0.000
0.204
0.037
0.000
0.066
0.374
0.591
0.146
77%
0.224
0,230
0.019
0.204
0.186
0.190
0.161
0.690
0.161
88%
0.246
0.252
0.334
0.001
0.211
0.021
0.236
-0.002
0.5
0.220
0.220
0.220
0.192
0.023
0.257
0.017
0.002
0. 198
0.396
0.753
0.179
100%
0.265
0.272
0.353
0.001
0.236
0.024
0.260
0.001
0.195
0.245
0.279
0.244
0.223
0.025
0.285
0.004
0.003
0.318
0.390
0.689
0.195
62

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 24, Model performance on Concrete dataset.
(a) Model performance of SC: experiment under RMSF on Concrete dataset.
Colume
PCC
LG18
XGB
CATH
DAN
MLP
NODE
RB5
SWI
NET
GUS
ENT
DCN
F71
GR0
5AT
SNN
TTF
TABB
NCA
LMA
AVE
5.003
5.033
3.936
14.138
17,054
17.13%
18.134
12.997
12.814
15.955
16,088
17.699
7.314
33.711
14.395
-0.108
5.294
5.157
17.19
14.199
17405
13.120
17.132
17,508
12.095
16:352
12,902
15,900
17.313
10.511
17.344
12.356
7.517
30438
14.214
0.135
7.600
7,520
6.101
17.108
14335
17.061
13.291
17.140
19.277
12.815
16.419
13.001
16.012
17.399
15.272
16.765
17.715
12.574
8.773
123
19.499
-0.165
5.426
5.384
4,880
17.106
142200
17405
13.170
17.146
18188
12815
16,011
12,990
15,40
17:354
15.318
16.291
17.635
12.253
7.558
25,855
14,102
5.512
1.562
174094
14.288
174054
17.515
12,743
16,013
16.766
17.606
7.745
13:821
-0.200
1.743
5.005
17,900
14.555
17.059
13.49
17.13
19.418
13,095
13.18
16,105
17.155
15.468
17,05
17.650
13.139
9.359
24-461
147701
0.329
14.79)
12.448
14823
17.136
14834
17.06%
141177
17.155
18,89
13889
16.367
14.046
16.264
17.164
15653
17:250
17.709
14:356
14.359
22,994
16:156
0.166
5.499
5.901
4.570
17.111
148866
0.078
13.888
17.136
17.756
13.446
16.533
13,205
16.357
17.19
15655
16.14
17.688
12 12.101
8.700
35.599
14,51
14878
16.293
15,00
050000
(b) Model performance of MC-M experiment under RMSE on Concrete dataset.
Degree
1.GB
XGB
CATB
DAN
MLP
NODE
RES
SW1
NET
G0S
INT
DCN
PTT
GRO
SAT
SNN
TTF
TABR
NCA
L.MA
AVE
5,003
3.936
17.102
14.178
17.054
12.976
17.138
15,955
17.150
15.310
16.058
17.699
12:451
7.374
33.711
14.385
11.116
10,919
4.093
17,100
14.878
17.800
14.287
17.113
18.776
13.414
15.983
134002
16.293
17.234
15.673
15.940
17.228
13,090
11,050
30.430
15.203
25%
11.441
11.157
6.279
13.109
15.685
17.005
15.343
17.130
19.424
14.526
16.651
14.620
16,703
17:255
16,145
16.126
17.200
13.049
11,643
38.463
16.179
37.5%%
20.014
16.971
6.586
17.121
16.263
17.118
16.158
17.142
19.998
15.736
17.050
15,610
16.993
17.264
16.571
17.382
17.717
16.044
17.057
72:247
19.358
17,742
7.655
17.105
16.770
17.120
16.774
17.133
20.135
16,574
16.59
17.153
17:221
16.887
19:937
17.68
18.390
18,043
36.396
62.5%
20.812
18.
8,404
17.103
15.874
17.118
16,952
17.127
19.227
16.794
17:452
16 500
17.218
17.215
16.952
19.828
17.65L
18,873
18.105
38.918
18:376
75%
20. 20.358
18,010
16.465
13.112
17.019
17.117
17.109
17.124
17.383
17.002
17 17.306
17.000
17.307
17.213
17.081
19,855
17.576
18.449
18.100
95.046
21.383
97.5%
21.49:
18,665
18.084
17.118
17.116
17.124
17.231
17.115
17.404
17.146
17:553
17.172
17.392
17.228
17.154
19,949
17.569
18.079
18:735
91.190
21.526
18.556
20.762
17.117
17.124
17.125
17.131
17.112
17.117
17,399
17.204
17.051
17,305
17,538
17.476
(c) Model performance of MC-L experiment under RMSE on Conerete datasct.
Degres
LGB
XGBB
CATB
DAN
MLP
.ODDE
RES
SW1
NET
GUS
DCN
PIT
GRO
S.T
SNN
TTP
TABB
NCA
LMA
AVE
3:936
17.102
14.13%
17.054
12.976
17.138
12.597
12.814
17.150
15.210
16.058
17.699
12,451
7.374
33.711
12.5%
5.290
5.157
4,092
17.105
14.139
17.058
13.120
17.132
17.508
12.886
16.352
12,902
15,966
17.113
15.233
16.511
17.251
12.365
7.517
30.438
14.274
25%
7,386
7.314
10.455
17.112
14.376
17.064
13.416
17.129
17.794
12.949
16.543
13,192
16,023
17.125
15,301
16.505
17.763
12.325
5.895
21.149
14.487
37.5%
7,586
7.454
18483
17.120
14.529
17:067
13,008
17.133
13:422
16.114
17.137
15,405
16.158
17.684
12.302
9.107
25.217
15.115
50%
7.949
7.777
19:251
17.106
14.641
17.065
13,804
17.134
17.041
13.356
16.350
13.531
16.174
17.097
15,469
16.555
17.690
12.388
9.517
25.540
15.376
62.5%
9.345
9,277
19555
17,055
14.998
17.069
14.171
17.123
19.525
13,672
16.76
14.056
16.330
17.143
15.714
15.154
17.764
11.368
10.435
24.127
15,522
75%
159
14.542
19 908
17,008
15.500
17.08 09083
15.049
17.168
18.549
14.751
16 697
14,003
16.641
17.179
16. 16.141
15.981
17.736
14,005
15,575
26. 1.6
16.855
57.5%
900
15.834
16.627
17.338
17.626
16,096
100%
20.876
18,556
20.782
17.117
17.124
17.125
17.131
17.112
17.277
17.124
174414
17.117
17.399
17.204
17.351
13.325
17.53
17.476
19.055
46.866
19.340
(d) Model performance of RD experiment under RMSE on Concrete dataset.
Degree
CATB
DAN
MLP
NOD0
RCS
SNK
TTP
TABR
NCA
LMA
5.003
5.33
3,936
17.102
14.138
17.054
12.976
17.138
12.814
15.955
17.190
15:310
16.088
17,699
12.451
7.374
33.711
12.5%
7.900
7.636
6.931
13.105
14.527
17.065
13.571
17.139
18.405
13.113
162288
13:226
16.130
17.162
15.481
16,63
17,600
12.815
9.437
9.514
13.586
25%
10.317
9.764
9.484
17.108
14.944
17078
14.368
17.133
18:365
13,771
163993
13.857
16.308
17.172
15.67
16.958
17.81
12.505
11:340
16.413
14.547
12.435
11,604
11:722
17.105
15.294
14.701
17.128
14,356
16.47
15:918
17.405
17.670
13:979
15.210
15:250
50%
14.345
13,214
13.729
17.107
15.673
17.093
15,208
17.124
18.407
14.855
16.73
14.959
16.653
17.183
16.163
17.671
17.698
14.382
14,250
15.029
15,892
62.5%
16. 110
14.712
15.360
109
16.053
11100
15.759
17.120
18:766
15.336
16.99
15:552
16.831
17.187
16.409
17.834
17.645
15:251
15.620
16.683
16.455
15%
17.770
16.077
17:209
17.112
16.415
17.109
16.251
17.115
17.969
16,080
17.173
16.118
17,009
17.194
16,655
17.864
17621
16.156
16,928
25.455
17.267
19.353
17,352
18814
17.114
16.771
17.117
16718
17.113
16.644
17.200
16.000
17.657
17.577
17.99]
100%
20.576
18.356
20.782
17.117
17.124
17.125
17.131
17.112
17.277
17.124
17.414
17.117
17.399
17.204
17.151
19.325
17.538
17.475
19.085
46.866
19.240
(c) Model performance gap A of RD experiment under RMSE on Conerete dataset.
Degree
LGB
XGB
CATB
DAN
MLP
NODE
RES
SWI
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
LMA
AVE
12.5%
0.579
0.515
0.761
0.000
0.027
0.001
0.046
0.000
0.015
0.041
0.009
0.032
0.011
0.001
0.015
0.036
-0.001
0.029
0.250
-0.718
-0.035
25%
1.062
0.910
1.109
0.000
0.057
0.001
0.100
0.000
0.013
0.093
0.017
0.081
0.023
0.001
0.031
0.055
-0.001
0.036
0.538
-0.513
0.032
37.5%
1.485
1.306
1.978
0,000
0.082
0.002
0.133
00.01
0.017
0.105
0.029
0.120
0.033
0.002
0.047
0.082
-0.002
0.123
0.734
-0.608
0.060
50%
1.867
1.630
2/183
0,000
0.109
0.002
0.174
-0.001
0.015
0.166
0.010
0.167
0.011
0.002
0.063
0.098
-0.002
0.195
0.932
-0.551
0.105
62.5%
7.220
1.923
2,953
0,000
0.135
0.033
0.215
-0.01
0.007
0.217
0.052
0.144
0.055
0.002
0,079
0.109
40.003
0.225
1.118
-0.505
0.144
75%
2.552
2.191
3.382
0,001
0.161
0.003
0.252
-0.001
-0.009
0.277
0.065
0.258
0.066
0.003
0.095
0.110
+0.004
0.298
1.295
-0.215
0.207
87.5%
2868
2,448
3.779
0,001
0,186
0.004
0.288
-0.00
-0,026
0,311
0.088
0.299
0.077
0.003
0.111
0.099
-0.007
0,381
1.471
-0.430
0.216
100%
3,172
2.687
4.279
0.001
0.211
0.004
40002
-0.047
0.359
0.000
0.336
0.003
0.128
0.077
-0.009
0.404
1.588
0.390
0.338
63

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Table 25. Model performance on Laptop datasct.
(a) Model performance of SC experiment under R.S... on Laptop dataset.
Colarn
PCC
14.08
XLII
LXIB
BAN
Mi.
5W1
NE.1
0225
1N1
D1.N
F11
0200
3.41
SSN
110
148K
NCA
L.A
AVE
171.955
276 276102
255.
718433
554,012
114.358
475.469
168
764.760
473 478.540
656,438
507.516
210
705.667
671.807
655
509:358
433
525-31
368.655
551.156
01862
381 552
265 477
353.334
704354
555.00 105
316.412
472471
735 109
8311431
4911571
633
515.005
677.164
131.531
677.616
665 121
4131146
407.000
515 -9.
310.99
587 247.448
0/67
265.064
278 629
255.00
710
554.799
714.373
477.734
725
746.665
481.354
654.233
500.30 306
676.95
106.643
071.554
074.010
710019
J00 564
325.00
355.375
551 110
0.125
700 856
301
R55
264.50
710 154
553
633
714.455
475.60
1351131
753.568
479 137
656.171
510.054
677 522
105.909
072.55 506
688
710.351
086 176
355 961
354.255
553 900
4.125
200
302
616
275.125
219.311
351385
114-400
473.13
725
750003
470186
658.375
500.251
077 m6
191161
675 269
659.23
100.192
401430
543906
341/330
554 141
0.137
789 759
760
917
267.554
719 191
555
711
676 297
515
087 796
656
101
678
170
196 100
133/25
411/90
457 700
40142
206.553
27
255
2354477
719.240
354005
734-436
478.59%
735-22
756.299
177117
658.770
507.794
077
705.433
675-711
609.077
211480
403115
222
243-22
553.55 553.992
172.41
222 (4)
155.13
1104451
114.490
47.565
135
366
159.57
251
655.55 59
510.947
072
105.774
072-37
650115
505.755
555.00
156.55
$36.35
500.775
9
0.180
271.111
275
151255
719
116.453
677-155
072355
155
402
06218-
651.900
116
116.953
677.555
105.011
673954
110.555
de 19
-55.77
58% 131
10
02212
100525
541
116851
180104
080101
104.11
482
655381
5351150
675 mm
-04.91
676-21
671313
511234
02211
55:36
564 WWW
11
0.251
353.55
278
555871
T10614
555
T14.330
(72391
115611
194.71
250.77L
655.500
511.375
675
108.518
072.047
663.09
110170
50..14
111:53
568355 0035511000
554 395
12
0.26
276229
510
254231
1
551.225
714-111
-2/229
115299
480 25
511.355
672.90
201230
605511
623110
244:29
13
0.91
271.28
119655
557 94
T11.771
225015
780251
677 063
150:359
07 266
206774
071.66
755-774
173111
111 01
115571
14
6.27
171.255
22622
255.22
717440
714-22
015299
67%
652402
511-224
03.24
201410
676.344
215:35
02211
222-28
302372
15
711-111
11113
018-79
112-945
201
-24.04
16
18
0-114
299,00
301
985
115451
718.45
563/05
364319
315-177
116.99
121511
611/09
111-129
516.45
516.288
N8
130226
1631077
56.56
91h
19
0.504
119,04
331919
314,315
71:111
551.50
714.778
755.442
155.43
483 351
658.113
516
683 714
101
662.337
668 756
110440
653.00
341.00
365.312
50
20
0,590
178.114
244 105
266/90
719300
575.40 4m
714.885
06..40
725
71844)
301 549
635-635
554.361
661 154
196/48
676.55
654.36
112:842
200.77
344916
162.021
587 194
71
8:593
372.967
741 973
764.07
719.50
551816
111.93
098 666
735 60
718.991
891812
658.55 090
550 355
683
676
677 464
705
505 231
545 750
478.478
961 196
119 804
333
141.75
719 (55
575 909
714.347
535
755
8M 316
400
653.117
553.376
60 731
713.587
061 581
A91 6M
300-931
400 167
410 9M
196.97
477
(b) Model performance of MC-M experiment under RMSE on Laptop dataset.
Degrve
CATR
IAN
ML.P
ND01
R115
SW1
NIT
GBS
INT
DCN
FTT
F100
5XT
5XN
TTV
TABR
NCA
1.M
AVE
0%
221.935
218.929
255 116
719-433
554111
7:4.19%
415.40 469
725.158
754.
478,341
635.838
517.61
631:200
110 361
631.3171
560.30 306
700:99
494.
822
513-2005 411
148-000
551.135
329.156
710
7:4.757
590.29
736-331
754.324
525.991
674.550
$71.35
688.221
712.85
705.275
419/023
341 596
575.444
-15.443
18.15%
597 1
571 014448800000
544 000
419.543
251 617
0192355
71947
139235
628667
656 716
716.119
596 716.116
7:5.33
471 09
725.424
734 532
354
718.578
576.955
900115
67% 312
650.054
602.111
500 490-000
659.579
667
716
71: 905
696.056
706.592
607 107
100 696
713 335
712.574
490.00 092.015
400.50
502
303.134
451 511
105 114
400344
005.045
511
431
736
444 19
21 11855%
640
N77 9822
725.455
750 151
907.33
107 107.31
614777
305.55
715.206
777
713.478
089 531
500.374
-
27.295
2M1 A21
$11
1922
219091
718-902
604 201
223:247
613.33
711 11
622:205
717.711
723.394
208.053
713.231
482 724
611:250
651 651
31.80%
201 537
533 1888
371 133
715811
663055
716.959
(46.5 573
225.671
720.50
634.777
714 365
618.558
711.5%4
T16.990
730.220
771 224
716.305
156775
637
637-564
500 100
645-764
36.15/
222 -24
433
222-00
713-116
188.016
223.005
722
032:244
214-355
214,342
712.40
711.014
005.272
715-213
466 22:
630-329
602009
50.00 40.00%
45.45%
533.90
07167
R10060
412465
001154
7:3.45
110
673.40
66
-14.450
721-11
154
111
11
131 055
718
512.55
626
215.491
516
715.00
116014
231410
110.008
156.75
100.004
103
717.475
712-333
13..50
150 625
406
116
195
655
579
54.5 54.55%
547.079
656 655.140
221559
015225
(9815)
715.46
661.70 1998080
725.230
(14.073
685115
132 132:227
(5145)
117.221
1
718
740.132
500100
714.213
522 045
671.3
071.39
514.544
59,00%
613
657.52
251.290
0
515579
714715
711345
T681
045
725.750
69
334.50 07
000 001.55
719.14
130
743.55
102335
115.96
510.04
304
731.900
554
(60.88
213111
100
725/51
211-200
155 22
210
TE530
241.22
100
713/445
231A11
551861
651.57
12129
16115
207.111
682-211
300 233
101220
215800
1
211850
112:54
020
112:01
725
771
201155
22
255
201.00
221-158
221/266
720
716.65
144331
145100
706 126
713-450
2 220
52 083
222-000
236-571
2 201-100
002011
683 005
715-21
166 11
723-111
201
20174
177
715-85
300 20
232-03
17.100
86.10%
703
113
100
721.977
735 115
191
751.778
751.008
771316
51
713-520
508
76-162
100
711,00
724092
95.45%
107%
731
813,542
$55.50 155
(22.765
718.444
718.16
71244
117103
718.544
7:30089
115
118
095
725.779
734600
116.61
730.96
115.335
139.144
357
754000
716.44
716812
722.459
732
721
221999
751.600
751.146
758 558
725.00
725.391
671.7L
755 945
262911
7640334
$40.592
500.25
741.22
735.055
(c) Model performance of MC-L experiment under RMSE on Laptop dataset.
Diggree
LX38
XL.8
CAFH
BIN
ML.F
1415
SW1
Nitr
GLU5
INI
DKN
1/11
1,80
SA1
55NN
F11
1AMM
NCA
LMA
AVE
0%
271.935
278.920
255 315.16
719.433
554.012
7.4.355
475.45 A47
725.158
764. 700
471.342
656.858
507.616
637.260
105.267
671.807
663.236
709.985
404.823
325-431
368,688
531.135
/51.135
0.00
1.50%
791
305
257.935
247.477
561
2613254
719.22
719 104
556647
205
7:4.112
7.4.397
472 871
235 560
725 010
746.073
433001
000.311
657857
654
515.291
511193
676962
607.161
708.00 103
100
673.656
631.616
673 660
065 131
710.126
710.14
531
33442
515
334
115-005
355.2
1111
557/244
561
27.19% 17.2%%
52 12.59
364
405
565 951
665
310
331-100
373.053
018851
110012
719173
1
554354
714.11
714.534
7.4.447
450
450 717
1915
725.34)
055 631
225.343
735.75
717.50
131
107.30 00
447
-47.334
257 097
451.664
455
654
653
172
519.86
$17.377
517.55 517.50
658.50 136
708 657.30
55
710 559
108 019
651.55
610
116
120.300
600.11
1000
710.471
719
711332
-
5TL 131133
553
354000
535-200
-
451372
564
56:
1
11 8300
36.20%
417.641
816
15h
1
<77
211
1
9
561.777
400 m1
114.556
711
14.8664000
486
486.21
725.555
234 106
T00
111 12
T00.000
10100000000000
200
03 155
00:541:5550
055 576
6524533
431 67%
5230024
639,792
033 50
T11 316
115 200
693,600
681 157
722:923
113
502 705
156
271-122
A00
45510
585544
111
40.91%
425967
#00
#02:100
56:511
113840
36110
103 714.755
480
576
-
TOO 600
291
550224
53:553
71
613/18
506
506.511
2 724/055
557 266
372.500
+31 25
387
45. 4%%
7:4-44
211-04
099
057-00
261 M89
1812216
711 221
085-25
29- 605
50,00%
59 00
5384%
461276
657-911
534.61
444
1
-10036
715858
561
580000
572273
715058
715.478
111
15-360
106 006
-26643
T0U
S19A11
202
901-130
515
050 650.966
649
648
107
571 541
041.44
663.44
654.661
714
116.212
116 505
692.910
687.566
690.117
566.915
650-144
615.46
74--244
400
166/941
442.54
157360
157511
442514
000043
615.65
614261
569
5A 196
607400
$74 100
110100
115
CTT.M 160
111111
569 074
mM 761
687001
712
K96 121
617001
17.00
761
405
606166
115006
665173
530
119617
005 1
616065
116006
315.05
558.44 145
195770
224.111
726.611
10 35
190000
671 333
S11.90
866 112
614mL
655,019
589-166
10
707.05
701 19
106.10
591
962
591
2M
710.113
723-801
725
315724
555.30
641693
(21
81822
8182%
035
216255
6313
415411
022-12
227-111
68%
108 254
31:
M0:30
977
677169
717-166
719.22
716.667
716
(00 334
731.441
100 28
222
785 551
646.33
706311
006
715
721-224
012200
15:555
086-276
655.871
140 140.00
455
118-151
626-21
711.00
361
724-230
550
715.441
672381
716 596
107 550
2338/2
191201
18L 00000
615-244
001.74
1005
753004
$56
774
718 164
11700
718.069
T18 618
774
150
718
537
732
751.54 144
TM 9M
731
T15 905
610
64

---

TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
(d) Model performance of RD experiment under RMSE on Laptop datasct.
Degree
1.28
CATR
BAN
NOFR
SW1
NIT
GIS
INT
DCN
FTT
(180
SNN
1TF
YARI
NCA
1MA
AVE
4.58%
358
363
719 364
555
714.412
871
810
680.57
657.657
515.09
637.761
707 531
613.6 636
18:344
370 3T0.61
557
9.99%
505.314
267.955
S01.722
239.22
556.643
714.397
480 264
725.810
146.872
433951
654 251
515.292
676,92
105.00 10
673.656
673.665
718.22
521 940
334,42
355.977
5e1 211
13.64%
564.353
310.210
573.053
719.75
556.1 190
714.447
400.705
225.755
717.599
494.254
652,70
513,373
657.20
708.17
634.2%6
100.17
710.77
552843
331.534
366.501
563 663.991
18.00 186
196.90 901
311 mm
401
119013
696 19
7-4.45
779 637
725.01 017
091981
656,066
710 73
637-766
700 10
527 007
555.75 776
18 331
568 193
22.759
205.751
241 700
+52
715.54
558 298
7:4.516
400.727
725
388
487081
655.56
517.230
678.456
710.25
679.406
620 222
710.474
544 064
355.200
400 30
574-249
21.277
805-285
168-800
224
218-51
714.314
485
225 263
403
654
509390
605.775
081.126 176
100
712-172
371
524
421 222
31,85%
416.199
237
715.54
725.705
713.50
499
655.576
678,500
706 760
683
600
712-520
556
425
23
40.91%
435
850
506
715.540
506 101
714.755
575
136.134
091.315
65n
536.55
650.000
715000
683.418
341
714.056
575.550
151
581
15.4%5 1.5%
617 360
490.0
718400
504 5014
714.349
405 141
734.764
The 10..00
090 1.00
447.066
561
681.16
711 43
686.22
615811
713-661
010
494
50.00%
443,900
444.300
563021
215.838
573
715.085
406.056
724655
TOL:45
502.135
648.63
$11.543
612.46
714.31
657.506
615.545
714.334
415.451
457.562
001
54 1M%
109-10 10
219016
400 261
715
110 831
726,953
596 140
511161
650
T15689
689.019
71
05:144
610411
59,099
692.276
504.001
219.200
582524
715,354
511.506
725.519
655
315341
581.222
683.547
716.50
690.447
182.714
714:119
307
455
437.11
015.599
462222
201
216254
212-415
224944
211 221
2014 29
716 291
091.410
766 V13
211-251
452211
611251
46.159
002-20
524
21
715.516
722/21
669
(03
687.001
712.29
694.750
500 -21
766-10
211
447013
017.05
73775
66--26
335
60)
21K.99
715.66
150 320
734-21
535
665
607.03
709
19
716-221
451.111
555.59
(21.9)
81.8%%
566055
831
696651
51
631
716.119
190 161
159
513.51
488
659
100 36
111
153 6M
647-917
661
86.36%
997.169
117.468
119329
116.267
663.94
716.867
124
731.241
59..44
505.551
646,33
X6.111
005
725.940
641012
721.234
15% 100
011.30
152.56
085.225
90911
670001
749.90 908
7711100
716.751
06 m33
717.377
635 6/9
775
T00 231
678A85
111541
677
718 190
18 91
809 M1
716.112
161 951
631.341
257.661
007 574
95.45%
600,772
764.171
723.157
715.275
686094
733.900
654 353
725.337
702 256
063033
733.921
081 897
716
007 550
735.519
(58.58
724.987
T64 100
075871
760
707.785
107%
751034
555
718.166
717104
738.069
118
730
715.355
337 50
716
733
V70
753 144
730 23
733.08
9
2640334
$49.92
741.225
(e) Model performance gap A of RD experiment under RMSE on Laptop dataset.
Degree
LGBB
XGB
CATB
DAN
MLP
.ODE
RES
SWI
NET
GOS
INT
DCN
FTT
GRO
SAT
SNN
TTF
TABR
NCA
L.MA
AVE
4.55%
0.003
0.061
0.075
0,000
0,011
0.000
0.019
0.000
0.003
0,000
0.004
0.017
0.003
0.001
0.006
0.004
0
0.000
0.053
.0.342
-0.001
9.09%
0.128
0.129
0.139
0.000
0.024
0.001
0.049
0.000
0.003
0.825
0.011
0.045
0.007
0.004
0.013
0.016
0.001
.0.007
0.137
.0.300
0.013
13.64%
0.195
0.204
0.221
0.000
0.024
0.001
0.059
0.000
-0.02
0.016
0.005
0.845
0.006
0.002
0.013
-0.002
0.002
0.010
0.111
-0.194
0.018
18.10 18%
0.261
0.281
0.314
0.000
0.057
0.001
0.112
0.001
-0.13
0.061
0.018
0.087
0.015
0.066
0.029
0.031
0.04
40011
0.305
-0.09
0.067
22. 795
0.35
.0.001
0.086
0.001
0.097
0.011
-0.07
0.021
0.022
0.097
0016
0004
0.91
0,33
0.03
40013
0.887
-0084
0.051
27.27%
0,468
0.000
0.076
0.002
0.146
0.001
0,086
0:029
0.123
0.021
0.037
0,000
0.003
0.026
-0.006
0.069
31.82%
0,533
0,517
0.078
0.002
0.000
0,115
0.024
0.150
0.020
0.004
00000
0.002
0.075
36 36%
0.641
.0.01
0.116
0.002
0.000
0,135
0.035
0.180
0.024
0.050
0,007
0012
0.230
0.097
40.91%
0.124
0.002
0,002
0.047
0.181
0.056
0.061
0.196
0.116
727
0.001
0.037
0.000
50.00%
0.985
0.951
0.134
0.003
0.149
54.55%
0.001
0.139
0.003
0.247
0.046
0.238
0.156
59, 39.09%
0 0.922
0.001
0.161
0.003
0.049
0.272
0.070
0.067
0.762
0.545
0.176
63. 6..644
1,032
1.265
00001
0.187
0,004
0.002
0.264
0,070
0.285
0.045
0.083
0.050
0.010
0.161
0.815
0.543
0.192
68. 189
1.128
-0.001
0.187
0,004
0.002
-0.075
0.255
0.068
0.307
0.044
0.017
0.084
0.066
0.009
0.878
0.666
0.211
72. 731
1.174
1,380
-0.001
0.207
0.004
0.363
0.001
-0.075
0.356
0.054
0.386
0.099
0016
0.097
0.047
0.010
0.293
1,029
0.674
0.232
77.27%
1.335
1,632
1.384
00.001
0.223
0,004
0.394
0.003
-0.066
0.390
0.079
0.345
0.054
0.188
0.099
0.099
0.006
0.276
1,071
0,702
0.252
81.82%
1,302
1.675
1,506
.0.001
0.239
0,004
0.415
0.001
-0.668
0.392
0,098
0.356
0,056
0.18
0.105
0.079
0.009
0.338
1.115
0.706
0.264
86.36%
1.433
1,739
1,586
.0.001
0,243
0,004
0.427
0.001
.0.644
03422
0.085
0.365
0.058
0.19
0.105
0.052
0,013
0.434
1.125
0.830
0.280
90.918
1.500
1,550
1.699
.0.001
0.365
0,005
0.453
0.003
40,65
03442
6.108
0.391
0.062
0019
0113
0.055
0,016
0.424
1,263
0.879
0.302
95 45.4
1.611
1.994
1.786
-0.001
0.240
0.005
0.487
0.003
40,60
0,677
0.35
0:401
0.06
0021
0.119
0.075
0.013
0.474
1,330
0.899
0.318
100%
1.696
2.104
1,859
-0.001
0.294
0.005
0,511
0.005
-0.058
0.502
0.125
0.412
0.067
0.021
0,121
0.083
0.019
0.507
1,409
1.304
0.347
65


Reviews:
[{"review_id": "BjXGiBo2dZ", "reviewer": "Reviewer_Q7p8", "summary": "This paper proposes a new benchmark “TabFSBench” for tabular data learning, especially for the feature shift in open environments. Feature shift means the feature could be decrement or increment. The authors select open-source datasets from OpenML and Kaggle’s dataset library, including three curated tasks of binary classification, multi-class classification, and regression, covering various domains such as finance, healthcare and geology. Four feature-shift scenarios including single shift, most/least relevant shift, and random shift. Some insightful empirical analysis are provided. Moreover, the paper provides callable APIs.\n\n## update after rebuttal\nMy problem has been solved, and I decided to keep my score.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Q7p8:\n\nThank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.\n\n**Q1: I hope the authors can release a leaderboard for the tabular data learning methods on this benchmark.**\n\nWe have currently implemented a leaderboard for TabFSBench and incorporated additional evaluation results for TabPFN v2. This **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/Leaderboard%20_%20TabFSBench.pdf)** provides access to the TabFSBench homepage we designed. Upon completion of the paper review process, public access will be granted. Furthermore, we will continuously update the benchmark with the evaluation results of new models. Researchers are also cordially invited to contribute their own evaluation results on additional datasets or models. Additionally, we'll be expanding TabFSBench in the future from below two parts:\n\n***1. For the dataset selected in TabFSbBench.***\n\nWe selected 12 datasets from Grinsztajn[1] and TabZilla[2]. They exhibit substantial heterogeneity in scale, domain, and task characteristics, deliberately encompassing diverse potential scenarios of feature shift. To comprehensively evaluate the challenges posed by feature shifts, we have implemented four distinct experimental configurations that collectively enhance both the breadth and depth of our benchmark assessment. Multiple experimental repetitions were conducted to mitigate stochastic variability.\n\nDue to rebuttal time constraints, we were unable to review the large table benchmark in a short period of time. We will incorporate evaluation results based on the Grinsztajn benchmark into the final version.\n\n***2. For the future work for TabFSBench.***\n\n  - We have established and maintained a project homepage and a comprehensive rank leaderboard. We continuously update the performance evaluation results of various newly released models or datasets (for example, the benchmark test data of the recently included TabPFN v2 model have been incorporated), ensuring that the research community can access the latest evaluation information in a timely manner. This homepage will be made open source in the subsequent phase.\n\n  - To enhance the scalability of cross-model comparison research, TabFSBench has been specifically designed with a feature-shift dataset export module. Through command-line parameter settings, this module can export datasets under various feature-shift scenarios, thereby effectively supporting researchers in conducting evaluations on their own models (for specific implementation details, please refer to our response to Reviewer 8NVr Q4).\n\n  - In the anonymized code repository provided in Supplementary Materials, TabFSBench has implemented a plug-and-play interface for importing new datasets and new models. This enables researchers to conveniently test their custom datasets or models within the TabFSBench framework (for specific implementation details, please refer to our response to Reviewer 8NVr Q4).\n\n  - We encourage and actively accept researchers to provide us with their own evaluation results. Subsequently, we will also be committed to constructing feature-shift datasets with practical significance and opening up both public and private evaluation rank leaderboards. This will further promote the research and evaluation of the feature-shift scenario within the community (for other relevant content, please refer to our response to Reviewer WumN Q1/3).\n\n**Q2: In the paper, the benchmark doesn’t consider the feature increment.**\n\nSection 2.2 of the paper indicates that this study primarily focuses on the impact of feature shifts on model performance. However, tabular machine learning models evaluated in this work are inherently incapable of handling newly added features in feature-increment scenarios, as they require consistent input and output dimensions. Consequently, in such scenarios, these models automatically disregard the new features while maintaining their original performance.\n\nIn the future work, we plan to extend the evaluation scope of TabFSBench by incorporating additional datasets and model evaluations. Specifically, we will include specialized models designed for feature increment scenarios (e.g., [3][4]) to assess their performance improvements under such conditions.\n\n[1] Léo, Grinsztajn, et al. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 2022: 507-520.\n\n[2] McElfresh, Duncan, et al. When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems, 2023: 76336-76369.\n\n[3]Zhang, Zhen-Yu, et al. Learning with feature and distribution evolvable streams. International Conference on Machine Learning, 2020.\n\n[4]You, Dianlong, et al. Online learning for data streams with incomplete features and labels. IEEE Transactions on Knowledge and Data Engineering, 2024.", "claims_and_evidence": "Yes, the claims are well supported.", "methods_and_evaluation_criteria": "Yes, the paper evaluates the performance of different models with ROC-AUC and RMSE, which is reasonable for the evaluation.", "theoretical_claims": "The paper doesn’t include theoretical results, but the empirical analysis from the results are reasonable and insightful.", "experimental_designs_or_analyses": "Yes, I have checked the experimental design and analyses. The authors conducted extensive to evaluate the performance of different methods on the feature shift scenarios. The compared methods include both tree-based methods, deep learning methods, LLM methods, tabular LLM methods. The experimental design is sound and the empirical analyses are insightful, pointing the challenges that current methods remain in the feature shift scenarios. I believe these results could help promote related research.", "relation_to_broader_scientific_literature": "This paper is related to the field of tabular data learning, especially in the benchmark for tabular learning. Different from previous methods or benchmarks, this paper is the first to study the tabular data learning under feature shift. The authors present a definition about feature shift in tabular data learning. This problem is important and unique to the tabular data (For images or text, there is no feature decrement or increment). Therefore, it is important to construct a benchmark to help evaluate the performance of tabular data learning methods under the feature shift problem. The benchmark could promote related research.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "It is good to see that the authors provided easy-tool-use API, I believe this could help related researchers. I hope the authors could release a leaderboard for the tabular data learning methods on this benchmark.", "other_comments_or_suggestions": "No.", "questions_for_authors": "In the paper, the benchmark doesn’t consider the feature increment. I think it is also a promising direction to study how to better exploit the new feature information.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYeMzGJF1Z", "reviewer": "Reviewer_8NVr", "summary": "# Summary\nThis paper proposes a benchmark to study \"feature shift\" in the tabular data setting -- where the set of features (i.e. columns) available to the model change at test time vs. training time. The authors identify a set of 12 datasets (four binary classification/multiclass classification/regression datasets), along with different shift settings (random, most/least relevant, single shift), and conduct a set of benchmarking experiments across these datasets, highlighting some empirical findings.\n\nOverall, this is a much-needed direction of high relevance to the tabular data research + practitioner community. However, I have some concerns about design decisions in the study, which seem to make it difficult to rely on the comclusions. At the very least, it is critical to see empirical validation that the authors' decisions do not overly bias the results. Additionally, the benchmark seems fairly small, and the empirical results are noisy, which can make the results feel somewhat unreliable. Some clarification is also needed on various details.\n\n# Major comments\n\n* In some sense, each \"table\" is a data point in this study. As such, the authors have 12 data points (four binary classification/multiclass classification/regression datasets), which makes it difficult to rely on the noisy results without clear trends (both Figure 3 and Figure 4 show data that clearly do *not* conform to the line of best fit drawn on the plot). The benchmark would feel much more reliable if it was comprised of more tables, or if the authors were at least able to better explain the results (for example, why are there multiple, apparent \"trajectories\" or lines in Figure 3?). I would suggest that the authors incorporate more high-quality tables from existing benchmarks, including some of those referenced in the paper (such as the Grinsztajn benchmark, TabZilla https://arxiv.org/abs/2305.02997, or the UniPredict suite https://huggingface.co/datasets/mlfoundations/tabula-8b-eval-suite/tree/main/unipredict).\n\n* API design: most users won't want to re-run your experiments (shown in \"Example Command\" -- this should be a latex code listing). Instead, they will want to get the *data*, and run their own algorithms on it. They may also want to contribute new data to the bechmark, and run your implementations against this. Please comment on how this is supported, and consider adding this to your documentation.\n\n* The authors make a significant design decision -- perfoming **mean-value imputation with missing features** -- which seems likely to have a major impact on the study's results. In particular, **replacing** values in the existing columns, after training on the true values, is not the same as \"removing\" the column. It revalues that column, and changes the model's predictions to p(x_j{j\\neq 1}|x1=x1) for some imputed feature x1. In this case, it seems an almost trivial observation that \"removing\" (imputing) important features leads to a linear drop, as a model is likely to learn to rely on these important features, and then mean-inputation actually distorts the distribution of X along that dimension, leading to an unsurpising performance drop. If the authors want to demonstrate that this is a reasonable design decision, I would suggest (1) providing evidence from other studies that this is a realistic and reliable method for feature \"removal\", and (2) show that the results of the study are robust to this decisino (for example, by performing other forms of removal or imputation and demonstrating that the key claims still hold). I, however, am extremely skeptical of the results without such validation.\n\n* Many methods used in this study are capable of natively modeling missing data. However, the authors do not investigate this decision, despite the fact that any practitioner would likely use this in practice if the schema was known to change. Please also include variants of models where they are trained with missing data (at train time, for example, by randomly masking x% of cells to missing) and then the target column is set to missing at test time.\n\n* It would be helpful to provide \"oracle\" measures of performance on the shifted data, by training a model directly on the shifted data and evaluating on a held-out performance. This would help understand how much of the perofrmance degradation is due to the information loss (an irredicible source of error), vs. how much is due to lack of robustness (it is this latter form of drop that the authors seem interested in minimizing/understanding).\n\n# Minor comments\n\n* I think the name \"feature shift\" is not an accurate descriptor of the setting being studied, and I am not aware of this terminology being used in prior tabular studies. Perhaps \"column shift\" or \"schema shift\" would be better names.\n\n* Details of the Pearson correlation method (L168) belong in the main text, not in the supplementary.\n\n* Why do the authors use a \"sampling with replacement\" approach to single shift, instead of simply removing one feature randomly? Please explain or revise.\n\n* While the hyperparameter grids are given, the HPO method is not - please clarify how hyperparameters are tuned.\n\n* The design of iteratively removing important/unimportant features is also in the Gardner et al. 2024 paper.\n\n# Typos ec\n\n* Abstract: \"Python codes\" --> Python code\n* L079: \"feature-shift challenge\" --> missing period.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8NVr:\n\nThank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.\n\n**Q1: It is difficult to rely on noisy results without clear trends.**\n\nWe would like to clarify that |$\\rho$| greater than 0.7 can be considered as a moderate linear correlation[1]. Tableshift[2] figure 5 also draws a conclusion of linear correlation based on $\\rho$ value of 0.7. In our paper, $\\rho$ value in Figure 3 is 0.7405, and $\\rho$ value in Figure 4 is 0.6. Therefore, despite the presence of noisy data, relevant conclusions can still be drawn based on $\\rho$ values. We will add the evaluation results of more datasets to mitigate the noise.\n\n**Q2: why are there multiple trajectories in Figure 3?**\n\nThe reason for multiple trajectories in Figure 3 is that each trajectory consists of points derived from the same dataset. During the fitting process, the results of all datasets were combined, resulting in each dataset forming a distinct trajectory. It is worth noting that these trajectories reflect the relationship between feature importance and model performance across different datasets. Specifically, each dataset's trajectory demonstrates how its specific feature importance influences the model's performance.\n\nTo better support our conclusions, this **[ link](https://anonymous.4open.science/r/icmlrebuttal-55BD/trajectories.pdf)** provides correlation and accuracy plots for 12 different datasets. Below are $\\rho$ values for the relationship between correlation and accuracy for each of the 12 datasets.\n|Dataset|Credit|Electricity|Heart|Miniboone|Eyemovements|Iris|Penguins|Jannis|Abalone|Bike|Concrete|Laptop|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|$\\rho$|0.8597|0.8939|0.9244|0.9482|0.9796|0.9646|0.9377|0.9891|0.6866|0.9656|0.8360|0.9896|\n\nIt indicates that although levels of feature importance vary across different datasets, they all support our research conclusion, namely, that there is a significant linear correlation between feature importance and model performance. These trajectories further confirm the generality and reliability of our conclusion.\n\n**Q3: Incorporate more high-quality tables.**\n\nPlease refer to our response to Reviewer B7Cf Q4.\n\n**Q4: API design**\n\nWe have already provided the --export_dataset option in our paper. Users can set this option to True to export specific versions of the datasets they wish to evaluate (e.g., single-column missing, missing to a certain extent, and all possible missing scenarios).\n\nThe README.md in Supplementary Material already explains how users can add new datasets and new models. We will further elaborate on the code functions in the final version.\n\n**Q5: Major comment 3&4.**\n\n1. Reviewer B7Cf has mentioned that \"The mean-value imputation strategy used is a common approach and therefore is a valid choice.\" We also found that benchmark datasets such as Talent[3] also use mean imputation as a method for handling missing values.\n\n2. We compared the performance of various models using their own imputation methods, random imputation, and mean imputation. We present the average performance of several representative models under different imputation methods.\n|CatBoost|NAN|Mean|Random|\n|-|-|-|-|\n|0%|0.845|0.851|0.818|\n|9%|0.820|0.826|0.807|\n|18%|0.792|0.801|0.767|\n|27%|0.763|0.775|0.739|\n|36%|0.733|0.747|0.702|\n|45%|0.702|0.716|0.665|\n|54%|0.670|0.682|0.630|\n|63%|0.637|0.643|0.608|\n|72%|0.600|0.596|0.574|\n|81%|0.538|0.541|0.502|\n\nOur experiments show that the model performs best under mean imputation, and its performance declines under all three types of imputation methods, indicating that the model still faces challenges from feature shift. Detailed data are presented in this **[ link](https://anonymous.4open.science/r/icmlrebuttal-55BD/imputation.pdf) and we strongly recommend that you view this link.**\n\n**Q6: Major comment 5.**\n\nPlease refer to our response to Reviewer WumN Q2.\n\n**Q7: Minor comment 1&2 & typos ec.**\n\nWe will make revisions in the final version.\n\n**Q8: Minor comment 3.**\n\nThe purpose of the single-shift experiment is to test the impact of the absence of features with different importance levels on model performance. \"Removing one feature randomly\" is the experimental setting for a feature shift degree of $\\frac{1}{n}$ (n = number of columns) in our random shift experiment.\n\n**Q9: Minor comment 4.**\n\nPlease refer to our response to Reviewer wA1o Q3.\n\n**Q10: Minor comment 5.**\n\nWe have already cited the paper you mentioned on lines 683-684 of Section A.3 on page 13 of our submitted paper.\n\n[1] Iversen, Gudmund R, et al. Statistics: The conceptual approach. Springer Science & Business Media, 2012.\n\n[2] Gardner, Josh, et al. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 2023: 53385-53432.\n\n[3] Liu, Si-Yang, et al. TALENT: A Tabular Analytics and Learning Toolbox. arXiv preprint arXiv:2407.04057, 2024.", "claims_and_evidence": "See above.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "See above.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "See above.", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itGhcEduUE", "reviewer": "Reviewer_WumN", "summary": "This paper establishes a new benchmark for tabular data, focusing on feature shift issues, where feature increments and decrements occur between the training and testing phases. This benchmark includes three types of tasks across twelve datasets and evaluates twenty-four tabular methods. The experimental results reveal three key observations that can guide future research in this area.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WumN:\n\nThank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.\n\n**Q1: The authors should provide datasets that naturally contain feature shift issues rather than simply analyzing performance on synthetic datasets.**\n\nRegrettably, there currently exists no dataset specifically designed for feature shift, unlike Tableshift [1] which was developed for distribution shifts. However, we have preliminarily constructed a feature-shifted dataset based on the Heart dataset. Given that different features in the original dataset require distinct measurement instruments, we categorized the features into three groups: basic features, electrocardiogram (ECG) features, and exercise stress test features.\n\nIn the constructed feature-shifted Heart dataset, both the training set and the test set step 0 contain all features. However, patients in the test set step 1 lack ECG measurements, resulting in the absence of RestingECG and ST_Slope features. Similarly, patients in the test set step 2 did not undergo an exercise stress test, leading to the absence of ExerciseAngina and Oldpeak. A subset of examples is provided in this **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/feature-shift%20heart%20dataset.pdf)**.\n\nAdditionally, we evaluated CatBoost on this dataset: \n|Metric|Test Set Step 0|Test Set Step 1|Test Set Step 2|\n|------------|----------------|----------------|----------------|\n|Accuracy| 0.879| 0.658| 0.823|\n\nNote that for meaningful evaluation of feature-shifted datasets, models must be assessed under specific partitioning schemes. Applying the four experimental settings proposed in our paper would undermine the unique characteristics and practical relevance of such datasets. In the final version, we will include a comprehensive evaluation of the feature-shifted Heart dataset and will open-source the dataset for broader research use.\n\n**Q2：What is the performance upper bound for each dataset when the feature shift issue occurs?**\n\nWe evaluated the performance upper bound of the models by training directly on the shifted data and assessing their performance on a held-out test set. Due to time constraints, we did not test the performance upper bounds of LLMs and tabular LLMs. The detailed performance for all evaluated models on 9 datasets is provided in this **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/upper%20bound.pdf)**, with the average performance upper bound and feature-shift performance on the Heart dataset presented here.\n\n|Model|Average Vanilla Performance|Average upper-bound performance|\n|-|-|-|\n|LightGBM|0.700|0.707|\n|XGBoost|0.698|0.761|\n|CatBoost|0.718|0.767|\n|TabPFN|0.778|0.804|\n|DANets|0.617|0.628|\n|MLP|0.775|0.783|\n|NODE|0.599|0.613|\n|ResNet|0.796|0.808|\n|SwitchTab|0.810|0.821|\n|TabCaps|0.743|0.777|\n|TabNet|0.604|0.616|\n|TANGOS|0.790|0.794|\n|AutoInt|0.629|0.642|\n|DCNv2|0.776|0.786|\n|FT-Transformer|0.661|0.671|\n|GrowNet|0.578|0.589|\n|Saint|0.770|0.778|\n|SNN|0.718|0.729|\n|TabTransformer|0.502|0.515|\n|TabR|0.806|0.818|\n|ModernNCA|0.781|0.819|\n\nThe results demonstrate that, under the same degree of feature shift, models trained on the original dataset exhibit inferior performance compared to those trained directly on the shifted dataset. This indicates that although training on the original dataset provides more comprehensive information, feature shift significantly compromises model robustness, resulting in performance degradation relative to models trained on the shifted data. Furthermore, as the degree of feature shift continues to increase, the performance gap between original and shifted datasets also becomes progressively larger, indicating a progressively diminishing robustness of the model.\n\n**Q3：Could the authors provide some examples of feature shift issues occurring in the real world?**\n\nSection 2.2 of our paper employs forest disease monitoring as a case study to demonstrate how sensor degradation leads to a reduction in available features. As further evidenced by the designed heart dataset in Q1, incomplete medical examinations may result in missing diagnostic indicators (features) due to the absence of specific equipment.\n\nThe feature shift phenomenon also manifests prominently in financial and transportation domains:\n\n- Finance: Stock prediction models trained on comprehensive features (e.g., financial ratios, macroeconomic indicators) may encounter missing features (e.g., market sentiment indices) during real-world deployment due to unforeseen events.\n\n- Transportation: Accident prediction models relying on features like road conditions and weather data may experience partial feature absence caused by sensor failures or insufficient data collection.\n\n[1] Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 2023: 53385-53432.", "claims_and_evidence": "The authors' major claims are based on three key observations from the experiments: (1) tabular models have limited performance in handling feature-shift scenarios; (2) feature importance shows a linear trend with performance degradation; (3) performance under feature shift is related to performance in a closed environment. The authors have demonstrated these key observations through their experiments.", "methods_and_evaluation_criteria": "This is a benchmark paper, and as such, it does not propose any new method to address the feature shift problem. The evaluation in this paper is comprehensive and accurate. The authors use accuracy and ROC-AUC for classification tasks and RMSE for regression tasks. Additionally, the performance gap is illustrated to demonstrate model robustness. Extensive analyses are provided to highlight the shortcomings of existing methods and suggest future directions.", "theoretical_claims": "The paper does not present theoretical claims.", "experimental_designs_or_analyses": "The experimental design and corresponding analyses are comprehensive because this is a benchmark paper. The three key observations are fully supported by the experimental results.", "relation_to_broader_scientific_literature": "This paper can advance tabular data research towards deployment in real-world applications. Specifically, this research has revealed a weakness in existing tabular methods, that is, feature shift leads to performance degradation issues, and established a benchmark to comprehensively evaluate the performance of each method, guiding future method development.", "essential_references_not_discussed": "No essential reference is omitted.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper studies a novel and practical problem for the tabular classification and regression problems. In tabular tasks, feature failure is a common problem. With the advance of this problem, the tabular methods can better deploy in the real-world applications. \n2.\tThe experimental results and corresponding analysis in this paper is comprehensive. The analyses have revealed three observations, which can guide the following researches. \n3.\tThe authors provide a large-scale benchmark for the studied problem, containing three types of tasks and twelve datasets evaluated by four metrics. This can benefit the whole tabular community. \n\nWeakness: \n\nAlthough the studied problem and conducted experiments offer a certain contribution, the authors should provide datasets that naturally contain feature shift issues rather than simply analyzing performance on synthetic datasets.", "other_comments_or_suggestions": "Please see the strengths and weaknesses above.", "questions_for_authors": "1.\tWhat is the performance upper bound for each dataset when the feature shift issue occurs? When feature shift happens, the optimal performance of each dataset should change, and understanding this optimal performance is helpful to gauge the gap between existing methods and a perfect method.\n2.\tCould the authors provide some examples of feature shift issues occurring in the real world? This would be helpful for understanding the importance of this problem.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FkFKk57Y2b", "reviewer": "Reviewer_wA1o", "summary": "The paper introduces TabFSBench, a benchmark designed to evaluate feature shifts in tabular data. The authors argue that while distribution shifts have been extensively studied, feature shifts remain an underexplored challenge in open environments. TabFSBench includes 12 datasets, four feature-shift scenarios, and evaluates various model categories, including tree-based models, deep learning approaches, and large language models (LLMs). The key findings from the study include:\n- Most tabular models perform poorly under feature shifts.\n- The importance of shifted features correlates linearly with performance degradation.\n- Model performance in closed environments is indicative of its robustness to feature shifts.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wA1o:\n\nThank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.\n\n**Q1: Weakness&Suggestion 1**\n\nWe adopt notations used in our paper and will provide a more detailed explanation in the final version.\n\n- Distribution Shift: During the training phase and the testing phase, the feature sets remain unchanged, i.e., $C_{train} = C_{test}$. However, there are three types of distribution shifts of the samples themselves: $p(x)$, $p(y|x)$, and $p(x|y)$, where $x \\in \\mathcal{X}$ and $y \\in \\mathcal{Y}$.\n\n- Feature Shift: During the training phase and the testing phase, the feature sets change while the data distribution of samples has no shift.\n  - Increment: $C_{train} \\subseteq C_{test}$. \n  - Decrement: $C_{test} \\subseteq C_{train}$. \n\n**Q2: W&S 2.**\n\nPlease refer to our response to Reviewer Q7p8 Q1.\n\n**Q3: W&S 3.**\n\nWe implement adaptive hyperparameter optimization based on the Optuna framework and following previous studies[1], fixing the batch size at 1024 and conducting 100 independent trials through train-validation splits to prevent test set leakage, with the best-performed hyperparameters fixed during the final 15 seeds.\n\n**Q4: W&S 4**\n\nPlease refer to our response to Reviewer B7Cf Q4.\n\n**Q5: W 5**\n\nSince our paper focuses on evaluating the robustness of tabular machine learning models in feature-shift scenarios, as Reviewer Q7p8 and WumN noted, our paper is a benchmark and doesn't propose new methods. Therefore, we did not propose a new method. However, through our response to reviewer B7Cf Q5, we explored the potential of AutoFE in addressing the challenges of feature shifts.\n\n**Q6: The justification for the Pearson correlation is weak.**\n\nWe calculated Kendall's $\\tau$ coefficient among Pearson, Spearman, SHAP, and mutual information to assess the consistency of feature importance rankings. The table showed a high degree of consistency across these metrics:\n|Metric|$\\tau$|\n|-|-|\n|Pearson|0.60|\n|Spearman|0.61|\n|SHAP|0.49|\n|Mutual Information|0.53|\n\nDespite Spearman's high consistency ($\\tau$ = 0.61), We chose Pearson for its widespread use and interpretability. The minor difference in consistency between Pearson and Spearman does not affect the conclusions of the analysis. Detailed data are provided in the **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/feature%20importance.pdf)**. We will further refine related experiments in the final version.\n\n**Q7: The claim that LLMs show potential in feature shifts is not rigorously validated.**\n\nWe evaluated LLM performance in scenarios without missing value imputation. Experimental results demonstrate that LLM can not only handle feature shifts but is also more robust than tree-based models. Please refer to our response to reviewer 8NVr Q5. Additionally, we tested the performance of LLM-based AutoFE in handling feature shifts. Please refer to our response to reviewer B7Cf Q5.\n\n**Q8: Observation 2 lacks theoretical justification.**\n\nThis observation is drawn from experimental results, thereby limiting the scope of the analytical section. To further investigate this observation, we conduct a theoretical justification.\n\nA high PCC signifies a strong feature-target relation, enabling the feature to provide substantial information increment and enhance model performance. Thus, removing important features leads to a significantly greater performance degradation than removing unimportant ones. Additionally, the model depends strongly on highly correlated features during training, and their absence markedly deteriorates performance (see single-shift experiment results).\n\nWe have already pointed out the limitations regarding the impact of interactions among input features on model performance in lines 433-435 in the Conclusion of our paper. Subsequently, we will conduct an in-depth exploration of this. For additional explanations regarding this observation, please refer to our response to Reviewer 8NVr Q2.\n\n**Q9: Not cite all relevant work.**\n\nThank you for your suggestion. Although existing Heterogeneous Domain Adaptation (HeDA) methods have achieved significant progress on feature shift of images, tabular data presents a fundamentally different pattern. The inherent structure of it presents challenges when attempting to directly implement HeDA on such datasets. Moreover, our review of the literature reveals that what is often referred to as \"feature shift\" in many papers is essentially a form of distribution shift. For example, [2] regards covariate shift as feature shift. Other related works on feature shift discussed in our paper, please refer to our response to Reviewer 8NVr Q10.\n\n[1] Liu, Si-Yang, et al. TALENT: A Tabular Analytics and Learning Toolbox. arXiv preprint arXiv:2407.04057, 2024.\n\n[2] He, Huan, et al. Domain adaptation for time series under feature and label shifts. International Conference on Machine Learning, 2023.", "claims_and_evidence": "The paper claims that feature shifts significantly degrade model performance, but the supporting evidence is limited by a small selection of datasets (12 datasets), unclear dataset selection criteria, and insufficient hyperparameter optimization (HPO) for baseline models. The claims regarding linear correlation between feature importance and performance degradation are presented with empirical results, but the justification for why Pearson correlation is the appropriate metric for feature importance is weak. Additionally, the claim that LLMs show potential for handling feature shifts is not rigorously validated.", "methods_and_evaluation_criteria": "The experimental design includes:\n\n- A benchmark with four feature-shift scenarios: single shift, most/least relevant shift, and random shift.\n- Comparison of various tabular models, including tree-based models, deep learning approaches, and LLMs.\n- Evaluation metrics: accuracy for classification, RMSE for regression, and a robustness metric based on performance drops.\n\nHowever, the dataset selection process is unclear, and the justification for using 12 datasets is missing. Many benchmarks for tabular learning typically use dozens or even hundreds of datasets to ensure robustness. Additionally, the HPO process for baseline models is underdeveloped, which raises concerns about whether the reported performance reflects true model capabilities.", "theoretical_claims": "The paper does not introduce novel theoretical claims but suggests a linear relationship between shifted feature importance and model degradation. While this is weakly supported by empirical evidence, no theoretical justification is provided. The Pearson correlation analysis is simplistic, and additional validation is needed.", "experimental_designs_or_analyses": "The experiments are not sufficiently rigorous due to:\n\n- Limited dataset variety: Only 12 datasets, whereas other tabular learning studies often use significantly more.\n- Weak HPO process: Many tabular models, particularly gradient-boosted decision trees (GBDTs), are highly sensitive to hyperparameter tuning, but details on the HPO space are lacking.\n- Overgeneralized conclusions: The claim that \"most models struggle with feature shifts\" is based on a narrow selection of datasets and does not consider variations across different domains.\n\nThe benchmark fails to justify why these 12 datasets are representative of real-world tabular learning tasks.", "relation_to_broader_scientific_literature": "The paper correctly identifies that distribution shifts have been extensively studied, while feature shifts remain relatively unexplored. However, it does not cite all relevant prior work on feature shift mitigation. The paper could benefit from a more detailed discussion of:\n\n- Domain adaptation and domain generalization techniques relevant to feature shifts.\n- Methods for assessing robustness of tabular models beyond standard performance metrics.", "essential_references_not_discussed": "Most of recent essential references are discussed.", "other_strengths_and_weaknesses": "Strengths:\n- Addresses an underexplored problem (feature shifts in tabular data).\n- Includes a diverse set of tabular models, including LLMs. (even though they are only loaded from the previous python library.)\n\nWeaknesses:\n- Lack of clarity in key definitions: The paper does not provide a formal, quantifiable definition of \"feature shift.\"; The explanation of distribution shift vs. feature shift is vague.\n- Limited dataset selection: Only 12 datasets are used without clear justification.\n- Insufficient hyperparameter tuning: HPO significantly affects tabular models, but the paper lacks a proper discussion of its impact.\n- Overgeneralized conclusions: Claims about limited applicability of tabular models are made based on a small dataset selection.\n- Weak novelty: The benchmark consists of existing datasets and models, with no novel methodological contribution.", "other_comments_or_suggestions": "- Clarify definitions: Provide formal mathematical definitions of feature shift and distribution shift.\n- Improve dataset selection justification: Explain why these 12 datasets were chosen and why they represent real-world tabular learning challenges.\n- Enhance HPO methodology: Describe the hyperparameter search space and ensure fair comparisons among models.\n- Avoid overgeneralized claims: Reword broad claims such as “most models struggle with feature shifts” to reflect limitations of the current benchmark.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JNbZRVBdX", "reviewer": "Reviewer_B7Cf", "summary": "The paper studies the problem of feature shifts in tabular learning. It introduces TabFSBench, a benchmark for evaluating four types of feature-shift scenarios, assessing the robustness of four categories of tabular models. The study shows among others that most tabular models have limitations in handling feature-shifts and the feature importance has linear relationship with performance degradation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer B7Cf:\n\nThank you very much for your valuable feedback on our paper. We will take your reviews seriously and make the necessary revisions or additions to the final version.\n\n**Q1: Shifted features vary across different examples.**\n\nWe appreciate your feedback on our experimental design. In our random shift experiments, to evaluate model performance under n/N% feature shift (where n is the number of shifted features and N is the total feature count), we follow this protocol:\n1. Randomly sample min(10,000, C(N,n)) distinct feature combinations for each shift magnitude.\n2. Remove each selected combination during testing to assess performance.\n3. The performance for a given n/N% shift is the mean across all combinations.\n\n**Q2: Considering cases with different types of feature shifts**\n\nRegarding this question, we would like to clarify two potential interpretations:\n- Different types of (feature shifts): Please refer to our response to Reviewer Q7p8 Q1. The first leaderboard provides a comprehensive evaluation of model robustness across various types of feature shifts.\n- Different types of (feature) shifts: We conducted evaluations of CatBoost performance on a single shift, categorized by: Categorical, Numerical and Boolean features. Results demonstrate that model performance exhibits the highest sensitivity to Categorical features, followed by Boolean features while showing the lowest susceptibility to Numerical features. The complete experimental results are available in the **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/feature%20types.pdf)**.\n|Metric|Raw|Categorical|Numerical|Boolean|\n|-|-|-|-|-|\n|Acc|0.792|0.769|0.787|0.778|\n\n**Q3: Feature-increment scenario.**\n\nPlease refer to our response to Reviewer Q7p8 Q2.\n\n**Q4: More datasets.**\n\nDue to time and computational constraints in the rebuttal phase, we couldn't complete evaluations on larger benchmarks. However, following OcTree, we conducted random shift experiments on two high-dimensional datasets (madelon and nomao) and present model performance under 10%, 20%, ..., and 100% feature shift degrees, limited by space. This **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/high%20dimension.pdf)** has detailed results.\n|Dataset|Metric|0%|10%|20%|30%|40%|50%|60%|70%|80%|90%|100%|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Nomao|Acc|0.931|0.926|0.918|0.907|0.880|0.871|0.845|0.822|0.831|0.765|0.747|\n|Madelon|Acc|0.590|0.581|0.572|0.568|0.573|0.564|0.555|0.543|0.526|0.513|0.524|\n\nThe table demonstrates that as the degree of feature shift increases, the model performance decreases significantly. Please refer to our response to Reviewer Q7p8 Q1 for further explanation. \n\n**Q5: Imputation methods.**\n\nWe appreciate your suggestions. As Reviewer Q7p8 and WumN noted, our paper is a benchmark study and needn't propose new methods. Thus, we did not explore AutoFE in feature shifts. Additionally, our focus on evaluating model robustness meant we did not extensively cover advanced imputation methods, as explained in Section 3.3.\n\nIn light of your proposal, we posit that AutoFE can address feature shift scenarios through two approaches. Detailed results can be found in the **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/autofe.pdf)**.\n- Imputation: CAAFE and OcTree generate rules for imputing specific features, while OpenFE's generated features do not match the originals. We tested single shifts on the heart dataset using CAAFE and OcTree for imputation. Results show that LLM-based AutoFE can effectively generate matching features, enhancing model robustness compared to mean imputation.\n|Model|Raw|CAAFE|OcTree|\n|-|-|-|-|\n|CatB|0.846|0.852|0.850|\n|PFN|0.859|0.869|0.870|\n\n- Generation: We used AutoFE to generate new features to offset the impact of missing original features on model performance. Tests on the heart dataset with random shifts via OpenFE, CAAFE, and OcTree (with the number of generated features matching the original dataset) showed that LLM-based AutoFE has significant potential in feature shift scenarios.\n|Model|Raw|OpenFE|CAAFE|OcTree|\n|-|-|-|-|-|\n|CatB|0.710|0.849|0.854|0.849|\n|PFN|0.763|0.859|0.867|0.865|\n\nAs the importance of features increases, both model performance declines, further corroborating Observation 2 of TabFSBench. Please refer to our response to reviewer wA1o Q7 on other imputation methods.\n\n**Q6: Typos.**\n\nWe will make revisions in the final version.\n\n**Q7: Questions For Authors.**\n\nFeature importance, calculated via PCC, correlates strongly with the impact of feature absence on performance. Higher importance results in greater performance decline, indicating that model dependency on features is consistent across closed environments and feature shift scenarios. This stability is due to the models' robust feature selection and weight allocation mechanisms. For more details, refer to our response to reviewer wA1o Q6 and the **[link](https://anonymous.4open.science/r/icmlrebuttal-55BD/heart%20single.pdf)**.", "claims_and_evidence": "The paper evaluates four types of feature shifts and their impact on model performance: (a) single shift, where a single feature with different correlations is shifted; (b) most-relevant shift, where features with the highest correlations are shifted; (c) least-relevant shift, where features with the lowest correlations are shifted; and (d) random shift, where a random subset of features are shifted.\n\nThe main claims, including that current state-of-the-art methods fall short in handling the four types of shifts in the 'feature-decrement' scenario and that the degree of performance degradation is closely related to feature correlation, are well supported by the experimental results. However, the scope of the evaluation is somewhat limited. For example, the paper only considers the 'feature-decrement' case, where some of the features are missing values, for all test example cases. The evaluation would be more comprehensive if the shifted features varied across different test examples and if the 'feature-increment' scenario were also considered, allowing an assessment of how much the model's performance degrades compared to when additional features are available.", "methods_and_evaluation_criteria": "TabFSBench is specifically designed to evaluate the types of feature-shift scenarios and tabular models considered in this paper, and so the benchmark is appropriate for the evaluation. However, it could have included more tabular tasks, e.g., those with substantially larger feature sets, to strengthen the evaluation. Moreover, the different types of tabular models considered are evaluated 'in isolation'. Instead, some of the approaches could be combined to potentially better handle the feature-shift scenarios. For example, traditional tree-based models that tend to do better when large training data is available could benefit from features imputed using LLMs as feature generators. Such new proposals for handling feature shifts and demonstrating those on the benchmark would have strengthened the evaluation substantially.", "theoretical_claims": "No substantial theoretical claims are made in the paper.", "experimental_designs_or_analyses": "The overall design of TabFSBench to include 12 tabular tasks, with the four types of feature shifts is sound. The mean-value imputation strategy used is a common approach and therefore is a valid choice. Again, the experiments could have been extended in several ways, such as considering cases where different test examples experience different types of feature shifts or exploring other imputation strategies, such as using LLMs as feature generators.", "relation_to_broader_scientific_literature": "While prior works tend to focus more on distribution shifts, this paper examines feature shifts in tabular learning, introducing a benchmark for assessing four types of feature shifts. In addition, the paper evaluates different types of tabular models on this benchmark, including both traditional tree-based models as well as more recent LLM-based approaches.", "essential_references_not_discussed": "While the paper discusses general-purpose and tabular LLMs and evaluates them on the benchmark, LLMs have also been used as feature generators in tabular learning, and discussion on those use cases are missing. Some relevant works in this regard include OpenFE [1], CAAFE [2], and OCTree [3].\n\n---\n[1] Zhang et al., OpenFE: Automated Feature Generation with Expert-level Performance.\\\n[2] Hollmann et al., Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering.\\\n[3] Nam et al., Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "Typos:\n- $C^{train}$ near line 192.", "questions_for_authors": "Q. Observation 3 states that models with superior performance in closed environments also perform better in feature-shift scenarios. This seems somewhat counterintuitive, as one might expect that models achieving high performance in closed environments rely more heavily on specific features, making them more vulnerable to feature shifts. Do you have any intuition about this? Or, does this suggest potential improvements to the benchmark to strengthen the evaluation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contains a new benchmark containing tabular datasets with feature shifts, whereby features are removed or added at test time.  The paper observes that performance degradation occurs as feature shifts happen and notes that some models may be more robust than others.  Reviewers criticized the statistical analysis, feature imputation methods as a choice for handling missing features, the fact that all feature shifts are synthetic, and dataset shortage.  I will also point out that another paper has looked at feature shifts for tabular data but in the transfer learning setting (“Transfer Learning with Deep Tabular Models”), and this is probably worth discussing. While the paper received mixed reviews, I am inclined to accept it since these criticisms seem non-terminal, and the authors addressed the criticisms during rebuttal.
Decision: Accept (poster)