A benchmark of categorical encoders for binary classification
Authors: Federico Matteucci
Vadim Arzamasov
Klemens Böhm
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: categorical data
encoder
benchmark
sensitivity analysis
replicability
generalizability
ranking
TL;DR: Our study shows encoder performance is highly sensitive to factors like ML models, metrics, and tuning, explaining the disagreement between related studies. We study the replicability of our results and provide recommendations for encoders

Abstract: Categorical encoders transform categorical features into numerical representations that are indispensable for a wide range of machine learning models.
Existing encoder benchmark studies lack generalizability because of their limited choice of (1) encoders, (2) experimental factors, and (3) datasets. 
Additionally, inconsistencies arise from the adoption of varying aggregation strategies.
This paper is the most comprehensive benchmark of categorical encoders to date, including an extensive evaluation of 32 configurations of encoders from diverse families, with 36 combinations of experimental factors, and on 50 datasets.
The study shows the profound influence of dataset selection, experimental factors, and aggregation strategies on the benchmark's conclusions~---~aspects disregarded in previous encoder benchmarks.
Our code is available at \url{https://github.com/DrCohomology/EncoderBenchmarking}.

A benchmark of categorical encoders for binary classification
Federico Matteucci!, Vadim Arzamasov and Klemens Böhm!
Karlsruhe Institute Technology
(federico .tw.huucccmtttuccc,,.coo vadim. arzamasov, klemens bowhm) edu
Abstract
Categorical encoders transform categorical features into numerical representa- tions that are indispensable for a wide range of machine learning models. Ex- isting encoder benchmark studies lack generalizability because of their limited choice of 1. encoders, 2. experimental factors, and 3. datasets. Addition- ally, inconsistencies arise from the adoption of varying aggregation strategies. This paper is the most comprehensive benchmark of categorical encoders to date, including extensive evaluation of 32 configurations of encoders from diverse families, with 48 combinations of experimental factors, and on 50 datasets. The study shows the profound influence of dataset selection, exper- imental factors, and aggregation strategies on the benchmark's conclusions aspects disregarded in previous encoder benchmarks. Our code is available attps://g/ http:/ggtttt...././ httpg///aaa.mm/////oooooooooo/// com/DrCohomology htt.a//oooo.ooooooaeeeeeenaaraaa
Introduction
Learning from categorical data poses additional challenges compared to numerical data, due to lack of inherent structure such order, distance, kernel. The conventional solution is to transform categorical attributes into a numerical form, i.c., encode them, before feeding them to a downstream Machine Learning (ML) model. Various encoders have been proposed, followed by several benchmark studies. However, their combined results remain inconclusive, as we now describe.
Many factors impact the generalizability (26) of a benchmark of encoders, including: 1. the compared encoders, the number of datasets, the quality metrics, 4. the MI models used, and the tuning strategy. We also hypothesize that 6. the aggregation strategy used to summarize the results of multiple experiments may affect the conclusions in Section 2 only partially control for these factors. First, none of these studies uses more than 15 datasets of a given type (regression or classification). Second, despite these studies collectively covering substantial number of encoders, they often focus specific encoder families, resulting in comparison gaps between the best encoders. For instance, the best perffrrinn encoders from (28) (Mean-Targe) Target) have not been studied together yet. Third, the results of existing studies are often not comparable due to variations in the selected quality metrics. For instance, (28 measures quality with ROC AUC, 4 with average precision, and [41) with accuracy. Fourth, existing studies tune ML models different ways, yielding incompatible evaluations. For instance, (28 41 do tune, while 4 5 8
study. Existing encoder benchmarks, reviewed
(Cross-Vaile Validated GLMM) and 4rl
tune but do not specify if they tune the ML model encoded data or if they tune the entire ML pipeline. Last, no benchmark study of categorical encoders explores the impact of aggregation strategies, which is substantial according our experiments. For instance, (5] ranks the encoders by average ranking across all datasets, while 288 computes the median ranking with Kemeny- Young aggregation 46)
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

This study offers taxonomy and a comprehensive experimental comparison of encoders for binary classification, taking into account the factors just mentioned. In particular, we consider: 32 encoder configurations, including all of the best-performing ones from the literature and three novel encoders; 50 datasets for binary classification; four quality metrics; 4. five widely used MI. models; 5. three tuning strategies; 6. 10 aggregation strategies gathered from existing categorical encoder benchmarks and from benchmarking methodology studies [27]9]. (9). This allows us to provide novel insights into the sensitivity of experimental results to experimental factors. In particular, we demonstrate how replicability (26) may not be ensured even for studies conducted on up to 25 datasets. For those combinations of experimental factors that show reproducible results, isolate and recommended the best encoders.
Paper outline:
Section 2 reviews existing works, Section presents a taxonomy of encoder families, Section describes the experimental setup, and Section features the results.
Table 1: Related work on categorical encoders for binary classification.
Ours
28
[4]
[8]
[5)
44
[1
# Binary classification datasets
50
10
5
3
6
# ML models
5
5
1
4
5
Identifier
Frequency-based
V
Contrast
Similarity
J
Encoder family
Simple target
Binning
Smoothing
Data-cons cottacconttaining
Precision-recall hased
/
Quality metric
Balanced accuracy
Accuracy
Full pipeline tuning
Tuning strategy
Model tuning
No tuning
Heuristic
Aggregation strategy
Friedman- Nemeeyyi
Kemeny- Kemeny-Young
2 Related work
Benchmarks of encoders. We focus on binary classification tasks, as they offer wider range of compatible encoders; indeed, we could conduct deeper replicability analysis while maintaining the computation feasible. Table summarrzzs the related work. The other benchmarks often consider few datasets and either do not tune the ML model or do not describe the tuning procedure. This limits their applicability and generalizability. Additionally, there are substantial differences in the experimental settings across articles, including the encoders considered, quality metrics employed, and aggregation strategies used to interpret results. Hence, the comparability of these findings
limited. For instance, (28) recommends a lata-constraining encoder, (41] both data-constraining
and contrast encoders, (5 similarity encoders, (8) an identifier encoder, and (44) a simple target
encoder. Other benchmarks of encoders are (36), which focuses regression tasks and faces similar
issues, and (3 14 20). that use only a single dataset.
Analysis of benchmarks. When designing our benchmark, we adhered to the best practices discussed in the literature on benchmark design and analysis. particular, [27] studies how choices of experimental factors impact the experimental results and advocates for benchmarks that consider a large variety of factors. Similarly, (9) suggests guidelines to mitigate the inconsistencies in the choices of data and evaluation metric. Finally, [2] proposes a methodology to account for variance in the design choices (randomization of sources of variation) and post-processing of the experimental results (significant and meaningful improvements).

---

3 Taxonomy of encoders
This section presents the essential terminology and discusses the considered encoders and their corresponding families. Appendix 7.1 provides formal and detailed descriptions of the encoders.
3.1 Notation and terminology
Consider a tabular dataset with target y taking values in {0. 1}, and be one of its attributes (columns). A is categorical if it represents qualitative properties and takes values in a finite domain QA Each w E A level of A. Categorical attributes do not support arithmetic operations like addition multiplication, and their comparison is not based on arithmetic relations. An encoder E replaces a categorical attribute A with set of numerical attributes, (A).. We write E(QAA) ) to indicate the domain of ((A). Encoders may encode different levels in A in the same way, encode in different ways different occurrences in the dataset of the same level. Encoders are either supervised or unsupervised: Supervised encoders require target column, while unsupervised encoders solely rely on A. In what follows, A always denotes the categorical attribute to encoded.
3.2 Unsupervised encoders
Identifier encoders assign unique vector identifier each level. The most recognized encoder is
15]. One-Hot both space- ince--nnffficeent and ineffective [28 4 5 Alternatives include Ordinal (Ord), which assigns unique consecutive identifier to each level, and Binary (Bin), which splits the base-2 representation
One-Hot (OH), the default encoder in most machine learning pipelines [5).
of Ord(A A) into its digits.
Frequency- -based encoders replace levels with some function of their frequency in the dataset. We use Count, which relies on absolute frequencies 28
Contrast encoders encode levels into (L -)-dimensional vectors so that the encodings of all levels sum up to (O..... 0) [1]] A constant intercept term, 1, is usually appended to the encoding of each level. Contrast encoders encode levels such that their coefficients represent the level's effect contrasted against a reference value. A common example is Sum, which contrasts against the target's average value.
Similarity encoders treat w E C as strings and map them into numeric space taking their similarity into account [5] 4). These encoders are particularly useful for handling "dirty" categorical datasets that may contain typos and redundancies. One example is Min-Hash (MH), which decomposes each level into a of n-grams, sequences of consecutive letters, and encodes to preserve the Jaccard similarity of the decompositions.
3.3 Supervised encoders
Simple target encoders encode levels with a function of the target. Prime examples are Mean- Marget (MT) (7). which encodes with the conditional average given A, and Weight of Evidence (WoE) (39), which encodes with the logit of MT(A). benefit from regularization. The following families of encoders are regularization for Mean- Target.
Target can lead to severe overfitting (88 3T. may
We propose Binning encoders, that regularize MT by partitioning either ΩA MT(Ω into bins. Pre-Binned MT (PBMT) partitions ΩA to maximize the number of bins such that each bin's relative frequency exceeds specified threshold, then encodes the binned attribute with MT. Discretized MT (DMT) partitions MT(22) into intervals of equal length, then encodes each level with the lower bound of the interval in which MT encoding falls.
Smoothing encoders blend MT(12A) with the overall average target. Notable examples are Mean- Estimate (ME) [22] which uses weighted average of the two, and Generalized Linear Mixed Model encoder (GLMM) 28 which encodes with the coefficients of generalized linear mixed model fitted the data.
Data-constraining encoders regularize MT(A) A) by restricting the amount of data used to encode each occurrence of level in the dataset. CatBoost (CB) (31) first randomly permutes the dataset's rows, then maps each occurrence level to the average target of its previous occurrences. Cross- Validated MT (CVMT) (28) splits the dataset into folds of equal size, then encodes each fold with an
3

---

MT trained on the other folds. We propose the Up variant of CVMT, BUMT, which trains an MT on each fold and uses them to encode the whole dataset. Related variants are Cross- Validated GLMM (CVGLMM) 28] and its BlowUp version (BUGLMM).
4 Experimental design
As there is intrinsic measure of encoder's quality, we proxy the latter with the quality of an ML model trained on encoded data. This procedure in line with the current literature on the topic, discussed in Section 2 Each experiment thus consists of the following steps. First, we fix combination of factors: a dataset, an ML model, quality metric, and a tuning strategy. Then, we partition the dataset using a 5-fold stratified cross-validation and pre-process the training folds by:
imputing missing values with median for numerical and mode for categorical attributes; scaling the numerical attributes; encoding the categorical attributes.
If funing is to be applied, we fine-tune the pipeline with nested pross-validation and output the average performance over the outer test folds. We used standard scikit-learn (99 procedures for scaling and missing values imputation.
We conducted experiments using Python 3.8 AMD EPYX 7551 machine with 32 cores and 128 GB RAM. We limit each evaluation to 100 minutes to handle the extensive workload. As described in Appendix 7.3.1 out of the 64000 cross-validated evaluations, 61812 finished on time without throwing errors. For the sensitivity, replicabilityy, and encoder comparison analysis, we ignored the missing evaluations. We did so since there is no clearly superior imputation method, and 2. to avoid introducing unnecessary variability in the analysis. Our preliminary experiments confirm that imputing the small number of missing evaluations does not significantly impact our analysis.
In what follows, we describe the datasets, ML models, quality metrics, and tuning strategies we use in our experiments. Then, we outline the different aggregation strategies. Appendix 7.2 provides further details about datasets and aggregation strategies.
4.1 Encoders
We used the category We sourced MH from the authors' implementation (4 51 We implemented DMT, GLMM, ME, MT, PBMT, CVMT, BUMT, CVGLMM, and BUGLMM, We also added a baseline encoder, Drop, which encodes every level with 1. For DMT, experimented with the number of bins: (2, 5, 10). for ME, with the regularization strength: 10.1, 1, 10). for PBMT, with the minimum frequency: (0.001, 0.01, 0.1), and for cross-validated encoders, such as CVMT, with the number of folds: {2,5, 5, 10}. We display hyperparameter values with subscripts, e.g., CV2MT.
category_ encoders implementations of Bin. CB, Count, Ord, OH, Sum, and WoE.
4.2
Datasets
We used binary classification datasets. This allows us to conduct in-depth analysis using the same ML models and quality metrics. Additionally, certain supervised encoders, e.g., WoE, specifically designed for binary classification tasks. We chose 50 datasets with categorical attributes from OpenML 42 including the suitable ones from the related work.
4.3
ML models
We experimented with diverse ML models that process data in different ways: decision trees (DT) and boasted trees (LGBM) exploit orderings, support vector machines (SVM) use kernels, k-nearest neighbors (k-NN) relies on distances, and logistic regression (LogReg) is "pseudo-linear" model. The LGBM implementation we used from the LightGBM module while the other models' implementations are from scikit-learn. Table 2obbpare our model choices with related work. We

---

Table 2: ML models used in related studies.
Ours
28
Tree ensembles
Linear
SVM
Model family
K-NN
DT
Neural
Naive Bayes
excluded neural models due to their inferior performance tabular data 15 and the absence of recommended architecture. We also did not use Naïve Bayes due to its lack of popularity.
4.4 Quality metrics and tuning strategies
We assessed an encoder's quality by evaluating an MI model trained on the encoded data. We use four quality metrics: balanced accuracy (BAce), F1-score (F1), accuracy (Acc), and Area Under the ROC Curve (AUC). We compared three tuning strategies:
no tuning;
model tuning: the entire training set is pre-processed before tuning the model;
fill tuning: the entire pipeline is tuned the training set, with each training fold of the nested cross-validation pre- processed independently.
We used Bayesian search from scikit-optimize" for full tuning, and for model tuning grid search from scikit-learn. Table f summarizes the tuning search space for different ML models. To mitigate
excessive runtime, chose not tune certain MI. models and limited the dataset selection the smallest 30 for full tuning, as Table 3 illustrates.
Table 3: Factors for different tuning strategies.
Models
Datasets
No tuning
DT, k-NN, LogReg, SVM, LGBM
Model tuning Full tuning
DT. k-NN, LogRreg
50
DT, k-NN, LogReg, SVM
30
Table 4: Tuning search space.
Hyperparameter
Interval
Grid
DT
max__eppl
2,....5
(2,5, None]
k-NN
n_ neighbors
2,..., 10]
(2,5, 10)
LogReg
C
0.2, 5
{0, 1,
C
0..1 2]
SVM
gamma
(0.1, 100]
4.5 Aggregating into a consensus ranking
A common practice for summarizing and interpreting the results of benchmark experiments is to aggregate them into a consensus ranking of alternatives (encoders in our case) [T0] 27 15]. To obtain a dataset-independen ranking of encoders, aggregate the results across different datasets while keeping all other factors fixed. We now present well-known aggregation strategies used in benchmarks.
Heuristics rank alternatives based on an aggregate score. Common aggregation heuristics include mean rank (R-M) [5], median rank (R-Md), mean quality (Q-M), median quality (Q-Md), rescaled
https://scihtt-oppiiie github. cocchub.oo//see io/stable/

---

mean quality [36 15 (Q-RM), the number of times the alternative was ranked the best (R-B) or
the worst (R-W) [1I, the number of times the alternative's 's quality is better than the best quality multiplied by threshold 0 < (Q-Tha).
Friedman-Nemenyi tests [T0] (R-Nemp-vllun First, one ranks alternatives separately for each dataset and then applies Friedman test to reject the hypothesis that all encoders have the same average rank. If the hypothesis rejected, pairwise Nemenyi post-hoc tests are conducted to compare pairs of alternatives. Finally, one uses the results of these post-hoc tests to construct the consensus ranking. This aggregation strategy requires the user to choose p-value.
Kemeny- Honeny-Young aggregation (21 47 (e- first ranks alternatives separately for each dataset. Then, it determines the consensus ranking that minimizes the of distances to the datasets' rankings. We adopt the approach described in [45]. with distance measure that accomodates ties and missing values in the rankings. We then formulate the optimization problem problem and solve it using a GUROBI solver with academic license 2 Kemeny- Young aggregation is much slower than the other aggregation strategies, taking minutes for each aggregation.
mixed integer linear
Results
This section summarizes the main results of our study, Appendix 7.3 further discusses the missing evaluations, run time, replicability, the ranks of the encoders and studies the effect of tuning on pipeline quality.
5.1 Sensitivity analysis
The relative performance of encoders, i.e., the ranking, can depend on the pick ML model, quality metric, and tuning strategy. More, the choice of an aggregation strategy impacts the consensus ranking. To quantify the influence of these choices, we calculate the similarity between rankings using the Jaccard index J for the of best encoders and the Spearman correlation coefficient p. Intuitively, measures if two experiments with different factor combinations agree on the best encoders, while takes the entire ranking into account. For both measures, values close indicate high agreement and low sensitivity. Conversely, values near 0 (or, for p. negative) suggest low consistency and high sensitivity.
5.1.1 Sensitivity to experimental factors
We evaluate the sensitivity of encoder rankings on individual datasets with respect to experimental factor (ML model, quality metric, or tuning strategy) by varying the factor of interest and keeping the other factors fixed, then calculating the similarity between pairs of rankings. After that, we average the result across all combinations of the other factors. Figures Db. Dp. and [ show the resulting values, with Spearman's p in the upper triangle and Jaccard index J in the lower triangle. For example, Spearman's 's between encoder rankings for DT and SVM, averaged across all datasets, tuning strategies, and quality metrics, is 0.3.
Our findings highlight the high sensitivity of results to experimental factors, for both the full rankings and the best encoders. They also explain why results from other studies are inconsistent, as choosing different values for any factor will lead to different results.
5.1.2 Sensitivity to aggregation strategy
To evaluate the impact of the aggregation strategy the consensus ranking, apply the same procedure as above to consensus rankings instead of rankings on individual datasets. Figure [ presents the results with the notation from Section[4.5 For example, Spearman's p between consensus rankings obtained with Q-M and Q-Md averaged across all ML models, tuning strategies, and quality metrics is 0.8.
While some aggregation strategies show strong similarities, different strategics yield very different consensus rankings in general. This is particularly evident for Jaccard index J, indicating the high sensitivity of the best encoders to the rank aggregation strategy.
httpp::// http://wwwwaaoooooooooooooooooos

---

K-NN
0.4
0.2
Q-M
0.8
0.4
0.6
0.2
0.6
0.7
0.7
0
0.5
LGBM
0.4
0.3
0.2
0.2
(a)
Q-Md
0.3
0.4
0.5
0.1
0.6
0.6
0.5
0
0.4
LogReg
0.4
0.5
0.4
0.4
Q-RM
0.3
0.3
0.3
0
0.5
0.5
0.3
0
0.4
SVM
0.3
0.5
0.3
0.4
Q Thoss
0.2
0.1
0.3
0.6
0.7
0.6
R-B
0.1
0.4
0.3
0.3
0.3
DF
k-NN
L.BBM
WAS
Kem
0.4
R-M
0.3
0.3
0.4
0.5
0.9
Acc
0.5
0.7
0.7
R-Md
0.4
0.3
0.3
0.3
0.5
0.5
0.5
0.3
(d)
AUC
0.5
0.6
0.5
(b)
R- R-Nemo.ot
0
0
0.1
0
0
0.8
0.8
BAcc
0.7
0.6
0.8
R-Nemoos
0
0
0
0.1
0
0
0.8
0.9
0.6
0.4
0.6
R- R-Nemo.1
0
0
0
0
0.1
0
0.7
0.9
Acc
ALC
BAce
Fi
R-W
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.2
0.2
0.1
full
0.6
0.7
QM
Q-Md
Q-M
Q-Tha.95
R-B
R-Kem
R-M
R-Md
R-Nema.01
R-Nemo.05
R-Nema.1
R-W
model
0.7
0.7
no
0.7
0.7
fall
model
no
Figure Sensitivity the average similarity between rankings, measured with (upper triangle) and J (lower triangle), computed between individual rankings for varying: (a) ML model, (b) quality metric, (c) tuning strategy, and between consensus rankings for varying (d) aggregation strategy.
5.2 Replicability
Replicability is defined as the property of a benchmark to produce consistent results from different data (26). This definition does not, however, provide quantifiable notion of replicability. To overcome this, we made the following modeling decisions. First, we fix factor combination: MI. model, quality metric, tuning strategy, and aggregation strategy. We excluded the R-Nem and R-Kem aggregation strategies due to their slower run time. Second, we model the result of benchmark on dataset sample S' with the consensus ranking aggregated across S. Third, we quantify replicability as the similarity between consensus rankings averaged over all factor combinations and 100 pairs of equal-sized disjoint sets of datasets. As discussed in Section 5.1 measure the similarity with p and J/ to capture the similarity between both the rankings and the best encoders. We refer to them as p-replicability and J J-replicability. respectively.
Figure shows the outcome for different tuning strategies, conditional on the MI model and the size of the dataset samples. We have studied additional factors Appendix 7.3.3 The shaded areas represent a bootstrapped 95% confidence interval. Our findings show an upward trend of p-replicability the size of the dataset samples increases. This observation confirms that, in general, considering a larger number of datasets yields more reliable experimental outcomes. It is, however, important
that this pattern does always hold for J-replicability. This suggests that, for some models, the best encoders might vary significantly even with a relatively large number of datasets. To conclude, the replicability of our results strongly depends on the ML model, with logistic regression exhibiting the highest replicability and decision trees the lowest.

---

DT
k-NN
LGBM model luning
LogReg
SVM
luning
full tuning
0.6
0.4
0.2
0.0
0.3
0.1
0.0
5
10
15
20
25
10
15
20
25
5
10
15
sample size
sample size
sample size
Figure 2: Replicability as the average similarity consensus rankings from disjoint subsets of datasets.
5.3 Comparing encoders
Based the outcome Section 5.2 we now examine the ranks of encoders limited to decision trees, logistic regression, and all ML models.
Figure 3a shows the rank of encoders from the experiments with decision trees across all datasets, quality metrics, and tuning strategies. One-Hot the hest-performing encoder; however, Nemenyi tests at significance level of 0.05 fail to reject that the average rank of One-Hot is the same that of the other encoders.
Figure 3b features the encoder ranks for logistic regression, where four encoders, namely One-Hot, Sum, Binary, and Weight Evidence, consistently achieve higher ranks compared to the others. Nemenyi tests confirm that this difference in ranks is significant. These results are in line with the ones from Section 5.2 which indicate low replicability of the results for decision trees and higher replicability for logistic regression.
Figure 3e presents the ranks of encoders across all datasets, ML models, quality metrics, and tuning strategies. Similarly to logistic regression, One-Hot, Sum, Binary, and Weight of Evidence consis- tently achieve significantly higher average ranks compared to the other encoders, again confirmed by Nemenyi tests. We recommend these four encoders as the preferred choices in practical applications. This conclusion contradicts other studies reporting a suboptimal performance of One-Hot (5) 28
Our findings also reveal that Drop performs significantly worse than all other encoders, i.e., encoding categorical attributes generally yields better results than dropping them.
5.4 Comparing to related work
In this section, we compare our results with the findings of other studies. To do so, we select subsets of our results that mimic the experimental settings in related work. In (28), CV;GLMM outperformed every competitor for boosted trees and k-NN, while GLMM was recommended for SVMs. However, our experiments, Sum outperformed GLMM for SVMs, One-Hot did better than CV: GLMM for boosted trees, and encoders are better than One-Ho One-Hot for boosted trees, subsequent research reported no significant difference between Min-hash and One-Hot on medium-sized tabular datasets 4 Our findings are in line with this latter result, as we could not find a performance difference between two encoders with t-test with a significance level of 0.05. In [1], Sum is reported as the best encoder on the Adult dataset for boosted trees, while a Data-constraining encoder reported as the worst. With the same setting, did not find significant performance difference for any encoder except for Drop, which
was better than CL.GLMM for k-NN. Next, while in (5) similarity

---

DT
LogReg
all models
WoF
Sum
WoF
OH
OH
OH
Bin
WoB
Bin
Bin
Sum
oB MT
BU
MT
PB MT MT
MU
PBo MT
CV
MT
BU
CV MT
MT
BUG
PB MT
BU.GLMM
CV.MT
CV
Pl MT
Count
PB MT
Sum
PB MT
BULGGLMM
GLMM
MI
CV:MT
BU OT
BLMM
D,MT
D:MT
D&MT
DIOTT
MEgg
MT
ME
D.MT
MT
MT
MT
BU,GLMM
ME
BU:GLMM
ME
DI
ME
D-MT
BU
ME
GLMM
Ord
D.MT
MEg
ME
Ord
BU:GLMM
CV&GLMM
CV:GLMM CB
PBo MT
BUIGLLMM
CV,GL.MM MH
CV CV1gGLMM
GLMM
BU:MT
CV.GLMM
CV,GLMM
CV,GLMM
CV MLLMLLMM
CV 1LLMMMM
BURGLMM
BU:GLMM
PB MT
Ord
CB
Count
Count
Drup
Drup
Drop
10 20
30
10
20
30
10
30
rank
rank
rank
(a) Decision tree
(b) Logistic regression
(c) All models
Figure 3: Ranks of encoders.
performed the worst. On the Bank marketing dataset, 8 showed that One- Hot and Mean- MerneTTrget outperformed Binary with logistic regression. In our experiments, Binary was slightly worse than One-Hot and Mean-Target. In [44), Dummy, an identifier encoder similar One-Hot, better than Mean-Target on the Tic-tac-toe dataset with boosted trees. We, instead, did not observe any significant difference between One-Hot and Mean-Target for these factors.
Limitations and conclusions
Limitations. First, we treated encoders as part the pre-processing, but certain encoders can be an integral component of specific ML models. For instance, CatBoost is derived from the homonymous boosted trees algorithm, which re-encodes the data multiple times during training. Second, we applied single encoder to all categorical attributes. Using different encoders based the cardinality of the attribute may sometimes yield favorable results [28] However, the selection of the optimal encoder for each attribute requires either domain knowledge of the attribute purpose- built tools, which falls outside the scope of our benchmark and therefore left as future work. We also did not include neural networks, due to the absence of recommended architecture and reported interior performance to tree-based models on tabular data 15
Conclusions.
In this study, we conducted an extensive evaluation of encoder performance across various experimen- tal factors, including ML models, quality metrics, and tuning strategies. Our results demonstrate high sensitivity of encoder rankings to these factors, both for the full rankings and the cest-performing encoders. This sensitivity explains the inconsistent results among related studies, as different choices any of these factors can lead to different outcomes. We also assessed the impact of aggregation strategies on consensus rankings, revealing significant variations in rankings depending on the chosen strategy. This emphasizes the importance of carefully considering the aggregation method when post-processing and interpreting results. Regarding replicability, we defined and quantified it using p-replicability and J-replicability. Our findings indicate that replicability is influenced by factors such the ML model, with logistic regression exhibiting the highest replicability and decision trees

---

the lowest. Additionally, larger dataset samples tend to yield more reliable experimental outcomes, although this trend does not always hold for J-replicabiiiiy. Based on our results, we recommend specific encoders for practical applications. For decision trees, Weight of Evidence performed the best, although statistical tests did not show a significant difference from other encoders. For logistic regression, Sum, One-Hot, Binary, and Weight of Evidence consistently achieved higher ranks, with statistically significant differences from other encoders. These findings contradict previous studies, highlighting the importance of considering a broad range of experimental factors. Finally, our comparative analysis with related work revealed discrepancies in encoder performance, suggesting that the breadth of our study may contribute to these differences. This emphasizes the need for caution when interpreting results from studies with more limited experimental settings. Overall, our study provides valuable insights into the sensitivity of encoder performance to experimental factors, as well as recommendations for practical encoder selection across different scenarios.
Acknowledgments
We thank Dmitriy Simakov for valuable discussions and Natalia Arzamasova for her algorithm and implementation of the PreBinnedEncoder. This work was supported in part by the German Research Foundation (Deutsche Forschungsemmeinchaat) project Charakterisierung, Modellierung und Homogenisierung von Vernetzungswerken mit Hilfe interpretierbarer Datenanalyyemetthtee and by the State of Baden- Württemberg, project Algorithm Engineering für die Scalability Challenge.
References
[1] David Aha. Tic-Tac- Toe Endgame. UCI Machine Learning Repository. 1991.
[2] Xavier Bouthillier al. "Accounting for Variance in Machine Learning Benchmarks". In: MLSys. mlsys.org. 2021.
[3] Laurent Candillier and Vincent Lemaire. Nomao. UCI Machine Learning Repository. 2012.
[4] Patricio Cerda and Gaël Varoquaux. "Encoding High-Cardinality String Categorical Variables".
In: IEEE Trans. Knowl. Data Eng. 34.3 (2022), pp. 1164-1176.
[5] Patricio Cerda, Gaël Varoquaux, and Balázs Kégl. "Similarity encoding for learning with dirty categorical variables". In: CoRR abs/ 806.00979 (2018).
[6]
Congressional Voting Records. UCI Machine Learning Repository. 1987.
[7] Don Coppersmith, Se June Hong, and Jomathan R. M. Hosking. "Partitioning Nominal At- tributes Decision Trees". In: Data Min. Knowl, Diseov. 3.2 (1999), pp. 197-217.
[8] Mwamba Kasongo Dahouda and Inwhee Joe. "A Deep-Learned Embedding Technique for Categorical Features Encoding". In: IEEE Access 9 (2021). pp. 1 113811-14331.
[9] Mostafa Dehgani et al. "The Benchmark Lottery". In: CoRR 0iss/107.00700) (2021).
[10] Janez Demsar. "Statistical Comparisons of Classifiers over Multiple Data Sets". In: Mach. Learn. Res. 7 (2006), pp. 1-30.
[11] Keyu Duan et al. "A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking".. . In: NeurIPS. 2022.
[12] Bob Evans. Cylinder Bands. UCI Machine Learning Repository. 1995.
[13] Farhad Soleimanian Gharehchopogh and Seyyed Reza Khaze. "Data mining application for cyber space users tendency in blog writing: a case study". In: CoRR abs/1307.7432 (2013).
[14] Sebastian Gnat. "Impact of Categorical Variables Encoding on Property Mass Valuation". In:
KES. Vol. 192. Procedia Computer Science. Elsevier, 2021, pp. 3542-3550.
[15] Léo Grinsztajn. Edouard Oyallon, and Gaël Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?" In: NewrIPS. 2022.
[16] C. Harleya, R. Reynolds, and M. Noordewier. Molecular Biology (Promoter Gene Sequences).
UCI Machine Learning Repository. 1990.
[17] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository. 1994.
[18] Ronald Iman and James Davenport. "Approximations of the critical region of the Friedman statistic". In: Communications in Statisticy-Theet and Methods 9 (Jan. 1980), pp. 571-595.
[19] Andras Janosi et al. Heart Disease. UCI Machine Learning Repository. 1988.
10

---

[20] Justin M. Johnson and Taghi M. Khoshgoftaar. "Encoding Techniques for Cagd-Caadnn Features and Ensemble Learners". In: IRI. IEEE, 2021, pp. 355-361.
[21]
John G Kemeny. "Mathematics without numbers". In: Daedalus 88.4 (1959), pp. 577-591.
[22] Daniele Micei-Barreca. "A Preprocessing Scheme for High-Cardinalle Categorical Attributes in Classification and Prediction Problems". In: SIGKDD Explor: 3.1 (2001), pp- 27-32.
[23] Erick Moreno-Centent and Adolfo R Escobedo. "Axiomatic aggregation of incomplete rank- ings". In: IIE Transactions 48.6 (2016), pp. 475-488.
[24] S Moro, P Rita, and P Cortez. Bank Marketing. UCI Machine Learning Repository. 2012.
[25]
Mushroom, UCI Machine Learning Repository. 1987.
[26]
Engineering National Academies of Sciences, Medicine, et Reproducibility and replicability
in science. National Academies Press, 2019.
[27] Christina Nießl al. "Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results". In: WIREs Data Mining Knowl. Discov. 12.2 (2022).
[28]
Florian Pargent al. "Regularized target encoding outperforms traditional methods in su- pervised machine learning with high cardinality features". In: Comput. Stat. 37.5 (2022), pp. 2671-2692.
[29] F. Pedregosa et al. "Scikit-learn Machine Learning in Python". In: Journal of Machine Learning Research 12 (2011), pp. 2825-2830.
[30] Kedar Potdar, Taher S Pardawala, and Chinmay D Pai. "A comparative study of categorical variable encoding techniques for neural network classifiers". In: International journal of computer applications 175.4 (2017), pp. 7-9.
[31] Liudmila Ostroumova Prokhorenkova et al. "CatBoost: unbiased boosting with categorical features". In: NeurlPS. 2018, pp. 6639-6649,
[32]
Quinlan and Quinlan. Credit Approval. UCI Machine Learning Repository.
[33] Ross Quinlan. Statlog (Australian Credit Approval). UCI Machine Learning Repository.
[34]
Ross Quinlan. Thyroid Disease. UCI Machine Learning Repository. 1987.
[35]
Jan N van Rijn and Jonathan K Vis. "Endgame Analysis of Dou Shou Qi".. In: ICGA Journal
37.2 (2014) 4), pp. 120-124.
[36] Diogo Seca and João Mendes-Moreira. "Benchmark of Encoders of Nominal Features for Regression". In: WorldCIST (1). Vol. 1365. Advances in Intelligent Systems and Computing. Springer, 2021, pp. 146-155.
[37]
Alen Shapiro. Chess (King-Rowe Rook vs. King-Pawn) UCI Machine Learning Repository. 1989. Peter Sprent and Nigel C Smeeton. Applied nonparametric statistical methods. CRC press, 2016.
[38]
[39] Gero Szepannek. "On the practical relevance of modern machine learning algorithms for credit scoring applications". In: WIAS Report Series 29 (2017), pp. 88-96.
[40] Muhammad Usman and Adeel Ahmed. Dresses Artrissttrrerttttte Sales. UCI Machine Learning Repository.201 2014.
[41] Eric Valdez- Valenzuela, Angel Kuri-Morales, and Helena Gómez-Adorno. "Measuring the Effect of Categorical Encoders in Machine Learning Tasks Using Synthetic Data". In: MICAI (1). Vol. 13067. Lecture Notes in Computer Science. Springer, 2021, pp. 92-107.
[42] Joaquin Vanschoren et al. "OpenML: networked science in machine learning". In: SIGKDD
Explor. 15.2 (2013), pp. 49-60.
[43] J. Wnek. MONK's 's Problems. UCI Machine Learning Repository. 1992.
[44] Marvin N Wright and Inke R König. "Splitting categorical predictors in random forests".
In: PeerJ 7 (2019), e6339.
[45] Yeawon Yoo and Adolfo R. Escobedo. "A New Binary Programming Formulation and Social
Choice Property for Kemeny Rank Aggregation". In: Decis. Anal. 18.4 (2021), pp. 296-3201
[46] H Peyton Young. "Condorcet's theory of voting". In: American Political science review 82.4
(1988), pp. 1231-1244.
[47] H Peyton Young and Arthur Levenglick. "A consistent extension of Condorcet's election principle". In: SIAM Journal on applied Mathematics 35.2 (1978), pp. 285-300.

---

[48] Maciej Zieba et al. "Boosted SVM for extracting rules from imbalanced data in application to prediction of the post-operative life expectancy in the lung cancer patients". In: Appl. Soft
Comput. 14 (2014), pp. 99-108.
12

---

7 Appendix
7.1 Encoders
This section presents reproducible description of the encoders discussed Section 3 following the structure outlined below. We discuss identifier, frequency- basee, contrast, and simple target encoders together Appendix
all of these encoders can be explicitly represented functions. Similarity, binning, smoothing, and contttonsg encoders have dedicated sections. Table 5 contains the notation used in this section.
Table 5: Notation for section 7.1
Symbol
Meaning
No
natural numbers including base-2 representation of x E Ng
(r)2 X
set of matrices with entries in X, rows and d columns indicator function binary classification dataset number of rows of D target attribute of D categorical domain (strings)
D
y {0, 13"h
01 twiteE)
A E SA
categorical attribute of D to be encoded such that A, = W|, encoder
4 E (1......II L}
E: A M Raxd
M Raxd
encoding of A, compact notation encoding of A with explicit encoder
E(A) E Baxd
E(RA)
unique values of rows ((A)
d = A)
number of columns of M
M
i-th row of M d > 1, i-th component of M if
Ie (1...., L}
index of levels row indices of M, A, or
i, h (1, (1.......
j E (1,...,d)
column index of M
7.1.1 Similarity encoders
Min-Hash treats E A as string, splits it into its set of character-level n-grams (substrings of n-consccutive characters), uses hash function encode each n-gram into an integer, and finally encodes w with the minimum value of the hash function on the set of n-grams. The process is repeated for d hash functions, vielding M Ra The default value of is 30, the authors report good performance with 300
7.1.2 Binning encoders
Pre-Binned Mean-Targe Target partitions QA into B buckets {P)) solve the optimization problem
Maximize
subject to 1 1 WEP 1=1
Vb < B
where 1 E [0 is user-defined threshold. Each bucket is then treated as new level and encoded with Mean-Targee Target, yielding an encoding E R".
Discretized Mean- Mean-Target partitions MT(ΩA) into intervals {I,,.., IB of equal length. Letting [([) be the interval that contains MT((w) (that is, the average target associated to wi). the encoding is M E R M, inf (l4). We experimented with B 2, 5, 10.
https://dirty-can httpa:/wwwa..aaaaaaaaaa/aa/aaaaa Mat.saeeeer...iillaaccccooooerre html
13

---

Table 6: Identifier encoders
A
((A)
Binary
{0, {0,1) [0g2(1]++1
M, = (11)2
Mi - [I(A; = =w))
j
Dummy
28
44
{0, 10,1)--1
10
j=L
One-
28
{0, 11
Mf; = 1(A, = =w))
Ordinal
28
No
M, 4
Table 7: Frequency-based encoders
Count 36
No
M, Σ, (A, =
Frequency
28
R
M, n I,, wi.
Table 8: Contrast encoders without intercept
[1A,
=)
j L
Sum
41
RL-1
My
-1
==
L-i
isj
Backward difference 41
RL-1
M
i>j
isj
Helmert 41
R/-1
Mj
i=j+1
J+1
i2j+2
Table 9: Simple target encoders
Mean-Target 28
R
M;
Σ
yn (A wi)
1=1
Weight of Evidence
39
28
No
M, = log
7.1.3
Smoothing target encoders
Mean-Estimate
Let TI Ei 1l (Ai wr) be the number of occurrences of wi in A.
TH MT(WW:) ) + Σ
M,
+ n4,
where w is a user-defined weight. Common choices are 1, 10.
GLMM 28 fits, for every E Ωs random intercept model
y/ Btg + ut Ei
where ~ N (0, 73) and s; ~ N(0,a²). The encoding is M E R" M, = 3,,
7.1.4 Identifier, frequency-based, contrast, simple target encoders
The descriptions are divided follows: iable for identifier encoders, Table iis for frequency- based encoders, Table is for contrast encoders, and Table 9 is for simple target encoders.
7.1.5
Data-constraining encoders
CatBoost uses a permutation æ of f1....,n] nn and encodes with M R such that
M (())
Σ
yn Il (Ap = Wirco)
h5*())
14

---

Table 10: Notation for section 7.2.2
Symbol
Meaning
T
missing evaluation rank indicator function
1
E;
encoder in table
E; E; + RU{
average cross- pross-validated quality on the j-th dataset, other factors fixed best quality on the j-th dataset, all other fac- tors fixed
h = max
{Φ; (Fi))
8 min;- I { (E₂)}
worst quality the j-th dataset, all other factors fixed
r3 E [1)
ranking obtained from $j adjacency matrix of rj consensus ranking adjacency matrix index of encoders index objects to aggregated
R (1,r S ry (Ba))) (0, 1)xm
c: N:E-NNu
(0,0,1) i,h, E (I....,,. n} j [m.
Cross-Validatee Validated MT 28) randomly partitions (1..... n) in folds of equal size. Let Da; be the fold that contains Then, every fold is encoded with Mean- Mean-Target trained on the other k folds:
M, = Σ 1 1 = wi,) yh
h=1
Common values for k are 2,5, 10.
Volss-valddted GLMM [28 works in a similar fashion as CVMT: it encodes each fold with GLMM trained on the other k folds.
BpowUp Cross- Validated MT randomly partitions {1...., n) in k folds Dı...., Dg of roughly equal size. Then, it encodes with M E R so that the j-th column is A encoded with Mean- Mean-Target trained on the j-th fold, yielding
Mj Σ (h DA) W1,) yh
We experimented with k = 2, 5, 10.
Blowup Cross-Vaas Validated GLMM is analogous to BUMT, but the j-th column of its encoding M is encode with E Bonk so that the j-th column of is encoded with GLMM trained on the j-th fold.
Experimental design
This section provides additional details about the datasets and aggregation strategies we discussed in Section 4
The notation we use in this section is summarized Table [0
7.2.1 Datasets
Table II
lists the datasets used in our experiments. The columns are as follows: is the OpenML identifier; n is the number of rows; d is the number of attributes; deat is the number of categorical attributes; ma the maximum categorical attribute cardinality; the "ft" flag denotes datasets used for full-tuning (cf. Section 4.4
15

---

Table 11: Datasets used in the study.
Name
Ref.
ID
d
deat
max max S2
ft
ada_prior
1037
4562
14
7
40
adult
1590
48842
14
7
42
airlines
1169
539383
7
4
293
amazon employee_aeeeeeeeee access
4135
32769
9
9
7518
Agrawall
1235
1000000
20
Australian
33
40981
690
14
4
14
bank-marketing
1461
45211
16
6
12
blogger
1463
100
5
3
5
Census-Incomer KDD
42750
199523
41
27
51
credit-approval
32
29
690
15
6
15
credit-g
31
1000
20
10
cylinder-bands
12
6332
540
37
17
71
dresses-sales
23381
500
12
11
25
heart-h
19
51
294
13
6
4
ibm-employee-attrtton
43896
1470
34
5
9
Dbn-employee-performaaa
43897
1470
33
5
9
irish
451
500
5
2
11
jungle_chess_ 2pcs......... elepphat
35
40999
2351
46
2
3
jungle_chess_ppg _p____pccs_.. Jion
35
41007
2352
46
2
3
jungle_chess_Dee ciaeeessppcs..___1
41005
3660
46
2
3
kdd internet usage
981
10108
68
20
129
KDDCup09 _at_upp0pppppeenecyye
1111
50000
230
33
15416
KDDCup09 _0e_upp00chummm___ churn
1112
50000
230
33
15416
KDDCup09 _
1114
50000
230
33
15416
KDD98
42343
82318
477
107
18543
kr-vs-kp
3
3196
36
3
kick
41162
72983
32
17
1063
ww--school-admissionn--aooo
43890
20800
11
1
molecular-biolog__ponne prromoterss
16
956
106
57
56
4
monks-problems-11
333
556
4
4
monks-problems-2
43
334
601
6
4
4
mv
881
40768
10
1
3
mushroom
25
43922
8124
22
16
12
43892
4908
16
4
29
nomao
1486
34465
118
27
3
nursery
959
12960
8
7
5
open_payment
42738
73558
5
4
4374
porto-seguro
41224
595212
57
13
104
profb
470
672
9
3
28
sick
34
38
3772
29
2
5
-ipiiccc-incieetts
42344
538638
21838
SpeedDating
40536
8378
120
58
260
students scores
43098
1000
7
2
6
telco-customer-chum
42178
7043
19
11
6531
thoracie-surgery surgery
1506
470
16
3
7
tic-tac-toe
50
958
9
9
3
Titanic
40945
1309
13
6
1307
vote
56
435
16
16
3
wholesale-customers
1511
440
1
3
WMO--Hariicane-SSuviviaa-D
43607
5021
22
21
4173
16

---

7.2.2 Aggregation strategies
This section presents the mathematical formulations of the aggregation strategies. As Section 4.5 explains, the results are aggregated across datasets, while keeping the other factors - ML model, tuning strategy, and quality metric fixed.
Heuristics. Heuristics aggregate by ranking encoders according to some score. Increasing heuristics assign the best rank to the encoder with the highest score, while the non non-increasing ones assign the best rank to the encoders with the lowest score. Table 12 contains the respective formulas. Any missing evaluations (L) are ignored during the computation.
The Friedman test is used rule out the null hypothesis that all encoders have, average, the same rank. The Friedman statistic adjusted for ties 38
Friedman-Nemenyi tests.
18 is
T (m- (S - -C) S
where S, = - ビn (E.)) - E 1i=1 ( T3 (E:)) and C m + 1)².
Under the null hypothesis that all encoders have the same rank, T is approximately distributed as an F-distribution with n 1 and (m-1)(m) 1) degrees of freedom.
If the Friedman hypothesis is rejected, one compare all pairs of encoders with n(n 1)/2 Nemenyi post-hoc tests [10). Nemenyi tests apply correction to control the error from testing multiple hypotheses. Two encoders Ej and Eg significatively different if
m(n+1)
Σ (E)()))- 2 4o
6m
where 1 /2 9a is the critical value based on Studentized range statistic 10
Kemeny- Young aggregation.
45
The consensus's adjacency matrix C is a solution to the mixed-integer linear problem
Maximize Σ S2C
subject to Cj CęN Ci 2 -1
Cj + Chi 2
Vi < h
C.hh E {0, 1}
Vi, h
where S
is cost matrix and nj Σ, 1 (r, + 1) is the number of encoders with evaluation dataset j. This formulation accounts for ties and missing ranks.
RC Σ (Ry i,
7.3 Results
This section complements Section 5
7.3.1 Missing evaluations
We successfully completed 61812 runs out of 64000, one per combination of encoder, dataset, ML model, tuning strategy, and quality metric. The 2188 failed evaluations are equally distributed among encoders, while tuning was the greatest influencing factor. Indeed, there were 4303 missing runs with no tuning, 1152 with model tuning, and 32 with full tuning. This is likely due to the bigger datasets used in no tuning and model tuning, ef. Section 4.4 and Table TT The total runtime for successful evaluations was 108 days.
17

---

Table 12: Scores of heuristics.
Score of E
Increasing
Mean rank
Ej r2(E) median
Median rank
(ry (E))
Rank best
1 (ry (B) =
Rank worst
Σ
TT (B) +
max (Bz)
ill....m
Mean quality
Ej (E)
Median quality
median
{ (b)
(F) 7
Rescaled mean quality
d-best quality
Σ 1 11 (E) 2 1 ))
7.3.2
Run time
We computed two scores for the runtimes of encoders. First is the time necessary to encode the dataset. The outcome, displayed in figure 4a that GLMM- encoders are the slowest. This happens because the bottleneck of these encoders is the fitting of the random intercept model, problem that we could only partially alleviate with our custom implementation. As expected, the model has no influence on the runtime.
Second, the time necessary tune the model-encoder pipeline, each tuning step requiring encoding the dataset and then fitting a model. FigureH4htellss a similar story as for encoding, with GLMM- encoders being the slowest. The other encoders, apart from Drop and Mean-Target, all show similar runtime.
10a
seconds
10
109
MT
Ord
MH
D,MT
D:MT
D1MTT
Sum
PBa. MTT
PB
BU:MT
CV3MT
Bin
WoB
Count
CV:MT
BUgMT
Co
BUGMTT
ME,
ME
ME
GLMM
BU:GLMM
CV.GLMM
BU:GLMM
CV.GLMM
GLMMM
mGLMMM
Encoding
10
Reconda
10
10"
Drop
MI
DTMT
DOMT
BU
D&MI
Ond
PB
MH
PBa.1M MT
CV2MT
00)
CV&MT
Bin
OH
WaE
MU
Sum
CB
CV MT
MEa
ME
MEj
BU
GLMM
CV.GLMM
BU,CLMM
W.GLMM
BU.GLMM
CV16GLMM
BUGGGLMM
(b) Tuning
Figure 4: Runtime of encoders (a) and full tuning pipelines (b).
18

---

7.3.3 Replicability
This section extends the replicability analysis of Section 5.2 showing the behavior of different quality metries in Figure and aggregation strategies Figure 5b
The quality metrics behave similarly for p-replicability. The notable exception is the AUC in model tuning, which is significantly better than the other metrics. Regarding J J-replicability, instead, accuracy is clearly the poorer choice. This hints that accuracy cannot discern the best encoder as well as the other metrics do and that is more sensitive the choice of dataset.
Among the aggregation strategies, rank best (R-B) shows higher replicability. possible explanation is that R-B produces consensus rankings with many encoders tied the best ones and few tiers in general.
Acc
AUC
BAcc
F1
no tuning
model tuning
full tuning
0.6
0.4
0.2
0.0
0.5
0.3
0.1
20
15
25
5
sample size
sample size
sample size
(a) Quality metric
Q-M
Q-RM
R-M
R-B
Q-Md
Q-Tho.95
R-Md
R-W
no tuning
model tuning
full tuning
0.75
0.50
0.25
Aommor
0.00
0.9
0.6
0.3
0.0
5
10
15
20
25
5
10
15
20
25
5
10
15
sample size
sample size
sample size
(b) Aggregation strategy
Figure 5: Average similarity of consensus rankings from disjoint subsets of datasets, conditional on (a) quality metric and (b) aggregation strategy.
19

---

7.3.4 Comparing encoders
This section expands on Section 5.3 and portrays Figure distribution of ranks of encoders. The best encoders are evident for LogReg (Sum, OH, WoE, Bin) and k-NN (WoE), confirmed by Nemenyi tests at 0.05 significance.
DT
LGBM
SVM
WoE
OH
OH
P MT
WoE
Bin
Count
Bin
MU
ooo MT
MH
Sum
PB MT
MT
BU,MT
CV2MT
CV&MT
BU.MT
MeMT
BUTMT
GLMM
PB
MT
PB PBogn MT
PB MT
&&MT MT
CV,GLMM
MEJ
Sum
BU:MT
PRacor M
GLMM
DIOTT
DT
MU
DcMT
ME
D.MT
O GLMM
BUGGLMM
ME
PBg MT
CV,MT
D.MT
BUGGLMM
D.MT
MT
CV&GLMM
MT
ME
MT
CV
Di
CV
GLMM
BU:MT
BU;GLMM
BU:GLMM
D.MT
CV,MT
ME
CV GLMM
D.MT
CV>GLMM
GLMM
GV.GLMM
BULGLMM
Sum
CV16GLMM
CV CVIGGLMM
CB
BU.MT
WoB
PBp MT
CV&GLMM
Ord
CVMGLMM
BU;GLMM
ME
BU,GLMM
ME
MH
CB
MFa
Drop
Drop
Drop
Count
10
20
30
0
10
20
30
10
20
30
rank
rank
rank
(a) DT
(b) LGBM
(c) SVM
k-NN
LogReg
all models
WoE
Sum
WoE
OH
OH
WoE
Sum
Bin
Sum
BU-MT
BU
BU.MT
PR
GLLM
MT
BU-MT
CV
CV
BU-MT
PB MT
BU.GLMM
CV.MT
ME
CV
oo MT
BU.MTT
PBea MT
CV CV
BU,GLMM
PBamn MT
GL
PB MT
DT
CV.MT
ME;
GLMM
D.MT
CV
D.MT
DiMMT
DI
MT
ME
CV.MT
MT
D.MT
MEa
BUGGLMM
MEG
BU:GLMM
ME
MT
MF;
D.MT
GLMM
ME
GLMM
PBo PT MT
D:MT
ME
D-
Ord
BU-GLMM
BU,GLMM
CV;GLMM
PB MT
CB
CB
CV₂GL.MM MH
CV,GLMM
GLMM
GLMM
CV-GLMM
CV&GLMM
CV CVLLLMM
CV GLM
CV CV
MH
MH
CB
Ord
MT
Ord
Count
Count
Count
Drop
Drop
Drop
30
20
30
10
rank
rank
rank
(d) K-NN
(e) LogReg
(I) models
Figure 6: Ranks of encoders.
20

---

7.3.5 Effect of tuning
This section investigates whether tuning leads to improvements in pipeline performance. The tuning strategies are described in Section 4.4 a pair of tuning strategies, we consider the factors they share and subtract the performance of the pipelines. Figure shows that full tuning is, in general, advantageous over no tuning and slightly better than model tuning.
full VS
full VS model
full VS
full VS model
0.10
0.04
0.05
0.02
0.00
0.00
-0.05
-0.02
DF
K-NN
LogReg
SVM
DT
NNN
LogReg
Acc
AUC.
BAcc
Fi
Ace
AUC
BAcc
(a) Model
(b) Scoring
0.04
0.02
0.00
0.002
Drop
MI
Ond
Bin
ME
MES
PBa0jMT
CV1MMT
Count
MEA.
Wol:
PBO MT
PB0.20/
OH
BU:MT
CV3MT
BUGMT
D
BU, BU,GLME CGLMM
GLMM
GLMMM
DioMT
D.MT
BU GLMM
BU-GLMM
BU
EV.GL.MM
CV3GLMM
CB
(c) Encoder full tuning VS no tuning
0.04
0.00
0.00
Drop
MI
ME3o
WoBl
BUSMT
BUmMT
MEj
Ond
DaMT
ME
OH
BUSMT
PRoaLMT
PB
MT
D&MT
DAMT
CVaMM
CV3MT
CV2MT
CB
BU,GLMM
CV.
VLMM
BU.GLMM
BU WU MM GLMM
CV WGMMM
(d) Encoder full tuning VS model tuning
Figure 7: Performance gain of full uning over no tuning and model tuning.
21


Reviews:
[{"review_id": "HInF4H6M9A", "reviewer": "Reviewer_siZ7", "summary": "", "questions": "", "limitations": "Authors have mentioned the limitations of their work.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1. More datasets (50 datasets) are used than existing benchmarks.\n\nS2. More factors are considered to measure the encoders, the quality metrics, the ML models used, the tuning strategy, the aggregation strategy.\n\nS3. The motivation and advantage are introduced clearly.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "5ZBuaFVkjF", "reviewer": "Reviewer_N1YT", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The evaluation is extensive and the experimental setup is varied across many different axes. The paper not only reports on the final ranking of encoding strategies, but clearly shows the various ways the effect of certain experimental design decisions can affect the rankings. The figures are easy to parse, and the text is well-written. It benchmarks many different encoding strategies.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Zbu0l64QGh", "reviewer": "Reviewer_QEyt", "summary": "", "questions": "", "limitations": "N/A", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "N/A", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors come up with an extensive benchmark of categorical encoders for binary classification. This is an important problem, where not much research has been done towards.

The average score is misleading: one reviewer did leave a 1 (because the anonymity) and was not available for correcting this mistake. 

It is hard to assess the quality of the reviews, since eventually only two reviewers have submitted a valid review. Additionally, one of these is quite shallow, and did not report back to the authors. Therefore, I will mostly base my assessment on the review of reviewer N1YT (who seems to have an excellent track record in benchmarking studies) and my own observations from reading the paper and assessing the benchmark.

While none of the reviewers identified dealbreakers or matters that should be improved (except for the discussion on AUC, which has been resolved) there remain some questions (also raised by reviewer N1YT) about the usability of the repository, and whether other people can easily analyze the results. Why not integrate these results with, for example, OpenML.org (the platform where all datasets came from)? This point was resolved by the remark that the readme will be further improved in the future, but in my opinion this is also something that should have been taken more seriously.

Finally, I would like to comment on the match of this work in the datasets and benchmark track. When looking at the scope of this track, it seems that this part would fall under "Benchmarks on new or existing datasets, as well as benchmarking tools" (on the part: existing). It seems that the authors have compiled a set of existing datasets, and benchmarked a wide range of categorical encoders on it. However, I personally would expected either a) a better integration with existing tools (such as the OpenML benchmark suites), such that other people could build upon this, or b) more clear analysis of the results, with clear conclusions.

Taking all into consideration, I believe that this paper could receive the benefit of the doubt, and can be recommended for a poster presentation at NeurIPS.
Decision: Accept (Poster)