SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3b features the encoder ranks for logistic regression, where four encoders, namely One-Hot, Sum, Binary, and Weight of Evidence, consistently achieve higher ranks compared to the others. Nemenyi tests confirm that this difference in ranks is significant. These results are in line with the ones from Section 5.2, which indicate low replicability of the results for decision trees and higher replicability for logistic regression.
Figure 3c presents the ranks of encoders across all datasets, ML models, quality metrics, and tuning strategies. Similarly to logistic regression, One-Hot, Sum, Binary, and Weight of Evidence consistently achieve significantly higher average ranks compared to the other encoders, again confirmed by Nemenyi tests. We recommend these four encoders as the preferred choices in practical applications. This conclusion contradicts other studies reporting a suboptimal performance of One-Hot [5, 28].
Our findings also reveal that Drop performs significantly worse than all other encoders, i.e., encoding categorical attributes generally yields better results than dropping them.
5.4 Comparing to related work
In this section, we compare our results with the findings of other studies. To do so, we select subsets of our results that mimic the experimental settings in related work. In [28],  $\mathrm{CV}_5\mathrm{GLMM}$  outperformed every competitor for boosted trees and k-NN, while GLMM was recommended for SVMs. However, in our experiments, Sum outperformed GLMM for SVMs, One-Hot did better than  $\mathrm{CV}_5\mathrm{GLMM}$  for boosted trees, and  $\mathrm{CV}_{10}\mathrm{GLMM}$  was better than  $\mathrm{CV}_5\mathrm{GLMM}$  for k-NN. Next, while in [5] similarity encoders are better than One-Hot for boosted trees, subsequent research reported no significant difference between Min hash and One-Hot on medium-sized tabular datasets [4]. Our findings are in line with this latter result, as we could not find a performance difference between the two encoders with a t-test with a significance level of 0.05. In [41], Sum is reported as the best encoder on the Adult dataset for boosted trees, while a Data-constraining encoder is reported as the worst. With the same setting, we did not find a significant performance difference for any encoder except for Drop, which
(a) Decision tree
Figure 3: Ranks of encoders.
(b) Logistic regression
(c) All models
performed the worst. On the Bank marketing dataset, [8] showed that One-Hot and Mean-Target outperformed Binary with logistic regression. In our experiments, Binary was slightly worse than One-Hot and Mean-Target. In [44], Dummy, an identifier encoder similar to One-Hot, was better than Mean-Target on the Tic-tac-toe dataset with boosted trees. We, instead, did not observe any significant difference between One-Hot and Mean-Target for these factors.
6 Limitations and conclusions
Limitations. First, we treated encoders as part of the pre-processing, but certain encoders can be an integral component of specific ML models. For instance, CatBoost is derived from the homonymous boosted trees algorithm, which re-encodes the data multiple times during training. Second, we applied a single encoder to all categorical attributes. Using different encoders based on the cardinality of the attribute may sometimes yield favorable results [28, 4]. However, the selection of the optimal encoder for each attribute requires either domain knowledge of the attribute or purpose-built tools, which falls outside the scope of our benchmark and is therefore left as future work. We also did not include neural networks, due to the absence of a recommended architecture and reported interior performance to tree-based models on tabular data [15].
In this study, we conducted an extensive evaluation of encoder performance across various experimental factors, including ML models, quality metrics, and tuning strategies. Our results demonstrate a high sensitivity of encoder rankings to these factors, both for the full rankings and the best-performing encoders. This sensitivity explains the inconsistent results among related studies, as different choices in any of these factors can lead to different outcomes. We also assessed the impact of aggregation strategies on consensus rankings, revealing significant variations in rankings depending on the chosen strategy. This emphasizes the importance of carefully considering the aggregation method when post-processing and interpreting results. Regarding replicability, we defined and quantified it using  $\rho$ -replicability and  $J$ -replicability. Our findings indicate that replicability is influenced by factors such as the ML model, with logistic regression exhibiting the highest replicability and decision trees
the lowest. Additionally, larger dataset samples tend to yield more reliable experimental outcomes, although this trend does not always hold for  $J$ -replicability. Based on our results, we recommend specific encoders for practical applications. For decision trees, Weight of Evidence performed the best, although statistical tests did not show a significant difference from other encoders. For logistic regression, Sum, One-Hot, Binary, and Weight of Evidence consistently achieved higher ranks, with statistically significant differences from other encoders. These findings contradict previous studies, highlighting the importance of considering a broad range of experimental factors. Finally, our comparative analysis with related work revealed discrepancies in encoder performance, suggesting that the breadth of our study may contribute to these differences. This emphasizes the need for caution when interpreting results from studies with more limited experimental settings. Overall, our study provides valuable insights into the sensitivity of encoder performance to experimental factors, as well as recommendations for practical encoder selection across different scenarios.
We thank Dmitriy Simakov for valuable discussions and Natalia Arzamasova for her algorithm and implementation of the PreBinnedEncoder. This work was supported in part by the German Research Foundation (Deutsche Forschungsgemeinschaft), project Charakterisierung, Modellierung und Homogenisierung von Vernetzungswerken mit Hilfe interpretierbarer Datenanalysemethoden, and by the State of Baden-Württemberg, project Algorithm Engineering für die Scalability Challenge.
[1] David Aha. Tic-Tac-Toe Endgame. UCI Machine Learning Repository. 1991.
[2] Xavier Bouthillier et al. "Accounting for Variance in Machine Learning Benchmarks". In: MLSys. mlsys.org, 2021.
[3] Laurent Candillier and Vincent Lemaire. Nomao. UCI Machine Learning Repository. 2012.
[4] Patricio Cerda and Gael Varouquaux. "Encoding High-Cardinality String Categorical Variables". In: IEEE Trans. Knowl. Data Eng. 34.3 (2022), pp. 1164–1176.
[5] Patricio Cerda, Gael Varoquaux, and Balázs Kégl. "Similarity encoding for learning with dirty categorical variables". In: CoRR abs/1806.00979 (2018).
[6] Congressional Voting Records. UCI Machine Learning Repository. 1987.
[7] Don Coppersmith, Se June Hong, and Jonathan R. M. Hosking. "Partitioning Nominal Attributes in Decision Trees". In: Data Min. Knowl. Discov. 3.2 (1999), pp. 197-217.
[8] Mwamba Kasongo Dahouda and Inwhee Joe. "A Deep-Learned Embedding Technique for Categorical Features Encoding". In: IEEE Access 9 (2021), pp. 114381-114391.
[9] Mostafa Dehghani et al. "The Benchmark Lottery". In: CoRR abs/2107.07002 (2021).
[10] Janez Demsrar. "Statistical Comparisons of Classifiers over Multiple Data Sets". In: J. Mach. Learn. Res. 7 (2006), pp. 1-30.
[11] Keyu Duan et al. "A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking". In: NeurIPS. 2022.
[12] Bob Evans. Cylinder Bands. UCI Machine Learning Repository. 1995.
[13] Farhad Soleimanian Gharehchopogh and Seyyed Reza Khaze. "Data mining application for cyber space users tendency in blog writing: a case study". In: CoRR abs/1307.7432 (2013).
[14] Sebastian Gnat. "Impact of Categorical Variables Encoding on Property Mass Valuation". In: KES. Vol. 192. Procedia Computer Science. Elsevier, 2021, pp. 3542-3550.
[15] Léo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?" In: NeurIPS. 2022.
[16] C. Harley, R. Reynolds, and M. Noordewier. Molecular Biology (Promoter Gene Sequences). UCI Machine Learning Repository. 1990.
[17] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository. 1994.
[18] Ronald Iman and James Davenport. “Approximations of the critical region of the Friedman statistic”. In: Communications in Statistics-Theory and Methods 9 (Jan. 1980), pp. 571–595.
[19] Andras Janosi et al. Heart Disease. UCI Machine Learning Repository. 1988.
[48] Maciej Zieba et al. "Boosted SVM for extracting rules from imbalanced data in application to prediction of the post-operative life expectancy in the lung cancer patients". In: Appl. Soft Comput. 14 (2014), pp. 99-108.
7 Appendix
7.1 Encoders
This section presents a reproducible description of the encoders discussed in Section 3, following the structure outlined below. We discuss identifier, frequency-based, contrast, and simple target encoders together in Appendix 7.1.4, as all of these encoders can be explicitly represented as functions. Similarity, binning, smoothing, and data-constraining encoders have dedicated sections. Table 5 contains the notation used in this section.
Table 5: Notation for section 7.1.
Symbol	Meaning
N0	natural numbers including 0
(x)2	base-2 representation of x ∈ N0
Xn×d	set of matrices with entries in X, n rows and d columns
1	indicator function
D	binary classification dataset
n	number of rows of D
y ∈ {0,1}n	target attribute of D
ΩA = {ωl}L	categorical domain (strings)
A ∈ ΩAn	categorical attribute of D to be encoded
li ∈ {1,...,L}	such that A_i = ωli
E: A → M ∈ Rn×d	encoder
M ∈ Rn×d	encoding of A, compact notation
E(A) ∈ Rn×d	encoding of A with explicit encoder
E(ΩA)	unique values of rows of E(A)
d = d(E, A)	number of columns of M
Mi	i-th row of M if d > 1, i-th component of M if d = 1
l ∈ {1,...,L}	index of levels
i, h ∈ {1,...,n}	row indices of M, A, or y
j ∈ {1,...,d}	column index of M
7.1.1 Similarity encoders [5, 4]
Min-Hash treats  $\omega \in \Omega_A$  as a string, splits it into its set of character-level n-grams (substrings of n-consecutive characters), uses a hash function to encode each n-gram into an integer, and finally encodes  $\omega$  with the minimum value of the hash function on the set of n-grams. The process is repeated for  $d$  hash functions, yielding  $M \in \mathbb{R}^{n \times d}$ . The default value of  $d$  is 30, the authors report good performance with  $300^6$ .
7.1.2 Binning encoders
Pre-Binned Mean-Target partitions  $\Omega_A$  into  $B$  buckets  $\{P_b\}_{b=1}^B$  to solve the optimization problem
Maximize  $B$
$$
\text {s u b j e c t} \frac {1}{n} \sum_ {\omega \in P _ {b}} \sum_ {i = 1} ^ {n} \mathbb {1} (\mathbf {A} _ {i} = \omega) \geq \vartheta \quad \forall b \leq B
$$
where  $\vartheta \in [0,1]$  is a user-defined threshold. Each bucket is then treated as a new level and encoded with Mean-Target, yielding an encoding  $M\in \mathbb{R}^n$
Discretized Mean-Target partitions  $MT(\Omega_A)$  into intervals  $\{I_1,\dots ,I_B\}$  of equal length. Letting  $I(l)$  be the interval that contains  $MT(\omega_l)$  (that is, the average target associated to  $\omega_{l}$ ), the encoding is  $\mathbf{M}\in \mathbb{R}^n:\mathbf{M}_i = \inf I(l_i)$ . We experimented with  $B = 2,5,10$
$^{6}$ https://dirty-cat.github.io/stable/generated/dirty_cat.MinHashEncoder.html
Table 6: Identifier encoders
E(ΩA)	E(A)
Binary [8]	{0,1}[log2(L)]+1	Mi=(li)2
Dummy [28, 44]	{0,1}L-1	Mij={1(Ai=ωj)0j≠L0j=L
One-Hot [28, 4, 8, 44, 41]	{0,1}L	Mij=1(Ai=ωj)
Ordinal [28, 44, 41]	N0	Mi=li
Table 7: Frequency-based encoders
Count [36]	N0	Mi = ∑j 1 (Aj = ωl)
Frequency [28]	R	Mi = 1/n ∑j 1 (Aj = ωl)
Table 8: Contrast encoders — without intercept
Sum [41]	RL-1	Mij = {1(Ai = j) j ≠ L -1 j = L
Backward difference [41]	RL-1	Mij = {−L-i/L i ≤ j 1/L i > j
Helmert [41]	RL-1	Mij = {−1/j+1 i ≤ j 1/j+1 i = j+1 0 i ≥ j+2
Table 9: Simple target encoders
Mean-Target [28, 8]	R	Mi = ∑n h=1 yh1(Ah = ωl i)
Weight of Evidence [39] [28]	N0	Mi = log(MT(A)/1-MT(A))
7.1.3 Smoothing target encoders
Mean-Estimate [41]. Let  $n_l = \sum_{i=1}^n \mathbb{1}(A_i = \omega_l)$  be the number of occurrences of  $\omega_l$  in  $\mathbf{A}$ .
$$
\mathbf {M} _ {i} = \frac {n _ {l _ {i}} M T (\omega_ {l _ {i}}) + \frac {w}{n} \sum_ {i = 1} ^ {n} y _ {i}}{w + n _ {l _ {i}}}
$$
where  $w$  is a user-defined weight. Common choices are 1, 10.
GLMM [28] fits, for every  $\omega_{l}\in \Omega_{A}$ , a random intercept model
$$
y _ {i} = \beta_ {l _ {i}} + u _ {l _ {i}} + \varepsilon_ {i}
$$
where  $u_{l}\sim N(0,\tau^{2})$  and  $\varepsilon_{i}\sim N(0,\sigma^{2})$ . The encoding is  $\mathbf{M}\in \mathbb{R}^n:\mathbf{M}_i = \beta_{l_i}$
7.1.4 Identifier, frequency-based, contrast, simple target encoders
The descriptions are divided as follows: Table 6 is for identifier encoders, Table 7 is for frequency-based encoders, Table 8 is for contrast encoders, and Table 9 is for simple target encoders.
7.1.5 Data-constraining encoders
CatBoost [41] uses a permutation  $\pi$  of  $\{1, \dots, n\}$  and encodes with  $\mathbf{M} \in \mathbb{R}^n$  such that
$$
\mathbf {M} _ {\pi (i)} = \sum_ {h \leq \pi (i)} y _ {h} \mathbb {1} \left(A _ {h} = \omega_ {l _ {\pi (i)}}\right)
$$
Table 10: Notation for section 7.2.2.
Symbol	Meaning
⊥	missing evaluation or rank
1	indicator function
Ei	encoder as in table 5
Φj: Ei → R ∪ {⊥}	average cross-validated quality on the j-th dataset, all other factors fixed
Φmaxj = maxi=1,...,n{Φj(Ei)}	best quality on the j-th dataset, all other fac-tors fixed
Φminj = mini=1,...,n{Φj(Ei)}	worst quality on the j-th dataset, all other factors fixed
rj: E → N0 ∪ {⊥}	ranking obtained from Φj
Rj = (1 (rj(Ei) ≤ rj(Eh))i,h=1n ∈ {0,1}n×n	adjacency matrix of rj
c: E → N0 ∪ {⊥}	consensus ranking
C ∈ {0,1}n×n	adjacency matrix of c
i,h,k ∈ {1,...,n}	index of encoders
j ∈ {1,...,m}	index of objects to be aggregated
Cross-Validated MT [28] randomly partitions  $\{1, \dots, n\}$  in  $k$  folds of equal size. Let  $D_{a_i}$  be the fold that contains  $i$ . Then, every fold is encoded with Mean-Target trained on the other  $k - 1$  folds:
$$
\mathbf {M} _ {i} = \sum_ {h = 1} ^ {n} \mathbb {1} (h \notin D _ {a _ {i}}) \mathbb {1} (\mathbf {A} _ {h} = \omega_ {l _ {i}}) y _ {h}
$$
Common values for  $k$  are 2, 5, 10.
Cross-Validated GLMM [28] works in a similar fashion as CVMT: it encodes each fold with GLMM trained on the other  $k - 1$  folds.
BlowUp Cross-Validated MT randomly partitions  $\{1, \ldots, n\}$  in  $k$  folds  $D_{1}, \ldots, D_{k}$  of roughly equal size. Then, it encodes with  $\mathbf{M} \in \mathbb{R}^{n \times k}$  so that the  $j$ -th column is  $\mathbf{A}$  encoded with Mean-Target trained on the  $j$ -th fold, yielding
$$
\mathbf {M} _ {i j} = \sum_ {h = 1} ^ {n} \mathbb {1} (h \in D _ {j}) \mathbb {1} (\mathbf {A} _ {h} = \omega_ {l _ {i}}) y _ {h}
$$
We experimented with  $k = 2,5,10$
Blowup Cross-Validated GLMM is analogous to BUMT, but the  $j$ -th column of its encoding  $\mathbf{M}$  is encode with  $\mathbf{M} \in \mathbb{R}^{n \times k}$  so that the  $j$ -th column of  $\mathbf{M}$  is  $\mathbf{A}$  encoded with GLMM trained on the  $j$ -th fold.
7.2 Experimental design
This section provides additional details about the datasets and aggregation strategies we discussed in Section 4. The notation we use in this section is summarized in Table 10.
7.2.1 Datasets
Table 11 lists the datasets used in our experiments. The columns are as follows: ID is the OpenML identifier;  $n$  is the number of rows;  $d$  is the number of attributes;  ${d}_{\text{cat }}$  is the number of categorical attributes;  $\max \left| {\Omega }_{A}\right|$  is the maximum categorical attribute cardinality; the "ft" flag denotes datasets used for full-tuning (cf. Section 4.4)
Table 11: Datasets used in the study.
Name	Ref.	ID	n	d	\( d_{cat} \)	max \( |\Omega_A| \)	ft
ada_prior	1037	4562	14	7	40	✓
adult	1590	48842	14	7	42	✓
airlines	1169	539383	7	4	293
amazon.employee_access	4135	32769	9	9	7518
Agrawal1	1235	1000000	9	3	20
Australian	[33]	40981	690	14	4	14	✓
bank-marketing	[24]	1461	45211	16	6	12
).(blogger	[13]	1463	100	5	3	5	✓
Census-Income-KDD	42750	199523	41	27	51
credit-approval	[32]	29	690	15	6	15	✓
credit-g	[17]	31	1000	20	11	10	✓
cylinder-bands	[12]	6332	540	37	17	71	✓
dresses-sales	[40]	23381	500	12	11	25	✓
heart-h	[19]	51	294	13	6	4
ibm-employee-attrition	43896	1470	34	5	9	✓
ibm-employee-performance	43897	1470	33	5	9	✓
irish	451	500	5	2	11	✓
jungle_chess_2pcs...elephant	[35]	40999	2351	46	2	3	✓
jungle_chess_2pcs...lion	[35]	41007	2352	46	2	3	✓
jungle_chess_2pcs...rat	[35]	41005	3660	46	2	3	✓
kddinternet_USAGE	981	10108	68	20	129	✓
KDDCup09_appetency	1111	50000	230	33	15416
KDDCup09_churn	1112	50000	230	33	15416
KDDCup09_upselling	1114	50000	230	33	15416
KDD98	42343	82318	477	107	18543
kr-vs-kp	[37]	3	3196	36	1	3	✓
kick	41162	72983	32	17	1063
law-school-admission-bianry	43890	20800	11	1	6
molecular-biology_promoters	[16]	956	106	57	56	4	✓
monks-problems-1	[43]	333	556	6	4	4	✓
monks-problems-2	[43]	334	601	6	4	4	✓
mv	881	40768	10	1	3	✓
mushroom	[25]	43922	8124	22	16	12	✓
national-longitudinal-survey-binary	[6]	43892	4908	16	4	29	✓
nomao	[3]	1486	34465	118	27	3
nursery	959	12960	8	7	5
openpayments	42738	73558	5	4	4374
porto-seguro	41224	595212	57	13	104
profb	470	672	9	3	28	✓
sick	[34]	38	3772	29	2	5	✓
sf-police-incidents	42344	538638	6	5	21838
SpeedDating	40536	8378	120	58	260	✓
students Scores	43098	1000	7	2	6	✓
telco-customer-churn	42178	7043	19	11	6531
thoracic-surgery	[48]	1506	470	16	3	7	✓
tic-tac-toe	[1]	50	958	9	9	3	✓
Titanic	40945	1309	13	6	1307
vote	[6]	56	435	16	16	3	✓
wholesale-customers	1511	440	8	1	3	✓
WMO-Hurricane-Survival-Dataset	43607	5021	22	21	4173
7.2.2 Aggregation strategies
This section presents the mathematical formulations of the aggregation strategies. As Section 4.5 explains, the results are aggregated across datasets, while keeping the other factors — ML model, tuning strategy, and quality metric — fixed.
Heuristics. Heuristics aggregate by ranking encoders according to some score. Increasing heuristics assign the best rank to the encoder with the highest score, while the non-increasing ones assign the best rank to the encoders with the lowest score. Table 12 contains the respective formulas. Any missing evaluations  $(\perp)$  are ignored during the computation.
Friedman-Nemenyi tests. The Friedman test is used to rule out the null hypothesis that all encoders have, on average, the same rank. The Friedman statistic adjusted for ties [38, 18] is
$$
T = \frac {(m - 1) (S _ {t} - C)}{S _ {r} - C}
$$
where  $S_{r} = \sum_{i = 1}^{n}\sum_{j = 1}^{m}r_{j}(E_{i})^{2}, S_{t} = \frac{1}{m}\sum_{i = 1}^{n}\left(\sum_{j = 1}^{m}r_{j}(E_{i})\right)^{2}$ , and  $C = \frac{1}{4} mn(n + 1)^2$
Under the null hypothesis that all encoders have the same rank,  $T$  is approximately distributed as an  $F$ -distribution with  $n - 1$  and  $(m - 1)(n - 1)$  degrees of freedom.
If the Friedman hypothesis is rejected, one can compare all pairs of encoders with  $n(n - 1) / 2$  Nemenyi post-hoc tests [10]. Nemenyi tests apply a correction to control the error from testing multiple hypotheses. Two encoders  $E_{1}$  and  $E_{2}$  are significantly different if
$$
\frac {1}{m} \sum_ {j = 1} ^ {m} \left(r _ {j} (E 1) - r _ {j} (E _ {2})\right) \geq q _ {\alpha} \sqrt {\frac {n (n + 1)}{6 m}}
$$
where  $\frac{1}{\sqrt{2}} q_{\alpha}$  is the critical value based on a Studentized range statistic [10].
Kemeny-Young aggregation. [21, 47, 45, 23]
The consensus's adjacency matrix  $\mathbf{C}$  is a solution to the mixed-integer linear problem
$$
\text {M a x i m i z e} \sum_ {i, h} \mathbf {S} _ {i h} \left(2 \mathbf {C} _ {i h} - 1\right)
$$
subject to  $\mathbf{C}_{ih} - \mathbf{C}_{kh} - \mathbf{C}_{ik}\geq -1$
$\mathbf{C}_{ih} + \mathbf{C}_{hi}\geq 1$ $\forall i <   h$
$\mathbf{C}_{ih}\in \{0,1\}$  （20  $\forall i,h$
where  $\mathbf{S} = \left(\sum_{j}\frac{\mathbf{R}_{ih}^{j}}{n_{j}(n_{j} - 1)}\right)_{i,h = 1}^{n}$  is a cost matrix and  $n_j = \sum_i\mathbb{1}(r_j(E_i)\neq \bot)$  is the number of encoders with evaluation on dataset  $j$ . This formulation accounts for ties and missing ranks.
7.3 Results
This section complements Section 5.
7.3.1 Missing evaluations
We successfully completed 61812 runs out of 64000, one per combination of encoder, dataset, ML model, tuning strategy, and quality metric. The 2188 failed evaluations are equally distributed among the encoders, while tuning was the greatest influencing factor. Indeed, there were 4303 missing runs with no tuning, 1152 with model tuning, and 32 with full tuning. This is likely due to the bigger datasets used in no tuning and model tuning, cf. Section 4.4 and Table 11. The total runtime for successful evaluations was 108 days.
Table 12: Scores of heuristics.
Score of E	Increasing
Mean rank	1/m∑j rj(E)
Median rank	median{rj(E)}m j=1
Rank best	∑j=1m 1 (rj(E)=1)	✓
Rank worst	∑j=1m 1 (rj(E)≠ max i=1,...,m r(Ej))
Mean quality	1/m∑j Φj(E)	✓
Median quality	median{Φj(E)}m j=1	✓
Rescaled mean quality	1/m ∑j=1m Φj(E)-Φjmin/Φjmax-Φjmin	✓
θ-best quality	∑j=1m 1 (Φj(E)≥θ·Φjmax)	✓
7.3.2 Run time
We computed two scores for the runtimes of encoders. First is the time necessary to encode the dataset. The outcome, displayed in figure 4a, is that GLMM-based encoders are the slowest. This happens because the bottleneck of these encoders is the fitting of the random intercept model, a problem that we could only partially alleviate with our custom implementation. As expected, the model has no influence on the runtime.
Second, the time necessary to tune the model-encoder pipeline, each tuning step requiring encoding the dataset and then fitting a model. Figure 4b tells a similar story as for encoding, with GLMM-based encoders being the slowest. The other encoders, apart from Drop and Mean-Target, all show a similar runtime.
(a) Encoding
(b) Tuning
Figure 4: Runtime of encoders (a) and full tuning pipelines (b).
7.3.3 Replicability
This section extends the replicability analysis of Section 5.2, showing the behavior of different quality metrics in Figure 5a and aggregation strategies in Figure 5b.
The quality metrics behave similarly for  $\rho$ -replicability. The notable exception is the AUC in model tuning, which is significantly better than the other metrics. Regarding  $J$ -replicability, instead, accuracy is clearly the poorer choice. This hints that accuracy cannot discern the best encoder as well as the other metrics do and that it is more sensitive to the choice of dataset.
Among the aggregation strategies, rank best (R-B) shows higher replicability. A possible explanation is that R-B produces consensus rankings with many encoders tied as the best ones and few tiers in general,
(b) Aggregation strategy
Figure 5: Average similarity of consensus rankings from disjoint subsets of datasets, conditional on (a) quality metric and (b) aggregation strategy.
7.3.4 Comparing encoders
This section expands on Section 5.3 and portrays in Figure 6 the distribution of ranks of encoders. The best encoders are evident for LogReg (Sum, OH, WoE, Bin) and k-NN (WoE), confirmed by Nemenyi tests at 0.05 significance.
(a) DT
Figure 6: Ranks of encoders.
7.3.5 Effect of tuning
This section investigates whether tuning leads to improvements in pipeline performance. The tuning strategies are described in Section 4.4 For a pair of tuning strategies, we consider the factors they share and subtract the performance of the pipelines. Figure 7 shows that full tuning is, in general, advantageous over no tuning and slightly better than model tuning.
(a) Model
(b) Scoring
(c) Encoder — full tuning VS no tuning
(d) Encoder — full tuning VS model tuning
Figure 7: Performance gain of full tuning over no tuning and model tuning.