SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Setup We compare STreeD with the MurTree algorithm from [19] on a set of binarized datasets with binary class as described in Table 7. The datasets are originally from the UCI repository [24] and from [6, 30, 41, 59, 81, 96, 97]. We use the binarized datasets from the MurTree repository. For each dataset, both algorithms need to find a tree of depth three and four that optimizes F1-score. We repeat the tests five times and report the average runtime.
Results Table 7 shows how STreeD compares to MurTree on optimizing nonlinear metrics. By using the geometric mean of the relative performance [31], we can see that for trees of depth three, STreeD is on average 7.1 times faster than MurTree; for trees of depth four, it is on average 6.8 times faster. This performance difference can be partly explained by the new upper and lower bound technique, as explained in Appendix B.
This same method could also be used to optimize other nonlinear metrics that are monotonic with respect to the false positive and negative scores, such as Matthews correlation coefficient.
F Group Fairness
In many cases, removing sensitive features is not sufficient to prevent discrimination [28, 70], and therefore fairness constraints are imposed on ML models. We here consider two versions of group fairness: demographic parity and equality of opportunity [16, 60]. We leave individual fairness notions as future work.
Group fairness metrics impose equality on groups defined by some sensitive feature. Demographic parity requires equal probability of a positive result for each group, and equality of opportunity requires equal true positive rate for each group. Both of these can be optimized using STreeD by adding the threshold constraints as defined in Eq. (11) for demographic parity and Eq. (71) for equality of opportunity, as introduced below.
The next section provides more related work on incorporating fairness constraints in decision tree search. We then derive how threshold constraints can be used to model fairness constraints in STreeD. The final section shows experimental results for imposing fairness constraints in STreeD.
<https://bitbucket.org/EmirD/MurTree-bi-objective>
F.1 Related Work
Fairness in decision trees was first considered by Kamiran and Calders [46] who proposed to preprocess the data to guide the decision tree training process to less discriminating trees. Later, Kamiran et al. [47] proposed new splitting criteria to enable top-down induction using a mix of information gain in the outcome label and the sensitive feature. However, they concluded that this was not sufficient and instead proposed after-the-fact relabeling to get good results.
Fairness constraints were first considered in MIP models for optimal decision trees by Verwer and Zhang [85]. Aghaei et al. [1] later developed a MIP model for optimal trees for either a group fairness or individual fairness constraint. Jo et al. [44] propose a new MIP model based on a max-flow formulation that scales better than previous models.
These MIP models, however, still do not scale well beyond small datasets and small tree depths. Therefore Wang et al. [90] propose to break down the problem and use MIP as a look-ahead procedure that decides on the best feature to split on, but does not solve the whole problem as a monolith. Their experiments show better scalability, but their method is no longer guaranteed to return optimal solutions.
Van der Linden et al. [53], on the other hand, propose an optimal DP method that optimizes a global fairness constraint by considering upper and lower bounds on the final discrimination value for subtrees to enable early comparison and pruning. Their results show order-of-magnitude improvements on runtime, and their method can handle much larger datasets than both the MIP methods and the iterative approach.
F.2 Optimization Task Formulation
In this section we derive threshold constraints for demographic parity and equality of opportunity.
Demographic parity As stated in the main text, demographic parity means that the probability of the positive outcome is the same for people of different groups:
$$
P (\hat {y} = 1 \mid a = 1) = P (\hat {y} = 1 \mid a = 0) \tag {67}
$$
Typically, this is simplified to limiting the difference between the probabilities to some small percentage  $\delta$ . Let  $\hat{N}(y,a)$  be the number of group  $a$  receiving the positive outcome, and  $\hat{N}(\bar{y},a)$  the number of group  $a$  receiving the negative outcome, etc. Then Eq. (67) becomes:
$$
\left| \frac {\hat {N} (y , a)}{N (a)} - \frac {\hat {N} (y , \bar {a})}{N (\bar {a})} \right| \leq \delta \tag {68}
$$
By observing that  $\hat{N} (\bar{y},a) / N(a) = 1 - \hat{N} (y,a) / N(a)$ , and  $\hat{N} (\bar{y},\bar{a}) / N(\bar{a}) = 1 - \hat{N} (y,\bar{a}) / N(\bar{a})$ , Eq. (68) can be rewritten to the two constraints shown in the main text:
$$
\frac {\hat {N} (y , a)}{N (a)} + \frac {\hat {N} (\bar {y} , \bar {a})}{N (\bar {a})} \leq 1 + \delta \quad \frac {\hat {N} (y , \bar {a})}{N (\bar {a})} + \frac {\hat {N} (\bar {y} , a)}{N (a)} \leq 1 + \delta \tag {69}
$$
Equality of opportunity Similarly, we can optimize for other group concepts of fairness, such as equality of opportunity, which is satisfied when:
$$
P (\hat {y} = 1 \mid y = 1, a = 1) = P (\hat {y} = 1 \mid y = 1, a = 0) \tag {70}
$$
This can be computed by two separable threshold constraints:
$$
g _ {a} (\mathcal {D}, \hat {k}) = \left\{ \begin{array}{l l} \frac {\left| \mathcal {D} _ {y , a} \right|}{N (y , a)} & \hat {k} = 1 \\ \frac {\left| \mathcal {D} _ {y , \bar {a}} \right|}{N (y , \bar {a})} & \hat {k} = 0 \end{array} \right. \quad g _ {\bar {a}} (\mathcal {D}, \hat {k}) = \left\{ \begin{array}{l l} \frac {\left| \mathcal {D} _ {y , a} \right|}{N (y , a)} & \hat {k} = 0 \\ \frac {\left| \mathcal {D} _ {y , \bar {a}} \right|}{N (y , \bar {a})} & \hat {k} = 1 \end{array} \right. \tag {71}
$$
In this equation  $\mathcal{D}_{y,a}$  and  $\mathcal{D}_{y,\bar{a}}$  are the number of instances with a positive outcome in the dataset for group  $a$  and  $\bar{a}$ , and  $N(y,a)$  and  $N(y,\bar{a})$  represent the number of group  $a$  and  $\bar{a}$  that have the positive outcome in the data.
Table 8: Runtime (s) results for optimizing with a demographic parity constraint. Timedouts (>600s) are marked with  ${}^{\prime } - {}^{\prime }$  .
Dataset	|D|	|F|	Aghaei [1]	Jo [44]	DPF [53]	STreeD
d=2	d=3	d=2	d=3	d=2	d=3	d=2	d=3
Adult	45222	17	-	-	-	-	< 1	< 1	< 1	2
Bank	45211	46	-	-	-	-	< 1	3	< 1	4
Com.&Crime	1994	97	-	-	-	-	< 1	5	< 1	29
COMPAS r.	6172	9	-	-	-	-	< 1	< 1	< 1	< 1
COMPAS v.r.	4020	9	-	-	-	-	< 1	< 1	< 1	< 1
Dutch	60420	58	-	-	-	-	< 1	6	< 1	8
German	1000	69	-	-	-	-	< 1	2	< 1	62
KDD	284556	117	-	-	-	-	1	39	3	26
OULAD	21562	45	-	-	-	-	< 1	3	< 1	19
Ricci	118	4	3	44	2	13	< 1	< 1	< 1	< 1
Stud. Math	395	55	-	-	473	-	< 1	< 1	< 1	2
Stud. Port.	649	55	-	-	-	-	< 1	< 1	< 1	3
Table 9: Runtime (s) results for optimizing with an equality of opportunity constraint. Timedouts (>600s) are marked with ' -'.
Dataset	|D|	|F|	STreeD
d=2	d=3
Adult	45222	17	<1	1
Bank	45211	46	<1	7
Com.&Crime	1994	97	<1	11
COMPAS r.	6172	9	<1	<1
COMPAS v.r.	4020	9	<1	<1
Dutch	60420	58	<1	7
German	1000	69	<1	50
KDD	284556	117	3	37
OULAD	21562	45	<1	19
Ricci	118	4	<1	<1
Stud. Math	395	55	<1	<1
Stud. Port.	649	55	<1	1
F.3 Experiments
In our experiments we compare STreeD with the two MIP methods Aghaei-Fair-MIP [1] and Jo-Fair-MIP [44]; and the DP method DPF [53]. The comparison focuses on scalability.
Setup We follow the setup from [53], by testing each method on 12 datasets, preprocessed and binarized as described in [50]. Categorical variables are binarized using one-hot encoding. Categorical variables with twenty or more categories are removed. For each of these datasets, the task is to find an optimal decision tree of depth 2-3 with at most  $1\%$  discrimination. A timeout of 600 seconds is used. Tests are repeated five times and we report averages. For all methods, we show results for enforcing a demographic parity constraint. For STreeD, we also show runtime results for optimizing with an equality-of-opportunity constraint.
References to the original datasets can be found here [6, 17, 24, 27, 49, 62, 77].
Results Table 8 shows the runtime results. Both MIP methods almost always hit the time limit, except for the smallest datasets. In contrast, both DP methods find optimal decision trees for depth two in less than one second except for the KDD-Census income dataset. For depth three DPF outperforms STreeD. This is partly because of how the fairness constraint is formulated for STreeD: as two threshold constraints. For better performance, an objective more like the one presented in DPF
Table 10: Runtime (s) for maximizing accuracy, with timeouts beyond 600s denoted with '-'. Best results are shown in bold. Average of five runs. Datasets for which all methods finish within one second are left out.
Dataset	d=4	DL8.5	MurTree	STreeD	d=5
DL8.5	MurTree	STreeD
Anneal	812	93	2	<1	<1	17	2	5
Audiology	216	148	4	<1	<1	<1	<1	<1
Australian-Credit	653	125	23	1	2	-	21	37
Breast-Wisconsin	683	120	4	<1	1	15	1	7
Diabetes	768	112	24	1	2	-	22	34
Fico	10459	17	<1	<1	<1	<1	1	2
German-Credit	1000	112	36	2	3	-	61	94
Heart-Cleveland	296	95	6	<1	<1	80	3	7
Hepatitis	137	68	<1	<1	<1	2	<1	<1
Hypothyroid	3247	88	5	1	<1	94	13	11
Ionosphere	351	445	-	69	152	-	146	112
Kr-vs-kp	3196	73	2	<1	<1	18	6	4
Letter	20000	224	547	200	145	-	-	-
Lymph	148	68	1	<1	<1	<1	<1	<1
Mushroom	8124	119	3	<1	<1	1	<1	<1
Pendigits	7494	216	160	60	74	-	240	109
Soybean	630	50	<1	<1	<1	6	<1	1
Splice-1	3190	287	-	148	156	-	-	-
Tic-Tac-Toe	958	27	<1	<1	<1	2	<1	<1
Vehicle	846	252	79	8	15	-	137	343
Vote	435	48	<1	<1	<1	5	<1	1
Yeast	1484	89	10	<1	1	280	13	18
Average rank	3.0	1.29	1.71	2.89	1.39	1.72
could be formulated for STreeD. However, formulating and implementing a fairness constraint in STreeD is much easier than developing a specialized method such as DPF. For equality of opportunity, STreeD shows similar results as when optimizing demographic parity.
In summary, STreeD shows performance not too far off from an existing specialized DP method; allows for easy formulation of new types of constraints, and outperforms both MIP methods by a large margin.
G Classification Accuracy
The most studied use case for optimal decision trees is maximizing accuracy. Since STreeD generalizes previous dynamic programming approaches for optimal decision trees, here we evaluate how STreeD performs compared to two state-of-the-art dynamic programming approaches to optimal decision trees: DL8.5 and MurTree.
DL8.5 [3, 4] is a dynamic programming approach that improves on DL8 [65, 66] by adding branch-and-bound and new caching techniques. We use their publicly available Python package. Note that we compared to their latest available version, which incorporates ideas from MurTree [20].
MurTree [20] is also a dynamic programming approach for optimal decision trees. Like DL8.5 it incorporates branch-and-bound and caching. Other improvements are a similarity lower bound and a special depth-two solver. We use the publicly available Python package. $^{8}$
<sup>7</sup>https://github.com/aia-uclovain/pyd18.5
<sup>8</sup>https://github.com/MurTree/pymurtree
Figure 6: Runtime (s) comparison. A timeout of 300 seconds is used. Note the logarithmic x-axis.
We test both methods and STreeD on a set of binarized datasets with binary class as described in Table 10. These are the same datasets as used in Appendix E. For each dataset, all three algorithms need to find a tree of depth four or five with minimum misclassification score. We repeat the tests five times and report the average runtime in Table 10. Runtime results are also shown in Fig. 6.
By using the geometric mean of the relative performance [31], we can see that MurTree on average is 1.25 times faster than STreeD and STreeD on average is 6.4 times faster than DL8.5. According to a Wilcoxon signed rank test with a significance level  $5\%$ , both results are significant.
Since STreeD outperforms DL8.5 and DL8.5 outperforms DL8, we can conclude that STreeD also outperforms DL8 in terms of scalability. Furthermore, Demirovic et al. [20] compared MurTree with GOSDT [52] and observed that GOSDT resulted in a timeout for  $65\%$  of the 68 datasets when computing optimal trees with maximum depth four, whereas MurTree did not result in a single timeout. Since STreeD's performance is only a factor of 1.25 slower than MurTree, we conclude that STreeD also outperforms GOSDT.
<sup>9</sup>https://bitbucket.org/EmirD/MurTree-bi-objective