SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. has a subtraction operator  $\ominus$ ;
2. does not have constraints;
3. is context independent; and
4. and the optimization task is per-instance (element-wise) additive.
Definition B.1 (Context independent). An optimization task is context independent if and only if the cost function does not depend on any of the parent nodes' branching decisions.
For example, cost-sensitive classification with discounts, as introduced in this paper, is not context independent.
Definition B.2 (Per-instance (element-wise) additive). An optimization task is per-instance (element-wise) additive, if the cost function for label assignments can be written as the sum of the costs from a per-instance cost function  $h$ :
$$
g \left(\left\langle \mathcal {D}, F \right\rangle , \hat {k}\right) = \sum_ {(x, k) \in \mathcal {D}} h (x, k, \hat {k}) \tag {48}
$$
If all these four conditions are met, the similarity bound can be used. Its definition is generalized from the definition given in [19]. Instead of subtracting the misclassification score per class from an existing solution, the worst-case value for instances of that class is subtracted by use of  $\Theta$ . Let  $w_{k}$  be the worst case contribution of a single instance with class  $k$  to the objective, and let  $n_k$  be the number of instances in a dataset  $\mathcal{D}$  that are not part of a dataset for a cached solution  $\mathcal{D}^c$ : i.e. if  $\mathcal{D}_k = \{(x,k') \in \mathcal{D} \mid k' = k\}$ , then  $n_k = |D_k \setminus D_k^c|$ . Now assume each of these  $n_k$  instances is misclassified and therefore  $w_{k}$  is added to the objective for each of these instances, then a lower bound can be computed from a cached solution  $\Theta^c$  as follows:
$$
\operatorname {l b} (\mathcal {D}) = \left\{v \ominus \sum_ {k \in \mathcal {K}} n _ {k} w _ {k} \mid v \in \Theta^ {c} \right\} \tag {49}
$$
In this equation, the sum operator uses  $\oplus$  to sum.
Once the lower bound for a search node is computed, it can be compared to the upper bound ub to check if any possible improvement exists, and if not, the node is pruned from the tree search. This happens whenever  $\mathrm{lb} > \mathrm{ub}$ , i.e., every element of the upper bound is reverse dominated by at least one element of the lower bound:
$$
\mathrm {l b} > \mathrm {u b} \Leftrightarrow \forall v \in \mathrm {u b}: \exists v ^ {\prime} \in \mathrm {l b}: v ^ {\prime} \prec v \tag {50}
$$
Computing these upper and lower bounds can be quite costly since it involves a product operation of two sets. To reduce the computation time for applying the subtraction or to combine two lower bounds for two subtrees, we apply a new procedure in which we first reduce the two sets to a representative subset using  $\triangle$  in such a way that the resulting bounds are still valid. These representative subsets are then combined to get a new valid upper or lower bound. This decreases the computation time of the bound but prunes slightly fewer nodes.
Sparse trees Optimal decision trees may overfit on the training data [22] and therefore typically a sparsity coefficient is introduced that adds a cost for every branching node, as for example done in [10, 36, 52]. In the work presented here, we do not consider such a sparsity coefficient, although it could trivially be added. Instead, STreeD optimizes directly for a given maximum number of nodes. This number of nodes can then be tuned (as explained below). In the DP formulation in the main text, this is left out for brevity. For completeness, the DP formulation should be changed by considering the following equation in a branching node  $(d > 0)$ , as is similarly proposed in [20]:
$$
\left. \operatorname {o p t} \left(\bigcup_ {f \in \mathcal {F}, i \in [ 0, n - 1 ]} \operatorname {m e r g e} \left(T (t (s, f), d - 1, i), T (t (s, \bar {f}), d - 1, n - i - 1), s, f\right)\right) \right. \tag {51}
$$
With this addition, STreeD can be tuned to optimize the number of nodes to prevent overfitting. This is done by using part of the training data as validation data and train STreeD on the rest of the training data for an increasing number of nodes. This is repeated five times and the number of nodes that yields on average the best result is used as the final maximum number of nodes.
Caching STreeD supports two forms of caching: branch caching and dataset caching. Dataset caching is typically slightly faster than branch caching [20]. However, dataset caching cannot be used for context-dependent optimization tasks, because dataset similarity is no longer sufficient to identify similar subproblems. Similarly, for some optimization tasks, branch caching should be replaced by equivalent state caching, which we leave as future work. In this paper, dataset caching is used, except for cost-sensitive classification and group fairness, which are context dependent and use branch caching.
Algorithm 1: Tree search of depth  $d$  for an optimization task  $\langle g, t, \succ, \oplus, c, s_0 \rangle$  for state  $s$ , for a feature set  $\mathcal{F}$  and a maximum number of nodes  $n$ .
$T(s,d,n,\mathrm{ub})$    
if  $d = 0\lor n = 0$  then return opt  $\left(\cup_{\hat{k}\in \mathcal{K}}g(s,\hat{k}),s,\mathrm{ub}\right)$    
if  $n > 2^d -1$  then return  $T(s,d,2^{d} - 1,\mathrm{ub})$    
if  $d > n$  then return  $T(s,d,d,\mathrm{ub})$ $\langle \Theta ,\mathrm{lb},\mathrm{stat}\rangle \gets \mathrm{cache}[s,d,n]$    
if  $\mathrm{lb} > \mathrm{ub}$  then return  $\emptyset$    
if stat  $=$  optimal then return opt  $(\Theta ,s,\mathrm{ub})$ $\Theta \gets \emptyset$    
for  $f\in \mathcal{F},n_L\in [0,n - 1]$  do  $n_R\gets n - n_L - 1$ $\mathrm{lb}_L\gets \mathrm{LB}(t(s,\bar{f}),d - 1,n_L)$ $\mathrm{lb}_R\gets \mathrm{LB}(t(s,f),d - 1,n_R)$ $\mathrm{lb}\gets \mathrm{merge}(\mathrm{lb}_L,\mathrm{lb}_R,s,f)$  if  $\mathrm{lb}>$  ub then continue  $\mathrm{ub}_L = \mathrm{ub}\ominus g(s,f)$ $\Theta_L\gets T(t(s,\bar{f}),d - 1,n_L,\mathrm{ub}_L)$  if  $\Theta_L = \emptyset$  then continue  $\mathrm{ub}_R = \mathrm{ub}\ominus \Theta_L\ominus g(s,f)$ $\Theta_R\gets T(t(s,f),d - 1,n_R,\mathrm{ub}_R)$  if  $\Theta_R = \emptyset$  then continue  $\Theta_{\mathrm{new}}\gets \mathrm{opt}(\mathrm{merge}(\Theta_L,\Theta_R,s,f),s)$ $\mathrm{ub}\gets \mathrm{opt}(\mathrm{ub}\cup \Theta_{\mathrm{new}},s)$ $\Theta \gets \mathrm{opt}(\Theta \cup \Theta_{\mathrm{new}},s,\mathrm{ub})$    
if  $\Theta = \emptyset$  then cache[s,d,n]  $\leftarrow \langle \emptyset ,\mathrm{ub},\right.$  lower bound return O   
cache[s,d,n]  $\leftarrow \langle \Theta ,\Theta ,$  optimal]   
return O
Pseudo-code Pseudo-code for STreeD is given in Algorithm 1. If a leaf node is reached, the cost function is used to compute optimal solutions. The next two checks in the pseudo-code check if the maximum number of nodes and the maximum depth are compatible. If they are, the cache is checked for optimal solutions or a lower bound. The search is stopped if the cached lower bound is dominated by the current upper bound. If the cached solution is optimal, it is returned.
Then, for every possible branching feature and every possible node division, the algorithm does the following. It computes lower bounds for the left and right subtree, merges these lower bounds, and skips this feature if the upper bound dominates the lower bound. Then a recursive call is performed to compute optimal solutions to the left subtree. The upper bound for the left tree is updated by subtracting the branching costs. Solutions from this recursive call are used to update the upper bound for the right subtree and the right subtree is also solved. Solutions from both subtrees are merged into a new set of solutions which are used to update the current set of solutions and the upper bound.
When all branching features have been checked and no feasible solution is found (better than the upper bound), the current upper bound is stored as a lower bound. If feasible solutions have been found, the current set of solutions is stored in the cache and the solutions are returned.
In case the comparison operator  $\succ$  is fully defined (a total ordering of all solutions exists), such as in the case with minimizing misclassification score, then the nondom operation always returns a set of precisely one solution (or zero if infeasible). In this case, nondom can be replaced by min (as defined by  $\succ$ ), simplifying the implementation and computation.
Depth-two solver Demirovic et al. [20] have shown that the runtime of optimal decision tree search can be improved considerably by use of a special solver for trees of maximum depth two. The core idea is to pre-compute class frequencies by looping over only the present features for every instance in a dataset. We generalize this concept in STreeD and explain it below.
STreeD's use of the depth-two solver is restricted under the following conditions:
1. the cost of a leaf node can be expressed as a function over the contributions of individual instances;
2. the cost of a leaf node is context independent (see Def. B.1, note that the branching costs may be context dependent); and
3. the branching costs are equal for datasets with the same size, i.e.,  $|\mathcal{D}_1| = |\mathcal{D}_2| \to g(\langle \mathcal{D}_1, F \rangle, f) = g(\langle \mathcal{D}_2, F \rangle, f)$  for all  $f \in \mathcal{F} \times \{0, 1\}$ .
For all optimization tasks considered in this paper, these conditions are met.
To use the depth-two solver, the following functions and values need to be defined:
1. a 'depth-two' solution space  $\mathcal{W}$
2. an 'empty' initial solution value  $w_0 \in \mathcal{W}$  (typically zero), the solution value of a leaf with zero instances;
3. a per-instance cost function  $j: \mathcal{X} \times \mathcal{K} \times \mathcal{K} \to \mathcal{W}$  such that  $j(x, k, \hat{k})$  returns the 'depth-two' costs of instance  $(x, k)$  when assigned label  $\hat{k}$ ;
4. a 'depth-two' combining operator  $\oplus^{\prime}:\mathcal{W}\times \mathcal{W}\to \mathcal{W}$  that combines two solution values into one;
5. a 'depth-two' subtract operator  $\Theta': \mathcal{W} \times \mathcal{W} \to \mathcal{W}$ , such that  $(w_1 \oplus' w_2) \ominus' w_2 = w_1$  for all  $w_1, w_2 \in \mathcal{W}$ ;
6. a transformation function  $q: \mathcal{W} \to \mathcal{V}$  that transforms a 'depth-two' solution value into a solution value in the original solution value space  $\mathcal{V}$ ;
7. a branching cost function  $r: \mathcal{S} \times \mathbb{N}^{+} \times \mathcal{F} \times \{0,1\} \to \mathcal{V}$ , such that  $r(s,n,f)$  returns the cost of branching on  $f$  in state  $s$  with  $n = |\mathcal{D}|$ .
In this paper, for all optimization tasks  $\mathcal{W} = \mathcal{V}$ ,  $\oplus' = \oplus$ , and  $j(x,k,\hat{k}) = g(\langle \{(x,k)\}, \emptyset \rangle, \hat{k})$ , but this is not necessarily true for future optimization tasks.
The function  $j$  provides a per-instance breakdown of the costs which are summed with the  $\oplus^{\prime}$  operator. These are used to precompute the costs of all possible leaf nodes. Let  $n(f_{i})$  and  $n(f_{i},f_{j})$  be the size of  $\mathcal{D}_{f_i}$  and  $\mathcal{D}_{f_i,f_j}$  respectively. Similarly, let  $w(\hat{k})$  be the cost of assigning all instances in  $\mathcal{D}$  label  $\hat{k}$ , and let  $w(\hat{k},f_i)$  and  $w(\hat{k},f_i,f_j)$  be the cost of assigning all instances in  $\mathcal{D}_{f_i}$  and  $\mathcal{D}_{f_i,f_j}$  respectively. By generalizing the result from Demirovic et al. [20], the following statements hold:
$$
n \left(\bar {f} _ {i}\right) = | \mathcal {D} | - n \left(f _ {i}\right) \tag {52}
$$
$$
n \left(f _ {i}, \bar {f} _ {j}\right) = n \left(f _ {i}\right) - n \left(f _ {i}, f _ {j}\right) \tag {53}
$$
$$
n \left(\bar {f} _ {i}, f _ {j}\right) = n \left(f _ {j}\right) - n \left(f _ {i}, f _ {j}\right) \tag {54}
$$
$$
n \left(\bar {f} _ {i}, \bar {f} _ {j}\right) = | \mathcal {D} | - n \left(f _ {i}\right) - n \left(f _ {j}\right) + n \left(f _ {i}, f _ {j}\right) \tag {55}
$$
$$
w (\hat {k}, \bar {f} _ {i}) = w (\hat {k}) \ominus^ {\prime} w (\hat {k}, f _ {i}) \tag {56}
$$
$$
w (\hat {k}, f _ {i}, \bar {f} _ {j}) = w (\hat {k}, f _ {i}) \ominus^ {\prime} w (\hat {k}, f _ {i}, f _ {j}) \tag {57}
$$
$$
w \left(\hat {k}, \bar {f} _ {i}, f _ {j}\right) = w \left(\hat {k}, f _ {j}\right) \ominus^ {\prime} w \left(\hat {k}, f _ {i}, f _ {j}\right) \tag {58}
$$
$$
w (\hat {k}, \bar {f} _ {i}, \bar {f} _ {j}) = w (\hat {k}) \ominus^ {\prime} w (\hat {k}, f _ {i}) \ominus^ {\prime} w (\hat {k}, f _ {j}) \oplus^ {\prime} w (\hat {k}, f _ {i}, f _ {j}) \tag {59}
$$
Furthermore, let  $f \in x$  denote that feature  $f$  is satisfied by  $x$ . The depth-two solver can now be generalized as shown in Algorithm 2. It first precomputes the dataset sizes and costs, then finds the best left and right solutions for each possible split, and finally returns the best solution.
C Cost-Sensitive Classification
For our cost-sensitive classification experiment, we consider a constant but asymmetric misclassification costs, test costs (feature costs) that are constant or may depend on previous selected tests. For
Algorithm 2: Tree search of depth two using a special depth-two solver.
$T_{d2}(s,\langle D,F\rangle)$    
// Pre-compute dataset sizes   
 $n(f_i)\gets 0\quad \forall f_i\in \mathcal{F}$ $n(f_i,f_j)\gets 0\quad \forall f_i,f_j\in \mathcal{F}$  s.t.  $i <   j$    
for  $(x,k)\in \mathcal{D}$  do for  $f_{i}\in x$  do  $n(f_i)\gets n(f_i) + 1$  for  $f_{j}\in x$  do  $n(f_i,f_j)\gets n(f_i,f_j) + 1$    
// Pre-compute costs   
 $w(\hat{k})\leftarrow w_0\quad \forall \hat{k}\in \mathcal{K}$ $w(\hat{k},f_i)\gets w_0\quad \forall \hat{k}\in \mathcal{K},f_i\in \mathcal{F}$ $w(\hat{k},f_i,f_j)\gets w_0\quad \forall \hat{k}\in \mathcal{K},f_i,f_j\in \mathcal{F}$  s.t.  $i <   j$    
for  $(x,k)\in \mathcal{D},\hat{k}\in \mathcal{K}$  do  $w(\hat{k})\leftarrow w(\hat{k})\oplus 'j(x,k,\hat{k})$    
for  $f_{i}\in x$  do  $w(\hat{k},f_i)\gets w(\hat{k},f_i)\oplus 'j(x,k,\hat{k})$    
for  $f_{j}\in x$  do  $w(\hat{k},f_i,f_j)\gets w(\hat{k},f_i,f_j)\oplus 'j(x,k,\hat{k})$    
// Find optimal subtrees   
for  $f_{i},f_{j}\in \mathcal{F}$  s.t.  $f_{i}\neq f_{j}$  do for  $\hat{k}_L,\hat{k}_R\in \mathcal{K}$  s.t.  $\hat{k}_L\neq \hat{k}_R$  do cost  $L\gets q(w(\hat{k}_L,\bar{f}_i,\bar{f}_j)\oplus q(w(\hat{k}_R,\bar{f}_i,f_j)$  cost  $L\gets \mathrm{cost}_{L}\oplus r(t(s,\bar{f}_{i}),n(\bar{f}_{i},\bar{f}_{j}),\bar{f}_{j})\oplus r(t(s,\bar{f}_{i}),n(\bar{f}_{i},f_{j}),f_{j})$  cost  $R\gets q(w(\hat{k}_L,f_i,\bar{f}_j)\oplus q(w(\hat{k}_R,f_i,f_j))$  cost  $R\gets \mathrm{cost}_{R}\oplus r(t(s,f_{i}),n(f_{i},\bar{f}_{j}),\bar{f}_{j})\oplus r(t(s,f_{i}),n(f_{i},f_{j}),f_{j})$  if cost  $L\succ$  best-cost  $L(f_{i})$  then best-cost  $L(f_{i})\gets \mathrm{cost}_{L}$  if cost  $R\succ$  best-cost  $R(f_{i})$  then best-cost  $R(f_{i})\gets \mathrm{cost}_{R}$    
return opt (UfeF merge(best-costL(f),best-costR(f),s,f))
costs, we follow the naming conventions of [83]. Misclassification costs or test costs that depend on individual cases or on feature values are easy extensions and also fit in the STreeD framework, but are not considered here.
The following sections present the related work, the experiment setup and results, and a discussion of the results. The objective formulation is given in Eq. (9) in the main text.
C.1 Related Work
Lomax and Vadera [55] provide an overview of methods for cost-sensitive classification with decision trees, most of which use top-down induction with varying cost functions for deciding on the best split [54, 68, 69, 80]. Each of these splitting criteria is mostly the same, except for a slight difference in the trade-off between accuracy and feature costs. This therefore is a good example of the drawback of top-down induction by splitting criteria: the objective is not directly translatable to a good splitting criterion. To address this issue Min and Zhu [61] generalize a number of those methods using a hyperparameter that decides on the trade-off between information gain and feature costs. ICET [82] does the same but uses a genetic algorithm to decide the hyperparameter. All of these methods except ICET, however, do not consider misclassification costs, and even ICET only does so indirectly.
Lomax and Vadera [56] co-optimize accuracy and costs by approaching the problem as a multi-armed bandit scenario. This means that in every branching node, they repeatedly try random splits (possibly
Table 1: Datasets used in cost-sensitive classification.  $|\mathcal{F}_{disc}|$  is the number of categorical features in the dataset as preprocessed by [57].  $|\mathcal{F}_{bin}|$  is the number of resulting binary features.
Dataset	|D|	|Fdisc|	|Fbin|	|K|
Annealing	898	24	82	5
Breast	277	9	38	2
Car	1728	6	21	4
Diabetes	768	6	11	2
Flare	323	10	25	3
Glass	214	7	17	6
Heart	297	11	20	2
Hepatitis	154	16	20	2
Iris	150	4	12	3
Krk	28056	6	40	18
Mushroom	8124	21	111	2
Nursery	8703	8	26	5
Soybean	562	35	80	15
Tictactoe	958	9	27	2
Wine	178	13	32	3
multiple consecutive random splits) and choose the split which on average has the best expected costs based on this random exploration.
Maliah and Shani [58] formulate the problem as a Partially Observable Markov Decision Process (POMDP). However, they observe that the action and state space of their model is too big, and therefore they present a smaller MDP that approximates the POMDP. For the MDP model, they generate trees for all possible subsets of features using a standard heuristic that ignores costs. The MDP's state space is the set of all leaf nodes of all those generated trees. The resulting MDP is acyclic, and therefore its value function can be computed in one bottom-up pass through the belief states.
As far as we know, Nijssen and Fromont [66] and Zubek and Dietterich [94] are the only ones to provide an optimal formulation. The latter do so by formulating it as an MDP. However, in their experiments, they decide to use a relaxed formulation instead because of memory limitations.
C.2 Experiment Setup
Table 1 lists the datasets used in this experiment [24, 41, 95]. These datasets were preprocessed and discretized according to the instructions in [57]. For methods that require binary features, the discretized features were binarized using one-hot encoding. Every dataset is split randomly 100 times in a train and test set, with  $80\%$  and  $20\%$  of the instances respectively.
We use the fixed feature costs as defined in [57]. The cost of misclassification strongly impacts the behavior of cost-sensitive algorithms. When misclassification costs are low, feature costs are comparatively high and branching is discouraged. Conversely, when misclassification costs are high, feature costs are comparatively low and therefore less important. In this case, overfitting also becomes more likely. Therefore, similar to the experiment setup in [57, 58] we vary the misclassification costs and generate misclassification matrices with low, middle, and high misclassification costs. These matrices are defined based on the feature costs and the class frequency as follows: First, calculate the maximum possible feature costs  $C$  by summing the costs for all features. The default cost  $C_{\text{def}}$  for low, middle, and high-cost experiments are defined as  $1/6C$ ,  $1/3C$ , and  $C$  respectively. By considering the relative frequency  $f_k$  of class  $k$ , the misclassification costs  $C_k$  for class  $k$  are defined as follows:
$$
C _ {k} = \frac {C _ {\text {d e f}}}{f _ {k} | \mathcal {K} |} \tag {60}
$$
For each run, a timeout of 300 seconds is used.
Normalized costs We report normalized costs, as defined in [82]. The normalized costs are calculated by dividing the obtained costs by the standard costs. As before, let  $C$  be the sum of all feature costs, let  $f_{k}$  be the relative frequency of class  $k$ , and let  $C_{k,\hat{k}}$  be the misclassification cost when an instance with true label  $k$  is assigned label  $\hat{k}$ . Then the standard costs is defined as follows:
$$
C + \min  _ {k} \left(1 - f _ {k}\right) \cdot \max  _ {k, \hat {k}} C _ {k, \hat {k}} \tag {61}
$$
These standard costs are lower than the maximum possible costs, but function as a good upper bound on the average costs [82].
In one aspect we deviate from the original definition. In the original definition,  $f_{k}$  was calculated based on the whole dataset, but here it is based on the dataset that is examined (the train or test set).
C.2.1 Methods
Apart from STreeD, the following methods are evaluated in our experiments:
C4.5 [73] is an extension of the ID3 algorithm [72] and uses top-down induction (TDI) based on an information gain measure. After construction, a pruning mechanism is used to remove branching nodes that are likely to result in overfitting. C4.5 is cost-insensitive and therefore functions as a baseline heuristic.
CSID3 [80] is a variant of ID3 with an updated splitting criterion. Instead of only information gain, it makes a trade-off between the information gain  $I(f)$  and the cost  $C(f)$  of feature  $f$  by the splitting criterion:
$$
\frac {I (f) ^ {2}}{C (f)}
$$
CSID3 does not consider varying misclassification costs.
EG2 [69] is also a variant of ID3 with an updated splitting criterion:
$$
\frac {2 ^ {I (f)} - 1}{(C (f) + 1) ^ {\omega}}
$$
The parameter  $\omega$ , with values  $0 \leq \omega \leq 1$ , determines the weight of the cost factor. Following Turney [82], we set  $\omega = 1$ . EG2 also does not consider varying misclassification costs.
IDX [68] is yet another variant of ID3 with the splitting criterion:
$$
\frac {I (f)}{C (f)}
$$
The look-ahead procedure of IDX is not implemented. IDX also does not consider varying misclassification costs.
MetaCost [23] can be used to turn any error-based classifier into a cost-sensitive classifier. It iteratively fits a model, computes class probabilities per sample, and updates the training input labels such that the expected costs are minimized based on the class probabilities. Note that MetaCost only considers misclassification costs and ignores feature costs. We run MetaCost with 10 iterations.
TMDP [58] formulates the problem as a Markov Decision Process (MDP). For every possible subset of features, it uses C4.5 to generate a normal cost-insensitive tree, and every leaf from the resulting tree is added as a state to the MDP. Each state is described by the path to the leaf. The state with the empty path is the starting state. In every state, the possible actions are to measure a feature or to classify a sample. If a feature is measured, the next state is the one described by the path that results from appending the measured feature to the current path. When classifying, the MDP moves to the terminal state. This MDP therefore is acyclic, and the values of the states can be computed in reverse topological order by using the training data. Based on these values, the tree can now be constructed by choosing the action with maximum value, beginning in the start node, and splitting into multiple nodes when selecting a feature to measure. This is repeated until all nodes are leaf nodes.
Because TMDP creates trees for every possible subset of the feature set, the total number of possible states explodes for an increasing number of features. Therefore, the authors
Table 2: Out-of-sample normalized costs (\%). Significantly  $(p < 5\%)$  best results are marked in bold. Timedouts are marked as ' -'.
Dataset	C4.5	CSID3	EG2	IDX	MetaCost	TMDP	STreeD
Annealing	5.6	5.6	2.2	2.2	5.7	1.6	1.2
Breast	43.9	43.4	41.0	40.7	54.8	21.0	20.4
Car	43.1	42.2	39.8	39.9	42.2	16.1	14.8
Diabetes	50.2	44.5	43.4	43.4	55.9	14.2	14.1
Flare	17.9	15.7	15.4	15.3	21.8	18.0	16.7
Glass	37.7	35.6	30.1	29.0	39.4	14.3	14.4
Heart	45.4	23.0	20.0	19.9	44.7	9.5	9.3
Hepatitis	32.2	24.3	20.0	19.7	32.2	16.2	15.6
Iris	38.8	34.0	29.5	29.7	40.9	12.8	12.8
Krk	6.5	6.2	6.3	6.3	6.5	-	1.7
Mushroom	16.4	8.3	5.6	4.5	16.4	-	1.9
Nursery	0.4	0.4	0.4	0.4	0.4	0.1	0.1
Soybean	12.8	16.3	19.8	21.1	12.6	15.4	11.4
Tictactoe	45.0	45.0	45.0	45.0	45.0	16.1	16.1
Wine	25.0	18.8	18.1	17.5	24.8	12.2	10.4
propose that for a larger number of features, TMDP only tests all possible subsets of the 10 most costly features together with all the cheap features considered by default. In the analysis of TMDP here, the same approach is used.