Title: SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Rashomon sets

decision trees

interpretable machine learning
Abstract: Sparse decision tree learning provides accurate and interpretable predictive models that are ideal for high-stakes applications by finding the single most accurate tree within a (soft) size limit. Rather than relying on a single “best” tree, Rashomon sets—trees with similar performance but varying structures—can be used to enhance variable importance analysis, enrich explanations, and enable users to choose simpler trees or those that satisfy stakeholder preferences (e.g., fairness) without hard-coding such criteria into the objective function. However, because finding the optimal tree is NP-hard, enumerating the Rashomon set is inherently challenging. Therefore, we introduce SORTD, a novel framework that improves scalability and enumerates trees in the Rashomon set in order of the objective value, thus offering anytime behavior. Our experiments show that SORTD reduces runtime by up to two orders of magnitude compared with the state of the art. Moreover, SORTD can compute Rashomon sets for any separable and totally ordered objective and supports post-evaluating the set using other separable (and partially ordered) objectives. Together, these advances make exploring Rashomon sets more practical in real-world applications.

SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration

Abstract

Sparse decision tree learning provides accurate and interpretable predictive models that are ideal for high-stakes applications by finding the single most accurate tree within a (soft) size limit. Rather than relying on a single 'best' tree, Rashomon sets-trees with similar performance but varying structures-can be used to enhance variable importance analysis, enrich explanations, and enable users to choose simpler trees or those that satisfy stakeholder preferences (e.g., fairness) without hard-coding such criteria into the objective function. However, because finding the optimal tree is NP-hard, enumerating the Rashomon set is inherently challenging. Therefore, we introduce SORTD, a novel framework that improves scalability and enumerates trees in the Rashomon set in order of the objective value, thus offering anytime behavior. Our experiments show that SORTD reduces runtime by up to two orders of magnitude compared with the state of the art. Moreover, SORTD can compute Rashomon sets for any separable and totally ordered objective and supports post-evaluating the set using other separable (and partially ordered) objectives. Together, these advances make exploring Rashomon sets more practical in real-world applications.

1 Introduction

Decision trees are widely regarded as one of the most interpretable models, as their decision paths are easy to follow and understand. While it is commonly believed that interpretability comes at the cost of accuracy, recent findings suggest that in some applications this trade-off may be negligible [1]. This makes decision trees even more appealing in practice, specifically for high-stakes domains.

Decision trees have been extensively studied, with early popular approaches such as CART [2] and C4.5 [3] focusing on greedy top-down induction. Because the size of a tree correlates with its comprehensibility [4], recent research has also examined optimal or sparse decision trees that maximise performance for a given size limit, thereby obtaining a better accuracy-interpretability trade-off [5, 6]. Consequently, the literature presents a variety of approaches to compute such optimal trees, including mixed-integer programming [7-9], Boolean satisfiability [10-12], constraint programming [13], branch-and-bound [14], and dynamic programming [15-18].

These methods share the property of returning a single decision tree that is constructed to perform well with respect to a loss function. However, there are typically many approximately equally good-but possibly very different-decision-tree models for a given learning problem. This phenomenon is known as the Rashomon effect [19], and the set of all models within a tolerance of the globally optimal loss value is called the Rashomon set .

Rashomon sets offer several advantages over relying on a single best model. In explainable AI, they enable the discovery of simpler models [20, 21], support diverse counterfactual explanations [22], address underspecification [23], also in explanations [24], and improve variable-importance analysis [25]. In addition, they allow evaluation of criteria that are difficult to encode directly into the objective function-such as fairness or robustness-by analysing properties across the set. In recidivism prediction, Fisher et al. [26] examined the influence of bias-associated variables, and Marx et al. [27] explored how similar models might yield conflicting predictions. Rashomon sets also provide a more flexible interface for stakeholders, enabling them to select models that best align with their domain-specific constraints. In healthcare, a diverse set of models obtained from the Rashomon set was used to support informed decision-making [28, 29].

Benefiting from the Rashomon set requires efficient evaluation, as the set size can be prohibitively large (with only ten features and a depth budget of four, it may exceed 10 12 trees). Practical use cases often require finding the most accurate trees, while also satisfying structural constraints (e.g., limits on tree size or mandatory/forbidden features) or attaining favourable scores on complementary objectives (e.g., fairness). This search task would be greatly simplified if candidate trees were generated in non-decreasing order of their objective value, preventing users from spending substantial time to examine the entire set to locate the desired models.

The current state-of-the-art for obtaining the Rashomon set of sparse decision trees [30] returns every tree whose performance falls within a user-chosen tolerance of the best model (e.g., 5% above the optimum) but the output is not ordered by the objective. If the search is terminated early, e.g., because of a time-out, there is no guarantee that the returned set contains the actually best models according to the objective. Furthermore, when the appropriate margin is uncertain, it is often more practical to select a larger value, which can result in the construction of an extremely large set; conversely, if a fixed number of top-ranked models is of interest, there is no systematic mechanism to terminate enumeration once that quantity has been reached.

To address these concerns, we propose a novel framework, SORTD ( So rted R ashomon Sets of T rees using D ynamic Programming). In doing so, we achieve three contributions. First, we compute the Rashomon set in order : trees with the best objective values are generated first. Ordered generation enables early termination when a specified number of high-quality models has been obtained, hence providing an anytime Rashomon set property. Our experiments show that having access to the best models early can speed up downstream evaluation tasks such as variable-importance analysis.

Our second contribution is improved scalability in the Rashomon set calculation . To reduce runtime, SORTD incorporates a specialised algorithm for trees of depth two or less. This design allows SORTD to scale well with both the number of features and the depth budget. In our evaluation on a variety of benchmark classification datasets, we show that SORTD significantly outperforms the state-of-the-art, achieving speed-ups of up to two orders of magnitude .

Finally, our third contribution is providing a general framework that computes Rashomon sets for separable and totally ordered objectives and evaluates them with any separable and partially ordered one . We demonstrate this by enumerating the Rashomon set also for regression trees in addition to classification, and evaluating the Rashomon set of decision trees using an additional fairness objective. Together, these contributions make SORTD a practical and scalable tool for decision tree training, evaluation, and selection.

The rest of the paper is organized as follows: Sec. 2 reviews related work; Sec. 3 covers preliminaries; Sec. 4 details the Rashomon set calculation; and Sec. 5 presents the experiments.

2 Related work

Methods Rashomon set computation has recently been explored for risk score models [29, 31], additive models [20, 24, 32], rule sets [33, 34], random forest [24], and kernel ridge regression [24]. For sparse decision trees-the focus of this work-the only dedicated approach is TreeFARMS [30],

which enumerates every tree within a given user-chosen tolerance (the 'Rashomon multiplier') but does not preserve a global ordering of trees with respect to the objective. However, this user tolerance is rarely known a priori: an overly large value may result in generating billions of trees, which is memory and time intensive; while a small value may return too few trees. Additionally, TreeFARMS is tailored to classification, and extension to other optimization tasks is non-trivial.

On the contrary, our method produces solutions iteratively in non-decreasing order, allowing the algorithm to stop as soon as a target number of high-quality trees is reached without relying on an accurately tuned tolerance. This gives SORTD an anytime behavior: stopping the search at any time yields a Rashomon set. Furthermore, a specialised depth-two solver reduces runtime and improves scalability with both the number of features and the depth budget. Finally, SORTD handles any separable and totally ordered loss function and supports post-hoc evaluation of separable and partially ordered objectives (e.g., multi-objective optimization), hence increasing flexibility in learning and evaluation.

Decision trees Early decision tree induction methods, such as AID [35] for recursive regression analysis, and CHAID [36] for classification, use top-down induction to infer the next best split. The two most popular approaches, CART [2] and C4.5 [3], share this paradigm. While these methods typically yield good results, their greedy nature may yield models that are arbitrarily larger than optimal [37]. Indeed, provably optimal trees that are obtained through exhaustive search on average obtain a better size-accuracy, and hence interpretability-accuracy, trade-off than greedy approaches [6, 7]. Although finding such optimal trees is NP-hard [38], the problem remains tractable for a limited number of features and small tree-size limits [39], and recent dynamic programming (DP) approaches can typically find optimal trees of limited size for real-world datasets in seconds [e.g., 16, 40].

Unlike these approaches, our work aims not to find a single best tree, but the set of all good trees. A key advantage is that this set can be explored to find optimal solutions for other objectives or constraints that are harder to optimize directly. For example, while Demirovi´ c et al. [41] develop a specialized DP algorithm for non-linear metrics such as F1-score, Xin et al. [30] obtain the optimal F1-score tree from the Rashomon set based on optimizing accuracy. Similarly, rather than building a custom method for each objective or constraint, such as for example, a demographic parity fairness constraint [42], we explore the Rashomon set instead to find trees that are both accurate and fair.

3 Preliminaries

Notation Given a set of binary features F and a set of labels K , a sample ( x i , k i ) is a pair of feature vector x i ∈ { 0 , 1 } | F | and label k i ∈ K . A dataset D = { ( x i , k i ) } i is the set of samples that can be used to train a prediction model. Since we assume that the features are binary, we use D ( f ) to indicate the set of samples where f is satisfied and D ( ¯ f ) is the set of samples where f is not satisfied.

Sparse tree objective Abinary tree is a function T : { 0 , 1 } | F | → K that recursively maps a feature vector x to a predicted label ˆ k . Starting with the root node, each internal node in T sends an instance left or right when its specified binary feature test is satisfied in x or not. The final leaf node then provides its assigned label as the return value. The optimization task in this work considers finding trees that optimize a given objective function. In Appendix A.9, we consider arbitrary separable and totally ordered objectives [40], but in the main text, we limit our discussion for brevity to finding optimal sparse classification trees, for which we need to find the tree that minimizes:

̸

This equation penalizes each misclassification and additionally, the number of leaf nodes N ( T ) by a complexity cost λ [43]. Given Eq. (1), T ∗ = argmin T ∈T ( d ) C ( T, D ) finds the optimal tree from the set of all trees T ( d ) of maximum depth d .

Rashomon set Given a Rashomon multiplier ε , the Rashomon set is the set of trees with an objective value within the Rashomon bound θ ( T ∗ , D, ε ) = (1 + ε ) C ( T ∗ , D ) . We obtain the Rashomon set:

Depth-two subroutine Aspecialized subroutine for finding optimal trees of depth two was proposed by Demirovi´ c et al. [16]. It has a significant computation advantage by exploiting precomputed frequency counts instead of repeatedly recursively splitting the dataset. Taking binary classification as example, with D + the set of positive samples, then Q + ( f i ) = |{ ( x, k ) : ( x, k ) ∈ D + ( f i ) }| and Q + ( f i , f j ) = |{ ( x, k ) : ( x, k ) ∈ D + ( f i ) ∩ D + ( f j ) }| are the number of occurrences of feature f i , and of f i and f j combined respectively in the positive samples, which can be counted efficiently by looping over sparse representations of the feature vectors. The frequency counts for other possible feature inclusions or exclusions of features f i and f j in a depth-two tree are calculated implicitly using only these two definitions. E.g., Q + ( f i , ¯ f j ) = Q + ( f i ) -Q + ( f i , f j ) represents the number of instances that satisfy feature f i , but not f j . The frequency counts Q -for negative labels can be calculated analogously. Combining both positive and negative frequency counts allows us to directly compute misclassification scores for all possible depth-two trees.

4 In-order Rashomon set calculation

4.1 High-level idea

Given a depth budget, we construct the Rashomon set by exploring decision trees in ascending (i.e., best-first) order of their objective values. We first identify the optimal tree and set the Rashomon bound (tolerance of the best model). We then iteratively build the Rashomon set by maintaining sorted lists of solutions for each node in the search tree. Note that our contribution lies in this second phase: efficiently identifying and ordering these additional solutions.

To compute the Rashomon set, we use a search tree. Each search node maintains a sorted solution list, starting with the optimal one(s), followed by the suboptimal solutions in order of their objective value. A search node contains a helper node for each possible split on all features f ∈ F , and one for creating a leaf node. Each helper node -branching or leaf- of the search node also maintains its own sorted solution list and a pointer to its next unprocessed solution with minimum objective value. Leaf nodes contribute a single solution while branching nodes generate multiple.

In essence, during Rashomon set computation, a search node either repeatedly returns its best next solution from its sorted list or explores new candidates through its helper nodes when no further solution is available. In this exploration phase, the node selects the best next solution among its helpers, and the chosen helper prepares its next candidate if one exists. The search node then returns this selected solution. This process continues until the Rashomon bound or the Rashomon set size is reached. This lazy enumeration strategy computes new solutions only when needed, thereby reducing computation time.

0

1

2

3

4

0

0

1

0

Fig. 2 shows how a branching node computes its next solution by combining the best solutions from its left and right child nodes. It computes the Cartesian sum { a + b | a ∈ A,b ∈ B } (see, e.g., [44,

45]), but iteratively and in sorted order, while the sets A and B (the solution lists of the child nodes) at the same time are also iteratively generated (and also in-order). The best solution of this branch node is formed by pairing the child search nodes' best solutions (their solutions with index 0). To find the next-best solution, we increment either the left or right index. The lower-valued combination is selected as the next solution, and the other is stored in the candidates queue. In each step, the next solution is either the minimum valued candidate or a new combination obtained by incrementing one of the indices of the current next solution.

0

0

1

0

1

0

Example 2. In Fig. 2, after adding the best solution of value 0 . 10 + 0 . 10 = 0 . 20 ( lindex = 0 , rindex = 0) , two new options with values 0 . 10 + 0 . 32 = 0 . 42 ( lindex = 0 , rindex = 1) and 0 . 40 + 0 . 10 = 0 . 50 ( lindex = 1 , rindex = 0) are considered. The former becomes the next solution, and the latter is stored in the candidates queue.

4.2 Algorithm

Phase 1 of our algorithm calculates the optimal tree given a dataset, and a depth budget and meanwhile also caches optimal solutions to subproblems. In Phase 2-the focus of our contribution-we create a search tree over subproblems, with each search node holding a sorted list of computed solutions (so far), and the value of the next best solution (initially set to the cached optimal partial solutions). Phase 2 then iteratively requests the next best solution from the root search node, which recursively constructs it by requesting the next best solutions from its child search nodes (see Appendix A.1).

Alg. 1 shows how a search node computes its next best solution. First, the helper node with the smallest next solution value node . next is found (Line 1). A solution group sol is then created to collect all solutions with the same objective value (Line 2). If the helper node is a branching node, additional solutions with the same value are retrieved from its candidates queue CQ (Lines 4-5). This queue is a heap of tuples ( sol . value , lindex , rindex ) , sorted by sol . value , while lindex and rindex are solution indices in the left and right child search nodes' solution lists. Using these indices, ExploreCandidates recursively explores each retrieved solution to determine whether incrementing the left or right index produces new solutions with the same objective value (Line 6). Solutions matching the solution group's value are added to the solution group (see Appendix A.2), while the others are added to the candidates queue (Lines 7-8). Then, the next solution value of the branching node becomes the minimum solution value in the candidates queue (Line 9).

If the helper node is a leaf node, its next solution and candidates queue become empty (Line 11). The helper node is removed from the list of helper nodes if its candidates queue is empty or if the value of its next solution is higher than the search node's upper bound (Lines 12-13). This is because it can no longer contribute new valid solutions in future iterations. Finally, the solution group is added to the helper node's sorted solution list SSL (Line 14), and it is returned as the next solution (Line 15).

Alg. 2 details the ExploreCandidates procedure. The list of solutions to explore is initialized with the same-valued solutions list (Line 2). For each solution in this list, new index pairs are generated by incrementing either the left or right index by one (Lines 6-7). If the resulting index pair has not been evaluated previously (Line 8), the left (right) child search node is queried to return its solution with the left (right) index (Lines 9-10). If the child search node does not have that solution yet, it is obtained by calling GetNextSolution (Alg. 1). These left and right solutions are combined into a new solution (Line 11, see Appendix A.3 for the details of the solution combination). If the combined solution has the same value as the branching node's current next solution, it is added to the same-valued solution list and is explored recursively in later iterations (Lines 13-15). Otherwise, it is added to the distinct-valued solutions list (Lines 16-18).

̸

Depth-two subroutine To improve scalability, Alg. 3 adapts the depth-two subroutine proposed by Demirovi´ c et al. [16] (see Sec. 3) to efficiently calculate all depth-two solutions of three branching nodes. It iterates over all pairs of features f i , f j (Lines 1-2), with f i the branching feature in the root, and f j in either the left or right branching nodes. It then computes the optimal solutions sol L and sol R for the left and right subtree (Line 4-5). While doing so, only the solutions with values within the upper bound are kept (Lines 6-9). For each left solution, the combinations obtained with the right solutions are iteratively evaluated and inserted into the node's sorted solution list until the combination exceeds the upper bound (Lines 11-14). See Appendix A.6 for computing trees with one or two branching nodes.

If a limit is set on the Rashomon set size, computing all depth-two trees may be unnecessary. We address this by rerunning the depth-two subroutine with an incrementally increased upper bound until the actual upper bound is reached. See Appendix A.7 for details.

4.3 Comparison to the state-of-the-art

TreeFARMS [30] calculates the Rashomon set of trees using a depth-first search and supports only classification tasks. It requires a predefined Rashomon multiplier. Its solutions can be sorted efficiently but only after full enumeration, making correct estimation of the multiplier necessary.

̸

In contrast, SORTD computes the Rashomon set in order of objective value using best-first search, enabling anytime behavior: it produces a valid Rashomon set at any point. This is particularly beneficial for high-dimensional datasets, ensuring search termination and stable performance. It additionally eliminates the need to guess the Rashomon multiplier, as the search can also stop based on a specified set size. When SORTD is run in the same way as TreeFARMS (i.e., with a fixed Rashomon bound), SORTD returns the whole Rashomon set up to two orders of magnitude faster than TreeFARMS, while using up to one order of magnitude less memory, as we will show in the next section. Furthermore, SORTD is not limited to classification tasks, supporting any separable and totally ordered objective for computation and any separable and partially ordered objective for post-evaluation.

Limitations Our approach is limited to binary features, which is a common limitation [e.g., 30]. And since finding even a single optimal tree is NP-hard, enumerating the Rashomon set for larger depth budgets, dataset sizes, or target set sizes becomes intractable. However, our scalability improvements extend the range of problem instances that can practically be solved.

5 Experimental evaluation

We conduct a series of experiments with the following aims: (1) to assess SORTD's runtime efficiency in computing Rashomon sets; (2) to showcase that a small number of high-quality trees-easily found by SORTD-may be informative for model evaluation via variable importance analysis; and (3) to demonstrate SORTD's flexibility in enumerating and analysing Rashomon sets under varying objective functions.

Experiment set-up For aims (1) and (2), we use the 30 benchmark binary classification datasets previously used to assess state-of-the-art methods [10, 15, 16, 30, 46]. For aim (3) we adopt common regression [47] and fairness benchmark datasets [48]. We implemented SORTD in C++ and provide it as a python package. 1 We use STreeD [40] to compute optimal trees in SORTD's first phase. All experiments are run single-threaded on an Intel Xeon E5-6448Y @ 2.1 GHz with 100 GB RAM [49], with a 300 seconds time limit. Further details are provided in Appendix B.

5.1 Runtime performance

We evaluated SORTD's runtime performance against the state-of-the-art method TreeFARMS [30] across varying Rashomon set sizes n T . Since TreeFARMS requires the Rashomon multiplier (or bound) to be specified in advance, we ensured a fair comparison by precomputing the Rashomon multipliers (see Appendix B.2) using the following procedure. We varied the depth budget d ∈ { 3 , 4 , 5 } and the complexity cost λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } . Using each (dataset, d , λ ) combination, we

ran SORTD to find the smallest multiplier ε that yields at least n T = 10 n trees for n ∈ { 1 , . . . , 6 } . We limited n to six, as we consider sets larger than 10 6 trees to be impractical for real-world analysis. Both methods then used these multipliers for Rashomon set enumeration, and runtime was measured to obtain the whole Rashomon set up to the specified bound.

Fig. 3 plots the empirical cumulative distribution of runtimes for finding the whole Rashomon set of the specified size, aggregated over all datasets and λ values (see Appendix B.2 for detailed results). Across every depth budget and Rashomon set size, SORTD consistently outperforms TreeFARMS, reaching speed-ups of up to two orders of magnitude. As the depth budget grows, SORTD's runtime increases only slightly-most sets are produced under 10 seconds even at the largest depth-whereas TreeFARMS slows by more than an order of magnitude and hits the time limit after depth budget three. Moreover, at higher depth limits, SORTD scales better for increasingly large Rashomon sets.

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

0

25

50

75

100

Total computed (%)

d = 3

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

d = 4

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

d = 5

TreeFARMS

n

T

= 10

2

n

T

= 10

4

n

T

= 10

6

We additionally investigated how feature dimensionality affects runtime. Fig. 4 shows the empirical cumulative distribution of runtimes at depth budget four and n T = 10 6 while varying the feature dimension of the input dataset. Runtime increases for both methods as dimensionality grows, but SORTD is impacted less. Once the feature count exceeds 30, TreeFARMS fails to finish computing some Rashomon sets within the time limit. In contrast, SORTD remains fast: even with 30 features, it completes all runs well below the time limit.

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

0

25

50

75

100

Total computed (%)

|

F

15

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

16

<

|

F

30

10

-

2

10

-

1

10

0

10

1

10

2

Runtime (s)

|

F

|

>

30

TreeFARMS

Furthermore, we evaluated the memory usage of SORTD. Fig. 5 shows the empirical cumulative distribution of memory usage in gigabytes. For most of the instances, SORTD's memory usage remains below 1 GB, leading to on order of magnitude less memory usage. In contrast, TreeFARMS shows substantially higher memory requirements, particularly at greater depths.

5.2 Variable importance analysis

To test whether SORTD's in-order solution generation facilitates downstream evaluation, we measured variable importance with the Leave-One-Feature-Out (LOFO) score [50] by computing the increase

10

-

1

10

0

10

1

10

2

Memory usage (GB)

0

25

50

75

100

Total computed (%)

d = 3

10

-

1

10

0

10

1

10

2

Memory usage (GB)

d = 4

10

-

1

10

0

10

1

10

2

Memory usage (GB)

d = 5

TreeFARMS

n

T

= 10

2

n

T

= 10

4

n

T

= 10

6

in the area under the Rashomon objective curve when that feature is omitted. Fig. 6 illustrates LOFO on the compas [51] and fico [52] datasets using the top-100 trees. As reported by Fisher et al. [26], 'priors > 3' is an important variable in compas . In our analysis, omitting this feature noticeably shifts the loss curve, and a similar effect is observed when excluding external risk estimate features in fico , highlighting their strong predictive influence.

We now start our test whether variable importance obtained through different Rashomon set sizes provides similar insights. We treated variable importance values derived from the top-10,000 trees as a reference, and compared them with the ones obtained from the top-1 and top-100 trees. Since importance estimates within Rashomon sets can vary under resampling [25], each dataset was bootstrapped 20 times. For each resample, we computed the Rashomon multiplier required to obtain at least 10,000 trees and constructed the corresponding set using the multipliers. We then evaluated the increase in area under the Rashomon objective curve using λ = 0 . 01 and a depth budget of four. Due to its high runtime requirement, the biodeg dataset was not used in this experiment.

We evaluated top-5 feature stability using the Jaccard index ( | A ∩ B | / | A ∪ B | ) and full ranking stability using Kendall's τ [53]. Comparing top-1 to top-10,000 trees, 19 of 29 datasets had a Jaccard index ≥ 0 . 8 ; this increased to 26 datasets for top-100. Similarly, τ ≥ 0 . 7 held for 5 datasets using top-1 trees, and for 17 using top-100 trees (see Appendix B.7). These results suggest that variable importance is relatively stable when using 100 trees, highlighting the potential of estimating the importance values from smaller Rashomon sets. This is particularly useful for large datasets, where computing large Rashomon sets is impractical. Developing theoretical guidelines for the minimum number of trees required to obtain stable importance estimates could further facilitate this analysis.

5.3 Evaluating other objectives

SORTD also supports objectives beyond accuracy (see Appendix A.9), optimizing separable totally ordered objectives directly and evaluating separable and partially ordered ones indirectly.

Regression For example, SORTD can directly find the Rashomon set of sparse regression trees with the totally ordered objective of minimal mean-squared error. We demonstrate this capability by comparing SORTD with the heuristic CART [2] and the optimal method STreeD [40] since no other method is known for generating this set. Both methods repeatedly compute trees for random samples of the data until a given time limit. Fig. 7 shows that SORTD finds trees in order until the 10% relative bound is reached, whereas, as observed in [30] for classification tasks, CART and STreeD find orders of magnitude fewer trees in the Rashomon set. See Appendix B.8 for further details.

10

0

10

2

10

4

10

6

10

8

Solution index

0

10

20

Relative loss (%)

Synchronous Machine

10

0

10

2

10

4

10

6

10

8

Solution index

10

0

10

2

10

4

10

6

10

8

Solution index

Seoul Bike Sharing Demand

CART

STreeD

Equality of opportunity SORTD can also evaluate partially ordered objectives such as multiobjective criteria indirectly. E.g., to find fair accurate trees, SORTD can first obtain the Rashomon set based on accuracy, and then evaluate it using another objective such as equality-of-opportunity , i.e., the difference in the true positive rate of two discrimination-sensitive groups. For example, Fig. 8 shows the top 10 7 trees in the Rashomon set for accuracy evaluated with the equality-of-opportunity metric. In Appendix B.9 we provide further details and a runtime comparison with STreeD.

83

.

50

83

.

75

84

.

00

Accuracy (%)

-

8

-

6

-

4

-

2

0

Discrimination (%)

Adult

89

.

25

89

.

50

89

.

75

90

.

00

Accuracy (%)

-

4

-

2

0

2

Discrimination (%)

Bank Marketing

66

.

25

66

.

50

66

.

75

67

.

00

Accuracy (%)

-

7

.

5

-

5

.

0

-

2

.

5

0

.

0

Discrimination (%)

Compas

2

3

4

5

6

7

8

9

10

11

12

13

Tree size

6 Conclusion

The Rashomon set of sparse decision trees offers many advantages over relying on a single model, provided it can be explored efficiently. SORTD makes such exploration practical: it enumerates trees in ascending objective order, allowing the highest-quality candidates to be retrieved and evaluated quickly. At the same time, its algorithmic design scales significantly better than the state-of-the-art and uses less memory as either the feature dimensionality or the depth budget grows. Finally, it supports separable totally ordered objectives for Rashomon set computation, as well as separable and partially ordered objectives for fast post-hoc evaluation. Together, these advances turn large-scale Rashomon-set analysis and model selection into a viable option for real-world applications.

7 Acknowledgments

This research was supported by the TU Delft AI Labs program.

References

Cynthia Rudin. 'Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead'. In: Nature machine intelligence 1.5 (2019), pp. 206-215.

Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. Classification and Regression Trees . Monterey, CA: Wadsworth and Brooks, 1984.

J. Ross Quinlan. C4.5: Programs for Machine Learning . San Francisco, CA: Morgan Kaufmann Publishers Inc, 1993.

Rok Piltaver, Mitja Luštrek, Matjaž Gams, and Sandra Martinˇ ci´ c-Ipši´ c. 'What makes classification trees comprehensible?' In: Expert Systems with Applications 62 (2016), pp. 333346.

Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, and Margo Seltzer. 'Generalized and Scalable Optimal Sparse Decision Trees'. In: Proceedings of ICML-20 . 2020, pp. 6150-6160.

Jacobus G. M. van der Linden, Daniël Vos, Mathijs M. de Weerdt, Sicco Verwer, and Emir Demirovi´ c. 'Optimal or Greedy Decision Trees? Revisiting their Objectives, Tuning, and Performance'. In: arXiv preprint arXiv:2409.12788 (2024).

Dimitris Bertsimas and Jack Dunn. 'Optimal classification trees'. In: Machine Learning 106.7 (2017), pp. 1039-1082.

Sicco Verwer and Yingqian Zhang. 'Learning decision trees with flexible constraints and objectives using integer optimization'. In: Proceedings of CPAIOR-17 . 2017, pp. 94-103.

Sina Aghaei, Andrés Gómez, and Phebe Vayanos. 'Strong Optimal Classification Trees'. In: Operations Research (2024).

Nina Narodytska, Alexey Ignatiev, Filipe Pereira, and Joao Marques-Silva. 'Learning Optimal Decision Trees with SAT'. In: Proceedings of IJCAI-18 . 2018, pp. 1362-1368.

Hao Hu, Mohamed Siala, Emmanuel Hebrard, and Marie-José Huguet. 'Learning Optimal Decision Trees with MaxSAT and its Integration in AdaBoost'. In: IJCAI-PRICAI 2020 . 2020, pp. 1170-1176.

Pouya Shati, Eldan Cohen, and Sheila A. McIlraith. 'SAT-based optimal classification trees for non-binary data'. In: Constraints 28.2 (2023), pp. 166-202.

Hélene Verhaeghe, Siegfried Nijssen, Gilles Pesant, Claude-Guy Quimper, and Pierre Schaus. 'Learning Optimal Decision Trees using Constraint Programming'. In: Constraints 25.3 (2020), pp. 226-250.

Rahul Mazumder, Xiang Meng, and Haoyue Wang. 'Quant-BnB: A Scalable Branch-andBound Method for Optimal Decision Trees with Continuous Features'. In: Proceedings of ICML-22 . 2022, pp. 15255-15277.

Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. 'Learning Optimal Decision Trees Using Caching Branch-and-Bound Search'. In: Proceedings of AAAI-20 . 2020, pp. 3146-3153.

Emir Demirovi´ c, Anna Lukina, Emmanuel Hebrard, Jeffrey Chan, James Bailey, Christopher Leckie, Kotagiri Ramamohanarao, and Peter J. Stuckey. 'Murtree: Optimal Decision Trees via Dynamic Programming and Search'. In: Journal of Machine Learning Research 23.26 (2022), pp. 1-47.

Hayden McTavish, Chudi Zhong, Reto Achermann, Ilias Karimalis, Jacques Chen, Cynthia Rudin, and Margo Seltzer. 'Fast Sparse Decision Tree Optimization via Reference Ensembles'. In: Proceedings of AAAI-22 . 2022, pp. 9604-9613.

C˘ at˘ alin E. Brit , a, Jacobus G. M. van der Linden, and Emir Demirovi´ c. 'Optimal Classification Trees for Continuous Feature Data Using Dynamic Programming with Branch-and-Bound'. In: Proceedings of AAAI-25 . 2025, pp. 11131-11139.

Leo Breiman. 'Statistical modeling: The two cultures (with comments and a rejoinder by the author)'. In: Statistical science 16.3 (2001), pp. 199-231.

Lesia Semenova, Cynthia Rudin, and Ronald Parr. 'On the Existence of Simpler Machine Learning Models'. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 2022, pp. 1827-1858.

Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. 'Position: Amazing Things Come From Having Many Good Models'. In: Proceedings of ICML-24 . 2024, pp. 42783-42795.

Hayden Andersen, Andrew Lensen, Will Browne, and Yi Mei. 'Producing Diverse Rashomon Sets of Counterfactual Explanations with Niching Particle Swarm Optimization Algorithms'. In: Proceedings of the Genetic and Evolutionary Computation Conference . 2023, pp. 393-401.

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 'Underspecification Presents Challenges for Credibility in Modern Machine Learning'. In: Journal of Machine Learning Research 23.226 (2022), pp. 1-61.

Gabriel Laberge, Yann Pequignot, Alexandre Mathieu, Foutse Khomh, and Mario Marchand. 'Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set'. In: Journal of Machine Learning Research 24.364 (2023), pp. 1-50.

Jon Donnelly, Srikar Katta, Cynthia Rudin, and Edward Browne. 'The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-Based Variable Importance'. In: Advances in NeurIPS-23 . 2023, pp. 6267-6279.

Aaron Fisher, Cynthia Rudin, and Francesca Dominici. 'All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously'. In: Journal of Machine Learning Research 20.177 (2019), pp. 1-81.

Charles T. Marx, Flavio du Pin Calmon, and Berk Ustun. 'Predictive Multiplicity in Classification'. In: Proceedings of ICML-20 . 2020, pp. 6765-6774.

Katarzyna Kobyli´ nska, Mateusz Krzyzi´ nski, Rafał Machowicz, Mariusz Adamek, and Przemysław Biecek. 'Exploration of the Rashomon Set Assists Trustworthy Explanations for Medical Data'. In: IEEE Journal of Biomedical and Health Informatics 28.11 (2024), pp. 64546465.

Chloe Qinyu Zhu, Muhang Tian, Lesia Semenova, Jiachang Liu, Jack Xu, Joseph Scarpa, and Cynthia Rudin. 'Fast and Interpretable Mortality Risk Scores for Critical Care Patients'. In: Journal of the American Medical Informatics Association 32.4 (2025), pp. 736-747.

Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. 'Exploring the Whole Rashomon Set of Sparse Decision Trees'. In: Advances in NeurIPS-22 . 2022, pp. 14071-14084.

Jiachang Liu, Chudi Zhong, Boxuan Li, Margo Seltzer, and Cynthia Rudin. 'FasterRisk: Fast and Accurate Interpretable Risk Scores'. In: Advances in NeurIPS-22 . 2022, pp. 17760-17773.

Chudi Zhong, Zhi Chen, Jiachang Liu, Margo Seltzer, and Cynthia Rudin. 'Exploring and Interacting with the Set of Good Sparse Generalized Additive Models'. In: Advances in NeurIPS-23 . 2023, pp. 56673-56699.

Martino Ciaperoni, Han Xiao, and Aristides Gionis. 'Efficient Exploration of the Rashomon Set of Rule-Set Models'. In: Proceedings Of The 30th ACM SIGKDD Conference On Knowledge Discovery And Data Mining . 2024, pp. 478-489.

Satoshi Hara and Masakazu Ishihata. 'Approximate and Exact Enumeration of Rule Models'. In: Proceedings of AAAI-18 . 2018, pp. 3157-3164.

James N. Morgan and John A. Sonquist. 'Problems in the Analysis of Survey Data, and a Proposal'. In: Journal of the American Statistical Association 58.302 (1963), pp. 415-434.

Gordon V. Kass. 'An Exploratory Technique for Investigating Large Quantities of Categorical Data'. In: Journal of the Royal Statistical Society. Series C (Applied Statistics) 29.2 (1980), pp. 119-127.

Michael R. Garey and Ronald L. Graham. 'Performance Bounds on the Splitting Algorithm for Binary Testing'. In: Acta Informatica 3.4 (1974), pp. 347-355.

Laurent Hyafil and Ronald L. Rivest. 'Constructing optimal binary decision trees is NPcomplete'. In: Information processing letters 5.1 (1976), pp. 15-17.

Sebastian Ordyniak and Stefan Szeider. 'Parameterized Complexity of Small Decision Tree Learning'. In: Proceedings of AAAI-21 . 2021, pp. 6454-6462.

Jacobus G. M. van der Linden, Mathijs de Weerdt, and Emir Demirovi´ c. 'Necessary and Sufficient Conditions for Optimal Decision Trees using Dynamic Programming'. In: Advances in NeurIPS-23 . 2023, pp. 9173-9212.

Emir Demirovi´ c and Peter J. Stuckey. 'Optimal Decision Trees for Nonlinear Metrics'. In: Proceedings of AAAI-21 . 2021, pp. 3733-3741.

Jacobus G. M. van der Linden, Mathijs M. de Weerdt, and Emir Demirovi´ c. 'Fair and Optimal Decision Trees: A Dynamic Programming Approach'. In: Advances in NeurIPS-22 . 2022, pp. 38899-38911.

Xiyang Hu, Cynthia Rudin, and Margo Seltzer. 'Optimal Sparse Decision Trees'. In: Advances in NeurIPS-19 . 2019, pp. 7267-7275.

Donald B. Johnson and Tetsuo Mizoguchi. 'Selecting the K th Element in X + Y and X 1 + X 2 + · · · + X m '. In: SIAM Journal on Computing 7.2 (1978), pp. 147-153.

Greg N. Frederickson and Donald B. Johnson. 'The Complexity of Selection and Ranking in X + Y and Matrices with Sorted Columns'. In: Journal of Computer and System Sciences 24.2 (1982), pp. 197-208.

Sicco Verwer and Yingqian Zhang. 'Learning Optimal Classification Trees Using a Binary Linear Program Formulation'. In: Proceedings of AAAI-19 . 2019, pp. 1625-1632.

Rui Zhang, Rui Xin, Margo Seltzer, and Cynthia Rudin. 'Optimal Sparse Regression Trees'. In: Proceedings of AAAI-23 . 2023, pp. 11270-11279.

Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. 'A survey on datasets for fairness-aware machine learning'. In: Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery (2022), e1452.

Delft High Performance Computing Centre (DHPC). DelftBlue Supercomputer (Phase 2) . https://www.tudelft.nl/dhpc/ark:/44463/DelftBluePhase2 . 2024.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. 'Distribution-free predictive inference for regression'. In: Journal of the American Statistical Association 113.523 (2018), pp. 1094-1111.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias . May 2016. URL: https://www.propublica.org/article/machine-bias-risk-assessments-incriminal-sentencing .

FICO, Google, Imperial College London, MIT, University of Oxford, UC Irvine, and UC Berkeley. Explainable Machine Learning Challenge . 2018. URL: https://community. fico.com/s/explainable-machine-learning-challenge .

Maurice G. Kendall. 'A New Measure of Rank Correlation'. In: Biometrika 30.1-2 (1938), pp. 81-93.

Siegfried Nijssen and Elisa Fromont. 'Optimal constraint-based decision tree induction from itemset lattices'. In: Data Mining and Knowledge Discovery 21.1 (2010), pp. 9-51.

Dheeru Dua and Casey Graff. UCI Machine Learning Repository . 2017. URL: http:// archive.ics.uci.edu/ml .

Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. 'A bayesian framework for learning rule sets for interpretable classification'. In: Journal of Machine Learning Research 18.70 (2017), pp. 1-37.

Christian Bessiere, Emmanuel Hebrard, and Barry O'Sullivan. 'Minimising decision tree size as combinatorial optimisation'. In: International Conference on Principles and Practice of Constraint Programming . 2009, pp. 173-187.

Mim van den Bos, Jacobus G. M. van der Linden, and Emir Demirovi´ c. 'Piecewise Constant and Linear Regression Trees: An Optimal Dynamic Programming Approach'. In: Proceedings of ICML-24 . 2024.

Dutch Central Bureau for Statistics. Dutch Census 2001 public use files (anonymized 1% samples from the microdata files) . DANS. 2001. DOI: https://doi.org/10.17026/dansxms-xc4a .

Paulo Cortez and Alice Maria Gonçalves Silva. 'Using data mining to predict secondary school student performance'. In: Proceedings of 5th FUture BUsiness TEChnology Conference . 2008, pp. 5-12.

Beata Strack, Jonathan P. DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, and John N. Clore. 'Impact of HbA1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records'. In: BioMed research international (2014).

Sérgio Moro, Paulo Cortez, and Paulo Rita. 'A data-driven approach to predict the success of bank telemarketing'. In: Decision Support Systems 62 (2014), pp. 22-31.

Jakub Kuzilek, Martin Hlosta, and Zdenek Zdrahal. 'Open university learning analytics dataset'. In: Scientific data 4 (2017), p. 170171.

Nathanael Jo, Sina Aghaei, Jack Benson, Andrés Gómez, and Phebe Vayanos. 'Learning Optimal Fair Decision Trees: Trade-offs Between Interpretability, Fairness, and Accuracy'. In: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society . 2023, pp. 181-192.

NeurIPS Paper Checklist

If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, we state the contributions including the key results in the abstract and introduction section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Enumerating the Rashomon set for larger depth budgets, dataset sizes, or target set sizes becomes intractable since even finding a single optimal tree is NP-hard. Our method also requires input datasets to be binary.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not have theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we provide the steps to reproduce main results of the paper by explaining the algorithm in high-level (Sec. 4.1) and in detail (Sec. 4.2, Appendix A.1-A.9) Additionally, we will provide open access to the code when the paper is published.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will provide open access to the code if the paper is published. We use open datasets for evaluations and their sources are cited in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We explain the instance generation steps and how the algorithms are run in Sec. 5 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard error of the runtime experiments in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resource is detailed in Sec. 5. We additionally provided the runtime results of the experiments in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We checked our paper with the Ethics Guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our method is a general machine learning model. It is not tailored to any specific application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not aware of any risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we have cited the original authors and publishers of all codes and datasets used in the paper. Furthermore, our code includes acknowledgments for any components developed by third parties.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide explanations in the code repository. We will provide open access to the code once the paper is published.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not involve crowdsourcing and did not use human subjects for this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not involve crowdsourcing and did not use human subjects for this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We did not use LLMs for core method development.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Detailed method description

A.1 Main algorithm of Rashomon set calculation

We generate the Rashomon set by iteratively computing trees in ascending order of their objective values. Alg. 4 summarizes how the Rashomon set is computed once the optimal solution has been found. The algorithm takes as input: the maximum depth d , the training dataset D , the optimal solution value sol ∗ , the Rashomon multiplier ε , the maximum number of trees n T , and a cache of optimal solutions of the subtrees. The Rashomon multiplier and the maximum number of trees are complementary: specifying either one is sufficient. If the Rashomon multiplier is omitted, a large default value is used to derive the bound.

Next, the root search node is initialized by setting its upper bound to the Rashomon bound and retrieving its optimal solution from the cache (Line 2). This optimal value is then compared with the bound to determine whether any solutions lie within the Rashomon set. If the condition is satisfied, additional solutions are iteratively added to the node's sorted solution list until a stopping criterion is met (Lines 4-6).

Algorithm 4 Main( d, D, sol ∗ , ε, n T , cache )

1: θ ← sol ∗ . value × (1 + ε )

2: root _ search _ node ← InitializeSearchNode( d, D, θ, cache )

3: sol ← root _ search _ node . sol ∗

4: while sol . value ≤ θ and | root _ search _ node . SSL | < n T do

5: sol ← root _ search _ node . GetNextSolution()

6: root _ search _ node . SSL ← root _ search _ node . SSL ∪ { sol }

7: return root _ search _ node . SSL

A.2 Solutions structure

A Rashomon set may contain well over a billion trees, and enumerating such a large set has a high computational and memory load. Therefore, we adopt the grouped solution structure used in [30]: Fig. 9 shows how (partial) solutions (i.e., subtrees) are stored in memory. Solutions are recursively grouped by their objective value and the splitting feature of the (subtree) root node, followed by a list of pairs of solutions for the left and right subtrees. The details of how same-valued solutions are grouped are given in Sec. 4.2.

Example 3. In Fig. 9, the left of the figure represents different solution values of a branching node. Because the first two solution values are the same, a solution group (right upper side of the image) with two solution pairs is created. As for the remaining solution value, a solution group (right lower side of the image) with one solution pair is created.

0

1

0

A.3 Incorporating the regularization cost

The regularization term in the objective function penalizes model complexity by adding λ for every leaf node (Eq. (1)). Consequently, creating a new split that raises the leaf node count by one increases the objective value even if predictive accuracy is unchanged. To capture this effect, we assign a fixed cost of λ to each branching decision. If the tree consists of a single root leaf, the same cost λ is added once for that leaf node.

We use the branching cost while combining left and right solutions and while determining the upper bounds of the child search nodes (see Appendix A.4).

To construct a solution for a branching (search) node, the algorithm first calculates solutions for its left and right child nodes. It then combines these partial solutions to obtain the parent solution (Alg. 2, Line 11). Since the learning objective imposes a regularization penalty of λ per additional leaf, we add the same branching cost of λ when combining the left and right subtree solutions as follows:

A.4 Upper bounding

To avoid exploring low-quality regions that cannot contribute to the Rashomon set, every search node is equipped with an upper bound on the solution value of any tree it may still yield. The root node's upper bound is initialized with the Rashomon bound, and each helper node simply inherits its parent's upper bound. Child search nodes, however, receive tighter bounds that account for the best solution already attainable in the complementary subtree. Given a branching node and its upper bound UB , the upper bounds of the child search nodes are calculated as follows:

Here, λ is to account for the branching cost (see Appendix A.3) while sol ∗ L and sol ∗ R are the optimal solution values of the left and right child search nodes, respectively. These optimal solutions are either retrieved from a cache of optimal solutions or computed using STreeD [40] if absent.

We also use the upper bounds of branching and search nodes for two additional purposes: (1) to filter out helper nodes whose next solution value exceeds its search node's upper bound (see Alg. 1); and (2) to determine whether a newly generated solution obtained by combining a left and a right solution should be accepted, by comparing it against the branching node's upper bound (see Alg. 2).

A.5 Caching

A subtree can be encountered multiple times during the search. To avoid recomputing its solutions in different parts of the search space, we cache both solutions themselves-stored in the search node's sorted solution list-and the objects required to compute them, namely the branching search nodes.

We adopt the caching mechanism of Demirovi´ c et al. [16], which supports two strategies: dataset caching , in which a subtree is identified by the set of samples it contains, and branch caching , in which it is identified by the features used in every branching decision from the root to that subtree. In our experiments, we used dataset caching.

When a new search node is created, the algorithm first queries the cache. If the subtree is present, the node reuses the cached solution list and branching nodes. The solution list is then extended-if not already exhausted-by all search nodes that share it.

A.6 Depth-two subroutine

We use a specialized algorithm to calculate the Rashomon set when the remaining depth budget is at most two. Sec. 4.2 details how, for a given search node, we enumerate all trees that contain three branching nodes. Below, we describe the cases with one and two branching nodes.

Alg. 5 outlines the procedure for generating all trees with a single branching node. The algorithm takes as input a search node node , a feature set F , class frequency counts Q + and Q -, and an upper bound UB . For each feature in F , the algorithm computes the label assignments for the left and right

children that minimize their respective subtree costs. These two partial solutions are then combined (Line 2; see Appendix A.3 for details). If the resulting solution value is within the upper bound, the solution is added to the search node's sorted solution list (Lines 3-4).

For trees with exactly two branching nodes, the feasible topology is restricted to one of two mirror images: either the left subtree is a leaf and the right subtree contains a single branching node, or vice versa. Alg. 6 handles the case in which the right child is a leaf; the symmetric case is treated analogously.

The procedure begins by selecting a feature for the root split and computing the solution of the right subtree being a leaf (Line 3). Next, it creates a second split in the left subtree using any remaining feature from F and evaluates every possible label assignment, selecting the minimal left-subtree solution value (Line 5). If this value does not exceed the upper bound, the left-subtree solution is stored (Lines 6-7). Each stored left solution is then combined with the right-leaf solution. The resulting trees are inserted into the search node's sorted solution list whenever its overall solution value remains within the upper bound (Lines 8-12).

̸

**The case where the left tree is a leaf node and the right tree has a branching node is calculated analogously.**

A.7 Gradual solution creation

When the Rashomon set is subject to a size limit (see Alg. 4), enumerating only a subset of the depth-two trees can suffice. We achieve this by tightening the upper bound, hence pruning more candidates, reducing memory usage and runtime. If the bound is too low, however, the algorithm may spend significant effort (e.g., computing frequency counts) only to yield a single solution; if it is too high, many unused trees are still generated. Empirically, a bound that guarantees the inclusion of the leaf solution and all trees with at most one branching node strikes a good balance. Therefore, we initialize the depth-two upper bound as

where D and UB denote the dataset and the search node's bound, T is the single-leaf tree at that node, and λ is the branching cost (Appendix A.3).

With ˆ UB set, Alg. 3, 5, and 6 generate the depth-two solutions. If the calculated solutions are exhausted before the Rashomon-set size limit is reached, we relax the bound as follows:

This increases the upper bound at a diminishing rate, as the number of solutions grows exponentially with respect to the changes in the upper bound.

A.8 Ignoring trivial extensions

By default, SORTD allows extending a tree node with two child nodes that share the same label. Following Xin et al. [30], we refer to these as trivial extensions . SORTD can optionally disregard such trees when calculating the Rashomon set. A trivial extension occurs when a branching node produces two leaf nodes whose labels are identical, so the branching does not change the objective value except for the added complexity cost. A trivial extension is detected during the depth-two subroutine (e.g., Alg. 3 Line 12) or branch-tracker iterations (Alg. 2 Line 11) if the following condition holds:

In some cases, at least one leaf label may be arbitrary: assigning one label or another does not affect the objective value. In this case, SORTD assigns one label arbitrarily, stores one of the alternatives, and when combining with another leaf node, selects the label that avoids creating a trivial extension.

Removing trivial extensions reduces the number of trees in a Rashomon set for a fixed Rashomon multiplier, or equivalently, increases the Rashomon multiplier attainable for a fixed set size. As shown in Appendix B.6, SORTD maintains performance improvements of up to two orders of magnitude also when trivial extensions are excluded.

A.9 Optimizing other objectives

SORTD supports computing Rashomon sets for separable and totally ordered objectives directly and post-evaluating other separable and partially-ordered objectives over an already computed Rashomon set indirectly .

Optimizing totally ordered objectives SORTD's Rashomon construction is based on the STreeD framework [40], and therefore easily generalizes to other objectives than the regularized misclassification score. However, since SORTD requires sorting partial solutions in its search nodes, it cannot support all optimization tasks that STreeD supports i.e., all separable and partially ordered tasks. Specifically, SORTD requires separable and totally ordered solutions. Additionally, in our implementation, we assume all objectives are additive , i.e., solutions from left and right subtrees can be combined using addition. SORTD therefore, has the same generalizability as DL8 [54].

To support other objectives, the only major change is to redefine the objective function C ( T, D ) . For example, to optimize regression trees by minimizing the sum of squared errors , we define:

To illustrate this, we show in Appendix B.8 SORTD's performance in finding the Rashomon set for regression trees.

Post-evaluating other objectives After generating a Rashomon set, the trees in this set can be evaluated using any objective by going over each tree one at a time. However, computing an objective for each tree individually can be expensive. Therefore, we provide a special procedure to post-evaluate objectives that are separable (as defined by Linden et al. [40]), i.e., we can compute the objective values separately for the left and right subtrees and combine them afterwards. This requires defining some cost function g ( D,k ) for leaf nodes with D the input dataset and k the assigned label. This cost function does not necessarily need to return a real-valued number, but can also, for example, return a tuple of values. Similarly, we need to define a combining operator ⊕ that takes a left and right solution and returns the cost of combining both (again, not necessarily a real-valued number).

SORTD uses these functions to construct a new Rashomon set, with equal-valued solutions grouped together (see Appendix A.2). However, in this post-evaluation, we no longer require the objective to be totally ordered, so we can no longer acquire the solutions in order of the second objective. In practice, we obtain the Rashomon set in order according to the optimization objective (e.g., regularized accuracy), by retrieving them in batches. We run Alg. 4 for a certain Rashomon set size; evaluate the obtained solutions using the second objective; and possibly rerun Alg. 4 with a larger

Rashomon set size, continuing where we left. This can be repeated until a certain time-out or some other user-defined stopping criterion.

To facilitate ease of evaluation, SORTD allows users to define the leaf cost function g and the combining operator ⊕ in Python and pass them to SORTD. Consider, for example, optimizing trees with a multi-objective criterion of regularized accuracy and the fairness metric equality-of-opportunity . The equality of opportunity loss is defined as the difference in the true positive rate between two discrimination sensitive groups. More formally, let y be the true label, ˆ y the predicted label, then equality of opportunity requires:

We can compute this by defining a leaf cost function that counts for both groups how many samples with the true positive label are also predicted with a positive label. Additionally, we count the total number of misclassifications, so that we can also compute the accuracy. Hence, the solution tuple returned will be ( misclassifications , positive _ count _ group _ 0 , positive _ count _ group _ 1 ) . The combining operator is then simply element-wise addition of two tuples. Finally, we need to provide a function that computes from these tuples in the root node the actual objective value. In this example, the final objective value is a tuple of the accuracy and the difference in the true positive rate for the two groups. These tuples can then be filtered to only keep the Pareto front. In Python, we can write:

B Experiment details

B.1 Classification datasets

We selected all 46 binary classification datasets with less than 100 binary features from [16] and [30]. 2 We excluded nine duplicate datasets and three trivially solvable ones (solved in under one second). We also removed four datasets that exceeded memory limits when calculating the Rashomon multipliers for specific Rashomon set sizes. This computation was memory intensive because the Rashomon multipliers were unknown, which required setting the upper bound to a high value.

After these exclusions, 30 datasets remained. From these, we eliminated duplicate and complementary features. Table 1 lists the datasets together with their sample size and resulting feature size, | D | and | F | , respectively. These datasets were used in runtime (Sec. 5.1) and variable importance analysis (Sec. 5.2). The original datasets can be obtained from the UCI Machine Learning repository [55] and from [51, 52, 56, 57].

B.2 Runtime performance

We generated benchmark instances by varying the complexity penalty λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } and the depth budget d ∈ { 3 , 4 , 5 } . For each ( dataset , λ, d ) combination, we use SORTD to compute the minimum Rashomon multiplier (and corresponding Rashomon bound) that yields at least 10 n trees for every n ∈ { 1 , . . . , 6 } . Because both methods employ a grouped solution structure, requesting 10 n solutions may sometimes yield more than 10 n +1 solutions, with the additional solutions sharing the same objective value. In such cases, we did not generate a separate instance for the 10 n +1 target, as it was already satisfied. Finally, we ran each instance five times.

We compare SORTD with the state-of-the-art algorithm TreeFARMS [30], under a 300second time limit per instance. Each experiment is repeated five times. We set rashomon_ignore_trivial_extensions = False for both methods (see Sec. B.6 for runtime results with trivial extensions ignored). Runtime results are presented in Tables 2-4. The results of the instances with depth budget three are not presented as both methods solved most of them within a second. Entries marked '-' indicate omitted runs due to excessive memory usage. We denote runtimes below one second with ' < 1 ', and timeouts with ' > 300 '.

As expected, increasing λ simplifies the search problem for both methods. A larger penalty on leaf usage encourages shallower trees, which effectively prunes deeper parts of the search space. In contrast, small λ values lead to significantly larger search spaces.

Across most instances, SORTD completes enumeration within one second for depth four and within ten seconds for depth five. In contrast, TreeFARMS exhibits high variability-even for depth four, runtimes range from below a second to full timeouts. Its performance degrades further with increasing depth, frequently timing out for λ = 0 . 001 and λ = 0 . 01 especially when the feature size of the dataset is high.

To evaluate the overall performance of SORTD compared to TreeFARMS, Tables 2-4 also report the geometric mean of the average runtime ratios t TreeFARMS /t SORTD, where timeout values are considered as 300 seconds. Results show that SORTD achieves up to two orders of magnitude improvement in runtime over TreeFARMS for Rashomon set enumeration. These findings further emphasize SORTD's scalability, particularly as problem complexity increases with respect to tree depth, dataset size, and regularization parameter λ .

Longer runtime evaluation for timeout instances Runtime results in Tables 2-4 show that several TreeFARMS [30] instances reached the 300-second time limit. To further examine these cases, we extended the runtime limit to 3600 seconds. Tables 5-6 present the results for depth budgets four and five, respectively.

With the extended limit, most depth-four instances complete within the allotted time. For depth five, TreeFARMS requires substantially more memory for some larger instances and longer runtimes for others, highlighting SORTD's improvements in runtime and memory efficiency.

B.3 Memory Performance

Using the same experimental setup described in Sec. B.2, we additionally evaluated the memory usage of both methods. Tables 7-9 show the detailed results (in gigabytes). Datasets for which both methods required less than 1 GB of memory are omitted from the tables, although their values are included in the geometric means of the average memory usage ratios m TreeFARMS /m SORTD reported at the end of each table.

The geometric mean results indicate that, for each depth and λ configuration, SORTD reduces memory usage by more than an order of magnitude compared to TreeFARMS. Combined with the runtime results, these findings demonstrate that SORTD achieves superior overall efficiency in both runtime and memory performance.

Across all datasets, maximum depths, and λ values, SORTD's memory usage consistently remains below 1 GB. In contrast, TreeFARMS shows substantially higher memory requirements, particularly at greater depths for λ = 0 . 001 and λ = 0 . 01 , and in datasets with larger feature spaces.

B.4 Rashomon multipliers

Table 10 reports the minimum Rashomon multiplier ε values computed by SORTD to yield at least 10 n trees for each n ∈ { 1 , . . . , 6 } for a subset of datasets. These results correspond to instances with λ = 0 . 01 and a depth budget of four. For readability, values are rounded to two decimal places.

The table reveals substantial variation in the Rashomon multipliers required to achieve a given set size across datasets. For example, while ε = 0 . 16 is necessary to obtain 10 trees for the hypothyroid

dataset, the same value yields over one million trees for the spect dataset. This illustrates the difficulty of selecting an appropriate Rashomon multiplier to target a desired set size. SORTD addresses this challenge through its ordered solution enumeration and anytime behavior: enumeration can be stopped once the desired number of models is reached, while still preserving the Rashomon set property-without requiring prior knowledge of the Rashomon multiplier (or bound).

B.5 Comparison of the tree ordering

A key difference between the state-of-the-art approach (TreeFARMS [30]) and SORTD lies in how trees are ordered within the Rashomon set. Fig. 10 illustrates this distinction. The figure reports the minimum number of trees that must be examined sequentially from the start of the Rashomon set to include all trees within the top x % of the lowest distinct objective values.

25

50

75

100

Top x% of lowest objective values

0

200

400

Largest index

Bar-7 (λ = 0.01, ε = 0.05)

TreeFARMS

25

50

75

100

Top x% of lowest objective values

0

500

1000

Largest index

Monk2 (λ = 0.01, ε = 0.1)

TreeFARMS

As the examples show, identifying trees within the top 20% of lowest objective values using the state-of-the-art method may require exploring nearly the entire set. In contrast, SORTD provides early access to high-quality solutions, greatly reducing the overall evaluation effort.

B.6 Runtime performance without trivial extensions

In Sec. B.2, we evaluated the runtime performance of SORTD while allowing trees with trivial extensions during Rashomon set computation. A trivial extension refers to a split that produces two leaf nodes with the same label (see Sec. A.8). We now assess how excluding such trees affects SORTD's runtime performance.

Following the same experimental setup as in Sec. B.2, we generated benchmark instances by varying the complexity penalty λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } and the depth budget d ∈ { 4 , 5 } . For each ( dataset , λ, d ) combination, SORTD was used to compute the minimum Rashomon multiplier (and

corresponding Rashomon bound) required to obtain at least 10 n trees for every n ∈ { 1 , . . . , 6 } . Using these multipliers, we constructed the respective Rashomon sets to evaluate the performance of SORTD against TreeFARMS.

Runtime results are presented in Tables 11-13. For a fixed Rashomon set size, ignoring trivial extensions can increase the required multiplier values, making some instances slightly more difficult. This is reflected in a modest rise in runtime. Nevertheless, SORTD's runtime performance remains stable, consistently achieving up to two orders of magnitude speed improvement over TreeFARMS.

B.7 Variable importance analysis

Variable importance on Rashomon sets was evaluated using the leave one feature out procedure [50] with λ = 0 . 01 , depth d = 4 . For each dataset, we generated 20 stratified bootstrap samples, computed the Rashomon multiplier required to obtain at least 10,000 trees, and constructed the corresponding set using these multipliers. For each bootstrap sample, we also calculated the top-1 and top-100 Rashomon sets and ranked features according to the increase in area under the tree-index versus objective-value curve when each feature was removed. Kendall's τ and Jaccard index were used to evaluate the stability of the full and top-5 variable ranking, respectively. Table 14 presents the results.

The stability of the top-1 tree versus the top-10,000 trees is low for most datasets with only five of them satisfying τ ≥ 0 . 7 , and 19 of them have a Jaccard index of value at least 0 . 8 . In contrast, for the top-100 versus top-10,000 trees, 17 datasets satisfy τ ≥ 0 . 7 and 26 exceed a Jaccard index exceeding 0.8, indicating that smaller Rashomon sets can yield similar variable importance values as larger Rashomon sets.

B.8 Regression results

This section provides the details on the regression experiment shown in Fig. 7 and additionally analyses the runtime performance of SORTD for regression.

Data For the regression experiments, we use the datasets from Zhang et al. [47] and also follow their binarization of non-binary features. 3 The datasets can also be obtained from the UCI Machine Learning repository [55]. For all datasets, we normalize the regression label by subtracting the mean and dividing by the standard deviation.

Comparison with CART and STreeD Since there are, to the best of our knowledge, no previous methods to enumerate the whole Rashomon set of regression trees, we follow the set-up by Xin et al. [30], who did a similar experiment for classification trees: we compare SORTD with both the heuristic CART [2] and the optimal method STreeD [40] when those are called repeatedly on random samples of the data. STreeD is a state-of-the-art optimal method for computing optimal regression trees [58]. 4

In this experiment, we impose a 60 seconds timeout for each method. CART and STreeD are run repeatedly within that time budget on random samples of 50% of the total dataset. All resulting unique trees are kept. SORTD, on the other hand, is run once and collects trees in the Rashomon set in order until time-out or until the Rashomon set is exhausted. We use the Rashomon multiplier ε = 0 . 1 , complexity cost λ = 0 . 001 , and maximum depth d = 4 .

Fig. 7 in the main text shows how SORTD can find orders of magnitude more trees in the Rashomon set than either CART or STreeD within the time limit. For the Synchronous Machine Dataset , SORTD finds the whole Rashomon set within two seconds, whereas both STreeD and CART find only a fraction in the allotted 60 seconds. For both Airfoil Self-Noise and Seoul Bike Sharing Demand , SORTD finds orders of magnitude more trees than either CART or STreeD in the given time limit. These results confirm what Xin et al. [30] also concluded for classification: enumerating the whole Rashomon set can be done best with a dedicated method.

Runtime Additionally, we analyse the runtime performance of SORTD to calculate the Rashomon set of regression trees. Here, we set the complexity cost λ = 0 . 01 , the Rashomon multiplier to ε = 1 . 0 , the maximum number of trees n T = 10 6 , and we test both with a maximum depth of four and five. We run all experiments five times on all datasets with a time-out of 300 seconds.

Table 15 shows that SORTD successfully enumerates the top one million trees for all benchmark datasets within the time limit for d = 4 , except for one, and for all but three for d = 5 .

In comparison to the experiments on classification trees shown above in Appendix B.2, the regression tree Rashomon set is more time intensive to compute. We think this is because the regression loss allows for many more unique loss values. Since SORTD combines solutions with the same value (see Appendix A.2), more unique loss values result in a higher runtime. Runtime performance can

likely be improved by setting a higher tolerance for which solution values are considered the 'same'. Currently, SORTD uses a tolerance of 10 -4 .

B.9 Equality-of-opportunity results

This section provides the details on the equality-of-opportunity experiment shown in Fig. 8 and additionally analyses the runtime performance of SORTD for evaluating such a second objective.

Data For the equality-of-opportunity experiments, we use the datasets from Le Quy et al. [48] and follow their suggested binarization of non-binary features. In Table 16, we report which feature is selected as the discrimination sensitive feature. References to the original datasets can be found here [51, 55, 59-63].

Pareto front Fig. 8 shows how SORTD can be used to generate a Rashomon set for sparse classification trees, which are then evaluated using a secondary objective: equality of opportunity. Since SORTD generates the solutions in increasing order of its objective, it examines the most accurate trees first. Therefore, when halting the search at any time, a (partial) Pareto front over the primary and secondary objectives can be computed. The results in Fig. 8 show that among the top 10 7 sparse depth-four classification trees for the Adult dataset, all trees have at least 2% discrimination, i.e., the true positive rate of one group is at least 2% higher than another. For the Bank Marketing dataset, a tree with zero discrimination is found among the top 10 7 trees. For the Compas dataset the most fair dataset in the top 10 7 trees still has 2.5% discrimination. It also shows that by reducing accuracy by only 0.5%, the discrimination score can be lowered from 7% to 2.5%.

Runtime comparison We evaluate SORTD's efficiency in evaluating a second objective such as equality of opportunity by comparing it with the optimal dynamic programming approach STreeD [40], which supports optimizing accuracy and equality of opportunity. We do not compare with the mixedinteger programming approach by Jo et al. [64] since Linden et al. [40] report runtimes several orders of magnitude lower than theirs, while computing the same optimal solutions.

We run SORTD by iteratively generating the next best n T trees according to the regularized accuracy objective. We then evaluate the newly generated trees as described in Appendix A.9 using the equality-of-opportunity metric. If a tree is found within the pre-set discrimination limit δ , we stop. Otherwise, SORTD generates the next batch of n T trees and repeats. Since SORTD yields trees in order of the regularized accuracy objective, the first tree it finds within the discrimination limit must be optimal with respect to the regularized accuracy objective.

In the comparison with STreeD, there are a couple of small differences that may influence runtime: (1) We run SORTD using the regularized accuracy objective, so each leaf node is penalized with

the sparsity penalty λ . In this experiment, we set λ = 0 . 01 . STreeD's equality-of-opportunity optimization task by default does not consider such regularization. (2) STreeD also considers nonmajority labels in leaf nodes, whereas SORTD by default does not. Because of these two differences, STreeD and SORTD are not guaranteed to find the same optimal solution, although we verified that the solutions are close. Since our main aim in this experiment is to validate that SORTD can successfully be used to post-evaluate a secondary objective, we consider these differences acceptable.

Table 16 shows the mean runtime performance of STreeD and SORTD to find fair and optimal trees with max-depth d = 3 , discrimination limit δ = 1% , and sparsity penalty λ = 0 . 01 (for SORTD). With SORTD, we iteratively produce and evaluate n T = 10 5 trees until a tree within the discrimination limit is found, or until the time-out of 300 seconds. The results show that SORTD remains close in performance to STreeD, and for some datasets, such as German credit and OULAD , even performs significantly better. These results are obtained by evaluating the secondary objective using callbacks to Python. Hence, if runtime performance was the main concern, these results could be further improved by computing this also in C++.

Concluding, these results show that using Rashomon sets to optimize one (totally ordered) objective, and later evaluating the Rashomon set (possibly iteratively) using a second objective (such that this multi-objective optimization task is only partially ordered) is promising.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work gives a method to enumerate the Rashomon set of sparse decision trees, over any totally ordered objective, in order of the trees' dataset objective value. The algorithm stores partial solutions at descending levels of the tree in two types of 'helper' nodes -- branching nodes, which enable further splits, and leaf nodes, which terminate the tree. Broadly speaking, the algorithm computes optimal solutions to subproblems in every branching node, and prunes subproblems by keeping running upper and lower bounds on the possible objective in a subproblem. The enumeration algorithm outputs the optimal tree at the top level (the global optimal tree), then in some sense \"percolates\" optimal solutions from the left and right subtrees up to the top of the tree. The algorithm stops when it has either enumerated all trees in the Rashomon set, has reached a time limit, or when it has enumerated a fixed number of trees. Thus, this algorithm gives two novel capabilities; first, it can enumerate the Rashomon set of sparse decision trees in order; second, it can compute the Rashomon set with respect to any totally ordered (and, for practical purposes, separable) objective function. The authors also include timing experiments to demonstrate an improvement over the current state-of-the-art in TreeFARMS, as well as case studies in variable importance, Rashomon sets of sparse regression trees, and fairness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The insight that the STreeD framework for decision tree optimization can be adapted to find all models in the Rashomon set of totally ordered (and perhaps additive) objective functions in order, and quickly at that, strikes me as both novel and significant. \n\n2. The speedups over TreeFARMS, in combination with the anytime enumeration capability and objective function flexibility, are fairly represented by this paper to be steps toward the practical application of Rashomon set methods. \n\n3. The paper was succinct, well-written, and easy to follow. The paper was well-situated among other literature in the related work section. The descriptions of the algorithms, both in the high-level idea and in the details in the paper and the appendix, were clear and I feel as if I understand both their motivation and implementation.\n\n\n## Weaknesses\n\n### Significant Weaknesses\n\nIn summary, I felt as if the case studies provided to demonstrate the practical utility of SORTD were somewhat superficial.\n\n1. The Model Class Reliance metric due to Fisher et al., 2019, already captures the essence of the LOFO case study. The resulting insights in the first example that \"priors > 3\" and \"ExternalRiskEstimate\" are important are not new, and are easily obtainable via existing approaches. \n\n2. The work of Donnelly et al., 2023: The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance gives a principled way to obtain a distribution of variable importances in the Rashomon set via bootstrapping.\nIn the second example, I feel as if the novel insight from this paper is not that Rashomon sets improve the stability of variable importance (which is thoroughly discussed by Donnelly et al., 2023), but that you don't need many models from the Rashomon set to significantly improve reliability. This point felt under-explored -- indeed, only 20/29 of the datasets analyzed met the 0.8 threshold for Jaccard index for top-100, which to me is not enough to indicate reliability. \n\n3. Similarly, the example given for generating sparse regression trees, while interesting for the fact that SORTD can indeed compute the Rashomon set of sparse regression trees, did not add much to the paper. TreeFARMS was already shown by Xin et al., 2022 to dominate methods which generate single optimal trees on bootstraps, and there is no reason to expect that another Rashomon set algorithm should be different. I think that the main point of this section was that SORTD can indeed effectively compute the Rashomon set of objective functions beyond misclassification error, and perhaps this could have been emphasized moreso than comparison to CART and STreeD.\n\n### Minor Weaknesses\n\n1. The abstract seems to imply that SORTD somehow bypasses the NP-hard optimal tree problem by enumerating Rashomon sets in order. This is an issue of wording, not content.\n\n2. I think that this work would benefit from a figure or analysis supporting the claim on line 243, that the scalability of SORTD derives from the shortening of solution lists, despite more branching nodes. This analysis seems plausible when adding bad or do-nothing features to the dataset, but when adding good features I would expect the number of models in the Rashomon set to explode, and concurrently for solution lists to get longer even though there are more branching nodes.\n\nQUESTIONS:\n1. Do you think that SORTD's speed advantage would persist if compared against TreeFARMS, but with the depth-2 solver? To rephrase -- is it the optimization framework itself that gives the advantage over TreeFARMS, or the incorporation of a very helpful extra algorithmic component?\n\n2. Is this algorithm able to easily exclude trivial extensions from the Rashomon set (as defined in the TreeFARMS algorithm; extra splits that don't change predictions, but also don't exceed the Rashomon bound)? Ignoring trivial extensions is a useful feature for TreeFARMS which also aids computation, so comparing to TreeFARMS without ignoring trivial extensions appears to be hamstringing TreeFARMS unduly, if SORTD is in fact unable to easily ignore them. \n\n3. How does the Rashomon set of sparse regression trees compare to the Rashomon set of sparse decision trees (say, on a threshold of the continuous regression label to turn it into a classification problem)? Are the Rashomon sets similar? Do the trees share many common prefixes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SORTD, a method to generate Rashomon Sets of sparse decision tree models.\n\nThe proposed algorithm uses dynamic programming with bounds, caching, and specialized subroutines to efficiently represent and explore the space of sparse decision trees in order of ascending (best first) objective function. Because it generates all sparse decision trees in the Rashomon set in ascending order of their objective function, SORTD provides anytime guarantees and does not require pre-fixing the Rashomon parameter $\\epsilon$ -- which are two important advances compared to the state-of-the-art. \n\nAn efficient procedure is also provided to compute partially ordered objectives in a post-hoc manner all over the Rashomon set, without having to explicitly compute it from scratch for each tree.\n\nThorough experiments confirm the efficiency of the proposed algorithm to generate Rashomon sets for any totally ordered objective and to evaluate them efficiently with respect to additional (partially ordered) desiderata (here, the equal opportunity statistical fairness metric).\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The problem is well-motivated, the related work is properly introduced and discussed. The proposed approach is technically solid, and the empirical evaluation is thorough. \n\nClarity: I found the paper pleasant to read. The different bounds, algorithms and subroutines are sufficiently detailed. Nice visualizations are provided for all experiments, allowing assessment of the results and their statistical significance.\n\n Significance: SORTD improves the state-of-the-art by (i) allowing an anytime behaviour (ii) avoiding pre-fixing a Rashomon parameter which could be too large or too small depending on the data and hypothesis class at hand and (iii) empirically improving scalability and efficiency of generating Rashomon sets for sparse decision trees (compared to state-of-the-art approaches). The ability to handle arbitrary objectives (as long as they are totally ordered and additive) is also a significant advantage of the method. I believe the provided implementation can be a useful tool for the interpretable machine learning community.\n\nOriginality: The proposed algorithm incorporates many building blocks from the literature on (optimal) sparse decision trees optimization, and on constructing their Rashomon sets. It results in a very technical contribution, with the conceptual/theoretical novelty being limited.\nNevertheless, the core idea of exploring the Rashomon set of sparse decision trees in ascending objective function order (i.e., best first) makes sense and empirically provides significant advantages (as aforementioned). \n\nScalability remains limited, as Rashomon sets are empirically explored only for decision trees with depth <= 5, and even in this setup, 100GB of RAM are necessary (with some experiments actually reaching the limit). But still, this is an improvement over the state-of-the-art. Maybe reporting some memory use values (in practice, are we far from using the allocated 100GB ?) could be worth it ?\n\n\nMinor:\n- (typo) Line 13 of Algorithm 2: what does \"if sol ′.value ← next then\" mean ? This looks like an assignment (with operator ←) rather than a comparison (as described in the main text on line 197). Also, \"next\" is undefined in this context.\n- (typo) Line 923: \"and a the fairness metric [...]\"\n- In Appendix B1, it is mentioned that 4 datasets were excluded because they \"exceeded memory limits while solving\". Could you clarify this ?(i.e., was the memory limit reached with the proposed SORTD, TreeFarms, or both ?)\n- (typo) Line 1073: \"[...], which are then evaluate [...]\"\n- (typo) Line 1109: \"[...] objective),\" (additional closing parenthesis)\n\nQUESTIONS:\nQ1/ \"Our analysis indicates that this scalability stems from a compensating effect in SORTD’s search: although more features increase the number of branching nodes per search node, they also shorten the sorted solution lists each search node maintains.\" (line 243) -> This sentence is a bit unclear to me. Could you please clarify how increasing the number of features (so basically the number of split possibilities) can actually shorten the solution lists ?\n\nQ2/ SORTD uses specialized subroutine for trees containing 1, 2 or 3 branching nodes, that are said to \"reduce runtime and improve\nscalability with both the number of features and the depth budget\". Did you perform ablation experiments to precisely quantify the speedup provided by using (or not) these subroutines ?\n\nQ3/ The fact that SORTD generates sparse decision trees within the Rashomon set in ascending order of their objective function is an important advantage as it provides nice anytime behaviour. Nevertheless, technically speaking, even though they are not generated, suboptimal solutions still need to be explored (e.g., in Figure 2, how to be sure that other splits are not better without verifying them ?). I guess this is done implicitly when calling STreeD to generate the optimal solution values of the left and right child search nodes of a given branching node ? Could you conceptually elaborate on that ?\n\nQ4/ What would be the main challenges towards extending the proposed approach to generate Rashomon sets of other types of interpretable models ? (e.g., rule lists, whose Rashomon set was already explored and for which efficient branch-and-bound algorithms were proposed such as CORELS)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend the STreeD decision tree optimization framework to support objective-value-ordered Rashomon set enumeration. The approach yields more than an order of magnitude improvement in runtime relative to state-of-the-art TreeFARMS, and allows extension of Rashomon sets to additive objectives with total orderings (significantly, giving us the first full Rashomon set enumeration algorithm for regression trees). The appendix details an additional benefit: the approach can compute other separable, partially ordered objectives while enumerating the Rashomon set, saving some computation time relative to computing those objectives afterwards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n-\tA flexible framework for Rashomon sets across many additive objectives, including regression\n-\tSubstantial runtime improvements\n-\tGiving the Rashomon set in sorted order, right away, makes finding the Rashomon set across a range of choices of epsilon more convenient. \n-\tI think the way of enumerating the set in objective order is quite clever, and of interest in general. \n\nWeaknesses: \n\n-\tThe prior state of the art allows random sampling from the Rashomon set, which I believe this approach does not\n-\tWhile this work is correct that TreeFARMS does not preserve objective ordering, my understanding is that this work is incorrect in stating there is no ordering at all to TreeFARMS trees. TreeFARMS trees are structured in relation to the splits used, allowing users to traverse the Rashomon set by split structures as in TimberTrek [1], rather than by objective ordering (when all trees in the Rashomon set are already near-equivalent in objective value). \n\nMinor Weaknesses: \n\n-\tWhile some applications clearly benefit from sorting the Rashomon set, not all do: the point of the Rashomon set for reasonable epsilon is that all models within it differ only trivially, so their relative ordering is not deeply significant. Giving a total ordering can imply meaningful differences among models that are, in practice, interchangeable.\n-\tWith the prior state-of-the-art, it was not particularly complicated to get a top-k set of trees, or the sorted ordering, from TreeFARMS when the full set is of the size this  paper defines as reasonable (10^6 or less). The sorting can be done as post-processing in reasonable time.\n-\tThe top 100 model approach for variable importance (section 5.2) shows some similarity to the conclusions with the top 10,000, but with limitations: a high jaccard similarity for 20 out of 29 datasets is better than the result for a single model, but still only just above 2/3 of datasets. (As a minor note - the top 100 threshold approach used here seems like it forces breaking ties arbitrarily, relative to just using a Rashomon set with a small epsilon) \n\n[1] Wang, Zijie J., et al. \"Timbertrek: Exploring and curating sparse decision trees with interactive visualization.\" 2022 IEEE Visualization and Visual Analytics (VIS). IEEE, 2022.\n\nQUESTIONS:\n- Could you please expand a bit on the benefit of the depth 2 solver? My understanding from MurTree is that asymptotically this solver is quite similar to a standard brute force search unless the binarization system is particularly sparse. (i.e. if each feature has ½ of its values true and the other half false, the runtime improvement is just a factor of 4). Since this work uses a sparsity term, lambda, there’s some limit on how sparse each reasonable feature is (since there’s no reason to consider features that have fewer than 2/lambda values true for binary classification). Clearly there’s a benefit to this approach relative to TreeFARMs, but it seems like it’s not an asymptotic improvement (and indeed, it’s sort of hard for me to tell from figure 3 that the scaling with depth is better than a constant factor improved, albeit an impressive constant factor).\n\n\n- How are you measuring partial progress in treefarms (i.e. how quickly each individual tree is generated), particularly when it times out? I’ve always run treefarms to completion before analyzing the set, and I’m not sure how to think about partial output from the algorithm (i.e. all the results for depth 5). It’s not clear to me from glancing at the experiment code for the treefarms comparison how intermediate results were collected, so a response here would be appreciated as it would save me some digging.\n\nLIMITATIONS:\nI'd be interested in seeing the results in 5.2 relative to a true Rashomon set, rather than the set of top 10,000 models. This is at most a minor limitation, though.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a scalable algorithm “SORTD” for enumerating the Rashomon set of sparse decision trees. (The Rashomon set is the set of models that are near-optimal, e.g., whose accuracy is within epsilon of the optimal model accuracy.) The authors build on TreeFarms, prior work that uses dynamic programming and specialized data structures to enumerate the Rashomon set of sparse trees. The distinguishing features of SORTD are that it is more scalable and it finds trees in best-to-worst order. Since trees are enumerated best-to-worst, there is “anytime” functionality (i.e., computations that only require part of the Rashomon set can be completed more quickly) and it is not necessary to know the Rashomon parameter epsilon a-priori.\n\nSTRENGTHS AND WEAKNESSES:\n###Strengths\n* Potential for high impact, because the code is available as a python package and this paper improves on the scalability bottleneck of existing methods for Rashomon set enumeration. \n* The experiments are comprehensive \n* As described in section 5.3 (regression trees and fairness metrics), the  approach can be used for real-world important tasks \n\n\n###Weaknesses\n* Inherent limitations to the approach include that it is limited to sparse decisions trees and it has high memory requirements. I think the high memory requirements is of particular concern because it would limit how widely this approach can be used. \n* Minor: is there a typo in Fig 2? (Swap lindex and rindex in “next” and “candidates queue”)\n\nQUESTIONS:\n- The number of features correlates with runtime (Fig. 4) but looking at Table 2 in the appendix, it doesn’t totally explain the variance between datasets (nor does dataset size). Are there other dataset properties, e.g., separability or maximal accuracy, that can help explain runtime variations?\n- Section 5.2 (Variable Importance Analysis) for top-5 feature stability using Jaccard and Kendall’s T, how much worse are these metrics if we use a random selection of 100 or 10000 Rashomon set members (e.g., terminating TreeFarms once the specified number of models have been sampled)? I.e., how beneficial is having the most-performant Rashomon set members for variable importance analysis on a subset of the Rashomon set?\n- Any workarounds for memory constraints? (Ok if this is for future work, but what general directions are promising?)\n\nLIMITATIONS:\nMostly — see comments about scaling and memory concerns", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRashomon sets of sparse decision trees are useful for downstream data analysis, such as variable importance analysis. However, getting the entire Rashomon set might be too time-consuming. To circumvent this difficulty, the authors propose to only get the top-$n$ sparse trees, using a best-first search approach so that the algorithm can exit early. To further improve the computational efficiency, the authors apply the depth-two subroutine, which has already been explored in previous works. Experimental results show that the proposed method, SORTeD, can run significantly faster than the SOTA TreeFARMS method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The writing for the introduction paragraph is pretty strong. The language is both concise and comprehensive.\n2. Clear run-time improvement over the baseline TreeFARMS.\n3. The main contribution is an engineering effort in C++ programming to make the whole program work for best-first search on the task of collecting the Rashomon set. Conceptually, the contribution is fair (See my #5 below for the weaknesses)\n\n**Weakness**\n\n3. The writing for the methodology section (Section 4) is a little bit too complicated in my opinion. I spent several hours without understanding in details of what is going on. The central idea is simple. The first phase is popping. We keep popping out the best solution in the queue. If we cannot certify the solutions in the queue are the best, we should switch to the second phase --- exploring (to certify optimality in the remaining search space). We alternate between the popping phase and exploration phase until we reach the Rashomon set limit. I suggest the authors give this big picture at the beginning of Section 4.\n\n4. To continue on this thread, I suggest the authors highlight/boldface this sentence \"If the child search node does not have that solution yet, it is obtained by calling GetNextSolution (Alg. 1).\" I believe this is the key step for understanding the big picture. We need to alternate between popping and further exploration.\n\n5. Conceptually, there aren't any big surprising innovations. Best-first search is one of the standard searching methods. Depth-two subroutine has already been proposed in previous works.\n\n6. This is a minor comment. In some sense, the substantial time-improvement isn't that too surprising. Firstly, the goals of SORTeD and TreeFARMS are different. SORTeD only care about the top-$n$ trees in the Rashomon set, while TreeFARMS want to collect ALL the trees in the Rashomon set. Secondly, SORTeD use the depth-two subroutine, which I believe play a big part in the time improvement. It is worthwhile to conduct some ablation study to see how much an impact this depth-two subroutine play by turning it off and just having the best-first search framework.\n\nQUESTIONS:\n1. Can you confirm whether my understanding (alternating between popping and exploring) is correct?\n2. Can you perform an ablation study by only using best-first search but NOT using depth-two routine? How does this compare with SORTeD in terms of running time?\n3. The time limit (300) is too small in my opinion. The idea of collecting the Rashomon set is to spend some time collecting the entire set. We only do this once and save all the trees. Afterwards, we can explore the entire space instantaneously by making queries in this Rashomon set space. In this sense, it is fine if practitioners can spend some time waiting for collection process to finish. Could you increase the running time (300-> 3600) and do the time comparison again? I am interested in seeing whether TreeFAMRS can at least do a descent job (by finish collecting all the trees in the Rashomon set or the top-$n$ trees) if practitioners are willing to wait.\n4. For variable importance analysis, it's unclear to me whether using the top-$n$ trees and all trees in the Rashomon set provide the same results. Can you elaborate on this? Can top-$n$ trees provide the same quantitative and qualitative variable importance analysis? Under what circumstances? Can we get any theories in this direction? If these two approaches give different variable importance analysis results, then it's unclear what benefit we get by obtaining only the top-$n$ trees. This is actually an important question. If getting the top-$n$ trees is not enough, then it undermines the motivation of this work.\n5. Unlike Figure 3, Figure 4 only contain solid lines. What happens when $n=10^4$ and $n=10^6$ for Figure 4?\n\nLIMITATIONS:\nThere isn't a specific paragraph discussing the limitations in the main paper. However, such a discussion is included in one of the checklist answers.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}