Title: Neural Rule Lists: Learning Discretizations, Rules, and Order in One Go
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

rule lists

differentiable relaxation
Abstract: Interpretable machine learning is essential in high-stakes domains like healthcare. Rule lists are a
popular choice due to their transparency and accuracy, but learning them effectively remains a challenge.
Existing methods require feature pre-discretization, constrain rule complexity or ordering, or struggle
to scale. We present NeuRules, a novel end-to-end framework that overcomes these limitations. At its
core, NeuRules transforms the inherently combinatorial task of rule list learning into a differentiable
optimization problem, enabling gradient-based learning. It simultaneously discovers feature conditions,
assembles them into conjunctive rules, and determines their order—without pre-processing or manual
constraints. A key contribution here is a gradient shaping technique that steers learning toward sparse
rules with strong predictive performance.  To produce ordered lists, we introduce a differentiable relaxation
that, through simulated annealing, converges to a strict rule list. Extensive experiments show
that NeuRules consistently outperforms combinatorial and neural baselines on binary as well as
multi-class classification tasks across a wide range of datasets.


Peer reviews:
comments: SUMMARY:
The paper proposes NEURULES, an end-to-end differentiable approach for learning rule lists, jointly handling discretization, rule composition, and ordering. Experiments demonstrate consistent improvements over existing methods across various datasets.

STRENGTHS AND WEAKNESSES:
Strengths:

	1. Technically solid contributions with clear and effective innovations (differentiable predicates, relaxed conjunction, and learned rule ordering).

	2. The writing is clear and well-organized, providing intuitive motivation and explanation of model architecture.

	3. Extensive experiments on diverse real-world datasets and multiple baselines demonstrate the effectiveness convincingly.

Weaknesses:

	1. Limited experiments on datasets with high feature dimensions, thus the scalability to high-dimensional scenarios needs further validation.

	2. The method requires pre-specifying the number of rules, which may cause inconvenience in practical applications.

QUESTIONS:
Although the overall framework is elegant and reasonable, based on my experience with differentiable rule learning methods, I still have concerns regarding the scalability of the proposed approach to high-dimensional datasets (e.g., feature dimension >100 or even >1000). The current experiments include only one dataset with dimension above 100. In scenarios of extremely high-dimensional data, classical feature preprocessing methods might show relative advantages. I’d appreciate it if the authors could provide additional experimental results or discussions on datasets with higher dimensionality to further validate scalability.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a differentiable relaxation of the problem of learning rule lists on tabular data. It jointly learns predicates (of the form $a < x < b$), conjunctions of those predicates to form rules, and the order of those conjunctions. The number of rules is held fixed. The differentiable relaxation of predicates yields soft predicates, which converge in the temperature limit to strict predicates -- this is exploited in the optimized algorithm by temperature annealing as training progresses. To learn conjunctions of rules, the authors use a generalization of the weighted harmonic mean to include a slack term, which simultaneously eliminates vanishing gradients and encourages sparse rules. Finally, to learn rule list order, the authors propose to learn a priority vector for the position of each rule in the list. The final form of a rule list is a sum over all classes of an indicator over the highest priority rule; however, this indicator is relaxed into a Gumbel-softmax approximation of the argmax, which allows gradients to flow beyond the single selected rule. This paper uses another form of temperature annealing to collapse this approximation later in training. The authors also propose a regularizer based on the minimum and maximum coverage of rules. Experiments comparing this method to other neural methods, as well as combinatorial discrete optimization methods, are presented.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The differentiable relaxations presented for each of the three layers of rule list learning are principled, justified by both intuition and thorough arguments for convergence and gradient stability, and clearly advance prior work on neural rule lists.\n\n2. The paper is written in a clear and concise manner. The issues with existing approaches, along with the authors' proposed solutions, are well motivated and clearly reasoned.\n\n3. The experiments effectively demonstrate that this approach is better on many datasets compared to previous neural methods.\n\n4. The easy extension to multi-class classification problems is a nice benefit of the proposed architecture.\n\n## Weaknesses\n\n### Major Weaknesses\n\n1. The comparisons to combinatorial methods, especially CORELs, are weak and do not present a fair comparison. The choice to only use 5 evenly spaced cut points on continuous features undercuts CORELs' ability to fit the data well -- in fact, due to the use of objective-based bounds, the use of fewer cut points may contribute to the surprising number of timeouts of CORELs in the experiments (even on small datasets). CORELs provides a feature selection method in both the paper and the code based on frequent itemset mining. In fact, existing prior work shows that, at least on the FICO dataset, CORELs outperforms the best metric presented in Table 1 (He et al., 2023; JAIR: Visualizing the Implicit Model Selection Tradeoff); FICO is listed as timing out in this paper.\n\n2. Similarly, the choice to perform hyperparameter optimization only on several held out datasets and not on each dataset individually is a weak-point of the evaluation in this paper. Especially for the discrete methods, the regularization is directly compared against misclassification error, and is sensitive to the class balance and particulars of each dataset. At least the regularization parameter should have been tuned for each dataset individually, ideally with nested cross validation (see Scikit-learn Nested vs. non-nested cross validation).\n\n### Minor Weaknesses\n\n1. There is little attention given to the actual complexity of the rule lists identified by this method. The authors demonstrate that this approach can find good rule lists which don't have many rules, but it is not clear how many conjunctions these rules have. A rough metric to compare to sparse methods would be the total number of predicates used in the rule list, rather than the length of the list.\n\n2. The maximum cardinality of 5 for CORELs is too high - a typical value for this is 2 or 3. Allowing length-5 predicates is probably another contributor to the timeouts observed in the results tables. Also, the max cardinality of 5 presented in this paper is not one of the options in the grid search (2, 3, 4) - I think this may be a typo on line 534.\n\n## Conclusion\n\nThis paper is technically solid, with reasonable and well-justified methods to make the rule list learning problem differentiable. However, the evaluations, for all methods in terms of hyperparameter optimization, and for CORELs in terms of continuous feature binning and hyperparameter choices, were incomplete and did not present a fair comparison.\n\nQUESTIONS:\n1. It appears that the rule ordering optimization method introduces discontinuities into the loss landscape of your neural architecture. When one priority value overtakes another, this results in an immediate swap of the rules and a corresponding change in the loss (say $t_{rl} \\approx 0$). I have a similar concern about the process to learn rules -- it seems that there are discontinuities introduced whenever a predicate achieves non-zero weight. My question is: do these discontinuities matter for your optimization process? Are they solved by the relaxations to achieve differentiability?\n\n2. How do the performance and sparsity (# of rules, and # of predicates per rule) of your method compare to CORELs when CORELs is given the same predicates to use as your method has learned? That is; how do the rule and order learning stages of your method compare to an optimal approach, conditioned on pre-selected predicates? If it performs comparably in both performance and sparsity across a variety of datasets, I will consider raising my score. \n\n3. I have several related questions relating to the implicit regularization in the rule construction process. How strong is the implicit regularization towards sparser rules in section 3.2? Is the strength of regularization tunable, for example by adjusting the slack magnitude? Does the regularizer encourage rules to be sparser, even when some terms have useful signal, or does it only eliminate useless terms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe rule learning method is essential for interpretable AI and highly trustworthy application domains. Current rule-learning methods need pre-discretized features, and some of the rule-learning techniques are limited in scalability. The paper proposed a novel framework for learning rules from the data without feature pre-discretization. The method mainly used a fully differentiable manner to construct the predicate, rule, and rule list ordered by the importance of rules. The experimental results indicate that the proposed method obtains the state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well-structured and presents its ideas using clear and rigorous mathematical formulations. The authors designed a unified method to learn Boolean conditions denoted as predicates in the paper, conjunctions, and the order of the rule list simultaneously.\n\nWeaknesses:\n\nThe paper omits preliminary explanations of fundamental concepts, such as the notion of a predicate, or misses the proper citations for these notions. Additionally, it lacks important details regarding the experimental setup, including the specific optimization methods employed. Besides, in the related work, the authors also ignore some discussion regarding the rule learning with the column generation algorithm and the first-order rule learning method.\n\nQUESTIONS:\n1. The notion of an atom in first-order logic (FOL) typically includes both terms and predicates. However, the concept of a “predicate” as described in your paper appears to differ slightly from its traditional definition in FOL. Could you please clarify the relationship between the predicate used in your paper and the standard predicate in FOL? \n\n2. Additionally, in the rule learning community, the term “soft predicate” (as described in [1]) carries a different meaning from the one presented in your work. Could you also elaborate on the connection or distinction between the soft predicate defined in your paper and that in [1]?\n\n3. Regarding Equation (4), could you explain how the parameters are chosen? For example, what is the rationale behind the formulation $(2x_j - a_{ij})$ and the exponential term $e^{\\frac{1}{t_{\\pi}(3x_j - a_{ij} - b_{ij})}}$?\n\n4. In the ablation studies, the paper does not appear to provide the specific values used for the temperature parameter $t_{\\pi}$. Does the value of $t_{\\pi}$ have a significant impact on the experimental results?\n\n5. Can you describe the connections, advantages, and disadvantages of the proposed rule learning method with other rule learning methods based on the column generation algorithm, such as [2].\n\n[1] Wang, W. Y., Mazaitis, K., & Cohen, W. W. (2015, July). A Soft Version of Predicate Invention Based on Structured Sparsity. In\nIJCAI (pp. 3918-3924).\n\n[2] Sanjeeb Dash, Oktay Günlük, Dennis Wei: Boolean Decision Rules via Column Generation. NeurIPS 2018: 4660-4670\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method called NeuRules, which is a differentiable method to obtain a rule list, an interpretable and easy to implement kind of decision tree. The author(s)' method takes data comprising of numeric (continuous or encoded as numeric) features, and predict one of the discrete output classes. Rather than applying combinatorial methods with discrete choices, they train such a rule-list using a differentiable relaxation, designing, implementing and running the relaxation. Their particular choices in this relaxation lead to rule lists that promote sparsity, and choose the orders of the rules (with a temperature annealing that tightens during the training). They implement and run their method on a variety of tabular datasets, they compare NeuRules with other existing methods using mainly F1 scores. Although Neurules doesn't score better on every dataset with this metric, on average it scores the best.\n\nSTRENGTHS AND WEAKNESSES:\nThe combination of various differentiable relaxations are able  to work in harmony to produce rule-lists which are a very discrete  object and able to order them organically. Even though rule lists are not as prevalent as, say, decision trees they are still important in areas which require high interpretability, so this work is significant in that it allows for flexible creation of rule lists that makes the modern machine learning practitioner feel at home with the use of gradients in its learning process. Thus I can also imagine this being integrated into other machine learning pipelines as a interpretable module (with possibly non-interpretable features).\n\nAs for weaknesses on originality, and significance The differentiable relaxations work (according to the experiments that were done), but there is no reason or heuristic argument that they are the best according to some metric.\n\nQUESTIONS:\nQ. How does one work with categorical features? How one converts such features to  continuous ones should be mentioned, or it should be mentioned in the limitations section the fact that the whole setup requires continuous features.\n\nQ. In lines 284-289 in the Ablation studies for rule ordering, the wording is not clear if Neurules is being compared to RLNet or a version of Neurules without ordering (such as deactivating the priority $p_i$ variables. Which one is it? In the former case we cannot really call it an ablation study, and table 6 seems to indicate that that was not what was done. But in any case the wording in section 5.3 doesn't make it clear exactly what manipulation was done. Could this be made clearer?\n\nQ. Gradient Shaping (GS) is not defined explicitly as a method. Does without GS in figure 4b means to take $\\epsilon = 0$? If yes then one should say so. \n\nQ. What does color represent in figure 4(c)?\n\nQ. things are not clear in equation (10) as to whether the logit coordinates $c_\\ell$ are learnable parameters? Also are the consequents integers $\\ell_i$ (which is what would come out of the argmax, or are they $c_{{\\ell}_i}$ (which is what is written in equation (11) for exmple, but doesn't make sense if the consequent is just a number). I think there is a notational confusion here. What is the notation?\n\nQ(cont). The confusion continues in equation (13), where (for the first time, if I'm not mistaken) $\\hat{c}_i$ is introduced.  What exactly are the differentiable relaxations of these consequents? Even if they are made numeric (and doesn't remain a symbolic \"yes\" or \"no\") by one-hot encoding, then what exactly does their differentiable relaxation mean? Why would we treat them as learnable? I think I understand the overall method, but  the notation here is confusing, if not an outright mistake.\n\nThe items below are not really questions but typos that should be fixed or certain suggestions for better presentation.\n\n- Typo on page 1, line 28 in \"i.e. the set of condtions that specify when a rule triggers.\" condtions--> conditions. Also better to say ``when a rule is triggered''.\n\n- Line 81, page 3, $\\hat{rl} \\to \\widehat{rl}$.\n\n- Table 1 should be in section 5, or better yet in section 5.1 if possible. And Table 1 would do well to include the lengths of the produced rule lists for methods which are rule lists.\n\n-In Appendix A1, the proof's first step is to divide the numerator and the denominator by $e^{\\frac{1}{t}b}$ in equation (18), as to bring us to (21). Which begs the question, then why was the soft predicate (16) given as such, and not directly as (21), where its form, and the purpose of each term would be clearer. If it is a numerical stability issue, then this should be mentioned.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}