Neural Rule Lists: Learning Discretizations, Rules, and Order in One Go
Authors: Sascha Xu
Nils Philipp Walter
Jilles Vreeken
Venue: NeurIPS 2025 poster ()
Keywords: interpretability
rule lists
differentiable relaxation
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a fully differentiable architecture for learning an interpretable rule list classifier.

Abstract: Interpretable machine learning is essential in high-stakes domains like healthcare. Rule lists are a
popular choice due to their transparency and accuracy, but learning them effectively remains a challenge.
Existing methods require feature pre-discretization, constrain rule complexity or ordering, or struggle
to scale. We present NeuRules, a novel end-to-end framework that overcomes these limitations. At its
core, NeuRules transforms the inherently combinatorial task of rule list learning into a differentiable
optimization problem, enabling gradient-based learning. It simultaneously discovers feature conditions,
assembles them into conjunctive rules, and determines their order—without pre-processing or manual
constraints. A key contribution here is a gradient shaping technique that steers learning toward sparse
rules with strong predictive performance.  To produce ordered lists, we introduce a differentiable relaxation
that, through simulated annealing, converges to a strict rule list. Extensive experiments show
that NeuRules consistently outperforms combinatorial and neural baselines on binary as well as
multi-class classification tasks across a wide range of datasets.

Neural Rule Lists: Learning Discretizations, Rules, and Order in One Go
Sascha Xu; Nils Philipp Walter' and Jilles Vreeken
CISPA Helmholtz Center for Information Security (sascha.xu, nils. walter, vreeken)@cispa.de
Abstract
Interpretable machine learning is essential in high-stakes domains like health- care. Rule lists are popular choice due to their transparency and accuracy, but learning them effectively remains challenge. Existing methods require feature pre-disreeizzttonn constrain rule complexity or ordering, or struggle to scale. We present NEURULES, novel end-to-end framework that overcomes these limita- tions. At its core, NEURULES transforms the inherently combinatorial task of rule list learning into differentiable optimization problem, enabling gradient-based learning. It simultaneously discovers feature conditions, assembles them into conjunctive rules, and determines their order-without pre-processing or manual constraints. A key contribution here is a gradient shaping technique that steers learning toward sparse rules with strong predictive performance. To produce ordered lists, we introduce differentiable relaxation that, through simulated an- nealing, converges a strict rule list. Extensive experiments show that NEURULES consistently outperforms combinatorial and neural baselines on binary well as multi-class classification tasks across wide range of datasets.
Introduction
Rule lists [Cohen 1995 are conceptually simple classifiers that are inherently interpretable and surprisingly effective [Rudin| 2019). As the name implies, rule lists are ordered lists of rules of the kind "if Thalassemia normal and Resting BP < 151 then predict yes". Each rule consists of a set conditions, when met, trigger a specific prediction. The model evaluates the rules order and applies the first one whose conditions are satisfied, yielding simple and interpretable decision path. We give an example rule list in Fig. [ Because of their transparency requirements, rule lists often used in high-stakes applications, c.g. stroke prediction [Letham et al., 2015). credit risk evaluation [hatore et al. 2020]. and criminal justice [Lakkaraju and Rudin 2017].
Inferring rule lists is a challenging combinatorial problem in which we face super-exponential search space with little exploitable structure. First, there the problem of which atomic rule conditions, or predicates, to use. Existing strategies rely on rre-dssretization, which incurs a loss of information. Second, there is the problem of combining these predicates into rule-heads, i.e. set of conditions that specify when a rule triggers. Existing approaches are either combinatorial, and have to restrict search space favor runtime [Cohen] 1995] Yang et al. 2017 Proenca and van Leeuwen, 2020 Angelino al. 2018), use differentiable relaxations that often lead to verbose rule-heads [Wang ct al. 2021 Qiao et al., 2021] Dierckx et al. 2023). Third, there is the order of the list. Except for Angelino al. 2018 existing methods learn rules greedily or consider fixed list order [Dierekx
et al. 2023 either of which can result in subpar accuracy.
'Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

if not (Chest pain) and not (Exercise chest pain) and (0.88 < ST depression < 5.24) then predict yes
else if not (Chest pain) and (45.5 Age 66. 4) and (Sex female)
then predict yes
else if (Resting BP < 151) and (ST depression < 2.65) and (Thal normal)
then predict
else predict yes
Figure 1: First three rules of the rule list learned with NEURULES on the Heart Disease dataset. NEURULES jointly optimizes thresholds, rule aggregation, and list order.
In this paper, propose novel approach that jointly addresses all of the above challenges. We introduce NEURULES, a differentiable end-to-end architecture that learns predicates, conjunctive rules, and the order of the rule list in single unified framework- witthut manual pre-processing structural constraints. Unlike prior work, we relax the discretization of features, the construction of rules, and the ordering of the rule list, allowing all components to be learned jointly. To overcome the limitations of existing methods, NEURULES builds on four key innovations:
Soft predicate learning. Instead of relying on fixed discretization, we learn soft predicates that converge to crisp thresholds via simulated annealing.
(ii) Differentiable rule composition. We use conjunction layer that supports robust composition of predicates while mitigating vanishing gradients.
(iii) Gradient shaping for sparsity. targeted mechanism promotes the learning of rule heads that are both sparse and accurate.
(iv) Learned rule ordering. We model rule order as a soft priority, which is annealed into strict ordering by the end of training.
Together, these elements yield crisp and succinct rule list upon convergence. NEURULES thus provides a holistic differentiable analog of rule lists that scales well and natively supports binary and multi-class classification. Extensive experiments show that NEURULES consistently outperforms combinatorial and neuro- neuro-symbolic baselines developed over the past 30 years.
Preliminaries
We consider a dataset of pairs [(x, y) consisting a descriptive feature vector x E RRd of real-valued features and discrete-valued target label y V. We assume each sample (x, y) to be realization of a pair of random variables (X, Y) ~ Px,Y, drawn iid.
Rules. We consider predictive rules X' + that consist of rule head h: X' - {0, 1) and
consequent e E y. A rule head consists of (a logical conjunction of) binary predicates क : R - {0, }} that represent the presence of characteristic, e.g. "18 Age < 65". We consider predicates threshold functions on feature x E IR parameterized by lower and upper bound, a, E R, as
be
(z; a, b) [a < x <],,
where a = - 00 resp. b= +00 are interpreted as no condition. Our formulation also works on binary features, that we accommodate both categorical features through one-hot encoding and learned predicates with more complex structure [Wang et al. 2015). We allow every rule its own predicates. That is, every rule r; has its own thresholds auj bij per feature Ij. We the shorthand Tij to denote (ay; aij, (ajy,ibb))) biz). We write मं to denote all predicates for rule rı, i.e. म. { )
We say a rule head holds iff (x; 0.) A, Ti (2y; aij biz) parameters of the predicates. A rule maps input x to label if the rule head holds, otherwise, its output undefined (c). Formally,
where 0, {ary biy) are the
if h( else
r r(x;,,0) -)

---

Input
Learned Predicates
Learned Rules
Learned Order aig mux bx(x;0:) .pi
Differentiable Rule-List
x x Rd
B4 xy by
Amay Rt
xl(x) = Σ (x)))
.F3
[(21; 1:::::1::0000 b11)
Wi
(x;x;;
P1
Ox
arg sort(p)
if (hmm(z)) then Ce(3)
( man, 6as)
W2
ha(x;
P2
6,
C2
else hoen then Ca(2)
else cdefauh
R(11:00 bas)
We
ha (xc 06)
Pk
p)
Figure NEURULES architecture: We learn binary predicates (2y, (ij. per feature and rule i. We combine these into rules (x; 0,), turning off unnecessary predicates through sparse weights Wi. We predict the label of a sample as o using the rule that has highest priority pt and is active, 0.) ps. NEURULES end-to-end differentiably optimizable.
L.e. arg max; (x;
[Cohen 1995 are if-then-elss classifiers made of rules. A rule list rl is an ordered set of k rules [he, ci) )=) with parameters è [0-))=1 To predict the label for input x, we traverse the list from top to bot and return the consequent of the first rule h; that holds, with the final rule hg serving as an "else" case ensure every sample can be assigned a label. Formally,
Rule Lists.
C1 if h1 (x: 01)
02 else if h2(x;; 02) =
r1(x; 0)
Cjs
clse
In the following, we will refer to binary-valued predicates and rule heads h as strict, and their differentiable relaxations and the interval [0, 1] as soft.
Differentiable Rule Lists
We now introduce NEURULES, short for Neural Rule Lists. We show its architecture in Fig. 2 The first module learns binary predicates (zz; dij, bıy) for each feature 2; and rule i. This gives NEURULES the freedom to discretize the same feature differently for each rule according to needs, e.g. "18 < Age < 30" for one and "16 < Age < 24" for another.
The second module combines these into sparse rules (x(x; 0,). These are differentiable functions that, for all practical purposes, behave like a logical conjunction. By learning a weight vector w; per rule, we turn off unnecessary predicates which functions as implicit feature selection.
The third module orders these into a rule list rl(x; O, p). We learn priority Pi per rule to govern the order in the rule list. We use the Gumbeel-Sotmax [Jang et al. 2017] approximate the striet indicator (x) that iff rule is the highest priority rule that is active, i.e. hx; 0.) Pi is maximal.
Next, we describe each of these modules in detail. Together, they form a differentiable relaxation that allows us jointly optimize rule list from predicates, rules, up to rule list order. We show that all our approximations converge towards strict logical operators at the end of training, i.c. NEURULES converges a strict rule list, and that through gradient shaping we achieve sparse rules.
3.1 Learning Predicates
We begin with how to learn the predicates that act the building blocks of the "if..." condition of rule. In particular, we consider thresholding functions (αx; a, b) = [a < x < b) on individual features Zj. To avoid the need for pre-discretization, seek to differentiably optimize per rule

---

0.1
0.05
0.025
0
|b 0) -((x; 0)]
0.2
0.1
0.6
0.5
0.4
0
0.2 0.4 0.6 0.8
0.5
0.2
0.15
0.1
0.05
Predicate temperature fa
(a) Soft predicate # ((t=((x) a decreasing temperature 1.
(x) under
(b) Differentiable logical conjunc- tion ix(;; 0) over two features.
(c) Softness rule (xx; for dif- ferent t. (variance in gray).
The soft predicate approaches the true thresholding with decreasing temperature t, + (a). Multiple soft predicates combined into conjunctive rule (b). Decreasing the temperature i results in increasingly strict, binary rules where hxx;0) z 1(x;; 0) (c).
Figure 3:
i respectively feature j the thresholds as and bij as part of the model. To this end, we require a differentiable equivalent of the thresholding funetion, as it is non-continuous at the bounds and hence unsuitable for gradient based optimization. In its place, we use an approximation that is differentiable, but yields soft predicates म R + [0, 1] Yang et al. 2018). For single feature 2j, have
* ( aiy. bys (x)
1 1 gie (au (a)-))) (1))
where the temperature i. controls the smoothness of the approximation. In Fig. 3a show म using different temperatures 4.- As we can see, in the limit f - the soft predicate converges to the true
thresholding function iT (zy; aij bij) (see Appx. A.I. i..
lim # (zy; aty, bij, (x) = π (zy; Dzj, bij)
(5)
t, +0
We start training with a high temperature 4, resulting in smoother predicates and avoids explod- ing/vanishing gradients with respect az and by.- To obtain strict predicates the end of training, we use temperature annealing to continuously decrease the temperature f, as the training progresses. In the limit, this leaves us strict predicates where Vi,j Aij E {0, 13, where write #e for a (Iz; Gij, bij, tx) whenever clear from context.
In sum, by using soft, differentiable predicates á with temperature annealing instead of static pre- processing, we are able to learn the rule and seature- predicates through backpropagation.
3.2 Learning Logical Conjunctions
Effectively, need the soft hú to behave like a logical conjunction that is differentiable, while the same time rewarding sparsity. Formally, we require
h : X + [0, Vj E [d] म. = h, 0.) =1, 3j E [d] म.j = 0 h; (x; 0,)
(6)
This means the differentiable conjunction evaluates to if all predicates are active, and to
at least one predicate is inactive. At first glance, the weighted harmonic mean h₂ 0,) =
Ed E myy E. "
Wij E R introduced by [Xu et al. 2024 is promising candidate as it fulfills the outlined criteria. If any predicate म. = 0, then the reciprocal お.J - oo and the rule evaluates to all predicates are active, i.e. Vj [d] R. 1, then fx(x;; 0,) = The weights Wiy E RJ control predicate inclusion, where setting Wij = 0 disables the predicate ăty. However, this formulation leads to vanishing gradients when predicates become inactive. Specifically, when any predicate म. + with Wis > 0, the gradients with respect to predicates and their weights approach zero, hindering learning. Thus predicates evaluating to cannot be deactivated, leading verbose rules and poor performance, the inactive predicate zeros out the entire conjunction.

---

Relaxed Conjunctions. To overcome this, we propose an elegant trick that resolves vanishing gradients while simultaneously acting as a gradient shaping mechanism that promotes sparse rules. To do so, we introduce a dynamic slack 7/s per rule.
2 Wij
(x; 0.)
T/
2j=1
1+th Wij 4%
2==1 Waj
d
where determines the slack magnitude. With the above, the conjunction no longer collapses to zero when predicate is inactive but assumes value in (x; 0.) [0, CJ. Thus, gradient gate remains open. The value C² depends on and the associated weight. In the worst case, it is bounded by
(,(x; 02) S
(8)
Wil +
For example, with 10-3 and wa 0.099, then C S 0.01. More generally, for desired level of slack C, we require way = 0 or wa > e/C. We provide a formal derivation in Appx. A.3
The slack formulation also guarantees stable gradients. The term W/y(1 + m// (mty + n.) is always bounded and non-zero and as result, the gradient 0 remains well-defined. More importantl, it induces a gradient shaping effect that promotes sparse rules. If ft = 0, the gradient is strictly negative, and the corresponding weight reduced. If Rt 1, the gradient is strictly positive, increasing the weight. For intermediate values of #t between 0 and 1, the gradient balances both effects: weakly activated predicates are discouraged while more strongly activated ones are reinforced. We give detailed derivation in Appx. B.2
This relaxation not only enables reliable gradient flow but also introduces a principled, self- regularizing mechanism for differentiable rule learning. By implicitly pruning unused predicates and amplifying relevant ones, it yields shorter rules than other neural approaches without the need for any explicit regularization term. We evaluate this in Section 5.3 where we find that the relaxed conjunction leads to consistent improvements in performance.
Differentiable Rules. With the predicates and the logical conjunction in place, the entire rule head
lim Oxxx;0,) - (o- 40
if waj 0 n.j =1
else
can be differentiably learned, where 0+ denotes a value slightly bigger than 0 due to the slack. However, this does not matter in practice since the activations of the rules only depend on h(x) which is exactly met. As the rule consequent, which is the "then... part of a rule, we use a logit vector e IR m in the classification setting with m classes. A single rule is then learned as
if (x: 0,) = 1 then arg max C2 (e(......m)
(10)
Next, we describe how the model learns the order of individual rules.
3.3 Learning Rule List Order
To complete NECRULES, we learn the order in which the rules should be applied, rather than assuming a fixed sequence or relying on greedy construction. We achieve this by introducing a learnable priority vector p RR where each pj defines the position of rule j in the list. A sample x is classified using the prediction of the first applicable rule with the highest priority, i.c.
rl x: O. C such that (x(x;,,)
A (Vj A i h3 (x) 0 0V pj)
(11)
To cast this combinatorial problem into a differentiable form, we rewrite the rule list as a linear combination of the rule consequents c, using the product of h; (x; 0t) Pt to determine which e; to select. Under mild conditions, this formulation is equivalent to the original rule list (see Appx. A.4) A.4)
Proposition 1 Given binary rule heads he( (x:x: 02) 1) and unique priorities ps E Ro, L.e. Vi i : Pi # Pj. the rule list rl(x; θ p) from Eq. 17 is equivalent to weighted sum of
&
rl(x; θ
Σ a
= arg max(h,,(x 0,) pp)
2
1=1
5

---

Differentiable Relaxation. While arg max can be used with gradient routing, it produces zero gradients for all but the selected rule, hindering learning. Moreover, relying solely on the current top rule can cause the model to converge prematurely to suboptimal solutions. To address both issues, we relax the indicator function using the Gumbel-SSoftax as
hx(;; -
[x t.)) = softmax;
g Gumbel 1)*
(12)
Ld
The Gumbel-Softmax yields differentiable approximation to the arg max and converges to it with high probability as tu + O. We anneal t, during training to gradually sharpen rule selection. This relaxation allows gradients to flow to multiple rules early in training, improving stability and enabling exploration of different rule configurations. Even rules with low current priority continue to receive updates and can become active later. Altogether, NEURULES models differentiable rule list as
fx(: O, p,
Σ [xx b.r) C,,
(13)
At convergence, we convert the model into strict rule list by fixing the learned discretization parameters azy and by. and construct each rule (x: 0.) as conjunction over all predicates with wi We then order the rules by their learned priorities pa, and compute predictions using = softmax((;) over learned class logits, yielding the final discrete rule list r((x; p).
3.4 Overall Objective
We train NEURULES in supervised learning setting, optimizing parameters based labeled data samples ((x, )) from the data distribution Рx,y. We minimize a classification loss {() y). for which we choose the cross-entropy loss. Formally, minimize the classification loss / and regularization term with respect to all rule list parameters θ = (a1, bi,.... ........, Og. dg, Wi,..., Wx) and p, where f,1 and , are controlled by annealing schedule. The overall objective is given by
arg min: O.P
[(il(x O,p, p. f.), v) + OR(OOp,,t.)
(14)
n
To avoid overfitting and encourage meaningful rules, we add a coverage-based regularizer akin to the minimum support criterion in decision trees. To this end, we define the soft coverage of each rule as the average activation over the training set, using the soft rule selection indicator [x(x;; θ p, t,:). Formally, we define as covy E 0,
We then construct a regularizer R(O, p. fn) out of a user-specified minimum and maximum coverage CoUmin and coUmax that discourages rules that only rarely frequently active as per
1 T
(.O.p.))
max (0, co -oou)²² max (0, cov, coUmax)
(15)
where use the max(.) operator ensure that rules whose coverage is within the interval [coumin, mov incur no penalty. The regularizer activates only when coverage moves outside this desired range, applying quadratic penalty that encourages rule coverage to return to the targeted interval. This ensures minimal interference while effectively promoting the desired behavior.
Related Work
Rule lists were introduced by Cohen [1995] and often used in practice, e.g. in healthcare [Deo 2015 criminal justice [Angelino et al. 20188] and credit risk evaluation [Bhatore et al., 2020 based on their competitive performance and inherent interpretability [Rudin 2019]. Rule lists are closely related to decision rule learners for unordered DNF rule sets (Dash et al.; 2018). The main difference the inclusion of order that makes rule lists more compact and thus easier grasp.
Combinatorial Optimization. Rule lists are typically optimized using heuristic, combinatorial search [Cohen 1995]. Wang al. 2017 propose Bayesian rule lists-probabiliste models whose

---

complexity is controlled by user-defined priors, These priors typically yield compact rule lists but may degrade performance if poorly chosen. Proenca and van Leeuwen (2020) and Papagianni and
van Leeuwen [2033 propose to use MDL to address the trade-off between complexity and accuracy.
Other approaches use branch-and-bounde or II.P to learn optimal rule lists [Angelino et al.] [2018]
Aivodji et al. 2022]. but these exact methods are computationally costly and limited to small datasets. In general, combinatorial methods explicitly limit rule length and rule count to prevent exponential explosion of the search space. Moreover, they rely pre-discretizing continuous features, which is prone to remove important signal from the data.
Differentiable Optimization. Recently, neural methods have been proposed for learning rule-based classifiers. Qiao et al. [202T] introduced the first end-to-end approach. using neural architecture with continuous relaxations of logical operators. Drawing from fuzzy logic. this allows differentiable
optimization of operations like negation, disjunction, and conjunction [van Krieken et al. 2022]. After
training, rules are extracted from the network. [Wang et al. [2021] extend this to deeper architectures capable of learning more complex rules. For binary data, Walter et al. (2024) use a binary autoencoder to optimize both predictive and descriptive strength. Unlike NEURULES, none of these are equipped with a mechanism that removes unnecessary predicates and hence often return overly complex rules.
To learn rule lists, Dierckx et al. 2023) build on Qiao et al. [021] by imposing a fixed rule hierarchy, but their approach requires feature pre-discretization a limitation shared with combinatorial methods. Kusters et al. [2022) propose a differentiable rule learning method based on linear decision boundaries, but these cannot be directly interpreted as threshold-based rules. In contrast, NEURULES jointly learns threshold-basee baasdd predicates per rule and feature, learns sparse rule heads that are strict logical conjunctions after training, and optimizes the order of the rules in the list, all in one go.
NEURULES
RLNET
DRNET
GREEDY
CLASSY
CORELS
SURL
Adult
0.50 1: 0.01
0.76 2 0.01
0.77 0:03
0.78 1: 0.01
0.75 0.0
0,81 0.0
0.80 1: 0.0
01800 1 0.0
0.82 0.0
Andraid Malware
0.93 4: 0.01
0.95 + 0.00
0.92 0:0
0.95 :: 6.0
0.56 0.0
0.94 2 0.
0.86 + o.
0.33 : 0.01
16/a n/a
COMPAS
0.67
0.65 2.42
0.58 1
0.58 1
0.66 2.00
0.67 0.2
0.65 1
0.65
0.67 1
Covid
0.57 0.006
0.63 0:0
0.48 6.00
0.63
0.00
0.64 0.00
Credit Card
0.79 0.00
0.77 = 0.00
0.75 1: 0.05
0.76 :: 0.0L
0.79 =: 0.01
0.79 0.0
0.76 +: 0.05
n/u 22 m/u
0,80 :: 0.0
German Credit
0,72 0.04
0.71 +
0.71 +
0:15 + 0.07
0.67 +
0.67 0.00
0.71
+ 0.01
0.64 0.0.
0.50 +.001
Credit Screening
0.02
0.03
0:43 0.04
1 0.02
0.86
0.72
0.05
Diabetes
0.71 0.00
0.70 = 0.08
0.73 0007
0-44 :: 6.14
0.71 0.0%
0.70 0.44
0.74 0.00
0.71 0.00
0.71 0.04
Drug Response
0.73 1:: 0.04
0.74 = 0.05
0,76 0.01
0.65 1: 0.03
0.67 - 0.04
0.70 1: 0.04
0.69 1: 0.02
n/a 2: náa
1//a m/a
Electricity
0.75 1:
0.69
0.63
0.61 1
0.75
0.59 2
0.75
1
0.73 2
0.78 U.O.
0.70 0.01
0.03
0.56 0.02
0.02
0.70
Ivla
n.0:
Heart Discase
0.80 0.25
0.74 1: 0.02
0.72 I.04
0.51 1: 6.1
0.71 2: 0.06
0.77 0.00
0.80 :: 0,00
0.65 1: 0.06
0.64 0.04
Heparitis
0.81 0.20
0.77 =
0.78 + 0.05
0.18 + 0.04
0.73 =
0.74 +:
0.75 + 0.01
0.74 + 0.1
0.73 0.1
Juvenile
0,89
0.87
0.98 &
0.59 1:
0.63
0.98 0.00
0.03 1:
IVaa ná/a
náa m/a
Magic
0..00
0.77
0.72 0.03
0.73 6.05
0.75
0.74 0.00
0.77
0.00 0.0
0.0
Phishing
0.90
0.93 d: 0.0
0.80 0.00
0.93 =:: 6.01
0.50 =: 0.0
0.92 d: 0.03
0.80 + 0.0
0.27 = 0.01
0.40 0.03
Phoneme
0.80++00.0
0.71 +
0.72 +
0.64 +
0.76 176---00.0
0.79 0.002
0.77 +
0.73 + 0.01
0,81 +
QSAR
0.81 0.000
0.83
0.52 +
0.74
0.00
0.79 +
0.81
Ring
0.92
0.81 0.00
0.83
0094
0.33 1: 0.02
0.56 =2
0.65
0.05
0.74 +: 0.04
n/o =: m/o
0.83 0.01
Titanic
0.75-t-0.0
0.74 - 0.68
0.68
+ 0.05
0.45 + 0.00
0.78 + 0.08
0.77
+ 0.00
0.75 + 0.00
0.66 - 0.00
0.86 + 0.00
Tokyo
0,92
0.91 2
0.91
0.0
0.25 1:
0.85
2 0,01
0.92
0.02
0.92 1:
I/a máa
0.90 2
TuanDrond
0.96:::0.01
0.00
0.99
0.01
0.922 0.0L
0.0.0
n/a vola
Avg. Fl
0.79 = 0.02
0.77 =: 0.02
0.75 0.04
0.56 4: 0.04
0.75
= 0.02
0.76
0.03
0.73 :: 0.0
0.63 4: 0.04 0.53
0.73 0.02
Rank
2.59
4.50
5,18
7.00
5.34
4.14
4.07
4.06
Table 1: Binary Classification. F1 scores for 22 real-world datasets. Timed-our experiments (>24h) indicated by n/a. NEURULES consistently achieves the best or close to the best performance, obtaining the highest overall Fi (0.79) and rank (2.59).
5 Experiments
In this section, we empirically evaluate the performance of NEURULES by comparing it against range of rule-based and ceuro-symbolic baselines. including both classical and state-of-the-a models. These include optimal rule lists (CORELS, AAggelino et al. 2018), scalable Bayesian rule lists (SBRL, Yang et al. 2017), MDL-based rule lists (CLASSY, Proenca and van Leeuwen 2020), RIPPER (RIPPER, Cohen 1995). and greedily constructed rule lists (Greedy). We also consider the of the-art neural rule list model RLNET [Dierckx et al.) 2023 as well as two recent neuro-symbolic rule set methods RRL (Wang et al. 2021 and DRN ET QQao et al. 2021].
As implementations, we use the imodels library (Singh et al. 2021 for CORELS, SBRL, and GREEDY; Weka (Hall et al.] 2009 for RIPPER; and the original implementations by the authors for CLASSY, RLNET, RRL, and DRNET. We tune hyperparameters for all methods using grid search on held-out datasets (see Appx. D. All results are averaged over five-fold vross-validation. All source code and datasets used in our experiments are included in the Supplementary Material.
7

---

NEURULES RLNET RIPPER
NEURULES wro = 0)
CLASSY
GREEDY
RKL
0.8
200
F4 Abiation
NEURULES
CLASSY
Frequencyy
150
DRNET
RUPPER
0.75
100
0.5
RLNET
SBRI
50
CORELS
0.73
10
15
20
25
30
15
25
0
0.5
10
10a
10*
Number of rules
Rule length
Original
Runtime
(a) Average Pi score over all datasets for increasing no. of rules.
(b) NEURULES rule
(c) NEURULES Fi scores
(d) Average runtime over
lengths with and without gradient shaping (e 0).
with vs. without relaxed
all benchmarked datasets.
conjunctions.
(log scale)
Figure 4: NEURULES is accurate for both short and long rule lists (a). The learned rules lists (b). mostly succinct and some detailed rules. The relaxed conjunction h(x) is always better (blue area) and improves the F1 score on average by 0.35 (c). We report the avg. runtime (d).
5.1 Binary Classification
We start with a comprehensive evaluation on 22 real-world binary classification datasets that span variety of domains. We provide characteristies and sources in Appx. C report weighted Pi scores (accounting for class imbalance) for budget of 10 rules in Table accuracies in Appx. Table and average runtimes in Fig. 4d
Overall. NEURULES demonstrates consistently strong performance, achieving the best overall average Fi of 0.79 and best overall average rank of 2.59. In terms of F1, its closest competitors are RINET and CLASSY (0.76), and in terms of rank SBRI. (4.06). Despite extensive hyperparameter tuning, CORELS underperforms and does not terminate within a reasonable time for 6 datasets. In terms of runtime, NEURULES is the 4-th fastest with 46s average. On the same hardware, it compares favorably to RLNET (228s), in spite of RLNET neither learning list order nor predicates. NEURULES outperforms the next best method on the Ring dataset by 0.09 Fi points. Ring consists entirely of continuous features, highlighting the advantage of NEURULES in optimizing thresholds.
Number of Rules. Next, we evaluate how each method performs under varying rule budget. We compare their results across increasing limits (n = 10, 30) and show the results in Fig. 4a We that NEURULES consistently performs best across all budget settings, maintaining comfortable lead over the best neural approach, R RINET. The greedy methods, RIPPER and CLASSY, show increasing performance for larger budgets; CLASSY needs three times many rules NEURULES to match its performance in terms of F1, showing the advantage of jointly optimizing the rule list.
Rule Length. Finally, we analyze the lengths of the rule heads NEURULES learns, i.c. the number of predicates. We give the histogram of the lengths of the rules NEURULES learned over all datasets in Fig. 4b) showing both those with and without gradient shaping. We see that the vast majority of rules consist of few predicates, i.c. they are sparse, and that gradient shaping plays an essential role in doing In Appx. Table (e we that gradient shaping also strongly improves performance, and reduces average rule length compared to other neural methods (Appx. T
5.2 Multi-Class Classification
NEURULES
RLNET
CLASSY
Next, we evaluate RULES on 6 real-world multi-class classification datasets (see Appx. C). for which we set the dimension of the consequent c E m to the number of classes We compare to RLNET and CLASSY they are strong competitors that support multi-class classification, with CLASSY being combinatorial and RLNET continuous. We allocate each method a budget of rules per class, i.e. 5m rules in total for m-class classification.
0.53 c.6o
Ecoli
0.84 0.04
0.78 0.0
0.75 0.04
Iris
0.95 1: u.24
0.81 4: 0.0
0.94 1.942c..
Penguins
+
0.99
0.96
image
0.81
0.01
0.68 0.04
0.43 0.11
Yeast
0.55
1: 0.02
0.39 1: 0.04
0.52 -: 0.00
Avg. Fl Avg. Rank
0,88
0.82
0.78
1.17
2.33
2.50
Table 2:
Multi-class classification. Fi scores for six real-world datasets.
We give the results in Table 2 We find that NEURULES obtains the best average F1 0.88 and best average rank of 1.17, giving comfortable lead over its closest competitor, RLNET, which achieves
8

---

an average Fi of 0.82 and average rank of 2.33. The results demonstrate that NEURULES handles multi-class problems with ease whilst delivering state-of-the-art performance.
5.3
Ablation Studies
To show the necessity and impact of the core components of NEURULES, i.c. learned predicates, relaxed conjunction, and learned rule ordering, we perform an ablation study.
Learned Discretization. We first examine the effect of learning thresholds as opposed to pre-defining them. To this end, we re-run the experiments from Section [5.1 with pre-discretized data. We provide the results on all datasets in Table in the Appendix. On average, we find that the F1 of NEURULES is degraded by 0.04 and 0.03 resp. for uniform and t-means-based threshold selection. Furthermore, the difference is highly dataset and discretization dependent. For example, the performance on the Credit Card dataset drops by 0.05 using k-means, but by 0.08 for uniform thresholding. In general, while fixed binning can sometimes achieve reasonable results, it requires users to tune the discretization manually. On the other hand, the learned discretization by NEURULES performs at least as well or better as the fixed binning on most datasets.
Relaxed Conjunctions. Next, to test the efficacy of the relaxation of the logical conjunction, re-run NEURULES without slack, i.c. c c= 0 in Eq. (7). We plot the difference in F1 score in Figure On most datasets, the relaxed conjunction outperforms the strict one by large margin, and on average by 0.35 F1 points. In addition, the strict conjunction performs worse on every tested dataset. Using the strict conjunction leads best to drop in performance, and in the worst case to numerical instability resulting in overflows. The extent of improvement stresses the importance of non-vanishing gradients and shows the necessity of relaxing the logical conjunction.
4c
Rule Ordering. Lastly, we examine the impact of NEURULES' s flexible rule ordering, by using fixed priorities ps, i.e. they are not updated, which is similar to RLNET. We provide the results in Table in the Appendix. We find that the Fi of NEURULES degrades by 0.05 on average, with the largest drop on the Credit Card and Adult datasets. These datasets contain many samples (n >> 30 000) and likely allow to learn many rules with sufficient coverage. Based on this evidence, we postulate that learning the order of the rules is especially important for large datasets.
6 Conclusion
We propose NEURULES, a differentiable relaxation of the rule list learning problem that converges to strict rule list through temperature annealing. NEURULES jointly learns feature discretization, the construction of conjunctive rules, and their ordering without requiring any pre-processing or manual constraints. By assigning a learnable priority score to each rule, NEURULES learns the ordering as part of the end-to-end training process. This enables it to flexibly learn both simple and complex rules, and arrange them to maximize predictive performance. As result, NEURULES produces rule lists that are succinct and accurate, supporting trustworthy decision-makinn across a wide range of applications. Extensive experiments on real-world datasets show that NEURULES consistently outperforms both combinatorial and neuro-symbolic baselines.
Limitations. NEURULES requires the rule list length be specified in advance. Learning this length differentiably is not trivial as adding or removing a rule would require a gradient signal that captures how such change influences the ordering and the selection of predicates across the list. Additionally, learning rule lists is an inherently combinatorial problem. Addressing it through a differentiable and scalable approach necessarily introduces a set of hyperparameters, including those controlling the temperature schedule used to produce discrete rule lists. NEURULES shares standard hyperparameters common deep learning models, such as learning rate and batch size. While these parameters must be set, our ablations in Appendix E.1 show NEURULES is robust to their variation.
Future Work. Extending NEURULES regression tasks opens up wide range of new applications to benefit from interpretable rule lists. In that context, we also plan to derive a non-conformity score from the rule list model for conformal prediction. Another exciting direction of future work the adaptation of TEURULES to other data types, such as images or graphs. With appropriate predicate functions that extract meaningful concepts in those domains, rule list models could be used more interpretable and accountable deep learning models.

---

References
U. Aivodji, Julien Ferry, Sebastien Gambs, Marie-Jose Huguet, Mohamed, and Siala. Leveraging integer linear programming to learn optimal fair rule lists. In Integration of Al and OR Techniques in Constraint Programming, 2022.
Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin. Learning certifiably optimal rule lists for categorical data. Journal of Machine Learning Research, 18(234): 1-78, 2018.
Siddharth Bhatore, Lalit Mohan, and Raghu Reddy. Machine learning techniques for credit risk evaluation: a systematic literature review. Journal of Banking and Financial Technology, 4(1): 111-138, 2020.
William W Cohen. Fast effective rule induction. In Machine learning proceedings 1995, pages 115-123, Elsevier, 1995.
Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances in neural information processing systems, 31, 2018
Rahul C Deo. Machine learning in medicine. Circulation, 132(20):1920-1993) 2015.
Dua Dheeru and Karra Taniskidou Efi. Uci machine learning repository, 2017. URL http:// archive. ics. uci. edu/mm Accessed: 2023-10-01.
Lucile Dierckx, Rosana Veroneze, and Siegfried Nijssen. Rl-net: Interpretable rule learning with neural networks. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 95-107. Springer, 2023.
Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. The weka data mining software: update. ACM SIGKDD explorations newsletter, 11(1):10-18, 2009.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel-softmax. In International Conference on Learning Representations, 2017.
Remy Kusters, Yusik Kim. Marine Collery, Christian de Sainte Marie, and Shubham Gupta. Differen- tiable rule induction with learned relational features. In International Workshop on Sembol-Symbolic Learning and Reasoning, 2022.
Himabindu Lakkaraju and Cynthia Rudin. Learning cost-effective and interpretable treatment regimes. In Artificial intelligence and statistics, pages 166-175. PMLR, 2017.
Benjamin Letham, Cynthia Rudin, Tyler H McCormick, and David Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. 2015.
Ioanna Papagianni and Matthijs van Leeuwen. Discovering rule lists with preferred variables. In International Symposium Intelligent Data Analysis, pages 340-352. Springer, 2023.
Hugo M Proenca and Matthijs Leeuwen. Interpretable multiclass classification by mdl-based
rule lists. Information Sciences, 51 112:1372-339, 2020.
Litao Qiao, Weijia Wang, and Bill Lin. Learning accurate and interpretable decision rule sets from neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4303-4311, 2021.
Samuel Romano, Randal S. Olson, Jeremy Green, and Jason H. Moore. Pmlb: A large benchmark suite for machine learning evaluation and comparison, 2016. URL https://gittub. com/ EpistasisLab/pae pmlb Accessed: 2023-10-01.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, ((5):266-155) 2019.
Chandan Singh, Keyan Nasseri, Yan Shuo Tan, Tiffany Tang, and Bin Yu. imodels: a python package for fitting interpretable models, 2021. URL wttp::ttt://dlt org/ 10. 21105 /os11////oss..s 03192
10

---

Emile van Krieken, Erman Acar, and Frank van Harmelen. Analyzing differentiable fuzzy logic operators, Artificial Intelligence. 302: 2022.
Nils Philipp Walter, Jonas Fischer, and Jilles Vreeken. Finding interpretable class-specific patterns through efficient neural search. In Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence. AAAL 2024.
Tong Wang, Cynthia Rudin, Finale Doshi- Velez, Yimin Liu, Erica Klampfi, and Perry MacNeille A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(70)11-37,200 2017.
William Yang Wang, Kathryn Mazaitis, and William W Cohen. A soft version of predicate invention based on structured sparsity. In IJCA,, pages 3918-3924 2015.
Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. Scalable rule-based representation learning for interpretable classification. Advances in Neural Information Processing Systems, 34:30479 30491, 2021.
Sascha Xu, Nils Philipp Walter, Janis Kalofolias, and Jilles Vreeken. Learning exceptional subgroups by end-to-end maximizing kl-divergence. In Forty-first International Conference on Machine Learning, 2024.
Hongyu Yang, Cynthia Rudin, and Margo Seltzer. Scalable bayesian rule lists. In International conference on machine learning, pages 3921-3930. PMLR, 2017.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint 1806 06988, 2018
A
Convergence of Continuous Relaxations
We show that our continuous relaxations for predicate, logical conjunction, and rule list converge to their discrete counterparts.
A.1 Predicate
We restate the proof of Xu et al. [2024]. The soft predicate for a single feature x; is defined as
(z aij. bij tx)
(16)
1 + e 1. (ay- + t (x, (z))
We now show that the soft predicate converges to the hard predicate as t + 0, which is defined as
{1 if E [asj. biy] otherwise
<
(17)
Proof: Consider the following four cases:
1. x, aij² Then for the lower bound it holds that
lim (av-22)) 60
(18)
while for the upper bound it, which must be bij > Oij > Dzs it holds that
lim e +zz = -00 0.
(19)
Hence, in the limit of tx + O, we have
lim
1 et (asy 2z) biz) 0.

---

2. > bez: Then for the lower bound it holds that
lim l
e (ay-21)) 60000
(20)
while for the upper bound which must be bes 3z. it holds that
lim (as-ass) bes)) c0 = 00. =00
(21)
Hence, in the limit of lx - O, we have
lim
+ G (ary 1,) + e biy) 0
3. au < 2s bu:: Then for the lower bound it holds that
lim e (nu-24) E00 0 +0
(22)
while for the upper bound it, which must biz > Iss it holds that
lim e (a;-bis) = - = =0.
(23)
Hence, in the limit of t + O, we have
lim te
1 + e
(ais nz) + e (a, bbz))
4. 3; dij J4 bij: If 0i azj. then
lim t =x:) c =1 and lim e (z)-bas) = = =0, 6
(24)
whereas xf; = bij. then
lim e (a))
0 and lim ete (xi-by) = = =1.
(25)
In both cases, have
1
lim
(26)
1 + e (as 2z) + e (a, bb,)
1
2
To obtain the desired behavior at the boundaries, Lee +(x;)) = or +(xx)) = 0, the output must thus be either ceiled floored, which corresponds to choosing strict or non-strict inequalities (azj < Is < big or aij S 34 I biz)-
A.2 Logical Conjunction
The soft logical conjunction for a set of predicates (ft...., Ata), where Rij = #Iz;: bıj), is defined as
Σ/= Wij
(, 0.)
(27)
Given set of non-negative weights w, E (0, with at least one weight waj being positive, we first show that the soft logical conjunction takes values in [0, I| given d predicates म.j [0, 1].
The domain of the reciprocal is के) oo) This implies for the weighted sum of reciprocals that Σ/= i = waj 0, since there exists weight waj Then the soft logical conjunction is bounded by
Σ DE=1 Wij
0 5
h₂(x;00) $1
(28)
j==1 Wiy
2-1
12

---

In particular, if and only if all active predicates are 1, i... Vj, waj > 0: fay 1, the rule (x 0.) 1. In that case, Fi such that 2j=1 Why 2ー= Wijs hence showing that hx; 0.) 1.
On the other hand, if there exists an index j where Wij > O, then क.g² = 00 ++ Mt = and the rule (x; 0,) 0.
Let us now consider the limit 4 - O, for which in Appendix A.1 for all j, i.e. the predicates are binary. Above, we have shown that
we have shown that Rsj E {0, 13
1. Vj [d] : Maj V wij 0 = (x; 0.) 1
2. 3j [d] : Waj > 0 n.
(x; 0.) =
Then, h,(x; 0.) E {0, 1}. as either all predicates are or at least one predicate is 0, and h; mimics the logical conjunction of the predicates, i.e. (x; 0,) = A7=1, Tij.
A.3 Relaxed Conjunction
The relaxed conjunction (x 0,) is defined as
d
2==1 Whj
0,) = ΣØ=1
(29)
1+m
j=1 Wis
Wij a.5
We first show that for मय the resulting soft conjunction is upper bounded by ctwu"
Proof: Let नg Then the relaxed conjunction is
2j=1 Wij
h,( 0.) = EjSt Wij
(30)
147:
1+
Mag +
Wil
Σ j==1 Waj
0.) Σ
(31)
Wij Rey +
1+
+ Wa
T+
(32)
To upper bound h2(x), lower bound the denominator. Consider the minimum value of the denominator, which is obtained when all other predicates are active, i.e. fej = 1fr all j # I, where
Wii Rij +
1
+
Wsl
2
Wij
16
Wil
1 13
Σ 1
Wij + wi
1 + 1
(33)
1
TK
178
Using this lower bound, we can upper bound the relaxed conjunction as
Σj= Wij
(x) -
(34)
Σ jzt Wij
wit
1+9 Th
77 - Wij
d
(35)
176 プ Wii wx (1 +
1/: ΣØ= Wij
(36)
1ii Σ 1=1 Wii + Wal
d wis Σド=1 Wij
(37)
E was Σd_ Wij
+ waa
(38)
+ Wil
13

---

A.4 Proof of Proposition 1
Proposition 1 Given binary rule heads (x: 8.) {0. 1) and unique priorities ps E R20, i.e. Vi + j Pi # Pj. the rule list rl(x; θ from Eq. 17 is equivalent to weighted sum of
k
r1(x; O.
Σ
Gi
arg max(h,(x;;)) pi)
1=1
Proof: The rule list rx; O.p) P) uses the rule active rule hx(x; 0,) 1 with the highest priority p: to determine the output, i.e.
r1(; θ p) = C₁
(39)
s.r. (x( 81) = A Vj -
(40)
Let i be the index of arg max, (x; 0z) .P)). and assume that there exist a further active rule with > and hy (x; B1) = 1. Then (x, 01) pt > h,(x; 0,) Ps. which contradicts that is the index of arg max, (h3 (x: 0,) P2).
B Gradient Shaping of
We analyze the proposed relaxed conjunction from Eq. 7 terms of its gradient with respect to the predicate मt and the weight wit-
B.1 Derivative with respect to ft.
To compute its derivatives, will the quotient rule for differentiation, i.e.
d f(x) de g(x)
(((a))())= where
g(x)2
0f
0f 0 Ōāg
f(x, w) Σ
Wij
(41)
Owa
j=1
1 + T
0g 1 1+ m
0g
wa(1 + 1R)
g(x, w)
Σ
Wij
(42)
it.
+ T
Owa
नo + 7)
(#t + 1T)
1=1
Then, the partial derivative of the relaxed conjunction with respect to the predicate मय is
0h,
01E dl
1+m: Wij #:s an;t
) + (E wiy) ma(l+n) (Ra+m))
(43)
14h: Ej=n Wis an,t
(2 Wij) W1r(1 + Ti))
(44)
1+me 2
(ft + n
Wij
#g+n
Consider the where there exists predicate k included rule i that is not active for sample i.e. Wigk > 0 and irsh = 0. Then the derivative for any predicate Ru, I E [d] is
Oh,
E Why ( + 17))
lim
(45)
R:c 0 Dia
(#t + TT.) Wisk 1+
Ejtt Wij
1tm t
2
The term Wis Les is finite and positive so that the gradient does not vanish if a predicate is off. In the worst case that all predicates are off, i.e. क.j = 0 for all j, the derivative is
n
EE wij wir) (1 7h) vi Dgm Wij
(46)
1+
m
14

---

We note that 1 + 77s z for small l/. so that approximately the gradient is
Ohu Iig
(Ei=1 Way yoa
Wij j= Wij
lim M
(47)
h
(E wa +
That is, the gradient is only zero if the predicate not active, i.e. wa 0, and otherwise finite and positive, scaled by the weight of the predicate.
B.2 Derivative with respect to wit
To compute the derivative with respect to the weight wa, we additionally need to consider the dependence between 77 and wag Hence, we rewrite the expressions as
d
(x(x) = f
1
Σ
Wijs
g
1+ Wij
Rij +
T
J
e
1=1
Step 1: the -term introduced by the chain rule.
Since y depends on every way.
Önp Own
f²
For the inner fraction set
Y A₁ := Ri + m Hence, the derivative of A, with respect to wa is
1 (Mig 1k) (1 + m1)
Tij
Om
(its + 7
(fti + n)
0A, 0A, On
17.
1 - Mt5
Own
dm
Owa
(Ttj + m²
Step 2: 0g/ Owa. We re-write y as Σÿ- Wij WijAj. Then have that For each individual my Aj, we use the product rule. For j = obtain
8g Thea
Σ5=1 Tes
(wij Az).
AAr
Aw) = Ai + wa Dwa
Swa
while for Z the derivative is
aA₁
(wig An) = wij Dwn
Owa
Hence, summing up overall gives
Ög = A, + Wij Owa
OA, Oq:
+ 7t
+
m
Σ 5=1
Wij
1 - f1
Om,
Owa
Ru n
f
(ftes + 72)2
3=1
Step 3: quotient rule for 0h,/ /wwa-
Because 0f /0/ai 1,
d
1 1 Aii
fA - m
Σ
Wij
(fts) + 17))2
ah
j=1
Owa
9²
Step sign of the gradient.
Write
g = Σ wi Σ Wir(Ag - A1) + Σ
Wij At A|= = Σ Wij(AA - Ar) + f Ar
so the numerator splits naturally into
d
wiy (in Ai)
- mh
Σ Waj (ît.h + 772)2
1 #aj
Ti
Tb (< 0)
We make case analysis (CA) on whether the predicate fires not.
15

---

1 + 7 7
Case A Rua 0 (predicate / is inactive). For A
it holds that Vj E [d] : Ag - As,
T Σ way (Ay(Aj A1) 20
T2 <<0, 0,
hence N = T1 + T2 <, 0 for every inactive predicate:
th Owy
म =
< 0
Case B - nu 1
(predicate / is active).
We re- write the partial derivative
ah, Owa
1 Atj
Σ
Wij
A
171 (Tij + 17:)2
(48)
==1
1 + ?7:
1 +
1/: 3
Σ Waj
(49)
Rij + 7i
कe + 1/s
(#t + n1
(1 + 17, ( (Mt + 17))
+ 1)
7) - 17: 7,
Way
(50)
(fte) + 17:)3
+ T
(fis + m/)2
(Mty + 176 + #ty : + 17,
(n) n, 11
Wiy
(51)
(fig + 1x)
j=1
ftj + 2 n
Wiy
1
(52)
#2 + 2 + 17
1=1
2 0
(53)
Since Rij E [0, 1], Aij 2 # such that each term
8, 2 #2
2 0. In particular, 8wa
8h;
= 0 if
and only if V/j [d] Wij 0 #.j = 1 V ß.j 0. i.e. the predicates are strictly binary.
Oh: Own
2 0
Case C - 0 मम < 1 (predicate | is active).
1- 7/ Because the mapping z A(z) 2 7
How large can A, A be? E [0,1].
strictly decreasing on
sgn sgn(A, Ari) = sgn fr.j- - #t)
Hence is positive if th < Rt, zero if R.j = S.. and negative if #tj > 7t..
2. Splitting the index set Let
( < #a). 9= 1 > #a).
f Σ wey-
Then
7 = = Σ Wigl (Aj- A1) + Σ Wis (Aj - Ar)
JEC
JEG
=(1 + m))
Wij
(1 17) Σ
Wij
(#ti 7h (Fra TT.)
( + m1) (fra + m))
jEC
jeg
16

---

3. Bounding the negative term [T2]
Because 0 S fij 1,00
(81++%))
S 1//1/
T2
7
4. Condition for N > 0 or N <
Ru
13j
Rsj Ax (Taj
.N = (1 mh)
Σ
wij
(1 mz)
Σ
Wi
[T2]]
(tty + nh ( 1/s)
( + 17)))
JEC
jEG
positive "support"
nogative "drag"
> 0 + support > drag +
|22]
Le. the positive help from less-active literals must exceed both the drag from more-active ones and the extra penalty [T2]-
5. What the sign of N means in practice.
> 0 (weight increased). This occurs exactly when
1 Rij
support > drag+ [
Ejec (Ay- > Ejeo Wis(Ai-Aj+th) wij (#1z + m7)
Small 17/ << makes the penalty [Tyl tiny, so the inequality easier to satisfy.
2. A literal that often fires (large set L) quickly accumulates support and its weight is pushed up.
3. If no other literal is more active (G Ø) the drag term vanishes and the weight always grows.
N < 0 (weight is decreased). Happens when either
1. one or more more-active predicates already carry substantial weight (large drag), or the literal fires only rarely, the support term is too small to beat drag+/T2].
In that case, the optimization gradually suppresses the weight, allowing "comppting" literals to dominate the conjunction.
Interpretation (gradient shaping).
1. Selective amplification. For predicate that fires (Ru > 0), the gradient is positive iff" the support from less-active literals outweighs both the drag from more-active ones and the small T2 -penalty:
support > drag + [T2]-
This easiest to satisfy when 1); tiny when the literal fires frequently, so those literals grow and become the dominant parts of the rule.
2. Automatic pruning. Predicates that stay inactive (+. = 0) always get negative gradient
(N < 0); their weights shrink every update, driving irrelevant features
3. Self-balancing dynamics. Because both support and drag depend the current weights, literal can switch from negative to positive feedback (or vice versa) during training, letting the model continually re-allocate capacity toward the most informative predicates.
The optimisation therefore performs an implicit feature-selection: irrelevant literals are pruned while relevant ones are reinforced, producing sparse, interpretable rules without any explicit E penalty.
C Dataset Statisties
We retrieve the datasets from the UCI repository [Dheeru and Efil 2017). the imodels-data repository [Singh et al. 2021 and the pmlb repository [Romano et al. 2016) We give the base statistics for the 22 real-world binary classification datasets Table and those for the real-world multi-class classification datasets in Table
17

---

Samples
Features
Dataset
Total
Positive
Negative
Ratio
Total
Discrete
Continuous
Ratio
32561
24720
0.64
Android Malware
29332
14700
14632
0.50
86
86
0
0.00
COMPAS
Covid ICU
1494
809
685
0.54
16
15
0.06
Credit Card
30000
6636
23364
0.22
15
0.55 18
Credit Screening
690
307
383
0.44
0.47
Diabetes
768
268
500
0.35
8
8
1.00
Drug Response
Flectricity
45312
19237
26075
0.42
8
0.88
FICO
10459
5000
21
0.91
German Credit
1000
700
300
0.70
57
0.05
Heart Disense
270
120
150
0.44
10
0.33
Hopatitis
155
123
0.32
Juvenile
3640
487
3153
0.13
286
284
2
0.01
Magic
Phishing
1055
6157
4898
0.56
30
30
0
0.00
Phonome
1586
3818
0.29
5
1.00
QSAR
1055
356
699
0.34
12
29
0,71
Ring
7400
3736
3664
0.50
20
1.00
Titunic
Tokyo
959
613
346
0.64
44
37
0.84
TuanDrorn
Table Dataset statistics for the 22 real-world datasets used in our experiments. We say a feature is numerical if it has more than 10 unique values.
Features
Dataset
Samples
Classes
Total
Discrete
Continuous
Ratio
Car
728
4
D
0.00
Ecoli
327
5
5
Iris
150
3
4
1.00
Penguins
333
3
4
0.57
Sat. Image
6:435
6
36
1.00
Yeast
1479
9.
2
6
0.75
Table 4: Dataset statistics for the real-world datasets used in our experiments, We say a feature is numerical if it has more than 10 unique values.
D Hardware
For the hyperparameter tuning and the comparison, we used machines equipped with 2x AMD Epyc 7773x 2.2GHz (base), 3.5GHZ (max Boost) with 128 real cores and 2TB of memory. For RRL, we used an A A100-40G since the code the authors can only be run on GPU. For the ablation studies, we additionally used machines with Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz with 32 real cores and 256GB of memory.
E Hyperparameters
We optimize the hyperparameters of each method using grid search, for each choosing that con- figuration that achieves the best average Fi score on the validation datasets (eeg_eye eeegtte__taaee horse_colics ozone-level, pc1, and breast_cancer respective papers for the hyperparameters we test. We allow a maximum runtime of 24 hours per run. equal height binning with 5 cutpoints. For the neural methods
We extend the default setting from the
For the combinatorial methods, we use their inbuild discretization according to the paper.
For SBRL, we performed a grid search for the following hyperparameters: listlengthprior E [2,3, 4]; for listwidthprior E [1, 2,3]; for maxcardinality E [1, 2, 3 and for minsupport, we fixed value at 0.05. The number of monte-carlo sampling chains is set to 5 10. The best configuration is listlengthprior = 10, listwidthprior 2, maxcardinality 3, minsupport = 0.05, and chains 10.
18

---

For DRNET, we tested the following hyperparameters: Ir E [0.001, 0.01, and lam e [0.0001, 0.001, 0.01]; epochs E [1000, 2000, 3000]; and or lam e [0.0001, 0.001, 0.01]. We ob- tained Ir = 0.01, and_ _dd_lam- = 0.0001, epochs = 3000, and or or_aam 0.0001.
As GREEDY has only hyperparameter, we optimized max_depth [3,5, 7, 10) The optimal setting is max max_depth = 5.
tested the following hyperparameters: beam_width width [50, 100, 150, 200]; n_ cutpoints E [3,5, 10]; and max__epthh E [3,5, 10). The best parameters are beam_ width 100, cutpoints 5, and max d_s__eepphhh = 5.
For CLASSY,
For RIPPER, we set -N to n (2k) and varied the parameter [1, and -O in [1.2.3].
For CORELS, we performed a grid search with the following hyperparameters: с E [0.005, 0.01, 0.02]: n_iter E [5000, 10000, 15000]; max card E [2, 4, 5]: and min min_support [0.01, 0.02, 0.05). Grid search selected = 0.01, n_iter- = 10 000, max_card = 5, and min_support = 0.01.
For NEURULES, performed grid search with the following hyperparameters: n_epochs E (250,500]; min_support E [0.1, 0.2, 0.3]: max_ support E [0.8, 0.9]: lambd [0.4, 0.5, 0.6]: bs [25610242048] and Ir [0.002, 0.025, 0.05]. We use n_epochs 500, min suppout 0.2, max max_support = 0.9, lambd 0.4, bs = 2048, and Ir = 0.01.
For RLNET, we conducted a grid search with the following hyperparameters: Ir E [0.010.020.025]: lambda_an and
[0.0001, 1..001.0.01, n_epochs E 1000, 2000, 3000]; and 12 12_lambda [0, 0.001, 0.01]. The optimal configuration is Ir = 0.01, lambda_and = 0.001, n_epochs = 3000, and 12 lambda 0.001.
E.1 Hyperparameter Sensitivity
We re-run NEURULES on all tested datasets and vary one parameter time. We provide the results in Figure [ The only noticeable impact on performance is observed when increasing the learning rate 0.5, which adversely affects the performance of NEURULES. All other parameters, using our default values otherwise, do not significantly affect the performance of NEURULES on the tested datasets.
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.16
0.18
0.2
12
0.24
0.8
0.9
1.1
1.2
frt
Learning
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.45
0.5
0.55
0.6
400
600
800
1,000
500
1,000
1,500
2,000
4
TLopochs
Batch size
0.8
6C'
0.6
0.4
0.1
0.2
0.3
0.4
0.8
0.85
0.9
0.
400
600
800
1,000
coVmin
cov
Epochs
Figure 5: Average Fi score over all datasets for different hyperparameters.
E.2
Temperature Schedules
Temperature schedules are crucial in optimization problems involving soft approximations of discrete functions. They help gradually transitioning from a soft hard decision boundary, which can improve convergence and performance. By adjusting the temperature parameter, we control the
19

---

smoothness of the approximations, allowing the model to explore the solution space more effectively during the initial stages of training and then refine the solutions as training progresses.
We use a linear temperature decay during the second half of training for both temperatures. The temperature starts 1.0 and linearly decreases 0.1 for the rule priority temperature i, and ranges from 0.2 to 0.05 for the predicate temperature ty. These values were determined through hyperparameter optimization and are unchanged across all experiments. The temperature is updated at each epoch as follows:
temp_start = 1.0
temp_end 0.1
temp temp_stare step_size (temp_start - and)_end// (aamp_end/(tttaa_ppccchse (totaal_ppocss*)) If epoch total_epochs/2: temp temp step_size
This schedule allows the model to maintain high level of flexibility during the first half of training with multiple active rules and gradually focus on only a single rule per sample in the latter half.
We plot the impact of the relaxation with regard to differ- ent temperatures t, Figure O We start training with positive temperature t, = 1.0, where the rule with highest active priority has on average 0.75 of the weight, whilst the other rules contribute the remaining 0.25. We contin- uously decrease the temperature f, towards zero, so that the end the indicator fxx; O, p. bxx) of the actual rule dominates with weight of 0.99. Using our appropriate annealing schedule NEURULES starts training using a re- laxed rule list, which it can optimize, and continuously moves towards a strict rule list.
Amxx xx
0.5
0-
1.0
0.8
0.4
0.2
Temperature f,t
Figure 6: Weight of highest priority rule for decreasing i, (variance grey) training run on the Heart Disease dataset.
F Real World: Accuracy
We report the accuracy of the methods the real-world datasets in Table The conclusions about the performance the methods are consistent with the results obtained using the weighted Fl score. Nonetheless, report the weighted Fl score the main evaluation metric, as it is more informative about the performance the methods in the presence class imbalance.
G
Real World: Runtime
10
Runtime (s)
0.8
Runtime (8)
200
0.7
10a
15
0.4
150
2
0.6
0.2
10
0
100
0.5
3
10
20
25
15
20
CP
CP
(a) CORELS HEART.
(b) SBRL on HEART.
Runtime
B Fcocere
Figure 7: Runtime (blue, left axis) and Fj score (red, right axis) versus number of cut-points (CP) the Heart dataset for CORELS (a) and SBRL (b).
Lastly, we examine the scalability of NEURULES in contrast to other rule lists. We provide the average runtime of each method across all benchmarks in Figure 4d NEURULES on average takes 75s per
20

---

NEURULES
RLNET
DRNET
GREEDY
CORELS
Adult
0.81 + 0.01
0.81 0.0
0.77 = 0.04
0.82 = 6.0
0.50 2 0.0
0.82 162:2.00 0.0
0.82 2 0.0
0.82 0.0
0.83 =
Android Malware
0.93 1:
0.95
0.32 1:
0.95 1:
0.57 4:
0.94 4
0.87 4 6os
0.50 0.0
m/a
COMPAS
0,68 0.0
0.66
0.60 0.07
0.62 0.02
0.86 002
0.68 0.0
0.65 0.01
0.65 0.01
0.67 0.01
Covid ICU
0.60 + 0.05
0.61 = 0.00
0.63 = 0.03
0.19 = 0.07
0.63 : 0.02
0.61 0.04
0.64 + 0.02
0.63 2: 0.08
0.64 + 0.04
Credit Card
0.81
0.75
0.80
0.82
0.81 +
0.78
n/a
0.81
Gorman Credit
0,73 + 0.04
0.72 = 0.04
0.72 = 0.03
0.30 = 6.02
0.72 + 0.04
0.71 + 0.04
0.72 + 0.06
0.62 + 0.10
0.70 + 0.08
Credit Screening
0.86 di 0.122
0.84 d: 0.00
0.82 d: 9.14
0.50 =: 0.05
0.86 =: G.02
0.85 4: D.22
0.86 1: D012
0.73 d: 0.24
0.75 d: 0.00
Diabetes
0.74
0.73
0.75
0.45
0.72
0.73
0.75
0.74
0.71 0.00
Drug Response
0.77 + 0.03
0.77 + 0.08
0.76 + 0.01
0.75 + 0.02
0.73 + 0.04
0.7G + 0.02
0.72 + 0.08
+: nda
+ m/u
Electricity
0.76 1: 0.01
0.71 = 0.0L
0.65 + 9
0.67 + 6.0:
0.76 +
0.66 4: DVD
0.76 +: 0.01
(.73 4: 0.9
0.78 +
FICO
0.71 +:
0.68
0.65
0.59
0.70
0.68
0.70
m/a
0.70 0.01
Heart Discase
0.80 0.03
0.75 +
0.72 + 0.04
0.52 + 6.13
0.71 + 0.05
0.78 0.09
0,80 + 0.05
0.68 0.06
0.66
Hepatitis
0.81 : 0.05
0.79 = 0.00
0.79 + 9.07
0.21 = 6.05
0,50 + 0.05
0.78 +: D
0.78 : 0.07
0.77 : 0.09
0.74 + 0.1
Juvenile
0,90
0
0.13
Magic
0.82 1: 0.01
0.79 = 0.0L
0.74 = 0.12
0.77 2 0.03
0.71 = D001
0.77 1: DUD
0.78 +: 0.01
0.74 1: 0.0
0,83 1: 0.0
Phishing
0.50 4: 0.05
0.93 d: 0.01
0.80 + 0.0G
0.93 E: 0.01
0.89 +
0.92 + D
0.89 +:
0.44 d:
0.56 4: 0.00
Phoneme
0.81
0:09
0.76
0.81
001
QSAR
0.82 :: 0.02
0.84 :: 0.01
0.80 = 0.02
0.64 = 6.02
0.71 = D.DS
0.82 : DUGS
0.79 + 0.08
wála :: máa
0.82 + 0.01
Ring
0.92 d: 0.0:
0.81 1: 0.00
0.83 2: 9.04
0.50 :: 0.02
0.61 1 DM22
0.68 1: D422
0.75 4: 0:04
Ivaa d: m/a
0.83
1: 0.01
Titanic
0.78
0.77
0:72
0.45
0.79
0.79
0.78
0.71
0.71
Tokyo
0,92 =: 0.02
0.91 = 0.02
0.91 + 0.01
0.37 + 0.05
0.58 + 0.01
0.92 = 0.02
0,92 : 0.05
náaa +: nda
0.91
+ 0.00
TranDromdd
0.96 4: 0.01
0.98 d:
0.97 4: 9.01
0.99 d: 0.0
0,93 4: 0003
0,96 1:
0.93
1: 0.03
0,20 4: 0.21
ráa 1: ma
Avg. Ace
0.02
0.79 d: 0.02
0.76 d: 0.01
0.62 0.03
0.76
0002
0.78 0002
0.75 0.02
0.68 0.58
0.75 0.02 4.28
Rank
2.59
4.18
5.77
6.73
5.02
3.70
4.59
6.63
Table 5: We report the results on 22 real-world datasets stemming from domains such as medicine,,inance and criminal justice. We compare NEURULES against CORELS, SBRL, CLASSY, GREEDY, RLNET, RRL, DRNET, and XGBOOST. We report the accuracy averaged over 5-fold cross validation. The experiments were terminated after 24 hours, indicated by n/a. NEURULES performs the best with respect to the Acc score, indicated by the lowest rank.
dataset. This is faster than DRNET, RLNET, and SBRL, but significantly slower than the greedy approaches GREEDY, CLASSY and the neural RRL, which all take below 10s per dataset. In general, NEURULES incurs a computational overhead compared to the greedy methods but compensates for it in terms of classification accuracy. RRL optimizes only rule set instead of rule list and avoids the more costly rule list optimization, which explains its faster runtime.
H Rule Complexity
The sparsity of the discovered rules is interesting due to its correlation with the interpretability of rule. We provide summary statistics for all methods in Table NEURULES learns more succinet rules than other neural approaches (RLNET, DRNEt), showing the efficacy of our gradient shaping in that regard. Compared to combinatorial methods that impose limit on maximum cardinality, the rules we find are longer. We do not think this necessarily a weakness; there may exist datasets where rules with longer clauses make sense. In general, we observe power law-like trend of rule lengths learned by NEURULES, where the majority of rules have four or fewer clauses.
I Ablation Studies
We perform ablation study investigate the impact of the different components of our method. We provide the Fi score when using uniform, k-means pr-oceesssing of continuous features, a fixed rule priority p and with strict conjunction respectively in Table 7 We re-run NEURULES on the same datasets as in the main experiments, using the same hyperparameters as in the main experiments, but with the respec- tive components removed. We report the 5-fold cross-validated score for each dataset Ta- ble 7
Method
Median
NEURULES
4.00
7.85
13.45
RLNET
6.00
9.65
15.16
DRNET
21.00
53.96
97.02
CORELS
2.00
CLASSY
3.00
2.89
0.90
RIPPER
2.00
1.94
SURE
2.00
1.91
0.71
Table 6: Rule length statistics (number of predi- cates per rule) across datasets.
21

---

Thresholding
Dataset
Original
Striet Conjunction
Fixed Order
Uniform
K-means
Quantile
Adult
0.66
0.66
Android Malware
0.93
0.33
0.92
0.94
0.33
0.33
0.35
Covid ICU
0.57
0.29
0.63
0.56
0,58
0.58
Credit Card
0.79
0.68
0.68
0.71
0.68
0.68
German Crodit
0.72
0.14
0.70
0.58
Credit Screening
0.86
0.76
0.86
0,86
0.68
0.68
Diabeles
Electricity
0.75
0.42
0.75
0,63
0,70
0.70
FICO
0.31
0.69
0.70
Heart Discase
0.80
0.40
0.80
0.78
0.66
0.66
Hepatitis
0.81
0.07
0.78
0.81
0.70
0.70
Juvenile
Magic
0.81
0.51
0.77
0.75
0.76
0.76
Phishing
0,40
Phoneme
0.80
0.59
0.59
0.76
0.65
0.65
QSAR
0.53
0.80
0.77
0.79
Ring
0.92
0.33
0.56
0.63
0.63
Titunic
0.75
0.59
0.77
0.77
0.54
Tokyo
Avg.
-0.33
-0.05
-0.14
Table 7: Ablation study comparing the obtained F1 scores using a strict conjunction, fixed rule priority, uniform, kmeans and quantile based binning. NEURULES accuracy is negatively impacted by each's 's components removal.
Table 8: Fj scores across CORELS regularization X; NECRULES column shows our method. Best per row in bold.
CORELS
Dataset
NEURULES
A=0
0.001
0.005
0.01
0.02
0.03
0.04
0.1
Android Malware
0.33
0.33
0.33
0.33
0.33
FCCO
0.70
0.69
0.69
0.69
0.69
0.69
0.69
0.69
0.68
Hcart Discase
0.68
0.68
0.68
0.07
Hepatitis
0.81
0.76
0.76
0.76
0.76
0.76
0.76
0.48
0.63
Rin
0.63
0.63
0.63
0.63
0.63
0.63
0.54
Titanic
0.75
0.68
0.66
0.66
0.66
0.62
0.54
0.54
0.54
Avg.
Sensitivity of CORELS. Across wide sweep of regularization values À (0, 0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.1}. CORELS shows largely stable performance: per-dataset F, varies little for 0.04 (e.g., FICO stays 0.69 across nearly all settings: Hepatitis
remains 0.76; Android Malware stays 0.33). Only very strong regularization (A = 0.1) causes notable degradation
datasets (e.g., Heart Disease drops 0.07). Despite this stability, CORELS consistently underperforms our method: NEURULES achieves higher F1 on every dataset in the table (e.g., 0.93 vs. 0.33 on Android Malware, 0.70 vs. 0.69 on FICO. 0.81 vs. 0.76 on Hepatitis, 0.92 vs. 0.63 on Rin, 0.75 vs. S. 0.68 on Titanic). On average, NEURULES attains 0.82 F whereas CORELS ranges from 0.47 (A 0.1) 0.63 (A 0), never matching NEURULES.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated, introduce the first differentiable rule list learning algorithm that jointly learns the discretization of features, the rules, and their order.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in the conclusion section and state all assumptions in the main text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: We prove all statements in the Appendix and provide all assumptions in the main text. To the best of our knowledge, the proofs are complete and correct.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the source code with which the experimental results were obtained in the Supplemental Material. We use fixed random seed and clearly describe how our architecture is implemented.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the source code in the Supplemental Material. We provide references to the external datasets and their sources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss wwtpcc/pcccccpcccceooooooop / wwwibbmisssinnie ssinooliccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our evaluation procedure in the main text and hyperparameter search in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use 5-fold cross validation in all our experiments and provide standard deviation in the main tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the compute resources used in the Appendix and provide runtimes in the Experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///neerrps..cc///blt///e/ iat
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics and our research conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We propose well established, so that there is no new societal impact to discuss that extends beyond the scope of the existing model class.
architecture for rule list learning. The model class itself is
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cite all respective authors for method code and datasets used in experiments.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p//eearrrppcc//ooofeeeeeee/ /c/////oeeeeeeccc//22////.. for what should or should not be described.
29


Reviews:
[{"review_id": "KztITX57TJ", "reviewer": "Reviewer_Aito", "summary": "This paper introduces a method called NeuRules, which is a differentiable method to obtain a rule list, an interpretable and easy to implement kind of decision tree. The author(s)' method takes data comprising of numeric (continuous or encoded as numeric) features, and predict one of the discrete output classes. Rather than applying combinatorial methods with discrete choices, they train such a rule-list using a differentiable relaxation, designing, implementing and running the relaxation. Their particular choices in this relaxation lead to rule lists that promote sparsity, and choose the orders of the rules (with a temperature annealing that tightens during the training). They implement and run their method on a variety of tabular datasets, they compare NeuRules with other existing methods using mainly F1 scores. Although Neurules doesn't score better on every dataset with this metric, on average it scores the best.", "questions": "Q. How does one work with categorical features? How one converts such features to  continuous ones should be mentioned, or it should be mentioned in the limitations section the fact that the whole setup requires continuous features.\n\nQ. In lines 284-289 in the Ablation studies for rule ordering, the wording is not clear if Neurules is being compared to RLNet or a version of Neurules without ordering (such as deactivating the priority $p_i$ variables. Which one is it? In the former case we cannot really call it an ablation study, and table 6 seems to indicate that that was not what was done. But in any case the wording in section 5.3 doesn't make it clear exactly what manipulation was done. Could this be made clearer?\n\nQ. Gradient Shaping (GS) is not defined explicitly as a method. Does without GS in figure 4b means to take $\\epsilon = 0$? If yes then one should say so. \n\nQ. What does color represent in figure 4(c)?\n\nQ. things are not clear in equation (10) as to whether the logit coordinates $c_\\ell$ are learnable parameters? Also are the consequents integers $\\ell_i$ (which is what would come out of the argmax, or are they $c_{{\\ell}_i}$ (which is what is written in equation (11) for exmple, but doesn't make sense if the consequent is just a number). I think there is a notational confusion here. What is the notation?\n\nQ(cont). The confusion continues in equation (13), where (for the first time, if I'm not mistaken) $\\hat{c}_i$ is introduced.  What exactly are the differentiable relaxations of these consequents? Even if they are made numeric (and doesn't remain a symbolic \"yes\" or \"no\") by one-hot encoding, then what exactly does their differentiable relaxation mean? Why would we treat them as learnable? I think I understand the overall method, but  the notation here is confusing, if not an outright mistake.\n\nThe items below are not really questions but typos that should be fixed or certain suggestions for better presentation.\n\n- Typo on page 1, line 28 in \"i.e. the set of condtions that specify when a rule triggers.\" condtions--> conditions. Also better to say ``when a rule is triggered''.\n\n- Line 81, page 3, $\\hat{rl} \\to \\widehat{rl}$.\n\n- Table 1 should be in section 5, or better yet in section 5.1 if possible. And Table 1 would do well to include the lengths of the produced rule lists for methods which are rule lists.\n\n-In Appendix A1, the proof's first step is to divide the numerator and the denominator by $e^{\\frac{1}{t}b}$ in equation (18), as to bring us to (21). Which begs the question, then why was the soft predicate (16) given as such, and not directly as (21), where its form, and the purpose of each term would be clearer. If it is a numerical stability issue, then this should be mentioned.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The combination of various differentiable relaxations are able  to work in harmony to produce rule-lists which are a very discrete  object and able to order them organically. Even though rule lists are not as prevalent as, say, decision trees they are still important in areas which require high interpretability, so this work is significant in that it allows for flexible creation of rule lists that makes the modern machine learning practitioner feel at home with the use of gradients in its learning process. Thus I can also imagine this being integrated into other machine learning pipelines as a interpretable module (with possibly non-interpretable features).\n\nAs for weaknesses on originality, and significance The differentiable relaxations work (according to the experiments that were done), but there is no reason or heuristic argument that they are the best according to some metric.", "was_revised": true, "final_justification": "After reading through other referees' comments and the authors' responses, I have decided to maintain my score.", "rebuttal": "Dear Reviewer,\n\nThank you for your in-depth, constructive feedback. Below we address your questions:\n\n1. **Categorical features**: We use one-hot encoding for categorical features and treat them as continuous features. We will make this clear in the updated manuscript\n2. **Ablation rule ordering**: In the ablation, we compare to NeuRules with a fixed, predefined rule order, where $p_i$ is not learned during training. This is similar to using a fixed order as RLNet, but using our predicate and rule learning models. We will clarify this.\n3. **Gradient shaping**: yes, we disable the gradient shaping mechanism by setting $\\epsilon = 0$.\n4. **Fig. 4 colors**: The red color marks the area where the gradient shaping leads to a performance loss. The blue signals where an improvement is achieved. The color of the markers is scaled as per the performance of the ablated model.\n5. **Notation consequent**: Each rule has a consequent $\\hat{c} \\in \\mathbb{R}^m$ that is a learnable parameter and represents the logits, which are turned into class probabilities using the softmax function. We acknowledge that denoting the consequent of each rule i as $c_i$ and the individual probabilities as $c_l$ is confusing, and we will adapt the notation to avoid this.\nWe denote as $\\hat{c}$ the logit parameter vector, and denote as c is the probability distribution over the classes induced by the learned logits.\n6. **Predicate**: We agree that Equation (21) makes the predicate easier to understand, so we will substitute it in the main text.\n7. **Typos**: We will incorporate your feedback. Thank you for your thorough review.\n\nLastly, we would like to provide further context to the rationale behind our differentiable relaxations. We only use relaxations that converge to their strict counterparts. For rule lists, the methodology of learning rule ordering and predicates is the first of its kind. For the logical conjunction, we introduce a novel relaxation approach with the slack $\\epsilon$ and derive its gradient shaping (GS) properties. We evaluate GS in an ablation, showing significant improvements in accuracy and rule sparsity. We empirically find that Neurules as a whole performs better compared to other neural approaches, which employ different relaxations for logical conjunctions (RLNet, DRNet, RRL)."}, {"review_id": "YUNNZaUo5y", "reviewer": "Reviewer_m6Ct", "summary": "The rule learning method is essential for interpretable AI and highly trustworthy application domains. Current rule-learning methods need pre-discretized features, and some of the rule-learning techniques are limited in scalability. The paper proposed a novel framework for learning rules from the data without feature pre-discretization. The method mainly used a fully differentiable manner to construct the predicate, rule, and rule list ordered by the importance of rules. The experimental results indicate that the proposed method obtains the state-of-the-art performance.", "questions": "1. The notion of an atom in first-order logic (FOL) typically includes both terms and predicates. However, the concept of a “predicate” as described in your paper appears to differ slightly from its traditional definition in FOL. Could you please clarify the relationship between the predicate used in your paper and the standard predicate in FOL? \n\n2. Additionally, in the rule learning community, the term “soft predicate” (as described in [1]) carries a different meaning from the one presented in your work. Could you also elaborate on the connection or distinction between the soft predicate defined in your paper and that in [1]?\n\n3. Regarding Equation (4), could you explain how the parameters are chosen? For example, what is the rationale behind the formulation $(2x_j - a_{ij})$ and the exponential term $e^{\\frac{1}{t_{\\pi}(3x_j - a_{ij} - b_{ij})}}$?\n\n4. In the ablation studies, the paper does not appear to provide the specific values used for the temperature parameter $t_{\\pi}$. Does the value of $t_{\\pi}$ have a significant impact on the experimental results?\n\n5. Can you describe the connections, advantages, and disadvantages of the proposed rule learning method with other rule learning methods based on the column generation algorithm, such as [2].\n\n[1] Wang, W. Y., Mazaitis, K., & Cohen, W. W. (2015, July). A Soft Version of Predicate Invention Based on Structured Sparsity. In\nIJCAI (pp. 3918-3924).\n\n[2] Sanjeeb Dash, Oktay Günlük, Dennis Wei: Boolean Decision Rules via Column Generation. NeurIPS 2018: 4660-4670", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-structured and presents its ideas using clear and rigorous mathematical formulations. The authors designed a unified method to learn Boolean conditions denoted as predicates in the paper, conjunctions, and the order of the rule list simultaneously.\n\nWeaknesses:\n\nThe paper omits preliminary explanations of fundamental concepts, such as the notion of a predicate, or misses the proper citations for these notions. Additionally, it lacks important details regarding the experimental setup, including the specific optimization methods employed. Besides, in the related work, the authors also ignore some discussion regarding the rule learning with the column generation algorithm and the first-order rule learning method.", "was_revised": true, "final_justification": "The authors have properly given the rebuttal.  I recognize that the paper has good technical contributions, but there are still some issues on the presentation side.  I have already put a rather positive score, which is not very strong but I think is appropriate.", "rebuttal": "Dear Reviewer,\nthank you for your in-depth, constructive feedback. \n\nBelow we address your questions and concerns:\n\n1. **Predicate FOL**: Thank you for your comment. Although our predicates are parameterized and grounded in feature space, they serve an analogous role as in FOL: they act as the basic Boolean conditions used to construct a conjunctive rule, similar to how predicates in FOL form the structure of atomic formulas. \n\n   In our work, we restrict these atomic conditions to threshold-based conditions over a single feature  (e.g., $a_{ij}<x_j<b_{ij}$). In principle, for every individual feature $x_j$ there exist infinitely many predicates by combinations of $(a_{ij},b_{ij})$. Thus, in subgroup discovery/rule mining, the first step is usually to generate an initial set of predicates using fixed discretization. Instead, we propose a method that explicitly optimizes those, instead of pre-defining them a priori. After training, these learned predicates become fixed, binary-valued conditions that can be interpreted as atomic logical components. Both concepts are very similar in spirit but differ in the formalities. We will clarify this connection in the text.\n2. **Soft predicate**: In [1], soft predicates refer to latent logical abstractions that emerge through structured sparsity over clause weights. These predicates are not explicitly defined; rather, they represent shared parameter patterns across clauses, serving as a soft form of predicate invention. \nIn contrast, our soft predicates are explicit, parameterized threshold functions over real-valued features. Each outputs a value in [0,1], approximating binary inclusion and enabling differentiable rule learning. \n\n   Although our formulation resembles fuzzy predicates-in that both yield values in [0,1], the underlying motivation differs. In fuzzy logic, the fuzziness reflects graded truth, such as \"x is partially tall.\" In our case, the soft predicate is a training-time relaxation, used to enable end-to-end optimization via gradient descent. The goal of this relaxation is to learn the threshold by which we determine “x is tall”, by means of minimizing the predictive error.  The learned predicates are ultimately interpreted as binary threshold conditions, although the intermediate softness could have similar semantics, albeit having a different functional form as say in the weak conjunction of Łukasiewicz, its primary purpose is to facilitate learning. As temperature annealing progresses, these soft predicates converge to interpretable, discrete conditions similar to classical logical atoms. We will clarify these distinctions in the revised manuscript to avoid confusion and better position our work in relation to both prior neural-symbolic methods and fuzzy logic approaches.\n3. **Threshold parameters**: The lower and upper bound parameters $a_{ij}$ and $b_{ij}$ are learned via gradient descent. The formulation in Eq. (4) allows us to take gradients with respect to the bounds. In particular, in Appendix 1 we show Eq (4) is equivalent to Eq (21)\n$$\\hat{\\pi}(x_j;a_{ij},b_{ij},t_{\\pi})=\\frac{1}{e^{(x_j-a_{ij})/t_{\\pi}}+1+e^{(b_{ij}-x_j)/t_{\\pi}}}$$\nIn the limit of $t_{\\pi}\\to 0$, the predicate verifies whether $x_j > a_{ij}$ and $x_j<b_{ij}$, i.e. it behaves like a strict thresholding function would. We will clarify the intuition behind this formulation in the updated text.\n4. **Temperature**: In our evaluation, we use a starting temperature of $t_{\\pi}=0.2$, which is gradually annealed to $0.05$ (Appendix E.2). Additionally, in the hyperparameter sensitivity section (Appendix E.1), we re-run our experiments with different $t_{\\pi}$. The results displayed in the upper left corner of Figure 5 show that under reasonable values of  $t_{\\pi}$, Neurules performance is stable.\n5. **Connection to column generation**: Both [2] and Neurules seek to find accurate and intrepretable sets of rules, but differ in the kind of rule ensemble that is learned and scope of application. The column generation method focuses on learning CNF (AND-of-ORs) and DNF (OR-of-ANDs) rule sets on binary data. On the other hand, Neurules learns *ordered* rule lists (IF THEN; ELSE IF) on continuous data. \n\n   Column generation casts rule learning as a mixed-integer program with provable guarantees, but to that end requires pre-mined candidate rules. In contrast, our method learns thresholds, rules, and their ordering jointly in a fully differentiable framework, without pre-discretization or candidate enumeration. While our approach does not offer global optimality guarantees, our evaluation shows that Neurules outperforms combinatorial methods for rule list learning that first mine candidate rules (SBRL,CORELS), and in the case of CORELS come with optimality guarantees over that search space. We will include this distinction in the revised text."}, {"review_id": "QpFNLjWEK1", "reviewer": "Reviewer_JExq", "summary": "This paper presents a differentiable relaxation of the problem of learning rule lists on tabular data. It jointly learns predicates (of the form $a < x < b$), conjunctions of those predicates to form rules, and the order of those conjunctions. The number of rules is held fixed. The differentiable relaxation of predicates yields soft predicates, which converge in the temperature limit to strict predicates -- this is exploited in the optimized algorithm by temperature annealing as training progresses. To learn conjunctions of rules, the authors use a generalization of the weighted harmonic mean to include a slack term, which simultaneously eliminates vanishing gradients and encourages sparse rules. Finally, to learn rule list order, the authors propose to learn a priority vector for the position of each rule in the list. The final form of a rule list is a sum over all classes of an indicator over the highest priority rule; however, this indicator is relaxed into a Gumbel-softmax approximation of the argmax, which allows gradients to flow beyond the single selected rule. This paper uses another form of temperature annealing to collapse this approximation later in training. The authors also propose a regularizer based on the minimum and maximum coverage of rules. Experiments comparing this method to other neural methods, as well as combinatorial discrete optimization methods, are presented.", "questions": "1. It appears that the rule ordering optimization method introduces discontinuities into the loss landscape of your neural architecture. When one priority value overtakes another, this results in an immediate swap of the rules and a corresponding change in the loss (say $t_{rl} \\approx 0$). I have a similar concern about the process to learn rules -- it seems that there are discontinuities introduced whenever a predicate achieves non-zero weight. My question is: do these discontinuities matter for your optimization process? Are they solved by the relaxations to achieve differentiability?\n\n2. How do the performance and sparsity (# of rules, and # of predicates per rule) of your method compare to CORELs when CORELs is given the same predicates to use as your method has learned? That is; how do the rule and order learning stages of your method compare to an optimal approach, conditioned on pre-selected predicates? If it performs comparably in both performance and sparsity across a variety of datasets, I will consider raising my score. \n\n3. I have several related questions relating to the implicit regularization in the rule construction process. How strong is the implicit regularization towards sparser rules in section 3.2? Is the strength of regularization tunable, for example by adjusting the slack magnitude? Does the regularizer encourage rules to be sparser, even when some terms have useful signal, or does it only eliminate useless terms?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The differentiable relaxations presented for each of the three layers of rule list learning are principled, justified by both intuition and thorough arguments for convergence and gradient stability, and clearly advance prior work on neural rule lists.\n\n2. The paper is written in a clear and concise manner. The issues with existing approaches, along with the authors' proposed solutions, are well motivated and clearly reasoned.\n\n3. The experiments effectively demonstrate that this approach is better on many datasets compared to previous neural methods.\n\n4. The easy extension to multi-class classification problems is a nice benefit of the proposed architecture.\n\n## Weaknesses\n\n### Major Weaknesses\n\n1. The comparisons to combinatorial methods, especially CORELs, are weak and do not present a fair comparison. The choice to only use 5 evenly spaced cut points on continuous features undercuts CORELs' ability to fit the data well -- in fact, due to the use of objective-based bounds, the use of fewer cut points may contribute to the surprising number of timeouts of CORELs in the experiments (even on small datasets). CORELs provides a feature selection method in both the paper and the code based on frequent itemset mining. In fact, existing prior work shows that, at least on the FICO dataset, CORELs outperforms the best metric presented in Table 1 (He et al., 2023; JAIR: Visualizing the Implicit Model Selection Tradeoff); FICO is listed as timing out in this paper.\n\n2. Similarly, the choice to perform hyperparameter optimization only on several held out datasets and not on each dataset individually is a weak-point of the evaluation in this paper. Especially for the discrete methods, the regularization is directly compared against misclassification error, and is sensitive to the class balance and particulars of each dataset. At least the regularization parameter should have been tuned for each dataset individually, ideally with nested cross validation (see Scikit-learn Nested vs. non-nested cross validation).\n\n### Minor Weaknesses\n\n1. There is little attention given to the actual complexity of the rule lists identified by this method. The authors demonstrate that this approach can find good rule lists which don't have many rules, but it is not clear how many conjunctions these rules have. A rough metric to compare to sparse methods would be the total number of predicates used in the rule list, rather than the length of the list.\n\n2. The maximum cardinality of 5 for CORELs is too high - a typical value for this is 2 or 3. Allowing length-5 predicates is probably another contributor to the timeouts observed in the results tables. Also, the max cardinality of 5 presented in this paper is not one of the options in the grid search (2, 3, 4) - I think this may be a typo on line 534.\n\n## Conclusion\n\nThis paper is technically solid, with reasonable and well-justified methods to make the rule list learning problem differentiable. However, the evaluations, for all methods in terms of hyperparameter optimization, and for CORELs in terms of continuous feature binning and hyperparameter choices, were incomplete and did not present a fair comparison.", "was_revised": true, "final_justification": "After extensive discussion with the authors, I have raised my score from a borderline reject to an accept. My issues with the paper, and their resolutions, are as follows:\n\n---\n\n**Issue:** The evaluation of discrete rule list methods, like CORELS, were not run with enough cutpoints on continuous features. \n\n**Resolution:** The authors pointed me to a figure in their appendix, which showed that increasing the number of cutpoints did not improve the performance (as measured by f1 score) of CORELs on two different datasets. More cutpoints led to more features to choose from, and also overfitting and timeouts.\n\n---\n\n**Issue:** The choice to do hyperparameter tuning on only held out datasets, then to apply those hyperparameters uniformly to all evaluation datasets, is biased against methods which are sensitive to particular datasets.\n\n**Resolution:** The authors provided evidence, in the form of additional experiments, showing that the choice of regularization did not affect at least CORELS' results on six of the evaluation datasets. In this light, I find the paper's experiments to be more trustworthy than my first reading and review. I still encourage the authors to perform limited per-dataset hyperparameter evaluation so that doubt cannot be raised in this direction.\n\n---\n\n**Issue:** On average, the rules found by NeuRules are too complicated to be called interpretable.\n\n**Resolution:** Although the average rule length is quite long, the median is small, and there are several very large rules contributing to the high average. These large rules may be either spurious or meaningful, but the authors argued that they are easily pruned if they are deemed spurious using held out data.", "rebuttal": "Dear Reviewer, \n\nThank you for your in-depth, constructive feedback.\n\nBelow we address your concerns:\n1. **Comparisons**: We respectfully disagree with the suggestion that our setup for CORELS is not fair or misrepresents its capabilities.\nDue to the delicate balance between performance and runtime when choosing the parameters, we performed a series of experiments for CORELS where we systematically varied the number of cut points. In Appendix G, Figure 7, we report runtime and performance for CORELS with an increasing number of cutpoints. We observe that, contrary to intuition, an increase in cutpoints has a slight negative impact on $F_1$ score. On the other hand, increasing the number of cut points leads to an exponential growth in the number of frequent itemsets considered during CORELs’ preprocessing, resulting in an exponential runtime increase. In theory, increasing the number of cut points should expand the search space and allow CORELs to discover better classifiers. However, in our experiments, this increase in resolution often leads to overfitting and timeouts, not improved generalization.\n\n   With respect to the FICO dataset and the comparison to He et al. (2023), we note that the reported performance of CORELs is achieved under a very different experimental setup, including an extensive and carefully crafted discretization scheme and what appears to be significantly relaxed runtime constraints. Under such idealized and hand-tuned conditions, CORELs can outperform other models. But this underscores - rather than undermines - our central contribution: our method learns the discretization, rule structure, and ordering jointly in a single, scalable pipeline, requiring no manual pre-processing, and delivers competitive (often superior) performance in a fraction of the runtime.\n2. **Hyperparameters**: Our evaluation avoids fine-tuning hyperparameters per dataset to ensure fairness and realism across a broad range of methods. In particular, neural approaches like ours often involve many hyperparameters that, if tuned per dataset, could lead to overly optimistic results and poor generalization. \nMoreover, tuning regularization parameters per dataset for combinatorial methods such as CORELs is especially problematic due to their high computational cost. As noted in the paper, many of these methods already approach or exceed runtime limits, and performing nested cross-validation across all datasets would break our computational budget.\n\tInstead, we determine the best overall hyperparameter configuration on a few holdout datasets. With the best overall parameters, the subsequent evaluation is conducted across all datasets to better reflect each method's robustness and out-of-the-box performance.\n3. **Rule complexity**: The sparsity of the discovered rules is interesting due to its correlation with the interpretability of a rule. We provide in Figure 4b) the distribution of rule lengths, i.e., the number of predicates per rule, over all datasets. Below, we provide a table with statistics for all methods, which we will add to the Appendix:                    \n |Method                    |   median |   mean |   std |\n|:-------------------|---------:|-------:|------:|\n| Neurules      |     4.00 |   7.85 | 13.45 |\n| RLNet              |     6.00 |   9.65 | 15.16 |\n| DRNet                |    21.00 |  53.96 | 97.02 |\n| CORELS  |     2.00 |   1.83 |  0.86 |\n| CLASSY      |     3.00 |   2.89 |  0.90 |\n| RIPPER             |     2.00 |   1.94 |  1.37 |\n| SBRL |     2.00 |   1.91 |  0.71 |\n   Neurules learns more succinct rules than other neural approaches (RLNet, DRNet), showing the efficacy of our gradient shaping in that regard. Compared to combinatorial methods that impose a limit on maximum cardinality, the rules we find are longer. We don't think that this is necessarily a weakness; there may exist datasets where rules with longer clauses make sense. In general, we observe a power law-like trend of rule lengths learned by Neurules, where the majority of rules have four or fewer clauses.  \n4. **Cardinality**: We think that there are many datasets for which rules with five or more conditions are appropriate. Increasing the maximum cardinality does lead to a combinatorial explosion of possible rules and hence termination issues for CORELS. However, as a configuration with cardinality five achieved the best accuracy, it was selected in the hyperparameter optimization. Thank you for pointing out the typo in the Appendix. We will update the text.\n\nRegarding your questions:\n\n1. **Loss discontinuities**: You're right in observing that, in principle, discontinuities could arise from rule reordering or predicate activation. The idea of Neurules is to overcome the obstacle of discontinuity in a rule list using smooth, differentiable approximations. For rule ordering, we apply a Gumbel-Softmax relaxation, which smooths the argmax operation and allows gradients to flow even when priority values are near a crossover point via the additional Gumbel term in Eq. (12). Similarly, predicate inclusion is handled through soft conjunctions with slack, so the model can adjust predicate weights without abrupt changes in the loss.  \n\n   That said, as the temperature approaches zero, the approximations can become steep. Particularly, if an input value lies at the transition, the soft predicate is effectively discontinuous, such as e.g Fig. 3a for $x=0.3$. In very rare cases, this can lead to very large gradients. In practice, this has not posed a significant problem for us because we control the temperature annealing schedule and avoid hard transitions. While we did not find it necessary in our experiments, given that we precisely know when this happens, one practical safeguard would be to monitor this operation and stop gradient flow in this situation.\n\n2. **Comparison to CORELS**: Although CORELS offers formal optimality guarantees, these apply only to the regularized training objective and only within the fixed rule space defined by mined itemsets. Although the method and paper are impressively rigorous, the authors do not provide a tight generalization bound. One could derive such bounds using standard tools like Hoeffding’s inequality, since the hypothesis space is finite, but they degrade rapidly as the rule space grows. Yet this bound would only hold for the regularized objective and does not allow us to make any statements about the performance on the test set.\n\n   One can, as you suggest, use Neurules to extract predicates for CORELS. To test this idea, we take the FICO dataset and extract all boolean predicates that Neurules learns, excluding those with $w_{ij}=0$ and those that are always 0 or 1, respectively. We use these predicates to binarize the train and test sets for each cross-validation split. On this data, we re-run CORELS. We obtain an $F_1$ score of $0.68+-0.01$ with an average runtime of 5 minutes per fold compared to Neurules $0.70$. We hypothesize that the difference stems from the maximum cardinality constraint placed by CORELS, whereas Neurules predicates were learned in the context of longer, unrestricted rules. So while it is possible to combine Neurules predicate learning with CORELS search method, there are no immediate benefits in terms of performance, but potentially in sparsifying the learned rule list.\n3. **Sparsity**: The sparsity pressure in our method arises from the slack term\n$\\eta_i = \\frac{\\varepsilon}{\\sum_j w_{ij}}$ in Eq. (7). This slack appears in the denominator of the relaxed conjunction and controls how sharply low-activation predicates are penalized. When ${\\hat{\\pi}}{ij}$ is close to zero, a small $\\eta_i$ causes the denominator $(\\hat{\\pi}{ij} + \\eta_i)$ to be large, which results in strong negative gradients on $w_{ij}$, pushing that literal's weight toward zero. Conversely, if $\\hat{\\pi}{ij} \\approx 1$, the term contributes positively to the gradient, reinforcing the literal. The parameter $\\varepsilon$ thus controls the strength of this push-pull: smaller values lead to stronger sparsity pressure. As $\\varepsilon \\to 0$, this pressure increases, but if $\\varepsilon$ reaches exactly zero, the gradients can vanish or tend to infinity when $\\hat{\\pi}{ij} = 0$. In practice, we keep $\\varepsilon$ small but strictly positive to ensure effective pruning without numerical issues. This mechanism eliminates literals with minimal contribution while retaining informative ones, and $\\varepsilon$ provides a smooth, tunable control over the regularization strength."}, {"review_id": "yjnPm4PKrz", "reviewer": "Reviewer_oPq7", "summary": "The paper proposes NEURULES, an end-to-end differentiable approach for learning rule lists, jointly handling discretization, rule composition, and ordering. Experiments demonstrate consistent improvements over existing methods across various datasets.", "questions": "Although the overall framework is elegant and reasonable, based on my experience with differentiable rule learning methods, I still have concerns regarding the scalability of the proposed approach to high-dimensional datasets (e.g., feature dimension >100 or even >1000). The current experiments include only one dataset with dimension above 100. In scenarios of extremely high-dimensional data, classical feature preprocessing methods might show relative advantages. I’d appreciate it if the authors could provide additional experimental results or discussions on datasets with higher dimensionality to further validate scalability.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n\t1. Technically solid contributions with clear and effective innovations (differentiable predicates, relaxed conjunction, and learned rule ordering).\n\n\t2. The writing is clear and well-organized, providing intuitive motivation and explanation of model architecture.\n\n\t3. Extensive experiments on diverse real-world datasets and multiple baselines demonstrate the effectiveness convincingly.\n\nWeaknesses:\n\n\t1. Limited experiments on datasets with high feature dimensions, thus the scalability to high-dimensional scenarios needs further validation.\n\n\t2. The method requires pre-specifying the number of rules, which may cause inconvenience in practical applications.", "was_revised": true, "final_justification": "Thank you for your reply. I have no other questions and will keep my recommendation", "rebuttal": "Dear Reviewer,\nthank you for your in-depth, constructive feedback. \n\nWe want to comment further on Neurules's scalability and relative performance compared to other methods. In our experiments, the Juvenile dataset with 286 features has the highest dimensionality. On it, Neurules is the best performing method, with an $F_1$ score of $0.89$.\n\nFurthermore, we supplement results on two additional datasets from the UCI repository that have hundreds of features:  “Drug-induced autoimmunity” (477 samples x195 features) and “TUNADROMD” (4465 samples x242 features), which deals with Android malware. We use the same hyperparameters from the experiments and set the rule list size to $10$.\nWe provide the results for each in the two tables at the bottom, excluding CORELS and SBRL due to their inability to scale to higher dimensions. \n\nOn TUNADROMD, we achieve an $F_1$ of $0.97$ compared to a best of $0.98$ (Classy, RLNet). For drug-induced autoimmunity, Neurules is the best performing method with a substantial gap between it ($0.78$) and the next best (RLNet, $0.73$). Drug-induced autoimmunity consists mainly of continuous features, where Neurules threshold learning ability comes in handy, while TUNADROMD is only binary.\n\nIn general, we agree that scalability is an important aspect for the real-world deployment of any rule list model. The current results indicate that Neurules is on par for large-scale datasets of discrete features or even superior on those with mainly continuous features.\nNeurules runtimes scale favorably due to the gradient-based optimization. We will add these results to the updated manuscript and expand the discussion on scalability.\n## Drug-induced autoimmunity\n|  Method                |   F1 |   F1_std |   Accuracy |   Accuracy_std |   Runtime |   Runtime_std |\n|:-----------------|-----:|---------:|-----------:|---------------:|----------:|--------------:|\n| Neurules    | **0.77** |     0.02 |       0.79 |           0.01 |     29.35 |          2.08 |\n| RLNet            | 0.73 |     0.03 |       0.76 |           0.02 |    209.39 |         24.12 |\n| Classy    | 0.70 |     0.04 |       0.76 |           0.02 |     12.93 |          4.41 |\n| Greedy | 0.67 |     0.05 |       0.73 |           0.04 |      0.02 |          0.01 |\n| Ripper           | 0.69 |     0.02 |       0.72 |           0.03 |      1.35 |          0.28 |\n\n\n\n\n## Results on TUNADROMD\n|  Method                |   F1 |   F1_std |   Accuracy |   Accuracy_std |   Runtime |   Runtime_std |\n|:-----------------|-----:|---------:|-----------:|---------------:|----------:|--------------:|\n| Neurules    | 0.97 |     0.01 |       0.97 |           0.01 |    125.60 |         24.53 |\n| RLNet            | **0.98** |     0.01 |       0.98 |           0.01 |    248.77 |         31.77 |\n| Classy    | **0.98** |     0.00 |       0.98 |           0.00 |      3.38 |          0.21 |\n| Greedy | 0.92 |     0.01 |       0.93 |           0.01 |      0.03 |          0.01 |\n| Ripper           | 0.92 |     0.01 |       0.93 |           0.01 |      8.03 |          0.46 |"}]

Author rebuttal:

Meta-review: This paper introduces NeuRules, an end-to-end differentiable framework for learning rule lists. The method jointly learns discretization, predicate formation, conjunction, and ordering, using principled relaxations such as soft predicates, slack-based conjunctions, and Gumbel-softmax ordering. The framework aims to combine the interpretability of rule lists with the scalability and flexibility of gradient-based optimization.

Reviewers generally agreed that the paper is technically solid and makes a clear contribution to interpretable machine learning. The differentiable relaxations are well-motivated, carefully designed, and advance prior neural rule-learning approaches. The paper is clearly written, with thorough mathematical exposition, and is supported by extensive experiments across diverse tabular datasets. Results show that NeuRules achieves strong average performance compared to both neural and combinatorial baselines, while maintaining interpretability. Multiple reviewers highlighted the potential impact of NeuRules as a bridge between modern differentiable learning and interpretable symbolic models.

Therefore, my recommendation is Accept.
Decision: Accept (poster)