Title: Orthogonal Gradient Boosting for Interpretable Additive Rule Ensembles
Venue: NeurIPS 2023
Accepted: no
Keywords: Rule Learning

Gradient Boosting

Branch-and-bound
Abstract: Gradient boosting of decision rules is an efficient approach to find interpretable yet accurate machine learning models. However, in practice, interpretability requires to limit the number and size of the generated rules, and existing boosting variants are not designed for this purpose. Through their strict greedy approach, they can increase accuracy only by adding further rules, even when the same gains can be achieved, in a more interpretable form, by altering already discovered rules. Here we address this shortcoming by adopting a  weight correction step in each boosting round to maximise the predictive gain per added rule. This leads to a new objective function for rule selection that, based on orthogonal projections, anticipates the subsequent weight correction. This approach does not only correctly approximate the ideal update of adding the risk gradient itself to the model, it also favours the inclusion of more general and thus shorter rules. Additionally, we derive a fast incremental algorithm for rule evaluation, as necessary to enable efficient single-rule optimisation through either the greedy or the branch-and-bound approach. As we demonstrate on a range of classification, regression, and Poisson regression tasks, the resulting rule learner significantly improves the comprehensibility/accuracy trade-off of the fitted ensemble. At the same time, it has comparable computational cost to previous branch-and-bound rule learners.

Abstract

Gradient boosting of decision rules is an efficient approach to find interpretable yet accurate machine learning models. However, in practice, interpretability requires to limit the number and size of the generated rules, and existing boosting variants are not designed for this purpose. Through their strict greedy approach, they can increase accuracy only by adding further rules, even when the same gains can be achieved, in a more interpretable form, by altering already discovered rules. Here we address this shortcoming by adopting a weight correction step in each boosting round to maximise the predictive gain per added rule. This leads to a new objective function for rule selection that, based on orthogonal projections, anticipates the subsequent weight correction. This approach does not only correctly approximate the ideal update of adding the risk gradient itself to the model, it also favours the inclusion of more general and thus shorter rules. Additionally, we derive a fast incremental algorithm for rule evaluation, as necessary to enable efficient single-rule optimisation through either the greedy or the branch-andbound approach. As we demonstrate on a range of classification, regression, and Poisson regression tasks, the resulting rule learner significantly improves the comprehensibility/accuracy trade-off of the fitted ensemble. At the same time, it has comparable computational cost to previous branch-and-bound rule learners.

1 Introduction

Algorithms for learning additive rule ensembles (or rule sets) are an active area of research, because 20 they are intrinsically interpretable yet relatively accurate due to their modularity and ability to 21 represent interaction effects. While there is an emerging consensus that rule ensembles should 22 optimize the trade-off between statistical risk and cognitive complexity in terms of number and lengths 23 of rules (see Fig. 1), there is a multitude of diverse approaches for performing this optimization. 24 This ranges from computationally inexpensive generate-and-select approaches [10; 14], over more 25 expensive minimum-description length and Bayesian approaches [29], to expensive full-fledged 26 discrete optimization methods [6; 30]. Within this range of options, methods based on gradient 27 boosting [9] are of special interest because of their robustness against changes in the training data, 28 flexibility to adapt to various response variable types and loss functions, and finally their good model 29

performance relative to their computational cost. 30

On the other hand, state-of-the-art rule boosting approaches are based on design choices that compro31

mise their risk/complexity trade-off. The traditional gradient boosting adaption [8] resorts to greedy 32

optimization of the individual rules, which results in additional rules and additional conditions per 33

rule to reach a desired statistical risk level. The more recent optimal rule boosting approach [3] 34

partially addresses this issue, but it is based on the uncorrected weight updates of the extreme gradient 35

boosting framework [4]. This too results in the inclusion of unnecessary extra rules, especially for 36

0

20

40

60

80

100

10

2

10

1

10 0

Risk

Gradient Boosting / XGBoost:

+18418 if PS

120 & year

2005

+13919 if count

69 & km

125000 & PS

220 & year

2011

- 8545  if 120

powerPS

160 & 2005

year

2011

+2769  if True

- 4150  if count

18 & km

50000 & PS

120 & year

2005

+2826  if count

18 & km

100000 & year

2008

+5878  if count

150 & km

125000 & PS

220 & year

2005

- 10356 if km

125000 & PS=220 & year

2011

Proposed algorithm:

+5245  if PS

120 & year

2005

+6779  if km

125000 & year

2011

+7451  if count

150 & PS

220

+3932  if year

2000

+6001  if PS

160 & year

2008

150 & PS

160 & year

2015

used cars

SIRUS

GS

GB / XGB

FCOGB

train

test

0

20

40

60

80

100

Cognitive Complexity

Gradient Boosting:

- 3.03 if var

1.24

+3.47 if skew

7.62 & var

-0.50

+5.65 if curt

-2.18 & skew

7.62 & -0.50

var

3.38

- 4.01 if curt

-2.18 & skew

3.42 & -2.26

var

1.24

- 5.38 if curt

4.57 & skew

-3.38 & var

1.24

+3.41 if curt

4.57 & ent

-0.19 & skew

0.76 & -0.50

var

1.24

- 3.92 if curt

-2.18 & skew

7.62 & var

1.24

- 2.76 if curt

-0.18 & ent

-0.19 & skew

0.76 & -0.507

var

1.24

+5.05 if -2.18

curt

-0.18 & ent

0.54 & skew

3.42

& 1.24

var

3.38

- 3.47 if curt

1.40 & skew

7.62 & var

-2.26

Proposed algorithm:

- 4.80 if var

1.24

+5.20 if skew

7.62 & var

-0.50

+7.68 if curt

-2.18 & skew

7.62 & -0.50

var

3.38

- 4.36 if skew

3.42 & -2.26

var

1.24

- 6.81 if curt

4.57 & skew

-3.38 & var

1.24

+3.34 if curt

4.57 & ent

-0.19 & skew

0.76 &

-0.50

var

1.24

banknote

SIRUS

GS

GB

XGB

FCOGB

train

test

loss functions with unbounded second derivatives like the Poisson loss. Most importantly, both 37 approaches use the strict stagewise fitting approach where rules are not revised after they are added to 38 the ensemble. Thus, they can increase accuracy only by adding further rules, even when the same 39 gains can be achieved, in a more interpretable form, by altering those already present in the model. 40

Here we develop the first rule boosting algorithm that consistently optimizes the accuracy/complexity 41 trade-off of the produced rule sets. For that, we adopt the fully corrective boosting approach [26] 42 where all rule consequents are re-optimized in every boosting round, which can be done with 43 only little computational extra effort given the usual convex loss functions. We then derive a new 44 objective function for selecting individual rule bodies that anticipates the subsequent consequent 45 re-optimization. This function is based on considering only the part of a rule body orthogonal to the 46 already selected rules, which, as we show, correctly identifies the best approximation to the ideal space 47 for consequent optimization defined by the risk gradient. Finally, we derive a corresponding efficient 48 algorithm for cut-point search, which is crucial for, either greedy or branch-and-bound, single rule 49 optimization. As we demonstrate on a wide range of datasets, the resulting rule boosting algorithm 50 significantly outperforms the previous boosting variants in terms of risk/complexity trade-off, which 51 can be attributed to a better risk reduction per rule as well as an affinity to select simpler rules. At the 52 same time, the computational cost remains comparable to the previous branch-and-bound rule learner. 53

The paper is organized as follows. After giving a brief overview of the wider literature on intepretable 54 machine learning and additive rule ensembles (Sec. 2), we recall the formal basics of rule ensembles 55 and gradient boosting in Sec. 3. We then present our main technical contributions in Sec. 4 and their 56 empirical evaluation in Sec. 5, before concluding in Sec. 6. 57

58

2 Related Literature

In contrast to post-hoc explanations of blackbox models [e.g., 28; 23], which are often unfaithful to 59 the original model [21; 24; 13], interpretable machine learning methods aim to produce intrinsically 60 intelligible, yet accurate, models. Additive models that compose terms in a simple summation are 61 particularly useful in this context, because of their modularity, i.e., the possibility to comprehend the 62 terms in isolation. As long as the individual terms are not too numerous and simulatable, i.e., their 63 output can be approximately computed by a human, the resulting model is highly interpretable. 64

Good examples for this are (generalized) linear models [GLMs, 19], or generalized additive mod65 els [GAMs, 12; 16]. However, they do not model variable interactions, at least not of higher order. 66 Conjunctive propositional rules, on the other hand, have this ability, explaining their longstanding 67 popularity in machine learning and related fields. Additive rule ensembles, which are closely related 68

a2

6

a3

92

d3

β191

to non-modular rule lists [e.g., 31; 22], thus provide a unique combination of interpretability and 69 predictive power. There is a wide range of algorithms for learning additive rule ensembles. One 70 approach is to generate a candidate set and then sub-selecting a rule ensemble, e.g. via sub-modular 71 optimization [14; 32], or-as in RuleFit [10] or SIRUS [2]-via a sparse linear model, which is 72 especially computationally inexpensive. However, these approaches are typically highly sensitive to 73 the randomness in the generation of the candidate set. Alternatively, finding an optimal rule ensemble 74 can be expressed as an integer program. Its relaxation as linear problem can then be solved via the 75 column generation framework [6; 30], making the problem tractable. This approach is robust and 76 flexible, but the full optimization remains computationally expensive. 77

78

79

80

81

82

83

84

85

86

87

88

89

90

Early approaches to additive rule ensemble learning that avoid initial candidate generation are based on the separate-and-conquer framework [11] and later on the original boosting algorithm [5; 17]. However, the first typically leads to non-modular rule lists and the second are designed for specific learning tasks only, typically classification. This problem is overcome with the gradient boosting framework [9], which generalizes the original AdaBoost algorithm [25] and allows fitting arbitrary differentiable loss functions. With this framework, rules are fitted stagewise based on their effect on the training loss when added to the ensemble [7; 8]. Extreme gradient boosting [4] increases the scalability of gradient boosting by avoiding numerical weight optimization. It is applicable whenever the loss function is twice differentiable. Fully-corrective boosting recalculate the weight of all weak learners after adding one weak learner into the ensemble model [26; 27]. It overcomes the drawback of the original gradient boosting algorithm that the weak learners are not changed after being generated. However, it is a high-level framework and does not solve the problem of how to select individual base learners.

91

3 Rule Boosting

An additive ensemble of k rules can be represented by Boolean query functions q 1 , . . . , q k and a 92 weight vector β = ( β 1 , . . . , β k ) T ∈ R k that jointly describe a function f ( x ) = ∑ k i =1 β i q i ( x ) , the 93 output of which can be mapped to the conditional distribution of a target variable Y | X = x . That is, 94 the queries define the rule antecedents (rule bodies), and the coefficients β define the rule consequents, 95 i.e., the output of rule i for input x ∈ R d is β i if x satisfies the antecedent, i.e., q i ( x ) = 1 (and 0 96 otherwise). Moreover, each query function q i : R d →{ 0 , 1 } is a conjunction of c i propositions , i.e., 97 q i ( x ) = p i, 1 ( x ) p i, 2 ( x ) . . . p i,c i ( x ) where the p i,j are typically a threshold function on an individual 98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

input variable, e.g., p i,j ( x ) = δ ( x l ≥ t ) . We denote the set of available propositions by P and the query language of all conjunctions that can be formed from P as Q .

We are concerned with two properties of an additive rule ensemble: its (empirical) prediction risk 1 R ( f ) = 1 n ∑ n i =1 l ( f ( x i ) , y i ) , measured by some positive loss function l ( f ( x ) , y ) averaged over a dataset { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , and its cognitive complexity C ( f ) = k + ∑ k i =1 c i , measuring the cognitive effort required to parse all rule consequents and antecedents. Here we consider loss functions that can be derived as negative log likelihood (or rather deviance function) when interpreting the rule ensemble output as natural parameter of an exponential family model of the target variable, which guarantees that the loss function is strictly convex and twice differentiable. Specifically, we consider the cases of squared loss l sqr ( f ( x i ) , y i ) = ( f ( x i ) -y i ) 2 , the logistic loss l log ( f ( x i ) , y i ) = log(1 + exp( -y i f ( x i ))) , and the Poisson loss l poi ( f ( x i ) , y i ) = log y i -f ( x i ) -y i +exp( f ( x i )) .

Gradient boosting Gradient boosting [9] is a 'stagewise' fitting scheme for additive models that, in our context, produces a sequence of rule ensembles f (0) , f (1) , . . . , f ( k ) such that f (0) ( x ) = 0 and, for t ∈ [1 , k ] , f ( t ) ( x ) = f ( t -1) ( x ) + β t q t ( x ) . Specifically, the term β t q t ( x ) is chosen to perform an approximate gradient descent with respect to the risk function R ( f ) = R ( f ) considered as a function of the model output vector f = ( f ( x 1 ) , . . . , f ( x n )) . The exact gradient descent update would be -α ∗ g where g is the gradient vector with components g i = ∂l ( f ( x i ) , y i ) /∂f ( x i ) and α ∗ is the step length that minimizes the empicical risk R ( f -α g ) . However, since in general there is no query q for which the output vector q = ( q ( x 1 ) , . . . , q ( x n )) is equal to the gradient g , the goal is to select q ∗ that best approximates g in the sense that it minimizes the squared projection error

This is achieved by choosing q t to maximize the standard gradient boosting objective [8] obj gb ( q ) = | q T g | / ∥ q ∥ and to find β t = arg min β ∈ R R ( f + β q t ) via a line search. Note that this β t is not generally equal to the minimizing β in (1), because the optimal update in direction q can be better than the best geometric approximation to the gradient descent update in direction q . A derivation of this objective function is the gradient sum objective [8; 26] obj gs ( q ) = | q T g | , which always selects more general rules than the gradient boosting objective [8, Thm. 1], however, typically at the expense of an increased risk per rule, because the correction of data points with large gradient elements has to be toned down to avoid over-correction of other selected data points with small gradient elements. Finally, an adaption of 'extreme gradient boosting' [4] to rule ensembles yields the extreme boosting objective [3] obj xgb ( q ) = ( q T g ) 2 / q T h where h = diag ( ∇ 2 f ( x ) R ( f )) is the diagonal vector of the risk Hessian again with respect to the output vector f . This approach starts from the second order approximation of R ( f + β q ) for which also yields a closed form weight update β t = -q T g / q T h . This approach is well-defined for our loss functions derived from exponential family response models, which guarantee defined and positive h . For the squared loss, it is equivalent to standard gradient boosting, because the second order approximation is exact for l sqr and h is constant.

Single rule optimization While the rule optimization literature can be neatly divided into heuristic (greedy / beam search) and exact branch-and-bound search, these approaches are actually closely related: they can both be described as traversing a lattice on the query language Q imposed by a specialization relation q ⪯ q ′ that holds if the propositions in q ′ are a superset of those mentioned in q , and q ′ thus logically implies q . The difference between the approaches is under what conditions they discard specializations of candidate queries and how those specializations are generated.

Here, we build on the branch-and-bound framework presented in Boley et al. [3] that allows to efficiently search for optimal conjunctive queries in a condensed search space, given that there is an admissible, effective, and efficiently computable bounding function for the employed objective. Specifically, let obj : Q → R denote the objective function to be maximized. Then a bounding function bnd : Q → R is admissible if for all q ∈ Q it holds that bnd( q ) ≥ bst( q ) where bst( q ) = max q ⪯ q ′ ∈Q obj( q ′ ) denotes the objective value of the best specialization of q . A bounding function is effective in allowing to prune the search space if the difference bnd( q ) -bst( q ) tends

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

to be small, rendering bst( q ) itself the theoretically most effective bounding function. However, computing bst( q ) is as hard as the overall optimization problem.

Afrequently applied recipe for constructing an admissible bounding function that is also effective and efficiently computable is to relax the quantifier in the definition of bst and instead of bounding the value of the best specialization in the search space, bound the value of the best subset of data points of those selected by q [20]. This results in the tight bounding function when unaware of selectability

Here, q ≤ q ′ refers to the component-wise less or equals relation on the binary output vector of q and q ′ . This function can be efficiently computed for many objective functions by pre-sorting the data in time O ( n log n ) that has to be carried out only once per fitted rule [15]. For instance for the extreme boosting objective, the optimum q ′ ∈ { 0 , 1 } n can be found as a prefix or suffix of all data points after sorting them according to the ratio g i /h i of first and second order loss derivatives.

4 Fully-corrective Orthogonal Gradient Boosting

Having reviewed the existing rule boosting approaches, we now turn to improving them in terms of their risk/complexity trade-off. Our approach to this is to improve the risk reduction per rule added to the ensemble, which directly affects the number of rules needed to achieve a certain risk. As it turns out, this typically coincides with preferring the addition of more general and hence simpler rules. Thus, it also positively affects the cognitive complexity on the level of the lengths of individual rules.

4.1 Weight Correction and Subspace Approximations

A natural idea to reduce the ensemble risk per rule added is to relax the strict stagewise fitting approach of traditional gradient boosting and to allow the whole weight vector β to be adjusted in every round t , i.e., to set

where Q t = [ q 1 , . . . , q t ] is the n × t query matrix with the output vectors of all selected queries as columns. In contrast to a full joint optimization of queries and weights, this intermediate solution still retains the computational benefits of gradient boosting for small rule ensembles: Given that our loss function l and therefore the empirical risk R are convex, optimizing the weights in round t is a convex optimization problem in t variables, equivalent to fitting a small generalized linear model. Using Newton-Raphson ('iterated least squares') type algorithms, the computational cost of this is usually negligible compared to the more expensive query optimization step, especially when aiming for optimal individual queries for their reduced cognitive complexity.

Re-optimizing the weights, which is sometimes referred to as fully corrective boosting [26], effectively turns boosting into a form of forward variable selection for linear models. However, in contrast to conventional variable selection where all variables are given explicitly, we still have to identify a good query q t in each boosting iteration, and it turns out that finding the appropriate query is more complicated as in the case of single weight optimization characterized by (1). We still would like to add the direction of steepest descent, i.e., the negative gradient -g , to the subsequent risk optimization step and approximate as closely as possible the outcome [ Q t -1 ; g ] α ∗ where α ∗ ∈ R t are the risk minimizing weights for q 1 , . . . , q t -1 , g . Therefore, the best approximating query q is now given by

It is an important observation that the standard gradient boosting objective does not correctly identify this optimally approximating query. This is demonstrated by the example illustrated in Fig. 2. In the left sub-figure it can be seen that the green plane, span { q 1 , q 3 } , is a better approximation to span { q 1 , -g } (blue) than the orange plain, span { q 1 , q 2 } . However, the latter is selected by standard gradient boosting, because the angle between q 2 and -g is smaller than that between q 3 and -g .

4.2 An Objective Function to Identify the Best Approximating Subspace

The intuitive reason for the gradient boosting objective failing to identify the correct query in Fig. 2 is that selecting x 2 in addition to x 3 is not beneficial for the overall risk reduction if we are only

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

allowed to set the weight for the newly selected query. This is because then this weight has to be a compromise between the two different magnitudes of correction required for x 2 , which only needs a small positive correction, and x 3 , which needs a large positive correction. If we, however, are allowed to change the weight of the previously selected query this consideration changes, because we can now balance an over-correction for x 2 by adjusting the weight of the first rule. While on first glance it seems unclear how much of such re-balancing can be applied without harming the overall risk, it turns out that this is captured by a simple criterion based on the norm of the part of the newly selected query that is orthogonal to the already selected ones.

where for a vector v ∈ R n we denote by v ⊥ its projection onto the orthogonal complement of range Q . (All proofs of lemmas and theorems are in SI [1])

From this result we can directly derive | g T ⊥ q | / ∥ q ⊥ ∥ as suitable objective function for fully corrective gradient boosting. However, it is worth incorporating two further observations. Firstly, we can show that, after applying the weight correction step (3), the gradient vector satisfies g = g ⊥ , i.e., it is orthogonal to the subspace spanned by the selected queries, and therefore can be used in the objective function without projection.

Lemma 4.2. Let g be the gradient vector after the application of the weight correction step (3) for selected queries q 1 , . . . , q t . Then g ⊥ span { q 1 , . . . , q t } .

Moreover, the right hand side of Eq. (4) is technically undefined for redundant query vectors q that lie in range Q and therefore have ∥ q ⊥ ∥ = 0 . Through Lm. 4.2 we know that for such queries we also have g T q = 0 , which suggests to simply fix this issue by defining the objective value in this case to be 0 . However, this solution would not fix numerical instabilities when ∥ q ⊥ ∥ is close to zero. A better solution is therefore to add a small positive value ϵ to the denominator, which can be considered a regularization parameter . With this we arrive at the final form of our objective function, which we state along with some of its basic properties in the following theorem.

Theorem 4.3. Let Q = [ q 1 , . . . , q t -1 ] ∈ R n × ( t -1) be the selected query matrix and g the corresponding gradient vector after full weight correction, and let us denote by q = q ⊥ + q ∥ the orthogonal decomposition of q with respect to range Q . Then we have for a maximizer q ∗ of the orthogonal gradient boosting objective obj ogb ( q ) = | g T q | / ( ∥ q ⊥ ∥ + ϵ ) :

Intuitively, the orthogonal gradient boosting objective function measures the cosine of the angle between the gradient vector and the orthogonal projection of a candidate query vector q . This is in contrast to the standard gradient boosting objective, which considers the angle of the unprojected query vector instead. In the example in Fig. 2 we can observe that this difference leads to successfully identifying the best approximating subspace, and Thm. 4.3a) guarantees this property.

4.3 Efficient Implementation

To develop an efficient optimization algorithm for the orthogonal gradient boosting objective, we recall that projections q ⊥ on the orthogonal complement of range Q can be naively computed via q ⊥ = Q (( Q T Q ) -1 ( Q T q )) where we placed the parentheses to emphasize that only matrix-vector products are involved in the computation-at least once the inverse of the Gram matrix Q T Q is computed. This approach allows to compute projections, and thus objective values, in time O ( nt + t 2 ) per candidate query after an initial preprocessing per boosting round of cost O ( t 2 n + t 3 ) .

In a first step, this naive approach can be improved by maintaining an orthonormal basis of the range of the query matrix throughout the boosting rounds, resulting in a Gram-Schmidt-type procedure.

240

241

242

243

244

245

246

247

248

249

250

Algorithm 1 Fully-corrective Orthogonal Gradient Boosting

Since the projections q t ⊥ of the selected queries naturally form an orthogonal basis of range Q this only requires normalization, which can be done essentially without additional cost. Formally, by storing o t = q t ⊥ / ∥ q t ⊥ ∥ in all boosting rounds t , subsequent projections can be computed via q ⊥ = q -O ( O T q ) where O = [ o 1 , . . . , o t ] . This reduces the computational complexity per candidate query to O ( tn ) without requiring any additional preprocessing.

While this looks like the optimal complexity for evaluating obj ogb in isolation, it leads to a prohibitive complexity for large n for finding the optimal query in a given round. Specifically, branch-and-bound with the tight bounding function (2) evaluates O ( n ) queries per expanded search node and beam search even O ( d 2 n ) . In both cases, using the expression for q ⊥ above repeatedly results in a quadratic cost in n per search node. To circumvent this, we have to exploit the structure of candidate evaluations, similar to other efficient implementations of rule and tree learning algorithms [18].

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

The candidates evaluated per search node of both branch-and-bound and beam search have query vectors that are prefixes of an ordered sub-selection of data points, in beam search because optimum cut-off values are sought for each of the d input variables, in branch-and-bound because the optimum in Eq. (2) is attained or approximated by a prefix with respect to some presorting order. Hence, we need to solve the following prefix optimization problem : given an ordered sub-selection of l of the n data points, represented by an injective map σ : { 1 , . . . , l } → { 1 , . . . , n } , find the optimal prefix

where q (0) = 0 and q ( i ) = q ( i -1) + e σ ( i ) . The following proof shows how the computational complexity for solving (6) can be substantially reduced compared to the direct approach above. It uses an incremental computation of projections that works directly on the available orthonormal basis vectors o instead of computing matrix-vector products or, even worse, the whole projection matrix.

Theorem 4.4. Given a gradient vector g ∈ R n , an orthonormal basis o 1 , . . . , o t ∈ R n of the subspace spanned by the queries of the first t rules, and a sub-selection of l candidate points σ : [ l ] → [ n ] , the best prefix selection problem (6) can be solved in time O ( tn ) .

Proof sketch. We write the objective value of prefix i in terms of incrementally computable quantities:

In particular, the t sequences of norms ∥ o k o T k q ( i ) ∥ can be computed in time O ( n ) via cumulative summation of the k -th basis vector elements in the order given by σ :

We close this section with a pseudocode (Alg. 1) that summarizes the main ideas of the orthogonal gradient boosting and refer to the literature for details about the beam-search/branch-and-bound step.

5 Experiments

In this section, we present empirical results comparing the proposed fully corrective orthogonal gradient boosting (FCOGB) to the standard gradient boosting algorithms [8] using greedy optimization of obj gb (GB) and obj gs (GS), to extreme gradient booosting [3] using branch-and-bound optimisation of obj xgb (XGB), and finally to SIRUS [2] as the state-of-the-art generate-and-filter approach. We investigate the risk/complexity trade-off, the affinity to select general rules, as well as the computational complexity. The datasets used are those of Boley et al. [3] augmented by three additional classification datasets from the UCI machine learning repository and, to introduce a novel modelling task to the rule learning literature, five counting regression datasets from public sources. This results in a total of 34 datasets (13 for classification, 16 for regression, and 5 for counting/Poisson regression, see Tab. 1). All algorithms were run five times on all datasets using 5 random 80/20 train/test splits to calculate robust estimates of all considered metrics. In all cases, the number of gradient boosting iterations was chosen to produce ensembles with cognitive complexity of at most 50. The experiment code and further information about the datasets are available on GitHub[1].

Cognitive complexity versus risk Tab. 1 compares the complexity/risk trade-off of the boosting variants and SIRUS by the normalized risk averaged across all considered cognitive complexity levels (where normalization is performed by the risk of the empty rule ensemble). FCOGB has the smallest training risk for 26 of the 34 datasets, occasionally outperforming the second best algorithm by a wide margin ( tic-tac-toe , wine , banknote , insurance , boston , ships , smoking ). For the test risk the picture is more ambiguous, however, FCOGB retains a relative majority of datasets won. Performing one-sided paired t-tests at significance level 0.05 (with Bonferroni correction for 8 hypotheses) reveals that FCOGB significantly outperforms all other variants with a margin of at least 0.001 average normalized training risk (while there is no significant winner in terms of test risk-likely due to a lack of regularization for larger ensemble sizes).

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

0.0

0.5

1.0

Coverage Rate Gradient Boosting

0.0

0.2

0.4

0.6

0.8

1.0

Coverage Rate FCOGB

0.0

0.5

1.0

Coverage Rate XGBoosting

0.0

0.5

1.0

Coverage Rate Gradient Sum

0

10

20

30

40

50

cognitive complexity

0

100

200

300

400

500

600

700

800

900

time (seconds)

breast

Gradient Sum

Gradient Boosting

XGBoost

Naive FCOGB

FCOGB

0

10

20

30

40

50

cognitive complexity

0

2

4

6

8

10

12

diabetes

Coverage To compare the generality of the rules learned by the new objective function in comparison to the existing ones, we performed an additional experiment where we first used one of the previous objective functions to generate rule ensembles with ten rules for all datasets. Then for each partial rule ensemble, we applied the orthogonal gradient boosting objective function to find an alternative rule. Importantly, we used branch-and-bound with admissible bounding functions for all the alternative objectives to avoid confounding through sub-optimal greedy solutions. In Fig. 3 we compare the relative coverage, i.e., the relative number of selected data points ∥ q ∥ 2 /n , of the rules discovered by the original algorithms to the ones discovered by FCOGB. The outcome is that 81.1% of the FCOGB rules covers more data points than gradient boosting, and similarly 71.3% of its rules cover more data points than those generated by XGBoost. In contrast, only 47.2% of the FCOGB rules cover more datapoint than the ones discovered by gradient sum. These results are in alignment with the theoretical expectation in terms of the influence of the coverage on the objective values where gradient sum is completely unaffected, whereas orthogonal gradient boosting has a denominator that tends to grow with coverage albeit less than the one of gradient boosting.

Computation time We also compare the computational cost of generating rule ensembles with cognitive complexity 50 by different algorithms in Tab 1. Comparing the computational cost of FCOGB to XGB, the other algorithm utilizing the more expensive branch-and-bound search, the costs are in the same order of magnitude except for one extreme case (wine) where FCOGB is a factor of 26 slower. Comparing to the two greedy variants, FCOGB is in the same order of magnitude as gradient boosting for most datasets. Unsuprisingly, there are a few examples where greedy search vastly outperforms branch-and-bound, in one case (telco churn) by a factor of around 76. However, overall, the results confirm that branch-and-bound search is a practical algorithm in absolute terms: For 23 benchmark dataset, FCOGB is able to finish training a model of cognitive complexity of 50 within one minute. Most of the other experiments run within 15 minutes except one dataset (telco churn) which require longer running time. Finally, Fig. 3 shows the detailed computation time of all algorithms in terms of cognitive complexity, including the naive implementation of FCOGB for breast cancer and diabetes , which shows that the performance improvement through Thm. 4.4 is critical to bring the computational complexity on par with XGB.

6 Conclusion

The proposed fully corrective orthogonal boosting approach is a worthwhile alternative to previously 323 published boosting variants for rule learning, especially when targeting a beneficial risk-complexity 324 trade-off and an overall small number of rules. The present work provided a relatively detailed 325 theoretical analysis of the newly developed rule objective function. However, some interesting 326 questions were left open. While the presorting-based approach to the bounding function performs 327 extremely well in synthetic experiments, a theoretical approximation guarantee for this algorithm has 328 yet to be derived. Another interesting direction for future work is the extension of the introduced 329 approximating subspace paradigm to the extreme gradient boosting approach, which, due to the 330 utilization of higher order information, should principally be able to produce even better risk331 complexity trade-offs. 332

333

References

Anonymous. Fully-corrective orthogonal gradient boosting: Code, datasets, and supplementary 334 information. https://anonymous.4open.science/r/FCOGB-BF6D , 2023. 335

C. Bénard, G. Biau, S. Da Veiga, and E. Scornet. Interpretable random forests via rule extraction. 336 In International Conference on Artificial Intelligence and Statistics , pages 937-945. PMLR, 337 2021. 338

M. Boley, S. Teshuva, P. L. Bodic, and G. I. Webb. Better short than greedy: Interpretable models 339 through optimal rule boosting. In Proceedings of the 2021 SIAM International Conference on 340 Data Mining (SDM) , pages 351-359. SIAM, 2021. 341

T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd 342 acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 343 2016. 344

W. W. Cohen and Y. Singer. A simple, fast, and effective rule learner. AAAI/IAAI , 99(335-342): 345 3, 1999. 346

S. Dash, O. Gunluk, and D. Wei. Boolean decision rules via column generation. Advances in 347 neural information processing systems , 31, 2018. 348

K. Dembczy´ nski, W. Kotłowski, and R. Słowi´ nski. Maximum likelihood rule ensembles. In 349 Proceedings of the 25th international conference on Machine learning , pages 224-231, 2008. 350

351

352

K. Dembczy´ nski, W. Kotłowski, and R. Słowi´ nski. Ender: a statistical framework for boosting decision rules. Data Mining and Knowledge Discovery , 21(1):52-90, 2010.

353

354

355

356

357

358

J. H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001.

J. H. Friedman and B. E. Popescu. Predictive learning via rule ensembles. The annals of applied statistics , pages 916-954, 2008.

J. Fürnkranz. Separate-and-conquer rule learning. Artificial Intelligence Review , 13(1):3-54, 1999.

T. Hastie and R. Tibshirani. Generalized Additive Models . Routledge, 1990. ISBN 359 9780203753781. 360

I. Kumar, C. Scheidegger, S. Venkatasubramanian, and S. Friedler. Shapley residuals: Quantify361 ing the limits of the shapley value for explanations. Advances in Neural Information Processing 362 Systems , 34:26598-26608, 2021. 363

H. Lakkaraju, S. H. Bach, and J. Leskovec. Interpretable decision sets: A joint framework for 364 description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference 365 on knowledge discovery and data mining , pages 1675-1684, 2016. 366

F. Lemmerich, M. Atzmueller, and F. Puppe. Fast exhaustive subgroup discovery with numerical 367 target concepts. Data Mining and Knowledge Discovery , 30:711-762, 2016. 368

Y. Lou, R. Caruana, J. Gehrke, and G. Hooker. Accurate intelligible models with pairwise 369 interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge 370 discovery and data mining , pages 623-631, 2013. 371

D. Malioutov and K. Varshney. Exact rule learning via boolean compressed sensing. In 372 International conference on machine learning , pages 765-773. PMLR, 2013. 373

M. Mampaey, S. Nijssen, A. Feelders, R. Konijn, and A. Knobbe. Efficient algorithms for finding 374 optimal binary features in numeric and nominal labeled data. Knowledge and Information 375 Systems , 42:465-492, 2015. 376

P. McCullagh and J. A. Nelder. Generalized linear models . Routledge, 2019. 377

S. Morishita and J. Sese. Transversing itemset lattices with statistical metric pruning. In 378 Proceedings of the nineteenth ACM SIGMOD-SIGACT-SIGART symposium on Principles of 379 database systems , pages 226-236, 2000. 380

381

382

W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Interpretable machine learning: definitions, methods, and applications. arXiv preprint arXiv:1901.04592 , 2019.

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

H. M. Proença and M. van Leeuwen. Interpretable multiclass classification by mdl-based rule lists. Information Sciences , 512:1372-1393, 2020.

M. T. Ribeiro, S. Singh, and C. Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence , 1(5):206-215, 2019.

R. E. Schapire and Y. Freund. Boosting: Foundations and algorithms. Kybernetes , 42(1): 164-166, 2013.

S. Shalev-Shwartz, N. Srebro, and T. Zhang. Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM Journal on Optimization , 20(6):2807-2832, 2010.

C. Shen and H. Li. On the dual formulation of boosting algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence , 32(12):2216-2231, 2010.

E. Strumbelj and I. Kononenko. An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research , 11:1-18, 2010.

T. Wang, C. Rudin, F. Doshi-Velez, Y. Liu, E. Klampfl, and P. MacNeille. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research , 18(1):2357-2393, 2017.

D. Wei, S. Dash, T. Gao, and O. Gunluk. Generalized linear rule models. In International Conference on Machine Learning , pages 6687-6696. PMLR, 2019.

H. Yang, C. Rudin, and M. Seltzer. Scalable bayesian rule lists. In International conference on machine learning , pages 3921-3930. PMLR, 2017.

G. Zhang and A. Gionis. Diverse rule sets. In Proceedings of the 26th ACM SIGKDD 405 International Conference on Knowledge Discovery & Data Mining , pages 1532-1541, 2020. 406

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a gradient-boosting algorithmic framework for rule ensemble learning, emphasizing the interpretability of produced rule set. Various gradient-boosting algorithms are reviewed in the rule-learning context, and the authors argue that a specific boosting algorithm, called fully corrective orthogonal gradient boosting (FCOGB), is particularly suited for rule boosting. The intuition is that existing additive rule-boosting procedures operate in a strictly greedy fashion - the weight of each added rule is fixed in later iterations. In contrast, FCOGB allows the weights of preceding rules to be adjusted in each later iteration, which may help to reduce the number of required rules (to reach a certain accuracy) and thus the cognitive complexity of the final rule set. Based on FCOGB, the authors derive the stepwise boosting objective function for single rule search, which is similar to existing gradient boosting objectives but with a different regularization term. The overall algorithm looks like the conjugate gradient method - in each iteration, the rule aligning best with the gradient in the orthogonal complement of the subspace spanned by previous rules is added. The authors demonstrate the effectiveness of FCOGB through experimental comparison with existing rule-boosting algorithms on classification, regression, and Poisson regression tasks.\n\nSTRENGTHS:\n- Applying FCOGB to rule learning, to the best of my knowledge, is a novel idea, and the authors provide a comprehensible justification for this choice. Figure 2 is helpful in understanding the difference between FCOGB and existing rule-boosting algorithms.\n- How to search the optimal rule in each iteration is especially considered, which is a key step in rule boosting. The authors propose a strategy that exploits the nice structure in the boosting objective function to speed up the bound calculation in branch-and-bound search of optimal rules.\n- The proposed algorithm is evaluated on a wide range of datasets and tasks. The authors provide a detailed analysis of the results. Figure 1 clearly shows that FCOGB can achieve a better accuracy-risk trade-off than existing rule-boosting algorithms.\n\nWEAKNESSES:\n- The presentation of the paper can be improved. For example:\n  + In the \"Rule Boosting\" section, the \"Gradient boosting\" subsection mixes the description of general gradient boosting and the more specific rule boosting. This makes it hard to understand these objectives for readers who are not familiar with the rule-boosting literature. For example, obj_gb(q) = |g^T q|/||q|| is nonstandard in the general gradient boosting literature. It would be better to separate the general gradient boosting and rule boosting parts.\n  + The \"Single rule optimization\" subsection assumes too much prior knowledge about the rule learning literature. I would suggest the authors merge this subsection with the \"4.3 Efficient Implementation\" subsection to make the paper more fluent and self-contained.\n  + The authors should provide more details about the proposed algorithm, especially the BnB/beam search of a single rule.\n- Lack of comparison with rule induction algorithms based on column generation, e.g., [30] and [b]. In the column generation approach, the weights of all added rules are also adjusted in each iteration when solving the restricted master problem, which is similar to FCOGB. I am interested in how FCOGB compares with this approach experimentally.\n- The presentation of the prefix optimization problem is misleading. The authors claim that \"This function can be efficiently computed for many objective functions by pre-sorting the data in time O(n log n)\" in Section 3, but is this true for the objective function obj_{ogb}(q)?  The authors should clarify this point. I cannot immediately see how the optimal solution to (2) under this objective function is contained in the prefix of the data sorted by some (what?) criterion. If this is true, the authors should provide a proof or a reference to support this claim.\n- There is a mistake in Lines 233-234.\n- Missing references:\n  + [a] Jonathan Eckstein, Noam Goldberg. An Improved Branch-and-Bound Method for Maximum Monomial Agreement. INFORMS Journal on Computing, 2012.\n  + [b] Jonathan Eckstein, Ai Kagawa, Noam Goldberg. REPR: Rule-Enhanced Penalized Regression. INFORMS Journal on Optimization, 2019.\n  + [c] Fan Yang, et al. Learning Interpretable Decision Rule Sets: A Submodular Optimization Approach. NeurIPS 2021.\n\nQUESTIONS:\n- The test accuracy of FCOGB is not generally better than that of existing rule-boosting algorithms. Can this be explored in more detail?\n- How the averaged, normalised training risks in Table 1 are computed? An exact formula would be helpful.\n- It is well-known in recent rule learning literature ([6] and [c]) that the tic-tac-toe dataset can be perfectly learned by eight 3-CNF rules. Can FCOGB learn the perfect rule set on this dataset?\n\nLIMITATIONS:\nThe limitations of this work are not explicitly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to gradient boosting of decision rules for interpretable machine learning models. By incorporating a\nweight correction step and orthogonal projections, the method maximizes predictive gain per rule.\nTheir experimental evaluation on various classification, regression, and Poisson regression tasks confirms that the resulting rule learner\nenhances the trade-off between comprehensibility and accuracy in the fitted ensemble. Moreover, it maintains a comparable computational\ncost to previous branch-and-bound rule learners.\n\nSTRENGTHS:\n1. Originality: The paper introduces the first rule boosting algorithm that consistently optimizes the accuracy/complexity trade-off of produced rule sets. This represents a novel contribution to the field.\n2. Quality: The research exhibits high quality as it adopts the fully corrective boosting approach, which entails re-optimizing all rule consequents in each boosting round. The study's rigorous algorithm development provides a strong foundation for the research, ensuring\nthe reliability and robustness of the findings.\n3. Clarity: The paper explains the new objective function for selecting individual rule bodies, the corresponding efficient algorithm for cutpoint\nsearch along with some other algorithm details. The clear explanations contribute to the overall clarity of the research.\n4. Significance: The research demonstrates significant improvements over previous boosting variants in terms of the risk/complexity tradeoff.\nThe better risk reduction per rule and the affinity to select simpler rules contribute to the overall significance of the findings. Additionally,\nthe comparable computational cost to previous approaches adds to the practical relevance of the research.\n\nWEAKNESSES:\nIn terms of the compared established methods, SIRUS [1] is the most recent work included in the analysis, published in 2021. However, it is\nworth noting that some more recent publications, such as [2,3], are not included in the experiment section.\nOne limitation of the paper's presentation is the heavy reliance on text and equations, with less emphasis on the use of figures and intuitive\nexample case studies. This approach may hinder the reader's ability to grasp complex concepts and visualize the practical applications of\nthe proposed methods. Incorporating more visual aids, such as figures and illustrative examples, could enhance the clarity and accessibility\nof the research.\n[1] C. Bénard, G. Biau, S. Da Veiga, and E. Scornet. Interpretable random forests via rule extraction. In International Conference on Artificial\nIntelligence and Statistics, pages 937–945. PMLR, 2021.\n[2] Souza V F, Cicalese F, Laber E, et al. Decision Trees with Short Explainable Rules[J]. Advances in Neural Information Processing\nSystems, 2022, 35: 12365-12379.\n[3] Calzavara S, Cazzaro L, Lucchese C, et al. Explainable Global Fairness Verification of Tree-Based Classifiers[C]//2023 IEEE Conference\non Secure and Trustworthy Machine Learning (SaTML). IEEE, 2023: 1-17.\n\nQUESTIONS:\nCould you provide some insights into the reason why certain newly published works, such as [1,2], were not included in the experiment\nsection? Were there any specific criteria or limitations that influenced the selection of methods for comparison?\n[1] Souza V F, Cicalese F, Laber E, et al. Decision Trees with Short Explainable Rules[J]. Advances in Neural Information Processing\nSystems, 2022, 35: 12365-12379.\n[2] Calzavara S, Cazzaro L, Lucchese C, et al. Explainable Global Fairness Verification of Tree-Based Classifiers[C]//2023 IEEE Conference\non Secure and Trustworthy Machine Learning (SaTML). IEEE, 2023: 1-17.\n\nLIMITATIONS:\nA limitation of the study is that while it includes the most recent work, SIRUS [1], which was published in 2021, it does not incorporate some\nmore recent publications like [2,3] in the experiment section. This omission limits the comprehensiveness of the analysis and may overlook\npotential advancements or alternative approaches introduced in these newer works.\n[1] C. Bénard, G. Biau, S. Da Veiga, and E. Scornet. Interpretable random forests via rule extraction. In International Conference on Artificial\nIntelligence and Statistics, pages 937–945. PMLR, 2021.\n[2] Souza V F, Cicalese F, Laber E, et al. Decision Trees with Short Explainable Rules[J]. Advances in Neural Information Processing\nSystems, 2022, 35: 12365-12379.\n[3] Calzavara S, Cazzaro L, Lucchese C, et al. Explainable Global Fairness Verification of Tree-Based Classifiers[C]//2023 IEEE Conference\non Secure and Trustworthy Machine Learning (SaTML). IEEE, 2023: 1-17.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new algorithm for learning rule ensembles and claims that these are interpretable, but does not present any support.\n\nSTRENGTHS:\nThe proposed method is reasonable, but, in the context of other work in this area, not ground-shaking. The experimental evaluation is done well, but does not touch on interpretability.\n\nWEAKNESSES:\nThere is no evidence that the learned rule sets are interpretable.\nRule complexity has not much to do with cognitive complexity.\nThe efficiency of algorithm is over-stated in the paper.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThis is another paper that claims that rule ensembles are interpretable. No evidence is presented to that end, the claim is just derived from the fact that rules are, by themselves, interpretable. However, for example, random forests are well-known to be not interpretable, and they are also just a rule ensemble. In addition, the situation is even worse here, because in a random forest at least each individual rule is interpretable, and may be viewed as an explanation for all the examples it covers. In an additive boosting setting, this property also does not hold, because each rule corrects and refines predictions of previous rules, so rules can no longer be interpreted in isolation, but only in the context of all previous rules. Even a single example can not be easily explained by a gradient-boosted rule set, because one would have to understand the interaction of multiple rules.\n\nThe authors use the term \"cognitive complexity\" for something that is essentially the size of a rule set. Again, this is a complete misnomer, as the cognitive effort to parse a rule set does not only depend on the size of the theory. As explained above, there might be dependencies between rules, or rules may be considered in isolation (the latter having a much lower cognitive complexity). There are also factors such as the familiarity with the used concepts. For example, the cognitive effort required to read a page of text in your mother tongue is much lower than the cognitive effort required to read a page in a language that you are just learning, even though both, the content, as well as the syntactic length (essentially the author's measure of cognitive complexity) is the same.\n\nIt is a pity that they authors make such unfounded claims about intepretability, where they could simply present their work as an attempt to learn a simpler rule ensemble. As such, the work is reasonable, but also not great break-through. What they essentially propose (following previous work) is to re-optimize all weights once a new rule is added, and build an efficient algorithm around that idea. It gains a little in performance, as can be expected, but it is not great break-through.\n\nThe small advantage seems to be bought with an increase in computation time, which the authors interpret as \"in the same order of magnitude\" except for one case, where it is by a factor 26 slower. Actually, it seems to be the case that in most of the datasets, the algorithm is at least a factor of 2 smaller, sometimes worse. \n\nMinor comments:\n\nSome of the numbers in Table 1 are obviously wrong (e.g., testing risks of 109.5 or 4.115 for XGB).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Fully-Corrective Orthogonal Gradient Boosting (FCOGB), a novel algorithm aimed at facilitating interpretable rule learning. The study contends that existing rule learning algorithms often yield complex models that pose challenges for interpretation. FCOGB addresses this concern by generating simpler and more easily understandable models.\nFCOGB is an extension of the widely employed gradient boosting algorithm, utilized for constructing predictive models. It employs a branch-and-bound search algorithm to identify the optimal set of rules that minimize prediction errors.\n\nSTRENGTHS:\nThe proposed method is supported by theoretical justifications and intuitive explanations using figures. Additionally, the paper proposes algorithms with computational complexity analysis to efficiently implement the method, demonstrating practical applicability.\n\nWEAKNESSES:\nDespite an increase in the required training time (takes several times longer computation), the generalization performance does not improve. If this weakness is addressed, I believe it would become a very strong paper.\n\n(Minor comments)\n- Despite Figure 2 being referenced on page 5, the figure is actually inserted on page 3.\n- The scatters plot in Figure 3 are difficult to interpret due to overlapping points. Please set alpha (transparency).\n\nQUESTIONS:\n- Table 1 evaluates the risk for cognitive complexity values ranging from 1 to 50 and provides the averaged risk. However, I'm wondering if there are any trends or patterns in the variations. For example, do the superior algorithms change when cognitive complexity is fixed at 5 compared to when it is fixed at 25? Additionally, if the range is changed to 1 to 25 or extended to 1 to 100, would the results be different? I believe this perspective would enhance the sense of conviction, and it might provide an approach for improving test error in your method.\n\n- Are there any test error improvement strategies that can be considered? Otherwise, from the perspective of interpretability, are there cases where it is desirable for only the training error to decrease without an improvement in generalization performance (overfitting)?\n\n- While SIRUS is presented as state-of-the-art, as seen in Figure 1, its performance looks unsuitable as a comparative reference. Do you have any idea why SIRUS does not work well in this example?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework of fully corrective orthogonal boosting. The main algorithmic difference here is the objective for each next weak model. It is the cosine between the gradient (which is orthogonal to previous weak learners by construction) and the part of the new model that is orthogonal to previous models. Authors motivate their work by the need of interpretable models, so they restrict themselves to the case of rules as weak learners. Also, they use a variant of b&b algorithm for optimal weak learner search instead of commonly used greedy construction in depth. \n\nAuthors claim that this the paper proposes an algorithm for constructing shorter and more interpretable rules for Gradient Boosting of Decision Rules model. Experiments show that described method outperforms standard implementations of GB in case of using models of low complexity.\n\nAlthough theoretical part is sound, practical questions are not thoroughly addressed or answered.\n\nSTRENGTHS:\nThe main part of the paper is well-written, the terms, designations and ideas are clear. The proposed method is sound, reasonable and well described. The idea of orthogonal rule search in conjunction with fully-corrective goosting looks good. The theoretical part is described very well, the main formulations are correct, and the obtained contributions look important and are novel to the best of my knowledge. The proposed algorithm is justified and has the potential to compete with SOTA in the outlined formulation that refer to \"cognitive complexity\".\n\nThe main part of the paper is well-written. The terms, designations, and ideas are clear. \n\nThe only point I did not buy is the Poisson loss defined in line 109, in my opinion, incorrectly (or unclear), because, formally, from that definition, its minimum is at $f(x_i)=0$ independently on $y_i$.\n\nWEAKNESSES:\nI have the following concerns about the research direction itself. Claimed advantage of ensembles of rules over ensembles of trees is their human interpretability. However, I cannot agree that the decisions a rule ensemble makes can be treated as interpretable. Particularly, I argue that in the domains where interpretation is important summation of even two terms is usually not interpretable for humans. Most critical decisions in such domains like medicine and justice, partly science and risk management are usually based on several binary factors, not a sum of dozens of rules. Where ensembles of rules are really used in practice?\n\nSecond, I am disappointed that the term \"cognitive complexity\" was left without any background. I would expect references to some papers using this metric or explicit statement that this way to estimate models' complexity is originally proposed in the current paper. Futhermore, I would expect some consideration of actual research in psychology domain that address the problem of cognitive complexity of calculations.\n\nFor example, we can see in \"Human knowledge models: Learning applied knowledge from the data.\" Plos one, 2022, by E. Dudyrev et al., that a human decision is usually based on:\n-\tBoolean operators: OR, AND, NOT, and thresholded Boolean SUM (arithmetic sum of\nBoolean variables, compared to an integer threshold)\n-\tAt most four (Boolean) variables, where each variable is used at most once\\\n\nThese ideas are rather far from the concept of sums of dozens of rules\n\nSee also:\n\nLemonidis C., “Mental Computation and Estimation: Implications for mathematics education research, teaching and learning”, 2015,\n\nMarois R et al, \"Capacity limits of information processing in the brain,\" Trends in cognitive sciences, vol. 9, no. 6, pp. 296–305, 2005\n\nNys J. et al, \"Complex Mental Arithmetic: The Contribution of the Number Sense,\" Canadian journal of experimental psychology, vol. 64, no. 3, pp. 215–220, 2010.\n\nAt last, but not least, the experimental part spoils the impression of the work and requires improvements:\n\n- First of all, I see no hyperparameter tuning step description (e.g. regularization terms for XGB, number of boosting rounds, length of decision rules) in the section on experiments. Are there any hyperparameters which may have a significant impact on the performance of FCOGB? Where they left \"defaulted\" or were they were tuned by a separate step of an algorithm?\n\n- In the beginning of Section 5, it is mentioned that you use only 5 runs for each dataset with < 50 cognitive complexity (CC) limit. But then I see averaging over complexities between 1 and 50 in the description. What does it mean? I suppose that CC may alter in different runs but it is limited to 50, is it true? Or did authors perform exhaustive search of all possible CCs and averaged over them? If the first is true, I have a doubt that different model may have had different mean CC values, so that it is not quite fair comparison results. Is the second is true, then it is unclear why such an averaging can prove something\n\n- It would be interesting to see the dynamics of quality with respect to increasing CC. In particular, some graphs that plots quality vs CC to see which algorithm uses the CC limit more effectively.\n\n- It may be useful to provide comparison with other variants of fully-corrective boosting implementations since the quality gain may origin from described by this scheme only\n\n- Time limitations should be discussed more in terms of time per CC point and pareto curves (time to achieve the desired quality)\n\n- How should we interpret relatively low quality for regression problems?\n\n- This paper addresses interpretability of trained decision rules, so it would be profitable to demonstrate a difference in the simplicity of interpretation for FCOGB rules and, e.g., XGB rules\n\nQUESTIONS:\n- Equation (4) goes far from original GB. The point $(Q_{t-1};g) \\alpha^*$ in the functional space is used as a target for the next step of rule search. You can go one step further this way: use the strict minimizer $f^*$ of the loss functional as a target instead of $(Q_{t-1};g) \\alpha^*$. This looks a simpler choice, which can also increase boosting convergence and thus simplicity of the trained ensemble. Did you consider this way?\n\n- How b&b approach to rule search matches previous literature on rule gradient boosting?\n\nLIMITATIONS:\nI do not see any particular limitation of the proposed work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}