Title: Linear Mode Connectivity in Differentiable Tree Ensembles
Venue: NeurIPS 2024
Accepted: no
Keywords: Linear Mode Connectivity

Soft Tree
Abstract: Linear Mode Connectivity (LMC) refers to the phenomenon that performance remains consistent for linearly interpolated models in the parameter space. For independently optimized model pairs from different random initializations, achieving LMC is considered crucial for validating the stable success of the non-convex optimization in modern machine learning models and for facilitating practical parameter-based operations such as model merging.
While LMC has been achieved for neural networks by considering the permutation invariance of neurons in each hidden layer, its attainment for other models remains an open question.
In this paper, we first achieve LMC for soft tree ensembles, which are tree-based differentiable models extensively used in practice.
We show the necessity of incorporating two invariances: subtree flip invariance and splitting order invariance, which do not exist in neural networks but are inherent to tree architectures, in addition to permutation invariance of trees.
Moreover, we demonstrate that it is even possible to exclude such additional invariances while keeping LMC by designing decision list-based tree architectures, where such invariances do not exist by definition.
Our findings indicate the significance of accounting for architecture-specific invariances in achieving LMC.

Abstract

Linear Mode Connectivity (LMC) refers to the phenomenon that performance remains consistent for linearly interpolated models in the parameter space. For independently optimized model pairs from different random initializations, achieving LMC is considered crucial for validating the stable success of the non-convex optimization in modern machine learning models and for facilitating practical parameter-based operations such as model merging. While LMC has been achieved for neural networks by considering the permutation invariance of neurons in each hidden layer, its attainment for other models remains an open question. In this paper, we first achieve LMC for soft tree ensembles , which are tree-based differentiable models extensively used in practice. We show the necessity of incorporating two invariances: subtree flip invariance and splitting order invariance , which do not exist in neural networks but are inherent to tree architectures, in addition to permutation invariance of trees. Moreover, we demonstrate that it is even possible to exclude such additional invariances while keeping LMC by designing decision list -based tree architectures, where such invariances do not exist by definition. Our findings indicate the significance of accounting for architecture-specific invariances in achieving LMC.

1 Introduction

A non-trivial empirical characteristic of modern machine learning models trained using gradient methods is that models trained from different random initializations could become functionally almost equivalent, even though their parameter representations differ. If the outcomes of all training sessions converge to the same local minima, this empirical phenomenon can be understood. However, considering the complex non-convex nature of the loss surface, the optimization results are unlikely to converge to the same local minima. In recent years, particularly within the context of neural networks, the transformation of model parameters while preserving functional equivalence has been explored by considering the permutation invariance of neurons in each hidden layer [1, 2]. Notably, only a slight performance degradation has been observed when using weights derived through linear interpolation between permuted parameters obtained from different training processes [3, 4]. This demonstrates that the trained models reside in different, yet functionally equivalent, local minima. This situation is referred to as Linear Mode Connectivity (LMC) [5]. From a theoretical perspective, LMC is crucial for supporting the stable and successful application of non-convex optimization. In addition, LMC also holds significant practical importance, enabling techniques such as model merging [6, 7] by weight-space parameter averaging.

Although neural networks are most extensively studied among the models trained using gradient methods, other models also thrive in real-world applications. A representative is tree ensemble models, such as random forests [8]. While they are originally trained by not gradient but greedy algorithms, differentiable soft tree ensembles , which learn parameters of the entire model through gradient-based

optimization, have recently been actively studied. Not only empirical studies regarding accuracy 38

and interpretability [9-11], but also theoretical analyses have been performed [12, 13]. Moreover, 39

the differentiability of soft trees allows for integration with various deep learning methodologies, 40

including fine-tuning [14], dropout [15], and various stochastic gradient descent methods [16, 17]. 41

Furthermore, the soft tree represents the most elementary form of a hierarchical mixture of experts [1842

20]. Investigating soft tree models not only advances our understanding of this particular structure 43

but also contributes to broader research into essential technological components critical for the 44

development of large-scale language models [21]. 45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

A research question that we tackle in this paper is: 'Can LMC be achieved for soft tree ensembles?'. Our empirical results, which are highlighted with a green line in the top left panel of Figure 1, clearly show that the answer is 'Yes'. This plot shows the variation in test accuracy when interpolating weights of soft oblivious trees, perfect binary soft trees with shared parameters at each depth, trained from different random initializations. The green line is obtained by our method introduced in this paper, where there is almost zero performance degradation. Furthermore, as shown in the bottom left panel of Figure 1, the performance can even improve when interpolating between models trained on split datasets.

The key insight is that, when performing interpolation between two model parameters, considering only tree permutation invariance, which corresponds to the permutation invariance of neural networks, is not sufficient to achieve LMC, as shown in the orange lines in the plots. An intuitive understanding of this situation is also illustrated in the right panel of Figure 1. To achieve LMC, that is, the green lines, we show that two additional invariances beyond tree permutation, subtree flip invariance and splitting order invariance , which inherently exist for tree architectures, should be accounted for.

Moreover, we demonstrate that it is possible to exclude such additional invariances while preserving 68 LMC by modifying tree architectures. We realize such an architecture based on a decision list , a 69 binary tree structure where branches extend in only one direction. By designating one of the terminal 70 leaves as an empty node, we introduce a customized decision list that omits both subtree flip invariance 71 and splitting order invariance, and empirically show that this can achieve LMC by considering only 72 tree permutation invariance. Since incorporating additional invariances is computationally expensive, 73 we can efficiently perform weight-space averaging in model merging on our customized decision 74 lists. 75

Our contributions are summarized as follows: 76

First achievement of LMC for tree ensembles with accounting for additional invariances beyond 77 tree permutation. 78

Development of a decision list-based tree architecture that does not involve the additional invari79 ances. 80

A thorough empirical investigation of LMC across various tree architectures, invariances, and 81 real-world datasets. 82

2 Preliminary 83

We prepare the basic concepts of LMC and soft tree ensembles. 84

2.1 Linear Mode Connectivity 85

Let us consider two models, A and B , that have the same architecture. In the context of evaluating 86

LMC, the concept of a 'barrier' is frequently used [4, 23]. Let Θ A , Θ B ∈ R P be vectorized 87

parameters of models A and B , respectively, for P parameters. Assume that C : R P → R measures 88

the performance of the model, such as accuracy, given its parameter vector. If higher values of C ( · ) 89

Only Permutation

Ours

Target

Origin

Full dataset

80

60

Split dataset

90

80

70

Interpolation

Naive

TreePermutation

Ours

19

mean better performance, the barrier between two parameter vectors Θ A and Θ B is defined as: 90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

We can simply reverse the subtraction order if lower values of C ( · ) mean better performance like loss.

Several techniques have been developed to reduce barriers by transforming parameters while preserving functional equivalence. Two main approaches are activation matching (AM) and weight matching (WM). AM takes the behavior of model inference into account, while WM simply compares two models using their parameters. The validity of both AM and WM has been theoretically supported [24]. Numerous algorithms are available for implementing AM and WM. For instance, [4] uses a formulation based on the Linear Assignment Problem (LAP) to find suitable permutations, while [23] employs a differentiable formulation that allows for the optimization of permutations using gradient-based methods.

Existing research has focused exclusively on neural network architectures such as multi-layer perceptrons (MLP) and convolutional neural networks (CNN). No study has been conducted from the perspective of linear mode connectivity for soft tree ensembles.

2.2 Soft Tree Ensemble

Unlike typical hard decision trees, which explicitly determine the data flow to the right or left at each splitting node, soft trees represent the proportion of data flowing to the right or left as continuous values between 0 and 1. This approach enables a differentiable formulation.

We use a sigmoid function, σ : R → (0 , 1) to formulate a function µ m,ℓ ( x i , w m , b m ) : R F × R F ×N × R 1 ×N → (0 , 1) that represents the proportion of the i th data point x i flowing to the ℓ th leaf of the m th tree as a result of soft splittings:

where N denotes the number of splitting nodes in each tree. The parameters w m,n ∈ R F and b m,n ∈ R correspond to the feature selection mask and splitting threshold value for n th node in a m th tree, respectively. The expression 1 ℓ ↙ n (resp. 1 n ↘ ℓ ) is an indicator function that returns 1 if the ℓ th leaf is positioned to the left (resp. right) of a node n , and 0 otherwise.

If parameters are shared across all splitting nodes at the same depth, such perfect binary trees are called oblivious trees . Mathematically, w m,n = w m,n ′ and b m,n = b m,n ′ for any nodes n and n ′ at the same depth in an oblivious tree. Oblivious trees can significantly reduce the number of parameters from an exponential to a linear order of the tree depth, and they are actively used in practice [9, 11].

To classify C categories, the output of the m th tree is computed by the function f m : R F × R F ×N × R 1 ×N × R C ×L → R C as sum of the leaf parameters π m,ℓ weighted by the outputs of µ m,ℓ ( x i , w m , b m ) :

where L is the number of leaves in a tree. By combining this function for M trees, we realize the 121 function f : R F × R M × F ×N × R M × 1 ×N × R M × C ×L → R C as an ensemble model consisting of 122 M trees: 123

with the parameters w = ( w 1 , . . . , w M ) , b = ( b 1 , . . . , b M ) , and π = ( π 1 , . . . , π M ) being ran124 domly initialized. 125

(a)

(b)

Reordering

Leaf Swap

Inequality Sign Flip

Despite the apparent differences, there are correspondences between MLPs and soft tree ensemble 126 models. The formulation of a soft tree ensemble with D = 1 is: 127

When we consider the correspondence between π m, 1 -π m, 2 in tree ensembles and second layer 128 weights in the two-layer perceptron, the tree ensembles model matches to the two-layer perceptron. It 129 is clear from the formulation that the permutation of hidden neurons in a neural network corresponds 130 to the rearrangement of trees in a tree ensemble. 131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

3 Invariances Inherent to Tree Ensembles

In this section, we discuss additional invariances inherent to trees (Section 3.1) and introduce a matching strategy specifically for tree ensembles (Section 3.2). We also show that the presence of additional invariances varies depending on the tree structure, and we present tree structures where no additional invariances beyond tree permutation exist (Section 3.3).

3.1 Parameter modification processes that maintains functional equivalence in tree ensembles

First, we clarify what invariances should be considered for tree ensembles, which are expected to reduce the barrier significantly if taken into account. When we consider perfect binary trees, there are three types of invariance:

Tree permutation invariance. In Equation (4), the behavior of the function does not change even if the order of the M trees is altered. This corresponds to the permutation of internal nodes in neural networks, which has been a subject of active interest in previous studies on LMC.

Subtree flip invariance. When the left and right subtrees are swapped simultaneously with the inversion of the inequality sign at the split, the functional behavior remains unchanged, which we refer to subtree flip invariance . Figure 2(a) presents a schematic diagram of this invariance, which is not found in neural networks but is unique to binary tree-based models. Since σ ( -c ) = 1 -σ ( c ) for c ∈ R due to the symmetry of sigmoid , the inversion of the inequality is achieved by inverting the signs of w m,n and b m,n . [25] also focused on the sign of weights, but in a different way from ours. They pay attention to the amount of change from the parameters at the start of fine-tuning, rather than discussing the sign of the parameters.

Splitting order invariance. Oblivious trees share parameters at the same depth, which means that the decision boundaries are straight lines without any bends. With this characteristic, even if the splitting rules at different depths are swapped, functional equivalence can be achieved if the positions of leaves are also swapped appropriately as shown in Figure 2(b). This invariance does not exist for non-oblivious perfect binary trees without parameter sharing, as the behavior of the decision boundary varies depending on the splitting order.

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

Note that MLPs also have an additional invariance beyond just permutation. Particularly in MLPs that employ ReLU as an activation function, the output of each layer changes linearly with a zero crossover. Therefore, it is possible to modify parameters without changing functional behavior by multiplying the weights in one layer by a constant and dividing the weights in the previous layer by the same constant. However, since the soft tree is based on the sigmoid function, this invariance does not apply. Previous studies [3, 4, 23] have consistently achieved significant reductions in barriers without accounting for this scale invariance. One potential reason is that changes in parameter scale are unlikely due to the nature of optimization via gradient descent. Conversely, when we consider additional invariances inherent to trees, the scale is equivalent to the original parameters.

3.2 Matching Strategy

Here, we propose a matching strategy for binary trees. When considering invariances, it is necessary to compare multiple functionally equivalent trees and select the most suitable one for achieving LMC. Although comparing tree parameters is a straightforward approach, since the contribution of all the parameters in a tree is not equal, we apply weighting for each node for better matching. By interpreting a tree as a rule set with shared parameters as shown in Figure 3, we determine the weight of each splitting node

8

4

4

2

2

2

2

4

3

2

Parameter Sharing

Parameter Sharing

by counting the number of leaves to which the node affects. For example, in the case of the left example in Figure 3, the root node affects eight leaves, nodes at depth 2 affect four leaves, and nodes at depth 3 affect two leaves. This strategy can apply to even trees other than perfect binary trees. For example, in the right example of Figure 3, the root node affects four leaves, a node at depth 2 affects three leaves, and a node at depth 3 affects two leaves.

In this paper, we employ the LAP, which is used as a standard benchmark [4] for matching algorithms. The procedures for AM and WM are as follows. Detailed algorithms (Algorithms 1 and 2) are described in Section A in the supplementary material.

Activation Matching (Algorithm 1). In trees, there is nothing that directly corresponds to the activations in neural networks. However, by treating the output of each individual tree as an activation value of a neural network, it is possible to optimize the permutation of trees while examining their output similarities. Regarding subtree flip and splitting order invariances, it is possible to find the optimal pattern from all the possible patterns of flips and changes in the splitting order. Since the tree-wise output remains unchanged, the similarity between each tree, generated by considering additional invariances, and the target tree is evaluated based on the inner product of parameters while applying node-wise weighting.

Weight Matching (Algorithm 2). Similar to AM, WM also involves applying weighting while extracting the optimal pattern by exploring possible flipping and ordering patterns. Although it is necessary to solve the LAP multiple times for each layer in MLPs [4], tree ensembles require only a single run of the LAP since there are no layers.

The time complexity of solving the LAP is O ( M 3 ) using a modified Jonker-Volgenant algorithm without initialization [26], implemented in SciPy [27], where M is the number of trees. If only considering tree permutation, this process needs to be performed only once in both WM and AM. However, when considering additional invariances, we need to solve the LAP for each pattern generated by considering these additional invariances. In a non-oblivious perfect binary tree with depth D , there are 2 D -1 splitting nodes, leading to 2 2 D -1 possible combinations of sign flips. Additionally, in the case of oblivious trees, there are D ! different patterns of splitting order invariance. Therefore, for large values of D , conducting a brute-force search becomes impractical.

In Section 3.3, we will discuss methods to eliminate additional invariance by adjusting the tree structure. This enables efficient matching even for deep models. Additionally, in Section 4.2, we will present numerical experiment results and discuss that the practical motivation to apply these algorithms is limited when targeting deep perfect binary trees.

1

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

3.3 Architecture-dependency of the Invariances

In previous subsections, tree architectures are fixed to perfect binary trees as they are most commonly and practically used in soft trees. However, tree architectures can be flexible as we have shown in the right example in Figure 3, and here we show that we can specifically design tree architecture that has neither the subtree flip nor splitting order invariances. This allows efficient matching as considering such two invariances is computationally expensive.

Our idea is to modify a decision list shown on the left side of Figure 4, which is a tree structure where branches extend in only one direction. Due to this asymmetric structure, the number of parameters does not increase exponentially with the depth, and the splitting order invariance does not exist. Moreover, subtree flip invariance also does not exist for any internal nodes except for the terminal splitting node, as shown in the left

Invariance exists

Empty Node

side of Figure 4. To completely remove this invariance, we virtually eliminate one of the terminal leaves by leaving the node empty, that is, a fixed prediction value of zero, as shown on the right side of Figure 4. Therefore only permutation invariance exists for our proposed architecture. We summarize invariances inherent to each model architecture in Table 1.

4 Experiment

We empirically evaluate barriers in soft tree ensembles to examine LMC.

4.1 Setup

Datasets. In our experiments, we employed Tabular-Benchmark [28], a collection of tabular datasets suitable for evaluating tree ensembles. Details of datasets are provided in Section B in the supplementary material. As proposed in [28], we randomly sampled 10 , 000 instances for train and test data from each dataset. If the dataset contains fewer than 20 , 000 instances, they are randomly divided into halves for train and test data. We applied quantile transformation to each feature and standardized it to follow a normal distribution.

Hyperparameters. We used three different learning rates η ∈ { 0 . 01 , 0 . 001 , 0 . 0001 } and adopted the one that yields the highest training accuracy for each dataset. The batch size is set at 512 . It is known that the optimal settings for the learning rate and batch size are interdependent [29]. Therefore, it is reasonable to fix the batch size while adjusting the learning rate. During AM, we set the amount of data used for random sampling to be the same as the batch size, thus using 512 samples to measure the similarity of the tree outputs. As the number of trees M and their depths D vary for each experiment, these details will be specified in the experimental results section. During training, we minimized cross-entropy using Adam [16] with its default hyperparameters 1 . Training is conducted for 50 epochs. To measure the barrier using Equation (1), experiments were conducted by interpolating between two models with λ ∈ { 0 , 1 / 24 , . . . , 23 / 24 , 1 } , which has the same granularity as in [4].

Randomness. We conducted experiments with five different random seed pairs: r A ∈ { 1 , 3 , 5 , 7 , 9 } and r B ∈ { 2 , 4 , 6 , 8 , 10 } . As a result, the initial parameters and the contents of the data mini-batches during training are different in each training. In contrast to spawning [5] that branches off from the exact same model partway through, we used more challenging practical conditions. The parameters w , b , and π were randomly initialized using a uniform distribution, identical to the procedure for a fully connected layer in the MLP 2 .

6

Naive

Perm

0

5

10

15

20

M

=256 (Train)

Naive

Perm

0

5

10

15

20

D

=2 (Train)

Naive

Perm

Perm&Order

Perm&Flip&Order

0

5

10

15

20

Naive

Perm

Perm&Order

Perm&Flip&Order

0

5

10

15

20

Naive

Perm

0

5

10

15

20

M

=256 (Test)

Naive

Perm

0

5

10

15

20

D

=2 (Test)

Naive

Perm

Perm&Order

Perm&Flip&Order

0

5

10

15

20

D

=1, WM

D

=2, WM

D

=3, WM

D

=1, AM

D

=2, AM

D

=3, AM

Naive

Perm

Perm&Order

Perm&Flip&Order

0

5

10

15

20

M

=64, WM

M

=256, WM

M

=1024, WM

M

=64, AM

M

=256, AM

M

=1024, AM

Interpolation

70

75

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

55

60

65

Higgs

Interpolation

80

85

MagicTelescope

Interpolation

60

80

MiniBooNE

Interpolation

60

70

bank-marketing

Interpolation

75

80

85

california

Interpolation

50

60

70

covertype

Interpolation

70

75

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

60

70

80

electricity

Interpolation

54

56

58

eye_movements

Interpolation

65

70

heloc

Interpolation

75

80

85

house_16H

Interpolation

65

70

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Resources. All experiments were conducted on a system equipped with an Intel Xeon E5-2698 CPU 260 at 2.20 GHz, 252 GB of memory, and Tesla V100-DGXS-32GB GPU, running Ubuntu Linux (version 261 4.15.0-117-generic). The reproducible PyTorch [30] implementation is provided in the supplementary 262 material. 263

264

4.2 Results for Perfect Binary Trees

Figure 5 shows how the barrier between two perfect binary tree model pairs changes in each operation. 265 The vertical axis of each plot in Figure 5 shows the averaged barrier over datasets for each considered 266 invariance. The results for both the oblivious and non-oblivious trees are plotted separately in a 267 vertical layout. The panels on the left display the results when the depth D of the tree varies, keeping 268 M = 256 constant. The panels on the right show the results when the number of trees M varies, with 269 D fixed at 2 . For both oblivious and non-oblivious trees, we observed that the barrier significantly 270 decreases as the considered invariances increase. Focusing on the test data results, after accounting for 271 various invariances, the barrier is nearly zero, indicating that LMC has been achieved. In particular, 272 the difference between the case of only permutation and the case where additional invariances are 273 considered tends to be larger in the case of AM. This is because parameter values are not used during 274 the rearrangement of the tree in AM. Additionally, it has been observed that the barrier increases as 275 trees become deeper, and the barrier decreases as the number of trees increases. These behaviors 276 correspond to the changes observed in neural networks when the depth varies or when the width of 277 hidden layers increases [3, 4]. Figure 6 shows interpolation curves when using AM in oblivious trees 278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

75

80

85

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

70

75

bank-marketing

Interpolation

75

80

85

california

Interpolation

50

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

65

70

75

electricity

Interpolation

52.5

55.0

57.5

eye_movements

Interpolation

60

70

heloc

Interpolation

80

85

house_16H

Interpolation

65

70

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

with D = 2 and M = 256 . Other detailed results, such as performance for each dataset, are provided in Section C in the supplementary material.

Furthermore, we conducted experiments with split data following the protocol in [4, 31], where the initial split consists of randomly sampled 80% negative and 20% positive instances, and the second split inverts these ratios. There is no overlap between the two split datasets. We trained two model pairs using these separately split datasets and observed an improvement in performance by interpolating their parameters. Figure 7 illustrates the interpolation curves under AM in oblivious trees with parameters D = 2 and M = 256 . We can observe that considering additional invariances improves performance after interpolation. Note that the data split is configured to remain consistent even when the training random seeds differ. Detailed results for each dataset using WM or AM are provided in Section C of the supplementary material.

Table 2 compares the average test barriers of an MLP with a ReLU activation function, whose width is equal to the number of trees, M = 256 . The procedure for MLPs follows that described in Section 4.1. The permutation for MLPs is optimized using the method described in [4]. Since [4] indicated that WM outperforms AM in neural networks, WM was used for the comparison. Overall, tree models exhibit smaller barriers compared to MLPs while keeping similar accuracy levels. It is important to note that MLPs with D > 1 tend to have more parameters at the same depth compared to trees, leading to more complex optimization landscapes. Nevertheless, the barrier for the non-oblivious tree at D = 3 is smaller than that for the MLP at D = 2 , even with more parameters. Furthermore, at the same depth of D = 1 , tree models have a smaller barrier. Here, the model size is evaluated using F = 44 , the average input feature size of 16 datasets used in the experiments.

MLP

Barrier

Depth

Naive

Perm

[4]

Size

1

8.755 ± 0.877

0.491 ± 0.062

76.286 ± 0.094

12034

2

15.341± 1.125

2.997 ± 0.709

75.981 ± 0.139

77826

3

15.915 ± 2.479

5.940 ± 2.153

75.935 ± 0.117

143618

Non-Oblivious Tree

Barrier

Depth

Naive

Perm

Ours

Size

1

8.965 ± 0.963

0.449 ± 0.235

0.181 ± 0.078

76.464 ± 0.167

12544

2

6.801 ± 0.464

0.811 ± 0.333

0.455 ± 0.105

76.631 ± 0.052

36608

3

5.602 ± 0.926

1.635 ± 0.334

0.740 ± 0.158

76.339 ± 0.115

84736

Oblivious Tree

Barrier

Depth

Naive

Perm

Ours

Size

1

8.965 ± 0.963

0.449 ± 0.235

0.181 ± 0.078

76.464 ± 0.167

12544

2

7.881 ± 0.866

0.918 ± 0.092

0.348 ± 0.172

76.623 ± 0.042

25088

3

7.096 ± 0.856

1.283 ± 0.139

0.484 ± 0.049

76.535 ± 0.063

38656

In Section 3.2, we have shown that considering additional invariances for deep perfect binary trees is computationally challenging, which may suggest developing heuristic algorithms for deep trees. However, we consider it is rather a low priority, supported by our observations that the barrier tends to increase as trees deepen even if we consider invariances. This trend indicates that deep models are fundamentally less important for model merging considerations. Furthermore, deep perfect binary trees are rarely used in practical scenarios. [12] have demonstrated that generalization performance degrades with increasing depth in perfect binary trees due to the degeneracy of the Neural Tangent Kernel (NTK) [32]. This evidence further supports the preference for shallow perfect binary trees, and increasing the number of trees can enhance the expressive power while reducing barriers.

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

4.3 Results for Decision Lists

We present empirical results of the original decision lists and our modified decision lists, as shown in Figure 4. As we have shown in Table 1, they have fewer invariances.

Figure 8 illustrates barriers as a function of depth, considering only permutation invariance, with M fixed at 256 . In this experiment, we have excluded non-oblivious trees from comparison as the number of their parameters exponentially increases as trees deepen, making them infeasible computation. Our proposed modified decision lists reduce the barrier more effectively than both oblivious trees and the original decision lists. However, barriers of the modified

2

4

8

D

0.0

2.5

5.0

7.5

10.0

WM

Oblivious

Decision List

Decision List (Modified)

2

4

8

D

5

10

15

20

AM

decision lists are still larger than those obtained by considering additional invariances with perfect binary trees. Tables 3 and 4 show the averaged barriers for 16 datasets, with D = 2 and M = 256 . Although barriers of modified decision lists are small when considering only permutations (Perm), perfect binary trees such as oblivious trees with additional invariances (Ours) exhibit smaller barriers, which supports the validity of using oblivious trees as in [9, 11]. To summarize, when considering the practical use of model merging, if the goal is to prioritize efficient computation, we recommend using our proposed decision list. Conversely, if the goal is to prioritize barriers, it would be preferable to use perfect binary trees, which have a greater number of invariant operations that maintain the functional behavior.

5 Conclusion

Wehave presented the first investigation of LMC for soft tree ensembles. We have identified additional invariances inherent in tree architectures and empirically demonstrated the importance of considering these factors. Achieving LMC is crucial not only for understanding the behavior of non-convex optimization from a learning theory perspective but also for implementing practical techniques such as model merging. By arithmetically combining parameters of differently trained models, a wide range of applications such as task-arithmetic [33], including unlearning [34] and continual-learning [35], have been explored. Our research extends these techniques to soft tree ensembles that began training from entirely different initial conditions. We will leave these empirical investigations for future work.

This study provides a fundamental analysis of ensemble learning, and we believe that our discussion will not have any negative societal impacts.

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

References

Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers . 1990.

An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the Geometry of Feedforward Neural Network Error Surfaces. Neural Computation , 1993.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks. In International Conference on Learning Representations , 2022.

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. In The Eleventh International Conference on Learning Representations , 2023.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Leo Breiman. Random Forests. In Machine Learning , 2001.

Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data. In International Conference on Learning Representations , 2020.

Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The Tree Ensemble Layer: Differentiability meets Conditional Computation. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. NODE-GAM: Neural generalized additive model for interpretable deep learning. In International Conference on Learning Representations , 2022.

Ryuichi Kanoh and Mahito Sugiyama. A Neural Tangent Kernel Perspective of Infinite Tree Ensembles. In International Conference on Learning Representations , 2022.

Ryuichi Kanoh and Mahito Sugiyama. Analyzing Tree Architectures in Ensembles via Neural Tangent Kernel. In International Conference on Learning Representations , 2023.

Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie-Yan Liu. DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2019.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research , 2014.

Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware Mini399 mization for Efficiently Improving Generalization. In International Conference on Learning 400 Representations , 2021. 401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. In Proceedings of International Conference on Neural Networks , 1993.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-ofExperts Layer. In International Conference on Learning Representations , 2017.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations , 2021.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023.

Byron Roe. MiniBooNE particle identification. UCI Machine Learning Repository, 2010.

Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit Sinkhorn differentiation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

David F. Crouse. On implementing 2D rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems , 2016.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 2020.

Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't Decay the Learning Rate, Increase the Batch Size. In International Conference on Learning Representations , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems , 2019.

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations , 2023.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems , 2018.

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha452 jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International 453 Conference on Learning Representations , 2023. 454

455

456

457

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems , 2022.

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan 458 Ghasemzadeh. Linear Mode Connectivity in Multitask and Continual Learning. In Inter459 national Conference on Learning Representations , 2021. 460

A Detailed Algorithms 461

We present pseudo-code of algorithms for activation matching (Algorithm 1) and weight matching 462 (Algorithm 2). In these algorithms, if there is only one possible pattern for U ∈ N , which represents 463 the number of possible operations, and the corresponding operation does nothing in particular, it 464 becomes equivalent to simply considering tree permutations.

Algorithm 1: Activation matching for soft trees

Algorithm 2: Weight matching for soft trees

Here, we describe the specifications of the notations and functions used in Algorithms 1 and 2. In Section 2.1, Θ A and Θ B are initially defined as vectors. However, for ease of use, in Algorithms 1 and 2, Θ A and Θ B are represented as matrices of size R M × P Tree , where P Tree denotes the number of parameters in a single tree. Multidimensional array elements are accessed using square brackets [ · ]. For example, for G ∈ R I × J , G [ i ] refers to the i th slice along the first dimension, and G [: , j ] refers to the j th slice along the second dimension, with sizes R J and R I , respectively. Furthermore, it can also accept a vector v ∈ N l as an input. In this case, G [ v ] ∈ R l × J . The FLATTEN function converts multidimensional input into a one-dimensional vector format. As the LINEARSUMASSIGNMENT

465

466

467

468

469

470

471

472

473

function, scipy . optimize . linear _ sum _ assignment 3 is used to solve the LAP. In the ADJUSTTREE 474 function, the parameters of a tree are modified according to the u th pattern among the enumerated U 475 patterns. Additionally, in the WEIGHTING function, parameters are multiplied by the square root 476 of their weights defined in Section 3.2 to simulate the process of assessing a rule set. If the first 477 argument for the UPDATEBESTOPERATION function, the input inner product, is larger than any 478 previously input inner product values, then u ′ is updated with u , the second argument. If not, u ′ 479 remains unchanged. 480

B Dataset 481

C Additional Empirical Results

Tables 6, 7, 8 and 9 present the barrier for each dataset with D = 2 and M = 256 . By incorporating additional invariances, it has been possible to consistently reduce the barriers.

Tables 10 and 11 detail the characteristics of the barriers in the decision lists for each dataset with D = 2 and M = 256 . The barriers in the modified decision lists tend to be smaller.

482

483

484

485

486

487

488

489

Tables 12 and 13 show the barrier for each model when only considering permutations with D = 2 and M = 256 . It is evident that focusing solely on permutations leads to smaller barriers in the modified decision lists compared to other architectures.

Figures 9, 10, 11, 12, 13, 14, 15 and 16 show the interpolation curves of oblivious trees with D = 2 490 and M = 256 across various datasets and configurations. Significant improvements are particularly 491 noticeable in AM, but improvements are also observed in WM. These characteristics are also apparent 492 in the non-oblivious trees, as shown in Figures 17, 18, 19, 20, 21, 22, 23 and 24. Regarding split data 493 training, the dataset for each of the two classes is initially complete (100%). It is then divided into 494 splits of 80% and 20%, and 20% and 80%, respectively. Each model is trained using these splits. 495 Figures 13, 15, 21, and 23 show the training accuracy evaluated using the full dataset (100% for each 496 class). 497

Interpolation

70

80

90

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

60

80

100

Higgs

Interpolation

80

85

MagicTelescope

Interpolation

60

80

100

MiniBooNE

Interpolation

60

70

80

bank-marketing

Interpolation

75

80

85

california

Interpolation

60

80

covertype

Interpolation

70

75

credit

Interpolation

60

70

80

default-of-credit-card-clients

Interpolation

60

70

80

electricity

Interpolation

60

70

80

eye_movements

Interpolation

70

80

heloc

Interpolation

80

90

house_16H

Interpolation

80

100

Interpolation

80

90

100

pol

Naive

Tree Permutation

Ours

Interpolation

70

75

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

55

60

65

Higgs

Interpolation

80

85

MagicTelescope

Interpolation

60

80

MiniBooNE

Interpolation

60

70

bank-marketing

Interpolation

75

80

85

california

Interpolation

50

60

70

covertype

Interpolation

70

75

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

60

70

80

electricity

Interpolation

54

56

58

eye_movements

Interpolation

65

70

heloc

Interpolation

75

80

85

house_16H

Interpolation

65

70

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

80

90

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

70

80

90

Higgs

Interpolation

82

84

86

MagicTelescope

Interpolation

80

100

MiniBooNE

Interpolation

60

70

80

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

80

covertype

Interpolation

70

75

credit

Interpolation

60

70

80

default-of-credit-card-clients

Interpolation

70

75

80

electricity

Interpolation

60

70

80

eye_movements

Interpolation

70

75

heloc

Interpolation

80

90

house_16H

Interpolation

80

90

100

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

70

75

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

62.5

65.0

Higgs

Interpolation

82

84

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

60

70

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

70

75

credit

Interpolation

60

65

70

default-of-credit-card-clients

Interpolation

70

75

80

electricity

Interpolation

54

56

58

eye_movements

Interpolation

66

68

70

heloc

Interpolation

75

80

85

house_16H

Interpolation

70

72

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

65

70

75

Higgs

Interpolation

75

80

85

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

70

75

bank-marketing

Interpolation

75

80

85

california

Interpolation

50

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

55

60

65

eye_movements

Interpolation

60

70

heloc

Interpolation

80

85

90

house_16H

Interpolation

75

80

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

75

80

85

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

70

75

bank-marketing

Interpolation

75

80

85

california

Interpolation

50

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

65

70

75

electricity

Interpolation

52.5

55.0

57.5

eye_movements

Interpolation

60

70

heloc

Interpolation

80

85

house_16H

Interpolation

65

70

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

65

70

Higgs

Interpolation

75

80

85

MagicTelescope

Interpolation

80

90

MiniBooNE

Interpolation

70

75

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

65

70

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

55

60

65

eye_movements

Interpolation

60

70

heloc

Interpolation

80

85

house_16H

Interpolation

75

80

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

75

80

85

MagicTelescope

Interpolation

80

90

MiniBooNE

Interpolation

70

75

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

65

70

75

electricity

Interpolation

52.5

55.0

57.5

eye_movements

Interpolation

60

70

heloc

Interpolation

80

85

house_16H

Interpolation

65

70

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

70

80

90

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

70

80

90

Higgs

Interpolation

80.0

82.5

85.0

MagicTelescope

Interpolation

80

100

MiniBooNE

Interpolation

60

70

80

bank-marketing

Interpolation

80

85

california

Interpolation

60

80

covertype

Interpolation

60

70

80

credit

Interpolation

60

70

80

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

70

80

eye_movements

Interpolation

70

75

heloc

Interpolation

85

90

house_16H

Interpolation

80

90

100

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

65

70

75

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

82

84

86

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

60

70

bank-marketing

Interpolation

80

85

california

Interpolation

50

60

70

covertype

Interpolation

60

70

credit

Interpolation

60

65

70

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

56

58

eye_movements

Interpolation

65

70

heloc

Interpolation

80

85

house_16H

Interpolation

70

75

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

70

80

90

Bioresponse

Interpolation

58

60

62

Diabetes130US

Interpolation

70

80

90

Higgs

Interpolation

80.0

82.5

85.0

MagicTelescope

Interpolation

80

90

100

MiniBooNE

Interpolation

70

80

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

80

covertype

Interpolation

70

75

80

credit

Interpolation

60

70

80

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

70

80

eye_movements

Interpolation

72.5

75.0

77.5

heloc

Interpolation

85

90

house_16H

Interpolation

80

90

100

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

70

75

Bioresponse

Interpolation

55

60

Diabetes130US

Interpolation

62.5

65.0

Higgs

Interpolation

82

84

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

60

70

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

70

75

credit

Interpolation

60

65

70

default-of-credit-card-clients

Interpolation

70

75

80

electricity

Interpolation

54

56

58

eye_movements

Interpolation

66

68

70

heloc

Interpolation

75

80

85

house_16H

Interpolation

70

72

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

65

70

75

Higgs

Interpolation

70

80

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

70

80

bank-marketing

Interpolation

75

80

85

california

Interpolation

60

70

covertype

Interpolation

50

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

55

60

65

eye_movements

Interpolation

65

70

75

heloc

Interpolation

84

86

88

house_16H

Interpolation

75

80

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

70

80

MagicTelescope

Interpolation

70

80

90

MiniBooNE

Interpolation

65

70

75

bank-marketing

Interpolation

75

80

85

california

Interpolation

60

70

covertype

Interpolation

60

80

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

65

70

75

electricity

Interpolation

50

55

eye_movements

Interpolation

60

70

heloc

Interpolation

82

84

86

house_16H

Interpolation

65

70

Interpolation

70

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

70

75

Higgs

Interpolation

70

80

MagicTelescope

Interpolation

80

90

MiniBooNE

Interpolation

70

80

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

50

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

70

80

electricity

Interpolation

55

60

65

eye_movements

Interpolation

65

70

75

heloc

Interpolation

86

88

house_16H

Interpolation

75

80

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

Interpolation

60

70

Bioresponse

Interpolation

50

55

60

Diabetes130US

Interpolation

60

65

Higgs

Interpolation

70

80

MagicTelescope

Interpolation

80

90

MiniBooNE

Interpolation

65

70

75

bank-marketing

Interpolation

80

85

california

Interpolation

60

70

covertype

Interpolation

50

60

70

credit

Interpolation

60

70

default-of-credit-card-clients

Interpolation

65

70

75

electricity

Interpolation

52.5

55.0

57.5

eye_movements

Interpolation

60

70

heloc

Interpolation

82.5

85.0

87.5

house_16H

Interpolation

65

70

Interpolation

80

90

pol

Naive

Tree Permutation

Ours

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction consistently present our research on tree ensembles from LMC perspectives.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 3.2, we have discussed the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not provide theoretical results in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup is detailed in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

Answer: [Yes]

Justification: Reproducible source code is provided in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setup is detailed in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted experiments multiple times with different random seeds and have reported the results, including the variability.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources used in our experiment is described in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics and conducted our research in accordance with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have addressed societal impact in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We provide the source code as supplementary material; however, since the experiments concern the fundamental nature of machine learning models, we believe there are no risks involved.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have used open datasets, citing them in accordance with their license information.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not provide any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper neither engages in crowdsourcing nor research involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper neither engages in crowdsourcing nor research involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

810

811

For initial submissions, do not include any information that would break anonymity (if

applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extrapolates the concept of Linear Mode Connectivity (LMC) modulo model invariances to differentiable tree ensembles (DTE). The authors revealed that, in contrast to neural networks (NNs), permutation invariance is insufficient to provide LMC in DTE and propose two additional tree-specific invariances that enable LMC after taking them into account: subtree flip invariance and splitting order invariance. In addition, they provide a modified DTE architecture that does not posses these additional invariances, however still enjoys LMC with only permutation invariance akin to neural network models. This work proposes two algorithms for building LMC given two independently trained DTEs, based on similar methods from NN LMC literature. The claims are supported by a detailed empirical evaluation.\n\nSTRENGTHS:\nHonestly, I enjoyed reading this paper. Although I am not specialized in tree ensembles, I have certain expertise in LMC, and was pleased to find that it is also relevant for DTE models. I think that this contribution is novel and significant.\n\nThe paper is very well-structured. It was very easy to follow despite having no significant experience in decision trees, the authors did a good job preparing the reader in Sec. 2. \n\nSection 3 presents the main contributions of this work, which is done very well using both detailed and intuitive text description and auxiliary images illustrating the main concepts.\n\nEmpirical evaluation is excellent, involving multiple datasets, hyperparameter options, and random seeds. The authors tackled many important questions concerning the study of LMC in DTEs and even compared with NN LMC, which I specifically liked.\n\nWEAKNESSES:\nIt is hard for me to formulate substantial flaws in this work but a couple of remarks that I put in the next section. \n\nThe main weakness of this work is lack of theoretical support and practical implications. However, I acknowledge that these are the same limitations that are attributed to LMC in neural networks, which is a significantly more broad and well-studied field than LMC in tree ensembles. I hope that future work will address these disadvantages in some way. \n\nAlso, I believe that the text could be slightly polished to eliminate typos and small inaccuracies. For instance, the value $D$ in line 127 is not defined at its first occurrence.\n\nQUESTIONS:\nBelow I list some questions/comments related to the work.\n1. It is indeed surprising that mode connectivity can worsen compared to a naive interpolation after accounting for the permutation invariance (e.g., Fig. 1). To my view, the barriers of naive interpolation (realizing an identity permutation) must upper bound the barriers after permutation search. I would ask the authors to give a small comment on this.\n2. Is the weighting strategy described in Sec. 3.2 remains the same for oblivious DTEs that share the same parameters for all nodes at equal depth, so every parameter affects all leaves?\n3. Interestingly, according to Figure 5, non-oblivious DTE shows better LMC than oblivious DTE when accounting for the same invariances. I suppose that this benign over-parameterization effect is similar to increasing $M$ in DTE or width in NN. What do the authors think?\n4. Figure 7: I would suggest adding the result of common (combined data) training for comparison alike [1].\n5. Does LMC in DTE suffer from the same variance collapse issues reported by [2]? If do, how could it be repaired and could it additionally improve LMC in DTE?\n6. It is a little astonishing how much worse Activation Matching performs in DTE compared to Weight Matching. In NN LMC (based on literature and my personal experience), this difference is not so prominent. Could the authors comment on this a little more?\n7. At the same depth, (modified) decision lists contain less total parameters than oblivious trees and especially non-oblivious trees. However, all DTE architectures perform very similarly according to Tab. 3 and 4. Could the authors give any explanation of this effect?\n8. Would adding subtree flip invariance in the terminal splitting node of regular decision lists improve their LMC reported in Tab. 3 and 4? \n\n[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. In The Eleventh International Conference on Learning Representations, 2023. \n\n[2] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, 2023.\n\nLIMITATIONS:\nThe authors discuss the limitations of their methods in Section 3.2.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an analysis of types of neural networks called soft trees from the linear mode connectivity point of view. The authors enumerate 3 types of invariances inherent to soft trees and study linear mode connectivity between different solutions (by solution they understand a trained ensemble of soft tree models) after weights or activations matching that account for these invariances. They also study linear mode connectivity for a special case of soft trees - decision list-based tree - that has only one type of invariance.\n\nSTRENGTHS:\n- The paper is well written\n- Authors claim that it is the first paper to study linear mode connectivity for soft trees\n\nWEAKNESSES:\n## Insufficient contribution\n- In my opinion, the main contribution of this paper is a showcase that different architectures need to account for different invariances when LMC is analyzed, e.g. MLP and soft trees have different invariances. I think that this insight alone is not enough for a paper, because it sounds quite obvious even without analysis.\n\n## Questionable results\n- It is very important to make sure that interpolation results are not computed between the models which are almost identical (that can happen if there is not enough diversity in training recipes). Could you please provide results with distances (any kind of them, e.g. L2 or cosine similarity) between the solutions in Figure. 5 for \"Naive\", \"Tree Permutation\" and \"Ours\" parameter transformations?\n- I would expect decision list trees to be much weaker than soft trees because they have less parameters. Could you please report its performance or show me where I can find it?\n- Model merging is mentioned as one of the applications for linear mode connectivity (LMC), however, no results for model merging are provided.\n  - line 32: \"In addition, LMC also holds significant practical importance, enabling techniques such as model merging [6, 7] by weight-space parameter averaging.\"\n\n## Questionable explanation\n- I could not find a related work section.\n- What is \"Ours\" in Table 2?\n- I did not find in the main text any explanation (even after looking into algorithms in appendix, which I found very confusing) for the operation of weights matching (WM) and activation matching (AM) in case of such invariances as \"Perm\", \"Order\" and \"Flip\" (Notation is from Table 1). Since invariances are the main part of the whole analysis, could you please elaborate more?\n- Another important part of parameter transforms includes Linear Assignment Problem (LAP), but I could not find any details for it neither.\n\nQUESTIONS:\n- What do you mean by crucial for the stable success of non-convex optimization? Since this motivates your analysis, could you justify the main text, please?\n  - line 4: \"considered crucial for validating the stable success of the non-convex optimization\"\n  - line 30: “From a theoretical perspective, LMC is crucial for supporting the stable and successful application of non-convex optimization.”\n- Why does an MLP with depth 2 have lower accuracy than an MLP with depth 1 in Table 2?\n- Could you please explain or provide sources for how soft trees help in the development of LLMs?\n  - line 44: \"contributes to broader research into essential technological components critical for the development of large-scale language models”\n- Why is the accuracy of the interpolated model higher than the accuracy of the starting models in Figure 8 (e.g. for Bioresponse)?\n- Why do you say that if models are LMC then they are functionally equivalent? Doesn't it just mean that they are connected with a path in weights space along which loss value is lower than some threshold?\n  - line 29: “This demonstrates that the trained models reside in different, yet functionally equivalent, local minima. This situation is referred to as Linear Mode Connectivity (LMC) [5].”\n- What is the sum of leaf parameters? Doesn't leaf have only one parameter? And what is its shape? According to eq. (3) it should be vector, I think.\n  - line 119: \"as sum of the leaf parameters $\\pi_{m, \\ell}$\"\n- Why do you study LMC between ensembles of trees and not single trees? That would remove the need for accounting for permutation invariance.\n- Could you please explain why decision boundaries of oblivious trees are straight lines?\n  - line 152: \"which means that the decision boundaries are straight lines without any bends.\"\n\nLIMITATIONS:\n- There is no theoretical justification for why and in which scenarios linear mode connectivity exists for soft trees.\n- The paper does not propose any practical application for the linear mode connectivity between soft trees. While it can be argued that this paper is an analysis paper, some practical applications can be useful in motivating this kind of analysis.\n- I did not find the code of the project while in the survey it is written that code is provided in supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empirically shows that separately trained tree emsemble models can show Linear Mode Connectivity (LMC) when considering tree invariant operations.\n\nSTRENGTHS:\n- The exploration of LMC on tree emsemble models is interesting.\n- The computational process is clearly stated which makes this paper easy to follow.\n\n------\n\nAfter reading author rebuttal and discussion with other reviewers, I decide to increase my rating of this paper to borderline reject.\n\nWEAKNESSES:\nThis paper does not provide any insights into the question of LMC in neural networks, as it is exploring a totally different model. Although it is always interesting to consider LMC in another senario, I find the contribution of this paper rather insignificant and incremental, since it is basically applying the same idea of [1] to another model. I do not want to deny the author's valueable efforts in exploring symmetry in a new model and using it to achieve LMC, but I just feel that the contribution of this paper may not be sufficient for it to be accepted by this conference.\n\nOne possible direction I can suggest for the authors to enhance the current paper is, if any non-trivial theory about LMC can be made on the tree ensemble model setting, then this work will be much more exciting. The underlying reason why neural networks can be made linear connected is not yet clear, and the is hard to study due to the non-linear nature of deep NNs. If the authors can show that the tree ensemble model can be an alternative model to study LMC from a theoretical perspective, then this will make the current work more valuable and intresting.\n\n[1] Git Re-Basin: Merging Models modulo Permutation Symmetries\n\n**Regard writting**\nThe intro is kind of confusing for readers who are not familiar with the tree ensemble models. It's even unclear whether 1) it is a new model ensembling method for neural networks, or 2) it is a new model, or 3) it is a training method. Although those questions are addressed after reading the detailed definition of tree ensemble in Section 2.2, I think it is better to make it clear in the intro to avoid any confusing.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nAuthors discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper aims to achieve LMC for soft tree ensembles. Akin to achieve LMC for neural network after accounting for permutation invariance, the authors introduce three different kinds of invariance in soft tree ensembles: tree permutation invariance, subtree flip invariance, splitting order invariance. Additionally, the authors demonstrate that better LMC can be achieved after considering all three kinds of invariance.

STRENGTHS:
1. The idea of extending LMC from neural networks to differentiable tree ensembles is interesting. 
2. Invariances beyond permutation variance are identified for differentiable tree ensembles. The authors demonstrate the effectiveness of accounting for these invariances when doing matching.

WEAKNESSES:
1. I am not familiar with differentiable tree ensembles, therefore, I would suggest the authors put more efforts on explaining tree ensembles and illustrating the invariances.
2. Another concern is about the motivation. This study is motivated by the question "Can LMC be achieved for soft tree ensembles?" but why would we achieve LMC for the tree ensembles? I would expect more elaboration on the motivation.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17