Title: Learning Interpretable Characteristic Kernels via Decision Forests
Venue: NeurIPS 2023
Accepted: no
Keywords: kernel learning

random forest

hypothesis testing
Abstract: Decision forests are popular tools for classification and regression. These forests naturally generate proximity matrices that measure the frequency of observations appearing in the same leaf node. While other kernels are known to have strong theoretical properties such as being characteristic, there is no similar result available for decision forest-based kernels. In addition, existing approaches to independence and k-sample testing may require unfeasibly large sample sizes and are not interpretable. In this manuscript, we prove that the decision forest induced proximity is a characteristic kernel, enabling consistent independence and k-sample testing via decision forests. We leverage this to introduce kernel mean embedding random forest (KMERF), which is a valid and consistent method for independence and k-sample testing. Our extensive simulations demonstrate that KMERF outperforms other tests across a variety of independence and two-sample testing scenarios. Additionally, the test is interpretable, and its key features are readily discernible. This work therefore demonstrates the existence of a test that is both more powerful and more interpretable than existing methods, flying in the face of conventional wisdom of the trade-off between the two.

Abstract

Decision forests are popular tools for classification and regression. These forests naturally generate proximity matrices that measure the frequency of observations appearing in the same leaf node. While other kernels are known to have strong theoretical properties such as being characteristic, there is no similar result available for decision forest-based kernels. In addition, existing approaches to independence and k-sample testing may require unfeasibly large sample sizes and are not interpretable. In this manuscript, we prove that the decision forest induced proximity is a characteristic kernel, enabling consistent independence and k-sample testing via decision forests. We leverage this to introduce kernel mean embedding random forest (KMERF), which is a valid and consistent method for independence and k-sample testing. Our extensive simulations demonstrate that KMERF outperforms other tests across a variety of independence and two-sample testing scenarios. Additionally, the test is interpretable, and its key features are readily discernible. This work therefore demonstrates the existence of a test that is both more powerful and more interpretable than existing methods, flying in the face of conventional wisdom of the trade-off between the two.

1 Introduction

Decision forests are ensemble method popularized by Breiman [3]. It is highly effective in classifica18 tion and regression tasks, particularly in high-dimensional settings [5, 6, 37]. This is achieved by 19 randomly partitioning the feature set and using subsampling techniques to construct multiple decision 20 trees from the training data. To measure the similarity between two observations, a proximity matrix 21 can be constructed, defined as the percentage of trees in which both observations lie in the same leaf 22 node [4]. This proximity matrix serves as an induced kernel or similarity matrix for the decision 23 forest. In general, any random partition algorithm may produce such a kernel matrix. 24

As the complexity of datasets grow, it becomes increasingly necessary to develop methods that can 25

efficiently perform independence and k-sample testing. We also desire methods that are interpretable, 26

27

lending insight into how and why statistically significant results were determined. Parametric methods

are often highly interpretable, such as Pearson's correlation and its rank variants [22, 33, 16]. These 28

29

methods are still popular to detect linear and monotonic relationships in univariate settings, but

they are not consistent for detecting more complicated nonlinear relationships. Nonparametric 30

methods can be very powerful. The more recent distance correlation (Dcorr) [36, 35] and the 31

kernel correlation (HSIC) [10, 11] are consistent for testing independence against any distribution of 32

finite second moments for any finite dimensionality; moreover, the energy-based statistics (such as 33

Dcorr) and kernel-based statistics (such as HSIC) are known to be exactly equivalent for all finite 34

samples [21, 30]. The theory supporting universal consistency of these methods (which we refer to as 35

kernel methods hereafter, without loss of generality) depends on those kernels being characteristic 36

kernel [28, 18, 19, 30]. Unfortunately, the above tests do not attempt to further characterize the 37 dependency structure. To the best of our knowledge, very few tests exist[38, 15]. 38

In addition, although these methods all have asymptotic guarantees, for finite samples, performance 39 can be impaired by poorly choosing a particular characteristic kernel. Choosing an appropriate 40 kernel that properly summarize geometries within the data is often times non-obvious [27]. High41 dimensional data is particularly vexing [25, 38], and a number of extensions have been proposed to 42 achieve better power such as adaptive metric kernel choice [12], low-dimensional projections [14], 43 and marginal correlations [29]. 44

45

46

47

48

49

50

51

52

53

54

In this paper, we leverage the popular random forest method [3] and a recent chi-square test [32] for a more powerful and interpretable method for hypothesis testing. We prove that the random forest induced kernel is a characteristic kernel, and the resulting kernel mean embedding random forest (KMERF) is a valid and consistent method for independence and k-sample testing. We then demonstrate its empirical advantage over existing tools for high-dimensional testing in a variety of dependence settings, suggesting that it will often be more powerful than existing approaches in real data. As random forest can directly estimate feature importances [3], the outputs of KMERF are also interpretable, KMERF therefore flies in the face of conventional wisdom that one must choose between power and interpretability: KMERF is both empirically more powerful and more interpretable than existing approaches.

55

56

57

58

59

60

61

62

63

2 Preliminaries

2.1 Hypothesis Testing

The testing independence hypothesis is formulated as follows: suppose x i ∈ R p and y i ∈ R q , and n samples of ( x i , y i ) iid ∼ F XY , i.e., x i and y i are realizations of random variables X and Y . The hypothesis for testing independence is

glyph[negationslash]

Given any kernel function k ( · , · ) , we can formulate the kernel induced correlation measure as c n k ( x , y ) using the sample kernel matrices [10, 30], where x = { x i } and y = { y i } . When the kernel function k ( · , · ) is characteristic, it has been shown that c n k ( x , y ) → 0 if and only if x and y are independent [10].

The k-sample hypothesis is formulated as follows: let u j i ∈ R p be the realization of random variable 64 U j for j = 1 , . . . , l and i = 1 , . . . , n j . Suppose the l datasets that are sampled i.i.d. from F 1 , . . . , F l 65 and independently from one another. Then, 66

glyph[negationslash]

glyph[negationslash]

By concatenating the l datasets and introducing an auxiliary random variable, the kernel correlation measure can be used for k-sample testing [21].

2.2 Characteristic Kernel

Definition 1. Let X be a separable metric space, such as R p . A kernel function k ( · , · ) : X ×X → R measures the similarity between two observations in X , and an n × n kernel matrix for { x i ∈ X , i = 1 , . . . , n } is defined by K ( i, j ) = k ( x i , x j ) .

A kernel k ( · , · ) : X × X → R is positive definite if, for any n ≥ 2 , x 1 , . . . , x n ∈ X and a 1 , . . . , a n ∈ R , it satisfies

A characteristic kernel is a positive definite kernel that has the following property: for any two random variables X 1 and X 2 with distributions F X 1 and F X 2 ,

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

3 KMERF

The proposed approach for hypothesis testing, KMERF, involves the following steps:

Run random forest with m trees, with independent bootstrap samples of size n b ≤ n used to construct each tree. The tree structures (partitions) within the forest P are denoted as φ w ∈ P , where w ∈ 1 , . . . , m and φ w ( x i ) represents the partition assigned to x i .

Calculate the proximity kernel by

where I ( · ) is the indicator function that checks whether the two observations lie in the same partition in each tree.

Compute the unbiased kernel transformation [34, 32] on K x . Namely, let

glyph[negationslash]

Let K y be the Euclidean distance induced kernel by Shen and Vogelstein [30], or the proximity kernel in the case that dimensions of x and y is the same, that is p = q , and compute L y using the same unbiased transformation. Then the KMERF statistic for the induced kernel k is,

Compute the p-value via the following chi-square test [32]:

where χ 2 1 is the chi-square distribution of degree 1 . Reject the independence hypothesis if the p-value is less than a specified typer 1 error level, say 0 . 05 .

In the numerical implementation, the standard supervised random forest is used with m = 500 (which is also applicable to the unsupervised version or other random forest variants [2, 1, 37]). In the second step, we simply compute the proximity kernel defined by the random forest induced kernel. In the third step, we normalize the proximity kernel to ensure it obtains a consistent dependence measure; this is the KMERF test statistic. We found that utilizing the multiscale version of the kernel correlation [38, 31], which is equivalent for linear relationships while being better for nonlinear relationships, produced similar results to using distance correlation, but substantially increased runtimes.

Note that one could also compute a p-value for KMERF via the permutation test, which is a standard procedure for testing independence [9]. Specifically, first compute a kernel on the observed { x i } and { y i } . Then randomly permute the index of { y i } , repeat the kernel generation process for { y i } for each permutation. This process involves training a new random forest for each permutation. Finally, compute the test statistic for each of the permutations, and the p-value equals the percentage the permuted statistics that are larger than the observed statistic. However, the permutation test is very slow for large sample size and almost always yields similar results as the chi-square test.

4 Theoretical Properties

Here, we show that the random forest kernel characteristic, and the induced test statistic used in KMERF allows for valid and universally consistent independence and k-sample testing. All proofs are in appendix.

For a kernel to be characteristic, it first needs to be positive definite, which is indeed the case for the 111 forest-induced kernel: 112

Theorem 1. The random forest induced kernel K x is always positive definite. 113

114

115

116

This theorem holds because the forest-induced kernel is a summation of a permuted block diagonal matrix, with each matrix coming from individual tree, that is positive definite [7]; and a summation of positive definite matrices is still positive definite.

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

Next, we show the kernel is characteristic when the tree partition area converges to zero. A similar property is also used for proving classification consistency for k-nearest-neighbors [8], and we shall denote N ( φ w ) as the maximum area of each part.

Theorem 2. Suppose as n, m →∞ , N ( φ w ) → 0 for each tree φ w ∈ P and each observation x i . Then the random forest induced kernel K x is asymptotically characteristic.

Intuitively, for sufficiently many trees and sufficiently small leaf region, observations generated by two different distributions cannot always be in the same leaf region.

This leads to the validity and consistency result of KMERF:

Corollary 2.1. KMERF satisfies

with equality to 0 if and only if F XY = F X F Y . Moreover, for sufficiently large n and sufficiently small type 1 error level α , this method is valid and consistent for independence and k-sample testing.

By Gretton et al. [10], any characteristic-kernel based dependence measure converges to 0 if and only if X and Y are independent. Moreover, Shen et al. [32] showed that the chi-square distribution χ 2 1 -1 approximates and upper-tail dominates the true null distribution of any unbiased kernel when using distance correlation, making it a valid and consistent test.

5 Simulations

In this section we exhibit the consistency and validity of KMERF, and compare its testing power with other competitors in a comprehensive simulation set-up. We utilize the hyppo package in Python [20], which uses scikit-learn [23] random forest with 500 trees and otherwise default hyper-parameters, and calculate the proximity matrix from this. The KMERF statistic and p-value then computed via the process in Section 3. The mathematical details for each simulation type is in the Appendix C.

5.1 Testing Independence

In this section we compare KMERF to Multiscale Graph Correlation (MGC), Distance Correlation (Dcorr), Hilbert-Schmidt Independence Criterion (Hsic), and Heller-Heller-Gorfine (HHG) method, Canonical Correlation Analysis (CCA), and the RV coefficient. The HHG method has been shown to work extremely well against nonlinear dependencies [13]. The MGC method has been shown to work well against linear, nonlinear, and high-dimensional dependencies [31]. The CCA and RV coefficients are popular multivariant extensions of Pearson correlation. For each method, we use the corresponding implementation in hyppo with default settings.

We take 20 high-dimensional simulation settings [38], consisting of various linear, monotone, and strongly nonlinear dependencies with p increasing, q = 1 , and n = 100 . To estimate the testing power in each setting, we generate dependent ( x i , y i ) for i = 1 , . . . , n , compute the test statistic for each method, repeat for r = 10000 times. Via the empirical alternative and null distribution of the test statistic, we estimate the testing power of each method at type 1 error level of α = 0 . 05 . The power result is shown in Figure 1 shows that KMERF achieves superior performance for most simulation modalities, except a few like circle and ellipse.

5.2 Two Sample Testing

Here, we compare the performance in the two-sample testing regime. It has been shown that all independence measures can be used for two-sample testing [21, 30], allowing all previous independence testing methods to be compared here as well. Once again, we investigate statistical

3

1000

0

1

Linear

3

1000

Exponential

3

1000

Cubic

3

10

Joint Normal

3

20

Step

3

20

0

1

Quadratic

3

20

W-Shaped

3

20

Spiral

3

100

3

100

Logarithmic

3

20

0

1

Fourth Root

3

10

Sine 4

3

10

Sine 16

3

40

Square

3

20

Two Parabolas

3

20

0

1

Circle

3

20

Ellipse

3

40

Diamond

3

10

Multiplicative

3

100

Independence

KMERF

MGC

Dcorr

Hsic

HHG

CCA

RV

Multivariate Independence Testing (Increasing Dimension)

Dimension

Statistical Power

.

power differences with 20 simulation settings consisting of various linear and nonlinear, monotonic 158 and nonmonotonic functions with dimension increasing from p = 3 , . . . , 10 , q = 1 , and n = 100 159 We then apply a random rotation to this generated simulation and generate the second independent 160 sample (via a rigid transformation). 161

162

163

164

165

Figure 2 shows that, once again, for the majority of simulations settings, KMERF performs at or better than other tests in nearly all simulations and simulation dimensions. For certain simulation settings, especially the exponential, cubic, and fourth root, KMERF vastly outperforms other metrics as dimensions increases.

166

167

168

5.3 Interpretability

Not only does KMERF typically offer empirically better statistical power compared to alternatives, it also offers insights into which features are the most important within the data set. Figure 3 shows

normalized 95% confidence intervals of relative feature importances for each simulation, where the 169

black line shows the mean and the light grey line shows the 95% confidence interval. Mean and 170

individual tree feature importances were normalized using min-max feature scaling. The simulations 171

were modified such that the weighting of each feature decreased as feature importance increased, 172

with the expectation that the algorithm would detect a decrease in feature importance as dimension 173

increased. With these simulations, we are able to determine that exact feature importance trend, 174

3

10

0

1

Linear

3

10

Exponential

3

10

Cubic

3

10

Joint Normal

3

10

Step

3

10

0

1

Quadratic

3

10

W-Shaped

3

10

Spiral

3

10

3

10

Logarithmic

3

10

0

1

Fourth Root

3

10

Sine 4

3

10

Sine 16

3

10

Square

3

10

Two Parabolas

3

10

0

1

Circle

3

10

Ellipse

3

10

Diamond

3

10

Multiplicative

3

10

Independence

KMERF

MGC

Energy

MMD

HHG

CCA

RV

Multivariate Two-Sample Testing (Increasing Dimension)

Dimension

Statistical Power

except for a few of the more complex simulations. The process we used to generate this figure can be 175

6 Real Data 177

0

1

Linear

Exponential

Cubic

Joint Normal

Step

0

1

Quadratic

W-Shaped

Spiral

Logarithmic

0

1

Fourth Root

Sine 4

Sine 16

Square

Two Parabolas

1

5

0

1

Circle

1

5

Ellipse

1

5

Diamond

1

5

Multiplicative

1

5

Independence

Feature Importances

Dimension (Sorted by Feature Importance)

Estimated Feature Importance

10

5

10

3

10

1

p-values for Panc vs Norm

10

4

10

3

10

2

10

1

p-values for Panc vs All

neurogranin

False Positives

True Positives

0

2

4

6

8

# True / False Positives

Biomarker kNN Classification

KMERF

Hsic

HHG

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

7 Discussion

KMERF is, to the best of our knowledge, one of the first learned kernel that is proven to be characteristic. The empirical experiments presented here illustrate the potential advantages of learning kernels, specifically for independence and k-sample testing.

In fact, multiscale graph correlation [38, 31] can be thought of, in a sense, as kernel learning: given n samples, and a pair of kernel or distance functions, it chooses one of the approximately n 2 sparsified kernels, by excluding all but the nearest neighbors for each data point [38, 31]. Because random forest can be thought of as a nearest neighbor algorithm [17], in a sense, the forest induced kernel is a natural extension of Vogelstein et al. [38], which leads to far more data-adaptive estimates of the nearest neighbors using supervised information. Moreover, proving that the random-forest induced kernel is characteristic is a first step towards building lifelong learning kernel machines with strong theoretical guarantees [24, 39].

As the choice of kernel is crucial for empirical performance, this manuscript offers a new kernel construction that is not only universally consistent for testing independence, but also exhibits strong empirical advantages, especially for high-dimensional testing. What is unique to this choice of kernel is the robustness and interpretability. It will be worthwhile to further understand the underlying theoretical mechanism of the induced characteristic kernel, as well as evaluating the performance of these forest induced kernels on other learning problems, including classification, regression, clustering, and embedding [26].

References

S. Athey, J. Tibshirani, and S. Wager. Generalized random forests. Annals of Statistics , 47(2): 1148-1178, 2018.

R. Blaser and P. Fryzlewicz. Random rotation ensembles. Journal of Machine Learning Research , 17(4):1-26, 2016.

L. Breiman. Random forests. Machine Learning , 4(1):5-32, October 2001.

L. Breiman. Some infinity theory for predictor ensembles. Journal of Combinatorial Theory, Series A , 98:175-191, 2002.

R. Caruana and A. Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd international conference on Machine learning , pages 161-168. ACM, 2006.

R. Caruana, N. Karampatziakis, and A. Yessenalina. An empirical evaluation of supervised learning in high dimensions. Proceedings of the 25th International Conference on Machine Learning , 2008.

A. Davies and Z. Ghahramani. The random forest kernel and creating other kernels for big data from random partitions. arXiv:1402.4293v1 , 2014.

L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition . Springer, 1996.

P. Good. Permutation, Parametric, and Bootstrap Tests of Hypotheses . Springer, 2005.

A. Gretton, R. Herbrich, A. Smola, O. Bousquet, and B. Scholkopf. Kernel methods for measuring independence. Journal of Machine Learning Research , 6:2075-2129, 2005.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research , 13(25):723-773, 2012. ISSN 1533-7928.

A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in neural information processing systems 25 , pages 1205-1213, 2012.

R. Heller, Y. Heller, and M. Gorfine. A consistent multivariate test of association based on ranks of distances. Biometrika , 100(2):503-510, 2013.

C. Huang and X. Huo. A statistically and numerically efficient independence test based on 236 random projections and distance covariance. arXiv , 2017. 237

W. Jitkrittum, Z. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features 238 with maximum testing power. In Advances in Neural Information Processing Systems , pages 239 181-189, 2016. 240

M. G. Kendall. Rank Correlation Methods . London: Griffin, 1970. 241

Y. Lin and Y. Jeon. Random forests and adaptive nearest neighbors. J. Am. Stat. Assoc. , 101 242 (474):578-590, 2006. 243

R. Lyons. Distance covariance in metric spaces. Annals of Probability , 41(5):3284-3305, 2013. 244

R. Lyons. Errata to 'distance covariance in metric spaces'. Annals of Probability , 46(4): 245 2400-2405, 2018. 246

247

248

S. Panda, S. Palaniappan, J. Xiong, E. W. Bridgeford, R. Mehta, C. Shen, and J. T. Vogelstein. hyppo: A comprehensive multivariate hypothesis testing python package, 2020.

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

S. Panda, C. Shen, R. Perry, J. Zorn, A. Lutz, C. E. Priebe, and J. T. V ogelstein. Nonpar manova via independence testing, 2021.

K. Pearson. Notes on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London , 58:240-242, 1895.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research , 12(85):2825-2830, 2011. URL http://jmlr.org/papers/ v12/pedregosa11a.html .

A. Pentina and S. Ben-David. Multi-task and lifelong learning of kernels. In Algorithmic Learning Theory , pages 194-208. Springer International Publishing, 2015.

A. Ramdas, S. J. Reddi, B. Póczos, A. Singh, and L. Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In 29th AAAI Conference on Artificial Intelligence , 2015.

B. Schölkopf and A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002.

D. Sejdinovic, A. Gretton, B. Sriperumbudur, and K. Fukumizu. Hypothesis testing using pairwise distances and associated kernels (with appendix). arXiv preprint arXiv:1205.0411 , 2012.

D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. Annals of Statistics , 41(5):2263-2291, 2013.

C. Shen. Scaling up independence testing: Maximum marginal distance correlation and the chi-square test. https://arxiv.org/abs/2001.01095 , 2023.

C. Shen and J. T. V ogelstein. The exact equivalence of distance and kernel methods in hypothesis testing. AStA Advances in Statistical Analysis , 105(3):385-403, 2021.

C. Shen, C. E. Priebe, and J. T. Vogelstein. From distance correlation to multiscale graph correlation. Journal of the American Statistical Association , 115(529):280-291, 2020.

C. Shen, S. Panda, and J. T. Vogelstein. The chi-square test of distance correlation. Journal of Computational and Graphical Statistics , 31(1):254-262, 2022.

C. Spearman. The Proof and Measurement of Association between Two Things. The American Journal of Psychology , 15(1):72, 1904. ISSN 00029556. doi: 10.2307/1412159. URL http://www.jstor.org/stable/1412159?origin=crossref .

G. Szekely and M. Rizzo. Partial distance correlation with methods for dissimilarities. Annals 281 of Statistics , 42(6):2382-2412, 2014. 282

G. Székely and M. L. Rizzo. The distance correlation t-test of independence in high dimension. 283 Journal of Multivariate Analysis , 117:193-213, 2013. 284

G. Székely, M. L. Rizzo, and N. K. Bakirov. Measuring and testing independence by correlation 285 of distances. Annals of Statistics , 35(6):2769-2794, 2007. 286

T. Tomita, J. Browne, C. Shen, J. Chung, J. Patsolic, B. Falk, J. Yim, C. E. Priebe, R. Burns, 287 M. Maggioni, and J. T. Vogelstein. Sparse projection oblique randomer forests. Journal of 288 Machine Learning Research , 21(104):1-39, 2020. 289

J. T. V ogelstein, Q. Wang, E. W. Bridgeford, C. E. Priebe, M. Maggioni, and C. Shen. Discover290 ing and deciphering relationships across disparate data modalities. eLife , 8:e41690, 2019. 291

J. T. V ogelstein, J. Dey, H. S. Helm, W. LeVine, R. D. Mehta, T. M. Tomita, H. Xu, A. Geisa, 292 Q. Wang, G. M. van de Ven, C. Gao, W. Yang, B. Tower, J. Larson, C. M. White, and C. E. 293 Priebe. Representation ensembling for synergistic lifelong learning with quasilinear complexity, 294 2023. 295

E. A. Willemse, A. De Vos, E. M. Herries, U. Andreasson, S. Engelborghs, W. M. Van Der Flier, 296 P. Scheltens, D. Crimmins, J. H. Ladenson, E. Vanmechelen, et al. Neurogranin as cerebrospinal 297 fluid biomarker for alzheimer disease: an assay comparison study. Clinical chemistry , 64(6): 298 927-937, 2018. 299

J. Yang, F. K. Korley, M. Dai, and A. D. Everett. Serum neurogranin measurement as a 300 biomarker of acute traumatic brain injury. Clinical biochemistry , 48(13-14):843-848, 2015. 301

Peer reviews:
comments: SUMMARY:
The authors show the random forest induced kernel is characteristic, and they empirically study the validity of the kernel for independence and k-sample testing.

STRENGTHS:
The authors show the connection between the characteristic property of the kernel and random forest. The topic is interesting and important.

WEAKNESSES:
The presentation should be improved. Since detailed explanations about random forest are missing, how we can connect the kernel methods to random forest is not clear. I think that is the most important background of this paper, so it should be explained more. In addition, the novelty of the paper is not clear to me since the theoretical results in this paper seem to depend strongly on the random forest setting, but the setting is not clearly explained.

QUESTIONS:
- In Theorem 2, does $\phi_w$ become injective by the construction of the kernel through random forest? I think specific properties coming from the random forest setting should be explained clearly in the main text.

Minor comments:
- In the supplementary material, there are so many CSV files, but we only need codes and a PDF file for the appendix of the paper.

LIMITATIONS:
As I also mentioned in the weakness part, since the setting and the background are not sufficiently explained, the novelty and the nontrivial part of this paper are not clear.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

comments: SUMMARY:
The paper show how decision forest can be used to induce a kernel for k-sample testing.

STRENGTHS:
Better-than-SOTA results.

WEAKNESSES:
The paper is a extremely hard to follow, particularly Section 3, which I was unable to follow without going deep into the referenced papers. The same goes for related works, which are simply mentioned en-passant in Section 5.1. A reader not directly familiar with all of them is left wondering in what they differ, and why.
Additional comments:
- L18: "are an ensemble".
- L18: "They are highly effective".
- Given that we're dealing with matrices, I'd suggest using \mathds{1} to indicate indicator functions, rather than I, which is usually used for identity matrices.
- Figure 4 only reports 2 competitors: what about the others?

QUESTIONS:
Could not formulate questions due to difficulty in understanding the paper.

LIMITATIONS:
Limitedly comprehensible.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduced a new method called KMERF, which employs random forest for kernel construction. Through their algorithm, they were able to establish that the kernel they created has certain properties, namely being positive definite and asymptotically characteristic. The authors also demonstrated that KMERF has better statistical power than other independent methods. This makes KMERF one of the pioneering learned kernels that has been proven to be asymptotically characteristic.\n\nSTRENGTHS:\nI find the proposed method to be impressive and well-organized. The paper is also well-written. It's worth noting that the number of learned kernels with characteristic properties is limited, so KMERF could be a valuable contribution to the kernel learning field. While it's not unexpected for a learned kernel to show better statistical power, the decent false discovery rate of KMERF is intriguing.\n\nWEAKNESSES:\n1. There is concern about potential inflation from post-selection inference since the kernel being proposed is learned. To address this, I suggest that the authors conduct additional calibration tests of KMER. For instance, they could create various simulation scenarios (similar to those presented in Fig1 and Fig2) involving a mix of causal and null features, and demonstrate the well-controlled FPR/ FDR. \n\n2. Further details are required in Fig1 and Fig2, such as parameter values and exact equations, to fully understand how each simulation setting was carried out.\n\n3. It would be beneficial to have a more comprehensive analysis of the runtime.\n\nQUESTIONS:\nIn supplementary proof theorem 1, line 8, it is not clear to me why \"Each block matrix is always positive definite\".\n\nLIMITATIONS:\nKMERF has been proven to exhibit asymptotic characteristics when the sample size n and the number of trees m approach infinity. However, in practical applications, m is often limited to prevent overfitting and computational issues. Providing insights on KMERF's behavior under finite n and reasonable m would be greatly beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, random forest induced kernel/proximity is combined with distance correlation and a recently developed chi-square test method to form a hypothesis testing method that is useful for independence testing and k-sample testing. The authors prove that the kernel is asymptotically characteristic and therefore the proposed method is valid and consistent for a sufficiently large sample size. Through experimental evaluation of statistical power for independence and two-sample testing on synthetic data, the authors show that their method performs better than other tests for the majority of simulations settings, and is good at identifying important features. When applied to real biomarker data, the method successfully identifies a potentially valuable marker for pancreatic cancer.

STRENGTHS:
- The method is easy to implement and seems powerful. It inherits the advantages of random forest, e.g., working very well on small datasets, almost no need for parameter tuning and preprocessing, easy-to-access high-quality implementations, etc.
- The paper is well-written and easy to follow.
- Theoretical properties of the method are investigated. The results seem correct.
- The chi-square test is much more efficient than the permutation test.
- Theorem 2 seems novel.
- Experimental evaluation is well conducted and looks convincing.

WEAKNESSES:
- The novelty of this method is limited because the combination is straightforward. The random forest induced proximity is well-known in the literature.
- The related work on random forests for hypothesis testing is not surveyed.

QUESTIONS:
- When does the premise of Theorem 2 (part area goes to zero) hold?
- What are the labels for supervised RF training?

LIMITATIONS:
The limitations of this work have been discussed in the Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this study, the authors proposed a new kernel KMERF for independence testing.\nIn KMERF, multiple decision trees are constructed similar to Random Forest, and the number of trees in which two data points belong to the same leaf node is calculated.\nThis count is used as the kernel value between the two points to form a kernel matrix.\nThe p-value is then computed using a method similar to HSIC.\nThe authors proved that KMERF is a characteristic kernel in the limit as the number of data points and the number of decision trees approach infinity.\nThis property guarantees that KMERF is an effective kernel (asymptotically) for independence testing.\n\nThrough experiments using synthetic data, the authors reported that independence testing with KMERF outperforms other kernel-based methods in terms of statistical power.\nThey also showed that by estimating the feature importance of the Random Forest, they can estimate the input features contributing to the independence testing.\nThis ability of estimating feature importance is considered an advantage of KMERF.\n\nSTRENGTHS:\nThe strengths of this study is on the theoretical anaysis of KMERF and the experimental results.\nFor valid independence testing, it is required to demonstrate the (asymptotic) characteristic kernel property of KMERF.\nThe theoretical proof in this study is therefore essential.\nMoreover, the experiments report that the testing using KMERF exhibits higher statistical power compared to other methods using different kernels.\nAdditionally, a unique feature of KMERF is its ability to estimate the important features contributing to the testing by examining the feature importance of the Random Forest.\n\nWEAKNESSES:\nA weakness of this study is the absence of a review of existing random forest-based kernels.\nFor example, [7] presents a kernel based on random partitions similar to KMERF.\nMoreover, measuring the similarity of two data points using the number of trees with the same leaf node has been used for several data analysis tasks, and packages such as rfProximity have been publicly released.\nWhen compared to these existing methods, Steps 1 and 2 of KMERF can be considered equivalent to rfProximity.\n\nThe absence of the aforementioned review poses a problem in properly evaluating the novelty of this study.\nIn fact, due to the similarity with rfProximity, the novelty of this study lies not in the random forest-based kernel itself (Steps 1 and 2), but rather in the application to testing in Steps 3--5, as well as demonstrating the asymptotic characteristic kernel property of KMERF.\n\nQUESTIONS:\n* Q1. Please reveiw the existing random forest / partition-based kernels.\n* Q2. What is the novelty and the advantage of KMERF compared to the kernels in Q1?\n\n---\nI would like to thank the authors for the detailed reply. The proposed updates look reasonable.\n\nLIMITATIONS:\nThe authors mentioned some possible future directions that are not addressed in the current study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}