Supervised Random Feature Regression via Projection Pursuit
Authors: Jingran Zhou
Ling Zhou
shaogao lv
Venue: Submitted to ICLR 2023 (2023)
Keywords: Random Feature
multi-kernel
projection pursuit
semi-parametric regression
neural networks

Abstract: Random feature methods and neural network models are two popular nonparametric modeling methods, which are regarded as representatives of shallow learning and Neural Network, respectively. In practice random  feature methods are short of the capacity of feature learning, while neural network methods lead to computationally heavy problems. This paper aims at proposing  a flexible but computational efficient method for general nonparametric problems.  Precisely, our proposed method is a feed-forward two-layer nonparametric estimation, and the first layer is used to learn a series of univariate basis functions for each projection variable, and then search for their optimal linear combination for each group of these learnt functions. Based on all the  features derived in the first layer, the second layer attempts at learning a single index function with an unknown activation function. Our nonparametric estimation takes advantage of both random features and neural networks, and can be seen as an intermediate bridge between them.

Under review as a conference paper at ICLR 2023
SUPERVISED RANDOM FEATURE REGRESSION VIA PROJECTION PURSUIT
Anonymous authors Paper under double-blind review
ABSTRACT
Random feature methods and neural network models are two popular nonparametric modeling methods, which are regarded as representatives of shallow learning and Neural Network, respectively. In practice random feature methods are short of the capacity of feature learning, while neural network methods lead to computationally heavy problems. This paper aims at proposing flexible but computational efficient method for general nonparametric problems. Precisely, our proposed method a feed-forward two-layer nonparametric estimation, and the first layer is used learn a series of univariate basis functions for each projection variable, and then search for their optimal linear combination for each group of these learnt functions. Based on all the features derived in the first layer, the second layer attempts at learning a single index function with an unknown activation function. Our nonparametric estimation takes advantage of both random features and neural networks, and can be seen as an intermediate bridge between them.
INTRODUCTION
Kernel methods are one of the most powerful methods for nonlinear statistical learning problems attributed to their excellent statistical theories and flexible modeling framework. Using the randomized algorithms for approximating kernel matrices, random feature (RF) models attract increasing attention due to that they significantly reduce the extensive hand tuning form the user for training. but obtain similar or better prediction accuracy with limited data size compared
neural network models (Du et al.) 2022 Zhen et al. 2020 The RF model can be traced back to the work of (Rahimi & Recht 2007), and was successlully developed by (Li et al. 2019b). To be specific, for observations R² E R. RF models consider linear combination over set of prespecified nonlinear functions on a relatively low-dimensional randomized feature space predict y. That is,
(Ji,
y f(x.)
oyal(x, 0/1/v))=vvvvv 6x,
where N + 00, (ax, x) = Σ/= ajZy, and a(-) is a pre-specified functioo, like Relu or the Sigmoid function. Here, 0, is chosen randomly from a prespecified distribution, say, a unit ball, i.e. 0 Unif(S (VP)), where (rd-1)(r) denotes the sphere of radius r in d dimensions, and r = vd. Model equation [ jinoolvss unknown parameters j..j 1,... .N only. The coefficients a in the RF model can be estimated using the following ridge regression:
NA
&(A) arg min
(0,,
|a
(2)
Let Far(O) (s()) >N ((0.)) : a N). where θ E NxP is matrix whose i-th row is the vector 0,. When the number of random features, N, goes to infinity, under suitable bound on the Eg norm of the coefficients. FRF reduce to certain Reproducing Kernel Hilbert Space (RKHS) (Luu et al. 2020 Specifically, the ridge regression over the function class converges to kernel ridge regression (KRR) with respect to the kernel: H RT (x, X2) := h ((x), E ((( x1) (()) x Here, the expectation is with respect to 0. Clearly, distinct distributions generating 0, and different activation functions induce different RKHS spaces. For examples, when O follows a standard multivariate normal distribution, and the activation function is the ReLU activation (x)) maz(0, x), the kernel corresponds the first order arc-cosine kernel. Another example

---

Under review as a conference paper at ICLR 2023
is, if the activation function is a (2)) |cos (x).
sin(z))", the kernel corresponds to the Gaussian kernel (Rahimi & Recht 2007 Liu et al. 2020p. According to Bochner's s theorem, the spectral distribution μ4 of stationary kernel k is the finite measure induced by a Fourier transform, ie., k(æ - x') = f exp (i0 (æ: - x²)) p& (d00) However, it is known that the distribution and the activation function may meet misspecification issues on the function space leading to inefficient or even wrong Ssiimation(Sihha & Duchi 2016) Derakhshani et al. (2021).
Note that general kernel x'x, describes the distance || x'1l who converges constant quickly as the dimension increases (Liu el al. 2020). Such kind of locality in terms of stationary and monotonic properties result in that they can not reveal more important information in the feature spaces, which largely restricts the performance of kernel methods in complex tasks (Xue et al.) 2019). The RF models overcome this issue with the induce of the coefficients O and its associated spectral distribution. In specific, the RF model learns a kernel function based on the fixed activation function (+) indexed by (approximately) infinite random parameters from prespecified distribution. In terms of the algorithm and implementation. the RF model improves the quality of approximation and reduces the requirement on time and space compared with traditional kernel approximation methods (Liu et al. 2020). This is because that the RF model is able to map features into new space where the dot product can approximate the kernel accurately, thus improving the quality of the approximation (Yu et al. 2016). Comparing to other kernel methods that mapping æ: to high dimensional space, RF uses a randomized feature map to map x: to a low-dimensional Fuclidean inner product space. Consequently, we can simply use linear learning methods approximate the result of the nonlinear kernel machine (Rahimi & Recht, (007). which saves computation time and reduces computation complexity. Also, unlike Nystrom methods or other data dependent methods, RF is a typical data- independent method with an explicit feature mapping. Data-ind independent implies that RF does not need large samples to guarantee its approximation property (Liu et al.) 2020). However, still fails to provide satisfactory performance for complex tasks due to its representing of a simple stationary kernel only. In contrast, sampling from a mixture distribution would bring in extra computational complexity (Avron al.) (017)) On the other hand, recently, some work have been done via the kernel Neural Network (KDL), a combination of kernel methods and Neural Network, to overcome the limitation of the locality (Xue et al. 2019), , and adopt the kernel trick make computation tractable. In particular, KDL methods incorporate Neural Network methods to kernel functions, i.e., (g((r, 0), g(x 0)), where gax, 0) is non-linear mapping given by deep architecture. KDL trains a deep architecture g(: 0) indexed by finitely many fixed parameters and then plugs it into a simple kernel function such Gaussian kernel. In this way, KDL adaptively estimates basis functions with finitely many parameters at the price of requiring lots of hand tuning work (lack of a principled framework to guide parameter choices), and thus a large number of data size needed. In this paper, following similar spirit of KDI., we develop novel supervised RF method (SRF) to overcome the local kernel's limitation by first adaptively estimating basis functions through (approximately) infinite tuning-frre free kernel techniques based on low-dimensional variables in the form of (xx 0) with 0 from simple distribution, and then adaptively estimating the corresponding weights and the unknown link in supervised way. Most importantly, with the incorporation of the information from the outcome to learn the basis functions, the proposed SRF has excellent predictive performance with the limited data size, in addition to the advantage of being interpretable, and hand-tuning free. It is worth noting that standard RF only has one single layer, which may not thoroughly express the complexity of the data. Instead, SRF includes two layers, which makes it have stronger ability of expression. Moreover, unlike KDL, which only introduces the information of y at the last layer, SRF incorporates the information of y at each layer, leading to higher prediction power without abundant layers. This idea is very similar to the idea of Conditional Variational Autoencoders( for good performance on limited data size and being energetic efficient (Kingma Welling 2013) Sohn et al., 2015). Energetic efficiency is an important aspect of the SRF approach. Compared to CVAE, the proposed SRF method enjoys easier interpretation benefit via its flexible semi-parametric structure.
which is also known
The proposed SRF has the following contributions: First, computational simplicity. Conventional RF models including training the random features in the implicit kernel learning (Li et al.) 2019a). choosing random features via kernel alignments in kernel allegement method (Sinha & Duchi 2016
Cortes et al. 2010)) choosing random features by score functions in the kernel polarization method (Shahrampour et al.) 2018), among others, require a huge computational burden. Instead, the SRF model generates the random features randomly from a simple pre-specified distribution. In comparison
2

---

Under review as a conference paper at ICLR 2023
[( to single hidden-layer neural networks (NN-1.(Rumelharr et al. 1986)) f(x) = = by). where A is the number of units in the hidden layer. NN-1 requires to estimate pk parameters {w, 1 while RF models estimate N linear coefficients {a, J j=1- The Projection Pursuit Regression (IPPR.(Friedma & Stuetzle 1981)) combines GAM and NN-1 by estimating nonlinear functions fj
and projected directions w, simultaneously, that f(z) = Σド= (w, x), which requires extensive computations when p and/or & are large. Furthermore, it is known that we usually require large N to obtain good approximation features, .N, is large, directly estimating the combination coefficients of supervised random features using the ridge regression equation 2 is computationally burdensome. The proposed SPF divide all random features into K < N blocks. For each block, the ridge regression is adopted to obtain the outcome y. Then, PPR is used on the low-dimensional (K) predictors
the function space. However, when the number of random
initial predictions to obtain the final prediction. This step further improves the prediction accuracy by adaptively estimating the combination schemes, in addition to save computational time by avoiding directly running large dimensional ridge regression but in a scalable way. Second, model flexibility and automatic calibration Wilson et al. 2016). Similarly to generalized additive models (GAM,(Hastie 2017)). i.e, f(x) = وز RF models overcome the curse of dimensionality by mapping p-dimensional covariates into one dimensional random feature, i.c., (0.x). Different from GAM, the RF model has the capacity to model interactions between covariates using the projected direction O. The proposed SRF estimates the activation functions in a supervised way for each random feature, which avoids any subjective pre-specified fixed kernel space. It adaptively estimate each function and thus allows different function spaces each random feature. Therefore, the proposed SRF allows a more complex function space on the variables x without knowing the true space they belong to. Consequently, the proposed SRF model has more stable prediction errors in comparison to conventional RF models. Third, model simplicity. Different from multi-layer neural network, the SRF model needs two layers only to achieve good prediction accuracy. As described in the following section, the first layer is a 'nonparametric" random feature through the nonparametric regression method, and the second layer is the projection pursuit, a universal approximator in terms of theoretically approximating any continuous function in Rp very well, which is extremely useful for regression forecasting due to its semi-parametric structure. More importantly, the estimation of these two layers can be easily solved by using common statistical methods without the need for extensive manual tuning from the users. Finally, model interpretability. As is known to all, neural network is lack of a principled framework to guide choosing parameters, such as architecture, activation functions or optimizers (Wilson et al. 2016). This, combining with unidentification of parameters, leads to the uninterpretability of neural network. Fortunately, our SRF model enjoys good interpretability to some extent. For instance, as we mentioned before, RF model uses linear learning methods to replace nonlinear kernel methods. The biggest advantage of linear learning methods its interpretability in terms of the coefficient. Significant coefficients implies important directions (0, x) (Liu et al. 2020). which facilitates the interpretation and understanding the underlying important features.
The rest of the paper organized as follows. Section 2 iintroduces proposed SRF with details and algorithm. Section 3 compares the proposed SRF method with other statistical methods under various types of simulated data. Section 4 considers five RWD(Real World Data) examples to evaluate the performance of the proposed SRF method. In Section 5 we summarize this paper with concluding remarks.
2 SUPERVISED RANDOM FEATURE
Consider the problem:
Y= fo(x:) +
where x; E RPP is p-dimensional vector, and the function So is unknown. The random errors 1 S < n, are independent of each other and of x,. Assume ((E,) = 0 and E(=?) = o² < co. When the dimension p larger than 3, the curse of dimensionality in the nonparametric regression occurs. We now introduce the proposed supervised random feature model, denoted as SRF. Firstly, for each random feature, (0,, x), we calculate its prediction on the outcome Y. That is,
Y [5((0) x:)) Eig i 1,.., n.
(4)
3

---

Under review as a conference paper at ICLR 2023
where fz(-) is an unknown univariate non-parametric function. Denote its estimator as fy. an initial prediction, which can be obtained easily using any nonparametric tools, such as NearestNeihhbor(KNN),, and Kernel density estimation Kernel regression from Python package peessmodels.onnpraamee
It is worthy pointing out that for each RF (0). x), we estimate the activating function in supervised way. By doing this way, avoid the misspecification issue on the kernel space. Second, the adaptive way on the kernel space relaxes the restriction on the distribution of the random index parameter 0z. In other words, we can simply sampling 0₃ from a unit ball, and then adaptively estimate the corresponding activating function with the outcome information incorporated. Third, for each RF, the underlying kernel space may not be the same. Thus, with each function estimated independently, we actually obtain multi-kernel mixed space, which largely improve the prediction power compared to the single-kernel space especially for complex task. Secondly, we further refine the prediction in an aggregated way by minimizing the following ridge-type objective similar to conventional RF models:
N
Yi
Σ -a j=1
N p
1a12
(5)
i=1
Denote the prediction as
N fSRF-1 Σ (0,,(())
By treating each initial prediction f as candidate model, the SRF method shares similar idea as the stacking methods in model averaging literature (Yao al. 2018). That is, we aggregate each predictions fy through the weights oj. obtained by minimizing least-square type criterion. Clearly, weights ay's could be positive or negative. Instead of all positive weights in conventional modeling averaging methods, allowing positive and negative weights improves prediction power especially when candidate models do not cover the underlying true model (Arce 1998). Different from stacking methods, SRF involves random features 0, leading to a clear identification on important features whose corresponding coefficient ay is usually large, show in Simulation- Part three.
To avoid the computation complexity with large N. we further divide N into K blocks with equal dimension in each without loss of generality. Then within each block, we run equation to obtain raw predictions f(')(x,.) (x.) Ennn a (O,, x:)). Base on K predictors, f( (-), 1, K, obtain a further refined prediction by minimizing the following objective
K
) 2
Y1
9
Σ
20
(x:)
(6)
where g is an unknown nonparametric function. This step further improves prediction accuracy through the non-parametric aggregation link function g and additional weight parameters 8g. Specifi- cally, the non non- parametriic aggregation link function extracts interaction information for each features, and the product term a 3% enjoys the ability of extracting hierarchical information from each feature, which is similar to the two-layer NN. Different from the two-layer NN, who prespecifies an activation function and the final link function, the proposed SRF estimates each activating function f and the final link function g with the outcome information incorporated. Thus, it has higher prediction power due to the use of multi-kernel mixed space and flexible interaction expression with the non non-parametric form, in a supervised way. The estimator can be obtained by PPR. The final prediction is defined as
K 10 (x:)
fSRF-11 =
The entire procedure is given in the following Algorithm
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Algorithm for SRI-11
1: SRF( X; N, K [Input]
2: Randomly generate N directions 0,. 1, 1, ... N: 0j Unit(sr- /)).
3: Obtain supervised random features f.. = 1, 1, .N using equation
4: Obtain initial K raw esitmators fk K -k=1 1, 1,... K by minimizing equation
5: Obtain f(x) Ex 8k Sk (x)) by minimizing the objective equation 6
6: return f. {Output}
SIMULATION STUDIES
This section evaluates the performance of the proposed SRF method based on various
types of simulated data. We compare the prediction results with other statistical meth-
ods including Basic Random Feature Regression (Relu-I), One Layer Kernel Regression (SRF-I), Advanced Basic Random Feature Regression (Relu-II), Two Layers Kernel Re- gression Projection Pursuit (SRF-II), Random Forest, One Layer Neural Network (NN-1)
and Two Layers Neural Network (NN-2). We consider four settings for the regression function fo()): (a) Linear:
fo(X)
2X1 + X₂ + 3Xg (b) Composite: fo(X)
cos (X1 + cos( + X3 Xa/3 + Xx + cos(Xs) + 2Xx + X3 + XgX9 + X10):
(c) Nonlinear: fo(X)
(X1 X2 + X²)² 1; (d) More complex: fo(X)
cos [ + 22 + X3 sin(X)) + exp(X6)) exp(Xe + X7) + cos(Xs + X9 + X
sin + X3) Here, X; represents the ith dimension of X R".
Under all settings, we generate n = 300 data with p = 100 covariates. We generate covariates X from multivariate normal distribution, N NO0, Σ), with three correlation structures: (I) Independence, ie., Σ = I. (II) Fixed correlation structure, i.e., all off-diagonal component of E equals to 0.5. (III) Random correlation structure, that is, each off- dif-oiagonal component of Σ is randomly generated from a uniform distribution Unif(-1, The random error in the regression function Y = [o(X) + is generated from normal distribution, i.e., N(0, 0.1). We replicate each simulation 100 times.
To determine the number of random features N, extensive simulation results show that when N 12000 and N 24000 for independent and correlated covariates, respectively, the prediction accuracy keeps steady and larger N does not bring significant improvement, as show in the Figure 4 (See Appendix A). Thus, for computational simplicity, we take N 12000 and N 24000 for simplicity. The larger N required for the correlated covariates is understandable, since randomly generated RF's may be correlated and share similar information due to the correlation among covariates leading to a larger N to thoroughly capture covariates information. The regularization parameter À is determined by the model complexity. Basically, in order to ensure the stability of the model and the accuracy of the estimation, larger À is used as the model complexity increased. We compare the prediction performance in terms of predicted mean square error (MSE) .ic 1 E (0t - y and Scaled MSE,
ie v / 1. d: )2 with additional data with size 100. To ensure visualization of simulation results, we exclude 5% - 10% outliers, and the number 2 in the denominator of Scaled-MSE amplifies the results and is more conducive observing the difference in visualization.
The results for models (a)-(d) are showed in Table I and Figure | under independent covariates (I). Particularly, Table I summarizes the average of the Scaled-MSE and MSE based on 100 replications, and Figure T shows the box-plot for the Scaled-MSE. Based on Table | and Figure I. we can see that when the model is simple, as linear model in scenario (a), two-layer type methods does not have better prediction accuracy than their corresponding one-layer counterparts. Interestingly, Scaled-MSS and MSE have different preferences on models. SRF-I, SRF-II and NN-1 have smaller Scaled-MSFss while Relu-Il and Random Forest have smaller MSES. The reason for this problem may be that the linear model is too simple, and kernel regression and projection pursuit are prone to over-fitting SRF shows its advantage as the model complexity increased. For models (b) and (c). SRF-I and SRF-II have better performance than others in terms of both Scaled-MST and MSE. Also, because of the complexity of the composite function, all the two layers models perform better than their corresponding one layer counterparts. However, Neural Network do not show their advantages under models (b) and (c). Further, for more complex data model (d), Neural Network works well. SRF-1
5

---

Under review as a conference paper at ICLR 2023
and SRF-II defeat Relu-I and Relu-II in terms of Scaled-MSEB Random Forest still remains stable. is worth noting that under this scenario, the two-layer models have significantly better and more stable prediction performance than one-layer models besides Neural Network.
Table 1: MSSled-MSEs and MSEs for models (a)-(d) of various methods including Relu-I, SRF-I, Relu- II, SRF-II, Random Forest, NN-1 and NN-2. Two-layer type methods have no obvious advantages for linear models. As model complexity increases, for composite and non-linear models, SRF-I and SRF-II perform better than others for both Scaled-MSB and MSE. Further, for more complex model. SRF-I and SRF-II work well on Scaled-MSE. NN-1 and NN-2 has comparable performance under
model (d) setting.
Model
Lincar
Composite Scaled MSE
Non-Lincar
More Complex
Scaled MSE
MSB
MSB
Scaled MSE
Scaled MSE
Rolu-I
16.75
3,20
13.37
0.57
3.78
19.56
2.33
20.11
SRF-T
5.22
12.89
4.97
0.49
2.61
17.18
1.73
19.07
II
7.28
2.77
8.66
0.57
3.53
3.81
20.80
SAF 1AFF-L. 11
6,35
13.70
3.75
0.48
1.65
17.59
1.04
21.84
Random Fozest
9.20
NN-1
4.50
6.52
15.21
0.85
4.49
23.04
1.51
33.04
BN-22
7.93
8.16
8388
0.56
4.53
27.14
1.53
22.01
Use Oata
Compante Dats
Figure 1:
Box-plot of Scaled-MSEs for models (a)-(d) under independent covariates (I), Clearly, except under linear model setting (a), SRF-II has smallest Scaled-MSEs under all other models.
To show the effects of various correlation among covariates, we further compare the results under model (c) for both random and fixed covariance matrices (II) and (III). The results are shown in Figure 5 (See Appendix B). Obviously, SRF-II still has the best performance. Compared to Figure (independent scenario), both Scaled-MSEs and MSES are larger because of the effects of various correlation. Except that, all the models have similar performance under correlated covariates.
In order to show the interpretability, we consider the following two criterions under model (c). For the first criterion, we calculate the difference of sum of absolute weights covariates between the maximum absolute Omax = max; |atı and the minimum absolute Ominn = miny layl- Ideally, large absolute value of aj implies the more importance of the feature. Thus, we compare the proportion of the differences larger than out of 50 replicates for 20 a s at a time, termed as Maxmin. For comparison, we also compare the difference between randomly chosen ay and Otmin- termed as Ranmin. For the second criterion, we compare the significant w elements (the first three elements) with the non-significant elements. For comparison, we consider three non-significant w at prespecified fixed positions (Fixpos) or randomly chosen positions (Ranpos). Also, we compare the difference between randomly chosen three covariates and randomly chosen covariates (Ranran). Similarly, we calculate the proportions of the differences larger than 0 out of 50 replications for 20 a's at time. The results are show Figure 2 From Figurq2] we can see that, Ranpos and Fixpos have significantly larger proportions, up to 56% improvement average, than Ranran, indicating that important features do have larger values on w. Maxmin also has significantly larger proportion than Ranmin, up to 80% improvement average, implying that larger a, does represent an important direction. Therefore, SRF-II is meaningful to determine important directions with large value of agz.
the three important

---

Under review as a conference paper at ICLR 2023
The average times of differences larger than out of 50 replicates under five different indexes, Maxmin, Ranmin, Fixpos, Ranpos, and Ranran, respectively. Compared to Ranran and Ranmin, Larger times in Maxmin, Fixpos and Ranpos show that larger a values are related to important features.
Figure 2:
REAL DATA EXAMPLES
List of datasets. Our real data experiments consider the following datasets. All the datasets are publicly available. More details about these datasets, including the size of the data and the number of features, are provided in Table
Abalone was collected form UCI (University California-Irvine) Machine Learning Repository with data size 800. Its objective is predict the age of abalone (Number rings) based on individual abalone measurements. It contains seven features including length, diameter, height, whole weight, shucked weight, viscera weight, and shell weight.
Boston was collected from Sklearn Machine Learning Repository with data size n n= 478. This data set about Boston House Prices and is of the most famous regression task datasets. contains thirteen features, which are CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, and LSTAT(See Appendix C for details). The objective is to predict how those aforementioned features affected the house price, MEDV(Median value of owner-occupied homes in $ 1000's) ).
Wine was also collected from UCI Machine Learning Repository with data size n =1000. The white wine quality data contains eleven features, which are fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide. density, PH value, sulphates, and alcohol. These eleven independent variables are used to predict the quality(based on sensory data) of each white wine.
Auto MPG was collected from Kaggle.com This MPG data is about 393 automobile fuel consumptions in miles per gallon with three multi-valued discrete attributes and five continuous attributes. These eight attributes are MPG(miles per gallon), number of engine cylinders, engine displacement, horsepower, vehicle weight, acceleration, model year, and origin. The objective is to predict MPG based on the other seven features.
Song Popularity was collected from Kaggle.com. Recently, there has been increasing research work into the relationship between the popularity of song and its certain factors. The main goal is to predict song's 's popularity based on several factors. In this dataset, thirteen factors including song duration, acoustinness(elccronie music or not, to 1), danceability, energy, instrumentalness(pure music or not, 0 to 1), key, liveness, loudness, audio mode, speechiness, tempo, time signature and audio valence(positive or negative psychological feelings, 0 to 1), were collected. For ease of illustration, the data with key is considered leading to n= 1307 data size.
In the data preprocesses, min max normalization for each continuous variable is applied for all datasets except Auto MPG and Song Popularity, in which, z-score normalization is used.
7

---

Interpretability results. the previous simulation section, we have confirmed that SRF-II is mean- ingful to determine important directions with large value of az. Thus, for the real data examples, we identify the significant variables according to the magnitude of the absolute values of w. Particularly, Under review as a conference paper at ICLR 2023
Prediction results. All predicted MSEs and Scaled-MSEs (with standard deviation) reported in Table 2 It can be seen from Table 2 that, SRF-II has the best performance in terms of both MSEs and MSEs, with the reduction in Scaled-MSEs and MSEs compared to Relu-I ranges from 9 - 34% and 2 28%, respectively, for all datasets. With limited data size, NN-2 and NN-1 have instability issues. Particularly, for dataset Abalone, conventional RF models and the proposed SRF have comparable performance, and two-layer models do not show there advantages compared with one-layer counterparts. This is probably due to the underlying simple structure of the data. For the other four datasets, two-layer models have smaller prediction errors than their corresponding one-layer counterparts.
Table 2: Prediction results including Scaled-MSEs (with standard deviation) and MSEs for all five datasets as well as data size, number of features, and publicly available or not. In data Abalone, Proposed SRF methods do not show obvious advantage. For the other four datasets, two-layer models defeat their corresponding one-layer counterparts. The proposed SRF performs best on the dataset Boston,Auto MPG, and Song Popularity. For all the five datasets, the proposed SRF-II works particularly well. (+: Due to the extreme instability of two-layers Neural Network, part of the crror data was deleted.)
Dataset
Abalone
Boston
Wine
MPG
Song
Date. size
800
478
1000
393
1307
Numbex oz Features
13
Available (VE3/Ko)
VES
YES
VES
YIS
YES
Resulta
Scaled
MSB
MSE
Scaled
MSE
MSB
Scaled
MSB
MSE
Scaled
MSE
MSE
Scaled
MSE MSE
Kelu I
0.48 = 0.032
6.29
0.62 + 0.030
18.34
0.39
# 0.017
0.93
2.63
= 0:372
0.23
6.62
= 1,020
1.12
SRF-T
0.44 1 0.030
6.57
0.60
1 0.738
14.25
0.37
1 0:011
0.96
2.42
1::::00:00 0.372
0.20
6.01
1: L.199
1.11
II
0.44 10,016
6.82
0.48
+0.020
14.18
0.36
4 0.008
0.98
2.21
= 0.550
0.22
4.70
0.615
1.07
SR2-11 -11
0.44 + 0.020
6.32
0.47
+ 0.026
13.55
0.35
+ 0.011
0.89
1.73
- 0.608
0.25
4.35
+ 0.615
1.09
Ron Forest
1 0:001
12.18
0.001
34.62
0.000
1.97
0.001
0.24
3.84
MN-1
0.50 1 0.007
7.62
0.61
1. 0.003
19.77
0.39
1 0.002
0.58
3.29
:: 0.026
0.22
5.37
4: 0.677
0.99
NN-2
0.51 = 0.018
7.93
0.37
+ 0.040
24.86
0.36
= 0.005
0.88
3.31
= 0.238
0.28
6.89
+ 0.043
0.95
we identify significant variables by integrating ranking of 50 times for every dataset. The proportion
of each variable in the first few ranks (depending on the number of features) in each dataset is reported in Figure
Abalone Due to the characteristic of the dataset itself, there are no significant and non- significant variables, which means that each variable has certain effect on abalone age.
Boston RM,LSTAT and B are three significant variables in this dataset. The significance of INDUS and AGE were moderate. According to the analysis from github.com. LSTAT and RM have the biggest correlation coefficient with MEDV. Persons tend to have a lower proportion of low status people around their houses, and more rooms imply a bigger house. Interestingly, we found that B is also an important factor for MEDV unlike what others have found. Other variables have less obvious effects on MEDV.
Wine In this dataset, alcohol, volatile acidity, and chlorides are relatively significant com- pared to the others. Because of the nature of wine, the best quality is achieved by a balance of all variables, there are no particularly significant variables in this dataset. First of all, it is often said that the higher the alcohol content, the better the wine. Then, too much volatile acidity can cause the wine to smell pungent. At last, the right amount of chloride can extend the life of a wine, but too much can produce an unpleasant taste.
Auto MPG For this dataset, weight and displacement have obvious significant characteristic, and year and horsepower are kind of significant. Other variables are less likely to affect the value of MPG. Based on other people's 's correlation analysis of this dataset on Kaggle.com com, the largest absolute correlation coefficients with MPG are weight and displacement. This conclusion is consistent with our results. Through common sense, we clearly know that the heavier the car, the bigger the MPG, and then the bigger the displacement, the bigger the MPG. It is worth mentioning that the year will also have an effect on the value of MPG, because the higher the year, the more serious the aging of auto parts, which will lead to the increase of MPG. The other variables are not significant because their correlation coefficients with weight, displacement and horsepower are too high.
8

---

Under review as a conference paper at ICLR 2023
Song Popularity Our results exactly the same other people's correlations conclu- sion on Kaggle.com, but they are roughly the same. Audio valence and loudness are two significant variables in this dataset. Also, acousticness, danceability and instrumentalness are kind significant. Everyone loves to listen to songs with positive psychological feeling, it is easy understand that audio valence has great influence song popularity. And, few people like loud songs, so loudness is also significant variable. Electromic music, dance music and pure music have unique audiences, so they have some influence on song popularity. However, for other variables, such as liveness tempo, the audience doesn't t pay much attention those. Therefore, they are min--inifiiant variables.
Abalone
Boston
Wire
Faet
1el
WOMC:
100
1p4
Sat
141
ederuddy
KO
enbet
1.00
610
Song Popularity
Figure 3: The proportion of each variable in the first few ranks (depending the number of features) each dataset shown. A Longer color bar in the histogram represents a higher proportion, implying the significant level of the feature. It's easy to that Abalone and Wine don't have obviously significant variables. For Boston data, RM,.STAT and B are three significant variables. For Auto MPG, weight and displacement have obvious significant characteristic. Audio valence and loudness are two significant variables in the data Song Popularity.
CONCLUDING REMARKS
This paper proposed a novel and computational efficient method for general nonparametric problems. To the best of our knowledge, we are the first to propose combining the advantages of random features and neural networks to come up with a new feed-forward two-layer nonparametric estimation method. Extensive simulation results and experimental data show that SRF-II has excellent prediction performance and good interpretation. More specifically, the proposed SRF-II improved the prediction compared to Relu-I ranges from 9 34% and 2 28% across five datasets. More importantly, SRF-II performs well with limited data size and thus is energetic efficient. However, there are still three limitations in this paper. At first, for computational simplicity, 24000 in these cases. However, optimal choices or clear criterions are not clear yet. Secondly, in this paper, we only consider the regression problem, not the classification problem. Classification problems based on RF require extra computational burden due to the non linear structure compared least squares. Nevertheless, extending SRF-II to classification problems, especially for image classification problems, are very meaningful. In the end, we did not yet consider variable selection methods to generate more simplified model, which well deserved further studies improve the prediction accuracy and computational efficiency further.
take N 12000 and N

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gonzalo R Arce. A general weighted median filter structure admitting negative weights. IEEE Transactions on signal processing, 46(12):31953222) 1998.
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In International conference on machine learning, pp. 252-262. PMLR, 2017.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadch. Two-stage learning kernel algorithms. ICML 2010 Proceedings, 27th International Conference on Machine Learning, 2010.
Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao, and Cees Snoek. Kernel continual learning. In International Conference on Machine Learning, pp. 2621-2631, PMLR, 2021.
Yingjun Du, Haoliang Sun, Xiantong Zhen, Jun Xu, Yilong Yin, Ling Shao, and Cees GM Snoek. Metakernel: Learning variational random features with limited labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Jerome H Friedman and Werner Stuetzle. Projection pursuit regression. Journal of the American statistical Association, 76(376):817-833, 1981.
Trevor Hastie. Generalized additive models. In Statistical models in S, pp. 249-307. Routledge, 2017.
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Chun-Liang Li, Wei-Cheng Chang, Youssef Mrouch, Yiming Yang, and Barnabás Póczos. Implicit kernel learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2007-2016. PMLR, 2019a.
Zhu Li, Nean-Francoi, Ton, Dino Oglic, and Dino Sejdinovic. Towards unified analysis of random fourier features. International conference on machine learning. pp. 3905-3914. PMLR, 2019b.
Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approximation: A survey on algorithms, theory, and beyond. arXiv preprint arXiv:2004.11155 2020.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
David Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, 323(6688))533-5366 1986.
Shahin Shahrampour, Ahmad Beirami, and Vahid Tarokh. On data-dependent random features for improved generalization in supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Aman Sinha and John C Duchi. Learning kernels with random features. Advances in neural information processing systems, 29, 2016.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Hui Xue, Zhe Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In IJCAI, pp. 4019--004025 2019.
Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Using stacking to average bayesian predictive distributions (with discussion). Bayesian Analysis, 13(3)917-1007, 2018.
10

---

Under review as a conference paper at ICLR 2023
Felix Xinnan X Yu, Ananda Theertha Suresh, Krzysztof M Choromanski, Daniel N Holtmann Holtmann-Rice, and Sanjiv Kumar. Orthogonal random features. Advances in neural information processing systems, 29, 29,209, 2016.
Xiantong Zhen, Haoliang Sun, Yingjun Du, Jun Xu, Yilong Yin, Ling Shao, and Cees Snock. Learning to learn kernels with variational random features. In International Conference on Machine Learning, pp. 11409-1 1419. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Figure 4: The box figures of scaled predicted mean square error under model (c) for independent covariates generated from (I) with N 6000, 12000, 18000 and correlated covariates generated from (III) with 12000, 24000, 36000.
B APPENDIX
RenierDeeseeddd bee
Coantend Secber
Figure 5: Box figures for Random and Fixed correlated Non-linear Data. SRF-II continues to perform well in both cases. Both Scaled-MSEs and MSES are larger because of the correlation of covariates.
C APPENDIX
Boston House Prices Dataset Attribute Information (in order):
-CRM
per capita crime rate by town proportion of residential land zoned for lots over 25,000 sq.ft.
-ZN
INDUS
proportion of non-retail business acres per town
CHAS
Charles River dummy variable (= 1 if tract bounds river; otherwise) nitric oxides concentration (parts per 10 million) average number of rooms per dwelling
NOX
RM
AGE
proportion of owner-occupied units built prior to 1940 weighted distances to five Boston employment centres index of accessibility to radial highways full-value property-tat tax rate per $10,000 pupil-teachs ratio by town 1000(Bk - 0.63) where Bk the proportion of blacks by town % lower status of the population
DIS
RAAD
TAX
PTRATIO
-SSAT
12

---

Under review as a conference paper at ICLR 2023
MEDV
Median value of owner-occupied homes in $ 1000's
13


Reviews:
[{"review_id": "s2r_-feQjM", "reviewer": "Reviewer_hYRF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an adaptation of the random feature method approach for kernel methods akin to a two-layer neural network. ", "summary_of_the_review": "One would expect that a dual-layer extension of RFs would perform better than a single layer version. Furthermore there are no analytical contributions for the extension proposed. Finally, several portions of the paper are not sufficiently detailed to make a clear argument for the claimed new benefit of interpretability. ", "strength_and_weaknesses": "The literature on RF models uses a single layer for analytical tractability. This manuscript does not provide any analytical results on the two-layer extension.\n \nThe claimed interpretability aspects are not clear to me - the inherent use of random features would obscure the interpretation of the learning process. From the description of the numerical experiment it appears that the approach to interpretability may involve multitudes of RF models, but this does not seem like a comparable approach to that of other methods that rely on a single learned model.\n\nThe other claimed contributions are also present in the original RF method.\n\nTable 1 and Figure 1 are not surprising - two-layer models perform better than one when the learning problem is sufficiently difficult.\nFigures 1 and 3 should be larger (unreadable text).\n\nTable 2 also does not show a clear distinction between existing methods and the proposed on performance.\n\nUnfortunately it is not clear to me how these weaknesses can be addressed in a revision.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g4OMsBfEu2n", "reviewer": "Reviewer_iHFk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for boosting the effectiveness of kernelized Ridge regression by first learning a set of random features through a deep-learning inspired preprocessing step, then applying kernelized Ridge regression to the learned features.", "summary_of_the_review": "The paper should be proofread, and more intuition and more experiments should be added to argue the benefits relative to vanilla deep learning or expressive kernel methods.", "strength_and_weaknesses": "\n- **Strength:** While much work has been done on constructing arbitrarily expressive kernels [1, 2] and scaling up kernel-based regression methods, more work is needed to make learning more efficient in kernel methods. Borrowing from deep learning to learn expressive families of kernels, as attempted by this paper, is laudable.\n\n\n- **Weaknesses:**\n\n*Clarity:* The paper is hard to read and contains too many typos.\n\n*Comparison to Deep Learning:* It is unclear why this approach would perform better than vanilla deep learning. A strong intuition and substantially more experiments are needed to make this case.\n\n*Comparison to expressive kernel:* This paper is missing much of the literature on expressive kernel methods, especially Generalized Spectral Kernels [1]. In particular, [2] introduced kernel families (namely GSKs) that are general-purpose in that they contain kernel that can perform as well as any other kernel not in the family, stationary or non-stationary. Additionally, a flurry of methods have been developed to scale up kernel regression. It would have been interesting to discuss what benefits this approach has over GSKs.\n\n- **Additional Comments:** Page 1: No condition is required for Eq (1) to be Kernel Ridge regression. Basis function regression with Ridge penalty is always kernelized Ridge regression. The kernel implied by Eq (1) is random and the behavior as $N \\to \\infty$ pertains to the convergence of the random kernel to a deterministic kernel.\n\n\n[1] Samo, Y.L.K. and Roberts, S., 2015. Generalized spectral kernels. arXiv preprint arXiv:1506.02236.\n[2] Samo, Y.L.K., 2017. Advances in kernel methods: towards general-purpose and scalable models (Doctoral dissertation, University of Oxford).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bojnpyuuzr", "reviewer": "Reviewer_ZaoF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a supervised random features (SRF) regression method that combines the ideas of RF kernel learning and of (simple) neural networks (NNs) model.\nThe authors claimed that the proposed SRF approach improves a few previous efforts in that it yields better performance with a relatively small amount of data (when compared with, e.g., kernel neural networks), and can be computed very efficiently (when compared with, e.g., implicit kernel learning or kernel alignment methods mentioned at the end of the second page). However, none of these claims are well-supported by solid and rigorous theory.\nSome limited experiments were provided on a few datasets and a few simple models to illustrate the advantages of the proposed SRF approach.\n", "summary_of_the_review": "As I mentioned above, the theoretical and empirical contributions in the paper are of limited significance and novelty, and the presentation in the paper can be clearly improved. Please see my detailed comments as follows: \n* P1, abstract: my personal, and possibly naive, understanding of the neural network (NN) model is that it is parameterized by a sequence of weight matrices and bias terms, and therefore a parametric model in some sense.\n* P1, introduction: \"reduce the extensive hand tuning form the user for training\": I get confused by this sentence, does this mean classical kernel learning needs a lot of hand tuning and that can be avoided by applying RF techniques?\n* Please use $\\max$ instead of max, $\\sin$, and $\\cos$ for sin and cos activation functions, respectively.\n* P2: The author claimed that this contribution improves kernel neural networks (KDL), which need a huge number of data, by proposing a novel supervised RF approach that works well even with a limited amount of training data. This advantage, however, is only evaluated empirically, no solid theoretical arguments are provided in the paper.\n* P2: Most computational or statistical advantages of the proposed SRF approach are stated without any empirical or theoretical evaluations.\n* P2 and P3: the introduction contains extremely long and wordy paragraphs that try to discuss the advantages of the proposed SRF approach, which is hardly readable and not easy to understand. It would be helpful to at least divide them into subsections or paragraphs.\n* The figures are hardly visible, which clearly harms the readability of the paper.", "strength_and_weaknesses": "**Strength**: I do not see very strong points in the paper, from either a theoretical or empirical viewpoint.\n** Weaknesses**: There is (almost) no theory in the paper, and the baselines compared in the experiments are weak. The presentation of the paper is, in general, poor.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject