# Feature Screening via Mutual Information Learning Based on Nonparametric Density Estimation

Authors: Shengbin Zhou; Tao Wang; Yejin Huang
Journal: Journal of Mathematics
 (2022)

---

## Abstract

With the advent of the era of big data, feature selection in high- or ultra-high-dimensional data is increasingly important in statistics and machine learning fields. In this paper, we propose a marginal utility measure screening method MI-SIS based on mutual information. The proposed marginal utility measure has several appealing features compared with the existing independence screening methods. Firstly, the proposed procedure is model-free without specifying any relationship between the predictors and the response and is valid under a wide range of model settings including parametric and nonparametric models. Secondly, it is suitable for various combinations of the continuous and categorical of predictors and response in our new method. Finally, the new procedure has a good performance in discovering a weak signal in the finite sample and its computation is simple and easy to implement. We establish the sure screening property for the proposed procedure with mild conditions. Simulation experiments and real data applications are presented to illustrate the finite sample performance of the proposed procedures.


---

## Body

## 1. Introduction

With the development of the information and technology, more and more data were collected in many scientific areas. The number of predictorsp can be bigger than the number of samples n. Theoretically, classical results allow that p may diverge at exponential rate of the sample size. The difficulty in ultra-high-dimensional data analysis is obvious; the most typical difficulty is a large amount of calculation. Fan and Lv [1] proposed a relatively new approach to deal with this problem; they proposed a sure independent screening (SIS, hereafter) method based on the Pearson correlation. Since the work of Fan and Lv [1] on sure independent screening, there has been a lot of work on feature screening. Wang [2] proposed Forward Regression for linear models; Fan et al. [3] designed a screening method based on the marginal likelihood estimators for generalized linear models and robust regression. Fan and Song [4] established the sure independence screening property under the background of generalized linear models. Within the scope of the ultra-high-dimensional nonparametric modeling, various feature screening methods have also been proposed. Those methods include but are not limited to the following: nonparametric independence screening (Fan et al. [5]), conditional correlation sure independence screening (Liu et al. [6]), iterative nonparametric independence screening (Fan et al. [7]), and others. All these methods are based on specific model assumption.The model-based screening methods have a good performance when the underling model is correctly specified. However, it is challenging to specify a correct model for ultra-high-dimensional data analysis. Model-based screening methods may perform poorly in the case of model misspecification. In order to overcome this problem, great efforts have been made to relax the model assumption and make the screening methods less model-dependent. Thus, new model-free screening methods have been proposed in the latest literature. See the works of Zhu et al. [8], Li et al. [9], He et al. [10], Mai and Zou [11], Shao and Zhang [12], Cui et al. [13], and others. In particular, Zhu et al. [8] proposed a sure independent ranking and screening (SIRS) method to screen predictors in multi-index models. They further showed that SIRS enjoys the ranking consistency property. Li et al. [9] developed a distance correlation based screening methods (DC-SIS). DC-SIS does not require model specification, and it can deal with grouped predictor variables. They further demonstrated that DC-SIS procedure possessed the sure screening property. Shao and Zhang [12] proposed a martingale difference correlation based screening method (MDC-SIS), which is also a model-free screening method; MDC-based screening methods can extend to quantile regression, and the sure screening property was also established.In ultra-high-dimensional classification problem, Cui et al. [13] proposed a screening method for classification problem (MV-SIS); this method can be applied to the situation where one of the predictors and the response are categorical. Huang et al. [14] proposed a screening method based on Pearson Chi-square (PC-SIS); PC-SIS is applicable in case the response and the predictors are categorical; by categorizing the continuous variables, both MV-SIS and PC-SIS have a wider applicability. We were inspired by the simplicity of MV-SIS and PC-SIS; in this paper, we proposed a new screening method based on mutual information. It has advantages over PC-SIS and MV-SIS. It does not require categorizing the continuous variables; categorizing the continuous variables will lose some information especially in small samples case and we do not know how many categories we should use to categorize the continuous variables in advance. Cui et al. [13] thought that the number of categories could be treated as tuning parameter, and they could be determined by cross validation; obviously, the calculation will increase especially in the higher-dimensional case.Recently, the studies on the independent screening methods are still booming. We just list a few relevant researches; for example, Zhang et al. [15] proposed a Gini correlation screening (GCS) method to select the important variables in ultra-high-dimensional data. Zhou and Zhu [16] proposed a modified martingale difference correlation to improve some drawbacks of martingale difference correlation. Dai et al. [17] proposed a feature selection method based on kernel density estimation for interval-valued data. An et al. [18] proposed a new model for supervised multiclass feature selection which has the l2,1−norm in both the fidelity loss and the regularization terms with an additional l2,0−constraint. Cuong et al. [19] established fundamental qualitative properties of the minimum sum-of-squares clustering problem and proved that the problem always has a global solution and the global solution set is finite. For more details, we refer to the selective survey by Kamolov [20, 21].In this paper, we propose a new model-free screening method which has a wider application, the mutual information based screening method; we refer to our method as MI-SIS. Table1 shows the application scope and algorithm complexity of different feature screening methods. We systematically study the theoretical properties of MI-SIS and establish the sure screening property for the proposed procedure with mild conditions. The new procedure has a good performance in discovering a weak signal in the finite sample. MI-SIS is comparable with MV-SIS and PC-SIS corresponding to the application scope of these methods. In the case that both the predictors and response are categorical variables, the MI index is similar to the PC index proposed by Huang et al. [14]; thus, the efficiency of MI-SIS will be the same as that of PC-SIS in this situation. Moreover, to enhance the performance of the proposed method in finite sample, we conduct three Monte Carlo simulations and conduct a real data analysis. MI-SIS has a good performance in all the simulations.Table 1
Application scope and algorithm complexity of different feature screening methods.
Y continuousY continuousY categoricalY categoricalComplexityX continuousX categoricalX continuousX categoricalMV-SIS✓✓ORn2pPC-SIS✓OK2pMI-SIS✓✓✓✓On3pThe rest of this paper is organized as follows. In Section2, we propose MI-SIS procedure and further study the theoretical properties of the novel approach. In Section 3, we conduct three Monte Carlo simulation studies to examine the performance of MI-SIS in finite samples, especially in very small sample case. We also analyze real data, and the result is very impressive. All technical proofs are given in Appendix A.


## 2. Independence Screening Using Mutual Information

### 2.1. Mutual Information

The mutual information between two random variablesX and Y is defined in terms of their joint probability distribution pX,Y as(1)MIX;Y=∫px,ylogpx,ypxpydxdy.IX;Y is always nonnegative and IX;Y=0 if and only if X and Y are independent.The MI marginal measure can be estimated by lettingw^=M^IX,Y. The estimator of the mutual information based on a nonparametric density estimator is illustrated in the following. Let Xi,Yi:1≤i≤n be a random sample of size n from the population X,Y. We assume that X,Y has continuous joint pdf. Define(2)p^x=1nh∑i=1nKx−Xih,p^y=1nh∑i=1nKy−Yih,p^x,y=1n∑i=1n1h1h2Kx−Xih1,y−Xih2,w^=M^IX,Y=∑i=1n∑j=1np^Xi,Yjlogp^Xi,Yjp^Xip^Yj,where the function K⋅ is kernel function, h and hii=1,2 are bandwidth in nonparametric density estimate, and in practice the kernel functions we often use are Gaussian kernel function and Epanechnikov kernel function. MI marginal measure under other circumstances is given in the following. When X is continuous and Y is categorical,(3)w^=M^IX,Y=∑i=1n∑j=1Kp^Xi,jlogp^Xi,jp^Xip^j,where p^Xi,j=p^jp^Xi|Y=j and p^j=∑i=1nIYi=j/n; when X is categorical and Y is also categorical,(4)w^=M^IX,Y=∑k1=1K1∑k2=1K2p^Xk=k1,Y=k2logp^X=k1,Y=k2p^X=k1p^Y=k2,where p^X=k1,Y=k2=1/n∑i=1nIXi=k1IYi=k2. In this case, MI index is very similar to PC index [14]; when X is categorical and Y is continuous,(5)w^=M^IX,Y=∑i=1K∑j=1np^i,Yjlogp^i,Yjp^ip^Yj,where p^i,Yj=p^ip^Yj|X=i and p^i=∑j=1nIXj=i/n.


### 2.2. An Independence Ranking and Screening Procedure

We now propose a new model-free sure independence screening usingMIX,Y for ultra-high-dimensional data analysis. Let Y be the response with support Ψy and Y can be either discrete random variable or continuous random variable, and X=X1,…,Xp denotes the predictor vector, where p≫n and n is the sample size. Without specifying a regression model, define the active predictors index subset by(6)D=k:pY|Xfunctionally depends onXkfor somey∈Ψy,and define the inactive predictors index subset by(7)I=k:pY|Xdoes not functionally depend onXkfor anyy∈Ψy.With the above notation, we can specify the active predictors asXD=Xk:k∈D and the inactive predictors as XI=Xk:k∈I. Our main purpose is to accurately find the active predictors index subset D.We now set out to calculateMI of each predictor and the response, wk=MIXk,Y, k=1,…,p. Note that wk=0 only if Xk∈XI; this also implies that the predictor Xk is statistically independent of Y; thus we can use MI index as a dependence measure to screen the predictors. The MI-based method is model-free because it involves only marginal density and joint density of the random variables. This index can characterize both linear and nonlinear relationships between the response and predictors.The primary objective of feature screening in ultra-high-dimensional data analysis is to find a reduced model with a small scale which can contain the true modelD with high probability. In this paper, we propose using the index w^k to select a moderate model(8)D^=k:w^k≥cn−τ,for1≤k≤p,where c and τ are predetermined positive values. In practice, we often select the reduced model using another formula:D^∗=k:w^kis among the topdlargest of all. Obviously, Xk:k∈D^∗ are the most likely relevant predictors with the response. Thus, we can use the predictors in Xk:k∈D^∗ to estimate the true model. For ease of presentation, we call the above procedure MI-SIS procedure for short.In the following, we will establish the theoretical properties of the proposed independence screening procedure; Fan and Lv [1] and Ji and Jin [22] demonstrated that the sure screening property guaranteed the effectiveness of the class of independence screening procedure. Therefore, to establish the sure screening property for MI-SIS is essential. The three following conditions are assumed to guarantee that the MI-SIS procedure has sure screening property. They are imposed mainly to facilitate the technical proofs, although they may not be the weakest ones.(C1) Suppose that X=x1,…,xp, and xi come from the distribution Fi which is unknown but has a Lebesgue pdf fi,i=1,…,p, and some conditions present in Lemma A.3 in the Appendix.(C2) There exists a positive constant 0<κ<2, such that(9)sup1≤k≤p∑i=1nlogpXik,YipXikpYi=Onκ,a.e.(C3) There exists a positive constant c>0 and τ; the minimum MI of the active predictors satisfies mink∈Dwk≥2cn−τ.(C4) Both X and Y satisfy the subexponential tail probability uniformly in p. That is, there exists a positive constant μ0 such that, for all 0<μ≤μ0,(10)suppmax1≤k≤pEexpμXk12<∞,andEexpμYq2<∞.Theorem 1 (Sure Screening Property).
Under conditions (C1)-(C2), there exists the positive constantC1 such that(11)Pmax1≤k≤pw^k−wk≥cn−τ≤Opexp−C1n1−2τ.Further, we have that(12)PD⊆D^∗≥1−Osnexp−C1n1−2τ.In the above equation, sn is the cardinality of D.Theorem1 indicates that we can deal with the ultra-high-dimensional case with logp=On1−2τ,τ>0.


## 2.1. Mutual Information

The mutual information between two random variablesX and Y is defined in terms of their joint probability distribution pX,Y as(1)MIX;Y=∫px,ylogpx,ypxpydxdy.IX;Y is always nonnegative and IX;Y=0 if and only if X and Y are independent.The MI marginal measure can be estimated by lettingw^=M^IX,Y. The estimator of the mutual information based on a nonparametric density estimator is illustrated in the following. Let Xi,Yi:1≤i≤n be a random sample of size n from the population X,Y. We assume that X,Y has continuous joint pdf. Define(2)p^x=1nh∑i=1nKx−Xih,p^y=1nh∑i=1nKy−Yih,p^x,y=1n∑i=1n1h1h2Kx−Xih1,y−Xih2,w^=M^IX,Y=∑i=1n∑j=1np^Xi,Yjlogp^Xi,Yjp^Xip^Yj,where the function K⋅ is kernel function, h and hii=1,2 are bandwidth in nonparametric density estimate, and in practice the kernel functions we often use are Gaussian kernel function and Epanechnikov kernel function. MI marginal measure under other circumstances is given in the following. When X is continuous and Y is categorical,(3)w^=M^IX,Y=∑i=1n∑j=1Kp^Xi,jlogp^Xi,jp^Xip^j,where p^Xi,j=p^jp^Xi|Y=j and p^j=∑i=1nIYi=j/n; when X is categorical and Y is also categorical,(4)w^=M^IX,Y=∑k1=1K1∑k2=1K2p^Xk=k1,Y=k2logp^X=k1,Y=k2p^X=k1p^Y=k2,where p^X=k1,Y=k2=1/n∑i=1nIXi=k1IYi=k2. In this case, MI index is very similar to PC index [14]; when X is categorical and Y is continuous,(5)w^=M^IX,Y=∑i=1K∑j=1np^i,Yjlogp^i,Yjp^ip^Yj,where p^i,Yj=p^ip^Yj|X=i and p^i=∑j=1nIXj=i/n.


## 2.2. An Independence Ranking and Screening Procedure

We now propose a new model-free sure independence screening usingMIX,Y for ultra-high-dimensional data analysis. Let Y be the response with support Ψy and Y can be either discrete random variable or continuous random variable, and X=X1,…,Xp denotes the predictor vector, where p≫n and n is the sample size. Without specifying a regression model, define the active predictors index subset by(6)D=k:pY|Xfunctionally depends onXkfor somey∈Ψy,and define the inactive predictors index subset by(7)I=k:pY|Xdoes not functionally depend onXkfor anyy∈Ψy.With the above notation, we can specify the active predictors asXD=Xk:k∈D and the inactive predictors as XI=Xk:k∈I. Our main purpose is to accurately find the active predictors index subset D.We now set out to calculateMI of each predictor and the response, wk=MIXk,Y, k=1,…,p. Note that wk=0 only if Xk∈XI; this also implies that the predictor Xk is statistically independent of Y; thus we can use MI index as a dependence measure to screen the predictors. The MI-based method is model-free because it involves only marginal density and joint density of the random variables. This index can characterize both linear and nonlinear relationships between the response and predictors.The primary objective of feature screening in ultra-high-dimensional data analysis is to find a reduced model with a small scale which can contain the true modelD with high probability. In this paper, we propose using the index w^k to select a moderate model(8)D^=k:w^k≥cn−τ,for1≤k≤p,where c and τ are predetermined positive values. In practice, we often select the reduced model using another formula:D^∗=k:w^kis among the topdlargest of all. Obviously, Xk:k∈D^∗ are the most likely relevant predictors with the response. Thus, we can use the predictors in Xk:k∈D^∗ to estimate the true model. For ease of presentation, we call the above procedure MI-SIS procedure for short.In the following, we will establish the theoretical properties of the proposed independence screening procedure; Fan and Lv [1] and Ji and Jin [22] demonstrated that the sure screening property guaranteed the effectiveness of the class of independence screening procedure. Therefore, to establish the sure screening property for MI-SIS is essential. The three following conditions are assumed to guarantee that the MI-SIS procedure has sure screening property. They are imposed mainly to facilitate the technical proofs, although they may not be the weakest ones.(C1) Suppose that X=x1,…,xp, and xi come from the distribution Fi which is unknown but has a Lebesgue pdf fi,i=1,…,p, and some conditions present in Lemma A.3 in the Appendix.(C2) There exists a positive constant 0<κ<2, such that(9)sup1≤k≤p∑i=1nlogpXik,YipXikpYi=Onκ,a.e.(C3) There exists a positive constant c>0 and τ; the minimum MI of the active predictors satisfies mink∈Dwk≥2cn−τ.(C4) Both X and Y satisfy the subexponential tail probability uniformly in p. That is, there exists a positive constant μ0 such that, for all 0<μ≤μ0,(10)suppmax1≤k≤pEexpμXk12<∞,andEexpμYq2<∞.Theorem 1 (Sure Screening Property).
Under conditions (C1)-(C2), there exists the positive constantC1 such that(11)Pmax1≤k≤pw^k−wk≥cn−τ≤Opexp−C1n1−2τ.Further, we have that(12)PD⊆D^∗≥1−Osnexp−C1n1−2τ.In the above equation, sn is the cardinality of D.Theorem1 indicates that we can deal with the ultra-high-dimensional case with logp=On1−2τ,τ>0.


## 3. Numerical Studies

In this section, we first assess the finite sample performance of the proposed MI-SIS by Monte Carlo simulation studies. Then, we use real data to analyze the sure screening property of our proposed method. All of our simulation studies were performed in the R language.Example 1.
(X is continuous and Y is categorical). In this example, we simulate a quadratic discriminant analysis problem with ultra-high-dimensional predictors by following the similar idea in Cui et al. [13] or Pan et al. [23]. Our simulation example is slightly different from theirs; we conduct a quadratic discriminant analysis in which the categorical response Y comes from two distributions which have very small difference. We generate Y from a discrete uniform distribution with R categories, where PYi=r=1/R, with r=1,…,R; given Yi=r, the ith predictor Xi is then generated by letting Xi=μr+εi+ηi, where μr=μr1,…,μrp=EXi|Yi=r is the mean of the r categories, and its rth component μrr=μ, but its other components are all zeros. μ is relatively a small number; in this example, we conduct μ≤0.1, and εi=εi1,…,εip are the p−dimensional error terms, and we assume that εij∼N0,1,1≤j≤p, and ηi=0,…,ηrr,…,0, with only the rth component ηrr∼N0,3; ηrr can take other symmetric distributions, such as t-distributions; in this case, we only illustrate the simulation result with ηrr∼N0,3. The shape of the conditional density given Y is shown in Figure 1(a), and that of the marginal density of Xj,j=1,2,3, is shown in Figure 1(b).
In order to illustrate the performance of the novel approach, we compare our result with the existing methods PC-SIS [14] and MV-SIS [13]. The MV-SIS method can be directly applied in this situation. But the PC-SIS method only applies to the situation that both the response and the predictors are categorical variables; in this case, we need to categorize the predictors. Cui et al. [13] proposed a specific procedure to categorize continuous variables. The procedure can be described as follows: assuming that Xj is a continuous variable, we define a new vector Xj∗ using the percentiles τ1,…,τkn of Xj; let Xj∗=kIτk≤Xij≤τk+1,i=1,…,n,k=1,…,kn, where I⋅ denotes an indicator function. Then we immediately face a problem of how many categories we should use in categorizing Xj. In their paper, they suggest kn=On1/5. In this simulation, we take R=2,n=50,p=500 and R=2,n=100,p=1000 separately, and, in each case, we let μ=0.05 and kn=cn1/5,c=1,2,3. We repeat each experiment 100 times and define some evaluation index to illustrate their performance. MMMS is short for the median of the minimum model size to include all the active predictors; we also report the robust estimate of its standard deviation RSD (=IQR/1.34, IQR stands for interquartile range) in the parentheses. Pj denotes that an individual active predictor is selected for a given model size d=n/logn in the 100 simulations, where x denotes the integer part of x. Pa denotes that all active predictors are selected for a given model size d in the 100 simulations. The MMMS is an index to assess the model complexity of an underling procedure. The closer to the true model size it is, the better the screening procedure is. The sure screening property ensures that Pj and Pa are asymptotic to 1 when the estimated model size d is sufficiently large. We report the detailed result in Table 2.
Table2 implies that the procedure MI-SIS performs reasonably well when the sample size is relatively small; thus, we can conclude that MI-SIS can capture more subtle signals in this situation. Table 2 also shows that the PC-SIS and MV-SIS have a poor performance in this situation, and the results are basically unchanged, despite the fact that we select three kn values for PC-SIS. With the sample size becoming large, we can find that all the three procedures have a good performance. But the result of PC-SIS is related with the selected kn; note that kn cannot be too small.Figure 1
(a) The conditional density ofX1, PX1|Y=1, and the conditional density of X2, PX2|Y=2; note that PX1|Y=1 is identical to PX2|Y=2, so we only draw a line. (b) The marginal density of all the predictors. Under our assumption, the marginal densities of Xj are nearly all the same except a small mean difference.
(a)(b)Table 2
Simulation result for Example1.
MethodCategoriesP1P2PaMMMSP=500n=50d=12PC-SISc=10.100.070.01265.0 (137.3)c=20.640.690.4545.5 (17.9)c=30.750.790.5914.0 (8.2)MV-SIS—0.350.390.1873.0 (85.8)MI-SIS—0.980.950.942.0 (0.7)n=100d=21PC-SISc=10.100.060.01304.5 (126.3)c=20.970.960.932.0 (0.7)c=31.001.001.002.0 (0.0)MV-SIS—0.900.920.834.0 (1.5)MI-SIS—1.001.001.002.0 (0.0)P=1000n=50d=12PC-SISc=10.060.050.01393.0 (194.0)c=20.550.590.30127.0 (126.9)c=30.640.640.4099.0 (22.4)MV-SIS—0.160.150.02285.5 (96.3)MI-SIS—0.920.910.847.0 (4.4)n=100d=21PC-SISc=10.060.080.00453.0 (97.0)c=20.960.950.922.0 (0.7)c=31.000.980.982.0 (0.0)MV-SIS—0.840.820.729.5 (4.5)MI-SIS—1.001.001.002.0 (0.0)The user-specified numberkn=cn1/5.Example 2.
(both X and Y are continuous). This example is designed to compare the performance of MI-SIS with those of PC-SIS and MV-SIS in the case that both the response and the predictors are continuous variables. We consider the two following models:(2.a)Y=5X1+4X2+3X3+ε(2.b)Y=5X1X2+3IX2<0+4sinX3+εwhere (2.a) is a linear regression; the relationship between Xi,i=1,2,3, and Y is also linear, while in model (2.b) the relationship between Xi,i=1,2,3, and Y is nonlinear, and IX2<0 is an indicator function. In addition, model (2.b) contains a sine function of X3 and an interaction term X1X2. All of the three methods are model-free; thus they can be directly applied in these two models. But we need to categorize the continuous response for MV-SIS and categorize both the continuous response and the predictors for PC-SIS. In this example, we set kn=4 when we categorize the continuous variables. Let n,p=100,500 and εi∼N0,1, and, for each model, we consider two scenarios to assess the performances of the three methods: covX=I and covX=∑, where ∑=0.5i−j. Moreover, in order to show defects of categorizing, we use formula (3) to calculate MI; we also need to categorize the continuous response in this case, and we name this method MI-SIS2 in Table 3. Table 3 presents the simulation result for Pj,j=1,2,3, and Pa. The performances of the MV-SIS, PC-SIS, MI-SIS, and MI-SIS2 are very similar in model (2.a). But MI-SIS outperforms the other procedures in model (2.b). We can conclude that MI-SIS has a better performance than other procedures in the case where both the response and the predictors are continuous variables.Table 3
The proportions ofPj and Pa in Example 2.
ModelParametercov (X)MethodP1P2P3Pa(a)p=500n=100d=21k=4∑=IPC-SIS1.000.960.690.65MV-SIS1.001.000.870.87MI-SIS21.000.880.730.55MI-SIS1.000.980.580.57∑=0.5i−jPC-SIS1.001.000.980.98MV-SIS1.001.001.001.00MI-SIS21.001.000.960.96MI-SIS1.001.001.001.00(b)p=500n=100d=21k=4∑=IPC-SIS0.350.590.990.20MV-SIS0.390.561.000.28MI-SIS20.460.640.990.29MI-SIS0.991.000.940.93∑=0.5i−jPC-SIS0.830.800.910.57MV-SIS0.420.480.980.25MI-SIS20.800.820.920.58MI-SIS1.001.000.910.91Example 3.
(both X and Y are categorical). In this case, our proposed index reduces to the form in (4); in order to assess the effect of our index with PC-SIS, we borrow an example from Huang et al. [14]; in their paper, the following example was used to assess the effect of PC-SIS in the case of the predictors without interaction. The category response Yi∈1,2,3,4, and PYi=k=1/4, for every 1≤k≤4. Define the true model as ST=1,2,…,10 with ST=10. Next, conditional on Yi, we generate the predictor as PXij=1|Yi=k=θkj for every 1≤k≤4 and j∈ST. The specific values of θkj are shown in Table 4. For j∉ST, let θkj=0.5, for every 1≤k≤4.
Under this mechanism, we infer that the predictorsXj,j>10, are independent of the response, because PXij=1|Yi=k,k∉ST=θkj=0.5, and PYi=k=1/4; we can calculate PXij=1=∑PXij=1|Yi=kPYi=k=0.5; the conditional mass functions are identical to the unconditional mass functions, so Xj,j>10, and Y are independent. In this example, in order to systematically study the gradual equivalence about MI-SIS and PC-SIS proposed by Huang et al. (2014), we designed four cases and performed simulation 100 times for each case. The gradual equivalence is a very strong property which needs us to subsequently define several evaluation indexes to assess. Pa−MI denotes the proportion that all active predictors are included in D^MI∗ for a given model size d=n/logn, while Pa−PC denotes the proportion that all active predictors are included in D^PC∗ at the same model size. More specifically, we let PCS=∑ID^MI∗∩D=D^PC∗∩D/n denote the proportion of the identical correct predictors selected by the methods of PC-SIS and MI-SIS, while we let PFS=∑ID^MI∗∩Dc=D^PC∗∩Dc/n denote the proportion of the identical false selected predictors.
The detailed results are shown in Table5; the four simulation results are very similar; this result is not out of our expectation. Both the MI-SIS and the PC-SIS methods are relatively very efficient in selecting the active predictors; the values of MMMS (RSD) and Pa−MI and Pa−PC are very close to 1. The value of PCS is approximately equal to 1 as n goes to infinity; this result demonstrates that MI-SIS and the PC-SIS have gradual equivalence. Both can almost surely select the active predictors, even if they do not select all the true predictors. However, PFS has a relatively poor performance; the reason may be that when n goes to infinity, the cardinality of D^∗ will also be very large; due to the randomness, the probability of the false selected predictors will vary a lot.Table 4
Probability specification for Example3.
θkjj12345678910k=10.20.80.70.20.20.90.10.10.70.7k=20.90.30.30.70.80.40.70.60.40.1k=30.70.20.10.60.70.60.80.90.10.8k=40.10.90.60.10.30.10.40.30.60.4Table 5
Detailed simulation results for Example3.
pnMMMS (RSD)Pa−MIPa−PCPCSPFS100010010.0 (0.0)0.960.961.000.6220010.0 (0.0)1.001.001.000.67500010010.0 (0.74)0.860.860.960.3120010.0 (0.0)1.001.001.000.43Example 4.
(X is categorical and Y is continuous). In this example, we use the same setting as Example2 but add some new categorical variables. The models we considered are as follows:(2.a)Y=5X1+4X2+3X3+2D1+3D2+ε(2.b)Y=5X1X2+3IX2<0+4sinX3+2D1+3D2+εwhere Xj,j=1,2,3, are continuous variables and Dj,j=1,2, are categorical variables. We consider the following scenarios with n=100,p=500. Firstly, we generate p1 continuous variable with covX=Ip1 and covX=Σp1×p1, where Σp1×p1=0.5i−jp1×p1. Then, we generate p2 categorical variables with PDij=1=θj and θj follows the uniform distribution on 0 and 1. Note that p1+p2=p and p1/p2=2/3, 1, 0, 0. The simulation result is presented in Table 6 which shows that the MI-SIS method also outperforms its competitors when X is categorical and Y is continuous.Table 6
The proportions ofPj and Pa in Example 4.
ModelParametercov (X)MethodP1P2P3Pa(a)p=500n=100d=21k=4∑=IPC-SIS1.000.950.680.64MV-SIS1.000.990.860.86MI-SIS21.000.890.740.56MI-SIS1.000.970.590.58∑=0.5i−jPC-SIS1.001.000.980.98MV-SIS1.001.001.001.00MI-SIS21.001.000.960.96MI-SIS1.001.001.001.00(b)p=500n=100d=21k=4∑=IPC-SIS0.340.570.970.20MV-SIS0.380.541.000.26MI-SIS20.460.650.980.28MI-SIS0.981.000.950.92∑=0.5i−jPC-SIS0.840.810.900.56MV-SIS0.410.460.970.24MI-SIS20.810.830.910.57MI-SIS1.001.000.900.90


## 4. Real Data Analysis

In this section, we illustrate the proposed MI-SIS procedure with an application to detect the important features about the voice of Parkinson’s patients using LSVT dataset. The dataset is available athttp://archive.ics.uci.edu/ml/datasets/LSVT+Voice+Rehabilitation. The LSVT (Lee Silverman voice treatment) Voice Rehabilitation Dataset comes from UCI website; this dataset was created by Tsanas et al., and it was first analyzed by [24]. The goal of the study is to improve the effectiveness of rehabilitative speech treatment by appropriate statistical algorithms for LSVT Companion system. These algorithms can automatically be used for detecting whether the characteristics of the voice are acceptable or not. An efficient algorithm can automatically assess the effectiveness of the LSVT Companion system during use of software away from expert clinical guidance.This dataset contains 126 samples from 14 participants, 309 predictors, and one response. The predictors are the features of the voice. The response is a binary variable; one means “unacceptable” and zero means “acceptable.” “Unacceptable” means that a clinician thought the voice was not persisting during in-person rehabilitation treatment. More details about the predictors can be found in [24]. Therein, the authors demonstrated that the algorithm they proposed could correctly replicate the experts’ binary assessment with approximately 90% accuracy.Here, we use two penalized regression models to analyze this dataset. We first established a penalized logistic model using all the predictors by minimizing the penalized likelihood(13)β=argminβlβ=−Lβ+λ∑j=1pβj,Lβ=∑i=1nyilogpi+1−yilog1−pi,pi=eβ0+xiTβ1+eβ0+xiTβ.For the second model, we propose first applying MI-SIS to screeningd=2n/logn=30 predictors; thereafter, we extend the predictor variable space by adding the interaction terms of the screened predictors. Then we apply the penalized logistic model to the new feature space, and in this way, we can explore the nonlinear relationship of the screened predictors. In the paper, these two models will be referred to as penalized-logistic model and MI-SIS-penalized-logistic model, respectively, for simplicity.The top-right and bottom-right figures in Figure2 show the penalized-logistic coefficient paths about the two models. The top-left and bottom-left figures are the CV error for each λ; the hyperparameter λ is selected by 5-fold cross validation, and the best λ will be obtained at the minimum of the CV error curve. We summarize the classification result in Table 7. From the confusion matrix, we conclude that both the penalized-logistic model and MI-SIS-penalized-logistic model have a better performance. For penalized-logistic model, it finally selects only 13 predictors, with the best λ=e−3.4, and its classification rate is 91.27%; for MI-SIS-penalized-logistic model, it finally selects 25 predictors on the new feature space, with the best λ=e−4.4, and its classification rate is 96.83%. This example further demonstrates that the MI-SIS with a penalized-logistic model is more enjoyable in real data analysis.Figure 2
The top-right and bottom-right figures are estimated coefficient in penalized-logistic models. The top-left and bottom-left figures are the CV error during the selection procedure of hyperparameterλ.Table 7
The confusion matrix for classification problem in Example4.
MI-SIS-penalized-logistic modelPenalized-logistic modelY010103913541383780


## 5. Discussion

In this paper, we proposed a new independent feature screening method based on the mutual information, that is, MI-SIS. The proposed procedure is model-free, and the sure screening property was established when the number of the predictors diverges with an exponential rate of the samples. The new procedure has a good performance in discovering a weak signal in the finite sample. Similar to Fan and Lv [1], we select a cutoff d for MI-SIS. How to choose d is a very important and tough problem. How to choose d for MI-SIS more reasonably is a good topic desiring more discussion and reasonable d plays an important role in feature screening methods.Theoretically, we ignore the marginal dependence between the predictors; marginal dependent problem may be a trouble during the feature screening procedure. How to deal with the marginal dependence between the predictors remains a question. Similar to the ISIS, we adopt the idea to MI-SIS and develop an iterative procedure about MI-SIS. Due to the defect of nonparametric estimation in small sample, the iterative MI-SIS performs poorly, so we do not post the result in this paper. To overcome the marginal dependence problem, more novel research needs to be developed.



---

*Source: 7584374-2022-02-08.xml*