# Capped Asymmetric Elastic Net Support Vector Machine for Robust Binary Classification

Authors: Kai Qi; Hu Yang
Journal: International Journal of Intelligent Systems
 (2023)

---

## Abstract

Recently, there are lots of literature on improving the robustness of SVM by constructing nonconvex functions, but they seldom theoretically study the robust property of the constructed functions. In this paper, based on our recent work, we present a novel capped asymmetric elastic net (CaEN) loss and equip it with the SVM as CaENSVM. We derive the influence function of the estimators of the CaENSVM to theoretically explain the robustness of the proposed method. Our results can be easily extended to other similar nonconvex loss functions. We further show that the influence function of the CaENSVM is bounded, so that the robustness of the CaENSVM can be theoretically explained. Other theoretical analysis demonstrates that the CaENSVM satisfies the Bayes rule and the corresponding generalization error bound based on Rademacher complexity guarantees its good generalization capability. Since CaEN loss is concave, we implement an efficient DC procedure based on the stochastic gradient descent algorithm (Pegasos) to solve the optimization problem. A host of experiments are conducted to verify the effectiveness of our proposed CaENSVM model.


---

## Body

## 1. Introduction

Support vector machine (SVM), first proposed by Cortes and Vapnik [1], is a powerful binary classification tool and has been widely used in various fields, such as bioinformatics analysis [2, 3], industrial flaw detection [4], and financial forecasting [5]. On the one hand, the SVM can be easily understood in the geometric view, i.e., it aims to seek a single separating hyperplane for classifying datasets. On the other hand, there is a solid statistical theory basis behind to well guarantee the classification performance of the SVM [6–8]. Thus, it has been drawing much attention to study the SVM [9–15]. Although a host of literature demonstrate the advantages of support vector classifiers, there is still a room for improvement.One drawback is the sensitivity to feature noise or more specifically the instability for resampling. In fact, the SVM can be fit in the regularization framework of loss + penalty by adopting hinge loss, i.e.,lhingeu=maxu,0. Huang et al. [11] pointed out that the hinge loss-based SVM lacks resistance to feature noise and the final separating hyperplane is severely disturbed by the feature noise around the decision boundary. To tackle this problem, motivated by the quantile in the statistical field, Huang et al. [11] constructed the so-called pinball loss and applied it to the SVM to propose PinSVM. Later, Xu et al. [16] extended this idea to the twin support vector machine, which can simultaneously obtain a pair of nonparallel separating hyperplanes. There are two typical defects of the pinball loss. The first defect is the heavy optimization burden caused by the singularity of the pinball loss function at zero. Huang et al. [17] considered an asymmetric least squared loss, which is also stable for resampling but is smooth everywhere. A similar idea was studied by Liu et al. [18]. Unlike them, Li and Lv [19] utilized Chen–Harker–Kanzow–Smale function to construct a smooth approximation of the pinball loss. The second defect is the lack of sparseness. According to the pinball loss function, those correctly classified training samples still produce losses, which is completely different from the hinge loss and increases the training cost. To enhance the sparseness, Huang et al. [11] introduced the pinball loss with ϵ-insensitive zone, which can achieve sparsity and maintain the stability simultaneously. Shen et al. [20] truncated the left part of the pinball loss and proposed pin¯-SVM, providing a more flexible framework for the tradeoff between sparsity and stability. Yang and Xu [21] applied a safe screening rule for accelerating the PinSVM. More PinSVM-related work can be found in [22–27].Another drawback is the sensitivity to label noise (outliers). Sincelhingeu tends to infinite as u⟶∞, outliers often produce large losses, indicating that the resulting decision hyperplane is possibly deviated. Wu and Liu [28] suggested a truncated hinge loss, termed as ramp loss, to suppress the influences of outliers. Based on the ramp loss and motivated by the Huberized scheme, Wang et al. [29] proposed a smooth ramp loss function, which is twice differentiable and can be efficiently solved. Liu et al. [30] applied ramp loss to a nonparallel support vector machine. Tang et al. [13] combined the pinball loss with ramp loss to propose the valley loss, which is both stable for resampling and robust to outliers. For more related literature, one can refer to [31–34]. Another different strategy to improve robustness is using correntropy-induced loss (C-loss) [35]. Based on C-loss, Xu et al. [36] proposed the rescaled hinge loss. Due to the properties of an exponential function, the rescaled hinge loss is bounded and the induced support vector classifier is insensitive to label noise. Yang and Dong [37] applied the idea of the pinball loss to C-loss and proposed a new generalized quantile loss, which can be viewed as a rescaled version of the pinball loss. The generalized quantile loss inherits the stability for resampling from the pinball loss and the robustness to outliers from C-loss. Similarly, we recently proposed a joint rescaled asymmetric least squared (RaLS) loss and applied it to a nonparallel support vector machine [38]. The proposed RaLS loss is smooth everywhere and enjoys both stability and robustness.Inspired by the previous work, we construct a novel capped asymmetric elastic net (CaEN) loss and apply it to the SVM (CaENSVM) in this paper. As a generalization of the pinball loss and ramp loss, the designed CaEN loss is bounded and asymmetric, as well. However, it is more flexible than the previous truncated pinball loss functions. To demonstrate its advantages, we theoretically investigate several properties of the CaEN loss, including noise insensitivity, Bayes rule, and generalization error bound. The main contributions of this work can be summarized as follows:(i)
A novel capped asymmetric elastic net (CaEN) loss is proposed to achieve stability for resampling and robustness to outliers simultaneously. The advantages of the elastic net (EN) loss for the SVM were theoretically discussed in our recent work [3]. The derived VTUB significantly characterizes the advantage of the EN loss. Thus, it is meaningful to improve the performance of the EN loss under the framework of the SVM.(ii)
We derive the influence function (see Theorem1) to demonstrate the robustness of the CaENSVM. Though there are lots of literature on improving the robustness of the SVM by constructing a nonconvex function, they seldom theoretically show the robust property of the constructed functions. Obviously, theoretical results can ensure the effectiveness of similar works. We use the influence function in the statistics to show the robustness of our constructed CaEN loss. Our results can be easily extended to other similar loss functions, such as ramp-type losses and rescaled-type losses. We further show the influence function of the CaENSVM is bounded, so that the robustness of the CaENSVM can be theoretically explained.(iii)
CaEN loss is proved to be equivalent to the Bayes rule, and the corresponding generalization error bound based on Rademacher complexity well guarantees a good generalization capability.(iv)
CaENSVM is applied to deal with a real problem, i.e., handwritten digit recognition. Experimental results show that CaENSVM is superior to many state-of-the-art methods.The remainder of this paper is organized as follows: In Section2, we introduce several related studies. In Section 3, we first formulate the proposed CaENSVM. Then, an efficient DC procedure based on the stochastic gradient descent algorithm is implemented for optimizing the CaENSVM problem. Theoretical analysis on the properties of the CaENSVM, including noise insensitivity, Bayes rule, and generalization error bound, is carefully discussed in Section 4. We conducted lots of experiments in Section 5 to investigate the performance of the CaENSVM. In Section 6, a conclusion summarizes the main contributions and further potential directions.


## 2. Background

In this section, we review several related works. Considering a binary classification problem withn training samples and p features, let xi∈Rp×1 and yi∈+1,−1 be i-th instance and the corresponding label, respectively. All samples are reorganized as a data matrix X∈Rn×p. Without particular explanations, all vectors are in column form.


### 2.1. SVM with Elastic Net Loss

Recently, Qi et al. [39] proposed a so-called elastic net (EN) loss given as follows:(1)lENu;c1,c2=c12u2+c2u,u≥0,0,u<0,where c1 and c2 are both two positive tuning parameters. EN loss is a fusion of the standard hinge loss and the squared hinge loss [9]. Figure 1(a) shows the shapes of the EN loss with different sets of parameters.Figure 1
Different types of loss functions: (a) “EN loss” is an elastic net loss, (b) “pin loss” is a pinball loss, and (c) “ramp loss” is a ramp loss.
(a)(b)(c)Based on EN loss, Qi et al. [39] constructed the following elastic net support vector machine (ENSVM):(2)minw,b12w22+b2+∑i=1nlEN1−yiwTxi+b;c1,c2,where w∈Rp×1 and b∈R are the normal vector and the intercept of the separating hyperplane, respectively. After obtaining w and b from (2), the decision function of a new sample xnew is fxnew=signwTxnew+b, where sign⋅ is a sign function, which maps a real number to its sign and zero to zero.We can equivalently transform (2) into the following constrained optimization problem:(3)minw,b12w22+b2+c12ξTξ+c2eTξ,s.t.DXw+eb≥e−ξ,ξ≥0,where D=diagy1,⋯,yn∈Rn×n is a diagonal matrix, e∈Rn×1 is filled with ones, and ξ=ξ1,⋯,ξnT∈Rn is a slack variable. For ENSVM (3), one can clearly see that the ENSVM resembles the standard SVM [1] with c1=0 and reduces to the Lagrangian SVM [9] with c2=0. Thus, the ENSVM is more flexible. Moreover, in our recent work [3], we derived the so-called VTUB for the SVM with the EN loss to demonstrate its unique advantages. Thus, it is meaningful to improve the performance of the EN loss under the framework of the SVM.


### 2.2. SVM with Pinball Loss

Huang et al. [10] proved that the support vector classifiers with hinge-type losses, including ENSVM, are sensitive to feature noise, or specifically, are unstable for resampling. Motivated by the quantile in the statistical field, Huang et al. [10] proposed a novel pinball loss defined as(4)lPinu;τ=u,u≥0,−τu,u<0,where τ∈0,1 controls the level of stability for resampling. Figure 1(b) illustrates the shapes of the pin loss with different values of τ. As the figure shows, unlike the hinge-type losses, lPin also produces losses for u<0, which can benefit the classifier for balancing the disturbance of the feature noise around the decision boundary (see subsection 3.3 in the study by Huang et al. [10] for details).Following the method of formulating the hinge loss-based SVM, Huang et al. [10] constructed the pinball loss SVM (PinSVM) as(5)minw,b12w22+c∑i=1nlPin1−yiwTxi+b;τ,where c≥0 is a tuning parameter. We can also equivalently rewrite (5) as(6)minw,b12w22+b2+ceTξ,s.t.DXw+eb≥e−ξ,DXw+eb≤e+1τξ.Note that ifτ=0, the second constraint of problem (6) turns to ξ≥0, and PinSVM reduces to the standard SVM with hinge loss. In other words, the PinSVM can be viewed as a generalization of the standard SVM.


### 2.3. SVM with Ramp Loss

Since the values of losses produced by the hinge-type loss [1, 9, 39] and the pinball-type loss [11, 17, 18] functions tend to infinite when u⟶∞, the support vector classifiers induced by these losses are sensitive to label noise (outliers). To reduce the influence of label noise, Wu and Liu [28] truncated the hinge loss and proposed the so-called ramp loss, which is defined as(7)lRampu;s=0,u<0,u,0≤u≤s,s,u>s,where s>0 controls the truncation level. Figure 1(c) shows different shapes of the ramp loss. As compared with the hinge-type losses, ramp loss is upper bounded, such that it can weaken the disturbance of label noise.Applying ramp loss to the SVM, we can obtain RampSVM, i.e.,(8)minw,b12w22+c∑i=1nlRamp1−yiwTxi+b;s,where c>0 is a tuning parameter.


## 2.1. SVM with Elastic Net Loss

Recently, Qi et al. [39] proposed a so-called elastic net (EN) loss given as follows:(1)lENu;c1,c2=c12u2+c2u,u≥0,0,u<0,where c1 and c2 are both two positive tuning parameters. EN loss is a fusion of the standard hinge loss and the squared hinge loss [9]. Figure 1(a) shows the shapes of the EN loss with different sets of parameters.Figure 1
Different types of loss functions: (a) “EN loss” is an elastic net loss, (b) “pin loss” is a pinball loss, and (c) “ramp loss” is a ramp loss.
(a)(b)(c)Based on EN loss, Qi et al. [39] constructed the following elastic net support vector machine (ENSVM):(2)minw,b12w22+b2+∑i=1nlEN1−yiwTxi+b;c1,c2,where w∈Rp×1 and b∈R are the normal vector and the intercept of the separating hyperplane, respectively. After obtaining w and b from (2), the decision function of a new sample xnew is fxnew=signwTxnew+b, where sign⋅ is a sign function, which maps a real number to its sign and zero to zero.We can equivalently transform (2) into the following constrained optimization problem:(3)minw,b12w22+b2+c12ξTξ+c2eTξ,s.t.DXw+eb≥e−ξ,ξ≥0,where D=diagy1,⋯,yn∈Rn×n is a diagonal matrix, e∈Rn×1 is filled with ones, and ξ=ξ1,⋯,ξnT∈Rn is a slack variable. For ENSVM (3), one can clearly see that the ENSVM resembles the standard SVM [1] with c1=0 and reduces to the Lagrangian SVM [9] with c2=0. Thus, the ENSVM is more flexible. Moreover, in our recent work [3], we derived the so-called VTUB for the SVM with the EN loss to demonstrate its unique advantages. Thus, it is meaningful to improve the performance of the EN loss under the framework of the SVM.


## 2.2. SVM with Pinball Loss

Huang et al. [10] proved that the support vector classifiers with hinge-type losses, including ENSVM, are sensitive to feature noise, or specifically, are unstable for resampling. Motivated by the quantile in the statistical field, Huang et al. [10] proposed a novel pinball loss defined as(4)lPinu;τ=u,u≥0,−τu,u<0,where τ∈0,1 controls the level of stability for resampling. Figure 1(b) illustrates the shapes of the pin loss with different values of τ. As the figure shows, unlike the hinge-type losses, lPin also produces losses for u<0, which can benefit the classifier for balancing the disturbance of the feature noise around the decision boundary (see subsection 3.3 in the study by Huang et al. [10] for details).Following the method of formulating the hinge loss-based SVM, Huang et al. [10] constructed the pinball loss SVM (PinSVM) as(5)minw,b12w22+c∑i=1nlPin1−yiwTxi+b;τ,where c≥0 is a tuning parameter. We can also equivalently rewrite (5) as(6)minw,b12w22+b2+ceTξ,s.t.DXw+eb≥e−ξ,DXw+eb≤e+1τξ.Note that ifτ=0, the second constraint of problem (6) turns to ξ≥0, and PinSVM reduces to the standard SVM with hinge loss. In other words, the PinSVM can be viewed as a generalization of the standard SVM.


## 2.3. SVM with Ramp Loss

Since the values of losses produced by the hinge-type loss [1, 9, 39] and the pinball-type loss [11, 17, 18] functions tend to infinite when u⟶∞, the support vector classifiers induced by these losses are sensitive to label noise (outliers). To reduce the influence of label noise, Wu and Liu [28] truncated the hinge loss and proposed the so-called ramp loss, which is defined as(7)lRampu;s=0,u<0,u,0≤u≤s,s,u>s,where s>0 controls the truncation level. Figure 1(c) shows different shapes of the ramp loss. As compared with the hinge-type losses, ramp loss is upper bounded, such that it can weaken the disturbance of label noise.Applying ramp loss to the SVM, we can obtain RampSVM, i.e.,(8)minw,b12w22+c∑i=1nlRamp1−yiwTxi+b;s,where c>0 is a tuning parameter.


## 3. Capped Asymmetric Elastic Net Loss-Based SVM

### 3.1. The CaENSVM Model

In our recent work [3], we derived the so-called VTUB to demonstrate the unique advantages of the EN loss under the framework of the SVM. Thus, it is meaningful to improve the performance of the EN loss. Motivated by the pinball loss, we designed the following asymmetric elastic net (aEN) loss:(9)laENu;τ,θ=θ2u2+1−θu,u≥0,τθ2u2−1−θu,u<0,where θ∈0,1 corresponds to a tradeoff between L1 norm and L2 norm, τ∈0,1 is a tuning parameter controlling the bias of the penalization for positive and negative losses. Figure 2(a) illustrates different shapes of aEN losses with different sets of tuning parameters. According to definition (9), laEN can be regarded as a generalization of elastic net loss, pinball loss, and asymmetric least squared loss functions [17], since laEN reduces to lEN for τ=0, laEN becomes lPin for θ=0 and laEN is equivalent to laLS for θ=1.Figure 2
(a) “aEN loss” is an asymmetric elastic net loss. (b) “CaEN loss” is a capped asymmetric elastic net loss. (c) The convex decomposition of CaEN loss, wheres=1,τ=0.7, and θ=0.2.
(a)(b)(c)According to (9), laEN goes to infinity along with u⟶∞, so the proposed aEN loss is also sensitive to outliers (label noise). To improve its robustness against outliers, we further use capped trick to propose a capped asymmetric elastic net (CaEN) loss function, which is defined as(10)lCaENu;τ,θ,s=minlaENu,s,where s>0 is a thresholding parameter. Figure 2(b) depicts the shapes of the CaEN loss with different groups of parameters. As shown in Figure 2(b) , the proposed CaEN loss function is exactly upper bounded and concave. The truncation of the left part of the CaEN loss can increase the sparseness [13, 20], while the boundness of the right part of the CaEN loss can enhance its robustness against label noise. Detailed discussion on the properties of CaENSVM is provided in Section 4. However, the concavity of the CaEN loss often involves high optimization costs.Combining CaEN loss with the standard SVM, we propose a novel robust CaEN loss-based SVM (termed as CaENSVM), which is formulated as(11)minw12w22+cn∑i=1nlCaEN1−yiwTxi;τ,θ,s,where c>0 is the tuning parameter. We remove the intercept term b in (11) for simplicity, which can be achieved before training by centering features just like [10, 13]. After obtaining w from (11), the decision function of the linear CaENSVM for a new sample xnew is fxnew=signwTxnew.For nonlinear CaENSVM, we consider the following kernel-generated separating hyperplane [40, 41]:(12)wTϕx=0,where ϕx maps x to a high-dimensional Hilbert space. In application, we often utilize kernel trick, i.e., ϕx=KX,x, where K⋅,⋅ is a kernel function and KX,x=Kx1,x,⋯,Kxn,xT. Then, by replacing wTxi with wTKX,xi in (11), we can obtain the nonlinear CaENSVM model. For determining the class of a new sample xnew, we only need to replace xnew with KX,xnew in the linear decision function.


### 3.2. DC Algorithm for CaENSVM

The truncation for aEN loss results in a nonconvex loss, indicating that solving CaENSVM (11) involves nonconvex minimization, which is often difficult. Note that, though CaEN loss is concave, we can decompose lCaEN into the difference of two convex functions, i.e.,(13)lCaENu;τ,θ,s=laENu;τ,θ−laEN1u;τ,θ,s,where laEN1 corresponds to the so-called aEN1 loss, which is given as follows:(14)laEN1u;τ,θ,s=maxlaENu;τ,θ−s,0.Figure2(c) depicts the above convex decomposition of the CaEN loss. As the figure shows, both aEN and aEN1 losses are exactly convex. By calculating the difference of aEN and aEN1 losses, we can finally obtain the nonconvex CaEN loss. Using this property of the CaEN loss, we apply the DC (difference of convex functions) algorithm [42] to optimize problem (11).Considering the following optimization problem:(15)minΘfΘ=gΘ−hΘ,Θ∈Rm,where g and h are both convex functions on Rm. To solve problem (15), the DC algorithm turns to minimize a sequence of convex subproblems. A general framework of the DC algorithm for (15) is illustrated as Algorithm 1.Table 6
The mean accuracy (Acc.) and standard deviation (sd) with linear kernel for the PMU-UD dataset.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd2-vs-30.942 ± 0.0110.922 ± 0.0060.942 ± 0.0040.975 ± 0.0100.949 ± 0.0540.979 ± 0.0172-vs-40.947 ± 0.0260.925 ± 0.0150.925 ± 0.0200.948 ± 0.0200.951 ± 0.0220.951 ± 0.0142-vs-60.999 ± 0.0020.997 ± 0.0120.997 ± 0.0140.990 ± 0.0040.999 ± 0.0021.000 ± 0.0002-vs-70.999 ± 0.0020.992 ± 0.0090.993 ± 0.0030.993 ± 0.0060.999 ± 0.0020.999 ± 0.0022-vs-80.994 ± 0.0030.995 ± 0.0010.992 ± 0.0010.991 ± 0.0070.997 ± 0.0030.998 ± 0.0033-vs-40.981 ± 0.0020.933 ± 0.0020.953 ± 0.0050.971 ± 0.0080.983 ± 0.0120.976 ± 0.0115-vs-90.994 ± 0.0020.980 ± 0.0040.981 ± 0.0060.988 ± 0.0110.994 ± 0.0020.988 ± 0.0076-vs-70.962 ± 0.0120.970 ± 0.0230.965 ± 0.0200.985 ± 0.0120.960 ± 0.0560.973 ± 0.0436-vs-80.996 ± 0.0060.988 ± 0.0100.989 ± 0.0130.980 ± 0.0160.994 ± 0.0080.998 ± 0.0027-vs-80.973 ± 0.0060.983 ± 0.0080.970 ± 0.1210.960 ± 0.0100.971 ± 0.0110.986 ± 0.004The bold is the best one.Recalling (11), based on the decomposition (13), we can reformulate CaENSVM optimization problem as(16)w∗=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ⏟g−cn∑i=1nlaEN11−yiwTxi;τ,θ,s⏟h,where g and h are both convex functions on Rp.According to the DC algorithm, we have to calculate the derivative ofh with respect to w. Since laEN1 in (16) has sharp points, h is also nondifferentiable. Thus, we utilize the subgradient instead of the derivative. For a set of tuning parameters τ,θ,s, the subgradient of laEN1 with respect to w is given as follows:(17)∇laEN1wi=τθxixiTw+1−2θyixi,1−yiwTxi<u1,0,u1<1−yiwTxi<u2,θxixiTw−yixi,1−yiwTxi>u2,where u1 and u2u1<0<u2 are two sharp points of both laEN1u and lCaENu except for zero, which can be easily calculated from (9) and (14) and are given as follows:(18)u1=1−θ−1−θ2+2θs/τθ,u2=θ−1+1−θ2+2θsθ.Therefore, by Algorithm 1 and givenwk, the main optimized subproblem is(19)wk+1=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ−cn∑i=1n∇laEN1wkiTw−wk=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ−cn∑i=1n∇laEN1wkiTw.For the sake of scalability and efficiency, we apply a stochastic gradient descent algorithm, i.e., Pegasos [43], to solve problem (19). Let At⊂1,2,⋯,n and At=m be a subset of k samples, randomly chosen from the whole dataset for the t-th iteration during optimizing a problem (19). Thus, we consider the following approximate objective function:(20)Fv;At=12v22+cm∑i∈AtlaEN1−yivTxi;τ,θ−cm∑i∈At∇laEN1wkiTv,v∈Rp.Then, the subgradient ofFv;At with respect to v at vt is given as follows:(21)∇Fvt=vt+cm∑i∈Atδvti−cm∑i∈At∇laEN1wki,where vt is the optimal value at the t-th iteration and(22)δvti=τθxixiTvt+1−2θyixi,1−yixiTvt<0,θxixiTvt−yixi,1−yixiTvt>0.In line with Pegasos, the update can be written as(23)vt+1←vt−ηt∇Fvt,where ηt=c/t is the step size.Finally, based on the aforementioned results about the DC algorithm and Pegasos, we design a Pegasos-based DC procedure to solve the CaENSVM optimization problem (11), which is shown in Algorithm 2. Note that if we substitute the training sample matrix X with the kernelized form XK=KX,XT=KX,x1,⋯,KX,xn, we can directly apply the implemented Algorithm 2 to solve nonlinear CaENSVM.Table 7
The mean accuracy (Acc.) and standard deviation (sd) with Gaussian kernel for the PMU-UD dataset.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd2-vs-30.954 ± 0.0240.990 ± 0.0100.987 ± 0.0080.992 ± 0.0100.952 ± 0.0390.979 ± 0.0182-vs-40.914 ± 0.0290.994 ± 0.0070.972 ± 0.0140.995 ± 0.0050.955 ± 0.0230.996 ± 0.0022-vs-60.997 ± 0.0040.999 ± 0.0021.000 ± 0.0000.999 ± 0.0020.962 ± 0.0290.967 ± 0.0302-vs-71.000 ± 0.0001.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.995 ± 0.0041.000 ± 0.0002-vs-80.988 ± 0.0121.000 ± 0.0000.998 ± 0.0031.000 ± 0.0000.806 ± 0.2571.000 ± 0.0005-vs-90.987 ± 0.0110.998 ± 0.0030.993 ± 0.0050.998 ± 0.0030.947 ± 0.0290.949 ± 0.0626-vs-70.962 ± 0.0360.997 ± 0.0030.993 ± 0.0050.997 ± 0.0030.971 ± 0.0251.000 ± 0.0006-vs-80.899 ± 0.1980.995 ± 0.0040.994 ± 0.0060.939 ± 0.0950.996 ± 0.0040.998 ± 0.0037-vs-80.966 ± 0.0190.981 ± 0.0060.981 ± 0.0080.992 ± 0.0040.966 ± 0.0200.985 ± 0.010The bold is the best one.


## 3.1. The CaENSVM Model

In our recent work [3], we derived the so-called VTUB to demonstrate the unique advantages of the EN loss under the framework of the SVM. Thus, it is meaningful to improve the performance of the EN loss. Motivated by the pinball loss, we designed the following asymmetric elastic net (aEN) loss:(9)laENu;τ,θ=θ2u2+1−θu,u≥0,τθ2u2−1−θu,u<0,where θ∈0,1 corresponds to a tradeoff between L1 norm and L2 norm, τ∈0,1 is a tuning parameter controlling the bias of the penalization for positive and negative losses. Figure 2(a) illustrates different shapes of aEN losses with different sets of tuning parameters. According to definition (9), laEN can be regarded as a generalization of elastic net loss, pinball loss, and asymmetric least squared loss functions [17], since laEN reduces to lEN for τ=0, laEN becomes lPin for θ=0 and laEN is equivalent to laLS for θ=1.Figure 2
(a) “aEN loss” is an asymmetric elastic net loss. (b) “CaEN loss” is a capped asymmetric elastic net loss. (c) The convex decomposition of CaEN loss, wheres=1,τ=0.7, and θ=0.2.
(a)(b)(c)According to (9), laEN goes to infinity along with u⟶∞, so the proposed aEN loss is also sensitive to outliers (label noise). To improve its robustness against outliers, we further use capped trick to propose a capped asymmetric elastic net (CaEN) loss function, which is defined as(10)lCaENu;τ,θ,s=minlaENu,s,where s>0 is a thresholding parameter. Figure 2(b) depicts the shapes of the CaEN loss with different groups of parameters. As shown in Figure 2(b) , the proposed CaEN loss function is exactly upper bounded and concave. The truncation of the left part of the CaEN loss can increase the sparseness [13, 20], while the boundness of the right part of the CaEN loss can enhance its robustness against label noise. Detailed discussion on the properties of CaENSVM is provided in Section 4. However, the concavity of the CaEN loss often involves high optimization costs.Combining CaEN loss with the standard SVM, we propose a novel robust CaEN loss-based SVM (termed as CaENSVM), which is formulated as(11)minw12w22+cn∑i=1nlCaEN1−yiwTxi;τ,θ,s,where c>0 is the tuning parameter. We remove the intercept term b in (11) for simplicity, which can be achieved before training by centering features just like [10, 13]. After obtaining w from (11), the decision function of the linear CaENSVM for a new sample xnew is fxnew=signwTxnew.For nonlinear CaENSVM, we consider the following kernel-generated separating hyperplane [40, 41]:(12)wTϕx=0,where ϕx maps x to a high-dimensional Hilbert space. In application, we often utilize kernel trick, i.e., ϕx=KX,x, where K⋅,⋅ is a kernel function and KX,x=Kx1,x,⋯,Kxn,xT. Then, by replacing wTxi with wTKX,xi in (11), we can obtain the nonlinear CaENSVM model. For determining the class of a new sample xnew, we only need to replace xnew with KX,xnew in the linear decision function.


## 3.2. DC Algorithm for CaENSVM

The truncation for aEN loss results in a nonconvex loss, indicating that solving CaENSVM (11) involves nonconvex minimization, which is often difficult. Note that, though CaEN loss is concave, we can decompose lCaEN into the difference of two convex functions, i.e.,(13)lCaENu;τ,θ,s=laENu;τ,θ−laEN1u;τ,θ,s,where laEN1 corresponds to the so-called aEN1 loss, which is given as follows:(14)laEN1u;τ,θ,s=maxlaENu;τ,θ−s,0.Figure2(c) depicts the above convex decomposition of the CaEN loss. As the figure shows, both aEN and aEN1 losses are exactly convex. By calculating the difference of aEN and aEN1 losses, we can finally obtain the nonconvex CaEN loss. Using this property of the CaEN loss, we apply the DC (difference of convex functions) algorithm [42] to optimize problem (11).Considering the following optimization problem:(15)minΘfΘ=gΘ−hΘ,Θ∈Rm,where g and h are both convex functions on Rm. To solve problem (15), the DC algorithm turns to minimize a sequence of convex subproblems. A general framework of the DC algorithm for (15) is illustrated as Algorithm 1.Table 6
The mean accuracy (Acc.) and standard deviation (sd) with linear kernel for the PMU-UD dataset.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd2-vs-30.942 ± 0.0110.922 ± 0.0060.942 ± 0.0040.975 ± 0.0100.949 ± 0.0540.979 ± 0.0172-vs-40.947 ± 0.0260.925 ± 0.0150.925 ± 0.0200.948 ± 0.0200.951 ± 0.0220.951 ± 0.0142-vs-60.999 ± 0.0020.997 ± 0.0120.997 ± 0.0140.990 ± 0.0040.999 ± 0.0021.000 ± 0.0002-vs-70.999 ± 0.0020.992 ± 0.0090.993 ± 0.0030.993 ± 0.0060.999 ± 0.0020.999 ± 0.0022-vs-80.994 ± 0.0030.995 ± 0.0010.992 ± 0.0010.991 ± 0.0070.997 ± 0.0030.998 ± 0.0033-vs-40.981 ± 0.0020.933 ± 0.0020.953 ± 0.0050.971 ± 0.0080.983 ± 0.0120.976 ± 0.0115-vs-90.994 ± 0.0020.980 ± 0.0040.981 ± 0.0060.988 ± 0.0110.994 ± 0.0020.988 ± 0.0076-vs-70.962 ± 0.0120.970 ± 0.0230.965 ± 0.0200.985 ± 0.0120.960 ± 0.0560.973 ± 0.0436-vs-80.996 ± 0.0060.988 ± 0.0100.989 ± 0.0130.980 ± 0.0160.994 ± 0.0080.998 ± 0.0027-vs-80.973 ± 0.0060.983 ± 0.0080.970 ± 0.1210.960 ± 0.0100.971 ± 0.0110.986 ± 0.004The bold is the best one.Recalling (11), based on the decomposition (13), we can reformulate CaENSVM optimization problem as(16)w∗=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ⏟g−cn∑i=1nlaEN11−yiwTxi;τ,θ,s⏟h,where g and h are both convex functions on Rp.According to the DC algorithm, we have to calculate the derivative ofh with respect to w. Since laEN1 in (16) has sharp points, h is also nondifferentiable. Thus, we utilize the subgradient instead of the derivative. For a set of tuning parameters τ,θ,s, the subgradient of laEN1 with respect to w is given as follows:(17)∇laEN1wi=τθxixiTw+1−2θyixi,1−yiwTxi<u1,0,u1<1−yiwTxi<u2,θxixiTw−yixi,1−yiwTxi>u2,where u1 and u2u1<0<u2 are two sharp points of both laEN1u and lCaENu except for zero, which can be easily calculated from (9) and (14) and are given as follows:(18)u1=1−θ−1−θ2+2θs/τθ,u2=θ−1+1−θ2+2θsθ.Therefore, by Algorithm 1 and givenwk, the main optimized subproblem is(19)wk+1=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ−cn∑i=1n∇laEN1wkiTw−wk=argminw12w22+cn∑i=1nlaEN1−yiwTxi;τ,θ−cn∑i=1n∇laEN1wkiTw.For the sake of scalability and efficiency, we apply a stochastic gradient descent algorithm, i.e., Pegasos [43], to solve problem (19). Let At⊂1,2,⋯,n and At=m be a subset of k samples, randomly chosen from the whole dataset for the t-th iteration during optimizing a problem (19). Thus, we consider the following approximate objective function:(20)Fv;At=12v22+cm∑i∈AtlaEN1−yivTxi;τ,θ−cm∑i∈At∇laEN1wkiTv,v∈Rp.Then, the subgradient ofFv;At with respect to v at vt is given as follows:(21)∇Fvt=vt+cm∑i∈Atδvti−cm∑i∈At∇laEN1wki,where vt is the optimal value at the t-th iteration and(22)δvti=τθxixiTvt+1−2θyixi,1−yixiTvt<0,θxixiTvt−yixi,1−yixiTvt>0.In line with Pegasos, the update can be written as(23)vt+1←vt−ηt∇Fvt,where ηt=c/t is the step size.Finally, based on the aforementioned results about the DC algorithm and Pegasos, we design a Pegasos-based DC procedure to solve the CaENSVM optimization problem (11), which is shown in Algorithm 2. Note that if we substitute the training sample matrix X with the kernelized form XK=KX,XT=KX,x1,⋯,KX,xn, we can directly apply the implemented Algorithm 2 to solve nonlinear CaENSVM.Table 7
The mean accuracy (Acc.) and standard deviation (sd) with Gaussian kernel for the PMU-UD dataset.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd2-vs-30.954 ± 0.0240.990 ± 0.0100.987 ± 0.0080.992 ± 0.0100.952 ± 0.0390.979 ± 0.0182-vs-40.914 ± 0.0290.994 ± 0.0070.972 ± 0.0140.995 ± 0.0050.955 ± 0.0230.996 ± 0.0022-vs-60.997 ± 0.0040.999 ± 0.0021.000 ± 0.0000.999 ± 0.0020.962 ± 0.0290.967 ± 0.0302-vs-71.000 ± 0.0001.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.995 ± 0.0041.000 ± 0.0002-vs-80.988 ± 0.0121.000 ± 0.0000.998 ± 0.0031.000 ± 0.0000.806 ± 0.2571.000 ± 0.0005-vs-90.987 ± 0.0110.998 ± 0.0030.993 ± 0.0050.998 ± 0.0030.947 ± 0.0290.949 ± 0.0626-vs-70.962 ± 0.0360.997 ± 0.0030.993 ± 0.0050.997 ± 0.0030.971 ± 0.0251.000 ± 0.0006-vs-80.899 ± 0.1980.995 ± 0.0040.994 ± 0.0060.939 ± 0.0950.996 ± 0.0040.998 ± 0.0037-vs-80.966 ± 0.0190.981 ± 0.0060.981 ± 0.0080.992 ± 0.0040.966 ± 0.0200.985 ± 0.010The bold is the best one.


## 4. Properties of CaENSVM

In this section, we theoretically investigate several properties of the proposed CaENSVM, including noise insensitivity, Bayes rule, and generalization error bound. Since the following analysis involves the statistical distribution of the training dataset, we first make several notations and assumptions. Supposing the training samplesxiT,yii=1n are independently drawn from a probability measure ρ on X×Y, where Y∈+1,−1. Let Prob⋅ and Prob⋅⋅ be the probability and the conditional probability, respectively.


### 4.1. Noise Insensitivity

By the construction of the CaEN loss, it inherits the robustness to label noise (outliers) from ramp loss and the resampling stability to feature noise from the pinball loss. Therefore, we focus on the noise insensitivity of CaENSVM from two aspects: the robustness to label noise and the resampling stability to feature noise.


#### 4.1.1. Robustness to Label Noise

For the robustness to label noise, we show this property throughout proving the boundness of the influence function, which was first introduced by Hampel [44]. The influence function aims to measure the stability of estimators against an infinitesimal contamination. The influence function of a robust estimator should be bounded [44, 45]. Before giving the main result, we have to make the following assumption for the distribution of the training dataset, which is common in statistical analysis.Assumption 1.
The random variablex∈X has finite second moment.
We denote byx0T,y0T a sample point with mass probability distribution ∆x0,y0. Given the distribution F of xT,yT in Rp+1, let the mixed distribution of F and ∆x0,y0 be Fϵ=1−ϵF+ϵ∆x0,y0, where ϵ∈0,1 is the proportion parameter. Fixing τ,θ and s, let(24)w0∗=argminwc∫lCaEN1−ywTxdF+12w22,wϵ∗=argminwc∫lCaEN1−ywTxdFϵ+12w22.
Then, the influence function at a sample pointx0T,y0T is defined as(25)IFx0,y0;w0∗=limϵ⟶0+wϵ∗−w0∗ϵ,provided that the limit exists.Theorem 1.
(influence function). For linear CaENSVM (11) with τ,θ and s fixed, the influence function IFx0,y0;w0∗ at a sample point x0T,y0T is given by(26)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where W0=1/cI+∫xxT∇2lCaEN1−yxTw0∗dF, and(27)∇2lCaENu=0,u<u1,τθ,u1<u<0,θ,0<u<u2,0,u>u2,and(28)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ.Proof.
According to KKT conditions,wϵ∗ must satisfy(29)−c∫∇lCaEN1−yxTwϵ∗yxdFϵ+wϵ∗=0.
SinceFϵ=1−ϵF+ϵ∆x0,y0, equation (29) can be rewritten as(30)1cwϵ∗=1−ϵ∫∇lCaEN1−yxTwϵ∗yxdF+ϵ∇lCaEN1−y0x0Twϵ∗y0x0.
Differentiating with respect toϵ in both sides of (30) and letting ϵ⟶0, we have(31)1c∂wϵ∗∂ϵϵ=0=−∫∇lCaEN1−yxTw0∗yxdF−∫xxT∇2lCaEN1−yxTw0∗dF⋅∂wϵ∗∂ϵϵ=0+γ0+∇lCaEN1−y0x0Tw0∗y0x0,where(32)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ are from the results of the optimality condition (36).
Combining (30) and (31), we can obtain that(33)1cI+∫xxT∇2lCaEN1−yxTw0∗IFIFx0,y0;w0∗=∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where I is an identity matrix with a proper size. Let W0=∫xxT∇2lCaEN1−yxTw0∗dF. Since ∇2lCaENu is nonnegative by (27), W0 can be always invertible. Therefore, we finally obtain(34)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0.
The proof is completed.Corollary 1.
The influence functionIFx0,y0;w0∗ is bounded, i.e., the CaENSVM is robust to label noise.Proof.
First, sinceζiϵ,x,y,i=1,2,3 are bounded and continuous with respect to ϵ in closed intervals, their corresponding derivatives with respect to ϵ are also bounded. Then, by Assumption 1 and Theorem 1, we have(35)IFx0,y0;w0∗≤λminW0θu2+1−θx0+1cw0∗+γ0<∞.
Foru1≤1−y0x0Tw0∗≤u2, where λmin⋅ is the smallest eigenvalue of a matrix. Otherwise, by (36), ∇lCaEN1−y0x0Tw0∗=0, the boundness of IFx0,y0;w0∗ also holds.Remark 1.
According to Corollary1, the derivative of loss, i.e., ∇lCaENu, significantly relates to the characteristics of the influence function. In fact, because ∇lCaENu is bounded, we can easily deduce the bounds of the influence function. For those convex losses, such as elastic net loss and pinball loss, their derivatives are boundless, which means the corresponding estimators are not robust. In other words, Corollary 1 reveals the reason of the robustness of the CaEN loss, or those concave losses.


#### 4.1.2. Resampling Stability to Feature Noise

For the resampling stability to feature noise, we demonstrate this property in line with Huang et al. [11]. Recalling the CaEN loss (10), the subgradient of lCaEN with respect to u is given by(36)∇lCaENu=0,u<u1,τθu1−1−θ,0,u=u1,τθu−1−θ,u1<u<0,−τ1−θ,1−θ,u=0,θu+1−θ,0<u<u2,0,θu2+1−θ,u=u2,0,u>u2.Then, by KKT (Karush–Kuhn–Tucker) conditions, the solution of CaENSVM satisfies(37)0∈ncw−∑i=1n∇lCaEN1−yiwTxiyixi,where 0 is a proper length vector with all components equal to zero. For given w, the training sample index set can be partitioned into the following seven sets:(38)S0w=i:1−yiwTxi<u1,S1w=i:1−yiwTxi=u1,S2w=i:u1<1−yiwTxi<0,S3w=i:1−yiwTxi=0,S4w=i:0<1−yiwTxi<u2,S5w=i:1−yiwTxi=u2,S6w=i:1−yiwTxi>u2.Using the notations (38), the optimal conditions (37) can be equivalently rewritten as(39)ncw−∑i∈S1w∇lCaEN1−yiwTxiyixi−∑i∈S2w∇lCaEN1−yiwTxiyixi−∑i∈S3w∇lCaEN1−yiwTxiyixi−∑i∈S4w∇lCaEN1−yiwTxiyixi−∑i∈S5w∇lCaEN1−yiwTxiyixi=0.SinceS1w, S3w, and S5w are based on equalities, it is reasonable to see that S1w, S3w, and S5w have much smaller sizes than S2w or S4w. Thus, the contributions of S1w, S3w, and S5w to (39) are considerably weak. In other words, we can roughly determine w by S2w and S4w, i.e.,(40)ncw+τ∑i∈S2w1−θ−θ1−yiwTxiyixi−∑i∈S4wθ1−yiwTxi+1−θyixi≅0.With parametersτ and θ properly selected, equation (40) indicates that τ controls the sensitivity of the CaENSVM to feature noise. In fact, by (38), 1−θ−θ1−yiwTxi and θ1−yiwTxi+1−θ are both positive, which means that a large τ (close to 1) can well balance the size of S2w and S4w for zero mean feature noise. Therefore, the effect of zero mean feature noise is weakened, and the final separating hyperplane of the CaENSVM is stable for resampling. Along with τ decreasing (close to 0), by (38), the final separating hyperplane is gradually dominated by the instances in S4w. As a result, the classification results are significantly disturbed by the zero mean feature noise around the decision boundary.


### 4.2. Bayes Rule

LetPx=ProbY=1X=x be the conditional probability of the positive class given X=x. Lin [46] claimed that signPx−1/2 is the decision-theoretic optimal classification rule with the smallest generalization error, which is the so-called Bayes rule. We can also equivalently define Bayes rule as(41)fCx=+1,Proby=+1x≥Proby=−1x;−1,Proby=+1x<Proby=−1x.For any loss functionl⋅, we define the expected risk of a classifier f:X⟼Y as(42)Rl,ρf=∫X×Yl1−yfxdρ.By minimizing the expected risk over all measurable functions, we can obtain fl,ρx as(43)fl,ρx=argminμ∫Yl1−yμdρyx,∀x∈X,where ρyx is the conditional distribution of y given x. Note that ρyx is a binary distribution, corresponding to Proby=+1x and Proby=−1x.Huang et al. [11] proved that the pinball loss can lead to the Bayes classifier. In the following, we demonstrate that the Bayes rule also holds for our proposed capped asymmetric elastic net loss function.Theorem 2.
The decision functionflCaEN,ρ obtained by minimizing lCaEN-based expected risk over all measurable functions f:X⟶Y is equivalent to the Bayes rule, i.e., flCaEN,ρx=fCx,∀x∈X.Proof.
By simple calculating, we can obtain(44)∫YlCaEN1−yμdρyx=lCaEN1−μProby=+1x+lCaEN1+μProby=−1x.
According to (10), it follows that(45)lCaEN1−μ=s,μ<1−u2,θ21−μ2+1−θ1−μ,1−u2≤μ<1,τθ21−μ2−1−θ1−μ,1≤μ<1−u1,s,μ≥1−u1,and(46)lCaEN1+μ=s,μ≤u1−1,τθ21+μ2−1−θ1+μ,u1−1<μ≤−1,θ21+μ2+1−θ1+μ,−1<μ≤u2−1,s,μ>u2−1.
LetP+1=Proby=+1x and P−1=Proby=−1x, respectively. We denote by r1θ,μ=θ/21−μ2+1−θ1−μ, r2θ,μ=τθ/21−μ2−1−θ1−μ, r3θ,μ=θ/21+μ2+1−θ1+μ, and r4θ,μ=τθ/21+μ2−1−θ1−μ for convenience, respectively. We found that the expected risk relates to the value of u2. Thus, based on (45) and (46) and the continuity of lCaEN, we discuss all cases as follows:
For the case of0<u2<1, we have(47)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤−1,sP+1+r3θ,μP−1,−1<μ≤u2−1,sP+1+sP−1,u2−1<μ≤1−u2,r1θ,μP+1+sP−1,1−u2<μ≤1,r2θ,μP+1+sP−1,1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=1, u2−1=1−u2 and the interval u2−1,1−u2 disappears. One can easily verify that the minimal value is not affected. Supposing that P+1>P−1, the minimal value of μ is +1 according to the increase-decrease characteristics of lCaEN with respect to μ in each interval. Similarly, the minimal value of μ is −1 if P+1<P−1 or the minimal value of μ is +1 or −1 if P+1=P−1. Consequently, we have flCaENx=fCx when 0<u2≤1.
For the case of1<u2<2, we have(48)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤−1,sP+1+r3θ,μP−1,−1<μ≤1−u2,r1θ,μP+1+r3θ,μP−1,1−u2<μ≤u2−1,r1θ,μP+1+sP−1,u2−1<μ≤1,r2θ,μP+1+sP−1,1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=2, we have 1−u2=−1 and u2−1=1. Then, the intervals −1,1−u2, and 1,u2−1 both disappear, which has no effects on the minimum value. After simple calculations, one can find that the minimal value is the same as 0<u2≤1. Thus, we have flCaENx=fCx when 1<u2≤2.
For the case of2<u2<2−u1, we have(49)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤1−u2,r1θ,μP+1+r4θ,μP−1,1−u2<μ≤−1,r1θ,μP+1+r3θ,μP−1,−1<μ≤+1,r2θ,μP+1+r3θ,μP−1,1<μ≤u2−1,r2θ,μP+1+sP−1,u2−1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=2−u1, u1−1=1−u2, and u2−1=1−u1, the interval u1−1,1−u2, and u2−1,1−u1 disappear, which has no effects on the minimum value; the minimal value is the same as 0<u2≤1, i.e., we have flCaENx=fCx when 2<u2≤2−u1.
For the case ofu2>2−u1, we have(50)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤1−u2,r1θ,μP+1+sP−1,1−u2<μ≤u1−1,r1θ,μP+1+r4θ,μP−1,u1−1<μ≤−1,r1θ,μP+1+r3θ,μP−1,−1<μ≤+1,r2θ,μP+1+r3θ,μP−1,1<μ≤1−u1,sP+1+r3θ,μP−1,1−u1<μ≤u2−1,sP+1+sP−1,μ>u2−1.
One can easily obtain the same minimal value as0<u2≤1, i.e., we have flCaENx=fCx when u2>2−u1. With the abovementioned results, minimizing lCaEN-based expected risk over all measurable functions can lead to the Bayes rule. Thus, Theorem 2 is proved.


### 4.3. Generalization Error Bound

We have proved that the proposed CaENSVM is equivalent to the Bayes rule, which is also called classification-calibrated in the study of Bartlett et al. [7], indicating that the CaENSVM enjoys many good properties. Here, we further give the generalization error bound of the CaENSVM based on the empirical Rademacher complexity [6], where the empirical Rademacher complexity is defined as follows:Definition 1.
Supposing thatx1,⋯,xn are independently selected from X with a probability distribution ν, and F is a real-valued function class mapping from X to R, the empirical Rademacher complexity of F is a random variable defined as(51)R^nF=Eσsupf∈F2n∑i=1nσifxi:x1,⋯,xn,where σ=σ1,…,σnT is independently uniform ±1-valued (Rademacher) random variables and Eσ⋅ means the expectation over σ. The Rademacher complexity of F is(52)RnF=EνR^nF,where Eν⋅ means the expectation over ν.
According to (51), R^nF can be viewed as a correlation between f=fx1,⋯,fxnT and σ for given x1,⋯,xn. Thus, the higher R^nF is, the more complex F is. By (52), RnF depicts the average complexity of F based on ν instead of a set of particular samples.
In line with the abovementioned definitions and lemmas in the study of Bartlett and Mendelson [6], we provide the following theorem to yield the generalization error bound of the CaENSVM.Theorem 3.
Fixζ∈0,1 and B∈R+ and consider the binary classification problem on xiT,yii=1n drawn independently from a probability distribution F. Let F=ff:x↦wTϕx,w≤B and G=gg:y,fx↦−yfx,f∈F be function classes, respectively. If the CaEN loss (10) with s≥1/2 and the optimal w0∗ satisfies w0∗≤B, then with probability at least 1−ζ, the prediction function fx satisfies f∈F and(53)Probyfx≤0≤2n∑i=1nlCaEN1−yifxi+8Bθu2+1−θn∑i=1nKxi,xi+8ln2/ζn.Proof.
define the Heaviside functionΞ⋅ as(54)Ξu=1,u≥0,0,otherwise.
Then, we can easily obtain(55)Probyfx≤0=EρΞ−yfx.
By definingΨu=2lCaEN1+u;τ,θ,s=1/2, we can easily verify that u2≤1 and Ψu∈0,1 dominates the function Ξ⋅ on the support of ρ, we can obtain the following inequality by Theorem 8 in the study of Bartlett and Mendelson [6] as(56)EρΞgy,fx≤EρΨgy,fx≤E^nΨgy,fx+R^nΨ˜G+8ln2/ζn,where E^nΨgy,fx=1/n∑i=1nΨgfxi,yi and Ψ˜G=x,y↦Ψgy,fx−Ψgy,0:f∈F.
According to the CaEN loss (10) with the optimal w0∗, we have(57)E^nΨgy,fx=1n∑i=1nΨgyi,fxi=2n∑i=1nlCaEN1−yifxi.
By Theorem 14 in the study of Bartlett and Mendelson [6], and since Lipschitz function Ψu pertains to Lipschitz constant θu2+1−θ, we have(58)R^nΨ˜°G≤4θu2+1−θR^nG.
According to the definition (51), we can deduce that(59)R^nG=Eσsupg∈G2n∑i=1nσigyi,fxi:x1,⋯,xn=Eσsupf∈F2n∑i=1n−σiyifxi:x1,⋯,xn=Eσsupf∈F2n∑i=1nσifxi:x1,⋯,xn=R^nF.
Hence, by Lemma 22 in the study of Bartlett and Mendelson [6], R^nF satisfies(60)R^nF≤2Bn∑i=1nKxi,xi.
Finally, combining (56)–(58) and (60), we can reach the result of the theorem.Remark 2.
From Theorem3, if the number of training samples n⟶+∞, we have Probyfx≤0⟶0. Therefore, the generalization capability of the proposed CaENSVM can be theoretically guaranteed.Remark 3.
According to the proof of Theorem3, s=1/2 can lead to a tighter generalization error bound. However, experimental results indicate that larger s may produce a more satisfactory classifier.


## 4.1. Noise Insensitivity

By the construction of the CaEN loss, it inherits the robustness to label noise (outliers) from ramp loss and the resampling stability to feature noise from the pinball loss. Therefore, we focus on the noise insensitivity of CaENSVM from two aspects: the robustness to label noise and the resampling stability to feature noise.


### 4.1.1. Robustness to Label Noise

For the robustness to label noise, we show this property throughout proving the boundness of the influence function, which was first introduced by Hampel [44]. The influence function aims to measure the stability of estimators against an infinitesimal contamination. The influence function of a robust estimator should be bounded [44, 45]. Before giving the main result, we have to make the following assumption for the distribution of the training dataset, which is common in statistical analysis.Assumption 1.
The random variablex∈X has finite second moment.
We denote byx0T,y0T a sample point with mass probability distribution ∆x0,y0. Given the distribution F of xT,yT in Rp+1, let the mixed distribution of F and ∆x0,y0 be Fϵ=1−ϵF+ϵ∆x0,y0, where ϵ∈0,1 is the proportion parameter. Fixing τ,θ and s, let(24)w0∗=argminwc∫lCaEN1−ywTxdF+12w22,wϵ∗=argminwc∫lCaEN1−ywTxdFϵ+12w22.
Then, the influence function at a sample pointx0T,y0T is defined as(25)IFx0,y0;w0∗=limϵ⟶0+wϵ∗−w0∗ϵ,provided that the limit exists.Theorem 1.
(influence function). For linear CaENSVM (11) with τ,θ and s fixed, the influence function IFx0,y0;w0∗ at a sample point x0T,y0T is given by(26)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where W0=1/cI+∫xxT∇2lCaEN1−yxTw0∗dF, and(27)∇2lCaENu=0,u<u1,τθ,u1<u<0,θ,0<u<u2,0,u>u2,and(28)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ.Proof.
According to KKT conditions,wϵ∗ must satisfy(29)−c∫∇lCaEN1−yxTwϵ∗yxdFϵ+wϵ∗=0.
SinceFϵ=1−ϵF+ϵ∆x0,y0, equation (29) can be rewritten as(30)1cwϵ∗=1−ϵ∫∇lCaEN1−yxTwϵ∗yxdF+ϵ∇lCaEN1−y0x0Twϵ∗y0x0.
Differentiating with respect toϵ in both sides of (30) and letting ϵ⟶0, we have(31)1c∂wϵ∗∂ϵϵ=0=−∫∇lCaEN1−yxTw0∗yxdF−∫xxT∇2lCaEN1−yxTw0∗dF⋅∂wϵ∗∂ϵϵ=0+γ0+∇lCaEN1−y0x0Tw0∗y0x0,where(32)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ are from the results of the optimality condition (36).
Combining (30) and (31), we can obtain that(33)1cI+∫xxT∇2lCaEN1−yxTw0∗IFIFx0,y0;w0∗=∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where I is an identity matrix with a proper size. Let W0=∫xxT∇2lCaEN1−yxTw0∗dF. Since ∇2lCaENu is nonnegative by (27), W0 can be always invertible. Therefore, we finally obtain(34)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0.
The proof is completed.Corollary 1.
The influence functionIFx0,y0;w0∗ is bounded, i.e., the CaENSVM is robust to label noise.Proof.
First, sinceζiϵ,x,y,i=1,2,3 are bounded and continuous with respect to ϵ in closed intervals, their corresponding derivatives with respect to ϵ are also bounded. Then, by Assumption 1 and Theorem 1, we have(35)IFx0,y0;w0∗≤λminW0θu2+1−θx0+1cw0∗+γ0<∞.
Foru1≤1−y0x0Tw0∗≤u2, where λmin⋅ is the smallest eigenvalue of a matrix. Otherwise, by (36), ∇lCaEN1−y0x0Tw0∗=0, the boundness of IFx0,y0;w0∗ also holds.Remark 1.
According to Corollary1, the derivative of loss, i.e., ∇lCaENu, significantly relates to the characteristics of the influence function. In fact, because ∇lCaENu is bounded, we can easily deduce the bounds of the influence function. For those convex losses, such as elastic net loss and pinball loss, their derivatives are boundless, which means the corresponding estimators are not robust. In other words, Corollary 1 reveals the reason of the robustness of the CaEN loss, or those concave losses.


### 4.1.2. Resampling Stability to Feature Noise

For the resampling stability to feature noise, we demonstrate this property in line with Huang et al. [11]. Recalling the CaEN loss (10), the subgradient of lCaEN with respect to u is given by(36)∇lCaENu=0,u<u1,τθu1−1−θ,0,u=u1,τθu−1−θ,u1<u<0,−τ1−θ,1−θ,u=0,θu+1−θ,0<u<u2,0,θu2+1−θ,u=u2,0,u>u2.Then, by KKT (Karush–Kuhn–Tucker) conditions, the solution of CaENSVM satisfies(37)0∈ncw−∑i=1n∇lCaEN1−yiwTxiyixi,where 0 is a proper length vector with all components equal to zero. For given w, the training sample index set can be partitioned into the following seven sets:(38)S0w=i:1−yiwTxi<u1,S1w=i:1−yiwTxi=u1,S2w=i:u1<1−yiwTxi<0,S3w=i:1−yiwTxi=0,S4w=i:0<1−yiwTxi<u2,S5w=i:1−yiwTxi=u2,S6w=i:1−yiwTxi>u2.Using the notations (38), the optimal conditions (37) can be equivalently rewritten as(39)ncw−∑i∈S1w∇lCaEN1−yiwTxiyixi−∑i∈S2w∇lCaEN1−yiwTxiyixi−∑i∈S3w∇lCaEN1−yiwTxiyixi−∑i∈S4w∇lCaEN1−yiwTxiyixi−∑i∈S5w∇lCaEN1−yiwTxiyixi=0.SinceS1w, S3w, and S5w are based on equalities, it is reasonable to see that S1w, S3w, and S5w have much smaller sizes than S2w or S4w. Thus, the contributions of S1w, S3w, and S5w to (39) are considerably weak. In other words, we can roughly determine w by S2w and S4w, i.e.,(40)ncw+τ∑i∈S2w1−θ−θ1−yiwTxiyixi−∑i∈S4wθ1−yiwTxi+1−θyixi≅0.With parametersτ and θ properly selected, equation (40) indicates that τ controls the sensitivity of the CaENSVM to feature noise. In fact, by (38), 1−θ−θ1−yiwTxi and θ1−yiwTxi+1−θ are both positive, which means that a large τ (close to 1) can well balance the size of S2w and S4w for zero mean feature noise. Therefore, the effect of zero mean feature noise is weakened, and the final separating hyperplane of the CaENSVM is stable for resampling. Along with τ decreasing (close to 0), by (38), the final separating hyperplane is gradually dominated by the instances in S4w. As a result, the classification results are significantly disturbed by the zero mean feature noise around the decision boundary.


## 4.1.1. Robustness to Label Noise

For the robustness to label noise, we show this property throughout proving the boundness of the influence function, which was first introduced by Hampel [44]. The influence function aims to measure the stability of estimators against an infinitesimal contamination. The influence function of a robust estimator should be bounded [44, 45]. Before giving the main result, we have to make the following assumption for the distribution of the training dataset, which is common in statistical analysis.Assumption 1.
The random variablex∈X has finite second moment.
We denote byx0T,y0T a sample point with mass probability distribution ∆x0,y0. Given the distribution F of xT,yT in Rp+1, let the mixed distribution of F and ∆x0,y0 be Fϵ=1−ϵF+ϵ∆x0,y0, where ϵ∈0,1 is the proportion parameter. Fixing τ,θ and s, let(24)w0∗=argminwc∫lCaEN1−ywTxdF+12w22,wϵ∗=argminwc∫lCaEN1−ywTxdFϵ+12w22.
Then, the influence function at a sample pointx0T,y0T is defined as(25)IFx0,y0;w0∗=limϵ⟶0+wϵ∗−w0∗ϵ,provided that the limit exists.Theorem 1.
(influence function). For linear CaENSVM (11) with τ,θ and s fixed, the influence function IFx0,y0;w0∗ at a sample point x0T,y0T is given by(26)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where W0=1/cI+∫xxT∇2lCaEN1−yxTw0∗dF, and(27)∇2lCaENu=0,u<u1,τθ,u1<u<0,θ,0<u<u2,0,u>u2,and(28)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ.Proof.
According to KKT conditions,wϵ∗ must satisfy(29)−c∫∇lCaEN1−yxTwϵ∗yxdFϵ+wϵ∗=0.
SinceFϵ=1−ϵF+ϵ∆x0,y0, equation (29) can be rewritten as(30)1cwϵ∗=1−ϵ∫∇lCaEN1−yxTwϵ∗yxdF+ϵ∇lCaEN1−y0x0Twϵ∗y0x0.
Differentiating with respect toϵ in both sides of (30) and letting ϵ⟶0, we have(31)1c∂wϵ∗∂ϵϵ=0=−∫∇lCaEN1−yxTw0∗yxdF−∫xxT∇2lCaEN1−yxTw0∗dF⋅∂wϵ∗∂ϵϵ=0+γ0+∇lCaEN1−y0x0Tw0∗y0x0,where(32)γ0=∫yx∂∂ϵζ1ϵ,x,y+ζ2ϵ,x,y+ζ3ϵ,x,ydFϵ=0,where ζ1ϵ,x,y∈τθu2+1−θ,0, ζ2ϵ,x,y∈−τ1−θ,1−θ, and ζ3ϵ,x,y∈0,θu2+1−θ are from the results of the optimality condition (36).
Combining (30) and (31), we can obtain that(33)1cI+∫xxT∇2lCaEN1−yxTw0∗IFIFx0,y0;w0∗=∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0,where I is an identity matrix with a proper size. Let W0=∫xxT∇2lCaEN1−yxTw0∗dF. Since ∇2lCaENu is nonnegative by (27), W0 can be always invertible. Therefore, we finally obtain(34)IFx0,y0;w0∗=W0−1∇lCaEN1−y0x0Tw0∗y0x0−1cw0∗+γ0.
The proof is completed.Corollary 1.
The influence functionIFx0,y0;w0∗ is bounded, i.e., the CaENSVM is robust to label noise.Proof.
First, sinceζiϵ,x,y,i=1,2,3 are bounded and continuous with respect to ϵ in closed intervals, their corresponding derivatives with respect to ϵ are also bounded. Then, by Assumption 1 and Theorem 1, we have(35)IFx0,y0;w0∗≤λminW0θu2+1−θx0+1cw0∗+γ0<∞.
Foru1≤1−y0x0Tw0∗≤u2, where λmin⋅ is the smallest eigenvalue of a matrix. Otherwise, by (36), ∇lCaEN1−y0x0Tw0∗=0, the boundness of IFx0,y0;w0∗ also holds.Remark 1.
According to Corollary1, the derivative of loss, i.e., ∇lCaENu, significantly relates to the characteristics of the influence function. In fact, because ∇lCaENu is bounded, we can easily deduce the bounds of the influence function. For those convex losses, such as elastic net loss and pinball loss, their derivatives are boundless, which means the corresponding estimators are not robust. In other words, Corollary 1 reveals the reason of the robustness of the CaEN loss, or those concave losses.


## 4.1.2. Resampling Stability to Feature Noise

For the resampling stability to feature noise, we demonstrate this property in line with Huang et al. [11]. Recalling the CaEN loss (10), the subgradient of lCaEN with respect to u is given by(36)∇lCaENu=0,u<u1,τθu1−1−θ,0,u=u1,τθu−1−θ,u1<u<0,−τ1−θ,1−θ,u=0,θu+1−θ,0<u<u2,0,θu2+1−θ,u=u2,0,u>u2.Then, by KKT (Karush–Kuhn–Tucker) conditions, the solution of CaENSVM satisfies(37)0∈ncw−∑i=1n∇lCaEN1−yiwTxiyixi,where 0 is a proper length vector with all components equal to zero. For given w, the training sample index set can be partitioned into the following seven sets:(38)S0w=i:1−yiwTxi<u1,S1w=i:1−yiwTxi=u1,S2w=i:u1<1−yiwTxi<0,S3w=i:1−yiwTxi=0,S4w=i:0<1−yiwTxi<u2,S5w=i:1−yiwTxi=u2,S6w=i:1−yiwTxi>u2.Using the notations (38), the optimal conditions (37) can be equivalently rewritten as(39)ncw−∑i∈S1w∇lCaEN1−yiwTxiyixi−∑i∈S2w∇lCaEN1−yiwTxiyixi−∑i∈S3w∇lCaEN1−yiwTxiyixi−∑i∈S4w∇lCaEN1−yiwTxiyixi−∑i∈S5w∇lCaEN1−yiwTxiyixi=0.SinceS1w, S3w, and S5w are based on equalities, it is reasonable to see that S1w, S3w, and S5w have much smaller sizes than S2w or S4w. Thus, the contributions of S1w, S3w, and S5w to (39) are considerably weak. In other words, we can roughly determine w by S2w and S4w, i.e.,(40)ncw+τ∑i∈S2w1−θ−θ1−yiwTxiyixi−∑i∈S4wθ1−yiwTxi+1−θyixi≅0.With parametersτ and θ properly selected, equation (40) indicates that τ controls the sensitivity of the CaENSVM to feature noise. In fact, by (38), 1−θ−θ1−yiwTxi and θ1−yiwTxi+1−θ are both positive, which means that a large τ (close to 1) can well balance the size of S2w and S4w for zero mean feature noise. Therefore, the effect of zero mean feature noise is weakened, and the final separating hyperplane of the CaENSVM is stable for resampling. Along with τ decreasing (close to 0), by (38), the final separating hyperplane is gradually dominated by the instances in S4w. As a result, the classification results are significantly disturbed by the zero mean feature noise around the decision boundary.


## 4.2. Bayes Rule

LetPx=ProbY=1X=x be the conditional probability of the positive class given X=x. Lin [46] claimed that signPx−1/2 is the decision-theoretic optimal classification rule with the smallest generalization error, which is the so-called Bayes rule. We can also equivalently define Bayes rule as(41)fCx=+1,Proby=+1x≥Proby=−1x;−1,Proby=+1x<Proby=−1x.For any loss functionl⋅, we define the expected risk of a classifier f:X⟼Y as(42)Rl,ρf=∫X×Yl1−yfxdρ.By minimizing the expected risk over all measurable functions, we can obtain fl,ρx as(43)fl,ρx=argminμ∫Yl1−yμdρyx,∀x∈X,where ρyx is the conditional distribution of y given x. Note that ρyx is a binary distribution, corresponding to Proby=+1x and Proby=−1x.Huang et al. [11] proved that the pinball loss can lead to the Bayes classifier. In the following, we demonstrate that the Bayes rule also holds for our proposed capped asymmetric elastic net loss function.Theorem 2.
The decision functionflCaEN,ρ obtained by minimizing lCaEN-based expected risk over all measurable functions f:X⟶Y is equivalent to the Bayes rule, i.e., flCaEN,ρx=fCx,∀x∈X.Proof.
By simple calculating, we can obtain(44)∫YlCaEN1−yμdρyx=lCaEN1−μProby=+1x+lCaEN1+μProby=−1x.
According to (10), it follows that(45)lCaEN1−μ=s,μ<1−u2,θ21−μ2+1−θ1−μ,1−u2≤μ<1,τθ21−μ2−1−θ1−μ,1≤μ<1−u1,s,μ≥1−u1,and(46)lCaEN1+μ=s,μ≤u1−1,τθ21+μ2−1−θ1+μ,u1−1<μ≤−1,θ21+μ2+1−θ1+μ,−1<μ≤u2−1,s,μ>u2−1.
LetP+1=Proby=+1x and P−1=Proby=−1x, respectively. We denote by r1θ,μ=θ/21−μ2+1−θ1−μ, r2θ,μ=τθ/21−μ2−1−θ1−μ, r3θ,μ=θ/21+μ2+1−θ1+μ, and r4θ,μ=τθ/21+μ2−1−θ1−μ for convenience, respectively. We found that the expected risk relates to the value of u2. Thus, based on (45) and (46) and the continuity of lCaEN, we discuss all cases as follows:
For the case of0<u2<1, we have(47)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤−1,sP+1+r3θ,μP−1,−1<μ≤u2−1,sP+1+sP−1,u2−1<μ≤1−u2,r1θ,μP+1+sP−1,1−u2<μ≤1,r2θ,μP+1+sP−1,1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=1, u2−1=1−u2 and the interval u2−1,1−u2 disappears. One can easily verify that the minimal value is not affected. Supposing that P+1>P−1, the minimal value of μ is +1 according to the increase-decrease characteristics of lCaEN with respect to μ in each interval. Similarly, the minimal value of μ is −1 if P+1<P−1 or the minimal value of μ is +1 or −1 if P+1=P−1. Consequently, we have flCaENx=fCx when 0<u2≤1.
For the case of1<u2<2, we have(48)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤−1,sP+1+r3θ,μP−1,−1<μ≤1−u2,r1θ,μP+1+r3θ,μP−1,1−u2<μ≤u2−1,r1θ,μP+1+sP−1,u2−1<μ≤1,r2θ,μP+1+sP−1,1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=2, we have 1−u2=−1 and u2−1=1. Then, the intervals −1,1−u2, and 1,u2−1 both disappear, which has no effects on the minimum value. After simple calculations, one can find that the minimal value is the same as 0<u2≤1. Thus, we have flCaENx=fCx when 1<u2≤2.
For the case of2<u2<2−u1, we have(49)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤u1−1,sP+1+r4θ,μP−1,u1−1<μ≤1−u2,r1θ,μP+1+r4θ,μP−1,1−u2<μ≤−1,r1θ,μP+1+r3θ,μP−1,−1<μ≤+1,r2θ,μP+1+r3θ,μP−1,1<μ≤u2−1,r2θ,μP+1+sP−1,u2−1<μ≤1−u1,sP+1+sP−1,μ>1−u1.
Ifu2=2−u1, u1−1=1−u2, and u2−1=1−u1, the interval u1−1,1−u2, and u2−1,1−u1 disappear, which has no effects on the minimum value; the minimal value is the same as 0<u2≤1, i.e., we have flCaENx=fCx when 2<u2≤2−u1.
For the case ofu2>2−u1, we have(50)∫YlCaEN1−yμdρyx=sP+1+sP−1,μ≤1−u2,r1θ,μP+1+sP−1,1−u2<μ≤u1−1,r1θ,μP+1+r4θ,μP−1,u1−1<μ≤−1,r1θ,μP+1+r3θ,μP−1,−1<μ≤+1,r2θ,μP+1+r3θ,μP−1,1<μ≤1−u1,sP+1+r3θ,μP−1,1−u1<μ≤u2−1,sP+1+sP−1,μ>u2−1.
One can easily obtain the same minimal value as0<u2≤1, i.e., we have flCaENx=fCx when u2>2−u1. With the abovementioned results, minimizing lCaEN-based expected risk over all measurable functions can lead to the Bayes rule. Thus, Theorem 2 is proved.


## 4.3. Generalization Error Bound

We have proved that the proposed CaENSVM is equivalent to the Bayes rule, which is also called classification-calibrated in the study of Bartlett et al. [7], indicating that the CaENSVM enjoys many good properties. Here, we further give the generalization error bound of the CaENSVM based on the empirical Rademacher complexity [6], where the empirical Rademacher complexity is defined as follows:Definition 1.
Supposing thatx1,⋯,xn are independently selected from X with a probability distribution ν, and F is a real-valued function class mapping from X to R, the empirical Rademacher complexity of F is a random variable defined as(51)R^nF=Eσsupf∈F2n∑i=1nσifxi:x1,⋯,xn,where σ=σ1,…,σnT is independently uniform ±1-valued (Rademacher) random variables and Eσ⋅ means the expectation over σ. The Rademacher complexity of F is(52)RnF=EνR^nF,where Eν⋅ means the expectation over ν.
According to (51), R^nF can be viewed as a correlation between f=fx1,⋯,fxnT and σ for given x1,⋯,xn. Thus, the higher R^nF is, the more complex F is. By (52), RnF depicts the average complexity of F based on ν instead of a set of particular samples.
In line with the abovementioned definitions and lemmas in the study of Bartlett and Mendelson [6], we provide the following theorem to yield the generalization error bound of the CaENSVM.Theorem 3.
Fixζ∈0,1 and B∈R+ and consider the binary classification problem on xiT,yii=1n drawn independently from a probability distribution F. Let F=ff:x↦wTϕx,w≤B and G=gg:y,fx↦−yfx,f∈F be function classes, respectively. If the CaEN loss (10) with s≥1/2 and the optimal w0∗ satisfies w0∗≤B, then with probability at least 1−ζ, the prediction function fx satisfies f∈F and(53)Probyfx≤0≤2n∑i=1nlCaEN1−yifxi+8Bθu2+1−θn∑i=1nKxi,xi+8ln2/ζn.Proof.
define the Heaviside functionΞ⋅ as(54)Ξu=1,u≥0,0,otherwise.
Then, we can easily obtain(55)Probyfx≤0=EρΞ−yfx.
By definingΨu=2lCaEN1+u;τ,θ,s=1/2, we can easily verify that u2≤1 and Ψu∈0,1 dominates the function Ξ⋅ on the support of ρ, we can obtain the following inequality by Theorem 8 in the study of Bartlett and Mendelson [6] as(56)EρΞgy,fx≤EρΨgy,fx≤E^nΨgy,fx+R^nΨ˜G+8ln2/ζn,where E^nΨgy,fx=1/n∑i=1nΨgfxi,yi and Ψ˜G=x,y↦Ψgy,fx−Ψgy,0:f∈F.
According to the CaEN loss (10) with the optimal w0∗, we have(57)E^nΨgy,fx=1n∑i=1nΨgyi,fxi=2n∑i=1nlCaEN1−yifxi.
By Theorem 14 in the study of Bartlett and Mendelson [6], and since Lipschitz function Ψu pertains to Lipschitz constant θu2+1−θ, we have(58)R^nΨ˜°G≤4θu2+1−θR^nG.
According to the definition (51), we can deduce that(59)R^nG=Eσsupg∈G2n∑i=1nσigyi,fxi:x1,⋯,xn=Eσsupf∈F2n∑i=1n−σiyifxi:x1,⋯,xn=Eσsupf∈F2n∑i=1nσifxi:x1,⋯,xn=R^nF.
Hence, by Lemma 22 in the study of Bartlett and Mendelson [6], R^nF satisfies(60)R^nF≤2Bn∑i=1nKxi,xi.
Finally, combining (56)–(58) and (60), we can reach the result of the theorem.Remark 2.
From Theorem3, if the number of training samples n⟶+∞, we have Probyfx≤0⟶0. Therefore, the generalization capability of the proposed CaENSVM can be theoretically guaranteed.Remark 3.
According to the proof of Theorem3, s=1/2 can lead to a tighter generalization error bound. However, experimental results indicate that larger s may produce a more satisfactory classifier.


## 5. Numerical Studies

In this section, we conduct a host of experiments to investigate the performance of the proposed CaENSVM on both synthetic and benchmark datasets. For fair assessment, we compare with several famous or recent related SVMs, including ENSVM [39], PinSVM [11], RampSVM [28], Rhinge-SVM [36], and Valley-SVM [13]. Note that Tang et al. [13] first proposed the valley loss and applied it to the RSSVM for robust multiclass classification, we here combine the valley loss with the standard SVM to construct a novel robust Valley-SVM for binary classification. All experiments are carried out in R 4.0.5 on Ubuntu 18 running on a PC with system configuration AMD R9 5900x CPU (3.70 GHz) with 16 GB of RAM.For PinSVM and CaENSVM, we setτ∈0.3,0.5,0.7. For RampSVM, Valley-SVM, and CaENSVM, we optimize s in 1,1.5,2. Parameter η of Rhinge-SVM is tuned in 0.2,0.5,1,2 like [36] and parameter θ of CaENSVM is tuned in 0.1,0.5,0.9, respectively. For nonlinear cases, we consider Gaussian kernel, i.e., Kx1,x2=exp−γx1−x2, where γ>0 is the kernel parameter. If not otherwise specified, all remaining parameters are optimized in 2−8,2−7,⋯,27,28, including kernel parameter. We use the five-fold cross-validation strategy to search for the optimal parameters. Note that, for the implemented Pegasos-based DC algorithm, eps is fixed to 10−3, T1=10, and T2=500 through our experiments. According to the numerical studies, this setting can often lead to a satisfactory result.


### 5.1. Synthetic Datasets

We generate a two-dimensional synthetic dataset to test the robustness of CaENSVM. The training dataset consists of 60 equal positive and negative samples. The positive samples are independently drawn from a two-dimensional normal distribution with the mean vectorμ+=−0.4,1.0T and the covariance matrix V=diag0.02,0.06. The negative samples are independently drawn from a similar normal distribution with the mean vector μ−=0.4,0.2T and the same covariance matrix.Case 1.
In this case, we only add three extra outliers for positive samples. Figure3 shows the training samples and the separating lines (black solid lines) obtained by six SVMs. Note that the green “+” notation in each figure is the midpoint between the centers of two classes of samples. Since the distributions behind two classes of training samples only differ from the location of centers, it is reasonable for the obtained separating line crossing the midpoint, i.e., the green “+” notation in each figure. In other words, we can measure the level of disturbance caused by outliers by comparing the relative location between the obtained separating lines and the corresponding midpoints.
According to Figure3, our proposed CaENSVM is more robust to outliers and produces the most satisfactory classifier. Due to the infinity of EN losses, ENSVM is easily attracted by outliers. As Figure 3(a) shows, ENSVM distinguishes two classes of samples worst, and the obtained separating line is clearly away from the midpoint, indicating its high sensitivity to outliers. Though pinball loss is unbounded, the correctly classified samples also produce losses, which behave like a balance and can reduce the attraction of outliers. From Figure 3(b), PinSVM performs little better than the ENSVM. Note that the separating line induced by the PinSVM is also close to outliers, which means PinSVM is still sensitive to outliers. ramp, Rhinge, and valley losses are all concave and bounded; they can limit the influence of outliers and contribute to robust classifiers. The performances of RampSVM and Rhinge-SVM closely resemble each other. However, both provide totally different shapes of classifiers compared with others. According to Figures 3(c) and 3(d), the corresponding classifiers indicate their overfitness as well as the sensitivity to outliers. Valley-SVM and our proposed CaENSVM share analogical performance. According to Figures 3(e) and 3(f) and based on the midpoint, our CaENSVM seems slightly better than Valley-SVM.Figure 3
The separating hyperplanes (black solid lines) obtained by ENSVM, PinSVM, RampSVM, Rhinge-SVM, Valley-SVM, and CaENSVM, respectively. The green “+” notation is the midpoint between the centers of two classes of samples. (a) ENSVM. (b) PinSVM. (c) RampSVM. (d) Rhinge-SVM. (e) Valley-SVM. (f) CaENSVM.
(a)(b)(c)(d)(e)(f)Case 2.
In this case, we add three extra outliers for both positive and negative samples. Figure4 shows the training samples and the final decision lines (black solid lines) of six SVMs. The midpoint between the centers of two classes of samples is also marked in each figure to help measure the robustness of each classifier.
According to Figure4, our proposed CaENSVM still performs best compared with other five SVMs. ENSVM, RampSVM, and Rhinge-SVM present similar performances. They are all severely attracted by outliers and provide unsatisfactory decision lines. Though the decision line obtained by PinSVM is also deviated by outliers, it seems clearly better than those given by the ENSVM, RampSVM, and Rhinge-SVM. In our opinion, since pinball loss produces losses for corrected classified training samples, it can balance and reduce the influence of outliers to some extent. Valley-SVM performs competitively like PinSVM, but the decision line of Valley-SVM is also drawn by outliers. In comparison with other five SVMs, our proposed CaENSVM shows the best robustness against outliers as before.Figure 4
The separating hyperplanes (black solid lines) obtained by ENSVM, PinSVM, RampSVM, Rhinge-SVM, Valley-SVM, and CaENSVM, respectively. The green “+” notation is the midpoint between the centers of two classes of samples. (a) ENSVM. (b) PinSVM. (c) RampSVM. (d) Rhinge-SVM. (e) Valley-SVM. (f) CaENSVM.
(a)(b)(c)(d)(e)(f)Case 3.
In this case, we investigate the time cost of the implemented Pegasos-based DC procedure for the proposed CaENSVM. Specifically, we turn to generate from 50 to 5000 equal positive and negative training samples. For a fair assessment, we compare the PinSVM solved by clipDCD [47] and Rhinge-SVM solved by clipDCD-based half-quadratic optimization algorithm [36]. All tuning parameters are fixed to 0.5 for simplicity. Figures 5 and 6 show the one-run CPU time of PinSVM, Rhinge-SVM, and CaENSVM with linear and Guassian kernels, respectively.
According to Figures5 and 6, our implemented Pegasos-based DC procedure obviously runs the fastest in comparison with PinSVM and Rhinge-SVM. For the linear kernel, PinSVM and Rhinge-SVM consume time similarly, though Rhinge-SVM needs to iterate a clipDCD chunk many times. It may be that the adaptive weighting scheme and sparsity of Rhinge-SVM can help reduce the training cost. Our CaENSVM runs fast, and the time cost is free of the sample size, indicating the scalability of Pegasos [43]. For the Guassian kernel, Rhinge-SVM is clearly more time-consuming than PinSVM due to the outer iteration of the half-quadratic procedure. The training time of CaENSVM also depends on the sample size, but it is still efficient. According to the experimental results, our proposed CaENSVM can be easily applied in solving large-scale data classification problems.Figure 5
The one-run CPU time (in seconds) of PinSVM, Rhinge-SVM, and CaENSVM with linear kernel.x-coordinate is the log10(sample size), while y-coordinate is the training time (seconds).Figure 6
The one-run CPU time (in seconds) of PinSVM, Rhinge-SVM, and CaENSVM with Gaussian kernel.x-coordinate is the log10(sample size), while y-coordinate is the training time (seconds).


### 5.2. UCI Datasets

#### 5.2.1. Experimental Settings and Results

We select twelve UCI datasets to further demonstrate the advantages of our proposed CaENSVM. The detailed information of the chosen datasets is listed in Table1. To investigate the label noise insensitivity, we artificially add 15% and 25% label noises to the raw datasets, i.e., we randomly select 15% and 25% training samples and exchange their labels. The experimental results with linear and Gaussian kernels based on five-fold cross-validation criterion are shown in Tables 2 and 3, respectively.Algorithm 1: A general framework of the DC algorithm for (15).
Require: k=0; Θ0, the initial value of Θ.Ensure: optimal solution of (15).(1)
repeat(2)
Θk+1=argminΘgΘ−h′ΘkTΘ−Θk, h′ is the derivative of h with respect to Θ.(3)
until convergence.(4)
returnΘk+1.Algorithm 2: Pegasos-based DC procedure for CaENSVM.
Require:T1, T2, eps, w0, xiT,yii=1n, c, τ, θ, and s.Ensure: optimal solution of (11).(1)
Sett=0, v0=w0.(2)
whilet≤T2do(3)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(4)
Compute∇Fvt by (21) and (22).(5)
Setηt=c/t.(6)
Setvt+1←vt−ηt∇Fvt.(7)
end while(8)
Setw1=vt+1.(9)
Setk=0.(10)
while (k≤T1 & wk+1−wk≥eps) do(11)
Setk←k+1.(12)
Sett=0, v0=wk.(13)
whilet≤T2do(14)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(15)
Compute∇Fvt by (21) and (22).(16)
Setηt=c/t.(17)
Setvt+1←vt−ηt∇Fvt.(18)
end while(19)
Setwk+1=vt+1.(20)
end while(21)
returnwk+1.Table 1
The information of UCI datasets.
IDDataset#Obs#Fea1Absenteeism740192Autism702153baDS137254Banknote137145Forestfire121106ILPD579107Knowledge25158Messidor1151199Popfailure5401510Seed210811wdbc5693412Wine1301313Amphibians1892114Raisin900715Tripadvisor9809From Table2 our proposed CaENSVM with linear kernel outperforms others in most cases, according to the average prediction accuracies. For the case without label noise, our CaENSVM performs slightly better than the ENSVM, followed by Valley-SVM. Rhinge-SVM and RampSVM have competitive performances, while PinSVM seems to be the worst. Along with the ratio of label noise increasing, the performances of all SVMs seem roughly reduced. Specifically, the prediction accuracy of the ENSVM is most affected by label noise, since the elastic net loss lacks robustness. Though PinSVM performs little stably due to the asymmetric pinball loss, its prediction accuracy is often the most unsatisfactory and competitive with the ENSVM for the case with moderate and high label noise. Due to the robustness of ramp, Rhinge, and valley losses, RampSVM and Rhinge-SVM together with Valley-SVM present more clearly high prediction accuracies for the case with label noise in comparison with ENSVM and PinSVM. Because our designed CaEN loss enjoys both outlier insensitivity and resampling stability, the CaENSVM always achieves the highest average prediction accuracies for the cases with label noise. Therefore, the advantage of CaENSVM becomes more concrete, indicating its good robustness to label noise.From Table3, our proposed CaENSVM with Gaussian kernel slightly performs better than others according to the average prediction accuracies. For the raw datasets, the CaENSVM gives higher average prediction accuracies in more than half of the datasets. The PinSVM turns to perform well, only behind the CaENSVM. RampSVM also has a good performance, but it is clearly worse than PinSVM. The remaining three SVMs are competitive with each other. For the datasets with moderate and high label noise, the CaENSVM still has a weak superiority, followed by Rhinge-SVM and PinSVM. ENSVM and Valley-SVM seem to be the worst in most datasets. Compared with the predicting performance in the linear cases, the superiority of our proposed CaENSVM is not overwhelming. In our opinion, this situation is possibly caused by the Pegasos procedure, since the efficiency of Pegasos is reduced in the nonlinear cases [43]. We will consider designing efficient algorithms for nonlinear cases in future.


#### 5.2.2. Comparisons by Statistical Test

In this part, we conduct a statistical test on the experimental results of UCI datasets to further demonstrate the advantage of the CaENSVM. Specifically, we utilize the famous Friedman test with the corresponding post hoc test [48] to study whether there is a statistically significant difference among CaENSVM and other five compared SVMs. Firstly, we calculate the average ranks of each method with respect to different kernels and ratios of label noise in Tables 2 and 3. The results are presented in Table 4.Table 2
The mean accuracy (Acc.) and standard deviation (sd) with linear kernel for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd(a)0%label noiseAbsenteeism0.946 ± 0.0140.946 ± 0.0180.946 ± 0.0120.946 ± 0.0510.954 ± 0.0120.973 ± 0.013Autism0.836 ± 0.0230.734 ± 0.0240.734 ± 0.0250.729 ± 0.0830.810 ± 0.0670.993 ± 0.013baDS0.968 ± 0.0120.912 ± 0.0290.982 ± 0.0150.972 ± 0.0150.977 ± 0.0110.987 ± 0.007Banknote0.983 ± 0.0130.936 ± 0.0140.983 ± 0.0110.975 ± 0.0180.974 ± 0.0060.992 ± 0.005Forestfire0.870 ± 0.0820.652 ± 0.0150.652 ± 0.0090.809 ± 0.0730.835 ± 0.0570.922 ± 0.020ILPD0.649 ± 0.0160.287 ± 0.0250.287 ± 0.0130.381 ± 0.6870.447 ± 0.2200.716 ± 0.012Knowledge0.931 ± 0.0400.967 ± 0.0310.959 ± 0.0320.939 ± 0.0430.718 ± 0.1520.967 ± 0.031Messidor0.678 ± 0.0440.481 ± 0.0250.542 ± 0.1190.687 ± 0.0370.675 ± 0.0230.642 ± 0.073Popfailure0.735 ± 0.3640.084 ± 0.0130.916 ± 0.0090.490 ± 0.3800.084 ± 0.0120.929 ± 0.025Seed0.907 ± 0.0410.500 ± 0.0220.500 ± 0.0100.800 ± 0.0410.686 ± 0.1500.929 ± 0.025wdbc0.894 ± 0.0270.372 ± 0.0360.372 ± 0.0240.857 ± 0.0520.897 ± 0.0180.715 ± 0.178Wine0.888 ± 0.0520.560 ± 0.0110.560 ± 0.0120.872 ± 0.0520.904 ± 0.0610.912 ± 0.052Amphibians0.671 ± 0.0440.692 ± 0.0820.681 ± 0.0880.698 ± 0.0550.687 ± 0.0450.698 ± 0.070Raisin0.864 ± 0.0300.864 ± 0.0380.863 ± 0.0340.864 ± 0.0310.834 ± 0.0410.872 ± 0.013Tripadvisor0.733 ± 0.0120.732 ± 0.0050.735 ± 0.0090.731 ± 0.0080.740 ± 0.0090.741 ± 0.008(b)15%label noiseAbsenteeism0.935 ± 0.0240.946 ± 0.0120.946 ± 0.0080.795 ± 0.3310.946 ± 0.0200.948 ± 0.010Autism0.806 ± 0.0350.734 ± 0.0230.734 ± 0.0020.794 ± 0.0680.796 ± 0.0880.813 ± 0.028baDS0.973 ± 0.0170.915 ± 0.0470.973 ± 0.0120.974 ± 0.0070.968 ± 0.0120.977 ± 0.010Banknote0.969 ± 0.0110.905 ± 0.0510.979 ± 0.0140.975 ± 0.0120.972 ± 0.0110.979 ± 0.011Forestfire0.848 ± 0.0950.652 ± 0.0150.652 ± 0.0230.739 ± 0.1410.800 ± 0.1400.852 ± 0.050ILPD0.402 ± 0.1580.287 ± 0.0080.287 ± 0.0250.640 ± 0.1750.593 ± 0.1820.654 ± 0.161Knowledge0.947 ± 0.0310.972 ± 0.0230.972 ± 0.0230.976 ± 0.0270.690 ± 0.1140.979 ± 0.029Messidor0.585 ± 0.0960.470 ± 0.0220.548 ± 0.1280.578 ± 0.0260.635 ± 0.0520.636 ± 0.033Popfailure0.677 ± 0.3320.084 ± 0.0000.905 ± 0.0220.084 ± 0.0060.084 ± 0.0230.802 ± 0.099Seed0.900 ± 0.0590.500 ± 0.0250.500 ± 0.0190.807 ± 0.0570.857 ± 0.1240.824 ± 0.160wdbc0.810 ± 0.0380.372 ± 0.0310.372 ± 0.0150.780 ± 0.0420.832 ± 0.0450.864 ± 0.039Wine0.872 ± 0.0180.560 ± 0.0100.560 ± 0.0320.768 ± 0.1070.888 ± 0.0440.896 ± 0.036Amphibians0.660 ± 0.0240.665 ± 0.0530.681 ± 0.0480.665 ± 0.0310.638 ± 0.0650.692 ± 0.053Raisin0.860 ± 0.0310.855 ± 0.0250.870 ± 0.0260.862 ± 0.0260.856 ± 0.0190.875 ± 0.018Tripadvisor0.723 ± 0.0140.729 ± 0.0020.728 ± 0.0050.730 ± 0.0040.710 ± 0.0180.724 ± 0.004(c)25%label noiseAbsenteeism0.916 ± 0.0670.940 ± 0.0230.940 ± 0.0080.772 ± 0.3600.945 ± 0.0030.946 ± 0.008Autism0.746 ± 0.0500.734 ± 0.0100.763 ± 0.0640.816 ± 0.0920.779 ± 0.1360.780 ± 0.035baDS0.972 ± 0.0140.887 ± 0.0980.977 ± 0.0080.966 ± 0.0170.965 ± 0.0200.978 ± 0.013Banknote0.972 ± 0.0130.966 ± 0.0210.980 ± 0.0140.976 ± 0.0150.978 ± 0.0110.982 ± 0.011Forestfire0.791 ± 0.0480.652 ± 0.0250.652 ± 0.0120.661 ± 0.1420.783 ± 0.1020.800 ± 0.079ILPD0.421 ± 0.1860.287 ± 0.0110.346 ± 0.1320.515 ± 0.1790.657 ± 0.0440.546 ± 0.125Knowledge0.939 ± 0.0380.963 ± 0.0300.967 ± 0.0230.967 ± 0.0230.657 ± 0.1060.972 ± 0.030Messidor0.686 ± 0.0320.470 ± 0.0230.564 ± 0.0880.610 ± 0.0570.660 ± 0.0480.595 ± 0.101Popfailure0.673 ± 0.0380.084 ± 0.0340.914 ± 0.0200.084 ± 0.0250.090 ± 0.0130.693 ± 0.031Seed0.907 ± 0.0480.500 ± 0.0120.500 ± 0.0230.910 ± 0.1000.600 ± 0.0960.914 ± 0.074wdbc0.871 ± 0.0390.372 ± 0.0150.372 ± 0.0410.731 ± 0.0820.892 ± 0.0390.894 ± 0.047Wine0.852 ± 0.0660.560 ± 0.0200.560 ± 0.0260.664 ± 0.0610.856 ± 0.1080.860 ± 0.321Amphibians0.649 ± 0.0850.725 ± 0.0480.676 ± 0.0540.687 ± 0.0590.665 ± 0.0560.698 ± 0.052Raisin0.856 ± 0.0250.851 ± 0.0110.863 ± 0.0140.863 ± 0.0140.846 ± 0.0310.867 ± 0.006Tripadvisor0.723 ± 0.0310.731 ± 0.0070.736 ± 0.0150.736 ± 0.0130.732 ± 0.0080.737 ± 0.011The bold is the best one.Secondly, we need to obtain the Friedman statistic based on Table4. Let Dn and Ck be the total number of compared datasets and classifiers, respectively. By Tables 2 and 3, we have Dn=15 and Ck=6 for each type of kernel and ratio of label noise. Then, we consider the following F statistic:(61)FF=Dn−1χF2DnCk−1−χF2,where χF2 is the raw Friedman statistic defined as(62)χF2=12DnCkCk+1∑i=1CkRi2−CkCk+124,where Ri is the average rank of the i-th classifier for each type of kernel and ratio of label noise. The obtained FF statistic obeys F distribution with Ck−1 and Ck−1Dn−1 degrees of freedom. When the significant level is set to 0.1, we have F0.15,70=1.93. According to Table 4, we obtain that the values of FF are 8.57, 8.34, and 8.87 with respect to different ratios of label noise for linear cases, 19.27, 21.21, and 12.69 with respect to different ratios of label noise for nonlinear cases. All values of FF statistic are larger than the critical value 1.93, which means that there is indeed a statistical difference among those compared six SVMs.Thirdly, we apply the Nemenyi test for post hoc test to further distinguish the detailed differences of six classifiers. The critical domainCD of the difference of the average ranks of two SVMs given by Nemenyi is defined as(63)CD=q0.1CkCk+16Dn=1.769,where q0.1=2.589. If the absolute difference of two SVMs is larger than CD, it means that they perform statistically differently. Otherwise, they have no statistical differences with each other. Figure 7 shows the comparison of the average ranks of each SVM for different type of kernels and ratios of label noise.Figure 7
Comparison by the Nemenyi test under 0.1 significant level. (a)–(c) correspond to the linear cases with 0%, 15%, and 25% label noises, while (d)–(f) correspond to the nonlinear cases with 0%, 15%, and 25% label noises, respectively. (a) Case 1. (b) Case 2. (c) Case 3. (d) Case 4. (e) Case 5. (f) Case 6.
(a)(b)(c)(d)(e)(f)According to Figure7, there is no significant difference between CaENSVM and ENSVM with linear kernel. However, by Figures 7(a)–7(c), the difference between CaENSVM and ENSVM becomes larger as the ratio of label noise increases, which means the proposed CaENSVM is still better than others. Rhinge-SVM, RampSVM, and PinSVM always have no significant differences for linear cases, they all are significantly worse than CaENSVM. Valley-SVM shows good robustness for a high ratio of label noise, but there is still a gap between it and CaENSVM. For the cases with Gaussian kernel, our CaENSVM always presents slightly better performances than others, though the differences among it and Rhinge-SVM and PinSVM are not significant. The performances of Valley-SVM and ENSVM are significantly worse than those of other four methods, especially for high ratio of label noise. In short, our CaENSVM enjoys good performance in the statistical viewpoint.


### 5.3. Handwritten Digit Recognition

In this subsection, we apply CaENSVM to solve a real problem, i.e., handwritten digit recognition. The test dataset is the PMU-UD dataset from the study of Alghazo et al. [49], containing handwritten Urdu/Arabic numerals from 0 to 9. Each handwritten number is standardized as a 120×80 image. Figure 8 shows four selected images for every handwritten number. We choose the datasets corresponding to two different handwritten numbers each time to conduct binary classification. The information of all considered datasets is listed in Table 5.Figure 8
Selected images from the PMU-UD dataset.Table 3
The mean accuracy (Acc.) and standard deviation (sd) with Gaussian kernel for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd(a)0%label noiseAbsenteeism0.946 ± 0.0530.955 ± 0.0030.947 ± 0.0100.946 ± 0.1200.946 ± 0.0220.955 ± 0.003Autism0.921 ± 0.1060.978 ± 0.0120.997 ± 0.0040.978 ± 0.0890.903 ± 0.0810.999 ± 0.003baDS0.987 ± 0.0101.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.944 ± 0.051.000 ± 0.000Banknote0.980 ± 0.0100.996 ± 0.0080.993 ± 0.0100.944 ± 0.0180.957 ± 0.0240.998 ± 0.005Forestfire0.861 ± 0.0840.913 ± 0.0620.870 ± 0.0440.635 ± 0.1460.722 ± 0.0900.931 ± 0.039ILPD0.711 ± 0.0130.716 ± 0.0220.621 ± 0.1870.714 ± 0.2220.374 ± 0.1950.727 ± 0.018Knowledge0.882 ± 0.0700.951 ± 0.0230.963 ± 0.0270.926 ± 0.1480.718 ± 0.1250.968 ± 0.028Messidor0.585 ± 0.0330.649 ± 0.0380.603 ± 0.0170.529 ± 0.0430.497 ± 0.0400.619 ± 0.054Popfailure0.563 ± 0.4390.744 ± 0.3690.905 ± 0.0180.497 ± 0.2910.084 ± 0.0340.903 ± 0.022Seed0.936 ± 0.0460.971 ± 0.0300.936 ± 0.0460.949 ± 0.0360.943 ± 0.0540.971 ± 0.030wdbc0.727 ± 0.0790.917 ± 0.0180.680 ± 0.0400.907 ± 0.1140.372 ± 0.0230.908 ± 0.015Wine0.728 ± 0.1070.896 ± 0.0460.648 ± 0.0660.864 ± 0.0460.568 ± 0.0180.888 ± 0.033Amphibians0.633 ± 0.0930.789 ± 0.0350.698 ± 0.0670.730 ± 0.0510.568 ± 0.0000.692 ± 0.059Raisin0.865 ± 0.0230.860 ± 0.0280.852 ± 0.0160.860 ± 0.0160.828 ± 0.0430.868 ± 0.012Tripadvisor0.725 ± 0.0070.733 ± 0.0160.722 ± 0.0140.742 ± 0.0090.728 ± 0.0000.734 ± 0.014(b)15%label noiseAbsenteeism0.946 ± 0.0130.946 ± 0.0140.947 ± 0.0030.943 ± 0.0150.946 ± 0.0080.970 ± 0.053Autism0.873 ± 0.1270.951 ± 0.0250.996 ± 0.0040.964 ± 0.0080.846 ± 0.1050.997 ± 0.065baDS0.945 ± 0.0320.946 ± 0.0040.941 ± 0.0130.949 ± 0.0030.940 ± 0.0350.951 ± 0.055Banknote0.986 ± 0.0100.993 ± 0.0071.000 ± 0.0000.966 ± 0.0080.879 ± 0.1481.000 ± 0.000Forestfire0.818 ± 0.0950.913 ± 0.0690.905 ± 0.0780.910 ± 0.0720.739 ± 0.0870.912 ± 0.082ILPD0.556 ± 0.2310.717 ± 0.0260.696 ± 0.0490.701 ± 0.0290.287 ± 0.0210.698 ± 0.076Knowledge0.857 ± 0.0970.929 ± 0.0290.917 ± 0.0310.921 ± 0.0340.710 ± 0.1130.937 ± 0.024Messidor0.586 ± 0.0310.629 ± 0.0390.625 ± 0.0280.651 ± 0.0200.478 ± 0.0170.634 ± 0.062Popfailure0.222 ± 0.3090.731 ± 0.3640.725 ± 0.3600.392 ± 0.4230.084 ± 0.0230.735 ± 0.192Seed0.936 ± 0.0390.914 ± 0.0740.936 ± 0.0390.936 ± 0.0300.836 ± 0.1880.929 ± 0.061wdbc0.703 ± 0.0790.436 ± 0.0530.626 ± 0.1430.864 ± 0.0420.372 ± 0.0230.890 ± 0.066Wine0.648 ± 0.0870.812 ± 0.0720.648 ± 0.0870.820 ± 0.0630.560 ± 0.0100.844 ± 0.128Amphibians0.611 ± 0.0530.660 ± 0.0620.649 ± 0.0990.708 ± 0.0840.579 ± 0.0240.714 ± 0.056Raisin0.693 ± 0.1770.860 ± 0.0210.864 ± 0.0200.855 ± 0.0330.840 ± 0.0140.867 ± 0.026Tripadvisor0.726 ± 0.0110.718 ± 0.0190.691 ± 0.0390.726 ± 0.0220.656 ± 0.1020.728 ± 0.010(c)25%label noiseAbsenteeism0.945 ± 0.0030.936 ± 0.0130.943 ± 0.0080.945 ± 0.0080.946 ± 0.0230.946 ± 0.015Autism0.883 ± 0.0850.967 ± 0.0140.969 ± 0.0060.973 ± 0.0210.796 ± 0.0850.974 ± 0.012baDS0.933 ± 0.0650.972 ± 0.0040.971 ± 0.0100.975 ± 0.0060.956 ± 0.0220.978 ± 0.082Banknote0.987 ± 0.0040.990 ± 0.0080.990 ± 0.0020.995 ± 0.0060.953 ± 0.0160.993 ± 0.042Forestfire0.809 ± 0.1090.870 ± 0.0440.852 ± 0.0390.852 ± 0.0580.817 ± 0.1080.867 ± 0.021ILPD0.602 ± 0.1780.697 ± 0.0070.689 ± 0.0150.709 ± 0.0480.287 ± 0.0210.712 ± 0.091Knowledge0.841 ± 0.0600.972 ± 0.0230.943 ± 0.0340.980 ± 0.0140.669 ± 0.1700.814 ± 0.038Messidor0.537 ± 0.0530.617 ± 0.0270.583 ± 0.0400.604 ± 0.0270.502 ± 0.0430.628 ± 0.057Popfailure0.366 ± 0.3860.249 ± 0.3680.733 ± 0.3630.411 ± 0.4480.084 ± 0.0410.784 ± 0.098Seed0.914 ± 0.0650.936 ± 0.0640.929 ± 0.0670.929 ± 0.0440.914 ± 0.0540.871 ± 0.054wdbc0.681 ± 0.0540.864 ± 0.0660.492 ± 0.1710.844 ± 0.0550.372 ± 0.0230.774 ± 0.032Wine0.672 ± 0.0720.716 ± 0.0460.672 ± 0.0520.732 ± 0.1000.560 ± 0.0080.748 ± 0.059Amphibians0.606 ± 0.1220.714 ± 0.0560.735 ± 0.0480.714 ± 0.0410.563 ± 0.0120.714 ± 0.073Raisin0.819 ± 0.0480.852 ± 0.0210.853 ± 0.0170.858 ± 0.0150.747 ± 0.1480.860 ± 0.023Tripadvisor0.673 ± 0.1210.702 ± 0.0410.701 ± 0.0570.708 ± 0.0220.721 ± 0.0080.722 ± 0.008The bold is the best one.For a fair evaluation, we also compare our CaENSVM with ENSVM, PinSVM, RampSVM, Rhinge-SVM, and Valley-SVM. The average prediction accuracies with linear and Gaussian kernels based on the five-fold cross-validation criterion are presented in Tables6 and 7, respectively. According to Tables 6 and 7, our CaENSVM achieves the highest prediction accuracies in more than half of the cases, indicating its excellent performance in handwritten digit recognition problems.Table 4
Average ranks of each SVM with respect to different kernels and ratios of label noise for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVM(a) Linear kernelAvg. rank 0%3.2004.8334.2673.7333.5001.467Avg. rank 15%3.5674.9333.8333.4333.8001.433Avg. rank 25%3.8675.0673.7333.4003.4671.467(b) Gaussian kernelAvg. rank 0%4.3672.1003.6003.7005.5331.700Avg. rank 15%4.3333.0673.4002.9005.8001.500Avg. rank 25%4.8333.0333.5332.3675.2002.033Table 5
The information of the PMU-UD dataset.
IDDataset#Obs#Fea12-vs-3937960022-vs-41003960032-vs-61007960042-vs-71019960052-vs-81003960063-vs-4986960076-vs-71072960086-vs-81056960097-vs-810689600


## 5.1. Synthetic Datasets

We generate a two-dimensional synthetic dataset to test the robustness of CaENSVM. The training dataset consists of 60 equal positive and negative samples. The positive samples are independently drawn from a two-dimensional normal distribution with the mean vectorμ+=−0.4,1.0T and the covariance matrix V=diag0.02,0.06. The negative samples are independently drawn from a similar normal distribution with the mean vector μ−=0.4,0.2T and the same covariance matrix.Case 1.
In this case, we only add three extra outliers for positive samples. Figure3 shows the training samples and the separating lines (black solid lines) obtained by six SVMs. Note that the green “+” notation in each figure is the midpoint between the centers of two classes of samples. Since the distributions behind two classes of training samples only differ from the location of centers, it is reasonable for the obtained separating line crossing the midpoint, i.e., the green “+” notation in each figure. In other words, we can measure the level of disturbance caused by outliers by comparing the relative location between the obtained separating lines and the corresponding midpoints.
According to Figure3, our proposed CaENSVM is more robust to outliers and produces the most satisfactory classifier. Due to the infinity of EN losses, ENSVM is easily attracted by outliers. As Figure 3(a) shows, ENSVM distinguishes two classes of samples worst, and the obtained separating line is clearly away from the midpoint, indicating its high sensitivity to outliers. Though pinball loss is unbounded, the correctly classified samples also produce losses, which behave like a balance and can reduce the attraction of outliers. From Figure 3(b), PinSVM performs little better than the ENSVM. Note that the separating line induced by the PinSVM is also close to outliers, which means PinSVM is still sensitive to outliers. ramp, Rhinge, and valley losses are all concave and bounded; they can limit the influence of outliers and contribute to robust classifiers. The performances of RampSVM and Rhinge-SVM closely resemble each other. However, both provide totally different shapes of classifiers compared with others. According to Figures 3(c) and 3(d), the corresponding classifiers indicate their overfitness as well as the sensitivity to outliers. Valley-SVM and our proposed CaENSVM share analogical performance. According to Figures 3(e) and 3(f) and based on the midpoint, our CaENSVM seems slightly better than Valley-SVM.Figure 3
The separating hyperplanes (black solid lines) obtained by ENSVM, PinSVM, RampSVM, Rhinge-SVM, Valley-SVM, and CaENSVM, respectively. The green “+” notation is the midpoint between the centers of two classes of samples. (a) ENSVM. (b) PinSVM. (c) RampSVM. (d) Rhinge-SVM. (e) Valley-SVM. (f) CaENSVM.
(a)(b)(c)(d)(e)(f)Case 2.
In this case, we add three extra outliers for both positive and negative samples. Figure4 shows the training samples and the final decision lines (black solid lines) of six SVMs. The midpoint between the centers of two classes of samples is also marked in each figure to help measure the robustness of each classifier.
According to Figure4, our proposed CaENSVM still performs best compared with other five SVMs. ENSVM, RampSVM, and Rhinge-SVM present similar performances. They are all severely attracted by outliers and provide unsatisfactory decision lines. Though the decision line obtained by PinSVM is also deviated by outliers, it seems clearly better than those given by the ENSVM, RampSVM, and Rhinge-SVM. In our opinion, since pinball loss produces losses for corrected classified training samples, it can balance and reduce the influence of outliers to some extent. Valley-SVM performs competitively like PinSVM, but the decision line of Valley-SVM is also drawn by outliers. In comparison with other five SVMs, our proposed CaENSVM shows the best robustness against outliers as before.Figure 4
The separating hyperplanes (black solid lines) obtained by ENSVM, PinSVM, RampSVM, Rhinge-SVM, Valley-SVM, and CaENSVM, respectively. The green “+” notation is the midpoint between the centers of two classes of samples. (a) ENSVM. (b) PinSVM. (c) RampSVM. (d) Rhinge-SVM. (e) Valley-SVM. (f) CaENSVM.
(a)(b)(c)(d)(e)(f)Case 3.
In this case, we investigate the time cost of the implemented Pegasos-based DC procedure for the proposed CaENSVM. Specifically, we turn to generate from 50 to 5000 equal positive and negative training samples. For a fair assessment, we compare the PinSVM solved by clipDCD [47] and Rhinge-SVM solved by clipDCD-based half-quadratic optimization algorithm [36]. All tuning parameters are fixed to 0.5 for simplicity. Figures 5 and 6 show the one-run CPU time of PinSVM, Rhinge-SVM, and CaENSVM with linear and Guassian kernels, respectively.
According to Figures5 and 6, our implemented Pegasos-based DC procedure obviously runs the fastest in comparison with PinSVM and Rhinge-SVM. For the linear kernel, PinSVM and Rhinge-SVM consume time similarly, though Rhinge-SVM needs to iterate a clipDCD chunk many times. It may be that the adaptive weighting scheme and sparsity of Rhinge-SVM can help reduce the training cost. Our CaENSVM runs fast, and the time cost is free of the sample size, indicating the scalability of Pegasos [43]. For the Guassian kernel, Rhinge-SVM is clearly more time-consuming than PinSVM due to the outer iteration of the half-quadratic procedure. The training time of CaENSVM also depends on the sample size, but it is still efficient. According to the experimental results, our proposed CaENSVM can be easily applied in solving large-scale data classification problems.Figure 5
The one-run CPU time (in seconds) of PinSVM, Rhinge-SVM, and CaENSVM with linear kernel.x-coordinate is the log10(sample size), while y-coordinate is the training time (seconds).Figure 6
The one-run CPU time (in seconds) of PinSVM, Rhinge-SVM, and CaENSVM with Gaussian kernel.x-coordinate is the log10(sample size), while y-coordinate is the training time (seconds).


## 5.2. UCI Datasets

### 5.2.1. Experimental Settings and Results

We select twelve UCI datasets to further demonstrate the advantages of our proposed CaENSVM. The detailed information of the chosen datasets is listed in Table1. To investigate the label noise insensitivity, we artificially add 15% and 25% label noises to the raw datasets, i.e., we randomly select 15% and 25% training samples and exchange their labels. The experimental results with linear and Gaussian kernels based on five-fold cross-validation criterion are shown in Tables 2 and 3, respectively.Algorithm 1: A general framework of the DC algorithm for (15).
Require: k=0; Θ0, the initial value of Θ.Ensure: optimal solution of (15).(1)
repeat(2)
Θk+1=argminΘgΘ−h′ΘkTΘ−Θk, h′ is the derivative of h with respect to Θ.(3)
until convergence.(4)
returnΘk+1.Algorithm 2: Pegasos-based DC procedure for CaENSVM.
Require:T1, T2, eps, w0, xiT,yii=1n, c, τ, θ, and s.Ensure: optimal solution of (11).(1)
Sett=0, v0=w0.(2)
whilet≤T2do(3)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(4)
Compute∇Fvt by (21) and (22).(5)
Setηt=c/t.(6)
Setvt+1←vt−ηt∇Fvt.(7)
end while(8)
Setw1=vt+1.(9)
Setk=0.(10)
while (k≤T1 & wk+1−wk≥eps) do(11)
Setk←k+1.(12)
Sett=0, v0=wk.(13)
whilet≤T2do(14)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(15)
Compute∇Fvt by (21) and (22).(16)
Setηt=c/t.(17)
Setvt+1←vt−ηt∇Fvt.(18)
end while(19)
Setwk+1=vt+1.(20)
end while(21)
returnwk+1.Table 1
The information of UCI datasets.
IDDataset#Obs#Fea1Absenteeism740192Autism702153baDS137254Banknote137145Forestfire121106ILPD579107Knowledge25158Messidor1151199Popfailure5401510Seed210811wdbc5693412Wine1301313Amphibians1892114Raisin900715Tripadvisor9809From Table2 our proposed CaENSVM with linear kernel outperforms others in most cases, according to the average prediction accuracies. For the case without label noise, our CaENSVM performs slightly better than the ENSVM, followed by Valley-SVM. Rhinge-SVM and RampSVM have competitive performances, while PinSVM seems to be the worst. Along with the ratio of label noise increasing, the performances of all SVMs seem roughly reduced. Specifically, the prediction accuracy of the ENSVM is most affected by label noise, since the elastic net loss lacks robustness. Though PinSVM performs little stably due to the asymmetric pinball loss, its prediction accuracy is often the most unsatisfactory and competitive with the ENSVM for the case with moderate and high label noise. Due to the robustness of ramp, Rhinge, and valley losses, RampSVM and Rhinge-SVM together with Valley-SVM present more clearly high prediction accuracies for the case with label noise in comparison with ENSVM and PinSVM. Because our designed CaEN loss enjoys both outlier insensitivity and resampling stability, the CaENSVM always achieves the highest average prediction accuracies for the cases with label noise. Therefore, the advantage of CaENSVM becomes more concrete, indicating its good robustness to label noise.From Table3, our proposed CaENSVM with Gaussian kernel slightly performs better than others according to the average prediction accuracies. For the raw datasets, the CaENSVM gives higher average prediction accuracies in more than half of the datasets. The PinSVM turns to perform well, only behind the CaENSVM. RampSVM also has a good performance, but it is clearly worse than PinSVM. The remaining three SVMs are competitive with each other. For the datasets with moderate and high label noise, the CaENSVM still has a weak superiority, followed by Rhinge-SVM and PinSVM. ENSVM and Valley-SVM seem to be the worst in most datasets. Compared with the predicting performance in the linear cases, the superiority of our proposed CaENSVM is not overwhelming. In our opinion, this situation is possibly caused by the Pegasos procedure, since the efficiency of Pegasos is reduced in the nonlinear cases [43]. We will consider designing efficient algorithms for nonlinear cases in future.


### 5.2.2. Comparisons by Statistical Test

In this part, we conduct a statistical test on the experimental results of UCI datasets to further demonstrate the advantage of the CaENSVM. Specifically, we utilize the famous Friedman test with the corresponding post hoc test [48] to study whether there is a statistically significant difference among CaENSVM and other five compared SVMs. Firstly, we calculate the average ranks of each method with respect to different kernels and ratios of label noise in Tables 2 and 3. The results are presented in Table 4.Table 2
The mean accuracy (Acc.) and standard deviation (sd) with linear kernel for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd(a)0%label noiseAbsenteeism0.946 ± 0.0140.946 ± 0.0180.946 ± 0.0120.946 ± 0.0510.954 ± 0.0120.973 ± 0.013Autism0.836 ± 0.0230.734 ± 0.0240.734 ± 0.0250.729 ± 0.0830.810 ± 0.0670.993 ± 0.013baDS0.968 ± 0.0120.912 ± 0.0290.982 ± 0.0150.972 ± 0.0150.977 ± 0.0110.987 ± 0.007Banknote0.983 ± 0.0130.936 ± 0.0140.983 ± 0.0110.975 ± 0.0180.974 ± 0.0060.992 ± 0.005Forestfire0.870 ± 0.0820.652 ± 0.0150.652 ± 0.0090.809 ± 0.0730.835 ± 0.0570.922 ± 0.020ILPD0.649 ± 0.0160.287 ± 0.0250.287 ± 0.0130.381 ± 0.6870.447 ± 0.2200.716 ± 0.012Knowledge0.931 ± 0.0400.967 ± 0.0310.959 ± 0.0320.939 ± 0.0430.718 ± 0.1520.967 ± 0.031Messidor0.678 ± 0.0440.481 ± 0.0250.542 ± 0.1190.687 ± 0.0370.675 ± 0.0230.642 ± 0.073Popfailure0.735 ± 0.3640.084 ± 0.0130.916 ± 0.0090.490 ± 0.3800.084 ± 0.0120.929 ± 0.025Seed0.907 ± 0.0410.500 ± 0.0220.500 ± 0.0100.800 ± 0.0410.686 ± 0.1500.929 ± 0.025wdbc0.894 ± 0.0270.372 ± 0.0360.372 ± 0.0240.857 ± 0.0520.897 ± 0.0180.715 ± 0.178Wine0.888 ± 0.0520.560 ± 0.0110.560 ± 0.0120.872 ± 0.0520.904 ± 0.0610.912 ± 0.052Amphibians0.671 ± 0.0440.692 ± 0.0820.681 ± 0.0880.698 ± 0.0550.687 ± 0.0450.698 ± 0.070Raisin0.864 ± 0.0300.864 ± 0.0380.863 ± 0.0340.864 ± 0.0310.834 ± 0.0410.872 ± 0.013Tripadvisor0.733 ± 0.0120.732 ± 0.0050.735 ± 0.0090.731 ± 0.0080.740 ± 0.0090.741 ± 0.008(b)15%label noiseAbsenteeism0.935 ± 0.0240.946 ± 0.0120.946 ± 0.0080.795 ± 0.3310.946 ± 0.0200.948 ± 0.010Autism0.806 ± 0.0350.734 ± 0.0230.734 ± 0.0020.794 ± 0.0680.796 ± 0.0880.813 ± 0.028baDS0.973 ± 0.0170.915 ± 0.0470.973 ± 0.0120.974 ± 0.0070.968 ± 0.0120.977 ± 0.010Banknote0.969 ± 0.0110.905 ± 0.0510.979 ± 0.0140.975 ± 0.0120.972 ± 0.0110.979 ± 0.011Forestfire0.848 ± 0.0950.652 ± 0.0150.652 ± 0.0230.739 ± 0.1410.800 ± 0.1400.852 ± 0.050ILPD0.402 ± 0.1580.287 ± 0.0080.287 ± 0.0250.640 ± 0.1750.593 ± 0.1820.654 ± 0.161Knowledge0.947 ± 0.0310.972 ± 0.0230.972 ± 0.0230.976 ± 0.0270.690 ± 0.1140.979 ± 0.029Messidor0.585 ± 0.0960.470 ± 0.0220.548 ± 0.1280.578 ± 0.0260.635 ± 0.0520.636 ± 0.033Popfailure0.677 ± 0.3320.084 ± 0.0000.905 ± 0.0220.084 ± 0.0060.084 ± 0.0230.802 ± 0.099Seed0.900 ± 0.0590.500 ± 0.0250.500 ± 0.0190.807 ± 0.0570.857 ± 0.1240.824 ± 0.160wdbc0.810 ± 0.0380.372 ± 0.0310.372 ± 0.0150.780 ± 0.0420.832 ± 0.0450.864 ± 0.039Wine0.872 ± 0.0180.560 ± 0.0100.560 ± 0.0320.768 ± 0.1070.888 ± 0.0440.896 ± 0.036Amphibians0.660 ± 0.0240.665 ± 0.0530.681 ± 0.0480.665 ± 0.0310.638 ± 0.0650.692 ± 0.053Raisin0.860 ± 0.0310.855 ± 0.0250.870 ± 0.0260.862 ± 0.0260.856 ± 0.0190.875 ± 0.018Tripadvisor0.723 ± 0.0140.729 ± 0.0020.728 ± 0.0050.730 ± 0.0040.710 ± 0.0180.724 ± 0.004(c)25%label noiseAbsenteeism0.916 ± 0.0670.940 ± 0.0230.940 ± 0.0080.772 ± 0.3600.945 ± 0.0030.946 ± 0.008Autism0.746 ± 0.0500.734 ± 0.0100.763 ± 0.0640.816 ± 0.0920.779 ± 0.1360.780 ± 0.035baDS0.972 ± 0.0140.887 ± 0.0980.977 ± 0.0080.966 ± 0.0170.965 ± 0.0200.978 ± 0.013Banknote0.972 ± 0.0130.966 ± 0.0210.980 ± 0.0140.976 ± 0.0150.978 ± 0.0110.982 ± 0.011Forestfire0.791 ± 0.0480.652 ± 0.0250.652 ± 0.0120.661 ± 0.1420.783 ± 0.1020.800 ± 0.079ILPD0.421 ± 0.1860.287 ± 0.0110.346 ± 0.1320.515 ± 0.1790.657 ± 0.0440.546 ± 0.125Knowledge0.939 ± 0.0380.963 ± 0.0300.967 ± 0.0230.967 ± 0.0230.657 ± 0.1060.972 ± 0.030Messidor0.686 ± 0.0320.470 ± 0.0230.564 ± 0.0880.610 ± 0.0570.660 ± 0.0480.595 ± 0.101Popfailure0.673 ± 0.0380.084 ± 0.0340.914 ± 0.0200.084 ± 0.0250.090 ± 0.0130.693 ± 0.031Seed0.907 ± 0.0480.500 ± 0.0120.500 ± 0.0230.910 ± 0.1000.600 ± 0.0960.914 ± 0.074wdbc0.871 ± 0.0390.372 ± 0.0150.372 ± 0.0410.731 ± 0.0820.892 ± 0.0390.894 ± 0.047Wine0.852 ± 0.0660.560 ± 0.0200.560 ± 0.0260.664 ± 0.0610.856 ± 0.1080.860 ± 0.321Amphibians0.649 ± 0.0850.725 ± 0.0480.676 ± 0.0540.687 ± 0.0590.665 ± 0.0560.698 ± 0.052Raisin0.856 ± 0.0250.851 ± 0.0110.863 ± 0.0140.863 ± 0.0140.846 ± 0.0310.867 ± 0.006Tripadvisor0.723 ± 0.0310.731 ± 0.0070.736 ± 0.0150.736 ± 0.0130.732 ± 0.0080.737 ± 0.011The bold is the best one.Secondly, we need to obtain the Friedman statistic based on Table4. Let Dn and Ck be the total number of compared datasets and classifiers, respectively. By Tables 2 and 3, we have Dn=15 and Ck=6 for each type of kernel and ratio of label noise. Then, we consider the following F statistic:(61)FF=Dn−1χF2DnCk−1−χF2,where χF2 is the raw Friedman statistic defined as(62)χF2=12DnCkCk+1∑i=1CkRi2−CkCk+124,where Ri is the average rank of the i-th classifier for each type of kernel and ratio of label noise. The obtained FF statistic obeys F distribution with Ck−1 and Ck−1Dn−1 degrees of freedom. When the significant level is set to 0.1, we have F0.15,70=1.93. According to Table 4, we obtain that the values of FF are 8.57, 8.34, and 8.87 with respect to different ratios of label noise for linear cases, 19.27, 21.21, and 12.69 with respect to different ratios of label noise for nonlinear cases. All values of FF statistic are larger than the critical value 1.93, which means that there is indeed a statistical difference among those compared six SVMs.Thirdly, we apply the Nemenyi test for post hoc test to further distinguish the detailed differences of six classifiers. The critical domainCD of the difference of the average ranks of two SVMs given by Nemenyi is defined as(63)CD=q0.1CkCk+16Dn=1.769,where q0.1=2.589. If the absolute difference of two SVMs is larger than CD, it means that they perform statistically differently. Otherwise, they have no statistical differences with each other. Figure 7 shows the comparison of the average ranks of each SVM for different type of kernels and ratios of label noise.Figure 7
Comparison by the Nemenyi test under 0.1 significant level. (a)–(c) correspond to the linear cases with 0%, 15%, and 25% label noises, while (d)–(f) correspond to the nonlinear cases with 0%, 15%, and 25% label noises, respectively. (a) Case 1. (b) Case 2. (c) Case 3. (d) Case 4. (e) Case 5. (f) Case 6.
(a)(b)(c)(d)(e)(f)According to Figure7, there is no significant difference between CaENSVM and ENSVM with linear kernel. However, by Figures 7(a)–7(c), the difference between CaENSVM and ENSVM becomes larger as the ratio of label noise increases, which means the proposed CaENSVM is still better than others. Rhinge-SVM, RampSVM, and PinSVM always have no significant differences for linear cases, they all are significantly worse than CaENSVM. Valley-SVM shows good robustness for a high ratio of label noise, but there is still a gap between it and CaENSVM. For the cases with Gaussian kernel, our CaENSVM always presents slightly better performances than others, though the differences among it and Rhinge-SVM and PinSVM are not significant. The performances of Valley-SVM and ENSVM are significantly worse than those of other four methods, especially for high ratio of label noise. In short, our CaENSVM enjoys good performance in the statistical viewpoint.


## 5.2.1. Experimental Settings and Results

We select twelve UCI datasets to further demonstrate the advantages of our proposed CaENSVM. The detailed information of the chosen datasets is listed in Table1. To investigate the label noise insensitivity, we artificially add 15% and 25% label noises to the raw datasets, i.e., we randomly select 15% and 25% training samples and exchange their labels. The experimental results with linear and Gaussian kernels based on five-fold cross-validation criterion are shown in Tables 2 and 3, respectively.Algorithm 1: A general framework of the DC algorithm for (15).
Require: k=0; Θ0, the initial value of Θ.Ensure: optimal solution of (15).(1)
repeat(2)
Θk+1=argminΘgΘ−h′ΘkTΘ−Θk, h′ is the derivative of h with respect to Θ.(3)
until convergence.(4)
returnΘk+1.Algorithm 2: Pegasos-based DC procedure for CaENSVM.
Require:T1, T2, eps, w0, xiT,yii=1n, c, τ, θ, and s.Ensure: optimal solution of (11).(1)
Sett=0, v0=w0.(2)
whilet≤T2do(3)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(4)
Compute∇Fvt by (21) and (22).(5)
Setηt=c/t.(6)
Setvt+1←vt−ηt∇Fvt.(7)
end while(8)
Setw1=vt+1.(9)
Setk=0.(10)
while (k≤T1 & wk+1−wk≥eps) do(11)
Setk←k+1.(12)
Sett=0, v0=wk.(13)
whilet≤T2do(14)
ChooseAt⊂1,2,⋯,n, where At=m, uniformly at random.(15)
Compute∇Fvt by (21) and (22).(16)
Setηt=c/t.(17)
Setvt+1←vt−ηt∇Fvt.(18)
end while(19)
Setwk+1=vt+1.(20)
end while(21)
returnwk+1.Table 1
The information of UCI datasets.
IDDataset#Obs#Fea1Absenteeism740192Autism702153baDS137254Banknote137145Forestfire121106ILPD579107Knowledge25158Messidor1151199Popfailure5401510Seed210811wdbc5693412Wine1301313Amphibians1892114Raisin900715Tripadvisor9809From Table2 our proposed CaENSVM with linear kernel outperforms others in most cases, according to the average prediction accuracies. For the case without label noise, our CaENSVM performs slightly better than the ENSVM, followed by Valley-SVM. Rhinge-SVM and RampSVM have competitive performances, while PinSVM seems to be the worst. Along with the ratio of label noise increasing, the performances of all SVMs seem roughly reduced. Specifically, the prediction accuracy of the ENSVM is most affected by label noise, since the elastic net loss lacks robustness. Though PinSVM performs little stably due to the asymmetric pinball loss, its prediction accuracy is often the most unsatisfactory and competitive with the ENSVM for the case with moderate and high label noise. Due to the robustness of ramp, Rhinge, and valley losses, RampSVM and Rhinge-SVM together with Valley-SVM present more clearly high prediction accuracies for the case with label noise in comparison with ENSVM and PinSVM. Because our designed CaEN loss enjoys both outlier insensitivity and resampling stability, the CaENSVM always achieves the highest average prediction accuracies for the cases with label noise. Therefore, the advantage of CaENSVM becomes more concrete, indicating its good robustness to label noise.From Table3, our proposed CaENSVM with Gaussian kernel slightly performs better than others according to the average prediction accuracies. For the raw datasets, the CaENSVM gives higher average prediction accuracies in more than half of the datasets. The PinSVM turns to perform well, only behind the CaENSVM. RampSVM also has a good performance, but it is clearly worse than PinSVM. The remaining three SVMs are competitive with each other. For the datasets with moderate and high label noise, the CaENSVM still has a weak superiority, followed by Rhinge-SVM and PinSVM. ENSVM and Valley-SVM seem to be the worst in most datasets. Compared with the predicting performance in the linear cases, the superiority of our proposed CaENSVM is not overwhelming. In our opinion, this situation is possibly caused by the Pegasos procedure, since the efficiency of Pegasos is reduced in the nonlinear cases [43]. We will consider designing efficient algorithms for nonlinear cases in future.


## 5.2.2. Comparisons by Statistical Test

In this part, we conduct a statistical test on the experimental results of UCI datasets to further demonstrate the advantage of the CaENSVM. Specifically, we utilize the famous Friedman test with the corresponding post hoc test [48] to study whether there is a statistically significant difference among CaENSVM and other five compared SVMs. Firstly, we calculate the average ranks of each method with respect to different kernels and ratios of label noise in Tables 2 and 3. The results are presented in Table 4.Table 2
The mean accuracy (Acc.) and standard deviation (sd) with linear kernel for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd(a)0%label noiseAbsenteeism0.946 ± 0.0140.946 ± 0.0180.946 ± 0.0120.946 ± 0.0510.954 ± 0.0120.973 ± 0.013Autism0.836 ± 0.0230.734 ± 0.0240.734 ± 0.0250.729 ± 0.0830.810 ± 0.0670.993 ± 0.013baDS0.968 ± 0.0120.912 ± 0.0290.982 ± 0.0150.972 ± 0.0150.977 ± 0.0110.987 ± 0.007Banknote0.983 ± 0.0130.936 ± 0.0140.983 ± 0.0110.975 ± 0.0180.974 ± 0.0060.992 ± 0.005Forestfire0.870 ± 0.0820.652 ± 0.0150.652 ± 0.0090.809 ± 0.0730.835 ± 0.0570.922 ± 0.020ILPD0.649 ± 0.0160.287 ± 0.0250.287 ± 0.0130.381 ± 0.6870.447 ± 0.2200.716 ± 0.012Knowledge0.931 ± 0.0400.967 ± 0.0310.959 ± 0.0320.939 ± 0.0430.718 ± 0.1520.967 ± 0.031Messidor0.678 ± 0.0440.481 ± 0.0250.542 ± 0.1190.687 ± 0.0370.675 ± 0.0230.642 ± 0.073Popfailure0.735 ± 0.3640.084 ± 0.0130.916 ± 0.0090.490 ± 0.3800.084 ± 0.0120.929 ± 0.025Seed0.907 ± 0.0410.500 ± 0.0220.500 ± 0.0100.800 ± 0.0410.686 ± 0.1500.929 ± 0.025wdbc0.894 ± 0.0270.372 ± 0.0360.372 ± 0.0240.857 ± 0.0520.897 ± 0.0180.715 ± 0.178Wine0.888 ± 0.0520.560 ± 0.0110.560 ± 0.0120.872 ± 0.0520.904 ± 0.0610.912 ± 0.052Amphibians0.671 ± 0.0440.692 ± 0.0820.681 ± 0.0880.698 ± 0.0550.687 ± 0.0450.698 ± 0.070Raisin0.864 ± 0.0300.864 ± 0.0380.863 ± 0.0340.864 ± 0.0310.834 ± 0.0410.872 ± 0.013Tripadvisor0.733 ± 0.0120.732 ± 0.0050.735 ± 0.0090.731 ± 0.0080.740 ± 0.0090.741 ± 0.008(b)15%label noiseAbsenteeism0.935 ± 0.0240.946 ± 0.0120.946 ± 0.0080.795 ± 0.3310.946 ± 0.0200.948 ± 0.010Autism0.806 ± 0.0350.734 ± 0.0230.734 ± 0.0020.794 ± 0.0680.796 ± 0.0880.813 ± 0.028baDS0.973 ± 0.0170.915 ± 0.0470.973 ± 0.0120.974 ± 0.0070.968 ± 0.0120.977 ± 0.010Banknote0.969 ± 0.0110.905 ± 0.0510.979 ± 0.0140.975 ± 0.0120.972 ± 0.0110.979 ± 0.011Forestfire0.848 ± 0.0950.652 ± 0.0150.652 ± 0.0230.739 ± 0.1410.800 ± 0.1400.852 ± 0.050ILPD0.402 ± 0.1580.287 ± 0.0080.287 ± 0.0250.640 ± 0.1750.593 ± 0.1820.654 ± 0.161Knowledge0.947 ± 0.0310.972 ± 0.0230.972 ± 0.0230.976 ± 0.0270.690 ± 0.1140.979 ± 0.029Messidor0.585 ± 0.0960.470 ± 0.0220.548 ± 0.1280.578 ± 0.0260.635 ± 0.0520.636 ± 0.033Popfailure0.677 ± 0.3320.084 ± 0.0000.905 ± 0.0220.084 ± 0.0060.084 ± 0.0230.802 ± 0.099Seed0.900 ± 0.0590.500 ± 0.0250.500 ± 0.0190.807 ± 0.0570.857 ± 0.1240.824 ± 0.160wdbc0.810 ± 0.0380.372 ± 0.0310.372 ± 0.0150.780 ± 0.0420.832 ± 0.0450.864 ± 0.039Wine0.872 ± 0.0180.560 ± 0.0100.560 ± 0.0320.768 ± 0.1070.888 ± 0.0440.896 ± 0.036Amphibians0.660 ± 0.0240.665 ± 0.0530.681 ± 0.0480.665 ± 0.0310.638 ± 0.0650.692 ± 0.053Raisin0.860 ± 0.0310.855 ± 0.0250.870 ± 0.0260.862 ± 0.0260.856 ± 0.0190.875 ± 0.018Tripadvisor0.723 ± 0.0140.729 ± 0.0020.728 ± 0.0050.730 ± 0.0040.710 ± 0.0180.724 ± 0.004(c)25%label noiseAbsenteeism0.916 ± 0.0670.940 ± 0.0230.940 ± 0.0080.772 ± 0.3600.945 ± 0.0030.946 ± 0.008Autism0.746 ± 0.0500.734 ± 0.0100.763 ± 0.0640.816 ± 0.0920.779 ± 0.1360.780 ± 0.035baDS0.972 ± 0.0140.887 ± 0.0980.977 ± 0.0080.966 ± 0.0170.965 ± 0.0200.978 ± 0.013Banknote0.972 ± 0.0130.966 ± 0.0210.980 ± 0.0140.976 ± 0.0150.978 ± 0.0110.982 ± 0.011Forestfire0.791 ± 0.0480.652 ± 0.0250.652 ± 0.0120.661 ± 0.1420.783 ± 0.1020.800 ± 0.079ILPD0.421 ± 0.1860.287 ± 0.0110.346 ± 0.1320.515 ± 0.1790.657 ± 0.0440.546 ± 0.125Knowledge0.939 ± 0.0380.963 ± 0.0300.967 ± 0.0230.967 ± 0.0230.657 ± 0.1060.972 ± 0.030Messidor0.686 ± 0.0320.470 ± 0.0230.564 ± 0.0880.610 ± 0.0570.660 ± 0.0480.595 ± 0.101Popfailure0.673 ± 0.0380.084 ± 0.0340.914 ± 0.0200.084 ± 0.0250.090 ± 0.0130.693 ± 0.031Seed0.907 ± 0.0480.500 ± 0.0120.500 ± 0.0230.910 ± 0.1000.600 ± 0.0960.914 ± 0.074wdbc0.871 ± 0.0390.372 ± 0.0150.372 ± 0.0410.731 ± 0.0820.892 ± 0.0390.894 ± 0.047Wine0.852 ± 0.0660.560 ± 0.0200.560 ± 0.0260.664 ± 0.0610.856 ± 0.1080.860 ± 0.321Amphibians0.649 ± 0.0850.725 ± 0.0480.676 ± 0.0540.687 ± 0.0590.665 ± 0.0560.698 ± 0.052Raisin0.856 ± 0.0250.851 ± 0.0110.863 ± 0.0140.863 ± 0.0140.846 ± 0.0310.867 ± 0.006Tripadvisor0.723 ± 0.0310.731 ± 0.0070.736 ± 0.0150.736 ± 0.0130.732 ± 0.0080.737 ± 0.011The bold is the best one.Secondly, we need to obtain the Friedman statistic based on Table4. Let Dn and Ck be the total number of compared datasets and classifiers, respectively. By Tables 2 and 3, we have Dn=15 and Ck=6 for each type of kernel and ratio of label noise. Then, we consider the following F statistic:(61)FF=Dn−1χF2DnCk−1−χF2,where χF2 is the raw Friedman statistic defined as(62)χF2=12DnCkCk+1∑i=1CkRi2−CkCk+124,where Ri is the average rank of the i-th classifier for each type of kernel and ratio of label noise. The obtained FF statistic obeys F distribution with Ck−1 and Ck−1Dn−1 degrees of freedom. When the significant level is set to 0.1, we have F0.15,70=1.93. According to Table 4, we obtain that the values of FF are 8.57, 8.34, and 8.87 with respect to different ratios of label noise for linear cases, 19.27, 21.21, and 12.69 with respect to different ratios of label noise for nonlinear cases. All values of FF statistic are larger than the critical value 1.93, which means that there is indeed a statistical difference among those compared six SVMs.Thirdly, we apply the Nemenyi test for post hoc test to further distinguish the detailed differences of six classifiers. The critical domainCD of the difference of the average ranks of two SVMs given by Nemenyi is defined as(63)CD=q0.1CkCk+16Dn=1.769,where q0.1=2.589. If the absolute difference of two SVMs is larger than CD, it means that they perform statistically differently. Otherwise, they have no statistical differences with each other. Figure 7 shows the comparison of the average ranks of each SVM for different type of kernels and ratios of label noise.Figure 7
Comparison by the Nemenyi test under 0.1 significant level. (a)–(c) correspond to the linear cases with 0%, 15%, and 25% label noises, while (d)–(f) correspond to the nonlinear cases with 0%, 15%, and 25% label noises, respectively. (a) Case 1. (b) Case 2. (c) Case 3. (d) Case 4. (e) Case 5. (f) Case 6.
(a)(b)(c)(d)(e)(f)According to Figure7, there is no significant difference between CaENSVM and ENSVM with linear kernel. However, by Figures 7(a)–7(c), the difference between CaENSVM and ENSVM becomes larger as the ratio of label noise increases, which means the proposed CaENSVM is still better than others. Rhinge-SVM, RampSVM, and PinSVM always have no significant differences for linear cases, they all are significantly worse than CaENSVM. Valley-SVM shows good robustness for a high ratio of label noise, but there is still a gap between it and CaENSVM. For the cases with Gaussian kernel, our CaENSVM always presents slightly better performances than others, though the differences among it and Rhinge-SVM and PinSVM are not significant. The performances of Valley-SVM and ENSVM are significantly worse than those of other four methods, especially for high ratio of label noise. In short, our CaENSVM enjoys good performance in the statistical viewpoint.


## 5.3. Handwritten Digit Recognition

In this subsection, we apply CaENSVM to solve a real problem, i.e., handwritten digit recognition. The test dataset is the PMU-UD dataset from the study of Alghazo et al. [49], containing handwritten Urdu/Arabic numerals from 0 to 9. Each handwritten number is standardized as a 120×80 image. Figure 8 shows four selected images for every handwritten number. We choose the datasets corresponding to two different handwritten numbers each time to conduct binary classification. The information of all considered datasets is listed in Table 5.Figure 8
Selected images from the PMU-UD dataset.Table 3
The mean accuracy (Acc.) and standard deviation (sd) with Gaussian kernel for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVMAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sdAcc. ± sd(a)0%label noiseAbsenteeism0.946 ± 0.0530.955 ± 0.0030.947 ± 0.0100.946 ± 0.1200.946 ± 0.0220.955 ± 0.003Autism0.921 ± 0.1060.978 ± 0.0120.997 ± 0.0040.978 ± 0.0890.903 ± 0.0810.999 ± 0.003baDS0.987 ± 0.0101.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.944 ± 0.051.000 ± 0.000Banknote0.980 ± 0.0100.996 ± 0.0080.993 ± 0.0100.944 ± 0.0180.957 ± 0.0240.998 ± 0.005Forestfire0.861 ± 0.0840.913 ± 0.0620.870 ± 0.0440.635 ± 0.1460.722 ± 0.0900.931 ± 0.039ILPD0.711 ± 0.0130.716 ± 0.0220.621 ± 0.1870.714 ± 0.2220.374 ± 0.1950.727 ± 0.018Knowledge0.882 ± 0.0700.951 ± 0.0230.963 ± 0.0270.926 ± 0.1480.718 ± 0.1250.968 ± 0.028Messidor0.585 ± 0.0330.649 ± 0.0380.603 ± 0.0170.529 ± 0.0430.497 ± 0.0400.619 ± 0.054Popfailure0.563 ± 0.4390.744 ± 0.3690.905 ± 0.0180.497 ± 0.2910.084 ± 0.0340.903 ± 0.022Seed0.936 ± 0.0460.971 ± 0.0300.936 ± 0.0460.949 ± 0.0360.943 ± 0.0540.971 ± 0.030wdbc0.727 ± 0.0790.917 ± 0.0180.680 ± 0.0400.907 ± 0.1140.372 ± 0.0230.908 ± 0.015Wine0.728 ± 0.1070.896 ± 0.0460.648 ± 0.0660.864 ± 0.0460.568 ± 0.0180.888 ± 0.033Amphibians0.633 ± 0.0930.789 ± 0.0350.698 ± 0.0670.730 ± 0.0510.568 ± 0.0000.692 ± 0.059Raisin0.865 ± 0.0230.860 ± 0.0280.852 ± 0.0160.860 ± 0.0160.828 ± 0.0430.868 ± 0.012Tripadvisor0.725 ± 0.0070.733 ± 0.0160.722 ± 0.0140.742 ± 0.0090.728 ± 0.0000.734 ± 0.014(b)15%label noiseAbsenteeism0.946 ± 0.0130.946 ± 0.0140.947 ± 0.0030.943 ± 0.0150.946 ± 0.0080.970 ± 0.053Autism0.873 ± 0.1270.951 ± 0.0250.996 ± 0.0040.964 ± 0.0080.846 ± 0.1050.997 ± 0.065baDS0.945 ± 0.0320.946 ± 0.0040.941 ± 0.0130.949 ± 0.0030.940 ± 0.0350.951 ± 0.055Banknote0.986 ± 0.0100.993 ± 0.0071.000 ± 0.0000.966 ± 0.0080.879 ± 0.1481.000 ± 0.000Forestfire0.818 ± 0.0950.913 ± 0.0690.905 ± 0.0780.910 ± 0.0720.739 ± 0.0870.912 ± 0.082ILPD0.556 ± 0.2310.717 ± 0.0260.696 ± 0.0490.701 ± 0.0290.287 ± 0.0210.698 ± 0.076Knowledge0.857 ± 0.0970.929 ± 0.0290.917 ± 0.0310.921 ± 0.0340.710 ± 0.1130.937 ± 0.024Messidor0.586 ± 0.0310.629 ± 0.0390.625 ± 0.0280.651 ± 0.0200.478 ± 0.0170.634 ± 0.062Popfailure0.222 ± 0.3090.731 ± 0.3640.725 ± 0.3600.392 ± 0.4230.084 ± 0.0230.735 ± 0.192Seed0.936 ± 0.0390.914 ± 0.0740.936 ± 0.0390.936 ± 0.0300.836 ± 0.1880.929 ± 0.061wdbc0.703 ± 0.0790.436 ± 0.0530.626 ± 0.1430.864 ± 0.0420.372 ± 0.0230.890 ± 0.066Wine0.648 ± 0.0870.812 ± 0.0720.648 ± 0.0870.820 ± 0.0630.560 ± 0.0100.844 ± 0.128Amphibians0.611 ± 0.0530.660 ± 0.0620.649 ± 0.0990.708 ± 0.0840.579 ± 0.0240.714 ± 0.056Raisin0.693 ± 0.1770.860 ± 0.0210.864 ± 0.0200.855 ± 0.0330.840 ± 0.0140.867 ± 0.026Tripadvisor0.726 ± 0.0110.718 ± 0.0190.691 ± 0.0390.726 ± 0.0220.656 ± 0.1020.728 ± 0.010(c)25%label noiseAbsenteeism0.945 ± 0.0030.936 ± 0.0130.943 ± 0.0080.945 ± 0.0080.946 ± 0.0230.946 ± 0.015Autism0.883 ± 0.0850.967 ± 0.0140.969 ± 0.0060.973 ± 0.0210.796 ± 0.0850.974 ± 0.012baDS0.933 ± 0.0650.972 ± 0.0040.971 ± 0.0100.975 ± 0.0060.956 ± 0.0220.978 ± 0.082Banknote0.987 ± 0.0040.990 ± 0.0080.990 ± 0.0020.995 ± 0.0060.953 ± 0.0160.993 ± 0.042Forestfire0.809 ± 0.1090.870 ± 0.0440.852 ± 0.0390.852 ± 0.0580.817 ± 0.1080.867 ± 0.021ILPD0.602 ± 0.1780.697 ± 0.0070.689 ± 0.0150.709 ± 0.0480.287 ± 0.0210.712 ± 0.091Knowledge0.841 ± 0.0600.972 ± 0.0230.943 ± 0.0340.980 ± 0.0140.669 ± 0.1700.814 ± 0.038Messidor0.537 ± 0.0530.617 ± 0.0270.583 ± 0.0400.604 ± 0.0270.502 ± 0.0430.628 ± 0.057Popfailure0.366 ± 0.3860.249 ± 0.3680.733 ± 0.3630.411 ± 0.4480.084 ± 0.0410.784 ± 0.098Seed0.914 ± 0.0650.936 ± 0.0640.929 ± 0.0670.929 ± 0.0440.914 ± 0.0540.871 ± 0.054wdbc0.681 ± 0.0540.864 ± 0.0660.492 ± 0.1710.844 ± 0.0550.372 ± 0.0230.774 ± 0.032Wine0.672 ± 0.0720.716 ± 0.0460.672 ± 0.0520.732 ± 0.1000.560 ± 0.0080.748 ± 0.059Amphibians0.606 ± 0.1220.714 ± 0.0560.735 ± 0.0480.714 ± 0.0410.563 ± 0.0120.714 ± 0.073Raisin0.819 ± 0.0480.852 ± 0.0210.853 ± 0.0170.858 ± 0.0150.747 ± 0.1480.860 ± 0.023Tripadvisor0.673 ± 0.1210.702 ± 0.0410.701 ± 0.0570.708 ± 0.0220.721 ± 0.0080.722 ± 0.008The bold is the best one.For a fair evaluation, we also compare our CaENSVM with ENSVM, PinSVM, RampSVM, Rhinge-SVM, and Valley-SVM. The average prediction accuracies with linear and Gaussian kernels based on the five-fold cross-validation criterion are presented in Tables6 and 7, respectively. According to Tables 6 and 7, our CaENSVM achieves the highest prediction accuracies in more than half of the cases, indicating its excellent performance in handwritten digit recognition problems.Table 4
Average ranks of each SVM with respect to different kernels and ratios of label noise for UCI datasets.
ENSVMPinSVMRampSVMRhinge-SVMValley-SVMCaENSVM(a) Linear kernelAvg. rank 0%3.2004.8334.2673.7333.5001.467Avg. rank 15%3.5674.9333.8333.4333.8001.433Avg. rank 25%3.8675.0673.7333.4003.4671.467(b) Gaussian kernelAvg. rank 0%4.3672.1003.6003.7005.5331.700Avg. rank 15%4.3333.0673.4002.9005.8001.500Avg. rank 25%4.8333.0333.5332.3675.2002.033Table 5
The information of the PMU-UD dataset.
IDDataset#Obs#Fea12-vs-3937960022-vs-41003960032-vs-61007960042-vs-71019960052-vs-81003960063-vs-4986960076-vs-71072960086-vs-81056960097-vs-810689600


## 6. Conclusion

In this paper, we have proposed a novel robust support vector classifier (CaENSVM) with a capped elastic net loss function. Theoretical analysis is conducted to thoroughly demonstrate the properties of CaENSVM, including noise insensitivity, Bayes rule, and the generalization error bound based on the Rademacher complexity. It is worth noting that we use the influence function to explain well the robustness of CaENSVM. Though the constructed CaEN loss is nonconvex, the implemented Pegasos-based DC algorithm can efficiently solve the CaENSVM optimization problem. The results of numerical studies indicate the following: (1) our CaENSVM is robust to outliers and performs better than many similar state-of-the-art SVMs according to prediction accuracy. The superiority of the CaENSVM is also supported by the statistical test. (2) The performance of the CaENSVM for nonlinear cases is not overwhelming like the performance for linear cases in comparison with other methods, indicating that there may be a room for improving the efficiency of the algorithm. In fact, though Pegasos algorithm is scalable, the performance with nonlinear kernel is little unsatisfactory. Further work will focus on designing a more stable and efficient algorithm to achieve higher prediction accuracy, especially for nonlinear cases. Note that theR code of the proposed CaENSVM is developed by the authors and it is available at https://github.com/jiandan94/CaENSVM.



---

*Source: 2201330-2023-02-21.xml*