# Smooth Diagonal Weighted Newton Support Vector Machine

Authors: Jinjin Liang; De Wu
Journal: Mathematical Problems in Engineering
 (2013)

---

## Abstract

Based on diagonal weighted support vector machine, a smooth model with Newton algorithm is proposed and is called SDWNSVM for short. SDWNSVM introduces the entropy function to approximate the plus function of the slack in the diagonal weighted SVM and is thus different from traditional SSVM that treats a reformulation problem. SDWNSVM utilizes the dual technique to rewrite the objection function by the connotative relation between the primal and dual program, which induces an exact smooth program and differs from traditional SSVM that uses Lagrangian multipliers to roughly substitute for the hyperplane weight. SDWNSVM proves the equivalence between the obtained model and the original one and proposes Newton algorithm to figure out the optimal solution. Numerical experiments on UCI data demonstrate that SDWNSVM has higher accuracies and less iteration than existing methods.


---

## Body

## 1. Introduction

Based on statistical learning theory and dual programming, support vector machine (SVM) has been developed by Vapnik as an efficient method for small-sample data [1–3] and has gained wide application in classification and regression area. When the sample points are completely separable, the hard margin classifiers are introduced which can separate the positive class from the negative class with the maximal margin. When the sample points are nonseparable, the soft margin classifiers are introduced, which allow erroneousness of the misclassified points. Both 1-norm and 2-norm soft margin SVM are proposed, while the former is called box constrained SVM and the latter is called diagonal weighted SVM.Denote the nonnegative constraints in the plus function form for SVM error Lee and Mangasarian propose smooth support vector machine (SSVM) [4, 5], which has strong mathematical properties, such as strong convexity and infinitely often differentiability. SSVM applied smoothing techniques to one reformulation of the 2-norm soft margin SVM by appending an additional bias term. The approach was originally proposed to induce strong convexity and was later used in [6–8]; this was equivalent to adding a constant feature to the training data and finding a separating hyperplane passing through the origin, which had been proved to have little or no effect on the solutions to the original program. Based on SSVM, many scholars research from various aspects and presented diverse kinds of smooth models [9–20]. Some scholars proposed polynomial function based smooth approach [12–15], quarter penalty function based smooth approach [16], or rational function based smooth approach [10] and some scholars generalize to the regression or forecasting area [11, 17]. Among these methods, the widely used models are the fourth polynomial function based smooth approach called FPSSVM; however, the accuracies are limited. Wu and wang proposed a piecewise-smooth support vector machine PWSSVM for classification [9], which uses a twice continuously differentiable piecewise polynomial function to approximate the plus function and get satisfying results. A detailed study shows that the above smooth models have two drawbacks. (1) The 2-norm of the weight of the separating hyperplane is substituted for that of the Lagrangian multipliers vector. (2) The equivalences of the obtained programs and the original program are proved in none of these methods.A natural problem exists whether an exact model can be constructed for SVM and whether the proof can be given with regard to the equivalence between the obtained program and the original one. This paper investigates a smooth diagonal weighted newton support vector machine (SDWNSVM), which applies entropy function to the diagonal weighted support vector machine and proves the equivalence between the transformed program and the original one, by making use of the connotative relation between the primal and dual program. We begin from the linear space to obtain the unconstrained smooth model by directly adopting the entropy penalty function. As for the kernel space, we use the dual technique to rewrite the objection. Upon obtaining the smooth models, Newton algorithm is utilized to figure out the solution to SDWNSVM. Numerical experiments are carried out to demonstrate the effectiveness and superiority.This paper is organized as follows. Section2 derives the exact smooth model in the linear space. Section 3 proposes the two smooth models in the kernel space with the key transformations illustrated. Section 4 proposes Newton algorithm to determine the solutions to SDWNSVM. Numerical experiments are given in Section 5 to demonstrate the performances. The conclusions are made in the final section.


## 2. Linear Smooth Model

Given is the training setT={(x1,y1),(x2,y2),…,(xl,yl), in which xi∈Rn and yi∈{1,-1}. The training of standard diagonal weighted SVM equals the following program:
(1)min12wTw+C2(∑i=1lξi2),s.t.yi(wTxi+b)≥1-ξi,i=1,…,l,ξi≥0.Herew∈Rn is the normal to the separating hyperplane, b∈R is the bias, and ξ∈Rl is the misclassified error. The linear separating hyperplane is wTx+b=0, with x∈Rn as the input point.Denote the training set in the matrix form byAl×n=[x1,x2,…,xi,…,xl]T, and denote the labels yi∈{1,-1}(i=1,…,l) by the diagonal matrix Dl×l with ones or minus ones along its diagonal. Having these notations, we can rewrite (1) as
(2)min12wTw+C2ξTξ,s.t.D(Aw+be)≥e-ξ,ξ≥0.Forr∈R, we define the plus function as r+=max(r,0) and denote the entropy function with smooth parameter β>0(3)Pβ(r)=r+β-1ln(1+exp(-βr)).Lemma 1.
P
β
(
r
) is a strict convex function.Proof.
Figuring out the first order derivative forPβ(r) with respect to variable r gives the following equation, which guarantees that Lemma 1 holds:
(4)Pβ′(r)=1+-βexp(-βr)β(1+exp(-βr))>0.Lemma 2.
For anyr∈R and |r|<ρ:Pβ(r)2-(r+)2≤(ln2/β)2+(2ρ/β)ln2 holds, where Pβ(r) is defined as in (3) with smooth parameter β>0.Proof.
For0<r<ρ, we have
(5)Pβ(r)2-(r+)2=1β2ln2(1+ε-βr)+2rβln(1+ε-βr)≤(ln2β)2+2ρβln2.
For-ρ<r<0, Pβ(r) is a monotonically increasing function, and
(6)Pβ(r)2-(r+)2=Pβ(r)2≤Pβ(0)2=(ln2β)2.
ThusPβ(r)2-(r+)2≤(ln2/β)2+(2ρ/β)ln2 holds, which completes the proof.Corollary 3.
The entropy penalty function converges to the plus function; that is,limβ→∞Pβ(r)=r+=max(r,0).
By figuring out the limit ofPβ(r) as β approaches infinity, it is easy to show that Corollary 3 holds.Corollary3 guarantees that the entropy function can be used to approximate the plus function. It should be pointed out that, if we generalize r∈R to r=(r1,r2,r3,…,rn)∈Rn, the element ri∈R will still satisfy the corresponding results of Lemmas 1 and 2 and Corollary 3, and Lemma 2 becomes Lemma 4.Lemma 4.
For any vectorr=(r1,r2,r3,…,rn)∈Rn with |max1≤i≤n(ri)|<ρ, the inequality |||Pβ(r)||2-||r+||2|≤(ln2/β)2+(2ρ/β)ln2 holds, where Pβ(r) is defined as in (3) with parameter β>0.
Write the slack vector in the plus function form:(7)ξ=max(e-D(Aw+be),0)=(e-D(Aw+be))+.
Then one converts the diagonal weighted SVM into the equivalent unconstrained formulation:(8)minFβ(wb)=12wTw+C2∥Pβ(e-D(Aw+be))∥2.
For any vectorγ=(γ1,γ2,γ3,…,γn)∈Rn, the diagonal matrix is defined as follows with the elements of γ along the diagonal:
(9)Diag(γ)=[γ1⋱γn],
Now one figures out the gradient vector∇Fβ(wb) and the Hessian matrix ∇2Fβ(wb). For the sake of simplicity, five new vectors are introduced; namely, t=e-D(Aw-eγ),v=exp(-βt),M=(I+βDiag(v))·Diag(Pβ(t)),Q=DA, and Y=De:
(10)∇Fβ(wb)=[w0]+C·[-QT-YT]Diag(Pβ(t))Diag(e+v)-1·e,(11)∇2Fβ(wb)=[I000]+C·[-QT-YT]Diag(e+v)-2M[-Q-Y].
Following from (10) and (11), one concludes that Fβ(wb) is convex and strict convex. Theorem 5 guarantees the existences of the global minima for the convex functions.Theorem 5.
SupposeS⊂Rn is a nonempty convex set and f:S→R is a convex function; then the local minima of fon S are also global minima.Proof.
Respectively, denote byx∈S and y∈S the local and global minimal point of the function f; then f(y)<f(x) and the following inequality holds for any real number λ∈[0,1]:
(12)f(λy+(1-λ)x)≤λf(y)+(1-λ)f(x)<f(x).
It is noted thatlimλ→0+λy+(1-λ)x=x; for arbitrary ε>0 there exists λ0 sufficiently small such that
(13)λ0y+(1-λ0)x∈U0(x,ε).
HereU0(x,ε) is a small hollow region with x as the center and ε as the radius.
Take inequality (12) and (13) into account; we directly derive the following inequality, which implies that x is not a local minimal and thus contradicts the hypothesis:
(14)f(λ0y+(1-λ0)x)<f(x).
Now we prove Theorem 5.Theorem 6.
LetA=Al×n and b∈R. Define the two programs for the real valued functions F(wb) and Fβ(wb) in Rn+1 as follows:
(15)minF(wb)=12wTw+C2∥(e-D(Aw+be))+∥2,(16)minFβ(wb)=12wTw+C2∥Pβ(e-D(Aw+be))∥2,
where β>0 is the smooth parameter; then(1)
there, respectively, exist global minima ofminF(wb) and minFβ(wb);(2)
let(w¯b¯) and (w¯βb¯β) be the global minima of minFβ(wb) and minF(wb); the following upper bound inequality holds with ρ=max1≤i≤l|(e-D(Aw+be))i|:
(17)∥w¯β-w¯∥2+∥b¯β-b¯∥2≤l((ln2β)2+2ρβln2).
Thus,(w¯βb¯β) converges to (w¯b¯) as β approaches infinity.Proof.
(
1
) Firstly, we prove the existence of local minima.
Define the level set off as Lμ(f)={x∣f(x)≤μ} for any real number μ∈R.
SincePβ(x) dominates the plus function x+(x+≤Pβ(x)), the two level sets satisfy inequality Fβ(x)-F(x)≥0, and the two level sets satisfy the following for r∈R:
(18)LV(Fβ(x))⊆LV(F(x))⊆{x∣∥x∥22≤2r},
Hence, both LV(F(x)) and LV(Fβ(x)) are compact subsets in Rn+1.
It is easy to prove the convexity ofF(wb) and the strong convexity of Fβ(wb). Theorem 5 guarantees that there, respectively, exist global minima.
(
2
) Secondly, we establish the convergence.
Utilizing the first order optimality conditions of the convex functionF(wb) and the strict convex function Fβ(wb),
(19)F(w¯βb¯β)-F(w¯b¯)≥∇F(w¯b¯)(w¯β-w¯,b¯β-b¯)=0,Fβ(w¯b¯)-Fβ(w¯βb¯β)≥∇Fβ(w¯βb¯β)(w¯-w¯β,b¯-b¯β)+12∥(w¯-w¯β,b¯-b¯β)∥22=12∥(w¯-w¯β,b¯-b¯β)∥22.
Adding up the above two inequalities, we derive the following inequality:(20)12∥(w¯β-w¯,b¯β-b¯)∥22≤(Fβ(w¯b¯)-F(w¯b¯))-(Fβ(w¯βb¯β)-F(w¯βb¯β))≤Fβ(w¯b¯)-F(w¯b¯)=12∥Pβ(e-D(Aw¯+b¯e))∥22-12∥(e-D(Aw¯+b¯e))+∥22.
Define δ=max1≤i≤l|(e-D(Aw¯+b¯e))i|; we apply Lemma 2 to the right of and obtain the following inequality, which testifies the convergence:
(21)∥w¯β-w¯∥2+∥b¯β-b¯∥2≤l((ln2β)2+2ρβln2).


## 3. Kernel Smooth Model

In the kernel space, the following program is minimized, whereϕ(x) is the nonlinear map:
(22)min12wTw+C2(∑i=1lξi2),s.t.yi(wTϕ(xi)+b)≥1-ξi,i=1,…,l,ξi≥0.


### 3.1. Rough Kernel Smooth Model

Similar to SSVM, the 2-norm of the Lagrangian column vectorα=(α1,α2,…,αl)T∈Rl is used to replace that of the weight of the hyperplane. The matrix formation for kernel diagonal weighted SVM is as follows:
(23)min12αTα+C2ξTξ,s.t.D(K(A,AT)Dα+be)≥e-ξ,ξ≥0.The unconstrained program in the plus function form and the smooth form for the kernel 2-norm soft margin SVM is, respectively, illustrated as follows:(24)minF(αb)=12αTα+C2∥(e-D(K(A,AT)Dα+be))+∥2,(25)minFβ(αb)=12αTα+C2∥Pβ(e-D(K(A,AT)Dα+be))∥2.As for the gradient vector and Hessian matrix, we definet=e-D(K(A,AT)α+be) and Q=DK(A,AT)D; the gradient vector ∇Fβ(x) and the Hessian matrix ∇2Fβ(x) are computed according to (26) and (27), where v,M, and Y are the same as defined in Section 2:
(26)∇Fβ(αb)=[α0]+C[-QT-YT]Diag(Pβ(t))Diag(e+v)-1·e,(27)∇2Fβ(αb)=[I000]+C[-QT-YT]Diag(e+v)-2M[-Q-Y].Theorem 7.
LetA=Al×n and b∈R. For the two programs F(αb) and Fβ(αb) in (24) and (25),(1)
there, respectively, exist global minima ofminF(αb) and minFβ(αb).(2)
let(α¯b¯) and (α¯βb¯β) be the global minima of minFβ(αb) and minF(αb) one has the following inequality with ρ=max1≤i≤l|(e-D(K(A,AT)Dα+be))i|:
(28)∥α¯β-α¯∥2+∥b¯β-b¯∥2≤l((ln2β)2+2ρβln2).
Thus,(α¯βb¯β) converges to (α¯b¯) as β approaches infinity.Theorem7 can be proved similarly as Theorem 6; thus its detailed process is omitted.


### 3.2. Exact Kernel Smooth Model

Now we will propose the exact smooth model in the kernel space.Since the nonlinear mapϕ(x) is unknown, the connotative conditions between the primal program and the dual one are exploited.It is known that at the optimal solution(29)w=∑i=1lyiαiϕ(xi).Using this relation, we have the following two equations, in whichQij=yiyjK(xi,xj):
(30)yiwTϕ(xi)=∑j=1lyiyjαjϕ(xj)Tϕ(xi)=∑j=1lQijαj=(Qα)i,wTw=∑i=1lyiαiϕ(xi)Tyjαjϕ(xj)=∑i=1lαi(Qα)i=αTQα.Put the above two equations back into (22); we get the following program:
(31)min12αTQα+C2ξTξ,s.t.Qα+by≥e-ξ,ξ≥0.Though (31) is different from (22), the below theorem will prove that, for any optimal solution α* to (31), w*=∑i=1lyiαi*ϕ(xi) is also an optimal solution to (22).Theorem 8.
Let(α*,b*,ξ*) be the optimal solution to program (31); then (w*,b*,ξ*) is the optimal solution to program (22), where w* is defined as w*=∑i=1lyiαi*ϕ(xi).Proof.
Evidently, if(w,b,ξ) and (α,b,ξ) are the primal and dual optimal solutions, then (α,b,ξ) is feasible for (31).
For feasible(w,b,ξ) and (α,b,ξ), inequality (31) can be deduced from (29):
(32)12wTw+C2ξTξ=12αTQα+C2ξTξ≥12(α*)TQα*+C2(ξ*)Tξ*=12(w*)Tw*+C2(ξ*)Tξ*.
Thus,(w*,b*,ξ*) is the optimal solution to (31).
From the constraint in (31), we write the slack ξ in the form of plus function as
(33)ξ=(e-Qα-by)+=(e-D(K(A,AT)Dα+be))+.
Then we obtain the model in the plus function form and the model in the entropy function form in the kernel space:(34)minF(αb)=12αTDK(A,AT)Dα+C2∥(e-D(K(A,AT)Dα+be))+∥2(35)minFβ(αb)=12αTDK(A,AT)Dα+C2∥Pβ(e-D(K(A,AT)Dα+be))∥2.The gradient vector and the Hessian matrix of (35) can be computed as follows, where t and Q are the same as defined in Section 3.1, while v, M, and Y are the same as defined in Section 2:
(36)∇Fβ(αb)=[Qα0]+C·[-QT-YT]·Diag(Pβ(t))·Diag(e+v)-1·e,(37)∇2Fβ(αb)=[Q000]+C·[-QT-YT]·Diag(e+v)-2·(I+βDiag(v)·Diag(Pβ(t)))·[-Q-Y].Theorem 9.
LetQ=Ql×l(Q=DK(A,AT)D), α∈Rl, and b∈R. For the two programs F(αb) and Fβ(αb) in (36) and (37),(1)
there, respectively, exist global minima ofminF(αb) and minFβ(αb).(2)
let(α¯b¯) and (α¯βb¯β) be the global minima of minFβ(αb) and minF(αb), the following inequality holds with ρ=max1≤i≤l|(e-D(K(A,AT)Dα+be))i|;
(38)∥Q1/2(α¯-α¯β)∥22+∥b¯-b¯β∥2≤l((ln2β)2+2ρβln2).Corollary 10.
Asβ approaches infinity, the unique solution (α¯βb¯β) to minFβ(αb) converges to the unique solution (α¯b¯) to minF(αb); that is, limβ→∞(α¯βb¯β)=(α¯b¯).


## 3.1. Rough Kernel Smooth Model

Similar to SSVM, the 2-norm of the Lagrangian column vectorα=(α1,α2,…,αl)T∈Rl is used to replace that of the weight of the hyperplane. The matrix formation for kernel diagonal weighted SVM is as follows:
(23)min12αTα+C2ξTξ,s.t.D(K(A,AT)Dα+be)≥e-ξ,ξ≥0.The unconstrained program in the plus function form and the smooth form for the kernel 2-norm soft margin SVM is, respectively, illustrated as follows:(24)minF(αb)=12αTα+C2∥(e-D(K(A,AT)Dα+be))+∥2,(25)minFβ(αb)=12αTα+C2∥Pβ(e-D(K(A,AT)Dα+be))∥2.As for the gradient vector and Hessian matrix, we definet=e-D(K(A,AT)α+be) and Q=DK(A,AT)D; the gradient vector ∇Fβ(x) and the Hessian matrix ∇2Fβ(x) are computed according to (26) and (27), where v,M, and Y are the same as defined in Section 2:
(26)∇Fβ(αb)=[α0]+C[-QT-YT]Diag(Pβ(t))Diag(e+v)-1·e,(27)∇2Fβ(αb)=[I000]+C[-QT-YT]Diag(e+v)-2M[-Q-Y].Theorem 7.
LetA=Al×n and b∈R. For the two programs F(αb) and Fβ(αb) in (24) and (25),(1)
there, respectively, exist global minima ofminF(αb) and minFβ(αb).(2)
let(α¯b¯) and (α¯βb¯β) be the global minima of minFβ(αb) and minF(αb) one has the following inequality with ρ=max1≤i≤l|(e-D(K(A,AT)Dα+be))i|:
(28)∥α¯β-α¯∥2+∥b¯β-b¯∥2≤l((ln2β)2+2ρβln2).
Thus,(α¯βb¯β) converges to (α¯b¯) as β approaches infinity.Theorem7 can be proved similarly as Theorem 6; thus its detailed process is omitted.


## 3.2. Exact Kernel Smooth Model

Now we will propose the exact smooth model in the kernel space.Since the nonlinear mapϕ(x) is unknown, the connotative conditions between the primal program and the dual one are exploited.It is known that at the optimal solution(29)w=∑i=1lyiαiϕ(xi).Using this relation, we have the following two equations, in whichQij=yiyjK(xi,xj):
(30)yiwTϕ(xi)=∑j=1lyiyjαjϕ(xj)Tϕ(xi)=∑j=1lQijαj=(Qα)i,wTw=∑i=1lyiαiϕ(xi)Tyjαjϕ(xj)=∑i=1lαi(Qα)i=αTQα.Put the above two equations back into (22); we get the following program:
(31)min12αTQα+C2ξTξ,s.t.Qα+by≥e-ξ,ξ≥0.Though (31) is different from (22), the below theorem will prove that, for any optimal solution α* to (31), w*=∑i=1lyiαi*ϕ(xi) is also an optimal solution to (22).Theorem 8.
Let(α*,b*,ξ*) be the optimal solution to program (31); then (w*,b*,ξ*) is the optimal solution to program (22), where w* is defined as w*=∑i=1lyiαi*ϕ(xi).Proof.
Evidently, if(w,b,ξ) and (α,b,ξ) are the primal and dual optimal solutions, then (α,b,ξ) is feasible for (31).
For feasible(w,b,ξ) and (α,b,ξ), inequality (31) can be deduced from (29):
(32)12wTw+C2ξTξ=12αTQα+C2ξTξ≥12(α*)TQα*+C2(ξ*)Tξ*=12(w*)Tw*+C2(ξ*)Tξ*.
Thus,(w*,b*,ξ*) is the optimal solution to (31).
From the constraint in (31), we write the slack ξ in the form of plus function as
(33)ξ=(e-Qα-by)+=(e-D(K(A,AT)Dα+be))+.
Then we obtain the model in the plus function form and the model in the entropy function form in the kernel space:(34)minF(αb)=12αTDK(A,AT)Dα+C2∥(e-D(K(A,AT)Dα+be))+∥2(35)minFβ(αb)=12αTDK(A,AT)Dα+C2∥Pβ(e-D(K(A,AT)Dα+be))∥2.The gradient vector and the Hessian matrix of (35) can be computed as follows, where t and Q are the same as defined in Section 3.1, while v, M, and Y are the same as defined in Section 2:
(36)∇Fβ(αb)=[Qα0]+C·[-QT-YT]·Diag(Pβ(t))·Diag(e+v)-1·e,(37)∇2Fβ(αb)=[Q000]+C·[-QT-YT]·Diag(e+v)-2·(I+βDiag(v)·Diag(Pβ(t)))·[-Q-Y].Theorem 9.
LetQ=Ql×l(Q=DK(A,AT)D), α∈Rl, and b∈R. For the two programs F(αb) and Fβ(αb) in (36) and (37),(1)
there, respectively, exist global minima ofminF(αb) and minFβ(αb).(2)
let(α¯b¯) and (α¯βb¯β) be the global minima of minFβ(αb) and minF(αb), the following inequality holds with ρ=max1≤i≤l|(e-D(K(A,AT)Dα+be))i|;
(38)∥Q1/2(α¯-α¯β)∥22+∥b¯-b¯β∥2≤l((ln2β)2+2ρβln2).Corollary 10.
Asβ approaches infinity, the unique solution (α¯βb¯β) to minFβ(αb) converges to the unique solution (α¯b¯) to minF(αb); that is, limβ→∞(α¯βb¯β)=(α¯b¯).


## 4. Implementation

In this section, we will first prove the existences of the unique solutions to the smooth models (16), (25), and (35) and next propose Newton algorithm to figure out solutions to the smooth models of diagonal weighted SVM.Theorem 11.
Programs (16), (25), and (35) have unique solutions.Proof.
We definet=e-(DAw-Dbe) in the linear space and t=e-D(K(A,AT)α-be) in the kernel space.
Letv=exp(-βt),M=(I+βDiag(v)·Diag(Pβ(t)); then for any column vector γ∈Rn, the inequalities Diag(e+v)-2M>0 and γT·Diag(e+v)-2M·γ>0 hold.
Using the results for∇2Fβ(x) in (11), (27), and (37), we know that ∇2Fβ(x)>0 holds, which guarantee that programs (16), (25) and (35) are convex and they all have unique solutions. Now, we complete the proof.SDWNSVM AlgorithmStep
0 (initialization). Input training set T={(x1,y1),(x2,y2),…,(xl,yl)}, the initial point (α0b0)∈Rl+1, and ε>0. Calculate the gradient vector ∇Fβ(α0b0) and the Hessian matrix ∇2Fβ(α0b0). Set i=0.Step
1 (termination). Having (αibi), stop if the gradient vector is zero, that is, ||∇Fβ(αibi)||<ε; else, go to step 2.Step
2 (newton direction). Determine the direction di∈Rl+1 by setting the linearization of Fβ(αb) around (αibi) equal to zero, which gives l+1 linear equations in l+1 variables:
(39)∇2Fβ(αibi)di=-∇Fβ(αibi).Step
3 (Armijo stepsize). Find the search direction λi using Armijo stepsize, that is, choose λi=max{1,1/2,1/4,…}, such that for δ∈(0,1/2),
(40)Fβ(αibi)-F(αibi)+λidi≥-δλi∇Fβ(αibi)di.Step
4 (update). Update the point according to
(41)(αi+1bi+1)=(αibi)+λidi.Compute the gradient vector∇Fβ(αi+1bi+1). Let k:=k+1. Go to step 1.Here we are solving a linear system ofl+1 equations in SDWNSVM instead of solving a quadratic program, which is the case for classical SVM.Comments. Although the presented SDWNSVM looks similar to SSVM, they are different in nature. Their main differences can be summarized into two points.(1)
The unconstrained programs have different convexities in SDWNSVM and SSVM. The programF(αb) is convex in SDWNSVM, while it is strict convex in SSVM.(2)
The distances between(α¯βb¯β) and (α¯b¯) have different upper bounds equality in SDWNSVM and SSVM. By the first order optimality condition of the strict convex function, SSVM has the upper bound inequality as
(42)∥α¯β-α¯∥2+∥b¯β-b¯∥2≤l2((ln2β)2+2ρβln2).Obviously, with regard to the distance between(α¯βb¯β) and (α¯b¯), the upper bound of the rough kernel smooth model in SDWNSVM is twice that in SSVM, while the upper bound of the exact kernel smooth model in SDWNSVM is absolutely different form that in SSVM, as illustrated in  (38)


## 5. Experiments and Comparisons

We demonstrate now the effectiveness and speed of SDWNSVM on several UCI datasets. All the experiments are carried out on a PC with P4 CPU, 3.06 GHz, and 1 GB memory. The programs are written in pure MATALAB 7.01 language. The parameters are chosen for the optimal performances on a tuning set by the ten-fold cross validation.


### 5.1. Parameter Setting

In the linear space, only the penaltyC needs to be specified. In the kernel space, the penalty C and the kernel width σ need to be specified with respect to the radial basis kernel function K(x,y)=exp(-||x-y||2/σ2).The detailed procedure of ten-fold cross validation consists of the following three steps.(1)
Setting the Parameter-Ranges. Specify penalty C∈{0.01,0.1,0.5,0.8,1,5,10,50,100}, and specify σ∈{0.01,0.02,0.05,0.1,0.5,1,10,50,100} as the kernel width.(2)
Estimating the Performance. Select C or (σ,C) from the parameter-ranges, run ten-fold cross validation to train a classifier, and test its performances. Firstly, divide the training set into ten subsets; secondly, select nine subsets as the training set to construct a classifier; finally, take the one subset left as the tuning set to test the generalization performances of the obtained classifier.(3)
Selecting the Optimal Parameters. Select the parameters as the optimal parameters, which are corresponding to the highest ten-fold testing accuracies.Obviously, the training and testing procedures need to be repeated ten times. The ten-fold training accuracy is defined as the averaged accuracy obtained on the nine training subsets, while the ten-fold testing accuracy is defined as the averaged accuracy obtained on the tuning set.We carry out the experiments on the ionosphere data withC=5 in the linear space and illustrate performances of the ten-fold cross validation for SSVM and SDWNSVM.Data in Table1 lead to the following conclusions.(1)
SSVM and SDWNSVM both have satisfying training and testing accuracies. The former has the training and testing accuracy of about 93.86% and 89.48%, while the latter has the training and testing accuracy of about 94.21% and 89.19%.(2)
SSVM and SDWNSVM both have low iteration number and training time. The former has an iteration number of about 7.6 and a training time of about 0.14 while the latter has an iteration number of about 5 and a training time of about 0.12.(3)
SSVM and SDWNSVM both have small variances. SSVM has the training and testing accuracy variance of about 0.0062 and 0.2844, while SDWNSVM has the training and testing accuracy variance of about 0.0044 and 0.2803. SSVM has an iteration number variance of about 0.2667 and a training time variance of about 0.0032, while SDWNSVM has an iteration number variance of 0 and a training time variance of about 0.0031.(4)
The differences are not statistically significant in the ten-fold cross validation.Table 1
Performances of the ten-fold cross validation.
Fold
SSVM
SDWNSVM
Train
Test
Iteration
Time
Train
Test
Iteration
Time
1
92.09%
91.43%
7
0.06
94.30%
82.86%
5
0.05
2
94.30%
88.57%
8
0.08
93.35%
85.71%
5
0.05
3
93.35%
97.14%
7
0.11
95.25%
88.57%
5
0.08
4
94.30%
94.29%
8
0.09
93.99%
85.71%
5
0.08
5
93.35%
97.14%
8
0.13
94.30%
91.43%
5
0.09
6
93.67%
82.86%
8
0.14
93.67%
94.29%
5
0.13
7
93.99%
88.57%
7
0.17
93.67%
100.00%
5
0.14
8
94.30%
85.71%
7
0.19
94.62%
88.57%
5
0.17
9
94.30%
85.71%
8
0.20
93.67%
91.43%
5
0.19
10
94.92%
83.33%
8
0.23
95.24%
83.33%
5
0.20
Mean/variance
93.86%/0.0062
89.48%/0.2844
7.6/0.2667
0.14/0.0032
94.21%/0.0044
89.19%/0.2803
5/0
0.12/0.0031The differences among the ten folds are in Table2Table 2
Differences among the ten different folds.
Absolute values of differences
SSVM
SDWNSVM
Train
Test
Iteration
Time
Train
Test
Iteration
Time
1 versus 2
0.0221
0.0286
1
0.02
0.0095
0.0285
0
0
2 versus 3
0.0095
0.0857
1
0.03
0.0187
0.0286
0
0.03
3  versus 4
0.0095
0.0285
1
0.02
0.0186
0.0286
0
0
4  versus 5
0.0095
0.0285
0
0.04
0.0031
0.0572
0
0.01
5  versus 6
0.0032
0.1428
0
0.01
0.0063
0.0286
0
0.04
6  versus 7
0.0032
0.0571
1
0.03
0
0.0571
0
0.01
7  versus 8
0.0031
0.0286
0
0.02
0.0095
0.1143
0
0.03
8  versus 9
0
0
1
0.01
0.0095
0.0286
0
0.02
9  versus 10
0.0062
0.0238
0
0.03
0.0157
0.0810
0
0.01
10  versus 1
0.0283
0.0810
1
0.17
0.0094
0.0047
0
0.15Data in Table2 bring about the following observations, which demonstrate that the differences among the ten different folds are trivial.(1)
The differences of the training and testing accuracies are mostly not larger than 0.03.(2)
The differences of the iterations are mostly about 1 in SSVM and 0 in SDWNSVM.(3)
The differences of the training time are mostly not larger than 0.03.Tables1 and 2 demonstrate that the ten-fold cross validation has satisfying training and testing accuracies; also the differences among the ten different folds are trivial and they are not statistically significant.


### 5.2. Performances in the Linear and Kernel Space

The experiment is carried out in the liner space on several datasets. SDWNSVM is compared with RLP, SVM||1||, SVM||2||, and FSV in terms of ten-fold training and testing accuracies. The results are illustrated in Table 3 with the penalty parameter specified by C∈{0.01,0.1,0.5,0.8,1,3,5,8,10}.Table 3
Comparisons with various algorithms.
Data (sizem*n)
Ten-fold training accuracies (%)
Ten-fold testing accuracies (%)
SDWNSVM
RLP
SV
M
∥
1
∥
SVM
∥
2
∥
FSV
Bupa Liver (345*6)
71.60
68.98
67.83
70.57
71.21
70.19
64.34
64.03
69.86
69.81
Ionosphere (351*34)
94.94
94.78
88.92
92.96
94.87
89.18
86.04
86.10
89.17
86.76
Pima Indians (768*8)
78.13
76.48
75.52
77.92
77.91
77.99
76.16
74.47
77.07
76.96Obviously, SDWNSVM has the highest ten-fold training and testing correctness.SDWNSVM is also compared with SSVM, SOR (successive over relaxation), andSVMlight on the large-scale adult data in Figure 1, whose training sizes vary from 1605 to 32562. The linear kernel is used and the penalty parameter is selected from C∈{0.1,0.3,0.5,0.8,1,5,10,50,100} by the ten-fold cross validation.Figure 1
Training accuracies of SDWNSVM with various algorithms.Evidently, SDWNSVM has the highest ten-fold training accuracies.SDWNSVM is compared with SSVM on the Bupa Liver and Pima Indians data. The former data set is composed of 241 “train” examples and 104 “test” examples, containing five attributes for each sample. The latter Pima Indians diabetes data is composed of 500 plus examples and 268 minus examples, eight attributes for each example. The results are illustrated in Table4 with parameters selected by ten fold cross validation with C∈{0.1,0.5,1,5,10,15,30,50,80,100} and σ∈{0.01,0.05,0.3,0.5,0.8,1,1.2,3,5,8}.Table 4
Comparisons with various algorithms.
Algorithm
Assessment indexes
Armijo
Iterations
Time
Training accu.
Testing accu.
SSVM Bupa (345×6)
Linear
1
4.1
0.01
70.11%
68.08%
0
4.3
0.01
70.08%
67.83%
Kernel
1
2
0.21
100%
60.36%
0
2
0.20
100%
60.29%
SDWNSVM Bupa (345×6)
Linear
1
4.1
0.01
71.60%
70. 19%
0
4.3
0.01
71.37%
69.56%
Rough
1
2
0.21
100%
60.18%
0
2
0.20
100%
60.57%
Exact
1
3.3
0.73
100%
61.12%
0
3.6
0.67
100%
60.91%
SSVM Pima (768×8)
Linear
1
4.3
0.01
78.02%
77.34%
0
4.3
0.01
77.89%
77.21 %
Kernel
1
2
3.38
100%
65.11%
0
2
3.12
100%
65.09%
SDWNSVM Pima (768×8)
Linear
1
4.3
0.01
78.14%
77.48%
0
4.3
0.01
78.11%
77.47%
Rough
1
2
2.99
100%
65.11%
0
2
2.99
100%
65.19%
Exact
1
2
2.99
100%
65.11%
0
2
2.99
100%
65.19%The following conclusions are directly drawn from Table4.(1)
The iterations of SDWNSVM are identical to those of SSVM.(2)
The training time of SDWNSVM remains the same as that of SSVM in linear space, but a little higher than that of SSVM in the kernel space.(3)
The training and testing accuracies of SDWNSVM are higher than those of SSVM. In the linear space, SDWNSVM improves the training and testing accuracies by about 1.42% and 2.11%. In the kernel space, SDWNSVM improves the testing accuracies by about 0.96%.To further demonstrate the superiority of SDWNSVM, we will compare SDWNSVM with earlier SSVM algorithm, as well as the currently frequently used algorithms with good performance, namely, PWSSVM and FPSSVM. The training time, the training and testing accuracies are selected as the assessment indexes.The “tried and true” checkerboard data set is used, which is generated by uniformly discrediting the regions[0,1]×[0,1] to 1002=10000 points and labeling two classes “White” and “Black.” For the sake of comparisons, the same parameters are erected for various algorithms. The Gaussian kernel is used; the kernel width is set to be σ=0.5 and the penalty is set to be C=1. The training set is randomly sampling with increasing sizes while the others are selected as the testing set. The averaged results are presented in Table 5 of ten random samplings.Table 5
Comparisons with various algorithms.
Training size
Algorithm
Time (s)
Training accu. (%)
Testing accu. (%)
1000
SSVM
4.61
98.28
97.65
FPSSVM
4.65
99.64
98.08
PWSSVM
4.31
99.82
98.41
SDWNSVM
4.10
99.87
99.02
2000
SSVM
11.97
98.54
98.33
FPSSVM
10.20
99.11
98.29
PWSSVM
9.67
99.18
98.83
SDWNSVM
9.67
99.69
99.36
3000
SSVM
25.21
98.97
97.93
FPSSVM
23.21
98.91
98.57
PWSSVM
19.38
99.07
98.94
SDWNSVM
15.35
99.69
99.36It is obvious that SDWNSVM has the highest training and testing accuracies with the lowest training time among the four different algorithms.The final experiment is designed to show the accuracies and computational advantages over datasets with larger feature.The NDC data generator is used to produce data of various dimensional data with different scales. Randomly choose 20% as the training set and choose the others as the testing set. Select the Gaussian kernel; the averaged performances of ten random samplings are illustrated in Table6 for the four algorithms under the same parameters with the kernel width σ=0.1 and the penalty C=1.Table 6
Comparisons with various algorithms.
Scale
Dimension
Algorithm
Time (s)
Training accu. (%)
Testing accu. (%)
5000
50
SSVM
8.61
93.72
93.15
FPSSVM
12.31
93.46
93.08
PWSSVM
6.46
93.82
93.17
SDWNSVM
3.63
93.87
93.20
5000
200
SSVM
25.21
93.59
92.87
FPSSVM
32.57
92.31
92.26
PWSSVM
11.31
93.77
93.08
SDWNSVM
7.16
93.90
93.33
10000
50
SSVM
27.97
95.09
94.37
FPSSVM
43.27
94.93
94.58
PWSSVM
19.86
95.71
95.03
SDWNSVM
13.79
95.96
95.67
10000
200
SSVM
58.61
94.37
93.63
FPSSVM
96.41
94.33
93.49
PWSSVM
46.32
95.03
93.86
SDWNSVM
38.17
95.51
94.03From data in Table6, we draw the following conclusions.(1)
The training and testing accuracies increase with the training scales but decrease with the dimension by a certain degree for the four algorithms.(2)
For various scale and various dimensional data, SDWNSVM has the highest training and testing accuracies but has the lowest training time among the four different algorithms. Take 200 dimensional data of 10000, for example; SDWNSVM has a training accuracy, respectively, about 1.14%, 1.18%, and 0.48% higher than FPSSVM, FPSSVM, and PWSSVM, while it has a testing accuracy respectively about 0.4%, 0.54%, and 0.17% higher than SSVM, FPSSVM, and PWSSVM. SDWNSVM has a training time respectively about 20.44, 58.24, and 8.05 seconds lower than SSVM, FPSSVM, and PWSSVM.


## 5.1. Parameter Setting

In the linear space, only the penaltyC needs to be specified. In the kernel space, the penalty C and the kernel width σ need to be specified with respect to the radial basis kernel function K(x,y)=exp(-||x-y||2/σ2).The detailed procedure of ten-fold cross validation consists of the following three steps.(1)
Setting the Parameter-Ranges. Specify penalty C∈{0.01,0.1,0.5,0.8,1,5,10,50,100}, and specify σ∈{0.01,0.02,0.05,0.1,0.5,1,10,50,100} as the kernel width.(2)
Estimating the Performance. Select C or (σ,C) from the parameter-ranges, run ten-fold cross validation to train a classifier, and test its performances. Firstly, divide the training set into ten subsets; secondly, select nine subsets as the training set to construct a classifier; finally, take the one subset left as the tuning set to test the generalization performances of the obtained classifier.(3)
Selecting the Optimal Parameters. Select the parameters as the optimal parameters, which are corresponding to the highest ten-fold testing accuracies.Obviously, the training and testing procedures need to be repeated ten times. The ten-fold training accuracy is defined as the averaged accuracy obtained on the nine training subsets, while the ten-fold testing accuracy is defined as the averaged accuracy obtained on the tuning set.We carry out the experiments on the ionosphere data withC=5 in the linear space and illustrate performances of the ten-fold cross validation for SSVM and SDWNSVM.Data in Table1 lead to the following conclusions.(1)
SSVM and SDWNSVM both have satisfying training and testing accuracies. The former has the training and testing accuracy of about 93.86% and 89.48%, while the latter has the training and testing accuracy of about 94.21% and 89.19%.(2)
SSVM and SDWNSVM both have low iteration number and training time. The former has an iteration number of about 7.6 and a training time of about 0.14 while the latter has an iteration number of about 5 and a training time of about 0.12.(3)
SSVM and SDWNSVM both have small variances. SSVM has the training and testing accuracy variance of about 0.0062 and 0.2844, while SDWNSVM has the training and testing accuracy variance of about 0.0044 and 0.2803. SSVM has an iteration number variance of about 0.2667 and a training time variance of about 0.0032, while SDWNSVM has an iteration number variance of 0 and a training time variance of about 0.0031.(4)
The differences are not statistically significant in the ten-fold cross validation.Table 1
Performances of the ten-fold cross validation.
Fold
SSVM
SDWNSVM
Train
Test
Iteration
Time
Train
Test
Iteration
Time
1
92.09%
91.43%
7
0.06
94.30%
82.86%
5
0.05
2
94.30%
88.57%
8
0.08
93.35%
85.71%
5
0.05
3
93.35%
97.14%
7
0.11
95.25%
88.57%
5
0.08
4
94.30%
94.29%
8
0.09
93.99%
85.71%
5
0.08
5
93.35%
97.14%
8
0.13
94.30%
91.43%
5
0.09
6
93.67%
82.86%
8
0.14
93.67%
94.29%
5
0.13
7
93.99%
88.57%
7
0.17
93.67%
100.00%
5
0.14
8
94.30%
85.71%
7
0.19
94.62%
88.57%
5
0.17
9
94.30%
85.71%
8
0.20
93.67%
91.43%
5
0.19
10
94.92%
83.33%
8
0.23
95.24%
83.33%
5
0.20
Mean/variance
93.86%/0.0062
89.48%/0.2844
7.6/0.2667
0.14/0.0032
94.21%/0.0044
89.19%/0.2803
5/0
0.12/0.0031The differences among the ten folds are in Table2Table 2
Differences among the ten different folds.
Absolute values of differences
SSVM
SDWNSVM
Train
Test
Iteration
Time
Train
Test
Iteration
Time
1 versus 2
0.0221
0.0286
1
0.02
0.0095
0.0285
0
0
2 versus 3
0.0095
0.0857
1
0.03
0.0187
0.0286
0
0.03
3  versus 4
0.0095
0.0285
1
0.02
0.0186
0.0286
0
0
4  versus 5
0.0095
0.0285
0
0.04
0.0031
0.0572
0
0.01
5  versus 6
0.0032
0.1428
0
0.01
0.0063
0.0286
0
0.04
6  versus 7
0.0032
0.0571
1
0.03
0
0.0571
0
0.01
7  versus 8
0.0031
0.0286
0
0.02
0.0095
0.1143
0
0.03
8  versus 9
0
0
1
0.01
0.0095
0.0286
0
0.02
9  versus 10
0.0062
0.0238
0
0.03
0.0157
0.0810
0
0.01
10  versus 1
0.0283
0.0810
1
0.17
0.0094
0.0047
0
0.15Data in Table2 bring about the following observations, which demonstrate that the differences among the ten different folds are trivial.(1)
The differences of the training and testing accuracies are mostly not larger than 0.03.(2)
The differences of the iterations are mostly about 1 in SSVM and 0 in SDWNSVM.(3)
The differences of the training time are mostly not larger than 0.03.Tables1 and 2 demonstrate that the ten-fold cross validation has satisfying training and testing accuracies; also the differences among the ten different folds are trivial and they are not statistically significant.


## 5.2. Performances in the Linear and Kernel Space

The experiment is carried out in the liner space on several datasets. SDWNSVM is compared with RLP, SVM||1||, SVM||2||, and FSV in terms of ten-fold training and testing accuracies. The results are illustrated in Table 3 with the penalty parameter specified by C∈{0.01,0.1,0.5,0.8,1,3,5,8,10}.Table 3
Comparisons with various algorithms.
Data (sizem*n)
Ten-fold training accuracies (%)
Ten-fold testing accuracies (%)
SDWNSVM
RLP
SV
M
∥
1
∥
SVM
∥
2
∥
FSV
Bupa Liver (345*6)
71.60
68.98
67.83
70.57
71.21
70.19
64.34
64.03
69.86
69.81
Ionosphere (351*34)
94.94
94.78
88.92
92.96
94.87
89.18
86.04
86.10
89.17
86.76
Pima Indians (768*8)
78.13
76.48
75.52
77.92
77.91
77.99
76.16
74.47
77.07
76.96Obviously, SDWNSVM has the highest ten-fold training and testing correctness.SDWNSVM is also compared with SSVM, SOR (successive over relaxation), andSVMlight on the large-scale adult data in Figure 1, whose training sizes vary from 1605 to 32562. The linear kernel is used and the penalty parameter is selected from C∈{0.1,0.3,0.5,0.8,1,5,10,50,100} by the ten-fold cross validation.Figure 1
Training accuracies of SDWNSVM with various algorithms.Evidently, SDWNSVM has the highest ten-fold training accuracies.SDWNSVM is compared with SSVM on the Bupa Liver and Pima Indians data. The former data set is composed of 241 “train” examples and 104 “test” examples, containing five attributes for each sample. The latter Pima Indians diabetes data is composed of 500 plus examples and 268 minus examples, eight attributes for each example. The results are illustrated in Table4 with parameters selected by ten fold cross validation with C∈{0.1,0.5,1,5,10,15,30,50,80,100} and σ∈{0.01,0.05,0.3,0.5,0.8,1,1.2,3,5,8}.Table 4
Comparisons with various algorithms.
Algorithm
Assessment indexes
Armijo
Iterations
Time
Training accu.
Testing accu.
SSVM Bupa (345×6)
Linear
1
4.1
0.01
70.11%
68.08%
0
4.3
0.01
70.08%
67.83%
Kernel
1
2
0.21
100%
60.36%
0
2
0.20
100%
60.29%
SDWNSVM Bupa (345×6)
Linear
1
4.1
0.01
71.60%
70. 19%
0
4.3
0.01
71.37%
69.56%
Rough
1
2
0.21
100%
60.18%
0
2
0.20
100%
60.57%
Exact
1
3.3
0.73
100%
61.12%
0
3.6
0.67
100%
60.91%
SSVM Pima (768×8)
Linear
1
4.3
0.01
78.02%
77.34%
0
4.3
0.01
77.89%
77.21 %
Kernel
1
2
3.38
100%
65.11%
0
2
3.12
100%
65.09%
SDWNSVM Pima (768×8)
Linear
1
4.3
0.01
78.14%
77.48%
0
4.3
0.01
78.11%
77.47%
Rough
1
2
2.99
100%
65.11%
0
2
2.99
100%
65.19%
Exact
1
2
2.99
100%
65.11%
0
2
2.99
100%
65.19%The following conclusions are directly drawn from Table4.(1)
The iterations of SDWNSVM are identical to those of SSVM.(2)
The training time of SDWNSVM remains the same as that of SSVM in linear space, but a little higher than that of SSVM in the kernel space.(3)
The training and testing accuracies of SDWNSVM are higher than those of SSVM. In the linear space, SDWNSVM improves the training and testing accuracies by about 1.42% and 2.11%. In the kernel space, SDWNSVM improves the testing accuracies by about 0.96%.To further demonstrate the superiority of SDWNSVM, we will compare SDWNSVM with earlier SSVM algorithm, as well as the currently frequently used algorithms with good performance, namely, PWSSVM and FPSSVM. The training time, the training and testing accuracies are selected as the assessment indexes.The “tried and true” checkerboard data set is used, which is generated by uniformly discrediting the regions[0,1]×[0,1] to 1002=10000 points and labeling two classes “White” and “Black.” For the sake of comparisons, the same parameters are erected for various algorithms. The Gaussian kernel is used; the kernel width is set to be σ=0.5 and the penalty is set to be C=1. The training set is randomly sampling with increasing sizes while the others are selected as the testing set. The averaged results are presented in Table 5 of ten random samplings.Table 5
Comparisons with various algorithms.
Training size
Algorithm
Time (s)
Training accu. (%)
Testing accu. (%)
1000
SSVM
4.61
98.28
97.65
FPSSVM
4.65
99.64
98.08
PWSSVM
4.31
99.82
98.41
SDWNSVM
4.10
99.87
99.02
2000
SSVM
11.97
98.54
98.33
FPSSVM
10.20
99.11
98.29
PWSSVM
9.67
99.18
98.83
SDWNSVM
9.67
99.69
99.36
3000
SSVM
25.21
98.97
97.93
FPSSVM
23.21
98.91
98.57
PWSSVM
19.38
99.07
98.94
SDWNSVM
15.35
99.69
99.36It is obvious that SDWNSVM has the highest training and testing accuracies with the lowest training time among the four different algorithms.The final experiment is designed to show the accuracies and computational advantages over datasets with larger feature.The NDC data generator is used to produce data of various dimensional data with different scales. Randomly choose 20% as the training set and choose the others as the testing set. Select the Gaussian kernel; the averaged performances of ten random samplings are illustrated in Table6 for the four algorithms under the same parameters with the kernel width σ=0.1 and the penalty C=1.Table 6
Comparisons with various algorithms.
Scale
Dimension
Algorithm
Time (s)
Training accu. (%)
Testing accu. (%)
5000
50
SSVM
8.61
93.72
93.15
FPSSVM
12.31
93.46
93.08
PWSSVM
6.46
93.82
93.17
SDWNSVM
3.63
93.87
93.20
5000
200
SSVM
25.21
93.59
92.87
FPSSVM
32.57
92.31
92.26
PWSSVM
11.31
93.77
93.08
SDWNSVM
7.16
93.90
93.33
10000
50
SSVM
27.97
95.09
94.37
FPSSVM
43.27
94.93
94.58
PWSSVM
19.86
95.71
95.03
SDWNSVM
13.79
95.96
95.67
10000
200
SSVM
58.61
94.37
93.63
FPSSVM
96.41
94.33
93.49
PWSSVM
46.32
95.03
93.86
SDWNSVM
38.17
95.51
94.03From data in Table6, we draw the following conclusions.(1)
The training and testing accuracies increase with the training scales but decrease with the dimension by a certain degree for the four algorithms.(2)
For various scale and various dimensional data, SDWNSVM has the highest training and testing accuracies but has the lowest training time among the four different algorithms. Take 200 dimensional data of 10000, for example; SDWNSVM has a training accuracy, respectively, about 1.14%, 1.18%, and 0.48% higher than FPSSVM, FPSSVM, and PWSSVM, while it has a testing accuracy respectively about 0.4%, 0.54%, and 0.17% higher than SSVM, FPSSVM, and PWSSVM. SDWNSVM has a training time respectively about 20.44, 58.24, and 8.05 seconds lower than SSVM, FPSSVM, and PWSSVM.


## 6. Conclusion

This paper deals with diagonal weighted SVM and introduces smoothing technique to obtain an efficient training algorithm SDWNSVM. By using the connotative relation found by dual technique, SDWNSVM can be easily extended to the kernel space and has higher training accuracies; what is more, SDWNSVM proves the equivalence between the new smooth model and the original one. By introducing Newton algorithm to figure out the optimal solution, SDWNSVM has lower training time than existing methods. Numerical experiments on several UCI data demonstrate the effectiveness and superiority. Future work includes finding other smooth penalty functions or searching new efficient algorithms to figure out the optimal solution for the unconstrained smooth model.



---

*Source: 349120-2013-11-28.xml*