# Neural Network for Solving SOCQP and SOCCVI Based on Two Discrete-Type Classes of SOC Complementarity Functions

Authors: Juhe Sun; Xiao-Ren Wu; B. Saheya; Jein-Shan Chen; Chun-Hsu Ko
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

This paper focuses on solving the quadratic programming problems with second-order cone constraints (SOCQP) and the second-order cone constrained variational inequality (SOCCVI) by using the neural network. More specifically, a neural network model based on two discrete-type families of SOC complementarity functions associated with second-order cone is proposed to deal with the Karush-Kuhn-Tucker (KKT) conditions of SOCQP and SOCCVI. The two discrete-type SOC complementarity functions are newly explored. The neural network uses the two discrete-type families of SOC complementarity functions to achieve two unconstrained minimizations which are the merit functions of the Karuch-Kuhn-Tucker equations for SOCQP and SOCCVI. We show that the merit functions for SOCQP and SOCCVI are Lyapunov functions and this neural network is asymptotically stable. The main contribution of this paper lies on its simulation part because we observe a different numerical performance from the existing one. In other words, for our two target problems, more effective SOC complementarity functions, which work well along with the proposed neural network, are discovered.


---

## Body

## 1. Introduction

In optimization community, it is well known that there are many computational approaches to solve the optimization problems such as linear programming, nonlinear programming, variational inequalities, and complementarity problems; see [1–6] and references therein. These approaches include the method using merit function, interior point method, Newton method, nonlinear equation method, projection method, and its variant versions. All the aforementioned methods rely on iterative schemes and usually only provide “approximate” solution(s) to the original optimization problems and do not offer real-time solutions. However, real-time solutions are eager in many applications, such as force analysis in robot grasping and control applications. Therefore, the traditional optimization methods may not be suitable for these applications due to stringent computational time requirements.The neural network approach has an advantage in solving real-time optimization problems, which was proposed by Hopfield and Tank [7, 8] in the 1980s. Since then, neural networks have been applied to various optimization problems; see [9–30] and references therein. Unlike the traditional optimization algorithms, the essence of neural network approach for optimization is to establish a nonnegative Lyapunov function (or energy function) and a dynamic system that represents an artificial neural network. This dynamic system usually adopts the form of a first-order ordinary differential equation and its trajectory is likely convergent to an equilibrium point, which corresponds to the solution to the considered optimization problem.Following the similar idea, researchers have also developed many continuous-time neural networks for second-order cone constrained optimization problems. For example, Ko, Chen and Yang [31] proposed two kinds of neural networks with different SOCCP functions for solving the second-order cone program; Sun, Chen, and Ko [32] gave two kinds of neural networks (the first one is based on the Fischer-Burmeister function and the second one relies on a projection function) to solve the second-order cone constrained variational inequality (SOCCVI) problem; Miao, Chen, and Ko [33] proposed a neural network model for efficiently solving general nonlinear convex programs with second-order cone constraints. In this paper, we are interested in employing neural network approach for solving two types of SOC constrained problems, the quadratic programming problems with second-order cone constraints (SOCQP for short) and the second-order cone constrained variational inequality (SOCCVI for short), whose mathematical formats are described as below.The SOCQP is in the form of(1)min12xTQx+cTxs.t.Ax=bx∈Kwhere Q∈Rn×n, A is an m×n matrix with full row rank, b∈Rm, and K is the Cartesian product of second-order cones (SOCs), also called Lorentz cones. In other words,(2)K=Kn1×Kn2×⋯×Knq where n1,…,nq,q are positive integers, n1+⋯+nq=n, and Kni denotes the SOC in Rni defined by(3)Kni≔xi=xi1,xi2∈R×Rni-1∣xi2≤xi1.with K1 denoting the nonnegative real number set R+. A special case of (3) corresponds to the nonnegative orthant cone R+n, i.e., q=n and n1=⋯=nq=1. We assume that Q is a symmetric positive semidefinite matrix and problem (1) satisfies a suitable qualification [34], such as the generalized Slater condition that there exists x^ with strictly feasibility, then x is a solution to problem (1) if and only if there exists a Lagrange multiplier (μ,y)∈Rm×Rn such that(4)Ax-b=0c+Qx+ATμ-y=0K∋y⊥x∈KIn Section 3, we will employ two new families of SOC complementarity functions and use (4) to build up the neural network model for solving SOCQP.We say a few words about why we assume thatQ is a symmetric positive semidefinite matrix. First, it is clear that the symmetric assumption is reasonable because Q can be replaced by 1/2(QT+Q) which is symmetric. Indeed, with Q being symmetric positive definite matrix, the SOCQP can be recast as a standard SOCP. To see this, we observe that(5)12xTQx+cTx=12Q1/2x+Q-1/2c2-12cTQ-1c which is done by completing the square. Then, the SOCQP (with K=Kn) is equivalent to(6)minQ1/2x+Q-1/2cs.t.Ax=bx∈Kn which is also the same as(7)miny¯s.t.Q1/2x-y¯=-Q-1/2cAx=bx∈KnThis formulation is further equivalent to(8)miny1s.t.Q1/2x-y¯=-Q-1/2cAx=bx∈Kny1≥y¯Now, we let y≔(y1,y¯) which says y∈Kn+1 and denote (9)v^≔x,y∈Kn×Kn+1,c^≔0,e∈R2n+1,A^≔A00Q1/20In,b^≔bQ-1/2.Thus, the above reformulation (8) is expressed as a standard SOCP as follows:(10)minc^Tv^s.t.A^v^=b^v^∈Kn×Kn+1In view of this reformulation (10), we focus on SOCQP with Q being symmetric positive semidefinite in this paper.The SOCCVI, our another target problem, is to findx∈C satisfying(11)Fx,y-x≥0∀y∈C,where the set C is finitely representable and is given by(12)C=x∈Rn∣hx=0,-gx∈K.Here ·,· denotes the Euclidean inner product, F:Rn→Rn, h:Rn→Rl, and g:Rn→Rm are continuously differentiable functions, and K is a Cartesian product of second-order cones (or Lorentz cones), expressed as(13)K=Km1×Km2×⋯×Kmp,with l≥0, m1,m2,…,mp≥1, m1+m2+⋯+mp=m. When h is affine, an important special case of the SOCCVI corresponds to the KKT conditions of the convex second-order cone program (CSOCP):(14)minfxs.t.Ax=b-gx∈Kwhere A∈Rl×n has full row rank, b∈Rl, g:Rn→Rm, and f:Rn→R. Furthermore, when f is a convex twice continuously differentiable function, problem (14) is equivalent to the following SOCCVI which is to find x∈C such that(15)∇fx,y-x≥0,∀y∈C, where(16)C=x∈Rn∣Ax-b=0,-gx∈K. In fact, the SOCCVI can be solved by analyzing its KKT conditions:(17)Lx,μ,λ=0,gx,λ=0,-gx∈K,λ∈K,hx=0,where L(x,μ,λ)=F(x)+∇h(x)μ+∇g(x)λ is the variational inequality Lagrangian function, μ∈Rl and λ∈Rm. We also point out that the neural network approach for SOCCVI was already studied in [32]. Here we revisit the SOCCVI with different neural models. More specifically, in our earlier work [32], we had employed neural network approach to the SOCCVI problem (11) and (13), in which the neural networks were aimed at solving system (17) whose solutions are candidates of SOCCVI problem (11) and (13). There were two neural networks considered in [32]. The first one is based on the smoothed Fischer-Burmeister function, while the other one is based on the projection function. Both neural networks possess asymptotical stability under suitable conditions. In Section 4, in light of (17) again, we adopt new and different SOC complementarity functions to construct our new neural networks.As mentioned earlier, this paper studies neural networks by using two new classes of SOC complementarity functions to efficiently solve SOCQP and SOCCVI. Although the idea and the stability analysis for both problems are routine, we emphasize that the main contribution of this paper lies on its simulations. More specifically, from numerical performance and comparison, we observe a new phenomenon different from the existing one in the literature. This may suggest update choices of SOC complementarity functions to work with neural network approach.


## 2. Preliminaries

Consider the first-order differential equations (ODE):(18)w˙t=Hwt,wt0=w0∈Rn,where H:Rn→Rn is a mapping. A point w∗=w(t∗) is called an equilibrium point or a steady state of the dynamic system (18) if H(w∗)=0. If there is a neighborhood Ω∗⊆Rn of w∗ such that H(w∗)=0 and H(w)≠0∀w∈Ω∗∖{w∗}, then w∗ is called an isolated equilibrium point.Lemma 1.
Suppose thatH:Rn→Rn is a continuous mapping. Then, for any t0>0 and w0∈Rn, there exists a local solution w(t) to (18) with t∈[t0,τ) for some τ>t0. If, in addition, H is locally Lipschitz continuous at x0, then the solution is unique; if H is Lipschitz continuous in Rn, then τ can be extended to ∞.Letw(t) be a solution to dynamic system (18). An isolated equilibrium point w∗ is Lyapunov stable if for any w0=w(t0) and any ε>0, there exists a δ>0 such that ‖w(t)-w∗‖<ε for all t≥t0 and ‖wt0-w∗‖<δ. An isolated equilibrium point w∗ is said to be asymptotic stable if in addition to being Lyapunov stable, it has the property that w(t)→w∗ as t→∞ for all ‖w(t0)-w∗‖<δ. An isolated equilibrium point w∗ is exponentially stable if there exists δ>0 such that arbitrary point w(t) of (18) with the initial condition w(t0)=w0 and ‖w(t0)-w∗‖<δ is well defined on [0,+∞) and satisfies(19)wt-w∗≤ce-ωtwt0-w∗∀t≥t0, where c>0 and ω>0 are constants independent of the initial point.LetΩ⊆Rn be an open neighborhood of w¯. A continuously differentiable function V:Rn→R is said to be a Lyapunov function at the state w¯ over the set Ω for (18) if(20)Vw¯=0,Vw>0,∀w∈Ω∖w¯,V˙w≤0,∀w∈Ω∖w¯.The Lyapunov stability and asymptotical stability can be verified by using Lyapunov function, which is a useful tool for analysis.Lemma 2.
(a) An isolated equilibrium pointw∗ is Lyapunov stable if there exists a Lyapunov function over some neighborhood Ω∗ of w∗.
(b) An isolated equilibrium pointw∗ is asymptotically stable if there exists a Lyapunov function over some neighborhood Ω∗ of w∗ such that V˙(w)<0,∀w∈Ω∗∖{w∗}.For more details, please refer to any usual ODE textbooks, e.g., [35].Next, we briefly recall some concepts associated with SOC, which are helpful for understanding the target problems and our analysis techniques. We start with introducing the Jordan product and SOC complementarity function. For anyx=(x1,x2)∈R×Rn-1 and y=(y1,y2)∈R×Rn-1, we define their Jordan product associated with Kn as(21)x∘y=xTyy1x2+x1y2. The Jordan product ∘, unlike scalar or matrix multiplication, is not associative, which is a main source of complication in the analysis of SOC constrained optimization. There exists an identity element under this product, which is denoted by e≔(1,0,…,0)T∈Rn. Note that x2 means x∘x and x+y means the usual componentwise addition of vectors. It is known that x2∈Kn for all x∈Rn. Moreover, if x∈Kn, then there exists a unique vector in Kn, denoted by x1/2, such that (x1/2)2=x1/2∘x1/2=x. We also denote x≔(x2)1/2.A vector-valued functionϕ:Rn×Rn→Rn is called an SOC complementarity function if it satisfies(22)ϕx,y=0⇔x∘y=0,x∈Kn,y∈Kn.There have been many SOC complementarity functions studied in the literature; see [36–40] and references therein. Among them, two popular ones are the Fischer-Burmeister function ϕFB and the natural residual function ϕNR, which are given by (23)ϕFBx,y=x2+y21/2-x+y,ϕNRx,y=x-x-y+. Some existing SOC complementarity functions are indeed variants of ϕFB and ϕNR. Recently, Ma, Chen, Huang, and Ko [41] explored the idea of “discrete generalization” to the Fischer-Burmeister function which yields the following class of functions (denoted by ϕD-FBp):(24)ϕD-FBpx,y=x2+y2p-x+yp,where p>1 is a positive odd integer. Applying similar idea, they also extended ϕNR to another family of SOC complementarity functions, ϕNRp:Rn×Rn→Rn, whose formula is as follows:(25)ϕNRpx,y=xp-x-y+p,where p>1 is a positive odd integer and (·)+ means the projection onto Kn. The functions ϕD-FBp and ϕNRp are continuously differentiable SOC complementarity functions with computable Jacobian, which can be found in [41].


## 3. Neural Networks for SOCQP

In this section, we first show how we achieve the neural network model for SOCQP and prove various stabilities for it accordingly. Then, numerical experiments are reported to demonstrate the effectiveness of the proposed neural network.


### 3.1. The model and Stability Analysis

As mentioned in Section2, the KKT conditions are expressed in (4). With system (4) and using a given SOC complementarity function ϕ:Rn×Rn→Rn, it is clear to see that system (4) is equivalent to(26)Hu=Ax-bc+Qx+ATμ-yϕx,y=0, where u=(x,μ,y)∈Rn×Rm×Rn. Moreover, we can specifically describe ∇H(u) as the following:(27)∇Hu=ATQ∇xϕ0A00-I∇yϕ. Here ϕ is a continuously differentiable SOC complementarity function such as ϕD-FBp and ϕNRp introduced in Section 2. It is clear that if u∗ solves H(u)=0, then u∗ solves ∇1/2‖H(u)‖2=0. Accordingly, we consider a specific first-order ordinary differential equation as follows:(28)dutdt=-ρ∇12Hu2,ut0=u0,where ρ>0 is a time scaling factor. In fact, letting τ=ρt, then du(τ)/dτ=ρdu(τ)/dτ. Hence, it follows from (28) that du(τ)/dτ=-∇(1/2‖H(u∗)‖2). In view of this, we set ρ=1 in the subsequent analysis. Next, we show that the equilibrium of the neural network (28) corresponds to the solution to system (4).Lemma 3.
Letu∗ be an equilibrium of the neural network (28) and suppose that ∇H(u∗) is nonsingular. Then u∗ solves system (4).Proof.
Since∇(1/2‖H(u∗)‖2)=∇H(u∗)H(u∗) and ∇H(u∗) is nonsingular, it is clear to see that ∇(1/2‖H(u∗)‖2)=0 if and only if H(u∗)=0.Besides, the following results address the existence and uniqueness of the solution trajectory of the neural network (28).Theorem 4.
(a) For any initial pointu0=u(t0), there exists a unique continuously maximal solution u(t) with t∈[t0,τ) for the neural network (28).
(b) If the level setL(u0)≔u∣‖H(u)‖2≤‖H(u0)‖2 is bounded, then τ can be extended to ∞.Proof.
This proof is exactly the same as the one in [32, Proposition 3.4], so we omit it here.Now, we are ready to analyze the stability of an isolated equilibriumu∗ of the neural network (28), which means ∇(1/2Hu∗2)=0 and ∇(1/2‖H(u)‖2)≠0 for u∈Ω∖{u∗}, with Ω being a neighborhood of u∗.Theorem 5.
Letu∗ be an isolated equilibrium point of the neural network (28). (a)
If∇H(u∗) is nonsingular, then the isolated equilibrium point u∗ is asymptotically stable and hence Lypunov stable.(b)
If∇H(u) is nonsingular for all u∈Ω, then the isolated equilibrium point u∗ is exponentially stable.Proof.
The desired results can be proved by using Lemma3 and mimicking the arguments as in [32, Theorem 3.1].


### 3.2. Numerical Experiments

In order to demonstrate the effectiveness of the proposed neural network, we test three examples for our neural network (28). The numerical implementation is coded by Matlab 7.0 and the ordinary differential equation solver adopted here isode23, which uses Ruge-Kutta (2;3) formula. As mentioned earlier, in general the parameter ρ is set to be 1. For some special examples, the parameter ρ is set to be another value.Example 6.
Consider the following SOCQP problem:(29)minx1-32+x22+x3-12+x4-22+x5+12s.t.x∈K5After suitable transformation, it can be recast as an SOCQP withQ=2I5, c=[-6,0,-2,-4,2]T, A=0, and b=0. This problem has an optimal solution x∗=[3,0,1,2,-1]T. Now, we use the proposed neural network (28) with two cases ϕ=ϕD-FBp and ϕ=ϕNRp, respectively, to solve the above SOCQP and their trajectories are depicted in Figures 1–4. For the sake of coding needs and check, the following expressions are presented.Figure 1
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 6.Figure 2
Convergence comparison ofϕD-FBp function with different p value for Example 6.Figure 3
Transient behavior of the neural network withϕNRp function (p=3) in Example 6.Figure 4
Convergence comparison ofϕNRp function with different p value for Example 6.For case ofϕ=ϕD-FBp, we have (30)dutdt=-ρ∇HuHu,ut0=u0Hu=c+2x-yϕD-FBpx,y,u=x,y∇Hu=2I5∇xϕD-FBpu-I5∇yϕD-FBpu∇xϕD-FBpx,y=2Lx∇gsocw-2Lx+y∇gsocv,∇yϕD-FBpx,y=2Ly∇gsocw-2Lx+y∇gsocv.wx,y≔x2+y2=w1x,y,w2x,y=x2+y2,2x1x2+y1y2∈R×R4,vx,y≔x+y2=x+y2,2x1+y1x2+y2∈R×R4. Note that the element w=(w1,w2)∈R×R4 can also be expressed as(31)w≔λ1e1+λ2e2 where λi=w1+-1iw2 and ei=1/2(1,(-1)iw2/‖w2‖)(i=1,2) if w2≠0; otherwise ei=1/2(1,(-1)iν with ν being any vector in R4 satisfying ‖ν‖=1.For case ofϕ=ϕNRp, we replace ϕD-FBp(x,y) as ϕNRp(x,y), Hence, H(u) and ∇H(u) have the forms as follows:(32)Hu=c+2x-yϕNRpx,y,u=x,y∇Hu=2I5∇xϕNRpu-I5∇yϕNRpu∇xϕNRpx,y=∇hsocx-∇lsocx-y,∇yϕNRpx,y=∇lsocx-y.Figures1 and 3 show the transient behaviors of Example 6 for neural network model (28) based on smooth SOC complementarity functions ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. In Figure 2, we see the convergence comparison of the neural network model using ϕD-FBp function with different values of p=3,5,7. Figure 4 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function.Example 7.
Consider the following SOCQP problem:(33)min4x12+10x22+4x32+4x1x2+12x2x3-x1+x2+5x3s.t.2x1+x2-7=03x2+2x3-1=0x∈K3For this SOCP, we have(34)Q=840420120128,c=-115,b=71,A=210032. This problem has an approximate solution x∗=(2.6529,1.6943,-2.0414)T. Note that the precise solution is 22-37/6,-2+237/6,6-337/6T. Indeed, we have(35)Hu=Ax-bc+Qx+ATμ-yϕx,y,u=x,μ,y,∇Hu=ATQ∇xϕx,y0A00-I∇yϕx,y. We also report numerical experiments for two cases when ϕ=ϕD-FBp and ϕ=ϕNRp; see Figures 5–8.Figure 5
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 7.Figure 6
Convergence comparison ofϕD-FBp function with different p value for Example 7.Figure 7
Transient behavior of the neural network withϕNRp function (p=3) in Example 7.Figure 8
Convergence comparison ofϕNRp function with different p value for Example 7.Figures5 and 7 show the transient behaviors of Example 7 for neural network model (28) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0]T, respectively. Figure 6 provides the convergence comparison by using ϕD-FBp function with different values of p=3,5,7. Figure 8 shows the convergence of neural network model using ϕNRp function, which indicates that this class of ϕNRp functions performs not well for this problem.Example 8.
Consider the following SOCQP problem:(36)min52x12+2x22+52x32+3x1x2-2x2x3-x1x3-47x1-35x2+2x3s.t.x∈K3Here, we have(37)Q=53-134-2-1-25,c=-47,-35,2T, and A=0, b=0. This problem has an optimal solution x∗=(7,5,3)T.Figures9 and 11 show the transient behaviors of Example 8 for neural network model (28) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0], respectively. Figure 10 shows that there are no difference between the neural networks using ϕD-FBp function with p=3,5. Figure 12 elaborates that when p=5 the neural network based on ϕNRp function produces fast decrease of norm of error. We point out that the neural network does not converge when p=7 for both cases.Figure 9
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 8.Figure 10
Convergence comparison ofϕD-FBp function with different p value for Example 8.Figure 11
Transient behavior of the neural network withϕNRp function (p=3) in Example 8.Figure 12
Convergence comparison ofϕNRp function with different p value for Example 8.


## 3.1. The model and Stability Analysis

As mentioned in Section2, the KKT conditions are expressed in (4). With system (4) and using a given SOC complementarity function ϕ:Rn×Rn→Rn, it is clear to see that system (4) is equivalent to(26)Hu=Ax-bc+Qx+ATμ-yϕx,y=0, where u=(x,μ,y)∈Rn×Rm×Rn. Moreover, we can specifically describe ∇H(u) as the following:(27)∇Hu=ATQ∇xϕ0A00-I∇yϕ. Here ϕ is a continuously differentiable SOC complementarity function such as ϕD-FBp and ϕNRp introduced in Section 2. It is clear that if u∗ solves H(u)=0, then u∗ solves ∇1/2‖H(u)‖2=0. Accordingly, we consider a specific first-order ordinary differential equation as follows:(28)dutdt=-ρ∇12Hu2,ut0=u0,where ρ>0 is a time scaling factor. In fact, letting τ=ρt, then du(τ)/dτ=ρdu(τ)/dτ. Hence, it follows from (28) that du(τ)/dτ=-∇(1/2‖H(u∗)‖2). In view of this, we set ρ=1 in the subsequent analysis. Next, we show that the equilibrium of the neural network (28) corresponds to the solution to system (4).Lemma 3.
Letu∗ be an equilibrium of the neural network (28) and suppose that ∇H(u∗) is nonsingular. Then u∗ solves system (4).Proof.
Since∇(1/2‖H(u∗)‖2)=∇H(u∗)H(u∗) and ∇H(u∗) is nonsingular, it is clear to see that ∇(1/2‖H(u∗)‖2)=0 if and only if H(u∗)=0.Besides, the following results address the existence and uniqueness of the solution trajectory of the neural network (28).Theorem 4.
(a) For any initial pointu0=u(t0), there exists a unique continuously maximal solution u(t) with t∈[t0,τ) for the neural network (28).
(b) If the level setL(u0)≔u∣‖H(u)‖2≤‖H(u0)‖2 is bounded, then τ can be extended to ∞.Proof.
This proof is exactly the same as the one in [32, Proposition 3.4], so we omit it here.Now, we are ready to analyze the stability of an isolated equilibriumu∗ of the neural network (28), which means ∇(1/2Hu∗2)=0 and ∇(1/2‖H(u)‖2)≠0 for u∈Ω∖{u∗}, with Ω being a neighborhood of u∗.Theorem 5.
Letu∗ be an isolated equilibrium point of the neural network (28). (a)
If∇H(u∗) is nonsingular, then the isolated equilibrium point u∗ is asymptotically stable and hence Lypunov stable.(b)
If∇H(u) is nonsingular for all u∈Ω, then the isolated equilibrium point u∗ is exponentially stable.Proof.
The desired results can be proved by using Lemma3 and mimicking the arguments as in [32, Theorem 3.1].


## 3.2. Numerical Experiments

In order to demonstrate the effectiveness of the proposed neural network, we test three examples for our neural network (28). The numerical implementation is coded by Matlab 7.0 and the ordinary differential equation solver adopted here isode23, which uses Ruge-Kutta (2;3) formula. As mentioned earlier, in general the parameter ρ is set to be 1. For some special examples, the parameter ρ is set to be another value.Example 6.
Consider the following SOCQP problem:(29)minx1-32+x22+x3-12+x4-22+x5+12s.t.x∈K5After suitable transformation, it can be recast as an SOCQP withQ=2I5, c=[-6,0,-2,-4,2]T, A=0, and b=0. This problem has an optimal solution x∗=[3,0,1,2,-1]T. Now, we use the proposed neural network (28) with two cases ϕ=ϕD-FBp and ϕ=ϕNRp, respectively, to solve the above SOCQP and their trajectories are depicted in Figures 1–4. For the sake of coding needs and check, the following expressions are presented.Figure 1
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 6.Figure 2
Convergence comparison ofϕD-FBp function with different p value for Example 6.Figure 3
Transient behavior of the neural network withϕNRp function (p=3) in Example 6.Figure 4
Convergence comparison ofϕNRp function with different p value for Example 6.For case ofϕ=ϕD-FBp, we have (30)dutdt=-ρ∇HuHu,ut0=u0Hu=c+2x-yϕD-FBpx,y,u=x,y∇Hu=2I5∇xϕD-FBpu-I5∇yϕD-FBpu∇xϕD-FBpx,y=2Lx∇gsocw-2Lx+y∇gsocv,∇yϕD-FBpx,y=2Ly∇gsocw-2Lx+y∇gsocv.wx,y≔x2+y2=w1x,y,w2x,y=x2+y2,2x1x2+y1y2∈R×R4,vx,y≔x+y2=x+y2,2x1+y1x2+y2∈R×R4. Note that the element w=(w1,w2)∈R×R4 can also be expressed as(31)w≔λ1e1+λ2e2 where λi=w1+-1iw2 and ei=1/2(1,(-1)iw2/‖w2‖)(i=1,2) if w2≠0; otherwise ei=1/2(1,(-1)iν with ν being any vector in R4 satisfying ‖ν‖=1.For case ofϕ=ϕNRp, we replace ϕD-FBp(x,y) as ϕNRp(x,y), Hence, H(u) and ∇H(u) have the forms as follows:(32)Hu=c+2x-yϕNRpx,y,u=x,y∇Hu=2I5∇xϕNRpu-I5∇yϕNRpu∇xϕNRpx,y=∇hsocx-∇lsocx-y,∇yϕNRpx,y=∇lsocx-y.Figures1 and 3 show the transient behaviors of Example 6 for neural network model (28) based on smooth SOC complementarity functions ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. In Figure 2, we see the convergence comparison of the neural network model using ϕD-FBp function with different values of p=3,5,7. Figure 4 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function.Example 7.
Consider the following SOCQP problem:(33)min4x12+10x22+4x32+4x1x2+12x2x3-x1+x2+5x3s.t.2x1+x2-7=03x2+2x3-1=0x∈K3For this SOCP, we have(34)Q=840420120128,c=-115,b=71,A=210032. This problem has an approximate solution x∗=(2.6529,1.6943,-2.0414)T. Note that the precise solution is 22-37/6,-2+237/6,6-337/6T. Indeed, we have(35)Hu=Ax-bc+Qx+ATμ-yϕx,y,u=x,μ,y,∇Hu=ATQ∇xϕx,y0A00-I∇yϕx,y. We also report numerical experiments for two cases when ϕ=ϕD-FBp and ϕ=ϕNRp; see Figures 5–8.Figure 5
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 7.Figure 6
Convergence comparison ofϕD-FBp function with different p value for Example 7.Figure 7
Transient behavior of the neural network withϕNRp function (p=3) in Example 7.Figure 8
Convergence comparison ofϕNRp function with different p value for Example 7.Figures5 and 7 show the transient behaviors of Example 7 for neural network model (28) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0]T, respectively. Figure 6 provides the convergence comparison by using ϕD-FBp function with different values of p=3,5,7. Figure 8 shows the convergence of neural network model using ϕNRp function, which indicates that this class of ϕNRp functions performs not well for this problem.Example 8.
Consider the following SOCQP problem:(36)min52x12+2x22+52x32+3x1x2-2x2x3-x1x3-47x1-35x2+2x3s.t.x∈K3Here, we have(37)Q=53-134-2-1-25,c=-47,-35,2T, and A=0, b=0. This problem has an optimal solution x∗=(7,5,3)T.Figures9 and 11 show the transient behaviors of Example 8 for neural network model (28) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0], respectively. Figure 10 shows that there are no difference between the neural networks using ϕD-FBp function with p=3,5. Figure 12 elaborates that when p=5 the neural network based on ϕNRp function produces fast decrease of norm of error. We point out that the neural network does not converge when p=7 for both cases.Figure 9
Transient behavior of the neural network withϕD-FBp function (p=3) in Example 8.Figure 10
Convergence comparison ofϕD-FBp function with different p value for Example 8.Figure 11
Transient behavior of the neural network withϕNRp function (p=3) in Example 8.Figure 12
Convergence comparison ofϕNRp function with different p value for Example 8.


## 4. Neural Networks for SOCCVI

This section is devoted to another type of SOC constrained problem, SOCCVI. Like what we have done for SOCQP, in this section, we first show how we build up the neural network model for SOCCVI and prove various stabilities for it accordingly. Then, numerical experiments are reported to demonstrate the effectiveness of the proposed neural network.


### 4.1. The Model and Stability Analysis

Letϕ(x,y) be a SOC complementarity function like ϕD-FBp and ϕNRp defined as in (24) and (25), respectively. Mimicking the arguments described as in [42], we can verify that the KKT system (17) is equivalent to the following unconstrained smooth minimization problem:(38)minΨz≔12Sz2,where z=(x,μ,λ)∈Rn+l+m and S(z) is given by(39)Sz=Lx,μ,λ-hxϕ-gm1x,λm1⋮ϕ-gmqx,λmq, with gmi(x),λmi∈Rmi. In other words, Ψ(z) is a smooth merit function for the KKT system (17). Hence, based on the above smooth minimization problem (38), it is natural to propose a neural network for solving the SOCCVI as follows:(40)dztdt=-ρ∇Ψzt,zt0=z0,where ρ>0 is a scaling factor. To prove the stability of neural network (40), we need to present some properties of Ψ(·).Proposition 9.
LetΨ:Rn+l+m→R+ be defined as in (38). Then, Ψ(z)≥0 for z=(x,μ,λ)∈Rn+l+m. Moreover, Ψ(z)=0 if and only if (x,μ,λ) solves the KKT system (17).Proof.
The proof is straightforward.Proposition 10.
LetΨ:Rn+l+m→R+ be defined as in (38). Then, the following results hold. (a)
The functionΨ is continuously differentiable everywhere with(41)∇Ψz=∇SzSz,where(42)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiϕ-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiϕ-gmix,λmii=1q.(b)
If∇S(z) is nonsingular, then for any stationary point (x,μ,λ)∈Rn+l+m of Ψ, (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.(c)
Ψ(z(t)) is nonincreasing with respect to t.Proof.
( a ) It follows from the chain rule immediately.
( b ) From ∇Ψ(z)=∇S(z)S(z) and the fact that matrix ∇S(z) is nonsingular, it is clear that ∇Ψ(z)=0 if and only if S(z)=0. Hence, we see that (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem provided (x,μ,λ)∈Rn+l+m is a stationary point of Ψ.
( c ) From the definition of Ψ(z) and (40), it is easy to verify that(43)dΨztdt=∇ΨztTdztdt=-ρ∇Ψzt2≤0,which says Ψ(z(t)) is a monotonically decreasing function with respect to t.Now, we are ready to analyze the behavior of the solution trajectory of neural network (40) and establish three kinds of stabilities for an isolated equilibrium point.Proposition 11.
(a) If(x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem, then (x,μ,λ)∈Rn+l+m is an equilibrium point of neural network (40).
(b) If∇S(z) is nonsingular and (x,μ,λ)∈Rn+l+m is an equilibrium point of (40), then (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.Proof.
( a ) From Proposition 9 and (x,μ,λ)∈Rn+l+m being a KKT triple of SOCCVI problem, it is clear that S(x,μ,λ)=0, which implies ∇Ψ(x,μ,λ)=0. Besides, by Proposition 10, we know that ∇Ψ(x,μ,λ)≠0. This shows that (x,μ,λ) is an equilibrium point of neural network (40).
( b ) It follows from (x,μ,λ)∈Rn+l+m being an equilibrium point of neural network (40) that ∇Ψ(x,μ,λ)=0. In other words, (x,μ,λ) is the stationary point of Ψ. Then, the result is a direct consequence of Proposition 10(b).Proposition 12.
(a) For any initial statez0=z(t0), there exists exactly one maximal solution z(t) with t∈[t0,τ(x0)) for the neural network (40).
(b) If the level setL(z0)=z∈Rn+l+m∣Ψ(z)≤Ψ(z0) is bounded, then τ(x0)=+∞.Proof.
( a ) Since S(·) is continuous differentiable, it says that ∇S(·) is continuous. This means ∇S(·) is bounded on a local compact neighborhood of z, which implies that ∇Ψ(z) is locally Lipschitz continuous. Thus, applying Lemma 1 leads to the desired result.
( b ) This proof is similar to the proof of Case(i) in [10, Proposition 4.2], so we omit it.Remark 13.
A natural question arises here. When are the level sets(44)LΨ,γ≔z∈Rn+l+m∣Ψz≤γ bounded for all γ∈R? For the time being, we are not able to answer this question yet. We suspect that there needs more subtle properties of F, h, and g to finish it.Next, we investigate the convergence of the solution trajectory and stability of neural network (40), which are the main results of this section.Theorem 14.
(a) Letz(t) with t∈[t0,τ(z0)) be the unique maximal solution to the neural network (40). If τ(z0)=+∞ and {z(t)} is bounded, then limt→∞⁡∇Ψ(z(t))=0.
(b) If∇S(z) is nonsingular and (x,μ,λ)∈Rn+l+m is the accumulation point of the trajectory z(t), then (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.Proof.
With Proposition10(b) and 10(c) and Proposition 12, the arguments are exactly the same as those for [19, Corollary 4.3]. Thus, we omit them.Theorem 15.
Letz∗ be an isolated equilibrium point of the neural network (40). Then, the following results hold. (a)
z∗ is asymptotically stable and hence is also Lyapunov stable.(b)
If∇S(z) is nonsingular, then it is exponentially stable.Proof.
Again, the arguments are similar to those in [32, Theorem 3.1] and we omit them.To study the conditions for nonsingularity based onψD-FBp and ϕNRp, we need the following assumptions.Assumption 16.
( a ) The gradients {∇hjx∣j=1,…,l}∪{∇gi(x)∣i=1,…,m} are linear independent.
( b )  ∇xL(x,μ,λ) is positive definite on the null space of the gradients {∇hj(x)∣j=1,…,l}.When SOCCVI problem corresponds to the KKT conditions of a convex second-order cone program (CSOCP) problem as (14) where both h and g are linear, the above Assumption 16(b) is indeed equivalent to the well-used condition of ∇2f(x) being positive definite, e.g., [22, Corollary 1].Assumption 17.
Letα≔wmip/2 and β≔vmip/2, where wmi=gmi2+λmi2 and vmi=(gmi+λmi)2. For gmi(x),λmi∈Kmi, we have (a)
Lgmi2-LβLα-1Lgmi2Lα-1Lβ⪰0 or LαLβ-1Lgmi2Lβ-1Lα-Lgmi2⪰0;(b)
Lλmi2-LβLα-1Lλmi2Lα-1Lβ⪰0 or LαLβ-1Lλmi2Lβ-1Lα-Lλmi2⪰0.Theorem 18.
Suppose-gmi+λmi∈int⁡Kmi for i=1,2,…,p and that Assumptions 16 and 17 hold. Then, the matrix(45)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiψD-FBp-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiψD-FBp-gmix,λmii=1qis nonsingular.Proof.
We know that∇S(z) is nonsingular if and only if the following equation only has zero solution:(46)∇Szuvt=0,whereu,v,t∈Rn×Rl×Rm.To reach the conclusion, we need to prove u=0,v=0,t=0. First, plugging the components of ∇S(z) into (46), we have(47)∇xLTu-∇hxv-∇gxL-g+λLβ-1-L-gLα-1t=0(48)∇hxTu=0(49)∇gxTu+L-g+λLβ-1-LλLα-1t=0where (50)Lg+λ=diag⁡L-gm1+λm1,L-gm2+λm2,…,L-gmq+λmqLβ=diag⁡∇gsocvm1,∇gsocvm2,∇gsocvmqL-g=diag⁡L-gm1,L-gm2,…,L-gmqLα=diag⁡∇gsocwm1,∇gsocwm2,∇gsocwmqLλ=diag⁡Lλm1,Lλm2,…,Lλmqα=diag⁡αm1,αm2,…,αmqαmi=vmip/2=-gmi+λmip,i=1,2,…,qβ=diag⁡βm1,βm2,…,βmqβmi=wmip/2=-gmi2+λmi2p/2,i=1,2,…,q From (47) and (48), we see that(51)uT∇xLTu-uT∇gxL-g+λLβ-1-L-gLα-1t=0,while from (49), we have(52)tTL-g+λLβ-1-L-gLα-1T∇gxTu+tTL-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1t=0Next, we will claim that(53)tTL-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1t≥0To see this, we note that (54)L-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1=diag⁡L-gm1+λm1Lβm1-1-L-gm1Lαm1-1TL-gm1+λm1Lβm1-1-Lλm1Lαm1-1,…,L-gmq+λmqLβmq-1-L-gmqLαmq-1TL-gmq+λmqLβmq-1-LλmqLαmq-1. In view of this, to prove inequality (53), it suffices to show that(55)tiTL-gmi+λmiLβmi-1-L-gmiLαmi-1TL-gmi+λm1Lβmi-1-LλmiLαmi-1ti≥0,for i=1,2,…,q. For convenience, we denote X≔-gmi, Y≔λmi, A≔αmi, and B≔βmi. With these notations, we have (56)L-gmi+λmiLβmi-1-L-gmiLαmi-1TL-gmi+λm1Lβmi-1-LλmiLαmi-1=LX+YLB-1-LXLA-1TLX+YLB-1-LYLA-1=LB-1LX+Y-LBLA-1LXLX+Y-LYLA-1LBLB-1, which says that it is enough to show M≔(LX+Y-LBLA-1LX)(LX+Y-LYLA-1LB) is semipositive definite in order to prove inequality (55). To this end, we compute that(57)12LX+Y-LBLA-1LXLX+Y-LYLA-1LB+LX+Y-LBLA-1LXLX+Y-LYLA-1LBT=122LX+Y2-LX+Y2LA-1LB-LBLA-1LX+Y2+LBLA-1LXLY+LYLXLA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX+Y2-LBLA-1LX+Y2LA-1LB+LBLA-1LXLY+LYLXLA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX+Y2-LBLA-1LX2+LY2LA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX2-LBLA-1LX2LA-1LB+LY2-LBLA-1LY2LA-1LB+LXLY+LYLX.It can be verified that LX+Y2 and LXLY+LYLX are positive semidefinite. Then, from Assumption 17 and (57), we conclude that M≔(LX+Y-LBLA-1LX)(LX+Y-LYLA-1LB) is semipositive definite; and hence inequality (55) holds. Thus, inequality (53) also holds accordingly. Now, tt follows from (51), (52), and (53) that uT(∇xL)Tu=0 which implies that u=0. Then, (47) and (48) become(58)∇hxv+∇gxL-g+λLβ-1-L-gLα-1t=0(59)L-g+λLβ-1-LλLα-1t=0In light of Assumption 16(a) and (58), we know(60)v=0,L-g+λLβ-1-L-gLα-1t=0.Combining (59) and (60) together, it is clear to obtain(61)L-gLα-1t=LλLα-1tNote that -g and λ are strict complementary. Hence, it yields t=0. In summary, from equation (46), we deduce u=v=t=0, which says the matrix ∇S(z) is nonsingular.Theorem 19.
Suppose Assumption16 holds and(62)∇gmiϕNRp-gmix,λmi·∇λmiϕNRp-gmix,λmi⪰0.Then, the matrix(63)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiϕNRp-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiϕNRp-gmix,λmii=1q is nonsingular.Proof.
The proof can be done by following the similar arguments as in Theorem18.To close this subsection, we say a few words about the complexity of the proposed neural network. Since SOCQP can be transformed into an SOCCVI problem, we only take SOCCVI as an example to illustrate the complexity of the proposed neural network model. In light of the main ideas for constructing neural network (see [25] for details), we establish a specific first-order ordinary differential equation, i.e., an artificial neural network. More specifically, based on the gradient of the objective function, we employ the neural network for solving the KKT system (17) of SOCCVI with the differential equation (40), where ρ>0 is a time scaling factor. In fact, if τ=ρt, then dz(t)/dt=ρdz(τ)/dτ. Hence, it follows from (40) that dz(τ)/dτ=-∇Ψ(u). In view of this, for simplicity and convenience, we set ρ=1 in this paper. Indeed, the dynamic system (40) can be realized by an architecture with the two discrete-type classes of SOC complementarity functions ϕD-FBp and ϕNRp shown in Figure 13. Moreover, the architecture of this artificial neural network is categorized as a “recurrent” neural network according to the classifications of artificial neural networks as in [25]. The circuit for (40) requires n+l+m integrators, n processors for F(x), m processors for g(x), mn processors for ∇g(x), ln processors for ∇h(x), (l2+m2+m+l)n processors for ∇xL(x,μ,λ), 1 processor for ϕD-FBp and ϕNRp, n processors for ∇xϕD-FBp and ∇xϕNRp, m processors for ∇λϕD-FBp and ∇λϕNRp, n2+4ln+3mn+m2+m connection weights, and some summers.Figure 13
Block diagram of the proposed neural network withϕ (ϕ is ϕD-FBp or ϕNRp).


### 4.2. Numerical Experiments

In this subsection, to demonstrate effectiveness of the proposed neural networks, some illustrative SOCCVI problems are tested. The numerical implementation is coded by Matlab 2014b and the ordinary differential equation solver adopted isode23, which uses Runge-Kutta (2;3) formula. In the subsequent tests, the parameter ρ in neural networks of Example 22 is set to be 10 and others are set to be 1.Example 20.
Consider the SOCCVI problem where(64)Fx=2x1-4,ex1-1,2x3-4,-sin⁡x4T,C=x∈R5∣-gx=x∈K5.This problem has an approximate solutionx∗=[2,0,1.3333,0,0]T. Figures 14 and 16 show the transient behaviors of Example 20 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. Figure 15 depicts the convergence comparison of the neural network model using ϕD-FBp function with different values of p=3,5,7. From this, we see that the performance of p=3 is significantly better than in other cases. Figure 17 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function. Again, when p=3, the neural network based on ϕNRp function produces fast decrease of norm of error.Figure 14
Transient behavior of neural network withϕD-FBp function (p=3) in Example 20.Figure 15
Convergence comparison ofϕD-FBp function with different p value for Example 20.Figure 16
Transient behavior of the neural network withϕNRp function (p=3) in Example 20.Figure 17
Convergence comparison ofϕNRp function with different p value for Example 20.Example 21.
Consider the problem where(65)minx∈C⁡fx=2x1-32+sin⁡x1-3sin⁡x2+2x22+x32,C=x∈R5∣-gx=x∈K5.This problem has an approximate solutionx∗=[3,0,0]T. In light of the analysis of previous section, this CSOCP in Example 21 can be transformed into an SOCCVI. We use the proposed neural networks to solve the problem with the trajectory obtained by it shown in Figures 18–21.Figure 18
Transient behavior of neural network withϕD-FBp function (p=3) in Example 21.Figure 19
Convergence comparison ofϕD-FBp function with different p value for Example 21.Figure 20
Transient behavior of the neural network withϕNRp function (p=3) in Example 21.Figure 21
Convergence comparison ofϕNRp functions with different p value for Example 21.Figures18 and 20 show the transient behaviors of Example 21 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. From Figure 19, we see the convergence comparison of the neural network model using ϕD-FBp function with p=3,5,7. There is no significant difference when perturbing the values of p. Figure 21 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function.Example 22.
We consider the following SOCCVI problem:(66)Dx,y-x≥0,∀y∈C where(67)C=x∈R10∣Ax-a=0,Bx-b⪯0,D is an 10×10 symmetric matrix and(68)D=Dij10×10,whereDij=2,i=j1,i-j=10,otherwise,A=[I5×505×5]5×5, B=[05×5I5×5]5×10, a=05×5, b=[1,0,1,0,0]T∈K2×K3. Clearly, A and B are full row rank and rank⁡([ATBT])=10.The problem has an solutionx∗=[0,0,0,0,0,0,0,0,0,0]T. It can be verified that the Lagrangian function for this example is(69)Lx,μ,λ=Dx+ATμ+BTλ. Note that ∇xL(x,μ,λ) is positive definite.Figures22 and 24 show the transient behaviors of Example 22 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. Figures 23 and 25 depict the influence of the parameter p on the value of norm of error for neural network model based on ϕD-FBp and ϕNRp functions. We see that the influence of perturbing parameter p is tiny.Figure 22
Transient behavior of neural network withϕD-FBp function (p=3) in Example 22.Figure 23
Convergence comparison ofϕD-FBp functions with different p value for Example 22.Figure 24
Transient behavior of the neural network withϕNRp function (p=3) in Example 22.Figure 25
Convergence comparison ofϕNRp functions with different p value for Example 22.We summarize some observations based on the above experiments. First, we provide the simulation diagrams of convergence comparison ofϕD-FBp and ϕNRp functions with different p value for 6 examples. It can be seem from the Figures 2, 4, 6, 8, 10, 12, 15, 17, and 21 that the convergence speed is the fastest when p=3. Under the same calculation time, the norm error in the case of p=3 is generally smaller than that in the cases of p=5 and p=7. But from Figures 19, 23, and 25, we find that the convergence of case p=3, p=5, and p=7 is similar. In other words, we observe that, for our examples, p=3 is basically the best choice. However, the theoretical evidence is not clear yet, which will be left for future study.


## 4.1. The Model and Stability Analysis

Letϕ(x,y) be a SOC complementarity function like ϕD-FBp and ϕNRp defined as in (24) and (25), respectively. Mimicking the arguments described as in [42], we can verify that the KKT system (17) is equivalent to the following unconstrained smooth minimization problem:(38)minΨz≔12Sz2,where z=(x,μ,λ)∈Rn+l+m and S(z) is given by(39)Sz=Lx,μ,λ-hxϕ-gm1x,λm1⋮ϕ-gmqx,λmq, with gmi(x),λmi∈Rmi. In other words, Ψ(z) is a smooth merit function for the KKT system (17). Hence, based on the above smooth minimization problem (38), it is natural to propose a neural network for solving the SOCCVI as follows:(40)dztdt=-ρ∇Ψzt,zt0=z0,where ρ>0 is a scaling factor. To prove the stability of neural network (40), we need to present some properties of Ψ(·).Proposition 9.
LetΨ:Rn+l+m→R+ be defined as in (38). Then, Ψ(z)≥0 for z=(x,μ,λ)∈Rn+l+m. Moreover, Ψ(z)=0 if and only if (x,μ,λ) solves the KKT system (17).Proof.
The proof is straightforward.Proposition 10.
LetΨ:Rn+l+m→R+ be defined as in (38). Then, the following results hold. (a)
The functionΨ is continuously differentiable everywhere with(41)∇Ψz=∇SzSz,where(42)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiϕ-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiϕ-gmix,λmii=1q.(b)
If∇S(z) is nonsingular, then for any stationary point (x,μ,λ)∈Rn+l+m of Ψ, (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.(c)
Ψ(z(t)) is nonincreasing with respect to t.Proof.
( a ) It follows from the chain rule immediately.
( b ) From ∇Ψ(z)=∇S(z)S(z) and the fact that matrix ∇S(z) is nonsingular, it is clear that ∇Ψ(z)=0 if and only if S(z)=0. Hence, we see that (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem provided (x,μ,λ)∈Rn+l+m is a stationary point of Ψ.
( c ) From the definition of Ψ(z) and (40), it is easy to verify that(43)dΨztdt=∇ΨztTdztdt=-ρ∇Ψzt2≤0,which says Ψ(z(t)) is a monotonically decreasing function with respect to t.Now, we are ready to analyze the behavior of the solution trajectory of neural network (40) and establish three kinds of stabilities for an isolated equilibrium point.Proposition 11.
(a) If(x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem, then (x,μ,λ)∈Rn+l+m is an equilibrium point of neural network (40).
(b) If∇S(z) is nonsingular and (x,μ,λ)∈Rn+l+m is an equilibrium point of (40), then (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.Proof.
( a ) From Proposition 9 and (x,μ,λ)∈Rn+l+m being a KKT triple of SOCCVI problem, it is clear that S(x,μ,λ)=0, which implies ∇Ψ(x,μ,λ)=0. Besides, by Proposition 10, we know that ∇Ψ(x,μ,λ)≠0. This shows that (x,μ,λ) is an equilibrium point of neural network (40).
( b ) It follows from (x,μ,λ)∈Rn+l+m being an equilibrium point of neural network (40) that ∇Ψ(x,μ,λ)=0. In other words, (x,μ,λ) is the stationary point of Ψ. Then, the result is a direct consequence of Proposition 10(b).Proposition 12.
(a) For any initial statez0=z(t0), there exists exactly one maximal solution z(t) with t∈[t0,τ(x0)) for the neural network (40).
(b) If the level setL(z0)=z∈Rn+l+m∣Ψ(z)≤Ψ(z0) is bounded, then τ(x0)=+∞.Proof.
( a ) Since S(·) is continuous differentiable, it says that ∇S(·) is continuous. This means ∇S(·) is bounded on a local compact neighborhood of z, which implies that ∇Ψ(z) is locally Lipschitz continuous. Thus, applying Lemma 1 leads to the desired result.
( b ) This proof is similar to the proof of Case(i) in [10, Proposition 4.2], so we omit it.Remark 13.
A natural question arises here. When are the level sets(44)LΨ,γ≔z∈Rn+l+m∣Ψz≤γ bounded for all γ∈R? For the time being, we are not able to answer this question yet. We suspect that there needs more subtle properties of F, h, and g to finish it.Next, we investigate the convergence of the solution trajectory and stability of neural network (40), which are the main results of this section.Theorem 14.
(a) Letz(t) with t∈[t0,τ(z0)) be the unique maximal solution to the neural network (40). If τ(z0)=+∞ and {z(t)} is bounded, then limt→∞⁡∇Ψ(z(t))=0.
(b) If∇S(z) is nonsingular and (x,μ,λ)∈Rn+l+m is the accumulation point of the trajectory z(t), then (x,μ,λ)∈Rn+l+m is a KKT triple of the SOCCVI problem.Proof.
With Proposition10(b) and 10(c) and Proposition 12, the arguments are exactly the same as those for [19, Corollary 4.3]. Thus, we omit them.Theorem 15.
Letz∗ be an isolated equilibrium point of the neural network (40). Then, the following results hold. (a)
z∗ is asymptotically stable and hence is also Lyapunov stable.(b)
If∇S(z) is nonsingular, then it is exponentially stable.Proof.
Again, the arguments are similar to those in [32, Theorem 3.1] and we omit them.To study the conditions for nonsingularity based onψD-FBp and ϕNRp, we need the following assumptions.Assumption 16.
( a ) The gradients {∇hjx∣j=1,…,l}∪{∇gi(x)∣i=1,…,m} are linear independent.
( b )  ∇xL(x,μ,λ) is positive definite on the null space of the gradients {∇hj(x)∣j=1,…,l}.When SOCCVI problem corresponds to the KKT conditions of a convex second-order cone program (CSOCP) problem as (14) where both h and g are linear, the above Assumption 16(b) is indeed equivalent to the well-used condition of ∇2f(x) being positive definite, e.g., [22, Corollary 1].Assumption 17.
Letα≔wmip/2 and β≔vmip/2, where wmi=gmi2+λmi2 and vmi=(gmi+λmi)2. For gmi(x),λmi∈Kmi, we have (a)
Lgmi2-LβLα-1Lgmi2Lα-1Lβ⪰0 or LαLβ-1Lgmi2Lβ-1Lα-Lgmi2⪰0;(b)
Lλmi2-LβLα-1Lλmi2Lα-1Lβ⪰0 or LαLβ-1Lλmi2Lβ-1Lα-Lλmi2⪰0.Theorem 18.
Suppose-gmi+λmi∈int⁡Kmi for i=1,2,…,p and that Assumptions 16 and 17 hold. Then, the matrix(45)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiψD-FBp-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiψD-FBp-gmix,λmii=1qis nonsingular.Proof.
We know that∇S(z) is nonsingular if and only if the following equation only has zero solution:(46)∇Szuvt=0,whereu,v,t∈Rn×Rl×Rm.To reach the conclusion, we need to prove u=0,v=0,t=0. First, plugging the components of ∇S(z) into (46), we have(47)∇xLTu-∇hxv-∇gxL-g+λLβ-1-L-gLα-1t=0(48)∇hxTu=0(49)∇gxTu+L-g+λLβ-1-LλLα-1t=0where (50)Lg+λ=diag⁡L-gm1+λm1,L-gm2+λm2,…,L-gmq+λmqLβ=diag⁡∇gsocvm1,∇gsocvm2,∇gsocvmqL-g=diag⁡L-gm1,L-gm2,…,L-gmqLα=diag⁡∇gsocwm1,∇gsocwm2,∇gsocwmqLλ=diag⁡Lλm1,Lλm2,…,Lλmqα=diag⁡αm1,αm2,…,αmqαmi=vmip/2=-gmi+λmip,i=1,2,…,qβ=diag⁡βm1,βm2,…,βmqβmi=wmip/2=-gmi2+λmi2p/2,i=1,2,…,q From (47) and (48), we see that(51)uT∇xLTu-uT∇gxL-g+λLβ-1-L-gLα-1t=0,while from (49), we have(52)tTL-g+λLβ-1-L-gLα-1T∇gxTu+tTL-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1t=0Next, we will claim that(53)tTL-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1t≥0To see this, we note that (54)L-g+λLβ-1-L-gLα-1TL-g+λLβ-1-LλLα-1=diag⁡L-gm1+λm1Lβm1-1-L-gm1Lαm1-1TL-gm1+λm1Lβm1-1-Lλm1Lαm1-1,…,L-gmq+λmqLβmq-1-L-gmqLαmq-1TL-gmq+λmqLβmq-1-LλmqLαmq-1. In view of this, to prove inequality (53), it suffices to show that(55)tiTL-gmi+λmiLβmi-1-L-gmiLαmi-1TL-gmi+λm1Lβmi-1-LλmiLαmi-1ti≥0,for i=1,2,…,q. For convenience, we denote X≔-gmi, Y≔λmi, A≔αmi, and B≔βmi. With these notations, we have (56)L-gmi+λmiLβmi-1-L-gmiLαmi-1TL-gmi+λm1Lβmi-1-LλmiLαmi-1=LX+YLB-1-LXLA-1TLX+YLB-1-LYLA-1=LB-1LX+Y-LBLA-1LXLX+Y-LYLA-1LBLB-1, which says that it is enough to show M≔(LX+Y-LBLA-1LX)(LX+Y-LYLA-1LB) is semipositive definite in order to prove inequality (55). To this end, we compute that(57)12LX+Y-LBLA-1LXLX+Y-LYLA-1LB+LX+Y-LBLA-1LXLX+Y-LYLA-1LBT=122LX+Y2-LX+Y2LA-1LB-LBLA-1LX+Y2+LBLA-1LXLY+LYLXLA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX+Y2-LBLA-1LX+Y2LA-1LB+LBLA-1LXLY+LYLXLA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX+Y2-LBLA-1LX2+LY2LA-1LB=12I-LBLA-1LX+Y2I-LA-1LB+LX2-LBLA-1LX2LA-1LB+LY2-LBLA-1LY2LA-1LB+LXLY+LYLX.It can be verified that LX+Y2 and LXLY+LYLX are positive semidefinite. Then, from Assumption 17 and (57), we conclude that M≔(LX+Y-LBLA-1LX)(LX+Y-LYLA-1LB) is semipositive definite; and hence inequality (55) holds. Thus, inequality (53) also holds accordingly. Now, tt follows from (51), (52), and (53) that uT(∇xL)Tu=0 which implies that u=0. Then, (47) and (48) become(58)∇hxv+∇gxL-g+λLβ-1-L-gLα-1t=0(59)L-g+λLβ-1-LλLα-1t=0In light of Assumption 16(a) and (58), we know(60)v=0,L-g+λLβ-1-L-gLα-1t=0.Combining (59) and (60) together, it is clear to obtain(61)L-gLα-1t=LλLα-1tNote that -g and λ are strict complementary. Hence, it yields t=0. In summary, from equation (46), we deduce u=v=t=0, which says the matrix ∇S(z) is nonsingular.Theorem 19.
Suppose Assumption16 holds and(62)∇gmiϕNRp-gmix,λmi·∇λmiϕNRp-gmix,λmi⪰0.Then, the matrix(63)∇Sz=∇xLx,μ,λT-∇hx-∇gxdiag⁡∇gmiϕNRp-gmix,λmii=1q∇hxT00∇gxT0diag⁡∇λmiϕNRp-gmix,λmii=1q is nonsingular.Proof.
The proof can be done by following the similar arguments as in Theorem18.To close this subsection, we say a few words about the complexity of the proposed neural network. Since SOCQP can be transformed into an SOCCVI problem, we only take SOCCVI as an example to illustrate the complexity of the proposed neural network model. In light of the main ideas for constructing neural network (see [25] for details), we establish a specific first-order ordinary differential equation, i.e., an artificial neural network. More specifically, based on the gradient of the objective function, we employ the neural network for solving the KKT system (17) of SOCCVI with the differential equation (40), where ρ>0 is a time scaling factor. In fact, if τ=ρt, then dz(t)/dt=ρdz(τ)/dτ. Hence, it follows from (40) that dz(τ)/dτ=-∇Ψ(u). In view of this, for simplicity and convenience, we set ρ=1 in this paper. Indeed, the dynamic system (40) can be realized by an architecture with the two discrete-type classes of SOC complementarity functions ϕD-FBp and ϕNRp shown in Figure 13. Moreover, the architecture of this artificial neural network is categorized as a “recurrent” neural network according to the classifications of artificial neural networks as in [25]. The circuit for (40) requires n+l+m integrators, n processors for F(x), m processors for g(x), mn processors for ∇g(x), ln processors for ∇h(x), (l2+m2+m+l)n processors for ∇xL(x,μ,λ), 1 processor for ϕD-FBp and ϕNRp, n processors for ∇xϕD-FBp and ∇xϕNRp, m processors for ∇λϕD-FBp and ∇λϕNRp, n2+4ln+3mn+m2+m connection weights, and some summers.Figure 13
Block diagram of the proposed neural network withϕ (ϕ is ϕD-FBp or ϕNRp).


## 4.2. Numerical Experiments

In this subsection, to demonstrate effectiveness of the proposed neural networks, some illustrative SOCCVI problems are tested. The numerical implementation is coded by Matlab 2014b and the ordinary differential equation solver adopted isode23, which uses Runge-Kutta (2;3) formula. In the subsequent tests, the parameter ρ in neural networks of Example 22 is set to be 10 and others are set to be 1.Example 20.
Consider the SOCCVI problem where(64)Fx=2x1-4,ex1-1,2x3-4,-sin⁡x4T,C=x∈R5∣-gx=x∈K5.This problem has an approximate solutionx∗=[2,0,1.3333,0,0]T. Figures 14 and 16 show the transient behaviors of Example 20 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. Figure 15 depicts the convergence comparison of the neural network model using ϕD-FBp function with different values of p=3,5,7. From this, we see that the performance of p=3 is significantly better than in other cases. Figure 17 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function. Again, when p=3, the neural network based on ϕNRp function produces fast decrease of norm of error.Figure 14
Transient behavior of neural network withϕD-FBp function (p=3) in Example 20.Figure 15
Convergence comparison ofϕD-FBp function with different p value for Example 20.Figure 16
Transient behavior of the neural network withϕNRp function (p=3) in Example 20.Figure 17
Convergence comparison ofϕNRp function with different p value for Example 20.Example 21.
Consider the problem where(65)minx∈C⁡fx=2x1-32+sin⁡x1-3sin⁡x2+2x22+x32,C=x∈R5∣-gx=x∈K5.This problem has an approximate solutionx∗=[3,0,0]T. In light of the analysis of previous section, this CSOCP in Example 21 can be transformed into an SOCCVI. We use the proposed neural networks to solve the problem with the trajectory obtained by it shown in Figures 18–21.Figure 18
Transient behavior of neural network withϕD-FBp function (p=3) in Example 21.Figure 19
Convergence comparison ofϕD-FBp function with different p value for Example 21.Figure 20
Transient behavior of the neural network withϕNRp function (p=3) in Example 21.Figure 21
Convergence comparison ofϕNRp functions with different p value for Example 21.Figures18 and 20 show the transient behaviors of Example 21 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. From Figure 19, we see the convergence comparison of the neural network model using ϕD-FBp function with p=3,5,7. There is no significant difference when perturbing the values of p. Figure 21 depicts the influence of the parameter p on the value of norm of error for neural network model using ϕNRp function.Example 22.
We consider the following SOCCVI problem:(66)Dx,y-x≥0,∀y∈C where(67)C=x∈R10∣Ax-a=0,Bx-b⪯0,D is an 10×10 symmetric matrix and(68)D=Dij10×10,whereDij=2,i=j1,i-j=10,otherwise,A=[I5×505×5]5×5, B=[05×5I5×5]5×10, a=05×5, b=[1,0,1,0,0]T∈K2×K3. Clearly, A and B are full row rank and rank⁡([ATBT])=10.The problem has an solutionx∗=[0,0,0,0,0,0,0,0,0,0]T. It can be verified that the Lagrangian function for this example is(69)Lx,μ,λ=Dx+ATμ+BTλ. Note that ∇xL(x,μ,λ) is positive definite.Figures22 and 24 show the transient behaviors of Example 22 for neural network model (40) based on ϕD-FBp and ϕNRp with initial states x0=[0,0,0,0,0]T, respectively. Figures 23 and 25 depict the influence of the parameter p on the value of norm of error for neural network model based on ϕD-FBp and ϕNRp functions. We see that the influence of perturbing parameter p is tiny.Figure 22
Transient behavior of neural network withϕD-FBp function (p=3) in Example 22.Figure 23
Convergence comparison ofϕD-FBp functions with different p value for Example 22.Figure 24
Transient behavior of the neural network withϕNRp function (p=3) in Example 22.Figure 25
Convergence comparison ofϕNRp functions with different p value for Example 22.We summarize some observations based on the above experiments. First, we provide the simulation diagrams of convergence comparison ofϕD-FBp and ϕNRp functions with different p value for 6 examples. It can be seem from the Figures 2, 4, 6, 8, 10, 12, 15, 17, and 21 that the convergence speed is the fastest when p=3. Under the same calculation time, the norm error in the case of p=3 is generally smaller than that in the cases of p=5 and p=7. But from Figures 19, 23, and 25, we find that the convergence of case p=3, p=5, and p=7 is similar. In other words, we observe that, for our examples, p=3 is basically the best choice. However, the theoretical evidence is not clear yet, which will be left for future study.


## 5. Conclusions

In this paper, we propose a unified neural network model for solving two types of second-order cone optimization problems. We implement the neural network model with two classes of SOC complementarity functionsϕD-FBp and ϕNRp, which are freshly discovered. Overall, when p is larger, the performance of the neural network based on ϕD-FBp and ϕNRp gets poorer. This suggests that the parameter p=3 is the best choice to work with the neural network no matter ϕD-FBp or ϕNRp is employed. On the other hand, we observe that the neural network using the family of ϕD-FBp functions performs better than the one using ϕNRp, in general. This is a very interesting phenomenon and new discovery, which is different from the observation in [31]. Note that in [31], for standard SOCP (second-order cone program), it was observed that the neural network based on the NR function has better performance than the one based on the FB function in most cases (except for some oscillating cases). Our numerical experiments show that for SOCQP and SOCCVI, the family of ϕD-FBp (a variant of FB function) may be considered to be employed in the neural network approach.Another point that we want to clarify is the significance of this paper. By using two new SOC complementarity functions, this paper can be viewed as a follow-up of our previous works (see [31, 32]). From the natural feature of neural network, we pay more attention to the convergence path. This kind of convergence path varies with different SOC complementarity functions. In engineering, an important application of neural network is the design of robot arm. Therefore, compared with previous works, we achieve different convergence paths and the convergence paths are very good. In addition, we provide more detailed graphs regarding error analysis in this paper.



---

*Source: 4545064-2019-02-14.xml*