# A One-Layer Recurrent Neural Network for Solving Pseudoconvex Optimization with Box Set Constraints

Authors: Huaiqin Wu; Rong Yao; Ruoxia Li; Xiaowei Zhang
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

A one-layer recurrent neural network is developed to solve pseudoconvex optimization with box
constraints. Compared with the existing neural networks for solving pseudoconvex optimization, the proposed neural
network has a wider domain for implementation. Based on Lyapunov stable theory, the proposed neural network is
proved to be stable in the sense of Lyapunov. By applying Clarke’s nonsmooth analysis technique, the finite-time state
convergence to the feasible region defined by the constraint conditions is also addressed. Illustrative examples further
show the correctness of the theoretical results.


---

## Body

## 1. Introduction

It is well known that nonlinear optimization problems arise in a broad variety of scientific and engineering applications including optimal control, structure design, image and signal progress, and robot control. Most of nonlinear programming problems have a time-varying nature; they have to be solved in real time. One promising approach to solve nonlinear programming problems in real time is to employ recurrent neural networks based on circuit implementation.In the past two decades, neural networks for optimization are studied massively and many good results are obtained in the literature; see [1–19] and references therein. In particular, Liang and Wang developed a recurrent neural network for solving nonlinear optimization with a continuously differentiable objective function and bound constraints in [4]. A projection neural network was proposed for solving nondifferentiable nonlinear programming problems by Xia et al., in [20]. In [9, 19], Xue and Bian developed a subgradient-based neural network for solving nonsmooth convex or nonconvex optimization problems with a nonsmooth convex or nonconvex objective function.It should be noticed tha, many nonlinear programming problems can be formulated as nonconvex optimization problems, and among nonconvex programming, as a special case, pseudoconvex programmings are found to be more prevalent than other nonconvex programming. Pseudoconvex optimization problem has many applications in practice, such as fractional programming, computer vision, and production planning. Very recently, Liu et al. presented a one-layer recurrent neural network for solving pseudoconvex optimization subject to linear equality in [1]; Hu and Wang proposed a recurrent neural network for solving pseudoconvex variational inequalities in [10]. Qin et al. proposed a new one-layer recurrent neural network for nonsmooth pseudoconvex optimization in [21].Motivated by the works above, our objective in this paper is to develop a one-layer recurrent neural network for solving pseudoconvex optimization problem subject to a box set. The proposed network model is an improvement of the neural network model presented in [10]. To the best of our knowledge, there are few works treating of the pseudoconvex optimization problem with a box set constraint.For convenience, some notations are introduced as follows.ℝ denotes the set of real numbers, ℝn denotes the n-dimensional Euclidean space, and ℝm×n denotes the set of all m×n real matrices. For any matrix A, A>0(A<0) means that A is a positive definite (negative definite). A-1 denotes the inverse of A. AT denotes the transpose of A. λmax⁡(A) and λmin⁡(A) denote the maximum and minimum eigenvalue of A, respectively. Given the vectors x=(x1,…,xn)T,y=(y1,…,yn)T∈ℝn, ∥x∥=(∑i=1n‍xi2)1/2, xTy=∑i=1n‍xiyi. ∥A∥ denotes the 2-norm of A; that is, ∥A∥=λ(ATA), where λ(ATA) denotes the spectral radius of ATA. x˙(t) denotes the derivative of x(t).Given a setC⊂ℝn, K[C] denotes the closure of the convex hull of C.LetV:ℝn→ℝ be a locally Lipschitz continuous function. Clarke’s generalized gradient of V at x is defined by
(1)∂V(x)=K[{lim⁡i→∞∇V(xi):lim⁡i→∞xi=x,hhhhhhxi∈ℝn∖ΩV∪ℳlim⁡i→∞}],
where ΩV⊂ℝn is the set of Lebesgue measure zero, ∇V does not exist, and ℳ⊂ℝn is an arbitrary set with measure zero. The set-valued map G(·) is said to have a closed (convex, compact) image if for each x∈E, G(x) is closed (convex, compact).The remainder of this paper is organized as follows. In Section2, the related preliminary knowledge are given, and the problem formulation and the neural network model are described. In Section 3, the stability in the sense of Lyapunov and finite-time convergence of the proposed neural network is proved. In Section 4, illustrative examples are given to show the effectiveness and the performance of the proposed neural network. Some conclusions are drawn in Section 5.


## 2. Model Description and Preliminaries

In this section, a one-layer recurrent neural network model is developed to solve pseudoconvex optimization with box constraints. Some definitions and properties concerning the set-valued map and nonsmooth analysis are also introduced.Definition 1 (set-valued map).
Suppose that to each pointx of a set E⊆Rn, there corresponds a nonempty set F(x)⊂Rn. Then x→F(x) is said to be a set-valued map from EtoRn.Definition 2 (locally Lipschitz function).
A functionϕ: Rn→R is called Lipschitz near x0 if and only if there exist ɛ, ϵ>0, such that for any x1,x2∈B(x0,ϵ), satisfying ∥ϕ(x1)-ϕ(x2)∥≤ɛ∥x1-x2∥, where B(x0,ϵ)={x:∥x-x0∥<ϵ}. The function ϕ: Rn→R is said to be locally Lipschitz in Rn if it is Lipschitz near any point x∈Rn.Definition 3 (regularity).
A functionϕ: Rn→R, which is locally Lipschitz near x∈Rn, is said to be regular at x if there exists the one-sided directional derivative for any direction v∈Rn which is given by ϕ′(x;v)=lim⁡ξ→0+(ϕ(x+ξ×v)-ϕ(x))/ξ, and we have ϕ0(x;v)=ϕ′(x;v). The function ϕ is said to be regular in Rn if it is regular for any x∈Rn.Definition 4.
A regular functionf:Rn→R is said to be pseudoconvex on a set Ω, if, for for all x,y∈Ω,x≠y,γ(x)∈∂f(x), we have
(2)γ(x)T(y-x)≥0⟹f(y)≥f(x).Definition 5.
A functionF:Rn→Rn is said to be pseudomonotone on a set Ω, if, for all x,x′∈Ω,x≠x′, we have
(3)F(x)T(x′-x)≥0⟹F(x′)T(x′-x)≥0.
Consider the following optimization problem with box set constraint:(4)min⁡f(x),subjecttod≤Bx≤h,
where x,d,h∈Rn, and B∈Rn×n is nonsingular.
SubstitutingBx with z, then the problem (4) can be transformed into the following problem:
(5)min⁡f(B-1z),subjecttod≤z≤h.
Let
(6)D(z)=∑i=1n‍d(zi),i=1,2,…,n,
where d(zi) is defined as
(7)d(zi)={zi-hi,zi≥hi,0,di<zi<hi,di-zi,zi≤di.
Obviously, d(zi)≥0 and D(z)≥0.
Throughout this paper, the following assumptions on the optimization problem (4) are made.(A1)
The objective functionf(x) of the problem (4) is pseudoconvex and regular and locally Lipschitz continuous.(A2)
∂f(x) is bounded; that is,
(8)sup⁡γ(x)∈∂f(x)∥γ(x)∥≤lf,where lf>0 is a constant.
In the following, we develop a one-layer recurrent neural network for solving the problem (4). The dynamic equation of the proposed neural network model is described by differential inclusion system:
(9)dzdt∈-∂f(B-1z)-μK[g[d,h]](z),
where μ is a nonnegative constant, ∂f(B-1z)=B-1∂f(x), and g[d,h] is a discontinuous function with its components defined as
(10)gi[d,h](s)={1,s>hi,0,di<s<hi,-1,s<di,K[gi[d,h]](s)={1,s>hi,[0,1],s=hi,0,di<s<hi,[-1,0],s=di,-1,s<di.
Architecture of the proposed neural network system model (9) is depicted in Figure 1.Figure 1
Architecture of the neural network model (9).Definition 6.
z
¯
∈
R
n is said to be an equilibrium point of the differential inclusion system (9) if
(11)0∈B-1∂f(x¯)+μK[g[d,h]](z¯);
that is, there exist
(12)γ¯∈∂f(x¯),ξ¯∈K[g[d,h]](z¯),
such that
(13)B-1γ¯+μξ¯=0,
where x¯=B-1z¯.Definition 7.
A functionz(·):[0,T]→Rn is said to be a solution of the system (9) with initial condition z(0)=z0, if z(·) is absolutely continuous on [0,T], and for almost t∈[0,T],
(14)dz(t)dt∈-∂f(B-1z(t))-μK[g[d,h]](z(t)).
Equivalently, there exists measurable functions γ(x)∈∂f(x), ξ(z)∈K[g[d,h]](z) such that
(15)z˙(t)=-B-1γ(B-1z(t))-μξ(z(t)).Definition 8.
Suppose thatB⊂Rn is a nonempty closed convex set. The normal cone to the set B at z∈B is defined as NB(z)={v∈Rn:vT(z-y)≥0,forally∈B}.Lemma 9 (see [22]).
Ifϕi:Rn→R,i=1,2,…,m, is regular at x, then ∂(∑i=1mϕi(x))=∑i=1m(∂ϕi(x)).Lemma 10 (see [22]).
IfV:Rn→R is a regular function at x and x(·):R→Rn is differentiable at t and Lipschitz near t, then dV(x(t))/dt=〈ξ,x˙(t)〉, for all ξ∈∂V(x(t)).Lemma 11 (see [22]).
IfB1,B2⊂Rn are closed convex sets and satisfy 0∈int⁡(B1-B2), then for any z∈B1⋂‍B2, NB1⋂‍B2(z)=NB1(z)+NB2(z).Lemma 12 (see [22]).
Iff is locally Lipschitz near z and attains a minimum over Ω at z, then 0∈∂f(z)+NΩ(z).SetΩ={z∈Rn:d≤z≤h}. Let s∈int⁡(Ω); then there exists a constant r>0, such that Ω⊆B(s,r), where int⁡(·) denotes the interior of the set Ω, B(s,r)={z∈Rn:∥z-s∥≤r}. It is easy to verify the following lemma.Lemma 13.
For anyz∈Rn∖Ω and ξ∈K[g[d,h]](z), (z-s)Tξ>ω, where ω=min⁡1≤i≤n{hi-si,si-di}, and si is the ith element of s∈int⁡Ω.


## 3. Main Results

In this section, the main results concerned with the convergence and optimality conditions of the proposed neural network are addressed.Theorem 14.
Suppose that the assumptions(A1) and (A2) hold. Let z0∈B(s,r). If μ>(rlf/ω)∥B-1∥, then the solution z(t) of the network system (9) with initial condition z(0)=z0 satisfies z(t)∈B(s,r).Proof.
Set(16)ρ(t)=12∥z(t)-s∥2.
By Lemma 10 and (15), evaluating the derivation of ρ(t) along the trajectory of the system (9) gives(17)dρ(t)dt=z˙(t)T(z(t)-s)=[-B-1γ(x(t))-μξ(z(t))]T(z(t)-s)=-[B-1γ(x(t))]T(z(t)-s)-μξ(z(t))T(z(t)-s)≤∥B-1∥∥z(t)-s∥∥γ(x(t))∥-μ(z(t)-s)Tξ(z(t)).
If z(t)∈Ω, it follows directly that z(t)∈B(s,r). If z(t)∈B(s,r)∖Ω, according to Lemma 13, one gets that (z(t)-s)Tξ(z(t))>ω. Thus, we have
(18)dρ(t)dt<∥B-1∥∥z(t)-s∥∥γ(x(t))∥-μω≤rlf∥B-1∥-μω.
If μ>(rlf/ω)∥B-1∥, then dρ(t)/dt<0; this means that z(t)∈B(s,r). If not so, the state z(t) leaves B(s,r) at time t1, and when t=t1, we have ∥z(t1)-s∥=r. This implies that (dρ(t)/dt)|t=t1≥0, which is the contradiction.
As a result, ifμ>(rlf/ω)∥B-1∥, for any z0∈B(s,r), the state z(t) of the network system (9) with initial condition z(0)=z0 satisfies z(t)∈B(s,r). This completes the proof.Theorem 15.
Suppose that assumptions(A1) and (A2) hold. If μ>lf∥B-1∥, then the solution of neural network system (9) with initial condition z(0)=z0∈B(s,r) converges to the feasible region Ω in finite time T, where T=t*=D(z(0))/n(μ-lf∥B-1∥), and stays thereafter.Proof.
According to the definition ofD(z), D(z) is a convex function in Rn. By Lemma 10, it follows that
(19)dD(z)dt=ζT(t)dz(t)dt,∀ζ(t)∈∂D(z).
Noting that ∂D(z)=K[g[d,h]](z), we can obtain by (19) that for all z∈B(s,r)∖Ω, ∃γ(x)∈∂f(x), and ξ(t)∈K[g[d,h]](z) such that
(20)dD(z)dt=ξT(t)z˙(t)=ξT(t)(-B-1γ(x)-μξ(t))=-ξT(t)B-1γ(x)-μ∥ξ(t)∥2≤∥ξ(t)∥∥B-1∥∥γ(x)∥-μ∥ξ(t)∥2≤∥ξ(t)∥[∥B-1∥∥γ(x)∥-μ∥ξ(t)∥].
Since z∈Rn∖Ω and ξ(t)∈K[g[d,h]](z), there at least is one of the components of ξ(t) which is -1 or 1. So ∥ξ(t)∥≥1. Noting that ∥ξ(t)∥≤n, we have
(21)dD(z)dt≤n[lf∥B-1∥-μ].
Let α=n(μ-lf∥B-1∥). If μ>lf∥B-1∥, then α>0, and
(22)dD(z)dt≤-α.
Integrating (22) from 0 to t, we can obtain
(23)D(z(t))≤D(z(0))-αt.
Let t*=(1/α)D(z(0)). By (23), D(z(t*))=0; that is, zi(t*)=hi or zi(t*)=di. This shows that the state trajectory of neural network (9) with initial condition z(0)=z0 reaches Ω in finite time T=t*=D(z(0))/n(μ-lf∥B-1∥).
Next, we prove that whent≥t*, the trajectory stays in Ω after reaching Ω. If this is not true, then there exists t1>t* such that the trajectory leaves Ω at t1, and there exists t2>t1 such that for t∈(t1,t2), z(t)∈Rn∖Ω.
By integrating (22) from t1 to t2, it follows that
(24)∫t1t2‍dD(z(t))dtdt=D(z(t2))-D(z(t1))≤-α(t2-t1)<0.
Due to D(z(t1))=0, D(z(t2))<0. By the definition of D(z(t)), D(z(t))≥0 for any t∈[0,∞], which contradicts the result above. The proof is completed.Theorem 16.
Suppose that assumptions(A1) and (A2) hold. If μ>max⁡{lf∥B-1∥,(rlf/ω)∥B-1∥}, then the equilibrium point of the neural network system (9) is an optimal solution of the problem (4) and vice versa.Proof.
Denotez* as an equilibrium point of the neural network system (9); then there exist γ*∈∂f(x*), ξ*∈K[g[d,h]](z*) such that
(25)B-1γ*+μξ*=0,
where γ*=B-1z*. By Theorem 15, z*∈Ω; hence, ξ*=0. By (25), γ*=0. We can get the following projection formulation:
(26)z*=ϕ[d,h](z*-B-1γ*),
where ϕ[d,h]=(ϕ[d1,h1](y1),…,ϕ[dn,hn](yn)) with ϕ[di,hi](yi), i=1,…,n, defined as
(27)ϕ[di,hi](yi)={hi,yi≥hi,yi,di<yi<hi,di,yi≤di.
By the well-known projection theorem [17], (26) is equivalent to the following variational inequality:
(28)(B-1γ*)T(z-z*)≥0,∀z∈Ω.
Since f(x) is pseudoconvex, f(B-1z) is pseudoconvex on Ω. By (28), we can obtain that f(B-1z)≥f(B-1z*). This shows that z* is a minimum point of f(B-1z) over Ω.
Next, we prove the reverse side. Denotez* as an optimal solution of the problem, then z*∈[d,h]. Since z* is a minimum point of f(B-1z) over the feasible region Ω, according to Lemma 12, it follows that
(29)0∈∂f(B-1z*)+NΩ(z*).
From (29), it follows that there exist η*∈NΩ(z*), B-1γ*=-η*∈B-1∂f(x*), and ∥η*∥≤lf∥B-1∥. Noting that NΩ(z*)={vξ*:v≥0,ξ*∈K[g[d,h]](z*), and at least one ξi*is1 or -1}, there exist β≥0 and ξ*∈K[g[d,h]](z*) such that βξ*∈NΩ(z*), and η*=βξ*.
In the following, we prove thatη*∈μK[g[d,h]](z*). We say that β≤μ. If not, then β>μ. Since (z*-s)Tξ*=∑i=1n(zi*-si)ξi≥ω, we have
(30)(z*-s)Tη*=β(z*-s)Tξ*≥ωβ≥μω.
Thus ∥η*∥≥μω/∥z*-s∥>μω/r. By the condition of Theorem 16, μ>max⁡(lf∥B-1∥,(rlf/ω)∥B-1∥). Hence, ∥η∥>lf∥B-1∥, which contradicts with ∥η∥2≤lf∥B-1∥. This implies that βξ*∈μK[g[d,h]](z*); that is, 0∈∂F(z*)+μK[g[d,h]](z*) which means that z* is the equilibrium point of neural network system (9). This completes the proof.Theorem 17.
Suppose that assumptions(A1) and (A2) hold. If μ>max⁡{lf∥B-1∥,(rlf/ω)∥B-1∥}, then the equilibrium point of the neural network system (9) is stable in the sense of Lyapunov.Proof.
Denotez* as an equilibrium point of the neural network system (9); that is,
(31)0∈∂F(z*)+μK[g[d,h](z*)].
By Theorem 16, we get that z* is an optimal solution of the problem (4); that is, f(B-1z)≥f(B-1z*),z∈Ω. By Theorem 15, the trajectory z(t) with initial condition z(0)=z0∈B(s,r) converges to the feasible region Ω in finite time T=t*=D(z(0))/n(μ-lf∥B-1∥) and will remain in Ω forever. That is, for all t≥t*,z(t)∈Ω. Let
(32)V1(z)=f(B-1z)+μD(z).
Since z* is a minimum point of f(B-1z) on Ω, we can get that V1(z)≥V1(z*), for all z∈Ω.
Consider the following Lyapunov function:(33)V(z)=V1(z)-V1(z*)+12∥z-z*∥2.
Obviously, from (33), V(z)≥(1/2)∥z-z*∥2, and
(34)∂V(z)=∂V1(z)+z-z*.
Evaluating the derivation of V along the trajectory of the system (9) gives
(35)dV(z(t))dt=ξT(t)dz(t)dt,∀ξ(t)∈∂V(z(t)).
Since z˙(t)∈-∂V1(z(t)), by (34), we can set ξ(t)=-z˙(t)+z-z*. Hence,
(36)dV(z(t))dt=(-z˙(t)+z-z*)Tz˙(t)=-∥z˙(t)∥2+(z-z*)z˙(t)≤sup⁡θ∈∂V1(z)(-∥θ∥2)+sup⁡θ∈∂V1(z)[-(z-z*)Tθ]=-inf⁡θ∈∂V1(z)(∥θ∥2)-inf⁡θ∈∂V1(z)(z-z*)Tθ.
For any θ∈∂V1(z), there exists γ∈∂f(x),ξ∈K[g[d,h]](z) such that θ=B-1γ+μξ. Since f(x) is pseudoconvex on Ω, ∂f(x) is pseudomonotone on Ω. From the proof of Theorem 16, for any z∈Ω, (z-z*)TB-1γ≥0. By the definition of g[d,h](z), (z-z*)Tξ≥0. Hence, (z-z*)Tθ≥0. This implies that
(37)dV(z(t))dt≤-inf⁡θ∈∂V1(z)∥θ∥2≤0.Equation (37) shows that the neural network system (9) is stable in the sense of Lyapunov. The proof is complete.


## 4. Numerical Examples

In this section, two examples will be given to illustrate the effectiveness of the proposed approach for solving the pseudoconvex optimization problem.Example 1.
Consider the quadratic fractional optimization problem:(38)min⁡f(x)=xTQx+aTx+a0cTx+c0,subjecttod≤Bx≤h,
where Q is an n×n matrix, a,c∈Rn, and a0,c0∈R. Here, we choose n=3,
(39)Q=(5-12-15-12-13),B=(21111-161111-92-10),a=(1-2-2),c=(21-1),d=(-2-1-6),h=(226),a0=-2,c0=4.It is easily verified thatQ is symmetric and positive definite and consequently is pseudoconvex on Ω={d≤Bx≤h}. The proposed neural network in (9) is capable of solving this problem. Obviously, neural network (9) associated to (38) can be described as
(40)Bx˙(t)=-B-1∇f(x)-μξ(Bx(t)),
where
(41)∇f(x)=2Qx+acTx+c0-xTQx+aTx+a0(cTx+c0)2c,ξ(Bx(t))∈K[g[d,h]](Bx(t)).Letx^={-1,-1,-1}, s=Bx^∈int⁡(Ω), then we have ω=3. Moreover, the restricted region [d,h]⊂B(s,r) with r=23. An upper bound of ∂f(x) is estimated as lf=3. Then the designed parameter μ is estimated as μ>10.32. Let μ=11 in the simulation.We have simulated the dynamical behavior of the neural network by using the mathematical software whenμ=11. Figures 2, 3, and 4 display the state trajectories of this neural network with different initial values, which shows that the state variables converge to the feasible region in finite time. This is in accordance with the conclusion of Theorem 15. Meanwhile, it can be seen that the trajectory is stable in the sense of Lyapunov.Figure 2
Time-domain behavior of the state variablesx1, x2, and x3 with initial point x0=(0,0,0.3).Figure 3
Time-domain behavior of the state variablesx1, x2, and x3 with initial point x0=(-0.2,0,0.4).Figure 4
Time-domain behavior of the state variablesx1, x2, and x3 with initial point x0=(0.1,0,0.3).Example 2.
Consider the following pseudoconvex optimization:(42)min⁡f(x)=(x1-x3)2+(x2-x3)2,subjecttod≤Bx≤h,
where
(43)B=(126012231),d=(113),h=(245).In this problem, the objective functionf(x) is pseudoconvex. Thus the proposed neural network is suitable for solving the problem in this case. Neural network (9) associated to (42) can be described as
(44)Bx˙(t)=-B-1∇f(x)-μξ(Bx(t)),
where
(45)∇f(x)=(2(x1-x3)2(x2-x3)-2(x1+x2-2x3)),ξ(Bx(t))∈K[g[d,h]](Bx(t)).Letx^={0,1,0}, s=Bx^∈int⁡(Ω); then we have ω=2. Moreover, the restricted region [d,h]⊂B(s,r) with r=13. An upper bound of ∂f(x) is estimated as lf=46. Then the designed parameter μ is estimated as μ>29.4. Let μ=30 in the simulation.Figures5 and 6 display the state trajectories of this neural network with different initial values. It can be seen that these trajectories converge to the feasible region in finite time as well. This is in accordance with the conclusion of Theorem 15. It can be verified that the trajectory is stable in the sense of Lyapunov.Figure 5
Time-domain behavior of the state variablesx1, x2, and x3 with initial point x0=(5.4,6.5,-1.5).Figure 6
Time-domain behavior of the state variablesx1, x2, and x3 with initial point x0=(-16.3,19.6,-2.3).


## 5. Conclusion

In this paper, a one-layer recurrent neural network has been presented for solving pseudoconvex optimization with box constraint. The neural network system model has been described with a differential inclusion system. The constructed recurrent neural network has been proved to be stable in the sense of Lyapunov. The conditions which ensure the finite time state convergence to the feasible region have been obtained. The proposed neural network can be used in a wide variety to solve a lot of optimization problem in the engineering application.



---

*Source: 283092-2014-02-27.xml*