# The Smoothing FR Conjugate Gradient Method for Solving a Kind of Nonsmooth Optimization Problem withl1-Norm

Authors: Miao Chen; Shou-qiang Du
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

We study the method for solving a kind of nonsmooth optimization problems withl1-norm, which is widely used in the problem of compressed sensing, image processing, and some related optimization problems with wide application background in engineering technology. Transformated by the absolute value equations, this kind of nonsmooth optimization problem is rewritten as a general unconstrained optimization problem, and the transformed problem is solved by a smoothing FR conjugate gradient method. Finally, the numerical experiments show the effectiveness of the given smoothing FR conjugate gradient method.


---

## Body

## 1. Introduction

In the last few years, the study work of finding sparsest solutions to undermined system of equations has been extensively done. Finding the sparsest solution of an undermined system of equations is equivalent to solving thel0-norm regularized minimization problem as the following: (1)minx0,s.t.x∈D,where D=x∣Ax=b,x∈Rn, A∈Rm×n(m≪n), b∈Rm, and ·0 denotes 0-norm. From [1–4], we know that the above problem is difficult to solve in a straight way. In order to solve the l0-norm problem effectively, an approximation model is to replace the l0-norm by the l1-norm to solve the Basis Pursuit problem, such as in [5, 6]: (2)minx∈Rnx1:Ax=b.Therefore the convex envelope of x0 is x1, where x1=∑i=1n|xi| is the l1-norm of x. And this problem is also NP-hard problem. When b contains some noise in practice application, the above problem is rewritten as the following nonsmooth optimization problem with l1-norm: (3)minx∈Rn12Ax-b22+τx1,where A∈Rm×n(m≪n), b∈Rm, τ>0, ·2 denotes 2-norm, and ·1 denotes 1-norm. For the minimization of the l1-norm having a good recovery, (3) is widely used in the sense compression, image processing, and other related fields in engineering technology, one can see [7, 8] and the references therein. For some τ>0, (1/2)Ax-b22+τx1 is a convex function but not a differentiable function. Recently, many scholars have studied the method for solving (3). For instance, gradient projection for sparse reconstruction was proposed by Figueiredo et al. in [9], a two-step iterative shrinkage thresholding (IST) method was proposed by Bioucas-Dias and Figueiredo in [10], a fast IST algorithm was presented by Beck and Teboulle in [11], SPGL1 (A solver for large-scale sparse reconstruction) was proposed by Ewout and Friedlander in [12], who consider a least-squares problem with l1-norm constraint and use a spectral gradient projection method, and ADM method was proposed by Yang and Zhang in [13]. Problem (3) was formulated to a convex quadratic problem in [14]. Among all the references mentioned above, there is no use of the relationship between the linear complementarity problems and the absolute value equations to solve (3). They do not use the structure of the absolute value equation to propose new method to solve (3). They also do not translate the original problem into an absolute value equation problem and use the effective methods of absolute value equations. Just recently, only in [15], a smoothing gradient method is given for solving (3) based on the absolute value equations. Therefore, in this paper, we study how to use this new transformation to solve (3) by the useful FR conjugate method.As we all know, the transformed linear complementarity problem is rewritten as the absolute value equation problem mainly based on the equivalence between the linear complementarity problem and the absolute value equation problem, such as [16–18]. The absolute value equation can be solved easily now. On the other hand, the conjugate gradient method is suitable for solving large-scare optimization problems and has sample structure and global convergence [19–25]. In addition, the smoothing methods are used to solve the related nonsmooth optimization problems, such as [26–28] and the references therein. Therefore, based on the above analysis, we present a new smoothing FR conjugate gradient method to solve (3); this is also our motivation to write this paper. The global convergence analysis of the given method is also presented. Finally, some computational results show that the smoothing FR conjugate gradient method is efficient in practice.The remainder of this paper is organized as follows: In Section2, the preliminaries are proposed, which include the description of how the linear complementarity problem is transformed into the absolute value equation problem. In Section 3, we present the smoothing FR conjugate gradient method and give its convergence analysis. Finally, in Section 4, we give some numerical results of the given method, which show the effectiveness of the given method.


## 2. Preliminaries

Firstly, we give the transformation form of (3). For any vector x∈Rn, it can be formulated to (4)x=u-v,u≥0,v≥0,where u,v∈Rn,ui=(xi)+, and vi=(-xi)+ for all i=1,…,n. By the definition of x1=∑i=1n|xi|, we get x1=enTu+enTv, where en=[1,1,…,1]T. Therefore, as in [13–15], problem (3) can be rewritten as (5)minz=u,vT≥012b-Az22+τenTu+τenTv.The above problem can be transformed to(6)min12zTHz+cTz,s.t.z≥0,where(7)z=u,vT,c=τe2n+-ctct,ct=ATb,H=ATA-ATA-ATAATA.Since H is a positive semidefinite matrix, problem (3) can be transformed into convex optimization problem. Then problem (6) can be transformed into a linear variable inequality problem, which is to find z∈R2n, such that (8)Hz+c,z~-z≥0,∀z~≥0.Given that the feasible region of z is a special structure (such as nonnegative orthant), (8) can be rewritten as the linear complementary problem, which is used to find z∈R2n, such that (9)z≥0,Hz+c≥0,zTHz+c=0.Now, we give some results about the absolute value equations and the linear complementarity problems as follows; one can see sources such as [16, 29, 30]. The absolute value equations have the form Ax-|x|=b, where A∈Rn×n, b∈Rn. The linear complementarity problems have the form 0≤z⊥Hz+c≥0, where H∈Rn×n, c∈Rn.Proposition 1.
(i) Conversely, if1 is not an eigenvalue of H, then the linear complementary problem can be reduced to the following equation: (10)H-I-1H+I12H-Iz+c-12H-Iz+c=H-I-1c.
(ii) The absolute value equations is equivalent to the bimultilinear program:(11)minA+Ix-bTA-Ix-b=0,A+Ix-b≥0,A-Ix-b≥0.
(iii) And the absolute value equations is equivalent to the generalized linear complementary problem:(12)A+Ix-b⊥A-Ix-b,A+Ix-b≥0,A-Ix-b≥0.Proposition 2.
Equation (9) can be transformed into the following absolute value equation problem, which is defined as (13)H+Iz+c=H-Iz+c.Proof.
Based on Proposition1 and (9), we know that (14)A+Ix-b=Hz+c,A-Ix-b=z.Then by (14), we have (15)x=A-I-1z+b,Hz+c=HA-Ix-Hb+c=A+Ix-b.To satisfy the last equation of (15) for all x∈Rn, denote (16)b=H-I-1c,A=H-I-1H+I.Substituting (16) in (15), we get(17)x=A-I-1z+b=H-I-1H+I-I-1z+H-I-1c=H+I-H-I-1H-Iz+c=12H-Iz+c.Due to the above (i) in Proposition 1, (9) can be reduced to the absolute value equations: (18)H-I-1H+Ix-x=H-I-1c,and substituting form (17) in (18), we get the following absolute value equation problem, which has the form (19)H+Iz+c=H-Iz+c.Thus, we get (13). Then, problem (3) can be transformed into the following problem: (20)minz∈R2nfz=12H+Iz+c-H-Iz+c22.


## 3. The Smoothing FR Conjugate Gradient Method

In this section, we give the smoothing FR conjugate gradient method for solving (20). Firstly, we give the definition of smoothing function and the smoothing approximation function of the absolute value function; one can see [15, 26, 27].Definition 3.
Letf:Rn→R be a local Lipschitz continuous function. We call f~:Rn×R+→R a smoothing function of f, if f(·,μ) is continuously differentiable in Rn for any fixed μ>0, and (21)limμ→0f~x,μ=fx.There are so many smoothing functions; for example, Chen and Mangasarian introduced a class of smooth approximations of the function (x)+. Let ρ:R→R+ be a density function satisfying (22)k≔∫-∞∞sρsds<∞,ρs=ρ-s.Then (23)ϕx,μ≔∫-∞∞x-μs+ρsds.Then ϕ is a smoothing function of (x)+.In this paper, we use the smoothing approximation function of the absolute value function as(24)Φiz,μ=H-Iz+ci2+μ2,μ∈R+,i=1,…,2n.And (24) is the smoothing approximation function of (20). Then, problem (20) is solved by smoothing conjugate gradient method. And formula (24) also satisfies (25)limμ→0Φiz,μ=H-Iz+ci,i=1,…,2n.Based on (24), we can get the following smoothing function of (20): (26)f~z,μ=12∑i=12nf~i2z,μ,where f~i(z,μ)=((H+I)z+c)i-Φi(z,μ), i=1,…,2n.Now, we give the smoothing FR conjugate gradient method for solving (20).Algorithm 4 (the smoothing FR conjugate gradient method).
Step 1. Choose 0<σ<1, 0<ρ<1, r>0, μ0>1, ϵ>0 and consider an initial point x0 in Rn. Let k=0; compute g~0=∇f~(z0,μ0). Let d0=-g~0.
Step 2. If ∇f~≤ϵ, then terminate the method; otherwise, let zk+1=zk+αkdk, where (27)dk=-g~k+βk-1FR·dk-1,g~k=∇f~zk,μk,βkFR=g~k+1Tg~k+1g~kTg~k.
Step 3. Compute αk by Armijo line search, where αk=max{ρ0,ρ1,ρ2,…} and ρi satisfies (28)f~zk+ρidk,μk≤f~zk,μk+σρig~kTdk.
Step 4. If ∇zf~(zk+1,μk)≥rμk, then set μk+1=μk; otherwise, let μk+1=σμk.
Step 5. Set k≔k+1; go to Step 2.Now, we give the convergence analysis of Algorithm4.Theorem 5.
Supposeg~kTdk-1<0, then g~kTdk≤-g~k2.Proof.
According to definitiong~kTdk=-g~k2+βkFRg~kTdk-1 and g~kTdk-1<0, we can obtain (29)g~kTdk=-g~k2+βkFRg~kTdk-1≤-g~k2.It is obviously that g~kTdk≤-g~k2.Theorem 6.
Supposef~(·,μ) is a smoothing function of f. And if for any constant μ>0, f~(·,μ) is bounded on the level set LX0=x∈Rn∣f(x)≤f(x0), then the {zk} generated by Algorithm 4 satisfies (30)limk→∞μk=0,liminfk→∞∇zf~zk,μk-1=0.Proof.
DefineK=K∣μk+1=σμk. If K is a finite set, then there exists an integer k¯, such that for all k>k¯(31)∇zf~zk,μk-1≥rμk-1.And μk=μk¯=μ¯ in Step 4 of Algorithm 4. By considering f~(·,μ¯) a smooth function, Theorems 1 and 2 in [26], the conjugate gradient method for solving (32)minf~z,μ¯satisfies (33)liminfk→∞∇zf~zk,μ¯=0,which contradicts (31). This shows that K must be infinite and (34)limk→∞μk=0.Because K is infinite, we can suppose K={k0,k1,…}, where k0<k1<⋯, and then we can get (35)limi→∞∇zf~zk+1,μk≤rlimi→∞μki=0.


## 4. Numerical Tests

In this section, we give numerical experiment results of Algorithm4. The numerical experiments are also considered in [9, 14, 15]. In computing Examples 1, 2, 3, and 4, we compare Algorithm 4 with the smoothing gradient method in [15]. In computing Example 5, we compare Algorithm 4 with GPSR, debiased and minimum norm methods proposed in [4, 9, 13]. The numerical results of all the examples illustrate that Algorithm 4 is effective. All codes for the test problems are finished in the MATLAB8.0. For example, 1–4, the parameters used in Algorithm 4, are choosen as σ=0.2, ρ=0.4, r=0.5, ϵ=10-6.Example 1.
Consider the following optimization problem:(36)minx12Ax-b22+τx1,where(37)τ=5,b=2,4,1,7′,A=358415296574347216896574.The problem has an optimal solutionx∗=(0.3461,0.0850,0,0,0.3719,0) in [14, 15]. The optimal solution of our method is x1∗=(0.3461,0.0850,0,0.0003,0.3720,-0.0001). The optimal solution of smoothing gradient method is x2∗=(0.3461,0.0850,0,0.0003,0.3720,-0.0001). In Figures 1 and 2, we, respectively, plot the evolution of the objective function versus iteration number when solving Example 1 with Algorithm 4 and smoothing gradient method in [15].Figure 1
Numerical results for solving Example1 with Algorithm 4.Figure 2
Numerical results for solving Example1 with smoothing gradient method.Example 2.
Consider the following optimization problem:(38)minx12Ax-b22+τx1,where (39)τ=10,b=1,1,…,1,A=4-10⋯01⋯1-14-1⋯⋮⋮⋯⋮0-14⋱⋮⋮⋯⋮⋮⋱⋱⋱-1⋮⋯⋮0⋯0-141⋯1n×n+10.Figures3 and 4 plotted the evolution of the objective function versus iteration number (m=100, n=110) when solving Example 2 with Algorithm 4 and smoothing gradient method. Figures 5 and 6 plotted the evolution of the objective function versus iteration number (m=200, n=210) when solving Example 2 with Algorithm 4 and smoothing gradient method. By comparison, we know that the number of iterations of the Algorithm 4 is less than that of the smoothing gradient method in [15].Figure 3
Numerical results for solving Example2 with Algorithm 4 (m=100, n=110).Figure 4
Numerical results for solving Example2 with smoothing gradient method (m=100, n=110).Figure 5
Numerical results for solving Example2 with Algorithm 4 (m=200, n=210).Figure 6
Numerical results for solving Example2 with smoothing gradient method (m=200, n=210).Example 3.
Consider the following optimization problem:(40)minx12Ax-b22+τx1,where (41)τ=10,b=1,1,…,1,A=2111⋯11⋯11200⋯0⋮⋯⋮1020⋯0⋮⋯⋮0002⋯0⋮⋯⋮⋮⋮⋮⋮⋱⋮⋮⋯⋮1000⋯21⋯1n×n+10.Figures7 and 8, respectively, show the results while m=100, n=110, the change in the number of iterations of the objective function when solving Example 3 with Algorithm 4 and smoothing gradient method. Figures 9 and 10, respectively, show the results while m=150, n=160, the change in the number of iterations of the objective function when solving Example 3 with Algorithm 4 and smoothing gradient method. By comparison, the objective function variants are faster in Algorithm 4 than that of the smoothing gradient method in [15].Figure 7
Numerical results for solving Example3 with Algorithm 4 (m=100, n=110).Figure 8
Numerical results for solving Example3 with smoothing gradient method (m=100, n=110).Figure 9
Numerical results for solving Example3 with Algorithm 4 (m=150, n=160).Figure 10
Numerical results for solving Example3 with smoothing gradient method (m=150, n=160).Example 4.
Consider the following optimization problem:(42)minx12Ax-b22+τx1,where (43)τ=10,b=1,1,…,1,A=1n000⋯01⋯102n00⋯0⋮⋯⋮003n0⋯0⋮⋯⋮0004n⋯0⋮⋯⋮⋮⋮⋮⋮⋱⋮⋮⋯⋮0000⋯11⋯1n×n+10.Figures11 and 12, respectively, show the objective function plotted against iteration number while m=100, n=110, when solving Example 4 with Algorithm 4 and smoothing gradient method. Figures 13 and 14, respectively, show the objective function plotted against iteration number while m=150, n=160, when solving Example 4 with Algorithm 4 and smoothing gradient method in [15]. By comparison, Algorithm 4 is more effective than the smoothing gradient method.Figure 11
Numerical results for solving Example4 with Algorithm 4 (m=100, n=110).Figure 12
Numerical results for solving Example4 with smoothing gradient method (m=100, n=110).Figure 13
Numerical results for solving Example4 with Algorithm 4 (m=150, n=160).Figure 14
Numerical results for solving Example4 with smoothing gradient method (m=150, n=160).Example 5.
We consider a typical CS scenario, which is also considered in [1–4, 13–15]. The goal is to reconstruct a length-n sparse signal from m observations, where m<n. In this example, we choose m=23, n=25, σ=0.1, ρ=0.5, r=0.5, ϵ=10-6, and the original signal contains 15 randomly generated ±1 spikes. The observation b is generated according to b=Ax+ne, where b is contaminated by noise. Furthermore, the m×n matrix A is obtained by first filling it with independent samples of a standard Gaussian distribution and then orthonormalizing its rows. In this example, we choose σ2=10-4 and τ=0.1ATy∞ as suggested in [14]. Figure 15 shows the results of the signal reconstruction.Figure 15
Numerical results for solving Example5 with Algorithm 4.


## 5. Conclusion

Compared with the GPSR method and other methods in [4, 9, 13–15], the smoothing FR conjugate gradient method is simple and needs small storage. The establishment and continuous improvement of the smoothing method for (3) provide a very useful tool to meet the challenges of many practical problems. For example, Figure 15 shows that the smoothing FR conjugate gradient method works well, and it provides an efficient approach to denoise sparse signals. Compare with the smoothing gradient method in [15], the smoothing FR conjugate gradient method is significantly faster than the smoothing gradient method, especially in large-scale iterations. We have also shown that, under weak conditions, the smoothing FR conjugate gradient method converges globally.



---

*Source: 5817931-2018-01-23.xml*