# Diagonal Hessian Approximation for Limited Memory Quasi-Newton via Variational Principle

Authors: Siti Mahani Marjugi; Wah June Leong
Journal: Journal of Applied Mathematics
 (2013)

---

## Abstract

This paper proposes some diagonal matrices that approximate the (inverse) Hessian by parts using the variational principle that is analogous to the one employed in constructing quasi-Newton updates. The way we derive our approximations is inspired by the least change secant updating approach, in which we let the diagonal approximation be the sum of two diagonal matrices where the first diagonal matrix carries information of the local Hessian, while the second diagonal matrix is chosen so as to induce positive definiteness of the diagonal approximation at a whole. Some numerical results are also
presented to illustrate the effectiveness of our approximating matrices when incorporated within the L-BFGS algorithm.


---

## Body

## 1. Introduction

Our investigation begins by seeking effective ways to diagonally scale an identity matrix, which is often used to initialize the L-BFGS method. For this purpose, it is useful to state the weak quasi-Newton equation:(1)skTMk+1sk=skTyk,
where the n-dimensional vector sk=xk+1-xk denotes the step corresponding to two different points xk and xk+1, yk=gk+1-gk denotes the gradient change corresponding to the gradients gk and gk+1 at the two points, Mk+1 is a full n×n matrix that approximates the Hessian of f. Both s and y are used explicitly and O(n2) storage is required for the matrix Mk+1. If M is further chosen to be a diagonal matrix, say Dk+1, then we can establish the so-called quasi-Cauchy relation denwol 1993, [1] by which
(2)skTDk+1sk=skTyk.
Different from the standard quasi-Newton which requires O(n2) of storage, here only O(n) storage is required to store the diagonal update that satisfies the quasi-Cauchy relation. In addition, we suppose the matrix Dk+1 to be positive definite by which it is able to define a metric.The first example of diagonal updating that satisfies the quasi-Cauchy relation is the well-known Oren-Luenberger OREN1974 scaling matrix, given by(3)Dk+1=skTykskTskI,
where I is the identity matrix. Expression (3) would be obtained from the quasi-Cauchy relation with the further restriction where the diagonal matrix is a scalar multiple of the identity matrix. Therefore, scaling matrices that are derived from the quasi-Cauchy relation are a natural generalization of Oren-Luenberger scaling.In general, a procedure to obtain diagonal updating formulae via quasi-Cauchy relation can be summarized as follow. Suppose thatDk>0 is a positive definite diagonal matrix and Dk+1 is the updated version of D, which is also diagonal. Then the updated Dk+1 is required to satisfy the quasi-Cauchy relation. It is also essential to require the deviation between Dk and Dk+1 being minimized under some variational principle, which in return will encourage numerical stability. Very often the Frobenius matrix norm is used to measure this deviation. As noted earlier, a diagonal matrix uses the same computer storage as a vector, resulting in potential use in limited memory algorithms.This paper is structured as follows. In Section2, we consider a briefly review on the diagonal updating via quasi-Cauchy relation and the preconditioning strategy. New diagonal initial approximation for L-BFGS method is introduced in this section as well. It follows that in Section 3, the convergence properties of the proposed L-BFGS methods are investigated. Numerical results are presented in Section 4, on a large set of unconstrained minimization problems, mainly from the collections of Moré et al. [2], CUTE [3], and Toint [4]. In Section 5, through the numerical results, we do the discussion regarding the new diagonal initial approximating which matrix is more effective at improving the performance of the L-BFGS update compared to the standard preconditioner in the literature. Finally, Section 6 ends the paper by presenting a summary and conclusion.


## 2. Diagonal Updating via Quasi-Cauchy Relation

The performance of the L-BFGS method is depending on a good approximations of the actual Hessian. In the basic implementation of the L-BFGS method, the correction pairs are as follows:(4)(si,yi)fork-m^+1≤i≤k,
to correct Hk0. The choices of Hk0 often influences the behaviour of the method. It is worth the investigation on the choices of Hk0.Throughout this section, when we mention a direct initial matrix, we mean a matrix that is a rough approximation to the Hessian; otherwise an initial matrix is an approximation to the inverse of Hessian.Our approach is inspired by [5, 6] which employed a variational technique that is analogue to the one used to derive the Powell Symmetric Broyden (PSB) quasi-Newton update (see, e.g, Dennis and Schnabel [7]). The resulting update for approximating the Hessian matrix diagonally is derived as follows:
(5)Dk=Dk-1+(sk-1Tyk-1-sk-1TDk-1sk-1)tr(Ek-12)Ek-1,
where Ek-1=diag(sk-1,12,sk-1,22,…,sk-1,n2), sk-1,i denotes the ith component of the vector sk-1, and tr is the trace operator.Note that whensk-1Tyk-1<sk-1TDk-1sk-1, the resulting Dk+1 is not necessarily positive definite. Thus, like their counterpart of PSB update in the quasi-Newton setting, the foregoing update may suffer from the loss of positive definiteness and it is not appropriate for use within a quasi-Newton-based algorithm.In this study, our approach in finding an efficient diagonal Hessian approximation is done through letting the diagonal approximating matrix be a combination of two diagonal matrices. This gives us a freedom to incorporate curvature information into one of these diagonal matrices, while the property of hereditary positive definiteness is carried over to the second matrix.To begin, suppose that the Hessian matrix of an objective function has positive diagonal elements. Let us divide the Hessian matrix into two parts:(6)∇2f(x)=∇2fA(x)+∇2fB(x),
where ∇2fA(x) is a diagonal matrix consisting the diagonal entries of the Hessian and ∇2fB(x) would resemble the actual Hessian except that its diagonal entries are all zero. Thus, we intend to form two diagonal approximating matrices to approximate each part of the Hessian, respectively; that is,
(7)∇2f(x)≈D0=Ψ1+Ψ2.
Since it is assume that the entries of the actual Hessian are all positive, an excellent choice would be to let a positive definite diagonal matrix, says Ψ1, to approximate ∇2fA(x). Meanwhile, ∇2fB(x) is expected to be dense and expensive to compute and would be approximated by Ψ2. To preserve positive-definiteness, we introduce additional term in the form of θI, that is, a Levenberg-Marquat-like step to maintain positive definite of Ψ2 in a way that Ψ2 is expressed as Ψ2=θI+Ψ3. Thus, the technique for calculating Dk0 subjected to the weak-quasi-Newton relation is as follows.Theorem 1.
Suppose thatsk-1≠0; then the optimal solution of the following minimization problem:
(8)min12∥Ψ3∥F2,s.t.sk-1T(Ψ1+(θI+Ψ3))sk-1=sk-1Tyk-1,Ψ3isdiagonal,
is given by
(9)Ψ3=(sk-1Tyk-1-sk-1TΨ1sk-1-θsk-1Tsk-1)tr(Ek-12)Ek-1.Proof.
Since the objective function in (8) is convex and the feasible set is also convex, then (8) has a unique solution. Its Lagrangian function is given by
(10)L(Ψ3,μ)=12∥Ψ3∥F2+μ(sk-1TΨ3sk-1+θsk-1Tsk-1+sk-1TΨ1sk-1-sk-1Tyk-1),
where μ is the Lagrange multiplier associated with the constraint. Differentiating (10) with respect to each of elements of Ψ3 and setting the results to zero yields,
(11)Ψ3=-μEk-1.
Pre- and postmultiplying (11) by sk-1 and invoking the constraint,
(12)sk-1TΨ3sk-1=sk-1Tyk-1-sk-1TΨ1sk-1-θsk-1Tsk-1=-μsk-1TEk-1sk-1.
We have
(13)μ=sk-1TΨ1sk-1+θsk-1Tsk-1-sk-1Tyk-1sk-1TEk-1sk-1.
Finally, by substituting (13) into (11) and using the fact that sk-1TEk-1sk-1=tr(Ek-12) yields (9),
(14)Ψ3=-μEk-1=-(sk-1TΨ1sk-1+θsk-1Tsk-1-sk-1Tyk-1)tr(Ek-12)Ek-1.A direct result of Theorem1 leads to the following diagonal preconditioning formulation:
(15)Dk0=Ψ1+(θI+sk-1Tyk-1-sk-1TΨ1sk-1-θsk-1Tsk-1tr(Ek-12)Ek-1).
An analogue of the above approach can be used to derive the initial inverse Hessian approximation H0, which is more useful for algorithmic purpose. Once again by letting the inverse Hessian of an objective be separated by parts;
(16)(∇2f(x))-1=H(1)+H(2),
then the initial approximating matrix H0 can be expressed as
(17)H0=Δ1+(θ^I+Δ3).
Since it is our intention to derive an initial approximation for the L-BFGS method, then an excellent choice of Δk+1,i for approximating the diagonal entries of the inverse Hessian would be the diagonalized inverse BFGS formula of Gilbert and Lemaréchal [8] updated from a multiple of identity matrix; that is,
(18)Δk+1,i=Δk,i+(1ykTsk+ykTΔk,iyk(ykTsk)2)(sk,i)2-2Δk,isk,iyk,iykTsk,
where we choose Δk,i=ykTsk/ykTyk as the Oren-Luenberger scalar. Thus, diagonal matrix Δ1 can be obtained in (18). Note that, to safeguard very small or very large ∥sk-1∥, we impose the additional following condition: if ε2≤∥sk-1∥≤ε1, for some small and large positive ε1 and ε2, we set B0=I. Here, we used ε1=108 and ε2=10-8.By interchanging the role ofs and y in Theorem 1, one can obtain the formula for H0 at step k as follows:
(19)Hk0=Δ1+(θI+yk-1Tsk-1-yk-1TΔ1yk-1-θyk-1Tyk-1tr(Yk-12)Yk-1),
where Δ1 is given by (18) and Yk-1=diag(yk-1,12,…,yk-1,n2) with yk-1,i is the ith component of the vector yk-1.For purposes of numerical illustrations, the latter diagonal formula (19) is used to initialize the L-BFGS method, although the potential use of (15) should not be neglected. Furthermore, one can observe that Δ3 involved in the solution of the variational problem is isolated from the solution and its value does not affect the quality of the solution. This allows us to choose the value of θ freely to ensure that Hk0 are positive definite while satisfying the weak-secant equation.Note that maintaining positive definiteness forHk0 is crucial for L-BFGS method to generate descent direction. For this purpose, the following lemma suggests a possible choice on θ.Lemma 2.
Assume thatyk≠0 for all k. Then Hk0 is a positive definite, if
(20)θ=min(1,yk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1).Proof.
Note that to keep positive definiteness ofHk0, we should choose θ such that
(21)yk-1Tsk-1-yk-1TΔ1yk-1-θyk-1Tyk-1≥0,
which also implies
(22)θ≤yk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1.
Therefore by our choice onθ as (20), we have the following.
Case
1.  If (yk-1Tsk-1-yk-1TΔ1yk-1)/yk-1Tyk-1≥1, we will let θ=1, and thus,
(23)Hk0=Δ1+I+yk-1Tsk-1-yk-1TΔ1yk-1-yk-1Tyk-1tr(Yk-12)Yk-1.
Sinceyk-1Tsk-1-yk-1TΔ1yk-1≥yk-1Tyk-1, then Hk0 would be positive definite.
Case
2. If (yk-1Tsk-1-yk-1TΔ1yk-1)/yk-1Tyk-1<1, θ1=(yk-1Tsk-1-yk-1TΔ1yk-1)/yk-1Tyk-1 is set, which leads to
(24)Hk0=Δ1+yk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1I.
It is clear that for both cases, Hk0 would maintain positive definiteness.Hence,Hk0 can be expressed in the following form:
(25)Hk0={Δ1+yk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1I,ifyk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1<1,Δ1+I+yk-1Tsk-1-yk-1TΔ1yk-1-yk-1Tyk-1tr(Yk-12)Yk-1,hhhhhhhhhhllhhlifyk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1≥1.
Now, we can set up the basic algorithm of our L-BFGS methods using Hk0 as (25).


### 2.1. LBFGS-USD Algorithm

Step 1.
Choose an initial pointx0∈ℝn and a positive definite matrix Hk0=I. Let k=0.Step 2.
Computedk=-Hkgk; then xk+1=xk+αkdk where αk satisfies the Wolfe conditions given by
(26)f(xk+αkdk)≤f(xk)+δ1αkgkTdk,g(xk+αkdk)Tdk≥δ2gkTdk
(we always try the step length αk=1).Step 3.
Letm^=min{k,m-1}. Update Hk0m^+1 times using the pairs {yj,sj}j=k-m^k; that is, let
(27)Hk+1=(VkT⋯Vk-m^T)Hk0(Vk-m^⋯Vk)+ρk-m^(VkT⋯Vk-m^+1T)sk-m^sk-m^T(Vk-m^+1⋯Vk)+ρk-m^+1(VkT⋯Vk-m^+2T)×sk-m^+1sk-m^+1T(Vk-m^+2⋯Vk)⋮+ρkskskT.Step 4.
Setk:=k+1 and return to Step 2.Remark 3.
The LBFGS-USD algorithm is exactly the L-BFGS algorithm of Liu and Nocedal [9], except that Hk0 is computed by (25).


## 2.1. LBFGS-USD Algorithm

Step 1.
Choose an initial pointx0∈ℝn and a positive definite matrix Hk0=I. Let k=0.Step 2.
Computedk=-Hkgk; then xk+1=xk+αkdk where αk satisfies the Wolfe conditions given by
(26)f(xk+αkdk)≤f(xk)+δ1αkgkTdk,g(xk+αkdk)Tdk≥δ2gkTdk
(we always try the step length αk=1).Step 3.
Letm^=min{k,m-1}. Update Hk0m^+1 times using the pairs {yj,sj}j=k-m^k; that is, let
(27)Hk+1=(VkT⋯Vk-m^T)Hk0(Vk-m^⋯Vk)+ρk-m^(VkT⋯Vk-m^+1T)sk-m^sk-m^T(Vk-m^+1⋯Vk)+ρk-m^+1(VkT⋯Vk-m^+2T)×sk-m^+1sk-m^+1T(Vk-m^+2⋯Vk)⋮+ρkskskT.Step 4.
Setk:=k+1 and return to Step 2.Remark 3.
The LBFGS-USD algorithm is exactly the L-BFGS algorithm of Liu and Nocedal [9], except that Hk0 is computed by (25).


## 3. Convergence Analysis

We shall also establish the convergence of the LBFGS-USD algorithm. The following standard assumptions are made on the objective function.(a)
The objective functionf is twice continuously differentiable.(b)
The level set£=x∈ℝn∣f(x)<f(x0) is convex, and there exist positive constants m1 and m2 such that,
(28)m1∥z∥2≤zTG(x)z≤m2∥z∥2,for allz∈ℝn and x∈£.Theorem 4.
Letx0 be a starting point for which f satisfies assumptions above, and assume that the matrices Bk0 are chosen so that {∥Bk0∥}({∥(Hk0)-1∥}) are bounded. Subsequently, for any positive definite B0 (most often, B0=I is chosen), LBFGS-USD algorithm generates a sequence {xk} which converges to x*. Moreover, there is a constant 0≤r≤1 such that
(29)f(xk)-f(x*)≤rk[f(x0)-f(x*)],
which implies that {xk} converges ℝ-linearly.Proof.
See [9].Theorem4 suggests that as long as Hk0 is chosen such that ∥Hk0∥ is bounded for all k, then the corresponding L-BFGS algorithm restarts by Hk0 would generate {xk} that converges globally and ℝ-linearly. For this purpose, we give the following result which ensures that ∥Hk0∥ where Hk0 is given by (25) is upper and lower bounded by some constants.Lemma 5.
Letx0 be a starting point for which f satisfies assumptions (a)-(b). Consider the sequence {xk} generated by L-BFGS algorithms subject to the diagonal initial approximation, Hk0 given by (25). If yk≠0 for all k, then ∥Hk0∥ is upper and lower bounded for all k.Proof.
LetG¯k-1=∫01‍G(xk-1+tsk-1)dt, then yk-1=G¯k-1sk-1, and assumption (a) implies that
(30)m1≤sk-1TG¯k-1sk-1∥sk-1∥2≤m2.
Subsequently, (28) also gives
(31)sk-1Tyk-1∥yk-1∥2=sk-1TG¯k-1sk-1sk-1TG¯k-12sk-1≤m2∥sk-1∥2m12∥sk-1∥2,
and therefore,
(32)sk-1Tyk-1≤m2m12∥yk-1∥2.
First, we begin by showing that every component ofΔ1 in (18) is bounded, so that ∥Δ1∥ is bounded. It is more convenient to show that each part of Δ1, namely,  (i)
σk-1=yk-1Tsk-1/yk-1Tyk-1,(ii)
(1/yk-1Tsk-1+yk-1Tσk-1yk-1/(yk-1Tsk-1)2)(sk-1Tei)2, and(iii)
-(2σk-1(yk-1Tei)(sk-1Tei))/yk-1Tsk-1, is bounded, such that Δ1 is bounded at whole.(i)
Since(33)1m2≤yk-1T(G¯k-1)-1yk-1yk-1Tyk-1≤1m1,
then, we obtain
(34)1m2≤σk-1≤1m1.(ii)
Note that(35)(sk-1Tei)2=sk-1,i2,
where skk,i is the ith component of sk-1. Then,
(36)(1yk-1Tsk-1+yk-1Tσk-1yk-1(yk-1Tsk-1)2)(sk-1Tei)2≥σk-1yk-1Tyk-1(yk-1Tsk-1)2(sk-1Tei)2≥σk-1yk-1Tyk-1∥yk-1∥2∥sk-1∥2(sk-1Tei)2≥σk-1ε22∥sk-1∥2≥σk-1ε22ε12≥1m2ε22ε12.
On the other hand,
(37)(1yk-1Tsk-1+yk-1Tσk-1yk-1(yk-1Tsk-1)2)(sk-1Tei)2≤∑i=1n(sk-1,i)2yk-1Tsk-1+∥yk-1∥2∥sk-1∥2m1(yk-1T(G¯k-1)-1yk-1)2≤∥sk-1∥2sk-1TG¯k-1sk-1+1m1m22∥sk-1∥2∥yk-1∥2≤1m1+m22m1yk-1TG¯k-1-2yk-1yk-1Tyk-1≤1m1+m22m11m12=1m1+m22m13.
Therefore, it gives, in overall,
(38)1m2ε22ε12≤(1yk-1Tsk-1+yk-1Tσk-1yk-1(yk-1Tsk-1)2)(sk-1Tei)2≤1m1+m22m13.(iii)
In a similar way, we can establish(39)-2σk-1(yk-1Tei)(sk-1Tei)yk-1Tsk-1≥-2m1yk-1Tsk-1yk-1Tsk-1≥-2m1.
and subsequently,
(40)-2σk-1(yk-1Tei)(sk-1Tei)yk-1Tsk-1≤2σk-1(yk-1Tei)(sk-1Tei)yk-1Tsk-1≤2(1/m1)∑i=1nyk-1,isk-1,iyk-1Tsk-1=2m1.
Then, we have
(41)-2m1≤-2σk-1(yk-1Tei)(sk-1Tei)yk-1Tsk-1≤2m1.
Following that, the component ofΔ1, namely Δ1,i, is bounded as follow:
(42)1m2+1m2ε22ε12-2m1≤Δ1,i≤4m1+m22m13,
Note that, by the definition of Hk0, we have
(43)Hk,i0=Δ1,i+yk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1hhhhhlifyk-1Tsk-1-yk-1TΔ1yk-1yk-1Tyk-1<1.
Hence, Hk,i0 satisfies
(44)1m2+1m2ε22ε12-2m1≤Δ1,i≤Hk,i0≤Δ1,i+1≤4m1+m22m13+1.
Whereas, when (yk-1Tsk-1-yk-1TΔ1yk-1)/yk-1Tyk-1≥1, we have
(45)1m2+1m2ε22ε12-2m1≤Δ1,i≤Hk,i0≤Δ1,i+1+yk-1Tsk-1trYk-12yk-1,M2≤4m1+m22m13+1+m2∥yk-1∥2m12∑i=1n(yk-1,i4)yk-1,M2yk-1,M4≤4m1+m22m13+1+nm2yk-1,M2m12yk-1,M2=1+4m1+m22m13+nm2m12,∀i,
where yk-1,M2 is the largest component among all yk-1,i2. Therefore, we can conclude that ∥(Hk,i0)∥ is upper and lower bounded for all k.


## 4. Numerical Experiences

Our test used a large set of unconstrained minimization problem consisting of 50 test problems where the list of problems is given in Table1. These test problems are selected from Moré et al. [2], CUTE [3], Toint [4], and various other test function collections such as in [10]. The subroutine of test problems is available at http://camo.ici.ro/forum/SCALCG/evalfg.for (accessed on Jan 2012). The method tested is as follows:(1)
LBFGS-I: L-BFGS method with the initial matrix,Hk0=I,(2)
LBFGS-mI: L-BFGS method with the initial matrix, Hk0=θI where θk is the Oren-Luenberger scaling at k,(3)
LBFGS-USD: L-BFGS method with the initial matrix,Hk0 is given by (19).Table 1
Selected test problems.
TRIGONOMETRIC, SROSENBR, Extended White and Holst,
Perturbed Quadratic, Raydan 1, Raydan 2, Diagonal 1, Diagonal 2, Diagonal 3, Hager,
Generalized Tridiagonal 1, Extended Tridiagonal 1,
Extended Three Expo Terms, Generalized Tridiagonal 2,
Diagonal 5, HIMMELBC, Generalized PSC1, Extended PSC1,
Extended Block-Diagonal BD1, Extended Maratos, Extended Hiebert,
Quadratic QF1, Extended Quadratic Penalty QP1, Quadratic QF2,
Extended EP1, Extended Tridiagonal 2, NONDIA, DIXMAANA,
DIXMAANB, DIXMAANC, DIXMAANE, Broyden Tridiagonal,
Almost Perturbed Quadratic, EDENSCH, STAIRCASE S1,
Diagonal 6, DIXON3DQ, DIXMAANF, DIXMAANG, DIXMAANI,
DIXMAANJ, DIXMAANK, DIXMAANL, ENGVAL1, COSINE, DENSCHNB,
DENSCHNF, BIGGSB1, Scaled Quadratic SQ1, Scaled Quadratic SQ2.


## 5.  Discussion

In general, Figure1 indicates that the new diagonal initial approximating matrix are substantially better, followed by both standard initializations of the L-BFGS method in terms of number of iterations, function/gradient calls, and CPU time, respectively. To better study the effect of our initial approximation, we include Tables 2, 3, and 4 that give ratio of function/gradient calls over iteration counts for the L-BFGS method with standard initial matrices and our diagonal initial approximating matrix. As conclusion, our diagonal initial approximation, LBFGS-USD, performs better in which the ratio is close to one and hence would likely to accept unit step length compared to the LBFGS-I (without any preconditioning) method.Table 2
Ratio of function/gradient calls over iteration counts form=3.
LBFGS-I
LBFGS-mI
LBFGS-USD
1.822477996
1.078519988
1.183809586Table 3
Ratio of function/gradient calls over iteration counts form=5.
LBFGS-I
LBFGS-mI
LBFGS-USD
1.764504202
1.057316205
1.181762512Table 4
Ratio of function/gradient calls over iteration counts form=7.
LBFGS-I
LBFGS-mI
LBFGS-USD
2.000371968
1.051121217
1.194642288Performance profile of LBFGS-USD based on iterations, function/gradient call, and CPU time for all choices ofm and n.
(a)
(b)
(c)Moreover for LBFGS-USD method, it requires, in general less iteration counts as much as 38% and 19% than LBFGS-I and LBFGS-mI, respectively. Meanwhile, in terms of number of function/gradient, LBFGS-MTD needs 46% and 16% less function/gradient counts, respectively. Finally, LBFGS-MTD requires 33% and 6% less CPU time in second, respectively. In conclusion, the numerical results for a broad class of the test problems show that the LBFGS-USD algorithm is efficient and vast superior in solving small to large size problems.


## 6. Concluding Remarks

We proposed technique that exploits the presence of the Hessian in the diagonal matrix form. Under some standard assumption on the objective function, we observe that the convergence of the diagonal initial approximation of the L-BFGS scheme isℝ-linear. Based on our numerical results, we believe that the following conclusions can be made on the diagonal approximation that is derived in this study.(i)
Our diagonal approximating matrix is able to maintain positive definiteness in a very simple way and giveO(n) storage requirement.(ii)
The numerical experiments show that our diagonal initial approximating matrix is generally effective for the L-BFGS method compared to the standard initial matrices in the literature.



---

*Source: 523476-2013-12-25.xml*