Title: Block Broyden's Methods for Solving Nonlinear Equations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Broyden's method

nonlinear equations
Abstract: This paper studies quasi-Newton methods for solving nonlinear equations. We propose block variants of both good and bad Broyden's methods, which enjoy explicit local superlinear convergence rates. Our block good Broyden's method has faster condition-number-free convergence rate than existing Broyden's methods because it takes the advantage of multiple rank modification on the Jacobian estimator. On the other hand, our block bad Broyden's method directly estimates the inverse of the Jacobian provably, which reduces the computational cost of the iteration. Our theoretical results provide some new insights on why good Broyden's method outperforms bad Broyden's method in most of the cases. The empirical results also demonstrate the superiority of our methods and validate our theoretical analysis.

Block Broyden's Methods for Solving Nonlinear Equations

Abstract

This paper studies quasi-Newton methods for solving nonlinear equations. We propose block variants of both good and bad Broyden's methods, which enjoy explicit local superlinear convergence rates. Our block good Broyden's method has a faster condition-number-free convergence rate than existing Broyden's methods because it takes the advantage of multiple rank modification on Jacobian estimator. On the other hand, our block bad Broyden's method directly estimates the inverse of the Jacobian provably, which reduces the computational cost of the iteration. Our theoretical results provide some new insights on why good Broyden's method outperforms bad Broyden's method in most of the cases. The empirical results also demonstrate the superiority of our methods and validate our theoretical analysis.

1 Introduction

In this paper, we consider solving the following nonlinear equation systems:

where x ∈ R d , F ( x ) def = [ F 1 ( x ) , · · · , F d ( x )] ⊤ : R d → R d and each F i ( x ) is differentiable. Solving nonlinear equations is one of the most important problems in scientific computing [38]. It has various applications including machine learning [3, 4, 12, 15, 44], game theory [18, 39], economics [2] and control systems [5, 37].

Newton's method and its variants [16, 25, 26] such as the Gauss-Newton method [19, 38], the Levenberg-Marquart method [17, 28, 34, 36] and the trust region method [40, 53] are widely adopted to solve the systems of nonlinear equations. These methods usually enjoy fast local superlinear rates.

Luo Luo

Newton's method takes iterates of form

where J ( x ) ∈ R d × d is the Jacobian at x . Since computing the inverse of the exact Jacobian matrix requires O ( d 3 ) running time, Newton's method suffers from expensive computation especially when solving the large-scale nonlinear equations [20, 47, 54].

Quasi-Newton methods have been proposed for avoiding the heavy computational cost of Newtontype methods while preserving good local convergence behaviour [7-11, 13, 24, 31, 32, 41-43, 45, 51]. Among these quasi-Newton methods, the Broyden's methods [6], including the good and the bad schemes [1, 30, 35], are considered to be the most effective methods for solving nonlinear equations. The Broyden's good method 2 approximates the Jacobian J ( x t ) by an estimator B t and updates the Jacobian estimator in each round as B t +1 = B t + ∆ t . Here ∆ t is a rank1 updating matrix constructed by the curvature information. Broyden et al. [11], Kelley and Sachs [27] proved that the good Broyden's method can achieve asymptotic local superlinear rates.

The bad Broyden's method approximates the inverse of the Jacobian by H t and updates the approximate matrix directly. Although the bad Broyden's method enjoys less computational cost than good Broyden's method in each iteration, it does not perform as well as the good method in most cases [1]. Lin et al. [29] show that both the good and bad Broyden's methods have superlinear rates of O ((1 / √ t ) t ) and provide some insights on the difference between their empirical performance.

Ye et al. [50] proposed a new variant of good Broyden's method by conducting ∆ t with a greedy or random strategy. Their method achieves a better explicit convergence rate of O ((1 -1 /d ) t ( t -1) / 4 ) . However, it remains unknown whether this convergence rate can be further improved by leveraging block updates which increase the reuse rate of the data in cache and take advantage of parallel computing [14]. Gower and Richt´ arik [21] studied several random quasi-Newton updates including the Broyden's updates for approximating the inverse of matrices, but they only provide implicit linear rates for their methods. Liu et al. [33] established explicit convergence rates for several block quasi-Newton updates, but they focus on approximating positive definite matrices.

In this paper, we propose two random block Broyden's methods for solving nonlinear equations and provide their explicit superlinear convergence rates. We compare the theoretical results of proposed methods with existing Broyden's methods in Table 1 and summarize our contribution as follows:

We provide explicit convergence rates for the block good Broyden's udpate and the block bad Broyden's update proposed by Gower and Richt´ arik [21]. Our results show that the block good Broyden's update can approximate a nonsingular matrix A with a linear rate of (1 -k/d ) t which improves the previous rate of (1 -1 /d ) t where k def = rank( ∆ t ) . We also show that the 'bad' update can approximate the inverse matrix A -1 with an linear rate of (1 -k/ ( d ˆ κ 2 )) t where ˆ κ is the condition number of A . To the best of our knowledge, this is the first explicit convergence rate for the block bad Broyden's update.

We propose the block good Broyden's method with convergence rate O ((1 -k/d ) t ( t -1) / 4 ) where k is the rank of the updating matrix ∆ t . This rate reveals the advantage of block update and improves previous results. Our method also relaxes the initial conditions stated in Ye et al. [50].

We propose the block bad Broyden's method with convergence rate O ((1 -k/ (4 dκ 2 )) t ( t -1) / 4 ) . We also study the initial conditions of two proposed block variants. Our analysis shows that bad Broyden's method is only suitable for the cases where the condition number of the Jacobian is small, while good Broyden's method performs well in most cases.

Paper Organization In Section 2, we introduce the notation and assumptions as the preliminaries of this paper. In Section 3, we introduce the block good or bad Broyden's updates for approximating the general matrix. In Section 4, we propose the block good or bad Broyden's methods with explicit local superlinear rates. In Section 5, we discuss the behavior difference of the good and bad methods. We validate our methods by numerical experiments in Section 6. Finally, we conclude our results in Section 7. All proofs are deferred to appendix.

2 Preliminaries

We let [ d ] def = { 1 , 2 · · · , d } . We use ∥ · ∥ F to denote the Frobenius norm of a given matrix, ∥ · ∥ 2 to denote the spectral norm of a vector and Euclidean norm of a matrix respectively. The standard basis for R d is presented by { e 1 , · · · , e d } and I d is the identity matrix. We denote the trace, the largest singular value, and the smallest singular value of a matrix by tr ( · ) , σ min ( · ) , and σ max ( · ) respectively.

We use x ∗ to denote the solution of the nonlinear equation (1) and J ∗ to denote the Jacobian matrix at x ∗ , i.e., J ∗ def = J ( x ∗ ) . We let µ def = σ min ( J ( x ∗ )) , L def = σ max ( J ( x ∗ )) and then define the condition number of J ∗ as κ def = L/µ . We also use ˆ κ def = σ max ( A ) /σ min ( A ) to present the condition number of given matrix A .

Then we present two standard assumptions on the nonlinear equations (1), which is widely used in previous works [16, 29, 50].

Assumption 2.1. The solution x ∗ of the nonlinear equation (1) is unique and nondegenerate, i.e.,

Assumption 2.2. The Jacobian J ( x ) satisfies

The following proposition shows that if x is in some local region of x ∗ , the Jacobian matrix J ( x ) has a bounded condition number.

Proposition 2.3. Suppose Assumptions 2.1 and 2.2 hold. For all x satisfies ∥ x -x ∗ ∥ 2 ≤ µ 2 / (6 LM ) , we have

We present two notations for the block Broyden's Update.

Definition 2.4 (Block Good Broyden's Update) . Let A , B ∈ R d × d . For any full column rank matrix U ∈ R d × k , we define

Definition 2.5 (Block Bad Broyden's Update) . Let A , H ∈ R d × d . For any full column rank matrix U ∈ R d × k , we define

3 The Block Broyden's Updates for Approximating Matrices

In this section, we provide the linear convergence rates of the block good and bad Broyden's updates for approximating matrices. The theoretical results is summarized in Table 2.

(a). the result holds for k = 1 and it is unknown when k > 1 .

(b). Gower and Richt´ arik [21] only prove that ρ ∈ [0 , k d ] , but do not provide the explicit value of ρ .

The block good Broyden's update, which aims to compute an approximation of matrix A , can be written as:

The following theorem presents a linear convergence rate of (1 -k/d ) t which is better than the rate (1 -1 /d ) t provided by Gower and Richt´ arik [21], Ye et al. [50].

Theorem 3.1. Assume that A ∈ R d × d and B 0 ∈ R d × d . If we select U t = [ e i 1 , e i 2 , · · · , e i k ] ∈ R d × k , where { i 1 , · · · , i k } are uniformly chosen from { 1 , 2 , · · · , d } without replacement at each round, then for any nonsingular matrix C ∈ R d × d , the block good Broyden's update satisfies

On the other hand, the bad Broyden's update which targets to approximate A -1 can be written as:

Gower and Richt´ arik [21] provide an implicit rate of (1 -ρ ) t for the above scheme with ρ ∈ [0 , k/d ] , but their analysis cannot guarantee an explicit ρ . In the following theorem, we show that the block bad Broyden's update can approximate H t to A -1 with an explicit linear rate of (1 -k/ (ˆ κ 2 d )) t .

Theorem 3.2. Assume that A ∈ R d × d and H 0 ∈ R d × d . If we select U t = [ e i 1 , e i 2 , · · · , e i k ] ∈ R d × k where { i 1 , · · · , i k } are uniformly chosen from { 1 , 2 , · · · , d } without replacement at each round, then for any nonsingular matrix C ∈ R d × d , the block bad Broyden's update satisfies

and and

Remark 3.3 . If we choose C = I d in Theorem 3.1 and Theorem 3.2, then the measures in these two theorems are exactly the same as the one in Section 8.5 and Section 8.3 of [21]. Besides, the rate of Theorem 3.1 recovers the convergent rates of Section 8.5 in [21] and Lemma 4.1 in [50] when we take k = 1 .

4 The Block Broyden's Methods

In this section, we propose two block Broyden's methods for solving the nonlinear equation (1). We present our algorithms in section 4.1 and the corresponding convergence results in Section 4.2.

Algorithm 1 Block Good Broyden's Method (BGB)

1: Input: Initial estimator B 0 , initial point x 0 and block size k .

2: for t = 0 , 1 . . .

3: x t +1 = x t -B -1 t F ( x t ) .

4: Choose { i 1 , i 2 , · · · , i k } by uniformly select k items from { 1 , · · · , d } without replacement.

5: U t = [ e i 1 , · · · , e i k ] ∈ R d × k .

6: B t +1 = Block-G-Broyden ( B t , J ( x t +1 ) , U t ) .

7: end for

Algorithm 2 Block Bad Broyden's Method (BBB)

1: Input: Initial estimator H 0 , initial point x 0 and block size k .

2: for t = 0 , 1 . . .

3: x t +1 = x t -H t F ( x t ) .

4: Choose { i 1 , · · · , i k } by uniformly select k items from { 1 , · · · , d } without replacement.

5: U t = [ e i 1 , · · · , e i k ] ∈ R d × k .

6: H t +1 = Block-B-Broyden ( H t , J ( x t +1 ) , U t ) .

7: end for

4.1 Algorithms

By using the block Broyden's updates in Section 3, we propose two novel algorithms called Block Good Broyden's Method (BGB) and Block Bad Broyden's Method (BBB) for solving nonlinear equations.

Wepresent the BGB algorithm in Algorithm 1 which updates the Jacobian estimator B t by the block good Broyden's update in each iteration. Notice that the inverse of B t can be computed efficiently by adopting Sherman-Morrison-Woodbury formula [46]. On the other hand, the BBB algorithm, which is presented in Algorithm 2, approximates the inverse of the Jacobian directly by using the block bad Broyden's update. It usually has a lower computational cost than the BGB algorithm in each round because the BBB algorithm does not need to compute the inverse of the estimator H t .

Remark 4.1 . Algorithms 1 and 2 do not require full information of the Jacobian. We construct U t by subsampling the columns of the identity matrix. When updating the Jacobian estimator by the block updates, we need to calculate J t +1 U t which is only the partial information of J t +1 (columns of J t +1 ). Since we have k ≪ d , it is not expensive to access the partial information of the Jacobian.

4.2 Convergence Analysis for the Block Broyden's Methods

We provide the convergence analysis for Algorithm 1 and Algorithm 2 in Section 4.2.1 and Section 4.2.2 respectively. We denote the Jabcobian matrix at x t as J t . As previous works [16, 29, 50], we make an assumption on the estimator matrices in Algorithm 1 and Algorithm 2 as follows:

Assumption 4.2. We assume the sequence { B t } ∞ t =0 generated by Algorithm 1 (and { H t } ∞ t =0 generated by Algorithm 2) are well-defined and nonsingular.

4.2.1 Analysis for Block Good Broyden's Methods

In this subsection, we use the following measures for our convergence analysis,

The r t measures the distance between x t and the solution x ∗ and σ t measures how well does the estimator matrix B t approximate the Jacobian at x ∗ .

The following lemma provides upper bound of σ t after one block Broyden's update.

Lemma 4.3. Performing Algorithm 1 under Assumptions 2.1, 2.2 and 4.2, we have

Based on Lemma 4.3, we present the superlinear convergence rate for Algorithm 1.

Theorem 4.4. Suppose Assumptions 2.1, 2.2 and 4.2 hold and the initial condition of Algorithm 1 satisfies

for arbitary q ∈ (0 , 1) . Then for any k ∈ [ d -1] , the output of Algorithm 1 satisfies

Theorem 4.4 implies the following high probability bound for Algorithm 1.

Corollary 4.5. Performing Algorithm 1 under the same assumption and initial condition as Theorem 4.4, with probability at least 1 -δ , we have

and and

Comparison with [50] Compare Theorem 4.4 with Theorem 4.3 of [50], we can find that the convergence rate of our BGB algorithm is better than greedy and randomized good Broyden's methods [50] if we choose k > 1 .

On the other hand, the initial condition of greedy and randomized good Broyden's methods [50] is

while the condition of Theorem 4.4 can be reformulated as

Since

condition (13) can implies condition (14). However, the reverse is not always true. For example, we can choose B 0 = 1 . 5 J ∗ and suppose

Overall, compared with the greedy or randomized good Broyden's method [50], Theorem 4.4 not only gives a faster convergence superlinear rate by leveraging the idea of block update, but also weakens the initial condition by using different measures in the analysis.

4.2.2 Analysis for Block Bad Broyden's Methods

This subsection gives the convergence analysis for Algorithm 2. We use the following measures to describe the convergent behavior

The R t measures the distance between x t and the solution x ∗ and τ t measures how well does the estimator H t approximate the matrix J -1 ∗ .

Using the convergence results for the block bad Broyden's update in Theorem 3.2, we are able to tackle the difference between the estimator H t and the matrix J -1 ∗ after one block update in Algorithm 2.

Lemma 4.6. Performing Algorithm 2 under Assumptions 2.1, 2.2 and 4.2 and suppose the sequence { x t } ∞ t =0 generated by Algorithm 2 satisfies that ∥ x t -x ∗ ∥ 2 ≤ µ 2 / (6 LM ) , we have

We can establish the superlinear convergence of the block bad Broyden's method based on Lemma 4.6.

Theorem 4.7. Suppose Assumptions 2.1, 2.2 and 4.2 hold and the initial condition of Algorithm 2 satisfies

for arbitrary q ∈ (0 , 1) . Then for k ∈ [ d ] , the output of Algorithm 2 satisfies

Similar to Corollary 4.5, we can also obtain the high probability bound for Algorithm 2.

Corollary 4.8. Performing Algorithm 2 under the same assumption and initial condition as Theorem 4.7, with probability at least 1 -δ , we have

and and

5 Discussion

In this section, we discuss the performance difference between the good and bad Broyden's methods which is considered as an important open problem in the field of nonlinear equations [35].

We first discuss the different performance of the block Broyden's methods (Algorithm 1 and 2). Notice that the 'good' method enjoys a condition-number-free superlinear rate of O ((1 -k/d ) t ( t -1) / 4 ) and the initial conditions of B 0 and x 0 are ∥ J -1 ∗ ( B 0 -J ∗ ) ∥ F = O (1) and ∥ x 0 -x ∗ ∥ 2 = O ( µ M √ d ) respectively. On the other hand, both the superlinear rate O ((1 -k/ (4 dκ 2 )) t ( t -1) / 4 ) and initial conditions ∥ J ∗ ( H 0 -J -1 ∗ ) ∥ F = O (min { 1 , √ d/κ } ) , ∥ J ∗ ( x 0 -x ∗ ) ∥ 2 = O ( µ 2 / ( M √ d )) for H 0 , x 0 of the 'bad' method depend on κ heavily. Thus we think these two block Broyden's methods are suitable for different scenarios:

The 'good' method is more suitable for the cases of large condition number ( κ ≫ 1 ) because its convergence rate is condition-number-free and its initial condition has weaker dependency on κ than the 'bad' method.

The 'bad' method may have better performance when κ = O (1) because under this case the convergence rates do not differ much between the 'good' and 'bad' method while the latter one usually has a cheaper computational cost per iteration.

The condition number is very large in most of the cases which means the 'good' method generally outperforms the 'bad' one. We summarize the different convergence rates, initial conditions and suitable scenes of the block good and bad Broyden's methods in Table 3.

The similar phenomenon also holds for the classical good and bad Broyden's methods [29], whose iterations can be reformulated as

and

respectively, where u t = x t +1 -x t , ˆ J t +1 = ∫ 1 0 J ( x t + s u t )d s and y t = F ( x t +1 ) -F ( x t ) . The different convergent behavior of the block Broyden's updates helps us understand the performance difference between the classical good and bad Broyden's methods for the similarity of their frameworks.

6 Experiments

We validate our methods on the Chandrasekhar H-equation which is well studied in the previous literature [25, 29, 50] as follows

where x = [ x 1 , · · · , x N ] ⊤ ∈ R N and F ( x ) = [ F 1 ( x ) , · · · , F N ( x )] ⊤ ∈ R N . We denote GB-Cl and BB-Cl as the classical good and bad Broyden's methods respectively [1, 29]. We denote GB-Gr and GB-Ra as the greedy and randomized Broyden's methods [50] respectively. Our experiments are conducted on a PC with Apple M1 and all algorithms are implemented in Python 3.8.12.

Our first experiment considers three cases: N = 200 , N = 300 , N = 400 . We set c = 1 -10 -12 for the H-equation and choose the block size k = N/ 10 for the proposed methods. In all cases, we use the same inputs B 0 = 0 . 1 I N ( H 0 = 10 I N ) for all algorithms. We use classical Newton method as the warm-up algorithm to obtain x 0 which satisfies the local condition and take it as the

0

150

300

10

13

10

9

10

5

10

1

GB-Cl

BB-Cl

GB-Gr

BGB

BBB

0

200

400

10

13

10

9

10

5

10

1

0

300

600

10

13

10

9

10

5

10

1

(a)

N

= 200

(iteration)

(b)

N

= 300

(iteration)

(c)

N

= 400

(iteration)

0.0

0.1

0.2

10

13

10

9

10

5

10

1

0.0

0.2

0.4

10

13

10

9

10

5

10

1

0.0

0.3

0.6

10

13

10

9

10

5

10

1

(d)

N

= 200

(time)

(e)

N

= 300

(time)

(f)

N

= 400

(time)

initial point for all methods. We compare the proposed BGB and BBB algorithm with baselines and present the results of iteration number against ∥ F ( x ) ∥ 2 and running time against ∥ F ( x ) ∥ 2 in Figure 1. We observe that the proposed block good Broyden's method (BGB) outperforms the baselines in all cases, but the block bad Broyden's method (BBB) does not perform very well. This is mainly because κ is very large in this setting ( κ ≈ 10 6 ). We also note that the classical Broyden's methods (GB-Cl and BB-Cl) are numerical unstable. Specifically, they do not guarantee the descent of ∥ F ( x t ) ∥ 2 and encounter nan value during the iterations. The BB-Cl algorithm even fails to converge after some iterations. Such instability of the classical Broyden's methods is also observed in the previous literature [29, 50].

Our second experiment explores the performance of the proposed block Broyden's methods with different block size. We also study whether BBB algorithm has good performance for the nonlinear equation which Jacobian of the solution small condition number. By fixing N = 400 and setting c = { 1 -10 -1 , 1 -10 -3 , 1 -10 -5 } , we obtain different condition numbers of (21) as κ = 2 , 31 , 327 . We present the results in Figure 2. For each κ , we also vary the block size k = { 1 , 10 , 100 } for BGB and BBB algorithms. We observe that when κ = O (1) , BBB outperforms BGB in terms of the CPU time (Figure 2 (d), (e)). which matches our analysis in section 5. We also find that larger block size k will lead to faster convergence in terms of the iterations ((a), (b), (c) of Figure 2), which verifies our theoretical results in section 4.2.

7 Conclusion

In this paper, we have proposed the block Broyden's methods for solving nonlinear equations. The proposed block good Broyden's method enjoys a faster superlinear rate than all of the existing Broyden's methods. We have also shown that the block bad Broyden's update approximates the inverse of the object matrix with an explicit linear rate and proposed the block bad Broyden's method accordingly. The established convergence results for the block good and bad methods bring us new understanding on the performance difference between the good and bad Broyden's methods. Especially, they can explain why good Broyden's method generally outperforms the 'bad' one.

For the future work, it is possible to incorporate the safeguard mechanism in Wang et al. [48] to remove the assumption on the Jacobian estimator (Assumption 4.2). It will also be interesting to study the global behavior based on the recent advance in Jiang et al. [23] and design efficient stochastic or sketched algorithms [48, 49, 52] for solving nonlinear equations.

0

3

6

10

14

10

10

10

6

10

2

BBB,

k

=1

BBB,

k

=10

BBB,

k

=100

BGB,

k

=1

BGB,

k

=10

BGB,

k

=100

0

6

12

10

14

10

10

10

6

10

2

0

40

80

10

14

10

10

10

6

10

2

(a)

κ

= 2

(iteration)

(b)

κ

= 31

(iteration)

(c)

κ

= 327

(iteration)

0.000

0.008

0.016

10

14

10

10

10

6

10

2

0.00

0.05

0.10

10

14

10

10

10

6

10

2

0.0

0.1

0.2

10

14

10

10

10

6

10

2

(d)

κ

= 2

(time)

(e)

κ

= 31

(time)

(f)

κ

= 327

(time)

References

Mehiddin Al-Baali, Emilio Spedicato, and Francesca Maggioni. Broyden's quasi-Newton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems. Optimization Methods and Software , 29(5):937-954, 2014.

Lucian-Liviu Albu. Non-linear models: applications in economics. Available at SSRN 1565345 , 2006.

Meysam Alizamir, Sungwon Kim, Ozgur Kisi, and Mohammad Zounemat-Kermani. A comparative study of several machine learning based non-linear regression methods in estimating solar radiation: Case studies of the usa and turkey regions. Energy , 197:117239, 2020.

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems , 32, 2019.

Eloıse Berthier, Justin Carpentier, and Francis Bach. Fast and robust stability region estimation for nonlinear dynamical systems. In 2021 European Control Conference (ECC) , pages 14121419. IEEE, 2021.

Charles G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation , 19(92):577-593, 1965.

Charles G. Broyden. Quasi-Newton methods and their application to function minimisation. Mathematics of Computation , 21(99):368-381, 1967.

Charles G. Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics , 6(1):76-90, 1970.

Charles G. Broyden. The convergence of a class of double-rank minimization algorithms: 2. the new algorithm. IMA journal of applied mathematics , 6(3):222-231, 1970.

Charles G. Broyden. On the discovery of the 'good Broyden' method. Mathematical programming , 87:209-213, 2000.

Charles G. Broyden, J. E. Dennis, and Jorge J. Mor´ e. On the local and superlinear convergence of quasi-Newton methods. IMA Journal of Applied Mathematics , 12(3):223-245, 1973.

Caterina Buizza, C´ esar Quilodr´ an Casas, Philip Nadler, Julian Mack, Stefano Marrone, Zainab Titus, Cl´ emence Le Cornec, Evelyn Heylen, Tolga Dur, Luis Baca Ruiz, et al. Data learning: integrating data assimilation and machine learning. Journal of Computational Science , 58: 101525, 2022.

Richard H. Byrd, Jorge Nocedal, and Ya-Xiang Yuan. Global convergence of a cass of quasiNewton methods on convex problems. SIAM Journal on Numerical Analysis , 24(5):11711190, 1987.

Timothy A. Davis. Block matrix methods: Taking advantage of high-performance computers. Technical report, Technical Report TR-98-024, 1998.

Alexandre D´ efossez and Francis Bach. Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions. In Artificial Intelligence and Statistics , pages 205-213. PMLR, 2015.

John E. Dennis Jr and Robert B. Schnabel. Numerical methods for unconstrained optimization and nonlinear equations . SIAM, 1996.

Jin-yan Fan and Ya-Xiang Yuan. On the quadratic convergence of the Levenberg-Marquardt method without nonsingularity assumption. Computing , 74:23-39, 2005.

J. Frehse and A. Bensoussan. Nonlinear elliptic systems in stochastic game theory. Journal f¨ ur die reine und angewandte Mathematik , 350:23-67, 1984.

Philip E. Gill and Walter Murray. Algorithms for the solution of the nonlinear least-squares problem. SIAM Journal on Numerical Analysis , 15(5):977-992, 1978.

Nick Gould, Dominique Orban, and Philippe Toint. Numerical methods for large-scale nonlinear optimization. Acta Numerica , 14:299-361, 2005.

Robert M. Gower and Peter Richt´ arik. Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms. arXiv preprint arXiv:1602.01768 , 2016.

Andreas Griewank. Broyden updating, the good and the bad. Optimization Stories, Documenta Mathematica. Extra Volume: Optimization Stories , pages 301-315, 2012.

Ruichen Jiang, Qiujiang Jin, and Aryan Mokhtari. Online learning guided curvature approximation: A quasi-Newton method with global non-asymptotic superlinear convergence. In Annual Conference Computational Learning Theory , 2023.

Qiujiang Jin and Aryan Mokhtari. Non-asymptotic superlinear convergence of standard quasiNewton methods. Mathematical Programming , pages 1-49, 2022.

Carl T. Kelley. Iterative methods for linear and nonlinear equations . SIAM, 1995.

Carl T. Kelley. Solving nonlinear equations with Newton's method . SIAM, 2003.

Carl T. Kelley and Ekkehard W. Sachs. A new proof of superlinear convergence for Broyden's method in Hilbert space. SIAM Journal on Optimization , 1(1):146-150, 1991.

Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics , 2(2):164-168, 1944.

Dachao Lin, Haishan Ye, and Zhihua Zhang. Explicit superlinear convergence rates of Broyden's methods in nonlinear equations. arXiv preprint arXiv:2109.01974 , 2021.

Dachao Lin, Haishan Ye, and Zhihua Zhang. Explicit convergence rates of greedy and random quasi-Newton methods. Journal of Machine Learning Research , 23(162):1-40, 2022.

Chengchang Liu and Luo Luo. quasi-Newton methods for saddle point problems. Advances in Neural Information Processing Systems , 35:3975-3987, 2022.

Chengchang Liu, Shuxian Bi, Luo Luo, and John CS Lui. Partial-quasi-Newton methods: Efficient algorithms for minimax optimization problems with unbalanced dimensionality. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1031-1041, 2022.

Chengchang Liu, Cheng Chen, and Luo Luo. Symmetric rankk methods. arXiv preprint arXiv:2303.16188 , 2023.

Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics , 11(2):431-441, 1963.

Jos´ e Mario Martınez. Practical quasi-Newton methods for solving nonlinear systems. Journal of computational and Applied Mathematics , 124(1-2):97-121, 2000.

Konstantin Mishchenko. Regularized Newton method with global O (1 /k 2 ) convergence. arXiv preprint arXiv:2112.02089 , 2021.

Jorge J. Mor´ e. A collection of nonlinear model problems. Technical report, Argonne National Lab., IL (USA), 1989.

Yu. Nesterov. Modified Gauss-Newton scheme with worst case guarantees for global performance. Optimisation methods and software , 22(3):469-483, 2007.

Mojtaba Nourian and Peter E. Caines. ϵ -Nash mean field game theory for nonlinear stochastic dynamical systems with major and minor agents. SIAM Journal on Control and Optimization , 51(4):3302-3331, 2013.

M.J.D. Powell. A hybrid method for nonlinear equations. Numerical Methods for Nonlinear Algebraic Equations , 1970.

Anton Rodomanov and Yurii Nesterov. Greedy quasi-Newton methods with explicit superlinear convergence. SIAM Journal on Optimization , 31(1):785-811, 2021.

Anton Rodomanov and Yurii Nesterov. New results on superlinear convergence of classical quasi-Newton methods. Journal of optimization theory and applications , 188(3):744-769, 2021.

Anton Rodomanov and Yurii Nesterov. Rates of superlinear convergence for classical quasiNewton methods. Mathematical Programming , pages 1-32, 2021.

Damien Scieur, Edouard Oyallon, Alexandre d'Aspremont, and Francis Bach. Online regularized nonlinear acceleration. arXiv preprint arXiv:1805.09639 , 2018.

David F. Shanno. Conditioning of quasi-Newton methods for function minimization. Mathematics of computation , 24(111):647-656, 1970.

Jack Sherman and Winifred J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics , 21(1): 124-127, 1950.

Ph L. Toint. On large scale nonlinear least squares calculations. SIAM Journal on Scientific and Statistical Computing , 8(3):416-435, 1987.

Xiao Wang, Shiqian Ma, Donald Goldfarb, and Wei Liu. Stochastic quasi-Newton methods for nonconvex stochastic optimization. SIAM Journal on Optimization , 27(2):927-956, 2017.

Xiaoyu Wang, Xiao Wang, and Ya-Xiang Yuan. Stochastic proximal quasi-Newton methods for non-convex composite optimization. Optimization Methods and Software , 34(5):922-948, 2019.

Haishan Ye, Dachao Lin, and Zhihua Zhang. Greedy and random Broyden's methods with explicit superlinear convergence rates in nonlinear equations. arXiv preprint arXiv:2110.08572 , 2021.

Haishan Ye, Dachao Lin, Xiangyu Chang, and Zhihua Zhang. Towards explicit superlinear convergence rate for SR1. Mathematical Programming , pages 1-31, 2022.

Rui Yuan, Alessandro Lazaric, and Robert M. Gower. Sketched Newton-Raphson. SIAM Journal on Optimization , 32(3):1555-1583, 2022.

Ya-Xiang Yuan. Trust region algorithms for nonlinear equations. Information , 1:7-20, 1998.

Ya-Xiang Yuan. Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numerical algebra, control & optimization , 1(1):15, 2011.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors introduce block variants of both good and bad Broyden's methods, which exhibit explicit local superlinear convergence rates. The block good Broyden's method, in particular, demonstrates a faster convergence rate compared to existing Broyden's methods that are not dependent on the condition number. This is achieved by leveraging multiple rank modifications on the Jacobian estimator. On the other hand, the block bad Broyden's method directly estimates the inverse of the Jacobian, resulting in reduced computational costs during the iteration process. The theoretical findings offer new insights into why the good Broyden's method tends to outperform the bad Broyden's method in most cases. Empirical results further validate the superiority of the proposed methods and affirm the theoretical analysis conducted by the authors.\n\nSTRENGTHS:\n* The authors provide explicit convergence rates for the block good Broyden’s update and the block bad Broyden’s update\n* The block good Broyden’s update can approximate a nonsingular matrix A with a linear rate of $(1-k/d)^t$  and the “bad” update can approximate the inverse matrix $A^{−1}$ with an linear rate of $(1-k/(d \\hat{\\kappa}^2))^t$\n* They propose the first explicit convergence rate for the block bad Broyden’s update.\n* The assumptions are supported by theory and experiments.\n\nWEAKNESSES:\nIn the experiment section Figure 1 and Figure 2 are a little bit confusing as for example Figure 1(a) and Figure 1(d) represents the experiments for the same N so they can be grouped under the same subfigure.\n\nQUESTIONS:\n* Can you compare the proposed method with the results presented by [1]?\n*  Why did you choose Chandrasekhar H-equation to conduct the experiments for the given method? \n\n\n[1] Robert M Gower and Peter Richtarik. Randomized quasi-newton updates are linearly convergent matrix inversion algorithms. arXiv preprint arXiv:1602.01768, 2016.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the Broyden family quasi-Newton method into block setting and shows explicit local linear convergence rate under mild conditions. More specifically, the authors studied both the “good” and “bad” Broyden algorithms, namely the update on the Hessian/Jacobian and the inverse Hessian/Jacobian respectively. The authors provided some insights on why the “good” update is better than the “bad” update and provided numerical experiments to support their findings.\n\nSTRENGTHS:\nThe theory is well-rounded and the method is consistent with existing works. The analysis on Algorithm 2 is interesting which incorporates the condition number of the Jacobian and brings insights on why the convergence of “bad” Broyden method is worse in practice.\n\nWEAKNESSES:\n(Please reply to the Questions section directly) First the “good” Broyden family still computes the inverse of matrix in the update; Second, the Assumption 4.1 is imposed toward the iterates directly, which is not very good; Third, it’s not very clear why we consider the block update. Also the numerical experiment is not adequate.\n\nQUESTIONS:\n1. Perhaps the biggest question I have toward the comparison of “good” and “bad” Broyden is that for the good Broyden method, still it computes the inverse of $B_{t}^{-1}$, which means that the dependency on the condition number is implicitly incorporated. I’d appreciate to hear from the author on how this problem is addressed (and how previous literature deals with this problem);\n2. In assumption 4.1, the assumption is imposed toward the sequence $\\{B_{t}\\}$, which is not very good. Is there a possible safeguard mechanism so that the Jacobians are well-defined? This is pretty common in quasi-Newton literatures, such as [1].\n3. Another problem is that the block update seems to be lack of motivations. Isn’t the case $k=1$ already enough to show the dependency of $\\kappa$ for the “bad” Broyden update? Certainly the $k$ in each of the convergence result and the numerical experiments show the efficiency of block updates, but usually block updates are for bigger targets such as parallelization or decentralized update. Could the authors give some comment on this direction;\n4. The numerical experiments on Chandrasekhar H-equation is not sufficient to show the efficiency of the proposed method. It would be interesting to see applications and experiments on problems with real-world data. In particular, would the “bad” Broyden method be better when the problem dimension is relatively large?\n\n\nReferences:\n[1] Wang, Xiao, et al. \"Stochastic quasi-Newton methods for nonconvex stochastic optimization.\" SIAM Journal on Optimization 27.2 (2017): 927-956.\n\nLIMITATIONS:\nThe limitation is well stated in weakness and question sections.\n\nI’m not aware of any potential negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies block Broyden's methods for solving nonlinear equation systems and presents explicit local superlinear convergence rates. For the block good Broyden's method, the convergence rate is independent of the condition number of the Jacobian matrix at the solution, which that of the block bad Broyden's method depends on the condition number heavily. Numerical experiments validate the theoretical analysis.\n\nSTRENGTHS:\n- The authors provided explicit local superlinear convergence rates for  block good Broyden's method and block bad Broyden's method. The rate improves previous results and reveals the advantage of block update.\n\n- The established convergence results give new understanding on the performance difference between the good and bad Broyden's methods. \n\n- The paper is clearly written and well-organized.\n\nWEAKNESSES:\n- In Algotithm 1 and Algotithm 2, the Jacobian matrix is explicitly needed even when $k=1$, while that is not the case for classic Broyden's methods. \n- The convergence rate depends on the dimension. When $d\\gg 1$, the rate $1-1/d$  is close to $1$ and the convergence will be slow. \n- Essentially the algorithms and analysis is the block version of previous work [38]. Although the comparison with [38] is include, it is still not so clear if this generalization (from rank $1$ to rank $k$ ) is straight-forward.\n\nQUESTIONS:\n- The rank $k$ is not defined in the contribution 1 when it is firstly used.\n\n- What is the definition of $\\hat{\\kappa}$ in line 51, equation (8) and Table 2?\n\n- I suggest to clarify the meaning of $\\kappa$ in line 59. \n\n- In line 78, $x^*$ is defined as 'the solution'. I suggest to clarify the uniqueness of the solution for the problem (1) considered. The nonlinear equation (1) may have multiple solutions.\n\n- What is the meaning of the bracket notation, for example that used in Table 1?\n\n- What is the meaning of $\\operatorname{e}$'s in equation (11), (12), (17) and  (18)?\n\n- Typos: line 24, 'large-scale'; it seems to be '$d$' (other than '$\\operatorname{d}$') in  equation (11), (17) and (18).  \n\n- Typos: line 94 summarized **in** Table 2.\n\n- Typos: line 150, our BGB algorithm is better **than** greedy ...\n\n- Please check the format of the reference list, particularly the letter case (in [3], [5], [9], [13]) and the math symbol (in [15]), etc.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes variants of block Broyden's methods for solving nonlinear equations. Explicit convergence rates are established, and the numerical experiment validates the theoretical analysis.\n\nSTRENGTHS:\nThe paper provides explicit convergence rates of the proposed block Broyden's methods. The theoretical analysis is sound, and the results improve some previous results related to Broyden's methods. The claims are supported by the numerical results.\n\nWEAKNESSES:\nAlthough this paper makes a contribution to the theoretical study of Broyden's method, there are some outstanding weaknesses that may outweigh the strengths.\n\n1. The novelty of the proposed block Broyden's method is limited. The proposed methods are very similar to the ones given in [1] (Section 7 and Table 8.1 in [1]). The distinction should be clearly explained in the main paper.\n\n2. The theoretical analysis is limited. Only local convergence is established. There is no discussion about the global convergence.\n\n3. The implementation details of the algorithms are totally missing. It is likely that these algorithms cannot be efficiently applied to real-world applications. For example, each step in Algorithm 1 and Algorithm 2 needs information about the Jacobian matrix, which can incur significant overhead compared with the classical quasi-Newton methods based on secant equations (as shown in Figure 1).\n\n4. The memory cost can be one of the biggest obstacles for the proposed methods to solving high-dimensional problems since the approximate Jacobian matrix or the approximate inverse Jacobian matrix of dimension $d^2$ needs to be maintained. However, there is no discussion about this severe issue.\n\n5. The experimental part needs to be improved. The algorithms were only tested on a simple problem, so it is desirable to consider more different problems to make the claims more convincing. Besides, some well-known methods, e.g., the Jacobian-free Newton-Krylov method, were not compared.\n\nRegarding to the writing, it is better to reorganize the materials to focus on the main contributions. For example, it is rather strange to analyze the convergence of the Block Broyden's update first in Section 3, which is not the algorithm proposed in this paper.\n\nSome typos:\n1. Line 29: approximate → approximates, update → updates\n\n2. Line 150: better → better than\n\n3. Line 209: The book [31] has no content about the Chandrasekhar H-equation.\n\n4. Line 354: The equality is not correct.\n\n[1] R.M. Gower and P. Richtarik. Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms. SIAM J. Matrix Anal. Appl., 2017.\n\nQUESTIONS:\n1. The Assumption 4.1 seems to be a strong assumption. Is it possible that the nonsingularity of the matrices is guaranteed by the iterative schemes themselves?\n\n2. The condition (10) and condition (16) require that the initial Jacobian approximation is sufficiently close to the exact Jacobian matrix. Is it a realistic assumption?\n\nLIMITATIONS:\nThe biggest limitation of the proposed algorithms is the large memory usage and high computational cost, but there is no discussion of this issue. It is likely that these algorithms are not suitable for solving large-scale nonlinear equations in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}