SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can establish the superlinear convergence of the block bad Broyden's method based on Lemma 4.6.
Theorem 4.7. Suppose Assumptions 2.1, 2.2 and 4.2 hold and the initial condition of Algorithm 2 satisfies
$$
\frac {4 M \sqrt {d} R _ {0}}{\mu^ {2}} \leq \min  \left\{\frac {1 - q}{4}, \frac {q}{2}, \frac {\sqrt {d}}{3 \kappa} \right\} \quad a n d \quad \tau_ {0} \leq \frac {q}{2} \tag {16}
$$
for arbitrary  $q \in (0,1)$ . Then for  $k \in [d]$ , the output of Algorithm 2 satisfies
$$
\mathbb {E} \left[ \| \mathbf {J} _ {*} (\mathbf {H} _ {t} - \mathbf {J} _ {*} ^ {- 1}) \| _ {F} \right] \leq \mathrm {e} \left(1 - \frac {k}{4 d \kappa^ {2}}\right) ^ {t / 2},
$$
$$
\mathbb {E} \left[ \frac {\| \mathbf {J} _ {*} (\mathbf {x} _ {t + 1} - \mathbf {x} _ {*}) \| _ {2}}{\| \mathbf {J} _ {*} (\mathbf {x} _ {t} - \mathbf {x} _ {*}) \| _ {2}} \right] \leq 2 \mathrm {e} \left(1 - \frac {k}{4 d \kappa^ {2}}\right) ^ {t / 2}.
$$
Similar to Corollary 4.5, we can also obtain the high probability bound for Algorithm 2.
Corollary 4.8. Performing Algorithm 2 under the same assumption and initial condition as Theorem 4.7, with probability at least  $1 - \delta$ , we have
$$
\left\| \mathbf {J} _ {*} \left(\mathbf {H} _ {t} - \mathbf {J} _ {*} ^ {- 1}\right) \right\| _ {F} \leq \frac {8 \mathrm {e} d ^ {2} \kappa^ {4}}{\delta k ^ {2}} \left(1 - \frac {k}{4 d \kappa^ {2} + k}\right) ^ {t / 2}, \tag {17}
$$
$$
\left\| \mathbf {J} _ {*} \left(\mathbf {x} _ {t} - \mathbf {x} _ {*}\right) \right\| _ {2} \leq \left(\frac {1 6 \mathrm {e} d ^ {2} \kappa^ {4}}{k ^ {2} \delta}\right) ^ {t} \left(1 - \frac {k}{4 d \kappa^ {2} + k}\right) ^ {t (t - 1) / 4} \| \mathbf {J} _ {*} \left(\mathbf {x} _ {0} - \mathbf {x} _ {*}\right) \| _ {2}. \tag {18}
$$
In this section, we discuss the performance difference between the good and bad Broyden's methods which is considered as an important open problem in the field of nonlinear equations [35].
We first discuss the different performance of the block Broyden's methods (Algorithm 1 and 2). Notice that the "good" method enjoys a condition-number-free superlinear rate of  $\mathcal{O}((1 - k / d)^{t(t - 1) / 4})$  and the initial conditions of  $\mathbf{B}_0$  and  $\mathbf{x}_0$  are  $\| \mathbf{J}_*^{-1}(\mathbf{B}_0 - \mathbf{J}_*)\| _F = \mathcal{O}(1)$  and  $\| \mathbf{x}_0 - \mathbf{x}_*\| _2 = \mathcal{O}\left(\frac{\mu}{M\sqrt{d}}\right)$  respectively. On the other hand, both the superlinear rate  $\mathcal{O}((1 - k / (4d\kappa^2))^{t(t - 1) / 4})$  and initial conditions  $\| \mathbf{J}_*(\mathbf{H}_0 - \mathbf{J}_*^{-1})\| _F = \mathcal{O}(\min \{1,\sqrt{d} /\kappa \})$ ,  $\| \mathbf{J}_*(\mathbf{x}_0 - \mathbf{x}_*)\| _2 = \mathcal{O}(\mu^2 /(M\sqrt{d}))$  for  $\mathbf{H}_0$ ,  $\mathbf{x}_0$  of the "bad" method depend on  $\kappa$  heavily. Thus we think these two block Broyden's methods are suitable for different scenarios:
Table 3: Comparison between block good and bad Broyden's methods where  $r_0 = \|\mathbf{x}_0 - \mathbf{x}_*\|_2$ ,  $\sigma_0 = \|\mathbf{J}_*(\mathbf{B}_0 - \mathbf{J}_*)\|_F$ ,  $R_0 = \|\mathbf{J}_*(\mathbf{x}_0 - \mathbf{x}_*)\|_2$  and  $\tau_0 = \|\mathbf{J}_*(\mathbf{H}_0 - \mathbf{J}_*^{-1})\|_F$ .
Block Good Broyden's Method Algorithm 1	Block Bad Broyden's Method Algorithm 2
Initial Condition	M√dr0/μ = O(1), σ0 = O(1)	M√dR0/μ2 = O(1), τ0 = O(1 ∧ √d/κ)
Superlinear Rate	O((1 - k/d)t(t-1)/4)	O((1 - k/4dκ2)t(t-1)/4)
Suitable Scene	κ ≫ 1	κ = O(1)
- The "good" method is more suitable for the cases of large condition number ( $\kappa \gg 1$ ) because its convergence rate is condition-number-free and its initial condition has weaker dependency on  $\kappa$  than the "bad" method.
- The 'bad' method may have better performance when  $\kappa = \mathcal{O}(1)$  because under this case the convergence rates do not differ much between the "good" and "bad" method while the latter one usually has a cheaper computational cost per iteration.
The condition number is very large in most of the cases which means the "good" method generally outperforms the "bad" one. We summarize the different convergence rates, initial conditions and suitable scenes of the block good and bad Broyden's methods in Table 3.
The similar phenomenon also holds for the classical good and bad Broyden's methods [29], whose iterations can be reformulated as
$$
\left\{ \begin{array}{l} \mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \mathbf {B} _ {t} ^ {- 1} \mathbf {F} (\mathbf {x} _ {t}), \\ \mathbf {B} _ {t + 1} = \text {B l o c k - G - B r o y d e n} \left(\mathbf {B} _ {t}, \hat {\mathbf {J}} _ {t + 1}, \mathbf {u} _ {t}\right) = \mathbf {B} _ {t} + \frac {\left(\mathbf {y} _ {t} - \mathbf {B} _ {t} \mathbf {u} _ {t}\right) \mathbf {u} _ {t} ^ {\top}}{\mathbf {u} _ {t} ^ {\top} \mathbf {u} _ {t}} \quad , \end{array} \right. \tag {19}
$$
$$
\left\{ \begin{array}{l} \mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \mathbf {H} _ {t} \mathbf {F} (\mathbf {x} _ {t}), \\ \mathbf {H} _ {t + 1} = \text {B l o c k - B - B r o y d e n} \left(\mathbf {H} _ {t}, \hat {\mathbf {J}} _ {t + 1}, \mathbf {u} _ {t}\right) = \mathbf {H} _ {t} + \frac {\left(\mathbf {u} _ {t} - \mathbf {H} _ {t} \mathbf {y} _ {t}\right) \mathbf {y} _ {t} ^ {\top}}{\mathbf {y} _ {t} ^ {\top} \mathbf {y} _ {t}} \end{array} \right. \tag {20}
$$
respectively, where  $\mathbf{u}_t = \mathbf{x}_{t + 1} - \mathbf{x}_t$ ,  $\hat{\mathbf{J}}_{t + 1} = \int_0^1\mathbf{J}(\mathbf{x}_t + s\mathbf{u}_t)\mathrm{d}s$  and  $\mathbf{y}_t = \mathbf{F}(\mathbf{x}_{t + 1}) - \mathbf{F}(\mathbf{x}_t)$ . The different convergent behavior of the block Broyden's updates helps us understand the performance difference between the classical good and bad Broyden's methods for the similarity of their frameworks.
We validate our methods on the Chandrasekhar H-equation which is well studied in the previous literature [25, 29, 50] as follows
$$
F _ {i} (\mathbf {x}) = x _ {i} - \left(1 - \frac {c}{2 N} \sum_ {j = 1} ^ {N} \frac {\mu_ {i} x _ {j}}{\mu_ {i} + \mu_ {j}}\right) ^ {- 1}, \tag {21}
$$
where  $\mathbf{x} = [x_1, \dots, x_N]^\top \in \mathbb{R}^N$  and  $\mathbf{F}(\mathbf{x}) = [F_1(\mathbf{x}), \dots, F_N(\mathbf{x})]^\top \in \mathbb{R}^N$ . We denote GB-Cl and BB-Cl as the classical good and bad Broyden's methods respectively [1, 29]. We denote GB-Gr and GB-Ra as the greedy and randomized Broyden's methods [50] respectively. Our experiments are conducted on a PC with Apple M1 and all algorithms are implemented in Python 3.8.12.
Our first experiment considers three cases:  $N = 200$ ,  $N = 300$ ,  $N = 400$ . We set  $c = 1 - 10^{-12}$  for the H-equation and choose the block size  $k = N / 10$  for the proposed methods. In all cases, we use the same inputs  $\mathbf{B}_0 = 0.1\mathbf{I}_N$  ( $\mathbf{H}_0 = 10\mathbf{I}_N$ ) for all algorithms. We use classical Newton method as the warm-up algorithm to obtain  $\mathbf{x}_0$  which satisfies the local condition and take it as the
(a)  $N = 200$  (iteration)
(b)  $N = 300$  (iteration)
(c)  $N = 400$  (iteration)
(d)  $N = 200$  (time)
(e)  $N = 300$  (time)
(f)  $N = 400$  (time)
Figure 1: We demonstrate iteration numbers vs.  $\| \mathbf{F}(\mathbf{x})\| _2$  and CPU time (second) vs.  $\| \mathbf{F}(\mathbf{x})\| _2$  for H-equation with different equation numbers  $N$ .
initial point for all methods. We compare the proposed BGB and BBB algorithm with baselines and present the results of iteration number against  $\| \mathbf{F}(\mathbf{x})\| _2$  and running time against  $\| \mathbf{F}(\mathbf{x})\| _2$  in Figure 1. We observe that the proposed block good Broyden's method (BGB) outperforms the baselines in all cases, but the block bad Broyden's method (BBB) does not perform very well. This is mainly because  $\kappa$  is very large in this setting ( $\kappa \approx 10^{6}$ ). We also note that the classical Broyden's methods (GB-Cl and BB-Cl) are numerical unstable. Specifically, they do not guarantee the descent of  $\| \mathbf{F}(\mathbf{x}_t)\| _2$  and encounter nan value during the iterations. The BB-Cl algorithm even fails to converge after some iterations. Such instability of the classical Broyden's methods is also observed in the previous literature [29, 50].
Our second experiment explores the performance of the proposed block Broyden's methods with different block size. We also study whether BBB algorithm has good performance for the nonlinear equation which Jacobian of the solution small condition number. By fixing  $N = 400$  and setting  $c = \{1 - 10^{-1}, 1 - 10^{-3}, 1 - 10^{-5}\}$ , we obtain different condition numbers of (21) as  $\kappa = 2, 31, 327$ . We present the results in Figure 2. For each  $\kappa$ , we also vary the block size  $k = \{1, 10, 100\}$  for BGB and BBB algorithms. We observe that when  $\kappa = \mathcal{O}(1)$ , BBB outperforms BGB in terms of the CPU time (Figure 2 (d), (e)). which matches our analysis in section 5. We also find that larger block size  $k$  will lead to faster convergence in terms of the iterations ((a), (b), (c) of Figure 2), which verifies our theoretical results in section 4.2.
In this paper, we have proposed the block Broyden's methods for solving nonlinear equations. The proposed block good Broyden's method enjoys a faster superlinear rate than all of the existing Broyden's methods. We have also shown that the block bad Broyden's update approximates the inverse of the object matrix with an explicit linear rate and proposed the block bad Broyden's method accordingly. The established convergence results for the block good and bad methods bring us new understanding on the performance difference between the good and bad Broyden's methods. Especially, they can explain why good Broyden's method generally outperforms the "bad" one.
For the future work, it is possible to incorporate the safeguard mechanism in Wang et al. [48] to remove the assumption on the Jacobian estimator (Assumption 4.2). It will also be interesting to study the global behavior based on the recent advance in Jiang et al. [23] and design efficient stochastic or sketched algorithms [48, 49, 52] for solving nonlinear equations.
(a)  $\kappa = 2$  (iteration)
(b)  $\kappa = 31$  (iteration)
(c)  $\kappa = 327$  (iteration)
(d)  $\kappa = 2$  (time)
(e)  $\kappa = 31$  (time)
(f)  $\kappa = 327$  (time)
Figure 2: We demonstrate iteration numbers vs.  $\| \mathbf{F}(\mathbf{x})\| _2$  and CPU time (second) vs.  $\| \mathbf{F}(\mathbf{x})\| _2$  for H-equation with different condition number  $\kappa$ .
We would like to thank Haishan Ye for valuable discussion. Cheng Chen is supported by National Natural Science Foundation of China (No. 62306116) and the Dean's fund of Shanghai Key Laboratory of Trustworthy Computing. Luo Luo is supported by National Natural Science Foundation of China (No. 62206058) and Shanghai Sailing Program (22YF1402900). John C.S. Lui is supported in part by the Hong Kong GRC 14215722.
[1] Mehiddin Al-Baali, Emilio Spedicato, and Francesca Maggioni. Broyden's quasi-Newton methods for a nonlinear system of equations and unconstrained optimization: a review and open problems. Optimization Methods and Software, 29(5):937–954, 2014.
[2] Lucian-Liviu Albu. Non-linear models: applications in economics. Available at SSRN 1565345, 2006.
[3] Meysam Alizamir, Sungwon Kim, Ozgur Kisi, and Mohammad Zounemat-Kermani. A comparative study of several machine learning based non-linear regression methods in estimating solar radiation: Case studies of the usa and turkey regions. Energy, 197:117239, 2020.
[4] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
[5] Eloise Berthier, Justin Carpentier, and Francis Bach. Fast and robust stability region estimation for nonlinear dynamical systems. In 2021 European Control Conference (ECC), pages 1412-1419. IEEE, 2021.
[6] Charles G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 19(92):577-593, 1965.
[7] Charles G. Broyden. Quasi-Newton methods and their application to function minimisation. Mathematics of Computation, 21(99):368-381, 1967.
[8] Charles G. Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 1970.
[9] Charles G. Broyden. The convergence of a class of double-rank minimization algorithms: 2. the new algorithm. IMA journal of applied mathematics, 6(3):222-231, 1970.
[31] Chengchang Liu and Luo Luo. quasi-Newton methods for saddle point problems. Advances in Neural Information Processing Systems, 35:3975–3987, 2022.
[32] Chengchang Liu, Shuxian Bi, Luo Luo, and John CS Lui. Partial-quasi-Newton methods: Efficient algorithms for minimax optimization problems with unbalanced dimensionality. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1031–1041, 2022.
[33] Chengchang Liu, Cheng Chen, and Luo Luo. Symmetric rank- $k$  methods. arXiv preprint arXiv:2303.16188, 2023.
[34] Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics, 11(2):431-441, 1963.
[35] José Mario Martínez. Practical quasi-Newton methods for solving nonlinear systems. Journal of computational and Applied Mathematics, 124(1-2):97-121, 2000.
[36] Konstantin Mishchenko. Regularized Newton method with global  $O(1 / k^2)$  convergence. arXiv preprint arXiv:2112.02089, 2021.
[37] Jorge J. Moré. A collection of nonlinear model problems. Technical report, Argonne National Lab., IL (USA), 1989.
[38] Yu. Nesterov. Modified Gauss-Newton scheme with worst case guarantees for global performance. Optimisation methods and software, 22(3):469-483, 2007.
[39] Mojtaba Nourian and Peter E. Caines.  $\epsilon$ -Nash mean field game theory for nonlinear stochastic dynamical systems with major and minor agents. SIAM Journal on Control and Optimization, 51(4):3302-3331, 2013.
[40] M.J.D. Powell. A hybrid method for nonlinear equations. Numerical Methods for Nonlinear Algebraic Equations, 1970.
[41] Anton Rodomanov and Yurii Nesterov. Greedy quasi-Newton methods with explicit superlinear convergence. SIAM Journal on Optimization, 31(1):785–811, 2021.
[42] Anton Rodomanov and Yurii Nesterov. New results on superlinear convergence of classical quasi-Newton methods. Journal of optimization theory and applications, 188(3):744-769, 2021.
[43] Anton Rodomanov and Yurii Nesterov. Rates of superlinear convergence for classical quasi-Newton methods. Mathematical Programming, pages 1-32, 2021.
[44] Damien Scieur, Edouard Oyallon, Alexandre d'Aspremont, and Francis Bach. Online regularized nonlinear acceleration. arXiv preprint arXiv:1805.09639, 2018.
[45] David F. Shanno. Conditioning of quasi-Newton methods for function minimization. Mathematics of computation, 24(111):647-656, 1970.
[46] Jack Sherman and Winifred J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1): 124-127, 1950.
[47] Ph L. Toint. On large scale nonlinear least squares calculations. SIAM Journal on Scientific and Statistical Computing, 8(3):416-435, 1987.
[48] Xiao Wang, Shiqian Ma, Donald Goldfarb, and Wei Liu. Stochastic quasi-Newton methods for nonconvex stochastic optimization. SIAM Journal on Optimization, 27(2):927–956, 2017.
[49] Xiaoyu Wang, Xiao Wang, and Ya-Xiang Yuan. Stochastic proximal quasi-Newton methods for non-convex composite optimization. Optimization Methods and Software, 34(5):922-948, 2019.
[50] Haishan Ye, Dachao Lin, and Zhihua Zhang. Greedy and random Broyden's methods with explicit superlinear convergence rates in nonlinear equations. arXiv preprint arXiv:2110.08572, 2021.
[51] Haishan Ye, Dachao Lin, Xiangyu Chang, and Zhihua Zhang. Towards explicit superlinear convergence rate for SR1. Mathematical Programming, pages 1-31, 2022.
[52] Rui Yuan, Alessandro Lazaric, and Robert M. Gower. Sketched Newton-Raphson. SIAM Journal on Optimization, 32(3):1555-1583, 2022.
[53] Ya-Xiang Yuan. Trust region algorithms for nonlinear equations. Information, 1:7-20, 1998.
[54] Ya-Xiang Yuan. Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numerical algebra, control & optimization, 1(1):15, 2011.