SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jérôme Bolte
Toulouse School of Economics, Université Toulouse Capitole, Toulouse, France.
Edouard Pauwels
Toulouse School of Economics (IUF), Toulouse, France.
Samuel Vaiter
CNRS & Universite Cote d'Azur, Laboratoire J. A. Dieudonne. Nice, France.
In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as efficient as implicit differentiation for fast algorithms (e.g., superlinear optimization methods). We provide a complete theoretical approximation analysis with specific examples (Newton's method, gradient descent) along with its consequences in bilevel optimization. Several numerical examples illustrate the well-foundedness of the one-step estimator.
Differentiating the solution of a machine learning problem is an important task, e.g., in hyperparameters optimization [9], in neural architecture search [31] and when using convex layers [3]. There are two main ways to achieve this goal: automatic differentiation (AD) and implicit differentiation (ID). Automatic differentiation implements the idea of evaluating derivatives through the compositional rules of differential calculus in a user-transparent way. It is a mature concept [27] implemented in several machine learning frameworks [36, 16, 1]. However, the time and memory complexity incurred may become prohibitive as soon as the computational graph becomes bigger, a typical example being unrolling iterative optimization algorithms such as gradient descent [5]. The alternative, implicit differentiation, is not always accessible: it does not solely rely on the compositional rules of differential calculus (Jacobian multiplication) and usually requires solving a linear system. The user needs to implement custom rules in an automatic differentiation framework (as done, for example, in [4]) or use dedicated libraries such as [11, 3, 10] implementing these rules for given models. Provided that the implementation is carefully done, this is most of the time the gold standard for the task of differentiating problem solutions.
Contributions. We study a one-step Jacobian approximator based on a simple principle: differentiate only the last iteration of an iterative process and drop previous derivatives. The idea of dropping derivatives or single-step differentiation was explored, for example, in [24, 23, 39, 22, 40, 29] and our main contribution is a general account and approximation analysis for Jacobians of iterative algorithms. One-step estimation constitutes a rough approximation at first sight and our motivation to study is its ease-of-use within automatic differentiation frameworks: no custom Jacobian-vector products or vector-Jacobian products needed as long as a stop_grad primitive is available (see Table 1).
Algorithm 1: Automatic
Input:  $\theta \mapsto x_0(\theta)\in \mathcal{X},k > 0.$
Eval: with_gradient for  $i = 1,\dots ,k$  do  $x_{i}(\theta) = F(x_{i - 1}(\theta),\theta)$
Return:  $x_{k}(\theta)$
Differentiation: native autodiff on Eval.
Algorithm 2: Implicit
Input:  $x_0\in \mathcal{X}$ $k > 0$
Eval: stop_gradent for  $i = 1,\dots ,k$  do  $x_{i} = F(x_{i - 1},\theta)$
Return:  $x_{k}$
Differentiation: Custom implicit VJP / JVP.
Algorithm 3: One-step
Input:  $x_0\in \mathcal{X},k > 0$
Eval:
stop_gradient
$$
\text {f o r} i = 1, \dots , k - 1 \text {d o}
$$
$$
x _ {i} = F \left(x _ {i - 1}, \theta\right)
$$
with_gradient
$$
x _ {k} (\theta) = F \left(x _ {k - 1}, \theta\right)
$$
Differentiation:
native autodiff on Eval
Table 1: Qualitative comparison of differentiation strategies. Native autodiff refers to widespread primitives in differentiable programming (e.g. grad in JAX). Custom JVP/VJP refers to specialized libraries such as jaxopt [12] or qpth [4] implicit differentiation in specific contexts.
Algorithm	Implementation	Efficient
Automatic	Native autodiff	no
Implicit	Custom	yes
One step	Native autodiff	yes
We conduct an approximation analysis of one-step Jacobian (Corollary 1). The distance to the true Jacobian are produced by the distance to the solution (i.e., the quality of completion of the optimization phase) and the lack of contractivity of the iteration mapping. This imprecision has to be balanced with the ease of implementation of the one-step Jacobian. This suggests that one-step differentiation is efficient for small contraction factors, which corresponds to fast algorithms. We indeed show that one-step Jacobian is asymptotically correct for super-linearly convergent algorithms (Corollary 2) and provide similar approximation rate as implicit differentiation for quadratically convergent algorithms (Corollary 3). We exemplify these results with hypergradients in bilevel optimization, conduct a detailed complexity analysis and highlight in Corollary 4 the estimation of approximate critical points. Finally, numerical illustrations are provided to show the practicality of the method on logistic regression using Newton's algorithm, interior point solver for quadratic programming and weighted ridge regression using gradient descent.
Related works. Automatic differentiation [27] was first proposed in the forward mode in [44] and its reverse mode in [30]. The study of the behaviour of differentiating iterative procedure by automatic differentiation was first analyzed in [25] and [8] in the optimization community. It was studied in the machine learning community for smooth methods [35, 32, 37], and nonsmooth methods [14]. Implicit differentiation is a recent highlight in machine learning. It was shown to be a good way to estimate the Jacobian of problem solutions, for deep equilibrium network [5], optimal transport [33], and also for nonsmooth problems [13], such as sparse models [10]. Inexact implicit differentiation was explored in [19]. Truncated estimation of the "Neumann series" for the implicit differentiation is routinely used [32, 34] and truncated backpropagation was investigated in [42] for the gradient descent iterations. In the very specific case of min-min problems, [2] studied the speed of convergence of automatic, implicit, and analytical differentiation.
The closest work to ours is [22] - under the name *Jacobian-free backpropagation* - but differs significantly in the following ways. Their focus is on single implicit layer networks, and guarantees are qualitative (descent direction of [22, Theorem 3.1]). In contrast, we provide quantitative results on abstract fixed points and applicable to any architecture. The idea of "dropping derivatives" was proposed in [21] for meta-learning, one-step differentiation was also investigated to train Transformer architectures [23], and to solve bilevel problems with quasi-Newton's methods [39].
2 One-step differentiation
2.1 Automatic, implicit and one-step differentiation
Throughout the text  $F\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}^n$  denotes a recursive algorithmic map from  $\mathbb{R}^n$  to  $\mathbb{R}^n$  with  $m$  parameters. For any  $\theta \in \mathbb{R}^m$ , we write  $F_{\theta}:x\mapsto F(x,\theta)$  and let  $F_{\theta}^{k}$  denote  $k$  recursive composition of  $F_{\theta}$ , for  $k\in \mathbb{N}$ . The map  $F_{\theta}$  defines a recursive algorithm as follows
$$
x _ {0} (\theta) \in \mathbb {R} ^ {n} \quad \text {a n d} \quad x _ {k + 1} (\theta) = F (x _ {k} (\theta), \theta), \tag {1}
$$
We denote by  $J_{x}F_{\theta}$  the Jacobian matrix with respect to the variable  $x$ . The following assumption is sufficient to ensure a non degenerate asymptotic behavior.
Assumption 1 (Contraction) Let  $F\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}^n$  be  $C^1$ ,  $0\leq \rho < 1$ , and  $\mathcal{X}\subset \mathbb{R}^n$  be nonempty convex closed, such that for any  $\theta \in \mathbb{R}^m$ ,  $F_{\theta}(\mathcal{X})\subset \mathcal{X}$  and  $\| J_xF_\theta \|_{\mathrm{op}}\leq \rho$ .
Remark 1 The main algorithms considered in this paper fall in the scope of smooth optimization. The algorithmic map  $F_{\theta}$  is associated to a smooth parametric optimization problem given by  $f\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}$  such that  $f_{\theta}\colon x\mapsto f(x,\theta)$  is strongly convex, uniformly in  $\theta$ . Two examples of algorithmic maps are given by gradient descent,  $F_{\theta}(x) = x - \alpha \nabla f_{\theta}(x)$ , or Newton's  $F_{\theta}(x) = x - \alpha \nabla^{2}f_{\theta}(x)^{-1}\nabla f_{\theta}(x)$  for positive step  $\alpha >0$ . For small step sizes, gradient descent provides a contraction and Newton's method provides a local contraction, both fall in the scope of Assumption 1. Other examples include inertial algorithms such as the Heavy Ball method [38], which has to be considered in phase space and for which a single iteration is not contracting, but a large number of iteration is (see e.g. [38, 35]).
The following lemma gathers known properties regarding the fixed point of  $F_{\theta}$ , denoted by  $\bar{x}(\theta)$  and for which we will be interested in estimating derivatives.
Lemma 1 Under Assumption 1, for each  $\theta$  in  $\mathbb{R}^m$  there is a unique fixed point of  $F_{\theta}$  in  $\mathcal{X}$  denoted by  $\bar{x} (\theta)$ , which is a  $C^1$  function of  $\theta$ . Furthermore, for all  $k\in \mathbb{N}$ , we have  $\| x_{k}(\theta) - \bar{x} (\theta)\| \leq \rho^{k}\| x_{0}(\theta) - \bar{x} (\theta)\| \leq \rho^{k}\frac{\|x_{0} - F_{\theta}(x_{0})\|}{1 - \rho}$ .
This is well known, we briefly sketch the proof. The mapping  $F_{\theta}$  is a  $\rho$  contraction on  $\mathcal{X}$  - use the convexity of  $\mathcal{X}$  and the intermediate value theorem. Banach fixed point theorem ensures existence and uniqueness, differentiability is due to the implicit function theorem. The convergence rate is classical. We are interested in the numerical evaluation of the Jacobian  $J_{\theta}\bar{x} (\theta)$ , thus well-defined under Assumption 1.
The automatic differentiation estimator  $J^{\mathrm{AD}}x_{k}(\theta) = J_{\theta}x_{k}(\theta)$  propagates the derivatives (either in a forward or reverse mode) through iterations based on the piggyback recursion [26], for  $i = 1,\dots ,k - 1$ ,
$$
J _ {\theta} x _ {i + 1} (\theta) = J _ {x} F (x _ {i} (\theta), \theta) J _ {\theta} x _ {i} (\theta) + J _ {\theta} F (x _ {i} (\theta), \theta). \tag {2}
$$
Under assumption 1 we have  $J^{\mathrm{AD}}x_{k}(\theta) \to J_{\theta}\bar{x} (\theta)$  and the convergence is asymptotically linear [25, 8, 35, 41, 14].  $J^{\mathrm{AD}}x_{k}(\theta)$  is available in differentiable programming framework implementing common primitives such as backpropagation.
The implicit differentiation estimator  $J^{\mathrm{ID}}x_{k}(\theta)$  is given by application of the implicit function theorem using  $x_{k}$  as a surrogate for the fixed point  $\bar{x}$ ,
$$
J ^ {\mathrm {I D}} x _ {k} (\theta) = \left(I - J _ {x} F \left(x _ {k} (\theta), \theta\right)\right) ^ {- 1} J _ {\theta} F \left(x _ {k} (\theta), \theta\right). \tag {3}
$$
By continuity of the derivatives of  $F$ , we also have  $J^{\mathrm{ID}}x_{k}(\theta)\to J_{\theta}\bar{x} (\theta)$  as  $k\rightarrow \infty$ , (see e.g. [27, Lemma 15.1] or [12, Theorem 1]). Implementing  $J^{\mathrm{ID}}x_{k}(\theta)$  requires either manual implementation or dedicated techniques or libraries [4, 3, 45, 28, 20, 12] as the matrix inversion operation is not directly expressed using common differentiable programming primitives. A related estimator is the Inexact Automatic Differentiation (IAD) estimator which implements (2) but with Jacobians evaluated at the last iterates  $x_{k}$ , which can be seen as an approximation of  $J^{\mathrm{ID}}$  [35, 19].
The one-step estimator  $J^{\mathrm{OS}}x_{k}(\theta)$  is the Jacobian of the fixed point map for the last iteration
$$
J ^ {\mathrm {O S}} x _ {k} (\theta) = J _ {\theta} F (x _ {k - 1} (\theta), \theta). \tag {4}
$$
def F(x, theta):
    # here a gradient step
    return x - alpha * grad_f(x, theta)
def iterative Procedure(theta, x0):
    x = x0
    for _ in range(100):
        x = F(x, theta)
    return x
# Automatic differentiation
J_ad = jacfd(iterative Procedure)(theta, x0)
#Implicitdifferentiation   
J_id  $=$  #Customimplementation(e.g.jaxopt)   
#One-stepdifferentiation   
defiterative_procedure_with_stop_grad(theta,x0):  $\mathbf{x} = \mathbf{x}0$  for_inrange(99):  $\mathrm{x} = \mathrm{F}(\mathrm{stop\_gradient(x)},$  stop_gradient(theta))  $\mathrm{x} = \mathrm{F}(\mathrm{x},\mathrm{theta})$  returnx   
J_os  $=$  jacfd惘(differential_procedure)(theta,x0)
Figure 1: Implementation of Algorithms 1, 2 and 3 in jax.  $F$  is a gradient step of some function  $f$ . The custom implementation of implicit differentiation is not explicitly stated. The function stop_gradient is present in jax.lax and jacfwd computes the full Jacobian using forward-mode AD.
Contrary to automatic differentiation or implicit differentiation estimates, we do not have  $J^{\mathrm{OS}}x_{k}(\theta) \rightarrow J_{\theta}\bar{x} (\theta)$  in general as  $k \to \infty$ , but we will see that the error is essentially proportional to  $\rho$ , and thus negligible for fast algorithms for which the estimate is accurate.
From a practical viewpoint, the three estimators  $J^{\mathrm{AD}}$ ,  $J^{\mathrm{ID}}$  and  $J^{\mathrm{OS}}$  are implemented in a differentiable programming framework, such as jax, thanks to a primitive stop_grad, as illustrated by Algorithms 1, 2 and 3. The computational effect of the stop_grad primitive is to replace the actual Jacobian  $J_{x}F(x_{i}(\theta),\theta)$  by zero for chosen iterations  $i\in \{1,\dots ,k\}$ . Using it for all iterations except the last one, allows one to implement  $J^{\mathrm{OS}}$  in (4) using Algorithm 3. This illustrates the main interest of the one-step estimator: it can be implemented using any differentiable programming framework which provides a stop_grad primitive and does not require custom implementation of implicit differentiation. Figure 1 illustrates an implementation in jax for gradient descent.
2.2 Approximation analysis of one step differentiation for linearly convergent algorithms
The following lemma is elementary. It describes the main mathematical mechanism at stake behind our analysis of one-step differentiation.
Lemma 2 Let  $A\in \mathbb{R}^{n\times n}$  with  $\| A\|_{\mathrm{op}}\leq \rho < 1$  and  $B,\tilde{B}\in \mathbb{R}^{n\times m}$ , then
$$
(I - A) ^ {- 1} B - \tilde {B} = A (I - A) ^ {- 1} B + B - \tilde {B}.
$$
Moreover, we have the following estimate,
$$
\left\| (I - A) ^ {- 1} B - \tilde {B} \right\| _ {\mathrm {o p}} \leq \frac {\rho}{1 - \rho} \| B \| _ {\mathrm {o p}} + \| B - \tilde {B} \| _ {\mathrm {o p}}.
$$
Proof: First for any  $v \in \mathbb{R}^n$ , we have  $\| (I - A)v \| \geq \| Iv \| - \| Av \| \geq (1 - \rho) \| v \|$ , which shows that  $I - A$  is invertible (the kernel of  $I - A$  is trivial). We also deduce that  $\| (I - A)^{-1} \|_{\mathrm{op}} \leq 1 / (1 - \rho)$ . Second, we have  $(I - A)^{-1} - I = A(I - A)^{-1}$ , since  $((I - A)^{-1} - I)(I - A) = A$ , and therefore
$$
A (I - A) ^ {- 1} B + B - \tilde {B} = ((I - A) ^ {- 1} - I) B + B - \tilde {B} = (I - A) ^ {- 1} B - \tilde {B}.
$$
The norm bound follows using the submultiplicativity of operator norm, the triangular inequality and the fact that  $\| A\|_{\mathrm{op}}\leq \rho$  and  $\| (I - A)^{-1}\|_{\mathrm{op}}\leq 1 / (1 - \rho)$ .
Corollary 1 Let  $F$  and  $\mathcal{X}$  be as in Assumption 1 such that  $\theta \mapsto F(x, \theta)$  is  $L_{F}$  Lipschitz and  $x \mapsto J_{\theta} F(x, \theta)$  is  $L_{J}$  Lipschitz (in operator norm) for all  $x \in \mathbb{R}^n$ . Then, for all  $\theta \in \mathbb{R}^m$ ,
$$
\left\| J ^ {\mathrm {O S}} x _ {k} (\theta) - J _ {\theta} \bar {x} (\theta) \right\| _ {\mathrm {o p}} \leq \frac {\rho L _ {F}}{1 - \rho} + L _ {J} \| x _ {k - 1} - \bar {x} (\theta) \|. \tag {5}
$$
Proof : The result follows from Lemma 2 with  $A = J_{x}F(\bar{x} (\theta),\theta)$ $B = J_{\theta}F(\bar{x} (\theta),\theta)$  and  $\tilde{B} = J_{\theta}F(x_{k - 1},\theta)$  using the fact that  $\| B\|_{\mathrm{op}}\leq L_F$  and  $\| B - \tilde{B}\|_{\mathrm{op}}\leq L_J\| \bar{x} (\theta) - x_{k - 1}\|$
Remark 2 (Comparison with implicit differentiation) In [12] a similar bound is described for  $J^{\mathrm{ID}}$ , roughly under the assumption that  $x \to F(x, \theta)$  also has  $L_J$  Lipschitz Jacobian, one has
$$
\left\| J ^ {\mathrm {I D}} x _ {k} (\theta) - J _ {\theta} \bar {x} (\theta) \right\| _ {\mathrm {o p}} \leq \frac {L _ {J} L _ {F}}{(1 - \rho) ^ {2}} \| x _ {k} - \bar {x} (\theta) \| + \frac {L _ {J}}{1 - \rho} \| x _ {k} - \bar {x} (\theta) \|. \tag {6}
$$
For small  $\rho$  and large  $k$ , the main difference between the two bounds (5) and (6) lies in their first term which is of the same order whenever  $\rho$  and  $L_{J}\| \bar{x} -x_{k - 1}\|$  are of the same order.
Corollary 1 provides a bound on  $\| J^{\mathrm{OS}}x_k(\theta) - J_\theta \bar{x} (\theta)\|_{\mathrm{op}}$  which is asymptotically proportional to  $\rho$ . This means that for fast linearly convergent algorithms, meaning  $\rho \ll 1$ , one-step differentiation provides a good approximation of the actual derivative. Besides, given  $F$  which satisfies Assumption 1, with a given  $\rho < 1$ , not specially small, one can set  $\tilde{F}_{\theta} = F_{\theta}^{K}$  for some  $K\in \mathbb{N}$ . In this case,  $\tilde{F}$  satisfies assumption 1 with  $\tilde{\rho} = \rho^K$  and the one-step estimator in (4) applied to  $\tilde{F}$  becomes a  $K$ -steps estimator on  $F$  itself, we only differentiate through the  $K$  last steps of the algorithm which amounts to truncated backpropagation [42].
Example 1 (Gradient descent) Let  $f\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}$  be such that  $f(\cdot ,\theta)$  is  $\mu$ -strongly convex  $(\mu >0)$  with  $L$  Lipschitz gradient for all  $\theta \in \mathbb{R}^m$ , then the gradient mapping  $F\colon (x,\theta)\mapsto x - \alpha \nabla_{x}f(x,\theta)$  satisfies Assumption 1 with  $\rho = \max \{1 - \alpha \mu ,\alpha L - 1\}$ , smaller than 1 as long as  $0 < \alpha < 2 / L$ . The optimal  $\alpha = 2 / (L + \mu)$  leads to a contraction factor  $\rho = 1 - 2\mu /(L + \mu)$ . Assuming that  $\nabla_{x\theta}^{2}f$  is also  $L$  Lipschitz, Corollary 1 holds with  $L_{F} = L_{J} = 2L / (\mu +L)\leq 2$ . For step size  $1 / L$ , Corollary 1 holds with  $L_{F} = L_{J}\leq 1$  and  $\rho = 1 - \mu /L$ . In both cases, the contraction factor  $\rho$  is close to 0 when  $L / \mu \simeq 1$ , for well conditioned problems. As outlined above, we may consider  $\tilde{F}_{\theta} = F_{\theta}^{K}$  in which case Corollary 1 applies with a smaller value of  $\rho$ , recovering the result of [42, Proposition 3.1]
2.3 Superlinear and quadratic algorithms
The one-step Jacobian estimator in (4) as implemented in Algorithm 3 is obviously not an exact estimator in the sense that one does not necessarily have  $J^{\mathrm{OS}}x_k(\theta) \to J_\theta \bar{x}(\theta)$  as  $k \to \infty$ . However, it is easy to see that this estimator is exact in the case of exact single-step algorithms, meaning  $F$  satisfies  $F(x,\theta) = \bar{x}(\theta)$  for all  $x,\theta$ . Indeed, in this case, one has  $J_xF(x,\theta) = 0$  and  $J_\theta F(x,\theta) = J_\theta \bar{x}(\theta)$  for all  $x,\theta$ . Such a situation occurs, for example, when applying Newton's method to an unconstrained quadratic problem. This is a very degenerate situation as it does not really make sense to talk about an "iterative algorithm" in this case. It turns out that this property of being "asymptotically correct" remains valid for very fast algorithms, that is, algorithms that require few iterations to converge, the archetypal example being Newton's method for which we obtain quantitative estimates.
2.3.1 Super-linear algorithms
The following is a typical property of fast converging algorithms.
Assumption 2 (Vanishing Jacobian) Assume that  $F \colon \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}^n$  is  $C^1$  and that the recursion  $x_{k+1} = F_\theta(x_k)$  converges globally, locally uniformly in  $\theta$  to the unique fixed point  $\bar{x}(\theta)$  of  $F_\theta$  such that  $J_xF(\bar{x}(\theta), \theta) = 0$ .
Note that under Assumption 2, it is always possible to find a small neighborhood of  $\bar{x}$  such that  $\| J_xF_\theta \|_{\mathrm{op}}$  remains small, that is, Assumption 1 holds locally and Lemma 1 applies. Furthermore, it is possible to show that the derivative estimate is asymptotically correct as follows.
Corollary 2 (Jacobian convergence) Let  $F\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}^n$  be as in Assumption 2. Then  $J^{\mathrm{OS}}x_{k}(\theta)\rightarrow J_{\theta}\bar{x} (\theta)$  as  $k\to \infty$  , and  $J^{\mathrm{OS}}\bar{x} (\theta) = J_{\theta}\bar{x} (\theta)$
Proof: Since  $J_{x}F(\bar{x} (\theta),\theta) = 0$ , implicit differentiation of the fixed point equation reduces to  $J_{\theta}\bar{x} (\theta) = J_{\theta}F(\bar{x} (\theta),\theta)$ , and the result follows by continuity of the derivatives.
Example 2 (Superlinearly convergent algorithm) Assume that  $F$  is  $C^1$  and for each  $\rho > 0$ , there is  $R > 0$  such that  $\| F_{\theta}(x) - \bar{x}(\theta) \| \leq \rho \| x - \bar{x}(\theta) \|$  for all  $x, \theta$  such that  $\| x - \bar{x}(\theta) \| \leq R$ . Then Corollary 2 applies as for any  $v$
$$
J _ {x} F (\bar {x} (\theta), \theta) v = \lim _ {t \to 0} \frac {F (\bar {x} (\theta) + t v , \theta) - \bar {x} (\theta)}{t} = 0.
$$
2.3.2 Quadratically convergent algorithms
Under additional quantitative assumptions, it is possible to obtain more precise convergence estimates similar to those obtained for implicit differentiation, see Remark 2.
Corollary 3 Let  $F$  be as in Assumption 2 such that  $x \mapsto J_{(x,\theta)}F(x,\theta)$  (joint jacobian in  $(x,\theta)$ ) is  $L_J$  Lipschitz (in operator norm). Then, the recursion is asymptotically quadratically convergent and for each  $k \geq 1$ ,
$$
\left\| J ^ {\mathrm {O S}} x _ {k} (\theta) - J _ {\theta} \bar {x} (\theta) \right\| _ {\mathrm {o p}} \leq L _ {J} \| x _ {k - 1} (\theta) - \bar {x} (\theta) \|. \tag {7}
$$
Figure 2: Newton's method quadratic convergence.
Proof : Following the same argument as in the proof of Corollary 2, we have
$$
\left\| J ^ {\mathrm {O S}} x _ {k} (\theta) - J _ {\theta} \bar {x} (\theta) \right\| _ {\mathrm {o p}} = \left\| J _ {\theta} F (x _ {k} (\theta), \theta) - J _ {\theta} F (\bar {x} (\theta), \theta) \right\| _ {\mathrm {o p}} \leq L _ {J} \| x _ {k - 1} (\theta) - \bar {x} (\theta) \|. \tag {8}
$$
As for the quadratic convergence, we may assume that  $\bar{x} (\theta) = 0$  and drop the  $\theta$  variable to simplify notations. We have  $F(0) = 0$  and for all  $x$ ,
$$
F (x) = \int_ {0} ^ {1} J _ {x} F (t x) x d t \leq \| x \| \int_ {0} ^ {1} \| J _ {x} F (t x) \| _ {\mathrm {o p}} d t \leq \| x \| ^ {2} L _ {J} \int_ {0} ^ {1} t d t = \frac {L _ {J} \| x \| ^ {2}}{2}.
$$
Thus  $L_{J} / 2\| x_{k + 1}\| \leq [L_{J} / 2\| x_{k}\| ]^{2}$ , and asymptotic quadratic convergence follows.
Example 3 (Newton's algorithm) Assume that  $f\colon \mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}$  is  $C^3$  with Lipschitz derivatives, and for each  $\theta$ ,  $x\mapsto f(x,\theta)$  is  $\mu$ -strongly convex. Then Newton's algorithm with backtracking line search satisfies the hypotheses of Corollary 3, see [15, Sec. 9.5.3]. Indeed, it takes unit steps after a finite number of iterations, denoting by  $\bar{x} (\theta)$  the unique solution to  $\nabla_{x}f(x,\theta) = 0$ , for all  $x$  locally around  $\bar{x} (\theta)$
$$
F (x, \theta) = x - \nabla_ {x x} ^ {2} f (x, \theta) ^ {- 1} \nabla_ {x} f (x, \theta).
$$
We have,  $\nabla_{xx}^2 f(x,\theta)F(x,\theta) = \nabla_{xx}^2 f(x,\theta)x - \nabla_x f(x,\theta)$ , differentiating using tensor notations,