SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaewook J. Suh
Seoul National University jacksuhkr@snu.ac.kr
Jisun Park
Seoul National University  
colleenp0515@snu.ac.kr
Ernest K. Ryu
Seoul National University ernestryu@snu.ac.kr
Recently, the anchor acceleration, an acceleration mechanism distinct from Nesterov's, has been discovered for minimax optimization and fixed-point problems, but its mechanism is not understood well, much less so than Nesterov acceleration. In this work, we analyze continuous-time models of anchor acceleration. We provide tight, unified analyses for characterizing the convergence rate as a function of the anchor coefficient  $\beta(t)$ , thereby providing insight into the anchor acceleration mechanism and its accelerated  $\mathcal{O}(1/k^2)$ -convergence rate. Finally, we present an adaptive method inspired by the continuous-time analyses and establish its effectiveness through theoretical analyses and experiments.
Nesterov acceleration [51] is foundational to first-order optimization theory, but the mechanism and its convergence proof are not transparent. One approach to better understand the mechanism is the continuous-time analysis: derive an ODE model of the discrete-time algorithm and analyze the continuous-time dynamics [65, 66]. This approach provides insight into the accelerated dynamics and has led to a series of follow-up work [71, 62, 29].
Recently, a new acceleration mechanism, distinct from Nesterov's, has been discovered. This anchor acceleration for minimax optimization and fixed-point problems [35, 75, 54] has been an intense subject of study, but its mechanism is understood much less than Nesterov acceleration. The various analytic techniques developed to understand Nesterov acceleration, including continuous-time analyses, have only been applied in a very limited manner [59].
Contribution. In this work, we present continuous-time analyses of anchor acceleration. The continuous-time model is the differential inclusion
$$
\dot {X} \in - \mathbb {A} (X) - \beta (t) (X - X _ {0})
$$
with initial condition  $X(0) = X_0 \in \operatorname{dom} \mathbb{A}$ , maximal monotone operator  $\mathbb{A}$ , and scalar-valued function  $\beta(t)$ . The case  $\beta(t) = \frac{1}{t}$  corresponds to the prior anchor-accelerated methods APPM [35], EAG [75], and FEG [42].
We first establish that the differential inclusion is well-posed, despite the anchor coefficient  $\beta(t)$  blowing up at  $t = 0$ . We then provide tight, unified analyses for characterizing the convergence rate as a function of the anchor coefficient  $\beta(t)$ . This is the first formal and rigorous treatment of this anchored dynamics, and it provides insight into the anchor acceleration mechanism and its accelerated  $\mathcal{O}(1/k^2)$ -convergence rate. Finally, we present an adaptive method inspired by the continuous-time analyses and establish its effectiveness through theoretical analyses and experiments.
1.1 Preliminaries and notation
We provide the organization for prior works in Appendix A. Here, we review standard definitions and set up the notation.
Monotone and set-valued operators. We follow the standard definitions of Bauschke and Combettes [15], Ryu and Yin [58]. For the underlying space, consider  $\mathbb{R}^n$  with standard inner product  $\langle \cdot ,\cdot \rangle$  and norm  $\| \cdot \|$ . Define domain of  $\mathbb{A}$  as  $\operatorname {dom}\mathbb{A} = \{x\in \mathbb{R}^n\mid \mathbb{A}x\neq \emptyset \}$ . We say  $\mathbb{A}$  is an operator on  $\mathbb{R}^n$  and write  $\mathbb{A}\colon \mathbb{R}^n\Rightarrow \mathbb{R}^n$  if  $\mathbb{A}$  maps a point in  $\mathbb{R}^n$  to a subset of  $\mathbb{R}^n$ . We say  $\mathbb{A}\colon \mathbb{R}^n\Rightarrow \mathbb{R}^n$  is monotone if
$$
\langle \mathbb {A} x - \mathbb {A} y, x - y \rangle \geq 0, \quad \forall x, y \in \mathbb {R} ^ {n},
$$
where the notation means that  $\langle u - v, x - y \rangle \geq 0$  for all  $u \in \mathbb{A}x$  and  $v \in \mathbb{A}y$ . For  $\mu \in (0, \infty)$ , say  $\mathbb{A} \colon \mathbb{R}^n \Rightarrow \mathbb{R}^n$  is  $\mu$ -strongly monotone if
$$
\langle \mathbb {A} x - \mathbb {A} y, x - y \rangle \geq \mu \| x - y \| ^ {2}, \quad \forall x, y \in \mathbb {R} ^ {n}.
$$
Write  $\operatorname{Gra} \mathbb{A} = \{(x, u) \mid u \in \mathbb{A}x\}$  for the graph of  $\mathbb{A}$ . An operator  $\mathbb{A}$  is maximally monotone if there is no other monotone  $\mathbb{B}$  such that  $\operatorname{Gra} \mathbb{A} \subset \operatorname{Gra} \mathbb{B}$  properly, and is maximally  $\mu$ -strongly monotone if there is no other  $\mu$ -strongly monotone  $\mathbb{B}$  such that  $\operatorname{Gra} \mathbb{A} \subset \operatorname{Gra} \mathbb{B}$  properly.
For  $L\in (0,\infty)$ , single-valued operator  $\mathbb{T}\colon \mathbb{R}^n\to \mathbb{R}^n$  is  $L$ -Lipschitz if
$$
\| \mathbf {T} x - \mathbf {T} y \| \leq L \| x - y \|, \quad \forall x, y \in \mathbb {R} ^ {n}.
$$
Write  $\mathbb{J}_{\mathbf{A}} = (\mathbb{I} + \mathbb{A})^{-1}$  for the resolvent of  $\mathbb{A}$ , while  $\mathbb{I} \colon \mathbb{R}^n \to \mathbb{R}^n$  is the identity operator. When  $\mathbb{A}$  is maximally monotone, it is well known that  $\mathbb{J}_{\mathbf{A}}$  is single-valued with  $\operatorname{dom} \mathbb{J}_{\mathbf{A}} = \mathbb{R}^n$ .
We say  $x_{\star} \in \mathbb{R}^{n}$  is a zero of  $\mathbb{A}$  if  $0 \in \mathbb{A}x_{\star}$ . We say  $y_{\star}$  is a fixed-point of  $\mathbb{T}$  if  $\mathbb{T}y_{\star} = y_{\star}$ . Write  $\mathrm{Zer}\mathbb{A}$  for the set of all zeros of  $\mathbb{A}$  and Fix  $\mathbb{T}$  for the set of all fixed-points of  $\mathbb{T}$ .
Monotonicity with continuous curves. We say an operator is differentiable if it is single-valued, continuous, and differentiable as a function. If a differentiable operator  $\mathbb{A}$  is monotone and  $X\colon [0,\infty)\to \mathbb{R}^n$  is a differentiable curve, then taking limit  $h\rightarrow 0$  of
$$
\frac {1}{h ^ {2}} \langle \mathbb {A} (X (t + h)) - \mathbb {A} (X (t)), X (t + h) - X (t) \rangle \geq 0
$$
leads to
$$
\left\langle \frac {d}{d t} \mathbb {A} (X (t)), \dot {X} (t) \right\rangle \geq 0. \tag {1}
$$
Similarly if  $\mathbb{A}$  is furthermore  $\mu$ -strongly monotone, then
$$
\left\langle \frac {d}{d t} \mathbb {A} (X (t)), \dot {X} (t) \right\rangle \geq \mu \| \dot {X} (t) \| ^ {2}. \tag {2}
$$
2 Derivation of differential inclusion model of anchor acceleration
2.1 Anchor ODE
Suppose  $\mathbb{A}\colon \mathbb{R}^n\Rightarrow \mathbb{R}^n$  is a maximal monotone operator and  $\beta :(0,\infty)\to [0,\infty)$  is a twice differentiable function. Consider differential inclusion
$$
\dot {X} (t) \in - \mathbb {A} (X (t)) - \beta (t) \left(X (t) - X _ {0}\right) \tag {3}
$$
with initial condition  $X(0) = X_0 \in \operatorname{dom}(\mathbb{A})$ . We refer to this as the anchor ODE. We say  $X \colon [0, \infty) \to \mathbb{R}^n$  is a solution, if it is absolutely continuous and satisfies (3) for  $t \in (0, \infty)$  almost everywhere.
Denote  $S$  as the subset of  $[0, \infty)$  on which  $X$  satisfies the differential inclusion. Define
$$
\tilde {\mathbf {A}} (X (t)) = - \dot {X} (t) - \beta (t) (X (t) - X _ {0})
$$
for  $t \in S$ . Since  $\tilde{\mathbf{A}}(X(t)) \in \mathbb{A}(X(t))$  for  $t \in S$ , we say  $\tilde{\mathbf{A}}$  is a selection of  $\mathbb{A}$  for  $t \in S$ . If  $\| \tilde{\mathbf{A}}(X(t)) \|$  is bounded on all bounded subsets of  $S$ , then we can extend  $\tilde{\mathbf{A}}$  to  $[0, \infty)$  while retaining certain favorable properties. We discuss the technical details of this extension in Appendix E.1. The statements of Section 3 are stated with this extension.
Strictly speaking, this is a differential inclusion, not a differential equation, but we nevertheless refer to it as an ODE.
2.2 Derivation from discrete methods
We now show that the following instance of the anchor ODE
$$
\dot {X} (t) = - \mathbb {A} (X (t)) - \frac {1}{t} (X (t) - X _ {0}), \tag {4}
$$
where  $X(0) = X_0$  is the initial condition and  $\mathbb{A} \colon \mathbb{R}^n \to \mathbb{R}^n$  is a continuous operator, is a continuous-time model of APPM [35], EAG [75], and FEG [42], which are accelerated methods for monotone inclusion and minimax problems.
Consider APPM with operator  $h\mathbb{A}$
$$
x ^ {k} = \mathbb {J} _ {h \mathbb {A}} y ^ {k - 1}
$$
$$
y ^ {k} = \frac {k}{k + 1} \left(2 x ^ {k} - y ^ {k - 1}\right) + \frac {1}{k + 1} y ^ {0} \tag {5}
$$
with initial condition  $y^0 = x^0$ . Assume  $h > 0$  and  $\mathbb{A} \colon \mathbb{R}^n \to \mathbb{R}^n$  is a continuous monotone operator. Using  $y^{k - 1} = x^k + h\mathbb{A}x^k$  obtained from the first line, substituting  $y^k$  and  $y^{k - 1}$  in the second line we get,
$$
x ^ {k + 1} + h \mathbb {A} x ^ {k + 1} = \frac {k}{k + 1} \left(x ^ {k} - h \mathbb {A} x ^ {k}\right) + \frac {1}{k + 1} x ^ {0}.
$$
Then reorganizing and dividing both sides by  $h$ , we have
$$
\frac {x ^ {k + 1} - x ^ {k}}{h} = - \mathbb {A} x ^ {k + 1} - \frac {k}{k + 1} \mathbb {A} x ^ {k} - \frac {1}{h (k + 1)} (x ^ {k} - x ^ {0}).
$$
Identifying  $x^0 = X_0$ ,  $2hk = t$ , and  $x^{k} = X(t)$ , we have  $\frac{k}{k + 1} = 1 - \frac{h}{hk + h} = 1 + \mathcal{O}(h)$  and so
$$
2 \dot {X} (t) + \mathcal {O} (h) = - \mathbb {A} (X (t + 2 h)) - (1 + \mathcal {O} (h)) \mathbb {A} (X (t)) - \frac {2}{t + \mathcal {O} (h)} (X (t) - X _ {0}).
$$
Taking limit  $h \to 0^+$  and dividing both sides by 2, we get the anchor ODE (4). The correspondence with EAG and FEG are provided in Appendix D.4.
The following theorem establishes a rigorous correspondence between APPM and the anchor ODE for general maximal monotone operators.
Theorem 2.1. Let  $\mathbb{A}$  be a (possibly set-valued) maximal monotone operator and assume  $\mathrm{Zer}\mathbb{A} \neq \emptyset$ . Let  $x^k$  be the sequence generated by APPM (5) and  $X$  be the solution of the differential inclusion (3) with  $\beta(t) = \frac{1}{t}$ . For all fixed  $T > 0$ ,
$$
\lim_{h\to 0 + }\max_{0\leq k\leq \frac{T}{2h}}\left\| x^{k} - X(2kh)\right\| = 0.
$$
We provide the proof in Appendix D.2.
2.3 Existence of the solution for  $\beta (t) = \frac{\gamma}{tp}$
To get further insight into the anchor acceleration, we generalize anchor coefficient to  $\beta(t) = \frac{\gamma}{t^p}$  for  $p, \gamma > 0$ . We first establish the uniqueness and existence of the solution.
Theorem 2.2. Consider (3) with  $\beta(t) = \frac{\gamma}{tp}$ , i.e.
$$
\dot {X} (t) \in - \mathbb {A} (X (t)) - \frac {\gamma}{t ^ {p}} (X (t) - X _ {0}). \tag {6}
$$
for  $p, \gamma > 0$ . Then solution of (6) uniquely exists.
We provide the proof in Appendix B.
2.4 Additional properties of anchor ODE
We state a regularity lemma of the differential inclusion (3), which we believe may be of independent interest. In particular, we use this result several times throughout our various proofs.
Lemma 2.3. Let  $X(\cdot)$  and  $Y(\cdot)$  are solutions of the differential inclusion (3) respectively with initial values and anchors  $X_0$  and  $Y_0$ . Then for all  $t \in [0, \infty)$ ,
$$
\left\| X (t) - Y (t) \right\| \leq \left\| X _ {0} - Y _ {0} \right\|.
$$
We provide the proof in Appendix B.1.
Boundedness of trajectories is an immediate corollary of Lemma 2.3. Specifically, suppose  $X(\cdot)$  is the solution of differential inclusion (3) with initial value  $X_0$ . Then for all  $X_{\star} \in \mathrm{ZerA}$  and  $t \in [0,\infty)$ ,
$$
\left\| X (t) - X _ {\star} \right\| \leq \left\| X _ {0} - X _ {\star} \right\|.
$$
This follows from setting  $Y_{0} = X_{\star}$  in Lemma 2.3.
3 Convergence analysis
We now analyze the convergence rate of  $\left\| \tilde{\mathbf{A}} (X(t))\right\|^2$  for the anchor ODE (3) with  $\beta (t) = \frac{\gamma}{t^p}$  and  $\gamma ,p > 0$ . The results are organized in Table 1.
Case	p=1,γ≥1	p=1,γ<1	p<1	p>1
||A(X(t))||2	O(1/t2)	O(1/t2γ)	O(1/t2p)	O(1)
Table 1: Convergence rates of Theorem 3.1.
Let  $\beta$  be the anchor coefficient function of (3). Define  $C\colon [0,\infty)\to \mathbb{R}$  as  $C(t) = e^{\int_v^t\beta (s)ds}$  for some  $v\in [0,\infty ]$ . Note that  $\dot{C} = C\beta$  and  $C$  is unique up to scalar multiple. We call  $\mathcal{O}(\beta (t))$  the vanishing speed and  $\mathcal{O}\left(\frac{1}{C(t)}\right)$  the contracting speed, and we describe their trade-off in the following.
Loosely speaking, the contracting speed describes how fast the anchor term alone contracts the dynamical system. Consider  $\dot{X}(t) = -\beta(t)(X(t) - a)$  for  $a \in \mathbb{R}^n$ , a system only with the anchor. Then,  $X(t) = \frac{C(0)}{C(t)} (X(0) - a) + a$  is the solution, so the flow contracts towards the anchor  $a$  with rate  $\frac{1}{C(t)}$ . Intuitively speaking, this contracting behavior leads to stability and convergence. On the other hand, the anchor must eventually vanish, since our goal is to converge to an element in  $\operatorname{Zer}\mathbb{A}$ , not the anchor. Thus the vanishing speed must be fast enough to not slow down the convergence of the flow to  $\operatorname{Zer}\mathbb{A}$ .
This observation is captured in Figure 1. Consider a monotone linear operator  $\mathbb{A} = \left( \begin{array}{cc}0 & 1\\ -1 & 0 \end{array} \right)$  on  $\mathbb{R}^2$  and  $\beta (t) = \frac{\gamma}{t^p}$  with  $\gamma = 1$  and  $p > 0$ . Note if there is no anchor, the ODE reduces to  $\dot{X} = -\mathbb{A}(X)$  which do not converge [31, Chapter 8.2]. Figure 1 shows that with  $p > 1$ , the anchor vanished too early before the flow is contracted enough to result in converging flow. With  $p < 1$ , the flow does converge but the anchor vanished too late, slowing down the convergence. With  $p = 1$ , the convergence is fastest.
Figure 1: Flows of the solution of (6) with  $\mathbb{A} = \left( \begin{array}{cc}0 & 1\\ -1 & 0 \end{array} \right),\gamma = 1,X_0 = (1,0)$  and various  $p$  . Flow is from  $t = 0$  to  $t = 100$  . The marker is plotted every O.8 units of time until  $t = 9.6$  . Note the last marker of the flow for  $p = \frac{1}{2}$  is farther from the optimal point  $\star$  than that of the flow for  $p = 1$
The following theorem formalizes this insight and produces the results of Table 1.
Theorem 3.1. Suppose  $\mathbb{A}$  is a maximal monotone operator with  $\operatorname{Zer} \mathbb{A} \neq \emptyset$ . Consider (3) with  $\beta(t) = \frac{\gamma}{tp}$ . Let  $\tilde{\mathbb{A}}(X(t))$  be the selection of  $\mathbb{A}(X(t))$  as in Section 2.1. Then,
$$
\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} = \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right) + \mathcal {O} (\beta (t) ^ {2}) + \mathcal {O} (\dot {\beta} (t)).
$$
$$
C (t) = \left\{ \begin{array}{l l} t ^ {\gamma} & p = 1 \\ e ^ {\frac {\gamma}{1 - p}} t ^ {1 - p} & p \neq 1. \end{array} \right.
$$
We expect the convergence rate of Theorem 3.1 to be optimized when the terms are balanced. When  $\beta(t) = \frac{1}{t}$ ,
$$
\frac {1}{C (t) ^ {2}} = \frac {1}{(e ^ {\int_ {1} ^ {t} \frac {1}{s} d s}) ^ {2}} = \frac {1}{t ^ {2}} = \beta (t) ^ {2} = - \dot {\beta} (t)
$$
and all three terms are balanced. Indeed, the choice  $\beta(t) = \frac{1}{t}$  corresponds to the optimal discrete-time choice  $\frac{1}{k + 2}$  of APPM or other accelerated methods.
3.1 Proof outline of Theorem 3.1
The proof of Theorem 3.1 follows from Lemma 3.4, which we will introduce later in this section. To derive Lemma 3.4, we introduce a conservation law.
Proposition 3.2. Suppose  $\tilde{\mathbb{A}}$  is Lipschitz continuous and monotone. For  $t_0 > 0$ , define  $E:(0,\infty)\to \mathbb{R}$  as
$$
\begin{array}{l} E = \frac {C (t) ^ {2}}{2} \left(\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + 2 \beta (t) \left\langle \tilde {\mathbf {A}} (X (t)), X (t) - X _ {0} \right\rangle + (\beta (t) ^ {2} + \dot {\beta} (t)) \| X (t) - X _ {0} \| ^ {2}\right) \\ - \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(\frac {C (s) ^ {2} \dot {\beta} (s)}{2}\right) \| X (s) - X _ {0} \| ^ {2} d s + \int_ {t _ {0}} ^ {t} C (s) ^ {2} \left\langle \frac {d}{d s} \tilde {\mathbb {A}} (X (s)), \dot {X} (s) \right\rangle d s. \\ \end{array}
$$
Then  $E$  is a constant function.
The proof of Proposition 3.2 uses dilated coordinate  $W(t) = C(t)(X(t) - X_0)$  to derive its conservation law in the style of Suh et al. [67]. We provide the details in Appendix E.2.
Recall from (1) that  $\left\langle \frac{d}{ds}\tilde{\mathbf{A}} (X(s)),\dot{X} (s)\right\rangle \geq 0$  , the integrand of the last term of  $E$  is nonnegative. This motivates us to define
$$
V (t) = E - \int_ {t _ {0}} ^ {t} C (s) ^ {2} \left\langle \frac {d}{d s} \tilde {\mathbf {A}} (X (s)), \dot {X} (s) \right\rangle d s
$$
as our Lyapunov function.
Corollary 3.3. Let  $\mathbb{A}$  be maximal monotone and  $\beta(t) = \frac{\gamma}{tp}$  with  $p > 0$ ,  $\gamma > 0$ . Let  $\tilde{\mathbb{A}}(X(t))$  be the selection of  $\mathbb{A}(X(t))$  as in Section 2.1. For  $t_0 \geq 0$ , define  $V:[0,\infty) \to \mathbb{R}$  as
$$
\begin{array}{l} V (t) = \frac {C (t) ^ {2}}{2} \left(\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + 2 \beta (t) \left\langle \tilde {\mathbf {A}} (X (t)), X (t) - X _ {0} \right\rangle + \left(\beta (t) ^ {2} + \dot {\beta} (t)\right) \left\| X (t) - X _ {0} \right\| ^ {2}\right) \\ - \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(\frac {C (s) ^ {2} \dot {\beta} (s)}{2}\right) \| X (s) - X _ {0} \| ^ {2} d s. \\ \end{array}
$$
for  $t > 0$  and  $V(0) = \lim_{t\to 0 + }V(t)$ . Then  $V(t)\leq V(0)$  holds for  $t\geq 0$ .
A technical detail is that all terms involving  $\frac{d}{ds} \tilde{\mathbb{A}}(X(s))$  have been excluded in the definition of  $V$  and this is what allows  $\mathbb{A}$  to not be Lipschitz continuous. We provide the details in Appendix E.3.
Lemma 3.4. Consider the setup of Corollary 3.3. Assume  $\mathrm{Zer}\mathbb{A} \neq \emptyset$ . Then for  $t > 0$  and  $X_{\star} \in \mathrm{Zer}\mathbb{A}$ ,
$$
\begin{array}{l} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} \leq 4 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2} + \frac {4 V (0)}{C (t) ^ {2}} - 2 (\beta (t) ^ {2} + \dot {\beta} (t)) \| X (t) - X _ {0} \| ^ {2} \\ + \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) \| X (s) - X _ {0} \| ^ {2} d s. \tag {7} \\ \end{array}
$$
Proof outline of Lemma 3.4. Define
$$
\Phi (t) = \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + 2 \beta (t) \left\langle \tilde {\mathbf {A}} (X (t)), X (t) - X _ {0} \right\rangle .
$$
Then, from monotonicity of  $\tilde{\mathbb{A}}$  and Young's inequality,
$$
\begin{array}{l} \Phi (t) \geq \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + 2 \beta (t) \left\langle \tilde {\mathbf {A}} (X (t)), X _ {\star} - X _ {0} \right\rangle \\ \geq \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} - 2 \left(\left\| \frac {1}{2} \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + \| \beta (t) (X _ {\star} - X _ {0}) \| ^ {2}\right) \\ = \frac {1}{2} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} - 2 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2}. \tag {8} \\ \end{array}
$$
By Corollary 3.3,  $\frac{2V(0)}{C(t)^2} -\frac{2V(t)}{C(t)^2} +\Phi (t)\geq \Phi (t)$  for  $t > 0$  . Applying (8) and organizing, we can get the desired result. The details are provided in Appendix E.4.
Proof outline of Theorem 3.1. It remains to show that last integral term of Lemma 3.4 is  $\mathcal{O}\left(\frac{1}{C(t)^2}\right) + \mathcal{O}\left(\beta (t)^2\right) + \mathcal{O}\left(\dot{\beta} (t)\right)$ . The details are provided in Appendix E.5.
Before we end this section, we observe how our analysis simplifies in the special case  $\beta(t) = \frac{1}{t}$ . In this case,
$$
V (t) = \frac {t ^ {2}}{2} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} + t \left\langle \tilde {\mathbf {A}} (X (t)), X (t) - X _ {0} \right\rangle ,
$$
and this corresponds to the Lyapunov function of [59, Section 4] for the case  $\gamma = 1$ . As  $V(0) = 0$ , the conclusion of Lemma 3.4 becomes
$$
\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} \leq \frac {4}{t ^ {2}} \left\| X _ {0} - X _ {\star} \right\| ^ {2} = \mathcal {O} \left(\frac {1}{t ^ {2}}\right),
$$
which to the best rate in Table 1.
3.2 Point convergence
APPM is an instance of the Halpern method [54, Lemma 3.1], which iterates converge to the element in  $\mathsf{ZerA}$  closest to  $X_0$  [34, 73]. The anchor ODE also exhibits this behavior.
Theorem 3.5. Let  $\mathbb{A}$  be a maximal monotone operator with  $\operatorname{Zer} \mathbb{A} \neq \emptyset$  and  $X$  be the solution of (3). If  $\lim_{t \to \infty} \left\| \tilde{\mathbf{A}}(X(t)) \right\| = 0$  and  $\lim_{t \to \infty} 1 / C(t) = 0$ , then, as  $t \to \infty$ ,
$$
X(t)\to \operatorname *{argmin}_{z\in \mathrm{Zer}\mathsf{A}}\| z - X_{0}\| .
$$
We provide the proof in Appendix E.6.
4 Tightness of analysis
In this section, we show that the convergence rates of Table 1 are actually tight by considering the dynamics under the explicit example  $\mathbb{A} = \left( \begin{array}{cc}0 & 1\\ -1 & 0 \end{array} \right)$ . Throughout this section, we denote  $\mathbb{A}$  as  $A$  when the operator is linear.
4.1 Explicit solution for linear  $A$
Lemma 4.1. Let  $A\colon \mathbb{R}^n\to \mathbb{R}^n$  be a linear operator and let  $\beta (t) = \frac{\gamma}{t}$ . The series
$$
X (t) = \sum_ {n = 0} ^ {\infty} \frac {(- t A) ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0},
$$
where  $\Gamma$  denotes the gamma function, is the solution for (3) with  $\mathbb{A} = A$ .
Note that when  $\gamma = 0$ , this is the series definition of the matrix exponential and  $X(t) = e^{-tA}$ . The solution also has an integral form, which extends to general  $\beta(t)$ .
Lemma 4.2. Suppose  $A\colon \mathbb{R}^n\to \mathbb{R}^n$  is a monotone linear operator. Then
$$
X (t) = \frac {e ^ {- t A}}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) \beta (s) d s + C (0) I\right) X _ {0} \tag {9}
$$
is the solution for (3) with  $\mathbb{A} = A$ .
See Appendix F.1.1 and Appendix F.1.2 for details.
4.2 The rates in Table 1 are tight
First, we consider  $p > 1$  for  $\beta (t) = \frac{\gamma}{t^p}$ .
Theorem 4.3. Suppose  $\lim_{t\to \infty}\frac{1}{C(t)}\neq 0$ , i.e., suppose  $\beta (t)\in L^{1}[t_{0},\infty)$  for some  $t_0 > 0$ . Then there exists an operator  $\mathbb{A}$  such that
$$
\lim _ {t \to \infty} \left\| \tilde {\mathbf {A}} (X (t)) \right\| \neq 0,
$$
where  $X$  is the solution of (3).
Note that  $\frac{\gamma}{t^p} \in L^1[t_0, \infty)$  when  $p > 1$ . The proof of Theorem 4.3 considers  $\mathbb{A} = 2\pi \xi \left( \begin{array}{cc}0 & 1 \\ -1 & 0 \end{array} \right)$  for  $\xi \in \mathbb{R}$  and uses the Fourier inversion formula. See Appendix F.2 for details.
Next, we consider  $\beta (t) = \frac{\gamma}{t^p}$  for cases other than  $p > 1$
Theorem 4.4. Let  $A = \left( \begin{array}{cc}0 & 1\\ -1 & 0 \end{array} \right),\beta (t) = \frac{\gamma}{t^p},0 < p\leq 1,$  and  $\gamma >0$ . Let  $X$  be the solution given by (9) and  $X_0\neq 0$ . Let
$$
r (t) = \left\{ \begin{array}{l l} t ^ {2} & f o r p = 1, \gamma \geq 1 \\ t ^ {2 \gamma} & f o r p = 1, \gamma <   1 \\ t ^ {2 p} & f o r 0 <   p <   1. \end{array} \right.,
$$
$$
\lim _ {t \to \infty} r (t) \left\| A (X (t)) \right\| ^ {2} \neq 0.
$$
We provide the proof in Appendix F.3.
5 Discretized algorithms
In this section, we provide discrete-time convergence results that match the continuous-time rate of Section 3.
Theorem 5.1. Suppose  $\mathbb{A}$  be a maximal monotone operator,  $p > 0$ , and  $\gamma > 0$ . Consider
$$
x ^ {k} = \mathbb {J} _ {\mathrm {A}} y ^ {k - 1}
$$
$$
y ^ {k} = \frac {k ^ {p}}{k ^ {p} + \gamma} (2 x ^ {k} - y ^ {k - 1}) + \frac {\gamma}{k ^ {p} + \gamma} x ^ {0}
$$
for  $k = 1,2,\ldots$ , with initial condition  $y^0 = x^0\in \mathbb{R}^n$ . Let  $\tilde{\mathbf{A}} x^{k} = y^{k - 1} - x^{k}$  for  $k = 1,2,\dots$ . Then this method exhibits the rates of convergence in Table 2.
Case	p=1,γ≥1	p=1,γ<1	p<1	p>1
||A(xk)||2	O(1/k2)	O(1/k2γ)	O(1/k2p)	O(1)
Table 2: Rates for the discrete-time method of Theorem 5.1.
Note that the method of Theorem 5.1 reduces to APPM when  $\gamma = 1, p = 1$ .