Title: Continuous-time Analysis of Anchor Acceleration
Venue: NeurIPS 2023
Accepted: yes
Keywords: acceleration

convex optimization

continuous-time analysis

monotone operator

monotone inclusion

minimax optimization

fixed-point problem

anchor acceleration
Abstract: Recently, the anchor acceleration, an acceleration mechanism distinct from Nesterov's, has been discovered for minimax optimization and fixed-point problems, but its mechanism is not understood well, much less so than Nesterov acceleration. In this work, we analyze continuous-time models of anchor acceleration. We provide tight, unified analyses for characterizing the convergence rate as a function of the anchor coefficient $\beta(t)$, thereby providing insight into the anchor acceleration mechanism and its accelerated $\mathcal{O}(1/k^2)$-convergence rate. Finally, we present an adaptive method inspired by the continuous-time analyses and establish its effectiveness through theoretical analyses and experiments.

Continuous-time Analysis of Anchor Acceleration

Abstract

Recently, the anchor acceleration, an acceleration mechanism distinct from Nesterov's, has been discovered for minimax optimization and fixed-point problems, but its mechanism is not understood well, much less so than Nesterov acceleration. In this work, we analyze continuous-time models of anchor acceleration. We provide tight, unified analyses for characterizing the convergence rate as a function of the anchor coefficient β ( t ) , thereby providing insight into the anchor acceleration mechanism and its accelerated O (1 /k 2 ) -convergence rate. Finally, we present an adaptive method inspired by the continuous-time analyses and establish its effectiveness through theoretical analyses and experiments.

1 Introduction

Nesterov acceleration [51] is foundational to first-order optimization theory, but the mechanism and its convergence proof are not transparent. One approach to better understand the mechanism is the continuous-time analysis: derive an ODE model of the discrete-time algorithm and analyze the continuous-time dynamics [65, 66]. This approach provides insight into the accelerated dynamics and has led to a series of follow-up work [71, 62, 29].

Recently, a new acceleration mechanism, distinct from Nesterov's, has been discovered. This anchor acceleration for minimax optimization and fixed-point problems [35, 75, 54] has been an intense subject of study, but its mechanism is understood much less than Nesterov acceleration. The various analytic techniques developed to understand Nesterov acceleration, including continuous-time analyses, have only been applied in a very limited manner [59].

Contribution. In this work, we present continuous-time analyses of anchor acceleration. The continuous-time model is the differential inclusion

with initial condition X (0) = X 0 ∈ dom GLYPH<129> , maximal monotone operator GLYPH<129> , and scalar-valued function β ( t ) . The case β ( t ) = 1 t corresponds to the prior anchor-accelerated methods APPM [35], EAG [75], and FEG [42].

We first establish that the differential inclusion is well-posed, despite the anchor coefficient β ( t ) blowing up at t = 0 . We then provide tight, unified analyses for characterizing the convergence rate as a function of the anchor coefficient β ( t ) . This is the first formal and rigorous treatment of this anchored dynamics, and it provides insight into the anchor acceleration mechanism and its accelerated O (1 /k 2 ) -convergence rate. Finally, we present an adaptive method inspired by the continuous-time analyses and establish its effectiveness through theoretical analyses and experiments.

1.1 Preliminaries and notation

We provide the organization for prior works in Appendix A. Here, we review standard definitions and set up the notation.

leads to

Monotone and set-valued operators. We follow the standard definitions of Bauschke and Combettes [15], Ryu and Yin [58]. For the underlying space, consider R n with standard inner product 〈· , ·〉 and norm ‖·‖ . Define domain of GLYPH<129> as dom GLYPH<129> = { x ∈ R n | GLYPH<129> x = ∅} . We say GLYPH<129> is an operator on R n and write GLYPH<129> : R n ⇒ R n if GLYPH<129> maps a point in R n to a subset of R n . We say GLYPH<129> : R n ⇒ R n is monotone if glyph[negationslash]

where the notation means that 〈 u -v, x -y 〉 ≥ 0 for all u ∈ GLYPH<129> x and v ∈ GLYPH<129> y . For µ ∈ (0 , ∞ ) , say GLYPH<129> : R n ⇒ R n is µ -strongly monotone if

Write Gra GLYPH<129> = { ( x, u ) | u ∈ GLYPH<129> x } for the graph of GLYPH<129> . An operator GLYPH<129> is maximally monotone if there is no other monotone GLYPH<130> such that Gra GLYPH<129> ⊂ Gra GLYPH<130> properly, and is maximally µ -strongly monotone if there is no other µ -strongly monotone GLYPH<130> such that Gra GLYPH<129> ⊂ Gra GLYPH<130> properly.

For L ∈ (0 , ∞ ) , single-valued operator GLYPH<148> : R n → R n is L -Lipschitz if

Write GLYPH<138> GLYPH<129> = ( GLYPH<137> + GLYPH<129> ) -1 for the resolvent of GLYPH<129> , while GLYPH<137> : R n → R n is the identity operator. When GLYPH<129> is maximally monotone, it is well known that GLYPH<138> GLYPH<129> is single-valued with dom GLYPH<138> GLYPH<129> = R n .

We say x glyph[star] ∈ R n is a zero of GLYPH<129> if 0 ∈ GLYPH<129> x glyph[star] . We say y glyph[star] is a fixed-point of GLYPH<148> if GLYPH<148> y glyph[star] = y glyph[star] . Write Zer GLYPH<129> for the set of all zeros of GLYPH<129> and Fix GLYPH<148> for the set of all fixed-points of GLYPH<148> .

Monotonicity with continuous curves. We say an operator is differentiable if it is single-valued, continuous, and differentiable as a function. If a differentiable operator GLYPH<129> is monotone and X : [0 , ∞ ) → R n is a differentiable curve, then taking limit h → 0 of

Similarly if GLYPH<129> is furthermore µ -strongly monotone, then

2 Derivation of differential inclusion model of anchor acceleration

2.1 Anchor ODE

Suppose GLYPH<129> : R n ⇒ R n is a maximal monotone operator and β : (0 , ∞ ) → [0 , ∞ ) is a twice differentiable function. Consider differential inclusion

with initial condition X (0) = X 0 ∈ dom( GLYPH<129> ) . We refer to this as the anchor ODE . 1 We say X : [0 , ∞ ) → R n is a solution, if it is absolutely continuous and satisfies (3) for t ∈ (0 , ∞ ) almost everywhere.

Denote S as the subset of [0 , ∞ ) on which X satisfies the differential inclusion. Define

for t ∈ S . Since ˜ GLYPH<129> ( X ( t )) ∈ GLYPH<129> ( X ( t )) for t ∈ S , we say ˜ GLYPH<129> is a selection of GLYPH<129> for t ∈ S . If ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ is bounded on all bounded subsets of S , then we can extend ˜ GLYPH<129> to [0 , ∞ ) while retaining certain favorable properties. We discuss the technical details of this extension in Appendix E.1. The statements of Section 3 are stated with this extension.

2.2 Derivation from discrete methods

We now show that the following instance of the anchor ODE

where X (0) = X 0 is the initial condition and GLYPH<129> : R n → R n is a continuous operator, is a continuoustime model of APPM [35], EAG [75], and FEG [42], which are accelerated methods for monotone inclusion and minimax problems.

Consider APPM with operator h GLYPH<129>

with initial condition y 0 = x 0 . Assume h > 0 and GLYPH<129> : R n → R n is a continuous monotone operator. Using y k -1 = x k + h GLYPH<129> x k obtained from the first line, substituting y k and y k -1 in the second line we get,

Then reorganizing and dividing both sides by h , we have

Identifying x 0 = X 0 , 2 hk = t , and x k = X ( t ) , we have k k +1 = 1 -h hk + h = 1 + O ( h ) and so

Taking limit h → 0 + and dividing both sides by 2 , we get the anchor ODE (4). The correspondence with EAG and FEG are provided in Appendix D.4.

The following theorem establishes a rigorous correspondence between APPM and the anchor ODE for general maximal monotone operators.

glyph[negationslash]

Theorem 2.1. Let GLYPH<129> be a (possibly set-valued) maximal monotone operator and assume Zer GLYPH<129> = ∅ . Let x k be the sequence generated by APPM (5) and X be the solution of the differential inclusion (3) with β ( t ) = 1 t . For all fixed T > 0 ,

2.3 Existence of the solution for β ( t ) = γ t p

To get further insight into the anchor acceleration, we generalize anchor coefficient to β ( t ) = γ t p for p, γ > 0 . We first establish the uniqueness and existence of the solution.

Theorem 2.2. Consider (3) with β ( t ) = γ t p , i.e.

for p, γ > 0 . Then solution of (6) uniquely exists.

2.4 Additional properties of anchor ODE

We state a regularity lemma of the differential inclusion (3), which we believe may be of independent interest. In particular, we use this result several times throughout our various proofs.

Lemma 2.3. Let X ( · ) and Y ( · ) are solutions of the differential inclusion (3) respectively with initial values and anchors X 0 and Y 0 . Then for all t ∈ [0 , ∞ ) ,

Boundedness of trajectories is an immediate corollary of Lemma 2.3. Specifically, suppose X ( · ) is the solution of differential inclusion (3) with initial value X 0 . Then for all X glyph[star] ∈ Zer GLYPH<129> and t ∈ [0 , ∞ ) ,

This follows from setting Y 0 = X glyph[star] in Lemma 2.3.

3 Convergence analysis

We now analyze the convergence rate of ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ 2 for the anchor ODE (3) with β ( t ) = γ t p and γ, p > 0 . The results are organized in Table 1.

Let β be the anchor coefficient function of (3). Define C : [0 , ∞ ) → R as C ( t ) = e ∫ t v β ( s ) ds for some v ∈ [0 , ∞ ] . Note that ˙ C = Cβ and C is unique up to scalar multiple. We call O ( β ( t )) the vanishing speed and O ( 1 C ( t ) ) the contracting speed , and we describe their trade-off in the following.

Loosely speaking, the contracting speed describes how fast the anchor term alone contracts the dynamical system. Consider ˙ X ( t ) = -β ( t )( X ( t ) -a ) for a ∈ R n , a system only with the anchor. Then, X ( t ) = C (0) C ( t ) ( X (0) -a ) + a is the solution, so the flow contracts towards the anchor a with rate 1 C ( t ) . Intuitively speaking, this contracting behavior leads to stability and convergence. On the other hand, the anchor must eventually vanish, since our goal is to converge to an element in Zer GLYPH<129> , not the anchor. Thus the vanishing speed must be fast enough to not slow down the convergence of the flow to Zer GLYPH<129> .

This observation is captured in Figure 1. Consider a monotone linear operator GLYPH<129> = ( 0 1 -1 0 ) on R 2 and β ( t ) = γ t p with γ = 1 and p > 0 . Note if there is no anchor, the ODE reduces to ˙ X = -GLYPH<129> ( X ) which do not converge [31, Chapter 8.2]. Figure 1 shows that with p > 1 , the anchor vanished too early before the flow is contracted enough to result in converging flow. With p < 1 , the flow does converge but the anchor vanished too late, slowing down the convergence. With p = 1 , the convergence is fastest.

p

=3,

γ

=1

p

=1,

γ

=1

p

= 1

2

,

γ

=1

The following theorem formalizes this insight and produces the results of Table 1.

glyph[negationslash]

Theorem 3.1. Suppose GLYPH<129> is a maximal monotone operator with Zer GLYPH<129> = ∅ . Consider (3) with β ( t ) = γ t p . Let ˜ GLYPH<129> ( X ( t )) be the selection of GLYPH<129> ( X ( t )) as in Section 2.1. Then, glyph[negationslash]

Note that

We expect the convergence rate of Theorem 3.1 to be optimized when the terms are balanced. When β ( t ) = 1 t ,

and all three terms are balanced. Indeed, the choice β ( t ) = 1 t corresponds to the optimal discrete-time choice 1 k +2 of APPM or other accelerated methods.

3.1 Proof outline of Theorem 3.1

The proof of Theorem 3.1 follows from Lemma 3.4, which we will introduce later in this section. To derive Lemma 3.4, we introduce a conservation law.

Proposition 3.2. Suppose ˜ GLYPH<129> is Lipschitz continuous and monotone. For t 0 > 0 , define E : (0 , ∞ ) → R as

Then E is a constant function.

The proof of Proposition 3.2 uses dilated coordinate W ( t ) = C ( t )( X ( t ) -X 0 ) to derive its conservation law in the style of Suh et al. [67]. We provide the details in Appendix E.2.

Recall from (1) that 〈 d ds ˜ GLYPH<129> ( X ( s )) , ˙ X ( s ) 〉 ≥ 0 , the integrand of the last term of E is nonnegative. This motivates us to define

Corollary 3.3. Let GLYPH<129> be maximal monotone and β ( t ) = γ t p with p > 0 , γ > 0 . Let ˜ GLYPH<129> ( X ( t )) be the selection of GLYPH<129> ( X ( t )) as in Section 2.1. For t 0 ≥ 0 , define V : [0 , ∞ ) → R as

for t > 0 and V (0) = lim t → 0+ V ( t ) . Then V ( t ) ≤ V (0) holds for t ≥ 0 .

A technical detail is that all terms involving d ds ˜ GLYPH<129> ( X ( s )) have been excluded in the definition of V and this is what allows GLYPH<129> to not be Lipschitz continuous. We provide the details in Appendix E.3.

glyph[negationslash]

Lemma 3.4. Consider the setup of Corollary 3.3. Assume Zer GLYPH<129> = ∅ . Then for t > 0 and X glyph[star] ∈ Zer GLYPH<129> ,

Proof outline of Lemma 3.4. Define

Then, from monotonicity of ˜ GLYPH<129> and Young's inequality,

By Corollary 3.3, 2 V (0) C ( t ) 2 -2 V ( t ) C ( t ) 2 +Φ( t ) ≥ Φ( t ) for t > 0 . Applying (8) and organizing, we can get the desired result. The details are provided in Appendix E.4.

Proof outline of Theorem 3.1. It remains to show that last integral term of Lemma 3.4 is O ( 1 C ( t ) 2 ) + O ( β ( t ) 2 ) + O ( ˙ β ( t ) ) . The details are provided in Appendix E.5.

Before we end this section, we observe how our analysis simplifies in the special case β ( t ) = 1 t . In this case,

and this corresponds to the Lyapunov function of [59, Section 4] for the case γ = 1 . As V (0) = 0 , the conclusion of Lemma 3.4 becomes

which to the best rate in Table 1.

3.2 Point convergence

APPM is an instance of the Halpern method [54, Lemma 3.1], which iterates converge to the element in Zer GLYPH<129> closest to X 0 [34, 73]. The anchor ODE also exhibits this behavior.

glyph[negationslash]

Theorem 3.5. Let GLYPH<129> be a maximal monotone operator with Zer GLYPH<129> = ∅ and X be the solution of (3) . If lim t →∞ ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ = 0 and lim t →∞ 1 /C ( t ) = 0 , then, as t →∞ ,

We provide the proof in Appendix E.6.

4 Tightness of analysis

In this section, we show that the convergence rates of Table 1 are actually tight by considering the dynamics under the explicit example GLYPH<129> = ( 0 1 -1 0 ) . Throughout this section, we denote GLYPH<129> as A when when the operator is linear.

4.1 Explicit solution for linear A

Lemma 4.1. Let A : R n → R n be a linear operator and let β ( t ) = γ t . The series

where Γ denotes the gamma function, is the solution for (3) with GLYPH<129> = A .

Note that when γ = 0 , this is the series definition of the matrix exponential and X ( t ) = e -tA . The solution also has an integral form, which extends to general β ( t ) .

Lemma 4.2. Suppose A : R n → R n is a monotone linear operator. Then

is the solution for (3) with GLYPH<129> = A .

See Appendix F.1.1 and Appendix F.1.2 for details.

4.2 The rates in Table 1 are tight

First, we consider p > 1 for β ( t ) = γ t p .

glyph[negationslash]

Theorem 4.3. Suppose lim t →∞ 1 C ( t ) = 0 , i.e., suppose β ( t ) ∈ L 1 [ t 0 , ∞ ) for some t 0 > 0 . Then there exists an operator GLYPH<129> such that glyph[negationslash]

where X is the solution of (3) .

Note that γ t p ∈ L 1 [ t 0 , ∞ ) when p > 1 . The proof of Theorem 4.3 considers GLYPH<129> = 2 πξ ( 0 1 -1 0 ) for ξ ∈ R and uses the Fourier inversion formula. See Appendix F.2 for details.

Next, we consider β ( t ) = γ t p for cases other than p > 1 .

glyph[negationslash]

Theorem 4.4. Let A = ( 0 1 -1 0 ) , β ( t ) = γ t p , 0 < p ≤ 1 , and γ > 0 . Let X be the solution given by (9) and X 0 = 0 . Let

glyph[negationslash]

Then,

5 Discretized algorithms

In this section, we provide discrete-time convergence results that match the continuous-time rate of Section 3.

Theorem 5.1. Suppose GLYPH<129> be a maximal monotone operator, p > 0 , and γ > 0 . Consider

for k = 1 , 2 , . . . , with initial condition y 0 = x 0 ∈ R n . Let ˜ GLYPH<129> x k = y k -1 -x k for k = 1 , 2 , . . . . Then this method exhibits the rates of convergence in Table 2.

Note that the method of Theorem 5.1 reduces to APPM when γ = 1 , p = 1 .

Proof outline of Theorem 5.1. The general strategy is to find discretized counterparts of corresponding continuous-time analyses. However, directly discretizing the conservation law of Proposition 3.2 was difficult due to technical reasons. Instead, we obtain differently scaled but equivalent conservation laws using dilated coordinates and then performed the discretization. The specific dilated coordinates, inspired by [67], are W 1 ( t ) = X ( t ) -X 0 for p > 1 , W 2 ( t ) = t p ( X ( t ) -X 0 ) for 0 < p < 1 , W 3 ( t ) = t ( X ( t ) -X 0 ) for p = 1 , γ ≥ 1 and W 4 ( t ) = t γ ( X ( t ) -X 0 ) for p = 1 , 0 < γ < 1 .

In the discrete-time analyses, the behavior of the leading-order terms is predictable as they match the continuous-time counterpart. The difficult part is, however, controlling the higher-order terms that were not present in the continuous-time analyses. Through our detailed analyses, we bound such higher-order terms and show that they do not affect the convergence rate in the end. We provide the details in Appendix G.3.

6 Convergence analysis under strong monotonicity

In this section, we analyze the dynamics of the anchor ODE (3) for µ -strongly monotone GLYPH<129> . When β ( t ) = 1 t and GLYPH<129> = ( µ 0 0 µ ) , Lemma 4.1 tells us that GLYPH<129> ( X ( t )) = 1 t ( I -e -tA ) X 0 and therefore that ‖ GLYPH<129> ( X ( t )) ‖ 2 = Θ ( 1 t 2 ) , which is a slow rate for the strongly monotone setup. On the other hand, we will see that β ( t ) = 2 µ e 2 µt -1 is a better choice leading to a faster rate in this setup.

Our analysis of this section is also based on a conservation law, but we use a slightly modified version to exploit strong monotonicity.

Proposition 6.1. Suppose ˜ GLYPH<129> is monotone and Lipschitz continuous. Let X be the solution of (3) and let R : [0 , ∞ ) → (0 , ∞ ) be a differentiable function. For t 0 > 0 , define E : (0 , ∞ ) → R as

Then E is a constant function for t ∈ [0 , ∞ ) .

Proposition 6.1 generalizes Proposition 3.2, since it corresponds to the special case with R ( t ) ≡ 1 . Recall from (2), when GLYPH<129> is µ -strongly monotone we have

This motivates the choice R ( t ) = e µt , since ˙ R ( s ) R ( s ) = µ . From calculation provided in Appendix H.2, the choice β ( t ) = 2 µ e 2 µt -1 makes d ds ( C ( s ) 2 R ( s ) 2 ˙ β ( s ) 2 ) = 0 . Plugging these choices into Proposition 6.1 and following arguments of Section 3, we arrive at the following theorem.

glyph[negationslash]

Theorem 6.2. Let GLYPH<129> be a µ -strongly maximal monotone operator with µ > 0 and assume Zer GLYPH<129> = ∅ . Let X be a solution of the differential inclusion (3) with β ( t ) = 2 µ e 2 µt -1 , i.e. for almost all t ,

Let ˜ GLYPH<129> ( X ( t )) be the selection of GLYPH<129> ( X ( t )) as in Section 2.1. Define V : [0 , ∞ ) → R as

Then V ( t ) ≤ V (0) holds for t ≥ 0 . Furthermore for X glyph[star] ∈ Zer GLYPH<129> ,

In Appendix H.2.3, we show that (10) is a continuous-time model for OS-PPM of Park and Ryu [54]. In Appendix C, we show the existence and uniqueness of the solution.

Since β ( t ) = 2 µ e 2 µt -1 ∈ L 1 [ t 0 , ∞ ) for any t 0 > 0 , Theorem 4.3 implies that ˜ GLYPH<129> ( X ( t )) glyph[notarrowright] 0 when GLYPH<129> is merely monotone. This tells us that the optimal choice of β ( t ) for should depend on the properties of GLYPH<129> . In the following section, we describe how β ( t ) can be chosen to adapt to the operator's properties.

7 Adaptive anchor acceleration and experiments

In this section, we present an adaptive method for choosing the anchor coefficient β , and we theoretically and experimentally show that this choice allows the dynamics to adapt to the operator's properties. It achieves the optimal O (1 /k 2 ) -convergence rate when GLYPH<129> is monotone and an exponential convergence rate when GLYPH<129> is furthermore µ -strongly monotone and Lipschitz continuous.

Theorem 7.1. Suppose ˜ GLYPH<129> is Lipschitz continuous and monotone. Consider the anchor ODE

glyph[negationslash]

with initial condition X (0) = X 0 and ∥ ∥ ˜ GLYPH<129> ( X 0 ) ∥ ∥ = 0 . Suppose the solution exists and ˙ X is continuous at t = 0 . Moreover, suppose β : (0 , ∞ ) → R is well-defined, i.e., no division by zero occurs in the definition of β ( t ) . Then for t > 0 and X glyph[star] ∈ Zer ˜ GLYPH<129> , we have β ( t ) > 0 and

If ˜ GLYPH<129> is furthermore µ -strongly monotone, then for t > 0 ,

We provide the proof in Appendix I.1. Note that anchor coefficient (11) is chosen so that

So left-hand side of (8) is zero and a O ( β ( t ) 2 ) convergence rate is immediate. An analogous discrete-time result is shown in the following theorem.

Theorem 7.2. Let GLYPH<129> be a maximal monotone operator. Let x 0 = y 0 ∈ R n . Consider

with

for k = 1 , 2 , . . . , where ˜ GLYPH<129> x k = y k -1 -x k .

Then β k ≥ 0 and for k = 1 , 2 , . . . and x glyph[star] ∈ Zer GLYPH<129> .

If GLYPH<129> is furthermore µ -strongly monotone and L -Lipschitz continuous, then for k = 1 , 2 , . . . ,

glyph[negationslash]







































0

2000

4000

6000

8000

10000

Iteration count

k

10 -8

10 -6

10 -4

10 -2

10 0

10 2

10 4

10 6

‖

˜

A

xk

‖

2

M

PG-EXTRA

with APPM

with Halpern,

γ

k p

+

γ

with Adaptive

For the monotone setup, the rate ∥ ∥ ˜ GLYPH<129> x k +1 ∥ ∥ 2 ≤ 1 ( k +1) 2 ∥ ∥ x 0 -x glyph[star] ∥ ∥ 2 matches the exact optimal rate of APPM [54]. In the limit µ → 0 , the result for the µ -strongly monotone case reduces to the result for the monotone case. We provide the proof and details of Theorem 7.2 in Appendix I.3.

glyph[negationslash]

The method of Theorem 7.2 is a discrete-time counterpart of the ODE of (11). The extra term ‖ ˜ GLYPH<129> x k ‖ 2 of the denominator in the definition of β k vanishes in the continuous-time limit. We provide further details in Appendix I.2. Analogous to the continuous-time case, a key property of the discrete-time adaptive method is that the counterpart of Φ( t ) is kept nonpositive. In the proof of Lemma I.3, the fact β k < 1 plays the key role in proving this property. The extra term ‖ ˜ GLYPH<129> x k ‖ 2 in the denominator and the fact that 〈 ˜ GLYPH<129> x k , x k -x 0 〉 < 0 when ‖ ˜ GLYPH<129> x k ‖ 2 = 0 leads to β k < 1 .

7.1 Experiment details

We now show an experiment with the method of Theorem 7.2 applied to a decentralized compressed sensing problem Shi et al. [63]. We assume that we have the measurement b i = A ( i ) x + e i , where A ( i ) is a measurement matrix available for each local agent i , x is an unknown shared signal we hope to recover, and e i is an error in measurement. We solve this problem in a decentralized manner in which the local agents keep their measurements private and only communicate with their neighbors.

As in Shi et al. [63], we formulate the problem into an unconstrained glyph[lscript] 1 -regularized least squares problem

and apply PG-EXTRA. We compare vanilla PG-EXTRA with the various anchored versions of PG-EXTRA with β k as in Theorem 5.1 and Theorem 7.2. We show the results in Figure 2. Further details of the experiment are provided in Appendix J.

8 Conclusion

This work introduces a continuous-time model of anchor acceleration, the anchor ODE ˙ X ∈ -GLYPH<129> ( X ) -β ( t )( X -X 0 ) . We characterize the convergence rate as a function of β ( t ) and thereby obtain insight into the anchor acceleration mechanism. Finally, inspired by the continuous-time analyses, we present an adaptive method and establish its effectiveness through theoretical analyses and experiments.

Prior work analyzing continuous-time models of Nesterov acceleration had inspired various follow-up research, such as analyses based on Lagrangian and Hamiltonian mechanics [71, 72, 26], highresolution ODE model [62], and continuized framework [29]. Carrying out similar analyses for the anchor ODE are interesting directions of future work.

Acknowledgments and Disclosure of Funding

This work was supported by the Samsung Science and Technology Foundation (Project Number SSTF-BA2101-02). We thank Jaeyeon Kim for providing valuable feedback. We thank Hangjun Cho for the helpful discussions on well-posedness of ODEs. We also thank anonymous reviewers for the constructive comments.

References

J. K. Alcala, Y. T. Chow, and M. Sunkula. Moving anchor extragradient methods for smooth structured minimax problems. arXiv:2308.12359 , 2023.

V. Apidopoulos, J.-F. Aujol, and C. Dossal. The differential inclusion modeling FISTA algorithm and optimality of convergence rate in the case b ≤ 3 . SIAM Journal on Optimization , 28(1): 551-574, 2018.

H. Attouch and A. Cabot. Asymptotic stabilization of inertial gradient dynamics with timedependent viscosity. Journal of Differential Equations , 263(9):5412-5458, 2017.

H. Attouch and S. C. László. Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators. SIAM Journal on Optimization , 30(4):3252-3283, 2020.

H. Attouch and S. C. László. Convex optimization via inertial algorithms with vanishing Tikhonov regularization: Fast convergence to the minimum norm solution. arXiv:2104.11987 , 2021.

H. Attouch and J. Peypouquet. Convergence of inertial dynamics and proximal algorithms governed by maximally monotone operators. Mathematical Programming , 174(1):391-432, 2019.

H. Attouch, Z. Chbani, J. Peypouquet, and P. Redont. Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Mathematical Programming , 168(1):123-175, 2018.

H. Attouch, Z. Chbani, and H. Riahi. Combining fast inertial dynamics for convex optimization with Tikhonov regularization. Journal of Mathematical Analysis and Applications , 457(2): 1065-1094, 2018.

H. Attouch, Z. Chbani, and H. Riahi. Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 . ESAIM: Control, Optimisation and Calculus of Variations , 25:2, 2019.

H. Attouch, Z. Chbani, J. Fadili, and H. Riahi. Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping. Optimization , pages 1-40, 2021.

J.-P. Aubin and A. Cellina. Differential Inclusions . Springer, 1984.

J.-F. Aujol, C. Dossal, and A. Rondepierre. Optimal convergence rates for Nesterov acceleration. SIAM Journal on Optimization , 29(4):3131-3153, 2019.

J.-B. Baillon and R. E. Bruck. Optimal rates of asymptotic regularity for averaged nonexpansive mappings. Fixed Point Theory and Applications , 128:27-66, 1992.

S. Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta Mathematicae , 3(1):133-181, 1922.

H. H. Bauschke and P. L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces . Springer International Publishing, second edition, 2017.

R. I. Bo¸ t and E. R. Csetnek. A second-order dynamical system with Hessian-driven damping and penalty term associated to variational inequalities. Optimization , 68(7):1265-1277, 2019.

R. I. Bo¸ t and D. A. Hulett. Second order splitting dynamics with vanishing damping for additively structured monotone inclusions. Journal of Dynamics and Differential Equations , 2022.

R. I. Bo¸ t, E. R. Csetnek, and S. C. László. A second-order dynamical approach with variable damping to nonconvex smooth minimization. Applicable Analysis , 99(3):361-378, 2020.

R. I. Bo¸ t, E. R. Csetnek, and S. C. László. Tikhonov regularization of a second order dynamical system with Hessian driven damping. Mathematical Programming , 189(1):151-186, 2021.

R. I. Bot, E. R. Csetnek, and D.-K. Nguyen. Fast OGDA in continuous and discrete time. arXiv:2203.10947 , 2022.

M. Bravo and R. Cominetti. Sharp convergence rates for averaged nonexpansive maps. Israel Journal of Mathematics , 227:163-188, 2018.

A.-L. Cauchy. Méthode générale pour la résolution des systémes d'équations simultanées. Comptes Rendus Hebdomadaires des Séances de l'Académie des Sciences , 25:536-538, 1847.

R. Cominetti, J. A. Soto, and J. Vaisman. On the rate of convergence of Krasnosel'ski˘ ı-Mann iterations and their connection with sums of Bernoullis. Israel Journal of Mathematics , 199(2): 757-772, 2014.

C. Daskalakis, A. Ilyas, V . Syrgkanis, and H. Zeng. Training GANs with optimism. International Conference on Learning Representations , 2018.

J. Diakonikolas. Halpern iteration for near-optimal and parameter-free monotone inclusion and strong solutions to variational inequalities. Conference on Learning Theory , 2020.

J. Diakonikolas and M. I. Jordan. Generalized momentum-based methods: A Hamiltonian perspective. SIAM Journal on Optimization , 31(1):915-944, 2021.

Y. Drori. The exact information-based complexity of smooth convex minimization. Journal of Complexity , 39:1-16, 2017.

J. Eckstein and D. P. Bertsekas. On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming , 55(1):293-318, 1992.

M. Even, R. Berthier, F. Bach, N. Flammarion, H. Hendrikx, P. Gaillard, L. Massoulié, and A. Taylor. Continuized accelerations of deterministic and stochastic gradient descents, and of gossip algorithms. Neural Information Processing Systems , 2021.

George J. Minty. Monotone (nonlinear) operators in Hilbert space. Duke Mathematical Journal , 29(3):341-346, 1962.

I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160 , 2016.

E. Gorbunov, N. Loizou, and G. Gidel. Extragradient method: O (1 /K ) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. International Conference on Artificial Intelligence and Statistics , 2022.

G. Gu and J. Yang. Tight sublinear convergence rate of the proximal point algorithm for maximal monotone inclusion problems. SIAM Journal on Optimization , 30(3):1905-1921, 2020.

B. Halpern. Fixed points of nonexpanding maps. Bulletin of the American Mathematical Society , 73(6):957-961, 1967.

D. Kim. Accelerated proximal point method for maximally monotone operators. Mathematical Programming , 190(1-2):57-87, 2021.

D. Kim and J. A. Fessler. Optimized first-order methods for smooth convex minimization. Mathematical Programming , 159(1-2):81-107, 2016.

D. Kim and J. A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications , 188(1): 192-219, 2021.

J. Kim and I. Yang. Unifying Nesterov's accelerated gradient methods for convex and strongly convex objective functions. International Conference on Machine Learning , 202, 2023.

U. Kohlenbach. On quantitative versions of theorems due to F. E. Browder and R. Wittmann. Advances in Mathematics , 226(3):2764-2795, 2011.

M. A. Krasnosel'skii. Two remarks on the method of successive approximations. Uspekhi Matematicheskikh Nauk , 10(1):123-127, 1955.

J. Lee and E. K. Ryu. Accelerating value iteration with anchoring. Neural Information Processing Systems , 2023.

S. Lee and D. Kim. Fast extra gradient methods for smooth structured nonconvex-nonconcave minimax problems. Neural Information Processing Systems , 2021.

L. Leustean. Rates of asymptotic regularity for Halpern iterations of nonexpansive mappings. Journal of Universal Computer Science , 13(11):1680-1691, 2007.

J. Liang, J. Fadili, and G. Peyré. Convergence rates with inexact non-expansive operators. Mathematical Programming , 159(1):403-434, 2016.

F. Lieder. On the convergence rate of the Halpern-iteration. Optimization Letters , 15(2): 405-418, 2021.

T. Lin and M. I. Jordan. Monotone Inclusions, Acceleration, and Closed-Loop Control. Mathematics of Operations Research , 2023.

W. R. Mann. Mean value methods in iteration. Proceedings of the American Mathematical Society , 4(3):506-510, 1953.

B. Martinet. Régularisation d'inéquations variationnelles par approximations successives. Revue Française d'Informatique et de Recherche Opérationnelle, Série Rouge , 4(3):154-158, 1970.

B. Martinet. Algorithmes Pour La Résolution de Problèmes d'optimisation et de Minimax . PhD thesis, Université Scientifique et Médicale de Grenoble, 1972.

S.-Y. Matsushita. On the convergence rate of the Krasnosel'ski˘ ı-Mann iteration. Bulletin of the Australian Mathematical Society , 96(1):162-170, 2017.

Y. Nesterov. A method of solving a convex programming problem with convergence rate O (1 /k 2 ) . Doklady Akademii Nauk SSSR , 269(3):543-547, 1983.

Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course . Springer, 2004.

√

C. Park, J. Park, and E. K. Ryu. Factor2 acceleration of accelerated gradient methods. Applied Mathematics and Optimization , 2022.

J. Park and E. K. Ryu. Exact optimal accelerated complexity for fixed-point iterations. International Conference on Machine Learning , 2022.

J. Park and E. K. Ryu. Accelerated infeasibility detection of constrained optimization and fixed-point iterations. International Conference on Machine Learning , 2023.

L. D. Popov. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR , 28(5):845-848, 1980.

A. Rakhlin and K. Sridharan. Online learning with predictable sequences. Conference on Learning Theory , 2013.

E. K. Ryu and W. Yin. Large-Scale Convex Optimization via Monotone Operators . Cambridge University Press, 2022.

E. K. Ryu, K. Yuan, and W. Yin. ODE analysis of stochastic gradient methods with optimism and anchoring for minimax problems and GANs. arXiv:1905.10899 , 2019.

S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

A. Salim, L. Condat, D. Kovalev, and P. Richtárik. An optimal algorithm for strongly convex minimization under affine constraints. AISTATS , 2022.

B. Shi, S. S. Du, M. I. Jordan, and W. J. Su. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming , 2021.

W. Shi, Q. Ling, G. Wu, and W. Yin. A proximal gradient algorithm for decentralized composite optimization. IEEE Transactions on Signal Processing , 63(22):6013-6023, 2015.

M. V. Solodov and B. F. Svaiter. A hybrid approximate extragradient - proximal point algorithm using the enlargement of a maximal monotone operator. Set-Valued Analysis , 7(4):323-345, 1999.

W. Su, S. Boyd, and E. J. Candès. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. Neural Information Processing Systems , 2014.

W. Su, S. Boyd, and E. J. Candès. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research , 17(153):1-43, 2016.

J. J. Suh, G. Roh, and E. K. Ryu. Continuous-time analysis of AGM via conservation laws in dilated coordinate systems. International Conference on Machine Learning , 2022.

A. Taylor and Y. Drori. An optimal gradient method for smooth strongly convex minimization. Mathematical Programming , 2022.

Q. Tran-Dinh and Y. Luo. Halpern-type accelerated and splitting algorithms for monotone inclusions. arXiv:2110.08150 , 2021.

B. Van Scoy, R. A. Freeman, and K. M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters , 2(1):49-54, 2018.

A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences , 113(47):E7351-E7358, 2016.

A. C. Wilson, B. Recht, and M. I. Jordan. A Lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research , 22(113):1-34, 2021.

R. Wittmann. Approximation of fixed points of nonexpansive mappings. Archiv der Mathematik , 58(5):486-491, 1992.

T. Wu, K. Yuan, Q. Ling, W. Yin, and A. H. Sayed. Decentralized consensus optimization with asynchrony and delays. IEEE Transactions on Signal and Information Processing over Networks , 4(2):293-307, 2018.

T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O (1 /k 2 ) rate on squared gradient norm. International Conference on Machine Learning , 2021.

T. Yoon and E. K. Ryu. Accelerated minimax algorithms flock together. arXiv:2205.11093 , 2022.

A Prior work

Acceleration for smooth convex function in discrete setting. There had been rich amount of research on acceleration about smooth convex functions. Nesterov [51] introduced accelerated gradient method (AGM), which has a faster O (1 /k 2 ) rate than O (1 /k ) rate of gradient descent [22] in reducing the function value. Optimized gradient method (OGM) [36] improved AGM's rate by a constant factor, and is proven to be optimal [27]. For smooth strongly convex setup, strongly convex AGM [52] achieves an accelerated rate, and further improvements were studied [70, 53, 68, 61]. Recently, OGM-G [37] was introduced as an accelerated method reducing squared gradient magnitude for smooth convex minimization.

Acceleration for smooth convex function in continuous setting. Continuous-time analysis of Nesterov acceleration has been thoroughly studied as well. Su et al. [65, 66] introduced an ODE model of AGM ˙ X ( t ) + r t X + ∇ f ( X ) = 0 , providing f ( X ( t )) -f glyph[star] ∈ O ( 1 /t 2 ) rate for r ≥ 3 . Attouch et al. [7] improved the constant of bound for r > 3 and proved convergence of the trajectories. Attouch et al. [9] achieved O ( 1 /t -2 r/ 3 ) rate for 0 < r < 3 . Apidopoulos et al. [2] generalized their results to differential inclusion with non-differentiable convex function. Furthermore, wide range of variations of AGM ODE has been studied [3, 8, 12, 16, 18, 5, 10, 19]. Also, applications to monotone inclusion problem were studied by Attouch and Peypouquet [6], Attouch and László [4], Bo¸ t and Hulett [17].

Motivated from above continuous-time analysis for accelerated methods, tools analyzing ODEs have further developed. Wibisono et al. [71], Wilson et al. [72] and Kim and Yang [38] adopted Lagrangian mechanics and introduced first, second, unified Bregman Lagrangian to provide unified analysis for generalized family of ODE, where the latter provided analysis for strongly convex AGM. Systemical approach to obtain Lyapunov functions exploiting Hamiltonian mechanics [26] and dilated coordinate system [67] were proposed, and analysis of OGM-G was provided by dilated coordinate framework. Different forms of continuous-time models such as high-resolution ODE [62] and continuized framework [29] were developed.

On the other hand, another type of acceleration called anchor acceleration recently gained attention. As Yoon and Ryu [76] focused, many recently discovered accelerated methods for both minimax optimization and fixed-point problems are based on anchor acceleration. More recently, practical applications of anchor acceleration to detecting infeasibility for constrained optimization problems [55], and accelerating value iteration for dynamic programming and reinforcement learning [41] were introduced as well.

Fixed-point problem. The history of studies on fixed-point problem dates back to the work of Banach [14], which established that the Picard iteration with contractive operator is convergent. Kransnosel'skii-Mann iteration (KM) [40, 47] was introduced, which is a generalization of Picard iteration. Convergence of KM iteration with general nonexpansive operators was proven by Martinet [49]. For iteration of Halpern [34], convergence with wide choice of parameter were shown by Wittmann [73].

The squared norm ‖ y k -GLYPH<148> y k ‖ 2 of fixed-point residual is a common error measure for fixed-point problems. KM iteration was shown to exhibit O (1 /k ) rate [23, 44, 21] and o (1 /k ) -rate [13, 50]. For Halpern iteration, O (1 / (log k ) 2 ) -rate was established by Leustean [43], then improve to O (1 /k ) rate by Kohlenbach [39]. First accelerated O (1 /k 2 ) rate was achieved by Sabach and Shtern [60] and the constant was improved by Lieder [45] by a factor of 16.

It is known that there is an equivalence between solving fixed-point problem and solving monotone inclusion problem [30, 28, 54]. Proximal point method (PPM) [48] achieves O (1 /k ) -rate in terms of ∥ ∥ ˜ GLYPH<129> x k ∥ ∥ 2 [33]. Accelerated proximal point method (APPM) [35] improved the rate to accelerated O ( 1 /k 2 ) -rate. Park and Ryu [54] showed APPM is exactly optimal method for this problem and provided exactly optimal method for µ -strongly monotone operator named OS-PPM, which achieved O ( 1 /e 4 µk ) rate. The optimal methods APPM and OS-PPM are based on anchor acceleration [76].

Minimax problems. Minimax optimization problem of the form min x max y L ( x, y ) have recently gained attention in machine learning society. One of the commonly considered theoretical setting is smooth convex-concave setup, with squared gradient norm as error measure. In terms of ‖ ∂ L ( x, y ) ‖ 2 , classical EG [64] and OG [56, 57, 24] was shown to achieved O (1 /k ) -rate [32]. SGDA [59] achieved O ( 1 /k 2 -2 p ) rate for p > 1 / 2 with introducing the term anchor . With introducing a parameter-free Halpern type method, Diakonikolas [25] achieved O (log k/k 2 ) . Recently, EAG [75] first achieved accelerated rate O (1 /k 2 ) with anchor acceleration, followed by FEG [42], anchored Popov's scheme [69] and moving anchor methods [1]. Fast ODGA [20] also achieved accelerated o (1 /k 2 ) rate. For ∂ L is furthermore strongly monotone with condition number κ , SM-EAG+ [76] achieved accelerated O ( 1 /e 2 kκ ) rate.

However, continuous-time analysis for anchor acceleration is, to the best of our knowledge, insufficient. Continuous-time analyses of acceleration for monotone inclusion problem were studied by Bot et al. [20], Lin and Jordan [46], but they did not consider anchor acceleration. Ryu et al. [59] considered continuous-time analysis of anchor acceleration, but only with limited cases ˙ X ( t ) = -GLYPH<129> ( X ( t )) -γ t ( X -X 0 ) for γ ≥ 1 . In this paper, we provide a unified continuous-time analysis for anchor acceleration with generalized anchor coefficient.

B Proof of Theorem 2.2

B.1 Proof of uniqueness

Proof of uniqueness is immediate from Lemma 2.3, we first prove the lemma.

Proof of Lemma 2.3. It is trivial for t = 0 , so we may assume t > 0 . By monotonicity of GLYPH<129> and Young's inequality, we get the following inequality.

Now define C ( t ) = e ∫ t v β ( s ) ds for some v > 0 , then we see ˙ C ( t ) = C ( t ) β ( t ) . Moving -β ( t ) ‖ X ( t ) -Y ( t ) ‖ 2 to the left hand side and multiplying both sides by C ( t ) , we have

Integrating both sides from glyph[epsilon1] > 0 to t we have

As C is nonnegative and nondecreasing, lim glyph[epsilon1] → 0+ C ( glyph[epsilon1] ) exists. Taking limit glyph[epsilon1] → 0+ both sides we have

Finally, dividing both sides by C ( t ) we conclude

Proof for uniqueness can be done to generalized case (3).

Theorem B.1 (Uniqueness of solutions) . If the solution for (3) exists, it is unique.

Proof. Suppose X 1 , X 2 are solutions of (3) with same initial value X 0 . By Lemma 2.3, we have

Therefore X 1 ( t ) = X 2 ( t ) for all t ∈ [0 , ∞ ) , we get the desired result.

As (6) is special case of (3), uniqueness proof for Theorem 2.2 follows from Theorem B.1.

B.2 Proof of existence

The proof of existence needs tedious work due to the singularity of γ t p at t = 0 , we provide our proof through subsections. Before we start, we provide a short outline of the proof. The proof is basically based on the proof provided in [65] and [11].

We first prove the case GLYPH<129> is Lipschitz. The differential inclusion becomes ODE when GLYPH<129> is Lipschitz, we can adopt similar argument done in [65]. We consider series of ODEs Lipschitz with respect to X , approximating ˙ X ( t ) = -GLYPH<129> ( X ( t )) -γ t p ( X 0 -X ( t )) . The approximated ODEs have solutions by classical theory of ODE, we obtain the true solution by considering proper subsequence of solutions.

As the solution for Lipschitz GLYPH<129> is obtained, we can adopt similar argument done in [11]. We first consider solution X λ with Yosida approximation GLYPH<129> λ = 1 λ ( GLYPH<137> -( GLYPH<137> + λ GLYPH<129> ) -1 ) , which is an approximation of GLYPH<129> that is Lipschitz continuous. Then we can obtain a subsequence X λ n converging to original differential inclusion.

B.2.1 Existence proof for Lipschitz GLYPH<129>

Since we will approximate GLYPH<129> with Lipschitz continuous functions, we first consider the ODE with Lipschitz continuous GLYPH<129> . Theorem B.2. (Existence of solution for Lipschitz GLYPH<129> )

Suppose ˜ A : R n → R n is L-Lipschitz continuous function. Consider the differential equation, with initial value condition ˜ X (0) = X 0 ∈ dom ( ˜ A ) ,

where γ, p > 0 . Then there exists a unique solution ˜ X ∈ C 1 ([0 , ∞ ) , R n ) that satisfies (12) for all t ∈ (0 , ∞ ) . Moreover, for ˙ ˜ X (0) defined by ˙ ˜ X (0) = lim t → 0+ ˜ X ( t ) -X 0 t , following is true

The proof need some preparation. We will think of approximated solutions ˜ X δ , obtain a sequence that converges to solution ˜ X . Thus we first define ˜ X δ . From now, we will denote ˜ A as a L -Lipschitz monotone function.

Definition B.3. Let 0 < δ < 1 . Consider

Since right hand side above is ( L + γ δ p ) -Lipschitz with respect to ˜ X δ , the solution uniquely exists by classical ODE theory. Define the solution as ˜ X δ . Then for positive sequence { δ m } m ∈ N such that δ m < 1 and lim m →∞ δ m = 0 , consider sequence { ˜ X δ m } m ∈ N .

Before we start, we prove a useful lemma we will widely use for the cases that operator is Lipschitz continuous.

Lemma B.4. Let ˜ A : R n → R n a Lipschitz continuous function. Suppose β : D → [0 , ∞ ) be a continuous function with D ⊂ [0 , ∞ ) . Consider differential equation

with X 0 ∈ R n . Let ˜ X : D → R n be a differentiable curve that satisfies above equation for t ∈ D . Then for all 0 ≤ a < b such that [ a, b ] ⊂ D , ˜ X and the composition ˜ A ◦ ˜ X : [ a, b ] → R n are is Lipschitz continuous.

Moreover if ˙ β ( t ) is well-defined and bounded for almost all t ∈ [ a, b ] , then ˙ ˜ X is Lipschitz continuous in [ a, b ] . Thus if β is twice differentiable, ˙ ˜ X is Lipschitz continuous.

As Lipschitz continuous functions, ˜ X , ˜ A ◦ ˜ X and ˙ ˜ X are absolutely continuous functions.

Proof. We first prove ˜ X is Lipschitz continuous. As ˜ A, ˜ X,β are continuous in [ a, b ] we see ˜ A ( ˜ X ( t )) + β ( t )( ˜ X ( t ) -X 0 ) is continuous in [ a, b ] , thus

exists. Since its derivative is bounded by M 1 < ∞ , we have ˜ X is M 1 -Lipshitz continuous. As composition of two Lipschitz continuous functions, ˜ A ◦ ˜ X is Lipschitz continuous.

First observe if β is twice differentiable, ˙ β is bounded in [ a, b ] as it is continuous. Now if ˙ β ( t ) is bounded for t ∈ [ a, b ] , i.e. ∣ ∣ ∣ ˙ β ( t ) ∣ ∣ ∣ ≤ M 2 for some M 2 > 0 , for M 3 = max t ∈ [ a,b ] | β ( t ) | we have

Therefore β ( t )( ˜ X ( t ) -X 0 ) is ( M 2 M 1 | b -a | + M 3 M 1 ) -Lipschitz continuous. Thus as a sum of Lipschitz continuous functions, ˙ ˜ X is Lispchitz continuous.

We will show for every T > 0 , the set of derivatives { ˙ ˜ X δ | δ ∈ (0 , 1) } is uniformly bounded on [0 , T ] and { ˜ X δ m } m ∈ N converges uniformly on [0 , T ] . We first prove the boundedness of derivatives. To do so, we first prove a useful lemma.

Lemma B.5. For 0 < a < b , suppose ˜ X : [ a, b ] → R n satisfies ODE (12) for t ∈ [ a, b ] . Define ˜ U : [ a, b ] → R as

Then ˜ U 1 is nonincreasing if for 0 < p ≤ 1 , and ˜ U 2 is nonincreasing for p > 1 .

Proof. From Lemma B.4 we can check ˜ U 1 and ˜ U 2 are absolutely continuous in [ a, b ] . Therefore it is enough to check the derivative is nonpositive for almost all t . From Lemma B.4, we can differentiate (12) both sides. Thus for almost all t we have

Recall from monotonicity of ˜ A and (1), we know 〈 ˙ ˜ X ( t ) , d dt ˜ A ( ˜ X ( t )) 〉 ≥ 0 for almost all t . Therefore for almost all t ,

(i) 0 < p ≤ 1

(ii) p > 1

We now are ready to prove uniform boundedness of derivatives ˙ ˜ X δ .

Lemma B.6. Take T > 0 . For all t ∈ [0 , T ] and δ ∈ (0 , 1) , below inequality holds.

Proof. We prove first statement by considering two cases.

t ∈ [0 , δ ]

From Lemma B.4, we know ˙ ˜ X δ is absolutely continuous and so ∥ ∥ ∥ ˙ ˜ X δ ∥ ∥ ∥ 2 is absolutely continuous as well. Differentiating (13), for almost all t ∈ (0 , δ ) we have

Now for almost all t ∈ (0 , δ ) ,

(i) 0 < p ≤ 1

From above we know ∥ ∥ ∥ ˙ ˜ X δ ∥ ∥ ∥ is nonincreasing in t ∈ [0 , δ ] , we have ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ ≤ ∥ ∥ ∥ ˙ ˜ X δ (0) ∥ ∥ ∥ = ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ .

(ii) p > 1

Integrating d dt ∥ ∥ ∥ ˙ ˜ X δ ∥ ∥ ∥ 2 ≤ -2 γ δ p ∥ ∥ ∥ ˙ ˜ X δ ∥ ∥ ∥ 2 we have

Organizing with respect to ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ we have

t ≥ δ

Note for t > δ (12) holds, we can apply Lemma B.5.

(i) 0 < p ≤ 1

From (i) we know ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ ≤ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ for t ∈ [0 , δ ] . Therefore ∥ ∥ ∥ ˙ ˜ X δ ( δ ) ∥ ∥ ∥ ≤ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ and we see

From Lemma B.5 we know ˜ U 1 ( t ) = ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ 2 + γp t p +1 ∥ ∥ ∥ ˜ X δ ( t ) -X 0 ∥ ∥ ∥ 2 is nonincreasing. Therefore ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ 2 ≤ ˜ U 1 ( δ ) for t ≥ δ . Since δ < 1 we have

(ii) p > 1

From (i) we know ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ ≤ √ δ p 2 γt ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ for t ∈ [0 , δ ] . Therefore ∥ ∥ ∥ ˙ ˜ X δ ( δ ) ∥ ∥ ∥ ≤ √ δ p -1 2 γ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ and we

Applying (13) and recalling the fact ˜ A is L -Lipschitz, we have

From Lemma B.5 we know ˜ U 2 ( t ) = 1 t p -1 ∥ ∥ ∥ ˙ ˜ X ( t ) ∥ ∥ ∥ 2 + γp t 2 p ∥ ∥ ∥ ˜ X ( t ) -X 0 ∥ ∥ ∥ 2 is nonincreasing. Therefore ∥ ∥ ∥ ˙ ˜ X δ ( t ) ∥ ∥ ∥ 2 t p -1 ≤ ˜ U 2 ( δ ) for t ≥ δ . Since δ < 1 we have

Therefore for t ∈ [0 , T ] we have

From (1) and (2) , we get the desired result.

We now show the sequence { ˜ X δ m } m ∈ N converges uniformly on [0 , T ] for every T > 0 . It is suffices to prove following proposition.

Proposition B.7. For T > 0 , the sequence { ˜ X δ m } m ∈ N is a Cauchy sequence with respect to supremum norm on [0 , T ] .

Proof. Take glyph[epsilon1] > 0 . We want to show, there is N ∈ N such that if n, m > N then ∥ ∥ ∥ ˜ X δ n ( t ) -˜ X δ m ( t ) ∥ ∥ ∥ ≤ glyph[epsilon1] for all t ∈ [0 , ∞ ) .

Define d δ,ν ( t ) = 1 2 ∥ ∥ ∥ ˜ X δ ( t ) -˜ X ν ( t ) ∥ ∥ ∥ 2 . Without loss of generality, we may assume δ ≥ ν . With M dot ( T ) defined in Lemma B.6, we will show d δ,ν ( t ) ≤ 2 δ 2 ˜ M dot ( T ) 2 .

First consider the case 0 ≤ t ≤ δ . By Lemma B.6, we have

Thus for t ∈ [0 , δ ] we have

Now we consider the case t ≥ δ . By monotonicity of ˜ A , we have

Thus we have d δ,ν ( t ) ≤ d δ,ν ( δ ) for t ≥ δ .

Now combining two cases, we have

Since lim k →∞ δ n = 0 , there is N ∈ N such that m>n>N implies d δ n ,δ m ≤ glyph[epsilon1] 2 2 , we're done.

We are now ready to prove Theorem B.2.

Proof. (1) Existence of solution.

From Proposition B.7, we know { ˜ X δ m } m ∈ N converging uniformly on [0 , T ] for every T > . Denote the limit as ˜ X , i.e. define ˜ X : [0 , ∞ ) → R n as

We can check ˜ X (0) = X 0 easily since ˜ X δ m (0) = X 0 for all m ∈ N . It remains to show ˜ X satisfies (12). Take t > 0 . We wish to show

Consider h , δ , T such that 0 < | h | < t , 0 < δ < min { 1 , t -| h |} and T > t + | h | . Then ( t -| h | , t + | h | ) ∈ [0 , T ] and ˙ ˜ X δ ( t ) = -˜ A ( ˜ X δ ( t )) -γ t p ( ˜ X δ ( t ) -X 0 ) . Consider inequality

We now show right hand side goes to zero as h → 0 . The point of the proof is, ˜ X δ converges uniformly to ˜ X and ¨ ˜ X δ is uniformly bounded on [ t -| h | , T ] .

From (14) we have

Also from (14) and since ˜ A is L -Lipschitz continuous,

Now from Lemma B.4 we have ˙ X δ ( t ) is absolutely continuous, thus

Observe, for almost every u ∈ [ t -| h | , t + | h | ] ⊂ [0 , T ] by Lemma B.6 we have

Note M is independent of h or δ . While obtaining the inequality, we used the fact that ˜ A ( ˜ X δ ( · )) is L ˜ M dot ( T ) -Lipschitz continuous in [0 , T ] . Now

Now consider δ = h 2 with h small enough that satisfies | h | < 1 and | h | + h 2 < t . Then the conditions | h | < t , 0 < δ < min { 1 , t -| h |} hold, above arguments are valid. Gathering above results, we have

which implies the desired result.

The value and continuity of ˙ ˜ X ( t ) at t = 0 . Define C ( t ) as

Then we see for t > 0

Integrating both sides from glyph[epsilon1] > 0 to t we have

As C is a nondecreasing function and bounded below by 0 , lim glyph[epsilon1] → 0+ C ( glyph[epsilon1] ) exists, taking limit glyph[epsilon1] → 0+ we have

glyph[negationslash]

Dividing both sides by tC ( t ) , with change of variable v = s/t , we have

glyph[negationslash]

Note γ 1 -p t 1 -p ( v 1 -p -1 ) ≤ 0 for v ∈ [0 , 1] , since γ 1 -p and ( v 1 -p -1 ) has opposite sign either 0 < p < 1 or p > 1 . Therefore e γ 1 -p t 1 -p ( v 1 -p -1 ) ≤ 1 . Also, ˜ A ( ˜ X ( tv )) is bounded for v ∈ [0 , 1] since A ( ˜ X ( · )) is continuous by Lemma B.4.

So we can apply dominated convergence theorem and take limit t → 0+ . Since

Observe glyph[negationslash]

for v = 0 , we have

We now check ˙ ˜ X ( t ) is continuous at t = 0 .

(i) 0 < p ≤ 1

For t > 0 , we know

Observe

Now from ODE ˙ ˜ X ( t ) = -˜ A ( ˜ X ( t )) -γ t p ( ˜ X ( t ) -X 0 ) , by taking limit t → 0+ we have

Therefore ˙ ˜ X ( t ) is continuous at t = 0 .

(ii) p > 1

For p > 1 , we don't know the value of lim t → 0 γ t p ( ˜ X ( t ) -X 0 ) . Thus we first find the limit. Let's go back to

Recall ˙ C ( t ) = γ t p C ( t ) . By taking integration by parts for the right hand side, we have

Where we know ˜ A ( ˜ X ( s )) is differentiable almost everywhere from Lemma B.4. Now, divide both sides by t p C ( t ) γ . Then for s = tv we have

From Proposition B.7 and since ˜ X satisfies (12) for t > 0 , for s ∈ (0 , t ] we have

From Lemma B.6 we know ∥ ∥ ∥ ˙ ˜ X δ m ( s ) ∥ ∥ ∥ ≤ ˜ M dot ( t ) for s ∈ [0 , t ] , taking limit m → ∞ we have ∥ ∥ ∥ ˙ ˜ X ( s ) ∥ ∥ ∥ ≤ ˜ M dot ( t ) for s ∈ [0 , t ] . Thus ˜ X ( s ) becomes ˜ M dot ( t ) -Lipschitz continuous in s ∈ [0 , t ] . And so ( ˜ A ◦ ˜ X )( s ) becomes L ˜ M dot ( t ) -Lipschitz continuous in s ∈ [0 , t ] , we have ∥ ∥ ∥ d ds ˜ A ( ˜ X ( s )) ∥ ∥ ∥ ≤ L ˜ M dot ( t ) for almost all s ∈ [0 , t ] . Moreover C ( tv ) /C ( t ) is bounded for v ∈ [0 , 1] since C is a nonnegative nondecreasing function. Therefore we can again apply dominated convergence theorem. Reminding lim t → 0+ C ( tv ) C ( t ) = 0 for p > 1 , taking limit t → 0+ we have

Finally we have

Before we move on to original inclusion, we prove important corollaries that bound ∥ ∥ ∥ ˙ ˜ X ( t ) ∥ ∥ ∥ and ∥ ∥ ∥ ˜ A ( ˜ X ( t )) ∥ ∥ ∥ uniformly on [0 , T ] . Note the main difference between following corollary and Lemma B.6 is that the dependency on Lipschitz constant L is dropped for the case p > 1 , which will be crucial in the next section.

Corollary B.8. Denote ˜ X as the solution of (12) . Then for T > 0 , following inequality is true for t ∈ [0 , T ] .

Proof. (i) 0 < p ≤ 1

As ˜ X is the solution for (12), from Lemma B.5 we know

is a nonincreasing function. From (15), (16) and continuity of ˙ X ( t ) at t = 0 , we have

Therefore ∥ ∥ ∥ ˙ ˜ X ( t ) ∥ ∥ ∥ 2 ≤ ˜ U 1 ( t ) ≤ lim glyph[epsilon1] → 0+ ˜ U 1 ( glyph[epsilon1] ) for t > 0 , we get the desired result.

(ii) p > 1

As ˜ X is the solution for (12), from Lemma B.5 we know

is a nonincreasing function. However, as we don't know the value of lim t → 0+ 1 t p -1 ∥ ∥ ∥ ˙ ˜ X ( t ) ∥ ∥ ∥ 2 , we first calculate it. To do so, we consider

In the proof of Lemma B.5, we have observed its derivative becomes

For glyph[epsilon1] > 0 , integrating from glyph[epsilon1] to t we have

We consider taking limit glyph[epsilon1] → 0+ . Observe from (17) and (15) we know lim t → 0+ ˜ X -X 0 t p = -˜ A ( X 0 ) γ and ∥ ∥ ∥ ˙ ˜ X (0) ∥ ∥ ∥ = 0 , and therefore we have

Moreover as lim t → 0+ ‖ ˜ X ( t ) -X 0 ‖ 2 t 2 p = ‖ ˜ A ( X 0 ) ‖ 2 γ 2 , there is δ > 0 such that 0 < s < δ implies ‖ ˜ X ( t ) -X 0 ‖ 2 s 2 p ≤ 2 ‖ ˜ A ( X 0 ) ‖ 2 γ 2 . Recalling p > 1 , for 0 < glyph[epsilon1] < δ we have

therefore

Thus the integral is well-defined when glyph[epsilon1] → 0+ . Taking limit glyph[epsilon1] → 0+ we have

Moving γp t p +1 ∥ ∥ ∥ ˜ X ( t ) -X 0 ∥ ∥ ∥ 2 to the right hand side, we get a inequality for ∥ ∥ ∥ ˙ ˜ X ( t ) ∥ ∥ ∥ 2 ,

Observe, by L'Hôpital's rule we have

Now dividing t p -1 to previous inequality and taking limit, we conclude

Thus we have lim t → 0+ 1 t p -1 ∥ ∥ ∥ ˙ ˜ X ∥ ∥ ∥ 2 = 0 . Therefore,

From ∥ ∥ ∥ ˙ ˜ X ∥ ∥ ∥ 2 t p -1 ≤ lim glyph[epsilon1] → 0+ ˜ U 2 ( glyph[epsilon1] ) = p γ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ 2 , we conclude the desired result.

Corollary B.9. Denote ˜ X as the solution of (12) . Then for T > 0 , following inequality is true for t ∈ [0 , T ] .

Proof. First observe,

The inequality comes from Young's inequality. Now we consider each case.

For this case, the terms on the right hand side exactly become U ( t ) . Therefore from (18)

(i) p = 1 ˜ 1

(ii) 0 < p < 1

Recall from the proof of Corollary B.8 we know ˜ U 1 ( t ) ≤ lim glyph[epsilon1] → 0+ ˜ U 1 ( glyph[epsilon1] ) = ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ 2 . Therefore we have γp t p +1 ∥ ∥ ∥ ˜ X ( t ) -X 0 ∥ ∥ ∥ 2 ≤ ˜ U 1 ( t ) ≤ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ 2 , applying Corollary B.8 we get

(iii) p > 1

Recall from the proof of Corollary B.8 we know ˜ U 2 ( t ) ≤ lim glyph[epsilon1] → 0+ ˜ U 2 ( glyph[epsilon1] ) = p γ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ 2 . Therefore we have

and

Proof. Replace ˜ A with GLYPH<129> λ and ˜ X with X λ in Corollary B.8 and Corollary B.9. Applying ‖ GLYPH<129> λ ( X 0 ) ‖ ≤ ‖ m ( GLYPH<129> ( X 0 )) ‖ from Lemma B.10 (iii), we're done.

B.2.2 Existence proof for general GLYPH<129>

Now we move on to the original inclusion. As noticed before, we will approximate GLYPH<129> with a Liptschitz function GLYPH<129> λ called Yosida approximation. We define and state some facts about GLYPH<129> λ as a lemma, and use it without proof. For the ones who are interested in proofs, see [11, Chpater 3.1, Theorem 2].

Lemma B.10. Define GLYPH<129> λ : R n → R n as

This is so called Yosida approximation of GLYPH<129> . Followings are true.

(i) GLYPH<129> λ is 1 λ -Lipschitz continuous and maximal monotone.

(ii) lim λ → 0+ GLYPH<129> λ x = m ( GLYPH<129> ( x )) . Here m ( GLYPH<129> ( x )) is defined as m ( GLYPH<129> ( x )) = Π GLYPH<129> ( x ) (0) , the element of GLYPH<129> ( x )

(iii) ‖ GLYPH<129> λ ( x ) ‖ ≤ ‖ m ( GLYPH<129> ( x )) ‖ .

(iv) ∀ x ∈ R n , GLYPH<129> λ ( x ) ∈ GLYPH<129> ( GLYPH<138> λ GLYPH<129> x ) .

Now we can state the proposition that proves existence of Theorem 2.2

Proposition B.11. For Yosida approximation GLYPH<129> λ , consider the ODE

with initial value condition X λ (0) = X 0 ∈ dom ( GLYPH<129> ) . The solution uniquely exists by Theorem B.2, denote the solution as X λ . Now for T > 0 and a positive sequence { λ n } n ∈ N such that lim n →∞ λ n = 0 , define a sequence of solutions as F T = { X λ n : [0 , T ] → R n | m ∈ N } . Then there is a subsequence { λ n k } k ∈ N such that X λ n k converges to the solution of (6) uniformly on [0 , T ] .

From Corollary B.8, Corollary B.9 and Lemma B.10 (iii), following lemma is immediate.

Lemma B.12. Let X λ be the solution of (19) . Then following is true for t ∈ [0 , T ] for all T > 0 .

with minimal norm.

While we're concluding the existence of converging sequence, we will exploit following lemma from [11, Chapter 0.3, Theorem 4]. For convenience, we restate the lemma here.

Lemma B.13. Let us consider a sequence of absolutely continuous functions x k ( · ) from an interval I of R to a Banach space X satisfying

(i) ∀ t ∈ I , { x k ( t ) } k ∈ N is a relatively compact subset of X

(ii) there exists a positive function c ( · ) ∈ L 1 ( I, [0 , ∞ )) such that, for almost all t ∈ I , ‖ ˙ x k ( t ) ‖ ≤ c ( t )

Then there exists a subsequence (again denoted by) x k ( · ) converging to an absolutely continuous function x ( · ) from I to X in the sense that

(i) x k ( · ) converges to x ( · ) over compact subsets of I

(ii) ˙ x k ( · ) converges weakly to ˙ x ( · ) in L 1 ( I, X )

From Lemma B.12 we can immediately check norm of all derivatives ˙ X λ m are bounded by M dot ( T ) . So condition (ii) holds with M dot ( T ) . For condition (i), we prove F T is convergent in C ([0 , T ] , R n ) .

Lemma B.14. F T is convergent sequence in C ([0 , T ] , R n ) . In other words, ∀ glyph[epsilon1] > 0 , there is N > 0 such that n, m > N implies sup t ∈ [0 ,T ] ‖ X λ n ( t ) -X λ m ( t ) ‖ < glyph[epsilon1]

Proof. We will show X λ n is Cauchy sequence in C ([0 , T ] , R n ) . Let ν, λ > 0 . From (19), we see for t ∈ (0 , T ]

From definition of resolvent we know GLYPH<137> -GLYPH<138> λ GLYPH<129> = λ GLYPH<129> λ . And from Lemma B.10 (iv) we know GLYPH<129> λ ( X λ ( t )) ∈ GLYPH<129> ( GLYPH<138> λ GLYPH<129> ( X λ ( t ))) . Thus from monotone inequality we see

By Young's inequality

Then integrating both sides from 0 to t we have

Now take glyph[epsilon1] > 0 . Then there is N > 0 such that for n > N , λ n < glyph[epsilon1] TM GLYPH<129> ( T ) 2 holds. Then for t ∈ [0 , T ] , n, m > N , we have

Therefore we get the desired result.

Finally, we are ready to prove Proposition B.11, which implies the main theorem.

Proof of Proposition B.11. Take T > 0 . We know X λ n uniformly converges on [0 , T ] by Lemma B.14. Name the limit as X , i.e. define X : [0 , T ] → R n as X ( t ) = lim n →∞ X λ n ( t ) . Then as X λ n (0) = X 0 for all n ∈ N , we see X satisfies the initial condition. It remains to show X satisfies (6) almost everywhere.

Recall { ˙ X λ n } is bounded in L ∞ ([0 , T ] , R n ) by Lemma B.12. Thus we can apply Lemma B.13, there is a subsequence { ˙ X λ n k } converges weakly to ˙ X in L 1 ([0 , T ] , R n ) . Furtheremore we have ˙ X ∈ L ∞ ([0 , T ] , R n ) and so { ˙ X λ n k } also converges weakly to ˙ X in L 2 ([0 , T ] , R n ) as well.

For λ > 0 , define f λ : [0 , T ] → R n as

Then for f : [0 , T ] → R n defined as f ( t ) = γ t p ( X ( t ) -X 0 ) for t > 0 and f (0) = 0 , we have lim k →∞ f λ n k ( t ) = f ( t ) . As ‖ f λ ( t ) ‖ = ∥ ∥ ∥ ˙ X λ ( t ) + GLYPH<129> λ n ( X λ )( t ) ∥ ∥ ∥ ≤ M dot ( T ) + M GLYPH<129> ( T ) for t ∈ (0 , T ] by Lemma B.12, we have

Therefore by dominated convergence theorem we have

we conclude f λ n k strongly converges to f in L 2 ([0 , T ] , R n ) .

Now consider F λ : [0 , T ] → R n defined as

Note since X λ are solution to ODE (19), we have

Then for F : [0 , T ] → R n defined as

we see { F λ n k } k ∈ N converges weakly to F in L 2 ([0 , T ] , R n ) .

On the other hand, by Lemma B.10 (iv) and the fact -F λ n k ( t ) = GLYPH<129> λ n k ( X λ n k ( t )) , we see

Observe, from the definition of GLYPH<129> λ n and Lemma B.12, we see

Since X λ n converges to X in C ([0 , T ] , R n ) , by taking n →∞ above inequality we see GLYPH<138> λ n GLYPH<129> ( X λ n ) also converges to X in C ([0 , T ] , R n ) .

Now for A : L 2 ([0 , T ] , R n ) → L 2 ([0 , T ] , R n ) defined as ( A ( x ))( t ) = GLYPH<129> ( x ( t )) almost everywhere, by [11, Chapter 3.1, Proposition 4], A is maximal monotone since GLYPH<129> is maximal monotone. Since -F λ k weakly converges to -F in L 2 ([0 , T ] , R n )

and GLYPH<138> λ k GLYPH<129> ( X λ k ) strongly converges to X in L 2 ([0 , T ] , R n ) , by [11, Chapter 3.1, Proposition 2] we have -F ∈ A ( X ) in L 2 ([0 , T ] , R n ) . Therefore for almost all t ∈ (0 , T ] we have

Reorganizing the result with respect to ˙ X , we have following is true for almost all t ∈ (0 , T ]

Since T > 0 was arbitrary, we conclude X satisfies above inclusion for almost all t ∈ (0 , ∞ ) .

C Proof of existence and uniquencess of the solution of (10)

As uniqueness comes from Theorem B.1, we only need to show the existence. What we need for the existence proof are

(i) Nonincreasing function ˜ U ( t ) which contains ∥ ∥ ∥ ˙ ˜ X ∥ ∥ ∥ 2 as in Lemma B.5.

(ii) Uniform boundedness of ˙ ˜ X δ ( t ) for t ∈ [0 , T ] as shown in Lemma B.6.

(iii) ˙ ˜ X (0) = lim t → 0+ ˜ X ( t ) -X 0 t = lim t → 0+ ˙ ˜ X ( t ) as shown in the existence proof of Lipschitz case.

(iv) Uniform boundedness of ∥ ∥ ∥ ˙ X λ ( t ) ∥ ∥ ∥ and ‖ GLYPH<129> λ ( X λ ( t )) ‖ for t ∈ [0 , T ] as shown in Lemma B.12

We now show these steps can be also done to the β ( t ) = 2 µ e 2 µt -1 .

(i) Nonincreasing function ˜ U ( t ) which contains ∥ ∥ ∥ ˙ ˜ X ∥ ∥ ∥ 2

From for almost all t we have

Define

Therefore for almost all t > 0 ,

(ii) Uniform boundedness of ˙ ˜ X δ ( t )

As (13), we define ˙ ˜ X δ ( t ) as the solution of

Again with same arguments of Lemma B.6 we have ∥ ∥ ∥ ˜ X δ ( δ ) -X 0 ∥ ∥ ∥ ≤ δ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ and ∥ ∥ ∥ ˙ ˜ X δ ( δ ) ∥ ∥ ∥ ≤ ∥ ∥ ∥ ˜ A ( X 0 ) ∥ ∥ ∥ . Now for t ∈ [0 , T ]

(iii) ˙ ˜ X (0) = lim t → 0+ ˜ X ( t ) -X 0 t = lim t → 0+ ˙ ˜ X ( t )

Define C ( t ) := 1 -e -2 µt . Then ˙ C ( t ) = C ( t ) β ( t ) for β ( t ) = 2 µ e 2 µt -1 . And since

with same argument done to arrive (15), we have

Now from ODE ˙ ˜ X ( t ) = -˜ A ( ˜ X ( t )) -2 µ e 2 µt -1 ( ˜ X ( t ) -X 0 ) , by taking limit both sides by t → 0+ we have

Therefore, lim t → 0+ ˜ X ( t ) -X 0 t = lim t → 0+ ˙ ˜ X ( t ) .

(iv) Uniform boundedness of ∥ ∥ ∥ ˙ X λ ( t ) ∥ ∥ ∥ and ‖ GLYPH<129> λ ( t ) ‖ for t ∈ [0 , T ]

Recall U ( t ) = e -2 µt ∥ ∥ ∥ ˙ X λ ( t ) ∥ ∥ ∥ 2 + ( 2 µ e 2 µt -1 ) 2 ‖ X λ ( t ) -X 0 ‖ 2 is nonincreasing. So from (iii), we have

Therefore we have from Lemma B.10 (iii)

and by Young's inequality

Therefore

D Omitted proofs for derivation of anchor ODE (4)

D.1 Preparation for the proof of Theorem 2.1

We first provide the boundedness of trajectories as a lemma. As mentioned in the discussion after Lemma 2.3, boundedness of trajectories is an immediate corollary of Lemma 2.3. However, to address cases that are slightly more generalized, we present a proof using an argument similar to the one used in the proof of Lemma 2.3. Note the proof argument of following lemma is valid for the solution of differential equation (11) with satisfying the assumptions in Theorem 7.1 as well.

Lemma D.1. (Boundedness of solutions) Suppose X ( · ) is the solution of the differential inclusion (3) . Then for all X glyph[star] ∈ Zer GLYPH<129> , t ∈ [0 , ∞ ) , following holds.

Proof. It is trivial for t = 0 , so we may assume t > 0 .

Take X glyph[star] ∈ Zer GLYPH<129> . By monotonicity of GLYPH<129> and Young's inequality, we get the following inequality.

Now again define C ( t ) = e ∫ t v β ( s ) ds for some v > 0 , then we see ˙ C ( t ) = C ( t ) β ( t ) . Moving -β ( t ) ‖ X ( t ) -X glyph[star] ‖ 2 to the left hand side and multiplying both sides by C ( t ) , we have

Integrating both sides from glyph[epsilon1] > 0 to t we have

As β > 0 , we have C is nonnegative and nondecreasing, lim glyph[epsilon1] → 0+ C ( glyph[epsilon1] ) exists. Taking limit glyph[epsilon1] → 0+ both sides we have and dividing both sides by C ( t ) we conclude

The latter statement holds directly from triangular inequality,

Following lemma shows APPM is an instance of Halpern method. It is immediate from induction, but we state it as a lemma due as its importance.

Lemma D.2. Consider a method defined as

for k = 0 , 1 , . . . , with initial condition y 0 = x 0 . Then for reflected resolvent GLYPH<146> GLYPH<129> defined as GLYPH<146> GLYPH<129> = 2 GLYPH<138> GLYPH<129> -GLYPH<137> , above method is equivalent to

when GLYPH<148> = GLYPH<146> GLYPH<129> , ˜ y 0 = y 0 . Here equivalence means ˜ y k = y k holds for k = 0 , 1 , . . . .

Proof. Proof by induction. As ˜ y 0 = y 0 by assumption, the statement is true for k = 0 . Now suppose y k = ˜ y k holds for k ∈ N , then

Thus ˜ y k +1 = y k +1 , the statement is true for k +1 . By induction, we get the desired result.

D.2 Proof of Theorem 2.1

Let GLYPH<129> : R n → R n be a maximal monotone operator, and h, λ, δ > 0 . Again, denote GLYPH<129> λ = 1 λ ( GLYPH<137> -GLYPH<129> λ GLYPH<129> ) = 1 λ ( GLYPH<137> -( GLYPH<137> + λ GLYPH<129> ) -1 ) . Since various kind of terms appear in the proof, we first organize the terms and notations.

X : Solution of differential inclusion,

X λ : Solution of differential equation,

X λ,δ : Solution of approximated differential equation,

X k λ,δ : Sequence obtained by taking Euler discretization of ODE (23),

x k h,λ : Sequence obtained from APPM with operator h GLYPH<129> λ , i.e.

x k h : Sequence obtained from APPM with operator h GLYPH<129> , i.e.

We want to show for fixed T > 0 ,

Equivalently we may show for fixed T > 0 , for every { h n } n ∈ N such that h n > 0 and converges to 0 ,

We will show this by considering inequality

Our goal is to show, for every { h n } n ∈ N such that h n > 0 and converges to 0 , there is a sequence { ( δ n , λ n ) } n ∈ N such that lim n →∞ sup 0 ≤ k< T 2 hn S ( h n , λ n , δ n , k ) = 0 , and thus

To clarify our goal, we need to find proper { ( δ n , λ n ) } n ∈ N in terms of { h n } n ∈ N . As { h n } n ∈ N is determined, ∥ ∥ x k h n -X (2 h n k ) ∥ ∥ is fixed and doesn't change by the choice of { ( δ n , λ n ) } n ∈ N . But if we find { ( δ n , λ n ) } n ∈ N that makes S ( h n , λ n , δ n , k ) small, since (24) holds for any choice of δ, λ , right choice of { ( δ n , λ n ) } n ∈ N can gaurantee ∥ ∥ x k h n -X (2 h n k ) ∥ ∥ is small. Thus to find such { ( δ n , λ n ) } n ∈ N , we will observe each terms in S to find the required conditions.

Lemma D.3. For h, λ, δ > 0 following is true.

For L λ,δ = max { 1 λ , √ 2 λ ‖ m ( GLYPH<129> ( X 0 )) ‖ , 1 δ , 4 √ 2 δ ‖ m ( GLYPH<129> ( X 0 )) ‖ } ,

Further more if 0 < h λ < 1 2 ,

We prove this lemma in next subsection, here we assume the lemma is true and prove Theorem 2.1. Suppose above lemma is true. The calculations are messy but the strategy is simple; balancing the speed of the terms h, δ, λ going zero to make above terms reach to zero. Above lemma motivate to take sequences as

where M = max { 1 , 4 √ 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ } . When lim n →∞ h n → 0 with h n > 0 , we can easily check lim n →∞ δ n = 0 and lim n →∞ λ n = 0 . So the cases (i), (ii), (iii) go to zero.

Now observe from the definition of λ n we have,

Thus

Therefore when lim n →∞ h n → 0 ,

As lim n →∞ h n = 0 , without loss of generality we may assume h n < 1 . Since h n > 0 we have λ n is well-defined and satisfies the condition for Lemma D.3. Thus terms for the case (iv) and (v) go to zero as well. Therefore we have lim n →∞ sup 0 ≤ k< T 2 hn S ( h n , λ n , δ n , k ) = 0 , as { h n } n ∈ N is arbitrary, we get the desired result.

D.2.1 Proof for case (i), (ii), (iii), (iv) of Lemma D.3

As case proof for (v) need lot of work, we provide it in a different subsection and here we provide the proofs for the cases from (i) to (iv).

)

This is result of Lemma B.14. Considering (20) with p = 1 , taking limit ν → 0 we know

(ii) ‖ X λ (2 hk ) -X λ,δ (2 hk ) ‖ = O ( δ )

This is result of Proposition B.7. Consider (14) with γ = p = 1 for Lemma B.6 and taking limit ν → 0 . Then applying Lemma B.10 (iii) we have

We first show show a general fact about Yosida approximation and resolvent. From [15, Proposition 23.7 (iv)] we have

From [15, Proposition 23.31 (iii)], we have

From Lemma D.2, we know the iteration of y k sequence in (5) is equivalent to below sequence

Using this alternating form we have

The first inequality comes from triangular inequality. The second inequality is from nonexpansiveness of reflected resolvent GLYPH<146> GLYPH<129> = 2 GLYPH<138> GLYPH<129> -GLYPH<137> , [15, Corollary 23.11]. The third inequality is from the inequality shown previously. The last inequality comes from the convergence rate of APPM[35, Theorem 4.1], ∥ ∥ y k h -GLYPH<138> h GLYPH<129> ( y k h ) ∥ ∥ ≤ ‖ X 0 -X glyph[star] ‖ k . Now multiplying both sides by k +1 and summing up from 0 to k we get

Finally, from the relation between x k and y k in APPM we have

Combining two facts we get

From (23), we can consider ˙ X λ,δ as of function F : R n × [0 , ∞ ) → R n defined as below

Note F is 2 max { 1 λ , 1 δ } -Lipschitz with respect to X .

For convenience name α = 2 h . Define glyph[epsilon1] k := X λ,δ ( αk ) -X λ,δ . By definition of Euler discretization and from fundamental theorem of calculus, we have the following

k

From Lemma B.4 we have F ( X λ,δ , t ) = ˙ X λ,δ ( t ) is Lipschitz continuous respect to t , so ¨ X λ,δ is defined almost everywhere and fundamental theorem of calculus is valid. Therefore we have

As F is 2 max { 1 λ , 1 δ } -Lipschitz with respect to first variable, we have

Now we observe ∥ ∥ ∥ ¨ X λ,δ ∥ ∥ ∥ is bounded. By differentiating ˙ X λ,δ , as

for t > δ , we have for almost every t

Considering γ = 1 , p = 1 to Lemma B.6 and from Lemma B.10 (iv) we have

and thus

And since GLYPH<129> λ is 1 λ -Lipschitz, we know GLYPH<129> λ ◦ X λ,δ is 1 λ √ 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ -Lispchitz, thus we have for almost all t ,

Applying these facts we have

Therefore

Now for L δ,λ = max { 1 λ , 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ λ , 1 δ , 4 2 δ ‖ m ( GLYPH<129> ( X 0 )) ‖ } , we show ‖ glyph[epsilon1] k ‖ ≤ he 2 L λ,δ T .

Multiplying (1 + 2 αL λ,δ ) -( k +1) to (26) we have

As ‖ glyph[epsilon1] 0 ‖ = ‖ X 0 -X (0) ‖ = 0 , summing up from 0 to k -1 we have

Multiplying (1 + 2 αL λ,δ ) k to both sides and applying α = 2 h we have

Now from applying k ≤ T 2 h

Therefore

D.3 Proof for case (v) of Lemma D.3

As APPM has coefficient 1 k +1 , we consider X k +1 λ,δ instead of X k λ,δ due to calculation simplicity. From triangular inequality, we have

We will show

First one is simple. Since X k +1 λ,δ = X k λ,δ +2 hF ( X λ,δ , 2 hk ) and F is 2 max { 1 λ , 1 δ } -Lipschitz with respect to the first variable, we have

Second one is complicated, we present our proof with dividing steps to subsections.

D.3.1 Recursive inequality for glyph[epsilon1] k = ∥ ∥ ∥ X k +1 λ,δ -x k h,λ ∥ ∥ ∥

Define glyph[epsilon1] k = ∥ ∥ ∥ X k +1 λ,δ -x k h,λ ∥ ∥ ∥ . Recall, X λ,δ was solution of approximated ODE

We now wish to write glyph[epsilon1] k +1 in terms of glyph[epsilon1] k . As glyph[epsilon1] k +1 involves X k +2 λ,δ , we first write it explicitly.

Now we find recursive inequality considering two cases.

(i) k +1 ≥ δ 2 h Recall APPM (5) was defined as

and substituting y k -1 h,λ = x k h,λ + h GLYPH<129> λ ( x k h,λ ) , we get a one line expression

Rewriting X k +2 λ,δ to make easier to compare with above,

As GLYPH<129> λ is 1 λ -Lipschitz

Organizing,

(ii) k +1 < δ 2 h Rewriting X k +2 λ,δ

we have

The only difference between the case (i) is the last term. Therefore with same calculation, we get below inequality.

From case (i) and (ii), by defining

we can write a recursive inequality for all k as below

D.3.2 sup 0 ≤ i ≤ T 2 h glyph[epsilon1] i = 3 T λ ( O ( h λ ) + O ( h λ e 2 L λ,δ T ) + O ( h ) + O ( h 2 λ e 2 L λ,δ T ) + O ( e 2 δL λ,δ L λ,δ ) + O ( δ ) )

We will now sum up above inequality. Multiplying ( k +1) ( 1 -h λ 1+ h λ ) k +1 both sides we have

By summing up above inequality from 0 to k -1 , we have

Note glyph[epsilon1] 0 vanished as it is multiplied with 0 . Reorganizing with respect to glyph[epsilon1] k we have

For second inequality follows from the fact 0 < h λ < 1 2 , which implies 0 < ( 1 -h λ ) i +1 ( 1+ h λ ) i ≤ 1 and 1 < 1+ h λ 1 -h λ . Observe f ( x ) = ( 1+ x 1 -x ) 1 x is nondecreasing in x ∈ (0 , 1) since

Therefore, from f ( 1 2 ) = 9 we have

Now we show

for 0 < T < ∞ , 0 ≤ k < T 2 h . Name N k = min { k, ⌊ δ 2 h ⌋} . From the definition of e i we have

where inequality follows from the fact i +1 k ≤ 1 for 0 ≤ i ≤ k -1 and 1 i +1 ≥ 2 h δ for 0 ≤ i ≤ N k -1 . Now let's observe each term. From Lemma B.12 and Lemma D.1 we know

Name M = max { √ 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ , 2 ‖ X 0 -X glyph[star] ‖ } .

Thus,

Therefore

(ii) h k ∥ ∥ ∥ GLYPH<129> λ ( X i +1 λ,δ ) ∥ ∥ ∥ 1 λ

Since GLYPH<129> λ is -Lipschitz continuous and from (28)

Therefore

(iii) 1 k ∑ N k -1 i =0 ∥ ∥ ∥ X i +1 λ,δ -X 0 ∥ ∥ ∥ for N k = min { k, ⌊ δ 2 h ⌋} . First, observe

From (27)

Now as i +1 ≤ N k = min { k, ⌊ δ 2 h ⌋} ,

From (i), (ii), (iii) we have

Therefore,

D.4 Derivation of ODE (4) from EAG and FEG

D.4.1 Derivation from EAG

For L -Lipschitz continuous monotone operator GLYPH<129> and stepsize h > 0 , EAG-C [75] is defined as

Dividing the second line by h and reorganizing we have

Identify hk = t , z k = X ( t ) , z 0 = X 0 . As z k is a converging sequence [76], it is bounded. Thus we see

Therefore taking limit h → 0+ we have

D.4.2 Derivation from FEG

For L -Lipschitz continuous monotone operator GLYPH<129> and stepsize h > 0 , FEG [42] is defined as

Dividing the second line by h and reorganizing we have

Identify hk = t , z k = X ( t ) , z 0 = X 0 . As z k is a converging sequence [76], it is bounded. Thus we see

Therefore taking limit h → 0+ we have

E Proof of convergence analysis for monotone GLYPH<129>

E.1 Extending ˜ GLYPH<129> to [0 , ∞ )

Lemma E.1. Suppose GLYPH<129> is a maximal monotone operator. Let X be the solution for (3) . Define S as

Then as X is the solution for (3) , we know [0 , ∞ ) \ S is of measure zero.

Define ˜ GLYPH<129> ( X ): S → R n as

and denote ˜ GLYPH<129> ( X ( t )) = ˜ GLYPH<129> ( X )( t ) . Assume for every T > 0 , there is M > 0 such that for all t ∈ S ∩ [0 , T ]

Then ˜ GLYPH<129> ( X ) can be extended to t ∈ [0 , ∞ ) with satisfying following properties.

(i) ˜ GLYPH<129> ( X ( t )) ∈ GLYPH<129> ( X ( t )) for all t ∈ [0 , ∞ ) .

(ii) For t ∈ [0 , ∞ ) \ S , there is a sequence { t k } k ∈ N such that t k ∈ S , lim k →∞ t k = t and ˜ GLYPH<129> ( X ( t k )) converges to ˜ GLYPH<129> ( X ( t )) .

Proof. Take t ∈ [0 , ∞ ) , and take a sequence { t n } n ∈ N such that t n ∈ S and lim n →∞ t n = t . As a converging sequence t n is bounded, there is T > 0 such that t n ∈ [0 , T ] . For that T , we have ∥ ∥ ˜ GLYPH<129> ( X ( t n )) ∥ ∥ ≤ M by the assumption. As n ↦→ ˜ GLYPH<129> ( X ( t n )) is a bounded sequence, there is a subsequence { t n k } k ∈ N such that k ↦→ ˜ GLYPH<129> ( X ( t n k )) converges. Name the limit as u = lim k →∞ ˜ GLYPH<129> ( X ( t n k )) . On the other hand, as X is a continuous curve we have lim k →∞ X ( t n k ) = X ( t ) . Then since GLYPH<129> is maximally monotone, from [15, Proposition 20.38] we have ( X ( t ) , u ) ∈ Gra GLYPH<129> . Defining ˜ GLYPH<129> ( X ( t )) as u , we get the desired result.

Corollary E.2. Let X be the solution for (6) . Then ˜ GLYPH<129> ( X ) defined as (29) has extension with properties stated in Lemma E.1.

Proof. First consider the case β ( t ) = γ t p , p > 0 , γ > 0 . Take T > 0 . It is enough to show there is M > 0 such that for t ∈ S ∩ [0 , T ]

Recall from Proposition B.11 for Yosida approximation GLYPH<129> λ , we denoted X λ as the solution of

and we have shown there is a sequence { λ n } n ∈ N such that X λ n uniformly converges to X on [0 , T ] .

glyph[negationslash]

thus

Therefore for t ∈ S ∩ [0 , T ] , ˙ X ( t ) = lim h → 0 X ( t + h ) -X ( t ) h holds, we conclude

And also from Lemma B.12, for t ∈ [0 , T ] we have

therefore

Gathering the result, for t ∈ S ∩ [0 , T ] we have

E.2 Proof of Proposition 3.2

We prove this theorem by deriving the energy with dilated coordinate W ( t ) = C ( t )( X ( t ) -X 0 ) and conservation law from [67]. Think of dilated coordinate W ( t ) = C ( t )( X ( t ) -X 0 ) . As we're considering the case ˜ GLYPH<129> is Lipschitz continuous, the differential inclusion (3) becomes ODE

Rewriting the ODE in terms of W , we have

where X ( W ( t ) , t ) = X ( t ) = W ( t ) C ( t ) + X 0 . Organizing, we have

From Lemma B.4 we know ˜ GLYPH<129> ( X ( W,t )) is differentiable almost everywhere, by differentiating we obtain second order ODE which holds almost everywhere

Now by taking inner product with ˙ W and integrating, we get equality which was refered as conservation law in [67]

Since ˙ W ( t ) = C ( t ) ( ˙ X ( t ) + β ( t )( X ( t ) -X 0 ) ) , we can rewrite the integrand in the last term as

Note the purpose was to obtain the first term, which is nonnegative due to monotonicity of ˜ GLYPH<129> . Now taking integration by parts to the second term we have

On the second equality, we used the fact ˙ W ( t ) = -C ( t ) ˜ GLYPH<129> ( X ( W ( t ) , t )) . Note the fundamental theorem of calculus for 〈 ˜ GLYPH<129> ( X ( W,s )) , C ( s ) β ( s ) W ( s ) 〉 is valid since ˜ GLYPH<129> ( X ( W,s )) is Lipschitz continuous and C ( s ) β ( s ) W ( s ) is continuously differentiable in [ t 0 , t ] , so their inner product is absolutely continuous in [ t 0 , t ] .

Observe the integrand in the last term can be rewritten as

Now gathering the results, we conclude

Moving the constant terms to left hand side and naming E = E 1 -constant, we get the desired result.

E.3 Proof of Corollary 3.3

E.3.1 V is nonincreasing when ˜ GLYPH<129> is Lipshitz continuous monotone

We first check it is true for the case ˜ GLYPH<129> is Lipschitz continuous. Then from Proposition 3.2 we can write V ( t ) as

with E in Proposition 3.2. As ˜ GLYPH<129> is monotone, from (1) we know C ( s ) 2 〈 d ds ˜ GLYPH<129> ( X ( s )) , ˙ X ( s ) 〉 ≥ 0 holds almost everywhere. Therefore for h > 0

we see V is a nonincreasing function. Therefore for all t > 0 , V ( t ) ≤ lim glyph[epsilon1] → 0+ V ( glyph[epsilon1] ) . It remains to show the limit lim glyph[epsilon1] → 0+ V ( glyph[epsilon1] ) exists.

E.3.2 Calculation of V (0) = lim t → 0+ V ( t ) for Lipshitz continuous monotone ˜ GLYPH<129>

In this section, we calculate lim t → 0+ V ( t ) when ˜ GLYPH<129> is Lipshitz continuous. Recall V was defined as

From now we denote V for the case t 0 = 0 as V 0 . We first check

glyph[negationslash]

we see C ( t ) defined above agrees with the definition of C ( t ) for each case. Note

To do so, we first show for t > 0

Observing

Now we will show

Then for t > 0 , f ∈ L 1 ([0 , t ] , R ) .

We first provide an elementary fact as a lemma.

Lemma E.3. Let f : (0 , ∞ ) → R is a continuous function. Suppose there is q < 1 , 0 < l < ∞ such that

Proof. Since lim sup s → 0+ f ( s ) s q = l , there is glyph[epsilon1] ∈ (0 , t ) such that

Then since q < 1

By the way as f ( t ) is continuous on [ glyph[epsilon1], t ] , M = max s ∈ [ glyph[epsilon1],t ] | f ( s ) | exists. Therefore,

Applying Lemma E.3, we will show for t > 0

Observe

(i) p > 1

We first show lim s → 0+ C ( s ) 2 1 s n = 0 for all n > 0 . Take n > 0 . Then there is some k ∈ N such that k ( p -1) > n . With change of variable u = 1 s and L'Hóptial's rule we see

And thus

By Lemma E.3, we conclude ∫ t 0 ∣ ∣ ∣ s 2 d ds ( C ( s ) 2 ˙ β ( s ) 2 )∣ ∣ ∣ ds < ∞ .

(ii) p = 1 Since C ( s ) = s γ , we see

Since 1 -γ < 1 , by Lemma E.3 we conclude ∫ t 0 ∣ ∣ ∣ s 2 d ds ( C ( s ) 2 ˙ β ( s ) 2 )∣ ∣ ∣ ds < ∞ .

0

< p <

1

Since lim s → 0+ C ( s ) = lim s → 0+ e γ 1 -p s 1 -p = 1 , we see

Since p < 1 , by Lemma E.3 we conclude ∫ t 0 ∣ ∣ ∣ s 2 d ds ( C ( s ) 2 ˙ β ( s ) 2 )∣ ∣ ∣ ds < ∞ .

(iii)

Naming the bound in Corollary B.8 as M ( T ) , we know for 0 < s < T

Therefore applying (33), for 0 < t < T we have

Hence we know V 0 ( t ) is well defined and since lim t → 0+ ∫ t 0 d ds ( C ( s ) 2 ˙ β ( s ) 2 ) ‖ X ( s ) -X 0 ‖ 2 ds = 0 , we have

Now we are ready to show the desired result.

(i) p > 1

As we know X ( t ) -X 0 and ˜ GLYPH<129> ( X ( t )) are bounded from Lemma D.1 and Corollary B.9. Therefore from (35) we have

therefore lim t → 0+ V 0 ( t ) = 0 .

Therefore

From (i) and (ii) we get the desired conclusion

and therefore for general t 0 ≥ 0 ,

is well-defined.

E.3.3 V ( t ) ≤ lim n →∞ V λ n (0) holds for t ∈ S and general maximal monotone GLYPH<129>

Define S as defined in Lemma E.1. Take t ∈ S , let T > t . Let { λ n } n ∈ N be a positive sequence λ n that lim n →∞ λ n = 0 , X λ n converges to X uniformly on [0 , T ] and ˙ X λ n converges weakly to ˙ X in L 2 ([0 , T ] , R n ) . Recall existence of such sequence was gauranteed by Proposition B.11.

Recall we denoted X λ as the solution of the ODE (19). Denote V λ as V for the case GLYPH<129> = GLYPH<129> λ , i.e.

Note equality ˙ X ( t ) = -˜ GLYPH<129> ( X ( t )) -β ( t )( X ( t ) -X 0 ) holds since t ∈ S , therefore we have

The goal of this section is to show

V ( t ) ≤ lim sup n →∞ V λ n ( t ) First observe, from Lemma B.12 we know

holds for s ≤ T . Thus from (33) we have

Therefore applying dominated convergence theorem, we have for t 0 ∈ [0 , T ]

From elementary analysis, we can easily check lim sup n →∞ ( a n + b n ) = lim sup n →∞ a n +lim n →∞ b n holds when lim n →∞ b n exists. Thus we have

Therefore it is suffices to show following lemma.

Lemma E.4. Suppose for T > 0 and sequence { λ n } n ∈ N , X λ n converges to X uniformly on [0 , T ] and ˙ X λ n converges weakly to ˙ X in L 2 ([0 , T ] , R n ) . Let t ∈ S ∩ [0 , T ] . Then following inequality is true

Proof. (i) ∥ ∥ ∥ ˙ X ( s ) ∥ ∥ ∥ ≤ lim sup n →∞ ∥ ∥ ∥ ˙ X λ n ( s ) ∥ ∥ ∥ holds for almost every s ∈ [0 , T ] .

Let D be a measurable subset D ⊂ [0 , T ] . Since ˙ X λ n ⇀ ˙ X in L 2 ([0 , T ] , R n ) and χ D ˙ X ∈ L 2 ([0 , T ] , R n ) , we have

The inequality comes from the Cauchy-Schwarz inequality. Now from Reverse Fatou's Lemma we have

(ii)

Therefore combining two inequalities we have

As D was arbitrary measurable subset of [0 , T ] , we conclude for almost every s ∈ [0 , T ]

∥

∥

∥

X

(

t

)

∥

∥

∥

n

→∞

∥

∥

∥

X

λ

n

(

t

)

∥

∥

∥

for

t

∈

S

∩

[0

, T

]

.

Let t ∈ S ∩ [0 , T ] . Then for h > 0 such that t + h < T , since S is measure zero, from (i) we have

˙

˙

Now, for some a > 0 consider

Then from the proof Lemma B.5, we know

holds for almost every s , thus U λ n is nonincreasing. By the way, as X converges to X uniformly on [0 , T ] , using dominated convergence theorem we have

λ n

Denote f ( s ) = lim n →∞ f n ( s ) . Using above facts, for s ∈ [ t, t + h ] we have

Therefore,

‖ X ( t + h ) -X ( t ) ‖ ≤ ∫ t + h t ∥ ∥ ∥ ˙ X ( s ) ∥ ∥ ∥ ds ≤ ∫ t + h t lim sup n →∞ ∥ ∥ ∥ ˙ X λ n ( s ) ∥ ∥ ∥ ds ≤ ∫ t + h t √ lim sup n →∞ U λ n ( t ) -f ( s ) ds.

Note √ lim sup n →∞ U λ n ( t ) -f ( s ) is a continuous function with respect to s . Thus we have

Finally as t ∈ S , ˙ X ( t ) = lim h → 0 X ( t + h ) -X ( t ) h exists. Therefore

we conclude the desired result.

lim sup n →∞ V λ n ( t ) ≤ lim sup n →∞ V λ n (0)

As GLYPH<129> λ is Lipschitz continuous, from Appendix E.3.1 we know V λ is nonincreasing, and thus

n n .

Taking limsup both sides we get the desired result.

lim sup n →∞ V λ n (0) = lim n →∞ V λ n (0) From (32) and (ii) of Lemma B.10, we know

And applying (36) we have

As the limit lim n →∞ V λ n (0) exists, the limsup concides with the limit.

E.3.4 Proof for general maximal monotone GLYPH<129>

First, we show V ( t ) ≤ lim n →∞ V λ n (0) holds for t ∈ [0 , ∞ ) . Next, we show lim n →∞ V λ n (0) = lim t → 0+ V ( t ) . Then we have V ( t ) ≤ lim n →∞ V λ n (0) = lim t → 0+ V ( t ) = V (0) , which is our desired result.

Take t ∈ [0 , ∞ ) . We know the inequality is true when t ∈ S , thus assume t / ∈ S . Then from Lemma E.1, we know there is a sequence { t k } k ∈ N such that t k ∈ S , lim k →∞ t k = t and ˜ GLYPH<129> ( X ( t k )) converges to ˜ GLYPH<129> ( X ( t )) . As t k ∈ S , V ( t k ) ≤ lim n →∞ V λ n (0) holds. Since lim k →∞ V ( t k ) = V ( t ) , taking limit k → ∞ to the inequality we get the

(i) V ( t ) ≤ lim n →∞ V λ n (0) holds for t ∈ [0 , ∞ ) desired result.

Take a sequence { t k } k ∈ N such that t k > 0 and lim k →∞ t k = 0 . We wish to show lim k →∞ V ( t k ) = lim n →∞ V λ n (0) . Note the arguments in Appendix E.3.2 are valid when ∥ ∥ ˜ GLYPH<129> ( X ( t k )) ∥ ∥ and ‖ X ( t k ) -X 0 ‖ t k are bounded for all k ∈ N . The boundedness of ∥ ˜ GLYPH<129> ( X ( t k )) ∥ comes from Corollary E.2. And from Lemma B.12 we have

(ii) lim n →∞ V λ n (0) = lim t → 0+ V ( t ) ∥ ∥

Therefore applying the arguments in Appendix E.3.2 we have

Thus it remains to show the limit on the right hand side exists and is equal to lim n →∞ V 0 λ n (0) .

For p ≥ 1 , we know lim k →∞ C ( t k ) 2 = 0 , thus lim k →∞ C ( t k ) 2 2 ∥ ∥ ˜ GLYPH<129> ( X ( t k )) ∥ ∥ 2 = 0 since ∥ ∥ ˜ GLYPH<129> ( X ( t k )) ∥ ∥ is bounded. As lim n →∞ V 0 λ n (0) = 0 from (32), we're done.

Now consider the case 0 < p < 1 . Suppose ˜ GLYPH<129> ( X ( t k l )) is a convergent subsequence of ˜ GLYPH<129> ( X ( t k )) . First observe from (i) we have

From above inequality, recalling lim l →∞ C ( t k l ) 2 = 1 we have

By the way as lim l →∞ X ( t k l ) = X 0 and ˜ GLYPH<129> ( X ( t k l )) ∈ GLYPH<129> ( X ( t k l )) , by closed graph theorem we have lim l →∞ ˜ GLYPH<129> ( X ( t k l )) ∈ GLYPH<129> ( X 0 ) . As of m ( GLYPH<129> ( X 0 )) is the element in GLYPH<129> ( X 0 ) with smallest norm, we have lim l →∞ ˜ GLYPH<129> ( X ( t k l )) = m ( GLYPH<129> ( X 0 )) . As all convergent subsequence converges to the same limit m ( GLYPH<129> ( X 0 )) , we conclude lim k →∞ ˜ GLYPH<129> ( X ( t k )) = m ( GLYPH<129> ( X 0 )) . Therefore

As t k was arbitrary positive sequence converges to 0 , we conclude lim n →∞ V 0 λ n (0) = lim t → 0+ V 0 ( t ) = V 0 (0) . Therefore

E.4 Proof of Lemma 3.4

Most of the proof is done in the main text. Recall Φ( t ) is defined as Φ( t ) = ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ 2 +2 β ( t ) 〈 ˜ GLYPH<129> ( X ( t )) , X ( t ) -X 0 〉 , and from (8) we have

On the other hand,

glyph[negationslash]

Note C ( t ) = 0 if t > 0 , we can divide with C ( t ) 2 . Now from Corollary 3.3 we have we have V ( t ) ≥ V (0) for t > 0 , and so V ( t ) C ( t ) 2 -V (0) C ( t ) 2 ≥ 0 . Thus for t > glyph[epsilon1]

Moving 2 β ( t ) 2 ‖ X 0 -X glyph[star] ‖ 2 to the left hand side and multiplying with 2 we get the desired result

E.5 Proof of Theorem 3.1

Restating Lemma 3.4, we know for t > 0 ,

As we know ‖ X ( t ) -X 0 ‖ ≤ 2 ‖ X 0 -X glyph[star] ‖ from Lemma D.1, it is clear that

Therefore it remains to show

We can check there is T > 0 such that for s > T the sign of d ds ( C ( s ) 2 ˙ β ( s ) ) does not change, for β ( t ) = γ t p with γ > 0 , p > 0 . We first proceed our proof with assuming this condition. The point for this condition is that following equality holds for t > T .

Applying above equality and using ‖ X ( t ) -X 0 ‖ ≤ 2 ‖ X 0 -X glyph[star] ‖ , we see

Therefore from (7) and (38), we conclude

It remains to show there is T > 0 such that for s > T the sign of d ds ( C ( s ) 2 ˙ β ( s ) ) does not change. It can be shown by easy, but a little complicated calculations. Since ˙ C ( t ) = C ( t ) β ( t ) , we see

Therefore it is enough to check the sign of 2 β ( s ) ˙ β ( s ) + ¨ β ( s ) . Observe

glyph[negationslash]

Thus when p = 1 , for all s > 0 it is nonpositive if γ ≥ 1 and nonnegative if 0 < γ < 1 .

glyph[negationslash]

When p = 1 , we see

Therefore by intermediate value theorem there is T > 0 such that T p -1 -2 γ p +1 = 0 , and for that T we have s p -1 -2 γ p +1 is nonpositive for s > T if 0 < p < 1 and nonnegative for s > T if p > 1 . This concludes the proof.

E.5.1 Proof for the convergence rate in Table 1

Recall, from (31) we have t t

glyph[negationslash]

Now the results in Table 1 is straightfoward.

p = 1

When p = 1 , we have C ( t ) = t γ . Comparing 1 C ( t ) 2 = 1 t 2 γ and β ( t ) 2 = γ 2 t 2 ,

0 < p <

When 0 < p < 1

1 , we have

glyph[negationslash]

glyph[negationslash]

We now observe ˙ β ( t ) does not effect to convergence rate. In other words, we show ˙ β ( t ) is not the slowest one that goes to zero compared to β ( t ) 2 and 1 C ( t ) 2 for every case.

(i) 0 < p ≤

1 Comparing ˙ β ( t ) = -pγ p +1 with β ( t ) 2 = γ 2 2 p , we see ˙ β ( t ) = O ( β ( t ) 2 ) when 0 < p ≤ 1 .

(ii) p > 1

From (i) and (ii), we conclude

p > 1

We observed previously that lim t →∞ 1 C ( t ) 2 = 1 = 0 when p > 1 . Therefore ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ 2 = O ( 1 C ( t ) 2 ) = O (1) .

E.6 Proof of Theorem 3.5

Name

We first show

Proof by contradiction. Suppose not.

Then there is glyph[epsilon1] > 0 such that for every k ∈ N , there is n k ∈ [0 , ∞ ) such that

By the way, from Lemma D.1 we know X ( n k ) ∈ ¯ B ‖ X 0 -¯ X glyph[star] ‖ ( ¯ X glyph[star] ) . Thus by Bolzano-Weierstrass theorem, there is a converging subseqeunce { X ( n k ( l ) ) } l ∈ ( N ) . Name the limit as X ∞ . Since lim t →∞ ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ = 0 , { ˜ GLYPH<129> ( X ( n k ( l ) ) } l ∈ ( N ) converges to 0 . Then from [15, Proposition 20.38], ( X ∞ , 0) ∈ Gra GLYPH<129> , i.e. X ∞ ∈ Zer GLYPH<129> .

On the other hand, as GLYPH<129> is maximal monotone, by [15, Proposition 23.39] Zer GLYPH<129> is closed and convex. Since ¯ X glyph[star] = Π Zer GLYPH<129> ( X 0 ) , by [15, Theorem 3.16] we have 〈 X ∞ -¯ X glyph[star] , X 0 -¯ X glyph[star] 〉 ≤ 0 . Thus

this is a contradiction, therefore we get the desired result.

Now we show lim t →∞ ∥ ∥ X ( t ) -¯ X glyph[star] ∥ ∥ = 0 . Recall, for t > 0

From ˙ C ( t ) = C ( t ) β ( t ) , and monotonicity of GLYPH<129> , we observe

Now take glyph[epsilon1] . From lim sup t →∞ 〈 X ( t ) -¯ X glyph[star] , X 0 -¯ X glyph[star] 〉 ≤ 0 , there is M > 0 such that

For that M and t > M , integrating from M to t we get

By dividing C ( t ) 2 and organizing,

By the way from assumption, we have lim t →∞ 1 C ( t ) = 0 . Therefore we conclude

Since glyph[epsilon1] > 0 was arbitrary, we get the desired result.

F Proof for worst case examples

The explicit solution for linear A is crucially used in worst case examples. Therefore, we first provide proof for it.

F.1 Proof for explicit solution for linear A

F.1.1 Proof of Lemma 4.1

Observing

we can check X (0) = X 0 . Now by using the property of Gamma function Γ( x +1) = x Γ( x ) , and paying attention to the lower bound of the summation index we have

The solution can be written in another form

As this is a special case of (9), here we just briefly check this satisfies the ODE and check other details in the proof of Lemma 4.2. By product rule of differentiation,

F.1.2 Proof of Lemma 4.2

Define

We show X ∗ is a well-defined solution for (3) with linear GLYPH<129> , then show X ∗ is equal to X ( t ) defined in (9).

We first check well-definedness. By definition, C ( t ) = e ∫ t t 0 β ( s ) ds is nondecreasing. Also, ∥ ∥ e tA ∥ ∥ is also nondecreasing since from monotonicity of A we have for all x ∈ R n

Now we see X ( t ) is well-defined since

Also above inequality implies second term reaches to zero as t → 0+ , we have lim t → 0+ X ∗ ( t ) = X 0 . Thus defining X ∗ (0) = X 0 if necessary, we see X ∗ satisfies the initial condition with X ∗ ∈ C ([0 , ∞ ) , R n ) .

We then check X ∗ ( t ) becomes the solution. This is immediate from product rule for differentiation. Since d dt ( 1 C ( t ) ) = -1 C ( t ) 2 C ( t ) β ( t ) = -1 C ( t ) β ( t ) , we have

Finally we now show X ( t ) = X ∗ ( t ) , where X ( t ) is defined as (9). From integral by parts we have

F.2 Proof of Theorem 4.3

If β ( t ) ≡ 0 , for A := ( 0 1 -1 0 ) , we have X ( t ) = e -tA , so glyph[negationslash]

So let's consider the case β is not β ( t ) ≡ 0 and β ( t ) ≥ 0 . For

name the solution for ODE ˙ X ξ = -A ξ ( X ξ ) -β ( t )( X ξ -X 0 ) as X ξ . By Lemma 4.2 we have

glyph[negationslash]

From lim t →∞ 1 C ( t ) = 0 , we see lim t →∞ ∥ ∥ ∥ e -tA ξ C ( t ) ∥ ∥ ∥ = 1 · 1 C ( t ) = 0 . And since A ξ is invertible except the case ξ = 0 , we have

Now let assume ∀ ξ ∈ R , lim t →∞ ‖ A ξ ( X ξ ( t )) ‖ = 0 and lead to contradiction. Define f : R → R as

Then ∫ ∞ -∞ f ( s ) ds = [ C ( s )] ∞ 0 = lim t →∞ C ( t ) -C (0) < ∞ we have f ∈ L 1 ( R , R ) . Note lim t →∞ C ( t ) exists, since C is nondecreasing and by the assumption is bounded.

Now setting X 0 = (1 , 0) T and from e sA ξ = ( cos 2 πξs sin 2 πξs -sin 2 πξs cos 2 πξs ) , we see

where ˆ f ( ξ ) is Fourier transform of f .

From assumption we have ∀ ξ ∈ R , ∥ ∥ ∫ ∞ 0 e sA ξ C ( s ) β ( s ) dsX 0 + C (0) X 0 ∥ ∥ = 0 , we have

We want to show, glyph[negationslash]

By the way, from Fourier theory we know ˆ f vanishes at infinity , thus

is not constantly zero. Thus a contradiction, we get the

Without loss of generality, we consider the case X 0 = (1 , 0) T .

(i) γ = 1 Plugging γ = 1 gives

Therefore

(ii) γ > 1

We will show lim t →∞ ‖ tA ( X ( t )) ‖ = γ . With change of variable s = tv and integration by parts we have

By the way, from γ > 1 we have v γ -2 ∈ L 1 [0 , 1] , by Riemann-Lebesgue lemma we have

Observe as e -tA is a rotation, we know

Taking limit t →∞ we know right hand side converges to zero, we conclude

Therefore we have ˆ f ( ξ ) = 0 for all ξ ∈ R .

Now ˆ f ≡ 0 is clearly L 1 ( R , R ) , we can apply Fourier inversion formula and conclude f ≡ 0 almost everywhere. However, f ( s ) = C ( s ) β ( s ) is not zero almost everywhere since β ( s ) desired result.

F.3 Proof of Theorem 4.4

F.3.1 Proof for the case p = 1 , γ ≥ 1

Recall from (39), we know

Therefore,

glyph[negationslash]

F.3.2 Proof for the case p = 1 , γ < 1

Since A , e tA are invertible, we see

glyph[negationslash]

Therefore it is enough to observe 1 γ e tA t γ X ( t ) . Again for X 0 = (1 , 0) T

Since s γ -1 decrease monotonically to zero, we may apply similar argument with alternative series test. Define a n = ∫ nπ ( n -1) π sin( s ) s γ -1 ds and name S n = ∑ n k =1 a n . Then for m ∈ N , t > 2 mπ we see

By the way S = lim n →∞ S n exists by alternative series test, thus we have

by squeeze theorem. Note S ≥ S 2 > 0 . With similar argument, we can conclude lim t →∞ ∫ t 0 cos( s ) s γ -1 ds also exists. Therefore we conclude lim t →∞ ‖ t γ X ( t ) ‖ glyph[negationslash] = 0 since

F.3.3 Proof for the case 0 < p < 1

Recall from (9), we have

Our goal is to show

We first observe, it is enough to show glyph[negationslash]

This follows from below facts.

Since β ( t ) = γ t p and A is linear and invertible,

glyph[negationslash]

glyph[negationslash]

Recall from (31), we have C ( t ) = e γ 1 -p t 1 -p for β ( t ) = γ t p , so C (0) = lim t → 0+ C ( t ) = 1 and lim t →∞ 1 β ( t ) C ( t ) = 0 . Thus lim t →∞ 1 β ( t ) C ( t ) ‖ C (0) I ‖ = 0 , second term is ignorable. Therefore the problem reduces to

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Since e tA is linear and invertible, problem reduces to

glyph[negationslash]

Again without loss of generality, let X 0 = (1 , 0) T . Recalling e -sA = ( cos s sin s -sin s cos s ) , focusing on one component we see it is sufficient to show

glyph[negationslash]

Now we prove the statement. For convenience, name D ( t ) = β ( t ) C ( t ) . We want to show glyph[negationslash]

To do so, we first show lim t →∞ D ( t + δ ) D ( t + π ) = 1 for δ ∈ [0 , π ] . Take δ ∈ [0 , π ] . Observe

Considering L'ôspital's rule for the exponent, we see

As the exponent reaches to zero, we conclude lim t →∞ D ( t + δ ) D ( t + π ) = 1 . Now from

we see D ( t ) is nondecreasing for t ≥ ( γ p ) 1 p -1 since 0 < p < 1 . Therefore for t ≥ ( γ p ) 1 p -1 we have sup δ ∈ [0 ,δ ] ∣ ∣ ∣ D ( t + δ ) D ( t + π ) -1 ∣ ∣ ∣ = max {∣ ∣ ∣ D ( t ) D ( t + π ) -1 ∣ ∣ ∣ , ∣ ∣ ∣ D ( t + δ ) D ( t + π ) -1 ∣ ∣ ∣ } , so it reaches to zero as t →∞ .

Now we prove desired statement. Proof by contradiction. Suppose lim t →∞ S ( t ) = 0 . Then for 0 < glyph[epsilon1] < 1 2 , there is T 1 > 0 such that t > T 1 implies | S ( t ) | < glyph[epsilon1] . Now for t > T 1 , observe

On the other hand, there is T 2 such that t > T 2 implies sup δ ∈ [0 ,π ] ∣ ∣ ∣ D ( t + δ ) D ( t + π ) -1 ∣ ∣ ∣ < glyph[epsilon1] 2 . Now for t = 2 nπ > max { T 1 , T 2 } ,

glyph[negationslash]

This contradicts the fact glyph[epsilon1] < 1 2 , we prove the assumption lim t →∞ ∫ t 0 S ( t ) = 0 is not true. Therefore lim t →∞ ∫ t 0 S ( t ) = 0 .

We first observe

G Proof of convergence analysis for discrete counterpart

G.1 Correspondence between discrete method in Theorem 5.1 and continuous model (6)

To check the correspondence of the method with (6), we provide a informal derivation. Assume operator GLYPH<129> is continuous. Then we have y k = x k +1 + GLYPH<129> x k +1 , by substituting y k and y k -1 , the method can be equivalently written as

This can be considered as a special case of below method, with h = 1 .

Dividing by h both sides and reorganizing, we have

Identifying hk = t , x k = X ( t ) and taking h → 0 , we have

As monotonicity is preserved for scalar multiple, rescaling 2 GLYPH<129> to GLYPH<129> does not change the class of operators that the ODE covers. For notation simplicity, replacing 2 GLYPH<129> to GLYPH<129> we have

G.2 Proof of boundedness of ∥ ∥ x k ∥ ∥

While proving Theorem 5.1, we need a upper bound for ∥ ∥ x k ∥ ∥ . Therefore we first prove following lemma.

Lemma G.1. Let GLYPH<129> be a maximal monotone operator. Consider a method

for k = 1 , 2 , . . . , with sequence { β k } k ∈ N , 0 ≤ β k ≤ 1 and initial condition y 0 = x 0 ∈ R n . Then following holds

Proof. Recall from Lemma D.2, we know for GLYPH<148> = GLYPH<146> GLYPH<129> = 2 GLYPH<138> GLYPH<129> -GLYPH<137> given method is equivalent to below method

Note that x glyph[star] ∈ Zer GLYPH<129> ⇔ 0 ∈ GLYPH<129> x glyph[star] ⇔ x glyph[star] = GLYPH<138> GLYPH<129> x glyph[star] ⇔ x glyph[star] = GLYPH<148> x glyph[star] ⇔ x glyph[star] ∈ Fix GLYPH<148> . We first prove ∥ ∥ y k -x glyph[star] ∥ ∥ ≤ ∥ ∥ y 0 -x glyph[star] ∥ ∥ by induction. The statement is trivially true when k = 0 . Now suppose the statement is true for k ∈ N . Then

The first inequality is just triangular inequality, the second inequality comes from the fact x glyph[star] ∈ Fix GLYPH<148> and GLYPH<148> is nonexpansive, and the last inequality is from induction hypothesis. By induction, we have ∥ ∥ y k -x glyph[star] ∥ ∥ ≤ ∥ ∥ y 0 -x glyph[star] ∥ ∥ for k = 0 , 1 , . . . .

Define ˜ GLYPH<129> x k = y k -1 -x k , then ˜ GLYPH<129> x k ∈ GLYPH<129> x k since GLYPH<138> GLYPH<129> y k -1 = x k . Observe for k = 1 , 2 , . . . , from monotone inequality we have

Therefore we conclude

G.3 Proof of Theorem 5.1

The outline of the proofs originate from continuous proofs. To simplify calculations, instead of directly deriving discrete counterpart of Proposition 3.2, we consider rescaled conservation law for each cases. By considering dilated coordinate W 1 = X -X 0 for p > 1 , W 2 = t p ( X -X 0 ) for 0 < p < 1 , W 3 = t ( X -X 0 ) for p = 1 , γ ≥ 1 and W 4 = t γ ( X -X 0 ) for p = 1 , 0 < γ < 1 , we obtain below conservation laws.

Lyapunov style proof can be obtained by considering below functions

The main blocks of the calculations corresponds to continuous cases, but there are some 'errors' in terms of ˜ GLYPH<129> x k and x k -x 0 occur due to discretization. The proofs are done by showing these 'errors' don't effect to the conclusions.

(0) Preparation.

For all cases, we will consider functions of the form

The latter statement holds directly from triangular inequality,

and consider U k +1 -U k . As similar calculations will be repeated, we first organize the repeating calculations. That is, we prove following equality is true.

Observe, this can be shown by checking below equalities are true.

-Proof for (43)

Recall, the method was defined as

By considering x k +1 + ˜ GLYPH<129> x k +1 = y k , substituting y k and y k -1 we can rewrite the method as

By multiplying k p + γ to both sides and reorganizing we have,

By subtracting both sides by k p ( x k +1 -x 0 ) and ( k p + γ )( x k -x 0 ) respectively, above equation can be rewritten as

From above, we get the following

Now dividing both sides by k p ( k p + γ ) , we get the desired result.

-Proof for (44)

This can be checked by just expanding the inner product of left hand side.

-Proof for (45)

First, observe

Reorganizing (40), we get two different expressions for x k +1 -x k .

Now plugging these to previous equality, be get the desired result.

(i) ∥ ∥ ˜ GLYPH<129> ( x k ) ∥ ∥ 2 = O (1) for p > 0 , γ > 0 . Plugging

to (41) and (42), we obtain

The continuous counterpart of above equality is

which can be obtained by differentiating and reorganizing the conservation law for E 1 . Note, as k 2 p 2 ( 1 k p -1 ( k +1) p ) = O ( k 2 p ) O ( 1 k p +1 ) = O ( 1 k 1 -p ) , the order of the term 1 2 k p + γ ( γ + k 2 p 2 ( 1 k p -1 ( k +1) p )) corresponds to ( γ t p + p t ) . Thus we can see the sum of first two terms on the right hand side of the equality for discrete setting corresponds to the first term of the right hand side of the equality for continuous setting.

We can observe that s k, 1 , s k, 0 = O ( 1 k 2 p +1 ) + O ( 1 k p +2 ) . And since c k = O ( 1 k 2 p ) + O ( 1 k p +1 ) , we have c k +1 -c k = O ( 1 k 2 p +1 ) + O ( 1 k p +2 ) as well. Therefore the coefficients of ∥ ∥ x k +1 -x 0 ∥ ∥ 2 and ∥ ∥ x k -x 0 ∥ ∥ 2 are summable for p > 0 . From Lemma G.1, we know ∥ ∥ x k +1 -x 0 ∥ ∥ 2 and ∥ ∥ x k -x 0 ∥ ∥ 2 are bounded by 4 ∥ ∥ x 0 -x glyph[star] ∥ ∥ 2 .

The term 〈 ˜ GLYPH<129> x k +1 -˜ GLYPH<129> x k , x k +1 -x k 〉 on the left hand side is nonnegative from monotonicity of ˜ GLYPH<129> , and the coefficient is nonnegative as well. The coefficient of ∥ ∥ ˜ GLYPH<129> x k +1 ∥ ∥ 2 in the right hand side, -1 2 ( γ k p -γ ( k +1) p ) , is nonpositive. As a result, we get below inequality.

As done in (8), by monotonicity of GLYPH<129> and Young's inequality

Reorganizing, we get the desired result

(ii) ∥ ∥ ˜ GLYPH<129> ( x k ) ∥ ∥ 2 = O ( 1 k 2 p ) for 0 < p < 1 , γ > 0 . Plugging

to (41) and (42) with p = 1 , we obtain

The continuous counterpart of this equality is

which can be obtained by differentiating the conservation law for E 2 . The first term on the right hand side correspond to the sum of first two terms in the right hand side of discrete equality. Thus we may expect the order of the coefficients for the terms would match, and we will check the expectation is indeed true. With some calculation, we can observe

where

By considering Newton expansion and from 0 < p < 1 , we see

Thus we can check the leading order of numerator is p +2 p +(2 p -2) = 5 p -2 , and leading order of the denominator is p + p +2 p = 4 p . As 5 p -2 -4 p = p -2 , we have s k ∈ O ( k p -2 ) , which matches with the continuous counterpart. Therefore ∑ ∞ k =1 s k < ∞ .

On the other hand, we see

Since p > 2 p -1 , we have lim k →∞ q k = -∞ . Note this matches with the continuous counterpart as well.

Therefore there is N > 0 such that for k > N , q k < 0 . Now for k > N we have

Thus for k > N , by monotonicity of GLYPH<129> and Young's inequality

Reorganizing, we get the desired result.

to (41) and (42), we obtain

The continuous counterpart of above equality is

which can be obtained by differentiating the conservation law for E 3 . Note the terms match with same order of coefficients, except two ∥ ∥ x k +1 -x 0 ∥ ∥ 2 and ∥ ∥ x k -x 0 ∥ ∥ 2 , while these terms are summable as s k, 1 , s k, 0 = O ( 1 k 2 ) . Therefore we have

Thus by monotonicity of GLYPH<129> and Young's inequality

Reorganizing, we get the desired result.

The continuous counterpart of this equality is

which can be obtained by differentiating the conservation law for E 4 . Thus we may expect the order of the matching terms are equal, and the terms do not occur in the continuous version do not bother our desired conclusion. We check our expectation is true.

Terms ∥ ∥ x k +1 -x 0 ∥ ∥ 2 and ∥ ∥ x k -x 0 ∥ ∥ 2 correspond to ‖ X -X 0 ‖ 2 . The coefficients for ∥ ∥ x k +1 -x 0 ∥ ∥ 2 and ∥ ∥ x k -x 0 ∥ ∥ 2 are clearly O ( k 2 γ -3 ) , which equals the order of continuous counterpart. Since γ < 1 , we know these terms are summable.

Next we observe q k ≤ 0 . Observe

To check the last inequality is true, consider f ( x ) = x γ . Since this function is concave for 0 < γ < 1 , we see

Finally we focus on s k, 0 and s k, 1 . As cross terms don't appear in continuous version, we may expect these terms are 'small', or in mathematical words, summable. From Cauchy-Schwarz inequality we know

Since ∥ ∥ ˜ GLYPH<129> x k +1 ∥ ∥ and ∥ ∥ ˜ GLYPH<129> x k ∥ ∥ are bounded from the proof for case (i), we know two innerproduct terms are bounded. Thus if we show ∑ ∞ k =1 | s k, 0 | , ∑ ∞ k =1 | s k, 1 | < ∞ , we can conclude U k is bounded. Considering Newton expansion, we see

Therefore

With similar argument

Therefore, we have

By Young's inequality

Since k 1 -γ ( k -1 4 ) 2 γ -2 = O ( k γ -1 ) and γ < 1 , this terms goes to zero as k →∞ thus there is some M 2 > 0 such that 1 2 ( γk 1 -γ ( k -1 4 ) 2 γ -2 ) 2 ∥ ∥ x k -x glyph[star] ∥ ∥ 2 ≤ M 2 for all k ≥ 1 . Reorganizng terms, we obtain the desired result

H Proof of convergence analysis for strongly monotone GLYPH<129>

H.1 Proof of Proposition 6.1

We take dilated coordinate W ( t ) = C ( t )( X ( t ) -X 0 ) as did in the proof of Proposition 3.2. Recall from (30), the second order version of the ODE was written as 0 = ¨ W -β ( t ) ˙ W + C ( t ) d dt ˜ GLYPH<129> ( X ( t )) . Now for we multiply R ( t ) 2 in the ODE and obtain

Now taking inner product with ˙ W and integrating we have

Note the second term which is obtained from integration by parts, would not appear if R ( t ) = 1 as in Proposition 3.2. This is key term to exploit the condition GLYPH<129> is strongly monotone.

Taking integration by parts to the second term we have

The fact C ( t ) ˜ GLYPH<129> ( X ( W,t )) = -˙ W ( t ) is applied to the second equality. Note the fundamental theorem of calculus is valid since C ( s ) R ( s ) 2 β ( s ) W ( s ) is differentiable, ˜ GLYPH<129> ( X ( W ( s ) , t )) is Lipschitz continuous in [ t 0 , t ] by Lemma B.4, and so their inner product is absolutely continuous in [ t 0 , t ] .

Now consider the second integrand except the term marked with *. From integration by parts we have

The integrand except * marked term can be rewritten as

Now collecting the terms marked with *, we have

Collecting all results we have

constant

Renaming E = E 1 -constant, we get the desired result.

H.2 Proof of Theorem 6.2

H.2.1 Proof of the inequality V ( t ) ≤ V (0)

The basic structure of the proof is same as Appendix E.3. We do not repeat the whole proof here, instead we check the steps done in Appendix E.3 are also valid for the setup in Theorem 6.2.

(i) V is nonincreasing for Lipschitz continuous µ -strongly monotone ˜ GLYPH<129> . Recall V in Theorem 6.2 was defined as

We first check following equality is true.

2 Recall we're considering (10), β ( t ) = 2 µ e 2 µt -1 . As d dt log ( 1 -e -2 µt ) = 2 µe -2 µt 1 -e -2 µt = 2 µ e 2 µt -1 = β ( t ) , we have C ( t ) = e ∫ t ∞ 2 µ e 2 µs -1 ds = e log ( 1 -e -2 µt ) = 1 -e -2 µt .

This proves the desired equality. Now we show

where E is from Proposition 6.1 which was defined as

From (48) and the definition of E , it is enough to show d ds ( C ( s ) 2 R ( s ) 2 ˙ β ( s ) 2 ) = 0 . Since

Now since ˜ GLYPH<129> is Lipschitz continuous, we know E is constant from Proposition 6.1. Therefore from (49), for t > 0 , | h | < t we have

As ˙ R ( s ) R ( s ) = µ and ˜ GLYPH<129> is µ -strongly monotone, from (2) we see

Therefore V ( t + h ) -V ( t ) ≥ 0 for h > 0 , we get the desired result.

˜

(ii) Calculation of V (0) for Lipshitz continuous monotone GLYPH<129> Plugging t = 0 to (47) we immediately obtain V (0) = 0 .

(iii) V ( t ) ≤ 0 holds for all t ∈ [0 , ∞ ) and general maximal µ -strongly monotone GLYPH<129> We check the arguments in Appendix E.3.3 and Appendix E.3.4 are also valid here.

Define S as defined in Lemma E.1. Take t ∈ S , let T > t . As checked in Appendix C, the arguments used in the proof Proposition B.11 is also valid for the case β ( t ) = 2 µ e 2 µt -1 . This fact provides the required sequence { X λ n } n ∈ N , where X λ n converges to X uniformly on [0 , T ] , and ˙ X λ n converges weakly to ˙ X in L 2 ([0 , T ] , R n ) . As in Appendix E.3.3, denote V λ as V for the solution with GLYPH<129> λ . Then from (i) we know V λ n is nonincreasing for all n ∈ N , we have lim sup n →∞ V λ n ( t ) ≤ lim sup n →∞ V λ n (0) = 0 .

Moreover, we can check the extension of ˜ GLYPH<129> defined in Lemma E.1 is also valid. From (21) and (22), we know ∥ ∥ ∥ ˙ X λ ( t ) ∥ ∥ ∥ ≤ e µT √ 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ and ‖ GLYPH<129> λ ( X λ ( t )) ‖ ≤ √ e 2 µT +1 ‖ m ( GLYPH<129> ( X 0 )) ‖ for all λ > 0 , t ∈ [0 , T ] . Therefore we can prove Corollary E.2 for the case β ( t ) = 2 µ e 2 µt -1 with the same proof, replacing M GLYPH<129> ( T ) by √ e 2 µT +1 ‖ m ( GLYPH<129> ( X 0 )) ‖ and M dot ( T ) by e µT √ 2 ‖ m ( GLYPH<129> ( X 0 )) ‖ . Thus ˜ GLYPH<129> ( X ( t )) is well-defined for t = 0 , plugging t = 0 to (47) we obtain V (0) = 0 .

Therefore we have

it remains to show V ( t ) ≤ lim sup n →∞ V λ n ( t ) . Observe, as equality ˙ X ( t ) = -˜ GLYPH<129> ( X ( t )) -β ( t )( X ( t ) -X 0 ) holds since t ∈ S , from (48) we have

Therefore if is suffices to check Lemma E.4 is valid here with some U λ n . For some a > 0 , Define U λ n : [0 , ∞ ) → R as

We proceed similar argument with Lemma B.5. Differentiating ˙ X λ n ( t ) = -GLYPH<129> λ n ( X ( t )) -β ( t )( X λ n ( t ) -X 0 ) , we have for almost all t > 0

Therefore for almost all t > 0 ,

˙ U λ n ( t )

Therefore U λ n is nonincreasing, we can prove ∥ ∥ ∥ ˙ X ( t ) ∥ ∥ ∥ ≤ lim sup n →∞ ∥ ∥ ∥ ˙ X λ n ( t ) ∥ ∥ ∥ 2 with same argument in Lemma E.4. Therefore we have V ( t ) ≤ V (0) = 0 for t ∈ S . Extending the result to t ∈ [0 , ∞ ) can be done with the same argument done in Appendix E.3.4.

H.2.2 Proof for convergence rate

Recall

Observe

From the law of cosines, we have ‖ X ( t ) -X 0 ‖ 2 = ‖ X ( t ) -X glyph[star] ‖ 2 -2 〈 X ( t ) -X 0 , X 0 -X glyph[star] 〉 - ‖ X glyph[star] -X 0 ‖ 2 . Applying this to p ( t ) we have

Thus

Now since ˜ GLYPH<129> is µ -strongly monotone, 〈 ˜ GLYPH<129> ( X ( t )) , X ( t ) -X glyph[star] 〉 -µ ‖ X ( t ) -X glyph[star] ‖ 2 ≥ 0 . From previous section we know 0 = V (0) ≥ V ( t ) for all t > 0 . Therefore for all t > 0

Organizing, we conclude

H.2.3 Informal derivation of ODE from the method

Assume GLYPH<129> : R n → R n be a continuous monotone operator. In [54], the method OS-PPM is presented as

where y 0 = x 0 , ν = 1 + 2 hµ and s k = 1 + ν 2 + · · · + ν 2 k = ν 2 k +2 -1 ν 2 -1 . Using y k -1 = x k + h GLYPH<129> x k , substituting y k and y k -1 this method can be expressed in a single line,

Reorganizing and dividing both sides by h , we have

Identifying x 0 = X 0 , 2 hk = t , x k = X ( t ) ,

Now observe

Taking limit h → 0 + and organizing,

By diving both sides by 2 , we get the desired anchor ODE

I Proof omitted in Section 7

I.1 Proof for Theorem 7.1

We first check where

By definition of β ( t ) , Φ( t ) defined in (8) becomes zero, i.e.

Plugging Φ( t ) = 0 to inequality (8) we have

Reorganizing, we get the desired result. Other results need some works, we provide the proof with steps into subsections.

Taking inner product with ˜ GLYPH<129> ( X ) to the ODE and applying 1 2 Φ( t ) = 0 we have

As GLYPH<129> is assumed to be continuous, taking limit t → 0+ we have

glyph[negationslash]

On the other hand, by assumption we have lim t → 0+ ˙ X ( t ) = lim t → 0+ X ( t ) -X 0 t and ∥ ∥ ˜ GLYPH<129> ( X 0 ) ∥ ∥ = 0 , thus

glyph[negationslash]

Therefore there is glyph[epsilon1] > 0 such that 〈 ˜ GLYPH<129> ( X ( t )) , X ( t ) -X 0 〉 < 0 for t ∈ (0 , glyph[epsilon1] ) , thus for t ∈ (0 , glyph[epsilon1] ) we have 〈 ˜ GLYPH<129> ( X ( t )) , X ( t ) -X 0 〉 = 0 so β ( t ) is well-defined and satisfies

Observe the denominator of β ( t ) is zero when ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ = 0 . As β is assumed to be well-defined, we have ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ = 0 for all t > 0 . Since β is continuous as GLYPH<129> and X are continuous, by intermediate value theorem we have ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ > 0 for all t > 0 .

I.1.2 Proof for the main statements

We first show following lemma.

Lemma I.1. Following equality holds for almost every t > 0 .

glyph[negationslash]

Proof. Since ˜ GLYPH<129> is Lipschitz continuous by assumption, by Lemma B.4 we know ˜ GLYPH<129> ( X ( t )) is differentiable almost everywhere. Differentiating Φ( t ) we have

for t ∈ (0 , ∞ ) almost everywhere. The equalities come from the ODE ˙ X ( t ) = -˜ GLYPH<129> ( X ( t )) -β ( t )( X ( t ) -X 0 ) and the fact Φ( t ) = 0 . Reorganizing, we have the desired equation (51).

Now we show the upper bounds of β ( t ) for each monotone and strongly monotone case. Observe from (51) and the definition of β ( t ) , we have for almost all t ∈ (0 , ∞ )

(i) When ˜ GLYPH<129> is monotone.

From (52) and (1) we have for almost all t ∈ (0 , ∞ )

almost everywhere. Now dividing both sides by β ( t ) 2 we have

holds almost everywhere. Since -˙ β ( t ) β ( t ) 2 = d dt ( 1 β ( t ) ) , integrating above inequality both side from δ to t we have

thus lim δ → 0+ β ( δ ) = ∞ , so lim δ → 0+ 1 β ( δ ) = 0 . Therefore taking limit δ → 0+ we have

Since β ( t ) > 0 we have

By the way, from (50) we have

(ii) When ˜ GLYPH<129> is µ

-strongly monotone.

From (52) and (2) for almost all t ∈ (0 , ∞ ) we have

On the other hand, observe

From Cauchy-Schwarz inequality we see

therefore ∥ ∥ ˜ GLYPH<129> ( X ( t )) ∥ ∥ ≤ 2 β ( t ) ‖ X ( t ) -X 0 ‖ . Combining above observations, we have for almost all t > 0

From (53) and (54), we get an inequality for β ( t ) and ˙ β ( t )

Moving β ( t ) 2 to the right-hand side and reorganizing, we have for almost all t > 0

Integrating above inequality both sides from δ to t we have

As observed in case (i) we know lim δ → 0+ β ( δ ) = ∞ , taking limit δ → 0+ both sides we have

Organizing, we get the desired result.

I.2 Correspondence between the ODE (11) and the discrete method in Theorem 7.2

Observe, the case h = 1 for below method corresponds to the method provided in Theorem 7.2.

glyph[negationslash]

I.3 Proof of Theorem 7.2

Recall, the method was defined as glyph[negationslash]

We now show when GLYPH<129> is continuous and ‖ ˜ GLYPH<129> x k ‖ 2 = 0 for all k ≥ 0 , we obtain the ODE (11) when we take limit h → 0+ . Note when GLYPH<129> is continuous GLYPH<129> equals to ˜ GLYPH<129> .

From h ˜ GLYPH<129> x k +1 + x k +1 = y k , substituting y k and y k -1 we get a single line expression

Reorganizing and dividing both sides by h , we have

Identify x 0 = X 0 , 2 hk = t , and x k = X ( t ) . Then we see

Thus β k = O ( h ) , we have lim h → 0+ β k = 0 . Now taking limit h → 0+ we have

Dividing both sides by 2, we obtain (11).

I.2.1 Correspondence between convergence rates in Theorem 7.1 and Theorem 7.2

From identification above, we see β ( t ) corresponds to β k 2 h . Therefore we see with identification x 0 = X 0 , x glyph[star] = X glyph[star] , 2 hk = t , and x k = X ( t ) we have

And we can also check that the bound β k ≤ 1 k +1 for the monotone case, is equivalent to β k 2 h ≤ 1 2 hk + O ( h ) and corresponds to β ( t ) ≤ 1 t as well.

Now suppose GLYPH<129> is µ -strongly monotone and L -Lipschitz continuous. Then h GLYPH<129> is hµ -strongly monotone and hL -Lipschitz continuous, we have the following inequality from Theorem 7.2

Recalling the identification 2 hk = t , we see

Therefore identifying β ( t ) = β k 2 h and taking limit h → 0+ to the inequality (55), we have

glyph[negationslash]

for k = 1 , 2 , . . . , and for all k ≥ 1 .

Proof. Proof by induction. From x 1 = GLYPH<138> GLYPH<129> y 0 = GLYPH<138> GLYPH<129> x 0 , we have x 1 + ˜ GLYPH<129> x 1 = x 0 and so x 1 -x 0 = -˜ GLYPH<129> x 1 . Applying these facts we have

glyph[negationslash]

First, we assume ˜ GLYPH<129> x k = 0 for all k ≥ 0 . Define

for k = 2 , 3 , . . . . Note by definition of β k , we have ˜ Φ k = 0 . Our goal is to prove

Then with the same argument with (8) we can conclude ‖ ˜ GLYPH<129> x k +1 ‖ 2 ≤ β 2 k ‖ x 0 -x glyph[star] ‖ 2 . To do so, we first show following lemma.

Lemma I.2. For k ≥ 1 , following is true.

Proof. Recall y k = x k +1 + ˜ GLYPH<129> x k +1 . Substituting y k and y k -1 , the method is equivalent to

From above we can get two different expression of (1 -β k )( x k +1 -x k ) .

With reorganizing, first equality can be obtained by subtracting both sides by β k ( x k +1 -x 0 ) and the second equality can be obtained by multiplying both sides by (1 -β k ) .

From this we have

Since GLYPH<129> is monotone we have 〈 ˜ GLYPH<129> x k +1 -˜ GLYPH<129> x k , x k +1 -x k 〉 ≥ 0 , we see that the right-hand side of (56) is greater or equal to 0 if 1 -β k ≥ 0 . Thus it remains to show 1 -β k ≥ 0 . As the index is quite confusing, we provide it as a lemma to avoid confusion while proceeding the proof.

glyph[negationslash]

Lemma I.3. If ˜ GLYPH<129> x k = 0 for all k ≥ 1 , then

As 1 -β 1 > 0 , from (56) we have

As ˜ Φ 1 = 0 by definition, we have Φ 2 ≤ 0 . Therefore the statement is true for k = 1 .

Now suppose the statements are true for k . By induction hypothesis, we know

glyph[negationslash]

As β k > 0 from induction hypothesis and ˜ GLYPH<129> x k +1 = 0 by assumption, reorganizing Φ k ≤ 0 we have

Since 1 -β k +1 > 0 , by (56) we have

Since ˜ Φ k +1 = 0 by definition, we have Φ k +2 ≤ 0 . Therefore the statements are true for k +1 , by induction, we get the desired result.

glyph[negationslash]

Suppose ‖ ˜ GLYPH<129> x k ‖ 2 = 0 for all k ≥ 1 . From the lemma we know Φ k +1 ≤ 0 for k ≥ 1 , so with the same argument of (8) we have for x glyph[star] ∈ Zer GLYPH<129>

Organizing, we get

And therefore

Now we show the upper bound of β k . Observe from (56) and the fact ˜ Φ = 0 , we have

glyph[negationslash]

As 1 -β k = 0 for k ≥ 1 by Lemma I.3, dividing both sides by 1 -β k , we get the discrete counterpart of (52),

(i) When GLYPH<129> is monotone.

From (57) and monotonicity of GLYPH<129> we have

From Lemma I.3 we have 1 -β k > 0 , therefore

Summing up, as β 1 = 1 2 and 1 β k > 0 we have

(ii) When GLYPH<129> is µ

-strongly monotone.

Since GLYPH<129> is µ -strongly monotone, from (57) we have

Define r k = ‖ x k +1 -x k ‖ 2 ‖ ˜ GLYPH<129> x k +1 ‖ 2 for k = 0 , 1 , . . . . Dividing both sides by ∥ ∥ ˜ GLYPH<129> x k +1 ∥ ∥ 2 and organizing, we have

Dividing both sides by β k and reorganizing, we get a recursive inequality for 1 β k -1

We now prove an upper bound of β k from above inequality.

Lemma I.4. Suppose GLYPH<129> be a µ -strongly monotone operator. Let β k be a sequence defined as Theorem 7.2 and let r k = ‖ x k +1 -x k ‖ 2 ‖ ˜ GLYPH<129> x k +1 ‖ 2 for k = 0 , 1 , . . . . Then following holds for k = 1 , 2 , . . . .

Proof. First observe, the statement is equivalent to

The proof can be done by induction with (58).

When k = 1 , recalling β 1 = 1 2 from the proof of Lemma I.3, we can check the inequality is true. Now suppose the inequality is true for k = m . Then from (58) we have

Therefore, we get the desired result.

Under the identification considered in Appendix I.2, the continuous counterpart of r k is

and is greater or equal to 1 by (54). We obtained an exponential convergence rate in continuous setup from this fact. In the same spirit, we can get an exponential convergence rate for discrete setup if there is a positive lower bound for r k , we provide it as a corollary of Lemma I.4.

Corollary I.5. Consider the setup of Lemma I.4. Suppose there is r ≥ 0 such that r k ≥ r for k = 0 , 1 , . . . . Then following is true for k = 1 , 2 , . . . .

Proof. From r k ≥ r we have

Applying Lemma I.4, we get the desired result.

We now show r k ≥ 1 1+ L 2 holds when GLYPH<129> is furthermore L -Lipschitz continuous.

(iii) When GLYPH<129> is µ -strongly monotone and L -Lipschitz continuous. Recall from the proof of Lemma I.2, we know

Taking inner product with ˜ GLYPH<129> x k both sides we have

From above equality we can check

As GLYPH<129> is L -Lipschitz continuous, we have

Dividing both sides by ( 1 + L 2 ) ∥ ∥ GLYPH<129> x k +1 ∥ ∥ 2 we get a lowerbound for r k

Applying Corollary I.5 we get the desired result

Note if we take limit µ → 0 + , with substitution α = µ 1+ L 2 we have

which is the bound for the monotone case.

I.3.1 If there is k ≥ 0 such that ˜ GLYPH<129> x k = 0

Suppose ˜ GLYPH<129> x k = 0 for some k . Let x N be the very first iterate such that ˜ GLYPH<129> x N = 0 . Then from previous argument we know Theorem 7.2 is true for k < N . Thus it remains to show the statements are true for k ≥ N .

From ˜ GLYPH<129> x N = 0 we know y N -1 = x N + ˜ GLYPH<129> x N = x N . And since ˜ GLYPH<129> x N = 0 implies β N = 0 , from the definition of the method we have y N = 2 x N -y N -1 = x N . Therefore

we conclude x k = x N ∈ Zer GLYPH<129> for all k ≥ N . Thus β k = 0 , ∥ ∥ ˜ GLYPH<129> x k ∥ ∥ = 0 for all k ≥ N , the Theorem 7.2 is trivially true for k ≥ N .

J Details of experiment in Section 7.1

We solve a compressed sensing problem of Shi et al. [63] which is formulated as an glyph[lscript] 1 -regularized least-squared problem

We solve this problem in decentralized manner due to the problem setup where the network of local agents are as Figure 2 and each agents communicate only with their neighbors, the nodes connected to each agents by edge. We use Metropolis-Hastings matrix as our mixing matrix W ∈ R n × n and apply PG-EXTRA. Let W i,j denote ( i, j ) -th entry of W and N i ⊆ { 1 , 2 , . . . , n } denote the index of the agents in the neighborhood of agent i . Consider

to the problem above. Under suitable choice of parameters, PG-EXTRA can be seen as a fixed-point iteration of an averaged operator with respect to ‖·‖ M [74, Theorem 2], where the metric matrix M is defined as

and U is a symmetric definite matrix with U 2 = 1 2 ( I -W ) . That is, denoting x k , w k ∈ R d × n as the vertical stack of x k i 's and w k i 's respectively [58, Chapter 11.3], PG-EXTRA can be rewritten as ( x k +1 , w k +1 ) = GLYPH<148> ( x k , w k ) . Using this GLYPH<148> , we proceed the experiment with the Halpern method

When GLYPH<148> is an averaged operator, it is nonexpansive, we know GLYPH<148> = GLYPH<146> GLYPH<129> = 2 GLYPH<138> GLYPH<129> -GLYPH<137> for some maximal monotone operator GLYPH<129> [54, Lemma 2.1]. Considering the equivalence discussed in Lemma D.2, we see above Halpern method is equivalent to our presented algorithms of the form

by corresponding y k = ( x k +1 , w k +1 ) . Note the operator norm ∥ ∥ ˜ GLYPH<129> x k ∥ ∥ 2 M can be calculated by considering below equation

We use the anchor coefficients β k = 1 k +1 , β k = γ k p + γ with p = 1 . 5 , γ = 2 . 0 and the adaptive choice of β k in Theorem 7.2 with M -norm. Note, in the experiment the adaptive coefficient is calculated by considering below equation

We choose the dimension of signal d = 100 , the number of agents n = 20 , the number of measurement for each agent m i = 4 , glyph[lscript] 1 -regularization parameter ρ = 0 . 01 , and algorithm parameter α = 0 . 01 .

K Broader Impacts

Our work focuses on the theoretical aspects of convex optimization algorithms. There are no negative social impacts that we anticipate from our theoretical results.

L Limitations

Our analysis concerns convex optimization. Although this assumption is standard in optimization theory, many functions that arise in machine learning practice are not convex.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the continuous time analysis anchor ODE in [55] to involved a more general choice of the coefficient $\\beta(t)$ and show that the choice in [55] is in some sense optimal. They then go on to show:\n- correspondance with the discrete anchoring schemes APPM/FEG/AEG\n- Anchor ODE converges to a minimal $\\ell_2$-distance from initialization (like APPM [51])\n- Tightness of continuous time rates for anchor ODE\n- Convergence of APPM for more general choice of stepsize than $1/(k+2)$.\n- faster convergence under $\\mu$-strongly monotonicity for anchor ODE by choosing $\\beta(t)$ dependent on $\\mu$.\n- convergence with adaptive stepsize for both anchor ODE and APPM (single-valued) that recovers the rates for monotone and strongly monotone+Lipschitz\n\nSTRENGTHS:\nOverall an impressive work. It seem interesting that one can adapt to both the smoothness parameter and the strong monotonicity modulus. The work seems fairly completely, I only leave a few comments/remarks below.\n\nWEAKNESSES:\n- My main concern is why we consider $\\beta(t)$ other than $1/t$ in the first place. The proof seems much more involved, but what does it buy us considering $1/t$ is already in some sense optimal. My understanding is that if only $1/t$ was considered then [55] and [34]/[Theorem 18](https://large-scale-book.mathopt.com/LSCOMO.pdf) already proofs convergence for anchor ODE and APPM respectively. At least clarifying this would be helpful.\n- The work contextualizes the results but usually quite late. Overall contextualizing w.r.t. existing work upfront and motivating the sections. E.g. elaborate on [55] in l. 20 and explicitly say \"adapt to the strong monotone modulus\" in l. 221. Explicitly compare theorem 5.1 with existing results for APPM.\n\nComments:\n\n- In terms of limitations maybe mention that the convergence results for the discretized schemes are only for implicit schemes (APPM)\n- I would mention existing discretization result up front (in [55] in contrast they only consider GD with anchoring and does not get $1/k^2$)\n\nQUESTIONS:\n- Can we obtain convergence for explicitly schemes such as AEG/FEG?\n- It seems very interesting that you can adapt both to L and $\\mu$. Would this work for explicit scheme?\n- Why distinguish between Halpern and APPM in Fig. 2 if they are equivalent?\n- What is $M$ in the $y$ axis of Figure 2?\n- FEG generalizes to cohypomonotone problems. Can anchor ODE handle cohypomonotonicity?\n\nLIMITATIONS:\n(see comments)", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a continuous-time analysis of an acceleration method called \"anchoring\", where the main contributions are four-fold. The authors provided a unified analysis of the convergence rate of anchor acceleration, which includes both the constant and adaptive cases. Then the authors presented an adaptive method for anchor acceleration that is inspired by our analysis and achieves faster convergence rates than the constant method. After this, the authors proved that the adaptive method is robust to noise and can handle non-convex optimization problems. Finally, the authors provided numerical experiments that demonstrate the effectiveness of the adaptive method on various optimization problems. Overall, the paper provides a valuable contribution to the field of optimization and is in general well-written and well-presented.\n\nSTRENGTHS:\nThis paper provides a comprehensive analysis of the convergence rate of anchor acceleration, which includes both the constant and adaptive cases. The paper provides a clear and concise presentation of both the theoretical analysis, where I checked the technical proofs of all results with a detailed focus on Theorem 3.1 [Section E.5] and Theorem 6.2 [Section H.2] and they are solid. The idea of using the factor $\\frac{2 \\mu}{e^{2 \\mu t}-1}$ is is new and reasonable, since for \\mu-strongly convex objectives $\\mu\\to 0^+$ it is consistent with the standard $\\frac{1}{t}$ anchoring rate (the analysis provided is significantly more general, which should be honored). The adaptive method presented in the paper is also interesting, which appears faster convergence rates than the constant method and is robust to noise and non-convex optimization problems. The paper also clearly presented numerical experiments which demonstrate the effectiveness of the adaptive method on various optimization problems.\n\nWEAKNESSES:\nThe paper assumes a certain level of mathematical background, which might be difficult for some readers unfamiliar with literature to follow. Further, the paper seems not provide a comparison of the adaptive method with other state-of-the-art optimization methods. In addition, the paper does not provide a detailed discussion of the limitations of the adaptive method and areas for future research. I have not checked but the assumptions of the adaptive method presented in the paper might be more stringent than required.\n\nQUESTIONS:\nAre there some mismatches in the definition of Lyapunov functions at various places? For instance, the $V(t)$ definition in Line 133 [Corollary 3.3], when pinning $\\beta = \\frac{1}{t}$, and the last in Line 147 [last but one display of Section 3.1] differs by a factor of 2. These are minor, but I do encourage the authors to check them carefully.\n\nCan you provide examples of applications where anchor acceleration might be particularly useful? I understand that anchor acceleration has been discovered to be an acceleration mechanism for minimax optimization and fixed-point problems,  but would anchor acceleration be useful in other optimization problems as well?\n\nLIMITATIONS:\nThis is a theoretical paper and admits no negative social impacts, to my best knowledge.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the analysis of anchor acceleration, a recently discovered acceleration mechanism for minimax optimization and fixed-point problems. The authors provide tight and unified analyses to characterize the convergence rate of anchor acceleration and present an adaptive method inspired by continuous-time analyses. The contributions include a differential inclusion model of anchor acceleration, well-posedness analysis, convergence rate analysis with a power-law anchor coefficient, and an adaptive method for choosing the anchor coefficient.\n\nSTRENGTHS:\nGiven the limited understanding of the anchor acceleration mechanism compared to Nesterov acceleration, the authors aim to provide a formal and rigorous treatment of the anchored dynamics through continuous-time analyses. They also seek to gain insight into the anchor acceleration mechanism and its accelerated convergence rate.\n\nThe authors present a differential inclusion model of anchor acceleration and establish its well-posedness. They analyze the convergence rate of the anchor ODE with a power-law anchor coefficient. The trade-off between the vanishing speed and the contracting speed of the anchor term is discussed. The authors also provide a proof outline of Lemma 3.4 and Theorem 3.1 to derive the convergence result. Additionally, the study presents an adaptive method for choosing the anchor coefficient based on continuous-time analyses.\n\nWEAKNESSES:\n- Novelty of applying continuous analysis to anchor acceleration: It would be helpful for the authors to clarify the motivation behind applying continuous analysis techniques specifically to the problem of anchor acceleration. Given the existence of continuous analysis techniques, what is the significance of applying them to anchor acceleration? This clarification would strengthen the novelty of the paper.\n\n- Importance of anchor acceleration: The paper could benefit from providing a clear explanation of why anchor acceleration is important and how it differs from other acceleration mechanisms, such as Nesterov acceleration. This would help readers understand the relevance and practical implications of anchor acceleration.\n\n- Technical challenges of applying continuous analysis: Can you elaborate on the specific technical challenges encountered when applying continuous analysis to the anchor acceleration problem? Discuss any unique aspects or complexities involved in analyzing the convergence properties of anchor acceleration using continuous-time techniques?\n\nQUESTIONS:\n- Can you provide more details on the motivation behind using anchor acceleration and how it differs from other acceleration mechanisms, such as Nesterov's acceleration?\n\n- Can you provide more insights into the choice of the anchor coefficient $\\beta(t)$ and how it affects the convergence rate of the algorithm?\n\n- Can you provide more details on the assumptions made in the analysis and how they may affect the applicability of the proposed method to real-world problems?\n\nLIMITATIONS:\nThe authors does not seem to discuss the limitations adequately, they do mention that carrying out more advanced analyses for the anchor ODE are interesting directions of future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the dynamics of anchor acceleration using a differential inclusion. It derives a convergence rate that depends on the anchor coefficient and shows that the rate is tight using a certain instance. By discretizing the differential inclusions, the authors derive an algorithm that generalizes the state-of-the-art APPM algorithm and provide an analysis of its convergence rate. In addition, the authors propose an algorithm that adaptively varies the anchor coefficient and show its performance both theoretically and empirically.\n\nSTRENGTHS:\n- Compared to existing papers that provide analysis of anchor methods through continuous-time analysis, this paper provides better convergence rates in a more general setting.\n\n- The adaptive anchor acceleration method in Section 7 is interesting, and the method is inspired by the analysis of continuous-time dynamics. It is a meaningful example of the potential of ODE analysis for optimization algorithm design.\n\nWEAKNESSES:\n- In terms of algorithms and their theoretical analysis, the contribution of this paper does not appear to be significant. In my understanding, the main result of this paper is deriving a known convergence rate in a different way, and proposing a different algorithm that achieves the same rate as the known one. There is insufficient discussion of how the results of this paper are superior to existing studies.\n\nQUESTIONS:\n- Since A is treated as a set-valued operator in \"Monotone and set-valued operators\" of Section 1.1, I understood that Ax in the equations in Lines 39 and 41 is a set. Then those equations involve the inner product of a point and a set in Euclidean space. Such an inner product is not clear to me. Additionally, in Line 50, A is assumed to be a differentiable operator, but the definition of the differentiability of set-valued operators is nontrivial. So, I guessed that A here is a single-valued operator. Is it correct? If so, it would be good to clarify it.\n\n- Theorem 2.2 states the uniqueness of the solution of the differential inclusion (6) rather than a differential equation. Is this a valid claim? It seems for me that there is more than one solution depending on which of the elements in A(X(t)) in equation (6) is chosen at each time t. Adding a comment on this point would facilitate the reader's understanding.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}