Title: Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Variational Inequalities

Quasi-Newton Methods

Min-Max Problems

Lower Bounds

Optimal Methods
Abstract: Variational inequalities represent a broad class of problems, including minimization and min-max problems, commonly found in machine learning. Existing second-order and high-order methods for variational inequalities require precise computation of derivatives, often resulting in prohibitively high iteration costs. In this work, we study the impact of Jacobian inaccuracy on second-order methods. For the smooth and monotone case, we establish a lower bound with explicit dependence on the level of Jacobian inaccuracy and propose an optimal algorithm for this key setting. When derivatives are exact, our method converges at the same rate as exact optimal second-order methods. To reduce the cost of solving the auxiliary problem, which arises in all high-order methods with global convergence, we introduce several Quasi-Newton approximations. Our method with Quasi-Newton updates achieves a global sublinear convergence rate. We extend our approach with a tensor generalization for inexact high-order derivatives and support the theory with experiments.

Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations

Abstract

Variational inequalities represent a broad class of problems, including minimization and min-max problems, commonly found in machine learning. Existing second-order and high-order methods for variational inequalities require precise computation of derivatives, often resulting in prohibitively high iteration costs. In this work, we study the impact of Jacobian inaccuracy on second-order methods. For the smooth and monotone case, we establish a lower bound with explicit dependence on the level of Jacobian inaccuracy and propose an optimal algorithm for this key setting. When derivatives are exact, our method converges at the same rate as exact optimal second-order methods. To reduce the cost of solving the auxiliary problem, which arises in all high-order methods with global convergence, we introduce several Quasi-Newton approximations. Our method with Quasi-Newton updates achieves a global sublinear convergence rate. We extend our approach with a tensor generalization for inexact high-order derivatives and support the theory with experiments.

1 Introduction

In this paper, we primarily address the problem of solving the Minty Variational Inequality (MVI) [77, 17]. Given a continuous operator F : X → R d , where X ⊆ R d is a closed bounded convex subset with a diameter D = max x,y ∈X ∥ x -y ∥ , the objective is to find a point x ∗ ∈ X such that

The solution to (1) is referred to as a weak solution of the Variational Inequality (VI) [37]. In contrast, the Stampacchia variational inequality problem [48] consists in finding a point x ∗ ∈ X such that

This solution is often called a strong solution to the variational inequality. When the operator F is both continuous and monotone, the weak and strong solutions are equivalent [37].

The operator F ( x ) is called monotone, if

Another useful assumption is L 1 -smoothness.

First-order methods. Variational inequalities encompass a wide range of problems, including minimization [19, 90, 93, 15], min-max problems, Nash equilibrium, differential equations, and others [37, 16]. The extensive research on VI methods dates back several decades, with a notable breakthrough in the 1970s-the development of the Extragradient method [62, 6]. Subsequently, it was demonstrated that this method achieves global convergence of O ( ε -1 ) [80, 43], matching the convergence rates of other first-order 2 methods such as optimistic gradient [92, 78, 63, 44], forward-backward splitting [95], and dual extrapolation [81]. These first-order methods collectively exhibit optimal convergence [89].

Second-order and high-order methods. To achieve further notable acceleration of methods for VIs, one can leverage information about higher-order derivatives. For instance, simply incorporating firstorder derivatives (Jacobian) can significantly enhance the convergence speed of the method. Following recent advancements in second-order and high-order methods with global rates for minimization [85, 82, 10, 79, 83, 46, 58, 14, 51, 64, 22], several high-order methods for VIs have been proposed [21, 68, 52, 88, 70, 84, 72, 5]. However, all these methods involve a line-search procedure, resulting in ˜ O ( ε -2 / 3 ) convergence for the case of first-order information (Jacobians). Recent works [69, 2] propose methods with improved rates O ( ε -2 / 3 ) and establish the lower bound Ω ( ε -2 / 3 ) , rendering these algorithms optimal.

Jacobian's approximation. In the last decade, VIs found new applications in machine learning. There are many problems that could not be reduced to minimization, including reinforcement learning [87, 55], adversarial training [75], GANs [42, 31, 41, 76, 28, 67, 91], classical learning tasks in supervised learning [56, 9], unsupervised learning [98, 8], image denoising [35, 27], robust optimization [13]. Applying second-order methods, without even mentioning high-order ones, described in a previous paragraph to machine learning problems could be a challenging task. Although these methods may theoretically converge faster, computing exact Jacobians and the per-iteration costs can be expensive. Therefore, it seems natural to introduce inexact approximations of the first-order derivatives. In the context of minimization, several works with inexact Hessians were introduced for both convex [40, 3, 7, 4] and nonconvex [24, 25, 23, 61, 99, 94, 74, 11, 12, 34] problems. Regarding VIs, Quasi-Newton (QN) methods [73, 14, 57] can be highlighted, though they, unfortunately, achieve only local convergence in the strongly monotone case [18, 38]. These methods are relatively less advanced for VIs compared to their counterparts in the field of minimization, where they are considered classics in optimization due to their effectiveness and practicality [86]. Modern research on QN approximations for minimization includes methods that exhibit global convergence [57, 54, 53]. A recent work [71] introduces the Newton-MinMax method for convexconcave unconstrained min-max optimization problems, demonstrating an optimal rate under special assumptions on the accuracy of the Jacobian approximation. However, the field of VIs lacks globally convergent inexact second-order methods with an explicit dependence on the accuracy of the Jacobian. This raises several natural questions:

What are the lower bounds for methods with inexact Jacobians? Can we construct an optimal method with inexact first-order information? What is the proper way to approximate the Jacobian to ensure global convergence and reduce the iteration complexity?

In our work, we attempt to answer these questions in a systematic manner.

Optimality measure. Most of our results are stated for the monotone setting (Assumption 1.1). In this context, the optimality of a point ˆ x ∈ X is typically measured by a gap function GAP ( · ) : X → R + [95, 80, 81, 78, 69], defined by

where ε ≥ 0 is the accuracy of solution. The boundedness of X and the existence of a strong solution ensure that the gap function is well-defined. If ε = 0 , we get by (1) that ˆ x is a weak solution of VI. We explore the performance of the proposed algorithm in scenarios involving nonmonotone operators F . In such cases, it is essential to assume that the operator satisfies the Minty condition to ensure that the problem is computationally manageable [32].

Assumption 1.3 The operator F ( x ) satisfies Minty condition, if there exists a point x ∗ such that

The range of applications of nonmonotone VIs satisfying Minty conditions is quite extensive [20, 29, 39, 36, 66, 60]. We note, that this condition is weaker than monotonicity [30, 50, 59] and guarantees the existence of at least one strong solution since F is continuous and X is closed and bounded [47]. To measure the optimality of point ˆ x we define the residue function RES ( · ) : X → R + [30, 50, 59]

The boundedness of X and the existence of a strong solution ensure that the residual function is well-defined. If ε = 0 , by (2), we get that ˆ x is a strong solution of VI.

Contributions. The main contribution of this paper lies in the development of a new second-order method robust to inexactness in the Jacobian, a common occurrence in machine learning. We demonstrate the algorithm's optimality in the monotone case by establishing a lower bound for this key setting. Expanding further:

We introduce a novel second-order algorithm, VIJI (Second-order Method for V ariational I nequalitues under J acoibian I nexactness), designed to handle δ -inexact 3 Jacobian information. Specifically, in the context of smooth and monotone VIs, VIJI achieves a convergence rate of O ( δD 2 T + L 1 D 3 T 3 / 2 ) to find weak solution. For smooth nonmonotone VIs satisfying the Minty condition, we demonstrate a convergence rate of O ( δD 2 √ T + L 1 D 3 T ) to identify strong solution. Notably, when δ ≤ L 1 D √ T , our method matches the convergence rates of optimal exact second-order

methods [2, 69].

We establish the optimal performance of our algorithm on monotone smooth operators by deriving a theoretical complexity lower bound of Ω ( δD 2 T + L 1 D 3 T 3 / 2 ) to find weak solution for the case of δ -inexact Jacobians.

Our algorithm involves solving a variational inequality subproblem. To tackle this challenge, we introduce an approximation condition, which makes the solution computationally feasible.

We introduce a new Quasi-Newton update for approximating the Jacobian, which significantly decreases the per-iteration cost of the algorithm while maintaining a global sublinear convergence rate. Numerical experiments demonstrate the practical benefits of our method.

We extend our algorithm for higher-order VIs with inexact high-order derivatives, resulting in O ( ∑ p -1 i =1 δ i D i +1 T ( i +1) / 2 + L p -1 D p +1 T ( p +1) / 2 ) rate for monotone and smooth VIs with δ i -inexact i -th derivative to find weak solution. Moreover, we extend our proposed high-order method to nonmonotone VIs.

We propose a restarted version of VIJI for strongly monotone VIs, which exhibits a linear rate.

Comparison with Lin, Mertikopoulos, and Jordan [71]. To the best of our knowledge, the work [71] is the most closely related to our research.The objective of [71] was to develop a method for convex-concave unconstrained min-max optimization under inexact Jacobian information with an optimal convergence rate O ( ε -2 / 3 ) matching the lower bound Ω( ε -2 / 3 ) [69]. The authors successfully achieve this goal by proposing a second-order algorithm Inexact-Newton-MinMax

method based on the Perseus [69]. To attain optimal convergence, they constrained the Jacobian inexactness with a function that decreases as the method converges and bounded the norm of Jacobian from above. While these assumptions might be suitable for randomized sampling in finite-sum and stochastic problems, they may not hold for many approximation strategies, such as Quasi-Newton algorithms. The aim of our work, however, is to study the impact of Jacobian inaccuracy on the convergence of second-order methods for VIs (a special case of which are min-max problems) and to identify the explicit dependence of the convergence rate on the inexactness. Compared to the work [71], the Jacobian inaccuracy directly affects the step sizes in our algorithm, allowing us to achieve a convergence rate of O ( ε -2 / 3 + δε -1 ) for any given δ . VIJI can be viewed as a generalization of Inexact-Newton-MinMax. With the same assumption on δ as in [71], our methods for min-max optimization are equivalent. The inexactness of the subproblem and the solution approach proposed in [71] remain valid for our method even with arbitrary large δ . Further details about application of our method to min-max problems can be found in Appendix J.

2 Preliminaries

Notation. Let R d be a finite-dimensional vector space with scalar product ⟨· , ·⟩ . For vector x ∈ R d we denote Euclidean norm as ∥ x ∥ . For X ∈ R d 1 × ... × d p , we define

and ∥ X ∥ op = max ∥ z i ∥ =1 , 1 ≤ j ≤ p X [ z 1 , · · · , z p ] . Fixing p ≥ 1 and letting F : R d → R d be a continuous and high-order differentiable operator, we define ∇ ( p ) F ( x ) as the p th -order derivative at a point x ∈ R d . To be more precise, letting z 1 , . . . , z k ∈ R d , we have

Taylor approximation and oracle feedback. The starting point for our method is the first-order Taylor polynomial of the operator F at point v : Φ v ( x ) = F ( v ) + ∇ F ( v )[ x -v ] . Since the computation of Jacobian ∇ F ( v ) could be a quite tiresome task, it seems natural to introduce an inexact approximation J ( v ) . Based on it we introduce inexact Taylor approximation - one of the main building blocks of our algorithm

where J ( x ) satisfies the following assumption.

Assumption 2.1 For given v ∈ X δ -inexact Jacobian satisfies:

As it was shown, e.g. in [52], Assumption 1.2 allows to control the quality of approximation of operator F by its Taylor polynomial

The next lemma is counterpart of (9) for the case of inexact Jacobian.

Lemma 2.2 Let Assumptions 1.2 and 2.1 hold. Then, for any x, v ∈ X

3 VIJI algorithm

In this section, extending on recent optimal high-order method for MVIs Perseus [69], we present our proposed method, dubbed as VIJI and detailed in Algorithm 1.

Algorithm 1 VIJI

.

The model of objective and subproblem's solution. We begin the description of the algorithm by introducing the inexact model of objective Ω η v ( x )

where η > 0 is given constant. Here, we introduced the additional regularization term ηδ ( x -v ) . As we will demonstrate, this term is crucial for ensuring that the method's subproblem has a solution. For brevity, we use a regularization constant of 5 L . Using larger coefficients would yield the same convergence rate. As other dual extrapolation-type methods, VIJI includes the following subproblem

First of all, the strong solution of this VI exists because Ω η v k +1 ( x ) is continuous, and X is a closed, bounded, and convex set. Next, we demonstrate that in a monotone setting VI (11) is also monotone.

Lemma 3.1 Let Assumptions (1.1) , (1.2) , (2.1) hold. Then for any x, v k +1 ∈ X VI (11) is monotone

Following the work [69], one can find a strong solution to such VI using mirror-prox methods from [1], achieving the following approximate condition

This ensures that the subproblem is computationally solvable in the monotone setting. In specific cases, such as minimax optimization, other efficient subsolvers can be employed [49, 2, 71].

Adaptive dual stepsizes. Adaptive stepsizes in dual space λ k are another core aspect of the algorithm. Due to inaccuracies in the Jacobian, applying the standard adaptive strategy, such as in the Perseus algorithm [69], can lead to excessively large steps, potentially slowing down the method. To address this issue, an additional term β k is incorporated into the adaptive strategy for selecting λ k . Thus, when ∥ x k -v k ∥ is small (indicating proximity to the optimum), β k has a greater influence on the choice of λ k , preventing the method from taking overly aggressive steps. Similar behavior can be observed in accelerated second-order methods for minimization with inexact Hessians from theoretical [3, Lemma 5], [4, Appendix B, Lemma E.3] and practical [4, Section 7] perspectives.

Convergence in monotone setting. Now, we are prepared to present the convergence theorem of Algorithm 1 in the monotone case.

Theorem 3.2 Let Assumptions 1.1, 1.2, 2.1 hold. Then, after T ≥ 1 iterations of Algorithm 1 with parameters β k = δ, η = 10 , opt = 0 , we get the following bound

The upper bound (13) consists of two terms. The first one corresponds to exact convergence and matches the lower bound for second-order VI methods [69]. The second term illustrates the impact

of the Jacobian's inexactness on the convergence rate, aligning with the lower bound for first-order methods [89]. We also notice that one can make the bound from (13) tighter for the so-called restricted gap-function [81] defined as RGAP ( y ) = sup x ∈C ⟨ F ( x ) , y -x ⟩ , where C ⊆ X and the solution set X ∗ of (2) satisfies X ∗ ⊆ C . In particular, following similar steps as in the proof of Theorem 3.2, one can derive O ( L 1 ̂ D 3 T 3 / 2 + δ ̂ D 2 T ) bound for RGAP (˜ x T ) , where ̂ D = sup x ∈C ∥ x -x 0 ∥ .

In the next theorem, we show that the method can achieve the optimal convergence rate of secondorder methods under additional assumption on δ .

Theorem 3.3 Let Assumptions 1.1, 1.2, 2.1 hold. Let { x k , v k } be iterates generated by Algorithm 1 and

Then, after T ≥ 1 iterations of Algorithm 1 with parameters β k = L 1 2 ∥ x k -v k ∥ , η = 10 , opt = 0 , we get the following bound

Note, that condition (14) is verifiable, indicating that the method can adjust to δ k in cases of controllable inexactness. Specifically, at each iteration, we can solve subproblem (12). If the assumption regarding δ k is not met, we can improve Jacobian approximation and repeat procedure. Moreover, similarly to the previous theorem, one can tighten the above bound for RGAP (˜ x T ) and get the dependence on ̂ D = sup x ∈C ∥ x -x 0 ∥ instead of D .

Convergence in nonmonotone setting. To begin with, in the nonmonotone case, the subproblem (11) may not exhibit monotonicity, and solving (12) becomes challenging [32]. Yet, in certain specific scenarios, such as unconstrained minimization tasks, it remains feasible to find a solution by leveraging the cubic structure of the subproblem [26]. The following theorem establishes the convergence of VIJI in the nonmonotone setting.

Theorem 3.4 Let Assumptions 1.2, 1.3, 2.1 hold. Then after T ≥ 1 iterations of Algorithm 1 with parameters β k = δ, η = 10 , opt = 2 we get the following bound

The convergence rate could be improved by additional assumption on inexact Jacobian.

Theorem 3.5 Let Assumptions 1.2, 1.3, 2.1 hold. Let { x k , v k } be iterates generated by Algorithm 1 that satisfy (14) . Then after T ≥ 1 iterations of Algorithm 1 with parameters β k = L 2 ∥ x k -v k ∥ , η = 10 , opt = 2 we get the following bound

4 The lower bound

In this section, we establish a theoretical lower bound for the complexity of first-order algorithms using inexact Jacobians for monotone MVIs. The proof technique draws inspiration from works [33, 4] and based on lower bounds from [69, 89].

We start by describing the available information and the method's structure. The considered class of algorithms relies on data provided by a first-order δ -inexact oracle, denoted as O : X → R d × R d × d . Given a point ¯ x ∈ X , the oracle returns

The method is able to generate points { x k } k ≥ 0 that satisfy the following condition

Next, we state the primary assumption concerning the method's ability to generate new points.

Assumption 4.1 The method generates a recursive sequence of iterates { x k } k ≥ 0 that satisfies the following condition: for all k ≥ 0 , we have that x k +1 ∈ X satisfies that ⟨ Ω ¯ x ( x k +1 -x k ) , x -x k +1 ⟩ ≥ 0 for all x ∈ X , where

As highlighted in [69], Assumption 4.1 is suitably satisfied by various dual extrapolation methods. However, it might not be applicable to alternative methods for variational inequalities, such as extragradient methods and their variants. We leave the generalization of inexact lower bounds for these algorithms to future research, as even lower bounds for exact algorithms [2, 69] do not address this case. Now, let us introduce the generalization of smoothness

Assumption 4.2 The operator F ( x ) is i -th-order L i -smooth ( i ≥ 0 ), if it has Lipschitz-continious i -th-order derivative

Finally, we present the lower bound theorem for first-order methods with inexact Jacobians.

Theorem 4.3 Let some first-order method M satisfy Assumption 4.1 and have access only δ -inexact first-order oracle 15. Assume the method M ensures for any L 0 -zero-order smooth and L 1 -first-order smooth monotone operator F the following convergence rate

5 Quasi-Newton Approximation

In this section, inspired by Quasi-Newton (QN) methods for Hessian approximation, we propose QN approximations for Jacobians. Our goal is to create a simple scheme to approximate the first-order derivative and thereby reduce the complexity of the subproblem. We compute J x using a QN update and use it as an inexact Jacobian in the model Ω η v ( x ) (10).

where r is a rank of approximation, J 0 ∈ R d × d ⪰ 0 , u i ∈ R d , v i ∈ R d are known. U ∈ R r × d and V ∈ R r × d are matrices of stacked vectors U = [ u 0 , . . . , u r -1 ] and V = [ v 0 , . . . , v r -1 ] and C ∈ R r × r is a diagonal matrix C = diag ([ c 0 , . . . , c r -1 ]) . If u i = v i the update becomes symmetric.

L-Broyden is a non-symmetric variant of QN approximation [45, 38] of the following form

In a view of (19), this update is obtained by setting u i = y i -J i s i , v i = s i , c i = 1 / ( s ⊤ i s i ) , and the rank r = m is equal to memory size m .

Damped L-Broyden is another option for QN approximation with non-symmetric damped update

By choosing u i = y i -J i s i , v i = s i , c i = 1 / (( m +1) s ⊤ i s i ) , r = m , we derive this update from (19). We define the matrix J r ( J 0 , U, V, C ) = J r ( J 0 , Y, S, C ) , where Y and S are formed by stacking the vectors [ y 0 , . . . , y m -1 ] and [ s 0 , . . . , s m -1 ] . The matrix J m ( J 0 , Y, S ) can be computed for any given pair ( Y , S ). Next, we describe two strategies for the choice of ( s, y ) pairs used in (20), (21).

QN with operator history is the well-known classic variant where operator differences are stored:

This approach is computationally efficient as it does not require additional operator calculations.

Then for all T ≥ 1 we have

QN with JVP sampling is based on fast computation of Jacobian-Vector Products (JVP):

where s i are random vectors uniformly distributed on the unit sphere such that ∥ s i ∥ = 1 and s 0 , . . . , s m -1 are linearly independent. Note, for m ≪ d , each s i is linearly independent with high probability. This approach requires only m operator/JVP computations per step, which is considerably fewer than the d JVPs needed for a full Jacobian. Utilizing the current Jacobian information allows to improve the accuracy of the approximation.

In the following theorem, we demonstrate that these approximations satisfy Assumption 2.1 and condition (8) for both the QN with operator history and JVP sampling methods.

Theorem 5.1 Let F ( x ) be L 0 -zero-order smooth operator. For m -memory L-Broyden approximation of the Jacobian J x = J m defined iteratively by (20) with 0 ⪯ J 0 ⪯ L 0 I , we have δ ≤ ( m +2) L 0 . For m -memory Damped L-Broyden approximation J x = J m of the Jacobian defined iteratively by (21) with 0 ⪯ J 0 ⪯ L 0 m +1 I , the condition δ ≤ 2 L 0 holds true.

With the primary toolkit for QN approximation in VIs established, we can now discuss efficient way of solving the subproblem (11), which takes the following form find y ∈ X such that ⟨ F ( x ) + ( J x + ηδI +5 L 1 ∥ y -x ∥ I )( y -x ) , z -y ⟩ ≥ 0 for all z ∈ X . (22) Let us introduce a parameter τ = ∥ y -x ∥ for a segment search in τ ∈ [0; D ] . To solve (22), we consider another problem

where A τ = J x +( ηδ +5 L 1 τ ) I . Problems (22) and (23) are equivalent when τ = ∥ y τ -x ∥ . The subproblem (23) can be reformulated as minimization problem

where (23) is an optimality condition for (24). The goal is to find y τ such that υ ( τ ) = | τ -∥ y τ -x ∥| ≤ ε . As υ ( τ ) is a continuous function of τ , we can find this solution via bisection segment-search with log 2 D ε iterations. This ray-search procedure is similar to the subproblem solution for the Cubic Regularized Newton subproblem.

For r -rank QN approximation J r from (19), we can effectively compute A -1 τ F ( x ) , where A τ = J r +( ηδ +5 L 1 τ ) I = U ⊤ CV + J 0 +( ηδ +5 L 1 τ ) I = U ⊤ CV + B and B = J 0 +( ηδ +5 L 1 τ ) I by using the Woodbury matrix identity[96, 97].

For computational efficiency, it is better to choose J 0 as a diagonal matrix, then inversion B -1 is computed by O ( d ) arithmetical operations, C -1 by O ( r ) operations. V B -1 U ⊤ requires O ( r 2 d ) for classical multiplication and can be improved by fast matrix multiplication. ( C -1 + V B -1 U ⊤ ) -1 can be computed by O ( r 3 ) , as an inverse of r -rank matrix. The rest of the operations are cheaper. Thus, the total number of arithmetic operations is O ( r 2 d ) instead of O ( d 3 ) for Jacobian inversion. We need to perform this inversion logarithmic number of times. Therefore, the total computational cost with the segment-search procedure is ˜ O ( r 2 d + r 3 log 2 ( D ε ) ) .

6 Strongly monotone setting

Assumption 6.1 The operator F : R d → R d is called strongly monotone if there exists a constant µ > 0 such that

To leverage the strong monotonicity of the objective function and achieve a linear convergence rate, we introduce the restarted version of Algorithm 1 dubbed as VIJI-Restarted. Restart techniques are widely utilized in optimization and typically preserve optimality. In other words, restarting an optimal method for convex functions or monotone problems effectively transforms it into an optimal method for strongly convex or strongly monotone problems. Within iteration of VIJI-Restarted listed as Algorithm 2, we execute VIJI for a predefined number of iterations (26). Next, the output of this run is used as initial point for next run of VIJI with parameters reset, and this iterative process continues.

Algorithm 2 VIJI-Restarted

Input: initial point z 0 ∈ X , D = max x,y ∈X , parameters L , δ .

Initialization : n = ⌈ log D ε ⌉ , R = D .

for i=1, ..., n do

Set x 0 = z i -1 , R i -1 = R 2 i -1 .

Run Algorithm 1 with β k = δ, η = 10 , opt = 0 for T i iterations, where

Set z i = x T i .

Theorem 6.2 Let Assumptions 1.2, 2.1, 6.1 hold. Then the total number of iterations of Algorithm 2 to reach desired accuracy ∥ z s -x ∗ ∥ ≤ ε , where x ∗ is the solution of (2) is

7 Tensor generalization

In this section, we generalize the results presented in Section 3 to the p -th order case. We consider a higher-order method with inexact high-order derivatives, satisfying the following assumption.

Assumption 7.1 For all x, v ∈ X , i ≥ 1 , i -th inexact derivative of F , which we denote as G i , satisfies

Based on inexact ( p -1) -th-order Taylor approximation Ψ p,v ( x ) = F ( v )+ ∑ p -1 i =1 1 i ! ∇ i G i ( v )[ x -v ] i , we introduce the inexact tensor model Ω p,v ( x ) of the objective

The tensor generalization of Algorithm 1, referred to as VIHI (High-order Method for V ariational I nequalitues under H igh-order derivatives I nexactness ) and detailed in Appendix G, involves the inexact solution of the subproblem, which satisfies the following condition:

Another difference in VIHI compared to VIJI (Algorithm 1) is the adaptive strategy for λ k +1 :

The other steps of Algorithm 1 remain unchanged for the higher-order method. Now, we are ready to present the convergence properties of VIHI.

Theorem 7.2 Let Assumptions 1.1, 4.2 with i = p -1 , and 7.1 hold. Then, after T ≥ 1 iterations of VIHI with parameters η i = 5 p , opt = 0 , we get the following bound

Finally, we extend our tensor generalization to nonmonotone case and obtain the following result.

Theorem 7.3 Let Assumptions 1.3, 4.2 with i = p -1 , and 7.1 hold. Then after T ≥ 1 iterations of VIHI with parameters η = 5 p, opt = 2 we get the following bound

8 Experiments

In this section, we present numerical experiments to demonstrate the efficiency of our proposed methods. We consider the cubic regularized bilinear min-max problem of the form:

where ρ > 0 , b = [1 , 0 , . . . , 0] ∈ R d , and A ∈ R d × d , with all 1 on the main diagonal and all -1 on the upper diagonal, the rest elements are 0 . To reformulate it as variational inequality, we define F ( x ) = [ ∇ x f ( x, y ) , -∇ y f ( x, y )] . This problem is inspired by the first-order lower bound function for variational inequalities and min-max problems and is commonly used to verify the convergence of high-order methods for VI [71, 52].

We implement our second-order method for V ariational I nequalities with Q uasi-Newton A pproximation (VIQA) as a PyTorch optimizer. The code is available in the OPTAMI package 4 [58]. VIQA Broyden refers to L-Broyden approximation (20) and VIQA Damped Broyden to (21), which are used as inexact Jacobians in VIJI (Algorithm 1). We compare them with the Extragradient method (EG) [62], first-order Perseus (Perseus1), and second-order Perseus with Jacobian (Perseus2).

0

20000

40000

60000

80000

100000

10

14

10

12

10

10

10

8

10

6

10

4

10

2

10 0

Gap

EG

Perseus1

Perseus2

VIQA Damped Broyden

VIQA Broyden

0

25000

50000

75000

100000

125000

150000

175000

200000

JVP/F(x) computations

10

14

10

12

10

10

10

8

10

6

10

4

10

2

10 0

Gap

EG

Perseus1

Perseus2

VIQA Damped Broyden

VIQA Broyden

Iteration, k

In Figure 1, one can see that second-order information in Perseus2 significantly accelerates the convergence compared to Perseus1. However, it is expensive to compute Jacobian and solve secondorder subproblems every iteration. VIQA with the proposed Damped Broyden approximation (21) is significantly faster than EG, Perseus1, and VIQA Broyden (20). It shows that this approximation improves the convergence of first-order Perseus1 and confirms the theoretical result that Damped Broyden is a more accurate approximation than classical Broyden from Theorem 5.1. The detailed parameters and setup are presented in Appendix I.

9 Conclusion

In this work, we introduced a second-order method specifically designed to handle Jacobian inexactness. We demonstrated its optimality in the monotone case by introducing a new lower bound and extended its applicability to tensor methods. However, similar to other high-order methods with global convergence properties, our algorithm involves a subproblem that necessitates an additional subroutine for its solution. To address this challenge, we proposed a computationally feasible criterion for solving the subproblem and implemented Quasi-Newton approximations for Jacobians, resulting in a significant reduction in per-iteration cost. Future investigations could explore incorporating inexactness within the operator itself and developing adaptive schemes to dynamically adjust for the level of inexactness encountered during the optimization process. Another open problem is a design of more accurate Quasi-Newton approximations specifically for Jacobians, focusing on non-symmetrical structure of Jacobian and specific inexactness criteria such as Assumption 2.1.

Acknowledgments.

The work was supported by MIPT based Center of National Technology Initiatives in the field of Artificial Intelligence for the purposes of the "road map" of Artificial Intelligence development up to 2030 and supported by NTI Foundation (agreement No.70-2021-00207 dated 22.11.2021, identifier 000000S507521QYL0002).

References

S. S. Ablaev, A. A. Titov, F. S. Stonyakin, M. S. Alkousa, and A. Gasnikov. Some adaptive first-order methods for variational inequalities with relatively strongly monotone operators and generalized smoothness. In International Conference on Optimization and Applications , pages 135-150. Springer, 2022.

D. Adil, B. Bullins, A. Jambulapati, and S. Sachdeva. Optimal methods for higher-order smooth monotone variational inequalities. arXiv preprint arXiv:2205.06167 , 2022.

A. Agafonov, D. Kamzolov, P. Dvurechensky, A. Gasnikov, and M. Takáˇ c. Inexact tensor methods and their application to stochastic convex optimization. Optimization Methods and Software , 39(1):42-83, 2024. doi: 10.1080/10556788.2023.2261604. URL https://doi. org/10.1080/10556788.2023.2261604 .

A. Agafonov, D. Kamzolov, A. Gasnikov, A. Kavis, K. Antonakopoulos, V. Cevher, and M. Takáˇ c. Advancing the lower bounds: an accelerated, stochastic, second-order method with optimal adaptation to inexactness. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=otU31x3fus .

M. M. Alves and B. F. Svaiter. A search-free O (1 /k 3 / 2 ) homotopy inexact proximal-newton extragradient algorithm for monotone variational inequalities. arXiv preprint arXiv:2308.05887 , 2023.

A. Antipin. Method of convex programming using a symmetric modification of lagrange function. Matekon , 14(2):23-38, 1978.

K. Antonakopoulos, A. Kavis, and V. Cevher. Extra-newton: A first approach to noise-adaptive accelerated second-order methods. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 29859-29872. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ c10804702be5a0cca89331315413f1a2-Paper-Conference.pdf .

F. Bach, J. Mairal, and J. Ponce. Convex sparse matrix factorizations. arXiv preprint arXiv:0812.1869 , 2008.

F. Bach, R. Jenatton, J. Mairal, G. Obozinski, et al. Optimization with sparsity-inducing penalties. Foundations and Trends® in Machine Learning , 4(1):1-106, 2012.

M. Baes. Estimate sequence methods: extensions and approximations. Institute for Operations Research, ETH, Zürich, Switzerland , 2(1), 2009.

S. Bellavia and G. Gurioli. Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic hessian accuracy. Optimization , 71:227-261, 2022. doi: 10.1080/02331934.2021.1892104. URL https://doi.org/10.1080/02331934.2021. 1892104 .

S. Bellavia, G. Gurioli, B. Morini, and P. Toint. Adaptive regularization for nonconvex optimization using inexact function values and randomly perturbed derivatives. Journal of Complexity , 68:101591, 2022. ISSN 0885-064X. doi: https://doi.org/10.1016/j.jco.2021.101591. URL https://www.sciencedirect.com/science/article/pii/S0885064X21000467 .

A. Ben-Tal, L. El Ghaoui, and A. Nemirovski. Robust optimization , volume 28. Princeton university press, 2009.

A. S. Berahas, M. Jahani, P. Richtárik, and M. Takáˇ c. Quasi-newton methods for machine learning: forget the past, just sample. Optimization Methods and Software , 37:1668-1704, 2022. doi: 10.1080/10556788.2021.1977806. URL https://doi.org/10.1080/10556788.2021. 1977806 .

A. Beznosikov, A. Alanov, D. Kovalev, M. Takáˇ c, and A. Gasnikov. On scaled methods for saddle point problems. arXiv preprint arXiv:2206.08303 , 2022.

A. Beznosikov, B. Polyak, E. Gorbunov, D. Kovalev, and A. Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems: A survey. European Mathematical Society Magazine , (127):15-28, 2023.

A. Beznosikov, M. Takác, and A. Gasnikov. Similarity, compression and local steps: three pillars of efficient communications for distributed variational inequalities. Advances in Neural Information Processing Systems , 36, 2023.

J. F. Bonnans. Local analysis of newton-type methods for variational inequalities and nonlinear programming. Applied Mathematics and Optimization , 29:161-186, 1994.

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

L. Brighi and R. John. Characterizations of pseudomonotone maps and economic equilibrium. Journal of Statistics and Management Systems , 5(1-3):253-273, 2002.

B. Bullins and K. A. Lai. Higher-order methods for convex-concave min-max optimization and monotone variational inequalities. SIAM Journal on Optimization , 32(3):2208-2229, 2022.

Y. Carmon, D. Hausler, A. Jambulapati, Y. Jin, and A. Sidford. Optimal and adaptive monteirosvaiter acceleration. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 20338-20350. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/7ff97417474268e6b5a38bcbfae04944-Paper-Conference.pdf .

C. Cartis and K. Scheinberg. Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Mathematical Programming , 169:337-375, 2018. ISSN 1436-4646. doi: 10.1007/s10107-017-1137-4. URL https://doi.org/10.1007/ s10107-017-1137-4 .

C. Cartis, N. I. Gould, and P. L. Toint. Adaptive cubic regularisation methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mathematical Programming , 127(2):245-295, 2011.

C. Cartis, N. I. Gould, and P. L. Toint. Adaptive cubic regularisation methods for unconstrained optimization. part ii: worst-case function-and derivative-evaluation complexity. Mathematical programming , 130(2):295-319, 2011.

C. Cartis, N. I. Gould, and P. L. Toint. Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation and Perspectives . SIAM, 2022.

A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision , 40:120-145, 2011.

T. Chavdarova, G. Gidel, F. Fleuret, and S. Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. Advances in Neural Information Processing Systems , 32, 2019.

S. C. Choi, W. S. DeSarbo, and P. T. Harker. Product positioning under price competition. Management Science , 36(2):175-199, 1990.

C. D. Dang and G. Lan. On the convergence properties of non-euclidean extragradient methods for variational inequalities with generalized monotone operators. Computational Optimization and applications , 60:277-310, 2015.

C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng. Training GANs with optimism. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=SJJySbbAZ .

C. Daskalakis, S. Skoulakis, and M. Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478, 2021.

O. Devolder, F. Glineur, and Y. Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37-75, 2014. ISSN 1436-4646. doi: 10.1007/s10107-013-0677-5. URL https://doi.org/10.1007/s10107-013-0677-5 .

N. Doikov, E. M. Chayti, and M. Jaggi. Second-order optimization with lazy Hessians. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 8138-8161. PMLR, 9 2023. URL https://proceedings.mlr.press/v202/doikov23a.html .

E. Esser, X. Zhang, and T. F. Chan. A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences , 3 (4):1015-1046, 2010.

C. Ewerhart. Cournot games with biconcave demand. Games and Economic Behavior , 85: 37-47, 2014.

F. Facchinei and J.-S. Pang. Finite-dimensional variational inequalities and complementarity problems . Springer, 2003.

D. Fernández. A quasi-newton strategy for the ssqp method for variational inequality and optimization problems. Mathematical Programming , 137:199-223, 2013.

G. Gallego and M. Hu. Dynamic pricing of perishable assets under competition. Management Science , 60(5):1241-1259, 2014.

S. Ghadimi, H. Liu, and T. Zhang. Second-order methods with cubic regularization under inexact information. arXiv preprint arXiv:1710.05782 , 2017.

G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien. A variational inequality perspective on generative adversarial networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=r1laEnA5Ym .

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

E. Gorbunov, N. Loizou, and G. Gidel. Extragradient method: O (1 /k ) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. In International Conference on Artificial Intelligence and Statistics , pages 366-402. PMLR, 2022.

E. Gorbunov, A. Taylor, and G. Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. Advances in neural information processing systems , 35: 21858-21870, 2022.

J. Han and D. Sun. Newton-type methods for variational inequalities. In Advances in Nonlinear Programming: Proceedings of the 96 International Conference on Nonlinear Programming , pages 105-118. Springer, 1998.

S. Hanzely, D. Kamzolov, D. Pasechnyuk, A. Gasnikov, P. Richtárik, and M. Takáˇ c. A damped Newton method achieves global O ( 1 k 2 ) and local quadratic convergence rate. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 25320-25334. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ a1f0c0cd6caaa4863af5f12608edf63e-Paper-Conference.pdf .

P. T. Harker and J.-S. Pang. Finite-dimensional variational inequality and nonlinear complementarity problems: a survey of theory, algorithms and applications. Mathematical programming , 48(1):161-220, 1990.

P. Hartman and G. Stampacchia. On some non-linear elliptic differential-functional equations. Acta Mathematica , 115(1):271-310, 1966.

K. Huang, J. Zhang, and S. Zhang. Cubic regularized Newton method for the saddle point models: A global and local convergence analysis. Journal of Scientific Computing , 91(2):60, 2022.

A. N. Iusem, A. Jofré, R. I. Oliveira, and P. Thompson. Extragradient method with variance reduction for stochastic variational inequalities. SIAM Journal on Optimization , 27(2):686-724, 2017.

M. Jahani, X. He, C. Ma, A. Mokhtari, D. Mudigere, A. Ribeiro, and M. Takác. Efficient distributed hessian free algorithm for large-scale empirical risk minimization via accumulating sample strategy. In International Conference on Artificial Intelligence and Statistics , pages 2634-2644. PMLR, 2020.

R. Jiang and A. Mokhtari. Generalized optimistic methods for convex-concave saddle point problems. arXiv preprint arXiv:2202.09674 , 2022.

R. Jiang and A. Mokhtari. Accelerated quasi-newton proximal extragradient: Faster rate for smooth convex optimization. Advances in Neural Information Processing Systems , 36, 2024.

R. Jiang, Q. Jin, and A. Mokhtari. Online learning guided curvature approximation: A quasinewton method with global non-asymptotic superlinear convergence. In The Thirty Sixth Annual Conference on Learning Theory , pages 1962-1992. PMLR, 2023.

Y. Jin and A. Sidford. Efficiently solving mdps with stochastic mirror descent. In International Conference on Machine Learning , pages 4890-4900. PMLR, 2020.

T. Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd international conference on Machine learning , pages 377-384, 2005.

D. Kamzolov, K. Ziu, A. Agafonov, and M. Takáˇ c. Cubic regularization is the key! the first accelerated quasi-newton method with a global convergence rate of o ( k -2 ) for convex functions. arXiv preprint arXiv:2302.04987 , 2023.

D. Kamzolov, D. Pasechnyuk, A. Agafonov, A. Gasnikov, and M. Takáˇ c. Optami: Global superlinear convergence of high-order methods. arXiv preprint arXiv:2410.04083 , 2024.

A. Kannan and U. V. Shanbhag. Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants. Computational Optimization and Applications , 74(3):779-820, 2019.

B. Kleinberg, Y. Li, and Y. Yuan. An alternative view: When does sgd escape local minima? In International Conference on Machine Learning , pages 2698-2707. PMLR, 2018.

J. M. Kohler and A. Lucchi. Sub-sampled cubic regularization for non-convex optimization. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 1895-1904. PMLR, 5 2017. URL https://proceedings.mlr. press/v70/kohler17a.html .

G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

G. Kotsalis, G. Lan, and T. Li. Simple and optimal methods for stochastic variational inequalities, i: operator extrapolation. SIAM Journal on Optimization , 32(3):2041-2073, 2022.

D. Kovalev and A. Gasnikov. The first optimal acceleration of high-order methods in smooth convex optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 35339-35351. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/e56f394bbd4f0ec81393d767caa5a31b-Paper-Conference.pdf .

X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89, pages 983-992. PMLR, 4 2019. URL https://proceedings.mlr.press/v89/li19c.html .

Y. Li and Y. Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems , 30, 2017.

T. Liang and J. Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 907-915. PMLR, 2019.

T. Lin and M. I. Jordan. Monotone inclusions, acceleration, and closed-loop control. Mathematics of Operations Research , 48(4):2353-2382, 2023.

T. Lin and M. I. Jordan. Perseus: A simple and optimal high-order method for variational inequalities. Mathematical Programming , pages 1-42, 2024.

T. Lin, M. Jordan, et al. A continuous-time perspective on optimal methods for monotone equation problems. arXiv preprint arXiv:2206.04770 , 2022.

T. Lin, P. Mertikopoulos, and M. I. Jordan. Explicit second-order min-max optimization methods with optimal convergence guarantee. arXiv preprint arXiv:2210.12860 , 2022.

C. Liu and L. Luo. Regularized newton methods for monotone variational inequalities with h \ " older continuous jacobians. arXiv preprint arXiv:2212.07824 , 2022.

D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45:503-528, 1989. doi: 10.1007/BF01589116. URL https: //doi.org/10.1007/BF01589116 .

A. Lucchi and J. Kohler. A sub-sampled tensor method for nonconvex optimization. IMA Journal of Numerical Analysis , 43:2856-2891, 10 2023. ISSN 0272-4979. doi: 10.1093/ imanum/drac057. URL https://doi.org/10.1093/imanum/drac057 .

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=rJzIBfZAb .

P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, and G. Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=Bkg8jjC9KQ .

G. J. Minty. Monotone (nonlinear) operators in hilbert space. Duke Mathematical Journal , 29 (3), 1962.

A. Mokhtari, A. E. Ozdaglar, and S. Pattathil. Convergence rate of o(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

R. D. C. Monteiro and B. F. Svaiter. An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM Journal on Optimization , 23:1092-1125, 2013. doi: 10.1137/110833786. URL https://doi.org/10. 1137/110833786 .

A. Nemirovski. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Y. Nesterov. Dual extrapolation and its applications to solving variational inequalities and related problems. Mathematical Programming , 109(2):319-344, 2007.

Y. Nesterov. Accelerating the cubic regularization of Newton's method on convex problems. Mathematical Programming , 112:159-181, 2008. ISSN 1436-4646. doi: 10.1007/ s10107-006-0089-x. URL https://doi.org/10.1007/s10107-006-0089-x .

Y. Nesterov. Implementable tensor methods in unconstrained convex optimization. Mathematical Programming , 186:157-183, 2021. ISSN 1436-4646. doi: 10.1007/s10107-019-01449-1. URL https://doi.org/10.1007/s10107-019-01449-1 .

Y. Nesterov. High-order reduced-gradient methods for composite variational inequalities. arXiv preprint arXiv:2311.15154 , 2023.

Y. Nesterov and B. T. Polyak. Cubic regularization of Newton method and its global performance. Mathematical Programming , 108:177-205, 2006. doi: 10.1007/s10107-006-0706-8. URL https://doi.org/10.1007/s10107-006-0706-8 .

J. Nocedal and S. J. Wright. Numerical Optimization . Springer New York, 2 edition, 2006. ISBN 978-0-387-30303-1. doi: 10.1007/978-0-387-40065-5.

S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and J. Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning , pages 2681-2690. PMLR, 2017.

P. Ostroukhov, R. Kamalov, P. Dvurechensky, and A. Gasnikov. Tensor methods for strongly convex strongly concave saddle point problems and strongly monotone variational inequalities. arXiv preprint arXiv:2012.15595 , 2020.

Y. Ouyang and Y. Xu. Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems. Mathematical Programming , 185(1):1-35, 2021.

D. A. Pasechnyuk, A. Gasnikov, and M. Takáˇ c. Convergence analysis of stochastic gradient descent with adaptive preconditioning for non-convex and convex functions. arXiv preprint arXiv:2308.14192 , 2023.

W. Peng, Y.-H. Dai, H. Zhang, and L. Cheng. Training gans with centripetal acceleration. Optimization Methods and Software , 35(5):955-973, 2020.

L. D. Popov. A modification of the arrow-hurwitz method of search for saddle points. Mat. Zametki , 28(5):777-784, 1980.

A. Sadiev, A. Beznosikov, A. J. Almansoori, D. Kamzolov, R. Tappenden, and M. Takáˇ c. Stochastic gradient methods with preconditioned updates. Journal of Optimization Theory and Applications , 201:471-489, 2024. ISSN 1573-2878. doi: 10.1007/s10957-023-02365-3. URL https://doi.org/10.1007/s10957-023-02365-3 .

N. Tripuraneni, M. Stern, C. Jin, J. Regier, and M. I. Jordan. Stochastic cubic regularization for fast nonconvex optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/ paper/2018/file/db1915052d15f7815c8b88e879465a1e-Paper.pdf .

P. Tseng. A modified forward-backward splitting method for maximal monotone mappings. SIAM Journal on Control and Optimization , 38(2):431-446, 2000.

M. A. Woodbury. The stability of out-input matrices. Chicago, IL , 9:3-8, 1949.

M. A. Woodbury. Inverting modified matrices . Department of Statistics, Princeton University, 1950.

L. Xu, J. Neufeld, B. Larson, and D. Schuurmans. Maximum margin clustering. Advances in neural information processing systems , 17, 2004.

P. Xu, F. Roosta, and M. W. Mahoney. Newton-type methods for non-convex optimization under inexact Hessian information. Mathematical Programming , 184:35-70, 2020. ISSN 1436-4646. doi: 10.1007/s10107-019-01405-z. URL https://doi.org/10.1007/ s10107-019-01405-z .

Contents

□

Lemma 3.1 Let Assumptions (1.1) , (1.2) , (2.1) hold. Then for any x, v ∈ X VI (11) is monotone

Proof. For all x, v ∈ X

B Proofs of Theorems 3.2, 3.3

B.1 Proof of Theorem 3.2

In this section, we let L := L 1 . To show the convergence of Algorithm 1, we define the following Lyapunov function

Lemma B.1 Let Assumption 1.2, 2.1 hold. Then, for every integer T ≥ 1 , we have

Proof. By the definition of Lyapunov function (30) and Step 2 of Algorithm 1, we have

Then, we have

By the update formula for v k +1 , we get

Letting x = v k +2 in this inequality and using ⟨ a, b ⟩ = 1 2 ( ∥ a + b ∥ 2 -∥ a ∥ 2 -∥ b ∥ 2 ) , we have

Plugging Eq. (32) into Eq. (31) and using Step 5 of Algorithm 1, we obtain:

for any x ∈ X . Summing up this inequality over k = 0 , 1 , . . . , T -1 and changing the counter k +1 to k yields that

Using the update formula for s k +1 and letting s 0 = 0 d ∈ R d , we have

Since x k +1 ∈ X satisfies (12), we have

where Ω η v ( x ) : R d → R d is defined in (10). Letting x = v k +1 in (35), we have

Then,

Next, using ⟨ x k -v k , x k -v k +1 ⟩ ≥ ∥ x k -v k ∥ 2 - ∥ x k -v k ∥∥ v k -v k +1 ∥ and ∥ x k -v k +1 ∥ ≤ ∥ x k -v k ∥ + ∥ v k -v k +1 ∥ , we get

Next,

where the last inequality is due to the following choice of η = 10 and λ : 1 32 ≤ λ k ( L 2 ∥ x k -v k ∥ + δ ) ≤ 1 22 . Then,

Plugging (34) and (37) into (33) yields that

Lemma B.2 Let Assumptions 1.2, 1.3, 2.1 hold and let x ∈ X . For every integer T ≥ 1 , we have

where x ∗ ∈ X denotes the weak solution to the VI.

Proof. For any x ∈ X , we have

Since s 0 = 0 d , we have E 0 = 0 and

This together with Lemma B.1 yields that

which implies the first inequality. Since the VI satisfies the Minty condition, there exists x ∗ ∈ X such that ⟨ F ( x k ) , x k -x ∗ ⟩ ≥ 0 for all k ≥ 1 . Letting x = x ∗ in the above inequality yields the second inequality. □

Lemma B.3 Let Assumptions 1.2, 1.3, 2.1 hold. For every integer T ≥ 1 , we have

where x ∗ ∈ X denotes the weak solution to the VI.

Proof. Without loss of generality, we assume that x 0 = x ∗ . We have

̸

By the Hölder inequality, we have

Putting these pieces together yields that

Plugging this into the above inequality yields that

Theorem 3.2 Let Assumptions 1.2, 1.1, 2.1. Then, after T ≥ 1 iterations of VIJI with parameters β = δ, η = 10 , opt = 0 , we get the following bound

Proof. Letting x ∈ X , we derive from the monotonicity of F and the definition of ˜ x T (i.e., opt = 0 ) that

Combining this inequality with the first inequality in Lemma B.2 yields that

Since x 0 ∈ X , we have ∥ x -x 0 ∥ ≤ D and hence

Then, we combine Lemma B.3 and the fact that ∥ x ∗ -x 0 ∥ ≤ D to obtain that

By the definition of a gap function. (4), we have

B.2 Proof of Theorem 3.3

We directly follow the steps of the proof of Theorem 3.2. Lemmas B.1, B.2 remain the same. Because of the choice of β k +1 = L 1 2 ∥ x k +1 -v k +1 ∥ adaptive strategy for λ k +1 looks as follows: 1 32 ≤ L ∥ x k +1 -v k +1 ∥ ≤ 1 22 . Next Lemma is a counterpart of Lemma B.3.

where x ∗ ∈ X denotes the weak solution to the VI.

Proof. Without loss of generality, we assume that x 0 = x ∗ . We have

̸

By the Hölder inequality, we have

Putting these pieces together yields that

Plugging this into the above inequality yields that

Then, by following the rest of the proof of Theorem 3.2, we get

Theorem 3.3 Let Assumptions 1.2, 1.1 hold. Let { x k , v k } be iterates generated by Algorithm 1 and ∥ ( ∇ F ( v k ) -J ( v k ))[ z k -v k ] ∥ ≤ δ k ∥ z k -v k ∥ , δ k ≤ L 1 2 ∥ x k -v k ∥ .

Then, after T ≥ 1 iterations of Algorithm 1 with parameters β k = L 1 2 ∥ x k -v k ∥ , η = 10 , opt = 0 , we get the following bound

□

C Proofs of Theorems 3.4, 3.5

In this section, we let L := L 1 .

Theorem 3.4 Let Assumptions 1.2, 1.3, 2.1 hold. Then after T ≥ 1 iterations of Algorithm 1 with parameters β k = δ, η = 10 , opt = 2 we get the following bound

Proof. Since we consider opt = 2 , then

From (36) we get

Next, from triangle inequality we have

From this and (29) we get

Now, we can return to (43):

Next, from second inequality from (38) and from definition of x k T in Algorithm 1 we obtain

Since (44) holds for any x ∈ X , we get final result

Theorem 3.5 Let Assumptions 1.2, 1.3 hold. Let { x k , v k } be iterates generated by Algorithm 1 that satisfy (14) . Then after T ≥ 1 iterations of Algorithm 1 with parameters β k = L 2 ∥ x k -v k ∥ , η = 10 , opt = 2 we get the following bound

The proof of this theorem repeats the proof of Theorem 3.4 with δ ≤ L 2 ∥ x k -v k ∥ .

D Proof of Theorem 4.3

Theorem 4.3 Let some first-order method M satisfy Assumption 4.1 and have access only δ -inexact first-order oracle 15. Assume the method M ensures for any L 0 -zero-order smooth and L 1 -first-order smooth monotone operator F the following convergence rate

Proof. We prove this Theorem by contradiction. Assume the existence of a method M that satisfies the conditions of Theorem 4.3 and achieves faster rate in one of the terms (18).

First, suppose Ξ 1 ( T ) > T . Consider the first-order lower bound from [89], which is established using a quadratic min-max problem as the worst-case function. In this scenario, the operator has a 0 -Lipschitz continuous Jacobian. Applying first-order method M to this lower bound and using an inexact Jacobian J ( x ) = L 0 I d × d , yields the rate O ( L 0 D 2 Ξ 1 ( T ) ) , Ξ 1 ( T ) > T , which is faster than the lower bound Ω ( L 0 D 2 ) , contradicting our assumption.

T

Secondly, let us assume Ξ 2 ( T ) > T 3 / 2 . The lower bound for exact second-order methods is Ω ( L 1 D 3 T 3 / 2 ) [69]. By taking exact Jacobian in the method M ( δ = 0 ), we place M in the class of exact second-order methods. Consequently, we obtain a contradiction with the lower bound. □

E Proof of Theorem 5.1

Proof. The proof is common for both formulas (20) and (21), where α = 1 for classical L-Broyden approximation and α = m +1 for Damped L-Broyden approximation. First, from L 0 -zero-order smoothness, get

Now, we upper-bound ∥ J x ∥ op = ∥ J m ∥ op by induction:

where the last inequality is coming from L 0 -zero-order smoothness for operator difference or JVP. By summing up the previous inequality for i in 0 , . . . , m -1 , we get ∥ J m ∥ op ≤ ∥ J 0 ∥ op + mL 0 α . Finally, we prove the result of Theorem 5.1. □

F Proof of Theorem 6.2

In this section, we let L := L 1 .

Theorem 6.2 Let Assumptions 1.2, 2.1, 6.1 hold. Then the total number of iterations of Algorithm 2 to reach desired accuracy ∥ z s -x ∗ ∥ ≤ ε is

Then for all T ≥ 1 we have

Proof. From the definition of ˜ x T , Jensen inequality, strong monotonicity (25), definition of strong minty problem (2) and Lemmas B.2, B.3 we get

Denote R := D, R i = R 2 i , i ≥ 1 . Now we run Algorithm 1 in cycle for i = 1 , ..., n and restart it every time its distance to the solution becomes at least twice less than R i -1 . Thus, let T i be number of iterations we run Algorithm 1 inside cycle of Algorithm 2. In other words, let T i be such that ∥ ∥ ˜ x T i -1 -x ∗ ∥ ∥ ≤ R i -1 2 where ˜ x T i +1 is the point, where we restart Algorithm 1. Then the number of iterations before the i -th restart is

Deriving T i for each case under max , we get

Now we calculate the total number of restarts to reach ∥ ˜ x T n -x ∗ ∥ ≤ ε . From (47) we can get that

Thus,

Deriving n from last inequality, we get

Thus,

Now we provide auxiliary estimation, that we will need next:

Finally, we can get the total number of iterations of Algorithm 1 inside Algorithm 2:

This completes the proof.

G Tensor generalization with more details

G.1 Preliminaries

Algorithm 3 VIHI

Input: initial point x 0 ∈ X , parameters L 1 , η , sequence { δ i } p -1 i =1 , and opt ∈ { 0 , 1 , 2 } . Initialization: set s 0 = 0 ∈ R d .

Compute x k +1 ∈ X such that condition (53) holds true.

Compute λ k +1 such that

In this section, we provide more details on the generalization of Algorithm 1 with high-order derivatives. We provide the pseudocode of the resulting method in Algorithm 3. To show the convergence of Algorithm 3, we use the Lyapunov function (30).

Define the ( p -1) -th order approximations of the F

On each step, our method solves the following subproblem:

Additionally, we will need an auxiliary result from [52], based on Assumption 4.2. The authors show, that this Assumption allows to control the quality of approximation of operator F by its high-order Taylor polynomial:

G.2 Auxiliary lemmas

First of all, we provide high-order generalizations of auxiliary lemmas for second-order case from Section A.

Proof.

□

Lemma G.2 Let Assumptions 1.1, 7.1 and 4.2 with i = p -1 hold. Then for any x, v k +1 ∈ X VI (11) is relatively strongly monotone if η i ≥ p

Proof.

From monotonicity of F we know that 1 2 ( F ( x ) -F ( x ) T ) ⪰ 0 . Thus,

By rearranging the terms we get

Thus, if η i ≥ p , we get that 1 2 ( ∇ Ω p,v k +1 ( x ) + ∇ Ω p,v k +1 ( x ) T ) is relatively strongly monotone. □

G.3 Convergence in monotone case

In this subsection we provide theoretical results directly connected to convergence rate of Algorithm 3. Firstly, we need to introduce additional technical lemmas, that generalize corresponding lemmas in Section B.

Lemma G.3 Let Assumption 1.1 hold and η i = 5 p . Then, for every T ≥ 1 , we have

Proof. Since the first steps in the proof of this Lemma are the same as in Lemma B.1, we start our reasoning from (33):

Using the update formula for s k +1 and letting s 0 = 0 d ∈ R d , we have

Now consider II . Using (28) we get

From definition of the subproblem (53) we have that

From this we get

Now, if we choose λ k in a such way that

we get the following

Finally, combining estimations of I and II with (57), we get

Lemma G.4 Let Assumptions 4.2, 1.3 hold. For every T ≥ 1 we have

Proof. From Hölder inequality

Consider denominator:

For p ≥ 2 we have

Consider some nonnegative sequence { a i | a i ≥ 0 , ∀ i ∈ 1 , n } . For p = 2 from Jensen inequality we know that ( ∑ n i =1 a i ) 2 ≤ ∑ n i =1 na 2 i . From Lemma 7 of [65] we know that ∀ q ∈ [0 , 1] , x, y > 0 → ( x + y ) q ≤ x q + y q . Thus, for p ≥ 3 we can come to the same conclusion: ( ∑ n i =1 a i ) 2 p -1 ≤ ∑ n i =1 a 2 p -1 i ≤ ∑ n i =1 na 2 p -1 i . In other words,

Using this inequality, we get

Consider the second factor in this inequality. If we denote a k = ∥ x k -v k ∥ 2( i -1) p -1 , b k = 1 , c = p -1 i -1 , d = p -1 p -i , then we can use Hölder inequality:

From this we get

Thus,

Consider ( a + b ) p -1 2 , p ≥ 2 . For p = 2 from Lemma 7 in [65] we get ( a + b ) 1 2 ≤ a 1 2 + b 1 2 < 2( a 1 2 + b 1 2 ) . For p ≥ 3 we have from Jensen inequality ( a + b ) p -1 2 ≤ 2 p -1 2 -1 ( a p -1 2 + b p -1 2 ) < 2 p 2 ( a p -1 2 + b p -1 2 ) . In other words,

Thus,

Now, we can get the final result

Finally, we provide the convergence rate of Algorithm 3 in monotone case.

Theorem 7.2 Let Assumptions 1.1, 4.2 with i = p -1 , and 7.1 hold. Then, after T ≥ 1 iterations of VIHI with parameters η i = 5 p , opt = 0 , we get the following bound

Proof. Consider ∀ x ∈ X , opt = 0 .

Thus,

G.4 Convergence in nonmonotone case

To make our paper more self-contained, we provide convergence rate of Algorithm 3 in nonmonotone case.

Theorem 7.3 Let Assumptions 4.2, 1.3 and 7.1 with i = p -1 hold. Then after T ≥ 1 iterations of Algorithm 3 with parameters η = 5 p, opt = 2 we get the following bound

From definition of Ψ p,v k ( x k ) and triangle inequality we get

From this and (55) we get

Thus,

From second inequality of (38) and definition of x k T we get

Combining all these results, we get

H Subproblem solution

For r -rank QN approximation J r from (19), we can effectively compute A -1 τ F ( x ) , where A τ = J r +( ηδ + 5 L 2 τ ) I = U ⊤ CV + J 0 +( ηδ + 5 L 2 τ ) I = U ⊤ CV + B and B = J 0 +( ηδ + 5 L 2 τ ) I by using the Woodbury matrix identity[96, 97].

For J 0 = ιI , the identity could be simplified even more. Hence, B = ιI +( ηδ + 5 L 2 τ ) I = χI , where χ = ι + ηδ + 5 L 2 τ . Then,

I Experiment details

We consider the cubic regularized bilinear min-max problem of the form:

where ρ > 0 , b = [1 , 0 , . . . , 0] ∈ R d , and A ∈ R d × d such that

To reformulate it as variational inequality, we define F ( x ) = [ ∇ x f ( x, y ) , -∇ y f ( x, y )] . Following [71], we plot the restricted primal-dual gap (65), written in a closed form:

where z = ( x, y ) .

Setup. All methods and experiments were performed using Python 3.11.5, PyTorch 2.1.2, numpy 1.24.3 on a 16-inch MacBook Pro 2023 with an Apple M2 Pro and 32GB memory.

Parameters. For Figure 1, the dimension of the problem d = 50 . The regularizer ρ is set to ρ = 1 e -3 . The starting point ( x 0 , y 0 ) = (0 , 0) is all zeroes. We present results of last iteration of each method or opt=1 from Algorighm 1. The diameter β for the gap is set as β = 1 . For QN methods, we set memory size m = r = 20 . We perform a total of 100000 iterations for all methods except Perseus2 with 1000 iterations. We plot each 500 iterations for a better visualisation. In Figure 1, the left plot refers to the gap decrease per iteration and the right plot refers to the gap decrease per JVP/operator computations. EG, Perseus1, VIQA computes 2 operators per iteration and Perseus2 computes d +1 JVP/operators.

We finetuned learning rate and presented the run with the best results. For EG1: lr = 0 . 5 . For Perseus1: L 0 = 0 . 2334 . For Perseus2: L 1 = 0 . 0001 . For VIQA Broyden: L 1 = 0 . 001 , δ = J 0 = 0 . 4 . For VIQA Damped Broyden: L 1 = 0 . 001 , δ = J 0 = 0 . 22 . Note, that tuned L 1 = 0 . 001 is actually a theoretical smoothness constant for the problem (63) and can be chosen with such value without finetuning. Also, Perseus2 is working with a smaller constant than theoretical.

J Application to minmax problems

J.1 Preliminaries

In this section, we consider the problem of finding a global saddle point of the following min-max optimization problem:

i.e., a tuple ( x ∗ , y ∗ ) ∈ R m × R n such that

where the continuously differentiable objective function f is convex-concave: f ( x, y ) is convex in x for all y ∈ R n and concave in y for all x ∈ R m .

Algorithm 4 VIJI-MinMax

Input: initial point z 0 ∈ X , parameters L 1 , δ , η , τ . Initialization: set s 0 = 0 ∈ R d . for k = 0 , 1 , 2 , . . . , T do Set v k +1 = z 0 + s k +1 . Compute z k +1 ∈ R n + m such that condition (68) holds true. Compute λ k +1 such that 1 16 ≤ λ k +1 ( L 2 ∥ z k +1 -v k +1 ∥ + δ ) ≤ 1 12 . Compute s k +1 = s k -λ k +1 F ( z k +1 ) . .

Assumption J.1 The function f ( x, y ) ∈ C 2 has L -Lipschitz-continious second-order derivative if

Following [71], we define the restricted gap function to measure the optimality of point ˆ z = (ˆ x, ˆ y )

where β is sufficiently large such that ∥ ˆ z -z ∗ ∥ ≤ β .

Problem (64) is a special case of VI defined by the following operator

The Jacobian of F is defined as follows

The following lemma [80, 71] provides properties of operator F .

Lemma J.2 Let Assumption J.1 hold. Then

The operator F is monotone, i.e. satisfies Assumption 1.1.

The operator F is L -smooth, i.e. satisfies Assumption 1.2.

F ( z ∗ ) = 0 for any global saddle point z ∗ ∈ R n + m of the function f .

We assume that inexact approximation of Jacobian satisfies Assumption 2.1.

J.2 The method

Since we consider unconstrained minmax optimization, Step 2 of VIJI simplifies to v k +1 = z 0 + s k and the subproblem (11) changes to

Usually, to find the solution of this subproblem it is necessary to run some addition subroutine. Following the work [71], we introduce the following approximate condition:

where τ ∈ (0 , 1) is a tolerance parameter.

The version of VIJI for unconstrained min-max problems is referred to as VIJI-MinMax and is detailed in Algorithm 4. This subproblem can solved by strategy proposed in [71], resulting in O ( ( n + m ) ω ε -2 / 3 +( n + m ) 2 ε -2 / 3 log log(1 /ε ) ) complexity, where ω ≈ 2 . 3728 is the matrix multiplication constant.

J.3 Convergence analysis

The following theorem provides convergence rate for Algorithm 4.

Theorem J.3 Let Assumptions J.1, 2.1 hold. Then, after T ≥ 1 iterations of VIJI-MinMax with parameters η = 5 , we get the following bound

Now, let us introduce additional assumption on δ , to obtain the convergence with optimal rate O ( ε -2 / 3 ) .

Theorem J.4 Let Assumptions J.1 hold. Let

hold for iterates { z k , v k } generated by Algorithm 4, where

Then, after T ≥ 1 iterations of VIJI-MinMax with parameters η = 5 , we get the following bound

Note, that it is also possible to change adaptive strategy for λ k in this case to 1 16 ≤ 3 2 L ∥ z k -v k ∥ ≤ 1 12 (by introducing parameter β as in Algorithm 1) to eliminate the dependence on δ from the step size while achieving the same convergence rate.

Finally, let us show, that we mathch the convergence of [71] under the same assumptions on Jacobian's inexactness and subproblem's solution.

Assumption J.5 ([71]) Let

hold for iterates { z k , v k } generated by Algorithm 4, where

where τ 0 < L 4 .

Next, we show, that (69) follows from (70).

Let us consider two cases. If ∥ z k -v k ∥ ≤ 1 , then from (70), we get δ k ≤ τ 0 ∥ z k -v k ∥ ≤ L 2 ∥ z k -v k ∥ . Otherwise, if ∥ z k -v k ∥ ≥ 1 , we obtain from (68)

Using our assumptions, we get

Next,

Thus, the assumptions of Theorem J.4 hold, and Algorithm 4 achieves O ( ε -2 / 3 ) convergence.

J.4 Proof of Theorem J.3

Again, as in the proof of Theorem 3.2, we introduce the Lyapunov function

Note that the scalar product ⟨ s k , v -z 0 ⟩ can be omitted (see the proof of [71, Theorems 3.1, 4.1]), which would also eliminate the somewhat redundant Step 2 of Algorithm 4. However, we chose to retain it, as it does not affect the method's performance. We retain

Proof. Following the proof of Lemma B.1, we arrive at (35), where the proofs begin to slightly diverge due to the changes in the subproblem. At this juncture, we have:

Then,

Next,

where the last inequality is due to the following choice of η = 10 and λ : 1 16 ≤ λ k ( L ∥ z k -v k ∥ + δ ) ≤ 1 12 . Then,

Plugging (73) into (72) yields that

Next, Lemma B.2 can be applied. Next Lemma is a counterpart of Lemma B.3 for the current choice of λ .

where x ∗ ∈ X denotes the weak solution to the VI.

Proof. Without loss of generality, we assume that z 0 = z ∗ . We have

̸

By the Hölder inequality, we have

Putting these pieces together yields that

Plugging this into the above inequality yields that

By applying Young's ineqaulity and the fact that z 0 = v 0 , we get

Thus, we have ∥ v k -z 0 ∥ ≤ 2 ∥ z ∗ -z 0 ∥ . From Lemma B.2 we also have that ∥ z k -v k ∥ ≤ 2 ∥ z ∗ -z 0 ∥ . Thus,

Lemma B.2 also implies

□

By Proposition [71, Proposition 2.9], we have

Putting these pieces together yields

This together with Lemma J.7 yields

Since ∥ z k -z ∗ ∥ ≤ β for all k ≥ 0 , we have ∥ ˜ z T -z ∗ ∥ ≤ β . By the definition of the restricted gap function, we have

Therefore, we conclude from the above inequality that there exists some T > 0 such that the output ˆ z satisfies that gap(ˆ z, β ) ≤ ϵ .

J.5 Proof of Theorem J.4

Lemma (J.6) hold with slight modification in λ k adaptive strategy. Lemma B.2 also holds. Next we need slight modification of Lemma B.3 for the current choice of λ .

where x ∗ ∈ X denotes the weak solution to the VI.

̸

Proof. Without loss of generality, we assume that z 0 = z ∗ . We have

By the Hölder inequality, we have

Putting these pieces together yields that

Plugging this into the above inequality yields that

Next, directly following the steps of proof of Theorem J.3, we get the following rate

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

We proposed an algorithm for variational inequalities with inexact Jacobians in Section 3, supported its optimality by establishing the lower bound in Section 4, introduced QuasiNewton updates in Section 5, studied strongly-convex case in Section 6 and discussed tensor generalization in Section 7. Section 8 provides experimental results to validate our proposed approach.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: the assumptions used in this paper are classical for methods for variational inequalities [69, 21]. Our methods include the solution of auxilarary subproblem (as other globally convergent second-order methods for VIs [69, 21]), which requires subsolver or addditional subroutine, what we explicitly stated. Moreover, we introduced the way to reduce the complexity and provided the computational cost of the proposed procedure.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For all theoretical results, we provide proofs in the main paper and Appendix. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: our algorithm is listed in the paper (Algorithm 1) and the details are fully described (Sections 3 and 5). The expereimantal details for reproducibility are describes in Section 8 and Appendix I. The code is attached.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: we describe experimental details in Section 8 and Appendix I. The code is attached, where all the methods implemented as PyTorch optimizers. Our experiments do not include datasets, but we provide the explicit definition of test functions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we describe experimental details in Section 8 and Appendix I. The code is attached.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: we test our optimizer on deterministic functions. Hence, the iteration of the methods are deterministic and provide deterministic solution.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: we describe compute resourses in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: see Section 8.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops 2nd-order methods for variational inequalities under inexact Jacobian information. Existing methods requires exact derivative information which can be prohibitively expensive for large-problems limiting their practicality. To address this issue, the paper analyzes the impact of inexact derivatives on the convergence rate. When the objective is smooth and monotone, the paper establishes a lower-bound that depends upon the level inaccuracy and develop an optimal algorithm that attains this bound. The paper develops several inexact schemes based on a Quasi-Newton approximation that achieve a gloabl sublinear convergence rates.\n\nSTRENGTHS:\n1.) The paper is the first to systematically explore the effect the impact of inexact Jacobian information on the global convergence rate of 2nd-order methods for variational inequalities. In particular, the established bounds demonstrate the precise impact of the level inexactness on the speed of convergence. \n\n2.) In the important special case when F is smooth and monotone, the paper provides a lower-bound on the convergence rate that explicitly depends upon the level of inexactness. They then provide an algorithm that achieves this lower-bound. Thus, the analysis in the paper is quite tight.\n\n3.) On a practical level the authors develop inexact 2nd-order schemes that achieve global sublinear convergence based on quasi-Newton schemes. \n\n4.) The theory is supported by numerical simulation and comparision with existing methods.\n\n5.) The paper is clear and easy to read.\n\nOverall, I think the paper provides a strong contribution to optimization theory and should be accepted.\n\nWEAKNESSES:\nThe main weakness that I can see is that the paper focuses on deterministic inaccuracy in the Jacobian. Most practical ML algorithms use stochastic derivative information. Moreover, the batch size doesn't typically increase as the algorithm progresses, so generally only a noisy neighborhood of the optimum is reached. Thus, it is not immediately clear how relevant the algorithms and analysis in the paper are to this setting.\n\nQUESTIONS:\n1) If the algorithms in this paper only have access to stochastic Jacobian information  and assuming they don't minibatch, do the authors believe the conclusions of the paper still hold modulo the fact that convergence will only occur up to the noise level in the Jacobian? If the answer is yes, then I don't think the weakness above is that significant. \n\n2)  For the experiments results in Figure 1, how well do the proposed methods perform in terms of wall-clock time? Although they require more iterations then Perseus, I would expect them to be faster due to their use of inexactness. I think inclusion of this information in a table would make a nice addition to the paper.\n\nLIMITATIONS:\nI think the authors have addressed the main limitations of the work at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the impact of inexactness Jacobian in second-order methods for solving variational inequalities.\n\nThis article is very informative， including a novel inexact second-order framework which achieves a convergence rate that matches the proposed lower bound, feasible solution to the sub problem, combination of quasi-Newton methods, and the generalization to the higher-order inexactness.\n\nOverall, I found this paper well-written and the results are neat and  interesting.\n\nSTRENGTHS:\n1. This paper gives a novel second-order methods that allows the inexactness of the Jacobian. The convergence rate of the proposed methods  matches the lower bound given by the authors, which is a nice and neat result.\n\n2. This paper involves solving sub VI problem, and implementation to quasi-Newton methods, which is practical.\n\n3. This paper is generally well-written.\n\nWEAKNESSES:\nThere may lack some discussion on the related work:\n\nIn terms of the inexact Jacobian: It is very necessary to discuss the relation with one that considers the inexact Jacobian for convex optimization, especially the technical difficulties over them [3, 4].\n\nIn terms of the quasi-Newton methods for solving VI, the author may need to discuss [A] which does not require the symmetric QN update for saddle point problems.\n\nI will also be happy to see discussion on the **technical difficulties** in comparison with optimal second-order methods for solving VI, especially [59], where Algorithm 1 use very similar framework to.\n\n\n[A]. Liu C, Luo L. Quasi-Newton methods for saddle point problems. NeurIPS 2022.\n\nQUESTIONS:\nSee weakness.\n\nIn related work for convex optimization [3, 4], they also show the influence of inexact gradient on the convergence, it seems also important to discuss the behavior when $F(\\cdot)$ is inexact.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper presents a new second-order algorithm for variational inequalities that is applicable to inexact Jacobian. The complete convergence analysis and a theoretical lower bound are provided in the paper.

STRENGTHS:
1. A novel second-order algorithm is proposed for variational inequalities that is able to deal with inexact Jacobian.
2. Solid convergence analysis and a theoretical lower bounded are established.

WEAKNESSES:
The advantages of the proposed method over some related works are not stated clearly (see question 1).

QUESTIONS:
1. What is the advantage of the proposed method over [60] since the convergence rate is not improved? To my understanding, the assumption of Jacobian inexactness in [60] is not suitable for quasi-newton. But why "suitable to quasi-newton" can be claimed as an advantage? 
2. Will inexact Jacobian causes more iterations to converge than exact Jacobian, which eventually leads to a higher complexity than exact Jacobian?

LIMITATIONS:
I did not see any negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13