Average-case Acceleration for Bilinear Games and Normal Matrices

1 INTRODUCTION . The traditional analysis of optimization algorithms is a worst-case analysis ( Nemirovski , 1995 ; Nesterov , 2004 ) . This type of analysis provides a complexity bound for any input from a function class , no matter how unlikely . However , since hard-to-solve inputs might rarely occur in practice , the worst-case complexity bounds might not be representative of the observed running time . A more representative analysis is given by the average-case complexity , averaging the algorithm ’ s complexity over all possible inputs . This analysis is standard for analyzing , e.g. , sorting ( Knuth , 1997 ) and cryptography algorithms ( Katz & Lindell , 2014 ) . Recently , a line of work ( Berthier et al. , 2020 ; Pedregosa & Scieur , 2020 ; Lacotte & Pilanci , 2020 ; Paquette et al. , 2020 ) focused on optimal methods for the optimization of quadratics , specified by a symmetric matrix . While worst-case analysis uses bounds on the matrix eigenvalues to yield upper and lower bounds on convergence , average-case analysis relies on the expected distribution of eigenvalues and provides algorithms with sharp optimal convergence rates . While the algorithms developed in this context have been shown to be efficient for minimization problems , these have not been extended to smooth games . A different line of work considers algorithms for smooth games but studies worst-case optimal methods ( Azizian et al. , 2020 ) . In this work , we combine average-case analysis with smooth games , and develop novel average-case optimal algorithms for finding the root of a linear system determined by a ( potentially non-symmetric ) normal matrix . We make the following main contributions : 1 . Inspired by the problem of finding equilibria in smooth games , we develop average-case optimal algorithms for finding the root of a non-symmetric affine operator , both under a normality assumption ( Thm . 4.1 ) , and under the extra assumption that eigenvalues of the operator are supported in a disk ( Thm . 4.2 ) . The proposed method shows a polynomial speedup compared to the worst-case optimal method , verified by numerical simulations . 2 . We make a novel connection between average-case optimal methods for optimization , and average-case optimal methods for bilinear games . In particular , we show that solving the Hamiltonian using an average-case optimal method is optimal ( Theorem 3.1 ) for bilinear games . This result complements ( Azizian et al. , 2020 ) , who proved that Polyak Heavy Ball algorithm on the Hamiltonian is asymptotically worst-case optimal for bilinear games . 2 AVERAGE-CASE ANALYSIS FOR NORMAL MATRICES . In this paper we consider the following class of problems . Definition 1 . Let A ∈ Rd×d be a real matrix and x ? ∈ Rd a vector . The non-symmetric ( affine ) operator ( NSO ) problem is defined as : Find x : F ( x ) def = A ( x−x ? ) = 0 . ( NSO ) This problem generalizes that of minimization of a convex quadratic function f , since we can cast the latter in this framework by setting the operator F = ∇f . The set of solutions is an affine subspace that we will denote X ? . We will find convenient to consider the distance to this set , defined as dist ( x , X ? ) def= min v∈X ? ‖x− v‖2 , with X ? = { x ∈ Rd |A ( x− x ? ) = 0 } . ( 1 ) In this paper we will develop average-case optimal methods . For this , we consider A and x ? to be random vectors , and a random initialization x0 . This induces a probability distribution over NSO problems , and we seek to find methods that have an optimal expected suboptimality w.r.t . this distribution . Denoting E ( A , x ? , x0 ) the expectation over these random problems , we have that average-case optimal methods they verify the following property at each iteration t min xt E ( A , x ? , x0 ) dist ( xt , X ? ) s.t . xi ∈ x0 + span ( { F ( xj ) } i−1j=0 ) , ∀i ∈ [ 1 : t ] . ( 2 ) The last condition on xt stems from restricting the class of algorithms to first-order methods . The class of first-order methods encompasses many known schemes such as gradient descent with momentum , or full-matrix AdaGrad . However , methods such as Adam ( Kingma & Ba , 2015 ) or diagonal AdaGrad ( Duchi et al. , 2011 ) are not in this class , as the diagonal re-scaling creates iterates xt outside the span of previous gradients . Although we will focus on the distance to the solution , the results can be extended to other convergence criteria such as ‖F ( xt ) ‖2 . Finally , note that the expectations in this paper are on the problem instance and not on the randomness of the algorithm . 2.1 ORTHOGONAL RESIDUAL POLYNOMIALS AND FIRST-ORDER METHODS . The analysis of first-order methods simplifies through the use of polynomials . This section provides the tools required to leverage this connection . Definition 2 . A residual polynomial is a polynomial P that satisfies P ( 0 ) = 1 . Proposition 2.1 . ( Hestenes et al. , 1952 ) If the sequence ( xt ) t∈Z+ is generated by a first-order method , then there exist residual polynomials Pt , each one of degree at most t , verifying xt − x ? = Pt ( A ) ( x0 − x ? ) . ( 3 ) As we will see , optimal average-case method are strongly related to orthogonal polynomials . We first define the inner product between polynomials , where we use z∗ for the complex conjugate of z ∈ C. Definition 3 . For P , Q ∈ R [ X ] , we define the inner product 〈· , ·〉µ for a measure µ over C as 〈P , Q〉µ def = ∫ C P ( λ ) Q ( λ ) ∗ dµ ( λ ) . ( 4 ) Definition 4 . A sequence of polynomials { Pi } is orthogonal ( resp . orthonormal ) w.r.t . 〈· , ·〉µ if 〈Pi , Pi〉µ > 0 ( resp . = 1 ) ; 〈Pi , Pj〉µ = 0 if i 6= j . 2.2 EXPECTED SPECTRAL DISTRIBUTION . Following ( Pedregosa & Scieur , 2020 ) , we make the following assumption on the problem family . Assumption 1. x0 − x ? is independent ofA , and E ( x0 , x ? ) [ ( x0 − x ? ) ( x0 − x ? ) > ] = R 2 d Id . We will also require the following definitions to characterize difficulty of a problem class . Let { λ1 , . . . , λd } be the eigenvalues of a matrix A ∈ Rd×d . We define the empirical spectral distribution ofA as the probability measure µ̂A ( λ ) def = 1d ∑d i=1δλi ( λ ) , ( 5 ) where δλi is the Dirac delta , a distribution equal to zero everywhere except at λi and whose integral over the entire real line is equal to one . Note that with this definition , ∫ D dµ̂A ( λ ) corresponds to the proportion of eigenvalues in D. WhenA is a matrix-valued random variable , µA is a measure-valued random variable . As such , we can define its expected spectral distribution µA def = EA [ µ̂A ] , ( 6 ) which by the Riesz representation theorem is the measure that verifies ∫ f dµ = EA [ ∫ f dµA ] for all measureable f . Surprisingly , the expected spectral distribution is the only required characteristic to design optimal algorithms in the average-case . 2.3 EXPECTED ERROR OF FIRST-ORDER METHODS . In this section we provide an expression for the expected convergence in terms of the residual polynomial and the expected spectral distribution introduced in the previous section . To go further in the analysis , we have to assume thatA is a normal matrix . Assumption 2 . The ( real ) random matrixA is normal , that is , it verifiesAA > = A > A . Normality is equivalent to A having the spectral decomposition A = UΛU∗ , where U is unitary , i.e. , U∗U = UU∗ = I . We now have everything to write the expected error of a first-order algorithm applied to ( NSO ) . Theorem 2.1 . Consider the application of a first-order method associated to the sequence of polynomials { Pt } ( Proposition 2.1 ) on the problem ( NSO ) . Let µ be the expected spectral distribution ofA . Under Assumptions 1 and 2 , we have E [ dist ( xt , X ? ) ] = R2 ∫ C\ { 0 } |Pt|2 dµ , ( 7 ) Before designing optimal algorithms for certain specific distributions , we compare our setting with the average-case accelerating for minimization problems of Pedregosa & Scieur ( 2020 ) , who proposed optimal optimization algorithms in the average-case . 2.4 DIFFICULTIES OF FIRST-ORDER METHODS ON GAMES AND RELATED WORK . This section compares our contribution with the existing framework of average-case optimal methods for quadratic minimization problems . Definition 5 . Let H ∈ Rd×d be a random symmetric positive-definite matrix and x ? ∈ Rd a random vector . These elements determine the following random quadratic minimization problem minx∈Rd { f ( x ) def = 1 2 ( x−x ? ) > H ( x−x ? ) } . ( OPT ) As in our paper , Pedregosa & Scieur ( 2020 ) find deterministic optimal first-order algorithms in expectation w.r.t . the matrix H , the solution x ? , and the initialization x0 . Since they work with problem ( OPT ) , their problem is equivalent to ( NSO ) with the matrix A = H . However , they have the stronger assumption that the matrix is symmetric , which implies being normal . The normality assumption is restrictive in the case of game theory , as they do not always naturally fit such applications . However , this set is expressive enough to consider interesting cases , such as bilinear games , and our experiments show that our findings are also consistent with non-normal matrices . Using orthogonal residual polynomials and spectral distributions , they derive the explicit formula of the expected error . Their result is similar to Theorem 2.1 , but the major difference is the domain of the integral , a real positive line in convex optimization , but a shape in the complex plane in our case . This shape plays a crucial role in the rate of converge of first-order algorithms , as depicted in the work of Azizian et al . ( 2020 ) ; Bollapragada et al . ( 2018 ) . In the case of optimization methods , they show that optimal schemes in the average-case follow a simple three-term recurrence arising from the three-term recurrence for residual orthogonal polynomials for the measure λµ ( λ ) . Indeed , by Theorem 2.1 the optimal method corresponds to the residual polynomials minimizing 〈P , P 〉µ , and the following result holds : Theorem 2.2 . ( Fischer , 1996 , §2.4 ) When µ is supported in the real line , the residual polynomial of degree t minimizing 〈P , P 〉µ is given by the degree t residual orthogonal polynomial w.r.t . λµ ( λ ) . However , the analogous result does not hold for general measures in C , and hence our arguments will make use of the following Theorem 2.3 instead , which links the residual polynomial of degree at most t that minimizes 〈P , P 〉µ to the sequence of orthonormal polynomials for µ. Theorem 2.3 . [ Theorem 1.4 of Assche ( 1997 ) ] Let µ be a positive Borel measure in the complex plane . The minimum of the integral ∫ C |P ( λ ) | 2 dµ ( λ ) over residual polynomials P of degree lower or equal than t is uniquely attained by the polynomial P ? ( λ ) = ∑t k=0 φk ( λ ) φk ( 0 ) ∗∑t k=0 |φk ( 0 ) |2 , with optimal value ∫ C |P ? ( λ ) |2 dµ ( λ ) = 1∑t k=0 |φk ( 0 ) |2 , ( 8 ) where ( φk ) k is the orthonormal sequence of polynomials with respect to the inner product 〈· , ·〉µ . In the next sections we consider cases where the optimal scheme is identifiable .

Summary: The paper combines two recent areas of interest in optimization for machine learning: average-case analysis for acceleration and learning in bilinear games.  Average-case optimal methods are proposed for some distributions on eigenvalues for the Jacobian of the games vector-field. Also, a connection is made between average-case optimal methods in bilinear games and optimally solving the Hamiltonian.