Gradient Descent Ascent for Min-Max Problems on Riemannian Manifolds

1 INTRODUCTION . In the paper , we study a class of useful non-convex minimax ( a.k.a . min-max ) problems on the Riemannian manifoldM with the definition as : min x∈M max y∈Y f ( x , y ) , ( 1 ) where the function f ( x , y ) is µ-strongly concave in y but possibly nonconvex in x . Here Y ⊆ Rd is a convex and closed set . f ( · , y ) : M → R for all y ∈ Y is a smooth but possibly nonconvex real-valued function on manifoldM , and f ( x , · ) : Y → R for all x ∈ M a smooth and ( strongly ) concave real-valued function . In this paper , we mainly focus on the stochastic minimax optimization problem f ( x , y ) : = Eξ∼D [ f ( x , y ; ξ ) ] , where ξ is a random variable that follows an unknown distribution D. In fact , the problem ( 1 ) is associated to many existing machine learning applications : 1 ) . Robust Training DNNs over Riemannian manifold . Deep Neural Networks ( DNNs ) recently have been demonstrating exceptional performance on many machine learning applications . However , they are vulnerable to the adversarial example attacks , which show that a small perturbation in the data input can significantly change the output of DNNs . Thus , the security properties of DNNs have been widely studied . One of secured DNN research topics is to enhance the robustness of DNNs under the adversarial example attacks . To be more specific , given training data D : = { ξi = ( ai , bi ) } ni=1 , where ai ∈ Rd and bi ∈ R represent the features and label of sample ξi respectively . Each data sample ai can be corrupted by a universal small perturbation vector y to generate an adversarial attack sample ai + y , as in ( Moosavi-Dezfooli et al. , 2017 ; Chaubey et al. , 2020 ) . To make DNNs robust against adversarial attacks , one popular approach is to solve the following robust training problem : min x max y∈Y 1 n n∑ i=1 ` ( h ( ai + y ; x ) , bi ) , ( 2 ) where y ∈ Rd denotes a universal perturbation , and x is the weight of the neural network ; h ( · ; x ) is the the deep neural network parameterized by x ; and ` ( · ) is the loss function . Here the constraint Y = { y : ‖y‖ ≤ ε } indicates that the poisoned samples should not be too different from the original ones . Recently , the orthonormality on weights of DNNs has gained much interest and has been found to be useful across different tasks such as person re-identification ( Sun et al. , 2017 ) and image classification ( Xie et al. , 2017 ) . In fact , the orthonormality constraints improve the performances of DNNs ( Li et al. , 2020 ; Bansal et al. , 2018 ) , and reduce overfitting to improve generalization ( Cogswell et al. , 2015 ) . At the same time , the orthonormality can stabilize the distribution of activations over layers within DNNs ( Huang et al. , 2018 ) . Thus , we consider the following robust training problem over the Stiefel manifoldM : min x∈M max y∈Y 1 n n∑ i=1 ` ( h ( ai + y ; x ) , bi ) . ( 3 ) When data are continuously coming , we can rewrite the problem ( 3 ) as follows : min x∈M max y∈Y Eξ [ f ( x , y ; ξ ) ] , ( 4 ) where f ( x , y ; ξ ) = ` ( h ( a+ y ; x ) , b ) with ξ = ( a , b ) . 2 ) . Distributionally Robust Optimization over Riemannian manifold . Distributionally robust optimization ( DRO ) ( Chen et al. , 2017 ; Rahimian & Mehrotra , 2019 ) is an effective method to deal with the noisy data , adversarial data , and imbalanced data . At the same time , the DRO in the Riemannian manifold setting is also widely applied in machine learning problems such as robust principal component analysis ( PCA ) . To be more specific , given a set of data samples { ξi } ni=1 , the DRO over Riemannian manifoldM can be written as the following minimax problem : min x∈M max p∈S { n∑ i=1 pi ` ( x ; ξi ) − ‖p− 1 n ‖2 } , ( 5 ) where p = ( p1 , · · · , pn ) , S = { p ∈ Rn : ∑n i=1 pi = 1 , pi ≥ 0 } . Here ` ( x ; ξi ) denotes the loss function over Riemannian manifold M , which applies to many machine learning problems such as PCA ( Han & Gao , 2020a ) , dictionary learning ( Sun et al. , 2016 ) , DNNs ( Huang et al. , 2018 ) , structured low-rank matrix learning ( Jawanpuria & Mishra , 2018 ) , among others . For example , the task of PCA can be cast on a Grassmann manifold . To the best of our knowledge , the existing explicitly minimax optimization methods such as gradient descent ascent method only focus on the minimax problems in Euclidean space . To fill this gap , in the paper , we propose a class of efficient Riemannian gradient descent ascent algorithms to solve the problem ( 1 ) via using general retraction and vector transport . When the problem ( 1 ) is deterministic , we propose a new deterministic Riemannian gradient descent ascent algorithm . When the problem ( 1 ) is stochastic , we propose two efficient stochastic Riemannian gradient descent ascent algorithms . Our main contributions can be summarized as follows : 1 ) We propose a novel Riemannian gradient descent ascent ( RGDA ) algorithm for the deterministic minimax optimization problem ( 1 ) . We prove that the RGDA has a sample complexity of O ( κ2 −2 ) for finding an -stationary point . 2 ) We also propose a new Riemannian stochastic gradient descent ascent ( RSGDA ) algorithm for the stochastic minimax optimization . In the theoretical analysis , we prove that the SRGDA has a sample complexity of O ( κ4 −4 ) . 3 ) To further reduce the sample complexity , we introduce a novel momentum variancereduced Riemannian stochastic gradient descent ascent ( MVR-RSGDA ) algorithm based on a new momentum variance-reduced technique of STORM ( Cutkosky & Orabona , 2019 ) . We prove the MVR-RSGDA achieves a lower sample complexity of Õ ( κ4 −3 ) ( please see Table 1 ) , which reaches near the best known sample complexity for its Euclidean counterparts . 4 ) Extensive experimental results on the robust DNN training over Stiefel manifold demonstrate the efficiency of our proposed algorithms . 2 RELATED WORKS . In this section , we briefly review the minimax optimization and Riemannian manifold optimization research works . 2.1 MINIMAX OPTIMIZATION . Minimax optimization recently has been widely applied in many machine learning problems such as adversarial training ( Goodfellow et al. , 2014 ; Liu et al. , 2019 ) , reinforcement learning ( Zhang et al. , 2019 ; 2020 ) , and distribution learning ( Razaviyayn et al. , 2020 ) . At the same time , many efficient min-max methods ( Rafique et al. , 2018 ; Lin et al. , 2019 ; Nouiehed et al. , 2019 ; Thekumparampil et al. , 2019 ; Lin et al. , 2020 ; Yang et al. , 2020 ; Ostrovskii et al. , 2020 ; Yan et al. , 2020 ; Xu et al. , 2020a ; Luo et al. , 2020 ; Xu et al. , 2020b ; Boţ & Böhm , 2020 ; Huang et al. , 2020 ) have been proposed for solving these minimax optimization problems . For example , Thekumparampil et al . ( 2019 ) have proposed a class of efficient dual implicit accelerated gradient algorithms to solve smooth min-max optimization . Lin et al . ( 2019 ) have proposed a class of efficient gradient decent ascent methods for non-convex minimax optimization . Subsequently , accelerated first-order algorithms Lin et al . ( 2020 ) have been proposed for minimax optimization . Xu et al . ( 2020b ) have proposed a unified singleloop alternating gradient projection algorithm for ( non ) convex- ( non ) concave minimax problems . Ostrovskii et al . ( 2020 ) have proposed an efficient algorithm for finding first-order Nash equilibria in nonconvex concave minimax problems . Xu et al . ( 2020a ) ; Luo et al . ( 2020 ) have proposed a class of fast stochastic variance-reduced GDA algorithms to solve the stochastic minimax problems . More recently , Huang et al . ( 2020 ) have presented a class of new momentum-based first-order and zeroth-order descent ascent method for the nonconvex strongly concave minimax problems . 2.2 RIEMANNIAN MANIFOLD OPTIMIZATION . Riemannian manifold optimization methods have been widely applied in machine learning problems including dictionary learning ( Sun et al. , 2016 ) , matrix factorization ( Vandereycken , 2013 ) , and DNNs ( Huang et al. , 2018 ) . Many Riemannian optimization methods were recently proposed . E.g . Zhang & Sra ( 2016 ) ; Liu et al . ( 2017 ) have proposed some efficient first-order gradient methods for geodesically convex functions . Subsequently , Zhang et al . ( 2016 ) have presented fast stochastic variance-reduced methods to Riemannian manifold optimization . More recently , Sato et al . ( 2019 ) have proposed fast first-order gradient algorithms for Riemannian manifold optimization by using general retraction and vector transport . Subsequently , based on these retraction and vector transport , some fast Riemannian gradient-based methods ( Zhang et al. , 2018 ; Kasai et al. , 2018 ; Zhou et al. , 2019 ; Han & Gao , 2020a ) have been proposed for non-convex optimization . Riemannian Adam-type algorithms ( Kasai et al. , 2019 ) were introduced for matrix manifold optimization . In addition , some algorithms ( Ferreira et al. , 2005 ; Li et al. , 2009 ; Wang et al. , 2010 ) have been studied for variational inequalities on Riemannian manifolds , which are the implicit min-max problems on Riemannian manifolds . Notations : ‖ · ‖ denotes the ` 2 norm for vectors and spectral norm for matrices . 〈x , y〉 denotes the inner product of two vectors x and y . For function f ( x , y ) , f ( x , · ) denotes function w.r.t . the second variable with fixing x , and f ( · , y ) denotes function w.r.t . the first variable with fixing y . Given a convex closed set Y , we define a projection operation on the set Y as PY ( y0 ) = arg miny∈Y 12‖y − y0‖2 . We denote a = O ( b ) if a ≤ Cb for some constant C > 0 , and the notation Õ ( · ) hides logarithmic terms . Id denotes the identity matrix with d dimension . The operation ⊕ denotes the Whitney sum . Given Bt = { ξit } Bi=1 for any t ≥ 1 , let ∇fBt ( x , y ) = 1B ∑B i=1∇f ( x , y ; ξit ) .

Summary: In this paper, the authors present and analyze a class of gradient-descent algorithms for solving min-max problems when the first (minimization) variable is constrained to live on a Riemaniann manifold. In the case when i) a retraction and an isometric transport are available on the manifold; and ii) the objective is strongly convex and smooth in the second variable, the authors show convergence rates. Experiments are performed with the setting of minimizing losses of neural nets whose weights are constrained to live in the Stiefeld manifold while an attacker of small norm perturbs the input.