Learning A Minimax Optimizer: A Pilot Study

1 INTRODUCTION . Many popular applications can be formulated into solving continuous minimax optimization , such as generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) , distributionally robust learning ( Globerson & Roweis , 2006 ) , domain adaptation ( Ganin & Lempitsky , 2014 ) , distributed computing ( Shamma , 2008 ; Mateos et al. , 2010 ) , privacy protection ( Wu et al. , 2018 ; 2020 ) , among many more . This paper studies such problems : we consider a cost function f : Rm × Rn → R and the min-max game minxmaxy f ( x , y ) . We aim to find the saddle point ( x∗ , y∗ ) of f : f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) , ∀ ( x , y ) ∈ X × Y , ( 1 ) where X ⊂ Rm and Y ⊂ Rn . If X = Rm and Y = Rn , ( x∗ , y∗ ) is called a global saddle point ; if X × Y is a neighborhood near ( x∗ , y∗ ) , ( x∗ , y∗ ) is a local saddle point . The main challenge to solve problem ( 1 ) is the unstable dynamics of iterative algorithms . Simplest algorithms such as gradient descent ascent ( GDA ) can cycle around the saddle point or even diverge ( Benaım & Hirsch , 1999 ; Mertikopoulos et al. , 2018b ; Lin et al. , 2019 ) . Plenty of works have been developed recently to address this issue ( Daskalakis et al. , 2018 ; Daskalakis & Panageas , 2018 ; Liang & Stokes , 2019 ; Mertikopoulos et al. , 2018a ; Gidel et al. , 2018 ; Mokhtari et al. , 2019 ) . However , the convergence is still sensitive to the parameters in these algorithms . Even if the cost function is only changed by scaling , those parameters have to be re-tuned to ensure convergence . A recent trend of learning to optimize ( L2O ) parameterizes training algorithms to be learnable from data , such that the meta-learned optimizers can be adapted to a special class of functions and outperform general-purpose optimizers . That is particularly meaningful , when one has to solve a large number of yet similar optimization problems repeatedly and quickly . Specifically , for existing L2O methods that operate in the space of continuous optimization , almost all of them solve some ∗Equal Contribution . 1 minimization problem ( Andrychowicz et al. , 2016 ; Chen et al. , 2017 ; Li & Malik , 2016 ) , leveraging an LSTM or a reinforcement learner to model their optimizer . Different from classic optimization results that often provide worst-case convergence , most L2O methods have little or no convergence guarantees , especially on problem or data instances distinct from what is seen in training , leaving their generalizability in practice questionable ( Heaton et al. , 2020 ) . Motivated by L2O ’ s success in learning efficient minimization solvers from data , this paper seeks to answer : whether we could accomplish strong minimax L2O solvers as well ; and if yes , how generalizable they could be ? As it might look straightforward at first glance , such extension is highly nontrivial due to facing several unique challenges . Firstly , while continuous minimization has a magnitude of mature and empirically stable solvers , for general minimax optimization , even state-of-the-art analytical algorithms can exhibit instability or even divergence . To the best of our knowledge , most state-of-theart convergence analysis of minimax optimization is built on the convex-concave assumption ( Gidel et al. , 2018 ; Mokhtari et al. , 2019 ; Ryu et al. , 2019 ) , and some recent works relax the assumption to nonconvex-concave ( Lin et al. , 2019 ; 2020 ) . Convergence for general minimax problems is still open . That makes a prominent concern on whether a stable minimax L2O is feasible . Secondly , given the two groups of min and max variables simultaneously , it is unclear to what extent their optimization strategies can be modeled and interact within one unified framework – a new question that would never be met in minimization . Thirdly , the noisy and sometimes cyclic dynamics of minimax optimization will provide noisier guidance ( e.g. , reward ) to L2O ; not to say that , it is not immediately clear how to define the reward : for minimization , the reward is typically defined as the negative cumulative objective values along the history ( Li & Malik , 2016 ) . However , for minimax optimization the objective can not simply be decreased or increased monotonically . Contribution : This paper is a pilot study into minimax L2O . We start by establishing the first dedicated minimax L2O framework , called Twin-L2O . It is composed of two LSTMs sharing one objective-based reward , separately responsible for updating min and max variables . By ablations of the design options , we find this decoupled design facilitate meta-learning most , particularly when the min and max updates are highly non-symmetric . We demonstrate the superior convergence of Twin-L2O on several testbed problems , compared against a number of analytical solvers . On top of that , we further investigate how to enhance the generalizability of the learned minimax solver1 , and discuss two complementary alternatives with experimental validations . The first alternative is an empirical toolkit that is applicable for general minimax L2O . We introduce curriculum learning to training L2O models for the first time , by recognizing that not all problem instances are the same difficult to learn to solve . After plugging in that idea , we show that Twin-L2O can be trained to stably solve a magnitude more problem instances ( in terms of parameter varying range ) . The second alternative explores a theoretical mechanism called safeguarding , particularly for the important special case of convex-concave problems . When solving a testing instance , safeguarding identifies when an L2O failure would occur and provides an analytical fall-back option ( Diakonikolas , 2020 ) . That guarantees convergence for convex-concave problems and , in practice , converges faster even when the problem parameters are drawn from a different distribution from training . 2 RELATED WORK . 2.1 MINIMAX OPTIMIZATION . Following ( Neumann , 1928 ) , the problem ( 1 ) has been studied for decades due to its wide applicability . Simultaneous gradient descent ( SimGD ) or gradient descent ascent ( GDA ) ( Nedić & Ozdaglar , 2009 ; Du & Hu , 2019 ; Jin et al. , 2019 ; Lin et al. , 2019 ) is one of the simplest minimax algorithms , conducting gradient descent over variable x and gradient ascent over variable y . However , the dynamics of SimGD or GDA can converge to limit cycles or even diverge ( Benaım & Hirsch , 1999 ; Mertikopoulos et al. , 2018b ; Lin et al. , 2019 ) . To address this issue , Optimistic gradient descent ascent ( OGDA ) simply modifies the dynamics of GDA and shows more stable performance ( Daskalakis et al. , 2018 ; Daskalakis & Panageas , 2018 ; Liang & Stokes , 2019 ; Mertikopoulos et al. , 2018a ; Gidel et al. , 2018 ; Mokhtari et al. , 2019 ) . OGDA attracts more attention because of its empirical success in training GANs . ( Ryu et al. , 2019 ) theoretically studies OGDA by analyzing its continuous time dynamic and 1We differentiate the usages of two terms : parameters and variables , throughout the paper . For example , minu maxv ax 2 − by2 , we call a , b parameters and x , y variables . For simplicity , this paper only discusses the L2O generalizability when the testing instances ’ parameter distribution differs from the training . 2 proposes Anchored simultaneous gradient descent that shows good performance . Follow-the-Ridge ( Wang et al. , 2019 ) also addresses the limit cycling problem by introducing second-order information into the dynamic of GDA . Lately , K-Beam ( Hamm & Noh , 2018 ) stabilizes the convergence of GDA by duplicating variable y , yielding strong performance . At each iteration , it performs gradient ascent independently on K copies of y and greedily chooses the copy that leads to a large function value f , then it updates x based on the selected copies . 2.2 LEARNING TO OPTIMIZE . As a special instance of meta-learning , L2O has been studied in multiple contexts , with continuous optimization being one of its main playgrounds so far . The first L2O framework is introduced in ( Andrychowicz et al. , 2016 ) , where both the optimizee ’ s gradients and loss function values are formulated as the input features for an RNN optimizer . Due to the enormous number of parameters , a coordinate-wise design of RNN optimizer is adopted , where all optimization coordinates share the same updating strategy . ( Li & Malik , 2016 ) uses the gradient history and objective values as observations and step vectors as actions in their reinforcement learning framework . ( Chen et al. , 2017 ) leverages RNN to train a meta-optimizer to optimize black-box functions . Two effective training tricks , random scaling and objective convexifying , are presented in ( Lv et al. , 2017 ) . Wichrowska et al . ( 2017 ) presents an optimizer of multi-level hierarchical RNN architecture augmented with additional architectural features . Li et al . ( 2020 ) introduces a Jacobian regularization to L2O and enhances the domain adaptation performance of optimizees . Chen et al . ( 2020a ) proposes several improved training techniques to stabilize L2O training and ameliorate performance . You et al . ( 2020 ) ; Chen et al . ( 2020b ; c ) extend the application scope of L2O into various practical problems such as graph neural network training , domain generalization , and noisy label training . The above works address continuous minimization problems using single optimizer models . One exception , ( Cao et al. , 2019 ) , extends L2O to solving Bayesian swarm optimization . The author presents a novel architecture where multiple LSTMs jointly learn iterative update formulas for the swarm of particles , coordinated by attention mechanisms . We also notice that two recent efforts ( Jiang et al. , 2018 ; Xiong & Hsieh , 2020 ) introduce L2O to adversarial training , a renowned application of minimax optimization . However , both of them merely utilize L2O to solve the inner minimization of their minimax problems ( i.e. , generating attacks ) , while the outer maximization is still solved analytically . Neither of the two directly solves the full minimax optimization . 3 METHOD . 3.1 MAIN FRAMEWORK : TWIN LEARNABLE OPTIMIZERS ( TWIN-L2O ) . The main L2O framework we proposed is named Twin-L2O , where we use two learnable optimizers to alternate between min and max updates . Our design adopts the basic idea of ( Andrychowicz et al. , 2016 ) to use Long Short-Term Memory ( LSTM ) to model learnable optimizers , for solving target problems known as optimizees . At each step , LSTM outputs the update of the optimizee variables . The LSTM inputs are typically the current zero-order or first-order information of the optimizee ( Andrychowicz et al. , 2016 ; Lee & Choi , 2018 ) , plus the historic optimization trajectory information . In Twin-L2O , two LSTMs separately update x and y and record historical trajectory information of their own variables respectively . Formally , we consider the minimax problem minxmaxy f ( x , y ) . We use two LSTM optimizers , LSTM-Min and LSTM-Max , to updates the min variable x and the max variable y respectively . LSTM-Min is parameterized by φmin and LSTM-Max is parameterized by φmax . At each iteration t , Twin-L2O updates x and y in turns and yields the following rule : xt+1 =xt + ∆xt , where ( ∆xt , hmint+1 ) = LSTM-Min ( [ ∇xf ( xt , yt ) , ∇yf ( xt , yt ) ] , hmint , φmin ) , yt+1 =yt + ∆yt , where ( ∆yt , hmaxt+1 ) = LSTM-Max ( [ ∇yf ( xt+1 , yt ) , ∇xf ( xt+1 , yt ) ] , hmaxt , φmax ) , ( 2 ) where hmint and h max t are the historical trajectory information of LSTM-Min and LSTM-Max at time step t. This formulation is inspired by the SimGD/GDA-style algorithms ( Nedić & Ozdaglar , 2009 ; Du & Hu , 2019 ; Jin et al. , 2019 ; Lin et al. , 2019 ) that conduct simultaneous/alternative gradient descent over x and ascent over y . Figure A4 ( Appendix A1 ) conceptually illustrates the framework . The next question is to design the L2O reward . To train the LSTM optimizers , the loss function is often to penalize some type of cost , accumulated along the optimization trajectory for a horizon of T 3 steps ( also known as the unrolling length for LSTM ( Sherstinsky , 2018 ) ) L ( φmin , φmax ) = Ef [ T∑ t=1 wtR ( f , xt , yt ) ] , ( 3 ) wt is chosen to be all 1 following the basic setting in ( Andrychowicz et al. , 2016 ) , that might be tuned for better performance in future work . As a key design option , R ( f ) represents the reward to guide the L2O training . In existing L2O methods for continuous minimization ( Andrychowicz et al. , 2016 ; Lv et al. , 2017 ) , R ( f ) is usually simply set to R ( f , xt ) ) = f ( xt ) to encourage fast decrease of objective values over time . To extend this existing reward to the minimax scenario , we can not directly penalize the overall objective function value either way , since the min and max objectives are entangled . Also , different from pure minimization problems , the Twin-L2O updates ( 2 ) consist of two alternating steps governed by two different LSTM optimizers : each accounts for its own subproblem goal ( min or max updates ) , but the two also have to collaborate to explore/exploit the minimax landscape . We specifically design the following reward that implicitly addresses the above issue by setting a new reward function : L ( φmin , φmax ) = Ef [ T∑ t=1 { [ f ( xt , yt−1 ) − f ( xt , yt ) ] + [ f ( xt , yt−1 ) − f ( xt−1 , yt−1 ) ] } ] . ( 4 ) Analysis of the reward design In Eqn . 4 , the first and second terms always characterize two consecutive min and max updates . In more details , the value of f ( xt , yt ) −f ( xt , yt−1 ) solely reflects how effectively the t-step max update increases the objective f , while f ( xt , yt−1 ) − f ( xt−1 , yt−1 ) reflects the effectiveness of t-step min update in decreasing the objective f . Our goal is then to maximize the weighted accumulated sum for f ( xt , yt ) −f ( xt , yt−1 ) , while minimizing the weighted accumulated sum for f ( xt , yt ) − f ( xt , yt−1 ) , t = 1 , 2 , ... , T . Combining the two sub-goals together ( with a sign change to turn max into min ) yields our reward . One may also alternatively interpret Eqn . 4 as penalizing the loss change from f ( xt , yt ) along both x and y updating directions , which would encourage yielding stationary points . For the reward design , we provide a more detailed discussion in Appendix A6 . Specifically , we provide a comparison between the objective-based reward in Eqn . 4 , and another possible gradientbased reward . The latter was found to be ineffective in solving the problems presented in Section 4 . Rationale of the framework selection Another important design question is to what extent learning the min and max updates should be ( dis ) entangled : on the one hand , the two steps obviously interact with each other as they jointly explore the minimax landscape ; on the other hand , min and max steps commonly have asymmetric difficulty levels , that have been leveraged by previous algorithms . For example , ( Hamm & Noh , 2018 ) demonstrates the failure of alternating gradient descent in minimax optimization due to the multiple solution discontinuity of the inner maximization , and addresses that by simultaneously tracking K candidate solutions for the max step , while the outer minimization remains to take one descent step . Besides the joint reward ( 4 ) , the default Twin-L2O design leverages two independent LSTMs in Eqn . ( 2 ) , each dedicatedly handling min or max updates . In comparison , we also consider two other more `` entangled '' ways : ( a ) fully entangling the two optimizers , i.e . using one LSTM to simultaneously generate min and max outputs ; ( b ) weakly entangling the two optimizers , by using two LSTMs sharing weights , yet allowing either to maintain its own temporal hidden states . Our ablation experiments ( see Section 4.1 ) find that the default decoupled design in Eqn . ( 2 ) seems to facilitate the L2O learning most .

Summary: Classical iterative minimax optimization algorithms display the unstable dynamics. Their convergence is often sensitive to the parameters and needs to be re-tuned for different problems to ensure convergence. Therefore, there is a practical motivation to develop L2O for minimax problems, so that we could meta-learn and adapt optimization rules to a special class of functions.