Learning A Minimax Optimizer: A Pilot Study

1 INTRODUCTION . Many popular applications can be formulated into solving continuous minimax optimization , such as generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) , distributionally robust learning ( Globerson & Roweis , 2006 ) , domain adaptation ( Ganin & Lempitsky , 2014 ) , distributed computing ( Shamma , 2008 ; Mateos et al. , 2010 ) , privacy protection ( Wu et al. , 2018 ; 2020 ) , among many more . This paper studies such problems : we consider a cost function f : Rm × Rn → R and the min-max game minxmaxy f ( x , y ) . We aim to find the saddle point ( x∗ , y∗ ) of f : f ( x∗ , y ) ≤ f ( x∗ , y∗ ) ≤ f ( x , y∗ ) , ∀ ( x , y ) ∈ X × Y , ( 1 ) where X ⊂ Rm and Y ⊂ Rn . If X = Rm and Y = Rn , ( x∗ , y∗ ) is called a global saddle point ; if X × Y is a neighborhood near ( x∗ , y∗ ) , ( x∗ , y∗ ) is a local saddle point . The main challenge to solve problem ( 1 ) is the unstable dynamics of iterative algorithms . Simplest algorithms such as gradient descent ascent ( GDA ) can cycle around the saddle point or even diverge ( Benaım & Hirsch , 1999 ; Mertikopoulos et al. , 2018b ; Lin et al. , 2019 ) . Plenty of works have been developed recently to address this issue ( Daskalakis et al. , 2018 ; Daskalakis & Panageas , 2018 ; Liang & Stokes , 2019 ; Mertikopoulos et al. , 2018a ; Gidel et al. , 2018 ; Mokhtari et al. , 2019 ) . However , the convergence is still sensitive to the parameters in these algorithms . Even if the cost function is only changed by scaling , those parameters have to be re-tuned to ensure convergence . A recent trend of learning to optimize ( L2O ) parameterizes training algorithms to be learnable from data , such that the meta-learned optimizers can be adapted to a special class of functions and outperform general-purpose optimizers . That is particularly meaningful , when one has to solve a large number of yet similar optimization problems repeatedly and quickly . Specifically , for existing L2O methods that operate in the space of continuous optimization , almost all of them solve some ∗Equal Contribution . 1 minimization problem ( Andrychowicz et al. , 2016 ; Chen et al. , 2017 ; Li & Malik , 2016 ) , leveraging an LSTM or a reinforcement learner to model their optimizer . Different from classic optimization results that often provide worst-case convergence , most L2O methods have little or no convergence guarantees , especially on problem or data instances distinct from what is seen in training , leaving their generalizability in practice questionable ( Heaton et al. , 2020 ) . Motivated by L2O ’ s success in learning efficient minimization solvers from data , this paper seeks to answer : whether we could accomplish strong minimax L2O solvers as well ; and if yes , how generalizable they could be ? As it might look straightforward at first glance , such extension is highly nontrivial due to facing several unique challenges . Firstly , while continuous minimization has a magnitude of mature and empirically stable solvers , for general minimax optimization , even state-of-the-art analytical algorithms can exhibit instability or even divergence . To the best of our knowledge , most state-of-theart convergence analysis of minimax optimization is built on the convex-concave assumption ( Gidel et al. , 2018 ; Mokhtari et al. , 2019 ; Ryu et al. , 2019 ) , and some recent works relax the assumption to nonconvex-concave ( Lin et al. , 2019 ; 2020 ) . Convergence for general minimax problems is still open . That makes a prominent concern on whether a stable minimax L2O is feasible . Secondly , given the two groups of min and max variables simultaneously , it is unclear to what extent their optimization strategies can be modeled and interact within one unified framework – a new question that would never be met in minimization . Thirdly , the noisy and sometimes cyclic dynamics of minimax optimization will provide noisier guidance ( e.g. , reward ) to L2O ; not to say that , it is not immediately clear how to define the reward : for minimization , the reward is typically defined as the negative cumulative objective values along the history ( Li & Malik , 2016 ) . However , for minimax optimization the objective can not simply be decreased or increased monotonically . Contribution : This paper is a pilot study into minimax L2O . We start by establishing the first dedicated minimax L2O framework , called Twin-L2O . It is composed of two LSTMs sharing one objective-based reward , separately responsible for updating min and max variables . By ablations of the design options , we find this decoupled design facilitate meta-learning most , particularly when the min and max updates are highly non-symmetric . We demonstrate the superior convergence of Twin-L2O on several testbed problems , compared against a number of analytical solvers . On top of that , we further investigate how to enhance the generalizability of the learned minimax solver1 , and discuss two complementary alternatives with experimental validations . The first alternative is an empirical toolkit that is applicable for general minimax L2O . We introduce curriculum learning to training L2O models for the first time , by recognizing that not all problem instances are the same difficult to learn to solve . After plugging in that idea , we show that Twin-L2O can be trained to stably solve a magnitude more problem instances ( in terms of parameter varying range ) . The second alternative explores a theoretical mechanism called safeguarding , particularly for the important special case of convex-concave problems . When solving a testing instance , safeguarding identifies when an L2O failure would occur and provides an analytical fall-back option ( Diakonikolas , 2020 ) . That guarantees convergence for convex-concave problems and , in practice , converges faster even when the problem parameters are drawn from a different distribution from training . 2 RELATED WORK . 2.1 MINIMAX OPTIMIZATION . Following ( Neumann , 1928 ) , the problem ( 1 ) has been studied for decades due to its wide applicability . Simultaneous gradient descent ( SimGD ) or gradient descent ascent ( GDA ) ( Nedić & Ozdaglar , 2009 ; Du & Hu , 2019 ; Jin et al. , 2019 ; Lin et al. , 2019 ) is one of the simplest minimax algorithms , conducting gradient descent over variable x and gradient ascent over variable y . However , the dynamics of SimGD or GDA can converge to limit cycles or even diverge ( Benaım & Hirsch , 1999 ; Mertikopoulos et al. , 2018b ; Lin et al. , 2019 ) . To address this issue , Optimistic gradient descent ascent ( OGDA ) simply modifies the dynamics of GDA and shows more stable performance ( Daskalakis et al. , 2018 ; Daskalakis & Panageas , 2018 ; Liang & Stokes , 2019 ; Mertikopoulos et al. , 2018a ; Gidel et al. , 2018 ; Mokhtari et al. , 2019 ) . OGDA attracts more attention because of its empirical success in training GANs . ( Ryu et al. , 2019 ) theoretically studies OGDA by analyzing its continuous time dynamic and 1We differentiate the usages of two terms : parameters and variables , throughout the paper . For example , minu maxv ax 2 − by2 , we call a , b parameters and x , y variables . For simplicity , this paper only discusses the L2O generalizability when the testing instances ’ parameter distribution differs from the training . 2 proposes Anchored simultaneous gradient descent that shows good performance . Follow-the-Ridge ( Wang et al. , 2019 ) also addresses the limit cycling problem by introducing second-order information into the dynamic of GDA . Lately , K-Beam ( Hamm & Noh , 2018 ) stabilizes the convergence of GDA by duplicating variable y , yielding strong performance . At each iteration , it performs gradient ascent independently on K copies of y and greedily chooses the copy that leads to a large function value f , then it updates x based on the selected copies . 2.2 LEARNING TO OPTIMIZE . As a special instance of meta-learning , L2O has been studied in multiple contexts , with continuous optimization being one of its main playgrounds so far . The first L2O framework is introduced in ( Andrychowicz et al. , 2016 ) , where both the optimizee ’ s gradients and loss function values are formulated as the input features for an RNN optimizer . Due to the enormous number of parameters , a coordinate-wise design of RNN optimizer is adopted , where all optimization coordinates share the same updating strategy . ( Li & Malik , 2016 ) uses the gradient history and objective values as observations and step vectors as actions in their reinforcement learning framework . ( Chen et al. , 2017 ) leverages RNN to train a meta-optimizer to optimize black-box functions . Two effective training tricks , random scaling and objective convexifying , are presented in ( Lv et al. , 2017 ) . Wichrowska et al . ( 2017 ) presents an optimizer of multi-level hierarchical RNN architecture augmented with additional architectural features . Li et al . ( 2020 ) introduces a Jacobian regularization to L2O and enhances the domain adaptation performance of optimizees . Chen et al . ( 2020a ) proposes several improved training techniques to stabilize L2O training and ameliorate performance . You et al . ( 2020 ) ; Chen et al . ( 2020b ; c ) extend the application scope of L2O into various practical problems such as graph neural network training , domain generalization , and noisy label training . The above works address continuous minimization problems using single optimizer models . One exception , ( Cao et al. , 2019 ) , extends L2O to solving Bayesian swarm optimization . The author presents a novel architecture where multiple LSTMs jointly learn iterative update formulas for the swarm of particles , coordinated by attention mechanisms . We also notice that two recent efforts ( Jiang et al. , 2018 ; Xiong & Hsieh , 2020 ) introduce L2O to adversarial training , a renowned application of minimax optimization . However , both of them merely utilize L2O to solve the inner minimization of their minimax problems ( i.e. , generating attacks ) , while the outer maximization is still solved analytically . Neither of the two directly solves the full minimax optimization . 3 METHOD . 3.1 MAIN FRAMEWORK : TWIN LEARNABLE OPTIMIZERS ( TWIN-L2O ) . The main L2O framework we proposed is named Twin-L2O , where we use two learnable optimizers to alternate between min and max updates . Our design adopts the basic idea of ( Andrychowicz et al. , 2016 ) to use Long Short-Term Memory ( LSTM ) to model learnable optimizers , for solving target problems known as optimizees . At each step , LSTM outputs the update of the optimizee variables . The LSTM inputs are typically the current zero-order or first-order information of the optimizee ( Andrychowicz et al. , 2016 ; Lee & Choi , 2018 ) , plus the historic optimization trajectory information . In Twin-L2O , two LSTMs separately update x and y and record historical trajectory information of their own variables respectively . Formally , we consider the minimax problem minxmaxy f ( x , y ) . We use two LSTM optimizers , LSTM-Min and LSTM-Max , to updates the min variable x and the max variable y respectively . LSTM-Min is parameterized by φmin and LSTM-Max is parameterized by φmax . At each iteration t , Twin-L2O updates x and y in turns and yields the following rule : xt+1 =xt + ∆xt , where ( ∆xt , hmint+1 ) = LSTM-Min ( [ ∇xf ( xt , yt ) , ∇yf ( xt , yt ) ] , hmint , φmin ) , yt+1 =yt + ∆yt , where ( ∆yt , hmaxt+1 ) = LSTM-Max ( [ ∇yf ( xt+1 , yt ) , ∇xf ( xt+1 , yt ) ] , hmaxt , φmax ) , ( 2 ) where hmint and h max t are the historical trajectory information of LSTM-Min and LSTM-Max at time step t. This formulation is inspired by the SimGD/GDA-style algorithms ( Nedić & Ozdaglar , 2009 ; Du & Hu , 2019 ; Jin et al. , 2019 ; Lin et al. , 2019 ) that conduct simultaneous/alternative gradient descent over x and ascent over y . Figure A4 ( Appendix A1 ) conceptually illustrates the framework . The next question is to design the L2O reward . To train the LSTM optimizers , the loss function is often to penalize some type of cost , accumulated along the optimization trajectory for a horizon of T 3 steps ( also known as the unrolling length for LSTM ( Sherstinsky , 2018 ) ) L ( φmin , φmax ) = Ef [ T∑ t=1 wtR ( f , xt , yt ) ] , ( 3 ) wt is chosen to be all 1 following the basic setting in ( Andrychowicz et al. , 2016 ) , that might be tuned for better performance in future work . As a key design option , R ( f ) represents the reward to guide the L2O training . In existing L2O methods for continuous minimization ( Andrychowicz et al. , 2016 ; Lv et al. , 2017 ) , R ( f ) is usually simply set to R ( f , xt ) ) = f ( xt ) to encourage fast decrease of objective values over time . To extend this existing reward to the minimax scenario , we can not directly penalize the overall objective function value either way , since the min and max objectives are entangled . Also , different from pure minimization problems , the Twin-L2O updates ( 2 ) consist of two alternating steps governed by two different LSTM optimizers : each accounts for its own subproblem goal ( min or max updates ) , but the two also have to collaborate to explore/exploit the minimax landscape . We specifically design the following reward that implicitly addresses the above issue by setting a new reward function : L ( φmin , φmax ) = Ef [ T∑ t=1 { [ f ( xt , yt−1 ) − f ( xt , yt ) ] + [ f ( xt , yt−1 ) − f ( xt−1 , yt−1 ) ] } ] . ( 4 ) Analysis of the reward design In Eqn . 4 , the first and second terms always characterize two consecutive min and max updates . In more details , the value of f ( xt , yt ) −f ( xt , yt−1 ) solely reflects how effectively the t-step max update increases the objective f , while f ( xt , yt−1 ) − f ( xt−1 , yt−1 ) reflects the effectiveness of t-step min update in decreasing the objective f . Our goal is then to maximize the weighted accumulated sum for f ( xt , yt ) −f ( xt , yt−1 ) , while minimizing the weighted accumulated sum for f ( xt , yt ) − f ( xt , yt−1 ) , t = 1 , 2 , ... , T . Combining the two sub-goals together ( with a sign change to turn max into min ) yields our reward . One may also alternatively interpret Eqn . 4 as penalizing the loss change from f ( xt , yt ) along both x and y updating directions , which would encourage yielding stationary points . For the reward design , we provide a more detailed discussion in Appendix A6 . Specifically , we provide a comparison between the objective-based reward in Eqn . 4 , and another possible gradientbased reward . The latter was found to be ineffective in solving the problems presented in Section 4 . Rationale of the framework selection Another important design question is to what extent learning the min and max updates should be ( dis ) entangled : on the one hand , the two steps obviously interact with each other as they jointly explore the minimax landscape ; on the other hand , min and max steps commonly have asymmetric difficulty levels , that have been leveraged by previous algorithms . For example , ( Hamm & Noh , 2018 ) demonstrates the failure of alternating gradient descent in minimax optimization due to the multiple solution discontinuity of the inner maximization , and addresses that by simultaneously tracking K candidate solutions for the max step , while the outer minimization remains to take one descent step . Besides the joint reward ( 4 ) , the default Twin-L2O design leverages two independent LSTMs in Eqn . ( 2 ) , each dedicatedly handling min or max updates . In comparison , we also consider two other more `` entangled '' ways : ( a ) fully entangling the two optimizers , i.e . using one LSTM to simultaneously generate min and max outputs ; ( b ) weakly entangling the two optimizers , by using two LSTMs sharing weights , yet allowing either to maintain its own temporal hidden states . Our ablation experiments ( see Section 4.1 ) find that the default decoupled design in Eqn . ( 2 ) seems to facilitate the L2O learning most .

Summary: The paper introduces the _learning to optimize_ (L2O) framework into the solution of minimax problems. The base model is composed of two decoupled LSTMs with a shared objective, with the two LSTMs being respectively responsible for the update of the min and max variables. On top of this, the authors further investigate two possible improvements.  One consists in applying curriculum learning to improve the generalization capability of the solver while the other uses safeguarding to guarantee convergence in convex-concave problems.