Convex-Concave Min-Max Stackelberg Games

1 Introduction . Min-max optimization problems have attracted a great deal of attention recently because of their applicability to a wide range of machine learning problems . Examples of settings in which min-max optimization problems arise include , but are not limited to , reinforcement learning [ 15 ] , generative adversarial networks [ 66 ] , fairness in machine learning [ 14 , 23 , 48 , 68 , 82 ] , adversarial learning [ 72 ] , generative adversarial imitation learning [ 11 , 32 ] , and statistical learning ( e.g. , learning parameters of exponential families ) [ 14 ] . These applications often require solving a constrained min-max optimization problem ( with independent feasible sets ) : i.e. , minx2X maxy2Y f ( x , y ) , where f : X ⇥ Y ! R is continuous , and X ⇢ Rn and Y ⇢ Rm are non-empty and compact . A convex-concave constrained min-max optimization problem is one in which f is convex in x and concave in y . In the special case of convex-concave objective functions , the seminal minimax theorem holds : i.e. , minx2X maxy2Y f ( x , y ) = maxy2Y minx2X f ( x , y ) [ 55 ] . This theorem guarantees the existence of a saddle point ( i.e. , a point that is simultaneously a minimum of f in the x-direction and a maximum of f in the y-direction ) , and allows us to interpret the optimization problem as a simultaneous-move , zero-sum game , where y⇤ ( resp . x⇤ ) is a best-response of the outer ( resp . inner ) player to their opponent ’ s strategy x⇤ ( resp . y⇤ ) , in which case a saddle point is also called a minimax point or a Nash equilibrium . In this paper , we show that the computation of competitive equilibria in Fisher markets [ 10 ] , a central solution concept for one of the most well-studied market models in algorithmic game theory [ 57 ] , can be understood as a convex-concave min-max optimization problem , albeit with dependent 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feasible sets . We define a constrained min-max optimization problem with dependent feasible sets as an optimization problem of the following form : minx2X maxy2Y : g ( x , y ) 0 f ( x , y ) , where f : X ⇥ Y ! R is continuous , X ⇢ Rn and Y ⇢ Rm are non-empty and compact , and g ( x , y ) = ( g1 ( x , y ) , . . . , gK ( x , y ) ) T with gk : X ⇥ Y ! R.1 Unfortunately , certain desirable properties of standard convex-concave min-max optimization problems with independent feasible sets do not carry over to the more general dependent setting . First and foremost , the minimax theorem [ 55 ] does not hold , which in turn precludes the existence of Nash equilibrium in general : Example 1.1 . Consider the following constrained min-max optimization problem with dependent feasible sets : minx2 [ 1,1 ] maxy2 [ 1,1 ] : x+y0 x2 + y + 1 . The optimum is x⇤ = 1/2 , y⇤ = 1/2 , with value 3/4 . Now , consider the same problem , with the order of the min and the max reversed : maxy2 [ 1,1 ] minx2 [ 1,1 ] : x+y0 x 2 + y + 1 . The optimum is now x⇤ = 1 , y⇤ = 1 , with value 3 . Without a minimax theorem , constrained min-max optimization problems with dependent feasible sets are more appropriately viewed as two-player sequential , i.e. , Stackelberg , games where the outer player chooses x 2 X before the inner player responds with their choice of y ( x ) 2 Y s.t . g ( x , y ( x ) ) 0 . The relevant equilibrium concept is then a Stackelberg [ 78 ] equilibrium,2 in which the outer player optimizes their choice assuming the inner player will best-respond : i.e. , optimize their choice in turn . We thus refer to constrained min-max optimization problems with dependent feasible sets as min-max Stackelberg games . In such games , the value function3 V : X ! R is defined as V ( x ) = maxy2Y : g ( x , y ) 0 f ( x , y ) . This function represents the outer player ’ s loss , assuming the inner player chooses a feasible best-response , so it is the function the outer player seeks to minimize . The inner player seeks only to maximize the objective function . The first-order necessary and sufficient conditions for a tuple ( x⇤ , y⇤ ) 2 X ⇥ Y to be a Stackelberg equilibrium in a convexconcave min-max Stackelberg game are given by the KKT stationarity conditions for the two players ’ optimization problems , namely minx2X V ( x ) for the outer player , and maxy2Y : g ( x , y ) 0 f ( x , y ) for the inner player . In the independent strategy setting , Danskin ’ s theorem [ 17 ] states that rxV ( x ) = rxf ( x , y⇤ ) , where y⇤ 2 argmaxy2Y f ( x , y ) . In other words , when there is no dependence among the players ’ strategy sets , the gradient of the value function coincides with that of the objective function . The first-order necessary and sufficient conditions for a tuple ( x⇤ , y⇤ ) to be an ( interior ) saddle point are for it to be a stationary point of f ( i.e. , rxf ( x⇤ , y⇤ ) = ryf ( x⇤ , y⇤ ) = 0 ) . It is therefore logical for players in the independent/simultaneous-move setting to follow the gradient of the objective function . In the dependent/sequential setting , however , the direction of steepest descent ( resp . ascent ) for the outer ( resp . inner ) player is the gradient of their value function . Example 1.2 . Consider , once again , the problem posed in Example 1.1 , and recall Jin , Netrapalli , and Jordan ’ s gradient descent with max-oracle algorithm [ 41 ] : x ( t+1 ) = x ( t ) ⌘rxf xt , y⇤ ( x ( t ) ) , where y⇤ ( x ( t ) ) 2 argmaxy2Y : g ( x ( t ) , y ) 0 f ( x ( t ) , y ) for ⌘ > 0 . Applied to this sample problem , with ⌘ = 1 , this algorithm yields the following update rule : x ( t+1 ) = x ( t ) 2x ( t ) = x ( t ) . Thus , letting x ( 0 ) equal any feasible x , the output cycles between x and x , so that the average of the iterates converges to x⇤ = 0 ( with y⇤ = 0 ) , which is not a Stackelberg equilibrium , as the Stackelberg equilibrium of this game is x⇤ = 1/2 , y⇤ = 1/2 . Now consider an algorithm that updates based not on gradient of the objective function , but of the value function , namely x ( t+1 ) = x ( t ) ⌘rxV ( xt ) . The value function is V ( x ) = maxy2 [ 1,1 ] : x+y0 x 2+y+1 = x2 x+1 , with gradient V 0 ( x ) = 2x 1 . Thus , when ⌘ = 1 , this algorithm yields the following update rule : x ( t+1 ) = x ( t ) V 0 ( x ( t ) ) = x ( t ) 2x ( t ) +1 = x ( t ) +1 . If we run this algorithm from initial point x ( 0 ) = 1/8 , we get x ( 1 ) = 1/8+1 = 7/8 , x ( 2 ) = 7/8+1 = 1/8 , and so on . The average of the iterates 1/8 , 7/8 , 1/8 , . . . converges to x⇤ = 1/2 ( 1/8 + 7/8 ) = 1/2 , and correspondingly y⇤ = 1/2 , which is indeed the Stackelberg equilibrium . 1Although notationally similar , we note that this model is more general than that of Daskalakis , Skoulakis , and Zampetakis [ 18 ] . The authors state that under the assumptions of the minimax theorem [ 55 ] , a Nash equilibrium is guaranteed to exist in their model . When there is dependence among the players ’ feasible sets , however , this claim does not hold ( see Example 1.1 ) . 2In constrained convex-concave min-max optimization problems with independent feasible sets , the notions of Nash and Stackelberg equilibria coincide . They also coincide in the dependent setting , when Nash equilibria exist , but they do not coincide in general , since Nash equilibria need not exist . 3Note that this use of the term value function in economics is distinct from its use in reinforcement learning . In this paper , we introduce two first-order ( subgradient ) methods that solve min-max Stackelberg games—to our knowledge the first such methods . Our approach relies on a new generalization of a series of fundamental results in mathematical economics known as envelope theorems [ 1 , 50 ] . Envelope theorems generalize aspects of Danskin ’ s theorem , by providing explicit formulas for the gradient of the value function in dependent strategy settings , when a derivative is guaranteed to exist . To sidestep the differentiability issue , we introduce a generalized envelope theorem that gives an explicit formula for the subdifferential of the value function in dependent strategy settings . Our first algorithm follows Jin , Netrapalli , and Jordan [ 41 ] , assuming access to a max-oracle that returns y⇤ 2 argmaxy2Y : g ( x , y ) 0 f ( x , y ) , given x 2 X . Hence , our first algorithm solves only for an optimal x⇤ , while our second algorithm explicitly solves for both x⇤ and y⇤ . We show that both algorithms converge in polynomial time to a Stackelberg equilibrium of any convex-concave min-max Stackelberg game with nonempty , compact and convex dependent strategy sets . In Table 1 , we summarize the iteration complexities of our algorithms , i.e. , the number of iterations required to achieve an `` -approximate equilibrium , where `` is the desired precision . Finally , we apply our results to the computation of competitive equilibria in Fisher markets . In this context , our method for solving a min-max Stackelberg game reduces to solving the market in a decentralized manner using a natural market dynamic called tâtonnement [ 80 ] . We demonstrate the efficacy and efficiency of our algorithms in practice by running a series of experiments in which we compute competitive equilibria in Fisher markets with varying utility structures—specifically , linear , Cobb-Douglas , and Leontief . Although our theoretical results do not apply to all these Fisher markets—Leontief utilities , in particular , are not differentiable—tâtonnement converges in all our experiments . That said , the rate of convergence does seem to depend on the smoothness characteristics of the utility structures ; we observe slower convergence for Leontief utilities , and faster convergence than our theory predicts for Cobb-Douglas utilities , which are not only differentiable , but whose value function is also differentiable . Related Work Our model of min-max Stackelberg games seems to have first been studied by Wald , under the posthumous name of Wald ’ s maximin model [ 79 ] . A variant of Wald ’ s maximin model is the main paradigm used in robust optimization , a fundamental framework in operations research for which many methods have been proposed [ 6 , 56 , 61 ] . Shimizu and Aiyoshi [ 70 , 71 ] proposed the first algorithm to solve min-max Stackelberg games via a relaxation to a constrained optimization problem with infinitely many constraints , which nonetheless seems to perform well in practice . More recently , Segundo , Krohling , and Cosme [ 69 ] proposed an evolutionary algorithm for these games , but they provided no guarantees . As pointed out by Postek and Shtern , all prior methods either require oracles and are stochastic in nature [ 6 ] , or rely on a binary search for the optimal value , which can be computationally complex [ 56 ] . The algorithms we propose in this paper circumvent the aforementioned issues and can be used to solve a large class of convex robust optimization problems in a simple and efficient manner . Extensive-form games in which players ’ strategy sets can depend on other players ’ actions have been studied by Davis , Waugh , and Bowling [ 19 ] assuming payoffs are bilinear , and by Farina , Kroer , and Sandholm [ 28 ] for another specific class of convex-concave payoffs . Fabiani et al . [ 25 ] and Kebriaei and Iannelli [ 43 ] study more general settings than ours , namely non-zero-sum Stackelberg games with more than two players . Both sets of authors derive convergence guarantees assuming specific payoff structures , but their algorithms do not converge in polynomial time . Min-max Stackelberg games naturally model various economic settings . They are related to the abstract economies first studied by Arrow and Debreu [ 4 ] ; however , the solution concept that has been the focus of this literature is generalized Nash equilibrium [ 26 , 27 ] , which , like Stackelberg , is a weaker solution concept than Nash , but which makes the arguably unreasonable assumption that the players move simultaneously and nonetheless satisfy the constraint dependencies on their strategies imposed by one another ’ s moves . See Appendix A for a more detailed discussion of generalized Nash equilibria versus Stackelberg equilibria . Duetting et al . [ 22 ] study optimal auction design problems . They propose a neural network architecture called RegretNet that represents optimal auctions , and train their networks using Algorithm 3.4 . Optimal auction design problems can be seen as min-max Stackelberg games ; however , as their objectives are non-convex-concave in general , our guarantees do not apply . In this paper , we observe that solving for the competitive equilibrium of a Fisher market can also be seen as solving a ( convex-concave ) min-max Stackelberg game . The study of the computation of competitive equilibria in Fisher markets was initiated by Devanur et al . [ 20 ] , who provided a polynomial-time method for the case of linear utilities . Jain , Vazirani , and Ye [ 40 ] subsequently showed that a large class of Fisher markets could be solved in polynomial-time using interior point methods . Recently , Gao and Kroer [ 29 ] studied an alternative family of first-order methods for solving Fisher markets ( only ; not min-max Stackelberg games more generally ) , assuming linear , quasilinear , and Leontief utilities , as such methods can be more efficient when markets are large . See Appendix G for a detailed discussion of recent progress on solving min-max Stackelberg games , both in the convex-concave case and the non-convex-concave case .

Summary: The paper studies min-max optimization problems with dependent strategy sets, and introduces a first-order method that solves a large class of convex-concave min-max games with dependent strategy sets. In doing so, the authors provide a suitable envelope theorem which generalizes known results to the case of dependent strategies. That result is used to build two algorithms: max-oracle gradient-descent (which returns an approximation of the inner player's value function), and nested gradient-descent/ascent (which computes both players' strategies explicitly). The algorithms are shown to converge in polynomial time, and  they are experimentally evaluated through the computation of competitive equilibria in Fisher markets with varying utility structures.