Conic Blackwell Algorithm: Parameter-Free Convex-Concave Saddle-Point Solving

√ T ) average regret . Intuitively , our approach generalizes to other decision sets of interest ideas from the Counterfactual Regret minimization ( CFR+ ) algorithm , which has very strong practical performance for solving sequential games on simplexes . We show how to implement CBA+ for the simplex , ` p norm balls , and ellipsoidal confidence regions in the simplex , and we present numerical experiments for solving matrix games and distributionally robust optimization problems . Our empirical results show that CBA+ is a simple algorithm that outperforms state-ofthe-art methods on synthetic data and real data instances , without the need for any choice of step sizes or other algorithmic parameters . 1 Introduction . We are interested in solving saddle-point problems ( SPPs ) of the form min x∈X max y∈Y F ( x , y ) , ( 1 ) whereX ⊂ Rn , Y ⊂ Rm are convex , compact sets , and F : X×Y → R is a subdifferentiable convexconcave function . Convex-concave SPPs arise in a number of practical problems . For example , the problem of computing a Nash equilibrium of a zero-sum game can be formulated as a convex-concave SPP , and this is the foundation of most methods for solving sequential zero-sum games [ vS96 , ZJBP07 , TBJB15 , KWKKS20 ] . They also arise in imaging [ CP11 ] , ` ∞-regression [ ST18 ] , Markov Decision Processes [ Iye05 , WKR13 , ST18 ] , and in distributionally robust optimization , where the max term represents the distributional uncertainty [ ND16 , BTHKM15 ] . In this paper we propose efficient , parameter-free algorithms for solving ( 1 ) in many settings , i.e. , algorithms that do not require any tuning or choices of step sizes . Repeated game framework One way to solve convex-concave SPPs is by viewing the SPP as a repeated game between two players , where each step t consists of one player choosing xt ∈ X , the other player choosing yt ∈ Y , and then the players observe the payoff F ( xt , yt ) . If each player employs a regret-minimization algorithm , then a well-known folk theorem says that the uniform average strategy generated by two regret minimizers repeatedly playing an SPP against each other converges to a solution to the SPP . We will call this the “ repeated game framework ” ( see Section 2 ) . There are already well-known algorithms for instantiating the above repeated game framework ∗Julien Grand-Clément acknowledges the financial support of Hi ! Paris for this project . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for ( 1 ) . For example , one can employ the online mirror descent ( OMD ) algorithm , which generates iterates as follows for the first player ( and similarly for the second player ) : xt+1 = arg min x∈X 〈ηft , x〉+D ( x‖xt ) , ( 2 ) where ft ∈ ∂xF ( xt , yt ) ( ∂x denotes here the set of subgradients as regards the variable x ) , D ( ·‖· ) is a Bregman divergence which measures distance between pairs of points , and η > 0 is an appropriate step size . By choosing D appropriately for X , the update step ( 2 ) becomes efficient , and one can achieve an overall regret on the order of O ( √ T ) after T iterations . This regret can be achieved either by choosing a fixed step size η = α/L √ T , where L is an upper bound on the norms of the subgradients visited ( ft ) t≥0 , or by choosing adaptive step sizes ηt = α/ √ t , for α > 0 . This is problematic , as 1 ) the upper bound L may be hard to obtain in many applications and may be too conservative in practice , and 2 ) adequately tuning the parameter α can be time- and resourceconsuming , and even practically infeasible for very large instances , since we won ’ t know if the step size will cause a slow convergence or a divergence until late in the optimization process . This is not just a theoretical issue , as we highlight in our numerical experiments ( Section 4 ) and in the appendices ( Appendices F ) . Similar results and challenges hold for the popular follow the regularized leader ( FTRL ) algorithm ( see Appendix F ) . The above issues can be circumvented by employing adaptive variants of OMD or FTRL , which lead to parameter- and scale-free algorithms that estimate the parameters through the observed subgradients , e.g. , AdaHedge for the simplex setting [ DRVEGK14 ] or AdaFTRL for general compact convex decisions sets [ OP15 ] . Yet these adaptive variants have not seen practical adoption in largesale game-solving , where regret-matching variants are preferred ( see the next paragraph ) . As we show in our experiments , adaptive variants of OMD and FTRL perform much worse than our proposed algorithms . While these adaptive algorithms are referred to as parameter-free , this is only true in the sense that they are able to learn the necessary parameters . Our algorithm is parameter-free in the stronger sense that there are no parameters that even require learning . Formalizing this difference may be one interesting avenue for explaining the performance discrepancy on saddle-point problems . Regret Matching In this paper , we introduce alternative regret-minimization schemes for instantiating the above framework . Our work is motivated by recent advances on solving large-scale zero-sum sequential games . In the zero-sum sequential game setting , X and Y are simplexes , the objective function becomes F ( x , y ) = 〈x , Ay〉 , and thus ( 1 ) reduces to a bilinear SPP . Based on this bilinear SPP formulation , the best practical methods for solving large-scale sequential games use the repeated game framework , where each player minimizes regret via some variant of counterfactual regret minimization ( CFR , [ ZJBP07 ] ) . Variants of CFR were used in every recent poker AI challenge , where poker AIs beat human poker players [ BBJT15 , MSB+17 , BS18 , BS19 ] . The CFR framework itself is a decomposition of the overall regret of the bilinear SPP into local regrets at each decision point in a sequential game [ FKS19a ] . The key to the practical performance of CFR-based algorithms seems to be three ingredients ( beyond the CFR decomposition itself ) : ( 1 ) a particular regret minimizer called regret matching+ ( RM+ ) [ TBJB15 ] which is employed at each decision point , ( 2 ) aggressive iterate averaging schemes that put greater weight on recent iterates ( e.g . linear averaging , which weights iterate at period t by 2t/T ( T + 1 ) ) , and ( 3 ) an alternation scheme where the updates of the repeated game framework are performed in an asymmetric fashion . The CFR framework itself is specific to sequential bilinear games on simplexes , but these last three ingredients could potentially be generalized to other problems of the form ( 1 ) . That is the starting point of the present paper . The most challenging aspect of generalizing the above ingredients is that RM+ is specifically designed for minimizing regret over a simplex . However , many problems of the form ( 1 ) have convex sets X , Y that are not simplexes , e.g. , box constraints or norm-balls for distributionally robust optimization [ BTHKM15 ] . In principle , regret matching arises from a general theory called Blackwell approachability [ Bla56 , HMC00 ] , and similar constructions can be envisioned for other convex sets . However , in practice the literature has only focused on developing concrete implementable instantiations of Blackwell approachability for simplexes . A notable deviation from this is the work of [ ABH11 ] , who showed a general reduction between regret minimization over general convex compact sets and Blackwell approachability . However , their general reduction still does not yield a practically implementable algorithm : among other things , their reduction relies on certain black-box projections that are not always efficient . We show how to implement these necessary projections for the setting where X and Y are simplexes , ` p balls , and intersections of the ` 2 ball with a hyperplane ( with a focus on the case where an ` 2 ball is intersected with a simplex , which arises naturally as a confidence region ) . This yields an algorithm which we will refer to as the conic Blackwell algorithm ( CBA ) , which is similar in spirit to the regret matching algorithm , but crucially generalizes to other decision sets . Motivated by the practical performance of RM+ , we construct a variant of CBA which uses a thresholding operation similar to the one employed by RM+ . We call this algorithm CBA+ . Our contributions We introduce CBA+ , a parameter-free algorithm which achieves O ( √ T ) regret in the worst-case and generalizes the strong performances of RM+ for bilinear , simplex saddle-points solving to other more general settings . A major selling point for CBA+ is that it does not require any step size choices . Instead , the algorithm implicitly adjusts to the structure of the domains and losses by being instantiations of Blackwell ’ s approachability algorithm . After developing the CBA+ algorithm , we then develop analogues of another crucial components for large-scale game solving . In particular , we prove a generalization of the folk theorem for the repeated game framework for solving ( 1 ) , which allows us to incorporate polynomial averaging schemes such as linear averaging . We then show that CBA+ is compatible with linear averaging on the iterates . This mirrors the case of RM and RM+ , where only RM+ is compatible with linear averaging on the iterates . We also show that both CBA and CBA+ are compatible with polynomial averaging when simultaneously performed on the regrets and the iterates . Combining all these ingredients , we arrive at a new class of algorithms for solving convex-concave SPPs . As long as efficient projection operations can be performed ( which we show for several practical domains , including the simplex , ` p balls and confidence regions in the simplex ) , one can apply the repeated game framework on ( 1 ) , where one can use either CBA or CBA+ as a regret minimizer for X and Y along with polynomial averaging on the generated iterates to solve ( 1 ) at a rate of O ( 1/ √ T ) . We highlight the practical efficacy of our algorithmic framework on several domains . First , we solve two-player zero-sum matrix games and extensive-form games , where RM+ regret minimizer combined with linear averaging and alternation , and CFR+ , lead to very strong practical algorithms [ TBJB15 ] . We find that CBA+ combined with linear averaging and alternation leads to a comparable performance in terms of the iteration complexity , and may even slightly outperform RM+ and CFR+ . On this simplex setting , we also find that CBA+ outperforms both AdaHedge and AdaFTRL . Second , we apply our approach to a setting where RM+ and CFR+ do not apply : distributionally robust empirical risk minimization ( DR-ERM ) problems . Across two classes of synthetic problems and four real data sets , we find that our algorithm based on CBA+ performs orders of magnitude better than online mirror descent and FTRL , as well as their optimistic variants , when using their theoretically-correct fixed step sizes . Even when considering adaptive step sizes , or fixed step sizes that are up to 10 , 000 larger than those predicted by theory , our CBA+ algorithm performs better , with only a few cases of comparable performance ( at step sizes that lead to divergence for some of the other non parameter-free methods ) . The fast practical performance of our algorithm , combined with its simplicity and the total lack of step sizes or parameters tuning , suggests that it should be seriously considered as a practical approach for solving convex-concave SPPs in various settings . Finally , we make a brief note on accelerated methods . Our algorithms have a rate of convergence towards a saddle point of O ( 1/ √ T ) , similar to OMD and FTRL . In theory , it is possible to obtain a faster O ( 1/T ) rate of convergence when F is differentiable with Lipschitz gradients , for example via mirror prox [ Nem04 ] or other primal-dual algorithms [ CP16 ] . However , our experimental results show that CBA+ is faster than optimistic variants of FTRL and OMD [ SALS15 ] , the latter being almost identical to the mirror prox algorithm , and both achieving O ( 1/T ) rate of convergence . A similar conclusion has been drawn in the context of sequential game solving , where the fastest O ( 1/ √ T ) CFR-based algorithms have better practical performance than the theoretically-superior O ( 1/T ) -rate methods [ KWKKS20 , KFS18 ] . In a similar vein , using error-bound conditions , it is possible to achieve a linear rate , e.g. , when solving bilinear saddle-point problems over polyhedral decision sets using the extragradient method [ Tse95 ] or optimistic gradient descent-ascent [ WLZL20 ] . However , these linear rates rely on unknown constants , and may not be indicative of practical performance .

Summary: # Summary   In light of successful applications of regret matching (RM) -- an instance of Blackwell approachability -- and its variant regret matching+ (RM+) for saddle point optimization  in zero-sum matrix games with simplex decision sets, the paper proposes generalizations beyond this simple setting to address convex concave problems with compact decisions sets. In the way that RM+ improves RM the conic Blackwell algorithm+ is introduced to improve over the approach of conic blackwell approachability (CBA) from [1]. Furthermore, CBA and CBA+ require non trivial Euclidean projections, in this work explicit closed form or efficient approximations are introduced. CBA and CBA+ are extensively tested on a variety of domains, for which CBA+ seems to demonstrate improvements over CBA similar to how RM+ improves RM. Importantly, like RM and RM+, CBA and CBA+ are parameter-free requiring no tuning.