Title: Exploiting hidden structures in non-convex games for convergence to Nash equilibrium
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nash Equilibrium

Games

Gradient

Non-monotone VI

Natural Gradient

Precondition
Abstract: A wide array of modern machine learning applications – from adversarial models to multi-agent reinforcement learning – can be formulated as non-cooperative games whose Nash equilibria represent the system’s desired operational states. Despite having a highly non-convex loss landscape, many cases of interest possess a latent convex structure that could potentially be leveraged to yield convergence to an equilibrium. Driven by this observation, our paper proposes a flexible first-order method that successfully exploits such “hidden structures” and achieves convergence under minimal assumptions for the transformation connecting the players’ control variables to the game’s latent, convex-structured layer. The proposed method – which we call preconditioned hidden gradient descent (PHGD) – hinges on a judiciously chosen gradient preconditioning scheme related to natural gradient methods. Importantly, we make no separability assumptions for the game’s hidden structure, and we provide explicit convergence rate guarantees for both deterministic and stochastic environments.

Exploiting Hidden Structures in Non-Convex Games for Convergence to Nash Equilibrium

Abstract

Awide array of modern machine learning applications - from adversarial models to multi-agent reinforcement learning - can be formulated as non-cooperative games whose Nash equilibria represent the system's desired operational states. Despite having a highly non-convex loss landscape, many cases of interest possess a latent convex structure that could potentially be leveraged to yield convergence to an equilibrium. Driven by this observation, our paper proposes a flexible first-order method that successfully exploits such 'hidden structures' and achieves convergence under minimal assumptions for the transformation connecting the players' control variables to the game's latent, convex-structured layer. The proposed method - which we call preconditioned hidden gradient descent (PHGD) - hinges on a judiciously chosen gradient preconditioning scheme related to natural gradient methods. Importantly, we make no separability assumptions for the game's hidden structure, and we provide explicit convergence rate guarantees for both deterministic and stochastic environments.

1 Introduction

Many powerful AI architectures are based on the idea of combining conceptually straightforward settings coming from game theory with the expressive power of neural nets. Some prominent examples of this type include generative adversarial networks (GANs) [14], robust reinforcement learning [36], adversarial training [26], multi-agent reinforcement learning in games [35, 39, 41], and even multi-player games that include free-form natural-language communication [4]. Intuitively, in all these cases, the game-theoretic abstraction serves to provide a palpable, easy-to-understand target, i.e., an equilibrium solution with strong axiomatic justification. However, from a complexity-theoretic standpoint, such targets are excessively ambitious, requiring huge amounts of data to express and compute, even approximately. Because of this, the agents' policies must be encoded via a universal function approximator (such as a neural net) and training this architecture boils down to iteratively updating these parameters until the process - hopefully! - converges to the target equilibrium.

Unfortunately, despite the ubiquitousness of these settings, the design of algorithms with provable convergence guarantees is still relatively lacking. This deficit is not surprising if one considers that even the - comparatively much simpler - problem of equilibrium learning in finite games is hindered by numerous computational hardness [9, 10] as well as dynamic impossibility results

Emmanouil V. Vlatakis-Gkaragkounis ∗

Georgios Piliouras

[15, 16, 19, 24, 30, 31]. In this regard, our best hope for designing provably convergent algorithms is to focus on specific classes of games with some useful structure to exploit.

One of the most well-established frameworks of this type is the class of monotone games whose study goes back at least to Rosen [37]. As special cases, this setting includes single-agent convex minimization problems, two-player convex-concave min-max games, diagonally convex N -player games, etc. Owing to this connection, there has been a proliferation of strong positive results at the interface of game theory and optimization, see e.g., [28, 38] and references therein. In our case however, the agents do not play this monotone game directly, but can only access it indirectly via an encoding layer of control variables - like the weight parameters of a neural net that outputs a feasible strategy profile for the game in question. In this sense, from a machine learning perspective, the strategies of the game are latent variables , so we can think of each player as being equipped with a smooth mapping from a high-dimensional space of control variables to the strategy space of the game. Importantly, in contrast to the control variables, the latent variables are not directly accessible to the players themselves and should only be viewed as auxiliary variables - to all extents and purposes, the goal remains to find an operationally desirable control layer configuration. In this sense, the convex structure of the game becomes 'hidden' behind the control layer, which entangles multiple input/control variables into nonlinear manifolds of latent variables.

As a result, the convex structure of the underlying game is effectively destroyed, resulting in highly non-convex end-to-end interactions. This raises the following central challenge:

Can we design provably convergent algorithms for non-convex games with a hidden structure in the presence of general couplings between control and latent variables?

Prior work in the area has shown that this is a promising and, at the same time, highly challenging question. In [42], the setting of hidden bilinear games was introduced and a number of negative results were presented, to the effect that gradient-descent-ascent can exhibit a variety of non-convergent behaviors, even when the game admits a hidden bilinear structure. Subsequently, [13] provided an approximate minimax theorem for a class of two-agent games where the players pick neural networks, but did not provide any convergent training algorithm for this class of games. Instead, the first positive result on the dynamics of hidden games was obtained by [43] who established a series of non-local convergence guarantees to the von Neumann max-min solution of the game in the case of two-agent hidden strictly convex-concave games for all initial conditions satisfying a certain genericity assumption. This approach, however, only applied to continuous-time dynamics -not algorithms - and it further imposed strong separability assumptions on the representation of the game in the control layer. More recently, [32] established the first global convergence guarantees in hidden games but, once again, these apply only to continuous-time dynamics and a special case of two-agent convex-concave games (akin to playing a convex combination of hidden games with one-dimensional latent spaces). This paper is the closest antecedent to our work as it introduces a dynamical system, called Generalized Natural Gradient Flow, which makes the L 2 norm between the equilibrium in the hidden game and the current set of latent variables a Lyapunov function for the system.

Our results & techniques. Our paper seeks to provide an affirmative answer to the key challenge above under minimal assumptions on the coupling between latent and control variables. To that effect, we only assume that each agent is able to affect a measurable change along any latent variable by updating their control variables appropriately; without this assumption spurious equilibria can emerge due to the deficiency of the control layer architecture. Importantly however, even though the map from control to latent variables is known to the players, we do not assume that it can be efficiently inverted (e.g., to solve for a profile of control variables that realizes a profile of latent variables). Otherwise, if it could, the entire game could be solved directly in the latent layer and then ported back to the control layer, thus rendering the whole problem moot - and, indeed, when working with realistic neural net architectures, this inversion problem is, to all intents and purposes, impossible.

For intuition, we begin by designing a new continuous-time flow, that we call preconditioned hidden gradient dynamics, and which enjoys strong convergence properties in games with a hidden strictly monotone structure (Proposition 1). Similarly to [32], this is achieved by using the L 2 norm in the latent layer as a Lyapunov function in the control layer; however, the similarities with the existing literature end there. Our paper does not make any separability or low-dimensionality assumptions, and is otherwise purely algorithmic: specifically, building on the continuous-time intuition, we provide a concrete, implementable algorithm, that we call preconditioned hidden gradient descent (PHGD);

· · ·

softmax

θ

R

P

S

R

P

S

softmax

ϕ

· · ·

· · ·

· · ·

· · ·

· · ·

this algorithm is run with stochastic gradients , and enjoys a series of strong, global convergence guarantees in hidden games.

First, as a baseline, Theorem 1 shows that a certain averaged process achieves an O (1 / √ t ) convergence rate in all games with a hidden monotone structure and Lipschitz continuous loss functions. If the hidden structure is strongly monotone, Theorem 2 further shows that this rate can be improved to O (1 /t ) for the actual trajectory of the players' control variables; and if the algorithm is run with full, deterministic gradients, the rate becomes geometric (Theorem 3). To the best of our knowledge, these are the first bona fide algorithmic convergence guarantees for games with a hidden structure.

2 Problem setup and preliminaries

Throughout the sequel, we will focus on continuous N -player games where each player, indexed by i ∈ N := { 1 , . . . , N } , has a convex set of control variables θ i ∈ Θ i := R m i , and a continuously differentiable loss function ℓ i : Θ → R , where Θ := ∏ i Θ i denotes the game's control space . For concreteness, we will refer to the tuple Γ ≡ Γ( N , Θ , ℓ ) as the base game .

The most relevant solution concept in this setting is that of a Nash equilibrium , i.e., an action profile θ ∗ ∈ Θ that discourages unilateral deviations. Formally, we say that θ ∗ ∈ Θ is a Nash equilibrium of the base game Γ if

where we employ the standard game-theoretic shorthand ( θ i ; θ -i ) to distinguish between the action of the i -th player and that of all other players in the game. Unfortunately, designing a learning algorithm that provably outputs a Nash equilibrium is a very elusive task: the impossibility results of Hart & Mas-Colell [15, 16] already preclude the existence of uncoupled dynamics that converge to a Nash equilibrium in all games; more recently, Milionis et al. [31] established a similar impossibility result even for possibly coupled dynamics (in both discrete and continuous time), while Daskalakis et al. [9, 10] has shown that even the computation of an approximate equilibrium can be beyond reach.

In view of this, our work focuses on games with a hidden, latent structure that can be exploited to compute its Nash equilibria. More precisely, inspired by [42, 43] we have the following definition.

Definition 1. We say that the game Γ ≡ Γ( N , Θ , ℓ ) admits a latent - or hidden -structure if:

Each player's control variables can be mapped faithfully to a closed convex set of latent variables x i ∈ X i ⊆ R d i ; formally, we posit that there exists a Lipschitz smooth map χ i : Θ i →X i with no critical points and such that cl( χ i (Θ i )) = X i .

Each player's loss function factors through the game's latent space X := ∏ i X i as

for some Lipschitz smooth function f i : X → R called the player's latent loss function .

For concreteness, we will refer to the product map χ ( θ ) = ( χ i ( θ i )) i ∈N as the game's representation map , and the tuple G ≡ G ( N , X , f ) will be called the hidden / latent game . To simplify notation later on, we also assume that X i has nonempty topological interior, so, in particular, dim( X i ) = d i ≤ m i .

We illustrate the above notions in two simple - but not simplistic - examples below; for a schematic representation, cf. Fig. 2.

θ

·

Control variables

x

=

χ

(

θ

)

·

Latent variables

χ

R

d

R

m

R

f

ℓ

=

f

◦

χ

Example 2.1. Consider a single-agent game ( N = 1 ) where the objective is to minimize the nonconvex function ℓ ( θ ) = ∑ |D| α =1 (sigmoid( θ α ) -s α ) 2 over a dataset D = { s α } . This problem can be recast as a hidden convex problem with f ( x ) = ∑ |D| α =1 ( x α -s α ) 2 and χ ( θ ) = sigmoid( θ ) .

Example 2.2. A more complex scenario involves non-convex / non-concave min-max optimization problems of the form min θ 1 max θ 2 χ 1 ( θ 1 ) ⊺ Cχ 2 ( θ 2 ) where χ 1 and χ 2 are preconfigured multi-layer perceptrons (MLPs) constructed with smooth activation functions (such as CeLUs). This problem can be reformulated as a hidden bilinear problem by setting f 1 ( x 1 , x 2 ) = x ⊺ 1 Cx 2 = -f 2 ( x 1 , x 2 ) .

Before moving forward, there are some points worth noting regarding the above definitions.

Remark 1 . First, the requirement cl( χ (Θ)) = X means that all latent variable profiles x ∈ X can be approximated to arbitrary accuracy in the game's control space. The reason that we do not make the stronger assumption χ (Θ) = X is to capture cases like the sigmoid map: if χ ( θ ) = [1 + exp( -θ )] -1 for θ ∈ R , the image of χ is the interval (0 , 1) , which is convex but not closed.

Remark 2 . Second, the requirement that χ has no critical points simply means that, for any control variable configuration θ ∈ Θ , the Jacobian Jac( χ ( θ )) of χ at θ has full rank. By the implicit function theorem [23], this simply means that χ locally looks like a projection (in suitable coordinates around θ ), so it is possible to affect a measurable change along any feasible latent direction by updating each player's control variables appropriately. This is no longer true if χ has critical points, so this is a minimal requirement to ensure that no spurious equilibria appear in the game's latent space.

To proceed, we will assume that the latent game is diagonally convex in the sense of Rosen [37], a condition more commonly known in the optimization literature as monotonicity [12]. Formally, let

denote the individual gradient of the latent loss function of player i ∈ N , and write g ( x ) = ( g 1 ( x ) , . . . , g N ( x )) for the profile thereof. We then say that the latent game G is:

̸

Strictly monotone if (3a) holds as a strict inequality whenever x ′ = x. (3b)

Strongly monotone if ⟨ g ( x ′ ) -g ( x ) , x ′ -x ⟩ ≥ µ ∥ x ′ -x ∥ 2 for some µ > 0 and all x, x ′ . (3c)

Clearly, strong monotonicity implies strict monotonicity, which in turn implies monotonicity; on the other hand, when we want to distinguish between problems that are monotone but not strictly monotone, we will say that g is merely monotone . Finally, extending the above to the base game, we will say that Γ admits a hidden monotone structure when the latent game G is monotone as above (and likewise for strictly / strongly monotone structures).

Examples of monotone games include neural Kelly auctions and Cournot oligopolies [5, 6, 20, 25], covariance matrix optimization problems and power control [27, 29, 38], certain classes of congestion games [34], etc. In particular, in the case of convex minimization problems, monotonicity (resp. strict / strong monotonicity) corresponds to convexity (resp. strict / strong convexity) of the problem's objective function. In all cases, it is straightforward to check that the image x ∗ = χ ( θ ∗ ) of a Nash equilibrium θ ∗ ∈ Θ of the base game satisfies a Stampacchia variational inequality of the form

In turn, by monotonicity, this characterization is equivalent to the Minty variational inequality

To avoid trivialities, we will assume throughout that the solution set X ∗ of (SVI)/(MVI) is nonempty. This is a standard assumption without which the problem is not well-posed - and hence, meaningless from an algorithmic perspective.

Notation. To streamline notation (and unless explicitly mentioned otherwise), we will denote control variables by θ i and θ , and we will write x i = χ i ( θ i ) and x = χ ( θ ) for the induced latent variables. We will also write m := ∑ i m i for the dimensionality of the game's control space Θ and d := ∑ i d i for the dimensionality of the latent space X (so, in general, m ≥ d ). Finally, when the representation map χ is clear from the context, we will write J i ( θ i ) := Jac( χ i ( θ i )) ∈ R d i × m i for the Jacobian matrix of χ i at θ i ∈ Θ i , and J ( θ ) = ⊕ i J i ( θ i ) for the associated block diagonal sum.

3 Hidden gradients and preconditioning

We are now in a position to present our main algorithmic scheme for equilibrium learning in games with a hidden monotone structure. In this regard, our aim will be to overcome the following limitations in the existing literature on learning in hidden games: ( i ) the literature so far has focused exclusively on continuous-time dynamics, with no discrete-time algorithms proven to efficiently converge to a solution; ( ii ) the number of players is typically limited to two; and ( iii ) the representation maps are separable in the sense that the control variables are partitioned into subsets and each subset affects exactly one latent variable. 1

We start by addressing the last two challenges first. Specifically, we begin by introducing a gradient preconditioning scheme that allows us to design a convergent continuous-time dynamical system for hidden monotone games with an arbitrary number of players and no separability restrictions for its representation maps. Subsequently, we propose a bona fide algorithmic scheme - which we call preconditioned hidden gradient descent (PHGD) - by discretizing the said dynamics, and we analyze the algorithm's long-run behavior in Section 4.

3.1. Continuous-time dynamics for hidden games. To connect our approach with previous works in the literature, our starting point will be a simple setting already captured within the model of [32], min-max games with a hidden convex-concave structure and unconstrained one-dimensional control and latent spaces per player, i.e., Θ i = R = X i for i = 1 , 2 . We should of course note that this specific setting is fairly restrictive and not commonly found in deep neural network practice: in real-world deep learning applications, neural nets generally comprise multiple interconnected layers with distinct activation functions, so the input-output relations are markedly more complex and intertwined. Nevertheless, the setting's simplicity makes the connection with natural gradient methods particularly clear, so as in [32], it will serve as an excellent starting point.

Concretely, Mladenovic et al. [32] analyze the natural hidden gradient dynamics

The reason for this terminology - and the driving force behind the definition of (NHGD) - is the observation that, in one-dimensional settings, a direct application of the chain rule to the defining relation ℓ i = f i ◦ χ of the game's latent loss functions yields ∂ℓ i /∂θ i = χ ′ i ( θ i ) · ∂f i /∂x i so, in turn, (NHGD) becomes

where, for simplicity, we are tacitly assuming that χ i ( θ i ) ′ > 0 .

This expression brings two important points to light. First, even though (NHGD) is defined in terms of the actual, control-layer gradients ∂ℓ i /∂θ i of Γ , Eq. (4) shows that the dynamics are actually driven by the latent-layer, 'hidden gradients' g i ( x ) = ∂f i /∂x i of G . Second, the preconditioner | χ ′ i ( θ i ) | -2 in (NHGD) can be seen as a Riemannian metric on Θ : instead of defining gradients relative to the standard Euclidean metric of Θ ≡ R N , (NHGD) can be seen as a gradient flow relative

to the Riemannian metric g ij ( θ ) = δ ij χ ′ i ( θ i ) , which captures the 'natural' geometry induced by the representation map χ . 2

From an operational standpoint, the key property of (NHGD) that enabled the analysis of [32] is the observation that the L 2 energy function

between the latent representation x = χ ( θ ) of θ and a solution x ∗ of (SVI) / (MVI) is a Lyapunov function for (NHGD). Indeed, a straightforward differentiation yields

with the penultimate step following from (4) and the last one from (MVI). It is then immediate to see that the latent orbits x ( t ) = χ ( θ ( t )) of (NHGD) converge to equilibrium in strictly monotone games.

The approach in the example above depends crucially on the separability assumption which, among others, trivializes the problem. Indeed, if there is no coupling between control variables in the game's latent space, the equations x i = χ i ( θ i ) can be backsolved easily for θ (e.g., via binary search), so it is possible to move back-and-forth between the latent and control layers, ultimately solving the game in the latent layer and subsequently extracting a solution configuration in the control layer. Unfortunately however, extending the construction of (NHGD) to a non-separable setting is not clear, so it is likewise unclear how to exploit the hidden structure of the game beyond the separable case.

To that end, our point of departure is the observation that the Lyapunov property (6) of the energy function E ( θ ) is precisely the key feature that enables convergence of (NHGD). Thus, assuming that control variables are mapped to latent ones via a general - but possibly highly coupled - representation map χ : Θ →X , we will consider an abstract preconditioning scheme of the form

denotes the individual loss gradient of player i , while the preconditioning matrix P i ( θ i ) ∈ R m i × m i is to be designed so that (6) still holds under (7). In this regard, a straightforward calculation (which we prove in Appendix A) yields the following:

Lemma 1. Under the dynamics (7) , we have

where, as per Section 2, J i ( θ i ) ∈ R d i × m i denotes the Jacobian matrix of the map χ i : Θ i → X i . More compactly, letting P := ⊕ i P i denote the block-diagonal ensemble of the players' individual preconditioning matrices P i ( and suppressing control variable arguments for concision ) , we have

In view of Lemma 1, a direct way to achieve the target Lyapunov property (6) would be to find a preconditioning matrix P such that JPJ ⊺ = I . However, since J is surjective (by the faithfulness assumption for χ ), the Moore-Penrose inverse J + of J will be a right inverse to J , i.e., JJ + = I . Hence, letting P = ( J ⊺ J ) + = J + [ J + ] ⊺ , we obtain:

In this way, unraveling the above, we obtain the preconditioned hidden gradient flow

By virtue of design, the following property of (PHGF) is then immediate:

Proposition 1. Suppose that Γ admits a latent strictly monotone structure in the sense of (3b) . Then the energy function (5) is a strict Lyapunov function for (PHGF) , and every limit point θ ∗ of θ ( t ) is a Nash equilibrium of Γ .

where

Proposition 1 validates our design choices for the preconditioning matrix P as it illustrates that the dynamics (PHGF) converge to Nash equilibrium, without any separability requirements or dimensionality restrictions; in this regard, Proposition 1 already provides a marked improvement over the corresponding convergence result of [32] for (NHGD). To streamline our presentation, we defer the proof of Lemma 1 and Proposition 1 to Appendix A, and instead proceed directly to provide a bona fide, algorithmic implementation of (PHGF).

3.2. Preconditioned hidden gradient descent. In realistic machine learning problems, any algorithmic scheme based on (PHGF) will have to be run in an iterative, discrete-time environment; moreover, due to the challenges posed by applications with large datasets and optimization objectives driven by the minimization of empirical risk, we will need to assume that players only have access to a stochastic version of their full, deterministic gradients. As such, we will make the following blanket assumptions that are standard in the stochastic optimization literature [17, 21]:

Assumption 1. Each agent's loss function is an expectation of a random function L i : Θ × Ω → R over a complete probability space (Ω , F , P ) , i.e., ℓ i ( θ ) = E [ L i ( θ ; ω )] . We further assume that:

L i ( θ ; ω ) is measurable in ω and β -Lipschitz smooth in θ (for all θ ∈ Θ and ω ∈ Ω respectively).

The gradients of L i have bounded second moments, i.e., sup θ ∈ Θ E [ ∥∇ L i ( θ ; ω ) ∥ 2 ] ≤ M 2 .

Taken together, the two components of Assumption 1 imply that each ℓ i is differentiable and Lipschitz continuous (indeed, by Jensen's inequality, we have ∥ E [ ∇ L i ( θ ; ω )] ∥ 2 ≤ E [ ∥∇ L i ( θ ; ω ) ∥ 2 ] ≤ M 2 ). Moreover, by standard dominated convergence arguments, ∇ L i ( θ ; ω ) can be seen as an unbiased estimator of ∇ ℓ i ( θ ) , that is, ℓ i ( θ ) = E [ ∇ L i ( θ ; ω )] for all θ ∈ Θ . In view of this, we will refer to the individual loss gradients ∇ i L i ( θ ; ω ) of player i ∈ N as the player's individual stochastic gradients .

With all this in mind, the preconditioned hidden gradient descent algorithm is defined as the stochastic first-order recursion

where:

θ i,t denotes the control variable configuration of player i ∈ N at each stage t = 1 , 2 , . . .

γ t > 0 is a variable step-size sequence, typically of the form γ t ∝ 1 /t p for some p ∈ [0 , 1] .

P i,t := P i ( θ i,t ) = [ J i ( θ i,t ) ⊺ J i ( θ i,t )] + is the preconditioner of player i ∈ N at the t -th epoch.

V i,t := ∇ i L ( θ i,t ; ω t ) is an individual stochastic loss gradient generated at the control variable configuration θ t by an i.i.d. sample sequence ω t ∈ Ω , t = 1 , 2 , . . .

The basic recursion (PHGD) can be seen as a noisy Euler discretization of the continuous flow (PHGF), in the same way that ordinary stochastic gradient descent can be seen as a noisy discretization of gradient flows. In this way, (PHGD) is subject to the same difficulties underlying the analysis of stochastic gradient descent methods; we address these challenges in the next section.

4 Convergence analysis and results

In this section, we present our main results regarding the convergence of (PHGD) in hidden monotone games. Because we are interested not only on the asymptotic convergence of the method but also on its rate of convergence, we begin this section by defining a suitable merit function for each class of hidden monotone games under consideration; subsequently, we present our main results in

Section 4.2, where we also discuss the main technical tools enabling our analysis. To streamline our presentation, all proofs are deferred to Appendix B.

4.1. Merit functions and convergence metrics. In the variational context of (SVI) / (MVI), the quality of a candidate solution ˆ x is typically evaluated by means of the restricted merit function

where the 'test domain' C is a relatively open subset of X [3]. The raison d'être of this definition is that, if g is monotone and x ∗ ∈ C is a solution of (SVI) / (MVI), we have

so the supremum in (B.6) cannot be too positive if ˆ x is an approximate solution of (SVI) / (MVI). This is encoded in the following lemma, which, among others, justifies the terminology 'merit function':

Lemma 2. Suppose that g is monotone. If ˆ x is a solution of (SVI) / (MVI) , we have Gap C (ˆ x ) = 0 whenever ˆ x ∈ C . Conversely, if Gap C (ˆ x ) = 0 and C is a neighborhood of ˆ x in X , then ˆ x is a solution of (SVI) / (MVI) .

Lemma 2 extends similar statements by Auslender & Teboulle [3] and ? ]; the precise variant that we state above can be found in [ ? ], but, for completeness, we provide a proof in Appendix B.

Now, since a latent variable profile x ∗ = χ ( θ ∗ ) solves (SVI) if and only if the control variable configuration θ ∗ is Nash equilibrium of the base game, the quality of a candidate solution ˆ θ ∈ Θ with ˆ x = χ ( ˆ θ ) can be assessed by the induced gap function

Indeed, since ˆ x = χ ( ˆ θ ) ∈ X , Lemma 2 shows that Gap( ˆ θ ) ≥ 0 with equality if and only if ˆ x is a Nash equilibrium of the base game Γ . In this regard, Eq. (14) provides a valid equilibrium convergence metric for Γ , so we will use it freely in the sequel as such; for a discussion of alternative convergence metrics, we refer the reader to Appendix B.

4.2. Convergence results. We are now in a position to state our main results regarding the equilibrium convergence properties of (PHGD). To streamline our presentation, we present our results from coarser to finer, starting with games that admit a hidden merely monotone structure and stochastic gradient feedback, and refining our analysis progressively to games with a hidden strongly monotone structure and full gradient feedback. 3 In addition, to quantify this distortion between the game's latent and control layers, we will require a technical regularity assumption for the game's representation map χ : Θ →X .

Assumption 2. The singular values of the Jacobian J ( θ ) of the representation map χ are bounded as

for some σ min , σ max ∈ (0 , ∞ ) and for all θ ∈ Θ .

With all this in hand, we begin by studying the behavior of (PHGD) in games with a hidden monotone structure.

Theorem 1 (PHGD in hidden monotone games) . Suppose that players run (PHGD) in a hidden monotone game with learning rate γ t ∝ 1 /t 1 / 2 . Then, under Assumptions 1 and 2, the averaged process ¯ θ t ∈ χ -1 ( t -1 ∑ t s =1 x s ) enjoys the equilibrium convergence rate

As far as we are aware, Theorem 1 is the first result of its kind in the hidden games literature - that is, describing the long-run behavior of a discrete-time algorithm with stochastic gradient input. At the same time, it is subject to two important limitations: the first is that the averaged state ¯ θ t cannot be efficiently computed for general representation maps; second, even if it could, the O (log t/ √ t )

convergence rate is relatively slow. The two results that follow show that both limitations can be overcome in games with a hidden strongly monotone structure.

In this case (SVI)/(MVI) admits a (necessarily) unique solution x ∗ in the game's control space, so we will measure convergence in terms of the latent equilibrium distance

With this in mind, we have the following convergence results:

Theorem 2 (PHGD in hidden strongly monotone games) . Suppose that players run (PHGD) in a hidden µ -strongly monotone game with γ t = γ/t for some γ > µ . Then, under Assumptions 1 and 2, the induced sequence of play θ t ∈ Θ , t = 1 , 2 , . . . , enjoys the equilibrium convergence rate

This rate is tight, even for standard strongly monotone games. To improve it further, we will need to assume that (PHGD) is run with full, deterministic gradients, i.e., V t = g ( θ t ) . In this case, we obtain the following refinement of Theorem 2.

Theorem 3 (PHGD with full gradient feedback in hidden strongly monotone games) . Suppose that players run (PHGD) in a hidden µ -strongly monotone game with full gradient feedback, and a suffciently small learning rate γ > 0 . Then, under Assumptions 1 and 2, the induced sequence of play θ t ∈ Θ , t = 1 , 2 , . . . , converges to equilibrium at a geometric rate, i.e.,

for some constant ρ ∈ (0 , 1) that depends only on the primitives of Γ and the representation map χ .

Importantly, up to logarithmic factors, the convergence rates of Theorems 1-3 mirror the corresponding rates for learning in monotone games. This take-away is particularly important as it shows that, when it exists, a hidden convex structure can be exploited to the greatest possible degree, without any loss of speed in convergence relative to standard, non-hidden convex problems.

The proofs of Theorems 1-3 are quite involved, so we defer the details to Appendix B. That said, to give an idea of the technical steps involved, we provide below (without proof) two lemmas that play a pivotal role in our analysis. The first one hinges on a transformation of the problem's defining vector field v ( θ ) = ( ∇ i ℓ i ( θ )) i ∈N which, coupled with the specific choice of preconditioner P i ( θ i ) = [ J i ( θ i ) ⊺ J i ( θ i )] + in (PHGD) allows us to effectively couple the latent and control layers of the problem in a 'covariant' manner:

Lemma 3. Fix some ˆ x ∈ X , and consider the energy function E ( θ ; ˆ x ) = (1 / 2) ∥ χ ( θ ) -ˆ x ∥ 2 . Then, for all θ ∈ Θ , we have

Our second intermediate result builds on Lemma 3 and provides a 'template inequality' for the energy function E ( θ ; ˆ x ) in the spirit of [7, 11, 18].

Lemma 4 (Template inequality) . Suppose that Assumptions 1 and 2 hold. Then, with notation as in Lemma 3, the sequence E t := (1 / 2) ∥ χ ( θ t ) -ˆ x ∥ 2 , t = 1 , 2 , . . . , satisfies

where x t := χ ( θ t ) , ϕ t := ( J ( θ t ) P ( θ t ) V t -g ( x t )) ⊺ ( x t -ˆ x ) and ψ t is a random error sequence with sup t E [ ψ t ] < ∞ .

This inequality plays a pivotal role in our analysis because it allows us to couple the restricted merit function (B.8) in the game's control space with the evolution of the algorithm's quasi-Lyapunov function in the game's latent space. We provide the relevant details and calculations in Appendix B.

5 Experiments

This section demonstrates our method's applicability in a couple of different and insightful setups. Technical details of those setups, as well as additional experimental results are deferred to the supplementary material. We start with a regularized version of Matching Pennies zero-sum game where the players' strategies are controlled by two individual preconfigured differentiable MLPs. Each MLP acts as the player's representation map χ i , which for each input θ i outputs a uni-dimensional latent variable x i = χ i ( θ i ) guaranteed to lie in X i ≡ [0 , 1] ; the player's latent space.

Figure 4 illustrates the trajectory of (PHGD), represented by the black curve. The algorithm employs a constant step-size of 0 . 01 and is initialized at the arbitrary state (1 . 25 , 2 . 25) in the control variables' space. The color map in the figure serves as a visual representation of the level sets associated with the proposed energy function (5). Notably, the trajectory of the algorithm intersects each of the energy function's level sets at most once, indicating its non-cycling behavior, which is an issue that shows up often in the equilibration task. Due to the design of our hidden maps, the stabilization at the point (0 , 0) corresponds to the ( χ 1 (0) , χ 2 (0)) = ( 1 2 , 1 2 ) the unique equilibrium of the game.

In the second, more complex, example, we consider a strongly-monotone regularized modification of an (atomic) El Farol Bar congestion game among N = 30 players [1,

-5

0

5

-5

-4

-3

-2

-1

0

1

2

3

4

5

44]. In this setup, we let the control space of each player i be multi-dimensional, namely, Θ i ≡ R 5 , i ∈ N , and, as in the previous example, the representation map of each player is instantiated by some preconfigured differentiable MLP whose output x i := χ i ( θ i ) is guaranteed to lie in [0 , 1] . The MLP's output is going to be the probability with which player i visits the El Farol bar. For the interested reader, further details, including technical specification of the MLPs, and loss functions of the games, can be found in the supplementary material. Figure 5 provides a comparative analysis of

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Time (

t

)

10 -15

10 -10

10 -5

10 0

PGD

0

100

200

300

400

500

600

700

800

900

1000

Time (

t

)

10 -1

10 0

PGD

the performance between (PHGD), and the standard GD method in the aforementioned two game scenarios. In Figure 5 (left) we explore the Matching Pennies game, where GD exhibits slightly erratic behavior. Despite eventually converging to the game's equilibrium point, GD's convergence rate, in this case, can be described as linear at best. In Figure 5 (right), we examine the El Farol Bar game. Interestingly, in this highly complex setup, GD fails to reach the equilibrium point entirely. In contrast, (PHGD) not only manages to converge in both of these setups, but it also consistently maintains an exponential rate of convergence. This stark difference underscores the efficacy and robustness of our algorithm.

6 Conclusion

This paper proposed a new algorithmic framework with strong formal convergence guarantees in a general class non-convex games with a latent monotone structure. Our algorithmic method which we call preconditioned hidden gradient descent - relies on an appropriately chosen gradient preconditioning scheme akin to natural gradient ideas. Our class of games combines the useful structure of monotone operators as well as the notion of latent/hidden variables that arise in neural networks and can thus model numerous AI applications. Our results indicate the possibility of deep novel algorithmic ideas emerging at the intersection of game theory, non-convex optimization and ML and offers exciting directions for future work.

Acknowledgments and Disclosure of Funding

This research has been partially supported by the National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Program (AISG Award No: AISG2-RP-2020-016), grant PIESGP-AI-2020-01, AME Programmatic Fund (Grant No.A20H6b0151) from A*STAR and Provost's Chair Professorship grant RGEPPV2101, project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program, and the French National Research Agency (ANR) in the framework of the PEPR IA FOUNDRY project (ANR-23-PEIA-0003), the 'Investissements d'avenir' program (ANR-15-IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), and MIAI@Grenoble Alpes (ANR-19-P3IA-0003). PM is also with the Archimedes Research Unit - Athena RC - University of Athens.

References

Arthur, W. B. Complexity in economic and financial markets. Complexity , 1(1):20-25, 1995.

Auslender, A. Optimisation: Méthodes numériques . Masson, 1976.

Auslender, A. and Teboulle, M. Interior projection-like methods for monotone variational inequalities. Mathematical Programming , 104:39-68, 2005.

Bakhtin, A., Wu, D. J., Lerer, A., Gray, J., Jacob, A. P., Farina, G., Miller, A. H., and Brown, N. Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. arXiv preprint arXiv:2210.05492 , 2022.

Beltratti, A., Margarita, S., Terna, P., et al. Neural networks for economic and financial modelling . International Thomson Computer Press London, UK, 1996.

Bichler, M., Fichtl, M., Heidekrüger, S., Kohring, N., and Sutterer, P. Learning equilibria in symmetric auction games using artificial neural networks. Nat. Mach. Intell. , 3(8):687-695, 2021. doi: 10.1038/ s42256-021-00365-4. URL https://doi.org/10.1038/s42256-021-00365-4 .

Bravo, M., Leslie, D. S., and Mertikopoulos, P. Bandit learning in concave N -person games. In NeurIPS '18: Proceedings of the 32nd International Conference of Neural Information Processing Systems , 2018.

Chung, K.-L. On a stochastic approximation method. The Annals of Mathematical Statistics , 25(3): 463-483, 1954.

Daskalakis, C., Goldberg, P. W., and Papadimitriou, C. H. The complexity of computing a Nash equilibrium. Communications of the ACM , 52(2):89-97, 2009.

Daskalakis, C., Skoulakis, S., and Zampetakis, M. The complexity of constrained min-max optimization. In STOC '21: Proceedings of the 53rd annual ACM SIGACT symposium on the Theory of Computing , 2021.

Duvocelle, B., Mertikopoulos, P., Staudigl, M., and Vermeulen, D. Multi-agent online learning in timevarying games. Mathematics of Operations Research , 48(2):914-941, May 2023.

Facchinei, F. and Pang, J.-S. Finite-Dimensional Variational Inequalities and Complementarity Problems . Springer Series in Operations Research. Springer, 2003.

Gidel, G., Balduzzi, D., Czarnecki, W. M., Garnelo, M., and Bachrach, Y. Minimax theorem for latent games or: How i learned to stop worrying about mixed-nash and love neural nets. AISTATS , 2021.

Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NIPS '14: Proceedings of the 28th International Conference on Neural Information Processing Systems , 2014.

Hart, S. and Mas-Colell, A. Uncoupled dynamics do not lead to Nash equilibrium. American Economic Review , 93(5):1830-1836, 2003.

Hart, S. and Mas-Colell, A. Stochastic uncoupled dynamics and Nash equilibrium. Games and Economic Behavior , 57:286-303, 2006.

Hazan, E. A survey: The convex optimization approach to regret minimization. In Sra, S., Nowozin, S., and Wright, S. J. (eds.), Optimization for Machine Learning , pp. 287-304. MIT Press, 2012.

Hsieh, Y.-G., Iutzeler, F., Malick, J., and Mertikopoulos, P. On the convergence of single-call stochastic extra-gradient methods. In NeurIPS '19: Proceedings of the 33rd International Conference on Neural Information Processing Systems , pp. 6936-6946, 2019.

Hsieh, Y.-P., Mertikopoulos, P., and Cevher, V. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In ICML '21: Proceedings of the 38th International Conference on Machine Learning , 2021.

Kelly, F. P., Maulloo, A. K., and Tan, D. K. H. Rate control for communication networks: shadow prices, proportional fairness and stability. Journal of the Operational Research Society , 49(3):237-252, March 1998.

Lan, G. First-order and Stochastic Optimization Methods for Machine Learning . Springer, 2020.

Lee, J. M. Riemannian Manifolds: an Introduction to Curvature . Number 176 in Graduate Texts in Mathematics. Springer, 1997.

Lee, J. M. Introduction to Smooth Manifolds . Number 218 in Graduate Texts in Mathematics. SpringerVerlag, New York, NY, 2003.

Letcher, A. On the impossibility of global convergence in multi-loss optimization. In ICLR '21: Proceedings of the 2021 International Conference on Learning Representations , 2021.

Liu, X., Yu, C., Zhang, Z., Zheng, Z., Rong, Y., Lv, H., Huo, D., Wang, Y., Chen, D., Xu, J., Wu, F., Chen, G., and Zhu, X. Neural auction: End-to-end learning of auction mechanisms for e-commerce advertising. In Zhu, F., Ooi, B. C., and Miao, C. (eds.), KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021 , pp. 3354-3364. ACM, 2021. doi: 10.1145/3447548.3467103. URL https://doi.org/10.1145/3447548.3467103 .

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR '18: Proceedings of the 2018 International Conference on Learning Representations , 2018.

Mertikopoulos, P. and Moustakas, A. L. Learning in an uncertain world: MIMO covariance matrix optimization with imperfect feedback. IEEE Trans. Signal Process. , 64(1):5-18, January 2016.

Mertikopoulos, P. and Zhou, Z. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1-2):465-507, January 2019.

Mertikopoulos, P., Belmega, E. V., Negrel, R., and Sanguinetti, L. Distributed stochastic optimization via matrix exponential learning. IEEE Trans. Signal Process. , 65(9):2277-2290, May 2017.

Mertikopoulos, P., Hsieh, Y.-P., and Cevher, V. A unified stochastic approximation framework for learning in games. Mathematical Programming , forthcoming, 2023.

Milionis, J., Papadimitriou, C., Piliouras, G., and Spendlove, K. An impossibility theorem in game dynamics. Proceedings of the National Academy of Sciences , 120(41):e2305349120, 2023.

Mladenovic, A., Sakos, I., Gidel, G., and Piliouras, G. Generalized natural gradient flows in hidden convex-concave games and gans. In International Conference on Learning Representations , 2021.

Nikaido, H. and Isoda, K. Note on non-cooperative convex games. Pacific Journal of Mathematics , 5: 807-815, 1955.

Orda, A., Rom, R., and Shimkin, N. Competitive routing in multi-user communication networks. IEEE/ACM Trans. Netw. , 1(5):614-627, October 1993.

Perolat, J., De Vylder, B., Hennes, D., Tarassov, E., Strub, F., de Boer, V ., Muller, P., Connor, J. T., Burch, N., Anthony, T., et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science , 378(6623):990-996, 2022.

Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Robust adversarial reinforcement learning. In ICML '17: Proceedings of the 34th International Conference on Machine Learning , 2017.

Rosen, J. B. Existence and uniqueness of equilibrium points for concave N -person games. Econometrica , 33(3):520-534, 1965.

Scutari, G., Facchinei, F., Palomar, D. P., and Pang, J.-S. Convex optimization, game theory, and variational inequality theory in multiuser communication systems. IEEE Signal Process. Mag. , 27(3):35-49, May 2010.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Singla, S. and Feizi, S. Fantastic four: Differentiable and efficient bounds on singular values of convolution layers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=JCRblSgs34Z .

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gulcehre, C., Wang, Z., Pfaff, T., Wu, Y ., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575:350-354, 2019.

Vlatakis-Gkaragkounis, E.-V., Flokas, L., and Piliouras, G. Poincaré recurrence, cycles and spurious equilibria in gradient-descent-ascent for non-convex non-concave zero-sum games. In Advances in Neural Information Processing Systems , pp. 10450-10461, 2019.

Vlatakis-Gkaragkounis, E.-V., Flokas, L., and Piliouras, G. Solving min-max optimization with hidden structure via gradient descent ascent. Advances in Neural Information Processing Systems , 34:2373-2386, 2021.

Whitehead, D. et al. The el farol bar problem revisited: Reinforcement learning in a potential game. ESE discussion papers , 186, 2008.

Zhang, G., Martens, J., and Grosse, R. B. Fast convergence of natural gradient descent for overparameterized neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Organization of the appendix

A Omitted proofs from Section 3

In this first appendix, we provide the technical proofs of our results for the continuous-time dynamics (PHGF), namely Lemma 1 and Proposition 1. For convenience, we restate the relevant results as needed.

Lemma 1. Under the dynamics (7) , we have

where, as per Section 2, J i ( θ i ) ∈ R d i × m i denotes the Jacobian matrix of the map χ i : Θ i → X i . More compactly, letting P := ⊕ i P i denote the block-diagonal ensemble of the players' individual preconditioning matrices P i ( and suppressing control variable arguments for concision ) , we have

Proof. Observe that if we expand the dynamical system of equations in (PHGF) with respect to each individual coordinate, we have that for each player i ∈ N , for all profiles θ ∈ Θ , and for all coordinates α ∈ { 1 , . . . , m i } :

Now, since Γ has a hidden monotone structure, and due to the decomposition ℓ i ( θ ) = f i ( x ) , we can expand the above as

Furthermore, if we expand the left-hand side (LHS) of (9) we also get

where each of the summands of the above expression can also be expanded further to

Putting everything together (9) follows by some trivial substitutions.

■

Proposition 1. Suppose that Γ admits a latent strictly monotone structure in the sense of (3b) . Then the energy function (5) is a strict Lyapunov function for (PHGF) , and every limit point θ ∗ of θ ( t ) is a Nash equilibrium of Γ .

Proof. First, observe that for any player i ∈ N , and θ i ∈ Θ i , since the player's representation map χ i is faithful, i.e., J i ( θ i ) is maximal rank, it holds:

That is, ∑ m i α =1 ∑ m i β =1 ∂x il ∂θ iα ∂x ir ∂θiβ P iαβ ( θ i ) = δ lr for all l, r ∈ { 1 , . . . , d i } , where δ lr is the Kronecker delta.

Now, by Lemma 1 we have that:

which is negative, since x ∗ is an optimizer of f , i.e., it satisfies the (MVI) due to monotonicity of g . Additionally, since g is strictly monotone, x ∗ is the unique optimizer of f , and, therefore, the above condition holds with equality if and only if x = x ∗ . ■

B Auxiliary results from Section 4

B.1. Convergence metrics and merit functions. In this appendix, we provide the technical scaffolding required for the analysis of(PHGD), namely Lemmas 2-4 in Section 3.2. As before, we restate the relevant results as needed.

Lemma 2. Suppose that g is monotone. If ˆ x is a solution of (SVI) / (MVI) , we have Gap C (ˆ x ) = 0 whenever ˆ x ∈ C . Conversely, if Gap C (ˆ x ) = 0 and C is a neighborhood of ˆ x in X , then ˆ x is a solution of (SVI) / (MVI) .

Proof of Lemma 2. Let ˆ x ∈ X be a solution of (SVI)/(MVI) so ⟨ g (ˆ x ) , x -ˆ x ⟩ ≥ 0 for all x ∈ X . Then, by the monotonicity of g , we get:

so Gap C (ˆ x ) ≤ 0 . On the other hand, if ˆ x ∈ C , we also get Gap(ˆ x ) ≥ ⟨ g (ˆ x ) , ˆ x -ˆ x ⟩ = 0 , so we conclude that Gap C (ˆ x ) = 0 .

For the converse statement, assume that Gap C (ˆ x ) = 0 for some ˆ x ∈ C and suppose that C contains a neighborhood of ˆ x in X . We then claim that ˆ x is a solution of (MVI) over C , i.e.,

To see this, assume to the contrary that there exists some x 1 ∈ C such that

so, in turn, we get a contradiction.

With this intermediate 'local' result in hand, we are now in a position to prove that ˆ x solves (SVI). Indeed, if we suppose to the contrary that there exists some z 1 ∈ X such that ⟨ g (ˆ x ) , z 1 -ˆ x ⟩ < 0 , then, by the continuity of g , there exists a neighborhood U ′ of ˆ x in X such that

Hence, assuming without loss of generality that U ′ ⊂ U ⊂ C (the latter assumption due to the assumption that C contains a neighborhood of ˆ x ), and taking λ > 0 sufficiently small so that x = ˆ x + λ ( z 1 -ˆ x ) ∈ U ′ , we get that ⟨ g ( x ) , x -ˆ x ⟩ = λ ⟨ g ( x ) , z 1 -ˆ x ⟩ < 0 , in contradiction to (B.2). We thus conclude that ˆ x is a solution of (SVI) - and hence, by monotonicity, also of (MVI). ■

For intuition, we discuss below some other merit functions that could be considered as valid convergence metrics. In this regard, the first thing to note is that the definition of Gap( ˆ θ ) effectively goes through the game's latent space , so it is natural to ask whether a similar merit function can be defined directly on the game's control space . To do so, it will be more convenient to start with a linearized variant of Gap X defined over the cone of tangent directions at ˆ x , namely the so-called 'tangent residual gap'

i.e., the maximum 'ascent' step ⟨-g ( x ) , z ⟩ over all admissible displacement directions z from ˆ x . 4 Just like Gap C , this linearized merit function correctly identifies solutions of (SVI)/(MVI):

Lemma B.1. For all ˆ x ∈ X , we have TGap X (ˆ x ) ≥ 0 with equality if and only if ˆ x solves (SVI) / (MVI) .

Proof. Let ˆ x ∈ X be some arbitrary profile of latent variables. By definition, we have that TGap X (ˆ x ) = -min z ∈ TC X (ˆ x ) , ∥ z ∥≤ 1 ⟨ g (ˆ x ) , z ⟩ , where TC X (ˆ x ) is the tangent cone to X at ˆ x . Observe that since TC X (ˆ x ) is a cone, 0 ∈ TC X (ˆ x ) , so TGap X (ˆ x ) ≥ -⟨ g (ˆ x ) , 0 ⟩ = 0 . Now assume that TGap X (ˆ x ) = 0 . Then, the following are equivalent:

where the last equivalence follows because TC X (ˆ x ) is a cone. Rearranging the terms, we may equivalently write ⟨-g (ˆ x ) , z ⟩ ≤ 0 for all z ∈ TC X (ˆ x ) . Notice that, by the definition of the tangent cone, the latter is equivalent to ⟨-g (ˆ x ) , ˆ x -z ⟩ ≤ 0 for all z ∈ X . Therefore, TGap X (ˆ x ) = 0 if and only if ˆ x satisfies (SVI). ■

Now, if θ ∗ is a Nash equilibrium of the base game, the latent variable configuration x ∗ = χ ( θ ∗ ) solves (SVI)/(MVI), so TGap X is a valid equilibrium convergence metric for Γ . However, since TGap X is still defined on the game's latent space, it does not give a straightforward way of definining the quality of a candidate solution directly on the game's control space. To that end, one could consider the gap function

where v ( θ ) := ( ∇ i ℓ i ( θ )) i ∈N collects the players' loss gradients relative to their control variables, and ∥·∥ ∗ denotes its dual norm (so TGap Θ ( ˆ θ ) is evaluated directly on the game's control space). Of course, as control variables are mapped to latent variables, χ introduces a certaint distortion (due to nonlinearities). This distortion can be quantified by the following lemma (see also Fig. 6 above):

Lemma B.2. Let ˆ x = χ ( ˆ θ ) for some ˆ θ ∈ Θ . We then have

In particular, TGap Θ ( ˆ θ ) ≥ 0 for all ˆ θ ∈ Θ , with equality if and only if ˆ θ is a Nash equilibrium of Γ .

J

i

(

θ

)

σ

min

(

θ

)

σ

(

θ

)

∥

z

σ

min

∥

z

σ

∥

η

1

Proof. To begin with, let us define the balls B Θ ≡ { η ∈ R m : ∥ η ∥ ≤ 1 } , and B X ≡ { z ∈ R d : ∥ z ∥ ≤ 1 } . Now, let ˆ θ ∈ Θ be some arbitrary profile of control variables. By Assumption 2, we have that the singular values of the Jacobian J ( ˆ θ ) of the representation map χ are bounded from above and below by σ max < ∞ and σ min > 0 respectively. That is, σ min ∥ η ∥ ≤ ∥ J ( ˆ θ ) η ∥ ≤ σ max ∥ η ∥ for all η ∈ R m , which implies that σ min B X ⊆ J ( ˆ θ ) B Θ ⊆ σ max B X . By definition, we also have that, v ( ˆ θ ) = J ( ˆ θ ) ⊺ g ( χ ( ˆ θ ) ) = J ( ˆ θ ) ⊺ g (ˆ x ) , and, hence

Now, recall that the set of Nash equilibria Θ ∗ ⊂ Θ is non-empty. In particular, since the Θ ≡ R m is an open set, the solution set lies in an open set, which, in conjunction with the fact that the representation maps are faithful representations of Θ to X , also implies that χ (Θ ∗ ) lies in the interior of X . Finally, since χ is smooth, and in conjunction with the previous observations, it follows that TGap Θ ( ˆ θ ) = ∥ v ( ˆ θ ) ∥ and TGap X (ˆ x ) = ∥ g (ˆ x ) ∥ , hence it is easy to see from the above discussion that

and our proof is complete.

■

In view of the above discussion, the reader may wonder why not use the tangent gap TGap Θ directly as a performance metric. The reason for this is that, if the solutions x ∗ of (SVI)/(MVI) do not belong to the image of χ , the dual norm of v ( ˆ θ ) may be too stringent as a performance metric as it does not vanish near an equilibrium (e.g., think of the operator g ( x ) = x for x between 0 and 1 ). In this case, equilibria can be approximated to arbitrary accurarcy but never attained, so the latent gap functions Gap and Err are more appropriate as performance measures.

B.2. Convergence analysis and template inequalities.

Lemma 3. Fix some ˆ x ∈ X , and consider the energy function E ( θ ; ˆ x ) = (1 / 2) ∥ χ ( θ ) -ˆ x ∥ 2 . Then, for all θ ∈ Θ , we have

Proof. Let i ∈ N be some arbitrary player, let θ ∈ Θ be some arbitrary profile of control variables, and let η ∈ R d i be some arbitrary vector. Then for each coordinate α ∈ { 1 , . . . , m i } , and latent profile ˆ x ∈ X , we have that

That is ∇ θ i E ( y ; ˆ x ) = J i ( θ i ) ⊺ ( x i -ˆ x i ) . Now, we can multiply both sides of the above expression with P i ( θ i ) to get

where in the last equality we used the fact that the i -th player's representation map χ i is a faithful representation, i.e., J i ( θ i ) is maximal rank. Finally, by expanding the LHS of (20), and applying the above simplification the result follows. ■

Lemma 4 (Template inequality) . Suppose that Assumptions 1 and 2 hold. Then, with notation as in Lemma 3, the sequence E t := (1 / 2) ∥ χ ( θ t ) -ˆ x ∥ 2 , t = 1 , 2 , . . . , satisfies

where x t := χ ( θ t ) , ϕ t := ( J ( θ t ) P ( θ t ) V t -g ( x t )) ⊺ ( x t -ˆ x ) and ψ t is a random error sequence with sup t E [ ψ t ] < ∞ .

Proof. For each t = 1 , 2 , . . . we expand x t +1 using its second-order Taylor approximation at θ t ; i.e., for each player i = 1 , . . . , N and each coordinate l = 1 , . . . , d i :

where H il ( ˜ θ i,t ) is the Hessian of the latent map χ il at some convex combination ˜ θ i,t of θ i,t and θ i,t +1 . Then, further expanding θ t +1 , we also get

Plugging the above expansion to E ( θ t +1 ; ˆ x ) for arbitrary state ˆ x we get

where the second-order term η t is given by the formula

For the second-order term, observe that by Assumption 2, we have that [ J ⊺ t J t ] + ⪯ 1 σ 2 min I . Using that fact, it follows that

Furthermore, since the representation maps χ i are β -Lipschitz smooth for some Lipschitz modulus β , we have that H i,l ( ¯ θ i,t ) ⪯ β I for each player i and coordinate l . Consequently, we have that

Let D = diam( χ ) , then, by applying the Cauchy-Schwarz inequality, we also get that

where γ = sup t =1 , 2 ,... γ t , and d = ∑ N i =1 d i .

Finally, let us consider the first-order term of (B.16). Let v t = ( ∇ i ℓ i ( θ t )) i ∈N , and g t = g ( x t ) . Then, by definition, we also have that v t = J ⊺ t g t . Moreover, recall that, by construction, J t P t J ⊺ t = I . Consequently, we can write

which concludes the proof.

C Proofs of Theorems 1-3

We are now in a position to prove Theorems 1-3 regarding the convergence properties of (PHGD). We proceed sequentially, restating the relevant results as needed.

Theorem 1 (PHGD in hidden monotone games) . Suppose that players run (PHGD) in a hidden monotone game with learning rate γ t ∝ 1 /t 1 / 2 . Then, under Assumptions 1 and 2, the averaged process ¯ θ t ∈ χ -1 ( t -1 ∑ t s =1 x s ) enjoys the equilibrium convergence rate

Proof. Let θ 1 ∈ Θ be some arbitrary initialization of the algorithm, and let ˜ x ∈ X be some arbitrary profile of latent variables. Next, by Lemma 4, we have that for all t ≥ 1 :

Rearranging the terms, the above is equivalent to

Furthermore, by the monotonicity of g , we also have that 〈 g ( x t ) -g (˜ x ) , x t -˜ x 〉 ≥ 0 , and by combining the two, we get that for all t ≥ 1 :

■

Summing up the above terms in both sides of the inequality, we also get that

Dividing all terms by ˜ γ t := ∑ t s =1 γ s , also yields

where ¯ x t := ∑ t s =1 γ s ˜ γ t x s is the time-average.

Next, let us define the following auxiliary process that will assist us in further bounding the above expression:

where η t := J ( θ t ) P ( θ t ) V t -g ( x t ) for all i ∈ N . Observe that

Let us, first, focus on the former of the two terms. Notice that for all t ≥ 1 , we can write

Furthermore, by the optimality of y t +1 , t ≥ 2 , we have also have that

That is, γ t ⟨ η t , y t +1 -˜ x ⟩ ≤ ⟨ y t -y t +1 , y t +1 -˜ x ⟩ .

Let us also recall a couple of useful quadratic identities, namely, we have that ∥ y t -˜ x ∥ 2 = ∥ y t -y t +1 + y t +1 -˜ x ∥ 2 = ∥ y t -y t +1 ∥ 2 +2 ⟨ y t -y t +1 , y t +1 -˜ x ⟩ + ∥ y t +1 -˜ x ∥ 2 , and ∥ γ t η t -( y t -y t +1 ) ∥ 2 = γ 2 t ∥ η t ∥ 2 -2 γ t ⟨ η t , y t -y t +1 ⟩ + ∥ y t -y t +1 ∥ 2 . Then, in conjunction with the above, we get that for all t ≥ 1 :

Finally, summing both sides of the above inequalities, we also get that

With the above established, let us reconsider the quantity 〈 g (˜ x ) , ¯ x t -˜ x 〉 . Specifically, due to the above derivations, we have that for all t ≥ 1 :

Considering the mean of supremum over ˜ x ∈ X of the LHS, we end up with

where the last inequality is direct consequence of the bounded second moment of the stochastic gradient in Assumption 1 ■

Theorem 2 (PHGD in hidden strongly monotone games) . Suppose that players run (PHGD) in a hidden µ -strongly monotone game with γ t = γ/t for some γ > µ . Then, under Assumptions 1 and 2, the induced sequence of play θ t ∈ Θ , t = 1 , 2 , . . . , enjoys the equilibrium convergence rate

Proof. Let θ 1 ∈ Θ be some arbitrary initialization of the algorithm, and let γ ≥ 1 2 µ be arbitrary. Since the map f is µ -strongly monotone, we have, by the definition of strong monotonicity, that

Next, by Lemma 4, we have that for all t ≥ 1 :

Let H t = { θ s : s = 1 , . . . , t } be the history of play up to iteration t . In expectation, it follows from the above that, for all t ≥ 1 :

Note that x t is H t -measurable; hence, we have that

Furthermore, since, by Assumption 1, we have that E [ ∥ V t ∥ 2 | H t ] ≤ M 2 , it also holds that

Next, by some simple substitutions, we have

Finally, since γ > 1 2 µ , we may apply Chung's lemma [8] to get that, for all t ≥ 1 :

■

Theorem 3 (PHGD with full gradient feedback in hidden strongly monotone games) . Suppose that players run (PHGD) in a hidden µ -strongly monotone game with full gradient feedback, and a suffciently small learning rate γ > 0 . Then, under Assumptions 1 and 2, the induced sequence of play θ t ∈ Θ , t = 1 , 2 , . . . , converges to equilibrium at a geometric rate, i.e.,

for some constant ρ ∈ (0 , 1) that depends only on the primitives of Γ and the representation map χ .

Proof. Due to the absence of randomness, and therefore the absence of noise, Lemma 4 may be simplified to

for some constants κ > 0 , where v t := ( ∇ i ℓ i ( θ t )) i ∈N .

Let β be the modulus of Lipschitz continuity of the gradients v t = J ( θ t ) ⊺ g ( x t ) , and let µ 2 be the modulus of the strong monotonicity of g . Recall that, by Assumption 2, we have that the singular

That is

values of the Jacobian J ( θ ) of the representation map χ are bounded from above and below by σ max < ∞ and σ min > 0 respectively. Therefore, since v t is β -Lipschitz, it follows that

In conjunction to the above, we then also have

Restricting γ t such that

we finally have that, for the choice of γ t = ˆ γ := σ 2 min µ 8 κβ 2 , it holds

Leveraging the equivalence of Err Θ ( θ t ) and Err X ( x t ) = E t completes the proof.

D Experiments

This section demonstrates the method's applicability in a series of different applccations. Along with revisiting the established examples of Section 5 in greater detail, we also present how the PHGD method performs in a couple of additional settings of interest. We begin with a high-level description of the common test suite, and afterward, we move to the definitive details of each of the presented applications.

In each example, we define a base game Γ among two or more players N ≡ { 1 , . . . , N } . Each player i control a m -dimensional vector of control variables θ i ∈ R m , which they feed in an individual differentiable preconfigured MLP with two hidden layers that act as the player's representation map χ i : R m → X i . The dimensions of each layer of the MLPs vary among the different examples. However, the MLP's output x i = χ i ( θ i ) is guaranteed to be a representation of a discrete probability distribution among the player's actions in some normal-form game, e.g., a Rock-Paper-Scissors game. The actual latent game G of Γ , is given by the 'hidden' loss functions f i : X → R , i ∈ N . Each loss function f i is a regularized variation of the aforementioned normal-form game, tuned by some µ 2 -strongly convex regularizer h µ ( x ) = µ 2 · (∑ i ∈N ∥ x i -x ∗ i ∥ 2 ) , where x ∗ is the normal-form game's unique equilibrium point. The modulus µ is a hyper-parameter, which we tune such that the vector field g ( x ) = ( ∇ x i f i ( x ) ) i ∈N to be strongly monotone, and to avoid finite-precision numerical errors, which can potentially arise during the computation of x .

A Hidden Game of Matching Pennies. As a first example, we revisit the two-player hidden game of Matching Pennies we introduced in Section 5. Here, each player's i control variables θ i are uni-dimensional, and are fed to each player's individual MLP given by the representation maps

where α (1) i , α (2) i ∈ [ -1 , 1] are randomly chosen. Although the definition of the activation functions sigmoid : R → [0 , 1] , and CeLU : R → ( -1 , ∞ ) are widely common, we give them below for reference; that is:

■

-5

0

5

-5

-4

-3

-2

-1

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-5

0

5

-5

-4

-3

-2

-1

0

1

2

3

4

5

0

0.2

0.4

0.6

0.8

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Time (

t

)

10 -15

10 -10

10 -5

10 0

PGD

In this, and the following examples, without any loss of the generality, we deliberately set the bias of each of the MLP's hidden layers to zero, in order for the base game's unique equilibrium to lie in θ ∗ = ⃗ 0 , and to simplify the notation. Each of the MLP's output x i = χ ( θ i ) is the i -th player's probability of playing Heads in a regularized game of Matching Pennies, given by the loss functions

The latent game's unique equilibrium, in this case, is at x = χ ( θ ) = ( , ) .

In this particular example it is possible to visualize the trajectories of PHGD and gradient descent with reference, both, the space of control variables R 2 , and the space of latent variables [0 , 1] 2 . In Fig. 7a we depict the trajectory of PHGD, with step-size 0 . 01 , in the above game, initialized at the arbitrary state (1 . 25 , 2 . 25) with respect to, both, the space of control variables (left), and the space of latent variables (right), over the sub-level sets of the energy function in (5). A similar trajectory of the GD algorithm is depicted in Fig. 7b. Observe that the PHGD's crosses the sub-level sets of the energy function at most once until it asymptotically reaches the game's equilibrium point at θ ∗ , as opposed to the trajectory of GD, which exhibits an erratic behavior. In particular, as is depicted in the semi-log plot in Fig. 7c, PHGD converges to the game's equilibrium point at an exponential rate as opposed to the rate of GD, which, at best, can be described as linear.

A Hidden Game of Rock-Paper-Scissors. In the next example we consider a regularized hidden game of Rock-Paper-Scissors between two players. In a standard Rock-Paper-Scissors game each player may choose among three strategies, namely, Rock, Paper, or Scissors. No strategy dominates over the others, with Rock ruling over Scissors, Paper ruling over Rock, and Scissors ruling over Paper. In this example, each player i controls a 5 -dimensional vector θ i ∈ R 5 of control variables, which, once again, they may feed their individual differentiable preconfigured MLP, whose output x i = χ i ( θ i ) lies in the 2 -dimensional simplex that encodes the space of probability distributions among the three strategies of the latent game. The two MLPs are given by the following representation maps:

where the matrices A (1) i ∈ [ -1 , 1] 4 × 5 , and A (2) i ∈ [ -1 , 1] 3 × 4 are random, and the activation function CeLU : R → ( -1 , ∞ ) , given as in (D.2b), is applied pair-wise. The definition of softmax : R d → ∆ d -1 , where ∆ d -1 is the ( d -1) -dimensional simplex is given, for reference, by

In this case, the latent game's loss functions are given by the following polynomial system of equations, whose unique equilibrium lies at the uniform distributions x ∗ i = χ i ( θ ∗ i ) = ( 1 3 , 1 3 , 1 3 ) , i = 1 , 2 :

Although, in this example, it is not possible to visualize the trajectories of PHGD and GD in the space of control variables due to the large dimensionality of the base game, we may still get a glimpse of the trajectories' behavior in the space of latent variables. Fig. 8a depicts an arbitrary trajectory of PHGD with step-size 0 . 01 , as it evolves in the simplices of the two players. Notice, that the trajectory, clearly, converges to the equilibrium of the latent game. A trajectory of GD, with the same step size and initialization point, is depicted in Fig. 8b. In this case, the erratic behavior of GD is more apparent than in the previous example. A more in-depth comparison of the algorithms in the current setup is depicted in the semi-log plot of Fig. 8c, where we visualize the mean distance, and confidence bounds, to the equilibrium point across 100 random trajectories of the two algorithms. Observe, that PHGD exhibits an exponential rate of convergence, as opposed to the linear rate of GD.

A Hidden Shapley's Game. The next game we are interested in is a hidden Shapley's game. The standard Shapley's game is a two-player normal-form game with payoff matrices:

R

P

S

R

P

S

R

P

S

R

P

S

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Time (

t

)

10 -10

10 -5

10 0

PGD

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Time (

t

)

10 -5

10 0

PGD

for some β ∈ (0 , 1) . The setup of this example is quite similar to the one in the previous example. However, there are a few noteworthy differences. To begin with, the hidden Shapley's game is not a zero-sum game. Specifically, the latent game's loss functions are given by the following polynomial system of equations:

As in the hidden Rock-Paper-Scissors game, this game's unique equilibrium also lies in x ∗ i = χ i ( θ ∗ i ) = ( 1 3 , 1 3 , 1 3 ) , i = 1 , 2 . A second difference is the small modulus, µ = 0 . 2 , that we choose for the strongly-convex regularizer of this game. In Fig. 9 we depict a semi-log plot of the mean performance of PHGD and GD in the above game constructed using the same parameters as in the case of the hidden Rock-Paper-Scissors game. Notice that although the behaviour of PHGD is similar in both games, the confidence bounds of GD are drastically larger in the current example.

0

100

200

300

400

500

600

700

800

900

1000

Time (

t

)

10 -1

10 0

PGD

A Hidden El Farol Bar Game. As a final example, we are going to revisit and describe the details of the hidden El Farol Bar game we introduced in Section 5. The El Farol Bar game is a famous congestion game that is often described as a game between populations. However, in this example, we are interested in its atomic N -playe variant. In the standard El Farol Bar game each player is given the option of visiting a specific bar in El Farol, and the outcome of the game is determined based on the number of players that decided to visit the bar. From the perspective of the player, there are three possible situations they may encounter, which carry some respective payoff. If the player

decides to do not to visit the bar, then, independently of the number of bar tenants, the player receives a payoff S . On the other hand, if the player decides to visit the El Farol bar, then depending on how crowded the bar is, they may receive a payoff that is strictly smaller, or strictly larger than S . If the bar is crowded, i.e., more than C ≥ 0 other players have visited the bar at the same time, then each of them receives payoff B < S . However, if the bar is not crowded, i.e., the total number of tenants is less than C , then they receive payoff G > S . It's not difficult to verify that the El Farol Bar game has a unique mixed Nash equilibrium, where each player chooses to visit the bar with probability C N .

We are going to consider a hidden variant of the above game among N = 30 players. Each player i controls a 5 -dimensional vector of control variables θ i ∈ R 5 , which feeds to an individual, differentiable, and preconfigured MLP. The MLP's output x i = χ i ( θ i ) is guaranteed to lie in [0 , 1] and is interpreted as the probability of player i visiting the El Farol bar. Regarding the MLP configuration, we follow a similar structure as in the previous examples. Specifically, the representation maps χ i : R 5 → [0 , 1] are defined as

where A (1) i ∈ [ -0 . 85 , 0 . 85] 4 × 5 , and α (2) i ∈ [ -1 , 1] 4 are randomly chosen. The additional restriction to the domain of A (1) reduces the chance for over-flow numerical errors of the sigmoid activation function. The activation function CeLU : R → ( -1 , ∞ ) is given as in (D.2b) and is applied pair-wise to the output of A (1) i × θ i . The loss functions of this game follow the standard variant's definitions, namely, we have that

̸

and the latent game's unique equilibrium is at x ∗ i = C N , i ∈ N .

The large dimensionality of the above game prohibits the usage of a detailed visualization as opposed to the previous examples. However, sufficient information about the behavior PHGD, and GD can be gathered by the performance log-plot across 100 random trajectories in Fig. 10. Observe that, regardless of the increased number of players, the behavior of PHGD is unaffected, i.e., it converges to the game's equilibrium at an exponential rate. On the other hand, the GD fails to converge in the game's equilibrium. In fact it eventually maintains a constant distance from it, unable to procceed further; an indication of a cycling behavior.

Peer reviews:
comments: SUMMARY:
This paper proposes a preconditioned hidden gradient descent to provide strong formal convergence guarantees in a general class
of multi-agent settings which are referred to as call hidden monotone games. Theoretical analyses and synthetic experiments are also provided.

STRENGTHS:
1. The method seems novel to handle the proposed problem of hidden monotone games.
2. Theoretical analyses and synthetic experiments are also provided to demonstrate the effectiveness of the proposed method.

WEAKNESSES:
1. Experiments on real-world datasets (e.g., MNIST or multi-agent reinforcement learning environments) are absent. 
2. Experiments on adversarial generation, adversarial attack, and adversarial transfer learning are absent.
3. Computational complexity analyses are absent.

QUESTIONS:
1. Is it feasible for this method to be applied to common neural networks, such as ResNet50, BERT, etc?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is known that a convergence guarantee exists for monotone games. However, most games are not monotone. This paper considers a new scenario where the monotone structure is presented in latent space. It then proposes the 'Preconditioned Hidden Gradient Dynamics' to design the preconditioned hidden gradient descent algorithm. It is demonstrated that the proposed algorithm will converge to the desired Nash equilibrium. The theoretical result is also verified using empirical experiments.\n\nSTRENGTHS:\nNovelty: Existing works have shown the difficulties of finding the equilibrium in hidden-monotone games. Other results usually require additional assumptions on the structure. This work proposes a new perspective to design the preconditioned hidden gradient descent and solves the problem that has not been addressed before.  \n\nSignificance: Most of games are not monotone. Studying the hidden-monotone structure will significantly extend the existing literatures on monotone games, which makes me believe this work opens a new direction of the resarch. The positive result obtained in this work and the new design in the continous-flow will also help understanding on the hidden structure.\n\nWEAKNESSES:\nThis paper is very well-written. But it may lack some real-world examples to help the reader understand the importance of studying the hidden games.\n\nQUESTIONS:\nThis paper has mentioned many times \"minimal assumptions\" or \"without additional assumptions\". I am probably interested in how strong other assumptions are, such as separability or low-dimensionality assumptions. Do you have any simple examples saying these assumptions are not satisfied. \n\nFigure 1 is very helpful to understand the concept of hidden games. Do you have any practical example or real-world example that would have such Rock-Paper-Scissors hidden structure?\n\nLIMITATIONS:\nThis is a purely theoretical work so no negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses hidden structure to provide continuous time and algorithmic theoretical learning guarantees for certain non-convex games. Specifically, they provide Preconditioned Hidden Gradient Flow and its discrete-time variant Preconditioned Hidden Gradient Descent that can be proved to converge when analyzed under a specific Lyapunov function.\n\nSTRENGTHS:\nThe theoretical contribution of the paper is significant, compared to past work, the setting is significantly more general. The paper is very well presented, with figures explaining the basic theoretical ideas behind the paper. The theory is solid and rigorous, while I have not checked the proofs line by line, they seem to be correct. Furthermore, significant motivation is provided both visually and by argumentation for the proof strategies employed.\n\nExperiments are also provided, and in toy examples the algorithm is demonstrably better than alternatives. Overall, the paper solves an important problem in algorithmic game theory.\n\nWEAKNESSES:\nI think this is a strong paper with a solid contribution. Some possible weaknesses:\n\n* A discussion of the relevance of the faithfulness assumption on the hidden map could be discussed. For instance, the experiments conducted seem to be on particular toy problems, it’s not immediately clear if there are large classes of problems with hidden structure that satisfy the given conditions.\n* The authors make an effort to present the ideas behind the proofs, which are the main contributions of the paper. Still, the presentation in the main body could benefit from a condensed roadmap of the proof strategy. While Section 4 contains main ideas, the overall picture was difficult to grasp.\n* The assumption on the representation map (e.g. faithfulness) is difficult to locate, the assumption could be made more explicit for ease of reading.\n* Experiments are limited to somewhat simple cases, an evaluation of PHGD on for instance GAN training or other relevant problems in practice could be a stronger empirical argument.\n\nQUESTIONS:\nThe examples mentioned in the experiment are interesting but also seem to be restricted to very specific cases. Are there larger classes of problems besides Examples 2.1, 2.2 that could be relevant in applications? For instance, has there been work on when neural network parameterization could satisfy the faithful map condition? While being possibly out of the scope of the main contribution of the paper, a discussion of these points could be interesting.\n\nAs a possible limitation, is the implied access to the Jacobian of representation map standard in the literature?\n\nThe paper mentions dimension reduction as a motivation of the representation map. It is not immediate to me that dimension reduction could be achieved while simultaneously preserving the faithfulness assumptions, due to topological considerations. A discussion of this point could be interesting. For instance, does the El Farol Bar game parameterization satisfy the faithfulness assumptions?\n\n*Minor questions:*\n\nLine 195 -> Is the message that $\\mathbf{P}_i$ will be designed to achieve certain properties of was it defined before? Was not clear on the first read.\n\nLines 177 - 189 -> The connection between the two paragraphs was not clear. Was NHGD shown to not require the mentioned assumptions?\n\n*Possible minor Typos:*\nLine 116: $\\Theta$ -> $\\Theta_i$?\nLine 164: For the Jacobian notation, should it depend on $\\theta_i$\nLine 172: Here, it is possible that the notation is confusing with the previous separation of the control variables of each player. As far as I understand, the individual coordinates of the controls of one player is meant here.\nLine 254: an closed -> a closed\n\nLIMITATIONS:\nNo major limitations. As mentioned above: empirical validation could be extended to relevant problems in application.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on studying non-convex games with hidden structures, where latent variables can be seen as a function of control variables and are decoupled. The authors propose a discrete algorithm called Preconditioned Hidden Gradient Descent (PHGD) to exploit the hidden structure and achieve convergence to Nash equilibrium. The method establishes a connection with natural gradient methods through the selection of gradient preconditioning schemes. The paper provides the first discrete convergence analysis on hidden convex concave games and extends the separable assumption to a more general multi-player setting. Convergence is proven under minimal assumptions, both in deterministic and stochastic environments.\n\nSTRENGTHS:\n- Originality: The paper introduces a novel algorithm, PHGD, to address non-convex games with hidden structure, providing the first discrete convergence analysis in this context. The connection established with natural gradient methods adds an original perspective to the research.\n\n- Quality: The paper demonstrates rigorous analysis by providing explicit convergence rates under minimal assumptions. It extends the previous work by considering a more general multi-player setting.\n\n- Clarity: The summary provided is clear and understandable, indicating good clarity in the paper.\n\n- Significance: The paper addresses an important problem by leveraging hidden structure in non-convex games, which has implications in various domains. The developed algorithm and its convergence properties contribute to the understanding and applicability of game theory.\n\nWEAKNESSES:\n- Lack of diverse examples: The paper could benefit from including additional examples beyond the matching Pennis zero-sum game to illustrate the applicability of the proposed method in different settings. The example shown in this paper can be included within previous 1-dimensional assumptions presented Mladenovic et al. Are there significant instances that this paper covers while previous results do not?\n\n- Lack of comparison with existing methods: It would be helpful if the authors provided a more detailed explanation of the technical challenges associated with discrete analysis compared to continuous ones. For the continuous dynamics, does it reduces to the exact natural gradient flow in hidden convex-concave games in Mladenovic et al., \n\n- It seems that the faster convergence is at the expense of high computational complexity.\n\nQUESTIONS:\n- Preconditioning method and its relation to natural gradient: The authors claim that \"a judiciously chosen gradient preconditioning scheme bears an unexpected connection to natural gradient method\". However, the connection has not been well explained in the main context. What is the literature of the preconditioning method? Why is the previous natural gradient approach not seen as preconditioning?\n\n\n- Technical difference in extending to the multi-dimensional case: What are the main technical differences or challenges encountered when extending the approach to the multi-dimensional case?\n\n- Minor issue regarding notation: In Line 195, where $P_i(\\theta_i)$ appears before introducing what $P_i$ represents.\n\nLIMITATIONS:\nThe limitations of this paper have not been addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a hidden preconditioned stochastic gradient descent method for finding Nash equilibrium in games with hidden monotone structures. It demonstrates non-asymptotic convergence bound of the proposed algorithm. The complexity bound of the hidden monotone game matches that of monotone games. The topic is very interesting as it provides a way to handle non-monotone stochastic games, which motivates me to read in details. Overall, it is an interesting paper. However, the presentation quality needs serious improvement. It is hard to judge the correctness with so many typos.\n\nSTRENGTHS:\nThe motivation of the hidden gradient descent design is clear. \n\nThe convergence rate result for hidden monotone stochastic games seems to be new and matches the optimal complexity bound for stochastic games.\n\nWEAKNESSES:\nThe presentation of the paper is poor with lots of typos. Without the Errata provided by the authors in the Appendix, when reading the main context for the first time, I found many mathematical mistakes. After reading the authors own corrections, there still remain many typos, making it hard to understand. The notation system is very confusing. I have tried my best to go through the proof. The overall idea seems correct but I cannot guarantee details. See more comments below. \n\nThe examples used to motivate the problems seems to be too simple. Example 2.1 is too simple. Example 2.2, it is unclear why one does not first solve the bilinear problem and find a corresponding mapping, which is not even needed actually. The example provided in figure 1 does not really need a neural network to approximate the probability space. \n\nFor Assumption 1, the paper claims that $\\theta$ and $x$ does not need to be within the same space, i.e., the dimension of them can be different. I wonder if $\\Theta$ is the whole space on $R^m$ and if $\\mathcal{X}$ is the whole space on $R^d$. Please specify the conditions for the Jacobian has positive and finite minimal and maximum singular values when $m\\not=d$.\n\nPlease give more concrete examples when the Jacobian can be exactly computed as required in the PHGD algorithm. If the Jacobian can be exactly computed, why one does not solve the hidden problem to optimality first and then find a mapping from control to optimal decision directly?\n\nMissing literature: there exists some literature on hidden convex structures in optimization and reinforcement learning with non-asymptotic convergence guarantee. The hidden gradient descent also appears previously. See literature [1-2] and reference therein. \n\n[1] Zhang, Junyu, et al. \"Variational policy gradient method for reinforcement learning with general utilities.\" Advances in Neural Information Processing Systems 33 (2020): 4572-4583.\n\n[2] Chen, Xin, et al. \"Efficient Algorithms for Minimizing Compositions of Convex Functions and Random Functions and Its Applications in Network Revenue Management.\" arXiv preprint arXiv:2205.01774 (2022).\n\nQUESTIONS:\nP2 Line 77, the \"[]\" should be deleted. \n\nP3 Line 99, Here $F$ is defined over $\\theta$. $F$ is later defined over $x$. Making it super confusing. Although $F$ is represent $l$ here, later $f$ is also used to denote $l$. Another notation $L$ is used to denote the stochastic counterpart of $l$. And then $f$ is used to denote a stochastic counterpart of $L(\\chi)$. Please read these notations to see how confusing they are.\n\nP5 Line 195, it is unclear why $P$ is defined over $\\theta_i$ rather than $\\theta$. \n\nP5 Lemma 1, what is a hidden smooth structure? What assumptions are exactly needed on $F$ and $\\chi$ function to ensure the format of $P$. \n\nP6, in Lemma 2, I suppose $\\chi(\\theta)=x$ by definition? \n\nP6 Line 230, about the statement for the first moment is totally wrong.\n\nP6 Line 238 & 239, $f_i$ is previously defined for deterministic function. Now $f_i$ becomes a random function. \n\nWhat is exactly the algorithm that one should run? I suppose that one should run PHGD with $\\hat g_{i,t}$ defined by $L$ rather than $f_i$ right?\n\nLine 308, there should be a space between \"games\" and \"This\".\n\nLemma 6, the notation is broken. \n\nEquation (C.5j) missing 1/2 in the third line.\n\nThere are many mismatch notations in the main context and the appendix.\n\nLIMITATIONS:\nThe motivation for using the method is not adequately discussed. Some more explanation on the limitation of some assumptions is definitely welcomed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}