SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Exploiting Hidden Structures in Non-Convex Games for Convergence to Nash Equilibrium
Iosif Sakos*
Engineering Systems and Design (SUTD)
iosif_sakos@mymail.sutd.edu.sg
Emmanouil V. Vlatakis-Gkaragkounis*
emvlatakis@cs.columbia.edu
Panayotis Mertikopoulos
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP
LIG & Archimedes RU, NKUA
panayotis.mertikopoulos@imag.fr
A wide array of modern machine learning applications – from adversarial models to multi-agent reinforcement learning – can be formulated as non-cooperative games whose Nash equilibria represent the system's desired operational states. Despite having a highly non-convex loss landscape, many cases of interest possess a latent convex structure that could potentially be leveraged to yield convergence to an equilibrium. Driven by this observation, our paper proposes a flexible first-order method that successfully exploits such "hidden structures" and achieves convergence under minimal assumptions for the transformation connecting the players' control variables to the game's latent, convex-structured layer. The proposed method – which we call preconditioned hidden gradient descent (PHGD) – hinges on a judiciously chosen gradient preconditioning scheme related to natural gradient methods. Importantly, we make no separability assumptions for the game's hidden structure, and we provide explicit convergence rate guarantees for both deterministic and stochastic environments.
Many powerful AI architectures are based on the idea of combining conceptually straightforward settings coming from game theory with the expressive power of neural nets. Some prominent examples of this type include generative adversarial networks (GANs) [14], robust reinforcement learning [36], adversarial training [26], multi-agent reinforcement learning in games [35, 39, 40], and even multi-player games that include free-form natural-language communication [4]. Intuitively, in all these cases, the game-theoretic abstraction serves to provide a palpable, easy-to-understand target, i.e., an equilibrium solution with strong axiomatic justification. However, from a complexity-theoretic standpoint, such targets are excessively ambitious, requiring huge amounts of data to express and compute, even approximately. Because of this, the agents' policies must be encoded via a universal function approximator (such as a neural net) and training this architecture boils down to iteratively updating these parameters until the process – hopefully! – converges to the target equilibrium.
Unfortunately, despite the ubiquitousness of these settings, the design of algorithms with provable convergence guarantees is still relatively lacking. This deficit is not surprising if one considers that even the – comparatively much simpler – problem of equilibrium learning in finite games is hindered by numerous computational hardness [9, 10] as well as dynamic impossibility results
[15, 16, 19, 24, 30, 31]. In this regard, our best hope for designing provably convergent algorithms is to focus on specific classes of games with some useful structure to exploit.
One of the most well-established frameworks of this type is the class of monotone games whose study goes back at least to Rosen [37]. As special cases, this setting includes single-agent convex minimization problems, two-player convex-concave min-max games, diagonally convex  $N$ -player games, etc. Owing to this connection, there has been a proliferation of strong positive results at the interface of game theory and optimization, see e.g., [28, 38] and references therein. In our case however, the agents do not play this monotone game directly, but can only access it indirectly via an encoding layer of control variables - like the weight parameters of a neural net that outputs a feasible strategy profile for the game in question. In this sense, from a machine learning perspective, the strategies of the game are latent variables, so we can think of each player as being equipped with a smooth mapping from a high-dimensional space of control variables to the strategy space of the game. Importantly, in contrast to the control variables, the latent variables are not directly accessible to the players themselves and should only be viewed as auxiliary variables - to all extents and purposes, the goal remains to find an operationally desirable control layer configuration. In this sense, the convex structure of the game becomes "hidden" behind the control layer, which entangles multiple input/control variables into nonlinear manifolds of latent variables.
As a result, the convex structure of the underlying game is effectively destroyed, resulting in highly non-convex end-to-end interactions. This raises the following central challenge:
Can we design provably convergent algorithms for non-convex games with a hidden structure in the presence of general couplings between control and latent variables?
Prior work in the area has shown that this is a promising and, at the same time, highly challenging question. In [41], the setting of hidden bilinear games was introduced and a number of negative results were presented, to the effect that gradient-descent-ascent can exhibit a variety of non-convergent behaviors, even when the game admits a hidden bilinear structure. Subsequently, [13] provided an approximate minimax theorem for a class of two-agent games where the players pick neural networks, but did not provide any convergent training algorithm for this class of games. Instead, the first positive result on the dynamics of hidden games was obtained by [42] who established a series of non-local convergence guarantees to the von Neumann max-min solution of the game in the case of two-agent hidden strictly convex-concave games for all initial conditions satisfying a certain genericity assumption. This approach, however, only applied to continuous-time dynamics – not algorithms – and it further imposed strong separability assumptions on the representation of the game in the control layer. More recently, [32] established the first global convergence guarantees in hidden games but, once again, these apply only to continuous-time dynamics and a special case of two-agent convex-concave games (akin to playing a convex combination of hidden games with one-dimensional latent spaces). This paper is the closest antecedent to our work as it introduces a dynamical system, called Generalized Natural Gradient Flow, which makes the  $L^2$  norm between the equilibrium in the hidden game and the current set of latent variables a Lyapunov function for the system.
Our results & techniques. Our paper seeks to provide an affirmative answer to the key challenge above under minimal assumptions on the coupling between latent and control variables. To that effect, we only assume that each agent is able to affect a measurable change along any latent variable by updating their control variables appropriately; without this assumption spurious equilibria can emerge due to the deficiency of the control layer architecture. Importantly however, even though the map from control to latent variables is known to the players, we do not assume that it can be efficiently inverted (e.g., to solve for a profile of control variables that realizes a profile of latent variables). Otherwise, if it could, the entire game could be solved directly in the latent layer and then ported back to the control layer, thus rendering the whole problem moot – and, indeed, when working with realistic neural net architectures, this inversion problem is, to all intents and purposes, impossible.
For intuition, we begin by designing a new continuous-time flow, that we call preconditioned hidden gradient dynamics, and which enjoys strong convergence properties in games with a hidden strictly monotone structure (Proposition 1). Similarly to [32], this is achieved by using the  $L^2$  norm in the latent layer as a Lyapunov function in the control layer; however, the similarities with the existing literature end there. Our paper does not make any separability or low-dimensionality assumptions, and is otherwise purely algorithmic: specifically, building on the continuous-time intuition, we provide a concrete, implementable algorithm, that we call preconditioned hidden gradient descent (PHGD);
Figure 1: A hidden game of Rock-Paper-Scissors with strategies encoded by two multi-layer perceptrons (MLPs), whose 4-dimensional input is dictated by the two players (cf. [32]). The nonlinearity of the MLP representation maps leads to a highly non-convex non-concave zero-sum game. However, by employing PHGD, both players' MLP control variables accurately identify the  $(1/3, 1/3, 1/3)$  equilibrium in the game's latent space.
this algorithm is run with stochastic gradients, and enjoys a series of strong, global convergence guarantees in hidden games.
First, as a baseline, Theorem 1 shows that a certain averaged process achieves an  $\mathcal{O}(1 / \sqrt{t})$  convergence rate in all games with a hidden monotone structure and Lipschitz continuous loss functions. If the hidden structure is strongly monotone, Theorem 2 further shows that this rate can be improved to  $\mathcal{O}(1 / t)$  for the actual trajectory of the players' control variables; and if the algorithm is run with full, deterministic gradients, the rate becomes geometric (Theorem 3). To the best of our knowledge, these are the first bona fide algorithmic convergence guarantees for games with a hidden structure.
2 Problem setup and preliminaries
Throughout the sequel, we will focus on continuous  $N$ -player games where each player, indexed by  $i \in \mathcal{N} \coloneqq \{1, \dots, N\}$ , has a convex set of control variables  $\theta_i \in \Theta_i \coloneqq \mathbb{R}^{m_i}$ , and a continuously differentiable loss function  $\ell_i \colon \Theta \to \mathbb{R}$ , where  $\Theta \coloneqq \prod_i \Theta_i$  denotes the game's control space. For concreteness, we will refer to the tuple  $\Gamma \equiv \Gamma(\mathcal{N}, \Theta, \ell)$  as the base game.
The most relevant solution concept in this setting is that of a Nash equilibrium, i.e., an action profile  $\theta^{*}\in \Theta$  that discourages unilateral deviations. Formally, we say that  $\theta^{*}\in \Theta$  is a Nash equilibrium of the base game  $\Gamma$  if
$$
\ell_ {i} \left(\theta^ {*}\right) \leq \ell_ {i} \left(\theta_ {i}; \theta_ {- i} ^ {*}\right) \quad \text {f o r a l l} \theta_ {i} \in \Theta_ {i}, i \in \mathcal {N} \tag {NE}
$$
where we employ the standard game-theoretic shorthand  $(\theta_{i};\theta_{-i})$  to distinguish between the action of the  $i$ -th player and that of all other players in the game. Unfortunately, designing a learning algorithm that provably outputs a Nash equilibrium is a very elusive task: the impossibility results of Hart & Mas-Colell [15, 16] already preclude the existence of uncoupled dynamics that converge to a Nash equilibrium in all games; more recently, Millionis et al. [31] established a similar impossibility result even for possibly coupled dynamics (in both discrete and continuous time), while Daskalakis et al. [9, 10] has shown that even the computation of an approximate equilibrium can be beyond reach.
In view of this, our work focuses on games with a hidden, latent structure that can be exploited to compute its Nash equilibria. More precisely, inspired by [41, 42] we have the following definition.
Definition 1. We say that the game  $\Gamma \equiv \Gamma (\mathcal{N},\Theta ,\ell)$  admits a latent - or hidden - structure if:
1. Each player's control variables can be mapped faithfully to a closed convex set of latent variables  $x_{i} \in \mathcal{X}_{i} \subseteq \mathbb{R}^{d_{i}}$ ; formally, we posit that there exists a Lipschitz smooth map  $\chi_{i} \colon \Theta_{i} \to \mathcal{X}_{i}$  with no critical points and such that  $\operatorname{cl}(\chi_i(\Theta_i)) = \mathcal{X}_i$ .
2. Each player's loss function factors through the game's latent space  $\mathcal{X} \coloneqq \prod_{i} \mathcal{X}_{i}$  as
$$
\ell_ {i} (\theta) = f _ {i} \left(\chi_ {1} \left(\theta_ {1}\right), \dots , \chi_ {N} \left(\theta_ {N}\right)\right) \tag {1}
$$
for some Lipschitz smooth function  $f_{i}\colon \mathcal{X}\to \mathbb{R}$  called the player's latent loss function.
For concreteness, we will refer to the product map  $\chi(\theta) = (\chi_i(\theta_i))_{i \in \mathcal{N}}$  as the game's representation map, and the tuple  $\mathcal{G} \equiv \mathcal{G}(\mathcal{N}, \mathcal{X}, f)$  will be called the hidden/latent game. To simplify notation later on, we also assume that  $\mathcal{X}_i$  has nonempty topological interior, so, in particular,  $\dim(\mathcal{X}_i) = d_i \leq m_i$ .
We illustrate the above notions in two simple - but not simplistic - examples below; for a schematic representation, cf. Fig. 2.
Figure 2: Schematic representation of a game with a hidden /latent structure.
Example 2.1. Consider a single-agent game  $(N = 1)$  where the objective is to minimize the nonconvex function  $\ell (\theta) = \sum_{\alpha = 1}^{|\mathcal{D}|}(\mathrm{sigmoid}(\theta_{\alpha}) - s_{\alpha})^{2}$  over a dataset  $\mathcal{D} = \{s_{\alpha}\}$ . This problem can be recast as a hidden convex problem with  $f(x) = \sum_{\alpha = 1}^{|\mathcal{D}|}(x_{\alpha} - s_{\alpha})^{2}$  and  $\chi (\theta) = \mathrm{sigmoid}(\theta)$ .
Example 2.2. A more complex scenario involves non-convex/non-concave min-max optimization problems of the form  $\min_{\theta_1}\max_{\theta_2}\chi_1(\theta_1)^\top C\chi_2(\theta_2)$  where  $\chi_{1}$  and  $\chi_{2}$  are preconfigured multi-layer perceptrons (MLPs) constructed with smooth activation functions (such as CeLUs). This problem can be reformulated as a hidden bilinear problem by setting  $f_{1}(x_{1},x_{2}) = x_{1}^{\top}Cx_{2} = -f_{2}(x_{1},x_{2})$
Before moving forward, there are some points worth noting regarding the above definitions.
Remark 1. First, the requirement  $\operatorname{cl}(\chi(\Theta)) = \mathcal{X}$  means that all latent variable profiles  $x \in \mathcal{X}$  can be approximated to arbitrary accuracy in the game's control space. The reason that we do not make the stronger assumption  $\chi(\Theta) = \mathcal{X}$  is to capture cases like the sigmoid map: if  $\chi(\theta) = [1 + \exp(-\theta)]^{-1}$  for  $\theta \in \mathbb{R}$ , the image of  $\chi$  is the interval  $(0,1)$ , which is convex but not closed.
Remark 2. Second, the requirement that  $\chi$  has no critical points simply means that, for any control variable configuration  $\theta \in \Theta$ , the Jacobian  $\mathrm{Jac}(\chi(\theta))$  of  $\chi$  at  $\theta$  has full rank. By the implicit function theorem [23], this simply means that  $\chi$  locally looks like a projection (in suitable coordinates around  $\theta$ ), so it is possible to affect a measurable change along any feasible latent direction by updating each player's control variables appropriately. This is no longer true if  $\chi$  has critical points, so this is a minimal requirement to ensure that no spurious equilibria appear in the game's latent space.
To proceed, we will assume that the latent game is diagonally convex in the sense of Rosen [37], a condition more commonly known in the optimization literature as monotonicity [12]. Formally, let
$$
g _ {i} (x) = \nabla_ {i} f _ {i} (x) := \nabla_ {x _ {i}} f _ {i} (x) \tag {2}
$$
denote the individual gradient of the latent loss function of player  $i \in \mathcal{N}$ , and write  $g(x) = (g_1(x), \ldots, g_N(x))$  for the profile thereof. We then say that the latent game  $\mathcal{G}$  is:
- Monotone if  $\langle g(x') - g(x), x' - x \rangle \geq 0$  for all  $x, x' \in \mathcal{X}$ . (3a)
- Strictly monotone if (3a) holds as a strict inequality whenever  $x' \neq x$ . (3b)
- Strongly monotone if  $\langle g(x') - g(x), x' - x \rangle \geq \mu \| x' - x \|^2$  for some  $\mu > 0$  and all  $x, x'$ . (3c)
Clearly, strong monotonicity implies strict monotonicity, which in turn implies monotonicity; on the other hand, when we want to distinguish between problems that are monotone but not strictly monotone, we will say that  $g$  is merely monotone. Finally, extending the above to the base game, we will say that  $\Gamma$  admits a hidden monotone structure when the latent game  $\mathcal{G}$  is monotone as above (and likewise for strictly / strongly monotone structures).
Examples of monotone games include neural Kelly auctions and Cournot oligopolies [5, 6, 20, 25], covariance matrix optimization problems and power control [27, 29, 38], certain classes of congestion games [34], etc. In particular, in the case of convex minimization problems, monotonicity (resp. strict/strong monotonicity) corresponds to convexity (resp. strict/strong convexity) of the problem's objective function. In all cases, it is straightforward to check that the image  $x^{*} = \chi(\theta^{*})$  of a Nash equilibrium  $\theta^{*} \in \Theta$  of the base game satisfies a Stampacchia variational inequality of the form
$$
\langle g \left(x ^ {*}\right), x - x ^ {*} \rangle \geq 0 \quad \text {f o r a l l} x \in \mathcal {X}. \tag {SVI}
$$
In turn, by monotonicity, this characterization is equivalent to the Minty variational inequality
$$
\langle g (x), x - x ^ {*} \rangle \geq 0 \quad \text {f o r a l l} x \in \mathcal {X}. \tag {MVI}
$$
To avoid trivialities, we will assume throughout that the solution set  $\mathcal{X}^*$  of (SVI)/(MVI) is nonempty. This is a standard assumption without which the problem is not well-posed – and hence, meaningless from an algorithmic perspective.
Notation. To streamline notation (and unless explicitly mentioned otherwise), we will denote control variables by  $\theta_{i}$  and  $\theta$ , and we will write  $x_{i} = \chi_{i}(\theta_{i})$  and  $x = \chi (\theta)$  for the induced latent variables. We will also write  $m\coloneqq \sum_{i}m_{i}$  for the dimensionality of the game's control space  $\Theta$  and  $d\coloneqq \sum_{i}d_{i}$  for the dimensionality of the latent space  $\mathcal{X}$  (so, in general,  $m\geq d$ ). Finally, when the representation map  $\chi$  is clear from the context, we will write  $\mathbf{J}_i(\theta_i)\coloneqq \mathrm{Jac}(\chi_i(\theta_i))\in \mathbb{R}^{d_i\times m_i}$  for the Jacobian matrix of  $\chi_{i}$  at  $\theta_{i}\in \Theta_{i}$ , and  $\mathbf{J}(\theta) = \bigoplus_{i}\mathbf{J}_{i}(\theta_{i})$  for the associated block diagonal sum.
3 Hidden gradients and preconditioning
We are now in a position to present our main algorithmic scheme for equilibrium learning in games with a hidden monotone structure. In this regard, our aim will be to overcome the following limitations in the existing literature on learning in hidden games: (i) the literature so far has focused exclusively on continuous-time dynamics, with no discrete-time algorithms proven to efficiently converge to a solution; (ii) the number of players is typically limited to two; and (iii) the representation maps are separable in the sense that the control variables are partitioned into subsets and each subset affects exactly one latent variable.<sup>1</sup>
We start by addressing the last two challenges first. Specifically, we begin by introducing a gradient preconditioning scheme that allows us to design a convergent continuous-time dynamical system for hidden monotone games with an arbitrary number of players and no separability restrictions for its representation maps. Subsequently, we propose a bona fide algorithmic scheme – which we call preconditioned hidden gradient descent (PHGD) – by discretizing the said dynamics, and we analyze the algorithm's long-run behavior in Section 4.
3.1. Continuous-time dynamics for hidden games. To connect our approach with previous works in the literature, our starting point will be a simple setting already captured within the model of [32], min-max games with a hidden convex-concave structure and unconstrained one-dimensional control and latent spaces per player, i.e.,  $\Theta_{i} = \mathbb{R} = \mathcal{X}_{i}$  for  $i = 1,2$ . We should of course note that this specific setting is fairly restrictive and not commonly found in deep neural network practice: in real-world deep learning applications, neural nets generally comprise multiple interconnected layers with distinct activation functions, so the input-output relations are markedly more complex and intertwined. Nevertheless, the setting's simplicity makes the connection with natural gradient methods particularly clear, so as in [32], it will serve as an excellent starting point.
Concretely, Mladenovic et al. [32] analyze the natural hidden gradient dynamics
$$
\dot {\theta} _ {i} = - \frac {1}{| \chi_ {i} ^ {\prime} (\theta_ {i}) | ^ {2}} \frac {\partial \ell_ {i}}{\partial \theta_ {i}} \quad \text {f o r a l l} i \in \mathcal {N}. \tag {NHGD}
$$
The reason for this terminology - and the driving force behind the definition of (NHGD) - is the observation that, in one-dimensional settings, a direct application of the chain rule to the defining relation  $\ell_{i} = f_{i}\circ \chi$  of the game's latent loss functions yields  $\partial \ell_i / \partial \theta_i = \chi_i^\prime (\theta_i)\cdot \partial f_i / \partial x_i$  so, in turn, (NHGD) becomes
$$
\dot {\theta} _ {i} = - \frac {1}{\chi_ {i} ^ {\prime} \left(\theta_ {i}\right)} \frac {\partial f _ {i}}{\partial x _ {i}} \tag {4}
$$
where, for simplicity, we are tacitly assuming that  $\chi_i(\theta_i)' > 0$ .
This expression brings two important points to light. First, even though (NHGD) is defined in terms of the actual, control-layer gradients  $\partial \ell_i / \partial \theta_i$  of  $\Gamma$ , Eq. (4) shows that the dynamics are actually driven by the latent-layer, "hidden gradients"  $g_{i}(x) = \partial f_{i} / \partial x_{i}$  of  $\mathcal{G}$ . Second, the preconditioner  $|\chi_i'(\theta_i)|^{-2}$  in (NHGD) can be seen as a Riemannian metric on  $\Theta$ : instead of defining gradients relative to the standard Euclidean metric of  $\Theta \equiv \mathbb{R}^N$ , (NHGD) can be seen as a gradient flow relative
The separability assumption also rules out the logit representation  $\exp (\theta_{i\alpha}) / \sum_{\beta}\exp (\theta_{i\beta})$  that is standard when the latent structure expresses mixed strategies in a finite game.
to the Riemannian metric  $g_{ij}(\theta) = \delta_{ij}\chi_i'(\theta_i)$ , which captures the "natural" geometry induced by the representation map  $\chi$ .<sup>2</sup>
From an operational standpoint, the key property of (NHGD) that enabled the analysis of [32] is the observation that the  $L^2$  energy function
$$
E (\theta) = \frac {1}{2} \| \chi (\theta) - x ^ {*} \| ^ {2} \tag {5}
$$
between the latent representation  $x = \chi(\theta)$  of  $\theta$  and a solution  $x^{*}$  of (SVI) / (MVI) is a Lyapunov function for (NHGD). Indeed, a straightforward differentiation yields
$$
\dot {E} (\theta) = \frac {1}{2} \frac {d}{d t} \| x - x ^ {*} \| ^ {2} = \sum_ {i} \dot {x} _ {i} \cdot \left(x _ {i} - x _ {i} ^ {*}\right) = - [ g (\chi (\theta)) ] ^ {\mathsf {T}} (\chi (\theta) - x ^ {*}) \leq 0 \tag {6}
$$
with the penultimate step following from (4) and the last one from (MVI). It is then immediate to see that the latent orbits  $x(t) = \chi(\theta(t))$  of (NHGD) converge to equilibrium in strictly monotone games.
The approach in the example above depends crucially on the separability assumption which, among others, trivializes the problem. Indeed, if there is no coupling between control variables in the game's latent space, the equations  $x_{i} = \chi_{i}(\theta_{i})$  can be back solved easily for  $\theta$  (e.g., via binary search), so it is possible to move back-and-forth between the latent and control layers, ultimately solving the game in the latent layer and subsequently extracting a solution configuration in the control layer. Unfortunately however, extending the construction of (NHGD) to a non-separable setting is not clear, so it is likewise unclear how to exploit the hidden structure of the game beyond the separable case.