SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\langle g (\tilde {x}), \bar {x} _ {t} - \tilde {x} \right\rangle = \left\langle g (\tilde {x}), \sum_ {s = 1} ^ {t} \frac {\gamma_ {s}}{\tilde {\gamma} _ {t}} x _ {s} - \tilde {x} \right\rangle \\ = \sum_ {s = 1} ^ {t} \frac {\gamma_ {s}}{\tilde {\gamma} _ {t}} \left\langle g (\tilde {x}), x _ {s} - \tilde {x} \right\rangle \tag {C.5} \\ \leq \frac {E _ {1}}{\tilde {\gamma} _ {t}} + \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} \phi_ {s}}{\tilde {\gamma} _ {t}} + \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \psi_ {s}}{\tilde {\gamma} _ {t}}, \\ \end{array}
$$
where  $\bar{x}_t\coloneqq \sum_{s = 1}^t\frac{\gamma_s}{\tilde{\gamma}_t} x_s$  is the time-average.
Next, let us define the following auxiliary process that will assist us in further bounding the above expression:
$$
y _ {1} = x _ {1}
$$
$$
y _ {t + 1} = \underset {x \in \mathcal {X}} {\arg \min } \| y _ {t} - \gamma_ {t} \eta_ {t} - x \| \quad \text {f o r a l l} t \geq 2, \tag {C.6}
$$
where  $\eta_t \coloneqq \mathbf{J}(\theta_t)\mathbf{P}(\theta_t)V_t - g(x_t)$  for all  $i \in \mathcal{N}$ . Observe that
$$
\gamma_ {t} \phi_ {t} = \gamma_ {t} \left\langle \eta_ {t}, x _ {t} - \tilde {x} \right\rangle = \gamma_ {t} \left\langle \eta_ {t}, y _ {t} - \tilde {x} \right\rangle + \gamma_ {t} \left\langle \eta_ {t}, x _ {t} - y _ {t} \right\rangle \quad \text {f o r a l l} t \geq 1. \tag {C.7}
$$
Let us, first, focus on the former of the two terms. Notice that for all  $t \geq 1$ , we can write
$$
\gamma_ {t} \left\langle \eta_ {t}, y _ {t} - \tilde {x} \right\rangle = \gamma_ {t} \left\langle \eta_ {t}, y _ {t + 1} - \tilde {x} \right\rangle + \gamma_ {t} \left\langle \eta_ {t}, y _ {t} - y _ {t + 1} \right\rangle . \tag {C.8}
$$
Furthermore, by the optimality of  $y_{t + 1}$ ,  $t \geq 2$ , we have also have that
$$
\left\langle y _ {t + 1} - y _ {t} + \gamma_ {t} \eta_ {t}, y _ {t + 1} - \tilde {x} \right\rangle \leq 0. \tag {C.9}
$$
That is,  $\gamma_t\langle \eta_t,y_{t + 1} - \tilde{x}\rangle \leq \langle y_t - y_{t + 1},y_{t + 1} - \tilde{x}\rangle$
Let us also recall a couple of useful quadratic identities, namely, we have that  $\| y_{t} - \tilde{x} \|^{2} = \| y_{t} - y_{t + 1} + y_{t + 1} - \tilde{x} \|^{2} = \| y_{t} - y_{t + 1} \|^{2} + 2\langle y_{t} - y_{t + 1}, y_{t + 1} - \tilde{x} \rangle + \| y_{t + 1} - \tilde{x} \|^{2}$ , and  $\| \gamma_{t}\eta_{t} - (y_{t} - y_{t + 1}) \|^{2} = \gamma_{t}^{2}\| \eta_{t}\|^{2} - 2\gamma_{t}\langle \eta_{t}, y_{t} - y_{t + 1} \rangle + \| y_{t} - y_{t + 1}\|^{2}$ . Then, in conjunction with the above, we get that for all  $t \geq 1$ :
$$
\begin{array}{l} \gamma_ {t} \langle \eta_ {t}, y _ {t} - \tilde {x} \rangle = \gamma_ {t} \langle \eta_ {t}, y _ {t + 1} - \tilde {x} \rangle + \gamma_ {t} \langle \eta_ {t}, y _ {t} - y _ {t + 1} \rangle \\ \leq \left\langle y _ {t} - y _ {t + 1}, y _ {t + 1} - \tilde {x} \right\rangle + \gamma_ {t} \left\langle \eta_ {t}, y _ {t} - y _ {t + 1} \right\rangle \\ = \frac {1}{2} \| y _ {t} - \tilde {x} \| ^ {2} - \frac {1}{2} \| y _ {t + 1} - \tilde {x} \| ^ {2} - \frac {1}{2} \| \gamma_ {t} \eta_ {t} - (y _ {t} - y _ {t + 1}) \| ^ {2} + \frac {\gamma_ {t} ^ {2}}{2} \| \eta_ {t} \| ^ {2} \tag {C.10} \\ \leq \frac {1}{2} \| y _ {t} - \tilde {x} \| ^ {2} - \frac {1}{2} \| y _ {t + 1} - \tilde {x} \| ^ {2} + \frac {\gamma_ {t} ^ {2}}{2} \| \eta_ {t} \| ^ {2} \\ \end{array}
$$
Finally, summing both sides of the above inequalities, we also get that
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \gamma_ {s} \langle \eta_ {s}, y _ {s} - \tilde {x} \rangle \leq \frac {1}{2} \sum_ {s = 1} ^ {t} \| y _ {s} - \tilde {x} \| ^ {2} - \frac {1}{2} \sum_ {s = 1} ^ {t} \| y _ {s + 1} - \tilde {x} \| ^ {2} + \frac {1}{2} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} \\ = \frac {1}{2} \| y _ {1} - \tilde {x} \| ^ {2} - \frac {1}{2} \| y _ {t + 1} - \tilde {x} \| ^ {2} + \frac {1}{2} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} \\ \leq \frac {1}{2} \| y _ {1} - \tilde {x} \| ^ {2} + \frac {1}{2} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} \tag {C.11} \\ = \frac {1}{2} \| x _ {1} - \tilde {x} \| ^ {2} + \frac {1}{2} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} \\ = E _ {1} + \frac {1}{2} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} \\ \end{array}
$$
With the above established, let us reconsider the quantity  $\langle g(\tilde{x}),\bar{x}_t - \tilde{x}\rangle$ . Specifically, due to the above derivations, we have that for all  $t\geq 1$
$$
\begin{array}{l} \left\langle g (\tilde {x}), \bar {x} _ {t} - \tilde {x} \right\rangle \leq \frac {E _ {1}}{\tilde {\gamma} _ {t}} + \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} \phi_ {s}}{\tilde {\gamma} _ {t}} + \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \psi_ {s}}{\tilde {\gamma} _ {t}} \\ = \frac {E _ {1}}{\tilde {\gamma} _ {t}} + \frac {1}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {t} \langle \eta_ {t}, y _ {t} - \tilde {x} \rangle + \frac {1}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {t} \langle \eta_ {t}, x _ {t} - y _ {t} \rangle + \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \psi_ {s}}{\tilde {\gamma} _ {t}} \\ \leq \frac {3 E _ {1}}{2 \tilde {\gamma} _ {t}} + \frac {1}{2 \tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| \eta_ {s} \| ^ {2} + \frac {1}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {t} \langle \eta_ {t}, x _ {t} - y _ {t} \rangle + \frac {\kappa}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| V _ {s} \| ^ {2}. \\ \end{array}
$$
Considering the mean of supremum over  $\tilde{x} \in \mathcal{X}$  of the LHS, we end up with
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {\tilde {x} \in \mathcal {X}} \left\langle g (\tilde {x}), \bar {x} _ {t} - \tilde {x} \right\rangle \right] \leq \frac {3 E _ {1}}{2 \tilde {\gamma} _ {t}} + \frac {1}{2 \tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \mathbb {E} [ \| \eta_ {s} \| ^ {2} ] + \frac {\kappa}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \mathbb {E} [ \| V _ {s} \| ^ {2} ] \\ \leq \frac {3 E _ {1}}{2 \tilde {\gamma} _ {t}} + \frac {M ^ {2}}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} + \frac {\kappa M ^ {2}}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \\ = \frac {3 E _ {1}}{2 \tilde {\gamma} _ {t}} + (1 + \kappa) \frac {M ^ {2}}{\tilde {\gamma} _ {t}} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \tag {C.12} \\ = \mathcal {O} \left(\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} / \tilde {\gamma} _ {t}\right) \\ = \mathcal {O} (\log t / \sqrt {t}), \\ \end{array}
$$
where the last inequality is direct consequence of the bounded second moment of the stochastic gradient in Assumption 1
Proof. Let  $\theta_{1} \in \Theta$  be some arbitrary initialization of the algorithm, and let  $\gamma \geq \frac{1}{2\mu}$  be arbitrary. Since the map  $f$  is  $\mu$ -strongly monotone, we have, by the definition of strong monotonicity, that
$$
\left\langle g \left(x _ {t}\right), x _ {t} - x ^ {*} \right\rangle \geq \mu \| x - x ^ {*} \| ^ {2} \quad \text {f o r a l l} x \in \mathcal {X}. \tag {C.13}
$$
Next, by Lemma 4, we have that for all  $t \geq 1$ :
$$
\phi_ {t} = \left(\mathbf {J} \left(\theta_ {t}\right) \mathbf {P} \left(\theta_ {t}\right) V _ {t} - g \left(x _ {t}\right)\right) ^ {\intercal} \left(x _ {t} - \hat {x}\right) \tag {C.14}
$$
$$
\begin{array}{l} E _ {t + 1} \leq E _ {t} - \mu \gamma_ {t} \| x _ {t} - x ^ {*} \| _ {2} ^ {2} + \gamma_ {t} \phi_ {t} + \gamma_ {t} ^ {2} \psi_ {t} \\ = E _ {t} - 2 \mu \gamma_ {t} E _ {t} + \gamma_ {t} \phi_ {t} + \gamma_ {t} ^ {2} \psi_ {t} \tag {C.15} \\ = (1 - 2 \mu \gamma_ {t}) E _ {t} + \gamma_ {t} \phi_ {t} + \gamma_ {t} ^ {2} \psi_ {t}. \\ \end{array}
$$
Let  $\mathcal{H}_t = \{\theta_s : s = 1, \dots, t\}$  be the history of play up to iteration  $t$ . In expectation, it follows from the above that, for all  $t \geq 1$ :
$$
\begin{array}{l} \mathbb {E} \left[ E _ {t + 1} \right] \leq (1 - 2 \mu \gamma_ {t}) \mathbb {E} \left[ E _ {t} \right] + \gamma_ {t} \mathbb {E} \left[ \phi_ {t} \right] + \gamma_ {t} ^ {2} \mathbb {E} \left[ \psi_ {t} \right] \\ = \left(1 - 2 \mu \gamma_ {t}\right) \mathbb {E} \left[ E _ {t} \right] + \gamma_ {t} \mathbb {E} \left[ \mathbb {E} \left[ \phi_ {t} \mid \mathcal {H} _ {t} \right] \right] + \gamma_ {t} ^ {2} \mathbb {E} \left[ \mathbb {E} \left[ \psi_ {t} \mid \mathcal {H} _ {t} \right] \right]. \tag {C.16} \\ \end{array}
$$
Note that  $x_{t}$  is  $\mathcal{H}_t$ -measurable; hence, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \phi_ {t} \mid \mathcal {H} _ {t} \right] = \left(\mathbf {J} \left(\theta_ {t}\right) \mathbf {P} \left(\theta_ {t}\right) \mathbb {E} \left[ V _ {t} \mid \mathcal {H} _ {t} \right] - g \left(x _ {t}\right)\right) ^ {\top} \left(x _ {t} - \hat {x}\right) \\ = \left(\mathbf {J} (\theta_ {t}) \mathbf {P} (\theta_ {t}) v (\theta_ {t}) - g (x _ {t})\right) ^ {\intercal} (x _ {t} - \hat {x}) \\ = \left(\mathbf {J} \left(\theta_ {t}\right) \mathbf {P} \left(\theta_ {t}\right) \mathbf {J} \left(\theta_ {t}\right) ^ {\top} g \left(x _ {t}\right) - g \left(x _ {t}\right)\right) ^ {\top} \left(x _ {t} - \hat {x}\right) \tag {C.17} \\ = \left(g (x _ {t}) - g (x _ {t})\right) ^ {\intercal} \left(x _ {t} - \hat {x}\right) \\ = 0. \\ \end{array}
$$
Furthermore, by Assumption 1, we have that  $\mathbb{E}[\| V_t\|^2 \mid \mathcal{H}_t] \leq M^2$ , it also holds that
$$
\mathbb {E} \left[ \psi_ {t} \mid \mathcal {H} _ {t} \right] = \kappa \mathbb {E} \left[ \| V _ {t} \| ^ {2} \mid \mathcal {H} _ {t} \right] \leq \kappa M ^ {2} \tag {C.18}
$$
Next, by some simple substitutions, we have
$$
\begin{array}{l} \mathbb {E} \left[ E _ {t + 1} \right] \leq \left(1 - 2 \mu \gamma_ {t}\right) \mathbb {E} \left[ E _ {t} \right] + \kappa M ^ {2} \gamma_ {t} ^ {2} \\ = \left(1 - \frac {2 \mu \gamma}{t}\right) \mathbb {E} \left[ E _ {t} \right] + \frac {\kappa M ^ {2} \gamma^ {2}}{t ^ {2}} \tag {C.19} \\ \end{array}
$$
Finally, since  $\gamma >\frac{1}{2\mu}$ , we may apply Chung's lemma [8] to get that, for all  $t\geq 1$
$$
\mathbb {E} \left[ E _ {t} \right] \leq \frac {\kappa M ^ {2} \gamma^ {2}}{2 \mu \gamma - 1} \cdot \frac {1}{t} + \mathcal {O} \left(\frac {1}{t ^ {2}} + \frac {1}{t ^ {2 \mu \gamma}}\right). \tag {C.20}
$$
Proof. Due to the absence of randomness, and therefore the absence of noise, Lemma 4 may be simplified to
$$
E _ {t + 1} \leq E _ {t} - \gamma_ {t} \left\langle g \left(x _ {t}\right), x _ {t} - x ^ {*} \right\rangle + \kappa \gamma_ {t} ^ {2} \| v _ {t} \| ^ {2} \tag {C.21}
$$
for some constants  $\kappa >0$ , where  $v_{t}\coloneqq (\nabla_{i}\ell_{i}(\theta_{t}))_{i\in \mathcal{N}}$ .
Let  $\beta$  be the modulus of Lipschitz continuity of the gradients  $v_{t} = \mathbf{J}(\theta_{t})^{\intercal}g(x_{t})$ , and let  $\frac{\mu}{2}$  be the modulus of the strong monotonicity of  $g$ . Recall that, by Assumption 2, we have that the singular
values of the Jacobian  $\mathbf{J}(\theta)$  of the representation map  $\chi$  are bounded from above and below by  $\sigma_{\mathrm{max}} < \infty$  and  $\sigma_{\mathrm{min}} > 0$  respectively. Therefore, since  $v_{t}$  is  $\beta$ -Lipschitz, it follows that
$$
\left\| v _ {t} - v ^ {*} \right\| ^ {2} \leq \beta^ {2} \left\| \theta_ {t} - \theta^ {*} \right\| ^ {2} \leq \frac {\beta^ {2}}{\sigma_ {\min } ^ {2}} \left\| x _ {t} - x ^ {*} \right\| ^ {2} \tag {C.22}
$$
In conjunction to the above, we then also have
$$
\begin{array}{l} E _ {t + 1} \leq E _ {t} - \gamma_ {t} \big <   g (x _ {t}), x _ {t} - x ^ {*} \big > + \kappa \gamma_ {t} ^ {2} \| v _ {t} \| ^ {2} \\ = E _ {t} - \gamma_ {t} \left\langle g \left(x _ {t}\right), x _ {t} - x ^ {*} \right\rangle + \kappa \gamma_ {t} ^ {2} \| v _ {t} - v ^ {*} \| ^ {2} \\ \leq E _ {t} - \mu \gamma_ {t} \frac {1}{2} \| x _ {t} - x ^ {*} \| ^ {2} + \frac {\kappa \gamma_ {t} ^ {2} \beta^ {2}}{\sigma_ {\min} ^ {2}} \| x _ {t} - x ^ {*} \| ^ {2} \tag {C.23} \\ = E _ {t} - \mu \gamma_ {t} E _ {t} + \frac {2 \kappa \gamma_ {t} ^ {2} \beta^ {2}}{\sigma_ {\mathrm {m i n}} ^ {2}} E _ {t} \\ = E _ {t} \Big (1 - \mu \gamma_ {t} + \frac {2 \kappa \gamma_ {t} ^ {2} \beta^ {2}}{\sigma_ {\mathrm {m i n}} ^ {2}} \Big) \\ \end{array}
$$
Restricting  $\gamma_{t}$  such that
$$
1 - \mu \gamma_ {t} + \frac {2 \kappa \gamma_ {t} ^ {2} \beta^ {2}}{\sigma_ {\min } ^ {2}} <   1 \Longleftrightarrow \gamma_ {t} <   \frac {\sigma_ {\min } ^ {2} \mu}{2 \kappa \beta^ {2}}, \tag {C.24}
$$
we finally have that, for the choice of  $\gamma_t = \hat{\gamma} \coloneqq \frac{\sigma_{\min}^2 \mu}{8\kappa \beta^2}$ , it holds
$$
E _ {t} = \mathcal {O} \left(\rho^ {t}\right) \quad \text {w h e r e} \rho := \left(1 - \frac {3 \sigma_ {\min } ^ {2} \mu}{3 2 \kappa \beta^ {2}}\right) <   1. \tag {C.25}
$$
Leveraging the equivalence of  $\mathrm{Err}_{\Theta}(\theta_t)$  and  $\mathrm{Err}_{\mathcal{X}}(x_t) = E_t$  completes the proof.
This section demonstrates the method's applicability in a series of different applications. Along with revisiting the established examples of Section 5 in greater detail, we also present how the PHGD method performs in a couple of additional settings of interest. We begin with a high-level description of the common test suite, and afterward, we move to the definitive details of each of the presented applications.
In each example, we define a base game  $\Gamma$  among two or more players  $\mathcal{N} \equiv \{1, \dots, N\}$ . Each player  $i$  controls a  $m$ -dimensional vector of control variables  $\theta_i \in \mathbb{R}^m$ , which they feed in an individual differentiable preconfigured MLP with two hidden layers that act as the player's representation map  $\chi_i: \mathbb{R}^m \to \mathcal{X}_i$ . The dimensions of each layer of the MLPs vary among the different examples. However, the MLP's output  $x_i = \chi_i(\theta_i)$  is guaranteed to be a representation of a discrete probability distribution among the player's actions in some normal-form game, e.g., a Rock-Paper-Scissors game. The actual latent game  $\mathcal{G}$  of  $\Gamma$  is given by the "hidden" loss functions  $f_i: \mathcal{X} \to \mathbb{R}$ ,  $i \in \mathcal{N}$ . Each loss function  $f_i$  is a regularized variation of the aforementioned normal-form game, tuned by some  $\frac{\mu}{2}$ -strongly convex regularizer  $h_\mu(x) = \frac{\mu}{2} \cdot \left( \sum_{i \in \mathcal{N}} \| x_i - x_i^* \|^2 \right)$ , where  $x^*$  is the normal-form game's unique equilibrium point. The modulus  $\mu$  is a hyper-parameter, which we tune such that the vector field  $g(x) = (\nabla_{x_i} f_i(x))_{i \in \mathcal{N}}$  to be strongly monotone, and to avoid finite-precision numerical errors, which can potentially arise during the computation of  $x$ .
A Hidden Game of Matching Pennies. As a first example, we revisit the two-player hidden game of Matching Pennies we introduced in Section 5. Here, each player's  $i$  control variables  $\theta_{i}$  are uni-dimensional, and are fed to each player's individual MLP given by the representation maps
$$
\chi_ {i} \left(\theta_ {i}\right) = \operatorname {s i g m o i d} \left(\alpha_ {i} ^ {(2)} \cdot C e L U \left(\alpha_ {i} ^ {(1)} \cdot \theta_ {i}\right)\right) \quad \text {f o r a l l} i = 1, 2, \tag {D.1}
$$
where  $\alpha_{i}^{(1)},\alpha_{i}^{(2)}\in [-1,1]$  are randomly chosen. Although the definition of the activation functions sigmoid:  $\mathbb{R}\to [0,1]$ , and  $CeLU:\mathbb{R}\rightarrow (-1,\infty)$  are widely common, we give them below for reference; that is:
$$
\operatorname {s i g m o i d} (x) = \left(1 + \exp (- x)\right) ^ {- 1} \tag {D.2a}
$$
$$
C e L U (x) = \max  \{0, x \} + \min  \{0, \exp (x) - 1 \}. \tag {D.2b}
$$
In this, and the following examples, without any loss of the generality, we deliberately set the bias of each of the MLP's hidden layers to zero, in order for the base game's unique equilibrium to lie in  $\theta^{*} = \vec{0}$ , and to simplify the notation. Each of the MLP's output  $x_{i} = \chi (\theta_{i})$  is the  $i$ -th player's probability of playing Heads in a regularized game of Matching Pennies, given by the loss functions
$$
f _ {1} (x) = - f _ {2} (x) = - \left(2 x _ {1} - 1\right) \cdot \left(2 x _ {2} - 1\right) + h _ {0. 7 5} (x). \tag {D.3}
$$
The latent game's unique equilibrium, in this case, is at  $x^{*} = \chi (\theta^{*}) = (\frac{1}{2},\frac{1}{2})$ .
In this particular example it is possible to visualize the trajectories of PHGD and gradient descent with reference, both, the space of control variables  $\mathbb{R}^2$ , and the space of latent variables  $[0,1]^2$ . In Fig. 7a we depict the trajectory of PHGD, with step-size 0.01, in the above game, initialized at the arbitrary state (1.25, 2.25) with respect to, both, the space of control variables (left), and the space of latent variables (right), over the sub-level sets of the energy function in (5). A similar trajectory of the GD algorithm is depicted in Fig. 7b. Observe that the PHGD's crosses the sub-level sets of the energy function at most once until it asymptotically reaches the game's equilibrium point at  $\theta^{*}$ , as opposed to the trajectory of GD, which exhibits an erratic behavior. In particular, as is depicted in the semi-log plot in Fig. 7c, PHGD converges to the game's equilibrium point at an exponential rate as opposed to the rate of GD, which, at best, can be described as linear.
A Hidden Game of Rock-Paper-Scissors. In the next example we consider a regularized hidden game of Rock-Paper-Scissors between two players. In a standard Rock-Paper-Scissors game each player may choose among three strategies, namely, Rock, Paper, or Scissors. No strategy dominates over the others, with Rock ruling over Scissors, Paper ruling over Rock, and Scissors ruling over Paper. In this example, each player  $i$  controls a 5-dimensional vector  $\theta_{i} \in \mathbb{R}^{5}$  of control variables, which, once again, they may feed their individual differentiable preconfigured MLP, whose output  $x_{i} = \chi_{i}(\theta_{i})$  lies in the 2-dimensional simplex that encodes the space of probability distributions among the three strategies of the latent game. The two MLPs are given by the following representation maps:
$$
\chi_ {i} \left(\theta_ {i}\right) = \operatorname {s o f t m a x} \left(A _ {i} ^ {(2)} \times C e L U \left(A _ {i} ^ {(1)} \times \theta_ {i}\right)\right) \quad \text {f o r a l l} i = 1, 2, \tag {D.4}
$$
where the matrices  $A_{i}^{(1)} \in [-1,1]^{4 \times 5}$ , and  $A_{i}^{(2)} \in [-1,1]^{3 \times 4}$  are random, and the activation function  $CeLU: \mathbb{R} \to (-1,\infty)$ , given as in (D.2b), is applied pair-wise. The definition of  $\text{softmax}: \mathbb{R}^d \to \Delta_{d-1}$ , where  $\Delta_{d-1}$  is the  $(d-1)$ -dimensional simplex is given, for reference, by
$$
\operatorname {s o f t m a x} _ {j} (x) = \frac {\exp \left(x _ {j}\right)}{\sum_ {k = 1} ^ {d} \exp \left(x _ {k}\right)} \quad \text {f o r a l l} j = 1, \dots , d. \tag {D.5}
$$
In this case, the latent game's loss functions are given by the following polynomial system of equations, whose unique equilibrium lies at the uniform distributions  $x_{i}^{*} = \chi_{i}(\theta_{i}^{*}) = (\frac{1}{3},\frac{1}{3},\frac{1}{3}), i = 1,2$ :
$$
f _ {1} (x) = - f _ {2} (x) = - x _ {1} ^ {\top} A x _ {2} + h _ {0. 2} (x) \quad \text {w h e r e} A = \left[ \begin{array}{r r r} 0 & - 1 & 1 \\ 1 & 0 & - 1 \\ - 1 & 1 & 0 \end{array} \right]. \tag {D.6}
$$
Although, in this example, it is not possible to visualize the trajectories of PHGD and GD in the space of control variables due to the large dimensionality of the base game, we may still get a glimpse of the trajectories' behavior in the space of latent variables. Fig. 8a depicts an arbitrary trajectory of PHGD with step-size 0.01, as it evolves in the simplices of the two players. Notice, that the trajectory, clearly, converges to the equilibrium of the latent game. A trajectory of GD, with the same step size and initialization point, is depicted in Fig. 8b. In this case, the erratic behavior of GD is more apparent than in the previous example. A more in-depth comparison of the algorithms in the current setup is depicted in the semi-log plot of Fig. 8c, where we visualize the mean distance, and confidence bounds, to the equilibrium point across 100 random trajectories of the two algorithms. Observe, that PHGD exhibits an exponential rate of convergence, as opposed to the linear rate of GD.
A Hidden Shapley's Game. The next game we are interested in is a hidden Shapley's game. The standard Shapley's game is a two-player normal-form game with payoff matrices:
$$
A = \left[ \begin{array}{l l l} 1 & 0 & \beta \\ \beta & 1 & 0 \\ 0 & \beta & 1 \end{array} \right] \quad \text {a n d} \quad B = \left[ \begin{array}{c c c} - \beta & 1 & 0 \\ 0 & - \beta & 1 \\ 1 & 0 & - \beta \end{array} \right], \tag {D.7}
$$
Figure 9: The mean distance and confidence bounds of  $x$  to the latent game's equilibrium point  $x^{*}$  along similar trajectories of PHGD and GD in a regularized Shapley game. The distance is depicted in a logarithmic scale in order to reveal the exponential rate of convergence of PHGD.
for some  $\beta \in (0,1)$ . The setup of this example is quite similar to the one in the previous example. However, there are a few noteworthy differences. To begin with, the hidden Shapley's game is not a zero-sum game. Specifically, the latent game's loss functions are given by the following polynomial system of equations:
$$
f _ {1} (x) = - x _ {1} ^ {\intercal} A x _ {2} + h _ {0. 2} (x)
$$
$$
f _ {2} (x) = - x _ {2} ^ {\intercal} B ^ {\intercal} x _ {1} + h _ {0. 2} (x). \tag {D.8}
$$
As in the hidden Rock-Paper-Scissors game, this game's unique equilibrium also lies in  $x_{i}^{*} = \chi_{i}(\theta_{i}^{*}) = (\frac{1}{3},\frac{1}{3},\frac{1}{3})$ ,  $i = 1,2$ . A second difference is the small modulus,  $\mu = 0.2$ , that we choose for the strongly-convex regularizer of this game. In Fig. 9 we depict a semi-log plot of the mean performance of PHGD and GD in the above game constructed using the same parameters as in the case of the hidden Rock-Paper-Scissors game. Notice that although the behaviour of PHGD is similar in both games, the confidence bounds of GD are drastically larger in the current example.
Figure 10: The mean distance and confidence bounds of  $x$  to the latent game's equilibrium point  $x^{*}$  along similar trajectories of PHGD and GD in a regularized El Farol Bar game. The distance is depicted in a logarithmic scale in order to reveal the exponential rate of convergence of PHGD.
A Hidden El Farol Bar Game. As a final example, we are going to revisit and describe the details of the hidden El Farol Bar game we introduced in Section 5. The El Farol Bar game is a famous congestion game that is often described as a game between populations. However, in this example, we are interested in its atomic  $N$ -playe variant. In the standard El Farol Bar game each player is given the option of visiting a specific bar in El Farol, and the outcome of the game is determined based on the number of players that decided to visit the bar. From the perspective of the player, there are three possible situations they may encounter, which carry some respective payoff. If the player
decides to do not to visit the bar, then, independently of the number of bar tenants, the player receives a payoff  $S$ . On the other hand, if the player decides to visit the El Farol bar, then depending on how crowded the bar is, they may receive a payoff that is strictly smaller, or strictly larger than  $S$ . If the bar is crowded, i.e., more than  $C \geq 0$  other players have visited the bar at the same time, then each of them receives payoff  $B < S$ . However, if the bar is not crowded, i.e., the total number of tenants is less than  $C$ , then they receive payoff  $G > S$ . It's not difficult to verify that the El Farol Bar game has a unique mixed Nash equilibrium, where each player chooses to visit the bar with probability  $\frac{C}{N}$ .