SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Higher-Order Uncoupled Dynamics Do Not Lead to Nash Equilibrium — Except When They Do
Sarah A. Toonsi
Industrial and Enterprise Systems Engineering
Urbana, Illinois, USA
stoonsi2@illinois.edu
Jeff S. Shamma
jshamma@illinois.edu
The framework of multi-agent learning explores the dynamics of how an agent's strategies evolve in response to the evolving strategies of other agents. Of particular interest is whether or not agent strategies converge to well-known solution concepts such as Nash Equilibrium (NE). In "higher-order" learning, agent dynamics include auxiliary states that can capture phenomena such as path dependencies. We introduce higher-order gradient play dynamics that resemble projected gradient ascent with auxiliary states. The dynamics are "payoff-based" and "uncoupled" in that each agent's dynamics depend on its own evolving payoff and has no explicit dependence on the utilities of other agents. We first show that for any polymatrix game with an isolated completely mixed-strategy NE, there exist higher-order gradient play dynamics that lead (locally) to that NE, both for the specific game and nearby games with perturbed utility functions. Conversely, we show that for any higher-order gradient play dynamics, there exists a game with a unique isolated completely mixed-strategy NE for which the dynamics do not lead to NE. Finally, we show that convergence to the mixed-strategy equilibrium in coordination games can come at the expense of the dynamics being inherently internally unstable.
The field of learning in games explores how game-theoretic solution concepts emerge as the outcome of dynamic processes where agents adapt their strategies in response to the evolving strategies of other agents (1; 2; 3; 4). There is a multitude of specific cases of learning dynamics/game combinations that result in a range of outcomes, including convergence, limit cycles, chaotic behavior, and stochastic stability (5; 6; 7; 8; 9; 10; 11; 12; 13). The emphasis in the literature is on simple adaptive procedures, called "natural" dynamics in (14), that can result in various solution concepts (e.g., Nash equilibrium, correlated equilibrium, and coarse correlated equilibrium). (A separate concern is the complexity associated with such computations (e.g. (15; 16)).)
One "natural" restriction for learning is that the dynamics of one agent should not depend explicitly on the utility functions of other agents. This restriction was referred to as "uncoupled" dynamics in (17), where the authors constructed a specific anti-coordination matrix game for which no uncoupled learning dynamics could converge to the unique mixed-strategy NE. The dynamics considered in that setting were of fixed order, i.e., the order of the learning dynamics was restricted to match the dimension of the strategy space. More recent work showed that specific instances of fixed order uncoupled learning dynamics can never lead to mixed-strategy NE (18). Furthermore, there exist games for which any fixed order learning dynamics are bound to have an initial condition starting from which the dynamics do not converge to NE (19).
The restriction on the order of the learning dynamics in learning mixed-strategy NE turns out to be essential. In particular, by introducing additional auxiliary states, (20) showed that higher-order learning could overcome the obstacle of convergence to NE in the same anti-coordination game considered in (17) while remaining uncoupled.
Higher-order learning in games can be seen as a parallel to higher-order optimization algorithms, such as momentum-based or optimistic gradient algorithms (e.g., (21; 22)). Such algorithms utilize history to update the underlying search parameter. In this way, there is a path dependency on the trajectory of information. An early utilization of higher-order learning is in (23), in which a player's strategy update uses two stages of history of an opponent's strategies in a zero-sum setting to eliminate oscillations. Similar ideas were used in (24). Reference (25) modified gradient-based algorithms through the introduction of a cumulative (integral) term. In (20), higher-order dynamics were used to create a myopic forecast of the action of other agents. Authors in (26) introduce a version of higher-order replicator dynamics and show that, unlike fixed order replicator dynamics, weakly dominated strategies become extinct. Reference (27) utilizes the system theoretic notion of passivity to analyze a family of higher-order dynamics.
In this paper, we further explore the implications of learning dynamics that are uncoupled. We address "payoff based" dynamics, in which the learning dynamics depend on the evolution of a payoff vector that is viewed as an externality. When players are engaged in a game, then the payoff stream of one agent depends on the actions of other agents. However, the learning dynamics themselves do not change based on the source of the payoff streams.
First, we show that for any polymatrix game with a mixed-strategy NE, there exist payoff-based dynamics that converge locally to that NE. This result is established by making a connection between convergence to NE and the existence of decentralized stabilizing control (28; 29). A consequence of the payoff-based structure is that the dynamics also converge to the NE of nearby perturbations of the original game. The form of higher-order learning used for this stability result is higher-order gradient play, which generalizes gradient ascent. Next, we show that for any such dynamics, there exists a game with a unique mixed-strategy NE that is unstable under given dynamics. The tool utilized is a classical analysis method in feedback control systems known as root-locus (e.g., (30), (31)), which characterizes the locations of the eigenvalues of a matrix as a function of a scalar parameter. A combination of the above results suggests the lack of universality on the side of both learning dynamics and games. While any mixed-strategy NE can be stabilized by suitable higher-order gradient dynamics, any such dynamics can be destabilized by a suitable anti-coordination game. Finally, we examine the implications of higher-order dynamics being able to converge to the mixed-strategy NE of a  $2 \times 2$  coordination game, which has two pure NE and one mixed-strategy NE. We show that such higher-order gradient play dynamics must have an inherent internal instability, which makes them unsuitable, if not irrational, as a model of learning.
2 Payoff-based learning dynamics
2.1 Finite polymatrix games
We consider finite (normal form) games over mixed-strategies. There are  $n$  players. The strategy space of player  $i \in \{1, 2, \dots, n\}$  is the probability simplex,  $\Delta(k_i)$ , where  $k_i$  is a positive integer and  $\Delta(\cdot)$  is defined as  $\Delta(\kappa) = \left\{s \in \mathbb{R}^{\kappa} \mid s_j \geq 0, j = 1, \dots, \kappa, \& \sum_{j=1}^{\kappa} s_j = 1\right\}$ . The utility function of player  $i$  is a function  $u_i: \Delta(k_1) \times \ldots \times \Delta(k_n) \to \mathbb{R}$ . We sometimes will write  $u_i(x_1, \ldots, x_n) = u_i(x_i, x_{-i})$  for  $x_i \in \Delta(k_i)$  and  $x_{-i} \in \mathcal{X}_{-i}$ , where  $\mathcal{X}_{-i} = \Delta(k_1) \times \ldots \times \Delta(k_{i-1}) \times \Delta(k_{i+1}) \times \ldots \times \Delta(k_n)$ .
For convenience, we will restrict our discussion to pairwise interactions, also called polymatrix games. That is, the utility function of player  $i$  is defined as
$$
u _ {i} \left(x _ {1}, \dots , x _ {n}\right) = x _ {i} ^ {\mathrm {T}} \sum_ {\substack {j = 1 \\ j \neq i}} ^ {n} M _ {i j} x _ {j}, \tag{1}
$$
for matrices,  $M_{ij}, j = 1, \ldots, n, j \neq i$ . We can write the utility function of player  $i$  as the inner product  $u_i(x_1, \ldots, x_n) = x_i^{\mathrm{T}} P_i(x_{-i})$  where  $P_i(x_{-i}) = \sum_{\substack{j=1 \\ j \neq i}}^{n} M_{ij} x_j \in \mathbb{R}^{k_i}$ . Accordingly, each element of  $P_i(x_{-i})$  can be viewed as a payoff that is associated with a component of player  $i$ 's strategy vector,  $x_i$ .
Figure 1: Payoff-based learning dynamics  $(\mathrm{LD}_i)$  in feedback with game matrices  $(M_{ij})$
A Nash equilibrium (NE) is a tuple  $(x_{1}^{*},\dots,x_{n}^{*})\in \mathcal{X}$  such that for all  $i = 1,\ldots ,n$
$$
u _ {i} \left(x _ {i} ^ {*}, x _ {- i} ^ {*}\right) \geq u _ {i} \left(x _ {i}, x _ {- i} ^ {*}\right), \quad \forall x _ {i} \in \Delta \left(k _ {i}\right).
$$
A completely mixed-strategy NE is such that each  $x_{i}^{*}$  is in the interior of the simplex.
2.2 Fixed order learning
Our model of learning is a dynamical system that relates trajectories of a payoff vector,  $p_i(t)$ , to trajectories of the strategy,  $x_i(t)$ . In particular, learning dynamics for player  $i$  are specified by a function  $f_i: \Delta(k_i) \times \mathbb{R}^{k_i} \to \mathbb{R}^{k_i}$  according to
$$
\dot {x} _ {i} (t) = f _ {i} \left(x _ {i} (t), p _ {i} (t)\right),
$$
where  $x_{i}(t) \in \Delta(k_{i})$  and  $p_{i} : \mathbb{R}_{+} \to \mathbb{R}^{k_{i}}$ . We assume implicitly that  $f_{i}$  and  $p_{i}$  are such that there exists a unique solution whenever  $x_{i}(0) \in \Delta(k_{i})$ . We further assume that the dynamics satisfy the invariance property that
$$
x _ {i} (0) \in \Delta \left(k _ {i}\right) \Rightarrow x _ {i} (t) \in \Delta \left(k _ {i}\right), \quad \forall t \geq 0. \tag {2}
$$
Note that we define learning dynamics without specifying the source of the payoff vector,  $p_i(t)$ , hence the terminology "payoff-based". Only once a player is coupled with other players in a game through their own (possibly heterogeneous) learning dynamics is when we make the connection  $p_i(t) = P_i(x_{-i}(t))$ .
This formulation is illustrated in Figure 1, where the  $\mathrm{LD}_i$  denote payoff-based learning dynamics that are interconnected through the game matrices,  $M_{ij}$ . Note that such learning dynamics are uncoupled by construction since each player can only access its own payoff vector. There is no dependence on the payoff stream of other players. Indeed, there is no dependence on the parameters of one's own utility function.
2.3 Higher-order learning
The learning dynamics described in the previous section have a fixed order associated with the dimension of the strategy space. Higher-order learning dynamics allow for the introduction of auxiliary states as follows. For any fixed order learning dynamics,  $f_{i}$ , we can define a higher-order version as
$$
\dot {x} _ {i} (t) = f _ {i} \left(x _ {i} (t), p _ {i} (t) + \phi_ {i} \left(p _ {i} (t), z _ {i} (t)\right)\right) \tag {3a}
$$
$$
\dot {z} _ {i} (t) = g _ {i} \left(p _ {i} (t), z _ {i} (t)\right). \tag {3b}
$$
As before,  $x_{i}(t) \in \Delta(k_{i})$  and  $p_{i}: \mathbb{R}_{+} \to \mathbb{R}^{k_{i}}$ . The new variable  $z_{i} \in \mathbb{R}^{\ell_{i}}$  represents  $\ell_{i}$  dimensional auxiliary states that evolve according to the  $p_{i}$ -dependent dynamics,  $g_{i}$ . These enter into the original fixed order dynamics through  $\phi_{i}$ . Accordingly, we can view  $p_{i}(t) + \phi_{i}(p_{i}(t), z_{i}(t))$  as a modified payoff stream that captures path dependencies in  $p_{i}$ , and the original learning dynamics react to this modified payoff stream.
As with the fixed order counterparts, there is no specification of game parameters in higher-order learning dynamics. In order to enforce that the auxiliary states have no effect on the equilibria of games, we make the following assumption.
Assumption 2.1. If  $p_i^*$  and  $z_i^*$  are an equilibrium of the higher-order dynamics, i.e.,
$$
0 = g _ {i} \left(p _ {i} ^ {*}, z _ {i} ^ {*}\right)
$$
$$
\phi_ {i} (p _ {i} ^ {*}, z _ {i} ^ {*}) = 0.
$$
This assumption assures that the auxiliary states represent purely transient phenomena that disappear at equilibrium.
Example 1 (Anticipatory higher-order dynamics). A special case of higher-order dynamics is
$$
\dot {z} _ {i} = \lambda \left(p _ {i} - z _ {i}\right)
$$
$$
\phi_ {i} = \gamma \lambda \left(p _ {i} - z _ {i}\right),
$$
where the higher-order modification is the linear system (see Appendix A.1)
$$
\left( \begin{array}{c c} - \lambda I & \lambda I \\ \hline - \gamma \lambda I & \gamma \lambda I \end{array} \right),
$$
and  $\gamma, \lambda \in \mathbb{R}_+$ . Intuition behind the connection to anticipation can be seen by viewing  $\lambda(p_i - z_i)$  as an approximation of  $\dot{p}_i$ , and so  $p_i(t) + \gamma \lambda(p_i(t) - z_i(t)) \approx p_i(t + \gamma)$ . Similar higher-order dynamics were used in reference (20) for smooth fictitious play to overcome the lack of convergence of uncoupled dynamics to NE in the anti-coordination game analyzed by (17). See also reference (32) for an analysis with replicator dynamics.
Anticipatory higher-order dynamics can also be linked to optimistic optimization algorithms (e.g., (22)). An Euler discretization of step size,  $h$ , results in
$$
z _ {i} ^ {+} = z _ {i} + h \lambda (p _ {i} - z _ {i}),
$$
with a modified payoff stream of
$$
p _ {i} + \phi_ {i} = p _ {i} + \gamma \lambda (p _ {i} - z _ {i}).
$$
Setting  $h = 1 / \lambda$  and  $\gamma = 1 / \lambda$  results in
$$
\begin{array}{l} p _ {i} + \phi_ {i} = p _ {i} + \left(p _ {i} - z _ {i}\right) \\ = p _ {i} + \left(p _ {i} - p _ {i} ^ {-}\right), \\ \end{array}
$$
where the superscripts  $+$  and  $-$  indicate the next and previous discrete time steps, respectively. There are also optimistic variants of discrete-time no-regret learning algorithms (33; 34; 35) that guarantee faster convergence rates to coarse correlated equilibria compared to the standard versions.
3 Gradient play
The main results of the paper will examine the behavior of gradient play and higher-order gradient play, which are the focus of this section.
3.1 Fixed order gradient play
In gradient play dynamics, a player adjusts its strategy in the direction of the payoff stream, i.e.,
$$
\dot {x} _ {i} = \Pi_ {\Delta} \left[ x _ {i} + p _ {i} \right] - x _ {i}, \tag {4}
$$
where  $\Pi_{\Delta}[x]: \mathbb{R}^n \to \Delta(n)$  is the projection of  $x$  into the simplex, i.e.,  $\Pi_{\Delta}(x) = \arg \min_{s \in \Delta(n)} \| x - s \|$ .
The terminology "gradient play" stems from the gradient of an agent's utility function in (1) with respect to its own strategy,  $x_{i}$ , namely  $\nabla_{x_i}u_i(x_i,x_{-i}) = P_i(x_{-i}) = \sum_{\substack{j = 1\\ j\neq i}}^{n}M_{ij}x_j$ . As was done in the description of payoff-based learning, we replace  $P_{i}(x_{-i})$  with the payoff stream  $p_i$  without regard to the game matrices  $M_{ij}$ .
Our primary concern will be studying these dynamics near a completely mixed-strategy NE. To this end, let  $x^{*} = (x_{1}^{*},\ldots ,x_{n}^{*})$  be an isolated completely mixed-strategy NE. The strategy vector  $x_{i}$
Figure 2: Cascade representation of linear higher-order dynamics for gradient play.
evolves on the simplex, which is a subset of dimension  $k_{i} - 1$ . Hence, the local behavior of the dynamics around  $x_{i}^{*}$  is characterized by evolution on a lower-dimensional subset. Thus, we can write
$$
x _ {i} = x _ {i} ^ {*} + N _ {i} w _ {i} \tag {5}
$$
$$
\mathbf {1} ^ {\mathrm {T}} N _ {i} = 0 \& N _ {i} ^ {\mathrm {T}} N _ {i} = I. \tag {6}
$$
Therefore,  $w_{i}\in \mathbb{R}^{(k_{i} - 1)}$  represents deviations from  $x_{i}^{*}$  and satisfies  $w_{i}(t) = N_{i}^{\mathrm{T}}(x_{i}(t) - x_{i}^{*})$ . When all players utilize fixed order gradient play, the collective dynamics near a completely mixed-strategy NE take the form
$$
\dot {w} = \mathcal {M} w, \tag {7}
$$
$$
\mathcal {M} = \mathcal {N} ^ {\mathrm {T}} \left( \begin{array}{c c c c c} 0 & M _ {1 2} & M _ {1 3} & \dots & M _ {1 n} \\ M _ {2 1} & 0 & M _ {2 3} & \dots & M _ {2 n} \\ M _ {3 1} & M _ {3 2} & 0 & \dots & M _ {3 n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ M _ {n 1} & M _ {n 2} & M _ {n 3} & \dots & 0 \end{array} \right) \mathcal {N}, \quad \mathcal {N} = \left( \begin{array}{c c c c c} N _ {1} & & & \\ & \ddots & \\ & & & N _ {n} \end{array} \right). \tag {8}
$$
Given the zero trace of  $\mathcal{M}$ , standard gradient play is always unstable at a completely mixed-strategy NE (see Appendix A.3 for stability conditions). Also, for this equilibrium to be isolated,  $\mathcal{M}$  must be non-singular. Otherwise,  $\mathcal{M}$  has a non-trivial null space leading to an equilibrium subspace.
3.2 Higher-order gradient play
We will be interested in a specific form of higher-order gradient play that uses the following linear structure of higher-order dynamics:
$$
\dot {x} _ {i} = - x _ {i} + \Pi_ {\Delta} \left[ x _ {i} + p _ {i} + N _ {i} \left(G _ {i} \xi_ {i} + H _ {i} \left(N _ {i} ^ {T} p _ {i} - v _ {i}\right)\right) \right]
$$
$$
\dot {\xi} _ {i} = E _ {i} \xi_ {i} + F _ {i} \left(N _ {i} ^ {\mathrm {T}} p _ {i} - v _ {i}\right)
$$
$$
\dot {v} _ {i} = N _ {i} ^ {\mathrm {T}} p _ {i} - v _ {i},
$$
for some matrices  $E_{i}$ ,  $F_{i}$ ,  $G_{i}$  and  $H_{i}$ . Here, the auxiliary states are  $z_{i} = (v_{i},\xi_{i})$ , which enter into the dynamics through  $\phi_i(p_i,\xi_i,v_i) = N_i(G_i\xi_i + H_i(N_i^{\mathrm{T}}p_i - v_i))$ , where  $N_{i}$  is defined as in (6).
The motivation behind this structure, illustrated in Figure 2, assures the enforcement of Assumption 2.1. The payoff stream is first preprocessed by a specific linear system to produce  $y_{i}$  and then by a general linear system  $K_{i}$  to produce  $u_{i}$ . The matrices  $(E_{i}, F_{i}, G_{i}, H_{i})$  create the dynamical system
$$
K _ {i} \sim \left( \begin{array}{c c} E _ {i} & F _ {i} \\ \hline G _ {i} & H _ {i} \end{array} \right) \tag {9}
$$
that maps  $y_{i} = N_{i}^{\mathrm{T}}p_{i} - v_{i}$  to  $u_{i} = G_{i}\xi_{i} + H_{i}y_{i}$  via  $\dot{\xi}_i = E_i\xi_i + F_iy_i$ . The preprocessing system has the property (see "washout filters" in Appendix A.2) that if  $p_i(t)$  converges to a constant, then  $y_{i}(t)$  converges to zero. Accordingly, when  $E_{i}$  is non-singular, the preprocessing system guarantees Nash stationarity, i.e., if
$$
\lim  _ {t \rightarrow \infty} (x _ {i} (t), \xi_ {i} (t), v _ {i} (t)) = (x _ {i} ^ {*}, \xi_ {i} ^ {*}, v _ {i} ^ {*}) \quad \forall i
$$
then  $x^{*} = (x_{1}^{*},\ldots ,x_{n}^{*})$  is a NE. To conclude, the linear system  $K_{i}$  is the central entity that performs the payoff modification, and the preprocessing system ensures compliance with Assumption 2.1.
3.3 Local stability analysis
As before, we can analyze the behavior near a completely mixed-strategy NE  $x^{*}$  through the variable  $w_{i}$  defined as in (5). Using the fact that  $N_{i}^{\mathrm{T}}\sum_{j\neq i}M_{ij}x_{j}^{*} = 0$ , we can write the local dynamics of a player as
$$
\dot {w} _ {i} = N _ {i} ^ {\mathrm {T}} \sum_ {j \neq i} M _ {i j} N _ {j} w _ {j} + G _ {i} \xi_ {i} + H _ {i} \left(N _ {i} ^ {\mathrm {T}} \sum_ {j \neq i} M _ {i j} N _ {j} w _ {j} - v _ {i}\right)
$$
$$
\dot {\xi} _ {i} = E _ {i} \xi_ {i} + F _ {i} \left(N _ {i} ^ {\mathrm {T}} \sum_ {j \neq i} M _ {i j} N _ {j} w _ {j} - v _ {i}\right)
$$
$$
\dot {v} _ {i} = N _ {i} ^ {\mathrm {T}} \sum_ {j \neq i} M _ {i j} N _ {j} w _ {j} - v _ {i}.
$$
The collective dynamics near a mixed-strategy NE can be written as
$$
\left( \begin{array}{c} \dot {w} \\ \dot {\xi} \\ \dot {v} \end{array} \right) = \left( \begin{array}{c c c} (I + H) \mathcal {M} & G & - H \\ F \mathcal {M} & E & - F \\ \mathcal {M} & 0 & - I \end{array} \right) \left( \begin{array}{c} w \\ \xi \\ v \end{array} \right), \tag {10}
$$
where  $E, F, G$ , and  $H$  are block diagonal matrices with appropriate dimensions and  $\mathcal{M}$  is defined in (8). Local stability of a completely mixed NE is determined by whether the above collective dynamics are stable, i.e., the dynamics matrix in (10) is a stability matrix.
4 Uncoupled dynamics that lead to mixed-strategy NE
4.1 Decentralized control formulation
The stability of a mixed-strategy equilibrium is tied to the existence of  $K_{1}$ ,  $K_{2}$ , ...,  $K_{n}$  so that the linear system in (10) is stable. When the  $K_{i}$  have yet to be determined, we can rewrite (10) as
$$
\left( \begin{array}{l} \dot {w} \\ \dot {v} \end{array} \right) = \left( \begin{array}{c c} \mathcal {M} & 0 \\ \mathcal {M} & - I \end{array} \right) \left( \begin{array}{l} w \\ v \end{array} \right) + \left( \begin{array}{l} I \\ 0 \end{array} \right) u, \tag {11a}
$$
$$
y = \left(\mathcal {M} - I\right) \binom {w} {v}, \tag {11b}
$$
where  $u = \begin{pmatrix} u_1 \\ \vdots \\ u_n \end{pmatrix}$ ,  $y = \begin{pmatrix} y_1 \\ \vdots \\ y_n \end{pmatrix}$ , and the  $y_i$  and  $u_i$  are to be related through  $K_i$ .
4.2 Decentralized stabilization
$$
\mathcal {P} \sim \left( \begin{array}{c c} \mathcal {A} & \mathcal {B} \\ \hline \mathcal {C} & 0 \end{array} \right)
$$
$$
\mathcal {A} = \left( \begin{array}{c c} \mathcal {M} & 0 \\ \mathcal {M} & - I \end{array} \right), \quad \mathcal {B} = \left( \begin{array}{c} I \\ 0 \end{array} \right), \quad \mathcal {C} = \left(\mathcal {M} - I\right). \tag {12}
$$
We first establish that  $\mathcal{P}$  can be stabilized by verifying the conditions for stabilizability and detectability (see Appendix A.3). The assumption that  $\mathcal{M}$  is non-singular stems from our interest in isolated NE.
Proposition 4.1. For  $\mathcal{M}$  non-singular, the pair  $(\mathcal{A},\mathcal{B})$  is stabilizable, and the pair  $(\mathcal{A},\mathcal{C})$  is detectable.
While Proposition 4.1 establishes that  $\mathcal{P}$  can be stabilized, that property alone is inadequate for our purposes. In particular, for the learning dynamics to be uncoupled, we seek to establish decentralized stabilization (see Appendix A.4) according to the partition
$$
\binom {\dot {w}} {\dot {v}} = \mathcal {A} \binom {w} {v} + \sum_ {i = 1} ^ {n} \mathcal {B} _ {i} u _ {i}, \quad y _ {i} = \mathcal {C} _ {i} \binom {w} {v}, \tag {13}
$$
$$
\mathcal {A} = \left( \begin{array}{c c} \mathcal {M} & 0 \\ \mathcal {M} & - I \end{array} \right), \quad \mathcal {B} _ {i} = \left( \begin{array}{c} \mathcal {E} _ {i} \\ 0 \end{array} \right), \quad \mathcal {C} _ {i} = \left( \begin{array}{c c} \mathcal {M} _ {i •} & - \mathcal {E} _ {i} ^ {\mathrm {T}} \end{array} \right). \tag {14}
$$
Here,  $\mathcal{M}_{i\bullet}$  denotes the  $i^{\mathrm{th}}$  block row of  $\mathcal{M}$ , i.e.,
$$
\mathcal {M} _ {i \bullet} = \left( \begin{array}{c c c c c c c} N _ {i} ^ {\mathrm {T}} M _ {i 1} N _ {1} & ... & N _ {i} ^ {\mathrm {T}} M _ {i (i - 1)} N _ {i - 1} & 0 & N _ {i} ^ {\mathrm {T}} M _ {i (i + 1)} N _ {i + 1} & ... & N _ {i} ^ {\mathrm {T}} M _ {i n} N _ {n} \end{array} \right)
$$
$$
\mathcal {E} _ {i} ^ {\mathrm {T}} = \left( \begin{array}{c c c c c c c} 0 & \ldots & 0 & \underbrace {I} _ {i ^ {\text {t h}} \text {p o s i t i o n}} & 0 & \ldots & 0 \end{array} \right),
$$
where  $I$  has a dimension (suppressed in the notation) of  $k_{i} - 1$ .
Theorem 4.1. For any isolated (i.e.,  $\mathcal{M}$  is non-singular) completely mixed-strategy NE, there exist uncoupled higher-order gradient play dynamics such that (10) is stable.
The proof of Theorem 4.1 relies on the conditions of Theorem A.1 and is presented in Appendix B.1.
Theorem 4.1 should be viewed as a statement regarding whether uncoupled learning in itself is a barrier to learning dynamics leading to NE. The theorem makes no claim that the higher-order learning dynamics are interpretable (e.g., as in anticipatory learning). Nor does the theorem offer guidance on how agents may construct the matrices of higher-order learning that lead to convergence. In the next section, we will see that, while the structure is universal, any specific set of parameters is not universal in that one can construct a game for which they do not lead to NE. Despite the lack of universality, there is an inherent robustness that is a consequence of stability. The following follows from standard arguments on linear systems.
Proposition 4.2. Let the  $K_{i}$  and  $M_{ij}$ ,  $i = 1, \dots, n$  and  $j = 1, \dots, n$ , be such that (10) is stable. Then there exists a  $\delta > 0$  such that (10) is stable with the  $M_{ij}$  replaced by any  $\tilde{M}_{ij}$  as long as  $\left\| \tilde{M}_{ij} - M_{ij} \right\| < \delta$  for all  $i = 1, \dots, n$  and  $j = 1, \dots, n$ .