Preliminaries
Differential Game. The theory of differential games was initially proposed by Isaacs ( 1965 ) , aiming to expand the scope of sequential game theory to encompass continuous-time scenarios.
Through the lens of machine learning, we formalize the differential game, as shown in Definition 3.1 .

A differential game could be defined as a tuple \{{\mathcal{N}},{\bm{w}},{\bm{\ell}}\} , where {\mathcal{N}}=\{1,\dots,n\} delineates the assembly of players. The parameter set {\bm{w}}=[{\bm{w}}_{i}]^{n}\in\mathbb{R}^{d} is defined, each with {\bm{w}}_{i}\in\mathbb{R}^{d_{i}} and d=\sum_{i=1}^{n}d_{i} . Here, {\bm{\ell}}=\{\ell_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}\}_{i=1}^{n} represents the corresponding losses. These losses are assumed to be at least twice differential. Each player i\in{\mathcal{N}} is equipped with a policy, parameterized by {\bm{w}}_{i} , aiming to minimize its individual loss \ell_{i} .

We rewrite the simultaneous gradient {\bm{\xi}}({\bm{w}}) of a differential game as {\bm{\xi}}({\bm{w}})=(\nabla_{{\bm{w}}_{1}}\ell_{1},\dots,\nabla_{{\bm{w}}_{n}}\ell_{n})\in\mathbb{R}^{d}, which is the gradient of the losses with respect to the parameters of the respective players.
Furthermore, Hessian matrix {\bm{H}} mentioned in a differential game is the Jacobian matrix of the simultaneous gradient.

The learning dynamics of differential game often refers to the process of sequential updates over {\bm{w}} . The learning rule for each player is defined as the operator, {\bm{w}}\leftarrow{\bm{w}}-\gamma{\bm{\xi}}, where {\bm{\xi}} is a feasible direction, and \gamma is a step size (learning rate) to determine the distance to move for each update.

Gradient Adjustment Optimization. The stable fixed point, a criterion initiated from stability theory, corresponds to the nature of local Nash equilibria (Balduzzi et al., 2018 ) , as defined within the field of game theory. More specifically, a stable fixed point, exhibiting its stability (robustness) to minor perturbations from environments, making it applicable to many real-world scenarios. The concept of stable fixed point prevailing in general-sum and zero-sum games, is defined as follows.

A point {\bm{w}}^{\star} is a fixed point if {\bm{\xi}}({\bm{w}}^{\star})=0 . If {\bm{H}}({\bm{w}}^{\star})\succeq 0 and {\bm{H}}({\bm{w}}^{\star}) is invertible, the fixed point {\bm{w}}^{\star} is called stable fixed point. If {\bm{H}}({\bm{w}}^{\star})\prec 0 , the point is called unstable.

A naive idea to steer the dynamic towards convergence at stable fixed points involves minimizing \frac{1}{2}\|{\bm{\xi}}({\bm{w}})\|^{2} . Assuming the Hessian {\bm{H}}({\bm{w}}) is invertible, then \nabla(\frac{1}{2}\|{\bm{\xi}}({\bm{w}})\|^{2})={\bm{H}}^{T}{\bm{\xi}}=0 holds true if and only if {\bm{\xi}}=0 .
However, it could fail to converge to stable fixed point (Mescheder et al., 2017 ) .
Hence, the consensus optimization method has been proposed, incorporating gradient adjustment (Mescheder et al., 2017 ) , shown as follows:

\tilde{{\bm{\xi}}}={\bm{\xi}}+\lambda\cdot abla\frac{1}{2}\|{\bm{\xi}}({\bm{w}})\|^{2}={\bm{\xi}}+\lambda\cdot{\bm{H}}^{T}{\bm{\xi}}

For simplicity, we will refer to the consensus gradient adjustment as CGA .
While CGA proves effective in certain specific scenarios, such as two-player zero-sum games, it unfortunately falls short in general games (Balduzzi et al., 2018 ) .
To address this shortage, symplectic gradient adjustment ( SGA ) (Balduzzi et al., 2018 ; Letcher et al., 2019 ) is introduced to find the stable fixed point in general sum games, such that \tilde{{\bm{\xi}}}={\bm{\xi}}+\lambda\cdot{\bm{A}}^{T}{\bm{\xi}}. Herein, {\bm{A}} represents the antisymmetric component of the generalized Helmholtz decomposition of {\bm{H}} , {\bm{H}}({\bm{w}})={\bm{S}}({\bm{w}})+{\bm{A}}({\bm{w}}) , where {\bm{S}}({\bm{w}}) denotes the symmetric component. The hyperparameter \lambda is determined by the following formula: \lambda=\operatorname{sign}\left(\frac{1}{d}\langle\bm{\xi},{\bm{H}}^{T}{\bm{\xi}}\rangle\langle\mathbf{A}^{\top}\bm{\xi},{\bm{H}}^{T}{\bm{\xi}}\rangle+\epsilon\right), where \epsilon is a small positive number.