SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Cai
yang.cai@yale.edu
haipeng1@usc.edu
Chen-Yu Wei
Weiqiang Zheng
weiqiang.zheng@yale.edu
We revisit the problem of learning in two-player zero-sum Markov games, focusing on developing an algorithm that is uncoupled, convergent, and rational, with non-asymptotic convergence rates to Nash equilibrium. We start from the case of stateless matrix game with bandit feedback as a warm-up, showing an  $\mathcal{O}(t^{-\frac{1}{8}})$  last-iterate convergence rate. To the best of our knowledge, this is the first result that obtains finite last-iterate convergence rate given access to only bandit feedback. We extend our result to the case of irreducible Markov games, providing a last-iterate convergence rate of  $\mathcal{O}(t^{-\frac{1}{9 + \varepsilon}})$  for any  $\varepsilon >0$ . Finally, we study Markov games without any assumptions on the dynamics, and show a path convergence rate, a new notion of convergence we define, of  $\mathcal{O}(t^{-\frac{1}{10}})$ . Our algorithm removes the coordination and prior knowledge requirement of [WLZL21a], which pursued the same goals as us for irreducible Markov games. Our algorithm is related to [CMZ21, CWC21] and also builds on the entropy regularization technique. However, we remove their requirement of communications on the entropy values, making our algorithm entirely uncoupled.
In multi-agent learning, a central question is how to design algorithms so that agents can independently learn (i.e., with little coordination overhead) how to interact with each other. Additionally, it is desirable to maximally reuse existing single-agent learning algorithms, so that the multi-agent system can be built in a modular way. Motivated by this question, decentralized multi-agent learning emerges with the goal to design decentralized systems, in which no central controller governs the policies of the agents, and each agent learns based on only their local information – just like in a single-agent algorithm. In recent years, we have witnessed significant success of this new decentralized learning paradigm. For example, self-play, where each agent independently deploys the same single-agent algorithm to play against each other without further direct supervision, plays a crucial role in the training of AlphaGo  $\left[\mathrm{SSS}^{+}17\right]$  and AI for Stratego  $\left[\mathrm{PDVH}^{+}22\right]$ .
Despite the recent success, many important questions remain open in decentralized multi-agent learning. Indeed, unless the decentralized algorithm is carefully designed, self-play often falls short of attaining certain sought-after global characteristics, such as convergence to the global optimum or stability as seen in, for example, [MPP18, BP18].
In this work, we revisit the problem of learning in two-player zero-sum Markov games, which has received extensive attention recently. Our goal is to design a decentralized algorithm that resembles
standard single-agent reinforcement learning (RL) algorithms, but with an additional crucial assurance, that is, guaranteed convergence when both players deploy the algorithm. The simultaneous pursuit of independence and convergence has been advocated widely [BV01, AY16, WLZL21a,  $\mathrm{SZL}^{+}21$ ], while the results are still not entirely satisfactory. In particular, all of these results rely on assumptions on the dynamics of the Markov game. Our paper takes the first step to remove such assumptions.
More specifically, our goal is to design algorithms that simultaneously satisfy the following three properties (the definitions are adapted from [BV01, DDK11]):
- Uncoupled: Each player  $i$ 's action is generated by a standalone procedure  $\mathcal{P}_i$  which, in every round, only receives the current state and player  $i$ 's own reward as feedback (in particular, it has no knowledge about the actions or policies used by the opponent). There is no communication or shared randomness between the players.
- Convergent: The policy pair of the two players converges to a Nash equilibrium.
- Rational: If  $\mathcal{P}_i$  competes with an opponent who uses a policy sequence that converges to a stationary one, then  $\mathcal{P}_i$  converges to the best response of this stationary policy.
The uncoupledness and rationality property capture the independence of the algorithm, while the convergence property provides a desirable global guarantee. Interestingly, as argued in [WLZL21a], if an algorithm is uncoupled and convergent, then it is also rational, so we only need to ensure that the algorithm is uncoupled and convergent. Regarding the notion of convergence, the standard definition above only allows last-iterate convergence. Considering the difficulty of achieving such convergence, in the related work review (Section 2) and in the design of our algorithm for general Markov games (Section 6), we also consider weaker notions of convergence, including the best-iterate convergence, which only requires that the Cesaro mean of the duality gap is convergent, and the path convergence, which only requires the convergence of the Cesaro mean of the duality gap assuming minimax/maximin policies are followed in future steps. The precise definitions of these convergence notions are given at the end of Section 3.
The main results in this work are as follows (see also Table 1 for comparisons with prior works):
- As a warm-up, for the special case of matrix games with bandit feedback, we develop an uncoupled algorithm with a last-iterate convergence rate of  $\mathcal{O}\left(t^{-\frac{1}{8}}\right)$  under self-play (Section 4). To the best of our knowledge, this is the first algorithm with provable last-iterate convergence rate in the setting.
- Generalizing the ideas from matrix games, we further develop an uncoupled algorithm for irreducible Markov games with a last-iterate convergence rate of  $\mathcal{O}(t^{-\frac{1}{9 + \varepsilon}})$  for any  $\varepsilon > 0$  under self-play (Section 5).
- Finally, for general Markov games without additional assumptions, we develop an uncoupled algorithm with a path convergence rate of  $\mathcal{O}(t^{-\frac{1}{10}})$  under self-play (Section 6).
Our algorithms leverage recent advances on using entropy to regularize the policy updates [CWC21, CMZ21] and the Nash-V-styled value updates [BJY20]. On the one hand, compared to [CWC21, CMZ21], our algorithm has the following advantages: 1) it does not require the two players to exchange their entropy information, which allows our algorithm to be fully uncoupled; 2) it does not require the players to have coordinated policy updates, 3) it naturally extends to general Markov games without any assumptions on the dynamics (e.g., irreducibility). On the other hand, our algorithm inherits appealing properties of Nash-V [BJY20], but additionally guarantees path convergence during execution.
The study of two-player zero-sum Markov games originated from [Sha53], with many other works further developing algorithms and establishing convergence properties [HK66, PAI69, VDW78, FT91]. However, these works primarily focused on solving the game with full knowledge of its parameters (i.e., payoff function and transition kernel). The problem of learning in zero-sum games was first formalized by [Lit94]. Designing a provably uncoupled, rational, and convergent algorithm
Table 1: (Sample-based) Learning algorithms for finding NE in two-player zero-sum games. Our results are shaded. A halfcheck “ $\checkmark$ ” in the convergent column means that the policy convergence is proven only for one player (typically this is a result of asymmetric updates). (L) and (B) stand for last-iterate convergence and best-iterate convergence, respectively. (P) stands for path convergence, a weaker convergence notion we introduce (see Section 3, 6.1).
*: While [WLZL21a] also proposes an uncoupled and convergent algorithm for irreducible Markov games, their algorithm requires coordinated updates and some prior knowledge of the game, while ours does not. See Section 2.1 for a more detailed discussion.
Setting	Algorithm	Uncoupled?	Convergent?
Matrix Game	EXP3 vs. EXP3	✓	✗
Algorithm 1	✓	✓(L)
Markov game + assumptions on dynamics	[DFG20]	✓	✓(B)
[ZTLD22, AVHC22]	✓	✓(L)
[SZL+21]	✓	✗
[CMZ21]	✗	✓(L)
[WLZL21a]	✓*	✓(L)
Algorithm 2	✓	✓(L)
Markov Game	[WHL17, JJJN21, HLOWY22]	✗	✓(B)
[JLY22, XZS+22]
[BJ20, XCWY20]	✗	✓(B)
[LYBJ21, CZG22]
[BJY20, JLOWY21]	✓	✗
Algorithm 3	✓	✓(P)
is challenging, with many attempts [SL99, BV01, HW03, CS07, AY16, SPO22] falling short in one aspect or another, often lacking either uncoupledness or convergence. Moreover, these works only establish asymptotic convergence without providing a concrete convergence rate.
2.1 Non-asymptotic convergence guarantees
Recently, a large body of works on learning two-player zero-sum Markov games use regret minimization techniques to establish non-asymptotic guarantees. They focus on fast computation under full information of payoff and transitions [CWC21, CCDX23, ZLW+22, SLY23, YM23], though many of their algorithms are decentralized and can be viewed as the first step towards the learning setting.
With rationality and uncoupledness satisfied, [DFG20] established one-sided policy convergence for players using independent policy gradient with asymmetric learning rates. Such an asymmetric update rule is also adopted by [ZTLD22, AVHC22] to establish one-sided policy convergence guarantees. When using a symmetric update rule,  $\mathrm{[SZL^{+}21]}$  developed a decentralized-Q learning algorithm. However, the convergence is only shown for the  $V$ -function maintained by the players instead of the policies being used, so the policies may still cycle and are not provably convergent in our definition.  $\mathrm{[ELS^{+}23]}$  studied regret minimization in general-sum Markov games and provided an algorithm with sublinear regret under self-play and average-iterate convergence rates to equilibria, while our work focuses on last-iterate convergence rates to Nash equilibria.
To our knowledge, [WLZL21a] first provided an uncoupled, rational, and convergent algorithm with non-asymptotic convergence guarantee, albeit only for irreducible Markov game. They achieved this via optimistic gradient descent/ascent. Despite satisfying all our criteria, their algorithm still has unnatural coordination between the players and a requirement on some prior knowledge of the game such as the maximum revisiting time of the Markov game. Our algorithm removes all these extra requirements. A follow-up work by [CMZ21] improved the rate of [WLZL21a] using entropy regularization; however, this requires their players to inform the opponent about the entropy of their own policy, making the algorithm coupled again. We show that such an exchange of information is unnecessary under entropy regularization.
2.2 Further handling exploration
The algorithms introduced above all require full information or some assumption on the dynamics of the Markov game. To handle exploration, some works design coupled learning algorithms
which guarantee that the player's long-term payoff is at least the minimax value [BT02, WHL17, XCWY20, HLWY22, JLY22, JJJN21,  $\mathrm{XZS}^{+}22$ ]. Interestingly, as shown in [WHL17, HLWY22, JLY22,  $\mathrm{XZS}^{+}22$ ], if the player is paired with an optimistic best-response opponent (instead of using the same algorithm), the first player's strategy can converge to the minimax policy. [XCWY20, BJ20, LYBJ21, CZG22] developed another coupled learning framework to handle exploration, but with symmetric updates on both players. In each round, the players need to jointly solve a general-sum equilibrium problem due to the different exploration bonus added by each player. Hence, the execution of these algorithms is more similar to the Nash-Q algorithm by [HW03].
So far, exploration has been handled through coupled approaches that are also not rational. To our knowledge, the first uncoupled and rational algorithm that handles exploration is the Nash-V algorithm by [BJY20]. Nash-V can output a nearly-minimax policy through weighted averaging [JLWY21]; however, it is not provably convergent during execution. A major remaining open problem is whether one can design a natural algorithm that is provably rational, uncoupled, and convergent with exploration capability. Our work provides the first progress towards this goal.
2.3 Other works on last-iterate convergence
Uncoupled Learning dynamics in normal-form games with provable last-iterate convergence rate receives extensive attention recently. Most of the works assume that the players receive gradient feedback, and convergence results under bandit feedback remain sparse. Linear convergence is shown for strongly monotone games or bilinear games under gradient feedback [Tse95, LS19, MOP20, WLZL21b] and sublinear rates are proven for strongly monotone games with bandit feedback [BLM18, HIMM19, LZBZ21, TK22, DFR22, HH23]. Convergence rate to strict Nash equilibrium is analyzed by [GVGM21]. For monotone games that includes two-player zero-sum games as a special case, the last-iterate convergence rate of no-regret learning under gradient feedback has been shown recently [GPD20, COZ22, GTG22, CZ23]. With bandit feedback, [MPS20] showed an impossibility result that certain algorithms with optimal  $\mathcal{O}(\sqrt{T})$  regret do not converge in last-iterate. To the best of our knowledge, there is no natural uncoupled learning dynamics with provable last-iterate convergence rate in two-player zero-sum games with bandit feedback.
Basic Notations Throughout the paper, we assume for simplicity that the action set for the two players are the same, denoted by  $\mathcal{A}$  with cardinality  $A = |\mathcal{A}|$ . We usually call player 1 the  $x$ -player and player 2 the  $y$ -player. The set of mixed strategies over an action set  $\mathcal{A}$  is denoted as  $\Delta_{\mathcal{A}} := \{x : \sum_{a \in \mathcal{A}} x_a = 1; 0 \leq x_a \leq 1, \forall a \in \mathcal{A}\}$ . To simplify notation, we denote by  $z = (x, y)$  the concatenated strategy of the players. We use  $\phi$  as the entropy function such that  $\phi(x) = -\sum_{a \in \mathcal{A}} x_a \ln x_a$ , and KL as the Kullback-Leibler (KL) divergence such that  $\mathrm{KL}(x, x') = \sum_{a \in \mathcal{A}} x_a \ln \frac{x_a}{x'_a}$ . The all-one vector is denoted by  $\mathbf{1} = (1, 1, \dots, 1)$ .
Matrix Games In a two-player zero-sum matrix game with a loss matrix  $G \in [0,1]^{A \times A}$ , when the  $x$ -player chooses action  $a$  and the  $y$ -player chooses action  $b$ , the  $x$ -player suffers loss  $G_{a,b}$  and the  $y$ -player suffers loss  $-G_{a,b}$ . A pair of mixed strategy  $(x^{\star},y^{\star})$  is a Nash equilibrium for  $G$  if for any strategy profile  $(x,y) \in \Delta_{\mathcal{A}} \times \Delta_{\mathcal{A}}$ , it holds that  $(x^{\star})^{\top}Gy \leq (x^{\star})^{\top}Gy^{\star} \leq x^{\top}Gy^{\star}$ . Similarly,  $(x^{\star},y^{\star})$  is a Nash equilibrium for a two-player zero-sum game with a general convex-concave loss function  $f(x,y): \Delta_{\mathcal{A}} \times \Delta_{\mathcal{A}} \to \mathbb{R}$  if for all  $(x,y) \in \Delta_{\mathcal{A}} \times \Delta_{\mathcal{A}}$ ,  $f(x^{\star},y) \leq f(x^{\star},y^{\star}) \leq f(x,y^{\star})$ . The celebrated minimax theorem [vN28] guarantees the existence of Nash equilibria in two-player zero-sum games. For a pair of strategy  $(x,y)$ , we use duality gap defined as  $\mathrm{GAP}(G,x,y) \triangleq \max_{y'} x^\top Gy' - \min_{x'} x'^\top Gy$  to measure its proximity to Nash equilibria.
Markov Games A generalization of matrix games, which models dynamically changing environment, is Markov games. We consider infinite-horizon discounted two-player zero-sum Markov games, denoted by a tuple  $(S, \mathcal{A}, (G^s)_{s \in S}, (P^s)_{s \in S}, \gamma)$  where (1)  $S$  is a finite state space; (2)  $\mathcal{A}$  is a finite action space for both players; (3) Player 1 suffers loss  $G_{a,b}^s \in [0,1]$  (respectively player 2 suffers
<sup>1</sup>We make this assumption only to simplify notations; our proofs can be easily extended to the case where the action sets of the two players are different.
loss  $-G_{a,b}^{s}$ ) when player 1 chooses action  $a$  and player 2 chooses action  $b$  at state  $s$ ; (4)  $P$  is the transition function such that  $P_{a,b}^{s}(s^{\prime})$  is the probability of transiting to state  $s^{\prime}$  when player 1 plays  $a$  and player 2 plays  $b$  at state  $s$ ; (5)  $\gamma \in [\frac{1}{2}, 1)$  is a discount factor.
A stationary policy for player 1 is a mapping  $S \to \Delta_{\mathcal{A}}$  that specifies player 1's strategy  $x^{s} \in \Delta_{\mathcal{A}}$  at each state  $s \in S$ . We denote  $x = (x^{s})_{s \in S}$ . Similar notations apply to player 2. We denote  $z^{s} = (x^{s}, y^{s})$  as the concatenated strategy for the players and  $z = (x, y)$ . The value function  $V_{x,y}^{s}$  denotes the expected loss of player 1 (or the expected payoff of player 2) given a pair of stationary policy  $(x, y)$  and initial state  $s$ :
$$
V _ {x, y} ^ {s} = \mathbb {E} \bigg [ \sum_ {t = 1} ^ {\infty} \gamma^ {t - 1} G _ {a _ {t}, b _ {t}} ^ {s _ {t}} | s _ {1} = s, a _ {t} \sim x ^ {s _ {t}}, b _ {t} \sim y ^ {s _ {t}}, s _ {t + 1} \sim P _ {a _ {t}, b _ {t}} ^ {s _ {t}} (\cdot), \forall t \geq 1 \bigg ].
$$
The minimax game value on state  $s$  is defined as  $V_{\star}^{s} = \min_{x}\max_{y}V_{x,y}^{s} = \max_{y}\min_{x}V_{x,y}^{s}$ . We call a pair of policy  $(x_{\star},y_{\star})$  a Nash equilibrium if it attains the minimax game value of a state  $s$  (such policy pair necessarily attains the minimax game value over all states). The duality gap of  $(x,y)$  is  $\max_s\left(\max_{y'}V_{x,y'}^s -\min_{x'}V_{x',y'}^s\right)$ . The  $Q$ -function on state  $s$  under policy pair  $(x,y)$  is defined via  $Q_{x,y}^{s}(a,b) = G_{a,b}^{s} + \gamma \cdot \mathbb{E}_{s'\sim P_{a,b}^{s}(\cdot)}[V_{x,y}^{s'}]$ , which can be rewritten as a matrix  $Q_{x,y}^{s}$  such that  $V_{x,y}^{s} = x^{s}Q_{x,y}^{s}y^{s}$ . We denote  $Q_{\star}^{s} = Q_{x,\star ,y\star}^{s}$  the  $Q$ -function under a Nash equilibrium  $(x_{\star},y_{\star})$ . It is known that  $Q_{\star}^{s}$  is unique for any  $s$  even when multiple equilibria exist.
Uncoupled Learning with Bandit Feedback We assume the following uncoupled interaction protocol: at each round  $t = 1, \dots, T$ , the players both observe the current state  $s_t$ , and then, with the policy  $x_t$  and  $y_t$  in mind, they independently choose actions  $a_t \sim x_t^{s_t}$  and  $b_t \sim y_t^{s_t}$ , respectively. Both of them then observe  $\sigma_t \in [0,1]$  with  $\mathbb{E}[\sigma_t] = G_{a_t, b_t}^{s_t}$ , and proceed to the next state  $s_{t+1} \sim P_{a_t, b_t}^{s_t}(\cdot)$ . Importantly, they do not observe each other's action.
Notions of Convergence For Markov games with the irreducible assumption (Assumption 1), given players' history of play  $(s_t, x_t, y_t)_{t \in [T]}$ , the best-iterate convergence rate is measured by the average duality gap  $\frac{1}{T} \sum_{t=1}^{T} \max_{s,x,y} (V_{x_t,y}^s - V_{x,y_t}^s)$ , while the stronger last-iterate convergence rate is measured by  $\max_{s,x,y} (V_{x_T,y}^s - V_{x,y_T}^s)$ , i.e., the duality gap of  $(x_T, y_T)$ . For general Markov games, we propose the path convergence rate, which is measured by the average duality gap at the visited states with respect to the optimal  $Q$ -function:  $\frac{1}{T} \sum_{t=1}^{T} \max_{x,y} (x_t^{s_t^\top} Q_\star^{s_t} y^{s_t} - x_s^{s_t^\top} Q_\star^{s_t} y_t^{s_t})$ . We remark that the path convergence guarantee is weaker than the counterpart of the other two notions of convergence in general Markov games, but still provides meaningful implications (see detailed discussion in Section 6.1 and Appendix F).
4 Matrix Games
In this section, we consider two-player zero-sum matrix games. We propose Algorithm 1 for decentralized learning of Nash equilibria. We only present the algorithm for the  $x$ -player as the algorithm for the  $y$ -player is symmetric.
Algorithm 1 Matrix Game with Bandit Feedback
1: Define:  $\eta_t = t^{-k_\eta}$ ,  $\beta_t = t^{-k_\beta}$ ,  $\epsilon_t = t^{-k_\epsilon}$  where  $k_\eta = \frac{5}{8}$ ,  $k_\beta = \frac{3}{8}$ ,  $k_\epsilon = \frac{1}{8}$ .
$$
\Omega_ {t} = \left\{x \in \Delta_ {\mathcal {A}}: x _ {a} \geq \frac {1}{A t ^ {2}}, \forall a \in \mathcal {A} \right\}.
$$
2: Initialization::  $x_{1} = \frac{1}{A}\tilde{\mathbf{1}}$ .
3: for  $t = 1,2,\ldots$  do
4: Sample  $a_{t} \sim x_{t}$ , and receive  $\sigma_t \in [0,1]$  with  $\mathbb{E}[\sigma_t] = G_{a_t,b_t}$ .
5: Compute  $g_{t}$  where  $g_{t,a} = \frac{\mathbf{1}[a_t = a]\sigma_t}{x_{t,a} + \beta_t} +\epsilon_t\ln x_{t,a},\forall a\in \mathcal{A}.$
6: Update  $x_{t+1} \gets \operatorname{argmin}_{x \in \Omega_{t+1}} \left\{ x^{\top} g_t + \frac{1}{\eta_t} \mathrm{KL}(x, x_t) \right\}$ .
7: end for
The algorithm is similar to the Exp3-IX algorithm by [Neu15] that achieves a high-probability regret bound for adversarial multi-armed bandits, but with several modifications. First (and most
importantly), in addition to the standard loss estimators used in [Neu15], we add another negative term  $\epsilon_t\ln x_{t,a}$  to the loss estimator of action  $a$  (see Line 5). This is equivalent to the entropy regularization approach in, e.g., [CWC21, CMZ21], since the gradient of the negative entropy  $-\phi(x_{t})$  is  $(\ln x_{t,a} + 1)_{a\in \mathcal{A}}$  and the constant 1 takes no effect in Line 6. Like [CWC21, CMZ21], the entropy regularization drives last-iterate convergence; however, while their results require full-information feedback, our result holds in the bandit feedback setting. The second difference is that instead of choosing the players' strategies in the full probability simplex  $\Delta_{\mathcal{A}}$ , our algorithm chooses from  $\Omega_{t}$ , a subset of  $\Delta_{\mathcal{A}}$  where every coordinate is lower bounded by  $\frac{1}{At^2}$ . The third is the choices of the learning rate  $\eta_{t}$ , clipping factor  $\beta_{t}$ , and the amount of regularization  $\epsilon_{t}$ . The main result of this section is the following last-iterate convergence rate of Algorithm 1.
Theorem 1 (Last-Iterate Convergence Rate). Algorithm 1 guarantees with probability at least  $1 - \mathcal{O}(\delta)$ , for any  $t \geq 1$ ,
$$
\max  _ {x, y \in \Delta_ {A}} \left(x _ {t} ^ {\top} G y - x ^ {\top} G y _ {t}\right) = \mathcal {O} \left(\sqrt {A} \ln^ {3 / 2} (A t / \delta) t ^ {- \frac {1}{8}}\right).
$$
Algorithm 1 also guarantees  $\mathcal{O}(t^{-\frac{1}{8}})$  regret even when the other player is adversarial. If we only target at an expected bound instead of a high-probability bound, the last-iterate convergence rate can be improved to  $\mathcal{O}(\sqrt{A}\ln^{3/2}(At)t^{-\frac{1}{6}})$ . The details are provided in Appendix C.
4.1 Analysis Overview
We define a regularized zero-sum game with loss function  $f_{t}(x,y) = x^{\top}Gy - \epsilon_{t}\phi (x) + \epsilon_{t}\phi (y)$  over domain  $\Omega_t\times \Omega_t$ , and denote by  $z_{t}^{\star} = (x_{t}^{\star},y_{t}^{\star})$  its unique Nash equilibrium since  $f_{t}$  is strongly convex-strongly concave. The regularized game is a slight perturbation of the original matrix game  $G$  over a smaller domain  $\Omega_t\times \Omega_t$ , and we prove that  $z_{t}^{\star}$  is an  $\mathcal{O}(\epsilon_t)$ -approximate Nash equilibrium of the original matrix game  $G$  (Lemma 9). Therefore, it suffices to bound  $\mathrm{KL}(z_t^\star ,z_t)$  since the duality gap of  $z_{t}$  is at most  $\mathcal{O}(\sqrt{\mathrm{KL}(z_t^\star,z_t)} +\epsilon_t)$ .
Step 1: Single-Step Analysis We start with a single-step analysis of Algorithm 1, which shows:
$$
\mathrm {K L} (z _ {t + 1} ^ {\star}, z _ {t + 1}) \leq (1 - \eta_ {t} \epsilon_ {t}) \mathrm {K L} (z _ {t} ^ {\star}, z _ {t}) + \underbrace {2 0 \eta_ {t} ^ {2} A \ln^ {2} (A t) + 2 \eta_ {t} ^ {2} A \lambda_ {t}} _ {\text {i n s t a b i l i t y p e n a l t y}} + \underbrace {\eta_ {t} \xi_ {t} + \eta_ {t} \zeta_ {t}} _ {\text {e s t i m a t i o n e r r o r}} + v _ {t}
$$