SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weichao Mao
University of Illinois Urbana-Champaign weichao2@illinois.edu
Haoran Qiu
University of Illinois Urbana-Champaign haoranq4@illinois.edu
Chen Wang
chen.wang1@ibm.com
Hubertus Franke
frankeh@us.ibm.com
Zbigniew Kalbarczyk
kalbarcz@illinois.edu
Ravishankar K. Iyer
University of Illinois Urbana-Champaign rkiyer@illinois.edu
Tamer Başar
University of Illinois Urbana-Champaign  
basar1@illinois.edu
Multi-agent reinforcement learning (MARL) has primarily focused on solving a single task in isolation, while in practice the environment is often evolving, leaving many related tasks to be solved. In this paper, we investigate the benefits of meta-learning in solving multiple MARL tasks collectively. We establish the first line of theoretical results for meta-learning in a wide range of fundamental MARL settings, including learning Nash equilibria in two-player zero-sum Markov games and Markov potential games, as well as learning coarse correlated equilibria in general-sum Markov games. Under natural notions of task similarity, we show that meta-learning achieves provable sharper convergence to various game-theoretical solution concepts than learning each task separately. As an important intermediate step, we develop multiple MARL algorithms with initialization-dependent convergence guarantees. Such algorithms integrate optimistic policy mirror descents with stage-based value updates, and their refined convergence guarantees (nearly) recover the best known results even when a good initialization is unknown. To our best knowledge, such results are also new and might be of independent interest. We further provide numerical simulations to corroborate our theoretical findings.
Many real-world sequential decision-making problems involve multiple agents interacting in a shared environment, a scenario commonly captured by game theory and addressed using multi-agent reinforcement learning (MARL). Existing research in MARL has primarily focused on solving a single task (i.e., a game) independently. In practice, however, one often needs to collectively solve a set of similar tasks due to the dynamically evolving environment. For example, in sponsored search auctions [48], the advertising spaces and search results are dynamic, and each bidder with an active bid will participate in a sequence of related auctions. In multi-robot cooperation [31, 26], the learning agents are often first pre-trained in simplified environments and are then asked to quickly adapt to more complicated ones. In cloud computing [53, 70], a learning-based autoscaling policy needs to achieve fast model adaptation to deal with varied application workloads or constantly evolving cloud infrastructures. All of these intriguing applications call for the development of intelligent multi-agent systems that can continuously build on previous experiences to enhance the learning of new tasks.
Meta-learning, or learning-to-learn [64, 56, 65, 58], is a rapidly developing approach that is particularly suitable for learning in a set of related tasks. In essence, meta-learning studies the use of data from existing tasks to learn representations or model parameters that enable quick adaptation to new tasks. By exploiting the knowledge obtained from prior tasks, the meta-learner can ideally solve an unseen task using much fewer training samples than learning from scratch, especially when the tasks share some inherent similarities. Despite many empirical successes [71, 31, 26], the theoretical results of meta-learning in multi-agent scenarios are still relatively lacking. It remains elusive whether meta-learning can provably expedite the convergence of MARL, and if so, what the proper task similarity assumptions to impose are. In fact, it is even unclear whether a meta-learner converges at all in a highly non-stationary system with loosely-coupled learning agents and diverse task setups.
In this paper, we make an initial attempt toward characterizing some of the central theoretical properties of meta-learning in a wide range of fundamental MARL settings. We focus on the classic model-agnostic meta-learning (MAML) [20] type of algorithms that aim to learn a good initialization for quick adaptation to new tasks. To study the convergence rate of MAML, an important prerequisite is to understand how the convergence of MARL algorithms depends on the quality of policy initialization. However, the convergence guarantees of most existing MARL algorithms are initialization-independent: They fail to track how the sub-optimality of the initial policy propagates during the learning process, and only provide pessimistic guarantees with respect to worst-case initialization. As a crucial intermediate step to meta-MARL, we need to establish refined initialization-dependent convergence guarantees for MARL. Our main contributions are thus summarized as follows.
Contributions. 1) For learning Nash equilibria (NE) in two-player zero-sum Markov games, we first propose an MARL algorithm blessed with a refined convergence analysis that explicitly characterizes the dependence on policy initialization (Section 3.1). Our algorithm runs optimistic online mirror descent for policy optimization and performs stage-based value function updates. Even when initialized with random policies, our algorithm still matches the best-known convergence rates in the literature except for an extra logarithmic term. Our algorithm and analysis appear to be new and might be of independent interest. 2) Based on such refined analysis, we show that meta-learning provably achieves faster convergence to NE when learning a sequence of "similar" zero-sum games collectively, where our similarity metric naturally depends on the closeness of the games' NE policies (Section 3.2). 3) For learning NE in Markov potential games (MPGs), we show that a simple refinement of an existing algorithm suffices to provide initialization-dependent guarantees. We establish sharper convergence rates of meta-learning when the MPGs have similar potential functions (Section 4.1). In addition, with a properly chosen policy update rule, we prove the non-asymptotic convergence of the exact MAML algorithm in MPGs (Section 4.2), despite the convoluted learning dynamics of multiple loosely-coupled agents. 4) For learning coarse correlated equilibria (CCE) in general-sum Markov games (Section 5), we analogously start by designing an initialization-dependent MARL algorithm, and then establish the sharper convergence rate of meta-learning under natural similarity metrics. 5) We provide numerical results to corroborate our theoretical findings (Section 6).
Related Work. Gradient-based meta-learning is a simple and effective approach that can be easily applied to any learning problem trained with gradient descent. The seminal MAML method [20] tries to learn a good model parameter initialization that leads to quick model adaptation. Theoretical properties of MAML have been investigated in a series of works [54, 17, 66, 18, 30]. In particular, [17, 30] have established the convergence of MAML to first-order stationarity for non-convex objectives. [18] has designed an unbiased gradient estimator for MAML in reinforcement learning tasks. Various first-order approximations [20, 49, 17] of MAML have been proposed to avoid the heavy computation of the Hessian. Meta-learning has also been studied in online convex optimization [21, 4, 14, 35], where regret bounds have been established under different metrics of task similarity. Another line of research [29, 51, 41] views meta-learning through the lens of task inference, where an RL policy is conditioned on a belief over tasks and perform Bayesian updates through interactions to adapt to different tasks.
MARL has been widely studied under the formulation of stochastic games (i.e., Markov games) [57]. Due to the fundamental difficulty of computing NE in generic games [10], most MARL research has focused on learning NE in games with special structures (such as zero-sum Markov games [67, 2, 69, 3, 12, 68, 7, 72, 73] and Markov potential games [43, 38, 76, 25, 22, 15, 78]) or learning weaker solution concepts such as (coarse) correlated equilibria [40, 44, 59, 33, 47, 16, 13]. The most relevant works are [77, 72], which have studied the convergence of optimistic no-regret
learning and smooth value updates in MARL with full-information feedback. For learning NE in MPGs, [38, 76, 15] have studied independent policy gradient methods and established their sample complexity results. These works have focused on learning a single game in isolation but have not considered exploiting the connections between multiple games to expedite the learning process.
Most related to ours, [27] has studied meta-learning in normal-form games. Under different notions of game similarities, [27] has shown faster convergences of meta-learning in zero-sum, general-sum, and Stackelberg games. [75] has investigated no-regret learning in time-varying zero-sum normal-form games. Compared to [27, 75], we consider meta-learning in the more generic and challenging Markov game setup with state transitions. Other related works include meta-learning for regret minimization in a distribution of games [61] and meta-safe RL for quick adaptation in constrained Markov decision processes (CMDPs) under task similarity [34]. Finally, meta-learning has also been empirically applied to many important MARL scenarios, including multi-intersection traffic signal control [71], multi-agent communication with natural language [26], and multi-agent collaboration with first-person pixel observations in open-ended tasks [63].
Markov game. An  $N$ -player episodic Markov game is defined by a tuple  $\mathbb{G} = (\mathcal{N}, H, S, \{\mathcal{A}_i\}_{i=1}^N, \{r_i\}_{i=1}^N, P)$ , where (1)  $\mathcal{N} = \{1, 2, \dots, N\}$  is the set of agents; (2)  $H \in \mathbb{N}_+$  is the number of time steps in each episode; (3)  $\mathcal{S}$  is the finite state space; (4)  $\mathcal{A}_i$  is the finite action space for agent  $i \in \mathcal{N}$ ; (5)  $r_i: [H] \times S \times \mathcal{A}_{\mathrm{all}} \to [0,1]$  is the reward function for agent  $i$ , where  $\mathcal{A}_{\mathrm{all}} = \times_{i=1}^{N} \mathcal{A}_i$  is the joint action space; and (6)  $P: [H] \times S \times \mathcal{A}_{\mathrm{all}} \to \Delta(S)$  is the transition kernel. The agents interact in an unknown environment for  $T$  episodes. Without loss of generality, we make a standard assumption [33, 59] that each episode starts from a fixed initial state  $s_1$ . Our results can be easily generalized to the setting where the initial state is sampled from a fixed distribution. At each time step  $h \in [H]$ , the agents observe the state  $s_h \in S$ , and take actions  $a_h, i \in \mathcal{A}_i, i \in \mathcal{N}$  simultaneously. Agent  $i$  then receives its reward  $r_{h,i}(s_h, a_h)$ , where  $a_h = (a_{h,1}, \dots, a_{h,N})$ , and the environment transitions to the next state  $s_{h+1} \sim P_h(\cdot | s_h, a_h)$ . Let  $S = |\mathcal{S}|, A_i = |\mathcal{A}_i|, \forall i \in \mathcal{N}$ , and  $A_{\max} = \max_{i \in \mathcal{N}} A_i$ .
Policy and Nash equilibrium. A (Markov) policy  $\pi_i \in \Pi_i : [H] \times S \to \Delta(\mathcal{A}_i)$  for agent  $i \in \mathcal{N}$  is a mapping from the time index and state space to a distribution over its own action space. Each agent seeks to find a policy that maximizes its own cumulative reward. A joint, product policy  $\pi = (\pi_1, \dots, \pi_N) \in \Pi$  induces a probability measure over the sequence of states and joint actions. We use the subscript  $-i$  to denote the set of agents excluding agent  $i$ , i.e.,  $\mathcal{N} \setminus \{i\}$ . We can rewrite  $\pi = (\pi_i, \pi_{-i})$  using this convention. For a joint policy  $\pi$ , and for any  $h \in [H]$ ,  $s \in S$ , and  $a \in \mathcal{A}_{\mathrm{all}}$ , we define the value function and Q-function for agent  $i$  as
$$
V _ {h, i} ^ {\pi} (s) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}, i} \left(s _ {h ^ {\prime}}, \boldsymbol {a} _ {h ^ {\prime}}\right) \mid s _ {h} = s \right], Q _ {h, i} ^ {\pi} (s, \boldsymbol {a}) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}, i} \left(s _ {h ^ {\prime}}, \boldsymbol {a} _ {h ^ {\prime}}\right) \mid s _ {h} = s, \boldsymbol {a} _ {h} = \boldsymbol {a} \right].
$$
For agent  $i$ , a policy  $\pi_i^\dagger$  is a best response to  $\pi_{-i}$  if  $V_{1,i}^{\pi_i^\dagger, \pi_{-i}}(s_1) = \sup_{\pi_i} V_{1,i}^{\pi_i, \pi_{-i}}(s_1)$ . A joint (product) policy  $\pi = (\pi_i, \pi_{-i}) \in \Pi$  is a Nash equilibrium (NE) if  $\pi_i$  is a best response to  $\pi_{-i}$  for all  $i \in \mathcal{N}$ . Similarly, for any  $\varepsilon > 0$ , a joint policy  $\pi = (\pi_i, \pi_{-i})$  is an  $\varepsilon$ -approximate NE if  $V_{1,i}^{\pi_i, \pi_{-i}}(s_1) \geq V_{1,i}^{\pi_i^\dagger, \pi_{-i}}(s_1) - \varepsilon, \forall i \in \mathcal{N}$ .
Correlated policy and coarse correlated equilibrium. We define  $\pi = \{\pi_h : \mathbb{R} \times (\mathcal{S} \times \mathcal{A})^{h-1} \times \mathcal{S} \to \Delta(\mathcal{A})\}_{h \in [H]}$  as a (non-Markov) correlated policy, where for each  $h \in [H]$ ,  $\pi_h$  maps from a coordination device  $z \in \mathbb{R}$  and a history of length  $h-1$  to a distribution over the joint action space. Let  $\pi_i$  and  $\pi_{-i}$  be the proper marginal distributions of  $\pi$  whose outputs are restricted to  $\Delta(\mathcal{A}_i)$  and  $\Delta(\mathcal{A}_{-i})$ , respectively. The value functions for non-Markov correlated policies at step  $h=1$  are defined in a similar way as for product policies. Given the PPAD-hardness of calculating NE in general [11], people often study a relaxed solution concept named coarse correlated equilibrium (CCE), which allows possible correlations in the policies: In particular, for any  $\varepsilon > 0$ , a correlated policy  $\pi = (\pi_i, \pi_{-i})$  is an  $\varepsilon$ -approximate CCE if  $V_{1,i}^{\pi_i, \pi_{-i}}(s_1) \geq V_{1,i}^{\pi_i^\dagger, \pi_{-i}}(s_1) - \varepsilon, \forall i \in \mathcal{N}$ .
Two-player zero-sum Markov game. An important special case of Markov games is (two-player) zero-sum Markov games, where there are two players ( $N = 2$ ) with exactly opposite rewards
$(r_{1} = -r_{2})$ . In a zero-sum game, we simply use  $r, V$ , and  $Q$  to denote the reward and (Q-)value functions for the max-player, i.e., agent 1. Correspondingly, the min-player has  $-r, -V$ , and  $-Q$ . For notational convenience, we denote the action space for the max-player (resp. min-player) by  $\mathcal{A}$  (resp.  $\mathcal{B}$ ), and let  $A = |\mathcal{A}|$ ,  $B = |\mathcal{B}|$ . We also write their policies  $(\pi_{1}, \pi_{2})$  as  $(\mu, \nu)$  for short. In zero-sum games, it is known that although the NE policy  $(\mu^{\star}, \nu^{\star})$  may not be unique, all the NE have the same values. We use  $V_{h}^{\star}$  and  $Q_{h}^{\star}$  to denote the NE value function and the NE Q-function. For any fixed  $(h, s) \in [H] \times S$  and an arbitrary function  $Q: S \times \mathcal{A} \times \mathcal{B} \to \mathbb{R}$ , we may consider  $Q(s, \cdot, \cdot)$  as an  $A \times B$  matrix. Then, for any policy pair  $(\mu_{h}, \nu_{h})$  at step  $h \in [H]$ , we can write in shorthand:
$$
\left[ \mu_ {h} ^ {\top} Q \nu_ {h} \right] (s) := \mathbb {E} _ {a \sim \mu_ {h} (\cdot | s), b \sim \nu_ {h} (\cdot | s)} [ Q (s, a, b) ] = \langle \mu_ {h}, Q \nu_ {h} \rangle (s),
$$
$$
\left[ \mu_ {h} ^ {\top} Q \right] (s, \cdot) := \mathbb {E} _ {a \sim \mu_ {h} (\cdot | s)} [ Q (s, a, \cdot) ], \text {a n d} \left[ Q \nu_ {h} \right] (s, \cdot) := \mathbb {E} _ {b \sim \nu_ {h} (\cdot | s)} [ Q (s, \cdot , b) ].
$$
Given the transition function  $P$  and an arbitrary function  $V: \mathcal{S} \to \mathbb{R}$ , we define
$$
\left[ P _ {h} V \right] (s, a, b) := \mathbb {E} _ {s ^ {\prime} \sim P _ {h} (\cdot | s, a, b)} \left[ V \left(s ^ {\prime}\right) \right].
$$
The Bellman equations can hence be rewritten more succinctly as
$$
V _ {h} ^ {\mu , \nu} (s) = \left[ \mu_ {h} ^ {\top} Q _ {h} ^ {\mu , \nu} \nu_ {h} \right] (s), \text {a n d} Q _ {h} ^ {\mu , \nu} (s, a, b) = r _ {h} (s, a, b) + \left[ P _ {h} V _ {h + 1} ^ {\mu , \nu} \right] (s, a, b).
$$
Markov potential game. Another important class of games is Markov potential games [43, 39, 76]. MPGs cover Markov teams [36], a fully cooperative setting where all agents share the same rewards. A Markov game is an MPG if there exists a global potential function  $\Phi : \Pi \times S \to [0, \Phi_{\max}]$  that can capture the variations of the agents' individual values: Specifically,  $\forall i \in \mathcal{N}$  and  $s \in S$ ,
$$
\Phi_ {s} \left(\pi_ {i}, \pi_ {- i}\right) - \Phi_ {s} \left(\pi_ {i} ^ {\prime}, \pi_ {- i}\right) = V _ {1, i} ^ {\pi_ {i}, \pi_ {- i}} (s) - V _ {1, i} ^ {\pi_ {i} ^ {\prime}, \pi_ {- i}} (s), \forall \pi_ {i}, \pi_ {i} ^ {\prime} \in \Pi_ {i}, \pi_ {- i} \in \Pi_ {- i}.
$$
Throughout the paper, we consider the classic full-information feedback setting [23, 8, 62, 68, 7], where the players are assumed to have exact information of the consequences of each of their candidate actions. In the case of zero-sum games, this implies that for any  $(h,s)$ , the max-player and min-player can query  $[Q_h\nu_h](s,\cdot)$  and  $[\mu_h^\top Q_h](s,\cdot)$ , respectively. Our meta-learning results can be easily extended to the stochastic bandit feedback setting using standard techniques as in [3, 44, 33, 15].
Meta-learning. Let  $\mathcal{G} = \{\mathbb{G}^k\}$  be a set of different Markov games. Each game is defined by  $\mathbb{G}^k = (\mathcal{N}, H, S, \{\mathcal{A}_i\}_{i=1}^N, \{r_i^k\}_{i=1}^N, P^k)$ , where we assume without loss of generality that the games share the same agent set and state & action spaces, but can have different transition and reward functions. Most of our results are established in the online learning setting where we encounter a sequence of  $K$  games  $(\mathbb{G}^1, \ldots, \mathbb{G}^K)$  one by one. To achieve faster convergence, the learning agents should use the knowledge obtained from previous games to expedite the learning process in future games.
The underlying principle of MAML [20] is to learn a good initialization such that running a few training steps from this initialization can lead to well-performing model parameters on any new task. An MAML-type algorithm in the context of RL typically involves two nested stages. The inner stage (or "base algorithm")  $\psi$  performs  $T$  iterations of policy updates to optimize for an individual task  $\mathbb{G}^k$ :
$$
\pi^ {k, t} \leftarrow \psi \left(\pi^ {k, t - 1}; \mathbb {G} ^ {k}\right), \forall t \in [ T ]. \tag {1}
$$
When task  $\mathbb{G}^k$  is completed, the outer stage (or "meta-algorithm")  $\Psi$  learns to form a good initialization  $\pi^{k + 1,0}$  for a new task  $\mathbb{G}^{k + 1}$  using all the knowledge obtained from all previous tasks:
$$
\pi^ {k + 1, 0} \leftarrow \Psi \left(\left\{\pi^ {k ^ {\prime}, t} \right\} _ {k ^ {\prime} \in [ k ], t \in [ T ]}; \mathbb {G} ^ {1}, \dots , \mathbb {G} ^ {k}\right). \tag {2}
$$
In this paper, we seek to properly instantiate both the base algorithm  $\psi$  and the meta-algorithm  $\Psi$  for a variety of MARL problems. We aim to show that a proper design of the meta-learning procedure  $(\psi, \Psi)$  can largely reduce the number of iterations  $T$  required to find NE or CCE in a new game.
3 Meta-Learning for Two-Player Zero-Sum Markov Games
In this section, we study meta-learning for Nash equilibria in zero-sum Markov games, where players are fully competitive. Since MAML-type algorithms seek to learn a good initialization for quick adaptation, it is crucial to explicitly characterize how the convergence behavior of an MARL algorithm depends on the initial policy. To our best knowledge, such results are not directly achievable using existing algorithms. For this reason, in Section 3.1, we start by proposing a new base algorithm (1) for zero-sum Markov games that has a refined initialization-dependent convergence guarantee. Based on that, we present our meta-algorithm (2) in Section 3.2 and establish its sharper convergence rates.
3.1 Initialization-Dependent Convergence in an Individual Zero-Sum Markov Game
Algorithm 1 presents our optimistic online mirror descent algorithm with stage-based value updates for learning NE in a zero-sum Markov game. To establish initialization-dependent convergence, Algorithm 1 performs optimistic online mirror descent (OMD) [55, 62] for policy updates (Lines 5 and 6), in contrast to the popular optimistic follow the regularized leader (FTRL) method in recent MARL policy optimization [77, 72]. We choose the negative entropy as our regularizer  $R$ , in which case the Bregman divergence  $D_R(\cdot, \cdot)$  reduces to the Kullback-Leibler divergence and optimistic OMD becomes an optimistic variant of the classic multiplicative weights update (MWU) algorithm.
Algorithm 1: Optimistic Online Mirror Descent for Zero-Sum Markov Games
1 Input: Initial policies  $\tilde{\mu} : [\bar{\tau}] \times [H] \times \mathcal{S} \to \Delta(\mathcal{A})$  and  $\tilde{\nu} : [\bar{\tau}] \times [H] \times \mathcal{S} \to \Delta(\mathcal{B})$ ;
2 Set stage index  $\tau \gets 1$ ,  $t_{\tau}^{\mathrm{start}} \gets 1$ , and  $L_{\tau} \gets H$ ;
3 Initialize:  $\mu_h^0 = \hat{\mu}_h^0\gets \tilde{\mu}_h^1,\nu_h^0 = \hat{\nu}_h^0\gets \tilde{\nu}_h^1$  , and  $Q_{h}^{\tau}\gets 0,\forall h\in [H]$
4 for iteration  $t \gets 1$  to  $T$  do
5 Auxiliary policy update: for each step  $h\in [H]$  and state  $s\in S$  ..
$$
\hat {\mu} _ {h} ^ {t} (\cdot | s) \leftarrow \operatorname * {a r g m a x} _ {\hat {\mu} \in \Delta (A)} \eta \left\langle \hat {\mu}, [ Q _ {h} ^ {\tau} \nu_ {h} ^ {t - 1} ] (s, \cdot) \right\rangle - D _ {R} (\hat {\mu}, \hat {\mu} _ {h} ^ {t - 1} (\cdot | s));
$$
$$
\hat{\nu}_{h}^{t}(\cdot |s)\leftarrow \operatorname *{argmax}_{\hat{\nu}\in \Delta (\mathcal{B})}\eta \left\langle \hat{\nu},[(\mu_{h}^{t - 1})^{\top}Q_{h}^{\tau}](s,\cdot)\right\rangle -D_{R}(\hat{\nu},\hat{\nu}_{h}^{t - 1}(\cdot |s));
$$
Policy update: for each step  $h \in [H]$  and state  $s \in S$ :
$$
\mu_ {h} ^ {t} (\cdot | s) \leftarrow \operatorname * {a r g m a x} _ {\mu \in \Delta (\mathcal {A})} \eta \left\langle \mu , \left[ Q _ {h} ^ {\tau} \nu_ {h} ^ {t - 1} \right] (s, \cdot) \right\rangle - D _ {R} (\mu , \hat {\mu} _ {h} ^ {t} (\cdot | s));
$$
$$
\nu_{h}^{t}( \cdot |s)\gets \operatorname *{argmax}_{\nu \in \Delta (\mathcal{B})}\eta \left\langle \nu ,[(\mu_{h}^{t - 1})^{\top}Q_{h}^{\tau}](s,\cdot)\right\rangle -D_{R}(\nu ,\hat{\nu}_{h}^{t}( \cdot |s));
$$
7 if  $t - t_{\tau}^{start} + 1 \geq L_{\tau}$  then
8  $t_{\tau}^{\mathrm{end}} \gets t, t_{\tau + 1}^{\mathrm{start}} \gets t + 1, L_{\tau + 1} \leftarrow \lfloor (1 + 1 / H)L_{\tau}\rfloor;$
9 Value update: for each  $h\in [H],s\in \mathcal{S},a\in \mathcal{A},b\in \mathcal{B}$
$$
Q _ {h} ^ {\tau + 1} (s, a, b) \leftarrow \frac {1}{L _ {\tau}} \sum_ {t ^ {\prime} = t _ {\tau} ^ {\mathrm {s t a r t}}} ^ {t _ {\tau} ^ {\mathrm {e n d}}} \left(r _ {h} + P _ {h} \left[ \left(\mu_ {h + 1} ^ {t ^ {\prime}}\right) ^ {\top} Q _ {h + 1} ^ {\tau} \left(\nu_ {h + 1} ^ {t ^ {\prime}}\right) \right]\right) (s, a, b);
$$
10  $\tau \gets \tau + 1$ ;  $\mu_h^t = \hat{\mu}_h^t$ $\tilde{\mu}_h^\tau, \nu_h^t = \hat{\nu}_h^t$ $\forall h \in [H]$ ;
11 Output policy:  $\bar{\mu}_h(\cdot |s) = \frac{1}{T}\sum_{t = 1}^{T}\mu_h^t (\cdot |s)$  and  $\bar{\nu}_{h}(\cdot |s) = \frac{1}{T}\sum_{t = 1}^{T}\nu_{h}^{t}(\cdot |s),\forall s\in S,h\in [H].$