SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We prove in Theorem 4.8 that CSP approximates the optimal risk-free offline adaptation policy by some constant upon convergence. The gap depends on the maximum KL-divergence between the learned proxy  $\mu$  and the target  $\pi_B$  on  $s\in D$ . The gap decreases as  $\mu$  becomes a more precise estimation of  $\pi_B$  within  $D$ . As a sanity check, the gap vanishes when  $\mu$  perfectly matches  $\pi_B$  in  $D$ .
Theorem 4.8. Let  $\pi^{*}$  be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective 2, and let  $\tilde{\pi}$  be the policy at the convergence of objective 5. Then the worst-case adaptation performance of  $\tilde{\pi}$  is near-optimal:
$$
\min  _ {\mu \in \mathcal {C} _ {D}} J \left(\pi^ {*}, \mu\right) \geq \min  _ {\mu \in \mathcal {C} _ {D}} J (\tilde {\pi}, \mu) \geq \min  _ {\mu \in \mathcal {C} _ {D}} J \left(\pi^ {*}, \mu\right) - R _ {M} \sqrt {2 \delta} \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {6}
$$
5 Didactic Maze Example
In this section, we use an adversarial grid-world maze game to demonstrate the challenges posed by offline policy adaptation and explain the superiority of our algorithm. Figure 2 displays the maze environment, where Player 1 (P1) and Player 2 (P2) start from the labeled positions and can move in four directions within the maze. The grids marked with numbers are terminal states with indicated rewards. There are five corridors (2 for P1 and 3 for P2), each with different lengths and exponential final rewards $^2$ . For simplicity, we use the notation  $\mathrm{P}2 \to 16$  to denote P2's policy which going downwards to the state with terminal reward 16. The first agent reaching any terminal state wins the reward, and the opponent receives the negative reward as a penalty. The game ends immediately. We assume that P1 wins if P1 and P2 arrive at terminal states simultaneously. If no one reaches a terminals in 10 steps, they both get a penalty of -500. Both agents have full observation of the game states. Assume that P1 is the opponent, and we want to find exploiting policies for P2.
We consider four baselines: (1) BC-First as explained in section 1; (2) Self-Play (max-min): approximating the pure max-min policy of P2 using self-play, which does not exploit; (3) Dataset-Only: completely following the dataset; and (4) Oracle: training an exploitation policy directly against the real target  $\pi_B$ . Please refer to Appendix D for detailed learning curves in this environment.
Figure 2: The didactic maze example. The solid line indicates the testing trajectory of adaptation policy produced by CSP, while the dashed line indicates sub-optimal adaptation trajectories: (left) BC-First in maze case 1 and (right) Self-Play in maze case 2.
Case 1: Inadequacy of Non-Conservative Methods We show in this case that non-conservative algorithms easily overfit to an erroneous proxy model, lured to exploit "fake weaknesses" of the
2Rewards are set exponentially for efficient exploration of practical deep reinforcement learning algorithms.
Table 1: The training and test performances in maze environment (left) case 1 and (right) case 2.
CASE 1	Train	Test (against πB)	CASE 2	Train	Test (against πB)
BC-First	16	-4	BC-First	16	16
Self-Play	1	1	Self-Play	1	1
Dataset-Only	-64	-64	Dataset-Only	16	16
CSP (Ours)	1	1	CSP (Ours)	16	16
Oracle	1	1	Oracle	16	16
opponent. Suppose that P1's real policy  $\pi_B$  is as follows: (1) if P1 observes  $P2 \to 16$ ,  $P1 \to 4$ ; (2) if P1 observes  $P2 \to 256$ ,  $P1 \to 64$ ; (3) otherwise P1 does not move. This is a reasonably strong policy for P1 which tries to exploit P2. Suppose the dataset is collected with  $\pi_B$  and P2's rollout policy is  $P2 \to 256$ . Therefore, in the dataset, P1 is only observed going downwards. The proxy model trained by BC thus erroneously assumes that  $P1 \to 64$  under any conditions. The results are shown on the left side of Table 1. The BC-First method makes a disastrous generalization and responds with a  $P2 \to 16$  policy. Trying to exploit the "fake weaknesses" of the opponent results in failure in testing against the real opponent. However, CSP discovers that  $P2 \to 16$  is unsafe and learns to safely win the game by playing  $P2 \to 1$ . Both CSP and max-min policy achieve the optimal solution. The Dataset-Only method is overly conservative and only gets -64. The trajectories of CSP and BC-First when exploiting the real target are shown on the left side of figure 2.
Case 2: Performing Risk-Free Adaptation In this case, we show that CSP outperforms pure Self-Play (which does not exploit the opponent) by discovering risk-free exploitation opportunities. Suppose that P1 always plays an aggressive policy  $\pi_B$ :  $P1 \to 64$ . The dataset  $D$  is collected by  $\pi_B$  with P2 playing  $P2 \to 16$  and  $P2 \to 256$  randomly. The results are shown on the right side of Table 1, and the trajectories of CSP and Self-Play are shown on the right side of Figure 2. The max-min policy produced by Self-Play cannot exploit P1's reckless policy. The generalization of the BC model is indeed correct in this specific situation, so BC-First is also optimal. Our algorithm recognizes that  $P2 \to 16$  is a risk-free exploitation because the dataset has witnessed that P1 still plays  $P1 \to 64$  even if  $P2 \to 16$  and successfully exploits the opponent.
It is noteworthy that we use the same hyperparameters for CSP in both cases and our algorithm learns different adaptation policies based solely on the different datasets used. It further validates that our algorithm is able to extract useful information from the dataset, and to perform risk-free adaptations.
We conduct extensive experiments on the predator-prey in MPE [39], a competitive two-player MuJoCo environment [2, 10] that requires continuous control, and Google Football [21]. In all experiments, we compare CSP with (1) BC-First, and (2) Self-Play, which produces an approximate max-min conservative policy of the game. Self-Play can also be viewed as an ablation of CSP which does not utilize an offline dataset to adapt to targets. We also compare with multi-agent imitation learning (MAIL), and previous opponent exploitation method RNR [17], which does not perform conservative offline exploitation. More experiment details are available in Appendix G. We focus on performing opponent exploitation in competitive games. However, we also discuss the use of CSP for cooperative tasks in Appendix E.
6.1 Experiments in Predator-Prey and MuJoCo
In the predator-prey environment, there are three adversarial agents, one good agent, and two obstacles. The good agent is penalized for being hit by adversaries, whereas the adversaries are rewarded. In experiments presented in Table 2, our agent controls either the good agent or the adversaries, respectively. We use a pre-trained model as target  $\pi_B$ . We report the episode rewards in evaluation with  $\pi_B$ . We use three types of datasets (10R, 100R, 100S) to demonstrate the impact of dataset quality in offline adaptation. The prefix number indicates the number of trajectories in the dataset. Letter "R" indicates that the dataset is collected by  $\pi_B$  playing with a random policy, while "S" indicates that the dataset is collected by  $\pi_B$  playing with another well-trained opponent.
Table 2: Testing episode rewards of adaptation policy in predator-prey of MPE.
Controls Good Agent	Controls Adversaries
Dataset	10R	100R	100S	10R	100R	100S
CSP(Ours)	-16.7±0.4	-17.5±0.4	-17.1±0.1	34.6±1.6	33.6±3.2	34.7±4.1
BC-First	-29.8±4.0	-21.8±1.6	-18.5±0.4	21.3±0.8	25.2±0.9	27.8±1.9
Self-Play	-17.6±1.5	-17.6±1.5	-17.6±1.5	29.7±2.8	29.7±2.8	29.7±2.8
MAIL	-59.0±0.6	-54.2±1.0	-37.9±1.4	4.3±0.3	4.1±0.2	25.8±0.7
Oracle	-16.0±0.3	-16.0±0.3	-16.0±0.3	34.3±6.2	34.3±6.2	34.3±6.2
Table 3: Experiments in MuJoCo YouShallNotPassHumans environment. We show the winning rates and episode rewards against 4 independent target opponents..
Method	1	2	3	4
Winning Rate	CSP(Ours)	0.72 ± 0.06	0.65 ± 0.04	0.85 ± 0.08	0.88 ± 0.02
BC-First	0.65 ± 0.13	0.46 ± 0.14	0.71 ± 0.17	0.72 ± 0.07
Self-Play	0.54 ± 0.20	0.59 ± 0.14	0.52 ± 0.02	0.60 ± 0.19
Episode Reward	CSP(Ours)	878 ± 166	676 ± 100	1203 ± 209	1274 ± 73
BC-First	693 ± 347	179 ± 456	848 ± 373	858 ± 185
Self-Play	419 ± 527	500 ± 395	318 ± 106	601 ± 514
Intuitively, 10R has the lowest dataset quality, while 100S has the highest. We train the Oracle directly against  $\pi_B$  as an upper bound. Self-Play and Oracle do not use datasets. MAIL imitates the dataset collection policy. It fails to learn meaningful behavior on datasets 10R and 100R where the dataset collection policy is a random policy. Table 2 shows that CSP outperforms baselines and achieves comparable performances with Oracle, regardless of the dataset quality. CSP effectively learns to exploit the opponent safely, while the non-conservative method BC-First fails to do so. Moreover, BC-First is significantly more sensitive to dataset quality. In the controlling adversaries scenario, even with the 100S dataset, BC-First still performs worse than Self-Play, which does not exploit at all. It shows that the non-conservative method overfits the proxy model severely.
The YouShallNotPassHumanoids environment [2, 10] is a MuJoCo-based two-player competitive game where the runner's objective is to pass the blocker and reach the red line on the opposite side. In our experiments, our agent acts as the runner, and the blocker is the target opponent for exploitation. In Table 3, we use four independently pre-trained blocker models as opponent targets. Since the exploitability of different opponent models can vary significantly, results against different opponents are not directly comparable. So we report results for all targets. CSP successfully exploits the opponent, and significantly outperforms the baselines for all targets. Additionally, we include a visualization of CSP's learned behavior in Appendix G.
6.2 Experiments in Google Football
Google Football [21] is a popular and challenging benchmark for MARL. In our experiments, the adaptation policy controls all players in one team, while the opponent target policy controls the other team. We conduct experiments in 4 scenarios: academy_3.vs_1_with_keeper (3vs1, where our agent acts as either the defender or the attacker), academy_run_pass_and_shoot_with_keeper (RPS, defender), and academy counterattack_easy (defender). We report the winning rates of adaptation policies for five independently pre-trained opponent targets.
The experiment results in Table 4 show that our algorithm CSP outperforms baselines for almost all targets in all scenarios. Notably, the BC-First algorithm even
Figure 3: The average test-train performance gap in four scenarios of Google Football. A negative gap indicates the occurrence of unsafe exploitation.
Table 4: Winning rates of offline adaptation policy in 4 scenarios of Google Football: 3vs1 (defender), 3vs1 (attacker), RPS (defender), and Counterattack Easy (defender). For each scenario, we experiment with 5 independent target opponents.
Scenario	Method	1	2	3	4	5
3vs1 defender	CSP(Ours)	0.9 ± 0.06	0.6 ± 0.12	0.45 ± 0.04	0.32 ± 0.11	0.76 ± 0.16
BC-First	0.64 ± 0.04	0.46 ± 0.09	0.2 ± 0.01	0.16 ± 0.04	0.56 ± 0.09
Self-Play	0.29 ± 0.09	0.34 ± 0.1	0.26 ± 0.16	0.29 ± 0.18	0.3 ± 0.13
3vs1 attacker	CSP(Ours)	0.81 ± 0.14	0.88 ± 0.02	0.83 ± 0.07	0.84 ± 0.05	0.78 ± 0.08
BC-First	0.83 ± 0.03	0.74 ± 0.21	0.68 ± 0.07	0.79 ± 0.11	0.71 ± 0.15
Self-Play	0.75 ± 0.15	0.76 ± 0.08	0.73 ± 0.13	0.7 ± 0.21	0.74 ± 0.08
RPS defender	CSP(Ours)	0.51 ± 0.33	0.71 ± 0.07	0.56 ± 0.13	0.38 ± 0.07	0.79 ± 0.09
BC-First	0.51 ± 0.24	0.41 ± 0.07	0.34 ± 0.18	0.38 ± 0.04	0.71 ± 0.02
Self-Play	0.57 ± 0.14	0.36 ± 0.04	0.25 ± 0.04	0.37 ± 0.1	0.76 ± 0.03
Counter-attack defender	CSP(Ours)	0.93 ± 0.02	0.88 ± 0.04	0.81 ± 0.25	0.81 ± 0.02	0.75 ± 0.06
BC-First	0.7 ± 0.17	0.52 ± 0.07	0.53 ± 0.09	0.69 ± 0.14	0.5 ± 0.09
Self-Play	0.55 ± 0.13	0.41 ± 0.1	0.46 ± 0.09	0.36 ± 0.08	0.36 ± 0.07
performs worse than Self-Play, which does not utilize the opponent's information to exploit. It indicates that previous non-conservative algorithms are not guaranteed to use the opponent's information safely in offline adaptation.
In Figure 3, we show the gap of performances between testing and training:  $J(\pi, \pi_B) - J(\pi, \mu)$ . A negative gap indicates that although  $\pi$  learns to exploit  $\mu$  during training, such exploitation is risky and can not be guaranteed in testing. The results demonstrate the significant challenges that offline policy adaptation poses for baseline exploitation algorithms. BC-First overfits the proxy model in training and experiences a severe performance drop in testing. Conversely, CSP has a positive gap on average, indicating its ability to find conservative exploitation opportunities. Because CSP optimizes for the worst-case performance, it tends to produce a proxy model that is possibly stronger than the real target. Thus, its evaluation performances are even higher than those in training.
6.3 Comparison with Non-Conservative Opponent Exploitation
We compare CSP with the previous safe exploitation method RNR [17], which minimizes the distance to NE when exploiting. RNR assumes that the opponent plays an NE policy with probability  $1 - p$  and follows an estimated opponent model with  $p$ . It then learns an exploitation policy. However, RNR does not optimize the conservative offline adaptation objective, and it still fails to handle offline policy adaptation due to the neglect of possible errors in the estimated opponent model. In Figure 4, the results show that CSP outperforms RNR for every dataset and every value of  $p$ . RNR(0) is equivalent to Self-Play, and RNR(1) is equivalent to BC-First. Results for these two have already been reported in Table 2. The results further support our claim that previous methods struggle with offline policy adaptation and highlight the efficiency and significance of our algorithm.
Figure 4: Comparison with RNR [17] in predator-prey, where our agent controls (left) adversaries and (right) good agent respectively. The y-axis represents the episode reward in testing.
We provide a thorough analysis of the unique challenges in offline policy adaptation, which are often neglected by previous literature. We propose the novel learning objective of conservative offline adaptation that optimizes the worst-case performance against any dataset-consistent opponent proxy. We propose an efficient algorithm that learns near-optimal conservative adaptation policies. Extensive empirical results in Maze, predator-prey, MuJoCo, and Google Football demonstrate that our algorithm significantly outperforms baselines.
One limitation of our work is that optimizing worst-case performance in real-world applications may not always be necessary since participants are not fully rational in practice. One future direction to settle this problem is to extend offline adaptation to non-stationary opponents, which is orthogonal to our current study. Those methods typically focus on adapting policy according to inferred opponent information in evaluation, while our algorithm prevents performance degradation in testing against a consistent opponent. One way is to use our algorithm to train a population of risk-free exploitation policies against opponents of different types and adapt exploitation policy in testing according to the opponent's posterior distribution. We discuss potential negative social impacts in Appendix F.
The authors would like to thank the anonymous reviewers for their insightful discussions and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030 – “New Generation Artificial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
[1] Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
[2] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent Complexity via Multi-Agent Competition. In International Conference on Learning Representations, 2018.
[3] Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining Deep Reinforcement Learning and Search for Imperfect-Information Games. In Advances in Neural Information Processing Systems, volume 33, pages 17057–17069. Curran Associates, Inc., 2020.
[4] Imre Csiszár and János Körner. Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition. Cambridge University Press, 2011.
[5] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1942-1951. PMLR, June 2019.
[6] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[7] Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, and Wei Yang. Greedy when Sure and Conservative when Uncertain about the Opponents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 6829-6848. PMLR, July 2022.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, June 2019.
[9] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A Divergence Minimization Perspective on Imitation Learning Methods. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1259-1277. PMLR, November 2020.
[45] Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, Yinyuting Yin, Bei Shi, Liang Wang, Tengfei Shi, Qiang Fu, Wei Yang, Lanxiao Huang, and Wei Liu. Towards Playing Full MOBA Games with Deep Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pages 621-632. Curran Associates, Inc., 2020.
[46] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6672-6679, April 2020. Section: AAAI Technical Track: Machine Learning.
[47] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. In The Eleventh International Conference on Learning Representations, 2023.
[48] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games. CoRR, abs/2103.01955, 2021. arXiv: 2103.01955.
[49] Chongjie Zhang and Victor R. Lesser. Multi-Agent Learning with Policy Prediction. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press, 2010.
[50] Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. AlphaHoldem: High-Performance Artificial Intelligence for Heads-Up No-Limit Poker via End-to-End Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4689–4697, June 2022. Section: AAAI Technical Track on Domain(s) Of Application.
[51] Rui Zhao, Jinming Song, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination. CoRR, abs/2112.11701, 2021. arXiv: 2112.11701.
Proposition 4.2 The performance gap of evaluating policy profile  $(\pi, \mu)$  and  $(\pi, \pi_B)$  at state  $s$  is  $\varepsilon(s) = V^{\pi, \pi_B}(s) - V^{\pi, \mu}(s)$ , which can be decomposed in a Bellman-like recursion:
$$
\begin{array}{l} \varepsilon (s) = V ^ {\pi , \pi_ {B}} (s) - V ^ {\pi , \mu} (s) \\ = \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi \left(a _ {1} | s\right) P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left(\pi_ {B} \left(a _ {2} \mid s\right) - \mu \left(a _ {2} \mid s\right)\right) \left(r \left(s, \boldsymbol {a}, s ^ {\prime}\right) + \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right)\right) \tag {7} \\ + \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi (a _ {1} | s) \pi_ {B} (a _ {2} | s) P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \gamma \varepsilon \left(s ^ {\prime}\right) \\ \end{array}
$$
$$
\begin{array}{l} \varepsilon (s) = V ^ {\pi , \pi_ {B}} (s) - V ^ {\pi , \mu} (s) \\ = \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} \pi_ {B} (a _ {2} | s) \sum_ {s ^ {\prime}} P _ {M} (s ^ {\prime} | s, \boldsymbol {a}) [ r (s, \boldsymbol {a}, s ^ {\prime}) + \gamma V ^ {\pi , \mu} (s ^ {\prime}) + \gamma \varepsilon (s ^ {\prime}) ] \\ - \sum_ {a _ {1}} \pi \left(a _ {1} \mid s\right) \sum_ {a _ {2}} \mu \left(a _ {2} \mid s\right) \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left[ r (s, \boldsymbol {a}, s ^ {\prime}) + \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right) \right] \tag {8} \\ = \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} [ (\pi_ {B} (a _ {2} | s) - \mu (a _ {2} | s)) A + B ], \\ \end{array}
$$
$$
A = \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) r (s, \boldsymbol {a}, s ^ {\prime}), \tag {9}
$$
$$
\begin{array}{l} B = \pi_ {B} (a _ {2} | s) \sum_ {s ^ {\prime}} P _ {M} (s ^ {\prime} | s, \boldsymbol {a}) \gamma (V ^ {\pi , \mu} (s ^ {\prime}) + \varepsilon (s ^ {\prime})) - \mu (a _ {2} | s) \sum_ {s ^ {\prime}} P _ {M} (s ^ {\prime} | s, \boldsymbol {a}) \gamma V ^ {\pi , \mu} (s ^ {\prime}) \\ = \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left[ \pi_ {B} \left(a _ {2} \mid s\right) \gamma \left(V ^ {\pi , \mu} \left(s ^ {\prime}\right) + \varepsilon \left(s ^ {\prime}\right)\right) - \mu \left(a _ {2} \mid s\right) \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right) \right] \tag {10} \\ = \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left[ \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right) \left(\pi_ {B} \left(a _ {2} \mid s\right) - \mu \left(a _ {2} \mid s\right)\right) + \gamma \pi_ {B} \left(a _ {2} \mid s\right) \varepsilon \left(s ^ {\prime}\right) \right]. \\ \end{array}
$$
Putting equation 9 and 10 back, we have
$$
\begin{array}{l} \varepsilon (s) = V ^ {\pi , \pi_ {B}} (s) - V ^ {\pi , \mu} (s) \\ = \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} \left[ \left(\pi_ {B} (a _ {2} | s) - \mu (a _ {2} | s)\right) \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) r (s, \boldsymbol {a}, s ^ {\prime}) + \right. \\ \left. \sum_ {s ^ {\prime}} P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left[ \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right) \left(\pi_ {B} \left(a _ {2} \mid s\right) - \mu \left(a _ {2} \mid s\right)\right) + \gamma \pi_ {B} \left(a _ {2} \mid s\right) \varepsilon \left(s ^ {\prime}\right) \right] \right] \tag {11} \\ = \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi (a _ {1} | s) P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left(\pi_ {B} \left(a _ {2} \mid s\right) - \mu \left(a _ {2} \mid s\right)\right) \left(r \left(s, \boldsymbol {a}, s ^ {\prime}\right) + \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right)\right) \\ + \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi (a _ {1} | s) \pi_ {B} (a _ {2} | s) P _ {M} (s ^ {\prime} | s, \boldsymbol {a}) \gamma \varepsilon (s ^ {\prime}). \\ \end{array}
$$