## 3 Bayes-Lab
### Online Oponent Detection and Policy Reuse  
During the online stage, the agent needs to accurately detect unknown opponent types and reuse the optimal response policies accordingly. Algorithm 2 outlines the above process. ```
0: agent policy library \(\Pi\), opponent policy library \(\mathcal{T}\), opponent model library \(\mathcal{X}\), performance models \(P(U|\mathcal{T},\Pi)\), reconstruction accuracy models \(\Omega(W|\mathcal{T},\Pi,\mathcal{X})\), planning similarity models \(\Theta(Q|\mathcal{T},\Pi,\mathcal{X})\), episodic belief \(\beta^{k}(\tau)\), the maximum number of episodes \(M\), a basic environment \(env_{\text{b}}\), a parallel environment \(env_{\text{p}}\)
0: response policy
1: Initialize \(\beta^{k}(\tau)\) with uniform distribution
2:forepisode \(k\)=1,2,3,...,M do
3: Use BPR-EI to select a response policy \(\pi^{*}_{k}\) based on \(\beta^{k}(\tau)\)
4: Select an opponent model \(x^{*}_{k}\) based on \(\pi^{*}_{k}\)
5: Use \(\pi^{*}_{k}\) to play against unknown opponent policy
6: Collect the episodic reward \(u^{k}\), the agent's local observations \(\varphi_{1}\)
7: Use \(x^{*}_{k}\) to reconstruct the opponent's actions \(a^{*}_{-1,recon}\) based on \(\varphi_{1}\)
8:if\(a^{*}_{-1}\) are available then
9: Compute the reconstruction accuracy \(w\) based on \(a^{*}_{-1}\) and \(a^{*}_{-1,recon}\)
10: Use the episodic reward \(u^{k}\) and \(w^{k}\) to update \(\beta^{k}(\tau)\)
11:elseif\(a^{*}_{-1}\) are unavailable then
12:\(env_{\text{b}}\) executes actions \(\left(a^{*}_{1},a^{*}_{-1}\right)\)
13: Sample the agent's observations \(o^{*}_{1}\) from \(env_{\text{b}}\)
14:\(env_{\text{p}}\) executes actions \(\left(a^{*}_{1},a^{*}_{-1,recon}\right)\)
15: Sample the agent's observation \(o^{*}_{1,p}\) from \(env_{\text{p}}\)
16: Compute the similarity \(q\) between \(o^{*}_{1,p}\) and \(o^{*}_{1}\)
17: Use the episodic reward \(u^{k}\) and \(q^{k}\) to update \(\beta^{k}(\tau)\)
18:endif
19:endfor
```  
**Algorithm 2** Online opponent detection and policy reuse  
Before starting the interaction, the belief \(\beta(\tau)\) is initialized with a uniform distribution. Based on the current belief \(\beta^{k}(\tau)\), the agent selects an optimal response policy \(\pi^{*}_{k}\) by using BPR-EI as follows (line 3 in Algorithm 2):  
\[\pi^{*}_{k}\text{=}\underset{\pi\in\Pi}{\text{argmax}}\ \int_{\Omega}^{U^{max}} \sum\tau\in\tau\beta^{k}(\tau)P\big{(}U^{+}|\tau,\pi\big{)}dU^{+} \tag{8}\]  
where the optimal estimated utility \(\bar{U}=\max_{\pi\in\Pi}\Sigma_{\tau\in\mathcal{T}}\beta^{k-1}(\tau)\mathbb{E }[U|\tau,\pi]\), and \(\pi^{*}_{k}\) is adopted until the end of episode \(k\). In addition, the agent uses a corresponding opponent model \(x^{*}_{k}\) to reconstruct the opponent's actions \(a^{*}_{-1,recon}\) only based on its local observations \(\varphi_{1}\) (line 7 in Algorithm 2). We propose two detecting modes to handle different opponent information, and we assume that the opponent's observations are always unavailable. Specifically, when the opponent's actions are observable, the agent calculates the reconstruction accuracy \(w^{k}\) by comparing \(a^{*}_{-1,recon}\) and \(a^{*}_{-1}\), and then jointly uses the episodic reward \(u^{k}\) and \(w^{k}\) to update the episodic belief as follows (line 10 in Algorithm 2):  
\[\beta^{k}(\tau)=\frac{\Omega\Big{(}w^{k}|\tau,\pi^{k},x^{k}\Big{)}P\Big{(}u^{ k}\Big{|}\tau,\pi^{k}\Big{)}\beta^{k-1}(\tau)}{\sum\limits_{\tau\in\mathcal{T}} \Omega\Big{(}w^{k}|\tau,\pi^{k},x^{k}\Big{)}P\big{(}u^{k}|\tau,\pi^{k}\big{)} \beta^{k-1}(\tau)} \tag{9}\]  
Otherwise, a planning method is proposed to infer the opponent's type if the opponent's actions cannot be observed. In detail, the opponent executes the real action \(a^{*}_{-1}\) in \(env_{b}\), while in \(env_{p}\), the opponent executes the reconstructed actions \(a^{*}_{-1,recon}\). The agent starts planning with a known initial state, and then reasons about its observations \(o^{*}_{1,p}\) in \(env_{p}\).