## 3 Bayes-Lab
### Offline Policy Learning and Model Generation
Furthermore, the offline learning mechanism of Bayes-Lab is relatively similar to LIAM [37], but there are significant differences: LIAM combines multiple opponent policies randomly and uses VAE to learn the general features of the combined policies. As a result, it only trains a single VAE model to deal with different opponent policies. The drawback of LIAM is that the agent can only learn from scratch and cannot reuse the existing policies during online interactions. In contrast, Bayes-Lab trains several VAE models, and each  
Figure 2: Diagram of offline learning architecture. VAE model corresponds to a particular opponent type. Therefore, it can reuse the learned policies based on the detection results. After learning the response policies and the opponent models, we formulate the corresponding performance model, the reconstruction accuracy model, and the planning similarity model, which are defined as Gaussian distributions. Algorithm 1 outlines the above process. ```
0: agent policy library \(\Pi\), opponent policy library \(\mathcal{T}\), opponent model library \(\mathcal{X}\)
0: planning similarity model \(\Theta(Q|\mathcal{T},\Pi,\mathcal{X})\), reconstruction accuracy model \(\Omega(W|\mathcal{T},\Pi,\mathcal{X})\), performance model \(P(U|\mathcal{T},\Pi)\)
1:for each \(\tau\in\mathcal{T}\)do
2:for each \(\pi\in\Pi\), \(x\in\mathcal{X}\)do
3: Use policy \(\pi\) to play against policy \(\tau\) multiple episodes
4: Collect the episodic reward \(u\), the agent's local observations \(\varphi_{1}\), the opponent's actions \(a^{*}_{-1}\)
5: Fit \(u\) to generate the performance model \(P(U|\tau,\pi)\)
6: Use \(x\) to reconstruct the opponent's actions \(a^{*}_{-1,recon}\) based on \(\varphi_{1}\)
7:if\(a^{*}_{-1}\) are available then
8: Compute reconstruction accuracy \(w\) based on \(a^{*}_{-1}\) and \(a^{*}_{-1,recon}\)
9: Fit \(w\) to generate the reconstruction accuracy model \(\Omega(W|\tau,\pi,x)\)
10:else if\(a^{*}_{-1}\) are unavailable then
11: Create a basic environment \(env_{\text{b}}\) and a parallel environment \(env_{\text{p}}\)
12:\(env_{\text{b}}\) executes actions \(\left(a^{*}_{1},a^{*}_{-1}\right)\)
13: Sample the agent's observations \(o^{*}_{1}\) from \(env_{\text{b}}\)
14:\(env_{\text{p}}\) executes actions \(\left(a^{*}_{1},a^{*}_{-1,recon}\right)\)
15: Sample the agent's observation \(o^{*}_{1,p}\) from \(env_{\text{p}}\)
16: Compute the similarity \(q\) between \(o^{*}_{1,p}\) and \(o^{*}_{1}\)
17: Fit \(q\) to generate the planning similarity model \(\Theta(Q|\tau,\pi,x)\)
18:endif
19:endfor
20:endfor
```  
**Algorithm 1** Offline models' generation  
For all policies in \(\mathcal{T}\) and \(\Pi\), we simulate them playing against each other for multiple episodes. Moreover, the generated episodic rewards are modeled as the performance model \(P(U|\mathcal{T},\Pi)\). In addition, the agent's local observations \(\varphi_{1}\), the opponent's real actions \(a^{*}_{-1}\), and the reconstructed actions \(a^{*}_{-1,recon}\) are also collected for further calculations. We propose two generating modes to fit different models, where the opponent's observations cannot be obtained by the agent. Specifically, when the opponent's actions are available, we compute the reconstruction accuracy \(w\) by comparing \(a^{*}_{-1}\) with \(a^{*}_{-1,recon}\), and then use \(w\) to generate the reconstruction accuracy model \(\Omega(W|\mathcal{T},\Pi,\mathcal{X})\) (line 9 in Algorithm 1). Otherwise, we propose a planning method when the opponent's actions are unavailable. In detail, we create a basic environment \(env_{b}\) and a parallel environment \(env_{p}\), in which the initial states are identical and known. In \(env_{b}\), the opponent follows policy \(\tau\), while in \(env_{p}\), the agent assumes that the opponent chooses the reconstructed actions \(a^{*}_{-1,recon}\), which is used to infer its own observations \(o^{*}_{1,p}\).