## 3 Bayes-Lab
### Offline Policy Learning and Model Generation  
During the offline stage, Bayes-Lab first needs to learn the response policy \(\pi\) and the opponent model \(x\) and then use them to constitute the response policy library \(\Pi\) and the opponent model library \(\mathcal{X}\), respectively. In addition, we use the VAE networks to represent the opponent model. As shown in Figure 2, Bayes-Lab updates the A2C networks and the VAE networks simultaneously, and the VAE networks consist of an encoder and a decoder. The encoder learns a variational Gaussian distribution \(\mathcal{N}(\mathbf{\mu}_{\sigma},\mathbf{\Sigma}_{\sigma})\) with parameters \(\sigma\) and is denoted as \(q_{\sigma}(z|\varphi_{1})\), which is used to approximate the true posterior distribution \(p(z|\varphi_{-1})\). The approximate posterior distribution \(q_{\sigma}(z|\varphi_{1})\) can map the agent's local observations \(\varphi_{1}\) to a hidden variable space \(\mathcal{Z}\) from which the hidden variable \(z\) is sampled as the representation of the current opponent policy. In addition, the KL divergence \(D_{KL}(q_{\sigma}(z|\varphi_{1})\|p(z|\varphi_{-1})\,)\) is used to measure the similarity between the approximate posterior distribution and the real posterior distribution. Figure 1: An overview of Bayes-Lab. The decoder learns a generative distribution \(p_{\alpha}(\varphi_{-1}|z)\) with parameters \(\alpha\) that maps the hidden variable \(z\) to the opponent's local information \(\varphi_{-1}\), which is used as a supervision signal to update the encoder-decoder networks. As a result, the opponent model can reconstruct \(\varphi_{-1}\) conditioned only on \(\varphi_{1}\). Moreover, we use the \(MMD^{2}\) distance [55,56] for regularization. The loss of the encoder-decoder networks is formulated as follows:  
\[\mathcal{L}(\varphi_{-1},\varphi_{1};\alpha,\sigma)=-\mathbb{E}_{z\sim q_{c}( z|\varphi_{1})}[\log p_{\alpha}(\varphi_{-1}|z)]+\lambda D_{MMD}(q_{c}(z| \varphi_{1})\|p(z)) \tag{6}\]  
where \(\lambda\) is a scaling parameter. In the original A2C algorithm, the input of the policy networks is only the agent's observations, and the obtained policy can be expressed by \(\pi(a|o_{1})\). Obviously, it cannot be directly applied in a multi-agent environment for not considering the impact of the opponent's policy on its policy. To explicitly consider the coexisting opponents in the environment, we propose using the latent variable \(z\), which is sampled as the low-level feature of the current opponent policy. Specifically, the agent's observation \(o_{1}\) augmented with \(z\) can be used as input signals to update the A2C networks, and the learned policy can be expressed by \(\pi(a|o_{1},z)\). We emphasize that the VAE networks are updated by supervised learning, while the A2C networks are updated by reinforcement learning. Therefore, A2C is a separate network, not another decoder. One advantage of this architecture is that each opponent's policy \(\tau_{i}\) can correspond to an optimal response policy \(\tau_{i}\) and a specific opponent model \(x_{i}\), which is essential for reusing the response policies based on the opponent types. Although A2C and VAE are updated simultaneously, they use different optimizers and learning rates. As a result, the loss generated by A2C will not affect the parameters of VAE through back-propagating the gradient. Given a batch of the agent's behaviors data, the objective of A2C is:  
\[ \begin{split}\mathcal{L}_{A2C}&=\mathbb{E}_{(o_{1},r,\,a_{1},\,r_{1}+,\,o_{1},\,t)\sim B}\bigg{[}\frac{1}{2}\big{(}r_{1,t+1}+ \gamma V_{\varphi}(o_{1,t+1},z_{t+1})-V_{\varphi}(o_{1,t},z_{t})\big{)}^{2}\\ &\quad-\hat{A}\log\pi_{\theta}(a_{1,t}|o_{1,t},z_{t})-\beta H(\pi _{\theta}(a_{1,t}|o_{1,t},z_{t}))\big{]}\end{split}  \tag{7}\]  
where \(B\) is the batch of transitions, and \(\hat{A}\) is the basic advantage term.