## 1 Introduction
Specifically, we perform an explicit opponent categorization and reuse the learned response policies without the opponent's local information. We follow the centralized training with decentralized execution (CTDE) paradigm and propose an algorithm called Bayesian policy reuse with LocAl objServations (Bayes-Lab), which incorporates VAE into BPR. Specifically, we train VAE during the offline phase to extract the latent opponent's policy features only based on the agent's local observations, and then use VAE to reconstruct the opponent's actions during online interactions. In addition, we collect the reconstructed actions and episodic rewards to fit the reconstruction accuracy model, planning similarity model, and performance model, which are defined as Gaussian distributions. Using the generated models, we update the agent's belief about the opponent's types during the online phase. Finally, we reuse the best response policy based on the updated belief. Compared with most BPR-based algorithms [22; 23; 40; 43; 44] that infer the opponent's policy from their behaviors, Bayes-Lab can track the opponent's policy only conditioned on the agent's local observations. The most significant contribution of Bayes-Lab is that it can efficiently detect the non-stationary opponents under partial observability and accurately reuse the offline response policies to improve the online performance. The main novelty of this paper is as follows: (1) An opponent model is introduced to reconstruct the opponent's local observations only based on the agent's local observations. Bayes-Lab uses VAE as an opponent model to learn the relationship between the agent's local observations and the opponent's local observations. Then, the learned opponent model can be used to reconstruct the opponent's actions under partial observability. (2) An offline learning architecture is introduced to simultaneously learn the response policy and the opponent model. Bayes-Lab uses VAE to extract the opponent's policy features which are integrated with the agent's observations to learn the response policy. The advantage of this architecture is that each opponent's policy can correspond to an optimal response policy and a specific opponent model, which is essential for detecting the opponent's type and reusing the response policy. (3) Bayes-Lab is the first to propose integrating VAE as an opponent model into BPR. Specifically, it combines the ability of BPR to improve online performance by using offline knowledge with the capability of VAE to reason about the opponent'sbehaviors under partial observability. It can continuously infer the opponent's policies during online interactions without the opponent's local observations. Compared with other BPR-based methods that can only be used in the globally observable environment, Bayes-Lab can effectively apply BPR in the partially observable environment. (4) The reconstruction accuracy model and planning similarity model are introduced to improve the detection accuracy during online interactions. Bayes-Lab can use these models to detect the opponent's policy and whether or not the opponent's actions can be observed. The rest of the paper is organized as follows: Section 2 presents the formal model of RL, BPR, and VAE. Section 3 introduces the proposed Bayes-Lab algorithm. Section 4 presents a validation environment, experimental settings, and corresponding experimental results. Section 5 concludes our contributions and problems that need to be solved.