## 1 Introduction
For each of these policies, the robot needs to use an appropriate response policy to achieve optimal performance on the task [40]. It is very inefficient if the robot has to learn new coping strategies from scratch during online interactions. More commonly, the robot maintains an offline response policy library and reuses the response policies directly based on human online behaviors. Compared with the implicit modeling methods that can only learn the response policy from scratch during online interactions, our algorithm can reuse the offline learned policy to accelerate online decisions using explicit modeling methods. One feasible direction for implementing explicit modeling is to exploit offline knowledge to improve online performance [39]. Type-based reasoning methods [30; 41] assume that the opponent switches between several known types and the agent maintains a prior belief about the probability distribution of the opponent's types. During online interactions, the belief is updated based on any task-related signals, and then the agent can reuse the existing response policy based on the updated belief. Bayesian policy reuse (BPR) [39] is an explicit opponent modeling framework based on type-based reasoning. It was initially applied to single-agent domains, where the agent can detect task types and reuse the learned best response policies when faced with multiple tasks. Regarding different opponent's policies as different task types in BPR, BPR+ [40] first extended BPR to multi-agent scenarios to detect dynamic changes in the opponent's policies. In addition, Bayes-Pepper [25] combines BPR with Pepper [42] to improve detection accuracy by using the opponent's state-action pairs as an additional signal during belief updating. However, as table-based algorithms, BPR+ and Bayes-Pepper are spatially inefficient and even infeasible when dealing with complex scenarios. To overcome this problem, Deep BPR+ [43] proposes using a deep neural network as a value function approximator and an opponent model, which can be used to rectify the belief model in BPR+ to improve the detection accuracy. However, Bayes-Pepper and Deep BPR+ may suffer from incomplete opponent information. The above algorithms [22; 23; 40; 43] assume that the opponent switches randomly among a class of stationary policies, but the opponent may be equally capable of reasoning, and thus adopts a more complex policy-switching approach. To address this issue, Bayesian theory of mind on policy (Bayes-ToMop) [44] assumes that the opponent also uses BPR. However, Bayes-ToMop needs to detect whether the opponent is using BPR through additional interactions when the opponent switches strategies randomly, which can be extremely time-consuming. Most BPR-based algorithms reviewed here assume that the agent can observe the opponent's local information during online interactions, including its observations and actions. In practice, many tasks are characterized by partial observability, in which the agent can only obtain incomplete and uncertain observations about the opponent, making accurate policy detection more challenging. In this work, we consider how to act optimally against non-stationary opponents that switch among several stationary strategies under partial observability. According to [45], a strength of VAE is that it can reason about the opponent's policies without the opponent's local observations. However, this method cannot reuse the learned response policies during online interactions for lacking explicit classification of the opponent's policies. On the contrary, the BPR-based methods explicitly classify the opponent's policies, and the optimal learned response policies can be reused during online interactions. However, most BPR-based methods require the opponent's local observations to accurately identify the opponent's policies. We consider that it is possible to combine these two forms of modeling to improve online performance under partial observability.