## 1 Introduction  
Recently, deep neural networks have provided excellent performance in image recognition [1], pattern recognition [2], and security [3; 4; 5]. They are usually combined with reinforcement learning (RL) and have achieved great success [6; 7; 8; 9]. Many advanced deep RL algorithms have been proposed and used to solve complex decision-making problems such as game playing [9; 10], robotics [11; 12; 13; 14], and autonomous driving [15; 16; 17; 18]. In multi-agent systems [19; 20], if an opponent's policy is fixed, the agent can treat it as part of the stationary environment and learn the response policy using single-agent RL algorithms [21; 22; 23]. However, in many cases, the opponent's behaviors may change over time and affect the objectives of the others, which implies a non-stationary environment [24]. As a result, it is essential for the agent to adapt its policy by taking into account the non-stationary opponent [25; 26; 27; 28], which is regarded as the second sophisticated challenge categorized in [26]. In this paper, we focus on dealing with the non-stationary opponents in a partially observable environment, where we control an agent with a limited horizon and the other agents (opponents) select their fixed policies from a policy library. In our setting, the controlled agent is required to reason about the opponent's policy and adjust its policy to achieve optimal rewards. One effective way to address the non-stationarity is opponent modeling [22; 27; 29; 30]. According to [30], the opponent modeling methods can be divided into implicit modeling and explicit modeling. A natural idea for implicit modeling is to use deep neural networks to encode the opponent's policy features [27; 29; 31; 32]. Specifically, such models usually consist of two networks [33]: one is to extract the latent policy features from supervision signals such as the opponent's actions [31], goals [34], and beliefs [27], and the other learns the response policy by taking into account these policy features. The deep reinforcement opponent network (DRON) [29] uses an additional network to extract the opponent's policy features, and it incorporates a mixed-of-expert structure with a deep Q-network (DQN) [35] to handle different opponent policies. However, it cannot guarantee optimality against each particular opponent's policy and assumes that the agent can observe the opponent's actions. Deep policy inference Q-network (DPIQN) [36] learns policy features directly from the opponent's raw observations. These features are learned by auxiliary tasks that provide additional supervision signals, such as the opponent's actions. Self-other modeling (SOM) [34] proposes using the agent's policy to predict the opponent's goal. Despite its simplicity, it takes longer to train, as an additional optimization step is required to consider the opponent's actions. Most studies reviewed here assume that the opponent's observations and actions can be accessed during online interactions. However, this assumption is unlikely to hold in practical scenarios. To address this problem, local information agent modeling (LIAM) [37] first proposed the use of a variational autoencoders (VAE) [38] model to learn the opponent's policy features in RL. This method can extract the hidden opponent features during training conditioned only on the agent's local observations. In other words, it does not require the opponent's local information during execution. A mutual limitation of the above studies is that they cannot reuse the existing offline response policies to accelerate online decisions for lacking explicit opponent categorization. When the opponent switches between several known policies, the agent has to learn new response policies from scratch, which is inefficient [30]. The above problem is motivated by many application domains where the agent requires fast online decision making [39]. For example, a service robot has to interact with people with different policies (behaviors).