## 2 Preliminaries
### Bayesian Policy Reuse  
BPR [39] was originally proposed as a framework for an agent to determine the best response policy to execute when facing an unknown task in Markov Decision Process (MDP) [46, 47]. Given a set of previously solved tasks \(\mathcal{T}\) and current task \(\tau^{*}\), the agent is expected to detect the type of task efficiently and reuse the optimal response policy \(\pi^{*}\) from the policy library \(\Pi\) accordingly based on the observation signal \(\sigma\) within a small number of interactions. The signal \(\sigma\) can be any information related to the performance of a policy (e.g., episodic rewards, encountered state-action-state tuples). In addition, a performance model \(P(U|\tau,\pi)\) is a probability distribution over the utility \(U\) generated by applying a policy \(\pi\) to a previously solved task \(\tau\). BPR uses the concept of belief \(\beta\), which is a probability distribution over the tasks \(\mathcal{T}\), to estimate the matching degree between \(\pi^{*}\) and the known tasks based on the signal \(\sigma\). After each trial episode, \(\beta^{k}(\tau)\) is updated using a Bayes' rule as follows:  
\[\beta^{k}(\tau)=\frac{P\Big{(}\sigma^{k}\Big{|}\tau,\pi^{k}\Big{)}\beta^{k-1} (\tau)}{\sum_{\tau^{\prime}\in\mathcal{T}}P\big{(}\sigma^{k}\big{|}\tau^{ \prime},\pi^{k}\big{)}\beta^{k-1}(\tau^{\prime})} \tag{2}\]where \(\beta^{k-1}(\tau)\) is the belief after episode \(k-1\). In addition, belief updating can be regarded as identifying task types. For example, suppose the agent's policy library includes two learned response policies, \(\pi_{1}\) and \(\pi_{2}\), which are used to handle tasks \(\tau_{1}\) and \(\tau_{2}\), respectively. If the agent uses \(\pi_{1}\) to interact with an unknown task which is one of \(\tau_{1}\) or \(\tau_{2}\), the agent will receive a signal \(\sigma\) that can be combined with the performance model \(P\) to detect the task type. Specifically, \(P(\sigma|\tau_{1},\tau_{1})\) reflects the possibility that the unknown task is \(\tau_{1}\). Similarly, \(P(\sigma|\tau_{2},\pi_{1})\) reflects the possibility that the unknown task is \(\tau_{2}\). By comparing \(P(\sigma|\tau_{1},\pi_{1})\) and \(P(\sigma|\tau_{2},\pi_{1})\), the agent can identify the unknown task type. Moreover, the received signal will be different if the task switches from one to another, resulting in changes in the performance model. As a result, the agent's belief \(\beta(\tau)\) can be rectified based on Equation (2), leading the agent to reselect its response policy.  
Several variants of BPR with exploration heuristics are available to choose the optimal response policy, among which BPR using an expected improvement (BPR-EI) presents the best performance. Assume that \(\Omega=\max_{\tau\in\Pi}\Sigma_{\tau\in\mathcal{T}}\beta(\tau)E[U|\tau,\pi]\) is the optimal estimated utility following the existing belief. BPR-EI selects the policy most likely to lead to any utility improvement \(\Omega<U^{+}<U^{\text{max}}\) as follows:  
\[\pi^{*}= \underset{\pi\in\Pi}{\text{argmax}}\,\int_{\Omega}^{U^{\text{max}}} \,\sum_{\tau\in\mathcal{T}}\beta(\tau)P\big{(}U^{+}|\tau,\pi\big{)}dU^{+} \tag{3}\]