## 3 Background  
Our investigation uses a deterministic and fully observable Markov decision process (MDP) model. The MDP state at time \(t\), denoted as \(s_{t}\in S\), and the MDP action (control input) at time \(t\), represented by \(a_{t}\in A\), are key components of this model. We define a trajectory \(\xi\in\Xi\) as a finite sequence of states and actions, expressed as \(((s_{t},a_{t}))_{t=0}^{T}\), where \(T\) indicates the time horizon of the MDP. Given the deterministic nature of the MDP, the trajectory \(\xi\) can be succinctly expressed as \(\Lambda=(s_{0},a_{0},\ldots,s_{T},a_{T})\), which includes the initial state and the series of subsequent actions. Depending on the context, we may use the symbols \(\xi\) and \(\Lambda\) interchangeably to describe the trajectories.  
We assume a reward function \(R:\Xi\rightarrow\mathbb{R}\) to specify the desired system behavior. The primary goal of this study is to elucidate the learning process for \(R\). Specifically, we define \(R\) as a linear combination of chosen features \(\Phi:\Xi\rightarrow\mathbb{R}^{d}\), such that \(R(\xi)=\omega^{\top}\cdot\Phi(\xi)\). Learning \(R\) requires the acquisition of the human-specified parameter \(\omega\). In the context of this research, a preference query \(Q\) is represented by the set \(\{Q:=\xi_{1},\xi_{2},\ldots,\xi_{k}\}\), encompassing a collection of \(k\) elements. Participants express their preferences within our framework by selecting preferred trajectories from this set; in our particular case, \(k=2\). Our methodology for learning the reward parameter \(\omega\) involves interpreting human responses to comparative queries between two trajectories.  
Our objective is to ascertain the precise reward parameter using the minimum number of preference queries and to identify a sequence of questions that maximizes the expected information gain regarding \(\omega\). An optimal query sequence should effectively identify the human in the loop and generate \"easy\" questions, reducing the probability of negative feedback. Unfortunately, this question often falls into the category of NP-hard problems [29], a classification that denotes problems for which no known polynomial-time algorithm can guarantee a solution in all instances Consequently, we use a greedy strategy, i.e., we initialize the distribution over \(\omega\) and iteratively alternate between posing a single query and updating the distribution based on human responses. Certain query-generation policies employ mutual information algorithms.  
We developed belief models to represent the \(\omega\) values or reward weight parameters characterized by the distribution over \(\omega\). These belief models are revised to incorporate human responses. The probability that a human will select trajectory \(q\) from query \(Q\), given the reward parameter \(\omega\), is denoted by \(P(q|Q,\omega)\). Utilizing the prior distribution \(P(\omega)\) and the human's response \(q\) to query \(Q\), we compute the posterior probability of \(\omega\) as shown in Equation (1).  
\[P(\omega|Q,q)\propto P(q|Q,\omega)P(\omega) \tag{1}\]  
The primary objective of this research is to develop a learning model for the weight parameter \(\omega\). To achieve this, we construct a Bayesian framework predicated on \(\omega\) and refine the posterior distribution of \(\omega\) through active learning from preference queries. Our aim is to formulate a reward function incorporating the \(\omega\) weights that are consistent with human preferences. Subsequent sections delineate the methodology for generating preference queries and the procedure for Bayesian posterior updating of \(\omega\).