# Time-Varying Preference Bandits for Robot Behavior Personalization
## 4 Methods
### Absolute Upper Confidence Bound  
Suppose that the parameter \(\hat{\theta}_{t}\) is estimated, which will be explained later. At round \(t\), DPB chooses an action based on the following action selection rule  
\[X_{t}:=\arg\max_{X\in\mathcal{X}}\ |X^{\mathsf{T}}\hat{\theta}_{t-1}|+\alpha_{t- 1}\|X\|_{V_{t-1}^{-1}}, \tag{4}\]  
where \(V_{t-1}:=\sum_{s=1}^{t-1}\gamma^{t-1-s}X_{s}X_{s}^{\mathsf{T}}+\lambda I_{d}\), \(X_{s}\) is a past query vector from \(s=1\) to \(t-1\), \(\lambda\) is a regularization coefficient, \(I_{d}\) is an identity matrix, and \(\alpha_{t}\) is a scale parameter that controls the importance between the first and second term.  
The first term indicates the absolute difference of the estimated rewards between two trajectories. Since we construct a context vector from two trajectories, \(X\) and \(-X\) contain the same information, i.e., \(X=\Phi(\xi_{A})-\Phi(\xi_{B})\) and \(-X=\Phi(\xi_{B})-\Phi(\xi_{A})\). Hence, computing the reward via the absolute value ensures the equivalence between selecting \(X\) and \(-X\). Based on this trick, the query selection method (4) employs the upper confidence bound (UCB) [20]. The second term in (4) represents the confidence bound that magnifies the amount of the uncertainty of the first term \(|X^{\mathsf{T}}\hat{\theta}_{t-1}|\). Particularly, \(V_{t}\), called a design matrix, embodies the empirical covariance of \(X\), and the parameter \(\gamma\) of \(V_{t}\) is a discount factor in \((0,1]\) which penalizes an effect of past data. Intuitively, as \(t\) grows, additional query vectors are added into \(V_{t}\), thereby augmenting the minimum eigenvalue of \(V_{t}\) and, thus, diminishing the term \(\alpha_{t}\|X\|_{V_{t-1}^{-1}}\). Consequently, the confidence bound eventually decreases.  
Our proposed query selection method simultaneously considers two key factors by leveraging UCB. The first factor evaluates how much the chosen query contributes to learning the relevant parameters. The second factor considers how much the user will like the query when presented as a demonstration. While conventional approaches focus on the first factor [4; 5; 6; 7], the proposed approach incorporates the second factor by applying the UCB method. The consideration of the quality of queries experienced by the user [5] is vital because it may help build user familiarity and trust with the robot. If users consistently encounter undesirable queries, it might lead to mistrust in the robot's behavior. Therefore,our approach carefully balances these factors, adjusting the trade-off between two factors via \(a_{t}\), which will be further examined in Section 5.  
While the query selection rule selects a single query, choosing a batch of queries is more efficient in practice. In PBL, extended durations for query generation and parameter updates can challenge users, particularly those who are less patient. Thus, we adopt a simple batched version by selecting the top \(b\) queries based on the UCB score (4), where \(b\) is the number of queries in a single batch. To approximate the solution for (4), we prepare a finite set of trajectory pairs by randomly generating and selecting two trajectories. The feature vectors derived from this set are used to compute (4) and to select the top-\(b\) queries among the finite set.