# Time-Varying Preference Bandits for Robot Behavior Personalization
## 6 Experimental Settings  
**Simulation Setup.** We validate our work in three simulation environments: _Driver_[1], _Tosser_[21], and _Avoiding_. The _Driver_ environment aims to drive while aware of the other vehicle and the _Tosser_ environment learns to put the ball in a certain basket with diverse trajectories. The features utilized are identical to [5], distance to the closest lane, speed, heading angle, and distance to the other vehicles for _Driver_ and maximum horizontal range, maximum altitude, the sum of angular displacements at each timestep, and final distance to the closest basket for _Tosser_. We newly created an _Avoiding_ environment where the robot moves the object over the laptop to place the final target pose, similarly to [22]. Four-dimensional hand-coded features: The height of the end-effector from the table, the distance between the end-effector and the laptop, the moving distance, and the distance between the end-effector and the user, are utilized in _Avoiding_. Optimal parameters were randomly generated for each seed.  
**Dataset.** To discretize a trajectory space, a query set \(\mathcal{X}\) is predefined in _Driver_ and _Tosser_ by sampling \(K\) trajectories with uniformly random controls. In _Avoiding_, RRT* is used to create trajectories after randomly sampling the passing midpoint through the fixed start and target point. We set \(K\) to _20,000_ for _Driver_ and _Tosser_, and to _5000_ for _Avoiding_.  
**Evaluation Metrics.** In our experiments, we use the following three suitable metrics: the cosine similarity \(m_{CS}\), the simple regret \(m_{SR}\), and the cumulative regret \(m_{CR}\). First, cosine similarity \(m_{CS}=(\hat{\theta}_{t})\tau_{t}^{\theta_{t}}/\|\hat{\theta}_{t}\|_{2}\| \theta_{t}^{*}\|_{2}\) is measured as the alignment metric leveraged in most existing research [4; 5; 6]. Simple regret is defined as \(m_{SR}=\Phi(\hat{g}_{t}^{*})\tau_{t}^{\theta_{t}}-\Phi(\hat{g}_{t}^{*})\tau_{t} ^{\theta_{t}}\), where \(\hat{g}_{t}^{*}\) is an optimal trajectory of learned parameter, i.e., \(\hat{g}_{t}^{*}:=\arg\max_{\chi\in\Xi}\Phi(\xi)\hat{\tau}_{t}\). The quality of the optimized trajectory can be measured by the simple regret, with smaller values indicating better performance. Cumulative regret \(m_{CR}\), defined in Section 5, illustrates howmuch reward will be lost by exploration. Minimizing cumulative regret is often the object of the bandit framework.  
**Baselines.** We compare the performance of DPB with other methods using different criteria for query selection, such as batch active learning [5; 7], information gain [6], and maximum regret [3]. All these baseline algorithms were adapted into batch selection versions to select the top-\(b\) queries to ensure fair comparisons.  
* _Greedy:_ This method selects top-\(b\) queries based on conditional entropy in a greedy way [5].
* _Medoid:_ This method first selects top-\(B\) queries based on conditional entropy where \(B>b\). \(B\) samples are clustered into \(b\) subsets via k-medoids [23] and selects \(b\) representative queries from the subsets.
* _DPP:_ This method naturally considers diversity and informativeness simultaneously by using k-DPP [24].
* _Information:_ A query is selected with the highest information gain for easy labeling by the user [6].
* _Max Regret:_ A query is selected with the highest regret over the solution space. Note that regret in [3] uses the same terminology but does not adhere to the conventional notion of regret like (6).
* _Random:_ We select uniformly random \(b\) queries.