Mudit Verma, Subbarao Kambhampati School of Computing & AI, Arizona State University {muditverma, rao} @ asu.edu
Objectives
Preference Based Reinforcement Learning has gained immense popularity for several tacit tasks however even State of the art algorithms have high variance in agent performance. We investigate reasons for this high variance and tie it t o high bias during reward initialization. We show that this bias occurs with popular initialization techniques like Xavier-Uniform and Orthonormal and present a simple data driven initialization method to mitigate this.
PbRL Objective
•   Approximating the Human Reward Model > Objective: Compute probability for the human preferring trajectory τ 1 over tշ: > Objective: The human reward model can then by learnt by minimizing the cross-entropy between the predictions made by the supervised learner and the ground truth human labels as follows:
Motivation & Method
•   Issue of Reward Initialization > Reward functions already suffer from the issue of degenracy. While this can be a source of high variance in the obtained policy, we find that initial bias of reward functions can also have unintended impacts the agent performance. > We find that data-independent initialization techqniues are limited to ensuring that only the parameters of the reward model are uniform, while the co-domain of the reward model is s till “patchy” and biased. •	 Proposed Data Driven Reward Initialization > Pre-training of the Reinforcement Learning agent is common in PbRL setups like PEBBLE and SURF. > We make use of the pretraining phase and the randomly collected trajectories to collect “diverse” state-action samples. > We can then simply ensure that the reward model predicts some pre- defnined constant over all these states. Kambhampati's research is supported by the J.P. Morgan Faculty Research Award, ONR grants N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840, N00014-9-1-2119, AFOSR grant FA9550- 18-1-0067 and DARPA SAIL-ON grant W911NF19-2-0006. Investigate via Empirical Evaluation Experiments & Results > Question 1: Empirically verify that seed of the experiment impacting reward initialization yields substantially different reward models. (a) Reward values for 7x7 Gridworld states over different seeds via Uniform Initialization > Question 2: Whether the proposed data-driven reward initialization scheme improves / stabilises the performance? > Empirical results on 7x7 and 15x15 gridworlds with backbone PEBBLE are shown in figures (b-d). > Using the data-driven initialization has zero cost to the human in the loop, and requires zero additional trajectory sampling (for algorithms that have a pre- training phase). (b) Reward values for 7x7 Gridworld states after data-driven Reward Initialization > We compare against baselines : Ones, Zeros, Xavier-Uniform, Orthonormal (c) Learning curves for 7x7 gridworld across various initialization schemes. (d) Learning curves for 15x15 gridworld comparing Ours v/s Uniform. 1. 2. Reward learning and Policy learning is s sensitive to Reward Initialization. Empirically, we show that several initialization schemes for the reward model results in high variance in agent performance. 3. We propose a pre-training data-driven reward initialization step that forces the reward-model co-domain to be uniform, minimizing any prior bias over trajectories. Key Takeaways