SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Experimental Results of Residual-Q Policy Customization
Env.	Policy	Basic Task	Add-on Task
Succ. Rate	Basic Reward	ex	Add-on Reward
CartPole	RL Prior Policy	100%	427.21 ± 2.25	0.15 ± 0.11	-31.11 ± 22.82
RL Customized	100%	425.73 ± 2.74	0.05 ± 0.05	-11.18 ± 10.12
IL Prior Policy	100%	422.17 ± 4.19	0.19 ± 0.12	-38.57 ± 24.26
IL Customized	100%	423.37 ± 4.28	0.07 ± 0.06	-14.12 ± 12.09
RL Full Policy	100%	425.27 ± 3.10	0.05 ± 0.04	-10.72 ± 7.43
Env.	Policy	Succ. Rate	Basic Reward	n neg	Add-on Reward
Cont. Mt. Car	RL Prior Policy	100%	95.78 ± 0.64	42.26 ± 1.29	-4.23 ± 0.13
RL Customized	100%	95.61 ± 0.43	37.90 ± 0.75	-3.79 ± 0.07
IL Prior Policy	100%	92.84 ± 2.35	64.40 ± 21.03	-6.44 ± 2.10
IL Customized	100%	94.41 ± 0.06	41.08 ± 1.01	-4.11 ± 0.10
RL Full Policy	100%	95.14 ± 0.21	27.86 ± 3.87	-2.79 ± 0.39
Env.	Policy	Succ. Rate	Basic Reward	I lane	Add-on Reward
Highway	RL Prior Policy	97.22%	44.67 ± 2.43	0.46 ± 0.28	9.17 ± 5.54
RL Customized	97.19%	42.23 ± 1.55	0.97 ± 0.05	19.42 ± 0.99
RL ME-MCTS	98.00%	42.71 ± 1.65	0.96 ± 0.07	19.19 ± 1.38
IL Prior Policy	96.04%	40.03 ± 0.02	0.50 ± 0.40	10.06 ± 8.07
IL Customized	93.00%	40.36 ± 0.34	0.99 ± 0.01	19.88 ± 0.24
IL ME-MCTS	97.00%	40.68 ± 0.46	0.99 ± 0.01	19.85 ± 0.30
RL Full Policy	96.35%	41.98 ± 1.53	0.97 ± 0.01	19.49 ± 1.24
Env	Policy	Succ. Rate	Basic Reward	γno-violation	Add-on Reward
Parking	RL Prior Policy	99.09%	-7.08 ± 2.65	57.61%	-1.55 ± 3.71
RL Customized	98.73%	-7.60 ± 3.07	96.09%	-0.03 ± 0.20
IL Prior Policy	70.91%	-12.37 ± 9.05	11.27%	-4.94 ± 5.52
IL Customized	83.59%	-7.93 ± 4.84	66.07%	-0.32 ± 0.73
RL Full Policy	99.34%	-6.81 ± 2.39	74.24%	-0.38 ± 0.72
The evaluation results are computed over 4000 episodes for model-free policies and 100 episodes for maximum-entropy MCTS (ME-MCTS). The statistics with  $\pm$  are in the format of mean  $\pm$  std.
As discussed in Sec. 5.2, one limitation of our current algorithms is that the performance of policy customization is bottlenecked by the imitative prior policy. Policy customization not only requires it to perform well on the trajectory distribution under the imitative prior policy itself but, more importantly, we need the imitative policy to accurately indicate desired behavior with respect to the basic task objective on the trajectory distribution under the customized policy. Hence, without careful design, our algorithms would suffer from the distributional shift issue. While, in practice, this issue can be mitigated by partially incorporating some basic task objectives into the add-on reward if available, a more principled and fundamental solution is to obtain a diverse imitative prior that is best suited for customization. Generative models, such as normalizing flows [48] and, more recently, diffusion models [23, 6, 20], have been shown promising in constructing such a diverse behavior prior. Besides, incorporating human experts in the loop has also been shown to be effective in improving the generalizability of imitation learning [27, 38, 41, 33]. We will investigate their applications in the policy customization problem in future work. Also, we only demonstrate MCTS online customization with an ideal perfect dynamics model in our current experiments. In future work, we will combine residual maximum-entropy MCTS with offline learned predictive models to achieve zero-shot online customization in more practical scenarios.