SlowGuess/NeurIPS_2023_OCR huggingface dataset:
N.8 Stationary and non-stationary policies
One important ablation for CSIL is evaluating the performance with standard MLP policies like those used by the baselines. We found that this setting was not numerically stable enough to complete an ablation experiment. Due to the spurious behavior (e.g., as shown in Figure 3), the log-likelihood values used in the reward varied significantly in magnitude, destabilizing learning significantly, leading to policies that could no longer be numerically evaluated. This result means that stationarity is a critical component of CSIL.
N.9 Effectiveness of constant rewards
Many imitation rewards typically encode an inductive bias of positivity or negativity, which encourage survival or 'minimum time' strategies respectively [40]. We ablate CSIL with this inductive bias for survival tasks (Hopper-v2 and Walker-v2) and minimum-time tasks (robomimic). Figure 47 shows that positive-only rewards are sufficient for Hopper-v2 but not for Walker-v2. Figure 48 shows that negative-only rewards is effective for the simpler tasks (Lift) but not the harder tasks (NutAssemblySquare).
N.10 Importance of behavioral cloning pre-training
Figure 49 shows CSIL without any policy pre-training, so the reward is learned only though the finetuning objective. CSIL without pre-training does not appear to work for any of the tasks. This poor performance could be attributed to the lack of hyperparameter tuning, since the reward learning hyperparameters were chosen assuming a good initialization. It could also due to the maximum likelihood fitting of the coherent reward, since the refinement does not use the 'faithful' objective (Section D) that improves the regression fit.
Figure 46: Normalized performance of CSIL for online MuJoCo Gym without regularizing the critic Jacobian. Uncertainty intervals depict quartiles over ten seeds. The benefit of this regularization is somewhat environment dependent, but clearly regularizes learning to the demonstrations effectively in many cases. Performance reduction is likely due to underfitting during policy evaluation due to the regularization.  $n$  refers to demonstration trajectories.
Figure 47: Online MuJoCo Gym tasks with absorbing states where CSIL is given a constant  $+1$  reward function. Uncertainty intervals depict quartiles over ten seeds. While this is sometimes enough to solve the task (due to the absorbing state), the CSIL reward performs equal or better.  $n$  refers to demonstration trajectories.
Figure 48: Online robomimic tasks where CSIL is given a constant -1 reward function. Uncertainty intervals depict quartiles over ten seeds. While a constant reward is sometimes enough to solve the task (due to the absorbing state), the CSIL reward performs roughly equal or better, with lower performance variance.  $n$  refers to demonstration trajectories.
Figure 49: Online MuJoCo gym tasks where CSIL has no BC pre-training. Uncertainty intervals depict quartiles over 5 seeds.  $n$  refers to demonstration trajectories.