SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use the Tianshou [Weng et al., 2022] implementation of PPO [Schulman et al., 2017] as our baseline policy optimizer. Classical Lagrangian methods exactly follow the gradient update shown in Algorithm 1, but they are susceptible to oscillating learning dynamics and constraint-violating behavior during training. The PID Lagrangian method [Stooke et al., 2020] extends the naive gradient update of  $\lambda_{i}$  with a proportional and derivative term to dampen oscillations and prevent cost overshooting.
We find that augmenting the policy state with the raw value of the constraint function to be crucial for successful policy training. For CRL, this corresponds to the violation of the ground-truth constraint.
B.2 Expert Demonstrations
For all of our experiments, our experts are CRL policies trained to satisfy a ground truth constraint. The expert demonstrations we use in ICL are generated by rolling out these policies and collecting 20 trajectories. To simulate suboptimal experts, we generate noisy trajectories by adding zero-mean Gaussian noise (with standard deviation  $0.7 / 0.5$  for velocity / position) to the policy's action at every timestep.
B.3 Single-Task ICL
Similar to CRL, we augment the policy state with the raw value of the constraint function. For ICL, this corresponds to the violation of the learned constraint.
To reduce the amount of interaction required to solve the inner optimization problem, we warm-start our policy in each iteration by behavior cloning against the given expert demonstrations. We zero-out the augmented portion of the state when behavior cloning to avoid leaking constraint violation information from the expert.
When using CRL as part of ICL, we set the constraint threshold used in the Lagrangian update to be the expert's constraint violation. However, when starting with degenerate constraints, this can prevent policy optimization from learning at all as the expert's violation under the constraint can be arbitrarily low. To circumvent this issue, we set the Lagrangian constraint threshold to use the expert's violation plus a cost limit buffer, which we anneal over the course of training to 0. This ensures that our learned policy satisfies the learned constraint as much as the expert does as desired.
Because ICL requires learning a constraint, we represent our constraints as neural networks, mapping from the state space of our agent to a bounded scalar in the range  $[0, 1]$ . To update this constraint, we solve the optimization problem using a regression objective. Learner and expert constraint values are labeled with 1 and -1 respectively, and we optimize a mean squared error loss. We found that weighting data equally (instead of via returned Lagrange multipliers) was sufficient for good performance and therefore utilize this simpler strategy.
For both CRL and ICL, we find that using a log-activation on top of the raw value of the constraint is an important detail for stable training.
B.4 Multi-Task ICL
For the multi-task maze setting, we consider 10 distinct tasks corresponding to unique goal locations, each with 2 starting locations.
Our solver combines a waypoint planner with a low-level controller. Waypoints are calculated by discretizing the maze into a 10 by 10 grid and running Q-value iteration, while low-level actions are computed using CRL experts trained to move in a single cardinal direction. To generate a trajectory, we follow a sequence of waypoints from the planner using the controller.
For ICL, we visualize our learned constraints by uniformly sampling 100,000 points from every grid cell, averaging the constraint predictions, and thresholding at a limit of 0.5.
B.5 Hyperparameters
The key experimental hyperparameters are shown in Table 1. The exact configuration we use for our experiments is available at https://github.com/konwook/mtic1/blob/main/mtic1/utils/config.py.
Hyperparameter	Value
PPO Learning Rate	0.0003
PPO Value Loss Weight	0.25
PPO Epsilon Clip	0.2
PPO GAE Lambda	0.97
PPO Discount Factor	0.99
PPO Batch Size	512
PPO Hidden Sizes	[128, 128]
P-update Learning Rate	0.05
I-update Learning Rate	0.0005
D-update Learning Rate	0.1
Constraint Batch Size	4096
Constraint Learning Rate	0.05
Constraint Update Steps	250
Steps per Epoch	20000
CRL Velocity Epochs	50
CRL Position Epochs	100
ICL Expert Demonstrations	20
ICL Velocity Cost Limit	20
ICL Position Cost Limit	100
ICL Anneal Rate	10
ICL Velocity Outer Epochs	20
ICL Position Outer Epochs	10
ICL Velocity Epochs	10
ICL Position Epochs	50
Table 1: Experiment hyperparameters.
C Comparison to Chou et al. [2020]
C.1 Experimental Setup
We attempt to faithfully implement the method of Chou et al. on the problems we consider as a baseline for single-task ICL. At a high level, their method requires 2 steps:
1. For each expert trajectory, perform a random search in trajectory space starting from the demo to try and compute the set of trajectories that are higher reward than the demo.
2. Solve a constrained optimization problem over a known parametric family that labels each expert trajectory with  $+1$  and each learner trajectory with 0.
Note that in contrast to our method, Chou et al. perform a single constraint estimation, rather than an iterative procedure.
For Step 1), Chou et al. [2020] use hit-and-run sampling that ensures good coverage over this set of trajectories in the limit of infinite sampling. However, for tasks of the horizon  $(1000+)$  and state-space dimension (30) we consider, one would need a rather large number of samples to ensure good coverage (i.e. an  $\epsilon$ -net would require  $\frac{30^{1000}}{\epsilon}$  samples).
As search in the space of trajectories is infeasible, we instead search in the space of policies, as is much more standard on long-horizon problems. We therefore initialize the policy with behavioral cloning (similar to starting from the expert trajectories), performing maximum entropy RL (a search procedure with good coverage properties), and return the set of trajectories from the replay buffer that are of a higher reward than that of the expert.
For Step 2), Chou et al. [2020] use a variety of mixed-integer LP solvers. In essence however, they are trying to solve a classification problem. Of course, with their assumption of an optimal expert (or a "boundedly suboptimal" expert with a known sub-optimality gap), they can solve the strict feasibility problem. Because we assume that the expert is safe in expectation (rather than uniformly), it is natural to consider a relaxation of this problem (i.e. a convex relaxation of the  $0/1$  loss).
We experimented with several such relaxations when developing our method and found the  $\ell_2$  relaxation (i.e. treating the problem as a regression problem with different targets for learner and expert demos) to work the best in practice. We therefore use the same relaxation for their method. For our single-task experiments, it is clear how to specify the "parametric family" the constraint fits in (e.g. a linear threshold), so we believe the above is a faithful implementation.
So, in short, we perform the same constraint learning procedure over a different source of learner data.
C.2 Baseline Comparison Results
We implemented the above method on our velocity and position tasks and display the results (averaged over 3 seeds) below.
Algo.	|c*-c| (↓)	J(πE,r) - J(π,r) (↓)	J(πE, c*) - J(π, c*) (↑)
ICL (iter 1)	0.052	569.277	270.111
ICL (iter 10)	0.000	-72.149	133.59
Chou et al.	0.311	384.772	-1958.372
Table 2: Baseline Comparison (Position Constraint)
Algo.	|c*-c| (↓)	J(πE,r) - J(π,r) (↓)	J(πE, c*) - J(π, c*) (↑)
ICL (iter 1)	0.249	12.267	-244.25
ICL (iter 10)	0.006	-46.771	-21.602
Chou et al.	0.246	-28.315	-134.58
Table 3: Baseline Comparison (Velocity Constraint)
We see that our method performs better in terms of all 3 performance criteria we evaluate under. As implemented, Chou et al.'s method appears to produce an inaccurate constraint that leads to a significant safety violation (the bottom right cell in both tables).