SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Constraints. For the setting with constraints, we apply box constraints for the cart position and controls. Specifically, we enforce  $|y| \leq y_{\mathrm{max}}$  and  $|u| \leq u_{\mathrm{max}}$ . We let constraint parameters  $\theta_{\mathrm{constr}} = [y_{\mathrm{max}}, u_{\mathrm{max}}]^{\top}$ . The final parameter vector is given by  $\theta = [\theta_{\mathrm{dyn}}^{\top}, \theta_{\mathrm{cost}}^{\top}, \theta_{\mathrm{constr}}^{\top}]^{\top}$ .
A.2 6-DoF Quadrotor Manipowering
See Section E of the appendix in [20] for the full definition of the quadrotor maneuvering problem. The objective is to drive the quadrotor using propeller thrusts to a goal configuration. The state is given by  $[p^{\top},\dot{p}^{\top},q^{\top},\omega^{\top}]^{\top}\in \mathbb{R}^{13}$ , where  $p\in \mathbb{R}^3$  represents the position of the center-of-mass of the quadrotor,  $\dot{p}\in \mathbb{R}^3$  represents linear velocity,  $q\in \mathbb{R}^4$  is a unit quaternion representation of the quadrotor attitude and  $\omega \in \mathbb{R}^3$  represents angular velocity. For the inequality constrained setting, non-linear (squared-norm) constraints on quadrotor position, given by  $\| p\| _2^2\leq r_{\mathrm{max}}$  and thrust limits are applied.
A.3 2-Link Robot Arm
See Section E in the appendix in [20] for a reference to the full description of the robot arm problem. The objective of this OC problem is to move the robot arm to a goal configuration. The state is given by the orientation of the base and elbow joint, as well as respective velocities  $[q_1, q_2, \dot{q}_1, \dot{q}_2]^\top \in \mathbb{R}^4$ . The control inputs  $u = [u_q, u_2]^\top \in \mathbb{R}^2$  relate to applying torques at each joint. Constraints for the inequality constrained setting relate to box constraints on the joint positions, as well as torque limits. The cost function is similar to cartpole, i.e., a quadratic penalty to a goal state and over the controls.
A.4 6-DoF Rocket Landing
See Section I in the appendix in [20] for the full definition of the 6-DoF rocket landing problem. The objective of this task is to land a rocket modeled as a rigid body (softly) onto a goal position, using thrusters located at the tail. A non-linear constraint over the tilt angle is applied in the inequality constrained setting. The cost function penalizes deviations from the goal configuration, as well as fuel cost.
Figure 6: Results for the imitation learning task with no inequality constraints present. The y-axis is the (percentage) relative difference in the imitation loss, defined to be (IDOC loss - PDP loss) / PDP loss (lower is better). For Cartpole, locations where the PDP gradient fails is marked by stars. Shaded regions indicate min/max values over five trials.
(b) 6-DoF Quadrotor
(c) Robotarm
(d) 6-DoF Rocket
B Experimental Setup for Synthetic Benchmark
We discuss in more detail how blocks for  $H, A, B, C$  are generated in this section. Elements for the Jacobian and Hessian blocks for the objective function and constraints are sampled independently and identically distributed (i.i.d.) from a standard Gaussian. Hessian blocks are made symmetric using  $H^{\mathrm{sym}} = (H + H^{\top}) / 2$ , where  $H$  is the initially generated random matrix. Each block's condition number is modified by applying an SVD to each block and adjusting the diagonal entries. Elements of the downstream loss gradients  $v = \mathrm{D}_{\xi} \mathcal{L}(\xi)^{\top}$  are also sampled i.i.d. from a standard Gaussian.
C Additional Results for the Imitation Learning/LfD Tasks
In this section, we provide additional detailed analysis around the imitation learning/LfD setting with no inequality constraints. Figure 6 illustrates the results of these experiments. We plot the percentage difference in loss across the full training curve between IDOC and PDP for five random initializations for  $\theta$ . For the quadrotor, robotarm and rocket environments, we observe almost no difference in learning curves. This is expected since IDOC and PDP are computing the same trajectory derivative. For cartpole however, we see that PDP fails catastrophically (resulting in undesirable spikes in the imitation loss) on two occasions across the five trials. We suspect this is due to numerical instability of the Riccati equations, arising from the local geometric properties of the dynamics at solved trajectories.
D Additional Timings for Simulation Experiments
In this section, we provide additional compute time experiments for the imitation learning/LfD tasks evaluated in the simulation environments. Given the relatively small size for the COC problems, there is very limited advantage with respect to compute time for parallel evaluation of trajectory derivatives. However, as a proxy to parallelization, IDOC can be vectorized across timesteps using the numerical linear algebra library Numpy [16]. All experiments in this section are run on a single thread of an AMD Ryzen 9 7900X 4.7Ghz desktop CPU.
The experimental setting where inequality constraints are present is challenging to vectorize compared to the setting with only equality constraints, because different timesteps may have different numbers of active inequality constraints. While matrices  $A$  and  $C$  presented in Equation 4 are still block structured, the blocks may not be of uniform size in this setting. We present in Figure 7a and 7b the mean computation time (along with the upper/ lower bound) per iteration across five trials with 10k iterations for each trial for all environments except for rocket (1k). We observe around a  $2\times$  speedup over PDP by computing our IFT/DDN gradients, even before direct computation of vector-Jacobian products. Computing VJPs directly yields a further (approx.)  $10\%$  saving in compute time over constructing the trajectory derivative explicitly.
In our non-optimized Python implementation of IDOC, we batch together and vectorize computations involving blocks with an identical number of active constraints. As expected, trajectory derivatives for inequality constrained problems are slightly slower to compute compared to the equivalent without (hard) inequality constraints (log-barrier approximation), due to the necessary computational overheads required for identifying and batching blocks.
Note that Safe-PDP [22] (for hard inequality constraints) and PDP [20] (used for differentiating through log-barrier problems) derivatives are implemented using a custom LQR and equality constrained LQR solver, respectively. These solvers have markedly different implementations, which explains differences in computation time between Safe-PDP and PDP.
(a) Hard Constraints
(b) Log-Barrier
Figure 7: Computation times for trajectory derivatives (per trajectory, per iteration) for the CIOC task. As expected, the log-barrier method with no "hard" inequality constraints affords faster derivative computation.