SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The results clearly show that the non-Riemannian method struggles to reach a success rate of 1 for the reaching task and the collision-avoidance task. Furthermore, we observe a high variance over different runs in the same settings (see Fig. 9 and Fig. 10). We attribute this to the fact that the our method takes exact gradient steps in the direction of steepest descent w.r.t. the underlying BW metric, whereas the implicit scheme only approximates this direction. For this reason the non-Riemannian method is much more noisy, which in turn leads to the aforementioned high variance. Nevertheless, the multiple-goal task constitutes an exception. Here we observed a similar performance for our approach and the ablated method. The reason for this is that the optimization of this task is mainly dominated by the weight updates, which are identical for both methods. This result is therefore expected and confirms that correctness of our ablation strategy.
Figure 9: Variance of the success rate over 5 runs for our method (WGF) and the ablated method (non-BW) on the reaching task (left), the collision avoidance task (middle) and the multiple-goal task (right). The violin plots are overlaid with box plots, quartile lines and a swarm plot, where dots indicate the success rates of individual runs. The time steps at which we determined the variance are 80000, 90000, 85000.
Figure 10: Variance of the success rate over 5 runs for our method (WGF) and the ablated method (non-BW) on the reaching task (left), the collision avoidance task (middle) and the multiple-goal task (right). The violine plots are overlaid with box plots, quartile lines and a swarm plot, where dots indicate the success rates of individual runs. The time steps at which we determined the variance are (80000, 400000), (90000, 200000), (85000, 90000).
A.6.3 Additional experiments with 7-DoF robotic manipulator
We carried out two additional experiments to show that our method can be employed on tasks performed by off-the-shelf robotic manipulators (e.g. a 7-DoF Franka Emika Panda robot). Specifically, we extended the collision-avoidance task described in § 4 to a 3D environment, where the state and action depend on the task space representation. The first task was represented in the robot operational space, i.e., the state  $s = x \in \mathbb{R}^3$  and the action  $\pmb{a} = \dot{\pmb{x}} \in \mathbb{R}^3$ . The second task was represented in the robot joint space, consequently the state  $s = q \in \mathbb{R}^7$  and the action  $\pmb{a} = \dot{\pmb{q}} \in \mathbb{R}^7$ . This experiment was aimed at assessing the capabilities of our approach to adapt robot motion policies in state-action spaces of higher dimensions. The initial 3-components GMM policy was trained using 10 human demonstrations featuring linear reaching 3D trajectories. For policy optimization, we used a sparse reward defined as a function of the position error between the robot end-effector position and the target at the end of the rollout. Moreover, two sparse penalty terms were added to punish collision with obstacles and divergent trajectories.
Similarly to the planar task reported in the main paper, we tested whether our method was able to adapt a trajectory tracking skill in order to avoid collisions with newly added obstacles. This means that the robot end-effector needed to pass through a narrow path between two spherical obstacles. For the operational space representation, the robot end-effector pose was controlled using a full-pose Cartesian velocity controller at a frequency of  $100\mathrm{Hz}$ , where the end-effector orientation was kept constant. For the joint space representation, the robot joint configuration was controlled using a joint velocity controller at a frequency of  $100\mathrm{Hz}$ . Figure 11 shows the results for 3D operational space representation, where our method reached a success rate of 1.0 very quickly, taking approximately
Figure 11: The success rate of our method applied to the 3D narrow-path task performed by the 7-DoF Panda robotic manipulator. The shaded area indicates the standard deviation over 5 runs.
Figure 12: The success rate of our method (WFG) and the baselines on a 3D narrow-path task performed by a simulated 7-DoF Panda robotic manipulator. The task is represented in the robot joint space, therefore the state  $s = q \in \mathbb{R}^7$ , and the action  $\pmb{a} = \dot{\pmb{q}} \in \mathbb{R}^7$ , correspond to the joint position and velocity, respectively. The shaded area indicates the standard deviation over 15 runs.
20000 environment steps<sup>3</sup>. Moreover, the solution variance of our method was also very low, which is consistent with our observations concerning the performance of our policy optimization on the three planar tasks analyzed in the main paper. Figure 12 shows the results of the collision avoidance task using the joint space representation. As observed, our approach was able to adapt the robot motion policy so that the robot end-effector safely passes through a narrow path defined by two spherical obstacles (the narrow-path task description is provided in Sec. 4.1). It is evident that our approach outperformed all the baselines in this simulated robotic task, providing evidence that our approach scales and it is able to adapt robot motion policies in higher-dimensional tasks.
A.6.4 Initial GMM policies
For the sake of completeness, Fig. 13 provides 2D projections of the initial GMM policies learned from demonstrations for the three robotic settings considered in the main paper: the reaching motion skill, the collision-free trajectory tracking, and the multiple-goal task. Figure 13 also provides the demonstration data used to train the initial policies. Note that these models are then adapted according to the policy optimization approach introduced in § 3.2.
<sup>3</sup>As the baselines underperformed in the 2D case, they were not tested in this specific setting.
(a) Reaching motion skill
(b) Collision-free trajectory tracking
(c) Multiple-goal task
Figure 13: Green Gaussian components (  $\mathbf{\Theta}$  ) represent the initial GMM policy learned from demonstrations, projected on the Cartesian position (left) and velocity (left) spaces. The recorded position and velocity data are depicted as black dots  $(\bullet)$ .