On-Robot Learning With Equivariant Models
Authors: Dian Wang
Mingxi Jia
Xupeng Zhu
Robin Walters
Robert Platt
Venue: CoRL 2022 Poster (2022)
Keywords: Manipulation
Reinforcement Learning
Equivariance

Abstract: Recently, equivariant neural network models have been shown to improve sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which a policy must be learned entirely on a physical robotic system without reference to a model, a simulator, or an offline dataset. We focus on applications of Equivariant SAC to robotic manipulation and explore a number of variations of the algorithm. Ultimately, we demonstrate the ability to learn several non-trivial manipulation tasks completely through on-robot experiences in less than an hour or two of wall clock time. 

On-Robot Learning With Equivariant Models
Dian Wang
Mingxi Jia
Xupeng Zhu
Robin Walters Robert Platt
Khoury College of Computer Sciences
Northeastern University
Boston, MA 02115, USA
[wang.dian, jia. ming, zhu.xupa xup, r. walters, r p.p.aat/enoorhhhsttemm..m
Recently, equivariant neural network models have been shown im- prove sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which policy must be learned entirely on a physical robotic system without reference to a model, simulator, or an offline dataset. We focus on applications of Equivariant SAC to robotic manipulation and explore number of variations of the algorithm. Ultimately, we demonstrate the ability to learn several non-trivial manipulation tasks completely through on-robot experiences in less than an hour or two of wall clock time.
Abstract:
Introduction
Training directly on a physical robot is challenging because it takes long time to gather experiences: an environmental step a physical robot system is often at least one or two orders of magnitude slower than an environmental step the simulation. As a result, it is not unusual for researchers who want to learn physical robotic system to spend hundreds of hours of robot time to learn even simple manipulation control policies, c.g. [1, 2, 3J. However, recent work has shown that policy learning using equivariant models with rotation, translation, and reflection symmetries yield much higher sample efficiency than conventional approaches [4, 5].
This paper demonstrates that with the right choices for symmetry group and data augmentation strategy, these equivariant approaches can be so sample efficient that it becomes feasible learn simple robotic manipulation policies directly on a physical system (we call this on-robot learning). This newfound ability to learn simple manipulation skills quickly and directly on robot also gives us new perspective on the problem of the sim2real gap, the small differences between simulation of the real world and the real world itself [6]. Our results suggest that it is not always worthwhile to train policy in simulation first before fine-tuning in the real world, least in the context of simple manipulation tasks where equivariant models can learn quickly.
This paper makes three contributions. First, we find that equivariance with respect to discrete sym- metry groups leads to better performance than equivariance with respect to continuous groups. Al- though prior work [4, 5] makes it clear that equivariant policy learning can be much more sample efficient than learning with non-equivariant models, not clear what symmetry groups are most appropriate in robotic domains with rotation and reflection symmetries. Based on recent work, we know we can encode continuous (O( and O(2) symmetries using the irreducible representations of the group [7]. However, approximating continuous symmetry using discrete subgroups may also work well. This paper evaluates these alternatives and finds that even though the continuous group more closely reflects the actual problem symmetries, discrete rotation and reflection groups like DA and Cg still have better performance.
Second, we show data augmentation further improves models equivariant to discrete groups. Since equivariant models hard code problem symmetries into the neural network model, one might assume that data augmentation would no longer be helpful. In the case of discrete groups, which only approximate the full domain symmetry, this turns out to untrue. Here, we evaluate equivariant policy learning with and without various types of data augmentation and find that even something as simple as buffer augmentation much improves performance.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Third, we perform a series of evaluations of on-robot learning using equivariant methods. We show that the equivariant models are so sample efficient that they can learn policies for solving various ma- nipulation tasks from scratch within one or two hours. Furthermore, we demonstrate that sim2real pre-training unnecessary for equivariant policy learning and is sometimes harmful. Since equiv- ariant policy learning makes it possible to learn simple manipulation policies directly on a physical robot efficiently, it is worth asking whether the sim2real approach is still useful in these applica- tions. We compare training exclusively on the robot with sim2real strategy where pre-train in a PyBullet simulation and then transfer onto the physical robot. We find in the four repre- sentative manipulation applications explored here that while there is often a benefit to training in simulation first before transferring to the robot, this is by no means necessary. Moreover, it is sometimes the case that the simulation and physical agents learn qualitatively different things, potentially leading to negative sim2real transfer, i.e., the situation where the pre-trained policy ac- tually impedes learning on the physical system. Supplementary video and code are available at https: / po ntw. gith.b. io/ requiqii_ http:a/io///p_pooo___pepee _robot_pagee....
2 Related Work
Equivariant Learning: The first equivariant neural networks introduced were G-Convolution [8]
and Steerable CNNs [9], which improved the sample efficiency of traditional convolutional neu-
ral networks by injecting symmetries in the structure of the neural network. Weiler and Cesa [7] proposed a framework for implementing general E(2)-Steerable CNNs. Recent work showed en- couraging results for applying equivariant networks in various computer vision [10, 11] and dy-
namics [12, 13] tasks. They have also been applied to deep RL [14, 4] and robotic manipula- tion [15, 5, 16, 17] with compelling results. However, to our knowledge, equivariant methods have never been explored in the context of on-robot reinforcement learning.
On-Robot Learning: The most common approach robotic policy learning is to train in simulation and then transfer to a real world application [18, 19, 20, 21, 22). Nevertheless, there have been several efforts develop methods that enable agent to learn policy directly physical robotic system. Gu et al. [23] trained manipulation skills in fixed environments with multiple physical robot workers. Singh et al. [24] developed method that learned manipulation skills within 1-4 hours in the real world but required a user to respond to queries for labels. Kalashnikov et al. [2] trained a grasping policy with seven robots and over 800 robot hours. Zeng et al. [25, 26] demonstrated on-robot learning by encoding the Q function using fully convolutional network, but only in the context of open-loop tasks where the gripper performed pre-defined behavior. FERM [27] performed on-robot learning using SAC [28] in combination with contrastive learning objective [29, 30], but only for tasks where the orientation of the gripper was fixed. Relative to the work above, our method is most comparable to FERM [27], and we therefore benchmark our method against that.
Background
Equivariance Over the Rotation Group: Many robotics problems display rotational and reflec- tional symmetry in the plane perpendicular to gravity. These are captured by the group O(2) which contains all continuous planar rotations Rot, about the origin and reflections through lines through the origin. It contains the subgroup of rotations SO(2) = {Rot : 0 < 0 < 2x).. Sometimes, we are interested in discrete subsets, for example the cyclic subgroup C,, = [Rot. : 0 E { 2 0 < i < n)) or the Dihedral group D, which contains the n rotations of C, as well as reflections through n evenly spaced lines through the origin. Domain symmetries can be described as invariance or equivariance of task functions. A function f is G-invariant if when its input is transformed by a symmetry group element g E G, its output stays the same, fggz = fxx) A function f is G-equivariant if when its input is transformed by symmetry group element g G. its output transforms accordingly by 9. f(zz) = (z).
n
Group Invariant MDPs: Equivariant policy learning uses symmetries in the MDP to structure the neural network model used to represent the policy and value function. Let M = (S, A, T, R) denote an MDP and let G denote an element of a symmetry group G (e.g., G G= 0(2)). We will say that MDP M is G-invariant if both the transition function and the reward function are invariant: T(s, a, s')
Tggs, ga, gs') and R(s, a) R(gs, ga) for all g E G [5]. This type of symmetry is

---

good fit for robotics problems that are invariant over rotation and reflection. In many manipulation problems, for example, the objective is perform some task (e.g., open drawer or insert part) regardless of the respective poses of the parts involved.
Equivariant SAC: Equivariant Soft Ac- tor Critic (Equivariant SAC) [5] is ver- sion of SAC [28] that uses equivariant neural models to encode the symmetries of G-invariant MDP. implements the critic using an G-invariant model, ((gs, ga) g(s, a). and the actor us- ing an G-equivariant model, Tggs) = gr( This is illustrated in Figure for a manipulation domain where G = SO(2), state is an image = Fs, and action is vector
R
90
90'
900
90'
aloke.y
+
8f,
ga
sf,
(z, U, X), where (x, y, 2) is the gripper position displacement, 0 is the gripper orienta- tion displacement about the z-axis, and A is the gripper aperture. The action is partitioned into (x,y) E Aequi and (z, 0, Aine that ga rotates the (I, y) component action but leaves the other action dimensions unchanged. Figure left shows the equivariance of the actor. When the state image rotates by 90 degrees, the x. y components of action rotate but the 2, 0, X components remain unchanged. The right side shows the invariance of the critic. Corresponding rotations of state and action do not change the output Q value.
(a) Equivariant Actor
(b) Invariant Critic
Figure I: Illustration of the Equivariant SAC. (a): the equivariant actor's output action rotates as the input state rotates. (b): the invariant critic's 's output doesn't t change when the input state and action are rotated simultane- ously.
4 Symmetry Group and Augmentation Strategy
The specific choice of symmetry group and data augmentation strategy have a significant impact on the sample efficiency of the algorithm. Before evaluating our algorithms on the robot, we evaluate those different algorithmic choices in simulation. Here, we experiment in the context of Equivariant SAC for the four tabletop manipulation tasks shown in Figure 2.
All tasks have sparse rewards, i.e., +1 reward for reaching the goal, and 0 otherwise. We use a 2-channel image as the observation. The first channel is a top-down depth image centered with respect to the robot gripper. The gripper is drawn at the center of the depth image with its current aperture and orientation. The second channel is a binary channel (i.c., the values of all pixels are either O 1), indicating if the gripper is holding an object. The action space is: rr V. E [-0.05m, 0.05m]] 0 E |- 7: T] A E [0,1] (0 means fully close and means fully open). 20 episodes of expert demonstration are added to the replay buffer before the start of training (see the ablation study about the effect of expert demonstrations in Appendix G). See the detailed description of the environments in Appendix B and the training details in Appendix D.
4.1 Choice of Symmetry Group
In Equivariant SAC, we must select a symmetry group with which to parameterize the actor and critic models. This symmetry group reflects the underlying symmetry that we assume exist in the problem. One might expect that larger symmetry groups would always be better because they would enable the model to generalize to a greater number of different situations. However, as shown in Weiler and Cesa [7]. a larger group does not necessarily improve the performance of the model. It therefore important to ask which symmetry groups are most helpful in our robotic manipulation domains. We compare the performance of Equivariant SAC when the actor/critic models are pa- rameterized by each of the following five different symmetry groups: 1) Cs: the cyclic group that encodes discrete rotations every 45 degrees; 2) Dy: the Dihedral group that encodes rotations every 90 degrees and reflection; 3) D3: the Dihedral group that encodes rotations every 45 degrees and reflection; 4) (O( the group of continuous planar rotations; 5) 0(2): the group of continuous pla- nar rotations and reflections. In all baselines, we use data augmentation in the replay buffer where four extra transitions with random SO(2) is generated for each experienced transition.
3

---

(a) Block Picking
Clutter Grasping
Block Pushing
(d) Block in Bowl
(e) Block Picking
(f) Clutter Grasping
(g) Block Pushing
(h) Block in Bowl
Figure 2: (a)-(d): Our simulation environments implemented in PyBullet [31]. The left images in each environment show the initial state of the environment; the right images in each environment show the goal state. (e)-(h): Our on-robot learning environments.
Block Picking
Clutter Grasping
Block Pushing
Block Bowl
Discourted
24
04
training steps
CB
D6
(12)
S012)
(a) Comparison of Equivariant SAC defined with different symmetry groups.
Block Picking
Clutter
Block Pushing
Block Bowl
1.2
30
Diecounted
02
30
2000
7500
10000
05
2000
5000
7500
13000
250
3000
7500
1000
00
2500
1000
750
3000
Number of training steps
aug
(b) Comparison of Equivariant SAC equipped with different data augmentation techniques.
Figure 3: The plots show the performance of the behavior policy in terms of the discounted reward. Each point the average discounted reward the previous 500 steps. Results are averaged over four runs. Shading denotes standard error.
Figure 3a shows the result. In all four environments, Da and Da show the best performance the five different groups in terms of convergence speed and converged performance, where D2 has marginal improvement over D3 even though D& encodes more rotations than D4. addition, we find that incorporating reflection symmetry generally improves the performance (comparing D4 vs Cg and O(2) SO(2)). However, both the SO(2) and O(2) networks underperform the Da network. We hypothesize that this is because the D4 network has access to the regular representation of the symmetry group as the hidden layer of the network, whereas the models defined over the continuous groups (2( and O(2) do not. The regular representation explicitly encodes the feature maps over all elements the finite group concurrently, making an informative representation for the hidden layers. Moreover, the regular representation is compatible with component-ve wise activation functions and component-wir max pooling, which are most commonly used and known to work well in deep networks.

---

4.2 Choice of Data Augmentation Strategy
The results of Section 4.1 indicate that the equivariant models that perform best in our domains use the D C ((2) symmetry group, the group of 90 degree rotations and reflections. As result, it could still make sense to use data augmentation to enable the model learn rotational symmetry within a continuous range (0, )). Here, we compare three different approaches to data augmentation with random rotations: SAC + buffer aug, SAC + aux loss, and SAC+ + rot RAD. SAC + buffer aug, add four extra transitions with random S2(2) rotations to the replay buffer for each experienced transition. (See Appendix for an comparison of different amounts augmentation.) In SAC + aux loss, add extra loss terms to encourage the model to learn continuous rotational equivariant representations: Laux"" 1 9 SO(2) and (F..a) is the sampled state-action pair in the minibatch. In SAC not RAD, use RAD [32] to perform rotational data augmentation at each sample and training step. Figure 3b shows the comparison between the three data augmentation approaches and vanilla Equivariant SAC (green). Even though Equivariant SAC already encodes D4 equivariance within the structure of the network, adding S2( data augmentations to the algorithm can still help by substantial margin, especially in the more challenging tasks like clutter grasping. block pushing, and block in bowl. However, notice that all three data augmentations approaches perform similarly. As such, we adopt the buffer augmentation method in the remainder of this paper because it is the simplest method.
2 (gFF - (F))) and Canx = 2 (q(9F.. ga) q(Fe.. a))?. where
5 On-Robot Learning
Figure 4 illustrates our on-
robot learning setup. We use a Universal Robots URS arm equipped with a
Depth Camera
Depth Camera
Acquiring
Robotiq 2F-85 parallel-jaw gripper. Since we need to be able to see the ob- jects on the table beneath the robot hand and arm, we mount two depth sensors that view the scene from or- thogonal directions each 45 degrees with respect to the bins (see Figure 4 left). One of these sensors is an Occipital Structure Sensor and the other is Microsoft Azure Kinect DK. The out- put of each sensor is con- verted to a partial point cloud, fused into a single combined point cloud, and projected into a depth image viewed from top-down direction (Figure 4 right). We use the same 2-channel observation as in Section 4 including depth channel and binary channel for the gripper state. Our workstation has Intel Core i7-9700k CPU (3.60GHz) and Nvidia RTX 2080Ti GPU. All together, this setup enables us to train at a rate of approximately 1.1 seconds per environmental transition. This includes the time it takes to take single transition the physical robot well to perform single SGD) step in the model (we do this in parallel with the robot motion), but it does not include the time it takes to reset the environment between episodes.
Orthographic Projection
Work Space
Our experimental set up for on-robot learning. The obser- vation (bottom right) generated by first acquiring point clouds from two depth cameras above the workspace then creating an orthographic projection at the gripper's position. The gripper is drawn at the center of the observation (in yellow) with its current aperture and orientation.
Figure 4:
5.1 Physical Tasks
All physical experiments are performed using the same four tasks described in Section 4 (see Fig- ure 2) ! In tasks involving only a single object (i.e., Block Picking and Block Pushing), the robot uses only one bin as the workspace. In tasks involving multiple objects (i.e., Clutter Grasping and Block in Bowl), the robot alternately uses one of the two bins as the active workspace and then uses
Note that in Block Pushing, we use a virtual goal drawn on the observation, so the goal is visible to the agent but not the human observer.
5

---

Task
Block Picking
Clutter Grasping
Block Pushing
Block in Bowl
Number of training steps
2000
2000
2000
4000
Approximate time for training
45 mins
45 mins
hr
hrs 40 mins
Evaluation success rate
100% (50/50)
96% (48/50)
92% (46/50)
92% (46/50)
Sim2real transfer success rate
(50/05)))0)
86%(43/50)
70%(35/5))
72%(36/50)
Top: the number of training steps, approximate time for training, and the evaluation success rate of the trained policy of on-robot learning. Bottom: the evaluation success rate of the model trained in simulation. All succes rates are averaged over 50 episodes.
Table I:
the other bin reset the environment. We implement an automated resetting process for all four environments. In Block Picking and Block Pushing, the robot will reset the environment through picking up the block and randomly placing it inside the workspace. In Block in Bowl, the robot will move both the bowl and the block from the active workspace random positions in the reset bin, then switch the active workspace to the reset bin. In Clutter Grasping, after the robot successfully grasps one object from the active workspace, it will drop the object to random position in the reset bin. Once the robot grasps all objects from the active workspace, the robot will switch the active workspace to the reset bin. See Appendix C for more details.
5.2 Performance of the On-Robot Learned Policy
First, we evaluate the ability of Equivariant SAC to learn these tasks entirely in the on-robot setting. iee, entirely on the robot without pre-training. Based on the findings described in Section 4, we configure both our actor and critic models use the D4 symmetry group, and we use the "SAC buffer aug" strategy described in Section 4.2. The blue line in Figure 5 shows the learning curve of training, and the third row of Table I shows the greedy performance of the learned policy after training, averaged over 50 test episodes.
In Block Picking, the robot succeeded in all 50 trials. In Clutter Grasping, the robot failed to find an appropriate grasp point in two out of the 50 episodes. In Block Pushing, the robot failed in four out of 50 episodes. In two of these, it failed move towards the block. In the other two failures, the robot kept pushing down from the top of the block and triggered the URS safety system. In Block in Bowl, the robot failed in four out of the 50 trials. In three of these, the robot grasped the block but did not move towards the bowl. In the other failure, the robot grasped the bowl and the block together in single grasp and failed to let go.
In Block Picking, Clutter Grasping, and Block Pushing, training lasts 2000 transitions. In Block in Bowl, training lasts 4000 transitions because it is a harder task to learn. In terms of wall clock time, training tasks a total of 45 minutes in Block Picking and Clutter Grasping; one hour in Block Pushing (because of the additional time required to physically reset that environment); and 2 hours and 40 minutes in Block in Bowl. For Block in Bowl, this time can be decomposed into approximately 73 minutes (1 x 4000) of environmental steps and 107 minutes to reset the environment.
5.3 Baseline Comparison
There are very few methods in the literature that can learn efficiently in the on-robot setting. Here, we baseline against the Framework for Efficient Robotic Manipulation (FERM) [27]. FERM utilizes contrastive learning with random crop augmentations improve sample efficiency. We implement FERM such that the image encoder has a similar amount of trainable parameters as the equivariant network (1.5M vs I.).). See Appendix E for details. Figure 5 compares the learning curves of Equivariant SAC (blue) and FERM (red) where both methods learn on-robot. In Block Picking, Equivariant SAC masters the task after about 1000 steps, while FERM learns much more slowly. In Clutter Grasping, FERM performs better at the beginning phase of learning but fails to converge to good policy at the end of learning. We hypothesize that this is because: 1) the pre-training of the encoder in FERM helps the network to learn a better feature representation at the early phase of learning. 2) this task does not require accurate manipulation as the other three tasks since the objects are all deformable (this also explains why the on-robot learning is faster than the simulation

---

Grasping
Discounted rewerd
022
2000
250
Number of training steps
scrasch
SAC
FERM
Figure 5: Comparison of Equivariant SAC trained from scratch (blue), Equivariant SAC with sim- to-real fine-tuning (green), and FERM (red) in real world. The plots show the performance of the behavior policy in terms of the discounted reward. Each point is the average discounted reward in the previous 200 steps. Results are averaged over three runs. Shading denotes standard error.
(a)
(b)
Figure 6: The different strategy between the simulation agent and the on-rohot learning agent. (a): In simulation, the agent presses on the edge the block and bounces it away. (b): In the real world, the agent pushes the block.
in this task). Block Pushing and Block in Bowl, FERM fails to learn any good policy all, while Equivariant SAC solves the task within 2000 and 4000 steps, respectively.
5.4 Sim2real Comparison
Here, compare direct on-robot learning with a sim2real strategy where we first train the agent to convergence in simulation and then transfer to the physical system by directly copying model parameters. We used the PyBullet simulation environment described in Section 4. The last row of Table 1 shows the performance of the agent trained in simulation when that policy is executed on the physical robot. In general, the sim2real policy performs significantly worse than the on-robot trained policy: 86% versus 96% in Clutter Grasping, 70% versus 92% in Block Pushing, and 72% versus 92% in Block in Bowl. The exception Block Picking which both the sim2real and on-robot policies achieve 100% success. This result explicitly reflects the sim2real performance gap.
5.5 Fine-tuning Sim2real on Physical Robots
We also evaluated our ability to improve the policy learned simulation by fine-tuning on the phys- ical robot. The green line in Figure 5 shows the learning curve of this fine-tuning agent. Before fine-tuning begins, the models used by this agent are loaded with the model parameters learned in simulation. This should be compared with the performance of the on-robot agent (no pre-training), shown in blue Figure 5. Block Picking, Clutter Grasping and Block in Bowl, both the training from scratch and sim2real fine-tuning converge to the same level of performance at the end of train- ing, where the training from scratch agent is about 500 steps slower than the fine-tuning agent in terms of converging speed. This suggests that while pre-training helps in these tasks, it is not critical to successfully learning the policy.
In Block Pushing, on the other hand, the fine-tuning agent performs very poorly and it does not recover during training (green versus blue line Figure 5). To understand why this occurs, we checked the policies typically learned by the different agents for Block Pushing. Whereas the sim-
7

---

ulation agent learns to move the block by pushing down on the block from the top (see Figure 6 (a)), the on-robot agent learns policy that pushes the block from the side (see Figure (b)). We hypothesize that this is what impedes the sim2real fine-tuning agent on this task. In order to suc- ceed on the physical robot, the sim2real agent must first unlearn the policy that was successful in simulation. This is an example of negative transfer where the pre-trained policy actually impedes on-robot learning. The fact that the simulation agent learns to push down on the block (and then to push sideways) whereas the on-robot agent learns push from the side is probably the result of differences between the contact dynamics the simulator and the real robot. The PyBullet simula- tor has slightly softer contact compliance that enables it to push down without generating too much force and triggering the URS safety system. Also, the friction coefficient between the gripper and the block is larger than between the block and the ground, allowing the simulator to slide the block. Notice that would be very hard to close this gap by improving the simulator because it is difficult to measure and model physical friction and compliance accurately.
6 Discussion
This paper proposes an approach to on-robot learning using Equivariant SAC in combination with data augmentation that can learn to solve simple manipulation tasks in a couple of hours without pre-training. Using the new method, we find that may sometimes be unnecessary to pre-train in simulation before training physical robot and that pre-training in simulation can sometimes be harmful because it causes negative transfer.
Limitations
A key limitation of the approach is that the use of equivariant models requires making assumptions about which symmetries are present in the domain. While many robotics problems have rotation, translation, and reflection symmetries, these symmetries may not be present in all regions of the state space. The current approach requires the system designer model these asymmetries ex- plicitly. However, the system would ideally have the ability to learn where symmetries are present without hand coding. Similarly, many problems have domain-specific symmetries beyond rotation and reflection symmetries that can be hard for the system designer to recognize. We would like our system to able to identify which symmetries are present and then incorporate them into the relevant neural models. Another area for future work is to understand when incorrect symmetry assumptions are still useful. We have observed that equivariant models can still speed up learning, even when the symmetry assumptions to have a theory for understanding this phenomenon more precisely. Third, though our method is not limited to depth images, this work demonstrates equivariant learning using depth images only because our tasks do not require RGB information. In the future work, we will incorporate RGB image improve the robustness of our method to transparent reflective objects.
sometimes violated in specific domain. We would like
Acknowledgments
This work supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, and NASA 80NSSC19K1477 1474. R. Walters is supported by the Roux Institute and the Harold Alfond Foundation and NSF grants 2107256 and 2134178.
References
[1] I.. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406-3411 3413. IEEE, 2016.
[2] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv:1806. 10293, 2018.
8

---

[3] L. Berscheid, C. Friedrich, and T. Kröger. Robot learning of dof grasping using model- based adaptive primitives. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4474-4480. IEEE, 2021.
[4] E. van der Pol, D. Worrall, H. van Hoof, F. Olichoek, and M. Welling. Máp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Pro- cessing Systems, 33, 2020.
[5] D. Wang, R. Walters, and R. Platt. S0(2)-cquivarian reinforcement learning. In Inter- national Conference on Learning Representations, 2022. URL https: oppnreview.eeeeeeeee net/
[6] S. Höfer, K. Bekris, A. Handa, J. C. Gamboa, F. Golemo, M. Mozifian, C. Atkeson, D. Fox, K. Goldberg, J. Leonard, et al. Perspectives on sim2real transfer for robotics: A summary of the Ss 2020 workshop. arXiv preprint arXiv:2012.03806, 2020.
[7] M. Weiler and G. Cesa. arXiv: 190::/911.000 1.08251, 2019.
General e ({2) (e)euivvari steerable cnns.
arXiv preprint
[8] T. Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pages 2990-2999. PMLR, 2016.
[9] S. Cohen and M. Welling. Steerable cnns. arXiv preprint arXiv:1612.008488, 2016.
[10] G. Benton, M. Finzi. P. Izmailov, and A. G. Wilson. Learning invariances in neural networks. arXiv preprint #rXiv:2010.11882 2020.
[11] N. Dey, A. Chen, and S. Ghafurian. Group equivariant generative adversarial networks. arXiv preprint WiXiv::00550168.. 2020.
[12] R. Wang, Walters, and Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.0306)/ 2020.
[13] R. Walters, J. Li, and R. Yu. Trajectory prediction using equivariant continuous convolution. arXiv preprint arXiv:2010. 11344, 2020.
[14] A. K. Mondal, P. Nair, and K. Siddiqi. Group equivariant deep reinforcement learning. arXiv preprint arXiv:2007.088. 0040077..337.... 2020.
[15] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant Sq$ learning in spatial action spaces. In 5th Anmual Conference on Robot Learning, 2021. URL ptlttps://ppeeree/// wwt.reeeeeeeee///ee/eee forum? 2wt.:iddd=i22aaaeeem
[16] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. Proceedings of Robotics: Science and Systems (RSS), 2022.
[17] H. Huang, D. Wang, R. Walters, and R. Platt. Equivariant transporter network. In Proceedings of Robotics: Science and Systems (RSS), 2022.
[18] A. A. Rusu, M. Vežerík, T. Rothörl, N. Heess, R. Pascanu, and Hadsell, Sim-to-real robot learning from pixels with progressive nets. In Conference on Robot Learning, pages 262-270. PMLR, 2017.
[19] Y. Zhu, Z. Wang. J. Merel, A. RUsu, T. Erez, Cabi, Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, et al. Reinforeement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 1802.89004. 2018.
[20] M. Gualtieri and R. Platt. Learning manipulation skills via hierarchical spatial attention. IEEE Transactions on Robotics, 3674):0677-1078, 2020.
[21] D. Wang, C. Kohler, and R. Platt. Policy learning in se (3) action spaces. arXiv preprint w.Xiv:2010.02798, 2020.
[22] O. Biza, D. Wang, R. Platt, J.-W. van de Meent, and I. Wong. Action priors for large action spaces in robotics. arXiv preprint 2rXiv.2101.04178. 2021.

---

[23] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-py updates. 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396 IEEE, 2017.
[24] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07833 2019.
[25] A. Zeng. S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4238- 4245. IEEE, 2018.
[26] A. Zeng, S. Song, J. Lee, Rodriguez, and T. Funkhouser. Tossingbot: Learning throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4):1307-1319, 2020.
[27] A. Zhan, Zhao, L. Pinto, P. Abbeel, and Laskin. A framework for efficient robotic manipulation. arXiv preprint arXin:2012.07975, 2020.
[28] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[29] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprini /rXiv: /807.03748, 2018.
[30] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[31] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. Gitllub repository, 2016.
[32] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
[33] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
[34] W. Wohlkinger, A. Aldoma, B. Rusu, and M. Vineze. 3dnet: Large-scale object class recog- nition from cad models. In 2012 IEEE international conference on robotics and automation, pages 5391-5391. TEFE, 2012.
[35] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[36] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.1 13649, 2020.
10


Reviews:
[{"review_id": "HeLDnOaCtpr", "reviewer": "Reviewer_vBAF", "summary": "This paper proposes an on-robot equivariant model to handle manipulation tasks. The work leverages the task structures such as symmetry explicitly for efficient learning of a control policy.  Moreover, the paper finds that data augmentation is still a powerful tool together with equivariant models. The experiments demonstrate the effectiveness of the work as well as compare different types of equivariant models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written.\nThe paper demonstrates the effectiveness of equivariant models for on-robot learning. While the idea itself is not new, the overall system is valuable. \nThe authors carefully study the variants of different equivariant models. This shed light on future research in this direction.\n\nWeaknesses:\nThe limitation of this work stands out. The requirement of the task structure prevent the method from scaling up to a general method. \nThe tasks are relatively on the easy end of the family. But this is a minor issue.\nThe robustness of the proposed method is not proved. For example, would this method work with arbitrary camera angle？\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work aligns well with the scope of CORL. It demonstrates the effectiveness of equivariant models on real robots. The authors test with various implementations and provide useful information for future works.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The key issue is that the method might not be scalable to different tasks or camera settings. \nThe lack of in-depth analysis of data augmentation in equivariant models. The reason why it works is not clear."}}, {"review_id": "6FbJfHZfOit", "reviewer": "Reviewer_QgU1", "summary": "The paper applies the recent progress of equivariant policy learning to top-down grasping using trained on real robot arms. The experiments show that the effectiveness of high sample efficiency of the equivariant policy learning. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe on-robot results are pretty impessive in terms of the sample-efficiency of learning process. The results are compared to one of the state-of-the-art methods FERM (Framework for Efficient Robotic Manipulation) in top-down grasping leanring, and show superior performance. In addition to purely on-robot learning bootstrapped from 20 human demonstrations, sim2real results are shown and analyzed. It's also worth-noting that this paper discussed the limitations of using equiviance in policy learning very well in the limitation section. \n\n## Weakness\n\n1. The delta of this paper's methodology from the prior work, Equivariant SAC (SO(2)-equivariant reinforcement learning), that this paper is largely based on, is small. Indeed, authors discovered that using discrete rotation and reflection groups for equivariant sac training can lead to better performance than the continuous counterpart, and found some interesting phenomena in sim2real transfer. However, the change in equivariant groups doesn't seems like a crucial part of the success of fast on-robot learning, since Figure 3 (a) only demonstrates minor performance boost by not using SO(2). \n\n2. Similar to the prior work Equivariant SAC, depth images are used as input to the policy. One limitation of using depth images is that they are not robust to many objects (e.g. plastic bottle, reflective silverwares) that are common in daily life. One solution to this is to directly use RGB images like what FERM did. It will be interesting to see a discussion about why not using RGB for equivariant policy learning. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the methodology of is this work is largely based on the prior work (Equivariant SAC), I would still recommend for a weak accept. I appreciate the large amount of work done by authors to make on-robot learning work really well, which aligns with this conference by creating learning methods that actually work on real robots and real problems. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. More details about what D_n and C_n groups mean in the context of depth images\n2. Discussion about why using depth and why not using RGB images. Is it simply because depth images give better sample efficiency, or there is some difficulties to prevent RGB images from being using by Equivariant SAC?\n3. Clearly state this work is based on Equivariant SAC (SO(2)-equivariant reinforcement learning) and E2CNN (General E(2)-Equivariant Steerable CNNs). "}}, {"review_id": "2P5t1o6pW64", "reviewer": "Reviewer_MYEX", "summary": "The paper presents a study on using equivariant models for deep reinforcement learning of robotic manipulation policies. It studies a variant of a well-known deep RL algorithm, soft actor-critic, when using rotation and reflection equivariant neural architectures for the learned components, as in prior work on equivariant SAC. The authors experiment to determine which symmetry groups lead to improved performance, and also show that interesting, further data augmentation can improve sample efficiency on top of discrete equivariances. They use real-world robot learning experiments to demonstrate the sample efficiency afforded by equivariant models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The motivation is very strong – sample efficiency is a significant limiting factor for current deep RL algorithms when trying to learn robot control policies from scratch.\n- The authors are able to successfully demonstrate real world on-robot learning with impressive sample efficiency, which is quite a nice result. \n- The experimental contributions of the paper are quite useful for practitioners and I think will be of interest to the community. \n\nWeaknesses:\n- The methods evaluated in the paper have limited novelty, and are mostly drawn from prior work. However, the experiments and analysis of these methods still provide significant value.\n- The first claim is that $D_4$ is empirically the best symmetry group is hard to conclude given the results in Figure 3, since the error bars are quite wide. Also, the statement that “In all four environments, $D_4$ shows the best performance over the four different groups” seems unclear to me – which metric is this based on? For example, $D_8$ does not seem to be worse than $D_4$ for the “block in bowl” task. \n- As with all equivariant methods, much care must be taken to ensure that the action and state space satisfy the conditions necessary for invariance and equivariance. This limits the types of tasks that such methods can be applied to. In spite of this, I believe that many important robotic manipulation tasks can be framed in such a way.\n- The reset mechanism for the block picking task requires the robot to already be able to pick up the block which somewhat obviates the task, but this could have been done using a number of other reset mechanisms (which aren’t the focus of this work).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a nice empirical study on using methods like equivariant SAC for real robot learning. It is a good fit for the CoRL venue and I think it is well motivated and interesting. Some of the claims made in the paper do not seem to be backed by experimental results with significant effect size, which is my main concern. However, I think the demonstration of highly sample efficient on-robot deep model-free RL is quite impressive. I currently vote for weak acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- I think it would be helpful to see the performance of non-equivariant SAC with the expert demonstrations in the buffer as well as non-equivariant SACfD, both in the simulated and real experiments. The claim in the paper is made that the sample efficiency of the equivariant method is significantly improved, but it’s never shown in any experiments. I would agree that in general on-robot learning using RL from scratch can be extremely time-consuming, but it would depend on the action space, reward, expert demonstrations provided, etc. so it would be good to be able to see that comparison here as well.\n- Why were 20 expert demonstrations chosen for the real-robot experiments? It seems from the experimental results in Figure 13 that 10 demonstrations doesn’t seem significantly worse. \n- Is the input to the networks in the real robot experiments a depth image, or is it a two-channel image like in the simulated experiments which also includes the binary channel indicating whether the gripper is holding the object? \n- It would be helpful for clarity to include that the plots in Figure 3(a) are generated using buffer augmentation as the additional data augmentation strategy (if I am not mistaken). \n\n"}}]

Author rebuttal:

Decision: Accept (poster)