Equivariant $Q$ Learning in Spatial Action Spaces
Authors: Dian Wang
Robin Walters
Xupeng Zhu
Robert Platt
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Equivariance
Manipulation

Abstract: Recently, a variety of new equivariant neural network model architectures have been proposed that generalize better over rotational and reflectional symmetries than standard models. These models are relevant to robotics because many robotics problems can be expressed in a rotationally symmetric way. This paper focuses on equivariance over a visual state space and a spatial action space -- the setting where the robot action space includes a subset of $\rm{SE}(2)$. In this situation, we know a priori that rotations and translations in the state image should result in the same rotations and translations in the spatial action dimensions of the optimal policy. Therefore, we can use equivariant model architectures to make $Q$ learning more sample efficient. This paper identifies when the optimal $Q$ function is equivariant and proposes $Q$ network architectures for this setting. We show experimentally that this approach outperforms standard methods in a set of challenging manipulation problems. 

Equivariant Q Learning in Spatial Action Spaces
Dian Wang
Robin Walters
Xupeng Zhu
Robert Platt
Khoury College of Computer Sciences Northeastern University
United States
(wang dian, r. r.walters, zhu. xup, r.platt/enorthhesteen.cc
Recently, a variety of new equivariant neural network model architec- tures have been proposed that generalize better over rotational and reflectional symmetries than standard models. These models are relevant to robotics because many roboties problems can be expressed in rotationally symmetric way. This paper focuses on equivariance over a visual state space and a spatial action space the setting where the robot action space includes subset of SE(2). In this sit- uation, we know a priori that rotations and translations the state image should result in the same rotations and translations in the spatial action dimensions of the optimal policy. Therefore, we can use equivariant model architectures to make Q learning more sample efficient. This paper identifies when the optimal Q fune- tion is equivariant and proposes Q network architectures for this setting. We show experimentally that this approach outperforms standard methods in set of chal- lenging manipulation problems.
Abstract:
Keywords: Reinforcement Learning, Equivariance, Manipulation
Introduction
A key question in policy learning for robotics is how to leverage structure present in the robot and the world to improve learning. This paper focuses on a fundamental type of structure present in visuo-motor policy learning for most robotics problems: translational and rotational invariance with respect to camera viewpoint. Specifically, the reward and transition dynamics of most robotics problems can be expressed in way that is invariant with respect to the camera viewpoint from which the agent observes the scene. In spite of the above, most visuo-motor policy learning agents do not leverage this invariance in camera viewpoint. The agent's value function or policy typically considers different perspectives on the same scene to be different world states. A popular way combat this problem through visual data augmentation, i.e., to create additional samples or experiences by randomly translating and rotating observed images [1] but keeping the same labels. This can be used in conjunction with contrastive term in the loss function which helps the system learn an invariant latent representation [2, 3]. While these methods can improve generalization, they require the neural network learn translational and rotational invariance from the augmented data.
Our key idea in this paper is to model rotational and translation invariance in policy learning us- ing neural network model architectures that are equivariant over finite subgroups of SE(2). These cquivariant model architectures reduce the number of free parameters using steerable convolutional layers [4]. Compared with traditional methods, this approach creates inductive bias that can sig- nificantly improve the sample efficiency of the model, the number of environmental steps needed to learn policy. Moreover, it enables to generalize respect to one camera viewpoint is automatically also represented in other camera perspectives via selectively tied parameters in the model architecture. We focus our work on Q learning in spatial action spaces, where the agent's 's action space spans SE(2) or SE(3). We make the following contri- butions. First, we identify the conditions under which the optimal Q function is SE(2) equivariant. Second, we propose neural network model architectures that encode SE(2) equivariance in the Q function. Third, since most policy learning problems are only equivariant some of the state vari- ables, we propose partially equivariant model architectures that can accommodate this. Finally, we compare equivariant models against non-equivariant counterparts in the context of several robotic
very precise way: everything learned with
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

manipulation problems. The results show that equivariant models are more sample efficient than non-equivariant models, often by a significant margin. Supplementary video and code are available at https: /ttp://poitttttt/ tithub. i 1 _tteei_____aae_e page.
2 Related Work
Data Augmentation: Data augmentation techniques have long been employed in computer vision to encode the invariance property of translation and reflection into neural networks [5, 6]. Recent work demonstrates the use of data augmentation improves the data efficiency and the policy's performance in reinforcement learning [7. 9]. In the context of robotics, data augmentation is often used to generate additional samples [1, 10, 1 111. In contrast to learning the equivariance property using data augmentation, our work utilizes the equívariant network to hard code the symmetries in the structure of the network to achieve better sample efficiency.
Contrastive Learning: Another approach to learning representation that is invariant to translation and rotation is to add a contrastive learning term to the loss function [2]. This idea has been applied to reinforcement learning in general [3] and robotic manipulation in particular [12]. While this approach can help the agent learn an invariant encoding of the data, it does not necessarily improve the sample efficiency of policy learning.
Equivariant Learning: Equivariant model architectures hard-code E(2) symmetries into the structure of the neural network and have been shown to be useful in computer vision [13, 4, 14]. In reinforce- ment learning, some recent work applies equivariant models structure-findin problems involving MDP homomorphisms [15, 16). In addition, Mondal et al. [17] recently applied an -quuaaqaua model Q learning in an Atari game domain, but showed limited improvement. To our knowledge, cquivariant model architectures have not been explored in the context of robotics applications.
Spatial Action Representations: Several researchers have applied policy learning in spatial action spaces to robotic manipulation. A popular approach is to do Q learning with a dense pixel action space using a fully convolutional neural network (this is the FCN approach we describe and extend in Section 4.2) [18, 19, 20, 21]. Variations on this approach have been explored in [22, 23]. The FCN approach has been adapted to a variety of different manipulation tasks with different action primitives [24, 25, 26, 27, 28, 1, 29, 30, 31]. In this paper, extend the work above by proposing new equivariant architectures for the spatial action space setting.
Problem Statement
We are interested in solving complex robotic manipulation problems such as the packing and con- struction problems shown in Fig 1. We focus on problems expressed in a spatial action space. This section identifies conditions under which the Q function is (B(())invariann) The next section de- scribes how these invariance properties translate into equivariance properties in the neural network.
Manipulation as an MDP in over a visual state space and a spatial action space: We assume that the manipulation problem is formulated as a Markov decision process (MDP): M = (S, A,T, R, 7)- We focus on MDPs in visual state spaces and spatial action spaces [29, 20, 31]. The state space is factored into the state of the objects in the world, expressed as an ne-caanel h w image E Sworld = 1uxhxx and the state of the robot (including objects held by the robot) Sybt E Sybt. expressed arbitrarily. The total state space is S Sworld X Sybt- The action space is expressed as a cross product of SE(() (hence it is spatial) and a set of additional arbitrary action variables: A = SE(2) x Aarb- The spatial component of action expresses where the robot hand is to move and the additional action variables express how it should move or what should do. For example, in the pick/place domains shown in Fig 1, Aarh = {PICK, PLACE), giving the agent the ability to move to a pose and close the fingers (pick) or move and open the fingers (place). We will sometimes de- compose the spatial component of action dup SE(() into its translation and rotation components, a.p = (x. 0). The goal of manipulation is to achieve a desired configuration of objects in the world, as expressed by a reward function R: S x A IR
Translation and Rotation in SE(2): We are interested in learning policies that are invariant to trans- lation and rotation of the state and action. To do that, we define rotation and translation of state and action follows. Let g E SE(2) be an arbitrary rotation and translation in the plane and let

---

(a) Block Stacking
(b) Bottle Arrangement
(c) House Building
Covid Test
(c) Box Palletizing
(f) Bin Packing
Figure 1:
The experimental environments implemented in PyBullet [32]. The left image in each sub
figure shows an initial state of the environment; the right image shows the goal state.
Sebe) E Sworld X Sibt be state. g operates by rotating and translating the image I, but leaving Sybt. unchanged: gs = (gI, Sybt). where gl denotes the image translated and rotated by (Asp: (arb), 9 rotates and translates Osp but Garb: ga (g0spp Carb). Notice that both S and A are closed under E SF(2), i.e. that Vy E SE((), a E A = ga E and S g8 E S.
g- For action
Assumptions: We assume that the reward and transition dynamics the system are invariant with respect to translation and rotation of state and action as defined above, and that the translation and rotation operations on state and action are invertible.
Assumption 3.1 (Goal Invariance). The manipulation objective achieve desired configuration of objects in the world without regard to the position and orientation of the scene. That is, R(s,a) R[gs,go for all g SE(2).
Assumption 3.2 (Transition Invariance). The oulcome of robot actions is invariant to translations and rotations of both the scene and the action. Specifically, T(s, s') T(gs, ga, gs') for all g E SE(2).
Assumption 3.3 (Invertibility). Translations and rotations in state and action are invertible. That is, Vg E SE(2). 9 -' (gs) and g
g- (ga) a.
Assumptions 3.1 and 3.2 are satisfied in problem settings where the objective and the transition dynamics can be expressed intrinsically to the world without reference to an external coordinate frame imposed by the system designer. These assumptions are satisfied in many manipulation do- mains including all those shown in Fig 1. In House Building, for example, the reward and transition dynamics of the system are independent of the coordinate frame of the image or the action space. Assumption 3.3 is needed to guarantee the Q function invariance described in the next section.
4 Approach
Assumptions 3.1, 3.2, and 3.3 imply that the optimal Q function is invariant to translations and rotations in SE(2).
Proposition 4.1. Given an MDP M = (S, A, R, 7) for which Assumptions 3.2, and 3.3 are satisfied, the optimal Q function is invariant to translation and rotation, i.e. Q* (s.a) = Q" (gs, ga), for all g E SE(2). (Proof in Appendix A.)
Our key idea is to the invariance property of Proposition 4.1 to structure Q learning (and make more sample efficient) by defining a neural network that is hard-wired to encode only invariant Q functions. However, in order to accomplish this in the context of DQN, we must allow for the fact that state is an input to the neural network while action values are an output. This neural network is therefore function S RA, where RA denotes the space of functions {A + R}. The invariance property Proposition 4.1 now becomes an equivariance property,
(a) (g
(I)
where (s) denotes the Q value action a in state s. We implement this constraint using equiv- ariant convolutional layers described below.

---

4.1 Equivariant Convolutions
Equivariance over a finite group: In order to implement the equivariance constraint, is standard in the literature to approximate SF(2) by a finite subgroup [4, 14). Recall that the spatial component of an action is Oup = (x, 0) E SE(2). We constrain position to be a discrete pair of positive integers x E {1... h x {1...uu 3 W C z² corresponding to a pixel in the input image /. We constrain orientation to be a member of finite cyclic group E Cu. i.e. one of u. discrete orientations. 2x. St dn Sx bn 7
For example, if u = 8, then Css = {0, 4: 4 4 4 4' 4 44.. J. Our finite approximation of
Asp E (2) is asp in the subgroup SEE(2) generated by translations 2² and rotations C..
Input and output of an equivariant convolutional layer: A standard convolutional layer h takes as input an n-channel feature map and produces an m-channel map output, hatandard : gaxxxx + RT gmxx We can construct an equivariant convolutional layer by adding additional dimension to the feature map that encodes the values for each element of group (C., in our case).' The equivari- ant mapping therefore becomes hequiv Ruxnxhxw + paxmxhxu for all layers except the first. The first layer of the network generally takes "flat" image as input: h : RRIxxxxx fuuxxxxxw
S1(2))
9l
N 12)
9 SE(2) trumalate
plae) Kar
Figure 2: Illustration of Q-map equiv- ariance when C, = C4. The output Q- map rotates and translates with the input image. The v-vector at each pixel does a circular shift, i.e., the optimal rotation changes from (the I st element of C4) to z (the 2nd element of C4)
Equivariance constraint: Let (z) denote the output of convolutional layer h at channel and pixel x given in- put T. For an equívariant layer, (a(())))) E CR describes feature values for each element of C.. For an element 9 E SEE)),, denote the rotational part by 9o E C... If we iden- tify functions RC= with vectors R", then the group action of go E C. on RC, becomes left multiplication by a per- mutation matrix p(ga) that performs circular shift on the vector in I Then the group action of SE(2) on a feature map hTT) R maxxxxaa xmxxxxu can be expressed as g(h: (T The mapping h is equivariant if and only if
h hx(g)(xx)= ((h,(t)) p(sa))i
for each E (1...m). This is illustrated in Fig 2. We can calculate the output feature map in the lower right corner by transforming the input by g and then doing the convolution (left side of Eq. 2) or by doing the convolution first and then taking the value of g "1 and circular-shifting the output vector (right side of Eq. 2). In order to create network that enforces the constraint of Eq. 1, we can simply stack equivariant convolutions layers that each satisfy Eq. 2.
The equivariance constraint of Eq. can be implemented by strategically tying weights together in the convolutional kernel [4]. Since the standard convolutional kernel is already translation equivariant [13]. we must only enforce rotational (C.)) equivariance [33]:
Kernel constraint:
K (gov) Pout (go) ) K (y) (ga)=
where Pin (ge( and Pout (ge) are the permutation matrix of the group element ge (note that for the first layer, K (y) will be a 1 x matrix, and (gs will be 1). More details are in Appendix B.
4.2 Equivariant Fully Convolutional Q Functions in SE(2)
A baseline approach to encoding the Q function over spatial action space is to fully convo- lutional network (FCN) that stacks convolutional layers to produce an output Q map with the same resolution the input image. If we ignore the non-image state variables Sebt and the non-spatial action variables darh, then we have all the tools we need we simply replace all convolutional layers with equivariant convolutions and the Q network becomes fully equivariant.
Partial Equivariance: Unfortunately, realistic roboties problems, Q function is generally not equiv- ariant with respect all state and action variables. For example, the non-equivariant parts of state In the language [14], this a steerable convolution between regular representations Cu. 2 This is stecrable convolution between the trivial representation and regular representation of C.,

---

and action in Section 3 are Sybt and aarb- We encode aurb by sim- ply having a separate output head for each. However, to encode Sybe, we need mechanism for insert- ing the non-equivariant informa- tion into the neural network model without "breaking" the equivari- ance property. We explored two approaches: the lift expansion ap- proach and the dynamic filter approach. In the lift expansion, tile the non-equivariant information across the equivariant dimensions of the feature map as additional channels (Fig 3a). In the dynamic filter approach [34], the non-equivariant data passed through separate pathway that outputs the weights of an equivariant kernel that is convolved into the main equivariant backbone. We constrain this filter to be equivariant by enforcing the kernel constraint of Eq. 3 (Fig 3b). We empirically find that both methods have similar performance (Appendix G.2). In the remainder of this paper, we use the dynamic filter approach because is more memory efficient.
Eronder
Dupal
Dup
Dipaaton
Aaa
Lift Expansion
(b) Dynamic Filter
Illustration of approaches to partial equivariance. Green blocks are equivariant and white blocks are not.
Figure 3:
Encoding Gripper Symmetry Using Quotient Groups: Another symmetry that we want to leverage the bilateral symmetry of the gripper. The outcome of a pick action performed using two- finger gripper in orientation 0 is the same as for the gripper in orientation 0 + ko for any integer k. Similarly, it often valid to assume that the outcome of place actions invariant 3. We model this invariance using the quotient group Cu/C2. The C2 = {0, πT action equates rotations which differ by multiples of π in /a,/ Coz The steerable layer defined under the quotient group is applied with the same constraint in Eq. 3, except that the output space will be in C2./C2.
Experimental Domains: We evaluate the equivariant FCN approach in the Block Stacking and Bottle Arrangement tasks shown in Fig 1. Both environments have sparse rewards (+1 goal and 0 other- wise). The world state is encoded by a 1 1-channel heightmap I E R R1xhx and robot state is encoded by an image patch H that describes the contents of the robotic hand. The non-spatial action variable Qarb E {PICK, PLACE} } is selected by the gripper state, i.c., Qarb = PLACE if the gripper is hold- ing an object, and PICK otherwise. The equivariant layers of the FCN are defined over group C12 where the output is with respect to the quotient group C12/C2 to encode the gripper symmetry. See Appendix C and for detail on the experimental domains and the FCN architecture respectively.
Experimental Comparison With Baselines: evaluate against the following baselines: 1) Con- ventional FCN: FCN with 1-channel input and 6-channel output where each output channel cor- responds to Q map for one rotation in the action space (similar to Satish et al. [19] but without the dimension). 2) RAD [7] FCN: same architecture as 1), while at each training step. we augment each transition in the minibatch with a rotation randomly sampled from C12- 3) DrQ [8] FCN: same architecture as 1), while at each training step, the Q targets and Q outputs are calculated by averaging over multiple augmented versions of the sampled transitions. Random rotations sampled from C12 are used for the augmentation. 4) Rot FCN: FCN with 1-channel input and I-channel output, the rotation is encoded by rotating the input and output for each 0 [21]. 5)
We
NAD PCN Tranpore
(a) Block Stacking
(b) Bottle Arrangement
Figure 4:
Comparison of Equivariant FCN (blue) with baselines. Results averaged over four runs. Shading denotes standard error.
Transporter Network [1], an FCN-based architecture with the last layer being a dynamic kernel generated by separate FCN with an input of an image crop the pick location. Baseline 2) and 3) are data augmentation methods that aim to learn the symmetry encoded in our equivariant network using rotational data augmentation sampled from the same symmetry group (C12) as used by our equivariant model. All baselines have the same action space as the proposal. More detail on the baselines is in Appendix E. All methods except the Transporter Network use SDQND, an approach to imitation learning in spatial action spaces that combines a TD loss term with penalties on non-expert actions [29]. (Transporter Network is a behavior cloning method.) Table shows the number of demonstration steps. Those expert transitions are augmented by 9 random
Strictly speaking, this is true only when the grasped object is also symmetric.
5

---

Block Siucking
Botile Arrangemnt
House Building
Box Palletizing
Covid Test
Bin Packing
expert steps
50
240
200
1000
2000
2000
equivalent episodes
8
20
20
28
111
125
Table 1: The number of expert steps and the (approximate) equivalent number of episodes.
SE(2) transformations. See Appendix F.2 for more parameter detail. Fig 4 shows the results. Our equivariant FCN outperforms all baselines in the block stacking task. Notice that in the Bottle Arrangement task, the equivariant network learns faster than the baselines but converges to a similar level as RAD and DrQ. This is because the domain itself is already partially rotationally equivariant because the bottles are cylindrical and therefore our network has less of an advantage.
4.3 Equivariant Augmented State Q Functions in SE(2)
The FCN approach does not scale well to challenging manipulation problems. Therefore, we design an equivariant version of the augmented state representation (ASR) method of [29], which has been shown to be faster and have better performance. The ASR method transforms the original MDP with a high dimensional action space into a new MDP with an augmented state space but lower dimensional action space. Instead of encoding the value of all dimensions of action in a single neural network, this model encodes the value of different factorized parts of the action space such as position and orientation using separate neural networks conditioned on prior action choices.
ASR in SE(2): We explain the ASR method in the example setting of the SF( action space. See Fig for an illustration. As be- fore, actions åsp = (z. f) are elements the space SE(2), the fi- nite approximation of SE(2) as in Section 4.1. However, the Q function is now computed using two separate functions, the position function (s, (sx (ar, 0)) and the orientation function Q2((s, 2), 0) = (ss, (rr,)), Q is encoded using a fully convolu- tional network qi Raxxxx + 1xx that takes an re-hannel image I as input and produces 1-channel Q map that describes (s. 2) for all x. We evaluate Q2 on the "augmented state" (s, x) which contains the state and the chosen The augmented state is encoded using the image patch P CROP(T,x)) RRuxh'''''
gm
Figure 5: Illustration of the
ASR approach SE(2).
cropped from T and centered at x. We model Q2 using the network 92 Raxh' xw" + R that takes input P and outputs Q2((s. x). 8) for all different orientation 0. These two networks are used together for both ac- tion selection and evaluation of target values during learning. We evaluate x² arg max, 91 (T), calculate P = CROP( 7, xx)". ). and then evaluate 0" = arg maxe 22 (P)) and Q = max, (P)). Note that Q maps produced by Qi and f2 are of size u + har, significantly smaller than the Q map in the FCN approach which is size uhw. Essentially the ASR method takes advantage of the fact that the optimal 0 depends only on the local patch P given an optimal position x.
Equivariant architecture for ASR in SE(2): We de- compose the SF(2) equivariance property of Eq. into two equivariance properties for 91 and f2. re- spectively: 9ı (z))((x)
qı (1) -x) where g E SiE(2), and q2 P) p(gg)g2(())))) (P) where go C.. The equivariance property of qı is similar to that of Eq. 2 except that the output of q has only one chan- nel which is invariant to rotations (since is maxi- mum over all rotations). This means we can rewrite the Q1 equivariance property as qı (g() = 941 (f), where g on the RHS of this equation translates and rotates the output Q map. In practice, we obtained the best performance for q1 by enforcing equivari- ance to the Dihedral group Da. which is generated by 90 degree rotation and reflections over the coordinate axis. For q2, we used an equivariant feature map that outputs a single u-dimensional vector of Q values corresponding to the finite cyclic group C. used. (We use C12/C2 and C in our experiments below). We handle the partial equivariance using the same strategies as earlier. Appendix D describes the model details.
vertee AGA
Kcal A50
(a) Block Stacking
(b) Bottle Arrangement
Figure Comparison between Equivariant ASR (blue) and Equivariant FCN (green). Re- sults averaged over four runs. Shading de- notes standard error.

---

NAC vil Turma
Fel
l'raporna
Redeec-- eta
400
Mid
200
(a) House Building
(b) Covid Test
(c) Box Palletizing
(d) Bin Packing
Figure 7: Comparison of Equivariant ASR (blue) with baselines. Results averaged over four runs.
Shading denotes standard error.
Figure 8: Top row: the robot finishing the Bottle Arrangement task. Bottom row: the robot finishing the Box Palletizing task. Full episodes are in Appendix I.
Experimental Comparison with Equivariant FCN: Fig 6 shows a comparison between equivariant ASR (this section) and equivariant FCN (Section 4.2) for the Block Stacking and Bottle Arrangement tasks. The network 22 is defined using C12 and its quotient group C12/C² match Section 4.2. The ASR method surpasses the FCN method in both tasks.
The equivariant ASR method is able to solve more chal- lenging manipulation tasks than equivariant FCN can. In particular, we could not run the FCN with as large a rotation space because requires more GPU memory. We evaluate on the following four additional domains: House Building, Covid Test, Box Palletizing (introduced in [1]), and Bin Pack- ing (Fig (cc-f)). All domains except Bin Packing have sparse rewards. In Bin Packing, the agent obtains positive reward inversely proportional to the highest point in the pile after packing all ob- jects. See Appendix for more details about the environments. We now define d using the group C32 and its quotient group C32/O>, i... now encode 16 orientations ranging from 0 to As in Section 4.2, we use the SDQID loss term to incorporate expert demonstrations (except for Trans- porter Net which uses standard behavior cloning exclusively). The number of expert transitions provided is shown in Table 1. 9 random SE(2) augmentations are applied to the expert transitions.
More Challenging Experimental Domains:
Experimental Comparison with Non-Equivarian Baselines: We compare equivariant ASR against the following non-equivariant baselines: 1) Conventional ASR: ASR in SF(() with conventional CNNs rather than equivariant layers. 2) RAD [7] ASR: same architecture as (1) but each minibatch is augmented with random rotation. 3): DrQ [8] ASR: same architecture (1) but each Q target and Q estimate are calculated by averaging over several augmented versions of the sampled transition. The augmentation in (2) and (3) is by random rotations sampled from C32, the same group used in the equivariant model. 4) the Transporter Network [1]. See Appendix E.2 for the network architecture for the baselines. The results in Fig show that equivariant ASR outperforms the other methods on all tasks, followed by DrQ, RAD, and Transporter Net, followed by Conventional ASR.
Robot Experiment: We evaluate the trained equivariant ASR models for Bottle Arrangement, House Building, and Box Pal- letizing on a Universal Robots UR5 arm equipped with Robo- tiq 2F-85 gripper. The observation is provided by Occipi- tal Structure sensor mounted on top of the workspace. Table 2 shows the results. In Bottle Arrangement, the robot shows 90% success rate. one of the two failures, the arrangement is not compact enough, leaving no enough space left for the last bottle. In the other failure, the robot arranges the bottles outside of the tray. In the House Building task, the robot succeeds in all 20 episodes. In the Box Palletizing task, the robot demonstrates 95% success
Environment
SR
Bottle Arrangement
(08(()
House Building
(00%()0020)
Box Palletizing
(1%
Table 2: Robot experiment result
7

---

rate. In the failure, the robot correctly stacks 16 of 18 boxes, but the 17th box's placement position offset slightly from the the rest of the stack and there is no room to place the last box. The same problem happens in another successful episode, where the fingers squeeze the boxes and make room for the last box. Fig shows two example episodes in the robot experiment.
4.4 Equivariant Augmented State Q Functions in SE(3)
A strength of the ASR method is that it can be ex- tended into SE(3) by adding networks similar 92 that encode Q values for additional dimensions of the action space [29]. Specifically, we add three networks to the (2(()--euivvarian architecture de-
(a) House Building
(b) Box Palletizing
scribed in Section 4.3, Q3- q4, and ys encoding Q values for Z (height above the plane), and angles & (rotation XZ plane) and V/ (rotation in YZ plane) dimensions of SE(3). Each of these networks takes as input stack of orthographic projections of point cloud along the coordinate planes. The point cloud is re-centered and rotated to encode the par- tial SE(3) actions. (see [29] for details).
Figure 9: The 6DOF experimental domains.
Carver Bena 3n8
Unfortunately, we cannot easily make these addi- tional networks equivariant using the same meth- ods we have proposed so far because they encode variation outside of SE(2). Instead, encoding that is approximately equivariant by ex- plicitly transforming the input to these networks in a way that corresponds to a set of candidate robot positions orientations (called "deictic" encod- ing in [22].. We will describe this idea using 92 as an example. Define (P)) E R to output the single Q value of the rotation encoded in the input image P. Then 92 be defined vector- valued function: 92 ( P)..... (2
(a) House Building
(b) Box Palletizing 18
Figure 10: Comparison of the equivariant net- work with the baseline in 6DOF tasks. Re- sults averaged over four runs Shading de- notes standard error.
create an
P)), where .1..... Pu. E C. O is approximately equivariant because everything learned by ob is automatically replicated in all orientations. We design deictic 93. q4. and Qs similarly by selecting a finite subset of (yI,...,
C SE(3) cor- responding to the dimension of the action space encoded by each g- qe can then be defined by evaluating network of over input P transformed by (g.) For q3, evaluate over 36 trans- lations 9x = 2 + (0.188/3)) + 0.02 where 0 S k S 35. For qs and qs. we use rotations Is E 1 8 12: 24. 0, 24 5) Note we use 42 for explanation, while our model uses equiv- ariant 91. 82 and deictic -3- For an ablated version using deictic (2. see Appendix G.4.3 and G.4.4. Comparison to -quivarivar Approaches: We evaluate ASR in SE(3) in the House Building and Box Palletizing domains. We modified those environments so that objects are presented randomly on bumpy surface with a maximum out of plane orientation of 15 degrees (Fig 9). order to succeed, the agent must correctly perform pick and place actions with the needed height and out of plane orientation. We evaluated the Equivariant ASR in comparison with a baseline Conventional ASR (same [29]). Both methods use SDQID with 2000 expert demonstration steps. The results are shown in Fig 10. Our proposed approach outperforms the baseline by a significant margin.
5 Discussion
In this paper, we show that equivariant neural network architectures can be used improve Q learning in spatial action spaces. We propose multiple approaches and model architectures that can be used to accomplish this and demonstrate improved sample efficiency and performance on several robotic manipulation applications both in simulation and on physical system. This work has several limitations and directions for future research. First, our methods apply directly only to problems in spatial action spaces. While many robotics problems can be expressed this way, it would clearly be useful develop equivariant models for policy learning that can be used in other settings. Second, although we extend our ASR approach from SE(() to SE(3) in the last section of this paper, this solution is not fully equivariant in SB()) and may be possible to do better by exploiting methods that are directly equivariant in SE..).
8

---

Acknowledgments
This work is supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, and NASA 8ONSSC19K141 1474. R. Walters is supported by a Postdoctoral Fellowship from the Roux Institute and NSF grants 2107256 and 2134178.
References
[1] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
[2] A. v. d. Oord, Li and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXiv:1807.08 03748, 2018.
[3] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[4] T. S. Cohen and M. Welling. Steerable cnns. arXiv preprint arXiv:1612.088498 2016.
[5] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278--233 1998.
[6] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1100 2012.
[7] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data, arXiv preprint arXiv:2004. 14990, 2020.
[8] Kostrikov, D. Yarats, and Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[9] N. Hansen and X. Wang. Generalization in reinforcement learning by soft data augmentation. arXiv preprint arXiv:2011.1338) 2020.
[10] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[11] Y. Lin, J. Huang, M. Zimmer, Y. Guan, J. Rojas, and Weng. Invariant transform experience replay: Data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters, 5(4):6615 6622, 2020.
[12] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. arXiv preprint arXiv:20/2.07975. 2020.
[13] T. Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pages 2990-2999. PMLR, 2016.
[14] M. Weiler and G. Cesa. 1911v:1911.0825/1 2019.
General <(2)-equivariant steerable enns.
arXiv preprint
[15] E. van der Pol, T. Kipf, F. A. Olichoek, and M. Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002.11966 2020.
[16] E. van der Pol, D. Worrall, H. van Hoof, F. Oliehoek, and M. Welling, Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Pro- cessing Systems, 33, 2020.
[17] A. K. Mondal, P. Nair, and K. Siddiqi. Group equivariant deep reinforcement learning. arXiv preprint arXiv:2007.03453 2020.
[18] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: real-time, generative grasp synthesis approach. arXiv preprint arXiv:1804.05172 2018.

---

[19] V. Satish, J. Mahler, and K. Goldberg. On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks. IEEE Robotics and Automation Letters, (2):1357-1364, 2019.
[20] A. Zeng. S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, al. Robotic pick-and-place of novel objects in clutter with -ulti-affordance grasp- ing and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757. IEEE, 2018.
[21] A. Zeng. S. Song. S. Welker, J. Lee, Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4238- 4245. IEEE, 2018.
[22] R. Platt, C. Kohler, and M. Gualtieri. Deictic image mapping: An abstraction for learning pose invariant manipulation policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8042-8049, 2019.
[23] M. Gualtieri and R. Platt. Learning manipulation skills via hierarchical spatial attention. IEEE Transactions on Robotics, 3664))0067-1078, 2020.
[24] L. Berscheid, P. Meißner, and T. Kröger. Robot learning of shifting objects for grasping in cluttered environments. arXiv preprini arXiv: 1907. 1005,, 2019.
[25] H. Liang, X. Lou, and C. Choi. Knowledge induced deep q-network for slide-to-wall object grasping. arXiv preprint 1rx.0:1910003781, 2019.
[26] A. Zeng. S. Song, J. Lee, A. Rodriguez, and Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. arXiv preprint arXiv:1903.112399 2019.
[27] K. Zakka, A. Zeng. J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly, arXiv preprint arXiv:1900.1 13675, 2019.
[28] L. Berscheid, P. Meißner, and T. Kröger. Self-supervised learning for precise pick-and-place without object model. IEEE Robotics and Automation Letters, 5(3):4828-4835, 2020.
[29] D. Wang, C. Kohler, and R. Platt. Policy learning in se (3) action spaces. arXiv preprint arXiv:2010.02798, 2020.
[30] O. Biza, D. Wang, R. Platt, J.-W. van de Meent, and L. Wong. Action priors for large action spaces in robotics. arXiv preprint arXiv:2101.044 2021.
[31] J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. arXiv preprint arXiv:2004.09144 2020.
[32] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. GitHub repository, 2016.
[33] T. Cohen, M. Geiger, and M. Weiler. A general theory of equivariant cnns on homogeneous spaces. arXiv preprint /rXiv: 1.02017, 2018.
[34] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. NIPS, 2016.
[35] W. Wohlkinger, A. Aldoma Buchaca, R. Rusu, and M. Vineze. 3DNet: Class Recognition from CAD Models. In IEEE International Conference on Roboties and Automation (ICRA), 2012.
Large-Scale Object
[36] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
[37] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.
10

---

[38] Т. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan,
A. Sendonaris, I. Osband, al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[39] D. P. Kingma and J. Ba. arXiv:1412.6980) 2014.
Adam: A method for stochastic optimization. arXiv preprint
[40] Huber. Robust estimation of a location parameter. The Annals of Mathematical Statis- fics, 35(1):73-101, 1964. ISSN 00034851. URL http: www. wtw..jjjsoo..orssttaleess. 2238020.
[41] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
[42] K. Perlin. Improving noise. In Proceedings the the 29th annual conference on Computer graph- ics and interactive techniques, pages 681 1682, 2002.


Reviews:
[{"review_id": "srcWOWuvXhT", "reviewer": "Reviewer_BRhk", "summary": "The paper aims at exploiting equivariances under SE(2) present in robotics manipulation tasks with spatial state and action spaces in order to improve sample efficiency of image-based deep Q-learning. \nIn particular, the paper proposes to exploit the equivariance of the Q-function wrt. simultaneous symmetry transformations in the state and action spaces. For that purpose the authors introduce two new architectures for the network parametrizing the Q-function, which are equivariant wrt SE(2) transformations of (spatial components of) the state and action space.\nThe authors compare the performance of their proposed architectures to several baselines, which either do not take into account equivariances or try to learn these via data augmentation, in several robotics manipulation tasks in simulation. Furthermore, the authors demonstrate the applicabilty of their approach in a real world setup.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts: \nThe paper is well written and mathematically sound. \nIt clearly outlines the pre-conditions under which the considered equivariance properties apply. \nIt proposes two novel architectures that directly encode the SE(2) equivariance into the structure of the network of the Q-function.\nThe experiments  are well chosen and results are properly documented and discussed. \n\nWeaknesses:\nsee detailed issues below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper discusses an the topic of equivariances, which is very important for robotics research and applications. The proposed approach to encode the equivariance directly into the network of the Q-function is promising and presumably easy to implement and might therefore have an impact on robotics research. \nThe paper is well written, mathematically sound and the experiments are well chosen and properly discussed. \nThere are some minor issues, which the authors should address. Inspite of these I would recommend the paper for acceptance to CORL, since the positive aspects clearly outweigh the negatives. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- line 29: The authors state that other methods cited in previous lines improve generalization but \"generally make policy learning harder\". I think this statement is too strong and not properly justified. \n\n- experiments: \n-- I wonder how the angles for the outpur of basesline 1 (line 216) are chosen. Do they have the same granularity and respect the gripper symmetry as the equivariant method (corresponding to a choice [0, 30, 60, 90, 120, 150])? The authors should state that. Also, if the authors chose a coarsere granularity (i.e. [0, 60, 120, 180, 240, 320]) I believe the comparison is not fair.  Similarly I wonder, why the authors chose to rotate the input of the baseline 2) only in terms of 90 degree, leading to a much coarser angel granularity. I believe this comparison is not really fair. \n-- The authors do not discuss the runtime of their approach. How much does the runtime suffer from the sequential approach the authors propose in section 4.3?\n--  I think, that the assumed SE(2) equivariance may be slightly violated in the considered experiments due to the fact that the robot is fixed to the workspace at some point. Some state-action pairs might therefore be more difficult to attain for the robot than an equivariant pair. Did the authors observe such effects?\n-- section 4.4: The architecture and approximation mentioned in lines 339ff is not sufficiently discussed. The reader has no chance to understand it without the appendix. I think that such important \"details\" should be part of the main paper and not be transferred to the appendix, otherwise a page limit would become meaningless.\n"}}, {"review_id": "Q4gIcQW0WNq", "reviewer": "Reviewer_GkB1", "summary": "This paper presents an architecture for learning rotation and translation equivariant policies in order to learn object manipulation policies more efficiently. The method also provides a channel for non-equivariant information. The experiments are on 6 simulated pick-and-place tasks with real world evaluations. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The topic is relevant to CoRL. Increasing learning efficiency is an important problem in robot learning. Structural biases such as rotational and translational equivariance are sensible and applicable to many manipulation problems. \n- The experimental results show the method is able to perform well on a variety of pick and place tasks.\n- The soft-equivariant baseline is a nice comparison of data enforced equivariance vs model enforced. Does \" augmented three fold for rotations every 90 degrees\" match the number of rotations in the model? If not, then having an experiment where they do match would be the best comparison.\n\nClarity:\n- The method is very confusing to read. The architecture is based of off ASR [29], but ASR is barely explained. It would be easier to understand if the equivariant aspect were first explained for a normal FCN policy, and then the ASR was added. My understanding is that with ASR, each dimension of the action space is predicted sequentially. From the video, the policy looks like a simple FCN applied to each rotation of the image, but this is not what I understood from reading the paper. I don't understand how this architecture differs from the one in Form2Fit, which outputs embeddings for each of 20 rotations (I do understand that algorithmically, this paper outputs Q values and Form2Fit outputs embeddings, is that the only difference?).\n- The theoretical backing for the method is overwrought for what seems to be in the a fairly simple method, consider cleaning it up to keep only what is necessary. \n\nExperiments:\n- To disentangle the contributions of Q learning vs this approach to equivariance, the comparison to transporter could be made more fair by either evaluating a BC version of the proposed model or a Q-learning method of the transporter networks. \n\nQuestion:\n- Is the model always applied to images where the gripper is not visible?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper's clarity needs to be improved. It is difficult to figure out exactly what the method is and how it differs from prior work. This also makes the novelty of the method difficult to determine. I find the general direction of the work to be impactful, although the method seems limited to non-dynamic pick and place tasks.\n\n\n-----------\n\nUpdate after author response and improved paper\nI find the paper clarity improved and the new experiments convincing. I am in favor of accepting this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "HHTY_HTh6OV", "reviewer": "Reviewer_HHEb", "summary": "This paper proposes to bake in the equivariance constraints in the state-action space in deep Q-learning methods for robotic manipulation problems. The method is to utilize equivariant convolutions in addition to other symmetries (such as gripper symmetries) and additional state fed into the network. These new q-function architectures are then evaluated on a number of problems and are shown to be better.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes to actually utilize symmetries in the Q-function training, which is interesting since it can be done by just using a different architecture.\n2. The empirical sample-efficiency results seem quite better compared to the baselines on a number of domains.\n\nWeaknesses:\n1. The paper does not compare to some obvious baselines such as enforcing these symmetries via data augmentation. While this is discussed in related work, a comparison on at least some domains is needed.\n2. I would suggest reorganizing the paper to have a clearly defined method section with a summarized table of various options and then experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea proposed in the paper is interesting, but I am not sure of the baselines (e.g., data augmentation) and how important the method will be in the long run -- is it likely that we will be able to just use simpler techniques without having to modify the architectures for Q-functions and obtain similar sample efficiency?  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Atleast the data augmentation baseline must be added.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "1d1DtAYrCmp", "reviewer": "Reviewer_3ab5", "summary": "The paper studies the relevance and benefits of using SE(2) equivariant models in Q-learning in robotic problems. The paper identifies the conditions under which the Q function is SE(2) equivariant (Proposition 4.1). The paper, via experimental evaluation on various robotic manipulation problems, demonstrates the sample efficiency of the equivariant models over other models considered in the literature. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper studies an important problem. The study of equivariant/invariant models for Q-learning and imposing structure gleaned from the problem, is a great direction to pursue. This paper presents a good step in that direction. Prior works have not used equivariant architectures to model Q functions and thus this paper is novel and makes a worthy contribution. The experimental results are thorough and convincing.\n\nWeaknesses: The paper could have been better written.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper studies the relevance and benefits of using SE(2) equivariant models in Q-learning in robotic problems. The paper identifies the conditions under which the Q function is SE(2) equivariant (Proposition 4.1). The paper, via experimental evaluation on various robotic manipulation problems, demonstrates the sample efficiency of the equivariant models over others considered in the literature. \nPrior works have not used equivariant architectures to model Q functions and thus this paper is novel and makes a worthy contribution. The paper is not very well written.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Following are some of the issues, but a major re-drafting of the paper write-up would help the paper make its case even better. \n\n- Figure 3 should be placed close to page 2 where it is first referred to. \n\n- In Sec 3, the paper states “learning policies that are invariant to translation and rotation.” This is misleading and confusing. The policies need to be invariant to translation and rotation to state space is perhaps what they mean.\n\n- The abstract, and the write-up in general, are very verbose. Examples:\n** “we identify the conditions under which the optimal Q function is SE(2) equivariant, and equivariant Q learning is therefore applicable.”\n** Sentences 3-5 in the abstract.\n\nIt would have been good to briefly describe sample efficiency earlier in the Introduction as it is referred to so many times in the paper.\n"}}]

Author rebuttal:

Meta-review: The authors propose in this paper a Q function neural architecture that is invariant to rotations. The consensus among reviewers is the following.

Pros:
- The problem is important and applicable to many tasks.
- The robotics task is well chosen and results are encouraging.

Cons:
- Concerns regarding the fairness of the experimental setup, especially the data augmentation baselines (R-H (Reviewer HHEb), R-G, R-B).
- The Q function architecture beyond the equivarient convolutions not well explained (R-G).

The authors are encouraged to actively engage with the reviewers and revise their submission if they see fit. Concerns regarding the fairness of the comparisons w.r.t. data augmentation (R-H, R-G, R-B) are major and should be properly addressed so as to improve the quality of the submission. Authors should justify their choice of a more limited set of rotations for the data augmentation baseline (R-G, R-B) and ideally provide a baseline with matching settings. As suggested by R-3, a clearer definition of sample efficiency in the related work section (e.g. when discussing data augmentation), and experiments that support the claims made in that section---that appear sometimes too strong (R-B), would greatly improve the submission.

I thank the authors for their updates and the new baseline results. The rebuttal addressed most of the reviewers' concern and I thus recommend the paper for acceptance. If possible, in the final version the authors could also discuss the computational complexity, especially since data augmentation alternatives would not increase computational cost at test time.
Decision: Accept (Poster)