Equivariant Reinforcement Learning under Partial Observability
Authors: Hai Huu Nguyen
Andrea Baisero
David Klee
Dian Wang
Robert Platt
Christopher Amato
Venue: CoRL 2023 Poster (2023)
Keywords: Partial Observability
Equivariant Learning
Symmetry

Abstract: Incorporating inductive biases is a promising approach for tackling challenging robot learning domains with sample-efficient solutions. This paper identifies partially observable domains where symmetries can be a useful inductive bias for efficient learning. Specifically, by encoding the equivariance regarding specific group symmetries into the neural networks, our actor-critic reinforcement learning agents can reuse solutions in the past for related scenarios. Consequently, our equivariant agents outperform non-equivariant approaches significantly in terms of sample efficiency and final performance, demonstrated through experiments on a range of robotic tasks in simulation and real hardware.

Equivariant Reinforcement Learning under Partial Observability
Hai Nguyen, Andrea Baisero, David Klee, Dian Wang, Robert Platt, Christopher Amato Khoury College of Computer Sciences, Northeastern University, Boston, MA, United States nguyen hww.haartheetteeteoooeeeeaeaeeee edu https://sites. ot
Abstract: Incorporating inductive biases is promising approach for tackling challenging robot learning domains with sample-efficient solutions. This paper identifies partially observable domains where symmetries can be useful induc- tive bias for efficient learning. Specifically, by encoding the equivariance regard- ing specific group symmetries into the neural networks, ment learning agents can reuse solutions in the past for related scenarios. Con- sequently, our equivariant agents outperform non-equivariant approaches signifi- cantly in terms of sample efficiency and final performance, demonstrated through experiments on range of robotic tasks in simulation and real hardware.
actor-critic reinforce-
Keywords: Partial Observability, Equivariant Learning, Symmetry
Introduction
A key challenge in robot learning is to improve ple efficiency, i.e., to reduce the number of experi- ences demonstrations needed learn good pol- icy. One way to do this is to identify domain sym- metries that can structure the policy space. Recent works have demonstrated that symmetty--peeservigg (equivariant) neural network models are particu- larly effective way of accomplishing this [1, 2, 3, 4, 5]. However, these works have focused primar- ily on fully observable Markov decision processes (MDPs) rather than partially observable systems en- coded as partially observable MDPs (POMDPs) [6]. The question arises whether symmetric neural mod- els can also be used to solve Partially Observable Re- inforcement Learning (PORL) problems. This paper identifies the theoretical conditions under which this is indeed the case and describes an equivariant recur- rent model that works well in practice.
Goal
Rutation & Tranatation
Livodkes
LnN Dolko
Figure 1: Prawin-Opening" This POMDP is rotationally symmetric in the that optimal solution to the problem on the left (in blue) can be rotated obtain optimal solution to rotated version of the problem the right (in red).
To motivate, Fig. illustrates the grawer-Opening problem where robot is presented with chest containing two drawers, one locked and one unlocked. To solve this task, the robot must determine which drawer is unlocked and then open that drawer, relying only on top-down image observations. This task reflects a common POMDP when physical properties (whether drawer is unlocked) are hidden from the visual input. The only way for the robot to distinguish between the two drawers to attempt open one of them. This is classic feature of POMDP that the agent must perform information gathering actions to obtain information needed to solve the task. Notice that this problem is rotationally symmetric in the sense that its optimal solution (the blue end-effector trajectory in Fig. la) rotates (the red trajectory in Fig. 1b) when the scene itself rotates and is an example of the type of symmetry that want agents embed in their architectures.
We make three contributions in this work. First, we extend the framework of group -ovurinnvariant Markov decision processes [1] to the partially observable setting, resulting in new theory and solution
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

method. Specifically, we prove the optimal policy and the value function must be equivariant and in- variant in this new setting. Second, backed by the proof, we introduce equivariant actor-critic agents that inherently embed the domain symmetry in their architectures. Finally, we apply the agents in realistic robot manipulation tasks with sparse rewards, where our agents are shown to significantly outperform non-equivariant approaches in both sample efficiency and final performance. Our ap- proach's effectiveness is shown through simulated and real-robot experiments with equivariant and recurrent versions of Advantage Actor-Critic (A2C) [7] and Soft Actor-Critic (SAC) [8].
2 Related Works
Learning under Partial Observability Unlike classical planning-based methods 19, 10, 11] that impractically require the complete dynamics of the environment, bearningbbssed methods [12, 13, 14, 15, 16, 17, 18, 19] utilize recurrent versions of common reinforcement learning (RL) algorithms for policy learning by directly interacting with the environment. To speed up learning, some methods leverage privileged information assumed available during training, such as the states, the belief about the environment states, the fully observable policy [20, 21, 22, 23], which are orthogonal to our approach. Only a few prior works exploited domain symmetries under partial observability. Kang and Kim [24], Doshi and Roy [25] leveraged the invariance of the value function of some POMDPs given a state permutation and experimented on a classical planning- baasenggbbased method [11] with the above limitations. Recently, Muglich al. [26] used equivariant networks enforce symmetry when multiple agents coordinate. In contrast, we use model- -feee RL agents in a single-agent setting.
Equivariant Learning Equivariant networks have been successfully applied to a range of tasks such as point cloud analysis [27] and molecular dynamics [28, 29). A common approach is to build net- [30] which equivariant arbitrary symmetry groups, such as 2D [31, 32] and 3D [27, 33, 34, 35] transformations. Recently, for MDPs, equivariant net- works have been applied to robotics [2, 5] and reinforcement learning [36, 37] to improve sample efficiency. Closest to our work is [1]. which formalized group-invariant MDPs and used equiv- ariant networks to perform robotic manipulation tasks. In contrast, this work extends equivariant reinforcement learning to partially observable environments, resulting in a new theory and method.
works with group equivariant convolutions
Equivariance Data Augmentation Both methods leverage known domain symmetry to improve learning. but in different ways. On the one hand, data augmentation artificially expands the training data distribution with transformed versions of the data using the symmetry (e.g.. rotating, cropping, or translating images [38, 39]); then training a non-equivariant model. On the other hand, an equiv- equivariant
ariant approach bakes the domain symmetry directly into the model's weights, model can automatically generalize across input transformations even before training. Compared to an equivariant approach, model trained using data augmentation alone is often less sample effi- cient [32, 34], generalizes worse [40], and requires a bigger architecture and longer training time for the same performance due the extra work of learning symmetry injected in the data.
3 Background
Here, we review some background about POMDPs, some specific group theories used in our work, and finally, the basis of our approach the framework of
MDPs [1].
3.1 Partially Observable Markov Decision Processes
A POMDP is defined by a tuple (S, by, T. R, O), where S, A, and Ω are the state space, the action space, and the observation space, respectively. bo AS is the starting state distribution (a.k.a. the initial belief), states change and observations are emitted according to the stochastic dynamics function T(s,, a, s') and the stochastic observation function O(a, s', o), respectively. Generally, an optimal agent may need to choose actions based on the entire observable action-observanl observation history a;-1, 0t) [41]. Denoting the space of all histories as H. the goal is to find a history- policy π: H + AA which maximizes the expected discounted return J E
h4 = (op, au
oc -t= 7 R(s,, ar),, where 7 E [0. 1) is a discounting factor. An important concept in POMDPs is the belief b(s) = Pr(s] h), which is the probability that the true state is given an observed history h. The belief state is sufficient statistic of the history, sufficient for optimal control. However, updating the belief state requires complete knowledge of the POMDP dynamic models, which are often hard to obtain.
I>

---

(1(T/2)
P. (m/2)
A (T/2)
Figure 2: Illustration of pixel-wise rotation (characterized by fixed representation pr) and channel-wise rotation (characterized by the representation p). When is T/2 CCW rotation, Ps always rotates the pixels while the effect of p varies, c.g., the effect when pt (a) being a trivial representation (pt) acting on a 1-channel feature map, (b) being a standard representation (px) acting on vector field, and (c) being regular representation (p,) acting on 2-channel feature map.
3.2 Cn and SO(2) Symmetry Groups
In this work, we are mainly concerned with the symmetry group G = SO(2) of continuous planar rotation, defined as SO(2) Rot, {0 - < 2m). For reduced computation complexity, we use the cyclic subgroup Cn s SO(2) to approximate SO(2), which is defined as Cn {Rot, 0 E ( 0 < i < n)}. In other words, C₂ defines n rotations (i.c., group elements), which are multiples of 2st For instance, C4 (0, π 2/2, 2 /2, 3r/ [ and Cg {0, T/8,....... ..... ............. 6t/8, 7 /8).
3.3 Group Representations
group representation is a mapping from group G to d-dimensional general linear (GL) group, i.e. p G + GL. by assigning each group element E G' with an invertible matrix p(g) E
When G C.., the effect of rotation g Cn signal (i.e., gæ) starts with pixel-wise rotation ( (with a fixed group representation Py), followed by channel-wise rotation, i.e., gz = (p) =) (with the choice of group representation p). this work, we consider three choices of the channel-wise representation p:
Trivial Representation (p = (:)): For Vgg G. Pt associates g with an identity matrix. For example, in Fig. 2a when g is x x/2 counter-clockwise (CCW) rotation, and x is a 1-channel feature map, Pf while Pt does not change the pixel values (i.e., the colors are unchanged).
rotates the pixels
Standard Representation (p = pa): For Vg E G, Ps associates with a rotational matrix, i.e., pa(g) - g. As in Fig. 2b, when y is T/ CCW rotation and is a vector field input, Py rotates the positions of vectors (denoted as colored arrows), and rotates their orientations.
pr)): For each G, when acting permute the coordinates z. Fig. 2c illustrates when is a π CCW rotation and x a 2-channel feature map, Pf rotates each channel's pixels and pr permutes the orders of the channels.
Regular Representation
input z, Pr will cyclically
Illustrative Example Combining the group and the group representation fully characterizes how signal will be transformed. For an illustrative example in grid-world domain, see Appendix A.
3.4 Equivariance, Invariance, and Group-invariant MDPs
Given o: X' + y and symmetric group G that acts on X' and V, we say that o is G-equivariant if ő(gz) go(x), and G-invariant if ó(gz) = (xx). For the remainder of this document, we drop the prefix G and simply refer to these properties invariance and equivariance.
These notions have been adopted in the framework of group-invariant MDPs [1]. Specifically, an MDP MG (S, A, T, R) is invariant if the transition and the reward function invariant, i.e., "(gs,ga, ) = T(s, a, s') and R(gs, ga) = R(s, a). Group-invariant MDPs are associated with an invariant optimal Q-function, i.e., Q" (gs, ga) = Q" (s, a), and least equivariant determin- istic optimal policy, i.e., π7 (gg)) grr These properties were exploited to build very sample- efficient equivariant agents under full observability [2. 3.
Groui-Invariant POMDPs
In this section, we extend the ideas from [1] to POMDPs and identify the basic set of assumptions that a POMDP needs to satisfy to have analogous invariance properties. We also note that while other assumptions might also lead to an invariant POMDP, ours are probably the most natural.
3

---

Continuous
Me Mean
Ja Std.
Equivariant Actor
Obs. Extractor
RNN
Outputter
Pr(AA Discrete Action
0ç-1
Act. Fasture Extractor
Obs. Feature Extractor
Q at)
Criti Outputter
Invariant Critic 02
Obs. Feature Extractor
RNN
01-1
Act. Feature Extractor
Figure 3: Our equívariant agent takes the commonly used structure of memory-based actor-critic agent [13, 42, 43, 44] but consists of an equivariant actor and an invariant critic, each constructed by equivariant modules. The actor's output can be learned means and standard deviations (for continu- ous action spaces) or a categorical distribution over the action space (for discrete action spaces).
Definition 1. We say POMDP Pa = (S,A, bo, T, R, O) is group-invariant with respect to group G it satisfies the following invariant properties for all g E
Tggs, ga, gs') = s') R(gs.ga) = R(s,aa)
Ogga, gs go) = 0a,,6',0) bolgs) = by(s).
This extends the definition of the group invariant MDP from [1] by incorporating additional con- straints on the observation function and the initial belief distribution. We also extend the group operations histories.
Definition 2. Group operation 9 acts on history h according to gh, = (g00. gao..... g1,-1.90;).
Finally, we show that group-invariant POMDPs exhibit similar properties and benefits as group- invariant MDPs.
Theorem A group-invariant POMDP has an invariant optimal Q-function Q' (gh, ga) ('hha) (h, an invariant optimal value function (ghh) = (hh,,, and at least one equivariant deter- ministic optimal policy xt" (gh) = ggT"((]
Proof. See Appendix B.
The above analysis allows us to constrain the value function and policy for a G-invariant POMDP to be invariant and equivariant, respectively, without eliminating optimal solutions.
5 Equivariant Actor-Critic RL for POMDPs
In this section, we introduce an equivariant agent that directly exhibits the desired properties of the optimal value function and policy, backed by the analysis in Theorem 1. Fig. 3 shows the agent, which takes a typical memory-based agent [13, 42, 43, 44] but has an equivariant actor and an invariant critic, each consisting of equivariant models. We later show that this very generic agent, when embedded with the domain symmetry, can outperform significantly strong POMDP methods.
5.1 Equivariant Modules
We describe the details of the equivariant modules within our agent below, with the core components being equivariant CNNs [45, 32]. For the implementation details, please see Appendix D.
Equivariant Feature Extractor This module takes observations or actions and outputs intermediate features for further processing. It comprises multiple equivariant CNN components chained sequen- tially as shown in Fig. 4a. Its input representation is the observation representation Po or the action representation Pa- The intermediate and output representations are chosen to the regular repre- sentation pr. which empirically outperforms other representations [32]. The input representation can be a single representation type or mixed, i.e., a sum of different representations. mixed represen- tation is necessary when the input has different components that transform differently under group

---

Act./Obs. Feature Extractor
Actor Outputter
01-1
PS
CNN
Pr
CNN
P
CNN
f
Critic Outputter
Mixed Input Representation
CNN
Pr.
CNN
CNN Pr
CNN
CNN
/
Pey
Pr
ai-1
CNN
CNN Pr PT(()
Figure 4: Equivariant Feature Extractor and Actor/Critic Outputter modules.
transformation, e.g., one component rotates with the transform, and one component unchanged. Such be seen in Appendix D.2 and is simplified in Fig. 4b, where Pa Ps Pt-
Equivariant Actor Outputter Fig. 4c shows the input represen- tation is regular because the signal coming in (the output of the RNN and the observation feature extractor modules) uses a regular representation. The output representation varies depending on the action type (discrete/continuous) and how a group transformation will affect action. For discrete actions, the module produces categorical distribution over the action space. In this case, the out- Pr
Pr
Pr
CNN
Pr. 9t
2;
Pr
&. ft ② _-1 + O tanh(
4 of @ tanb(c,)
put representation is the regular (discrete) action when the history is transformed (see Appendix for illustration). For continuous actions, this module outputs the means with some representation Pa and the standard deviations of actions with some representation Po (as in A2C [7], PPO [46], SAC [8]). The representations used for Pμ. and Par are mixed, as each action component might change differently under a group transformation (see Appendix D.2).
want to change the
Figure 5: Equi LSTM cell.
Equivariant Critic Outputter Because the optimal critic is invariant, this module (Fig. 4d) uses the trivial representation Pr at the output to keep the output the same under a group transformation. Its input representation is regular, enforced by the output of the RNN and the action feature extractor.
Equivariant Recurrent Neural Network This our contribution needed for constructing POMDP equivariant agent (there is another similar component in [26], but only model weights are released). We utilize an LSTM [47] for this module, but the approach can also be modified for other types of RNNs. Specifically, given an input (e.g., the concatenated obs-action feature) and the previous hidden state 41-1 the input gate i, the forget gate ft, the memory cell candidate Ir, and the output o are computed as follows with W's and bs being learnable weights and biases:
i = sigmoid Wai + Whi44- + b)) ft = sigmoid(W ff + 1
+ by)
O: = sigmoid W,.,, - Whoh,- + b,) St = tanh(Waghe + Whghh-1 + bg)
The above equations do not make an equivariant RNN module. To enforce the equivariance, we compute all equations at once using an equivariant CNN module (Fig. 5), similar to the ConvLSTM network [48]. The input representation p, determined by the output of the feature extractors (x:) and the previous hidden state (h,-1) and the output representation is also regular. Next, we compute the next hidden state h, and cell state & using the common L.STM equations with @ denoting the Hadamard product:
G ft 10- + tanh(g:)
h @ tanh(c.).
(3)
Finally, as the output of the RNN an approximation the helief state, to satisfy the condition of an invariant initial belief distribution, we Co and hn with vectors.
Experiments
We compare the performance of learning agents two grid-world domains (discrete actions and feature-based observations) and four robot domains (continuous actions and pixel observations).
6.1 Domains
We briefly describe domains below. Please refer to Appendix C for more specific details.
5

---

Goal
Good
Pb. Pegan
Dad
Immevatle
Pod
Undeckml
Mosdlin
Immovabre
Ooel
Vovatle
Mouslee
CarFlag-1D
CarFlag-20 2D
Block- Blcck-iicking Block-Pulling Block-Pishinggraaee--ppnni
Figure 6: Our domains. The first two domains have feature-based observation and discrete action spaces. The last four domains have pixel-based observations and continuous action spaces.
6.1.1 Grid-World Domains
There two versions of CarFlag [49] in Fig. 6, where agent must reach goal (green), whose position is visible only when the agent visits an unknown information region. For instance, in CarFlag-1D, the agent must visit the central blue flag to get the side (left/right) of the goal; in CarFlag-2D, the agent must visit the central blue region to see the coordinate of the goal cell. We also illustrate the domain symmetry in the figure: in these domains, when the starting position and the goal location are transformed (flipped in CarFlag- 1D or rotated by n// radians clockwise in CarFlag-2D, the optimal trajectories will be transformed similarly, i.e., black blue trajectories.
6.1.2 Robot Manipulation Domains
Fig. shows our robot manipulation domains (extended from the BulletArm suite [50]), where robot arm must perform individual manipulation tasks (i.e., picking, pulling, pushing, and opening) using top-down depth images win a sparse reward. In these domains, only one object is manipu- lable, but both objects are the same if only relying on current image. Therefore, actively check the objects' mobility and remember past interactions with the objects to determine the next action. Specifically, in Delock-Picktin Block-Pulling, the agent needs to pull the movable block to be in contact with the other block. In Block-Pushing, the goal is to push the movable block to a goal pad. In Drawer-Open the agent is tasked open unlocked drawer between locked and unlocked one.
agent must
the agent needs to pick the movable block up. In
In these domains, the transition function is invariant because the Newtonian physics applied to the interaction is invariant to the location of the reference frame. The reward function is invariant by def- inition. Using top-down depth images makes observation function invariant. If the initial belief is assumed invariant, then according to Definition these domains are group-invariant POMDPs.
6.2 Agents
We compare our proposed agents (instances the structure in Fig. 3 applied to A2C [7] and SAC [8]) against diverse set of baselines, including ppolcc//off-oolie//e bosdel-basedmmodel.e and generic/specialized POMDP methods (see Appendix and Appendix for more details).
6.2.1 Grid-world Domains
RA2C [51] is recurrent version of A2C [7]. Equi-RA2C is our proposed architecture applied to A2C. DPFRL [14] is statc-of-the-arr model-based POMDP baseline where an A2C agent is given features produced by differentiable particle filter. DreamerV2 [52] and DreamerV3 [53] are strong model-based methods that learn recurrent world model, thus, can work with POMDPs.
No Data Augmentation for All Since all methods on-policy algorithms, augmented data using the domain symmetry only becomes on-policy only for Equi-RA2C, dues to its unique symmetry- awareness. Therefore, for fair comparison, do not perform any data augmentation.
6.2.2 Robot Manipulation Domains
While on-policy RA2C or Dreamer-v2 can handle continuous action spaces in these domains, there no clear way to leverage expert demonstrations necessary efficiently solve the robot manipula- tion tasks with sparse rewards in Fig. 6. Thus, we switch to SAC (8] as the base RI. algorithm, where

---

CarFlag-ID ID
CarFlag-2D
Block-Picking
EquiRADC
EquARADC
Equi
(PKR)
RSAC
MAIC
Onvarrnnd2
MALC
Orvarrnnd2
0.8
Daxs bap a3de (L5in
0.6
0.2
Evallation Success
6.0
1.0K
1.KK
Block-Pulling
Pushing
Drawer- Opening
5.0
1.0
1.0
RESA
80.5
ROlD Coop wiid
Roe- TEuGRSAC (LSTRO)
AMM
IAA MAD-Crrp--ie AGAC
Cqpl PSAC
SLAC
DIC Siih- ROAC
0.4
0.4
EvO Stilt POAC
AKD Crop KSAC
S.AC
0.2
0.0
10k
10k 20k
30k
40k
Emvironment Step
Figure 7: Evaluated success rates (four seeds, shaded areas denote one std.). No data augmentation is used in CarFlag domains. Rotational data augmentation used for all agents in robot domains.
can pre- pre-populate its replay buffer with demonstration episodes. In our experiments, RSAC [13] is a non-equivariant recurrent SAC agent. Equi-RSAC is our proposed method applied to SAC. We also compare with recurrent versions of two strong data augmentation baselines: RAD-Crop- RSAC [54] and DrQ-Shift-RAAC [55]. These specific data augmentation techniques, i.e., random cropping and shifting (see Appendix G for visualizations), are chosen among others because they were reported to perform best [55]. To train RAD- MADD-Cro-RSAAC for each training episode, an auxil- iary episode is created by using the same random cropping for every depth image inside the original episode. DrQ-Shift-RSAC applies two random shifts to each depth image in a training episode to create two. The Q-target and the Q-values are then computed by averaging the values computed on the two episodes. Finally, SLAC [56] learns a latent model from pixels and then uses SAC on the latent space by using the observation-action history (instead of the latent state) for the actor and the latent state samples for the critic. This enables SLAC to scale to more difficult tasks.
Rotational Data Augmentation for All All replay buffers pre- pre-populated with 80 expert episodes to overcome the reward sparsity. Moreover, we augment the training data by applying the same random rotation for every action and observation inside training episode (see Appendix G for visualizations). Note that these rotational data augmentations are applied in addition to the existing data augmentation techniques in RAD-Crop-RSAC and DrQ-Shift-RSAC.
Demonstrations
6.3 Results
Grid-world Domains Fig. 7 shows that Equi- is significantly more sample efficient than the baselines. Morcover, the dominance of our method is also seen with variants of these domains with different sizes (see Appendix 1). DPFRL did not perform well potentially because of the reward sparsity, which was also previously reported in [57]. DreamerV2 and DreamerV3 also perform poorly, even with many more learnable parameters of the models, potentially indicating that learning good model under partial observability and sparse rewards might be more challenging than in the domains originally tested. For instance, most Atari games and locomotion tasks the DeepMind Control suite [58] have low levels of partial observability and provide dense rewards.
d1d
Equi-RA2C IM (t)
RA2C 1M (1)
d2D
Equi-RAZC 2M (1)
RA2C 2M (t)
-10
0.51 + 0.06
0.78 + 0,08
-2
0.38 + 0.11
0.31 + 0.04
0.44 + 0.07
0.70 + 0.22
2
0.41 + 0.12
0.28 + 0.02
-5
0.95 + 0.03
0.72 + 0.33
-1
0.58 + 0.24
0.38 + 0.13
5
0.99 + 0.03
0.76 + 0.24
0.71 + 0.14
0.30 + 0.03
Table 1: The convergent success rates (mean + one standard deviation) of Equi-RA2C and RA2C
agents in asymmetric variants of CarFlag (after IM and 2M training steps). dID and dep refer to the distance from the information region to the world center (see Appendix C for illustrations).
7

---

Robot Manipulation Domains Clearly from Fig. 7, Equi-RSCC strongly outperforms other base- lines in all domains, with itself being the only agent that can reach satisfactory performance. Across all domains, without the equivariant LSTM module (denoted as Non-Equi-RSAC (LSTM), which can be considered as a naive extension of [T], the performance degrades significantly, even though it starts pretty well. SLAC, surprisingly, performs the worst. A possible reason is that SLAC was originally only tested on domains with dense rewards and low levels of partial observability (e.g., locomotion domains in DeepMind Control Suite [58] and OpenAI Gym [59]). Another po- tential reason is the usage of concatenated feature vectors across an episode for the actor, which can be very high- eigh-dimensional for long episode. Moreover, also found that the trained latent model failed to sufficiently reconstruct the observation in Block- Boock-Pulling (see Appendix for more details).
Additional Results See Appendix for the performance when using a different group symmetry (O's instead of C4), utilizing symmetry partially (for either actor or critic only), and Co and ho being random instead of zero vectors. Other additional results are shown in Appendix
6.4 Using Equivariant Models on Domains with Imperfect Symmetry
We investigate the performance when the perfect symmetry does not hold in asymmetric variants of CarFlag, created by offsetting the information region distance from the world center (see Ap- pendix C). From the final success rates shown in Table 1 (see Appendix I.1 for learning curves), we can see that equivariant Equi-RA2C still outperforms non-equivariant RA2C when the domains are close to perfect symmetry, i.e., when d d1 5 in CarFlag- 1D or d d2n 1,2 in CarFlag-20 2D. However, a bigger symmetry gap might lead to the sub-optimality of equivariant agents. As evidence, Equi-RA2C performs worse than RA2C when d = 10, - 10 in CarFlag-1D.
6.5 Zero-shot Transfers to Real Hardware
Combiic Point Choes
Sapeh Comees
Domain
Success Sim / Real (1)
Orthoggaricc Projction
Block-Picking
1.00 / 0.90
Block-Pulling
1.00 0.88
Block-Pushing
0.96 / 0.92
Draver-Opening
0.95 / 0.80
Table 2: Average success rates of sim2real trans- fers over 50 episodes.
Figure& 8:
Experimental robot setup.
Because only agents can perform well in simulation, we transfer their best policies in simulation to a URS robot (see Fig. 8). We combine the point clouds from two side-view cameras to create a top-down depth image using a projection at the gripper's 's position. We roll out 50 episodes, divided equally into test cases when the agents first manipulate the immovable or movable objects. Table shows that the learned policies can be zero-shot transferred well in the real world regardless of small performance drops in all domains (see our supplementary video for policy visualizations). The biggest performance drop in Draw-Opening, in which the transferred policies sometimes clumsily move one drawer far away from the other. creating a novel scene never seen in simulation.
7 Conclusion and Limitations
Conclusion In this work, we introduced group-invariant POMDPs and proposed equivariant actor- critic RI. agents as an effective solution method. Through extensive experiments, our proposed cquivariant agents can tackle realistic and challenging robotic manipulation domains much better than non-equivariant approaches with learned policies zero-shot transferable to real robot.
Limitations limitation of most equivariant approaches, including ours, is the requirement of im- perfect symmetry, which might be present when images are affected by non-symmetric factors, e.g., side view instead of top-down view or asymmetric noises. Fortunately, under full observability, re- cent empirical [60, 61] and theory work [62] show that an equivariant model can still outperform non-equivariant approaches many such cases. Together with the results in Section 6.4, our ap- proach might still perform better than unstructured agents even under imperfect symmetry.
8

---

Acknowledgments
This material is supported by the Army Research Office under award number W91 the U.S. Office of Naval Research under award number N00014-19-1-2131; NSF grants 1816382, 1830425, 1724257, and 1724191.
References
[1] D. Wang, R. Walters, and Platt. SO(2) equivariant reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
[2] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant learning in spatial action spaces. In Conference on Robot Learning, pages 1713-1723. PMLR, 2022.
[3] D. Wang, M. Jía, X. Zhu, R. Walters, and Platt. On-robot learning with equivariant models. In Conference On Robot Learning, 2022.
[4] H. Huang, D. Wang, R. Walters, and R. Platt. Equivariant Transporter Network. In Proceedings of Robotics: Science and Systems, 2022.
[5] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and Platt. Sample efficient grasp learning using equivariant models. In Proceedings of Robotics: Science and Systems, 2022.
[6] K. Astrom. Optimal control of markov decision processes with incomplete state estimation.
J. Math. Anal. Applic., 10:1772205 1965.
[7] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pages 1928-1937. PMLR, 2016.
[8] T. Haarojja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[9] H. Kurniawati, D. Hsu, and W. S. Lee. Sarsop: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and systems, 2008.
[10] A. Somani, N. Ye, Hsu, and W. S. Lee. Despot: Online POMDP planning with regulariza- tion. Advances in neural information processing systems, 26, 2013.
[11] Pineau, Gordon, S. Thrun, et al. Point-based value iteration: An anytime algorithm for POMDPs. In Ijcai, volume 3, pages 1025-1032, 2003.
[12] M. Hausknecht and Stone. Deep recurrent Q-leaming for partially observable MDPs. In 2015 aaai fall symposium series, 2015.
[13] Ni. B. Eysenbach, and R. Salakhutdinov. Recurrent model-free RL can be strong baseline for many POMDPs. In International Conference Machine Learning, pages 16691-16723. PMIR, 2022.
[14] X. Ma, P. Karkus, D. Hsu, W. S. Lee, and N. Ye. Discriminative particle filter reinforcement learning for complex partial observations. International Conference on Learning Represen- tations (ICLR), 2020.
[15] D. Han, Doya, and Tani. Variational recurrent models for solving partially observable control tasks. In International Conference on Learning Representations (ICLR), 2020.
[16] Z. Yang and H. Nguyen. Recurrent off off-policy baselines for memory-based continuous control. In Deep RI. Workshop NeurlPS 2021, 2021.
[17] N. Heess, J. Hunt, Lillicrap, and D. Silver. Memory-based control with recurrent neural networks. arXiv preprint
2015.

---

[18] M. Igl. L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for POMDPs. In International Conference on Machine Learning, pages 2117-2126. PMLR, 2018.
[19] L. Meng, R. Gorbet, and D. Kulić. Memory-based deep reinforcement learning for POMDPS. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-5626. IEEE, 2021.
[20] H. Nguyen, B. Daley, X. Song. C. Amato, and R. Platt. Belief groendddrondedd networks for accel- erated robot learning under partial observability. In Proceedings of the 2020 Conference Robot Learning, volume 155, pages 1640-1653, PMLR, 2021.
[21] H. Nguyen, A. Baisero, D. Wang, C. Amato, and R. Platt. Leveraging fully observable policies for learning under partial observability. In 6th Annual Conference on Robot Learning, 2022.
[22] A. Baisero, B. Daley, and C. Amato. Asymmetric dqn for partially observable reinforcement learning. In The 38th Conference on Uncertainty Artificial Intelligence. 2022.
[23] A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial ob- servability. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 44-52, 2022.
[24] B. K. Kang and K.-E. Kim. Exploiting symmetries for single-and multi-agent partially observ- able stochastic domains. Artificial Intelligence, 182:32-57, 2012.
[25] F. Doshi and N. Roy. The permutable POMDP: fast solutions to POMDPs for preference elicitation. Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems- Volume 1, pages 493-500, 2008
[26] D. Muglich, C. Schroeder de Witt, E. van der Pol, Whiteson, and J. Foerster. Equivariant networks for zero-shot coordination. Advances in Neural Information Processing Systems, 35: 6410-6423, 2022.
[27] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and Riley. Tensor field networks: Rotation-an and canslation-eqqivarian neural networks for 3d point clouds. arXiv preprint 1802.0880 19, 2018
[28] V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. In International Conference on Machine Learning, pages 9323-9332. PMLR, 2021.
[29] S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky. E equuqiirrrii graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11, 2022.
[30] T. Cohen and M. Welling. Group equivariant convolutional networks. In International Con- ference on Machine Learning, pages 2990-2999. PMLR, 2016.
[31] Y. LeCun, Y. Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
[32] M. Weiler and G. Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Infor- mation Processing Systems, 32, 2019.
[33] H. Chen, S. Liu, W. Chen, H. Li, and R. Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 14514-14523, 2021.
[34] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical cnns. arXiv preprint arXiv: 1011010010........11111111110.11 10130, 2018.
[35] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. Vector neu- rons: A general framework for so (3)-equivariant networks. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 12200-1209,, 2021.
10

---

[36] A. K. Mondal, P. Nair, and K. Siddiqi. Group equivariant deep reinforcement learning. arXiv preprint arXiv:2007.03433 034337, 2020.
[37] E. van der Pol. D. Worrall, van Hoof, F. Oliehoek, and M. Welling. MDP homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Pro- 4210,, 2020.
cessing Systems, 33:4199 4210,
[38] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[39] A. Zhan, R. Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. Deep RL Workshop NeurIPS 2021, 2021.
[40] R. Wang, R. Walters, and R. Yu. Incorporating symmetry into deep dynamics models for improved generalization. In International Conference on Learning Representations (ICLR), 2021.
[41] S. P. Singh, T. Jaakkola, and M. I. Jordan. Learning without state-estimation in partially observ- able markovian decision processes. In Machine Learning Proceedings 1994, pages 284-292. Elsevier, 1994.
[42] D. Ha and J. Schmidhuber. World models. arXiv preprint
2018.
[43] L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: very good method for adaesiad deep rl via meta-learning In International Conference on Learning Representations (ICLR), 2020.
[44] C.-C. Hung, T. Lillicrap, J. Abramson, Y. Wu, M. Mirza, Carnevale, A. Ahuja, and G. Wayne. Optimizing agent behavior over long time scales by transporting value. Nature commurications, 10(1):1-12, 2019.
[45] M. Weiler and G. Cesa. General E(2)-Equivariant Steerable CNNs. Conference on Neural Information Processing Systems (NeurIPS), 2019.
[46] Schulman, Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
[47] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1735 1780, 1997.
[48] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural in- formation processing systems, 28, 2015.
[49] H. Nguyen. POMDP robot domains.
https://ggitub..comhaiiiee
pomdp-domains 2021.
[50] D. Wang, C. Kohler, X. Zhu, M. Jia, and R. Platt. Bulletarm: An open-source robotic ma- nipulation benchmark and learning framework. In The International Symposium of Robotics Research, 2022.
[51] I. Kostrikov. Pytorch implementations of reinforcement learning algorithms. https:// // github. wwmm//oostriooo/ptttoooo wwwpoorchhhhcpppppe --cc-po--ee ppw-ppppo--a---eeee acktr-gaill gail, 2018.
[52] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International Conference Learning Representations (ICLR), 2021.
[53] D. Hafner, Pasukonis, Ba, and Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301 2.Xx:22011.0404,, 2023.
[54] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances neural information processing systems, 33:19884-1989, 2020.

---

[55] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations (ICI.R), 2021.
[56] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein- forcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
[57] H. Nguyen, Z. Yang, A. Baisero, X. Ma, R. Platt, and C. Amato. Hierarchical reinforcement learning under mixed observability. In Algorithmic Foundations of Robotics XV: Proceed- ings of the Fifieenth Workshop on the Algorithmic Foundations of Robotics, pages 188-204. Springer, 2022.
[58] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:/80L.0069 2018.
[59] G. Brockman, V. Cheung, L.. Pettersson, Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint 1606.01 2016.
[60] D. Wang, J. Y. Park, N. Sortur, L. L. Wong, R. Walters, and R. Platt. The surprising effective- ness of equivariant models in domains with latent symmetry. arXiv preprint arXiv:2211.09231, 2022.
[61] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. arXiv preprint arXiv:2304.0120// 2023.
[62] D. Wang, X. Zhu, J. Y. Park, R. Platt, and R. Walters. A general theory of correct, incorrect, and extrinsic equivariance. arXiv preprint arXiv:2303. 04745, 2023.
[63] K. X. Nguyen. Converting POMDPs into MDPs using history representation. 2021.
[64] L. P. Kaclbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
[65] G. Cesa, L. Lang, and M. Weiler. program build E(N)-equivariant steerable CNNs. In International Conference on Learning Representations (ICLR), 2022.
[66] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[67] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
12


Reviews:
[{"review_id": "owAxYoyKmp", "reviewer": "Reviewer_jfAZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "this paper extent equivariant MDP to equivariant POMDP. A theoretical justification and a particular implementation with deep neural networks (recurrent, MLP and transforms) are provided. The method is demonstrated both in simulated and real robots. ", "summary_of_recommendation": "a complete story is provide: idea, it theoretical justification, and demonstration both on the simulators and a robot. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. line 102: what does \"fiber-wise rotation\" mean? is there a formal definition beyond an illustrative example?\n\n2. line 148: \".. memory-based agent..\". why is the agent memory-based? How does it follow from Theorem 1? Is it an assumption? Heuristics?\n\n3.  What is you design criteria for the scheme in Figure 3? Is it the only scheme? What are special in this scheme?\n\n4. \"We later show that this very general agent,\" is not a very general agent, but a memory-\nbased agent ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "see please \"Questions For Rebuttal\" ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "aH9XseJ01_", "reviewer": "Reviewer_Pu8X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper extends group invariant MDPs to the POMDP case. It introduces an equivariant LSTM cell and an actor critic agent based on that architecture, that can efficiently learn to solve POMDPs by leveraging symmetries in the environment. The authors experimentally show that their algorithm outperforms baselines and zero-shot transfer the learned policies to a real robot.", "summary_of_recommendation": "The paper is well-written and easy to follow. The experiments seem solid and show substantial improvement over the provided baselines. Overall I have very view minor notes on improving the presentation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "see above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of the paper is nice, the overall presentation seems solid, and the paper is easy to follow. The experiments seem good and even include transfer to real hardware.\n\nI only have some minor points:\n- I am very familiar with these POMDP tasks, but for me the sidenote in line 232, that expert demonstrations are necessary in order to solve the task, was a bit surprising. Yes, the tasks is of course much harder, but is Dreamer really not able to solve it at all without expert demonstrations? I think providing a learning curve for dreamer on these tasks would really improve the claim.\n- evaluation on four seeds is really not that many.\n- I think the presentation of Figure 7 could be quite a bit improved. As of now it is a bit messy:\n  - bottom left environment is missing the title\n  - move the legends out out the plots and use a shared legend among the different plots. Right now it's a but confusing at first sight, especially since colors are reused between different environments for different algorithms.\n  - Pulling out the env names into titles for each plot would also look more organized. \n- I believe aligning Figures 2,3,4 at the top of the corresponding pages looks nicer and does not break the flow of text as much.\n\n--------------\n\nAfter rebuttal\n\n--------------\n\nI thank the reviewers for their response and the effort they put into it. I will keep my current score of weak accept for the paper.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Ue9YBwVRAF", "reviewer": "Reviewer_78Fy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper extends the framework of $SO(2)$- Equivariant RL from [1]  to the POMDP setting. Therefore,  the theory is extended to cope with histories instead of states, and an equivariant LSTM structure is proposed. The paper shows strong results in various tasks ranging from simple grid-world tasks to robotic manipulation tasks. \n\n\n\n[1] D. Wang, R. Walters, and R. Platt. SO(2) equivariant reinforcement learning. In International\nConference on Learning Representations (ICLR), 2022.\n", "summary_of_recommendation": "I believe that this work makes a notable contribution to the framework of equivariant RL. Hence, I vote for acceptance, but I require my questions to be answered. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- why are symmetries particularly important in POMDP?\n- how do the agents perform under dense reward signals?\n- does the decrease in performance of other agents come from sparse rewards?\n- to my understanding, invariant POMDPs require invariant hidden state transition functions, similar to the environment dynamics. Are the dynamics of the hidden transition -- shown in Equations (2) and (3) -- really invariant? Additional clarification is required here. \n- did you use convolutional layers for feature extraction for RSAC (and the other approaches)?  \n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and easy to read. The authors also provide extensive ablation in the appendix. This paper provides a missing piece in the field of equivariant RL. The experiments are well chosen, and the results are promising. \n\n### Strengths\n- well-written and easy to read.\n- provides a missing piece in the equivariant RL framework.\n- interesting experiments with promising results. \n\n\n### Weaknesses \n- the requirement of symmetries is often very limiting in robotics.\n- the question of why symmetries are particularly important in POMDPs is not answered.\n- the experiments with a huge number of expert demonstrations limit the results' impressiveness. \n- some questions remain.\n\n### Minor Corrections\n- line 108, p instead of $\\rho$\n- Fig 7: no label on the lower left plot; I assume these are the Block-Pulling examples", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "QFyniOsqJw", "reviewer": "Reviewer_e6B6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper extends equivariant reinforcement learning to POMDPs. Specifically, it shows that in group-invariant POMDPs, the optimal value function must be invariant and the optimal policy must be equivariant. To implement such a solution, this work introduces an equivariant actor-critic method that embeds the domain symmetry into its encoder network, recurrent network, and actor/critic components. The experiments are conducted in feature-based gridworld and a simulated pixel-based robot manipulation suite based on BulletArm. The built-in equivariance significantly improves the performance and sample-efficiency of the learning agent, compared to non-equivariant networks and those trained with rotational data augmentation.", "summary_of_recommendation": "This paper studies equivariant RL in symmetric POMDPs, and presents the advantages of doing so both empirically and theoretically. There are a few remaining questions I have in terms of its generality and whether related solutions may be applicable here, but I generally think this paper would be a good fit for CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why is data augmentation not applied to the on-policy algorithms? (Lines 228-229)\n\n- Why do you think the data augmentation methods plateau at such low success rates? It seems like it may be more than just sample-inefficiency. Even Non-equi-RSAC seems to outperform the variant with data augmentation, which is not obvious why.\n\n- Why does the method take 1M environment steps to learn in the CarFlag environments, while it takes 40-80K steps in the robot manipulation domains? The episodes seem to have comparable lengths.\n\n- How would other POMDP methods that leverage privileged information at train time, e.g., [20-23] referenced in the paper, perform on these tasks?\n\n- Could this method take advantage of symmetry that exists even if there are also asymmetric components in the problem?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- This work introduces an equivariant reinforcement learning method for POMDPs, which is a novel approach to the best of my knowledge.\n- The problem and solution are presented very clearly, and the visualizations are effective in illustrating the method components.\n- The results are very strong and demonstrate the advantage of encoding symmetry into the agent, both theoretically and empirically in several POMDP settings.\n\nWeaknesses:\n- The condition under which this method excels is when there is perfect or near-perfect symmetry in the problem. The symmetry must be known beforehand in order to encode this symmetry into each component network of the agent.\n- The experimental analysis is quite thorough. Perhaps, the main missing component is a comparison to POMDP methods that leverage privileged information about the task at train time, e.g., of the hidden state, a task identifier. The problems studied in the experiments can also be viewed as meta-RL tasks, so I wonder how a meta-RL method such as DREAM [1] would perform on them.\n\n[1] Liu et al. Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices. ICML 2021.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper makes a worthy contribution to CoRL -- it introduces a method that incorporates equivariance into RL agents in partially-observed problems. The paper is written well, and the experimental results are compelling. I encourage the authors to incorporate the reviewer feedback in the final version.
Decision: Accept (Poster)