Continual Vision-based Reinforcement Learning with Group Symmetries
Authors: Shiqi Liu
Mengdi Xu
Peide Huang
Xilun Zhang
Yongkang Liu
Kentaro Oguchi
Ding Zhao
Venue: CoRL 2023 Oral (2023)
Keywords: Continual Learning
Symmetry
Manipulation

Abstract: Continual reinforcement learning aims to sequentially learn a variety of tasks, retaining the ability to perform previously encountered tasks while simultaneously developing new policies for novel tasks. However, current continual RL approaches overlook the fact that certain tasks are identical under basic group operations like rotations or translations, especially with visual inputs. They may unnecessarily learn and maintain a new policy for each similar task, leading to poor sample efficiency and weak generalization capability. To address this, we introduce a unique Continual Vision-based Reinforcement Learning method that recognizes Group Symmetries, called COVERS, cultivating a policy for each group of equivalent tasks rather than an individual task. COVERS employs a proximal-policy-gradient-based (PPO-based) algorithm to train each policy, which contains an equivariant feature extractor and takes inputs with different modalities, including image observations and robot proprioceptive states. It also utilizes an unsupervised task grouping mechanism that relies on 1-Wasserstein distance on the extracted invariant features. We evaluate COVERS on a sequence of table-top manipulation tasks in simulation and on a real robot platform. Our results show that COVERS accurately assigns tasks to their respective groups and significantly outperforms baselines by generalizing to unseen but equivariant tasks in seen task groups. Demos are available on our project page: https://sites.google.com/view/rl-covers/.

Continual Vision-based Reinforcement Learning with Group Symmetries
Shiqi Liu!*, Mengdi Xu Peide Huang', Xilun Zhang'
Yongkang Liu?, Kentaro Oguchi', Ding Zhao
'Carnegie Mellon University. PR&D, Toyota Motor North America, "equal contribution (shiqiliu, mengdixu, peideh, xilunz, dingzhao Candrew cmu. edu (yongkang. liu, kentaro. kentaro.oguchi] (guuch]]ttyyeel @toyota. com
Abstract: Continual reinforcement learning aims to sequentially learn variety of tasks, retaining the ability perform previously encountered tasks while simulta- neously developing new policies for novel tasks. However, current continual RL approaches overlook the fact that certain tasks are identical under basic group op- erations like rotations or translations, especially with visual inputs. They may un- necessarily learn and maintain a new policy for each similar task, leading to poor sample efficiency and weak generalization capability. To address this, we intro- duce unique Continual Vision-based Reinforcement Learning method that rec- ognizes Group Symmetries, called COVERS, cultivating policy for each group of equivalent tasks rather than individual task. COVERS employs proximal- policy-gradient-basee (PPO-based) algorithm to train each policy, which contains an equivariant feature extractor and takes inputs with different modalities, includ- ing image observations and robot proprioceptive states. It also utilizes an unsu- pervised task clustering mechanism that relics on 1-Wasserstein distance on the extracted invariant features. We evaluate COVERS on sequence of table-top manipulation tasks in simulation and on a real robot platform. Our results show that COVERS accurately assigns tasks their respective groups and significantly outperforms baselines by generalizing to unseen but equivariant tasks in seen task groups. Demos are available on our project page'.
Keywords: Continual Learning, Symmetry, Manipulation
INTRODUCTION
Quick adaptation unseen tasks has been key objective in the field of reinforcement learning (RL.) [1, 2, 3]. RL algorithms usually trained in simulated environments and then deployed in the real world. However, pre-trained RL agents are likely to encounter new tasks during their deployment due to the non-stationarity of the environment [4, 5]. Blindly reusing policies obtained during training can result in substantial performance drops and even catastrophic failures [6. 7, 8).
Continual RL (CRL), also referred to as lifelong RL, addresses this issue by sequentially learning a series of tasks. It achieves this by generating task-specific policies for the current task, while simul- taneously preserving the ability to solve previously encountered tasks [3, 10, 11, 12]. Existing CRL works that rely on task delineations to handle non-stationary initial states, dynamics, reward functions can greatly boost task performance, particularly when significant task changes occur [10]. However, in realistic task-agnostic settings, these delineations are unknown and have to be identified by the agents. In this work, we explore how to define and detect task delineations to enhance robots' learning capabilities in task task-agnostic CRL.
Project Page: https://sitees google www.ggle.coo/veee/rr-oeeeees
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Our key insight is that robotic control tasks typ- ically preserve certain desirable structures, such as group symmetries. Existing CRL approaches typically delineate task boundaries based on sta- tistical measures, such as maximum posterioni estimates and likelihoods [10, 11] in the original observation space. However, such implementa- tions overlook the geometric information inherent in task representations, which naturally emerge in robotic control tasks, as demonstrated in Fig- ure Consider the drawer-closing example: con- ventional CRL works using image inputs would treat each mirrored configuration a new task and learn the task from scratch. Yet, we, as hu- mans, understand that the mirrored task configu- ration can be easily resolved by correspondingly reflecting the actions. Learning the mirrored task from scratch hampers positive task transfer and limits the agent's 's adaptivity. To address this issue, our goal to exploit the geometric similarity among tasks in the task-agnostic CRL setting facilitate rapid adaptation to unseen but geometri- cally equivalent tasks.
Equaertent:
Policy
Network
Reflect Tank Configuration
Ruflect
Action
This example illustrates how group symmetry enhances adaptability. The robot is instructed to close drawers situated in two dis- tinct locations given top-down images. The op- timal control policies are equivalent but mir- rored because of the symmetry of the drawers' locations around the robot's position.
Figure 1:
In this work, we propose COVERS, a task-agnostic vision-based CRL algorithm with strong sample efficiency and generalization capability by encoding group symmetries in the state and action spaces. We define task group as the set that contains equivalent tasks under the same group operation, such as rotations and reflections. We state our main contributions as follows:
COVERS grows PPO-based [13] policy with an equivariant feature extractor for each task group, instead of single task, solve unseen tasks in seen groups in zero-shot manner.
2. COVERS utilizes a novel unsupervised task grouping mechanism, which automatically detects group boundaries based on 1- W-Wasserttein distance the invariant feature space.
3. In non-stationary table-top manipulation environments, COVERS performs better than baselines in terms of average rewards and success rates. Moreover, we show that (a) the group symmetric information from the equivariant feature extractor promotes the adaptiv- ity by maximizing the positive interference within each group, and (b) the task grouping mechanism recovers the ground truth group indexes, which helps minimize the negative interference among different groups
2 Related Work
Task-Agnostic CRL. CRL has been long-standing problem that aims to train RL agents adaptable
to non-stationary environments with evolving world models [14, 15, 16, 17, 18, 7, 19, 20, 21, 22]. In task-agnostic CRL where task identifications are unrevealed, existing methods have addressed the problem through a range of techniques. These include hierarchical task modeling with stochastic processes [10, 11], meta-learning [3, 23], online system identification [24, 25], learning a repre- sentation from experience [12, 26]. and experience replay [17, 27]. Considering that in realistic situations, the new task may not belong the same task distribution as past tasks, develop an ensemble model of policy networks capable of handling diverse unseen tasks, rather than relying a single network to model dynamics or latent representations. Moreover, prior work often depends on data distribution-wise similarity or distances between latent variables, implicitly modeling task relationships. In contrast, we aim to introduce beneficial inductive bias explicitly by developing policy networks with equivariant feature extractors to capture the geometric structures of tasks.
Symmetries in RL. There has been surge of interest in modeling symmetries in components of Markov Decision Processes (MDPs) to improve generalization and efficiency [28, 29, 30, 31, 32, 33,

---

Plate Slide
Button Press
Drawer Close
Goal Reach
Streaming Groups
Timesteps
Figure 2: The continual learning environment setup involves four task groups, including Plate Slide, Button Press, Drawer Close, and Goal Reach. Groups streamingly come in.
34, 35, 36, 37, 38, 39J. MDP homomorphic network [30] preserves equivariant under symmetries in the state-action spaces of an MDP by imposing equivariance constraint on the policy and value network. As a result, it reduces the RL agent's solution space and increases sample efficiency. This single-agent MDP homomorphic network is then extended to the multi-agent domain by factorizing global symmetries into local symmetries [31]. SO(2)-Equivariant RL [32] extends the discrete sym- metry group to the group of continuous planar rotations. SO(2), boost the performance in robotic manipulation tasks. In contrast, seek to exploit the symmetric properties improve the general- ization capability of task-agnostic CRL algorithms and handle inputs with multiple modalities.
Preliminary
Markoy decision process. We consider Markov decision process (MDP) as 5-tuple (8, A, I' 7). where S and A are the state and action space, respectively. T A A(S) the transition function, R A + IR is the reward function, and y is the discount factor. We aim to find an optimal policy Te S > A parameterized by 9 that maximizes the expected return Eysro
-H 7²r (82; m.). where is the episode length.
Invariance and equivariance. Let G be mathematical group. f X + 3/ is mapping function. For transformation Lg X' - X' that satisfies f(x) = [[L. [=]), Vg G. x E X, we say f is invariant to Lg- Equivariance is closely related to invariance. If we can find another transformation Kg y + y that fulfills Ky[f(=)] = [Lg[=)), Vg E G, a E then we say f is equivariant to transformation T.g. It's worth noting that invariance a special case of equivariance.
MDP with group symmetries. In MDPs with symmetries [28, 29, 30], we can identify at least one mathematical group G of a transformation Lg S + S and state-dependent action transformation
K; A + A, such that R(s, a) = (L, [s), , K [a]) T (s. a, s') T (L, [s]. [a] Ly [s') hold for all g E &, E S, a E A.
Equivariant convolutional layer. Let G be Euclidean group, with the special orthogonal group and reflection group as subgroups. We use the equivariant convolutional layer developed by Weiler and Cesa [40], where each layer consists of G-steerable kernels : TR2 - TR that satisfies k(gz) = Poul (g) (g Yg E G, E IR2. Pmn and Pour are the types of input vector field fin : R2 Re and output vector field fout : I22 + Reas respectively.
Equivariant MIP. An equivariant multi-layer perceptron (MLP) consists of both equivariant linear layers and equivariant nonlinearities. An equivariant linear layer is linear function W that maps from one vector space Vnn with type Pm to another vector space with type Pout for a given group G. Formally Vr Vín Vgg E G Poun (g)(g) W z = W (g) Here we use the numerical method proposed by Finzi et [41] to parameterize MLPs that are equivariant arbitrary groups.
4 Methodology
4.1 Problem Formulation
We focus on continual learning table-top manipulation environments, where various tasks sequentially presented. We hypothesize that the streaming tasks can be partitioned into task groups, each containing tasks that share symmetry with one another. We adopt realistic setting where
3

---

Continual Vision-based RL with Group Symmetries Input: Threshold de, initial frame number k. update interval Nu. rollout step size N₂ Output: collection of policies
Algorithm COVERS:
Initialization: Current policy Teur initialized random policy with policy data buffer B 2, policy collection II + {(Teur, B. number of episodes n. f- 0, online rollout buffer D t- Ø
1: while task not finish do
2: n1 <
3: if n%NN, 0 then
4:
Rollout buffer O 2
D Unsupervised Policy Assignment
5: Rollout N₂ steps with Teur and get trajectories (So, oo..... 8H-1,
6:
Append the first & frames of each episode to rollout buffer O < {(su...., 8k 8k-1)
7:
Append the whole episode trajectories Calculate the 1- I-Wasserstein distances d!W (O, Bi), VT B:} E II (Equation 2)
the online rollout buffer D
8:
9:
Get the minimum distance dl where j = arg min; d!W (O, if d d, then Initialize a new random policy v as well as its policy data buffer B <- O Teur f- n, II < IT [T 13)
10:
11:
12:
13:
else
Assign the existing policy and buffer with Teur f Tj. Bj + B; U Update Teur hased on online rollout buffer 'D (Equation 1) t- Equivariant Policy Update
15:
16:
D
17:
else
18:
Sample episode and append online rollout buffer
new task group may emerge at each episode, the total number of distinct groups remains unknown, and the group may arrive in random orders. The primary objective is devise online learning algorithm capable of achieving high performance across all tasks with strong data efficiency. We visualize our CRL setting with table-top manipulation environments in Figure 2.
4.2 Algorithm
We present the pseudocode for COVERS, task- aask-agnottic continual RL method with group sym- metries, in Algorithm 1. COVERS maintains a collection IT {(T, B)), each element of which comprising pair of policy and its respective data buffer B. Each policy independently manages one group of tasks, with B storing the initial frames of the group oversees, At fixed time intervals, COVERS collects N, steps in parallel under the current policy Teur and stores the first k frames from each episode in the rollout buffer O Based on O the algorithm then either (a) creates new policy for an unseen group and adds it to the collection II, or (b) recalls an existing policy from the collection if the group has been previously encountered. It is worth noting that we assign poli- cies based on the initial frames of each episode rather than the full episode rollout. This is because frames corresponding to later timesteps are heavily influenced by the behavior policy and could eas- ily lead unstable policy assignments. Only maintaining subset of the rollout trajectories also helps alleviate memory usage.
After the policy assignment, the selected policy Teur with parameters 0 is updated based on an online rollout buffer D and the PPO method [13] with loss in Equation 1. Åg is the estimated advantage, Pt n
(ae|st)) /a (ar|s) (s2)) the importance ratio and is the clip range.
H
LCLIP EY~D
Σ
minl min(p: clip( e, c)A.
4.3 Policy Network Architecture
COVERS utilizes equivariant policy network that comprises a policy network for predicting ac- tions, value network approximating values, and an equivariant feature extractor taking multiple modalities. We show the policy architecture in Figure 3 and additional details in Figure 10.

---

Initial Frame
Equivariant Feature
Group MaxPooling
Inveriant Features
Equivariant Convalutional Networks
Metric
Current
Equiveriant Features
Value
Equivariant Linear
Equiveriant
Action
State &
Networks
Auxiliary Info.
Figure 3: Equivariant policy network architecture.
Equivariant feature extractor. In manipulation tasks, the observations typically comprise multiple modalities, such image observations, robot proprioceptive states, and goal positions represented in vector form. To accommodate these diverse modalities, we design an equívariant feature ex- tractor , that employs an equivariant convolutional network ceCon [40] for image processing, coupled with an equivariant linear network MP [42] to handle vector inputs. The resulting equiv- ariant features from these two pathways are concatenated to form the output of the feature extractor. Formally, h (s) Concat(h" Conv (s), ()M))((s))
Invariant value and equivariant policy. In the context of MDPs involving robotic manipulation tasks with group symmetries, it is known that the optimal value function maintains group invari- ance, while the optimal policy displays group equivariance [32]. To attain this, both the policy and value networks utilize shared equivariant feature extractor, designed to distill equivariant features from observations. Subsequently, the value network leverages a group pooling layer to transform these equivariant features into invariant ones, before employing fully connected layer generate MooMMaxxoooiingeeeeeggaa The policy network, the other hand,
values. Formally, (s) processes the equivariant features with additional equivariant MLP network output actions.
4.4 Unsupervised Dynamic Policy Assignment
In COVERS, we propose to detect different groups of tasks based on distances the invariant feature space. Such a mechanism facilitates knowledge transfer between tasks in each group. At a fixed episode interval, COVERS selects the policy of the group, whose data buffer B has the minimal dis- tance in the invariant feature space to the rollout buffer O col- lected in the current environment. Note that the invariant fea- tures of both O and are obtained through the feature extrac- tor of ग as shown in Figure 4. Considering that O and B may have a different number of data pairs, we take probabilistic perspective by treating those data buffers as sample-based rep- resentations of two distributions and use the Wasserstein distance measure the distance between those two feature distributions [43]. The invariant features are obtained from the equivariant feature extractor via a group max- max- max-iooling operation as shown in Figure 3.
Heu
online rollout Butfer 12
roliout buffer
Figure 4:
Calculation 1- Wasserstein distance and update of selected policy πj, whose data buffer has minimal distance O.
Wasserstein distance on invariant feature space. Here we show how calculate the distance to group {T,, B.) II. Let X and Y be matrices constructed by invariant features extracted from the state buffer B, of size and the buffer O of size m. X (X1, X2, X2..., Xx)T TT, Xp h (sp), [n), B, and Y (Yi, ..., ) Yi h)"" E [m)], O. We use the I-Wasserstein distance [44] to measure the distance between two empirical distributions X and Y. Hence the distance between O and B, is
d.W (0, (O,B.) (XXXYY) min (7. M) s.t. 71 a, T 1
b, 20.
where Mn = 1|X, Vill2.
|1/n...,, b 1/m]. M is the metric cost matrix.
5

---

Simulation Experiments
We validate COVERS's performance in robot manipulation [45] tasks with nonstationary environ- ments containing different objects following different reward functions. We aim to investigate whether our method can (I) recall stored policy when facing a seen group, as well automatically initialize new policy when encountering unseen group, (2) achieve similar or better performance compared to baselines, and (3) understand the significance of key components of COVERS.
5.1 Environment
Real
Simulation setup. Our manipulation setup is composed of four groups of tasks. Each group contains four tasks, and all tasks within the same group exhibit rotational reflectional symmetry with respect to each other. We build environments based on the Meta- World benchmark [45]. Meta-World fea- tures a variety of table-top manipulation tasks that require in- teraction with diverse objects using a Sawyer robot. We show the four groups of tasks in Figure 2 including Goal Reach for reaching goal position, Button Press for pressing the button with gripper, Drawer Close for closing drawer with gripper, and Plate Slide for sliding the plate to goal position. The goal positions and object locations of tasks in each group are symmetrically arranged around the center of the table. In our experiments, the four task groups arrive cyclically in order, as shown in Figure 2. The task order within each group and the initial configuration of each task randomized. We provide additional setup details in Appendix B.3.
Environment Setup
Original Top-dawn Image
Processed down
States and actions. The agent receives four kinds of observa- tions: an RGB image captured by a top-down camera centered over the table at each timestep, an RGB image captured by the same camera at the beginning of the episode, the robot state, including gripper's 3D coordinates and opening angle, and auxiliary information. The RGB image at the initial step helps alleviate the occlusion problem caused by the movement of the robot. The auxiliary information contains 3D goal positions, which are only revealed to the agent in Goal Reach since the goal locations are not visualized in the captured image and are masked out for other groups. To close the sim-to-real gap, we prepossess the RGB images by inpainting robot arms motivated by [46], with details deferred to Section B. comparison of the original and processed images visualized in Figure The action is niu-dimensional vector containing the gripper's s 3D positions and its opening angle. Con- sidering that utilize distinct robots: Sawyer in the simulation and Kinova in the real world, such action space and the image preprocessing mechanism help improve transferability.
Figure 5: Image preprocessing to narrow down the sim-to-real gap.
5.2 Baselines and Ablations
We compare COVERS with different methods detailed as follows. 3RL [26], an acronym for Replay-based Recurrent RL, is state-of-the-art method in CRL with Meta- Meta-World tasks that in- tegrates experience replay [17] and recurrent neural networks [47]. Note that augment 3RL with a convolutional neural network (CNN) to handle image inputs. In contrast, CLEAR [17], a com- mon baseline of CRL, only utilizes the experience replay by maintaining a memory buffer to store the experience of the past tasks and oversamples the current tasks to boost the performance in the current one. Equi utilizes a single policy with an equivariant feature extractor to solve tasks. CNN utilizes single policy with CNN-based feature extractor as vanilla baseline. We provide the detailed implementation of baselines and hyperparameters in Section B.
We compare with two ablation methods. COVERS-GT uses ground truth group labels to assign policies to different groups, which helps ablate the performance of our proposed policy assignment mechanism. COVERS-CNN utilizes a vanilla CNN block as the image feature extractor to help ablate the effect of using equivariant feature extractors.

---

1.2
COVERS CLEAR
COVERS-GT
COVERS-CNN
Success rate
2000 4000
0006
18003 12000
Number of episodes
Figure 6: Training curves for COVERS and other methods. Each background color corresponds task group. Each curve is averaged over runs, and the shaded area shows the confidence interval of 95%. COVERS shows similar performance with COVERS-GI which utilizes additional ground truth group indices, and substantially outperforms other baselines.
Policies index
2008
6008
0000
1800
12000
Number of episodes
Figure 7: The selected policies at each episode of COVERS. Each background color corresponds to one task group. The assigned policy indexes remain in alignment with the ground truth ones.
Simulation Results and Ablations
6.1 Results
Dynamic policy assignments. Figure 7 shows that when the environment switches to new group, COVERS quickly detects changes and initializes new policy for the group. Our method also recalls the corresponding policy from the collection when facing the same group again. Overall, the dynamic policy assignments generated by COVERS align well with the ground truth group labels. However, observe some instances where the policy assignment does not match the ground truth. This could potentially be attributed to the fact that the feature extractor of each policy may not be able to capture representative features for each group during the early stages of training. Notably, the rate of such misclassifications significantly reduces as the number of training episodes increases.
Training performance. We show the training curves of all methods in Figure and the quantitative performance in Table 2, including the average success rates and mean rewards. COVERS achieves much higher episode reward and success rate consistently in different groups than baselines. It is worth noting that although 3RL performs worse than COVERS, it achieves better performance than baselines with implicit task representations, including Equi, CLEAR, and CNN. This indicates that the explicit task representation used by 3RL, which maps transition pairs to latent variables using an RNN, facilitates the revelation of partial task identifications, thereby enhancing performance. underscores the significance of task-specific representations CRI.. In the early stages training, there isn't significant performance difference between COVERS and Equi However, as training progresses, COVERS begins to outperform Equi This is because COVERS avoids the problem of forgetting through the retraining of policies for each previously encountered task group. A compari- son between CNN and Equi reveals that incorporating group symmetries as inductive bias within the cquivariant network significantly enhances sample efficiency. This is achieved by only optimizing the policy for the abstracted MDP of each task group.
6.2 Ablation Study
The effect group symmetric information. COVERS SOVERS-CNN without the invariant feature extrac- tor demonstrates lower episodic rewards and success rates when compared with COVERS as shown in Table and Figure 6. From these results, conclude that the equivariant feature extractor signif- icantly enhances performance by modeling group symmetry information by introducing beneficial inductive bias through its model architecture.
7

---

Table 1: Quantitative results showing performances at convergence for different methods, including the average performance over five runs as well as the confidence interval of 95%.
Methods
3KL
Equi
COVERS-CNN
Plate Slide
Ave. Keward
Success Rate
344.04 044
0.97 0.02
12, 89
101,20
0.28 1: 0.06
7.36
63.65
0.06 0.03
2:25
23.44 2: 1.14
0.00
6
0.02
347.44
0.01 1 1.03
= 13.87
232.25
0.62 0.05
14.24
Button Press
Ave. Reward
Success Kale
923.4
0.87 1 0.014
+ 3.48
260.80 6.5G
1,52 0,06
138
0,31 1 0.36
12.25
0.08 1:00
+..34
121.13= 7.02
0.00
330
0.87
1 0.04
+ 2.63
181.21 10.83
0.26 1 0.00
Drower Close
Rase
0.82
0.35
7,68
116.33
0.36
273.26
0:00
417.36
227.3
Reach
Success Kate
Ave. Rewanl
683.53 1.35
0.38
322.25
0.00 0.00
17.59
243.5
16.16
151.24 14.11
306.72
20.34
488 02 0.35
480.96
0.39
1.06
Average
Ave. Reward
Sucuess Kate
387.77 1. 5.02
0.01 7: 0.02
241.21 - 7.39
0.44 = 0.05
178.5 7.58
0.30 0.05
96.59 3.30
0-1 4.002
191.28 - 6.23
0.22 0.02
393. 35
0.365 T 0.01
1 5.19
250.43 &.
F.00
The effeec of the dynamic policy assignment module. In Figure 6, COVERS's training curve is similar COVERS-GT which uses ground truth group indexes extra prior knowledge. Table shows that the performance drop due misclassification is minor considering the small standard deviation and COVERS's performance is within one or two standard deviations of COVERS-GT.
7 Real-world Validation
Real-world setup. Our real-world experiment setup utilizes a Kinova GEN3 robotic arm with a Robotiq 2F- 85 gripper. The top-down RGB image is captured with an Intel RealSense D345f. Gripper's coordinates and opening angle obtained through the robot's internal sensors. The real robot setups are demonstrated Fig- ure 8. We directly deploy the trained policies in simu- lation to the real world. Table 2 shows average success rates across 20 trials and shows that trained policies have strong generalization capability to real-world sce- narios. The performance drop compared with simula- tion experiments may be due to the inconsistent visual features and different scales of robots' action spaces.
Goal Reach
Button Press
Task Groups
Success Rate
Plate Slide
0.45 0.15
Drawer Close
Plate Slide
Button Press
0.60 + 0.15
Drawer Close
0.65 + 0.15
Figure 8: The real Kinova GEN3 setup with four task groups. The goal point marked in the figure is only disclosed to the agent in Goal Reach auxiliary information.
Goal Reach
0.92 + 0.07
Table 2: Real-world validation results.
8 Conclusion
We propose COVERS, novel Vision- based CRL framework that leverages group symmetries to facilitate generalization to unseen but equivalent tasks under the same group operations. COVERS detects group boundaries in an unsupervised manner based on invariant features and grows policies for each group of equivalent tasks instead of a single task. We show that COVERS assigns tasks to different groups with high accuracy, has a strong generalization capability, and maintains the capability to solve seen groups, outperforming baselines by large margin.
Limitation: One limitation of COVERS is that the memory it occupies grows linearly with the number of task groups. However, it is worth noting that COVERS still occupies less memory than maintaining policy buffer for each task by only storing representative data frames such as the ini- tial frames for each task group. Another limitation is that although assuming a top-down camera with a fixed base is widely adopted in existing works, it is hard to fulfill outside of labs. would be interesting to incorporate more general group operations, such as affine transformation and do- main randomization techniques, handle deformed images. Moreover, we only experimented with groups with equivariance structures. COVERS's performance is unknown in more complex scenar- ios with both equivariant and non-equivariant tasks.

---

ACKNOWLEDGMENT
The authors gratefully acknowledge the support from the National Science Foundation (under grants CNS-2047454) and research grant from the Toyota Motor North America. The ideas, opinions, and conclusions presented in this paper are solely those of the authors.
References
[1] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135, PMLR, 2017.
[2] A. Nagabandi, C. Finn, and S. Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv::812..0677 2018.
[3] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint
2018.
[4] M. Xu, P. Huang, Li, J. Zhu, Qi, Oguchi, Huang, H. Lam, and D. Zhao. Scalable cafety-critical policy evaluation with accelerated rare event sampling. In 2022 IEEE/RSJ In- ternational Conference Intelligent Robots and Systems (IROS), pages 12919- 12926. IEEE, 2022.
[5] Huang, Xu, F. Fang, and D. Zhao. Robust reinforcement learning as stackelberg game via adaptively-rreuularzee adversarial training. arXiv preprint arXiv:2202.09514, 2022.
[6] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 737-744. IEEE, 2020.
[7] K. Khetarpal, M. Riemer, I. Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research. 75:1401-1476. 2022.
[8] Huang, X. Zhang, Z. Cao, S. Liu, M. Xu, W. Ding. J. Francis, Chen, and Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. arXiv preprint arXiv:2306.1586 2023.
[9] K. Khetarpal, M. Riemer, Rish, and D. Precup. Towards continual reinforcement learning: review and perspectives. arXiv preprint arXiv:2012. 13490, 2020.
[10] M. Xu, W. Ding, J. Zhu, Z. Liu, B. Chen, and D. Zhao. Task-agnostic online reinforcement learning with an infinite mixture gaussian processes. Advances Neural Information Pro- cessing Systems, 33:6429 6440, 2020.
[11] H. Ren, A. Sootla, T. Jafferjee, J. Shen, J. Wang, and H. Bou-Ammar. Reinforcement learning in presence of discrete markovian context evolution. arXiv preprint arXiv:2202.06557 2022.
[12] A. Xie, Harrison, and C. Finn. Deep reinforcement learning amidst continual structured non- stationarity. In International Conference Machine Learning, pages 11393-11408 PMLR, 2021.
[13] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1.X.:v1777.064433 2017.
[14] S. Thrun and T. M. Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15 (1-2):25-46, 1995.
[15] F. Tanaka and M. Yamamura. An approach to lifelong reinforcement learning through multiple environments, In 6th European Workshop Learning Robots, pages 93-99, 1997.

---

[16] Z. Chen and B. Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207, 2018.
[17] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning, Advances in Neural Information Processing Systems, 32, 2019.
[18] M. Xu, Z. Liu, Huang, W. Ding. Cen, B. Li, and Zhao. Trustworthy reinforce- ment learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209, 200002200000005, 2022.
[19] Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, Quan, T. Ramalho, A. Grabska-Band Barskkkk, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351133526 2017.
[20] S. Powers, E. Xing, E. Kolve, R. Mottaghi, and A. Gupta. Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents. In Conference on Lifelong Learning Agents, pages 705-743. PMLR, 2022.
[21] H. Ahn, S. Cha, D. Lee, and T. Moon. Uncertainty- continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
[22] R. Traoré, H. Dusee,, T. Lesort, T. Sun, G. Cai, N. Díaz Rodrigzd and D. Fil- liat. Discorl: Continual reinforcement learning via policy distillation. arXiv preprini arXiv:190.005855, 2019.
[23] S. Sæmunnssoo,, K. Hofmann, and M. Deisenroth. Meta reinforcement learning with latent variable gaussian processes, arXiv preprint arXiv:1803.0755/ 2018.
[24] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint 1702.02500 2017.
[25] M. Xu, P. Huang, Y. Niu, V. Kumar, J. Qiu, C. Fang, K.-H. Lee, X. Qi, H. Lam, B. Li, et al. Group distributionally robust reinforcement learning with hierarchical latent variables. In International Conference on Artificial Intelligence and Statistics, pages 2677-2703. PMLR, 2023.
[26] M. Caccia, J. Mueller, T. Kim, L. Charlin, and R. Fakoor. Task-agnostic continual reinforce- ment learning: In praise of simple baseline. arXiv preprint arXiv:2205, 14495, 2022.
[27] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, H. Torr, and M. Ran- zato. Continual learning with tiny episodic memories. 2019.
[28] B. Ravindran and A. G. Barto. Symmetries and model minimization in markov decision pro- cesses, 2001.
[29] B. Ravindran and A. G. Barto. Approximate homomorphisms: A framework for non-exact minimization in markov decision processes. 2004.
[30] E. van der Pol, D. Worrall, H. van Hoof, F. Olichoek, and M. Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Pro- cessing Systems, 33:4199 4210, 42100, 2020.
[31] E. van der Pol, H. van Hoof, F. A. Oliehock, and M. Welling. Multi-agent mdp homomorphic networks. arXiv preprint arXiv:21 2021.
[32] D. Wang, R. Walters, and R. Platt. So (2) equivariant reinforcement learning. In International conference on learning representations (ICIR), 2022.
[33] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant 9 learning in spatial action spaces. In Conference Robot Learning, pages 1713-1723. PMLR, 2022.
10

---

[34] L. Zhao, X. Zhu, L. Kong, R. Walters, and L. L. Wong, Integrating symmetry into differentiable planning with steerable convolutions. In The Eleventh International Conference on Learning Representations, 2023. 2023..
[35] D. Wang, J. Y. Park, N. Sortur, L. L. Wong, R. Walters, and R. Platt. The surprising effective- ness of equivariant models in domains with latent symmetry. arXiv preprint arXiv:2211.09231, 2022.
[36] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and Platt. Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.09448, 2022.
[37] Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pages 2990-2999. PMLR, 2016.
[38] F. Fuchs, D. Worrall, V. Fischer, and M. Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970- 1981, 2020.
[39] M. J. Hutchinson, C. Le Lan, S. Zaidi, E. Dupont, Y. W. Teh, and H. Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference on Machine Learning, pages 4533-4543. PMLR, 2021.
[40] M. Weiler and G. Cesa. General (2)-equivariant steerable cnns. Advances in Neural Infor- mation Processing Systems, 32, 2019.
[41] M. Finzi, M. Welling, and G. Wilson. practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, pages 1318-33283 PMLR, 2021.
[42] G. Cesa, L. Lang, and M. Weiler. A program to build e(n)-equivariant steerable CNNs. In International Conference on Learning Representations, 2022. URL http::://o/oeeeeiieer net/ftorun?dd-WAAqslll
[43] P. Huang, M. Xu, J. Zhu, L. Shi, F. Fang, and D. Zhao. Curriculum reinforcement learn- ing using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35:16666-06670 2022.
[44] V. I. Bogachev and A. V. Kolesnikov. The monge-kantorovich problem: achievements, con- nections, and perspectives. Russian Mathematical Surveys, 67(5):785, 2012.
[45] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100, PMLR, 2020.
[46] S. Bahl, A. Gupta, and D. Pathak. Human-to-robor imitation in the wild. arXiv preprint arXiv:2207.09450, 2022.
[47] B. Bakker. Reinforcement learning with long short-term memory. Advances in neural infor- mation processing systems, 14, 2001.
[48] P. I. Etingof, O. Golberg, S. Hensel, T. Liu, A. Schwendner, D. Vaintrob, and E. Yudovina. Introduction to representation theory, volume 59, American Mathematical Soc., 2011.
[49] D. Steinley. Properties of the hubert-arable adjusted rand index. Psychological methods, 9(3):
386. 2004.

---

A Brief Introduction to Group and Representation Theory
In this section, we briefly introduce Group and Representation Theory [48] to help understand the policy structure Section B.2.
Linear group representations describe abstract groups in terms of linear transformations on some vector spaces. In particular, they be used to represent group elements linear transformations (matrices) on that space. A representation of group G' vector space V is group homomor- phism from G to GL(VV, the general linear group on V. That is, representation is map
p: - GV),), such that p(9192) = p(g)))(()))) V91.92 E G.
(3)
Here V is the representation space, and the dimension of V is the dimension of the representation.
A.1 Trivial Representation
Trivial representation maps any group element to the identity, i.c.
Ag E G. p(g) =
(4)
A.2 Irreducible Representations
representation of group is said be irreducible (shorthand as irrep) if has no non-trivial invariant subspaces. For example, given a group G acting on vector space V, V is said to be irreducible if the only subspaces of V preserved under the action of every group element the subspace and V itself. The trivial representation is irredueible representation and is common to all groups.
A.3 Regular Representation
Given a group G, the regular representation is representation over vector space V which has basis indexed by the elements of G. In other words. if G has n elements (if G is finite), then the regular representation a representation on a vector space of dimension n. An important fact about the regular representation is that can be decomposed into irreducible representations in a very structured way.
A.4 Dihedral Group
The dihedral group D, the group of symmetries of regular n-sided polygon, including n rotations and n reflections. Thus, D,, has 2n elements. For example, the dihedral group of a square (D₂) includes 4 rotations and 4 reflections, giving 8 transformations in total.
B
Additional Experiment Details
B.1
Image Inpainting
To close the sim-to-real gap, we employ pre-processing technique camera images, which in- volves in-painting robotic arms. The process begins by capturing background image in which the robotic arm is absent from the camera's view. For every time step, a mask that represents the position of each robotic limb is generated, leveraging the 3D locations of individual joints and the projection matrix of the camera. With this mask, we can select all areas devoid the robotic arm, and subsequently update the background image accordingly. The images are subjected to a color correction process mitigate any potential color deviations attributable to lighting or reflection. Lastly, distinct blue circle is overlaid at the gripper's 's position the background image to indicate the gripper's 's location. The entire image in painting process is shown in Figure 9.
12

---

Current image
Mask
Background image
Color correction
Processed image
Update
Figure 9: Image inpainting process.
B.2 Detailed Policy Architecture
In this section, we present the detailed model architecture including the model sizes and the types of each layer Figure 10.
In order make our policy network equivariant under transformations from the finite group D2, we need to choose the appropriate representation for both the network input and output, while also ensuring that the network architecture and operations preserve this equivariance.
The image input is encoded using the trivial representation. The robot state, on the other hand, encoded with a mixture of different representations: the gripper's position on the z-axis and the gripper's open angle encoded with the trivial representation since they invariant group actions in D2. The gripper's location the x and y-axes, however, encoded with two different non-trivial irreducible representations because their values are equivariant group actions in D2.
The value output is encoded with the trivial representation since the optimal value function should be invariant to group actions [32]. Finally, the action output is encoded with mixture of different representations. For actions, the gripper movement along the z-axis and the gripper's opening angle encoded with the trivial representation, while the gripper's location the x and y-axes encoded with two different non-trivial irreducible representations, aligning with the input encoding. The distance metric is encoded with trivial representation through the group pooling operation.
Nobot Stare
MLP. 64R
MLP. 64R
ML P mix
Action
Intial fra me
3x3 Conw 4R
7x2 Max pooling
3x3 Conv BR
2x2 Max cooling
BRR
2x2 Max scconn
3x3 CoIN: 22R
PxD Maxk Sroolie
3x3 Comy, 64R
3x3 Coin.
FC, IT
Value
Current kame
FC,
Glotal
Distance
Metric
Figure 10: Detailed equivariant policy network architecture. ReLU nonlinearity is omitted in the figure. A layer with a suffix of R indicates the layer output the regular representation. A layer with a suffix of T indicates the layer output is in the trivial representation. A layer with a suffix of mmxx means the layer output combines different representations.
13

---

B.3 Randomness of Tasks
In each task across all groups, we introduce randomness to the object's initial position and goal posi- tion by adding a perturbation value sampled from uniform distribution with a range (-0.02, 0.02) meters. We list the perturbed features as follows:
Goal Reach: the æyz coordinate of the goal position.
Button Press: the ry coordinate of the button's initial location.
Drawer Close: the zy coordinate of the drawer's initial location.
Plate Slide: the xy coordinate of the plate's destination.
B.4 Implementation of CLEAR
The CLEAR algorithm [17] addresses the challenge of continual learning by putting data from preceding tasks a buffer, utilized subsequently for retraining. This method effectively decelerates the rate of forgetting by emulating continuous learning setting. The specific network architecture for CLEAR is illustrated in Figure 11. To make CLEAR able to process both images and robot state as input, we introduce feature extractor, which harmoniously integrates a CNN and an MLP network. This composite feature extractor is carefully designed to contain a similar quantity of learnable parameters to our Equivariant feature extractor.
(4)
(b)
Images
3xB Conv.
MLP. 64
Feature
2x2 Max pooling
Conk:
2x2 Mex pouling
Mar paoiiig
MLP 64
MLP 84
MLP 64
363 Coriv 32
MLP.
MLP G4
MLP. 64
pooling
MLF:4
MLP 1
Action
Velue
Value
Figure 11: Network architecture for CLEAR. In (a) we show the network architecture of the actor network and the critic network. In (b) we show the structure of the feature extractor, which consists of both a CNN network and an MLP network. ReLU nonlinearity is omitted in the figure.
B.5 Implementation of 3RL
The 3RI algorithm [26] can be seen as an improved version of CLEAR, wherein additional historical data provided to the actor and critic from a dedicated context encoder. This historical data includes (st, a, r;), and the context encoder extracted task specificities from the history data with an RNN network. The specific network architecture for 3RL illustrated in Figure 12.
B.6 Hyperparameters
We show the hyperparameters of our proposed COVERS in Table 3. Moreover, we show the hyper- parameters of baselines in Table 4.
14

---

(8)
State
History Data
State
History
Action
History State
History Action Rewards
Contost Encoder
Festuee Ditacior
Exiecior
Brcockrr
MIP
MiP
MLP.
MER
MUP. 1
M.P:
Action
9, Value
Q, Value
Figure 12: Network architecture for 3RL. In (a), we illustrate the structure of both the actor and critic networks, whereas (b) highlights the configuration of the context encoder, comprising feature extractor and GRUs. It's noteworthy that the feature extractor has the same architecture as the CLEAR algorithm, as shown in Figure 11.
Table 3: COVERS Hyperparameter
Hyperparameters
Value
Wasserstein distance threshold de
1.0
Initial frame number
4
Update interval N₂,
1000
Rollout buffer size N,
1000
Batch size
64
Number of epochs
Discount factor
0.99
Optimizer learning rate
0.0003
Likelihood ratio clip range c
0.2
Advantage estimation X
0.95
Entropy coefficient
0.001
Max KL divergence
0.05
Table 4: CLEAR and 3RL Hyperparameter
Hyperparameters
Value
Common hyperparameter
Replay buffer size
200000
Discount factor
0.95
Burn in period
20000
Warm up period
1000
Batch size
512
Gradient clipping range
1-1.0,+1.00
Learning rate
0.0003
Entropy regularization coefficient
0.005
3RL Specific Hyperparameters
RNN's number of layers
RNN's 's context size
30
RNN's context length
5
15

---

C Additional Ablation Study
C.1
Sensitivity Analysis of Different Metrics
In Section 4.4, we used the I-Wasserstein distance to measure the distance between those two fea- ture distributions. In this section, compared the 1-Wasserstein distance with two other metries: Euclidean distance and Mahalanobis distance. We present the qualitative results in Figure 13a, 13b and 13c.
Adjusted Rand Index (ARI). To evaluate how different metric affects the algorithm performance, besides evaluating the converged performance, we further evaluated the Adjusted Rand Index (ART) [49] of the policy ID and group ID during the training progress. The ARI value measures the similarity between two clusterings by considering all pairs of samples. It counts pairs assigned to the same or different clusters in both the predicted and true clusterings. Here we used the group index as the ground truth label for each episode, while the policy index as the predicted label. Then we compute the ARI value between two clustering of the entire training process. An ARI value closer to 1.0 indicates a more accurate clustering result.
Euclidean distance (or 1,2 norm). The Euclidean distance between point and is
d(p,g) v (p -9)²
To compute the Euclidean distance between features of the state buffer B; of size and the buffer O, we computed the mean vector and y of buffer X and Y, the Euclidean distance is simply d(æ, = (z y) Here X and Y be matrices constructed by invariant features extracted from the state buffer B; of size n and the buffer of size m, as shown in Section 4.4. Here we test five different threshold de and run three random seeds for each threshold. The quantitative converged performance shown in Table 5.
Table 5: Quantitative results showing performances at convergence for different Euclidean distance threshold d,, including the average performance over three runs as well as the confidence interval of 95%.
Threshold d,
0.3
0.4
0.5
0.6
0.7
Plate Slide
Success Rute
0.91 = 0.04
0.92 + 0.04
0.93 + 0.04
0.97 +: 0.03
0.96 +:: 0.03
Ave. Reward
332.27 19.4
338.96
19.51
342.46 19.2
346.63 = 20.03
348.02 4 20.68
Success Rate
0.83
0.06
0.07
0.77
0.95
0.97 0.00
Button Press
Ave. Reward 323.49
6,92
304.63 + 9.71
298.75 + 10.67
334.18 + 2.08
332.27 + 3.0
Success Rate 0.93
0.04
0.98 + 0.02
(173 + 0.07
U187 + 0.05
0.98 + 0.02
Drawer Close
Ave. Reward 413.04 1 9.03
432.22 +: 6.14
387.01 +: 11.63
409.37 - 10.19
454.7 1: 3.25
Goal Reach
Success Rate
0.99 = 0.01
0.90 0.00
199900000
0.00
U.82 0.06
Reward
488.43
0.18
486 486.82-4 0.59
488 89 0.11
488, 82 0.08
480.46 2.35
Success Rate
0.92
0.02
0.91 0.02
0.86 :: 0.03
0.95 :: 0.02
0.34 +: 0.02
Averuge
Ave. Reward
389.31 + 7.77
390.66 8.11
379.28 = 8.4
394.75 : 7.48
403 86 7.42
Mahalanobis distance. The Mahalanobis distance measures the distance between point and
distribution D. It is defined as:
das dar(x) - (oG-N)-S-(( da(x))-(a--w)--1 pi),
(6)
where æ is the point, is the mean of the distribution D, and E is the covariance matrix of D.
We compute Mahalanobis distance between the mean vector y of Y and distribution X. Similarly, we test five different threshold d, and run three random seeds for each d. The quantitative converged performance is shown in Table 6.
Wasserstein distance. In Table we analysed COVERS performance under 1- I-Wasserstein distance with threshold of de = 1.0. Here we test five different threshold de and run three random seeds for each d, The converged performance is shown in Table 7.
Analysis. Our results show that the Wasserstein distance is less sensitive to the hyperparameter and performs well across different parameters. Moreover, the L2 distance yield satisfactory
16

---

Table 6: Quantitative results showing performances at convergence for different Mahalanobis dis- tance threshold d,, including the average performance over three runs as well as the confidence interval of 95%.
Threshold d,
4.0
5.0
7.0
Plate Slide
Success Rate
0.94 = 0.04
0.96 0.03
0.95 4: 0.03
0.9 = 0.05
0.31 : 0.07
Ave. Reward
332.75 + 20.91
330.25 + 20.56
329.31 22.13
333.61 19.43
138.03
22.72
Success Rate
0.81 0.06
0.83 0.06
0.7 0.07
0.94 0.04
0.24 0.07
Button Press
Ave. Reward
323.0 +: 3.83
321.37 4: 4.34
306, 89 4: 7.62
336.77 +: 0.76
301.48 6,24
Drawer Close
Success Rate
00.82 0.00
0.85 = 0.06
0.75 0.07
0.72 0.07
0.62
Ave. Reward
392.73 9.48
420.41 8.77
399.00
10.31
390.82 10.78
363.15 L1.7
Success Rate
0.59 0.01
0.96 = 0.03
0.99 = 0.01
0.99 0.01
0.9 0,05
Goal Reach
Ave. Reward
487.85 + 0.27
486.54 1.15
488.54
0.16
486.05 + 1.46
484.0 + 2.26
Sucress Rate
0.9 +: 0.02
0.91 +: 0.02
U.86 + 0.03
0.89 0.02
0.52 +: 0.04
Average
Ave. Reward
384.09 7.84
389.69
380.95 8.54
386.81 7.44
321.66 11.96
Table Quantitative results showing performances convergence for different 1- Wassersteii dis- tance threshold de, including the average performance over three runs as well as the confidence interval of 95%.
Threshold d,
1.0
1.2
1.4
Plate Slide
Success Rate
+: 0.1
0.94 0.04
+: 0.04
0.95
0.04
+: 0.06
Ave. Reward
196.34 + 31,64
339.21 + 26.44
340.1 + 24.42
352.65
24.72
336.82 24.02
Success Rate
0.48 U.1
0.83 4 0.07
0.88 0.06
0.94 :: 0.04
0.75 E 0.08
Button Press
Ave. Reward 272.01 + 13:25
316.08 0.41
324.98 4.51
326. 22 : 5.81
298.83 = 13:23
Drawer Close
Success Rate
0.97
0.03
0,89 0.06
0.91 -:
0.88 0.06
0.76 =: 0.08
Ave. Reward
433.62
7.79
410.59 +: 13.92
411.99
11.26
425.68 4: 8.38
393.62 12.99
Goal Reach
Success Rate
U.38 0.02
0.00
0.96
0,98 = 0.02
U.98 = 0.02
Ave. Reward
488.72 0.11
487.93 0.16
486.9 0.96
487.62 + 0.4
488 08.05-22 0.36
Success Rate
0.74
0.92
0.94 7.944:0.00
0.95 = 0.02
0.86 = 0.03
Average
Ave. Reward 1
388.45 10.23 390.99 :: 9.29
398.05 1:: 9.12
379.33 10.31
performance with the optimal hyperparameter selection. Such observations show that the invariant feature is more important in group identification other than the metrics.
C.2 The Effect of Buffer Size
We conduct ablation study to show the effect of the buffer sizes. We select five buffer sizes, including 32, 64, 128, 256, and 512. We show the results in Table 8 and Figure 13d. Our results show that when buffer size equals 128, COVERS achieves the best performance.
D Additional Training Results
D.1 Evaluation over Different Levels of Camera Perturbations
In this section, we present the converged performance of our algorithm under different camera per- turbation levels. For experiments with perturbation distance dys we randomly shift the zy coordinate
Table 8: Quantitative results showing performances at convergence for different buffer sizes, includ- ing the average performance over three well as the confidence interval of 95%.
Buffer size
32
128
256
512
Plate Slide
Success Rate
0.61 +:
0.94 0.04
0.95 0.04
0.96 = 0.03
0.94 0.04
Ave. Reward
240).42
25.11
310.47 + 20.19
340.1 + 24.42
331.52 + 21.91
347.38 + 20.26
Success Rate
0.55 +: 0.08
0.94 0.04
0.88 +: 0.06
0.49 +: 0.08
0.77 +: 0.07
Button Press
Reward 269.27
14.27
324.98
4.54
221.73
23.32
311.55 6.15
Success Rate (1.9 + 0.05
0.73 0.007
0.91 0.05
0.86 + 0.05
0.79++0.00
Drawer Close
Ave. Reward 380.41
13.72
378.98 + 11.03
411.99 (9.9 + 11.26
398.73 5.54
394.83
10.06
Success Rate
0196 +: 0.03
0.99 + 0.01
0.95 0.04
0.99 =: 0.01
0.99 0.01
Goal Reach
Reward
486.83
0.74
488.62 0.1
486.9 0.96
488
0.16
488.3 - 0.43
Success Rate
0.76 + 0.03
0.9 0.02
0.94 0.02
0183 = 0.03
0.88 : 0.03
Averuge
Ave. Reward
344.23 11.17
377.96 7.95
390.99 + 9.29
360.08 1: 11.37
385.52 7.9
17

---

Adquuted Rand Indes (A81)
Adjuseed Rand Indro (A81)
Adjusted fand trdder (Abe))
Adjustdd Rand Indox
(a)
(b)
(c)
(d)
Figure 13: ART value for analyzing the sensitivity of different metrics and the buffer size.
COVERS
3RL
CLEAR
CNN
Equi
COVERS GT
COVERS CNN
Reward
640
W00
3000 Lock
Number of episodes
Figure 14: Training curves for COVERS and other methods. Each background color corresponds to one task group. Each curve is averaged over 5 runs, and the shaded area shows variance. COVERS shows similar performance with COVERS- GTG, which utilizes additional ground truth group indices, and substantially outperforms other baselines.
by adding noise sampled from uniform distribution of range (-dp. dp). The results are shown in Table 9. We choose five perturbation levels, including dp = 0.01, 0.02, 0.03, 0.04, 0.05 in meters. Our results show that our trained policy can still achieve high performance even with large camera position shifts when d, = 0.05, indicating strong robustness to the camera perturbations. The task that suffers the most when the camera position shifts the Button Press, which we conjecture may be due the contact-rich nature of the task. It is worth noting that even when the equivariance between tasks is imperfect due to camera position shift, our policy still achieve success rates higher than 0.5 in 3 out of tasks and average success rate of 0.59 when dp 0.05.
D.2 Additional Training Setups
Training devices. We conducted COVERS training and ablation studies on a cluster of servers, with
different hardware configurations. The CPU model includes AMD RYZEN 9 3900X, AMD RYZEN 9 3900X, AMD RYZEN 9 5900x, and AMD RYZEN 7900x. The GPU model includes NVIDIA GeForce RTX 2080 Ti, NVIDIA GeForce RTX 3090, and NVIDIA GeForce RTX 4090.
Training time and memory consumption. Here, we show the total time to train different methods. Note that the absolute training time highly depends on server hardware, and our results are best
Table 9: Evaluation over Different Levels of Camera Perturbations
Perturbations level dp (m)
0.01
0.02
0.03
0.04
0.05
Success Rate
0.92 + 0.05
0.86 +: 0.07
0.78 0.07
0.8 1: 0.07
0.69 + 0.08
Plate Slide
Ave. Reward
299.06 + 23.95
253.54
22.36
261.21
22.31
264.79 4: 22.13
252.06
21.32
Success
0.45 0.09
0.17 0.07
0.24 0.07
0.14 1114:0000
0.08 0.05
Button Press
Ave. Reward
254.15 + 18.14
143.74
17.59
162.13 + 18.8
133.01 + 16.47
144.12
15.86
Success Rate
0.89
0.08
0.75 + 0.08
U.67 0.08
U.67 + 0.08
0.6 + 0.09
Drawer Close
Ave. Reward
365.87 + 18.03
313.79
19.11
287 09.0 20.57
270.05 +: 20.28
235.91
21.67
Goal Reach
Success Rate
0.98
0.02
0.98 +
0.98 0.02
0.98 0.02
0.95 0.02
Ave. Reward
488.06 0.7
488.56 + 0.27
488.61 0.23
488 63 0.13
488.49 0.45
Average
Success Rate
0.80 = 0.04
0.09 + 0.04
0.66 0.04
0.65 = 0.04
0.59 = 0.04
Ave. Reward
351.78 + 11.79
307. + 13.93
299.76 13.86
289.12 + 14.27
280.14 + 14.24
18

---

understood when comparing them relative to each other. The averaged training time for COVERS, CNN, Equi, COVERS-GT, COVERS-CNN is roughly the same, about 34 hours, while CLEAR and 3RL take 10 and 30 hours to train, respectively. For memory consumption, COVERS, CNN, Equi, COVERS-GT, COVERS-CNT CNN are roughly the same, about Gigabytes. For 3RL and CLEAR, the memory consumption is about 250 Gigabytes since they are off-policy algorithms and consist of large replay buffer that stores state-action pair. This could be problematic in setup since images as part of the state that dramatically increases memory consumption.
D.3 Qualitative Visualization using Training Rewards
Similar to Figure 6 that shows the success rates along training, we provide qualitative visualization using the task rewards in Figure 14.
19


Reviews:
[{"review_id": "mh-fezu7iJp", "reviewer": "Reviewer_WPhV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a continual reinforcement learning algorithm for robotic manipulation tasks that utilizes symmetries in the task space to improve sample efficiency and generalization. An equivariant feature extractor is utilized to facilitate training under symmetries within a task group, and subsequently an invariant latent representation of the initial rollout is used to determine if the current episode is from a task group that is new or already encountered. Wasserstein distance is used to determine whether the current task is significantly different from the previously encountered ones, in which case a new random policy is instantiated.", "summary_of_recommendation": "The topic addressed by the paper is very relevant for current robot learning research as its focus is on important challenges in the field such as sample efficiency, representation learning and continual learning. The paper will be beneficial to the community as it provides unified methodology that to some degree addresses all these challenges.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1) The explicit initialization of new policy for new task group prevents re-usage of useful features between task groups. Could we start from the closest policy and fine-tune it when we instantiate a new policy for a task instead of randomly initialized policy?\n\n2) In your experiments, are the 4 tasks of each group fixed (for example, the 4 tasks are 4 fixed target positions for the 4 Goal Reach tasks from the Goal Reach Group), or is a random position sampled for each training/test episode of this task?\n\n3) How many training runs do you do? What is the shaded area in Fig. 6 and the interval in the Table 1 representing (e.g., 95% confidence interval, standard deviation …)?\n\n4) How is the order of presentation during each training run? For example, are the most similar tasks from each group always presented one after the other (e.g. Plate Slide Top Left followed by Button Press Top Left), or is the order during different training runs randomized?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and well-structured, it contains sufficient experiments on hardware and explains the limitations correctly. The authors compare the proposed approach with other methods and variants of their method as baselines and show that it outperforms the other models. However, the experiment scenario feels quite simplified, there are several questions related to this below. \n\nIn the reviewer’s opinion there is a limited novelty as the methodology combines several existing approaches, however it combines them in a new and convincing way by addressing the concept of continual learning.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "lH2yvxROCKB", "reviewer": "Reviewer_JbJQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors consider the problem of continual reinforcement learning, i.e. learning from a changing stream of tasks, from the perspective of equivariance. The authors argue that identifying symmetries in observed tasks is crucial for generalization. Following this insight, they propose a continual reinforcement learning approach that continually grows the library of equivariant skills. For this purpose, the authors propose an architecture based on an equivariant MLP and an equivariant CNN. By comparing the 1-Wasserstein distance of the states in the invariant feature space, they decide whether to apply an existing policy to the current task or to add a new one. Finally, the authors test the proposed method in simulation and in real-life experiments.", "summary_of_recommendation": "Although the paper has certain drawbacks, I find the idea of using equivariance in continual reinforcement learning compelling. The experiments are also quite convincing, although additional ablation studies would make the picture clearer. As such, I am going with \"weak accept\" for now.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the points in the weaknesses section. In particular, I would like to see:\n- Additional ablation studies\n- A separate limitations section\n- Discussion of the complexity of the method in terms of computation and memory.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- An equivariant approach to growing new tasks is an interesting perspective on continual reinforcement learning problems. The idea to adaptively group together different tasks with symmetries is intuitive and quite elegant.\n- The evaluation protocol is mostly satisfactory. The empirical results show that the proposed approach works quite well. The authors also conduct ablation studies on the role of equivariant components\n- The real-world experiments are good to see, even if rather brief.\n- The paper is nicely written and easy to understand.\n\n\nWeaknesses:\n- Although the empirical evaluation shows sufficiently that the proposed method works in the desired setup, I think the paper would benefit from additional experiments that examine particular assumptions and implementation details. In particular, I would be interested in the following ablation studies:\n    - For example, the $d_\\epsilon$ hyperparameter used to decide whether to assign the current task to an existing group or create a new group seems to be quite crucial, so it would be great to see how sensitive it is to set in practice and if there are any hints on how to set it.\n    - How big do the buffers have to be in order for the method to work? How small can we make it before the model will break down?\n    - The top-down assumption is quite strong here. I would be interested to see how it behaves when the camera is at least slightly disturbed (e.g. not centered perfectly).\n    - Related to the previous point, seeing how the model behaves when the equivariance assumption is broken would also be nice. E.g. imagine a task where you have to do execute different actions depending on whether the object is on the left or on the right. It seems like this model would completely break down. Can we deal with it somehow, e.g. by using a non-equivariant MLP on top of the equivariant features?\n- The method introduces some operations that seem to be expensive both in terms of computation as well as memory. I would appreciate a more thorough discussion of the complexity this approach introduces.\n- The authors describe the limitations of their work in the \"Conclusion\" section, but I think it would be good to create a separate section for that purpose.\n- This is more \"nice to have\" than a weakness, since it extends beyond the paper's scope, but I would be interested in additional experiments in a multi-task setting, i.e. we see all the tasks at once. In general this setup is easier for CL methods, but in this case it might actually be more tricky since we build the representations for each policy group at the same time.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "FedvYYySep", "reviewer": "Reviewer_iFGd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "this paper proposes to learn a single policy to a class of symmetry equivalent tasks. It is achieved by equivariant DNNs with 1-Wasserstein distances between distributions. The results are convincingly remonstrated both on simulated and real robot. ", "summary_of_recommendation": "good paper with demonstration on both simulated and real robot", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. \"...unique Continual Vision-based Reinforcement Learning...\" in what sense is it unique? \n\n2. \"..Existing CRL approaches typically delineate task boundaries based on statistical measures...\" then \n\"...However, these measures overlook the geometric information inherent in task representations,..\".\n\nDo you have evidence that statistical measures overlook geometric information? \"Maximum a posteriori estimates and likelihoods\" can indicate symmetries, and can be invariant to symmetries of the system.\n\n3. \"..can be easily resolved by correspondingly reflecting the actions..\" is that true for an arbitrary symmetry? or for trivial ones only such as \"reflection\"/ rotation by angle $\\theta$?\n\n\n\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "good paper.  I have a few comment in \"Questions For Rebuttal\".", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "FZAH_0QHNMf", "reviewer": "Reviewer_bj9h", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel continual reinforcement learning (CRL) algorithm that effectively recognizes a new group of tasks and generalizes to SO(2) variations of the observation. Specifically, the method utilizes 1-Wasserstein distance on the extracted latent features clustering tasks. Moreover, leveraging equivariant models, the method generalizes to SO(2) perturbed tasks.", "summary_of_recommendation": "This paper addresses the erroneous clustering of geometrically similar tasks by using SO(2) equivariant networks and effectively recognizing new tasks. Experiments in simulation and real robots verified the effectiveness of the method.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please give an introduction and intuition of 1-Wasserstein distance. Why not use trivial distance measurement, e.g., l2 distance or cosine similarity to cluster tasks?\n\nI assume the x-labels of Fig. 6. 7 are (task1, task2, task3, task4, task1, ...), is it correct?\n\nIn lines 179 and 180, h^{inv} seems should have a subscript j to represent policy j.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Evaluation of the quality, clarity, originality, and significance of this work\n\nThe method address task clustering and SO(2) perturbations challenges in the CRL. The presentation is clear and the method is novel. This work could be applied to various reinforcement learning applications and real-world learning setups. \n\nstrengths\nNovel task clustering method that also generalizes to SO(2) rotated task.\n\nweaknesses\n1-Wasserstein distance is the key to detecting new groups of tasks and assigning a new policy. However, there is a very limited introduction and intuition of this distance.\n\nFigures 6 and 7 could be improved. Fig. 6, it can be breakdown into two figures, one for baseline comparison and the other for ablation. For Fig. 6. 7, the X- axis seems to be (task1, task2, task3, task4, task1, ...), this could be added to the x-labels.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2023. The reviewers have spent considerable time working with this paper and the current version is recommended for acceptance.

Please follow up with promised improvements, experiments and additional suggestions from the discussion phase. These include in particular improved clarity of the method section and further experiments and ablations.
Decision: Accept (Oral)