PointPatchRL - Masked Reconstruction Improves Reinforcement Learning on Point Clouds
Authors: Balazs Gyenes
Nikolai Franke
Philipp Becker
Gerhard Neumann
Venue: CoRL 2024 (2024)
Keywords: Point Clouds
Self-Supervised Learning
Reinforcement Learning
TL;DR: Recent methods for representation learning on point clouds significantly improve RL on point clouds for challenging simulated robotic tasks

Abstract: Perceiving the environment via cameras is crucial for Reinforcement Learning (RL) in robotics. While images are a convenient form of representation, they often complicate extracting important geometric details, especially with varying geometries or deformable objects. In contrast, point clouds naturally represent this geometry and easily integrate color and positional data from multiple camera views. However, while point-cloud processing with deep learning has seen many recent successes, RL on point clouds is under-researched, with only the simplest encoder architecture considered in the literature. We introduce PointPatchRL (PPRL), a method for RL on point clouds that builds on the common paradigm of dividing point clouds into overlapping patches, tokenizing them, and processing
the tokens with transformers. PPRL provides significant improvements compared with other point-cloud processing architectures previously used for RL. We then complement PPRL with masked reconstruction for representation learning and show that our method outperforms strong model-free and model-based baselines on image observations in complex manipulation tasks containing deformable objects and variations in target object geometry.

PointPatchRL Masked Reconstruction Improves Reinforcement Learning on Point Clouds
Balázs Gyenes .2, Nikolai Franke', Philipp Becker', Gerhard Neumann
Institute for Anthropomatics and Rohotics, Karlsruhe Institute of Technology. Germany HHDSS44ealtt Helmholtz Information and Data Science School for Health,
Karlsnuhe/Heidelberg, Germany
Abstract: Perceiving the environment via cameras is crucial for Reinforcement Learning (RL) in robotics. While images are a convenient form of representation. they often complicate extracting important geometric details, especially with vary- ing geometries or deformable objects. In contrast, point clouds naturally represent this geometry and easily integrate color and positional data from multiple camera views. However, while deep learning on point clouds has seen many recent suc- cesses, RL on point clouds is under-researched, with only the simplest encoder architecture considered the literature. We introduce PointPatchRL (PPRL), a method for RI. on point clouds that builds on common paradigm of divid- ing point clouds into overlapping patches, tokenizing them, and processing the tokens with transformers. PPRL provides significant improvements compared with other point-cloud processing architectures previously used for RL. We then complement PPRL. with masked reconstruction for representation learning and show that our method outperforms strong model-free and model-based baselines on image observations in complex manipulation tasks containing deformable ob- jects and variations in target object geometry. Videos and code are available alrhub, wite
Keywords: Point Clouds, Self-Supervised Learning, Reinforcement Learning
1 Introduction
In recent years, Reinforcement Learning (RL) for robots enabled solving increasingly complex tasks [1, 3]. A key factor contributing to these advancements is the development of representa- tion learning techniques for encoding high-dimensional sensor data such camera images or depth maps into compact, low-dimensional representations [4, 5, 6]. There are variety of learning objec- tives to train visual representations for RL, such as image reconstruction [7], multi-view contrastive loss [8]. or multi-view consistency [9]. However, these methods struggle to extract 3D geometric information about the scene and leave the handling of depth measurements under-explored. Not being able to provide the policy with 3D inductive bias for resolving ambiguities due to occlusions camera movement presents challenges for representations learned purely from pixels [10]. Point clouds provide natural representation of 3D geometry and can straightforwardly combine color and positional information. In contrast to depth images, point clouds leverage the camera parameters to obtain a common geometric representation from one or more camera views, alleviating the issues of pixel-based approaches. While this requires measuring the relative poses between cameras, in many cases this can done without tracking markers or SLAM algorithms. For example, if all cameras are mounted on the moving robot, the forward kinematics alone are sufficient. In addition, policies trained in simulation on point clouds instead of color images may be easier transferable the real world [2, 11]. since shapes are inherently easier to simulate than textures.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Grudient Flow
Critic
Loss
Enceder
Pooling
Sequence
Critic
Actor
Loss
Tokenizer
Actor
Masking
Encoder
Decoder
Prediction
Chamfer
+
Color
Gradient Flow
Figure 1: Schematic of Poin(PtthhRLPPPR) with an auxiliary masked reconstruction loss trained end-to-end for RL. Top: We train a patching- based tokenizer and transformer encoder to compute latent embedding for the RL policy and snute-action-valus estimation using sequence pooling, The entire pipeline learns using the critic's 's gradients while we detach the latent embedding before pro- viding it to the actor. Bottom: We augment the policy learning using masked reconstruction. Using the token sorting, masking, and transformer decoder introduced by PointGPT [17], we minimize the Chamfer distance for the point's ' positions and the mean squared reconstruction error for colors. This auxiliary loss provides an additional training signal for shared encoder and tokenizer, improving RL performacee and sample efficiency.
Since the introduction of PointNet [12] for processing raw point clouds directly, the number of methods in this field has grown considerably [13, 14, 15, 16, 17]. However, the of these encoder architectures in the context of reinforcement learning is relatively under-explored.
In this work, we introduce PointPatchRL (PPRL), which builds on recent patching-based methods for point clouds such as Point-BERT [15], Point-MAE [16] and PointGPT [17]. These methods first divide the point clouds into overlapping patches, which are then tokenized and processed by Transformer encoder [18]. By following this approach, PPRL leverage the intrinsic geometric properties of point clouds to learn dense representations that capture the task-relevant features of the environment directly from raw point cloud data. We show that providing this representation to Soft Actor-Critic [19] agent and training the encoder and tokenizer jointly with the critic already gives successful results compared to most point cloud-based and image-based baselines. We then extend PPRL with components from PointGPT [17] and introduce masked reconstruction objective to increase the geometric information encoded in the latent representation. This modification requires adapting the padding and matching procedure to support point clouds of any size and integrating point-level color features. We demonstrate that PPRL combined with this auxiliary loss further improves performance, especially in more challenging mobile manipulation environments and tasks with varying object geometries and moving cameras. A schematic overview of our architecture and learning pipeline is given in Figure
To summarize our main contributions: (i) we introduce PPRL, novel, patching- based approach for RL point clouds. (ii) We augment PPRL using masked reconstruction learning objective that further significantly improves the performance of our agents. (iii) We empirically demonstrate the capability of our method to solve complex simulated robotics tasks conditioned on color and depth information relative to other point cloud-based and image-based approaches.
Related Work
Image Representations for RL. Images are natural and versatile observation modality when using Reinforcement Learning (RL) in unstructured environments. However, naively using images as inputs to actor and critic networks often results in poor sample complexity, poor performance, or

---

both. Previous works propose sophisticated approaches such as image augmentation [20, 21, 22], autoencoding- based reconstruction [23, 7, 4], or other self-supervised methods [24, 25J. Driess et al. [9] learn a representation with the help of a multi-view consistency objective using Neural Radiance Fields (NeRF). Of particular importance to our work are approaches using masked autoencoding [26] on images for RL. Xiao et [5] use pre-trained masked autoencoder for on on-policy RL. Seo al. [27] train Dreamer world model [7] in the latent space of masked autoencoder. They show that naively masking image patches can be suboptimal for RL and instead mask convolutional features. In subsequent work, they extend their approach to handle multiple viewpoints and make it robust to viewpoint perturbations [28], properties which are shared implicitly by our point cloud-based approach. While these approaches can be extended to depth images, we argue that point clouds are a more natural representation of the scene's geometry, as they leverage the camera geometry and straightforwardly merge points from multiple cameras into the same coordinate frame.
Point Cloud Representation Learning. Since the advent of PointNet [12], many neural network ar- chitectures for point clouds have been developed, mainly for classification and segmentation tasks. PointNet++ [13] improves upon PointNet by using the Farthest Point Sampling algorithm to hi- erarchically partition the point cloud. PointTransformer [14] introduces a transformer backbone for point cloud processing. iteratively applying vector attention on point-level features within a small neighborhood and downsampling similar to PointNet++. Point-MAE [16] adapts the popu- lar masked autoencoding paradigm learn point cloud embeddings, dividing the point cloud into patches, masking them, and using transformer to reconstruct the masked patches. The network is trained to minimize the Chamfer distance between the reconstruction and the ground truth in the masked point patches. Point-BERT [15] uses block masking instead of random masking and pre- dicts discrete point tokens instead continuous ones. Point-GPT [17] addresses several weaknesses in Point-MAF. and leverages the successful auto-regressive inference paradigm used natural lan- guage processing. Despite their success, many of these architectures have never been used for RL, and we show that using the more modern point patching approach with an auxiliary objective in PointGPT [17] improves performance challenging (mobile) manipulation tasks.
RL with Point Clouds. Despite recent advances in point cloud processing, RL. on point clouds for robotics is under-eese,,ched, with the most common application being dexterous manipulation of objects [29, 3, 30]. Huang et al. [29] use pretrained PointNet feature extractor to learn a general dexterous manipulation policy capable of in-hand orientation of various objects. Chen et al. [3] train sparse 3D CNN feature extractor end-to-end using student-teacher learning method, where the teacher has access to privileged state information, while the student only relies point clouds and proprioception. Wu et al. [30] use imitation learning to solve the dexterous manipulation task, also employing pretrained PointNet that is finetuned during behavior cloning. Liu et al. [31] explore the impact of different coordinate frames on tasks from ManiSkil12 [32] with point cloud observations combined with proprioception. Ling et al. [33] study the effectiveness of point cloud-based RI. com- pared to image-based RL on a mix of two-dimensional and three-dimensional environments. They find that point cloud observations offer benefits over image observations in environments where un- derstanding agent-object and object-object relationships is important and that PointNet encoder outperforms a 3D SparseConvNet [34]. Wan et al. [35] make use of Transsfftttasemmerr [36), but this requires a segmented point cloud, which is not always available. Peri et al. [37] investigate the use of an encoder based on PointConv [38] for RL in latent imagination. However, a systematic comparison of point cloud encoder architectures is missing in the literature. In particular, there are no works that make use of the point patching paradigm, which we show empirically to powerful approach complex 3D (mobile) manipulation tasks.
3 PointPatchRL
PointPatchRIL (PPRL) assumes point clouds X E R.mx3 with m points. Here, the size m can vary between the individual point clouds X encountered during RL and we thus design an encoder to compute a fixed-length embedding containing all task-relevant information provided by X. We then provide this embedding to actor and critic which train using soft actor-critic (SAC) [19].
3

---

Our point cloud encoder is based on the point patching paradigm [15, 16, 17] with transformer backbone, adapted to handle point clouds arbitrary size. If additional color information is avail- able for the points, we can include it by appending corresponding features to each point. We use a sequence pooling layer to obtain the fixed-length embedding required for RL. Besides training the encoder solely using the critic loss, we consider augmenting training by explicit representation learning. We do so using PointGPT [17], which learns representations via the reconstruction of masked point patches. During training, we separately compute the reconstruction loss with masking and the RI. loss based on an embedding without masking, as illustrated in Figure 1.
3.1 Point Patch RL (PPRL)
Tokenization. To process the point cloud X using transformer, we first need to tokenize it. Here, grouping many points into a single token is desirable, point clouds have low information density. Thus, sample points centroids C' (X), C' E 1x* using Farthest Point Sampling (FPS) [13], which ensures good coverage and even distribution of the centroids. Subsequently, we conduct K-Nearest Neighbours (k-NN) search to select the k nearest points of each centroid to obtain (potentially overlapping) point patches P = knn(X,
P E TRxxxx3 The coordinates of all points in a patch are normalized relative to the centroid's position. lightweight version of PointNet [12], consisting of two MLPs and two max-pooling layers, converts point matches into input tokens T " of size D.
Transformer Architecture. A standard transformer encoder receives the sequence of tokens T and produces new sequence of encoded tokens. To this end, we also need to encode the position of each centroid and add to the respective token. As in PointGPT, we add sinusoidal positional encodings of the centroids' locations to the tokens.
Reinforcement Learning. We utilize soft-actor critic (SAC) which is popular algorithm due to its sample efficiency. SAC learns policy and state-action value function network, which both require fixed-length embedding as input vectors. We use sequence pooling layer [39] to obtain this embedding. This layer computes a weighted sum over all input tokens, where the weight of each token a function that token, i.c.,
pooooled "T with ur = softmax(g(T)) and Trpooled Rdd
where g is a linear layer. Compared with using an explicit query token to compute a fixed-length embedding, sequence pooling can improve the performance of small transformer networks [39].
If low-dimensional state observations are also available, they are projected with a linear layer to the same dimensionality as the point cloud embedding, and then concatenated with the embedding. Ac- tor and critic networks share a single point cloud encoder, which is updated only with the gradients from the critic loss, is standard practice in off- policy RL [24, 22].
3.2 Representation Learning Using Masked Reconstruction
We extend PPRL with masked reconstruction objective, to benefit from an additional geometric learning signal. By adding sorting, masking, and transformer decoder to our architecture, we obtain a modified version of PointGPT. Unless noted, all details below follow PointGPT.
After tokenization, the tokens sorted according to the Morton order [41] (Z-order curve), which maps n-dimensional data to one dimension while preserving the locality of the data points. Because nearby point patches remain close to each other after sorting, the model can rely on neighboring tokens to inform the reconstruction. In contrast to the transformer encoder, the transformer de- coder does not receive the encodings of the absolute positions of point patches, but only the relative direction between subsequent patches is encoded, i.e.,
C - (
C -
= where C is the ith centroid of the ordered list of centroids. This avoids leaking global informa- tion about the point cloud structure through the positional embeddings to the decoder. The masking
for i (2,...,n), and = C₁

---

PPRL + Aux
PPRL
PointTransformrr
PointNet++
PointNet
Dreamer
MV-MWM
DrQ-v2
ThreadInHole
DeflectSpheres
PushChair
Success Rate
0.8
0.6
0.4
Orewreri
OpenCabinetDoor
TurnFaucet
Success Rate
0.8
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x (19)
Environment Steps (x10%)
Environment Steps (x10%)
Figure 2: Average success rates and 95% Bootstrap Confidence Intervals [40] for all methods sofaenv and ManiSkil12 environments. Our method, PPRI. + Aux, achieves top performance on 5 of 6 environments. On DeflectSpheres, ThreadInHole, and PushChair, PPRL without represen- tation learning achieves roughly the same success as PPRL + Aux, demonstrating the effectiveness of neural network architecture even without auxiliary learning objectives. On OpenCabinet- Drawer and OpenCabinetDoorr two challenging tasks with non-trivial variations in scene geometry. adding auxiliary reconstruction loss significantly improves learning, and required for solving the task. On TurFFaucet, DrQ-v2 outperforms our method, potentially due to the availability of static camera, making the learning task easier for image-based methods.
combines random and causal masking as detailed in Appendix B. Finally, a one-layer prediction head reconstructs point patches from decoded tokens. The loss is the symmetric Chamfer dis- tance [15] between the reconstructed and ground truth patches, which measures the average distance of each point in the ground truth patch the closest point in the reconstructed patch and vice-versa.
Variable Point Cloud Size. While standard representation learning commonly assumes fixed-sized point clouds, in RL, their sizes can vary by orders magnitude, especially in environments with moving camera. For point clouds smaller than a fixed minimum size, PointGPT duplicates random points, which may distort objects' point density and add redundant information. To alleviate this problem, add padding tokens to the input sequence and adjust the masking procedure maintain constant ratio of masked and visible tokens among non-padding tokens while ensuring consistent length for all sequences in batch.
Reconstruction of Color Features. Although most point cloud representation learning methods ignore level features such as color, color may be useful many tasks. We generalize PPRI. and PointGPT include color by adding color features to the points, which are used as inputs by the tokenizer well by the reconstruction loss. We will denote = (, ar*h),, point consisting of the coordinate component x R and the color component w R3, The color reconstruction minimizes the MSE between the predicted color of each point in the predicted point cloud pnd of patch and the color of the nearest point in the ground truth point cloud pst i.e.,
R6 single
Crbb (rad P!!)
Σ P
(a
nn(a (a.))
ppd
where 2x pet) returns the color of the nearest point to xyz the ground truth point cloud pet These correspondences are already available through the Chamfer distance computation. The final auxiliary loss (referred to Aux, i.e. PPRL Aux) is then the sum of Chamfer loss and the color reconstruction loss.
5

---

Figure 3: Visualization of successful PPRL + Aux trajectories on the OpenCabinetDoorr environ- ment from static rendering camera that the agent does not have access to. Each column shows single episode, Agents trained with PPRL + Aux adapt to varying geometries, including handle size and orientation, and whether the door opens the left right. Our method is able coordinate the movements of the gripper and the base and generalize well over these factors.
Experiments
We compare method with the auxiliary representation learning loss (PPRL + Aux) and without it (PPRL) on 6 challenging visual robotic manipulation tasks from sofaenv [42] and ManiSkil12 [32]. These environments highlight PPRL's ability to infer non-trivial geometric infor- mation from multiple viewpoints and moving cameras. We compare several ablations using different point cloud architectures to showcase the benefits of using point patching with transformer. Fur- ther, compare against SOTA image-based RL approaches to demonstrate the merits of using point clouds. We report the average success rate and 95% Bootstrapped Confidence Intervals [40]. All results are averaged over 8 seeds, except for methods that fail' where we only run 4.
Environments. sofaenv [42] provides environments for robot-assisted surgery, focusing on tasks involving deformable objects. For these tasks, the camera's position and orientation change for each trajectory, mimicking viewpoint changes encountered during surgery with endoscopic cameras. This is exceptionally challenging for most image-based approaches but reflects the desiderata of deploying learning agent in real-world system without tedious camera calibration. Furthermore, the deformable nature of the objects involved prevents access concise state information.
ManiSkil12 [32] provides challenging robot manipulation tasks, including varying object geome- tries by sampling object models randomly from PartNet [43] (e.g., cabinets, chairs, faucets, etc.). PushChair. OpenCabinetDrawerr and OpenCabinetDoor are mobile manipulation tasks, using cameras mounted above the mobile robot. TurnFaucet exhibits static manipulation and uses cam- eras, mounted to the mobile gripper and the static base. Point clouds are given in the egocentric perspective, which is especially challenging for mobile manipulation, as this reference frame is not static. The agent additionally has access to proprioception and the target location and pose.
We train on ThreadInHole, PushChair, and TurnFaucet without color, while we use color features for DeffectSpheres, CaenCabinetDrawer,, and CabnCabinet where color is either necessary or helpful for detecting the target. Appendix A provides details for all environments.
Ablations and Baselines. from the literature. We consider PointNet [12], which has previously been used in the literature for RI. on point clouds, as well as its successor PointNet++ [13]. To investigate the benefit of transformer-basse architectures, we consider PoinTTransformer [14].
We compare our encoder against several popular point cloud architectures
Failing means that of the seeds reached 10% success.

---

OrenCabinetDrawer
OpenCabinetDoor
PushChair
TurnFaucet
0.8
0.6
0.4
15
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
1.0
Environment Stegs (x 10")
Revircement Stegs (x 10")
Eavironment Stegs (x 10")
Favironment Stepe 10")
Average success rates and 95% Bootstrap Confidence Intervals for agents trained with PPRL + Aux on selected environments with varying numbers of object models. Although there is general trend that more object models reduce success rates, the effect is not always strong, showing that the encoder can generalize well. OpenCabinet performs approximately the same for 5, 10, 15 object models, and only begins to decrease for 25.
Figure 4:
Additionally, we compare PPRL against several SOTA image-based RL approaches, to showcase the benefits of using point clouds. First, we compare against the model-free DrQ-v2 [22] and the model-based Dreamer [7] approaches. To ensure all approaches receive the same information, both get the full RGBD images from all available cameras. Finally, we compare PPRL. against Multi- View Masked World Models (MV-MWM) [28], which tailored integrate image observations from different camera perspectives and uses Preamer-basee world model to do sensor fusion in latent space. Appendix B provides details on the hyperparameters and training of the baselines.
4.1 Results
Figure shows the results for all environments tested. PPRL Aux outperforms all baselines on 5 of 6 environments tested, while DrQ outperforms method the TurnFaucet task although the final performance is similar. While PPRL also works DeflectSpheres, ThreadInHole, and PushChair without representation learning, representation learning leads to substantial benefits in the challenging OrenCarinetDrawer, OpenCabinetDoor, and TurnFaucet tasks. In these tasks, object geometry is crucial for policy execution, showing that representation learning is helpful for the policy to understand the differences object geometry. For DeflectSpheres and ThreadInHole, representation learning still speeds up learning, while there is slight decrease in performance on the PushChair task. These tasks either have rather simple object geometries the difference in the object geometry does not require different strategies for manipulation (PushChair).
PPRL + Aux solves Oreneer and OpenCabinetDoor with success rates of 60% and 50%, respectively, while PPRL without representation learning and the baseline methods are unable to solve them. These environments are challenging mobile manipulation tasks that necessitate adapting the strategy to the target geometry. The handles of each cabinet model have different sizes and poses, and must therefore be grasped differently, and doors must be actuated according to their direction of opening and turning radius. Additionally, all cameras in these environments mobile, which presents a much more difficult perception problem for image-based methods than for point cloud methods. Figure visualizes the policies for the CaenCabinetDoor environment, showing how the agent responds to variations in the cabinet geometry.
In TurnFaucet, DrQ-v2 has higher initial success rate than our method and comparable final suc- cess rate. We speculate that this is partly because of the cameras is static, allowing image- based methods to learn representations without needing to generalize across viewpoints. Dreamer and Multi-View Masked World Models also appear to benefit from the static camera in TurnFaucet. All other environments contain no completely static cameras, either due to variation between episodes or the movement the robot base, leading to the general tendency for point cloud-base meth- ods to outperform image-based methods. Interestingly, PointNet++ also performs competitively on DeflectSpheres, PushChair, and TurnFaucet, despite its relative simplicity. This may be due to how it processes point cloud data at multiple geometric scales, using neighborhoods generated by FPS to iteratively downsample. Surprisingly, PointTransformer performs poorly across all environ-
7

---

SefrectSpheres
OpenCabinetDrawerr
OpenCabinet
Success Rate
0.8
0.6
0.4
Color Loss
0.2
No Color Loss
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps (x (x10")
Environment Steps (x10%)
Environment Steps (x 1000)
Figure 5: Average success rates and 95% Bootstrap Confidence Intervals for agents trained with and without color reconstruction loss, on all environments with color point cloud observations. When training without color reconstruction loss, color is still observed, but only the signal from RL en- courages the agent to condition on color features. In particular the hard Cabinet tasks profit from explicitly reconstructing the color.
ments, despite containing many components common to more successful networks such PPRL and PointNet++, namely neighborhood computation via FPS and transformer backbone.
Object Model Count. We test the generalization capabilities of our method by training on ManiSkil12 environments with varying numbers of object models. Figure shows the effect of increasing numbers object models on the sample efficiency. Although more models generally result more difficult learning task, the effect is weak below a certain number of models (15 for OpenCabinetDrawer, 10 for OpenCabinetDoorr) This suggests that the encoder's capacity is suffi- cient for learning more object models, and the limiting factor may rather be the burden of exploring and finding generalizable RI. policies with increasing number of geometries.
Color Reconstruction. To investigate the effect of color reconstruction we remove color recon- struction for all 3 enviromments that use color point clouds. Without the reconstruction loss, only the loss signal from RI. encourages the agent to incorporate color into the embedding. The results shown in Figure 5. Surprisingly, the color reconstruction objective provides no significant ben- efit for the 3 environments tested, despite the fact that DeflectSpheres encodes some task-relevant information solely through color. Although it is important to provide color to the policy to solve the task, the learning signal from RI. alone enough to learn any color features required. Furthermore, this shows that the Chamfer distance loss is informative enough to learn much of the tesk-relevant information the point cloud without any additional modifications.
Conclusion
We present PPRL., a novel method for RL on point clouds, and an accompanying auxiliary recon- struction objective based on PointGPT. PPRL exploits the inherent properties of point clouds to combine color and positional information from all available cameras and is robust to changing and moving viewpoints. We evaluate PPRI. on series of challenging visual robot (mobile) manipulation tasks from sofaenv and Mani ManiSkil12 and find thatPPRL outperforms popular baselines, even with- out the auxiliary reconstruction objective. In combination with the auxiliary representation learning objective, our method improves sample efficiency and final success rates in most environments com- pared to both point cloud-basee and image-based baselines. These advantages are more significant for more challenging environments, such OpenCabinelDrawer and OpenCabinetDoor, A key sight that point cloud methods offer robustness changing viewpoints in environments without static camera, domain where image-based methods are known struggle [10].
Limitations. While we demonstrated the effectiveness of existing representation learning methods for RL on point clouds, findings from image-representaatin learning with masked reconstruction for RL [27] suggest that special treatment may further improve the performance. While the convolu- tional feature masking method of [27] straightforwardly applicable point clouds, similar extensions could be investigated. Furthermore, have not included other sensor modalities such as proprioception in the representation learning, which has shown benefits for image-based RI. [44].

---

Acknowledgments
We would like to thank Paul Maria Scheikl for his help with sofaenv and the LapGym environment suite. The present contribution is supported by the Helmholtz Association under the joint research school "HIDSS4Health Helmholtz Information and Data Science School for Health." The authors acknowledge support by the state of Baden-Wiürttember through bwHPC, as well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden- Wiinon-brrttebbbrggg and by the German Federal Ministry of Education and Research.
References
[1] Luo, Z. Hu, C. Xu, S. Gadipudi, A. Sharma, R. Ahmad, S. Schaal, C. Finn, A. Gupta, and Levine. SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition CoRL2023,
Oct. 2023. URL https ptttp:://ppeseee //tpe//veeeeeeeeeeeee/eeeeeeeeee wwt. wett/foruur1ia7722gg007..
[2] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation. In Proceedings of The 6th Conference on Robot Learning, pages 605.005. PMLR, Mar. 2023. URL https:
/pproceedings.. mlr. press/v205/qinaa /ain/a/. html. ISSN: 2640-3498.
[3] T. Chen, J. Xu, and P. Agrawal. A System for General In- aa-Hand Object Re-Orientation. In Proceedings of the 5th Conference on Robot Learning, pages 297-307. PMLR, Jan. 2022. URL https:
B http://pooceee....///// mlr.
wttp:ss.mm.ppp.rs....m www.pprrsssea.. 22664//hh222a... html. ISSN: 2640-3498.
[4] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering Diverse Domains through World Models, Jan. 2023. URL httt:://tariiii hwtp://rrggbbb////////// 04104. arXiv:2301.04104 [cs, stat]
version: 1.
[5] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked Visual Pre-training for Motor Con- trol, Mar. 2022. URL http: arxiv. org/abs//200 06173. arXiv:2203.06173 [cs].
[6] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3M: A Universal Visual Rep- resentation for Robot Manipulation. In Proceedings of The 6th Conference on Robot Learn- ing, pages 892-909. PMLR, Mar. 2023. URL httpp://prroceednggs mlr hat..prres//25/// nair23a. html. ISSN: 2640-3498.
[7] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to Control: Latent Imagination. In International conference on learning representations, Mar. 2020. URL meet:/formmiidd111000e444
Learning Behaviors by
arXiv: 1912.01603.
[8] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang. S. Schaal, S. Levine, and G. Brain. Time- Contrastive Networks: Self- Supervupe Learning from Video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141, May 2018. doi:10.1109/ R.AA.2018.446889 ISSN: 2577-087X.
[9] D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement Learning with Neural Radiance Fields. Advances in Neural Information Processing Systems, 35:16931- 16945, Dec. 2022. URL https: ptttoee//proceeeiingg. neurips. ptrprppp.cpppppprfffiiieeeeeeeee 10222//aa/ccc2o11eeeee7788eeee22
Conftrreonneeeecceeoo html.
[10] S. M. A. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. S. Morcos, M. Gamelo, A. Ruderman, A. A. Rusu, Danihelka, Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukeuoglu, and D. Hassabis. Neural scene representation and rendering. Science, 360 (6394):1204-1210, June 2018. doi: 120.1.11122/cccccc.aarr////// URL https //www. science. org/doi/ful110.I1 111........iiineeeeeeeee aar6170. Publisher: American Association for the Ad- vancement of Science.

---

[11] S. Christen, W. Yang, C. Pérez-D' Arpino, O. Hilliges, D. Fox. and Y.-W. Chao. Learn- ing Human-to-Robot Handovers From Point Clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9634-9664, 2023. URL whecaaaccceesseeeeee.eeeeeeeeee twectheeeeccce /ttlev/..co//mmtee////////////tl Homen-to-Roooo__aaaaaa cot.rrrree..r.r.ooooooooooooo... CVVPR__222__ppprrr html.
[12] C. R. Qi, Su, K. Mo, and J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition (CVPR), July 2017. URL https:://ppeaaccesss thecvf com/ content_cypi 2017/attm/O1__oiintetet_te Pon.tieeDDeepLLaannnng.......LL. _appee htmll. arXiv: 1612.00593 version: 2.
[13] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. In Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https:/ http:/oeeeennnn....rrr......... wwwpripp.cpppppr////p 2017
html.
[14] H. Zhao, L. Jiang. J. Jia, H. S. Torr, and V. Koltun. Point Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), pages 16259- 16268. 2021. URL http::ttt//e/cccoccccc Tias__Trasffroroo www.orrreeeeeeeeeaeeeeeeeeeeeeee 2021 20w100aaeerepp___11 www.r.ppppp........ html.
thecvf http:///oooeee//////////////////
[15] X. Yu, Tang, Rao, T. Huang. J. Zhou, and J. Lu. Point-BERT: Pre-Training 3D Point Cloud Transformers With Masked Point Modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
2022.
URL. httpps://ppenaccess. thecvf. com http///oonee/////2/22////////// httpp/////222////////// / Paotttnn//oooPPttBRR Sre-Traaining_3DPone Bacpi_oooot___ieeeeeeeeee Saennt_Coouud___ wrw.oo__raanfoommrrsr...mmm With Mastiha_MseeeePPitt_ddddddd CVPR_2022_ cactl2_apperhtat. html.
[16] Y. Pang, W. Wang, F. E. H. Tay, W. Liu, Y. Tian, and L. Yuan. Masked Autoencoders for Point Cloud Self-supervised Learning. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and Hassner, editors, European conference on computer vision 2022, pages 604-621, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20086-1. doi: 35.
10.1007/978-3-031-22
[17] G. Chen, M. Wang, Y. Yang, K. Yu, Yuan, and Yue. PointGPT: Auto-regressively Genera- tive Pre-training from Point Clouds, May 2023. htL:ttt://arrtt wwtp:///.wa/ab//////b//// 1.0...1144 11487.
arXiv:2305.1 1487 14487 [es].
[18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, NIPS 2017, pages 5998-6008. Curran Associates, Inc., 2017. URL http::/ nips cc/cc////p//o cc/paper/ 1il-attention----llllyoottt you.ll
[19] T. Haarnoja, A. Zhou, Abbeel, and Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. 10X..0200 [es, stat), Aug. 2018. URL http://arxiv org/ abs 18011 01290. arXiv: 1801.01290.
[20] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and Srinivas. Reinforcement Learning with Augmented Data. arXiv:2004. 14990 [cs, stat), Nov. 2020. URL http://arxiv. org/ abs/2004 14990. arXiv: 2004.14990.
[21] D. Yarats, A. Zhang, Kostrikov, Amos. J. Pineau, and Fergus. Improving Sample Efficiency in Model-Free Reinforcement Learning from Images. arXiv:1910.01441 [cs, stat), July 2020. URL http:://tee /ttp:////arii.aa/ wttpiii..rrrab///111100 01741. arXiv: 1910.01741.
10

---

[22] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning, July 2021. URL httttt/pp//aiii wrw. arggabbs/117.. 09645. arXiv: 2107.09645.
[23] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic Latent Actor- Critic: Deep Reinforcement Learning with Latent Variable Model. In Advances in Neural Information Processing Systems, volume 33, pages 741-752. Curran As- sociates, Inc., 2020. URL https://proceediin wwwrips.cc/ppppeeeeepeee wwtpccpppppp/.///aaaeeeeee
0000515004452200011188003333 Abstract. html.
[24] M. Laskin, A. Srinivas, and P. Abbeel. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In International Conference Machine Learning, pages 5639-
5650. PMLR, Nov. 2020. URL http::/pt/ooo http://prooceednns mlr. http:ssmw.....s....e wttppp/rss/seeeeeeeeeeeeee 11ss///./1/laan0n... html. ISSN: 2640-3498.
[25] A. Stooke, K. Lee, Abbeel, and M. Laskin. Decoupling Representation Learning from Reinforcement Learning. DXXiv:009.088199 [cs, stat], Sept. 2020. URL http://arxt org/abs/20053 08011 arXiv: 2009.08319.
[26] K. He, X. Chen, S. Xie, Y. Li. Dollár, and R. Girshick. Masked Au- toencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022. URL https://openaccess. caappeaaccess.hhev...... cwm/ /wtp///ttt////////////////////// _MMkedd Autoencoders wwaeroddrrssrree_eeeee Are
[27] Y. Seo, D. Hafner, H. Liu, F. Liu, James, K. Lee, and P. Abbeel. Masked World Models for Visual Control. In Proceedings of The 6th Conference on Robot Learning, pages 1332-1344 PMLR, Mar. 2023. URL https::/prooceeeeeee http:/eeiinnn......eeessss///e/e 2640-3498.
mlr. pppppp//00//ee020//3 htmll.l. ISSN:
[28] Y. Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel. Multi-View Masked World Models for Visual Robotic Manipulation, May 2023. URL http://arxiv, wrgg/ba//302..0222222/ arXiv:2302.0240) [cs].
[29] W. Huang. I. Mordatch, P. Abbeel, and D. Pathak. Generalization in Dexterous Manipulation via Geometry-Awant Multi-Tase Learning, Nov. 2021. URL http::/tt:/i/ittt////////tt www.aawwwwww......////.r.a..taet 03062. arXiv:2111 Ntt:2111003022 [cs, eess].
[30] Y.-H. Wu, Wang, and Wang. Learning Generalizable Dexterous Manipulation from Hu- man Grasp Affordance. In Proceedings of The 6th Conference Robot Learning, pages 618- 629. PMLR, Mar. 2023. URL https: htttps:/pproceet http://ppcceeee.....// mlr press//2005/mm33.. htmll. ISSN: 2640-3498.
[31] M. Liu, Li, Z. Ling, Y. Li, and H. Su. Frame Mining: Free Lunch for Learning Robotic Manipulation from 3D Point Clouds, Oct. 2022. URL 07442v1.
[32] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su. ManiSkil12: A Unified Benchmark for Generalizable Manip- ulation Skills, Feb. 2023. URL http:://a/aa: http://aaxxvv..r/abbeaaaee /2302 04659. arXiv:2302.04659 [cs].
[33] Z. Ling, Y. Yao, X. Li, and H. Su. On the Efficacy of 3D Point Cloud Reinforcement Learning, June 2023. URL https://arxie. wwwpeiiivoorraaaeo arg//bss/3066 067999....
[34] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han. Searching efficient 3d architec- tures with sparse point-voxc convolution. In European conference computer vision, pages 685-702. Springer, 2020.

---

[35] W. Wan, H. Geng, Y. Liu, Z. Shan, Y. Yang, L. Yi, and H. Wang. UniDexGraspt+: Improving Dexterous Grasping Policy Learning via Geometry-Aware Curriculum and Iterative Generalist- Specialist Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3891-3902, 2023.
[36] T. Mu, Z. Ling, F. Xiang, D. C. Yang. X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations. In Thirty- Fifth Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round 2), Aug. 2021.
[37] S. Peri, I. Lee, C. Kim, L. Fuxin, T. Hermans, and S. Lee. Point cloud models improve visual robustness in robotic learners. International Conference on Robotics and Automation, 2024.
[38] W. Wu, Z. Qi, and Fuxin. PointConv: Deep Convolutional Networks 3D Point Clouds. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9631-9630, 2019.
[39] A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and Shi. Escaping the Big Data Paradigm with Compact Transformers, June 2022. URL http://arxive wwwpxi..org/bb///1004 05704. arXiv:2104.05704 [cs].
[40] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, Deep reinforce- ment learning at the edge of the statistical precipice. Advances in neural information process- ing systems, 14:29304-29322 2021.
[41] G. M. Morton. A computer oriented geodetic data base and new technique in file sequencing, 1966. Publisher: International Business Machines Company New York.
[42] P. M. Scheikl, B. Gyenes, R. Younis, C. Haas, G. Neumann, M. Wagner, and F. Mathis- Ullrich LapGyym An Open Source Framework for Reinforcement Learning in Robot-Assisted La- paroscopic Surgery. Journal of Machine Learming Research, 24(368):1-42, 2023. ISSN 1533- wwtpppas////////2 00ttmmmm00...........
7928. URL http: / jmlr.
[43] K. Mo. S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su. PartNet: A Large- Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[44] P. Becker, S. Mossburger, F. Otto, and G. Neumann. Combining reconstruction and contrastive methods for multimodal representations in RL. Reinforcement Learning Journal, 4:1619- 1655, 2024.
[45] D. Hafner, K.-H. Lee, I. Fischer, and Abbeel. Deep Hierarchical Planning from Pixels, June 2022. URL http:://rxiii http://rxiv.oorgabbaaa/aaaaa wttg/aa////aaeeeee 04114. arXiv:2206.044 201122006011144 14 [cs, stat].
[46] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International conference on learning representations, 2021. URL wttttht:///peeeeiiiiie net//forum?id=00aabyyt00
12

---

A Environments
Figure 6: ThreadInHole
Figure 7: DeflectSpheres
Figure 8: PushChair
Figure OrerCabinetDrawee Figure 10: OpenCabinetDDoor
Figure 11: TurnFaucet
Image renderings of the 6 environments used. The ManiSkil12 environments rendered from a dedicated, fixed rendering camera that the agent does not have access to.
Figure 12:
Task Descriptions: All 6 environments are rendered in Figure Figure 12. In ThreadInHole, the goal is to feed the deformable thread into the (rigid) hole. In DeflectSpheres, the goal is to deflect the red or blue sphere with the tool of the corresponding color. Spheres turn green after they are deflected. In PushChair, the goal to push the chair to the position marked by the red ball (not visible to the agent), without knocking over. OpenCabinetDrawee (OpenCabinetDoor) the goal grab the drawer (door) handle and open the drawer (door), then stabilize the open position. In TurFFaucet, the goal is turn faucet the open position.
sofaenv: All sofaenv environments have point cloud observations from only a single camera, without low-dimensional state. Upon each reset, the camera position and look-at perturbed by vector uniformly sampled from 2cm, 2cm] in all directions. Reward weights are taken from the reference implementation.
ManiSkil12: All mobile manipulation tasks (OpenCabinetDrawee, OpenCabinetDoor, and PushChair) have 3 cameras mounted above the robot base facing outwards, spaced radially at 120". In these tasks, the camera FOV was increased to 1.5, such that the cabinet/chair is almost always visible from at least one perspective. In OoenCabinetDoor, only the first 10 cabinet models are used for training and evaluation. In addition, the target door is not randomized but fixed for each cabinet model. OreaCari because is slightly easier, is not changed, using 25 models and a randomly chosen target drawer for each episode. TurnFaucet is statie manipulation task with only 2 cameras, mounted on the (static) base and one inside the gripper. In TurnFaucet, only the first 10 faucet models are used for training and evaluation. In all environments, joint position
13

---

delta control is used for the robot arm, and joint velocity control is used for the robot base, if ap- plicable. Observations contain point clouds and a low-dimensional state vector containing robot proprioception and the target location.
Discounts and Time Limits: Discounts for all tasks are: 0.99 for ThreadInHole and PushChair, 0.9 for DeflectSpheres, and 0.85 for OpenCabinelDoor, OpenCabinellrawer, and TurnFaucet. The time limits for all tasks are: 300 for ThreadInHole, 500 for DeflectSpheres, and 200 for all ManiSkil12 tasks. In OpenCabinetDraweer, OpenCabinetDoorr, and TurFFaucet, trajectories are ended only on timeout, not task success. In ThreadInHole, DeflectSpheres, and PushChair, trajec- tories are ended either on a timeout or task success. PushChair is the only ManiSkil12 environment treated like this, because we found that ending on task success is crucial for training.
Point Clouds: All environments are rendered with a resolution of 128x128, except PushChair, which has resolution of 64x64. Egocentric point clouds created from depth images using the linear transformations given by the camera intrinsic (focal length) and extrinsic (position and orientation) parameters. Point clouds are then post-processed using the following steps: cropping, appending target points, downsampling, and normalization (note that not all environments use all of these steps). First, points beyond the boundaries of the scene are discarded. In ManiSkil12 environ- ments, the floor is also cropped out, and furthermore in OpenCabinetDrawee and OpenCabinetDoor, any points more than 0.5m front of the cabinet are also cropped out. DreeaarinetDraaeer,, OpenCabinetDoor, and PushChair, 50 additional points are appended the point cloud, sampled from uniform cube (of side length 7em for OpenCabinetDrawerr and OpenCabinetDooor and 1 14m for PushChair). Although this information is also present in the state vector observation, these points serve to indicate the visual target position within the observation directly. In ThreadInHole, Deflect- Spheres, and PushChair, we use voxel grid sampling (with a voxel size of 5mm for sof aenv and 5cm for PushChair) to downsample the point clouds, followed by random downsampling (without replacement) to a maximum point cloud size of 1000 points. In OpenCabinetDrawer, OpenCab- randomly downsample each point cloud (without replacement) to
inetDoor, and TurnFaucet, maximum point cloud size of 800 points. For DeflectSpheres, we normalize all observed point clouds by subtracting fixed center point and dividing by fixed scale factor. The scale and center point are chosen such that the extrema of point cloud from typical initial observation are roughly within [-1, 1]. For ThreadInHole, we normalize each observed point cloud independently by sub- tracting the mean and then dividing by the maximum absolute value over all 3 coordinates, such that the longest axis is mapped to [-1, 1]. This method of normalization (compared to that of Deflect- Spheres) was crucial for training. Point clouds for Mani mSki 112 environments are not normalized as they already lie roughly within [-1, 1].
14

---

1
An example of an attention mask for single token sequence. Black circles are masked (not visible), and white circles are unmasked. The ith row shows the tokens that are visible when encoding the ith token. The jth column shows which tokens may attend the jth token. The first column is the start-of-sequence token, not the token of the first point patch. A token is masked if it either randomly selected with a probability of m [0, 1] (random masking) or in the present or future (causal masking). A token is always allowed to attend the token immediately preceding it. Additionally, a certain fraction of the tokens at the start of each sequence are never masked out.
Figure 13:
B Hyperparameters and Architectures
PPRL We use hyperparameters for the tokenizer, masking, encoder, decoder, and prediction head as defined PointGPT. We reduce the number of transformer layers to 3 to speed up inference. For environments with color, the point feature dimension is set to 3 in the tokenizer and the prediction head; otherwise is O. We disable dropout.
Actor and Critic Networks Actor and critic networks are fully-connected networks with 3 layers and 1024 neurons per layer, using ELU as the non-linearity.
Masking Figure 13 provides an overview of the attention mask used for the auxiliary reconstruc- tion loss.
SAC We use SAC with a replay buffer of 500000. We use a replay ratio of 64, except for sofaenv environments with a replay ratio of 32, and PushChair where we use We update target networks after every learning update with a of 0.005, except PushChair, which uses a of 0.01. By default, the batch size is 1024 when training without auxiliary loss and 256 when training with auxiliary loss, except for PushChair, where we use 128 in both cases. Due to compute constraints, we reduce the batch size for PointNet++ to 256, and for PointTransformer to 512. Also due to compute constraints, we reduce the replay ratio for PointNet++ on OpenCabiinetDrawer and OpenCabinettooor to 32, as these environments have the largest point clouds on average. The learning rate is Ic-4, except for DeflectSpheres and PushChair with 5e-5, and we use the Adam optimizer. The starting entropy coefficient is 0.1 except for DeflectSpheres with 0.2 and PushChair for 0.05. The learning rate for the entropy coefficient is le-4, except for PushChair with 2c-5.
For all baselines, we use hyperparameters as defined by the baselines, except for the environment- dependent discount, initial entropy coefficient, and entropy learning rate, where we use the same values as for our method.
DrQ-v2 Each camera uses a dedicated CNN encoder, and the features are concatenated together, resulting in an image encoding of size 11 117600. Both the image embedding and the state are projected down to dimensionality of 384 (to match the dimensionality used for PPRL) using a single fully- connected layer and then concatenated.
Dreamer and Multi- View Masked World Models For these world model-based methods, we follow the approach of [45, 4] to handle the additional state information. They first encode both image and state using separate encoders and concatenate their outputs before providing them to the RSSM. For training, they then consider two separate output heads reconstruct both the image and state. Dreamer uses mostly the hyperparameters proposed by [46]. However, assume Gaussian latent variable and use the same discount and entropy control as PPRL., For Multi-
15

---

Masked World Models, we run training in multi-view mode, which masks random viewpoints and reconstructs them. We disable the hehavioural cloning loss, as our method does not have access to demonstrations.
B.1
Model Parameter Counts
Table provides comparison of the model sizes by listing parameter counts for the proposed method and all ablations. Because the model size changes slightly based on the observation size, and therefore the environment, we use OpenCabinetDrawer for this comparison.
Encoder
Actor
Critic
Total
PPRL+Aux
9.47M
2.91M
5.80M
18.18M
PPRL
4.95M
2.91M
5,80M
13.66M
PoinfTransformer
2.20M
2.91M
5.80M
10.91M
PointNet++
0.38M
2.91M
5.80M
9.09M
PointNet
1.55M
2.91M
5.80M
10.26M
DrQ-v2
0.08M
5.08M
6.25M
11.39M
Dreamer
2.69M
0.26M
0.25M
3.13M
MW-MWM
4.72M
0.84M
0.83M
6.39M
Table Comparison of parameter counts for each method.
C Wallclock Runtimes
Thread InHole
Deflect Spheres
Push Chair
Open Cabinet
Open Cabinet
Turn Faucet
Average
Drawer
Door
PPRL+Aux
14.1
19.3
8.7
40.9
39.3
39.6
26.0
PPRL
7.4
10.7
7.4
23.2
22.5
22.3
15.0
PoinTTransformer
6.6
11.9
7.8
33.0
28.5
25.7
19.3
PointNet++
7.7
5.9
7.3
25.1
24.1
47.7
16.4
PointNet
4.1
4.5
6.1
11.1
10.8
9.8
7.1
DrQ-v2
29.4
15.0
36.5
46.2
33.5
21.5
30.3
Dreamer
25.5
15.7
28.6
27.5
26.6
17.7
24.0
MW-MWM 32.3
13.3
23.3
22.0
21.4
14.8
21.2
Table 2: Comparison of wallclock runtime for each method (in hours).
Table provides a comparison of the average wallelock runtimes for all methods considered. All runs used a single Nvidia A100 GPU, 19 Intel Xeon Platinum 8368 CPUs, and 128Gb of RAM (maximum memory, average). In some cases, different encoders perparameters such replay ratio, which has subsantial effect on the runtime, making direct comparisons difficult. However, we note that image-based methods are among the slowest of the methods we considered, while PointNet is the fastest method by a wide margin. While there are many factors that affect wallelock time, point cloud methods have the advantage that background pixels are already filtered out, which may reduce the compute required for given observation.
run with different hy-
16

---

D Further Ablations
D.1 Normalization Type
ThreadInHole
DeflectSpheres
PushChair
0.8
0.8
0.8
Success
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
OpenCabineeDrawer
OpenCabinetDoor
TurnFaucet
0.8
0.8
0.8
per pointcloud
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Enviromennt Steps (x10%)
Envinonment Stepe (x 10")
Envizonment (x10%)
Figure 14: Average success rates and 95% Bootstrap Confidence Intervals for agents trained with fixed and per- per-pointcloud normalization on select environments.
To show the effect of different methods for point cloud normalization, we train agents using the opposite normalization type for each environment. Per-pointcloud normalization scales each point cloud independently by subtracting the mean and dividing by the maximum value over all 3 coor- dinates, such that the longest axis is mapped to [-,, Per-pointcloud normalization is used for ThreadInHole. Static normalization subtracts a fixed center point from each point cloud and divides by a fixed scale factor, where the center and scale are chosen so that initial observations fall roughly within [-1, 1]. Static normalization used for all other environments. A special case of static nor- malization selects a center the original and scale of 1, resulting in no change, which is used for all ManiSkil12 environments.
Figure 14 shows the effect of normalization type. ThreadInHole depends greatly on per-pointcloud normalization, while other environments appear to be robust to this design choice.
D.2 Point Patch Size
After a point cloud tokenized into point patches, the structure of the patches is captured by the positional encoding, while the structure within point patch can only be captured by the PointNet- based feature extractor. As such, the point patch size determines how fine- fine-grained the agent's per- ception of the scene is. To investigate the effect of the point patch size (also known as kt, because it is used in the KNN step), we train PPRL+Aux agents on select environments using [8, 16, 32), where h = 32 the default value. The runtime l: = 64 is too slow to be practical except for PushChair.
The results of this ablation are shown in Figure 15. We note that learning performance is relatively robust the value of k in all 3 tasks investigated. We suspect this is because the tokenizer is able to extract all relevant point patch features up to size of 64. Future work may investigate the optimal trade-off between adding transformer layers and increasing the point patch size.
17

---

PushChair
OpenCebinelDrawer
TurnFaucet
0.8
0.8
0.8
16
Suceea Bate
32
64
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Enviroament (xeps(x100) (x100))
Environment Steps (x 10%)
Environment Steps (x10%)
Average success rates and 95% Bootstrap Confidence Intervals for agents trained with varying point patch sizes (k value) on select environments.
Figure 15:
18


Reviews:
[{"review_id": "4P4AVKXkYT", "reviewer": "Reviewer_HVuz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper investigated the use of point cloud representations in training RL learners and demonstrated its effectiveness compared to other image-based representations. The authors adopted a patching-based point cloud representation developed in prior work [15-17] for training RL learners in this paper, where a point cloud is divided into local patches, and a transformer encoder processes each patch as a token. The authors successfully trained this point cloud encoder using the critic's gradients and additionally showed the benefits of training with an additional patch reconstruction loss in challenging simulated robotics tasks.", "summary_of_recommendation": "I recommend a weak accept for this paper. Although the paper lacks real-world demonstrations, the findings are still insightful and can lead to more interesting RL papers using point cloud representations in the future. Post-rebuttal: I thank the authors for their efforts in the additional experiments and answers to the reviewers' questions. While I agree with some concerns regarding the DrQ baseline experiment raised by the other reviewer, I am satisfied with the new results the authors provided with the improved DrQ baseline (i.e., the new results didn't change the original claims made in the paper significantly). The strengths I mentioned in my original review remain the same even after checking other reviews and authors' responses, so I will keep my original score.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "I would like to see answers to 1-3 that I wrote in the weakness section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KDZ2cbGDSz", "reviewer": "Reviewer_eyuT", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposed PointPatchRL (PPRL), an RL method specifically for point clouds that is inspired by prior works that divide the point clouds into overlapping patches and tokenize them. The authors extend this to include masking these patched point clouds and reconstruction -- on the lines of PointMAE (and masked autoencoders in the case of images) specifically for RL. They perform experiments on 5 robotic tasks and show the sample efficiency of the PPRL framework.", "summary_of_recommendation": "I am currently voting for a weak reject of the paper on the basis of (a) lack of fair baselines, (b) and incorrect claims regarding contributions. However, I will make my final decision after reading all the other reviews + authors' rebuttal and engaging with authors during the rebuttal.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Experiments to conduct: I would like to see authors address my concerns about the baseline (Weakness #2) and Wall clock time analysis (Weakness #4).\n\n**Additional Question**\n\n7. Have authors experimented with PointTransformer-v2 and v3 and observed any significant differences in the sample-efficient and/or wall clock speed of RL policies.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iojRuH3BVN", "reviewer": "Reviewer_Gb3L", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Inspired by advances in image representation learning, the paper proposes PointPatchRL (PRRL), a patch-based representation for 3D point-clouds that is learned via Transformer architecture and is further improved by masking parts of the input. The representations are learned end-to-end in downstream reinforcement learning tasks and demonstrate favorable performance in several RL environments compared to other representation approaches.", "summary_of_recommendation": "Overall, I am very fond of the ideas presented in this paper, and I largely agree with the claim that working with point clouds in RL is under-explored. I find several key contributions that are novel with moderate impact and significance. I have raised several concerns in the “Weaknesses” and “Questions” parts of my review that I would like the authors to address, but I think the paper is relevant and would be interesting to the CoRL community. As such, my current recommendation is “Weak Accept”, but I’m willing to increase my score given convincing answers to my concerns and the concerns that might be raised by the other reviewers. **Post-Rebuttal**:  I strongly believe that these additions help make the paper clearer and more complete. I am fairly convinced by the authors response to my review and I acknowledge that I have read the other reviews. The other reviewers were concerned by the DrQ baseline and the applicability of the method to real-world settings, which I largely agree with; however, I don't think that it shadows the contribution of this paper as it is focused on robotics, just in simulation. Overall, if all the modifcations and experiments end-up in the revised paper, I believe this paper should be accepted. As such, I'm happy to recommend acceptance..", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "* The representation is trained end-to-end on specific tasks. What are the benefits of this approach? Many methods today pre-train a representation model which is then utilized (frozen/fine-tuned) in downstream tasks. Do you think such a 2-stage approach can work in your case (e.g., pre-training your model with the reconstruction loss)?\n* The environments in the experiments are simulated and synthetic, how do you think your method would fare with more real-world scenarios where there are also background elements? What would be required to be added to the learning pipeline in order to adapt to such settings?\n* If a static camera for image observations seems to provide a useful signal for learning, is it possible to combine image inputs with PPRL to benefit from both worlds?\n* Additional training details: what kind of hardware was used to train the models? How long does training take for these computational resources? What are the model sizes?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: PRE REBUTTALS:

High level summary of reviews:

Strengths:

- The research aligns with the growing interest in point cloud data for robotics, offering an effective encoder design choice.
- The paper is clear, easy to follow, and presents a straightforward and technically sound approach.
- The method outperforms state-of-the-art image-based approaches in specific tasks.
- The paper explores an important problem regarding encoding choices for point clouds in RL.
- The method is conceptually simple and demonstrates favorable performance in RL environments.
- Sharing of code.
 
Weaknesses:

- The assumption of known camera poses is a limitation, as it's often challenging to obtain accurate poses in real-world scenarios.
- The lack of real-world experiments raises questions about the generalizability of the findings.
- The comparison with DrQ-v2 may be unfair.
- The claim that PointNet is the only prior work using point clouds is incorrect.
- The ablation study on the impact of the number of objects is limited to two tasks.
- The lack of visualizations of successful and failed rollouts makes it hard to evaluate the learned policies.
- The paper lacks thorough discussion/comparison of design choices (e.g., normalization, pooling, masking).

POST REBUTTAL:

The authors added additional DrQ-v2 experiments and made a large number of clarifications and modifications to the manuscript during rebuttal. The paper is a stronger submission as a result. There were some concerns raised around the additional DrQ-v2 experiment (pertaining to aggressive dimensionality reduction using only a single linear layer as well as some details around non-shared encoders for multi-view) however, the concerns pertaining to these experiments boiled down to h-parameter choice - which is clearly documented - and so the experiments are a welcome addition. Despite the modifications, reviewer scores were unchanged and still quite high variance post rebuttal. Given the relevance to the community and the additional experiments, this paper meets the threshold for acceptance.
Decision: Accept