PointPatchRL - Masked Reconstruction Improves Reinforcement Learning on Point Clouds

Authors: balázs  gyenes
nikolai  franke
philipp  becker
gerhard  neumann
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 21:32:45

Abstract: Perceiving the environment via cameras is crucial for Reinforcement Learning (RL) in robotics. While images are a convenient form of representation, they often complicate extracting important geometric details, especially with varying geometries or deformable objects. In contrast, point clouds naturally represent this geometry and easily integrate color and positional data from multiple camera views. However, while point-cloud processing with deep learning has seen many recent successes, RL on point clouds is under-researched, with only the simplest encoder architecture considered in the literature. We introduce PointPatchRL (PPRL), a method for RL on point clouds that builds on the common paradigm of dividing point clouds into overlapping patches, tokenizing them, and processing
the tokens with transformers. PPRL provides significant improvements compared with other point-cloud processing architectures previously used for RL. We then complement PPRL with masked reconstruction for representation learning and show that our method outperforms strong model-free and model-based baselines on image observations in complex manipulation tasks containing deformable objects and variations in target object geometry.

Main paper: 
 introduction: 
 In recent years, Reinforcement Learning (RL) for robots enabled solving increasingly complex tasks [1, 2, 3] . A key factor contributing to these advancements is the development of representation learning techniques for encoding high-dimensional sensor data such as camera images or depth maps into compact, low-dimensional representations [4, 5, 6] . There are a variety of learning objectives to train visual representations for RL, such as image reconstruction [7] , multi-view contrastive loss [8] , or multi-view consistency [9] . However, these methods struggle to extract 3D geometric information about the scene and leave the handling of depth measurements under-explored. Not being able to provide the policy with a 3D inductive bias for resolving ambiguities due to occlusions or camera movement presents challenges for representations learned purely from pixels [10] . Point clouds provide a natural representation of 3D geometry and can straightforwardly combine color and positional information. In contrast to depth images, point clouds leverage the camera parameters to obtain a common geometric representation from one or more camera views, alleviating the issues of pixel-based approaches. While this requires measuring the relative poses between cameras, in many cases this can be done without tracking markers or SLAM algorithms. For example, if all cameras are mounted on the moving robot, the forward kinematics alone are sufficient. In addition, policies trained in simulation on point clouds instead of color images may be easier transferable to the real world [2, 11] , since shapes are inherently easier to simulate than textures.both. Previous works propose sophisticated approaches such as image augmentation [20, 21, 22] , autoencoding-based reconstruction [23, 7, 4] , or other self-supervised methods [24, 25] . Driess et al. [9] learn a representation with the help of a multi-view consistency objective using Neural Radiance Fields (NeRF). Of particular importance to our work are approaches using masked autoencoding [26] on images for RL. Xiao et al. [5] use a pre-trained masked autoencoder for on-policy RL. Seo et al. [27] train a Dreamer world model [7] in the latent space of a masked autoencoder. They show that naively masking image patches can be suboptimal for RL and instead mask convolutional features. In subsequent work, they extend their approach to handle multiple viewpoints and make it robust to viewpoint perturbations [28] , properties which are shared implicitly by our point cloud-based approach. While these approaches can be extended to depth images, we argue that point clouds are a more natural representation of the scene's geometry, as they leverage the camera geometry and straightforwardly merge points from multiple cameras into the same coordinate frame.Point Cloud Representation Learning. Since the advent of PointNet [12] , many neural network architectures for point clouds have been developed, mainly for classification and segmentation tasks. PointNet++ [13] improves upon PointNet by using the Farthest Point Sampling algorithm to hierarchically partition the point cloud. PointTransformer [14] introduces a transformer backbone for point cloud processing, iteratively applying vector attention on point-level features within a small neighborhood and downsampling similar to PointNet++. Point-MAE [16] adapts the popular masked autoencoding paradigm to learn point cloud embeddings, dividing the point cloud into patches, masking them, and using a transformer to reconstruct the masked patches. The network is trained to minimize the Chamfer distance between the reconstruction and the ground truth in the masked point patches. Point-BERT [15] uses block masking instead of random masking and predicts discrete point tokens instead of continuous ones. Point-GPT [17] addresses several weaknesses in Point-MAE and leverages the successful auto-regressive inference paradigm used in natural language processing. Despite their success, many of these architectures have never been used for RL, and we show that using the more modern point patching approach with an auxiliary objective as in PointGPT [17] improves performance on challenging (mobile) manipulation tasks.RL with Point Clouds. Despite recent advances in point cloud processing, RL on point clouds for robotics is under-researched, with the most common application being dexterous manipulation of objects [29, 3, 30] . Huang et al. [29] use a pretrained PointNet feature extractor to learn a general dexterous manipulation policy capable of in-hand orientation of various objects. Chen et al. [3] train a sparse 3D CNN feature extractor end-to-end using a student-teacher learning method, where the teacher has access to privileged state information, while the student only relies on point clouds and proprioception. Wu et al. [30] use imitation learning to solve the dexterous manipulation task, also employing a pretrained PointNet that is finetuned during behavior cloning. Liu et al. [31] explore the impact of different coordinate frames on tasks from ManiSkill2 [32] with point cloud observations combined with proprioception. Ling et al. [33] study the effectiveness of point cloud-based RL compared to image-based RL on a mix of two-dimensional and three-dimensional environments. They find that point cloud observations offer benefits over image observations in environments where understanding agent-object and object-object relationships is important and that a PointNet encoder outperforms a 3D SparseConvNet [34] . Wan et al. [35] make use of PointNet+Transformer [36] , but this requires a segmented point cloud, which is not always available. Peri et al. [37] investigate the use of an encoder based on PointConv [38] for RL in latent imagination. However, a systematic comparison of point cloud encoder architectures is missing in the literature. In particular, there are no works that make use of the point patching paradigm, which we show empirically to be a powerful approach on complex 3D (mobile) manipulation tasks. 
pointpatchrl: 
 PointPatchRL (PPRL) assumes point clouds X ∈ R m×3 with m points. Here, the size m can vary between the individual point clouds X encountered during RL and we thus design an encoder to compute a fixed-length embedding containing all task-relevant information provided by X. We then provide this embedding to an actor and a critic which we train using a soft actor-critic (SAC) [19] .Our point cloud encoder is based on the point patching paradigm [15, 16, 17] with a transformer backbone, adapted to handle point clouds of arbitrary size. If additional color information is available for the points, we can include it by appending corresponding features to each point. We use a sequence pooling layer to obtain the fixed-length embedding required for RL. Besides training the encoder solely using the critic loss, we consider augmenting training by explicit representation learning. We do so using PointGPT [17] , which learns representations via the reconstruction of masked point patches. During training, we separately compute the reconstruction loss with masking and the RL loss based on an embedding without masking, as illustrated in Figure 1 .Tokenization. To process the point cloud X using a transformer, we first need to tokenize it. Here, grouping many points into a single token is desirable, as point clouds have low information density. Thus, we sample n points as centroids C = fps(X), C ∈ R n×3 using Farthest Point Sampling (FPS) [13] , which ensures good coverage and an even distribution of the centroids. Subsequently, we conduct a K-Nearest Neighbours (k-NN) search to select the k nearest points of each centroid to obtain (potentially overlapping) point patches P = knn(X, C), P ∈ R n×k×3 . The coordinates of all points in a patch are normalized relative to the centroid's position. A lightweight version of PointNet [12] , consisting of two MLPs and two max-pooling layers, converts point matches P into input tokens T ∈ R n×D of size D.Transformer Architecture. A standard transformer encoder receives the sequence of tokens T and produces a new sequence of encoded tokens. To this end, we also need to encode the position of each centroid and add it to the respective token. As in PointGPT, we add sinusoidal positional encodings of the centroids' locations to the tokens.Reinforcement Learning. We utilize soft-actor critic (SAC) which is a popular algorithm due to its sample efficiency. SAC learns a policy and a state-action value function network, which both require a fixed-length embedding as input vectors. We use a sequence pooling layer [39] to obtain this embedding. This layer computes a weighted sum over all input tokens, where the weight of each token is a function of that token, i.e.,T pooled = w T T with w = softmax(g(T )) andT pooled ∈ R d .where g is a linear layer. Compared with using an explicit query token to compute a fixed-length embedding, sequence pooling can improve the performance of small transformer networks [39] .If low-dimensional state observations are also available, they are projected with a linear layer to the same dimensionality as the point cloud embedding, and then concatenated with the embedding. Actor and critic networks share a single point cloud encoder, which is updated only with the gradients from the critic loss, as is standard practice in off-policy RL [24, 22] .We extend PPRL with a masked reconstruction objective, to benefit from an additional geometric learning signal. By adding sorting, masking, and a transformer decoder to our architecture, we obtain a modified version of PointGPT. Unless noted, all details below follow PointGPT.After tokenization, the tokens are sorted according to the Morton order [41] (Z-order curve), which maps n-dimensional data to one dimension while preserving the locality of the data points. Because nearby point patches remain close to each other after sorting, the model can rely on neighboring tokens to inform the reconstruction. In contrast to the transformer encoder, the transformer decoder does not receive the encodings of the absolute positions of point patches, but only the relative direction between subsequent patches is encoded, i.e.,E r i = ϕ C s i -C s i-1 ∥ C s i -C s i-1 ∥ 2, for i ∈ {2, . . . , n}, andE r 1 = C S 1where C s i is the ith centroid of the ordered list of centroids. This avoids leaking global information about the point cloud structure through the positional embeddings to the decoder. The masking [40] for all methods on 2 sofaenv and 4 ManiSkill2 environments. Our method, PPRL + Aux, achieves top performance on 5 of 6 environments. On DeflectSpheres, ThreadInHole, and PushChair, PPRL without representation learning achieves roughly the same success as PPRL + Aux, demonstrating the effectiveness of our neural network architecture even without auxiliary learning objectives. On OpenCabinet-Drawer and OpenCabinetDoor, two challenging tasks with non-trivial variations in scene geometry, adding an auxiliary reconstruction loss significantly improves learning, and is required for solving the task. On TurnFaucet, DrQ-v2 outperforms our method, potentially due to the availability of a static camera, making the learning task easier for image-based methods.combines random and causal masking as detailed in Appendix B. Finally, a one-layer prediction head reconstructs point patches from the decoded tokens. The loss is the symmetric Chamfer distance [15] between the reconstructed and ground truth patches, which measures the average distance of each point in the ground truth patch to the closest point in the reconstructed patch and vice-versa.Variable Point Cloud Size. While standard representation learning commonly assumes fixed-sized point clouds, in RL, their sizes can vary by orders of magnitude, especially in environments with a moving camera. For point clouds smaller than a fixed minimum size, PointGPT duplicates random points, which may distort objects' point density and add redundant information. To alleviate this problem, we add padding tokens to the input sequence and adjust the masking procedure to maintain a constant ratio of masked and visible tokens among non-padding tokens while ensuring a consistent length for all sequences in a batch.Reconstruction of Color Features. Although most point cloud representation learning methods ignore point-level features such as color, color may be useful in many tasks. We generalize PPRL and PointGPT to include color by adding color features to the points, which are used as inputs by the tokenizer as well as by the reconstruction loss. We will denote x = [x xyz , x rgb ], x ∈ R 6 as a single point consisting of the coordinate component x xyz ∈ R 3 and the color component x rgb ∈ R 3 . The color reconstruction minimizes the MSE between the predicted color of each point in the predicted point cloud P pd i of patch i and the color of the nearest point in the ground truth point cloud P gt i , i.e., L rgb (P pd i ,EQUATIONwhere nn(a xyz , P gt i ) returns the color of the nearest point to a xyz in the ground truth point cloud P gt i . These correspondences are already available through the Chamfer distance computation. The final auxiliary loss (referred to as Aux, i.e. PPRL + Aux) is then the sum of Chamfer loss and the color reconstruction loss.  
experiments: 
 We compare our method with the auxiliary representation learning loss (PPRL + Aux) and without it (PPRL) on 6 challenging visual robotic manipulation tasks from sofaenv [42] and ManiSkill2 [32] . These environments highlight PPRL's ability to infer non-trivial geometric information from multiple viewpoints and moving cameras. We compare several ablations using different point cloud architectures to showcase the benefits of using point patching with a transformer. Further, we compare against SOTA image-based RL approaches to demonstrate the merits of using point clouds. We report the average success rate and 95% Bootstrapped Confidence Intervals [40] . All results are averaged over 8 seeds, except for methods that fail 1 where we only run 4.Environments. sofaenv [42] provides environments for robot-assisted surgery, focusing on tasks involving deformable objects. For these tasks, the camera's position and orientation change for each trajectory, mimicking viewpoint changes encountered during surgery with endoscopic cameras. This is exceptionally challenging for most image-based approaches but reflects the desiderata of deploying a learning agent in a real-world system without tedious camera calibration. Furthermore, the deformable nature of the objects involved prevents access to concise state information.ManiSkill2 [32] provides challenging robot manipulation tasks, including varying object geometries by sampling object models randomly from PartNet [43] (e.g., cabinets, chairs, faucets, etc.). PushChair, OpenCabinetDrawer, and OpenCabinetDoor are mobile manipulation tasks, using 3 cameras mounted above the mobile robot. TurnFaucet exhibits static manipulation and uses 2 cameras, mounted to the mobile gripper and the static base. Point clouds are given in the egocentric perspective, which is especially challenging for mobile manipulation, as this reference frame is not static. The agent additionally has access to proprioception and the target location and pose.We train on ThreadInHole, PushChair, and TurnFaucet without color, while we use color features for DeflectSpheres, OpenCabinetDrawer, and OpenCabinetDoor, where color is either necessary or helpful for detecting the target. Appendix A provides details for all environments.Ablations and Baselines. We compare our encoder against several popular point cloud architectures from the literature. We consider PointNet [12] , which has previously been used in the literature for RL on point clouds, as well as its successor PointNet++ [13] . To investigate the benefit of transformer-based architectures, we consider PointTransformer [14] . TurnFaucet Figure 4 : Average success rates and 95% Bootstrap Confidence Intervals for agents trained with PPRL + Aux on selected environments with varying numbers of object models. Although there is a general trend that more object models reduce success rates, the effect is not always strong, showing that the encoder can generalize well. OpenCabinetDrawer performs approximately the same for 5, 10, or 15 object models, and only begins to decrease for 25.Additionally, we compare PPRL against several SOTA image-based RL approaches, to showcase the benefits of using point clouds. First, we compare against the model-free DrQ-v2 [22] and the model-based Dreamer [7] approaches. To ensure all approaches receive the same information, both get the full RGBD images from all available cameras. Finally, we compare PPRL against Multi-View Masked World Models (MV-MWM) [28] , which is tailored to integrate image observations from different camera perspectives and uses a Dreamer-based world model to do sensor fusion in a latent space. Appendix B provides details on the hyperparameters and training of the baselines.Figure 2 shows the results for all 6 environments tested. PPRL + Aux outperforms all baselines on 5 of 6 environments tested, while DrQ outperforms our method on the TurnFaucet task although the final performance is similar. While PPRL also works on DeflectSpheres, ThreadInHole, and PushChair without representation learning, representation learning leads to substantial benefits in the challenging OpenCabinetDrawer, OpenCabinetDoor, and TurnFaucet tasks. In these tasks, object geometry is crucial for policy execution, showing that representation learning is helpful for the policy to understand the differences in object geometry. For DeflectSpheres and ThreadInHole, representation learning still speeds up learning, while there is a slight decrease in performance on the PushChair task. These tasks either have rather simple object geometries or the difference in the object geometry does not require different strategies for manipulation (PushChair). PPRL + Aux solves OpenCabinetDrawer and OpenCabinetDoor with success rates of 60% and 50%, respectively, while PPRL without representation learning and the baseline methods are unable to solve them. These environments are challenging mobile manipulation tasks that necessitate adapting the strategy to the target geometry. The handles of each cabinet model have different sizes and poses, and must therefore be grasped differently, and doors must be actuated according to their direction of opening and turning radius. Additionally, all cameras in these environments are mobile, which presents a much more difficult perception problem for image-based methods than for point cloud methods. Figure 3 visualizes the policies for the OpenCabinetDoor environment, showing how the agent responds to variations in the cabinet geometry.In TurnFaucet, DrQ-v2 has a higher initial success rate than our method and a comparable final success rate. We speculate that this is partly because one of the cameras is static, allowing image-based methods to learn representations without needing to generalize across viewpoints. Dreamer and Multi-View Masked World Models also appear to benefit from the static camera in TurnFaucet. All other environments contain no completely static cameras, either due to variation between episodes or the movement of the robot base, leading to the general tendency for point cloud-based methods to outperform image-based methods. Interestingly, PointNet++ also performs competitively on DeflectSpheres, PushChair, and TurnFaucet, despite its relative simplicity. This may be due to how it processes point cloud data at multiple geometric scales, using neighborhoods generated by FPS to iteratively downsample. Surprisingly, PointTransformer performs poorly across all environ- Figure 5 : Average success rates and 95% Bootstrap Confidence Intervals for agents trained with and without color reconstruction loss, on all environments with color point cloud observations. When training without color reconstruction loss, color is still observed, but only the signal from RL encourages the agent to condition on color features. In particular the hard Cabinet tasks profit from explicitly reconstructing the color.ments, despite containing many components common to more successful networks such as PPRL and PointNet++, namely neighborhood computation via FPS and a transformer backbone.Object Model Count. We test the generalization capabilities of our method by training on ManiSkill2 environments with varying numbers of object models. Figure 4 shows the effect of increasing numbers of object models on the sample efficiency. Although more models generally result in a more difficult learning task, the effect is weak below a certain number of models (15 for OpenCabinetDrawer, 10 for OpenCabinetDoor). This suggests that the encoder's capacity is sufficient for learning more object models, and the limiting factor may rather be the burden of exploring and finding generalizable RL policies with an increasing number of geometries.Color Reconstruction. To investigate the effect of color reconstruction we remove color reconstruction for all 3 environments that use color point clouds. Without the reconstruction loss, only the loss signal from RL encourages the agent to incorporate color into the embedding. The results are shown in Figure 5 . Surprisingly, the color reconstruction objective provides no significant benefit for the 3 environments tested, despite the fact that DeflectSpheres encodes some task-relevant information solely through color. Although it is important to provide color to the policy to solve the task, the learning signal from RL alone is enough to learn any color features required. Furthermore, this shows that the Chamfer distance loss is informative enough to learn much of the task-relevant information in the point cloud without any additional modifications. 
conclusion: 
 We present PPRL, a novel method for RL on point clouds, and an accompanying auxiliary reconstruction objective based on PointGPT. PPRL exploits the inherent properties of point clouds to combine color and positional information from all available cameras and is robust to changing and moving viewpoints. We evaluate PPRL on a series of challenging visual robot (mobile) manipulation tasks from sofaenv and ManiSkill2 and find thatPPRL outperforms popular baselines, even without the auxiliary reconstruction objective. In combination with the auxiliary representation learning objective, our method improves sample efficiency and final success rates in most environments compared to both point cloud-based and image-based baselines. These advantages are more significant for more challenging environments, such as OpenCabinetDrawer and OpenCabinetDoor. A key insight is that point cloud methods offer robustness to changing viewpoints in environments without a static camera, a domain where image-based methods are known to struggle [10] .Limitations. While we demonstrated the effectiveness of existing representation learning methods for RL on point clouds, findings from image-representation learning with masked reconstruction for RL [27] suggest that special treatment may further improve the performance. While the convolutional feature masking method of [27] is not straightforwardly applicable to point clouds, similar extensions could be investigated. Furthermore, we have not included other sensor modalities such as proprioception in the representation learning, which has shown benefits for image-based RL [44] . ManiSkill2: All mobile manipulation tasks (OpenCabinetDrawer, OpenCabinetDoor, and PushChair) have 3 cameras mounted above the robot base facing outwards, spaced radially at 120 • . In these tasks, the camera FOV was increased to 1.5, such that the cabinet/chair is almost always visible from at least one perspective. In OpenCabinetDoor, only the first 10 cabinet models are used for training and evaluation. In addition, the target door is not randomized but fixed for each cabinet model. OpenCabinetDrawer, because it is slightly easier, is not changed, using all 25 models and a randomly chosen target drawer for each episode. TurnFaucet is a static manipulation task with only 2 cameras, one mounted on the (static) base and one inside the gripper. In TurnFaucet, only the first 10 faucet models are used for training and evaluation. In all environments, joint position delta control is used for the robot arm, and joint velocity control is used for the robot base, if applicable. Observations contain point clouds and a low-dimensional state vector containing robot proprioception and the target location. 
 
 Appendix: Discounts and Time Limits: Discounts for all tasks are: 0.99 for ThreadInHole and PushChair, 0.9 for DeflectSpheres, and 0.85 for OpenCabinetDoor, OpenCabinetDrawer, and TurnFaucet. The time limits for all tasks are: 300 for ThreadInHole, 500 for DeflectSpheres, and 200 for all ManiSkill2 tasks. In OpenCabinetDrawer, OpenCabinetDoor, and TurnFaucet, trajectories are ended only on a timeout, not task success. In ThreadInHole, DeflectSpheres, and PushChair, trajectories are ended either on a timeout or task success. PushChair is the only ManiSkill2 environment treated like this, because we found that ending on task success is crucial for training.Point Clouds: All environments are rendered with a resolution of 128x128, except PushChair, which has a resolution of 64x64. Egocentric point clouds are created from depth images using the linear transformations given by the camera intrinsic (focal length) and extrinsic (position and orientation) parameters. Point clouds are then post-processed using the following steps: cropping, appending target points, downsampling, and normalization (note that not all environments use all of these steps). First, points beyond the boundaries of the scene are discarded. In ManiSkill2 environments, the floor is also cropped out, and furthermore in OpenCabinetDrawer and OpenCabinetDoor, any points more than 0.5m in front of the cabinet are also cropped out. In OpenCabinetDrawer, OpenCabinetDoor, and PushChair, 50 additional points are appended to the point cloud, sampled from a uniform cube (of side length 7cm for OpenCabinetDrawer and OpenCabinetDoor and 14cm for PushChair). Although this information is also present in the state vector observation, these points serve to indicate the visual target position within the observation directly. In ThreadInHole, Deflect-Spheres, and PushChair, we use voxel grid sampling (with a voxel size of 5mm for sofaenv and 5cm for PushChair) to downsample the point clouds, followed by random downsampling (without replacement) to a maximum point cloud size of 1000 points. In OpenCabinetDrawer, OpenCab-inetDoor, and TurnFaucet, we randomly downsample each point cloud (without replacement) to a maximum point cloud size of 800 points. For DeflectSpheres, we normalize all observed point clouds by subtracting a fixed center point and dividing by a fixed scale factor. The scale and center point are chosen such that the extrema of a point cloud from a typical initial observation are roughly within [-1, 1]. For ThreadInHole, we normalize each observed point cloud independently by subtracting the mean and then dividing by the maximum absolute value over all 3 coordinates, such that the longest axis is mapped to [-1, 1] . This method of normalization (compared to that of Deflect-Spheres) was crucial for training. Point clouds for ManiSkill2 environments are not normalized as they already lie roughly within [-1, 1].Masked World Models, we run training in multi-view mode, which masks random viewpoints and reconstructs them. We disable the behavioural cloning loss, as our method does not have access to demonstrations. 2 provides a comparison of the average wallclock runtimes for all methods considered. All runs used a single Nvidia A100 GPU, 19 Intel Xeon Platinum 8368 CPUs, and 128Gb of RAM (maximum memory, not average). In some cases, different encoders were run with different hyperparameters such as replay ratio, which has a subsantial effect on the runtime, making direct comparisons difficult. However, we note that image-based methods are among the slowest of the methods we considered, while PointNet is the fastest method by a wide margin. While there are many factors that affect wallclock time, point cloud methods have the advantage that background pixels are already filtered out, which may reduce the compute required for a given observation.Failing means that none of the 4 seeds ever reached > 10% success. 


Decision: PRE REBUTTALS:

High level summary of reviews:

Strengths:

- The research aligns with the growing interest in point cloud data for robotics, offering an effective encoder design choice.
- The paper is clear, easy to follow, and presents a straightforward and technically sound approach.
- The method outperforms state-of-the-art image-based approaches in specific tasks.
- The paper explores an important problem regarding encoding choices for point clouds in RL.
- The method is conceptually simple and demonstrates favorable performance in RL environments.
- Sharing of code.
 
Weaknesses:

- The assumption of known camera poses is a limitation, as it's often challenging to obtain accurate poses in real-world scenarios.
- The lack of real-world experiments raises questions about the generalizability of the findings.
- The comparison with DrQ-v2 may be unfair.
- The claim that PointNet is the only prior work using point clouds is incorrect.
- The ablation study on the impact of the number of objects is limited to two tasks.
- The lack of visualizations of successful and failed rollouts makes it hard to evaluate the learned policies.
- The paper lacks thorough discussion/comparison of design choices (e.g., normalization, pooling, masking).

POST REBUTTAL:

The authors added additional DrQ-v2 experiments and made a large number of clarifications and modifications to the manuscript during rebuttal. The paper is a stronger submission as a result. There were some concerns raised around the additional DrQ-v2 experiment (pertaining to aggressive dimensionality reduction using only a single linear layer as well as some details around non-shared encoders for multi-view) however, the concerns pertaining to these experiments boiled down to h-parameter choice - which is clearly documented - and so the experiments are a welcome addition. Despite the modifications, reviewer scores were unchanged and still quite high variance post rebuttal. Given the relevance to the community and the additional experiments, this paper meets the threshold for acceptance.

Reviews: review: title: title: It's a well-written paper with a simple but effective approach.
paper_summary: paper_summary: The paper investigated the use of point cloud representations in training RL learners and demonstrated its effectiveness compared to other image-based representations. The authors adopted a patching-based point cloud representation developed in prior work [15-17] for training RL learners in this paper, where a point cloud is divided into local patches, and a transformer encoder processes each patch as a token. The authors successfully trained this point cloud encoder using the critic's gradients and additionally showed the benefits of training with an additional patch reconstruction loss in challenging simulated robotics tasks.
main_review: main_review: Strengths
1. The research direction aligns with the growing interest in using point cloud data in the robotics community, providing a good encoder design choice for processing point cloud data.
2. The paper is well-written and easy to follow. The proposed approach is also straightforward, effective, and technically sound.
3. The authors demonstrated that the proposed method works even when other state-of-the-art image-based approaches struggle in certain tasks, such as ThreadInHole, DeflectSpheres, OpenCabinetDrawer, and OpenCabinetDoor.

Weaknesses
1. The proposed work assumes known camera poses, which is a strong assumption given that it is often hard to obtain camera poses in real-world environments. Depending on applications, visual markers can't be used in real-world environments, so often, one would need to rely on SLAM algorithms to estimate camera poses from real-world videos. This is prone to errors and can negatively affect the registration of point clouds in a common reference frame later. This limitation should be clearly specified, as the paper does not currently address the potential challenges of applying the proposed method in real-world environments.
2. All experiments are only done using simulation data. The paper's claims regarding the use of point cloud representations could have been strengthened significantly with real-world results, especially given the argument in lines 31-33 that there is a potentially smaller sim-to-real gap in point cloud data than in RGB images.
3. In the TurnFaucet result in Figure 2, DrQ-v2 seems to be trained with an additional static camera view. I feel this is not a fair comparison if the proposed method only took an egocentric view as input. The comparison would be fairer if DrQ-v2 were also trained with the same view as the proposed method. 
4. The ablation study on a variable number of objects (lines 267-273) is conducted with only two tasks, making it difficult to draw general conclusions. Depending on the task, the effect may be more significant, especially if the task requires understanding interactions between multiple objects. It would be helpful to note this limitation in the ablation study to provide a clearer understanding of its scope.
questions: questions: I would like to see answers to 1-3 that I wrote in the weakness section above.
limitations: limitations: 2
review_summary: review_summary: I recommend a weak accept for this paper. Although the paper lacks real-world demonstrations, the findings are still insightful and can lead to more interesting RL papers using point cloud representations in the future. Post-rebuttal: I thank the authors for their efforts in the additional experiments and answers to the reviewers' questions. While I agree with some concerns regarding the DrQ baseline experiment raised by the other reviewer, I am satisfied with the new results the authors provided with the improved DrQ baseline (i.e., the new results didn't change the original claims made in the paper significantly). The strengths I mentioned in my original review remain the same even after checking other reviews and authors' responses, so I will keep my original score.
title: title: Good motivation, need more concrete experiments and fair+stronger baselines
paper_summary: paper_summary: This work proposed PointPatchRL (PPRL), an RL method specifically for point clouds that is inspired by prior works that divide the point clouds into overlapping patches and tokenize them. The authors extend this to include masking these patched point clouds and reconstruction -- on the lines of PointMAE (and masked autoencoders in the case of images) specifically for RL. They perform experiments on 5 robotic tasks and show the sample efficiency of the PPRL framework.
main_review: main_review: **Strengths:**

1. The work investigates an important problem that is very much relevant to the robot learning community on what should be the choice of encoding for point clouds to facilitate sample-efficient RL policies.

2. I appreciate the authors sharing their code for this work. I haven't run it but, I have glanced over the main parts of the code.

**Weaknesses:**

1. Claim that only PointNet has been used: This claim has been made by the authors several times (L35-37, L89-91, L105-107) -- however works like [1] use PointNet+Transformer and [2] use PointConv. Hence, the claim that authors are the first to investigate this (to the best of their knowledge) is incorrect. I'd ask the authors to kindly remove the claim.

2. DrQ-v2 baseline architecture: DrQ-v2, a CNN baseline used for comparison has only a **single** CNN encoder for all the multi-camera images. This is a poor choice of encoder and an unfair baseline. Instead, each camera should have a separate CNN encoder as the filters learned would be specific to each camera.
Additionally, details on how this 117600 dim vector is projected down to 50 are missing. 50-dimensional vector (concatenated with another 50-dim for state encoding) to the policy seems to be a **very small** latent. The typical latent size in literature is of the order of 256/512/1024. I believe the authors need to tune the baseline carefully to have a meaningful comparison and claim that PPRL does better. Given that DrQ-v2 had better sample efficiency with this poor architecture on the TurnFaucet task, I'd expect its performance to significantly increase after accommodating the above-mentioned changes.

3. Instead of only having successful trajectories in Figure 3, I would also like to see the patch-reconstruction quality being evaluated both quantitatively as well as qualitatively in the main paper. Please provide some reconstruction results for PPRL + Aux.

4. Wall clock time analysis: I would expect that point cloud encoders in general to be slower than CNNs for RGB-D images for RL. I would like to see a comparison of wall clock time between **all** the baselines (different point cloud encoders as well as CNNs). This analysis would give a better idea of how much gap well-performing point cloud encoders (such as PointNet++ and PPRL) have compared to PointNet and CNNs.

5. Color Reconstruction experiment: On L277-278, the authors claim that "color reconstruction tends to work better 277 on OpenCabinetDoor" -- I'm not fully on board with this claim. In OpenCabinetDoor, there is a significant overlap of the 95% Bootstrap Confidence Intervals for a policy that uses Color Loss in the Aux. loss and one that does not. So, it is hard to make the claim that color information is significantly helpful.

6. [Very minor comment -- not included to make the decision] I believe the addition of Chamfer loss + color loss is the Auxiliary loss, however, Sec 3.2 that talks about it does not make a reference to the term "Aux. loss" or "Auxiliary loss" and introduction of the word "Auxiliary" directly in Sec 4 (Experiments) is confusing to the reader. Please include a reference to "Auxiliary/Aux." in Sec 3.2.

----

**References**

[1] UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning, Weikang Wan et al., CVPR 2023

[2] Point Cloud Models Improve Visual Robustness in Robotic Learners, Skand Peri et al., ICRA 2024
questions: questions: Experiments to conduct: I would like to see authors address my concerns about the baseline (Weakness #2) and Wall clock time analysis (Weakness #4).

**Additional Question**

7. Have authors experimented with PointTransformer-v2 and v3 and observed any significant differences in the sample-efficient and/or wall clock speed of RL policies.
limitations: limitations: 3
review_summary: review_summary: I am currently voting for a weak reject of the paper on the basis of (a) lack of fair baselines, (b) and incorrect claims regarding contributions. However, I will make my final decision after reading all the other reviews + authors' rebuttal and engaging with authors during the rebuttal.
title: title: Interesting paper tackling the under-explored problem of 3D representation learning in RL
paper_summary: paper_summary: Inspired by advances in image representation learning, the paper proposes PointPatchRL (PRRL), a patch-based representation for 3D point-clouds that is learned via Transformer architecture and is further improved by masking parts of the input. The representations are learned end-to-end in downstream reinforcement learning tasks and demonstrate favorable performance in several RL environments compared to other representation approaches.
main_review: main_review: **Strengths**:
* Well-motivated: tackles an interesting representation learning problem for utilizing 3D inputs in RL.
* Conceptually simple (but the technical implementation can get complex).
* Sufficient and convincing experimental benchmark.
* Open-source code (!)

**Weaknesses**:
* Visualizations: it has become common to provide videos of successful and failed rollouts of the learned policies as it is hard to evaluate it visually (e.g., how natural are the learned motions?) from a short sequence of images. However, I couldn’t find any in the supplementary files or in an anonymous webpage (e.g., Google sites).
* Ablations of design choices: as the authors mention in the appendix, the choice of normalization was crucial to get the reported performance. I think a more thorough discussion/comparison of design choices (even if it ends up in the appendix) would complement this paper. For example, choice of PC normalization, pooling mechanism (line 139), masking strategy, number (K) of nearest neighbors and etc… 
* Missing training details (See “Questions”).

**Limitations**: Most of the limitations are clearly addressed. A discussion of failure cases would be appreciated as it might help gain insights on further improvements to the proposed approach.
questions: questions: * The representation is trained end-to-end on specific tasks. What are the benefits of this approach? Many methods today pre-train a representation model which is then utilized (frozen/fine-tuned) in downstream tasks. Do you think such a 2-stage approach can work in your case (e.g., pre-training your model with the reconstruction loss)?
* The environments in the experiments are simulated and synthetic, how do you think your method would fare with more real-world scenarios where there are also background elements? What would be required to be added to the learning pipeline in order to adapt to such settings?
* If a static camera for image observations seems to provide a useful signal for learning, is it possible to combine image inputs with PPRL to benefit from both worlds?
* Additional training details: what kind of hardware was used to train the models? How long does training take for these computational resources? What are the model sizes?
limitations: limitations: 3
review_summary: review_summary: Overall, I am very fond of the ideas presented in this paper, and I largely agree with the claim that working with point clouds in RL is under-explored. I find several key contributions that are novel with moderate impact and significance. I have raised several concerns in the “Weaknesses” and “Questions” parts of my review that I would like the authors to address, but I think the paper is relevant and would be interesting to the CoRL community. As such, my current recommendation is “Weak Accept”, but I’m willing to increase my score given convincing answers to my concerns and the concerns that might be raised by the other reviewers. **Post-Rebuttal**:  I strongly believe that these additions help make the paper clearer and more complete. I am fairly convinced by the authors response to my review and I acknowledge that I have read the other reviews. The other reviewers were concerned by the DrQ baseline and the applicability of the method to real-world settings, which I largely agree with; however, I don't think that it shadows the contribution of this paper as it is focused on robotics, just in simulation. Overall, if all the modifcations and experiments end-up in the revised paper, I believe this paper should be accepted. As such, I'm happy to recommend acceptance..