CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects
Authors: Yoonyoung Cho
Junhyek Han
Yoontae Cho
Beomjoon Kim
Venue: ICLR 2024 poster (2024)
Keywords: pretraining
robotics
manipulation
object representation
representation learning
Primary area: applications to robotics, autonomy, planning
TL;DR: contact-based representation learning for nonprehensile robotic manipulation on objects with general geometry, with zero-shot real-world transfer.

Abstract: Nonprehensile manipulation is essential for manipulating objects that are too thin, large, or otherwise ungraspable in the wild. To sidestep the difficulty of contact modeling in conventional modeling-based approaches, reinforcement learning (RL) has recently emerged as a promising alternative. However, previous RL approaches either lack the ability to generalize over diverse object shapes, or use simple action primitives that limit the diversity of robot motions. Furthermore, using RL over diverse object geometry is challenging due to the high cost of training a policy that takes in high-dimensional sensory inputs. We propose a novel contact-based object representation and pretraining pipeline to tackle this. To enable massively parallel training, we leverage a lightweight patch-based transformer architecture for our encoder that processes point clouds, thus scaling our training across thousands of environments. Compared to learning from scratch, or other shape representation baselines, our representation facilitates both time- and data-efficient learning. We validate the efficacy of our overall system by zero-shot transferring the trained policy to novel real-world objects. We highly recommend the video attached in the supplementary material. Code and videos are available at \url{https://sites.google.com/view/contact-non-prehensile}.

Published as a conference paper at ICLR 2024
CORN: CONTACT-BASED OBJECT REPRESENTATION FOR NONPREHENSILE MANIPULATION OF GENERAL UNSEEN OBJECTS
Yoonyoung Cho; Junhyek Han;' Yoontae Cho, Beomjoon Kim
Korea Advanced Institute of Science and Technology {yoonyoung. cho, junhyek.har yoontae.cho) beomjoon, kim Okiisttaast.aa kr
ABSTRACT
Nonprehensile manipulation is essential for manipulating objects that are too thin, large, or otherwise ungraspable in the wild. To sidestep the difficulty of con- tact modeling in conventional modeling ing (RL) has recently emerged as promising alternative. However, previous RL approaches either lack the ability to generalize over diverse object shapes, or use simple action primitives that limit the diversity of robot motions. Furthermore, using RL over diverse object geometry is challenging due to the high cost of train- ing a policy that takes in high- dinensiinnsii sensory inputs. We propose a novel contact- -htact-baase object representation and pretraining pipeline to tackle this. To enable massively parallel training, we leverage lightweight patch-based trans- former architecture for our encoder that processes point clouds, thus scaling our training across thousands of environments. Compared to learning from scratch, or other shape representation baselines, our representation facilitates both time- and data-efficient learning. We validate the efficacy of our overall system by zero-shot transferring the trained policy to novel real-world objects. Code and videos are available https://sites.gont
approaches, reinforcement learn-
INTRODUCTION
Robot manipulators can transform day-to-day lives by alleviating humans from taxing physical labor. Despite this potential, they have remained largely confined to picking and placing objects in limited settings such manufacturing logistics facilities. One key reason for such confined usage is their limited action repertoire: while prehensile manipulation has made great improve- ments (Mahler et al., 2017; Wan al., 2023), there are several situations in the wild where pick- and-place is not an option. For instance, consider the scenarios in Figure 1, where the robot is tasked manipulate diverse household objects a target pose. While the effective strategy differs for each object, the predominant approach of grasping and placing objects is not viable. To manipulate such objects in the wild, the robot must perform nonprehensile actions such as pushing, dragging, and toppling by reasoning about the object geometry (Kao et al., 2016).
Traditionally, this problem has been tackled using planning algorithms that model the dynamics of the system and compute motions using optimization (Posa et al., 2014) or tree search (Cheng et al., 2022). However, this necessitates full system information about object geometry and inertial param- eters, which generally unavailable from sensory observations, especially for novel objects. Even when such models are available, the complex search space renders planning algorithms impractically slow. Recently, deep reinforcement learning (RL) has surfaced
promising alternative, where nonprehensile policy trained in a simulator (Kim et al., 2023; Zhou et al., 2023), then zero-shot transferred the real world. However, algorithm offers the level of generality over object shapes and diversity in motions necessary for our problem: they either generalize over objects but are con- fined to rudimentary motions such as poking (Zhou al., 2023) or affords rich motions but uses rudimentary object representation that precludes generalization over shapes (Kim et al., 2023).
"qqual contribution

---

Published as a conference paper at ICLR 2024
Figure I: Snapshot our robot performing diverse real-world object manipulation tasks. The first column shows the initial and goal object pose (transparent) of the task, and the green region marks the robot's workspace. Row 1: Raising a cup bottom-side up. The cup's grasp and placement poses occluded by the collision with the table; since the cup is dynamically unstable, the robot must also support the object to avoid toppling the object out of the workspace during manipulation. Row 2: Flipping the wipe dispenser upside down, which is too wide to grasp. Since may roll erratically, frequent re-contacts and dense closed-loop motions are required to stabilize the object during manipulation. Row 3: Moving book that is too thin
grasped; to drag the book, the robot must finely adjust the pressure to allow for both reorientation and translation. Row 4: Flipping a toy crab. Given its concave geometry, the robot must utilize the interior contacts to pivot the crab.
(a) Case where the robot to topple cup.
(b) Case where robot to push block.
Figure 2: Even among similar-looking states, the interaction outcome varies drastically depending on the presence of contact. (a-left) the gripper passes near the cup, yet not quite contact. As the robot narrowly misses the object, the object remains still. (a-right) the gripper engages with the cup, leading to a successful topple. (b-left) The robot relocates to the left the block to push it to the goal(dark). By avoiding unintended collision, it is well-positioned to push the object. (b-right) due to spurious contact, the gripper accidentally topples the block, making the goal farther to reach.
We propose a novel object representation learning algorithm for nonprehensile manipulation that can be jointly used with the action-space from Kim et (2023) achieve both generality over shapes and diversity in motions. The key challenge is defining the pretraining task. Given that our objec- employ self-supervised shape-learning,
tive is generalize across shapes, one naive approach where the geometric representation is trained on contrastive or shape-reconstrrution objectives (Pang al.. 2022; Yu et al., 2022; Xie al., 2020: Zeid et al., 2023). However, reconstructing the full shape representation is needlessly complex: the full shape includes many intricate features, such as internal geometry, concavities, or surface details, which may not directly influence the performance of the manipulation policy. Even for complex object, only subset of its surface interacts with the gripper. Thus, while the contactable parts of an object must be represented at high fidelity, the remainder be omitted without influencing the performance of the downstream policy.
Our key insight is that recognizing the presence and location of contacts between the object and the robot's hand is crucial for nonprehensile manipulation. For RL, good representation must necessarily distinguish between states with high and low values, and in contact-rich manipulation, this is primarily determined by what forces and torques be applied to the object in the given state.
2

---

Published as a conference paper at ICLR 2024
This, in turn, depends on the presence and position of the contact between the object and the robot's hand. This intuition is demonstrated in Figure 2. Based on this insight, we propose a pretraining task where the point cloud encoder is trained to predict what part of the object is in collision with the robot's end-effector
Our other contribution is the design of efficient architecture that can process high-dimensional
point clouds, in a way that can scale to massively parallel GPU-backed simulations. Existing well- known point-cloud processing architectures such as PointNet (Qi al., 2016; 2017) inherently incur substantial inference time and memory usage while processing individual points. Inspired by the re- cent success of patch-based vision architectures (Dosovitskiy et al., 2020; Yu et al., 2022: Pang et al., 2022: Zhang et al., 2022; Zeid et al., 2023), we leverage a patch-based transformer to efficiently en- code the point cloud. Since single point in a point cloud does not carry significant information, grouping local points into single unit(patch) (Yu et al., 2022) can effectively reduce the computa- tional burden of launching thousands of point-wise operations as in PointNets (Qi et al., 2016) or Point-based Transformers (Zhao et al., 2021), resulting in significant computational gains.
We call our framework Contact-based Object Representation for Nonprehensile manipulation (CORN). We show that by leveraging CORN, we can efficiently train nonprehensile manipula- tion policy that can execute dexterous closed-loop joint-space maneuvers without being restricted to predefined motion primitives. Further. by employing an efficient patch-based point-cloud processing backbone, we can achieve highly time-efficient training by allowing the policy to scale to massively parallel environments. By adopting our full system, we demonstrate state-of-the-art performance in nonprehensile manipulation of general objects in simulated environments and zero-shot transfer to unseen real-world objects.
2 RELATED WORK
Planning algorithms In planning-basse based approaches, nonprehensile manipulation problem is often tackled using gradient-based optimization or graph search. In optimization, the primary challenge arises from optimizing the system of discontinuous dynamics, stemming from contact mode tran- sitions. To combat this, prior works resort to softening contact mode decision variables (Mordatch al., 2012) or introducing complementarity constraints (Posa al., 2014; Moura al., 2022). However, due to the imprecision from the smooth approximation of contact mode transitions, inex- act contact dynamics models, and difficulty in precisely satisfying contact constraints, the resulting motions are difficult to realize in the real world. On the other hand, graph-search approaches han- dle discontinuous system dynamics by formulating the problem as search over a graph on nodes encoding states such as the robot's configuration and the object's contact mode, and the edges en- coding transition feasibility and relative motion between the nodes (Maeda & Arai, 2005; Maeda et al., 2001; Miyazawa et al., 2005). As result of handling discrete dynamics transitions without approximation, these methods can output more physically realistic motions that can be deployed in the real world (Cheng et al., 2022: Liang al., 2022). However, to make the search tractable, they resort to strong simplifications such as quasi-static assumption on the space of motions (Cheng et al., 2022; Hou& & Mason, 2019), or restricting the edges to predefined primitives contact modes (Zito et al., 2012). As a result, these works have been limited to tasks that only require simple motions and sparse contact-mode transitions.
Moreover, because both of these approaches must compute plan in a large hybrid search space with discontinuous dynamics, they are too slow to use in practice. Further, they also require the physical parameters of objects, such as mass and friction coefficients, to be known a priori, which undermines the practicality of these methods. Since the objects may vary significantly in the wild (Figure 5), acquiring ground-truth geometric and physical attributes of target objects is intractable.
Reinforcement learning algorithms Recently, several works have proposed to leverage learning to directly map sensory inputs to actions in nonprehensile manipulation tasks, circumventing the computational cost of planning, inaccuracy of hybrid dynamics models, and difficulty in system identification from high dimensional sensory inputs (Yuan et al., 2018; 2019; Lowrey et al., 2018; Peng et al., 2018: Ferrandis et al., 2023; Kim et al., 2023; Zhou & Held, 2023: Zhou et al., 2023). In Kim et al. (2023). they propose an approach that outputs diverse motions and effectively performs lime-varying hybrid force and position control (Bogdanovic et al., 2020), by using the end-effector target pose and controller gains as action space. However, since they represent object geometry via
3

---

Published as a conference paper at ICLR 2024
its bounding box, their approach has limited generalization capability across shapes. Similarly, other approaches only handle simple shapes such as cuboids (Yuan et al., 2018; 2019; Ferrandis et al., 2023; Zhou & Held, 2023), pucks (Peng et al., 2018), eylinders (Lowrey et al., 2018).
In HACMan (Zhou et al., 2023), they propose an approach that generalizes to diverse object shapes using end-to-end training. However, they are limited to 3D push primitives defined fixed-direction poking motions applied on a point on the point cloud. Due to this, they must retract the arm after each execution to observe the next potential contact locations. Further, end-to-end training with point clouds requires a significant memory footprint, it is difficult to use with massively parallel GPU-based simulations (Makoviychuk et al., 2021): the large resource consumption significantly limits the number of environments that can be simulated in parallel, undermining the parallelism and damaging the training time. Moreover, end-to-end RI. on high-dimensional inputs is prone to instability (Eysenbach et al., 2022) and sample-ineffiiiency noisy gradients from RL serve as poor training signals for the representation model (Banino et al., 2022), and distributional shift that occurs whenever the encoder gets updated (Shah & Kumar, 2021) slows down policy training. We believe for these reasons, they had to resort to simple primitives that make exploration easier. We use pretraining and efficient architecture to get around these problems.
Along with HACMan, other methods use simple primitives (Yuan et al., 2019; Zhou et al., 2023). But for problems we are contemplating, use of such open-loop primitives precludes dense feedback motions. This is problematic for highly unstable (e.g. rolling or pivoting) objects that move errat- ically, which require the robot to continuously adjust the contact. For instance, in Figure (row 1, column 3), the robot must adjust the contact to prevent spurious pivoting of the ceramic cup while rolling the handle the other side; in Figure (row 2, column 6), the robot must align the contact with the rotational axis while lifting the wipe dispenser to prevent accidentally toppling the dis- penser to either side. In our work, we adopt controller with end-effector subgoals with variable gains (Bogdanovic et al., 2020; Kim et al.. 2023) as our action space, which allows us to perform dense, closed-loop control of the object.
Representation learning on point clouds Diverse pretraining tasks have been proposed for learn- ing representation for point cloud inputs, such as PointContrast (Xie et al., 2020), which uses contrastive loss point correspondences, cluded points from partial views. More recently, inspired by advances from NLP (Devlin et al., 2018) and 2D vision (He et al., 2022; Baevski et al., 2022), self-supervised representation learning methods on masked transformers have gained attention (Yu et al., 2022: Zhang et al., 2022; Pang et al., 2022; Zeid et al., 2023). While Point-BERT (Yu et al., 2022), PointM2AF (Zhang et al., 2022) and PointMAF (Pang et al., 2022) reconstruct the masked points from the remaining points, Vec (Zeid al., 2023) follows Data2 (Baevski et al., 2022) and instead estimate the latent patch embeddings, outperforming above baselines in shape classification and segmentation.
OcCo (Wang et al., 2021), which reconstructs oc-
As a result, these methods learn highly performant shape representations, demonstrating state-of- the-art performance across tasks such as shape classification or segmentation (Zeid et al., 2023). However, the same impressive performance is often inaccessible with smaller models (Fang et al., 2021: Shi et al., 2022) due to the difficulty of the pretraining task (Fang et al., 2021) or model collapse (Shi et al., 2022). While this necessitates large models, adopting a large encoder for policy training in GPU-based simulators is undesirable, as it can dominate the compute-cost and memory footprint during policy rollout, limiting the parallelism in massively-paralle RL setups.
In robotics, several alternative representation learning approaches have been proposed. Huang et al. (2021) pretrains their point cloud encoder by predicting object class and relative offset from the goal orientation. Similarly, Chen et al. (2022) pretrains sparse 3D CNN on object class, relative orientation from the goal, and the joint positions of the robot hand. While these methods have been proven effective on in-hand re-orientation tasks for inferring the object's 's pose from point cloud observations, the benefits are smaller for nonprehensile manipulation tasks which are more sensitive to the specific presence and location of contacts (see Figure 2)'. In comparison, we directly represent an object with an embedding that is used to predict such information.
'hiile (Chen et al., 2022) demonstrated that in hn-dand reorientation task can be solved solely from pose- based observations without observing the underlying object geometry, the same success cannot be achieved in our domain (Figure 6)
4

---

Published as a conference paper at ICLR 2024
Gareve
Carou1
Camera
(a) Real-world setup.
(b) Simulation setup,
Figure 3: Our real-world (left) and simulated (right) domains.
CONTACT-BASED OBJECT REPRESENTATION FOR NON-PREHENSILE MANIPULATION
We tackle the problem rearranging an object of arbitrary geometry on a tabletop to a specified 6D relative pose from the initial pose using nonprehensile manipulation, such pushing, toppling, and pivoting. Our setup includes a manipulator with simple gripper, equipped with propri- oceptive sensor and table-mounted depth cameras. We assume that a rigid-body target object is well-singulated on the table, and both the initial and goal poses are stable at rest. Figure 3 illustrates environment setup.
3.1 OVERVIEW
Our framework goes through three stages of training before being deployed in the real world: (1) pretraining, (2) teacher policy training with privileged information, and (3) student policy training with partial information only available in the real world. These three stages happen entirely in simulation based on Isaac Gym (Makoviychuk et al., 2021). Our overall system is shown in Figure 4.
fornic
Conteit
Decoder
Tolario MEP
Encoder
Contact Network
Jahnt State
Arter
[Studend]
Duers
Attenbinn
Rei Goal
Conot
Teacher Polley
Polic::
Plys
Distil
Figure 4: Our system and model architecture. The contact network consists of point cloud encoder (red) and contact-prediction decoder (green), passing the point cloud embeddings the teacher policy module (blue). Student module (orange, omitted) detailed Section A.1.
In the pretraining phase, we train our encoder module (Figure 4, red), which takes in point-cloud observations [g512x3 and the hand state e R (position and 6D orientation (Zhou et al., 2019)). and outputs patch-wise embedding of the object cloud. To train the network, we pass the embeddings through contact prediction header(Figure 4, green) that classifies whether each patch intersects with the gripper. During this phase, the point-cloud encoder and collision decoder are trained jointly.
During policy training, discard the collision decoder and freeze the encoder, so that only the weights of the policy module are learned. The policy consumes the object point cloud embedding from the encoder, and other task-relevant inputs comprising robot's joint position and velocity, the
5

---

Published as a conference paper at ICLR 2024
previous action (end-effector subgoal and controller gains), the object's pose relative to the goal, and its physical properties (mass, friction, and restitution). These inputs are tokenized via an MLP layer to produce query tokens for the cross-attention layer against the point cloud embeddings to extract task-relevant features. These features are concatenated again with the task-relevant inputs and processed by a shared MI.P before being split into actor and critic networks. We train the policy module via PPO (Schulman et al., 2017). Since full-cloud observations and physics parameters of objects are not available in the real world, we distill the teacher policy into a student policy. We use DAgger (Ross et al., 2011) to imitate the teacher's actions solely based on information available in the real world: the partial point cloud of the object, robot's joint states, the previous action, and the relative object pose from the goal. We utilize nvdiffrast (Laine et al., 2020) to efficiently render the partial point cloud of the scene on the GPU during training.
3.2 LEARNING CORN
Algorithm outlines the data generation proce- dure for our network, inspired by (Son & Kim, 2023). We first sample the SE(3) transforms within the workspace for the gripper and ob- ject (L1-2). We then compute the nearest dis- placement between the surface points of the ob- ject and the gripper (L3) and move the gripper in that direction plus a small Gaussian noise (L4-6) to approach a configuration near or in collision. To compute the labels, we sam- ple points from the object's 's surface (L7), then whether they in-
Algorithm Dataset generation for CORN.
Require:
object shape O, gripper shape G. workspace W, noise level a Ensure: gripper pose Tc, object cloud X, con- tact label I.
1: To, To +- SumplePoses(M)
2: O' t To O; To
3: 8 + Nearest Displaacecllcccee((.......
4: o
5: Ta laxa -s- 5
label each point according tersect with the gripper (L8-10). This process generates about half of the gripper poses in col- lision with the object, and the other half in near-collision pose with the object. To disam- biguate these cases, the model must precisely reason about the object's geometry.
0
1
6: G" + G
7: ~ SampleSur CompleeSuffcce face face (o)
8: for x; X do
9: L, t PointInMesh(a,..)
end for
Afterward, the model is trained by classifying whether each patch of the object intersects with the gripper at the given pose. A patch is labeled as positive if any point within it intersects with the gripper. We use binary cross-entropy between the patchwise decoder logits and the collision labels to train the (Figure 4, green block)
3.3 PATCH-BASED ARCHITECTURE FOR CORN
To efficiently train policy by simultaneously simulating thousands of environments using GPUs, we leverage a patch-based transformer to encode the object point cloud shown in Figure 4 (red), which pass the patch embeddings the policy module (blue).
Patch Tokenization. In line with previous work (Pang et al., 2022), we divide the point cloud into set of patches (Figure 4, red block, top left). To define each patch, we identify N representative points using farthest-poin sampling (FPS). The k-nearest neighbor (kNN) points from these rep- resentatives constitute a patch, which is normalized by subtracting its center from its constituent points (Pang et al., 2022; Zeid et al., 2023; Yu et al., 2022) to embed the local shape. These patches are then tokenized via a Multi-Layer Perceptron (MIP) as input tokens to a transformer.
Unlike past methods that use small PointNet for tokenizing the patches (Pang et al., 2022; Zeid et al., 2023), we tokenize the patches with lightweight MLP. Using an MLP is sufficient since the patch size is constant, fully determined by the size of k during kNN lookup. By sorting the points within each patch by their distance from the patch center, we can ensure that points within a patch always follow a consistent order. Thus, we avoid using costly permutation ianvaration-invariant tokenizer. After computing the local patch embeddings, we add learnable positional embeddings of the patch centers to restore the global position information (Pang et al., 2022).
invariant PointNet

---

Published as a conference paper at ICLR 2024
Additional Encoder Inputs. By leveraging transformer backbone, architecture can easily mix heterogeneous task-related inputs as extra tokens, such as hand state, together with raw point-cloud observations as shown in Figure 4. Unlike PointNet-derived models that can only concatenate such inputs after the point-processing layers, this has an advantage in that the network can pay attention to parts of the point cloud in accordance with the hand state, such the ones near the robot's hand that would interact with the gripper.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Why
Figure 5: Set of 16 real-world objects that we test in the real world.
Training setup For each scene, we randomly draw an object from a subset of 323 geometrically diverse objects from DexGraspNet dataset (Wang et al., 2023), detailed in Section A.6.2. For each episode, we first place the object in random stable pose on the table. Then, we reset the robot arm at joint configuration uniformly sampled within predefined joint bounds slightly above the workspace. By doing so, we can avoid initializing the robot in spurious collision with the table or the object. Afterward, we sample a 6D stable goal pose randomly on the table, spaced at least 0. Im away from the initial pose to prevent immediately achieving success at initialization. To sample valid initial and goal poses for each object, pre-compute set of stable poses for each object. This procedure is elaborated in Section A.
Action, Rewards, and Success Criteria Following Kim et al. (2023), the action space of the robot comprises the subgoal residual of the hand, and the joint-space gains of a variable impedance con- troller (Details in Section A.2). We also follow the same curriculum learning scheme on the subgoal residual from Kim et al. (2023) to facilitate transferring our policy to the real world. The reward in domain is defined
T'uc + Treach + Teontoct Cenergy comprising the task success reward T Tauc- goal-reaching reward Treach, contact-inducing reward Teomtact. and energy penalty Cenergy- The task success reward Fauc L is given when the pose of the object within 0.05m and 0.1 radians of the target pose. We add dense rewards Treach and Fcontact to facilitate learning, based on potential function (Ng et al., 1999) and has the form r = 70(5') - ((s) where y E [o, 1) the discount factor. For Treach. we have &reach
k d.. and for Tcontact. we have (contact (s) = ke where kg. kid, k, E IR are scaling coefficients; do.g(s) is the distance from the current object pose to the goal pose, measured using the bounding-box based distance measure from Allshire et al. (2022); (s) is the hand-object distance between the object's center of mass and the tip of the gripper. Energy penalty term is defined as Cenergy ke ET Ti di, where k E IR is a scaling coefficient, and ds are joint torque and velocity of the ith joint. The scaling coefficients for the rewards are included in Table A.
Domain Randomization. During training, we randomize the mass, scale of the object, and friction and restitution of the object, table, and the robot gripper. We set the scale of the object by setting the largest diameter of the object within predefined range. To facilitate real-world transfer, also add a small noise to the torque command, object point cloud, and the goal pose when training the student. Detailed parameters for domain randomization described in Table A.4.
Real- World Setup. Our real-world setup is shown in Figure 3a. We use three RealSense D435 cameras extract the point clouds, and use Iterative Closest Point (ICP) To estimate the error between the object's current and target poses. We further use April tags (Wang & Olson, 2016) to track symmetric objects that cannot be readily tracked via ICP. We also wrap the gripper of the
7

---

Published as a conference paper at ICLR 2024
robot with high-ffrict frictffrittion glove match the simulation. We emphasize that all training is done in simulation, and the policy is zero-shot transferred to the real world.
4.2 RESULTS
We'd like to evaluate the following claims: (1) Pretraining CORN enables training a nonprehensile manipulation policy in time- and data-efficient manner. (2) Patch-based transformer architecture enables efficiently scaling to thousands of point-cloud based RL agents in parallel. (3) Our system can generalize to domain shifts such as unseen objects in the real world. To support claims I and 2, we compare our pretraining task and architecture (OURS) against the baseline encoders in Figure 6.
Compared to ours, P2V uses a pretrained Point2Vec (Zeid et al., 2023) encoder, using the author's weights; P2V-LITE is pretrained as in p2v, but uses the same architecture as OURS; ROT also shares the architecture with OURS, but is pretrained with relative rotation estimation as in (Huang et al., 2021: Chen et al., 2022); E2E also shares the architecture, but is trained in an end-to-end manner without pretraining: PN uses three-layer PointNet encoder, pretrained on point-wise collision pre- diction, analogous to CORN; and STATE trains geometry-unawan policy that only receives object bounding box inputs as the object representation, rather than point clouds. Note that P2v, P2V-LITE, ROT, E2E have the same downstream policy architectures as ours. For PN and STATE, they also have the same policy architecture except that they do not use cross-attention.
0.8
Baseline
Architecture
pretraining
OURS
PT
CORN
P2V
PT(arge)
p2v
Success
0.4
OURS
PN
PointNet
CORN-pt p2v
P2V-LITE
PT
ROT
PT
rot
0.2
E2B
PT
X
STATE
X
x
0.5
1.0
1.5
2.0
50
100
150
200
# Interactions
1e9
Timeth our)
Figure 6: Training progression and baselines. Plots show the mean (solid) and standard deviations (transparent) for each baseline. (Left) Success rate vs, number of interaction steps. Interaction steps are reported as the total number of steps aggregated across 4096 parallel environments. (Middle) Success rate vs. number of hours. All baselines are allowed to interact at most 2e9 time steps. (Right) Comparison of baselines. PT: Patch Transformer.
Figure shows the progression of success rates across different baselines in simulation. To support claim 1, we consider the performance of the policy after 2 billion interactions (in steps) and 24 hours (in time), respectively. First, E2E takes significantly longer compared to OURS, terms of both steps and time: by pretraining CORN, OURS achieves 88.0% suc.rate (steps) and 88.5% (time), while E2E only reaches 31.4% (in time) and 54.4% (in steps), respectively. This is due to the overhead of jointly training the encoder and policy networks in R2F. Among pretrained representations, we first compare with p2v. While P2V is similar to OURS in steps(85.6%% it is significantly slower in time(3.55%) due to the inference overhead of the large representation model. To isolate the effect of the pretraining task, we also compare with P2V-LITE: pretrained as in P2v, yet with same architec- ture as OURS. While the resulting policy is reasonably time-efficient due faster rollouts (73.4%, in time), it is less data-efficient model (Shi al., 2022: Fang et al.. 2021). Another baseline, ROT from in-hand reorientation litera- ture (Huang et al., 2021: Chen et al., 2022) performs slightly worse in both measures (64.2%% (steps); 67.0% (time)): we believe this due to the increased demand for the awareness of object geometry in our task. To see this, we note that the pose-only Policy(STATE). unaware of object geometry, cannot learn at all in our domain, in contrast to the positive result from Chen et al. (2022). This highlights the fact that our task requires greater awareness of the object's geometric features: just knowing the object's pose is insufficient, and a deeper understanding of its geometry is necessary.
(68.8% in steps) due to the reduced performance of the smaller
To see CORN learns suitable representation for manipulation, we inspect the attention scores of the policy amid different motions in Figure 7. For pushing and toppling motions (row 1,3), our policy primarily attends to the current or future contact between the hand and the object, which is consistent
8

---

Published as a conference paper at ICLR 2024
100
0.75
Roll
-0.5
Topple
0.25
0.0
Figure 7: Visualization of the scores from the cross-attentin layers of the policy, summed over all heads. We colorize the attention for each patch normalized to range (0, 1), then project the patch- wise colors the surface of the object for visualization with VIRIDIS colormap.
with the premise of our eppresentation-laamiig scheme. While pivoting or rolling motions (row 2,4) share the same trend, policy also attends the pivot points of the object against the table.
To support claim 2, we show that our patch-transte tranfrrranfoomme architecture more performant in both compute- and memory-cost compared to PointNet (Qi et al., 2016). In particular, evaluate PN, where the primary difference is the use of PointNet in place the patch transformer. While the performance degradation from the architecture change only modest in steps (88.3% 70.2%), the large computational overhead of PointNet renders it slowest in time(0.223%) among all baselines. To support claim 3, we evaluate our policy in the real-world setting as described Section 4.1. Figure shows the objects that we evaluate in our real-world experiments, exhibiting wide diversity terms of shape, mass, and materials; Figure shows four representative real-world episodes.
Table 1: Real world run results. All of the objects in the real-world setup are unseen except for 3D 3D-printed objects marked with f.
Object Success/ Irial
Object name
Success/Tra
Object name
Success/
Object name
Success/ /rial
Red Card
4/5
Cube Block
475
Ceramic Cup
425
Red Dripper
375
Toy Puppyt
Iced
Nutella Bottle
Holder
Toy Bulldozer
4/5
Wipe Dispenser
5/5
Coffee Jar
2/5
Thick Book
4/5
Blue
Toy Tabler
Lemona
Candy
Success rale
$7780 (71.3%
The results from our policy execution in the real-world organized in Table 1. Overall, our policy demonstrates 71.3% success rate in the real world. These results indicate that, with distillation, our system can zero-shot transfer to the real world despite only training in a simulation.
Limitations. We find that our policy struggles more with concave objects holdserr-hoddr, red drip- per; 60%) or unstable objects (iced box, coffee jar, red dripper; 53%). Concave objects cause perception challenge in the real world due to severe ambiguity when occluded by the robot, leading to timeouts as the agent oscillates between actions or stops amid a maneuver. Unstable objects tend to abruptly spin out of the workspace after minor collisions, and the dexterity of our robot platform and speed our real-worle perception pipeline cannot catch up with such rapid motions.
CONCLUSION
In this work, we present a system for effectively training a non-prehensile object manipulation policy that generalize over diverse object shapes. We show that by pretraining contact-informed representation of the object, coupled with an efficient pateh- transformer architecture, we can train the policy in data- and time-efficient manner. With student-teacher distillation, find that our policy can zero-shot transfer to the real world, achieving success rates 71.3% in the real world across both seen and unseen objects despite only training in simulation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (MVT)) (No.2019-0-00075, Artificial In- telligence Graduate School Program(KAISTT) (No.2022-0-00311, Development of Goal-Oriented Reinforeement Learning Techniques for Contact-Rich Robotic Manipulation Everyday Objects), (No. 2022-0-0066 12, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AT).
REPRODUCIBILITY STATEMENT
We describe our real-world system and perception pipeline in detail in Section A.4. The procedure for data generation and pretraining for model and other basclines is detailed in Section 3,2 and Section A.3, respectively. We include the set of 323 training objects from DexGraspNet (Wang al., 2023) that we used for policy training supplementary materials. We include the student- model architecture and distillation pipeline for real-world transfer in Section A.1. We describe the simulation setup regarding data preprocessing(Section A.6), and describe the hyperparameters associated with our network architecture A.2), domainnraandommizaiinnohhhb A.4), and policy training(Section A.3). We also release code at https: / thub. com/ cont/ct -non-nttttoo-nnn- -no--p rehensile/con corn.
REFERENCES
Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, and Animesh Garg. Transferring dex- terous manipulation from gpu simulation remote real-world trifinger. In 2022 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 18822-11809, IEEE, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312. PMLR, 2022.
Andrea Banino, Adria Puigdomenech Badia, Jacob C Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In International Confer- ence on Learning Representations, 2022.
Miroslav Bogdanovic, Majid Khadiv, and Ludovic Righetti. Learning variable impedance control for contact sensitive tasks. IEEE Robotics and Automation Letters, 5(4):6129-6136, 2020.
Samuel Buss. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE Journal of Robotics and Automation, 17(1-19):16, 2004.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rict 3D Model Repository. Technical Report arXiv:1512.03011 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211.11744, 2022.
Xianyi Cheng, Eric Huang, Yifan Hou, and Matthew T Mason. Contact Mode Guided Motion Plan- ning for Quasidynamic Dexterous Manipulation in 3D. In International Conference on Robotics and Automation, 2022.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using pacoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv:1810.04480 011.8880000000,, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learn- ing as goal-conditioned reinforcement learning. Advances in Neural Information Processing Sys- fems, 04.35603-356221 2022.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In ICLR, 2021.
Juan Del Aguila Ferrandis, João Moura, and Sethu Vijayakumar. Nonprehensile planar manipu- lation through reinforcement learning with multimodal categorical exploration. arXiv preprint arXiv:2308.00 02459, 2023.
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Yifan Hou and Matthew T. Mason. Robust execution of contact-rich motion plans by hybrid force- velocity control. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, may 2019. doi: 10. 1099/caa.019..8794888 URL https: /tt.///o.i.oo org.lo..r111.. wttp:ooo////......1....... 1109#2Ficra ..211..88 8794366.
Jingwei Huang, Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint arXiv:1802.01698, 2018.
Wenlong Huang, Igor Mordatch, Pieter Abbeel, and Deepak Pathak. Generalization in dexterous manipulation via geometry-awan multi-task learning. arXiv preprint arXiv:2111.03006 2021.
Imin Kao, Kevin M. Lynch, and Joel W. Burdick. Contact Modeling and Manipulation. In Springer Handbook of Robotics, 2016.
Minchan Kim, Junhyek Han, Jaehyung Kim, and Beomjoon Kim. Pre-and post-contact policy de- composition for non-prehensile manipulation with zero-shot sim-to-real transfer. arXiv preprint arXiv:2309.02754, 2023.
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for pigh-performance differentiable rendering, ACM Transactions on Graphics (TOG), 39(6):1-14, 2020.
Jacky Liang, Xianyi Cheng, and Oliver Kroemer. Learning Preconditions of Hybrid Force-Velocity Controllers for Contact-Rich Manipulation. In Conference on Robot Learning, 2022.
Kendall Lowrey, Svetoslav Kolev, Jeremy Dao. Aravind Rajeswaran, and Emanuel Todorov. Rein- forcement Learning for Non-Prehen Prehenneee Manipulation: Transfer from Simulation to Physical Sys- tem. In International Conference Simulation, Modeling, and Programming for Autonomous Robots, 2018
Yusuke Maeda and Tamio Arai. Planning of graspless manipulation by multifingered robot hand. Advanced Robotics, 1955)5011552
2005. 2005.
Yusuke Maeda, Hirokazu Kijimoto, Yasumichi Aiyama, and Tamio Arai. Planning of graspless ma- nipulation by multiple robot fingers. In Proceedings 2001 ICRA. IEEE International Conference Robotics and Automation (Cat. No. 01CH37164), volume 3, pp. 2474-2479. IEEE, 2001.

---

Published as a conference paper at ICLR 2024
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Apari- cio, and Ken Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts, July 2017. doi: 10. 15607//SS..0117.IIII
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint 1XXi:22108.10700 2021.
Kiyokazu Miyazawa, Yusuke Maeda, and Tamio Arai. Planning of graspless manipulation based on rapidly-e random trees. In (ISATP 2005). The 6th IEEE International Symposium on Assembly and Task Planning: From Nano to Macro Assembly and Manufacturing. 2005., PP-
7-12. IEEE, 2005.
Igor Mordatch, Zoran Popović, and Emanuel Todorov. Contact-Invariant Optimization for Hand Manipulation. In ACM SIGGRAPI/ELmograpicc Symposium Computer Animation, 2012.
João Moura, Theodoros Stouraitis, and Sethu Vijayakumar. Non-Prehensile Planar Manipulation via Trajectory Optimization with Complementarity Constraints. In International Conference on Robotics and Automation, 2022.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278-287. Citeseer, 1999.
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked au- toencoders for point cloud self-supervised learning. In European conference on computer vision, pp. 604-221. Springer, 2022.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. In International Conference on Robotics and Automation, 2018.
Michael Posa, Cecilia Cantu, and Russ Tedrake. A Direct Method for Trajectory Optimization of Rigid Bodies Through Contact. The International Journal of Robotics Research, 2014.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1677077707 2017.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In Inter- national Conference on Machine Learning. PMLR, 2021.
Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, and Yueting Zhuang. On the efficacy of small self-supervised contrastive models without distillation signals. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Dongwon Son and Beomjoon Kim. Local object crop collision network for efficient simulation of non-convex objects in GPU-based simulators. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi:
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidex- grasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iter- ative generalist-speeialiss learning. arXiv preprint arXiv:2304. 00464, 2023.
12

---

Published as a conference paper at ICLR 2024
Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matthew J. Kusner. Unsupervised point cloud pre-training via occlusion completion. In International Conference on Computer Vision, ICCV, 2021.
John Wang and Edwin Olson. AprilTag 2: Efficient and robust fiducial detection, In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), October 2016.
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dex- graspnet: A large-scale robotic dexterous grasp dataset for general objects based simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11359-11366. IEEE, 2023,
Xinyue Wei, Minghua Liu, Zhan Ling, and Hao Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG), 41(4):1-18, 2022.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Rision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt III 16, pp- 574-591. Springer, 2020.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Weihao Yuan, Johannes A Stork, Danica Kragic, Michael Y Wang, and Kaiyu Hang. Rearrange- ment with Nonprehensile Manipulation Using Deep Reinforcement Learning. In International Conference Robotics and Automation, 2018.
Weihao Yuan, Kaiyu Hang. Danica Kragic, Michael Y Wang, and Johannes Stork. End-to- End Nonprehensile Rearrangement with Deep Reinforcement Learning and Simulation-to-Realli Transfer. Robotics and Autonomous Systems, 2019.
Karim Abou Zeid, Jonas Schult, Alexander Hermans, and Bastian Leibe. Point2vec for self- supervised representation learning on point clouds. arXiv preprint arXiv:2303.16570, 2023.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang. Bin Zhao, Dong Wang, Yu Qiao, and Hong- sheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems, 35:27061-27074, 2022.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021.
Wenxuan Zhou and David Held. Learning to grasp the ungraspable with emergent extrinsic dexterity. In Conference on Robot Learning. pp. 150-160. PMLR, 2023.
Wenxuan Zhou, Bowen Jiang, Fan Yang, Chris Paxton, and David Held. Hacman: Learning hybrid actor-critic maps for 6d non-prehensile manipulation. In Conference on Robot Learning. PMLR, 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
Claudio Zito, Rustam Stolkin, Marek Kopicki, and Jeremy Wyatt. Two-Level RRT Planning for Robotic Push Manipulation. In International Conference Intelligent Robots and Systems, 2012.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 STUDENT MODEL
Trans former
Panchily
Hand State
DAgger
teacher antions
Joint
Encoder
Prev Action
Rell Pose Goal
Learned Querr Taleen
Attention xQ
Concat
Policy (cauu
actions
Student Policy
Figure A.1: Student Model Architecture in DAgger.
The student policy shares similar architecture with the teacher policy, with two distinctions. Unlike teacher policy, which employs joint state, previous action, and goal inputs for generating query tokens for cross-attention, the encoder to facilitate extracting task-relevant features from partial point clouds. Given that these inputs are already incorporated in the input of the encoder, we use learned query tokens for cross- attention.
incorporate these tokens into the input of the patch transformer within
Since the student policy can only perceive limited shape information of objects at any given moment due to occlusion, we incorporate a two-layer Gated Recurrent Unit (GRU) (Cho et al., 2014) on top of the student point cloud encoder. This enables the student model to aggregate partial observations of object geometry based on historical observations.
We train the student with DAgger with truncated backpropagation through time (BPTT), In par- ticular, we run the student for 8 time-steps and update its parameters minimize KL-divergence between the action distribution the teacher policy and the student policy.
A.2 DETAILS ON POLICY ACTION SPACE
The action space of our policy consists of the subgoal residual of the end-effector AT, and joint- space gains. The subgoal residual is parameterized by the translational offset At E R and axis- angle representation of the residual rotation in the world frame Ar R3, The joint-space gains parameterized by proportional terms (kp- RT) and damping factors (p E IR) of variable impedance controller. Based on this, we compute the joint position target by solving inverse kine- matics with damped least squares method (Buss, 2004) Starget qi K ((AT..). The torque command for each joint is computed based on the joint position error and the predicted joint-space gains and damping factors as kip (ytarget - 4t) kaqt, kid = p Vkp.
A.3 DATA GENERATION AND PRETRAINING PIPELINES
P2V-LITE. For pretraining P2V-LITE, we follow the same procedure as Zeid et al. (2023), except use the same architecture as OURS. Due to the use of significantly lighter model, we apply two modifications to the original training setup, as follows: first, since we only use a two-layer transformer, instead of averaging over the last 6 layers of the teacher, take the result after layer- normalization of the final layer, which keeps the ratio consistent (50% of layers). Second, we replace DropPath layers with dropout since the model is shallow. Otherwise, follow the original hyper-
14

---

Published as a conference paper at ICLR 2024
parameters from the paper and the author's colchh::pp://giittab..omm// nt2vec).
ROT. For pretraining ROT, follow the procedure described in Chen et al. (2022). Since the authors report that adding the classification objective does not influence the encoder quality, we omit the classification terms. For the training data, We apply two different SO(3) transformations to the same point cloud, and pass it through the encoder obtaín the patchwise embeddings. We concatenate the patchwise embeddings from both streams, then pass it through a 3-layer MLP (512, 64, 6). We train the encoder on the regression task, where the task for the decoder is to regress the parameters for a 6D rotation Zhou et al. (2019) as recommended from Chen et al. (2022).
ShapeNet Chang et al. (2015) dataset.
OURS-PN. For pretraining PN, we use the same dataset from OURS. Since PointNet does not pro- duce patchwise embeddings, we instead adopt U-Net architecture based on PyG Fey & Lenssen (2019). To inform the model about the end-effector, we concatenate the hand pose (position and orientation Zhou et al. (2019)) each point in the channel dimension.
A.4 REAL WORLD SYSTEM AND PERCEPTION PIPELINE
Point Cloud Segmentation. In the point cloud segmentation phase, we integrate three RealSense D435 cameras. Points outside the predefined 3D bounding-box workspace above table are first re- moved, and those within a distance less than the threshold = 1 cm from the table surface are removed next. These parameters were determined empirically to handle outliers without removing too much of the object. For thin objects such as cards, we alternatively employ color-based segmen- tation to remove the table from the point cloud, avoiding the risk of inadvertently removing object points.
After removing the table, only the object and the robot remains. To remove the robot, we compute the transform of all robot links based on the robot kinematics, then query the points as to whether it is contained within each link mesh. For efficiency, we first apply convex decomposition on the robot geometry, and employ point-convex intersection to check whether the point belongs to the robot. We initially tried using robot-points removal based on raycasting the point-to-robot mesh distances, but we found it was much slower than the convex-hull based method.
To reduce noise in the point cloud, we employ a radius outlier removal logic with r = 2 cm and Amin 96 points, which is the minimum number of points that the inlier should contain within the neighborhood. Finally, to identify the object point cluster, we used DBSCAN clustering algorithm. Clusters were computed with e=lmm and the number of neighbors 4 to form the core-set.
To enhance processing speed, we optimized the entire segmentation pipeline to utilize tensor oper- ations executed on the GPU. The output object point cloud of this segmentation process serves as input for both our policy and object pose tracking pipeline.
Object Pose Tracking. To trade-off tracking speed and performance, we first subsample the object point clouds randomly to Purck = 2048 points. Empirically. we found that this avoids compromising the quality of the ICP(Iterative Closest Point) algorithm, while remaining reasonably fast.
We employ ICP for tracking. Given the object point cloud C, at current time step t, we conduct point-to-plane ICP registrations with point cloud of previous timestep Ct-1, resulting in a transfor- mation Pt,--1 = ICP(C,, C.-1). The current object pose To, is obtained by recursively applying pairwise transformations: To, = Ptt--1 To, 1. the case that the transform drifts over time, we also perform an additional ICP match with the initial object point cloud Pr. ICP(C,, Co) to correct the error. If the fitness score Fitness exceeds 60%, current object pose is calculated as To. Pr. To..
For thin objects, we utilize point-to-poin ICP matching since their point cloud nearly forms a single plane. While the ICP tracker performs well for large and non-symmetrical objects, it struggles with highly symmetrical objects, or objects with point-matcin matching ambiguity under occlusion (e.g. when cup handles occluded by the robot).
To address this challenge, we tried based ICP to resolve the ambiguity based on surface tex- ture. However, found that it did not improve tracking performance and, in some cases,
15

---

Published as a conference paper at ICLR 2024
worsened it. As such, used april-tag -tag tracker to maintain object pose tracking accuracy for highly symmetrical objects.
A.5 EXAMPLE TRAJECTORIES
Example trajectories can be observed from the video in the supplementary material, from http s: st//sites... https/tttssooggle...e.eeoo
A.6
SIMULATION SETUP
A.6.1 STABLE POSES GENERATION
To sample stable poses for training, drop the objects in simulation and extract the poses after stabilization. In 80% of the trials, we drop the object 0.2m above the table in a uniformly random orientation. the remaining 20%, we drop the objects from their canonical orientations, to increase the chance of landing in standing poses, as they are less likely to occur naturally. If the object remains stationary for 10 consecutive timesteps, and its center of mass projects within the support polygon of the object, then consider the pose be stable. We repeat this process to collect least 128 initial candidates for each object, then keep the unique orientations by pruning equivalent orientations that only differ by planar rotation about the z-axis.
A.6.2 OBJECTS FOR TRAINING
We sample 323 objects from the DexGraspNett dataset (Wang et al., 2023), shown in Figure A.2. To increase the simulation speed and stability, we apply convex decomposition on every mesh using CoACD (Wei et al., 2022), after generating the watertight mesh with simplification using Mani- fold (Huang et al., 2018).
yiu
E
Figure A.2: 323 object meshes that were used for training.
16

---

Published as a conference paper at ICLR 2024
A.7 HYPERPARAMETERS
Table A.I: Hyperparmeters for reward computation.
Parameter
Value
Description
kg 0.302
Goal-reaching reward cod coefficient
k 0.0604
Hand-reaching reward cuefficient
Re 0.0001
Energy penalty coefficient
kg
243.12
Decuy factor for potential-ban -basd reward
Table A.2: Hyperparameters for Encoder and Policy.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
Shared
Num. points
512
Hidden dim.
128
MLP (512, 256, 128)
ML.P
Num. patches
16
Num. layer Num. attention head in self aelf-attention
Actor
MLP (64. 1)
Patch size
32
Critic
MLP(64, 20)
Num. attention head
Concatenate context input
True
cross-attention (teacher/studentt
16/4
Table A.3: Hyperparameters for PPO.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
Max Num. epoch
8
Base learning
0,0003
GAE parameter
0.95
Num, environment
4096
Earlystopppngg KL target
0.024
Adaptive-LR KL target
0.016
Discount factor
0.99
Episode length
300
Entropy regularization
0
Learning rale schedule
adaptive
PPO clip range
0.3
Update frequency
8
Policy loss coeff. 2
Value loss coeff.
0.5
Bound loss coeff.
0.02
Table A.4: Range for domain randomization. U U(min, max] denotes uniform distribution, and N Ju, o denotes Normal distribution. Point cloud noise and Goal pose noise are added to the nor- malized input.
Parameter
Range
Object mass (kg)
0.0.
Object scale (m)
0.0.1,0.3
Object friction
1.0.7,
Table friction
4/0.3, 0.8
Gripper friction
2[1.0, 1.5]
Torque noise
N.O.0, 0.03)
Point cloud noise
A/ 0.005
Goal pose noise
0.005
17


Reviews:
[{"review_id": "lVkXfCjclz", "reviewer": "Reviewer_4HBK", "summary": "This paper presents a novel method for non-prehensile manipulation, i.e. manipulating an object that is not graspable via poking, pivoting and toppling from an initial pose to a goal pose. There are two major technical contributions:\n- A novel pre-training objective on predicting which parts of the object point cloud are in contact with the gripper\n- A novel patch-based transformer architecture that allows efficient encoding of point clouds and other modalities such as robot gripper state.\n\nThe paper also provides a new dataset on non-prehensile manipulation with over 300 different objects.", "questions": "- What is the coordinate frame of the hand pose input? Since the hand pose is sampled near the object to ensure good data balance for contact/no contact during pre-training, will the hand pose go out of distribution during policy learning?\n- In Figure 7, is the scale of the success rate the same for the two plots? It seems that the final success rate doesn’t match up for the green, red, purple and brown curves.\n- Which simulator is used? I didn’t find it in the paper.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The model is trained entirely in simulation and achieves over 70% success rate in the real world with zero-shot sim-to-real transfer.\n- The model finishes 2 million steps training in less than a day.\n- The objects tested have good diversity. Over 20 objects are tested in the real world and over 300 objects are used in simulation. The test objects have significant geometric difference from the training objects.\n- The paper addresses an important and difficult problem in contact-rich manipulation, which has significant impact on expanding a robot’s manipulation ability to more diverse objects in the real world.\n- The experiments are very comprehensive, covering all the key parts of the design, including the point encoder and the pre-training objective.\n- The hyperparameters are well documented in the appendix, which is important for reproducing the results.", "weaknesses": "- The model takes the difference between current object pose and target object pose as input. This can bring significant engineering challenge since object segmentation and pose tracking in the real world can be difficult. However, the authors documented their approach very well in Sec. A.3.\n- The method only works on object well separated from clutter on a tabletop. This is related to the above assumption for the object pose, since having more than one object in close contact will make object segmentation and pose tracking even more challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nABov1Sp1e", "reviewer": "Reviewer_LafH", "summary": "The paper introduces CORN, a Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects. The system combines deep reinforcement learning with a novel contact-based object representation to effectively manipulate a variety of shapes and sizes of objects. The key innovation lies in the use of a lightweight patch-based Transformer architecture to process point clouds, enabling large-scale parallel training across thousands of environments. The efficacy of CORN is validated through a series of experiments, demonstrating zero-shot transferability to novel real-world objects.", "questions": "- The paper strongly emphasizes the role of \"contact\" in nonprehensile manipulation. How does this emphasis differ in importance from its role in grasping tasks?\n- The description of the policy network is relatively concise. Could the authors provide more details on its design and working principles?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces an approach to nonprehensile manipulation of general unseen objects, a challenging area in robotics. \n- The methodology is laid out with a clear structure, and the paper provides a set of experimental results to back its proposals. \n- The work has potential implications for the field of robotics, particularly in object manipulation, although the full extent of its impact may require further exploration and validation.", "weaknesses": "- Lack of Unique Design for Complex Operations: The paper emphasizes in the introduction that its approach can execute more complex robotic arm operations than prior grasping work. However, in the method description, there appears to be no clear unique design to directly support this motivation, raising doubts about the novelty and effectiveness of the method.\n- Overemphasis on \"Contact\": While the authors place particular emphasis on the importance of \"contact\" in nonprehensile operations, it seems that the equally vital role of \"contact\" in grasping tasks has not been adequately considered. This imbalanced emphasis may impact the comprehensiveness of the method and its practical applicability.\n- Questioning the Novelty of Point Cloud Processing: The paper utilizes a Transformer-based architecture to process point cloud data, but this approach does not appear groundbreaking, emphasizing the need for a more detailed description of the policy network design.\n- Insufficient Description of the Policy Network: The paper provides a relatively concise description of the policy network, lacking in-depth details, which might hinder readers from fully understanding the workings of the method and its potential advantages.\n- Inadequate Experimental Setup: The paper does not offer detailed information related to actions and reward settings in the experiments, which is crucial for evaluating the effectiveness of the reinforcement learning portion.\nIn summary, these points highlight potential shortcomings in the paper concerning method description, evidence of novelty, and experimental setup. They offer specific directions for improvement to the authors, aiming to enhance the persuasiveness of the paper and provide a clearer conveyance of their research outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8GkrLGc73K", "reviewer": "Reviewer_dbmK", "summary": "The paper presents a novel method for nonprehensile manipulation using reinforcement learning (RL). Traditional RL struggles with diverse object shapes and high-dimensional sensory inputs. The authors introduce a contact-based object representation and pretraining pipeline, using a patch-based transformer encoder for point clouds, enabling scalable training. Their approach offers time- and data-efficient learning, with successful zero-shot transfers to real-world objects.", "questions": "See the weakness section above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper motivates well, and the techniques presented are sound.\n2. The paper is well-written and easy to follow.\n3. The presented method shows good generalization ability to unseen objects and sim-to-real scenarios.", "weaknesses": "1. The novelty of this work is marginal. basically, it just applies point-cloud-based reinforcement learning to nonprehensile manipulation tasks.\n\n2. The generation of the collision label is questionable with the coverage of the collision states, i.e., if this work truly aims for generalization on unseen objects, I wonder how the collision prediction network be generalized to the unseen geometry.\n\n3. Besides, if the contact network is trained as a guide for the encoder, how does it guarantee to generate a collision-free policy? The collision decoder itself is not perfect (due to the coverage issue and neural network prediction), and the influence on the encoder is indirect (leads to less perfect), not to mention the policy is distilled from the teacher network to the student network (even less perfect).\n\n4. Since this method is \"contact-based\"? In a broad sense, all the manipulation is contact-based, but I assume the word choice here is for the embedding induced by the contact network. Then, though the authors have conducted extensive baseline methods on the encoder side, in my opinion, they should also discuss different decoder schemes to justify the \"contact-based\" name. For example:\n\n(a). the encoder is directly trained by the policy network? That is, no contact decoder to pretrain the encoder.\n\n(b). the decoder is a geometry reconstruction network. In this sense, the encoder can capture the geometry information of the object point cloud and hand. I see no apparent reason why the \"action and rewards\" cannot be done with such kinds of decoders.\n\n5. Another possibility of the naming is due to the r_contact in the reward, but the contact potential is not given exactly, since the essential d_h,o is built upon the distance from the CoM of the object to the tip. Which does not prevent collision/intersection between the gripper and hand.\n\nTherefore, if the contact is not necessarily determinant or explicitly modeled for policy training in both network design and reward shaping, the only sane way to interpret the contact-based would be in the broadest sense, i.e., all the manipulation is contact-based. In this sense, I would suggest removing the \"contact-based\" in the title, which would give wrong expectations to the readers. Or the paper can present stronger relevance of \"contact\" to the object representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S1QjVu1qVk", "reviewer": "Reviewer_UC4D", "summary": "This paper presents an innovative contact-based representation for non-prehensile manipulators, which aims to enhance the robot's ability to manipulate objects. A pre-training model is utilized to predict the contact between the gripper and the object, thereby providing the policy with a more detailed understanding of the robot-object interaction. The state-based policy is then distilled into a vision-based one for implementation in real-world scenarios.", "questions": "1. Could you elaborate on the \"success rate\" metric used in your experiments? What specific criteria are used to determine a successful manipulation?\n2. Can you explain how do you measure the physical parameters in the real world and make sure they are aligned or well-simulated in simulations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The creation of this representation is commendable, as it underscores the importance of using the full functionality of the gripper for contact. The algorithm designed for its training is intriguing and appears to be well thought out.\n2. The experimental results convincingly demonstrate the model's capability, efficiency, and superiority compared to alternative methods. Furthermore, experiments using real robots with zero-shot transfer highlight the model's robustness, thereby solidifying the research.", "weaknesses": "1. The scope of manipulation tasks in this research is restricted to single-object state maneuvering using a closed gripper. Incorporating more intricate robot-object interactions, such as grasping, could help to fully utilize the robot's kinematic capabilities and cover more complex scenarios.\n2. The paper measures the success of the tasks using a \"success rate.\" However, necessary details, like the specific criteria used to determine successful manipulation, lacks clarity. Moreover, the goal state illustrated in the manipulation videos (or their screenshots) appears to be a snapshot of a future state, which does not accurately represent the actual desired outcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)