HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation
Authors: Wenxuan Zhou
Bowen Jiang
Fan Yang
Chris Paxton
David Held
Venue: CoRL 2023 Oral (2023)
Keywords: Action Representation
Reinforcement Learning with 3D Vision
Non-prehensile Manipulation

Abstract: Manipulating objects without grasping them is an essential component of human dexterity, referred to as non-prehensile manipulation. Non-prehensile manipulation may enable more complex interactions with the objects, but also presents challenges in reasoning about gripper-object interactions. In this work, we introduce Hybrid Actor-Critic Maps for Manipulation (HACMan), a reinforcement learning approach for 6D non-prehensile manipulation of objects using point cloud observations. HACMan proposes a temporally-abstracted and spatially-grounded object-centric action representation that consists of selecting a contact location from the object point cloud and a set of motion parameters describing how the robot will move after making contact. We modify an existing off-policy RL algorithm to learn in this hybrid discrete-continuous action representation. We evaluate HACMan on a 6D object pose alignment task in both simulation and in the real world. On the hardest version of our task, with randomized initial poses, randomized 6D goals, and diverse object categories, our policy demonstrates strong generalization to unseen object categories without a performance drop, achieving an 89% success rate on unseen objects in simulation and 50% success rate with zero-shot transfer in the real world. Compared to alternative action representations, HACMan achieves a success rate more than three times higher than the best baseline. With zero-shot sim2real transfer, our policy can successfully manipulate unseen objects in the real world for challenging non-planar goals, using dynamic and contact-rich non-prehensile skills. Videos can be found on the project website: https://hacman-2023.github.io.

HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation
Wenxuan Zhou Bowen Jiang', Fan Yang', Chris Paxton David Held
Manipulating objects without grasping them is essential component of human dexterity, referred to as non-prehensile manipulation. Non-prehensile manipulation may enable more complex interactions with the objects. but also presents challenges in reasoning about gripper-object interactions. In this work, we introduce Hybrid Actor- Croo-C Maps for Manipulation (HACMan), reinforce- ment learning approach for 6D non-prehensile manipulation of objects using point cloud observations. HACMan proposes a comporally-abstraat grounded object-centric action representation that consists of selecting a contact location from the object point cloud and a set of motion parameters describing how the robot will move after making contact. We modify an existing off off-policy RL al- gorithm to learn this hybrid discrete-continuous action representation. We eval- uate HACMan on a 6D object pose alignment task in both simulation and in the real world. On the hardest version of our task, with randomized initial poses, ran- domized 6D goals, and diverse object categories, our policy demonstrates strong generalization unseen object categories without a performance drop, achieving 89% success rate on unseen objects in simulation and 50% success rate with zero-shot transfer in the real world. Compared to alternative action representa- tions, HACMan achieves success rate more than three times higher than the best baseline. With zero-shot sim2real transfer, our policy can successfully manipulate unseen objects in the real world for challenging non-planat goals, using dynamic and contact-rich non prehensile skills. Videos can be found on the project website: https: http://acmaaaa22222 wtt..0itttu....e. github.io.
Abstract:
and spatially-
Keywords: Action Representation, Reinforcement Learning with 3D Vision, Non-prehensile Manipulation
Introduction
The ability to manipulate objects in ways beyond grasping a critical aspect of human dexterity. Non-prehensile manipulation, such pushing, flipping, toppling, and sliding objects, is essential for a wide variety tasks where objects are difficult to grasp or where workspaces are cluttered or confined. However, non-prehensile manipulation remains challenging for robots; previous work has only shown results with limited object generalization [1, 2] or limited motion complexity, such planar pushing or manipulating articulated objects with limited degrees of freedom [3, 4, 5]. We propose a method that generalizes across object geometries while showing versatile interactions for complex noe-peeee manipulation tasks.
We present Hybrid Actor-Critic Maps for Manipulation (HACMan), reinforcement learning (RL) approach for non-prehensile manipulation from point cloud observations. 'he first technical con- tribution of HACMan is to propose an object-centric action representation that is temporally- abstracted and spatially-grounde eters determining the trajectory should take after making contact. The contact location is selected from the observed object point cloud which provides spatial grounding. At the same time, the robot decisions become more semporally-abstracee because we focus on only learning the contact-rich portions of the action.
The agent selects a contact location and set of motion param-
The second technical contribution HACMan is to incorporate the proposed action representation in an actor-critic RL framework. Since the contact location is defined over discrete action space (selecting a contact point among the points in the object point cloud) and the motion parameters
Robotics Institute, Carnegie Mellon University Meta 'Equal Advising
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Target Object Pose
Figure I: We propose HACMan (Hybrid Actor-Critic Maps for Manipulation), which allows non- prehensile manipulation of unseen objects into arbitrary stable poses. With HACMan, the robot learns to push, tilt, and flip the object to reach the target pose, which is shown in the first column and in the top row with transparency. The policy allows for dynamic object motions with complex contact events in both simulation (top) and in the real world (bottom). The performance of the policy best understood from the vidcos on the website: https: /hhcman 2023. gi hub. io.
(defining the trajectory after contact) are defined over a continuous action space, our action repre- sentation is in hybrid discrete-continun action space. In HACMan, the actor network outputs per-point continuous motion parameters and the critic network predicts per-point Q-values over the object point cloud. Different from common continuous action space RL algorithms [6, 7]. the per- point Q-values are used both to update the actor and also compute the probability for selecting the contact location. We modify the update rule of an existing off- off-policy RL algorithm to incorporate such a hybrid action space.
We apply HACMan to 6D object pose alignment task with randomized initial object poses, ran- domized 6D goal poses, and diverse object geometries (Fig. 1). In simulation, our policy generalizes to unseen objects without a performance drop, obtaining an 89% success rate on unseen objects. In addition, HACMan achieves training success rate more than three times higher than the best base- line with an alternative action representation. We also perform real robot experiments with zero-shot sim2real transfer, in which the learned policy performs dynamic object interactions over unseen ob- jects of diverse shapes with non non-planar goals. Our contributions include:
We propose novel object-centric action representation based on 3D spatial action maps learn complex non prn-prehensile interactions. We also modify an existing off off-policy RL algorithm incorporate such hybrid discrete-continuogg action space.
The proposed action representation demonstrates substantive improvements of perfor- mance over the baselines and shows strong generalization to unseen objects.
The learned policy showcases complex contact-rich and dynamic manipulation skills, in- cluding pushing, tilting, and flipping, shown both in simulation and with a real robot.
2 Related Work
Non-prehensile manipulation. Non- Non-prehensile manipulation is defined manipulating objects without grasping them [8]. Many non-prehensile manipulation tasks involve complex contact events among the robot, object, and the environment, which lead to significant challenges in state- estimation, planning and control [1, 2, 9, 5]. Recent work has applied learning-based methods in non-pree noe-eneheesile manipulation, but they are limited in terms of either skill complexity [10, 3, 4, 11] or object generalization [12, 11]. In contrast, our work shows 6D object manipulation involving more complex object interactions while also generalizing to large variety of unseen object geometries.
Visual Reinforcement Learning with Point Clouds. Recent research has explored various ways of incorporating point clouds into RL [13, 14]. To overcome the optimization difficulties, previous work has tried pre- tra-train the feature extractor with an auxiliary loss [15), initializing the RL policy with behavior cloning [16), or using student-teacher training [17, 18] (see detailed discussion in Appendix F). Our method does not require these additional training procedures due to the benefits the experiments, we show that the baselines following the most relevant previous work [13, 17, 18] struggles when the task becomes more complex.
of the proposed action representation.

---

Spatial action maps. Similar to our method, recent work has explored spatial action maps that are densely coupled with visual input instead of compressing into a global embedding, based on images [19, 4, 20], point clouds [21, 3, 22], or voxels [23]. Unlike previous works with spatial action maps that consider one-shot decision making (similar to bandit problem) [3, 4, 10] or rely on expert demonstrations with ground truth actions [19, 21, 23], our method reasons over multi-step sequences with no expert demonstrations. For example, Xu et al. [4] only chooses a single contact location followed by action sequence, rather than sequence of contact interactions. Unlike previous work in spatial action maps that uses DQN with discrete actions [24, 25, I0, 20], our hybrid discrete-continuous action space allows the robot to perform actions without discretization. Furthermore, we demonstrate the benefit of spatial action representations when applied to a 6D non- prehensile manipulation task, which is more challenging than the pick-and -place and articulated object manipulation tasks in previous work.
RL with hybrid discrete-continuout action spaces. Most RI. algorithms focus on either a discrete action space [26] or a continuous action space [7, 6, 27]. However, certain applications are defindd over a hybrid action space where the agent selects discrete action and continuous parameter for the action [28, 29, 30J. Unlike previous work, our hybrid action space uses a spatial action representation in which the discrete actions are defined over a map of visual inputs. The closest to our work is Feldman et al. [20] in terms of applying RL to spatial action maps, but they only consider a finite horizon of 2. We include formal definitions of the policies over the hybrid action space and modify the loss functions and exploration accordingly.
3 Preliminaries
A Markov Decision Process (MDP) models a sequential stochastic process. An MDP can be repre- sented as tuple (S, A, P)r, where S is the state space; A is the action space; P(* 188, 0z) is the transition probability function, which models the probability of reaching state $441 from state 8, by taking action ag; r(st, ag, $1+1) is the immediate reward at time t. The objective is to maximize the return Rg, defined as the cumulative discounted reward Rg Di=on' T' Given a policy æ, the
Q-function is defined as Q" (T a) Ex [Rc]s:
a, = a).
HACMan is built on top of Q-learning------ based off-policy algorithms with continuous action space [6, 27, 29]. In these algorithms, we define a deterministic policy parameterized by 0 and a Q- function Qo parameterized by o. Note that since the policy is deterministic, we use epsilon-gress during exploration. Given dataset D with transitions (St, at 8t41), the Q-function loss is defined according to the Bellman residual:
l(b) E2 (0 - )),
(1)
where y is defined as:
Vt rr + (8)++1))
(2)
The policy loss for Ty is defined to maximize the fufuctioo::
((0) -Ba (st, az) (m)=)))))))
4 Problem Statement and Assumptions
We focus on the task of 6D object pose alignment with non-prehensile manipulation. The objective of the robot is to perform sequence of prehenehe actions (i.e. pushing, flipping) to move an object on the table into a target goal pose. We assume that the goals are stable object poses on the table. The robot policy observes the point cloud of the scene from depth cameras, denoted X. We assume that the point cloud observation is segmented between the background and the object to be manipulated. Thus, the full point cloud X' consists of the object point cloud gobi and the background point cloud X²b The feature for each point is a 4-dimensional vector, including a 1-dimensional segmentation mask and a 3-dimensional goal flow vector (will be defined in Section 5.3).
Method
5.1 Action Representation
We propose an object-centric action space that consists of two parts: a contact location Lee on an object and motion parameters a which define how the robot moves to interact with the object after contact. As shown in Fig. 2, to execute an action, the end-effector will first move to location in
3

---

free space near location aoo after which it will interact with the object using the motion parameters a". After the interaction, the end-effector will move away from the object, a new observation is obtained, and the next action can be taken.
Specifically, given the object point cloud Xaay = {T; I - .e......y N). where 2; E R are the point locations, the contact location ahe is chosen from among the points in X Thus, dlee is defined over a discrete action space of dimension N. We assume collision-free mo-
(a) Before action execution
(b) Move to the contact location
[c) Execute the mation parameters
tion planner to move the gripper to the contact location aooe (see Appendix A.6 for details). In contrast, the motion parameters amm, which de- fine how the gripper interacts with the object after contact, are defined in continuous action space. Furthermore, we define a" as the end-effector delta position movement from the contact position, hence a" Th E R Our experiments show that translation-ons onlly movements are sufficient enable complex 6D object manipulation in our task. We also include additional experiments on extending motion parameters enable rotations in Appendix C.4.
Figure 2:
Illustration of our action space.
The proposed action representation has two benefits compared to previous work. First, it is temporally-abstracter We "abstract" sequence of lower-level gripper movements of approach- ing the contact and executing the motion parameters into one action decision step in the RI problem definition. Compared to the common action space of end-effector delta movements [31, 32, 12, 33], the agent with our action space can avoid wasting time learning how to move in free space and in- stead focus learning contact-rich interactions. Second, it is spatially-groundee since the agent selects a contact location from the observed object point cloud.
5.2 Hybrid RL Algorithm
The proposed action space is hybrid discrete-continuous action space: the contact location aee is discrete while the motion parameters am are continuous. We propose a way adapt existing off- policy algorithms designed for continuous action spaces [6, 27] to this hybrid action space. First, consider the simpler case of an action space that only has the continuous motion parameters am. In this case, we can directly apply existing off- off-policy algorithms as described in Section 3. Given the observation s (which is the point cloud X' our task), we can train an actor to output the motion parameters (s(s) = am Similarly, the critic Qø(s, a")) outputs the Q-value given the observation 8 and the motion parameters am"; the Q-value can be used update the actor according to Eqn. 3.
To additionally predict the contact location a loc we also need the policy to select a point among the discrete set of points in the object point cloud gtabj Our insight that we can embed such a discrete component of the action space into the critic by training the critic to output a per-point Q-value Q: for each point x; over the entire point cloud. The Q-value at each point on the object represents the estimated return after selecting this point as the contact location. These Q-values can thus be used not only to update the actor, but also to select the contact location the point with the highest Q-value. Additionally, we train the actor to also output per-point motion parameters a," for each point x4. If point I; is selected as the contact location Loss then the motion parameters at this point at" will be used as the gripper motion after contact.
The overall architecture shown in Fig. 3. The actor Ty receives as input the point cloud observation and outputs per-point motion parameters (x) {am" (X) | i per per-point output an "Actor Map". The critic also receives as input the point cloud observation. first calculates per-point features f(X) (f: i = 1...N). N). point feature /i (Section 4) with the corresponding per-point motion parameter a. and inputs the concatenated vector an MI.P The output of the MI.P is per-point Q-value: Q, Qo a!"), which scores the action of moving the gripper to location I; and executing motion parameters a!". We call this per-point output a "Critic Map". In this way, the critic is able reason jointly about the contact location (via the feature fi) as well as the motion parameters at". In our implementation, both the actor and the critic use segmentation-styl PointNet++ architecture [34] (Appendix B).
I...NN. We call this
The critic then concatenates each per-
At inference time, we can select the point I; within the object points ohj with the highest Q- value Qi as the contact location and use the corresponding motion parameters at" For exploration during training, we define a policy Loc which selects the contact location based on softmax of the

---

HACMan: Hybrid Actor-Critic Maps for Manipulation
Actor Map
Seomantatin-strle Network
Point Cloud Observation X
Action Guiput
Per Point Motion Parameters
Critic Map
Targat 8E(3)
Transformation
Segmentetin-allle Point Cloud
Higheet
Best Contact
Network
Point Location
Comeaonninng Motion Poneriear Movorsot)
Doal (dne=]]
Segrecriation (1ine11)
Q-value
Figure 3: An overview the proposed method. The point cloud observation includes the location of the points and point features. The goal is represented as per-point flow of the object points. The actor takes the observation as input and outputs an Actor Map of per-point motion parameters. The Actor Map is concatenated with the per-point critic features to generate the Critic Map of per-point Q-values. Finally, we choose the best contact location according to the highest value in the Critic Map and find the corresponding motion parameters in the Actor Map.
Q-values over all of the object points. The probability of point 24 being selected as the contact location is thus given as:
exp(Qi/B) exp(Qx/3)
1 (x, =8)= (xx (x)
=
(4)
is the temperature of the softmax which controls the exploration of the contact location. Note that the background points X are included in the observation s = X = {X², xr but are excluded when choosing the contact location. We modify the update rules of the off-policy algorithm for this hybrid policy. Given s = X, we first define the per-point loss for updating the actor (s)) at location 2; according to Eqn. 3:
( = - (o.. = - (o (s).
(5)
fi is the feature corresponding point z,, The total objective of the actor is then computed as an expectation over contact locations:
((0) = Eng [1((0]] Entone (x, 18) J(()).
(6)
g (x: s) is the probability of sampling contact location 2;, defined in Eq. 4. The difference between Eqn. 6 and the regular actor loss in Eqn. 3 is that we use the probability of the discrete action to weight the loss for the continuous action. To take into account Tie during the critic update, the Q-target ys from Eqn. 2 is modified to be:
y - Blo (Q (i (o,,+))))))
5.3 Representing the Goal Per-Point Goal Flow
As described in Section 4, the objective of our task is to move an object to a given goal pose. Instead of concatenating the goal point cloud to the observed point cloud [17, 18), we represent the goal as per-point "goal flow": Suppose that point in the goal point cloud; then the goal flow is given by Az; x I;.- The goal flow Az;
in the initial point cloud corresponds to point af
3D vector which is included as the feature of the point cloud observation (concatenated with segmentation label, resulting
4-dimensional feature vector). This flow representation of goal also used to calculate the reward and success rate for the pose alignment task (Appendix A.3). In the real robot experiments, we estimate the goal flow using point cloud registration (Appendix D). Our ablation experiments suggest that utilizing the flow representation of the goal drastically enhances performance compared to directly concatenating the goal point cloud (Appendix C.I).
Experiment Setup
We evaluate our method on the 6D object pose alignment task as described in Section The objective is to perform sequence of non-prehensile actions (i.e. pushing, flipping) to move the object to a given goal pose. this section, we describe the task setpp in simulation, used for training and simulation evaluation (see Section 8 for real robot experiments).
5

---

1.00
Select Location [Ours]
0.75
Select Contact Location [Ours Greedyl
Regress Contact Location (Point Cloud]
0.50
Regress Contact Location (State)
Contact Location (Point Cloud)
0.25
No Contact Location (State)
0.00
Single Object
Cylindrical Objects
Single Object
Cylindrical Objects Gools
All Objects Goals
Planar Coals
Planar
6D Goals
Task Variants
Our approach outperforms the baselines and the ablations, with a larger margin for more challenging tasks on the right. Success rates for simple tasks pushing a single object to an in-plane goal are high for all methods, but only HACMan achieves high success rates for 6D alignment of diverse objects.
Figure 4: Baselines and Ablations.
Task Setup. The simulation environment is built on top of Robosuite [35] with MuJoCo [36]. We include 44 objects with diverse geometries from Liu et al. [37]. Details and visualizations of the object models are included in Appendix A.I. The object dataset is split into three mutually exclusive sets: training set (32 objects). unseen instances (7 objects) and unseen categories (5 objects). The unseen instances consist of objects from categories included in the training set, whereas the 5 objects in "unseen categories" consist of novel object categories. An episode is considered success if the average distance between the corresponding points of the object and the goal is less than 3 cm. More details on our simulation environment setup can be found in Appendix A.
Task Variants. To analyze the limitations of different methods, we design the object pose alignment task with varying levels of difficulty. We consider three types of object datasets: An All Objects dataset that includes the full object dataset, a Cylindrical Objects dataset consisting of only cylin- drical objects, and Single Object dataset consists of just single cube. We also try different task configurations: In the Planar goals experiments, the object starts from a fixed initial pose at the center of the bin, and the goal pose randomized planar translation of the starting pose. In the 6D goals experiments, both the object initial pose and goal pose are randomized SE(3) stable poses, not limited planar transformations. This task requires SE(3) object movement achieve the goal which imposes challenges in spatial reasoning. These different task variations are used to show at what level of difficulty each of the baseline methods stop being able to complete the task.
7 Simulation Results
In this section, we demonstrate the effectiveness of HACMan compared the baselines and abla- tions. Fig. 4 summarizes the performance of each method after being trained with the same number of environment interactions. The training curves, tables, and additional results can be found in Appendix C. Implementation details of all methods are included in Appendix B.
Effect of action representations. We compare our method with two alternative action representations, summarized in Ta- ble 1. In Regress Contact Location, the policy directly regresses to a contact lo- cation and motion parameters, instead of choosing a contact point from the point cloud as in HACMan. The No Contact Location baseline directly regresses to a delta end-effector movement at each timestep. For every action, the robot continues from its position from the previous action, instead of first moving the gripper to a selected contact location. This is the most common action space in manipulation [17, 18, 12, 13, 32, 31J. As input for these two baselines, we use either point cloud observations or ground-trum state, establishing four basclines in total. The bascline that regresses motion parameters from point cloud observations is a common action representation used in prior work in RL from point clouds such as Qin et al. [13]. The bascline that regresses motion parameters from ground-truit state is the most common approach in prior work, such as in Zhou and Held [12] as well as the teacher policies in Chen et al. [17, 18] (see Appendix F for more discussion).
Table I: Features of the proposed action representation compared to the baselines.
Spatially Grounded?
Temporally
Abstracted?
Sclect Contact Location [Ours] Regress Contact Location No Contact Locution [17, 18, 12, 13, 32, 31]
As shown in Fig. 4, these baseline action representations struggle with the more complex task vari- ants. For the most challenging task variant "All Objects + 6D goals," " our method achieves a success rate 61% better than the best baseline (see Table 5 for numbers). As mentioned in Section 5, the proposed action representation benefits from being groundddd and temporally-abstriee The comparison against the baselines demonstrates the importance of each of these two features

---

Figure 5: Qualitative results for the object pose alignment task. HACMan shows complex non-
prehensile behaviors that move the object to the goal pose (shown the transparent object).
(a) Planar - thin object (b) Planar - thick object (c) Planar Multimodal solution
(d) Knock over
(e) Flip up
Figure 6: Goal-conditioned Critic Maps. Blue: goal point cloud. Color map: observed object
point cloud. Lighter colors indicate higher Critic Map scores, Red arrows: motion parameters at
selected location. The policy uses different contact locations based on object geometries and goals.
(Table 1). The "Regress Contact Location" baseline still benefits from being temporally-abstratted because the gripper starts from location chosen by the policy at each timestep; however, this action because it regresses to contact location which might not
representation is not be on the object surface, unlike our approach which selects the contact location among the points in the point cloud observation. Thus, the "Regress Contact Location" baseline suffers from training difficulties with more diverse objects (last two variants in Fig. 4). The "No Contact Location" base- line is neither -agoully-goounded nor cemporally-abstractee work [17, 18, 12, 13, 32, 31] of regressing an end-effectoo delta motion at each timestep. While this the most common action space in prior work, it has close to performance with 6D goals.
it follows the usual approach from prior
Effect of Multi-step Reasoning, To test the necessity of multi-step reasoning for the pose alignment task, we experiment with a "Greedy" version of HACMan by setting the discount factor in the RL algorithm to y 0. This forces the algorithm to optimize for greedy actions for each step. Using RL for multi-step reasoning is one of the important differences between our method and previous work such as Where2Act [3] and UMPNet [4] which optimize for one-step contact locations. Fig. indicates that greedy actions might work for planar goals, but suffer from poor performance for 6D goals that requires multi-step non-greedy interactions. For example, the last row in Fig. shows an example of method pushing the object away from the goal position to prepare for flipping it to the correct orientation, demonstrating non- non-greede greedy behavior. In contrast, we find that the greedy ablation often results in local optima of only trying to match the object position but not its orientation.
Generalization unseen objects. The evaluation of our method over unseen ob- jects with 6D goals is summarized in Ta- ble 2. Our method generalizes well unseen object instances and unseen cate- gories without performance drop. When we increase the maximum episode length from 10 steps to 30 steps, our method achieves 95.1% success unseen categories (Appendix C.7). Table shows that, comparing the same set object categories ("common categories"). the suc- cess rates of the training instances similar to the unseen instances. The differences in geometry comparing the training objects and the unseen objects are visualized in Appendix A.1.
Table 2: Generalization to unseen objects.
Object Set Split
Success Rate
# of Objects
Train
0.833 .018
32
Train (Common Categories)
0.887 = .04
13
Unseen Instance (Common Categories)
0.891
Unseen Cutegory
0.827 047
5
Goal-conditioned Object Affordance and Multimodality. We visualize the Critic Map. which computes the score of each contact location of the object (Fig. 6). The Critic Maps capture goal- conditioned object affordances which describe how the object can be moved to achieve the goal. Fig. 6(a) and (b) are two scenarios of performing translation object motions for different object heights: For thin object, the Critic Map highlights the region of the top of the object (dragging) or
7

---

Object Name
Planar Goads
Non-planar
Total
(a) Blac cup
4/7
4/13
8/20
(b) Milk bottle
6/7
10113
16/2
(c) Rox
2/5
10115
12/20
(d) Red bottle
477
0/13
4/20
[a] Blue
(bI Milk
Red bottle
Hook
[e]
(1) Black mug
4/7
0/13
4/20
(g) Red mug
5/7
3/13
8/20
(5) Wood block
6/7
6/13
12/20
Toy
( Toy block
2/2
10/18
12/20
Total
47/67
53/133
100/200
(P) Black
[h) Wood block
Toy bridge
block
Percentage
70%
4176
50%
Figure 7:
Real robot experiments. HACMan achieves 50% overall success rate over unseen objects with different geometries and physical properties, with 6D goal poses.
from the back side of the object (pushing). For a thick object, it prefers to push from the bottom to avoid the object falling over. In Fig. 6(c), the hammer needs to be rotated by 180 degrees. The Critic Map predicts a multimodal solution pushing from either end the object. Fig. 6(d) and (e) show out-of-plane motions of the object of knocking
and flipping up the objects.
Additional Results. We include additional experiment results in Appendix C, including additional ablations, extending the motion parameters, cluttered scenes, longer training steps, longer episode lengths, and success rate breakdown for each object category.
Real robot experiments
In the real robot experiments, we aim evaluate the ability of the trained policy to generalize to novel objects and execute dynamic motions in the real world. We evaluate the policy with diverse of objects with different shapes, surface frictions, and densities (Fig. 7). We use random initial poses and random 6D goals (referred to in the previous section as "All Objects + 6D Goals"), For example, the red mug (Fig. 7g)) has goal poses of being upright on the table or lying on the side, episode is considered a success if the average distance of corresponding points between the object and the goal smaller than em; we also mark an episode as failure there is a failure in the point cloud registration between the observation and the goal. Implementation details of the real robot experiments can be found in Appendix D.
Fig. 7 (right) summarizes the quantitative results of the real robot experiments. We run the evalua- tions without manual reset, which may create uneven numbers of planar versus out-of-pland goals. It achieves 70% success rate on planar goals and 40% success rate on non-planar goals. Non-planar goals are more difficult than the planar goals because they require dynamic motions to interact with the object. A small enror in the action may result in large changes in the object movement. Videos of the real robot experiments can be found on the website: https: //ha cman 20003. g thub. The real robot experiments demonstrate that the policy is able to generalize to novel objects in the real world, despite the sim2real gap of the simulator physics and inaccuracies point cloud regis- tration for estimating the goal transformation. More discussion can be found in Appendix D.
Limitations
Since the contact location in our action space is defined over the point cloud observation. our method requires relatively accurate depth readings and camera calibration. Further, the contact location currently limited to the observed part of the object. In addition, for this goal- coal-conditioned task, we represent the goal as per-point flow (Section 5.3) which relies on point cloud registration algorithms. Inaccuracies in the registration algorithm sometimes lead to failure cases in real robot experiments. More discussion
failure cases can be found in Appendix D.3. In addition, finetuning the policy on the real robot can potentially improve the success rate further.
10 Conclusion
In this work, we propose to learn Hybrid Actor-Critic Maps with reinforcement learning for non- prehensile manipulation. The learned policy shows complex object interactions and strong general- ization across unseen object categories. Our method achieves a significantly higher success rate than alternative action representations, with a larger performance gap for more difficult task variants. In addition, it would be interesting to explore alternative 3D representations other than point clouds such implicit representation [38, 39). We hope the proposed method and the experimental results can pave the way for future work on more skillful robot manipulation over diverse objects.

---

Acknowledgments
We thank Thomas Weng, Ben Eisner, Daniel Seita, Mrinal Kalakrishnan, Homanga Bharadhwaj, Patrick Lancaster, Vikash Kumar, Jay Vakil, and Priyam Parashar for the insightful feedback throughout this project.
References
[1] x. Cheng, E. Huang, Y. Hou, and M. T. Mason. Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d. In 2022 International Conference on Robotics and Automation (ICRA), pages 2730-2733 IEEE, 2022.
[2] Y. Hou and M. T. Mason. Robust execution of contact-rich motion plans by hybrid force- velocity control. In 2019 International Conference on Robotics and Automation (ICRA), pages 1933-1939. IEEE, 2019.
[3] K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6813-6823, October 2021.
[4] 7. Xu, H. Zhanpeng, and S. Song. Umpnet: Universal manipulation policy network for articu- lated objects. IEFE Robotics and Automation Letters, 2022.
[5] K.-T. Yu, M. Bauza, N. Fazeli, and A. Rodriguez. More than a million ways to be pushed. high-fidelity experimental dataset of planar pushing- In 2016 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 30-37, 2016. doi:
[6] Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[7] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 10w::559900000 2015.
[8] M. T. Mason. Progress in nonprehensile manipulation. The International Journal of Robotics Research, (8(11):1129-1 111,, 1999.
[9] I. Mordatch, E. Todorov, and Z.. Popović. Discovery of complex behaviors through contact- invariant optimization. ACM Transactions on Graphics (TOG), 31(4):1-8, 2012.
[10] J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. Proceedings of Robotics: Science and Systems (RSS), 2020.
[11] Liang, Cheng, and O. Kroemer. Learning preconditions of hybrid force-velocity con- trollers for contact-rich manipulation. arXiv preprint arXiv:2206. 2022.
[12] W. Zhou and D. Held. Learning to grasp the ungraspable with emergent extrinsic dexterity. In Conference on Robot Learning (CoRL), 2022.
[13] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. Conference on Robot Learning (CoRL), 2022.
[14] M. Liu, X. Li, Z. Ling, Y. Li, and H. Su. Frame mining: free lunch for learning robotic manipulation from 3d point clouds. In 6th Annual Conference Robot Learning, 2022.
[15] W. Huang, I. Mordatch, Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv preprint arXiv:2111.03060 2021.
[16] L. Wang, Y. Xiang, W. Yang. A. Mousavian, and D. Fox. Goal-auxiliary actor-critic for 6d robotic grasping with point clouds. In The Conference on Robot Learning (CoRL), 2021.
[17] Chen, J. Xu, and Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning, 2021.

---

[18] Т. Chen, M. Tippur. S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:221..1174., 2022.
[19] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robol Learning (CoRL), 2020.
[20] Z. Feldman, H. Ziesche, N. A. Vien, and D. Di Castro. A hybrid approach for learning to shift and grasp with elaborate motion primitives. In 2022 International Conference on Robotics and Automation (ICRA), pages 6365-666666666 0755-631. IEEE, 2022.
[21] D. Seita, Y. Wang, Shetty, E. Li, Z. Erickson, and D. Held. ToolFlowNet: Robotic Ma- nipulation with Tools via Predicting Tool Flow from Point Clouds. In Conference on Robot Learning (CoRL), 2022.
[22] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1 -3444, 2021.
[23] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[24] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4238- 4245. IEEE, 2018.
[25] A. Hundt, B. Killeen, N. Greene, H. Wu, H. Kwon, C. Paxton, and G. D. Hager. "good robot!": Efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE Robotics and Automation Letters, 5(4)67224 6731, 2020.
[26] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 131..503, 2013.
[27] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[28] M. Neunert, A. Abdolmaleki, M. Wulfmeier, T. Lampe. T. Springenberg, R. Hafner, F. Ro- mano, J. Buchli, N. Heess, and M. Riedmiller. Continuous-discrete reinforcement learning for hybrid control in robotics. In Conference Robot Learning, pages 735 751. PMLR, 2020.
[29] M. Hausknecht and P. Stone. Deep reinforcement leann in parameterized action space. arXiv preprint arXiv:1511.00 1,.4111,111111111111111 2015.
[30] J. Xiong, Q. Wang, 7. Yang, P. Sun, L. Han, Y. Zheng, H. Fu, T. Zhang, J. Liu, and H. Liu. Parametrized deep q-networks learning: Reinforcement learning with discrete-comtinuous hy- 2018.
brid action space, arXiv preprint 1810v:1110.0633
[31] T. Mu, Z. Ling, F. Xiang, D. C. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. Man- iskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[32] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https: pettps:/aarxee tttp//awwwww.gaa.////o/aaooootsa wttp:////p/./............
[33] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv: /62.0006,, 2016.
[34] C. R. Qi, I.. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. arXiv preprint arXiv: 1706.02413, 2017.
10

---

[35] Y. Zhu, J. Wong, A. Mandlekar, R. Martin, A. Joshi, S. Nasiriany, and Y. Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. /2293, 2020.
[36] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012. doi:10.1 100.1100//RS2220..6350000
[37] W. Liu, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. arXiv preprini arXiv:2211.04666 2022.
[38] J.-. Ha, D. Driess, and M. Toussaint. Deep visual constraints: Neural implicit models for manipulation planning from visual input, 2022.
[39] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se(ii-aquivariant object representations for manipulation, 2021.
[40] S. Hinterstoisser, Lepetit, Ilie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. Computer Vision-ACCV 2012: 11th Asian Conference on Computer Vision, Dae- jeon, Korea, November 5-9, 2012, Revised Selected Papers, Part I 11, pages 2013.
Springer,
[41] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes, Robotics: Science and Systems (RSS), 2018.

---

Appendix
Table of Contents
A Simulation Environment
12
Ohject dataset preprocessing
A.2 Collecting goal poses
A.3 Representing the goal per-point goal flow
................................
13
A.4 Success rate definition
14
A.5 Observation
14
A.6 Action
15
B Algorithm Training Details
B.I HACMan (Ours)..
15
B.2 Baselines
15
C Supplementary Experiment Results
17
C.I Additional ablations
17
C.2 Training curves and tables
17
Additional baseline: Global feature with query contact location
C.4 Extending Motion Parameters
C.S Experiments in cluttered environments
20
C.6 Effect of longer training time
20
C.7 Effect of longer episode lengths
20
C88 Per-category result breakdown.
21
C.9 Final Distance to Goal
21
D Real Robot Experiments
22
D.1 Real robot setup
................................
22
D.2 Analysis...
23
D.3 Failure cases
24
More discussion on non-prehensile manipulation
24
F More discussion the related work
24
F.1 Compared to Chen et al. [17, 18]
24
F22 Compared to Cheng et al. [1], Hou and Mason [2]
25
A Simulation Environment
A.1 Object dataset preprocessing
We use the object models from Liu et al. [37]. Before importing the object models to Mu- Joco, we perform convex decomposition using V-HACD v- v-hacd) and generate watertight meshes using Manifold (https: //gttub com/ h hjwdzh// Manifolde The objects first scaled to 10 cm according to the maximum lengths along x, y, and z axis. The object sizes are randomized with an additional scale within [0.8, 1.2] for the "All Objects" task variants.
We filter out a part the objects the original dataset due to simulation artifacts such as wall penetration and unstable contact behaviors. For example, some of the long and thin objects can be pushed into the walls and bounce back like springs. Some of the objects cannot remain stable on the table. The filtering procedure proceeds as follows: 1) we drop object with arbitrary quaternion
12

---

Figure 8: Training objects. 32 objects used in training.
Evaluation objects (unseen in- stance). 7 objects used in unseen instance eval- uations. These instances are from the same cat- egories as the training objects.
Figure 9:
Figure 10: Evaluation objects (unseen cate- gory). 5 objects used in unseen category eval- uations. They come from randomly chosen categories.
and translation for 100 times; 2) calculate the percentage of rollouts where the objects still unstable after 80 simulation steps; 3) we filter out objects with larger than 10% instability rate. We also filter out flat objects because they hard to flip. Flat objects defined objects for which the ratio between the second smallest dimension to the smallest dimension is larger than 1.5. After filtering, are left with 44 objects. We split the 44 objects into three datasets: train (32 objects), unseen instances (7 objects), and unseen categories (5 objects). The object models of the three datasets are visualized Fig. 8, Fig. and Fig. 10 respectively. The Cylindrical Objects used in the experiments is a subset of the All Objects dataset. Cylindrical Objects consist of 9 train objects, 3 unseen instance objects, and unseen category objects.
A.2 Collecting goal poses
To collect stable goal poses, we sample an SE(3) object pose the air above the center of bin, drop the object in the bin, and then wait until it becomes stable to record the pose. We collect 100 goal poses for each object. At the beginning of each episode, goal is sampled from the list of stable poses. Furthermore, we randomize the location of the sampled stable goal pose within the bin.
A.3 Representing the goal as per-point goal flow
As mentioned in Section 5.3, we represent the goal as the "goal flow" of each object point from the current point cloud to the corresponding point in the transformed goal point cloud. In other words, suppose that point z; in the initial point cloud corresponds to point of in the goal point cloud; then
13

---

Camera
Camara B
Camera
Figure 11: Camera locations simulation.
the goal flow is given by Ar; = = to the other features of the input point cloud to represent the goal. In the ablations in Appendix C.I. show that such representation of the goal significantly improves training, compared other goal representations such as concatenating the goal point cloud with the observed point cloud.
Iz. The goal flow Ar; is 3D vector which is concatenated
In order to compute the flow to the goal, need estimate correspondences between the obser- vation and the goal. simulation, we calculate the goal flow based on the ground truth correspon- dences, based the known object pose and goal pose. In the real robot experiments, estimate the correspondences using point cloud registration methods (see Appendix for details).
Further, for training the RL algorithm, we need some measure of the distance between the initial pose and the goal pose as the reward. Rather than computing a weighted average of the translation and rotation distance (which requires a weighting hyperparameter), we instead define the reward at the negative of the average goal flow: rn
each timestep denotes the L2 distance and Az; is the "goal flow" as defined above. This computation is similar to the "matching score" [40] or "PLoss" [41] used in previous work, except here it a reward function.
Σ (ee1llAza ||sll in which ||
A.4 Success rate definition
episode is marked as a success when the average distance of the corresponding points between the object and the goal is smaller than 3 cm. More specifically, this is calculated by the average norm the per-point goal flow vectors as described in Appendix A.3. The episode terminates when it reaches a success. If the episode does not reach success within 10 steps, it is marked as failure. We include an additional experiment on longer episode length in Appendix C.7.
A.5 Observation
The observation space includes point cloud of the entire scene X. It contains background points d and object points Xoby Note that we move the gripper to reset pose after every action before taking the next observation. Thus, the gripper not observed in the point cloud. To get the point cloud, we set three cameras around the bin (Fig. 11). The depth readings from the cameras are converted to set of point locations in the robot base frame and combined.
The object points are then downsampled with a voxel size of 0.005 m x 0.005 m X 0.005 m and the background points are downsampled with a voxel size of 0.02 x 0.02 0.02 m. We empirically find that using a slightly denser object point cloud may increase the performance. More specifically, using 0.005 x 0.01 m. We suspect that the policy can perform more precise manipulation of the object with a denser point cloud.
0.005 0.005 voxel downsample is slightly better than 0.01 x 0.01 m
After downsampling, we estimate the normals of the object points using Open3I (http:/ http://wwww. open3d. org/ docs/o. 0 httppythooaaappppeeeepeoop appp////d/dppppppppp//mm/mm geometry.
14

---

estimate_ satpteee_noaalsseeeeee htmml. The estimated normals will be used during action execution (discussed in the next section).
As mentioned in Section 5.3 and Appendix A.3, the feature of each point contains the goal flow and the segmentation mask (foreground vs background). The goal flow of the object point is cal- culated according to Section A.3. The goal flow of the background point is set to zero. We obtain the segmentation labels of the object points and the background points from Robosuite[35] dur- ing simulation. Details of obtaining segmentation labels in real robot experiments are discussed in Appendix D.
A.6 Action
As mentioned in Section 5, the proposed method uses an action space with a contact location selected from the object points and set of motion parameters. We discuss the implementation details of executing such an action in the simulation environment this section. Note that we use a floating gripper as the robot in simulation since only focus on gripper interactions with the objects.
Once the policy selects point on the object point cloud, we obtain the corresponding location and estimated normal of the point as described in the previous section. The robot first moves to a "pre-contact" location which is 2 em away from the contact location along the surface normal. In simulation, this is implemented by directly setting the gripper to the desired pose. In real ex- periments, we adopt a workaround solution discussed in Appendix D. If the gripper encounters a collision the desired pose, we mark this action failure and skip the remaining action execution procedure. After reaching the pre-contact location, the gripper will approach to the desired contact location using a low-level controller.
After that, the robot will execute the motion parameters which the end-effector delta position command that was output by the policy. For the delta actions, we use an action scale of 2 cm. The delta action is executed with an action repeat 3. We use Operation Space Controller with relatively low gains to allow compliant contact-rich motions with the object. Note that we only consider translation commands (3 dimensions) without rotation in the main experiments because it leads to sufficiently complex object motion for our task. Appendix C.4 discusses the effect of including rotation the gripper movements.
The gripper may not exactly reach the desired location in both sim and real, due to the compliant low-level controller and the gripper geometry. We consider this imperfect execution as part of the environment dynamics. We do not enforce assumptions such as keeping the contact while executing the motion parameter or avoiding other contact points. Avoiding such assumptions on contacts is strength of the proposed method compared to some of the classical methods [1, 2].
B Algorithm and Training Details
B.1 HACMan (Ours)
HACMan is implemented as modification on top of TD3 [6] based on the implementation from Stable-Baselines3 (https: /tttttbtttttt///////////. com/ DLR- RM--RM/SS0 st sttaalee-bel wwweeeaallleee (aaseliness) We use PointNet+ segmentation--stlee backbones for both the actor and the critic using the implementation from PyG settps::/pytorrchie wwwerrch--emmerriee readthedocs: io). Weights are not shared between the actor and the critie. Hyperparameters are included in Table 3. The actor and the critic use the same network size and the same learning rate. To improve the stability of policy training, we clamp the target Q-values according to an estimated upper and lower bound of the return for the task. The location policy temperature /3 is described in Eqn. 4.
B.2 Baselines
The baselines share the same code framework HACMan. We discuss their differences with HAC- Man in this section.
Regress Contact Location. Unlike HACMan, this baseline does not use the object surface for contact point selection. Instead, it directly predicts a location (3 dimensions) and motion parameter
15

---

Table Hyperparameters.
Hyperparameters
Values
Initial timesteps
10000
Batch size
64
Discount factor (7)
0.99
Critic update freq per env step
2
Actor update freq per eny step
0.5
Target update freq per env step
0.5
Learning rate
0,0001
MLP size
[128. 128, 128]
Critic clamping
[-20, 0]
Location policy temperature (3)
0.1
(3 dimensions, represented as a delta end-effector movement). For each action execution, the end- effector moves the selected location, moves according the motion parameters, and then resets to the default pose. To improve the performance of this baseline, we project the contact location output to be within the bounding box of the object. Thus, in this baseline, for location output of the policy, a value of corresponds to the center of the object along specific dimension, while 1 and -1 represent the maximum and minimum boundaries of the bounding box along that dimension, respectively. Since the location output is no longer point selected from the object surface, we can no longer use the surface normal vector to determine the approach direction as in HACMan. Instead, this baseline always approaches the location from the top at height equal to the maximum side length of the object bounding box.
No Contact Location. This baseline does not the idea of a contact point. Instead, the policy only predicts a motion parameter (3 dimensions, represented as a delta end-effector movement). For each action execution, the end-effector moves according the motion parameter starting from where ends after the previous action, without resetting to the default pose. To reduce the exploration difficulties, we make
additional changes: 1) always start the end-effector right above the object (at a height equal to the maximum side length of the object bounding box) at the beginning of episode, and 2) we add an extra term to the reward function that penalizes the end-effector for being too far from the object,
{ Adiaa - 0.05), dmin 0.05 m
Jais
0, otherwise
where dmnin is the minimum distance from the end-effector to the object point cloud vertices, and Adist is the weight for this reward term.
Point Cloud. Unlike HACMan, these point cloud baselines use PointNet++ classification-styy -tyle backbones from PyG (https: cloud, it extracts single global feature vector instead of per-point feature.
readt hedoc. io). For each point
State. In the state-based baselines, the input consists of the pose of the current object, the goal, and optionally the end-effector if the baseline is using "No Contact Location".. Each pose is vector (dim=7) that consists of position (dim=3) and a quaternion (dim=4). The model concatenates all the pose vectors into single vector as the input to an ML..
We report the best results of the baselines in the paper by searching over different hyperparameters for each baseline, including learning rate, actor update frequency, initial timesteps, and EE distance weight Adiet- The best hyperparameters for each baseline that are different from HACMan are summarized in Table 4; any hyperparameter not listed in Table is the same as our method (Table 3).
Table 4: Baseline-specific Hyperparameters.
Baselines
Hyperparameters
Values
Regress Contact Location (Point Cloud)
Actor update freq per eav step
0.25
No Contact Location (Point Cloud)
Actor update freq per env step
0.25
EE Distance Weight Asaa
No Contact Location (State)
RF Distance Weight Agii
16

---

C
Supplementary Experiment Results
C.1
Additional ablations
We perform additional ablation studies to analyze each component of the proposed method with all
the variants of the object pose alignment task. The results of the ablations summarized in Fig. 12.
1.00
Success Rate
Ours
No Actor Map
(Goal Transformation)
No Flow (Goal PC)
Random Location
0.25
0.00
Single Object
Cyliddricl Objects Planar Goals
Single Object
Cylinérical Objects
All Objects
Planar Goals
6D Goals
6D Goals
60 Goals
Task Variants
Figure 12: Additional ablations. All of components of our method are essential to achieve the best performance when the task becomes more difficult.
Effeet of Contact Location: To test the hypothesis that contact location matters for non non-prehensile manipulation, design "Random Location" ablation: the policy randomly selects contact location on the object instead of learning to predict contact location. From Fig. 12, we observe a performance drop for not predicting the contact location for the simplest task variant.
Effect of Goal Representations: As described in Section 5.3 and Appendix A.3, in our method, represent the goal by first computing the correspondence between the observation and goal point clouds and concatenating a per-point "goal flow" to the observation. We include two alternative goal representations to justify the use of goal flow in our pipeline: "No Flow (Goal PC") concatenates the goal point cloud with the observed point cloud [17, 18]. We use an additional segmentation label in the point features distinguish the goal points from the observed points. From Fig. 12, this ablation only works well on planar goals for this task. In "No Flow (Goal Transformation)", represent the goal as the transformation between the current observation pose and the goal pose. We represent this transformation as a 7D vector that includes a translation vector and a quaternion. We concatenate the 7D goal pose the observation all of the object points. Note that, similar to our method, this baseline also requires computing correspondences between the observation and the goal. This approach performs well but slightly worse than our method in the last two task variants.
Effect of Actor Map: Instead of using an Actor Map which has per-point outputs, this ablation uses actor that outputs single vector of motion parameters while keeping the Critic Map. This is different from the baselines in the previous section that remove both the Actor and Critic Maps. In observe relatively minor performance drop compared the
the "No Actor Map" experiments, per-point action output from an Actor Map instead of a single full method. Nonetheless, using the per-po output may allow the agent reason effectively about different actions for different contact locations, such as the multimodal solution shown in Fig. (middle).
C.2 Training curves and tables
In this section, include the full training results for all the methods with additional task variants. Fig. 13 and Fig. 14 include the training curves for the baselines and the ablations. Table and Table are recorded at 200k environment interaction steps from the training curves for all the methods. The numbers in the tables are used to generate the bar plots in Fig. 4 and Fig. 12.
Note we also interpolate between the tasks "Planar Goals" and "6D Goals" and include additional task configuration with a fixed initial object pose and a randomized 6D goal, "6D Goals (Fixed Init)". This task configuration is combined with the Single Object dataset and the Cylindrical Object dataset. Thus, we include 7 variants in total (5 variants in the main paper).
As discussed in Section 7, the baselines and ablations have poor performance when the task becomes more challenging. Our method achieves the best converged performance across all task variants while being more sample efficient.
17

---

Single Object Goals
Cylindrical Goals
Goals
6D Goals (Fixed Ini))
1.0
UU
1.0
1.0
1.0
Success
0.6
0.6
0.4
0.4
0.4
0.4
150 Environmmnnt Step (1=3)
50
100
200
50
100
150
200
50
100
150
200
5.
100
150
200
Environment Step [1m3]
Envinonment Step [1w33)
En Envnrmmennt Step (1+33)
Cylindrical Objects
Cylindrical Objects
All Objects
(Fixed Init)
60
1.0
100
1.0
Ours
0.8
0.8
0.8
Regress Contaci
Location (Point Cloud)
Regress Contact
Location (Stane)
0.4
0.4
0.4
No Contact Location cloud)
0.2
0.2
0.2
Cantact Location
(Statel
50
100
150
200
50
100
15.0
2000
150 Environment Step (1e3)
50
100
200
Environeennt Stop (163)
Environment Stop (1s3)
Figure 13: Baselines. It shows success rates on the train dataset over environment steps. The shaded area represents the standard deviation across three training seeds.
Single Object Planar Goals
Single Object
Single Object 6D Goals
Cylindrical Objects Planar Goals
6D Goals (Fixed Init)
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.2
0.2
M
0.2
0.2
0.0
0.0
0.0
0.0
150
100
100
150
Step 11e1)
Step
Step
Cylindrical Objects GD Goals (Fixed Init)
Cylindrical Objects
All Objects
6D Goals
6D Goals
0.8
0.6
0.6
0.5
flow (Goal PCI No Flow (Soal Transtermation)
0.4
Groody
0.0
0.0
0.0
150 Environeent Stop (163)
100
200
100
150
2000
150 Enviromment Step (1m3)
100
Environment Step 11a3)
Figure 14: Ablations. It shows success rates on the train dataset over environment steps. The shaded area represents the standard deviation across three training seeds.
Table Baselines. We compare our method with baselines with different action representations and observations. Our approach outperforms the baselines, with larger margin for more challenging tasks. The success rate is reported with the mean and standard deviation across three seeds.
Methods
Object Dataset
Task Configuration
No Contact Location
Regress Contact Location
Ours
State
Point Cloud
State
Point Cloud
Planar Goal
0.812 + .012
0.973 .016
1,000 .000
0.996 .005
1.000 .000
Single Object
6D Goal (Fixed Init)
0.003 + .000
0.020 + 0022
0.060 # 0144
0.971 + .005
0.982 + .004
6D Goal
0.000 # .000
0.009 = .000
0,573 .015
0.991 # .004
0.997 = .003
Planar Goal
0.361 + .019
0.107 000
0.990 .002
0.924 .027
0.961 + .003
Cylindrical Objects
6D Goal (Fixed Init)
0.001 # .001
0.021 # 000
0,264 = .017
0.324 2 .014
0.885 = .004
6D Goal
0.006 + .002
0.035 = .002
0.258 = .010
0.187
.012
0.879 = .014
All Objects
6D Goal
0.012 + .004
0.016 + .009
0.094 + .018
0.243 + .028
0.854 = .028
18

---

Table Ablations. We show that all of the components are essential achieve the best performance when the task becomes more difficult. Each success rate is reported with the mean and standard deviation across three seeds.
Methods
Object Dataset
Task Configuration
Random Location
Greedy
(Goal PC)
No Flow
(Goal Pose)
Action Map
No
Ours
Goal
0.323
1,000 +:
.000
1,000
.000
1.000 .000
Single Object
6D Goal (Fixed Init)
0.075 + 005
0.754 .23
0.198 + 025
1,000 2 .000
0.991 2 .002
0,982 2 .004
6D Goal
0.037 003
0.633 + 014
0.181 018
0.994 000
0.989
.00
0.997
.003
Planar Goal
0.158 +. 006
0.767 a. .017
0,949 a .003
0.927 a .012
0.925
.012
0.961 e .003
Cylindrical Objocts
6D Goal (Fixed Init)
0.097
0,346 2
0.189
0,746
0.805 006
0.885
.004
6D Goal
0.093 + 004
0,262 a. .011
0.216 a .008
0.631 a: 016
0.775 .018
0.879 .014
Objects
6D Goal
0.147
.28
0.835 .017
0.854 .088
C.3
Additional baseline: Global feature with query contact location
We consider an additional baseline in this section where both the actor and the critic use global point cloud feature and a query contact location as input. The query contact location represented as 3D coordinate (I, y, =). More specifically, the actor takes as input global feature and contact location and outputs a continuous vector of motion parameters. The critic takes as input a global feature and contact location and outputs a Q-value. Since both the actor and the critic require contact location as input, we still need a way of selecting the query contact location. We follow similar way our method use the observed points select the contact location based on the highest Q-value. In this way, the action space remains a discrete-continune action space segmentation-style per-point feature.
the object as candidate queries and
method, but it uses a global point cloud feature rather than
As shown in Fig. 15, this alternative baseline performs worse than our method. We hypothesize that a segmentation-styy point cloud network can reason about the local point features more effectively than global feature extractor due to skip connections.
Single Object
Objects
1.0
1.0
Sucoess
0.4
2
0.4
ParPoint Feature (Ours)
0.2
0.2
Global Feature Query Contact Location
50
100
150
200
20
40
60
80
100
Invironment Step (1e3)
Em romment Step (1e3)
Figure 15: Comparison between our method and the additional baseline with query contact locations. The left figure shows the success rate of the simplest task variant single object with planar goals. The right figure shows the most challenging task variant all objects with 6D goals. The shaded area represents the standard deviation across three training seeds. Our method performs better than the baseline in both cases.
C.4 Extending Motion Parameters
The motion parameters the main results are defined as a 3D vector that describes the translation motion of the gripper. In this section, we extend the motion parameters in different ways:
6D Contact. The motion parameters also predict the orientation of the gripper when the gripper approaches the contact location. The orientation is in the form of ZYX Euler angles. To account for the physical constraints of our task setup, we restrict the y and angles to the range of [-0.5,, 0.5mT. and the z angle to the range of [-
6D Motion. We introduce the ability for the gripper to change orientation while executing the motions after making contact. Similar to the translation motion parameters, the rotation motion parameters (ZYX Fuler angles) represent the delta rotation at each action repeat step.
Per-point Contact Location Offset. We conduct an experiment where the agent learns a per-point contact location offset combined with 3D motion. The agent's continuous action space is defined as
19

---

(contact offset, 3D motion parameters). For each action on given point at location the agent uses (x Poffset) as the contact location. Notably, the ooffset value is mapped scale with the bounding box since it ranges between [-,,
'able 7 compares the performance of HACMan with the modified action spaces. The success rates are re- ported along with their corresponding standard devi- ations. We find that including 6D motion in the mo- tion parameters results a slightly higher success rate. However, 6D contact or contact offset does not seem provide significant benefits. We also try to include 6D motion in the Regress Contact Location baseline. As shown Table 7, 6D motion improves the performance of the baseline, but the success rate still much worse than our method.
Table 7: Success rates with different mo- tion parameters. All methods are evaluated on all train objects with 6D goals.
Method
Success Rate
HACMan Default
0.833 .018
with 6D Motion
0,866 090
with 6D Contact
0.819 + .077
with Contact Offset
0.800 .011
Regress Contact Location Default with 6D Motion
0.243 + .028
0.356 + .133
C.5 Experiments in cluttered environments
We can directly apply HACMan to a setting of ma- nipulating objects in cluttered scenes. We con- duct preliminary experiments in which we introduce varying numbers of scene objects into the bin. The scene objects serve as obstacles that add challenges the task. We train HACMan under two condi- tions: with one scene object and with five scene objects, and we compare the results with the perfor- mance achieved in the absence of any scene objects (default setting). From Table as expected, the task becomes more challenging when there are more obstacles in the bin. As illustrated in Fig 16, the policy tends to push the object directly toward the goal by pushing the scene object aside.
Table 8: Success rates under different clut- tered scenes. All methods are evaluated with 6D goals.
# of Scene Ohjects
Success Rate
0 (Default)
0.833
1
0.773
5
0.580
Figure 16: Qualitative results for object pose alignment tasks in cluttered environments. HAC- Man shows complex non-prehensile behaviors that move objects to goal poses (shown as the trans- parent objects). The scene objects are colored in brown to distinguish from the target object to be manipulated the goal pose.
C.6 Effect of longer training time
Although report the success rate 200k training steps for all the results due to computational limitations, our method continues to improve performance with longer training (Figure 17). The graph illustrates the success rate achieved by method the number of training steps increases. Notably, after 500k training steps, our method achieves a success rate of 91.1 + 7.3%, significantly improving from the 83.3% success rate reported in the main text (at 200k training steps).
C.7 Effect of longer episode lengths
We conducted an additional experiment to explore the relationship between success rates and max- imum episode length. In the main paper, our episodes were limited to maximum of 10 steps, and
20

---

Success rate over training Steps
9. Rate
0.6
0.2
100
300 Environment Step 11e3)
200
500
Figure 17: Success rate with extended training. The success rate our method reaches 91.1 + 7.3% after 500k training steps, compared 83.3% after 200k training steps.
vs. Episode Length
1.0
0.8
Success Rate
0.6
0.4
0.2
Train (Commont
Instance (Common)
Unseen Categoty
20
Episode Longth
Figure 18: Success rates various maximum episode lengths. This line plot shows the success rates of HACMan evaluated the four datasets. It is worth noting that the success rates for Unseen Instance (Common) and Train (Common) are marginally higher compared to Train and Unseen Category, similar to the pattern in Table 2.
any episode exceeding this limit was deemed a failure. During this additional evaluation, we relaxed the episode length restrictions and allowed the agent to operate with maximum episode length of 30. As shown in Fig 18, HACMan achieves more than 95% success rates across all datasets (Train 96.6%, Train (Common) 99.4%, Unseen Instance (Common) 99. 7%, Unseen Category 95. 1%) when the maximum episode length extended to 30. This suggests that providing the agent with a longer time horizon enables it to achieve higher success rates without the need for retraining.
C.8 Per-category result breakdown
Fig. 19 shows the breakdown of the results for each object category. Although our method demon- strates consistent performance across the majority of objects, there certain objects with geome- tries that pose intrinsic challenges for our approach. For example, our method is limited to poking a bowl from the top due to occlusions, making it difficult flip upward-facing bowl downwards.
C.9 Final Distance to Goal
To further analyze the performance of our method, Fig. 20 visualizes the distribution distances to goal of our method the end of the episodes for the "All Objects 6D Goals" task variant. These distances are computed as the norms of the flow distances between the objects and their respective goals.
21

---

Success Rate per Object Category
2el botfle (X1)
bottle
Iuncc bag (UC)
bontle
Object Category
Denncil Cane
Fauhlight
Alant
plans
Aand bel
bitcher
marcer 1003
haaaphone
bool
0.0
0.2
0.4
0.6
0.8
1.0
Success Rate
Train
Unseen Instance (UI)
Unseen Category (UC)
Figure 19: Results breakdown. Object categories in the unseen instance set (orange) can be com- pared to the same object categories in the train set (blue) to see the level of instance generalization.
Histogram of Episode-End Flow Distances
success threshold
of Episedess
6
0.10
Flow Distance (m)
Figure 20: Distribution of distances to the goal at the end of the episode for our method in the "All Objects 6D Goals" experiment. The vertical dashed line represents the success threshold at distance of 0.03m. The distribution has median of 2.57cm, mean of 3.66cm, and standard deviation 4.27cm.
D Real Robot Experiments
D.I Real robot setup
The robot setup is shown in Fig. 21. We use three cameras on the real robot to get combined point cloud. We follow similar procedure as Appendix to process the point cloud and to execute the action except the following details: We segment the object points from the full point cloud
22

---

Camera A
Camera B
Camera
ROBOTICS
Figure 21: Real robot setup.
based on the location and the dimension of the bin instead of using the ground truth segmentation labels from Robosuite. To move the gripper to the pre-contact location, first move the robot to location above the pre-contact location and then move down to the pre-contact location, instead of "teleporting" the gripper in simulation.
To obtain goals for the real world evaluation, we record 10 goal point clouds for each object by manually setting the objects into different stable poses. During each timestep, we use point cloud registration algorithm to estimate the goal transformation to calculate the goal flow. Specifically, we use the global registration implementation from Open3D (http: /www 3d. org/ docs// release/ release/tutore /et/uuuoriie /pipel ines /inee/lllssi globalbll.. wwwooaaeegissraatlaeeeeeeeaeaaae ionn html) and then use Iterative Closest Point (ICP) for local refinement. Note that only match the shapes of the object instead of matching both the colors and the shapes due to the limitation the registration algorithms.
Note that the evaluation process can be done automatically without any manual resets. The reward and the episode termination condition (Appendix A.4) are both calculated automatically.
For the real robot experiments, we use the policy trained with "6l) goals" and the "All Objects" dataset. We perform zero-shot sim2real transfer without finetuning or additional domain random- ization. We have tried to add noise to the contact location execution and add noise to the point cloud observation. However, these modifications did not result in better real robot performance.
D.2 Analysis
We include additional analysis the real robot results in this section. The proposed method assumes an estimated goal transformation as input. To estimate the transformation from the object to the goal, point cloud registration, described above. However, the estimation of the transformation might not be perfect in the real world. To better understand the performance of our system, we define two types of evaluation criteria: The "flow success" is automatically calculated based on the estimated point cloud registration according to the evaluation metric in Appendix A.4. Hence, "flow success" will sometimes mark an episode as a success or failure incorrectly due to errors in the point cloud registration. For the "actual success" evaluation metric, we manually mark as failures the cases among the flow success episodes where the goal estimation is significantly wrong. Thus, "flow success" indicates the performance of the trained policy (assuming perfect point cloud registration termination) while the "actual success" indicates the performance of the full system (accounting for errors in the point cloud registration). Fig. 7 in the main text reports the actual success. We include both success metrics in Table 9 below. The policy achieves a 61 % success rate based on the flow success, indicating that some of our errors are due to failures in point cloud registration.
23

---

Table 9: Additional analysis on the real robot experiments. An episode considered "flow success" if the average norm of the estimated flow is less than cm. An episode is considered as an "actual success" if the object aligned with the goal pose without point cloud registration failure.
Planar Goals
Non-planar Goals
Total
Object Name
Flow
Actual
Flow
Actual
Flow
Actual
(4) Blue cup
4/7
4/7
7/13
4/13
5/20
4/20
(b) Milk carton
6/7
6/7
10/13
10/13
16/20
16/20
(c) Box
2/5
2/5
10/15
10/15
12/20
12/20
(d) Red bottle
7/7
6/13
0/13
13/20
4/20
(e) Hook
5/8
5/12
10/20
10/20
(f) Black mug
4/7
2/13
0/13
6/20
4/10
(g) Red mug
5/7
5/7
7/13
3/13
12/20
8/20
(h) Wood block
6/7
6/7
8/13
6/13
14/20
12/20
(i) Toy bridge
9/10
9/10
7/10
5/10
16/20
14/20
(j) Toy block
2/2
10/18
10/18
12/20
12/20
Total Percentage
50/67
47/67
72/133
53/133
122/200
100/200
75%
70%
54%
61%
50%
D.3 Failure cases
We discuss the failure cases of the real robot experiments in this section and include the videos on our website: https://hacman-2223 o.t.aammmaa......ii.uubb.bb The most noticeable failure cases are due to the errors of point cloud registration. The challenges of the registration methods come from noisy depth readings and partial point clouds. The error of the point cloud registration methods will lead to unexpected actions during the episode. In addition, it may end the episode early because the episode termination depends on the goal estimation. This motivates us to separate out the success criteria in Table 9 based on the failures the registration method.
github.io/.
On the action side, both the contact location and the motion parameters might have execution errors. Since the contact location selected from the observed point cloud, when the camera calibration is not accurate enough, the robot might not be able to reach the desired contact location of the object. In addition, since we use a compliant low-level controller to execute the motion parameters, the robot might not be able to execute the desired motion the same way in the simulation.
In addition, the object dynamics might be different from the simulation due the surface friction and the density of the object. The performance of our method could be further improved with domain randomization over the physical parameters.
E
More discussion non-prehensile manipulation
Non- Non-prehensile manipulation is an important aspect of the robot's capabilities, particularly in sce- narios where grasping encounters limitations, as demonstrated in Fig. 22. This section discusses the importance of non-prehensile actions in two key contexts:
Environment Occlusion. The first row of Figure 22 shows example where the potential grasp poses of the object are occluded by the wall. -oneeeeee moves, like nudging objects to better position, offer practical solution in such scenario.
Oversized Objects. The last two rows of Figure 22 include scenarios where certain dimensions of the object are larger than the width of the gripper.
F
More discussion on the related work
F.1 Compared Chen et al. [17, 18]
Our work is substantially different from Chen et al. [17, 18] from the follow aspects:
The approach in Chen et al. [17, 18] follows a student-teacher training pipeline. The teacher training is equivalent to the "No Contact Location" baseline with "states" observations in paper. The policy takes all the relevant robot state and object state information and outputs delta
Approach:

---

Figure 22: Examples showcasing limitations of prehensile manipulation. The frames where prehensile manipulation is challenging highlighted. The first row shows cube placed at the corner of the bin, where any grasp is obstructed by the bin wall. Both the second and third rows depict instances where objects too large be grasped specific poses.
robot actions. Note that they train single teacher policy across all shapes without using the point cloud which results in state-observattin policy that is "robust" to shapes instead of "adaptive" to shapes (see Discussion section in Chen et al. [18]). As shown in Table II, this baseline performs significantly worse than our method in our task because lacks shape information from the point cloud and the robot-centric action space is not as efficient our object-centric action space. On the other hand, although the student policy in Chen al. [17, 18] takes point cloud observation, trained using imitation learning from the teacher, so its performance is upper bounded by the teacher policy which has been shown to be worse than our proposed method.
We investigate completely different task and thus the numbers are not really comparable with the numbers from previous work [17. 18]. First, we use simple gripper instead dexterous hand. Second, we consider matching the orientation and position of the goal pose while Chen et al. [17, 18] only considers orientation.
Task:
F.2 Compared Cheng al. [1], Hou and Mason [2]
Unlike Cheng et al. [1]. Hou and Mason [2], our method not limited to a simplified gripper model and does not require the knowledge of object environment contact modes which are challenging to estimate during real robot execution. In addition, Cheng et al. [1]. Hou and Mason [2] are restricted by quasi-static assumptions. Although our method requires static point cloud observations in be- tween robot actions, the robot interaction with the object is not restricted to quasi-static motion. As shown in Figure 23, the flipping motion could be non-quasi-statte However, also point out that static point cloud observations may limit the method from more complex dynamic motions.
Figure 23: An example of non-quasi-static motion. The figure shows an example of executing the motion parameters flip mug upright. After the gripper pushes against the edge of the mug (first two images), it relies on the inertia of the mug to finish the motion which is not quasi-static (last two images).
25


Reviews:
[{"review_id": "leuGf5Lzr8U", "reviewer": "Reviewer_h5wU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper \"Hybrid Actor-Critic Maps for Manipulation (HACMan): Learning 6D Non-Prehensile Manipulation using Point Cloud Observations\" introduces a reinforcement learning approach for non-prehensile manipulation of objects. \nHACMan proposes a novel action representation for non-prehensile manipulation, which is temporally-abstracted and spatially-grounded. The action representation consists of selecting a contact location from the object's point cloud and a set of motion parameters describing the robot's movements after making contact. The authors modify an existing off-policy reinforcement learning algorithm to learn in this hybrid discrete-continuous action representation.\nComparisons with alternative action representations show that HACMan outperforms the best baseline by achieving a success rate more than three times higher.", "summary_of_recommendation": "Having carefully reviewed this work, I am pleased to express my strong acceptance of this paper. The authors present a compelling and innovative reinforcement learning approach that addresses the challenging problem of achieving 6D non-prehensile manipulation using point cloud observations. Their results are truly remarkable, showcasing an impressive 89% success rate on unseen objects and a commendable 50% success rate in real-world experiments with zero-shot transfer.\n\nThe paper is well-written, exhibiting a clear and concise presentation of the research, methodologies, and findings. Moreover, the authors have provided a complete and rigorously verified package of their hybrid actor-critic method, making it an excellent reference for the reinforcement learning community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I do not really have questions for this paper. However, please write more insights how can you tackle the problem of only 50% success rate in real-world experiment? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper titled 'Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation' presents a novel reinforcement learning approach for achieving 6D non-prehensile manipulation of objects by leveraging point cloud observations. The results of the study are highly promising, demonstrating an impressive 89% success rate on previously unseen objects and a noteworthy 50% success rate in real-world experiments using zero-shot transfer. Overall, the paper is well-written and effectively conveys its contributions.\n\nWhile the hybrid actor-critic method is not entirely new to the reinforcement learning community, this work delivers a comprehensive and thoroughly validated package of their approach, which serves as a valuable reference for the entire community. I have thoroughly reviewed all the equations presented in the paper, and they demonstrate a high level of technical soundness.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "iHVHOORVPq", "reviewer": "Reviewer_Qqvw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses non-prehensile object manipulation within an RL framework. The authors propose to define the input observation space of an actor-critic RL method to be surface points obtained from a point cloud observations. This way, both the Q function and the actor network are defined on the surface of an object. At inference time, the method selects both a contact location on the object surface and an action applied at this point to the object. Results in simulation and with a real robot indicate that defining the actor and Q function networks on the surface of objects is advantageous compared to multiple baselines.", "summary_of_recommendation": "As written in the review above, I encourage the authors to improve the presentation of the paper to clarify some parts of the method section.\nI am happy to raise my score if my points are addressed.\n\nPost rebuttal: raised score to strong accept", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, the idea of directly defining the Q values on the surface of an object is interesting and promising. Also, the way the authors propose to train the respective models is sound.\n\nMy main issue with this paper is that the method could be explained more clearly.\n\nThe actor is defined as a function that takes the point cloud as input and outputs a set of actions. I assume this is realized with the per-point features of a pointnet++ style architecture? Similarly for the Q function.\nThis raises the question why the authors call the approach \"hybrid\". The fact that the method selects from a discrete set of surface points is an implementation detail and not fundamental to the problem. For example, one alternative way of implementing those mappings is to compute a global point cloud feature and then input a \"query contact location\". This still is using contact locations from obtained point cloud, but the resulting Q function/actor would also be able to interpolate between points from the point cloud. A baseline testing this would be highly appreciated.\n\n\n\"The goal flow is a 3D vector which is concatenated to the other features of the point cloud observation\". I.e. each point in the point cloud is a 6 (or 7 dim?) vector? Maybe I missed it, what is the segmentation dim?\n\n\nSmall comments:\n- Please consider alternatives to \"hybrid\". This is not a hybrid problem.\n- Punctuation around equations could be improved (full stop after equation)\n- \"temporally-abstracted\" and \"spatially-grounded\" are wrongly used in this context in my opinion.\n- \"our method does not rely on quasi-static assumptions\" In my understanding, the method only takes as input observations of a single time slice, hence it by definition cannot properly reason about processes that involve inertia or second order dynamics. I agree that no explicit quasi-static assumptions are made in the method.\n\nRelevant other papers:\n- \"Neural Descriptor Fields: SE (3)-Equivariant Object Representations for Manipulation\", A. Simeonov et al.\n- \"Deep visual constraints: Neural implicit models for manipulation planning from visual input\", J-S. Ha et al.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "VXZHGyHhsZ", "reviewer": "Reviewer_sYWT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses a task where the robot must manipulate an object into a 6-DoF goal pose without grasping it. To address this, the proposed method incorporates a novel action representation which selects an interaction point and predicts a gripper movement to perform this interaction, and integrates this into an RL framework. The paper’s claims are extensively validated through simulation benchmarking, a range of ablation studies, qualitative visualisations, and real robot experiments.", "summary_of_recommendation": "This paper proposes a novel and sensible idea for action parameterization, and integrates it elegantly into an RL algorithm. The results are impressive, particularly for multi-step, difficult, non-planar manipulation sequences. The experiments also contain some valuable and surprising insights, e.g. on generalization to novel objects. The paper could be further improved through additional techniques which reduce the sim2real gap for this task, in order to improve real-world performance. Overall, this paper would be a great contribution to CoRL and will be widely appreciated by the community for its insights.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see the main review for questions for rebuttal.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths:\n1. This paper tackles a challenging and widely useful task: general 6-DoF manipulation without grasping.\n2. The chosen action representation is clever and novel, making it possible to learn this task with pure RL autonomously, without requiring demonstrations or multi-stage training as in prior work. This action representation, combined with this RL method, may have significant impact beyond non-prehensile manipulation.\n3. Training with a hybrid discrete and continuous action space is non-trivial, but this method resolves this in a neat way. Weighting the actor loss for the continuous action by the probability of selecting the discrete action is quite elegant and intuitive.\n4. The experiments are well-designed and answer interesting research questions. For example, a baseline which follows the reward greedily performs less well (Figure 4), showing the importance of the multi-step reasoning capabilities of this RL method. The Critic Map visualisations showing how the method captures multi-modal distributions (often difficult for alternative methods which use behavioral cloning to bootstrap from demonstrations) are great to see. Some of the results are quite surprising as well as impressive (e.g. on generalization to new objects).\n\n### Weaknesses:\n1. The paper may benefit from slightly stronger motivation for non-prehensile manipulation, compared to grasping approaches. Intuitively, I am persuaded by the paper’s Introduction which says that non-prehensile interaction is important for human dexterity, e.g. nudging an object into place when grasping is difficult due to clutter. However, the experiments in the paper use a clear workspace: that is, they do not yet show non-prehensile manipulation in the setting where it is most needed, compared with grasping. A small demo in a more cluttered scene where grasping fails but non-prehensile manipulation succeeds may help the paper to persuade more people in the community to build on this approach and work on this kind of manipulation.\n2. In the experiments, the paper uses a 3cm error margin to determine whether an episode succeeds. This is quite reasonable in general, but it would also be interesting to see exactly how precise this non-prehensile manipulation can be (e.g. by also reporting the median error, for one experiment). This is because humans usually use this non-prehensile manipulation for millimetre-level “nudging”, and so precision seems quite important for this approach.\n3. The approach uses only depth measurements, and no RGB, which may help with sim2real but also means that it may be difficult to get this to work for objects where depth measurements are poor, such as highly reflective or thin or transparent objects (as the authors discuss in the limitations section).\n4. Nevertheless, a sim2real gap remains for this method, likely due to differences in contact dynamics between the simulator and real objects. This leads to a drop in success rate (down to 40% success rate in the real world on non-planar goals). This is perhaps the main limitation / bottleneck in the method’s performance in the real world. However, the simulation performance and the complex beheviors exhibited are very impressive.\n\n### Minor issues:\n1. The phrase “temporally abstracted” is an intriguing concept but the link between the literal meaning of the words in the phrase and what it means in this context could be made clearer. If I understand correctly, temporally abstracted here means that the action is not tied to a specific timestep in seconds, since the robot can move around in free space for as long as it needs before reaching the right point and executing this contact action. So, an extra explanation along these lines which refers to timesteps could be useful in justifying the name “temporal abstraction”. Otherwise, the “temporal” nature of this concept is not so clear.\n2. One point which may benefit from a little more explanation for clarity is this: in the preliminaries, the policy pi_theta is deterministic (Line 109), but nevertheless during exploration a stochastic policy is still used as in Equation 4 (since this is an off-policy method). Just flagging this distinction (that a different, stochastic policy may still be used for exploration) in the Preliminaries may be helpful, when it is mentioned that the policy is deterministic.\n\n### Questions for rebuttal:\n1. Can this same method be used for prehensile manipulation? A single predicted contact point is already immediately applicable for suction grippers. Have the authors considered whether future work can extend this method to predicting contact locations for parallel-jaw gripper? This would be helpful to better assess the full impact of this work on future research.\n2. It is fortunate but surprising that as per Table 2, there is no performance drop for unseen object instances or even new categories. This strong generalization is particularly interesting because there are not that many objects in the training set (32). Intuitively, one might expect that the new test-time point clouds observed would be quite unlike anything seen by the point cloud networks during training, and so they might give strange predictions. Would the authors be able to shed some more light on what makes this possible? Is there an ablation study which could be conducted to further explore which components contribute more to this strong generalization performance?\n3. For future work, have the authors considered using this non-prehensile manipulation as a second stage after pick-and-place, in order to “nudge” an object into place very precisely?\n4. It seems like the paper focuses on object rearrangement tasks, with rigid objects. The method could theoretically be extended to articulated objects. What would be required to make this possible?\n5. The experimental setup seems suitable for quite sample-efficient, reset-free real world RL. Have the authors considered this approach? Maybe fine-tuning on real world episodes would help further close the sim2real gap.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "V2N05l34kK0", "reviewer": "Reviewer_3gnN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper examines the challenges inherent in non-prehensile robotic manipulation. The authors utilize an object-centric action representation to simplify the process by constraining the robot's actions to translational movements upon contact with the object. The robot selects a contact point and a set of motion parameters, which determine the trajectory post-contact, based on the observed object point cloud. This approach enables the robot to focus on learning the intricacies of the contact-rich segments of the action. This action representation is then integrated into an actor-critic reinforcement learning framework. The contact location is defined in a discrete action space and the motion parameters in a continuous action space. The actor-network generates continuous motion parameters on a per-point basis, while the critic network predicts per-point Q-values over the object point cloud.", "summary_of_recommendation": "This work offers substantial experimental details to demonstrate that the proposed problem representation performs effectively for tabletop single-object repositioning tasks. These implementation details will undoubtedly prove beneficial to the community. However, the current problem setup lacks novelty and offers limited insights that can be generalized to more complex tasks, as described in the Review section. Therefore, the reviewer deems this work as a borderline accept.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Is the feature f_i mentioned in line 187 just the goal flow discussed in sec 5.3? If so, it would be clearer to at least mention it in Sec.5.2.\n- The action space in the \"regress contact\" baseline constrains the end-effector's orientation to always point straight downward. The loss of certain degrees of freedom regarding the gripper orientation could potentially affect the robot's performance. Would allowing the policy to specify the gripper orientation in such a baseline action space provides a fairer comparison with the proposed action space?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n- This work specifically targets the repositioning of single objects via non-prehensile manipulation. The main contribution lies in exploring how the design of the action space influences data efficiency for such manipulation. The authors propose an action space that is temporally abstracted and spatially grounded. This means that a single action extends over various lengths of time and is associated with specific locations in physical space. Specifically, the authors suggest using an object-centric action representation to simplify the manipulation. This approach constrains the robot's actions to translational movements upon contact with the object. The authors also propose to represent the goal as a \"goal flow\" within the object's point cloud. This goal flow is determined by the difference between the initial point and the corresponding point in the goal point cloud. In the simulation experiment, the authors illustrated how each of these three components enhances the robot's performance in the task of non-prehensile manipulation of a single object. They also demonstrated how their proposed action and goal representations significantly bolster learning efficiency for 6D object repositioning tasks within a reinforcement learning process. The proposed concepts have been well executed and substantiated with extensive experimental results.\n\nWeakness:\n- A primary limitation of this work is its narrow focus on a very specific task space -- table-top single object repositioning. While the authors provide extensive experimental results proving the efficacy of their proposed action and goal representation for this particular task, these results aren't surprising. This is because the representations are specifically designed to simplify the learning problem in this context. Consequently, the paper does not present sufficient evidence to support the claim made in line 338, which suggests that this action space could be extended to a wider variety of manipulation.' The reviewer posits that the assumptions inherent in the proposed action space might even lead to decreased performance in other more complex manipulation tasks. For instance, in a multi-object repositioning scenario, an action not spatially grounded to a single target object could potentially move two or more non-contacting neighboring objects toward their goal poses in a single action, thereby reducing the total control effort.\n- The reviewer finds the most intriguing aspect of this paper to be the exploration of how a temporally-abstracted and/or spatially-grounded action space can improve learning efficacy. However, while the action space proposed in the study exemplifies these characteristics, it does not fully represent them. Future research could benefit from comparing different action spaces that incorporate either or both of these traits. For instance, it would be interesting to investigate what might happen if the temporally-abstracted action involved a more complex set of carefully designed parameterized motion primitives.\n- Without a comparison to other baseline methods, the real-robot experiment offers limited useful information. Considering the relatively high tolerance for imprecision in the success condition (i.e., an average distance of 3 cm between corresponding points of the object and the goal), a 70% success rate on planar goals and a 40% success rate on non-planar goals is not particularly impressive or informative. The experiment would have been more interesting if it had compared the impact of the proposed action space on zero-shot transfer to at least one baseline method.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the paper is well-written, receives three Strong Accept and one Weak Accept reviews. The rebuttal has addressed major concerns and issues. Please revise the paper in line with the rebuttal discussion for the camera-ready submission.

**Strengths**:

- The paper's writing is clear and effective.

- The core idea of directly defining Q values on contact points holds interest and promise.

- The combination of this novel action representation with the RL method may extend its influence beyond non-prehensile manipulation.

- The experiments are well-designed and the results are impressive, such as the generalization capability to unseen objects.

- The proposed critic map representation showcases an ability to handle multi-modal distributions.

**Weaknesses**:

- The central notion of defining Q values on an object's surface is not novel, has precedent in related literature, such as Contact-Graspnet for grasping, and the paper [20].

- Further clarity is needed to discern the primary contributors to the strong generalization performance with unseen objects.

- As outlined in the limitation section, the approach's reliance on depth measurements and the assumption of precise goal point tracking may restrict its applicability in real-world robotic scenarios.
Decision: Accept (Oral)