DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation
Authors: Yan Zhao
Ruihai Wu
Zhehuan Chen
Yourong Zhang
Qingnan Fan
Kaichun Mo
Hao Dong
Venue: ICLR 2023 poster (2023)
Keywords: Visual Actionable Representation for Robotics
Visual Understanding of 3D Shapes

Abstract: It is essential yet challenging for future home-assistant robots to understand and manipulate diverse 3D objects in daily human environments. Towards building scalable systems that can perform diverse manipulation tasks over various 3D shapes, recent works have advocated and demonstrated promising results learning visual actionable affordance, which labels every point over the input 3D geometry with an action likelihood of accomplishing the downstream task (e.g., pushing or picking-up). However, these works only studied single-gripper manipulation tasks, yet many real-world tasks require two hands to achieve collaboratively. In this work, we propose a novel learning framework, DualAfford, to learn collaborative affordance for dual-gripper manipulation tasks. The core design of the approach is to reduce the quadratic problem for two grippers into two disentangled yet interconnected subtasks for efficient learning. Using the large-scale PartNet-Mobility and ShapeNet datasets, we set up four benchmark tasks for dual-gripper manipulation. Experiments prove the effectiveness and superiority of our method over three baselines. We will release code and data upon acceptance. 

Published as a conference paper at ICLR 2023
DUALAFFORD: LEARNING COLLABORATIVE VISUAL AFFORDANCE FOR DUAL-GRIPPER MANIPULATION
Yan Zhao! 1.6
Ruihai Wu Zhehuan Chen'
Yourong Zhang'
Qingnan Fan
Kaichun Mo
Hao Dongl.2.6
1,2,6t
CCFCS, CS Dept., PKU PAIIT, PKU
Stanford University
*NVIDIA Research
'Tencent AI Lab 'BAII
(zhaoyan790, wuruihai, acmlezh, minor_miiniiiieee hao. dong) y 0pku. edw..edduucoo en fanchina gma i comm kmo@nvidia.com
ABSTRACT
is essential yet challenging for future home-assistant robots to understand and manipulate diverse 3D objects in daily human environments. Towards building scalable systems that can perform diverse manipulation tasks over various 3D shapes, recent works have advocated and demonstrated promising results learning visual actionable affordance, which labels every point over the input 3D geometry with an action likelihood of accomplishing the downstream task (e.g., pushing or picking-up). However, these works only studied single-gripper manipulation tasks, yet many real-world tasks require two hands to achieve collaboratively. In this work, we propose novel learning framework, DualAfford, to learn collaborative affordance for dual-gripper manipulation tasks. The design of the approach is to reduce the quadratic problem for two grippers into two disentangled yet intercon- nected subtasks for efficient learning. Using the large-scale PartNet-Mobille and ShapeNet datasets, we set up four benchmark tasks for dual-gripper manipulation. Experiments prove the effectiveness and superiority of our method over baselines.
INTRODUCTION
We, humans, spend little or no effort perceiving and interacting with diverse 3D objects to accomplish everyday tasks in our daily lives. It is. however, an extremely challenging task for developing artificial intelligent robots to achieve similar capabilities due to the exceptionally rich 3D object space and high complexity manipulating with diverse 3D geometry for different downstream tasks. While researchers
have recently made many great advances in 3D shape recognition (Chang et al. 2015: Wu et al.
2015), pose estimation (Wang et al.
2019 Xiang et al. 2017), and semantic understandings (Hu et 2018 Mo et al. 2019) Savva et al. 2015) from the vision community, as well as grasping (Mahler
al. 2019 Pinto& Gupta 2016) and manipulating 3D objects (Chen al.) 2021 Xu et | 2020) on the robotic fronts, there are still huge perceptioon-inteactiin
gaps (Batra al. 2020 Gadre et al.
2021 [hen et al. 2021 Xiang et al., 2020) to close for enabling future home-assistant autonomous systems in the unstructured and complicated human environments.
One of the core challenges in bridging the gaps is figuring out good visual representations of 3D objects that are generalizable across diverse 3D shapes at large scale and directly consumable by downstream planners and controllers for robotic manipulation. Recent works (Mo et al., 2021 Wu
al. 2022) have proposed novel perception-interaction handshaking representation for 3D objects visual actionable affordance, which essentially predicts an action likelihood for accomplishing the given downstream manipulation task at each point on the 3D input geometry. Such visual actionable affordance, trained across diverse 3D shape geometry (e.g., refrigerators, microwaves) and for a specific downstream manipulation task (e.g., pushing), is proven to generalize to novel unseen objects (e.g., tables) and benefits downstream robotic executions (e.g., more efficient exploration).
Though showing promising results, past works (Mo et al. 2021 Wu al.) 2022) are limited to engle-gripper manipulation tasks. However, future home-assistan robots shall have two hands just like us humans, if not more, and many real-world tasks require two hands to achieve collaboratively. For example (Figure []. to steadily pick up heavy bucket, two grippers need to grasp it at two top
'Equal contribution
I Cooresponning author; Project page: httpss://ypprrpparee-lbb
io/Dual Affor

---

Published as a conference paper at ICLR 2023
0.25
0,5
0.75
(b)
Push
Topple
(4)
Rotate
Pick-up
(a)
bb)
Real-world
Pick-up
Data
Figure 1: Given different shapes and manipulation tasks (e.g., pushing the keyboard in the direction indicated by the red arrow), our proposed DualAfford framework predicts dual collaborative visual actionable affordance and gripper orientations. The prediction for the second gripper (b) dependent on the first (a). We can directly apply our network to real-world data.
edges and move in the same direction; to rotate a display anticlockwise, one gripper points downward to hold and the other gripper moves the other side. Different manipulation patterns naturally emerge when the two grippers collaboratively attempt to accomplish different downstream tasks.
In this paper, we study the dual-gripper manipulation tasks and investigate learning collaborative visual actionable affordance. It is much more challenging to tackle dual gripppr manipulation tasks than single-gripe gripper ones as the predictions are required due to the addition of the second gripper. Besides, the pair of affordance maps for the two grippers needs be learned collaboratively. As we can observe from Figure [ the affordance for the second gripper is dependent on the choice of the first gripper action. How to design the learning framework to learn such collaborative affordance a non-trivial question.
offrreooo- in action spaces is doubled and two affordance
We propose novel method DualAfford to tackle the problem. At the core of our design, DualAfford disentangles the affordance learning problem of two grippers into two separate yet highly coupled subtasks, reducing the complexity of the intrinsically quadratic problem. More concretely, the first part of the network infers actionable locations for the first gripper where there exist second-gin -giipeee actions cooperate, while the second part predicts the affordance for the second gripper conditioned on a given first-gripper action. The two parts of the system are trained as a holistic pipeline using the interaction data collected by manipulating diverse 3D shapes in physical simulator.
We evaluate the proposed method on four diverse dual-gripper manipulation tasks: toppling and picking-up. We set up benchmark for experiments using shapes from PartNet- dataset (Mo et al. 2019 Xiang et al. 2020) and ShapeNet dataset (Chang et al. 2015). Quantitative comparisons against baseline methods prove the effectiveness of the proposed framework. Qualitative results further show that our method successfully learns interesting and reasonable dual-gripper collaborative manipulation patterns when solving different tasks. To summarize, in this paper,
pushing, rotating,
We propose a novel architecture DualAfford to learn collaborative visual actionable affor- dance for dual-grriped manipulation tasks over diverse 3D objects:
We set up benchmark built upon SAPIEN physical simulator (Xiang et al. 2020) using the
PartNet-Mobility and ShapeNet datasets (Chang al. 2015 Mo et al.) 2019 Xiang et al.)
2020> for four dual-gripper manipulation tasks;
We show qualitative results and quantitative comparisons against three baselines to validate the effectiveness and superiority of the proposed approach.
2 RELATED WORK
Dual-gripper Manipulation. Many studies, from both computer vision and robotics communities, have been investigating dual-gripper or dual-arm manipulation (Chen et al. 2022 Simeonov et al.
2020 Weng et al. 2022 Chitnis al. 2020 Xie et al. 2020 Liu & Kitani
2021 L et al. 2022)
2

---

Published as a conference paper at ICLR 2023
Vahrenkamp et al. (2009) presented two strategies for dual-arm planning: J+ and IK-RRT. Cohen et al. (2014) proposed a heuristic search-based approach using a manipulation lattice graph. Ha et al. (2020) presented closed-loop and decentralized motion planner to avoid collision. Multi-arm
manipulation has also been investigated in various applications: grasping (Pavlichenko et al. 2018). pick-and-place (Shome& & Bekriss 2019), and rearrangement (Shome al. [021]] Harrtannn et al.) 2021). Our work pays more attention to learning object-centric visual actionable affordance heatmaps for dual-arm manipulation tasks, while previous works focus more on the planning and control sides.
Gadre et al. (2021) learns affordances but for interactively part segmentation, they use one gripper to simply hold one articulated part, and use the other gripper to move the other articulated part.
Visual Affordance Prediction. Predicting affordance plays an important role in visual under- standing and benefits downstream robotic manipulation tasks, which has been widely used in many
previous works (Jiang al. 2021b Kokic al. 2017 2020] Mandikal & Grauman, 202T; Redmon & Angelova 2015; Wang et al. 2021 Wu et al. 2022) For example, Kokic et al. (2017) used CNN
to propose binary map indicating contact locations for task-specific grasping. Jiang et all (2021a) proposed the contact maps by exploiting the consistency between hand contact points and object contact regions. Following Where2Act (Mo et al. 202T), we use dense affordance maps to suggest action possibilities at every point 3D scan. In our work, we extend by learning two collaborative affordance maps for two grippers that are in deep cooperation for accomplishing downstream tasks.
3 PROBLEM FORMULATION
General Setting, We place a random 3D object from a random category on the ground, given its partially scanned point cloud observation
RR and specific task the network required to propose two grippers actions #1 (p). R1) and N2 = (p2, R2). in which p is the contact point and R the manipulation orientation. All inputs and outputs are represented in the camera base coordinate frame, with the z-axis aligned with the up direction and the x-axis points to the forward direction, which is in align with real robot's s camera coordinate system.
Task Formulation. We formulate four benchmark tasks: pushing, rotating, toppling and picking-up which are widely used in manipulation benchmarks (Andrychowicz et al. [017 Kumar et al. 2016 Mousavian et al., 2019 OpenAl et al. 202T) and commonly used as subroutines in object grasping and relocation (Chao et al. 2021 Mahler et al.) 2019 Mandikal & Grauman 2022 Rajeswaran et al. 2018, Zeng et al.] | 2020). We set different success judgments for difference tasks, and here we describe the pushing task as example. Task E IR' is unit vector denoting the object's goal pushing direction. An object is successfully pushed (1) its movement distance is over 0.05 unit-length, (2) the difference between its actual motion direction I' and goal direction is within 30 degrees, (3) the object should be moved steadily, i.e., the object can not be rotated or toppled by grippers.
4
METHOD
4.1 OVERVIEW OF DualAfford FRAMEWORK
presents the overview of our proposed DualAfford framework. Firstly, we collect large amount of interaction data supervise the perception networks. Since it is costly to collect human annotations for Sual-gripper manipulations, we use an interactive simulator named SAPIEN Xiang et al. 2020). We sample offline interactions by using either a random data sampling method or an optional reinforcement-leanee learning (RL) augmented data sampling method described in Sec. 4.5
Figure 2
We propose the novel Perception Module to learn collaborative visual actionable affordance and interaction policy for dual-gripper manipulation tasks over diverse objects. To reduce the complexity of the intrinsically quadratic problem of dual-gripper manipulation tasks, we disentangle the task into two separate yet highly coupled subtasks. Specifically, let N denote the point number of the point cloud, and 0g denote the gripper orientation space on one point. If the network predicts the two gripper actions simultaneously, the combinatorial search space will be (((O)) )((0)(NxN) (Nxx)). However, our Perception Module sequentially predicts two affordance maps and gripper actions in a conditional manner, which reduces the search space to ((RR) Therefore, we design two coupled submodules in the Perception Module: the First Gripper Module Me (left) and the Second Gripper Module 22 (right), and each gripper module consists of three networks (Sec. 4.2)
The training and inference procedures, respectively indicated by the red and blue arrows in Figure 2 share the same architecture but with reverse dataflow directions. For inference, the dataflow direction is intuitive: M, proposes H1, and then Mll proposes U2 conditioned on U1 Although such dataflow guarantees the second gripper plays along with the first during inference, it cannot guarantee the first.
3

---

Published as a conference paper at ICLR 2023
Perception Module
Environment
Training
Update
Physies Simulation
Collaborative Adaptation
Gripperl Module
Gripper2 Module
Self-supervised Efficient
Collection (Optional RL)
Real World
Predict
Inference
Figure 2: Our proposed DualAfford framework, first collects interaction data points in physics simulation, then uses them to train the Perception Module, which contains the First Gripper Module and the Second Gripper Module, and further enhances the cooperation between two grippers through the Collaborative Adaption procedure. The training and the inference procedures, as respectively indicated by the red and blue arrows, share the architecture but with opposite dataflow directions.
gripper's s action is suitable for the second to collaborate with. To tackle this problem, for training. we employ the reverse dataflow: Mr trained first, and then An is trained with the awareness of the trained- Mr.. Specifically, given diverse in training dataset, MM2 is first trained to propose u2 collaborative with them. Then, with the trained Mr able propose иą collaborative with different "1, 4 learns to propose ii that are easy for Mr to propose successful collaborations. In this way, both 14 and Mr are able propose actions easy for the other to collaborate with.
Although such design encourages two grippers to cooperate, the two gripper modules are separately trained using only offline collected data, and their proposed actions are never truly executed whole, so they are not explicitly taught if their collaboration is successful. To further enhance their cooperation, we introduce the Collaborative Adaptation procedure (Sec. (44), in which we execute two grippers' actions simultaneously in simulator, using the outcomes to provide training supervision.
4.2 PERCEPTION MODULE AND INFERENCE.
To reduce the complexity the intrinsically quadratic problem and relieve the learning burden of our networks, we disentangle the dual-gripper learning problem into two separate yet coupled subtasks. We design a conditional perception pipeline containing two submodules shown in Figure u2 is proposed conditioned #1 during inference, while Mn is trained conditioned on the trained Mr during training. There are three networks in each gripper module: Affordance Network s Proposal Network @ and Critic Network &. First, the gripper action can be decomposed into contact point and a gripper orientation, we design Affordance Network and Proposal Network to respectively predict them. Also, to evaluate whether action of the gripper is suitable for collaboration, design Critic Network for this purpose. Below we describe the design of each module.
which
Backbone Feature Extractors. The networks in Perception Module may receive four kinds of input entities or intermediate results: point cloud O, task I, contact point and gripper orientation R. different submodules, the backbone feature extractors share the same architectures. We use a segmentation-vertt PointNet++ (Qi al. 2017) to extract per-poin point feature fs E IR 128 from O, and employ three MLP networks to respectively encode I, p, and into fi E 323 Sp E R.32 and fR E 32
4.2.1 THE FIRST GRIPPER MODULE
The First Gripper Module contains three sequential networks. Given object and task configuration, the Affordance Network si indicates where to interact by predicting affordance map. the Proposal Network 9 suggests how interact by predicting manipulation orientations, and the Critic Network '6, evaluates the per-action success likelihood.
Affordance Network. This network o predicts an affordance score ai E [0, for each point p. indicating the success likelihood when the first gripper interacts with the point, with the assumption that there exists an expert second gripper collaborating with it. Aggregating the affordance scores, acquire affordance map A₁ over the partial observation, from which
filter out low-rated
4

---

Published as a conference paper at ICLR 2023
afforcance mapi
point
affordance map2
fs
1
select pr
First Gripper
seleci po
Module
tack f
fn
g 2
select R
seleec
several R
fe
62
First Gripper Module
Second Gripper Module
Figure 3: Architecture details of the Perception Module. Given 3D partial scan and a specific task, network sequentially predicts the first and second grippers' affordance maps and manipulation actions in a conditional manner. Each gripper module is composed of 1) an Affordance Network of indicating where to interact; 2) Proposal Network P suggesting how to interact; 3) Critic Network & evaluating the success likelihood of an interaction.
proposals and select contact point Pi for the first gripper. This network implemented as single-layer MLP that receives the feature concatenation of fi and fp..
Proposal Network. This network P models the distribution of the gripper's orientation R on the given point p1. is implemented as a conditional variational autoençoder (Sohn et al.) 2015). where encoder maps the gripper orientation into a Gaussian noise z R.32 decoder reconstructs it from 2. Implemented as MLPs, they both take the feature concatenation of fs. fi, and fp, as the condition.
Critic Network. This network 6, rates the success likelihood of each manipulation orientation on each point by predicting scalar Cι [0, 1]. A higher ej indicates higher potential for the second gripper to collaboratively achieve the given task. It is implemented as single-layer MLP that consumes the feature concatenation of fs. fI. Sp, and fRJ-
4.2.2 THE SECOND GRIPPER MODULE
Conditioned on the first gripper action 41 = (p1,R1) proposed by MI. first generates point- level collaborative affordance A for the second gripper and samples contact point p2. Then, Mr proposes multiple candidate orientations, among which we can choose a suitable one as R2. The design philosophy and implementations of .12 the (oh2 P and (6) take the first gripper's action (j, ie., and R1, as the additional input.
Mi, except that all three networks
4.3 TRAINING AND LOSSES
As shown in Figure 2 during inference (indicated by blue arrows), the first gripper predicts actions without seeing how the second gripper will collaborate. enable the first gripper to propose actions easy for the second to collaborate with, train the Perception Module in the dataflow direction indicated by red arrows, as described Sec. 4.1 We adopt the Critic Network E of the first gripper bridge connect two gripper modules. '6 scores whether action of the first gripper is easy for .12 to propose collaborative actions. With the trained '6, Mi will propose actions with the assumption that there exists an expert gripper to cooperate with. Therefore, Mn and Ml will both learn to collaborate with each other.
Critic Loss. It relatively easy to train the second Critic Network E2. Given the interaction data with the corresponding ground-truth interaction result r, where r means positive and 0 means negative, we can train '62 using the standard binary cross-entropy loss. For simplicity, we use fr to denote each network's input feature concatenation. as mentioned in Sec 4.2
r = (o0 (2(( - (1-1 (orr))o)o(11 62 ())))
However, for the first Critic Network 6,, since only know the first gripper's action иi R1). we can not directly obtain the ground-trute interaction outcome of single action i1. To tackle this problem, given the first gripper's action, evaluate it by estimating the potential for the second
5

---

Published as a conference paper at ICLR 2023
Network Pg
gripper2 orientations
Network 61 Q: How good is (p, R,)?
Network 2n gripper2 affordance
R
R 3/
Pat
p.
Pap P21 pm Pio
Raat
R
Pis
Network 62 evaluate the score 22,1300 2zww and calculate the mean
Figure To train that evaluates how the first action can collaborate with the trained Second Gripper Module M22 we comprehensively use the trained on 9 and 62 of M2 to provide supervision.
gripper to collaboratively accomplish the given task. As shown in Figure 4 we comprehensively use the trained 2 2 and '62 of the Second Gripper Module Mr. Specifically, to acquire the ground- truth action score for the first gripper, we first use s to predict the collaborative affordance map A2 and sample n contact points: P2,1, P2,- then we use P to sample m interaction orientations Finally,
each contact point i: R2. (2.,1 C2.m and calculate their average value. Thus we acquire the ground-truth score of 6,, and apply Li loss measure the error between the prediction and the ground-truth:
use 62 to rate the scores of these actions:
m ΣΣ
êpp
(f 62 pay: f
P2 zR));
Le, = ((() Ep.l-
(2)
Proposal Loss. P and 92 are implemented eVE (Sohn al. 2015). For the i-th gripper, we apply geodesic distance loss to measure the eπror between the reconstructed gripper orientation R and ground-truth R; and KI. Divergence to measure the difference between two distributions:
Lg = geoo (R, R.) + DxI. [ (l, 1)).
Affordance Loss. Similar to Where2Act (Mo et al.) 2021), for each point, we adopt the "affordance" score as the expected success rate when executing action proposals generated by the Proposal Network P. which can be directly evaluated by the Critic Network & Specifically, to acquire the ground-trunt truth affordance score à for the i-th gripper, we sample n gripper orientations on the point p: using P. and calculate their average action scores rated by '6 We apply Li loss to measure the error between the prediction and the ground-truth affordance score on certain point:
åp:
La (( ap:
4.4 COLLABORATIVE ADAPTATION PROCEDURE
Although the above training procedure can enable two gripper modules to propose affordance and actions collaboratively, their collaboration is limited, because they trained in separate and sequential way using only offline collected data, without any real and simultancous executions of proposed actions. To further enhance the collaboration between the two gripper modules, we introduce the Collaborative Adaptation procedure, in which the two modules are trained in a simultancous manner using online executed and collected data, with loss functions the same as in Sec. 4.3 this procedure, the proposed dual-grips gripper actions are simultaneously executed in the simulator, using interaction outcomes to update two gripper modules. this way, the two gripper modules can better understand whether their proposed actions successful or not they are aware of interaction results, and thus the two separately trained modules are integrated into one collaborative system.
4.5 OFFLINE DATA COLLECTION
Instead of acquiring costly human annotations, we use SAPIEN (Xiang et al. 2020) to sample large amount of offline interaction data. For each interaction trial with each object, sample two gripper actions in 42, and test the interaction result r. We define a trial to be positive when: (1) the two grippers successfully achieve the task, e.g., pushing a display over threshold length without rotating it; (2) the task can be accomplished only by the collaboration of two grippers, i.e., when we replay each gripper action without the other, the task not be achieved. We represent each interaction data as (0.1, PI. m.PP P2, R2) + r. and balance the number positive and negative interactions. Here we introduce data collection methods: random and RL augmented data sampling.

---

Published as a conference paper at ICLR 2023
Train Categories
0.25
Test Categories
Pun
Topple
Pickcanp
Figure 5: Qualitative results of Affordance Networks. In each block, we respectively show (1) task represented by a red arrow, (2) object which should be moved from transparent to solid, (3) the first affordance map predicted by .41 (4) the second affordance map predicted by Aa22 conditioned on the first action. Left shapes are from training categories, while right shapes are from unseen categories.
Random Data Sampling. We can efficiently sample interaction data by parallelizing simulation environments across multiple CPUs. For each data point, we first randomly sample two contact points on the object point cloud, then randomly sample two interaction orientations from the hemisphere above the tangent plane around the point, and finally test the interaction result.
RL Augmented Data Sampling. For tasks with complexity, such as picking-up, is nearly impossible for random policy to collect positive data. To tackle this problem, we propose the RL method. We first leverage Where2Act (Mo et al.] 202T) to propose a prior affordance map, highlighting where to grasp. After sampling two contact points, use SAC (Haarnoja al. (018) with the manually designed dense reward functions to efficiently predict interaction orientations.
EXPERIMENTS
5.1 RESULTS AND ANALYSIS
We perform large-scale experiments under four dual-gripper manipulation tasks, and set up three baselines for comparisons. Results prove the effectiveness and superiority of our proposed approach.
5.2
ENVIRONMENT SETTINGS AND DATASET
We follow the environment settings of Where2Act (Mo et al. 2021 except that we Panda Flying grippers. We conduct our experiments on SAPIEN (Xiang et al.) (020) simulator with the large-scale PartNet-Mobility (Mo al. 2019) and ShapeNet (Chang et al. (0015) dataset. To analyze whether the learned representations can generalize to novel unseen categories, we reserve some categories only for testing. See Supplementary See, C for more details.
Franka
5.3 EVALUATION METRICS, BASELINES AND ABLATION
Evaluation Metrics. To quantitatively evaluate the action proposal quality, we run interaction trials in simulation and report sample-success-e (Mo al. 2021), which measures the percentage of successful interactions among all interaction trials proposed by the networks.
Baselines. We compare our approach with three baselines and one ablated version: (1) random approach that randomly selects the contact points and gripper orientations. (2) A heuristic approach in which we acquire the ground-true truth object poses and hand-engineer a set rules for different tasks. For example, for the picking-up up task, right top edges and set the two gripper orientations the same as the given picking-up direction. (3) M-Where2Act: dual-gripper Where2Act (Mo al. 2021 approach. While Where2Act initially
the contact points the objects' left and

---

Published as a conference paper at ICLR 2023
Train Categories
Test Categories
Rotate
Topple
Pick-uip
I
Figure 6: The per-point action scores predicted by Critic Networks E1 and E2. In each result block, from left to right, we show the task, the input shape, the per-point success likelihood predicted by 'i given the first gripper orientation, and the per-point success likelihood predicted by '62 given the second gripper orientation, conditioned on the first gripper's 's action.
considers interactions for a single gripper, we adapt it as a baseline by modifying each module in Where2Act consider the dual grippers combination, and assign task to it as well. (4) Ours w/o CA: an ablated version of our method that removes the Collaborative Adaptation procedure.
Figure presents the dual affordance maps predicted by our Affordance Networks s4 and 2/2 as well as the proposed grippers interacting with the high-rated points. We can observe that: (1) the affordance maps reasonably highlight where interact (e.g., for picking-up, the grippers can only grasp the top edge); (2) the affordance maps embody the cooperation between the two grippers (e.g., collaboratively push display, the two affordance maps sequentially highlight its left and right half part, that the display can be pushed steadily.) Besides, we find that our method has the ability to generalize to novel unseen categories.
In Figure we additionally visualize the results of Critic Networks 11 and 62. Given different gripper orientations, the Critic Networks propose the per-point action scores over the whole point cloud. We can observe that our network is aware of the shape geometries, gripper orientations and tasks. For example, in the Rotate- Train-Catee -atte-Crteesiiess block, the first map highlights part chair surface since the first gripper is downward, and the second map accordingly highlights the chair back on the other side given the second -ripperr orientation, which collaboratively ensures the chair is rotated clockwise. It noteworthy that in the first map the chair surface has higher scores than the arm, because the chair tends to skid when selecting the arm as a fulcrum for rotation.
Figure (a) visualizes the diverse collaborative actions proposed by Proposal networks 9 and P on an example display. Our networks can propose different orientations on the same points.
Table I: Baseline comparison on the sample-success-radl metric.
Train Categories
Test Categories
pushing
rotating
toppling
picking up
pushing
rotating
toppling
picking-up
6.00
Heuristic
32.40
24.20
54.00
31.93
25.80
21.80
38.00
37.90
M-Where2Act
28.00
15.67
36.60
5.00
23.40
10.67
25.60
13.80
Ours CA
17.53
56.00
28.87
15.33
Ours
48.76
33.73
65.53
40.33
42.93
35.07
41.80
54.33
Table presents the sample-success-ee different methods over the four challenging tasks. We can that method outperforms three baselines over all comparisons.
8

---

Published as a conference paper at ICLR 2023
6.25
0,5
0.75
(8)
(c)
w/l CA
CA
Fash
Push
Retste
(b)
Tupple
Rolace
Pick-up
Figure 7: (a) The diverse and collaborative actions proposed by the Proposal Networks P and S2 (b) The promising results testing on real-world data. (c) The actionable affordance maps of the ablated version that removes the Collaborative Adaptation procedure (left) and ours (right).
For the heuristic baseline, it gains relatively high numbers since it proposes actions with the ground object poses and orientations. However, the inter- and intra-category shape geometries exceptionally diverse, and Moreover, in the real world, this approach needs more effort to acquire ground-truth information.
not expect the hand-engineered rules to work for all shapes.
For M-Where2Act, it learns the dual contact points and orientations performance. comparison, our method disentangles the collaboration learning problem and reduces the complexity. Besides, M-Where2Act consumes nearly quadratic time give proposals for the reason that it has to query the affordance scores of all the + pair combinations of points.
combination and has worse
For Ours w/o CA, this ablated version of method shows that the Collaborative Adaption procedure helps boost the performance. Figure 7 (c) visualizes the affordance maps without (left) and with (right) Collaborative Adaption procedure. We find that the affordance maps become more refined. For example, to push the display, the affordance scores of the base become lower since it is difficult interact with; collaboratively topple the dishwasher, in the second affordance map, the left front surface receives lower scores while the right maintains relatively higher.
Table 2 shows the success rate of the Random Data Sampling and RL Augmented Data Sampling method. The RI. method significantly improves data collection efficiency on each object category.
Figure I and Figure (b) show qualitative results that our networks can directly transfer the learned affordance to real-world data. We show more real-robot experiments in supplementary Sec.
Table The success rate of data collection in the picking-up task.
Train Categories
Test Categories
eyeglasses
bucket
trash can pliers
basket
display
box
kitchen pul scissors
laptop
Saaudom-Sampiing
0.06
0.12
0.04
0.09
0.03
|0.03
0.06
0.01
0.01
RL-Sampling
6.12
9.65
5.26
5.79
6.41
9.78
5.12
9.18
6.38
7.13
CONCLUSION
In this paper, we proposed a novel framework DualAfford for learning collaborative actionable affordance for dual-ge dual-gripper manipulation diverse 3D shapes. We set up large-scale scale benchmarks for four dual- dual-gripper manipulation tasks using the PartNet-Mon Mobiilty and ShapeNet datasets. Results proved the effectiveness of the approach and its superiority over the three baselines.

---

Published as conference paper at ICLR 2023
Acknowledgements. Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Hao Dong were supported by National Natural Science Foundation of China (No. 62136001).
Ethics Statements. Our project helps build future home-assistant robots. The visual affordance predictions be visualized and analyzed before use, which is less vulnerable to potential dangers. The training data used may introduce data bias, but this a general concern for similar methods. We do not any particular major harm issue work may raise up.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Dhruv Batra, Angel Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied arXiv preprint arXiv:2011.0197. 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv::512.03012, 2015.
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: benehmark for capturing hand grasping of objects. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning, 2021.
Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Hao Dong, Zongqing Lu, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning, 2022.
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manip- ulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1149-1155. IEEE, 2020.
Sachin Chitta, Joan Sucan, and Steve Cousins. Moveit! IEEE Robotics & Automation Magazine, 19 (1):18-19, 2012.
Benjamin Cohen, Sachin Chitta, and Maxim Likhachev. Single- and dual-arm motion planning with heuristic search. The International Journal of Robotics Research, 33(2):305-320, 2014. doi: 10.1 10.1177702233369915777733 URL https::::: /ttppp:://oo.... //ooi///./// www.orggg./..... 1177 1111111777777949999
Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/VV International Conference on Computer Vision (ICCV), pp. 15752-15761, October 2021.
Huy Ha, Jingxi Xu, and Shuran Song. Learning a decentralized multi-arm motion planner. In Conference Robotie Learning (CoRL), 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/ Mss/1812.05900 2018. URL
org. wtt.:aawwwiv..aaaa////aaaaaaaa//
05905
Valentin Noah Hartmann, Andreas Orthey, Danny Driess, Ozgur S. Oguz, and Mare Toussaint. Long- horizon multi-robol rearrangement plamning for construction assembly. CoRR, abs/2106.02489, 2021. URL htt.httts://// htttps://arxttit
06. /2486 02489
10

---

Published as a conference paper at ICLR 2023
Ruizhen Hu, Manolis Savva, and Oliver van Kaick. Functionality representations and applications for shape analysis. In Computer Graphics Forum, volume 37, pp. 603-624. Wiley Online Library, 2018.
Hanwen Jiang. Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. CoRR, abs/2104.03304, 2021a. URL buttps://arx:: org/ abs 22///11/ 04.044.4. 03304
Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. CoRR, abs/2104.01542, 202 URL htt.httts///xxii.coomm hbwpg///bb///....../// 01542
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Mia Kokic, Johannes A Stork, Joshua A Haustein, and Danica Kragic. Affordance detection for task-specific grasping using deep learning. In 2017 IEEE-RAS 17th International Conference Humanoid Robotics (Humanoids), pp. 91-98. IEEE, 2017.
Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352-3359) 2020. doi: 10. 109/LRA.2020 2975706.
Vikash Kumar, Abhishek Gupta, Emanuel Todorov, and Sergey Levine. Learning dexterous manipu- lation policies from experience and imitation. arXiv preprint arXiv: 1e/.:1/1110000 1.05095, 2016.
Junjia Liu, Yiting Chen, Zhipeng Dong, Shixiong Wang, Sylvain Calinon, Miao Li, and Fei Chen. Robot cooking with stir-fry: Bimanual non-prehensile manipulation of semi-fluid objects. IEEE Robotics and Automation Letters, 7(2):515--5166, 2022.
Xingyu Liu and Kris M Kitani. V-mao: Generative modeling for multi-arm manipulation of articulated objects. In 5th Annual Conference on Robot Learning. 2021.
Jeffrey Mahler, Matthew Matl, Vishal Satish, Michael Danielezuk, Bill DeRose, Stephen McKinley, and Ken Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26):
eaau4984, 2019.
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6169-6176, 2021.
doi: 10.1 00.100///RA44450622222
802.
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. CoRR, abs/2202.00164, 2022. URL https: org. abs//
2202.00164
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical pert-level 3d object understanding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 909-918, 2019.
Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6813-6823, October 2021.
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2901-2910, 2019.
OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, 1lge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique P Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv: 001.04822 2021.
11

---

Published as a conference paper at ICLR 2023
Dmytro Pavlichenko, Diego Rodriguez, Max Schwarz, Christian Lenz, Arul Selvam Periyasamy, and Sven Behnke. Autonomous dual-arm manipulation of familiar objects. In 2018 IEEE- RAS 18th International Conference on Humanoid Robots (Humanoids), pp. 1-9, 2018. doi: 10.1 1A110//HAAOOOOSS 8624422.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406-3413. IEEE, 2016.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng. et al. Ros: open-source robot operating system. In ICRA workshop open source software, volume 3, pp. 5. Kobe, Japan, 2009.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforeement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Joseph Redmon and Anelia Angelova. Real-time grasp detection using convolutional neural networks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1316-1322, 2015. doi: 10.1 10.1099/CRA.2015.733399
Manolis Savva, Angel X Chang, and Pat Hanrahan. Semantically-enriched 3d models for common- sense knowledge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 24-31, 2015.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez.DD' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. accepted. IEEE, 2021.
Rahul Shome and Kostas E. Bekris. Anytime multi-arm task and motion planning for pick-and- place of individual objects via handoffs. In 2019 International Symposium on Multi-Roboo and 2012.8901009,
Aulti-Agent Systems (MRS), pp. 37-43, 2019, doi:
Rahul Shome, Kiril Solovey, Jingjin Yu, Kostas Bekris, and Dan Halperin. Fast, high-quality two-arm rearrangement in synchronous, monotone tabletop setups. IEEE Transactions on Automation Science and Engineering, 18(3):888-90) 901, 2021. doi: 10. 1091119E.22202113055555 144.
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois R. Hogan, Joshua Tenenbaum, Pulkit Agrawal, and Alberto Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. In Conference on Robot Learning (CoRL), 2020. URL https //anthonysimmonor wwt.:wg.itu......... htt.:ooopooo-ppaiiigggooooo.. franneook--f framework/
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28:3483-3499 2015.
Nikolaus Vahrenkamp. Dmitry Berenson, Tamim Asfour, James Kuffner, and Rüdiger Dillmann. Humanoid motion planning for dual-arm manipulation and re- re-grasping tasks. In 2009 JEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 2464 2464-2470, 2009. doi: 10.1109/
0008.2009.335462
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for cete 6d object pose and size estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2642-2651, 2019.
12

---

Published as a conference paper at ICLR 2023
Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. arXiv preprint arXiv:2112.002444 2021.
Thomas Weng, Sujay Man Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fabricflownet: Bimanual cloth manipulation with a flow-based policy, In Conference on Robol Learning, pp. 192-202. PMLR, 2022.
Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022. URL https:tt///o//... htpp:st///peeeeeee...eee net oetp:/orrumlaa-11i0PPooo
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp-
11097-11107, 2020.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint 1rXiv:171..00199 2017.
Fan Xie, Alexander Chowdhury, M De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. Advances in neural information processing systems, 33:2327-253 2337, 2020.
Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manipulation. In Conference Robotic Learning (CoRL), 2020.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A REAL-ROBOT SETTINGS AND EXPERIMENTS
For real-robot experiments, we set up two Franka Panda robots, and place various target objects like boxes, buckets and displays in front of them. One RealSense i435 camera is mounted behind the two robots. The camera captures partial 3D point cloud data as inputs to our learned models.
We control the two robots using Robot Operating System (ROS) (Quigley al. 2009). The two robots are programmed to execute the actions proposed by our method. We use Movelt. (Chitta et al. 2012) for the motion planning.
In Figure 8] we present some promising results by directly testing our pre-trained model over real-world scans. We observe that our model has good generalization capability.
Please check our video for more results.
0.25
0.5
0.75
Push
Rotute
Topple
Pick-up
Figure 8: We present some promising results by directly testing model on real-world scans. In each block, from left to right, we respectively show the task represented by a red arrow, the input shape, the actionability heatmap predicted by network si, and the actionability heatmap predicted by network ae conditioned on the action of the first gripper. Please check our video for more results.
B MORE DETAILS ABOUT EXPERIMENT SETTING AND TASK FORMULATION
EXPERIMENT SETTING
All imputs and outputs are represented in the camera base coordinate frame, with the z-axis aligned with the up direction and the x-axis points to the forward direction, which is in align with real robot's camera coordinate system.
B.2 TASK FORMULATION
We formulate four benchmark tasks: pushing, rotating, toppling and picking-up up. We have explained the pushing task in the main paper, and here describe the remaining tasks:
14

---

Published as a conference paper at ICLR 2023
For rotating, / R denotes the object's goal rotation angle the horizontal plane. 0 0 denotes the object is rotated anticlockwise, object is successfully rotated if its actual rotation degree O' is over 10 degrees, its rotation direction is the same as (i.e., 0 * 0r > 0), and the difference between O' and less than 30 degrees.
0 denotes the object is rotated clockwise. An
For toppling, / E IR is unit vector denoting the object's goal toppling direction. An object is successfully toppled if its toppling angle is over 10 degrees, the difference between its actual toppling direction I' and / is less than 30 degrees, and the object not be rotated.
For picking up, / E TR3 is unit vector denoting the object's goal motion direction. An object is successfully picked up if its movement height is over 0.05 unit-length, the difference between its actual motion direction I' and / is less than 45 degrees, and the object should keep steady.
C DATA STATISTICS
We perform our experiments using the large-scale PartNet-Mobility/Me et al. 2019) dataset and ShapeNet(Chang Chang et al. 2015) dataset. We select object categories suitable for each dual-gripper manipulation task and remove objects that can be easily manipulated by single gripper (e.g., small objects like USB drives that are easy to push). To analyze whether the learned representations can generalize to novel unseen categories, the rest categories only for testing. We further split the training set into training shapes and testing shapes, and only used the training shapes from the training categories to train our framework.
randomly select some categories for training and reserve
We summarize the shape counts from the training categories in Table and the shape counts from the novel unseen categories in Table 4 In Table the number before slash is the training shape counts from the training categories, and the number after slash is the testing shape counts from the training categories. We label the categories from the ShapeNet dataset with *, and the remaining categories are from ParbNettMobillty dataset.
Figure visualizes one example for each object category from the dataset use in our experiment.
Table 3: The shape counts from the training categories. We further spilt the training into training shapes (before slash) and testing shapes(after slash), and only used the training shapes from the training categories to train our framework.
All
Box
Dishwasher
Display
Bench*
Keyboard*
pushing
272 / 992
21/7
35/12 12
28/8
141 /48
47 17
All
Box
Dishwasher
Display
Bench*
Keyboard*
rotating
272 / 992
21/7
3
28
141 /4
47 / 17
All
Box
Bucket
Dishwasher
Display
Bench*
Bowl*
toppling
383 126
21/7
23/7
35 /
28 /8/8
141 /48
135 /
All
Eyeglasses
Bucket
Trash Can
Pliers
Basket*
Display
picking-up
215 85
49 16
17/8
42 / 16
19/6
66/ /6/31
22/8 02/8
D MORE DETAILS ABOUT PERCEPTION MODULE
The Perception Module has submodules: the First Gripper Module and the Second Gripper Module. Below we will explain the network architecture details of the First Gripper Module, which consists of four input encoders and three parallel output networks.
Regard the four input encoders
Thee partial point cloud 0 E Rx3 is fed through segmentationn PointNet++ (Qi et al. 2017) extract the per-point feature f, I128
15

---

Published as a conference paper at ICLR 2023
Table 4: The shape counts from the novel unseen categories.
All
Kitchen Pot
Toaster
Basket"
pushing
164
25
25
114
All
Kitchen Pot
Toaster
Basket*
rotating
164
25
114
toppling
All
Kitchen Pot
Toaster
Basket'
164
25
25
114
picking-up
All
Box
Kitchen Pot
Scissors
Laptop
84
27
19
16
22
Display
Laptop
Dishwasher
Toaster
PartNet-Mobilli Dataset
Bucket
Kitchen Pot
Trash Can
Eyeglasses
Pliers
Scissors
Bench
Basket
Keyboard
Bowl
ShapeNet Dataset
Figure 9: We visualize example for each object category used in work.
The task E 3 (for pushing, toppling, picking-up) E E. (for rotating) is fed through fully connected layer (3 + 32 or + 32) to extract the feature fp E 3232
The contact point E IR3 is fed through fully connected layer (3 + 32) to extract the feature fp 32
The manipulation orientation R E R is represented by the first two orthonormal axes in the x rotation matrix, following the proposed 6l-rotation representation in (Zhou et al.. 2019h. We fed the orientation through fully connected layer (3 + 32) to extract the feature fR E 32
Regarding the three output networks
The affordance network of the first gripper oi concatenates the feature fs fi. and fp, to form a high-dimensional vector, which is fed through 2-layer MLP(192 128 + to predict per-point affordance score ai E [0, I].
The proposal network of the first gripper P is implemented as a conditional variational nutoencoder(cVAR (Sohn et al. 2015), which takes the feature concatenation of f.. fi. and fp, as conditions. The encoder is implemented as a 3-layer MLP (224 - 128 -
32 + 32) that takes the orientation feature fR, and the given conditions to estimate Gaussian distribution (μ E 32 R22). The decoder is implemented as 2- MI.P
16

---

Published as a conference paper at ICLR 2023
(224 - 128 6) that takes the concatenation of the sampled Gaussian noise E IR and the given conditions to recover the manipulation orientation R
The critic network of the first gripper '1 concatenates the feature fs. fi. f. and fR, to form a high- lige--imensional vector, which is fed through a 2-layer MLP(224 + 128 + 1) to predict the success likelihood ej E [0, 1].
For the Second Gripper Module, the architecture of its input encoders and output networks are the same those in the First Gripper Module, except that they take the first gripper's action i.e., Pi and R1 as the additional input. Therefore, the input dimension of the three parallel output networks should be added with 64, while other feature dimensions remain unchanged.
The hyper-pane parameters used to train the Perception Module are as following: 32 (batch size); 0.001 (learning rate of the Adam optimizer (Kingma & Ba 2014)) for all three networks.
E MORE DETAILS ABOUT RL AUGMENTED DATA SAMPLING
E.1 THE RL POLICY FOR INTERACTIVE TRAJECTORY EXPLORATION
It is too difficult for random policies to collect positive data for the picking-up tasks, since the objects and tasks are too complex. Therefore, we train an RL policy using SAC (Haarnoja et al., (018)) replacing the random policy to collect enough positive dual-gripper interactions. We access the ground-truth state information the simulation environment, such as object poses and orientations, and utilize them for the RL training.
State Space. The RL state is the concatenation of the object position Pa E RR3, the object orientation qu E RR and the two contact points PI. P2 E TR order select contact points more reasonable positions, we apply the pre-trained Where2Act (Mo et al. (02I) pulling affordance on the observation and randomly sample two different points with the top 10% affordance scores, Since the grippers have only one chance to interact with objects in one trial, the second state will be the stopping criterion.
Action Space. The RL policy predicts two grippers' orientations RI, E SO(3), which is the 3D rotation group about the origin of three-dimensional Euclidean space under the operation of composition. Since 6-dimensional real vector is enough to restore the 3 x 3 orientation matrix R, the dimension of the vector proposed by RL policy can be reduced to 12. Moreover, the actual actions of grippers are defined as: first generating two grippers at certain distance de away from contact points in the gripper fingers' orientation, then synchronously moving forward to the contact points, closing fingers, and finally moving back to the original positions.
Reward Design. The reward is the summation of: 1) task reward, which is assigned 1 if the interaction is successful(as defined in B). the object keeps steady (but its movement is less than the threshold), we give the reward of 20d, so the longer it moves, the higher reward it will get. If the movement over the threshold(but the object rotated or toppled), will acquire a small reward of 0.15. 2) interaction reward, which is determined by whether the object is grasped during the whole interaction. Successful grasps at the beginning and in the end will respectively give a reward of 0.5.
E.2 IMPLEMENTATION AND TRAINING DETAILS
We leverage SAC (Haarnoj et al. 2018) to train the RL policy. It consists of a policy network and two soft-Q networks, all of which are implemented policy network receives the state as input (R.5), and predicts the grippers' orientations as action (R!2). Its network architecture is implemented as a 4-layer MIP (13 + 512 - 512 - 512 512), followed by two separate fully connected layers (512 + 12) that estimate the mean and variance of action probabilities, respectively. The Q-value network receives both the state and action as input 2R25), and predicts a single Q value (R'). Its network architecture is implemented (25 > 512 + 512 + 512 + 1).
Multi-layer Perceptions (MLP). The
4-layer MLP
We the following parameerseeters to train the RL policy: 16384 (buffer size); 512 (batch size); 0.0002 (learning rate of the Adam optimizer (Kingma & Ba 2014) for both the policy and Q-value network);
17

---

Published as a conference paper at ICLR 2023
F MORE DISCUSSIONS ON COLLABORATIVE ADAPTATION PROCEDURE
In this section, we will provide more discussion on why the Collaborative Adaptation procedure can further improve the performance.
Before the Collaborative Adaptation procedure, we only use the collected offline data to train the networks. After such training stage, the proposed affordance maps and actions may not be completely and perfectly consistent with the desired distributions of collected offline data, and thus will exist some errors. The errors of the two separately trained modules would accumulate and lead to worse performance, and thus chances are that the dual-gripper manipulation system will propose many actions (i.e., false positive actions) that cannot actually fulfill the tasks.
Therefore, we propose the Collaborative Adaptation procedure, let two grippers perform actions end-to-end in simulation, and use the actual action results to adapt the dual-gripper manipulation system with the above explained cumulative errors. During this procedure, those false positive cases would be tuned to be negative, and thus the system will be adapted into better collaborative system.
G COMPUTATIONAL COSTS, TIMING AND ERROR BARS
It takes around 4.5-6 days per experiment single Nvidia 3090 GPU. Specifically, it takes around 3.5 days and day to respectively train the networks in Perception Module and in the subsequent Collaborative Adaptation procedure, while extra 1.5 days is needed if we apply the RL Augmented Data Sampling.
Our model only consumes 1,600 MB memory of the GPU during the inference time.
For each task, for the reason that we randomly sample 500 seeds initializing 500 random con- figurations, representing 500 different targets and object states, and execute 3 trials for each configuration to compute the average success rate, the errors may be quite small.
H MORE DETAILS ABOUT BASELINES
H.1 HEURISTIC BASELINE
For the rule-based heuristic baseline, we hand-craft set of rules for different tasks. We now describe the details in the following:
Pushing: to choose where to manipulate, select two contact points pl and P2 that are halfway up the given object, where p1 is at the left one-fifth of the length the object, and P2 is at the right -ifff of the length of the object. Then, we initialize the grippers with the same orientation as the given task, which is a unit vector denoting the object's goal pushing direction.
Rotating: if the given task is to rotate an object clockwise on the horizontal plane, we will choose the first contact point # at the one-tenth of the length at the right top, setting the first orientation downward, and thus the point is held as a fulcrum for rotation. Then we choose the second point /2 halfway up the object, the greater the given rotation angle is, the closer it will be to the fulerum, and the second orientation is set horizontally forward. On the contrary, the given task to rotate an object anticlockwise, the fulcrum will be chosen at the one-tenth of the length and the left top.
Toppling: we select two contact points P) and p2 that are one-fifth up the given object, where Pi the left fifth of the length of the object, and p2 is at the right fifth of the length of the object. Then, we set the two grippers with the same orientation as the given task.
Picking-up: we select the two contact points pi and p2 on the objects' left and right top edges and set the two gripper orientations the same as the given picking up direction.
H.2 M-WHERE2ACT BASELINE
Since Where2Act Mo et al. 202T) is used for single-griper and task-less interactions, we should adapt it as a baseline by modifying its network structure.
18

---

Published as a conference paper at ICLR 2023
For its Critic network, it receives as input the observation O, the given task 1, the two contact points Pt. P2. and the two orientations of the two grippers R1, R2 as a whole, and outputs a score c E [0, indicating the success likelihood of accomplishing the task when the two grippers collaboratively manipulate this object.
To implement the Proposal network, employ a conditional variational autoencoder (cVAE) (Sohn et al., 2015) instead using single MLP used in Where2Act. This module receives as input the observation O, the given task I, and both of the grippers' actions in R1), u2 - (2), proposes both gripper's orientations Ri, R2 as a combination.
For its Affordance network, it receives input the observation O, the given task 1, and the two contact points Pl P2. and outputs a score a E [0, 1], indicating the success likelihood when grippers manipulate with this pair of contact points. The ground truth supervision of this module is provided by the modified Critic network and the modified Proposal network.
The loss functions are similar to our framework.
H.3 M-FABRICFLOWNET BASELINE
We additionally compare method with another baseline method named M-FabricFlowNet (M- FFN). We will first give brief introduction to FabricFlowNet (Weng et al. 2022). and then provide the experimental results different tasks.
FabricFlowNet uses two grippers to manipulate the cloth with goal-conditioned flow-based policy. Given the initial and the target observations, FabricFlowNet can (I) use the FlowNet predict the 2D optical flow, which represents the correspondence between two observations, (2) use the PickNet to take in the flow input and predict the two pick points, (3) the predicted flow at the pick points to find the two corresponding place points. Note that if the distance between pick points (or place points) smaller than threshold, they will use only single gripper for manipulation avoid the collision.
In our implementation, we adapt FabricFlowNet as baseline with some modifications: (1) for optical flow, we calculate the ground-trute 3D flow between the initial observation and the target observation, and the ground-truth flow will be directly fed the following PickNet. This means, we have the perfect flow. (2) For the PickNet, given the ground-truth flow, we adapt it from a 2l>vvesioon to 3D-version by changing the backbone from FCN to PointNet++, and keeping other settings the same. (3) For the placing position, we also use the flow direction at the pick point to find the place point. However, since our task is in 3D space, apart from the place position, we also require to set the two grippers' orientations. For fair comparison, we grippers' orientations.
pretrained Actor Networks to propose the two
T the sample-success-rue and reason is that, M-FFN only disentangles the learning of the two grippers' actions, without further designs for the collaboration. M-FFN only learns the second action conditioned on the first, while method makes both the two actions conditioned on each other. Specifically, M-FFN uses ground-trutt data to directly supervise the first action, and then uses ground-truth data to supervise the second action conditioned on the first. As for our method, we also use ground-truth data to supervise the second action conditioned on the first, but further, we use the previously trained second gripper module to supervise the first action conditioned on the second (as shown in Figure 4 in our main paper). Therefore, our method makes the two gripper modules aware of each other, while in M-FFN, only the second action module is aware of the first.
that our method outperforms M-FFN. The
Table 5:
Baseline comparison on the sample-success-ratl metric.
Train Categories
Test Categories
pushing
rotating
toppling
picking-up
pushing
rotating
toppling
picking-up
M-FFN
33.67
19.60
47.20
19.30
25.40
14.80
36.47
26.90
Ours
48.76
33.73
65.53
40.33
42,93
35.07
41.80
54.33
19

---

Published as a conference paper at ICLR 2023
1
MORE RESULTS AND ANALYSIS
Train Categories
Test Categories
Rotate
Topple
Pick-un
Figure 10: We present additional qualitative results of the learned Affordance networks of and af to augment F in the main paper. In each block, from left to right, we respectively show the task represented by a red arrow, the object which needs to moved from transparent to solid, the actionability heatmap predicted by network 21, and the actionability heatmap predicted by network sn conditioned on the action of the first gripper.
1.1
MORE QUALITATIVE RESULTS
In Figure 10 and Critic networks C.
we respectively show more results on the learned Affordance networks of and
1.2
FAILURE CASES
Figure 12 visualizes some failure cases, which demonstrate the difficulty of our tasks and some ambiguous cases that are hard for robots to figure out. See the caption for detailed descriptions.
J DISCUSSION OF LIMITATIONS
Though our work takes one important step toward the dual-gripper collaborative affordance prediction, it focuses on the short-term interaction while there exist many long-term tasks in daily life, such as lifting and moving heavy pot to target pose with two grippers, opening a drawer with one gripper and putting an object inside with the other gripper. These long-term tasks are much more difficult and challenging, and we leave this topic for future work.
limitation of our work is that we use the flying grippers instead of the robot arms to interact with objects. Following Where2Act (Mo et al., (021)) we abstract away the control and planning complexity of having robot arms, and focus on learning the collaborative affordance for dual- -ripper manipulation tasks. Though our real-robot experiments show that the proposed actions by our network can apply to real robot arms with the help of the motion planning Movelt! (Chitta et al., 2012). it is meaningful and more realistic to the robot arms. Because, in the real-robot experiments, we should additionally consider the degree limitations of each arm joint, which means some contact points may be out of reach. Moreover, the joints of different robots to ensure safety, which will be more challenging compared to simply avoiding the collisions between two flying grippers.
must take into consideration the collisions amongst
20

---

Published as a conference paper at ICLR 2023
Train Categories
0:75
Test Categories
Pushh
Topple
Pick-up
Figure 11: We visualize additional qualitative results of the per-point action scores predicted by our Critic networks 6, and ' to augment Fig.5 in the main paper. each result block, from left right, show the task represented by red arrow, the input shape, 61 's predictions of success likelihood applying the first action over all points, and 62 's predictions success likelihood applying the second action over all points given the first action.
(a)
(b)
(c)
(d)
We visualize some failure cases, which demonstrate the difficulty of the tasks and some ambiguous cases that are hard for robots to figure out. (a) To push a keyboard on the ground, the two grippers may have collisions with the ground and can move on. (b) To topple a cylinder like bucket, it is easy to make the cylinder rotate during toppling. (c) The door is open due to gravity, so we can hardly find suitable contact points for interaction. (d) failed picking-uy up attempt since the lifting handle is a little oversized for the gripper to grasp.
Figure 12:
21


Reviews:
[{"review_id": "qbymLCrivIN", "reviewer": "Reviewer_yumu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to learn a bimanual perceptual affordance model, provides a simulated benchmark to evaluate the model and compares its performance against a number of sensible baselines.", "summary_of_the_review": "Very nice paper overall, useful to the community with a few ideas that could apply beyond the narrow setting of affordance modeling.", "strength_and_weaknesses": "Strengths:\n- very important topic, extremely relevant to representation learning for robotics. While it's been relatively easy to learn single-arm grasping directly from manipulation data, without having to learn an explicit affordance representation, doing so in the context of leads to an explosion in complexity which makes learning an independent proposal model for affordances very attractive.\n- the model appears sound, well-motivated, and immediately useful to derive further research on the topic.\n- the approach to combating the combinatorial explosion of having to learn two joint affordance models together is well-motivated, clever, and of practical relevance to any setting where chained conditional models have to be learned efficiently.\n\nWeaknesses:\n- the main weakness of the approach, in that it doesn't take the robot kinematics into account, is well documented and not a showstopper for future work.\n- the benchmark developed as part of this work would be exceedingly valuable to the community, particularly since the success rate on this task seems to hover around ~50%, which indicates it's in the sweet spot of not being too difficult or too easy. I saw no discussion of providing it as an open-source benchmark, which would greatly enhance the value of this submission.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "C_dufGoRzY", "reviewer": "Reviewer_VYuj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It is known that dual gripper is a challenging yet valuable robotic problem because there are daily tasks that are hard to complete with a single gripper. When robots are deployed for home-assistive applications, it becomes even more difficult because of the diverse 3D objects in the daily environment. The authors propose a collaborative visual affordance to facilitate dual-gripper manipulation to overcome the hurdle. Dual-gripper manipulation has the problem of a high degree of freedom. The authors propose a learning framework that disentangles the two gripper pose prediction subtasks and optimizes the two subtasks via a collaborative adaptation procedure. The authors conduct experiments on a proposed benchmark based on the PartNet-Mobility and ShareNet datasets. Four diverse dual-gripper manipulation tasks, i.e., pushing, rotating, toppling, and picking up, are studied. The proposed method achieves good performance compared with three baselines (random approach, heuristic approach, and a modification to Where2Act [Mo et al., 2021]). Additional experiments involve (1) showing the effectiveness of an RL-based data collection strategy and (2) real-world robot demonstrations.", "summary_of_the_review": "The authors propose a collaborative visual affordance learning framework for dual-gripper manipulation. In addition, a benchmark is proposed to validate the effectiveness of different algorithms for dual-gripper manipulation. In addition, a real-world demonstration is presented to prove the effectiveness of the proposed approach. However, several concerns are raised in the Weakness section, including the novelty of the proposed framework and benchmark and insufficient ablative studies.", "strength_and_weaknesses": "Strength:\n1. The authors tackle a challenging problem, i.e., dual-gripper manipulation. The work is timely toward home-assistive robots.\n2. . The reviewer believes the work can motivate the community of visual affordance learning and dual-gripper manipulation to stimulate new ideas toward home-assistive applications.\n3. The authors propose a collaborative visual affordance learning framework to address the challenges of dual-gripper manipulation due to high-degree freedoms. Specifically, they propose to disentangle the dual-gripper learning problem into two separate yet coupled subtasks, i.e., the second gripper module is conditioned on the output of the first module. With the proposed design, it can reduce the search space. In addition, they propose a collaborative adaptation procedure to enhance cooperation. The proposed collaborative adaptation procedure is shown to be effective in Table 1, where the success rate on both the training and testing category is higher than the one architecture without a collaborative adaptation procedure.\n4. The authors demonstrate the feasibility of the proposed method on real robots that demonstrate dual-gripper manipulation on the following tasks, i.e., picking up containers, rotating boxes, pushing displays, and toppling buckets.\n\nWeaknesses:\n1. An essential piece of this work is the concept of collaborative visual affordance. However, the authors did not propose any metrics and benchmarks for this part. The reviewer found it critical as this is the \"direction\" the authors aim to promote. Otherwise, the authors should compare other bimanual manipulation approaches (e.g., Weng et al., 2021 and Xie et al., 2020)\n2. If visual affordance learning \"direction\" is the aim of this work, what are the limitations of other visual actionable approaches for the tasks tackled in this work? For instance, Weng et al. propose a flow-based method for bimanual cloth manipulation (Weng et al., FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy, CoRL 2021). The work shows the feasibility of bimanual cloth manipulation. While the task domain of Weng et al., 2021 is different from this work, the idea of disentangling has been discussed in Figure 3. In addition, the pick point prediction is another strategy for manipulation. What is the advantage of the proposed approach? The authors should discuss the work and the difference.\n3. One of the contributions is the benchmark based on SAPIEN. However, the authors did not discuss the \"value\" of the proposed benchmark over others (e.g., Xie et al., 2020, Chitnis et al., 2020, Chen et al., 2022). Please comment.\n4. Lack of detailed ablative studies: This work has several components for collaborative visual affordance learning. Specifically,\n    1. Disentangled gripper module design with conditioning. What is the performance without conditioning? Does the model fail to converge?\n    2. Collaborative adaptation procedure\n    3. RL data sampling vs. Random sampling\n5. Currently, only (b) is evaluated and shown its effectiveness. The other two aspects do not prove their values. The question regards (c) can be found in the next point. Please comment.\n6. Table 2 results are confusing: Specifically, based on the reviewer's understanding, the results aim to show that the success rate of the RL-based sampling method is more effective than a random method. However, the notion in Table 2 is \"Ours w/o RL\" and \"Ours,\" which is confusing. Moreover, the results are for the picking-up task. How about other tasks? Please comment.\n7. Diversity of data collected using RL policy: while the authors show the success rate of RL-based sampling in Table 2, the reviewer is interested in the diversity of the RL-based sampling. What is the coverage of the sampled points based on the RL-based method? Could we visualize them?\n8. The reviewer suggests that the authors can conduct experiments in the real world where a policy is trained on training categories but test on the testing categories to further strengthen the proposed method's value.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BrG7W_Do9w", "reviewer": "Reviewer_RF6q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the visual actionable affordance learning method of Where2Act (Mo et. al. 2021) for dual-gripper manipulation. To deal with the quadratic complexity of dual action space, the authors propose to sequence action predictions from two grippers and conditioning the second on the first. This reduces the action space to linear complexity and aids in ensuring collaborative policies of the two grippers. To train the network components of their architecture, the authors generate in offline fashion dense affordance and gripper orientation maps from a simulation environment by sampling random policies, as well as with a soft actor-critic RL method. The paper also proposes a collaborative adaptation procedure to enhance collaboration between the predicted actions by training the network components end-to-end. Experiment evaluation showcases that the proposed method outperforms task-specific heuristics and a quadratic complexity extension of Where2Act, while being able to generalize in unseen object geometries.", "summary_of_the_review": "The paper introduces a novel method for dual-gripper manipulation based on visual affordances and it also publishes an evaluation suite for four collaborative tasks. I am fairly confident that the contributions are novel, impactful, and well-supported by the experimental section. However, I believe that the paper would benefit from some extra analysis and discussions on some aspects (see issues below):\n\n1. I would suggest adding more discussion in Sec. 4.4 (see also weaknesses). The authors mention: ” [...] In this way, the two gripper modules can better understand whether their proposed actions are successful or not as they are aware of interaction results, and thus the two separately trained modules are integrated into one collaborative system”. How is this ”better understanding” achieved? How does training together add to a better understanding of collaboration, given that separate gripper-wise annotations are also collaboration-aware?\n\n2. Some type of experimental confirmation of utilizing a dual-arm system vs a single gripper is missing. The authors motivate this well through the nature of the concerned task, but I think a single gripper could also achieve a subset of such tasks with extra effort (more actions). The benefits of then using a dual-gripper system are obvious, however, it would be nice if a comparison is included in the experimental section.\n\n3. Comparison with other recent dual-gripper manipulation systems (e.g. Gadre et. al. 2021).\n\n4. I would like to see a comparative analysis of the proposed vs. quadratic complexity version of Where2Act by training data size. How fast does your method pick up on the collaborative aspect compared to a holistic version?\n\n5. There are a few typos, e.g. in the Conclusion section ”[...] its superiority of the three baselines” should be ”over” etc. I believe an extra iteration by the authors would resolve these issues.", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written, motivates its purposes, and presents its ideas in a clear fashion.\n2. The proposed method addresses the complexity bottleneck of dual-arm manipulation by sequencing predictions from two grippers and conditioning the latter on the first.\n3. The paper also proposes (and experimentally evaluates) an RL-based method for data generation.\n4. Evaluation includes a study of generalization properties (novel shapes to be manipulated), which greatly improves its experimental\n5. The proposed method is integrated with a real robot and includes demonstrations in real environments.\n6. The authors generate evaluation testbeds in a simulation environment and claim that will release the dataset, which is a great contribution given the sparsity of resources for dual-arm manipulation.\nWeaknesses\n1. I would like more discussion on Sec. 4.4 (Collaborative Adaption Procedure), as it was also shown experimentally that it is essential for achieving collaboration between the grippers. However, the process is not 100% clear to me. It seems that the authors pair the learning networks with the robot to perform actions end-to-end in simulation, but how/why does that contribute more than training the components separately in an offline fashion, given that ground truth offline data are generated with annotations for the desired collaboration?\n2. Evaluation baselines include a random policy, heuristics, and a dual-gripper-enhanced version of Where2Act (Mo et. al. 2021). I think the paper would benefit by comparing their method also with non-visual affordance-based methods for dual-arm manipulation. For example, how does this method compare with the mentioned related work keeping one gripper fixed method (Gadre et. al. 2021)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster