Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions
Authors: Ruihai Wu
Kai Cheng
Yan Zhao
Chuanruo Ning
Guanqi Zhan
Hao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Affordance for Robotics
Articulated Object Manipulation
Occlusion Handling
TL;DR: We propose the affordance learning task that considers environment constraints, and propose a novel and data-efficient learning framework.

Abstract: Perceiving and manipulating 3D articulated objects in diverse environments is essential for home-assistant robots. Recent studies have shown that point-level affordance provides actionable priors for downstream manipulation tasks. However, existing works primarily focus on single-object scenarios with homogeneous agents, overlooking the realistic constraints imposed by the environment and the agent's morphology, e.g., occlusions and physical limitations. In this paper, we propose an environment-aware affordance framework that incorporates both object-level actionable priors and environment constraints. Unlike object-centric affordance approaches, learning environment-aware affordance faces the challenge of combinatorial explosion due to the complexity of various occlusions, characterized by their quantities, geometries, positions and poses. To address this and enhance data efficiency, we introduce a novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations. Experiments demonstrate the effectiveness of our proposed approach in learning affordance considering environment constraints.

Learning Environment-Aware Affordance for 3D
Articulated Object Manipulation under Occlusions
Ruihai Wu 1,4+ Kai Cheng?*
Yan Shen Chuanruo Ning Guanqi Zhan 3 Hao Dong 1,41
School of EECS, PKU NNatoonal Key Laboratory for Multimedia Information Processing, School of CS, PKU
CFCS, School of CS, PKU
University of Oxford
Abstract
Perceiving and manipulating 3D articulated objects in diverse environments essential for home-assistani robots. Recent studies have shown that point-level affordance provides actionable priors for downstream manipulation tasks. However, existing works primarily focus single-object scenarios with homogeneous agents, overlooking the realistic constraints imposed by the environment and the agent's s morphology, e.g., occlusions and physical limitations. In this paper, propose an environment-avern affordance framework that incorporates both object-level actionable priors and environment constraints. Unlike object-centric affordance approaches, learning environment-awann affordance faces the challenge of combi- natorial explosion due the complexity of various occlusions, characterized by their quantities, geometries, positions and poses. To address this and enhance data efficiency, introduce novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations. Experiments demonstrate the effectiveness of our proposed approach in learning affordance considering environment constraints.
Introduction
Articulated objects, such as doors and drawers, exist everywhere in our daily life. Perceiving and manipulating these objects present crucial yet challenging tasks in computer vision and robotics. Unlike rigid objects, articulated objects exhibit diverse articulation types and functionally important articulated parts crucial for human and robot interactions. Numerous research endeavors have been investigating articulated objects broadly, encompassing joint parameters estimation [4] 47), part pose estimation 22 23), kinematic structure estimation
34 33
digital twins generalization 13
articulated part robotic manipulation
25 44
46
and few-shot policy adaptation [42
However, most existing works for manipulating articulated objects primarily focus on single-object scenarios with homogeneous agents, such as flying grippers [46 25 or exed-position robot arms ( Consequently, these approaches tend to develop object-centric representations and policies, neglecting the realistic constraints imposed by both the environment and the agent's morphology. These constraints are commonplace in real-world scenarios and their oversight limits the applicability and performance of the manipulation tasks. For example, successfully opening cabinet door that is obstructed by occluders not only depends on the properties the target door but also heavily relies on the robot's position and the way it interacts (e.g., colliding bypassing) with the occluders.
We take significant step towards manipulating articulated objects in a more realistic setting, i.e., considering constraints imposed by the environment and robot. Such a task encounters the combi-
"Equal contribution. Author ordering determined by coin flip.
i "orresponniig author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Train
Test
Affordance
Manipulation
Pris
00
0.3
0.25
Pull
Target Point
Significant Part
Affordance
Guide
Real-World
Manipulation
Scan
Figure/ Learning Environment-Aware Affordance for Articulated Object Manipulation. As the complexity of occluder combinations grows exponentially, we leverage the property that target manipulation point (Red Point) conditioned significant parts of occluders that impact the manipulation usually have limited local areas (Red Point's corresponding Red Box) even in complex scenes. Aware of such significant parts, our model can be trained on one-oceluder scenes (Train) and generalize to multiple occluder combinations (Test, Affordance). The learned affordance provides actionable information for articulated object manipulation (Manipulation). Our model predicts reasonable affordance on real-world scanned point clouds (Real-world Scan).
natorial explosion challenge in complexity. To be specific, the substantial variability of occluders. characterized by their numbers, geometries, positions, and poses, leads to exponential complexity in scene data distribution [49] 18). This introduces significant obstacle to training model that can comprehensively understand the diversity and intricacies of different scenes within the limits of avail- able data. Moreover, sampling-bess based motion planning [19 (35, kind of commonly used approach for planning and manipulation, also faces performance degradation with increasing occluders (38 as the probability of valid sampling rapidly diminishes.
We propose the use of point-level representations to tackle the challenge outlined above. Specifically, we exploit notable property: even in situations where the combinations of occluders are complex, given the target manipulation point and the robot, the occluder parts that significantly affect the manipulation are typically confined to limited area. This is clearly illustrated in Figure (Row 1, Column 2), where, in the case of pushing drawer obstructed by multiple occluders, the portions of the occluders that could potentially collide with the robot are confined to specific area (shown as areas in the two boxes with bright colors in the heatmap). This confinement results from the predetermined position of the robot and the target manipulation point, which limits the range of possible trajectories for the manipulation. Hence, when we contemplate the manipulation of the target object at the point level, the complexity of these significant scene components remains manageable, despite the possible addition of unseen occluders.
Further, we take inspiration from affordance for robotic manipulation, which provides actionable priors of the target at the point level and thus guides manipulation policies, and has exhibited remarkable efficacy in 3D articulated object manipulation [25]
6 and other manipulation tasks [54] 26 43). Unlike these works that focus solely on single objects, we introduce the concept of environment environment-awan affordance for articulated object manipulation, which integrates object-centric actionable priors with environment constraints the point level. Specifically, in scene featuring target articulated object, diverse occlusion objects, and robot at different positions, we aim to learn the per-point actionable information of the target object, taking into account the constraints imposed by the environment and the agent. Furthermore, in light of Gibson's 's theory of affordance
articulated

---

(the originator of affordance theory), where affordance is defined as the different possibilities of action that the environment offers an agent 77. our proposed task is not only more realistic but also more closely aligned with this foundational theory compared to preceding works.
The point-level representation we propose above shows the potential to train significant-part-awee model in simple scenes while capable of generalizing to more complex scenes. This approach leverages the aforementioned point-level significant parts, whose complexity remains manageable across a spectrum from simple to complicated scenes. To facilitate learning of Sggifficant-part-wwae scene representations, we design a tobot-target conditioned occlusion field.
Further, we introduce a contrastive learning method that refines our learned representations to selectively disregard insignificant elements while maintaining sensitivity similar, significant parts across varied scenes. Consequently, our proposed learning framework can be trained on scenes with single occluder, yet generalizable scenes with numerous novel occluder combinations. This strategy effectively mitigates the issue of combinatorial explosion in a data-efficient manner.
We conduct experiments using SAPIEN physics simulator [45 equipped with large-scale PartNet- Mobility [27] and ShapeNet i1 datasets. To assess our framework's performance and data efficiency, we initially train it on scenes containing only one occluder, with additional augmented contrastive scenes that incorporate an extra occluder. We then test our framework on significantly more complex scenes, featuring diverse combinations of novel occluders. Experimental results yield convincing evidence of the effectiveness and data-efficiency of our proposed framework.
In summary, we make the following contributions:
We explore the task of manipulating articulated objects within environment constraints and formulate the task environment--waare affordance learning for manipulating 3D articu- lated objects, incorporating object-centric per-point priors and environment constraints.
To tackle the combinatorial explosion problem in scene complexity, we propose a data- efficient framework capable of training on scenes featuring single occluder and generalizing to scenes with complex occluder combinations, leveraging contrastive learning and point- level local significant part representations.
We establish benchmarking multi-objec full-robot (as opposed to flying grippers) envi- ronments in SAPIEN simulator. Results show our framework learns environment-aware affordance generalizable well to novel scenes with complex novel occluder combinations.
2 Related Work
2.1
Visual Affordance for Robotic Manipulation
Visual affordance 7 is a kind of representation that indicates possible ways for robots to interact with the target and complete tasks. Many works study affordance for the classic grasping task in robotics [2 28 3. 17 16 50), while there exist many current works on point-level affordance
indicating object geometries for articulated object manipulation [25 44 42 5
6 29 dual-gripper
collaboration (54) and object to object interaction (26). Tracing back to Gibson 77 the proposer of
affordance, however, affordance notates the opportunities of interactions that involve consideration of the environment constraints and the robot, such as the occluders and the robot morphology. and thus ought to be aware of the robot and environment. Therefore, we propose the environment-un awarr affordance learning task, incorporating both the object actionable priors and environment constraints.
2.2 Occlusion Handling
Occlusion is significant challenge for current computer vision and robotics systems. In computer vision, there have been previous works trying to handle occlusion in different tasks, including object
detection [39 14 52]. instance segmentation 48 14 53 and tracking (9] Besides, to intrinsically handle the occlusion, another series of work aim to recover the entire shape of occluded objects, i.e. amodal segmentation
[55 [2]].
In robotics, the occlusion problem mainly exists in object retrieval 20 12 grasping 40 51 36 or
rearrangement 2 21 in clutters. In our paper, we study the task of learning the manipulation of 3D articulated objects with cluttered occlusions in front of the target object, which involves consideration of not only the observations of occlusions and the target, but also the agent's 's relationship with them.
3

---

Problem Formulation
We formulate new challenging task, Environment-Aware Affordance, aiming infer point- level affordance to indicate the actionable information for full robot arm in different positions to manipulate 3D articulated objects under diverse occlusions. While previous point-level affordance learning works only consider flying grippers [25 42 44 [4] or fixed arms [6] to manipulate the target object without occluders, further consider the robot location well scene occlusions.
Specifically, in our formulated task, given as input a 3D scene point cloud S with n points P1. P2.P2..... .... ... containing target articulated object T with points Tp. Tp .... Tpm object point cloud O1, O2, Oz..., Ok, and the robot position R E TR the model is required to predict the ..... point-level affordance score oT, RR.S s on each target point Tp, of T, indicating how likely the point interactable. The point-level affordance is able guide 3D articulated object manipulation tasks.
l occluder
As the complexity of occluder combinations grows exponentially, making difficult and time- consuming collect enough occluder combinations for training, propose data-efficient method that trains on only a few scenes with single occluder, which can generalize to scenes with multiple occluder combinations.
Per-point Feature
Target Encoder Er118
pip p2ss
pais
Pushing Novel Scene Generalization
Scene Point Cloud
Affordance Predictor Ar,ins
FATTa
Scone Eacoder
0.2
Robot Position
Pulling
Novel Scene Generalization
Robot Engoder ER
Figure Our Proposed Data-efficient Framework for Learning Environment-Awan Affordance. Our model takes occluded scene point cloud and robot position as input. Our framework generates per-point occlusion fields indicating most significant local parts occluders. Then, it predicts per-point affordance using extracted features of each target point, its corresponding occlusion field, and robot position. The trained model can generalize to novel multi-occluder scenes. Red points denote manipulation points over the target object.
4
Method
4.1
Overview
Our framework is composed of four networks: Robot Encoder &R. Target Encoder Er,,s,, Scene Encoder Es nt and Affordance Predictor Ar, While εn and Er, extracts the robot and target representations fa and JT, (Sec. 4.2 the most important component of our framework is learning the robot-target conditioned scene representations that are sensitive the significant local parts in the scene for manipulation (Sec. 4.3 which uses ESIT,, to extract the scene representations SSIT,.. from a designed robot-target conditioned occlusion field. This component makes the model sensitive to robot-target point conditioned significant parts and thus empowers the model with the generalization novel complicated scenes. Finally, AI,IR, takes fn. ST,1 and fSYI, and predicts the point-level environment-awann affordance score aT, R.S Tp (Sec. 4.4).

---

4.2 Robot and Target Representations
Both the robot position and the target manipulation point (including its position and local geometry) will affect the environment-ave aware affordance. For example, when the robot position changes, the affordance will change according to the robot's new reach area and interaction modes with occluders. Similarly, when the target manipulation point changes, the originally interactable point will become non-interactive for various reasons, such as robot being unable reach the new point, the local geometry of the target point not supporting the desired action (e.g., smooth surface does not support pulling), or the robot colliding with occluders while attempting to move to the new point. So we design Robot Encoder εR and a Target Encoder Erp1s to extract relevant information.
As shown in Figure 2 we employ a MLP network as &R, which encodes the robot position f into a latent representation fR E R 128 and a Segmentation-verssonn of PointNet++ [31] that encodes the scene point cloud into per-point features, where fr,1s R 128 represents the feature of Tp
4.3 Robot-Target Conditioned Scene Representations
In scenes with multiple occluders, the quantity, geometry and arrangement of occluders demonstrate rich diversity. However, given the robot position and the target manipulation point, most points in the scene are unlikely to significantly affect the manipulation, and the area of the significant parts that do have an impact (e.g., the area around occluders where potential collisions may occur during robot movement) is usually not large. By focusing on these significant parts of the scene, the change of occluder number and geometry will matter, and model can effectively learn generalize more complex scenes with multiple novel occlusion combinations.
To this end, design the tobot-target conditioned occlusion field to facilitate learning significant scene representations conditioned on the robot and the target point (See. (.3.I. and propose the robot- target conditioned contrastive learning to further enhance learned scene representations (Sec. 4.3.2)
4.3.1 Robot- Target Conditioned Occlusion Field
Although a scene may contain a large number of points (of occlusions and the target object), when we know the target manipulation point and the robot position, the point number of the significant part is not large, and the distance between these points and either the robot or the target manipulation point is not far. Therefore, design the robot- robot-target conditioned occlusion field that maps the scene into a vector field conditioned on both the target point and the robot, facilitating learning scene representations highlighting significant parts for representations.
We define the Occlusion Field as conditional continuous vector field F on an open and connected C 3 which can be represented by value function F R.T,, G R
set G 3 Object
F FRI, 2) VR x VT,
(xr TR, yn, =RR) @ (x IT,, VT,, 2T,))
F2. E.
The field factors VR and VT, are seperately conditional on R and Tp- For any point (ar, ys in G, its value vector F r, (xr, y. is caleulated as the eross product (8) of the Euclidean vectors from (x, y, to robot R and target Tp. two fixed points of F R.T,,. Since the field is continuous, we have:
lim F FR,T, (x. 2) lim F FR.T, (x, y.=) 0. p-I,
p-R
So the model can filter the unimportant points whose field values too large.
We employ PointNet (30) network as the Scene Encoder Esyr, It takes as input the conditional vector field F R.T, mapped from the scene point cloud S selected with small field values, and then outputs the robot-target conditioned scene representations SSy,,
4.3.2 Robot-Target Conditioned Contrastive Learning
The above-learned robot-target conditioned scene representations JsiT, should be sensitive to the meaningful local parts of the occluders for manipulation, while agnostic to unimportant occluders and their local parts, despite the number, geometry and combinations of them. Specifically, the learned
5

---

scene representations should have the following properties: (I) given the same target point, when new occluder occurs in the scene while affecting the manipulation, the learned representations will keep the same; (2) given the same scene, when the target point changes, the meaningful local parts of scene as well as the learned representations will correspondingly change.
To better empower scene representations with such properties, which can further boost the per- formance and data-efficiency affordance prediction, propose the robot-target conditioned contrastive learning method for learning scene representations.
As shown in Figure for predicting the point affordance iT,,RRS for TR.s(2 (p) E XR.s task in single-oceluder scene S, the target point Tр and the robot R. correspond- ingly generate a positive and a negative task. Specifi- cally, generate the task of predicting the point affor- dance ( with the same scene S drawn from aug- mentation distribution A(- TR)s(p)), robot R and dif- ferent target point Tp drawn from marginal distribution A() (p) A(. ((.ITR.s(p)) (P)ss))). for the reason that although the scene keeps unchanged, the target point changes and thus the significant part in occluders will correspondingly change, and thus the scene representations should change. Also, we generate the task of predicting the point affordance @r,, Ittss* with the same robot R, target point Tp- and new scene which adds to the original scene with an occluder that uninfluential to the manipulation. Even though the scene changes from S to S", the significant part the occlud- keeps unchanged, and thus the target-robot conditioned scene representations should keep the same. We leverage this prior to conduct our contrastive learning, which performs better representation learning for the occlusion field.
Scene Eacoder
Stangwrighr
Positive
Scene Encoder
Tripler
Original
Wright
Encoder
Negative
Figure 3: Robot- Target Conditioned Contrastive Learning. For each point affordance prediction task (Mid- dle, ir,,nn.s). respectively generate its corresponding positive (Up, same target point with an extra occluder âr, (R,s') and negative (Down, differ- ent target point âT, |R,s) paired tasks, and use triplet loss learn the scene representations.
The original task and its positive and negative paired tasks constitute triplets, so we use triplet loss [32] learn the scene representations of them contrastively (a is the bound- ary constant):
CCL T,RRS = || - SSTI,,
[STEp.R - SSYT +
4.4 Affordance Prediction
We propose the Affordance Predictor Ar, n.s that aggregates the information of the target, the robot and the scene to predict the point-level environment-awar affordance.
We employ a MIPP the Affordance Predictor AT,RR, that takes fn. JT, and fSiT R as input, and predicts the affordance score åT,IR, s R on each target point T, conditioned on R and
We apply Li loss to measure the error between the affordance prediction ir,|R, and the ground-truth affordance score ar, R.S on a certain target point Tp:
CAFF
Li R.S. âT, (n,s) T,IRS
(4)
The total loss for the whole framework is then defined
Ctotal CAFF + XOL CCL TyRRt L,,n,, R,,
(5)
where ÀcL is balancing coefficient.

---

Experiments
5.1 Settings
For tasks, we follow Where2Act [25] and use the point-level affordance predictions of pushing and pulling articulated parts (doors and drawers) and robot, the ground-truth affordance score of a target point is set to be 0 / I when the robot fails succeeds in pushing pulling the target point without colliding occluders.
tasks. In scene with occluders, target point
For simulation and dataset, we use SAPIEN [45] as our simulation environment, equipped with large- scale Partnet-Mobility dataset [27] and ShapeNet [ dataset, with oc- cluder data statistics as shown in Ta- Besides, we use cabinets and
Train- Catn.
All
Basket
Bottle
Bowl
Box
Train-Data Test-Data
367
77
16
128
17
128
31
4
44
Bucket
Chair
Pot
TrashCan
ble 4 tables target objects, for the reason that in the real world there often exist many occluders in front of them.
27
61
16
25
9
20
5
10
Novel-Cats.
All Dispenser 4
Jar
Kettle
FoldChair
For training, we collect interactions in one-occluder scenes. Specifically, we collect 900 successful and 900 failure interactions for pushing, and 300 successful and 2500 failure in- teractions for pulling. Pulling needs more failure interactions as most points not pullable. For each data, we additionally collect a positive and a negative paired interaction for contrastive learning. For testing, use multi-occluder scenes respectively in training category test shapes and novel categories.
Test-Data 589
528
26
26
Table 1: Occluder Dataset Statistics. We use 1,084 different
shapes in ShapeNet [I] and PartNet-Mobillty (27). covering 12 commonly seen indoor occluder categories. We use 8 training categories (split into 367 training shapes and 128 test shapes), and 4 novel categories with 589 shapes.
For evaluation, to evaluate the accuracy of affordance prediction, we follow Where2Act and use F-Score and Average Precision. To evaluate predicted affordance's capability in providing actionable priors for manipulation scenes, we introduce Sample Manipulation Accuracy metric sma:
sma successful interactions
(6)
total interactions
The total interactions are proposed based affordance predictions the whole scene. We randomly adopt proposals with a confidence threshold and compute the manipulation accuracy.
Baselines and Ablations
We compare method with the following baselines that learn affordance for manipulation:
Where2Act (W2A) 25 that abstracts the robot arm into flying gripper and learns point- level actionable affordance for manipulating articulated objects.
W2A-R 255 that uses robot arm instead of flying gripper in Where2Act's setting, and adds robot encoder in Where2Act's ' networks.
020-Afford (020) 26 that learns point-level affordance for object-to-objed interactions. We use the robot and the scene as two input objects.
020-M that trains O20-Afford using scenes containing multiple occluders.
To ensure fair comparison, train the baselines using both originally generated data and their corresponding positive and negative paired data collected for contrastive learning.
Moreover, we compare our method with Collision Avoidance RTT Planner (RTT-CA) 35 8. commonly used sampling-based planner for robotic manipulation, to demonstrate the capability of point-level affordance in guiding manipulation in complicated scenes.
In addition, we compare our method with two versions that respectively ablate one core component:
Ours w/o OF that learns environment-aware affordance without the occlusion field.
Ours w/o CL that learns environment- affordance without contrastive learning.

---

5.3 Results and Analysis
Table 2 and Figure 4 demonstrate the superiority our framework quantitatively and qualitatively,
Results of W2A (highlighting every actionable point) demonstrate the necessity in considering the robot instead of flying gripper in more realistic settings.
Results W2A-R and 020 (they have similar qualitative results and we show visualization of W2A-R) that consider robot arms demonstrate that, our framework can better recognize the novel occluder part (e.g. the back of foldingchair that hinders pulling the handles in the middle doors, shown in Line 2, Column 3 and 5 in Figure in novel complicated scenes.
Results of 020-M demonstrate that, when the amount of training data is not large, training on complex multi-occcluer scenes cannot achieve performance comparable to our data-efficient method.
Results of Ours w/o OF and Ours w/o CL demonstrate the necessities of two components. With occlusion field, our model better recognizes target areas that may exist collision with novel-shaped occluders (e.g., the drawer board hindered by the basket in Line the door edge hindered by in Line 2, Figure 5). With contrastive learning, our model better discriminates between actionable and non-actionable points, while Ours w/o CL predicts reasonable affordance but with less certainty.
Comparisons between RTT-CA and affordance methods show that, while sampling-bas baaedd planners may face difficulty in complicated scenes, point-level affordance may be more suitable by directly providing per- per-point actionable priors for manipulating articulated objects within complex constraints.
Push
Pull
W2A
W2A-R
020-M
Ours
Figure 4: Qualitative Comparisons between Our Method and Baselines.
Push
Pull
Ours w/o OF
Ours w/o CL
Ours
Figure 5: Qualitative Comparisons between Our Method and Ablations.
Task
RRT-CA
W2A
020
020-M
W2A-R
Ours w/o OF
Ours w/o CL
Ours
pushing
23.65
17.80
34.60
37.33
34.76
39.43
36.63
43.52
pulling
10.31
3.82
29.93
29.36
27.55
33.19
28.02
36.19
Table 2: Sample Manipulation Accuracy( (%).

---

Method
pushing
pushing (novel)
pulling
pulling (novel)
W2A
64.70 / 52.97
62.80 / 47.07
66.42 37.59
60.37 40.73
W2A-R
68.66 / 71.31
64.86 69.55
34.06 72.56
26.79 73.79
020
65.18 / 77.07
59.04 72.18
46.03 / 74.06
43.42 73.56
020-M
61.06 / 67.82
67.43 68.54
39.76 / 66.42
33.29 / 61.47
Ours w/o OF
64.28/ 69.02
58.31 66.94
36.36 67.19
33.33 64.30
Ours w/o
68.59 / 74.02
64.70 /1.00/71.6.
65.74 75.40
55.25 77.92
Ours
77.30 / 86.11
76.11 / 83.58
71.28 75.96
61.32 75.18
Table 3: Quantitative Evaluations and Comparisons with Baselines and Ablated Versions. In each entry, we report F-Score( %) and Average Precision( %) before and after slash.
position!
position2
position3
Figure 6: Predicted Affordance Changes Conditioned on Different Robot Rositions.
Push
Pull
Figure Predicted Affordance on Real-Worle Scans.
Additionally, Figure shows our method generates different environment-awant affordance condi- tioned different robot positions in the same scene. Figure deemonttrrtes framework generates promising results by testing on different real-world scenes. It is worth mentioning, as denoted in the red circles, our model only learns constraints from occluders, but also learns avoid manipulating points that may lead to collisions with other parts of the object (i.e., self occlusion).
Conclusion
We introduce enviromment-awarc affordance for manipulating 3D articulated objects within envi- ronment constraints, leveraging point-level representations to address the combinatorial explosion challenge scene complexity. Using an interactive environment built upon SAPIEN and the Mobillty and ShapeNet datasets, we train neural networks that predict per-point actionable information for manipulating articulated 3D objects under occlusions. We present extensive quantita- tive evaluations and qualitative analyses of the proposed method. Results show that the learned priors are highly localized and thus generalizable to novel scenes with unseen occluder combinations.
Ethics Statement. Our work has the potential to enable robots on articulated object manipulation complicated scenes. The learned environment-awe affordance avoids collisions in manipulation, and thus reduces the risk of accident. We do not see work has any particular harm or issue.

---

7 Acknowledgment
This work was supported by National Natural Science Foundation of China General Program (62376006) and The National Youth Talent Support Program (8200800081).
References
[1] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 151.00252..0312,, 2015.
[2] Sang Hun Cheong, Brian Y Cho, Jinhwi Lee, ChangHwan Kim, and Changjoo Nam. Where to relocate?: Object rearrangement inside cluttered and confined environments for robotic manipulation. In 2020 JEEE International Conference on Robotics and Automation (ICRA), pages 7791-7797. IEEE, 2020.
[3] Enrie Corona, Alhert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5031-5041, 2020.
[4] Ben Eisner*, Harry Zhang*, and David Held. Flowbot3d: Learning articulation flow to manipulate articulated objects. In Robotics: Science and Systems (RSS), 2022.
[5] Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15752-15761, October 2021.
[6] Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to- end affordance learning for robotic manipulation. International Conference on Robotics and Automation (ICRA), 2023.
[7] James Gibson. The theory of affordances. Hilldale, USA, 1(2)x77-22, 1977.
[8] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint (.XXi:23322.466555 2023.
[9] Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, and Carl Vondrick. Tracking through containers and occluders in the wild, 2023.
[10] Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In JEEE International Conference on Robotics and Automation (ICRA), 2023.
[11] Yuan-Ting Hu, Hong-Shuo Shuo Chen, Kexin Hui, Jia-Bin Huang. and Alexander Schwing. Sail- vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3105-3115, 2019.
[12] Baichuan Huang, Shuai D Han, Jingjin Yu, and Abdeslam Boularias. Visual foresight trees for object retrieval from clutter with nonprehensile rearrangement, IEEE Robotics and Automation Letters, 7(1):231-23) 2021.
[13] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[14] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion- instance segmentation with overlapping bilayers. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition (CVPR), pages 4019-4022 2021.
[15] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. The 3rd International Conference for Learning Representations, 2015.
10

---

[16] Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352-3359, 2020.
[17] Mia Kokie, Johannes A. Stork, Joshua A. Haustein, and Danica Kragic. Affordance detection for task-specific grasping using deep learning. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoidss, pages 91-98, 2017.
[18) Adam Kortylewski, Qing Liu, Angtian Wang, Yihong Sun, and Alan Yuille. Compositional convolutional neural networks: robust and interpretable model for object recognition under occlusion. International Journal of Computer Vision, 129:736-760, 2021.
[19] James Kuffner and Steven M LaValee. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995-1001. IEEE, 2000.
[20] Andrey Kurenkov, Joseph Taglic, Rohun Kulkarni, Marcus Dominguez-Kuhne Animesh Garg, Roberto Martín- Martín, and Silvio Savarese. Visuomotor mechanical search: Learning to retrieve target objects in clutter. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 8408- 8414. IEEE, 2020.
[21] Jinhwi Lee, Younggil Cho, Changjoo Nam, Jonghyeon Park, and Changhwan Kim. Efficient obstacle rearrangement for object manipulation tasks in cluttered environments. In 2019 International Conference on Robotics and Automation (ICRA), pages 183-189. IEEE, 2019.
[22] Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A Lynn Abbou, and Shuran Song. Category- level articulated object pose estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3706-3715, 2020.
[23] Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but geometric constraints: model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088 2020.
[24] Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[25] Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In International Conference Computer Vision (ICCV), 2021.
[26] Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O20-Afford: Annotation- free large-scale object-object affordance learning. In Conference on Robot Learning (CoRL), 2021.
[27] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[28] Luis Montesano and Manuel Lopes. Learning grasping affordances from local visual descriptors. In 2009 IEEE 8th international conference development and learning, pages 1-6. IEEE, 2009.
[29] Chuanruo Ning, Ruihai Wu, Haoran Lu, Kaichun Mo, and Hao Dong. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[30] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:/612.0059) 2016.
[31] Charles Ruizhongtai Qi, Yi, Hao Su, and Leonidas J Guibas. Pointnet++: feature learning on point sets in metric space. In Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
Deep hierarchical
11

---

[32] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 815-823, 2015.
[33] Rafal Staszak, Milena Molska, Kamil Mtodzikowski, Justyna Ataman, and Dominik Belter. Kinematic structures estimation on the rgb-d images. In 2020 25th IEEE International Confer- ence on Emerging Technologies and Factory Automation (ETFA), volume 1, pages 675-681.
IEFE, 2020.
[34] Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research, 41:477-526, 2011.
[35] loan A Sucan, Mark Moll, and Lydia E Kavraki. The open motion planning library. IEEE Robolics & Automation Magazine, 19(4):72-2,, 2012.
[36] Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automution (ICRA), pages 13438-13444, IEEE, 2021.
[37] Bingjie Tang and Gaurav S Sukhatme. Selective object rearrangement in clutter. In Conference on Robot Learning, pages 1001-1010. PMLR, 2023.
[38] Hao Tian, Chaoyang Song, Changbo Wang, Xinyu Zhang, and Jia Pan. Sampling- -batd planning for retrieving near-cylindrical objects in cluttered scenes using hierarchical graphs. IEEE Transactions on Robotics, 2022.
[39] Angtian Wang. Yihong Sun, Adam Kortylewski, and Alan L Yuille. Robust object detection under occlusion with context-aware compositionalnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12645-12654, 2020.
[40] Chenxi Wang, Hao-Shu Fang, Minghao Gou, Hongjie Fang. Jin Gao, and Cewu Lu. Graspness discovery in clutters for fast and accurate grasp detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15964-15973, 2021.
[41] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[42] Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. European conference on computer vision (ECCV 2022), 2022.
[43] Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[44] Ruihai Wu, Yan Zhao, Kaichun Mo. Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022.
[45] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated past- interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[46] Zhenjia Xu, He Zhanpeng, and Shuran Song. Umpnet: Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters, 2022.
[47] Zhihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver van Kaick, Hao Zhang, and Hui Huang. RPM-NET: Recurrent prediction of motion and parts from point cloud. ACM Trans. on Graphics, Articee 240, 2019.
12

---

[48] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11141-11150, 2021.
[49] Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? International Journal of Computer Vision, 129:781 802, 2021.
[50] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757, IEEE, 2018.
[51] Andy Zeng. Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pack-and-place of novel objects in clutter with aalti-affordaace grasping and cross-domaie image matching, The International Journal of Robotics Reseurch, 41(7):600-705 2022.
[52] Guanqi Zhan, Weidi Xie, and Andrew Zisserman. tri-layer plugin to improve occluded detection. British Machine Vision Conference, 2022.
[53] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self- supervised scene de-occlusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2020.
[54] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual dual-gripper object manipulation. International Conference on Learning Representations (ICLR), 2023.
[55] Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1464-1472, 2017.
13

---

Appendix
A
More Details on Simulation and Settings
Following Where2Act 25 we design our interactive simulation environment based on SAPIEN, using the same of simulation parameters for all interaction trials.
For general simulation settings, we use frame rate 500 fps, tolerance length 0.001, tolerance speed 0.005, solver iterations 20 (for constraint solvers related to joints and contacts), with Persistent Contact Manifold (PCM) disabled (for better simulation stability), with disabled sleeping mode (i.e. no locking for presumably still rigid bodies in simulation), and all the other settings default in SAPIEN release.
For physical simulation, use the standard gravity 9.81, 1,, static friction coefficient 4.0, dynamic friction coefficient 4.0, and restitution coefficient 0.01. For the object articulation dynamics simulation, use stiffness O and damping 10.
For the rendering, we OpenGL-based rasterization rendering for the fast speed of simulation. We set three point lights around the object (one at the front, one from back-left and one from back-right) for lighting the scene, with mild ambient lighting as well. The camera is set have near plane 0.1, far plane 100, resolution 448, and field of view 35..
For 3D partial point cloud sean inputs, back- pock-project the depth image into foreground point cloud, by rejecting the far-away background depth pixels, and then perform furthest point sampling get 10K-size point cloud sean.
For robot arm movement, we use RRT Planner 35 and execute a certain path towards the target.
equipped with PID controller to generate
For interaction trial be considered successful, it not only needs cause considerable part motion along intended direction. To avoid the extreme data unbalance in pulling data, we manually handle mask on our simulator and assign half of the interactions on the handles. To simplify the consideration of different interaction directions' impact on affordance, we set every interaction to move along the normal direction of the target point.
B More Data Details and Visualization
In Table 4 we summarize data statistics. In Fig. visualize our simulation assets from ShapeNet [T] and PartNet 227 that we use in this work.
Train-Cats
All
Basket
Bottle
Bowl
Box
Bucket
Chair
Pot
TrashCan
Train-Date 367
77
16
128
17
27
61
16
25
Test-Data
128
44
5
20
5
10
Test-Cats All Dispenser
Jar
Kettle
FoldingChair
Test-Data 589
528
26
26
Table 4: Occluder Dataset Statistics. We use 1,084 difffernnt shapes in ShapeNet [ and PartNet- Mobility (27). covering 12 commonly seen indoor occluder categories. We use 8 training categories (split into 367 training shapes and 128 test shapes), and 4 test categories with 589 shapes networks have never seen in training.
14

---

Basket
Bottle
Bowl
Box
Training Categories
Bucket
Chair
Pot
Trash Can
Testing Categories
Dispenser
Jar
Kettle
Folding Chair
Figure 8: Our simulation assets from ShapeNet [ and PartNet [27].
C More Training Details
C.1 Hyper-parameters
We set the batch size 30, and Adam Optimizer 155 with 0.001 the initial learning rate.
We use const 2.00 as the boundary constant in a contrastive learning, and 1.00 as the balancing coefficient ÀoL in the total loss.
C.2 Computing Resources
We use Py Torch as our Deep Learning framework, and single RTX GeForce 3090 (20GB GPU) for training and inference.
D
More Results and Analysis
Fig.
12 demonstrate comparions with baselines and ablations. Fig. 13 shows the whole occlusion fields. Fig. 14 shows real-world demonstrations with analysis the caption.
10
11
E Discussion and Visualization on Failure Cases
We present some interesting failure cases in Fig. 15 From these examples, we see the difficulty of the task. Also, given the current problem formulation, there are some intrinsically ambiguous cases that are generally hard for robots to figure out from a single static snapshot. Moreover, our affordance is trained on data generated with robots deploying naive policy, e.g., pushing the normal direction of the target plane. Better policies may introduce better affordance.
15

---

FRESAAR
Ours
V2A
CONZA
MAODO
Figure
Quaniissiree More 9;
in Basshines and Nettioli Our beeween Compaprtzonnn
Puussinge
OF wio Compartiona Ours Duauiaatte
Ours betteen Neciooi Our and Abhatonn in Pushing
CL ViO
Ours
More 10. Frgure
WVZ
WPAAR
020AN
Ours
Figure
11: More
Conperstunn Duaiifattre
berveen Our Mettuoi and Basciinee
Pollinne in
16

---

Ours w/o OF
Ours w/o CL
Ours
Figure 12: More Qualitative Comparisons between Our Method and Ablations in Pulling.
pushing
pulling
Input
Occlusion Field
Occlusion Field
Input
Figure 13: Visualization of the whole Occlusion Fields.
17

---

Push
Door
Collide with Drawer
Successful
Pull
Door
Collide with
Successful
Pull Drawer
W2A Object-Centric
Ours Environment-Aware
Successful
Manipulation
Manipulation
Figure 14: Real-World Demonstrations Manipulation Policy Guided by Object-Centric Affordance and Our Proposed Environment-Aware Affordance. It is clear that environment- aware affordance can help avoid out-of-reach situations and collisions with other self self-parts objects.
Target Point
Collision
Figure 15: Failure cases. Up: robot initialized close to occluders. Down: Motion planning method leads the robot collide with occluders when approaching plausible target point.
F Future Work on Robot- Robot-Target Conditioned Contrastive Learning
Limited to simulator configuration, our contrastive learning method only considers limited aug- mentation distribution A(- .)) for each anchor scene T E X' while the marginal distribution A(-) = A( F) is complete. The augmentation distribution A(- x) only includes one more occluder at the edge of h. and neglects the potential augmentation methods by choosing similar target points. Future methods can be applied with a better similarity metric of comparing different things and improve self- sele-super learning paradigm.
18


Reviews:
[{"review_id": "7Juv9GBdxi", "reviewer": "Reviewer_gQVm", "summary": "This paper introduces a framework for learning environment-aware affordance in occluded 3D object manipulation. It tackles combinatorial explosion and improves data efficiency. The authors showcase their superior framework using benchmarking multi-object full-robot environments in the SAPIEN simulator. Their contributions involve studying articulated object manipulation within environment constraints, proposing a data-efficient framework, and establishing benchmarking environments.", "questions": "- L38/267 mention that sampling-based motion planning faces performance degradation with increasing occluders. Could you provide a benchmark of the time and cost of different methods to address this issue?\n-  L147 mention E_R encodes the robot position R into a latent representation. What about the geometry or configuration of the robot? How do you ensure that the scene and the configuration of the robot are plausible?\n- How would the proposed framework generalize to different types of robots and scenes?\n- In real-time executions, the state of point clouds is constantly changing, and the heatmap also changes. How do you plan to handle this issue?", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework proposed addresses the learning of environment-aware affordances for 3D articulated object manipulation, even under occlusions.\n- The paper focuses on affordance learning and introduces benchmarking environments to enhance robotic system performance in real-world settings.\n- The paper includes a comprehensive description of the framework, establishes benchmarking environments, and highlights its superior performance.\n- It is well-structured, easy to comprehend, and effectively presents experimental results.\n", "weaknesses": "- The proposed framework has only been evaluated on simulated environments, and there is a lack of real-world experiments from different perspectives and scenarios.\n- This work primarily examines a static scene, which raises concerns about the applicability of the proposed framework in real-world scenarios that involve more complex and dynamic environments.\n- The authors have not provided a detailed analysis of the failure cases of the proposed framework.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We have addressed them below, and hope to hear back from you if you have further questions!\n\n> Lack of real-world exp from different perspectives and scenarios.\n\nWe have provided **3 pairs of comparisons of affordance and their corresponding manipulation policies in real-world scenarios** in Figure 7 in the supplementary pdf and the supplementary video (3:35 - 4:09), to show the superiority of our method over baselines in the real world. Additionally, we have scanned **more real-world point clouds in different perspectives and scenarios** and provided the predicted affordance in Figure C in the Global Response PDF. Since point clouds of different perspectives can be rotated into the same perspective with differences in the vacancy of point clouds given camera intrinsics and extrinsics, our method is robust to the complexity of perspectives (except for orthogonal and behind views). Besides, apart from our work, a series of papers have demonstrated point-level affordance can generate promising results when directly deployed on different real-world point scans [1, 3, 4, 5] (specifically, Figure 6 in [1], Figure 5 in [3], Figure 6 in [4], Figure 1, 7 in [5] and Figure 8 in supplementary in [4]).\n\n\n\n> The applicability of the proposed framework in real-world scenarios that involve more complex and dynamic environments.\n\nFollowing the settings of previous affordance learning works [1, 2, 3, 4, 5], we evaluate our framework in static environments. We have made the scene more complex and realistic by introducing **combinations of occluders** in the environment, and then integrate object-centric actionable priors with environment constraints. As for **manipulation in a dynamic scene**, a simple solution is to manipulate step-by-step with the affordance of each frame, which is demonstrated in [6, 7]. As shown in Figure E in the **Global Response PDF**, we construct a dynamic scene with the movement of the target articulated part and one occluder. The predicted affordance in such a dynamic scene shows **consistency** in consecutive frames as most points do not change much. We believe making affordance more consistent in such dynamic scenes is a future direction worth studying. As for real-time dynamic scenes with movable robots, we believe it is a promising direction for future work, and our work has laid a solid foundation in this direction. Thanks for this valuable comment! We will add more results and discussions in the paper.\n\n> Changing states of point clouds and heatmaps in real-time executions.\n\nPlease refer to the previous response.\n\n\n\n> Detailed analysis of failure cases.\n\nSee the **Global Response TEXT** and Figure D in the **Global Response PDF** for details.\n\n\n> Benchmark of the time and cost of different methods on the issue of performance degradation with increasing occluders.\n\nPerformance degradation with increasing occluders is a common issue for planning, as it costs more to find a successful path. In contrast, the time cost of our method remains constant, because the complexity of different input point clouds does not affect the inference time of neural networks at a fixed point sampling number. Below we provide the benchmark of time and cost of different methods: the commonly used sampling-based method RRT for planning and our method. For RRT, when the number of occlusions increases from 0 to 5, the inference time in one scene is 0.04, 0.27, 1.38, 2.29, 2.75, 3.08 seconds respectively, while the inference time of our method remains around 0.18 seconds.\n\n\n\n> Geometry or configuration of the robot.\n\nPlease see the **Geometry or configuration of the robot** section in **Global Response TEXT**.\n\n\n\n> Generalization to different types of robots and scenes.\n\nFor different robot types, please see the **Geometry or configuration of the robot** section in **Global Response TEXT**. For different scenes, our environment-aware affordance has **demonstrated good generalization** towards scenes different from those in training. Specifically, a scene is mainly composed of the target object and occluders. For **target objects**, point-level affordance has demonstrated the generalization towards different target objects [1, 2, 3, 4, 5], because it focuses on the local geometry of each target point instead of the whole object, which is more generalizable across different object categories sharing similar actionable local parts. Apart from the two categories (Cabinet and Table) in our paper, we further demonstrate the affordance of **more categories** in Figure B in the **Global Response PDF**. For **occluders**, our proposed Robot-Target Occlusion Field filters out most unimportant points and focuses on the most important part of the scene, facilitating generalization to novel scenes with unseen and more occluders. As a result, while trained on scenes with **only one occluder** sampled from 8 kinds of 367 objects, our model can generalize to **combinations** of 128 **unseen objects** in training categories and 589 objects in 4 **unseen categories** in a scene. Statistics and example occluders are shown in Table 1 in the main paper and Figure 1 in the supplementary PDF.\n\n[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.\n\n[2] O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning. CoRL, 2021.\n\n[3] VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. ICLR, 2022. \n\n[4] AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. ECCV, 2022.\n\n[5] DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Object Manipulation. ICLR, 2023.\n\n[6] UMPNet: Universal Manipulation Policy Network for Articulated Objects. ICRA, 2022.\n\n[7] FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects. RSS, 2022.\n"}, {"review_id": "Gt4aW6y9gN", "reviewer": "Reviewer_eVFQ", "summary": "The paper presents a novel environment-aware affordance prediction framework for robot interactions (pushing/pulling) with parts of articulated objects (drawers and doors). The proposed approach consists primarily of two novel improvements: a robot-target conditioned occlusion field and a contrastive learning formulation for improved data efficiency. The paper evaluates the proposed approach against where2act and O2O with metrics from those works and demonstrates a fair improvement (~6% overall and 2-10% in all individual categories). Real world trials demonstrate the usefulness of predicted point-level affordances for robotics tasks. Qualitative results demonstrate several clear wins for the approach related to prediction of occluding object constraints.", "questions": "I understand the value of robot and target encoder networks, but I didn’t find any satisfying details or citations to provide insight into what these networks produce/encode or how they are trained. Are these empty MLPs initialized with random weights and allowed to fill freely, or is there something else to them? Are they described more in prior art or specific to this work?\nI would like to better understand the intuition behind the Robot-Target Conditioned Occlusion Field. Why does the vector field of cross products of point-to-robot and point-to-target vectors provide a good heuristic? Some expanded text here would be valuable.\n\tWhat is a “novel” scene for the evaluation test metrics? Is it a scene with additional occluders that was not present in training?\nWhy does the pulling performance on novel scenes only improve 1-2%? Is this significantly harder to predict occlusions or is there a common failure mode that escapes the assumptions of the heuristic for the approach?\n", "limitations": "Discussion of limitations seemed light. Only the lack of mobile base was mentioned. Given the maximum performance of 43/36% push/pull I expected to learn more about what wasn’t working or what additional challenges and failure modes were observed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Based on a known simulation system and asset data, the evaluation and metrics soundly demonstrate the advantage of the proposed approach over prior art.\nTrials on down-stream manipulation tasks with a real robot platform validate the usefulness of the point-level affordances predicted by the approach.\nThe proposed improvements over prior art (contrastive learning and robot-target conditioned occlusion fields) appear novel and constitute interesting solutions to the challenges posed.\nThe advantage of training on simple occlusions and generalizing to more complex occlusion scenarios is a clear strength of the approach.\n", "weaknesses": "Some areas of the paper could be improved. I felt there was some information missing leaving me with questions. The high-level concepts were repeated often and could be deduplicated to make room for more details.\nThe line was slightly blurry between novel work and prior art. Clearly this approach builds upon W2A and O2O, but the paper would benefit from clarifying carry-over vs. novel formulations.\n\tSMA metric appears to be the ssr metric from W2A with a different name.\n\tSome discussion of F-Score and Average Precision metrics would help interpretation. Could be appendix if repeated from prior art.\n\tOverall accuracy improvement over SoTA was somewhat small (6% over prior methods to achieve 43/36 push/pull). \n\tThe paper talks up the approach without mentioning remaining challenges or failure modes.\n\nExposition could be improved with further polish and proofreading.\nGrammar(G)/Typos(T) (listed with line number) :\nG - 18 - perceiving and manipulating these objects present\nT - 39 - also faces experiences\nT - 107 - objectc\nT - 148 - extra period after fR\nG - 255 - We randomly adopt proposed\nT - 267 - Moreover to we \n274 - Table 3 2 (formatting? Comma or conjunction?)\nG - 284 - of that may exist collision\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We have addressed them below. Hope to hear back from you if you have further questions!\n\n> High-level concepts were repeated often. \n\nThanks for your suggestion! We will prune repetitive concepts to make space for crucial details.\n\n\n> Clarifications on carry-over vs. novel formulations.\n\nGiven the overlap of many settings from W2A[1] and O2O[2] and limited pages, we didn't describe many details in the paper. We will further polish the paper and provide more details. Here are more clarifications.\n**Sample Manipulation Accuracy (SMA)** mirrors the Sample Successful Rate (SSR) in W2A. We think the name SMA can better reflect the function of affordance (providing actionable priors for manipulation), and thus use this name. \nFor **F-Score and Average Precision**, we borrowed them from W2A and O2O. Our F1 Score calculation: $2 * (precision * recall) / (precision + recall)$. The definitions for precision and recall remain standard.\n\n\n\n> Challenges and failure modes.\n\nPlease see **Global Response TEXT** and Figure D in the **Global Response PDF**.\n\n\n> Improvements over SoTA.\n\n**First**, baselines put up in the paper are not actual prior methods except for W2A. O2O, O2O-M, W2A-R have been modified by us to fit the new setting, which have been improved by adding consideration of robot constraints. Otherwise, prior works perform unsatisfactorily with a successful rate of 10%-23%. Besides, O2O-M is trained on data of scenes with multiple occluders, while our model is trained on data with only 1 occluder.\n**Second**, as described in *Challenges and failure modes*, 80% failures come from the initial close distances between the robot and some occluders, or the following motions of some robot joints given a plausible target point. These cases are quite difficult and require a policy to produce fine-grained robot joint variable sequences equipped with our environment-aware actionable priors that propose plausible target points. We recognize this as a prospective avenue for future research. Our environment-aware affordance serves as a pioneering step in this direction, highlighting the significance of identifying credible target points.When excluding such failure modes, our method performs well in most cases and the improvement margin over baselines becomes larger.\n\n\n\n> Robot and target encoder networks.\n\nThe robot encoder E_R is a 3-layer MLP (3->32->64->128) to represent the robot’s 3D position relative to target objects. The target encoder is a Segmentation-version PointNet++[3] encoder (architecture same as [3]). Following prior works [1, 2], all networks are initialized with random weights and allowed to fill freely.\n\n> Why Robot-Target Conditioned Occlusion Field provides a good heuristic.\n\nThe intuition of Robot-Target Conditioned Occlusion Field is to highlight points near the robot or target, while sidelining distant ones. Therefore, we design the vector field in a manner that represents the spatial relationships of points in the scene *relative to both the robot and the target*.\n\nBy defining the field factors $V_R$ and $V_{T_p}$ as vectors pointing from any scene point to the robot and the target respectively, the vectors inherently encode the spatial relation of the scene points to the robot and the target. The cross product of two vectors provides a new vector that is orthogonal to the plane formed by the original vectors. By taking the cross product of $V_R$ and $V_{T_p}$, the resulting field captures the interaction between a point's relationship with the robot and the target.\nIt emphasizes points that have significant relationships with both the robot and the target.\nThe field value approaches zero as a point gets closer to robot/target. This property ensures the points that have a large cross product of $V_R$ and $V_{T_p}$ can be considered less significant since their spatial relationship to both the robot and target is more distant or oblique while keeping near important points.\n\nMoreover, **Figure 6 in supplementary** visualizes many occlusion fields showing the influence on field values caused by different robot positions / target points.\n\n> “Novel” scenes for the evaluation test metrics.\n\nNovel refers to **unseen object categories**. Our model is trained on objects in 8 categories, tested on combinations of objects in 4 unseen categories. Details and statistics are shown in Table 1 in the paper. Thanks for this comment, and we will further clarify this setting.\n\n\n> Pulling performance on novel scenes.\n\nSuch performance improves 1-2% only for **F-Score metric compared with W2A**, and **Average Precision metric compared with W2A-R and O2O**. For **W2A**, as it only considers a flying gripper without considering the constraints of the robot arm, more points are predicted as actionable (positive), which leads to the recall (true positive rate) super higher than other methods, leading F-score to be high, while with a much lower Average Precision (false positive rate) than all other methods. For **W2A-R and O2O**, as most points are not pullable, models tend to give negative predictions, and only actions that are more likely to succeed will be predicted as positive by these models and ours. As Average Precision is calculated among actions predicted as positive, the performance of these models and ours in this metric will be similar in the pulling task. With similar precision, as our method has a higher recall than W2A-R and O2O, it has a much higher F-Score. Overall our model is robust and performs well in 4 tasks as a whole.\n\n> Grammar / Typos and improvement on exposition.\n\nThanks for your attention to details! We will fix the problems and carefully improve the exposition.\n\n\n\n[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.\n\n[2] O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning. CoRL, 2021.\n\n[3] PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NeurIPS, 2017.\n"}, {"review_id": "d9ZLK0XPnD", "reviewer": "Reviewer_zyYG", "summary": "Given a 3D point cloud and a robot, the method predicts per-point actionable affordance. More specifically it predicts whether a given point is pushable or pullable by the robot in a given position. The method takes into account environment occlusion. The paper presents a data-efficient training framework that allows training on a single occlusion and generalize to multiple occlusions. In addition, the paper presents a novel contrastive learning approach.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Acounting for occlusion when predicting affordance is a step up from previous work. It has not been tackled before as far as I know.\n- The paper presents several novelties: problem setup and formulation, data-efficient framework, and contrastive learning approach\n- The method outperforms previous work. \n- Paper is well written and easy to follow. \n- The results are well presented and accompanied by a good analysis", "weaknesses": "- The action space is too small. Only pushing and pulling are considered.\n- The test setting seems too simple. Most of the results are shown on very similar desks. \n- Related work section is too short and does not clearly explain what is the gap in previous work, nor how the paper is filling that gap. \n- Section 4.3.1 is lacking a supporting figure\n- The robot geometry is not taken into account. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you! We have provided responses below.\n\n> The action space is too small. Only pushing and pulling are considered.\n\nWe follow the settings of a series of affordance learning works [1, 3, 4] and use pushing and pulling as the action primitives, as they are the most common action primitives, and many actions can be the combination of them. Besides, the affordance predictions on these two actions can already demonstrate the contribution of superiority of our proposed framework in learning environment-aware affordance.\n\n\n\n> The test setting seems too simple. Most of the results are shown on very similar desks.\n\nThe main focus and contribution of our work is to learn affordance with environment constraints, and thus the generalization is mainly demonstrated in the environment constraints (occluder numbers, categories, shapes and combinations).\nWhile trained on scenes containing **only one occluder** in 8 kinds of 367 different objects, our model is tested on scenes containing **combinations of occluders** from **128 unseen objects** in those categories and 589 objects in **4 unseen categories**. Detailed statistics and example occluders are shown in Table 1 in the main paper and Figure 1 in the supplementary PDF. Therefore, the test setting is actually quite difficult, which requires the strong generalization ability towards novel complex scenes with different occluder numbers, categories, shapes and combinations.\nAs for the category of the target object,  previous point-level affordance works have demonstrated the generalization ability towards different *target* objects [1, 3, 5].\nWe choose two categories (Cabinets and Tables) as they are the most complicated categories with various self-occlusion parts in the SAPIEN simulation. Inferring environment-aware affordance on Cabinets and Tables needs not only consideration of the interaction constraints from occluders and robots, but also the measure of self-occlusion from different parts of the object itself (e.g., pulling a drawer without colliding with another drawer nearby). Besides, in the real world there often exist many occluders in front of them, making our setting very practical. Additionally, we have trained a model on three new categories with good affordance predictions (shown in Figure B in the **Global Response PDF**), showing that the framework can handle scenes with more diverse target objects.\n\n\n\n\n> Related work section is too short and does not clearly explain what is the gap in previous work, nor how the paper is filling that gap.\n\nThank you for pointing out this. We will add more discussions on the gap between our work and previous works (integrating object-centric actionable priors in previous works with environment constraints in our work) and how we fill this gap (proposing robot representations, robot-target conditioned occlusion field to efficiently learn environment-aware affordance).\n\n\n> Section 4.3.1 is lacking a supporting Figure.\n\nSection 4.3.1 describes the Robot-target Conditioned Occlusion Field, and the 3 heatmaps in Figure 2 (Left), 3 heatmaps in Figure 3 show the most important area of the occlusion field. Additionally, the 8 heatmaps in Figure 6 in supplementary show the visualization of the whole occlusion fields, demonstrating how changing the target point and the robot position will influence the heatmap. Thank you for pointing out this! We will add more descriptions in Section 4.3.1 on the relationships between Section 4.3.1 and the above figures.\n\n\n\n> The robot geometry is not taken into account.\n\nPlease see the **Geometry or configuration of the robot** section in **Global Response TEXT**.\n\n\n\n\n[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.\n\n[2] O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning. CoRL, 2021.\n\n[3] VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. ICLR, 2022. \n\n[4] AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. ECCV, 2022.\n\n[5] DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Object Manipulation. ICLR, 2023.\n\n[6] PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NeurIPS, 2017.\n"}, {"review_id": "0sUpwoQKAT", "reviewer": "Reviewer_ntD4", "summary": "The paper introduces the environment-aware affordance task, taking into account both object-centric affordance and environmental constraints. More specifically, the proposed task is designed to incorporate various occlusion cases and the robot's position.  In order to tackle the inherent challenge of combinatorial explosion, the paper introduces a novel framework that utilizes a local robot-target conditioned occlusion field and contrastive losses. This framework enables effective training on simpler scenarios with only one occluder, while also facilitating generalization to more complex scenes.", "questions": "* What’s the performance of the trained model on scenes without any occlusion? And what will be the performance compared to the prior work in this scenario?\n* When inference, is the segmentation of the target object and all occluded objects are given? For the real case, how do you get the segmentation? \n* Is the number of points of the target object fixed for the network? How about the number of points for the occluded objects? If the number of points is not fixed, how to balance the number of points in each train example?", "limitations": "The authors mention the limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed task that takes the environment constraints into account is novel and insightful. Occlusion and robot position conditioned affordance is intuitive and potentially useful for downstream robot tasks.\n* The idea of utilizing a local robot-target occlusion field to enable generalization to more complex occlusion scenarios is straightforward and effective based on the experimental results.\n* The strategy employed to construct positive and negative pairs for the contrastive loss is intriguing, and the ablation studies confirm its efficacy in improving the scene encoder.", "weaknesses": "* In the proposed environment-aware affordance task, both the segmentation of the target object and the occluded objects are provided. For previous work like W2A, which only consider the object-centric affordance, it’s unclear whether the addition of simple heuristics can yield positive results. One possible heuristic is to examine the predicted affordance from W2A and determine if there are any points close to the points of occluder, subsequently filtering the affordance based on this observation. While I acknowledge the significance of environment-aware affordance for downstream tasks, I remain uncertain about the complexity and realism of the current proposed task. In addition, for the target objects, there are only cabinets and tables, it’s also unclear if the train and test scenario uses the same instance or different instances from the same category. Only two categories may not be challenging enough for the task.\n* For the robot position, it’s unclear how many positions are chosen in the train and test set. From the supplement video, there seems to be 4 different robot positions. It’s unclear if the method can generalize to new robot positions well.\n* For the push and pull action, it’s unclear if the default action direction is the surface normal. There should be more descriptions on the action selection and the dataset construction, including how to pick the occluder, how to add the occluder and how to judge if the action succeed considering the occluded objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for valuable comments! We have addressed them below. Hope to hear back from you if you have further questions!\n\n> Whether the addition of simple heuristics can yield positive results.\n\nWe have conducted the corresponding Where2Act with the heuristic experiment with the successful rates of *28.35%* in pushing, while our method achieves a successful rate of *43.52%*.\nThe reason for such heuristic method’s failure is, only the distance between occlusions and the point on which the gripper interacts with the object could not fully avoid the whole robot’s collisions. Specifically, as the robot is composed of many links connected by joints, when the gripper is moving to the target point, the movements of other links are not free, and those links may have different poses and touch different occlusions in moving. In other words, in the proposed Where2Act+heuristic, even though the final pose of the gripper could avoid collision with occlusions, during the movement of the robot, chances are that some part of the robot may touch the occlusions.\n\n\n> Clarifications on instances and categories in training and test, and challenges of our setting.\n\nThe main focus and contribution of our work is to learn affordance with environment constraints, thus the generalization is **mainly demonstrated in the environment constraints**, as the generalization of point-level affordance towards more target object categories has been comprehensively demonstrated in previous works[1, 3, 5]. As for our setting, while trained on 8 kinds of 367 different occluders, our model is tested on combinations of 128 **unseen objects** in those categories and 589 objects in 4 **unseen categorie**s in the scene. Please see statistics and example occluders in Table 1 in main paper and Figure 1 in the supplementary PDF.\nMoreover, the model is trained on **scenes with only one occluder**, while tested on scenes with **combinations of different numbers and kinds of objects**, which can demonstrate the data efficiency of our framework.\nTherefore, the test setting is actually quite challenging.\nAs for the instances and categories of the **target object**, the chosen two categories (Cabinets and Tables) are the most complicated categories with various self-occlusion parts in the SAPIEN simulation. Inferring environment aware affordance on Cabinets and Tables needs not only consideration of the interaction constraints from occluders and robots, but also the measure of self-occlusion from different parts of the object itself that may affect the target part. Besides, in the real world there often exist many occluders in front of them.\nAdditionally, we have trained a model on **three new categories** with good affordance predictions (shown in Figure B in the **Global Response PDF**), showing that the framework can handle scenes with more diverse target objects.\n\n> How many positions are chosen in the train and test set.\n\nThe robot position in our setting is continuous in $R^3$ for both training and test, and our method can generalize to robot positions within reach and out of reach of the target.\nBesides, Figure 6 in the main paper shows affordance when robots are in more different positions.\n\n> Clarification on action directions, more descriptions on the action selection and dataset construction.\n\nWe used the surface normal as the action direction since it can work in most cases and our work mainly focuses on occlusion and robot constraints instead of action directions.\nAs for more descriptions of action selection and the dataset construction, we followed the setting of [1]. The dataset is constructed self-supervisedly in SAPIEN. Specifically, the occluder is randomly chosen from Table 1 in the main paper, and randomly added in a wide area (X: (-1.0m, 1.0m); Y: (0, 1.5m)) in front of the target object (0, 0). An action is judged to be successful when it successfully moves the target part at a threshold (greater than 0.01 unit-length or 0.5 relative to the total motion range of the articulated part) without colliding with the occluded objects.\nDue to the page limit, we did not include many such descriptions in the submission, and we will add more descriptions in the main paper and supplementary. Thank you for this comment!\n\n\n\n> What’s the performance of the trained model on scenes without any occlusion? And what will be the performance compared to the prior work in this scenario?\n\n\nWe have conducted the corresponding experiment and show the successful rates of our method and previous works on the pushing task: *W2A / O2O / Ours: 40.29% / 43.37% / 63.42%*.\n\nThe results show that our model performs well on scenes without any occlusion.\nEven though it is a simple setting, prior object-centric works (Where2Act, O2O-Afford) do not consider the robot arm’s constraint so they still fall behind us.\n\n> When inference, is the segmentation of the target object and all occluded objects are given? For the real case, how do you get the segmentation?\n\nSee **Clarification on the segmentation mask** Section in **Global Response TEXT**.\n\n> Number of points of the target object.\n\nThe number of points of the target object and that of the occluded objects are not respectively fixed. Instead, we fix the number of points of the whole scene to be 30000.\nSpecifically, we use Furthest Point Sampling (FPS) to sample 30000 points from the scanned point cloud.\n\n[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.\n\n[2] O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning. CoRL, 2021.\n\n[3] VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. ICLR, 2022. \n\n[4] AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. ECCV, 2022.\n\n[5] DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Object Manipulation. ICLR, 2023.\n\n[6] PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NeurIPS, 2017.\n"}, {"review_id": "q5RhwFmN1W", "reviewer": "Reviewer_wNB2", "summary": "This work formulates environment-aware affordance for articulated object manipulation, which incorporates both object-level per-point actionable priors and environment constraints. First, this work identifies limitations of previous works on object-level affordance learning, including over-simplified robot manipulators and environment occlusion; it proposes to incorporate robot and environment constraints into the per-point affordance score. Next, this work proposes a contrastive learning method to allow the learned affordance prediction model to generalize to complex and unseen occluder combinations. This work evaluates the success rate of the proposed method with open door and drawer tasks in simulated environments and show improvements over previous affordance learning methods, and it demonstrates that the learned affordance can be useful in real robotic tasks by showing real robot demos.", "questions": "1. How are the simulation assets pre-processed? What convex decomposition is used? \n\n   - With default PartNet-Mobility and SAPIEN simulator, all assets would be processed into convex shapes and form a large gap between physics and rendering. Manipulation experiments performed on such data would not reflect a realistic scenario. So it is important to describe the data pre-processing method.\n\n2. Why robot position $R$ is used as input instead of transforming the point cloud? Is there some important choice on canonical frame that is not described in the text? How do you ensure the model prediction is invariant to a rigid transformation of the entire scene?\n\n3. What are the setup for the real-world experiments? How are the models trained and deployed? Is there finetuning on real-world data? Is there domain gaps when the model is trained on perfect point clouds in simulation and deployed on noisy data in the real world? \n\n4. What is the inference time for the method when deployed on real scenes?", "limitations": "The paper provides a description of some limitations and I have raised questions of other potential limitations in the previous *Questions* section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work has a clear goal and motivation. It addresses several important limitations of previous affordance learning works (over-simplified robots and environments), allowing per-point affordance prediction to be deployed on real robots to complete tasks in more complex and realistic environments.\n\n- This work provides adequate quantitative and qualitative experiments and comparison with baselines to show the effectiveness of the proposed method. It also justifies the necessity of 2 significant components, occlusion field and contrastive learning, with ablation studies.", "weaknesses": "1. This paper does not provide a clear definition of per-point affordance in problem formulation. I think this paper needs to provide a more comprehensive definition of \"affordance\" and \"interactable\" in introduction or section 3, preferably with concrete examples. Currently the motivation and goal of this paper is hard to understand without reading a previous work such as Where2Act.\n\n2. The problem formulation is also missing several key points.\n\n    - What is the scene point cloud? Based on the experiment section, it seems to be a colored point cloud obtained from a depth camera. This information needs be explained clearly in the formulation.\n   - How is the point cloud segmented into target point cloud and occluder point cloud? This information is available in the simulation, but how is it done in the real-world experiments?\n\n3. The robot-target conditioned occlusion field is not described clearly. I find section 4.3.1 very confusing. The occlusion field is defined on a open and connected set $S$, but I cannot see how $S$ is constructed. I assume that $S$ is the space left after *filtering the unimportant points whose field values are too large*, but the field is constructed with cross-products, and it is not clear which norm is used to define *large*. Moreover, $S$ clash with the notation of the scene point cloud (which is clearly not open and not connected). I believe this section needs to be reworked to provide a clear explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you! We have provided responses below.\n\n> A clear definition of per-point affordance in problem formulation.\n\nThanks for your suggestion! We follow [1] to define per-point affordance as an actionability score of each point on the object. Interactable means literally movable towards the intended direction to a threshold when interacted by a robot without collision. We will further provide more details in the introduction and problem formulation.\n\n\n> Clarification on the scene point cloud.\n\nFollowing settings of [1, 2, 3, 4], we use point clouds (without color) obtained from a depth camera as scene point cloud. Thanks for this comment and we will further clarify it in problem formulation (line 118).\n\n\n> Segmentation mask.\n\nPlease see the **Clarification on the segmentation mask** Section in **Global Response TEXT** for clarification.\n\n\n> Clarification on robot-target conditioned occlusion field.\n\nSorry for making you confused. For the **definition and property** of the robot-target conditioned occlusion field $F$, $F$ is defined in the whole $R^3$ space excluding the object $O$ (i.e., $R^3$\\\\ $O$). We query each occlusion point $P \\in R^3$ \\\\ $O$ to get field values on the occlusions, and use the queried results to filter out the unimportant points in the scene.\nThus, we notate the robot-target conditioned occlusion field as open and connected since $R^3$ \\\\ $O$ is open and connected, not that the space that occlusion objects possess is open and connected.\nFor the **NORM to define field value**, our field value is defined as the NORM of the vector of cross-products. We use this to filter the unimportant points whose field values (NORM of the vector of cross-products) are too large.\nThanks again for pointing out and we will refine our expression. To avoid the clash with the notation of the scene point cloud $S$, we will use symbol $G$ to notate the space $R^3$ \\\\ $O$  in our refined paper.\n\n> Convex decomposition to pre-process simulation assets.\n\nTo reduce the gap between physics and rendering, we use the Voxelized Hierarchical Approximate Convex Decomposition (V-HACD) algorithm to decompose the 3D surface into a set of \"near\" convex parts, which is also used in the affordance learning work [5]. Specifically, V-HACD involves the following steps: 1. convert the 3D model into a voxel representation; 2. progressively merge voxels into larger ones while retaining the essential geometric features; 3. transform the merged convex voxels into convex polygons to further simplify the representation. After this procedure, the gap between physics and rendering is reduced and the manipulation can be conducted well. Thanks for this question! We will add more descriptions in the paper.\n\n> Robot position R v.s. transforming the point cloud. More details about the input point cloud.\n\nWe take **robot position R** as input instead of transforming the point cloud because we want that the model can be deployed before a robot is observed in a scene, and thus we can pre-choose different positions for predictions and find a suitable one, which is essential for future work of movable robots and arm-based cameras. Our canonical frame is based on the target object’s center for convenience. For rigid translation, since we normalize the scene point cloud centered at the object’s center, our model prediction is invariant to rigid translation of the entire scene. For rigid rotation, since point clouds of different perspectives can be rotated into the same perspective with differences in the vacancy of point clouds given camera intrinsics and extrinsics, our method is robust to the complexity of perspectives (Figure C in the **Global Response PDF** provides affordance predictions on point clouds scanned from different perspectives with promising consistency).\n\n\n> Setup and more details of the real-world experiments.\n\nFor **point cloud**, we use Microsoft’s Azure Kinect Depth Scanner to scan the depth image of the scene, and project the depth image to point cloud. Instead of fine-tuning our model on real-world data (collecting real-world data in diverse scenarios is costly), we directly deploy our model pre-trained on prefect point clouds in real-world point clouds, and get promising affordance. A series of papers have demonstrated point-level affordance learned in simulation is robust and can generate promising results when directly deployed on real-world point clouds (one possible reason: the noises do not influence a lot on the actionable information reflected by the local geometry of each point, which is what point-level affordance focuses on): Figure 6 in [1], Figure 5 in [2], Figure 6 in [3], Figure 1, 7 in [4] and Figure 8 in supplementary in [4]. Moreover, compared to the ZED-MINI Scanner and RealSense Scanner used in [2, 3, 4], point clouds scanned by our Microsoft’s Azure Kinect Depth Scanner are less noisy (which can be observed from the scanned point clouds shown in Figure 7 in our paper and scanned point clouds shown in above-mentioned figures in [2, 3, 4]), and thus the sim2real gap is smaller for our work on real-world point clouds. For **robot and control**, we use Franka Panda as the robot arm (shown in Figure 7 in the main paper, Figure 7 in supplementary pdf and 3:35 - 4:09 in the video), the same as the robot in simulation. We use the Robot Operating System (ROS) to control the robot.\n\n> Inference time when deployed on real scenes.\n\nOur inference is very quick thanks to the wonderful architecture of PointNet++. It only takes 0.18s for one scene inference on average.\n\n\n[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.\n\n[2] VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. ICLR, 2022. \n\n[3] AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. ECCV, 2022.\n\n[4] DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Object Manipulation. ICLR, 2023."}]

Author rebuttal:
We extend our gratitude to reviewers for their careful reading, meticulous feedback and valuable insights.
We are glad that reviewers unanimously agree that our work (1) proposes meaningful environment-aware affordance (zyYG: “a step up from previous work”, ntD4: “novel and insightful”, wNB2: “address several important limitations”), (2) contains many insightful designs (eVFQ:  “novel and interesting”, zyYG: “several novelties”, ntD4: “intriguing”) and (3) shows comprehensive experiments (eVFQ: “soundly demonstrate advantage”, zyYG: “good analysis”, wnB2: “adequate”).

In this **Global Response TEXT**, we provide clarifications to 3 common concerns.


> Geometry or configuration of the robot. (Reviewer gQVm&zyYG)

We clarify this concern by respectively clarifying (1.1) using 1 robot type is a realistic setting that previous works employ; (1.2) in this setting, the training data without the robot point cloud implicitly reflects robot geometry/configuration; (2) our framework can accommodate scenarios with multiple robot types when encoding robot point clouds.

For (1.1), contrary to various targets, robot types are generally limited, and thus existing robotic manipulation works [1, 3, 4, 5] and real-world scenarios usually use 1 type of robot/gripper. We follow them and use Franka Panda.

For (1.2), in this single-robot-type setting, affordance can be learned without inputting robot point clouds. Given that only one robot type features across scenes, both its initial configuration and point cloud (geometry) remain constant, and the only difference comes from the robot's position R. So R is enough to represent robots’ difference across different scenes. The interaction results between the robot and different targets can reflect its geometry and configuration. Specifically, supervised by actionable information of robot-object interaction, the learned affordance includes the robot’s geometry and configuration from interaction results implicitly. For this reason, previous manipulation works [1, 3, 4, 5] learn the manipulation affordance/policy without encoding the robot point cloud.

For (2), when existing many types of robot, we have extended our model to take the robot point cloud as extra input, on 3 different types (xArm7, Jaco2 and Franka Panda). Illustrated in Figure A of **Global Response PDF**, our learned affordance adeptly mirrors robots’ configuration and geometry. We provide a detailed analysis of affordance visualization in the figure caption.

Such **ability of PointNet++ encoders to reflect geometry of two interacting objects** is also proven in O2OAfford [2] which studies affordance between objects.



> Challenges and failure modes. (Reviewer gQVm&eVFQ)

After analyzing the visualization gallery of our failed cases, our failures basically follows 2 modes:

(1) The starting position of the robot is closely adjacent to some occluders, and the robot initially collides with occlusions even in small-scale movement. **This case is demonstrated in the first row of Figure D in the Global Response PDF**. This mode accounts for about *55%* of the failure cases.

(2) Though the environment-aware affordance model proposes a plausible target point, the following actions of robot joints lead to collisions. Given the target point (i.e., the final position of the gripper), we employ inverse kinematics (IK), a prevalent method in robotics to calculate the robot joint configurations that would result in a desired end-effector pose.
However, IK can produce varied joint configurations that lead to the same endpoint for the gripper. Some samples may cause a robot link to have large rotations/translations when approaching target. Consequently, these links risk colliding with occluders. **This case is demonstrated in the second row of Figure D in the Global Response PDF**. This mode accounts for about *25%* of the failure cases. Note that, even though IK is the most commonly used for solving the joint variables and works well with RRT in non-occlusion scenarios, we believe in our novel difficult setting with multiple occluders, the joint motions should be better planned. As all baselines use IK for moving joints, it is clear that our proposed environment-aware affordance has already proposed more plausible points as potential targets.


These 2 modes together account for about *80%* of failure cases. Equipped with our environment-aware actionable priors to propose plausible target manipulation points, these cases further require a policy to produce fine-grained action sequences of each robot joint. We recognize this as a prospective avenue for future research. Our environment-aware affordance serves as a pioneering step in this direction, highlighting the significance of identifying credible target points.



> Segmentation mask. (Reviewer ntD4&wNB2)

As our framework needn’t consider each occluder one by one, we only need 1 mask of the target object, and the points left are on occluders. This setting is similar to previous affordance learning works [1, 3, 4] that require the mask of target parts.
In simulation, the mask can be acquired by the simulator. For real cases, we can easily acquire the mask of the target object using segmentation models like SAM [6].


We appreciate the valuable and insightful comments from reviewers! We will add more details and discussions in the paper.

[1] Where2Act: From Pixels to Actions for Articulated 3D Objects. ICCV, 2021.

[2] O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning. CoRL, 2021.

[3] VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. ICLR, 2022. 

[4] AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions. ECCV, 2022.

[5] UMPNet: Universal Manipulation Policy Network for Articulated Objects. ICRA, 2022.

[6] Segment Anything. ArXiv, 2023.

Meta-review: This paper received positive reviewer opinions after the discussion period.  The main concerns expressed during the discussion revolved around more clearly explaining in the writeup some important elements: the problem definition, technical details of the encoder network architecture, and delineating the contributions of this work relative to prior work.  The author responses during the disucssion alleviated these concerns.  The AC finds no basis to overturn the reviewer consensus and therefore recommends acceptance.  The AC strongly recommends a thorough editing pass to incorporate the clarifications and improvements from this discussion period when preparing the camera ready version.
Decision: Accept (poster)