Title: Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Affordance for Robotics

Articulated Object Manipulation

Occlusion Handling
Abstract: Perceiving and manipulating 3D articulated objects in diverse environments is essential for home-assistant robots. Recent studies have shown that point-level affordance provides actionable priors for downstream manipulation tasks. However, existing works primarily focus on single-object scenarios with homogeneous agents, overlooking the realistic constraints imposed by the environment and the agent's morphology, e.g., occlusions and physical limitations. In this paper, we propose an environment-aware affordance framework that incorporates both object-level actionable priors and environment constraints. Unlike object-centric affordance approaches, learning environment-aware affordance faces the challenge of combinatorial explosion due to the complexity of various occlusions, characterized by their quantities, geometries, positions and poses. To address this and enhance data efficiency, we introduce a novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations. Experiments demonstrate the effectiveness of our proposed approach in learning affordance considering environment constraints.

Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions

Abstract

Perceiving and manipulating 3D articulated objects in diverse environments is essential for home-assistant robots. Recent studies have shown that point-level affordance provides actionable priors for downstream manipulation tasks. However, existing works primarily focus on single-object scenarios with homogeneous agents, overlooking the realistic constraints imposed by the environment and the agent's morphology, e.g. , occlusions and physical limitations. In this paper, we propose an environment-aware affordance framework that incorporates both object-level actionable priors and environment constraints. Unlike object-centric affordance approaches, learning environment-aware affordance faces the challenge of combinatorial explosion due to the complexity of various occlusions, characterized by their quantities, geometries, positions and poses. To address this and enhance data efficiency, we introduce a novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations. Experiments demonstrate the effectiveness of our proposed approach in learning affordance considering environment constraints.

1 Introduction

Articulated objects, such as doors and drawers, exist everywhere in our daily life. Perceiving and manipulating these objects present crucial yet challenging tasks in computer vision and robotics. Unlike rigid objects, articulated objects exhibit diverse articulation types and functionally important articulated parts crucial for human and robot interactions. Numerous research endeavors have been investigating articulated objects broadly, encompassing joint parameters estimation [41, 47], part pose estimation [22, 23], kinematic structure estimation [34, 33], digital twins generalization [13, 10], articulated part robotic manipulation [25, 44, 46, 4] and few-shot policy adaptation [42].

However, most existing works for manipulating articulated objects primarily focus on single-object scenarios with homogeneous agents, such as flying grippers [46, 25, 44] or fixed-position robot arms [6]. Consequently, these approaches tend to develop object-centric representations and policies, neglecting the realistic constraints imposed by both the environment and the agent's morphology. These constraints are commonplace in real-world scenarios and their oversight limits the applicability and performance of the manipulation tasks. For example, successfully opening a cabinet door that is obstructed by occluders not only depends on the properties of the target door but also heavily relies on the robot's position and the way it interacts ( e.g. , colliding or bypassing) with the occluders.

We take a significant step towards manipulating articulated objects in a more realistic setting, i.e. , considering constraints imposed by the environment and robot. Such a task encounters the combi-

Affordance

Train

Push

Pull

Manipulation

Real-World

Scan

Manipulation

Affordance

Guide

Test

Target Point

Significant Part

0.75

0.5

0.25

natorial explosion challenge in complexity. To be specific, the substantial variability of occluders, characterized by their numbers, geometries, positions, and poses, leads to exponential complexity in scene data distribution [49, 18]. This introduces a significant obstacle to training a model that can comprehensively understand the diversity and intricacies of different scenes within the limits of available data. Moreover, sampling-based motion planning [19, 35], a kind of commonly used approach for planning and manipulation, also faces performance degradation with increasing occluders [38] as the probability of valid sampling rapidly diminishes.

We propose the use of point-level representations to tackle the challenge outlined above. Specifically, we exploit a notable property: even in situations where the combinations of occluders are complex, given the target manipulation point and the robot, the occluder parts that significantly affect the manipulation are typically confined to a limited area. This is clearly illustrated in Figure 1 (Row 1, Column 2), where, in the case of pushing a drawer obstructed by multiple occluders, the portions of the occluders that could potentially collide with the robot are confined to a specific area (shown as areas in the two boxes with bright colors in the heatmap). This confinement results from the predetermined position of the robot and the target manipulation point, which limits the range of possible trajectories for the manipulation. Hence, when we contemplate the manipulation of the target object at the point level , the complexity of these significant scene components remains manageable, despite the possible addition of unseen occluders.

Further, we take inspiration from affordance for robotic manipulation, which provides actionable priors of the target at the point level and thus guides manipulation policies, and has exhibited remarkable efficacy in 3D articulated object manipulation [25, 44, 42, 6] and other manipulation tasks [54, 26, 43]. Unlike these works that focus solely on single objects, we introduce the concept of environment-aware affordance for articulated object manipulation, which integrates object-centric actionable priors with environment constraints at the point level. Specifically, in a scene featuring a target articulated object, diverse occlusion objects, and a robot at different positions, we aim to learn the per-point actionable information of the target object, taking into account the constraints imposed by the environment and the agent. Furthermore, in light of Gibson's theory of affordance

(the originator of affordance theory), where affordance is defined as the different possibilities of action that the environment offers to an agent [7], our proposed task is not only more realistic but also more closely aligned with this foundational theory compared to preceding works.

The point-level representation we propose above shows the potential to train a significant-part-aware model in simple scenes while capable of generalizing to more complex scenes. This approach leverages the aforementioned point-level significant parts, whose complexity remains manageable across a spectrum from simple to complicated scenes. To facilitate learning of significant-part-aware scene representations, we design a robot-target conditioned occlusion field.

Further, we introduce a contrastive learning method that refines our learned representations to selectively disregard insignificant elements while maintaining sensitivity to similar, significant parts across varied scenes. Consequently, our proposed learning framework can be trained on scenes with a single occluder, yet generalizable to scenes with numerous novel occluder combinations. This strategy effectively mitigates the issue of combinatorial explosion in a data-efficient manner.

We conduct experiments using SAPIEN physics simulator [45] equipped with large-scale PartNetMobility [27] and ShapeNet [1] datasets. To assess our framework's performance and data efficiency, we initially train it on scenes containing only one occluder, with additional augmented contrastive scenes that incorporate an extra occluder. We then test our framework on significantly more complex scenes, featuring diverse combinations of novel occluders. Experimental results yield convincing evidence of the effectiveness and data-efficiency of our proposed framework.

In summary, we make the following contributions:

We explore the task of manipulating articulated objects within environment constraints and formulate the task of environment-aware affordance learning for manipulating 3D articulated objects, incorporating object-centric per-point priors and environment constraints.

To tackle the combinatorial explosion problem in scene complexity, we propose a dataefficient framework capable of training on scenes featuring a single occluder and generalizing to scenes with complex occluder combinations, leveraging contrastive learning and pointlevel local significant part representations.

We establish benchmarking multi-object full-robot (as opposed to flying grippers) environments in SAPIEN simulator. Results show our framework learns environment-aware affordance generalizable well to novel scenes with complex novel occluder combinations.

2 Related Work

2.1 Visual Affordance for Robotic Manipulation

Visual affordance [7] is a kind of representation that indicates possible ways for robots to interact with the target and complete tasks. Many works study affordance for the classic grasping task in robotics [24, 28, 3, 17, 16, 50], while there exist many current works on point-level affordance indicating object geometrics for articulated object manipulation [25, 44, 42, 5, 6, 29], dual-gripper collaboration [54] and object to object interaction [26]. Tracing back to Gibson [7], the proposer of affordance, however, affordance notates the opportunities of interactions that involve consideration of the environment constraints and the robot, such as the occluders and the robot morphology, and thus ought to be aware of the robot and environment. Therefore, we propose the environment-aware affordance learning task, incorporating both the object actionable priors and environment constraints.

2.2 Occlusion Handling

Occlusion is a significant challenge for current computer vision and robotics systems. In computer vision, there have been previous works trying to handle occlusion in different tasks, including object detection [39, 14, 52], instance segmentation [48, 14, 53] and tracking [9]. Besides, to intrinsically handle the occlusion, another series of work aim to recover the entire shape of occluded objects, i.e. , amodal segmentation [55, 11, 52].

In robotics, the occlusion problem mainly exists in object retrieval [20, 12], grasping [40, 51, 36] or rearrangement [37, 2, 21] in clutters. In our paper, we study the task of learning the manipulation of 3D articulated objects with cluttered occlusions in front of the target object, which involves consideration of not only the observations of occlusions and the target, but also the agent's relationship with them.

3 Problem Formulation

We formulate a new challenging task, Environment-Aware Affordance , aiming to infer pointlevel affordance to indicate the actionable information for a full robot arm in different positions to manipulate 3D articulated objects under diverse occlusions. While previous point-level affordance learning works only consider flying grippers [25, 42, 44, 54] or fixed arms [6] to manipulate the target object without occluders, we further consider the robot location as well as scene occlusions.

Specifically, in our formulated task, given as input a 3D scene point cloud S with n points p 1 , p 2 , ..., p n , containing a target articulated object T with m points T p 1 , T p 2 , ..., T p m , k occluder object point cloud O 1 , O 2 , ..., O k , and the robot position R ∈ R 3 , the model is required to predict the point-level affordance score a T p i | R,S on each target point T p i of T , indicating how likely the point is interactable. The point-level affordance is able to guide 3D articulated object manipulation tasks.

As the complexity of occluder combinations grows exponentially, making it difficult and timeconsuming to collect enough occluder combinations for training, we propose a data-efficient method that trains on only a few scenes with a single occluder, which can generalize to scenes with multiple occluder combinations.

Robot Feature

Scene

Encoder

Robot

Encoder

Affordance

Predictor

···

Per-point Feature

···

···

···

···

Novel Scene Generalization

R

Novel Scene Generalization

···

Pulling

Pushing

𝐅

𝑹,𝑻

𝒑𝒌

𝐅

𝑹,𝑻

𝒑𝟏

𝐅

𝑹,𝑻

𝒑$

Scene

𝓔

𝑹

Target

Encoder

𝒇

𝑹

𝑨

𝑻

𝒑

|𝑹,𝑺

𝓔

𝑺&𝐅

𝐓 𝒑 ,𝑹

𝒇

𝒑𝟏|𝑺

𝑺

𝒇

𝒑𝟐|𝑺

𝒇

𝒑𝒌|𝑺

Robot

Position

𝓔

𝑻

𝒑

|𝑺

0.75

0.25

4 Method

4.1 Overview

Our framework is composed of four networks: Robot Encoder E R , Target Encoder E T p | S , Scene Encoder E S | T p ,R and Affordance Predictor A T p | R,S . While E R and E T p extracts the robot and target representations f R and f T p | S (Sec. 4.2), the most important component of our framework is learning the robot-target conditioned scene representations that are sensitive to the significant local parts in the scene for manipulation (Sec. 4.3), which uses E S | T p ,R to extract the scene representations f S | T p ,R from a designed robot-target conditioned occlusion field. This component makes the model sensitive to robot-target point conditioned significant parts and thus empowers the model with the generalization in novel complicated scenes. Finally, A T p | R,S takes f R , f T p | S and f S | T p ,R , and predicts the point-level environment-aware affordance score a T p | R,S on T p (Sec. 4.4).

4.2 Robot and Target Representations

Both the robot position and the target manipulation point (including its position and local geometry) will affect the environment-aware affordance. For example, when the robot position changes, the affordance will change according to the robot's new reach area and interaction modes with occluders. Similarly, when the target manipulation point changes, the originally interactable point will become non-interactive for various reasons, such as robot being unable to reach the new point, the local geometry of the target point not supporting the desired action ( e.g. , a smooth surface does not support pulling), or the robot colliding with occluders while attempting to move to the new point. So we design a Robot Encoder E R and a Target Encoder E T p | S to extract relevant information.

As shown in Figure 2, we employ a MLP network as E R , which encodes the robot position R into a latent representation f R ∈ R 128 and a Segmentation-version of PointNet++ [31] that encodes the scene point cloud into per-point features, where f T p | S ∈ R 128 represents the feature of T p .

4.3 Robot-Target Conditioned Scene Representations

In scenes with multiple occluders, the quantity, geometry and arrangement of occluders demonstrate rich diversity. However, given the robot position and the target manipulation point, most points in the scene are unlikely to significantly affect the manipulation, and the area of the significant parts that do have an impact ( e.g. , the area around occluders where potential collisions may occur during robot movement) is usually not large. By focusing on these significant parts of the scene, the change of occluder number and geometry will not matter, and a model can effectively learn to generalize to more complex scenes with multiple novel occlusion combinations.

To this end, we design the robot-target conditioned occlusion field to facilitate learning significant scene representations conditioned on the robot and the target point (Sec. 4.3.1), and propose the robottarget conditioned contrastive learning to further enhance learned scene representations (Sec. 4.3.2).

4.3.1 Robot-Target Conditioned Occlusion Field

Although a scene may contain a large number of points (of occlusions and the target object), when we know the target manipulation point and the robot position, the point number of the significant part is not large, and the distance between these points and either the robot or the target manipulation point is not far. Therefore, we design the robot-target conditioned occlusion field that maps the scene into a vector field conditioned on both the target point and the robot, facilitating learning scene representations highlighting significant parts for representations.

We define the Occlusion Field as a conditional continuous vector field F on an open and connected set G = R 3 \ Object ⊂ R 3 , which can be represented by a value function F R,T p : G → R 3 ,

The field factors V R and V T p are seperately conditional on R and T p . For any point ( x, y, z ) in G , its value vector F R,T p ( x, y, z ) is calculated as the cross product ( ⊗ ) of the Euclidean vectors from ( x, y, z ) to robot R and target T p , two fixed points of F R,T p . Since the field is continuous, we have:

So the model can filter the unimportant points whose field values are too large.

We employ a PointNet [30] network as the Scene Encoder E S | T p ,R . It takes as input the conditional vector field F R,T p mapped from the scene point cloud S selected with small field values, and then outputs the robot-target conditioned scene representations f S | T p ,R .

4.3.2 Robot-Target Conditioned Contrastive Learning

The above-learned robot-target conditioned scene representations f S | T p ,R should be sensitive to the meaningful local parts of the occluders for manipulation, while agnostic to unimportant occluders and their local parts, despite the number, geometry and combinations of them. Specifically, the learned

scene representations should have the following properties: (1) given the same target point, when a new occluder occurs in the scene while not affecting the manipulation, the learned representations will keep the same; (2) given the same scene, when the target point changes, the meaningful local parts of scene as well as the learned representations will correspondingly change.

To better empower scene representations with such properties, which can further boost the performance and data-efficiency in affordance prediction, we propose the robot-target conditioned contrastive learning method for learning scene representations.

As shown in Figure 3, for predicting the point affordance ˆ a T p | R,S for T R,S ( p ) ∈ X R , S task in single-occluder scene S , the target point T p and the robot R , we correspondingly generate a positive and a negative task. Specifically, we generate the task of predicting the point affordance ˆ a T p ′ | R,S with the same scene S drawn from augmentation distribution A ( · | T R,S ( p )) , robot R and a different target point T p ′ drawn from marginal distribution A ( · ) = E T R,S ( p ) A ( · | T R,S ( p )) , for the reason that although the scene keeps unchanged, the target point changes and thus the significant part in occluders will correspondingly change, and thus the scene representations should change. Also, we generate the task of predicting the point affordance ˆ a T p | R,S ′ with the same robot R , target point T p , and a new scene S ′ which adds to the original scene S with an occluder that is uninfluential to the manipulation. Even though the scene changes from S to S ′ , the significant part in the occluders keeps unchanged, and thus the target-robot conditioned scene representations should keep the same. We leverage this prior to conduct our contrastive learning, which performs better representation learning for the occlusion field.

The original task and its positive and negative paired tasks constitute triplets, so we use triplet loss [32] to learn the scene representations of them contrastively ( α is the boundary constant):

Scene

Encoder

Scene

Encoder

Scene

Encoder

Triplet

Loss

Share Weight

Share Weight

Positive

Negative

Original

4.4 Affordance Prediction

We propose the Affordance Predictor A T p | R,S that aggregates the information of the target, the robot and the scene to predict the point-level environment-aware affordance.

We employ a MLP as the Affordance Predictor A T p | R,S that takes f R , f T p | S and f S | T p ,R as input, and predicts the affordance score ˆ a T p | R,S ∈ R on each target point T p conditioned on R and S .

We apply L 1 loss to measure the error between the affordance prediction ˆ a T p | R,S and the ground-truth affordance score a T p | R,S on a certain target point T p :

The total loss for the whole framework is then defined as:

where λ CL is a balancing coefficient.

5 Experiments

5.1 Settings

For tasks , we follow Where2Act [25] and use the point-level affordance predictions of pushing and pulling articulated parts (doors and drawers) as our tasks. In a scene with occluders, a target point and a robot, the ground-truth affordance score of a target point is set to be 0 / 1 when the robot fails / succeeds in pushing or pulling the target point without colliding occluders.

For simulation and dataset , we use SAPIEN [45] as our simulation environment, equipped with largescale Partnet-Mobility dataset [27] and ShapeNet [1] dataset, with occluder data statistics as shown in Table 4. Besides, we use cabinets and tables as target objects, for the reason that in the real world there often exist many occluders in front of them.

For training , we collect interactions in one-occluder scenes. Specifically, we collect 900 successful and 900 failure interactions for pushing, and 300 successful and 2500 failure interactions for pulling. Pulling needs more failure interactions as most points are not pullable. For each data,

Table 1: Occluder Dataset Statistics. We use 1,084 different shapes in ShapeNet [1] and PartNet-Mobility [27], covering 12 commonly seen indoor occluder categories. We use 8 training categories (split into 367 training shapes and 128 test shapes), and 4 novel categories with 589 shapes.

we additionally collect a positive and a negative paired interaction for contrastive learning. For testing , we use multi-occluder scenes respectively in training category test shapes and novel categories.

For evaluation , to evaluate the accuracy of affordance prediction, we follow Where2Act and use F-Score and Average Precision . To evaluate predicted affordance's capability in providing actionable priors for manipulation in scenes, we introduce Sample Manipulation Accuracy metric sma :

The total interactions are proposed based on affordance predictions over the whole scene. We randomly adopt proposals with a confidence threshold and compute the manipulation accuracy.

5.2 Baselines and Ablations

We compare our method with the following baselines that learn affordance for manipulation:

Where2Act (W2A) [25] that abstracts the robot arm into a flying gripper and learns pointlevel actionable affordance for manipulating articulated objects.

W2A-R [25] that uses a robot arm instead of a flying gripper in Where2Act's setting, and adds a robot encoder in Where2Act's networks.

O2O-Afford (O2O) [26] that learns point-level affordance for object-to-object interactions. We use the robot and the scene as two input objects.

O2O-M that trains O2O-Afford using scenes containing multiple occluders.

To ensure fair comparison, we train the baselines using both originally generated data and their corresponding positive and negative paired data collected for contrastive learning.

Moreover, we compare our method with Collision Avoidance RTT Planner ( RTT-CA ) [19, 35, 8], a commonly used sampling-based planner for robotic manipulation, to demonstrate the capability of point-level affordance in guiding manipulation in complicated scenes.

In addition, we compare our method with two versions that respectively ablate one core component:

Ours w/o OF that learns environment-aware affordance without the occlusion field.

Ours w/o CL that learns environment-aware affordance without contrastive learning.

5.3 Results and Analysis

Table 2, 3 and Figure 4 demonstrate the superiority of our framework quantitatively and qualitatively.

Results of W2A (highlighting every actionable point) demonstrate the necessity in considering the robot instead of a flying gripper in more realistic settings.

Results of W2A-R and O2O (they have similar qualitative results and we show visualization of W2A-R ) that consider robot arms demonstrate that, our framework can better recognize the novel occluder part ( e.g. the back of foldingchair that hinders pulling the handles in the middle doors, shown in Line 2, Column 3 and 5 in Figure 4) in novel complicated scenes.

Results of O2O-M demonstrate that, when the amount of training data is not large, training on complex multi-occluder scenes cannot achieve performance comparable to our data-efficient method.

Results of Ours w/o OF and Ours w/o CL demonstrate the necessities of two components. With occlusion field , our model better recognizes target areas that may exist collision with novel-shaped occluders ( e.g. , the drawer board hindered by the basket in Line 1, the door edge hindered by in Line 2, Figure 5). With contrastive learning , our model better discriminates between actionable and non-actionable points, while Ours w/o CL predicts reasonable affordance but with less certainty.

Comparisons between RTT-CA and affordance methods show that, while sampling-based planners may face difficulty in complicated scenes, point-level affordance may be more suitable by directly providing per-point actionable priors for manipulating articulated objects within complex constraints.

W2A

W2A-R

O2O-M

Ours

Push

Pull

Ours

Ours w/o OF

Ours w/o CL

Push

Pull

position1

position2

Push

Pull

Additionally, Figure 6 shows our method generates different environment-aware affordance conditioned on different robot positions in the same scene. Figure 7 demonstrates our framework generates promising results by testing on different real-world scenes. It is worth mentioning, as denoted in the red circles, our model not only learns constraints from occluders, but also learns to avoid manipulating points that may lead to collisions with other parts of the object ( i.e. , self occlusion).

6 Conclusion

We introduce environment-aware affordance for manipulating 3D articulated objects within environment constraints, leveraging point-level representations to address the combinatorial explosion challenge in scene complexity. Using an interactive environment built upon SAPIEN and the PartNet-Mobility and ShapeNet datasets, we train neural networks that predict per-point actionable information for manipulating articulated 3D objects under occlusions. We present extensive quantitative evaluations and qualitative analyses of the proposed method. Results show that the learned priors are highly localized and thus generalizable to novel scenes with unseen occluder combinations.

Ethics Statement. Our work has the potential to enable robots on articulated object manipulation in complicated scenes. The learned environment-aware affordance avoids collisions in manipulation, and thus reduces the risk of accident. We do not see our work has any particular harm or issue.

7 Acknowledgment

This work was supported by National Natural Science Foundation of China - General Program (62376006) and The National Youth Talent Support Program (8200800081).

References

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Sang Hun Cheong, Brian Y Cho, Jinhwi Lee, ChangHwan Kim, and Changjoo Nam. Where to relocate?: Object rearrangement inside cluttered and confined environments for robotic manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 7791-7797. IEEE, 2020.

Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5031-5041, 2020.

Ben Eisner*, Harry Zhang*, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. In Robotics: Science and Systems (RSS) , 2022.

Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15752-15761, October 2021.

Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-toend affordance learning for robotic manipulation. International Conference on Robotics and Automation (ICRA) , 2023.

James J Gibson. The theory of affordances. Hilldale, USA , 1(2):67-82, 1977.

Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659 , 2023.

Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, and Carl Vondrick. Tracking through containers and occluders in the wild, 2023.

Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G Schwing. Sailvos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3105-3115, 2019.

Baichuan Huang, Shuai D Han, Jingjin Yu, and Abdeslam Boularias. Visual foresight trees for object retrieval from clutter with nonprehensile rearrangement. IEEE Robotics and Automation Letters , 7(1):231-238, 2021.

Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion-aware instance segmentation with overlapping bilayers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 4019-4028, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. The 3rd International Conference for Learning Representations , 2015.

Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters , 5(2):3352-3359, 2020.

Mia Kokic, Johannes A. Stork, Joshua A. Haustein, and Danica Kragic. Affordance detection for task-specific grasping using deep learning. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids) , pages 91-98, 2017.

Adam Kortylewski, Qing Liu, Angtian Wang, Yihong Sun, and Alan Yuille. Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion. International Journal of Computer Vision , 129:736-760, 2021.

James J Kuffner and Steven M LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) , volume 2, pages 995-1001. IEEE, 2000.

Andrey Kurenkov, Joseph Taglic, Rohun Kulkarni, Marcus Dominguez-Kuhne, Animesh Garg, Roberto Martín-Martín, and Silvio Savarese. Visuomotor mechanical search: Learning to retrieve target objects in clutter. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8408-8414. IEEE, 2020.

Jinhwi Lee, Younggil Cho, Changjoo Nam, Jonghyeon Park, and Changhwan Kim. Efficient obstacle rearrangement for object manipulation tasks in cluttered environments. In 2019 International Conference on Robotics and Automation (ICRA) , pages 183-189. IEEE, 2019.

Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song. Categorylevel articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3706-3715, 2020.

Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but geometric constraints: A model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088 , 2020.

Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In IEEE International Conference on Robotics and Automation (ICRA) , 2021.

Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In International Conference on Computer Vision (ICCV) , 2021.

Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O2O-Afford: Annotationfree large-scale object-object affordance learning. In Conference on Robot Learning (CoRL) , 2021.

Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Luis Montesano and Manuel Lopes. Learning grasping affordances from local visual descriptors. In 2009 IEEE 8th international conference on development and learning , pages 1-6. IEEE, 2009.

Chuanruo Ning, Ruihai Wu, Haoran Lu, Kaichun Mo, and Hao Dong. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593 , 2016.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

Rafał Staszak, Milena Molska, Kamil Młodzikowski, Justyna Ataman, and Dominik Belter. Kinematic structures estimation on the rgb-d images. In 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) , volume 1, pages 675-681. IEEE, 2020.

Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research , 41:477-526, 2011.

Ioan A Sucan, Mark Moll, and Lydia E Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine , 19(4):72-82, 2012.

Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13438-13444. IEEE, 2021.

Bingjie Tang and Gaurav S Sukhatme. Selective object rearrangement in clutter. In Conference on Robot Learning , pages 1001-1010. PMLR, 2023.

Hao Tian, Chaoyang Song, Changbo Wang, Xinyu Zhang, and Jia Pan. Sampling-based planning for retrieving near-cylindrical objects in cluttered scenes using hierarchical graphs. IEEE Transactions on Robotics , 2022.

Angtian Wang, Yihong Sun, Adam Kortylewski, and Alan L Yuille. Robust object detection under occlusion with context-aware compositionalnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12645-12654, 2020.

Chenxi Wang, Hao-Shu Fang, Minghao Gou, Hongjie Fang, Jin Gao, and Cewu Lu. Graspness discovery in clutters for fast and accurate grasp detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15964-15973, 2021.

Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8876-8884, 2019.

Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. European conference on computer vision (ECCV 2022) , 2022.

Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV) , 2023.

Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations , 2022.

Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Zhenjia Xu, He Zhanpeng, and Shuran Song. Umpnet: Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters , 2022.

Zhihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver van Kaick, Hao Zhang, and Hui Huang. RPM-NET: Recurrent prediction of motion and parts from point cloud. ACM Trans. on Graphics , 38(6):Article 240, 2019.

Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11141-11150, 2021.

Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? International Journal of Computer Vision , 129:781-802, 2021.

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3750-3757. IEEE, 2018.

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research , 41(7):690-705, 2022.

Guanqi Zhan, Weidi Xie, and Andrew Zisserman. A tri-layer plugin to improve occluded detection. British Machine Vision Conference , 2022.

Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Selfsupervised scene de-occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3784-3792, 2020.

Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper object manipulation. International Conference on Learning Representations (ICLR) , 2023.

Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1464-1472, 2017.

Appendix

A More Details on Simulation and Settings

Following Where2Act [25], we design our interactive simulation environment based on SAPIEN, using the same set of simulation parameters for all interaction trials.

For general simulation settings , we use frame rate 500 fps, tolerance length 0.001, tolerance speed 0.005, solver iterations 20 (for constraint solvers related to joints and contacts), with Persistent Contact Manifold (PCM) disabled (for better simulation stability), with disabled sleeping mode ( i.e. no locking for presumably still rigid bodies in simulation), and all the other settings as default in SAPIEN release.

For physical simulation , we use the standard gravity 9.81, static friction coefficient 4.0, dynamic friction coefficient 4.0, and restitution coefficient 0.01. For the object articulation dynamics simulation, we use stiffness 0 and damping 10.

For the rendering , we use OpenGL-based rasterization rendering for the fast speed of simulation. We set three point lights around the object (one at the front, one from back-left and one from back-right) for lighting the scene, with mild ambient lighting as well. The camera is set to have near plane 0.1, far plane 100, resolution 448, and field of view 35 ◦ .

For 3D partial point cloud scan inputs , we back-project the depth image into a foreground point cloud, by rejecting the far-away background depth pixels, and then perform furthest point sampling to get a 10K-size point cloud scan.

For robot arm movement , we use RRT Planner [35, 8, 19] equipped with PID controller to generate and execute a certain path towards the target.

For an interaction trial to be considered successful, it not only needs to cause considerable part motion along intended direction. To avoid the extreme data unbalance in pulling data, we manually set handle mask on our simulator and assign half of the interactions on the handles. To simplify the consideration of different interaction directions' impact on affordance, we set every interaction to move along the normal direction of the target point.

B More Data Details and Visualization

In Table 4, we summarize our data statistics. In Fig. 8, we visualize our simulation assets from ShapeNet [1] and PartNet [27] that we use in this work.

Train-Cats

All

Basket

Bottle

Bowl

Box

Bucket

Chair

Pot

TrashCan

Train-Data

367

77

16

128

17

27

61

16

25

Test-Data

128

31

4

44

5

9

20

5

10

Test-Cats

All

Jar

Kettle

FoldingChair

Test-Data

589

9

528

26

26

Training Categories

Testing Categories

Basket

Chair

Bucket

Box

Bowl

Bottle

Pot

Trash Can

Jar

Kettle

Folding Chair

C More Training Details

C.1 Hyper-parameters

We set the batch size to 30, and use Adam Optimizer [15] with 0.001 as the initial learning rate.

We use const 2.00 as the boundary constant in α contrastive learning, and 1.00 as the balancing coefficient λ CL in the total loss.

C.2 Computing Resources

We use PyTorch as our Deep Learning framework, and single RTX GeForce 3090 (20GB GPU) for training and inference.

D More Results and Analysis

Fig. 9 10 11 12 demonstrate comparions with baselines and ablations. Fig. 13 shows the whole occlusion fields. Fig. 14 shows real-world demonstrations with analysis in the caption.

E Discussion and Visualization on Failure Cases

We present some interesting failure cases in Fig. 15. From these examples, we see the difficulty of the task. Also, given the current problem formulation, there are some intrinsically ambiguous cases that are generally hard for robots to figure out from a single static snapshot. Moreover, our affordance is trained on data generated with robots deploying a naive policy, e.g. , pushing the normal direction of the target plane. Better policies may introduce better affordance.

W2A

ConW2A

M-O2O

Ours

L

Ours w/o CL

Ours w/o OF

ConW2A

Ours

W2A

W2A-R

O2O-M

Ours

68866860860830

Ours

Ours w/o OF

Ours w/o CL

pushing

Input 2

Occlusion Field 2

Occlusion Field 1

Input 1

pulling

7

Push

Door

Pull

Door

Collide with Drawer

Collide with Pot

Successful

Successful

Out of Reach

W2A

Object-Centric

Manipulation

Pull

Drawer

Ours

Environment-Aware

Manipulation

Successful

A

Target Point

Collision Area

F Future Work on Robot-Target Conditioned Contrastive Learning

Limited to simulator configuration, our contrastive learning method only considers a limited augmentation distribution A ( · | ¯ x ) for each anchor scene ¯ x ∈ X while the marginal distribution A ( · ) = E ¯ x A ( · | ¯ x ) is complete. The augmentation distribution A ( · | ¯ x ) only includes one more occluder at the edge of ¯ x , and neglects the potential augmentation methods by choosing similar target points. Future methods can be applied with a better similarity metric of comparing different things and improve our self-supervised learning paradigm.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work formulates environment-aware affordance for articulated object manipulation, which incorporates both object-level per-point actionable priors and environment constraints. First, this work identifies limitations of previous works on object-level affordance learning, including over-simplified robot manipulators and environment occlusion; it proposes to incorporate robot and environment constraints into the per-point affordance score. Next, this work proposes a contrastive learning method to allow the learned affordance prediction model to generalize to complex and unseen occluder combinations. This work evaluates the success rate of the proposed method with open door and drawer tasks in simulated environments and show improvements over previous affordance learning methods, and it demonstrates that the learned affordance can be useful in real robotic tasks by showing real robot demos.\n\nSTRENGTHS:\n- This work has a clear goal and motivation. It addresses several important limitations of previous affordance learning works (over-simplified robots and environments), allowing per-point affordance prediction to be deployed on real robots to complete tasks in more complex and realistic environments.\n\n- This work provides adequate quantitative and qualitative experiments and comparison with baselines to show the effectiveness of the proposed method. It also justifies the necessity of 2 significant components, occlusion field and contrastive learning, with ablation studies.\n\nWEAKNESSES:\n1. This paper does not provide a clear definition of per-point affordance in problem formulation. I think this paper needs to provide a more comprehensive definition of \"affordance\" and \"interactable\" in introduction or section 3, preferably with concrete examples. Currently the motivation and goal of this paper is hard to understand without reading a previous work such as Where2Act.\n\n2. The problem formulation is also missing several key points.\n\n    - What is the scene point cloud? Based on the experiment section, it seems to be a colored point cloud obtained from a depth camera. This information needs be explained clearly in the formulation.\n   - How is the point cloud segmented into target point cloud and occluder point cloud? This information is available in the simulation, but how is it done in the real-world experiments?\n\n3. The robot-target conditioned occlusion field is not described clearly. I find section 4.3.1 very confusing. The occlusion field is defined on a open and connected set $S$, but I cannot see how $S$ is constructed. I assume that $S$ is the space left after *filtering the unimportant points whose field values are too large*, but the field is constructed with cross-products, and it is not clear which norm is used to define *large*. Moreover, $S$ clash with the notation of the scene point cloud (which is clearly not open and not connected). I believe this section needs to be reworked to provide a clear explanation.\n\nQUESTIONS:\n1. How are the simulation assets pre-processed? What convex decomposition is used? \n\n   - With default PartNet-Mobility and SAPIEN simulator, all assets would be processed into convex shapes and form a large gap between physics and rendering. Manipulation experiments performed on such data would not reflect a realistic scenario. So it is important to describe the data pre-processing method.\n\n2. Why robot position $R$ is used as input instead of transforming the point cloud? Is there some important choice on canonical frame that is not described in the text? How do you ensure the model prediction is invariant to a rigid transformation of the entire scene?\n\n3. What are the setup for the real-world experiments? How are the models trained and deployed? Is there finetuning on real-world data? Is there domain gaps when the model is trained on perfect point clouds in simulation and deployed on noisy data in the real world? \n\n4. What is the inference time for the method when deployed on real scenes?\n\nLIMITATIONS:\nThe paper provides a description of some limitations and I have raised questions of other potential limitations in the previous *Questions* section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the environment-aware affordance task, taking into account both object-centric affordance and environmental constraints. More specifically, the proposed task is designed to incorporate various occlusion cases and the robot's position.  In order to tackle the inherent challenge of combinatorial explosion, the paper introduces a novel framework that utilizes a local robot-target conditioned occlusion field and contrastive losses. This framework enables effective training on simpler scenarios with only one occluder, while also facilitating generalization to more complex scenes.\n\nSTRENGTHS:\n* The proposed task that takes the environment constraints into account is novel and insightful. Occlusion and robot position conditioned affordance is intuitive and potentially useful for downstream robot tasks.\n* The idea of utilizing a local robot-target occlusion field to enable generalization to more complex occlusion scenarios is straightforward and effective based on the experimental results.\n* The strategy employed to construct positive and negative pairs for the contrastive loss is intriguing, and the ablation studies confirm its efficacy in improving the scene encoder.\n\nWEAKNESSES:\n* In the proposed environment-aware affordance task, both the segmentation of the target object and the occluded objects are provided. For previous work like W2A, which only consider the object-centric affordance, it’s unclear whether the addition of simple heuristics can yield positive results. One possible heuristic is to examine the predicted affordance from W2A and determine if there are any points close to the points of occluder, subsequently filtering the affordance based on this observation. While I acknowledge the significance of environment-aware affordance for downstream tasks, I remain uncertain about the complexity and realism of the current proposed task. In addition, for the target objects, there are only cabinets and tables, it’s also unclear if the train and test scenario uses the same instance or different instances from the same category. Only two categories may not be challenging enough for the task.\n* For the robot position, it’s unclear how many positions are chosen in the train and test set. From the supplement video, there seems to be 4 different robot positions. It’s unclear if the method can generalize to new robot positions well.\n* For the push and pull action, it’s unclear if the default action direction is the surface normal. There should be more descriptions on the action selection and the dataset construction, including how to pick the occluder, how to add the occluder and how to judge if the action succeed considering the occluded objects.\n\nQUESTIONS:\n* What’s the performance of the trained model on scenes without any occlusion? And what will be the performance compared to the prior work in this scenario?\n* When inference, is the segmentation of the target object and all occluded objects are given? For the real case, how do you get the segmentation? \n* Is the number of points of the target object fixed for the network? How about the number of points for the occluded objects? If the number of points is not fixed, how to balance the number of points in each train example?\n\nLIMITATIONS:\nThe authors mention the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
Given a 3D point cloud and a robot, the method predicts per-point actionable affordance. More specifically it predicts whether a given point is pushable or pullable by the robot in a given position. The method takes into account environment occlusion. The paper presents a data-efficient training framework that allows training on a single occlusion and generalize to multiple occlusions. In addition, the paper presents a novel contrastive learning approach.

STRENGTHS:
- Acounting for occlusion when predicting affordance is a step up from previous work. It has not been tackled before as far as I know.
- The paper presents several novelties: problem setup and formulation, data-efficient framework, and contrastive learning approach
- The method outperforms previous work. 
- Paper is well written and easy to follow. 
- The results are well presented and accompanied by a good analysis

WEAKNESSES:
- The action space is too small. Only pushing and pulling are considered.
- The test setting seems too simple. Most of the results are shown on very similar desks. 
- Related work section is too short and does not clearly explain what is the gap in previous work, nor how the paper is filling that gap. 
- Section 4.3.1 is lacking a supporting figure
- The robot geometry is not taken into account.

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel environment-aware affordance prediction framework for robot interactions (pushing/pulling) with parts of articulated objects (drawers and doors). The proposed approach consists primarily of two novel improvements: a robot-target conditioned occlusion field and a contrastive learning formulation for improved data efficiency. The paper evaluates the proposed approach against where2act and O2O with metrics from those works and demonstrates a fair improvement (~6% overall and 2-10% in all individual categories). Real world trials demonstrate the usefulness of predicted point-level affordances for robotics tasks. Qualitative results demonstrate several clear wins for the approach related to prediction of occluding object constraints.\n\nSTRENGTHS:\nBased on a known simulation system and asset data, the evaluation and metrics soundly demonstrate the advantage of the proposed approach over prior art.\nTrials on down-stream manipulation tasks with a real robot platform validate the usefulness of the point-level affordances predicted by the approach.\nThe proposed improvements over prior art (contrastive learning and robot-target conditioned occlusion fields) appear novel and constitute interesting solutions to the challenges posed.\nThe advantage of training on simple occlusions and generalizing to more complex occlusion scenarios is a clear strength of the approach.\n\nWEAKNESSES:\nSome areas of the paper could be improved. I felt there was some information missing leaving me with questions. The high-level concepts were repeated often and could be deduplicated to make room for more details.\nThe line was slightly blurry between novel work and prior art. Clearly this approach builds upon W2A and O2O, but the paper would benefit from clarifying carry-over vs. novel formulations.\n\tSMA metric appears to be the ssr metric from W2A with a different name.\n\tSome discussion of F-Score and Average Precision metrics would help interpretation. Could be appendix if repeated from prior art.\n\tOverall accuracy improvement over SoTA was somewhat small (6% over prior methods to achieve 43/36 push/pull). \n\tThe paper talks up the approach without mentioning remaining challenges or failure modes.\n\nExposition could be improved with further polish and proofreading.\nGrammar(G)/Typos(T) (listed with line number) :\nG - 18 - perceiving and manipulating these objects present\nT - 39 - also faces experiences\nT - 107 - objectc\nT - 148 - extra period after fR\nG - 255 - We randomly adopt proposed\nT - 267 - Moreover to we \n274 - Table 3 2 (formatting? Comma or conjunction?)\nG - 284 - of that may exist collision\n\nQUESTIONS:\nI understand the value of robot and target encoder networks, but I didn’t find any satisfying details or citations to provide insight into what these networks produce/encode or how they are trained. Are these empty MLPs initialized with random weights and allowed to fill freely, or is there something else to them? Are they described more in prior art or specific to this work?\nI would like to better understand the intuition behind the Robot-Target Conditioned Occlusion Field. Why does the vector field of cross products of point-to-robot and point-to-target vectors provide a good heuristic? Some expanded text here would be valuable.\n\tWhat is a “novel” scene for the evaluation test metrics? Is it a scene with additional occluders that was not present in training?\nWhy does the pulling performance on novel scenes only improve 1-2%? Is this significantly harder to predict occlusions or is there a common failure mode that escapes the assumptions of the heuristic for the approach?\n\nLIMITATIONS:\nDiscussion of limitations seemed light. Only the lack of mobile base was mentioned. Given the maximum performance of 43/36% push/pull I expected to learn more about what wasn’t working or what additional challenges and failure modes were observed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for learning environment-aware affordance in occluded 3D object manipulation. It tackles combinatorial explosion and improves data efficiency. The authors showcase their superior framework using benchmarking multi-object full-robot environments in the SAPIEN simulator. Their contributions involve studying articulated object manipulation within environment constraints, proposing a data-efficient framework, and establishing benchmarking environments.\n\nSTRENGTHS:\n- The framework proposed addresses the learning of environment-aware affordances for 3D articulated object manipulation, even under occlusions.\n- The paper focuses on affordance learning and introduces benchmarking environments to enhance robotic system performance in real-world settings.\n- The paper includes a comprehensive description of the framework, establishes benchmarking environments, and highlights its superior performance.\n- It is well-structured, easy to comprehend, and effectively presents experimental results.\n\nWEAKNESSES:\n- The proposed framework has only been evaluated on simulated environments, and there is a lack of real-world experiments from different perspectives and scenarios.\n- This work primarily examines a static scene, which raises concerns about the applicability of the proposed framework in real-world scenarios that involve more complex and dynamic environments.\n- The authors have not provided a detailed analysis of the failure cases of the proposed framework.\n\nQUESTIONS:\n- L38/267 mention that sampling-based motion planning faces performance degradation with increasing occluders. Could you provide a benchmark of the time and cost of different methods to address this issue?\n-  L147 mention E_R encodes the robot position R into a latent representation. What about the geometry or configuration of the robot? How do you ensure that the scene and the configuration of the robot are plausible?\n- How would the proposed framework generalize to different types of robots and scenes?\n- In real-time executions, the state of point clouds is constantly changing, and the heatmap also changes. How do you plan to handle this issue?\n\nLIMITATIONS:\nThe authors have addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}