SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Train-Cats.	All	Basket	Bottle	Bowl	Box
Train-Data	367	77	16	128	17
Test-Data	128	31	4	44	5
Bucket	Chair	Pot	TrashCan
9	20	5	10
Novel-Cats.	All	Dispenser	Jar	Kettle	FoldChair
Test-Data	589	9	528	26	26
Table 1: Occluder Dataset Statistics. We use 1,084 different shapes in ShapeNet [1] and PartNet-Mobility [27], covering 12 commonly seen indoor occluder categories. We use 8 training categories (split into 367 training shapes and 128 test shapes), and 4 novel categories with 589 shapes.
we additionally collect a positive and a negative paired interaction for contrastive learning. For testing, we use multi-occluder scenes respectively in training category test shapes and novel categories.
For evaluation, to evaluate the accuracy of affordance prediction, we follow Where2Act and use F-Score and Average Precision. To evaluate predicted affordance's capability in providing actionable priors for manipulation in scenes, we introduce Sample Manipulation Accuracy metric sma:
$$
s m a = \frac {\# \text {s u c c e s s f u l} \text {i n t e r a c t i o n s}}{\# \text {t o t a l} \text {i n t e r a c t i o n s}} \tag {6}
$$
The total interactions are proposed based on affordance predictions over the whole scene. We randomly adopt proposals with a confidence threshold and compute the manipulation accuracy.
5.2 Baselines and Ablations
We compare our method with the following baselines that learn affordance for manipulation:
- Where2Act (W2A) [25] that abstracts the robot arm into a flying gripper and learns point-level actionable affordance for manipulating articulated objects.
- W2A-R [25] that uses a robot arm instead of a flying gripper in Where2Act's setting, and adds a robot encoder in Where2Act's networks.
- O2O-Afford (O2O) [26] that learns point-level affordance for object-to-object interactions. We use the robot and the scene as two input objects.
- O2O-M that trains O2O-Afford using scenes containing multiple occluders.
To ensure fair comparison, we train the baselines using both originally generated data and their corresponding positive and negative paired data collected for contrastive learning.
Moreover, we compare our method with Collision Avoidance RTT Planner (RTT-CA) [19, 35, 8], a commonly used sampling-based planner for robotic manipulation, to demonstrate the capability of point-level affordance in guiding manipulation in complicated scenes.
In addition, we compare our method with two versions that respectively ablate one core component:
- Ours w/o OF that learns environment-aware affordance without the occlusion field.
- Ours w/o CL that learns environment-aware affordance without contrastive learning.
Table 2, 3 and Figure 4 demonstrate the superiority of our framework quantitatively and qualitatively. Results of W2A (highlighting every actionable point) demonstrate the necessity in considering the robot instead of a flying gripper in more realistic settings.
Results of W2A-R and O2O (they have similar qualitative results and we show visualization of W2A-R) that consider robot arms demonstrate that, our framework can better recognize the novel occluder part (e.g. the back of foldingchair that hinders pulling the handles in the middle doors, shown in Line 2, Column 3 and 5 in Figure 4) in novel complicated scenes.
Results of O20-M demonstrate that, when the amount of training data is not large, training on complex multi-occluder scenes cannot achieve performance comparable to our data-efficient method.
Results of Ours w/o OF and Ours w/o CL demonstrate the necessities of two components. With occlusion field, our model better recognizes target areas that may exist collision with novel-shaped occluders (e.g., the drawer board hindered by the basket in Line 1, the door edge hindered by in Line 2, Figure 5). With contrastive learning, our model better discriminates between actionable and non-actionable points, while Ours w/o CL predicts reasonable affordance but with less certainty.
Comparisons between RTT-CA and affordance methods show that, while sampling-based planners may face difficulty in complicated scenes, point-level affordance may be more suitable by directly providing per-point actionable priors for manipulating articulated objects within complex constraints.
Figure 4: Qualitative Comparisons between Our Method and Baselines.
Figure 5: Qualitative Comparisons between Our Method and Ablations.
Ours w/o OF
Ours w/o CL
Task	RRT-CA	W2A	O2O	O2O-M	W2A-R	Ours w/o OF	Ours w/o CL	Ours
pushing	23.65	17.80	34.60	37.33	34.76	39.43	36.63	43.52
pulling	10.31	3.82	29.93	29.36	27.55	33.19	28.02	36.19
Table 2: Sample Manipulation Accuracy(%)
Method	pushing	pushing (novel)	pulling	pulling (novel)
W2A	64.70 / 52.97	62.80 / 47.07	66.42 / 37.59	60.37 / 40.73
W2A-R	68.66 / 71.31	64.86 / 69.55	34.06 / 72.56	26.79 / 73.79
O2O	65.18 / 77.07	59.04 / 72.18	46.03 / 74.06	43.42 / 73.56
O2O-M	61.06 / 67.82	67.43 / 68.54	39.76 / 66.42	33.29 / 61.47
Ours w/o OF	64.28 / 69.02	58.31 / 66.94	36.36 / 67.19	33.33 / 64.30
Ours w/o CL	68.59 / 74.02	64.70 / 71.62	65.74 / 75.40	55.25 / 71.92
Ours	77.30 / 86.11	76.11 / 83.58	71.28 / 75.96	61.32 / 75.18
Table 3: Quantitative Evaluations and Comparisons with Baselines and Ablated Versions. In each entry, we report F-Score(%) and Average Precision(%) before and after slash.
Figure 6: Predicted Affordance Changes Conditioned on Different Robot Rositions.
Figure 7: Predicted Affordance on Real-World Scans.
Additionally, Figure 6 shows our method generates different environment-aware affordance conditioned on different robot positions in the same scene. Figure 7 demonstrates our framework generates promising results by testing on different real-world scenes. It is worth mentioning, as denoted in the red circles, our model not only learns constraints from occluders, but also learns to avoid manipulating points that may lead to collisions with other parts of the object (i.e., self occlusion).
We introduce environment-aware affordance for manipulating 3D articulated objects within environment constraints, leveraging point-level representations to address the combinatorial explosion challenge in scene complexity. Using an interactive environment built upon SAPIEN and the PartNet-Mobility and ShapeNet datasets, we train neural networks that predict per-point actionable information for manipulating articulated 3D objects under occlusions. We present extensive quantitative evaluations and qualitative analyses of the proposed method. Results show that the learned priors are highly localized and thus generalizable to novel scenes with unseen occluder combinations.
Ethics Statement. Our work has the potential to enable robots on articulated object manipulation in complicated scenes. The learned environment-aware affordance avoids collisions in manipulation, and thus reduces the risk of accident. We do not see our work has any particular harm or issue.
This work was supported by National Natural Science Foundation of China - General Program (62376006) and The National Youth Talent Support Program (8200800081).
[48] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11141-11150, 2021.
[49] Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? International Journal of Computer Vision, 129:781-802, 2021.
[50] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757. IEEE, 2018.
[51] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research, 41(7):690-705, 2022.
[52] Guanqi Zhan, Weidi Xie, and Andrew Zisserman. A tri-layer plugin to improve occluded detection. British Machine Vision Conference, 2022.
[53] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervised scene de-occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3784–3792, 2020.
[54] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper object manipulation. International Conference on Learning Representations (ICLR), 2023.
[55] Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dólar. Semantic amodal segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1464-1472, 2017.
A More Details on Simulation and Settings
Following Where2Act [25], we design our interactive simulation environment based on SAPIEN, using the same set of simulation parameters for all interaction trials.
For general simulation settings, we use frame rate 500 fps, tolerance length 0.001, tolerance speed 0.005, solver iterations 20 (for constraint solvers related to joints and contacts), with Persistent Contact Manifold (PCM) disabled (for better simulation stability), with disabled sleeping mode (i.e. no locking for presumably still rigid bodies in simulation), and all the other settings as default in SAPIEN release.
For physical simulation, we use the standard gravity 9.81, static friction coefficient 4.0, dynamic friction coefficient 4.0, and restitution coefficient 0.01. For the object articulation dynamics simulation, we use stiffness 0 and damping 10.
For the rendering, we use OpenGL-based rasterization rendering for the fast speed of simulation. We set three point lights around the object (one at the front, one from back-left and one from back-right) for lighting the scene, with mild ambient lighting as well. The camera is set to have near plane 0.1, far plane 100, resolution 448, and field of view  $35^{\circ}$ .
For 3D partial point cloud scan inputs, we back-project the depth image into a foreground point cloud, by rejecting the far-away background depth pixels, and then perform furthest point sampling to get a 10K-size point cloud scan.
For robot arm movement, we use RRT Planner [35, 8, 19] equipped with PID controller to generate and execute a certain path towards the target.
For an interaction trial to be considered successful, it not only needs to cause considerable part motion along intended direction. To avoid the extreme data unbalance in pulling data, we manually set handle mask on our simulator and assign half of the interactions on the handles. To simplify the consideration of different interaction directions' impact on affordance, we set every interaction to move along the normal direction of the target point.
B More Data Details and Visualization
In Table 4, we summarize our data statistics. In Fig. 8, we visualize our simulation assets from ShapeNet [1] and PartNet [27] that we use in this work.
Train-Cats	All	Basket	Bottle	Bowl	Box	Bucket	Chair	Pot	TrashCan
Train-Data	367	77	16	128	17	27	61	16	25
Test-Data	128	31	4	44	5	9	20	5	10
Test-Cats	All	Dispenser	Jar	Kettle	FoldingChair
Test-Data	589	9	528	26	26
Table 4: Occluder Dataset Statistics. We use 1,084 different shapes in ShapeNet [1] and PartNet-Mobility [27], covering 12 commonly seen indoor occluder categories. We use 8 training categories (split into 367 training shapes and 128 test shapes), and 4 test categories with 589 shapes networks have never seen in training.
Figure 8: Our simulation assets from ShapeNet [1] and PartNet [27].
C More Training Details
C.1 Hyper-parameters
We set the batch size to 30, and use Adam Optimizer [15] with 0.001 as the initial learning rate.
We use const 2.00 as the boundary constant in  $\alpha$  contrastive learning, and 1.00 as the balancing coefficient  $\lambda_{CL}$  in the total loss.
C.2 Computing Resources
We use PyTorch as our Deep Learning framework, and single RTX GeForce 3090 (20GB GPU) for training and inference.
D More Results and Analysis
Fig. 9 10 11 12 demonstrate comparisons with baselines and ablations. Fig. 13 shows the whole occlusion fields. Fig. 14 shows real-world demonstrations with analysis in the caption.
E Discussion and Visualization on Failure Cases
We present some interesting failure cases in Fig. 15. From these examples, we see the difficulty of the task. Also, given the current problem formulation, there are some intrinsically ambiguous cases that are generally hard for robots to figure out from a single static snapshot. Moreover, our affordance is trained on data generated with robots deploying a naive policy, e.g., pushing the normal direction of the target plane. Better policies may introduce better affordance.
Figure 9: More Qualitative Comparisons between Our Method and Baselines in Pushing.
W2A
ConW2A
M-O2O
Figure 10: More Qualitative Comparisons between Our Method and Ablations in Pushing.
Figure 11: More Qualitative Comparisons between Our Method and Baselines in Pulling.
W2A-R
O2O-M
Figure 12: More Qualitative Comparisons between Our Method and Ablations in Pulling.
Occlusion Field 1
Figure 13: Visualization of the whole Occlusion Fields.
Occlusion Field 2
Input 2
Collide with Painter
Collide with Pot
W2A
Object-Centric
Out of Reach
Manipulation
Ours
Environment-Aware
Successful
Manipulation
Figure 14: Real-World Demonstrations of Manipulation Policy Guided by Object-Centric Affordance and Our Proposed Environment-Aware Affordance. It is clear that environment-aware affordance can help avoid out-of-reach situations and collisions with other self-parts or objects.
Figure 15: Failure cases. Up: robot initialized close to occluders. Down: Motion planning method leads the robot to collide with occluders when approaching a plausible target point.
- Target Point
$\bigcirc$  Collision Area
F Future Work on Robot-Target Conditioned Contrastive Learning
Limited to simulator configuration, our contrastive learning method only considers a limited augmentation distribution  $A(\cdot \mid \bar{x})$  for each anchor scene  $\bar{x} \in \mathcal{X}$  while the marginal distribution  $A(\cdot) = \mathbb{E}_{\bar{x}}A(\cdot \mid \bar{x})$  is complete. The augmentation distribution  $A(\cdot \mid \bar{x})$  only includes one more occluder at the edge of  $\bar{x}$ , and neglects the potential augmentation methods by choosing similar target points. Future methods can be applied with a better similarity metric of comparing different things and improve our self-supervised learning paradigm.