SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruihai Wu $^{1,4*}$  Kai Cheng $^{2*}$
Yan Shen $^{1,4}$  Chuanruo Ning $^{2}$  Guanqi Zhan $^{3}$  Hao Dong $^{1,4\dagger}$
$^{1}$ CFCS, School of CS, PKU  $^{2}$ School of EECS, PKU  $^{3}$ University of Oxford
$^{4}$ National Key Laboratory for Multimedia Information Processing, School of CS, PKU
Perceiving and manipulating 3D articulated objects in diverse environments is essential for home-assistant robots. Recent studies have shown that point-level affordance provides actionable priors for downstream manipulation tasks. However, existing works primarily focus on single-object scenarios with homogeneous agents, overlooking the realistic constraints imposed by the environment and the agent's morphology, e.g., occlusions and physical limitations. In this paper, we propose an environment-aware affordance framework that incorporates both object-level actionable priors and environment constraints. Unlike object-centric affordance approaches, learning environment-aware affordance faces the challenge of combinatorial explosion due to the complexity of various occlusions, characterized by their quantities, geometries, positions and poses. To address this and enhance data efficiency, we introduce a novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations. Experiments demonstrate the effectiveness of our proposed approach in learning affordance considering environment constraints.
Articulated objects, such as doors and drawers, exist everywhere in our daily life. Perceiving and manipulating these objects present crucial yet challenging tasks in computer vision and robotics. Unlike rigid objects, articulated objects exhibit diverse articulation types and functionally important articulated parts crucial for human and robot interactions. Numerous research endeavors have been investigating articulated objects broadly, encompassing joint parameters estimation [41, 47], part pose estimation [22, 23], kinematic structure estimation [34, 33], digital twins generalization [13, 10], articulated part robotic manipulation [25, 44, 46, 4] and few-shot policy adaptation [42].
However, most existing works for manipulating articulated objects primarily focus on single-object scenarios with homogeneous agents, such as flying grippers [46, 25, 44] or fixed-position robot arms [6]. Consequently, these approaches tend to develop object-centric representations and policies, neglecting the realistic constraints imposed by both the environment and the agent's morphology. These constraints are commonplace in real-world scenarios and their oversight limits the applicability and performance of the manipulation tasks. For example, successfully opening a cabinet door that is obstructed by occluders not only depends on the properties of the target door but also heavily relies on the robot's position and the way it interacts (e.g., colliding or bypassing) with the occluders.
We take a significant step towards manipulating articulated objects in a more realistic setting, i.e., considering constraints imposed by the environment and robot. Such a task encounters the combi-
*Equal contribution. Author ordering determined by coin flip.
Figure 1: Learning Environment-Aware Affordance for Articulated Object Manipulation. As the complexity of occluder combinations grows exponentially, we leverage the property that target manipulation point (Red Point) conditioned significant parts of occluders that impact the manipulation usually have limited local areas (Red Point's corresponding Red Box) even in complex scenes. Aware of such significant parts, our model can be trained on one-occluder scenes (Train) and generalize to multiple occluder combinations (Test, Affordance). The learned affordance provides actionable information for articulated object manipulation (Manipulation). Our model predicts reasonable affordance on real-world scanned point clouds (Real-world Scan).
natorial explosion challenge in complexity. To be specific, the substantial variability of occluders, characterized by their numbers, geometries, positions, and poses, leads to exponential complexity in scene data distribution [49, 18]. This introduces a significant obstacle to training a model that can comprehensively understand the diversity and intricacies of different scenes within the limits of available data. Moreover, sampling-based motion planning [19, 35], a kind of commonly used approach for planning and manipulation, also faces performance degradation with increasing occluders [38] as the probability of valid sampling rapidly diminishes.
We propose the use of point-level representations to tackle the challenge outlined above. Specifically, we exploit a notable property: even in situations where the combinations of occluders are complex, given the target manipulation point and the robot, the occluder parts that significantly affect the manipulation are typically confined to a limited area. This is clearly illustrated in Figure 1 (Row 1, Column 2), where, in the case of pushing a drawer obstructed by multiple occluders, the portions of the occluders that could potentially collide with the robot are confined to a specific area (shown as areas in the two boxes with bright colors in the heatmap). This confinement results from the predetermined position of the robot and the target manipulation point, which limits the range of possible trajectories for the manipulation. Hence, when we contemplate the manipulation of the target object at the point level, the complexity of these significant scene components remains manageable, despite the possible addition of unseen occluders.
Further, we take inspiration from affordance for robotic manipulation, which provides actionable priors of the target at the point level and thus guides manipulation policies, and has exhibited remarkable efficacy in 3D articulated object manipulation [25, 44, 42, 6] and other manipulation tasks [54, 26, 43]. Unlike these works that focus solely on single objects, we introduce the concept of environment-aware affordance for articulated object manipulation, which integrates object-centric actionable priors with environment constraints at the point level. Specifically, in a scene featuring a target articulated object, diverse occlusion objects, and a robot at different positions, we aim to learn the per-point actionable information of the target object, taking into account the constraints imposed by the environment and the agent. Furthermore, in light of Gibson's theory of affordance
(the originator of affordance theory), where affordance is defined as the different possibilities of action that the environment offers to an agent [7], our proposed task is not only more realistic but also more closely aligned with this foundational theory compared to preceding works.
The point-level representation we propose above shows the potential to train a significant-part-aware model in simple scenes while capable of generalizing to more complex scenes. This approach leverages the aforementioned point-level significant parts, whose complexity remains manageable across a spectrum from simple to complicated scenes. To facilitate learning of significant-part-aware scene representations, we design a robot-target conditioned occlusion field.
Further, we introduce a contrastive learning method that refines our learned representations to selectively disregard insignificant elements while maintaining sensitivity to similar, significant parts across varied scenes. Consequently, our proposed learning framework can be trained on scenes with a single occluder, yet generalizable to scenes with numerous novel occluder combinations. This strategy effectively mitigates the issue of combinatorial explosion in a data-efficient manner.
We conduct experiments using SAPIEN physics simulator [45] equipped with large-scale PartNet-Mobility [27] and ShapeNet [1] datasets. To assess our framework's performance and data efficiency, we initially train it on scenes containing only one occluder, with additional augmented contrastive scenes that incorporate an extra occluder. We then test our framework on significantly more complex scenes, featuring diverse combinations of novel occluders. Experimental results yield convincing evidence of the effectiveness and data-efficiency of our proposed framework.
- We explore the task of manipulating articulated objects within environment constraints and formulate the task of environment-aware affordance learning for manipulating 3D articulated objects, incorporating object-centric per-point priors and environment constraints.
- To tackle the combinatorial explosion problem in scene complexity, we propose a data-efficient framework capable of training on scenes featuring a single occluder and generalizing to scenes with complex occluder combinations, leveraging contrastive learning and point-level local significant part representations.
- We establish benchmarking multi-object full-robot (as opposed to flying grippers) environments in SAPIEN simulator. Results show our framework learns environment-aware affordance generalizable well to novel scenes with complex novel occluder combinations.
2.1 Visual Affordance for Robotic Manipulation
Visual affordance [7] is a kind of representation that indicates possible ways for robots to interact with the target and complete tasks. Many works study affordance for the classic grasping task in robotics [24, 28, 3, 17, 16, 50], while there exist many current works on point-level affordance indicating object geometrics for articulated object manipulation [25, 44, 42, 5, 6, 29], dual-gripper collaboration [54] and object to object interaction [26]. Tracing back to Gibson [7], the proposer of affordance, however, affordance notates the opportunities of interactions that involve consideration of the environment constraints and the robot, such as the occluders and the robot morphology, and thus ought to be aware of the robot and environment. Therefore, we propose the environment-aware affordance learning task, incorporating both the object actionable priors and environment constraints.
2.2 Occlusion Handling
Occlusion is a significant challenge for current computer vision and robotics systems. In computer vision, there have been previous works trying to handle occlusion in different tasks, including object detection [39, 14, 52], instance segmentation [48, 14, 53] and tracking [9]. Besides, to intrinsically handle the occlusion, another series of work aims to recover the entire shape of occluded objects, i.e., amodal segmentation [55, 11, 52].
In robotics, the occlusion problem mainly exists in object retrieval [20, 12], grasping [40, 51, 36] or rearrangement [37, 2, 21] in clutters. In our paper, we study the task of learning the manipulation of 3D articulated objects with cluttered occlusions in front of the target object, which involves consideration of not only the observations of occlusions and the target, but also the agent's relationship with them.
We formulate a new challenging task, Environment-Aware Affordance, aiming to infer point-level affordance to indicate the actionable information for a full robot arm in different positions to manipulate 3D articulated objects under diverse occlusions. While previous point-level affordance learning works only consider flying grippers [25, 42, 44, 54] or fixed arms [6] to manipulate the target object without occluders, we further consider the robot location as well as scene occlusions.
Specifically, in our formulated task, given as input a 3D scene point cloud  $S$  with  $n$  points  $p_1, p_2, \ldots, p_n$ , containing a target articulated object  $T$  with  $m$  points  $T_{p_1}, T_{p_2}, \ldots, T_{p_m}$ ,  $k$  occluder object point cloud  $O_1, O_2, \ldots, O_k$ , and the robot position  $R \in \mathbb{R}^3$ , the model is required to predict the point-level affordance score  $a_{T_{p_i}|R,S}$  on each target point  $T_{p_i}$  of  $T$ , indicating how likely the point is interactable. The point-level affordance is able to guide 3D articulated object manipulation tasks.
As the complexity of occluder combinations grows exponentially, making it difficult and time-consuming to collect enough occluder combinations for training, we propose a data-efficient method that trains on only a few scenes with a single occluder, which can generalize to scenes with multiple occluder combinations.
Figure 2: Our Proposed Data-efficient Framework for Learning Environment-Aware Affordance. Our model takes an occluded scene point cloud and robot position as input. Our framework generates per-point occlusion fields indicating most significant local parts of occluders. Then, it predicts per-point affordance using extracted features of each target point, its corresponding occlusion field, and robot position. The trained model can generalize to novel multi-occluder scenes. Red points denote manipulation points over the target object.
Our framework is composed of four networks: Robot Encoder  $\mathcal{E}_R$ , Target Encoder  $\mathcal{E}_{T_p|S}$ , Scene Encoder  $\mathcal{E}_{S|T_p,R}$  and Affordance Predictor  $A_{T_p|R,S}$ . While  $\mathcal{E}_R$  and  $\mathcal{E}_{T_p}$  extracts the robot and target representations  $f_R$  and  $f_{T_p|S}$  (Sec. 4.2), the most important component of our framework is learning the robot-target conditioned scene representations that are sensitive to the significant local parts in the scene for manipulation (Sec. 4.3), which uses  $\mathcal{E}_{S|T_p,R}$  to extract the scene representations  $f_{S|T_p,R}$  from a designed robot-target conditioned occlusion field. This component makes the model sensitive to robot-target point conditioned significant parts and thus empowers the model with the generalization in novel complicated scenes. Finally,  $A_{T_p|R,S}$  takes  $f_R$ ,  $f_{T_p|S}$  and  $f_{S|T_p,R}$ , and predicts the point-level environment-aware affordance score  $a_{T_p|R,S}$  on  $T_p$  (Sec. 4.4).
4.2 Robot and Target Representations
Both the robot position and the target manipulation point (including its position and local geometry) will affect the environment-aware affordance. For example, when the robot position changes, the affordance will change according to the robot's new reach area and interaction modes with occluders. Similarly, when the target manipulation point changes, the originally interactable point will become non-interactive for various reasons, such as robot being unable to reach the new point, the local geometry of the target point not supporting the desired action (e.g., a smooth surface does not support pulling), or the robot colliding with occluders while attempting to move to the new point. So we design a Robot Encoder  $\mathcal{E}_R$  and a Target Encoder  $\mathcal{E}_{T_p|S}$  to extract relevant information.
As shown in Figure 2, we employ a MLP network as  $\mathcal{E}_R$ , which encodes the robot position  $R$  into a latent representation  $f_{R} \in \mathbb{R}^{128}$  and a Segmentation-version of PointNet++ [31] that encodes the scene point cloud into per-point features, where  $f_{T_p|S} \in \mathbb{R}^{128}$  represents the feature of  $T_{p}$ .
4.3 Robot-Target Conditioned Scene Representations
In scenes with multiple occluders, the quantity, geometry and arrangement of occluders demonstrate rich diversity. However, given the robot position and the target manipulation point, most points in the scene are unlikely to significantly affect the manipulation, and the area of the significant parts that do have an impact (e.g., the area around occluders where potential collisions may occur during robot movement) is usually not large. By focusing on these significant parts of the scene, the change of occluder number and geometry will not matter, and a model can effectively learn to generalize to more complex scenes with multiple novel occlusion combinations.
To this end, we design the robot-target conditioned occlusion field to facilitate learning significant scene representations conditioned on the robot and the target point (Sec. 4.3.1), and propose the robot-target conditioned contrastive learning to further enhance learned scene representations (Sec. 4.3.2).
4.3.1 Robot-Target Conditioned Occlusion Field
Although a scene may contain a large number of points (of occlusions and the target object), when we know the target manipulation point and the robot position, the point number of the significant part is not large, and the distance between these points and either the robot or the target manipulation point is not far. Therefore, we design the robot-target conditioned occlusion field that maps the scene into a vector field conditioned on both the target point and the robot, facilitating learning scene representations highlighting significant parts for representations.
We define the Occlusion Field as a conditional continuous vector field  $\mathbf{F}$  on an open and connected set  $\mathbf{G} = \mathbb{R}^3\setminus \mathbf{Object}\subset \mathbb{R}^3$ , which can be represented by a value function  $\mathbf{F}_{R,T_p}:\mathbf{G}\to \mathbb{R}^3$
$$
\begin{array}{l} \mathbf {F} _ {R, T _ {p}} (x, y, z) = \mathbf {V} _ {\mathbf {R}} \times \mathbf {V} _ {\mathbf {T} _ {\mathbf {p}}} \\ = \left\langle x - x _ {R}, y - y _ {R}, z - z _ {R} \right\rangle \otimes \left\langle x - x _ {T _ {p}}, y - y _ {T _ {p}}, z - z _ {T _ {p}} \right\rangle \\ = \left\langle F _ {1}, F _ {2}, F _ {3} \right\rangle . \forall (x, y, z) \in \mathbf {G}. \tag {1} \\ \end{array}
$$
The field factors  $\mathbf{V}_{\mathbf{R}}$  and  $\mathbf{V}_{\mathbf{T_p}}$  are separately conditional on  $R$  and  $T_{p}$ . For any point  $(x,y,z)$  in  $\mathbf{G}$ , its value vector  $\mathbf{F}_{R,T_p}(x,y,z)$  is calculated as the cross product  $(\otimes)$  of the Euclidean vectors from  $(x,y,z)$  to robot  $R$  and target  $T_{p}$ , two fixed points of  $\mathbf{F}_{R,T_p}$ . Since the field is continuous, we have:
$$
\lim  _ {p \rightarrow T _ {p}} \mathbf {F} _ {R, T _ {p}} (x, y, z) = \lim  _ {p \rightarrow R} \mathbf {F} _ {R, T _ {p}} (x, y, z) = 0. \tag {2}
$$
So the model can filter the unimportant points whose field values are too large.
We employ a PointNet [30] network as the Scene Encoder  $\mathcal{E}_{S|T_p,R}$ . It takes as input the conditional vector field  $\mathbf{F}_{R,T_p}$  mapped from the scene point cloud  $S$  selected with small field values, and then outputs the robot-target conditioned scene representations  $f_{S|T_p,R}$ .
4.3.2 Robot-Target Conditioned Contrastive Learning
The above-learned robot-target conditioned scene representations  $f_{S|T_p,R}$  should be sensitive to the meaningful local parts of the occluders for manipulation, while agnostic to unimportant occluders and their local parts, despite the number, geometry and combinations of them. Specifically, the learned
scene representations should have the following properties: (1) given the same target point, when a new occluder occurs in the scene while not affecting the manipulation, the learned representations will keep the same; (2) given the same scene, when the target point changes, the meaningful local parts of scene as well as the learned representations will correspondingly change.
To better empower scene representations with such properties, which can further boost the performance and data-efficiency in affordance prediction, we propose the robot-target conditioned contrastive learning method for learning scene representations.
As shown in Figure 3, for predicting the point affordance  $\hat{a}_{T_p|R,S}$  for  $T_{R,S}(p)\in \mathcal{X}_{\mathcal{R},S}$  task in single-occluder scene  $S$  , the target point  $T_{p}$  and the robot  $R$  , we correspondingly generate a positive and a negative task. Specifically, we generate the task of predicting the point affordance  $\hat{a}_{T_{p^{\prime}}|R,S}$  with the same scene  $S$  drawn from augmentation distribution  $A(\cdot \mid T_{R,S}(p))$  ,robot  $R$  and a different target point  $T_{p^{\prime}}$  drawn from marginal distribution  $A(\cdot) = \mathbb{E}_{T_{R,S}(p)}A(\cdot \mid T_{R,S}(p))$  ,for the reason that although the scene keeps unchanged, the target point changes and thus the significant part in occluders will correspondingly change, and thus the scene representations should change. Also, we generate the task of predicting the point affordance  $\hat{a}_{T_p|R,S'}$  with the same robot  $R$  ,target point  $T_{p}$  ,and a new scene  $S^{\prime}$  which adds to the original scene  $S$  with an occluder that is uninfluential to the manipulation. Even though the scene changes from  $S$  to  $S^{\prime}$  ,the significant part in the occluders keeps unchanged, and thus the target-robot conditioned scene representations should keep the same. We leverage this prior to conduct our contrastive learning, which performs better representation learning for the occlusion field.
The original task and its positive and negative paired tasks constitute triplets, so we use triplet loss [32] to learn the scene representations of them contrastively ( $\alpha$  is the boundary constant):
Figure 3: Robot-Target Conditioned Contrastive Learning. For each point affordance prediction task (Middle,  $\hat{a}_{T_p|R,S}$ ), we respectively generate its corresponding positive (Up, same target point with an extra occluder  $\hat{a}_{T_p|R,S'}$ ) and negative (Down, different target point  $\hat{a}_{T_{p'}|R,S}$ ) paired tasks, and use triplet loss to learn the scene representations.
$$
\mathcal {L} _ {T _ {p} | R, S} ^ {C L} = \left\| f _ {S \mid T _ {p}, R} - f _ {S ^ {\prime} \mid T _ {p}, R} \right\| _ {2} ^ {2} - \left\| f _ {S \mid T _ {p}, R} - f _ {S \mid T _ {p ^ {\prime}}, R} \right\| _ {2} ^ {2} + \alpha \tag {3}
$$
4.4 Affordance Prediction
We propose the Affordance Predictor  $A_{T_p|R,S}$  that aggregates the information of the target, the robot and the scene to predict the point-level environment-aware affordance.
We employ a MLP as the Affordance Predictor  $A_{T_p|R,S}$  that takes  $f_{R}$ ,  $f_{T_p|S}$  and  $f_{S|T_p,R}$  as input, and predicts the affordance score  $\hat{a}_{T_p|R,S} \in R$  on each target point  $T_{p}$  conditioned on  $\hat{R}$  and  $S$ .
We apply  $\mathcal{L}_1$  loss to measure the error between the affordance prediction  $\hat{a}_{T_p|R,S}$  and the ground-truth affordance score  $a_{T_p|R,S}$  on a certain target point  $T_p$ :
$$
\mathcal {L} _ {T _ {p} | R, S} ^ {A F F} = \mathcal {L} _ {1} \left(a _ {T _ {p} | R, S}, \hat {a} _ {T _ {p} | R, S}\right). \tag {4}
$$
The total loss for the whole framework is then defined as:
$$
\mathcal {L} _ {T _ {p} | R, S} ^ {\text {t o t a l}} = \mathcal {L} _ {T _ {p} | R, S} ^ {A F F} + \lambda_ {C L} \cdot \mathcal {L} _ {T _ {p} | R, S} ^ {C L}, \tag {5}
$$
where  $\lambda_{CL}$  is a balancing coefficient.
For tasks, we follow Where2Act [25] and use the point-level affordance predictions of pushing and pulling articulated parts (doors and drawers) as our tasks. In a scene with occluders, a target point and a robot, the ground-truth affordance score of a target point is set to be  $0 / 1$  when the robot fails / succeeds in pushing or pulling the target point without colliding occluders.
For simulation and dataset, we use SAPIEN [45] as our simulation environment, equipped with largescale Partnet-Mobility dataset [27] and ShapeNet [1] dataset, with occluder data statistics as shown in Table 4. Besides, we use cabinets and tables as target objects, for the reason that in the real world there often exist many occluders in front of them.
For training, we collect interactions in one-occluder scenes. Specifically, we collect 900 successful and 900 failure interactions for pushing, and 300 successful and 2500 failure interactions for pulling. Pulling needs more failure interactions as most points are not pullable. For each data,