Title: Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects
Venue: NeurIPS 2023
Accepted: yes
Keywords: articulated object manipulation

few-shot learning

visual affordance for robotics
Abstract: Articulated object manipulation is a fundamental yet challenging task in robotics. Due to significant geometric and semantic variations across object categories, previous manipulation models struggle to generalize to novel categories. Few-shot learning is a promising solution for alleviating this issue by allowing robots to perform a few interactions with unseen objects. However, extant approaches often necessitate costly and inefficient test-time interactions with each unseen instance. Recognizing this limitation, we observe that despite their distinct shapes, different categories often share similar local geometries essential for manipulation, such as pullable handles and graspable edges - a factor typically underutilized in previous few-shot learning works. To harness this commonality, we introduce 'Where2Explore', an affordance learning framework that effectively explores novel categories with minimal interactions on a limited number of instances. Our framework explicitly estimates the geometric similarity across different categories, identifying local areas that differ from shapes in the training categories for efficient exploration while concurrently transferring affordance knowledge to similar parts of the objects. Extensive experiments in simulated and real-world environments demonstrate our framework's capacity for efficient few-shot exploration and generalization.

Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects

Abstract

Articulated object manipulation is a fundamental yet challenging task in robotics. Due to significant geometric and semantic variations across object categories, previous manipulation models struggle to generalize to novel categories. Few-shot learning is a promising solution for alleviating this issue by allowing robots to perform a few interactions with unseen objects. However, extant approaches often necessitate costly and inefficient test-time interactions with each unseen instance. Recognizing this limitation, we observe that despite their distinct shapes, different categories often share similar local geometries essential for manipulation, such as pullable handles and graspable edges - a factor typically underutilized in previous few-shot learning works. To harness this commonality, we introduce 'Where2Explore', an affordance learning framework that effectively explores novel categories with minimal interactions on a limited number of instances. Our framework explicitly estimates the geometric similarity across different categories, identifying local areas that differ from shapes in the training categories for efficient exploration while concurrently transferring affordance knowledge to similar parts of the objects. Extensive experiments in simulated and real-world environments demonstrate our framework's capacity for efficient few-shot exploration and generalization.

1 Introduction

Articulated objects, such as doors, drawers, scissors, and faucets, are ubiquitous in our daily lives. Therefore, the ability of robots to manipulate these objects is of critical importance. Many previous works have been done on perceiving and manipulating articulated objects [25, 35, 38, 3]. However, due to the significant variance in the objects' structure, 3D geometry, and articulation types across categories, developing efficient perception and manipulation systems that can generalize to those variations remains challenging [7, 25, 41].

An intuitive solution to equip models with generalized manipulation knowledge is training them on large-scale datasets. However, conducting real-world interactions with diverse objects or acquiring 3D models encompassing potential categories can be prohibitively time-consuming and costly. Moreover, this approach could still fail with the emergence of new object categories or designs ( e.g. , a cup with novel geometries resembling a gourd as shown in Figure 1).

Since encountering novel objects is inevitable in real-world applications, few-shot learning, which allows robots to propose interactions with novel objects and adapt their understanding to them, has emerged as a promising solution. However, previous few-shot learning works for articulated object manipulation usually focus on instance-level exploration and adaptation, requiring test-time interactions on each novel object [35, 27]. This limitation hinders the efficiency and safety of real-world applications of robots.

This paper investigates the open question of cross-category few-shot learning, in which the model is required to understand how to manipulate a novel category via interactions with a limited number of objects. After the few-shot exploration, the model should be able to manipulate other unseen objects within the same category without further test-time interactions.

Similarity-guided Exploration

Similarity

Pull the lid

Zero-shot to Unseen Object

1

2

3

Part motion

No part motion

No part motion

Training categories

Other categories

0            1

Similar Local Geometries Across Categories

Supporting Set

Geometries similar to supporting set

Geometries dissimilar to supporting set

Known

geometry

Part motion

Pulling

Pulling

Pulling

Pulling

Different from instance-level few-shot learning that focuses on discovering kinematic and dynamic information of a specific object, cross-category few-shot learning proposes a more demanding requirement for the exploration strategy to select informative interactions on different object categories. Considering the substantial semantic and geometric gap between known shapes and novel categories, forming an efficient exploration strategy for out-of-distribution objects is challenging. However, we point out that despite distinct overall shapes, different categories often share similar local geometries crucial for manipulation ( e.g. , pullable handles, pushable boards, and graspable edges, as shown in Figure 1). This property of possessing similar significant geometries across different categories is typically ignored or underutilized by previous instance-level few-shot learning studies.

To effectively discover these critical local geometries for cross-category few-shot learning, we propose a 'Where2Explore' framework by explicitly requesting the system to estimate the semantic similarity of geometries on novel objects with geometries already known by the model (As shown in the top part of Figure 1). When faced with objects from a novel object category ( e.g. , mugs), our framework identifies the uncertain yet important areas on the novel objects to interact with (Bottom Left of the Figure). Via fine-tuning our network with the interactions on novel objects, the model could generalize to unseen objects within this novel category (Bottom Right).

Our system is built in several steps. Firstly, we require a representation that encapsulates diverse semantic and geometric information from known categories to form a supporting set from which we could expand the learned knowledge to novel categories. The desired properties in our system are fulfilled by point-level affordance, which provides per-point manipulation priors with detailed semantic and geometric information on various objects [25, 38, 35, 46]. Next, we implement a 'similarity network' to measure the geometric similarity between shapes in different categories and those in the supporting set. This is achieved by partitioning the training set, exposing our similarity module to a wider range of categories, and supervising the learning of similarity in a cross-category manner. Finally, we perform few-shot learning on novel categories by proposing interactions on the low-similarity areas, indicating unseen yet significant geometries.

We evaluate our framework by training our model on constrained object categories and applying few-shot learning to novel categories with limited shapes. After the exploration, we evaluate our model on unseen objects in the novel categories. The results demonstrate our framework's capability to efficiently explore novel categories by exploiting geometric similarity. Additionally, we examine our framework's robustness across diverse combinations of training and testing categories, yielding consistent results.

In summary, the contributions of this paper include:

Exploring the challenging task of cross-category few-shot learning for articulated object manipulation, requiring the model to capture fine-grained geometric information from an entirely new category using a few interactions with limited instances.

Introducing the 'Where2Explore' affordance learning framework that explicitly measures the semantic similarity of local geometries across different categories, which successfully guides the exploration on novel categories with only a few interactions.

Our experiments, both in a simulator and the real world, show that our proposed framework can efficiently explore novel categories and generalize to unseen instances.

2 Related Work

Perceiving Articulated Objects for Manipulation. Future home-assistant robots need to possess the ability to perceive and manipulate a wide variety of articulated objects within human environments. Previous studies have made significant advancements in estimating and tracking the segmentation of the articulated parts [18, 43, 8, 15, 33], the 6-DoF part poses [20, 21, 36, 22, 9], the joint parameters [32, 34, 45, 13], and digital twins [26, 16, 12, 14]. Based on these perceptual signals, subsequent motion planners and controllers [31, 4, 2, 24, 1] can manipulate the articulated objects. Our work focuses on learning dense manipulation affordance heatmaps over the input 3D object, augmenting the basic part and joint parameters with fine-grained geometry-to-action mappings.

Affordance Learning on Articulated Objects. The concept of affordance [11] plays a pivotal role in facilitating the manipulation of articulated objects. In the literature, researchers have been exploring learning manipulation affordance on articulated objects [19, 17, 23, 6, 25, 38, 41, 30, 10, 42, 40, 5]. Although these works can accurately predict manipulation affordances for articulated objects, they frequently encounter difficulties handling novel unseen objects significantly different from the training data, which is the core problem we study in this paper. A noteworthy piece of research closely aligned with ours is the AdaAfford system [35] that learns to fine-tune the affordance prediction of novel objects using a limited number of interactions. While AdaAfford focuses on rapidly adapting to a single novel instance within known training object categories, our work addresses the more arduous challenge of generalizing to an entirely new category of articulated objects.

3 Problem Formulation

In line with prior affordance learning studies [25, 38, 46, 37], given an N-point 3D partial point cloud observation of an articulated object O ∈ R N × 3 and a set of action directions and gripper orientations { R p 1 , R p 2 , R p 3 , · · · | R p i ∈ SO (3) } on each point, the visual manipulation affordance is defined as a dense prediction Aff ∈ R N × 3 , where a ∈ [0 , 1] N on each point indicates whether the action on that point would result in a part motion. This definition forms a fine-grained geometry-to-action mapping.

During the few-shot exploration, the model needs to propose a few interactions I = { I 1 , I 2 , · · ·} sequentially. Each interaction I i = ( O i , p i , R i ) represents a task-specific hard-coded trajectory defined in Where2Act [25], parametrized by the interaction point p i ∈ O i and the action direction and gripper orientation R i ∈ SO (3) . The interaction will result in a part motion m i . The model will be fine-tuned by the proposed interactions I and corresponding outcomes { m 1 , m 2 , · · ·} .

4 Method

As shown in Figure 2, we propose the 'Where2Explore' framework to explicitly leverage the similar semantics on local geometries shared across different categories for cross-category fewshot exploration. To achieve this, we divide the training categories into two parts - the affordance category C aff and the similarity categories C sim . We begin by learning point-level manipulation affordance on the affordance category to create a supporting set containing semantic and geometric information on known shapes (Figure 2, Left) 4.1. Next, we introduce the 'similarity module' to form a representation that connects the geometries in the supporting set with geometries across category boundaries. This module is trained by comparisons between the geometries in C aff with those in C sim . (Middle) 4.2. Then, to expand the supporting set along the similarity representation we built, we perform few-shot learning on novel categories with the guidance of the similarity module, which transfers its prediction to novel categories through shared local geometries (Right) 4.3. Finally, we describe the network and training strategy 4.4.

4.1 Affordance Learning for Building Supporting Set

To conduct the cross-category few-shot exploration task, firstly, we need to build a supporting set from which we can expand our knowledge to a broader range of categories. This requires a representation that encapsulates the learned semantic and geometric information in known categories.

Similarity Learning

1

0

Supporting set

Affordance category

Few-shot Learning Loop

Similarity Categories

Novel category

Dissimilar

Similar

Affordance (pulling)

Similarity

Similarity

Transferred

Similarity

Interaction

Adapted affordance

Action

Proposal

Adapt

Break

Shared local

geometries

1

3

2

4

Visual manipulation affordance has been proven by previous works to have fine-grained manipulation information and is able to generalize to unseen objects within the same category [25, 38, 35]. We exploit these valuable properties to build up our supporting set. Following Where2Act [25], we build a module to predict the per-point affordance. Given a specific action R i on a point p i of a partial point cloud O i , the affordance module is required to predict whether the given action will result in a part motion. We train the affordance module on training categories to build up its semantic understanding of geometries in the training categories.

However, previous affordance learning works suffer a dramatic performance drop when tested on novel categories. To enable our framework to smoothly expand the supporting set to include wider object categories, we need a cross-category representation that links similar geometries across categories.

4.2 Cross-category Similarity Learning

We propose to explicitly estimate the similarity between geometries from different categories with the learned geometries in the supporting set. The similarity should act as a set of 'bridges' connecting shapes in the supporting set and the geometries from different categories. To be specific, the proposed similarity should be equipped with the following properties. Firstly, the similarity should be conditioned on specific actions (the action type, action direction, and gripper orientation) because even geometrically similar areas can have distinct semantic meanings when the action is different. For example, handles are significant in pulling whereas less important in pushing, and a horizontal handle could not be grasped by a gripper whose pose is also horizontal. Besides, similarity should be based on the current knowledge of affordance, showing high similarity when the learned affordance could directly generalize to a given geometry, which indicates this geometry is semantically similar to the supporting set and verse versa.

To achieve the first property, as shown in the middle of Figure 3, we propose a 'similarity module' to predict the semantic similarity. The similarity module is designed to take a partial point cloud of an object O i ∈ R 3 × N , a set of action directions and gripper orientations { R i } on each point, and is required to predict per-point similarity Sim ∈ R N based on these inputs. This design enables the similarity to be aware of specific actions and thus has the potential to contain manipulation semantics instead of only relying on the shape geometry.

In order to acquire the second property, the key is to expose our similarity module to categories that are broader than what our affordance module is trained on. As shown in the 3 (Left), we divide the training categories into two parts - the affordance category C aff that only contains one category and the similarity categories C sim that contains three categories (which also contains the affordance category). We train our affordance on C aff and supervise the similarity module using C sim .

During training, the interactions on the affordance category ( e.g. , cabinets) are used to supervise the affordance module, as shown in the previous section. However, interactions on similarity categories

"##$%&'()*

+$&,-*

./0/-'%/12

+$&,-*

Similarity

Affordance

Gripper

Pose

Training categories

Similarity

Categories

GT Interaction

Action

Direction

0            1

Affordance

Category

Split

Affordance learning

Similarity learning

GT Interaction

Affordance

Comparison

Inference-only

( e.g. , faucets, and windows) are kept from updating the affordance module. Instead, we compare the affordance prediction on objects from similarity categories with the ground-truth interaction results and use the accuracy of affordance prediction during training to supervise the learning of the similarity module, which is:

where Sim and Aff stand for the similarity and affordance prediction given a specific action. The Accu is computed using the accuracy in predicting the affordance score of an action during training.

As straightforward as this approach may appear, the learned similarity holds valuable properties for cross-category exploration. Since the similarity is trained in a cross-category manner, it forces the network to focus on local geometries shared across different objects, which is beneficial for generalizing to novel categories. Moreover, the learned similarity is based on the current knowledge of affordance, highlighting the areas where the current model is uncertain while saving the unnecessary exploration on geometries that the affordance could directly generalize to. Finally, similarity defined through this approach could reveal the semantic meaning of geometries with regards to manipulating them instead of only relying on the similarity of shapes.

4.3 Few-shot Learning Loop

After the similarity linking the supporting set and different geometries are learned, we perform a few-shot exploration of novel categories (unseen by both modules) with the learned similarity module.

As shown in the right part of figure 2, when faced with a novel category, our framework will first predict the similarity of the objects. Thanks to the property that similarity is conditioned on action directions and gripper orientations, we could sample interactions in diverse directions and poses R i to indicate which action on what geometry p i ∈ O i contains the most informative semantics. Then, by choosing the action with the lowest similarity prediction, the model performs a short-term manipulation trajectory and observes the result of the interaction as a part motion m i (the right part of the figure). Finally, both the affordance module and the similarity module will be updated by this interaction ( O i , p i , R i , m i ) and be ready for the next prediction on the object to indicate where to explore. This loop will break if the similarity on the instance reaches a bar or the interaction budget is reached.

Through just a few interactions on the instances from an unseen category, our model explores the semantic and geometric significant areas of a novel category and captures the common features shared with already learned categories. Via these few-shot explorations, the knowledge of manipulation affordance on training categories is transferred to novel categories. Thanks to the generalization ability of affordance within one category, our adapted affordance could manipulate unseen objects from this category without additional interactions.

4.4 Network Architecture and Training Strategy

Our network consists of two modules - the affordance module and the similarity module. We use a PointNet++ segmentation network [29] encoder for extracting features from 3D partial point clouds. The encoder will output a per-point feature of 128 dimensions. We let the two modules share the same encoder since we want the similarity module to be based on affordance prediction. We employ Multilayer Perceptrons (MLP) with one hidden layer of size 128 to implement both decoders.

Affordance Loss. To supervise the learning of the affordance network, We deploy a binary crossentropy loss, which measures the error between the affordance prediction of a given interaction and a binary label indicating whether the action results in a part motion ( i.e. , m i reaches the threshold of 0.01).

Similarity Loss. To train the similarity module, we use an L 1 loss to measure the distance between Similarity prediction and the ground truth accuracy. This accuracy is calculated as the ratio of correct predictions to total prediction attempts for each interaction during training.

We balance the positive and negative interactions during training and sampling the same amount of instances from different object categories equally. We train both modules simultaneously to learn the proposed affordance and similarity until they converge. Please see supplementary for more details.

5 Experiments

To demonstrate the ability of our framework to propose informative interactions for cross-category exploration efficiently. We intentionally train our model with constrained training categories. Then, we perform few-shot learning on a wide variety of categories using only a few instances. Finally, we test our fine-tuned model on unseen instances in novel categories to demonstrate that our model learns the general semantic and geometric information. We set up three baselines for comparisons. We also conduct ablation studies to prove the efficiency of our exploration strategy.

5.1 Data and Settings

Data. Following [25, 35], we use SAPIEN [39] with NVIDIA PhysX [28] as our simulator. We use 942 articulated 3D objects covering 14 categories to show our cross-category exploration ability.

To simulate the challenging cross-category few-shot task in the real world. We divide the categories into only three training categories and 11 novel categories . The shapes in all the categories are further divided into two disjoint sets of training and testing shapes. See supplementary for more details.

Experiment Settings. We conduct experiments under two different manipulation action types (pushing and pulling). We first train the networks on three training categories and then perform few-shot learning on 11 novel categories. Finally, we evaluate the model by testing objects from the novel categories.

For the training stage, to filter out randomness and prove the universal effectiveness of our framework, we conduct experiments using 4 different training category combinations , which are {cabinet, faucet, window}, {cabinet, switch, refrigerator}, {table, Faucet, refrigerator} and {table, switch, window}. These category combinations are chosen because they could cover representative articulations (revolute and prismatic joints). For baselines, we train the models using all training objects in training categories, whereas we divide the training categories into two parts to train our framework, as mentioned in the method. The first category in each combination is the training category.

For few-shot learning, we perform exploration on objects from the novel categories ( i.e. , the rest 11 categories). We conduct two experiments. The first experiment is few-shot learning on all novel categories simultaneously, which requires the model to choose the objects and interactions that contain uncertain yet significant semantic properties. We also perform few-shot learning on each novel category separately to match the real-world scenario.

It is worth noticing that we only choose 10 instances from each novel category for few-shot learning and evaluate the model on unseen objects in the novel categories, which challenges the model's ability to explore wisely in order to learn the semantic and geometric information that could generalize to unseen objects.

Environment Settings. Following Where2Act and AdaAfford [25, 35], we abstract away the robot arm and only use a Franka Panda flying gripper as the robot actuator. The input partial point cloud is assumed to be cleanly segmented out. To generate the point clouds, we mount an RGB-D camera with known intrinsic parameters 5-unit-length away pointing to the center of the target object.

5.2 Baselines, Ablations, and Metrics

Baselines and Ablations. We compare our framework with several baselines:

Where2Act [25]: an affordance learning framework predicting the visual actionable affordance using a partial point cloud. During few-shot learning, Where2Act will sample interaction on the points with an affordance score closest to 0.5.

AdaAfford [35]: an affordance learning method that explores test-time few-shot adaptation. During few-shot exploration, the interactions are proposed by a curiosity module that is optimized for discovering the dynamic information of a specific object.

PointEncoder [44]: a pre-trained transformer framework that takes point cloud as input and could perform few-shot learning on classification and segmentation tasks. This baseline uses the pre-trained transformer encoder to extract features for few-shot affordance learning.

We select Where2Act as a baseline to compare the exploration ability of certainty represented by our similarity with the certainty defined in classification tasks. We use AdaAfford to evaluate the ability of instance-level exploration strategy on cross-category few-shot learning. We select PointEncoder to compare our framework with a network pre-trained on large-scale datasets.

Besides, we compare to ablated versions of our method to verify our exploration strategy:

No-explore (lower bound) : our affordance model directly evaluated on novel categories without few-shot exploration, which represents the lower bound of few-shot learning.

Explore-random : a variant of our proposed framework that explores novel objects through random interactions.

Explore-noSim : a variant of our proposed framework that uses the same exploration strategy as Where2act instead of using the similarity module.

Full-data (upper bound) : our affordance model trained on all categories with abundant data. We choose this ablation to represent the upper bound of affordance learning.

Evaluation Metrics. Following Where2Act [25] and AdaAfford [35], we use the F-score, balancing the precision and recall, to evaluate the predictions of the visual affordance and use the sample successful rate to evaluate the ability of the learned affordance to propose successful actions. We calculate the sample success rate by randomly selecting one action predicted as successful by the affordance module, performing the interaction, and observing the result. The final rate is reported as the percentage of successful interactions in the simulation. For both the F-score and sample success rate, we use the average score of the four different training category combinations.

5.3 Quantitative Results and Analysis

Table 1: Few-shot learning on novel categories using different interaction budget (1, 2, 5).

Table 1 shows the results of few-shot learning on novel categories using different interaction budgets. Our method outperforms others in all metrics, particularly in 'pulling' actions. The pulling action is more demanding in selecting informative geometries to interact with ( e.g. , drawer handles, kettle lids). Specifically, compared to Where2Act , our framework could more efficiently explore novel categories, proving that the cross-category similarity is a better representation of certainty on novel geometries than certainty defined in classification tasks. Compared to the AdaAfford , our results suggest that instance-level exploration strategies which focus on dynamic information for a single object fail to generalize well across categories. Notably , AdaAfford requires test-time interaction for affordance prediction, making the comparison unfair since our model predicts affordance from only visual observation. Compared with PointEncoder , we show that our framework better understands the semantic information for manipulation than a pre-trained encoder, even if it is trained on a large-scale dataset and achieves generalization ability on several tasks.

Sample successful rate (%)

We also conduct few-shot affordance learning on representative categories separately to match the real-world scenario. Table 2 presents the quantitative comparisons against the baselines showing that our method achieves the best performance in most entries (especially in pulling tasks).

In Table 3, we compare our full method against several variants of our framework. Compared with No-explore , we observe that the proposed similarity could smoothly guide the cross-category generalization of affordance. Compared with other exploration strategies Explore-random and Explore-noSim that fail to discover important local areas, our strategy is dramatically more effective and efficient. Our framework also achieves comparable performance compared with Full-data , which is trained on all categories with abundant data. Considering that our framework only uses 0.3% of the original data, it further proves the efficiency of our exploration strategy.

5.4 Qualitative Results and Analysis

Affordance

Similarity

Where2Act strategy

Affordance

Similarity

AdaAfford strategy

Affordance

Similarity

AdaAfford strategy

Affordance

Similarity

Pushing

Pulling

Where2Act strategy

一

一

1

Figure 4 compares the visualization of our proposed similarity with exploration strategies of Where2Act [25] and AdaAfford [35] on novel categories. Compared with Where2Act , our proposed similarity is more geometric-aware, successfully discovering uncertain geometries ( e.g. , whether the handle joint is at its limit) to interact with and familiar shapes ( e.g. , windows) to save exploration budget. Compared with AdaAfford , which fails to generalize to novel categories, our framework could still propose reasonable exploration strategies on novel categories leveraging local similarity.

Affordance on novel object

Similarity on novel object

Exploration on novel object

Affordance after exploration

Affordance on novel object

Similarity on novel object

Exploration on novel object

Affordance after exploration

Affordance on novel object

Similarity on novel object

Fail

Fail

Success

Fail

Exploration on novel object

Affordance after exploration

Success

Similarity-guided Exploration

1

Part motion

Franka

Emika

Similarity

prediction

Azure Kinect DK

2

No part motion

3

Fail to grasp

Manipulation

After Exploration

Similarity

prediction

Adapted

affordance

Pulling

Part motion

Figure 5 shows the visualization of our proposed affordance and similarity on novel categories. While affordance fails to directly generalize to novel objects (Left), the similarity module can still discover areas that contain uncertain yet important semantic information to interact with (Middle). With a few similarity-guided interactions on the novel category, the affordance could capture the geometric information of novel objects. Note that although we choose the same novel object for visualization, the adapted manipulation affordance could directly generalize to other unseen objects in this category.

Figure 6 shows the qualitative results in the real world. We require our model, which is only trained on cabinets, windows, and faucets, to perform a few-shot exploration on four mugs and manipulate another mug after the exploration. We use pulling as our action primitive since it's more challenging. All objects are fixed to the tabletop to ensure the observed motion is a part motion instead of an entire movement. Please refer to the supplementary for more details.

6 Conclusion

We investigate the critical yet challenging task of cross-category few-shot affordance learning. The proposed 'Where2Explore' framework leverages the similarities in geometries across different categories to guide exploratory interactions on uncertain yet significant areas during few-shot learning. This study of cross-category few-shot exploration is beneficial to the real-world application of robots by empowering them to understand and manipulate novel categories through minimal interactions.

Limitations and Future Works. Our performance drops on categories with significant shape variance. Future works could use 'similarity' to divide local geometries into parts according to their manipulation properties, which might improve the generalization ability towards more objects.

Ethics Statement. Our work can empower future robots with the ability to explore and understand unseen objects. We do not see our work has any particular major harm or issue.

7 Acknowledgment

This work was supported by National Natural Science Foundation of China - General Program (62376006) and The National Youth Talent Support Program (8200800081).

References

Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipulation with articulated objects. arXiv preprint arXiv:2305.05706 , 2023.

Felix Burget, Armin Hornung, and Maren Bennewitz. Whole-body motion planning for manipulation of articulated objects. In 2013 IEEE International Conference on Robotics and Automation , pages 1656-1662. IEEE, 2013.

Kai Cheng, Ruihai Wu, Yan Shen, Chuanruo Ning, Guanqi Zhan, and Hao Dong. Learning environment-aware affordance for 3d articulated object manipulation under occlusions. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Sachin Chitta, Benjamin Cohen, and Maxim Likhachev. Planning for autonomous door opening with a mobile manipulator. In 2010 IEEE International Conference on Robotics and Automation , pages 1799-1806. IEEE, 2010.

Leiyao Cui, Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Yixin Zhu. Strap: Structured object affordance segmentation with point supervision. arXiv preprint arXiv:2304.08492 , 2023.

Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1778-1787, 2021.

Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.04382 , 2022.

Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15752-15761, 2021.

Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. arXiv preprint arXiv:2211.05272 , 2022.

Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to-end affordance learning for robotic manipulation. arXiv preprint arXiv:2209.12941 , 2022.

James J Gibson. The theory of affordances. Hilldale, USA , 1(2):67-82, 1977.

Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic reconstruction of articulated objects. pages xxxx-xxxx, 2023.

Nick Heppert, Toki Migimatsu, Brent Yi, Claire Chen, and Jeannette Bohg. Categoryindependent articulated object tracking with factor graphs. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022 , pages 3800-3807. IEEE, 2022.

Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. OPD: Single-view 3D openable part detection. In ECCV , 2022.

Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542 , 2021.

Dov Katz, Moslem Kazemi, J Andrew Bagnell, and Anthony Stentz. Interactive segmentation, tracking, and kinematic modeling of unknown 3d articulated objects. In 2013 IEEE International Conference on Robotics and Automation , pages 5003-5010. IEEE, 2013.

Hedvig Kjellström, Javier Romero, and Danica Kragi´ c. Visual object-action recognition: Inferring object affordances from human demonstration. Computer Vision and Image Understanding , 115(1):81-90, 2011.

Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song. Categorylevel articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3706-3715, 2020.

Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but geometric constraints: A model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088 , 2020.

Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised categorylevel articulated object pose estimation with part-level se (3) equivariance. arXiv preprint arXiv:2302.14268 , 2023.

Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and automation (ICRA) , pages 6169-6176. IEEE, 2021.

Mayank Mittal, David Hoeller, Farbod Farshidian, Marco Hutter, and Animesh Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1647-1654. IEEE, 2022.

Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6813-6823, 2021.

Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13001-13011, 2021.

Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learning interactions for articulated object 3d structure discovery. arxiv , 2022.

NVIDIA. Nvidia.physx.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Giulio Schiavi, Paula Wulkop, Giuseppe Rizzi, Lionel Ott, Roland Siegwart, and Jen Jen Chung. Learning agent-aware affordances for closed-loop interaction with articulated objects. arXiv preprint arXiv:2209.05802 , 2022.

Andreas J Schmid, Nicolas Gorges, Dirk Goger, and Heinz Worn. Opening a door with a humanoid robot using multi-sensory tactile feedback. In 2008 IEEE International Conference on Robotics and Automation , pages 285-291. IEEE, 2008.

Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research , 41:477-526, 2011.

Ruiqi Wang, Akshay Gadi Patil, Fenggen Yu, and Hao Zhang. Coarse-to-fine active segmentation of interactable parts in real scene images. arXiv preprint arXiv:2303.11530 , 2023.

Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8876-8884, 2019.

Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas J Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 90-107. Springer, 2022.

Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13209-13218, 2021.

Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV) , 2023.

Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106.14440 , 2021.

Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11097-11107, 2020.

Chao Xu, Yixin Chen, He Wang, Song-Chun Zhu, Yixin Zhu, and Siyuan Huang. Partafford: Part-level affordance discovery.

Zhenjia Xu, Zhanpeng He, and Shuran Song. Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters , 7(2):2447-2454, 2022.

Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d object affordance from 2d interactions in images. arXiv preprint arXiv:2303.10437 , 2023.

Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. arXiv preprint arXiv:1809.07417 , 2018.

Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19313-19322, 2022.

Vicky Zeng, Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2443-2450. IEEE, 2021.

Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper object manipulation. arXiv preprint arXiv:2207.01971 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first proposes a new task, cross-category few-shot learning for articulated object manipulation. And it further proposes a framework that first explictly measures the semantic similarity of local geometries,  interacts with the objects of novel categories according to the similarity, uses the interaction results to update the affordance estimation network, and finally manipulate the target object. Authors perform experiments in simulation and the proposed method outperforms existing methods. They show a successful demo on a real robot.\n\nSTRENGTHS:\nThe strengths of the paper are:\n1. The paper formulates a new problem, cross-category few-shot affornace learning, which may be useful for robot applications;\n2. The paper proposes a framework to address the proposed problem, which leverages the geometric similarity to guide the exploration on objects of novel categories. The paper designs a cross-category similarity learning pipeline to learn the similarity metric.\n\nWEAKNESSES:\n1. Statistics, eg. std, are not provided in the experimental results, especially the variance casued by the choose of few-shot learning instances.\n2. While the results of the proposed method are highed than baselines, the successful rate is still not statisfactory, especially the result of `full-data' is still low. More discussion on the results and illustration of failure cases can be added.\n3. The assumption of the paper that similar geometry indicate similar affordance for novel categories may be not solid.\n\nQUESTIONS:\n1. The `semantic' used in this paper is ambigious. You may add more explaination or use another term.\n2. The similarity learning part needs more details. How do you use Eq(1) to learn the similairty?\n3. How do you decide to stop the few-shot training?\n4. What is sliding joint in L235?\n\nLIMITATIONS:\nThe limiation of low success rate is not discurssed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the task of predicting affordances given a single-frame observation of an articulated object. The authors utilize few-shot learning to address the novel object adaption problem. However, previous approaches to few-shot learning require per-instance interaction during test time, which can be both costly and inefficient. To address this issue, this paper introduces the Where2Explore framework, which estimates geometric similarity across categories to offer more efficient interaction proposals. Additionally, it enables the transfer of learned affordance to novel cases. Experimental results demonstrate that this method surpasses previous approaches in various aspects.\n\nSTRENGTHS:\n* The motivation to explore more efficient few-shot learning is intriguing. The concept of leveraging affordance similarities across categories is intuitive and seems to be effective based on the experimental results.\n* The training strategy to learn the cross-category geometry-aware affordance similarity is straightforward and effective based on the quantitative visualization.\n* This paper conducts extensive experiments to compare with prior work and includes sensible ablation studies to demonstrate the effectiveness of the Where2Explore framework.\n\nWEAKNESSES:\n* There are not enough descriptions of the hard-coded intersection sets. Even if it’s proposed in the prior work, it’s better to include more details to be self-contained in this paper. This paper only chooses two actions, “push” and “pull” from the original 6 actions defined in the prior work. For the interaction definition, the action direction should be in the positive hemisphere of the surface normal. It’s not clear if it holds true for the real data. (Because the real data similarity visualization is not based on the surface normal) \n* In the few-shot adaption stage, it’s unclear how many interactions for each point are sampled for the similarity. In addition, the authors mention the similarity is conditioned on the action. It’s unclear if the action is still conditioned in the object/camera/world coordinate or some local coordinate to the interaction points. If the action is still parameterized in the global coordinate, it’s hard to see if the model can really learn the similarity.\n* For the few-shot learning, it’s hard to see the usefulness of the models in the supporting set. The current set of experiments cannot show the usefulness of the supporting set. Is this possible to learn from scratch with the few-shot learning strategy?\n* For the affordance and similarity training, there seems no reason to train similarity at the start, because the affordance predictions are still quite bad. It’s also not clear if training affordance and similarity in two-stage can achieve better performance.\n\nQUESTIONS:\n* Why for the visualizations of the similarity prediction in figure 5, the action directions are different for synthetic data and the real data. Does the actual inference process have any difference for synthetic data and real data? For example, the sampled actions for few-shot learning are defined differently for the synthetic and the real data?\n* For the interaction budget, it seems that 1, 2, 5 get better and better performance, is there some performance top bar, and what’s the number of the budgets?\n* In the similarity visualization, why the dark blue is not considered as potential interactions (they should have the lowest similarity score?) Like the pot in figure 5 (the second row). It seems that the visualization mix the affordance and the similarity, which is a bit confusing.\n\nLIMITATIONS:\nThe authors mention the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of interacting with novel articulated objects. More specifically, a few-shot learning based approach is proposed to adapt to unseen instances at test time. The methodology explicitly leverages similar local geometries, which may be shared across objects from distinct categories. Experimentation reveals superior performance relative to prior work.\n\nSTRENGTHS:\n- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Ablations are insightful. The ablations adequately test the utility of the design choices of the proposed methodology.\n- Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The fact that the proposed methodology achieves comparable performance to full-data while using only 0.3% of the data is impressive.\n\nWEAKNESSES:\n- Simplistic problem setting. The problem setting seems to be simplistic. For instance, the robot arm is abstracted away. This completely side-steps the problem of finding a motion plan which actually solves the task in a collision-free manner, limiting real world deployment. Furthermore, the input point cloud is assumed to be cleanly segmented out. It is unclear whether this is a realistic assumption, especially for when the models are deployed to the real world.\n- Limited real world evaluation. It is not clear whether the model truly generalizes to the real world. Furthermore, the section on real world experimentation (both in the main paper and the supplementary) seems to be quite limited. For instance, how is the grasping (gripper open / close) done, as this doesn't seem to be an output of the system? Or, how is the motion plan obtained from the end-effector poses? More extensive (quantitative and qualitative) experimentation on real world data would be preferred to be convincing that the sim2real generalizes.\n\nQUESTIONS:\n- Are \"action directions / gripper orientations\" often the normals? Is it possible to constrain the action space to just be normals?\n- Is a separate affordance model trained for each category, such that the similarity model can be trained on \"held-out\" categories?\n- Other alternatives / potential ablations:\n  - As an alternative to the L1 loss for training the similarity module, could a cross-entropy loss have been used for whether the current affordance model was correct on the unseen object? This would also provide a notion of uncertainty of the model (via confidence), which could be used to guide exploration.\n  - What if you returned the output of the affordance model for points with the highest similarity, without any further exploration? Wouldn't this correspond to points for which the model is the most confident in being correct?\n  - Could you train an affordance model which only operates on smaller point clouds (ie smaller field of view, e.g. a patch)? This may force the model to only look at the local geometries which should be shared across categories?\n- \"During few-shot learning, Where2Act will sample interaction on the points with an affordance score closest to 0.5.\" --> Why is this the case? Isn't 0 push and 1 pull? So shouldn't the highest confident action be sampled (ie farthest away from 0.5)?\n\nLIMITATIONS:\nThere is a small section on limitations of the work, which could be expanded.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on few-shot affordance learning and articulated object manipulation. It introduces a framework named Where2Explore, which consists of the affordance module and the similarity module. Both modules share the same architecture, a Where2Act [24]-like network based on PointNet++ [28]. While the affordance module is trained on a single category, the similarity model is trained across categories to understand similarity in local geometry and manipulation semantics. The model is benchmarked on 942 instances from 14 (3 for train, 11 for test) object categories in PartNet-Mobility [37]. It outperforms previous works [24, 34] in simulation.\n\nSTRENGTHS:\n1. The methodology is well motivated: Learning similar local geometry across object categories could benefit few-shot generalization in manipulation.\n2. It is impressive to observe the stark differences between the seen categories and the unseen categories. \n3. Under the same settings, Where2Explore exhibits a clear edge over previous works [24, 34] quantitatively.\n4. The demo video is well-prepared.\n\nWEAKNESSES:\n1. The method is limited to pulling and pushing actions, and the action trajectories are short-term and hard-coded. These limitations inherited from Where2Act could result in a significant simulation-to-real gap.\n2. There is only a single example of real-world manipulation, which is insufficient to demonstrate the method’s capabilities. Besides, according to the video, it seems that the gripper may not always hold mugs firmly enough. More real-world experiments should be conducted, and their success rates should be reported.\n3. Few improvements on the architecture. The current module largely follows the design of [24, 34]. While it is commendable to demonstrate new insights using a minimal network design, the current task may benefit from more fine-grained cross-category transfer learning. There is room for improvement in the methods.\n4. Neural Descriptor Fields (Simeonov *et al*.) is also capable of capturing local geometry information using few shots. It would be valuable to discuss and even adapt them for comparison.\n\nQUESTIONS:\n1. [38, 8] share a similar spirit and motivation in learning local geometry across categories and generalizing to the novel, which should be discussed in the related works.\n2. Table 3: The successful rate of pulling drops for the full-data compared to ours. Any explanation for this observation?\n3. Is the interaction budget (1, 2, 5) per instance or per category (10 instances)? This term seems not well defined.\n4. L233: Which category is the affordance category ($C_{aff}$) in each combination?\n\nLIMITATIONS:\nThe paper includes a section dedicated to limitations; however, it provides minimal discussion on the specific limitations. Please incorporate a comprehensive discussion of failure cases and limitations, including those in the Weaknesses section. To enhance the credibility of this work, it would be beneficial to incorporate quantitative results from real-world experiments and quantitative comparison to other works that follow a different paradigm, such as Neural Descriptor Fields and GAPartNet [8]. This would provide a more convincing evaluation of the proposed methodology.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores a few-shot learning strategy to predict the affordance of objects in novel categories. The key idea of the paper is to use a similarity module to transfer the knowledge from known categories to novel categories. Then the affordance prediction on the novel object can be learned with just a few exploration actions. The paper is compared with several existing methods on standard benchmarks.\n\nSTRENGTHS:\nThe paper addresses an important and interesting problem of transferring affordance knowledge from known categories to novel categories.\n\nThe paper proposes a similarity module for effective affordance knowledge transfer and achieves quantitativel  better results than existing methods.\n\nThe paper is overall easy to follow.\n\nWEAKNESSES:\nMethod:\n\nI have a couple of questions regarding the learning procedure and intuition.\n\n1. In Figure 2, to train the similarity module, the affordance module predicts the affordance map on the novel object and the similarity module predicts the per-point similarity. In my understanding, is the similarity module exactly the same with the affordance module but just learned with different data and learning objectives? If so, in eq (1), is the Aff the affordance module or similarity module in Figure 2? I am not sure if eq (1) and Figure 2 are aligned, maybe explaining in more detail could be helpful.\n\n2. It is not clear to me how the affordance prediction is updated during exploration/inference. Is it more of a test-time-adaptation? What module parameters are updated?\n\n3. I am not sure if the intuition of generalization using local geometries is well supported by the design. Is the similarity module responsible for this? If so, how do local geometries play a part? Isn’t it still predicted for all N observed points with global information? Since this builds the key idea of cross-category generalization, I think further clarification is necessary.\n\n4. In the paper, 10 instances from each novel category are used for few-shot learning. I wonder, for example, for Where2Act baseline, is it also trained with only 10 instances? There is also no ablation on the number of instances required, could it be fewer?\n\n5. The paper only explores push and pull. Maybe more actions can be studied to validate the method.\n\nQUESTIONS:\nI think overall the paper addressed an interesting problem and proposed an effective solution, but I think further clarification on the method is necessary. Please see my comments above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}