Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects
Authors: Chuanruo Ning
Ruihai Wu
Haoran Lu
Kaichun Mo
Hao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: articulated object manipulation
few-shot learning
visual affordance for robotics
TL;DR: Leveraging similarity among local geometries across object categories to perform cross-category few-shot affordance learning for articulated object manipulation.

Abstract: Articulated object manipulation is a fundamental yet challenging task in robotics. Due to significant geometric and semantic variations across object categories, previous manipulation models struggle to generalize to novel categories. Few-shot learning is a promising solution for alleviating this issue by allowing robots to perform a few interactions with unseen objects. However, extant approaches often necessitate costly and inefficient test-time interactions with each unseen instance. Recognizing this limitation, we observe that despite their distinct shapes, different categories often share similar local geometries essential for manipulation, such as pullable handles and graspable edges - a factor typically underutilized in previous few-shot learning works. To harness this commonality, we introduce 'Where2Explore', an affordance learning framework that effectively explores novel categories with minimal interactions on a limited number of instances. Our framework explicitly estimates the geometric similarity across different categories, identifying local areas that differ from shapes in the training categories for efficient exploration while concurrently transferring affordance knowledge to similar parts of the objects. Extensive experiments in simulated and real-world environments demonstrate our framework's capacity for efficient few-shot exploration and generalization.

Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects
Chuanruo Ningl.3 School of EECS, PKU 2School of CS, PKU $CFCS, School of CS, PKU "NVIDIA National Key Laboratory for Multimedia Information Processing, School of CS, PKU
Ruihai 2.3.3,5 Haoran Lu Kaichun Mo
Hao Dong 3,5
Abstract
Articulated object manipulation is fundamental yet challenging task in robotics. Due to significant geometric and semantic variations across object categories, previ- ous manipulation models struggle generalize to novel categories. Few-shot learn- ing is promising solution for alleviating this issue by allowing robots perform a few interactions with unseen objects. However, extant approaches often necessitate costly and inefficient test-time interactions with each unseen instance. Recognizing this limitation, we observe that despite their distinct shapes, different categories often share similar local geometries essential for manipulation, such as pullable handles and graspable edges factor typically underutilized in previous fhot learning works. To harness this commonality, we introduce "Where2Explore" affordance learning framework that effectively explores novel categories with minimal interactions on a limited number of instances. Our framework explicitly estimates the geometric similarity across different categories, identifying local areas that differ from shapes in the training categories for efficient exploration while concurrentyy transferring affordance knowledge to similar parts of the objects. Extensive experiments in simulated and real-world environments demonstrate framework's 's capacity for efficient few-shot exploration and generalization.
Introduction
Articulated objects, such as doors, drawers, scissors, and faucets, ubiquitous in our daily lives. Therefore, the ability of robots to manipulate these objects is of critical importance. Many previous However, due the significant variance the objects' structure, 3D geometry, and articulation types across categories, developing efficient perception and manipulation systems that can generalize to those
works have been done perceiving and manipulating articulated objects [25] 35 38 3
variations remains challenging 25
An intuitive solution to equip models with generalized manipulation knowledge is training them on large-scale datasets. However, conducting real-world interactions with diverse objects or acquiring 3D) models encompassing potential categories this approach could still fail with the emergence of new object categories or designs (e.g., a cup with novel geometries resembling a gourd as shown in Figure
be prohibitively time-consuming and costly. Morcover,
Since encountering novel objects is inevitable in real-world applications, few-shot learning, which allows robots propose interactions with novel objects and adapt their understanding to them, has emerged as a promising solution. However, previous few-shot learning works for articulated object manipulation usually focus instance-level exploration and adaptation, requiring test-time interactions on each novel object [35] 27]. This limitation hinders the efficiency and safety of real-world applications of robots.
This paper investigates the open question of cross-category few-shot learning, in which the model is required to understand how to manipulate a novel category via interactions with a limited number of objects. After the few-shot exploration, the model should be within the same category without further test-time interactions.
able to manipulate other unseen objects
"Correponninng author.
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

Supporting Set
Similar Local Geometries Across Categories
Geometries similar supporting
Known geometry
Training categories
Geometries dissimilar supporting set
Other categories
1
Similarity-gunt Exploration
Zero-shot to Unseen Object
Pulling
Pulling
POtinE
Part motion
Similarity
Part mutton
motion
No part motien
Pull
Figure I: 2xere2Explore framework. Our model, solely trained on training categories (Top Left) and having never seen mugs, utilizes the underlying similarity in local geometries across different categories (Top Right), enabling it to identify uncertain yet important areas for interaction (Bottom Left). After minimal interactions, our model could manipulate unseen objects (Bottom Right) in this category.
Different from instance-level few-shot learning that focuses on discovering kinematic and dynamic information of specific object, cross-category few-shot Icarning proposes more demanding requirement for the exploration strategy to select informative interactions different object categories. Considering the substantial semantic and geometric gap between known shapes and novel categories, forming an efficient exploration strategy for out-of-distribution objects is challenging. However, point out that despite distinct overall shapes, different categories often share similar local geometries crucial for manipulation (e.g., pullable handles, pushable boards, and graspable edges, shown in Figure This property possessing similar significant geometries across different categories is typically ignored or underutilized by previous instance-level few-shot learning studies.
To effectively discover these critical local geometries for cross-category few-shot learning, we propose 'Where2Explore' framework by explicitly requesting the system estimate the semantic similarity of geometries on novel objects with geometries already known by the model (As shown in the top part of Figure [ When faced with objects from novel object category (e.g., mugs), our framework identifies the uncertain yet important areas on the novel objects interact with (Bottom Left of the Figure). Via fine-tuning our network with the interactions on novel objects, the model could generalize to unseen objects within this novel category (Bottom Right).
Our system built in several steps. Firstly, we require representation that encapsulates diverse semantic and geometric information from known categories to form a supporting set from which could expand the learned knowledge to novel categories. The desired properties in system are fulfilled by point-level affordance, which provides per-point manipulation priors with detailed semantic and geometric information "similarity network' to measure the geometric similarity between shapes in different categories and those in the supporting set. This is achieved by partitioning the training set, exposing our similarity module to wider range of categories, and supervising the learning of similarity in cross-category manner. Finally, we perform few-shot learning on novel categories by proposing interactions on the low-similarity areas, indicating unseen yet significant geometries.
various objects (25] 38 35 46). Next, we implement
We evaluate our framework by training our model on constrained object categories and applying few-shot learning to novel categories with limited shapes. After the exploration, we evaluate our model on unseen objects in the novel categories. The results demonstrate our framework's capability to efficiently explore novel categories by exploiting geometric similarity. Additionally, examine our framework's 's robustness across diverse combinations of training and testing categories, yielding consistent results.
In summary, the contributions of this paper include:

---

Exploring the challenging task of cross- category few-shot learning for articulated object manip- ulation, requiring the model to capture fine-grained geometric information from an entirely new category using a few interactions with limited instances.
Introducing the 'Where2Explore" affordance learning framework that explicitly measures the semantic similarity of local geometries across different categories, which successfully guides the exploration on novel categories with only a few interactions.
Our experiments, both in a simulator and the real world, show that our proposed framework can efficiently explore novel categories and generalize to unseen instances.
2 Related Work
Perceiving Articulated Objects for Manipulation. Future Nome-assistant robots need to possess the ability to perceive and manipulate wide variety of articulated objects within human environments. Previous studies have made significant advancements in estimating and tracking the segmentation
of the articulated parts [18
43
15
333, the 6-DoF part poses 20
21
36 22 9 the joint
parameters [32 45 13). and digital twins 126
16 12
14). Based on these perceptual signals, 24 can manipulate the articulated objects.
subsequent motion planners and controllers [31 4
Our work focuses learning dense manipulation affordance heatmaps over the input 3D object, augmenting the basic part and joint parameters with fine- ane-grained geometry-to-action mappings.
Affordance Learning on Articulated Objects. The concept of affordance [1] plays pivotal role
in facilitating the manipulation of articulated objects. In the literature, researchers have been exploring
S]. Although these works can accurately predict manipulation affordances for articulated objects, they frequently encounter difficulties handling novel unseen objects significantly different from the training data, which is the core problem we study in this paper. A noteworthy piece of research closely aligned with ours is the AdaAfford system [BS] that learns to fine-tune the affordance prediction of novel objects using a limited number of interactions. While AdaAfford focuses on rapidly adapting to a single novel instance within known training object categories, our work addresses the more arduous challenge of generalizing to an entirely new category of articulated objects.
learning manipulation affordance on articulated objects 19. 17 23 25
38
30 10
3 Problem Formulation
In line with prior affordance learning studies 25
given an N-point 3D partial point cloud observation of an articulated object O E Nx3 and a set of action directions and gripper orientations each point, the visual manipulation affordance is defined
38
46 37
{RT, Rg ... |R S3(3) dense prediction Afʃ E R N x where E [0, 1]" 1 on each point indicates whether the action on that mapping. point would result a part motion. This definition forms fine tne-ggrined geometry-to-actt
During the few-shot exploration, the model needs to propose a few interactions = {/1, I2, ...3 sequentially. Each interaction 4;- (O,, Pt, R₂) represents a task-specifie hard-coded trajectory defined in Where2Act (25), parametrized by the interaction point p; O, and the action direction and gripper orientation R₂ E SO(3). The interaction will result in a part motion m,, The model will be fine-tuned by the proposed interactions / and corresponding outcomes {mi, m2:-.!)
4 Method
As shown in Figure 2 we propose the 'Where2Explore' framework to explicitly leverage the similar semantics local geometries shared across different categories for cross-category few- shot exploration. To achieve this, we divide the training categories into two parts the affordance category Caff and the similarity categories Ceim- We begin by learning point-level manipulation affordance the affordance category create supporting set containing semantic and geometric information on known shapes (Figure 2 Left) 4.1 Next, we introduce the 'similarity module' to form representation that connects the geometries in the supporting set with geometries across category boundaries. This module is trained by comparisons between the geometries Caff with those in Csim- (Middle) 4.2 Then, to expand the supporting set along the similarity representation we built, we perform few-shot learning on novel categories with the guidance of the similarity module, which transfers its prediction to novel categories through shared local geometries (Right) 4.3 Finally, we describe the network and training strategy 4.4
4.1 Affordance Learning for Building Supporting Set
To conduct the cross-category few-shot exploration task, firstly, need to build supporting set from which we can expand our knowledge to broader range of categories. This requires a representation that encapsulates the learned semantic and geometric information in known categories.
3

---

Supporting set
Similarity Learning
Few-shot Learning Loop
Dissimilar
Similar
Shared local geometries
Affordance category
Similarity Categories
Novel category
Adapted affordance
Break
Action
Proposal
Adapt
Transferred Similarity
Affordance (pulling)
Similarity
Similarity
Interaction
Figure 2: Method overview: We first employ affordance learning on the affordance category to form our supporting set (Left), Then, we estimate the semantic similarity between learned geometries and geometries from the similarity categories (Middle). Finally, leveraging the shared local geometries in novel categories, we conduct few-shot learning with the transferred similarity prediction (Right).
Visual manipulation affordance has been proven by previous works to have fine-grained manipulation information and is able to generalize to unseen objects within the same category [25]
[8 35
We
exploit these valuable properties to build up our supporting set. Following Where2Act 255 build
module to predict the per-point affordance. Given specific action R, on point of partial point
cloud O,, the affordance module is required predict whether the given action will result in part motion. We train the affordance module on training categories to build up its semantic understanding of geometries in the training categories.
However, previous affordance learning works suffer dramatic performance drop when tested on novel categories. To enable our framework to smoothly expand the supporting set include wider object categories, we need cross-category representation that links similar geometries across categories.
4.2 Cross-category Similarity Learning
We propose to explicitly estimate the similarity between geometries from different categories with learned geometries in the supporting set. The similarity should act set 'bridges' connecting shapes in the supporting set and the geometries from different categories. To be specific, the proposed similarity should be equipped with the following properties. Firstly, the similarity should be conditioned on specific actions (the action type, action direction, and gripper orientation) because even geometrically similar areas can have distinct semantic meanings when the action is different. For example, handles are significant pulling whereas less important in pushing, and horizontal handle could not be grasped by gripper whose pose is also horizontal. Besides, similarity should be based the current knowledge of affordance, showing high similarity when the learned affordance could directly generalize to a given geometry, which indicates this geometry semantically similar to the supporting set and verse versa.
To achieve the first property, shown in the middle of Figure to predict the semantic similarity. The similarity module designed take partial point cloud of an object O R3xN set of action directions and gripper orientations { RR } each point, and is required predict per-point similarity Sim E N based these inputs. This design enables the similarity be aware of specific actions and thus has the potential contain manipulation semanties instead of only relying on the shape geometry.
propose 'similarity module'
In order to acquire the second property, the key is to expose our similarity module to categories that are broader than what our affordance module trained on. As shown the 3 (Left), divide the training categories into two parts the affordance category Caff that only contains one category and the similarity categories Csim that contains three categories (which also contains the affordance category). We train our affordance on Caff and supervise the similarity module using Csi--
During training, the interactions on the affordance category (e.g., cabinets) are used to supervise the affordance module, shown in the previous section. However, interactions similarity categories

---

Training categories
Affordance learning
Affordacce
Module
Affordance
GT Interaction
Affordance Category
Similarity learning
Comparison
Split
Gripper
Action
Pose
Direction
GT Interaction
Similarity
Similarity
Module
Inferenee-only
Categories
Similarity
Figure 3: Cross-category similarity learning. We use a similarity module to predict the similarity conditioned specific actions (Middle). While the affordance category used to train the affordance representation (Top Right), the split similarity categories are used to supervise the learning of the similarity module by comparing the GT interactions with the affordance prediction (Bottom Right),
(e.g., faucets, and windows) are kept from updating the affordance module. Instead, we compare the affordance prediction on objects from similarity categories with the ground-trutt interaction results and use the accuracy of affordance prediction during training to supervise the learning of the similarity module, which is:
Sim
Accu(Aff(O; R.), m.), O₂ E Csim
(I)
where Sim and Aff stand for the similarity and affordance prediction given specific action. The Accu is computed using the accuracy in predicting the affordance score of an action during training.
As straightforward this approach may appear, the learned similarity holds valuable properties for cross-category exploration. Since the similarity is trained in a cross-category manner, it forces the network to focus on local geometries shared across different objects, which is beneficial for generalizing to novel categories. Moreover, the learned similarity is based on the current knowledge of affordance, highlighting the areas where the current model is uncertain while saving the unnecessary exploration on geometries that the affordance could directly generalize to. Finally, similarity defined through this approach could reveal the semantic meaning of geometries with regards manipulating them instead of only relying on the similarity of shapes.
4.3 Few-shot Learning Loop
After the similarity linking the supporting set and different geometries are learned, we perform few-shot exploration of novel categories (unseen by both modules) with the learned similarity module.
As shown the right part of figure 2 when faced with a novel category, our framework will first predict the similarity of the objects. Thanks to the property that similarity is conditioned on action directions and gripper orientations, we could sample interactions in diverse directions and poses R; to indicate which action on what geometry p: E O, contains the most informative semantics. Then, by choosing the action with the lowest similarity prediction, the model performs short-term manipulation trajectory and observes the result of the interaction as part motion m; (the right part of the figure). Finally, both the affordance module and the similarity module will be updated by this interaction (O;, Pi, R2, m&) and be ready for the next prediction on the object indicate where explore. This loop will break the similarity on the instance reaches bar or the interaction budget reached.
Through just few interactions on the instances from unseen category, model explores the semantic and geometric significant areas a novel category and captures the common features shared with already learned categories. Via these few-shot explorations, the knowledge of manipulation affordance on training categories is transferred novel categories. Thanks the generalization ability of affordance within one category, our adapted affordance could manipulate unseen objects from this category without additional interactions.
5

---

4.4 Network Architecture and Training Strategy
Our network consists of two modules the affordance module and the similarity module. We use PointNet++ segmentation network [29] encoder for extracting features from 3D partial point clouds. The encoder will output per-point feature of 128 dimensions. We let the two modules share the same encoder since we want the similarity module to be based on affordance prediction. We employ Multilayer Perceptrons (MLP) with one hidden layer of size 128 to implement both decoders.
Affordance Loss. To supervise the learning of the affordance network, We deploy binary cross- entropy loss, which measures the error between the affordance prediction of a given interaction and binary label indicating whether the action results in a part motion (i.e., m, reaches the threshold of 0.01).
Similarity Loss. To train the similarity module, we use an Li loss to measure the distance between Similarity prediction and the ground truth accuracy. This accuracy is calculated as the ratio of correct predictions to total prediction attempts for each interaction during training.
We balance the positive and negative interactions during training and sampling the same amount of instances from different object categories equally. We train both modules simultaneously to learn the proposed affordance and similarity until they converge. Please see supplementary for more details.
5 Experiments
To demonstrate the ability of our framework to propose informative interactions for cross-category exploration efficiently. We intentionally train our model with constrained training categories. Then, we perform few-shot learning on wide variety of categories using only few instances. Finally, we test our fine-tuned model on unseen instances in novel categories to demonstrate that our model learns the general semantic and geometric information. We set up three baselines for comparisons. We also conduct ablation studies to prove the efficiency of our exploration strategy.
5.1 Data and Settings
Data. Following 25/:31 we use SAPIEN 39
with NVIDIA PhysX (28 as our simulator. We use 942 articulated 3D objects covering 14 categories to show our cross-category exploration ability.
To simulate the challenging cross-category few-shot task in the real world. We divide the categories into only three training categories and 11 novel categories. The shapes in all the categories further divided into two disjoint sets of training and testing shapes. See supplementary for more details.
Experiment Settings. We conduct experiments under two different manipulation action types (pushing and pulling). We first train the networks on three training categories and then perform few-shot learning on 11 novel categories. Finally, we evaluate the model by testing objects from the novel categories.
For the training stage, to filter out randomness and prove the universal effectiveness of our framework, we conduct experiments using 4 different training category combinations, which are {cabinet, faucet, window), (cabinet, switch, refrigerator), (table, Faucet, refrigerator) and (table, switch, window). These category combinations are chosen because they could cover representative articula- tions (revolute and prismatic joints). For baselines, we train the models using all training objects in training categories, whereas we divide the training categories into two parts to train our framework, as mentioned in the method. The first category in each combination is the training category.
For fho-shot learning, we perform exploration on objects from the novel categories (i.e., the rest 1l categories). We conduct two experiments. The first experiment is few-shot learning on all novel categories simultaneously, which requires the model to choose the objects and interactions that contain uncertain yet significant semantic properties. We also perform sew-shot learning on each novel category separately to match the real-world scenario.
is worth noticing that we only choose 10 instances from each novel category for few-shot learning and evaluate the model on unseen objects in the novel categories, which challenges the model's ' ability explore wisely in order to learn the semantic and geometric information that could generalize to unseen objects.
Settings. Following Where2Act and AdaAfford [25 35], we abstract away the robot arm and only use Franka Panda flying gripper as the robot actuator. The input partial point cloud is assumed to be cleanly segmented out. To generate the point clouds, we mount an RGB-D camera with known intrinsic parameters 5-unit-length away pointing the center of the target object.
Environment

---

5.2 Baselines, Ablations, and Metrics
Baselines and Ablations. We compare our framework with several baselines:
affordance learning framework predicting the visual actionable affordance using a partial point cloud. During few-shot learning, Where2Act will sample interaction on the points with an affordance score closest to 0.5.
Where2Act [5
AdaA fford 35 an affordance learning method that explores test-time few-shot adaptation. During few-shot exploration. the interactions are proposed by curiosity module that optimized for discovering the dynamic information of a specific object.
PointEncoder [44]:
pre-trained transformer framework that takes point cloud as input and could perform few-shot learning classification and segmentation tasks. This baseline uses the pre-trained transformer encoder to extract features for few-shot affordance learning.
We select Where2Act as baseline to compare the exploration ability of certainty represented by our similarity with the certainty defined in classification tasks. We use AdaAfford to evaluate the ability of instance-level exploration strategy on cross-category few-shot learning. We select PointEncoder to compare our framework with a network pre-trained on large-scale datasets.
Besides, we compare to ablated versions of our method to verify our exploration strategy:
No-explore (lower bound); our affordance model directly evaluated on novel categories without few-shot exploration, which represents the lower bound of few-shot learning.
Explore-random: variant of our proposed framework that explores novel objects through random interactions.
Explore-noSing a variant our proposed framework that uses the same exploration strategy as Where2act instead of using the similarity module.
Full-data (upper bound): our affordance model trained on all categories with abundant data. We choose this ablation to represent the upper bound of affordance learning.
Evaluation Metrics. Following Where2Act [25 and AdaAfford [35], we use the F-score, balancing the precision and recall, to evaluate the predictions of the visual affordance and use the sample successful rate to evaluate the ability of the learned affordance to propose successful actions. We calculate the sample success rate by randomly selecting one action predicted successful by the affordance module, performing the interaction, and observing the result. The final rate is reported as the percentage of successful interactions in the simulation. For both the F-score and sample success rate, we use the average score of the four different training category combinations.
5.3 Quantitative Results and Analysis
F-score
Sample successful rate Pushing
Method
Pushing
Pulling
Pulling
Where2Act
28.6/ 30.4
6.4/ 8.5
15.7 17.0/ 19.9
3.9/4.3 6.2
AdaAfford
27.5 32.0
3.7/4.0/4.4
27.2/ 37.1
/
PointEncoder
19.4 / 29.9
2.9/4.6/599
11.6 10.9 29.9
1.8 3.1 9.2
Ours
41.6
12.1 24.2
31.3 39.5
11.5 13.4 14.9
Table I: Few-shot learning on novel categories using different interaction budget (I, 5).
Table [ shows the results of few-shot learning on novel categories using different interaction budgets. Our method outperforms others in all metrics, particularly in 'pulling' actions. The pulling action is more demanding in selecting informative geometries to interact with (e.g., drawer handles, kettle lids). Specifically, compared to Where2Act, our framework could more efficiently explore novel categories, proving that the cross-category similarity a better representation of certainty on novel geometries than certainty defined in classification tasks. Compared to the AdaAfford, our results suggest that instance- level exploration strategies which focus on dynamic information for single object fail to generalize well across categories. Notably, AdaAfford requires test-time interaction for affordance prediction, making the comparison unfair since our model predicts affordance from only visual observation. Compared with PointEncoder, semantic information for manipulation than a pre- pre-trained encoder, even if it is trained on a large-scale dataset and achieves generalization ability on several tasks.
show that our framework better understands the
7

---

Methods
Pushing unseen instances in novel categories
Pulling unseen instances in novel categories
W
Where2Act
22.1
10.5
42.8
43.4
31.2
47.4
51.7
8.9
6.0
13.1
12.1
2.5
5.4
8.3
AdaAfford
24.4
7.5
50.1
48.8
25.5
44.3
52.2
9.2
4.3
14.0
11.3
2.7
7.8
9.2
PointEncoder
20.4
14.2
29.3
24.1
22.7
26.8
29.8
3.9
9.6
7.7
7.8
4.7
8.9
9.0
Ours
36.5
15.6
60.5
48.5
39.7
61.5
66.0
26.6
15.8
28.8
19.1
8.7
16.4
13.8
F-score (%)
Where2Act
14.1
5.9
42.4
35.7
22.2
34.8
39.4
7.4
5.3
18.2
18.2
1.5
3.0
4.5
AdaAfford
14.4
7.5
47.1
47.4
24.2
40.4
43.2
7.7
7.5
25.0
11.3
1.3
3.1
5.4
PointEncoder
13.1
3.4
18.7
17.3
12.4
17.5
21.0
3.0
3.9
4.3
7.8
0.6
4.3
3.6
Ours
29.5
9.6
54.5
41.9
32.8
49.2
54.7
17.1
16.0
35.5
21.4
15.1
11.3
15.4
Sample successful rate (%)
Table 2: Evaluation of few-shot learning on different categories separately interaction budget)
F-score
Sample successful rate Pushing
Method
Pushing
Pulling
Pulling
No-explore
20.6
3.9
12.1
3.2
Explore-random
28.9 29.0
4.0/6.1/9.2
15.0/ 18.4 21.6
3.4/ 5.1
Explore-noSim
24.8 24.8/ 30.2 30.2
6.8/6.71 8.4
15.0/ 19.2 25.5
5.4/ 6.2 5.3
Ours
38.5/ 41.6
12.1 / 12.5/ 24.2
39.5
11.5 13.4/ 14.9
Full-data
47.9
27.1
40.3
13.9
Table Ablations the exploration strategy using different interaction budget (1, 2, 5).
We also conduct few-shot affordance learning representative categories separately match the real-world scenario. Table presents the quantitative comparisons against the baselines showing that our method achieves the best performance in most entries (especially in pulling tasks).
In Table 3, we compare our full method against several variants of our framework. Compared with No-explore, we observe that the proposed similarity could smoothly guide the cross-category generalization affordance. Compared with other exploration strategies Explore-random and Explore-noSim that fail discover important local areas, our strategy is dramatically more effective and efficient. Our framework also achieves comparable performance compared with Full-data. which is trained on all categories with abundant data. Considering that our framework only uses 0.3% of the original data, further proves the efficiency of exploration strategy.
5.4 Qualitative Results and Analysis
Pushing
Pulling
Affordance
Similarity
WhereZAct strategy
Affordance
Similarity
strategy
Similarity
strategy
Affordance
Similarity
AdaAfford strategy
Figure 4: Visualization of different exploration strategies on novel objects. The action directions are set to the normal direction of each point in this visualization.
Figure 4 compares the visualization of proposed similarity with exploration strategies of Where2Act (25) and AdaAfford (35) on novel categories. Compared with Where2Act, our proposed similarity is more geometric-aware, successfully discovering uncertain geometries (e.g.. whether the handle joint is at its limit) interact with and familiar shapes (e.g., windows) to exploration budget. Compared with AdaAfford. which fails to generalize to novel categories, our framework could still propose reasonable exploration strategics novel categories leveraging local similarity.

---

Fail
Affordance novel object
Similarity novel object
Exploration novel object
Affordance after exploration
Affordance novel object Similarily novel object
Exploration novel
Affordance after exploration
Affordance on novel object Similarity on novel object
Exploration on novel object
Affordance after exploration
Figure 5: Pushing (top) and pulling (middle and bottom) affordance and similarity prediction on novel object categories. Although Affordance fails directly generalize to novel categories (Left) via interacting on low-similarity areas (Middle), our framework could learn the semantic information them (Right). In this visualization, for objects in the simulator, the action directions set to the normal direction of each point. For objects from the real world, the direction the vertically up direction.
Similarity-guided Exploration
Manipulation
Prenka Emilca
After Exploration
Fanda Boboot
Similarity
Similarity prediction
Adapted affordance
prediction
Pulling
Part soxine
Na pant medias
THI poup
Figure 6: Real-world experiment set up (Left), iimilarity-guider exploration (Middle), and manipula- tion after exploration (Right). The pulling direction is vertical in this visualization.
Figure shows the visualization of our proposed affordance and similarity novel categories. While affordance fails to directly generalize to novel objects (Left), the similarity module can still discover areas that contain uncertain yet important semantic information to interact with (Middle). With few similarity- information of novel objects. Note that although we choose the same novel object for visualization, the adapted manipulation affordance could directly generalize to other unseen objects in this category.
interactions the novel category, the affordance could capture the geometric
Figure 6 shows the qualitative results in the real world. We require our model, which is only trained on cabinets, windows, and faucets, to perform few-shot exploration on four mugs and manipulate another mug after the exploration. We use pulling as our action primitive since it's more challenging. All objects are fixed the tabletop ensure the observed motion is part motion instead an entire movement. Please refer to the supplementary for more details.
Conclusion
We investigate the critical yet challenging task of cross-category few-shot affordance learning. The proposed 'Where2Explore' framework leverages the similarities geometries across different categories to guide exploratory interactions on uncertain yet significant areas during few-shot learning. This study of cross-category few-shot exploration is beneficial the real-world application of robots by empowering them to understand and manipulate novel categories through minimal interactions.
Limitations and Future Works. Our performance drops categories with significant shape variance. Future works could use "similarity' to divide local geometries into parts according to their manipulation properties, which might improve the generalization ability towards more objects.

---

Ethics Statement. Our work can empower future robots with the ability to explore and understand unseen objects. We do not see our work has any partieular major harm issue.
7 Acknowledgment
This work was supported by National Natural Science Foundation of China General Program (62376006) and The National Youth Talent Support Program (8200800081).
References
[1] Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipulation with articulated objects. arXiv preprint arXivv2305.057006 2023.
[2] Felix Burget, Armin Hornung, and Maren Bennewitz. Whole-body motion planning for manipulation of articulated objects. In 2013 IEEE International Conference on Robotics and Automation, pages 1656-1662. IEEE, 2013.
[3] Kai Cheng, Ruihai Wu, Yan Shen, Chuanruo Ning, Guanqi Zhan, and Hao Dong. Learning environment-awwre affordance for 3d articulated object manipulation under occlusions. In Advances Neural Information Processing Systems (NeurIPS), 2023.
[4] Sachin Chitta, Benjamin Cohen, and Maxim Likhachev. Planning for autonomous door opening with mobile manipulator. In 2010 IEEE International Conference on Robotics and Automation, pages 1799-1806. IEEE, 2010.
[5] Leiyao Cui, Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Yixin Zhu. Strap: Structured object affordance segmentation with point supervision. arXiv preprint arXiv:2304.08449 2023.
[6] Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1778-1787, 2021.
[7] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.04382. 2022.
[8] Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15752-15761. 2021.
[9] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generrlizzalee object perception and manipulation via gener- alizable and actionable parts. arXiv preprint arXiv:2211.05277 2022.
[10] Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to-end affordance learning for robotic manipulation. arXiv preprint arXiv:2209. /2944 2022.
[11] James J Gibson. The theory of affordances. Hilldale, USA, 1(2):67-82, 1977.
[12] Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic reconstruction of articulated objects. pages XXXX-XXXX, 2023.
[13] Nick Heppert, Toki Migimatsu, Brent Yi, Claire Chen, and Jeannette Bohg. Category- independent articulated object tracking with factor graphs. In JEEE/RSJ International Confer- ence Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 3800-3807. IEEE, 2022.
[14] Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In IEEE International Conference Robotics and Automation (ICRA), 2023.
[15] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. OPD: Single-view 3D openable part detection. In ECCV, 2022.
10

---

[16] Zhenyu Jiang, Cheng- Chann-Chunnuu Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu, Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542. 2021.
[18) Dov Katz, Moslem Kazemi, J Andrew Bagnell, and Anthony Stentz. Interactive segmentation, tracking, and kinematic modeling of unknown 3d articulated objects. In 2013 IEEE International Conference Robofics and Automation, pages 5003-5010. IEFF, 2013.
[19] Hedvig Kjellström, Javier Romero, and Danica Kragié. Visual object-action recognition: Infer- ring object affordances from human demonstration. Computer Vision and Image Understanding, 115(1):81-99) 1-,,, 2011.
[20] Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A Lynn Abbott, and Shuran Song. Category- level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3706-3715, 2020.
[21] Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but geometric constraints: A model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088 2020.
[22] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised category- level articulated object pose estimation with part-level (3) equivariance. arXiv preprint arXiv:2302. 14268, 2023.
[23] Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference robotics and automation (ICRA), pages 6169-6176. IEEE, 2021.
[24] Mayank Mittal, David Hoeller, Farbod Farshidian, Marco Hutter, and Animesh Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation. In 2022 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1647-1654. IEEE, 2022.
[25] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6813-6823, 2021.
[26] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representa- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001-13011, 2021.
[27] Neil Nic, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learning interactions for articulated object 3d structure discovery. arxiv, 2022.
[28] NVIDIA. Nvidia.physx.
[29] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[30] Giulio Schiavi, Paula Wulkop, Giuseppe Rizzi, Lionel Ott, Roland Siegwart, and Jen Jen Chung. Learning agent-aware affordances for closed-loop interaction with articulated objects. arXiv preprint 22Xiv:2209.005
2022.
[31] Andreas J Schmid, Nicolas Gorges, Dirk Goger, and Heinz Worn Opening a door with a humanoid robot using multi-sensory tactile feedback. In 2008 IEEE International Conference on Robotics and Automation, pages 285-299 291. IEEE, 2008
[32] Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Inteligence Research, 41:477-526.
2011.
11

---

[33] Ruiqi Wang, Akshay Gadi Patil, Fenggen Yu, and Hao Zhang. Coarse-to fine active segmenta- tion of interactable parts in real scene images. arXiv preprint arXiv:2303. 1530, 2023.
[34] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8876-8884, 2019.
[35] Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas J Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 90-107. Springer, 2022.
[36] Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209-13218, 2021.
[37] Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[38] Ruihai Wu, Yan Zhao, Kaichun Mo. Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprini arXiv:2 14440, 2021.
[39] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive en- vironment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1009-11107, 2020.
[40] Chao Xu, Yixin Chen, He Wang, Song-Chun Zhu, Yixin Zhu, and Siyuan Huang. Partafford: Part-level affordance discovery.
[41] Zhenjia Xu, Zhanpeng He, and Shuran Song. Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters, 7(2):2447-2454, 2022.
[42] Yuhang Yang. Wei Zhai, Hongchen Luo, Yang Cao,, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d object affordance from 2d interactions in images. arXiv preprint arXiv:2303, 10437, 2023.
[43] Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep 2018.
part induction from articulated object pairs. arXiv preprint
[44] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19322,, 2022.
[45] Vicky Zeng, Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. In 202/ IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2443-2450. IEEE, 2021.
[46] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingman Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual- qual-gripper object manipulation. arXiv preprint arXiv:2207.01977, 2022.
12


Reviews:
[{"review_id": "hX4SihUqXG", "reviewer": "Reviewer_ztRo", "summary": "The paper explores a few-shot learning strategy to predict the affordance of objects in novel categories. The key idea of the paper is to use a similarity module to transfer the knowledge from known categories to novel categories. Then the affordance prediction on the novel object can be learned with just a few exploration actions. The paper is compared with several existing methods on standard benchmarks.", "questions": "I think overall the paper addressed an interesting problem and proposed an effective solution, but I think further clarification on the method is necessary. Please see my comments above.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important and interesting problem of transferring affordance knowledge from known categories to novel categories.\n\nThe paper proposes a similarity module for effective affordance knowledge transfer and achieves quantitativel  better results than existing methods.\n\nThe paper is overall easy to follow.\n", "weaknesses": "Method:\n\nI have a couple of questions regarding the learning procedure and intuition.\n\n1. In Figure 2, to train the similarity module, the affordance module predicts the affordance map on the novel object and the similarity module predicts the per-point similarity. In my understanding, is the similarity module exactly the same with the affordance module but just learned with different data and learning objectives? If so, in eq (1), is the Aff the affordance module or similarity module in Figure 2? I am not sure if eq (1) and Figure 2 are aligned, maybe explaining in more detail could be helpful.\n\n2. It is not clear to me how the affordance prediction is updated during exploration/inference. Is it more of a test-time-adaptation? What module parameters are updated?\n\n3. I am not sure if the intuition of generalization using local geometries is well supported by the design. Is the similarity module responsible for this? If so, how do local geometries play a part? Isn’t it still predicted for all N observed points with global information? Since this builds the key idea of cross-category generalization, I think further clarification is necessary.\n\n4. In the paper, 10 instances from each novel category are used for few-shot learning. I wonder, for example, for Where2Act baseline, is it also trained with only 10 instances? There is also no ablation on the number of instances required, could it be fewer?\n\n5. The paper only explores push and pull. Maybe more actions can be studied to validate the method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and your positive feedback. Thank you for the valuable questions! Here, we will address the raised questions and comments in the same order as they appear in the reviews.  Please Feel free to follow up if you have more questions.\n\n> is the similarity module exactly the same with the affordance module but just learned with different data and learning objectives? If so, in eq (1), is the Aff the affordance module or similarity module in Figure 2?\n\n  Sorry for not being clear. We will make this clearer in the final paper.\nThe two modules share the same encoder but have separate decoders. Their outputs are also defined differently. The Eq(1) defines the supervision of the similarity module, which is the accuracy of the prediction from the affordance module. The \"Aff\" in Eq(1) refers to the affordance module in Figure 3 (top). As shown in figure 3, the similarity module learns from\nthe comparison of affordance prediction and ground truth (bottom right), which correspond to $Accu(Aff(O_i,p_i,R_i),m_i)$ in Eq(1). As for the Figure 2, it describes the general pipeline of our system, which is training affordance module on supporting set (left) and learning similarity through cross-category comparison (middle) and then perform few-shot learning under the guidance of the similarity module (right). Any further questions are welcomed!\n\n> how the affordance prediction is updated during exploration/inference?\n\nAs written in line 187, both the affordance module and the similarity module are adapted using the proposed interactions and their corresponding results. The parameters of the affordance module are updated by the interaction results (successful or not) same as the training process. After the affordance module is updated, the similarity module will be supervised by the accuracy of affordance prediction on these interactions.\n\n> I am not sure if the intuition of generalization using local geometries is well supported by the design.\n\nThanks for your insightful comment! While we don’t explicitly measure the similarity of local geometries, our architecture and the training method exploits this intuition in an implicit yet effective way. Firstly, the backbone (PointNet++) of the similarity module does not only depend on global features. PointNet++ leverages a hierarchical structure to extract features at various scales, with each point having features containing both its local and global information, and thus is able to capture local geometric information. Such ability is demonstrated in other affordance learning works that require learning local geometric information (Where2Act, DualAfford [1, 2]). Second, we train the similarity module in a cross-category manner, and the optimization target implicitly trains the network to focus on local geometries (i.e., recognizing known and unfamiliar local geometries) in order to achieve better generalization towards unseen categories. Finally, qualitative results empirically show that the learned similarity has a clear focus on local geometries. For example, in figure 5, the pot lid (middle) and the top surface of drawers (top) are predicted to be unfamiliar, which shows a clear geometric boundary with parts that the model is familiar with.\n\n> Is the Where2Act baseline also trained with only 10 instances?\n\nYes. Data setting is the same for all baseline comparisons for a fair evaluation. Additionally, we compare our method with the original Where2Act that is trained on abundant data ('full-data' in Table 3). our method achieves comparable performance with only 0.3% data of the \"full-data\" model.\n\n> There is also no ablation on the number of instances required, could it be fewer?\n\nThanks for your advice! We have conducted two more experiments respectively with 2 instance and 5 instances, the results are shown here. We could see the performance slightly drop for pushing and drop dramatically for pulling due to fewer positive supervision and instance-level variance.\n\n|   F-score   | pushing | pulling |\n|-------------|---------|---------|\n| Ours (10 instances)| 41.6%   | 24.2%   | \n| 5 instances | 40.3%   | 14.5%   |\n| 2 instances | 33.9%   | 12.8%   |\n\n> The paper only explores push and pull. Maybe more actions can be studied to validate the method.\n\nThanks for your advice! We use push and pull as they are the most commonly used action primitives in Where2Act and follow-up papers [1, 2, 3]. Following Where2Act, we focus on learning dense visual prior for short-term manipulations and most of the short-term actions start with push or pull. \n\nMoreover, we have conducted two more experiments on two new action primitives following Where2Act: dragging (pulling object parts to the left) and scratch (pushing object parts to the left side using friction). These action primitives are commonly used in the real world (e.g. scratch sliding windows to open them, dragging a laptop to close it). **Please see the global reply for detailed results**. We could see our method still outperforms the baselines. Any further suggestions are welcomed!\n\n[1] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani.\nWhere2act: From pixels to actions for articulated 3d objects. ICCV 2021.\n\n[2] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative\nvisual affordance for dual-gripper object manipulation. ICLR 2023.\n\n[3] Wang, Y., Wu, R., Mo, K., Ke, J., Fan, Q., Guibas, L. J., & Dong, H. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. ECCV 2022."}, {"review_id": "ylowkk67DW", "reviewer": "Reviewer_8WHp", "summary": "The paper focuses on few-shot affordance learning and articulated object manipulation. It introduces a framework named Where2Explore, which consists of the affordance module and the similarity module. Both modules share the same architecture, a Where2Act [24]-like network based on PointNet++ [28]. While the affordance module is trained on a single category, the similarity model is trained across categories to understand similarity in local geometry and manipulation semantics. The model is benchmarked on 942 instances from 14 (3 for train, 11 for test) object categories in PartNet-Mobility [37]. It outperforms previous works [24, 34] in simulation.", "questions": "1. [38, 8] share a similar spirit and motivation in learning local geometry across categories and generalizing to the novel, which should be discussed in the related works.\n2. Table 3: The successful rate of pulling drops for the full-data compared to ours. Any explanation for this observation?\n3. Is the interaction budget (1, 2, 5) per instance or per category (10 instances)? This term seems not well defined.\n4. L233: Which category is the affordance category ($C_{aff}$) in each combination?", "limitations": "The paper includes a section dedicated to limitations; however, it provides minimal discussion on the specific limitations. Please incorporate a comprehensive discussion of failure cases and limitations, including those in the Weaknesses section. To enhance the credibility of this work, it would be beneficial to incorporate quantitative results from real-world experiments and quantitative comparison to other works that follow a different paradigm, such as Neural Descriptor Fields and GAPartNet [8]. This would provide a more convincing evaluation of the proposed methodology.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The methodology is well motivated: Learning similar local geometry across object categories could benefit few-shot generalization in manipulation.\n2. It is impressive to observe the stark differences between the seen categories and the unseen categories. \n3. Under the same settings, Where2Explore exhibits a clear edge over previous works [24, 34] quantitatively.\n4. The demo video is well-prepared.", "weaknesses": "1. The method is limited to pulling and pushing actions, and the action trajectories are short-term and hard-coded. These limitations inherited from Where2Act could result in a significant simulation-to-real gap.\n2. There is only a single example of real-world manipulation, which is insufficient to demonstrate the method’s capabilities. Besides, according to the video, it seems that the gripper may not always hold mugs firmly enough. More real-world experiments should be conducted, and their success rates should be reported.\n3. Few improvements on the architecture. The current module largely follows the design of [24, 34]. While it is commendable to demonstrate new insights using a minimal network design, the current task may benefit from more fine-grained cross-category transfer learning. There is room for improvement in the methods.\n4. Neural Descriptor Fields (Simeonov *et al*.) is also capable of capturing local geometry information using few shots. It would be valuable to discuss and even adapt them for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts and your thoughtful feedback. Thank you for the valuable questions and suggestions!\nHere, we will address the questions and comments in the same order as they appear in the reviews. Please feel free to follow up if you have more questions. \n\n\n> The method is limited to pulling and pushing actions, and the action trajectories are short-term and hard-coded. These limitations inherited from Where2Act could result in a significant simulation-to-real gap.\n\nThanks for your nice comment! We use push and pull as they are the most commonly used action primitives in Where2Act and follow-up papers. Moreover, we have conducted two more experiments on two new action primitives following Where2Act: dragging (pulling object parts to the left) and scratching (pushing object parts to the left using friction). These action primitives are commonly used in the real world (e.g. scratch sliding windows to open them, dragging a laptop to close it). **Please refer to the global reply for detailed results.**\nAs for the short-term issue, we follow the setting of Where2Act that aims to learn an object-centric dense visual prior. Although they are learned based on short-term interactions, such dense prior could guide the first step of long-term tasks. Current works like VAT-Mart[1] and UMPNet[2] have leveraged this kind of prior to perform long-term manipulations, which proves that the learned knowledge could help the downstream tasks.\n\n\n> There is only a single example of real-world manipulation.\n> Besides, according to the video, it seems that the gripper may not always hold mugs firmly enough. More real-world experiments should be conducted, and their success rates should be reported.\n\nWe conduct more real-world experiments, which include affordance and similarity prediction on more novel categories (e.g. bucket, kettle and pot). **Please see the global reply for detailed results.**\nThe problem that the gripper may not always hold mugs firmly enough comes from the small surface-friction between the gripper and the mug, and changing the gripper type could solve this problem.  Thanks for bringing up this point and we will make this clear in the final paper.\n\n> Few improvements on the architecture.\n\nThanks for this comment! As you mentioned, we largely follow the network designs of previous works to demonstrate the new insights with a new learning system. We believe adding more designs to the network architectures is promising for future works to explore. We could let the model explicitly group the points into parts and reason the similarity among these parts across categories (e.g. the body part of a cup and a bottle is similar whereas the handle on the cup is unfamiliar). Such modification to the network might help the reasoning of similarity in part level. \n\n> Neural Descriptor Fields (Simeonov et al.) is also capable of capturing local geometry information using few shots.\n> It would be valuable to discuss and even adapt them for comparison.\n\nThanks for bringing up this related work to us! We will certainly add discussion to this paper in the final revision. Neural Descriptor Fields leverage correlation among different instances in the same category to perform category-level few-shot imitation learning whereas our method focuses on cross-category few-shot learning. Specifically, NDF learns from the demonstration on a mug and transfers the knowledge to unseen mugs. We learn affordance on cabinets and windows and transfer to novel categories like tables and mugs with 5 interactions.\n\n> [38, 8] share a similar spirit and motivation in learning local geometry across categories and generalizing to the novel, which should be discussed in the related works.\n\nThanks for your suggestion! Gapartnet and PartAfford both propose novel supervisions (part-level annotations or sparse labels) on parts to learn the affordance that can generalize to unseen categories, which are very interesting and useful. Different from these works, we do not receive explicit part-level supervisions, instead, our model learns similarity among local geometries through interaction results (i.e. parts are learned by the network itself). We will discuss more about these works in our related works.\n\n> The successful rate of pulling drops for the full-data compared to ours. Any explanation for this observation?\n\nWe observe that models trained with abundant data could learn some corner cases (e.g., pulling the drawer inner surface by friction to make it open), which are not always successful, but included in the dataset as long as the number of random interactions is large enough. In contrast, our method with self-proposed interactions tends to explore geometries that are more meaningful (e.g. the handle of the drawer), thus achieving even higher performance in the successful rate.\n\n> Is the interaction budget (1, 2, 5) per instance or per category (10 instances)? This term seems not well defined.\n\nThe interaction budget refers to the per-instance budget here. We will make it clearer in the paper. Thanks for pointing this out!\n\n> Which category is the affordance category in each combination?\n\nThe affordance category is the first one of each category. We will clarify it in the paper.\n\n> More specific failure cases and limitations\n\nThanks for your valuable suggestion. We will add more failure cases analysis. For example, the similarity module may fail on categories that have significant shape variance, which is shown in the PDF attached. As for the limitation, while our method has dramatically improved the data efficiency, we still need abundant data in the supporting set to form a fine-grained manipulation prior before we can perform few-shot learning. **Please see the global reply for detailed descriptions.**\n\n[1] Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. ICLR 2022.\n\n[2] Universal manipulation policy network for articulated objects. RA-L, 2022.\n"}, {"review_id": "V2EYZ15axT", "reviewer": "Reviewer_WVQD", "summary": "This paper studies the problem of interacting with novel articulated objects. More specifically, a few-shot learning based approach is proposed to adapt to unseen instances at test time. The methodology explicitly leverages similar local geometries, which may be shared across objects from distinct categories. Experimentation reveals superior performance relative to prior work.", "questions": "- Are \"action directions / gripper orientations\" often the normals? Is it possible to constrain the action space to just be normals?\n- Is a separate affordance model trained for each category, such that the similarity model can be trained on \"held-out\" categories?\n- Other alternatives / potential ablations:\n  - As an alternative to the L1 loss for training the similarity module, could a cross-entropy loss have been used for whether the current affordance model was correct on the unseen object? This would also provide a notion of uncertainty of the model (via confidence), which could be used to guide exploration.\n  - What if you returned the output of the affordance model for points with the highest similarity, without any further exploration? Wouldn't this correspond to points for which the model is the most confident in being correct?\n  - Could you train an affordance model which only operates on smaller point clouds (ie smaller field of view, e.g. a patch)? This may force the model to only look at the local geometries which should be shared across categories?\n- \"During few-shot learning, Where2Act will sample interaction on the points with an affordance score closest to 0.5.\" --> Why is this the case? Isn't 0 push and 1 pull? So shouldn't the highest confident action be sampled (ie farthest away from 0.5)?", "limitations": "There is a small section on limitations of the work, which could be expanded.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Ablations are insightful. The ablations adequately test the utility of the design choices of the proposed methodology.\n- Strong results. The proposed approach outperforms baselines from prior work as well as the ablations. The fact that the proposed methodology achieves comparable performance to full-data while using only 0.3% of the data is impressive.", "weaknesses": "- Simplistic problem setting. The problem setting seems to be simplistic. For instance, the robot arm is abstracted away. This completely side-steps the problem of finding a motion plan which actually solves the task in a collision-free manner, limiting real world deployment. Furthermore, the input point cloud is assumed to be cleanly segmented out. It is unclear whether this is a realistic assumption, especially for when the models are deployed to the real world.\n- Limited real world evaluation. It is not clear whether the model truly generalizes to the real world. Furthermore, the section on real world experimentation (both in the main paper and the supplementary) seems to be quite limited. For instance, how is the grasping (gripper open / close) done, as this doesn't seem to be an output of the system? Or, how is the motion plan obtained from the end-effector poses? More extensive (quantitative and qualitative) experimentation on real world data would be preferred to be convincing that the sim2real generalizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts and your feedback. Thank you for the valuable questions!\nHere, we will address the questions and comments in the same order as they appear in the reviews.\n\n> The robot arm is abstracted away. The input point cloud is assumed to be cleanly segmented out.\n\nWe follow the settings of a series of manipulation affordance works like Where2Act and VAT-Mart [1, 2] and focus on learning an object-centric visual prior that could guide the downstream tasks. Therefore we abstract away the robot arm to learn all possible interactions on each object. Current control systems (e.g. Inverse kinematics solver [3]) of robotic arms could achieve motion planning given the position of the end-effector, which facilitates our real-world application. It is also proved by our real-world experiments, where we use impedance control [4] to plan for the arm motion. As for the segmentation issue, since our model consumes partial point clouds following Where2Act and other works, segmentation could be easily performed by using the segmentation mask on rgb images and applying it to the depth maps. We can use SAM [5] to produce such segmentation masks in the real world. \n\n> Limited real world evaluation. How is the grasping (gripper open / close) done? How is the motion plan obtained from the end-effector poses?\n\nThanks for your suggestions! We provide more real-world experiments in the global reply, which includes visualization of affordance and similarity prediction on novel categories (e.g. bucket, kettle and pot). The results demonstrate the generalization ability of our method in real scenarios. **Please see the global reply for detailed results.**\nThe open and close of the gripper is controlled by hard-coded interaction sets defined in Where2Act. The degree of closing is controlled by the force sensor at the end-effector. When the pressure on the gripper is higher than a threshold, the gripper will stop closing. We use Impedance control system [4] for the real-world experiments. Given the poses of the end-effectors, the system could plan the motion trajectory. We will release the code and add more details in the final paper.\n\n> Are \"action directions / gripper orientations\" often the normals? Is it possible to constrain the action space to just be normals?\n\nThanks for your nice question! The action directions refer to the motion directions and the gripper orientations are the poses of the gripper. We do not limit them to the normals since we want a dense prior that is conditioned on specific action parameters. This design enjoys many benefits. For example, a horizontal handle should be grasped by a gripper whose pose is vertical. When pushing a window that moves prismatically, the direction of action should also not be constrained to the normal to make it move.\n\n> Is a separate affordance model trained for each category, such that the similarity model can be trained on \"held-out\" categories?\n\nAs we want one affordance model to learn all categories in the end (which is more practical in the real-world application), we perform few-shot learning on \"held-out\" categories by adapting the same model under the guidance of the similarity module.\n\n> As an alternative to the L1 loss for training the similarity module, could a cross-entropy loss have been used?\n\nAs accuracy is a continuous number in [0, 1], we want to learn a continuous indicator of the uncertainty, thus precisely revealing the degree of uncertainty and finding the geometries with highest uncertainty to interact with. A cross-entropy loss might not guarantee this property. Previous work (Confidence-Aware Learning for Deep Neural Networks [6]) also proves that L1 is better in making the predicted numbers more smooth, meaningful and continuous than CE loss.\n\n> What if you returned the output of the affordance model for points with the highest similarity, without any further exploration?\n\nFollowing Where2Act, we aim to learn the dense pixel to actions mapping instead of just performing a task, which means we need to know which action could work as well as which couldn’t. Such visual priors will be helpful in diverse tasks. Besides, we want to learn affordance on novel geometries when faced with a novel category (e.g. handles of a mug), which requires exploration instead of just choosing the affordance with highest similarity.\n    \n> Could you train an affordance model which only operates on smaller point clouds?\n\nSome kinematic information could only be discovered when seeing the global shapes (i.e. Whether the joint is at limit, whether the surface moves horizontally or vertically). Our network backbone (PointNet++) leverages a hierarchical structure to extract features at various scales, with each point having features containing both its local and global information, which benefits the prediction accuracy as well as the generalization ability.\n\n> Why does Where2Act sample interaction on the points with an affordance score closest to 0.5?\n\nWe train separate models for different action primitives (e.g. pushing and pulling). In each primitive, the affordance\nprediction indicates whether one action will be successful (0 for fail and 1 for success). Seeing from a classification aspect, 0.5 indicates uncertainty.\n\n> Limitations could be expanded.\n\nThanks for your valuable suggestion! We add more failure cases and limitation analysis. **Please see the global reply for more information.**\n\n[1] Where2act: From pixels to actions for articulated 3d objects. ICCV 2021\n\n[2] Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. ICLR 2022\n\n[3] Wikipedia contributors. \"Inverse kinematics.\" Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia\n\n[4] Wikipedia contributors. \"Impedance control.\" Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia\n\n[5] Segment anything. arXiv preprint arXiv:2304.02643. 2023\n\n[6] Confidence-aware learning for deep neural networks. ICML 2020\n"}, {"review_id": "QGjVf29Vqt", "reviewer": "Reviewer_2ur3", "summary": "The paper focuses on the task of predicting affordances given a single-frame observation of an articulated object. The authors utilize few-shot learning to address the novel object adaption problem. However, previous approaches to few-shot learning require per-instance interaction during test time, which can be both costly and inefficient. To address this issue, this paper introduces the Where2Explore framework, which estimates geometric similarity across categories to offer more efficient interaction proposals. Additionally, it enables the transfer of learned affordance to novel cases. Experimental results demonstrate that this method surpasses previous approaches in various aspects.", "questions": "* Why for the visualizations of the similarity prediction in figure 5, the action directions are different for synthetic data and the real data. Does the actual inference process have any difference for synthetic data and real data? For example, the sampled actions for few-shot learning are defined differently for the synthetic and the real data?\n* For the interaction budget, it seems that 1, 2, 5 get better and better performance, is there some performance top bar, and what’s the number of the budgets?\n* In the similarity visualization, why the dark blue is not considered as potential interactions (they should have the lowest similarity score?) Like the pot in figure 5 (the second row). It seems that the visualization mix the affordance and the similarity, which is a bit confusing.", "limitations": "The authors mention the limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation to explore more efficient few-shot learning is intriguing. The concept of leveraging affordance similarities across categories is intuitive and seems to be effective based on the experimental results.\n* The training strategy to learn the cross-category geometry-aware affordance similarity is straightforward and effective based on the quantitative visualization.\n* This paper conducts extensive experiments to compare with prior work and includes sensible ablation studies to demonstrate the effectiveness of the Where2Explore framework.", "weaknesses": "* There are not enough descriptions of the hard-coded intersection sets. Even if it’s proposed in the prior work, it’s better to include more details to be self-contained in this paper. This paper only chooses two actions, “push” and “pull” from the original 6 actions defined in the prior work. For the interaction definition, the action direction should be in the positive hemisphere of the surface normal. It’s not clear if it holds true for the real data. (Because the real data similarity visualization is not based on the surface normal) \n* In the few-shot adaption stage, it’s unclear how many interactions for each point are sampled for the similarity. In addition, the authors mention the similarity is conditioned on the action. It’s unclear if the action is still conditioned in the object/camera/world coordinate or some local coordinate to the interaction points. If the action is still parameterized in the global coordinate, it’s hard to see if the model can really learn the similarity.\n* For the few-shot learning, it’s hard to see the usefulness of the models in the supporting set. The current set of experiments cannot show the usefulness of the supporting set. Is this possible to learn from scratch with the few-shot learning strategy?\n* For the affordance and similarity training, there seems no reason to train similarity at the start, because the affordance predictions are still quite bad. It’s also not clear if training affordance and similarity in two-stage can achieve better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts and your thoughtful feedback. Thank you for the valuable questions and suggestions!\nHere, we will address the questions and comments in the same order as they appear in the reviews.\n\n> There are not enough descriptions of the hard-coded intersection sets. This paper only chooses two actions, “push” and “pull” from the original 6 actions defined in the prior work. For the interaction definition, the action direction should be in the positive hemisphere of the surface normal. It’s not clear if it holds true for the real data.\n\nThanks for your advice. For pushing, a closed gripper first touches the surface and then pushes 0.05 unit-length forward. For pulling, an open gripper approaches the surface by moving forward for 0.04 unit-length, performs grasping by closing the gripper, and pulls backward for 0.05 unit-length. We will add more detail about the action sets in the final paper. \n\nAs for the other action primitives, we also provide experiments on two new action primitives: dragging (pulling object parts to the left) and scratching (pushing object parts to the left side using friction). The results show that our method still outperforms other method, indicating that our system has potential to be applied to more actions. **Please see the global reply for detailed results**\n\nFor the action direction issue, the affordance of actions with directions in the negative hemispheres are supervised by zero during training. Therefore, in the real scenario, our model can distinguish the actions that are in the negative hemisphere and give negative affordance predictions.\n\n> It’s unclear how many interactions for each point are sampled for the similarity. It’s unclear if the action is still conditioned in the object/camera/world coordinate or some local coordinate to the interaction points.\n\nThanks for pointing this out! We randomly sample ten directions for the whole objects and let the similarity module choose the point with lowest prediction. We will clarify it in the paper. As for the action coordinate issue, we only parameterize actions’ directions and gripper orientations. The starting point and ending point are hardcoded regarding the interaction point (e.g. pushing is defined as a closed gripper first touches the point and then pushes 0.05 unit-length forward). Therefore, our model could capture the local similarity conditioned on actions.  \n\n> For the few-shot learning, it’s hard to see the usefulness of the models in the supporting set. Is this possible to learn from scratch with the few-shot learning strategy?\n\nSupporting set provides the affordance module with manipulation knowledge in one category, only with which the transfer of knowledge using the similarity is possible. Besides, the similarity is defined as similarity between geometries in the supporting set and those in novel categories. If no supporting set is provided, both the manipulation knowledge and the exploration strategy can not be learned.\n\n> For the affordance and similarity training, there seems no reason to train similarity at the start.\n\nThanks for your valuable advice! We conduct more experiments on the two-stage training strategy. We observe no evident performance improvements (as shown below). Our observation is that affordance prediction will be stable and reasonable after about 10 epochs\nin the original training. We choose our current training strategy because it is easier to train and more efficient during few-shot learning, as the two modules share the same encoder which can be trained in two modules simultaneously.\n\n| F-score | pushing | pulling |\n|--|--|-|\n|Ours (One-stage) | 41.6%| 24.2% |\n|Two-stages| 42.4% | 22.3% |\n\n> Why for the visualizations of the similarity prediction in figure 5, the action directions are different for synthetic data and the real data. Does the actual inference process have any difference for synthetic data and real data?\n\nThe inference process for real data and synthetic data are the same. We visualize differently simply because it is hard to obtain precise normal directions from the real data, so we choose to use \"up\" direction in the real-world visualization. However, note that our model is capable of predicting actions with any given directions.\n\n> For the interaction budget, it seems that 1, 2, 5 get better and better performance, is there some performance top bar, and what’s the number of the budgets?\n\nThe number of the budgets means the number of interactions on each instance. As shown in the ablation study (line 302), \"full-data\" could be seen as the top bar of the performance since it is trained with abundant data (100 interactions on each shape). The results show using 5 as the interaction budget can achieve comparable performance of the top bar. We also tried interaction budget 10 and the performance did not show an increase (less than 3% improvement in both pushing and pulling). We will clarify it in the paper.\n\n> In the similarity visualization, why the dark blue is not considered as potential interactions?\n\nFollowing the setting of Where2Act, VAT-Mart, DualAfford [1, 2, 3], points that cannot result in the movement of an articulated part (i.e., the points that can only lead to the movement of the whole object) are segmented out, resulting in dark blue. Thanks for pointing this out! We will clarify it in the paper.\n\n[1] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani.\nWhere2act: From pixels to actions for articulated 3d objects. ICCV 2021.\n\n[2] Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong.\nVat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. ICLR 2022.\n\n[3] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative\nvisual affordance for dual-gripper object manipulation. ICLR 2023.\n"}, {"review_id": "jBGrnjOmwM", "reviewer": "Reviewer_3SiD", "summary": "This paper first proposes a new task, cross-category few-shot learning for articulated object manipulation. And it further proposes a framework that first explictly measures the semantic similarity of local geometries,  interacts with the objects of novel categories according to the similarity, uses the interaction results to update the affordance estimation network, and finally manipulate the target object. Authors perform experiments in simulation and the proposed method outperforms existing methods. They show a successful demo on a real robot.  ", "questions": "1. The `semantic' used in this paper is ambigious. You may add more explaination or use another term.\n2. The similarity learning part needs more details. How do you use Eq(1) to learn the similairty?\n3. How do you decide to stop the few-shot training?\n4. What is sliding joint in L235?", "limitations": "The limiation of low success rate is not discurssed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strengths of the paper are:\n1. The paper formulates a new problem, cross-category few-shot affornace learning, which may be useful for robot applications;\n2. The paper proposes a framework to address the proposed problem, which leverages the geometric similarity to guide the exploration on objects of novel categories. The paper designs a cross-category similarity learning pipeline to learn the similarity metric.", "weaknesses": "1. Statistics, eg. std, are not provided in the experimental results, especially the variance casued by the choose of few-shot learning instances.\n2. While the results of the proposed method are highed than baselines, the successful rate is still not statisfactory, especially the result of `full-data' is still low. More discussion on the results and illustration of failure cases can be added.\n3. The assumption of the paper that similar geometry indicate similar affordance for novel categories may be not solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts and your thoughtful feedback. Thank you for the valuable questions and suggestions!\nHere, we will address the questions and comments in the same order as they appear in the reviews. Further questions are welcomed!\n\n\n> Statistics, eg. std, are not provided in the experimental results.\n\nThe detailed results (144 items) of different kinds of category combinations are shown in Table 1 in the supplementary material,\ndifferent combination shows consistent results, which demonstrate the performance of our method is not affected by the choice of few-shot learning instrances.\n\n> While the results of the proposed method are higher than baselines, the successful rate is still not satisfactory.\n\nWe follow Where2Act [1] and choose the sample successful rate to evaluate our prediction because it balances the precision and recall of successful predictions, which aims to evaluate coverage along with accuracy.\n\n> More discussion on failure cases and limitations.\n\nWe add more failure cases analysis. For example, the similarity module may fail on categories that have significant shape variance (especially if the quality of the point cloud is low), which is shown in the PDF attached. When the size of uncertain geometry is very small, similarity might fail to capture the geometric information. **Please see the global reply for detailed discussion.**\n\n>  The assumption of the paper that similar geometry indicates similar affordance for novel categories may be not solid.\n\nThanks for your thoughtful comment! Our similarity does not only depend on geometries but also on global features and action directions. As shown in the supplementary, the surface of a door is predicted to be similar to the front surface of a microwave but is different to the top surface of the microwave even if they are both flat surfaces. Moreover, our few-shot learning could further explore whether the geometry is similar through further interactions. We are happy to discuss more if you have more questions.\n\n> The 'semantic' used in this paper is ambiguous. You may add more explanation or use another term.\n\n'Semantic' is used to clarify that the similarity here is not only based on geometries, but also based on the given action type (push or pull), which contains manipulation semantic information for different actions. For example, handles are significant in pulling whereas less important in pushing, and a horizontal handle could not be grasped by a gripper whose pose is also horizontal. We will add more explanations to make it more clear or change the term. Thanks for pointing it out!\n\n> The similarity learning part needs more details. How do you use Eq(1) to learn the similarity?\n\nThe Eq(1) defines the supervision of the similarity module. Specifically, given a partial observation $O_i$, point to interact $p_i$ and the action direction $R_i$, the similarity is defined as an estimation of the accuracy of affordance prediction on this input $Aff(Oi,pi,Ri)$. The accuracy is computed by the correctness of affordance prediction during training $Accu(Aff(Oi,pi,Ri))$. The module is supervised using a L1 loss as described in Sec. 4.4. Any further questions are welcomed!\n\n> How do you decide to stop the few-shot training?\n\nAs described in line 43 of the supplementary, the exploration will stop if the average score of similarity prediction exceeds a bar (0.9) or the interaction budget is reached.\n\n> What is sliding joint in L235?\n\nSliding joint is the prismatic joint, thanks for pointing out this typo!\n\n[1] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani.\nWhere2act: From pixels to actions for articulated 3d objects. ICCV 2021.\n\n\n\n\n"}]

Author rebuttal:
We thank you all for your efforts and insightful comments! We thank reviewers for appreciating our motivation: "The methodology is well motivated", "The motivation to explore more efficient few-shot learning is intriguing", and design: "The concept of leveraging affordance similarities across categories is intuitive", "The paper proposes a similarity module for effective affordance knowledge transfer".
Below we clarify some common concerns. Thanks again for valuable comments! Any further questions are welcomed!

> More real-world evaluations

We conduct more real-world experiments on diverse objects, which includes affordance and similarity prediction on more novel categories (10 instances including buckets, pots and kettles). We choose the vertically pulling up as our action for the real-world experiments and give 5 interaction budget for each instance. While our model is only trained on cabinet, window and faucet and the affordance module fails to give reasonable predictions, the result indicates that the similarity module could still provide reasonable predictions on novel objects in the real world, guiding the exploration of uncertain yet important geometries. After the exploration, the adapted affordance could reveal the dense kinematic information on the novel objects. **Please see the PDF attached for detailed visualization.**

> More action primitives

We use push and pull as they are the most commonly used action primitives in Where2Act [1] and follow-up papers. Moreover, we have conducted two more experiments on two new action primitives following Where2Act: dragging (pulling object parts to the left) and scratching (pushing object parts to the left side using friction). These action primitives are commonly used in the real world (e.g. scratch sliding windows to open them, dragging a laptop to close it). We could see our method still outperforms previous baselines (especially in dragging, which is more demanding). **Please see the PDF attached for detailed results.**

> More discussion on failure cases and limitations

We add more failure cases analysis. For example, the similarity module may fail on categories that have significant shape variance (especially if the quality of the point cloud is low), which is shown in the PDF attached. Besides, when the size of uncertain geometry is very small, the similarity module might fail to capture the geometric information. **Please see the PDF attached for detailed visualization.**
As for the limitation, while our method has dramatically improved the data efficiency, we still need abundant data in the supporting set to form a fine-grained manipulation prior before we can perform few-shot learning. Besides, in order to extend our method to long-term manipulations, we might need to rely on some recent works such as VAT-Mart [2] and UMP-Net [3]. 

[1] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani.
Where2act: From pixels to actions for articulated 3d objects. ICCV 2021.

[2] Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong.
Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. ICLR 2022.

[3] Xu Z, He Z, Song S. Universal manipulation policy network for articulated objects[J]. IEEE Robotics and Automation Letters, 2022.


Meta-review: This paper studies a novel approach on affordance with articulated objects. The goal is aiming at generalization to novel categories. All the reviewers agree to accept the paper after the rebuttal and the AC agrees.
Decision: Accept (poster)