PartAfford: Part-level Affordance Discovery
Authors: Chao Xu
Yixin Chen
He Wang
Song-Chun Zhu
Yixin Zhu
Siyuan Huang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Understanding what objects could furnish for humans—learning object affordance—is the crux of bridging perception and action. In the vision community, prior work has primarily focused on learning object affordance with dense (e.g., at a per-pixel level) supervision. In stark contrast, we humans learn the object affordance without dense labels. As such, the fundamental question to devise a computational model is: What is the natural way to learn the object affordance from geometry with humanlike sparse supervision? In this work, we present the new task of part-level affordance discovery (PartAfford): Given only the affordance labels for each object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part of the object corresponds to a certain affordance category. We propose a novel learning framework that discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization without dense supervision. To learn and evaluate PartAfford, we construct a part-level, cross-category 3D object affordance dataset, annotated with 24 affordance categories shared among >25, 000 objects. We demonstrate through extensive experiments that our method enables both the abstraction of 3D objects and part-level affordance discovery, with generalizability to difficult and cross-category examples. Further ablations reveal the contribution of each component.

Under review as a conference paper at ICLR 2023
PARTAFFORD: PART-LEVEL AFFORDANCE DISCOVERY
Anonymous authors
Paper under double-blind review
Trained with per-shape sparse annotations (affordance label set)
Infer part decomposition and corresponding part affordances
shape
backrest
sittable
framework
sittable
sittable
label
sittable
backrest
backrest
Input: 3D shape
Output: Discovered 3D parts w/ their affordances
framework
framework
armrest
framework
Figure 1: The proposed PartAfford: discover 3D object part affordances by learning contrast affordance compositions. During training (left), given weak annotations (per-shape affordance label set), a learning framework is devised ground affordance (e.g., backrest) to 3D part (e.g., sofa back) through learning cross- category, affordance-rrlated shapes (e.g.. chair, sofa) with various affordance compositions. At time (right), the learned model decomposes 3D object into parts and infers the part-level affordances.
ABSTRACT
Understanding what objects could furnish for humans learning object affordance- is the crux of bridging perception and action. In the vision community, prior work has primarily focused on learning object affordance with dense (e.g., at a per-pixel level) supervision. In stark contrast, we humans learn the object affordance without dense labels. As such, the fundamental question to devise a computational model is: What the natural way to learn the object affordance from geometry with humanlike weak supervision? In this work, we present the new task of part-level affordance discovery (PartAfford): Given only the affordance labels for each object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part of the object corresponds to certain affordance category. We propose a novel learning framework that discovers leve- representations by leveraging only the affordance set supervision and geometric primitive regularization without dense supervision. To learn and evaluate PartAfford, construct a part-level, cross- category 3D object affordance dataset, annotated with 24 affordance categories shared among > 25, 000 objects. We demonstrate through extensive experiments that our method enables both the abstraction of 3D objects and part-level affordance discovery, with generalizability to difficult and cross-category examples. Further ablations reveal the contribution of each component.
INTRODUCTION
The human vision system could swiftly locate the functional part upon using object for specific tasks (Land et al., 1999). Such a critical capability in object interaction requires fine-grained ob- ject affordance understanding. Affordance, coined and originally theorized by Gibson (Gibson & Carmichael, 1966; Gibson, 1979), characterizes how humans interact with human-made objects and environments. As such, affordance understanding of objects and scenes has significant influence bridging visual perception and holistic scene understanding (Huang et al., 2018 8b;a; Chen et al., 2019) with actionable information (Soatto, 2013; Han et al., 2022).
Object affordances have two main characteristics. First, object affordances are not defined in terms of conventional categorical labels in computer vision; instead, they defined by the associated actions

---

Under review as a conference paper at ICLR 2023
for various tasks and are naturally cross-category. For example, both chair and sofa can be sat on, which indicates they share the sittable affordance. Similarly, desktop and bookshelf share the support affordance. Second, object affordances are intrinsically part-based. We could easily associate sittable affordance with the seats of chairs and sofas, and support with the boards of desktop and bookshelf. As such, the ability learn part-based, cross-category affordance is essential demonstrate the general object affordance understanding.
In passive affordance learning, prior literature follows the supervised learning paradigm, in which dense affordance annotation on the objects is fed as supervised signals (Deng et al., 2021). However, this line of thought depends heavily on the quality of dense annotation, which significantly deviates from how we humans learn to understand affordance. Humanlike supervision would be: "you can sit on this chair and rest your arm," "you can open the lid and hold water with the cup." In this paper, we try to answer: How to distinguish each object part while recognizing corresponding affordances with such weak and natural supervisions?
To tackle this problem, we present PartAfford, new task of part-level affordance discovery, which learns the object affordance with the natural supervision of the affordance set. As shown in Fig. I, by providing only the set of affordance labels for each object, the algorithm is tasked to decompose the 3D shapes into parts and discover how each part corresponds to a certain affordance category. which is challenging and under- ander-explored in the area of generalizable part-level object understanding and affordance learning.
To address this, we propose novel method that discovers part-level representations with self- supervised 3D reconstruction, affordance set supervision, and primitive regularization. The proposed approach consists of two main components. The first component an encoder with slot attention for unsupervised clustering and abstraction. Specifically, we encode the 3D object into visual features and abstract the low-level features into a set of slot variables (Locatello et al., 2020). The second component is a decoder built upon the learned slot features. It has three output branches that jointly reconstruct the 3D parts and object, predict the affordance labels, and regularize the learned part-level shapes with cuboidal primitives. Our method does not rely on dense supervision but instead learns from the weak set supervision. discovers the part-level affordance by learning the correspondence between affordance labels and abstracted 3D object parts.
Learning and evaluating PartAfforl demands collections of 3D objects and their affordance labels for object parts. Prior work on visual affordance learning (Hassanin et al., 2021) either focuses on 2D objects and scenes or lacks part-based annotation (Deng et al., 2021). Hence, we construct a part-level, cross-category 3D object affordance dataset annotated with 24 affordance categories shared among over 25, 000 3D objects. The 3D objects are collected from PartNet dataset (Mo et al., 2019b) and the PartNet-Mobiligg dataset (Xiang et al., 2020). The 24 part affordance categories are defined in terms of adjectives (e.g.. "sittable") or nouns (e.g.. "armrest"); : they describe how object parts could afford human daily actions and actívities. We annotate the part-level object affordances by manually mapping the fine-grained object part defined in PartNet to the part affordances defined in this work.
By experimenting on this newly constructed PartAfford dataset, we empirically demonstrate that our method jointly enables the abstraction of 3D objects and part-level affordance discovery. Our model also shows strong generalizability on hard and cross-category objects. Further experiments and ablations analyze each component's 's contribution and point out future directions.
In summary, our work makes four main contributions:
We present a new PartAfford task for part-level affordance discovery. Compared to the prior densely- supervised learning paradigm, PartAfford learns the visual object affordance more naturally. We propose a novel learning framework for tackling PartAfford, which jointly abstracts 3D objects into part-level representations and discovers affordances by learning the affordance correspondence.
We build the benchmark for learning and evaluating PartAfford by curating a dataset consisting of
3D objects and annotating part-level affordances.
We empirically demonstrate the efficacy and generalization capability of the proposed method and analyze each component's 's significance via a suite of ablation studies. Code and data will be released for research purposes.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Affordance Learning Affordance learning is a multidisciplinary research field of vision, cognition. and robotics. In general, "affordance" is first perceived from images (Gupta et al., 2011; Kjellström et al., 201 I; Zhu et al., 2015; Myers et al., 2015; Roy & Todorovic, 2016) or videos (Xie et al., 2013; Zhu et al., 2016; Fang et al., 2018; Nagarajan et al., 2020), followed by cognitive reasoning (Zhu et al., 2015; 2020), and finally serves for task and motion planning in robotics (Nagarajan & Grauman, 2020; Mo et al., 2022). Prior work tackles affordance at various scales and representationss Although affordance has been studied at the scene level (Zhao & Zhu, 2013; Gupta et al., 2015; Roy & Todorovic, 2016), object level (Nguyen et al., 2017; Mo et al., 2021; Gadre et al., 2021), ) and associated with generated human poses (Zhu et al., 2015; Wang et al., 2017b), few attempts study affordance as 3D shape analysis task (Yu et al., 2015; Liang et al., 2016; Zhu et al., 2016; Wang al., 2017a) since it would normally require large-scale, high-quality 3D data. A recent work (Deng et al., 2021) benchmarks several affordance estimation tasks on PartNet (Mo et al., 2019b) with dense affordance heatmap supervisions, annotated by densely selecting keypoints without considering affordance compositionality. Prior work also tackles affordance learning through interaction based methods, either from human demonstration videos (Kjellström et al., 2011; Nagarajan al., 2019; 2020) or iimulation-based active learning (Wang et al., 2022; Mo et al., 2021). The first usually infers high-level and coarse 2D affordance, and the second is often restricted to basic manipulations in specific domains. In comparison, PartAfford studies affordance in weakly supervised manner, such that the affordance discovery will be guided by affordance set matching and geometry abstraction. The new affordance dataset we construct provides fine-grained, pert- 3D affordance annotations, tailored for the weak supervision setting and affordance compositionality study.
Object-centric Learning Object discovery has been studied in iterative end-to-end fashion (Greff et al., 2017; Van Steenkiste et al., 2018; Burgess et al., 2019; Engelcke et al., 2019; Greff al., 2019; Du al., 2021). Recently, Locatello et al. (2020) presents the slot attention module, an efficient and generic framework for object-centric representation extraction. It is capable of modeling compositional nature in synthetic scenes with multiple simple geometry shapes (Kabra et al., 2019). Subsequently, Stelzner et al. (2021) and Yu et al. (2021) apply slot attention on unsupervised 3D- aware scene decomposition, integrating NeRF (Mildenhall et al., 2020) as object representations. They demonstrate that slot-based bottleneck could perform reasonably datasets with a textureless background. In the reakky-suupevised regime, methods have been proposed for 3D semantic segmentation with scene-level labels (Wei et al., 2020; Ren et al., 2021). However, these methods rely on additional input (e.g., color, normal) and abstract object-level fcatures in different data domains. Our work takes one step further to tackle the challenge of part-level affordance discovery of 3D objects; part discovery is more complex than object discovery, primarily due to the ambiguity in the object part segmentation without applying additional constraints. Fortunately, for man-made objects, affordances are attached to objects at the part level. This observation implies the possibility of combining part discovery and affordance learning with minimal supervision. In this work, we integrate part discovery with affordance estimation, hoping that affordance information would help discover object parts sharing similar affordances.
synthetic multi-view RGB
Unsupervised Geometric Primitive Modeling Whereas supervised geometric primitive abstrac- tion methods (Mo et al., 2019a; Yang et al., 2020) require dense hierarchical annotations, unsuper- vised frameworks using cuboid-based (Tulsiani et al., 2017; Sun et al., 2019), superquadrics based (Paschalidou al., 2019; 2020), or other genus-zero-shen shape (Deng al., 2020; Paschalidou et al., 2021) primitives discover structural information naturally embedded in the geometry. Recently, Yang Chen (2021) unsupervisedly learn the cuboid-based shape abstraction with shape co-segmentation. Yet, it relies heavily the ground-truie truth point normals for accurate abstraction and lacks seman- tic representation for object understanding. In our affordance discovery framework, we leverage the cuboidal regularization to refine the reconstructed affordance part, which distinguishes densely connected 3D parts by providing geometric prior, thus improving the affordance part discovery.
3 TASK DEFINITION
We formulate the new task PartAfford as discovering the part-level object affordance with the affordance set supervision. We define K = 24 common affordance categorics S= {*+} 1 as "sittable" and "openable," for object understanding. Input is given collection of N objects
k== such
3

---

Under review as a conference paper at ICLR 2023
(a) Abstraction Encoder
T-iteration 3D Slot Attention
Attention Mask Visualization (Stots Allend to Parts)
(Sicts Compete for Parts)
CNN Encoder
LayerNorm
SonMaax
Input shape
feal + pas_ambes Lambedd
GRU
alott
Part Reconstruction Decoder
Affordance Prediction Decoder
(d) Cuboidal Prinitive Regularization Decoder
Set Matching
Park Cuboid Consistency
sittable
beckreett
3D ONN Deccc
Armores
amtrost
rolition
backrestt
sitable
framework
framswork
Recon. Object
scale
Affordiacces Affordances
Prod.
Figure 2: Illustration of the proposed method for PartAfford. Our model contains two main components: abstraction encoder and affordance decoder. (a) Abstraction encoder takes 3D object as input, extracts features with 3D convolutional neural networks, and abstracts it into several slots. Affordance decoder with three branches jointly (b) reconstructs the 3D parts, (c) predicts affordance labels, and (d) regularizes cuboidal primitives. Each predicted cuboid in (d) wraps around the corresponding predicted object part in (b) tightly.
{0:) and their corresponding affordance set labels {A:) where A, = {a == of E and J4 represents the number of distinct assigned affordances for each object PartAfford requires an algorithm to decompose each object into parts and discover the affordance corresponding to each object part. Fig. illustrates the PartAfford task.
METHOD
We propose a novel framework for affordance discovery from 3D objects. integrates unsupervised part discovery with affordance set prediction and geometric primitive abstraction; see Fig. 2. Given 3D shape represented by voxel grids V of resolution 323, our method first encodes the 3D shape into visual features and abstracts it into M slots; each slot represents an abstracted high-level feature for downstream tasks. Next, we utilize decoder with three branches to jointly (i) decode the features into parts, (ii) predict the affordance label, and (iii) regularize the parts with cuboidal primitives. Below, we describe detail how each module is constructed and the loss design.
4.1 ABSTRACTION ENCODER
The encoder takes a 3D shape input and abstracts part-centric latent codes in unsupervised manner. It consists of feature extraction module and a 3D slot attention module; see Fig. 2a.
The feature embedding backbone encodes the input voxels and generates D = 64 dimensional feature for each voxel. Following (Mescheder et al., 2019), voxels are encoded by five layers of 3D convolutional neural networks. The embedded feature is then augmented with absolute positional embedding (Locatello et al., 2020).
Feature Extraction
3D Slot Attention The 3D slot attention architecture, adapted from Locatello al. (2020), serves as the part-centric representational bottleneck between the 3D feature embedding network and the downstream decoders. The encoded feature of 3D shape is fed into an iterative attention module, where M randomly initialized slots are updated for T = 3 iterations through GRU (Cho et al., 2014). During each iteration, the attention coefficients are calculated by applying softmax normalization over the slots on the dot-produc similarity between queries (i.e., linearly-mapper and keys (i.e., eeearly-mapped input features). The attention coefficients are then applied as weights for aggregating the values (i.e., linearly-mappe input feature) and updating the slots.
3D slot features)
Since the inputs-to-slots sloss attention assignment is normalized over the 3D slots, those slots compete to attend to clustering of the input 3D shape. Such clusterings are similar to couman-defined parts on common objects. 3D slot attention masks naturally segment the object through iterations. An example of the learned 3D attention masks is shown in Fig. 2a.
4

---

Under review as a conference paper at ICLR 2023
4.2 AFFORDANCE DECODER
Shown in Fig. 2b-d, the affordance decoder takes part-centric slot features as input, followed by three branches for 3D part reconstruction, affordance prediction, and primitive regularization. The decoder parameter is shared across slots.
3D Part Reconstruction We design a 3-layer 3D transposed convolutional decoder followed by single MLP layer reconstruct voxel values y and a voxel mask for each slot. The mask is normalized across slots with softmax, which generates normalized mask Å Th E R32x32x33 It is then used to compute the weighted sum of voxel values across slots and combine the reconstructed parts (DM)M 1==1 into full 3D shape V: D - Em=1_ Am"mm. The 3D part reconstruction branch is self-supervised by the reconstruction loss between original voxels and reconstructed voxels V; we use the binary cross-entropy (BCE): Lrcon = BCE(V, V).
Affordance Prediction We predict one-hot affordance label for each slot with two-layer MLP with sharing weights across slots for classification.
The affordance prediction branch is weakly-superviste each voxel. Instead, only the affordance label set for the entire object is used as the supervision signal. The model is tasked to learn the alignment between the abstracted parts and the affordance labels from set supervision.
we do not provide affordance labels for
As defined in Sec. the ground-truth set of affordance labels for input 3D object is denoted A. We denote Á the set of slot affordance predictions. Ag is permutation of elements in Á, where a E B and 6 represent all M! possible permutations. Lmatch is the pairwise matching cost between two sets, which can be calculated by mean square error (MSE) or cross-entropy:
Pped min maatch (A, Az). o
Due to the order-invariant nature of slot modules, we apply the Hungarian matching algorithm (Kuhn, 1955), with Huber loss as the pair-wise matching cost, to calculate the set-based (Carion et al., 2020) affordance prediction loss in Eq. (1).
Cuboidal Primitive Regularization As a generalized soft k-means algorithm, the slot attention mechanism heavily relies on visual cues, such as the clustering of pixel colors on the image. As such, it cannot perform precisely in a crowded scene with overlapping objects even (Locatello et al., 2020). In the 3D voxel regime, segmenting objects into parts is challenging since every voxel is connected to neighboring voxels without distinguishable visual appearances.
toy image dataset
Therefore, we introduce the cuboidal primitive regularization module, providing a geometric prior for segmentation: Human-mace objects usually have geometric regularity, and cuboid is a concise structural representation for abstraction.
From each slot embedding, the cuboid abstraction module predicts cuboid parametrized by two vectors (Yang& & Chen, 2021): a scale vector E RR3 and a quaternion vector r R for 3D rotation. Of note, calculate the cuboid center from the weighted mean of voxel positions in the slot.
To evaluate how the predicted cuboid fits the reconstructed part in the m-th slot, we first compute the Euclidean distance d'd from each voxel pº" its closest cuboid face. Next, we calculate the weighted sum distance for all voxels, where the weight o fmm is the reconstructed voxel value within Jo. 1]. Additionally, we designed binary surface mask (() that masks out internal voxels in the loss. We also regularize the cuboid loss by adding a cuboid scale penalty term, i.e., Lı norm for the scale vector. Thus, the loss encourages cuboid to wrap around a solid object part tightly. The regularization loss for all the slots is defined as:
Leuivoid Σ
Ascate || 8
Σ
(i) d:
Total Loss Taking together, the total training loss is the sum of 3D reconstruction loss, affordance prediction loss, and primitive regularization loss:
Lusual Arewonneecca + Apuredt Epredd + Saudnoia&
(3)
5

---

Under review as a conference paper at ICLR 2023
(a) Our affordance desorittion for "openablc"
(b) An ecample from Parthle
hanging
Eumination
eerroeal
backneal
consaimment
framswork
sitable
twstablle
Openable:
Affordance Annotation
Iquidcontaliment
step
cutling
Dettrstonn Parts could moved
operaable
tyable
pounsbiu
lover
Hkige-ike mechariam antoclatedd abject.
handle
heacrest
support
diplay
- Deetalls Usually, handle found attached to
an opennable part (more in supp)
+ MMal-tttee Openatia given or by ooer
openable
pinchable
potental co exicting affordamoes like contaiment
Tever
wneggrop
.... coorturrboee boord,
audibl
(c) clallwaehh sugmanieed from SAPIEN
openable
containeent
openable
inment
(d) cross-calegory, fina-graned affordance labals
Part affordance dataset. (a) Description for the "openable" affordance construct the mapping. (b) Given the part hierarchy of a door from PartNet (Mo al.. 2019b), we annotate its affordance labels by manual mapping and inspection. (c) Given a dishwasher from PartNer- Mobility (SAPIEN) (Xiang et al., 2020) and its kinematics, we rotate the door frame to include 3D objects with different articulation states. (d) Some exemplar 3D object models with color-coded affordance visualization.
Figure 3:
where Arecon Acuhoid, and Apred are balancing coefficients.
Of note, with the current architecture design, for the first time, we demonstrate the capability of part-level affordance discovery from set labels. The exploration of more complex and practical modules is left for future work.
5 PART AFFORDANCE DATASET
To benchmark PartAfford and facilitate the research in affordance understanding, we construct part-level 3D object affordance dataset. We focus on 24 cross-category, fine-grained affordance labels as shown in Fig. 3. The dataset is annotated with over 25, 000 3D CAD models from the PartNet dataset (Mo et al., 2019b) and 625 articulated objects among categories from the PartNet-Mobility dataset in SAPIEN (Xiang et al., 2020). Below, we describe how to define part affordances and generate affordance annotation. See appendix for more details.
Part affordances in our dataset are defined in terms of adjectives (e.g., sittable) or nouns (e.g., armrest), which describe how object parts could afford human daily actions and activities. We adopt certain common affordance categories from comprehensive survey of visual affordance (Hassanin et al., 2021), e-g., containment, sittable, support, openable, rollable, display, and wrapgrasp. However, they grarss--rriee-eiihher the object level or scene level. For example, "openable" only indicates whether an object can be opened, not on which object part can afford the object to be opened.
To pursue fine-grained understanding of object affordance, we manually construct one-to-multiple mapping from 479 kinds of object part labels defined at the finest granularity in Mo et al. (2019b) to 24 potential affordance labels, given the detailed affordance description. We provide expert-defined descriptions for 24 affordances to guarantee the quality and consistency of the mapping construction. An example is shown in Fig. 3a.
Given the part hierarchy of 3D object, we can get the corresponding affordance annotation by mapping. We also perform a manual inspection to correct the affordance labels, especially for some fine-graines parts, according to their specific geometry and physics properties. For instance, different door handles will be mapped to different affordance labels (twistable, lever, etc.) according to how they should be operated (Fig. 3b).
The PartNet dataset does not contain articulation information, making affordances such as openahle not geometrically distinguishable. Therefore, generate set of shapes with openable affordance from the Parbiit-Mobility dataset by capturing 3D shapes with various opening angles (Fig. 3c).
As can be seen from Fig. 3d, each affordance type-due to its cross-category nature-may be found on various object part instances. For example, openable is usually afforded by rotatable doors for unobstructed access. Under such criteria, the door frame of dishwasher and the surface board of a door are both mapped to openable. Please refer to the appendix for a full list of all affordance categories, descriptions, and mapping examples.

---

Under review as a conference paper at ICLR 2023
openable
framework
handle
support
sittable
amrest
backrest
(b)
(
Input
GT
Recon.
Slot
Siot2
Blot3
Slot
Input
GT
Recon.
Skor
Slat2
Siot
Slor4
Figure Qualitative results on three curated subsets: "openable" (a-b), "support" (c-e), and "sittable" (f-h),
EXPERIMENTS
In this section, we design and conduct comprehensive experiments to evaluate the proposed method. Fig. 4 visualizes our main results. We present both quantitative and qualitative comparisons of baseline models and other variants. We also evaluate the model generalization on novel shapes, analyze failure cases, and propose potential improvement directions. Please refer to the appendix for additional experimental settings, results, and analyses.
6.1 EXPERIMENTAL. SETTINGS
Benchmarks To benchmark PartAfford, we curate different subsets samples from our constructed dataset. Specifically, we study the subsets related to the most representative affordance categories "sittable," "support," and "openable" separately, where the subsets are created by collecting all cross- category objects that have the corresponding affordance label in our dataset. They contain kinds of affordances from object categories, covering 17, 842/ 25K 71% instances. For each subset, we learn to distinguish all the affordance labels appear in the 3D objects. Note that although part can have multiple affordances as mentioned in Sec. 5, we only keep the most prioritized affordance for each part ease the ambiguities in learning.
Evaluation Metrics In PartAfford, we evaluate the performances of part discovery (clustering similarity), 3D reconstruction, and affordance prediction.
Part Discovery: We use the Intersection over Union (IoU) to evaluate the part similarity. Specifically, we employ Hungarian matching to find the best matches between the reconstructed parts and ground truth parts using voxel loU as the matching score. Then we compute the mean loU (mloU) by
averaging the IoU between best matches.
3D Reconstruction: We evaluate the shape reconstruction quality by Mean Squared Error (MSE). Affordance Prediction: Following Locatello et al. (2020), we use Average Precision (AP) to evaluate the prediction accuracy. A correct prediction means an exact match of the affordance label set.
Baselines and Ablations Since we are the first to propose and formulate PartAfford, we have no previous work to make direct comparisons. Therefore, we compare our method with two designed baseline models and three variants:
Slot MIP: simple MI.P-based baseline where we replace Slot Attention with an MLP that maps from the learned feature maps (resized and flattened) to the (now ordered) slot representation.
IODINE: baseline where replace the Slot Attention with an object-centric learning method
IODINE (Greff et al., 2019) to abstract and cluster the encoded feature.
Ours w/o Afford & Cuboid: our model variant that only keeps the 3D part reconstruction branch.
Ours w/o Afford: our model variant without the affordance prediction branch.
Ours w/o Cuboid: our model variant that discards the cuboidal primitive regularization branch.
Ours Full: our full model with all branches.
6.2 IMPLEMENTATION DETAILS
Learning Strategy To stabilize the training, we split the training into two stages. In the first stage, train the decoder only with 3D part reconstruction and affordance prediction branches. In the
7

---

Under review as a conference paper at ICLR 2023
recon, onlyinul
fremework
backrest
sittable
ammrest
support
StoMMLP
Ours (full)
Input
Recm
Siot
Slat
3
Skor
Inport
Stor
Slot
(a) Comparisons between the Slot MLP and model.
viC AI
wib Out.
ww ATT..
Wi
wib At:
W Ar.
Irput
Recon
Stoc
Slat
Skot
Siet4
Recon.
Siot
Slat
Slot
(b) Comparisons between models without and affordance prediction branch. The affordance predic- tion facilitates the abstraction of 3D shape (rows 1-2) and elimination spare slots (rows 3-4).
(c) Comparisons between the models without and with cuboidal primitive regularization branch. The cuboidal primitive regularization helps to guide and improve segmentation.
Figure 5: Qualitative comparison results.
second stage, the cuboidal primitive regularization branch incorporated into joint training with a lower learning rate.
We set learning rate as x 10-4 for the first stage, x 10-4 for the second stage, and apply Adam optimizer (Kingma & Ba, 2014) for optimization. takes 5 5-+ hours on RTX A6000 GPUs for two-stage full-model training of "sitabbee-rellated -related objects. For slot attention, we empirically set the number of GRU iterations T 3. We the number of slots the maximal number of affordance labels that appear in each training subset. For example, we learn the "sittable" with 4 slots, "support" with slots, and "openable" with 3 slots. Appendix C.1 further discusses choices of the number of slots. For the joint loss weight, we set Ayecon = 1.0, Apred = 0.5, Acuboid = 0.1.
Hyperparameter
6.3 RESULTS AND ANALYSIS
Tabs. I and tabulate the quantitative performances of all the models under different settings. Fig, qualitatively shows capability our method and Fig. compares different models. Below, we summarize some key findings:
1. The proposed method achieves the best overall performance on the PartAfford task, especially in the part discovery (mean IoU) where it outperforms the baselines by a large margin. This demonstrates the outstanding abstraction capability of our approach given the weak supervision. From Fig. 4,
method can discover the detailed part-level representation with their aligned affordances for the 3D objects.
The most challenging part of PartAfford lies in the part discovery; also where our model differentiates from other baselines. For example, Tab. shows that Slot MLP achieves the best affordance prediction performance (AP) but fails in the part discovery (mean IoU) and 3D reconstruction (MSE). As also shown in Fig. 5a, the Slot MLP cannot segment the object input to parts due the lack of abstraction capability. 3. Affordance prediction branch significantly escalates the part discovery performance since it helps to learn part-affordance correspondence from the affordance composition, which provides contrasts to distinguish different parts among the training objects. Our qualitative results also show that the affordance prediction facilitates the abstraction 3D shape (e.g., rows 1-2 of Fig. 5b) and elimination of spare slots (e.g., rows 3-4 of Fig. 5b);
Cuboidal primitive regularization branch also boosts the part discovery, especially when affor- dance prediction unavailable. This demonstrates that geometric priors play crucial role in
8

---

Under review as a conference paper at ICLR 2023
framework
backrest
sitable
armresi
support
Novel Siltable
farewwrk backrest
sitable
armrest
support
Support
Openabie
Support Sitabie
Sitiable Support
Stor
Sloe3
2
3
Figure 7: Failure cases.
Figure 6: Generalization results.
segmentation when data are not diverse enough. From Fig. Sc, we can see the cuboidal primitive regularization helps to segment better primitives and avoid scattered voxels.
6.4 MODEL GENERALIZATION
Table I: Quantitative results "sittable." We evaluate mean IoU (mloU), mean squared error (MSE), and average precision (AP) on included objects.
With the cross-category nature of affordance, qualitatively test how the learned model can
Model
(%)
MSE 1
AP (P)t
be generalized to novel objects and unseen cate-
Slot MLP
21.5
0.0150
94,5
gories. We conduct model generalization exper-
IODINE
49.2
0.0102
92.5
Ours wlo Afford & Cuboid
31.5
0.0112
N/A
iments by testing hard examples or objects from
Ours w/o Afford
39.4
0.0100
N/A
other categories. Examples from Fig. 6 demon- strate the learned model could be generalized to
Ours Cuboid
55.3
0.0102
92.7
Ours (full)
57.3
0,0097
92.9
objects with diverse shapes. We show the results of testing the learned model on novel object shape (bean bag) (a) from Fu et al. (2021) and unseen categories (b-d). For example, (b) shows the result of learning with "support" and testing on an "openable" object (i.e., microwave). Despite the imperfect reconstructions, partly due to the reconstruction bottleneck's ' impact disentanglement quality (Engelcke et al., 2020), the learned model can identify the functional parts given novel objects.
6.5 FAILURE CASES
We show some failure cases of our method in Fig. 7. For "sittable" and "support," " the failures are commonly caused by (i) the difficulties in reconstructing the fine-grained details of 3D objects with novel shapes; (ii) certain parts that violate the cuboid assumption, and thus hurt other components.
For objects in "openable" category, our model cannot discover and reconstruct "handle," as shown in Fig. 4. This is because the objects with related affordances come from various object categories with diverse shapes, making it chal- lenging for the model to capture such complex mixtures of distributions and reconstruct fine- grained 3D shapes, especially tiny parts This points out future directions to better understand object parts (e.g., segment, reconstruct), and po- tentially interactive learning framework to learn beyond geometry and appearance.
Table 2: Quantitative results "support" and "open- able."
Model
mloU (%)
MSE 1
AP (%)
Slot MLP
36.8
0.0099
91.6
Ours who Afford
34.8
0.0092
N/A
support
Ours w/o Cuboid
51.3
0.0087
95.2
Ours (full)
52.7
0.0085
95.1
Slore ML.P
21.0
0.0130
70.8
openable
Qurs win Cuhoid
w/o Afford
46.7
19.9
0.0104
0.0097
N/A
55.8
Ours (full)
47.6
0.0093
60.4
CONCLUSION
We present PartAfford, new task in visual affordance research that aims at discovering part-level affordances from 3D shapes. We propose a novel learning framework that discovers part-level affordances leveraging only the affordance With comprehensive experiments and analyses, we point out potential directions for incorporating visual appearance facilitate better shape abstraction and combining it with an active learning approach for efficient affordance learning.
supervision and geometric primitive regularization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Christopher P Burgess, Loic Mathhe, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 2019. 3
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer
Vision (ECCV), 2020.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03312, 2015. 20
Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holistic++ scene understanding: Single-viers 3d holistic scene parsing and human pose estimation with human- object interaction and physical commonsense. Conference on Computer Vision and Pattern Recognition (CVPR). 2019. 1
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprini arXiv: /406. 1078, 2014. 4
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2, 3,
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv preprint 1907 13052, 2019. 3
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in object-centric generative models. arXiv preprint arXiv:2007.0622 2020.
Kuan Fang. Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph Lim. Demo2vec: Reasoning object affordances from online videos. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision (ICV), 129 (12):3313-3337, 2021. 9
Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. In International Conference Computer Vision (ICCV), pp.
15752-15761, 2021. 3
Caelan Reed Garrett, Tomás Lozano-Péres and Leslie Pack Kaelbling. Pddistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference Automated Planning and Scheduling, volume 30, pp. 440 448, 2020. 19
Caclan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pére Integrated task and motion planning. Annual review of control, robotics, and autoromous systems, 4:265-295: 2021. 19
10

---

Under review as a conference paper at ICLR 2023
James Jerome Gibson. The ecological approach to visual perception. Houghton, Mifflin and Company, 1979.
James Jerome Gibson and Leonard Carmichael. The senses considered as perceptual systems, volume 2. Houghton Mifflin Boston, 1966.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization.
arXiv preprint arXiv: /700: 03448, 2017. 3
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML), 2019. 3,
Abhinav Gupta, Scott Satkin, Alexei Efros, and Martial Hebert. From 3d scene geometry to human workspace. In Conference Computer Vision and Pattern Recognition (CVPR), 2011. 3
Saurabh Gupta, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Indoor scene understanding with rgb-d images: Bottom-up segmentation, object detection and semantic segmentation. International Journal of Computer Vision (IJCV), 112(2):133-)) 149, 2015. 3
Denis Hadjivelichkov, Sicelukwanda Zwane, Mare Deisenroth, Lourdes Agapito, and Dimitrios Kanoulas. One-shot transfer of affordance regions? affcorrs! arXiv preprint arXiv:2209.077 022009001444 077, 2022. 19
Muzhi Han, Zeyu Zhang, Ziyuan Jiao, Xu Xie, Yixin Zhu, Song-Chun Zhu, and Hangxin Liu. Scene reconstruction with functional objects for robot autonomy. International Journal of Computer Vision (IJCV), pp. 1-22, 2022.
Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael Black. Populating 3d scenes by learning human-scene interaction. In Conference Computer Vision and Pattern Recognition (CVPR), 2021. 20
Mohammed Hassanin, Salman Khan, and Murat Tahtali. Visual affordance and function understand- ing: survey. ACM Computing Surveys (CSUR), 54(3):1 -35, 2021.
Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2018a.
Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. European Conference on Computer Vision (ECCV), 2018b.
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https//gthha..cmdeemmmmdm object-datasets/, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Hedvig Kjellström, Javier Romero, and Danica Kragić. Visual object-action recognition: Inferring object affordances from human demonstration. Computer Vision and Image Understanding, 115 (1):81-90, 3.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955. 5
Michael Land, Neil Mennie, and Jennifer Rusted. The roles of vision and eye movements in the control of activities of daily living. Perception, 28(11):131 13111-133,, 1999.
Wei Liang, Yibiao Zhao, Yixin Zhu, and Song-Chun Zhu. What is where: Inferring containment relations from videos. In International Joint Conference Artificial Intelligence (IJCAI), 2016.
11

---

Under review as a conference paper at ICLR 2023
Francesco Locatello, Dirk Weissenborn Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 3, 5, 7, 18, 21
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019,
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020. 3
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J Mitra, and Leonidas J Guibas. Structurenet: hierarchical graph networks for 3d shape generation. ACM Transactions on Graphies (TOG), 38(6):1111 2019a. 3
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. PartNet: large-scale benchmark for fine- geeeggrrindd and hierarchical part-level 3D object understanding. In Conference Computer Vision and Puttern Recognition (CVPR), 2019b. 2,
6, 20, 24
Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. WhereZact: From pixels to actions for articulated 3d objects. In International Conference on Computer Vision (ICCV), 2021. 3
Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O20-afford: Annotation-free large-scale object-object affordance learning. In Conference on Robot Learning, pp. 1666-1677. PMLR, 2022. 3
Austin Myers, Ching Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In International Conference on Robotics and Automation (ICRA), pp. 1374 1381. IEEE, 2015. 3
Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d environments. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object in- teraction hotspots from video. In International Conference Computer Vision (ICCV), 2019.
Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Envi- ronment affordances from egocentric video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3
Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In International Conference Intelligent Robots and Systems (IROS), 2017.
Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadries revisited: Learning 3d shape parsing beyond cuboids. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Despoina Paschalidou, Luc Van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition of 3d objects from single rgb image. In Conference Computer Vision and Pattern Recognition (CVPR), 2020. 3
Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
Zhongzheng Ren, Ishan Misra, Alexander G Schwing, and Rohit Girdhar. 3d spatial recognition without spatially labeled 3d. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13204-13215 2021.
12

---

Under review as a conference paper at ICLR 2023
Anirban Roy and Sinisa Todorovic. A multi-scale enn for affordance segmentation in rgb images. In European Conference on Computer Vision (ECCV), 2016. 3
Stefano Soatto. Actionable information in vision. In Machine Learning for Computer Vision, pp.
17-48. Springer. 2013. 1
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence (AAAT), 2017. 21
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2 148, 2021.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, al. The replica dataset: digital replica of indoor spaces. arXiv preprint
2019. 15, 20
Yhun-Yu Sun, Qian-Fang Zou, Xin Tong, and Yang Liu. Learning adaptive hierarchical cuboid abstractions of 3d shape collections. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
Mare Toussaint, Kelsey R Allen, Kevin A Smith, and Joshua Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning-extende abtract. In IJCAI, pp. 6231-6235, 2019. 19
Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353 2018. 3
Hanqing Wang, Wei Liang, and Lap-Fai Yu. Transferring objects: Joint inference of container and human pose. In International Conference on Computer Vision (ICCV), 2017a. 3
Xiaolong Wang, Rohit Girdhar, and Abhinav Gupta. Binge watching: Scaling affordance learning from sitcoms. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017b. 3
Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas J Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In European Conference on Computer Vision (ECCV), pp. 90-107. Springer, 2022. 3
Jiacheng Wei, Guosheng Lin, Kim-Hui Yap, Tzu-Yi Hung, and Lihua Xie. Multi-path region mining for weakly supervised 3d semantic segmentation on point clouds. In Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 4384-4393, 2020. 3
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. SAPIEN: A simulated part-based interactive environment. In Conference Computer Vision and Patlern Recognition (CVPR), 2020. 2, 6, 24
Dan Xie, Sinisa Todorovic, and Song- Chun Zhu. Inferring" dark matter" and" dark energy" from videos. In International Conference on Computer Vision (ICCV), 2013. 3
Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Lin Gao. Dsg-net: Disentangled structured mesh net for controllable generation of fine geometry. arXiv preprint Nxv:20008.054400 2020. 3
Kaizhi Yang and Xuejin Chen. Unsupervised learning for cuboid shape abstraction via joint segmen- tation from point clouds. ACM Transactions on Graphics (TOG), 2021. 3,
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905, 2021. 3
Lap-Fai Yu, Noah Duncan, and Sai-Kit Yeung. Fill and transfer. A simple physics- approach for containability reasoning. In International Conference Computer Vision (ICCV), 2015. 3
13

---

Under review as a conference paper at ICLR 2023
Yibiao Zhao and Song- Chun Zhu. Scene parsing by integrating function, geometry and appearance models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
Yixin Zhu, Yibiao Zhao, and Song-Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 3
Yixin Zhu, Chenfanfu Jiang, Yibiao Zhao, Demetri Terzopoulos, and Song- Chun Zhu. Inferring forces and learning human utilities from videos. Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 3
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, (3):330-345,, 2020. 3
14

---

Under review as a conference paper at ICLR 2023
sittblee
framework
backrest
Real-world Mesh
Reconstructed
Siot
Slot 2
Sitt
Slot
Figure 8: Qualitative results on real-world objects (chairs and sofas) from Replica (Straub al., 2019).
support
framowork
Reak-world Meah
Inpul
Recorstructed
Siot
Skx
Figure Qualitative results real- world objects (tables and cabinets) from Replica (Straub al., 2019).
15

---

Under review as a conference paper at ICLR 2023
display
wrapgrasp
pressable
Input
Ground Truth
Reconstructed
Slot
Slot
Slot 3
Figure 10: Qualitative results objects (laptops and displays) "display" subset.
16

---

Under review as a conference paper at ICLR 2023
cutting
handle
Input
Ground Truth
Reconstructed
Slot
Slot
Figure 11: Qualitative results on objects (knives and scissors) in our "cutting" subset.
17

---

Under review as a conference paper at ICLR 2023
contamiicl
10 4R0
Iquidoomrat
Elum
Iquidco R0
heg
hgng
2300 5000 7500 10000 12500 15000 17500 20000
(a) Number of object categories covered by cach affor- dance category.
(b) Number of object instances covered by each affor- dance category.
Figure Statisties of our part affordance dataset.
A EXPERIMENTS ON MORE AFFORDANCES
We conduct further experiments and report per- formance on more affordance categories on the long tail of the dataset distribution. More specifi- cally, we include the subsets "display" and "cut- ting". The "display" set includes part-level af- fordances (display, wrapgrasp, pressable) from 1,441 object instances, most of which are lap- tops and displays. The "cutting" set includes affordances (cutting, handle) from 641 object
Table 3:
Quantitative results the "display" subset and the "cutting" subset.
Model
into (8)
MSE
AP (S)
MLo
17.2
00183
97.2
display
Ours w/o Cahoid
wño Cuboid
47.0
0.0146
0.0400
80.6
Ouns (full)
47.6
0.0135
$2.0
Slat MLP
21.6
0.031
97.7
wlo Afford Cuboid
cutting
Ours wlo Caboid
28.1
0.0027
96.9
Ours (full)
29.8
0.0026
97.7
instances, most of which are knives and scissors on the long tail of the dataset distribution. Quantita- tive results are shown in Tab. 3 and qualitative results shown in Fig. 10 and Fig. 11. Our method still shows a significant advantage over the baseline methods on two core metrics: mIoU for part affordance discovery and MSE for 3D reconstruction.
B ATTENTION VISUALIZATION
Following the settings in Locatello et al. (2020), train the models using T' = attention iterations in the slot attention module. In Fig. 13, we observe that each slot gradually attends to the correct part of the object the number of attention iterations increases. The attention mask t = 3 depicts the silhouette of the reconstructed shape and parts.
C FURTHER ANALYSIS & DISCUSSION
C.I NUMBER OF SLOTS
We set the number of slots the maximal number of affordance labels that appear in subset, which is different from previous object-centric learning algorithm (Locatello et al., 2020), where the number of slots could be arbitrary. This is because when increase the number of slots, increase the ambiguities of the affordance composition and set matching at the same time. It prevents the model from learning accurate correspondence between affordance labels and parts. As shown in Fig. 14, the model learns the "sittable" in a "null" slot.
also
C.2 SELECTION OF STUDIED AFFORDANCES
Although we annotate objects with 24 affordance categories, we benchmark PartAfford only on three subsets with 7 kinds of affordances in this work. The main reason is that we find it much more challenging to discover part-level affordance for some other affordance categories. For example, "rollable" and "cutting" usually connect to tiny object parts that are challenging to be segmented from objects, "illumination" and "display" cannot be distinguished from the objects without deeper understanding of the visual appearance and reflections, "pressable" and "pinchable" require richer interactions to be discovered. summary, it is either especially challenging to segment or requires more than geometric information (e.g., active interactions) to discover the other affordance categories.
18

---

Under review as a conference paper at ICLR 2023
sittable
framework
backrest
armrest
Input
GT
Recon.
Attn. t=1
Attn. t=2
Attn. t=3
Figure 13: Example affordance discovery model trained with T = 3 attention iterations. Attention is visualized in various colors and point radius. The point is colored according to the predicted affordance label of the slot which attends the most the point. Point radius positively depends the maximal attention value across the slots. We use trilinear upsampling to rescale the attention mask to the input resolution (32 x 32 x 32).
null
framework
backrest
sittable
Input
GT
Recon.
Slot
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Figure 14: Model performance when the number slots increases. The model learns the "sittable" part in "null" slot.
We believe that future solutions to our task could involve those interaction cues take the best of both worlds, further improving the performance and extending the scope. Our solution could also provide an intuitive prior for active learning methods, which could boost learning efficiency compared to learning from scratch. We hope to further explore the learning of these affordances in the future.
C.3 AMBIGUITIES IN AFFORDANCE LEARNING
Affordance is naturally ambiguous since the functions of object parts are rich. This work provides well-defined benchmark to study how to learn affordance from accurate affordance definition and sparse set supervision. In another work, (Deng et al., 2021) models the multiple affordances with a mixture of distributions. Our dataset has supported learning multiple affordances per part; we annotate a single part with multiple affordances (see details in Sec. 5). Yet, the first step towards affordance discovery, our current method focuses on learning the most important affordance. We believe our current learning framework could also be extended to learn multiple affordances by switching the one-hot affordance label to multi-hot labels. We also suggest further fine-tuning the multi-hot affordance prediction based on the one-hot version, since the affordance prediction of single-hot labels appears significant guide the slot competition from ablation study.
C.4 POTENTIAL IMPACTS FOR VISION & ROBOTICS COMMUNITY
By learning to discover the lert-level affordance, our model could facilitate the understanding of human-object interaction and object manipulation. As shown in Fig. 15, the learned affordance could be applied to synthesize potential human actions and interactions with various 3D objects. Additionally, our work could provide priors for robotic cross-category object manipulation and task planning (Toussaint et al., 2019; Garrett et al., 2020: 2021). For example, Hadjivelichkov al. (2022) demonstrates single reference image of an object with annotated affordance regions can help the robot to use the affordance skill in the real-world setting, where our method can provide generalizable affordance information the 3D shapes without the need for manual annotation.
19

---

Under review as a conference paper at ICLR 2023
sittable
support
frame.
contain.
liquid.
openable 58.62
display 87.26
cutting
Orig- Scaled
72.44
82.92
74.74
54.01
20.34
83.29
74.95
83.68
77.87
75.01
69.42
66.26
90.03
79.67
backrest 80.51
armrest 73.19 73.63
press.
handle
illum.
wrapgrasp
lyable
headrest
Orig.
78.70
38.09
24.57
63.07
46.24
34.73
Scaled
81.36
83.57
49.31
24.59
71.48
45.32
32.76
rollable
pourable
twist.
lever
pinch.
audible
Orig. 53.33
68.15
40.24
60.34
62.07
56.67
Scaled
58.43
73.89
53.97
64.24
66.84
46.11
Table 4: Supervised affordance segmentation results (category mloU %). "Orig." refers to the original PartNet dataset with 3D shapes scaled to unit bounding sphere. "Scaled" refers to rescaling 3D shapes to real-world dimensions. "avg" refers shape average Intereection-over-Umnioo (IoU).
Figure 15: 3D Human synthesis conditioned on inferred part affordance using (Hassan al., 2021).
D SUPERVISED AFFORDANCE ESTIMATION
Apart from the PartAfford task, also benchmark the state-of-the-art 3D auto-encoder for the super- vised affordance segmentation task in cross-category fashion on our proposed dataset. Evaluation reported on the Intersection-over-nniioo (IoU) metric following Mo al. (2019b), as shown in Table 4.
Note that the baseline is trained and tested on object and affordance categories. In contrast, semantic segmentation in PartNet (Chang et al., 2015; Mo et al., 2019b) is trained on each object category separately. Results show that the algorithm could still achieve high performance, which demonstrates that our affordance annotation is reasonable and consistent.
PartNet normalizes every object shape into a unit bounding sphere. This not realistic since objects in different categories may have very different dimensions. For example, a mug is much smaller than a bed. normalized in the same way, cross-category training performance may be hurt to some extent. Thus, calculate the average real-world 3D dimensions of each object category from metadata of ShapeNet (Chang et al., 2015) and scale the point cloud in our part affordance dataset according to its object category.
E MODEL GENERALIZATION TO UNSEEN OBJECTS
To further demonstrate the generalizability of our methods qualitatively, additionally provide examples for real scanned and more diverse objects from the Replica (Straub al., 2019) dataset, results shown in Fig. 8 and Fig. 9. Although the scan quality is not perfect, our learned model generalizes well and can reconstruct and identify the functional parts given novel objects.
F DETAILS ON EXPERIMENTS
Below, we describe the statistics of objects and the related affordances for these subsets; we discuss detailed reasons about why choose these three subsets in Appendix C.2.
Benchmarks
"Sittable": We collect all object instances that have affordance "sittable"; most of them are chairs and sofas. Their part-level affordances belong to the set {sittable, backrest, armrest, framework). We split the training, validation, and test set in the ratio 7: 1 In total, we have 5, 093 instances for training and 1, 457 for test.
20

---

Under review as a conference paper at ICLR 2023
We collect objects with affordance "support", mainly from categories table and cabinet. Their affordances belong to (support, framework]. There are 7, 974 instances for training and
"Support":
2, 279 instances for test.
This subset contains objects from frige, dishwasher, washing machine, and microwave. Their affordances belong {openable, framework, handle). There are 807 instances for training and 232 instances for test.
"Openable":
Data Augmentation To enrich affordance compositions, we augment the training data by randomly removing certain object parts with corresponding affordance labels.
G MORE MODULE ABLATION
G.1 SOFT K-MEANS
Slot attention, as generalized soft k-means algorithm, could be reduced to the soft k-means algorithm according to Locatello et al. (2020). It turns out that the reduced model can achieve 23.0% Mean IoU on "sittable" objects, which is slightly better than the Slot MLP baseline but significantly worse than our full model using slot attention (57.3%). reconstructs the whole shape with a quality (MSE: 0.0096) par with the full model (MSE: 0.0097). Its affordance set prediction accuracy is the lowest among all models (AP: 0.87). Overall, the soft k-means algorithm cannot effectively segment the shape or discover affordance parts.
H ADDITIONAL EXPERIMENTAL RESULTS
We show additional qualitative results for our full model's affordance discovery results on "sittable" (Fig. 16), "support" (Fig. 17), "openable" (Fig. 18) objects, respectively.
I PART AFFORDANCE DATASET
Here we report more statistics and details on how we define and annotate the affordance labels in our dataset. As shown in Fig. 12, most of the affordance categories annotated on more than kinds of objects. Our curated subsets cover the most common affordance categories (top 7), as well as some rare affordance categories (e.g., "'uutiig")) on the long tail.
1.1 PRINCIPLES AFFORDANCE ANNOTATION
To keep annotations consistent across object categories, we design guideline for affordance annota- tion. Below are some general principles to annotate a leaf part of an object instance with affordances.
Multiple affordances. A part can afford multiple kinds of human actions. For example, the seat of chair could afford sittable for resting human body or support if one wants to place some books on it. We refer to ConceptNet (Speer et al., 2017), a giant knowledge database, for annotating common usage of object parts.
Prioritized fine- fne-grained affordances. When there are multiple affordances labels for part, we give the more fine-grad treiggr affordance label higher priority. For the chair seat in the example above, sittable is prioritized compared with support it is fine-grained support affordance for body resting.
Articalation-relatee affordances. The PartNet dataset does not contain articulation information, which makes affordances such as openable not geometrically distinguishable. Thus, we also generate set of shapes with openable affordance from the PartNet-Mooillty dataset by capturing 3D shapes with various opening angles. More geometric variation helps models to learn articulation-relate affordances.
I.2 AFFORDANCE DESCRIPTIONS
Our description for each affordance contains brief definition, some supplemental clarification and priority statements if needed, and some example leaf nodes in the part hierarchy of various reasonable objects (full path from root leaf).
21

---

Under review as a conference paper at ICLR 2023
Indicates whether the object can be used for sitting. Anything sittable of course affords support, and the requirement for a supporting object to be sittable is that it must be both comfortable and safe for human seating. For example, table is not sittable despite affording support because it is not comfortable. Sittable is given priority over potential co-existing affordances like support.
sittable:
E.g., chhiirchaiiic hetrrr/hhhaissettseaate peat//eeatssuffie
support: A trait of objects which can safely keep other objects on top of themselves. Common characteristics of support-affording objects are that they flat and can support multiple objects at once. A key distinction between support-affording objects and non-supporting objects is that support-affording objects will remain stable when other objects are placed table is a supporting object because it is just as stable with objects on it as it is without, However, a stack of plates is not supporting because they become more unstable you add more plates.
them. For example,
E.g. hoopllrreglar_age Baclilar_bbbl/taeee _able/tableetop. tabletop, storage_ cormiture/cabineer /selff bed/bed http//ddduittbbbbe unit/bed sleep_ area.
openable: Parts which may be moved with hinge-like mechanism on an articulated object. Openable objects do not need to afford handles, but usually handles can be found attached to the openable part. Openable parts are distinct from other moving parts that they need to swing to some degree to be moved. Openable is given priority over potential co-existing affordances like containment.
g.g. door/door_booe saurface_boarte dish_washer/ / microwave/boode /boddy dooo.
backrest: Objects which are reasonably designed for providing support to a person's 's back. By reasonably designed, mean either specifically (as in the back of chair) can afford back support a person wanted to sit upright (like a headboard).
E.g., chair/ back baackbbac_suppooss support.
armrest: Objects which are specifically designed to support an arm. For example, table can support variety of things and is thus not armrest. A chair's arm is the perfect size for a human arm, so it must be an armrest.
E.g., chair/chair /chair arm.
handle: An object extension which affords the ability to open an attached 'openable' part. Handles mostly grabbed with hand-wray wrapping. so it's important to only afford 'handle' to parts which are specifically involved in an opening mechanism, like a door handle.
E.g.,
storage_furniture furniture/cale /cabinet//caaiiiieie /ttp//caabntoooooeeeee handle, iable/regular table/table /table base/drawer__brr
bobinet_door/hanee
mug/ hhandl,
dish washer/ bote///o/////// /ooor hhandle. bag/bay handle.
framework: Any object segment which either: a) helps to define the shape of the object as a whole or b) is an unaffording extension of the object or connector for other segments. For example, hinge affords framework because is integral to connects the door the door frame. Overall, this affordance is the most general of all, so it should only be used when it clearly applies to either case of definition. For example, most handles do not afford framework because is both a small part of object's shape and already affords handle. It has the least priority among potential co-existing affordances.
E.g., chair/chair base.
containment: An affordance of object which can store physical items. The size of the physical items does not matter, so long they are not too small. For example, anything that can only contain objects smaller than say a marble do not afford containment. Also, items that afford containment must afford security to the items they contain, such that they will not fall out.
22

---

Under review as a conference paper at ICLR 2023
E.g., table /regulr _ttall/tabbbeeeeeeee base /drawer base bass//drawy drawer, storage furniture/ /ct//nettb/eeceeee drawer, mug/ /ong/continer,
trash_can/ /cnnctnnine refrigerator /body,
bowl/container, bag/ bag/bag.u her//baabbooy bottle/ bottllooott/bbbbeeeeeeeeeeeeee
liquidcontainment: A more specific version of the containment affordance. Objects that afford liquid-containn containmen must be able safely contain liquid. Examples of these are bottles, bath tubs, etc.
E.g.. muod/
bottle /normall caatmmallbott///eee bottle/body.
display: Something which visualizes information for a useful purpose. Examples of these would be monitor screens or a clock surface.
E.g., display/dise /dii//diissspeeeee screen/screen, screen.
laptop/ screen /scree
side/ side/screen, screen, block/table, saclle_llock// block_body..
body/surface.. /surface.
cutting: The quality of being able to slice through other objects. Certain things that can cut are not considered to have cutting affordance if it was used against its intended purpose, like smashing a glass vase. Cutting is only afforded to objects which are specifically designed for cutting, like blade-edge.
E.g., cutting irrtriinniisttumet/eeee scissors/ https///bbbdd_aaaeeeeeeeeee /laade
side,
pressable: A mechanical feature of objects which either have buttons or can interact with a finger. Good examples of these are keyboard keys.
E.g. keyboard/kes
hanging: A part which can be hung on another object. These parts almost always only serve the purpose of hanging the rest of the entire object. An example of this would be shoulder strap for handbag.
E.g. hag/shoulder_stren
The 'wrap-grasp' trait is afforded by parts which are explicitly meant to be grabbed in hrad-igg motion. Just because a hand can wrap around an object part does not mean it affords suap-gras.. It must be useful to grip the part in this way. An example of this would be ladder rung, which a person is meant to wrap their hand around to climb the ladder.
wrapgrasp:
E.g., cup,
bed bed/ladder/rrug.
illumination: The affordance of light emission. This only applies to object parts which meant to light up a broad area. For example, a monitor screen does not afford illumination despite emitting light because it is not supposed to be used light up the area around it.
E.g., tampetable_aae ooroor__oooo f__fflor____a srlffoorolamp///// lamp/lamp_um /amp_unit. haamp_hed/llee
bulb.
lyable: Indicates that a human can comfortably rest his / /he entire body on the object. These objects usually flat with soft surface. Lyable is given priority over potential co-existing affordances such as sittable and support.
E.g., bed/bed leeed_unt//bbr htap:/dddd_seepppeeeee sree_aree//ie mattress.
headrest: An extension of an object which oriented so a human head can rest comfortably on Examples of these are chair headrests or bedframe headrests.
E.g., bed/ bed/bed. _od_uunttbeed___e bed/bed codd__nit/bbed
23

---

Under review as a conference paper at ICLR 2023
step: A part which affords the human foot climbing resting functionality. For example, ladder rung is a step because it affords climbing with both hands and feet. foot pedestal also step because it can be stood feet can be rested on it.
E.g., bed/ hoddallder/uuma /rung.
pourable: Meant for parts which liquid flow out of. Things that pourable may also be dependent on a mechanism for controlling flow, like a bottle cap or a knob.
E.g., bottle normal bottle/mouth, bottle/jug/bot
twistable: These objects either be detached provide special functionality by twisting them in a clockwise or counterclockwise motion. Examples include bottle caps and knobs.
E.g., htttorrmabbottl///oeeeee bottle pottle/normar normal_bottle/ bottle/moutte /mouth.
rollable: A part which can roll move around. Exceptions to this affordance are objects which roll but stay fixed in place, like a rocking chair.
E.g., wheel.
lever: Any handle which can rotate up to a point. For example, knobs rotate but are not levers because they do not provide handles. Levers must be treated differently from twistable objects or handles because if they are twisted too much they will break.
E.g., lever.
pinchable: An object which is small enough such that it can be manipulated by pinching with two more fingers. Things that are pinchable must not be heavy, and they usually fit inside the palm of hand.
E.g., earbud.
audible: Anything which emits sound. This does not include sound emitted indirectly, such as door creaking when opened, which makes sound side-effect.
E.g., beadphonee/padinng
1.3 LICENSE
Our dataset is annotated based on PartNet (v0) (Mo al., 2019b) and PartNet- Mobillity (v2.0) (Xiang et al., 2020), both of which are licensed under the terms of the MIT License.
J CODE AND DATA
Code, data, and instructions needed to reproduce the main experimental results are available in the supplementary materials.
24

---

Under review as a conference paper at ICLR 2023
sittable
framework
backrest
armrest
Input
GT
Recon.
Slot
Slot
Slot
Slot
Figure 16: Additional qualitative results objects in our "sittable" subset.
25

---

Under review as a conference paper at ICLR 2023
support
framework
Input
GT
Recon.
Slot
Slot 2
Figure 17: Additional qualitative results on objects our "support" subset.
26

---

Under review as a conference paper at ICLR 2023
openable
framework
handle
Input
GT
Recon.
Slot 1
Slot 2
Slot 3
Figure 18: Additional qualitative results on objects in our "openable" subset.
27


Reviews:
[{"review_id": "bDuH4bngdPt", "reviewer": "Reviewer_iNCK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work focuses on learning object affordance. The authors present a new task called part-level affordance discovery (PartAfford). Given only the affordance labels for each object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part of the object corresponds to a certain affordance category. The authors propose a slot-attention-based framework with 3D part reconstruction, affordance prediction, and cuboidal primitive regularization. They construct a part affordance dataset and benchmark their method with several baselines they designed. ", "summary_of_the_review": "Overall, the reviewer agrees that the proposed task is challenging. However, it is questionable to motivate the task from object affordance. In addition, the experimental section is not convincing. The reviewer looks forward to the author's feedback on the questions. ", "strength_and_weaknesses": "Strength\n\n1. The paper studies object affordance, a critical problem with various applications such as scene understanding, object understanding, and object manipulation. Toward the goal, a part-level affordance discovery task is proposed. Compared to the existing densely-labeled paradigm, the proposed task offers a new challenge to the community that studies affordance detection, weakly-supervised segmentation, and part segmentation. Note that the authors propose a part affordance dataset with 24 affordance categories shared among > 25, 000 objects.\n2. Figure 5 demonstrates the effectiveness of the proposed components. In particular, the authors successfully demonstrate that the proposed task cannot be tackled simply with the slot-attention algorithm (Locatello et al., 2020). In addition, the effectiveness of the cuboidal primitive regularization branch can be observed in Figure 5 (c).\n\nWeaknesses\n\n1. Task formulation: The authors motivate the readers that the work aims to answer a question: what is the natural way to learn object affordance from geometry with humanlike sparse supervision? Then, the authors answer the question from a labeling perspective, i.e., use part-level affordance instead a densely-labeled strategy. It makes sense to reduce the annotation efforts by proposing the task. However, the reviewer is not convinced that the task tries to answer the question directly. Specifically, other affordance-related works (e.g., Kjellström et al., CVIU 2011 and Nagarajan et al., ICCV 2019, Wang et al., ECCV 2022) that study affordance from an interaction perspective. With a detailed discussion with them, the reviewer has a hard time appreciating the value of the work.\n    1. Kjellström et al., Visual object-action recognition: Inferring object affordances from human demonstration, CVIU 115 (2011) 81-90\n    2. Nagarajan et al., Grounded Human-Object Interaction Hotspots from Video, ICCV 2019\n    3. Wang et al., “AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions,” ECCV 2022\n2. Part affordance dataset: the authors did not recognize the existing dataset (i.e., Myers et al., 2015). Note that they also consider affordance detection in cluttered environments. The authors should discuss the difference and why the dataset Myers et al., 2015 built cannot serve the need for evaluation. It is worth noting that the number of affordance categories in the proposed dataset is larger than Myers et al., 2015. However, the authors only conduct experiments on seven affordances. The experiments weaken the advantage of the proposed dataset.\n    1. Myers et al., \"Affordance Detection of Tool Parts from Geometric Features,\" ICRA 2015\n3. The proposed algorithm is highly relevant to 3D weakly supervised semantic segmentation with scene-level labels (Wei et al., CVPR 2020 and Ren et al., CVPR 2021). They could serve as the baseline for the proposed task. The authors should justify why these methods are not considered in the experiments. Please comment.\n    1. Wei et al., Multi-Path Region Mining For Weakly Supervised 3D Semantic Segmentation on Point Clouds, CVPR 2020\n    2. Ren et al., 3D Spatial Recognition without Spatially Labeled 3D, CVPR 2021\n4. Limited quantitative evaluation: The part affordance dataset has various objects and affordance categories. However, the authors only report a subset (i.e., sittable, support, and openable) in Tables 1 and 2. In addition, the paper also reports similar qualitative analysis on similar objects. Currently, the experimental results are insufficient to justify the proposed method's effectiveness in a wide range of tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ER7yohrh4D", "reviewer": "Reviewer_Xneh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of discovering part affordances. This is an important topic in many applications. Previous methods focused on learning object affordances with dense supervision. In this work, the paper proposes a new task of part affordance discovery. This task is interesting and challenging The paper constructs a part-level, cross-category 3D object affordance dataset with 24 affordance categories shared among 25,000 objects.", "summary_of_the_review": "Please see the comments in the two boxes above.", "strength_and_weaknesses": "Strengths\n1. The proposed task is interesting yet challenging and has value for many research communities.\n2. Code is provided.\n3. The video is very clear.\n\nWeaknesses\n1. Given that this paper is really interesting, it is very important to have a section that talks about limitations and future work in other problem domains, e.g., robotics.\n2. I would also suggest that in camera ready it would be valuable to create a website that has an interactive viewer for people to view the dataset. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BeCAEfA0Ep-", "reviewer": "Reviewer_X6Ff", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the new task of part-level affordance discovery: it is a joint task of decomposing 3D shapes into their parts and predicting how each part corresponds to affordances. A learning framework has been proposed for this task. It learns to segment 3D shapes into parts from weak shape-level labels. It also associates affordances with predicted parts. The framework is powered by 3D position-embedded features and 3D slot attention. To facilitate this new task, a novel dataset that features part-level cross-category 3D object affordances is constructed. Extensive evaluation and ablation studies have been performed on the dataset.", "summary_of_the_review": "Overall, I like this paper as it defines an interesting yet challenging task, i.e., jointly finding 3D parts and their affordance, and provides a viable solution. The new 3D part affordance dataset is also a nice contribution; it will promote more interesting research in this direction. There are a few places the paper could be improved as mentioned in the \"Weakness\" above. But I think the paper will be a good contribution to ICLR 2023 in its current form. ", "strength_and_weaknesses": "### Strength\n- The new task, PartAfford, is interesting and I believe will promote new research in this direction.\n- The new dataset is also a valuable contribution to the community. I especially like the openable affordance shapes (Figure 3c).\n- The proposed method to discover 3D parts and associate affordance in a weakly supervised setting is interesting. \n- I like Section 6.4 which evaluates the generalization power of the proposed method on unseen objects.\n\n\n### Weakness\n- Although the 3D part affordance dataset contains 24 affordance labels, the experiment section only considers three: 'sittable', 'support', and 'openable'. I understand that they are very useful functions for everyday objects but I wish the method is evaluated on more categories for a comprehensive understanding of the task.\n\n- I know the paper proposes a new task. But I think it could compare its individual components to other methods mentioned in the related work. For example, (unsupervised) 3D part discovery work could be compared. Also, even a fully supervised dense affordance learning scheme can be compared as a reference too because it is common to have such a method in the evaluation as the performance upper bound reference.\n\n- Section 6.1 mentions that the current setting only allows one affordance per part. I think quite frequently an object part can afford multiple functions. It is not clear to me if the method is able to handle multiple affordance per part. \n\n- Comparing to Table 1, Table 2 doesn't have results of Slot MLP and IODINE. A complete table will be better.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "81OFzmIgep", "reviewer": "Reviewer_2e64", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a large-scale 3D object affordance learning and part discovery dataset, paired with a baseline method. The proposed task is important for 3D object understanding and would be impactful in many fields. The dataset consists of more than 25,000 objects and each object has a set of affordance labels. The objects mainly come from PartNet and PartNet-Mobility. The task requires the machine to discover the parts of these human-made objects and classify their affordance under a weakly-supervised learning setting. The proposed method uses slot attention as the main component with a multi-task head design to address this task. In experiments, this method achieves decent performance compared with some basic methods.", "summary_of_the_review": "Overall, this is an interesting paper with a lot of data, sound methods and experiment designs, and good writing. Though there is some room to improve (listed above), I think it is OK to be accepted.", "strength_and_weaknesses": "Pros:\n+ The aim and motivation of the benchmarks are sound and well-defined to make the community move on.\n\n+ The dataset is large-scale and useful as a good complementary to PartNet.\n\n+ The proposed method follows a sound design and uses slot attention as the feature bottleneck to finish three sub-tasks. And its performance looks well.\n\n+ Code and data licenses are available.\n\nCons:\n- The classes of affordance are limited. Even the dataset is large with a lot of objects. Furthermore, in tests, only several affordances are tested.\n\n- Though there are analyses about the mutual effect of the three tasks, deeper insight is lacking to guide future study. I suggest a more profound discussion about the possible design and methods for future study, as this is a benchmark paper.\n\n- Lacking a longtail distribution data analysis, e.g., affordances, object classes, parts, effect on performance, etc.\n\n- More visualizations according to the above analysis would make this paper more solid.\n\n- \"We only keep the most prioritized affordance for each part to ease the ambiguities in learning\", is somehow oversimplified.\n\n- Possibilities scaling to some real scanned objects and more diverse objects?\n\n- typo: in robotics (Nagarajan & Grauman, 2020; Mo et al., 2022).Prior --> ). Prior", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject