Assembly Fuzzy Representation on Hypergraph for Open-Set 3D Object Retrieval
Authors: Yang Xu
Yifan Feng
Jun Zhang
Jun-Hai Yong
Yue Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Hypergraph
3D Object Retrieval
Open-Set Learning
3D Part Assembly
Fuzzy Representation
Primary area: machine_vision

Abstract: The lack of object-level labels presents a significant challenge for 3D object retrieval in the open-set environment. However, part-level shapes of objects often share commonalities across categories but remain underexploited in existing retrieval methods. In this paper, we introduce the Hypergraph-Based Assembly Fuzzy Representation (HARF) framework, which navigates the intricacies of open-set 3D object retrieval through a bottom-up lens of Part Assembly. To tackle the challenge of assembly isomorphism and unification, we propose the Hypergraph Isomorphism Convolution (HIConv) for smoothing and adopt the Isomorphic Assembly Embedding (IAE) module to generate assembly embeddings with geometric-semantic consistency. To address the challenge of open-set category generalization, our method employs high-order correlations and fuzzy representation to mitigate distribution skew through the Structure Fuzzy Reconstruction (SFR) module, by constructing a leveraged hypergraph based on local certainty and global uncertainty correlations. We construct three open-set retrieval datasets for 3D objects with part-level annotations: OP-SHNP, OP-INTRA, and OP-COSEG. Extensive experiments and ablation studies on these three benchmarks show our method outperforms current state-of-the-art methods.

Assembly Fuzzy Representation on Hypergraph for Open-Set 3D Object Retrieval
Yang Xu', Yifan Feng', Jun Zhang", Jun-Hai Yong', and Yue Gao BNRist, THUIBCS, KLISS, BLBCI, School of Software, Tsinghua University, China
TTencett AI Lab
(xuyang9610, evanfeng97) com, june jzannnttennentttccmmmmm
(anngliiggooueetttinnnee edu. cn
Abstract
The lack of object-level labels presents significant challenge for 3D object re- trieval the open-set environment. However, part-level shapes of objects often share commonalities across categories but remain underexploited in existing re- trieval methods. this paper, we introduce the Hypergraph-Bane Based Assembly Fuzzy Representation (HAFR) framework, which navigates the intricacies of open-set 3D object retrieval through a bottom-up lens Part Assembly. To tackle the challenge of assembly isomorphism and unification, propose the Hypergraph Isomorphism Convolution (HIConv) for smoothing and adopt the Isomorphic Assembly Embed- ding (IAE) module generate assembly embeddings with coometric-semantie consistency. To address the challenge of open-set category generalization, our method employs high-order correlations and fuzzy representation to mitigate dis- tribution skew through the Structure Fuzzy Reconstruction (SFR) module, by constructing leveraged hypergraph based on local certainty and global uncertainty correlations. We construct three open-set retrieval datasets for 3D objects with part- level annotations: OP-SHNP, OP-INTRA, and OP-COSEG. Extensive experiments and ablation studies on these three benchmarks show our method outperforms current state-of-the-art methods.
Introduction
With the growing accessibility of 3D data, 3D object retrieval (3DOR) has emerged as an important area of interest in computer vision [19 [2] The key objective of 3DOR is to establish connections between query and target samples via model training. Despite significant progress in recent years enhancing the development of 3DOR, most existing methods are still based on the closed-set as- sumption, where all categories encountered in the testing phase have been seen during training [6]. However, training sets can not cover all potential categories in real-world applications [32] which hinders accurate retrieval for unseen categories. Although object-level categories are difficult to cover, the part-level shapes of objects across object-level categories often share commonalities [17] among objects, which may provide sufficient semantic information of the object (29 [3] However, this part-level method still remains underexploited on assembly-based representation for open-set retrieval.
3D parts, as essential components in shape representation and analysis, have the capability to represent potential information for both semantic 30 and geometric (25] level tasks. Recently, there has been an increasing application of assembly-based methods in the realm of 3D vision 24 37.. From a geometric perspective, current methods for 3D part assembly impose strict constraints on the
Coorrspooniing author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Query Set
Parts
Intra-Object
Inter- Inter-Category
Target Set
Assembly
Seen
Unseen
Isomorphic Assembly Embedding
Structured
Unseen Categories
Fuzzy Reconstruction
Unseen Categories
Figure I: Illustration of the assembly-based open-set 3DOR task and proposed HAFR framework. Given 3D objects of unseen categories, our method takes several part features and generates the as- sembly embedding isomorphically for each object. Then fuzzy embeddings are generated via leverage propagation and fuzzy reconstruction for open-set retrieval with unseen categories generalization.
categorics or quantities of parts 45 presenting a substantial challenge for implementation in an open-set environment. These methods often overlook the isomorphism and correlations between parts of the same object [20] during semantic embedding. This oversight diminishes the generalization capabilities of the representation model and exacerbates the distribution skew of unseen categories during open-set learning [7 [27] In this paper, we explore the part-assembly representation method from both semantic and geometric perspectives to mitigate the distribution skew of unseen categories, aiming to enhance the generalization performance for open-set 3D object retrieval.
Distinct from existing 3D shape assembly open-set learning methods, the assembly- based open-set retrieval task emphasizes global-level semantics in object representation and requires enhanced generalization of geometric assembly. This leads several challenges for assembly-bascl open-set retrieval, including: First, the difficulty of achieving assembly isomorphism for part features. While part features effectively capture the categorical information of objects, direct fusion for multiple parts may lead to inconsistencies arising from geometric factors, including the order of input and the presence of repeated parts. Consequently, there strong motivation to achieve assembly isomorphism for part features with geometric-semantic consistency. Second, the difficulty in achieving assembly unification across different parts entails mapping and integrating part embeddings from the local-part space to the global-object space. Third, the difficulty in open- category generalization against distribution skew, which requires spatial propagation and generalization for object embeddings from the seen certainty to unseen uncertainty space.
Addressing the aforementioned challenges, explore a method for open-set retrieval tasks through a bottom-up lens Part Assembly. As shown in Figure [ we introduce the Hypergraph-bert based As- sembly Fuzzy Representation (HAFR) framework for assembly-based open-set set 3DOR. On one hand, to tackle the challenge of assembly isomorphism and unification, we first propose the Hypergraph Isomorphism Convolution (HIConv) layer for feature smoothing, and then adopt the Isomorphic Assembly Embedding (IAE) module for embedding integration with geometric-semantic consistency. On the other hand, overcome the difficulty in category generalization, construct leverage hypergraph based on the local-certainty and potential leveraged correlations between seen and unseen categories for propagation. Besides, we adopt the Structure Fuzzy Reconstruction (SFR) module to exploit the fuzzy representation approach for seen-set category generalization. Our contributions are summarized as follows:
correlations. This structure captures
We explore a method to navigate the intricacies of open-set 3D object retrieval through bottom-up lens of Part Assembly, and we construct three 3D point cloud datasets with multiple part annotations for benchmarking.
We propose the HAFR framework for assembly-based open-set 3D object retrieval tasks, including the Isomorphic Assembly Embedding (IAE) and the Structured Fuzzy Reconstruc- tion (SFR) modules, which are designed to generate assembly embeddings with geometric- semantic consistency and overcome the distribution skew of unseen categories.

---

We propose the Hypergraph Isomorphism Convolution (HIConv) and a leverage hypergraph structure to capture the high-order correlations within and among objects, utilizing them for assembly isomorphism and open-set category generalization.
Extensive experiments are conducted on the three benchmarks for evaluation. demonstrating the superiority of HAFR over current state-of of-tte-art 3D object retrieval methods.
Related Work
2.1 3D Object Retrieval
Most current 3D object retrieval methods operate under the closed-set assumption, meaning that the training and testing sets exhibit the same distribution of categories. These methods are divided into single-modal and mudti-modal types. Single-modal 3D object retrieval focuses identifying similar objects within a single modality of 3D data. For example, [31] and 35 use view-based graph model to generate aggregated embeddings from multi-view data. 13 introduces a triplet-center loss to cluster objects of the same category and separate those different categories. HGNN employs hypergraph-base -baseed structure to capture high-order correlations among objects for improved embeddings. Multi-modal retrieval methods [26 22
use weighted fusion or feature fusion networks to aggregate embeddings from different modality-speciif features. Additionally, CMCL [15) proposes a cross-moda center loss to minimize differences across various 3D modalities using common center embeddings.
2.2
Open-Set Learning
Open-set learning focuses on conducting machine learning research in scenarios where key factors
variable 47 32
introduces the Semantic Shift Benchmark (SSB) for open-set recognition.
46
proposes "none-of-above" classifier to determine if a sample belongs to known categories. (5]
presents an adversarial method minimize the overlap between known and unknown distributions. Additionally, several open-set recognition methods for 3D object learning have been proposed [3]
16 I 48 However, retrieval tasks in open-set scenarios are more practical than recognition due
to the fundamentals of representation. Only few methods [8] 23 40 39 38) tackle the open-set 3DOR task, focusing on structure learning networks while neglecting the intricacies of the open-set environment.
2.3 3D Shape Assembly
Much research in computer graphics has concentrated on assembly representation for the reconstruc- tion [77, analysis [34], and generation [17 of 3D shapes. As for the assembly-related method for retrieval, most existing methods primarily focus on establishing relationships between parts one object and parts of other objects
(1O), then utilize these connections for shape analysis [2]] or part retrieval [4]. Although these methods have achieved satisfactory results, they rarely concentrate on the higher-order semantic connections from parts to objects, which makes them challenging apply in 3D object retrieval. Additionally, their geometric constraints also limit their application open-set environments.
3 Problem Setup
3.1 Open-Set 3D Object Retrieval
Given 3D objects from the query set D,, the 3D object retrieval (3DOR) task is to find similar samples from the target set D, The core approach for the 3DOR task is to learn the relationship between query samples and target samples from the training set Dern- Each 3D object can be denoted as (st, v.), the y/ y {c, ) is the category label associated with the 3D object sample 8;.
In the open-set environment 3DOR, all categories of samples the testing set have not been learned in the training set, each retrieval sample is from unseen categories for the model, termed as Open-Set Set Retrieval. Specifically, the open-set settings means that the testing set Dies = (D,. D.} and the training set D are drawn from the different distributions. For the testing set Dies

---

Isomorphic Assemby Embedding (IAE)
Structured Fuzzy Reconstruction (SFR)
Isomerphism Hlyeeggrph
Integration
Leverage Propagation
Local
Giobal
HONNNCom
Assemblly Embeddings
Certainty
Unertainty
Leverage
Hypergraph
Infer-Object
Uiaseen
Propagation
Memory Bank
Projection
yca
Fuzzy Emheddingg
Parts
Rosic Port Featines
Reconstruction
Isomorphism Smoothing
Assembly Embedding
Fuzzy Reconstruction
Similar Objects
Figure 2: An overview of the Tlypergraph-Basee Assembly Fuzzy Representation (HAFR) framework for assembly-based open-set 3D object retrieval. Our framework composed of the Assembled Isomorphism Embedding (IAE) and Structured Fuzzy Reconstruction (SFR) modules, which are designed for -cometric-semantic
semantic consistent integration and fuzzy-aware generalization, respectively.
(ss 10))=111 (D,, D.) and the training set Dern (()) the category space of them are not the same indicating 0: E ý {cz))= y1 E = {c,))= and Y ny = Ø.
3.2 Assembly Representation for Retrieval
Typically, each 3D object can be decomposed into multiple parts based on its shape and semantic information, and it can be regarded model assembled from these parts, i.e., the doors, roofs, hoods, and wheels cars, the seats, backs, arms, and legs of the chairs. Face the emergence of the open-set environment with incomplete missing class labels, the assembly representation based on multiple local parts may provide enough semantic information than global object features for retrieval. We termed this presentation for open-set 3DOR Assembly Representation:
s; E S (p
(I)
where S denote the set of 3D objects, = [p])-- denotes 3D object represented by P semantic parts, NN denotes the number of samples in S.
Consequently, the assembly- -based open-set 3DOR aims to design a method to retrieve similar samples of query in the testing set Dies {((r))/= 9.)) (D,, D.}. based on the data and knowledge
in the training set Denn {(1r])=- (1),,,))) W The assembly-based open-set 3DOR by multiple
parts aims to minimize the expected risk:
f ar argmin D, FEH
X10 -
+X - -0
where D; ((p])p= ni, Di) and Dj the query set D, and target set Dr. X(-) denotes the indicator function, which evaluates to when the specified condition holds true and 0 otherwise. The embedding function f := {p] - U maps multiple parts {p of an object to an assembly representation vector E IRd facilitating similarity-based retrieval. denotes the hypothesis space of the embedding function. The L2 norm function ||2 is used as a distance metric to measure the Euclidean distance between two vectors.
ny Új) are the object samples selected from
Methodology
4.1 Overall Framework
As shown Figure the proposed Hypergraph-Banr Based Assembly Fuzzy Representation (HAFR) framework is composed of two modules: Isomorphic Assembly Embedding (IAE) and Structured

---

Fuzzy Reconstruction (SFR). The framework takes the basic features of different parts as input. In the IAE stage, the multiple features are assembled isomorphically by the Hypergraph Isomorphism Convolution (HIConv), and the assembly embeddings are generated from multiple parts. Next, in the SFR stage, the leverage hypergraph structure is constructed based on the local-certainty and global- uncertainty correlations. Guided by this structured open-set distribution, hypergraph convolution is adopted for propagation implicitly from seen categories to unseen categories. Finally, the memory bank is adopted for fuzzy-aware reconstruction and fuzzy embedding generation to further overcome openness during open-set retrieval.
4.2 Isomorphic Assembly Embedding
The IAE module is designed here to obtain assembly embeddings with geometric-semantic consistency from multiple parts. Specifically, the IAE comprises a hypergraph-bame morphism layer and assembly auto- encoders as shown in the left side of Figure 2 We proposed the Hypergraph Isomorphism Convolu- tion (HIConv) in the layer to im- part geometric-semantic consistency to part features, overcoming the bias introduced by the order of parts during assembly representation. The assem- bly auto-encoders are utilized to get the assembly embedding from multi- ple part features.
Part Segmentation Labels
HARF
Input
iso-
Points from
Pre-Trained
00000000
Top-n Labels
Foundation Models
Basic Part
Query Object (Point Cloud)
Other Points
Fentures
Point-War Point-Wise Average
Features
Figure 3: Illustration of input basic part features for HAFR.
The IAE module takes the basic part features ( J C Rxxd)) of N objects with P parts. In this paper, we use the average point feature of each region extracted by 3D point cloud part segmentation network (28) shown in Figure The hypergraph isomorphism layer constructs an assembly hypergraph structure 9a and generates isomorphism embeddings of under the guidance of this structure. The assembly auto-encoders A. encode the multi-part features of 3D objects and get the assembly embeddings u; by integration function.
4.2.1 Isomorphism Smoothing
Although part features have the potential to adequately represent the category information of every object, direct fusion for multiple parts may lead to inconsistencies arising from geometric factors, including the order of input and the presence of repeated parts. The HIConv are utilized to impart geometric-semantic consistency during part assembly, aiming generate isomorphism embeddings from independent-par to correlated-object distribution, by the correlation-based smoothing under the guidance of the assembly hypergraph structure.
hypergraph can be represented as 9 = {V,E), where V and È are the vertex set and the hyperedge set, respectively. In the hypergraph of HIConv, the basic part features fl are treated the vertices
x = U J Then the isomorphism hyperedges E, are constructed as the subset vertices that are from the same object:
E, (0.(1) (1, ,N)
where (i) denotes all vertices of the i-th object. In this way, obtain N hyperedges and N is the number of objects. The assembly hypergraph is constructed as G, = Eo) after getting vertices and hyperedges.
After the construction of the assembly hypergraph, the HIConv designed to bridge the geometry and semantic correlations for isomorphic smoothing. For the convenience of convolution, the incidence matrix H E (0, [0,1] 1D to represent the hypergraph, where the hyperedges are the columns of H. and ((e, e) if vertex v are contained in hyperedge e. Inspired by [18 and [. the HICony is designed to leverage the geometric-semantic collaborative information under the
5

---

guidance of assembly hypergraph:
x MLP (1 + HD, "DD'''' D, 0
where H denotes the incidence matrix of the hypergraph Ga. respectively. D. and De are the diagonal degree matrices for vertices and hyperedges, respectively. denotes learnable parameter and ① #IConm is the learnable matrix for HIConv. After the HIConv, the isomorphism embeddings X, = ((c] == L. are generated from independent-part to correlated-ohject space.
4.2.2 Assembly Embedding
The IAE module first utilizes assembly auto-encoders A. to encode the isomorphism embeddings into a latent code space for each part, then employs the isomorphism loss to pull the different part codes together ensure encoded embeddings from the same object together. Besides, the intra-part and inter-part reconstruction loss are proposed to reduce information loss during compression.
Specifically for the Aa, we have n. 10'" (cf) and 2 4" (uf), where cj E Rd denote isomorphism embeddings of parts, u, E Rd, denote the unified embeddings compressed from isomorphism embeddings of different parts. The encoder and decoder are defined as Vi" := Sy + S. and
d' = S. + Sp. which map the representation between local-part space S to global-assemle space Sa.
After the generation of unified embeddings ot for each part, the IAE module employs an integration function B()) obtain the assembly embeddings integrated from all parts of the same object:
= B(1
(5)
4.2.3 Loss Function for the LAE
For better isomorphism smoothing and assembly embedding, we adopt the Assembly Loss Cas and the Cross-Part Loss Lzp for the IAE module, which is designed pull the distance and prompt the generalization performance across parts, respectively:
2
k=1 E
R
Las = R(R 1)
|| utlle.
(6)
Cup R(R 1) E (1c+ - lle  (v* (4)))l2)
2 D
where |1-1/2 is the L2 norm, ut and u² are both the unified embeddings but from different parts of the same object, 4 is the encoder of k-tth part and &' is the decoder of (-th part.
The loss function for IAE is constructed by the balanced combination of the two losses: LIAE aLas + (1-a) where a is the hyper-parameter to trade-off between them.
4.3 Structured Fuzzy Reconstruction
Although the IAE module generated the assembly embeddings from multiple parts, the distribution skew across seen and unseen categories still affects the performance of open-set retrieval. As shown in Figure 2] proposed the SFR module for generalization. The SFR module first constructs leverage hypergraph to model the local-certainty and global-uncertantyy correlations. Then SFR employs hypergraph convolution for propagation from categories unseen categories based on the implicit leveraged structure. After that, the memory bank adopted to reconstruct the propagation embeddings into fuzzy space further overcome openness during open-set retrieval.
4.3.1 Leverage Propagation
To get the most out potential correlations from seen categories, the leverage hypergraph is designed here. As shown in Figure? 2 the assembly embeddings u, treated the vertices X,, {u, N in the leverage hypergraph, and the hyperedges are constructed from two perspectives: local-certainty and global-uncertainty correlations.
The local-certainty hyperedges &, are constructed based on the category observability, which defined E. (v) )), where (v(y) denotes the subset of vertices that belong to the

---

seen categories y. For the global-uncertaie uncertainty hyperedges Eu, we construct them by linking each vertex and its K neighbor vertices: &, {KKNN, (v) v V}. where KKNN, (t) denotes the top-k nearest neighbor set of vertex v. In this way, we obtain one local-certainty hyperedge and N global-uncertainty hyperedge. The leverage hypergraph is constructed by 9iev = ,X & U E.).
After the construction of the leverage hypergraph, we utilize the modified hypergraph convolution from for embedding propagation.
(8) where denotes the incidence matrix of the leverage hypergraph Giev. D, and D are the diagonal degree matrices for vertices and hyperedges, respectively. Iiev denotes the learnable matrix for the HGNNConv, X are the propagation embeddings and X. {pr).
X = D
4.3.2 Fuzzy Reconstruction
To overcome the distribution skew caused by the open-set environment, the memory bank is adopted here to reconstruct the propagation embeddings to fuzzy space. The memory bank is designed to store large amount of fuzzy representations with uniform distribution. Specifically, the memory bank M is composed of Z invariant memory anchors m, for 3D objects.
M {m, 1 1.. _}}
Given the propagation embedding ü, of the each 3D object, the activation score $t calculated for every memory anchor m, in .M $t = ||ū - myl12. where || is the C2 norm for distance metric, $t, denotes the activation score of each anchor. Then scores s/ to rebuild the propagation embedding into fuzzy space and get fuzzy embeddings 2; by:
the normalization of activation
2
21
(10)
4.3.3 Loss Function for the SFR
In the SFR stage, we adopt the Cross-Entropy Loss Lee and Fuzzy Reconstruction Loss fzz
Ci = [ zi|l2
(11)
Lee - k=1
n log((()) + nn, mlg.log()..))
(12)
where Pi,k Exsr M1. and qi,k Elzr eA1,E denote the prediction scores of fuzzy embeddings
PLA
ctikk
that the image belongs the k-th category. nj, the k-th value of one-hot encoded seen category labels, Y is the number of seen categories.
The loss function for SER is constructed by the balanced combination of the two losses: LSFR fC,; + +1 Cee, where 3 the ppprmmpr to trade-off between them.
Experiments
5.1
Experimental Settings
OpenPart Datasets. We construct three datasets for assembly-based open-set 3D ob- ject retrieval (OpenPart datasets), including OP-SHNP, OP-INTRA, OP-COSEG based
Table 1: The statistics of the three OpenPart datasets.
Dataset
OP-SHNP
OP-INTRA
OP-COSEG
on the public dataset ShapeNetPart [42] In-
Average Parts/Sample
3.2
3.1
2.3
trA [41] and COSEG (34). We sampled the point cloud from the triangular surface for each dataset. As shown in Table I the classes of these datasets are split into seen and un- seen classes for training and testing, respec- tively. Each class contains three to five parts. Specifically, the detailed descriptions of the datasets and parts segmentation are shown in Appendix B
All
3
Category
Unseen
10
2
Train
5804
116
240
Retrieval
6142
2318
802
Number
Query
302
Target
5197
2016
717

---

Table 2: Quantitative results of retrieval the OP-SHNP, OP-INTRA, and OP-COSEG datasets.
OP-SHNP
OP-INTRA
OP-COSEG
Method
MAPt
NDCG
ANMRR
MAPt
NDCG1
ANMRR
mAPt
NDCG1
ANMRR,
MMJN
0.5685
0.5856
0.2599
0.5465
0,5898
0.5053
0.6394
0.7623
0.4314
TCL
0.5683
0.5861
0.2608
0.5467
0.5970
0.5059
0,6285
0.7543
0,4401
SDML
0.5699
0.5870
0.2593
0.5456
0.5944
0.5064
0.6328
0.7638
0,4422
MMSAE
0.5637
0,5824
0.2659
0,5452
0.5919
0.5056
0.6350
0.7555
0.4334
PROSER
0.5687
0,5861
0.2607
0,5462
0,5943
0.5059
0.6343
0.7605
0,4348
HGM"R
0.5736
0,5886
0.2549
0.5545
0.6019
0.4928
0.6452
0.7627
0.4355
Ours
0.5947
0.5916
0.2239
0.5750
0.6382
0.4797
0.7015
0.7629
0.3604
1.0
0.6
0.8
0.8
Precision
Precision
Precision
0.6
0.4
MMINN
TCL
0.4
MMIN
TCL
0.4
MMIN
TCL.
SDML
MMSAE
SDML
MMSAE
SOML
MMSAE
PROSER
HGM-2R
0.2
0.2
Ours
Our
Curs
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
Recall
(a) PR Curve on OP-SHNP.
(b) PR Curve on OP-INTRA.
(c) PR Curve on OP-COSEG.
Figure The Precision- Curves for comparison on the three datasets, respectively.
Implemental Details. The random seeds are fixed to 2024 this paper for a fair comparison. The basic features are extracted by PointNet [28] and each part feature is obtained by the average point feature of each region, we use the top four parts for each object in this paper. We set a 0.5 in LIAE and 3 0.9 in LSFR. The IAE is trained for 40 epochs on learning rate Ir 0.1, and the SFR is trained for 30 epochs on Ir 0.001. The detailed implements of the HAFR framework are provided in Appendix C
5.2 Retrieval Performance
Compared Methods. Since methods are specifically designed for assembly- open-set 3DOR, we refine the current state-of-the-art close-set 3DOR methods (MMJN (26.. TCL. [3] SDML [4, MMSAE (36)), and open-set 3D learning methods (PROSER (46). HGMPR (8), then we refine the multi-modal fusion module with multi-part fusion for each method. We provide detailed implements of compared methods in Appendix
Evaluation Metrics. For fair comparison, we use standard retrieval metrics, including Mean Average Precision (mAP), Normalized Discounted Cumulative Gain (NDCG), Average Normalized Modified Retrieval Rank (ANMRR), and the Precision- Recall Curve (PR-Curve). For the mAP and NDCG metrics, higher scores are better. For the ANMRR metric, a lower score is better.
Comparsion Analysis. As shown Table we evaluate the assembly-basee open-set retrieval results from HAFR framework and other state-of-the-arr compared methods. Quantitative results demonstrate the superiority our method over the other methods on all three datasets. In particular, on the OP-COSEG dataset, our method achieves 0.7015 mAp with about 8.7% improvements Curves
compared with the second-best method (HGMPR). We also compare the Precision-Redd (PR-Curve) which evaluate the trade-off between precision and recall different thresholds. A larger area between the curve and the axes indicates better performance for retrieval, our method outperforms all other retrieval methods shown in Figure
The superior performance in the comparison indicates that by the proposed IAE and SFR modules, the proposed HAFR framework can better utilize the potential semantic information among part

---

Table 3: Ablation studies of retrieval the OP-SHNP, OP-INTRA, and OP- COSEE datasets.
OP-SHNP
OP-INTRA
OP-COSEG
Ablation
MAPt
NDCGf
ANMRR
MAPt
NDCGt
ANMRR
MAPt
NDCGf
ANMRR
HICony GII
0.5761
0.5826
0.2465
0.5567
0.6050
0.4947
0.6782
0.7625
0.4001
HIConv-MMLP
0.5837
0.5854
0.2357
0.5523
0.6035
0.4948
0.6505
0.7664
0.4334
IAE w/o fas 0.5767
0.5838
0.2464
0.5560
0.6077
0,4946
0.68299
0.7542
0.3823
IAE w/o Cxp
0.5767
0.5840
0,2464
0,5561
0.6078
0.4944
0.6828
0.7542
0,3824
SER w/o Gtev
0.5755
0.5825
0.2474
0.5545
0.6019
0,4928
0.6887
0.7540
0.3768
MLP-based SFR
0.5636
0.5851
0,2665
0,5429
0.5814
0,4997
0,6662
0.7610
0,4062
GCN-based SFR
0.5679
0.5845
0,2603
0,5430
0.5854
0.4982
0.6674
0.7618
0.4072
IAF+SFR
0.5947
0.5916
0.2239
0.5750
0.6382
0.4797
0.7015
0.7629
0.3604
0.6
0.8
0.8
Precision
Precision
Precision
0.4
0.6
0.6
0.2
HBC -MLP
JAB wio Las SER G Jey
B0 -GIN
JAE AN L_
0.4
IAE woo Los
IAE wio L_sp
HlCorr -GEN
JAE wil L SFR wi C Jev
IAE wit
ClCo CCN
MIP SFR
SER w Jey
MLP bosed SFR
0.2
MLP baod SFR
CCNoowed 9FR
CCNiooed SPR
Oues
SrCchase STR
Oan
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
Recall
(a) PR Curve on OH-S
(b) PR Curve on OP-INTRA.
(c) PR Curve on OP-COSEG.
Figure 5: The Precision- Recall Curves of the ablation studies on the three datasets, respectively.
features to fully represent 3D objects and generalize them to unseen categories. Notably, for the OP-INTRA dataset with only category and the least parts in the training set, our method still achieves performance improvements in the retrieval of unseen categories as shown in Table and Figure This indicates that our method has minimal dependence the performance of classifiers during basic feature extraction. In open-set environments, where training on all potential categories is infeasible, our HAFR method through Part Assembly demonstrates superior open-set adaptability. We provide more visualizations and results in Appendix D.
5.3 Ablation Studies
We conduct ablation studies to verify the effectiveneess of modules in the proposed framework. For the IAE stage, we first remove the Hypergraph Isomorphism Convolution (HIConv) layer for comparison. Specifically, we replace it with the Graph Isomorphism Convolution (HIConv-GIIN) and MLP (HIConv-MMLP) Then we compared with the method without the Assembly Loss (IAE w/o Las) or the Cross-Part Loss(IAE w/o Czp). As shown in Table 3 and Figure 3 replacing HIConv or removing parts of the LIAE loss significantly degrades the performance of IΛE. For the HIConv, this indicates that naive part assembly is dependent on geometric or semantic factors such as order and quantity. Removing isomorphism smoothing results in geometric or semantic inconsistencies, thereby degrading the retrieval performance by the worse assembly embeddings. As for the SFR module, we compare the hypergraph structure without the leveraged structure (SFR w/o Glev) and we also replace the hypergraph- based structure learning with MLP (MLP-baser SFR) and GCN (GCN-based SFR). Besides, we remove the memory bank to verify the effectiveness of fuzzy reconstruction.
Table 3 and Figure show that the proposed full SFR module outperforms all the other ablative struc- tures, these results show that the proposed leverage propagation and fuzzy reconstruction approach can effectively utilize the high-order correlations between seen and unseen categories for open-set generalization. Besides, we can observe that the complete combination of IAE and SFR yields the best performance. These quantitative and qualitative results indicate that the proposed HAFR framework effectively achieves leve- assembly isomorphism and unification while mitigating distribution skew from seen certainty to unseen uncertainty at the object level.

---

6 Conclusion
In this paper, we introduce the Hypergraph-Bass Assembly Fuzzy Representation (HAFR) frame- work, which navigates the intricacies of open-set 3D object retrieval through bottom-up lens of Part Assembly. Specifically, we propose the Hypergraph Isomorphism Convolusion (HIConv) and adopt the Isomorphic Assembly Embedding (IAE) module for assembly isomorphism and unification, generating the integration embeddings with geometric-semane
consistency. Besides, we employ the Structure Fuzzy Reconstruction (SFR) approach to exploit high-order correlations among objects and fuzzify representations for open-set category generalization. This module constructs a lever- aged hypergraph based on local-certainty and global-uncertainty correlations mitigate distribution skew. We construct three open-set retrieval datasets for 3D objects with part-level annotations, i.e. OP-SHNP, OP-INTRA, and OP-COSEG, Extensive experiments and ablation studies on these three benchmarks show our method outperforms current state-of-the-a methods. However, due to data limitations, this paper does not currently consider the assembly fuzzy representation for varying numbers parts, which will be a focus of our future research. We believe this paper provides a novel perspective for open-set set retrieval by exploring from local to global levels.
7
Acknowledgement
This work was supported by Beijing Natural Science Foundation (No. L242167), CCF-Tencent Open Research Fund, and Jiangxi Provincial Natural Science Foundation (20244CC2118002)
References
[1] Antonio Alliegro, Francesco Cappio Borlino, and Tatiana Tommasi. Towards open set learning: benchmark on object point clouds. arXiv preprint arXiv:2207.11554 2022.
[2] Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 1 110: 107637,
[3] Abhijit Bendale and Terrance Boult. Towards open set deep networks. /EEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1563-1572, 2016.
[4] Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Probabilistic reasoning for assembly-based 3d modeling. In ACM SIGGRAPH, pages 1-10, ACM New York, NY, USA, 2011.
[5] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions Pottern Analysis and Machine Intelligence, 44(1) 10:65-5065---- 8081, 2021.
[6] Wei Chen, Yu Liu, Weiping Wang, Erwin M Bakker, Theodoros Georgiou, Paul Fieguth, Li I. Liu, and Michael Lew. Deep learning for instance retrieval: survey. IEFE Transactions Pattern Analysis and Machine Intelligence, 2022.
Yihe Dong, Will Sawin, and Yoshua Bengio. Hnhn: Hypergraph networks with hyperedge neurons. arXiv preprint arx:220006. 12278, 2020.
Yifan Feng, Shuyi Ji, Yu-Shen Liu, Shaoyi Du, Qionghai Dai, and Yue Gao. Hypergraph- multi- modal representation for open-set 3d object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(4):2206-2223, 2023.
[9] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI Conference on Artificial Intelligence, pages 3558-3565, 2019.
[10] Thomas Funkhouser, Michael Kazhdan, Philip Shilane, Patrick Min, William Kiefer, Ayellet Tal, Szymon Rusinkiewicz, and David Dobkin. Modeling by example. ACM Transactions on Graphics, 23(3):652 663,,,
2004.
[11] Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji. Hgnn+: General hypergraph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3181-311 2022.
[12] Yue Gao, Meng Wang, Dacheng Tao, Rongrong Ji, and Qionghai Dai. 3-d object retrieval and recognition with hypergraph analysis. IEEE Transactions Image Processing, 22(9):4290-44 4303, 2012.
10

---

[13] Xinwei He, Yang Zhou, Zhichao Zhou. Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1945-1954, 2018.
[14] Peng Hu, Liangli Zhen, Dezhong Peng, and Pei Liu. Scalable deep multimodal learning for cross-modal retrieval. In Annual Conference of the Association for Computing Machinery Special Interest Group in Information Retrieval, pages 635 644, 2019.
[15] Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian, Cross-modal center loss for 3d cross- modal retricval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3142-3151, 2021.
[16] Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3830-5840, 2021.
[17] Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun, A probabilistic model for component- shape synthesis. ACM Transactions on Graphics, 31(4):1-11, 2012.
[18] Xu Keyulu, Hu Weihua, Leskovec Jure, and Jegelka Stefanie. How powerful are graph neural networks? In International Conference on Learning Representations. OpenReview.net, 2019.
[19] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d ohject representations for fine-grained
categorization. In International Conference on Computer Vision Workshop, pages 554-561, 2013.
[20] Jun Li, Chengjie Niu, and Kai Xu. Learning part generation and assembly for structure-aware shape synthesis. In AAAI conference on artificial intelligence, pages 1 11362-11369 2020.
[21] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics, 36(4):1-14, 2017.
[22] Qi Liang, Mengmeng Xiao, and Dan Song. shape recognition based on multi-modal information fusion. Multimedia Tools and Applications, 80: 10173316111 2021.
[23] Minghua Liu, Ruoxi Shi, Kaiming Kuang. Yinhao Zhu, Xuanlin Li, Shizhong Han. Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open- understanding. Annual Conference on Neural Information Processing Systems, 36, 2024.
[24] Katia Lupinetti, Phillipe Peroo,, Marina Monti, and Franca Giannini. Content- cad assembly model retrieval: Survey and future challenges. Computer-Aider Design, 113:62-81 81, 2019.
[25] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J Mitra, and Leonidas J Guibas. Struc- turenet: hierarchical graph networks for 3d shape generation. ACM Transactions on Graphics, 3-(6):1119, 2019.
[26] Weizhi Nie, Qi Liang, An-An Liu, Zhendong Mao, and Yangyang Li. Mmjn: Multi-modal joint networks for shape recognition. In ACM International Conference on Multimedia, pages 908-916, 2019.
[27] Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Open-world machinc learning: applications, challenges, and opportunities. ACM Computing Surveys, 55(10):1-33 2023.
[28] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[29] Chao-Hui Shen, Hongbo Fu, Kang Chen, and Shi-Min Hu. Structure recovery by part assembly, ACM Transactions on Graphics, (1(6):111,, 2012.
[30] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEFF Transactions on Pattern Analysis and Machine Intelligence, 4368):2647-266 2020.
[31] Yuting Su, Yuqian Li, Weizhi Nie, Dan Song, and An-An Liu. Joint heterogeneous feature learning and distribution alignment for 2d image-based 3d object retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 30(10):365-3376, 2019.
[32] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need? arXiv preprint arXiv:2110.06207 2021.
11

---

[33] Kai Wang, Paul Guerrero. Vladimir G Kim, Siddhartha Chaudhuri, Minhyuk Sung. and Daniel Ritchie. The shape part slot machine: Contact-based reasoning for generating 3d shapes from parts. In European Conference on Computer Vision, pages 610-626, Springer, 2022.
[34] Yunhai Wang. Shmulik Asafi, Oliver Van Kaick, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. Active co-analysis of of shapes. ACM Transactions on Graphics, 31(6):1-10, 2012.
[35] Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1850-1859, 2020.
[36] Yiling Wu, Shuhui Wang, and Qingming Huang. Multi-modal semantic autoencoder for cross-modal retrieval. Neurocomputing. 331:165-175, 1011.
[37] Xianghao Xu, Paul Guerrero, Matthew Fisher. Siddhartha Chaudhuri, and Daniel Ritchie. Unsupervised 3d shape reconstruction by part retrieval and assembly. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8559-8567, 8567, 2023.
[38] Yang Xu, Yifan Feng. and Lin Bie. Triadic elastic structure representation for open-set incremental 3d object retrieval. In ACM International Conference on Multimedia Retrieval, pages 20 28. 2024.
[39] Yang Xu, Yifan Feng, and Yue Gao. Negative prompt driven complementary parallel representation for open-world 3d object retrieval. In International Joint Conference on Artificial Intelligence, pages 1498-1506, 2024.
[40] Yang Xu, Yifan Feng. and Yu Jiang. Structure-aware residual-center representation for self-supervised open-set 3d cross-modal retrieval. In IEEE International Conference on Multimedia and Expo. pages 1-6. 2024.
[41] Xi Yang, Ding Xia. Taichi Kin, and Takeo Igarashi. Intra: 3d intracranial aneurysm dataset for deep learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3656-2666, 2020.
[42] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cowu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics, 35(6):1-12, 2016.
[43] Haoxuan You, Yifan Feng. Rongrong Ji, and Yue Gao. Pvnet: joint convolutional network of point cloud and multi-view for 3d shape recognition. In ACM International Conference on Multimedia, pages 1310-1318, 2018.
[44] Haoxuan You, Yifan Feng, Xibin Zhao, Changqing Zou, Rongrong Ji, and Yue Gao. Pvrne:: Point-view relation neural network for 3d shape recognition. In AAAI Conference on Artificial Intelligence, pages 9119-9126, 2019.
[45] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Annual Conference Neural Information Processing Systems, 33:6315-6326, 2020.
[46] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4401-4410, 2021.
[47] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 9()) 2022.
[48] Chenming Zhu, Wenwei Zhang. Tai Wang, Xihui Liu, and Kai Chen. Object2scene: Putting objects in context for open- -oca-vocyyylary 3d detection. arXiv preprint
2023.
A Assembly-Based Open-Set 3DOR
In practical scenarios, the categories of 3D objects can be labeled from various perspectives such as geometric and semantic, or from different levels of granularity. From geometric perspective, 3D object is naturally composed of multiple part-level shapes. For instance, 3D model of car can be deconstructed into several part-level shapes, such as wheels, chassis, doors, and windows. Each of these parts represents a distinct geometric part that contributes to the formation of the complete 3D object. This modular decomposition facilitates more efficient manipulation, analysis, and reconstruction, as each part can be independently modified replaced while preserving the overall structural integrity of the automobile. These parts not only enhance hierarchical structure and diversity of labels but also play a crucial role in the representation of 3D models. Consequently, they find applications in various fields, such as computer-aie aided design (CAD). where precise modeling of individual components is essential: virtual reality (VR), which relies detailed and interactive 3D environments: and
12

---

Table 4: The category splitting on seen and unseen categories for the three datasets.
Dataset
Seen Categories (Training Set)
Unseen Categories (Testing Set)
OP-SHNP
airplane, guitar, lamp, laptop, mug, skateboard
bag, cap, car, chair, earphone, knife, mo- torbike, pistol, rocket, table chairs, fourleg, guitars, irons, lamps, vases aneurysm, vessel
OP-INTRA OP-COSEG
candelabra, goblets, telealiens mixed
Table 5: Detailed statistic of OP-SHNP.
Category
airplane
guitar
lamp
laptop
mug
skatehoard
bag
cap
Samples
2690
787
1546
445
184
152
76
55
Parts
4
2
3
Category
car
chair
earphone
knife
motorbike
pistol
rocket
table
Samples
898
3746
69
392
202
275
66
363
Parts
3
2
3
Table 6: Detailed statistic of OP-COSEG.
Category
candelabra
goblets
telealiens
chairs
fourleg
guitar
irons
lamps
vase
Samples
28
12
200
400
44
18
300
Parts
4
3
4
5
3
3
3
medical imaging, where accurate representations of anatomical structures critical for diagnosis and treatment planning.
As for 3D object retrieval (3DOR) task, individual parts and combinations of multiple parts can serve as crucial clues for the representation and analysis of a complete 3D shape. The greater the number of parts, the higher the accuracy and uniqueness of the object representation. This characteristic is particularly advantageous for open-set 3DOR through bottom-up lens of Part Assembly. Given the incomplete object-level labels in open-set set settings, employing a multi-layered representation and analysis of individual and associated parts can significantly enhance the accuracy open-set 3D object retrieval (3DOR). Furthermore, may reduce the dependency on extensive training data, addressing a common limitation in open-set environments.
B OpenPart Dataset Generation
generate three part-assembly driven open-set 3D object retrieval (OpenPart) datasets, including OP-SHNP, OP-INTRA, OP-COSEG based on the public dataset ShapeNetPart
Table 7: Detailed statistic of OP-INTRA.
42 IntrA [. and COSEG (34. We sampled point cloud from the triangular surface for each dataset. Specifically, the point number for point clouds in OP-SHNP, OP- INTRA. and OP-COSEG respectively, the OpenPart datasets, we use part segmentation annotations of the original dataset, the classes are split into seen unseen classes for training and testing as shown in Tab. Besides, we provide more detailed statistics of the three datasets in Table Table and Table 7
Category mixed
aneurysm
vessel
Samples 116
1290
1028
Parts
3
2
2
2048, 2048, and 1024,
C Implemental Details
The proposed Hypergraph-Bass Assembly Fuzzy Representation (HAFR) framework is composed of two modules: Isomorphic Assembly Embedding (IAE) and Structured Fuzzy Reconstruction (SFR). The basic features for input extracted by PointNet (8 and each part feature is obtained by the average point feature of each region, use the top four parts each object in this paper. The IAE utilizes Hypergraph Isomorphism Convolution (HIConv) and assembly auto auto-encoders A, to generate assembly embeddings from the basic part features. The implemented details of the IAE and SFR modules are provided in Algorithm and Algorithm 2 respectively.
13

---

Algorithm Training the IAE module
Input: Basic part features {fr) Parameter: 0.5.
Output: Assembly embeddings {G/
Let epoch
2: Initialize assembly hypergraph 9a = (X,, Eo];
3: Construct vertices x = ( (
!== 4: Construct isomorphism hyperedge & = (Op i (1,.,,,,,)) ,N)]:
5: Calculate diagonal degree matrices D and De:
6: Calculate incidence matrix H of Ga:
7: Initialize learnable HIConv parameters HICono of Gai
8: Initialize assembly auto-encoder Aa = 13
9: Initialize aggregation function B;
10: while epoch s 40 do
Isomorphism embeddings x, MLP((1 + e) oD, HD. x HICono)):
11:
12:
Reshape isomorphism embeddings {[]] IN = X,;
13:
Get unified embeddings of each part {[u[)=])
Get assembly embeddings of each object (ur = {B((u] rr=
i= (V(llc IN
IN
14:
Get mixed embeddings if each part {[AT)p
P
- ("(((c)) N
16:
Calculate the Assembly Loss Las I[[R-1]
Ek=l Eis411 |u w/llz.
17:
Calculate the Cross-Part Loss Lxp = R(R-1) 2k=1, (1o) 22 + |0
2
Φ (v (2))
Calculate loss for the IAE module LIAE aLas + (1 a) Cap.
19:
if LLAE does not converges then
Update parameters of HICone and Ag by LIAE.
21:
epoch+ =
22: else
23:
Break.
end if
25:
end while
26:
return Assembly embeddings {u:
Our experiments were conducted Tesla V100-32G GPU and an Intel(R) Xcon(R) Silver 4210 CPU @ 2.20GHz. The hyper-parameters "k" in the SFR module 20, 6, and 40 for OP-SHNP, OP-INTRA, and OP-COSEG, respectively.
D
More Results
Since no methods are specifically de-
signed for part-assembly driven open-set
3DOR, refine the current state-of-the-
Table 8: The hyper-parameters of the HAFR framework.
art close-set 3DOR methods (MMJN (26).
IAE
SFR
TCL [13. SDML (14), MMSAE (36)).
and open-set set 3D learning methods
Optimizer
SGD
SGD
(PROSER 446. HGMRR (B), then fine the multi-modal fusion module with multi-part fusion for each method. Specif- ically, these compared methods are imple- mented by:
Learning Rate
0.1
0.001
Momentum
0.9
0.9
Weight Decay
0.0001
Scheduler
Cosine Annealing
Cosine Annealing
Tmax
40
60
etamin
0.00001
0.00001 30
MMJN 26 MMJN is a multi-modal joint network that employs weighted fu- sion to integrate features across multiple modalities for retrieval. Specifically, we generate the assembly embeddings by auto-encoders and utilize them in the classification fusion part of the MMJN network.
Max Epoches
40
14

---

Algorithm 2 Training the SFR module
Assembly embeddings {ur) Parameter: 3 = 0.9. Output: Fuzzy embeddings {
Input:
Let epoch =
2: Initialize leverage hypergraph Giev = (Xu, Elevl;
3: Construct vertices X., iim
4: Construct
hyperedge & = (C,()) 1 }); 5: Construct global-uncertainty hyperedge & (KKNN, (v) lee
6: Construct hyperedges Etev = & U Exi
7: Calculate diagonal degree matrices D, and De:
8: Calculate incidence matrix H of Gievi
9: Initialize HGNNConv parameters Oeev of Gtevi
10: Construct memory bank M:
11: while epoch < 120 do
Get propagation embeddings X., - D "HH ".D
13:
Reshape propagation embeddings {p:} X.. Get activation ši,j || m2j
15:
Get fuzzy embeddings [ = |Σ/=1 &1, m
16:
Calculate the Cross-Entroppy Loss Lee Σ k= (1g..lo((P.))) +g,.
Calculate the Fuzzy Reconstruction Loss f ū; 2: lg
18:
Calculate loss for the SFR module LEFR fL (1 Loc
19:
if LSFR does not converges then
Update parameters of iew and M by LSFR.
21:
epoch+
22: else
23:
Break.
end if
25:
end while
26:
return Fuzzy embeddings {=r),-]
Retrieval by fuzzy embeddings {z} J=1-
TCL. [33:
TCI. is method hased metric learning, combining triplet and center loss get unified fusion embeddings from different modalities. We use center-based method for object center embedding from different parts
SDML. is a metric-lcamming- hased method for cross-modal retrieval, which learns projection functions for different modalities independently. We construct the assembly embeddings by auto-encoders and the joint supervision aligned with its triplet center loss.
SDML 14]
MMSAE 36 MMSAE multi-men multi-modal retrieval method using auto-encoders. It trains encoders with reconstruction loss function align embeddings from various modalities into unified latent space. We add auto-encoders to generate assembly embeddings, and we use them for alignment with semantic code vectors for 3D object retrieval.
PROSER [46]: PROSER is an open-world recognition method that extends the closed-set classifier to determine if sample belongs seen categories not. We construct auto-encoders for assembly embedding, and we mix up the assembly embeddings with the multiple dummy classifiers in the PROSER.
open-world 3D multi-modal retrieval method, which retrieves the objects from unseen categories through structure-aware learning. construct auto-encoders for assembly embedding of 3D parts, and we use the KNN- based hyperedges in the SAIKL module.
HGM'R R 3 "GM"R
We provide the visualized results of assembly embeddings and fuzzy embeddings in Fig. and Fig. 7 The visualizations indicate that the SFR module is capable of accurately representing open-set categories, which can effectively enhance retrieval performance.
15

---

chairs
Tourieg
guitars
trans
lamps
VOIES
Chais
Toureg
guitars
Irons
lamps
V02e5
(a) Assembly embeddings on OP-INTRA.
(b) Fuzzy embeddings on OP-INTRA.
Figure 6: The t-SNE visualization of the embeddings from unseen categories in the OP-INTRA and OP-SHNP datasets.
bag
car
earphone
matorbike
racket
bag
sarahone
motorbike
rocket
knff
pistol
table
cap
knife
(a) Assembly embeddings OP-SHNP.
(b) Fuzzy embeddings OP-SHNP.
Figure 7: The I-SNE visualization of the embeddings from unseen categories in the OP-SHNP dataset.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We provide comprehensive motivations and detailed introductions of all the contributions in the abstract and introduction.
Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
16

---

Answer: [Yes]
Justification: discussed the current limitations work the conclusion.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does paper provide full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provided a formalized, step-by-step explanation each procedure methodol-
ogy.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification:
We provide detailed experiment and reproduction settings in the experiments and
appendix.
5. Open access to data and code
Question: Does the paper provide open the data and code, with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Answer: [No]
Justification: are the preparation of the codes, models, and datasets, and they will be released the earliest opportunity.
6. Experimental DeetinggDetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all experimental settings and details settings in the experiments and appendix.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We explain the evaluation protocols of the experiments.
Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the sufficient information on the computer resources in appendix.
Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies
Answer: [Yes]
Justification:
We have carefully read and adhered to the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts of this work in the conclusion.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
17

---

Justification: This paper poses such risks.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have obtained and adhered to the permissions for the datasets used and have cited them the paper.
13. New Assets
Question: Are new assets introduced paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the dataset and provide the corresponding documentation immediately upon the paper's acceptance.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor rescarch with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed the suhjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "1JFubbntBz", "reviewer": "Reviewer_3mhr", "summary": "This paper proposes a framework for open-set 3D object retrieval, called the Hypergraph-Based Assembly Fuzzy Representation (HAFR) framework. This model leverages an Isomorphic Assembly Embedding (IAE) to integrate geometric and semantic consistency. Furthermore, a Structured Fuzzy Reconstruction (SFR) is used to overcome the distribution skew of unseen categories. On three point cloud datasets constructed by the authors, this model outperforms the state-of-the-art.", "questions": "Please refer to paper weaknesses.", "limitations": "The authors discusseds the limitations in the conclusion section. But I did not find the societal impacts mentioned in the conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation for this work is well-established.\n- The idea of using hypergraph structures to achieve high-order correlations both within and between objects is novel.\n- Sufficient quantitative and qualitative comparisons verify the effectiveness of the proposed model.", "weaknesses": "- In structured fuzzy reconstruction, the value of k in the k-nearest neighbors seems to determine the global uncertainty hyperedge. However, the paper lacks explanation or experiments to clarify the selection of k value.\n\n- While HGM2R [1] employs a multimodal approach, the IAE component appears to be similar to the Multi-Modal 3D Object Embedding in HGM2R. What are the differences and unique contributions of IAE compared to the embedding technique used in HGM2R?\n\n-In Table 2, although HGM2R also utilizes hypergraphs, it shows only slight improvements over previous methods in most metrics. For example, the mAP scores on three datasets are only about 0.1 higher. However, the method proposed in this paper demonstrates a significant improvement over HGM2R on the OP-COSEG dataset, with an increase of nearly 0.6. How can this result be explained?\n[1] Hypergraph-Based Multi-Modal Representation for Open-Set 3D Object Retrieval. TPAMI 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer 3mhr**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for us to enhance the rigor and coherence of our paper and directed the focus of our future work.\n\n1. **About the ablation study on $K$-value (Answer for Weakness 1)**:\nWe further conduct ablation studies on hyper-parameters $K$ to validate the influence of uncertainty hyperedge in the leverage hypergraph. As shown in Fig. R1 of the rebuttal PDF, as the $K$ varies, the performance of the proposed method remains stable and outperforms the compared method in most ranges. However, the performance ceases to improve and may even slightly decrease after reaching a certain value, indicating that the extraction and utilization of high-order correlations have reached saturation. Besides, we can find that different datasets have different peak values of $K$. A larger K-value indicates the requirements for more extensive and deeper capturing of high-order correlations. Your suggestions have inspired our future work, and we will focus on balancing the performance and complexity of the structure-aware network to achieve optimal relationship modeling. We will also include this part of the experiment and analysis in the paper.\n\n2. **About the unique contributions of IAE (Answer for Weakness 2)**:\nThe IAE module is designed to obtain assembly embeddings from multiple part features. Compared to the multimodal embedding method in HGM<sup>2</sup>R, which generates **averaged embeddings of different modalities from a global perspective**, **the IAE module aims at assembled embeddings with geometric-semantic consistency for all parts from a global-local collaborative perspective**. Specifically:\n\n    a) The IAE module constructs a structure (**the Isomorphism Hypergraph**) to capture geometric correlations, such as the order and quantity among the input parts, and then utilizes them for structure-aware fusion. However, HGM<sup>2</sup>R only uses an averaged-guided fusion method with simple auto-encoders.\n\n    b) Guided by the isomorphism hypergraph, the IAE module designs **the Hypergraph Isomorphism Convolution (HIConv) layer** that combines geometric and semantic information to generate embeddings collaboratively. However, HGM<sup>2</sup>R uses a naive MLP for feature mapping and fusion, which loses the high-order information within the input.\n\n    Although the HGM<sup>2</sup>R designs a satisfactory approach for multimodal embedding, its semantic-only averaged embedding paradigm is not suitable for part assembly, which requires the collaborative use of information from different domains. Based on HGM<sup>2</sup>R, the IAE module of our method introduces the Isomorphism Hypergraph and Hypergraph Isomorphism Convolution to achieve part assembly with geometric-semantic consistency. Experimental results in both the paper and the rebuttal PDF demonstrate the necessity and superiority of the IAE module.\n\n3. **About the improvement on OP-COSEG (Answer for the last Weakness)**:\nAlthough HGM<sup>2</sup>R uses a hypergraph structure, they only construct an inter-object hypergraph at the feature smoothing stage (stage 2), without considering the intra-object correlations between different parts of an object, which directly determine the accuracy of the embeddings. Our method constructs hypergraphs for capturing both intra-object and inter-object correlations, as mentioned in the Answer 2 above. We provide examples of the three datasets in Fig. R4, we can find that compared to the OP-SHNP and OP-INTRA datasets, the objects in the OP-COSEG dataset exhibit stronger symmetry, meaning that the isomorphism between parts is more significant. Ignoring geometric information during assembly embedding may lead to inconsistencies within the same category. These challenges result in only slight improvements over non-hypergraph methods. However, our method tackles the challenge of assembly isomorphism and unification, achieving part assembly with geometric-semantic consistency, leading to significant performance improvements on the OP-COSEG dataset.\n\n4. **About societal impacts (Answer for Limitations)**\nAs shown in line 158-159 and Appendix C Algorithm 1, the proposed HARF framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data and such as e.g. text, audio, video, and their pieces. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets, code, configs, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.\n\nThank you again for your valuable suggestions, especially your professional advice on future work in assembly-based open-set learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnhZyQSp30", "reviewer": "Reviewer_Kfog", "summary": "This paper presents a method for finding similar samples from a set of 3D objects given query objects in an open setting, where objects can belong to both already seen and new categories. This method is based on considering 3D objects as hypergraphs consisting of individual geometric and semantic parts of objects. The hypergraph is used to form Isomorphic Assembly Embedding. The second part of the proposed HAFR framework is the Structured Fuzzy Representation module that constructs a hypergraph based on local certainty and global uncertainty correlation to enable transfer from seen to unseen categories. The authors propose a new layer, HIConv, which improves the quality of the generated representation. The authors demonstrate the effectiveness of their approach on three datasets that they constructed for this task.", "questions": "1. How are fuzzy embeddings used to find similar objects in the target set?\n\n2. What is the size of the memory anchors bank? Will the method remain effective if the dataset contains more than 16 categories?\n\n3. The method is described as open-set, but it requires GT segmentation of the object into parts. How do you see its applicability in real-world scenarios where GT segmentation might not be available for any object? How much would the quality metrics decrease if we used a neural network model for part segmentation?", "limitations": "The authors discuss limitations in the conclusion regarding the use of the assembly fuzzy representation for a varying number of object parts. In my opinion, another limitation is the need to segment the point cloud into parts to use this method.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea that one can understand the whole object shape from its parts sounds interesting and reasonable.\n- The description of Isomorphic Assembly Embedding and Structured Fuzzy Reconstruction is formal and rather clear.\n- The authors conduct extensive ablation studies of their method.", "weaknesses": "- Based on the provided experiments, it is unclear if HAFR can generalize well to an unseen domain. Are the results in Table 2 provided for the same suite of model weights?\n- The literature review does not include existing methods for open-set 3d object retrieval and recent methods for closed-set 3d object retrieval.\n- When comparing with other methods, the authors use their own modification of existing multimodal methods. A comparison with modern methods for open-set 3d object retrieval, such as [1], is necessary to demonstrate the effectiveness of this particular method of object representation.\n- The method's description lacks an explanation of how the resulting fuzzy embeddings are used to find similar objects. Additionally, the description contains undefined concepts like isomorphism loss and integration function. If these concepts are not introduced by the authors, please include references to articles where they are defined.\n\n[1] Zhou, J., Wang, J., Ma, B., Liu, Y. S., Huang, T., & Wang, X. (2023). Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Generalization ability and comparison (Weakness 1)**  \nAll categories of the testing set are unseen during training (widely accepted of open-set retrieval [1-2]), the retrieval results in this paper are experimented on the unseen categories. The compared results between different methods are conducted under the same settings, and training weights.  \n    We have explained this open-set setting in Section 5.1 (lines 254-255) and Appendix A (Table 4). We will provide a more detailed explanation in the revised version of this paper. Quantitative and qualitative results show that HARF achieves significant improvements over existing methods. Under this open-set setting, this improvement sufficiently demonstrates the superiority of HARF in terms of generalization capability.  \n    [1] Open-environment machine learning. National Science Review, 2022.  \n    [2] Hypergraph-based multi-modal representation for open-set 3d object retrieval. TPAMI, 2023.  \n2. **Related works (Weakness 2)**  \nIn our submission, We have provided a brief review of closed-set 3D object retrieval methods in lines 78-88, and a summary of recent open-set 3D object retrieval methods in lines 95-97.\n3. **More comparisons (Weakness 3)**  \nWe provide more comparison between modern methods[3][4] for open-set 3d object retrieval. As shown in Tab. R1 of the rebuttal PDF, the results indicate that these two methods have similar performance to the existing multimodal method (HGM$^2$R). Our HAFR framework shows significant improvements over all these SOTA open-set retrieval methods. This improvement highlights the limitations of existing retrieval paradigms in open-set environments and demonstrates the superiority of our assembly-based open-set retrieval paradigm.  \n    [3] Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.  \n    [4] Openshape: Scaling up 3d shape representation towards open-world understanding. NIPS, 2023.  \n4. **The approach for finding similar objects (Weakness 4 and Question 1)**  \nAfter obtaining the fuzzy embeddings (feature vectors) for all objects, we follow the common distance-based approach in the multimedia retrieval field to find similar objects, which is a widely accepted practice in recent decades [5-7]. Given a feature vector (fuzzy embedding) of query object:\na) Calculate the Euclidean distance between the query feature vector and all target object feature vectors.  \nb) Sort these distances in ascending order to get the top-$n$ nearest objects.\nc) Determine whether the class labels of the top-$n$ nearest objects are the same as the query object label and calculate metrics such as mAP, NDCG, ANMRR, and PR-Curve, where $n$ denote the hyper-parameter of the evaluation metrics and can be chosen based on the specific scenario.  \n    [5] A survey of content-based image retrieval with high-level semantics. PR, 2007.  \n    [6] 3-D object retrieval and recognition with hypergraph analysis. TIP, 2012.  \n    [7] Triplet-center loss for multi-view 3d object retrieval. CVPR, 2018.  \n5. **More ablation on the Memory Bank (For Question 2)**  \nThe memory bank we used has 128 anchors and 512 dimensions. The memory bank is a commonly used knowledge distillation method in deep learning. Specifically, it constructs several anchors (feature vectors) and then learns the activation scores of the target embeddings relative to all anchors. The memory bank is independent of the classification layer, and its size usually does not affect the performance of the network when the number of categories changes. This has been validated in methods of multiple fields[8]. We have conducted more ablation studies on the memory bank. As shown in Tab. R1 of the rebuttal PDF, changes in the memory size have almost no impact. We believe HARF remains effective even if the dataset contains more than 16 categories.  \n    [8] Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. CVPR, 2021.\n6. **Methods for part feature extraction (For Question 3)**  \nWe have adopted a neural network model for part segmentation to obtain part features (as described in lines 259-263 and lines 476-478 of the submission). However, instead of extracting features from the segmented parts of the point cloud, we use a segmentation network to obtain point-wise features for each point and then average these point-wise features to obtain part features. As shown in Fig. R2 of the rebuttal PDF, the steps are as follows:  \n    a) Input the point clouds of an object.\n    b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network.\n    c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework. Where $n$ is the hyper-parameters in this paper.  \nTherefore, in our framework, we do not need to know how many parts each object should be segmented into. We have conducted more comparisons on the segmentation method. Compared with the SOTA point cloud segmentation methods [9][10], the features obtained using PointNet did not significantly affect the results. Therefore, we believe the method remains applicable in real-world scenarios and is not influenced by the choice of the segmentation network.  \n    [9] Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NIPS, 2022.\n    [10] Segment any point cloud sequences by distilling vision foundation models. NIPS, 2024.\n7. **Undefined concepts (Weakness 4)**  \nWe apologize for the lack of sufficient visualization results. The isomorphism loss mentioned in line 189 is a typo, and it should mean the loss function for the IAE module (section 4.2.3). The integration function of line 189 is an averaged function for multiple features followed [2].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGxb5XsCcR", "reviewer": "Reviewer_ZtnM", "summary": "This paper proposes to utilize the part-assembly representation method to mitigate the distribution skew of unseen categories, enhancing the generalization performance for open-set 3D object retrieval. Compared to previous methods, this paper benefits from part-level representation learning rather than object-level representation, obtaining in a good generalization on unseen categories. To utilize the part-level representation, this paper introduces Isomorphic Assembly Embedding (IAE) and the Structured Fuzzy Reconstruction (SFR) modules. The former can generate the assembly embedding isomorphically for each object, and the latter is used for generating the fuzzy representation thus overcoming the distribution skew of unseen categories.", "questions": "The quantitative performance comparisons in Table 2 show the superiority of the proposed method. However, this paper only surpassed the second place by a little bit in some metrics, and there is no sufficient statistical information to prove the significance of the results, such as p-values.", "limitations": "The paper should add discussions on limitations and possibly show some failure cases.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is well-motivated and the solution seems working well. The results are good. The paper also contributes three 3D point cloud datasets with multiple part annotations for benchmarking. Extensive experiments on the three benchmarks demonstrate the superiority of the proposed method over current state-of-the-art 3D object retrieval methods.", "weaknesses": "1. The datasets OP-INTRA and OP-COSEG mentioned in the paper may have limitations in category diversity, number of parts, and dataset size, which may affect the generalization ability of the model.\n2. The framework comprises many sub-architectures, such as the HIConv layer, multiple auto-encoders, fuzzy embeddings, and memory bank, it seems to be relatively complex. However, this paper does not explicitly discuss the computational efficiency of the model, including training and inference time, and computational cost.\n3. Though the paper proposes a solution to the open set problem, the datasets are all virtual. Its generalization ability to unseen categories in real-world applications still needs further verification.\n4. The ablation studies show the effect of the HIConv layer. However, only comparisons with MLP and GIN are performed, but no comparisons with other neural layers such as KAN, nor is the number of HIConv layers ablated.\n5. The experiments are only conducted on the proposed datasets. The generalization ability of the model on a wider data distribution requires more verification. It would be better to add some experiments on previous public datasets or datasets without open-set settings to demonstrate generalization capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer ZtnM**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for us to enhance the rigor and coherence of our paper and directed the focus of our future work.\n\n1. **About the generalization ability of the model (Answer for Weakness 1 and Weakness 3)**:  \nThanks for your valuable suggestion. As an early exploration of open-set learning in a fine-grained way, we believe this paper should focus on exploring the new assembly-based retrieval task and designing a novel open-set learning paradigm based on both inter-object and intra-object correlations. Therefore, we selected the three 3D object datasets with typical geometrical structures for experiments, which is representative of a fine-grained way for an open-set environment. Experimental results demonstrate the necessity and effectiveness of the assembly-based paradigm and framework. However, further improving the generalization ability of our framework and extending it to encompass more intertwined factors in complex open-set environments is one of the key directions for future work. Specifically, we have preliminarily experimented with an adaptive Scale isomorphism Computation method for generalization across datasets and domains, inspired by [1] and [2]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of parts and labels inspired by [3].  \n    [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024.  \n    [2] Zhou J, et al. Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.  \n    [3] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.  \n\n2. **About module-wise computational requirements (Answer for Weakness 2)**:  \nOur experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.\n\n3. **About more ablation studies (Answer for Weakness 4)**:  \nThanks for your valuable suggestion. We conducted more ablation studies, especially on HIConv and other neural layers. Our method employs only a single layer of HIConv, therefore, we explored the impact of more layers of HIConv. Besides, we replaced the HGNN with KAN network [4] in the SFR module. As shown in Tab. R1 of the rebuttal PDF. The full version of our framework yields the best performance, these results indicate the effectiveness of our design in assembly-based open-set 3D object retrieval.\n\n4. **About the public datasets (Answer for Weakness 5)**:  \nThanks for your valuable suggestion. The proposed datasets are constructed based on public datasets without open-set settings, making them suitable for open-set retrieval experiments. As mentioned in Answer 1, we will continue to explore assembly-based retrieval research in our extension version, by constructing more datasets and conducting more experiments.\n\n5. **About the significance of the proposed method (Answer for Questions)**:  \nThanks for your valuable suggestion. We provide statistics of our framework (HAFR) and the second-best method (HGM$^2$R). As shown in Tab. R3 of the rebuttal PDF, statistical information proves the significance of the results.\n\n6. **About failures cases and limitations (Answer for Limitations)**:  \nThanks for your valuable suggestion. We provide some failure cases in Fig. R3 of the rebuttal PDF. In these failure cases, the query objects (rocket, pistol) and the wrong-matched target objects (car, motorbike) share a certain similarity in their part segmentation. Although the significant performance improvement of the HARF framework demonstrates the necessity of assembly-based research in open-set learning, these cornet cases also indicate the necessity of the equilibrium between different levels of labels, which needs the balance of global semantic and local geometry information. This issue is the same as the generalization ability mentioned in Weakness 1. However, this paper focuses more on the fundamental challenge brought by part assembly for open-set retrieval. Therefore, we only consider typical segmentations in this study. As mentioned in Answer 2 above, we are currently conducting research to address these more complex environments. Thank you for your keen observations and academic insights.\n\nThank you again for your valuable suggestions, especially your professional advice on future work in assembly-based open-set learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5aRJhX39O0", "reviewer": "Reviewer_zjgH", "summary": "The manuscript introduces a framework (HAFR) for addressing the challenge of open-set 3D object retrieval. The authors propose a bottom-up approach focusing on part assembly, leveraging both geometric and semantic information of object parts to enhance retrieval performance across categories, including those unseen during training.\n\nThe HAFR framework consists of two main modules: Isomorphic Assembly Embedding (IAE) and Structured Fuzzy Reconstruction (SFR). The IAE module utilizes Hypergraph Isomorphism Convolution (HIConv) and assembly auto-encoders to generate embeddings with geometric-semantic consistency. The SFR module tackles distribution skew in open-set retrieval by constructing a leveraged hypergraph based on local and global correlations and employs a memory bank for fuzzy-aware reconstruction.\n\nThe authors have created three datasets, OP-SHNP, OP-INTRA, and OP-COSEG, to benchmark their approach. Extensive experiments demonstrate the superiority of HAFR over current state-of-the-art methods in open-set 3D object retrieval tasks.", "questions": "The manuscript presents a contribution to the field of 3D object retrieval, particularly in the open-set scenario. The proposed HAFR framework is innovative and has been demonstrated to be effective through rigorous experimentation. However, there are areas where the manuscript could be improved, particularly in terms of computational efficiency, limitations on various parts, and other minor issues. Addressing these points would likely enhance the manuscript's impact and applicability in the field.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a method for open-set 3D object retrieval that cleverly integrates part-level information using hypergraphs, which is a unique and promising direction in the field. The HAFR framework is well-thought-out, with clearly defined modules (IAE and SFR) that address different aspects of the retrieval task, from assembly isomorphism to distribution skew mitigation.\n- The construction of three new datasets with part-level annotations provides a valuable resource for the research community and supports the validation of the proposed method.\n- The methodology is clearly described, and the algorithms are well-structured, making it relatively easy for readers to follow the technical contributions.\n- The paper is well-written and easy to follow.", "weaknesses": "- The paper does not address scenarios with varying numbers of parts per object. Expanding the framework to handle flexibility in the number of parts could improve its applicability.\n- The manuscript could benefit from a discussion on the computational complexity and efficiency of the proposed methods, especially when scaling to larger datasets or higher-dimensional part features.\n- Why not evaluate on the PartNet(https://partnet.cs.stanford.edu/)?\n- Although the paper claims state-of-the-art performance, they do not achieve the best (SDML is the best on OP-COSEG for NDCG metric), what is the reason?\n- Some implementation details, such as network architecture specifics and hyperparameter settings, could be better elaborated to ensure reproducibility.\n- The paper mentions that data and code will be made available upon acceptance, which is good practice. - However, providing this information upfront or during the review process could enhance transparency and reproducibility. For the three datasets, the detailed construction is missing and encourages the authors to publicize the data, facilitating the community.\n- The limitations and failure cases should be discussed comprehensively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Varying numbers of parts (Weakness 1)**  \nHAFR takes 4 part features as input for each object in this manuscript for now. As shown in Fig. R2 of the rebuttal PDF, the steps for part features generation are as follows:  \n    a) Input the point clouds of an object.  \n    b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network.  \n    c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework ( $n=3$ for this paper).  \n    We conducted more experiments with the number of input part features set to 3 ($n=2$) and 5 ($n=4$). As shown in Tab. R1 of the rebuttal PDF, both the 4-part (ours) and 5-part settings show certain performance improvements over the 3-part, indicating that more detailed segmentation provides richer information for assembly-based retrieval. However, our 4-part setting currently exhibits the best performance (better than the 5-part). We can infer that once the number of parts reaches a certain level, the utilization of part information also becomes saturated.\n2. **Computational complexity and efficiency (Weakness 2)**  \nOur experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF. The SFR module occupies a very small parameter space (less than 3\\%) and is directly affected by the size of datasets. The IAE module only consumes up to 34\\%, which determines the effectiveness with high-dimensional features. We believe that our method can remain effective when scaling to larger datasets or higher-dimensional part features.\n3. **PartNet (Weakness 3)**  \nThe ShapeNet Part [1] dataset we used and the PartNet [2] dataset are both fine-grained annotated subsets of ShapeNet, with PartNet having a greater variance in the number of parts per object. As an early exploration of open-set learning in a fine-grained way, we believe this paper should focus on exploring the new assembly-based retrieval task and designing a novel open-set learning paradigm based on both inter-object and intra-object correlations. Therefore, we selected the ShapeNet Part with more evenly distributed parts rather than PartNet, which can reflect the core challenges of a fine-grained approach in an open-set environment. We have preliminarily experimented with an adaptive Scale isomorphism computation method for better generalization, inspired by [3] and [4]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of parts and labels inspired by [5] for the future work.  \n    [1] A scalable active framework for region annotation in 3d shape collections[J]. ACM ToG, 2016.  \n    [2] Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding[C]. CVPR, 2019.  \n    [3] Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024.\n    [4] Uni3D: Exploring Unified 3D Representation at Scale.   ICLR, 2024.  \n    [5] Hypergraph dynamic system[C]. ICLR, 2024.  \n4. **SDML on OP-COSEG (Weakness 4)**  \nSDML proposes a scalable multimodal learning paradigm for retrieval by predefining a common subspace, aiming to minimize the intra-class difference. In the context of assembly-based retrieval, the SDML method projects all parts into a class-related common space to achieve unification. However, this approach results in an overall feature shift, leading to the loss of unique geometrical information of parts and the correlations among them. As shown in Fig. R3 of the rebuttal PDF, we can find that compared to the OP-SHNP and OP-INTRA datasets, the objects in the OP-COSEG dataset exhibit stronger symmetry, leading to less influence of this biased unification. Furthermore, NDCG is a metric that considers the global perspective, thus SDML achieves a slight advantage (0.11\\%) of NDCG on OP-COSEG through this globally biased unification. However, the significantly worse results on both other datasets and other commonly use metric demonstrate the limitation of this method.\n5. **Reproducibility and open access (Weakness 5 and 6)**  \nWe have provided a brief description of implementation details and dataset generation in the Appendix. Besides, we are well prepared and will release the datasets, code, configurations, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We are willing to share our experiences on this (OpenReview) or other open-source platforms.\n5. **Failure cases (Weakness 7)**  \nWe provide some failure cases in Fig. R4 of the rebuttal PDF. In these failure cases, the query objects (rocket, pistol) and the wrong-matched target objects (car, motorbike) share a certain similarity in their part segmentation. Although the significant performance improvement of the HARF framework demonstrates the necessity of assembly-based research in open-set learning, these cornet cases also indicate the necessity of the equilibrium between different levels of labels, which needs the balance of global semantic and local geometry information. This issue is the same as the generalization ability mentioned in Weakness 1. However, this paper focuses more on the fundamental challenge brought by part assembly for open-set retrieval. Therefore, we only consider typical segmentations in this study. As mentioned in Answers 1 and 3 above, we are currently conducting research to address these more complex environments.\n7. **Writing (Limitations)**:  \nWe apologize for the typos and writing issues in this manuscript. We will conduct a thorough review and revision of the entire paper to ensure the clarity and rigor.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3qEmmW82p", "reviewer": "Reviewer_7vQy", "summary": "This paper presents a novel 3D object retrieval method. First, to facilitate this task, the authors build 3 datasets for training and evaluation, which may significantly benefit the community. Then the paper propose the Isomorphic Assembly Embedding (IAE) and the Structured Fuzzy Reconstruction (SFR) modules, which are designed to generate assembly embeddings with geometric-semantic consistency and overcome the distribution skew of unseen categories. Besides, HIConv is proposed to capture high-order correlations within and among objects. Extensive experiments show that the method achieves sota performance.", "questions": "see weakness", "limitations": "The authors didn't discuss the limitations", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "1. This paper builds 3 datasets for the task, which may facilitate future research. \n2. The paper proposes several novel modules to capture the part-level and inter-object features for object retrieval.\n3. The task itself is important in shape understanding.", "weaknesses": "1. No visualization results.\n2. The presentation is hard to understand. There are quite some complex equations, like Eq 2 and Eq 4. Please briefly explain what they mean and how they work.\n3. In Fig. 1, it shows that intra-object features are extracted before inter-category features. But in Fig. 2, I only see Inter-object features? It's hard for me to match them up.\n4. I still don't understand the input. So you need dense point cloud with ground truth 3D part segmentation as input, right? If the segmenation is not perfect, will the method collapse? if the point cloud undergoes SE(3)-transformation, will the method collapse? Can this method handle partial point cloud input, like the point cloud back-projected from depth map?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Visualization (Weakness 1)**\nWe apologize for the lack of sufficient visualization results. We provide some visualized examples of the retrieval results in Fig. R3 of the rebuttal PDF and we will provide more.\n2. **Equations (Weakness 2)**\nWe have revised the expression and explanation of these two equations. Eq 2. is the goal of the open-set retrieval task (minimize the expected risk), which is a widely accepted type for task definition [1-3] in the retrieval field. Eq 4. is the definition of HIConv layer, which followed the commonly used type for graph-based convolution [4-6].  \n\n    [1] View-based 3-D object retrieval[M]. Morgan Kaufmann, 2014.  \n    [2] Hypergraph-based multi-modal representation for open-set 3D object retrieval. TPAMI, 2023.  \n    [3] SHREC’22 track: Open-set 3D object retrieval. 2022.  \n    [4] How powerful are graph neural networks? ICLR, 2019.  \n    [5] Hypergraph neural networks. AAAI, 2019.  \n    [6] Hgnn+: General hypergraph neural networks. IEEE TPAMI, 2022.  \n\n3. **Framework (Weakness 3)**:\nThanks for your valuable suggestion, We apologize for the typos in Figure 2, the \"inter-object\" should be corrected as \"intra-object\". We have restructured the presentation of the proposed framework, which consists of two sequentially connected modules: IAE and SFR.\n    a) The IAE module takes basic part features (as explained in Answer 4) as input. This module employs a structure-aware convolution layer and a set of auto-encoders to achieve assembly fusion of different parts within an object. In this module, the structure-aware convolution layer is implemented by constructing an isomorphism hypergraph and a hypergraph isomorphism convolution function (as explained for Eq. 4 in Answer 2).  \n    b) The SFR module takes assembly embedding for each object as input, utilizing structure-aware feature smoothing and distillation through hypergraph convolution and memory bank reconstruction, respectively. Finally, this module generates the final features (fuzzy embeddings) for similar object matching based on feature distance, thereby enabling retrieval.\n\n4. **Input (Weakness 4)**\nThanks for your valuable suggestion. The inputs for our framework are the part features rather than dense point clouds, and we do not need ground truth 3D part segmentation as input (as described in lines 259-263 and lines 476-478 of the submission). Instead of extracting features from the segmented parts of the point cloud, we use a segmentation network to obtain point-wise features for each point and then average these point-wise features to obtain part features. As shown in Fig. R2 of the rebuttal PDF, the steps are as follows:\n\n    a) Input the point clouds of an object.  \n    b) For each point, obtain its point-wise feature and part labels through a pre-trained point cloud part segmentation network.  \n    c) Select the points belonging to the top-$n$ most frequent part categories and then average the point-wise features with the same top-$n$ part label. Then we calculate the average feature of other points. In this way, we obtain $n+1$ part features for each object as the input for our HAFR framework.  \n\n    We do not need dense point clouds nor the segmentation ground truth for input. As shown in Tab. R1 of rebuttal, compared with the SOTA point cloud segmentation methods [7][8], the features obtained using PointNet did not significantly affect the results. Therefore, we believe our method will not collapse if the segmentation is not perfect. Besides, our framework is a feature-driven method (as described in lines 157-158) and does not directly process raw data (dense point clouds). Thus, its robustness to SE(3) transformations and partial back-projection of point clouds is equivalent to that of the point feature extraction network. These backbone networks have rotation equivariant and adaptability to partial data, as proven in famous works such as [7-11]. Therefore, we believe the method will not collapse when confronted with SE(3) transformations and partial data.  \n    [7] Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NIPS, 2022.  \n    [8] Segment any point cloud sequences by distilling vision foundation models. NIPS, 2024.  \n    [9] Pointnet: Deep learning on point sets for 3d classification and segmentation. CVPR, 2017.  \n    [10] Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NIPS, 2017.  \n    [11] Uni3D: Exploring Unified 3D Representation at Scale. ICLR, 2024.  \n\n4. **Limitations (Limitations)**\nWe have provided a brief discussion in line 330-332. \n\n    a) Experiments of various parts. We only conduct limited experiments on the influence of varying parts in the manuscript. We conducted more experiments with the number of input part features set to 3 ($n=2$) and 5 ($n=4$). As shown in Tab. R1 of the rebuttal PDF, both the 4-part (ours) and 5-part settings show performance improvements over 3-part, indicating that more detailed segmentation provides richer information for assembly-based retrieval. However, our 4-part setting currently exhibits better than 5-part. We can infer that once the number of parts reaches a certain level, the utilization of part information also becomes saturated.\n\n    b) Discussions of societal impacts. As shown in line 158-159 and Appendix C Algorithm 1, the proposed HARF framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data and such as e.g. text, audio, video, and their pieces. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets and code immediately after the anonymous review period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for your insightful feedback and for your valuable time and effort. We try to answer all the questions and weaknesses of each reviewer in the rebuttal section below. The attached PDF contains our additional experimental results and figures.

Meta-review: Congratulations! After reading the rebuttal, the reviewers' concerns were alleviated. Note that the reviewer "7vQy" changed the rating from reject to borderline accept during the discussion (the rating has not been updated in the form). Thus, given support for acceptance from all the reviewers, the AC recommends acceptance. The AC strongly encourages the authors to include the ablation study, clarifications and the rest of the rebuttal/discussion material in the final version of the paper.
Decision: Accept (poster)