Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph
Authors: Yang Xu
Yifan Feng
Jun Zhang
Jun-Hai Yong
Yue Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Hypergraph
3D Object Retrieval
Open-Set Learning
Multi-Modal Fusion
Multi-Label Learning
Primary area: machine_vision

Abstract: Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets, leading to contradictory optimization for superposed categories. In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for this task. Specifically, we propose the Hierarchical Retrace Embedding (HRE) module to overcome the global disequilibrium of unseen categories by fully leveraging the multi-level category information. Besides, tackling the feature overlap and class confusion problem, we perform the Structured Equilibrium Tuning (SET) module to utilize more equilibrial correlations among objects and generalize to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we generate four semi-open 3DOR datasets with multi-level labels for benchmarking. Results demonstrate that the proposed method can effectively generate the hierarchical embeddings of 3D objects and generalize them towards semi-open environments.

Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph
Yang Xu', Yifan Feng', Jun Zhang", Jun-Hai Yong', and Yue Gao BNRist, THUIBCS, KLISS, BLBCI, School of Software, Tsinghua University, China
TTencett AI Lab
(xuyang9610, evanfeng97) com, june www.ejgtennennnt.commm
Banggih.gaoyueettiinnnoo edu. cn
Abstract
Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets, leading to contradictory optimization for superposed categories. In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share partial label space for coarse categories but are completely disjoint from fine categories. We propose the HlyeegraahhBasee Hierarchical Equilibrium Representation (HERT) framework for this task. Specifically, we propose the Hierarchical Retrace Embed- ding (HRE) module overcome the global disequilibrium of unseen categories by fully leveraging the multi-level category information. Besides, tackling the fea- overlap and class confusion problem, perform the Structured Equilibrium Tuning (SET) module to utilize more equilibrial correlations among objects and generalize unseen categories, by constructing superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we generate four semi-open 3DOR datasets with multi-level labels for benchmarking. Results demonstrate that the proposed method can effectively generate the hicrarchical embeddings of 3D objects and generalize them towards semi-open environments.
Introduction
3D objects are of paramount significance, finding extensive applications from computer graphics to security B1] and autonomous robotics (2). As the fundamental task of data acquisition, 3D object retrieval (3DOR) 22 1O plays a pivotal role in the computer vision community (3. 3DOR methods learn to represent 3D objects from the training set and then extract features from query objects to effectively align similar samples. According the category overlap between training and testing sets, existing 3DOR algorithms can be divided into closed-set and open-set types. The former conducts
retrieval for objects whose categories have been seen in the training set [32 36), while the latter
handles objects of unseen categories 7
Existing open-set 3D learning methods are based on the assumption that the labels of object categories are at single level 46 23 In practical scenarios, objects are typically described by multiple hierarchical labels. This leads categories in training and testing sets showing varying degrees of overlap at different levels. As illustrated in Figure [[a] we term this scenario, where the training and testing set share a partial label space coarse categories but are completely disjoint from fine categories, Semi-Oper Environment. Besides, existing methods typically extract basic features using pre-trained models, followed by further open-set learning and optimization. However, for
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training Set
Retricval Set
Implicit Coarse Labels
Recoogslan Cuti Prin
Mincolanarze
Revolttonn
Shapes
Hierarchical Labels
Overlap
Fine Labok
Basic Feature
Pull
Disjoint
Open O Semi Open
Open-Set Learning
Push
Contradictory
Disjoint
Labels
Revdatior
Rectongscar
Afacolbrasa
Fine Labels
Overlap
Seen Categorics (Tmin)
Unseen Categories (Test)
(a) Hierarchical labels semi-open environment.
Optimization contradiction hierarchical labels.
Illustration of motivation from open-set to semi-open 3DOR. Objects may be more accurately described by hierarchical labels than single-level labels in real-world scenarios. In this semi-open setting, the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories.
Figure 1:
categories with the same coarse label but different fine labels, the training of the basic model and the open-set learning will result contradictory optimizations as shown in Figure ((b)
Specifically for 3D object retrieval techniques, only a few methods 7
322 have been explored
for open-set retrieval. These methods consider only the single-layer labels of objects and strictly assume overlap between the training and testing data distributions. The existing open-set 3DOR method (7) treats objects from seen and unseen categories as isomorphic vertices and constructs graph model based global correlations. On one hand, this isomorphic model overlooks hierarchical correlations inherent in multi-level categories, particularly the shared coarse labels. Consequently, the embedding distribution from structure-awane learning tends to be unbalanced towards the feature space of seen categories in the training set. On the other hand, this structure focuses on global correlations between seen and unseen categories, neglecting intricate local correlations within unseen categories themselves. This lack of local attention may result in issues like feature overlap and class confusion within these unseen categories.
Focusing on this practical semi-open environment for retrieval, where the training and testing set share partial label space for coarse categories but are completely disjoint from fine categories, we introduce the semi-open 3D object retrieval task and construct four datasets with multi-level labels to expand the application of 3DOR. We propose the Hypergraph-Band -Based Hicrarchical Equilibrium Representation (HERT) framework for semi-open 3DOR. To overcome the global disequilibrium of unseen categories, propose the Hierarchical Retrace Embedding module (HRE) to achieve balanced representation across multi-level categories. This module generates multi-level retrace embeddings for capturing the hierarchical semantics of objects. To tackle the feature overlap and class confusion problem, we propose the Structured Equilibrium Tuning (SET) module. This module utilizes high-order correlations among objects for unseen category generalization, by constructing superposed Hypergraph based on local coherent and global entangled correlations. In summary, our main contributions are fourfold:
We introduce the semi-open 3D object retrieval task to refine the setting of the 3D ob- ject retrieval task in real-world scenario applications, and we construct four datasets for benchmarking downstream tasks.
We propose the Hypergraph- Based Hierarchical Equilibrium Representation (HERT) frame- work for the semi-open 3DOR task, including the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules, which are designed to overcome the distribution disequilibrium and confusion of unseen categories.
We propose a superposed Hypergraph structure to capture high-order correlations among objects, under the guidance of local coherent correlations and global entangled correlations from hierarchical category information.
Experimental results on the four datasets demonstrate that our method can outperform ofte-offthe- of-the-rrt art retrieval methods towards the semi-open environment.

---

2 Related Work
3D Object Retrieval. 3D object retrieval (3DOR) aims to find the relevant objects from the target set for the query objects, most 3DOR methods construct feature alignment models for 3D objects through metric learning. MMJN (00] proposes a discrimination loss minimize the distance between objects
belonging to the same categories, and learn discriminative embeddings for retrieval. MIFN [17]
constructs fusion networks by weighted concatenation for -ppecit features. PVNet 442
proposes a joint network for the fusion of multi-view and point cloud features. PVRNet 43 proposes an attention mechanism to generate the unified embedding of different modalities. CMCL [14 designs a cross-modia center loss to compress features of different modalities to a invaralinvariant space. However, most existing methods mainly focus on close-set retrieval. Currently, only a few methods 7 32 39 38 37 have explored open-set retrieval, but they typically assume no overlap between training and testing data distributions, which is at odds with the semi-open setting.
Open-Set Learning, Open-set learning (OSL) methods can be roughly separated into two cate- gories [45], i.e., discriminative methods and generative models. In discriminative models, traditional methods achieve the classification of unknown and known categories based on the Support Vector Machine (SV 27 23 or the Extreme Value Theory (EVT)
26 Recently, deep learning methods for OSL. have made remarkable rapid progress. OpenMax 4 is the first algorithm proposed to replace the SoftMax layer and calibrate the output probability with the Weibull distribution. Then, [29] and [41] utilize the one-vs-rest units and reconstructed latent representation for unknown de- tection. Generative OSL models are designed anticipate the distribution of novel classes through
12
training. PROSER 45 allocates placeholders for both data and classifier to detect the unknown classes, C2AE 21 propose a two-step framework to tackle open-set recognition problem with close-set training and open-set training, respectively. Both types of open-set learning methods [9 are based on the assumption that there no overlap between known and unknown data distribution, thus facilitating the design of classifiers for unseen object detection, but are challenging to generalize to practical scenarios that involve hierarchical superposed categories.
Multi-Label Learning. Multi-label learning aims to create a model that can assign multiple labels for each instance simultaneously. Existing multi-label learning methods are mostly designed for
recognition tasks, and based on one-vs-all classifiers 40 24 and embeddings [4]] 18]. As for the
retrieval task, DMSSPH [33 proposes a multi-level preserving hashing network, and AMD GCN [66] designs a cross-modal attention mechanism at each layer for multi-label embeddings. Although these methods construct effective models for image embedding, they overlook the hierarchical relationship and dependencies between different categories. In this paper, we aim to model the correlations between hierarchical label information and generalize to unseen categories.
network for multi-label pattern image retrieval. TranGCN [15 proposes a
3 Problem Setup
3.1 3D Object Retrieval
The goal of the 3D object retrieval (3DOR) is to develop a method using the training set Strn = {(o,, sf, v Ma which is then employed to identify similar objects in the retrieval (testing) set Sret {(o,, if H which is comprised of the query set Sg and the target set S,. Here, L and R represent the number of samples in the training and testing (retrieval) sets, respectively. The expressions of yd 1 and yf E y = -
denote the fine and coarse category labels 3D object o,, respectively. Typically, yf provides the category label from a more general (coarse) level, such as the basic geometric shapes or other common attributes between Strn and Syet. whereas coarse categories
ye provides a semantic-specifie (fine) category label. Generally, the number ye is much smaller than the number of fine categories, which is ye & YJ, For 3D objects, The 0 = {m, M denotes the representation by M modalities, i.e., multi-view, point cloud and voxel.
3.2 Semi-Oper 3D Object Retrieval
In traditional open-set 3DOR, the category spaces of the training set and the retrieval set are not the same fine indicating of E yd (c of E 5z (6/) and y' + $5. y and y denote
3

---

Coarse Labels
Training Categorics
Testing Caregories
Similar Objects
Miscellasooa
Redespala-
(Trainble Panamatie))
Ressce Essading
Coherent
Entangled
Rectieaal
Retrace Embedding
Entungled
Mensey
Bank
Hypenggaph Contulaior
Basic Features
Multi Modal Auto Encoder Am
Retrace Auto Encoder M,
Evro
Superposed Hypergraph
Structare-Aware
Tuning
Hierarchical Retrace Embedding (HRE)
Structured Equilibrium Tuning (SET)
Figure 2: An overview of the proposed Hypergraph framework (HERT) framework for semi-open 3D object retrieval. Our framework is composed of the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules, which are designed for multi-level semantic embedding and hierarchical structure-aware tuning.
Baaerapph-BBedddd Based Hierarchical Equilibrium Representation
the numbers of fine categories in the training set and the retrieval set, respectively. In the practice of semi-open 3DOR, the training set and the testing set share the same space of coarse category,
which yf E y = [ci)k if E y { km and y =v, Y and ý denote the numbers of coarse categories in the training set and the testing set, respectively. During the retrieval phase, each query object is provided with a coarse category label. For better representation, the semi-open 3DOR defines the retrieval model r == (o,1y ) 2, that maps the 3D object og into semantic embedding z; E Rd under coarse category condition yf- The task seeks to minimize the expected risk under the conditional constraints of coarse labels:
(S (IGR * 06. Di)e ((r(adl))) -uoo! -e - ((r(o)be (I)
where S; = (or, vl, 0/() and S₁ (oj, vf. 0f) are object instances sampled from the query set Sy and target set St of and of denote the predicted fine labels, yf and yf are the conditional constraint, which are coarse labels. I(-) is the indicator function, which returns if the expression holds and 0 otherwise. H is the hypothesis space of map r(-l-). D(zi, zz) is the distance metric function between different embeddings.
4 Methodology
4.1 Overall Framework
The overall framework of the Sppergraph-Basee Hicrarchical Equilibrium Representation (HERT) framework illustrated in Figure 2 HERT is composed of the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules. Given 3D objects represented by multiple modalities, common-use used backbones are used to extract the basic features for each modality. Then, the HRE module is introduced to generate the multi-level retrace embeddings of hierarchical semantic information. Next, the SET module, the superposed hypergraph constructed based on the local coherent and global entangled correlations. Finally, the hypergraph convolution and memory bank under the superposed structure are used smooth and distill for feature generalization between seen and unseen categories.
4.2 Hierarchical Retrace Embedding
To obtain fully multi-level embeddings of objects based on hierarchical categories, the HRE module designed here. Specifically, the HRE utilizes two hierarchical auto-encoders in series as shown in Figure 2 The multi-modal auto-encoder Am encodes the multi-modal basic features of 3D objects

---

to get the unified embeddings. The retrace auto-encoder A, encodes the coarse label aligning with the unified embeddings get retrace embeddings.
Given N 3D objects { and feature extractors { JM kk= the basic feature of M modalities
{F* lk=1 = F* (o can be generated, where F E Nxdy As shown in Fig. 2 we first space and generate the unified embeddings from multi-modal auto-encoder Am. Specifically, for the basic features ! of object os, the unified embedding of it be denoted q; - T (Am (U/)) ( where X' denotes the aggregation function among modalities and C E IRd
compress the input basic features from M modalities into the latent modal-invarian
After getting the unified embedding C, the retrace auto-encoder A, takes the coarse category label yf into retrace encoding e;, As shown in Fig. 2 then A, compresses the unified embedding c; aligned with e, into the retracte space S, and does the reverse reconstruction to the mixed space S., which can be defined as follows: V/ = g + y and & :- S - S.. where (-) is the encoder that maps the unified embedding G aligned with e, into the retracte space S,. the retrace embedding ry can be generated by as r4- L (c: et). Rd., (-)) is the decoder that maps the retrace embedding to reconstruction feature m (r:)), m, Rdd..
Through the HRE stage, we got the unified embedding c, retrace embedding r,, and mixed feature rh for each 3D object.
4.3 Structured Equilibrium Tuning
To endow the unified and retrace embeddings of 3D objects with the ability to generalize unseen categories, we introduce the SET module as shown in Figure 2 Specifically, the Superposed Hypergraph structure employed capture the local coherent and global entangled correlations under the constraint of hierarchical category information. Then, hypergraph convolution is employed to utilize the collaborative high-order correlation under the guidance of this superposed Hypergraph. Finally, we use the hypergraph convolution and memory bank to generalize the structure-aware knowledge generate unbiased features for unseen categories.
4.3.1 Superposed Hypergraph
Despite the lack of fine labels in objects from unseen categories, the implicit information within these categories can enhance the generalization capabilities of 3D object embeddings, especially when prompted by coarse labels. To establish the high-order correlations between 3D objects from both seen and unseen categories under common general space, we design a superposed hypergraph structure. The hypergraph in this paper can be represented as G = (V,E), where V and & are the vertex set and the hyperedge set, respectively.
We construct the vertices in our superposed hypergraph by the multi-level retrace embeddings. Specif- ically, we combine the unified embedding c and retrace embedding r; to generate the heterogeneous retrace vertices U; E IR- xde, which can be defined as follows:
0 Ac + (1 (1-A)r;
where À is the papee--arrmeter to trade-off between embeddings of different levels.
Distinct from the traditional hypergraph g we construct superposed hypergraph with two types of conditional hyperedges: coherent hyperedges and entangled hyperedges.
Coherent Hyperedge. The coherent hy hyperedges capture the local coherent correlations in the coarse category space. We define the coherent hyperedge as e &. under the condition of coarse labels:
E = (S.())) of E y*
(3)
where (3 denotes the vertex subset that shares the same coarse label y and y denotes the coarse label space of 3D objects.
Entangled Hyperedge. The entangled hyperedges model the global collaborative correlations among objects between seen and unseen categories. For each vertex, we define its entangled hyperedge as the set of nearest K neighboring vertices. Specifically,
ε, = (DKNN. [
(4)
5

---

where DKNN. (v) denotes the k-nearest neighbors in the feature space of vertex
In this way, construct y coherent hyperedges and N entangled hyperedges, where y is the number of coarse categories and is the number of vertex in the superposed hypergraph. Finally, we combine the sets of hyperedges to obtain complete superposed hypergraph for structure-aware tuning.
& U
(5)
where & and & are the set of coherent and entangled hyperedges, respectively.
4.3.2 Structure-Aware Tuning
To get better generalization unseen categories, hypergraph convolution and memory bank adopted here for feature smoothing and distillation.
For the convenience of computation, the hypergraph H E (0, 1|1Vix E1, where the i-th hyperedge is the i-th column of H, and H(v, e) 1 if the hyperedge e contains the vertex W E RIEX is diagonal matrix, where W denotes the weight of the i-th hyperedge.
be represented by the incidence matrix
To learn the conditional embeddings V E Nxde from multi-level embedding V Nxde under the guidance of hypergraph, the hypergraph convolution (HGNNConv [8] can be represented as:
v D HWD HHWD.'H''. D &
(6)
where D, and D, are the diagonal degree matrices for vertex and hyperedge, respectively. θ E Rduxdu xdu the trainable parameter for the HGNNConv layer (8].
To increase the generalization ability of the SET module, we construct a memory bank M that contains L invariant memory anchors for conditional embedding ú; the 3D object o,, we compute the activation score for each memory anchor in the memory bank by t; Dm (0; az), where aj denotes the anchor and Dm (-:-) denotes the distance metric function. We rebuild the aligned multi- level embedding of each object by 2; Σド= ag 24 E Rdd,, where 1 denotes the normalized values of activation score.
4.4 Training Objective
Loss Function for the HRE. To get better representation of multi-level label information, first use the Homology Loss Lhomo and Bi-reconstrruction Loss Lar followed 7 for Am to leverage the collaborative information across modalities, then we adopt the Retrace Cross-Entropy loss Loe to guide the retrace embedding of coarse category information, which can be defined as follows:
Lee = - - Y
k=1 f log(pt)))
e Arc
where Pi,k D " is the predicted probability score the 3D object o, in k-th coarse category for the reconstructed mixed feature vi4 vf,K is the k-th value the one-hot encoded ground truth coarse label of o;, and y the number of the coarse categories.
In the hierarchical retrace embedding stage, the overall loss function is given:
CHRE = p(Cromo Lor) + (1 - fc
(8)
where μ is the hyper-parameter to trade-off between the loss of multi-modal and multi-level represen- tations.
Loss Function for the SET. To train the hypergraph convolution and learnable memory anchors, adopt Memory Reconstruction Loss Cur and the Cross-entropy Loss Lee
Pmr | -21122
(9)
Lce = Σr k (v..log(eexk) + ve. v1 (.xxlo((p.)))
(10)

---

Table 1: Comparisons of retrieval performance on SO-ESB and SO-NTU dataset.
SO-ESB
SO-NTU
Method
mAPt
Recallf
NDCG
ANMRR
MAPt
Recallf
NDCG1
ANMRR
SDML.
0.4947
0.8027
0.1858
0,5430
0.4384
0.7009
0.1937
0,5764
CMCL
0,4990
0.8154
0.1880
0.5457
0.4440
0.7053
0.1946
0.5721
MMSAE
0.5036
0,8503
0.1931
0.5523
0,4454
0.7046
0.1935
0.5745
TranGCN
0,5063
0,9011
0.1968
0,5408
0.4548
0.7121
0.1961
0,5624
C2AE
0,4809
0.7863
0.1824
0,5501
0.4303
0.6987
0.1915
0,5828
HGMRR
0,5049
0.8831
0.1939
0,5551
0.4821
0.7364
0.2026
0,5438
Ours
0.5756
0.9346
0.2045
0,4874
0.5678
0.8116
0.2251
0,4677
1.0
1.0
1.0
1.0
0.5
0.8
Precision
0.6
Precision
0.6
Precision
0.6
Presision
0.6
0.4
0.4
0.2
2014
0.2
2002
0.0
Trustts
0.0
0.0
0.2
0.4
0.8
1.0
0.2
0.4
1.0
0.4
0.5
0.0
0.1
0.6 Recall
0.8
Recall
Recall
Recall
(a) PR-C on SO-ESB.
(b) CR-C on SO-NTU.
(c) PR-Co SO-MN40.
(d) PR-C SO-ABO.
Figure 3: The Precision-Rede Recall Curves (PR-C) of the proposed method and compared methods the four datasets, respectively.
Pikk
where Pi,k =
Em=r m. and Pi,k 2 f. is the predicted probability score of the 3D object Og in k-th fine category for the multi-level embedding i and memory reconstruction embedding 2;- yE. is the k-th value of the one-hot encoded ground truth fine label of o;, and Y is the number of the coarse categories.
In the structured equilibrium tuning stage, the overall loss function is given by:
LEET = T/Crr + (1 - Lce
(11)
where 1) is the
for trade-off between them.
Experiments
5.1 Dataseta and Evaluation Metrics
Datasets. We generate four semi-open 3DOR datasets, including SO-ESB, SO-NTU, SO-MN40, and SO-ABO, based on the public datasets ESB [33 NTU [3) ModelNet40 B5 and ABO 6) respectively. We add coarse category labels for each object based on the basic geometric shapes such as solid of revolution, rectangular-cubie etc. Also, we remove some objects that are difficult to categorize based on their shapes. Then, we split the fine categories into seen categories for training and unseen categories for testing, the training and testing sets share the same coarse label space according to the semi-open environment setting. Each object has three modalities including multi-view, voxel, and point cloud. Specifically, the detailed descriptions of dataset generation and setting are shown in Appendix B
Evaluation Protocols. As for the evaluation criteria, We employ commonly used retrieval metrics for comparison, including Mean Average Precision (mAP), Recall, Normalized Discounted Cumula- tive Gain (NDCG), Average Normalized Modified Retrieval Rank (ANMRR), and the Precision-Recall Curve (PR-C). For the mAP, Recall, and NDCG metrics, the higher scores better. For the ANMRR metric, the lower score is better.
7

---

Table 2: Comparisons of retrieval performance on SO-MN40 and SO-ABO dataset.
SO-MN40
SO-ABO
Method
mAPt
Recall
NDCG1
ANMRR,
MAPt
Recall
NDCG1
ANMRR,
SDML.
0.5018
0.3241
0.6082
0.5106
0.4380
0.3425
0.4726
0.5564
CMCL.
0.5086
0.3281
0.6128
0.5060
0.4520
0.3657
0,4816
0.5458
MMSAE
0.5189
0.3335
0,6226
0,4938
0,4783
0.3863
0,4929
0.5264
TranGNN
0.5188
0.3358
0.6131
0,4957
0.5175
0.3956
0,5127
0.4801
C2AE
0.4865
0.3152
0,5977
0.5231
0.4669
0,3674
0.4794
0.5313
HGM
0.5779
0.3698
0.6482
0.4407
0.6069
0,4675
0.5463
0.4154
Ours
0.6336
0.3993
0.6874
0.3972
0.6339
0,4793
0.5622
0.3836
Table 3: Ablation studies on SO-ESB and SO-NTU dataset.
SO-ESB
SO-NTU
Method
mAPt
Recall
NDCG1
ANMRR
mAP
Recall
NDCG1
ANMRR,
HRE w/o ReFnz
0.5159
0.9086
0.1953
0.5431
0.4913
0.7534
0.2053
0.5355
HRE w/o Lee
0.5133
0.8738
0.1934
0.5365
0.5161
0.7902
0.2162
0.5162
SET w/o & GCN-based SET
0.5358
0.8957
0.1975
0.5184
0.5285
0.7898
0.2184
0,4986
0.5405
0.8999
0.2003
0.5192
0.5144
0.7703
0.2140
0.5138
MI.P-hased SET
0.5014
0.8483
0.1930
0.5476
0.4689
0.7304
0.2023
0.5561
HRE+SET
0.5756
0.9346
0.2045
0.4874
0.5678
0.8116
0,2251
0.4677
5.2 Experimental Settings
Implemental Details. our experiments, we choose three modes of multi-view (12 views), point cloud (1024 points), and voxel (32 dimensions) as the representation of 3D objects. The basic features for framework input are extracted by MVCNN [30.. PointNet [25]. and 3D ShapeNets [35, respectively. The HRE and SET modules trained separately with 40 and 120 epochs. The SGD optimizers are used for both two modules with learning rates of 0.1 and 0.001, respectively. As for the hyper-parameters in HERT, set 0.5. μ 0.8, and y 0.9. Detailed implemental settings for our framework are provided in Appendix C
Compared Methods. Under this semi-open setting, since there is no 3D object retrieval method designed specifically for this muli-level settings, we choose the current state-of-the-an methods of close-set 3D retrieval (SDML. [1 CMCL [14], MMSAE 34 ) close-set multi-label retrieval(Trann
GCN (15),, and open-set 3D learning (C2AE (21), HGM (7]). For each method, we add
multi-label learning mechanism [15] on their basis for comparison. We provide more implemented
details of compared methods Appendix D
5.3 Comparison with the State-of-the-Arts
To evaluate the effectiveness of the proposed HERT framework, we conduct experiments on SO-ESB, SO-NTU, SO-MN40, and SO-ABO datasets. The comparison of quantitative results is presented in Table [ and Table respectively. Results demonstrate that the proposed HERT outperforms the state-of-the-ar methods on all four datasets. Especially on the SO-NTU and SO-MN40 datasets, our method achieves mAP of 0..567/00665 /0.6336 with about 17. 17.7% 99.6% improvements compared with the second-ber best method, and achieves Recall 0.8116/00.33 /0.399 with about (0.2%/77.9 improvements compared with the second-best method. Besides, also provide the qualitative results through precision-recall curves as shown Figure in which the larger area below the curve indicates better performance.
The better results indicate that the proposed HERT framework has the capability understand and generalize unseen fine categories under the guidance of coarse labels. The proposed HRE and SET modules can fully leverage the hierarchical category information into multi-level retrace embedding and generalize them to unseen categories. This approach better captures the hierarchical semantic correlations in the wild and provides practical framework for the representation learning of multi-label tasks in semi-open environments. We provide more visualized results in Appendix

---

Table 4: Ablation studies on SO-MN40 and SO-ABO dataset.
SO-MN40
SO-ABO
Method
mAPt
Recall
NDCG
ANMRR
mAP
Recall
NDCG
ANMRR !
HRE w/o ReEnz
0.5791
0.3710
0.6479
0.4410
0.6055
0.4523
0.5535
0.4062
HRE w/o Liee 0.5967
0.3783
0.6756
0.4309
0.5885
0.4269
0,5413
0.4230
SET w/o & 0,5913
0.3757
0.6669
0.4347
0.6006
0.4263
0,5494
0.4132
GCN-based SET
0,5602
0,3573
0.6410
0.4628
0,5686
0.4253
0,5314
0.4415
MLP-based SET
0.5088
0.3290
0.6149
0.5073
0.4880
0.3787
0.5023
0,5159
HRE+SET
0.6336
0.3993
0.6874
0.3972
0.6339
0.4793
0.5622
0.3836
1.0
1.0
1.0
1.0
0.8
0.8
0.3
Precision
0.6
Precision
0.6
Precision
0.0
Precision
0.4
94
0.4
0.4
0.2
000
0.0
SET
0.0
201
0.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
Recall
Recall
Recall
Recall
(a) PR-C on SO-ESB.
(b) PR-C on SO-NTU.
(c) PR-C on SO-MN40.
(d) PR-C on SO-ABO.
Figure 4: The Precision-Recht Curves (PR-C) of the ablation studies on four datasets, respectively.
5.4 Ablation Study
We conduct ablation experiments on each module of HERT to demonstrate their effectiveness. First, we remove the retrace encoding e (HRE w/o ReEnz) and cross-entropy loss in the HRE module (HRE w/o Lee), where "w/o" denotes "without". This is equivalent to using naive hierarchical embedding approach for coarse labels. As shown in Table 3 4 and Figure 4 the proposed retrace embedding approach in the HRE module achieves an mAP improvement of 9.4% 6.22% in SO-MN40 dataset and 7.8%/4.7% 7.7% in SO-ABO dataset. These results effectively demonstrate the effectiveness of the HRE module for hierarchical categories.
As for the SET module, remove the proposed coherent hyperedges (SET w/o &.) for comparison, also we replace the hypergraph convolution with Graph Convolutional Networks (GCN-based SET) and Multilayer Perceptron (MLP-based SET). Ablation results in Table 3 4 and Figure 4 show that the proposed SET outperforms all other structure learning methods, and the combination of the HRE and the SET yields the best performance. These results demonstrate the proposed SET can effectively utilize the semi-superposed correlations among objects.
6 Conclusion
In this paper. we introduce more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share partial label space for coarse categories but are completely disjoint from fine categories. We propose the Hypergraph- Based Hierarchical Equilibrium Representation (HERT) framework for semi-open 3D object retrieval. Specifically, to overcome the global disequilibrium of unseen categories, we propose the Hierarchical Retrace Embedding (HRE) module fully leverage the multi-level category information. Besides, we perform the Structured Equilibrium Tuning (SET) module to tackle the feature overlap and class confusion problem. This module utilizes more equilibrial correlations among objects and generalizes to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, category labels for semi-open 3DOR tasks, i.e., SO-ESB, SO-NTU, SO-MN40, and SO-ABO, Results demonstrate that the proposed method can effectively generate and generalize the hierarchical embeddings of 3D objects in semi-open environments. However, due to dataset limitations, we are currently unable to verify the balanced representation effect on more than three levels of labels, which is one of our future research directions. We believe this paper can provide new insights for future research in more practical scenarios of open-set learning.
construct four 3D object datasets with multi-level

---

7 Acknowledgement
This work was supported by Beijing Natural Science Foundation (No. L.242167), CCF- Open Research Fund, and Jiangxi Provincial Natural Science Foundation (20224ACB218002))
References
Ferran Argelaguct and Carlos Andujar. A survey of 3d object selection techniques for virtual environments. Computers Graphics, 1713)122 2013.
[2] Eduardo Amold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis. A survey on 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportation Systems, 20(10):3782-3755 2019.
[3] Song Bai, Peng Tang, Philip HS Torr, and Longin Jan Latecki. Re-ranking via metric fusion for object re- trieval and person re-identification. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 740-749, 2019.
[4] Abhijit Bendale and Terrance Boult. Towards open set deep networks. In /EBE/CVF Conference on Computer Vision and Pattern Recognition, pages 1563 1572, 2016.
[5] Ding- Yun Chen, Xiao Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. Computer graphics forum, 22(3):223 232, 2003.
Jasmine Collins, Shubham Goel, Kenan Deng. Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas Yago Vicente, Thomas Dideriksen, Himanshu Arora, Abo: Dataset and benchmarks for real-world 3d object understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 126 21136, 2022.
[7] Yifan Feng, Shuyi Ji, Yu-Shen Liu, Shaoyi Du Qionghai Dai, and Yue Gao. Hypergraph- based multi- modal representation for open-set 3d object retrieval. IEEE Transactions on Puttern Analysis and Machine Intelligence, (66(4):2206-22)) 2023.
[8] Yue Gao, Yifan Feng. Shuyi Ji, and Rongrong Ji. Hgan+: General hypergraph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3181 3199, 2022.
[9] Zong Yuan Ge. Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-elass open set classification. arXiv preprint arXiv: 1707.07777. 2017.
[10] Alexander Grabner, Peter M Roth, and Vincent Lepetit. 3d pose estimation and 3d model retrieval for objects the wild. JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3022-3031, 2018.
[11] Peng Hu, I Liangli Zhen, Dezhong Peng, and Liu. Scalable deep multimodal learning for cross-modal retrieval. In Annual Conference of the Association for Computing Machinery Special Interest Group in Information Retrieval, pages 635 644, 2019.
[12] Lalit P Jain, Walter J Scheirer, and Terrance E Boult. Multi-class open set recognition using probability of inclusion. In European Conference on Computer Vision, pages 393-409. Springer, 2014.
[13] Subramaniam Jayanti, Yagnanarayanan Kalyanaraman, Natraj Iyer, and Karthik Ramani. Developing an engineering shape benchmark for cad models. Computer-Aided Design, 38(9):939-95 953,3, 2006.
[14] Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian, Cross-modal center loss for 3d cross- modal retrioval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3142-3151, 2021.
[15] Ying Li. Chunming Guan, Rui Cai, Ye Erwan, Ding Yuxiang, and Jiaquan Gao. Tran-gen: Multi- lahel pattern image retrieval via transformer driven graph convolutional network. In ACM International Conference on Multimedia, pages 6301-6310, 2023.
[16] Ying Li, Hongwei Zhou, Yeyu Yin, and Jiaquan Gao. Multi-label pattern image retrieval via attention mechanism driven graph convolutional network. In ACM International Conference on Multimedia, pages 300-308, 2021.
[17] Qi Liang, Mengmeng Xiao, and Dan Song. 3d shape recognition based on multi modal information fusion. Multimedia Tools and Applications. 80:16173- 16184, 2021.
10

---

[18] Weiwei Liu, Donna Xu. Ivor W Tsang, and Wenjie Zhang. Metric learning for multi-output tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 41(2):408 422, 2018.
[19] Lawrence Neal, Matthew Olson, Xiaoli Femn Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. European Conference on Computer Vision, pages 613-628, 2018.
[20] Weizhi Nie, Qi Liang, An-An Liu, Zhendong Mao, and Yangyang Li. Mmjn: Multi-modal joint networks for 3d shape recognition. In ACM International Conference on Multimedia, pages 908-916, 2019.
[21] Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2019.
[22] Panagiotis Papadakis, Ioannis Pratikakis, Theoharis Theoharis, and Stavros Perantonis. Panorama: A 3d shape descriptor based on panoramic views for unsupervised 3d object retrieval, International Journal of Computer Vision, 89:177-192, 2010.
[23] Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Open- machine learning: applications, challenges, and opportunities. ACM Computing Surveys, 55(10):1-37, 2023.
[24] Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In The International Conference of World Wide Web, pages 993-1002, 2018.
[25] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[26] Ethan M Rudd, Lalit Jain, Walter J Scheirer, and Terrance E Boult. The extreme value machine. IEEE Transactions on Puttern Analysis and Machine Intelligence, 40(3):762-768, 2017.
[27] Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 35(7):1757-1772, 2012.
[28] Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(11):2317 2324, 2014.
[29] Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. arXiv preprint 1eXiv:1709.087/6 2017.
[30] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 945-953, 2015.
[31] Jian Wang, Fan Li, Xuchong Zhang, and Hongbin Sun. Adversarial ohstacle generation against lidar-based 3d object detection. IEEF Transactions on Multimedia, 2023.
[32] Shijie Wang, Jianlong Chang, Haojie Li, Zhihui Wang, Wanli Ouyang, and Tian. Open-set fine-grained retrieval via prompting vision- -anguage evaluator. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19381 19391, 2023.
[33] Dayan Wu, Zheng Lin, Bo Li. Mingzhen Ye, and Weiping Wang. Deep supervised hashing for multi-label and large-scale image retrieval. In International Conference on Multimedia Retrieval, pages 150-158. 2017.
[34] Yiling Wu, Shuhui Wang, and Qingming Huang. Multi-modal semantic autoencoder for cross-modal retrieval. Nearccopputing 331:165-175, 2019.
[35] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1912-1991 1920, 2015.
[36] Zhi Xiong, Dayan Wu, Wen Gu, Haisu Zhang, Bo Li, and Weiping Wang. Deep discrete attention guided hashing for face image retrieval. In International Conference on Multimedia Retrieval, pages 136-144, 2020.
[37] Yang Xu, Yifan Feng. and Lin Bie. Triadic elastic structure representation for open-set incremental 3d object retrieval. In ACM International Conference Multimedia Retrieval, pages 20 10-28, 2024.
11

---

Coarse Labels
Rectangular-cuble prism
Solids of revolution
Miscellaneous
Point Cloud
Voxel
Multi- Multi-Viey View
Desk
Sofa
Bed
Bottle
Vase
Plant
Toilet
Labels
Figure 5: Visualizations of the hierarchical labels and multi-modal representations of 3D objects in the SO-MN40 datasets.
[38] Yang Xu, Yifan Feng, and Yue Gao. Negative prompt driven complementary parallel representation for open-world 3d object retrieval. In International Joint Conference on Artificial Intelligence, pages 1498-1506, 2024.
[39] Yang Xu, Yifan Feng, and Yu Jiang. Structure-aware residual-center representation for self-supervised open-set 3d cross-modal retrieval. In IEEE International Conference on Multimedia and Expo, pages 1-6, 2024.
[40] EH Yen, Xiangru Huang. Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. Ppdsparse: parallel primal-dual sparse method for extreme classification. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 453-553,44 2017.
[41] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Nacmura. classfiiation-rrconstttiiii learning for open-set recognition. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4016-4025, 2019.
[42] Нaoxuan You, Yifan Feng. Rongrong Ji, and Yue Gao. Pvnet: joint convolutional network of point cloud and multi-view 3d shape recognition. In ACM International Conference on Multimedia, pages 1310-1318, 2018.
[43] Haoxuan You, Yifan Feng, Xibin Zhao, Changqing Zou, Rongrong Ji, and Yue Gao. Pymnt: Point-view relation neural network for 3d shape recognition. In AAAI Conference Artificial Intelligence, pages 9119- 9126, 2019.
[44] Hsiang-Fu Fu Yu, Prateck Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International Conference Machine Learning, pages 593-601, PMLR, 2014.
[45] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2021.
[46] Zhi-Hua Zhou. Open-environmmnt machine learning. National Science Review, 98) 2022.
A
Semi-Open Environment
practical scenarios, the categories of objects can be labeled from various perspectives such appearance and functionality, or from different levels of granularity, which means hierarchical labels. For example, consider a 3D object, which can simultaneously be labeled as "sphere" and "blue"a at the basic geometric and color level. Moving up the semantic hierarchy, the same object may be labeled as "transportation. "car". and "red""o indicate
12

---

Table 5: The statistics of the semi-open 3DOR datasets.
SO-ESB
SO-NTU
SO-MN40
SO-ABO
Coarse
3
3
3
3
Fine
41
67
40
21
Categories
Seen
17
13
8
4
Unseen
24
54
32
17
Training
98
378
2821
1082
Retrieval
457
1232
7591
4432
Number
Query
96
216
128
68
Target
361
1016
7463
4364
its coarse semantic, fine semantic, and color. Further up the hierarchy, labels could include descriptors related to the object's specific function detailed features. For instance, 3D architectural model might labeled as "residential". "multi-story building". and "with garage" tto convey functional and structural attributes. These hierarchical labels not only enhance the descriptive richness of 3D object representations but also play a crucial role in applications spanning computer graphics. virtual reality, machine learning, and game development. They enable more comprchensive understanding of an object's characteristies and context, ultimately enhancing its usability and interpretability across diverse domains.
Most open-set learning methods based the assumption that the labels of object categorics are a single level. This setting implics that cxisting methods consider category visibility only between the training and testing sets at a single layer, ignoring the distribution overlap among categories at other layers. Besides, the existing strict setting of the single-level open-set also brings more deviation bias to the feature space of unseen categories. In this paper, do not follow the strict assumption the existing open-set approach regarding the disjoint distribution between training and testing sets, instead leveraging the different overlaps across multi-leve categories to enhance representation and retrieval performance. we term this scenario, where the training and testing set share partial label space of coarse categories but are completely disjoint from fine categories, as Semi-Open Environment.
B Semi-Open Dataset Generation
Considering that there is no existing dataset for semi-open 3DOR task, we generate four semi-open 3DOR datasets, including SO-ESB, SO-NTU, SO-MN40, and SO-ABO, based the public datasets ESB [13] NTU [5). ModelNet40 35). and ABO (6), respectively. to the basic geometric shape and remove some objects that are difficult to categorize based on their shapes. Thus, the number of 3D objects in the released datasets is smaller than that of the original datasets. Specifically, the coarse labels for the four datasets are:
add coarse category labels for each object according
SO-ESB: Flat-thin wall components, Rectangular prism, Solids of revolution.
SO-NTU: Rectangular-cubic prism, Solids of revolution, Miscellaneous shape.
SO-MN40: Rectangular- prism, Solids of revolution, Miscellancous shape.
SO-ABO: Rectangular- prism, Solids of revolution, Miscellancous shape.
Then, split the fine categories into seen categories for training and unseen categories for testing, the training and lesting sets share the coarse lahel space according the semi-open environment setting. The staties of the four semi-open 3DOR datasets are shown Table Each object has three modalities including multi-view, voxel, and point cloud. The multi-view data are rendered by Blender to get 12 images for each object. The point number for point cloud is 1024, and the dimension for voxel data is 32 x 32 32, Point cloud and voxel data sampled by Open31 0. 13. The illustration of hierarchical labels and multiple modalities of the semi-open 3DOR datascts shown in Figure
C Method Details
The overall framework of HERT is composed of the Hierarchical Retrace Embedding (HRE) and the Conditional Structure Learning (SET) modules. Given 3D objects represented by multiple modalities, common
httpp:///ww. httpp:://www.bbeedrr www.:e/wbbeded....m .og
https://wwe www.wwopend..oom org
13

---

Algorithm Training the HRE module
Input: Basic features ()") )==1 and their coarse labels { of N instances {o:1--1- Parameter: f = 0.5.
Output I: Unified embeddings { Output Retrace embeddings {fr LN
1: Let epoch = 0:
2: Initialize multi-modal auto-encoder Ami
Initialize aggregation function T:
4: Initialize retrace auto-encoder A, { }
5: Encoding coarse labels as retrace encoding {er
6: while epoch < 40 do
7: Get unified embedding for each modality (c) 9. c2) Am (f.
8: Get unified embedding for each object {ci / TT(c 1
(())))
9:
Calculate the Homology Loss Lhomo = Chomo (1c) (()))))
10:
Calculate the Bi-reconstruction Loss Lor = ((c))=))
11:
Get retrace embedding of each object {r; = (fc + e (mm))
12:
Get mixed feature each object { =
13:
Calculate the Retrace Cross-Entropy Loss Lce = Loo ( (=m))
14:
Calculate loss for the HRE module LHRE pa [Chomo Lor) (1 p.)
15:
if CHRE does not converges then
16:
Update parameters of Am. T, and A, by LHRE-
17:
epoch+
18:
else
19:
Break.
20:
end if
21:
end while
return Unified embeddings {c.) and retrace embeddings {ri)
backbones are used to extract the basic features for each modality, Specifically, the basic features of multi-view, point cloud, and voxel are extracted by the pre-trained MVCNN 330, PointNet (25). and 3D ShapeNets (35), respectively.
The implemental details of the HRE module are provided in Algorithm I the HRE utilizes two hierarchical auto-encoders in series. The multi-modal auto-encoder Am encodes the multi-modal basic feature of 3D objects get unified embeddings. The retrace auto-encoder A. encodes the coarse label aligning with the unified embeddings to get retrace embeddings.
As shown in Algorithm 2 we provide the implemental details of the SET module, the superposed hypergraph structure G is employed capture the local coherent and global entangled correlations under the constraint of hierarchical category in- formation. Then, hypergraph convolution employed utilize the collaborative high-order correlation under the guidance of this superposed hypergraph. Finally, we use the hypergraph convolution (HGN- NConv) and memory bank M to general- ize the structure-aware knowledge to gen- erate unbiased features for unseen cate- gories.
Table 6: The hyper-parameters of the HERT framework.
HRE
SET
Optimizer
SGD
SGD
Learning Rate
0.1
0.001
Momentum
0.9
0.9
Weight Decay
0.1
0
Scheduler
Cosine Annealing 40
Cosine Annealing
Tmax
60
ctamin
0.00001
0.00001
Max Epoches
40
120
Our experiments were conducted on a Tesla V100-32G GPU and an Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. Besides, We provide more SET module set to 12, 10, 50, and 50 for SO-ESB, SO-NTU, SO-MN40, and SO-ABO, respectively.
of HERT in T The hyper-parameter "k" in the
14

---

Algorithm 2 Training the SET module
Unified embeddings {cih- retrace embeddings {ft)= and coarse labels {o of N instances {or [o=)==1- Parameter: X = 0.8, 0.9. Output: Final embeddings {
Input:
1: Let epoch
2: Initialize superposed hypergraph 9 (V, W &};
3: Construct vertices V = (rr /. (Ac (1 (1-)) =
4: Construct coherent hyperedge & = (B) oe E }
5: Construct entangled hyperedge & (DKNN, (v) lue V);
6: Construct hyperedges & = Ug;
7: Initialize weight diagonal matrix W of g;
8: Calculate diagonal degree matrices D, and D.:
9: Calculate incidence matrix H of g;
10: Initialize HGNNConv parameters O of g;
11: Construct memory bank M;
12: while epoch < 120 do
13: Get { = (D. H. 1.V ii=1 θ);
14: Calculate the Cross-Entte Eotoso- Loss Lce = Lee (1 [ (1)))=))
Rebuild {& ] to get { by M:
Calculate the Momery Reconstruction Loss Pmr Lmr (fū;
17:
Calculate loss for the SET module LSET = nLmr + - Cce
if LSET does converges then
19:
Update parameters of O and M by LSET.
epoch+
21: else
22: Break.
23: end if
24: end while
25: return Final embeddings (
26: Retrieval by (6:)-11-
D Comparison Implement
Under this semi-open setting, since there is no 3D object retrieval method designed specifically for this muli-level settings, we choose the current state-of-the-arr methods of close-set 3D retrieval (SDML M.I. CMCL [14), MM- SAE (3)). close-set multi-label retrieval(Tran- GCN (15)). and open-set 3D learning (C2AE (21). HGM (7). For each method, add multi-label learning mechanism [15] on their basis for comparison. The coarse embeddings are supervised by the coarse labels by cross-entropy loss.
SDML MI: SDML is a metric learning based method for multi-modal retrieval, which learns projection functions for different modalities independently. We implement the multi-label embedding by adding auto-encoder after the encoder of the DSAE module.
CMCL [14] CMCL is a typical 3DOR network based on cross modal center loss, which is designed to eliminate cross-modal discrepancy. We construct an auto-encoder and take the center representation of each object to generate the coarse embeddings.
MMSAE 34 MMSAE is an auto-encoder based method that compresses features from different modalities into unified space. We take the semantic code vector of the MMSAE network as the input and generate coarse embeddings by another auto-encoder.
TranGCN [15] TranGCN is multi-label retrieval method based on a graph convolutional network. We treat the fine and coarse lahels as the two labels supervision.
C2AE (2:: C2AE is open-set 3D object recognition method with two-step structure, which is designed to identify objects of the generate the coarse embeddings after first step.
categories. We add auto-encoder
HGM²R is hypergraph based 3DOR method specifically designed for the open-set environment, which assumes distribution overlap between training and testing set. We construct an
HGM'R 7
15

---

Baclangula-cubic
rade
mome
3600
bend
(a) Retrace embeddings.
(b) Final embeddings.
Figure The t-SNE visualization of the embeddings from unseen categories in the OS-MN40 dataset.
auto-encoder for coarse embedding after the MM3DOE module, and construct only the knn-based hyperedges in the SAIKL module.
E
Performance Result
The HRE module designed to obtain multi-level embeddings objects based on hicrarchical categories. Therefore, we provide two visualized results of the retrace embeddings and final conditional embeddings of unseen categories in Figure 6(a)aand and Figure 6(b] respectively. These two t-SNE visualizations show that the proposed HRE and SET modules can fully leverage the hierarchical category information into multi-level retrace embedding and generalize them correlations in wild and provides practical framework the representation learning of multi-label tasks towards a semi-open environment.
categories. This approach better captures the hierarchical semantic
NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We provide comprehensive motivations and detailed introductions of all the contributions in the abstract and introduction.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the current limitations of this work in the conclusion.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does paper provide the full set assumptions complete (and correct) proof?
Answer: [Yes]
Justification: We provided formalized, step-by- by-step explanation of each procedure in the methodol- ogy.
16

---

4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide detailed experiment and reproduction settings in the experiments and appendix.
Justification:
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [No]
Justification: We in the preparation of the codes, models, and datasets, and they will be released the earliest opportunity.
6. Experimental Setting/Detan Details
Question: Does the paper specify the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: We provide experimental settings details settings the experiments and appendix.
7. Experiment Statistical Significance
Question: Does paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We explain the evaluation protocols of the experiments.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide the sufficient information on the computer resources in appendix.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, every respect, with the NeurlPS Code
of Ethics ptctps://neuripee: http:///eiirrp...cpppe//////eele cttp:///ece///iiccccc.........eo
Answer: [Yes]
Justification:
We have carefully read and adhered to the NeurIPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts of this work in the conclusion.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper poses such risks.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification; We have obtained and adhered to permissions the datasets used have cited them the paper.
13. New Assets
17

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the dataset and provide the corresponding documentation immediately upon the paper's acceptance.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "d1sfMwLjrY", "reviewer": "Reviewer_nF63", "summary": "This paper introduces a Semi-Open Environment setting for open-set 3D object retrieval, addressing the limitation of existing methods that only consider single-layer labels and assume no overlap between training and testing sets. The authors propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework, which includes the Hierarchical Retrace Embedding (HRE) module to balance representations across multi-level categories and the Structured Equilibrium Tuning (SET) module to handle feature overlap and class confusion through high-order correlations in a superposed hypergraph. They also create four semi-open 3D object retrieval datasets with hierarchical labels to benchmark their approach. Experimental results show that their method effectively generates and generalizes hierarchical embeddings of 3D objects, outperforming current state-of-the-art retrieval methods in semi-open environments.", "questions": "please address the concerns raised above. \n\nTo be specific, I'd like to see a total 25 randomly sampled hierarchy examples from the proposed SO-ESB, SO-NTU, SO-MN40, and SO-ABO.", "limitations": "The current version discussed limitations in #line 311~314.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I generally appreciate the study angle around the fine-grained structure across different 3D object categories, which could bring more insights for related 3D research. Overall, the proposed architecture is composed of reasonable components along with reasonable loss functions. \n\nMotivating by potential contradict optimization for open-set learning, they proposed semi-open 3D object retrieval task. To examine the performance, they also design 4 datasets based on existing 3D object datasets. The proposed framework outperforms all the aselines.", "weaknesses": "I have several major questions and concerns:\n\n- What does the exact coarse labels mean by author in Figure 1? I failed to see the relationship among solid of revolution, rectangular-cubic, and helicopters. I doubt the meaningful of the proposed multi-levels of 3D objects: it is too coarse to be a intermediate level. Can the author show several qualitative visualization around several 3D objects which share different fine-grained levels but with the same coarse level? I'd like to check the results with randomly sampled from SO-ESB, SO-NTU, SO-MN40, and SO-ABO, respectively. Generally, for each dataset, randomly sampled 5 sets of objects with the different fine-grained-level labels while the same coarse-level label would be very helpful. \n- The proposed HRET framework is a bit ad-hoc: consider if we have multiple levels (more than 2 levels) shared across a large amount of 3D objects, we will need more levels of auto-encoders given the current design logic.\n- As mentioned in line 448-449, some of the 3D objects will be throwed away due to the improper design of coarse labels. I failed to this the use of basic geometric shape to fit nowadays 3D vision research. For example, Objaverse(-xl) have a lot of comprehensive objects consisting of multiple different basic geometric shape. Those \"complex\" object may lie in the interest for us to perform retrieve tasks. \n\n---\nHere are some minor points:\n- How do you determine the corase label if one object actually contains multiple separate simple shape?\n- 3DOR is first mentioned in #line15 without full form. This is unfriendly to readers who are not familiar with this direction. \n- In Figure 6 (b), there are some overlap fine-grained objects. Could you please show some? For example, in the left brown ones, they are mixed with yellow and light-purple dots; in the top pink group, there is a green dot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer nF63**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.\n\n1. **About coarse labels and datasets (Answer for Weakness Major 1, 3, Minor Q1, and the Questions)**:  \nThe coarse labels in Figure 1 mean the basic shape of the object as a whole. As shown in Fig. R1 of the rebuttal PDF, we provide examples of four datasets, each with three coarse classes and five fine classes. We annotate the coarse labels according to the geometry-based shape of each object as a whole (\\#line 242-243, \\#line 447-449), while ignoring the part assembly relationships within an object in this paper. This paper is an early exploration of semi-open learning. Therefore, we selected two typical levels of labels based on different criteria: geometry-based coarse shape category and semantic-based fine category, which is representative of a semi-open environment. As shown in the right side of Fig. R1, the objects that were removed during dataset construction are those with multiple separated parts and cannot be considered as a whole. We believe that the use of these corner-case samples would intertwine other issues such as graphics and foundation models, and would not reflect the key problem of contradicted hierarchical labels in semi-open learning. Therefore, we decided to exclude them from this early exploration and will address these more complex cases with multiple levels of labels in future work. Additionally, we provide the split of some datasets in Answer 5-7.\n\n2. **About multiple levels (Answer for Weakness Major Q2)**:  \nAs an early exploration of semi-open learning, we believe this paper should focus on exploring the new semi-open learning task and designing a novel collaborative learning paradigm based on hierarchical correlations. Based on this paradigm, the HERT framework is implemented with two layers temporarily in this paper, aiming to use two of the most representative layers to validate the necessity and performance of machine learning research in typical semi-open environments. However, one of our future work directions is to extend the HERT framework to encompass more intertwined factors in complex semi-open environments. This involves addressing challenges such as domain adaptation associated with multiple levels while balancing complexity, efficiency, and performance. Specifically, we have preliminarily experimented with a hypergraph-based isomorphism computation method to address the increase in parameters brought by higher levels, inspired by [1]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of labels at each layer inspired by [2].  \n    [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2023.  \n    [2] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.\n\n3. **About failure cases (Answer for Weakness Minor Q3)**:  \nWe provide the visualization of failure cases in Fig. R3 of the rebuttal PDF. In these failure cases, the query objects (bench, TV stand, bookshelf) and the wrong-matched target objects (mantel, laptop) share a certain similarity in their shapes and belong to the same coarse category (Rectangular-Cubic Prism). Although the significant performance improvement of the HERT framework demonstrates the necessity of research in semi-open learning and the effectiveness of our method, these corner cases also indicate the necessity of utilizing finer-level information such as the part-assembly of objects. This issue is the same as the multiple levels issue mentioned in Weakness Major Q2. However, this paper focuses more on the fundamental differences brought by semi-open hierarchical labels. Therefore, we only consider two layers of labels in this study. As mentioned in the Answer 2 above, we are currently conducting research to address these more complex environments. Thank you for your keen observations and academic insights.\n\n4. **About the writing (For Weakness Minor Q2)**:  \nThanks for your thorough review and suggestions. We will conduct a comprehensive review of the entire paper, especially focusing on the use of abbreviations.\n\n5. **Splits of SO-NTU**:  \n5.1 Train  \nRectangular-Cubic Prism: headstone,table square  \nSolids of Revolution:ball,ballon,cannon,watch  \nMiscellaneous: book,plant with pot,cold weapon stick,frame,gun pistol,plane delta wing,plant leaf  \n5.2 Test  \nRectangular-Cubic Prism: Bed,truck,chair common,sofa,man,chair,computer,container,tank  \nSolids of Revolution: fish,bottle,knife,cup,helmet,hydrant,insect fly,insect polypod,missle,orchestral,pen,plane backswept wing,plane forwardswept wing,tree,weed,ring,table round,hammer,screwdriver,wheel,zeppelin  \nMiscellaneous: dinosaur,dog,duck,tetrapods,bird,car common,chair swivel,chess,chip,clock,cold weapon long,sword,cycle bike,cycle moto,door,gun musket,gun submachine,human stand,floorlamp,table lamp,giant,helicopter,straight wing,flower,galleon,ship modern\n\n6. **Splits of SO-MN40**:  \n6.1 Train  \nRectangular-Cubic Prism: table,night stand,sink, monitor  \nSolids of Revolution: Glass box,flower pot  \nMiscellaneous: Keyboard, airplane  \n6.2 Test  \nRectangular-Cubic Prism: Mantel,tv stand,desk,sofa,bed,bookshelf,chair,bathtub,wardrobe,dresser, radio,piano,bench,xbox,range hood  \nSolids of Revolution: Bottle,bowl,cup,stool,vase,cone,tent  \nMiscellaneous: Toilet,curtain,car,guitar,stairs,door,person,laptop,plant,lamp  \n\n7. **Splits of SO-ABO**:  \n7.1 Train  \nRectangular-Cubic Prism: table\nSolids of Revolution: tent\nMiscellaneous: Mirror,Plant or flower pot  \n7.2 Test  \nRectangular-Cubic Prism: chair,cart,shelf,cabinet,dresser,bed,bench,ladder,sofa\nSolids of Revolution: exercise weight,container or basket,vase,ottoman,pillow\nMiscellaneous: picture frame or painting,lamp,fan\n\nThank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7Fx2JWHg5", "reviewer": "Reviewer_YDem", "summary": "This paper introduces a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. A novel framework, HERT, is proposed for this task. The HRE module is designed to overcome the global disequilibrium of unseen categories. Besides, the SET module is designed to utilize more equilibrial correlations among objects and generalize to unseen categories. Furthermore, four semi-open 3DOR datasets are generated with multi-level labels for benchmarking. The proposed method achieves good performance.", "questions": "Please clarify the technical insights.", "limitations": "The writing can be improved.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is easy to read. This paper targets an interesting problem, open-set 3D object retrieval. The proposed method is also interesting.", "weaknesses": "The proposed method is a little simple, making the technical contribution unclear. Maybe the writing should be improved to highlight the technical insights.\n\nThis paper introduces a new task. It would be better to discuss the difference between this new task and existing ones in the technical aspect. It also helps if some promising research directions for this new task could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer YDem**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.\n\n1. **About the technical contribution (Answer for Weakness 1 and the Questions)**:  \na) **A newly general task for real-world practical machine learning**. We demonstrated the limitations of naive open-set learning tasks and methods through experiments and proposed a more practical semi-open learning task. Additionally, we constructed four semi-open datasets for benchmarking.  \nb) **An early explored paradigm and framework for semi-open learning**. Specifically, we proposed the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework, including the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules, which are designed to overcome the distribution disequilibrium and confusion of unseen categories in semi-open 3D object retrieval.  \nc) **A flexible high-order structure for semi-open learning**. We propose a superposed Hypergraph structure to capture high-order correlations among objects, under the guidance of local coherent correlations and global entangled correlations from hierarchical category information.  \nd) **Extensive experiments and analysis**. Experimental results on the four datasets demonstrate that our method can outperform state-of-the-art retrieval methods in the semi-open environment.\n\n2. **About the task comparison (Answer for Weakness 2)**:  \nWe discuss the difference between this new task (semi-open learning) and existing ones (open-set learning) in Tab. R1 of the rebuttal PDF. Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets[1][2]. However, in practical real-world scenarios, objects are typically described by multiple hierarchical labels, and the training set and testing set often share a partial space of coarse labels. We expand the number of label levels in the semi-open learning task, where testing categories are unseen at one level but seen at other levels. The label spaces are disjoint at only one level and have some overlap at other levels.\n\n    [1] Zhou Z. Open-environment machine learning[J]. National Science Review, 2022.  \n    [2] Parmar J, et al. Open-world machine learning: applications, challenges, and opportunities[J]. ACM Computing Surveys, 2023.\n\nThank you again for your valuable suggestions, especially your professional advice on presentation and future work in semi-open learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VyPrAo2XmE", "reviewer": "Reviewer_oLzP", "summary": "The paper introduces a novel framework called the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) for semi-open 3D object retrieval. The proposed framework addresses the practical scenario of semi-open environments where the training and testing sets share a partial label space for coarse categories but are completely disjoint for fine categories. The HERT framework comprises two main modules: Hierarchical Retrace Embedding (HRE) and Structured Equilibrium Tuning (SET). The authors also generate four semi-open datasets to benchmark their approach and demonstrate its effectiveness through extensive experiments.", "questions": "- How does the HERT framework perform in scenarios with more than three levels of hierarchical labels?  \n&nbsp;\n- What are the computational requirements for training and deploying the HERT framework, especially in terms of time and resources?  \n&nbsp;\n- Can the proposed method be extended to other domains beyond 3D object retrieval, such as text or image retrieval?", "limitations": "The authors have provided discussions about the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Novel Framework**: The introduction of the HERT framework for semi-open 3D object retrieval is a novel contribution that fills a gap in the current literature.  \n&nbsp;\n- **Hierarchical Approach**: The use of hierarchical labels to better capture the multi-level semantics of 3D objects is innovative and aligns well with real-world scenarios.  \n&nbsp;\n- **Comprehensive Experiments**: The authors conducted extensive experiments on four newly generated datasets, providing strong empirical evidence of the effectiveness of their approach.  \n&nbsp;\n- **Clear Problem Definition**: The paper clearly defines the semi-open environment and distinguishes it from traditional open-set and closed-set scenarios.", "weaknesses": "- **Complexity**: The proposed framework is quite complex, which might make it difficult for practitioners to implement and extend.  \n&nbsp;\n- **Lack of Baseline Comparisons**: While the paper compares HERT against state-of-the-art methods, more diverse baseline comparisons, including simpler methods, could provide a clearer picture of the improvements.  \n&nbsp;\n- **Citation**: Please add citations for the methods you compared against in the tables (Tab. 1, 2, 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer oLzP**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.\n\n1. **About the framework (Answer for Weakness 1)**:  \nBased on your suggestions, we have restructured the presentation of the proposed framework. Specifically, **the proposed framework consists of two sequentially connected modules**: HRE and SET.  \na) The HRE module takes basic features of different modalities as input. This module employs two sets of auto-encoders sequentially to achieve modality fusion and category space retrace, and generates unified embeddings and retrace embeddings for each object.  \na) The SET module takes two types of embeddings from the last module as input, utilizing structure-aware feature smoothing and distillation through hypergraph convolution and memory bank reconstruction, respectively. Finally, this module generates the final features for similar object matching based on feature distance, thereby enabling retrieval.\n\n2. **About more comparisons (Answer for Weakness 2)**:  \nInspired by your suggestions, we added three simpler methods as compared baselines to make our experiments more comprehensive. We provide the additional experimental results in Tab. R3 of the rebuttal PDF. From the results, we can observe that the low performance of simpler methods like MLP and GCN demonstrates the complexity of the semi-open environment and the necessity of research in semi-open learning. The significant improvement achieved by our method also proves its effectiveness. We will include these results and analyses in the revised version of the paper. \n\n3. **About citation and writing (Answer for Weakness 3)**:  \nWe have added citations for the compared methods as shown in Tab. R3 of the rebuttal PDF, and we have made similar revisions for the tables in the paper.\n\n4. **About the level of labels (Answer for Question 1)**:  \nThis paper is an early exploration of semi-open learning. Therefore, we selected the two typical levels of labels based on different criteria: geometry-based coarse shape category and semantic-based fine category, which is representative of a semi-open environment. This two-layer framework is also a typical implementation of the collaborative learning paradigm based on hierarchical correlations. When the levels of hierarchical labels increase, a natural implementation is to construct more Retrace Auto-Encoders, and each auto-encoder is designed to retrace one level of the category. This involves addressing challenges such as domain adaptation associated with multiple levels while balancing complexity, efficiency, and performance. Specifically, we have preliminarily experimented with a hypergraph-based isomorphism computation method to address the increase in parameters brought by higher levels, inspired by [1]. Additionally, we have developed a hypergraph-based dynamic system approach to manage the increasing number of labels at each layer inspired by [2].  \n    [1] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2023.  \n    [2] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.\n\n5. **About the computational requirements (Answer for Question 2)**:  \nOur experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.\n\n6. **About the data extension (Answer for Question 3)**:  \nAs shown in #line 152-153 and Appendix C Algorithm 1, the proposed HERT framework is a feature-driven framework and exclusively relies on the input of basic features, rather than utilizing raw data through the end-to-end approach. This feature-driven representation approach preserves extensibility to other common multimedia data such as e.g. text, audio, video, 3d, etc. We believe this paper can provide a general theoretical foundation and methodological reference for the application of multimedia retrieval in practical real-world scenarios. We will release the datasets, code, configs, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.\n\nThank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMp8YdprcV", "reviewer": "Reviewer_vaz9", "summary": "This paper extends the open-set 3D object retrieval problem to a semi-open situation where hierarchical semantic labels are considered. The authors leverage the multi-level category information with a proposed hypergraph-based Hierarchical Equilibrium Representation (HERT) framework. This framework consists of a Hierarchical Retrace Embedding module for encapsulating hierarchical semantic information and a Structured Equilibrium Tuning module for learning generalizable features according to the constructed superposed hypergraph from both local coherent and global entangled correlations.", "questions": "1. In Section 3 of the problem setup, the traditional 3D Object Retrieval setting contains no multi-level labels, right? Bringing this concept with the proposed multi-level labels without explanation is unsuitable, which may lead to confusion.\n2. The authors construct four datasets for evaluating their framework in their proposed task. However, as listed in Appendix B, each dataset is equipped with only 3 coarse categories according to the shape of the objects, which increases my concerns about the efficiency of the proposed hierarchical semantic graph in more complex situations. Are there any possible coarse categories to be included in the real scenes? \n3. As the proposed framework is trained in a two-stage scheme, It would be better to include a comparison of model complexity and training/inference times, \n4. Will the proposed datasets be released?", "limitations": "The authors have mentioned the potential limitations.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed semi-open 3D object retrieval setting is reasonable and more practical for real-world applications. \nThe idea of introducing a hierarchical semantic graph is suitable for this task and the proposed framework is technically sound. \nExperiment results could demonstrate the effectiveness of the proposed framework, the ablation study comprehensively shows the functionality of the coarse label.", "weaknesses": "1. The mathematical symbols within this article are chaotic, giving the following examples:\n\n    1.1. The notations of basic features in Line 152 ($\\mathcal{F}$) and Line 155 ($f$)are inconsistent. Does the aggregation function $\\mathcal{X}$ in Line 156 have the same meaning as $\\mathcal{T}$ in Line 155? \n\n    1.2. The description in Lines 157-163 does not match the illustrated pipeline in Figure 2, which shows that $\\mathcal{A}_r$ takes $e_i$ as a side input (with a concatenate operation? This is also confusing. I think $e_i$ is concat with $c_i$ before inputting into $\\mathcal{A}_r$). How exactly can the Retrace Encoding $e_i$ be obtained? Where are the reconstruction features $\\hat{m}_i$ mentioned in Line 163 be used? Does the $\\hat{m}_i$ actually the $f_i$ in Figure 2? However, $f$ has been utilized to represent the basic feature in Line 155. I suggest adding more essential symbols and legends in Figure 2 to align with the descriptions. Similar things happen in the Appendix C Algorithm 1.\n\n    1.3. Besides, it is not recommended to over-defined some ''spaces'' that are not utilized in the rest of the article, such as ''retrace space $\\mathbb{S}_r$''  (Line 159 retracte -> retrace? Line 160 $\\mathbb{S}_f$ -> $\\mathbb{S}_r$?) and ''mixed space $\\mathbb{S}_x$''. These may bring potential typos and are unhelpful in understanding the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response for Reviewer vaz9**\n\nWe sincerely thank you for the valuable comments and advice, which provided important guidance for the presentation of this paper and clarified the direction for future work.\n\n1. **About the writing and symbols (Answer for Weakness 1)**:  \nWe apologize for these typos. We will conduct a thorough review and revision of the entire paper to ensure the clarity and rigor of the writing.  \n1.1 *(For Weakness 1.1)* The typos $\\mathbf{F}^k$ in line 152 and $\\mathcal{T}$ in line 155 should be correctly written as $f^k$ and $\\mathcal{X}$, which denote the basic features of $k$-th modality and aggregation function, respectively.  \n1.2 *(For Weakness 1.2)* We have restructured the data flow in the HRE module for each object:  \n$\\{ f^k_i \\}^M_{k=1} \\rightarrow \\mathcal{A}_m \\rightarrow \\mathcal{X} \\rightarrow c_i$  \n$ (c_i+e_i) \\rightarrow \\Psi \\rightarrow r_i \\rightarrow \\Phi \\rightarrow \\hat{m}_i$  \nwhere $e_i$ is an encoding of the coarse label, which is a learnable vector of the same dimension as $c_i$. Specifically, retrace encoding is implemented by nn.Parameter for each coarse category, objects within the same coarse label share the same retracing encoding. During computation, it is element-wise added to $c_i$ and then input to $\\mathcal{A}_r$. $\\hat{m}_i$ is used solely for loss calculation in the HRE module, using coarse labels for supervision to guide the accuracy of the retrace embedding representation. It does not participate in the computations of the SET module.  \nBased on your suggestions, we have revised the pipeline diagram and added more symbols, as shown in Fig. R2 of the rebuttal PDF.  \n1.3 *(For Weakness 1.3)* We removed the presentation of space in lines 157-163 to enhance the readability of the paper. Specifically, we revise this paragraph as follows:  \n*...then $\\mathcal{A}_r$ compresses the unified embedding $c_i$ aligned with $e_i$ into retrace embedding $r_i$ and does the reverse reconstruction $\\hat{m}_i$ for supervision...*  \n\n2. **About the traditional 3D object retrieval method (Answer for Question 1)**:  \nTraditional 3D object retrieval methods, both closed-set [1-3] or open-set[4][5] methods, consider only single-layer labels of objects. Besides, traditional open-set methods strictly assume no overlap between the training and testing sets[5][6]. However, in practical real-world scenarios, objects are typically described by multiple hierarchical labels, and the training set and testing set often share a partial space of coarse labels. As shown in Tab. R1 of the rebuttal PDF, we expand the number of label levels in the semi-open learning task, where testing categories are unseen at one level but seen at other levels. The label spaces are disjoint at only one level and have some overlap at other levels.  \n    [1] Gao Y, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE TIP, 2012.  \n    [2] He X, et al. Triplet-center loss for multi-view 3d object retrieval[C]. IEEE CVPR, 2018.  \n    [3] Collins J, et al. Abo: Dataset and benchmarks for real-world 3d object understanding[C]. IEEE CVPR, 2022.  \n    [4] Zhou Z. Open-environment machine learning[J]. National Science Review, 2022.  \n    [5] Feng Y, et al. Hypergraph-based multi-modal representation for open-set 3d object retrieval[J]. IEEE TPAMI, 2023.  \n    [6] Parmar J, et al. Open-world machine learning: applications, challenges, and opportunities[J]. ACM Computing Surveys, 2023.\n\n3. **About the efficiency of graphs (Answer for Question 2)**:  \nThis paper is an early exploration of semi-open learning. Therefore, we selected the three typical categories for the geometry-based coarse label, which is significantly different from the semantic-based fine category. These three coarse categories and two levels of hierarchical labels are representative of a semi-open environment, helping us focus on exploring the new semi-open learning task and designing a novel collaborative learning paradigm based on hierarchical correlations. Specifically, when the category of coarse labels increases, a natural implementation is to construct a hypergraph structure with more hyperedges to capture more complex correlations. This involves addressing challenges such as complex network representation associated with multiple labels while balancing complexity, efficiency, and performance. Tackling these challenges, we have preliminarily experimented with a hypergraph-based isomorphism computation method for structure compression and complexity reduction, inspired by [7]. Besides, we also have developed a hypergraph-based dynamic system approach to manage the dynamic increase of categories and levels of the label inspired by [8].  \n    [7] Feng Y, et al. Hypergraph isomorphism computation[J]. IEEE TPAMI, 2024.  \n    [8] Yan J, et al. Hypergraph dynamic system[C]. ICLR, 2024.\n\n4. **About the computational requirements comparison (Answer for Question 3)**:  \nOur experiments are conducted on a computing server with one Tesla V100-32G GPU and one Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. We provide a detailed comparison of model parameters, training time, and inference time for the two stages in Tab. R2 of the rebuttal PDF.\n\n5. **About the open access to datasets (Answer for Question 4)**:  \nThanks for your interest in our work. We are well prepared and will release the datasets, code, configurations, and pre-trained models immediately after the anonymous review period of NeurIPS 24. We also look forward to engaging and collaborating with more researchers on both theoretical and applied studies of semi-open learning across different fields. Additionally, we are willing to share our experiences on this (OpenReview) or other open-source platforms.\n\nThank you again for your valuable suggestions, especially your professional advice on future work in semi-open learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for your insightful feedback and for your valuable time and effort. We try to answer all the questions and weaknesses of each reviewer in the rebuttal section below. The attached PDF contains our additional experimental results and figures.

Meta-review: This paper receives 1 strong accept, 2 week accept, and 1 borderline reject. Most reviewers agree that the proposed semi-open setting could be interesting to future research in the field when objects are getting more complicated. Some reviewers raised concerns regarding the complexity of the proposed method and lack of strong baseline comparisons. But these concerns are addressed after rebuttal. Reviewer nF63 still holds the concerns regarding the coarse-label partition is too coarse to be practical after thoughtful discussion. After reading the paper and the discussion, AC finds that the proposed new setup and dataset is still quite beneficial to the field. The authors are encouraged to make the above mentioned necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)