Title: Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph
Venue: NeurIPS 2024
Accepted: yes
Keywords: Hypergraph

3D Object Retrieval

Open-Set Learning

Multi-Modal Fusion

Multi-Label Learning
Abstract: Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets, leading to contradictory optimization for superposed categories. In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for this task. Specifically, we propose the Hierarchical Retrace Embedding (HRE) module to overcome the global disequilibrium of unseen categories by fully leveraging the multi-level category information. Besides, tackling the feature overlap and class confusion problem, we perform the Structured Equilibrium Tuning (SET) module to utilize more equilibrial correlations among objects and generalize to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we generate four semi-open 3DOR datasets with multi-level labels for benchmarking. Results demonstrate that the proposed method can effectively generate the hierarchical embeddings of 3D objects and generalize them towards semi-open environments.

Semi-Open 3D Object Retrieval via Hierarchical Equilibrium on Hypergraph

Abstract

Existing open-set learning methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing sets, leading to contradictory optimization for superposed categories. In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for this task. Specifically, we propose the Hierarchical Retrace Embedding (HRE) module to overcome the global disequilibrium of unseen categories by fully leveraging the multi-level category information. Besides, tackling the feature overlap and class confusion problem, we perform the Structured Equilibrium Tuning (SET) module to utilize more equilibrial correlations among objects and generalize to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we generate four semi-open 3DOR datasets with multi-level labels for benchmarking. Results demonstrate that the proposed method can effectively generate the hierarchical embeddings of 3D objects and generalize them towards semi-open environments.

1 Introduction

3D objects are of paramount significance, finding extensive applications from computer graphics [1] to security [31] and autonomous robotics [2]. As the fundamental task of data acquisition, 3D object retrieval (3DOR) [22, 10] plays a pivotal role in the computer vision community [3]. 3DOR methods learn to represent 3D objects from the training set and then extract features from query objects to effectively align similar samples. According to the category overlap between training and testing sets, existing 3DOR algorithms can be divided into closed-set and open-set types. The former conducts retrieval for objects whose categories have been seen in the training set [32, 36], while the latter handles objects of unseen categories [7].

Existing open-set 3D learning methods are based on the assumption that the labels of object categories are at a single level [46, 23]. In practical scenarios, objects are typically described by multiple hierarchical labels. This leads to categories in training and testing sets showing varying degrees of overlap at different levels. As illustrated in Figure 1(a), we term this scenario, where the training and testing set share a partial label space of coarse categories but are completely disjoint from fine categories, as Semi-Open Environment . Besides, existing methods typically extract basic features using pre-trained models, followed by further open-set learning and optimization. However, for

Fine

Coarse

Semi

Open

Open

Solid of

Revolution

Miscellaneous

Rectangular

-Cubic

Overlap

Disjoint

Solid of

Revolution

Miscellaneous

Shapes

Rectangular-

Cubic Prism

Disjoint

Push

Pull

Contradictory

Overlap

Seen Categories (Train)

Unseen Categories (Test)

Open-Set Learning

Basic Feature

(a) Hierarchical labels in semi-open environment.

(b) Optimization contradiction of hierarchical labels.

categories with the same coarse label but different fine labels, the training of the basic model and the open-set learning will result in contradictory optimizations as shown in Figure 1(b).

Specifically for 3D object retrieval techniques, only a few methods [7, 32] have been explored for open-set retrieval. These methods consider only the single-layer labels of objects and strictly assume no overlap between the training and testing data distributions. The existing open-set 3DOR method [7] treats objects from seen and unseen categories as isomorphic vertices and constructs a graph model based on global correlations. On one hand, this isomorphic model overlooks hierarchical correlations inherent in multi-level categories, particularly the shared coarse labels. Consequently, the embedding distribution from structure-aware learning tends to be unbalanced towards the feature space of seen categories in the training set. On the other hand, this structure focuses on global correlations between seen and unseen categories, neglecting intricate local correlations within unseen categories themselves. This lack of local attention may result in issues like feature overlap and class confusion within these unseen categories.

Focusing on this practical semi-open environment for retrieval, where the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories, we introduce the semi-open 3D object retrieval task and construct four datasets with multi-level labels to expand the application of 3DOR. We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for semi-open 3DOR. To overcome the global disequilibrium of unseen categories, we propose the Hierarchical Retrace Embedding module (HRE) to achieve balanced representation across multi-level categories. This module generates multi-level retrace embeddings for capturing the hierarchical semantics of objects. To tackle the feature overlap and class confusion problem, we propose the Structured Equilibrium Tuning (SET) module. This module utilizes high-order correlations among objects for unseen category generalization, by constructing a superposed Hypergraph based on local coherent and global entangled correlations. In summary, our main contributions are fourfold:

We introduce the semi-open 3D object retrieval task to refine the setting of the 3D object retrieval task in real-world scenario applications, and we construct four datasets for benchmarking downstream tasks.

We propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework for the semi-open 3DOR task, including the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules, which are designed to overcome the distribution disequilibrium and confusion of unseen categories.

We propose a superposed Hypergraph structure to capture high-order correlations among objects, under the guidance of local coherent correlations and global entangled correlations from hierarchical category information.

Experimental results on the four datasets demonstrate that our method can outperform state-of-the-art retrieval methods towards the semi-open environment.

2 Related Work

3D Object Retrieval. 3D object retrieval (3DOR) aims to find the relevant objects from the target set for the query objects, most 3DOR methods construct feature alignment models for 3D objects through metric learning. MMJN [20] proposes a discrimination loss to minimize the distance between objects belonging to the same categories, and learn discriminative embeddings for retrieval. MIFN [17] constructs fusion networks by weighted concatenation for modality-specific features. PVNet [42] proposes a joint network for the fusion of multi-view and point cloud features. PVRNet [43] proposes an attention mechanism to generate the unified embedding of different modalities. CMCL [14] designs a cross-modal center loss to compress features of different modalities to a modal-invariant space. However, most existing methods mainly focus on close-set retrieval. Currently, only a few methods [7, 32, 39, 38, 37] have explored open-set retrieval, but they typically assume no overlap between training and testing data distributions, which is at odds with the semi-open setting.

Open-Set Learning. Open-set learning (OSL) methods can be roughly separated into two categories [45], i.e. , discriminative methods and generative models. In discriminative models, traditional methods achieve the classification of unknown and known categories based on the Support Vector Machine (SVM) [27, 28] or the Extreme Value Theory (EVT) [12, 26]. Recently, deep learning methods for OSL have made remarkable rapid progress. OpenMax [4] is the first algorithm proposed to replace the SoftMax layer and calibrate the output probability with the Weibull distribution. Then, [29] and [41] utilize the one-vs-rest units and reconstructed latent representation for unknown detection. Generative OSL models are designed to anticipate the distribution of novel classes through training. PROSER [45] allocates placeholders for both data and classifier to detect the unknown classes, C2AE [21] propose a two-step framework to tackle open-set recognition problem with close-set training and open-set training, respectively. Both types of open-set learning methods [9, 19] are based on the assumption that there is no overlap between known and unknown data distribution, thus facilitating the design of classifiers for unseen object detection, but are challenging to generalize to practical scenarios that involve hierarchical superposed categories.

Multi-Label Learning. Multi-label learning aims to create a model that can assign multiple labels for each instance simultaneously. Existing multi-label learning methods are mostly designed for recognition tasks, and based on one-vs-all classifiers [40, 24] and embeddings [44, 18]. As for the retrieval task, DMSSPH [33] proposes a multi-level preserving hashing network, and AMD-GCN [16] designs a GCN-based network for multi-label pattern image retrieval. TranGCN [15] proposes a cross-modal attention mechanism at each layer for multi-label embeddings. Although these methods construct effective models for image embedding, they overlook the hierarchical relationship and dependencies between different categories. In this paper, we aim to model the correlations between hierarchical label information and generalize to unseen categories.

3 Problem Setup

3.1 3D Object Retrieval

The goal of the 3D object retrieval (3DOR) is to develop a method using the training set S trn = { ( o i , y c i , y f i ) } L i =1 , which is then employed to identify similar objects in the retrieval (testing) set S ret = { ( o i , ˆ y c i , ˆ y f i ) } R i =1 , which is comprised of the query set S q and the target set S t . Here, L and R represent the number of samples in the training and testing (retrieval) sets, respectively. The expressions y f i ∈ Y f = { c f j } Y f j =1 and y c i ∈ Y c = { c c k } Y c k =1 denote the fine and coarse category labels of 3D object o i , respectively. Typically, y c i provides the category label from a more general (coarse) level, such as the basic geometric shapes or other common attributes between S trn and S ret , whereas y i provides a semantic-specific (fine) category label. Generally, the number of coarse categories Y c is much smaller than the number of fine categories, which is Y c ≪ Y f . For 3D objects, The o i = { m r } r M =1 denotes the representation by M modalities, i.e. , multi-view, point cloud and voxel.

3.2 Semi-Open 3D Object Retrieval

In traditional open-set 3DOR, the category spaces of the training set and the retrieval set are not the same fine indicating y f i ∈ Y f = { c f j } Y f j =1 , ˆ y f i ∈ ˆ Y f = { ˆ c f j } ˆ Y f j =1 , and Y f = ˆ Y f . Y f and ˆ Y f denote

̸

Hierarchical Retrace Embedding (HRE)

Structured Equilibrium Tuning (SET)

GLYPH(cmap:d835)

Multi-Modal Auto-Encoder

Retrace Auto-Encoder

Global Entangled

Local Coherent

Structure-Aware

Tuning

Basic

Features

Similar Objects

Retrace Embedding

Solid of

Revolution

Miscellaneous

Rectangular-

Cubic

Training Fine Categories

Testing Fine Categories

Retrace Encoding

(Trainable Parameters)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

⨁

Superposed Hypergraph

GLYPH(cmap:d835)

GLYPH(cmap:d835)

⨁

Retrieval

Hypergraph

Convolution

Memory

Bank

Coherent

Entangled

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

the numbers of fine categories in the training set and the retrieval set, respectively. In the practice of semi-open 3DOR, the training set and the testing set share the same space of coarse category, which y c i ∈ Y c = { c c k } Y c k =1 , ˆ y c i ∈ ˆ Y c = { ˆ c c k } ˆ Y c k =1 , and Y c = ˆ Y c , Y c and ˆ Y c denote the numbers of coarse categories in the training set and the testing set, respectively. During the retrieval phase, each query object is provided with a coarse category label. For better representation, the semi-open 3DOR defines the retrieval model r := ( o i | y c i ) ↦→ z i that maps the 3D object o i into a semantic embedding z i ∈ R d under coarse category condition y c i . The task seeks to minimize the expected risk under the conditional constraints of coarse labels:

̸

where S i = ( o i , y c i , ˆ y f i ) and S j = ( o j , y c j , ˆ y f j ) are object instances sampled from the query set S q and target set S t , ˆ y f i and ˆ y f j denote the predicted fine labels, y c i and y c j are the conditional constraint, which are coarse labels. I ( · ) is the indicator function, which returns 1 if the expression holds and 0 otherwise. H is the hypothesis space of map r ( ·|· ) . D ( z i , z j ) is the distance metric function between different embeddings.

4 Methodology

4.1 Overall Framework

The overall framework of the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework is illustrated in Figure 2. HERT is composed of the Hierarchical Retrace Embedding (HRE) and the Structured Equilibrium Tuning (SET) modules. Given 3D objects represented by multiple modalities, common-used backbones are used to extract the basic features for each modality. Then, the HRE module is introduced to generate the multi-level retrace embeddings of hierarchical semantic information. Next, in the SET module, the superposed hypergraph is constructed based on the local coherent and global entangled correlations. Finally, the hypergraph convolution and memory bank under the superposed structure are used to smooth and distill for feature generalization between seen and unseen categories.

4.2 Hierarchical Retrace Embedding

To obtain fully multi-level embeddings of objects based on hierarchical categories, the HRE module is designed here. Specifically, the HRE utilizes two hierarchical auto-encoders in series as shown in Figure 2. The multi-modal auto-encoder A m encodes the multi-modal basic features of 3D objects

to get the unified embeddings. The retrace auto-encoder A r encodes the coarse label aligning with the unified embeddings to get retrace embeddings.

Given N 3D objects { o i } N i =1 and feature extractors {F k } M k =1 , the basic feature of M modalities { F k } M k =1 = F k ( { o i } N i =1 ) can be generated, where F k ∈ R N × d f . As shown in Fig. 2, we first compress the input basic features from M modalities into the latent modal-invariant space and generate the unified embeddings from multi-modal auto-encoder A m . Specifically, for the basic features f k i of object o i , the unified embedding of it can be denoted as c i = T ( A m ( { f k i } M k =1 ) ) , where X denotes the aggregation function among modalities and c i ∈ R d c .

After getting the unified embedding c i , the retrace auto-encoder A r takes the coarse category label y c i into retrace encoding e i . As shown in Fig. 2, then A r compresses the unified embedding c i aligned with e i into the retracte space S r and does the reverse reconstruction to the mixed space S x , which can be defined as follows: Ψ := S x → S f and Φ := S f → S x , where Ψ( · ) is the encoder that maps the unified embedding c i aligned with e i into the retracte space S r , the retrace embedding r i can be generated by as r i = Ψ( c i + e i ) , r i ∈ R d r . Φ( · ) is the decoder that maps the retrace embedding to reconstruction feature ˆ m i = Φ( r i ) , ˆ m i ∈ R d c .

Through the HRE stage, we got the unified embedding c i , retrace embedding r i , and mixed feature ˆ m i for each 3D object.

4.3 Structured Equilibrium Tuning

To endow the unified and retrace embeddings of 3D objects with the ability to generalize to unseen categories, we introduce the SET module as shown in Figure 2. Specifically, the Superposed Hypergraph structure is employed to capture the local coherent and global entangled correlations under the constraint of hierarchical category information. Then, hypergraph convolution is employed to utilize the collaborative high-order correlation under the guidance of this superposed Hypergraph. Finally, we use the hypergraph convolution and memory bank to generalize the structure-aware knowledge to generate unbiased features for unseen categories.

4.3.1 Superposed Hypergraph

Despite the lack of fine labels in objects from unseen categories, the implicit information within these categories can enhance the generalization capabilities of 3D object embeddings, especially when prompted by coarse labels. To establish the high-order correlations between 3D objects from both seen and unseen categories under common general space, we design a superposed hypergraph structure. The hypergraph in this paper can be represented as G = {V , E} , where V and E are the vertex set and the hyperedge set, respectively.

We construct the vertices in our superposed hypergraph by the multi-level retrace embeddings. Specifically, we combine the unified embedding c i and retrace embedding r i to generate the heterogeneous retrace vertices v i ∈ R N × d c , which can be defined as follows:

where λ is the hyper-parameter to trade-off between embeddings of different levels.

Distinct from the traditional hypergraph [8], we construct a superposed hypergraph with two types of conditional hyperedges: coherent hyperedges and entangled hyperedges.

Coherent Hyperedge. The coherent hyperedges capture the local coherent correlations in the coarse category space. We define the coherent hyperedge as e c ∈ E c under the condition of coarse labels:

where S v ( y c ) denotes the vertex subset that shares the same coarse label y c and Y c denotes the coarse label space of 3D objects.

Entangled Hyperedge. The entangled hyperedges model the global collaborative correlations among objects between seen and unseen categories. For each vertex, we define its entangled hyperedge as the set of nearest K -1 neighboring vertices. Specifically,

where D KNN k ( v ) denotes the k-nearest neighbors in the feature space of vertex v .

In this way, we construct Y c coherent hyperedges and N entangled hyperedges, where Y c is the number of coarse categories and N is the number of vertex in the superposed hypergraph. Finally, we combine the two sets of hyperedges to obtain a complete superposed hypergraph for structure-aware tuning.

where E c and E t are the set of coherent and entangled hyperedges, respectively.

4.3.2 Structure-Aware Tuning

To get better generalization on unseen categories, hypergraph convolution and memory bank are adopted here for feature smoothing and distillation.

For the convenience of computation, the hypergraph can be represented by the incidence matrix H ∈ { 0 , 1 } |V|×|E| , where the i -th hyperedge is the i -th column of H , and H ( v, e ) = 1 if the hyperedge e contains the vertex v . W ∈ R |E|×|E| is a diagonal matrix, where W i,i denotes the weight of the i -th hyperedge.

To learn the conditional embeddings ˜ V ∈ R N × d c from multi-level embedding V ∈ R N × d c under the guidance of hypergraph, the hypergraph convolution (HGNNConv [8]) can be represented as:

where D v and D e are the diagonal degree matrices for vertex and hyperedge, respectively. Θ ∈ R d u × d u is the trainable parameter for the HGNNConv layer [8].

To increase the generalization ability of the SET module, we construct a memory bank M that contains L invariant memory anchors for conditional embedding ˜ v i of the 3D object o i , we compute the activation score for each memory anchor in the memory bank by t ij = D m (˜ v i , a j ) , where a j denotes the anchor and D m ( · , · ) denotes the distance metric function. We rebuild the aligned multilevel embedding of each object by z i = ∑ L j =1 t n i,j a j , z i ∈ R d c , where t n ij denotes the normalized values of activation score.

4.4 Training Objective

Loss Function for the HRE. To get a better representation of multi-level label information, we first use the Homology Loss L homo and Bi-reconstruction Loss L br followed [7] for A m to leverage the collaborative information across modalities, then we adopt the Retrace Cross-Entropy loss L ce to guide the retrace embedding of coarse category information, which can be defined as follows:

where p i,k = e ˆ mi,k ∑ Y c k =1 e ˆ mi,k is the predicted probability score of the 3D object o i in k -th coarse category for the reconstructed mixed feature ˆ m i . y c i,k is the k -th value of the one-hot encoded ground truth coarse label of o i , and Y c is the number of the coarse categories.

In the hierarchical retrace embedding stage, the overall loss function is given:

where µ is the hyper-parameter to trade-off between the loss of multi-modal and multi-level representations.

Loss Function for the SET. To train the hypergraph convolution and learnable memory anchors, we adopt Memory Reconstruction Loss L mr and the Cross-entropy Loss L ce :

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

ion

0.6

lon

0.6

0.6

ion

0.6

S

S

S

S

C

0.4

C

0.4

C

0.4

C

0.4

P

0.2

SDML

P

0.2

SDML

P

0.2

SDML

←C2AE

P

0.2

SDML

0.0

MMSAE

TranGCN

HGM2R

←Ours

0.0

一

一

MMSAE

TranGCN

→Ours

HGM2R

0.0

MMSAE

TranGCN

HGM2R

snO-

0.0

1

+

MMSAE

TranGCN

Ours

HGM2R

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

Recall

Recall

(a) PR-C on SO-ESB.

(b) PR-C on SO-NTU.

(c) PR-C on SO-MN40.

(d) PR-C on SO-ABO.

where ˜ p i,k = e ˜ u i,k ∑ Y m =1 e ˜ u i,m and p i,k = e z i,k ∑ Y m =1 e z i,m is the predicted probability score of the 3D object o i in k -th fine category for the multi-level embedding ˜ v i and memory reconstruction embedding z i . y i,k is the k -th value of the one-hot encoded ground truth fine label of o i , and Y is the number of the coarse categories.

In the structured equilibrium tuning stage, the overall loss function is given by:

where η is the hyper-parameter for trade-off between them.

5 Experiments

5.1 Dataseta and Evaluation Metrics

Datasets. We generate four semi-open 3DOR datasets, including SO-ESB, SO-NTU, SO-MN40, and SO-ABO, based on the public datasets ESB [13], NTU [5], ModelNet40 [35], and ABO [6], respectively. We add coarse category labels for each object based on the basic geometric shapes such as solid of revolution, rectangular-cubic, etc . Also, we remove some objects that are difficult to categorize based on their shapes. Then, we split the fine categories into seen categories for training and unseen categories for testing, the training and testing sets share the same coarse label space according to the semi-open environment setting. Each object has three modalities including multi-view, voxel, and point cloud. Specifically, the detailed descriptions of dataset generation and setting are shown in Appendix B.

Evaluation Protocols. As for the evaluation criteria, We employ commonly used retrieval metrics for comparison, including Mean Average Precision (mAP), Recall, Normalized Discounted Cumulative Gain (NDCG), Average Normalized Modified Retrieval Rank (ANMRR), and the Precision-Recall Curve (PR-C). For the mAP, Recall, and NDCG metrics, the higher scores is better. For the ANMRR metric, the lower score is better.

5.2 Experimental Settings

Implemental Details. In our experiments, we choose three modes of multi-view (12 views), point cloud (1024 points), and voxel (32 dimensions) as the representation of 3D objects. The basic features for framework input are extracted by MVCNN [30], PointNet [25], and 3D ShapeNets [35], respectively. The HRE and SET modules are trained separately with 40 and 120 epochs. The SGD optimizers are used for both two modules with learning rates of 0 . 1 and 0 . 001 , respectively. As for the hyper-parameters in HERT, we set λ = 0 . 5 , µ = 0 . 8 , and η = 0 . 9 . Detailed implemental settings for our framework are provided in Appendix C.

Compared Methods. Under this semi-open setting, since there is no 3D object retrieval method designed specifically for this muli-level settings, we choose the current state-of-the-art methods of close-set 3D retrieval (SDML [11], CMCL [14], MMSAE [34]), close-set multi-label retrieval(TranGCN [15]), and open-set 3D learning (C2AE [21], HGM 2 R [7]). For each method, we add a multi-label learning mechanism [15] on their basis for comparison. We provide more implemented details of compared methods in Appendix D.

5.3 Comparison with the State-of-the-Arts

To evaluate the effectiveness of the proposed HERT framework, we conduct experiments on SO-ESB, SO-NTU, SO-MN40, and SO-ABO datasets. The comparison of quantitative results is presented in Table 1 and Table 2, respectively. Results demonstrate that the proposed HERT outperforms the state-of-the-art methods on all four datasets. Especially on the SO-NTU and SO-MN40 datasets, our method achieves mAP of 0 . 5678 / 0 . 6336 with about 17 . 7% / 9 . 6% improvements compared with the second-best method, and achieves Recall of 0 . 8116 / 0 . 3993 with about 10 . 2% / 7 . 9% improvements compared with the second-best method. Besides, we also provide the qualitative results through precision-recall curves as shown in Figure 3, in which the larger area below the curve indicates better performance.

The better results indicate that the proposed HERT framework has the capability to understand and generalize unseen fine categories under the guidance of coarse labels. The proposed HRE and SET modules can fully leverage the hierarchical category information into multi-level retrace embedding and generalize them to unseen categories. This approach better captures the hierarchical semantic correlations in the wild and provides a practical framework for the representation learning of multi-label tasks in semi-open environments. We provide more visualized results in Appendix E.

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

ion

0.6

lon

0.6

lon

0.6

ion

0.6

P

S

0.4

0.2

P

S

0.4

0.2

P

S

0.4

0.2

P

S

0.4

0.2

HRE w/o ReEnc

+ GCN-based SET

HRE w/o ReEnc

* GCN-based SET

HRE w/o ReEnc

* GCN-based SET

HRE w/o ReEnc

*GCN-based SET

0.0

HRE w/o Lce

MLP-based SET

0.0

HRE w/o Lce

MLP-based SET

0.0

HRE w/o Lce

MLP-based SET

0.0

—HRE w/o Lce

MLP-based SET

SET w/o Es

HRE+SET

SET w/o Es

HRE+SET

SET w/o Es

HRE+SET

SET w/o Es

HRE+SET

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

Recall

Recall

(a) PR-C on SO-ESB.

(b) PR-C on SO-NTU.

(c) PR-C on SO-MN40.

(d) PR-C on SO-ABO.

5.4 Ablation Study

We conduct ablation experiments on each module of HERT to demonstrate their effectiveness. First, we remove the retrace encoding e i (HRE w/o ReEnz) and cross-entropy loss in the HRE module (HRE w/o L ce ), where 'w/o' denotes 'without'. This is equivalent to using a naive hierarchical embedding approach for coarse labels. As shown in Table 3, 4, and Figure 4, the proposed retrace embedding approach in the HRE module achieves an mAP improvement of 9 . 4% / 6 . 2% in SO-MN40 dataset and 7 . 8% / 4 . 7% in SO-ABO dataset. These results effectively demonstrate the effectiveness of the HRE module for hierarchical categories.

As for the SET module, we remove the proposed coherent hyperedges (SET w/o E c ) for comparison, also we replace the hypergraph convolution with Graph Convolutional Networks (GCN-based SET) and Multilayer Perceptron (MLP-based SET). Ablation results in Table 3, 4, and Figure 4 show that the proposed SET outperforms all other structure learning methods, and the combination of the HRE and the SET yields the best performance. These results demonstrate the proposed SET can effectively utilize the semi-superposed correlations among objects.

6 Conclusion

In this paper, we introduce a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. We propose the HypergraphBased Hierarchical Equilibrium Representation (HERT) framework for semi-open 3D object retrieval. Specifically, to overcome the global disequilibrium of unseen categories, we propose the Hierarchical Retrace Embedding (HRE) module to fully leverage the multi-level category information. Besides, we perform the Structured Equilibrium Tuning (SET) module to tackle the feature overlap and class confusion problem. This module utilizes more equilibrial correlations among objects and generalizes to unseen categories, by constructing a superposed hypergraph based on the local coherent and global entangled correlations. Furthermore, we construct four 3D object datasets with multi-level category labels for semi-open 3DOR tasks, i.e. , SO-ESB, SO-NTU, SO-MN40, and SO-ABO. Results demonstrate that the proposed method can effectively generate and generalize the hierarchical embeddings of 3D objects in semi-open environments. However, due to dataset limitations, we are currently unable to verify the balanced representation effect on more than three levels of labels, which is one of our future research directions. We believe this paper can provide new insights for future research in more practical scenarios of open-set learning.

References

Ferran Argelaguet and Carlos Andujar. A survey of 3d object selection techniques for virtual environments. Computers & Graphics , 37(3):121-136, 2013.

Eduardo Arnold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis. A survey on 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportation Systems , 20(10):3782-3795, 2019.

Song Bai, Peng Tang, Philip HS Torr, and Longin Jan Latecki. Re-ranking via metric fusion for object retrieval and person re-identification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 740-749, 2019.

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1563-1572, 2016.

Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. Computer graphics forum , 22(3):223-232, 2003.

Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21126-21136, 2022.

Yifan Feng, Shuyi Ji, Yu-Shen Liu, Shaoyi Du, Qionghai Dai, and Yue Gao. Hypergraph-based multimodal representation for open-set 3d object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(4):2206-2223, 2023.

Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji. Hgnn+: General hypergraph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3181-3199, 2022.

ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. arXiv preprint arXiv:1707.07418 , 2017.

Alexander Grabner, Peter M Roth, and Vincent Lepetit. 3d pose estimation and 3d model retrieval for objects in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3022-3031, 2018.

Peng Hu, Liangli Zhen, Dezhong Peng, and Pei Liu. Scalable deep multimodal learning for cross-modal retrieval. In Annual Conference of the Association for Computing Machinery Special Interest Group in Information Retrieval , pages 635-644, 2019.

Lalit P Jain, Walter J Scheirer, and Terrance E Boult. Multi-class open set recognition using probability of inclusion. In European Conference on Computer Vision , pages 393-409. Springer, 2014.

Subramaniam Jayanti, Yagnanarayanan Kalyanaraman, Natraj Iyer, and Karthik Ramani. Developing an engineering shape benchmark for cad models. Computer-Aided Design , 38(9):939-953, 2006.

Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian. Cross-modal center loss for 3d cross-modal retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3142-3151, 2021.

Ying Li, Chunming Guan, Rui Cai, Ye Erwan, Ding Yuxiang, and Jiaquan Gao. Tran-gcn: Multilabel pattern image retrieval via transformer driven graph convolutional network. In ACM International Conference on Multimedia , pages 6301-6310, 2023.

Ying Li, Hongwei Zhou, Yeyu Yin, and Jiaquan Gao. Multi-label pattern image retrieval via attention mechanism driven graph convolutional network. In ACM International Conference on Multimedia , pages 300-308, 2021.

Qi Liang, Mengmeng Xiao, and Dan Song. 3d shape recognition based on multi-modal information fusion. Multimedia Tools and Applications , 80:16173-16184, 2021.

Weiwei Liu, Donna Xu, Ivor W Tsang, and Wenjie Zhang. Metric learning for multi-output tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(2):408-422, 2018.

Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In European Conference on Computer Vision , pages 613-628, 2018.

Weizhi Nie, Qi Liang, An-An Liu, Zhendong Mao, and Yangyang Li. Mmjn: Multi-modal joint networks for 3d shape recognition. In ACM International Conference on Multimedia , pages 908-916, 2019.

Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2307-2316, 2019.

Panagiotis Papadakis, Ioannis Pratikakis, Theoharis Theoharis, and Stavros Perantonis. Panorama: A 3d shape descriptor based on panoramic views for unsupervised 3d object retrieval. International Journal of Computer Vision , 89:177-192, 2010.

Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Open-world machine learning: applications, challenges, and opportunities. ACM Computing Surveys , 55(10):1-37, 2023.

Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In The International Conference of World Wide Web , pages 993-1002, 2018.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Ethan M Rudd, Lalit P Jain, Walter J Scheirer, and Terrance E Boult. The extreme value machine. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(3):762-768, 2017.

Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(7):1757-1772, 2012.

Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(11):2317-2324, 2014.

Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. arXiv preprint arXiv:1709.08716 , 2017.

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 945-953, 2015.

Jian Wang, Fan Li, Xuchong Zhang, and Hongbin Sun. Adversarial obstacle generation against lidar-based 3d object detection. IEEE Transactions on Multimedia , 2023.

Shijie Wang, Jianlong Chang, Haojie Li, Zhihui Wang, Wanli Ouyang, and Qi Tian. Open-set fine-grained retrieval via prompting vision-language evaluator. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19381-19391, 2023.

Dayan Wu, Zheng Lin, Bo Li, Mingzhen Ye, and Weiping Wang. Deep supervised hashing for multi-label and large-scale image retrieval. In International Conference on Multimedia Retrieval , pages 150-158, 2017.

Yiling Wu, Shuhui Wang, and Qingming Huang. Multi-modal semantic autoencoder for cross-modal retrieval. Neurocomputing , 331:165-175, 2019.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1912-1920, 2015.

Zhi Xiong, Dayan Wu, Wen Gu, Haisu Zhang, Bo Li, and Weiping Wang. Deep discrete attention guided hashing for face image retrieval. In International Conference on Multimedia Retrieval , pages 136-144, 2020.

Yang Xu, Yifan Feng, and Lin Bie. Triadic elastic structure representation for open-set incremental 3d object retrieval. In ACM International Conference on Multimedia Retrieval , pages 20-28, 2024.

Multi-View

Voxel

Desk

Sofa

Bed

Bottle

Vase

Plant

Toilet

Coarse

Fine

Rectangular-cubic prism

Solids of revolution

Miscellaneous

米口

Yang Xu, Yifan Feng, and Yue Gao. Negative prompt driven complementary parallel representation for open-world 3d object retrieval. In International Joint Conference on Artificial Intelligence , pages 1498-1506, 2024.

Yang Xu, Yifan Feng, and Yu Jiang. Structure-aware residual-center representation for self-supervised open-set 3d cross-modal retrieval. In IEEE International Conference on Multimedia and Expo , pages 1-6, 2024.

Ian EH Yen, Xiangru Huang, Wei Dai, Pradeep Ravikumar, Inderjit Dhillon, and Eric Xing. Ppdsparse: A parallel primal-dual sparse method for extreme classification. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 545-553, 2017.

Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura. Classification-reconstruction learning for open-set recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4016-4025, 2019.

Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao. Pvnet: A joint convolutional network of point cloud and multi-view for 3d shape recognition. In ACM International Conference on Multimedia , pages 1310-1318, 2018.

Haoxuan You, Yifan Feng, Xibin Zhao, Changqing Zou, Rongrong Ji, and Yue Gao. Pvrnet: Point-view relation neural network for 3d shape recognition. In AAAI Conference on Artificial Intelligence , pages 9119-9126, 2019.

Hsiang-Fu Yu, Prateek Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In International Conference on Machine Learning , pages 593-601. PMLR, 2014.

Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4401-4410, 2021.

Zhi-Hua Zhou. Open-environment machine learning. National Science Review , 9(8):nwac123, 2022.

A Semi-Open Environment

In practical scenarios, the categories of 3D objects can be labeled from various perspectives such as appearance and functionality, or from different levels of granularity, which means hierarchical labels. For example, consider a 3D object, which can simultaneously be labeled as 'sphere'and 'blue'at the basic geometric and color level. Moving up the semantic hierarchy, the same object may be labeled as 'transportation', 'car', and 'red'to indicate

its coarse semantic, fine semantic, and color. Further up the hierarchy, labels could include descriptors related to the object's specific function or detailed features. For instance, a 3D architectural model might be labeled as 'residential', 'multi-story building', and 'with a garage'to convey its functional and structural attributes. These hierarchical labels not only enhance the descriptive richness of 3D object representations but also play a crucial role in applications spanning computer graphics, virtual reality, machine learning, and game development. They enable a more comprehensive understanding of an object's characteristics and context, ultimately enhancing its usability and interpretability across diverse domains.

Most open-set learning methods are based on the assumption that the labels of object categories are at a single level. This setting implies that existing methods consider category visibility only between the training and testing sets at a single layer, ignoring the distribution overlap among categories at other layers. Besides, the existing strict setting of the single-level open-set also brings more deviation bias to the feature space of unseen categories. In this paper, we do not follow the strict assumption of the existing open-set approach regarding the disjoint distribution between training and testing sets, instead leveraging the different overlaps across multi-level categories to enhance representation and retrieval performance. we term this scenario, where the training and testing set share a partial label space of coarse categories but are completely disjoint from fine categories, as Semi-Open Environment .

B Semi-Open Dataset Generation

Considering that there is no existing dataset for semi-open 3DOR task, we generate four semi-open 3DOR datasets, including SO-ESB, SO-NTU, SO-MN40, and SO-ABO, based on the public datasets ESB [13], NTU [5], ModelNet40 [35], and ABO [6], respectively. We add coarse category labels for each object according to the basic geometric shape and remove some objects that are difficult to categorize based on their shapes. Thus, the number of 3D objects in the released datasets is smaller than that of the original datasets. Specifically, the coarse labels for the four datasets are:

SO-ESB : Flat-thin wall components, Rectangular-cubic prism, Solids of revolution.

SO-NTU : Rectangular-cubic prism, Solids of revolution, Miscellaneous shape.

SO-MN40 : Rectangular-cubic prism, Solids of revolution, Miscellaneous shape.

SO-ABO : Rectangular-cubic prism, Solids of revolution, Miscellaneous shape.

Then, we split the fine categories into seen categories for training and unseen categories for testing, the training and testing sets share the same coarse label space according to the semi-open environment setting. The statics of the four semi-open 3DOR datasets are shown in Table 5. Each object has three modalities including multi-view, voxel, and point cloud. The multi-view data are rendered by Blender 3.0 2 to get 12 images for each object. The point number for point cloud is 1024, and the dimension for voxel data is 32 × 32 × 32 . Point cloud and voxel data are sampled by Open3D 0.13.0 3 . The illustration of hierarchical labels and multiple modalities of the semi-open 3DOR datasets are shown in Figure 5.

C Method Details

The overall framework of HERT is composed of the Hierarchical Retrace Embedding (HRE) and the Conditional Structure Learning (SET) modules. Given 3D objects represented by multiple modalities, common-used

Algorithm 1 Training the HRE module Input : Basic features { r k } M k =1 and their coarse labels { y c i } N i =1 of N instances { o i } N i =1 . Parameter : µ = 0 . 5 . Output 1 : Unified embeddings { c i } N i =1 . Output 2 : Retrace embeddings { f i } N i =1 . 1: Let epoch = 0 ; 2: Initialize multi-modal auto-encoder A m ; 3: Initialize aggregation function T ; 4: Initialize retrace auto-encoder A r = { Ψ , Φ } ; 5: Encoding coarse labels as retrace encoding { e i } N i =1 ; 6: while epoch ≤ 40 do 7: Get unified embedding for each modality { c 1 i , c 2 i , c 3 i } N i =1 = A m ( { f 1 i , f 2 i , f 3 i } N i =1 ) . 8: Get unified embedding for each object { c i } N i =1 = T ( { c 1 i , c 2 i , c 3 i } N i =1 ) . 9: Calculate the Homology Loss L homo = L homo ( { c i } N i =1 ) . 10: Calculate the Bi-reconstruction Loss L br = L br ( { c i } N i =1 ) . 11: Get retrace embedding of each object { r i } N i =1 = Ψ( { c i + e i } N i =1 ) . 12: Get mixed feature of each object { ˆ m i } N i =1 = Φ( { f i } N i =1 ) . 13: Calculate the Retrace Cross-Entropy Loss L ce = L ce ( { ˆ m i } N i =1 ) . 14: Calculate loss for the HRE module L HRE = µ ( L homo + L br ) + (1 -µ ) L ce . 15: if L HRE does not converges then 16: Update parameters of A m , T , and A r by L HRE . 17: epoch + = 1 18: else 19: Break. 20: end if 21: end while 22: return Unified embeddings { c i } N i =1 , and retrace embeddings { r i } N i =1

backbones are used to extract the basic features for each modality. Specifically, the basic features of multi-view, point cloud, and voxel are extracted by the pre-trained MVCNN [30], PointNet [25], and 3D ShapeNets [35], respectively.

The implemental details of the HRE module are provided in Algorithm 1, the HRE utilizes two hierarchical auto-encoders in series. The multi-modal auto-encoder A m encodes the multi-modal basic feature of 3D objects to get the unified embeddings. The retrace auto-encoder A a encodes the coarse label aligning with the unified embeddings to get retrace embeddings.

As shown in Algorithm 2, we provide the implemental details of the SET module, the superposed hypergraph structure G is employed to capture the local coherent and global entangled correlations under the constraint of hierarchical category information. Then, hypergraph convolution is employed to utilize the collaborative high-order correlation under the guidance of this superposed hypergraph. Finally, we use the hypergraph convolution (HGNNConv) and memory bank M to generalize the structure-aware knowledge to generate unbiased features for unseen categories.

Our experiments were conducted on a Tesla V100-32G GPU and an Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. Besides, We provide more hyper-parameters of HERT in Table 6. The hyper-parameters 'k' in the SET module are set to 12, 10, 50, and 50 for SO-ESB, SO-NTU, SO-MN40, and SO-ABO, respectively.

D Comparison Implement

Under this semi-open setting, since there is no 3D object retrieval method designed specifically for this muli-level settings, we choose the current state-of-the-art methods of close-set 3D retrieval (SDML [11], CMCL [14], MMSAE [34]), close-set multi-label retrieval(Tran-GCN [15]), and open-set 3D learning (C2AE [21], HGM 2 R [7]). For each method, we add a multi-label learning mechanism [15] on their basis for comparison. The coarse embeddings are supervised by the coarse labels by cross-entropy loss.

SDML [11]: SDML is a metric learning based method for multi-modal retrieval, which learns projection functions for different modalities independently. We implement the multi-label embedding by adding an auto-encoder after the encoder of the DSAE module.

CMCL [14]: CMCL is a typical 3DOR network based on cross-modal center loss, which is designed to eliminate cross-modal discrepancy. We construct an auto-encoder and take the center representation of each object to generate the coarse embeddings.

MMSAE [34]: MMSAE is an auto-encoder based method that compresses features from different modalities into a unified space. We take the semantic code vector of the MMSAE network as the input and generate coarse embeddings by another auto-encoder.

TranGCN [15]: TranGCN is a multi-label retrieval method based on a graph convolutional network. We treat the fine and coarse labels as the two labels of supervision.

C2AE [21]: C2AE is an open-set 3D object recognition method with a two-step structure, which is designed to identify objects of the out-of-the-distribution categories. We add an auto-encoder to generate the coarse embeddings after the first step.

HGM 2 R [7]: HGM 2 R is a hypergraph-based 3DOR method specifically designed for the open-set environment, which assumes no distribution overlap between training and testing set. We construct an

(a) Retrace embeddings.

(b) Final embeddings.

Rectangular-cubicprism

·Solidsofrevolution

desk

curtain

guitar

tent

bed

door

bowl

vase

dn3

lamp

wardrobe

laptop

cone

toilet

xbox

●stairs

ty_stand

·bottle

bookshelf

·plant

·sofa

●dresser

·stool

car

●mantel

qnueq

piano

·bench

·person

auto-encoder for coarse embedding after the MM3DOE module, and we construct only the knn-based hyperedges in the SAIKL module.

E Performance Result

The HRE module is designed to obtain multi-level embeddings of objects based on hierarchical categories. Therefore, we provide two visualized results of the retrace embeddings and final conditional embeddings of unseen categories in Figure 6(a) and Figure 6(b), respectively. These two t-SNE visualizations show that the proposed HRE and SET modules can fully leverage the hierarchical category information into multi-level retrace embedding and generalize them to unseen categories. This approach better captures the hierarchical semantic correlations in the wild and provides a practical framework for the representation learning of multi-label tasks towards a semi-open environment.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: We provide comprehensive motivations and detailed introductions of all the contributions in the abstract and introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the current limitations of this work in the conclusion.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provided a formalized, step-by-step explanation of each procedure in the methodology.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed experiment and reproduction settings in the experiments and appendix.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are in the preparation of the codes, models, and datasets, and they will be released at the earliest opportunity.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all experimental settings and details settings in the experiments and appendix.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We explain the evaluation protocols of the experiments.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the sufficient information on the computer resources in appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read and adhered to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the broader impacts of this work in the conclusion.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have obtained and adhered to the permissions for the datasets used and have cited them in the paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release the dataset and provide the corresponding documentation immediately upon the paper's acceptance.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the open-set 3D object retrieval problem to a semi-open situation where hierarchical semantic labels are considered. The authors leverage the multi-level category information with a proposed hypergraph-based Hierarchical Equilibrium Representation (HERT) framework. This framework consists of a Hierarchical Retrace Embedding module for encapsulating hierarchical semantic information and a Structured Equilibrium Tuning module for learning generalizable features according to the constructed superposed hypergraph from both local coherent and global entangled correlations.\n\nSTRENGTHS:\nThe proposed semi-open 3D object retrieval setting is reasonable and more practical for real-world applications. \nThe idea of introducing a hierarchical semantic graph is suitable for this task and the proposed framework is technically sound. \nExperiment results could demonstrate the effectiveness of the proposed framework, the ablation study comprehensively shows the functionality of the coarse label.\n\nWEAKNESSES:\n1. The mathematical symbols within this article are chaotic, giving the following examples:\n\n    1.1. The notations of basic features in Line 152 ($\\mathcal{F}$) and Line 155 ($f$)are inconsistent. Does the aggregation function $\\mathcal{X}$ in Line 156 have the same meaning as $\\mathcal{T}$ in Line 155? \n\n    1.2. The description in Lines 157-163 does not match the illustrated pipeline in Figure 2, which shows that $\\mathcal{A}_r$ takes $e_i$ as a side input (with a concatenate operation? This is also confusing. I think $e_i$ is concat with $c_i$ before inputting into $\\mathcal{A}_r$). How exactly can the Retrace Encoding $e_i$ be obtained? Where are the reconstruction features $\\hat{m}_i$ mentioned in Line 163 be used? Does the $\\hat{m}_i$ actually the $f_i$ in Figure 2? However, $f$ has been utilized to represent the basic feature in Line 155. I suggest adding more essential symbols and legends in Figure 2 to align with the descriptions. Similar things happen in the Appendix C Algorithm 1.\n\n    1.3. Besides, it is not recommended to over-defined some ''spaces'' that are not utilized in the rest of the article, such as ''retrace space $\\mathbb{S}_r$''  (Line 159 retracte -> retrace? Line 160 $\\mathbb{S}_f$ -> $\\mathbb{S}_r$?) and ''mixed space $\\mathbb{S}_x$''. These may bring potential typos and are unhelpful in understanding the paper.\n\nQUESTIONS:\n1. In Section 3 of the problem setup, the traditional 3D Object Retrieval setting contains no multi-level labels, right? Bringing this concept with the proposed multi-level labels without explanation is unsuitable, which may lead to confusion.\n2. The authors construct four datasets for evaluating their framework in their proposed task. However, as listed in Appendix B, each dataset is equipped with only 3 coarse categories according to the shape of the objects, which increases my concerns about the efficiency of the proposed hierarchical semantic graph in more complex situations. Are there any possible coarse categories to be included in the real scenes? \n3. As the proposed framework is trained in a two-stage scheme, It would be better to include a comparison of model complexity and training/inference times, \n4. Will the proposed datasets be released?\n\nLIMITATIONS:\nThe authors have mentioned the potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework called the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) for semi-open 3D object retrieval. The proposed framework addresses the practical scenario of semi-open environments where the training and testing sets share a partial label space for coarse categories but are completely disjoint for fine categories. The HERT framework comprises two main modules: Hierarchical Retrace Embedding (HRE) and Structured Equilibrium Tuning (SET). The authors also generate four semi-open datasets to benchmark their approach and demonstrate its effectiveness through extensive experiments.\n\nSTRENGTHS:\n- **Novel Framework**: The introduction of the HERT framework for semi-open 3D object retrieval is a novel contribution that fills a gap in the current literature.  \n&nbsp;\n- **Hierarchical Approach**: The use of hierarchical labels to better capture the multi-level semantics of 3D objects is innovative and aligns well with real-world scenarios.  \n&nbsp;\n- **Comprehensive Experiments**: The authors conducted extensive experiments on four newly generated datasets, providing strong empirical evidence of the effectiveness of their approach.  \n&nbsp;\n- **Clear Problem Definition**: The paper clearly defines the semi-open environment and distinguishes it from traditional open-set and closed-set scenarios.\n\nWEAKNESSES:\n- **Complexity**: The proposed framework is quite complex, which might make it difficult for practitioners to implement and extend.  \n&nbsp;\n- **Lack of Baseline Comparisons**: While the paper compares HERT against state-of-the-art methods, more diverse baseline comparisons, including simpler methods, could provide a clearer picture of the improvements.  \n&nbsp;\n- **Citation**: Please add citations for the methods you compared against in the tables (Tab. 1, 2, 3).\n\nQUESTIONS:\n- How does the HERT framework perform in scenarios with more than three levels of hierarchical labels?  \n&nbsp;\n- What are the computational requirements for training and deploying the HERT framework, especially in terms of time and resources?  \n&nbsp;\n- Can the proposed method be extended to other domains beyond 3D object retrieval, such as text or image retrieval?\n\nLIMITATIONS:\nThe authors have provided discussions about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper introduces a more practical Semi-Open Environment setting for open-set 3D object retrieval with hierarchical labels, in which the training and testing set share a partial label space for coarse categories but are completely disjoint from fine categories. A novel framework, HERT, is proposed for this task. The HRE module is designed to overcome the global disequilibrium of unseen categories. Besides, the SET module is designed to utilize more equilibrial correlations among objects and generalize to unseen categories. Furthermore, four semi-open 3DOR datasets are generated with multi-level labels for benchmarking. The proposed method achieves good performance.

STRENGTHS:
This paper is easy to read. This paper targets an interesting problem, open-set 3D object retrieval. The proposed method is also interesting.

WEAKNESSES:
The proposed method is a little simple, making the technical contribution unclear. Maybe the writing should be improved to highlight the technical insights.

This paper introduces a new task. It would be better to discuss the difference between this new task and existing ones in the technical aspect. It also helps if some promising research directions for this new task could be provided.

QUESTIONS:
Please clarify the technical insights.

LIMITATIONS:
The writing can be improved.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Semi-Open Environment setting for open-set 3D object retrieval, addressing the limitation of existing methods that only consider single-layer labels and assume no overlap between training and testing sets. The authors propose the Hypergraph-Based Hierarchical Equilibrium Representation (HERT) framework, which includes the Hierarchical Retrace Embedding (HRE) module to balance representations across multi-level categories and the Structured Equilibrium Tuning (SET) module to handle feature overlap and class confusion through high-order correlations in a superposed hypergraph. They also create four semi-open 3D object retrieval datasets with hierarchical labels to benchmark their approach. Experimental results show that their method effectively generates and generalizes hierarchical embeddings of 3D objects, outperforming current state-of-the-art retrieval methods in semi-open environments.\n\nSTRENGTHS:\nI generally appreciate the study angle around the fine-grained structure across different 3D object categories, which could bring more insights for related 3D research. Overall, the proposed architecture is composed of reasonable components along with reasonable loss functions. \n\nMotivating by potential contradict optimization for open-set learning, they proposed semi-open 3D object retrieval task. To examine the performance, they also design 4 datasets based on existing 3D object datasets. The proposed framework outperforms all the aselines.\n\nWEAKNESSES:\nI have several major questions and concerns:\n\n- What does the exact coarse labels mean by author in Figure 1? I failed to see the relationship among solid of revolution, rectangular-cubic, and helicopters. I doubt the meaningful of the proposed multi-levels of 3D objects: it is too coarse to be a intermediate level. Can the author show several qualitative visualization around several 3D objects which share different fine-grained levels but with the same coarse level? I'd like to check the results with randomly sampled from SO-ESB, SO-NTU, SO-MN40, and SO-ABO, respectively. Generally, for each dataset, randomly sampled 5 sets of objects with the different fine-grained-level labels while the same coarse-level label would be very helpful. \n- The proposed HRET framework is a bit ad-hoc: consider if we have multiple levels (more than 2 levels) shared across a large amount of 3D objects, we will need more levels of auto-encoders given the current design logic.\n- As mentioned in line 448-449, some of the 3D objects will be throwed away due to the improper design of coarse labels. I failed to this the use of basic geometric shape to fit nowadays 3D vision research. For example, Objaverse(-xl) have a lot of comprehensive objects consisting of multiple different basic geometric shape. Those \"complex\" object may lie in the interest for us to perform retrieve tasks. \n\n---\nHere are some minor points:\n- How do you determine the corase label if one object actually contains multiple separate simple shape?\n- 3DOR is first mentioned in #line15 without full form. This is unfriendly to readers who are not familiar with this direction. \n- In Figure 6 (b), there are some overlap fine-grained objects. Could you please show some? For example, in the left brown ones, they are mixed with yellow and light-purple dots; in the top pink group, there is a green dot.\n\nQUESTIONS:\nplease address the concerns raised above. \n\nTo be specific, I'd like to see a total 25 randomly sampled hierarchy examples from the proposed SO-ESB, SO-NTU, SO-MN40, and SO-ABO.\n\nLIMITATIONS:\nThe current version discussed limitations in #line 311~314.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}