OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields
Authors: Lisa Weijler
Sebastian Koch
Fabio Poiesi
Timo Ropinski
Pedro Hermosilla
Venue: NeurIPS 2025 poster ()
Keywords: 3D scene understanding
open vocabulary
NeRF
hyperbolic embeddings
Primary area: deep_learning
TL;DR: We present a hierarchical representation learning method using hyperbolic spaces for Neural Radiance Field.

Abstract: Modeling the inherent hierarchical structure of 3D objects and 3D scenes is highly desirable, as it enables a more holistic understanding of environments for autonomous agents. Accomplishing this with implicit representations, such as Neural Radiance Fields, remains an unexplored challenge. Existing methods that explicitly model hierarchical structures often face significant limitations: they either require multiple rendering passes to capture embeddings at different levels of granularity, significantly increasing inference time, or rely on predefined, closed-set discrete hierarchies that generalize poorly to the diverse and nuanced structures encountered by agents in the real world. To address these challenges, we propose OpenHype, a novel approach that represents scene hierarchies using a continuous hyperbolic latent space. By leveraging the properties of hyperbolic geometry, OpenHype naturally encodes multi-scale relationships and enables smooth traversal of hierarchies through geodesic paths in latent space. Our method outperforms state-of-the-art approaches on standard benchmarks, demonstrating superior efficiency and adaptability in 3D scene understanding.

OpenHype: Hyperbolic Embeddings for Hierarchical Open gpen-Vocabulary Radiance Fields
Lisa Weijler
Sebastian Koch Ulm University
Fabio Poiesi Fondazione Bruno Kessler
Timo Ropinski Ulm University
TU Wien
Pedro Hermosilla
TU Wien
Abstract
Modeling the inherent hierarchical structure of 3D objects and 3D scenes is highly desirable, as it enables more holistic understanding of environments for au- tonomous agents. Accomplishing this with implicit representations, such as Neural Radiance Fields, remains an unexplored challenge. Existing methods that explicitly model hierarchical structures often face significant limitations: they either require multiple rendering passes to capture embeddings at different levels of granularity, significantly increasing inference time, or rely on predefined, closed-set discrete hierarchies that generalize poorly to the diverse and nuanced structures encountered by agents in the real world. To address these challenges, a novel approach that represents scene hierarchies using a continuous hyperbolic latent space. By leveraging the properties of hyperbolic geometry, OpenHype natu- rally encodes multi-scale relationships and enables smooth traversal of hierarchies through geodesic paths in latent space. Our method outperforms state-of-the- art approaches on standard benchmarks, demonstrating superior efficiency and adaptability in 3D scene understanding.
propose OpenHype,
Introduction
Understanding 3D scenes requires capturing their inherent semantic hierarchical structure, as objects and their relationships are naturally organized in a multi-scale fashion [28]. For example, objects composed of multiple parts, such as the arms and legs of chair, and can also be semantically grouped at a higher level, e.g. a fridge and cabinets within a kitchen, or couch and TV within living room. Preserving this hierarchical organization is critical for range of applications, including semantic segmentation, scene reconstruction, and object detection.
Natural language provides human-readable interface for interacting with computational representa- tions of 3D scenes. However, enabling such an interaction requires more than the mere processing of language input aligned with visual representations; it also requires the ability to model semantics multiple scales and to reason about long-tail and abstract concepts. This requirement partic- ularly pronounced in the task of 3D vocabulayy segmentation from Neural Radiance Fields (NeRF) 229, where a holistic understanding of the scene is essential for handling queries varying granularity. Despite its importance, most existing methods overlook the hierarchical nature of 3D scenes. Instead, they treat 3D scenes as flat collections of independent elements and process the entire scene uniformly, generating independent embeddings for each spatial position [19 15 25 7). often resulting in suboptimal performance when hierarchical understanding is required [
Iweijler htcpller@@vl..tvevvev www.://wwwwww...ccccm lisawejlee.github.ii
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

One promising direction for improving 3D scene understanding is the incorporation of hierarchical representations [1] 35 T. Introducing such inductive biases during training has been shown to
enhance performance on tasks like semantic segmentation and object localization [5 I. However, existing NeRF-based approaches often require multiple rendering passes to obtain embeddings at different resolutions [15 [35] or depend on predefined, discrete hierarchical levels [5 D. These constraints limit the applicability in complex real-world scenarios, where hierarchical structures are not neatly constrained by fixed or discrete resolutions.
To overeome these challenges, we propose OpenHype, a novel method for open-vocabulary segmen- tation on NeRFs that leverages hyperbolic spaces to embed hierarchical structures in continuous manner. Thanks to the exponential expansion rate of hyperbolic spaces [34). we are able to naturally embed hierarchies continuously, and later navigate such hierarchies by simply traversing geodesic paths within the latent space. OpenHype naturally accommodates varying levels of abstraction with- out the need for explicit discrete hierarchies or additional rendering passes, offering a more flexible and efficient framework for 3D scene understanding. We evaluate OpenHype both quantitatively and qualitatively on existing benchmarks, where we consistently outperform ofte-of-the-arr methods.
2 Related Work
Open-vocabulary 3D scene understanding. Open-vocabulary scene understanding has gained considerable attention in recent years, shifting the paradigm in object detection and segmentation In 2023, this concept was 33 OpenScene jointly embeds 3D point features with image pixels and text to enable zero-shot 3D scene understanding. Follow-up works (2) 19
from a closed set of object classes an open set 88 23 45 47
popularized in the field of 3D scene understanding with OpenScene
13 7 25 35 extend this idea to other 3D representations, such as Neural Radiance Fields (NeRF) 29 Distilled Feature Fields (DFFs) [9] addresses the challenge of semantic scene decomposition in NeRF, enabling query- local editing of 3D scenes. Later, Language Embedded Radiance Fields (LERF) [5] integrated language embeddings from foundation models like CLIP K7 into NeRF, enabling open-ended language queries in 3D. LERF learns dense, multi-scale language field within NeRF by volume- renderin CLIP embeddings along training rays, and incorporates DINO features (3] for language regularization. Similar to LERF, OpenNeRF 77] encodes CLIP features within NeRF. However, OpenNeRR utilizes pixel-wise CLIP features instead of global CLIP features, resulting in simpler architecture that eliminates the need for DINO regularization. Unlike NeRF- based methods, LangSplat (35] leverages the 3D Gaussian Splatting (3DGS) [4] representation to encode language features distilled from CLIP into each 3D Gaussian, enabling efficient rendering and open-vocabulary 3D understanding. Concurrently, Feature 3DGS (50) extended 3DGS toward more general multi-modal feature learning. While LangSplat employs an autoencoder to compress high- dimensional CLIP features before integrating them into the Gaussian representation, Feature 3DGS introduces a lightweight upsampling network that enables training low-dimensional feature fields jointly with radiance information. In contrast, Semantic Gaussians [9) bypass explicit distillation during training by projecting 2D semantic features onto 3D Gaussians through pixel-Gaussin correspondence and multi-view feature aggregation. While it demonstrates fine-grained capabilities such as part segmentation through features inherited from 2D models like VLPart, its semantics remain per-Gaussian and non- nin-bierarlhical, lacking explicit multi-level organization as enabled by our approach. Recently, several methods also extend 3DGS approaches with feature quantization methods: Shi et al. [40], Wu et al. [44]. In this paper, we extend these works by directly learning continuous hierarchical embeddings within NeRF.
Hierarchical 3D scene understanding. Hierarchical representations have long been central to how neural networks interpret complex data, mirroring the multi-level abstraction observed in human perception. In computer vision, convolutional neural networks (CNNs) naturally build hierarchical features, progressing from low-level edges high-level object parts and categories 21 38 I.. However, despite their success on downstream tasks, those hierarchies are fixed, defined by the network architecture, and treated as a black box, making these models difficult to interpret. In 3D scene understanding, hierarchical representations have also gained attention, and some approaches do incorporate such inductive biases. Search3D [42] for example, segments 3D scene entities, such as object instances or object parts, from point clouds, based on arbitrary textual queries. Thus, it can construct hierarchical open-vocabut vocabulary 3D scene representations with two pre-defined levels, enabling 3D search at two semantic granularity levels. DeCompositional Consensus [31 also incorporated

---

hierarchical understanding in 3D point clouds, by maximizing agreement between segmentation hypotheses and their decomposed parts with a part segmentation network. However, those methods depend on large data sets for training and cannot be directly applied to NeRF representations.
Hierarchical representations in radiance fields. In NeRF, several methods have been proposed to I 44]. GARField [16] decom- poses 3D scenes into a hierarchy of semantically meaningful groups by optimizing a ceale-conditioned 3D affinity feature field, allowing point in space to belong to different groups at different scales. This field is optimized using 2D masks obtained from Segment Anything (SAM) [7] foundation model. More related to our work, LERF [15] incorporates hierarchical language embeddings into NeRF by conditioning the 3D feature field with a scale parameter. Supervision of such a hierarchical model was achieved by computing CLIP embeddings with bounding boxes of multiple sizes. During inference, multiple rendering passes are required to obtain multiple feature maps. LangSplat (35) leverages SAM to learn hierarchical semantics at three different scales, reducing the extensive language field queries to three. Concurrent to our work and building on LangSplat, Hi-L-Splat [49] further refines hierarchical understanding through multi-scale latent-space disentanglement strategy. Hi-LSplat constructs a hierarchical 3D semantic tree via clustering and contrastive learning, yet still relies on the three predefined scale levels of SAM. Recently, N2F2 [ has proposed to avoid relying scale-conditioned language fields and instead learn a single feature field that contains hierarchical information. N2F2 learns single feature using Matryoshka Representation Learning [22] that can be later processed to derive the language embedding at the desired granularity. However, the number of hierarchical levels is fixed and equal to the number of dimensions of the embedding, which might lead to insufficient representation power the number of dimensions is too small or too large. The paper also proposes to learn weighted sum of the different hierarchical embeddings from general concepts to reduce the number of queries to one; however, as our experiments will show, this results in subpar understanding compared to OpenHype.
incorporate hierarchical representations during learning [16 15 35
In contrast to previous approaches, we propose to learn single feature field in hyperbolic space that naturally encodes the hierarchy of the scene, allowing for continuous navigation of such hierarchy by interpolating the embedding along a geodesic path. Furthermore, contrary to LERF ([5), LangSplat (35), and Hi-LSplat [49], our method only requires a single rendering pass, and our continuous hierarchical embeddings in hyperbolic space lift the constraints of discrete hierarchical levels inherent to N2F2 [1]
3 Preliminaries
In this section, we briefly introduce the key concepts of hyperbolic ge- ometry on which OpenHype relies. Hyperbolic spaces are Riemannian manifolds with constant negative curvature, which causes the volume of ball grow exponentially with its radius. By contrast, Euclidean space has zero curvature and therefore exhibits only polynomial volume growth. Intuitively, hyperbolic spaces provide "more room" than Eu- clidean ones, which makes them well-suited for embedding trees with arbitrarily small distortion (39 Several isometric models exist for rep- resenting hyperbolic spaces 34. among which the Poincaré ball and the Lorentz model are the most widely used for hierarchical embeddings. In OpenHype, we use the Lorentz model L", also known as the hyper- boloid, due its numerical stability, computational efficiency during
Lorentz
Geodesic
Tangent
Boundary
optimization 30 32), and a more stable distance function 34 32
which we use in our hierarchical
loss.
Geodesics. The shortest path between two points in LL" is called a geodesic. These paths naturally curve inward, rather than staying straight in Euclidean geometry, due to the curvature of the space. In our approach, we use geodesics between the origin of the hyperboloid and feature embeddings to traverse object hierarchies. Distances on the Hyperboloid are measured using the Lorentzian geodesic distance d (x. y)
Exponential and logarithmic maps. Another important concept of hyperbolic geometry is exp. and log. maps, which allow us to move from the tangent space, at point T.", onto the Hyperboloid LK and back, respectively. Since tangent spaces are Euclidean spaces, the outputs of Euclidean
3

---

Auto-encoder training
(c) NeRF inference
Hyp-
Euci
Hyp-
am
R.G.B
Network
Dencity
CLIP
D
feat
Rendering
CLIP
D
(b) NeRF training
Prompl
"door of washing machine"
mnn Nerwark
R.G.8
4F13
BBC
0
Density
Sintarity
DOB
Hyp. four
Figure I: Method overview figure. (a) Our auto-encoder transforms a hierarchy of language-aligned features into hyperbolic space. (b) Then, we supervise a NeRF model predict the hierarchical hyperbolic features together with color and density. (c) Lastly, during inference, we can traverse in continuous manner the scene hierarchy by following a geodesic path hyperbolic space and perform open-vocabulary queries for a given text prompt.
network layers can be interpreted as vectors in tangent space of the Lorentz model. These vectors can then be easily mapped onto the hyperboloid. enabling the learning of embeddings in hyperbolic space. Similar to Desai et al. [6), in OpenHype, we only consider the tangent space of the Hyperboloid's origin O [0, /1/c]. We refer the reader to the supplementary material for additional formal definitions and equations.
Method
Our hierarchical framework OpenHype takes as input a collection images representing the scene with the associated camera parameters, and trains NeRF model to produce 3D consistent hierarchical semantic features Fig. [ However, contrary to existing approaches, OpenHype features are hyperbolic embeddings and thus allow for continuous hierarchy traversal along geodesic paths. As illustrated in features, extracted
Fig. T (a), OpenHype utilizes an auto-encoder that transforms language-aligne from the scene images, to the hyperbolic space, which encodes the hierarchical structure of the scene (Sec. 4.1]. Our NeRF model is supervised with hyperbolic features (Sec. 4.2 and Fig. (b)), which then, during inference, can be queried in continuous manner to traverse multiple levels of the scene hierarchy to acquire multi-scale responses for given input text prompt (Sec. 4.3 and Fig. (c).
4.1 Hyperbolic auto-encoder
Our auto-encoder takes as input language-aligned D-dimensional features, F E NXD. of N different objects (parts) present in the scene, O, and the hierarchical relationship between them, R E N where Rij = 1 if object is the parent of object j and R, = 0 otherwise. The encoder processes F,, mapping it to a hyperbolic latent H -ddimnnsional representation. H. E N this hyperbolic space, object oj, with associated feature h,. is ancestor of object o₁, with associated feature h4, if dy (h,, O) < (1u, O) and ahh, h4) R 0, i... h2 is closer to the origin than h; and hj lies close to the geodesic path connecting h, and the origin O. This results in leaf nodes (e.g. parts of objects) embedded close to the boundary of the hyperbolic space and objects high in the hierarchy embedded close to the origin O. At the same time, elements of a branch in the hierarchy are embedded close to the same geodesic path connecting O and point the boundary of the hyperbolic space. The decoder then takes the hyperbolic features and reconstructs them back to the original language features living in Euclidean space. Fig. (a) illustrates this process.
Note that this isn't I boundary of the space in mathematical sense but rather a "boundary orbit" that we introduced by limiting the maximum geodesic distance to the origin to numerical stability.

---

Loss. In order to enforce such structure in the hyperbolic space, we employ contrastive learning to train the auto-encoder. From batch of features processed. we compute the following loss:
f= La + Lu + L. La is the standard contrastive learning loss 4 37 6 but minimizing hyperbolic distances instead of maximizing cosine similarities. L.a uses the same contrastive approach as La but applied to the exterior angle, ा 2Oh,h; a(h,, A4)) Fig. (a) illustrates the exterior angle. Lastly, C, is the reconstruction loss of the decoded language-aligee -allgned features, defined as the Mean Squared Error (MSE) between the reconstructed feature f and the input feature ft.
Hierarchical data creation. In order to train the auto- anto-encoder and later supervise the NeRF model, we need to extract the set of objects O associated features F, and the hierarchical relationships R from the collection of input images I. We leverage a recent foundation model for general object segmentation, Semantic SAM (4). to generate set of object masks M from the input images. For each mask m,, we extract the language-alignan the image and processing it with CLIP 37 The cropped image only contains pixels within the bounding box of the object mask my. Moreover, for each image, we compute a hierarchy representing the relationship between objects present in the image. First, the largest masks, those that are not contained in any other mask, are directly associated with the root node, defining the level 4 of the hierarchy. Then, we iterate over all the remaining masks in the image and select those that are only contained within the masks of level 4, These new masks are associated with the corresponding mask of l1, defining the new level l2 in the hierarchy. The process is repeated until there no masks left in the image. Note that, even the same object appears in multiple images, each image will have its own instance of the same object and will be part of different hierarchy. However, due to the generalization power of visual-language foundation models such as CLIP, the resulting features will be similar. As we will show later in our experiments, during the auto-encoding training process, these features will be transformed to the same region of the hyperbolic space, consolidating the hierarchies of multiple images and creating a unique hierarchical structure for the scene.
features f; associated with object oj by cropping
4.2 3D lifting
radiance field defines a function that maps a 3D point x E RR3 and a viewing direction d E S to both a color value e E [o. 1| and a volume density E [0, Mildenhall et al. (29 introduced NeRF, which models this function implicitly using an Multi Layer Perceptron (MLP). The network, parameterized by 0, is trained to regress the radiance values from a set of multi-view images of 20
scene. Formally, the network defined as: fo(x, d) -> (c, o). Later, several works 35 have suggested extending such a design by including additional semantic outputs, enabling zero-shot segmentation capabilities. In this paper, we follow this design and extend radiance fields to produce a hierarchical feature in hyperbolic space, h. Fig. D (b) illustrates this training process.
Loss. Since our features live in hyperbolic space, we require a loss that properly minimizes the distance of such features. Therefore, instead of the commonly used MSE or cosine similarity, we propose to minimize the geodesic distance between the embeddings, di (h,,h,h) include an additional regularization loss that encourages both embeddings to have exactly the same norm, (h,, O)- dr. (h, O))2. This additional loss helps to mitigate errors for embeddings far from O where a small displacement in tangent space leads to a big displacement in hyperbolic space.
Moreover, we
Extrapolated features. To supervise the hyperbolic features produced by the radiance field for a given pixel, we could select the hyperbolic feature of the object associated with it. However, since each image contains hierarchy of objects, each pixel might be associated with multiple objects from different levels in the hierarchy. Additionally, each image contains its own hierarchy of the scene, therefore making it difficult to select consistent feature level for supervision, which might translate to contradictory signal during training. Therefore, we propose to extrapolate the hyperbolic features of the lowest mask in the hierarchy to the boundary of the hyperbolic space and use this extrapolated feature for supervision instead. Assuming that the sub-hierarchies present in different images are encoded in the same region of the hyperbolic space, extrapolating features will produce consistent target hyperbolic features for the supervision of the radiance field.
5

---

4.3
Hierarchy traversal
Once the NeRF model trained, we can generate novel views of the scene with associated hyperbolic features per pixel. These features can then be used for open-vocabulary segmentation of objects at different granularities. Fig. (c) illustrates this process open- vpccbvocabulary query.
Hyperbolic space traversal. The hyperbolic space embeds the hierarchical structure of the scene, where leaf nodes are embedded close the boundary of the hyperbolic space, and objects high the hierarchy are embedded close to the origin. Since the NeRF has been trained to produce features h on the boundary of the hyperbolic space, to traverse the scene hierarchy back to the root node, we can simply follow the geodesic path connecting the feature h to the origin O. This continuous traversal, contrary to existing discrete approaches [13] 35 I. allows us to dynamically sample the geodesic path with the desired number of samples 7 at a low computational cost, resulting in a discrete number of hyperbolic features for each pixel i, {h The set of features
then be decoded using the
auto-encoder described in Sec, 4.1 to obtain a set of aagguaee-alinned features {f [fu)"= describing the different objects of this particular branch of the hierarchy. We can compute the similarity between these features and features from a given text prompt using cosine similarity, resulting set of object-text similarity {s
Aggregation method. Previous works relying on discrete hierarchies have aggregated such similarity vectors using a max operation [15] (35 or by a fixed learned weighted sum operation []. While a max operation can produce good masks for simple queries, such as objects, complex compositional queries require a combination of several multi-scale responses. For such queries, in this work, we advocate for softmax-weighted sum operation, where the list of similarity scores is normalized
by softmax operation before being used to aggregate the scores: d(i) = = 85t where 8 is the original similarity score for pixel at level i and &, is the same similarity score after being transformed by the softmax operation. This approach helps to remove noise the set {5x before aggregation, increasing high similarity scores while decreasing low ones. Unless otherwise stated, we perform all our experiments using softmax.
4.4 Implementation details
In order to reduce the noise in the resulting masks, following LangSplat [35] and LERF [15], we also compute the similarity of the embeddings to four neutral terms: "object", "things", "stuf"" and "texture". " We compute the softmax between the resulting similarity for our query and the similarity of the four neutral terms and pick the one for which the neutral softmax value is the lowest. As in LangSplat (35), the final segmentation mask is defined as the pixels with an adjusted similarity score (after softmax) higher than 0.4.
Experiments
5.1 Hierarchical feature evaluation
To highlight the hierarchical understanding of our method, in the first experiment, we evaluate its ability to successfully process hierarchical vocabulayy queries. Here, models aim to localize and segment objects and object parts in the scene, given hierarchical query.
Dataset. We adapt the recent Search3D dataset 42 to the tasks of open-vocabulary segmentation and localization from radiance fields. This dataset provides instance segmentation masks at two levels of granularity: objects and object parts. The dataset provides annotations for two different sets 3D scans, MultiScan [27] and ScanNet++ [46). We focus on the ScanNet+ scenes, since this dataset supports novel view synthesis evaluation. In particular, we use 20 scenes from different types of rooms and select 30 unique objects and 33 object parts. As our test frames, use the novel view synthesis split of ScanNet++.
Hierarchical queries. For each object-part pair, we define two prompt queries, one to identify the object and another to identify the part of the object. The object query is created by using the object class label "/OBJJ", which aligns with previous work on vecebvocabulary segmentation. The part query is more challenging and specifically designed to measure the hierarchical understanding of the models. This prompt is constructed by combining the labels of the object with the label of the

---

Table 1: Quantitative results on the Search3D 42
dataset (a) and in the LERF [[] dataset (b).
Results show that our hierarchical approach outperforms all baselines in almost all experiments.
Method
ram.
fig.
tea.
kit.
Avg.
Object
Part
Avg.
LSeg
21.7
29.9
16.6
Method
IoU
Acc
loU
Acc
IoU
Acc
OpenSeg
27.7
22.6
50.6
39.0
34.1
LERF
15
25.4
75.0
06.2
26.6
15.4
50.0
LERF 151
28.2
38.6
45.0
37.9
37.4
OpenNeRF 7
28.3
69.0
09.7
38.7
18.2
53.3
OpenNeRF
?
33.7
24.3
59.1
39.4
39.0
SangSplat
35
41.6
68.9
11.0
19.3
25,8
43.3
LangSplat
35
51.2
44.7
65.1
44.5
51.4
N2F2 I
56.6
47.0
69.2
47.9
54.4
Ours
51.4
89.7
19.9
50.8
35.1
69.5
Ours
43.9
59.8
71.2
51.7
54.6
(a)
(b)
object part, resulting in prompts with the following form: "[PART] of a [OBJ]". For some pairs, incorporate additional information in the query to differentiate the object from other objects in the scene, such as "leg of the blue chair".
Metric. We report results on the task of segmentation using Mcan Intersection Over Union (mloU) computed between the predicted mask and the annotated masks, well as performance on the task of localization, quantified as accuracy. For localization, we follow LERF [55 and consider a true prediction if the pixel with the highest similarity score is inside the bounding box of the object.
Baselines. We compare our approach against state-of-ther the-trr methods for open- vocabulary segmenta- tion from radiance fields. We select LERF [15] OpenNerf (7]. and LangSplat (35) as representative baselines. LERF and OpenNeRF exemplify NeRF-based representations, while LangSplat represents approaches based on Gaussian Splatting.
Quantitative results. Segmentation and localization results are reported in Tab. la evaluated on the adapted Search3D benchmark 42. Results show that our method consistently achieves more accurate localization and segmentation predictions than existing approaches across all experiments. For object queries, which involve relatively simple tasks, all methods perform reasonably well in both segmentation and localization. OpenHype achieves improvements with gains of +9.8 in mloU and +14.7 in accuracy. Composition queries, which are designed to detect object parts, pose greater challenge as they require a hierarchical understanding of the scene. All baseline methods show a significant reduction in performance, with some achieving single-digit mloU scores, e.g., LERF [55 obtains 6.2, and OpenNeRF 7 reaches 9.7. In contrast, OpenHype maintains robust performance, outperforming baselines by +8.9 in mloU and +12.1 in accuracy. These results indicate that OpenHype can better capture hicrarchical relationships within the scene than existing methods, effectively mitigating the bag-of-worrd effect that some bascline methods suffer from [I].
Qualitative results. Fig. | provides examples of the results of our method compared all baselines. While most methods are able to segment objects, our method is able to provide cleaner segmentation masks with fewer artifacts. However, when more complex queries describing object parts are evaluated, all methods fail to provide an adequate mask and segment the full object instead. This an indication that existing methods suffer from the "bag of words" problem. Our method, on the other hand, is able to properly detect individual object parts.
Feature visualization. Unlike our basclines. which encode open-vocabulary features in Euclidean space, we leverage a hyperbolic latent space. To gain insights into its structure, we provide visualiza- tions of the hyperbolic latent space. We use CO-SNE [10 reduce the dimension of the features to two and therefore be able to visualize them as images. Fig. 3 presents such visualization for different images of the same scene. First, we can observe that features from different SAM masks that are part of the same hierarchy are embedded within the same geodesic path. Moreover, when we color-code the points with the level the mask hierarchy they belong to, we can see that higher levels are embedded closer to the origin, while lower levels are embedded closer to the boundary of the hyperbolic space. Lastly, we can see that the features of masks of the same object from different views are also embedded in the same geodesic path. These visualizations provide empirical validation
7

---

Ours
LERF
OpenNeRF
LangSplat
printe
printer
ciry
laptop
laptop keyboarrd
Figure Qualitative results on Search3D dataset. While other methods struggle to segment object parts, method successfully segments these complex compositional prompts.
Image
Image
Hierarchical embedding
Multi-view embedding
[Root]
Lmg.
[Leaf]
Img. 2
Figure 3: Visualization of hierarchies embedded on hyperbolic features. We visualize the hyper- bolic embeddings of the hierarchies of two different images of the same scene using CO-SNE (10). By color-coding the embeddings, we can see that hierarchies are properly embedded in the hyperbolic space even if the embeddings come from two different images of the same scene.
that our method is able embed the mask features following the hierarchical structure defined by our masks.
5.2 LERF dataset
In addition to the hierarchical evaluation using Search3D, we provide open-vocabulary segmen- tation and localization experiments on the well-established LERF dataset [15). This evaluation
provides further insights into our improved open-vocabulary representation, aided by our hierarchical
representation learning.
Dataset. For this task, we select the benchmark proposed by LERF 15 and later refined by LangSplat (35). This dataset is composed of four scenes representing different environments. The mloU of the predicted segmentation mask w.r.t. the ground truth annotated masks is reported.
Baselines. We reproduce the results of the same baselines in previous experiment, and also compare against N2F2, a Gaussian Splatting- based approach that the most similar to our method. Since the code for N2F2 [I] was not available at the time of submission, we report results for this dataset as provided the original paper. Moreover, we compare to the image-based approach, LSeg 23 and OpenSeg [8]
Quantitative results. The quantitative evaluation on the LERF dataset reported in Tab. Tb Results show that, even if the dataset is not focused on evaluating hierarchical capabilities of the models, OpenHype is able to provide better segmentation results than all baselines, even surpassing N2F2 0I. recent work with discrete hierarchical representations.

---

Table Ablation study of different components of OpenHype on the task of segmentation on five representative scenes from the Search3D dataset. Rows with the grey background indicate the configuration used in our final method.
Overall
Obj.
Part
Overall
Obj.
Part
Overall
Obj.
Part
CLIP
10.9
16.4
6.4
No Neg.
28.7
45.5
14.9
Dist
28.8
50.6
10.1
Hyp. Tan.
24.7
44.2
8.7
Aggregated
30.7
50.4
14.5
Angle
NaN
NaN NaN
Hyperbolic
26.4
46.6
9.9
Stepwise
30.9
50.4
14.9
Dist Angle
30.9
50.4 14.9
+ Extrap.
30.9
50.4
14.9
(b) Negative prompts.
(c) AE Loss.
(a) Nerf supervision.
Overall
Obj.
Part
Overall
Obj.
Part
Overall
Obj.
Part
Max
29.8
52.5
11.3
19.6
30.9
10.3
10
30.7
50.9
14.4
Mean
30.3
48.8
15.2
16
30.3
49.2
14.8
20
30.9
50.4
14.9
Sum
24.3
40.0
11.3
32
30.9
50.4
14.9
30
30.8
50.5
14.7
Softmax
30.9
50.4
14.9
64
28.4
47.0
13.3
(f) Number of steps.
(d) Path aggregation.
(e) Feature dimension.
5.3
Ablation study
We conduct comprehensive ablation studies to systematically analyze the influence of individual components within our framework, highlighting the effectiveness of our design choices. For the evaluation, we use five scenes (showing different room types and objects) from Search3D.
NeRF supervision. We begin by evaluating the impact of different supervision types used during NeRF training, specifically the pixel-level features employed for distillation. We compare a model trained using CL.IP embeddings from the final mask in the hierarchy of each image (CLIP) with a model trained using hyperbolic embeddings projected into the tangent space (Hyp. Tan.). Both models are supervised using MSE objective function. Morcover, with hyperbolic embeddings directly in hyperbolic space using our hyperbolic loss (Hyp.). Lastly, report results for our full model, which is also trained in hyperbolic space but includes feature extrapolation toward the boundary of the hyperbolic space (Hyp. + Extrap.). Tab. 2a summarizes the results of this experiment. Hyperbolic embeddings outperform standard CLIP features. Training directly in hyperbolic space further improves performance compared to training in the tangent space. The best results are achieved with our extrapolated features, highlighting the importance of effectively addressing multi-view feature inconsistencies for achieving strong performance.
evaluate model trained
Negative prompts. We investigate the impact of using negative prompts final model performance. We compare the results of the same model under three conditions: without negative prompts, with negative prompts combined after aggregation, and with negative prompts integrated at each step of the geodesic path. Specifically, in the first approach, we compute similarity scores separately for the positive and negative prompts along the geodesic path, then combine the results after path aggregation. In the last approach, we combine the similarities of positive and negative prompts at each step along the geodesic path, before aggregation. Tab. [b shows that incorporating negative prompts improves model performance by reducing noise and variation in the similarity maps. Integrating positive and negative prompts each step yields slightly better results than combining them after geodesic path aggregation, suggesting more effective alignment in the inference process.
Auto-encoder loss. We evaluate the contribution of each component in the loss function used to train our auto-encoder. Specifically, we compare models trained with contrastive learning based on the geodesic distance between masks, the exterior angle, and combination of both. Tab. 2c shows that using only the geodesic distance yields high performance. In contrast, training with the exterior angle alone leads to collapse of the network, indicating that it is insufficient standalone supervision signal. The best performance is achieved when both losses are combined, showing that they provide complementary information for learning robust representations.
Path aggregation. We also evaluate different variants of the aggregation method used to combine similarities along the geodesic path. Specifically, compare selecting the maximum cosine

---

similarity, as done in LERF [5] (Max), summing all similarities (Sum), and computing their average (Mean). Moreover, we compare these methods our proposed upproach: weighted sum based on noftmax-normalizer scores. Tab. shows that while averaging (Mean) can yield competitive results, our weighted sum consistently achieves the best performance, highlighting the benefit of adaptively weighting contributions along the geodesic path. For simpler prompts, as the Object prompts, we observed that Max produces slightly better performance than Softmax.
Feature dimension. We investigate the impact of the hyperbolic feature dimension. We train four outo-encoders with feature dimensions of 3 (as used LangSplat [35)) 16, 32, and 64. Tab. shows that, while our method remains relatively robust across different feature dimensions, a dimension of 32 yields the best performance. In contrast, a oow-dimeniional embedding such as 3, as used in LangSplat, appears insufficient for capturing the hierarchical structure present in complex scenes.
Interpolation steps. Lastly, we evaluate the effect of the number of steps used during geodesic path traversal on model performance. Specifically, we test our model with 10, 20, and 30 steps. Tab. 2 shows that our method demonstrates strong robustness the number of traversal steps, maintaining similar performance across all configurations.
Conclusions
In this paper, we introduce OpenHype, a novel framework for open open-vocabulary segmentation of NeRF that leverages the geometry of hyperbolic space to model the hierarchical structure of 3D scenes in continuous and flexible manner. By modeling hierarchies through geodesic traversal hyperbolic space, OpenHype supports reasoning across multiple levels of abstraction without requiring multiple rendering passes or rigid, discrete resolutions. Our extensive evaluations across standard benchmarks demonstrate that OpenHype consistently outperforms existing methods in both quantitative metrics and qualitative analyses. These results highlight the value incorporating continuous hierarchical priors for semantic understanding in 3D NeRF-basee representations. OpenHype is agnostic to the underlying radiance field representation and is equally applicable to Gaussian Splatting. Future work could explore how our hyperbolic space can be adapted to this representation.
Acknowledgements
We acknowledge the EuroHPC Joint Undertaking for awarding this project (EHPCDDEV-2024D11) 029) access to the EuroHPC supercomputer LEONARDO, hosted by CINECA, and the HPC system Vega at the Institute of Information Science, through an EuroHPC Development Access call.
10

---

References
[1] Yash Bhalgat, Iro Laina, João F Henriques, Andrew Zisserman, and Andrea Vedaldi. N2f2: Hicrarchical scene understanding with nested neural feature fields. In European Conference on Computer Vision (ECCV), 2024.
[2] Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jianing Yang. Ayush Jain, Ada Martin, Vincent-Pierre Berges, Paul Vay, Ruslan Partsey, Aravind Rajeswaran, et al. From thousands to billions: 3d visual language grounding via render-supervised distillation from 2d vlms. In International Conference on Machine Learning (ICML), 2025.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self- Self-Suuee Vision Transformers In International Conference on Computer Vision (ICCV). 2021.
[4] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
[5] Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabricl Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[6] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Ramakrishna Vedantam. Hyper- bolic Image-Text Representations. In International Conference on Machine Learning (ICML), 2023.
[7] Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Mare Pollefeys, and Federico Tombari. Opennerf: Open set 3d neural scene segmentation with pixel-wise features and rendered novel views. In International Conference on Learning Representations (ICLR), 2024.
[8] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocahulary image segmentation with image-level labels. In European Conference on Computer Vision (ECCV), 2022.
[9] Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: open-vocabulary scene understanding with 3d gaussian splatting. arXiv preprint arXiv:2403. 15624, 2024.
[10] Yunhui Guo, Haoran Guo, and Stella X. Yu. Co-sne: Dimensionality reduction and visualization for hyperbolic data. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[12] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 2016.
[13] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graphics (SIGGRAPH), 2023.
[15] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Chung Min Kim, Mingxuan Wu, Justin Kerr, Matthew Tancik, Ken Goldberg, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yer Lo, Piotr Dollár, and Ross Girshick. Segment anything. In International Conference on Computer Vision (ICCV), 2023.
[18] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladien Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions Graphicx, 2017.
[19] Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
11

---

[20] Sebastian Koch, Johanna Wald, Mirco Colosi. Narunas Vaskevicius, Pedro Hermosilla, Federico Tombari, and Timo Ropinski. Relationfield: Relate anything in radiance fields. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2025.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NewrIPS), 2012.
[22] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[23] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl, Language-driven semantic segmentation. In International Conference on Learning Representations (ICLR), 2022.
[24] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang. Lei Zhang, and Jianfeng Gao. Segment and recognize anything at any granularity. In European Conference on Computer Vision (ECCV), 2024.
[25] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open-vocabulary segmentation. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations (ICLR), 2019.
[27] Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel X Chang, and Manolis Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[28] Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: A survey. International Journal of Computer Vision (IJCV), 2024.
[29] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[30] Gal Mishne, Zhengehao Wan, Yusu Wang. and Sheng Yang. The numerical stability of hyperbolic representation learning. In International Conference on Machine Learning (ICMI.), 2023.
[31] Muhammad Ferjad Nacem, Evin Pinar Ömek, Yongqin Xian, Luc Van Gool, and Federico Tombari. 3d compositional zero-shi shot learning with decompositional consensus. European Conference on Computer Vision (ECCV), 2022.
[32] Maximillian Nickel and Douwe Kicla. Learning continuous hicrarchies the lorentz model of hyperbolic geometry. In International Conference on Machine Learning (ICML), 2018.
[33] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Opensceme: 3d scene understanding with open vocabularies. /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[34] Wei Peng. Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on pattern analysis and machine intelligence, 2021.
[35] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: language gaussian splatting. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[36] Yansong Qu, Shaohui Dai, Xinyang Li, Jianghang Lin, Liujuan Cao, Shengchuan Zhang, and Rongrong Ji. Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic- hyp hyperplane. In ACM International Conference on Multimedia, 2024.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), 2021.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
[39] Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International symposium graph drawing, 2011.
12

---

[40] Jin-Chuan Shi, Miao Wang, Hao-BBn Duan, and Shao-Hua Guan. Language embedded 3d gaussians for opea-vocabylary scene understanding. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[41] Leslie N Smith and Nicholay Topin. Sonverconv:e Very fast training of neural networks using large learning rates, Artificial intelligence and machine learning for multi-domain operations applications. 2019.
[42] Ayca Takmaz, Alexandros Delitzas, Robert W Sumner, Francis Engelmann. Johanna Wald, and Federico Tombari. Search3d: Hierarchical open- vecabvocyyulary 3d segmentation. IEEE Robotics and Automation Letters, 2025.
[43] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular framework for neural radiance field development. In ACM Trans, Graphics (SIGGRAPH), 2023.
[44] Yanmin Wu, Jiarui Meng. Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Jian Zhang. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. In Advances Neural Information Processing Systems (NeurIPS), 2024.
[45] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang- Groupvit: Semantic segmentation emerges from text supervision. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[46] Chandan Yeshwanth, Yuch- Tuch-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor seenes. In International Conference on Computer Vision (ICCV), 2023.
[47] Nir Zabari and Yedid Hoshen. Open-vocabulary semantic segmentation using test-time distillation. In European Conference on Computer Vision (ECCV), 2022.
[48] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre wre-training. In International Conference on Computer Vision (ICCV), 2023.
[49] Chenlu Zhan, Yufei Zhang, Gaoang Wang, Hongwei Wang. Hi-lsplat: Hierarchical language gaussian splatting. arXiv preprint arXiv:2506.06822 2025.
[50] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide description in the abstract and introduction that reflects the empirical results obtained in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
14

---

Justification: Due to page limitations, we provided such limitations section in the supple- mentary material. Assumptions and realistic results were discussed the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper, and especially the supplementary material, provides all details to reproduce the experiments presented the paper. However, we will also provide the code upon acceptance.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., . a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will make the code publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc cttccc/pppo wwwpc//pppcc/aeeeeeeeeeeeeeeeeee www.://gggeeeeeeeeeeeeeee wwtp///oodussssiiioooo
wwm.iissssiiiiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, we provide the necessary background to understand the main findings in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides experimental results computed as the average over 5 runs. However, due space restrictions, we do not provide standard deviations in the main paper. We will include the full numerical results in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
17

---

Justification: We include in the supplementary material all the details of the hardware re- quired to run our experiments. However, since this work overfits each scene independently. it does not require a large clusters and can be run in standard GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't Itdn't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripps /ttpa////.////////........o/ppaa ctw.e/ieeeiiciicicieeeeenee
Question:
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We include such discussion in the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We do not release large pre-trained model but a methodology to embed hier- archical structures into Neural Radiance Fields. Therefore, there is no safeguard necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use publicly available datasets and credit the authors properly in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, phcors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
19

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: we only use CLIP models but not LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non stn-standard components.
Please refer to LLM policy for what should or should not be described.
20

---

A Theoretical background
In this section, we provide additional information and a formal introduction of hyperbolic geometry concepts used in this work.
Lorentz model. The Lorentz model L is the upper half of two-sheeted Hyperboloid in IR++1_ More formally, this model is the n-dim "unit sphere" in Minkowski space defined as
{ar IR*+1 (ax, w))r. -1/c,, 0},
(A.1)
where c >> denotes the curvature magnitude and (xe a)11 = -1030 + Σi=1 J. is the Lorentzian inner product. The Lorentzian norm is defined ||||| (xx x) The Lorentzian geodesic distance is defined (x. y) vI/c- arccosh(-c(aa y)).
Exponential and logarithmic maps. The tangent space at a point x E IL" consists of Euclidean vectors orthogonal to æ regarding the Lorentzian inner product. The exponential map allows to map vectors from the tangent space of æ, T L", to the Hyperboloid IL". It is defined as
exp_map map (z) cosh( cosb(vC(=l|LLx sinh sinh(Vcl(zll..
(A.2)
|
Its inverse, the logarithmic map. maps = exp_map (z), on the Hyperboloid, back to 2 E T&L" t in the tangent space. is defined
log ma__mapp. (y) arccosh(-c(a, B) (y + cx (a, y)..
(A.3)
Velar, y)))²
Traveling along geodesics. We use geodesic paths to traverse up and down our embedded hierarchy. For this, interpolation (and extrapolation) in the Lorentz model. Let æz y E IL" be points on the Hyperboloid, z E TTT.," tangent vector of the tangent space of æ pointing in the direction of y then move along the geodesic starting going in the direction of using
(tt) cosh(tv sinh(tv/C
t [0,1]. 1].
(A.4)
Vel==l""
Exterior angle. The exterior angle a used for part of our hierarchical loss C, is defined as π LOay, where x denotes the parent of y. More specifically, as introduced by the authors of 6
Yo + soo(( y)
arccos
(A.5)
|| Fn Fnll (c(æ y).
As mentioned in |secionn 3] we only consider the tangent space of the Hyperboloid's origin [0, / I/c]. Feature vectors from the last encoder layer Zene can be interpreted using = = [Zenos 0] E R=++1 as elements of the surrounding Minkowski space. Since (O, 2)1. 0, those vectors are orthogonal to O and hence naturally lie in the tangent space of the origin O ( We further fix the curvature of the Hyperboloid to - 11, being standard choice for working with hyperbolic
spaces. Note that only considering the tangent space of the origin O significantly simplifies the formulas for implementation as shown in [6)
B Limitations
Our work is not without limitations. First, the quality and completeness of the extracted hierarchy and corresponding language-alignet features depend on the performance of the underlying 2D segmentation model. If object parts not segmented in any of the 2D views, they cannot be detected or represented in the final 3D scene hierarchy.
Second, our approach relies on pretrained vision-language models (e.g., CLIP) to align visual regions with open- vocabulary text concepts. These models, while powerful, are trained on internet-scale data and may reflect biases, fail recognize domain-speciffe terms, produce inconsistent representations
21

---

for visually similar parts. This can impact the quality of the resulting scene hierarchy features and query results.
Third, inconsistent representations for visually similar parts introduced by vision-language models, as mentioned previously, introduce noise when supervising the language field across multi-view images. While we mitigate this by supervising with extrapolated features, some inconsistencies remain. A promising direction for future work incorporate uncertainty estimation, allowing the model to filter supervision signals and prioritize views with higher semantic agreement.
Lastly, the fidelity of the rendered and interpolated features relies the quality of the learned hyperbolic latent space. Our approach assumes that semantics and hierarchies are smoothly distributed in this space. While quantitative and qualitative results demonstrate that this holds to reasonable extent, we observe some noise along geodesic paths between rendered features and the origin O which can affect semantic coherence. An interesting direction for future work is to explore hyperbolic variational autoencoders, which extend hyperbolic autoencoders with probabilistic modeling that could improve the smoothness and robustness of latent inter- and extrapolations.
C Broader impact
This work presents a novel method for hierarchical open-vocabulary scene understanding with Neural Radiance Fields. Potential positive societal impacts include applications in robotics, assistive technologies, and augmented and virtual reality. For instance, enhanced scene understanding could improve navigation for autonomous agents, accessibility tools for individuals with visual impairments, and more intuitive human-computer interaction.
However, this technology also presents potential risks. The ability reconstruct and label real- world scenes from images may raise privacy concerns, particularly in uncontrolled or unauthorized environments. Additionally, reliance on large-scale langua-a models introduces risks of bias propagation in scene interpretation.
We encourage responsible data collection, and fair evaluation practices. Careful deployment and user consent mechanisms will be essential to ensuring that the societal impact of this technology remains positive.
D
Experimental details
This section provides additional details on our experimental setups and the resources necessary.
D.1 Compute resources and efficiency
Our method does not necessitate high-performance specialized compute resources for training or inference; most experiments were conducted on a standard GPU setup using an NVIDIA RTX A5000.
In Table D.I. the average running time on a NVIDIA RTX A5000 the main components of our method for an image of size 543x979 with 20 interpolation steps is shown. We can observe that the most computationally expensive task is rendering, which is inherited from Nerfstudio and hence the same for OpenNerf. Note that the rendering time also depends on the eval eval_num_raye wawpu_rraa_pee_chutte variable; in our measurements, it was set to @wal_num__at mww_uuas_ppeeeeeeeeeeee chunk 4096. Interpolation only requires 473 milliseconds for 20 steps along the geodesic path, plus 622 milliseconds to transform hyperbolic embeddings into CLIP em- beddings with decoder. A naive discrete approach would instead require rendering of each level of the hierarchy, multiplying the rendering time by the number of levels in the hierarchy, i.e., 20 in this case. Additionally, our approach only requires single training process, unlike methods that train multiple models.
Regarding memory usage, method does not introduce additional overhead when sampling one level in the hierarchy. However, memory consumption increases with the number of interpolation steps, generating steps feature fields, particularly when decoding features in high-dim. CLIP space. That said, decoding can be performed in batches to manage memory constraints effectively.
22

---

Overall, our method remains substantially more efficient than training multiple models, even in memory-limites limited scenarios.
Table D.1: Time measurements in milliseconds for the main inference steps.
Task
Time (ms)
rendering
1992.214
interpolating
473.471
decoding
622.364
D.2 Hyperbolic auto-encoder
Architecture. We use symmetric auto-encoder with a five-layer encoder and decoder. The feature dimensions for encoder and decoder per layer are [512, 256, 128, 64, 32] and [32, 64, 128, 256, 512), respectively, and the activation function used is GELU [12]
Training details. All auto-encoders are trained for 1000 epochs using Adam W (26) as optimizer with
weight decay value of 1-1 and OneCycleLr [41] as learning rate scheduler. The initial and final division factors are 10 and 1000, the percentage of the cycle (in number of steps) spent increasing the learning rate is 5%. We
batch size of 10 images, where all object features, O, involved in at least one parent-child relationship are used. Since the number of object features, |O], varies between images, the batch size in number of object features varies for each pass (around 60 100 extracted masks/objects per image).
The output feature vectors of the last encoder layer can be interpreted as features on the tangent space of the Hyperboloid's origin O We use the exponential map, exp_mapo- defined in Equation A.2 to project them onto the Hyperboloid before calculating the hierarchical loss parts La L. Before decoding the latent features, the log_mapp. defined in Equation A.3) is applied to map them back to the tangent space. In theory, the hyperbolic space does not have boundary as depicted in the schematic illustration of (section 3] However, since distances grow exponentially when moving away from the origin, create "boundary orbit" by limiting the maximum geodesic distance to the origin to ensure numerical stability.
Hyperbolic latent.
Loss. The loss consists of three parts, L = the latent vectors on the Hyperboloid. As contrastive loss for an object og in an image, we use
La La L.. La and La are contrastive losses applied
(xp(s(fi,$t//))
Lcontrast (o.) = -log
(D.I)
+ NN_neg exp(s(fe,5))/
maximize the similarity ((,,)) between positive, f and minimize between negative pairs, f7- For La, the similarity the negative geodesic distance the Hyperboloid, s -dg, and for La, the similarity is the negative exterior angle, as defined in Equation A.5 s -a. For the temperature, we 0.2. Each object o; used for the contrastive loss calculation has exactly namely its direct parent. The number of negatives denoted as y_neg, varies per object as it depends on the depth of the hierarchy the object is involved in. We use all objects of the same image that not part of the object's og direct hierarchy as negatives. In other words, we ignore the children or higher-level parents (e.g., parent of the direct parent) of object o, for its contrastive loss calculation, but we include siblings (objects on the same level that have the same parent) and their children as negatives avoid collapse of siblings on the same level the same geodesic path. As final C, and La losses we average over the respective loss Loontrast (o:) of all objects.
positive example,
Regarding the reconstruction loss, normalize the input f. and reconstructed features f of the autoencoder before we calculate MSE between them, since retrieval happening with cosine similarity, where the magnitude doesn't 't matter. Note that if there is object without child or parent, its embedding is solely supervised by L,, The training duration is approximately. 40 minutes for one scene on the specified hardware.
23

---

D.3 NeRF model
We implement OpenHype Nerfstudio and build upon the Nerfacto model. For our OpenHype vision- Laooonla field, we follow OpenNerf 7 and use an MLP with one hidden layer and a feature dimension of 256. We follow LERF [5] for the hashgrid representing language features. For optimizers and learning rate schedulers of proposal networks and fields of the underlying Nerfacto model, we use the same settings as LERF 15 and OpenNerf 77.
We train all models for 30000 steps
(approx. 60 min. per scene).
D.4 Language-aligned feature extraction
We apply Semantic SAM 24 using all granularity levels (the default setting) to extract segmentation masks of objects from a given image. We use tight bounding box around the segmentation mask to create crops and set pixels outside of the segmentation mask to zero. Those crops are processed with CLIP [37]. more specifically, we use the OpenClip [ [3 ViT-B-16 model trained on the LAION-2B dataset (laion2b_ss seioo22bs334b____ _34b_b88k) to extract features. In order to provide a balance between object and context, we extract second crop for each object by extending the bounding box by a factor of 0.1 and process it with the vision-language model without zeroing out the pixels outside of the mask. The final feature fi for each is then the average of the two crop features.
D.5 Querying OpenHype
As described in Sec. 4.3, querying OpenHype can be divided into 4 steps: 1) interpolation of rendered features in hyperbolic space, 2) decoding interpolated features to CLIP space, 3) computing relevancy scores for each decoded feature (representing different levels of granularity), and 4) aggregating the relevancy scores to one single score per pixel.
For interpolation on the Hyperboloid, [qqaation A.4 is used to sample equidistant features along the geodesic from the rendered feature to the origin O.
2) The sampled features are mapped to the tangent space and processed with the decoder of the auto-encoder, resulting in a set of language- aligned features {fi,, == N..ps) per pixel with Nsscps being the number of interpolation steps.
3) The relevancy score Si, for each of these features fú, is computed using the formula introduced explff, fyvempt)
in [5]: minj exp(S, Peey promppp -x,x "prompe)' where fneg_prompt are the dision-languee em- beddings of the negative prompts (also called canonical prompts) "object", "things", "stuff", and "texture".
4) The softmax- mean aggregation of the relevancy scores 8i, per pixel is computed using Rep Bi. 8i.. with 8. exp(ss,). 3/E /Ener exp(s;,). Here, 8, can also be interpreted as attention weights computed the softmax over the ss, values. Our ablation results Table 2d show that plain wean-aggregatton yields similar results, especially for part queries. These lindings are consistent with the intuition that for querying parts, the relevancy scores are high at part-level and object-level, distinguishing pixels that are included in the part segmentation mask from pixels included solely in the object segmentation mask. -aggreataggrggttioon,, on the other hand, yields higher scores for objects as objects are easier to detect, and using the maximum mitigates noise on the geodesic path, which is included when aggregating using mean or softmax-weighter mean aggregation. The adapted LERF [15 dataset of LangSplat (35) has only a few prompts per scene, consisting mostly of objects. Since SpngSplat and LERF use the maximum for scale selection, we follow their practice on this dataset for fair comparison. Moreover, use the evaluation code of the public repository of LangSplat for comparability in all experiments.
Σk=1
D.6 Dataset
To adapt the ScanNet++ subset of the Search3D dataset 42 to the tasks of open-vocabulary segmen- tation and localization from radiance fields, we project the 3D annotations to the test frames of the novel view synthesis split of ScanNet++. We work with undistorted images and sample a maximum of 250 train frames per scene evenly across all images. Note that while train and test frames are
24

---

strictly separated for training and evaluating our models, ground truth poses are used for camera parameters fairly compare against baselines.
E
Additional ablations and results
In this section, provide further ablations and results that complement the findings in the main paper.
Additional baselines based on Gaussian splats
Since we focus on radiance fields, our main baselines are based radiance fields as well. Yet, we include highest performing approach based on Gaussian splats, LangSplat [35]. main results
Table 1b and Table Ta We give additional Gaussian splatting baselines in [Table E. OpenHype outperforms the additional baselines on out of 4 scenes and on average all other methods by margin of 3.2 mloU points.
Table E.1: Additional comparison baselines based Gaussian splatting the LERF 15] dataset. Results show that our hierarchical approach outperforms all baselines on average and in almost all single scene results.
Method
fig.
Tea.
kit.
Avg.
LEGau. 40
13.8
27.6
45.2
23.7
27.6
GOI
36
35.1
36.9
66.6
45.2
45.9
OpenGau.
44
21.1
69.7
63.4
34,9
47.3
Splat
35
51.2
44.7
65.1
44.5
51.4
Ours
43.9
59.8
71.2
51.7
54.6
E,2 Variation between runs
As stated in Appendix B OpenHype is dependent on the quality of the latent space embeddings, where we experience some noise. Our results reported are the average over 5 runs. In Table E.2 we report the standard deviation over those 5 runs. We can see that the standard deviation remains low leaving significant gap between OpenHype and other methods on ScanNet++ dataset.
Table E.2: Standard deviation of the results presented in the main paper in Table lb and Table la
Dataset
Exp.
loU
Acc
Avg.
+1.0
#4.2
ScanNet++
Obj.
112
+0.0
Part
=1.
#8.0
LERF
Avg.
#0.6
E.3
Oracle ablation
We noticed that, in some cases, LangSplat has, qualitatively, clean-looking maps but selects the "wrong" level. To evaluate if OpenHype's superiority is mainly attributed to aggregation, we conduct an oracle experiment, where pick the level that gives the highest loU score for each prompt. The experiment conducted on the same scenes also used for ablations in Table Results in Ta ble E.3 show that the standard OpenHype experiment outperforms the oracle version of Lang showing the effectiveness of the continuous traversal of geodesics in our hyperbolic hierarchical embeddings. The oracle version of OpenHype depicts significant improvements suggesting that exploring alternative aggregation methods are an interesting direction for future work. FFiurre E. presents the relevancy maps for different prompts of objects and object parts the 3 different levels of SpngSplat and at sampled levels the continuous OpenHype hierarchy path. We can see that while
25

---

LangSplat produces qualitatively good relevancy maps even for parts, results in Table E.3 that the capacity of hard-coded 3 level hierarchy is limited. Further, the working of OpenHype's hierarchy. For the prompt "laptop keyboard" on fine granularity levels (further away form the origin O) the standard keyboard as well as the laptop keyboard highlighted; when moving up along the hierarchy the relevancy map starts focusing on the laptop keyboard only. For the prompt "printer" on fine gramularity levels highlights only lightly parts, which can be clearly associated with "printer" such as the scanner-top. When moving closer to the origin O the relevancy map gets stronger as the prompt corresponds more with object-levels than part-levels.
suggest E. gives insights in
Table E.3: Results for the oracle experiment using 5 scenes of the ScanNet++ subset of Search3D 42) The level that gives the highest IoU score chosen for evaluation per prompt.
Method
Object
Part
Avg.
LangSplat
35
38.3
5.2
20.1
LangSplat
35
Oracle
41.3
10.4
24.3
Ours
50.4
14.9
30.9
Ours Oracle
68.7
23.7
44.0
laptop
laptop keybound
OpenHyype
Level
Lenel
Lavel2
LangSplat
printer
printer control pamel
OpeaHypo
Level
Load
Lod
Level2
LangSplat
Figure E.1: Qualitative results on Search3D dataset of relevancy maps for different levels of the hierarchy of LangSplat 351 and OpenHype.
E.4 Different foundation models
OpenHype depends pretrained 2D vision- laniuuge model and pretrained image segmentation model. the following, we investigate the impact of the model choice on OpenHype's performance using the 5 scenes of the ScanNet++ subset of Search3D 42 used in all ablation experiments.
Vision-language model. We use a relatively small CLIP 13 5 model (ViT-B/16) for efficiency, and our approach still outperforms others that use the same SpangSplat, LERF) larger base models (OpenNerf). To test the impact of more powerful lision-l model, we conducted an ablation study using SigLIP 43), more specifically, ViT-S0400M- -100000------ppp- 14- The results shown in Table E.4a demonstrate that our method benefits from newer models, further widening its performance advantage state-of-the-art approaches.
At first glance, the question of why not use pre-trained hyperbolic CL.IP model might arise. pre- traierd hyperbolic CLIP encoder, such MERU [6), is trained to encode text and images
26

---

following a visual-semantic hierarchy, where more general concepts are closer to the origin. For example, for the semantic hierarchy "Taj Mahal" + "monument" + "architecture". "architecture" is the closest to the origin. In contrast to that, we use the AE to train latent embedding that encodes the scene-specific hierarchy. This means that we can leverage the geodesic path from a rendered pixel feature to the origin in hyperbolic space, which equals traversing vision-language feature hierarchy that encodes parts of objects or objects in the scene that the pixel is part of. The semantic hierarchies of MERU do not necessarily encode object-part relationships. Nevertheless, we have conducted an ablation using the MERU model, instead of our AE, to see if it additionally stores some object part semantic relationships that could be helpful for hierarchical scene understanding, However, as shown in Table E.4a we observed poor performance, confirming that encoding our custom hyperbolic embedding space is key.
Segmentation model. In our experiments, we use the default contiguration of Semantic SAM 24 which provides six granularity prompt levels and is specifically designed for multi-level mask generation. Table E.4b
comparison with the native SAM [77.. where we built hierarchies by combining masks from all three available levels. As shown, Semantic SAM enables the construction of more meaningful hierarchies. When comparing our results based on native SAM to LangSplat- - which also leverages multiple mask levels from the native SAM it becomes evident that, while OpenHype benefits from the richer multi-granularity masks of Semantic SAM, consistently outperforms existing baselines across the evaluated metrics. This demonstrates that our approach is not only robust to the choice of segmentation model but also able to fully exploit finer mask hierarchies to produce superior scene representations.
Table E.4: Ablation study of different foundation models used for OpenHype on the task of segmen- tation on five representative scenes from the Search3D dataset.
Object
Part
Avg.
Object
Part
Avg.
MERU 6
26.0
6.7
15.4
Spaat 35
38.3
5.2
20.1
CLIP ViT-B/16 37
50.4
14.9
30.9
native SAM 17
48.0
10.8
27.6
SigLIP VIT SO400M- 14 48
52.7
21.6
35.6
Semantic SAM
24
50.4
14.9
30.9
(a) Vision- Masoun-annguage model.
(b) Segmentation model.
E.5 Comparison with RelationField
Recently, RelationField [20] has been introduced for modeling semantic relationships. employs a querying strategy that separately processes nouns and relationships, subsequently combining activations. This method assumes that a prompt consists of explicitly stated relationships, such as "on top of," which makes it not compatible with the Search3D benchmark.
Yet, we provide a qualitative comparison with the "is part of" relationship querying of RelationField, which comes closest to hierarchical scene understanding. The white dot in Figuure F.2 indicates the pixel of interest, whose corresponding object and object parts aim identify. The results of RelationField show the similarity map of its feature field with the query "is part of" with respect to this pixel of interest.
For our method, we examine the lowest-level feature of the pixel (located near a boundary) and traverse up its hierarchy by following the geodesic path to the origin of the hyperbolic space. The highlighted areas in the image correspond to pixels whose features lie within or close to the entailment cone, as introduced in (6) of each parent feature of our pixel of interest when moving up the hierarchy. The magnitude of the heatmap is given by the negative entailment loss 6 which equals zero when the embedding vector lies entirely within the entailment cone.
Figure E.2 that while RelationField highlights only part of the fridge and handles only single- level hierarchies, OpenHype naturally supports hierarchical grouping by leveraging the structure of our hyperbolic embeddings, without requiring explicit queries.
27

---

Ours
RelationFicld
Interpelatoo
Figure E.2: Comparison of OpenHype and RelationField [20] for the pixel of interest (white dot). RelationField highlights only part-level relations via the "is part of" query, whereas OpenHype uncovers hierarchical groupings by traversing the geodesic path in hyperbolic space, guided by entailment cones 6
E.6 2D 3D
To demonstrate that lifting to 3D is beneficial, we conducted an ablation that uses only Semantic SAM with the extracted hierarchy and mask features directly the evaluation frames. We compare this to the full OpenHype pipeline. Although the 2D-only variant can still segment object parts thanks to the mask hierarchy, incorporating multi-view supervision and lifting to 3D yields clear and substantial performance improvement as shown in Table E.5
Table E.5: Comparison between applying the Semantic SAM hierarchy and its mask features directly to the evaluation frames, versus lifting them to 3D and decoding the rendered features with the hyperbolic autoencoder.
Object
Part
Avg.
Semantic SAM hicrarchy 24 + CLIP 37 Ours
39.2
9.3
22.7
50.4
14.9
30.9
E.7 Robustness study of noisy hierarchies
Since the feature field learns from multiple views (~ 250 per scene), it is able present robust features even if some views contain noisy hierarchies. In order to evaluate it, we provide a robustness study using 5 representative scenes used in all ablations. Segmentation noise essentially results in noisy features used to supervise our model. To analyze the robustness of our model to the number of noisy views, added Gaussian noise the features of a number of training frames sampled randomly for training our hyperbolic auto-encoder. We use 0.2 and 0.4 standard deviation as noise level, which would result in cosine similarity of 0.22 and 0.13, significant drop when measuring the similarity of normalized vector and the same vector but noised.
Results show that model is quite robust noisy supervision. As seen in Table E.6a the Gaussian noise with a = 0.2 even works as data augmentation, given slightly improved results up to 80 corrupted images. When exceeding than half of the images to be comrupted, performance starts to decrease, yet not catastrophically.
E.8 Results on outdoor seene
Our main results reported indoor scene datasets. To enable comparison with baselines in different setting, we also annotated the Truck scene from the Tanks and Temples dataset [18]. This scene presents several notable challenges, including reflections in the vehicle windows and large depth range with multiple elements in view. We annotated "truck", "truck door", "wheel", "mudffap", "front bumper", "windshield" object queries and "window of truck door". "logo on door". "wooden rails of truck bed", "wheel rim", "truck door handle", "truck bed", "truck cabin" as part queries.
28

---

Table E.6: Robustness study results for different amount of corrupted images using Gaussian noise with different values as standard deviation.
# corr. img.
Object
Part
Avg.
corr. img.
Object
Part
Avg.
50.4
14.9
30.9
50.4
14.9
30.9
10
49.5
15.0
30.5
10
49.1
14.4
30.0
40
52.8
15.5
32.3
40
48.6
14,4
29.8
80
51.3
14.9
31.3
80
46.9
14.6
29.2
160
42.6
14.6
27.2
47.1
(a)
0,
0.2.
(b) μa 0, a 0.4.
Table E.7 shows that OpenHype also outperforms our baselines for part as well as object annotations on this outdoor scene. Notably, the biggest difference of more than 8 IoU points can be seen for the part-level annotations.
Table E.7: Results on the Truck scene of the Tanks and Temples dataset on the segmentation task using IoU metric.
Object
Part
Avg.
OpenNerf 7
24.98
18.74
21.7
LpngSplat 35
35.82
27.34
31.36
Ours
38.41
35.76
37.01
29

---

List of assets
We use publicly available datasets and repositories and credit the authors properly in the paper. A full list of assets used in this work, including the license, is given below.
Neristudio and its
Nerfacto model v1.1.5
(https://ittub.coom com/
Apache License 2.0
OpenClip [S] 13 v2.29.0 http:://ggttuu..ooooommoomoomoop MIT
License
ScanNet++ subset of Search3D
42
46
released under the original data of use
Semantic SAM 24]
SAM [17
Apache
License, Version 2.0, Caccoooeeeaooooo/////onnoonnnnnn
2004
MERU
6
Attribution-
NonCommercial 4.0 4 ln
OpenNerf http:////atttthoooo////e///////r MIT License
LERF
15
| License
LangSplat 35
custom
research-only created by Inria and the Max Planck Planit for Informatik (MPII)
OpenSeg
Apache License 2.0
CO-SNE [10 https://gtthbb..cmmmmmuiiiig
30


Reviews:
[{"review_id": "zibnGcyl6G", "reviewer": "Reviewer_f2nb", "summary": "This paper presents OpenHype, a method for open-vocabulary segmentation in Neural Radiance Fields (NeRF) that encodes 3D scene hierarchies in a continuous hyperbolic space. Unlike prior approaches that rely on fixed, discrete hierarchies or require multiple rendering passes, OpenHype leverages geodesic traversal in hyperbolic latent space to efficiently model and query semantic structures at varying levels of abstraction. The method combines a hyperbolic auto-encoder for learning language-aligned hierarchical embeddings and a NeRF model supervised directly in this space, enabling accurate segmentation for compositional queries. Extensive experiments on the Search3D and LERF benchmarks demonstrate significant improvements, particularly in fine-grained part-level segmentation, over state-of-the-art baselines.", "questions": "1. Could the authors clarify why the proposed method also improves performance on the \"object\" queries in Table 1(a)? It is intuitive that modeling hierarchy in hyperbolic space would help with fine-grained queries like object parts, but it's less clear why it would also lead to significant gains on higher-level object queries, which existing methods already handle reasonably well. \n\n2. This paper [2] discusses another side of the \"bag of words\": CLIP cannot tell the spatial relationships between objects. I am curious, could the proposed method solve this problem?\n\n3. Not a weakness, not an issue. There are many new papers (including [2]) discussing open-vocabulary grounding. It might be good to cite as well.\n\n[2] From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs, ICML 2025\n\n[3] LargeSpatialModel: End-to-end Unposed Images to Semantic 3D, Neurips 2024.\n\n[4] Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields, CVPR 2024\n\n[5] Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting\n\n[6] SAB3R: Semantic-Augmented Backbone in 3D Reconstruction,\n...", "limitations": "I don't see limitation discussed in the main paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. I appreciate the core idea of the paper: language queries in 3D inherently involve different levels of granularity, and instead of relying on multi-scale CLIP features—which can be computationally expensive—a natural solution is to construct the feature field in hyperbolic space. This allows for efficient geodesic traversal to obtain multi-scale features at low cost. The central insight is both intuitive and compelling.\n\n2. The performance improvements are clear and significant Table 1,2, showing that the proposed method work well.\n\n## Weakness\n\n1. The paper would benefit significantly from improved writing. While I understand that hyperbolic representations are not commonly discussed and can be challenging to describe, the current presentation lacks clarity and important implementation details. In particular, the description of the CLIP-based encoder-decoder is vague. It is unclear whether this module is trained jointly across the entire dataset (which I assume is the case) or separately for each scene. If it is trained across scenes, the authors should specify which dataset is used for training and provide relevant training details such as supervision strategy, epochs, and optimization settings.\n\n2. I remain unconvinced by the current approach for constructing hierarchical data and training the CLIP-based VAE in hyperbolic space. The method relies on Semantic SAM to extract object masks and builds hierarchies by detecting smaller masks (level-2) contained within larger ones (level-1). However, this strategy results in very shallow hierarchies—typically only two levels—which limits the expressiveness and potential of the hyperbolic framework. There are several alternatives that could yield richer hierarchical structures. For example: (1) SAM natively provides masks at multiple scales (large, medium, small), which could naturally form a multi-level hierarchy; (2) one could iteratively crop within large masks to discover finer structures, enabling deeper semantic levels. I am not sure if the proposed way is a good enough way to construct the data.\n\n3. Related to the previous issue, I’m also curious why the authors chose to train a hyperbolic CLIP encoder-decoder from scratch rather than leveraging existing pretrained hyperbolic CLIP models (e.g., from prior work like Hyperbolic CLIP [1] or similar). While I can imagine potential reasons, it would be helpful to clarify this design choice and, ideally, include a comparison showing whether using a pretrained hyperbolic model yields better, worse, or comparable performance. This would help isolate whether the gains come from the hyperbolic space itself or from task-specific training.\n\n4. The \"Extrapolated features.\" look strange to me. Wouldn't it destroy many information since all pixels inside one single big masks share the same features in this way?\n\n\n[1] Hyperbolic Image-Text Representations", "was_revised": true, "final_justification": "The rebuttal addresses most of my questions, so I increased the score.", "rebuttal": "We thank the reviewer for highlighting our hyperbolic embeddings for hierarchical structuring in radiance fields as _\"intuitive and compelling\"_, and for recognizing our _\"extensive experiments\"_. We also greatly appreciate the reviewer’s valuable feedback, which, in our opinion, greatly improves the clarity and insights into our architectural choices.\n\n\n### *W1: Description of clip-based encoder-decoder is vague*\nA detailed description of the AE setup is given in the supplementary material, including all training details and hyperbolic supervision losses. \nAs suggested, we will move the relevant details to the main paper. The AE is trained per scene (on the NeRF training frames) to capture scene-specific hierarchies. Besides a cleaner representation of the scene hierarchies, a per-scene training enables a fair comparison with LangSplat, which also trains an AE per scene. However, in contrast to our novel hyperbolic formulation, LangSplat facilitates a conventional Euclidean embedding space and further requires training 3 models for each scene. Please find below a comparison of scene-specific AEs with a global AE below. The results indicate that a scene-specific AE is better suited to encode object hierarchies in hyperbolic latent space than a global AE, since the scene-specific AE can use the full hyperbolic space to encode a single scene hierarchy while the global AE needs to embed all scene hierarchies simultaneously.\n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| scene-specific | 14.9 | 50.4 | 30.9 |\n| global | 14.1 | 39.9 | 25.7 | \n\n### *W2: Concerns about shallow hierarchies*\nWe use the default setting of Semantic-SAM, which has 6 granularity prompt levels, not 2, and was specifically designed for multi-granularity level masks. We additionally provide a comparison with the native SAM below, whereby we construct hierarchies using masks of all 3 levels combined. As can be seen, Semantic-SAM encodes more meaningful hierarchies. We will include this experiment in our final version.\n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| native SAM | 10.8 | 48.0 | 27.6 | \n| Semantic-SAM | 14.9 | 50.4 | 30.9 |\n\n### *W3: Why not a pretrained hyperbolic CLIP encoder*\nWe use the AE to train a latent embedding that encodes the scene-specific hierarchy. This means that we can leverage the geodesic path from a rendered pixel feature to the origin in hyperbolic space, which equals traversing a vision-language feature hierarchy that encodes parts of objects or objects in the scene that the pixel is part of. A pre-trained hyperbolic CLIP encoder, such as MERU [1], is trained to encode text and images following a visual-semantic hierarchy, where more general concepts are closer to the origin. For example, for the semantic hierarchy \"Taj Mahal\" -> \"monument\" -> \"architecture\", \"architecture\" is the closest to the origin. However, those semantic hierarchies do not necessarily encode object-part relationships. We have also conducted an ablation using the MERU model, instead of our AE, to see if it additionally stores some object part semantic relationships that could be helpful for hierarchical scene understanding. However, we observed poor performance, confirming that encoding our custom hyperbolic embedding space is key.\n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| MERU | 6.7 | 26.0 | 15.4 |\n| OpenHype | 14.9 | 50.4 | 30.9 |\n\n[1] Desai, Karan, et al. \"Hyperbolic image-text representations.\" ICML, 2023\n\n### *W4: Clarification on extrapolated features*\nThe VL-features of objects (parts) can be at different levels in the hierarchy depending on the view, since the mask hierarchy is created for each train view, independent of others.\nBy extrapolation, we set each leaf mask used for supervision of the NeRF model to the lowest level in the global hierarchy, meaning we obtain the features with higher granularity. Note that the interpolation path from the extrapolated feature to the origin leads through the original feature; hence, extrapolating does not destroy any information.\n\n### *Q1: Could the authors clarify why the proposed method also improves performance on the \"object\" queries in Table 1(a)? It is intuitive that modeling hierarchy in hyperbolic space would help with fine-grained queries like object parts, but it's less clear why it would also lead to significant gains on higher-level object queries, which existing methods already handle reasonably well.*\nOur intuition is that our representations for entire objects can be more informed by explicitly modeling the underlying part-whole hierarchy within the hyperbolic space. The optimization process in hyperbolic space encourages the object's representation to be positioned relative to its learned parts in a geometrically meaningful way, enriching the final feature field with more information about the object. \nFurther, our perception is that traditional Euclidean representations are effective for general object recognition, but tend to represent entities in a relatively flat space. This can lead to a mixing of representations where distinct hierarchical levels (e.g., parts and the entire object) are not optimally disentangled or organized. In contrast, the hyperbolic space has an inherent ability to efficiently embed hierarchical structures due to its exponential growth, which naturally encourages a more organized and disentangled feature representation. \n\n### *Q2: This paper [2] discusses another side of the \"bag of words\": CLIP cannot tell the spatial relationships between objects. I am curious, could the proposed method solve this problem?*\nWe focus on compositional relationships or, more generally, on tree-structured data. Spatial relationships do, in general, not obey hierarchies, and have therefore to be modelled by a general graph, which is currently not supported by OpenHype. Nevertheless, we find the suggestion by the reviewer very interesting and might investigate in future work how hyperbolic feature representations can be used to model object relationships.\n\n### *Q3: Not a weakness, not an issue. There are many new papers (including [2]) discussing open-vocabulary grounding. It might be good to cite as well.*\nAs suggested, we will cite them in the final version.\n\n\n#### We hope that our responses have addressed the concerns raised, and we would be grateful if the revisions are reflected in the score."}, {"review_id": "hCWGhF1Dri", "reviewer": "Reviewer_1m1q", "summary": "This paper proposes using hyperbolic space for hierarchical embedding in NeRF, which naturally captures multi-scale relationships and allows for smooth traversal of the hierarchy via geodesic paths. Experiments on open-vocabulary segmentation from radiance fields demonstrate the effectiveness of this continuous hierarchical structure.", "questions": "1. Could the authors elaborate on the computational cost or time efficiency of sampling along the geodesic?\n\n2. Are there additional downstream tasks or applications that could benefit from this type of embedding?\n\n3. Could the authors clarify why the neural network first learns features in the tangent space and then projects them to the hyperbolic space, rather than learning the embeddings directly in hyperbolic space?\n\n4. Please refer to the weaknesses section for further comments.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work in the supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is well-written and effectively communicates its ideas, even to readers unfamiliar with the domain, aided by clear and illustrative figures.\n\n2. To the best of my knowledge (though I am not an expert in this area), this is the first work to propose a continuous hierarchy for semantic embeddings.\n\n3. The experimental results are impressive, and the paper includes a thorough and extensive ablation study.\n\nWeakness: \n1. I suggest the authors give more discussion on choosing the Lorentz model but not other models, even for example, why not hemisphere, if the authors could give reasonable ideas or give ablations on that, it could complete the whole idea of the paper. \n\n2. As the paper stated in the limitations, this work depends on the quality of the language-vision model, the previous segmentation model, and the assumption that the embedding is smoothly distributed. I'm wondering whether the authors could conduct an ablation to show the robustness of the model to different vision-language models.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and appreciate their comments about our _\"impressive experimental results\"_ as well as _\"well-written\"_ manuscript and our _\"illustrative figures\"_. Below, we provide more discussions requested by the reviewer; we hope this makes our design choices clearer.\n\n### *W1: Choice of hyperbolic model*\nSeveral isomorphic models of hyperbolic space exist (e.g., Poincaré ball, Lorentz, Hemisphere) [1], so in theory, any of these models could be used to embed hierarchical data. Our primary reasons for selecting the Lorentz model are its favorable numerical stability and computational efficiency during optimization [2,3], as well as a numerically more stable distance function [1,3], which we use in our hierarchical loss. The mentioned hemisphere model is almost entirely absent from practical DL literature, and hence, less support exists; it is mostly used for visualization [1].\n\n[1] Peng, Wei, et al. \"Hyperbolic deep neural networks: A survey.\" IEEE TPAMI, 2021\n\n[2] Mishne, Gal, et al. \"The numerical stability of hyperbolic representation learning.\" ICML, 2023\n\n[3] Nickel, Maximillian, and Douwe Kiela. \"Learning continuous hierarchies in the lorentz model of hyperbolic geometry.\" ICML, 2018.\n\n### *W2: Different vision-language model*\nWe use a relatively small CLIP model (ViT-B/16) for efficiency, and while performance improves with more advanced architectures, our approach still outperforms others that use same (LangSplat, LERF) or larger base models (OpenNerf).\nIn response to the reviewer’s suggestion, we conducted an experiment using a more powerful vision-language model, ViT-SO400M-14-SigLIP. The results demonstrate that our method benefits from newer models while remaining robust across different VL models. We will include experiments with additional models in the camera-ready version. \n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| SigLIP ViT-SO400M-14 (ours) | 21.6 | 52.7 | 35.6 |\n| CLIP ViT-B/16 (ours, reported in the paper) | 14.9 | 50.4 | 30.9 |\n\n### *Q1: Could the authors elaborate on the computational cost or time efficiency of sampling along the geodesic?*\nAs suggested, we have measured the average running time on a NVIDIA RTX A5000 of the main components of our method for an image of size 543x979 with 20 interpolation steps. From the numbers below, we can observe that the most computationally expensive task is rendering, which is inherited from OpenNerf. Interpolation only requires 473 milliseconds for 20 steps along the geodesic path, plus 622 milliseconds to transform hyperbolic embeddings into CLIP embeddings with our decoder.\nA naive discrete approach would instead require rendering of each level of the hierarchy, multiplying the rendering time by the number of levels in the hierarchy, i.e., 20 in this case.\nAdditionally, our approach only requires a single training process, unlike methods that train multiple models.\n\nRegarding memory usage, our method does not introduce additional overhead by sampling one level in the hierarchy.\nHowever, memory consumption increases with the number of interpolation steps, generating n_steps feature fields, particularly when decoding features in CLIP space.\nThat said, decoding can be performed in batches to manage memory constraints effectively. Overall, our method remains substantially more efficient than training multiple models, even in memory-limited scenarios.\n\n| Task | Time - ms | \n|----------|----------|\n| rendering | 1992.214 | \n| interpolating | 473.471 |\n| decoding | 622.364 |\n\n### *Q2: Are there additional downstream tasks or applications that could benefit from this type of embedding?*\nA concrete application scenario could be affordance segmentation of small parts like light switches, handles, and buttons in living spaces. This is particularly useful when an agent has to fetch something from inside a cabinet. Here, the agent must first recognize the cabinet itself, and then identify the cabinet handle to open and close it.\nEstablished methods like OpenNerf or LangSplat struggle with small parts (as shown in the experiments in the main paper) and do not represent the connection between the complete object and its parts. In contrast, OpenHype could perform well in this scenario due to its strong performance on part segmentation and hierarchical structuring.\n\n### *Q3: Could the authors clarify why the neural network first learns features in the tangent space and then projects them to the hyperbolic space, rather than learning the embeddings directly in hyperbolic space?*\n\nOur approach learns features in the tangent space (locally Euclidean) primarily due to the computational tractability and well-established optimization properties of Euclidean geometry. Gradient-based optimization and standard neural network operations are highly optimized for Euclidean spaces. \nAlthough there are fully hyperbolic networks, those are computationally demanding and tend to be unstable [1]. Additionally, deep learning methods in non-Euclidean settings remain relatively underdeveloped, making Euclidean architectures more practical for training and experimentation.\nFollowing previous work [2], we thus learn the embeddings in the Euclidean space, where there is rich support from DL frameworks, and then project them into hyperbolic space to compute the loss. \n\n[1] Peng, Wei, et al. \"Hyperbolic deep neural networks: A survey.\" IEEE TPAMI, 2021\n\n[2] Desai, Karan, et al. \"Hyperbolic image-text representations.\" ICML, 2023"}, {"review_id": "pgMpw7FNxj", "reviewer": "Reviewer_pVMv", "summary": "This paper aims to improve 3d object and scene modeling through using the properties of hyperbolic geometry to encode hierarchical structure in latent space in a single pass. The authors compare their method with LERF, OpenNeRF, and LangSplat, finding that their method is the best when evaluated with the Search3D and LERF datasets.", "questions": "- Is inference more expensive than the baselines due to hyperbolic embeddings? If so, how would results differ if you improved the baselines with more compute?\n- Do the baselines and your results use the same base models?\n- Could you include additional 2D baselines, include those based on a powerful large model like SAM?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Writing is clear and structured nicely\n- Qualitative and quantitative results are better than baselines\n- Ablation studies help improve understanding\n\nWeaknesses\n- Limited evaluation in terms of breadth of datasets / tasks\n- Lack of detail regarding baselines", "was_revised": true, "final_justification": "I maintain my score for now because my main concern (weakness 1) has not been adequately addressed.", "rebuttal": "We thank the reviewer for their review, highlighting our improved performance over recent baselines. In the following, we highlight the strengths of our method further by providing additional experiments and discussions requested by the reviewer.\n\n### *W1: Breadth of datasets/tasks*\nWe want to highlight that we provide results on 2 different datasets, and compare against a total of 7 recent works (LERF, OpenNeRF, LangSplat, N2F2, LEGaussian, GOI, OpenGaussian) as well as 2D approaches for open-vocabulary segmentation (OpenSeg, LSeg). Additionally, we provide 6 main ablations in the paper.\n\nMoreover, following the suggestion of reviewer LDqQ, during the limited time of the rebuttal period, we included an additional data sample by annotating and using the Truck scene of the Tanks and Temples dataset. We annotated the following components in the scene:\n\n- truck door\n- truck\n- truck door window\n- truck door logo\n- wheel\n- wheel rim\n- mudflap\n- truck cabin\n- truck bed\n- wooden rails of truck bed\n- truck door handle\n- front bumper\n- windshield\n\nWe evaluate our method and OpenNerf in this scene, whereby OpenHype outperformed OpenNerf for this challenging outdoor scene. In the camera-ready, we will provide more comparisons to other baselines.\n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| OpenNerf | 18.74 | 24.98 | 21.7 |\n| OpenHype | 35.76 | 38.41 | 37.01 |\n\n\n### *W2: Details regarding baselines*\nThank you for the suggestion. We compare against multiple methods that compute semantic feature fields. While some of them, such as LERF and Langsplat, provide a discrete set of hierarchical levels, most of them rely on only a single feature. Relying on only one feature level means that the results are bound to detecting objects of one fixed granularity; further, the model is not able to detect compositional prompts such as \"table leg\"; due to the \"bag of word\" issue it would highlight all tables and all legs in the scene. For methods using discrete set of granularity levels, the model is bound to those specific granularities, whereas with our approach we can encode a continuous hierarchy without predefined granularity levels. We will include additional details of all the baselines in the supplementary material of the final version of the paper.\n\n### *Q1: Is inference more expensive than the baselines due to hyperbolic embeddings? If so, how would results differ if you improved the baselines with more compute?*\nInference is, in general, not more expensive due to hyperbolic embeddings. Rendering is the same as for OpenNerf, the additional compute comes from interpolation and decoding into CLIP space from our latent embeddings. We provide average running time on an NVIDIA RTX A5000 for an image of size 543x979 with 20 interpolation steps. From the numbers below, we can observe that the most computationally expensive task is rendering, which is inherited from OpenNerf. Interpolation only requires 473 milliseconds for 20 steps along the geodesic path, plus 622 milliseconds to transform hyperbolic embeddings into CLIP embeddings with our decoder.\nA naive discrete approach would instead require rendering of each level of the hierarchy, multiplying the rendering time by the number of levels in the hierarchy, i.e., 20 in this case.\nAdditionally, our approach only requires a single training process, unlike methods that train multiple models.\n\n| Task | Time - ms | \n|----------|----------|\n| rendering | 1992.214 | \n| interpolating | 473.471 |\n| decoding | 622.364 |\n\nOur approach used the same base vision-language model (LangSplat, LERF) or a smaller one (OpenNerf) than the baselines, yet still outperforms them. Using a more powerful VL-model would most likely increase the results of baselines, but so does our method, as can be seen in the ablation with a bigger VL-model SigLIP ViT-SO400M-14.\n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| SigLIP ViT-SO400M-14 (ours) | 21.6 | 52.7 | 35.6 |\n| CLIP ViT-B/16 (ours, reported in the paper) | 14.9 | 50.4 | 30.9 |\n\n\n### *Q2: Do the baselines and your results use the same base models?*\nWe use the same vision language model as LangSplat and LERF (CLIP ViT B/16) and a smaller model than OpenNerf (CLIP ViT L/14@336). \n\n### *Q3: Could you include additional 2D baselines, include those based on a powerful large model like SAM?*\n\n Following your suggestion, we compare a 2D baseline based on Semantic-SAM and CLIP directly applied to the test images. We use the 2D mask hierarchy that can be constructed from the extracted Semantic-SAM masks of the test images. We can see that although also being able to segment object parts due to the mask hierarchy, using multi-view supervision and lifting to 3D yields a great performance improvement. \n We will add this ablations to the camera ready version.\n \n\n| Method | IoU - part | IoU - obj. | IoU - overall |\n|----------|----------|----------|----------|\n| Semantic-SAM hierarchy + CLIP | 9.27 | 39.22 | 22.74 |\n| OpenHype | 14.9 | 50.4 | 30.9 |"}, {"review_id": "BCCStJebFT", "reviewer": "Reviewer_LDqQ", "summary": "The submission introduces OpenHype, a Neural Radiance Field that embeds an open-vocabulary semantic hierarchy in hyperbolic (Lorentz) space, enabling continuous part → object → region queries with a single render pass. Experiments on Search3D and LERF scenes report sizable mIoU gains over LangSplat and OV-NeRF, and ablations explore curvature, dimension, and step length. While the hyperbolic formulation is conceptually novel relative to Euclidean/tree approaches such as Hi-LSplat and ReasonGrounder, the empirical study omits several state-of-the-art baselines, lacks real-time metrics, and provides little analysis of hyperbolic optimisation stability.", "questions": "See weakness.", "limitations": "Although single-pass rendering should be faster, the paper omits FPS and memory numbers. GARField—which also addresses hierarchical grouping—reports detailed runtime statistics; a similar disclosure would let readers weigh the practical trade-offs", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The authors abandon the usual Euclidean or spherical feature spaces and embed the whole open-vocabulary tree in Lorentz hyperbolic space, whose negative curvature naturally accommodates exponentially expanding trees. This idea has been explored in 2-D and vision-language models for representing hierarchies, but—so far as the literature shows—has not been applied to NeRF or 3-D Gaussian Splatting before.\n2. By distilling CLIP features into the hyperbolic field, the method inherits the free-form text querying of OV-NeRF yet pushes it further down to “wheel”, “doorknob” or even finer part semantics—something earlier systems struggled to do consistently.\n3. Training hyper-parameters, compute budgets and a link to code are given; releasing a complex NeRF variant with reproducible instructions is still the exception rather than the rule in this young sub-field, so this is a practical virtue of the paper.\n\nWeakness:\n1. The paper does not evaluate or, at least discuss, against Hi-LSplat or RelationField all targeting hierarchical or relational semantics in 3-D radiance fields. Without these numbers, the superiority of the proposed method remains unproven.\n2. Experiments are confined to indoor ScanNet-style scenes. Outdoor environments (e.g., Tanks & Temples) or dynamic captures would test whether hyperbolic supervision copes with lighting extremes, motion blur and large depth ranges.\n3. Semantic-SAM is powerful yet still error-prone with occlusions or mirror reflections; because those masks seed the entire hierarchy, segmentation noise could cascade through the hyperbolic field, but the paper offers no robustness study. \n\nIf the authors address most concerns, I will reupdate the score and comment.", "was_revised": true, "final_justification": "The final justification is BA, because the author provides more explanation for some ambiguous points that were unclear in the original version. Even though the author still misses comparisons with some recently published works, these works are considered concurrent ones. Therefore, this will not degrade the novelty of this work.", "rebuttal": "We thank the reviewer for their detailed review and for highlighting our novel approach, _\"which pushes open-vocabulary understanding to finer part semantics\"_. In the following, we provide additional experimental evidence supporting the impact of our approach.\n\n\n### *W1: Comparison with concurrent work*\nWe thank the reviewer for suggesting two relevant works (Hi-LSplat and RelationField). Hi-LSplat appeared on arXiv after the NeurIPS submission, is currently unreviewed, and there is no available code, preventing fair comparisons. \nRelationField (CVPR 2025) does provide code, and we examined its potential comparison with OpenHype. RelationField models semantic relationships but does not capture hierarchical structures. Additionally, RelationField employs a querying strategy that separately processes nouns and relationships, subsequently combining activations. This method assumes that a prompt consists of explicitly stated relationships, such as \"on top of,\" which makes it not compatible with the Search3D benchmark. In our camera-ready version, we will include Hi-LSplat in the related works as concurrent work and will provide a qualitative comparison with RelationField (which we cannot provide in this rebuttal due to the lacking option for media attachments).\nAdditionally, we would like to clarify that our method outperforms all recent baselines present at the time of the submission, such as N2F2, which also targets the same task of hierarchical decomposition.\n\n\n### *W2: Only indoor scenes*\nAs suggested, we carefully analysed the Tanks and Temples dataset and discovered that it does not contain semantic labels. Therefore, given the limited time of the rebuttal period, we annotated the Truck scene as it contains some interesting challenges, for example, there are reflections in vehicle windows, and it presents a large depth range with multiple elements in the scene.\nWe annotated the following components:\n- truck door\n- truck\n- truck door window\n- truck door logo\n- wheel\n- wheel rim\n- mudflap\n- truck cabin\n- truck bed\n- wooden rails of truck bed\n- truck door handle\n- front bumper\n- windshield\n\nWe evaluate our method and OpenNerf on this scene, whereby OpenHype outperforms OpenNerf for this challenging outdoor scene. In the camera-ready, we plan to provide more comparisons to other baselines.\n\n| Method | iou - part | iou - obj. | iou - overall |\n|----------|----------|----------|----------|\n| OpenNerf | 18.74 | 24.98 | 21.7 |\n| OpenHype | 35.76 | 38.41 | 37.01 |\n\n\n### *W3: Robustness analysis of noisy hierarchies*\nSince the feature field learns from multiple views (~250 per scene), it is able to present robust features even if some views contain noisy hierarchies. In order to evaluate it, following the reviewer's suggestion, we provide a robustness study. Segmentation-noise essentially results into noisy features used to supervise our model. To analyse the robustness of our model to the number of noisy views, we added Gaussian noise to the features of a number of training frames sampled equally spaced throughout the scene. \nWe use 0.2 standard deviation as noise level, which would result in a cosine similarity of ~0.2, a significant drop, when measuring the similarity of a vector and a vector-noised.\nResults show that our model is quite robust to noisy supervision, with almost no performance drop with 80 corrupted images. When exceeding more than half of the images to be corrupted, performance starts to decrease, yet not catastrophically. We will include this and additional experiments with different noise levels in the supplementary materials. \n\n| # corrupted img | iou - part | iou - obj. | iou - overall |\n|----------|----------|----------|----------|\n| 0 | 14.8 | 45.1 | 26.93 |\n| 20 | 14.4 | 42.7 | 25.7 |\n| 40 | 14.9 | 41.1 | 25.4 |\n| 80 | 14.7 | 40.7 | 25.1 |\n| 160 | 12.0 | 37.5 | 22.2 |\n\n\n### *W4: FPS and memory numbers*\nThank you for your comment. We agree that runtime statistics are important to disclose; we have measured the average running time of the main components of our method on an NVIDIA RTX A5000 for an image of size 543x979 with 20 interpolation steps, and will add this to the final version. From the numbers below, we can observe that the most computationally expensive task is rendering, which is inherited from OpenNerf. Interpolation only requires 473 milliseconds for 20 steps along the geodesic path, plus 622 milliseconds to transform hyperbolic embeddings into CLIP embeddings with our decoder.\nA naive discrete approach would instead require rendering of each level of the hierarchy, multiplying the rendering time by the number of levels in the hierarchy, i.e., 20 in this case.\nAdditionally, our approach only requires a single training process, unlike methods that train multiple models (e.g. LangSplat).\n\nRegarding memory usage, our method does not introduce additional overhead sampling one level in the hierarchy.\nHowever, memory consumption increases with the number of interpolation steps, generating n_steps feature fields, particularly when decoding features in CLIP space.\nThat said, decoding can be performed in batches to manage memory constraints effectively. Overall, our method remains substantially more efficient than training multiple models like previous state-of-the-art, such as LangSplat.\n\n| Task | Time - ms | \n|----------|----------|\n| rendering | 1992.214 | \n| interpolating | 473.471 |\n| decoding | 622.364 |"}]

Author rebuttal:

Meta-review: This paper investigates hyperbolic latent embeddings to enable hierarchical open-vocabulary 3D scene understanding. The reviewers were in their initial reviews positive about the novelty. Integrating hyperbolic embeddings in radiance fields is interesting and new. The reviewers also agree that the paper is clearly presented.

The reviewers also pointed out a list of open questions, rengaing from using more VLMs, using outdoor scenes, why hyperbolic VLMs are not used, and much more. The AC has gone through the questions and responses and notes that the rebuttal clearly answers the concerns of the reviewers. The reviewers have given good pointers, which have been taken seriously by the authors. As a result, all reviewers vote for weak accept. The AC agrees and urges the authors to incorporate the outcomes of the discussions in the manuscript. Some of the questions were about hyperbolic space specifically (why Lorentz, why not MERU, etc), which can be resolved by including a part on hyperbolic learning e.g., in related work, which is currently focused on radiance fields only.
Decision: Accept (poster)