Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
Authors: Yash Sanjay Bhalgat
Iro Laina
Joao F. Henriques
Andrea Vedaldi
Andrew Zisserman
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Neural Radiance Fields
Instance Segmentation
Metric Learning
Clustering
3D Computer Vision
TL;DR: Our paper presents a novel "slow-fast" contrastive fusion method to lift 2D predictions to 3D for scalable instance segmentation, achieving significant improvements without requiring an upper bound on the number of objects in the scene.

Abstract: Instance segmentation in 3D is a challenging task due to the lack of large-scale annotated datasets. In this paper, we show that this task can be addressed effectively by leveraging instead 2D pre-trained models for instance segmentation. We propose a novel approach to lift 2D segments to 3D and fuse them by means of a neural field representation, which encourages multi-view consistency across frames. The core of our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with a large number of objects. Unlike previous approaches, our method does not require an upper bound on the number of objects or object tracking across frames. To demonstrate the scalability of the slow-fast clustering, we create a new semi-realistic dataset called the Messy Rooms dataset, which features scenes with up to 500 objects per scene. Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets, as well as on our newly created Messy Rooms dataset, demonstrating the effectiveness and scalability of our slow-fast clustering method.

Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
Yash Bhalgat
Iro Laina
João F. Henriques Andrew Zisserman
Andrea Vedaldi
Visual Geometry Group
University of Oxford
fyashsb, iro, joao, az, vedaldi)@roooota ox. o..ac. uk
Abstract
Instance segmentation in 3D is challenging task due to the lack of large-scale annotated datasets. In this paper, we show that this task can be addressed effectively by leveraging instead 2D pre-trained models for instance segmentation. We propose a novel approach lift segments to and fuse them by means of neural field representation, which encourages multi-view consistency across frames. The core our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with large number of objects. Unlike previous approaches, our method does not require an upper bound on the number of objects or object tracking across frames. To demonstrate the scalability of the slow-fast clustering, we create new semi-realistic dataset called the Messy Rooms dataset, which features scenes with up to 500 objects per scene. Our approach outperforms the state-of the-art art on challenging scenes from the ScanNet, Hypersim, and Replica datasets, as well effectiveness and scalability of our slow-fast clustering method.
our newly created Messy Rooms dataset, demonstrating the
Introduction
While the content of images is three-dimensional, image understanding has largely developed by treating images as two-dimensional patterns. This was primarily due to the lack of effective machine learning tools that could model content 3D. However, recent advancements in neural field methods [5, 40, 42, 49, 67] have provided an effective approach for applying deep learning to 3D signals. These breakthroughs enable to revisit image understanding tasks in 3D, accounting for factors such as multi-view consistency and occlusions.
In this paper, study the problem of object instance segmentation in 3D. Our goal is extend 2D instance segmentation to the third dimension, enabling simultancous 3D reconstruction and 3D instance segmentation. Our approach is extract information from multiple views of scene independently with pre-trained 2D instance segmentation model and fuse it into a single 3D neural field. Our main motivation is that, while acquiring densely labelled 3D datasets is challenging, annotations and pre-trained predictors for 2D data are widely available. Recent approaches have also capitalized on this idea, demonstrating their potential for 2D-to-3D semantic segmentation [31, 37, 60, 67] and distilling seneral-purpose 2D features in 3D space [29, 55]. When distilling semantic labels or features, the information to fused is inherently consistent across multiple views: semantic labels are viewpoint invariant, and 2D features across views are typically learned with the same loss function. Additionally, the number of labels or feature dimensions is predetermined. Thus, 3D fusion amounts to multi-view aggregation.
When it comes to instance segmentation, however, the number of objects in 3D scene is not fixed or known, and can indeed be quite large compared to the number semantic classes. More importantly, when objects are detected independently in different views, they are assigned different
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Images
Before Lifting
After Lifting
2D Panoptic Instance Segmenter
Contrastive Lift
Figure 1: Contrastive Lift takes as input several views of a scene (left), as well as the output of panoptic 2D segmenter (middle). It then reconstructs the scene in 3D while fusing the 2D segments, which are noisy and generally labelled inconsistently between views, when no object association (tracking) is assumed. Our method represents object instances in 3D space by Now-dimensional continuous embedding which can be trained efficiently using a contrastive formulation that is agnostic to the inconsistent labelling across views. The result (right) is consistent 3D segmentation of the objects, which, once imaged, results in more accurate and consistent 2D segmentations.
and inconsistent identifiers, which cannot be aggregated directly. The challenge is thus how to fuse information that is not presented in a viewpoint-consistenn manner.
Recently, Panoptic Lifting [48] proposed resolve the lack of multi-view consistency by explicitly fitting a permutation that aligns labels extracted from multiple views. Although this yields good results, there are two drawbacks this approach. Firstly, determining the permutation matrix involves solving linear assignment problem using Hungarian Matching for every gradient computation. The cost of this increases cubically with the number of identifiers, which may limit scalability when dealing with large number of object instances. Secondly, the canonical label space, where the permutation maps each 2D label, may need be extensive to accommodate a large number of objects.
In this study, we propose a more efficient formulation, which also leads to more accurate results. To understand our approach, consider first a 2D image segmenter: it takes an image / as input and produces mapping y that assigns each pixel ur E R²2 to an object instance label y(u) E {1..... L). is natural extend this mapping 3D by introducing function Y that associates each 31) point æ E R with the label Y (x) of the corresponding object. To account for the fact that labels are arbitrary and thus inconsistent between views, Panoptic Lifting [48] seeks an image-dependent permutation matrix /' such that Y (x) = P yu)). where u is the projection of a onto the image.
To address the aforementioned challenges with the mmeer-assignment basdd approach, identify the labels yuu) with coordinate vectors in the Euclidean space R., The functions y(u) can be reconstructed, up to label permutation, from the distances d(y(u), (u), v(u))) lly(u)) y(u Ill of such vectors, as they tell whether labels of two pixels (u, u') are the same or different, without considering the specific labelling. Notably, similar to compressed sensing, we seek lower-dimensional projections of the vectors y that preserve this information. With this mind, we replace the 3D) labelling function Y with low-dimensional Euclidean embedding ((z) IR". Then, we supervise the embeddings such that their distances d(O(z), O(z')) =s d(y(u), v(u') are sufficiently similar to that of corresponding 2D label embeddings.
This approach has two advantages. First, only requires learning vectors dimensionality D & L which is independent of the number of objects L. Second, learning this function does not require solving an assignment problem; rather, it only considers pairwise distances. Hence, the complexity of computing the learning objective is independent of the number of objects in the scene.
We translate this idea into a neural fusion field framework, which we call Contrastive Lift. We build the recent progress in self-supervised learning, and combine two key ideas: the usage of

---

contrastive loss, and the usage of a slow-fast learning scheme for minimizing the latter in a stable manner. We believe to be the first to introduce these two ideas in the context of neural fields.
We compare our method to recent techniques including Panoptic I Lifting [48] on standard 31) instance segmentation benchmarks, viz. ScanNet [13], Replica [50], and Hypersim [46]. To better demonstrate the scalability of our method to very large number of object instances, we introduce a seai-realistic Messy Rooms dataset featuring scenes with up to 500 objects.
2
Related Work
Neural Radiance Fields (NeRFs). NeRF [40] and its numerous variants [2, 5, 34, 36, 42] have achieved breakthrough results in generating photorealistic 3D reconstructions from 2D images of scene. These systems typically represent the scene as a continuous volumetric function that can be evaluated any 3D point, enabling high-quality rendering of novel views from any viewpoint.
Objects and Semantics in NeRF While NeRF by default offers low-level modelling of radiance and geometry, recent methods have expanded the set of tasks that can be addressed in this con- text to include semantic 3D modelling and scene decomposition. Some works use neural scene representations to decompose scenes into foreground and background without supervision or from weak signals [17. 41, 47, 56, 62, 63], such as text or object motion. Others exploit readily available annotations for 2D datasets to further extend the capabilities of NeRF models. For example, Semantic NeRF [67] proposes to incorporate a separate branch predicting semantic labels, while NeSF [60] predicts semantic field by feeding density field as input to 3D semantic segmentation model.
Closer to our work are methods that employ NeRFs to address the problem of 3D panoptic segmen- tation [19, 26, 31, 48, 61]. Panoptic NeRF [19] and Instance-NeRI [26] make use of 3D instance supervision. Panoptic Neural Fields [31], each instance represented with its own MLP but dynamic object tracking is required prior to training the neural field. In this work, we focus on the problem of lifting 2D instance segmentation to 3D without requiring any 3D masks or object tracks. A paper most related to work is Panoptic Lifting [48], which also seeks to solve the same problem, using linear assignment to make multi-view annotations consistent. Here, we propose a more efficient and effective technique based on learning cermutation-invarian embedding vectors instead.
Fusion with NeRF. The aforementioned works, such as Semantic NeRF [67] or Panoptic Lifting [48], are also representative of recent research direction that seeks to fuse the output of 2D analysis into 3D space. This is not new idea; multi-view semantic fusion methods [25, 35, 37, 38, 51, 59] predate and extend beyond NeRF. The main idea is that multiple 2D semantic observations (e.g.. noisy or partial) can be combined in 3D space and re-rendered to obtain clean and multi-view consistent labels. Instead of assuming 3D model, others reconstruct semantic map incrementally using SLAM [32, 43, 53]. Neural fields have greatly improved the potential of this idea. Instead of 2D labels, recent works, such as FFD [29], N3F [55], and LERF [27]. apply the 3D fusion idea directly to supervised and unsupervised dense features; in this manner, unsupervised semantics can be transferred to 3D space, with benefits such zero-shot 3D segmentation.
Slow-fast contrastive learning. Many self sulf-super learning methods are based on the idea of learning representations that distinguish different samples, but are similar for different augmenta- tions of the same sample. Some techniques build on InfoNCE [54, 57] and, like MoCo [23] and SimCLR [7], use contrastive objective. Others such as SWaV [3] and DINO [4] are based on online pseudo-labelling Many of these methods stabilise training by using mean- weaa-teeccher [52]. also called momentum encoders [23]. The idea is to have two versions of the same network: fast "student" network supervised by pseudo-labels generated from a slow "teacher" network, which is in turn updated as the moving average of the student model. Our formulation is inspired by this idea and extends it to learning neural fields.
Clustering operators for segmentation. Some works [14, 18, 30,45] have explored using clustering of pixel-level embeddings to obtain instance segment assignments. Recent works [64, 65] learn a pixel-cluster assignment by reformulating cross-attention from clustering perspective. Our proposed method, Contrastive Lift, similar spirit, although we learn the embeddings (and cluster centers) using volumetric rendering from 2D labels.
3

---

Sampled points 7,
Slow NeRF U
Renderer
Pixel embeddings 8g = R(
EMA update
Point embeddings
((r)).(r))
image I
Fast NeRF
Renderer
Contrastive
byr
Color e(r)) Density p(r))
0₂
Pairwise RBF kemel
4
Scene volume
Renderer
Concentration loss baae
Photometric lass Lygan
Real pixel color (())
Pixel color R(u(c.p)
Figure Overview of the Contrastive Lift architecture. See Section for details.
3 Proposed Method: Contrastive Lift
Here and in Fig. describe Contrastive Lift, our approach for fusing 2D instance segmentation in 3D space. An image is a mapping I
R3, where Ω is pixel grid in R2, and the values are RGB colours. We have set of images I captured in the same scene and, for each image E I, have its camera pose E (3) as well as object identity labels 2D instance segmentation model for the image I. The labels y assigned the 3D objects one image / and the labels y' in another image I' are in general not consistent. Furthermore, these 2D label maps be noisy across views.
Ω - (1,... (1..... L} obtained from
We use this data to fit a neural field. The latter is a neural network that maps 3D coordinates RS to multiple quantities. The first two quantities density, denoted by p R + [0, 1]. and radiance (colour), denoted by c R x S2 + [0, 1]3. Following the standard neural radiance field approach [36], the colour c(x: d) also depends the viewing direction d S The third quantity is a D-diiennsional instance embedding (vector) denoted as θ : R + RD, Each 3D coordinate also the semantic classes.
mapped to semantic embedding that represents distribution
Differentiable rendering. The neural field associates attributes (density, colour, and embedding vectors) to each 3D point E IR$. These attributes are projected onto an image taken from viewpoint π via differentiable ray casting. Given pixel location u E S2 in the image, we take N successive 3D samples E R3, i O, ..... N 1 along the ray from the camera center through the pixel (so that (u, f) & π (r.) where f is the focal length). The probability that a photon is not absorbed when travelling from sample to sample r1+1 is exp(-p(r:)6o) where d; |rr+11 rslla the distance between points, The transmittance т; exp( E'zo (6,)) is the probability that the photon travels through sample rú. The projection of any neural field onto pixel is thus given by the rendering equation:
N-1 Σ f(r)(( - -T))) = Σ f(ri)n 1=0
N-1
R(uff,
1=0
In particular, the colour of a pixel is reconstructed as I(u) ~ R(ulc(, d.). where the viewing direction d, = ro/ll roll2. The photometric loss is thus:
CROu (c, p\l)
1
Σ uEN
Ω
Instance embeddings and slow-fast contrastive learning. The photometric loss (2) learns the colour and density fields (c, p) from the available 2D views I. Now we turn to learning the instance embedding field θ R + RD_ As noted in Section 1, the goal of the embeddings is to capture the (binary) distances between pixel labels sufficiently well. By that, we mean that the segments can be recovered, modulo a permutation of their labels, by simply clustering the embeddings posteriori.

---

We cast learning the embeddings as optimising the following contrastive loss function:
Ewen =v())=((w)) exp(sim(Ou, (w:)))
Connt Senntr(,, (0, =
log
8,. - R( p.
[S2]
Ewen exp(sim (0,, (2))))
uE
where 1 is the indicator function, and sim 7 exp(-> x'1l3) is Gaussian RBF kernel used to compute the similarity between embeddings in Euclidean space. Therefore, pixels that belong to the same segment are considered positive pairs, and their embeddings are brought closer, while the embeddings of pixels from different segments are pushed apart. It is worth emphasizing that, since the object identity labels obtained from the underlying 2D segmenter are not consistent across images, Lcomt is only applied to positive and negative pixel pairs sampled from the same image.
While Eq. (3) is logically sound, we found it to result in gradients with high variance. To address this, we draw inspiration from momentun--eacher approaches [1, 4, 24] and define slowly-updated instance embedding field é, with parameters that are updated with an exponential moving average of the parameters of θ, instead of gradient descent. With this, we reformulate Eq. (3) as:
Ewen, v(u)=wtw) (O,, Euren, exp(sim(Ou: 0w))))
Car(O,ply, = [221
log uE
(4)
where 0, R(u|R, P3 TT, and 0.. R(u' @ p,,, Here, we randomly partition the pixels Q into two non-overlapping sets 22 and f22, one for the "fast" embedding field θ, and another for the "slow" field é. This avoids the additional cost of predicting and rendering each pixel's embedding using both models, and allows the computational cost remain the same as for Eq. (3).
Concentration loss. In order to further encourage the separation of the embedding vectors and thus simplify the extraction of the objects via posteriori clustering, introduce a loss function that further encourages the embeddings to form concentrated clusters for each object:
O, Enreo, 12(w) Ewen, In(()=((w))
Ceone O Plu = [2
Σ wEB
This loss computes a centroid (average) embedding as predicted by the "slow" field Ô and penalizes the squared error between each embedding (as predicted by the "fast" field +) and the corresponding centroid. While this loss reduces the variance of the clusters, it is not sufficient training objective by itself as it does encourage the separation of different clusters, as done by Eqs. (3) and (4).
Semantic segmentation. For semantic segmentation, we follow the same approach as Semantic NeRF [67], learning additional embedding dimensions (one per semantic class), rendering labels in the same manner as Eq. (I), and using the cross-entropy loss for fitting the semantic field. Additionally, also leverage the segment consistency loss introduced in [48] which encourages the predicted semantic classes to be consistent within an image segment.
Architectural details. Our neural field architecture is based on TensoRF [6]. For the density, we use a single-channel grid whose values represent the scalar density field directly. For the colour, multi-channel grid predicts an intermediate feature which is concatenated with the viewing direction and passed to a shallow 3-layer MLP to predict the radiance field. The viewing directions are encoded using a frequency encoding [40, 58]. For the instance embedding field θ (and also the "slow" field é which has the exact same architecture the "fast" field θ), we use shallow 5-layer MLP that predicts an embedding given an input 3D coordinate. The same architecture is used for the semantic field. We use raw 3D coordinates directly without frequency encoding for the instance and semantic components. More details are provided in the supplementary material.
Rendering instance segmentation maps. After training is complete, we sample 10y pixels from 100 random viewpoints (not necessarily training views) and render the fast instance field at these pixels using the corresponding viewpoint pose. The rendered 10 D embeddings are clustered using HDBSCAN [39] obtain centroids, which are cached. Now, for any novel view, the field θ rendered and for each pixel, the label of the centroid nearest to the rendered embedding is assigned.
5

---

De preco
Figure Messy Rooms dataset visualization. Left: physically realistic static 3D scene with N objects from GSO [15]. Middle: M camera viewpoints sampled in dome-shaped shell. Right: ground-truth RGB and instance IDs, and instance segmentations obtained from Detic [68].
4 Messy Rooms Dataset
In order to study the scalability of our method to scenes with a large number of objects, we generate a semi-realistic dataset using Kubric [20]. To generate a scene, we first spawn N realistically textured objects, randomly sampled from the Google Scanned Objects dataset [15], without any overlap. The objects are dropped from their spawned locations and a physics simulation is run for a few seconds until the objects settle in natural arrangement. The static scene is rendered from M camera viewpoints randomly sampled dome-shaped shell around the scene. Background, floor, and lighting are based on 360 HDRI textures from PolyHaven [66] projected onto dome.
Specifically, create scenes with N 25, 50, 100, and 500 objects. The number of viewpoints, M set to min (1200, [600 22/25)), and the rendered image resolution is 512x512 To ensure that the focus is on the added objects, we use background textures old_ old_romm and crrge_corridrr from PolyHaven that do not contain any objects. total of scenes are generated. The use of realistic textures for objects and background environments makes them representative of real-world scenarios.
Additionally, would like maintain consistent number of objects per image the total number of objects so that the performance of the 2D segmenter is not a factor in the final performance. Firstly, ensure that the floor area of the scene scales proportionally with the number of objects, preventing objects from becoming densely packed. Secondly, the cameras move further away from the scene as its extent increases. To ensure that the same number of objects visible in each image, regardless of the scene size, we adjust the focal length of the cameras accordingly, i.e., f = 35.0 x / 1/25, creating an effect similar to magnification. This approach ensures a comparable object distribution in each image, while enabling us to study the scalability of our method.
increase
We render the instance IDs from each camera viewpoint to create ground-truth instance maps. These ground-truth instance IDs remain consistent (tracked) across views, they are rendered from the same 3D scene representation.' Figure shows illustrative examples from the dataset, which we name Messy Rooms. For evaluation (Section 5), semantic maps are required. As there is large variety of different object types in Kubric, there is no off-the-shelf detector that can classify all of these, and since we are interested in the instance segmentation problem, rather than the semantic classes, we simply lump all object types in a single "foreground" class, which focuses the evaluation on the quality of instance segmentation. More details about the dataset are provided in Appendix A.
Experiments
Benchmarks and baselines. We train and evaluate proposed method challenging scenes from the ScanNet [13]. Hypersim [46], and Replica [50] datasets. We compare our method with Panoptic Lifting (PanopLi) [48], which is the current state-of-the-art for lifting 2D panoptic predictions to 3D, along with other 3D panoptic segmentation approaches: Panoptic Neural Fields [31] and
In all experiments, tracked ground-trute truth instance maps used only for evaluation and not train models.

---

Table Results on ScanNet, Hypersim, and Replica datasets. The performance of all prior work has been sourced from [48]. For each dataset, we report the PQ metric.
Method
ScanNet [13]
HyperSim [46]
Replica [50]
DM-NeRF [61]
41.7
51.6
44.1
PNF [31]
48.3
44.8
41.1
PNF + GT BBoxes
54.3
47.6
52.5
PanopLi [48]
58.9
60.1
57.9
Vanilla (Ours)
60.5
60.9
57.8
Slow- (Ours)
62.3
62.3
59.1
Table Results on the Messy Rooms dataset. PQ metric is reported on "old room" and "large corridor" environments with increasing number of objects in the scene (N 25, 50, 100, 500).
Old Room environment
Large Corridr environment
Method
25 Objects
50 Objects
100 Objects
500 Objects
25 Objects
50 Objects
100 Objects
500 Objects
PanopLi [48]
73.2
69.9
64.3
$1.0
65.5
71.0
61.8
49.0
Vanilla (Ours)
74.1
71.2
63.6
49.7
67.9
69.3
62.2
47.2
Slow-Fast (Ours)
78.9
75,8
69.1
55.0
76.5
75,5
68.7
52.5
DM-NeRF [61]. We follow PanopLi [48] for the data preprocessing steps and train-test splits for each scene from these datasets. We also evaluate our proposed method and PanopLi on our Messy Rooms dataset (Section 4) that features scenes with up to 500 objects. These experiments aim to demonstrate the scalability of our proposed method as compared to the lincar-assignan assignment approach.
We compare two variants of our Contrastive Lift method: (1) Vanilla: uses the simple contrastive loss (Eq. (3)), and (2) Slow-Fast: uses slow-fast contrastive (Eq. (4)) and concentration (Eq. (5)) losses.
Metrics. The metric used in our evaluations is the scene-level Panoptic Quality (PQ metric introduced in [48]. PQ is scene-level extension of standard PQ [28] that takes into account the consistency of instance IDs across views/frames (aka tracking). In PQuceme, predicted/groumc truth segments with the same instance ID across all views are merged into subsets and all pairs of predicted/ground--ruut subsets are compared, marking them as a match if the loU is greater than 0.5.
Implementation Details. We train neural field model for 400k iterations all scenes. Optiiizttion-reltte hyper-parameters can be found in Appendix B.2. The density grid is opti- mised using only the photometric loss (CRgn). While rendering the instancc/semantic fields and computing associated losses (Eqs. (3) to (5)), gradients are stopped from flowing to the density grid.
For experiments on ScanNet, Hypersim and Replica, we use Mask2Former (M2F) [8] as the 2D seg- menter to obtain the image-level semantic labels and instance identities. Although any 2D segmenter can be used, using M2F allows direct comparisons with other state-of-the-art approaches [48]. We follow the protocol used in [48] to map the COCO [33] vocabulary to 21 classes ScanNet.
For experiments on Messy Rooms, we use Detic [68] instead since the object categories are not isomorphic the COCO vocabulary M2F uses. We use the LVIS [21] vocabulary with Detic. To show the scalability of our method compared to a linear-assie wwwaaaassiinnbbbaaae
approach, we train the PanopLi [48] model on this dataset. For fair comparison, we first train the density, colour and semantic fields, which are identical in PanopLi and our approach. We then separately train the instance field using the respective linear-assipen assignmettt and slot-fast contrastive losses, with all other components frozen, ensuring that performance is only influenced by the quality of the learned instance field.
5.1 Results
In Table 1, we compare the performance of our proposed approach with existing methods on three datasets: ScanNet [13], HyperSim [46], and Replica [50]. Since the semantic field and underlying TensoRF [5] architecture we use is similar to Semantic-NeRF [67] and PanopLi [48], we only report the PQsseme metric here and have added an additional table to Appendix where we show that the mloU and PSNR of our method match the performance of prior methods expected. We observe that the proposed Slow-Fast approach consistently outperforms the baselines on all three datasets, while also outperforming the state-of-the-arr Panoptic Lifting [48] method by +3.9, + +1.4 and +0.8
7

---

GT
DETIC (35)
Panopli [23]
Ours
(a) Messy Rooms: large_corridor (25 objects)
(b) Messy Rooms: old_room (25 objects)
Figure 4: Qualitative comparisons of our method with PanopLi [48] and Detic [68] (underlying 2D segmenter model) on scenes from our Messy Rooms dataset. Colour coding: regions where PanopLi performs poorly are highlighted with red boxes, while regions where both PanopLi and our method exhibit poor performance are marked with blue boxes. Additionally, red arrows indicate instances where PanopLi i fails to distinguish between different objects. Please zoom in to observe finer details.
Table Ablations of different variants of the Contrastive Lift method. PQueme metric averaged over the scenes of ScanNet and Messy Rooms datasets is reported. Embedding size of is used.
Dataset
Lst + Loone
Est
Coontr
Lecont + Loone (fast)
ScanNet [13]
62.0
61.3
60.5
55.2
Messy Rooms
69.0
66.5
63.2
51.7
PQuceme points on these datasets respectively. We note that the Vanilla version of our method also performs comparably with PanopLi and outperforms other methods on all datasets.
Table 2 shows comparisons between method and PanopLi [48] on scenes from Messy Rooms dataset with 25, 50, 100, and 500 objects. We see that the margin improvement achieved by Contrastive Lift over PanopLi even larger these scenes, which shows that the proposed method scales favorably to scenes with a large number of objects. Fig. 4 shows qualitative results on two of these scenes. Even though the 2D segments obtained using Detic [68] are noisy (sometimes over- segmented) and generally labelled inconsistently between views, the resulting instance segmentations rendered Contrastive Lift are clearer and consistent across views. We also note that PanopLi sometimes fails to distinguish between distinct objects as pointed out in Fig. 4b.
5.2 Ablations
Different variants of Contrastive Lift. Our proposed method uses Lat (Eq. (4)) and Leune (Eq. (5)) to optimise the instance embedding field. To study the effect these losses, we design a comprehensive of variants of the proposed method: (1) Proposed (C.r Ceone), (2) Proposed without Concentration loss (La),, (3) Vanilla contrastive (Lcont), (4) Vanilla contrastive with Concentration loss applied to "fast" field since there no "slow" field (Ccomir + (fast)). Table 3 shows these ablations.
Effect embedding size on performance. We investigate the impact of varying the instance embedding size on the performance of our proposed Contrastive Lift method. Specifically, we evaluate the effect of different embedding sizes using the PQ metric on ScanNet, Hypersim and Replica datasets. As shown in Fig. 5, we find that an embedding size as small as 3 is already almost optimal. Based on this, we use an embedding size of 24 for experiments with these datasets (c.f. Table 1). For experiments with Messy Rooms dataset (c.f. Table 2), we keep the embedding size to 3.

---

0.62
0.58
0.62
80
0.61
0.58
0.61
0.59
0.56
0.59
0.55
3
12
24
48
12
24
48
96
192
12
24
48
102
Embedding size
Embedding size
Embedding size
(a) ScanNet [13]
(b) Replica [50]
(c) Hypersim [46]
Figure 5: Impact of the embedding size the performance (PQueme) of the instance module.
Contrastive
Slow-Fast Contrastive
Vanilla Contrastive
Stow-Fast Contrastive
(a) ScanNet scene0300__ scene
(b) Messy Rooms large_ corridorr 25 scene
Figure 6: Embeddings obtained using vanilla (plain) contrastive learning and our proposed Slow-Fast contrastive learning. We use LDA [22] to project the embeddings to 2D for the illustration here.
Qualitative evaluation: Slow-fast vanilla contrastive learning. Fig. shows how the embed- dings are distributed in Euclidean space when learned using our proposed slow-fast contrastive loss (Eqs. (4) and (5)) and the vanilla contrastive loss (Eq. (3)). Embeddings learned with the slow-fast method are clustered more compactly and are easy to distinguish using any post post-processing algorithm, such HDBSCAN [39] which is used in this example.
Comparison to underlying 2D instance segmentation model with tracking. Before lifting, the predictions of the underlying 2D instance segmentation model (c.g.. Mask2Former [8] or Detic [68]) are not consistent (aka tracked) across frames/views. To achieve consistency and to allow comparisons with our approach, post-process the 2D segmenter's predictions using Hungarian Matching for cross-frame tracking as follows:
1. w/ Hungarian matching (2D IoU): Given sets predicted segments (P, and P.-1) from consec- utive frames, compute IoU matrix by comparing all segment pairs in P. P.+1- Apply Hungarian matching to the IoU matrix to associate instance segments across frames.
w/ Hungarian matching based on IoU after depth-aware pose-warping: Use ground-truth pose and depth for warping (i (i+ 1) frame's 's segmentation to frame i. Compute IoU matrix using warped segmentations and apply Hungarian matching.
3. w/ Hungarian matching using ground-truth pointcloud: Using only consecutive frames leads to errors in long-range tracking. To address this, starting from the first frame, unproject segments into the 3D point cloud. Iteratively fuse segments in 3D using Hungarian matching. This way, segments from preceding frames along with 3D information are used for tracking.
The last two baselines use 3D groundtruth for tracking. Table 4 shows that despite 3D information being used for matching, Contrastive Lift still significantly improves over the underlying 2D model.
Frame-level improvement on underlying 2D segmentation models. In addition to generating consistent (tracked) instance segmentations, our method also improves the per-frame quality (i.e., not considering tracking) of the underlying 2D segmentation model. To show this, we train Contrastive Lift on ScanNet scenes with different 2D models, viz. Mask2Former [8], MaskFormer [9] and Detic [68]. Table 5 we report the Panoptic Quality (PQ) metric (computed per frame) for these 2D models and for method when trained with segments from each corresponding model.

---

Table Comparison of our approach with the underlying 2D segmentations on ScanNet [13]. For
M2F predictions [8], consistency across frames is obtained with different tracking variants.
Method
PQ
Mask2Former [8] (M2F) (non-tracked)
32.3
M2F w/ Tracking method (1)
33.7
M2F w/ Tracking method (2)
34.0
M2F w/ Tracking method (3)
41.0
Contrastive Lift (ours trained w/ Mask2Former lahels)
62.3
Table 5: Improvement of per-frame segmentation quality as measured by Panoptic Quality (PQ).
Method
PQ
MaskFormer [9]
41.1
Contrastive Lift (w/ MaskFormer labels)
61.7
Mask2Former [8]
42.0
Contrastive Lift (w/ Fask2Former labels)
61.6
Detic [68]
43.6
Contrastive Lift (w/ Detic labels)
62.1
Comparison of training speed with the linear-assignmett loss method. While the exact number of objects present in a scene is unknown, linear assignment-basee methods typically require hyperparameter that specifies the maximum number of objects. Solving the linear assignment problem in PanopLi's 's loss is O(K3) [48]. Our method is agnostic to object count, eliminating the need for such parameter. Our approach does rely on the size of the embedding size, but, as shown above, even very small size suffices. In the slow-fast contrastive loss computation, the Softmax function dominates more than the pairwise similarity matrix calculation. Consequently, we find that the training speed of Contrastive Lift is largely unaffected by the choice of embedding size.
Table compares the training speed, measured on NVIDIA A40 GPU, between PanopLi and our method, showing that PanopLi iterations become slower as K increases. We only optimise the instance embedding field with associated losses, while the densitylcolouur/smante fields are frozen.
Table 6: Training speed iterations/sccond. Mean error margin measured over 8 runs.
Panoptic Lifting [48]
Contrastive Lift
K = 25
K = 50
K 100
K = 500
16.06 +: 2.34
13.01 + 1.26
12.53 + 0.92
12.10 + 1.07
9.41 + 0.60
6
Limitations
Contrastive Lift improves noisy 2D input segmentations, but cannot recover from catastrophic failures, such entirely missing object classes. It also requires the 3D reconstruction work reliably. As result, we have focused on static scenes, as 3D reconstruction remains unreliable in a dynamic setting. Contrastive Lift is a useful building block in applications, but has no particular direct societal impact. The datasets used in this paper are explicitly licensed for research and contain no personal data.
7 Conclusion
We have introduced Contrastive Lift, method for fusing the outputs of 2D instance segmenter using a 3D neural fields. It learns 3D vector field that characterises the different object instances manner which is invariant
the scene. This field is fitted to the output of the 2D segmenter to permutation of the object labels, which are assigned independently and arbitrarily in each input image. Compared to alternative approaches that explicitly seek to make multi-view labels compatible, Contrastive Lift is more accurate and scalable, enabling future work on larger object collections.
10

---

Acknowledgments and Disclosure of Funding
We are grateful for funding from EPSRC AIMS CDT EP/S024050/1 and AWS (Y. Bhalgat), ERC- CoG UNION 101001212 (A. Vedaldi and I. Laina), EPSRC VisualAI EP/T028572/1 (I. Laina, A. Vedaldi and Zisserman), and Royal Academy of Engineering RF200819M181163 (J. Henriques).
Data Ethics. We use the Google Scanned Objects, ScanNet, Hypersim and Replica datasets fol- lowing their terms and conditions. These datasets do not contain personal data. For further de- tails on ethics, data protection, and copyright please see https: /ttp:://wwwww www.wrrobot....m ox. ac uk/"vedaldi research/ /niion/ ceiiion/thiiee ethiics...... .tml.
References
[1] Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. Coke: Contrastive learning for robust keypoint detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 65 74, 2023. 5
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualle and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In Proc. NewrIPS, 2020. 3
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. ICCV, 2021. 5, 17
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In arXiv, 2022.
3,3,7
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey E. Hinton. simple framework for contrastive learning visual representations. Pruc. JCML, 2020. 3
[8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022. 7, 7,9, 10
[9] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. 2021. 9, 10
[10] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 JEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (05), volume 1, pages 339-546. IEEE, 2005. 17
[11] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence, 24(5):603 619, 2002. 16
[12] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 25
[13] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017. 6, 7, 8, 9, 10, 16, 17
[14] Bert De Brabandere, Davy Neven, and Lue Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv: (t0.::70..2251111 2017.
[15] Laura Downs, Anthony Francis. Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned houschold items. In 2022 International Conference on Robotics and Automation (ICRA), pages
2553-2560. IEEE, 2022.
[16] Martin Ester, Hans- Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226 231, 1996. 16

---

[17] Zhiwen Fan, Peihao Wang, Yifan Jiang. Xinyu Gong, Dejia Xu, and Zhangyang Wang. NeRF-SOS: Any- view self-superrised object segmentation on complex scenes. In The Eleventh International Conference on Learning Reprexentations, 2023.
[18] Alireza Fathi, Zhigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin P Murphy. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv: 1703. 10277, 2017.
[19] Xiao Fu, Shangzhan Zhang. Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic NeRF: 3d-to-2d label transfer for panoptic urban scene segmentation. arXivees. abs/2203. 152224, 2022. 3
[20] Klaus Greff, Francois Bellctti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. 2022.
[21] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A datasct for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3356-5364, 2019. 7
[22] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. arXiv.es. cs. abs/1911.05722. 2019.
[24] Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jacgle, Andrew Zisserman, João Carreira, and Relja Arandjelovié. Object discovery and representation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pages 123-143. Springer, 2022. 5
[25] Alexander Hermans, Georgios Floros, and Bastian Leibe. Dense 3d semantic mapping of indoor scenes from rgb-d images. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 2631 2631.2638. IEEE, 2014. 3
[26] Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. arXiv preprint arXiv:2304.0439)) 04395, 2023. 3
[27] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.0955: 2023. 3
[28] Alexander Kirillov, Kaiming Hc, Ross B. Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In Proc. CVPR, 2019. 7
[29] Sasuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. arXiv.ex, 2022. 1,3
[30] Shu Kong and Charless C Fowlkes. Recurrent pixel embedding for instance grouping. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9018-9028, 2018.
[31] Abhijit Kundu, Kylc Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Lconidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871-1288 2022. 1, 6,7
[32] Abhijit Kundu. Yin Li, Frank Dellaert, Fuxin Li, and James Rehg. Joint semantic segmentation and 3d reconstruction from monocular vidco. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 703-718. Springer, 2014. 3
[33] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014. Proceedings. Part V 13, pages 740-755. Springer, 2014.7 7
[34] Lingjee Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems,
2020. 3
12

---

[35] Lingni Ma, Jörg Stückler, Christian Kerl, and Daniel Cremers. Multi-view deep learning for consistent semantic mapping with RGB-D cameras. In Pro,.RRO,, 2017.
[36] Ricardo Martin-Brualla Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proc. CVPR, volume abs/2008. 022668. 2021. 3,4
[37] Ruben Mascaro, Lucas Teixeira, and Margarita Chli. Diffuser: Multi-view 2D-to-3D label diffusion for semantic scene segmentation. In Proc. ICRA, 2021. 1,3
[38] John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pages 4628 4635. IEEE, 2017. 3
[39] Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11)) 2017. 5,9, 15, 16
[40] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Proc. ECCV, 2020. 1,
[41] Ashkan Mirzaei, Yash Kant, Jonathan Kelly, and Igor Gilitschenski, Laterf: Label and text driven object radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III. pages 20-36. Springer, 2022. 3
[42] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies primitives with multiresolution hash encoding. In Proc. SIGGRAPH, 2022. 1,3
[43] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In 2019 JEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pages 4205-4212 12. IEEE, 2019. 3
[44] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. Advances in neural information processing systems, 30, 2017. 17
[45] David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Semi-convolutional operators for instance segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 86-102, 2018. 3
[46] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webh, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10912-10922, 2021. 3, 6. 7, 9, 15, 16
[47] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T. Freeman. Fredo Durand, Joshua B. Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural scene representations from a single image. In The Eleventh International Conference on Learning Reprexentations, 2023.
[48] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9043-9052, June 2023. 2, 5, 6, 7, 8, 10, 15, 16, 17
[49] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure aware neural scene representations. Proc. NeurIPS, 2019.
[50] Julian Straub. Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint 1rXiv:/906.000 05797, 2019. 3, 6, 7,9, 16
[51] Niko Sünderhauf, Trung T Pham, Yasir Latif, Michael Milffor, and Ian Reid. Meaningful maps with object-oriented semantic mapping. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 0079-5085. IEEE, 2017.
[52] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proc. NeurIPS, 2017. 3
[53] Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. Cnn-slam: Real-time time dense monocular slam with learned depth prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6243-6252, 2017.
13

---

[54] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In Proc. ICLR, 2020.
[55] Vadim Tschemezki,, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In Proceedings of the International Conference on Vision (3DV), 2022. 3,3
[56] Vadim Tschemezki, Diane Larlus, and Andrea Vedaldi. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In Proceedings of the International Conference on 3D Vision (3DV), 2021. 3
[57] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, Pruc. NewrIPS, 2019.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017. 5
[59] Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A Prisacariu, Olaf Kähler, David W Murray, Shahram Izadi, Patrick Pérez, al. Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction. In 2015 IEEE international conference on robotics and automation (ICRA), pages 75 82, IEEE, 2015. 3
[60] Suhani Vora, Noha Radwan, Klaus Greff, Heaning Meyer. Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic segmentation of 3D scenes. arXiv.cs, abs/21 11.13260, 2021. 1,3
[61] Bing WANG, Lu Chen, and Bo Yang. DM-neRF: 3d scene geometry decomposition and manipulation from 2d images. In The Eleventh International Conference Learning Representations, 2023. 3.7
[62] Christopher Xie, Keunhong Park, Ricardo Martin-Brualla and Matthew Brown, Fig-nerf: Tigure-ground neural radiance fields for 3d object category modelling. In 2021 International Conference on 3D Vision (3DV), pages 962 971. IEEE, 2021.
[63] Hong- Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In International Conference on Learning Representations. 2022. 3
[64] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang Chich Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2560-2570 2022. 3
[65] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. k-means mask transformer. In European Conference on Computer Vision, pages 288 00--307.666 Springer, 2022. 3
[66] Greg Zaal, Rob Tuytel, Rico Cilliers, James Ray Cock, Andreas Mischok, Sergej Majhoroda, Dimitrios Savva, and Jurita Burger. Polyhaven: curated public asset library for visual effects artists and game designers. https: httpp:://polyaaeeeee com/hdris, 2021.
[67] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15838-15847, 2021. 1,3, 5, 7, 15, 16
[68] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh,, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. ECCV, 2022. 6, 7, 8, 10
14

---

A Messy Rooms dataset
The full Messy Rooms dataset introduced in this work can be accessed at this link: https:/ figshare comaarrecco/ceeeooo b wewcceeeeeaaaf7776500.
We show some representative examples from this dataset below in Fig. 14, 15, 16, and 17, which illustrate scenes with 25, 50, 100 and 500 objects respectively. Notice how the density of "number of objects per image" remains similar as the number of objects increases from 25 to 500. In Fig. 18, we show the corresponding 3D scenes used to generate the datasets.
B Implementation Details
Here, we provide further implementation details for our method in addition to the details mentioned Sections and 5 of the main paper.
B.1 Architectural details
Our neural field architecture is similar to [48] for fairness of comparisons. The density and color grids are initialized with resolution of 128 128 128 which is progressively increased up to 192 x 192 x 192 by the end of training. The density and color grids use 16 and 48 components respectively. The output of the color grid is projected 27 dimensions which are then processed by 3-layer MLP with 128 hidden units per layer to output the RGB color. The fast and slow instance fields use a 256 hidden size in their MLP, while the semantic field uses a hidden size of 128.
B.2 Training details
We follow a schedule for training our neural field model as follows: (1) For the first 40k iterations, the model is trained only with the RGB reconstruction loss (CRGB). In this initial phase, the density field optimized to reach a reasonable quality such that can be used to render the instance/semantic field. (2) At 40k iterations, the semantic segmentation loss (i.e., cross-entropy loss, as in [48, 67J) is activated and used for the rest of the training iterations. (3) At 160k iterations, the instance embedding loss (i.e., Lg Loone for the slow-fast version of our method Leont for the vanilla baseline) is activated. (4) At 280k iterations, the segment consistency loss (proposed in [48]) is activated. For scenes from the Hypersim dataset [46], we activate the segment consistency loss at 200k iterations instead. In our proposed slow-fast clustering framework, the slow field parameters are updated using an exponential moving average with momentum m = 0.9, i.e. Ô = Ô x m + θ (1 - -m).
The RGB reconstruction loss, semantic segmentation loss, instance embedding loss, and segment consistency loss are balanced using weights of 1.0, 0.1, and 1.0 respectively. However, we empirically observe that the final performance is not very sensitive to these choices. A learning rate of 5-10- 4 is used for MLPs and 0.01 for the grids. A batch-size of 2048 used to train all models.
B.3 Post-processing Clustering details
Given the learned instance embedding field, a clustering mechanism (e.g.. HDBSCAN [39]) can be used to obtain cluster centroids and generate instance segmentation maps. We have chosen HDBSCAN since it does not require the number of objects to be known a priori. Generally, clusters obtained by HDBSCAN recommended. But, our proposed method results in highly compact clusters which makes this simple method effective. We perform clustering as follows.
non-convex and assigning the label of the nearest centroid is
Hierarchical clustering using semantic predictions. An advantage of our method is that we can model's 's semantic predictions to guide the clustering of the instance embeddings. "Instance" segmentation requires separating instances of the same semantic class. Based on this, we perform hierarchical clustering follows:
1. After training, sample 10a pixels from 100 random viewpoints and render the fast instance field θ and semantic field for these pixels.
2. Group the 103 rendered embeddings based on predicted semantic labels, forming S groups
15

---

3. Cluster the embeddings within each group separately using HDBSCAN, caching cluster-centrosd for each group, assigning unique instance label each centroid.
4. For novel view, render the instance field and semantic field, assigning each pixel an instance embedding and semantic class. Obtain the instance label for pixel by finding the closest centroid to the rendered instance embedding within the group of the same semantic label as the pixel.
Tuning clustering hyperparameter. Despite HDBSCAN's robustness to hyperparameter selection, we found that it is beneficial to specify a minimum cluster size. Since always sample and render 10 pixels for clustering, the expected cluster size per object decreases as the number of objects increases. To determine optimal value, perform hyperparameter sweep using 10% of the training data, which includes training viewpoints and associated segments from the 2D segmenter. We then this identified optimal value to perform clustering described above.
C Comparison between different clustering algorithms.
We compare HDBSCAN with other unsupervised clustering algorithms, viz. MeanShift [1]] and DBSCAN [16]. We tune the bandwidth parameter with MeanShift, and the epsilon parameter with DBSCAN. However, we note that MeanShift struggles to converge for embedding sizes greater than 10. For fair comparison,
train model with embedding size of 3. Table show that both MeanShift and DBSCAN perform slightly worse but remain comparable to HDBSCAN. Generally, any unsupervised clustering method that doesn" 't require prior knowledge of the number of clusters is suitable for use with our method.
Table 7: Performance (PQucene) achieved with DBSCAN [16], MeanShift [11] and HDBSCAN [39].
ScanNet [13]
Messy Rooms
w/ DBSCAN
61.8
68.2
MeanShift
62.0
68.6
w/ HDBSCAN
62.0
69.0
D Quality of our semantic and radiance field
In Tables 1 and in the main paper, evaluate the quality and consistency (aka tracking) of the instance segmentation maps obtained by the various tested methods. The semantic field and density/color field architecture of our method is based on Panoptic Lifting [48], which in turn is modification of Semantic-Nele [67] for the semantic component. As a sanity check, we compare the quality of rendered semantic and RGB maps obtained by these methods with ours. Table shows the mean Intersection over Union (mloU) and seak-signallto-noise ratio (PSNR) metrics. As expected, the mIoU and PSNR obtained by our method is nearly the same Panoptic Lifting.
For the Messy Rooms dataset we have explicitly ensured that the density and semantic model used by both Panoptic Lifting and method are the same and the only factor influencing the final performance is the quality of the learned instance field. This is done by pre-training the same density and semantic fields for both methods and subsequently training the instance field using the respective objective functions.
Table Comparisons of the rendered semantic and RGB maps. Performance numbers for [48, 67] are sourced from [48].
ScanNet [13]
HyperSim [46]
Replica [50]
Method
mloU
PSNR
mloU
PSNR
mloU
PSNR
Semantic-NeRF [67]
58.9
26.6
58.5
24.8
59.2
26.6
PanopLi [48]
65.2
28.5
67.8
30.1
67.2
29.6
Ours
65.2
28.3
67.9
30.0
67.0
29.3
16

---

E
Comparisons to other metric learning loss functions
While we employ a contrastive loss formulation learn the instance embeddings, there are many alternative loss functions proposed in the metric learning literature. For comparison, we also train our instance embedding field with the Associative Embedding (AE) loss [44] and the margin-based contrastive loss [10].
To compute the AE Loss, we divide the batch Ω into groups based segment ID. If there K groups/segments, G1 Gx. then
1 |
LAE(O,ply)
Σ
Σ
10
0xl12 +
10
12.
ON
On
(6)
K
|G
NEGb
uEGz
The margin-based contrastive loss is defined as:
Lmargin (margin(O,,pl9)) = [22/2
Σ M,WER
[o()) w(())=(aa]]]l Our 112 1 ((()))((())) max(0, c- --110, - 0w (7)
Here, 0₂ RRu O pT is the rendered instance field at pixel Note that, the slow-fast field formulation is not used in these comparisons. In Table we compare the proposed objective (slow- fast) to these baselines. We observe that both the vanilla contrastive, as well the slow-fast version of our method, outperform the alternatives.
DINO-style loss. Since method is inspired by momentum teacher approaches, e.g. DINO [4], design a baseline with DINO-style learning mechanism. Two pixels from the same instance segment fed into the slow and fast fields. This is akin DINO, where two random image transformations are fed to the student and teacher networks. A Centering layer is applied to the slow field embedding. Projection module (with proj_dim 512) is added to both the slow and fast fields followed by Softmax and a cross-entropy loss is used. After training, embeddings from the fast field are used for clustering, Results in Table 9 demonstrate that this baseline performs worse than the metric-learning losses on ScanNet. We do not evaluate this baseline on the Messy Rooms dataset.
Table 9: Comparing Associative Embedding Loss and Triplet Loss with proposed losses (vanilla and slow-fast). An embedding size is used in all cases. PQuceme is reported here.
ScanNet [13]
Messy Rooms
Panoptic Lifting [48]
58.9
63.2
Ours w/ AE loss (CAE)
60.0
62.4
Ours w/ Margin loss (Lmargin)
60.1
62.9
Ours w/ DINO-style loss
54.7
Ours w/ Vanilla contrastive loss (Coomr)
60.5
63.1
Ours w/ Slow-Fast losses (proposed)
62.0
69.0
Stability of Slow-Fast loss compared to Vanilla contrastive loss
We found that training with the vanilla contrastive loss resulted in gradients with higher variance. The usage of a slowly-updated embedding field in the slow-fast loss formulation mitigates this problem and leads more stable training. We quantitatively verify this by computing the relative variance (which is Var()/Mean()) in the gradients of the loss w.r.t. to the instance embeddings (i.e. dL/de). Figure shows that the vanilla loss exhibits spikes with a maximum relative variance around 10 whereas the slow-fast version remains around much controlled range of around 10º
G More qualitative visualizations
In Figures 8, 9, 10, 11, 12 and 13, we visualize the predictions of our proposed method on scenes from ScanNet [13] and Messy Rooms. Left-most columns show instance labels obtained after
17

---

Vanilla Loss
SlowFast Loss
Vanilla
SlowFast
Relative Variance of dL/de
Relative Variance of dL/de
2c+07
1c+07
Se+06
0c+00
10000
20000
30000
40000
10000
20000
30000
40000
Iterations
Iterations
Figure 7: Relative variance (i.e. Var (/a(aa/aaa)())) in loss gradients w.r.t. embeddings (i.e. di./ao).
clustering, which as we can see are consistent across different views. To understand how well the embeddings are clustered, we visualize heatmaps of distance of rendered embeddings from cluster- centroids. Specifically, we choose centroids, and for each centroid e; and each pixel u, we plot (u)) = log( c,ll) normalized to [0, 1]. where 0, is the rendered embedding.
Note that, instance labels only computed for pixels belonging to the "thing" semantic categories (as predicted by the semantic field)?. The "stuff" pixels are masked out.
As be seen in all these visualizations, the heatmaps locations and close to zero elsewhere which indicates the embeddings are compactly clustered around the corresponding centroid for each object. In Fig. 13, we can see that even in scene with 50 objects, the embeddings for each instance are distinctly separable.
peaked at the corresponding object
"The thing and stuff categories for Messy Rooms dataset simply "foreground" and "background".
18

---

Predicted labels
Centroid 1
Centroid
Centroid
Centroid
08
View
04
04
02
View
04
62
0.0
View
0.
Figure 8: ScanNet scene0300_00 01: Visualized instance segmentation and clustering heatmaps.
Predicted labels
Centroid
Centroid 2
Centroid
Centroid 4
View
02
0.
100
088
View 2
16
08
View
0.6
64
Figure 9: ScanNet scene0050_0m 02: Visualized instance segmentation and clustering heatmaps.
Predicted labels
Centroid
Centroid
Centroid
Centroid
1.
08
View
1.0
0a
View
066
64
View3
066
0A
62
00
Figure 10: ScanNet scene0422 02: Visualized instance segmentation and clustering heatmaps.
19

---

Predicted labels
Centroid
Centrold 2
Centroid
Centrold
View
00
14
08
View 2
08
08
View
04
64
02
Figure 11: Messy Rooms large_conie 2ang__corrido__55 Visualized instance segmentation and clustering heatmaps.
Predicted labels
Centroid
Centroid 2
Centroid
Centroid
View
18
08
View
066
64
View
TRRKE
0.6
6A
02
0.0
Figure 12: Messy Rooms old oldo_ooddooooooo reo_room_22___ 25: Visualized instance segmentation and clustering heatmaps.
Predicted labels
Centroid
Centroid 2
Centroid
Centroid 4
0.8
View
0.6
04
92
3.0
View2 2
02
0.
View 3
Figure 13: Messy Rooms old room 50: Visualized instance segmentation and clustering heatmaps.
20

---

RGB
GT labels
DETIC preds
RGB
GT labels
DETIC preds
(a) Sarge_corridors 25 objects
(b) old_room: 25 objects
Figure 14: Illustrative examples from Messy Rooms dataset. Here, we show scenes with 25 objects.
21

---

RGB
GT labels
DETIC preds
RGB
GT labels
DETIC preds
0?
(a) Large_corridor: 50 objects
(b) room: 50 objects
Figure 15: Illustrative examples from Messy Rooms dataset. Here, we show scenes with 50 objects.
22

---

RGB
GT labels
DETIC preds
RGB
GT labels
DETIC preds
(a) large_ sarge_corridor: 100 objects
(b) old_room 100 objects
Figure 16: Illustrative examples from Messy Rooms dataset. Here, we show scenes with 100 objects.
23

---

RGB
GT labels
DETIC preds
RGB
GT labels
DETIC preds
(a) Large_corridee 500 objects
(b) old_room 500 objects
Figure 17: Illustrative examples from Messy Rooms dataset. Here, we show scenes with 500 objects.
24

---

(a) Scenes with large_coms corridor environment. Top left: 25 objects. Top right: 50 objects. Bottom left: 100 objects. Bottom right: 500 objects.
(b) Scenes with old room environment. Top left: 25 objects. Top right: 50 objects. Bottom left: 100 objects. Bottom right: 500 objects.
Figure 18: We show (using Blender [12]) the actual 3D scenes without texture that used to render/generate the Messy Rooms scenes. Note that the surface area of the scene is increased proportionally to the number of objects.
25


Reviews:
[{"review_id": "Zh5CFI12QH", "reviewer": "Reviewer_zMUp", "summary": "The paper studies the 3D object instance segmentation inside a 3D NeRF space. Specifically, to train the model, conservative loss between features generated by slow and fast NeRF models are computed to 1) maximize the feature distance between different semantic regions, 2) minimize the feature distance within the same semantic regions. Also, a dataset is introduced namely Messy Room, which consists of rendering real captured objects from Google Scanned Objects. Experiments on the dataset show a reasonable improvement in comparison to baselines.", "questions": "1) How the ground plane is determined and set in the messy Room dataset?\n\n2) The author mentions in line 148 that \"gradients with high variance\". What is a high variance on gradients? The author should quantitatively study how the variance is controlled using the proposed SF loss in comparison to vanilla contrastive loss.\n\n3) A clustering operator is conducted on the feature vectors for segmentation, is this similar to this work? [2]\n\n4) In the visualization, how does the trained feature vectors convert into semantic labels?\n\n5) Some typos: in figure 2 L_cntr but in equation 5 L_conc.\n\n\n[2] Yu, Q., Wang, H., Qiao, S., Collins, M., Zhu, Y., Adam, H., ... & Chen, L. C. (2022, October). k-means Mask Transformer. In European Conference on Computer Vision (pp. 288-307). Cham: Springer Nature Switzerland.\n", "limitations": "The authors include and discuss limitations of this work. There seems no potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed contrastive learning for 3D semantic segmentation on NeRF is elegant and novel. I would believe such a structure is considered to be better in comparison to previous works that directly output the semantic labels from the NeRF network. Also, the introduced messy rooms dataset is believed to be useful for the community, despite the dataset is partially synthetic. The overall performance improvements from the baseline of the proposed method are not huge but still reasonable.", "weaknesses": "1) The proposed method is only evaluated on the half-synthetic dataset with small objects on the table. It is necessary to evaluate the proposed method on some real images, at least qualitatively.\n\n2) The performance of the segmentations before lifting is not reported in the experiment section, it is unclear how the proposed method improves from the 2D segmentation.\n\n3) The contrastive training pipeline is partially similar to this work [1], it would be better to include it in discussion.\n\n[1] Bai, Y., Wang, A., Kortylewski, A., & Yuille, A. (2020). Coke: Localized contrastive learning for robust keypoint detection. arXiv preprint arXiv:2009.14115.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and provide feedback!\n\n---\n\n**Response to Weakness 1**: We do evaluate on real scenes from ScanNet [8] dataset (please refer L211-212 and Table 1). We also evaluate using Replica [41], which comprises real 3D scans, and Hypersim [36], which is synthetic but designed to match the complexity of real scenes. In addition to quantitative results in Tables 1,2 and 3, Figures 1,2,3 in the supplementary show visualizations on ScanNet scenes. We will add more visualizations from these datasets in the final version.\n\n---\n\n**Response to Weakness 2**: The performance before lifting is not shown because these instance segmentations (obtained from a 2D segmenter) are *not consistent (aka tracked) across frames/views*. Therefore, the $PQ^{scene}$ metric for these 2D methods is very low, because the metric is designed to reflect the cross-frame consistency of the predictions.\n\n**Following the reviewer’s suggestion, we evaluate performance of 2D segmentations (i.e., before lifting) in 2 ways:**\n1. Table 1 (see below) reports the _frame-level_ PQ (Panoptic Quality) score for both the 2D segmenter and our method. Note that PQ only evaluates segmentation quality frame-by-frame, regardless of inter-frame consistency. The table shows that our method significantly improves the frame-level 2D segmentation quality.\n2. Table 2 (see below) reports the PQ$^{\\text{scene}}$ metric, which considers the consistency of instance segmentations across frames. To achieve consistency, we post-process the 2D segmenter's predictions using Hungarian Matching for cross-frame tracking as follows: \n    * **w/ Hungarian matching (2D IoU):** Given sets of predicted segments ($P_i$ and $P_{i+1}$) from consecutive frames, compute IoU matrix by comparing all segment pairs in $P_i\\times P_{i+1}$. Apply Hungarian matching to the IoU matrix to associate instance segments across frames.\n    * **w/ Hungarian matching based on IoU after depth-aware pose-warping**: Use *ground-truth* pose and depth for warping $(i+1)$-th frame's segmentation to frame $i$. Compute IoU matrix using warped segmentations and apply Hungarian matching.\n    * **w/ Hungarian matching using the 3D ground-truth pointcloud**: Using only consecutive frames leads to errors in long-range tracking. To address this, starting from 1st frame, un-project 2D segmentations into a 3D point cloud. Iteratively fuse these segments in 3D using Hungarian matching. This way, segments from *all preceding frames* along with 3D information are used for tracking.\n\nWe note that the last two baselines use the _3D ground-truth_ for matching. The table below shows that, even when 3D information is used, our method still significantly outperforms the baseline 2D segmentation approach.\n\n**Table 1**:\n| Method                         | PQ (on ScanNet [8]) |\n| ------------------------------ | ------------------: |\n| Mask2Former [7] (2D segmenter) | 43.6                |\n| Contrastive Lift (**ours** trained w/ Mask2Former labels) | **62.1**|\n\n**Table 2**:\n| Method | PQ$^{\\text{scene}}$ (on ScanNet) |\n| -------- | --------: |\n| Mask2Former (*M2F*) (non-tracked 2D segmentations)     | 32.3     |\n| M2F + Hungarian matching (2D IoU)     | 33.7     |\n| M2F + Hungarian matching based on IoU after ***depth-aware pose-warping***     | 34.0     |\n| M2F + Hungarian matching using the ***3D ground-truth pointcloud***     | 41.0     |\n| Contrastive Lift (**ours** trained w/ Mask2Former labels)    | **62.3**     |\n\n---\n\n**Response to Weakness 3**: Thank you for bringing this method to our attention. CoKe is an interesting method that uses contrastive learning along with moving average updates to learn keypoint prototypes. We will include CoKe in the discussion of related methods.\n\n---\n\n**Response to Question 1:** The Messy Rooms dataset is generated using the Kubric [12] simulator. We configure the \"gravity\" vector along the \"-z\" axis and designate the xy-plane as the ground-plane. For details, see the code in `dataset/kubric_panopli_generator_final.py`.\n\n---\n\n**Response to Question 2:** In L148, we refer to the variance in loss gradients w.r.t. the embedding field ($\\Theta$), i.e. variance of $\\nabla_{\\Theta} L$. Empirically, we compute the relative variance ($\\frac{Var(\\cdot)}{Mean(\\cdot)}$), finding significantly higher values (across training iterations) with the vanilla loss as compared to the slow-fast version. Please see Figure 1 in the attached PDF. The vanilla loss shows spikes with a peak relative variance near $10^7$, while the slow-fast variant maintains around $10^1$.\n\n---\n\n**Response to Question 3:** *kMaX-DeepLab* learns a pixel-cluster assignment by reformulating cross-attention from a clustering perspective. So, it is indeed similar in spirit to our approach, although it's worth noting that the embeddings (and cluster centers) in our method (defined as $\\mathbb{R}^3 \\rightarrow \\mathbb{R}^D$) are learnt using *differentiable volumetric rendering* from 2D labels. We will include a discussion of *kMaX-DeepLab* (along with “CMT-Deeplab: Clustering Mask Transformers for Panoptic Segmentation”, Yu et al. and “Semi-convolutional Operators for Instance Segmentation”, Novotny et al.) in the revised paper.\n\n---\n\n**Response to Question 4:** For the ***semantic*** labels, since the features were trained with a cross-entropy loss (similar to [38,57]), we obtain the label as the `argmax` of rendered logits.\nFor the ***instance*** labels, as described in lines 177-181 of main paper and in section 2.3 of supplementary material:\n1.  the rendered instance features are clustered with HDBSCAN [31], an unsupervised density-based clustering algorithm, to obtain cluster centroids.\n2. for pixels in any novel view, the label of the centroid nearest to the rendered embedding is assigned as the instance label.\n\n---\n\n**Response to Question 5:** Thank you for pointing this out. It should be $L_{conc}$ in figure 2. We will fix this in the revised paper."}, {"review_id": "tKLZ4mfHVI", "reviewer": "Reviewer_KUUX", "summary": "The proposed method tries to solve the problem of reconstructing a 3D scene together with the underlying instance segmentation. Prior work required either GT tracking data or concurrent work a less efficient way to assign instances. From a set of images a Neural Radiance Field is reconstructed together with a feature field, that represents an embedding of the instance. Instance embeddings are guided by a contrastive loss function, that pushes embeddings that are projected into pixels from the same segment in a 2D segmentation mask closer and embeddings projecting into different masks apart. To improve the stability of the training, the authors propose an additional loss with a jointly trained slowly-updated instance embedding field updated with a moving average over the parameters of the faster field instead of SGD.\nInstances are later computed by clustering embeddings, which is supported by the third loss term, which uses an average embedding from the slowly updated field to penalize the difference for the fast-field predictions. Specific values in the embedding vector are assigned to semantic classes for semantic segmentation and are directly supervised with the 2D semantic maps.  \nAdditionally, the authors propose a novel dataset with up to 500 objects for evaluating future 3D instance segmentation ", "questions": "- It seems like a fixed number of clusters is given or how is the number of clusters decided?", "limitations": "The authors address most of the limitations I can think of and ablate the use of different methods they rely on, such as the clustering method in the supplement.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "So far 3D instance segmentation methods, such as Neural Panoptic Fields (referenced by the authors as well) required a tracking algorithm or GT tracking data to reconstruct instance labels of the 3D scene and this method presents a light, optimization-based approach, that directly learns an instance embedding from semantic maps through alignment. \n\nThe authors describe their method in a way that is understandable and design choices, especially for the loss function and the learning paradigm are reasonable and justified by ablation and additional experiments.\n\nIn general, this is a well-designed method that leverages the current state of the art and adds interesting new components to allow a jointly learning of the radiance field, instance and semantic embedding.\n\nA big plus of the presented method is the additional dataset with up to 500 objects and the submission of the code, that allows the reproducibility of the results.\n", "weaknesses": "While the presented evaluation of the proposed dataset shows a clear advantage over state-of-the-art methods on existing and their own synthetic dataset, as well as ScanNet, methods like Panoptic Fields have also shown results on complex outdoor scenes, such as the KITTI dataset. Therefore an evaluation in such a complex outdoor setting would further strengthen the paper and usability in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to study our work and provide thoughtful feedback!\n\n**Response to Weaknesses:** Thank you for suggesting the idea to assess the performance of our approach on outdoor scenes, e.g. *KITTI* or *KITTI-360* datasets. We agree that it would strengthen the paper. Unfortunately, we could not produce the results on the KITTI dataset in the given limited time for rebuttal, but we will include results on these outdoor scenes in the final version of the paper. As you have already pointed out, we show in Table 1 that our method outperforms Panoptic Neural Fields on indoor scenes.\n\n**Response to Questions:** We ***do not*** assume knowledge of the number of clusters. Instead, we use HDBSCAN (Hierarchical DBSCAN) [31] which is a density-based clustering algorithm. HDBSCAN does not require the number of clusters to be known. It computes a “mutual reachability distance” matrix between all pairs of datapoints and then uses a single linkage algorithm to find clusters in the data. The only hyperparameter in HDBSCAN is the minimum cluster size, which can be set to any reasonable value (e.g. $10^3$ if we have $10^5$ datapoints) or can be grid-searched on a fraction of the training dataset."}, {"review_id": "ZyR7vZ6tye", "reviewer": "Reviewer_4dod", "summary": "This paper utilizes contrastive learning for lifting 2D segments to 3D and fuses the learned embedding by means of a neural field representation, namely Contrastive Lift. The authors further propose a slow-fast clustering objective function, which makes the method scalable for scenes with a large number of objects. To further validate the ability of the method, this paper also introduces a new dataset, Messi Rooms, which includes up to 500 objects as a benchmark for instance segmentation with a large number of objects. The experiments show that the proposed approach outperforms former SOTA on ScanNet, Hypersim, Replica, and Messy Rooms.", "questions": "- The random partition of pixels into two non-overlapping sets needs further explanation. (Randomly choose pixels? Randomly divide images with pre-defined boundaries? What is the proportion between two sets? etc.)", "limitations": "The authors have addressed some limitations in Sec. 6. There is no negative social impact in this work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach employs a low-dimensional Euclidean embedding to represent a 3D instance. The dimensionality D is far less than the number of objects L, making the approach more efficient and easily extended to larger numbers of objects. Most importantly, using the 3D instance embedding implicitly ensures multi-view consistency. It avoids the assignment problem that exists in Panoptic Lifting [43].\n\n- Using contrastive learning and the clustering strategy makes the proposed approach independent of the number of objects, which is more suitable for scenes with a large number of objects.\n\n- The slow-fast contrastive learning is scalable for different object numbers and stabilizes the training phase. And the proposed concentration loss ensures the concentration of embeddings within the same cluster. It improves the more complete instance segmentation results.\n\n- The proposed semi-realistic dataset, Messy Rooms, provides a novel benchmark for testing the performance on scenes with large object numbers. \n", "weaknesses": "- In Tab. 1 and Tab.2, the metric used for evaluation only uses PQ^scene^, which is cherry-picked. In Panoptic Lifting [43], mIoU and PSNR are also used for evaluation. A comprehensive comparison according to different metrics should be included.\n\n- For semantic segmentation, Contrastive Lift should append a new branch to predict the semantic labels and be supervised by the segment consistency loss in [43]. I suppose that the model should be trained specifically for semantic segmentation. Panoptic Lifting can predict semantic and instance labels simultaneously. I am curious about whether the additional supervision on semantic labels would influence the instance segmentation results. Please explain it. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing feedback and taking the time to review our work!\n\n---\n\n**Response to Weakness 1**: We report the mIoU and PSNR of our method (and also for Panoptic Lifting and SemanticNeRF) in Table 2 of the **supplementary material**. In Tables 1 and 2 of the main paper, we only report PQ$^\\text{scene}$ for brevity, as it is the most relevant metric for assessing instance segmentation quality, which is the main contribution of our approach. \n\nSince the semantic field and density/color field architecture of our method is identical to Panoptic Lifting [38], we expect to achieve similar mIoU (semantic segmentation performance) and PSNR (view synthesis performance) as [38]. This can be verified from Table 2 of supplementary material.\n\n---\n\n**Response to Weakness 2**: We would like to clarify the following points about our architecture and loss functions:\n1. As described in lines 168-176, our architecture has separate branches for density, color, semantic and instance prediction. Our architecture is the same as the one used in Panoptic Lifting, which also has these 4 separate branches. We purposely use the same underlying architecture so that we can fairly compare Panoptic Lifting’s matching-based instance loss with our proposed “slow-fast” loss formulation with instance embeddings (*while keeping all other things equal*).\n2. We do have supervision on semantic labels, including the segment consistency loss (line 166 in the main paper). We will make this part more clear in the revised paper.\n3. Note that the gradients from the semantic loss(es) do NOT propagate to the instance branch (and vice versa), as these are parallel/independent branches. That being said, supervising semantic labels does not affect instance segmentation performance but it affects panoptic segmentation which considers both semantic and instance predictions.\n\n---\n\n**Response to Questions**: These pixels are chosen purely randomly from $\\Omega$. Both of these non-overlapping sets are of equal size. To be more precise: say we want to sample two non-overlapping sets, each of size $N$. We first sample a batch of $2N$ pixels from $\\Omega$ and then simply split it into two sets of size $N$."}, {"review_id": "zgiEFpF2cb", "reviewer": "Reviewer_Vjqu", "summary": "This paper introduces a novel Contrastive Lift method for 2D segment lifting to 3D reconstruction and instance segmentation. The authors fuse multiview representations obtained from pre-trained 2D models into a unified 3D neural field. They propose a scalable slow-fast clustering objective function that enables segmenting without an upper bound on the number of objects. Additionally, a new semi-realistic dataset is created to evaluate the proposed method, which demonstrates superior performance compared to previous state-of-the-art approaches on both public datasets and the newly introduced dataset.", "questions": "Please refere to Weaknesses.", "limitations": "Please refere to Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research motivation and key challenges are generally well illustrated and summarized. \n2. The newly constructed framework and dataset fits well within the 3D vision field, particularly the flexible and scalable design for lifting 2D segments to 3D.\n3. The proposed method is technically sound and achieves SOTA performance on the newly proposed dataset and challenging scenes on public datasets.", "weaknesses": "1. The paper lacks a discussion on the complexity of the proposed method and its potential impact.\n2. Ablations on different 2D segmenters are not included in the paper.\n3. The analysis of the ablations and their corresponding results is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback!\n\n ---\n\n**Response to Weakness 1**: Lines 278-288 of the main paper provide an empirical analysis of the time complexity and training speed of our method and compares it to the linear-assignment matching-based method. In summary, the complexity of our proposed method is agnostic to the number of objects in the scene, while the compared matching-based method has a $O(K^3)$ complexity where $K$ is a hyperparameter that specifies the maximum number of objects.\n\n---\n\n**Response to Weakness 2**: Thank you for the suggestion. Based on the reviewer's suggestion, we have conducted experiments with two more 2D instance segmentation models (*MaskFormer [Cheng et al., \"Per-Pixel Classification is Not All You Need for Semantic Segmentation\"] and Detic* [59]) in addition to *Mask2Former* [7].\n\nThe Table below demonstrates the Panoptic Quality (PQ) metric achieved on the ScanNet dataset [8] by each of these 2D models and by our method when trained with predictions from these 2D segmentation models. In all cases, we can see that our method improves the PQ metric significantly.\n\n| Method | PQ | \n| -------- | -------- |\n| MaskFormer     |  41.1    |\n| Contrastive Lift (trained with *MaskFormer* labels)     |  61.7    |\n| Detic [59]     |   42.0   |\n| Contrastive Lift (trained with *Detic* labels)     |   61.6   |\n| Mask2Former [7]     |  43.6    |\n| Contrastive Lift (trained with *Mask2Former* labels)     |  **62.1** |\n\n---\n\n**Response to Weakness 3**: Thank you for the suggestion. We will include the above ablation regarding the effect of different 2D segmenter models in the revised paper. We shall also expand the discussion in the ablation studies included in Section 5.2 of the main paper as well as Sections 3 and 5 of the supplementary material."}, {"review_id": "7t07bfKbxJ", "reviewer": "Reviewer_RRt9", "summary": "This paper aims to solve the task of 3D instance segmentation by leveraging pre-trained 2D instance segmentation models. The authors propose a novel approach to lift 2D segments to 3D via a neural field. This idea is not completely new [38, 51], but the authors propose a contrastive loss that replaces the Hungarian-algorithm-based loss used in [38]. Moreover, the authors propose a synthetic dataset where the number of objects can be controlled, and show that the Hungarian-algorithm-based loss slows down substantially as the number of objects increase. Finally, the authors augment the contrastive loss with a momentum-teacher component (similar to [16]).\n\n_[16] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman,\nJoão Carreira, and Relja Arandjelović. Object discovery and representation networks. In Computer\nVision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part\nXXVII, pages 123–143. Springer, 2022._\n\n_[38] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. arXiv.cs, abs/2212.09802,\n2022._\n\n_[51] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot,\nAndrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic\nsegmentation of 3D scenes. arXiv.cs, abs/2111.13260, 2021_", "questions": "Q1) A prior work [1002] might be relevant to the 3D semantic segmentation background. The references tackles that problem and predates the references [23, 29, 51, 57]. \n\nQ2) How does the proposed approach compare to [38]? See W3.\n\nQ3) What does the statement on l290 mean? How does the proposed approach improve noisy 2D semgmentations?\n\nQ4) On l291, what does it mean for 3D reconstruction to work properly?\n\n_[1002] Lawin, Felix Järemo, et al. \"Deep projective 3D semantic segmentation.\" Computer Analysis of Images and Patterns: 17th International Conference, CAIP 2017, Ystad, Sweden, August 22-24, 2017, Proceedings, Part I 17. Springer International Publishing, 2017._", "limitations": "The authors provide a discussion that clarifies some important aspects, for instance that the proposed approach supports only static scenes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1) The proposed scheme to deal with instance label ambiguity is novel.\n\nS2) The paper is mostly well-written.\n\nS3) The authors show that the proposed contrastive momentum-teacher loss gives good performance and is computationally lighter than the Hungarian-algorithm-based loss.\n\nS4) The problem of adapting 2D instance segmentation methods to 3D is useful to real-world applications.", "weaknesses": "W1) The notation has some issues, is confusing in some places, and deviates a bit from prior work. \n- It is not common to let $y$ represent a mapping. It is common to say that $f$ is a mapping such that $y = f(x)$. Same goes for $Y$. The label $y(u)$ is sometimes referred to as a label and sometimes as a function, which is a bit confusing.\n- I think $u$ should be the location of a pixel, rather than the actual pixel (which would be in $\\mathbb{R}^3$).\n- $Y(x)$ is not presented as a function of $x$, but as a function of $u$.\n- $\\Theta$ is introduced as both a mapping $\\Theta: \\mathbb{R}^2\\rightarrow \\mathbb{R}^D$ and $\\Theta: \\mathbb{R}^3\\rightarrow \\mathbb{R}^D$.\n- Also $\\rho$, $c$, and $\\Theta$ are introduced as both mappings and actual elements of the codomain of the mapping.\n- Usually, $x$ and $d$ are used for 3D location and ray direction. While it is not a crime to change notation, I cannot see a reason for replacing $d$ with $v$.\n- Equation (1) does not link to the input to the nerf ($x$ and $v$). The link can only be found in the text in l128-l129. It would be easier to understand if $R$ was actually described as a function of $u$.\n\nW2) The name Slow-Fast is confusing.\n- The loss component name slow-fast could have been named _student-teacher_ or something with _exponential moving average_. Currently, the name makes it easy to confuse with Slow-Fast [1001].\n\nW3) The prior work [38] also adapts 2D instance segmentation models to 3D instance segmentation using NeRFs. The proposed approach replaces the matching-based loss and the \"slow-fast\"-component seems necessary for good performance. Is the comparison to [38] completely fair? Except for the proposed changes, are all other things equal, e.g., the underlying instance 2D segmentation approach, NeRF architecture, or training scheme? I find this difficult to tell from the paper text. It is clear that the proposed approach yields good performance and trains faster than [38], but it is not clear whether this is purely do to the proposed changes.\n\nMinor remarks\n- \"much smaller\" on l62 should probably be removed.\n\n\n_[1001] Feichtenhofer, Christoph, et al. \"Slowfast networks for video recognition.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019._", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments which have helped us improve our paper!\n\n---\n\n**Response to Weakness 1**: We are thankful for the reviewer's detailed feedback on the notation and presentation. \n* We will clarify that $u$ is the pixel location.\n* $\\Theta$ should be $\\Theta:\\mathbb{R}^3\\rightarrow\\mathbb{R}^D$. And $\\theta$, which is defined as $\\mathcal{R}(u|\\Theta,\\rho,\\pi)$ in Equation (3), should be $\\theta:\\mathbb{R}^2\\rightarrow\\mathbb{R}^D$. We will correct this typo.\n* We acknowledge the concern regarding the dual usage of {$\\rho, c, \\Theta$} as both mappings and codomain elements. Our intention was to maintain notation for ease of understanding. We'll ensure to clarify the context whenever dual usage occurs, aiming to minimize any confusion.\n* Thank you for the suggestion. We will change the notation to use $d$ for viewing direction, as in common in the literature.\n* We will ensure that the relationships between equations, especially Equation (1), and their corresponding inputs are clearly stated and explained.\n\n---\n\n**Response to Weakness 2**: We acknowledge that the term \"slow-fast\" is a bit overloaded (e.g. *Feichtenhofer, Christoph, et al. \"Slowfast networks for video recognition.\"*). However, as described in L149-154, we use a *slowly-updated* field to compute the centroid embeddings and encourage compact clustering in the *fast* field, which is a key element of our approach. We hope that this explanation in the paper can justify the usage of the term \"slow-fast contrastive fusion\" in our method.\n\n---\n\n**Response to Weakness 3**: We have made every effort to ensure that the comparison to Panoptic Lifting [38] is completely fair. This can be noted as follows:\n1. We purposely use the same underlying architecture as [38] so that we can fairly compare Panoptic Lifting’s matching-based instance loss with our proposed “slow-fast” formulation (*while keeping all other things equal*). \n2. Apart from the instance segmentation related losses (namely, matching-based loss for [38] and our slow-fast loss), all the other losses and training scheme are identical to [38].\n3. We use the same underlying 2D panoptic (semantic+instance) segmenter for all the methods (which includes Panoptic Lifting [38] and our method). For example, we use Mask2Former [7] for experiments with ScanNet, Replica and Hypersim datasets. And use Detic [59] for experiments with our Messy Rooms dataset.\n\nWe will highlight these aspects in the revised paper so that they are clear to the reader.\n\n---\n\n**Response to minor remark**: Thank you. We will remove the phrase “much smaller” and only state that D<<L, which is more clear.\n\n---\n\n**Response to Question 1**: Thank you for bringing this method to our attention. It is definitely an influential work in the direction 2D-to-3D semantic fusion to achieve 3D segmentation. We will include this in the discussion of related works.\n\n**Response to Question 2**: Please refer to our \"**response to weakness 3**\" above which clarifies how we ensure a fair comparison with Panoptic Lifting [38].\n\n---\n\n**Response to Question 3**: The proposed method fuses 2D segments in 3D space, promoting multi-view consistency across frames. By fusing information from multiple views, the approach is able to overcome the limitations of individual noisy 2D segmentations and produce more accurate and robust instance segmentation *in the 3D space*. As a result, when rendering 2D views from the 3D neural field model, we observe an improvement in quality compared to the original 2D segmentations that were used to train the model in the first place. To clarify this further, we quantify this improvement in the Table below. We report the PQ metric achieved on ScanNet [8] by three different 2D segmentation models: MaskFormer [Cheng et al., “Per-Pixel Classification is Not All You Need for Semantic Segmentation”], Detic [59] and Mask2Former [7]. We compare this to the PQ score achieved by our method when trained with each of these 2D models' predictions. We can see that our method improves the Panoptic Quality (PQ) significantly in each case.\n\n| Method | PQ | \n| -------- | -------- |\n| MaskFormer     |  41.1    |\n| Contrastive Lift (trained with *MaskFormer* labels)     |  61.7    |\n| Detic [59]     |   42.0   |\n| Contrastive Lift (trained with *Detic* labels)     |   61.6   |\n| Mask2Former [7]     |  43.6    |\n| Contrastive Lift (trained with *Mask2Former* labels)     |  **62.1** |\n\n---\n\n**Response to Question 4**: For 3D reconstruction to work properly, it means that the _learned density field_ accurately represents the geometry of the scene. The proposed method fuses the 2D segmentations from multiple views into a 3D neural field, and this fused field is obtained via differentiable volumetric rendering using the density field (denoted as $\\rho$ in Section 3). Hence, to ensure multi-view consistency during the fusion, accurate geometry or density field reconstruction is crucial."}]

Author rebuttal:
We thank all the reviewers for their time and valuable feedback. We are pleased to see positive responses from all reviewers, who acknowledge the novelty [*zMUp*,*RRt9*], design [*KUUX*,*4dod*,*Vjqu*], efficiency [*4dod*,*RRt9*] and performance [*Vjqu*,*RRt9*] of our approach as well as the usefulness of the proposed dataset [*zMUp*, *KUUX*, *4dod*]. 

We have answered each reviewer's concerns/questions **separately in the individual responses below**. Additionally, we have attached a PDF that contains one figure which addresses a question from *Reviewer zMUp*.

Meta-review: This paper proposes an approach for 3D instance segmentation in Nerfs by lifting multiple 2D views of scenes with available instance segmentation labels. It proposed a novel slow-fast contrastive learning approach to solve the instance association problem, which greatly simplifies the the multi-view assignment problem for different instances across views and also makes it easier to scale up to a large number of object instances. Additionally, a new Messy-Rooms dataset is proposed with many object instances to evaluate the proposed approach.

The paper received the scores (4A, 1BA). All reviewers appreciated the work for its novel and elegant method, clear improvements in quality over the existing state of the art and for appreciated the proposed new dataset as being a valuable contribution to the research community. Most reviewers' concerns were also successfully addressed by the authors' responses. 

The AC recognizes the paper's significant contribution to 3D neural instance segmentation and recommends acceptance. Congratulations!

The authors are strongly encouraged to incorporate the changes that they've promised in the rebuttal to the final manuscript.
Decision: Accept (spotlight)