Title: Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Radiance Fields

Instance Segmentation

Metric Learning

Clustering

3D Computer Vision
Abstract: Instance segmentation in 3D is a challenging task due to the lack of large-scale annotated datasets. In this paper, we show that this task can be addressed effectively by leveraging instead 2D pre-trained models for instance segmentation. We propose a novel approach to lift 2D segments to 3D and fuse them by means of a neural field representation, which encourages multi-view consistency across frames. The core of our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with a large number of objects. Unlike previous approaches, our method does not require an upper bound on the number of objects or object tracking across frames. To demonstrate the scalability of the slow-fast clustering, we create a new semi-realistic dataset called the Messy Rooms dataset, which features scenes with up to 500 objects per scene. Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets, as well as on our newly created Messy Rooms dataset, demonstrating the effectiveness and scalability of our slow-fast clustering method.

Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion

Abstract

Instance segmentation in 3D is a challenging task due to the lack of large-scale annotated datasets. In this paper, we show that this task can be addressed effectively by leveraging instead 2D pre-trained models for instance segmentation. We propose a novel approach to lift 2D segments to 3D and fuse them by means of a neural field representation, which encourages multi-view consistency across frames. The core of our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with a large number of objects. Unlike previous approaches, our method does not require an upper bound on the number of objects or object tracking across frames. To demonstrate the scalability of the slow-fast clustering, we create a new semi-realistic dataset called the Messy Rooms dataset, which features scenes with up to 500 objects per scene. Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets, as well as on our newly created Messy Rooms dataset, demonstrating the effectiveness and scalability of our slow-fast clustering method.

1 Introduction

While the content of images is three-dimensional, image understanding has largely developed by treating images as two-dimensional patterns. This was primarily due to the lack of effective machine learning tools that could model content in 3D. However, recent advancements in neural field methods [5, 40, 42, 49, 67] have provided an effective approach for applying deep learning to 3D signals. These breakthroughs enable us to revisit image understanding tasks in 3D, accounting for factors such as multi-view consistency and occlusions.

In this paper, we study the problem of object instance segmentation in 3D. Our goal is to extend 2D instance segmentation to the third dimension, enabling simultaneous 3D reconstruction and 3D instance segmentation. Our approach is to extract information from multiple views of a scene independently with a pre-trained 2D instance segmentation model and fuse it into a single 3D neural field. Our main motivation is that, while acquiring densely labelled 3D datasets is challenging, annotations and pre-trained predictors for 2D data are widely available. Recent approaches have also capitalized on this idea, demonstrating their potential for 2D-to-3D semantic segmentation [31, 37, 60, 67] and distilling general-purpose 2D features in 3D space [29, 55]. When distilling semantic labels or features, the information to be fused is inherently consistent across multiple views: semantic labels are viewpoint invariant, and 2D features across views are typically learned with the same loss function. Additionally, the number of labels or feature dimensions is predetermined. Thus, 3D fusion amounts to multi-view aggregation.

When it comes to instance segmentation, however, the number of objects in a 3D scene is not fixed or known, and can indeed be quite large compared to the number of semantic classes. More importantly, when objects are detected independently in different views, they are assigned different

Images

BeforeLifting

AfterLifting

and inconsistent identifiers, which cannot be aggregated directly. The challenge is thus how to fuse information that is not presented in a viewpoint-consistent manner.

Recently, Panoptic Lifting [48] proposed to resolve the lack of multi-view consistency by explicitly fitting a permutation that aligns labels extracted from multiple views. Although this yields good results, there are two drawbacks to this approach. Firstly, determining the permutation matrix involves solving a linear assignment problem using Hungarian Matching for every gradient computation. The cost of this increases cubically with the number of identifiers, which may limit scalability when dealing with a large number of object instances. Secondly, the canonical label space, where the permutation maps each 2D label, may need to be extensive to accommodate a large number of objects.

In this study, we propose a more efficient formulation, which also leads to more accurate results. To understand our approach, consider first a 2D image segmenter: it takes an image I as input and produces a mapping y that assigns each pixel u ∈ R 2 to an object instance label y ( u ) ∈ { 1 , . . . , L } . It is natural to extend this mapping to 3D by introducing a function Y that associates each 3D point x ∈ R 3 with the label Y ( x ) of the corresponding object. To account for the fact that labels are arbitrary and thus inconsistent between views, Panoptic Lifting [48] seeks an image-dependent permutation matrix P such that Y ( x ) = P · y ( u ) , where u is the projection of x onto the image.

To address the aforementioned challenges with the linear-assignment-based approach, we identify the labels y ( u ) with coordinate vectors in the Euclidean space R L . The functions y ( u ) can be reconstructed, up to a label permutation, from the distances d ( y ( u ) , y ( u ′ ))= ∥ y ( u ) -y ( u ′ ) ∥ 2 of such vectors, as they tell whether labels of two pixels ( u, u ′ ) are the same or different, without considering the specific labelling. Notably, similar to compressed sensing, we can seek lower-dimensional projections of the vectors y that preserve this information. With this in mind, we replace the 3D labelling function Y with a low-dimensional Euclidean embedding Θ( x ) ∈ R D . Then, we supervise the embeddings such that their distances d (Θ( x ) , Θ( x ′ )) ≈ d ( y ( u ) , y ( u ′ )) are sufficiently similar to that of corresponding 2D label embeddings.

This approach has two advantages. First, it only requires learning vectors of dimensionality D ≪ L which is independent of the number of objects L . Second, learning this function does not require solving an assignment problem; rather, it only considers pairwise distances. Hence, the complexity of computing the learning objective is independent of the number of objects in the scene.

We translate this idea into a neural fusion field framework, which we call Contrastive Lift . We build on the recent progress in self-supervised learning, and combine two key ideas: the usage of a

contrastive loss, and the usage of a slow-fast learning scheme for minimizing the latter in a stable manner. We believe to be the first to introduce these two ideas in the context of neural fields.

We compare our method to recent techniques including Panoptic Lifting [48] on standard 3D instance segmentation benchmarks, viz. ScanNet [13], Replica [50], and Hypersim [46]. To better demonstrate the scalability of our method to a very large number of object instances, we introduce a semi-realistic Messy Rooms dataset featuring scenes with up to 500 objects.

2 Related Work

Neural Radiance Fields (NeRFs). NeRF [40] and its numerous variants [2, 5, 34, 36, 42] have achieved breakthrough results in generating photorealistic 3D reconstructions from 2D images of a scene. These systems typically represent the scene as a continuous volumetric function that can be evaluated at any 3D point, enabling high-quality rendering of novel views from any viewpoint.

Objects and Semantics in NeRF. While NeRF by default offers low-level modelling of radiance and geometry, recent methods have expanded the set of tasks that can be addressed in this context to include semantic 3D modelling and scene decomposition. Some works use neural scene representations to decompose scenes into foreground and background without supervision or from weak signals [17, 41, 47, 56, 62, 63], such as text or object motion. Others exploit readily available annotations for 2D datasets to further extend the capabilities of NeRF models. For example, Semantic NeRF [67] proposes to incorporate a separate branch predicting semantic labels, while NeSF [60] predicts a semantic field by feeding a density field as input to a 3D semantic segmentation model.

Closer to our work are methods that employ NeRFs to address the problem of 3D panoptic segmentation [19, 26, 31, 48, 61]. Panoptic NeRF [19] and Instance-NeRF [26] make use of 3D instance supervision. In Panoptic Neural Fields [31], each instance is represented with its own MLP but dynamic object tracking is required prior to training the neural field. In this work, we focus on the problem of lifting 2D instance segmentation to 3D without requiring any 3D masks or object tracks. Apaper most related to our work is Panoptic Lifting [48], which also seeks to solve the same problem, using linear assignment to make multi-view annotations consistent. Here, we propose a more efficient and effective technique based on learning permutation-invariant embedding vectors instead.

Fusion with NeRF. The aforementioned works, such as Semantic NeRF [67] or Panoptic Lifting [48], are also representative of a recent research direction that seeks to fuse the output of 2D analysis into 3D space. This is not a new idea; multi-view semantic fusion methods [25, 35, 37, 38, 51, 59] predate and extend beyond NeRF. The main idea is that multiple 2D semantic observations ( e.g. , noisy or partial) can be combined in 3D space and re-rendered to obtain clean and multi-view consistent labels. Instead of assuming a 3D model, others reconstruct a semantic map incrementally using SLAM [32, 43, 53]. Neural fields have greatly improved the potential of this idea. Instead of 2D labels, recent works, such as FFD [29], N3F [55], and LERF [27], apply the 3D fusion idea directly to supervised and unsupervised dense features; in this manner, unsupervised semantics can be transferred to 3D space, with benefits such as zero-shot 3D segmentation.

Slow-fast contrastive learning. Many self-supervised learning methods are based on the idea of learning representations that distinguish different samples, but are similar for different augmentations of the same sample. Some techniques build on InfoNCE [54, 57] and, like MoCo [23] and SimCLR [7], use a contrastive objective. Others such as SWaV [3] and DINO [4] are based on online pseudo-labelling. Many of these methods stabilise training by using mean-teachers [52], also called momentum encoders [23]. The idea is to have two versions of the same network: a fast 'student' network supervised by pseudo-labels generated from a slow 'teacher' network, which is in turn updated as the moving average of the student model. Our formulation is inspired by this idea and extends it to learning neural fields.

Clustering operators for segmentation. Some works [14, 18, 30, 45] have explored using clustering of pixel-level embeddings to obtain instance segment assignments. Recent works [64, 65] learn a pixel-cluster assignment by reformulating cross-attention from a clustering perspective. Our proposed method, Contrastive Lift, is similar in spirit, although we learn the embeddings (and cluster centers) using volumetric rendering from 2D labels.

Camera

Slow NeRF

෩ Θ

Pixel color

ℛ(𝑢|𝑐,𝜌)

Real pixel

color

I

(

u

)

Scene volume

Fast NeRF

Θ

EMA

update

Sampled

points

r

i

Photometric

loss

L

RGB

Contrastive

loss

L

sf

Concentration

loss

L

conc

Renderer

Renderer

Point embeddings

෩

Θ 𝑟 𝑖

, Θ

𝑟 𝑖

Renderer

෨

𝜃𝑢

Θ, 𝜌)

Color

𝑐 𝑟 𝑖

Density

𝜌(𝑟 𝑖

)

Pairwise

RBF kernel

𝜃𝑢

3 Proposed Method: Contrastive Lift

Here and in Fig. 2, we describe Contrastive Lift, our approach for fusing 2D instance segmentation in 3D space. An image is a mapping I : Ω → R 3 , where Ω is a pixel grid in R 2 , and the values are RGB colours. We have a set of images I captured in the same scene and, for each image I ∈ I , we have its camera pose π ∈ SE (3) as well as object identity labels y : Ω →{ 1 , . . . , L } obtained from a 2D instance segmentation model for the image I . The labels y assigned to the 3D objects in one image I and the labels y ′ in another image I ′ are in general not consistent. Furthermore, these 2D label maps can be noisy across views.

We use this data to fit a neural field. The latter is a neural network that maps 3D coordinates x ∈ R 3 to multiple quantities. The first two quantities are density, denoted by ρ : R 3 ↦→ [0 , 1] , and radiance (colour), denoted by c : R 3 × S 2 ↦→ [0 , 1] 3 . Following the standard neural radiance field approach [36], the colour c ( x, d ) also depends on the viewing direction d ∈ S 2 . The third quantity is a D -dimensional instance embedding (vector) denoted as Θ : R 3 ↦→ R D . Each 3D coordinate is also mapped to a semantic embedding that represents a distribution over the semantic classes.

Differentiable rendering. The neural field associates attributes (density, colour, and embedding vectors) to each 3D point x ∈ R 3 . These attributes are projected onto an image I taken from a viewpoint π via differentiable ray casting. Given a pixel location u ∈ Ω in the image, we take N successive 3D samples r i ∈ R 3 , i = 0 , . . . , N -1 along the ray from the camera center through the pixel (so that ( u, f ) ∝ π -1 ( r i ) where f is the focal length). The probability that a photon is not absorbed when travelling from sample r i to sample r i +1 is exp( -ρ ( r i ) δ i ) where δ i = ∥ r i +1 -r i ∥ 2 is the distance between points. The transmittance τ i = exp( -∑ i -1 j =0 ρ ( r j ) δ j ) is the probability that the photon travels through sample r i . The projection of any neural field f onto pixel u is thus given by the rendering equation:

In particular, the colour of a pixel is reconstructed as I ( u ) ≈ R ( u | c ( · , d u ) , ρ, π ) where the viewing direction d u = r 0 / ∥ r 0 ∥ 2 . The photometric loss is thus:

Instance embeddings and slow-fast contrastive learning. The photometric loss (2) learns the colour and density fields ( c, ρ ) from the available 2D views I . Now we turn to learning the instance embedding field Θ : R 3 ↦→ R D . As noted in Section 1, the goal of the embeddings is to capture the (binary) distances between pixel labels sufficiently well. By that, we mean that the segments can be recovered, modulo a permutation of their labels, by simply clustering the embeddings a posteriori.

We cast learning the embeddings as optimising the following contrastive loss function:

where 1 is the indicator function, and sim( x, x ′ ; γ ) = exp( -γ ∥ x -x ′ ∥ 2 ) is a Gaussian RBF kernel used to compute the similarity between embeddings in Euclidean space. Therefore, pixels that belong to the same segment are considered positive pairs, and their embeddings are brought closer, while the embeddings of pixels from different segments are pushed apart. It is worth emphasizing that, since the object identity labels obtained from the underlying 2D segmenter are not consistent across images, L contr is only applied to positive and negative pixel pairs sampled from the same image.

While Eq. (3) is logically sound, we found it to result in gradients with high variance. To address this, we draw inspiration from momentum-teacher approaches [1, 4, 24] and define a slowly-updated instance embedding field ˜ Θ , with parameters that are updated with an exponential moving average of the parameters of Θ , instead of gradient descent. With this, we reformulate Eq. (3) as:

where θ u = R ( u | Θ , ρ, π ) , and ˜ θ u ′ = R ( u ′ | ˜ Θ , ρ, π ) . Here, we randomly partition the pixels Ω into two non-overlapping sets Ω 1 and Ω 2 , one for the 'fast' embedding field Θ , and another for the 'slow' field ˜ Θ . This avoids the additional cost of predicting and rendering each pixel's embedding using both models, and allows the computational cost to remain the same as for Eq. (3).

Concentration loss. In order to further encourage the separation of the embedding vectors Θ and thus simplify the extraction of the objects via a posteriori clustering, we introduce a loss function that further encourages the embeddings to form concentrated clusters for each object:

This loss computes a centroid (average) embedding as predicted by the 'slow' field ˜ Θ and penalizes the squared error between each embedding (as predicted by the 'fast' field Θ ) and the corresponding centroid. While this loss reduces the variance of the clusters, it is not a sufficient training objective by itself as it does not encourage the separation of different clusters, as done by Eqs. (3) and (4).

Semantic segmentation. For semantic segmentation, we follow the same approach as Semantic NeRF [67], learning additional embedding dimensions (one per semantic class), rendering labels in the same manner as Eq. (1), and using the cross-entropy loss for fitting the semantic field. Additionally, we also leverage the segment consistency loss introduced in [48] which encourages the predicted semantic classes to be consistent within an image segment.

Architectural details. Our neural field architecture is based on TensoRF [6]. For the density, we use a single-channel grid whose values represent the scalar density field directly. For the colour, a multi-channel grid predicts an intermediate feature which is concatenated with the viewing direction and passed to a shallow 3-layer MLP to predict the radiance field. The viewing directions are encoded using a frequency encoding [40, 58]. For the instance embedding field Θ (and also the 'slow' field ˜ Θ which has the exact same architecture as the 'fast' field Θ ), we use a shallow 5 -layer MLP that predicts an embedding given an input 3D coordinate. The same architecture is used for the semantic field. We use raw 3D coordinates directly without a frequency encoding for the instance and semantic components. More details are provided in the supplementary material.

Rendering instance segmentation maps. After training is complete, we sample 10 5 pixels from 100 random viewpoints (not necessarily training views) and render the fast instance field Θ at these pixels using the corresponding viewpoint pose. The rendered 10 5 × D embeddings are clustered using HDBSCAN [39] to obtain centroids, which are cached. Now, for any novel view, the field Θ is rendered and for each pixel, the label of the centroid nearest to the rendered embedding is assigned.

RGB

GTlabels

DETICpreds

View1

View2

View n-1

Viewn

4 Messy Rooms Dataset

In order to study the scalability of our method to scenes with a large number of objects, we generate a semi-realistic dataset using Kubric [20]. To generate a scene, we first spawn N realistically textured objects, randomly sampled from the Google Scanned Objects dataset [15], without any overlap. The objects are dropped from their spawned locations and a physics simulation is run for a few seconds until the objects settle in a natural arrangement. The static scene is rendered from M inward-facing camera viewpoints randomly sampled in a dome-shaped shell around the scene. Background, floor, and lighting are based on 360 ◦ HDRI textures from PolyHaven [66] projected onto a dome.

Specifically, we create scenes with N = 25 , 50 , 100 , and 500 objects. The number of viewpoints, M is set to min(1200 , ⌊ 600 × √ N / 25 ⌉ ) , and the rendered image resolution is 512 × 512 . To ensure that the focus is on the added objects, we use background textures old_room and large_corridor from PolyHaven that do not contain any objects. A total of 8 scenes are generated. The use of realistic textures for objects and background environments makes them representative of real-world scenarios.

Additionally, we would like to maintain a consistent number of objects per image as we increase the total number of objects so that the performance of the 2D segmenter is not a factor in the final performance. Firstly, we ensure that the floor area of the scene scales proportionally with the number of objects, preventing objects from becoming densely packed. Secondly, the cameras move further away from the scene as its extent increases. To ensure that the same number of objects is visible in each image, regardless of the scene size, we adjust the focal length of the cameras accordingly, i.e. , f = 35 . 0 × √ N / 25 , creating an effect similar to magnification. This approach ensures a comparable object distribution in each image, while enabling us to study the scalability of our method.

We render the instance IDs from each camera viewpoint to create ground-truth instance maps. These ground-truth instance IDs remain consistent (tracked) across views, as they are rendered from the same 3D scene representation. 1 Figure 3 shows illustrative examples from the dataset, which we name Messy Rooms . For evaluation (Section 5), semantic maps are required. As there is a large variety of different object types in Kubric, there is no off-the-shelf detector that can classify all of these, and since we are interested in the instance segmentation problem, rather than the semantic classes, we simply lump all object types in a single 'foreground' class, which focuses the evaluation on the quality of instance segmentation. More details about the dataset are provided in Appendix A.

5 Experiments

Benchmarks and baselines. We train and evaluate our proposed method on challenging scenes from the ScanNet [13], Hypersim [46], and Replica [50] datasets. We compare our method with Panoptic Lifting (PanopLi) [48], which is the current state-of-the-art for lifting 2D panoptic predictions to 3D, along with other 3D panoptic segmentation approaches: Panoptic Neural Fields [31] and

DM-NeRF [61]. We follow PanopLi [48] for the data preprocessing steps and train-test splits for each scene from these datasets. We also evaluate our proposed method and PanopLi on our Messy Rooms dataset (Section 4) that features scenes with up to 500 objects. These experiments aim to demonstrate the scalability of our proposed method as compared to the linear-assignment approach.

We compare two variants of our Contrastive Lift method: (1) Vanilla : uses the simple contrastive loss (Eq. (3)), and (2) Slow-Fast : uses slow-fast contrastive (Eq. (4)) and concentration (Eq. (5)) losses.

Metrics. The metric used in our evaluations is the scene-level Panoptic Quality (PQ scene ) metric introduced in [48]. PQ scene is a scene-level extension of standard PQ [28] that takes into account the consistency of instance IDs across views/frames ( aka tracking). In PQ scene , predicted/groundtruth segments with the same instance ID across all views are merged into subsets and all pairs of predicted/ground-truth subsets are compared, marking them as a match if the IoU is greater than 0 . 5 .

Implementation Details. We train our neural field model for 400k iterations on all scenes. Optimization-related hyper-parameters can be found in Appendix B.2. The density grid is optimised using only the photometric loss ( L RGB). While rendering the instance/semantic fields and computing associated losses (Eqs. (3) to (5)), gradients are stopped from flowing to the density grid.

For experiments on ScanNet, Hypersim and Replica, we use Mask2Former (M2F) [8] as the 2D segmenter to obtain the image-level semantic labels and instance identities. Although any 2D segmenter can be used, using M2F allows direct comparisons with other state-of-the-art approaches [48]. We follow the protocol used in [48] to map the COCO [33] vocabulary to 21 classes in ScanNet.

For experiments on Messy Rooms, we use Detic [68] instead since the object categories are not isomorphic to the COCO vocabulary M2F uses. We use the LVIS [21] vocabulary with Detic. To show the scalability of our method compared to a linear-assignment-based approach, we train the PanopLi [48] model on this dataset. For fair comparison, we first train the density, colour and semantic fields, which are identical in PanopLi and our approach. We then separately train the instance field using the respective linear-assignment and slow-fast contrastive losses, with all other components frozen, ensuring that performance is only influenced by the quality of the learned instance field.

5.1 Results

In Table 1, we compare the performance of our proposed approach with existing methods on three datasets: ScanNet [13], HyperSim [46], and Replica [50]. Since the semantic field and underlying TensoRF [5] architecture we use is similar to Semantic-NeRF [67] and PanopLi [48], we only report the PQ scene metric here and have added an additional table to Appendix D where we show that the mIoU and PSNR of our method match the performance of prior methods as expected. We observe that the proposed Slow-Fast approach consistently outperforms the baselines on all three datasets, while also outperforming the state-of-the-art Panoptic Lifting [48] method by +3 . 9 , +1 . 4 and +0 . 8

GT

DETIC[35]

PanopLi[23]

(a) Messy Rooms: large_corridor (25 objects)

(b) Messy Rooms: old_room (25 objects)

PQ scene points on these datasets respectively. We note that the Vanilla version of our method also performs comparably with PanopLi and outperforms other methods on all datasets.

Table 2 shows comparisons between our method and PanopLi [48] on scenes from our Messy Rooms dataset with 25 , 50 , 100 , and 500 objects. We see that the margin of improvement achieved by Contrastive Lift over PanopLi is even larger on these scenes, which shows that the proposed method scales favorably to scenes with a large number of objects. Fig. 4 shows qualitative results on two of these scenes. Even though the 2D segments obtained using Detic [68] are noisy (sometimes oversegmented ) and generally labelled inconsistently between views, the resulting instance segmentations rendered by Contrastive Lift are clearer and consistent across views. We also note that PanopLi sometimes fails to distinguish between distinct objects as pointed out in Fig. 4b.

5.2 Ablations

Different variants of Contrastive Lift. Our proposed method uses L sf (Eq. (4)) and L conc (Eq. (5)) to optimise the instance embedding field. To study the effect of these losses, we design a comprehensive set of variants of the proposed method: (1) Proposed ( L sf + L conc ), (2) Proposed without Concentration loss ( L sf ), (3) Vanilla contrastive ( L contr ), (4) Vanilla contrastive with Concentration loss applied to 'fast' field since there is no 'slow' field ( L contr + L conc ( fast ) ). Table 3 shows these ablations.

Effect of embedding size on performance. We investigate the impact of varying the instance embedding size on the performance of our proposed Contrastive Lift method. Specifically, we evaluate the effect of different embedding sizes using the PQ scene metric on ScanNet, Hypersim and Replica datasets. As shown in Fig. 5, we find that an embedding size as small as 3 is already almost optimal. Based on this, we use an embedding size of 24 for experiments with these datasets ( c.f. Table 1). For experiments with Messy Rooms dataset ( c.f. Table 2), we keep the embedding size to 3 .

1

3

6

12

24

48

96

192

Embedding size

0.58

0.59

0.60

0.61

0.62

0.63

PQ scene

(a) ScanNet [13]

1

3

6

12

24

48

96

192

Embedding size

0.55

0.56

0.57

0.58

0.59

0.60

PQ scene

1

3

6

12

24

48

96

192

Embedding size

0.58

0.59

0.60

0.61

0.62

0.63

PQ scene

(c) Hypersim [46]

VanillaContrastive

Slow-FastContrastive

VanillaContrastive

Slow-FastContrastive

(a) ScanNet scene0300_01 scene

(b) Messy Rooms large_corridor_25 scene

Figure 6: Embeddings obtained using vanilla (plain) contrastive learning and our proposed Slow-Fast contrastive learning. We use LDA [22] to project the embeddings to 2D for the illustration here.

Qualitative evaluation: Slow-fast vs vanilla contrastive learning. Fig. 6 shows how the embeddings are distributed in Euclidean space when learned using our proposed slow-fast contrastive loss (Eqs. (4) and (5)) and the vanilla contrastive loss (Eq. (3)). Embeddings learned with the slow-fast method are clustered more compactly and are easy to distinguish using any post-processing algorithm, such as HDBSCAN [39] which is used in this example.

Comparison to underlying 2D instance segmentation model with tracking. Before lifting, the predictions of the underlying 2D instance segmentation model (e.g., Mask2Former [8] or Detic [68]) are not consistent ( aka tracked) across frames/views. To achieve consistency and to allow comparisons with our approach, we post-process the 2D segmenter's predictions using Hungarian Matching for cross-frame tracking as follows:

w/ Hungarian matching (2D IoU) : Given sets of predicted segments ( P i and P i +1 ) from consecutive frames, compute IoU matrix by comparing all segment pairs in P i × P i +1 . Apply Hungarian matching to the IoU matrix to associate instance segments across frames.

w/ Hungarian matching based on IoU after depth-aware pose-warping : Use ground-truth pose and depth for warping ( i +1) -th frame's segmentation to frame i . Compute IoU matrix using warped segmentations and apply Hungarian matching.

w/ Hungarian matching using ground-truth pointcloud : Using only consecutive frames leads to errors in long-range tracking. To address this, starting from the first frame, unproject 2D segments into the 3D point cloud. Iteratively fuse segments in 3D using Hungarian matching. This way, segments from preceding frames along with 3D information are used for tracking.

The last two baselines use 3D groundtruth for tracking. Table 4 shows that despite 3D information being used for matching, Contrastive Lift still significantly improves over the underlying 2D model.

Frame-level improvement on underlying 2D segmentation models. In addition to generating consistent (tracked) instance segmentations, our method also improves the per-frame quality (i.e., not considering tracking) of the underlying 2D segmentation model. To show this, we train Contrastive Lift on ScanNet scenes with different 2D models, viz. Mask2Former [8], MaskFormer [9] and Detic [68]. In Table 5 we report the Panoptic Quality (PQ) metric (computed per frame) for these 2D models and for our method when trained with segments from each corresponding model.

Comparison of training speed with the linear-assignment loss method. While the exact number of objects present in a scene is unknown, linear assignment-based methods typically require a hyperparameter K that specifies the maximum number of objects. Solving the linear assignment problem in PanopLi's loss is O ( K 3 ) [48]. Our method is agnostic to object count, eliminating the need for such a parameter. Our approach does rely on the size of the embedding size, but, as shown above, even a very small size suffices. In the slow-fast contrastive loss computation, the Softmax function dominates more than the pairwise similarity matrix calculation. Consequently, we find that the training speed of Contrastive Lift is largely unaffected by the choice of embedding size.

Table 6 compares the training speed, measured on a NVIDIA A40 GPU, between PanopLi and our method, showing that PanopLi iterations become slower as K increases. We only optimise the instance embedding field with associated losses, while the density/colour/semantic fields are frozen.

6 Limitations

Contrastive Lift improves noisy 2D input segmentations, but cannot recover from catastrophic failures, such as entirely missing object classes. It also requires the 3D reconstruction to work reliably. As a result, we have focused on static scenes, as 3D reconstruction remains unreliable in a dynamic setting. Contrastive Lift is a useful building block in applications, but has no particular direct societal impact. The datasets used in this paper are explicitly licensed for research and contain no personal data.

7 Conclusion

We have introduced Contrastive Lift, a method for fusing the outputs of 2D instance segmenter using a 3D neural fields. It learns a 3D vector field that characterises the different object instances in the scene. This field is fitted to the output of the 2D segmenter in a manner which is invariant to permutation of the object labels, which are assigned independently and arbitrarily in each input image. Compared to alternative approaches that explicitly seek to make multi-view labels compatible, Contrastive Lift is more accurate and scalable, enabling future work on larger object collections.

Acknowledgments and Disclosure of Funding

We are grateful for funding from EPSRC AIMS CDT EP/S024050/1 and AWS (Y. Bhalgat), ERCCoG UNION 101001212 (A. Vedaldi and I. Laina), EPSRC VisualAI EP/T028572/1 (I. Laina, A. Vedaldi and A. Zisserman), and Royal Academy of Engineering RF\201819\18\163 (J. Henriques).

Data Ethics. We use the Google Scanned Objects, ScanNet, Hypersim and Replica datasets following their terms and conditions. These datasets do not contain personal data. For further details on ethics, data protection, and copyright please see https://www.robots.ox.ac.uk/~vedaldi/ research/union/ethics.html .

References

Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. Coke: Contrastive learning for robust keypoint detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 65-74, 2023. 5

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021. 3

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. NeurIPS , 2020. 3

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. ICCV , 2021. 3, 5, 17

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In arXiv , 2022. 1, 3, 7

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV) , 2022. 5

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proc. ICML , 2020. 3

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022. 7, 9, 10

Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. 2021. 9, 10

Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 1, pages 539-546. IEEE, 2005. 17

Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on pattern analysis and machine intelligence , 24(5):603-619, 2002. 16

Blender Online Community. Blender - a 3D modelling and rendering package . Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 25

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017. 3, 6, 7, 8, 9, 10, 16, 17

Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551 , 2017. 3

Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2553-2560. IEEE, 2022. 6

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd , volume 96, pages 226-231, 1996. 16

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. NeRF-SOS: Anyview self-supervised object segmentation on complex scenes. In The Eleventh International Conference on Learning Representations , 2023. 3

Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin P Murphy. Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277 , 2017. 3

Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic NeRF: 3d-to-2d label transfer for panoptic urban scene segmentation. arXiv.cs , abs/2203.15224, 2022. 3

Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. 2022. 6

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5356-5364, 2019. 7

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009. 9

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv.cs , abs/1911.05722, 2019. 3

Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelovi´ c. Object discovery and representation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII , pages 123-143. Springer, 2022. 5

Alexander Hermans, Georgios Floros, and Bastian Leibe. Dense 3d semantic mapping of indoor scenes from rgb-d images. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 2631-2638. IEEE, 2014. 3

Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. arXiv preprint arXiv:2304.04395 , 2023. 3

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553 , 2023. 3

Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proc. CVPR , 2019. 7

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. arXiv.cs , 2022. 1, 3

Shu Kong and Charless C Fowlkes. Recurrent pixel embedding for instance grouping. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9018-9028, 2018. 3

Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12871-12881, 2022. 1, 3, 6, 7

Abhijit Kundu, Yin Li, Frank Dellaert, Fuxin Li, and James M Rehg. Joint semantic segmentation and 3d reconstruction from monocular video. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 703-718. Springer, 2014. 3

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 7

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020. 3

Lingni Ma, Jörg Stückler, Christian Kerl, and Daniel Cremers. Multi-view deep learning for consistent semantic mapping with RGB-D cameras. In Proc.IROS , 2017. 3

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proc. CVPR , volume abs/2008.02268, 2021. 3, 4

Ruben Mascaro, Lucas Teixeira, and Margarita Chli. Diffuser: Multi-view 2D-to-3D label diffusion for semantic scene segmentation. In Proc. ICRA , 2021. 1, 3

John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA) , pages 4628-4635. IEEE, 2017. 3

Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. J. Open Source Softw. , 2(11):205, 2017. 5, 9, 15, 16

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV , 2020. 1, 3, 5

Ashkan Mirzaei, Yash Kant, Jonathan Kelly, and Igor Gilitschenski. Laterf: Label and text driven object radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III , pages 20-36. Springer, 2022. 3

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. In Proc. SIGGRAPH , 2022. 1, 3

Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4205-4212. IEEE, 2019. 3

Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. Advances in neural information processing systems , 30, 2017. 17

David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Semi-convolutional operators for instance segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 86-102, 2018. 3

Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10912-10922, 2021. 3, 6, 7, 9, 15, 16

Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T. Freeman, Fredo Durand, Joshua B. Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural scene representations from a single image. In The Eleventh International Conference on Learning Representations , 2023. 3

Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9043-9052, June 2023. 2, 3, 5, 6, 7, 8, 10, 15, 16, 17

Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. Proc. NeurIPS , 2019. 1

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019. 3, 6, 7, 9, 16

Niko Sünderhauf, Trung T Pham, Yasir Latif, Michael Milford, and Ian Reid. Meaningful maps with object-oriented semantic mapping. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5079-5085. IEEE, 2017. 3

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proc. NeurIPS , 2017. 3

Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. Cnn-slam: Real-time dense monocular slam with learned depth prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6243-6252, 2017. 3

Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In Proc. ICLR , 2020. 3

Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In Proceedings of the International Conference on 3D Vision (3DV) , 2022. 1, 3

Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In Proceedings of the International Conference on 3D Vision (3DV) , 2021. 3

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In Proc. NeurIPS , 2019. 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017. 5

Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A Prisacariu, Olaf Kähler, David W Murray, Shahram Izadi, Patrick Pérez, et al. Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction. In 2015 IEEE international conference on robotics and automation (ICRA) , pages 75-82. IEEE, 2015. 3

Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic segmentation of 3D scenes. arXiv.cs , abs/2111.13260, 2021. 1, 3

Bing WANG, Lu Chen, and Bo Yang. DM-neRF: 3d scene geometry decomposition and manipulation from 2d images. In The Eleventh International Conference on Learning Representations , 2023. 3, 7

Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown. Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling. In 2021 International Conference on 3D Vision (3DV) , pages 962-971. IEEE, 2021. 3

Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In International Conference on Learning Representations , 2022. 3

Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2560-2570, 2022. 3

Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In European Conference on Computer Vision , pages 288-307. Springer, 2022. 3

Greg Zaal, Rob Tuytel, Rico Cilliers, James Ray Cock, Andreas Mischok, Sergej Majboroda, Dimitrios Savva, and Jurita Burger. Polyhaven: a curated public asset library for visual effects artists and game designers. https://polyhaven.com/hdris , 2021. 6

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15838-15847, 2021. 1, 3, 5, 7, 15, 16

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twentythousand classes using image-level supervision. In ECCV , 2022. 6, 7, 8, 9, 10

A Messy Rooms dataset

The full Messy Rooms dataset introduced in this work can be accessed at this link: https:// figshare.com/s/b195ce8bd8eafe79762b . We show some representative examples from this dataset below in Fig. 14, 15, 16, and 17, which illustrate scenes with 25 , 50 , 100 and 500 objects respectively. Notice how the density of 'number of objects per image' remains similar as the number of objects increases from 25 to 500 . In Fig. 18, we show the corresponding 3D scenes used to generate the datasets.

B Implementation Details

Here, we provide further implementation details for our method in addition to the details mentioned in Sections 3 and 5 of the main paper.

B.1 Architectural details

Our neural field architecture is similar to [48] for fairness of comparisons. The density and color grids are initialized with a resolution of 128 × 128 × 128 which is progressively increased up to 192 × 192 × 192 by the end of training. The density and color grids use 16 and 48 components respectively. The output of the color grid is projected to 27 dimensions which are then processed by a 3-layer MLP with 128 hidden units per layer to output the RGB color. The fast and slow instance fields use a 256 hidden size in their MLP, while the semantic field uses a hidden size of 128 .

B.2 Training details

We follow a schedule for training our neural field model as follows: (1) For the first 40k iterations, the model is trained only with the RGB reconstruction loss ( L RGB). In this initial phase, the density field is optimized to reach a reasonable quality such that it can be used to render the instance/semantic field. (2) At 40k iterations, the semantic segmentation loss (i.e., cross-entropy loss, as in [48, 67]) is activated and used for the rest of the training iterations. (3) At 160k iterations, the instance embedding loss (i.e., L sf + L conc for the slow-fast version of our method or L contr for the vanilla baseline) is activated. (4) At 280k iterations, the segment consistency loss (proposed in [48]) is activated. For scenes from the Hypersim dataset [46], we activate the segment consistency loss at 200k iterations instead. In our proposed slow-fast clustering framework, the slow field parameters are updated using an exponential moving average with momentum m = 0 . 9 , i.e. ˜ Θ = ˜ Θ × m +Θ × (1 -m ) .

The RGB reconstruction loss, semantic segmentation loss, instance embedding loss, and segment consistency loss are balanced using weights of 1 . 0 , 0 . 1 , 0 . 1 , and 1 . 0 respectively. However, we empirically observe that the final performance is not very sensitive to these choices. A learning rate of 5 · 10 -4 is used for all MLPs and 0 . 01 for the grids. A batch-size of 2048 is used to train all models.

B.3 Post-processing Clustering details

Given the learned instance embedding field, a clustering mechanism (e.g., HDBSCAN [39]) can be used to obtain cluster centroids and generate instance segmentation maps. We have chosen HDBSCAN since it does not require the number of objects to be known a priori . Generally, clusters obtained by HDBSCAN are non-convex and assigning the label of the nearest centroid is not recommended. But, our proposed method results in highly compact clusters which makes this simple method effective. We perform clustering as follows.

Hierarchical clustering using semantic predictions. An advantage of our method is that we can use our model's semantic predictions to guide the clustering of the instance embeddings. 'Instance' segmentation requires separating instances of the same semantic class. Based on this, we perform hierarchical clustering as follows:

After training, sample 10 5 pixels from 100 random viewpoints and render the fast instance field Θ and semantic field for these pixels.

Group the 10 5 rendered embeddings based on predicted semantic labels, forming S groups.

Cluster the embeddings within each group separately using HDBSCAN, caching cluster-centroids for each group, assigning a unique instance label to each centroid.

For a novel view, render the instance field and semantic field, assigning each pixel an instance embedding and semantic class. Obtain the instance label for a pixel by finding the closest centroid to the rendered instance embedding within the group of the same semantic label as the pixel.

Tuning clustering hyperparameter. Despite HDBSCAN's robustness to hyperparameter selection, we found that it is beneficial to specify a minimum cluster size . Since we always sample and render 10 5 pixels for clustering, the expected cluster size per object decreases as the number of objects increases. To determine an optimal value, we perform a hyperparameter sweep using 10% of the training data, which includes training viewpoints and associated segments from the 2D segmenter. We then use this identified optimal value to perform clustering as described above.

C Comparison between different clustering algorithms.

We compare HDBSCAN with other unsupervised clustering algorithms, viz. MeanShift [11] and DBSCAN [16]. We tune the bandwidth parameter with MeanShift, and the epsilon parameter with DBSCAN. However, we note that MeanShift struggles to converge for embedding sizes greater than 10. For fair comparison, we train our model with an embedding size of 3. Table 7 show that both MeanShift and DBSCAN perform slightly worse but remain comparable to HDBSCAN. Generally, any unsupervised clustering method that doesn't require prior knowledge of the number of clusters is suitable for use with our method.

D Quality of our semantic and radiance field

In Tables 1 and 2 in the main paper, we evaluate the quality and consistency ( aka tracking) of the instance segmentation maps obtained by the various tested methods. The semantic field and density/color field architecture of our method is based on Panoptic Lifting [48], which in turn is a modification of Semantic-NeRF [67] for the semantic component. As a sanity check, we compare the quality of rendered semantic and RGB maps obtained by these methods with ours. Table 8 shows the mean Intersection over Union (mIoU) and peak-signal-to-noise ratio (PSNR) metrics. As expected, the mIoU and PSNR obtained by our method is nearly the same as Panoptic Lifting.

For the Messy Rooms dataset we have explicitly ensured that the density and semantic model used by both Panoptic Lifting and our method are the same and the only factor influencing the final performance is the quality of the learned instance field. This is done by pre-training the same density and semantic fields for both methods and subsequently training the instance field using the respective objective functions.

E Comparisons to other metric learning loss functions

While we employ a contrastive loss formulation to learn the instance embeddings, there are many alternative loss functions proposed in the metric learning literature. For comparison, we also train our instance embedding field with the Associative Embedding (AE) loss [44] and the margin-based contrastive loss [10].

To compute the AE Loss, we divide the batch Ω into groups based on segment ID. If there are K groups/segments, G 1 . . . G K , then

The margin-based contrastive loss is defined as:

̸

Here, θ u = R ( u | Θ , ρ, π ) is the rendered instance field at pixel u . Note that, the slow-fast field formulation is not used in these comparisons. In Table 9 we compare the proposed objective (slowfast) to these baselines. We observe that both the vanilla contrastive, as well as the slow-fast version of our method, outperform the alternatives.

DINO-style loss. Since our method is inspired by momentum-teacher approaches, e.g. DINO [4], we design a baseline with a DINO-style learning mechanism. Two pixels from the same instance segment are fed into the slow and fast fields. This is akin to DINO, where two random image transformations are fed to the student and teacher networks. A Centering layer is applied to the slow field embedding. A Projection module (with proj_dim = 512 ) is added to both the slow and fast fields followed by Softmax and a cross-entropy loss is used. After training, embeddings from the fast field are used for clustering. Results in Table 9 demonstrate that this baseline performs worse than the metric-learning losses on ScanNet. We do not evaluate this baseline on the Messy Rooms dataset.

F Stability of Slow-Fast loss compared to Vanilla contrastive loss

We found that training with the vanilla contrastive loss resulted in gradients with higher variance. The usage of a slowly-updated embedding field in the slow-fast loss formulation mitigates this problem and leads to more stable training. We quantitatively verify this by computing the relative variance (which is V ar ( · ) / Mean ( · ) ) in the gradients of the loss w.r.t. to the instance embeddings ( i.e. dL / d Θ ) . Figure 7 shows that the vanilla loss exhibits spikes with a maximum relative variance around 10 7 , whereas the slow-fast version remains around a much controlled range of around 10 1 .

G More qualitative visualizations

In Figures 8, 9, 10, 11, 12 and 13, we visualize the predictions of our proposed method on scenes from ScanNet [13] and Messy Rooms. Left-most columns show instance labels obtained after

0

10000

20000

30000

40000

Iterations

0e+00

5e+06

1e+07

2e+07

2e+07

2e+07

3e+07

Relative Variance of

dL

/

d

Vanilla

0

10000

20000

30000

40000

Iterations

15

10

5

0

5

10

Relative Variance of

dL

/

d

SlowFast Loss

SlowFast

clustering, which as we can see are consistent across different views. To understand how well the embeddings are clustered, we visualize heatmaps of distance of rendered embeddings from clustercentroids. Specifically, we choose 4 centroids, and for each centroid c i and each pixel u , we plot H ( u ) = -log( ∥ θ u -c i ∥ ) normalized to [0 , 1] , where θ u is the rendered embedding.

Note that, instance labels are only computed for pixels belonging to the ' thing ' semantic categories (as predicted by the semantic field) 2 . The ' stuff ' pixels are masked out.

As can be seen in all these visualizations, the heatmaps are peaked at the corresponding object locations and close to zero elsewhere which indicates the embeddings are compactly clustered around the corresponding centroid for each object. In Fig. 13, we can see that even in a scene with 50 objects, the embeddings for each instance are distinctly separable.

Predictedlabels

Centroid 1

Centroid 2

Centroid3

Centroid4

1.0

0.8

View1

0.6

0.4

0.2

0.0

1.0

0.8

View2

0.6

0.4

0.2

0.0

1.0

0.8

View3

0.6

0.4

0.2

0.0

Predictedlabels

Centroid1

Centroid2

Centroid3

Centroid4

1.0

0.8

View1

0.6

0.4

0.2

0.0

1.0

0.8

View 2

0.6

0.4

0.2

0.0

1.0

0.8

View 3

0.6

0.4

0.2

0.0

Predictedlabels

Centroid1

Centroid 2

Centroid3

Centroid4

1.0

View1

0.8

0.6

0.4

0.2

0.0

1.0

0.8

View 2

0.6

0.4

0.2

0.0

1.0

0.8

View3

0.6

0.4

0.2

0.0

Predictedlabels

Centroid1

Centroid 2

Centroid3

Centroid4

1.0

0.8

View1

0.6

0.4

0.2

0.0

1.0

0.8

View2

0.6

0.4

0.2

0.0

1.0

View3

0.8

0.6

0.4

0.2

0.0

Predictedlabels

Centroid1

Centroid 2

Centroid3

Centroid4

1.0

View1

0.8

0.6

0.4

0.2

0.0

1.0

0.8

View2

0.6

0.4

0.2

0.0

1.0

0.8

View3

0.6

0.4

0.2

0.0

Predictedlabels

Centroid1

Centroid 2

Centroid3

Centroid 4

1.0

0.8

View1

0.6

0.4

0.2

0.0

1.0

0.8

View2

0.6

0.4

0.2

0.0

1.0

0.8

View3

0.6

0.4

0.2

0.0

RGB

GTlabels

DETICpreds

RGB

GT labels

DETIC preds

(a)

large_corridor

: 25 objects

(b)

old_room

: 25 objects

RGB

GTlabels

DETIC preds

RGB

GT labels

DETIC preds

(a)

large_corridor

: 50 objects

(b)

old_room

: 50 objects

RGB

GTlabels

DETIC preds

RGB

GT labels

DETIC preds

CC

(a)

large_corridor

: 100 objects

(b)

old_room

: 100 objects

RGB

GTlabels

DETIC preds

RGB

GT labels

DETIC preds

(a)

large_corridor

: 500 objects

(b)

old_room

: 500 objects

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the task of 3D instance segmentation by leveraging pre-trained 2D instance segmentation models. The authors propose a novel approach to lift 2D segments to 3D via a neural field. This idea is not completely new [38, 51], but the authors propose a contrastive loss that replaces the Hungarian-algorithm-based loss used in [38]. Moreover, the authors propose a synthetic dataset where the number of objects can be controlled, and show that the Hungarian-algorithm-based loss slows down substantially as the number of objects increase. Finally, the authors augment the contrastive loss with a momentum-teacher component (similar to [16]).\n\n_[16] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman,\nJoão Carreira, and Relja Arandjelović. Object discovery and representation networks. In Computer\nVision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part\nXXVII, pages 123–143. Springer, 2022._\n\n_[38] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. arXiv.cs, abs/2212.09802,\n2022._\n\n_[51] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot,\nAndrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic\nsegmentation of 3D scenes. arXiv.cs, abs/2111.13260, 2021_\n\nSTRENGTHS:\nS1) The proposed scheme to deal with instance label ambiguity is novel.\n\nS2) The paper is mostly well-written.\n\nS3) The authors show that the proposed contrastive momentum-teacher loss gives good performance and is computationally lighter than the Hungarian-algorithm-based loss.\n\nS4) The problem of adapting 2D instance segmentation methods to 3D is useful to real-world applications.\n\nWEAKNESSES:\nW1) The notation has some issues, is confusing in some places, and deviates a bit from prior work. \n- It is not common to let $y$ represent a mapping. It is common to say that $f$ is a mapping such that $y = f(x)$. Same goes for $Y$. The label $y(u)$ is sometimes referred to as a label and sometimes as a function, which is a bit confusing.\n- I think $u$ should be the location of a pixel, rather than the actual pixel (which would be in $\\mathbb{R}^3$).\n- $Y(x)$ is not presented as a function of $x$, but as a function of $u$.\n- $\\Theta$ is introduced as both a mapping $\\Theta: \\mathbb{R}^2\\rightarrow \\mathbb{R}^D$ and $\\Theta: \\mathbb{R}^3\\rightarrow \\mathbb{R}^D$.\n- Also $\\rho$, $c$, and $\\Theta$ are introduced as both mappings and actual elements of the codomain of the mapping.\n- Usually, $x$ and $d$ are used for 3D location and ray direction. While it is not a crime to change notation, I cannot see a reason for replacing $d$ with $v$.\n- Equation (1) does not link to the input to the nerf ($x$ and $v$). The link can only be found in the text in l128-l129. It would be easier to understand if $R$ was actually described as a function of $u$.\n\nW2) The name Slow-Fast is confusing.\n- The loss component name slow-fast could have been named _student-teacher_ or something with _exponential moving average_. Currently, the name makes it easy to confuse with Slow-Fast [1001].\n\nW3) The prior work [38] also adapts 2D instance segmentation models to 3D instance segmentation using NeRFs. The proposed approach replaces the matching-based loss and the \"slow-fast\"-component seems necessary for good performance. Is the comparison to [38] completely fair? Except for the proposed changes, are all other things equal, e.g., the underlying instance 2D segmentation approach, NeRF architecture, or training scheme? I find this difficult to tell from the paper text. It is clear that the proposed approach yields good performance and trains faster than [38], but it is not clear whether this is purely do to the proposed changes.\n\nMinor remarks\n- \"much smaller\" on l62 should probably be removed.\n\n\n_[1001] Feichtenhofer, Christoph, et al. \"Slowfast networks for video recognition.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019._\n\nQUESTIONS:\nQ1) A prior work [1002] might be relevant to the 3D semantic segmentation background. The references tackles that problem and predates the references [23, 29, 51, 57]. \n\nQ2) How does the proposed approach compare to [38]? See W3.\n\nQ3) What does the statement on l290 mean? How does the proposed approach improve noisy 2D semgmentations?\n\nQ4) On l291, what does it mean for 3D reconstruction to work properly?\n\n_[1002] Lawin, Felix Järemo, et al. \"Deep projective 3D semantic segmentation.\" Computer Analysis of Images and Patterns: 17th International Conference, CAIP 2017, Ystad, Sweden, August 22-24, 2017, Proceedings, Part I 17. Springer International Publishing, 2017._\n\nLIMITATIONS:\nThe authors provide a discussion that clarifies some important aspects, for instance that the proposed approach supports only static scenes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper introduces a novel Contrastive Lift method for 2D segment lifting to 3D reconstruction and instance segmentation. The authors fuse multiview representations obtained from pre-trained 2D models into a unified 3D neural field. They propose a scalable slow-fast clustering objective function that enables segmenting without an upper bound on the number of objects. Additionally, a new semi-realistic dataset is created to evaluate the proposed method, which demonstrates superior performance compared to previous state-of-the-art approaches on both public datasets and the newly introduced dataset.

STRENGTHS:
1. The research motivation and key challenges are generally well illustrated and summarized. 
2. The newly constructed framework and dataset fits well within the 3D vision field, particularly the flexible and scalable design for lifting 2D segments to 3D.
3. The proposed method is technically sound and achieves SOTA performance on the newly proposed dataset and challenging scenes on public datasets.

WEAKNESSES:
1. The paper lacks a discussion on the complexity of the proposed method and its potential impact.
2. Ablations on different 2D segmenters are not included in the paper.
3. The analysis of the ablations and their corresponding results is insufficient.

QUESTIONS:
Please refere to Weaknesses.

LIMITATIONS:
Please refere to Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes contrastive learning for lifting 2D segments to 3D and fuses the learned embedding by means of a neural field representation, namely Contrastive Lift. The authors further propose a slow-fast clustering objective function, which makes the method scalable for scenes with a large number of objects. To further validate the ability of the method, this paper also introduces a new dataset, Messi Rooms, which includes up to 500 objects as a benchmark for instance segmentation with a large number of objects. The experiments show that the proposed approach outperforms former SOTA on ScanNet, Hypersim, Replica, and Messy Rooms.\n\nSTRENGTHS:\n- The proposed approach employs a low-dimensional Euclidean embedding to represent a 3D instance. The dimensionality D is far less than the number of objects L, making the approach more efficient and easily extended to larger numbers of objects. Most importantly, using the 3D instance embedding implicitly ensures multi-view consistency. It avoids the assignment problem that exists in Panoptic Lifting [43].\n\n- Using contrastive learning and the clustering strategy makes the proposed approach independent of the number of objects, which is more suitable for scenes with a large number of objects.\n\n- The slow-fast contrastive learning is scalable for different object numbers and stabilizes the training phase. And the proposed concentration loss ensures the concentration of embeddings within the same cluster. It improves the more complete instance segmentation results.\n\n- The proposed semi-realistic dataset, Messy Rooms, provides a novel benchmark for testing the performance on scenes with large object numbers.\n\nWEAKNESSES:\n- In Tab. 1 and Tab.2, the metric used for evaluation only uses PQ^scene^, which is cherry-picked. In Panoptic Lifting [43], mIoU and PSNR are also used for evaluation. A comprehensive comparison according to different metrics should be included.\n\n- For semantic segmentation, Contrastive Lift should append a new branch to predict the semantic labels and be supervised by the segment consistency loss in [43]. I suppose that the model should be trained specifically for semantic segmentation. Panoptic Lifting can predict semantic and instance labels simultaneously. I am curious about whether the additional supervision on semantic labels would influence the instance segmentation results. Please explain it.\n\nQUESTIONS:\n- The random partition of pixels into two non-overlapping sets needs further explanation. (Randomly choose pixels? Randomly divide images with pre-defined boundaries? What is the proportion between two sets? etc.)\n\nLIMITATIONS:\nThe authors have addressed some limitations in Sec. 6. There is no negative social impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method tries to solve the problem of reconstructing a 3D scene together with the underlying instance segmentation. Prior work required either GT tracking data or concurrent work a less efficient way to assign instances. From a set of images a Neural Radiance Field is reconstructed together with a feature field, that represents an embedding of the instance. Instance embeddings are guided by a contrastive loss function, that pushes embeddings that are projected into pixels from the same segment in a 2D segmentation mask closer and embeddings projecting into different masks apart. To improve the stability of the training, the authors propose an additional loss with a jointly trained slowly-updated instance embedding field updated with a moving average over the parameters of the faster field instead of SGD.\nInstances are later computed by clustering embeddings, which is supported by the third loss term, which uses an average embedding from the slowly updated field to penalize the difference for the fast-field predictions. Specific values in the embedding vector are assigned to semantic classes for semantic segmentation and are directly supervised with the 2D semantic maps.  \nAdditionally, the authors propose a novel dataset with up to 500 objects for evaluating future 3D instance segmentation\n\nSTRENGTHS:\nSo far 3D instance segmentation methods, such as Neural Panoptic Fields (referenced by the authors as well) required a tracking algorithm or GT tracking data to reconstruct instance labels of the 3D scene and this method presents a light, optimization-based approach, that directly learns an instance embedding from semantic maps through alignment. \n\nThe authors describe their method in a way that is understandable and design choices, especially for the loss function and the learning paradigm are reasonable and justified by ablation and additional experiments.\n\nIn general, this is a well-designed method that leverages the current state of the art and adds interesting new components to allow a jointly learning of the radiance field, instance and semantic embedding.\n\nA big plus of the presented method is the additional dataset with up to 500 objects and the submission of the code, that allows the reproducibility of the results.\n\nWEAKNESSES:\nWhile the presented evaluation of the proposed dataset shows a clear advantage over state-of-the-art methods on existing and their own synthetic dataset, as well as ScanNet, methods like Panoptic Fields have also shown results on complex outdoor scenes, such as the KITTI dataset. Therefore an evaluation in such a complex outdoor setting would further strengthen the paper and usability in future work.\n\nQUESTIONS:\n- It seems like a fixed number of clusters is given or how is the number of clusters decided?\n\nLIMITATIONS:\nThe authors address most of the limitations I can think of and ablate the use of different methods they rely on, such as the clustering method in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the 3D object instance segmentation inside a 3D NeRF space. Specifically, to train the model, conservative loss between features generated by slow and fast NeRF models are computed to 1) maximize the feature distance between different semantic regions, 2) minimize the feature distance within the same semantic regions. Also, a dataset is introduced namely Messy Room, which consists of rendering real captured objects from Google Scanned Objects. Experiments on the dataset show a reasonable improvement in comparison to baselines.\n\nSTRENGTHS:\nThe proposed contrastive learning for 3D semantic segmentation on NeRF is elegant and novel. I would believe such a structure is considered to be better in comparison to previous works that directly output the semantic labels from the NeRF network. Also, the introduced messy rooms dataset is believed to be useful for the community, despite the dataset is partially synthetic. The overall performance improvements from the baseline of the proposed method are not huge but still reasonable.\n\nWEAKNESSES:\n1) The proposed method is only evaluated on the half-synthetic dataset with small objects on the table. It is necessary to evaluate the proposed method on some real images, at least qualitatively.\n\n2) The performance of the segmentations before lifting is not reported in the experiment section, it is unclear how the proposed method improves from the 2D segmentation.\n\n3) The contrastive training pipeline is partially similar to this work [1], it would be better to include it in discussion.\n\n[1] Bai, Y., Wang, A., Kortylewski, A., & Yuille, A. (2020). Coke: Localized contrastive learning for robust keypoint detection. arXiv preprint arXiv:2009.14115.\n\nQUESTIONS:\n1) How the ground plane is determined and set in the messy Room dataset?\n\n2) The author mentions in line 148 that \"gradients with high variance\". What is a high variance on gradients? The author should quantitatively study how the variance is controlled using the proposed SF loss in comparison to vanilla contrastive loss.\n\n3) A clustering operator is conducted on the feature vectors for segmentation, is this similar to this work? [2]\n\n4) In the visualization, how does the trained feature vectors convert into semantic labels?\n\n5) Some typos: in figure 2 L_cntr but in equation 5 L_conc.\n\n\n[2] Yu, Q., Wang, H., Qiao, S., Collins, M., Zhu, Y., Adam, H., ... & Chen, L. C. (2022, October). k-means Mask Transformer. In European Conference on Computer Vision (pp. 288-307). Cham: Springer Nature Switzerland.\n\nLIMITATIONS:\nThe authors include and discuss limitations of this work. There seems no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}