SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\text {c o r r}} = \sum_ {h w i j} \left(\operatorname {C o r r} _ {-} F _ {h w i j} - b\right) \times \operatorname {C o r r} _ {-} D _ {h w i j}, \tag {12}
$$
Table 1: Quantitative comparisons. We report the mIoU(↑) scores and the Accuracy(↑) scores of the following methods in 6 scenes and highlight the best, second-best, and third-best scores. Our method outperforms both 2D and 3D methods without any segmentation annotations in training.
Methods	bed	sofa	lawn	room	bench	table
mIoU	Accuracy	mIoU	Accuracy	mIoU	Accuracy	mIoU	Accuracy	mIoU	Accuracy	mIoU	Accuracy
2D	LSeg [8]	56.0	87.6	04.5	16.5	17.5	77.5	19.2	46.1	06.0	42.7	07.6	29.9
ODISE [16]	52.6	86.5	48.3	35.4	39.8	82.5	52.5	59.7	24.1	39.0	39.7	34.5
OV-Seg [19]	79.8	40.4	66.1	69.6	81.2	92.1	71.4	49.1	88.9	89.2	80.6	65.3
3D	FFD [4]	56.6	86.9	03.7	09.5	42.9	82.6	25.1	51.4	06.1	42.8	07.9	30.1
Sem(ODISE) [45]	50.3	86.5	27.7	22.2	24.2	80.5	29.5	61.5	25.6	56.4	18.4	30.8
Sem(OV-Seg) [45]	89.3	96.7	66.3	89.0	87.6	95.4	53.8	81.9	94.2	98.5	83.8	94.6
LERF [2]	73.5	86.9	27.0	43.8	73.7	93.5	46.6	79.8	53.2	79.7	33.4	41.0
Ours	89.5	96.7	74.0	91.6	88.2	97.3	92.8	98.9	89.3	96.3	88.8	96.5
where  $b$  is a hyper-parameter denoting that we consider the segmentation probabilities of two spatial locations  $(h, w)$  and  $(i, j)$  to be similar if their DINO features' similarity is larger than  $b$  and distant if less than  $b$ . Nonetheless, the correlation loss  $\mathcal{L}_{corr}$  introduces significant instability due to the diverse shapes of distributions with large divergence from a target distribution, making the loss assign wrong labels to the segmented parts. Conversely, when a distribution displays a low JS divergence with the target distribution, it consistently demonstrates a similar shape to the target distribution, as shown in Fig. 3. Based on this observation, we propose re-balancing the weights associated with similar and dissimilar DINO features. Specifically, we allocate a much greater weight to the correlation loss arising from similar DINO features and a smaller weight to that of dissimilar DINO features, thereby mitigating the instability caused by the correlation loss. Thus the Feature-Distribution Alignment (FDA) loss can be formulated with:
$$
\operatorname {p o s} _ {-} F = \operatorname {c l a m p} \left(\operatorname {C o r r} _ {-} F - b, \min  = 0\right), \quad \operatorname {n e g} _ {-} F = \operatorname {c l a m p} \left(\operatorname {C o r r} _ {-} F - b, \max  = 0\right), \tag {13}
$$
$$
\mathcal {L} _ {F D A} = \lambda_ {p o s} \sum_ {h w i j} (p o s \_ F _ {h w i j} \times C o r r \_ D _ {h w i j}) / \text {c o u n t} _ {\text {n o n z e r o}} (p o s \_ F) +
$$
$$
\lambda_ {n e g} \sum_ {h w i j} \left(n e g _ {-} F _ {h w i j} \times C o r r _ {-} D _ {h w i j}\right) / \text {c o u n t} _ {-} \text {n o n z e r o} (n e g _ {-} F), \tag {14}
$$
where  $\mathrm{clamp}(\min / \max = 0)$  is to clamp all the elements smaller/greater than 0 to 0, thus making  $pos\_F \geq 0$  and  $neg\_F \leq 0$ , count_nonzero() is to count the number of non zero elements, and  $\lambda_{pos}, \lambda_{neg}$  are the weights associated with similar and dissimilar DINO features, which are set as  $\lambda_{pos} = 200 \gg \lambda_{neg} = 0.2$  by default.
The total training loss is:  $\mathcal{L} = \mathcal{L}_{\text{supervision}} + \mathcal{L}_{RDA} + \mathcal{L}_{FDA}$ , where  $\mathcal{L}_{\text{supervision}}, \mathcal{L}_{RDA}$  are calculated with randomly sampled rays and  $\mathcal{L}_{FDA}$  is calculated with randomly sampled patches.
We evaluate our method on 3D open-vocabulary segmentation, showing that our method can recognize long-tail classes and produce highly accurate object boundaries even with limited input data. We employ TensoRF [42] as the backbone and extract 3 scales of pixel-level CLIP features. More implementation details and experiments are in the appendix.
Dataset. Existing 3D segmentation datasets predominantly focus on either restricted scenes with a narrow range of object classes [5, 70], or individual objects [71-73], thereby limiting their capacity to fully assess the task of 3D open-vocabulary segmentation. Thus following [2], we create a dataset comprising 10 distinct scenes. Each scene features a set of long-tail objects situated in various poses and backgrounds. Ground truth masks for the test views are manually annotated, enabling both qualitative and quantitative evaluation of our segmentation methods. We also evaluate our method on more diverse datasets which include human body, human head, indoor scenes with low-quality images [70, 74], and a complex scene from LERF datasets [2], as shown in the appendix.
Baselines. We benchmark our method with three NeRF-based methods capable of 3D open-vocabulary segmentation: FFD [4], Semantic-NeRF (abbreviated as Sem) [45], and LERF [2].
Figure 4: Qualitative comparisons. Visualization of the segmentation results in 3 scenes. Our method successfully recognizes long-tail classes and produces the most accurate segmentation maps.
FFD, a pioneering effort in 3D open-vocabulary segmentation, applies the feature maps of the 2D open-vocabulary segmentation method, LSeg [8], to achieve its segmentation results. For Sem, we adapt it to the open-vocabulary segmentation context by distilling segmentation results from two state-of-the-art 2D open-vocabulary segmentation models: the diffusion-model-based ODISE [16], and the CLIP-based OV-Seg [19]. LERF closely aligns with our proposed approach due to its use of knowledge distillation from CLIP and DINO. However, its primary focus is on object localization rather than segmentation. We use the same scale level number and patch sizes in LERF for fair comparisons. We also include results obtained by independently segmenting each test view using the aforementioned 2D models. Note that under our settings, FFD, Sem are fully supervised methods using segmentation annotations.
We present the qualitative results in Fig. 4 and quantitative results in Tab. 1. Our proposed method outperforms all other techniques, including those which heavily rely on extensive segmentation annotations, such as LSeg, ODISE, OV-Seg. In particular, ODISE and FFD underperform in our evaluation, as they are unable to identify many long-tail classes, suggesting that the fine-tuning process of LSeg and ODISE may have led to a significant loss of the open-vocabulary knowledge originally encapsulated by CLIP and Stable Diffusion [23]. OV-Seg attempts to retain CLIP's knowledge by leveraging a mined dataset, however, it requires a mask proposal model which produces inconsistent segmentation across views, making Sem(OV-Seg) produce noisy and imprecise segmentation. LERF
Figure 5: Studies. Visualization of the studies on ablations and limited input.
also fails to capture precise object boundaries due to its usage of a relatively naive regularization loss, which fails to fully exploit the object boundary information within the DINO features. In contrast, our method exhibits robust performance, successfully recognizing long-tail classes and generating accurate and well-defined boundaries for each class. However, LERF allows querying any object without the need for running optimization again, which is an advantage over our method.
4.2 Studies
Ablations. We conduct ablation studies to evaluate the individual contributions of the RDA loss and the FDA loss to the overall performance of our proposed method. As shown in Tab. 2, both RDA loss and FDA loss are crucial to our method, without each of which can result in severe performance degradation. As illustrated in Fig. 5, without the RDA loss, the model does not resolve the ambiguities of the CLIP features, leading to misclassifications. For instance, it fails to distinguish between an orange cat and a Portuguese egg tart, and confuses a mini offroad car with wood. Without the FDA loss, although our method can correctly locate each class, it fails to segment precise object boundaries. When discarding the re-balancing in the FDA loss, i.e. using the correlation loss [68], the model produces accurate boundaries but assigns each cluster the wrong label due to the instability brought by diverse distribution shapes.
Table 2: Studies. We report the mean mIoU(↑) scores and the Accuracy(↑) scores in our studies.
mIoU	Accuracy
w/o RDA loss	57.2	79.4
w/o FDA loss	58.2	82.7
w/o re-balance	44.9	74.3
50% views	85.7	95.7
10% views	79.1	94.6
single scale	85.2	95.5
single & 10%	77.1	94.6
full model	86.2	95.8
Limited input. Given the substantial computational and storage demands of extracting hierarchical CLIP features for each view (exceeding 1GB for 3 scales in the full model), we explore whether reducing input CLIP features would yield similar results, as shown in Tab. 2 and Fig. 5. We test two modifications: reducing views for feature extraction and using a single scale of CLIP features rather than three. Halving the input views for feature extraction leads to negligible performance degradation  $(< 1\%)$  and minor visual differences compared to the full model. When reducing to only  $10\%$  of input views, equivalent to 2-3 views in our dataset, we observe a modest  $9\%$  drop in the mIoU score and a  $1\%$  decrease in the Accuracy score, while retaining accurate segmentation across most classes. Using a single scale of CLIP features also only incurs minimal degradation  $(< 1\%)$ . Even under extreme conditions, i.e., extracting a single scale of features from  $10\%$  of input views (total only  $1/30$  input of the full model), performance degradation is limited to  $10\%$ . This efficient approach even outperforms LERF [2] which utilizes all input views and scales, highlighting our method's robustness.
The limitations of our method are twofold. First, unlike LERF [2], our method requires text labels before training. To perform segmentation with new text labels, our method needs to be retrained.
Inferring accurate boundaries with open vocabularies is challenging for implicit representations like NeRF, as NeRF learns a continuous representation rather than a discrete one. It is promising to learn object-level discrete representation using NeRF in future work.
Second, since our method has never seen any segmentation maps during training (only weakly supervised by the text labels), it fails to segment complex scenes like indoor datasets [70, 74] with high precision, as shown in the appendix. Our method distills pixel-level CLIP features in a patch-based fashion with a strong inductive bias for compact objects with an aspect ratio close to 1. For objects with large complex shapes and unobvious textures, our method would fail to recognize them.
In this study, we address the challenge of 3D open-vocabulary segmentation by distilling knowledge from the pre-trained foundation models CLIP and DINO into reconstructed NeRF in a weakly supervised manner. We distill the open-vocabulary multimodal knowledge from CLIP with a Selection Volume and a novel Relevancy-Distribution Alignment loss to mitigate the ambiguities of CLIP features. In addition, we introduce a novel Feature-Distribution Alignment loss to extract accurate object boundaries by leveraging the scene layout information within DINO features. Our method successfully recognizes long-tail classes and produces precise segmentation maps, even when supplied with limited input data, suggesting the possibility of learning 3D segmentation from 2D images and text-image pairs.
We sincerely thank Zuhao Yang, Zeyu Wang, Weijing Tao, and Kunyang Li for collecting the dataset. This project is funded by the Ministry of Education Singapore, under the Tier-1 project scheme with project number RT18/22.
[1] Nur Muhammad Mahi Shafullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clipfields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022.1
[2] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553, 2023. 1, 3, 7, 9, 16, 17
[3] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023. 1, 3
[4] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nef for editing via feature field distillation. arXiv preprint arXiv:2205.15585, 2022. 1, 3, 4, 7, 17
[5] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 1, 3, 7
[6] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99-106, 2021. 1, 3, 4
[7] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211.15654, 2022. 1, 3
[8] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546, 2022. 1, 3, 7, 8
[9] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Lianian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022. 1, 3
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from
[72] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. arXiv preprint arXiv:2301.07525, 2023.
[73] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901-10911, 2021. 7
[74] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10912-10922, 2021. 7, 10, 17
[75] Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 16
[76] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019. 16
This document provides supplementary materials for Weakly Supervised 3D Open-vocabulary Segmentation in implementation details (Appendix A), more ablations (Appendix B), more evaluations (Appendix C), and more results (Appendix D).
A.1 Model Architecture
Figure 6: Model architecture.
We use TensoRF [42] as our base NeRF architecture for efficiency, the plane size is also the same as the default setting of TensoRF. The RGB and CLIP feature branches share the same volume and use the same intermediate features. The selection volume and density volume are two other independent volumes. We directly use the features extracted from the selection volume and density volume as the selection vector and the density value, as they have low dimensions and are view-independent. We use the original MLP architecture in TensoRF to extract the view-dependent RGB value and use another MLP which discards view direction input to extract the rendered CLIP feature. The architecture is illustrated in Fig. 6.
We set  $\tau = 0.2$  to get the shaper segmentation probability distribution  $\hat{P}$ . The offset  $b$  is set to 0.7 to measure the similarities of the DINO features, meaning that two DINO features are considered similar if their cosine similarity is larger than 0.7, and different if less than 0.7. We use 3 scales of CLIP features, and the patch sizes of each scale are set as  $s/5$ ,  $s/7$ , and  $s/10$ , where  $s$  is the smaller value in the width and height of the input image  $I$ . In the ablation studies, we use  $s/7$  as the patch size of the single-scale CLIP feature input. The weights associated with similar and dissimilar DINO features in  $\mathcal{L}_{FDA}$  are set as  $\lambda_{pos} = 200$  and  $\lambda_{neg} = 0.2$  by default. In certain scenes, we find that setting  $\lambda_{neg}$  to 0.22 or 0.18 can produce better results. We use ViT-B/16 CLIP model to extract the image and text features and use version 1 dino_vitb8 model to extract the DINO features because it employs the smallest downsampling factor of 8 which is advantageous for high-precision segmentation.
A.3 Training
To reconstruct a NeRF from multiview images of a scene, we follow the same training settings as TensoRF. For segmentation training, we train the model for 15k iterations. In the first 5k iterations,
Table 3: Dataset. We list the collected 10 scenes and the corresponding text labels. The background labels are in Italic font.
Scene	Text Labels
bed	red bag, black leather shoe, banana,hand, camera, white sheet
sofa	a stack of UNO cards, a red Nintendo Switch joy-con controller,Pikachu, Gundam, Xbox wireless controller, grey sofa
lawn	red apple, New York Yankees cap, stapler,black headphone, hand soap, green lawn
room	shrilling chicken, weaving basket, rabbit,dinosaur, baseball, wood wall
bench	Portuguese egg tart, orange cat, green grape,mini offroad car, dressing doll, pebbled concrete wall, wood
table	a wooden ukulele, a beige mug, a GPU card with fans,a black Nike shoe, a Hatsune Miku statue, lime wall
office desk	the book of The Unbearable Lightness of Being, a can of red bull drink,a white keyboard, a pack of pocket tissues, desktop, blue partition
blue sofa	a bottle of perfume, sunglasses, a squirrel pig doll,a JBL bluetooth speaker, an aircon controller, blue-grey sofa
snacks	Coke Cola, orange juice drink, calculator,pitaya, Glico Picky chocolate, biscuits sticks box, desktop
covered desk	Winnie-the-Pooh, Dove body wash, gerbera,electric shaver, canned chili sauce, desktop
we freeze the shared volume and density volume, and train the selection volume and the CLIP feature branch. For the rest 10k iterations, we further finetune the shared volume and the RGB branch. We use Adam optimizer with  $betas = (0.9, 0.99)$ . The learning rates for training the volume and MLP branch are respectively set to 0.02 and  $1e - 4$ . For finetuning the volume and the MLP, the learning rates are set to  $5e - 3$  and  $5e - 5$ . We also employ a learning rate decay with a factor of 0.1.
The multi-scale pixel-level CLIP features of training views are pre-computed before training and the DINO features are computed with sampled patches on the fly during training. When computing  $\mathcal{L}_{supervision}$  and  $\mathcal{L}_{RDA}$ , we randomly sample rays with a batch size of 4096. When computing  $\mathcal{L}_{FDA}$  we randomly sample patches of size  $256 \times 256$  with a batch size of 8. We use a downsampling factor of 8 when sampling rays and a factor of 5 when sampling patches. The model is trained on an NVIDIA A5000 GPU with 24G memory for  $\sim1\mathrm{h}30\mathrm{min}$  for each scene.
A.4 Dataset
We capture 10 scenes using smartphones and use Colmap [75] to extract camera parameters for each image. We capture  $20 \sim 30$  images for each scene and the resolution of each image is  $4032 \times 3024$ . We follow the data structure of LLFF [76]. We manually annotate the segmentation maps of 5 views for each scene as the ground truth for evaluation.
We list the text labels used in our experiments in Tab. 3. Note that we sometimes choose to add a general word to describe the whole background, such as wall, desktop, following LERF [2]. The text labels in our dataset contain many long-tail classes, which can be used to fully assess the open-vocabulary capability.
Figure 7: More ablations.
Figure 8: Evaluation on human body dataset (left). Evaluation on human head dataset (right).
B More Ablations
We perform two more ablation studies on the Selection Volume and the FDA loss, as shown in Tab. 4 and Fig. 7. Without the Selection Volume, we simply average the multi-scale CLIP features rather than learning the appropriate scale. We can see that both the mIoU score and the Accuracy score are inferior to the full model. We could discard the dissimilar part  $neg\_F$  since dissimilar DINO features often impair the stability of the correlation loss. However,  $neg\_F$  encourages different segmentation probabilities for different semantic regions and it plays a crucial role for precise object boundary extraction.
C More Evaluations
We additionally perform evaluations on human body, human head, indoor datasets with low-quality images [70, 74], and a complex scene from LERF datasets [2]. We compare with the concurrent work LERF qualitatively due to the lack of labels or the defective annotations as pointed out in [4]. We also perform experiments with different text prompts. We use the same scale level number and patch sizes in all comparisons.
Human body and head. As shown in Fig. 8, our method segments more precise parts than LERF. Specifically, LERF fails to segment the "head" in the human body and the "black T-shirt" in the human head. In contrast, our method can recognize and segment these parts correctly because our designed RDA loss addresses the ambiguity of the CLIP features effectively.
Indoor scenes with low-quality images. Fig. 9 shows experiments on the indoor datasets [70, 74], where many images are unrealistically rendered with less photorealistic appearances (as indicated in [4]) and have limited spatial resolution ( $640 \times 480$  or  $1024 \times 768$ ). Due to these data constraints,
Table 4: More Ablations.
mIoU	Accuracy
w/o neg_F	76.9	92.4
w/o Selection	84.8	95.3
full model	86.2	95.8
Figure 9: Evaluation on indoor datasets with lower quality images.
Figure 10: Evaluation on a complex scene from LERF datasets (left). Evaluation on a scene with multiple instances of a same class (right).
our method sometimes confuses labels with similar appearances. However, we can see that our method still outperforms LERF by successfully segmenting more labels.
Complex scenes. Fig. 10 (left) shows the segmentation of one challenging sample from the LERF dataset, where the scene has complex geometry as well as many objects of varying sizes. It can be observed that LERF cannot segment most objects due to the ambiguities of CLIP features while our method can segment more objects correctly with more precise boundaries. Fig. 10 (right) shows a scene with multiple instances of a same class. Since instances of the same class often share similar appearance, texture, etc., they also have similar DINO features. As a result, FDA will not mistakenly segment them into different classes. The RDA loss will further help by assigning all these instances to the same text label. In the experiment, we observed that our method successfully segments all three apples into the same class with accurate boundaries.
Figure 11: Evaluation on rephrased texts.
Segmentation with different text prompts. We conduct experiments to segment scenes with different text prompts. In the experiments, we replaced the original texts with different languages (e.g., Portuguese egg tart -> Pastel de Nata), names (e.g., dressing doll -> Barbie), and actions (e.g., hand soap -> wash hand, black headphone -> listen to music). As Fig. 11 shows, with the rephrased text prompts, our method can still segment the scenes reliably. The experiments are well aligned with the quantitative experiments as shown in Tab. 5.