arXiv:2310.17504v2 [cs.CV] 19 Feb 2024

Three Pillars improving Vision Foundation Model Distillation for Lidar

Gilles Puy¹

Spyros Gidaris¹

Alexandre Boulch¹

Oriane Siméoni¹

Corentin Sautier¹,³

Patrick Pérez²*

Andrei Bursuc¹

Renaud Marlet¹,³

¹valeo.ai, Paris, France

²Kyutai, Paris, France

³LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

**Abstract**

Self-supervised image backbones can be used to address complex 2D tasks (e.g., semantic segmentation, object discovery) very efficiently and with little or no downstream supervision. Ideally, 3D backbones for lidar should be able to inherit these properties after distillation of these powerful 2D features. The most recent methods for image-to-lidar distillation on autonomous driving data show promising results, obtained thanks to distillation methods that keep improving. Yet, we still notice a large performance gap when measuring the quality of distilled and fully supervised features by linear probing. In this work, instead of focusing only on the distillation method, we study the effect of three pillars for distillation: the 3D backbone, the pretrained 2D backbones, and the pretraining dataset. In particular, thanks to our scalable distillation method named ScaLR, we show that scaling the 2D and 3D backbones and pretraining on diverse datasets leads to a substantial improvement of the feature quality. This allows us to significantly reduce the gap between the quality of distilled and fully-supervised 3D features, and to improve the robustness of the pretrained backbones to domain gaps and perturbations. The code is available at https://github.com/valeoai/ScaLR.

**1. Introduction**

Lidars capture the 3D geometry of a scene with high accuracy while being little sensitive to adverse light conditions, which is useful for advanced driver assistance systems. However, annotating lidar point clouds to train deep neural networks is notoriously long and expensive [5].

More frugal learning can be achieved by pretraining backbone networks with self-supervision on a pretext task [8, 9, 21, 23, 24] and a non-annotated dataset. The pretrained network can then be finetuned on various down-

stream tasks and other datasets, with much less supervision for the same level of performance, or with a higher performance than when trained from scratch. Such pretraining has been particularly successful for image backbones [9, 12, 21]. In particular, the gap between supervised and self-supervised representations, as evaluated by linear probing, has been closed on ImageNet [8, 9, 12, 20, 21, 23].

On autonomous driving (AD) data, we distinguish mainly two categories of self-supervised methods for 3D backbones: (1) methods leveraging only lidar data and defining a pretext task at the level of a single [6, 43, 77] or multiple scans [44, 58, 63, 66], and (2) methods exploiting images acquired in synchronization with the point clouds and distilling self-supervised image representations to a 3D backbone [37, 38, 40, 57]. Our focus here is on the second category of methods, thanks to which 3D backbones should be able to inherit the powerful properties of self-supervised image backbones.

Image-to-lidar distillation has improved a lot recently, in particular by designing better distillation losses. Nevertheless, we still observe a large gap between distilled and supervised 3D representations when directly measuring their quality by linear probing: respectively 45.0% and 74.7% mIoU in linear probing on the validation of nuScenes in [38] for a MinkUNet [14] with cylindrical voxels [80].

In this work, in order to improve the quality of distilled features, we explore the effect of three other pillars, instead of focusing only on the distillation method. These pillars are: (i) the 3D backbone, (ii) the pretrained 2D backbones and (iii) the pretraining dataset. In particular, we show that scaling the 2D and 3D backbones and pretraining on diverse datasets leads to a considerable improvements of the feature quality. The role of these pillars is actually more important than the distillation method itself, which we propose to simplify for easier scaling.

After proposing and studying a scalable distillation method, which we call ScaLR for Scalabale Lidar Representation, we make the following contributions.

First, we are able to significantly reduce the gap between

<footnote>*Work done at valeo.ai.</footnote>
<figure><img src="image_1.png" /><figcaption>Figure 1. **Correlation properties of distilled 3D features.** Correlation maps with a point located on a car on four different scenes extracted from nuScenes [7], SemanticKITTI [5], Pandar64 and PandarGT [65], respectively. The features used to compute these maps are extracted from a *single* pretrained backbone on all four datasets with ScaLR. Color goes from blue to red for low and high values.</figcaption></figure>

distilled and supervised lidar representations. We reach an mIoU of 67.8% in linear probing on the validation set of nuScenes with a WaffleIron-48-768 backbone [50], i.e., an increase of 22.8 points compared to the score obtained with the best distillation method [38]. We recall that the best<fnref n="1" /> reported mIoU on the validation of nuScenes obtained under supervision is 78.4% [32], to the best of our knowledge. We are thus only about 10 points away from this upper bound.

Second, we show that it is possible to pretrain a *single* backbone on a mixture of datasets, performing similarly or better than different backbones specialized on each dataset individually. The capacity of this backbone in providing good features across multiple datasets in illustrated in Fig. 1. For each scene in this figure, we pick a point located on a car and present the feature correlation map with respect to this point. We notice that the most correlated points also belong to cars on all datasets, illustrating the capacity of our single pretrained backbone to correctly distinguish objects on multiple datasets.

Third, we thoroughly study the properties of our distilled features. We show that they are robust to both domain gaps and perturbations, actually leading to new state-of-the-art results on the benchmark of Robo3D [30]. We also show that pretraining on diverse datasets improves the robustness.

Finally, we show that a possible way to get even better features is to distill the knowledge from multiple vision foundation models at the same time, which can be easily done with our scalable distillation strategy.

## 2. Related work

**Pretraining 2D backbones.** Supervised pretraining of 2D networks [19, 22, 31] with large-size data [16] produces powerful image representations that can transfer well to various downstream tasks in the image domain. Alternatively, self-supervised pretraining has been proven able to rival or surpass supervised pretraining while using only raw unlabeled image data. Two prominent approaches conduct self-supervised pretraining via either discriminative tasks

that train a 2D network to extract features invariant to augmentations [9, 11, 21, 23], or via masked image modeling tasks [4, 24] that hide part of an image and then train a network to reconstruct the missing fragment. An emerging pretraining paradigm is via language-image contrastive learning (CLIP) [52], in which the 2D network is trained with a text network to match image data with their paired captions.

**Pretraining 3D backbones.** Several self-supervised techniques for training 3D backbones have appeared recently. While the first techniques were often limited to dense scans of single objects [13, 49, 56, 77], contrastive self-supervision has enabled significant improvements in self-supervision on large indoor and outdoor datasets [35, 43, 44, 66, 70, 77]. Instead of working on a single scan, some works also take advantage of the temporal dimension to construct their contrastive pretext tasks [27, 44, 58, 63]. Finally, other strategies consist in reconstructing missing information hidden either artificially [25, 42, 46, 73, 74], or intrinsically because of the sparse point sampling in lidar acquisitions [6].

**Using 2D pretrained features to train 3D backbones.** Exploiting pretrained image backbones to train 3D networks has drawn a lot of attention recently. For example, in [17, 76], 3D autoencoders are pretrained on object-level point clouds where features extracted from a 2D backbone are used to build the supervision signal. Several methods [37, 38, 40, 57] distill the knowledge of 2D self-supervised backbones on large indoor or outdoor point clouds thanks to a contrastive loss that aligns 2D and 3D representations for pairs of corresponding point and pixel, while making the representation as dissimilar as possible for non matching point-pixel pairs. The same principle can also be applied for object-level point cloud [1]. Another type of strategy consists in pseudo-labeling 3D point clouds with class labels provided by 2D backbones trained under supervision [72]. Recently, several methods, e.g., [10, 48, 75], proposed to distill the knowledge of supervised vision-language models for open-vocabulary recognition or semantic segmentation. In addition to distillation, these methods also need to preserve alignment with the text representations. While our

<footnote n="1">¹without test time augmentation.</footnote>
focus is not on open-vocabulary semantic segmentation, it is probable that some recipes discovered in this work can improve the distillation of vision-language models as well.

**Robustness.** Making sure that trained models are robust to various type of perturbations is important for safety-critical applications, such as advanced driver assistance systems. Recently, several works [2, 18, 34, 68, 71] have proposed ways to evaluate or improve robustness of 3D perception models. In this work, we evaluate the robustness of our pretrained models on the Robo3D benchmark [30], which considers eight corruption types of different intensities meant to mimic different cases and degrees of distribution shifts from the original training distribution. As mentioned earlier, we show that distilling 2D features on data collected from multiple different lidars improves the robustness of our semantic segmentation models.

**Domain generalization.** 3D backbones trained on lidar data are sensitive to domain shifts such as the one induced by the different lidar sensors. Several techniques have thus been developed to improve the generalization capabilities of 3D models, e.g., [29, 54, 55, 64]. Alternatively, some techniques adapt these models to a new target domain using non-annotated target lidar data only, e.g., [41, 53, 69], or with the help of image data, e.g., [28, 36, 47]. While our method is not a domain generalization or adaptation technique per se, we consider the protocol used in this related literature to evaluate the capacity of our pretrained backbones to generalize to different lidar sensors.

# 3. Scalable Distillation Strategy

## 3.1. Motivation and Principle

Our goal is to study the benefit of using high-capacity 2D and 3D backbones, and of mixing data acquired by multiple lidars. To facilitate this study, it is important to use a distillation strategy that is scalable, with no or few hyperparameters to tune on each dataset, while remaining competitive.

While the state-of-the-art methods [37, 38, 40, 57] use a contrastive loss defined at the level of semantically coherent segments, in this work we relate pixel and point features directly using a simple cosine similarity-based loss, whose usage actually appears naturally when distilling CLIP features [48]. This loss has the advantage to have no hyperparameters and, as we will see later, is very competitive without the cost of pre-computing segments.

Finally, AD datasets often contain images captured by multiple cameras synchronized with the lidar. The best practice for distillation is, for each lidar scan, to load all available synchronized images during batch loading. Nevertheless, the number of cameras varies from one dataset to the other, which creates some implementation difficulties when pretraining on a mix of different datasets, as in this work. Instead, we propose to have batches where each

sample is created as follows: one camera is selected at random and we only load the corresponding image and points viewed in this camera. Therefore, data loading becomes strictly identical whatever the number of cameras available in the pretraining dataset, which makes multi-datasets pretraining easier to implement. Furthermore, this strategy saves memory, making it easier to train large backbones.

## 3.2. Formal description

**Backbones.** The 3D backbone $\phi_{3D}$ takes as input a point cloud $(p_1, \dots, p_N) \in \mathbb{R}^{N \times (3+1)}$, where each point $p_i$ holds three Cartesian coordinates and the laser return intensity. It outputs deep features of dimension $F_{3D}$ for all input points: $(f_1, \dots, f_N) \in \mathbb{R}^{N \times F_{3D}}$. Similarly, the 2D backbone $\phi_{2D}$ takes as input an RGB image of $M$ pixels $(u_1, \dots, u_M) \in \mathbb{R}^{M \times 3}$ and outputs deep features of dimension $F_{2D}$ for all pixels: $(g_1, \dots, g_M) \in \mathbb{R}^{M \times F_{2D}}$. In practice, we use bilinear interpolation to increase the resolution of the 2D feature map to the resolution of the input image.

**3D Projection head.** A linear projection head is added at the end of the 3D networks. We denote this projection head by $\psi_{3D}$. At distillation time, it projects 3D features into the output space of the 2D backbone. The projection head $\psi_{3D}$ is removed after distillation and replaced by another one dedicated for the task of interest. We denote $(\tilde{f}_1, \dots, \tilde{f}_N) \in \mathbb{R}^{N \times F_{2D}}$ the output of $\psi_{3D} \circ \phi_{3D}$.

**Point-pixel mapping.** We assume the lidar and the camera (selected at random among all available cameras during batch creation) are calibrated so that we can put in correspondence 3D points and pixels. We can thus compute a point-to-pixel mapping $\rho: \{1, \dots, N\} \to \{-1\} \cup \{1, \dots, M\}$. We are then able to retrieve the pixel feature $g_{\rho(i)}$ associated to each point $p_i$. By convention, $\rho(i) = -1$ means that the point $p_i$ is not viewed in the camera.

**Similarity loss.** In our experiments, we use the following loss, which relies on pointwise cosine similarity

$$ \mathcal{L}_{\text{sim}} = \frac{1}{|\mathcal{V}|} \sum_{i \in \mathcal{V}} \| \tilde{f}_i - g_{\rho(i)} \|_2, \quad (1) $$

where $\tilde{f}_i$ and $g_j$ have been $l_2$-normalized beforehand, and $\mathcal{V}$ is the subset of all visible points in the camera, i.e., such that $\rho(i) \neq -1$.

## 3.3. Analysis of our Distillation Strategy

In this section, we conduct an analysis of different pretraining options to validate our choice of distillation strategy.

We conduct our experiments on 2 backbones: the MinkUNet [14] with cylindrical voxels [81] used in [38, 40, 57] and WaffleIron-48-256 [50] (48 layers with 256-dim. features). We perform our study on nuScenes [7]. We split the original 700 training scenes in a mini-train set and a mini-val set of 600 and 100 scenes, respectively (see
<table><caption>Table 1. **Analysis of distillation losses.** Effect on the linear probing performance (mIoU), memory usage, training speed of the distillation loss and number of images loaded per scan. We show relative gain in memory and training time with respect to the first row (lower value = less consumption). We report the standard deviation between parentheses evaluated over 3 different pretrainings for some selected experiments.</caption><thead><tr><th></th><th>2D head</th><th>Con.</th><th>Cos.</th><th>No. cam.</th><th>Mem. / Time ↓</th><th>mIoU% ↑</th></tr></thead><tbody><tr><td rowspan="4">MinkUNet</td><td>✓</td><td>✓</td><td>-</td><td>6</td><td>1.0 / 1.0</td><td>39.3</td></tr><tr><td>X</td><td>✓</td><td>-</td><td>6</td><td>1.6 / 1.0</td><td>43.4</td></tr><tr><td>X</td><td>-</td><td>✓</td><td>6</td><td>1.6 / 1.1</td><td>43.6</td></tr><tr><td>X</td><td>-</td><td>✓</td><td>1</td><td>0.8 / 0.3</td><td>42.1</td></tr><tr><td rowspan="3">WI-48-256</td><td>✓</td><td>✓</td><td>-</td><td>1</td><td>-</td><td>50.0 (±0.8)</td></tr><tr><td>X</td><td>✓</td><td>-</td><td>1</td><td>-</td><td>54.1</td></tr><tr><td>X</td><td>-</td><td>✓</td><td>1</td><td>-</td><td>56.7 (±0.9)</td></tr></tbody></table>

[57] for details). All backbones are pretrained and linearly probed on the mini-train set and the results are reported on the mini-val set. For all settings, we distill features obtained from the DINO-pretrained ViT-S/8 [9].

The results are presented in Tab. 1. We start from the baseline [37] which uses a contrastive loss and a 2D projection head on top of the 2D backbone. We then remove this 2D projection head and finally replace the loss by ours (Eq. 1). Note that the use of a 2D projection head is not compatible with (1) as an optimal but degenerated solution would be obtained by letting the parameters of the 2D and 3D projection heads go to zero. Finally, we measure the effect of loading only one image instead of six per batch. Based on the results of Tab. 1, we notice that our distillation strategy performs well thanks to the following advantages.

**Preservation of the 2D feature space.** The results in Tab. 1 show a clear detriment of the 2D projection for this unidirectional image-to-lidar distillation. The absence of 2D projection head preserves the structure of the 2D feature space: there is no loss of information on the 2D side anymore. Leveraging as much as possible the original 2D features, obtained after pretraining on very large image datasets, is thus key to produce good 3D features.

**Absence of false negatives.** Even in absence of 2D projection head, the loss (1) performs better than the contrastive loss. We believe that this can be explained by the presence of false negatives in the contrastive loss. Indeed, with the cosine similarity loss, a feature $f_i$ of a 3D point $p_i$ sampled, e.g., on a car will be made as similar as possible to the corresponding 2D feature $g_{\rho(i)}$ falling at the same location on the car. However, with the contrastive loss, an additional mechanism comes into play: the feature $f_i$ will also be made as dissimilar as possible to all other 2D features $g_{\rho(j)}$, $j \neq i$, even when these 2D features are originally similar, e.g., when they fall on the same object. This

known drawback of the contrastive loss seems to harm its performance significantly.

**Scalability to large point-pixel pairs.** A drawback of the contrastive loss is that it cannot be computed using all possible point-pixel pairs because of memory constraints. One needs to subsample these pairs, e.g., randomly but then increasing the risk of ignoring small objects in the loss. The effects of this sub-optimality have been reduced by constructing the loss at the level of semantically coherent segments in [38, 40, 57], but at the cost of extra pre-processing time and additional parameters to tune. This is a cost we wish to avoid to be able to pretrain more easily on multiple datasets. The cosine similarity loss does not require selecting point-pixel pairs or pre-extracting image/lidar segments.

**Reduced memory usage.** nuScenes provides images acquired from six different cameras, offering a 360° view of the scene. To reduce the memory requirement, we propose to use only one image (chosen at random among all available cameras) and only the points viewed in this camera when loading one element of a batch. Our result shows a slight drop of performance when batching with single cameras, but a significant gain in speed and memory. As explained in Sec. 3.1, this change has several benefits. First, the savings in memory and compute time make it easier to train large backbones, which, as we will see, is essential to reach good performance. Second, the data loading process becomes strictly identical whatever the number of cameras available in the pretraining dataset, which makes multi-datasets pretraining easier to implement.

To summarize, our distillation strategy does not suffer from the presence of false negatives, does not require a selection of point-pixel pairs, does not need extra pre-processing time (e.g., extracting segments), does not introduce extra hyperparameters to tune (such as the temperature in the contrastive loss or number of segments), and saves compute time and memory.

## 4. Experiments

In this section, we show that scaling the 2D and 3D backbones and pretraining on diverse datasets leads to a substantial improvement of the feature quality. First, we describe the backbones and datasets that we use. Second, we show that our scalable distillation strategy is competitive compared to other state-of-the-art methods. Third, we scale the 2D and 3D backbones and show that it results in distilled features of much better quality. Fourth, we pretrain a single backbone on multiple datasets captured by four different lidars and demonstrate that it performs as well as, or better than, backbones specialized to each lidar. Fifth, we evaluate the downstream finetuning performance, robustness and generalization properties of our pretrained backbones. Finally, we show that our method allows an easy combination of multiple 2D teachers to further boost the performance.
## 4.1. Three Pillars

**3D backbones.** We experiment with two very different architectures designed for 3D point clouds: MinkUNet [14] with cylindrical voxels [81] and WaffleIron (WI) [50]. For WI, we fix the depth to 48, as it led to the best results in [50], and vary the feature size: $F_{3D} \in \{96, 256, 384, 768\}$. We denote the corresponding backbone by $WI-F_{3D}$. Note that we noticed numerical instabilities during training at $F_{3D} = 768$, which we solved by replacing all batchnorms by layernorms (except in the embedding and classification layers) at this feature size.

**2D backbones.** We concentrate on the distillation of self-supervised ViT features obtained with DINO [9], DINOv2 [45] or MAE [24]. Nevertheless, we also conduct one experiment with the MaskCLIP ViT-B/16 [78] to show the benefit of using multiple 2D teachers.

**Pretraining datasets and lidars.** We conduct experiments by pretraining on data collected from four different lidars: Velodyne-32 of nuScenes [7], Velodyne-64 of SemanticKITTI [5], Pandar 64 and Pandar GT of PandaSet [65]. On nuScenes and SemanticKITTI, we use the official train/val split and list of classes. We detail the train/val split and list of classes we used on PandaSet in the supplementary material as we noticed different practices on this dataset in the literature.

## 4.2. Implementation Details

For all experiments conducted with MinkUNet, we use the implementation provided by [57] and the same hyperparameters, unless otherwise stated. We describe below the main implementation details used to pretrain, linear probe and finetune WI backbones. More information, such as learning rate schedules, weight decay, number of epochs are provided in the supplementary material.

**Data augmentation.** We do not apply any image augmentation, beyond systematic resizing to 224 × 448. During pretraining, finetuning and linear probing, we apply the following standard point cloud augmentations: random rotation around the z-axis, random flip of the x and y axes. During finetuning and linear/MLP probing, we also globally scale the coordinates by a random factor chosen uniformly in [0.9, 1.1].

**Linear probing.** We remove the projection layer $\psi_{3D}$ and replace it by a batch normalization layer directly followed by a linear classification layer. We denote the combination of these last layers by $\kappa$. Note that the combination of batch normalization followed by linear classification acts as a linear layer at inference. The batch normalization layer makes the results less sensitive to the choice of downstream learning rate [33]. While maintaining $\phi_{3D}$ fixed, we train $\kappa \circ \phi_{3D}$ using ground-truth labels.

<table><caption>Table 2. Comparison of different pretraining methods and effect of 2D/3D backbones. The mIoU reported in italic are found in the literature while the others are our production. All methods are pretrained on the same 600 scenes of nuScenes. Linear probing is done using the 700 training scenes of nuScenes. Finetuning is done using only 1% of nuScenes training dataset. All scores are reported on the official validation split.</caption><thead><tr><th>3D Back.</th><th>Method</th><th>2D Back.</th><th>Lin. Prob.</th><th>1%</th></tr></thead><tbody><tr><td rowspan="4">MinkUNet</td><td>PPKT [37]</td><td>ResNet-50</td><td>36.4</td><td>37.5</td></tr><tr><td>SLidR [57]</td><td>ResNet-50</td><td>38.8</td><td>39.0</td></tr><tr><td>ST-SLidR [40]</td><td>ResNet-50</td><td>40.5</td><td>40.8</td></tr><tr><td>Seal [38]</td><td>ResNet-50</td><td>45.0</td><td>45.8</td></tr><tr><td rowspan="3">MinkUNet</td><td>PPKT [37]</td><td>ViT-S/8</td><td>38.6</td><td>40.6</td></tr><tr><td>SLidR [57]</td><td>ViT-S/8</td><td>39.3</td><td>39.0</td></tr><tr><td><b>ScaLR</b> (ours)</td><td>ViT-S/8</td><td>42.4</td><td>40.5</td></tr><tr><td>WI-96</td><td><b>ScaLR</b> (ours)</td><td>ViT-S/8</td><td>46.8</td><td>38.8</td></tr><tr><td>WI-256</td><td><b>ScaLR</b> (ours)</td><td>ViT-S/8</td><td>54.2</td><td>41.4</td></tr></tbody></table>

**Finetuning.** For finetuning, we remove the distillation layer $\psi_{3D}$ and replace it by a batch normalization layer directly followed by a linear classification layer. As before, we denote the combination of these last layers by $\kappa$ and we train $\kappa \circ \phi_{3D}$ using ground-truth labels. The backbone is finetuned using a layer-wise learning rate decay [3, 15], a technique commonly used when finetuning pretrained ViT backbones [4, 24, 79].

## 4.3. Comparison of Distillation Methods

In this section, we verify that our distillation strategy remains competitive with respect to the state-of-the-art in image-to-lidar distillation. All experiments are conducted using the experimental protocol used in [38, 40, 57], i.e., pretraining on the reduced training set of nuScenes (600 scenes) defined in [57] and using the official train/val split for downstream linear probing and finetuning.

We start by distilling the representations of DINO ViT-S/8 in a MinkUNet. These representations are distilled using three different methods: PPKT [37], SLidR [57] and using our cosine-based similarity loss (but still loading all six images as in PPKT and SLidR). The MinkUNet backbones are pretrained with a batch size of 12 (we scale the learning rate proposed in [57] accordingly) during 25 epochs. The quality of the distilled features is assessed by linear probing. We also finetune the backbones using 1% of the complete training set of nuScenes. We report the corresponding scores in Tab. 2, where we also present the results obtained in the literature by distilling ResNet-50 representations.

First, we notice that changing the 2D backbones from ResNet-50 to ViT-S/8 helps both PPKT and SLidR, which both reach a higher mIoU in linear probing and finetuning after this change. The improvement seems however more
<table><caption>Table 3. **Effect of the 3D network width and of the 2D features' quality.** We report the mIoU% obtained by linearly probing distilled features. The networks are trained on nuScenes or SemanticKITTI using ScaLR. The scores are reported on the corresponding official validation split. The column Δ shows the gap with respect to the best fully supervised mIoU% reached with WI on the corresponding dataset.</caption><thead><tr><th rowspan="2">3D Back.</th><th rowspan="2">2D Back.</th><th colspan="2">nuScenes</th><th colspan="2">SemKITTI</th></tr><tr><th>LP</th><th>Δ↓</th><th>LP</th><th>Δ↓</th></tr></thead><tbody><tr><td colspan="6">No pretraining. Best fully supervised WI.</td></tr><tr><td>WI-F<sub>3D</sub></td><td>-</td><td>78.7 (F<sub>3D</sub> = 768)</td><td>-</td><td>65.1 (F<sub>3D</sub> = 256)</td><td>-</td></tr><tr><td colspan="6">MAE-pretrained</td></tr><tr><td>WI-256</td><td>ViT-B/16</td><td>47.8</td><td>30.9</td><td>-</td><td>-</td></tr><tr><td colspan="6">DINO-pretrained</td></tr><tr><td>WI-256</td><td>ViT-S/8</td><td>54.4</td><td>24.3</td><td>-</td><td>-</td></tr><tr><td>WI-384</td><td>ViT-S/8</td><td>57.8</td><td>20.9</td><td>-</td><td>-</td></tr><tr><td>WI-768</td><td>ViT-S/8</td><td>61.1</td><td>17.6</td><td>-</td><td>-</td></tr><tr><td colspan="6">DINOv2-pretrained</td></tr><tr><td>WI-256</td><td>ViT-S/14</td><td>57.7</td><td>21.0</td><td>-</td><td>-</td></tr><tr><td>WI-256</td><td>ViT-B/14</td><td>60.2</td><td>18.5</td><td>-</td><td>-</td></tr><tr><td>WI-256</td><td>ViT-L/14</td><td>61.6</td><td>17.1</td><td>48.3</td><td>16.8</td></tr><tr><td>WI-384</td><td>ViT-L/14</td><td>63.9</td><td>14.8</td><td>50.6</td><td>14.5</td></tr><tr><td>WI-768</td><td>ViT-L/14</td><td><strong>66.8</strong></td><td><strong>11.9</strong></td><td><strong>51.1</strong></td><td><strong>14.0</strong></td></tr></tbody></table>

significant for PPKT than for SLidR.

Second, we remark that **ScaLR leads to better 3D features than those obtained with PPKT and SLidR**: we reach a higher score in linear probing; we also obtain a better result in finetuning with 1% of the training data than SLidR and a level of performance similar to PPKT. This confirms that using the simple cosine-based similarity loss of Eq. (1) is a competitive choice.

Third, we also pretrained WI-96 and WI-256 backbones on this dataset using our full scalable distillation protocol. We notice that **changing MinkUNet to WI further boosts linear probing performance**. We remark that WI-96 has the same output feature dimension than the MinkUNet backbone but fewer trainable parameters (1M vs 35M). Hence, the improvement of performance in linear probing cannot be solely explained by the 3D feature dimension or the capacity of the backbone; other elements in the architecture must come into play but identifying the exact origin of this improvement is beyond the scope of this work.

Fourth, when finetuning on 1% of nuScenes training set, **our ScaLR WI-256 model surpasses prior methods**. The only exception is Seal [38] that leverages a supervised backbone (segment-everything-everywhere model [82]) to create segments. It is probable that we could reach even higher performance by using such extra supervision. Yet, we leave this research direction for future work as our scalable distillation strategy is already competitive enough for the purpose of our study.

<table><caption>Table 4. **Benefit of pretraining on diverse datasets – Linear probing.** Performance after linearly probing distilled features. The backbones are pretrained on nuScenes, SemanticKITTI, Pandar 64, Pandar GT, nuScenes & SemanticKITTI, or all these datasets together, and then linearly probed on each individual dataset. The reported mIoU% is computed on the val split of each dataset. The underlined mIoU% highlights the score obtained by pretraining and linear probing on the same dataset.</caption><thead><tr><th rowspan="2">Pretrain.<br>Dataset</th><th rowspan="2"></th><th colspan="4">Downstream & Test Dataset</th></tr><tr><th>nuScenes</th><th>SemKITTI</th><th>Pandar 64</th><th>Pandar GT</th></tr></thead><tbody><tr><td rowspan="2">WI-F<sub>3D</sub></td><td colspan="5"><em>No Pretraining - Best fully supervised WI</em></td></tr><tr><td>-</td><td>78.7</td><td>65.1</td><td>47.8</td><td>40.6</td></tr><tr><td></td><td></td><td colspan="4">F<sub>3D</sub> = 768 F<sub>3D</sub> = 256 F<sub>3D</sub> = 768 F<sub>3D</sub> = 256</td></tr><tr><td colspan="6"><em>Pretraining with DINO-ViT-S/8 and linear probing</em></td></tr><tr><td rowspan="5">WI-256</td><td>nuScenes</td><td><strong>54.4</strong></td><td>28.8</td><td>26.9</td><td>25.2</td></tr><tr><td>KITTI</td><td>39.5</td><td><u>46.6</u></td><td>25.3</td><td>25.7</td></tr><tr><td>Pandar 64</td><td>39.6</td><td>25.6</td><td><u>30.0</u></td><td>24.7</td></tr><tr><td>Pandar GT</td><td>29.9</td><td>26.9</td><td>23.5</td><td><u>28.5</u></td></tr><tr><td>nuSc. & KITTI</td><td>54.4</td><td>50.1</td><td>29.3</td><td>28.9</td></tr><tr><td></td><td>All datasets</td><td><strong>54.6</strong></td><td><strong>50.6</strong></td><td><strong>33.1</strong></td><td><strong>32.3</strong></td></tr><tr><td rowspan="2">WI-768</td><td colspan="5"><em>Pretraining with DINOv2-ViT-L/14 and linear probing</em></td></tr><tr><td>nuScenes</td><td><strong>67.8</strong></td><td>43.1</td><td>33.9</td><td>29.9</td></tr><tr><td></td><td>All datasets</td><td><strong>67.8</strong></td><td><strong>55.8</strong></td><td><strong>37.9</strong></td><td><strong>34.5</strong></td></tr></tbody></table>

Finally, we already notice that distilling features in WI backbones of increasing width helps both the linear probing and finetuning performance. We continue to study this property in the next section with WI, as this is the backbone which led to the best performance with our technique.

## 4.4. 2D Backbone Choice & 3D Backbone Scale

In this section, we show that increasing the width of WI leads to substantial improvements of the feature quality. This effect combined with a good choice of 2D backbone significantly reduces the gap between the quality of distilled 3D features and fully-supervised 3D features.

We conduct experiments with three different WaffleIron backbones: WI-256, WI-384, WI-768. We pretrain these backbones on the complete training set of nuScenes or SemanticKITTI, and evaluate the quality of the distilled features on the validation set of the respective datasets. The evaluation consists of linear probings using 100% of the annotated training scans. The results obtained by distilling MAE, DINO, and DINOv2 features are presented in Tab. 3.

First, we observe on nuScenes that pretraining WI-256 backbones with ViTs pretrained with DINOv2 instead of DINO consistently improves the scores. We also notice that the linear probing performance improves when distilling DINOv2 ViTs of increasing capacities. This indicates that 3D backbones pretrained with **our method can directly benefit from an improved quality of the image features.**
<table><caption>**Table 5. Finetuning performance.** The backbones are pretrained on nuScenes alone or combined with SemanticKITTI, Pandar 64, Pandar GT ('All'), and finetuned on the train split of each dataset for different amounts of data. The reported mIoU% is computed on the val split of each dataset.</caption><thead><tr><th rowspan="2">Pretrain.<br>Dataset</th><th colspan="3">nuScenes</th><th colspan="2">SemKITTI</th><th>Pan. 64</th><th>Pan. GT</th></tr><tr><th>1%</th><th>10%</th><th>100%</th><th>1%</th><th>100%</th><th>100%</th><th>100%</th></tr></thead><tbody><tr><td colspan="8" style="background-color: #f0f0f0; text-align: left; font-weight: bold;">No Pretraining - Fully supervised WI-768</td></tr><tr><td>-</td><td>35.2</td><td>62.2</td><td><strong>78.7</strong></td><td>49.6</td><td>63.4</td><td>47.8</td><td>40.0</td></tr><tr><td colspan="8" style="background-color: #f0f0f0; text-align: left; font-weight: bold;">Pretraining WI-768 with DINOv2-ViT-L/14 and finetuning</td></tr><tr><td>nuScenes</td><td><strong>51.0</strong></td><td><strong>70.5</strong></td><td>77.9</td><td>49.7</td><td>63.9</td><td><strong>49.4</strong></td><td><strong>42.1</strong></td></tr><tr><td>All</td><td>50.7</td><td>69.2</td><td>78.4</td><td><strong>56.8</strong></td><td><strong>65.8</strong></td><td>48.3</td><td>41.1</td></tr></tbody></table>

Interestingly, the MAE-pretrained ViT leads to the worst performance. We attribute this to the fact that MAE features are less linearly separable, as reported in [24].

Second, we notice that the **quality of our distilled features increases significantly with the WI width** for ViT-S/8 and ViT-L/14 on nuScenes and SemanticKITTI. Noticeably, we reach a mIoU of 66.8 by linear probing on nuScenes. This is only about 11.9 points away from the mIoU reached by WI-768 when trained under full supervision. Note that WI-768 is actually well-performing on this dataset as it reaches a mIoU of 78.7% while the best published score, without test time augmentations, we know of on this dataset is 78.4% (reported in [32]). It shows that **distilling 2D features without manual annotations can bridge the gap with the quality of fully supervised 3D features.**

## 4.5. Pretraining on Multiple Datasets

In this section, we show that we can pretrain a single backbone on multiple datasets that performs as well or better than individual backbones specialized to each dataset.

We conduct our main experiment with WI-256 and the DINO-pretrained ViT-S/8. The datasets we consider (nuScenes, SemanticKITTI, Pandar 64 and Pandar GT) have different sizes. To avoid drawing conclusions which could be explained by different amount of pretraining, we adjust the number of pretraining epochs on each dataset so that the backbones are pretrained for the same number of iterations (the batch size is fixed to 16).

We start by pretraining WI-256 on each dataset individually and evaluate the quality of the distilled features by linear probing on each of the datasets. The results are presented in Tab. 4. We notice that the pretrained backbones are quite sensitive to the domain gap induced by different lidars. For example, the backbone pretrained on nuScenes performs significantly worse on SemanticKITTI than the backbone pretrained on SemanticKITTI: 28.8% vs 46.6% in mIoU. The observation is valid whatever

<table><caption>**Table 6. Robustness to corruptions.** The evaluation is done on nuScenes-C from the Robo3D benchmark [30]. We report the mCE%, mRR%, and the mIoU% averaged over all eight corruptions. The scores in italic are obtained from [30, 38]. PPKT, SLidR and Seal use a MoCov2 ResNet-50. We use DINOv2 ViT-L/14.</caption><thead><tr><th>Method</th><th>3D Back.</th><th>Pretrain dataset</th><th>mCE% ↓</th><th>mRR% ↑</th><th>Avg. mIoU%</th></tr></thead><tbody><tr><td>-</td><td>Cylind3D [80]</td><td>-</td><td>105.6</td><td>78.1</td><td>57.4</td></tr><tr><td>-</td><td>2DPASS [67]</td><td>-</td><td>98.6</td><td>75.2</td><td>58.6</td></tr><tr><td>-</td><td>SPVCNN [61]</td><td>-</td><td>97.5</td><td>75.1</td><td>57.5</td></tr><tr><td>-</td><td>GFNet [51]</td><td>-</td><td>92.6</td><td>83.3</td><td>64.0</td></tr><tr><td>-</td><td>WI-768</td><td>-</td><td>90.9</td><td>80.6</td><td>63.5</td></tr><tr><td>PPKT</td><td>MinkUNet</td><td>nuScenes</td><td>105.6</td><td>76.1</td><td>56.6</td></tr><tr><td>SLidR</td><td>MinkUNet</td><td>nuScenes</td><td>106.1</td><td>76.0</td><td>56.8</td></tr><tr><td>Seal</td><td>MinkUNet</td><td>nuScenes</td><td>92.6</td><td>83.1</td><td>62.8</td></tr><tr><td><strong>ScaLR</strong></td><td>WI-768</td><td>nuScenes</td><td>89.1</td><td>83.7</td><td>65.2</td></tr><tr><td><strong>ScaLR</strong></td><td>WI-768</td><td>Multiple</td><td><strong>87.4</strong></td><td><strong>83.8</strong></td><td><strong>65.7</strong></td></tr></tbody></table>

pairs of pretraining/downstream datasets we take. However, by pretraining a backbone on both nuScenes and SemanticKITTI, we do not notice any drop in linear probing performance on nuScenes and achieve even better results on SemanticKITTI. The best overall performance is obtained when pretraining on all datasets together. This experiment shows that a **single backbone easily benefits from diverse lidar data without performance loss in linear probing.**

We conduct one more experiment using the best combination of 3D and 2D backbone discovered in the last section: WI-768 and DINOv2 ViT-L/14. We pretrain this backbone either on nuScenes alone or the combination of all considered datasets. Due to memory constraints at this scale when training on all datasets, we reduce the batch size to 8 for both pretrainings. We thus also used a longer training schedule. The results are presented in the last section of Tab. 4. We notice again that **pretraining on multiple datasets leads to better linear probing performance across all downstream datasets.** We also remark that reducing the batch size and training for longer allowed to gain 1 mIoU point when training only on nuScenes (see WI-768 in Tab. 3 and Tab. 4). The remaining gap with respect to the best fully-supervised WI baseline is less than 10.9 mIoU points on all datasets. Notably, on nuScenes, we reach 67.8 in mIoU while the best reported score so far in linear probing was 45.0 (see Tab. 8), reached by [38] with the help of fully supervised 2D backbones to create semantically coherent segments.

## 4.6. Properties of Distilled Features

In this section, we use the WI-768 backbones pretrained with ScaLR using DINOv2 ViT-L/14. We consider two pretraining datasets: (1) nuScenes alone and (2) the combination of nuScenes, SemanticKITTI, Pandar 64/GT (denoted
<table><caption>Table 7. Domain generalization properties (left) and benefit of pretraining with multiple teachers (right). Left: mIoU% on SemanticKITTI (abbreviated KITTI) and SemanticPOSS (abbreviated POSS) of WI-768 pretrained either on nuScenes alone or on all considered pretraining datasets, and then finetuned on nuScenes. The performance of WI-768 trained under full supervision on nuScenes without pretraining is also reported. Right: Linear probing mIoU% obtained on nuScenes after pretraining WI with ScaLR on nuScenes. The considered 2D backbones are DINOv2 ViT-L/14 and MaskCLIP ViT-B/16.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">KITTI</th><th rowspan="2">POSS</th><th>3D Back</th><th colspan="2">2D Back</th><th rowspan="2">LP</th></tr><tr><th>MaskCLIP</th><th>DINOv2</th></tr></thead><tbody><tr><td rowspan="3">Pretrain Dataset</td><td rowspan="3">28.6</td><td rowspan="3">54.0</td><td>WI-256</td><td>✓</td><td>✗</td><td>59.4</td></tr><tr><td>WI-256</td><td>✗</td><td>✓</td><td>61.6</td></tr><tr><td>WI-256</td><td>✓</td><td>✓</td><td>62.5</td></tr><tr><td>nuScenes</td><td>29.6</td><td>57.1</td><td>WI-384</td><td>✓</td><td>✗</td><td>60.9</td></tr><tr><td rowspan="2">Multiple</td><td rowspan="2">36.6</td><td rowspan="2">59.1</td><td>WI-384</td><td>✗</td><td>✓</td><td>63.9</td></tr><tr><td>WI-384</td><td>✓</td><td>✓</td><td>65.1</td></tr></tbody></table>

by 'Multiple' in pretraining dataset column of the tables).

**Performance in finetuning.** We present in Tab. 5 the results obtained by finetuning our pretrained WI-768 on different datasets for different amounts of data. First, we notice that **pretraining always improves the score compared to the non-pretrained fully supervised baseline**, except on nuScenes when using all the available annotations. We also notice that WI-768 pretrained on multiple datasets: (1) significantly outperforms the nuScenes-pretrained WI-768 when finetuning on SemanticKITTI; (2) performs nearly as well on nuScenes as the model nuScenes-pretrained WI-768, i.e., the model specialized to this dataset. Interestingly, WI-768 pretrained on nuScenes seems to have a small advantage for finetuning on Pandar. We recall nevertheless that the model pretrained on multiple datasets still improves the performance compared to no pretraining. More importantly, as will be seen in the next experiment, **multi-dataset pretraining produces more robust backbones against perturbations and domain gaps**.

**Robustness to corruptions.** We evaluate the robustness of the pretrained WI-768 to corruptions on the Robo3D benchmark [30]. The results are presented in Tab. 6. We notice that WI-768 trained under full supervision on nuScenes, without any pretraining, already performs well compared to the other 3D backbones trained in the same condition. WI-768 achieves the best results in terms of mean corruption error (mCE) and is second in terms of mean resilience rate (mRR). Then, we remark that pretraining always improves the robustness of WI-768. Finally, the best performance is achieved thanks to our multiple datasets pretraining with an mCE of 87.4% and a mRR of 83.8%, i.e., new state-of-the-art results on this benchmark.

<table><caption>Table 8. Progress made with ScaLR. We report the mIoU% obtained by linear probing (LP) or finetuning on different amounts of data. The scores in italic are reported from [38]. WI-768 is pretrained using DINOv2 ViT-L/14 on nuScenes, SemanticKITTI (abbreviated KITTI), Pandar 64, Pandar GT. The MinkUNet backbones are pretrained using MoCov2 ResNet-50 on nuScenes.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">3D</th><th rowspan="2">Pretrain</th><th colspan="4">nuScenes</th><th>KITTI</th></tr><tr><th>LP</th><th>1%</th><th>10%</th><th>100%</th><th>1%</th></tr></thead><tbody><tr><td>–</td><td>MinkUNet</td><td>–</td><td>8.1</td><td>30.3</td><td>56.2</td><td>74.7</td><td>39.5</td></tr><tr><td>–</td><td>WI-768</td><td>–</td><td>–</td><td>35.2</td><td>62.2</td><td><strong>78.7</strong></td><td>49.6</td></tr><tr><td>PPKT</td><td>MinkUNet</td><td>nuScenes</td><td>35.9</td><td>37.8</td><td>60.3</td><td>74.5</td><td>44.0</td></tr><tr><td>SLidR</td><td>MinkUNet</td><td>nuScenes</td><td>38.8</td><td>38.3</td><td>59.8</td><td>74.8</td><td>44.6</td></tr><tr><td>ST-SLidR</td><td>MinkUNet</td><td>nuScenes</td><td>40.5</td><td>40.8</td><td>60.8</td><td>75.1</td><td>44.7</td></tr><tr><td>Seal</td><td>MinkUNet</td><td>nuScenes</td><td>45.0</td><td>45.8</td><td>63.0</td><td>75.6</td><td>46.6</td></tr><tr><td><strong>ScaLR</strong></td><td>WI-768</td><td>Multiple</td><td><strong>67.8</strong></td><td><strong>50.7</strong></td><td><strong>69.2</strong></td><td><strong>78.4</strong></td><td><strong>56.8</strong></td></tr></tbody></table>

**Domain generalization properties.** We evaluate the capacity of our pretrained WI-768 to generalize to different lidars. The backbones are finetuned on nuScenes after pretraining. We measure the performance of the resulting models directly on the val set of SemanticKITTI (train set seen during pretraining) or SemanticPOSS (not seen during pretraining) using respectively 10 and 6 aggregated classes, as done, e.g., [41, 69]. We notice in Tab. 7 that pretraining always helps generalization and that the best performance are obtained when pretraining on multiple datasets.

## 4.7. Pretraining with multiple teachers

We show in Tab. 7 that using multiple 2D teachers is a way to further improve the performance. Distillation was done using the two modifications in ScaLR: (1) On the 2D side, we concatenate the pixel features obtained from both backbones; (2) On the 3D side, we use a 2-layer MLP for $\psi_{3D}$ with an output size matching the dimension of the concatenated pixel features. More details about this MLP are available in the supp. material.

## 5. Conclusion

We have seen that scaling 2D and 3D backbones and pretraining on multiple datasets lead to better and more robust distilled features. We summarize in Tab. 8 the scores obtained by [37, 38, 40, 57] and with ScaLR to highlight the significant progress we have made. The most notable improvement appears when linear probing and finetuning on small amounts of data. Note that we *do not* claim that the concurrent distillation methods cannot surpass our proposed scalable distillation method; the take-away message is that scaling the 2D and 3D backbones and pretraining on diverse datasets actually lead to significantly better pretrained backbones even with a simple distillation method.
These are three pillars that should not be overlooked to get good pretrained backbones for lidars. We hope that these findings, combined with more powerful distillation methods, will lead to even better 3D backbones in the future.

**Acknowledgement** This project has benefited from an access to the HPC resources of CINES under the allocation GDA2213 for the Grand Challenges AdAstra GPU made by GENCI. We thank Mathieu Cloirec and Jean-Christophe Penalva at CINES for helping us accessing the cluster and creating the good setup to run our experiments smoothly. This research also received the support of EXA4MIND project, funded by a European Union's Horizon Europe Research and Innovation Programme, under Grant Agreement N° 101092944. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT4Innovations, Czech Republic. We also acknowledge the support of the French Agence Nationale de la Recherche (ANR), under grant ANR-21-CE23-0032 (project MultiTrans).

## References

[1] Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. CrossPoint: Self-supervised cross-modal contrastive learning for 3D point cloud understanding. In CVPR, 2022. 2

[2] Fatima Albreiki, Sultan Abu Ghazal, Jean Lahoud, Rao Anwer, Hisham Cholakkal, and Fahad Khan. On the robustness of 3d object detectors. In ICM Asia, 2022. 3

[3] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv:2304.12210, 2023. 5

[4] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022. 2, 5

[5] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jürgen Gall. SemanticKITTI: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2019. 1, 2, 5

[6] Alexandre Boulch, Corentin Sautier, Björn Michele, Gilles Puy, and Renaud Marlet. ALSO: Automotive lidar self-supervision by occupancy estimation. In CVPR, 2023. 1, 2

[7] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beibom. nuScenes: A multi-modal dataset for autonomous driving. In CVPR, 2020. 2, 3, 5

[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning

of visual features by contrasting cluster assignments. In NeurIPS, 2020. 1

[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 4, 5

[10] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. CLIP2Scene: Towards label-efficient 3D scene understanding by CLIP. CVPR, 2023. 2

[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2

[12] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 1

[13] Ye Chen, Jinxian Liu, Bingbing Ni, Hang Wang, Jiancheng Yang, Ning Liu, Teng Li, and Qi Tian. Shape self-correction for unsupervised point cloud understanding. In ICCV, 2021. 2

[14] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4D spatio-temporal ConvNets: Minkowski convolutional neural networks. In CVPR, 2019. 1, 3, 5

[15] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020. 5

[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 2

[17] Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2D image transformers help 3D representation learning? ICLR, 2022. 2

[18] Yinpeng Dong, Caixin Kang, Jinlai Zhang, Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, and Jun Zhu. Benchmarking robustness of 3d object detection to common corruptions. In CVPR, 2023. 3

[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Jakob Uszkoreit, Mostafa Dehghani Neil Houlsby, Matthias Minderer, Georg Heigold, and Sylvain Gelly and. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 2

[20] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Kostodakis, Matthieu Cord, and Patrick Pérez. Obow: Online bag-of-visual-words generation for self-supervised learning. In CVPR, 2021. 1

[21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doversch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap Your Own Latent: A new approach to self-supervised learning. In NeurIPS, 2020. 1, 2

[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 1, 2

[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1, 2, 5, 7

[25] Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoder for self-supervised pre-training on lidar point clouds. In WACV, 2023. 2

[26] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, 2016. 13

[27] Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3D point clouds. In ICCV, 2021. 2

[28] Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emile Wirbel, and Patrick Pérez. xmuda: Cross-modal unsupervised domain adaptation for 3D semantic segmentation. In CVPR, 2020. 3

[29] Hyeonseong Kim, Yoonsu Kang, Changgyoon Oh, and Kuk-Jin Yoon. Single domain generalization for lidar semantic segmentation. In CVPR, 2023. 3

[30] Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3D: Towards robust and reliable 3D perception against corruptions. In ICCV, 2023. 2, 3, 7, 8, 12, 13

[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In NeurIPS, 2012. 2

[32] Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, and Jiaya Jia. Spherical transformer for lidar-based 3D recognition. In CVPR, 2023. 2, 7

[33] Jae-Hun Lee, Doyoung Yoon, ByeongMoon Ji, Kyungyul Kim, and Sangheum Hwang. Rethinking evaluation protocols of visual representations learned via self-supervised learning. arXiv:2304.03456, 2023. 5

[34] Shuangzhi Li, Zhijie Wang, Felix Juefei-Xu, Qing Guo, Xingyu Li, and Lei Ma. Common corruption robustness of point cloud detectors: Benchmark and enhancement. T-Multimedia, 2023. 3

[35] Kangcheng Liu, Aoran Xiao, Xiaoqin Zhang, Shijian Lu, and Ling Shao. FAC: 3D representation learning via foreground aware feature contrast. In CVPR, 2023. 2

[36] Wei Liu, Zhiming Luo, Yuanzheng Cai, Ying Yu, Yang Ke, José Marcato Junior, Wesley Nunes Gonçalves, and Jonathan Li. Adversarial unsupervised domain adaptation for 3D semantic segmentation with multi-modal learning. ISPRS, 2021. 3

[37] Yueh-Cheng Liu, Yu-Kai Huang, HungYueh Chiang, Hung-Ting Su, Zhe Yu Liu, Chin-Tang Chen, Ching-Yu Tseng, and Winston H. Hsu. Learning from 2D: Pixel-to-point knowledge transfer for 3D pretraining. arXiv:2104.04687, 2021. 1, 2, 3, 4, 5, 8, 13

[38] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment Any Point Cloud Sequences by Distilling Vision Foundation Models. In NeurIPS, 2023. 1, 2, 3, 4, 5, 6, 7, 8, 13

[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 12

[40] Anas Mahmoud, Jordan SK Hu, Tianshu Kuai, Ali Harakeh, Liam Paull, and Steven L Waslander. Self-supervised image-to-point distillation via semantically tolerant contrastive loss. CVPR, 2023. 1, 2, 3, 4, 5, 8

[41] Bjoern Michele, Alexandre Boulch, Gilles Puy, Tuan-Hung Vu, Renaud Marlet, and Nicolas Courty. SALUDA: Surface-based automotive lidar unsupervised domain adaptation. In 3DV, 2024. 3, 8

[42] Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai. Voxel-MAE: Masked autoencoders for pre-training large-scale point clouds. arXiv:2206.09900, 2022. 2

[43] Lucas Nunes, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. SegContrast: 3D point cloud feature representation learning through self-supervised segment discrimination. RA-L, 2022. 1, 2

[44] Lucas Nunes, Louis Wiesmann, Rodrigo Marcuzzi, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Temporal consistent 3D lidar representation learning for semantic perception in autonomous driving. In CVPR, 2023. 1, 2

[45] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. 5

[46] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV, 2022. 2

[47] Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dense feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3D semantic segmentation. In ICCV, 2021. 3

[48] Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. OpenScene: 3D scene understanding with open vocabularies. In CVPR, 2023. 2, 3

[49] Omid Poursaeed, Tianxing Jiang, Han Qiao, Nayun Xu, and Vladimir G Kim. Self-supervised learning of point clouds via orientation estimation. In 3DV, 2020. 2

[50] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Using a waffle iron for automotive point cloud semantic segmentation. In ICCV, 2023. 2, 3, 5, 12

[51] Haibo Qiu, Baosheng Yu, and Dacheng Tao. GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation. TMLR, 2022. 7, 13

[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2

[53] Cristiano Saltori, Fabio Galasso, Giuseppe Fiameni, Nicu Sebe, Elisa Ricci, and Fabio Poiesi. CoSMix: Compositional
semantic mix for domain adaptation in 3d lidar segmenta-
tion. In ECCV, 2022. 3

[54] Cristiano Saltori, Aljosa Osep, Elisa Ricci, and Laura Leal-Taixé. Walking your lidog: A journey through multiple domains for lidar semantic segmentation. In ICCV, 2023. 3

[55] Jules Sanchez, Jean-Emmanuel Deschaud, and François Goulette. Domain generalization of 3d semantic segmentation in autonomous driving. In ICCV, 2023. 3

[56] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. NeurIPS, 2019. 2

[57] Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In CVPR, 2022. 1, 2, 3, 4, 5, 8, 12, 13

[58] Corentin Sautier, Gilles Puy, Alexandre Boulch, Renaud Marlet, and Vincent Lepetit. BEVContrast: Self-supervision in bev space for automotive lidar point clouds. In 3DV, 2023. 1, 2

[59] Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonin Vobecky, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In CVPR, 2023. 12

[60] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021. 12

[61] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In ECCV, 2020. 7, 13

[62] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In CVPR, 2022. 12

[63] Yanhao Wu, Tong Zhang, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Spatiotemporal self-supervised learning for point clouds in the wild. In CVPR, 2023. 1, 2

[64] Aoran Xiao, Jiaxing Huang, Weihao Xuan, Ruijie Ren, Kangcheng Liu, Dayan Guan, Abdulmotaleb El Saddik, Shijian Lu, and Eric P. Xing. 3d semantic segmentation in the wild: Learning generalized models for adverse-condition point clouds. In CVPR, 2023. 3

[65] Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang, Yunlong Wang, and Diange Yang. PandaSet: Advanced sensor suite dataset for autonomous driving. In ITSC, 2021. 2, 5, 12

[66] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. PointContrast: Unsupervised pre-training for 3D point cloud understanding. In ECCV, 2020. 1, 2

[67] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2DPASS: 2D priors assisted semantic segmentation on lidar point clouds. In ECCV, 2022. 7, 13

[68] Xu Yan, Chaoda Zheng, Zhen Li, Shuguang Cui, and Dengxin Dai. Benchmarking the robustness of lidar semantic segmentation models. arXiv:2301.00970, 2023. 3

[69] Li Yi, Boqing Gong, and Thomas Funkhouser. Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds. In CVPR, 2021. 3, 8

[70] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Proposal-Contrast: Unsupervised pre-training for lidar-based 3D object detection. In ECCV, 2022. 2

[71] Kaicheng Yu, Tang Tao, Hongwei Xie, Zhiwei Lin, Tingting Liang, Bing Wang, Peng Chen, Dayang Hao, Yongtao Wang, and Xiaodan Liang. Benchmarking the robustness of lidar-camera fusion for 3d object detection. In CVPR, 2023. 3

[72] Ping-Chung Yu, Cheng Sun, and Min Sun. Data efficient 3D learner via knowledge transferred from 2D model. In ECCV, 2022. 2

[73] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre-training 3D point cloud transformers with masked point modeling. In CVPR, 2022. 2

[74] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-M2AE: multi-scale masked autoencoders for hierarchical point cloud pre-training. In NeurIPS, 2022. 2

[75] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xu-peng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. PointCLIP: Point cloud understanding by CLIP. In CVPR, 2022. 2

[76] Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hong-
sheng Li. Learning 3D representations from 2D pre-trained
models via image-to-point masked autoencoders. CVPR,
2023. 2

[77] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3D features on any point-cloud. In ICCV, 2021. 1, 2

[78] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In ECCV, 2022. 5

[79] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT pre-training with online tokenizer. In ICLR, 2022. 5

[80] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In CVPR, 2021. 1, 7, 13

[81] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3D convolution networks for lidar segmentation. In CVPR, 2021. 3, 5

[82] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Lin-
jie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and
Yong Jae Lee. Segment everything everywhere all at once.
arXiv:2304.06718, 2023. 6
# Three Pillars improving Vision Foundation Model Distillation for Lidar

## Supplementary Material

### A. Robustness

We present in Tab. 9 a detailed version of Tab. 6 with the mIoU% attained by for the eight different types of corrup-
tions considered in [30].

We observe that WI-768 pretrained on multiple datasets
is second or third for each type of corruption, except for
motion blur, which permits it to achieve the best mCE%
and mRR%.

### B. Training details

#### B.1. Pandaset

Pandaset [65] is made of scans acquired in San Francisco and along El Camino Real from Palo Alto to San Mateo. We use the scans collected in San Francisco as train set and the rest as validation set. We also separate the scans collected with the Pandar64 from the scans collected with the PandarGT and treat them as different datasets. Finally, we merge the fine-grained original classes into 17 classes similar to those used in nuScenes and SemanticKITTI. These classes are: road, traffic sign, barrier, pedestrian, vegetation, road marking, sidewalk, manmade, traffic cone, car, motorcycle, truck, bus, bicycle, other vehicle, ground, and driveway.

#### B.2. Pretraining

**Shared setting.** The point tokens (see [50] for details) in the WaffleIron backbones are computed using 16 nearest neighbors and the following point features: lidar intensity, 3D Cartesian *xyz* coordinates, radius/range. The field-of-view in the spatial mixing blocks is restricted to [-64 m, +64 m] along the *x* and *y* axes and [-8 m, +8 m] along the *z* axis; we use a grid of resolution 50 cm.

We pretrain the WaffleIron backbones using AdamW [39] with a weight decay of 3 × 10⁻⁴ and a learning rate linearly increasing from 0 to 0.002, then decreasing to 10⁻⁵ following a cosine schedule. The number of iterations or epochs corresponding to that maximum learning rate, as well as the total number of iterations or epochs, are described below, depending on the setting.

**Mono-dataset setting.** The mono-dataset setting is the setting used in Secs. 3.3, 4.3, 4.4, 4.7. We pretrain the WaffleIron backbones by distilling 2D features during 19 epochs, with a batch size of 16. The learning rate reaches its maximum value after 2 epochs. The MinkUNet backbone is pretrained following SLidR [57] protocol with the following adjustments: we use a batch size of 12, an initial

learning rate of 1.5 –the optimizer used in SLidR is SGD– and 25 epochs.

**Multi-dataset setting.** The multi-dataset setting is the setting used in Secs. 4.5, 4.6. The pretraining dataset contains: 28,130 scans for nuScenes, 19,130 scans for SemanticKITTI, 3,920 scans for Panda64 and 3,920 scans for PandaGT. We pretrain WI-256 by distilling 2D features with a batch size of 16 during 19 epochs on nuScenes, 28 epochs on SemanticKITTI, 136 epochs on Panda 64 or Panda GT, 11 epochs on nuScenes & SemanticKITTI, 10 epochs on the mix of all datasets. The number of epochs is adjusted so that the backbone is pretrained for (approximately) the same number of iterations. For WI-768, as the available GPU memory is not sufficient for some batches, we decrease the batch size to 8 and pretrain for 49 epochs on nuScenes and 25 epochs on the mix of all datasets. The learning rate reaches its maximum value after 3500 iterations in all cases.

**2D teacher.** All the results presented in this paper with the DINO-pretrained ViT-S/8 are obtained by distilling the keys at the last attention layer. This choice was guided by the fact that the last keys have properties that enable the design of unsupervised object discovery algorithms [59, 60, 62]. The results obtained with the DINOv2-pretrained ViTs are obtained by distilling the features before the last normalization layer.

**Multi-teacher distillation.** Let us denote the output feature dimension of both image teachers by $F_{2D}^{(1)}$ and $F_{2D}^{(2)}$, respectively. On the image side, we $\ell_2$-normalize the pixel features extracted by each teacher and concatenate them. On the point cloud side, the head $\psi_{3D}$ is a 2-layer MLP where the hidden linear layer has size $2 \times F_{3D}$ and is followed by a Layer Norm and a ReLU. The final linear layer of the MLP has size $F_{2D}^{(1)} + F_{2D}^{(2)}$ to match the size of the concatenated 2D features. These point features are then split into two parts of size $F_{2D}^{(1)}$ and $F_{2D}^{(2)}$, respectively. Each part is $\ell_2$-normalized independently. The normalized features are then re-concatenated. Finally, we distill the knowledge of the 2D features by applying Eq. (1) directly on the features of size $F_{2D}^{(1)} + F_{2D}^{(2)}$.

#### B.3. Linear probing

The linear head is trained with a batch size of 8, using AdamW with a weight decay of 3 × 10⁻³. The learning rate linearly increases from 0 to 0.001 during the first 2 epochs and then decreases to 10⁻⁵ following a cosine schedule. We use 20 epochs on nuScenes and SemanticKITTI, and 50 epochs on Pandar64 and PandarGT.
<table><caption>Table 9. Robustness to corruptions. The evaluation is conducted on nuScenes-C from the Robo3D benchmark [30]. We report the mCE%, mRR%, and the mIoU% attained for the eight corruptions, i.e., fog, wet ground, snow, motion blur, beam missing, crosstalk (among multiple sensors), incomplete echo, and cross-sensor (beam and point dropping). The scores in italic are obtained from [30, 38]. PPKT, SLidR and Seal use a MoCov2 ResNet-50. We use DINOv2 ViT-L/14.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">2D Back.</th><th rowspan="2">3D Back.</th><th rowspan="2">Pretrain. dataset</th><th rowspan="2">mCE% ↓</th><th rowspan="2">mRR% ↑</th><th colspan="8">Corruptions (mIoU% ↑)</th></tr><tr><th>Fog</th><th>Wet</th><th>Snow</th><th>Motion</th><th>Beam</th><th>Cross</th><th>Echo</th><th>Sensor</th></tr></thead><tbody><tr><td>–</td><td>–</td><td>Cylinder3D [80]</td><td>–</td><td>105.6</td><td>78.1</td><td>61.4</td><td>71.0</td><td>58.4</td><td>56.0</td><td>64.2</td><td>45.4</td><td>60.0</td><td>43.0</td></tr><tr><td>–</td><td>–</td><td>2DPASS [67]</td><td>–</td><td>98.6</td><td>75.2</td><td>64.5</td><td>76.8</td><td>54.5</td><td>62.0</td><td>67.8</td><td>34.4</td><td>63.2</td><td>45.8</td></tr><tr><td>–</td><td>–</td><td>SPVCNN [61]</td><td>–</td><td>97.5</td><td>75.1</td><td>55.9</td><td>74.0</td><td>42.0</td><td>74.6</td><td>69.0</td><td>28.1</td><td>65.0</td><td>51.6</td></tr><tr><td>–</td><td>–</td><td>GFNet [51]</td><td>–</td><td>92.6</td><td>83.3</td><td>69.6</td><td>75.5</td><td>71.8</td><td>59.4</td><td>64.5</td><td>66.8</td><td>61.9</td><td>42.3</td></tr><tr><td>–</td><td>–</td><td>WI-768</td><td>–</td><td>90.9</td><td>80.6</td><td>72.2</td><td>78.0</td><td>66.6</td><td>55.2</td><td>70.4</td><td>48.7</td><td>64.7</td><td>52.4</td></tr><tr><td>PPKT</td><td>[37]</td><td>ResNet-50</td><td>MinkUNet</td><td>nuScenes</td><td>105.6</td><td>76.1</td><td>64.0</td><td>72.2</td><td>59.1</td><td>57.2</td><td>63.9</td><td>36.3</td><td>60.6</td><td>39.6</td></tr><tr><td>SLidR</td><td>[57]</td><td>ResNet-50</td><td>MinkUNet</td><td>nuScenes</td><td>106.1</td><td>76.0</td><td>65.4</td><td>72.3</td><td>56.0</td><td>56.1</td><td>62.9</td><td>41.9</td><td>61.2</td><td>38.9</td></tr><tr><td>Seal</td><td>[38]</td><td>ResNet-50</td><td>MinkUNet</td><td>nuScenes</td><td>92.6</td><td>83.1</td><td>72.7</td><td>74.3</td><td>66.2</td><td>66.1</td><td>66.0</td><td>57.4</td><td>59.9</td><td>39.9</td></tr><tr><td>ScaLR (ours)</td><td>ViT-L/14</td><td>WI-768</td><td>nuScenes</td><td>89.1</td><td>83.7</td><td>70.8</td><td>77.2</td><td>67.1</td><td>55.9</td><td>70.0</td><td>65.7</td><td>63.9</td><td>51.1</td></tr><tr><td>ScaLR (ours)</td><td>ViT-L/14</td><td>WI-768</td><td>Multiple</td><td>87.4</td><td>83.8</td><td>72.2</td><td>77.9</td><td>69.1</td><td>57.4</td><td>70.1</td><td>62.7</td><td>64.0</td><td>52.2</td></tr></tbody></table>

## B.4. Finetuning

For finetuning the pretrained WaffleIron backbones, we use a batch size of 8, using AdamW without weight decay. The learning rate linearly increases from 0 to 0.002 during the first tenth of epochs and then decreases to 0 following a cosine schedule. During finetuning, we also use stochastic depth [26] with a layer drop probability of 0.2. We finetune the WaffleIron backbones for 45 epochs and a layer-wise learning rate decay parameter of 0.95 when using 1% and 10% of available data, and for 25 epochs and a layer decay parameter of 0.99 when all annotated data are available.

## C. Visual inspection

We provide a visualization of the features computed by a ScaLR-pretrained $\phi_{3D}$ backbone in Fig. 2. We use our WI-768 pretrained on nuScenes, SemanticKITTI, Pandar64 and PandarGT. In this figure, the features are projected onto the space spanned by their 3 principal components and used as RGB values to color the point clouds. Note that the PCA is done independently on each scan, which explains why the colors are not consistent from one scan to another.

We notice that the feature space of our pretrained backbone is correctly structured as we can distinguish rather easily the main urban constructions and objects in these figures. For example, we notice that the points belonging to road and sidewalk have similar colors (per scan) on nuScenes and SemanticKITTI. On Pandar64 and PandarGT, we also notice that the cars have similar colors (per scan) as well. Let us mention that the road on PandarGT scans appears in a less uniform color than on the other datasets. It could be explained by a higher density of points on the road for this lidar, which might lead to more subtle differences between features after distillation and/or PCA.

We continue our visual inspection of the distilled features by presenting feature similarity map with respect to class prototypes in Fig. 3. The features are extracted at the output of $\phi_{3D}$ and are $\ell_2$-normalized. The similarity maps are then obtained as follows. For each scan, we use the ground-truth labels to extract the point features of a class of interest (car, pedestrian, road or sidewalk). We average all the corresponding features to obtain a single class prototype for that class. Finally, we compute the similarity of all point features with respect to this class prototype. This is a similar procedure as the one used in Fig. 1 but using a mean feature instead of a single point feature.

In all cases, we notice that the most similar features to a class prototype belongs to corresponding class, as expected. This is another indication that the feature space is well structured where: the features of a same semantic class are close to each other; the features of two different semantic class are well separated. Nevertheless, when inspecting closely the similarity map, we notice sometimes some “leakage” around the objects of interest. This phenomenon is mostly visible for the class pedestrian. We believe that these artifacts are due to errors when projecting the points onto the camera plane, which affects the boundary of the objects. Finally, we remark as well that the similarity maps are less sharp on Pandar64 and PandarGT than on nuScenes and SemanticKITTI, likely because of the small number of scans available in Pandaset.

## D. Limitations

Our work raises the possibility of replicating undesirable biases present in the large pretrained 2D models used for distillation. These models are known to harbor problematic biases related, e.g., to geographic location, gender, skin tone, and age. When distilling these 2D vision models into
3D lidar models, there is a potential for these biases to be
amplified or mirrored. Our resulting lidar models may ex-
hibit varying performance across different geographical re-
gions, influenced by how these regions are represented in
the training datasets of the original 2D models and in the
2D-to-3D distillation training data. For real-world applica-
tions of this distillation strategy, practitioners are expected
to be mindful about the 2D foundation model used and the
nature of the data it was trained upon (e.g., potential biases,
privacy breaches, licenses, etc.)

Our study in Sec. 3.3 shows that the linear probing mIoU
has a standard deviation around 1.0 percentage point be-
tween different pretrainings. Some possibilities to reduce
these small fluctuations might be to explore longer pretrain-
ing schedules, or re-increase the number of loaded images
per scan (from 1 to 6).
<figure><img src="image_2.png" /><figcaption>Figure 2. **Distilled feature visualizations.** We project the features at the output of $\phi_{3D}$ into a three-dimensional space by PCA. The projected value serves as RGB value to color the point clouds, i.e., the first, second and third components are used as the red, green and blue channels, respectively. Note that the PCA is done independently for each scan, which explains why the colors are not consistent from one scan to another. In this figure, we used the WI-768 pretrained on nuScenes, SemanticKITTI, Pandar 64 and Pandar GT with ScaLR.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 3. Similarity map with class prototype. For each scan, we use the ground-truth labels (presented on even rows) of four classes (car, pedestrian, road, sidewalk) to compute a class prototype (mean feature of the point belonging to the considered class). We then compute the feature similarity map (presented on odd rows) with respect to that class prototype. Color goes from blue to red for low and high values.</figcaption></figure>