# Generalization abilities of foundation models in waste classification

Aloïs Babé, Rémi Cuingnet, Mihaela Scuturici, Serge Miguet

► To cite this version:

Aloïs Babé, Rémi Cuingnet, Mihaela Scuturici, Serge Miguet. Generalization abilities of foundation models in waste classification. Waste Management, 2025, 198, pp.187-197. 10.1016/j.wasman.2025.02.032. hal-05009162

HAL Id: hal-05009162

https://hal.science/hal-05009162v1

Submitted on 9 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Generalization abilities of foundation models in waste classification

Aloïs Babé*,¹,², Rémi Cuingnet², Mihaela Scuturici¹, and Serge Miguet¹

¹Université Lumière Lyon 2, CNRS, Ecole Centrale de Lyon, INSA Lyon, Université Claude Bernard Lyon 1, LIRIS, UMR5205, 69676 Bron, France

²Veolia Scientific & Technical Expertise Department, Maisons-Laffitte, France

Abstract

Industrial waste classification systems based on com-
puter vision require strong generalization abilities across
location and time period in order to be deployed. This study
investigates the potential of foundation models, known for
their adaptability to a wide range of tasks and promising
generalization capabilities, to serve as the basis for such
systems. To evaluate the generalization performance of
foundation models we use five waste classification datasets
spanning various domains, train the models on one dataset
and test them on all others. Additionally, we explore vari-
ous training procedures to optimize foundation model adap-
tation for this specific domain. Our findings reveal that
foundation models exhibit superior generalization abilities
compared to standard models and that good generaliza-
tion performance is correlated with the model size and the
size of the model pretraining dataset. Furthermore, we
demonstrate that elaborate classifier heads are not neces-
sary for extracting discriminative features from foundation
models. Both standard fine-tuning and Parameter-Efficient
Fine-tuning (PEFT) improve generalization performance,
with PEFT being particularly effective for larger models.
Simple data augmentation techniques were found to be in-
effective. Overall, application of foundation models to in-
dustrial waste classification holds very promising results.

1. Introduction

The imperative to recycle and efficiently manage waste
is driven by the need to conserve resources, save energy,
and reduce greenhouse gas emissions [19]. In the United
States alone, recycling of municipal solid waste contributed
to a reduction of approximately 193 million metric tons of
carbon dioxide equivalent in 2018 [19]. Despite these ef-
forts, significant amounts of recoverable materials continue

to be lost, exemplified by the $11 billion worth of packag-
ing materials landfilled in 2010 [35]. Effective recycling
necessitates precise sorting of waste, a process typically
conducted in Materials Recovery Facilities (MRFs). Opti-
mizing these sorting processes requires knowing the waste
stream composition at various points within the MRF. Tra-
ditional methods of stream inspection, which involve either
manual sampling and sorting of waste material or qualita-
tive visual inspections, are time-consuming and fail to pro-
vide real-time data [14]. The development of an automatic
stream composition analysis system, leveraging computer
vision technologies, could enhance the efficiency and ef-
fectiveness of waste sorting by providing real-time accurate
data on waste stream composition. Such a system should
be capable of being deployed across diverse geographical
locations and maintain its functionality over extended peri-
ods. This necessitates advanced generalization capabilities,
enabling the system to adapt to changes in waste composi-
tion, in packaging materials, and in the image acquisition
systems. Such robustness is crucial for ensuring the sys-
tem's long-term usability and effectiveness in a variety of
operational environments.

Foundation models are characterized by their vast num-
ber of parameters and are pre-trained on extensive datasets,
often using self-supervision, enabling them to be adapted to
a broad spectrum of tasks with minimal task-specific tun-
ing [8]. Initially developed in the field of natural language
processing (NLP) [9], these models have shown remark-
able capabilities [3, 39, 52, 53]. A notable example includes
the GPT [9, 39] models used by OpenAI in ChatGPT. Sim-
ilarly, foundation models have begun to make significant
inroads into the field of computer vision, although they are
still several orders of magnitude smaller in scale compared
to their NLP counterparts. While language model LLama
3 has 70 billion parameters and was trained on 15 trillion
tokens [53], Google Research has made strides with a 22
billion parameters vision model [15], and the largest pub-
licly available vision foundation model, DINOv2 [40], fea-

*Corresponding author: alois.babe@veolia.com
tures 1 billion parameters and was trained on 100 billion
tokens. Nevertheless these foundation models have demon-
strated state-of-the-art generalization capacities, even com-
peting with supervised models [30, 40, 42].

Therefore, foundation models and their generalization
abilities seem like good candidates for a computer vision
based waste classification system. To date, those abilities
have predominantly been tested on standard datasets such
as ImageNet, CIFAR, and Pascal VOC 2007 [40, 42, 55, 56],
which, while useful, do not necessarily reflect the complex-
ities of real-world domain-specific applications. This re-
search aims to contribute to bridge this gap by evaluating
the generalization ability of foundation models on the task
of waste classification.

To achieve this, we have tested a range of foundation models of varying sizes on several waste classification datasets. The generalization capabilities are assessed by training each model on a single dataset and then evaluating its performance across all others. Furthermore, to determine the optimal training procedures for a foundation model-based classifier, we will explore the impact of various adaptation techniques presented in Figure 1. These include the type of head classifier used, the extent of fine-tuning with both standard methods and Parameter-Efficient Fine-Tuning (PEFT) techniques [26], and the role of data augmentation.

## 2. Related work

### 2.1. Foundation models producing all purpose features

Foundation models producing all purpose visual features
build upon self-supervised learning techniques developped
for models such as MoCo [23], SimCLR [13], BYOL [21],
SwAV [10], DINO [11], MAE [22] or iBOT [62]. These
methods laid the groundwork for learning robust represen-
tations that could be effectively transferred to various down-
stream tasks, but those early models were all trained on Im-
ageNet [46], which contains only 1.3 million images. The
CLIP [42] model marked a milestone by being the first vi-
sion model to be trained on a very large dataset, consisting
of 400 million image-text pairs, with a high number of pa-
rameters (300 millions), and can thus be considered as a
foundation model [8]. CLIP employs two encoders—one
for images and another for text—and trains them using con-
trastive learning. During training the model is fed with
batches of (image, text) pairs and is trained to maximize
the similarity of the embeddings from real (image, text)
pairs while minimizing the similarity across incorrect pair-
ings. A similar approach is adopted by ImageBind [20],
which extends the modality to include audio, heat maps,
depth, and IMU data alongside images and text. Another
foundation model, DINOv2 [40], focuses solely on images.

Its training process involves feeding two different random
transformations of an input image to a teacher and a stu-
dent network with the same architecture. The student's pa-
rameters are learned via a combination of DINO [11] and
iBOT [62] losses and the teacher parameters are updated
with an exponential moving average of the student's pa-
rameters. A centering and sharpening of the momentum
teacher outputs is used to avoid model collapse ; a regular-
izer to spread features and a short high-resolution training
phase were also added. The training dataset for DINOv2,
LVD-142M, was constructed from an initial pool of 1.2 bil-
lion images sourced from the web, followed by deduplica-
tion and retrieval of images that closely align with those
in 15 curated datasets (like ImageNet). It is also worth
mentioning the emergence of diffusion models, which have
demonstrated remarkable capabilities in the domain of im-
age generation [43, 45, 47]. Among these, the Latent Diffu-
sion Mode (LDM) [45] stands out by incorporating an au-
toencoder, which functions as a type of feature extractor.
This autoencoder is used to facilitate the generation pro-
cess by working within a compressed latent space. Recent
research [41] has successfully explored using the LDM's
autoencoder as a feature extractor.

### 2.2. Generalization performances of foundation models

These foundation models exhibit generalization capa-
bilities that are now state-of-the-art. The generalization
capabilities of CLIP and DINOv2 have been rigorously
tested from their inception, albeit primarily on ImageNet-
like datasets such as ImageNetV2, ImageNet-A, ImageNet-
R, ImageNet-C, and Sketch [40,42] and never for a domain-
specific application. Under these conditions, Zero-shot
CLIP has demonstrated a significantly higher robustness
to distribution shift compared to standard ImageNet mod-
els [42]. DINOv2, on the other hand, has reported state-
of-the-art weakly supervised generalization performances
on some of these datasets [40]. Several studies have high-
lighted the superior performance of foundation models in
generalization. For instance, it has been claimed that high
train-test similarity alone cannot account for CLIP's per-
formance [38], implying that the model achieves a certain
level of robustness. Similarly, it has been demonstrated
the superior performance of few-shot CLIP over a few-shot
vision-only model in limited data environments that con-
tain realistic distribution shifts [55] and evaluations of CLIP
under natural distribution shift have shown robust perfor-
mance [56]. Furthermore, CLIP models have shown com-
petitive performance in out-of-distribution (OOD) detection
across commonly used benchmarks such as iNaturalist and
ImageNet-O [54], generally exhibiting better factor-level
robustness than other models.

However, some studies have challenged these general-
<figure><img src="image_2.png" /><figcaption>Figure 1. Four approaches used in this study to adapt pre-trained foundation models to a new domain. (a) Linear probing: training a single fully connected layer on top of frozen model features. (b) Adapter modules: learning a more complex network over frozen features. (c) Partial fine-tuning: unfreezing and training some of the model's last layers. (d) Parameter-Efficient Fine-Tuning (PEFT): enabling small parameter adjustments across all model layers, here Low-Rank Adaptation [26] (f) was used to adapt Vision Transformer (ViT) (e) based foundation models.</figcaption></figure>

For example, the generalization of foundation models across geography has been tested, revealing significant disparities in performance linked to the geographic origin of data [44]. For Imagebind, the authors tested the generalization capabilities of their model on audio and depth modalities and found that it outperforms its competitors [20].

## 2.3. Foundation models for real-world domain-specific applications

The adaptation of foundation models to real-world domain-specific applications has been a growing research subject. For instance, Chen et al. [12] have adapted foundation models to plant phenotyping, providing a comprehensive discussion on the optimal use-cases of different foundation models and tuning methods. In another study, Baha-
roon et al. [5] conducted a thorough evaluation of DINOv2 across various scenarios for multiple radiology modalities. The study demonstrated DINOv2's ability to either outperform or compete with supervised methods trained end-to-end.

Furthermore, Zhang et al. [61] benchmarked state-of-the-art vision models, including ResNet [24], ViT [18], DINOv2 [40], and ConvNeXt [33], for microscopic material analysis. The study found that DINOv2 exhibited the most stable performance, thereby demonstrating its capacity to generalize to rarely seen domains. Lastly, Huix et al. [27] applied foundation models to medical image classification, where DINOv2 was the only foundation model that consistently outperformed the standard practice of learning a standard model pretrained on ImageNet. All those studies adapt foundation models to their specific task, but do not test the generalization abilities of adapted models, to the best of our knowledge such a study has never been carried out.

## 2.4. Computer vision for waste classification

The literature on computer vision applications in waste management spans three primary research domains: recyclable material identification, trash pollution detection, and solid waste classification [57]. Within these domains, the tasks predominantly focus on classification, object detection, and image segmentation [57]. Convolutional Neural Networks (CNNs) have been the dominant approach for tackling these tasks, with various architectures [4, 7] such as ResNet, MobileNet, DenseNet [37] and EfficientNet [36] being extensively utilized. Recent studies have also explored the potential of hybrid models [1], vision transformers [2], and advanced techniques like feature fusion and attention mechanisms to enhance model performance [17, 60]. In the specific areas of detection and segmentation, object detectors such as YOLO [16, 49], Faster R-CNN [14, 29], and Mask R-CNN [31] have been prominently featured in recent research.

Despite these advancements, the field faces significant challenges related to the suitability and specificity of datasets. Current datasets often lack the granularity (e.g., specific types of plastics like PET, HDPE, PP, PS) needed for municipal solid waste sorting in Material Recovery Facilities (MRFs), with many featuring overly broad or irrelevant class labels [57]. Notably, there are only two public datasets designed for municipal waste sorting on conveyor belts—the ZeroWaste [6] and WaRP [59] datasets. The need for a robust system has been demonstrated by [14]: a drop in performance was noted when the model was tested on the same training waste stream but six months later. Furthermore, to the best of our knowledge, foundation models have not yet been explored within the framework of computer vision systems applied to waste management.

<table><caption>Table 1. Models used in this study, with their number of parameters, the dimension of their output features, and the average inference time for one image on a 16GB NVIDIA Tesla P100.</caption><thead><tr><td>Model</td><td>Number of parameters</td><td>Features dimension</td><td>Inference time (ms)</td></tr></thead><tbody><tr><td>DINO ViT-S/16</td><td>21,665,664</td><td>384</td><td>8.07</td></tr><tr><td>DINO ViT-S/8</td><td>21,670,272</td><td>384</td><td>22.69</td></tr><tr><td>DINO ViT-B/16</td><td>85,798,656</td><td>768</td><td>15.16</td></tr><tr><td>DINO ViT-B/8</td><td>85,807,872</td><td>768</td><td>59.81</td></tr><tr><td>DINOv2 ViT-S/14</td><td>22,056,576</td><td>384</td><td>10.02</td></tr><tr><td>DINOv2 ViT-B/14</td><td>86,580,480</td><td>768</td><td>19.0</td></tr><tr><td>DINOv2 ViT-L/14</td><td>304,368,640</td><td>1024</td><td>53.21</td></tr><tr><td>DINOv2 ViT-g/14</td><td>1,136,480,768</td><td>1536</td><td>203.91</td></tr><tr><td>CLIP-RN50</td><td>38,316,896</td><td>1024</td><td>8.2</td></tr><tr><td>CLIP-RN101</td><td>56,259,936</td><td>512</td><td>8.91</td></tr><tr><td>CLIP-ViT-B/16</td><td>86,192,640</td><td>512</td><td>8.52</td></tr><tr><td>CLIP-ViT-B/32</td><td>87,849,216</td><td>512</td><td>8.3</td></tr><tr><td>CLIP-RN50x4</td><td>87,137,080</td><td>640</td><td>11.1</td></tr><tr><td>CLIP-RN50x16</td><td>167,328,912</td><td>768</td><td>19.06</td></tr><tr><td>CLIP-ViT-L/14</td><td>303,966,208</td><td>768</td><td>16.2</td></tr><tr><td>CLIP-RN50x64</td><td>420,380,352</td><td>1024</td><td>43.95</td></tr><tr><td>ImageBind</td><td>629,680,640</td><td>1024</td><td>67.59</td></tr><tr><td>Stable Diffusion</td><td>34,163,592</td><td>3136</td><td>27.11</td></tr><tr><td>ConvNeXt-large</td><td>197,767,336</td><td>1536</td><td>27.23</td></tr><tr><td>ResNet-152</td><td>60,192,808</td><td>2048</td><td>13.44</td></tr><tr><td>ViT L/14</td><td>304,326,632</td><td>1280</td><td>209.83</td></tr></tbody></table>

# 3. Material and methods

## 3.1. Foundation models

In this study we compared four families of foundation models DINOv2, CLIP, ImageBind and Stable Diffusion.

DINOv2 [40] was pre-trained on LVD-142 M, a 142 million images dataset. In this work we will use the original version ViT-g/14, pre-trained from scratch, and smaller version obtained with distillation, keeping roughly the same pre-training procedure but using the bigger model as the teacher network [40]. For comparison we will also use different versions of the original DINO [11] model that were all pre-trained from scratch on ImageNet [46].

CLIP [42] was pre-trained on a private dataset of 400 million (image, text) pairs collected from publicly available sources on the Internet. The different model versions consist of five ResNet models and three ViT models. For the ResNet models, RN50x4, RN50x16, and RN50x64 are three models scaled using the EfficientNet methodology [51], utilizing approximately 4, 16, and 64 times the compute of a ResNet-50, respectively.

ImageBind [20] learned a joint embedding across six diverse modalities: images, text, audio, depth, thermal, and IMU data. Initially, the model employs aligned text and image encoders pre-trained on the LAION-2B dataset, a subset of LAION-5B [48] containing only English image-text
<figure><img src="image_3.png" /><figcaption>Figure 2. An example instance of each dataset used in this study.</figcaption></figure>

<table><caption>Table 2. Datasets used in this study.</caption><thead><tr><td>Dataset</td><td>Type</td><td># objects</td><td># classes</td></tr></thead><tbody><tr><td>Hall</td><td>designed waste stream</td><td>4600</td><td>24</td></tr><tr><td>Alu</td><td>real waste stream</td><td>6323</td><td>5</td></tr><tr><td>N&amp;P</td><td>real waste stream</td><td>7134</td><td>7</td></tr><tr><td>ZW</td><td>real waste stream</td><td>18000</td><td>6</td></tr><tr><td>TN</td><td>individual piece of waste</td><td>1889</td><td>4</td></tr></tbody></table>

pairs. To integrate additional modalities, separate encoders for audio, depth, thermal, and IMU data were pre-trained while the initial image and text encoders remained frozen.

Stable Diffusion [45] is a latent diffusion model trained on a subset of LAION-5B [48] and filtered for unsafe content. Here, only the encoder to the latent diffusion space will be used.

Traditional models ResNet, ViT and ConvNeXt were also used as a baseline. ResNet [24] introduced residual connections, enabling the training of networks with an unprecedented number of layers, reaching up to 1000. ViT [18] was developed as an alternative to convolutional neural networks (CNNs), while ConvNeXt [33] is a convolutional model inspired by transformers. These models have demonstrated state-of-the-art performance on image recognition benchmarks at the time of their publication. All three models are pre-trained on ImageNet. We reported in Table 1 all the models that were studied along with their number of parameters and the dimension of their output features space.

## 3.2. Datasets

This study utilized five datasets, presented in Figure 2 and Table 2, encompassing a diverse range of waste types, locations, and time periods. Three datasets are provided by us, capturing real-world waste streams in material recovery facilities (MRFs). Two of them includes respectively images from a News and Pams stream in France (N&P) and an aluminum (Alu) stream in England, captured by cameras

and lighting systems placed over conveyor belts. The third dataset (Hall) features isolated waste items from MRFs individually placed on a conveyor belt in a research hall, resulting in clean backgrounds. Additionally, two publicly available datasets were leveraged: ZeroWaste (ZW) [6], containing images of waste on conveyor belts in a paper stream in Massachusetts, and TrashNet (TN) [58], a dataset of manually collected recycled objects around Stanford with isolated waste against a white background. We used a modified version of the TrashNet dataset, removing 2 waste classes that are not found in MRFs (trash and glass) in order to be able to perform the label conversion we describe in the next paragraph. As we focus on classification, for the Hall, Alu, N&P and ZeroWaste datasets, we consider that a data point is defined as a bounding box of an object (piece of trash). These datasets introduce domain gaps due to variations in waste types, locations, and image capture methods. The detailed distribution of classes for each dataset can be found in the Supplementary Section B.

Models were trained on the research hall dataset, which has the most granular class labels. Labels from other datasets can be viewed as unions of these detailed labels. Hence, the predicted probability of a class is obtained by summing all the probabilities of the corresponding fine-grained research hall dataset classes.

## 3.3. Experimental protocol

This study aims to evaluate the generalization abilities of foundation models for real-world domain-specific computer vision tasks, specifically waste classification. To rigorously test generalization capabilities, we adopted a cross-dataset evaluation approach, training models on one dataset (the research hall dataset) and testing them on all remaining datasets. This allowed us to assess the models' ability to learn classification features that are robust to distribution shifts across diverse datasets, due to various geographies, time periods, and acquisition processes.

We also aimed at founding the optimal training pipeline for a foundation model-based classifier. The standard approach to extract features from the input data using foundation models is to use them “as is”, without further fine-tuning. These features are then used to train a separate classifier head for the specific task at hand (Figure 1a). We evaluated the foundation models we presented in Section 3.1 using this direct pipeline. Additionally, we explored potential modifications to enhance the pipeline’s performance. One approach involved incorporating an adapter network between the foundation model and the classifier head (as depicted in Figure 1b), which we achieved by adding layers to the classifier head. Another method we explored was standard fine-tuning (as shown in Figure 1c), and we also examined Parameter Efficient Fine-tuning (as illustrated in Figure 1d). We also assessed the impact of data augmenta-
tion techniques on the overall efficacy of the pipeline.

While our methodology primarily focuses on testing the
generalization ability of models, this makes direct compar-
ison with the state-of-the-art on the public TrashNet dataset
challenging, as existing studies directly train their models
on TrashNet. To facilitate a more direct comparison, we
also perform a regular training experiment on the TrashNet
dataset. Here, we extract features using a foundation model
and then train a two-layer MLP classifier to map these fea-
tures to the desired classes.

**Foundation model.** The influence of foundation model selection on feature extraction performance is still unclear. This study assessed the effectiveness of four different types of foundation models. It also explored how the pre-training dataset of the foundation model affects its performance and examined the effect of model size on performance. While smaller models are less computationally demanding and quicker to train, larger models may identify subtler features and achieve higher classification accuracy. In order to evaluate the quality of the features extracted by the foundation model, we utilized a k Nearest Neighbors (k-NN) classifier. This approach aligns with established protocols from previous studies [10, 11, 40, 62] that have employed k-NN to assess the quality of the features computed by a model. The intuition behind the use of k-NN is that a key indicator of effective feature extraction is the ability of the resulting feature space to reflect semantic similarity. Ideally, features extracted from semantically similar images should cluster closely together, while features from dissimilar images should be well-separated. This property can be evaluated using a k-Nearest Neighbors (k-NN) classifier. We evaluated the performance of the k-NN classifier for different values of k (5, 10, 20, 40) on the research hall validation set. The k-NN model with the best performance on the validation set is then used to evaluate the model performances on all other datasets.

**Classifier head.** To investigate the impact of classifier
size on performance, we compared the performance of three
different classifier architectures: a single linear layer, a
two-layer Multi-Layer Perceptron (MLP), and a three-layer
MLP. For each architecture, we conducted a hyperparame-
ter search using three learning rates (0.001, 0.0003, 0.0001)
with AdamW optimizer [34] (weight decay 0.0001, (beta1,
beta2) = (0.9, 0.999)) and SGD optimizer [50] (weight de-
cay 0.001, momentum 0.95). The best hyperparameters for
each architecture were selected based on the performance
of the linear layer on the research hall validation set. For
a given foundation model the same hyperparameters were
used for all classifiers.

**Fine-tuning.** While the extracted features are intended
to be directly usable, we investigated the potential benefits
of fine-tuning the foundation model for further performance
improvement. We compared two fine-tuning approaches:

standard fine-tuning and LoRa [26], a Parameter-Efficient
Fine-Tuning (PEFT) [32] technique, to see if fine-tuning
improves feature quality and classification accuracy. Stan-
dard fine-tuning involves unfreezing the last layers of the
foundation model and training a linear classifier on top of
these layers, as shown in Figure 1c. In contrast, PEFT fo-
cuses on training only a small set of parameters, either by
modifying a subset of the existing parameters or by intro-
ducing new parameters, thereby significantly reducing the
computational cost. LoRa is a specific PEFT technique
based on the hypothesis that the modifications required dur-
ing fine-tuning often have a low “intrinsic rank.” LoRa
exploits this by learning a rank decomposition of certain
weight matrices, as depicted in Figure 1d, allowing for min-
imal parameter updates. Specifically, for a projection matrix
W in the self-attention module of size (d, k), it is replaced
with W + BA, where B is of size (d, r) and A is of size
(r, k), with r chosen such that r ≪ min(d, k). Hence, the
resulting number of trained parameters is r(d+k) instead of
dk. Both A and B are learned during fine-tuning, while the
original W remains frozen. We employed AdamW for 100
epochs with a learning rate lr = 0.0001 with linear warmup
from 0.1lr for 5 epochs and linear decay to 0.1lr for the last
45 epochs, (β₁, β₂) = (0.9, 0.999) and a batch size of 8.

**Data Augmentation.** Given the robustness of foundation models, data augmentation may not be necessary for achieving optimal performance. This is particularly relevant for DINO models, which are specifically pretrained to learn representations that are invariant to data augmentations. To investigate this hypothesis, we compared the performance of models trained with and without data augmentation, aiming to determine whether it provides any additional benefit for foundation model-based feature extraction. We employed a standard augmentation pipeline that includes scaling and cropping images to 256x256, followed by random 224x224 cropping, rotations, flips, and color jittering (using the `ColorJitter` class from `torchvisionecess` with parameters: contrast=0.5, saturation=0.5, hue=0.05). We kept the foundation model frozen and used a linear classifier. We performed both offline augmentation, where each image is augmented 10 times before training, and online augmentation, where a new augmented version of each image is generated at each epoch. For each model, we reused the optimal hyperparameters identified during the linear layer evaluation.

4. Results

4.1. Comparison of foundation models

Models ability to produce discriminative features is esti-
mated through the accuracy of a k-NN classifier. Results are
presented in Table 3. DINOv2 and ImageBind emerged as
<table><caption>Table 3. Accuracies of various frozen foundation models using K-NN and linear classifiers across multiple datasets. The classifiers were added on top of the foundation models and trained exclusively on the Research Hall dataset, with the foundation models' parameters kept frozen. Evaluation was conducted on the test sets of all five datasets.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">K-NN</th><th colspan="5">Linear</th></tr><tr><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th></tr></thead><tbody><tr><td>DINOv2 ViT-g/14</td><td><strong>70.9</strong></td><td>78.0</td><td>48.5</td><td>20.5</td><td><strong>88.3</strong></td><td><strong>81.7</strong></td><td>80.2</td><td>52.5</td><td>33.5</td><td>87.2</td></tr><tr><td>ImageBind</td><td>68.9</td><td><strong>78.8</strong></td><td>49.8</td><td>21.8</td><td>83.0</td><td>80.4</td><td><strong>82.0</strong></td><td>54.7</td><td><strong>55.3</strong></td><td><strong>91.0</strong></td></tr><tr><td>CLIP ViT-L/14</td><td>68.3</td><td>70.6</td><td>50.5</td><td>14.7</td><td>85.1</td><td>80.0</td><td>77.5</td><td><strong>56.2</strong></td><td>25.5</td><td>88.8</td></tr><tr><td>ResNet 152</td><td>64.1</td><td>71.0</td><td><strong>51.5</strong></td><td>15.0</td><td>68.6</td><td>72.8</td><td>76.3</td><td>53.6</td><td>37.7</td><td>70.7</td></tr><tr><td>ConvNeXt large</td><td>61.3</td><td>61.1</td><td>48.7</td><td>17.0</td><td>68.6</td><td>77.8</td><td>77.1</td><td>51.2</td><td>31.6</td><td>74.5</td></tr><tr><td>ViT L/16</td><td>60.0</td><td>61.6</td><td>48.0</td><td><strong>28.4</strong></td><td>62.8</td><td>77.2</td><td>78.2</td><td>49.9</td><td>22.0</td><td>75.5</td></tr><tr><td>Stable Diffusion</td><td>19.1</td><td>25.5</td><td>34.8</td><td>11.4</td><td>20.2</td><td>24.3</td><td>27.8</td><td>29.2</td><td>19.6</td><td>34.0</td></tr></tbody></table>

the top performers on three datasets, while baseline models ResNet and ViT L/16 took the lead on two datasets. Interestingly, ResNet outperformed CLIP on three datasets, demonstrating its competitive performance despite being a non-foundation model. While CLIP fell behind ResNet in terms of accuracy on these datasets, the difference remained within a margin of 0.5 points. However, on datasets where CLIP surpassed ResNet, the accuracy gap could be as significant as 16 points. Furthermore, we observed that foundation models exhibited a notable advantage over baseline models on the TrashNet dataset, which significantly differs from the training research hall dataset (manually collected waste with uniform white background, not in the context of MRF). Stable Diffusion consistently ranked last across all datasets, indicating its limitations in capturing relevant features for image classification.

Continuing the analysis of foundation models with linear layer classifiers, similar trends can be observed on Table 3. Stable Diffusion consistently exhibited the poorest performance, falling behind all other models by a significant margin. Consequently, it was excluded from further analysis in this study. Interestingly, it can be observed that on the News and Pams datasets DINOv2 was outperformed by standard model ResNet. Similarly, on the ZeroWaste dataset, both DINOv2 and CLIP were outperformed by ResNet. However, on each dataset, the best performing model was always a foundation model. Notably, ImageBind consistently ranked among the top models, achieving the best performance on three datasets and never falling more than 1.5 accuracy points behind the best model on any dataset. Furthermore, the notable advantage of foundation models over baseline models on the TrashNet dataset is still present.

Significant variations in accuracy depending on the dataset can be observed in Table 3. These differences can be attributed to the unique characteristics of each dataset, which are of several types. Firstly, the number and granularity of classes vary. For example, the News & Pams dataset includes several types of paper, while the TrashNet

and ZeroWaste datasets only have a single paper class. Additionally, the datasets corresponding to real waste streams (Aluminum, News & Pams, and ZeroWaste) exhibit class imbalance, with more instances of the main waste types in each stream. This imbalance can impact performance, as good detection of the main classes leads to better overall performance. Both the News & Pams and ZeroWaste datasets contain dense scenes, where object bounding boxes may contain other objects from different classes. This can potentially cause disturbances for the classifier and reduce performance. Furthermore, the images from the ZeroWaste dataset exhibit more blur compared to the other datasets. Additionally, it is worth noting that the TrashNet dataset is unique in that its waste items are not presented on a conveyor belt but rather isolated on a clear background.

All those differences prevent direct performance comparison between datasets. Hence direct comparisons can only be made between models for each dataset. Those comparisons reveal that foundation models show superior performance, indicating their robustness in handling label distribution shift, cluttered background, and blur.

Inference time for the model that are used in this study are reported in Table 1, the times shown correspond to the average inference time for one image on a 16GB NVIDIA Tesla P100. It can be noted that, with this kind of hardware, not all models are suitable for real-time use, especially on dense streams, which may involve classifying dozens of waste pieces per second. Nevertheless, there are strategies to mitigate this issue, such as reducing the size of the model. For instance, the ViT-L/14 of CLIP and DINOv2 share the same architecture, but CLIP employs a 16-bit floating point instead of a 32-bit floating point, which makes it three times faster.

**Models size.** To investigate the influence of model size on performance, linear classifiers were trained on the features extracted by DINO and CLIP models (Figure 3). Interestingly, a consistent trend can be observed across both model families: larger models generally achieved better
<figure><img src="image_4.png" /><figcaption>Figure 3. Linear probing accuracy of pre-trained DINO and CLIP models of various sizes. Models were linearly probed on the Research Hall dataset and evaluated on test sets of five datasets. The last column shows the average rank of each model over all datasets. Exact numerical values are presented in Supplementary Tables 6 and 7.</figcaption></figure>

performance. Among the DINO models, DINOv2 ViT-g/14 and ViT-L/14, the two largest models, consistently ranked among the top performers, either achieving the best performance or falling within 0.1 accuracy points of the best model across the five datasets. When computing the average rank of DINO models over all datasets, DINOv2 ViT-L/14, ViT-g/14, and DINO ViT-B/8 emerged as the top performers. Notably, there was no clear superiority of DINOv2 models over their DINO counterparts of the same size. This trend of larger models achieving better performance was also evident with CLIP models. The two largest models, ViT-L/14 and RN50x64, with comparable sizes, achieved the best performance. However, ViT-L/14 exhibited a performance drop on the ZeroWaste dataset compared to other models. Furthermore, comparing ViT-L/14 with RN50x64 and ViT-B with RN50x4 suggests that ResNet models tend to perform worse than ViT models of the same size.

**Pre-training dataset.** Direct evaluation of the impact of the pre-training dataset on model performance is limited due to variations in model size and architecture. Yet it can be noted that the ImageBind model, despite not having the highest number of parameters, achieved the best performance in linear evaluation. A more controlled comparison can be conducted between two models with similar

architectures and parameter counts: ConvNeXt and CLIP RN50x16, both convolutional neural networks with approximately 197 and 167 million parameters, respectively. The CLIP model, pre-trained on a dataset more than 300 times larger than that of ConvNeXt, demonstrated overall superior performance.

## 4.2. Evaluation of classifier heads

To further explore the factors influencing model performance, the impact of classifier head size on the performance of DINO, CLIP and ImageBind models (Table 4) was studied. For smaller models, such as DINOv2 ViT-S/14, adding a second layer to the classifier head resulted in improved performance. However, for larger models like CLIP ViT-B/16, a single-layer classifier head achieved the best results. The effect of classifier size on CLIP ViT-L/14 was less clear, with no significant difference observed between different head sizes. ImageBind consistently benefited from a two-layer classifier head across all datasets. Notably, only the largest DINOv2 model, ViT-g/14, showed improvement with a three-layer classifier head. This last finding aside, based on the observed results, it is difficult to definitively explain the relationship between model size and optimal classifier head size.
<table><caption>Table 4. Accuracy of various frozen foundation models with an added MLP classifiers with 1, 2 and 3 layers on multiple datasets. Models were trained on the research Hall dataset and evaluated on the test sets of all five datasets. All foundation model parameters were kept frozen during training, with only the additional classifier being fine-tuned.</caption><thead><tr><th>Model</th><th># layers</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th></tr></thead><tbody><tr><td rowspan="3">DINOv2<br>ViT-S/14</td><td>1</td><td><strong>77.4</strong></td><td>78.3</td><td>48.4</td><td>31.7</td><td>89.9</td></tr><tr><td>2</td><td><strong>77.4</strong></td><td><strong>79.7</strong></td><td>48.7</td><td><strong>40.6</strong></td><td><strong>90.4</strong></td></tr><tr><td>3</td><td>75.7</td><td>79.3</td><td><strong>49.5</strong></td><td>35.2</td><td>87.8</td></tr><tr><td rowspan="3">DINOv2<br>ViT-g/14</td><td>1</td><td>81.7</td><td><strong>80.2</strong></td><td>52.5</td><td>33.5</td><td><strong>87.2</strong></td></tr><tr><td>2</td><td>82.2</td><td>79.3</td><td><strong>53.2</strong></td><td>33.5</td><td>86.2</td></tr><tr><td>3</td><td><strong>82.4</strong></td><td>79.4</td><td>51.9</td><td><strong>38.0</strong></td><td><strong>87.2</strong></td></tr><tr><td rowspan="3">CLIP<br>ViT-B/16</td><td>1</td><td><strong>73.9</strong></td><td><strong>73.4</strong></td><td><strong>51.3</strong></td><td>65.6</td><td><strong>76.6</strong></td></tr><tr><td>2</td><td>72.0</td><td>68.5</td><td>51.1</td><td><strong>66.6</strong></td><td>73.9</td></tr><tr><td>3</td><td>72.0</td><td>60.6</td><td>49.2</td><td>64.8</td><td>70.2</td></tr><tr><td rowspan="3">CLIP<br>ViT-L/14</td><td>1</td><td><strong>80.0</strong></td><td>77.5</td><td><strong>56.2</strong></td><td>25.5</td><td><strong>88.8</strong></td></tr><tr><td>2</td><td>78.3</td><td>78.2</td><td>54.8</td><td><strong>27.4</strong></td><td>88.3</td></tr><tr><td>3</td><td>77.2</td><td><strong>78.6</strong></td><td>52.3</td><td>35.0</td><td>81.9</td></tr><tr><td rowspan="3">ImageBind</td><td>1</td><td>80.4</td><td>82.0</td><td>54.7</td><td>55.3</td><td>91.0</td></tr><tr><td>2</td><td><strong>80.7</strong></td><td><strong>82.8</strong></td><td><strong>57.4</strong></td><td><strong>60.3</strong></td><td><strong>92.6</strong></td></tr><tr><td>3</td><td>77.4</td><td>80.2</td><td>54.8</td><td>56.0</td><td>89.4</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4. Accuracy of DINOv2 ViT-S/14 with a linear classifier when fine-tuning some of the last layers of the model. Models were trained on the Research Hall dataset and evaluated on the test sets of all five datasets. Exact numerical values are presented in Supplementary Table 8.</figcaption></figure>

### 4.3. Impact of fine-tuning techniques

To assess the impact of fine-tuning foundation models on performance, we experimented with different fine-tuning strategies. Our findings in Figure 5 revealed that both standard fine-tuning of the last 3 layers and PEFT consistently outperformed keeping all model layers frozen on almost all models and datasets. For DINOv2 ViT-S/14, Figure 4 shows that the number of unfrozen layers had an impact on performance. Unfreezing more than 3 layers did not result in any further gains, and fine-tuning all layers even led to a

<table><caption>Table 5. Classification accuracies on the TrashNet dataset. Each model was trained on the TrashNet training set and evaluated on its test set. For foundation models, linear probing was employed, training only an additional linear classifier. State-of-the-art results are included for comparison.</caption><thead><tr><th>Model</th><th>Accuracy</th></tr></thead><tbody><tr><td>CLIP RN50</td><td>84.1</td></tr><tr><td>CLIP ViT-B/16</td><td>89.2</td></tr><tr><td>CLIP RN50x16</td><td>88.4</td></tr><tr><td>CLIP ViT-L/14</td><td>93.2</td></tr><tr><td>CLIP RN50x64</td><td>91.2</td></tr><tr><td>DINOv2 ViT-S/14</td><td>91.2</td></tr><tr><td>DINOv2 ViT-B/14</td><td>94.0</td></tr><tr><td>DINOv2 ViT-L/14</td><td>94.0</td></tr><tr><td>DINOv2 ViT-g/14</td><td>96.0</td></tr><tr><td>ImageBind</td><td>94.8</td></tr><tr><td>Aral et al. [4]</td><td>95</td></tr><tr><td>Bircanoğlu et al. [7]</td><td>95</td></tr><tr><td>Mao et al. [37]</td><td><strong>99.6</strong></td></tr><tr><td>Alrayes et al. [2]</td><td>95.8</td></tr></tbody></table>

significant drop in performance. For larger models (ViT-L and ViT-g), PEFT almost always outperformed fine-tuning only the last 3 layers. For other models, no clear superiority of one fine-tuning method over the other was observed.

### 4.4. Impact of data augmentation

To evaluate the impact of data augmentation on model performance, both offline and online augmentation techniques were experimented. Figure 6 analysis revealed that the effect of data augmentation was limited and varied depending on the specific augmentation method and model architecture. Indeed, even if for the test set of the research hall dataset we observed that both offline and online data augmentation contributed to improved performance, when considering all other datasets used in our experiments, data augmentation techniques did not systematically produce additional enhancements in a model's ability to generalize. Notably, for both offline and online augmentation, the size of the model appears to influence its sensitivity to data augmentation only for DINOv2 models. It is noticeable on the ZeroWaste dataset, for which data augmentation has a strong negative impact, except for the two biggest DINOv2 models (ViT-L/14 and ViT-g/14).

### 4.5. Standard TrashNet classification

The state-of-the-art results on the TrashNet dataset hover around 95% accuracy. Both Aral et al. [4] and Bircanoğlu et al. [7] reported this figure using DenseNet121, while Alrayes et al. [2] reported a slightly higher accuracy of 95.8% using a Vision Transformer based on Multilayer Hybrid Convolution Neural Network. Mao et al. [37] achieved
<figure><img src="image_6.png" /><figcaption>Figure 5. Accuracy difference between fine-tuned and non-fine-tuned foundation models with an added linear layer. Models were trained on the Research Hall dataset and evaluated on the test sets of all five datasets. Fine-tuning was performed using standard methods (training the last 3 layers) or LoRA, a PEFT technique. Exact numerical values are presented in Supplementary Table 9.</figcaption></figure>

a reported peak accuracy of 99.6% by employing a ge-
netic algorithm to optimize the fully-connected-layer of
DenseNet121. In contrast, our results when learning on
the research hall dataset fall short of these figures, rang-
ing from 88.8% for CLIP ViT-L/14 to 91.0% for Imagebind
(Table 3). However, when learning directly on the Trash-
Net dataset, our approach demonstrates competitive perfor-
mance, as evidenced in Table 5. Notably, DINOv2 ViT-g/14
achieves an accuracy of 96.0%.

This demonstrates the capability of foundation models to
achieve results comparable to the state-of-the-art with min-
imal adaptation and hyperparameter tuning, requiring only
feature extraction and training a simple MLP, making them
natural candidates for waste classification tasks, even in sce-
narios with minimal variations in input data distributions.

5. Discussions

Foundation models demonstrated superior generaliza-
tion abilities compared to standard models, particularly on
the TrashNet dataset, which significantly differed from the
training data distribution, where foundation models scored
at least 12 accuracy points more than standard models.
These results are very promising for the application to waste
classification and allow to envisage the creation of robust
classifiers which could be deployed in numerous contexts
eliminating the need for retraining and ensuring long-term

effectiveness.

Our results confirm that foundation models are more
adaptable to diverse data distributions. In particular, Im-
ageBind emerged as the top performer across all datasets,
despite not being the largest model, but the one with the
largest pre-training dataset. This highlights the importance
of pre-training data size and quality in achieving high gen-
eralization performance. On the contrary, Stable Diffusion
encoder was found to be unsuitable as simple feature ex-
tractors. This is probably due to its inherent focus on re-
construction accuracy and thus local information preserva-
tion. This contrasts with other foundation models that pri-
oritize capturing global and semantic content, making their
features more discriminative for image classification tasks.

Standard models, ResNet and ConvNeXt, exhibited sim-
ilar performance levels. Notably, ConvNeXt underper-
formed compared to foundation models with similar pa-
rameter numbers and architecture, such as CLIP-RN50x16.
This also suggests that the pre-training dataset plays a cru-
cial role in model performance, independently of the under-
lying architecture or number of parameters.

Our analysis revealed a general trend of larger mod-
els achieving better performance. Interestingly, the perfor-
mance improvements observed with DINOv2 over DINO
models seemed to vanish when using distillation tech-
niques. This suggests that model size is a critical parameter
<figure><img src="image_7.png" /><figcaption>Figure 6. Accuracy difference between foundation models trained with and without data augmentation. Models were linearly probed on the Research Hall dataset and evaluated on the test sets of all five datasets. Data augmentation was performed using offline augmentation (each image is augmented 10 times prior to training) and online augmentation (each image is randomly augmented at each epoch). Exact numerical values are presented in Supplementary Table 10.</figcaption></figure>

to benefit from larger pre-training dataset. This behaviour was described in [28] and [25] for Large Language Models, it was found that model and dataset size should be scaled equally in order to reach compute optimally.

Furthermore, the comparison of CLIP models suggests that Vision Transformer architectures may offer better generalization capabilities compared to Convolutional Neural Networks. One possible explanation for this could be that the inductive bias of CNNs might limit their ability to learn certain types of robust discriminative features.

Our analysis revealed that the two largest models, ImageBind and ViT-g/14, benefited from using a classifier head with more than one layer. However, for other models, no clear rule emerged regarding the optimal classifier head size. This could be due to the relatively small number of classes (24) and training images (less than 5,000) in our dataset. With larger datasets and more classes, larger classifier heads might become necessary.

Fine-tuning foundation models generally led to performance improvements, suggesting that adapting the models to specific domains can be beneficial. Experiments with DINOv2 fine-tuning suggest that adaptation should be light, as fine-tuning all layers can lead to significant performance drops. PEFT appears to be particularly effective for larger models. This could be because larger models have

more layers and therefore benefit from the ability to modify deeper layers sparsely. In future studies, it would be interesting to investigate whether PEFT can be further optimized by focusing on specific layers or subsets of layers, in particular this would make it possible to check whether the adaptation of the first layers of the model is really useful.

A robust classification model should ideally be insensitive to data augmentation, as it should be able to generalize well to unseen data. In fact, DINO models [11, 40] were explicitly pre-trained to produce the same features for two augmented views of the same image. This may explain why the largest DINOv2 models were less sensitive to data augmentation in our experiments. The observed random effects of data augmentation on performance suggest that simple random augmentations may not be effective for improving model robustness.

## 6. Conclusion

The practical application of foundation models in Material Recovery Facilities (MRFs) for waste classification highlighted their robust generalization capabilities, which are crucial for developing models that can be widely applied and remain effective over time. They therefore constitute a promising avenue to serve as a basis for such systems. Our research has demonstrated that foundation models pos-
sess superior generalization abilities compared to standard models such as ResNet and ConvNeXt. This enhanced generalization performance is primarily attributed to their large number of parameters and the extensive datasets used during their pre-training. Notably, the deployment of foundation models in practical applications does not require complex classifiers; a simple two-layer MLP suffices for feature classification. Furthermore, fine-tuning techniques, including both standard methods and PEFT, have proven beneficial, even without extensive hyper-parameter optimization. PEFT, in particular, shows greater advantages over standard fine-tuning when dealing with big models (over 100M parameters). Our study also explored the efficacy of data augmentation, which was found to impact performance inconsistently, thus rendering it less useful for our specific application. Looking ahead, further research is needed to explore the potential of foundation models in the context of object detection within waste management systems.

## References

[1] Kashif Ahmad, Khalil Khan, and Ala Al-Fuqaha. Intelligent fusion of deep features for improved waste classification. *IEEE access*, 8:96495–96504, 2020. 4

[2] Fatma S Alrayes, Mashael M Asiri, Mashael S Maashi, Mohamed K Nour, Mohammed Rizwanullah, Azza Elneil Osman, Suhanda Drar, and Abu Sarwar Zamani. Waste classification using vision transformer based on multilayer hybrid convolution neural network. *Urban Climate*, 49:101483, 2023. 4, 9

[3] AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. *Claude-3 Model Card*, 1, 2024. 1

[4] Rahmi Arda Aral, Şeref Recep Keskin, Mahmut Kaya, and Murat Hacıömeroğlu. Classification of trashnet dataset based on deep learning models. In *2018 IEEE International Conference on Big Data (Big Data)*, pages 2058–2062. IEEE, 2018. 4, 9

[5] Mohammed Baharoon, Waseem Qureshi, Jiahong Ouyang, Yanwu Xu, Kilian Phol, Abdulrhan Aljouie, and Wei Peng. Towards general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. *arXiv preprint arXiv:2312.02366*, 2023. 4

[6] Dina Bashkirova, Mohamed Abdelfattah, Ziliang Zhu, James Akl, Fadi Alladkani, Ping Hu, Vitaly Ablavsky, Berk Calli, Sarah Adel Bargal, and Kate Saenko. Zerowaste dataset: Towards deformable object segmentation in cluttered scenes. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 21147–21157, 2022. 4, 5

[7] Cenk Bircanoğlu, Meltem Atay, Fuat Beşer, Özgün Genç, and Merve Ayyüce Kızrak. Recyclenet: Intelligent waste sorting using deep neural networks. In *2018 Innovations in intelligent systems and applications (INISTA)*, pages 1–7. Ieee, 2018. 4, 9

[8] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Roderigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jianjun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. 1, 2

[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. *Advances in neural information processing systems (NeurIPS)*, 33:1877–1901, 2020. 1

[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. *Advances in neural information processing systems*, 33:9912–9924, 2020. 2, 6

[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In *Proceedings of the IEEE/CVF international conference on computer vision (ICCV)*, pages 9650–9660, 2021. 2, 4, 6, 11

[12] Feng Chen, Mario Valerio Giuffrida, and Sotirios A Tsaftaris. Adapting vision foundation models for plant phenotyping. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 604–613, 2023. 3

[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In *International conference on machine learning (ICML)*, pages 1597–1607. PMLR, 2020. 2

[14] Remi Cuingnet, Yannik Ladegaillerie, Jérôme Jossent, Aude Maitrot, Julien Chedal-Anglay, Williams Richard, Marine Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Roderigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jianjun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. 1, 2
Bernard, Jake Woolfenden, Emmanuel Birot, and Damien Chenu. Portik: A computer vision based solution for real-time automatic solid waste characterization – application to an aluminium stream. Waste Management, 150:267–279, 2022. 1, 4

[15] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul-mohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), pages 7480–7512. PMLR, 2023. 1

[16] Demetris Demetriou, Pavlos Mavromatidis, Ponsian M Robert, Harris Papadopoulos, Michael F Petrou, and Demetris Nicolaides. Real-time construction demolition waste detection using state-of-the-art deep learning methods; single-stage vs two-stage detectors. Waste Management, 167:194–203, 2023. 4

[17] Hongjie Deng, Daji Ergu, Fangyao Liu, Bo Ma, and Ying Cai. An embeddable algorithm for automatic garbage detection based on complex marine environment. Sensors, 21(19):6391, 2021. 4

[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 4, 5

[19] EPA. Recycling basics and benefits, 2013. 1

[20] Rohit Girdhar, Alaaldeln El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180–15190, 2023. 2, 3, 4

[21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems (NeurIPS), 33:21271–21284, 2020. 2

[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 16000–16009, 2022. 2

[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 9729–9738, 2020. 2

[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016. 4, 5

[25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de

Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), pages 30016–30030, 2022. 11

[26] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. 2, 3, 6

[27] Joana Palés Huix, Adithya Raju Ganeshan, Johan Fredin Haslum, Magnus Söderberg, Christos Matsoukas, and Kevin Smith. Are natural domain foundation models useful for medical image classification? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7634–7643, 2024. 4

[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 11

[29] Hamidreza Karbasi, Adam Sanderson, Alireza Sharifi, and Carl Wilson. Robotic sorting of shredded e-waste: utilizing deep learning. In proceedings on the international conference on artificial intelligence (ICAI), pages 119–123. The Steering Committee of The World Congress in Computer Science, Computer ..., 2018. 4

[30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. 2

[31] Maria Koskinopoulou, Fredy Raptopoulos, George Papadopoulos, Nikitas Mavrakis, and Michail Maniadakis. Robotic waste sorting technology: Toward a vision-based categorization system for the industrial robotic separation of recyclable waste. IEEE Robotics & Automation Magazine, 28(2):50–60, 2021. 4

[32] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arxiv 2023. arXiv preprint arXiv:2303.15647, 2023. 6

[33] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 11976–11986, 2022. 4, 5

[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 6

[35] Conrad MacKerron. Unfinished business: The case for extended producer responsibility for post-consumer packaging. Technical report, As You Sow, July 2012. 1

[36] Meena Malik, Sachin Sharma, Mueen Uddin, Chin-Ling Chen, Chih-Ming Wu, Punit Soni, and Shikha Chaudhary. Waste classification for sustainable development using im-
age recognition with deep learning neural network models. Sustainability, 14(12):7222, 2022. 4

[37] Wei-Lung Mao, Wei-Chun Chen, Chien-Tsung Wang, and Yu-Hao Lin. Recycling waste classification using optimized convolutional neural network. *Resources, Conservation and Recycling*, 164:105132, 2021. 4, 9

[38] Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does CLIP's generalization performance mainly stem from high train-test similarity? In *The Twelfth International Conference on Learning Representations*, 2024. 2

[39] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1

[40] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. *Transactions on Machine Learning Research (TMLR)*, 2024. 1, 2, 4, 6, 11

[41] Koutilya Pnvr, Bharat Singh, Pallabi Ghosh, Behjat Sidiquie, and David Jacobs. Ld-znet: A latent diffusion approach for text-based image segmentation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 4157–4168, 2023. 2

[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *International conference on machine learning (ICML)*, pages 8748–8763. PMLR, 2021. 2, 4

[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. *arXiv preprint arXiv:2204.06125*, 1(2):3, 2022. 2

[44] Megan Richards, Polina Kirichenko, Diane Bouchacourt, and Mark Ibrahim. Does progress on object recognition benchmarks improve generalization on crowdsourced, global data? In *The Twelfth International Conference on Learning Representations (ICLR)*, 2024. 3

[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 10684–10695, 2022. 2, 5

[46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. *International journal of computer vision*, 115:211–252, 2015. 2, 4

[47] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep

language understanding. *Advances in neural information processing systems (NeurIPS)*, 35:36479–36494, 2022. 2

[48] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. *Advances in Neural Information Processing Systems (NeurIPS)*, 35:25278–25294, 2022. 4, 5

[49] Rahulkumar Sunil Singh, Subbu Venkata Satyasri Harsha Pathapati, Michael L. Free, and Prashant K Sarswat. Identification and Separation of E-Waste Components Using Modified Image Recognition Model Based on Advanced Deep Learning Tools, chapter 10, pages 115–127. John Wiley & Sons, Ltd, 2024. 4

[50] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In *International conference on machine learning (ICML)*, pages 1139–1147. PMLR, 2013. 6

[51] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In *International conference on machine learning (ICML)*, pages 6105–6114. PMLR, 2019. 4

[52] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. *arXiv preprint arXiv:2312.11805*, 2023. 1

[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*, 2023. 1

[54] Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (clip). *Advances in Neural Information Processing Systems (NeurIPS)*, 36, 2024. 2

[55] Kevin Vogt-Lowell, Noah Lee, Theodoros Tsiligkaridis, and Marc Vaillant. Robust fine-tuning of vision-language models for domain generalization. In *2023 IEEE High Performance Extreme Computing Conference (HPEC)*, pages 1–7. IEEE, 2023. 2

[56] Chenguang Wang, Ruoxi Jia, Xin Liu, and Dawn Song. Benchmarking zero-shot robustness of multimodal foundation models: A pilot study. *arXiv preprint arXiv:2403.10499*, 2024. 2

[57] Ting-Wei Wu, Hua Zhang, Wei Peng, Fan Lü, and Pin-Jing He. Applications of convolutional neural networks for intelligent waste identification and recycling: A review. *Resources, Conservation and Recycling*, 190:106813, 2023. 4

[58] Mindy Yang and Gary Thung. trashnet. https://github.com/garythung/trashnet, 2016. 5

[59] Dmitry Yudin, Nikita Zakharenko, Artem Smetanin, Roman Filonov, Margarita Kichik, Vladislav Kuznetsov, Dmitry Larichev, Evgeny Gudov, Semen Budennyy, and Aleksandr Panov. Hierarchical waste detection with weakly supervised segmentation in images from recycling plants. *Engineering Applications of Artificial Intelligence*, 128:107542, 2024. 4
[60] Can Zhang, Xu Zhang, Dawei Tu, and Ying Wang. Intel- ligent garbage detection system based on neural networks. In *International Conference on Image Processing and Intel- ligent Control (IPIC 2021)*, volume 11928, pages 252–257. SPIE, 2021. 4

[61] Jing Zhang, Irving Fang, Hao Wu, Akshat Kaushik, Al- ice Rodriguez, Hanwen Zhao, Juexiao Zhang, Zhuo Zheng, Radu Iovita, and Chen Feng. Luwa dataset: Learning lithic use-wear analysis on microscopic images. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 22563–22573, 2024. 4

[62] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image BERT pre- training with online tokenizer. In *International Conference on Learning Representations*, 2022. 2, 6
## A. Detailed Accuracy Metrics

<table><caption>Table 6. Linear probing accuracy of pre-trained DINO models of various sizes. Models were linearly probed on the Research Hall dataset and evaluated on test sets of five datasets. The last column shows the average rank of each model over all datasets</caption><thead><tr><th>DINO</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th><th>rk</th></tr></thead><tbody><tr><td>v2 ViT-L/14</td><td>79.3</td><td>80.9</td><td>50.1</td><td><strong>48.2</strong></td><td><strong>89.9</strong></td><td>2.6</td></tr><tr><td>v2 ViT-g/14</td><td><strong>81.7</strong></td><td>80.2</td><td>52.5</td><td>33.5</td><td>87.2</td><td>2.8</td></tr><tr><td>ViT-B/8</td><td>81.5</td><td><strong>81.0</strong></td><td>51.3</td><td>43.8</td><td>75.5</td><td>2.8</td></tr><tr><td>v2 ViT-B/14</td><td>79.1</td><td>78.3</td><td>49.1</td><td>35.6</td><td>84.6</td><td>4.6</td></tr><tr><td>RN 50</td><td>80.0</td><td>78.0</td><td><strong>52.6</strong></td><td>30.5</td><td>74.5</td><td>5.2</td></tr><tr><td>v2 ViT-S/14</td><td>77.4</td><td>78.3</td><td>48.4</td><td>31.7</td><td><strong>89.9</strong></td><td>5.4</td></tr><tr><td>ViT-B/16</td><td>78.5</td><td>78.5</td><td>50.9</td><td>19.8</td><td>78.2</td><td>5.6</td></tr><tr><td>ViT-S/8</td><td>77.4</td><td>77.8</td><td>48.9</td><td>34.8</td><td>74.5</td><td>6.8</td></tr><tr><td>ViT-S/16</td><td>77.0</td><td>74.8</td><td>46.7</td><td>27.0</td><td>75.5</td><td>8.2</td></tr></tbody></table>

<table><caption>Table 7. Linear probing accuracy of pre-trained CLIP models of various sizes. Models were linearly probed on the Research Hall dataset and evaluated on test sets of five datasets. The last column shows the average rank of each model over all datasets</caption><thead><tr><th>CLIP</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th><th>rk</th></tr></thead><tbody><tr><td>ViT-L/14</td><td><strong>80.0</strong></td><td><strong>77.5</strong></td><td><strong>56.2</strong></td><td>25.5</td><td><strong>88.8</strong></td><td>2.4</td></tr><tr><td>RN50x64</td><td>79.8</td><td>70.9</td><td>56.0</td><td>47.2</td><td>81.9</td><td>2.8</td></tr><tr><td>RN50x16</td><td>77.6</td><td>72.3</td><td>52.6</td><td>40.7</td><td>79.3</td><td>3.6</td></tr><tr><td>ViT-B/16</td><td>73.9</td><td>73.4</td><td>51.3</td><td><strong>65.6</strong></td><td>76.6</td><td>4.2</td></tr><tr><td>ViT-B/32</td><td>73.7</td><td>76.9</td><td>50.9</td><td>34.1</td><td>76.6</td><td>5.2</td></tr><tr><td>RN50x4</td><td>72.0</td><td>77.4</td><td>50.6</td><td>30.0</td><td>78.2</td><td>5.6</td></tr><tr><td>RN 101</td><td>73.0</td><td>69.6</td><td>48.2</td><td>37.0</td><td>79.8</td><td>5.8</td></tr><tr><td>RN 50</td><td>74.3</td><td>56.2</td><td>51.8</td><td>28.0</td><td>73.4</td><td>6.2</td></tr></tbody></table>

<table><caption>Table 8. Accuracy of DINOv2 ViT-S/14 with a linear classifier when fine-tuning some of the last layers of the model. Models were trained on the Research Hall dataset and evaluated on the test sets of all datasets.</caption><thead><tr><th>Unfrozen layers</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th></tr></thead><tbody><tr><td>0 layer</td><td>77.4</td><td>78.3</td><td>48.4</td><td>31.7</td><td><strong>89.9</strong></td></tr><tr><td>1 layer</td><td>80.0</td><td>79.3</td><td>50.8</td><td><strong>42.3</strong></td><td>83.0</td></tr><tr><td>2 layers</td><td><strong>80.2</strong></td><td>80.2</td><td><strong>54.0</strong></td><td>31.4</td><td>80.0</td></tr><tr><td>3 layers</td><td>79.6</td><td><strong>80.8</strong></td><td>52.9</td><td>42.2</td><td>78.9</td></tr><tr><td>6 layers</td><td>79.8</td><td>76.8</td><td>50.3</td><td>32.8</td><td>62.8</td></tr><tr><td>12 (all) layers</td><td>69.6</td><td>71.9</td><td>41.5</td><td>33.0</td><td>31.9</td></tr></tbody></table>

<table><caption>Table 9. Accuracy of fine-tuned and non-fine-tuned foundation models with an added linear layer. Models were trained on the Research Hall dataset and evaluated on the test sets of all five datasets. Fine-tuning was performed using partial fine-tuning (training the last 3 layers) or LoRA, a PEFT technique</caption><thead><tr><th>Model</th><th>FT</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th></tr></thead><tbody><tr><td rowspan="3">DINOv2<br>ViT-S/14</td><td>PEFT</td><td><strong>82.4</strong></td><td><strong>82.8</strong></td><td>49.5</td><td>39.6</td><td>87.2</td></tr><tr><td>3 layers</td><td>79.6</td><td>80.8</td><td>52.9</td><td>42.2</td><td>78.9</td></tr><tr><td>no FT</td><td>77.4</td><td>78.3</td><td>48.4</td><td>31.7</td><td><strong>89.9</strong></td></tr><tr><td rowspan="3">DINOv2<br>ViT-B/14</td><td>PEFT</td><td>81.3</td><td>80.7</td><td><strong>52.6</strong></td><td><strong>36.8</strong></td><td><strong>87.8</strong></td></tr><tr><td>3 layers</td><td><strong>81.7</strong></td><td><strong>81.5</strong></td><td>49.9</td><td>36.4</td><td>83.5</td></tr><tr><td>no FT</td><td>79.1</td><td>78.3</td><td>49.1</td><td>35.6</td><td>84.6</td></tr><tr><td rowspan="3">DINOv2<br>ViT-L/14</td><td>PEFT</td><td><strong>83.9</strong></td><td><strong>83.4</strong></td><td><strong>56.9</strong></td><td>41.4</td><td><strong>91.0</strong></td></tr><tr><td>3 layers</td><td>83.7</td><td>81.3</td><td>56.1</td><td>33.8</td><td>90.4</td></tr><tr><td>no FT</td><td>79.3</td><td>80.9</td><td>50.1</td><td><strong>48.2</strong></td><td>89.9</td></tr><tr><td rowspan="3">DINOv2<br>ViT-g/14</td><td>PEFT</td><td><strong>87.6</strong></td><td><strong>82.0</strong></td><td><strong>56.7</strong></td><td>35.1</td><td><strong>89.9</strong></td></tr><tr><td>3 layers</td><td>83.7</td><td>80.5</td><td>54.0</td><td><strong>46.6</strong></td><td>89.4</td></tr><tr><td>no FT</td><td>81.7</td><td>80.2</td><td>52.5</td><td>33.5</td><td>87.2</td></tr><tr><td rowspan="3">CLIP<br>ViT-B/16</td><td>PEFT</td><td><strong>80.0</strong></td><td>76.6</td><td><strong>54.4</strong></td><td>53.7</td><td><strong>85.6</strong></td></tr><tr><td>3 layers</td><td>78.7</td><td><strong>77.1</strong></td><td>51.9</td><td>49.1</td><td>82.4</td></tr><tr><td>no FT</td><td>73.9</td><td>73.4</td><td>51.3</td><td><strong>65.6</strong></td><td>76.6</td></tr><tr><td rowspan="3">CLIP<br>ViT-L/14</td><td>PEFT</td><td><strong>83.9</strong></td><td><strong>83.4</strong></td><td><strong>56.9</strong></td><td><strong>41.4</strong></td><td>91.0</td></tr><tr><td>3 layers</td><td>81.3</td><td>82.9</td><td>53.6</td><td>37.8</td><td><strong>91.5</strong></td></tr><tr><td>no FT</td><td>80.0</td><td>77.5</td><td>56.2</td><td>25.5</td><td>88.8</td></tr></tbody></table>
<table><caption>Table 10. Accuracy of foundation models trained with and without data augmentation. Models were linearly probed on the Research Hall dataset and evaluated on the test sets of all five datasets. Data augmentation was performed using offline augmentation (each image is augmented 10 times prior to training) and online augmentation (each image is randomly augmented at each epoch).</caption><thead><tr><th>Model</th><th>Aug.</th><th>Hall</th><th>Alu</th><th>N&P</th><th>ZW</th><th>TN</th></tr></thead><tbody><tr><td rowspan="3">DINOv2<br>ViT-S/14</td><td>offline</td><td><u>77.8</u></td><td><u>80.5</u></td><td><u>49.3</u></td><td><u>22.8</u></td><td><u>87.4</u></td></tr><tr><td>online</td><td><u>79.6</u></td><td><u>81.9</u></td><td><u>49.4</u></td><td><u>23.6</u></td><td><u>88.5</u></td></tr><tr><td>no aug.</td><td>77.4</td><td>78.3</td><td>48.4</td><td><u>31.7</u></td><td><u>89.9</u></td></tr><tr><td rowspan="3">DINOv2<br>ViT-B/14</td><td>offline</td><td><u>80.4</u></td><td>77.1</td><td>48.9</td><td><u>27.8</u></td><td><u>87.2</u></td></tr><tr><td>online</td><td><u>79.6</u></td><td>77.7</td><td><u>50.2</u></td><td><u>27.1</u></td><td><u>90.4</u></td></tr><tr><td>no aug.</td><td>79.1</td><td><u>78.3</u></td><td><u>49.1</u></td><td><u>35.6</u></td><td>84.6</td></tr><tr><td rowspan="3">DINOv2<br>ViT-L/14</td><td>offline</td><td><u>82.1</u></td><td><u>81.2</u></td><td><u>52.7</u></td><td><u>47.2</u></td><td><u>90.2</u></td></tr><tr><td>online</td><td><u>83.0</u></td><td><u>82.3</u></td><td><u>53.7</u></td><td><u>44.9</u></td><td><u>90.8</u></td></tr><tr><td>no aug.</td><td>79.3</td><td>80.9</td><td>50.1</td><td><u>48.2</u></td><td>89.9</td></tr><tr><td rowspan="3">DINOv2<br>ViT-g/14</td><td>offline</td><td><u>83.3</u></td><td>77.1</td><td>51.9</td><td><u>29.8</u></td><td><u>88.1</u></td></tr><tr><td>online</td><td><u>84.5</u></td><td><u>78.2</u></td><td><u>54.0</u></td><td><u>30.2</u></td><td><u>89.0</u></td></tr><tr><td>no aug.</td><td>81.7</td><td><u>80.2</u></td><td><u>52.5</u></td><td><u>33.5</u></td><td>87.2</td></tr><tr><td rowspan="3">CLIP<br>ViT-B/16</td><td>offline</td><td><u>74.6</u></td><td><u>78.9</u></td><td><u>51.2</u></td><td><u>49.6</u></td><td><u>78.5</u></td></tr><tr><td>online</td><td><u>75.6</u></td><td><u>79.9</u></td><td><u>54.0</u></td><td><u>48.6</u></td><td><u>79.4</u></td></tr><tr><td>no aug.</td><td>73.9</td><td>73.4</td><td><u>51.3</u></td><td><u>65.6</u></td><td>76.6</td></tr><tr><td rowspan="3">CLIP<br>ViT-L/14</td><td>offline</td><td>79.8</td><td><u>80.5</u></td><td>51.2</td><td>18.8</td><td>88.5</td></tr><tr><td>online</td><td><u>80.9</u></td><td><u>79.8</u></td><td><u>53.1</u></td><td><u>19.6</u></td><td><u>92.6</u></td></tr><tr><td>no aug.</td><td><u>80.0</u></td><td>77.5</td><td><u>56.2</u></td><td><u>25.5</u></td><td>88.8</td></tr><tr><td rowspan="3">ImageBind</td><td>offline</td><td><u>81.6</u></td><td><u>83.3</u></td><td><u>56.3</u></td><td><u>41.5</u></td><td>88.8</td></tr><tr><td>online</td><td><u>80.7</u></td><td><u>82.0</u></td><td><u>55.3</u></td><td><u>37.1</u></td><td><u>89.4</u></td></tr><tr><td>no aug.</td><td>80.4</td><td><u>82.0</u></td><td>54.7</td><td><u>55.3</u></td><td><u>91.0</u></td></tr></tbody></table>

## B. Labels repartition

<table><caption>Table 11. Repartition of the labels of the Aluminum stream dataset in both train (Research hall) and test (Aluminum stream) dataset.</caption><thead><tr><th></th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>Metal aerosol</td><td>0,063</td><td>0,022</td></tr><tr><td>Metal drink can</td><td>0,078</td><td>0,620</td></tr><tr><td>LDPE film</td><td>0,061</td><td>0,255</td></tr><tr><td>Tetrapack</td><td>0,041</td><td>0,051</td></tr></tbody></table>

<table><caption>Table 12. Repartition of the labels of the News & Pams stream dataset in both train (research hall) and test (News & Pams stream) dataset.</caption><thead><tr><th></th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>Graphic paper</td><td>0,072</td><td>0,309</td></tr><tr><td>Mixed-paper</td><td>0,041</td><td>0,187</td></tr><tr><td>Graphic news&amp;pams</td><td>0,15</td><td>0,289</td></tr><tr><td>Cardboard</td><td>0,124</td><td>0,111</td></tr><tr><td>Metal</td><td>0,178</td><td>0,004</td></tr><tr><td>Plastic</td><td>0,393</td><td>0,095</td></tr><tr><td>Tetrapack</td><td>0,041</td><td>0,006</td></tr></tbody></table>

<table><caption>Table 13. Repartition of the labels of the ZeroWaste dataset in both train (research hall) and test (ZeroWaste) dataset. Paper proportion is 0 because only intruder classes were annotated in this dataset.</caption><thead><tr><th></th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>Paper</td><td>0,263</td><td>0</td></tr><tr><td>Cardboard</td><td>0,124</td><td>0,679</td></tr><tr><td>Metal</td><td>0,178</td><td>0,012</td></tr><tr><td>Rigid plastic</td><td>0,374</td><td>0,062</td></tr><tr><td>Soft plastic</td><td>0,061</td><td>0,246</td></tr></tbody></table>

<table><caption>Table 14. Repartition of the labels of the Trashnet dataset in both train (research hall) and test (Trashnet) dataset.</caption><thead><tr><th></th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>Paper</td><td>0,263</td><td>0,298</td></tr><tr><td>Cardboard</td><td>0,124</td><td>0,229</td></tr><tr><td>Metal</td><td>0,178</td><td>0,207</td></tr><tr><td>Plastic</td><td>0,435</td><td>0,266</td></tr></tbody></table>
<table><caption>Table 15. Labels and their respective proportions in the research hall dataset, as well as the corresponding labels in all other datasets.</caption><thead><tr><th>Research hall</th><th>Prop.</th><th>Aluminum stream</th><th>News & Pams stream</th><th>Zero Waste</th><th>TrashNet</th></tr></thead><tbody><tr><td>Graphic paper</td><td>0,072</td><td>Other</td><td>Graphic paper</td><td>Paper</td><td>Paper</td></tr><tr><td>Mixed-paper colored-paper</td><td>0,004</td><td>Other</td><td>Mixed-paper</td><td>Paper</td><td>Paper</td></tr><tr><td>Mixed-paper other-indeterminate</td><td>0,011</td><td>Other</td><td>Mixed-paper</td><td>Paper</td><td>Paper</td></tr><tr><td>Mixed-paper various-cardboard</td><td>0,026</td><td>Other</td><td>Mixed-paper</td><td>Paper</td><td>Paper</td></tr><tr><td>Graphic news&amp;pams</td><td>0,150</td><td>Other</td><td>Graphic news&amp;pams</td><td>Paper</td><td>Paper</td></tr><tr><td>Cardboard corrugated</td><td>0,037</td><td>Other</td><td>Cardboard</td><td>Cardboard</td><td>Cardboard</td></tr><tr><td>Cardboards linerboard-thin</td><td>0,087</td><td>Other</td><td>Cardboard</td><td>Cardboard</td><td>Cardboard</td></tr><tr><td>Steel-aluminum tray</td><td>0,037</td><td>Other</td><td>Metal</td><td>Metal</td><td>Metal</td></tr><tr><td>Steel-aluminum aerosol</td><td>0,063</td><td>Metal aerosol</td><td>Metal</td><td>Metal</td><td>Metal</td></tr><tr><td>Steel-aluminum drink-can</td><td>0,078</td><td>Metal drink can</td><td>Metal</td><td>Metal</td><td>Metal</td></tr><tr><td>PET-clear food bottle</td><td>0,054</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PP non-food bottle</td><td>0,002</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>HDPE non-food container</td><td>0,022</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>HDPE-opaque milk-bottle</td><td>0,043</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>LDPE wrapping</td><td>0,061</td><td>LDPE film</td><td>Plastic</td><td>Soft plastic</td><td>Plastic</td></tr><tr><td>PET-opaque food-container</td><td>0,011</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PET-clear food tray</td><td>0,007</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PET-colored food bottles</td><td>0,028</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PET-opaque milk bottle</td><td>0,004</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PET-opaque food non-milk bottle</td><td>0,001</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PP food-container tray opaque</td><td>0,035</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PP yogurt/dairy pot</td><td>0,091</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>PS other-indeterminate</td><td>0,035</td><td>Other</td><td>Plastic</td><td>Rigid plastic</td><td>Plastic</td></tr><tr><td>Tetrapack</td><td>0,041</td><td>Tetrapack</td><td>Tetrapack</td><td>Rigid plastic</td><td>Plastic</td></tr></tbody></table>