fine-tuning with very large dropout

## Abstract

Abstract It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution. Several authors have recently used ensemble techniques to show how scenarios involving multiple data distributions are best served by representations that are both richer than those obtained by regularizing for the best in-distribution performance, and richer than those obtained under the influence of the implicit sparsity bias of common stochastic gradient procedures. This contribution investigates the use of very high dropout rates instead of ensembles to obtain such rich representations. Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups. This result has practical significance because the importance of the fine-tuning scenario has considerably grown in recent years. This result also provides interesting insights on the nature of rich representations and on the intrinsically linear nature of fine-tuning a large network using a comparatively small dataset.

## 1 Introduction

The practice of machine learning has been shaped by the assumption that training and testing examples are independently drawn from the same unknown probability distribution. This is seldom the case in modern settings, not only because this i.i.d. assumption breaks down for the problems of interest, but also because it is often convenient to use multiple datasets that are known to follow different distributions. For instance, we may pre-train a deep network on a large dataset, fine-tune it on a smaller dataset specific to the task of interest, and test on a collection of tasks designed to benchmark various aspects of the system.

Many of the tenets of machine learning should therefore be regarded with healthy suspicion. For instance, under the i.i.d. assumption, favoring solutions with sparse representations has well-known benefits on the generalization performance. Yet, several authors (Zhang et al., ; Zhang & Bottou, ; Chen et al., ) make the point that scenarios involving multiple distributions are best served by “*richer representations*” that contain redundant features, that is, features that do not improve the model performance on the training distribution, but could prove helpful when the distribution changes.

It would be nice to construct such rich representations by merely optimizing the expectation of a suitable loss function for a single training distribution, for instance using stochastic gradient techniques.
Alas, this hope is contradicted by the implicit sparsity bias of stochastic gradient algorithms (Andriushchenko et al., ; Blanc et al., ). In a nutshell, a feature only survives when it brings an incremental training error advantage relative to what can be achieved using all the other features already present in the network. We slightly abuse the terminology and call them “strongly relevant”. However, features that are not strongly relevant might nevertheless

- (a)
be incrementally useful when the data follows a different distributions of interest, or
- (b)
be useful under the training distribution when added to certain subsets of the other existing features instead of all of them (“weakly relevant”).

It is therefore tempting to “enrich" the representation with features of type (b), which can be found using the training data, and hope that some of these will turn out to also be features of type (a) whose inclusion helps when the data distribution changes.

The dropout technique (Srivastava et al., ) seems well suited to finding weakly relevant features because randomly masking units of a representation layer during training amounts to forming random subsets of all other available features. However, in order to form small subsets, one would have to use very high levels of dropout. Unfortunately, training a sizable deep network from scratch with such a large dropout is practically impossible. Instead, computationally demanding methods, such as adversarial sampling (Zhang et al., ; Chen et al., ) and representation ensembles (Zhang & Bottou, ), have been proposed to find weakly relevant features while training a network from scratch.

There is however a practically meaningful scenario in which we can use an extremely aggressive dropout: fine-tuning a pre-trained network using a comparatively small dataset. This is possible because such a fine-tuning operation makes only modest changes to the network weights. For example, several authors (Ramé et al., ; Wortsman et al., ) argue that fine-tuned networks remain “linearly connected”, that is averaging the parameters of multiple fine-tuned networks approximate the ensemble of these networks. Evci et al. () even show that a linear classifier on top of the union of internal-layer features of pre-trained residual networks can match or exceed the performance of fine-tuning.

In the present work, we adopt the out-of-distribution fine-tuning setup (three-distributions) of Ramé et al. (). In this framework, we have access to a model pre-trained using a large dataset for a task weakly related to the task of interest. This pre-trained model is then fine-tuned on datasets that illustrate the task of interest, and then tested on a dataset for the same task but with a different distribution.
However, instead of enriching the representations by constructing ensembles (Zhang & Bottou, ) or averaging weights (Ramé et al., ; ; Wortsman et al., ), we simply *fine-tune using very large dropout levels*, randomly masking above 90% of the units in the representation layer. We find that this simple approach *exceeds the performance of both ensemble and weight-averaging methods*. This result is not only *practically meaningful*, but also clarifies the idea of *richer representation*.

## 2 Related Work

##### Constructing versatile representations

Reusing or transferring features across related tasks has been commonplace for more than one decade (Collobert et al., ; Bottou, ; Sharif Razavian et al., ) and plays a fundamental role in the appeal of foundational models (Bommasani et al., ). However, once the optimization process has identified a set of features that is sufficient to achieve near-optimal performance on the training set, additional features are often discarded because they do not bring an incremental benefit to the training error, despite the fact that they may independently carry useful information (Zhang & Bottou, ).

Researchers have devised ways to obtain more versatile representations by engineering a diversity of datasets, architectures, and even hyper-parameters (Chen et al., ; Wang et al., ; Dvornik et al., ; Bilen & Vedaldi, ; Gontijo-Lopes et al., ; Li et al., ; ; Chowdhury et al., ), as an alternative to the most popular approach which consists of simply using ever larger datasets (Bommasani et al., ).

Interesting results have also been obtained without engineering diversity and without increasing the dataset sizes. Zhang et al. () and Chen et al. () propose to discover rich representation through multiple training episodes that adversarially reweigh the training dataset to impede the use of previously learned features. Zhang & Bottou () show that surprisingly good results can be obtained by concatenating the representations of multiple networks that are trained in exactly the same way, save for the random seed used in the stochastic gradient process.

##### Fine-tuning as a near-linear process

Although modern deep residual networks feature highly complex nonconvex cost functions, several authors have shown that their final training phase remains mostly confined to a nearly-convex attraction basin (Izmailov et al., ; Li et al., ; Frankle et al., ). The same observation holds when fine-tuning a large pre-trained network using a dataset whose size is considerably smaller than the dataset size one would need to train such a large network from scratch. As long as one starts from the same pre-trained model, Wortsman et al. () and Ramé et al. (; ) observe that averaging the weights of diverse fine-tuned models can reproduce the i.i.d. and o.o.d. performances of the ensemble of these models, implying that fine-tuning is a near-linear process.

Maddox et al. () and Mu et al. () propose instead to approximate the fine-tuning process with a first-order Taylor expansion, obtaining a linear system operating on top of the ntk features. Evci et al. () match the performance of fine-tuning by merely learning a strongly regularized linear model that takes all internal layer states as inputs. Meanwhile, Yu et al. () efficiently fine-tune large foundational language models by essentially restricting the weight updates to low dimensional manifolds.

##### Fine-tuning with very large dropout

Our contribution advocates using very large dropout in the fine-tuning scenario in order to force the learning algorithm to create a redundant representation without specifically engineering diversity. We do not seek to propose new dropout variations (Chu et al., ), understand dropout from either an overfitting/underfitting perspective (Liu et al., ) or from a Bayesian perspective (Gal & Ghahramani, ).

## 3 Fine-tuning and dropout

### 3.1 The three-distributions setup

The two-distributions setup is a commonly used for transfer learning. In this setup, features $\Psi$ are obtained by pre-training a network on a large training set associated with a first distribution $\mathcal{T}_{\text{p}}$. These features are then used to construct or initialize a new model $\omega_{\text{d}}\circ\Psi$, which is then trained using a smaller training set associated with a second distribution $\mathcal{T}_{\text{d}}$. The question is to determine which pre-training approach is most likely to make the features $\Psi$ useful for the transfer task $\mathcal{T}_{\text{d}}$.

The three-distributions setup (Ramé et al., ) views the pre-trained model as a base model that is assumed very rich but whose training process is beyond our control (e.g., a fundational model). The features $\Psi$ of the pre-trained model are then incorporated into a new model $\omega_{\text{d}}\circ\Psi$ that is fine-tuned using a second distribution $\mathcal{T}_{\text{d}}$ and eventually tested on a third distribution $\tilde{\mathcal{T}}_{\text{d}}$ illustrating the same general task as the second distribution (e.g., using the same classification labels.) The question is then to determine which fine-tuning approach is most likely to produce a model that will perform robustly under the eventual testing distribution $\tilde{\mathcal{T}}_{\text{d}}$.

### 3.2 Method

The key results described later in this paper have been obtained with a very simple method. The base model is a deep learning network with residual connections trained on data $\mathcal{T}_{\text{p}}$ that is related to but substantially larger than the datasets illustrating the task of interest. Some of these datasets ($\mathcal{T}_{d}$) are used to fine-tune the base model. Performance is reported on both held-out data from the fine-tuning datasets (i.i.d. performance on $\mathcal{T}_{d}$) and data from the remaining datasets (o.o.d. performance on $\tilde{\mathcal{T}}_{d}$).

We focus on residual networks because fine-tuning has been found to hardly change the inner layers of non-residual networks (Raghu et al., , fig 2).
In contrast, skip connections in residual networks expose the inner block features in such a manner that the fine-tuning process can utilize these features in a near-linear way (Evci et al., ).

Fine-tuning is carried out with a standard stochastic learning procedure (e.g. sgd or adam) after applying a very large dropout to the penultimate layer representation $\Phi$. Unlike (Srivastava et al., ), we only apply dropout on the penultimate layer representation $\Phi$,(^1^11Except in Figure 6 in the appendix where we investigate the effect of dropping entire inner residual blocks during fine-tuning.) because skip connections in residual networks expose many inner-layer features to the last linear layer, as illustrated by the decomposition of residual networks proposed by Veit et al. (),

$$ $\displaystyle\Phi(x)$ $\displaystyle=\underbrace{\vphantom{f_{1}(x)}x}_{\mathclap{\phi_{0}(x)}}+ \underbrace{f_{1}(x)}_{\phi_{1}(x)}+\underbrace{f_{2}(x+f_{1}(x))}_{\phi_{2}(x )}+\dots=\sum_{i\in[0,\dots,l]}\phi_{i}(x)\in\mathbb{R}^{p}\,,$ (1) $$

where $f_{i}$ represents the function implemented by the $i$-th residual block, and

$$ $\displaystyle\Phi_{\mathrm{dropout}}(x)=\frac{m(\lambda)}{1-\lambda}\odot\Phi( x)\,,$ (2) $$

where $\odot$ represents the component-wise product and $m(\lambda)$ is a vector of random Bernoulli variables equal to 0 0 with probability $\lambda$ and $1$ with probability $1-\lambda$. The additive decomposition of $\Phi(x)$ in equation equation 1 makes clear that applying dropout to $\Phi(x)$ simultaneously blocks the contributions $\phi_{i}(x)$ of all residual blocks.

## 4 Experiments

We perform most experiments using pacs (Li et al., ), vlcs (Fang et al., ), office home (Venkateswara et al., ), and terra  incognita (Beery et al., ) datasets. These datasets are part of the DomainBed suite which has been widely used for this kind of experiments.(^2^22[https://github.com/facebookresearch/DomainBed](https://github.com/facebookresearch/DomainBed)) With $9,991$ to $24,788$ examples, they are substantially smaller than the pre-training dataset ImageNet with 1.2m examples. We also use the larger domainnet dataset (Peng et al., ), 0.58m examples, to show that linear connectivity breaks down when the fine-tuning dataset size becomes comparable with the pre-training dataset size and justifies carrying out many more fine-tuning iterations.

Each of these datasets is divided into four sub-datasets that share the same target label categories but follow a different distribution. For example, one sub-dataset of pacs contains simple sketch images of ‘dog’ and ‘elephant’, while another sub-dataset contains real photos of ‘dog’ and ‘elephant’. This makes it possible to conveniently evaluate o.o.d. performance by fine-tuning on three sub-datasets and testing on the fourth one.

Fine-tuning experiments are carried out on two widely used residual architectures: convolutional residual networks, ResNets (He et al., ), and vision transformers, ViTs (Dosovitskiy et al., ).
Unless otherwise stated, all convolutional residual network experiments are carried out using the ResNet50 neural network pre-trained on ImageNet with substantial data augmentations(^3^33[https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/](https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/)) such as trivialaugment (Müller & Hutter, ), cutmix (Yun et al., ), and random erasings (Zhong et al., ). These augmentations mimic the properties of large foundational models that learn substantial diverse features using very large and diverse pre-training data. We refer to this network as ResNet50 #2 as opposed to the original ResNet50 #1 recipe described by He et al. ().
Similarly, all visual transformer fine-tuning experiments leverage a ImageNet pre-trained ViT-L-16 model(^4^44[https://github.com/pytorch/vision/tree/main/references/classification#vit_l_16](https://github.com/pytorch/vision/tree/main/references/classification#vit_l_16)) with 304m parameters.

Figure: Figure 1: o.o.d. performance comparison between very large dropout, ensembles, and weight averaging methods on four DomainBed tasks and two backbones (ResNet50 and ViT-L-16). Baseline results were obtained using plain fine-tuning with different hyperparameters. Weight averaging results either average the model weights collected every 300 iterations along each fine-tuning trajectory or the final model weights of all fine-tuning trajectories as in (Ramé et al., ). Ensemble results average instead the model outputs. Finally, large dropout results were obtained like the baseline results but using a 90% dropout rate on the penultimate layer. Each box summarizes the results obtained with 1296 hyper-parameters combinations (no hyper-parameter selection).
Refer to caption: x1.png

**Table 1: o.o.d. performance comparison between very large dropout, ensembles, and weight averaging methods after hyperparameter selection. The hyperparameter is selected according to the best i.i.d. performance.**
|  | dataset | baseline | weight avg<br>(single) | ensemble<br>(single) | large<br>dropout | weight avg<br>(multi) | ensemble<br>(multi) |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | vlcs | 78.3 | 79.4 | 79.6 | 80.1 | 78.8 | 79.1 |
|  | office home | 71.4 | 72.2 | 72.3 | 73.6 | 71.3 | 71.3 |
| ResNet | pacs | 87.3 | 86.9 | 87.3 | 88.5 | 87.0 | 87.1 |
| terra incognita | 51.0 | 53.1 | 52.3 | 53.9 | 52.0 | 52.5 |  |
| Average | 72.0 | 72.9 | 72.9 | 74.0 | 72.3 | 72.5 |  |
| vlcs | 78.1 | 78.1 | 77.9 | 79.0 | 78.4 | 78.4 |  |
| office home | 74.6 | 74.8 | 74.8 | 74.6 | 74.5 | 74.6 |  |
| ViT-L-16 | pacs | 85.0 | 84.2 | 84.3 | 86.0 | 84.7 | 84.8 |
| terra incognita | 44.4 | 45.1 | 44.8 | 45.8 | 44.1 | 44.0 |  |
| Average | 70.5 | 70.6 | 70.5 | 71.4 | 70.4 | 70.5 |  |

### 4.1 Very large dropout yields better o. o. d. performance

Using these same datasets, Gulrajani & Lopez-Paz () argue that plain Empirical Risk Minimization (erm) equals and often betters the o.o.d. performance of purposefully designed methods, such as coral (Sun & Saenko, ), vrex (Krueger et al., ), and irm (Arjovsky et al., ). More recently, Arpit et al. (), Cha et al. (), Ramé et al. (), and Ramé et al. () find that ensemble and weight averaging methods consistently outperform the o.o.d. performance of erm. We now show that fine-tuning with very large dropout outperforms the o.o.d. performance of these state-of-the-art methods.(^5^55Code: [https://github.com/TjuJianyu/verylarge_dropout](https://github.com/TjuJianyu/verylarge_dropout))

Following these earlier works, we focus on the o.o.d. performance of these methods because this is the testing performance that matters for the practical situations that the three-distribution setup targets. The i.i.d. performance must then be viewed as a secondary indicator, akin to the training set performance, which could be used to cast a light on the training process but cannot be used as a predictor of the system performance under its actual testing distribution.

Figure 1 reports the o.o.d. performances of the following approaches.

- •
Baseline results are obtained by fine-tuning our ResNet50 or ViT-L-16 using sgd with 0.9 momentum for $10,000$ iterations.(^6^66We use a batch size $32$ for all ResNet fine-tunings, and reduce the batch size to $16$ for all ViT-L-16 fine-tunings due to the vram constraint.) A 10% learning rate decay is applied at $5000^{th}$ iterations. For each choice of three training sub-datasets, we repeat three experiments for each combination of learning rate in $\{10^{-3},5.10^{-4}\}$ and L2 weight decay in $\{10^{-4},5.10^{-5},10^{-5}\}$. We measure the i.i.d. performance using 20% examples held from the training data, and we measure o.o.d. performance on the fourth sub-dataset. Each box summarizes the performances obtained using two learning rate choices and three weight decay choices, for four possible training dataset choices, resulting in $(2\times 3)^{4}=1296$ results. Following Gulrajani & Lopez-Paz (), we prevent overfitting by early-stopping at the best i.i.d. performance.
- •
Dropout results are obtained using the same protocol but using a 90% dropout rate on the penultimate layer representation.
- •
Ensemble results are obtained in two ways, either using an ensemble of checkpoints collected along each fine-tuning trajectory, or using the ensemble of the final checkpoints collected along all fine-tuning trajectories with different hyper-parameters.
- •
Weight averaging results approximate the corresponding ensembling results by averaging the model weights instead of averaging the model outputs.

As expected, both ensemble methods (Ueda & Nakano, ; Dietterich, )
and their weight averaging approximation (Ramé et al., ; Wortsman et al., )
improve on the o.o.d. baseline performance. However, fine-tuning with a very large dropout outperforms the o.o.d. performance of both ensemble and weight averaging methods. There is even a *large gap* between the worst dropout results and the best ensemble results for the office home and pacs datasets.
Figure 7 in the Appendix shows that the i.i.d. performance of the large dropout method lags behind that of ensembles, revealing that the o.o.d. performance improvement is not a secondary effect of some i.i.d. performance improvement (i.e. the o.o.d. performance gaps in o.o.d. Figure 1 do not come from i.i.d. overfitting/underfitting.).

The box plots in Figure 1 summarize the o.o.d. performance of all possible hyper-parameter combinations. Following again Gulrajani & Lopez-Paz (),
Table 1 features o.o.d. performances obtained using the hyper-parameter combination that provides the best i.i.d. performance. Because ResNet50 produces a better performance than ViT-L-16 on these o.o.d. fine-tuning tasks, our experiments in the following sections will be conducted on ResNet50.

#### 4.1.1 Optimal o. o. d. dropout rate

To the best of our knowledge, such large dropout rates (90% and above) are considered unsuitable for training a network from scratch and have not been previously used for fine-tuning either. This section illustrates how the optimal dropout rate can be very high in fine-tuning scenarios and falls to small values when one gets closer to training the network from scratch.

Figure 2 compares various dropout rates on the four DomainBed tasks. A $90\%$ dropout rate reliably produces good o.o.d. performance on all four tasks. The optimal dropout rate for o.o.d. performance ranges from 90% to 95% for vlcs and pacs task (with 10k examples). And becomes slightly smaller, about 90%, for the slighlty larger datasets office home and terra incognita (with 15k to 25k examples).

##### Increasing the fine-tuning dataset size to approach the pre-training dataset size

The larger the fine-tuning dataset, the more fine-tuning iterations we can make without overfitting. When the fine-tuning dataset size approaches the pre-training dataset size, the difference between fine-tuning and training from scratch becomes less clear, the linear connectivity property disappears, the linear approximation perspective on fine-tuning no longer holds, and the optimal dropout rate falls sharply. Figure 4 illustrates this effect using the larger domainnet dataset (Peng et al., ) that contains 586k examples (almost half as big as ImageNet) and requires $30,000$ fine-tuning iterations.

##### Training from scratch on the fine-tuning dataset

Figure 4 shows the effect of various dropout rates when one trains a network on the vlcs task from scratch, that is starting from a randomly initialized network without pretraining (i.e. the pretraining dataset size is zero). The optimal dropout rate falls to about zero. Dropout rates higher than 50% have a negative impact on both the i.i.d. and the o.o.d. performance of the network. This suggests again that high dropout rates make it difficult to create new features (a nonlinear operation), but does not prevent leveraging existing features that were possibly buried in the network inner layers (a linear operation). This is the idea of richer representation we discussed in section 1. We will provide other fine-tuning experiments on the richer representation idea in Section 4.3.

Figure: Figure 2: Effect of diverse dropout rates during fine-tuning. The best o.o.d. performances are attained using rates around or above 90%. A large dropout rate (e.g. 90%) reliably produces good o.o.d. performance on all four tasks.
Refer to caption: x9.png

Figure: Figure 3: Comparison of various dropout rates on the larger domainnet dataset (586K examples), whose size approaches the pretraining dataset size (ImageNet, 1.2M examples). The optimal dropout rate falls to about 50%, a value comparable to the dropout rates traditionally used when training from scratch.
Refer to caption: x13.png

### 4.2 Popular fine-tuning techniques do not substantially improve the o. o. d. performance of large dropouts

Various fine-tuning techniques have been proposed to improve the fine-tuning ability to leverage the representations learned by a pre-trained model, such
as using a larger learning rate on the last layer (Caron et al., ; Bardes et al., ; Kumar et al., ) or, as discussed above, using weight averaging and ensemble methods (Ramé et al., ; ; Arpit et al., ). We show in this section that using these techniques *in addition to very large dropout rates* do not yield much o.o.d. performance improvements over using large dropout rates alone. Note that we still expect some incremental benefits because both weight averaging and ensembles reduce the stochastic optimization noise and accelerate training in general (Polyak & Juditsky, ).

#### 4.2.1 Large learning rates for the last layer

Several authors routinely use a larger training rate on the last layer on the intuition that fine-tuning a pre-trained deep network on a different target task entails training a new last layer from scratch (Caron et al., ; Bardes et al., ; Kumar et al., ).

**Table 2: Incremental benefits achieved by applying a $10\times$ larger learning rate in the last layer. The first two columns show that this $10\times$ last-layer learning rate is helpful to baseline. Then the middle two columns show that using a large dropout rate vastly improves the o.o.d. performance of merely using the increased learning rate ($\sim$$1.3\%$). The last two columns reveals that using this $10\times$ larger last-layer training rate yields small or zero incremental improvements over only using a large dropout rate ($\sim$$0.2\%$).**
| dataset | baseline | baseline<br>+ 10$\times$ last-layer lr | baseline<br>+ large dropout | baseline<br>+ large dropout<br>+ 10$\times$ last-layer lr |
| --- | --- | --- | --- | --- |
| vlcs | 78.3 | 79.9 (+1.6) | 80.1 (+1.8) | 80.5 (+2.2) |
| office home | 71.4 | 71.8 (+0.4) | 73.6 (+2.2) | 73.3 (+1.9) |
| pacs | 87.3 | 87.0 (-0.3) | 88.5 (+1.2) | 88.3 (+1.0) |
| terra incognita | 51.0 | 52.2 (+1.2) | 53.9 (+2.9) | 54.9 (+3.9) |
| Average | 72.00 | 72.73 | 74.03 | 74.25 |

Table 2 follows a similar fine-tuning process as in Table 1 but uses a $10\times$ larger training rate for the last layer classifier. Comparing the last two columns in Table 2 shows that using this $10\times$ larger last layer training rate yields small or zero incremental improvements over only using a large dropout rate ($\sim$$0.2\%$). Comparing the middle two columns further shows that using a large dropout rate vastly improves the o.o.d. performance of merely using the increased learning rate ($\sim$$1.3\%$).

#### 4.2.2 Ensemble and weight averaging

Figure: Figure 5: Incremental benefits achieved by constructing ensembles or by averaging the weights of models fine-tuned with very large dropouts. The baseline and dropout results are the same as those reported in Figure 7. In contrast, the ensemble and weight averaging results are now obtained by averaging the output or the weights of models fine-tuned *with large dropouts*. Ensemble and weight averaging techniques provide a marginal o.o.d. performance improvement on vlcs or office home and a negligible o.o.d. performance improvement on pacs or terra incognita.
Refer to caption: x15.png

Figure 5 similarly explores the incremental benefits achieved by constructing ensembles or by averaging the weights of models fine-tuned with very large dropouts. The incremental improvements in o.o.d. performance achieved by these methods, if any, are much smaller than the improvement achieved by large dropout rates alone.
Comparing Figures 7 and 5 also shows that in contrast, fine-tuning with large dropout rates before computing ensembles or averaging model weights brings large o.o.d. performance improvements over fine-tuning without dropout.

**Table 3: Comparison of the o.o.d. performances obtained after fine-tuning two pre-trained networks: ResNet #1 and ResNet #2. Hyperparameters are selected according to the best i.i.d. performance. Compared with ResNet #1 (He et al., ), ResNet #2 was pre-trained with the vast array of data augmentation techniques. For each of these two pre-trained networks, we follow two fine-tuning approaches: 1) naive fine-tuning; 2) advanced fine-tuning including various tricks intended to improve the o.o.d. performance, e.g. large dropout (90%), weight averaging, and increased last-layer learning rate, using hyper-parameters are selected according to the i.i.d. performance. Despite all this technology, advanced fine-tuning of a pretrained ResNet #1 (2nd column) barely matches the performance of naive fine-tuning on ResNet #2 (3rd column).**
| dataset | ResNet #1<br>naive fine-tune | ResNet #1<br>advance fine-tune | ResNet #2<br>naive fine-tune | ResNet #2<br>advance fine-tune |
| --- | --- | --- | --- | --- |
| vlcs | 76.7 | 79.5 | 78.3 | 81.0 |
| office home | 68.9 | 71.8 | 71.4 | 74.3 |
| pacs | 86.2 | 87.5 | 87.3 | 89.2 |
| terra incognita | 48.2 | 49.7 | 51.0 | 55.8 |
| Average | 69.0 | 72.1 | 72.0 | 75.1 |

### 4.3 Richer pre-training beats sophisticated fine-tuning

We have demonstrated that the very-large dropout method delivers consistently better o.o.d. performance than computing ensembles or weight-averages of models fine-tuned without dropout. However we also have argued that fine-tuning does not create new representations but merely exploits the representations already present in the pre-trained model. Therefore the final o.o.d. performance of this fine-tuning process must strongly depend on the quality and the diversity of the features present in the pre-trained network (richer representation), even if these features are not exploited by the pre-trained network but buried in its hidden layers.

To validate this assertion, we compare the i.i.d. and o.o.d. performance obtained by various methods applied to ResNet50 networks pre-trained using the same ImageNet data but using different data augmentation schemes. As explained in the first paragraphs of section 4, the results reported so far use a network pre-trained using a broad array of data augmentation techniques, termed ResNet #2. We now compare its fine-tuning properties with network termed ResNet #1 pre-trained using the simpler protocol described in He et al. ().

Figure 3 compares the o.o.d. performances of both networks after regular fine-tuning and after fine-tuning with all the available tricks, that is, with dropout, with $10\times$ larger last layer learning rate, and after averaging the weights of checkpoints collected along the fine-tuning trajectory. This comparison makes clear that the quality of the pre-trained representation matters more than the sophistication of the fine-tuning techniques. This is consistent with the idea that fine-tuning only leverages the existing features of the pre-trained network and does not create new ones.

## 5 Discussion

The o.o.d. performance of fine-tuning with very large dropout consistently exceeds that achieved by popular techniques such as ensemble and by more recent techniques such as weight averaging.
Furthermore, ensemble and weight averaging techniques only bring a small incremental improvement when applied on top of fine-tuning with large dropout rates. This suggests that very large dropout implements a key factor that favors o.o.d. performance, which we believe is related to
seeking features of type (a) among features of type (b) as explained in the introduction.

Both ensemble and weight-averaging techniques can be used for training a network from scratch or for fine-tuning a pre-trained network. In contrast, very large dropout rates cannot be realistically used when training a network from scratch. We argue that they work for fine-tuning because fine-tuning is well approximated as a linear process that can leverage their existing or buried features of a pre-trained network but cannot create new ones. Using large dropout rates is akin to a form of L2 regularization, expressing a richer set of features even if redundant.

This result also illustrates how the i.i.d. and o.o.d. scenarios can call for very different techniques.
It is well known that sparse representations can be very helpful in the i.i.d. scenario,
and it is increasingly clear that rich representations are preferable in
the o.o.d. scenario (Zhang et al., ; Zhang & Bottou, ; Chen et al., ).
There are no reasons to expect that the many techniques designed for the i.i.d. scenarios will systematically help o.o.d. generalization. The very-large dropout case is one of many such examples.

## Appendix A Experiment details

### A.1 Fine-tuning DomainNet in Figure 4

The DomainNet o.o.d. fine-tuning experiment in Figure 4 follows the same pipeline as other o.o.d. fine-tuning experiments on vlcs, pacs, Office Home, and Terra Incognita datasets. Due to the larger size of DomainNet dataset, we use larger learning rates $\{3.10^{-3},5.10^{-3}\}$ and a longer training iteration $30,000$ with a 10% learning rate decay at $15,000$ and $25,000$.

### A.2 Training from scratch in Figure 4

The vlcs scratch training experiment in Figure 4 follows the same pipeline as o.o.d. fine-tuning experiments. But it uses larger learning rates $\{5.10^{-3},10^{-2}\}$ on a random initialized ResNet50 network (all weights are trainable).

### A.3 Compute Resources

All experiments are done on V100 GPUs with Intel(R) Xeon(R) Gold 6230
CPUs. One V100 GPU and less than 32GB RAM are enough to fine-tune one Domainbed dataset within a few hours.

## Appendix B Additional results

### B.1 Residual block dropout (stochastic depth) in o. o. d. fine-tuning

Huang et al. () shows dropping a residual block at random with probability $p$ helps train very deep residual neural networks from scratch. Dropping a residual block, for example $f_{1}(x)$, in $\Phi(x)$ blinds the feature $\phi_{1}(x)$ but also changes the input of successive layers:

$$ $\displaystyle\Phi^{\prime}(x)$ $\displaystyle=\underbrace{\vphantom{f_{1}(x)}x}_{\mathclap{\phi_{0}(x)}}+ \underbrace{\deleted{f_{1}(x)}}_{\deleted{\phi_{1}(x)}}+\underbrace{f_{2}(x+ \deleted{f_{1}(x))}}_{{\color[rgb]{0.1015625,0.5,0.734375}\definecolor[named]{ pgfstrokecolor}{rgb}{0.1015625,0.5,0.734375}\phi^{\prime}_{2}(x)}}+\dots$ $$

So an aggressive residual block dropout heavily disturbs the input of high residual blocks, hurts the fine-tuning process. Figure 6 showcase the effect of redisual block dropout in o.o.d. fine-tuning.

Figure: Figure 6: Residual block dropout (stochastic depth) and penultimate layer representation dropout comparison. Residual block dropout random drops a residual block with probability $\in\{0.1,0.2,0.3\}$. Mild residual block dropouts (0.1) provides some o.o.d. improvements (⚫), but lags behind dropout (◆) on vlcs, office home, and terra incognita. Aggressive residual block dropouts hurt fine-tuning (e.g. $\geq 0.3$ on office home and terra incognita), because dropping a residual block change the input of successive layers. Likewise the comparison of ensemble, weight averaging, and large last-layer learning rate in section 4.2, a proper residual block dropout helps fine-tuning in general, but is still a secondary factor compared with dropout.
Refer to caption: extracted/5947081/figures/sgd_v2_1_stodepth_recipe.png

### B.2 o. o. d. performance improvement is not a secondary effect of some i. i. d. performance improvement

Figure 7 provides both i.i.d. and o.o.d. performance of very large dropout, ensembles and weight averaging methods. The very large dropout method outperforms other methods on o.o.d. performance without any advantage on i.i.d. performance. It reveals that the o.o.d. performance gaps of different methods in Figure 7 and Figure 1 do not come from the i.i.d. overfitting/underfitting.

Figure: Figure 7: Performance comparison between very large dropout, ensembles, and weight averaging methods on four DomainBed tasks and ResNet50 #2 backbone. The horizontal axis denotes the i.i.d. performance and the vertical axis the o.o.d. performance. Baseline results were obtained using plain fine-tuning with different hyperparameters (1296$\times$⚫). Weight averaging results either average the model weights collected every 300 iterations along each fine-tuning trajectory (1296$\times$$\bigstar$) or the final model weights of all fine-tuning trajectories (1$\times$\contourblack$\bigstar$) as in (Ramé et al., ). Ensemble results average instead the model outputs (1296$\times$▲ and 1$\times$\contourblack$\blacktriangle$). Finally, large dropout results were obtained like the baseline results but using a 90% dropout rate on the penultimate layer (1296$\times$◆).
Refer to caption: x19.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Can the proposed high-dropout method generalize to tasks and architectures beyond classification, such as segmentation, text classification, or detection? Can you provide such evaluations?

* Why should evenly distributed information across dropped features lead to “weakly relevant” features? Is further theoretical support available?

* The underperformance of ensembles is somewhat surprising. A potentially useful baseline could involve training ensembles with a shared classification head, where all layers except the final one remain distinct for each model in the ensemble. This setup would encourage the ensemble to generate features from the same domain, allowing for feature averaging, which could improve OOD performance. Would that resolve the underperformance issue? 

* Could structured dropout methods, such as Masksembles, help address the issue of aggressive dropout leading to worse performance on in-distribution data? Could it help train models from scratch with high dropout rates without compromising generalization?
soundness: 2
strengths: * The paper is clearly written and easy to follow, with an intuitive and easily understandable idea. The related work section provides a sufficient discussion of existing approaches for generating rich feature representations suitable for OOD data.

* The method is straightforward to apply, requiring only a pretrained model and a simple fine-tuning procedure that is easy to implement and computationally efficient. This approach does not change the inference of the final model (compared to vanilla model baseline), making it significantly more computationally efficient than ensembling-based alternatives.

* The method achieves strong results across various classification benchmarks.
summary: In the paper, the authors propose a novel, effective fine-tuning method to fix reduced performance on out-of-distribution (OOD) data. In essence, they begin with a model pretrained on a larger dataset and then fine-tune it on a significantly smaller set of labelled data. Given the well-known tendency of modern neural networks to perform poorly on OOD samples, the proposed method aims to reduce this performance gap. The approach is straightforward: it fine-tunes the pretrained model with applying dropout (with high dropout rate) to the penultimate layer. This method improves the final model's performance on OOD data across multiple benchmarks (PACS, VLCS, Office Home, Terra Incognita), surpassing popular baselines like model ensembling and weight averaging in classification accuracy. Furthermore, it introduces significantly less computational and memory overhead and enables faster training than conventional ensembles. The proposed approach is also easy to apply across different architectures.
weaknesses: * The experiments are limited to classification tasks on fairly standard datasets, leaving it unclear whether this approach with high dropout rates generalizes effectively to other tasks, such as segmentation or even text classification, or to other architectures, like detection models and etc. This raises questions about the scalability and broader applicability of the proposed method beyond image classification. Can other modalities/tasks be added?  

* While the paper offers a high-level explanation of why applying high dropout during fine-tuning could improves generalization (by generating more "weakly relevant" features), it is not entirely clear why this should occur from a formal standpoint. Consequently, the high dropout rate forces the model to distribute information more evenly across dropped features. However, why a more even distribution would result in more "weakly relevant" features remains unclear. A more detailed discussion would be beneficial.

* As shown in Figure 7 of the appendix, the ensembling approach consistently improves performance on in-distribution data, whereas the proposed high-dropout approach does not always yield similar benefits (and sometimes even reduces performance compared to the baseline, which is somewhat expected). It appears that overly aggressive dropout can significantly undermine performance due to excessive randomness. Could more structured methods, such as Masksembles [1], address this issue? Such methods might also overcome the problem of training a model from scratch with high dropout (which often fails to converge due to randomness) by introducing a more structured way to drop features.

[1] Durasov, Nikita, et al. "Masksembles for uncertainty estimation." CVPR 2021.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see above.
soundness: 3
strengths: Originality: Very high dropout rates have not been explored because they tend not to work well when training from scratch. Exploring it in the context of modern residual networks and fine-tuning with an emphasis on evaluating OOD robustness is original, to my knowledge.

Clarity: The paper is very clearly written, and is easy to follow.

Quality: The ideas are more or less technically sound, building on previous observations. The experiments are adequate to support the claims made — high dropout is more effective for OOD generalization for a fine-tuned model relative to related ensemble methods. 

Significance: The results appear to be consistent, though marginal most often, improvements over the selected baselines. Given that the proposed method is simple and lightweight makes it practically interesting.
summary: The submission proposes a technique for fine-tuning pre-trained networks: apply a very large dropout rate (90%) on the penultimate layer of residual networks. This approach is shown to be effective at making fine-tuned networks more robust to an OOD dataset, when compared to two existing approaches.
weaknesses: While the intuitions motivating the method are interesting and seem to play the role of establishing weight-averaging and ensemble as the natural baselines, it is not very clear that these intuitions are sufficient to narrow the scope of comparison to only the selected baselines.

It seems that performance is somewhat sensitive to the precise value of the dropout rate — 90%. It is not clear that this rate was chosen without looking at test-time performance. 

It is not fully clear to me why applying large dropout only to the penultimate layer is necessarily the best implementation, given the motivation. Does performance suffer if such dropout is applied everywhere, and not in a “dimension-aligned” fashion?


L296-298 says “the i.i.d. performance of the large dropout method lags behind that of ensembles, revealing that the o.o.d. performance improvement is not a secondary effect of some i.i.d. performance improvement (i.e. the o.o.d. performance gaps in o.o.d. Figure 1 do not come from i.i.d. overfitting/underfitting.).” In general, one wishes to retain good IID performance while also improving upon OOD numbers. On balance, might a practitioner prefer one of the baselines?

Minor:

L150-L153 says: `We focus on residual networks because fine-tuning has been found to hardly change the inner layers of non-residual networks (Raghu et al., 2019, fig 2). In contrast, skip connections in residual networks expose the inner block features in such a manner that the fine-tuning process can utilize these features in a near-linear way (Evci et al., 2022).` The first citation seems to be about MAML, which is a specific setting and likely does not extend to all fine-tuning settings. The point in that work seems to be more about what meta-learning does in terms of feature reuse and does not make generic observations about fine-tuning in non-residual networks. The second citation doesn’t seem to make any claims about the usefulness of intermediate features being specific to models using skip connections. ResNets and ViTs are top classifiers currently, so it might not be necessary to motivate the choice of models here anyway.

L319-L321 says `When the fine-tuning dataset size approaches the pre-training dataset size, the difference between fine-tuning and training from scratch becomes less clear, the linear connectivity property disappears, the linear approximation perspective on fine-tuning no longer holds, and the optimal dropout rate falls sharply.` Has it been demonstrated in prior work that the linear connectivity property disappears when the dataset sizes get close (i.e. is there a citation for this claim)?

L447 says `Despite all this technology, advanced fine-tuning of a pretrained RESNET #1 (2nd column) barely matches the performance of naive fine-tuning on RESNET #2 (3rd column).` Just a nit, but usually one does not say “A barely matches B” when A > B.

Typos:

L129: “setup is a commonly used” —> drop “a”

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - How is the proposed idea performing on more efficient architectures, like MobileNet?
- Is the same approach valid for non CV-based tasks, like NLP? It is felt the high dropout is successful where architectures are notoriously over-parametrized (as it is for CV).
- How are the learning hyper-parameters found? Has a validation set been used?
soundness: 2
strengths: - The quantitative evaluation (for obtaining the main results) is quite comprehensive.
- The message is delivered clearly.
- The empirical result is quite surprising.
summary: In this work the authors make one simple observation: when fine-tuning in o.o.d. setups, using a very high dropout can yield surprisingly good performance. The evaluation is performed on popular downstream datasets and common architectures.
weaknesses: - There is a gap from the expected theory. In the limit of large sampling, there should not be any difference in terms of performance when varying the dropout probability (a-priori, large dropout should make convergence harder, when untouching the learning rate). The empirical observation in the specific setup chosen is interesting, but right now, considering it is the only contribution, it is felt there is a lack of grounding to allow practitioners to employ this solution without running an extensive grid search.
- It seems that there is a lot of engineered choices related to learning rate scaling at layer's scale, dropout etc. 
- Some statements are overly (and maybe, unnecessarily) strong. For example, "It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution": there are many cases in which inference happens in distribution.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. The training scheme of a large dropout is interesting.
2. The proposed large dropout obtains superior performance than previous methods.
summary: To improve the ensembling performance of weight averaging, the paper proposes to employ a large dropout rate at the penultimate layer to obtain rich representations with better OOD performance.
weaknesses: 1. The main problem of the paper does not provide a deep analysis of why large dropout is helpful. It simply provides some comparing experiments to demonstrate its effectiveness. The authors should provide a formal definition of the achieved "richer representation" (i.e., quantitative comparisons). Also, some visualizations also can be provided for a better analysis.
2. Since the experiments of the proposed method are conducted on the DomainBed, comparing them to the existing methods is also required.
3. The deep analysis of different behaviors of Weight Averaging, Deep Ensemble, and the proposed method on the ID and OOD performance is required for Figure 7.
4. Why not adopt the large dropout in the deeper layers?
5. The number of ensembling members is provided (1296? which seems too large for the practice). The influence of the number should also be considered.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Please see the above weaknesses section.
soundness: 2
strengths: This paper argues that the traditional assumption in ml—that training and testing data come from the same distribution—is no longer realistic. To handle real-world scenarios involving multiple, shifting data distributions, recent studies have highlighted the need for representations that are "richer" than those optimized for single-distribution performance.  and they suggest fine-tuning a large trained model via large dropout rates.
summary: The current manuscript challenges the traditional assumption that training and testing data share the same distribution, which is increasingly unrealistic in real-world applications. To handle diverse data distributions, recent work has used ensemble methods to create richer, more robust representations than those optimized for single-distribution performance. Instead, the authors propose fine-tuning large pre-trained models with large dropout rates, achieving better performance on OOD tasks. They compare the effective of their method with ensembles and weight-averaging methods.
weaknesses: **w1: novelty**: my major concern is the limited novelty of the proposed approach, as it closely resembles existing methods without introducing substantial new concepts or techniques.

**w2:limited experiments and results:** even the experimental setup is minimal, with limited datasets and benchmarks, which weakens the generalizability and impact of the findings. Expanding the experiments to include a broader range of datasets and baselines would strengthen the evaluation.

**w3:Lack of theoretical discussion:** the paper lacks a theoretical foundation or in-depth discussion that could clarify the underlying mechanisms or justify the observed empirical results.

Meta-review: This paper shows that fine-tuning large pre-trained models with a very high dropout rate applied to the penultimate layer improves out-of-distribution (OOD) generalization. The authors explain it as providing "richer" feature representations beneficial for OOD tasks. The approach is computationally efficient, requires fewer resources than ensembles, and performs robustly across different architectures (e.g., ResNets, Transformers). The paper is well-written, making the method and results accessible.

While the observation in this paper is simple, interesting, and potentially of broad interest to the community, the paper falls short in both practicality and theoretical insights. 

- Practicality: multiple reviewers raised the concern that the experiments are restricted to image classification, leaving open questions about the method’s applicability to other tasks (e.g., segmentation, NLP). The authors acknowledged this limitation but did not address it experimentally. In addition, multiple reviewers raised concerns about the method's drop in IID performance, which may limit its practicality in scenarios requiring both IID and OOD generalization. 

- The paper lacks a rigorous theoretical explanation of the observed phenomenon. For example, the authors resort to the notion of "richer representations" as an explanation but this does not have a quantitative definition. 

While not all of these issues are necessarily required for acceptance, addressing at least one of them in a significant way would greatly strengthen the paper. I encourage the authors to refine their manuscript along these lines for future submissions.

Decision: Reject