fine-tuning with very large dropout

Authors: Jianyu Zhang, Leon Bottou
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: out-of-distribution, fine-tuning, very large dropout

Abstract:
It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution. Several authors have recently used ensemble techniques to show how scenarios involving multiple data distributions are best served by representations that are both richer than those obtained by regularizing for the best in-distribution performance, and richer than those obtained under the influence of the implicit sparsity bias of common stochastic gradient procedures.

This contribution investigates the use of very high dropout rates instead of ensembles to obtain such rich representations. Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups. 

This result has practical significance because the importance of the fine-tuning scenario has considerably grown in recent years. This result also provides interesting insights on the nature of rich representations and on the intrinsically linear nature of fine-tuning a large network using a comparatively small dataset.

Decision: reject
Reviewer scores: 6, 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730492987024

thread_type:
official_review

<thread_object>
creator: reviewer_hny5

<content>
summary:
The current manuscript challenges the traditional assumption that training and testing data share the same distribution, which is increasingly unrealistic in real-world applications. To handle diverse data distributions, recent work has used ensemble methods to create richer, more robust representations than those optimized for single-distribution performance. Instead, the authors propose fine-tuning large pre-trained models with large dropout rates, achieving better performance on OOD tasks. They compare the effective of their method with ensembles and weight-averaging methods.

strengths:
This paper argues that the traditional assumption in ml—that training and testing data come from the same distribution—is no longer realistic. To handle real-world scenarios involving multiple, shifting data distributions, recent studies have highlighted the need for representations that are "richer" than those optimized for single-distribution performance.  and they suggest fine-tuning a large trained model via large dropout rates.

weaknesses:
**w1: novelty**: my major concern is the limited novelty of the proposed approach, as it closely resembles existing methods without introducing substantial new concepts or techniques.

**w2:limited experiments and results:** even the experimental setup is minimal, with limited datasets and benchmarks, which weakens the generalizability and impact of the findings. Expanding the experiments to include a broader range of datasets and baselines would strengthen the evaluation.

**w3:Lack of theoretical discussion:** the paper lacks a theoretical foundation or in-depth discussion that could clarify the underlying mechanisms or justify the observed empirical results.

questions:
Please see the above weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516224257

thread_type:
official_review

<thread_object>
creator: reviewer_lz8e

<content>
summary:
To improve the ensembling performance of weight averaging, the paper proposes to employ a large dropout rate at the penultimate layer to obtain rich representations with better OOD performance.

strengths:
1. The training scheme of a large dropout is interesting.
2. The proposed large dropout obtains superior performance than previous methods.

weaknesses:
1. The main problem of the paper does not provide a deep analysis of why large dropout is helpful. It simply provides some comparing experiments to demonstrate its effectiveness. The authors should provide a formal definition of the achieved "richer representation" (i.e., quantitative comparisons). Also, some visualizations also can be provided for a better analysis.
2. Since the experiments of the proposed method are conducted on the DomainBed, comparing them to the existing methods is also required.
3. The deep analysis of different behaviors of Weight Averaging, Deep Ensemble, and the proposed method on the ID and OOD performance is required for Figure 7.
4. Why not adopt the large dropout in the deeper layers?
5. The number of ensembling members is provided (1296? which seems too large for the practice). The influence of the number should also be considered.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645731962

thread_type:
official_review

<thread_object>
creator: reviewer_rzvb

<content>
summary:
In this work the authors make one simple observation: when fine-tuning in o.o.d. setups, using a very high dropout can yield surprisingly good performance. The evaluation is performed on popular downstream datasets and common architectures.

strengths:
- The quantitative evaluation (for obtaining the main results) is quite comprehensive.
- The message is delivered clearly.
- The empirical result is quite surprising.

weaknesses:
- There is a gap from the expected theory. In the limit of large sampling, there should not be any difference in terms of performance when varying the dropout probability (a-priori, large dropout should make convergence harder, when untouching the learning rate). The empirical observation in the specific setup chosen is interesting, but right now, considering it is the only contribution, it is felt there is a lack of grounding to allow practitioners to employ this solution without running an extensive grid search.
- It seems that there is a lot of engineered choices related to learning rate scaling at layer's scale, dropout etc. 
- Some statements are overly (and maybe, unnecessarily) strong. For example, "It is impossible today to pretend that the practice of machine learning is compatible with the idea that training and testing data follow the same distribution": there are many cases in which inference happens in distribution.

questions:
- How is the proposed idea performing on more efficient architectures, like MobileNet?
- Is the same approach valid for non CV-based tasks, like NLP? It is felt the high dropout is successful where architectures are notoriously over-parametrized (as it is for CV).
- How are the learning hyper-parameters found? Has a validation set been used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666650368

thread_type:
official_review

<thread_object>
creator: reviewer_hp51

<content>
summary:
The submission proposes a technique for fine-tuning pre-trained networks: apply a very large dropout rate (90%) on the penultimate layer of residual networks. This approach is shown to be effective at making fine-tuned networks more robust to an OOD dataset, when compared to two existing approaches.

strengths:
Originality: Very high dropout rates have not been explored because they tend not to work well when training from scratch. Exploring it in the context of modern residual networks and fine-tuning with an emphasis on evaluating OOD robustness is original, to my knowledge.

Clarity: The paper is very clearly written, and is easy to follow.

Quality: The ideas are more or less technically sound, building on previous observations. The experiments are adequate to support the claims made — high dropout is more effective for OOD generalization for a fine-tuned model relative to related ensemble methods. 

Significance: The results appear to be consistent, though marginal most often, improvements over the selected baselines. Given that the proposed method is simple and lightweight makes it practically interesting.

weaknesses:
While the intuitions motivating the method are interesting and seem to play the role of establishing weight-averaging and ensemble as the natural baselines, it is not very clear that these intuitions are sufficient to narrow the scope of comparison to only the selected baselines.

It seems that performance is somewhat sensitive to the precise value of the dropout rate — 90%. It is not clear that this rate was chosen without looking at test-time performance. 

It is not fully clear to me why applying large dropout only to the penultimate layer is necessarily the best implementation, given the motivation. Does performance suffer if such dropout is applied everywhere, and not in a “dimension-aligned” fashion?


L296-298 says “the i.i.d. performance of the large dropout method lags behind that of ensembles, revealing that the o.o.d. performance improvement is not a secondary effect of some i.i.d. performance improvement (i.e. the o.o.d. performance gaps in o.o.d. Figure 1 do not come from i.i.d. overfitting/underfitting.).” In general, one wishes to retain good IID performance while also improving upon OOD numbers. On balance, might a practitioner prefer one of the baselines?

Minor:

L150-L153 says: `We focus on residual networks because fine-tuning has been found to hardly change the inner layers of non-residual networks (Raghu et al., 2019, fig 2). In contrast, skip connections in residual networks expose the inner block features in such a manner that the fine-tuning process can utilize these features in a near-linear way (Evci et al., 2022).` The first citation seems to be about MAML, which is a specific setting and likely does not extend to all fine-tuning settings. The point in that work seems to be more about what meta-learning does in terms of feature reuse and does not make generic observations about fine-tuning in non-residual networks. The second citation doesn’t seem to make any claims about the usefulness of intermediate features being specific to models using skip connections. ResNets and ViTs are top classifiers currently, so it might not be necessary to motivate the choice of models here anyway.

L319-L321 says `When the fine-tuning dataset size approaches the pre-training dataset size, the difference between fine-tuning and training from scratch becomes less clear, the linear connectivity property disappears, the linear approximation perspective on fine-tuning no longer holds, and the optimal dropout rate falls sharply.` Has it been demonstrated in prior work that the linear connectivity property disappears when the dataset sizes get close (i.e. is there a citation for this claim)?

L447 says `Despite all this technology, advanced fine-tuning of a pretrained RESNET #1 (2nd column) barely matches the performance of naive fine-tuning on RESNET #2 (3rd column).` Just a nit, but usually one does not say “A barely matches B” when A > B.

Typos:

L129: “setup is a commonly used” —> drop “a”

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730602303

thread_type:
official_review

<thread_object>
creator: reviewer_azwu

<content>
summary:
In the paper, the authors propose a novel, effective fine-tuning method to fix reduced performance on out-of-distribution (OOD) data. In essence, they begin with a model pretrained on a larger dataset and then fine-tune it on a significantly smaller set of labelled data. Given the well-known tendency of modern neural networks to perform poorly on OOD samples, the proposed method aims to reduce this performance gap. The approach is straightforward: it fine-tunes the pretrained model with applying dropout (with high dropout rate) to the penultimate layer. This method improves the final model's performance on OOD data across multiple benchmarks (PACS, VLCS, Office Home, Terra Incognita), surpassing popular baselines like model ensembling and weight averaging in classification accuracy. Furthermore, it introduces significantly less computational and memory overhead and enables faster training than conventional ensembles. The proposed approach is also easy to apply across different architectures.

strengths:
* The paper is clearly written and easy to follow, with an intuitive and easily understandable idea. The related work section provides a sufficient discussion of existing approaches for generating rich feature representations suitable for OOD data.

* The method is straightforward to apply, requiring only a pretrained model and a simple fine-tuning procedure that is easy to implement and computationally efficient. This approach does not change the inference of the final model (compared to vanilla model baseline), making it significantly more computationally efficient than ensembling-based alternatives.

* The method achieves strong results across various classification benchmarks.

weaknesses:
* The experiments are limited to classification tasks on fairly standard datasets, leaving it unclear whether this approach with high dropout rates generalizes effectively to other tasks, such as segmentation or even text classification, or to other architectures, like detection models and etc. This raises questions about the scalability and broader applicability of the proposed method beyond image classification. Can other modalities/tasks be added?  

* While the paper offers a high-level explanation of why applying high dropout during fine-tuning could improves generalization (by generating more "weakly relevant" features), it is not entirely clear why this should occur from a formal standpoint. Consequently, the high dropout rate forces the model to distribute information more evenly across dropped features. However, why a more even distribution would result in more "weakly relevant" features remains unclear. A more detailed discussion would be beneficial.

* As shown in Figure 7 of the appendix, the ensembling approach consistently improves performance on in-distribution data, whereas the proposed high-dropout approach does not always yield similar benefits (and sometimes even reduces performance compared to the baseline, which is somewhat expected). It appears that overly aggressive dropout can significantly undermine performance due to excessive randomness. Could more structured methods, such as Masksembles [1], address this issue? Such methods might also overcome the problem of training a model from scratch with high dropout (which often fails to converge due to randomness) by introducing a more structured way to drop features.

[1] Durasov, Nikita, et al. "Masksembles for uncertainty estimation." CVPR 2021.

questions:
* Can the proposed high-dropout method generalize to tasks and architectures beyond classification, such as segmentation, text classification, or detection? Can you provide such evaluations?

* Why should evenly distributed information across dropped features lead to “weakly relevant” features? Is further theoretical support available?

* The underperformance of ensembles is somewhat surprising. A potentially useful baseline could involve training ensembles with a shared classification head, where all layers except the final one remain distinct for each model in the ensemble. This setup would encourage the ensemble to generate features from the same domain, allowing for feature averaging, which could improve OOD performance. Would that resolve the underperformance issue? 

* Could structured dropout methods, such as Masksembles, help address the issue of aggressive dropout leading to worse performance on in-distribution data? Could it help train models from scratch with high dropout rates without compromising generalization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732050437318

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## Question 1: Why does the very-large-dropout approach help OOD generalization? Is there a theoretical result?

Our paper explains the reason behind the success of the very-large-dropout approach in OOD fine-tuning as "rich representation" (Lines 043-063). This concept is also discussed in related works [1,2,3]. We want to emphasize that our approach and the idea of rich representation are counter-intuitive compared to traditional in-distribution views. As such, it may be challenging for those familiar with in-distribution machine learning to accept this new perspective.

However, we believe that our experimental results objectively demonstrate the superior performance of the very-large-dropout approach and the existence of a new view that differs from traditional in-distribution views. Even if our explanations fail to convince you, the superior performance of very-large-dropout is objective and contributive. It's worth noting that ICLR accepts experimental works, which aligns with the nature of our submission.


---
## Question 2: How does it work on other tasks and other architectures?

This concern is closely related to the previous question, as our proposed approach is counter-intuitive and may be challenging to accept subjectively. Consequently, one may require more support or evidence to fully accept it.

We understand this concern and acknowledge that it is reasonable. However, we would like to highlight that our paper has already provided a comprehensive evaluation of the very-large-dropout approach on two widely used architectures (ResNet, Transformer), three pre-trained models, five OOD tasks, and thousands of fine-tuning configurations. We believe that this extensive evaluation provides sufficient evidence to support the claims made in our paper.

While we agree that more experiments would be beneficial, we must also consider the page limit of the paper. As a nine-page submission, we have made a concerted effort to provide a clear and concise presentation of our research, while also providing sufficient evidence to support our claims.

---

## Question 3: IID performance drops down! Is this a drawback of the proposed very-large dropout approach?

We understand the concern about the drop in IID performance. However, we would like to emphasize that our work focuses on out-of-distribution (OOD) generalization, as stated in the DomainBed Benchmark setup and the abstract.

To put this into perspective, let's consider an analogy with traditional IID machine learning:
- In the traditional IID setup, one does not really care about the training accuracy but cares about the test accuracy. Because the true test examples are not necessarily the same as the training examples.

- In this OOD setup, one does not really care about the IID test accuracy, but cares about the OOD accuracy. Because the true test distribution is not necessarily the same as the training distribution.

In this context, the drop in IID performance is not necessarily a drawback, but rather a consequence of prioritizing OOD generalization. Our experimental results demonstrate the effectiveness of the very-large-dropout approach in achieving superior OOD performance, which is the primary focus of our work.

---

## Question 4: Why apply dropout on the penultimate layer, instead of everywhere (all layers)?

Applying high dropout rates to intermediate layers can significantly alter the input to subsequent layers, making fine-tuning more challenging. As we mentioned in line 066-072, fine-tuning with very-large dropout relies on the "simplicity" (nearly convex) of the fine-tuning process. By applying dropout to the penultimate layer, we preserve the simplicity of the fine-tuning process, which is essential for our approach to be effective.

[1] Zhang, J., & Bottou, L. (2023, July). Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning (pp. 40830-40850). PMLR.

[2] Zhang, J., Lopez-Paz, D., & Bottou, L. (2022, June). Rich feature construction for the optimization-generalization dilemma. In International Conference on Machine Learning (pp. 26397-26411). PMLR.

[3] Chen, Y., Huang, W., Zhou, K., Bian, Y., Han, B., & Cheng, J. (2024). Understanding and improving feature learning for out-of-distribution generalization. Advances in Neural Information Processing Systems, 36.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper shows that fine-tuning large pre-trained models with a very high dropout rate applied to the penultimate layer improves out-of-distribution (OOD) generalization. The authors explain it as providing "richer" feature representations beneficial for OOD tasks. The approach is computationally efficient, requires fewer resources than ensembles, and performs robustly across different architectures (e.g., ResNets, Transformers). The paper is well-written, making the method and results accessible.

While the observation in this paper is simple, interesting, and potentially of broad interest to the community, the paper falls short in both practicality and theoretical insights. 

- Practicality: multiple reviewers raised the concern that the experiments are restricted to image classification, leaving open questions about the method’s applicability to other tasks (e.g., segmentation, NLP). The authors acknowledged this limitation but did not address it experimentally. In addition, multiple reviewers raised concerns about the method's drop in IID performance, which may limit its practicality in scenarios requiring both IID and OOD generalization. 

- The paper lacks a rigorous theoretical explanation of the observed phenomenon. For example, the authors resort to the notion of "richer representations" as an explanation but this does not have a quantitative definition. 

While not all of these issues are necessarily required for acceptance, addressing at least one of them in a significant way would greatly strengthen the paper. I encourage the authors to refine their manuscript along these lines for future submissions.