# Workshop on Distribution Shifts: New Frontiers with Foundation Models

## Overview
This workshop focuses on distribution shifts in the context of foundation models.
Distribution shifts—where a model is deployed on a data distribution different from what it was trained on—pose significant robustness challenges in real-world ML applications. Such shifts are often unavoidable in the wild and have been shown to substantially degrade model performance in applications such as biomedicine, wildlife conservation, sustainable development, robotics, education, and criminal justice. For example, models can systematically fail when tested on patients from different hospitals or people from different demographics. Training models that are robust to such distribution shifts is a rapidly growing area of interest in the ML community, and the goal of our workshop is to foster discussions and further research on distribution shifts.

In recent years, foundation models—large pretrained models that can be adapted for a wide range of tasks—have achieved unprecedented performance on a broad variety of discriminative and generative tasks, including in distribution shift scenarios. Foundation models open up an exciting new frontier in the study of distribution shifts, raising many open research questions:
- Empirical trends. Foundation models can perform well under distribution shift—for instance, finetuned foundation models hold the state-of-the-art on several datasets in the WILDS benchmark of distribution shifts, although substantial gaps remain between in-distribution and out-of-distribution performance. What aspects of foundation models (e.g., pretraining data diversity, model scale, etc.) are driving this robustness? On what kinds of distribution shifts do these performance gains hold—e.g., are there shifts on which larger-scale models do more poorly?
- Pretraining. Foundation models are pretrained on diverse corpora that typically do not reflect the data distribution of a downstream task, and this shift is particularly drastic for specialized applications (e.g., medical NLP). How does this pretraining distribution shift affect performance on downstream tasks? How can we mitigate it when pretraining foundation models?
- Adaptation. For specialized tasks with poor few-shot performance, current foundation models must be adapted, e.g., by fine-tuning on a specialized dataset that differs significantly from the large pretraining dataset. However, prior work has shown that such fine-tuning can reduce the gains in distributional robustness that come from using foundation models, and these finetuned models incur substantial performance drops due to distribution shifts. What causes these phenomena, and how can we adapt models to downstream tasks without sacrificing robustness?
- Generation. Distribution shifts have been largely studied in discriminative settings, but many foundation models have unprecedented generative capabilities. How do distribution shifts affect generative settings, e.g., if a model is used with prompts that are under-represented in the training data? How do we generate samples from a distribution of interest that differs from the pretraining distribution? How can we measure the effects of such shifts and mitigate them? And how can we leverage these generative capabilities to address distribution shifts in discriminative settings, e.g., through data augmentation?

Many of these questions of distribution shift are also key challenges for developing better foundation models.  For example, foundation models are often adapted to be instruction-following and harmless using methods such as reinforcement learning from human feedback, and these are attempts to address the pretraining-to-downstream shift in a generative setting. Moreover, since today's foundation models are typically trained on data scraped from the Internet, adapting them to a broader set real-world applications (e.g., in biomedicine, conservation and sustainability, law, etc.) also requires grappling with the pretraining shift.

To this end, our workshop focuses on distribution shifts in the context of foundation models. We are broadly interested in methods, evaluations and benchmarks, and theory for distribution shifts, and we are especially interested in work that involve foundation models.