## Workshop on Mathematics of Modern Machine Learning

Deep learning has demonstrated tremendous success in the past decade, sparking a revolution in artificial intelligence.

However, the modern practice of deep learning remains largely an art form, requiring a delicate combination of guesswork and careful hyperparameter tuning. This can be attributed to the fact that classical machine learning theory fails to explain many deep learning phenomena, which inhibits its ability to provide effective guidance in practice.

As we enter the large model era of deep learning, this issue becomes even more critical since trial and error with billion- or trillion-size models can result in enormous costs of time and computation. There is a greater need than ever before for theory that can guide practice and provide principled ways to train large models...

## Topics
This workshop's main areas of focus include but are not limited to:

- **Reconciling Optimization Theory with Deep Learning Practice**

  - **Convergence analysis beyond the stable regime:** How do optimization methods minimize training losses despite large learning rates and large gradient noise? How should we understand the Edge of Stability (EoS) phenomenon? What could be more realistic assumptions for the loss landscape and gradient noise that foster training algorithms with faster convergence both in theory and practice?

  - **Continuous approximations of training trajectories:** Can we obtain insights into the discrete-time gradient dynamics by approximating them with a continuous counterpart, e.g., gradient flow or SDE? When is such an approximation valid?

  - **Advanced optimization algorithms:** Why does Adam optimize faster than SGD on Transformers? Under what theoretical models can we design advanced optimization methods (e.g., adaptive gradient algorithms, second-order algorithms, distributed training algorithms) that provably work better?

- **Generalization for Overparametrized Models**

  - **Implicit bias:** Whether and how do gradient-based algorithms implicitly pick the solution with good generalization, despite a rich set of non-generalizing minimizers?

  - **Generalization Measures:** What is the relationship between generalization performances and common generalization measures? (e.g., sharpness, margin, norm, etc.) Can we prove non-vacuous generalization bounds based on these generalization measures?

  - **Roles of Key Components in Algorithm and Architecture:** What are the roles of initialization, learning rate warmup and decay, and normalization layers?

- **Intriguing phenomena of foundation models**

  - **Pretraining:** What do foundation models learn in pretraining that allows for efficient finetuning? How does the choice of dataset/architecture affect this?

  - **Effect of Data:** How does the number of data passes affect training, and can we consolidate the empirical and theoretical understanding? How should the use of data differ during and after pretraining?

  - **Multimodal Representations:** How can we learn representations from multimodal data?

  - **Scaling Laws and Emergent Phenomena:** How and why does the performance scale with data, compute, and model size? What mathematical models should we use to understand emergent abilities such as in-context and few-shot reasoning?

  - **Diffusion Models:** What do we understand about the success and limitations of diffusion models and score-matching methods?

- **Provable Guarantees Beyond Supervised Learning Settings**

  - **Online Learning and Reinforcement Learning:** How is learning affected by various factors such as expert feedback quality or data coverage? How should theory tools be adapted to inform modern use cases such as RLHF?

  - **Representation Learning and Transfer Learning:** What properties of the source and target tasks allow for efficient transfer learning? What types of representations can be learned via self-supervised learning (e.g., contrastive learning)

  - **Multitask and Continual Learning:** What conditions are needed to adapt a model to new tasks while preserving the performance of old tasks? What view should we take to understand modern notions of multitask and continual learning, where assumptions could diviate greatly from classic theory?