# Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models

## About This Workshop

In the rapidly evolving landscape of AI, the development of scalable optimization methods to yield efficient and adaptive foundation models has significant demand in the space of their inference service. In specific, enabling model efficiency while allowing them to be adaptable to various new downstream tasks has multifold challenges.

Firstly, the model's ability to quickly learn adaptive and efficient sub-model selection on different tasks requires the capability to perform continual weight updates, compute- and memory-efficient fine-tuning, and personalized adaptation.

Secondly, with the increased demand for long context understanding and reasoning, the model needs to yield such efficient adaptation with the informative usefulness of the query-specific token fetching. For instance, imagine a model that continually learns from current news events, adapting to the ever-changing global landscape by integrating up-to-date knowledge. Such models may not only need efficient fine-tuning to new incoming data stream, but also understand efficient handling of the KV cache that may keep on growing with the requirement to handle longer contextual information. Additionally, the integration of retrieval-augmented generation (RAG) into foundation models can ensure that generated content is not only relevant, but also reflects the most current knowledge while costing the prefill size to go up.

Thirdly, with such growing demand for contextual adaptation, mixture of experts (MoE) models have also received significant traction that can perform test time adaptation via learned routing policy. In addition, the emergence of sub-quadratic models with constant KV states as opposed to KV caching of transformers, has opened up a new avenue of the model's adaptation ability in the context of information retention into compressive KV states. These capabilities rely on techniques for adapting foundation models, including fine-tuning, conversion, distillation, and in-context/few-shot learning.

This workshop aims to capture advances in scalable, adaptive fine-tuning, calibration, and conversion to yield inference efficient quadratic and sub-quadratic foundation models, focusing on methodologies across vision, language, and multi-modal domains. Hosting this workshop at ICLR aligns with the conference’s mission to advance the frontiers of machine learning. The workshop aims to bring together interdisciplinary researchers from core ML/DL, efficient ML, computer vision, and NLP.


## Topics:
The relevant topics of interest at this workshop include (but are not limited to):

- Efficient Long Context Understanding
- Sub-Quadratic Models for Foundational Tasks and Personalization
- Quadratic to Sub-Quadratic Model Conversion
- Task Specific Adaptive Foundation Models
- Retrieval Augmented Generation for Efficient Contextual Processing
- Efficient Sub-Quadratic Foundation Models
- Adaptive Fine-Tuning for Multimodal Foundation Models
- Efficient Fine-Tuning for Continual Adaptation and Personalization
- Model Optimization for Latency and Throughput Efficient Inference
- Adaptive Routing with Mixture of Experts