chchenhui/mlrbench-tasks huggingface dataset:
# Workshop on Open Science for Foundation Models

## About the Workshop
Foundation models (FMs) have transformed AI research but lack scientific transparency. The SCI-FM workshop aims to address this by fostering open science, reproducibility, and the sharing of open-source models and datasets. We invite contributions that explore key aspects of FMs, such as dataset curation, evaluation methodologies, and innovative training strategies. Join us in advancing the accessibility and transparency of foundation models for the global research community.

## Scope

We invite papers on topics including, but not limited to:

- Open Datasets: Acquisition, curation, and synthesis of pretraining, instruction, and preference datasets through manual or algorithmic methods. Open access to instruction and preference datasets for alignment research.
- Open Foundation Models: Pretraining strategies including data scaling, model architecture, multi-modal, and multi-task pretraining. Learning algorithms such as meta-learning, model fusion, model merging, and continual learning designed for open, scalable models. Inference algorithms like decoding, reasoning, search, and planning, tailored for foundation models.
- Open Training Protocols: Training dynamics research on scaling laws, interpretability, complexity analysis, emergent capabilities, and phenomena like grokking. Alignment techniques including prompt tuning, prefix tuning, instruction tuning, and reinforcement learning with human/AI feedback.
- Open Evaluation: Benchmark development and the creation of transparent evaluation protocols and metrics, including the open sharing of benchmark datasets and evaluation results across different foundation models.
- Open Compute Efficiency Techniques: Focus on model distillation, compression, quantization, and optimizing attention or memory mechanisms for improved compute efficiency in open foundation models.
- Open Multi-Modal Foundation Models: Expanding to modalities like vision, audio, and multi-modal foundation models, with extra emphasis on underexplored areas such as chemistry, medicine, and education.
- Open Interactive and Agent Systems: Open development of conversational AI, interactive learning models, multi-agent systems, and integration with external tools and APIs.
- Open Replication of Proprietary Systems: Efforts to replicate and openly share foundation models and systems that were previously proprietary, ensuring transparency and reproducibility for broader research and development.
## Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference

Large Language Models (LLMs) have emerged as transformative tools in both research and industry, excelling across a wide array of tasks. However, their growing computational demands especially during inference—raise significant concerns about accessibility, environmental sustainability, and deployment feasibility. At the same time, sparsity-based techniques are proving critical not just for improving efficiency but also for enhancing interpretability, modularity, and adaptability in AI systems. 

This workshop aims to bring together researchers and practitioners from academia and industry who are advancing the frontiers of sparsity in deep learning. Our scope spans several interrelated topics, including Mixture of Experts (MoEs), LLM inference and serving, network pruning, sparse training, distillation, activation sparsity, low-rank adapters, hardware innovations and quantization. A key objective is to foster connections and unlock synergies between traditionally independent yet highly related research areas, such as activation sparsity and sparse autoencoders (SAEs), or quantization and KV cache compression. Rather than focusing solely on efficiency, we aim to explore how sparsity can serve as a unifying framework across multiple dimensions of AI—driving advances in interpretability, generalization, and system design. 

By facilitating the fusion of ideas from different topics, the workshop will create new opportunities for innovation. We encourage participants to think beyond traditional constraints, exploring how different forms of sparsity can inform each other and yield new algorithms. Whether the goal is faster inference, modular architectures, or more interpretable models, our aim is to catalyze research that deepens the integration of sparsity within AI.

Topics of interest include, but are not limited to:

- Mixture of Experts (MoEs) and Modularity

- Parameter Sparsity/Pruning

- Interaction with Quantization and Distillation

- Activation Sparsity for Inference

- Sparsity for Interpretability

- Hardware Innovation for Sparsity

- Parameter Efficient Fine Tuning
# Will Synthetic Data Finally Solve the Data Access Problem?
Accessing large scale and high quality data has been shown to be one of the most important factors to the performance of machine learning models. Recent works show that large (language) models can greatly benefit from training with massive data from diverse (domain specific) sources and aligning with user intention. However, the use of certain data sources can trigger privacy, fairness, copyright, and safety concerns. The impressive performance of generative artificial intelligence popularized the usage of synthetic data, and many recent works suggest (guided) synthesization can be useful for both general purpose and domain specific applications.

Will synthetic data ultimately solve the data access problem for machine learning? This workshop seeks to address this question by highlighting the limitations and opportunities of synthetic data. It aims to bring together researchers working on algorithms and applications of synthetic data, general data access for machine learning, privacy-preserving methods such as federated learning and differential privacy, and large model training experts to discuss lessons learned and chart important future directions.


Topics of interest include, but are not limited to, the following:

- Risks and limitations of synthetic data.
- New algorithms for synthetic data generation.
- New applications of using synthetic data (e.g. in healthcare, finance, gaming and simulation, education, scientific research, or - autonomous systems).
- Synthetic data for model training and evaluation.
- Synthetic data for improving specific model capabilities (e.g., reasoning, math, coding).
- Synthetic data to address privacy, fairness, safety and other data concerns.
- Evaluation of synthetic data quality and models trained on synthetic data.
- Conditional and unconditional synthetic data generation.
- Fine-grained control of synthetic data generation.
- Data access with federated learning and privacy-preserving methods.
- New paradigm of accessing data for machine learning.
- Mixing synthetic and natural data.
# Workshop on GenAI Watermarking

A Dedicated Space for Watermarking in Generative AI

Watermarking is crucial in the age of generative AI, yet often gets lost in broader conversations around AI safety and security. Our workshop brings together experts from academia, industry, policy, and different communities to discuss advancements and challenges in watermarking technologies.

## Topics
- Algorithmic advances and new applications of watermarking

- Adversarial robustness and security-related topics

- Evaluation and benchmarks

- Industry requirements

- Policy, regulations and ethics landscapes
# Robot Learning Workshop: Towards Robots with Human-Level Abilities

## About This Workshop
The year 2024 has seen an explosion of interest in humanoid robots. However, recent systems for drone racing, playing table tennis, and others clearly demonstrate that the humanoid form-factor isn’t a requirement for human-level performance. In the 7th Robot Learning workshop, to be held at ICLR 2025, we will look beyond the humanoid embodiment and ask: how far are we from robots with human-level abilities? What do we need to improve about embodied learning, decision-making, perception, and data collection to train generally physically capable robots to robustly perform a wide range of activities such as cooking or tidying up a house – activities that people do without much thinking?

We believe many of the weaknesses of the current robotic systems to be a reflection of the shortcomings of general AI methods and models. As such, we seek diverse perspectives on the workshop theme from robotics-focused and robotics-orthogonal parts of the ICLR community alike, scientific contributions from academia and industry, as well as participants from a variety of backgrounds and career stages.

We welcome submissions of original research papers as well as systems papers focusing on algorithmic innovations, theoretical advancements, system design, or practical applications relevant to the workshop theme.

Specific areas of interest include but are not limited to:
- Novel ML algorithms and model architectures for robot control: techniques integrating large multi-modal models, sim-to-real bridging, safe policy optimization, and data efficiency.
- Human-robot interaction and collaboration: socially aware motion planning, adaptive interfaces, and trust-building strategies for seamless teamwork. Hardware innovations and system integration: advanced sensing and actuation, high-degree-of-freedom controllers, energy-efficient designs, and cohesive robotics architectures.
- Simulation, benchmarking, and evaluation methodologies: realistic simulation environments, standardized task suites, robust metrics, and cross-domain validation protocols.
- Applications in unstructured and dynamic environments: household assistance, mobile manipulation, industrial automation, healthcare, disaster response, and other real-world domains.
# XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge

## About This Workshop
Machine learning (ML) models are impressive when they work but they can also show unreliable, untrustworthy, and harmful dangerous behavior. Yet, such models are widely adopted and deployed, even though we do not understand why they work so well and fail miserably at times. Such rapid dissemination encourages irresponsible use, for example, to spread misinformation or create deep fakes, while hindering the efforts to use them to solve pressing societal problems and advance human knowledge.


Ideally, we want models to help us improve our understanding of the world and, at the very least, we want them to aid human knowledge and help us to further enrich it. Our goal in this workshop is to take a step in this direction by bringing together researchers working on understanding model behavior and using it to discover new human knowledge. The workshop will include theoretical topics on understanding model behavior, namely interpretability and explainability (XAI), but also three distinct scientific application areas: weather and climate, healthcare, and material science (ML4Science).

## Topics
A-priori (i.e., ante-hoc) interpretability and self-explainable models for understanding model’s behaviour


A-posteriori (i.e., post-hoc) interpretability and attribution methods for understanding model’s behaviour, including methods for evaluating the accuracy of post-hoc interpretability and attribution


Practical use of interpretability and explainability for knowledge discovery in

- ⁠Weather and climate science,
- ⁠Material science, and
- ⁠⁠Healthcare
# Artificial Intelligence and Human-Computer Interaction

## Workshop Overview

Artificial intelligence (AI) and Human-Computer Interaction (HCI) share common roots: early work on conversational agents has laid the foundation for both fields. Despite sharing a common history, economic and political influences have driven these fields to remain separate in subsequent decades. The recent rise of data-centric methods in machine learning has propelled few-shot emergent AI capabilities, resulting in a raft of practical tools. In particular, modern AI techniques now power new ways for machines and humans to interact. Recently, a wave of HCI tasks has been proposed to the machine learning community, which direct AI research by contributing new datasets and benchmarks, and challenging existing modeling techniques, learning methodologies, and evaluation protocols. Machine learning techniques have been developed for a variety of tasks, including user-interface understanding, UI generation, accessibility, and reinforcement learning from human feedback. This workshop offers a forum for researchers to discuss these new research directions, identify important challenges, showcase new computational and scientific ideas that can be applied, share datasets/tools that are already available, or propose those that should be further developed.

## Topics

We invite researchers both in academia as well as industry who work in the areas including but not limited to: 

- User interface modeling for understanding and generation
- Reinforcement learning with human feedback (RLHF)
- Explainable and interpretable machine learning methods
- Generative AI and creativity tools
- Human evaluation methods
- Personalizable and correctable machine learning models
- Novel human interactions with models
- Active learning and human-in-the-loop systems
- Ethics and fairness-based models, interactions, and evaluations
- Tools and datasets to accelerate works at the intersection of HCI and AI
- Challenges in working at the intersection of AI and HCI
## Challenges of Deploying Generative AI

Generative modeling has recently gained massive attention given high-profile successes in natural language processing and computer vision. However, there remain major challenges in deploying generative models for real-world impact in domains like healthcare and biology. This is a challenging agenda that requires collaboration across multiple research fields and industry stakeholders. This workshop aims to advance such interdisciplinary conversations around challenges in deploying generative models – the lessons learned by deploying large language models could be impactful for other high stakes domains. Specifically, we will solicit contributions that prioritize (1) Multimodal capabilities in generative modeling, (2) Deployment critical features in generative models such as Safety, Interpretability, Robustness, Ethics, Fairness and Privacy, and (3) Human facing evaluation of generative models.

## Topics

We seek papers from all topics related to recent advances in Generative AI, including all data modalities including language and vision models. We especially encourage submissions that focus on challenges when applying Generative AI to impactful, real-world, interdisciplinary problems. Potential topics can include, but are not limited to:

- Applications to challenging real-world problems
- Interpretability, Fairness, Robustness, and Safety
- Memorization, Unlearning, and Privacy
- Multi-modal generation
- Technical challenges of deployment and implementation
- Evaluation methodologies, metrics, human-facing evaluations
- Novel methods and architectures
## Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators

Gradients and derivatives are integral to machine learning, as they enable gradient-based optimization. In many real applications, however, models rest on algorithmic components that implement discrete decisions, or rely on discrete intermediate representations and structures. These discrete steps are intrinsically non-differentiable and accordingly break the flow of gradients. To use gradient-based approaches to learn the parameters of such models requires turning these non-differentiable components differentiable. This can be done with careful considerations, notably, using smoothing or relaxations to propose differentiable proxies for these components. With the advent of modular deep learning frameworks, these ideas have become more popular than ever in many fields of machine learning, generating in a short time-span a multitude of “differentiable everything”, impacting topics as varied as rendering, sorting and ranking, convex optimizers, shortest-paths, dynamic programming, physics simulations, NN architecture search, top-k, graph algorithms, weakly- and self-supervised learning, and many more.

## Scope

The technical topics of interest at this workshop include (but are not limited to):

- Continuous relaxations of discrete operations and algorithms (e.g., argmax, sorting, ranking, rendering, shortest-path, optimizers, if-else constructs, indexing, top-k, logics, etc.)
- Stochastic relaxations and gradient estimation methods (e.g., stochastic smoothing)
- Weakly- and self-supervised learning with differentiable algorithms, e.g., ranking supervision
- Optimization with diff. algorithms, e.g., regression of scene parameters via diff. rendering
- Systematic techniques for making discrete structures differentiable, e.g., smoothing
- Differentiable simulators such as differentiable fluid dynamics, differentiable particle simulators, differentiable optics, differentiable protein-folding, differentiable cloth simulations, etc.
- Differentiable architecture search, e.g., convolutions with diff. and learnable kernel sizes
- Applications of differentiable relaxations, e.g., in learning-to-rank and computer vision

The workshop does not cover “differentiable programming”, i.e., the programming paradigm of automatic differentiation and its technical implementations. Instead, the workshop covers cases where vanilla automatic differentiation fails or does not yield meaningful gradients.
## Duality Principles for Modern Machine Learning

The ICML Duality Principles workshop brings together researchers working on various duality concepts from many different fields to discuss new applications for modern machine learning, especially focusing on topics such as model understanding, explanation, and adaptation in deep learning and reinforcement learning.

Duality is a pervasive and important principle in mathematics. Not only has it fascinated researchers in many different fields but it has also been used extensively in optimization, statistics, and machine-learning, giving rise to powerful tools such as

- Fenchel duality in convex optimization,
- Representer theorems in kernel methods and Bayesian nonparametrics,
- Dually-flat spaces in information geometry.

Duality played an important role in the past, but lately we do not see much work on duality principles, especially in deep learning. For example, Lagrange duality can be useful for model explanation because it allows us to measure sensitivity of certain perturbations, but this is not yet fully exploited. This slowdown is perhaps due to a growing focus on nonconvex and nonlinear problems where duality does not seem to be directly applicable.

## Topics

We invite submissions of papers related (but not limited) to the following topics:

**Theory of duality principle:**

- Representer theorems
- Lagrange and Fenchel dualities, generalized conjugacy and abstract convexity
- Duality on manifolds or metric spaces, geodesic convexity
- Convex relaxations and duality for nonconvex problems
- Duality for optimization over measures, duality in optimal transport
- Other dualities in mathematics, e.g., information geometry, algebraic geometry

**Practical applications of duality principle:**

- Fast knowledge adaptation and transfer, lifelong learning, few shot learning etc.
- Model understanding, explanation and interpretation
- Differentiable programming, smoothing discontinuous/discrete maps
- Reinforcement learning, control, and deep learning in general
# Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities

## About the Workshop

Proposed in 2016 as a privacy enhancing technique, federated learning and analytics (FL & FA) made remarkable progress in theory and practice in recent years. However, there is a growing disconnect between theoretical research and practical applications of federated learning. This workshop aims to bring academics and practitioners closer together to exchange ideas: discuss actual systems and practical applications to inspire researchers to work on theoretical and practical research questions that lead to real-world impact; understand the current development and highlight future directions. To achieve this goal, we aim to have a set of keynote talks and panelists by industry researchers focused on deploying federated learning and analytics in practice, and academic research leaders who are interested in bridging the gap between the theory and practice. Topics of interest include, but are not limited to, the following:

**Federated Learning and Analytics:**

- Scalable and robust federated machine learning systems.
- Novel cross-device and cross-silo production applications.
- Training, fine-tuning, and personalizing (foundation) models in federated settings.
- Federated analytics vs. federated learning: synergies and differences in algorithms and systems (characteristics, constraints, and orchestration).
- Approaches for addressing distribution shifts and continual learning in federated settings.
- Autotuned federated algorithms for hyperparameters, model architectures, etc.
- Federated learning and analytics as part of an AI lifecycle.
- Open-source frameworks and community for federated learning and analytics.
- Theoretical studies with realistic assumptions for practical settings.

**Privacy and Security in Federated Settings:**

- Differential privacy and other privacy-preserving technologies in federated settings.
- Privacy attacks and empirical privacy auditing techniques in federated contexts.
- Security attacks and defenses in federated settings.
- Multi-party computation protocols & trusted execution environments for federated computations.

**Decentralized Networks and Trustworthiness:**

- Challenges in fully decentralized networks compared to federated settings.
- Trustworthy decentralized learning at scale.

**Fairness, Responsibility, and Social Impact:**

- Fairness and responsible models in federated settings.
- Social impact and privacy policies in federated settings.
## New Frontiers in Learning, Control, and Dynamical Systems

Recent advances in algorithmic design and principled, theory-driven deep learning architectures have sparked a growing interest in control and dynamical system theory. Complementary, machine learning plays an important role in enhancing existing control theory algorithms in terms of performance and scalability. The boundaries between both disciplines are blurring even further with the rise of modern reinforcement learning, a field at the crossroad of data-driven control theory and machine learning. This workshop aims to unravel the mutual relationship between learning, control, and dynamical systems and to shed light on recent parallel developments in different communities. Strengthening the connection between learning and control will open new possibilities for interdisciplinary research areas.

## Topics

We invite submissions related (but not limited) to the following topics:

- Optimal Transport
- Stochastic Processes
- Stochastic Optimal Control
- Dynamical Probabilistic Inference, e.g., MCMC, Variational Inference
- Diffusion Models
- Neural ODEs, SDEs, or PDEs
- Reinforcement Learning