chchenhui/mlrbench-tasks huggingface dataset:
## Workshop on Bayesian Decision-making and Uncertainty

Recent advances in ML and AI have led to impressive achievements, yet models often struggle to express uncertainty, and more importantly, make decisions that account for uncertainty. This hinders the deployment of AI models in critical applications, ranging from scientific discovery, where uncertainty quantification is essential, to real-world scenarios with unpredictable and dynamic environments, where models may encounter data vastly different from their training sets.

Through the use of probability, Bayesian methods offer a powerful framework to address these limitations by quantifying uncertainty, incorporating prior knowledge, enabling adaptive decision-making and information gathering in uncertain environments. These approaches have led to significant progress and success in relevant fields, tackling critical problems such as drug discovery, hyperparameter tuning and environmental monitoring. However, challenges remain in both theory and practice, such as establishing performance guarantees and scaling up these methods to handle the complexity and dimensionality of larger data and models. On the other hand, the development of frontier models (e.g., based on large language models) presents new opportunities to enhance Bayesian methods with stronger priors and tools not previously available.

This workshop aims to bring together researchers from different but closely related areas, including Bayesian optimization, active learning, uncertainty quantification, Gaussian processes, spatiotemporal modeling, and sequential experimental design. We seek to foster a vibrant exchange of ideas, showcase successful applications, and prompt fruitful discussion to collaboratively tackle the emerging challenges and shape the future directions of Bayesian decision-making and uncertainty in the new era of ML and AI.
## Workshop on Behavioral Machine Learning

Across many application areas, machine learning systems rely on human data. Yet these systems often leave unmodelled the psychological processes that generate human data.

Fortunately, there's a field full of insights about human behavior: the behavioral sciences. However, many of these  insights are qualitative. Integrating them into machine learning systems requires converting them into computational models and designing machine learning systems to incorporate them.

The goal of this workshop is to explore the incorporation of insights from the behavioral sciences into AI models/systems. We hope to bring together computer scientists across many subfields — e.g. AI, robotics, HCI — with behavioral scientists to drive progress in this interdisciplinary area.  

## Topics

- **Alignment:** Aligning LLMs and other large-scale generative models with models of human behavior inspired by the behavioral sciences

- **Evaluation:** Evaluating AI systems by incorporating models of human interaction

- **Computational cognitive science:** Incorporating formal models of human cognition into AI systems

- **Computational creativity:** Integrating psychological models of creativity into generative AI systems

- **Robotics:** Enhancing human-robot interaction through behavioral models

- **Interpretability:** Using behavioral models to improve the interpretability of AI systems
## Causality and Large Models
The remarkable capabilities and accessibility of recent large models, also known as “foundation models,” have sparked significant interest and excitement in the research community and beyond. In particular, large pre-trained generative models have demonstrated remarkable competencies in understanding and generating human-like text despite being trained on largely unstructured data using relatively simple self-supervised learning objectives. This raises the question: (A) Why do such large models work so well?

The impressive performance, sometimes even exceeding human experts, across a wide variety of benchmarks, together with the incorporation of multiple modalities such as images, text, and audio, makes these large models particularly versatile decision-making systems. However, the increased adoption of these models is not without challenges. The increasing size and complexity of these “black box” models raises concerns about their trustworthiness and reliability. For real-world applications, where distribution shifts are pervasive and sufficient high-quality data may be difficult or expensive to collect, it is crucial to systematically verify and enhance the robustness and generalization capabilities of these models. This is especially pertinent in safety-critical domains, such as healthcare and policy-making. Consequently, we must consider: (B) Under what circumstances can we trust these large models and how can this be improved?

Enter causality: a systematic framework to formalize “why?” and “how?” questions much like (A) or (B) and develop principled tools to address them. Causal inference is a powerful approach to describe a system’s behavior under interventions and reason over counterfactual scenarios. By relying on stable causal relationships, instead of potentially spurious statistical correlations, causal models can transparently elucidate a system’s behavior and enable performance guarantees beyond the training distribution, which is crucial for high-risk applications. However, translating the rigorous theoretical tools of causality into practical methods, especially in the large-scale regime with heterogeneous unstructured data as in large models, remains a notable challenge, despite the growing attention by the community.

## Topics
With the striking potential of causality and the enormous interest in tackling the many open questions about understanding and improving large models on the other, we propose a workshop that aims to explore the many exciting synergies between causality and large models. Specifically, we identify four main directions to cover in our workshop:

- Causality in large models: Assessing the causal knowledge captured by large models and their (causal) reasoning abilities.

- Causality for large models: Applying ideas from causality to augment and improve large models.

- Causality with large models: Leveraging large models to improve causal inference and discovery.

- Causality of large models: Investigating the causal structure of how large models work and how to make them more interpretable and controllable.
## Workshop on Machine Learning and Compression

The workshop solicits original research in the intersection of machine learning, data/model compression, and more broadly information theory.

Machine learning and compression have been described as “two sides of the same coin”, and the exponential amount of data being generated in diverse domains underscores the need for improved compression as well as efficient AI systems. Leveraging deep generative models, recent machine learning-based methods have set new benchmarks for compressing images, videos, and audio. Despite these advances, many open problems remain, such as computational efficiency, performance guarantees, and channel simulation. Parallel advances in large-scale foundation models further spurred research in efficient AI techniques such as model compression and distillation. This workshop aims to bring together researchers in machine learning, data/model compression, and information theory. It will focus on enhancing compression techniques, accelerating large model training and inference, exploring theoretical limits, and integrating information-theoretic principles to improve learning and generalization. By bridging disciplines, we seek to catalyze the next generation of scalable, efficient information-processing systems.

## Topics
Topics of interest include, but are not limited to,

- Improvements in learning-based techniques for compressing data, model weights, implicit/learned representations of signals, and emerging data modalities.
- Accelerating training and inference for large foundation models, potentially in distributed settings.
- Theoretical understanding of neural compression methods, including but not limited to fundamental information-theoretic limits, perceptual/realism metrics, distributed compression and compression without quantization.
- Understanding/improving learning and generalization via compression and information-theoretic principles.
- Information-theoretic aspects of unsupervised learning and representation learning.
## Workshop on Scalable Continual Learning for Lifelong Foundation Models

For the pursuit of increasingly general intelligence, current foundation models are fundamentally limited by their training on static data, leading to outdated encoded information, saturation in knowledge accumulation, and wasteful use of compute resources. The increasing size of machine learning (ML) models puts ever more emphasis on scalable learning since even fine-tuning large models is becoming increasingly resource-intensive and time-consuming. Continual learning (CL) now emerges as a crucial framework in this new era, essential for dealing with the evolving scale and complexity of ML models. Yet, even the most recent methods in CL fall short of effectively addressing the challenges posed by the current data and compute scales. 

At this workshop, we discuss recent advances in scalable CL that could potentially replace static foundation model (FM) training, enabling us to model dynamic real-world information. We bring together experts and researchers from various domains, including language, vision, speech, and multimodal ML to exchange ideas and foster collaboration. With invited and contributed talks by distinguished researchers in the area, the workshop will delve into the evolving definition of CL, and how CL can enable the efficient development of foundation models. 

## Topics
We welcome all contributions related to scaling the continual learning of foundation models. Potential areas of interest include but are not limited to:

- How should CL methods be utilized to avoid retraining large, foundation models?

- How can we address the challenge of catastrophic forgetting when fine-tuning FMs on considerably smaller and less diverse datasets in comparison to the extensive pretraining datasets?

- How can we address CL on a large scale when dealing with real-world problems with domain shifts and long-tailed data distributions?

- How can insights from other fields (online learning, meta-learning, reinforcement learning, neuroscience, AutoML, etc) inform and advance our CL of FMs?

- Does combining FMs with structured knowledge sources (databases, knowledge graphs, etc) help CL?

- What are the key considerations in designing benchmarks, evaluation protocols, and appropriate metrics for assessing CL of FMs?

- How can recent advances in FMs enhance CL techniques?

- What strategies can facilitate the seamless integration of CL and multi-modal learning systems?
## Causal Representation Learning Workshop

Advanced Artificial Intelligence (AI) techniques based on deep representations, such as GPT and Stable Diffusion, have demonstrated exceptional capabilities in analyzing vast amounts of data and generating coherent responses from unstructured data. They achieve this through sophisticated architectures that capture subtle relationships and dependencies. However, these models predominantly identify dependencies rather than establishing and making use of causal relationships. This can lead to potential spurious correlations and algorithmic bias, limiting the models’ interpretability and trustworthiness. In contrast, traditional causal discovery methods aim to identify causal relationships within observed data in an unsupervised manner. While these methods show promising results in scenarios with fully observed data, they struggle to handle complex real-world situations where causal effects occur in latent spaces when handling images, videos, and possibly text.

## Topics
Recently, causal representation learning (CRL) has made significant progress in addressing the aforementioned challenges, demonstrating great potential in understanding the causal relationships underlying observed data. These techniques are expected to enable researchers to identify latent causal variables and discern the relationships among them, which provides an efficient way to disentangle representations and enhance the reliability and interpretability of models. The goal of this workshop is to explore the challenges and opportunities in this field, discuss recent progress, and identify open questions, and provide a platform to inpire cross-disciplinary collaborations. This workshop will cover both theoretical and applied aspects of CRL, including, but not limited to, the following topics:

- Theory of causal representation learning
- Causal representation learning models
- Causal discovery with latent variables
- Causal generative models
- Causal Foundation Models
- Applications of causal representation learning, such as in biology, economics, image/video analysis, and LLMs
- Benchmarking causal representation learning
## Workshop on Data-driven and Differentiable Simulations, Surrogates, and Solvers

Recent advances in Machine Learning highlights promising solutions to aid simulation-based scientific discovery e.g., regulating nuclear fusion, synthesizing new molecules, designing chips. Since ML-based techniques are inherently learnable, they offer a promising solution to bridge the simulation-to-real gap and improve accuracy of simulations, and their differentiability addresses inverse problems by backpropagation through the simulation. Moreover, advances in novel architectures, optimization, and specialized hardwares might hold key to find better accuracy-speed trade-offs over conventional simulation softwares. Furthermore, probabilistic uncertainty quantification adds functionality for the estimation and control of epistemic, computational, and aleatoric uncertainty.

This workshop seeks to bring experts in Machine Learning working on relevant topics (like learnable surrogates, probabilistic simulation, operator-valued models) and connect them with practitioners and researchers in interdisciplinary topics from science (e.g., physics, climate, chemistry) and engineering (e.g., wireless, graphics, manufacturing). The workshop will provide a unique platform for ML and interdisciplinary researchers to expose challenges and opportunities of integrating ML methods and simulation techniques across these diverse domains.

## Topics
We are seeking submissions in topics including, but not limited to:

- Differentiable simulators and neural surrogates in various domains (e.g., graphics, EM-wave propagation, physics, molecular systems)
- Probabilistic Inverse Problems (e.g., simulation-based inference, posterior and likelihood estimation)
- Probabilistic Simulation (e.g. probabilistic ODE and PDE solvers, uncertainty quantification in operators, especially for data assimilation)
- Techniques to speed-up simulation (e.g., neural surrogates, efficient blackbox optimization)
- Improving simulation accuracy (e.g., mitigating sim2real gap, learnable formulation)
- Hybrid simulation and rendering approaches (e.g., neural fields)
- Generative modelling for simulation (e.g., biomolecule structure synthesis, material generation, image generation for autonomous vehicles)
- Datasets, gyms, and simulation softwares
## Evaluating Evaluations: Examining Best Practices for Measuring Broader Impacts of Generative AI

Generative AI systems are becoming increasingly prevalent in society, producing text, images, audio, and video content with far-reaching implications. While the NeurIPS Broader Impact statement has notably shifted norms for AI publications to consider negative societal impact, no standard exists for approaching these impact assessments.

This workshop addresses this critical gap by bringing together experts on evaluation science and practitioners who develop and analyze technical systems. We will share existing findings, develop future directions for effective community-driven evaluations, and create comprehensive frameworks for documenting and standardizing evaluation practices.

Key Focus: Breadth of Participation
A key focus of this workshop is broadening the expertise involved in shaping evaluations. Involving all participants and stakeholders in a system, not just Machine Learning and AI experts, can yield wide benefits. By encouraging collaboration among experts, practitioners, and the wider community, the workshop aims to create more comprehensive evaluations and develop AI community resources and policy recommendations.



## Topics

- Share existing findings and methodologies with the NeurIPS community
- Collectively develop future directions for effective community-built evaluations
- Address barriers to broader adoption of social impact evaluation of Generative AI systems
- Develop policy recommendations for investment in future directions for social impact evaluations
- Create a framework for documenting and standardizing evaluation practices
## Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability

This FITML workshop aims to contribute to the recent radical paradigm shift for fine-tuning in modern machine learning, theoretically, computationally, and systematically.

It encourages researchers to push forward the frontiers of theoretical understanding of fine-tuning, devising expeditious and resource-efficient inference and fine-tuning methods in machine learning systems, enabling their deployment within constrained computational resources.

This FITML workshop explores theoretical and/or empirical results for understanding and advancing modern practices for efficiency in machine learning.

# Topics

Key topics include but are not limited to:
- Exploration of new methodology for fine-tuning of various strategies, architectures and systems, from low-rank representation to sparse representation, from deep neural networks to LLMs, from algorithmic design to hardware design.
- Theoretical foundations of fine-tuning, e.g. approximation, optimization, and generalization from the perspective of transfer learning, deep learning theory, RLHF. Besides, theoretical understanding of low-rank representation from sketching and signal recovery are also welcome.
- Works that propose new experimental observations that can help advance our understanding of the underlying mechanisms of fine-tuning, a discrepancy between existing theoretical analyses and practice, explainability and interpretability of fine-tuning in scientific contexts.

The topics are not limited to fine-tuning, LLMs. Any topic on theoretical and/or empricial results for understanding and advancing modern practices for efficiency in machine learning is also welcome.
## Workshop on Large Foundation Models for Educational Assessment

The advanced generative artificial intelligence (AI) techniques, such as large language models and large multimodal models, are transforming many aspects of educational assessment. The integration of AI into education has the potential to revolutionize not only test development and evaluation but also the way students can learn. Over the past years, some successful adoptions of machine learning in this area are using natural language processing for automated scoring, or applying collaborative filtering to predict student responses. The rapid advances of large foundation models (e.g., ChatGPT, GPT-4, Llama, Gemini) demonstrate the potential of intelligent assessment with data-driven AI systems. These models could potentially benefit test construct identification, automatic item generation, multimodal item design, automated scoring, and assessment administration. Meanwhile, new research challenges arise in the intersection of AI and educational assessments. For instance, the explainability and accountability of current large foundations models are still inadequate to convince the stakeholders in the educational ecosystem, which limits the adoption of AI techniques in large-scale assessments. Also, it is still unclear whether the large foundation models are capable of assisting complex assessment tasks that involve creative thinking or high-order reasoning. Tackling these research challenges would require collaborative efforts from researchers and practitioners in both AI and educational assessment.



## Topics
This one-day workshop provides a forum for researchers from AI and educational assessment to review and discuss the recent advances of applying large foundation models for educational assessment. The workshop includes keynote talks and peer-reviewed papers (oral and poster). Original high-quality contributions are solicited on the following topics:

- Large foundation models for automated scoring
- Large foundation models for automated item generation
- Large foundation models for computerized adaptive testing
- Large foundation models for educational content generation
- Large foundation models for knowledge tracing
- Large foundation models for creating technology-enhanced items
- Knowledge augmentation of large models for educational assessment
- Knowledge editing of large models for educational assessment
- Finetune large foundation models for educational assessment
- Generative AI for assessment security and accountability
- Trustworthy AI (Fairness, Explainability, Privacy) for educational assessment
## GenAI for Health: Potential, Trust and Policy Compliance

Generative AI (GenAI) emerged as a strong tool that can revolutionize healthcare and medicine. Yet the public trust in using GenAI for health is not well established due to its potential vulnerabilities and insufficient compliance with health policies. The workshop aims to gather machine learning researchers and healthcare/medicine experts from both academia and industry to explore the transformative potential of GenAI for health. We will delve into the trustworthiness risks and mitigation of cutting-edge GenAI technologies applicable in health applications, such as Large Language Models, and multi-modality large models. By fostering multidisciplinary communication with experts in government policies, this workshop seeks to advance the integration of GenAI in healthcare, ensuring safe, effective, ethical, and policy-compliant deployment to enhance patient outcomes and clinical research.


## Topics

We invite paper submissions that have not been published, falling in but not limited to the following topics.
- Topic 1: GenAI use cases
    - For example, surveys of GenAI in healthcare, methodologies of using GenAI for data synthesis, simulation (e.g., digital twins), preliminary study, improving diagnosis accuracy, treatment assistance, and digital therapies.

- Topic 2: Trustworthiness and risks
    - For example, novel benchmarks of GenAI safety in specific or general health use cases, potential misuse, safeguarding techniques, reliability, and ethical disparities.

- Topic 3: Policy and compliance
    - For example, reviews of the latest policies in the association of AI and health, evaluation of the compliance of current GenAI applications, and pipelines to coordinate policymakers, GenAI developers, and security experts.

Papers will be submitted in three tracks: demonstration papers for the GenAI health applications, research papers for the policy-compliant GenAI trustworthiness in health or methodology of using GenAI for health, and position papers discussing policies and solutions for technical compliance. We encourage authors to involve multidisciplinary experts, especially the health community (e.g., stakeholders) and policymakers in writing the papers, which ensures the developed methods can address emerging stakeholders' and policymakers' concerns.