chchenhui/mlrbench-tasks huggingface dataset:
## Interpretable Machine Learning in Healthcare

Applying machine learning (ML) in healthcare is gaining momentum rapidly.  However, the black-box characteristics of the existing ML approach inevitably lead to less interpretability and verifiability in making clinical predictions. To enhance the interpretability of medical intelligence, it becomes critical to develop methodologies to explain predictions as these systems are pervasively being introduced to the healthcare domain, which requires a higher level of safety and security. Such methodologies would make medical decisions more trustworthy and reliable for physicians, which could ultimately facilitate the deployment. On the other hand, it is also essential to develop more interpretable and transparent ML systems. For instance, by exploiting structured knowledge or prior clinical information, one can design models to learn aspects more aligned with clinical reasoning. Also, it may help mitigate biases in the learning process or identify more relevant variables for making medical decisions.

In this workshop, we aim to bring together researchers in ML, computer vision, healthcare, medicine, NLP, and clinical fields to facilitate discussions including related challenges, definitions, formalisms, and evaluation protocols regarding interpretable medical machine intelligence. Additionally, we will seek possible solutions such as logic and symbolic reasoning over medical knowledge graphs, uncertainty quantification, composition models, etc. We hope that the proposed workshop is fruitful in offering a step toward building autonomous clinical decision systems with a higher-level understanding of interpretability. 		

## Topics

Topics include (but are not limited to):

- Definition of interpretability in healthcare    
- Identification of out-of-distribution/failure prediction 
- Uncertainty quantification for medical decision making
- Designing quantification and measurement of interpretability in healthcare
- Robustness and generalization of Medical ML systems
- Graph reasoning in healthcare
- Developing interpretable ML methods aligned with clinical reasoning
- Embedding medical knowledge in ML systems
- Application of interpretation methods to disease understanding
- Auditing and debugging algorithms for diagnosis system
- Visualization of explanation for model prediction
- Personalized vs. population-level interpretation methods
## Localized Learning Workshop

Despite being widely used, global end-to-end learning has several key limitations. It requires centralized computation, making it feasible only on a single device or a carefully synchronized cluster. This restricts its use on unreliable or resource-constrained devices, such as commodity hardware clusters or edge computing networks. As the model size increases, synchronized training across devices will impact all types of parallelism. 
Global learning also requires a large memory footprint, which is costly and limits the learning capability of single devices. Moreover, end-to-end learning updates have high latency, which may prevent their use in real-time applications such as learning on streaming video. 
Finally, global backpropagation is thought to be biologically implausible, as biological synapses update in a local and asynchronous manner. To overcome these limitations, this workshop will delve into the fundamentals of localized learning, which is broadly defined as any training method that updates model parts through non-global objectives.

## Topics

Relevant topics include but are not limited to:
- Forward-forward learning
- Greedy training
- Decoupled or early-exit training
- Iterative layer-wise learning
- Asynchronous model update methods
- Biologically plausible methods for local learning
- Localized learning on edge devices
- Self-learning or data-dependent functions
- New applications of localized learning
## The Many Facets of Preference-based Learning

Learning from human preferences, or preference-based learning, has been critical to major advances in AI and machine learning. It is based on the fact that humans are more reliable at providing relative feedback compared to numerical values. Therefore, preference feedback is usually easier to collect and less biased. A recent success story that showed the dormant potential of learning from preference feedback is fine-tuning of large language models with a reward function learned from human feedback and reinforcement learning to follow instructions in a dialogue context. There are other areas where preference-based learning yielded promising results, such as guided image generation, robotics and self-driving vehicles, games, collaborative filtering, simulated continuous control tasks, optimization and search problems, and healthcare. Despite these ground-breaking successes, the most exciting opportunities still lie ahead of us.

The broad objective of this workshop is twofold:

1. Bring together different communities where preference-based learning has played a major role.

2. Connect theory to practice by identifying real-world systems that can benefit from incorporating preference feedback.

The aim of this workshop is to create a suitable platform for sharing techniques and ideas, learning from each other, and potentially posing new and groundbreaking research questions.

## Topics

We cordially invite researchers who feel addressed by the theme of the workshop to submit their latest works to our workshop. Topics include but are not limited to:

- Collaborative filtering
- Control theory
- Convex optimization
- Dueling and preference-based bandit
- Econometrics and assortment selection
- Fairness
- Game theory, equilibria, and multiplayer games
- Marketing and revenue management
- Multi-objective optimization 
- Ranking aggregation
- Recommender systems 
- Reinforcement learning 
- Robotics
- Search engine optimization
- Social choice theory
# Neural Compression: From Information Theory to Applications

## Overview

The workshop solicits original research in the intersection of machine learning, data/model compression, and more broadly information theory.

Machine learning and compression have been described as “two sides of the same coin”, and the exponential amount of data being generated in diverse domains underscores the need for improved compression as well as efficient AI systems. Leveraging deep generative models, recent machine learning-based methods have set new benchmarks for compressing images, videos, and audio. Despite these advances, many open problems remain, such as computational efficiency, performance guarantees, and channel simulation. Parallel advances in large-scale foundation models further spurred research in efficient AI techniques such as model compression and distillation. This workshop aims to bring together researchers in machine learning, data/model compression, and information theory. It will focus on enhancing compression techniques, accelerating large model training and inference, exploring theoretical limits, and integrating information-theoretic principles to improve learning and generalization. By bridging disciplines, we seek to catalyze the next generation of scalable, efficient information-processing systems.

## Topics

Topics of interest include, but are not limited to,

- Improvements in learning-based techniques for compressing data, model weights, implicit/learned representations of signals, and emerging data modalities.
- Accelerating training and inference for large foundation models, potentially in distributed settings.
- Theoretical understanding of neural compression methods, including but not limited to fundamental information-theoretic limits, perceptual/realism metrics, distributed compression and compression without quantization.
- Understanding/improving learning and generalization via compression and information-theoretic principles.
- Information-theoretic aspects of unsupervised learning and representation learning.
# PAC-Bayes Meets Interactive Learning Workshop

## Scope

Interactive learning encompasses online learning, continual learning, active learning, bandits, reinforcement learning, and other settings where an algorithm must learn while interacting with a continual stream of data. Such problems often involve exploration-exploitation dilemmas, which can be elegantly handled with probabilistic and Bayesian methods. Deep interactive learning methods leveraging neural networks are typically used when the setting involves rich observations, such as images. As a result, both probabilistic and deep interactive learning methods are growing in popularity. However, acquiring observations in an interactive fashion with the environment can be costly. There is therefore great interest in understanding when sample-efficient learning with probabilistic and deep interactive learning can be expected or guaranteed. Within statistical learning theory, PAC-Bayesian theory is designed for the analysis of probabilistic learning methods. It has recently been shown to be well-suited for the analysis of deep learning methods. This workshop aims to bring together researchers from the broad Bayesian and interactive learning communities in order to foster the emergence of new ideas that could contribute to both theoretical and empirical advancement of PAC-Bayesian theory in interactive learning settings.

## Topics

We are interested in (but not limited to) the following topics:
- Explaining the success of existing interactive learning algorithms with PAC-Bayesian theory
- PAC-Bayesian analysis of exploration-exploitation trade-offs
- PAC-Bayes bounds under distribution shift
- PAC-Bayes bounds under adversarial corruptions
- Development of practically useful interactive learning algorithms using PAC-Bayesian theory.
## Workshop on Spurious Correlations, Invariance and Stability

The workshop brings together domain experts and researchers to facilitate discussions and forge collaborations on problems with spurious correlations, and instability of machine learning models. Models built without accounting for spurious correlations often break when deployed in the wild, despite excellent performance on benchmarks. In particular, models can learn to rely on apparently unnatural or irrelevant features. Such examples abound in recent literature:
1. In detecting lung disease from chest X-rays, models rely on the type of scanner and marks that technicians use in specific hospitals, instead of the physiological signals of the disease.
2. In Natural Language Processing, when reasoning whether a premise entails a hypothesis, models rely on the number of shared words rather than the subject’s relationship with the object.
3. In precision medicine, polygenic risk scores for diseases like diabetes and breast cancer rely on genes prevalent mainly in people of European ancestry, and are not as accurate in other populations.

Extensive work on resolving problems akin to spurious correlations has sprung up in several communities. These include works on invariance constraints and graph-based methods rooted in Causality, methods to avoid discrimination of compromised subgroups in Algorithmic Fairness, and stress testing procedures to discover unexpected model dependencies in reliable ML. Yet there is little consensus on best practices, useful formal frameworks, rigorous evaluations of models, and fruitful avenues for the future.

We invite work addressing all aspects of ML in the presence of spurious correlations, from formalization to deployment.

## Solicited Topics

We invite submissions that address discovery, learning, and unification in the presence of spurious correlations. We welcome a wide range of topics, including but not limited to: 
- Methods for discovering and diagnosing spurious correlations.
- Evaluation and stress tests of model stability.
- Impacts of different dataset shifts when learning exploits a shortcut/spurious correlation.
- Learning robust models in the presence of spurious correlations.
- Exploring relationships b/n methods from causal ML, algorithmic fairness, and OOD generalization.

Furthermore, we strongly encourage practitioners to submit examples of failure modes due to spurious correlations in real-world scenarios. We are particularly interested in submissions that can create new opportunities for collaboration, and motivate foundational research that is impactful in real-world applications.
# Sampling and Optimization in Discrete Space

## Overview

Sampling and optimization in discrete space are classical and important problems that arise in many applications, including physics, combinatorial optimization, compiler optimization, and many modern machine learning models like large language models and protein models. Examples include searching device placement strategies for distributed neural network training, or sampling from language model posteriors with arbitrary conditioning.

## Challenges

However, discrete space sampling/optimization is hard in general compared to continuous space. Although some independence structures can be leveraged for some special problems, in general discrete space sampling/optimization is slow. Recently, there have been new research trends in efficient discrete sampling and optimization via

- Leveraging the gradient information of objectives: the gradient based MCMC algorithms generalize the Langevin dynamics to discrete space.
- Embedding into a continuous space: The embedding methods first map the discrete space to continuous space, then apply an efficient sampling algorithm in continuous space, then map the new sample back into discrete space.
- Other effective proposal strategies in discrete space: for example the Stein variational method and GFlowNet. These methods have improved efficiency in many domains. With simulated annealing, they can also serve as optimization algorithms and demonstrate superior performance on many combinatorial optimization problems compared to existing learning-based methods. However, they might still be limited when applied to black-box objectives, or problems involving long-range and high-order correlations like in modern language models.

## Scope and Topics

Given this new direction of research and potential applications, we are organizing this workshop with the goals including, but not limited to:

- Syncing up on the latest research progress in discrete sampling and optimization.
- Discussing limitations of the current methods and brainstorm the new algorithm paradigms.
- Connecting to applications in domains such as language/protein modeling, physics simulation, and bio/chemical engineering, where improved sampling/optimization in discrete space would help, and explore the current gap between the application requirements and the capability of existing methods.

We hope this workshop will be a great opportunity for presenting and discussing new algorithms and applications with researchers and practitioners within or outside the domain of discrete sampling/optimization.
## Structured Probabilistic Inference & Generative Modeling

Probabilistic inference addresses the problem of amortization, sampling, and integration of complex quantities from graphical models, while generative modeling captures the underlying probability distributions of a dataset. Despite promising results, probabilistic methods face challenges when applied to highly structured data. We aim to bring experts from diverse backgrounds together, from both academia and industry, to discuss the applications and challenges of probabilistic methods, emphasizing challenges in encoding domain knowledge in these settings. We hope to provide a platform that fosters collaboration and discussion in the field of probabilistic methods.

## Scope

Relevant topics to this workshop include but are not limited to:

- Inference and generating methods for graphs, time series, text, video, and other structured modalities.
- Unsupervised representation learning of high dimensional structured data.
- Scaling and accelerating inference and generative models on structured data.
- Uncertainty quantification in AI systems.
- Applications and practical implementations of existing methods to areas in science.
- Empirical analysis comparing different architectures for a given data modality and application.
# Workshop on the Synergy of Scientific and Machine Learning Modeling

## Introduction

The Synergy of Scientific and Machine Learning Modeling Workshop (“SynS & ML”) is an interdisciplinary forum for researchers and practitioners interested in the challenges of combining scientific and machine-learning models. The goal of the workshop is to gather together machine learning researchers eager to include scientific models into their pipelines, domain experts working on augmenting their scientific models with machine learning, and researchers looking for opportunities to incorporate ML in widely-used scientific models.

The power of machine learning (ML), its ability to build models by leveraging real-world data is also a big limitation; the quality and quantity of training data bound the validity domain of ML models. On the other hand, expert models are designed from first principles or experiences and labelled scientific if validated on curated real-world data, often even harvested for this specific purpose, as advised by the scientific method since Galileo. Expert models only describe idealized versions of the world which may hinder their deployment for important tasks such as accurate forecasting or parameter inference. This workshop focuses on **the combination of two modelling paradigms: scientific and ML modelling.** Sometimes called hybrid learning or grey-box modelling, this combination should (1) unlock new applications for expert models, and (2) leverage the data compressed within scientific models to improve the quality of modern ML models. In this spirit, the workshop focuses on the symbiosis between these two complementary modelling approaches; it aims to be a “rendezvous” between the involved communities, spanning sub-fields of science, engineering and health, and encompassing ML, to allow them to present their respective problems and solutions and foster new collaborations.

## Topics

The workshop invites submissions on (but not limited to) the following topics:

- **Real-world application** of the combination of scientific and ML modelling: How can scientific models capitalise on ML to exploit raw data to broaden their applicability in the real world?
    - Astronomy
    - Biology
    - Chemistry
    - Geology
    - Robotics
    - Sub-domains of engineering
    - etc.
- **Methodological and theoretical study** on the combination of scientific and ML modelling: How can ML take advantage of the large amounts of data and human efforts hidden behind scientific models?
    - Model class and neural architectures
    - Learning algorithms
    - Data preparation
    - Theoretical analysis
## Topology, Algebra, and Geometry in Machine Learning Workshop

Much of the data that is fueling current rapid advances in machine learning is high dimensional, structurally complex, and strongly nonlinear. This poses challenges for researcher intuition when they ask (i) how and why current algorithms work and (ii) what tools will lead to the next big break-though. Mathematicians working in topology, algebra, and geometry have more than a hundred years worth of finely-developed machinery whose purpose is to give structure to, help build intuition about, and generally better understand spaces and structures beyond those that we can naturally understand. Following on the success of the first TAG-ML workshop in 2022, this workshop will showcase work which brings methods from topology, algebra, and geometry and uses them to help answer challenging questions in machine learning. Topics include mathematical machine learning, explainability, training schemes, novel algorithms, performance metrics, and performance guarantees.

**Topics Covered**
- Geometric Machine/Deep Learning 
- MathematicalMachine Learning
- Novel Algorithms
- Equivariant Models
- Explainability
- Interpretability
- Robustness 
- Performance
- Metrics 
- Performance Guarantees
- Training Methods
# Workshop on Theory of Mind in Communicating Agents

## Motivation

Theory of Mind (ToM) is the ability to reason about the minds of other agents. The main theme of our workshop is the computational modeling of ToM, with a special focus on the role of natural language in such modeling. Specifically, ToM 2025 pays attention to cognitive foundations and theories of ToM, the acquisition and relationship between language and ToM, leveraging ToM to improve and explain NLP and ML models, and using ToM for positive social impact. This workshop intends to promote the community of researchers that are interested in improving the ability of intelligent agents to reason about others' mental states. Our proposed program provides a space to discuss pathways for understanding and applying ToM in psycholinguistics, pragmatics, human value alignment, social good, model explainability, and many other areas of NLP. 

## Topics

**Potential topics include:**
- Leveraging ToM for Machine Learning Applications (e.g., NLP, Robotics, CV)
- Cognitive Science Perspectives of ToM
- ToM for HCI / Human-AI collaboration
- Surveys or replication of existing work
- Social Impacts of ToM