chchenhui/mlrbench-tasks huggingface dataset:
# What do we need for successful domain generalization?

## Workshop Description

The real challenge for any machine learning system is to be reliable and robust in any situation, even if it is different compared to training conditions. Existing general purpose approaches to domain generalization (DG) — a problem setting that challenges a model to generalize well to data outside the distribution sampled at training time — have failed to consistently outperform standard empirical risk minimization baselines. In this workshop, we aim to work towards answering a single question: what do we need for successful domain generalization? We conjecture that additional information of some form is required for a general purpose learning methods to be successful in the DG setting. The purpose of this workshop is to identify possible sources of such information, and demonstrate how these extra sources of data can be leveraged to construct models that are robust to distribution shift. Specific topics of interest include, but are not limited to:

- Leveraging domain-level meta-data
- Exploiting multiple modalities to achieve robustness to distribution shift
- Frameworks for specifying known invariances/domain knowledge
- Causal modeling and how it can be robust to distribution shift
- Empirical analysis of existing domain generalization methods and their underlying assumptions
- Theoretical investigations into the domain generalization problem and potential solutions
# Machine Learning for Materials

## Overview

Many of the world's most crucial challenges, such as access to renewable energy, energy storage, or clean water, are currently fundamentally bottlenecked by materials challenges. The discovery of new materials drives the development of key technologies like solar cells, batteries, and catalysis. Machine learning has significantly impacted the modeling of drug-like molecules and proteins, including the discovery of new antibiotics and the accurate prediction of 3D protein structures. Geometric deep learning methods, in particular, have made tremendous progress in modeling atomic structures and are a promising direction for solving open problems in computational materials science.

While there has been growing interest in materials discovery with machine learning, the specific modeling challenges posed by materials have been largely unknown to the broader community. In particular, compared with the domain of drug-like molecules and proteins, the modeling of materials has the two major challenges outlined below.

First, materials-specific inductive biases are needed to develop successful ML models. For example, materials often don't have a handy representation, like 2D graphs for molecules or sequences for proteins. Moreover, most materials are found in the condensed phase. This means they need to be represented under periodic boundary conditions, introducing challenges to both representation learning and generative models.

Second, there exists a broad range of interesting materials classes, such as inorganic crystals, polymers, catalytic surfaces, nanoporous materials, and more. Each class of materials demands a different approach to represent their structures and new tasks/data sets to enable rapid ML developments.

This workshop aims at bringing together the community to discuss and tackle these two types of challenges. In session A, we will feature speakers to discuss the latest progress in developing ML models for materials focusing on algorithmic challenges, covering topics like geometric deep learning and generative models. In particular, what can we learn from the more developed field of ML for molecules and proteins, and where might challenges differ and opportunities for novel developments lie? In session B, we will feature speakers to discuss unique challenges for each sub-field of materials design and how to define meaningful tasks that are relevant to the domain, covering areas including inorganic materials, polymers, nanoporous materials, and catalysis. More specifically, what are the key materials design problems that ML can help tackle?

## Topics

Example topics include (but not limited to):

- Representation of materials
- Generative models for materials
- Unique challenges in modeling materials with machine learning
- Physical inductive biases useful for machine learning models for materials
- Benchmark datasets and tools
- Machine learning potentials
- Automated experimental synthesis and characterization
- Integration of simulation and experimental data
- Language models on scientific literature
## Machine Learning & Global Health

During the Covid-19 pandemic, in spite of the impressive advances in machine learning in recent decades, the successes of this field were modest at best. Much work remains, for both machine learning and global health researchers, to deliver true progress in global health. This workshop will start a lasting and consistent effort to close the gap between advances in machine learning, practitioners and policy makers working in public health globally. It will focus on difficult public health problems and relevant machine learning and statistical methods.

We will use this opportunity to bring together researchers from different communities to share new ideas and past experiences. We will facilitate rapid communication of the latest methodological developments in machine learning to parties who are in positions to use them and establish feedback loops for assessing the applicability and relevance of methods that are available and gaps that exist. It will be a unique opportunity to challenge both research communities and demonstrate important, policy-relevant applications of sophisticated methods at one of the most prestigious annual machine learning conferences.

## Topics

This will be the first ever machine learning conference workshop on the topic ``Machine Learning & Global Health’’, sponsored by the Machine Learning & Global Health Network. By showcasing key applied challenges, along with recent theoretical advances, we hope to foster connections and prompt fruitful discussion. We will invite researchers to submit extended abstracts for contributed talks and posters along the themes of:

- What lessons can we learn from the COVID-19 pandemic?
- What sorts of questions in global health can machine learning be useful for? What sorts of questions in global health is machine learning unlikely to be useful for?
- The current limitations in the application of machine learning to solving global health problems and possible solutions to these limitations.
- How can we leverage machine learning in order to: promote public health worldwide; be proactive against future pandemics; understand and address inequalities in health.
- What types of data and data sharing practices would enable better machine learning and global health?

This workshop will start a lasting and consistent effort to close the gap between advances in machine learning, practitioners and policy makers working in public health globally. It will focus on difficult public health problems and relevant machine learning and statistical methods, which includes but is not limited to:

- Disease transmission models;
- Multi-agent modelling;
- Epidemiology and public health;
- Semi-mechanistic modelling of infectious disease dynamics; and
- Any work within the intersection of ML and global health
# Multimodal Representation Learning: Perks and Pitfalls

## About the workshop

Following deep learning, multimodal machine learning has made steady progress, becoming ubiquitous in many domains. Learning representations from multiple modalities can be beneficial since different perceptual modalities can inform each other and ground abstract phenomena in a more robust, generalisable way. However, the complexity of different modalities can hinder the training process, requiring careful design of the model in order to learn meaningful representations. In light of these seemingly conflicting aspects of multimodal learning, we must improve our understanding of what makes each modality different, how they interact, and what are the desiderata of multimodal representations. With this workshop, we aim to bring the multimodal community together, promoting work on multimodal representation learning that provides systematic insights into the nature of the learned representations, as well as ways to improve and understand the training of multimodal models, both from a theoretical and empirical point of view.

## Topics

We welcome submissions related to any aspects of multimodal representation learning, including but not limited to:

- Properties of multimodal representations.
- Insights on interactions across modalities.
- Novel applications regarding the nature and number of modalities.

In particular, we encourage submission that address the following questions:

- **Representation:** How do we identify useful properties of multimodal representations?
    - What semantic information is encoded in the learned representations?
    - How does the geometry of the representation space affect the quality of the learned representations?
    - What properties are leveraged for downstream tasks?
- **Training:** How can we promote useful properties of multimodal representations?
    - What are the limits of representation models, in regard to the number of modalities?
    - How do different learning objectives influence the resulting representations?
    - How do we promote the robustness of the representations to adversarial attacks, missing input modalities, and noise?
- **Modalities:** What makes a modality different? How can we improve their interactions?
    - How can we quantify the (dis)similarity between modalities?
    - How do different modalities contribute to the semantics of the learned representations?
    - What are the representation benefits of having multimodal observations as opposed to just a single modality?
    
The MRL workshop has the objective to bring together experts from the multimodal learning community in order to advance these fundamental questions and discuss the future of the field. We invite submissions that present analysis of the properties of multimodal representations, insights on interactions across modalities, as well as novel applications regarding the nature and number of modalities employed.
## Overview of Sparsity in Neural Networks

Deep networks with billions of parameters trained on large datasets have achieved unprecedented success in various applications, ranging from medical diagnostics to urban planning and autonomous driving, to name a few. However, training large models is contingent on exceptionally large and expensive computational resources. Such infrastructures consume substantial energy, produce a massive amount of carbon footprint, and often soon become obsolete and turn into e-waste. While there has been a persistent effort to improve the performance of machine learning models, their sustainability is often neglected. This realization has motivated the community to look closer at the sustainability and efficiency of machine learning, by identifying the most relevant model parameters or model structures. In this workshop, we examine the community’s progress toward these goals and aim to identify areas that call for additional research efforts. In particular, by bringing researchers with diverse backgrounds, we will focus on the limitations of existing methods for model compression and discuss the tradeoffs among model size and performance. 

## Topics 

The following is a non-exhaustive list of questions we aim to address through our invited talks, panels, and accepted papers:

- Where do we stand in evaluating and incorporating sustainability in machine learning? We make our models larger every day. Is this the right way to learn better?
- Do we need better sparse training algorithms or better hardware support for the existing sparse training algorithms? 
- Hardware seems to be behind in supporting sparse training. What are the challenges of hardware design for sparse and efficient training? Are GPUs the answer or do we need new designs?
- Our current theory can only analyze small neural networks. Can compression help us provide performance and reliability guarantees for learning?
- What are the tradeoffs between sustainability, efficiency, and performance? Are these constraints competing against each other? If so, how can we find a balance?
- Among different compression techniques, quantization has found more applications in industry. What is the current experience and challenges in deployment?
- How effective sparsity could be in different domains, ranging from reinforcement learning to vision and robotics?
## Overview of Scene Representations for Autonomous Driving

This workshop aims to promote the real-world impact of ML research toward self-driving technology. While ML-based components of modular stacks have been a huge success, there remains progress to be made in the development of integration strategies and intermediate representations. We invite contributions discussing the following topics, in order to empower the next generation of autonomous vehicles:

- Representation learning for perception, prediction, planning, simulation, etc
- Approaches that account for interactions between traditional sub-components (e.g., joint perception and prediction, end-to-end driving)
- ML / statistical learning approaches to facilitate safety / interpretability / generalization
- Driving environments / datasets for benchmarking ML algorithms
- New perspectives on the future of autonomous driving
## Trustworthy Machine Learning for Healthcare Workshop

Machine learning (ML) has achieved or even exceeded human performance in many healthcare tasks, owing to the fast development of ML techniques and the growing scale of medical data. However, ML techniques are still far from being widely applied in practice. Real-world scenarios are far more complex, and ML is often faced with challenges in its trustworthiness such as lack of explainability, generalization, fairness, privacy, etc. Improving the credibility of machine learning is hence of great importance to enhance the trust and confidence of doctors and patients in using the related techniques. We aim to bring together researchers from interdisciplinary fields, including but not limited to machine learning, clinical research, and medical imaging, etc., to provide different perspectives on how to develop trustworthy ML algorithms to accelerate the landing of ML in healthcare.

## Scope and Topics

Interested topics will include, but not be limited to:
- Generalization to out-of-distribution samples.
- Explainability of machine learning models in healthcare.
- Reasoning, intervening, or causal inference.
- Debiasing ML models from learning shortcuts.
- Fair ML for healthcare.
- Uncertainty estimation of ML models and medical data.
- Privacy-preserving ML for medical data.
- Learning informative and discriminative features under weak annotations.
- Human-machine cooperation (human-in-the-loop, active learning, etc.) in healthcare, such as medical image analysis.
- Multi-modal fusion and learning, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, pathology, genetics, electronical healthcare records, etc.
- Benchmarks that quantify the trustworthiness of ML models in medical imaging tasks.
## Pitfalls of limited data and computation for Trustworthy ML

Due to the impressive performance of ML algorithms, they are increasingly used in a wide range of applications that impact our daily lives. These include sensitive domains like healthcare, banking, social services, autonomous transportation, social media, advertisement, etc. However, ML algorithms that are deployed in the real world are restricted by a multitude of computational and statistical limitations. Often ignored in the ML research pipeline, these restrictions include
- **Statistical limitations:** lack of available data, limited availability of high-quality labelled data, and lack of data from different domains of interest
- **Computational limitations:** lack of high-speed hardware, lack of high memory hardware, extreme constraints on the computation time of ML algorithms during training or inference, and lack of hardware (e.g. hardware that cannot exploit sparsity) that is suitable for specific kinds of computations

It is necessary to understand the impact of such limitations on the performance of ML algorithms. As these algorithms are increasingly used for high-stakes decision-making in socially impactful domains, their trustworthiness is becoming an increasingly relevant design factor to consider. In recent years, several issues with the trustworthiness of ML algorithms have been identified:
- **Privacy:** Leaking private information about the training data.
- **Fairness:** Incurring disparate impact on sensitive subpopulations.
- **Miscalibration:** Giving a false sense of reliability through miscalibrated predictions.
- **Reproducibility:** Inconsistency across multiple runs of the ML pipeline.
- **Distribution shift:** Sensitivity to natural and adversarial test distribution shifts.
- **Robustness:** Vulnerability to noise in the training data.
- **Safety and Reliability:** Causing issues in the safety of resulting applications.
- **Explainability and Interpretability:** Identifying factors leading to predictions.
- **Auditing and Certifying ML systems:** Challenges of audit and certification under limited data and compute.

In this workshop, we want to invite theoretical and empirical researchers to come together and discuss barriers to trustworthy ML and algorithms that overcome them. To enable this, we will solicit submissions that address questions such as (but not limited to) the following:
- How does having less data or poor-quality data affect the trustworthiness of ML algorithms? Can these problems be mitigated with new algorithmic techniques (e.g. SSL, new DNN models, active learning)?
- How do computational limitations impact the trustworthiness of ML algorithms? What are some natural statistical tasks that exhibit fundamental trade-offs between computational efficiency (runtime, memory, etc.) and trustworthiness (fairness, privacy, robustness)? Are these trade-offs also observed in practice? 
- Do these limitations result in trade-offs between different aspects of trustworthiness? If yes, how can they be averted with relaxations or new algorithmic techniques?
## Workshop on Time Series Representation Learning for Health

Time series data have been used in many applications in healthcare, such as the diagnosis of a disease, prediction of disease progression, clustering of patient groups, online monitoring, and dynamic treatment regimes, to name a few. More and more methods build on representation learning to tackle these problems by first learning a (typically low-dimensional) representation of the time series and then use the learned representation for the corresponding downstream task.

Machine learning (ML) provides a powerful set of tools for time series data; however, its applicability in healthcare is still limited. As a result, the potential of time series analysis has yet to be fully realized. Our workshop on 'Time Series Representation Learning for Health' aims at bringing together the community to discuss cutting-edge research in this area, with a focus on the following themes:

- Labeling, in general and in particular of long-term recordings, is a nontrivial task requiring appropriate experts like clinicians who are restricted in their time
- Time series data acquired within real-life settings and novel measurement modalities are recorded without supervision, having no labels at all
- The high dimensionality of data from multimodal sources
- Missing values or outliers within acquired data or irregularity of measured data

This workshop focuses on these aspects and the potential benefits of integrating representation learning in time series applications. Our goal is to encourage a discussion around developing new ideas towards representation learning complemented with robust, interpretable, and explainable approaches which can provide a medical expert with more information than just a prediction result.

To make time series representation learning research actionable in clinical practice, we especially encourage discussions from application areas that tackle minority data groups and, thus, have their own unique challenges; for example, pediatrics, critical care (ICU), rare diseases like Alzheimer, HIV, fertility, and others.

## Topics

We solicit original paper submissions advancing research in representations learning with time series data, with a focus on healthcare applications. Under this premise, we encourage submissions touching topics such as:
- Robustness
- Explainable and interpretable methods
- Causality 
- Fairness
- Challenges of addressing time series data, such as 
    - labeling of real-world data, 
    - long-term recordings, 
    - handling high-dimensionality of data from multimodal sources, 
    - dealing with missing values and outliers in data or irregularlity of measured data
- Presenting novel open-access datasets 

Finally, we encourage work that is actionable in clinical practice, especially targeting application areas that tackle minority data groups and, thus, have their own specific, often under-explored, challenges. Such areas include, but are not limited to, pediatrics, critical care (ICU), rare diseases like Alzheimer, HIV, and fertility.
# How Far Are We From AGI

## Topics

This workshop aims to become a melting pot for ideas, discussions, and debates regarding our proximity to AGI. We invite submissions on a range of topics including, but not limited to:

1. **Frontiers of AGI research:** Examples include AI agents, embodied AI, retrieval-based and tool- augmented LLMs, knowledge-enhanced AI, and multi-agent AI.

2. **Classic AGI Attempts as Inspiration:** Delving into historical methods such as expert systems, symbolic AI, Type I and Type II reasoning for insights that can guide LLM research further.

3. **Interdisciplinary Insights for AGI:** Drawing parallels from fields like psychology, sociology, and neuroscience to inspire and inform the development of LLMs towards AGI.

4. **Fundamental Limitations of LLMs:** Analyzing the intrinsic capabilities or lack thereof in LLMs that might impede their progression to AGI. This includes discussions on reasoning, planning, and more.

5. **Practical Limitations of LLMs and Foundation models:** Addressing external challenges like system constraints, computational costs, data acquisition barriers, and privacy concerns.

6. **Safety, Ethics, and Regulation in AGI Development:** Exploring the complexity of moral, safety, and regulatory concerns that will shape AGI’s evolution.

7. **AGI’s Economic and Societal Impacts:** Probing the potential changes AGI might initiate into our societies, economies, and daily lives.