chchenhui/mlrbench-tasks huggingface dataset:
# Workshop on Generalization in Planning

Humans are good at solving sequential decision-making problems, generalizing from a few examples, and learning skills that can be transferred to solve unseen problems. However, these problems remain long-standing open problems in AI.

This workshop will feature a synthesis of the best ideas on the topic from multiple highly active research communities. On the one hand, recent advances in deep-reinforcement learning have led to data-driven methods that provide strong short-horizon reasoning and planning, with open problems regarding sample efficiency, generalizability and transferability. On the other hand, advances and open questions in the AI planning community have been complementary, featuring robust analytical methods that provide sample-efficient generalizability and transferability for long-horizon sequential decision making, with open problems in short-horizon control and in the design and modeling of representations.

# Topics

The workshop will focus on research related to all aspects of learning, generalization, and transfer in sequential decision-making (SDM). This topic features technical problems that are of interest not only in multiple sub-fields of AI research (including reinforcement learning, automated planning, and learning for knowledge representation) but also in other fields of research, including formal methods and program synthesis. We will welcome submissions that address formal as well as empirical issues on topics such as:
- Formulations of generalized SDM problems.
- Representations, learning, and synthesis for generalized plans and policies.
- Learning for transfer and generalization in reinforcement learning.
- Learning and representing hierarchical policies and behaviors for SDM.
- Learning and synthesis of generalizable solutions for SDM problem classes.
- Learning paradigms, representations, and algorithms for transferring learned knowledge and solutions to new SDM problems.
- Learning and representing generalized Q/V functions and heuristics for plan and policy generalization.
- Learning high-level models and hierarchical solutions for generalizable SDM.
- Neuro-symbolic approaches for generalization and transfer in SDM.
- Few-shot learning and transfer for SDM.
- Meta-learning for generalizable policies.
- Learning for program synthesis.
- Learning domain control knowledge and partial policies.
- Generalization and transfer in robot planning problems.
- Representation of solution structures that enable generalization and transfer.
# Heavy Tails in Machine Learning

Heavy-tailed distributions likely produce observations that can be very large in magnitude and far from the mean; hence, they are often used for modeling phenomena that exhibit outliers. As a consequence, the machine learning and statistics communities often associate heavy-tailed behaviors with rather negative consequences, such as creating outliers or numerical instability.

Despite their ‘daunting’ connotation, heavy tails are ubiquitous in virtually any domain: many natural systems have been indeed identified as heavy-tailed, and it has been shown that their heavy-tailed behavior is the main feature that determines their characteristics.

In the context of machine learning, recent studies have shown that heavy tails also naturally emerge in ML training in various ways, and, contrary to their perceived image, they can be in fact beneficial for the performance of an ML algorithm.

The ultimate goal of this workshop is to foster research and exchange of ideas at the intersection of applied probability, theory of dynamical systems, optimization and theoretical machine learning to make progress on practical problems where heavy tails, stability, or topological properties of optimization algorithms play an important role, e.g., in understanding learning dynamics. 

In our community, the emergence of heavy tails (and the edge of stability) is often perceived as a ‘phenomenon’, which essentially implies that they are rather ‘surprising’ or even ‘counterintuitive’.
We aim to break this perception and establish that such behaviors are indeed expected and the theory and methodology should be re-positioned accordingly.

# Topics
- Heavy tails in stochastic optimization
- Edge of stability
- Empirical scaling laws in large models
- Heavy-tailed auto-correlation
- Iterated function systems
- Heavy-tailed continuous dynamical systems
- Power-laws in ML
- Heavy tails and generalization
# Information-Theoretic Principles in Cognitive Systems

The InfoCog workshop is an interdisciplinary venue for exploring new avenues for progress toward an integrative computational theory of human and artificial cognition, by leveraging information-theoretic principles and formulations. To this end, we aim to bring together researchers from machine learning, cognitive science, neuroscience, linguistics, economics, and other fields, who are interested in information-theoretic approaches to cognitive systems, as well as researchers from information theory who focus on advanced methods for computation and estimation of information theoretic measures. 

# Topics

We invite submissions that present original work related to information theory and cognitive systems. We aim to bring together researchers from multiple disciplines (e.g., machine learning, cognitive science, neuroscience, linguistics, economics) who are interested in information-theoretic approaches to human and artificial cognition. This year will feature a special emphasis on connections with researchers focused on the computation/estimation of information-theoretic quantities, with the aim of tightening the collaboration across the machine learning, cognitive science, and information theory communities. We particularly encourage submissions of interdisciplinary work.

Examples of specific topics of interest include but are not limited to:

- Novel information-theoretic approaches to cognitive functions (e.g., perception, decision making, language, social reasoning,etc.)
- Method and approaches for the validation of information-theoretic formalism in human and artificial cognition
- Novel methods and approaches for the computation and/or estimation of information theoretic quantities and their application tohuman and artificial cognition
- Challenges and limitations of the use of information theory in studying cognitive systems
- Application of information theory to training human-aligned artificial agents, i.e., agents that can better communicate and cooperate with humans
# Instruction Tuning and Instruction Following

Recent advancements in training large language models (LLMs) to follow “instructions” have significantly increased their ability to comprehend open-ended language commands, encompassing a wide range of needs, preferences, and values.

This remarkable transformation has led to the creation of remarkable industrial models such as GPT-4 and Bard , as well as an increased focus within the open-source and research communities: creating new benchmark and resources, developing new training methods, and understanding the limitations of these methods. Furthermore, instruction following powered by LLMs has proven to be effective in multi-modal settings, with applications in image editing and robotic command execution.

# Topics
we invite submissions covering various topics, including but not limited to the list below:
- Modeling: algorithms and pipelines for learning from instructions and human feedback; designing training objectives and rewards; training and inference efficiency
- Data Collection: crowd-sourcing; synthetic data generation; data democratization
- Evaluation and Oversight: effective and reliable oversight over existing models; enforcing guardrails and guarantees for model behaviors; interpretability and analysis
- Engineering and Open-sourcing: best practice in training, evaluation and deployment; open-sourcing efforts; openness and reproducibility
- Applications: long-context, multi-round and personalized instruction-following models
- Multimodal and Multidisciplinary: instruction following models for computer vision, robotics, games, art, etc.
- Limitations, Risks and Safety: bias and fairness; factuality and hallucination; safety concerns arising from instruction-following models
- Other adjacent research topics (e.g., in-context learning, prompting, multi-task learning) that enable better responses to instructions in dynamic environments
# Mathematical Reasoning and AI 

Mathematical reasoning is a fundamental aspect of human cognition that has been studied by scholars ranging from philosophers to cognitive scientists and neuroscientists. Mathematical reasoning involves analyzing complex information, identifying patterns and relationships, and drawing logical conclusions from evidence. It is central to many applications in science, engineering, finance, and everyday contexts.

Recent advancements in large language models (LLMs) have unlocked new opportunities at the intersection of artificial intelligence and mathematical reasoning, ranging from new methods that solve complex problems or prove theorems, to new forms of human-machine collaboration in mathematics and beyond. 

Our proposed workshop is centered on the intersection of deep learning and mathematical reasoning, with an emphasis on, but not limited to, large language models.
Our guiding theme is: “To what extent can machine learning models comprehend mathematics, and what applications could arise from this capability?”

To address this question, we aim to bring together a diverse group of scholars from different backgrounds, institutions, and disciplines into our workshop. Our objective is to foster a lively and constructive dialogue on areas related, but not limited, to the following:
- Humans vs. machines: A comparative study of human-level mathematical reasoning and current AI techniques. How do they differ, complement one another, or intersect?
- Measuring mathematical reasoning: How do we design benchmarks which accurately evaluate mathematical reasoning abilities, especially in an era of large language models?
- New capabilities: How do we move beyond our current techniques?
- Education: What role can deep learning models play in mathematics education, especially in contexts with limited educational resources?
- Applications: What applications could AI systems enable in the near- and long-term? Example domains include software verification, sciences, engineering, finance, education, and mathematics itself.
# Medical Imaging

'Medical Imaging meets NeurIPS' is a satellite workshop established in 2017. The workshop aims to bring researchers together from the medical image computing and machine learning communities. The objective is to discuss the major challenges in the field and opportunities for joining forces. This year the workshop will feature online oral and poster sessions with an emphasis on audience interactions. In addition, there will be a series of high-profile invited speakers from industry, academia, engineering and medical sciences giving an overview of recent advances, challenges, latest technology and efforts for sharing clinical data.

Medical imaging is facing a major crisis with an ever increasing complexity and volume of data and immense economic pressure. The interpretation of medical images pushes human abilities to the limit with the risk that critical patterns of disease go undetected. Machine learning has emerged as a key technology for developing novel tools in computer aided diagnosis, therapy and intervention. Still, progress is slow compared to other fields of visual recognition which is mainly due to the domain complexity and constraints in clinical applications which require most robust, accurate, and reliable solutions. The workshop aims to raise the awareness of the unmet needs in machine learning for successful applications in medical imaging.

We invite submissions of extended abstracts for oral and poster presentation during the workshop. Submitting an abstract is an ideal way of engaging with the workshop and to showcase research in the area of machine learning for medical imaging. Submitted work can be of preliminary nature and we also invite perspectives and position papers to generate discussions about recent trends and major challenges.
# Machine Learning with New Compute Paradigms

Digital computing is approaching fundamental limits and faces serious challenges in terms of scalability, performance, and sustainability. At the same time, generative AI is fuelling an explosion in compute demand. There is, thus, a growing need to explore non-traditional computing paradigms, such as (opto-)analog, neuromorphic hardware, and physical systems. Expanding on last year's successful NeurIPS workshop, which was the first of its kind in this community, we aim to bring together researchers from machine learning and alternative computation fields to establish new synergies between ML models and non-traditional hardware. Co-designing models with specialized hardware, a feature that has also been key to the synergy of digital chips like GPUs and deep learning, has the potential to offer a step change in the efficiency and sustainability of machine learning at scale. Beyond speeding up standard deep learning, new hardware may open the door for efficient inference and training of model classes that have been limited by compute resources, such as energy-based models and deep equilibrium models. So far, however, these hardware technologies have fallen short due to inherent noise, device mismatch, a limited set of compute operations, and reduced bit-depth. As a community, we need to develop new models and algorithms that can embrace and, in fact, exploit these characteristics. This workshop aims to encourage cross-disciplinary collaboration to exploit the opportunities offered by emerging AI accelerators both at training and at inference.
# Overview
The central theme of the workshop will be the application of moral philosophy and moral psychology theories to AI practices. Our invited speakers are some of the leaders in the emerging efforts to draw on theories in philosophy or psychology to develop ethical AI systems. Their talks will demonstrate cutting-edge efforts to do this cross-disciplinary work, while also highlighting their own shortcomings (and those of the field more broadly). Each talk will receive a 5-minute commentary from a junior scholar in a field that is different from that of the speaker. We hope these talks and commentaries will inspire conversations among the rest of the attendees.

# Topics
Ideal submissions will show how a theory from moral philosophy or moral psychology can be applied in the development or analysis of ethical AI systems. For example:
- How can moral philosophers and psychologists best contribute to ethically-informed AI?
- What can theories of developmental moral psychology teach us about making AI?
- How do theories of moral philosophy shed light on modern AI practices?
- How can AI tools advance the fields of moral philosophy and psychology themselves?
- How can findings from moral psychology inform the trustworthiness, transparency or interpretability of AI decision-makers?
- What human values are already embedded in current AI systems?
- Are the values embedded in the current-day AI systems consistent with those in society at large?
- What pluralistic values are missing from current-day AI?
- Methodologically, what is the best way to teach an AI system human values? What are competitors to RLHF, reinforcement learning from human feedback?
- Concerning AI alignment, to which values are we to align? Is the current practice of AI alignment amplifying monolithic voices? How can we incorporate diverse voices, views and values into AI systems?
# Optimization for Machine Learning

Optimization lies at the heart of many machine learning algorithms and enjoys great interest in our community. Indeed, this intimate relation of optimization with ML is the key motivation for the OPT series of workshops. We aim to foster discussion, discovery, and dissemination of state-of-the-art research in optimization relevant to ML.

The focus of OPT 2024 is on "Scaling up optimization". The advent of large language models (LLMs) has changed our perceptions of the landscape of optimization and is resulting in the emergence of new interesting questions related to scaling. For instance, we can view optimization as a sequence of problems parameterized by the size of the model. Questions naturally arise around scaling and optimization. Are there natural model size dependent learning rates that allow extrapolation from smaller models to large ones, and therefore facilitating fine-tuning? Or given a fixed compute budget, how should one choose the hyper-parameters of the model (e.g., width size, depth size, architecture, batch) so as to minimize the loss function? How dependent are these scaling laws on the optimization algorithm? Answers to these questions would have a huge impact in AI – saving time and millions of dollars in training, plus helping reduce AI’s environmental impact through reducing energy costs. The new area of scaling laws and its deep ties to the optimization community warrants a necessary discussion.

# Topics

We particularly encourage submissions in the area of "scaling up optimization", with works contributing to bridging new and classical optimization methodology with challenges in large machine learning models and their scaling laws.

The main topics are, including, but not limited to:

- Adaptive Stochastic Methods
- Algorithms and techniques (higher-order methods, algorithms for nonsmooth problems, optimization with sparsity constraints, online - optimization, streaming algorithms)
- Approaches to Adversarial Machine Learning
- Average-case Analysis of Optimization Algorithms
- Combinatorial optimization for machine learning
- Deep learning optimization
- Federated learning
- Games; min/max theory
- Nonconvex Optimization
- Optimization software (integration with existing DL software, hardware accelerators and systems)
- Parallel and Distributed Optimization for large-scale learning
- Privacy and Optimization
- Scaling laws
- The Interface of Generalization and Optimization
# Optimal Transport and Machine Learning

Over the last decade, optimal transport (OT) has evolved from a prize-winning research area in pure mathematics to a recurring theme bursting across many areas of machine learning (ML). Advancements in OT theory, computation, and statistics have fueled breakthroughs in a wide range of applications, from single-cell genomics to generative modeling and the optimization of over-parametrized neural nets , among many others. The OTML workshop series has been instrumental in shaping this influential research thread. The OTML workshop aims to provide a unique platform to federate, disseminate, and advance current knowledge in this rapidly growing field.

# Topics
We invite researcher in optimal transport and machine learning to submit their latest works to our workshop.

Topics include but are not limited to:
- Optimal Transport Theory
    - OT with generalized choices of cost functions
    - Study of partial differential equations and Wasserstein gradient flows (theory + applications)
    - Limits of regularization schemes
- Generalizations of Optimal Transport
    - Unbalanced formulation (OT between measures of different mass)
    - Gromov-Wasserstein formulation (OT with rigid transformations)
    - Multi-marginal OT
    - Martingale OT (financial applications, etc)
- Computational and Statistical Optimal Transport
    - Estimation of Monge maps, couplings, etc.
    - Finite-sample convergence guarantees
    - Limit distribution theory
    - Study of complexity of OT algorithms
- Optimal Transport for Machine Learning and Applications
    - OT costs as a loss (e.g. GANs, minimization of Wasserstein distance between empirical and population measures)
    - OT to define data transformations (domain adaptation, clustering)
    - High-dimensional applications such as Natural Language Processing, computational biology, vision tasks, etc.
    - Low-dimensional applications such as graphics, shapes, imaging, etc.
# Workshop on Adaptive Experimental Design and Active Learning in the Real World

This workshop aims to bring together researchers from academia and industry to discuss major challenges, outline recent advances, and highlight future directions pertaining to novel and existing real-world experimental design and active learning problems. In addition, we aim to highlight new and emerging research opportunities for the machine learning community that arise from the evolving needs to make experimental design and active learning procedures that are theoretically and practically relevant for practical applications. Examples include protein design, causal discovery, drug design, and materials design, to name a few.

Whether in robotics, protein design, or physical sciences, one often faces decisions regarding which data to collect or which experiments to perform. There is thus a pressing need for algorithms and sampling strategies that make intelligent decisions about data collection processes that allow for data-efficient learning. Experimental design and active learning have been major research focuses within machine learning and statistics, aiming to answer both theoretical and algorithmic aspects of efficient data collection processes. The goal of this workshop is to identify missing links that hinder the direct application of these principled research ideas into practically relevant solutions. Progress in this area can provide immense benefits in using experimental design and active learning algorithms in emerging high-impact applications, such as materials design, computational biology, causal discovery, drug design, citizen science, etc.

# Topics

Technical topics of interest include (but are not limited to):

- Large-scale and real-world experimental design (e.g. drug design, physics, robotics, material design, protein design, causal discovery, citizen science).
- Efficient active learning and exploration.
- High-dimensional, scalable Bayesian and bandit optimization (e.g. contextual, multi-task).
- Effective off-policy evaluation and treatment-effect estimation.
- Effective exploration in high-dimensional spaces (e.g., through use of neural networks).
- Sample-efficient interactive learning, hypothesis, and A/B testing.
- Corrupted/indirect measurements, multi-fidelity, and multi-objective experimentation.
- Domain-knowledge integration (e.g., from physics, chemistry, biology, medicine).
- Safety and robustness during experimentation and of resulting designs.
- Experiment design/active learning in reinforcement learning.