chchenhui/mlrbench-tasks huggingface dataset:
# Workshop on Humans, Algorithmic Decision-Making and Society: Modeling Interactions and Impact

## About

With the widespread adoption of machine learning in social technologies, there are increasingly complex interactions between humans, algorithmic decision-makers, and society at large. For instance, algorithmic decisions influence the information and opportunities that are available to individuals, the news they read, the job listings they are matched to, the credit lines they receive, and the social circle they form. Such decisions can therefore affect societal outcomes such as social mobility, mental health, polarization etc. At the same time, humans also influence algorithmic decision-makers, for instance, by expressing their preferences through observed behaviors which might be inconsistent or strategic. To understand long-term individual and societal outcomes resulting from these interactions, and to develop algorithms that mitigate undesired outcomes, it has therefore become increasingly important to model these complex interactions as a whole.

The purpose of this workshop is to bring together researchers from both academia and industry working on the full spectrum of modeling interactions between AI systems, humans, and society, from theory to practice. We will invite speakers and solicit contributed papers and posters covering the various facets of these interactions. We are targeting different communities/fields such as machine learning, network science, social systems, algorithmic game theory, economics. We expect that the presence of these different communities will result in a fruitful exchange of ideas and stimulate an open discussion about the current challenges and possible solutions.

## Topics

We invite submissions that are related to the interplay of humans, algorithmic decision-making and society with a special focus on modeling interactions and their impact. In particular, we encourage submissions on the following topics:

- Feedback loops between human and algorithmic decisions, and their long-term impacts
- Strategic behavior and its impact on algorithmic decision-making
- Models for human utility/preferences in the presence of non-rational behavior
- Generative and foundation models for interpretable human behavior
- Emergent social phenomena and complex systems
- Modeling societal outcomes through multi-agent models, mean-field games, etc.
- Fairness and algorithmic approaches to mitigate disparate impact
## About

In-context learning (ICL) is an emerging capability of large-scale models, including large language models (LLMs) like GPT-3, to acquire new capabilities directly from the context of an input example without separate training or fine-tuning, enabling these models to adapt rapidly to new tasks, datasets, and domains. This workshop brings together diverse perspectives on this new paradigm to assess progress, synthesize best practices, and chart open problems. Core topics will include architectural and other inductive biases enabling in-context skill acquisition, and reliable evaluation of ICL in application domains including reinforcement learning, representation learning, and safe and reliable machine learning.

We invite submissions to the ICL 2024 workshop, focusing on the development of new architectures, algorithms, theoretical analysis, empirical studies, and applications of In-Context Learning (ICL). Submissions must present original research that has not been previously published.

## Topics

Specific topics of interest include, but are not limited to:

- architectures, training paradigms, and inductive biases that enable or improve ICL;
- theoretical analyses and guarantees for ICL methods;
- empirical evaluation of the performance of ICL on interpretability, controllability, and safety considerations for ICL systems;
- similarities and differences between ICL in large-scale language modeling systems and learned algorithms in other domains;
- the relationship between ICL and few-shot learning, meta-learning and automated machine learning (AutoML).
# Workshop on Long-Context Foundation Models

## Overview

Many challenging tasks for foundation models require synthesizing information over thousands to millions of individual pieces of data, which may take many forms, including images, text, audio, genomes, etc. Our workshop aims to convene researchers to address challenges in long-context foundation models, fostering discussions, developments, deployments, evaluation, and understanding of long-context foundation models across various AI disciplines. 

## Topics

The topics of this workshop include (but not limited to):

- New modeling, training, and data strategies.
- Efficiency techniques for (long-context) foundation models.
- Evaluation and understanding of long-context models.
- Retrieval-augmented foundation models.
- Interdisciplinary applications of LCFMs.
# Workshop on LLMs and Cognition

## About

Large Language Models (LLMs) have undoubtedly taken center stage in the AI revolution, showing impressive performance in a wide variety of tasks, including machine translation, standardized tests, and conversational chatbots. It is even more impressive to uncover that these models exhibit unpredictable capabilities in solving unseen tasks. This demonstration of emergent abilities, often credited to the scale of the parameters and data size in the case of LLMs, is being considered as the footprint of intelligence. The goal of this workshop is to assess and understand the position of current LLMs' abilities in the landscape of intelligent systems, with a strong focus on cognitive abilities. 

## Topics

By bringing in experts from different scientific disciplines, such as AI/ML, neuroscience, cognitive science, and psychology, we aim to discuss topics that include but not limited to:

- Where do LLMs stand in terms of performance on cognitive tasks, such as reasoning, navigation, planning, and theory of mind?
- What are the fundamental limits of language models with respect to cognitive abilities?
- How do LLMs fine-tuned on specific tasks end-to-end compare to augmented LLMs coupled with external modules?
- What are the similarities and differences between mechanistic interpretability approaches in AI and in neuroscience? What do they tell us about similarities and differences between LLMs and human brains?
- How can we improve existing benchmarks and evaluation methods to rigorously assess cognitive abilities in LLMs?
- Can multimodal and multiagent approaches address some of current limits of LLMs to cognitive tasks?
# Multi-modal Foundation Model meets Embodied AI 

## Overview

Multi-modal Foundation Model meets Embodied AI (MFM-EAI)In recent years, Multi-modal Foundation Models (MFM) such as CLIP, ImageBind, DALL·E 3, GPT-4V, and Gemini have emerged as one of the most captivating and rapidly advancing areas in AI, drawing significant attention and progressing swiftly. The open-source community for MFM has also seen vigorous growth, with the emergence of models and algorithms like LLaVA, LAMM, Stable Diffusion, and OpenFlamingo. These MFMs are now actively exploring ultimate application scenarios beyond traditional computer vision tasks.Recent studies have unveiled the immense potential these models hold in empowering embodied AI agents, marking the intersection of these fields with a multitude of open questions and unexplored territories. This workshop, MFM-EAI, is dedicated to exploring these critical challenges:

- How can we train and evaluate MFM in open-ended environments?

- What constitutes an effective system architecture for MFM-based Embodied AI Agents?

- How can MFM augment the perceptual and decision-making capabilities of these agents, balancing their high-level decision-making prowess with the nuanced requirements of low-level control in embodied systems?

## Topics

Topics include but are not limited to:
- Training and evaluation of MFM in open-ended scenarios

- Data collection for training Embodied AI Agents and corresponding MFM

- Framework design for MFM-powered embodied agents

- Decision-making in Embodied Agents empowered by MFM- Low-level control in Embodied Agents empowered by MFM

- Evaluation and simulation of Embodied Agents

- Limitations of MFM in empowering Embodied AI
# Workshop on Mechanistic Interpretability

## Overview

Aligning AI agents with human intentions and values is one of the main barriers to the safe and ethical application of AI systems in the real world, spanning various domains such as robotics, recommender systems, autonomous driving, and large language models. To this end, understanding human decision-making and interpreting human choices is fundamental for building intelligent systems that can interact with users effectively, align with their preferences, and contribute to the development of ethical and user-centric AI applications. 

Despite its vital importance for Human-AI Alignment, current approaches, such as Reinforcement Learning with Human Feedback (RLHF) or Learning from Demonstrations (LfD), rely on highly questionable assumptions about the meaning of observed human feedback and interactions. In fact, these assumptions remain mostly unchallenged by the community, and simplistic human feedback models are often being reused without any re-evaluation of their suitability. For example, we typically assume that a human acts rationally, that human feedback is unbiased, or that all humans provide similar feedback and have similar opinions. Of course, many of these assumptions are violated in practice, however, the role of such modeling assumptions has mostly been neglected in the literature on human-AI alignment. The goals of this workshop are: 
- to bring together different communities towards a better understanding of human feedback 
- to discuss different types of human feedback and discuss mathematical and computational models of human feedback and their shortcomings
- discuss important and promising future directions towards a better understanding of human feedback models and better AI alignment. 

## Topic

We invite researchers in machine learning, artificial intelligence, and related disciplines to submit their latest work to this workshop. We invite submissions related to the theme of the workshop. Relevant topics include, but are not limited to: 
- Learning from Demonstrations (Inverse Reinforcement Learning, Imitation Learning, ...)
- Reinforcement Learning with Human Feedback (Fine-tuning LLMs, ...)
- Human-AI Alignment, AI Safety, Cooperative AI 
- Robotics (Human-AI Collaboration, ...) 
- Preference Learning, Learning to Rank (Recommendation Systems, ...)
- Computation Social Choice (Preference Aggregation, ...) 
- Operations Research (Assortment Selection, ...)
- Behavioral Economics (Bounded Rationality, ...)
- Cognitive Science (Effort in Decision-Making, ...)
# Workshop on Machine Learning for Earth System Modeling 

## Summary

Climate change is a major concern for human civilization, yet significant uncertainty remains in future warming, change in precipitation patterns, and frequency of climate extremes. Proper adaptation and mitigation demands accurate climate projections capable of simulating the atmosphere, ocean, land, and their interactions. Numerical models exhaustively tuned by domain scientists have been the gold standard for modeling both weather and climate because of their interpretability and ability to simulate “what-if” scenarios not present in the historical record. Although AI forecasts have started to make operational progress in weather prediction, climate projections are a harder problem. For example, High Impact-Low Likelihood events are undersampled in ERA5 reanalysis data, and substantial decadal variability in modes of climate variability (like the El-Niño Southern Oscillation) limit the ability of AI forecasts to reliably extrapolate into the future. This workshop seeks to accelerate progress on using machine learning to improve climate projections, emphasizing areas that domain scientists have deemed amenable to machine learning approaches. Examples include
hybrid physics-ML climate models, where machine learning is used to emulate subgrid processes too expensive to resolve explicitly, and dynamical downscaling, where high-resolution climate variables are inferred from coarse-resolution models in a physically consistent manner. 

## Topics

We welcome submissions on machine learning topics that can advance earth system model development. Some examples include 
1. deep generative models
2. explainable AI
3. physics-informed neural networks
4. uncertainty quantification.
# Workshop ML for Life and Material Science: From Theory to Industry Applications

## Overview

This workshop aims to highlight translational ML research in biology and chemistry ML for real-world applications in life-and materials science. The goal is to bridge theoretical advanceswith practical applications and connect academic and industry researchers.

Biology and chemistry play a central role in understanding life, and are a fundamental pillar ofhuman well-being through their roles as medicines, materials, or agro-chemicals.

With increasingchallenges associated with climate change, growth of the global population, diseases associatedwith aging, and the global supply of food and energy, it is becoming increasingly urgent toaccelerate the pace at which technical discoveries can be made, and translated into practical solutions to these societal issues.

However, compared to other modalities such as images orlanguage, the study of biology and chemistry with machine learning is not as industriallyestablished. Multiple factors contribute to this delay. Different research questions require manylevels and scales of representation, from electronic structure to graph and point cloudrepresentations of (bio) molecules, to protein and nucleic acid sequences, crystals, omics data, celland tissue-level representations.

## Topics

We envision abalanced scientific industrial and academic attendance, and propose committees and a lineup that reflect a mix of top industry scientists, academic leaders and double-affiliated scientists, as well asemerging scientists and new voices in ML for healthcare, molecular-, life- and material sciences. We welcome a broad range of submissions, whose topics include
1. **dataset curation, analysis and benchmarking workhighlighting opportunities and pitfalls of current ML applications in health and materials**
2. **novelmodels and algorithms unlocking capabilities previously thought available only through non-MLapproaches**
# Next Generation of AI Safety

## Overview

In recent years, general-purpose AI has experienced a meteoric rise in capabilities and applications. This rise has continued to bring forth new safety challenges, requiring mitigation to ensure AI systems meet trustworthiness standards. 

In this workshop, we take a proactive approach to safety and focus on five emerging trends in AI and explore the challenges associated with deploying these technologies safely:
1. **Agentic AI**: As AI agents become more autonomous, concerns about unintended consequences, ethical issues, and adversary exploitation emerge. How do we ensure these agents respect privacy, and adhere to safety protocols?

2. **Multimodal**: With the evolution of AI systems to process and generate diverse modalities like audio, video, and images, concerns around content appropriateness, privacy, bias, and misinformation arise. How do we craft robust guidelines and security measures to tackle these challenges?

3. **Personalized Interactions**: As conversational agents evolve for social and personal interaction, risks like data privacy breaches and echo chambers grow. How do we balance tailored experiences with user safety?

4. **Sensitive Applications**: With AI’s integration into high-risk domains like legal, medical, and mental health, the stakes rise with risks such as overreliance on automation and potential catastrophic errors. How do we ensure that AI systems in these critical areas enhance decision-making without compromising human expertise and judgment?

5. **Dangerous Capabilities**: As AI's knowledge and understanding capabilities improve, these systems could be leveraged to extract or generate information about harmful applications or technologies, including bioweapons or cyber attack methods. How do we ensure that AI systems are designed with safeguards to prevent their misuse in creating or disseminating dangerous knowledge, while still allowing for beneficial research and innovation?
# Next Generation of Sequence Modeling Architectures Workshop at ICML 2024

## Description

This workshop will bring together various researchers to chart the course for the next generation of sequence modeling architectures. The focus will be on better understanding the limitations of existing models like transformers, recurrent neural networks, and state space models (e.g., S4, Mamba, LRU) and describing existing open problems. We will touch on topics such as memory, long-range context and in-context learning, optimization stability of these architectures, and their ability to represent different class problems. We will also cover interpretability and pragmatic aspects of making these models efficient and perform well: how they should be scaled up and the trade-offs and limitations imposed by current hardware. We will place additional emphasis on building both theoretical and also empirical understanding of the sequence models at scale; for example, this could be a better understanding of the scaling properties of these models concerning data, number of parameters, and amount of time the model spends at the inference. 

## Topics

We accept submissions on a diverse range of topics, including, but not limited to
- Memory: How to effectively discover or model long-range correlations? How to deal with long context? What types of memory behavior can these models exhibit?
- Theory: What are the limitations of current architectures? How can we understand the emerging properties of language models?
- Reasoning: Can we better understand and improve in-context learning and the chain of thought? Can current model reason or execute algorithms?
- Generalization: How does the sequence model generalize to different lengths and tasks? How robust are these models? What are different types of OOD generalization we should study, and how does generalization interact with memory or context? 
- Improving architectures: Some of the recent studies that would fall in this category are, for example, mixture of expert models such as Mixtral or hardware-aware architecture designs like FashAttention.
- Recurrent neural networks and state-space models: Some recent examples are Mamba, Griffin, Hawk, LRU, S4D, H3, etc.
- Scaling studies: Can we improve our understanding of scaling properties for different foundational models?
- Data-centric approaches to improve the performance of existing models such as data deduplication, diversification and curriculum.
- Downstream applications, such as language modeling, vision, biological data, and beyond.
# Workshop on Structured Probabilistic Inference & Generative Modeling

## Overview

The workshop focuses on theory, methodology, and application of structured probabilistic inference and generative modeling Probabilistic inference addresses the problem of amortization, sampling, and integration of complex quantities from graphical models, while generative modeling captures the underlying probability distributions of a dataset. Apart from applications in computer vision, natural language processing, and speech recognition, probabilistic inference and generative modeling approaches have also been widely used in natural science domains, including physics, chemistry, molecular biology, and medicine. Despite the promising results, probabilistic methods face challenges when applied to highly structured data, which are ubiquitous in real-world settings. We aim to bring experts from diverse backgrounds together, from both academia and industry, to discuss the applications and challenges of probabilistic methods, emphasizing challenges in encoding domain knowledge in these settings. We hope to provide a platform that fosters collaboration and discussion in the field of probabilistic methods. Topics include but are not limited to (see Call for Papers for more details):

* Inference and generative methods for graphs, time series, text, video, and other structured modalities
* Scaling and accelerating inference and generative models on structured data
* Uncertainty quantification in AI systems
* Applications in decision making, sampling, optimization, generative models, inference
* Applications and practical implementations of existing methods to areas in science
* Empirical analysis comparing different architectures for a given data modality and application
# Trustworthy Multi-modal Foundation Models and AI Agents (TiFA)

## Descriptions

Advanced Multi-modal Foundation Models (MFMs) and AI Agents, equipped with diverse modalities and an increasing number of available affordances (e.g., tool use, code interpreter, API access, etc.), have the potential to accelerate and amplify their predecessors’ impact on society.
MFM includes multi-modal large language models (MLLMs) and multi-modal generative models (MMGMs). MLLMs refer to LLM-based models with the ability to receive, reason, and output with information of multiple modalities, including but not limited to text, images, audio, and video. Examples include Llava, Reka, QwenVL, LAMM,and so on. MMGMs refer to a class of MFM models that can generate new content across multiple modalities, such as generating images from text descriptions or creating videos from audio and text inputs. Examples include Stable Diffusion , Sora, and Latte. AI agents, or systems with higher degree of agenticness, refer to systems that could achieve complex goals in complex environments with limited direct supervision. Understanding and preempting the vulnerabilities of these systems and their induced harms  becomes unprecedentedly crucial.
Building trustworthy MFMs and AI Agents transcends adversarial robustness of such models, but also emphasizes the importance of proactive risk assessment, mitigation, safeguards, and the establishment of comprehensive safety mechanisms throughout the lifecycle of the systems’ development and deployment. This approach demands a blend of technical and socio-technical strategies, incorporating AI governance and regulatory insights to build trustworthy MFMs and AI Agents.

## Topics

Topics include but are not limited to: 

- Adversarial attack and defense, poisoning, hijacking and security 
- Robustness to spurious correlations and uncertainty estimation
- Technical approaches to privacy, fairness, accountability and regulation
- Truthfulness, factuality, honesty and sycophancy 
- Transparency, interpretability and monitoring 
- Identifiers of AI-generated material, such as watermarking 
- Technical alignment / control , such as scalable overslight, representation control and machine unlearning
- Model auditing, red-teaming and safety evaluation benchmarks
- Measures against malicious model fine-tuning
- Novel safety challenges with the introduction of new modalities