chchenhui/mlrbench-tasks huggingface dataset:
##  Workshop on Regulatable ML

With the increasing deployment of machine learning in diverse applications affecting our daily lives, ethical and legal implications are rising to the forefront. Governments worldwide have responded by implementing regulatory policies to safeguard algorithmic decisions and data usage practices. However, there appears to be a considerable gap between current machine learning research and these regulatory policies. Translating these policies into algorithmic implementations is highly non-trivial, and there may be inherent tensions between different regulatory principles.

This workshop aims to provide a platform for: i) discussing various algorithmic, technical, and policy challenges that arise when operationalizing various guidelines outlined in existing regulatory frameworks, and ii) finding solutions to mitigate and address these challenges.


## Topics
With the widespread deployment of machine learning, there is a growing concern about the ethical and legal implications of these technologies. Governments worldwide have responded by implementing regulatory policies to safeguard algorithmic decisions and data usage practices. However, there is still a considerable gap between current machine learning research and these regulatory policies. Translating these policies into algorithmic implementations is highly non-trivial, and there may be inherent tensions between different regulatory principles.

The main focus of this workshop is to identify and bridge the gaps between ML research and regulatory principles. We encourage paper submissions relevant to (but not limited to) the following topics:

- Theoretical and/or empirical studies that highlight the operational gaps between existing regulations and SOTA ML research;
- Evaluation and auditing frameworks for ensuring that ML models comply with regulatory guidelines;
- Theoretical and/or empirical studies to highlight tensions between different desiderata (e.g., fairness, explainability, privacy) of ML models outlined by various regulatory frameworks;
- Novel algorithmic frameworks to operationalize the right to explanation, the right to privacy, the right to be forgotten, and to ensure fairness and robustness of ML models;
- Perspective/position papers that outline open problems and negative results relevant to ML regulation, or flawed research and development practices that misalign with regulatory policies;
- New regulation challenges posed by large generative models and methods to mitigate them, especially in the area of creative industries;
- Regulation needs for preventing catastrophic risks brought by artificial general intelligence (AGI).
## Safe Generative AI Workshop

In recent years, many AI researchers believe that advanced AI systems could potentially put human society at risk, especially if these systems become smarter than humans. Generative models have been the major driving force behind the development of advanced AI in the past two years. This workshop emphasizes AI safety concerns related to the use of generative models in basic machine learning research, scientific discoveries, and industrial/commercial applications. Generative models, including large language models, vision-language models, and diffusion models, have significantly aided various aspects of both academia and industry. In scientific discovery,these aspects encompass experimental design, hypothesis formulation, theoretical reasoning, and observation organization. In commercial applications, generative models such as large language models and diffusion algorithms have changed the lifestyles and workflows of billions around the world. However, these models have raised substantial concerns about potential misuse and negative scientific and social impacts.

## Topics

- Generation of harmful or biased content.
- Vulnerability to adversarial attacks.
- Privacy and security risks.
- Bias and fairness issues in generated content.
- Ethical implications of deploying generative AI.
- Limited robustness in out-of-distribution contexts.
- Overconfidence in the reliability of generated content.
## Workshop on Safe & Trustworthy Agents

This workshop aims to clarify key questions on the safety of agentic AI systems and foster a community of researchers working in this area.


## Topics
‍This workshop aims to clarify key questions on the trustworthiness of agentic AI systems and foster a community of researchers working in this area. We welcome papers on topics including, but not limited to, the following:

- Research into safe reasoning and memory. We are interested in work that makes LLM agent
reasoning or memory trustworthy, e.g., by preventing hallucinations or mitigating bias.
- Research into adversarial attacks, security and privacy for agents. As LLM agents interact
with more data modalities and a wider variety of input/output channels, we are interested in work
that studies or defends against possible threats and privacy leaks.
- Research into controlling agents. We are interested in novel control methods which specify goals,
constraints, and eliminate unintended consequences in LLM agents.
- Research into agent evaluation and accountability. We are interested in evaluation for LLM
agents (e.g., automated red-teaming) and interpretability + attributability of LLM agent actions.
- Research into environmental and societal impacts of agents. We are interested in research that
examines the environmental cost, fairness, social influence, and economic impacts of LLM agents.
- Research into multi-agent safety and security. We are interested in research that analyzes novel
phenomena with multiple agents: emergent functionality at a group level, collusion between agents, correlated failures, etc.
## Workshop on Scientific Methods for Understanding Deep Learning

While deep learning continues to achieve impressive results on an ever-growing range of tasks, our understanding of the principles underlying these successes remains largely limited. This problem is usually tackled from a mathematical point of view, aiming to prove rigorous theorems about optimization or generalization errors of standard algorithms, but so far they have been limited to overly-simplified settings. The main goal of this workshop is to promote a complementary approach that is centered on the use of the scientific method, which forms hypotheses and designs controlled experiments to test them. More specifically, it focuses on empirical analyses of deep networks that can validate or falsify existing theories and assumptions, or answer questions about the success or failure of these models. This approach has been largely underexplored, but has great potential to further our understanding of deep learning and to lead to significant progress in both theory and practice. The secondary goal of this workshop is to build a community of researchers, currently scattered in several subfields, around the common goal of understanding deep learning through a scientific lens.

## Topics
We invite researchers from machine learning and related fields to submit their latest work on the science of deep learning to the workshop. Accepted papers will be presented as posters during the poster sessions. Selected works will also be highlighted as contributed talks.

We encourage submissions that further our understanding of deep learning using the scientific method. Works that are a good fit for the workshop use empirical experiments on real-world datasets in order to:
- validate or falsify hypotheses about the inner workings of deep networks,
- make observations to inform or inspire theoretical models,
- evidence new phenomena or empirical regularities (e.g., scaling laws).

We invite studies that employ the scientific method of investigation in any field of application, including but not limited to:
- in-context learning in transformers,
- generalization properties of generative models,
- inductive biases of learning algorithms,
- (mechanistic) interpretability,
- empirical studies of loss landscapes, training dynamics, and learned weights and representations.

We explicitly welcome submissions that fall outside standard acceptance criteria, such as improving state-of-the-art performance or proving rigorous theorems, yet have a high impact potential by shedding light on deep network mechanisms.
## Statistical Foundations of LLMs and Foundation Models

Statistics has historically been the tool of choice for understanding and mitigating the operational risks of engineering deployments.

We need new statistical tools for the era of black-box models where the standard statistical ideas don't apply.

## Topics
Does your work intersect with any of the following topics as they relate to LLMs and foundation models?

- Benchmarks
- Measuring and correcting bias
- Automatic evaluation
- Watermarking
- Conformal prediction and other black-box uncertainty quantification techniques
- Privacy
- Auditing, safety, and risk analysis
## Workshop on Socially Responsible Language Modelling Research

The Socially Responsible Language Modelling Research (SoLaR) workshop at NeurIPS 2024 is an interdisciplinary gathering that aims to foster responsible and ethical research in the field of language modeling. Recognizing the significant risks and harms associated with the development, deployment, and use of language models, the workshop emphasizes the need for researchers to focus on addressing these risks starting from the early stages of development. The workshop brings together experts and practitioners from various domains and academic fields with a shared commitment to promoting fairness, equity, accountability, transparency, and safety in language modeling research.

## Topics
Given the wide-ranging impacts of LMs, our workshop will welcome a broad array of submissions. We briefly detail some specific topic areas and an illustrative selection of pertinent works:

- Security and privacy concerns of LMs [13, 30, 25, 49, 55].
- Bias and exclusion in LMs [12, 2, 26, 53, 44].
- Analysis of the development and deployment of LMs, including crowdwork [42, 50], deploy- ment protocols [52, 47], and societal impacts from deployment [10, 21].
- Safety, robustness, and alignment of LMs [51, 8, 35, 32, 7].
- Auditing, red-teaming, and evaluations of LMs [41, 40, 29, 15, 11].
- Examination of risks and harms from any novel input and/or output modalities that are introduced in LMs [14, 28, 54].
- Transparency, explainability, interpretability of LMs [39, 17, 3, 46, 22, 38].
- Applications of LMs for social good, including sector-specific applications [9, 31, 16] and LMs for low-resource languages [4, 5, 36].
- Perspectives from other domains that can inform socially responsible LM development and deployment [48, 1].
## Self-Supervised Learning - Theory and Practice

Self-supervised learning (SSL) is an approach of representation learning that does not rely on human-labeled data. Instead, it creates auxiliary tasks from unlabeled input data and learns representations by solving these tasks. SSL has shown significant success across various domains such as images (e.g., MAE, DINO, MoCo, PIRL, SimCLR), speech (e.g., wav2vec, Whisper), and text (e.g., BERT, GPT, Llama). It has also demonstrated promising results in other data modalities including graphs, time-series, and audio. Recent large language models—predominantly trained on web-scale data using self-supervised methods—have exhibited remarkable generalizability and are beginning to transform numerous research fields. SSL, without using human-provided labels, can achieve performance comparable to or even surpassing that of fully supervised methods. Furthermore, generative SSL techniques such as Imagen, Stable Diffusion, and SORA have significantly enhanced the artistic capabilities of AI models.

Existing research on self-supervised learning (SSL) has primarily concentrated on enhancing empirical performance without substantial theoretical underpinnings. Although SSL approaches are empirically effective across various benchmarks, their theoretical foundations and practical applications remain less explored. Key questions such as the reasons behind the superior performance of certain auxiliary tasks, the requisite amount of unlabeled data for learning effective representations, the impact of neural architectures on SSL performance, and the practical scenarios where SSL outperforms supervised models, are still largely unanswered.

In the 5th iteration of this workshop, we aim to address these gaps by fostering a dialogue between theory and practice, especially in the context of LLMs. We bring together SSL-interested researchers from various domains to discuss the theoretical foundations of empirically well-performing SSL approaches and how the theoretical insights can further improve SSL’s empirical performance.

## Topics
We invite submissions of both theoretical works and empirical works, and the intersection of the two. The topics include but are not limited to:

- Theoretical foundations of SSL
- SSL for computer vision, natural language processing, robotics, speech processing, time-series analysis, graph analytics, etc.
- Sample complexity of SSL methods
- Theory-driven design of auxiliary tasks in SSL
- Comparative analysis of different auxiliary tasks
- Comparative analysis of SSL and supervised approaches
- Information theory and SSL
- SSL for healthcare, social media, neuroscience, biology, social science, etc.
- Cognitive foundations of SSL
## Workshop on System-2 Reasoning at Scale

System 2 Reasoning At Scale focuses on improving reasoning in neural networks, particularly the challenges and strategies for achieving System-2 reasoning in transformer-like models. The workshop addresses issues like distinguishing memorization from rule-based learning, understanding, syntactic generalization, and compositionality. The workshop also covers the importance of understanding how systematic models are in their decisions for AI safety, integrating neural networks with symbolic reasoning, and developing new architectures for enhanced reasoning capabilities.

## Topics
The authors are welcome to submit paper that aims to answer the following questions:


- What do we need to imbue language models with System-2 reasoning capabilities?
- Do we need this kind of capability?
- Are scale and the “bitter lesson” going to dictate how the future of AI technology will unfold?
- Do we need a different mechanism for implementing System-2 reasoning, or should it be a property that emerges from a possibly different training method?
- Where should a system like this be implemented? Implicitly inside the model, or explicitly in some engineered system around the model, like search or graph of thought?
- How do we benchmark System-2-like generalization? How do we avoid data contamination?
# Workshop on Unifying Representations in Neural Models


### When, how and why do different neural models learn the same representations?
New findings in neuroscience and artificial intelligence reveal a shared pattern: whether in biological brains or artificial models, different learning systems tend to create similar representations when subject to similar stimuli.

The emergence of these similar representations is igniting a growing interest in the fields of neuroscience and artificial intelligence, with both fields offering promising directions for their theoretical understanding. These include analyzing the learning dynamics in neuroscience and studying the problem of identifiability in the functional and parameter space in artificial intelligence.

While the theoretical aspects already demand investigation, the practical applications are equally compelling: aligning representations allows for model merging, stitching and reuse, while also playing a crucial role in multi-modal scenarios. Furthermore, studying the features that are universally highlighted by different learning processes brings us closer to pinpoint the invariances that naturally emerge from learning models, possibly suggesting ways to enforce them.

The objective of the workshop is to discuss theoretical findings, empirical evidence and practical applications of this phenomenon, benefiting from the cross-pollination of different fields (ML, Neuroscience, Cognitive Science) to foster the exchange of ideas and encourage collaborations.

In conclusion, our primary focus is to delve into the underlying reasons, mechanisms, and extent of similarity in internal representations across distinct neural models, with the ultimate goal of unifying them into a single cohesive whole.
## Workshop on Touch Processing: From Data to Knowledge
The growing relevance of video-language models in both academia and industry highlights the necessity for a dedicated workshop to address the unique challenges and opportunities this field presents. This workshop is designed to accelerate the development and practical application of video foundation models, which are crucial for interpreting and utilizing the extensive amounts of video data that make up a significant portion of global data. These models are increasingly vital for a range of applications, from video search and content creation to surveillance and robotics.


## Topics
- First, the scarcity of high-quality, annotated video data is a significant barrier to progress. Unlike text and image data, which are abundant and often come with high-quality annotations, video data typically lacks such detailed annotations, limiting the development of advanced models.
- Second, the sheer volume of video data demands significant advancements in data processing techniques.. Modern video models must process hundreds to thousands of frames per video, with each frame requiring detailed analysis. Efficient processing methods are needed to handle this scale while maintaining detailed information capture.
- Third, the multimodal nature of video data requires sophisticated model designs that can integrate audio, visual, temporal, and textual data in a cohesive manner.
- Last but not least, the community still lacks robust video-language alignment benchmarks, which makes it hard to evaluate and compare the capabilities of video-language models.