AIS-Bench/AIS-Bench huggingface dataset:
The Logical Implication Steering Method for Conditional Interventions on
Transformer Generation
Damjan Kalajdzievski 1
Abstract
The field of mechanistic interpretability in pre-
trained transformer models has demonstrated sub-
stantial evidence supporting the “linear represen-
tation hypothesis”, which is the idea that high
level concepts are encoded as vectors in the space
of activations of a model. Studies also show that
model generation behavior can be steered toward
a given concept by adding the concept’s vector
to the corresponding activations. We show how
to leverage these properties to build a form of
logical implication into models, enabling trans-
parent and interpretable adjustments that induce
a chosen generation behavior in response to the
presence of any given concept. Our method, Log-
ical Implication Model Steering (LIMS), unlocks
new hand-engineered reasoning capabilities by
integrating neuro-symbolic logic into pre-trained
transformer models.
1. Introduction
In this work, we are motivated by advances in mechanis-
tic interpretability which deepen our understanding of the
representations within pre-trained transformer models, as
well as by the goal of bridging neuro-symbolic and algo-
rithmic capabilities with these representations. Harnessing
the extensive knowledge and intuitive processing embedded
in these models in a high-level, interpretable, and algorith-
mic way, holds the potential to unlock significantly greater
reasoning capabilities and control.
Driven by these motivations, we introduce the Logical Impli-
cation Model Steering (LIMS) method, which extracts and
uses concept vectors to allow a user to “program” logical
implication into model generation behavior in a transparent
and interpretable manner. LIMS achieves this by implement-
ing the atomic building blocks for a general neuro-symbolic
1Salesforce. Correspondence to: Damjan Kalajdzievski <dam-
jank7354@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning...
A-Mem: Agentic Memory for LLM Agents
Wujiang Xu1, Zujie Liang2, Kai Mei1, Hang Gao1, Juntao Tan1, Yongfeng Zhang1,3
1Rutgers University
2Independent Researcher
3AIOS Foundation
wujiang.xu@rutgers.edu
Abstract
While large language model (LLM) agents can effectively use external tools for
complex real-world tasks, they require memory systems to leverage historical
experiences. Current memory systems enable basic storage and retrieval but lack
sophisticated memory organization, despite recent attempts to incorporate graph
databases. Moreover, these systems’ fixed operations and structures limit their
adaptability across diverse tasks. To address this limitation, this paper proposes
a novel agentic memory system for LLM agents that can dynamically organize
memories in an agentic way. Following the basic principles of the Zettelkasten
method, we designed our memory system to create interconnected knowledge
networks through dynamic indexing and linking. When a new memory is added, we
generate a comprehensive note containing multiple structured attributes, including
contextual descriptions, keywords, and tags. The system then analyzes historical
memories to identify relevant connections, establishing links where meaningful
similarities exist. Additionally, this process enables memory evolution – as new
memories are integrated, they can trigger updates to the contextual representations
and attributes of existing historical memories, allowing the memory network to
continuously refine its understanding. Our approach combines the structured
organization principles of Zettelkasten with the flexibility of agent-driven decision
making, allowing for more adaptive and context-aware memory management.
Empirical experiments on six foundation models show superior improvement
against existing SOTA baselines.
Code for Benchmark Evaluation:
https://github.com/WujiangXu/AgenticMemory
Code for Production-ready Agentic Memory:
https://github.com/WujiangXu/A-mem-sys
1
Introduction
Large Lang...
Published as a conference paper at ICLR 2025
BREAKING MENTAL SET TO IMPROVE REASONING
THROUGH DIVERSE MULTI-AGENT DEBATE
Yexiang Liu1,2, Jie Cao1∗, Zekun Li3, Ran He1,2, Tieniu Tan1,2,4
1MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences
2School of Artificial Intelligence, University of Chinese Academy of Sciences
3University of California, Santa Barbara
4Nanjing University
liuyexiang2023@ia.ac.cn, jie.cao@cripac.ia.ac.cn
ABSTRACT
Large Language Models (LLMs) have seen significant progress but continue to
struggle with persistent reasoning mistakes. Previous methods of self-reflection
have been proven limited due to the models’ inherent fixed thinking patterns.
While Multi-Agent Debate (MAD) attempts to mitigate this by incorporating mul-
tiple agents, it often employs the same reasoning methods, even though assigning
different personas to models. This leads to a “fixed mental set”, where models rely
on homogeneous thought processes without exploring alternative perspectives. In
this paper, we introduce Diverse Multi-Agent Debate (DMAD), a method that en-
courages agents to think with distinct reasoning approaches. By leveraging diverse
problem-solving strategies, each agent can gain insights from different perspec-
tives, refining its responses through discussion and collectively arriving at the op-
timal solution. DMAD effectively breaks the limitations of fixed mental sets. We
evaluate DMAD against various prompting techniques, including self-reflection
and traditional MAD, across multiple benchmarks using both LLMs and Multi-
modal LLMs. Our experiments show that DMAD consistently outperforms other
methods, delivering better results than MAD in fewer rounds. Code is available at
https://github.com/MraDonkey/DMAD.
1
INTRODUCTION
In the past few years, Large Language Models (LLMs) have experienced unprecedented develop-
ment (Brown et al., 2020; Touvron et al., 2023a;b; Chowdhery et al., 2023; Anil et al., 2023; Achiam
et al., 2023; Open AI, 2024a),...
Riemann Tensor Neural Networks: Learning Conservative Systems with
Physics-Constrained Networks
Anas Jnini 1 Lorenzo Breschi 1 Flavio Vella 1
Abstract
Divergence-free symmetric tensors (DFSTs) are
fundamental in continuum mechanics, encoding
conservation laws such as mass and momentum
conservation. We introduce Riemann Tensor Neu-
ral Networks (RTNNs), a novel neural architec-
ture that inherently satisfies the DFST condition
to machine precision, providing a strong induc-
tive bias for enforcing these conservation laws.
We prove that RTNNs can approximate any suffi-
ciently smooth DFST with arbitrary precision and
demonstrate their effectiveness as surrogates for
conservative PDEs, achieving improved accuracy
across benchmarks. This work is the first to use
DFSTs as an inductive bias in neural PDE surro-
gates and to explicitly enforce the conservation
of both mass and momentum within a physics-
constrained neural architecture.
1. Introduction
Partial Differential Equations (PDEs)
Partial Differen-
tial Equations (PDEs) are central to the mathematical model-
ing of complex physical systems, including fluid dynamics,
thermodynamics, and material sciences. Traditional nu-
merical methods, such as finite element and spectral meth-
ods, often require fine discretization of the physical domain
to achieve high accuracy. These approaches can become
computationally expensive, particularly in engineering ap-
plications where systems must be solved repeatedly under
varying parameters or initial conditions. Recent advances in
machine learning (ML) have shown promise in addressing
these challenges by leveraging neural networks (NNs) as po-
tential alternatives or enhancements to traditional numerical
solvers (Kovachki et al., 2021; Li et al., 2020).
1 Department of Information Engineering and Computer Sci-
ence, University of Trento, Trento, Italy. Correspondence to: Anas
Jnini <anas.jnini@unitn.it>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancou...
Mahalanobis++: Improving OOD Detection via Feature Normalization
Maximilian M¨uller 1 Matthias Hein 1
Abstract
Detecting out-of-distribution (OOD) examples
is an important task for deploying reliable ma-
chine learning models in safety-critial applica-
tions. While post-hoc methods based on the Ma-
halanobis distance applied to pre-logit features
are among the most effective for ImageNet-scale
OOD detection, their performance varies signif-
icantly across models. We connect this incon-
sistency to strong variations in feature norms,
indicating severe violations of the Gaussian as-
sumption underlying the Mahalanobis distance es-
timation. We show that simple ℓ2-normalization
of the features mitigates this problem effectively,
aligning better with the premise of normally dis-
tributed data with shared covariance matrix. Ex-
tensive experiments on 44 models across diverse
architectures and pretraining schemes show that
ℓ2-normalization improves the conventional Ma-
halanobis distance-based approaches significantly
and consistently, and outperforms other recently
proposed OOD detection methods. Code is avail-
able at github.com/mueller-mp/maha-norm.
1. Introduction
Deep neural networks have demonstrated remarkable perfor-
mance across a variety of real-world tasks. However, when
faced with inputs that fall outside their training distribu-
tion, they can behave unpredictably and even result in high-
confidence predictions (Hendrycks & Gimpel, 2017; Hein
et al., 2019). These so-called out-of-distribution (OOD) in-
puts are often misclassified with high confidence as belong-
ing to the in-distribution (ID) classes, creating significant
risks for real-world deployments. OOD detectors aim to
identify and reject such anomalous inputs — potentially
prompting human intervention, transitioning to a safe state,
or declining to provide a prediction — while still allowing
1University of T¨ubingen and T¨ubingen AI Center.
Corre-
spondence to: Maximilian M¨uller <maximilian.mueller@...
From Pixels to Perception:
Interpretable Predictions via Instance-wise Grouped Feature Selection
Moritz Vandenhirtz 1 Julia E. Vogt 1
Abstract
Understanding the decision-making process of
machine learning models provides valuable in-
sights into the task, the data, and the reasons be-
hind a model’s failures. In this work, we propose
a method that performs inherently interpretable
predictions through the instance-wise sparsifica-
tion of input images. To align the sparsification
with human perception, we learn the masking in
the space of semantically meaningful pixel re-
gions rather than on pixel-level. Additionally, we
introduce an explicit way to dynamically deter-
mine the required level of sparsity for each in-
stance. We show empirically on semi-synthetic
and natural image datasets that our inherently in-
terpretable classifier produces more meaningful,
human-understandable predictions than state-of-
the-art benchmarks.
1. Introduction
Knowledge is power (Bacon, 1597). Knowing how ma-
chine learning models make their predictions empowers
users to understand the underlying patterns in the data, iden-
tify potential biases, reason about their safety, and build
trust in the models’ decisions. In high-stakes domains,
where model decisions can have significant consequences,
the field of interpretability is crucial to provide this under-
standing (Doshi-Velez & Kim, 2017). In this work, we focus
on providing inherent interpretability – a subfield where the
explanations are necessarily faithful to what the model com-
putes (Rudin, 2019) – through the lens of sparsity in the
input features (Lipton, 2016; Marcinkeviˇcs & Vogt, 2023).
This constraint makes it easier to understand which specific
inputs contribute to an output (Miller, 1956), leading to
more human-understandable predictions.
1Department
of
Computer
Science,
ETH
Zurich,
Switzerland.
Correspondence
to:
Moritz
Vandenhirtz
<moritz.vandenhirtz@inf.ethz.ch>.
Proceedings of the 42 nd International Conference on ...
HYGMA: Hypergraph Coordination Networks with Dynamic Grouping
for Multi-Agent Reinforcement Learning
Chiqiang Liu 1 Dazi Li 1
Abstract
Cooperative multi-agent reinforcement learning
faces significant challenges in effectively orga-
nizing agent relationships and facilitating infor-
mation exchange, particularly when agents need
to adapt their coordination patterns dynamically.
This paper presents a novel framework that in-
tegrates dynamic spectral clustering with hyper-
graph neural networks to enable adaptive group
formation and efficient information processing
in multi-agent systems.
The proposed frame-
work dynamically constructs and updates hyper-
graph structures through spectral clustering on
agents’ state histories, enabling higher-order rela-
tionships to emerge naturally from agent interac-
tions. The hypergraph structure is enhanced with
attention mechanisms for selective information
processing, providing an expressive and efficient
way to model complex agent relationships. This
architecture can be implemented in both value-
based and policy-based paradigms through a uni-
fied objective combining task performance with
structural regularization. Extensive experiments
on challenging cooperative tasks demonstrate
that our method significantly outperforms state-
of-the-art approaches in both sample efficiency
and final performance. The code is available at:
https://github.com/mysteryelder/HYGMA.
1. Introduction
Cooperative multi-agent reinforcement learning (MARL)
has emerged as a promising paradigm for addressing com-
plex real-world challenges that require coordinated decision-
making among multiple agents (Charbonnier et al., 2025;
Si et al., 2025). A fundamental challenge in MARL is the
1College of Information Science and Technology, Beijing Uni-
versity of Chemical Technology, Beijing, China. Correspondence
to: Dazi Li <lidz@mail.buct.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyri...
Adaptive Elicitation of Latent Information Using Natural Language
Jimmy Wang * 1 Thomas Zollo * 1 Richard Zemel 1 Hongseok Namkoong 1
Abstract
Eliciting information to reduce uncertainty about
a latent entity is a critical task in many appli-
cation domains, e.g., assessing individual stu-
dent learning outcomes, diagnosing underlying
diseases, or learning user preferences. Though
natural language is a powerful medium for this
purpose, large language models (LLMs) and ex-
isting fine-tuning algorithms lack mechanisms for
strategically gathering information to refine their
own understanding of the latent entity. To har-
ness the generalization power and world knowl-
edge of LLMs in developing effective informa-
tion gathering strategies, we propose an adaptive
elicitation framework that actively reduces un-
certainty on the latent entity. Since probabilistic
modeling of an abstract latent entity is difficult,
our framework adopts a predictive view of un-
certainty, using a meta-learned language model
to simulate future observations and enable scal-
able uncertainty quantification over complex nat-
ural language. Through autoregressive forward
simulation, our model quantifies how new ques-
tions reduce epistemic uncertainty, enabling the
development of sophisticated information gather-
ing strategies to choose the most informative next
queries. In experiments on the twenty questions
game, dynamic opinion polling, and adaptive stu-
dent assessment, our method consistently outper-
forms baselines in identifying critical unknowns
and improving downstream predictions, illustrat-
ing the promise of strategic information gathering
in natural language settings.
1. Introduction
The performance of many valuable services and systems
depends on the ability to efficiently elicit information and
1Columbia University.
Correspondence to:
Jimmy
Wang
<jw4209@columbia.edu>,
Thomas
Zollo
<tpz2105@columbia.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vanco...
CROW: Eliminating Backdoors from Large Language Models via Internal
Consistency Regularization
Nay Myat Min 1 Long H. Pham 1 Yige Li 1 Jun Sun 1
Abstract
Large Language Models (LLMs) are vulnera-
ble to backdoor attacks that manipulate out-
puts via hidden triggers.
Existing defense
methods—designed for vision/text classification
tasks—fail for text generation. We propose In-
ternal Consistency Regularization (CROW), a de-
fense leveraging the observation that backdoored
models exhibit unstable layer-wise hidden rep-
resentations when triggered, while clean models
show smooth transitions. CROW enforces consis-
tency across layers via adversarial perturbations
and regularization during finetuning, neutraliz-
ing backdoors without requiring clean reference
models or trigger knowledge—only a small clean
dataset. Experiments across Llama-2 (7B, 13B),
CodeLlama (7B, 13B), and Mistral-7B demon-
strate CROW’s effectiveness: it achieves signifi-
cant reductions in attack success rates across di-
verse backdoor strategies (sentiment steering, tar-
geted refusal, code injection) while preserving
generative performance. CROW’s architecture-
agnostic design enables practical deployment.
1. Introduction
Large Language Models (LLMs) have revolutionized natu-
ral language processing but face critical security risks from
backdoor attacks—hidden triggers that manipulate model
outputs during inference. While extensively studied in vi-
sion/text classification (Gu et al., 2019; Chen et al., 2021b),
these attacks pose unique challenges for generative LLMs
due to their complex output spaces and context sensitivity.
Recent work shows that attackers can implant backdoors via
data poisoning (Li et al., 2025) or prompt engineering (Yan
et al., 2024), enabling dangerous behaviors like toxic con-
1School of Computing and Information Systems, Singapore
Management University, Singapore. Correspondence to: Yige Li
<yigeli@smu.edu.sg>.
Proceedings of the 42 nd International Conference on Machine
...
Active Learning for Efficient Discovery of Optimal Combinatorial Perturbations
Jason Qin 1 Hans-Hermann Wessels 1 Carlos Fernandez-Granda 2 3 Yuhan Hao 1
Abstract
Combinatorial CRISPR screening enables large-
scale identification of synergistic gene pairs for
combination therapies, but exhaustive experimen-
tation is infeasible. We introduce NAIAD1, an ac-
tive learning framework that efficiently discovers
optimal gene pairs by leveraging single-gene per-
turbation effects and adaptive gene embeddings
that scale with the training data size, mitigating
overfitting in small-sample learning while cap-
turing complex gene interactions as more data is
collected. Evaluated on four CRISPR datasets
with over 350,000 interactions, NAIAD trained
on small datasets outperforms existing models by
up to 40%. Its recommendation system priori-
tizes gene pairs with maximum predicted effects,
accelerating discovery with fewer experiments.
We also extend NAIAD to optimal drug com-
bination identification among 2,000 candidates.
Overall, NAIAD enhances combinatorial pertur-
bation design and drives advances in genomics
research and therapeutic development in combi-
nation therapy. Our code is publicly available at:
https://github.com/NeptuneBio/NAIAD
1. Introduction
Targeting multiple genes through drug combinations or
polypharmacology offers a transformative therapeutic ap-
proach for developing effective treatments across diverse
medical fields, including oncology (Al-Lazikani et al., 2012;
Mokhtari et al., 2017), infectious diseases (Hammond et al.,
2022; Shyr et al., 2021), and metabolic disorders (Samms
et al., 2020; Jastreboff et al., 2023). Combinatorial gene per-
turbations can yield additive or synergistic effects, enhanc-
ing therapeutic outcomes beyond what is achievable with
1Neptune Bio, New York, NY, USA 2Center for Data Science,
New York University, New York, NY, USA 3Courant Institute
of Mathematical Sciences, New York University, New York, NY,
USA . Correspondence to:...
Reading Recognition in the Wild
Charig Yang1,2, Samiul Alam3, Shakhrul Iman Siam3, Michael J. Proulx1, Lambert Mathias1,
Kiran Somasundaram1, Luis Pesqueira1, James Fort1, Sheroze Sheriffdeen1, Omkar Parkhi1,
Carl Ren1, Mi Zhang3, Yuning Chai1, Richard Newcombe1, Hyo Jin Kim1
1Meta Reality Labs Research
2VGG, University of Oxford
3The Ohio State University
charig@robots.ox.ac.uk, kimhyojin@meta.com
https://www.projectaria.com/datasets/reading-in-the-wild/
Abstract
To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able
to keep a record of the user’s interactions with the world, including during reading.
In this paper, we introduce a new task of reading recognition to determine when
the user is reading. We first introduce the first-of-its-kind large-scale multimodal
Reading in the Wild dataset, containing 100 hours of reading and non-reading videos
in diverse and realistic scenarios. We then identify three modalities (egocentric
RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible
transformer model that performs the task using these modalities, either individually
or combined. We show that these modalities are relevant and complementary to
the task, and investigate how to efficiently and effectively encode each modality.
Additionally, we show the usefulness of this dataset towards classifying types of
reading, extending current reading understanding studies conducted in constrained
settings to larger scale, diversity and realism. Code, model, and data will be public.
Figure 1: Am I reading? The left figure shows a timeline as the user navigates the world. We aim to solve the
task of reading recognition to enable AI assistants in always-on wearables. We identify three modalities: eye
gaze (in colored dot patterns), RGB crop around gaze (in red box), and inertial sensors performs the task to high
accuracy (with Prediction and GT shown). Images from our Reading in the Wild dataset, which features 100
hours ...
The Brain’s Bitter Lesson:
Scaling Speech Decoding With Self-Supervised Learning
Dulhan Jayalath 1 Gilad Landau 1 Brendan Shillingford 2 Mark Woolrich3 Oiwi Parker Jones 1
Abstract
The past few years have seen remarkable progress
in the decoding of speech from brain activity,
primarily driven by large single-subject datasets.
However, due to individual variation, such as
anatomy, and differences in task design and scan-
ning hardware, leveraging data across subjects
and datasets remains challenging. In turn, the
field has not benefited from the growing number
of open neural data repositories to exploit large-
scale deep learning. To address this, we develop
neuroscience-informed self-supervised objectives,
together with an architecture, for learning from
heterogeneous brain recordings. Scaling to nearly
400 hours of MEG data and 900 subjects, our ap-
proach shows generalisation across participants,
datasets, tasks, and even to novel subjects. It
achieves improvements of 15-27% over state-of-
the-art models and matches surgical decoding
performance with non-invasive data. These ad-
vances unlock the potential for scaling speech
decoding models beyond the current frontier.
1. Introduction
In his Bitter Lesson, Richard Sutton argues that a major
conclusion of 70 years of AI research is that general methods
exploiting large-scale computation will outperform model-
based approaches as the availability of compute increases
(Sutton, 2019). The ability of deep learning to learn from
ever-larger datasets has enabled seemingly arbitrary scaling
with computation, leading to astounding advances across
a diverse set of domains (Jumper et al., 2021; Caron et al.,
2021; OpenAI, 2023; Radford et al., 2023).
In the domain of brain data, and of tasks like speech de-
coding, the bitter lesson has not yet been fully assimilated.
1PNPL
/ 3OHBA, University of Oxford 2Google DeepMind.
Correspondence to: <{dulhan, oiwi}@robots.ox.ac.uk>.
Proceedings of the 42 nd International Conference on...