AIS-Bench/AIS-Bench huggingface dataset:
Published as a conference paper at ICLR 2025
LEAN-STAR:
LEARNING TO INTERLEAVE THINKING AND PROVING
Haohan Lin2∗
Zhiqing Sun1
Sean Welleck1
Yiming Yang1
1Language Technologies Institute, Carnegie Mellon University
2Institute for Interdisciplinary Information Sciences, Tsinghua University
{haohanl,zhiqings,swelleck,yiming}@cs.cmu.edu
https://leanstar.github.io/
ABSTRACT
Traditional language model-based theorem proving assumes that by training on a
sufficient amount of formal proof data, a model will learn to prove theorems. Our
key observation is that a wealth of informal information that is not present in formal
proofs can be useful for learning to prove theorems. For instance, humans think
through steps of a proof, but this thought process is not visible in the resulting code.
We present Lean-STaR, a framework for training language models to produce
informal thoughts prior to each step of a proof, thereby boosting the model’s
theorem-proving capabilities. Lean-STaR uses retrospective ground-truth tactics to
generate synthetic thoughts for training the language model. At inference time, the
trained model directly generates the thoughts prior to the prediction of the tactics
in each proof step. Building on the self-taught reasoner framework, we then apply
expert iteration to further fine-tune the model on the correct proofs it samples and
verifies using the Lean solver. Lean-STaR significantly outperforming base models
(43.4% →46.3%, Pass@64). We also analyze the impact of the augmented
thoughts on various aspects of the theorem proving process, providing insights into
their effectiveness.
1
INTRODUCTION
Theorem proving is a fundamental aspect of mathematics, and mathematical reasoning is an important
part of artificial intelligence (Newell & Simon, 1956; Zhou, 2023). Formalized mathematics in
particular provides a challenging testbed for assessing mathematical reasoning capabilities. Since
theorems and proofs in this setting can be represented in the form of checkab...
Leveraging Online Olympiad-Level Math Problems for LLMs
Training and Contamination-Resistant Evaluation
Sadegh Mahdavi * 1 Muchen Li * 1 Kaiwen Liu 1 Christos Thrampoulidis 1 Leonid Sigal 1 2 3 4 Renjie Liao 1 2 3
Abstract
Advances in Large Language Models (LLMs)
have sparked interest in their ability to solve
Olympiad-level math problems. However, the
training and evaluation of these models are con-
strained by the limited size and quality of avail-
able datasets, as creating large-scale data for
such advanced problems requires extensive ef-
fort from human experts. In addition, current
benchmarks are prone to contamination, lead-
ing to unreliable evaluations. In this paper, we
present an automated pipeline that leverages the
rich resources of the Art of Problem Solving
(AoPS) forum, which predominantly features
Olympiad-level problems and community-driven
solutions.
Using open-source LLMs, we de-
velop a method to extract question-answer pairs
from the forum, resulting in AoPS-Instruct, a
dataset of more than 600,000 high-quality QA
pairs.
Our experiments demonstrate that fine-
tuning LLMs on AoPS-Instruct improves their
reasoning abilities across various benchmarks.
Moreover, we build an automatic pipeline that
introduces LiveAoPSBench, an evolving evalua-
tion set with timestamps, derived from the latest
forum data, providing a contamination-resistant
benchmark for assessing LLM performance. No-
tably, we observe a significant decline in LLM
performance over time, suggesting their success
on older examples may stem from pre-training
exposure rather than true reasoning ability. Our
work presents a scalable approach to creating
and maintaining large-scale, high-quality datasets
for advanced math reasoning, offering valuable
insights into the capabilities and limitations of
* Equal Contribution, order decided by a coin flip 1University
of British Columbia, Vancouver, Canada 2Vector Institute for AI,
Toronto, Canada 3Canada CIFAR AI Chair 4NSERC CRC Chair.
Correspo...
Latent Action Learning Requires Supervision in the Presence of Distractors
Alexander Nikulin 1 2 Ilya Zisman 1 3 4 Denis Tarasov 1 Nikita Lyubaykin 1 5 Andrei Polubarov 1 3 4
Igor Kiselev 6 Vladislav Kurenkov 1 5
Abstract
Recently, latent action learning, pioneered by
Latent Action Policies (LAPO), have shown re-
markable pre-training efficiency on observation-
only data, offering potential for leveraging vast
amounts of video available on the web for em-
bodied AI. However, prior work has focused on
distractor-free data, where changes between obser-
vations are primarily explained by ground-truth
actions. Unfortunately, real-world videos contain
action-correlated distractors that may hinder la-
tent action learning. Using Distracting Control
Suite (DCS) we empirically investigate the ef-
fect of distractors on latent action learning and
demonstrate that LAPO struggle in such scenario.
We propose LAOM, a simple LAPO modification
that improves the quality of latent actions by 8x,
as measured by linear probing. Importantly, we
show that providing supervision with ground-truth
actions, as few as 2.5% of the full dataset, dur-
ing latent action learning improves downstream
performance by 4.2x on average. Our findings
suggest that integrating supervision during Latent
Action Models (LAM) training is critical in the
presence of distractors, challenging the conven-
tional pipeline of first learning LAM and only
then decoding from latent to ground-truth actions.
1. Introduction
Recently, a new wave of approaches based on latent action
learning has emerged (Edwards et al., 2019), demonstrating
superior pre-training efficiency on datasets without action
labels in large-scale robotics (Ye et al., 2024; Chen et al.,
2024b;a; Cui et al., 2024; Bruce et al., 2024) and reinforce-
ment learning (Schmidt & Jiang, 2023). Latent Action Mod-
1AIRI 2MIPT 3Skoltech 4Research Center for Trusted Artificial
Intelligence, ISP RAS 5Innopolis University 6Accenture. Corre-
spondence to: Alexand...
Vintix: Action Model via In-Context Reinforcement Learning
Andrei Polubarov * 1 2 3 Nikita Lyubaykin * 1 4 Alexander Derevyagin * 1 5 Ilya Zisman 1 2 3 Denis Tarasov 1
Alexander Nikulin 1 6 Vladislav Kurenkov 1 4
Abstract
In-Context Reinforcement Learning (ICRL) rep-
resents a promising paradigm for developing gen-
eralist agents that learn at inference time through
trial-and-error interactions, analogous to how
large language models adapt contextually, but
with a focus on reward maximization. However,
the scalability of ICRL beyond toy tasks and
single-domain settings remains an open challenge.
In this work, we present the first steps toward scal-
ing ICRL by introducing a fixed, cross-domain
model capable of learning behaviors through
in-context reinforcement learning. Our results
demonstrate that Algorithm Distillation, a frame-
work designed to facilitate ICRL, offers a com-
pelling and competitive alternative to expert distil-
lation to construct versatile action models. These
findings highlight the potential of ICRL as a scal-
able approach for generalist decision-making sys-
tems. Code to be released at dunnolab/vintix.
1. Introduction
The pursuit of generalist control and decision-making agents
has long been a major target of the reinforcement learning
(RL) research community (Sutton & Barto, 1998). These
agents are envisioned to handle a diverse set of tasks while
exhibiting adaptation properties such as self-correction and
self-improvement based on reward functions. Notably, tradi-
tional online RL algorithms demonstrate these properties in
narrow domains and can achieve exceptional performance
(Berner et al., 2019; Badia et al., 2020; Schrittwieser et al.,
2020; Baker et al., 2022; Team et al., 2023). However,
their online nature of learning and frequent reliance on
environment-specific training is still considered a challenge
to overcome limiting the scalability (Dulac-Arnold et al.,
*Equal contribution
1AIRI 2Skoltech 3Research Center for
Trusted Artifi...
Published as a conference paper at ICLR 2025
MONET: MIXTURE OF MONOSEMANTIC EXPERTS FOR
TRANSFORMERS
Jungwoo Park1,3†, Young Jin Ahn2†, Kee-Eung Kim2⋆, Jaewoo Kang1,3⋆
1Korea University, 2KAIST, 3AIGEN Sciences
{jungwoo-park, kangj}@korea.ac.kr
{snoop2head, kekim}@kaist.ac.kr
ABSTRACT
Understanding the internal computations of large language models (LLMs) is
crucial for aligning them with human values and preventing undesirable behav-
iors like toxic content generation. However, mechanistic interpretability is hin-
dered by polysemanticity—where individual neurons respond to multiple, unre-
lated concepts.
While Sparse Autoencoders (SAEs) have attempted to disen-
tangle these features through sparse dictionary learning, they have compromised
LLM performance due to reliance on post-hoc reconstruction loss. To address
this issue, we introduce MIXTURE OF MONOSEMANTIC EXPERTS FOR TRANS-
FORMERS (MONET) architecture, which incorporates sparse dictionary learning
directly into end-to-end Mixture-of-Experts pretraining. Our novel expert decom-
position method enables scaling the expert count to 262,144 per layer while total
parameters scale proportionally to the square root of the number of experts. Our
analyses demonstrate mutual exclusivity of knowledge across experts and show-
case the parametric knowledge encapsulated within individual experts. Moreover,
MONET allows knowledge manipulation over domains, languages, and toxicity
mitigation without degrading general performance. Our pursuit of transparent
LLMs highlights the potential of scaling expert counts to enhance mechanistic
interpretability and directly resect the internal knowledge to fundamentally ad-
just model behavior. The source code and pretrained checkpoints are available at
https://github.com/dmis-lab/Monet.
1
INTRODUCTION
As large language models (LLMs) continue to scale and generalize (Radford et al., 2019; Brown
et al., 2020), understanding their internal computations becomes increasingly imperative. M...
Bandit Guided Submodular Curriculum for Adaptive
Subset Selection
Prateek Chanda∗
Prayas Agrawal∗
Saral Sureka
Lokesh Reddy Polu
Atharv Kshirsagar
Ganesh Ramakrishnan
Department of Computer Science and Engineering,
Indian Institute of Technology Bombay
{prateekch, prayas, ssaral
{lokeshreddypolu, atharvksagar, ganesh}@cse.iitb.ac.in
Abstract
Traditional curriculum learning proceeds from easy to hard samples, yet defining
a reliable notion of difficulty remains elusive. Prior work has used submodular
functions to induce difficulty scores in curriculum learning. We reinterpret adaptive
subset selection and formulate it as a multi-armed bandit problem, where each
arm corresponds to a submodular function guiding sample selection. We introduce
ONLINESUBMOD, a novel online greedy policy that optimizes a utility-driven re-
ward and provably achieves no-regret performance under various sampling regimes.
Empirically, ONLINESUBMOD outperforms both traditional curriculum learning
and bi-level optimization approaches across vision and language datasets, show-
ing superior accuracy-efficiency tradeoffs. More broadly, we show that validation-
driven reward metrics offer a principled way to guide the curriculum schedule. Our
code is publicly available at GitHub 2.
1
Introduction
Curriculum Learning (CL), inspired by cognitive development, posits that training machine learning
models by gradually exposing them to data of increasing complexity can significantly enhance
both learning efficiency and generalization performance [4, 59]. The underlying principle is that
mastering simpler concepts first provides a robust foundation for acquiring more complex ones,
leading to improved convergence and a more effective exploration of the hypothesis space [20].
Empirical evidence shows CL improves model training, particularly in areas like code understanding
[34], enhances graph embeddings through complexity-based ordering [57], mitigates catastrophic
forgetting [2, 27, 51, 45], and boosts le...
Published as a conference paper at ICLR 2025
LEARNING EFFICIENT POSITIONAL ENCODINGS WITH
GRAPH NEURAL NETWORKS
Charilaos I. Kanatsoulis1∗, Evelyn Choi1, Stephanie Jegelka2,3, Jure Leskovec1†,
Alejandro Ribeiro4†
1Stanford University
2MIT
3Technical University of Munich
4 University of Pennsylvania
ABSTRACT
Positional encodings (PEs) are essential for effective graph representation learning
because they provide position awareness in inherently position-agnostic trans-
former architectures and increase the expressive capacity of Graph Neural Net-
works (GNNs). However, designing powerful and efficient PEs for graphs poses
significant challenges due to the absence of canonical node ordering and the scale
of the graph. In this work, we identify four key properties that graph PEs should
satisfy: stability, expressive power, scalability, and genericness. We find that exist-
ing eigenvector-based PE methods often fall short of jointly satisfying these criteria.
To address this gap, we introduce PEARL, a novel framework of learnable PEs for
graphs. Our primary insight is that message-passing GNNs function as nonlinear
mappings of eigenvectors, enabling the design of GNN architectures for generating
powerful and efficient PEs. A crucial challenge lies in initializing node features
in a manner that is both expressive and permutation equivariant. We tackle this
by initializing GNNs with random node inputs or standard basis vectors, thereby
unlocking the expressive power of message-passing operations, while employing
statistical pooling functions to maintain permutation equivariance. Our analysis
demonstrates that PEARL approximates equivariant functions of eigenvectors
with linear complexity, while rigorously establishing its stability and high expres-
sive power. Experimental evaluations show that PEARL outperforms lightweight
versions of eigenvector-based PEs and achieves comparable performance to full
eigenvector-based PEs, but with one or two orders of magnitude lower com...
How Transformers Learn Structured Data:
Insights From Hierarchical Filtering
J´erˆome Garnier-Brun 1 2 Marc M´ezard 1 Emanuele Moscato 1 Luca Saglietti 1
Abstract
Understanding the learning process and the em-
bedded computation in transformers is becom-
ing a central goal for the development of inter-
pretable AI. In the present study, we introduce
a hierarchical filtering procedure for data mod-
els of sequences on trees, allowing us to hand-
tune the range of positional correlations in the
data. Leveraging this controlled setting, we pro-
vide evidence that vanilla encoder-only trans-
formers can approximate the exact inference al-
gorithm when trained on root classification and
masked language modeling tasks, and study how
this computation is discovered and implemented.
We find that correlations at larger distances, cor-
responding to increasing layers of the hierarchy,
are sequentially included by the network dur-
ing training. By comparing attention maps from
models trained with varying degrees of filtering
and by probing the different encoder levels, we
find clear evidence of a reconstruction of corre-
lations on successive length scales corresponding
to the various levels of the hierarchy, which we
relate to a plausible implementation of the exact
inference algorithm within the same architecture.
1. Introduction
Transformer-based large language models have revolution-
ized natural language processing, and have notably demon-
strated their capacity to perfectly assimilate the grammati-
cal rules of the languages they are trained on. While this ev-
idence shows that transformers can handle and exploit the
subtle long-range correlations that emerge in natural lan-
guage, their inner workings remain largely unclear.
1Department of Computing Sciences,
Universit`a Boc-
coni, Milan, Italy
2Department of Finance, Universit`a Boc-
coni, Milan, Italy.
Correspondence to: J´erˆome Garnier-Brun
<jerome.garnier@unibocconi.it>.
Proceedings of the 42 nd International Confer...
Published as a conference paper at ICLR 2025
DISENTANGLED REPRESENTATION LEARNING WITH
THE GROMOV-MONGE GAP
Theo Uscidda1,2∗
Luca Eyring2,3,4,∗
Karsten Roth2,4,5
Fabian Theis2,3,4
Zeynep Akata2,3,4,†
Marco Cuturi1,6,†
1CREST-ENSAE
2Helmholtz Munich
3TU Munich
4Munich Center of Machine Learning
5Tubingen AI Center
6Apple
theo.uscidda@ensae.fr
luca.eyring@tum.de
ABSTRACT
Learning disentangled representations from unlabelled data is a fundamental chal-
lenge in machine learning. Solving it may unlock other problems, such as general-
ization, interpretability, or fairness. Although remarkably challenging to solve in
theory, disentanglement is often achieved in practice through prior matching. Fur-
thermore, recent works have shown that prior matching approaches can be enhanced
by leveraging geometrical considerations, e.g., by learning representations that
preserve geometric features of the data, such as distances or angles between points.
However, matching the prior while preserving geometric features is challenging, as
a mapping that fully preserves these features while aligning the data distribution
with the prior does not exist in general. To address these challenges, we introduce a
novel approach to disentangled representation learning based on quadratic optimal
transport. We formulate the problem using Gromov-Monge maps that transport one
distribution onto another with minimal distortion of predefined geometric features,
preserving them as much as can be achieved. To compute such maps, we propose
the Gromov-Monge-Gap (GMG), a regularizer quantifying whether a map moves
a reference distribution with minimal geometry distortion. We demonstrate the
effectiveness of our approach for disentanglement across four standard benchmarks,
outperforming other methods leveraging geometric considerations.
1
INTRODUCTION
Learning low-dimensional representations of high-dimensional data is a fundamental challenge
in unsupervised deep learning (Bengio et al., 2014). Emphasis is put...
Published as a conference paper at ICLR 2025
CONGO:
COMPRESSIVE ONLINE GRADIENT OPTIMIZATION
Jeremy Carleton1∗
Prathik Vijaykumar1
Divyanshu Saxena2
Dheeraj Narasimha3
Srinivas Shakkottai1
Aditya Akella2
1 Texas A&M University, 2 The University of Texas at Austin, 3 Inria
ABSTRACT
We address the challenge of zeroth-order online convex optimization where the
objective function’s gradient exhibits sparsity, indicating that only a small number
of dimensions possess non-zero gradients. Our aim is to leverage this sparsity to
obtain useful estimates of the objective function’s gradient even when the only
information available is a limited number of function samples. Our motivation
stems from the optimization of large-scale queueing networks that process time-
sensitive jobs. Here, a job must be processed by potentially many queues in
sequence to produce an output, and the service time at any queue is a function of
the resources allocated to that queue. Since resources are costly, the end-to-end
latency for jobs must be balanced with the overall cost of the resources used. While
the number of queues is substantial, the latency function primarily reacts to resource
changes in only a few, rendering the gradient sparse. We tackle this problem by
introducing the Compressive Online Gradient Optimization framework which
allows compressive sensing methods previously applied to stochastic optimization
to achieve regret bounds with an optimal dependence on the time horizon without
the full problem dimension appearing in the bound. For specific algorithms, we
reduce the samples required per gradient estimate to scale with the gradient’s
sparsity factor rather than its full dimensionality. Numerical simulations and real-
world microservices benchmarks demonstrate CONGO’s superiority over gradient
descent approaches that do not account for sparsity.
1
INTRODUCTION
Online convex optimization (OCO) plays an important role in managing distributed applications like
web caching, advertise...
zip2zip: Inference-Time Adaptive Tokenization via
Online Compression
Saibo Geng1∗
Nathan Ranchin1∗
Yunzhen Yao1
Maxime Peyrard4
Chris Wendler1,2
Michael Gastpar1
Robert West1
1EPFL
2Northeastern University
4Université Grenoble Alpes, CNRS, Grenoble INP, LIG
{saibo.geng, nathan.ranchin, yunzhen.yao, michael.gastpar, robert.west}@epfl.ch
maxime.peyrard@univ-grenoble-alpes.fr
ch.wendler@northeastern.edu
Abstract
Tokenization efficiency plays a critical role in the performance and cost of large
language models (LLMs), yet most models rely on static tokenizers optimized on
general-purpose corpora. These tokenizers’ fixed vocabularies often fail to adapt
to domain- or language-specific inputs, leading to longer token sequences and
higher computational costs. We introduce zip2zip, a novel method for achieving
context-adaptive tokenization in LLMs at inference time. Leveraging an online
data compression algorithm (Lempel–Ziv–Welch), zip2zip dynamically expands
its active vocabulary at inference time by continuously replacing fragmented token
sequences with more compact hypertokens, which it can immediately output during
generation. zip2zip consists of three key components: (1) a tokenizer based on
Lempel–Ziv–Welch compression that incrementally merges co-occurring tokens
into reusable hypertokens on the fly; (2) a dynamic embedding (and unembedding)
layer that computes embeddings for newly formed hypertokens at runtime; and (3)
a variant of autoregressive language modeling that pretrains the model to handle
hypertokenized, compressed text sequences as inputs and outputs. We show that an
existing LLM can be uptrained for zip2zip in 10 GPU-hours via parameter-efficient
finetuning. The resulting LLM performs test-time adaptation, learning to use
hypertokens in unseen contexts and reducing input and output tokens by 15–40%.
Code and models are released at https://github.com/epfl-dlab/zip2zip.
1
Introduction
Large language models (LLMs) have shown impressive versatility across a...
PDEfuncta: Spectrally-Aware Neural Representation
for PDE Solution Modeling
Minju Jo1∗
Woojin Cho2∗
Uvini Balasuriya Mudiyanselage3
Seungjun Lee4
Noseong Park5
Kookjin Lee3†
1Sorbonne University,
2TelePIX,
3SCAI, Arizona State University,
4Brookhaven National Laboratory,
5KAIST
minnju42@gmail.com, woojin.py@gmail.com, ubalasur@asu.edu,
slee19@bnl.gov, noseong@kaist.ac.kr, kookjin.lee@asu.edu
Abstract
Scientific machine learning often involves representing complex solution fields that
exhibit high-frequency features such as sharp transitions, fine-scale oscillations,
and localized structures. While implicit neural representations (INRs) have shown
promise for continuous function modeling, capturing such high-frequency behavior
remains a challenge—especially when modeling multiple solution fields with a
shared network. Prior work addressing spectral bias in INRs has primarily focused
on single-instance settings, limiting scalability and generalization. In this work,
we propose Global Fourier Modulation (GFM), a novel modulation technique that
injects high-frequency information at each layer of the INR through Fourier-based
reparameterization. This enables compact and accurate representation of multiple
solution fields using low-dimensional latent vectors. Building upon GFM, we
introduce PDEfuncta, a meta-learning framework designed to learn multi-modal
solution fields and support generalization to new tasks. Through empirical studies
on diverse scientific problems, we demonstrate that our method not only improves
representational quality but also shows potential for forward and inverse inference
tasks without the need for retraining.
1
Introduction
Solving partial differential equations (PDEs) is essential to progress in many science and engineering
applications, including fluid dynamics, seismic inversion, climate modeling, and mechanical design.
Traditionally, these PDE problems are addressed using numerical methods (e.g. finite-difference [1],
finite-volume [2], fi...