AIS-Bench/AIS-Bench huggingface dataset:
TIMEPERCEIVER: An Encoder-Decoder Framework
for Generalized Time-Series Forecasting
Jaebin Lee
Sungkyunkwan University
jaebin.lee@skku.edu
Hankook Lee
Sungkyunkwan University
hankook.lee@skku.edu
Abstract
In machine learning, effective modeling requires a holistic consideration of how
to encode inputs, make predictions (i.e., decoding), and train the model. How-
ever, in time-series forecasting, prior work has predominantly focused on encoder
design, often treating prediction and training as separate or secondary concerns.
In this paper, we propose TIMEPERCEIVER, a unified encoder-decoder forecast-
ing framework that is tightly aligned with an effective training strategy. To be
specific, we first generalize the forecasting task to include diverse temporal pre-
diction objectives such as extrapolation, interpolation, and imputation. Since this
generalization requires handling input and target segments that are arbitrarily po-
sitioned along the temporal axis, we design a novel encoder-decoder architecture
that can flexibly perceive and adapt to these varying positions. For encoding,
we introduce a set of latent bottleneck representations that can interact with all
input segments to jointly capture temporal and cross-channel dependencies. For
decoding, we leverage learnable queries corresponding to target timestamps to
effectively retrieve relevant information. Extensive experiments demonstrate that
our framework consistently and significantly outperforms prior state-of-the-art
baselines across a wide range of benchmark datasets. The code is available at
https://github.com/efficient-learning-lab/TimePerceiver.
1
Introduction
Time-series forecasting is a fundamental task in machine learning, aiming to predict future events
based on past observations. It is of practical importance, as it plays a crucial role in many real-world
applications, including weather forecasting [1], electricity consumption forecasting [2], and traffic
flow prediction [3]. Despite decades of rap...
KernelBench: Can LLMs Write Efficient GPU Kernels?
Anne Ouyang * 1 Simon Guo * 1 Simran Arora 1 Alex L Zhang 2 William Hu 1 Christopher R´e 1
Azalia Mirhoseini 1
Abstract
Efficient GPU kernels are crucial for building per-
formant machine learning architectures, but writ-
ing them is a time-consuming challenge that re-
quires significant expertise; therefore, we explore
using language models (LMs) to automate kernel
generation. We introduce KernelBench, an open-
source framework for evaluating LMs’ ability to
write fast and correct kernels on a suite of 250
carefully selected PyTorch ML workloads. Ker-
nelBench represents a real-world engineering en-
vironment and making progress on the introduced
benchmark directly translates to faster practical
kernels. We introduce a new evaluation metric
fastp, which measures the percentage of gener-
ated kernels that are functionally correct and offer
a speedup greater than an adjustable threshold
p over baseline. Our experiments across vari-
ous state-of-the-art models and test-time meth-
ods show that frontier reasoning models perform
the best out of the box but still fall short overall,
matching the PyTorch baseline in less than 20% of
the cases. While we show that results can improve
by leveraging execution and profiling feedback
during iterative refinement, KernelBench remains
a challenging benchmark, with its difficulty in-
creasing as we raise speedup threshold p.
1. Introduction
AI relies on efficient GPU kernels to achieve high perfor-
mance and cost and energy savings; however, developing
kernels remains challenging. There has been a Cambrian
explosion of ML architectures (Tay et al., 2022; Peng et al.,
*Equal contribution 1Department of Computer Science, Stan-
ford University, Stanford, California, USA 2Department of Com-
puter Science, Princeton University, Princeton, New Jersey, USA.
Correspondence to: Anne Ouyang <aco@stanford.edu>, Simon
Guo <simonguo@stanford.edu>.
Proceedings of the 42 nd International Conferen...
3D-Prover: Diversity Driven Theorem Proving With
Determinantal Point Processes
Sean Lamont1,2, Christian Walder3, Amir Dezfouli4, Paul Montague2, Michael Norrish1
1Australian National University
2Defence Science and Technology Group
3Google DeepMind
4BIMLOGIQ
sean.lamont@anu.edu.au
Abstract
A key challenge in automated formal reasoning is the intractable search space,
which grows exponentially with the depth of the proof. This branching is caused
by the large number of candidate proof tactics which can be applied to a given
goal. Nonetheless, many of these tactics are semantically similar or lead to an
execution error, wasting valuable resources in both cases. We address the problem
of effectively pruning this search, using only synthetic data generated from previous
proof attempts. We first demonstrate that it is possible to generate semantically
aware tactic representations which capture the effect on the proving environment,
likelihood of success, and execution time. We then propose a novel filtering
mechanism which leverages these representations to select semantically diverse
and high quality tactics, using Determinantal Point Processes. Our approach, 3D-
Prover, is designed to be general, and to augment any underlying tactic generator.
We demonstrate the effectiveness of 3D-Prover on the miniF2F and LeanDojo
benchmarks by augmenting popular open source proving LLMs. We show that
our approach leads to an increase in the overall proof rate, as well as a significant
improvement in the tactic success rate, execution time and diversity. We make our
code available at https://github.com/sean-lamont/3D-Prover.
1
Introduction
Interactive Theorem Proving (ITP) traditionally involves a human guiding an ITP system to verify a
formal proposition. The applications range from secure software (Tan et al., 2019) to the verification
of mathematical results (Hales et al., 2017). There has been significant interest in automating this
process, with formalization efforts requiring ...
Published as a conference paper at ICLR 2025
PERIODWAVE: MULTI-PERIOD FLOW MATCHING FOR
HIGH-FIDELITY WAVEFORM GENERATION
Sang-Hoon Lee1,2,∗
Ha-Yeong Choi3,∗
Seong-Whan Lee4†
1Department of Software and Computer Engineering, Ajou University, Suwon, Korea
2Department of Artificial Intelligence, Ajou University, Suwon, Korea
3Gen AI Lab, KT Corp., Seoul, Korea
4Department of Artificial Intelligence, Korea University, Seoul, Korea
ABSTRACT
Recently, universal waveform generation tasks have been investigated conditioned
on various out-of-distribution scenarios. Although one-step GAN-based methods
have shown their strength in fast waveform generation, they are vulnerable to
train-inference mismatch scenarios such as two-stage text-to-speech. Meanwhile,
diffusion-based models have shown their powerful generative performance in other
domains; however, they stay out of the limelight due to slow inference speed in
waveform generation tasks. Above all, there is no generator architecture that can
explicitly disentangle the natural periodic features of high-resolution waveform
signals. In this paper, we propose PeriodWave, a novel universal waveform gener-
ation model from Mel-spectrogram and neural audio codec. First, we introduce
a period-aware flow matching estimator that effectively captures the periodic fea-
tures of the waveform signal when estimating the vector fields. Additionally, we
utilize a multi-period estimator that avoids overlaps to capture different periodic
features of waveform signals. Although increasing the number of periods can
improve the performance significantly, this requires more computational costs. To
reduce this issue, we also propose a single period-conditional universal estimator
that can feed-forward parallel by period-wise batch inference. Additionally, we
first introduce FreeU to reduce the high-frequency noise for waveform generation.
Furthermore, we demonstrate the effectiveness of the proposed method in neural
audio codec decoding task, and...
Published as a conference paper at ICLR 2025
PALU: KV-CACHE COMPRESSION WITH LOW-RANK
PROJECTION
Chi-Chih Chang1,3∗
Wei-Cheng Lin1∗
Chien-Yu Lin2∗
Chong-Yan Chen1
Yu-Fang Hu1
Pei-Shuo Wang1
Ning-Chi Huang1
Luis Ceze2
Mohamed S. Abdelfattah3
Kai-Chiang Wu1
1National Yang Ming Chiao Tung University
2University of Washington
3Cornell University
ABSTRACT
Post-training KV-Cache compression methods typically either sample a subset of
effectual tokens or quantize the data into lower numerical bit width. However,
these methods cannot exploit redundancy in the hidden dimension of the KV
tensors. This paper presents a hidden dimension compression approach called
Palu, a KV-Cache compression framework that utilizes low-rank projection to
reduce inference-time LLM memory usage. Palu decomposes the linear layers
into low-rank matrices, caches compressed intermediate states, and reconstructs
the full keys and values on the fly. To improve accuracy, compression rate, and
efficiency, Palu further encompasses (1) a medium-grained low-rank decompo-
sition scheme, (2) an efficient rank search algorithm, (3) low-rank-aware quan-
tization compatibility enhancements, and (4) optimized GPU kernels with oper-
ators fusion. Extensive experiments with popular LLMs show that Palu com-
presses KV-Cache by 50%, while maintaining strong accuracy and delivering up to
1.89× speedup on the RoPE-based attention module. When combined with quan-
tization, Palu’s inherent quantization-friendly design yields small to negligible
extra accuracy degradation, while saving additional memory than quantization-
only methods and achieving up to 2.91× speedup for the RoPE-based attention.
Moreover, it maintains comparable or even better accuracy (up to 1.19 lower
perplexity) compared to quantization-only methods. These results demonstrate
Palu’s superior capability to effectively address the efficiency and memory chal-
lenges of LLM inference posed by KV-Cache. Our code is publicly available at:
https://github.c...
Multi-Marginal Stochastic Flow Matching
for High-Dimensional Snapshot Data at Irregular Time Points
Justin Lee 1 Behnaz Moradijamei 2 Heman Shakeri 1
Abstract
Modeling the evolution of high-dimensional sys-
tems from limited snapshot observations at irreg-
ular time points poses a significant challenge in
quantitative biology and related fields. Traditional
approaches often rely on dimensionality reduction
techniques, which can oversimplify the dynam-
ics and fail to capture critical transient behaviors
in non-equilibrium systems. We present Multi-
Marginal Stochastic Flow Matching (MMSFM), a
novel extension of simulation-free score and flow
matching methods to the multi-marginal setting,
enabling the alignment of high-dimensional data
measured at non-equidistant time points without
reducing dimensionality. The use of measure-
valued splines enhances robustness to irregular
snapshot timing, and score matching prevents
overfitting in high-dimensional spaces. We val-
idate our framework on several synthetic and
benchmark datasets, including gene expression
data collected at uneven time points and an im-
age progression task, demonstrating the method’s
versatility.1
1. Introduction
Understanding cellular responses to perturbations is a fun-
damental challenge in quantitative biology, with significant
implications for fields such as developmental biology, can-
cer research, and drug discovery (Altschuler & Wu, 2010;
Saeys et al., 2016). Modeling these responses requires cap-
turing complex stochastic dynamics in high-dimensional
cellular states that evolve over time under the influence of
1School of Data Science, University of Virginia, Charlottesville
VA, USA. 2James Madison University, Harrisonburg VA, USA.
Correspondence to: Justin Lee <jgh2xh@virginia.edu>, Heman
Shakeri <hs9hd@virginia.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Code
available
at
https://github.com...
Published as a conference paper at ICLR 2025
TAILORING MIXUP TO DATA FOR CALIBRATION
Quentin Bouniot,1,2,3,4 Pavlo Mozharovskyi1 & Florence d’Alché-Buc1
1LTCI, Télécom Paris, Institut Polytechnique de Paris, France,
2Technical University of Munich,
3Helmholtz Munich,
4Munich Center for Machine Learning (MCML)
ABSTRACT
Among all data augmentation techniques proposed so far, linear interpolation of
training samples, also called Mixup, has found to be effective for a large panel of
applications. Along with improved predictive performance, Mixup is also a good
technique for improving calibration. However, mixing data carelessly can lead
to manifold mismatch, i.e., synthetic data lying outside original class manifolds,
which can deteriorate calibration. In this work, we show that the likelihood of
assigning a wrong label with mixup increases with the distance between data to
mix. To this end, we propose to dynamically change the underlying distributions
of interpolation coefficients depending on the similarity between samples to mix,
and define a flexible framework to do so without losing in diversity. We provide
extensive experiments for classification and regression tasks, showing that our
proposed method improves predictive performance and calibration of models,
while being much more efficient.
1
INTRODUCTION
The Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2000) improves on the well-known
Empirical Risk Minimization (ERM) (Vapnik, 1998) for training deep neural networks by drawing
virtual samples from a vicinity around true training data. This data augmentation principle is known to
improve the generalization ability of deep neural networks when the number of observed data is small
compared to the task complexity. In practice, the method of choice to implement it relies on hand-
crafted procedures to mimic natural perturbations (Yaeger et al., 1996; Ha & Bunke, 1997; Simard
et al., 2002). However, one counterintuitive but effective and less application...
KEA: Keeping Exploration Alive
by Proactively Coordinating Exploration Strategies
Shih-Min Yang 1 Martin Magnusson 1 Johannes A. Stork 1 Todor Stoyanov 1
Abstract
Soft Actor-Critic (SAC) has achieved notable suc-
cess in continuous control tasks but struggles in
sparse reward settings, where infrequent rewards
make efficient exploration challenging. While
novelty-based exploration methods address this
issue by encouraging the agent to explore novel
states, they are not trivial to apply to SAC. In par-
ticular, managing the interaction between novelty-
based exploration and SAC’s stochastic policy
can lead to inefficient exploration and redundant
sample collection. In this paper, we propose KEA
(Keeping Exploration Alive) which tackles the
inefficiencies in balancing exploration strategies
when combining SAC with novelty-based explo-
ration. KEA integrates a novelty-augmented SAC
with a standard SAC agent, proactively coordi-
nated via a switching mechanism. This coordina-
tion allows the agent to maintain stochasticity in
high-novelty regions, enhancing exploration effi-
ciency and reducing repeated sample collection.
We first analyze this potential issue in a 2D naviga-
tion task, and then evaluate KEA on the DeepSea
hard-exploration benchmark as well as sparse re-
ward control tasks from the DeepMind Control
Suite. Compared to state-of-the-art novelty-based
exploration baselines, our experiments show that
KEA significantly improves learning efficiency
and robustness in sparse reward setups.
1. Introduction
Despite the success of deep reinforcement learning (RL) in
continuous control tasks, such as robotic manipulation, these
methods often rely on manually designed dense rewards
(Zhou et al., 2023; Zhou & Held, 2023; Zhang et al., 2023;
Yang et al., 2024), which require task-specific expertise and
1Department of AASS, ¨Orebro University, ¨Orebro, Sweden.
Correspondence to: Shih-Min Yang <shih-min.yang@oru.se>.
Proceedings of the 42 nd International Conference on Ma...
Scaling Collapse Reveals Universal Dynamics in
Compute-Optimally Trained Neural Networks
Shikai Qiu 1 † Lechao Xiao 2 Andrew Gordon Wilson 1 Jeffrey Pennington 2 Atish Agarwala 2
Abstract
What scaling limits govern neural network train-
ing dynamics when model size and training time
grow in tandem? We show that despite the com-
plex interactions between architecture, training
algorithms, and data, compute-optimally trained
models exhibit a remarkably precise universality.
Specifically, loss curves from models of varying
sizes collapse onto a single universal curve when
training compute and loss are normalized to unity
at the end of training. With learning rate decay,
the collapse becomes so tight that differences in
the normalized curves across models fall below
the noise floor of individual loss curves across
random seeds, a phenomenon we term supercol-
lapse. We observe supercollapse across learning
rate schedules, datasets, and architectures, includ-
ing transformers trained on next-token prediction,
and find it breaks down when hyperparameters
are scaled suboptimally, providing a precise and
practical indicator of good scaling. We explain
these phenomena by connecting collapse to the
power-law structure in typical neural scaling laws,
and analyzing a simple yet surprisingly effective
model of SGD noise dynamics that accurately
predicts loss curves across various learning rate
schedules and quantitatively explains the origin
of supercollapse.
1. Introduction
As machine learning systems grow in scale, accurate predic-
tive models of their training dynamics become increasingly
valuable, both for interpreting costly experiments and for de-
signing robust, efficient training pipelines (Wortsman et al.,
†Work done partly during an internship at Google Deep-
Mind
1New York University 2Google DeepMind. Correspon-
dence to: Shikai Qiu <sq2129@nyu.edu>, Atish Agarwala
<thetish@google.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, C...
Temporal Misalignment in ANN-SNN Conversion and
its Mitigation via Probabilistic Spiking Neurons
Velibor Bojkovi´c * 1 Xiaofeng Wu * 2 Bin Gu 3
Abstract
Spiking Neural Networks (SNNs) offer a more
energy-efficient alternative to Artificial Neural
Networks (ANNs) by mimicking biological neu-
ral principles, establishing them as a promising ap-
proach to mitigate the increasing energy demands
of large-scale neural models. However, fully har-
nessing the capabilities of SNNs remains chal-
lenging due to their discrete signal processing and
temporal dynamics. ANN-SNN conversion has
emerged as a practical approach, enabling SNNs
to achieve competitive performance on complex
machine learning tasks. In this work, we identify a
phenomenon in the ANN-SNN conversion frame-
work, termed temporal misalignment, in which
random spike rearrangement across SNN layers
leads to performance improvements. Based on
this observation, we introduce biologically plau-
sible two-phase probabilistic (TPP) spiking neu-
rons, further enhancing the conversion process.
We demonstrate the advantages of our proposed
method both theoretically and empirically through
comprehensive experiments on CIFAR-10/100,
CIFAR10-DVS, and ImageNet across a variety
of architectures, achieving state-of-the-art results.
Codes are available at https://github.
com/shizukanaskytree/TPP-SNN.
1. Introduction
Spiking neural networks (SNNs), often referred to as the
third generation of neural networks (Maass, 1997), closely
mimic biological neuronal communication through dis-
crete spikes (McCulloch & Pitts, 1943; Hodgkin & Hux-
*Equal contribution
1Department of Machine Learning, Mo-
hamed bin Zayed University of Artificial Intelligence, Abu Dhabi,
United Arab Emirates 2Faculty of Data Science, City University
of Macau, Macau, China 3School of Artificial Intelligence, Jilin
University, China. Correspondence to: Velibor Bojkovic <veli-
bor.bojkovic@mbzuai.ac.ae>.
Proceedings of the 42 nd International Conference on Machine...
Stochastic Poisson Surface Reconstruction with One Solve
using Geometric Gaussian Processes
Sidhanth Holalkere 1 David S. Bindel 1 Silvia Sellán 2 3 Alexander Terenin 1
Abstract
Poisson Surface Reconstruction is a widely-used
algorithm for reconstructing a surface from an ori-
ented point cloud. To facilitate applications where
only partial surface information is available, or
scanning is performed sequentially, a recent line
of work proposes to incorporate uncertainty into
the reconstructed surface via Gaussian process
models. The resulting algorithms first perform
Gaussian process interpolation, then solve a set
of volumetric partial differential equations glob-
ally in space, resulting in a computationally ex-
pensive two-stage procedure. In this work, we
apply recently-developed techniques from geo-
metric Gaussian processes to combine interpo-
lation and surface reconstruction into a single
stage, requiring only one linear solve per sam-
ple. The resulting reconstructed surface samples
can be queried locally in space, without the use of
problem-dependent volumetric meshes or grids.
These capabilities enable one to (a) perform prob-
abilistic collision detection locally around the re-
gion of interest, (b) perform ray casting without
evaluating points not on the ray’s trajectory, and
(c) perform next-view planning on a per-ray ba-
sis. They also do not requiring one to approxi-
mate kernel matrix inverses with diagonal matri-
ces as part of intermediate computations, unlike
prior methods. Results show that our approach
provides a cleaner, more-principled, and more-
flexible stochastic surface reconstruction pipeline.
1. Introduction
Surface reconstruction algorithms are used to process
point cloud data—the most-common format in which real-
Code: HTTPS://GITHUB.COM/SHOLALKERE/GEOSPSR.
1Cornell University 2Columbia University 3MIT. Correspon-
dence to: Sidhanth Holalkere <sh844@cornell.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Va...
Multi-Objective Causal Bayesian Optimization
Shriya Bhatija 1 2 Paul-David Zuercher 2 3 Jakob Thumm 1 Thomas Bohn´e 2
Abstract
In decision-making problems, the outcome of
an intervention often depends on the causal
relationships between system components and
is highly costly to evaluate.
In such settings,
causal Bayesian optimization (CBO) exploits the
causal relationships between the system variables
and sequentially performs interventions to ap-
proach the optimum with minimal data. Extend-
ing CBO to the multi-outcome setting, we pro-
pose multi-objective causal Bayesian optimiza-
tion (MO-CBO), a paradigm for identifying Pareto-
optimal interventions within a known multi-target
causal graph.
Our methodology first reduces
the search space by discarding sub-optimal in-
terventions based on the structure of the given
causal graph. We further show that any MO-CBO
problem can be decomposed into several tradi-
tional multi-objective optimization tasks. Our
proposed MO-CBO algorithm is designed to iden-
tify Pareto-optimal interventions by iteratively ex-
ploring these underlying tasks, guided by relative
hypervolume improvement. Experiments on syn-
thetic and real-world causal graphs demonstrate
the superiority of our approach over non-causal
multi-objective Bayesian optimization in settings
where causal information is available.
1. Introduction
Decision-making problems arise in various domains, such
as healthcare, manufacturing, and public policy, and in-
volve manipulating variables to obtain an outcome of in-
terest. In many such domains, interventions are inherently
costly, and practical applications are subject to budgetary
constraints. Moreover, these systems are often governed
1Department of Computer Engineering, Technical University of
Munich, Munich, Germany 2Department of Engineering, Univer-
sity of Cambridge, Cambridge, United Kingdom 3The Alan Turing
Institute, London, United Kingdom. Correspondence to: Shriya
Bhatija <shriya.bhatija@tum.de>.
Proceedings ...