AIS-Bench/AIS-Bench huggingface dataset:
Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks
for Full-Range Timestep Deployment
Chengting Yu 1 2 † Xiaochen Zhao 2 † Lei Liu 2 Shu Yang 2 Gaoang Wang 2 Erping Li 1 2 Aili Wang 1 2 Abstract
Spiking Neural Networks (SNNs) are emerging
as a brain-inspired alternative to traditional Arti-
ficial Neural Networks (ANNs), prized for their
potential energy efficiency on neuromorphic hard-
ware. Despite this, SNNs often suffer from accu-
racy degradation compared to ANNs and face
deployment challenges due to fixed inference
timesteps, which require retraining for adjust-
ments, limiting operational flexibility.
To ad-
dress these issues, our work considers the spatio-
temporal property inherent in SNNs, and pro-
poses a novel distillation framework for deep
SNNs that optimizes performance across full-
range timesteps without specific retraining, en-
hancing both efficacy and deployment adaptabil-
ity. We provide both theoretical analysis and em-
pirical validations to illustrate that training guaran-
tees the convergence of all implicit models across
full-range timesteps.
Experimental results on
CIFAR-10, CIFAR-100, CIFAR10-DVS, and Im-
ageNet demonstrate state-of-the-art performance
among distillation-based SNNs training methods.
Our code is available at https://github.com/Intelli-
Chip-Lab/snn temporal decoupling distillation.
1. Introduction
Spiking Neural Networks (SNNs) are modeled after biologi-
cal neural systems and feature spiking neurons that replicate
the dynamics of biological neurons (Maass, 1997; Roy et al.,
2019). In contrast to Artificial Neural Networks (ANNs),
which utilize continuous data forms, SNNs employ a spike-
coding approach, using discrete binary spike trains for data
transmission (Panzeri & Schultz, 2001). This binary sig-
naling significantly reduces the multiply-accumulate op-
† Equal contribution.
1College of Information Science and
Electronic Engineering, Zhejiang University 2ZJU-UIUC Insti-
tute, Zhejiang Univ...
Hyper-Transforming Latent Diffusion Models
Ignacio Peis 1 2 Batuhan Koyuncu 3 4 Isabel Valera 3 Jes Frellsen 1 2
Abstract
We introduce a novel generative framework for
functions by integrating Implicit Neural Represen-
tations (INRs) and Transformer-based hypernet-
works into latent variable models. Unlike prior ap-
proaches that rely on MLP-based hypernetworks
with scalability limitations, our method employs
a Transformer-based decoder to generate INR pa-
rameters from latent variables, addressing both
representation capacity and computational effi-
ciency. Our framework extends latent diffusion
models (LDMs) to INR generation by replacing
standard decoders with a Transformer-based hy-
pernetwork, which can be trained either from
scratch or via hyper-transforming—a strategy that
fine-tunes only the decoder while freezing the pre-
trained latent space. This enables efficient adapta-
tion of existing generative models to INR-based
representations without requiring full retraining.
We validate our approach across multiple modali-
ties, demonstrating improved scalability, expres-
siveness, and generalization over existing INR-
based generative models. Our findings establish a
unified and flexible framework for learning struc-
tured function representations.
1. Introduction
Generative modelling has seen remarkable advances in re-
cent years, with diffusion models achieving state-of-the-art
performance across multiple domains, including images,
videos, and 3D synthesis (Ho et al., 2020; Dhariwal &
Nichol, 2021; Rombach et al., 2022). A key limitation of ex-
isting generative frameworks, however, is their reliance on
structured output representations, such as pixel grids, which
constrain resolution and generalisation across data modal-
1Department of Applied Mathematics and Computer Science,
Technical University of Denmark, Copenhagen, Denmark 2Pioneer
Centre for Artificial Intelligence, Copenhagen, Denmark 3Saarland
University, Saarbr¨ucken, Germany 4Zuse School ELIZA. C...
Model-Based Exploration in Monitored Markov Decision Processes
Alireza Kazemipour 1 2 Matthew E. Taylor 1 2 3 Michael Bowling 1 2 3
Abstract
A tenet of reinforcement learning is that the agent
always observes rewards. However, this is not
true in many realistic settings, e.g., a human ob-
server may not always be available to provide
rewards, sensors may be limited or malfunction-
ing, or rewards may be inaccessible during de-
ployment. Monitored Markov decision processes
(Mon-MDPs) have recently been proposed to
model such settings. However, existing Mon-
MDP algorithms have several limitations: they
do not fully exploit the problem structure, can-
not leverage a known monitor, lack worst-case
guarantees for “unsolvable” Mon-MDPs without
specific initialization, and offer only asymptotic
convergence proofs. This paper makes three con-
tributions. First, we introduce a model-based algo-
rithm for Mon-MDPs that addresses these short-
comings. The algorithm employs two instances
of model-based interval estimation: one to ensure
that observable rewards are reliably captured, and
another to learn a minimax-optimal policy. Sec-
ond, we empirically demonstrate the advantages.
We show faster convergence than prior algorithms
in more than four dozen benchmarks, and even
more dramatic improvements when the monitor-
ing process is known. Third, we present the first
finite sample-bound on performance. We show
convergence to a minimax-optimal policy even
when some rewards are never observable.
1. Introduction
Reinforcement learning (RL) is based on trial-and-error: in-
stead of being directly shown what to do, an agent receives
consistent numerical feedback in the form of rewards for
its decisions. However, this assumption is not always re-
1Department of Computing Science, University of Alberta
2Alberta Machine Intelligence Institute (Amii) 3Canada CIFAR
AI Chair, Amii.
Correspondence to:
Alireza Kazemipour
<kazemipo@ualberta.ca>.
Proceedings of the 42 nd International Confere...
Published as a conference paper at ICLR 2025
TULIP: TOKEN-LENGTH UPGRADED CLIP
Ivona Najdenkoska∗Mohammad Mahdi Derakhshani∗Yuki M. Asano
Nanne van Noord
Marcel Worring† Cees G. M. Snoek†
University of Amsterdam
Amsterdam, the Netherlands
ABSTRACT
We address the challenge of representing long captions in vision-language mod-
els, such as CLIP. By design these models are limited by fixed, absolute posi-
tional encodings, restricting inputs to a maximum of 77 tokens and hindering
performance on tasks requiring longer descriptions. Although recent work has
attempted to overcome this limit, their proposed approaches struggle to model
token relationships over longer distances and simply extend to a fixed new to-
ken length. Instead, we propose a generalizable method, named TULIP, able
to upgrade the token length to any length for CLIP-like models.
We do so
by improving the architecture with relative position encodings, followed by a
training procedure that (i) distills the original CLIP text encoder into an en-
coder with relative position encodings and (ii) enhances the model for aligning
longer captions with images. By effectively encoding captions longer than the
default 77 tokens, our model outperforms baselines on cross-modal tasks such
as retrieval and text-to-image generation.
The code repository is available at
https://github.com/ivonajdenkoska/tulip.
1
INTRODUCTION
Obtaining text representations for long captions in vision-language models is an open research chal-
lenge. Within the context of text-only large language models, this problem has been studied ex-
tensively (Chung et al., 2024; Touvron et al., 2023; Bai et al., 2023; Gu & Dao, 2023; Su et al.,
2024; Dubey et al., 2024; Pawar et al., 2024; Jiang et al., 2024; Lieber et al., 2024), yet, methods
for context length expansion have only scarcely made their way into the vision-language domain.
For instance, contrastive vision-language models like CLIP (Radford et al., 2021; Jia et al., 2021; Li
et al., 2022)...
Self-Improving Transformers Overcome Easy-to-Hard and Length
Generalization Challenges
Nayoung Lee * 1 Ziyang Cai * 1 Avi Schwarzschild 2 Kangwook Lee 1 Dimitris Papailiopoulos 1 3
Abstract
Large language models often struggle with length
generalization and solving complex problem in-
stances beyond their training distribution. We
present a self-improvement approach where mod-
els iteratively generate and learn from their own
solutions, progressively tackling harder problems
while maintaining a standard transformer archi-
tecture.
Across diverse tasks including arith-
metic, string manipulation, and maze solving,
our method enables models to solve problems far
beyond their initial training distribution—for in-
stance, generalizing from 10-digit to 100-digit
addition without apparent saturation.
We ob-
serve that filtering for correct self-generated exam-
ples leads to exponential improvements in out-of-
distribution performance across training rounds.
Additionally, starting from pretrained models sig-
nificantly accelerates this self-improvement pro-
cess for several tasks. Our results demonstrate
how controlled weak-to-strong curricula can sys-
tematically expand model capabilities while pre-
serving architectural simplicity.
1. Introduction
Despite the remarkable success of transformer-based lan-
guage models (Vaswani et al., 2017) across a wide range
of tasks, these models exhibit significant limitations in
length generalization—the ability to extrapolate to longer
sequences than those seen during training. Even in simple
algorithmic tasks such as arithmetic, standard transformer
models trained with autoregressive objectives struggle to
generalize to longer problem instances (Anil et al., 2022).
To address this, prior work has explored various modifica-
*Equal contribution
1University of Wisconsin-Madison
2Carnegie Mellon University
3Microsoft.
Correspondence
to:
Nayoung Lee <nayoung.lee@wisc.edu>,
Ziyang Cai
<zcai75@wisc.edu>.
Proceedings of the 42 nd Internatio...
Conformal Prediction as Bayesian Quadrature
Jake C. Snell 1 Thomas L. Griffiths 1 2
Abstract
As machine learning-based prediction systems are
increasingly used in high-stakes situations, it is
important to understand how such predictive mod-
els will perform upon deployment. Distribution-
free uncertainty quantification techniques such as
conformal prediction provide guarantees about
the loss black-box models will incur even when
the details of the models are hidden. However,
such methods are based on frequentist probability,
which unduly limits their applicability. We revisit
the central aspects of conformal prediction from
a Bayesian perspective and thereby illuminate the
shortcomings of frequentist guarantees. We pro-
pose a practical alternative based on Bayesian
quadrature that provides interpretable guarantees
and offers a richer representation of the likely
range of losses to be observed at test time.
1. Introduction
Machine learning systems based on deep learning are in-
creasingly used in high-stakes settings, such as medical
diagnosis or financial applications. These settings impose
unique constraints on the performance of these systems: we
want them to produce good outcomes in the aggregate, but
also do so fairly and with a guarantee of a low probability of
harm. However, predictive models based on deep learning
can be difficult to interpret, and commercial models increas-
ingly tend to offer little information about the techniques
used in training. This creates a new challenge: How can we
flexibly and reliably quantify the suitability of a model for
deployment without making too many assumptions about
how the model was trained or in which settings it will be
used?
Recent research on quantifying uncertainty has employed
methods based on conformal prediction (Vovk et al., 2005),
1Department of Computer Science, Princeton University
2Department of Psychology, Princeton University. Correspondence
to: Jake C. Snell <jsnell@princeton.edu>.
Proceedings of the 4...
Physics-Informed DeepONets for drift-diffusion on metric graphs: simulation
and parameter identification
Jan Blechschmidt 1 2 Tom-Christian Riemer 1 Max Winkler 1 Martin Stoll 1 Jan-F. Pietschmann 3
Abstract
We develop a novel physics informed deep learn-
ing approach for solving nonlinear drift-diffusion
equations on metric graphs. These models repre-
sent an important model class with a large number
of applications in areas ranging from transport in
biological cells to the motion of human crowds.
While traditional numerical schemes require a
large amount of tailoring, especially in the case
of model design or parameter identification prob-
lems, physics informed deep operator networks
(DEEPONETS) have emerged as a versatile tool
for the solution of partial differential equations
with the particular advantage that they easily in-
corporate parameter identification questions. We
here present an approach where we first learn
three DEEPONET models for representative in-
flow, inner and outflow edges, resp., and then
subsequently couple these models for the solu-
tion of the drift-diffusion metric graph problem
by relying on an edge-based domain decomposi-
tion approach. We illustrate that our framework
is applicable for the accurate evaluation of graph-
coupled physics models and is well suited for
solving optimization or inverse problems on these
coupled networks.
1. Introduction
Dynamic processes on graphs (Newman, 2018; Barab´asi,
2016) are crucial for understanding complex phenomena
1Department of Mathematics, TU Chemnitz, Chemnitz,
Germany 2Department of Mathematics and Computer Science,
TU Bergakademie Freiberg, Freiberg, Germany 3Department
of Mathematics, University of Augsburg, Augsburg, Germany
and Centre for Advanced Analytics and Predictive Sciences
(CAAPS), University of Augsburg, Universit¨atsstr. 12a, 86159
Augsburg, Germany.
.
Correspondence to:
Jan Blech-
schmidt <jan.blechschmidt@math.tu-chemnitz.de>, Martin Stoll
<martin.stoll@math.tu-chemnitz.de>.
...
Causal Logistic Bandits with Counterfactual Fairness Constraints
Jiajun Chen 1 Jin Tian 2 Christopher John Quinn 1
Abstract
Artificial intelligence will play a significant role
in decision making in numerous aspects of soci-
ety. Numerous fairness criteria have been pro-
posed in the machine learning community, but
there remains limited investigation into fairness
as defined through specified attributes in a sequen-
tial decision-making framework. In this paper, we
focus on causal logistic bandit problems where
the learner seeks to make fair decisions, under
a notion of fairness that accounts for counterfac-
tual reasoning. We propose and analyze an al-
gorithm by leveraging primal-dual optimization
for constrained causal logistic bandits where the
non-linear constraints are a priori unknown and
must be learned in time. We obtain sub-linear
regret guarantees with leading term similar to
that for unconstrained logistic bandits (Lee et al.,
2024) while guaranteeing sub-linear constraint
violations. We show how to achieve zero cumula-
tive constraint violations with a small increase in
the regret bound.
1. Introduction
Artificial intelligence (AI) models, using techniques from
statistics and machine learning, are increasingly being used
to make decisions that affect people’s lives. In light of this, a
plethora of formal fairness criteria have been proposed (Dar-
lington, 1971; Dwork et al., 2012; Hardt et al., 2016; Zhang
et al., 2016; Kusner et al., 2017; Zafar et al., 2017; Nabi
& Shpitser, 2018; Chiappa, 2019; Chouldechova & Roth,
2020; Imai & Jiang, 2023; Plecko & Bareinboim, 2024).
There has been growing interest in the sequential decision-
making community for accounting for fairness, including in
settings such as classic and contextual bandits (Joseph et al.,
2018), combinatorial bandits (Xu et al., 2020), bandits with
1Department of Computer Science, Iowa State University,
Ames, IA, USA 2Mohamed bin Zayed University of Artificial
Intelligence, Abu Dhabi, UAE. C...
Published as a conference paper at ICLR 2025
PHYSICS-INFORMED DIFFUSION MODELS
Jan-Hendrik Bastek
Dept. of Mechanical and Process Eng.
ETH Zurich
Zurich, Switzerland
jbastek@ethz.ch
WaiChing Sun
Dept. of Civil Eng. and Eng. Mechanics
Columbia University
New York, NY, USA
wsun@columbia.edu
Dennis M. Kochmann
Dept. of Mechanical and Process Eng.
ETH Zurich
Zurich, Switzerland
dmk@ethz.ch
ABSTRACT
Generative models such as denoising diffusion models are quickly advancing their
ability to approximate highly complex data distributions. They are also increasingly
leveraged in scientific machine learning, where samples from the implied data
distribution are expected to adhere to specific governing equations. We present
a framework that unifies generative modeling and partial differential equation
fulfillment by introducing a first-principle-based loss term that enforces generated
samples to fulfill the underlying physical constraints. Our approach reduces the
residual error by up to two orders of magnitude compared to previous work in a
fluid flow case study and outperforms task-specific frameworks in relevant metrics
for structural topology optimization. We also present numerical evidence that
our extended training objective acts as a natural regularization mechanism against
overfitting. Our framework is simple to implement and versatile in its applicability
for imposing equality and inequality constraints as well as auxiliary optimization
objectives.
1
INTRODUCTION
Denoising diffusion models (Sohl-Dickstein et al., 2015) are generative models that have gained
popularity due to their success in learning intricate data distributions across various modalities,
including images (Ho et al., 2020; Nichol et al., 2021; Rombach et al., 2021), videos (Ho et al.,
2022a;b), graphs (Niu et al., 2020; Hoogeboom et al., 2022), text (Li et al., 2022; Wu et al., 2023), and
audio waveforms (Kong et al., 2021). Originally popularized within the image generation community,
their outstandin...
Synonymous Variational Inference for Perceptual Image Compression
Zijian Liang 1 Kai Niu 1 2 Changshuo Wang 1 Jin Xu 1 Ping Zhang 3
Abstract
Recent contributions of semantic information the-
ory reveal the set-element relationship between
semantic and syntactic information, represented
as synonymous relationships. In this paper, we
propose a synonymous variational inference (SVI)
method based on this synonymity viewpoint to re-
analyze the perceptual image compression prob-
lem. It takes perceptual similarity as a typical syn-
onymous criterion to build an ideal synonymous
set (Synset), and approximate the posterior of its
latent synonymous representation with a paramet-
ric density by minimizing a partial semantic KL
divergence. This analysis theoretically proves
that the optimization direction of perception im-
age compression follows a triple tradeoff that
can cover the existing rate-distortion-perception
schemes. Additionally, we introduce synonymous
image compression (SIC), a new image compres-
sion scheme that corresponds to the analytical
process of SVI, and implement a progressive
SIC codec to fully leverage the model’s capabili-
ties. Experimental results demonstrate compara-
ble rate-distortion-perception performance using
a single progressive SIC codec, thus verifying the
effectiveness of our proposed analysis method.
1. Introduction
Image compression is a typical topic for lossy source cod-
ing, aiming to achieve the optimal tradeoff between recon-
structed image quality and coding rate. Following the rate-
distortion optimization instructed by Shannon’s classic in-
formation theory (1948; 1959), traditional image compres-
sion like JPEG (Wallace, 1991) and BPG (Bellard, 2015)
1Key Laboratory of Universal Wireless Communications, Min-
istry of Education, Beijing University of Posts and Telecommuni-
cations, Beijing, China 2Peng Cheng Laboratory, Shenzhen, China
3State Key Laboratory of Networking and Switching Technol-
ogy, Beijing University of Posts an...
Circumventing Backdoor Space via Weight Symmetry
Jie Peng 1 Hongwei Yang 1 Jing Zhao 1 Hengji Dong 1 Hui He 1 Weizhe Zhang 1 2 Haoyu He 3
Abstract
Deep neural networks are vulnerable to backdoor
attacks, where malicious behaviors are implanted
during training. While existing defenses can effec-
tively purify compromised models, they typically
require labeled data or speciﬁc training proce-
dures, making them difﬁcult to apply beyond su-
pervised learning settings. Notably, recent studies
have shown successful backdoor attacks across
various learning paradigms, highlighting a crit-
ical security concern. To address this gap, we
propose Two-stage Symmetry Connectivity (TSC),
a novel backdoor puriﬁcation defense that oper-
ates independently of data format and requires
only a small fraction of clean samples. Through
theoretical analysis, we prove that by leverag-
ing permutation invariance in neural networks
and quadratic mode connectivity, TSC ampliﬁes
the loss on poisoned samples while maintaining
bounded clean accuracy. Experiments demon-
strate that TSC achieves robust performance com-
parable to state-of-the-art methods in supervised
learning scenarios. Furthermore, TSC generalizes
to self-supervised learning frameworks, such as
SimCLR and CLIP, maintaining its strong defense
capabilities. Our code is available at https:
//github.com/JiePeng104/TSC.
1. Introduction
Modern classiﬁers require substantial data and computa-
tional resources to achieve high accuracy, providing adver-
saries with opportunities to implant backdoors into deep
neural networks (Gu et al., 2017; Chen et al., 2017). This
vulnerability arises from either injecting poisoned data (Gu
et al., 2017; Turner et al., 2019) (i.e., data-poisoning attack),
or manipulating the training process (Pang et al., 2019;
1School of Cyberspace Science, Harbin Institute of Technol-
ogy, Harbin, China.
2Pengcheng Laboratory, Shenzhen, China.
3Department of Data Science and AI, Faculty of IT, Monash
University, Melb...
SING: SDE Inference via Natural Gradients
Amber Hu∗
Stanford University
amberhu@stanford.edu
Henry D. Smith∗
Stanford University
smithhd@stanford.edu
Scott W. Linderman
Stanford University
swl1@stanford.edu
Abstract
Latent stochastic differential equation (SDE) models are important tools for the
unsupervised discovery of dynamical systems from data, with applications ranging
from engineering to neuroscience. In these complex domains, exact posterior
inference of the latent state path is typically intractable, motivating the use of
approximate methods such as variational inference (VI). However, existing VI
methods for inference in latent SDEs often suffer from slow convergence and
numerical instability. Here, we propose SDE Inference via Natural Gradients
(SING), a method that leverages natural gradient VI to efficiently exploit the un-
derlying geometry of the model and variational posterior. SING enables fast and
reliable inference in latent SDE models by approximating intractable integrals
and parallelizing computations in time. We provide theoretical guarantees that
SING approximately optimizes the intractable, continuous-time objective of inter-
est. Moreover, we demonstrate that better state inference enables more accurate
estimation of nonlinear drift functions using, for example, Gaussian process SDE
models. SING outperforms prior methods in state inference and drift estimation
on a variety of datasets, including a challenging application to modeling neural
dynamics in freely behaving animals. Altogether, our results illustrate the potential
of SING as a tool for accurate inference in complex dynamical systems, especially
those characterized by limited prior knowledge and non-conjugate structure.1
1
Introduction
Stochastic differential equations (SDEs) are powerful tools for modeling complex, time-varying
systems in scientific domains such as physics, engineering, and neuroscience [1, 2]. In many real-
world settings, we do not observe the state of the syste...