AIS-Bench/AIS-Bench huggingface dataset:
Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than
Extrapolation
Cheng Jin 1 Zhenyu Xiao * 1 Chutao Liu * 2 Yuantao Gu 1
Abstract
Classifier-free guidance (CFG) has emerged as a
pivotal advancement in text-to-image latent diffu-
sion models, establishing itself as a cornerstone
technique for achieving high-quality image syn-
thesis. However, under high guidance weights,
where text-image alignment is significantly en-
hanced, CFG also leads to pronounced color dis-
tortions in the generated images. We identify
that these distortions stem from the amplifica-
tion of sample norms in the latent space. We
present a theoretical framework that elucidates
the mechanisms of norm amplification and anoma-
lous diffusion phenomena induced by classifier-
free guidance. Leveraging our theoretical insights
and the latent space structure, we propose an An-
gle Domain Guidance (ADG) algorithm. ADG
constrains magnitude variations while optimizing
angular alignment, thereby mitigating color dis-
tortions while preserving the enhanced text-image
alignment achieved at higher guidance weights.
Experimental results demonstrate that ADG sig-
nificantly outperforms existing methods, generat-
ing images that not only maintain superior text
alignment but also exhibit improved color fidelity
and better alignment with human perceptual pref-
erences.
1. Introduction
The Stable Diffusion series (Rombach et al., 2022; Esser
et al., 2024), along with other latent-diffusion-based genera-
tive models (Ramesh et al., 2022; Labs, 2024), has revolu-
tionized text-to-image generation and related downstream
tasks (Zhang et al., 2023; Zhao et al., 2024; Zhang et al.,
*Equal contribution
1Department of Electronic Engineering,
Tsinghua University, Beijing, China 2Zhili College, Tsinghua
University, Beijing, China.
Correspondence to: Yuantao Gu
<gyt@tsinghua.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by t...
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Pierre Ablin 1 Angelos Katharopoulos 1 Skyler Seto 1 David Grangier 1
Abstract
Machine learning models are routinely trained on
a mixture of different data domains. Different
domain weights yield very different downstream
performances. We propose the Soup-of-Experts,
a novel architecture that can instantiate a model
at test time for any domain weights with mini-
mal computational cost and without re-training
the model. Our architecture consists of a bank of
expert parameters, which are linearly combined
to instantiate one model. We learn the linear com-
bination coefficients as a function of the input
domain weights. To train this architecture, we
sample random domain weights, instantiate the
corresponding model, and backprop through one
batch of data sampled with these domain weights.
We demonstrate how our approach obtains small
specialized models on several language modeling
tasks quickly. Soup-of-Experts are particularly
appealing when one needs to ship many different
specialist models quickly under a size constraint.
1. Introduction
Large Language Models (LLMs) work well on diverse tasks
because they have many parameters and are trained on gen-
eralist datasets (Brown et al., 2020; Bommasani et al., 2021).
However, they are costly to train and to serve, both in terms
of memory and inference cost.
Specialist language models hold fewer parameters; they
are, therefore, cheaper to store, send, and use at inference.
However, they must give up the generality of LLMs and
specialize in a few specific topics.
In cases where there is an abundance of specialization data,
training a small model on those data yields a good specialist.
However, in many settings, the specialization data is scarce:
for instance, it may come from a narrow topic of interest
1Apple.
Correspondence
to:
Pierre
Ablin
<p ablin@apple.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PM...
SEFE: Superficial and Essential Forgetting Eliminator for
Multimodal Continual Instruction Tuning
Jinpeng Chen * 1 Runmin Cong * 2 Yuzhi Zhao 1 Hongzheng Yang 3
Guangneng Hu 4 Horace Ho Shing Ip 1 Sam Kwong 5
https://github.com/jinpeng0528/SEFE/
Abstract
Multimodal Continual Instruction Tuning (MCIT)
aims to enable Multimodal Large Language Mod-
els (MLLMs) to incrementally learn new tasks
without catastrophic forgetting. In this paper, we
explore forgetting in this context, categorizing it
into superficial forgetting and essential forgetting.
Superficial forgetting refers to cases where the
model’s knowledge may not be genuinely lost,
but its responses to previous tasks deviate from
expected formats due to the influence of subse-
quent tasks’ answer styles, making the results
unusable. By contrast, essential forgetting refers
to situations where the model provides correctly
formatted but factually inaccurate answers, indi-
cating a true loss of knowledge. Assessing essen-
tial forgetting necessitates addressing superficial
forgetting first, as severe superficial forgetting can
obscure the model’s knowledge state. Hence, we
first introduce the Answer Style Diversification
(ASD) paradigm, which defines a standardized
process for transforming data styles across differ-
ent tasks, unifying their training sets into similarly
diversified styles to prevent superficial forgetting
caused by style shifts. Building on this, we pro-
pose RegLoRA to mitigate essential forgetting.
RegLoRA stabilizes key parameters where prior
knowledge is primarily stored by applying regu-
larization, enabling the model to retain existing
competencies. Experimental results demonstrate
that our overall method, SEFE, achieves state-of-
the-art performance.
*Equal contribution 1City University of Hong Kong 2Shandong
University
3The Chinese University of Hong Kong
4Xidian
University 5Lingnan University.
Correspondence to:
Yuzhi
Zhao <yzzhao2-c@my.cityu.edu.hk>, Sam Kwong <samk-
wong@lingnan.edu.hk>....
Published as a conference paper at ICLR 2025
MULTI-LABEL TEST-TIME ADAPTATION WITH BOUND
ENTROPY MINIMIZATION
Xiangyu Wu1,2, Feng Yu1, Qing-Guo Chen2, Yang Yang1∗, Jianfeng Lu1∗
1Nanjing University of Science and Technology
2Alibaba International Digital Commerce Group
ABSTRACT
Mainstream test-time adaptation (TTA) techniques endeavor to mitigate distri-
bution shifts via entropy minimization for multi-class classification, inherently
increasing the probability of the most confident class. However, when encoun-
tering multi-label instances, the primary challenge stems from the varying num-
ber of labels per image, and prioritizing only the highest probability class in-
evitably undermines the adaptation of other positive labels. To address this issue,
we investigate TTA within multi-label scenario (ML–TTA), developing Bound
Entropy Minimization (BEM) objective to simultaneously increase the confi-
dence of multiple top predicted labels. Specifically, to determine the number of
labels for each augmented view, we retrieve a paired caption with yielded tex-
tual labels for that view. These labels are allocated to both the view and cap-
tion, called weak label set and strong label set with the same size k. Follow-
ing this, the proposed BEM considers the highest top-k predicted labels from
view and caption as a single entity, respectively, learning both view and cap-
tion prompts concurrently. By binding top-k predicted labels, BEM overcomes
the limitation of vanilla entropy minimization, which exclusively optimizes the
most confident class. Across the MSCOCO, VOC, and NUSWIDE multi-label
datasets, our ML–TTA framework equipped with BEM exhibits superior perfor-
mance compared to the latest SOTA methods, across various model architec-
tures, prompt initialization, and varying label scenarios. The code is available
at https://github.com/Jinx630/ML-TTA.
1
INTRODUCTION
The advent of vision-language models (VLMs) (Radford et al., 2021; Li et al., 2023; Zeng et al.,
2024; Ya...
Compositional Causal Reasoning Evaluation in Language Models
Jacqueline R. M. A. Maasch 1 Alihan Hüyük 2 Xinnuo Xu 3 Aditya V. Nori 3 Javier Gonzalez 3
Abstract
Causal reasoning and compositional reasoning
are two core aspirations in AI. Measuring the ex-
tent of these behaviors requires principled eval-
uation methods. We explore a unified perspec-
tive that considers both behaviors simultaneously,
termed compositional causal reasoning (CCR):
the ability to infer how causal measures compose
and, equivalently, how causal quantities propagate
through graphs. We instantiate a framework for
the systematic evaluation of CCR for the average
treatment effect and the probability of necessity
and sufficiency. As proof of concept, we demon-
strate CCR evaluation for language models in the
Llama, Phi, and GPT families. On a math word
problem, our framework revealed a range of tax-
onomically distinct error patterns. CCR errors
increased with the complexity of causal paths for
all models except o1.
1. Introduction
Causal reasoning is a defining outcome of human evolution
(Goddu & Gopnik, 2024). Humans flexibly reason about
cause and effect in factual realities that can be observed and
intervened on, as well as imagined counterfactual worlds.
A causal lens enables humans and machines alike to learn
generalizable lessons about the mechanics of the universe
(Schölkopf et al., 2021). Thus, human-like AI might require
reasoning at all three levels of Pearl’s Causal Hierarchy: as-
sociational, interventional, and counterfactual (Pearl, 2000).
Human-like AI might also require compositional reasoning
(Lake et al., 2017): the capacity to recognize and synthe-
size novel combinations of previously observed concepts
(Xu et al., 2022). Compositionality is ubiquitous in the
physical world, symbolic systems, and human cognition
1Cornell
Tech
2Harvard
University
3Microsoft
Re-
search Cambridge.
Correspondence to:
Jacqueline Maasch
<maasch@cs.cornell.edu>. Work done while authors J. Maasch
an...
Unnatural Languages Are Not Bugs but Features for LLMs
Keyu Duan * 1 Yiran Zhao * 1 Zhili Feng 2 Jinjie Ni 1 Tianyu Pang 3 Qian Liu 3 Tianle Cai 4 Longxu Dou 3
Kenji Kawaguchi 1 Anirudh Goyal 5 J. Zico Kolter 2 Michael Qizhe Shieh 1
Abstract
Large Language Models (LLMs) have been ob-
served to process non-human-readable text se-
quences, such as jailbreak prompts, often viewed
as a bug for aligned LLMs. In this work, we
present a systematic investigation challenging
this perception, demonstrating that unnatural lan-
guages - strings that appear incomprehensible
to humans but maintain semantic meanings for
LLMs - contain latent features usable by mod-
els. Notably, unnatural languages possess latent
features that can be generalized across differ-
ent models and tasks during inference. Further-
more, models fine-tuned on unnatural versions
of instruction datasets perform on-par with those
trained on natural language, achieving 49.71 win
rates in Length-controlled AlpacaEval 2.0 in av-
erage across various base models. In addition,
through comprehensive analysis, we demonstrate
that LLMs process unnatural languages by filter-
ing noise and inferring contextual meaning from
filtered words. Our code is publicly available
at https://github.com/John-AI-Lab/
Unnatural_Language.
1. Introduction
Large Language Models (LLMs) (OpenAI, 2023; Touvron
et al., 2023; Dubey et al., 2024; anthropic, 2024) have shown
remarkable capabilities in understanding and generating
human-readable text, achieving impressive performance
across tasks, spanning from question answering (Bisk et al.,
2020; Ni et al., 2024) and mathematical reasoning (Cobbe
et al., 2021; Hendrycks et al., 2021; Gao et al., 2024) to
open-ended dialogue (Li et al., 2023). Such abilities are
*Equal contribution
1National University of Singapore
2Carnegie Mellon University 3Sea AI Lab 4Princeton Univer-
sity 5Mila, University of Montreal. Correspondence to: Keyu
Duan <k.duan@u.nus.edu>, Yiran Zhao <zhaoyiran@u.nus.edu>,
Mi...
Score Matching with Missing Data
Josh Givens 1 Song Liu 1 Henry W J Reeve 2
Abstract
Score matching is a vital tool for learning the dis-
tribution of data with applications across many
areas including diffusion processes, energy based
modelling, and graphical model estimation. De-
spite all these applications, little work explores
its use when data is incomplete. We address this
by adapting score matching (and its major exten-
sions) to work with missing data in a flexible set-
ting where data can be partially missing over any
subset of the coordinates. We provide two sep-
arate score matching variations for general use,
an importance weighting (IW) approach, and a
variational approach. We provide finite sample
bounds for our IW approach in finite domain set-
tings and show it to have especially strong perfor-
mance in small sample lower dimensional cases.
Complementing this, we show our variational ap-
proach to be strongest in more complex high-
dimensional settings which we demonstrate on
graphical model estimation tasks on both real and
simulated data.
1. Introduction
Over the last decade, score matching has established itself
as a powerful tool with downstream use in many areas of
machine learning. Examples include: energy based mod-
elling (Swersky et al., 2011; Bao et al., 2020; Li et al.,
2019b), mode-seeking clustering (Sasaki et al., 2014), and
perhaps most prominently of all Diffusion processes (Song
& Ermon, 2019; Song et al., 2021b; Tashiro et al., 2021;
Song et al., 2021a; Huang et al., 2021). Score matching
aims to learn the score of a distribution which is the gra-
dient of the log of the probability density function (PDF)
(s(x) = ∇x log p(x)). In contrast to modelling the density
directly, the score does not need to integrate to one meaning
there is no need to calculate a normalising constant. This
allows it to be much more more flexibly modelled than the
1School of Mathematics, University of Bristol, Bristol, UK
2School of Artificial Intelligence,...
NICE
Data Selection for Instruction Tuning in LLMs
with Non-differentiable Evaluation Metric
Jingtan Wang 1 2 Xiaoqiang Lin 1 Rui Qiao 1 3 Pang Wei Koh 4 Chuan-Sheng Foo 2 5 Bryan Kian Hsiang Low 1 3
Abstract
Curating data for instruction tuning is crucial
for enhancing the performance of large language
models (LLMs). This work aims to select training
data for instruction tuning to improve the LLM
performance on specific tasks. Existing methods
often rely on next-token prediction (NTP) loss as
a proxy for target task performance due to the non-
differentiable nature of performance evaluation
metrics. They select training data points that are
most helpful in reducing validation loss. However,
there is a discrepancy between minimizing NTP
loss and maximizing performance (e.g., code pass
rate in code generation). To remedy this, we intro-
duce a novel Non-differentiable evaluation metric-
based InfluenCe Estimation (NICE), which lever-
ages the policy gradient to select the training data
that improves the performance. Moreover, NICE
can perform data selection in the absence of labels
(ground-truth responses) when the evaluation met-
rics do not require labels (e.g., a reward model can
output reward scores without supervision from
labels). Experimental results show that our ap-
proach outperforms existing data selection base-
lines that use NTP loss in diverse and realistic sce-
narios. Notably, subsets selected by NICE often
produce models that outperform those trained on
the full dataset. Our code is available at https:
//github.com/JTWang2000/NICE.
1. Introduction
Instruction tuning (Bai et al., 2022; Ouyang et al., 2022) is
a fine-tuning paradigm that enables large language models
1National University of Singapore 2Institute for Infocomm Re-
search (I2R), A*STAR, Singapore 3Singapore-MIT Alliance for
Research and Technology 4University of Washington 5Centre for
Frontier AI Research (CFAR), A*STAR, Singapore. Correspon-
dence to: Bryan Kian Hsiang Low <lowkh@comp.nus...
Is Best-of-N the Best of Them?
Coverage, Scaling, and Optimality in Inference-Time Alignment
Audrey Huang 1 Adam Block 2 Qinghua Liu 2 Nan Jiang 1 Akshay Krishnamurthy 2 Dylan J. Foster 2
Abstract
Recent work on inference-time alignment has
established the beneﬁts of increasing inference-
time computation in language models,
but
naively scaling compute through techniques like
Best-of-N sampling can cause performance to de-
grade due to reward hacking. Toward a theoreti-
cal understanding of how to best leverage addi-
tional computation, we formalize inference-time
alignment as improving a pre-trained policys re-
sponses for a prompt of interest, given access to
an imperfect reward model. We analyze the per-
formance of inference-time alignment algorithms
in terms of (i) response quality, and (ii) compute,
and provide new results that highlight the impor-
tance of the pre-trained policys coverage over
high-quality responses for performance and com-
pute scaling: (1) We show that Best-of-N align-
ment with an ideal N can achieve optimal perfor-
mance under stringent notions of coverage, but
provably suffers from reward hacking when N
is large, and fails to achieve tight guarantees un-
der more realistic coverage conditions; (2) We
introduce InferenceTimePessimism, a new al-
gorithm which mitigates reward hacking through
deliberate use of inference-time compute, imple-
menting pessimism in the face of uncertainty; we
prove that its performance is optimal and scaling-
monotonic, i.e., does not degrade as N increases.
We complement our theoretical results with ex-
periments that demonstrate the practicality of our
algorithm across a variety of tasks and models.
1. Introduction
Inference-time computation has emerged as a new axis
for scaling in language models, which has led to dramatic
1Computer Science Department, University of Illinois at
Urbana-Champaign, Champaign, Illinois, USA 2Microsoft Re-
search, New York, New York, USA. Correspondence to: Audrey
Huang <audreyh5...
Published as a conference paper at ICLR 2025
GRAPHBRIDGE:
TOWARDS ARBITRARY TRANSFER
LEARNING IN GNNS
Li Ju, Xingyi Yang, Qi Li, Xinchao Wang∗
National University of Singapore
{jujulili,xyang,liqi}@u.nus.edu, xinchao@nus.edu.sg
ABSTRACT
Graph neural networks (GNNs) are conventionally trained on a per-domain, per-
task basis. It creates a significant barrier in transferring the acquired knowledge
to different, heterogeneous data setups. This paper introduces GraphBridge, a
novel framework to enable knowledge transfer across disparate tasks and domains
in GNNs, circumventing the need for modifications to task configurations or graph
structures. Specifically, GraphBridge allows for the augmentation of any pre-
trained GNN with prediction heads and a bridging network that connects the input
to the output layer. This architecture not only preserves the intrinsic knowledge of
the original model but also supports outputs of arbitrary dimensions. To mitigate
the negative transfer problem, GraphBridge merges the source model with a con-
currently trained model, thereby reducing the source bias when applied to the tar-
get domain. Our method is thoroughly evaluated across diverse transfer learning
scenarios, including Graph2Graph, Node2Node, Graph2Node, and graph2point-
cloud. Empirical validation, conducted over 16 datasets representative of these
scenarios, confirms the framework’s capacity for task- and domain-agnostic trans-
fer learning within graph-like data, marking a significant advancement in the field
of GNNs. Code is available at https://github.com/jujulili888/GraphBridge.
1
INTRODUCTION
With the explosive growth of graph data, the application of Graph Neural Networks (GNNs) has
become increasingly widespread in domains (Jing et al., 2022; 2021b; Wu et al., 2020b; Yu et al.,
2018; Shah et al., 2020) such as recommendation systems (Gao et al., 2022; Wu et al., 2019) and
biopharmaceutics(Zitnik et al., 2018; Rathi et al., 2019). Despite their growing popularity, the ...
Autoformulation of Mathematical Optimization Models Using LLMs
Nicol´as Astorga * 1 Tennison Liu * 1 Yuanzhang Xiao 2 Mihaela van der Schaar 1
Abstract
Mathematical optimization is fundamental to
decision-making across diverse domains, from
operations research to healthcare. Yet, translat-
ing real-world problems into optimization models
remains a difficult task, often demanding special-
ized expertise. This paper approaches the prob-
lem of autoformulation: the automated creation
of solver-ready optimization models from natu-
ral language problem descriptions. We identify
three core challenges of autoformulation: (1) the
vast, problem-dependent hypothesis space, (2) ef-
ficient and diverse exploration of this space un-
der uncertainty, and (3) evaluation of formulation
correctness against problem description. To ad-
dress these challenges, we present a novel method
leveraging Large Language Models (LLMs) with
Monte-Carlo Tree Search, exploiting the hierar-
chical nature of optimization modeling to gener-
ate and systematically explore possible formula-
tions. To enhance search efficiency, we introduce
symbolic pruning to eliminate trivially equiva-
lent search paths (branches), and employ LLM-
based evaluation of partial formulations to guide
search. Empirical analysis on linear and mixed-
integer programming benchmarks demonstrates
our method’s effectiveness, with significant per-
formance gains from both LLM-based value esti-
mation and symbolic pruning techniques.
1. Introduction
Mathematical optimization has long been a cornerstone of
decision-making processes across various domains, from
supply chain management (Bramel & Simchi-Levi, 1997)
and healthcare resource allocation (Delgado et al., 2022) to
portfolio optimization (Mokhtar et al., 2014). These prob-
*Equal contribution
1DAMTP, University of Cambridge,
Cambridge, UK 2ECE, University of Hawaii at Manoa, Hon-
olulu, USA. Correspondence to: Nicol´as Astorga, Tennison Liu
<{nja46,tl522}@cam.ac.uk>.
Proceedi...
Provable Meta-Learning with Low-Rank Adaptations
Jacob L. Block
UT Austin
jblock@utexas.edu
Sundararajan Srinivasan
UT Austin
sundararajans@utexas.edu
Liam Collins
Snap, Inc.
lcollins2@snapchat.com
Aryan Mokhtari
UT Austin & Google Research
mokhtari@austin.utexas.edu
Sanjay Shakkottai
UT Austin
sanjay.shakkottai@utexas.edu
Abstract
The power of foundation models (FMs) lies in their capacity to learn highly expres-
sive representations that can be adapted to a broad spectrum of tasks. However,
these pretrained models require additional training stages to become effective for
downstream applications. In the multi-task setting, prior works have shown em-
pirically that specific meta-learning approaches for preparing a model for future
adaptation through parameter-efficient fine-tuning (PEFT) can outperform standard
retraining methods, but the mechanism of the benefits of meta-learning has been
largely unexplored. We introduce a framework for generic PEFT-based meta-
learning to learn a model that can easily adapt to unseen tasks. For linear models
using LoRA, we show that standard retraining is provably suboptimal for finding
an adaptable set of parameters and provide strict performance guarantees for our
proposed method. We verify these theoretical insights through experiments on
synthetic data as well as real-data vision and language tasks. We observe significant
performance benefits using a simple implementation of our proposed meta-learning
scheme during retraining relative to the conventional approach.
1
Introduction
Foundation Models (FMs) learn rich representations that are useful for a variety of downstream tasks.
The first stage of FM training is referred to as pretraining, where a combination of massive public,
proprietary, and synthetic sources of data is used to learn a general-purpose model from scratch
[1–4]. However, due to the enormous cost of training state-of-the-art models on such large datasets,
pretraining is largely infeasible for most. Thus, the ...