NeurIPS 2025

Inference-time Alignment in Continuous Space
Yige Yuan1,2∗,
Teng Xiao3,4∗,
Yunfan Li1,2,
Bingbing Xu1†,
Shuchang Tao5,
Yunqi Qiu1,2,
Huawei Shen1,2,
Xueqi Cheng1,2
1Institute of Computing Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3University of Washington, 4Allen Institute for AI, 5Alibaba Group
yuanyige20z@ict.ac.cn, tengxiao01@gmail.com, liyunfan24s@ict.ac.cn
Abstract
Aligning large language models with human feedback at inference time has received
increasing attention due to its flexibility. Existing methods rely on generating
multiple responses from the base policy for search using a reward model, which
can be considered as searching in a discrete response space. However, these
methods struggle to explore informative candidates when the base policy is weak
or the candidate set is small, resulting in limited effectiveness. In this paper, to
address this problem, we propose Simple Energy Adaptation (SEA), a simple yet
effective algorithm for inference-time alignment. In contrast to expensive search
over the discrete space, SEA directly adapts original responses from the base policy
toward the optimal one via gradient-based sampling in continuous latent space.
Specifically, SEA formulates inference as an iterative optimization procedure on an
energy function over actions in the continuous space defined by the optimal policy,
enabling simple and effective alignment. For instance, despite its simplicity, SEA
outperforms the second-best baseline with a relative improvement of up to 77.51%
on AdvBench and 16.36% on MATH. Our code is publicly available at this link.
1
Introduction
The alignment of large language models (LLMs) plays a crucial role to ensure the model outputs
meet human expectations and reflect human values [1, 2, 3, 4, 5]. Reinforcement Learning from
Human Feedback (RLHF) [2, 6] has emerged as a widely adopted method for LLM alignment. RLHF
typically involves training a reward model based on human feedback and sub...