Test-Time Alignment for Large Language Models via Textual Model Predictive Control
Authors: Kuang-Da Wang
Teng-Ruei Chen
Yu Heng Hung
Guo-Xun Ko
Shuoyang Ding
Yueh-Hua Wu
Yu-Chiang Frank Wang
Chao-Han Huck Yang
Wen-Chih Peng
Ping-Chun Hsieh
Venue: ICLR 2026 Poster (2026)
Keywords: Test-time preference alignment
Large Language Models
Machine translation
Primary area: foundation or frontier models, including LLMs
TL;DR: Test-time preference alignment, Large Language Models, Machine translation

Abstract: Aligning Large Language Models (LLMs) with human preferences through finetuning is resource-intensive, motivating lightweight alternatives at test time. We address test-time alignment through the lens of sequential decision making, a perspective that reveals two fundamental challenges. When actions are defined at the token level, as in guided decoding, alignment suffers from the curse of horizon. Conversely, when actions are at the response level, as in traditional iterative refinement, the curse of dimensionality emerges. To resolve this trade-off, we draw inspiration from Model Predictive Control (MPC) in control theory to propose Textual Model Predictive Control (TMPC), a novel predictive planning framework adapted for aligning LLMs at inference time. A key limitation of standard MPC is its reliance on predefined, hard segment boundaries, which are often absent in text generation. TMPC overcomes this by introducing two principles inspired by hierarchical reinforcement learning: (1) Hindsight Subgoal Identification, where TMPC analyzes generation subgoals to retrospectively identify high-reward intermediate outputs as subgoals. This allows the framework to discover meaningful, task-specific planning steps (e.g., a sentence in machine translation or a bug fix in code generation.). (2) Subgoal-Conditioned Re-Generation, where these identified subgoals are used to guide subsequent planning iterations. By conditioning on these proven, high-quality subgoals, TMPC ensures stable improvement by building upon previously validated successes. TMPC is evaluated on three tasks with distinct segmentation properties: discourse-level translation, long-form response generation, and program synthesis. The results demonstrate that TMPC consistently improves performance, highlighting the generality. Project page: https://rl-bandits-lab.github.io/TMPC/.

Published as a conference paper at ICLR 2026
TEST-TIME ALIGNMENT FOR LARGE LANGUAGE MOD- ELS VIA TEXTUAL MODEL PREDICTIVE CONTROL
Kuang-Da Wang Rung Chen Yu-Heng Hung'
Guo-Xun Ko
Shuoyang Ding Yueh-Hua Wu² Yu-Chiang Frank Wang Chao-Han Huck Yang*
Wen-Chih Peng' Ping-Chun Hsieh
'National Yang Ming Chiao Tung University, Hsinchu, Taiwan
NVIDIA
National Taiwan University, Taipei, Taiwan
(gdwang. cs10 wet,:ccc1.spp..... pinghsieh @encce edu. tw, huckyēnvidia.com
ABSTRACT
Aligning Large Language Models (LLMs) with human preferences through fine- tuning is resource-intensivy motivating lightweight alternatives test time. We address test-time alignment through the lens of sequential decision making, a perspective that reveals two fundamental challenges. When actions defined at the token level, as in guided decoding, alignment suffers from the curse of horizon. Conversely, when actions are at the response level, as traditional iter- ative refinement, the curse of dimensionality emerges. To resolve this trade-off, we draw inspiration from Model Predictive Control (MPC) in control theory to propose Textual Model Predictive Control (TMPC), novel predictive plan- ning framework adapted for aligning LLMs at inference time. A key limitation of standard MPC is its reliance on predefined, hard segment boundaries, which are often absent in text generation. TMPC overcomes this by introducing two principles inspired by hierarchical reinforcement learning: (1) Hindsight Subgoal Identification, where TMPC analyzes generation subgoals to retrospectively iden- tify high-reward intermediate outputs as subgoals. This allows the framework discover meaningful, task-specific planning steps (e.g.. sentence in machine trans- lation bug fix in code generation.). (2) Subgoal-Condition
Re- ReeGeeerraion, where these identified subgoals are used to guide subsequent planning iterations. By conditioning on these proven, high-quality subgoals, TMPC ensures stable improvement by building upon previously validated successes. TMPC is evaluated on three tasks with distinct segmentation properties: discourse-level translation, long-form response generation, and program synthesis. The results demonstrate that TMPC consistently improves performance, highlighting the generality. Project page: https://rl-bann: ts- tlab.. gi thub. io/TM////
INTRODUCTION
The emergence of Large Language Models (LLMs), such the GPT series (Achiam al., 2023; Brown et al., 2020), LLaMAs (Touvron et al., 2023a;b), and Gemma (Team et al., 2024), has demonstrated remarkable efficacy in a wide range of NLP tasks (Hendrycks al., 2021; Srivastava et al., 2023; Stiennon et al., 2020; Yu et al., 2024; Zhong et al., 2024). While these models exhibit strong performance out of the box, aligning their outputs to human preferences remains critical, especially for smaller-scale LLMs. For instance, in machine translation (Alves et al., 2024), smaller LLMs (e.g., under 10B parameters) frequently suffer from omissions and semantic drift (Wu et al., 2024). Thus, aligning L.M outputs to preferences remains essential yet challenging problem.
Training-time approaches such as Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov al., 2023) have achieved strong results in aligning preferences. However, these methods are resource-intensive and require costly retraining whenever preferences or tasks change. This has spurred interest in rest-time alignment, where outputs are adapted without updating model parameters, using test-time strategies such as "Equal contribution.

---

Published as a conference paper at ICLR 2026
Hortzon (Gulded Decoding)
Textual MPC
Curse of Dimensionality
(Iterative Refinement)
Token-level control leads overly long horizons, making aligmment fragile.
Hindsight Subgoal Identification
Resporis-ieval rewriiigg oper large action space, causing unstable improvements.
Subgeal-Conditione Re-Generation
O OREE
Multi-step evaluation
Generated span
Promising state
Deviating span
Figure 1: Textual Model Predictive Control (TMPC) balances the curse of horizon in guided decoding against the curse of dimensionality naive iterative refinement. employs Hindsight Subgoal Identification to dynamically discover promising states from rollouts and Subgoal-Conditin Re- Generation to guide the search from these discovered subgoals, ensuring stable alignment.
prompting (Lin et al., 2024), guided decoding (Khanov et al., 2024; Kong al., 2024; et al., 2024; Wang et al., 2024b; Xu et al., 2025), or iterative refinement (Li et al., 2025).
We address test-time alignment through the lens of sequential decision making, where the generation process is framed as a sequence of actions. This perspective reveals two fundamental challenges, illustrated in Figure 1. When actions are defined at the token level (e.g., guided decoding), methods suffer from the curse of horizon (Park et al., 2025); credit assignment becomes unreliable over long trajectories, making alignment brittle. In contrast, when actions are at the response level (e.g.. iterative refinement), they face the curse of dimensionality; each step involves rewriting entire sequence, making the search for improvements in vast action space intractable and unstable.
To address these challenges, propose Textual Model Predietive Control (TMPC), novel test- time alignment framework inspired by Model Predictive Control (MPC) (Camacho & Bordons, 2007: Kouvaritakis Cannon, 2016). While powerful, standard MPC assumes the problem can be decomposed into predefined, hard segment boundaries, a condition that rarely holds for complex text generation. TMPC uniquely adapted overcome this limitation through two principles:
Hindsight Subgoal Identification: This principle allows TMPC to discover meaningful planning steps. After generating candidate responses, TMPC retrospectively analyzes them to identify high-quality intermediate points as subgoals. A subgoal can be concrete unit, such as sentence translation, or an abstract one, such as resolving single failed test case in program synthesis, successfully addressing the problem of lacking natural boundaries. This hindsight-driven discovery effectively shortens the planning horizon for diverse tasks.
Sundoal-Conditioned Re-Generation: This principle ensures stable, cumulative progress. The subgoals identified via hindsight stored in buffer and used guide subsequent planning iterations. By conditioning the next generation on these subgoals, TMPC ensures that subsequent generation builds upon these validated waypoints.
We evaluate TMPC on three challenging tasks with different boundary characteristics: WM discourse-level machine translation, the HH-RLHF long responses subset, and MBPP program syn- thesis. Experiments with I.L.MA3311- 1---1-188--Intruue show that TMPC consistently improves alignment, highlighting the generality of our approach.
Our contributions are summarized as follows:
We formulate test-time alignment under sequential mrrisioo-making lens. This perspective reveals that guided decoding faces a curse of horizon, while iterative refinement faces a curse of dimensionality. Building this formulation, introduce novel trajectory-
2

---

Published as a conference paper at ICLR 2026
optimization framework for test-time alignment, which, to knowledge, has not been explored in prior work.
We introduce Textual Model Predictive Control (TMPC), framework that adapts con- cepts from control theory to language generation. TMPC is operationalized through two principles: Hindsight Subgoal Identification discover subgoals from rollouts, and Subgoal- Conditioned Re- Generratiion to iteratively improve generation by building on subgoals.
We empirically demonstrate the effectiveness of TMPC. TMPC achieves substantial improve- ments across three distinct domains including long-form response generation, discourse-leve machine translation, and program synthesis, validating its ability to discover and leverage task-specific subgoals.
RELATED WORK
2.1 PREFERENCE ALIGNMENT THROUGH FINE-TUNING
Aligning large language models (LLMs) with human preferences has traditionally relied on post- training strategies. Supervised fine-tuning (SFT) (Ziegler al., 2019) and reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) (e.g.. through Proximal Policy Optimiza- tion (Schulman et al., 2017; Huang et al., 2024)) are widely used but computationally expensive. Direct Preference Optimization (DPO) (Rafailov et al., 2023) simplifies RLHF by converting into supervised learning objective, though it requires managing dual policies. More recent approaches like SimPO (Meng al., 2024) and Contrastive Preference Optimization (CPO) (Xu et al., 2024) reduce memory and resource demands using reference models and contrastive signals. Despite these improvements, fine-tuning methods remain rigid and slow adapt to changing data objectives, posing challenges in dynamic environments.
2.2 TEST-TIME PREFERENCE ALIGNMENT
Test-time preference alignment offers an efficient way to align frozen language models by guiding generation at inference, without requiring any parameter updates. Beyond simple prompting or in-context learning, guided decoding methods harness external signals to control the generation itself. ARGS (Khanov et al., 2024) is representative example that incorporates reward model guidance at the token level, and InferAligner (Wang et al., 2024b) adopts a similar strategy. Among guided decoding methods, there are also approaches that directly modify internal representations. For instance, RE-Control (Kong et al., 2024) trains value function on hidden states using the Bellman equation, and applies gradient-based optimization to align with preferences. TreeBoN (Qiu et al., 2025) and RAIN (Li et al., 2024) leverage tree-based structures: TreeBoN combines tree search with Best-of-N sampling, while RAIN performs self-evaluation without relying on reward model to align preferences. GenARM (Xu et al., 2025) enhances test-time aligmment by introducing an autoregressive reward model that predicts next-token reward signals conditioned on prior context, enabling efficient, token-wise guidance that is theoretically expressive under a KL-regularized RL framework. Test-Time Preference Optimization (TPO) (Li et al., 2025) takes distinct approach, translating reward feedback into textual critiques that serve as language- rewards. The model uses these iteratively refine its oututteffectivlly learning alignment on the fly.
2.3 SUBGOAL-BASED PLANNING
A long line of work has demonstrated that decomposing a complex task into intermediate subgoals can substantially improve long- long-horizon reasoning, exploration efficiency, and credit assignment (Ren et al., 2019). Language-man sorguugge-mdel-bbaee model-baaedd planners (Logeswaran et al., 2022) generate candidate subgoal sequences to reduce search complexity, while hierarchical RI. frameworks for vision-and-lan aad-language navigation (Wang et al., 2025a) discover intrinsic subgoals that guide agents through sparse-reward environments. Across these settings, subgoals act as local waypoints that stabilize planning and make long-range objectives more tractable. TMPC leverages the same intuition. We leverages intermediate waypoints to simplify long-form generation by uniquely identifying subgoals purely in hindsight from model rollouts at test time with frozen LLM, offering single, task- aask-agnottic mechanism across translation, response generation, and program synthesis, unlike prior work relying on explicit environments or learned policies.
3

---

Published as a conference paper at ICLR 2026
3 BACKGROUND
In the general setup of RI. for LLMMs, text generation can be formally modeled as a finite-horizon Markov Decision Process (MDP). We adopt a general notion of step as the basic unit of temporal progression, which can represent token, a segment at various granularities (e.g. phrase, sentence, or paragraph), or other linguistically or structurally meaningful units. Then, an MDP can be defined as M = (S,A, A. P. R. μ. T),, where (i) the state space consists of all possible text prefixes, (ii) the action space A corresponds to the set of all possible generation units, (iii) P denotes the transition function, (iv) R S x A + IR is the reward function that assigns scalar feedback to step-level or trajectory-level outcomes (e-g. measuring fluency, factuality, or alignment with user preferences), (v)
denotes the initial state distribution, and (vi) T E N is the episode length.
We define the initial state so as the initial prompt and let a, denote the partial response generated at step t. At each step t, the current state is the set of tokens from the initial prompt and the partial responses generated up step i, i.e., 8t (sg, 01, 1....... (t-1). Based on this construction, we know that the transition function is deterministic with P(5 ax) = 1. A policy T (a)s), parameterized by the language model, defines probability distribution over actions given the prefix E S. The generation of full text sequence of length T can therefore be viewed as trajectory
(su, du, 8T-1, 01,,, $T)
with the cumulative reward given by J(r) Σ R(st, at).
This perspective enables the application of RL methods to text generation in LLMs. Rather than relying solely maximum likelihood estimation, which optimizes local token-level likelihoods, the MDP formulation allows optimization with respect to long-horizon objectives such as coherence and alignment with human preferences. This provides the foundation for recent advances in preference- based fine-tuning and test-time alignment.
METHODOLOGY
4.1 TEST-TIME ALIGNMENT VIA TRAJECTORY OPTIMIZATION
Our key idea is to take a model- RL viewpoint to achieve test-time alignment for LLMs. Specifically, we propose to recast preference alignment as trajectory optimization and thereby employ receding-horins horizon control for iterative text generation.
Text Generation Optimization as Trajectory Optimization. Usually adopted by the model-based RL literature (Chua et al., 2018; Lowrey et al., 2019), the goal of trajectory optimization is to find an optimal sequence of actions is maximized. This matches the objective of LLM text generation in that the output response is generated to best align with the underlying preference. Recall from Section that we adopt a general notion of a step as the basic unit of temporal progression, which can be a segment at various granularities or other linguistically meaningful units. Again, we let 8o denote the initial prompt -T, CT-1, ST) denote trajectory generated under an action sequence
(ag: - such that the total trajectory-wise reward
and let 7 (so, ao, -0 := (ao, dI..... -1). Given an initial prompt so, the search for an optimal sequence a' (sn) can be formulated by the following optimization problem
a (5o) = arg max ao
Σ R(s, 1=0
Note that there is need to take expectation in (1) as the state transitions are deterministic given a0:T-1 in MDPs for text generation, as described in Section 3.
Textual Model Predictive Control for Text Generation. In general, direct optimization of (1) requires searching over all possible action sequences of length and is computationally intractable. As predictive planning method, MPC planner approximately solves (I) by iteratively solving local optimization problems (Hansen et al., 2022), instead of globally optimizing the total reward in one pass. Specifically, MPC planner determines the action of each step by estimating the optimal subsequence d on a moving horizon H (usually H is smaller than T), given the state st, i.e.,
H-
a MPC MPC(s) = arg max
Σ A(st,a));
(2)
4

---

Published as a conference paper at ICLR 2026
integra)
Hedight Sudged Herthoatee
Work
Polbett Trdletory
678
Trylettery
Weighted
Dather
Sudgual
Ra Ra-Ganerzion
Rakgod Contioooe RoGoresation
Patrod
377
De inon
Raffer
Figure 2: TMPC adapts the MPPI framework for test-time alignment by introducing two core principles. Hindsight Subgoal Identification: After generating multiple rollouts, the planner's aggregation function G selects subset of locally-optimal actions a @MMPC. This executed plan is retrospectively identified as a high-quality subgoal and stored threshold a. Sobgoal-Conditione and composing subgoals in the buffer B. This allows the planner to iteratively refine the full- horizon plan by building upon the best strategies discovered in previous iterations.
buffer B its utility meets a Re- Re-Geeeration: New rollouts are generated by sampling from
and then select subset of a (s)), denoted by (MPC for execution. In practice, aMPC(s,) can be selected as the first contiguous actions (1 < < H) or as set of non-contiguous actions (Cagienard et al., 2007). As a model-based approach, MPC solves (2) by employing (i) a learned predictive dynamics model and (ii) a proposal action distribution to jointly generate multiple H H-step predictive
rollouts {T( (8) (s ($ - )) and obtain an approximate maximizer based on these K
(2)
rollouts. As widely-used variant of MPC for continuous control, Model Predictive Path Integral (MPPI) (Williams et al., 2015) determines an approximate maximizer by performing a soft, utility- (E exp( J T(T( )a!"
weighted aggregated selection as J(T) denotes the cumulative reward of a rollout t and X >> 0 controls the exploration-exploiatio trade-off. Compared deterministic MPC that selects single maximizer, MPPI yields smoother updates by aggregating multiple high-reward rollouts while still biasing toward higher J.
(0)
/ exp( { IT(())) where
Inspired by MPPI for continuous control, better leverage MPC in text generation (inherently with discrete actions), we propose to define an aggregation function that determines the action sequence by aggregating multiple textual rollouts based on the corresponding cumulative rewards, i.e..
a TMPC (MPC
( - (1
where {T(1) are rollouts starting from s. Then, TMPC can leverage a sequence of non-contiguous actions, denoted by a TMMPC(s), (s), to be selected for actual use in subgoal generation. The detailed construction of G will be specified in Section 4.2.
Notably, TMPC enjoys two salient features that make it a particularly suitable method for test-time alignment of LLMs: (i) No additional model learning or fine-tuning needed: Recall that MPC- like methods typically require a learned dynamics model and a proposal distribution. In TMPC, a dynamics model is already available since in text-generation MDPs, the transition from s; to 8t+1 is known and deterministic under an action a, Moreover, a pre-trained frozen LLM can naturally play the role of a good proposal distribution for generating candidate texts. Hence, TMPC does not require any fine-tuning or model learning. (ii) Addressing curse of horizon and curse of dimensionality: TMPC addresses these two fundamental issues by iteratively solving local optimization problems. Concretely, TMPC evaluates rewards at the subgoal level, which shortens the effecttve credit horizon, and uses the subgoal buffer B to constrain search to high-reward regions, thereby reducing the size of the action space explored in each iteration. Compared to guided decoding and full-ret full-response iterative refinement, the design of TMPC can achieve a better balance between accurate credit assignment and the size of search space.
5

---

Published as a conference paper at ICLR 2026
4.2 TEXTUAL MODEL PREDICTIVE CONTROL FOR GENERAL TEMPORAL PROGRESSION
In this section, we extend the TMPC framework to text generation tasks with general temporal progression. Inherited from the classic MPC, TMPC described in Section 4.1 presumes that there already exists a basic unit as a discrete time step for planning. This requirement indeed holds for various tasks, such as viewing one output sentence as a step machine translation and text summarization. However, there also cxist text generation tasks without natural boundaries, such as code generation. Despite this, we present a more general version of TMPC that can achieve approximate trajectory optimization in text generation, with and without natural boundaries, by introducing subgoals, which can serve as a basic unit for temporal progression. More specifically, subgoals provide directional guidance for the LLM's generation, enabling efficient exploration toward the optimum. TMPC can be substantiated via two core principles, as illustrated in Figure 2.
Principle I: Hindsight Subgoal Identification. To achieve higher-quality generation, construct meaningful subgoals from continuous text by aggregating prior high-reward actions into a buffer B. This identification occurs after rollouts evaluated, hence hindsight, the planner discovers what constitutes successful step based on empirical outcomes. The update rule of the buffer is as follows:
if B < capacity, B B| < R(s, d(s,d() otherwise, for each a' E @.TMPC
B (M)
Principle 2: Subgool-Conddtioned Aggregation Function for Re-Generation. TMPC, the non-contiguous actions are generated from the following aggregation function:
T-TMPC (s) + 9 (( a,
(0) J R(.) BB = {a R(s, a) 2 a and a E (Te (K si=1s:
(5)
where (T ===1 are the rollouts generated from
LLM <(,, B). (.MPC(s) im- plicitly favors higher-reward outcomes by exploiting subgoals that serve as local optimizers over planning iterations, making validated and locally optimal action sequence with high utility.
This principle describes how TMPC leverages identified subgoals to refine the entire trajectory over multiple iterations. A single pass of optimization may yield suboptimal solution. TMPC overcomes this by performing planning iteratively. In the subgoal identification step, the planner populates the subgoal buffer B using the Hindsight Subgoal Identification described above. The re-generation step constructs new rollouts by explicitly leveraging the high-reward goals accumulated in B as conditioning signals. Rather than exploring from generic proposal distribution, the planner is encouraged to generate new candidate trajectories by composing and extending the high-quality subgoals from the buffer. The aggregation function ς thus plays crucial role: it not only selects high-reward action subset a!TMPC for the current iteration but also leverages the subgoal buffer B to inform the generation of rollouts for the next iteration. This iterative process allows TMPC to escape poor local optima and progressively construct globally high-utility response by combining the best building blocks (subgoals) discovered across all iterations.
At conceptual level, Principle 1 specifies which previously generated segments are treated as subgoals, while Principle determines how a new full trajectory is planned around them. In combination, they induce receding- horizon control loop over text: each iteration designates part of the trajectory as exccuted in hindsight and re-optimizes the remaining plan conditioned on these commitments. This mirrors the classical MPC structure of repeatedly solving finite-horizon problems while executing only part of the solution, and these two principles are what enable this structure to operate effectively for LLM test-time alignment.
EXPERIMENTS
We evaluate TMPC on three tasks with different structural properties to ensure its generality: (1) Paragraph-Level Machine Translation represents a a task with natural boundaries. The generated translation can be precisely aligned with the source text, allowing for sentence-level segments that are structurally anchored and easy to evaluate. (2) Long-Form Response Generation represents a task without natural boundaries. Without source for direct alignment, responses segmented

---

Published as a conference paper at ICLR 2026
by content into coherent chunks (e.g.. groups of sentences), each preserving semantic integrity. (3) Program Synthesis challenges conventional segmentation, representing a task where structural boundaries (e.g.. Abstract Syntax Tree nodes) are semantically too fragmented for effective planning. Our framework addresses this by defining a segment abstractly through a functional milestone: the successful resolution of single unit test.
5.1 PREFERENCE DATASET AND REWARD MODEL
Paragraph-Level MT Dataset. To construct suitable preference dataset for long-text MT, use the 2M Discourse-Level Literary Translation benchmark (Wang et al., 2024a) for our experiments. The available language pairs include: Chinese - English, Chinese - German, and Chinese + Russian. To fit within LLM context windows, each instance is segmented into up to 1024 tokens using GPT-4's tokenizer, ensuring paragraph-level MT remains within model limits.
The preference dataset is derived from the training set of the dataset. Each instance is segmented into paragraphs up to 1,024 tokens. From each translation direction, we sample 2,000 paragraphs, resulting in a total of 6,000 paragraphs for constructing the preference dataset. Translation outputs Gemma-2-99 2-BB, and GPT-4o. The translations are then evaluated with MetricX-24-XX X-2-XXL (Juraska et al., 2024) under the reference- freee evaluation mode, where no reference translation is supplied as input. Following the procedure CPO (Xu et al., 2024), we assign the translation with the highest score as the chosen response, the one with the lowest score as the re rejected response, and discard the middll-scoring translation. The resulting reward model achieves 88.53% validation accuracy. Further details the formation of preference data can be found in Appendix D, and detail of training can be found in Appendix G.2.
are generated using LLaMA-3.1 1......8---snuuuee
Forg-Form Response Dataset. We use the Bahoas/full-hhhrlhi dataset, which is widely adopted for LLM alignment. This dataset is designed to improve Al assistant behavior in terms of helpfulness and harmlessness. Fach sample consists of prompt and responses, with one labeled as preferred based on human judgments. Since the response lengths in the dataset vary significantly, we select samples based on the length of the chosen responses. Specifically, we construct the training set using the top 6K samples with the longest chosen responses from training set, and using the top 1024 longest chosen responses from the testing set to construct test set. We use the 6k size training set to train reward model, which achieves validation accuracy of 83.78%.
Program Synthesis Dataset. We evaluate performance on the official testing set of the Mostly Basic Python Programming (MBPP) dataset (Austin et al., 2021), which comprises 500 problems (Task IDs 11-510). Code generation offers a direct reward signal. The resulting pass rate serves as the direct reward signal, eliminating the need for separate reward model.
5.2 EVALUATION METRICS
Paragraph-Level MT. We use SEGALE (Wang et al., 2025b), a framework that extends existing metrics to long-text translation. Following CPO (Xu et al., 2024), we apply COMET* within the SEGALE framework, thereby extending COMET to the paragraph level. To better capture contextual quality, rather than feeding only source, translation, and reference sentences into COMET, we follow (Vemikks et al., 2022) and incorporate three concatenated sentences as inputs. We refer to this context-augmented version as SEGALEqumet. SEGALE further reports the Null Alignment (NA) Ratio, the proportion of source translation sentences that fail to align, often due to over- or under-translation.
Long-Form Responses. We evaluate response quality using two complementary metrics: Average Reward measures the mean score assigned by the reward model. This reflects the degree of alignment with helpfulness and harmlessness preferences. We introduce this metric to directly test whether TMPC achieves stronger alignment when the reward model and evaluation are consistent. To avoid the potential for "cheating" in reward-based scoring, we also report Win Rate, which captures the proportion of pairwise comparisons in which a model's 's response is preferred over reference response by GPT-4 (OpenAl al., 2024). Following the ARGS evaluation protocol (Khanov et al., 2024), GPT.
"https://huggiin http:://hggggnaaaee co/ hatpaa/oooddatttttt//a//a////l// Linbabel/wmt22-come
7

---

Published as a conference paper at ICLR 2026
4 is prompted to assess overall response quality, considering helpfulness, harmlessness, relevance, accuracy, depth, creativity, and detail. The full evaluation prompt is provided in Appendix F.
Program Synthesis. Following standard practice, directly report the Pass Rate, defined as the proportion of problems for which all associated test cases are passed.
5.3 BASELINES
We evaluate all training-time alignment methods on LLaMA-3.1-8--In-tee and also adopt as the backbone for all test-time alignment methods, including TMPC. Implementation details of TMPC, including parameters and prompt design, are provided in Appendix H.
Test-Time Alignment Methods. We compare TMPC against the following representative approaches. (I) ARGS (Khanov et al., 2024), a token-level decoding method that incorporates reward model guidance during inference. (2) RAIN (Li et al., 2024), which leverages tree-structured self-evaluation without relying on an external reward model. (3) RE-Control (Kong et al., 2024), which modifies internal representations by training value function on hidden states with the Bellman equation and applying gradient-based optimization to align preferences. (4) GenARM (Xu et al., 2025), an approach that trains an autoregressive reward model to assign token-level rewards conditioned on past tokens, and combines these reward scores with next-token probabilities during inference. (5) TPO (Li et al., 2025), which translates reward signals into textual critiques and uses an LLM to provide feedback for iterative refinement. (6) Best-of-N Sampling, widely adopted baseline that generates multiple candidates and selects the highest-scoring one.
To ensure fair comparison, ARGS and RE-Control are equipped with the same reward model as TMPC. RAIN requires neither a reward model nor additional training data. GenARM trains own autoregressive reward model using the same training data employed for TMPC's reward model. For TPO, we set the number of iterations to 4 to ensure it generates no fewer responses than TMPC, although this involves more LLM calls for textual losses and gradients. Further implementation details for all baselines, including a breakdown of TPO's s LLM calls, are provided in Appendix G.I.
Training-Tine Time Alignment Methods. We further compare TMPC with training-time alignment methods. We include supervised fine-tuning (SFT) on the same preference dataset, which often serves strong baseline in translation. In addition, we evaluate SimPO (Meng al., 2024) and DPO (Rafailov et al., 2023), which represent recent and mainstream approaches to preference- training-time alignment, respectively. Details of training procedures are reported in Appendix G.2.
Task-Specific Settings. For paragraph-level MT, we include two high-performance models for additional context: GPTT-0,, which serves strong upper bound despite being specialized for translation (Shahriar et al., 2024), and Qwen-2.5-14B, competitive open-source alternative for Chinese language tasks. For program synthesis, our comparison focuses Best-of-N sampling and TPO. Token-leve guided decoding methods are excluded as functional correctness is holistic property of the entire code sequence, making them ill-suited for this task.
5.4 QUANTITATIVE RESULTS
Results on Paragraph-Leve MT. As shown in Table 1, TMPC achieves the best or second-best performance across all translation directions. It notably surpasses a strong Best-of-60 baseline with a fraction of the computational budget, underscoring the efficiency of predictive planning over naive sampling. For the zh-ten direction, TMPC's performance even exceeds GPT-4o, highlighting its effectiveness on complex alignment tasks. TMPC's success stems from mitigating the failure modes of other paradigms. For instance, TPO exhibits inconsistent performance; while competitive in zh-rru, is prone to factual inconsistencies in zh zh-ten and zh dde, reflected high NA Ratios. Similarly, while RE-Com RE-Control is more stable than myopic methods like ARGS and RAIN, it still underperforms and lacks a strategic refinement mechanism. TMPC inherits the stability of response-leve refinement while avoiding the compounding errors of token-level guidance, striking a more effective balance.
Results on Long-Form Responses. We present the results in Figure 3. TMPC outperforms the strongest training-time (DPO) and test-time (Best-of-20) baselines in head-to-head comparisons judged by GPT-4. The efficiency of TMPC is particularly notable: TMPC requires only 3 iterations with 3 rollouts each, in addition to the initial LLM output, totaling 10 generations. In contrast,
8

---

Published as a conference paper at ICLR 2026
zh
zh de
Methods
Test-Time
SEGALE
Ratio
SEGALEE
Ratio
SEGALE
Ratio
GPT-4o 2024-08-06
94.58
0.10
93.74
0.00
94.54
0.00
Qwen-2.5 (148)
94.43
0.18
90.47
3.08
92.98
1.24
Llama-3.1 (8B)
84.36
10.47
86.28
4.19
88.97
4.43
Llama-3. SUT
93.54
0.34
89.11
1.92
93.47
0.19
3.ama-3.1suPPe
1.66
84.56
2.53
0.00
Llama-3.
90.23
1.33
82.15
6.62
93.48
0.00
Llama-3 LRGS
63.99
31.53
43.03
32,96
51.97
40.01
Llama-3.1 RAIN
37.18
27.79
27.15
Llama-3. IRE-Comml
86.39
7.06
84.97
5.83
87.16
5.96
Llama-3. IGMARM
61.18
34.73
55.67
39.52
60.96
34.58
Llama-3 3. 1pp
88.81
5.63
92.63
0.67
87.67
6.79
IJama-3. Dest of
90.97
3.58
84.86
3.89
82.74
10.78
Llarma-, 1. TMPC
0.00
1.19
2.40
Table I: Results the WM literary translation shared task (zh-xxx directions). Results are grouped into SoTA and base models, training-time alignment methods, and test-time alignment methods. For test-time methods, the best-performing results are bold, and the second-ber underlined. Proposed methods highlighted.
Best-of-20 produces twice as many outputs but still underperforms, showing that its advantage stems from TMPC rather than sheer sampling volume.
To ensure a fair comparison, Figure also includes TPO at 4 iterations, whose computational cost is double that of TMPC (20 vs. 10 LLM calls). At 2 iterations, TPO has exactly the same cost TMPC (both use 10 calls). This reveals a consistent trend: TMPC matches or outperforms TPO at equal cost, and TPO must spend roughly twice much compute close the gap, highlighting TMPC's superior sample efficiency. Furthermore, TMPC provides a more stable alignment path than other test-time paradigms. TMPC bypasses fragile textual critiques and mitigates error accumulation by iteratively planning from buffer of validated subgoals.
Reward of Methods
GPT-4 Evalustion
1024 prompts
3.95
Tyme
4.19
4.18
4.60
3.56
2.95
Teat-time
(420)
/125
2:24
1.99
Avg
Oum
25.0% [/90)]
$2.4% (5:0)
3%
Box 20
10.9
LLMMA-3.1-88 1-888
SPT
DPG
SimPo
RAIN
ABOS'
Re-conteel control
Genal RM
TPoliter
Trotieerck (e)
Bost-rl-l Bostal.2
TMPC
0%
30%
40%
60%
100%.
Figure 3: Results on the long-form responses. Left: Average reward across the base model, training- time baselines, and test-time alignment methods. Right: GPT-4 win rate of TMPC against DPO and Best-of-20. All methods use 1---A--.1--B-In-ee as the backbone for fair comparison.
Results on Program Synthesis. As shown in Figure 4, TMPC achieves 61% pass rate, outperforming all baselines. This result highlights the limitations of unstructured approaches. Best-of-N sampling, even with large budget (N 35), is
Pass Rate across Methods
80
61
Pass Rate 086)
60
constrained by the model's initial capabilities and relies on sampling chance. TPO shows only marginal gains with more iterations, reaching pass rate of just 48% after 4 iterations. In contrast, TMPC systematically explores solution pathways by building upon partially correctness. Instead of merely hop- ing for a correct answer, TMPC maximizes the possibility of constructing one, allowing completely solve problems.
50
40
20
Figure 4: The pass rates on MBPP

---

Published as a conference paper at ICLR 2026
(a) Hyperparameter
Reward Model
(c) Threshold
(d) Variants
Setting
Avg.
Setting
Avg.
Setting
Avg.
Setting
Avg.
buf=3, seg=3
4.595
Original RM
4.595
4.469
w/o Principle
4.264
buf=6, sog=3
4.482
Weaker RM
4.332
4.595
Principle 2
4.463
buf=3, sog=6 4.512
Noise (a"=1) 4.457
a =
4.539
Full TMPC
4.595
Table 2: Robustness and sensitivity of TMPC. (a) Robustness to hyperparameter choices, with performance varying by less than 0.1 across different buffer and segment sizes. (b) Robustness to imperfections in the reward model, including injected noise and lower accuracy. (c) Robustness the threshold used for selecting high-reward segments.
(d) Ablation TMPC's two principles. Removing Principle is approximated by disabling hindsight and making the buffer FIFO (First-In-First-One removing Principle 2 is approximated by minimizing subgoal conditioning (buffer size = 1).
5.5 ROBUSTNESS AND SENSITIVITY ANALYSIS
Table 2 illustrates TMPC's robustness and sensitivity on long-form responses (more numerical results are in Appendix 1. As shown in Table 2a), the framework is insensitive to its core hyperparameter choices; variations in buffer and segment size alter the average reward by less than 0.1 points, with performance consistently remaining superior to other test-time alignment methods. Table 2b) further tests the framework's 's robustness to reward model quality. Using a weaker reward model has limited negative impact despite disturbing the optimization direction, while injected reward noise has a much smaller effect. We employ GRM (Yang et al., 2024) as the weaker RM, using the Ray2333/GRM, 1aRma_ 1 1_8B_ @et_reawrdmddeeel-e checkpoint, which achieves 77.54% validation accuracy. This resilience to noise stems from TMPC's subgoal buffer, which progressively filters out low-quality subgoals. Table 2(c) evaluates sensitivity to the selection threshold Lowering to O admits low-quality segments early on, causing a slight drop, while increasing a to 5 restricts how easy segments are selected, reducing diversity and partially collapsing TMPC toward Best-of-N. Nevertheless, TMPC remains stable across wide range of thresholds because stronger segments eventually overwrite weaker ones in the buffer. Table 2(d) reports ablations of TMPC's two core principles. Removing Principle is approximated by disabling hindsight and forcing the buffer into First-In-Firs--O00 mode, which leads to a sharp drop because subgoals are no longer ranked by quality, pushing the optimization in the wrong direction. Removing Principle 2 is approximated by minimizing subgoal conditioning (buffer size = 1); we do not remove entirely, since that would collapse the method into Best-of-N, but
this weakened form still produces degradation.
For paragraph- MT, we analyze the zh- -en direc- tion to reduce confounds from the base model's famil- iarity with specific languages. Figure 5 reports iteration- wise performance. The early-iteration gains suggest that TMPC progressively strengthens its planning signal as the subgoal buffer have more high-reward segments. The results show that TMPC performance steadily improves up to three iteration, after which extra iterations lead to slight decline. In contrast, reducing TMPC naive it- erative refinement (buf=1, seg=1) yields no initial gains and fails to improve with more iterations, highlighting the importance of TMPC's two principles.
Performance across Itteationns (zhmen)
$4.13
03.95
$9.76
35.00
$4.53
10.52
[Buf=3, sog-3)
Refinement
Figure 5: Translation zh >en.
6 CONCLUSION
We introduced TMPC, test-time predictive planning framework for preference alignment. Under the sequential decision-making view, existing methods suffer from two fundamental limitations: guided decoding operates at the token level and faces the curse of horizon, while iterative refinement operates at the response level and suffers from the curse of dimensionality. TMPC strikes balance by identifying locally-optimal trajectory segments subgoals in hindsight, and then leveraging buffer of these subgoals to iteratively refine the full-horizon plan. This design mitigates both challenges and enables consistent improvements in long-form alignment without modifying the model parameters.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is based upon work partially supported by NVIDIA grant and technical involvement from NVIDIA Taiwan AI R&D Center (TRDC), This research is also partially supported by the National Science and Technology Council (NSTC) of Taiwan under Grant Numbers I 114-2628-E- A49-002 and 1-644---449-02- performance Computing (NCHC) for providing computational and storage resources. This research utilized H100 GPU computing resources generously donated by Wistron Corporation.
The authors also thank the National Center for High-
ETHICS STATEMENT
This work introduces Textual Model Predictive Control (TMPC), a general framework for the test-time alignment of large language models. Our primary goal is to develop more stable and efficient methods for aligning models with beneficial human preferences, such as helpfulness and harmlessness. All experiments were conducted on publicly available and widely used academic benchmarks (HH-RLHF, 24 and MBPP), and no new data involving human subjects was collected.
The primary ethical consideration of our work is that the alignment outcome is determined by the provided reward signal. While we have used for positive alignment, a malicious or biased reward signal could steer model toward generating harmful, unfair, toxic content. TMPC, like other alignment techniques, could potentially amplify biases present in the preference data used to train the reward model. We therefore stress the importance of careful design, auditing, and red-teaming of reward models before deploying systems using this technology sensitive, real-world applications. We believe that by providing more transparent and controllable test-time alignment mechanism, our work can contribute positively to the development of safer AI systems.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. Our implementation of TMPC, along with all experimental scripts, will be made publicly available in permissively licensed open-source repository upon publication.
The core methodology is described in Section 4, with detailed, task-agnostic algorithm provided in Algorithm 1. All datasets used in our Spprimennt--HH-RLHH WMT'24, and MBPP-are public benchmarks, with details on their specific versions and preprocessing steps provided in Section and Appendix G. All hyperparameters, prompt templates, and sasc-spec implementation details necessary to replicate our results for long-form response generation, machine translation, and programmatic synthesis are documented in Appendix H. We believe these resources provide clear and sufficient basis for the community to reproduce and build upon our findings.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. URL https::/arxiv.or / httppdff//333 /2003.2333.22 230..08774 08774.
Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. Tower: An Open Multilingual Large Language Model for Coanslation-ellatee Tasks. In First Conference on Language Modeling, 2024. URL https: //openreview.ner wwwp::/wwwweeeeneeieieeeea iwt.rrnnnniittsss
Jacob Austin, Augustus Odena, Maxwell I. Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732. 2021. URL https://arxxv.. http://aaaaiv..ogas///////// 07732.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan-
11

---

Published as a conference paper at ICLR 2026
guage Models are Few-shot Learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020. URL hnttps://pappeet nips. httpsscccppper..llle/papeeee/e/e (1570dd6ffcb49774 8bbbbacc 42f64a- pdd.
Raphael Cagienard, Pascal Grieder, Eric C Kerrigan, and Manfred Morari. Move blocking strategies in receding horizon control. Journal of Process Control, 17(6):63--570, 2007. URL https:
//www. www.ssccencdiieeeeee cww/science/ httpi//ee//erreeee///////////000
E. Camacho and C. Bordons. Constrained Model Predictive Control, pp. 177-216. Springer London, London, 2007. ISBN 978-0-85729- doi: 10.10077978---88522-3338 URL
https://doi. org/10. / 1--0 8572--89--2779-777777
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. Advances in neural
information processing systems, 31, 2018. URL https::/proceedding htwp:occeeeiiinsneeiiiis.i.. wwurips.cc/papp...
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal Difference Learning for Model Predictive Control. In International Conference on Machine Learning, pp. 8887-8406, 2022. URL https: pdff.
//proceedings. mlr. hww.:/rreee///l////nnnnn////a/a
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. International Conference on Learning Representations, 2021. URL https://opennreview. net net/forumiiddd7K3 3GmQ.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022. URI. https: http:///ppeeeveeeeee http://erreeeeeeeee/oor///m//// forrmid nZeVKeeFYf9.
Nai-Chieh Huang, Ping-Chun Hsieh, Kuo-Hao Ho, and I-Chen Wu. PPO-clip attains global optimality: Towards deeper understandings clipping. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12600-12607, 2024. URL https: //jjs aaai wwo//onderrr /php/ MML/artticle/ieew22111.
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24 -24: The Google Submission to the WMT 2024 Metrics Shared Task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference Machine Translation, pp. 492-504, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2224.val 10000244.ttt110 1.35. URL https: http://alaaaattooogoo/oo or 022/2024. wmt-1 1.35/
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as Reward- Search. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview net mett/formmidd-sggeeeeeee
Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: control perspective. In NeurIPS, 2024. URL http: //pppr/s//ppp./pppppppppp nips. cc ac/paper__p /tppr_filees html.
Basil Kouvaritakis and Mark Cannon. Model Predictive Control. Switzerland: Springer Inter- national Publishing, 38:13-56, 2016. URL https::/linn hww.://..r...ge....ooottttttttto wwwp///oneet///d////// 1007/978-3- 3-1778-3331----1 10099-2483-- pdf.
Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. In Forly-second International Conference Machine Learning, 2025. URL https::/ http:////ppppeeeeeeeeeee//o//oaa FAHEEE.
Yuhui Li. Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your Language
Models Can Align Themselves without Finetuning. In The Tweifth International Conference on Learning Representations, 2024. URL http:p////pppeeeeeeeeeeeeeoeeeee
12

---

Published as a conference paper at ICLR 2026
Bill Yuchen Lin, Abhilasha Ravichander. Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. In The Twelfth International Conference on Learning Representations. 2024. URL /ottps://operree eview. net/ fort/ wwwrrrummmdwwddooo wid==xxeeewxdaa....
I Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. Few-shot subgoal planning with language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5493-5506, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1t.1186611/222222 URL htttp::/aallanthllogg org wwwg/2222.or naacl maain.aa.n222..
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. International Conference on Learning Representations, 2019. URL http:////ppprreeeeeeeefffottetee Bkg6RiCqY7.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model--Bssd Control. In Interna- tional Conference on Learning Representations, 2019. URL id-Byey7n85FQ.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization With Reference-Free Reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /ttp:////ppeeeiiew... wttpeerreiiii.eeeeooumrttteeeeee
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman. Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Semaadet-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu. Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Auty Bleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo. Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Rejichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael Pokorny. Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel
13

---

Published as a conference paper at ICLR 2026
Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv arrg/bbs/303.000 08774.
Long Ouyang, Jeffrey Wu. Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Fol- low Instructions with Human Feedback. In Advances in Neural Information Processing wwweerriiilee/pee//222....
Systems, 2022. URL https://proceedd/e merreeeeeiiinn.eeiiiisssss b1efdde5bbe3647739144 FF pdf.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: http:///eeeeeeeeeeeeeeee net meet/forum?dd-hguuupul
Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Chenhao Zhu, Xinzhe Juan, Ling Yang, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. TreeBoN: Enhancing inference- time alignment with speculative tree-search and best-of-n sampling. In Christos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20894 20917. Suzhou, China, November 2025. doi: 10.186530 12.18855/112225. findings-emnlp.I 140. URL https: hwwps///hlaaaaaao.....o..oo org/ /225. (arsfffnniii findings-emnlp... 1 1140/.
Association for Computational Linguistics. ISBN 979-8-891 8917--87117--35-7 1400.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In seirth-seventh Conference on Neural Information Processing Systems, 2023. URL https: wtwprrrevweeeeeeooountttttettee
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13464-13474, 2019. URL https: /ttp://edooc........ neurips. cc/paper/201
/201
53aese221555544400 53 a html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:1707 2017. URL https://arxxv.orrg
abs/1707.06344
Sakib Shahriar, Brady D. Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Computing Research Repository, abs/2407.092 09519, 2024. URL https: / arxiv. org/pdf 2207009075
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta,
14

---

Published as a conference paper at ICLR 2026
Anna Gottardi, et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http://pwrrreeeeeeeeeeeooo/////a
Featured Certification.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. Advances in Neural Information P Processing Systems, 33:3008-3021, 2020. URL https: httttpp://prooeeeiingg.. pdf.
neurips. . ctt/////000000111111110212222209
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving Open Language Models a Practical Size. arXiv preprint arXiv:2408.000 2024. URL https://arxisn wwwpaaawww..oooooo//oo//a///p 020///00000011118 00118.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 13971, 2023a. URL
https://arxxv.orgg 22302. 13977.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open Foundation and Fine-tuned Chat Models. arXiv preprint 23XXi:2307.09288,, 2023b. URL https://arxiv.
org/pdf 22077 09288.
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into Document-Level Metric. Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Mattco Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference Machine Translation (WMT), pp. 118-128, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https: aacass//ooll org/ 22w..2222o22 wmt-1.6/
Jiawei Wang, Teng Wang, Lele Xu, Zichen He, and Changyin Sun. Discovering intrinsic subgoals for vision- and-language navigation via hierarchical reinforcement learning. IEEE Trans. Neural Networks Learn. Syst., 36(4):6516-6522 2025a. doi: J0.110//NNNIS https://doi, http:://doi.or//..... TTT.9//NTNS...O 2024. 3398300.
URL
Kuang-Dr Da Wang, Shuoyang Ding. Chao-Han Huck Yang. Ping-Chun Hsich, Wen-Chil Peng, Vitaly Lavrukhin, and Boris Ginsburg. Extending automatic machine translation evaluation to book-length documents. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, November 2025b. URL https: http///xxvv..rr...... abs 2509. 17249.
Longyue Wang, Siyou Liu, Chenyang Lyu, Wenxiang Jiao, Xing Wang. Jiahao Xu, Zhaopeng Tu, Yan Gu, Weiyu Chen, Minghao Wu, Liting Zhou, Philipp Koehn, Andy Way, and Yulin Yuan. Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 699-700. Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2000 201244.mtt111 1.58. URL https: http///aaaaaatloo////// org 2024. wmt- 58/.
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, and Xipeng Qiu. InferAligner: Inference-Time Alignment for Harmlessness through Cross- Model Guidance. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10460-10479,
Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 18653//1/0224cmmppmmiii
URL https: http///aaaaaatlloooeee/e
org/2024 emnlp-main 585/.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. CoRR, 2015. URL http:/ http://arxiv.or/ abs/1509. 01149.
/ttp://.xxxv/////////
15

---

Published as a conference paper at ICLR 2026
Qiyu Wu, Masaaki Nagata, Zhongtao Miao, and Yoshimasa Tsuruoka. Word Alignment Prefer- ence for Machine Translation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. Linguistics. org//024....// http//22222m....... emnlp-ma main..
3223-3239, Miami, Florida, USA, November 2024, Association for Computational I doi: 10. 10.16655/1/2022eemlp--mm 188/.
URL http::ttt///lla//
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. In Forty-first International Conference Machine Learning, 2024, URL https: https://oppnrevieww http:///pppppeeeeeeeee/eerrrrrrs www.:iiiiidioippipp iwkioZpn.
Yuancheng Xu, Udari Madhushani Schwag, Alec Koppel. Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025. URL htttps://openeviiiie www.rreiieieneooouniiiedeee J0qTpmbSbh.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttss://ppenrvieeee net/ffrummidd jwh9MHEfmY.
Jifan Yu, Xiaozii Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lx, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Kaifeng Yun, Linlu GONG, Nianyi Lin, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi. Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Xu Bin, Jie Tang, and Juanzi Li. KoLA: Carefully Benchmarking World Knowledge of Large Language Models. The Twelfth International Conference on Learning Representations, 2024. URL https: / wpwpreveew.eew.eeeeeep/p. net/ net/forummid=AgW23orrs
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamalactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024. URL https: // wcw.aaoloatttoooyyyooooooo 0rgg/024..00 ac1- 38. pdf.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Lingmistics: NAACL 2024, pp. 2299-23993 2414, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 11613//1/2024, finn24...iiing--aaa naael. 1499. URL https: httts://acllntthology... org/2024. httpg////...niiin...... findings-naacl. 149/
Daniel M. Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine- Fine-Tuning Language Models from Human Preferences. arXiv preprint arxiv: /w..:/9999.088 2019. URL http://xxxv.oogppddf.aa/ 08593.
16

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODELS USAGE
Throughout the preparation of this manuscript and the accompanying code, we utilized Large Language Models (LLMs) as assistive tools. During the writing process, LLMs were employed to improve the clarity and readability of the text by rephrasing sentences and refining wording, For software development, we used code editor with generative functionalities. This was primarily used to generate Python scripts for creating the figures presented in this paper, based on high-level descriptions of the desired plots. All core research ideas, experimental design, data analysis, and scientific conclusions were exclusively the work of the human authors.
B LIMITATIONS AND FUTURE WORK
While TMPC effectively improves output quality through test-time preference alignment, it is fundamentally constrained by the expressiveness of the underlying language model. It can only steer generation toward outputs already supported by the model's capabilities, and may struggle when the desired outputs lie far from the model's original distribution. future work, we aim to better understand the conditions under which iterative alignment shifts the output distribution effectively. This could enable stronger integration with training-ting time methods, such as lightweight fine-tuning or reward model co-optimization, to improve robustness in harder settings and under distribution shifts.
C BROADER IMPACTS
We anticipate that TMPC will contribute positively to the development of more controllable and aligned language models, particularly in low-resource or safety-critical applications where fine-tuning impractical. By enabling preference-aware generation at test time without modifyyng model weights, our method may lower the barrier to deploying LLMs in real-world scenarios. However, test-time alignment methods also introduce signals are poorly calibrated. To mitigate this, we recommend careful design and auditing reward models, especially in domains involving sensitive subjective outputs.
risks, such reinforcing spurious patterns if reward
D DETAILS OF PREFERENCE DATA
D.1 LONG-FORM RESPONSES
For the long-form response task, we construct a pairwise preference dataset using the Dahoas/full-hh-rll dataset, where each pair consists of chosen and a rejected response. To train the reward model, we select the first 6,000 chosen responses whose lengths are closest to 1,024 tokens. For evaluation, we use the first 1,024 chosen responses under the same criterion, ensuring that test examples also have continuation lengths close to 1,024 tokens. This setup provides both controlled training signal for the reward model and consistent benchmark for evaluating long-fom generation.
D.2 PARAGRAPII-LEVEL MT
For paragraph-Level machine translation task, we used MetricX-24-XL to labeled our pairwise preference datasets, the Chinese sources sentences are from WM Discourse-Leve Literary Translation benchmark, we employed generate translations across three language pairs; each language pair has 2000 records, with maximum length of 1024 tokens. The distribution of preferences, indicating the number of translation the best translation among the three, three, indicating the number of translation is the best translation among the three, given that MetricX-24 scores range from 0 to 25, where 0 is the best and 25 is the worst. We removed translations scoring above 20, and two out of three translations in a paragraph exceeded this threshold, we did not that paragraph. The details are in Table 3.
Gemma-2-9RR and GPT-4o to
17

---

Published as a conference paper at ICLR 2026
LLaMA Wins
Gemma Wins
GPT4 Wins
zh-
310
421
1269
zh-
82
1814
zh rr
32
127
1680
Table 3: The statistics of winning translations for each language pairs evaluated by MetricX-24.
E SUPPLEMENTARY RESULTS
E.1 ROBUSTNESS AND SENSITIVITY ANALYSIS
To complement the iteration-wise trends shown in Table 2, we report the full numerical results for robustness and sensitivity analysis in Table The analysis covers variations in hyperparameters (buffer size and segment length), reward model fidelity, injected noise, and number of iterations. These results provide more detailed view of how TMPC behaves under different conditions.
Category
Setting
Iter=I
Iter=2
Iter=3
default (buf=3, seg=3)
4.359
4.533
4.595
buf=6, seg=3
4.293
4,438
4,482
Hlperrparameters
buf=3, seg=6
4.317
4,463
4.512
original
4.359
4.533
4,595
RM Variants
weaker RM (GRM)
4.144
4.272
4.332
add noise (o² 1)
4.295
4.443
4.457
Table 4: Robustness of TMPC on HH-RLHF-RLHH (3 iterations). We vary buffer size, segment length, reward model quality, and injected noise. Performance converges around three iterations and remains stable under noisy or weaker supervision.
E.2 COMPARISON TO
HEURISTICS
System
zh-ten
zh->de
zh-tru
Fower-7B (sentence-level, concatenated)
92.8
91.1
92.5
LLaMA3. 1-B-1-88B-nnstuee Tixed-Boundas Heuristics
92.7
91.0
90.4
L.LaMA31-8B-Instrer w/ TMPC
94.6
91.7
91.5
Table 5: Comparison with fixed-boundary heuristics on WMT'24 paragraph-level MT. Sentence-level systems translate rewrite each sentence independently with fixed boundaries. TMPC dynamically identifies hindsight subgoals at test time, which may cross sentence boundaries.
Experimental setup. To contextualize TMPC against houndbruddrry heuristic method. The first is Tower-7B operating in sentence-by-sentooe translation mode: the source paragraph is segmented using spaCy, each sentence is translated independently, and the results are concatenated. The second baseline uses the same backbone TMPC (LLaMA-3.1 1.1------1--B)))) but performs sentence-leve rewriting: the model first produces full paragraph translation, then rewrites each sentence independently using fixed boundaries as subgoals, without hindsight subgoal identification cross-sentence planning.
Results and analysis. Table 5 reports results across WMT'24 paragraph-level MT. TMPC achieves the strongest performance in the zh-ten direction, where the backbone model exhibits greater linguistic familiarity; in these settings, dynamic subgoal identification allows TMPC restruc- ture multi-sentence units for improved coherence, giving a clear advantage over sentence-level translation and rewriting. TMPC also outperforms both baselines in zh dh.
TMPC discovers hindsight subgoals from model rollouts that may be sub-sentence, multi-sentence. cross-boundary spans. This flexibility enables the planner work with semantically meaningful
18

---

Published as a conference paper at ICLR 2026
chunks rather than rigid sentence splits, and to regenerate the entire paragraph conditioned these validated subgoals. As a result, TMPC can refine paragraph-level structure holistically instead of enforcing locality constraints imposed by fixed segmentation.
For zh-tru, Tower-7B attains slightly higher scores, which we attribute to the weaker Russian proficiency of the LLaMA-3.1 backbone: limited base-model capability constrains the headroom available for test-time refinement. Nevertheless, TMPC remains competitive and noticeably improves upon the fixed-boundary LLaMA translation.
E.3 COMPUTATIONAL COST
Table 6 reports the end-to-end wall-clock latency and throughput for generating single HH-RLHF long-form response.
Method
Latency (s response)
Throughput (responses min)
Base LLaMA-3.1-8B
7.50
RE-Control
40
1.50
ARGS
363
0.17
RAIN
1930
0.03
GenARM (H200 GPU)
3.75
TPO (2 iterations)
90
0,66
TPO (4 iterations)
148
0.40
TMPC (3 iterations)
108
0.55
Table 6: End-to-end latency and throughput for generating single HH-RLHF long-form response.
TMPC demonstrates a computational cost at 108 seconds. This time can be substantially reduced as the K candidate rollouts highly parallelizable, given sufficient hardware memory. In contrast, methods like ARGS (363s) and RAIN (1930s) are significantly slower because they require fully decoding responses be fed into an external reward model or the LLM itself for evaluation. The faster methods, RE-Control (40s) and GenARM (16s), achieve their efficiency by using their own integrated reward or value functions, which allows for more direct guidance during generation.
TPO exhibits similar behavior to TMPC: with 2 iterations it requires 90 seconds, and with 4 iterations increases to 148 seconds. Importantly, running TPO for iterations requires two A6000 GPUs to avoid out-of-memory errors. At equal compute budgets, TMPC reaches stronger alignment quality with fewer iterations, which contributes its favorable perforannr tradeoff.
It is important to note, however, that for our experiments, GenARM was executed on NVIDIA H200 Tensor Core GPU, while all other methods were benchmarked on an NVIDIA RTX 6000 Ada Generation GPU. This necessary because GenARM's 's memory footprint is too large for the A6000, which would have restricted its batch size and degraded its performance. The difference in hardware is primary contributor to GenARM's exceptionally low reported inference time.
F EVALUATION PROMPT OF GPT-4
Figure 6 is the prompt template for the GPT-4 evaluation, we follow the ARGS evaluation setup (Khanov et al., 2024), with slight modification:Our prompt explicitly instructs GPT-4 to consider how well each assistant addresses the user's question, rather than to evaluate the response in isolation. The prompt asks GPT-4 to rate responses based on overall performance, considering criteria such helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail
19

---

Published as a conference paper at ICLR 2026
SYSTEM PROMPT "! [Systen]
You are he-pful, harm_esa, and precise assistant for checking the quality of the answer. would like request your feedback the performance of asaistants in response to user question. Please rate the harmlessness, helpfulnsss, and level of detail of their responses.
Your evaluation should consider ractors such the helptuiness, harmlesaness, relevance, accurocy, depth, creativity, and level of detail of the response, You should base your evalusticn
user'a a question each assistant addresses the question.
provided responses, censidering well
Note chat it reaponse appears out oft at the end due to length constraints, this ahould nagotively impect the
Please firat output a single Line containing only two valuea indicating the scorea for Ansissant ard Assistant respectivaly. The subsequent lize, please provide comprehensive explanaticn of your evalustion, avoiding any patentisl biag and ensuring that the order in which the responses vere presented does not affect your judgment.
separated space. In the
USER PROMPT """ [Quastion] (question]
(The Assistant Answez) [anawerl]
(The Bad
I's Ansver]
(The Start of Assistant 2 Answer]
(anawer2)
(The Bad of Assistant 2 Answez]""" "M*
Figure 6: System prompt and user prompt of GPT-4 evaluation.
G IMPLEMENTATION DETAILS OF BASELINES
G.1 TEST-TIME ALIGNMENT METHODS
G.1.1 ARGS
We follow the setting of ARGS, We adopt the ARGS-greedy method in ARGS as our baseline. Following the setting of ARGS-greedy. We set 1.5 and 10. For fairness, replaced the backbone model with the same LLaMA3.1 -88--nsttrct as TMPC, and the reward model was also replaced with the reward model used by TMPC. Although ARGS settings indicate that using ARGS-greedy results in answers more closely aligned with the characteristics specified in the reward model, ARGS uses the weighted sum of logit of the token and the reward for token generation. Given that the number of tokens generated by ARGS ARGS-greedy does not exceed those produced by TMPC and RAIN, we included ARGS-stochastic for comparison and conducted best-of-n to optimize results, the choice of n was determined based on the average number of tokens required by TMPC and RAIN to generate a single translation.
However, ARGS-stochasiia 's best-of-n did not surpass ARGS-greedy in performance, leading to ultimately select ARGS -reedy as the baseline.
G.1.2 RAIN
In RAIN, we also replaced the backbone model with LLaMA3.J-8B-mm evaluation prompt in RAIN with text in the Figure 7. For parameters in RAIN, we set value threshold to 0.8 as the default setting of RAIN. We try four combinations of maximum and minimum number of search iterations for finding the parameter that generates the required number of tokens close to TMPC, which are [10,20]. The detailed configuration of the tokens generated in each parameter pairs in Table 7.
and replaced the self-
20

---

Published as a conference paper at ICLR 2026
(MinT.MaXT)
AVG. tokens
[6.12]
6575
[8,
8972
[10,20]
13401
[12,24]
17291
Table 7: The average number of tokens required to generate translation for RAIN in each maximum and minimum number of search iterations pairs.
(INST]
Consider the following source text (Source) and its translation (Translation). Determine if the translation is accurate.
Translations that deviate from the objective mearing of the source text, introduce speculative content, aLter the intended meaning are considered insccurate.
<generated text>
Options:
The translation is accurate.
(B) translation is Lnacourate. (/INST evaluation is: (
[TNST]
Consider the following source text (Source) and its translation (Translation). Determine if the translation is accurate.
Translations that deviate from the objective mearing of the source text, Introduce speculative content, or aiter the intended meaning are considered inaccurate.
<generated text>
Oytions:
(A) The translation is Inaccurate.
(B) The translation is accurate. 
 The evaluation is:
Figure 7: Prompt templates for RAIN.
G.1.3 GENARM
In GenARM, we utilize preference dataset to train an autoregressive reward model, which can predict the next token reward and support the LLM generation process to make the response more preferable. In our implementation, we used LLaMA3.1-80 1-B--.t--B-tntttu Instruct as our backbone model. For each task, the dataset includes both chosen and rejected responses. For example, in the translation task, for each source text, we have the corresponding chosen translation and the rejected translation. Then, we treat chosen response and rejected response responses with preference signals for training an autoregressive reward model.
G.1.4 RE-CONTROL
In RE-Control, we utilize the reward obtained from a reward model to train a value function for test-time alignment. In our implementation, throughout the training process and used our own reward models to generate reward labels to train the value function. The performance of RE-Control heavily depends the reward model since the training process requires the reward signals from a reward model. Sometimes, it is hard to train good value models for some reward models. For example, for the HH-RLHF dataset, given reward model, the value function did not train well under the framework of RE-Control. On the other hand, RE-control performed well with our reward model for the translation task.
used 1L.MA311-8B -n--rstt the LLM backbone
G.1.5 TPO
TPO uses textual optimization to mimic the behavior of numerical optimization (e.g, mimic loss computation and gradient computation) and to try to optimize the reward obtained from a reward model. In TPO, replaced the backbone model with ILaMA3.1 --strrcctctt and replaced the
21

---

Published as a conference paper at ICLR 2026
original reward model with our reward models. For our implementation of TPO, we set the number of iterations to 4 (2 for HH-RLHF because it would cause an out-of-memory issue for HH-RLHF) and the number of samples for each iteration to 3. Since TPO requires textual loss operation and a textual gradient operation, we need additional 2 operations of LLM. So, for each iteration, it requires 2 L.LM operations, and thus it costs x (2 + 3) LLM operations for each problem (2 x (2 + 3) for HH-RLHF). Because of the limitations of the VRAM of our GPUs and for the fairness compare the inference time between TMPC and TPO, we remove the vLLM library from the original implementation of TPO.
G.2 TRAINING-TIM ALIGNMENT METHODS AND REWARD MODEL.
We adopt 1-L.MA..--8B-Insteee as the backbone model for all experiments, including the reward model and training- time alignment methods. Model training is conducted using a single NVIDIA RTX 6000 Ada Generation GPU, with the implementation based on the LLaMA Factory library (Zheng et al., 2024). For training setups, the SFT model is trained solely on the preferred responses from the preference dataset, while the Reward Model, DPO, and SimPO are trained on the full preference data, which includes the input, chosen, and rejected responses, as detailed in Appendix D. All models are trained using identical hyperparameters: the Adam optimizer (Loshchilov Hutter, 2019) with gradient accumulation steps set to 8, sequence cutoff length of 2048 tokens, and maximum gradient norm clipped at 1.0 for stability. Training is performed using 6 precision, with a batch size of 2, for epoch. We apply LoRA (Hu et al., 2022) with rank of 16 and alpha of 128 across all models. The learning rates are configured as follows: 2e-5 for the Reward Model and SFT, and 5e-6 for both DPO and SimPO, in accordance with recommendations from Rafailov al. (2023) and Meng et al. (2024), Additionally, SimPO is trained with a gamma value of 0.4, while DPO uses beta value of 0.1, consistent with the original settings proposed in their respective works.
H IMPLEMENTATION DETAILS OF TMPC
The TMPC framework is operationalized by the iterative planning process detailed in Algorithm 1. The power of TMPC lies how these subgoals are identified and utilized. The UpdateBuffer function implements Hindsight Subgoal Identification. After a rollout, it analyzes the generated response to identify and extract subset of locally-optimal actions TMPC) that meet quality threshold. These validated subsequences are then stored in B as subgoals.
The rollout generation itself, (2x B), implements subgoals in the buffer are not treated as raw text to be merely concatenated, but as operational directives for constructing new prompts. A subgoal prompts the LLM to generate a new, complete response that is coherently anchored by past successes. This approach deliberately avoids generating and stitching disjointed fragments, a common cause of semantic incoherence. The instantiation of these subgoal conditione prompts is tailored to each task's structure:
Conditoood Re-Generation. The
In Paragraph-Level Machine Translation, the prompt reuses validated souree-target sentence pairs from the buffer as few-shot examples to anchor the generation, ensuring contextual consistency.
In Long- Response Generation, the prompt instructs the LLM to synthesize high- scoring ideas and phrases from the buffer into a single, comprehensive, and well-structured response.
In Programmatic Synthesis, the prompt provides validated code snippets from the buffer that passed a subset of unit tests, and asks the model to employ them subgoals to guide exploration toward higher-scoring solutions that satisfy more tests.
H.1 PARAGRAPH-LEVEL MACHINE TRANSLATION
H.1.1 PARAMETER SETTING
For translation tasks, each rollout is segmented into spans of 3 sentences. TMPC runs for 3 iterations, sampling 3 diverse candidate translations at each iteration. A quality threshold of a = 1 is used. To
Phttps:/ //gi thub. com/hhiyouga/LLaM--aaaae Factory
22

---

Published as a conference paper at ICLR 2026
Algorithm Textual Model Predictive Control
Require: Input prompt 2, base LLM evaluation function R, iterations T, rollouts per iteration K.
Ensure:
Final response TT.
1: Initialize subgoal buffer B <- 0
2: Initialize candidate set T <- 0 (Stores all full (response, score) pairs}
3: Generate initial response To +- :7(0))
4: Add (TO, R(To)) to T
(su) based on (5) 5: Compute dg @
6: B + Update Buffer(B,, a
TMPC (so), R( (sMP)(ss))) (Hindsight Subgoal Identification)
7: for 1 to 7 do
8: (Iterative Planning Loop)
9: Generate K rollouts conditioned on the current subgoal buffer
10: {T,
K
Fi=l < GGneeaaeRollouuss(T, S0, B, K)
11:
for each candidate 7,") do
Add (T,6), )) to
12:
II Identify new subgoals from the successful rollout and update the busfer
14: Compute a.MMPC (s0) based on (5)
UpdateBuffer(B, (.MPP((s))
(so). R( so. a: asMM)(ss))))
16: end for
end for
18: Select Tr < arg max+eT R(T)
return FT
ensure increasing reliance on accumulated subgoals, the number of sampled segments from B grows linearly: sampled segments iteration 5.
H.1.2 CONCRETE EXAMPLES SUBGOALS
In real paragraph-level MT tasks, a document typically contains 20-30 sentences. To keep the illustration concise, show only small excerpt from actual example while preserving the original structure and behavior of TMPC.
Consider the Chinese paragraph:
(1) b
(2) エ 12 3
(3) Г B 3
3
first-pass translation (iteration 0) might be:
(1) "To live on wholeheartedly!"
(2) "He wasn't arrogant person."
(3) "But since he had chosen to come, he had to try his best to resist!"
(7)-(7) "Momentarily, his legs resonated with the earth his strength resonated
with his arms!"
TMPC segments the translation into spans corresponding to the original source sentences:
Span A (for source sentence 1): target sentences (1)-(1)
Span B (for source sentence 2): target sentences (2)-(3)
Span C (for source sentence 3): target sentences (4)-(7)
Suppose Span and Span exceed the reward threshold, and Span does not. The buffer is populated as:
00 [1] Span A, [[0 Empty,
[30][3] Span C.
23

---

Published as a conference paper at ICLR 2026
In the next iteration, the model regenerates the entire paragraph while conditioning these subgoals, yielding a revised translation such as:
(1) "To live on wholeheartedly.""
(2) "He wasn't I some fool who didn't it know his limits; since he had chosen to come,
Lü Shu must have had the confidence to retreat safely."
(3) "In an instant, foot joined with ground, leg with waist, waist with arm, and arm
with force."
Here, the model preserves the high-quality subgoals while restructuring the weaker middle section.
H.1.3 PROMPT DESIGN
Three complementary system prompts encourage diverse perspectives: (i) sentence-by translation, (ii) precise and literal translation, and (iii) stylistically enriched translation. Examples of these prompts appear in Figures 9 and 10.
H.2 LONG-FORM RESPONSE GENERATION (HH-RLHF)
H.2.1 PARAMETER SETTING
For long-form response generation, each rollout is segmented into three spans of equal length, measured in sentences. We fix the segment size to 3 sentences and run TMPC for 3 iterations. At every iteration, the planner samples 3 diverse candidate responses using distinct system prompts. A reward threshold of a = 4 is used to select high-quality segments, which are then saved into the inference-time action buffer B.
H.2.2
CONCRETE EXAMPLES OF SUBGOALS
Consider the user prompt:
"What was the first renaissance fair held?"
TMPC generates three candidate rollouts, each split into three segments:
The first Renaissance fair held in the United States is matter of some debate, with multi- ple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963, which was called the "Renaissance Pleasure Faire."
Rollout 1
Seg-1: "The first Renaissance fair held the United States is matter of some debate, with multiple sources citing different events."
"However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963, which was called the "Renaissance Pleasure Faire."
Seg-2:
Seg-3:
empty (Since the response too short)
The first Renaissance fair held the United States matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held Agoura Hills, California, in 1963. This inaugural event was called the "Renaissance Pleasure Faire" and was inspired by the traditional English and Scottish festivals from the 16th century, which were known for their merriment, pageantry, and historical reenactments. The Renaissance Pleasure Faire founded by Phyllis and Ron Patterson, couple who were passionate about history and the arts. The success of the 1963 event helped establish Renaissance fairs staple of American cultural events, influencing similar festivals across the United States.
Rollout
Seg-1: "The first Renaissance fair held in the United States is matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963."
24

---

Published as a conference paper at ICLR 2026
Seg-2:
"This inaugural event called the "Renaissance Pleasure Faire" and was inspired by the traditional English and Scottish festivals from the 16th century, which were known for their merriment, pageantry, and historical reenactments."
Seg-3: "The Renaissance Pleasure Faire was founded by Phyllis and Ron Patterson, couple who were passionate about history and the arts. The success of the 1963 event helped establish Renaissance fairs as staple of American cultural events, influencing similar festivals across the United States."
The first Renaissance fair held the United States a matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963. This inaugural event was called the "Renaissance Pleasure Faire" and was inspired by traditional English and Scottish festivals of the 16th century. The Renaissance Pleasure Faire was founded by Phyllis and Ron Patterson, who drew inspiration from their own interests and the historical festivals they admired.
Rollout 3
Seg-1: "The first Renaissance fair held in the United States is matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963." "
"This inaugural event was called the "Renaissance Pleasure Faire" and was inspired by traditional English and Scottish festivals of the 16th century."
Seg-2:
Seg-3: "The Renaissance Pleasure Faire was founded by Phyllis and Ron Patterson, who drew inspiration from their own interests and the historical festivals they admired."
Resulting Buffer Since all three rollouts contain segments whose rewards exceed the threshold, TMPC stores every segment into the buffer:
B [[1] All three segments from Rollout
B
[ All three segments from Rollout
B 3] All three segments from Rollout 3
In the next iteration, TMPC samples segments such as:
"The first Renaissance fair held in the United States is matter of some debate, with multiple sources citing different events.
Seg-1:
Seg-2: "The first Renaissance fair held in the United States is matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held in Agoura Hills, California, in 1963.
Seg-3: "The Renaissance Pleasure Faire was founded by Phyllis and Ron Patterson, couple who were passionate about history and the arts.
and asks the model to rewrite full answer conditioned on these subgoals.
Given these subgoals, TMPC regenerated the entire answer from scratch:
"The first Renaissance fair held in the United States matter of some debate, with multiple sources citing different events. However, the most widely recognized and celebrated Renaissance fair is the one held Agoura Hills, California, 1963. This inaugural event was called the "Renaissance Pleasure Faire" and was inspired by the traditional English and Scottish festivals from the 16th century, which were known for their merriment, pageantry, and historical reenactments. The Renaissance Pleasure Faire was founded by Phyllis and Ron Patterson, a couple who were passionate about history and the arts. The fair has since become a beloved annual tradition in California, attracting thousands of visitors each year with its blend of history, entertainment, and merriment. It also inspired numerous other Renaissance fairs across the United States, helping popularize immersive historical reenactments and cultural celebrations.
25

---

Published as a conference paper at ICLR 2026
H.2.3 PROMPT DESIGN
To promote diversity in generation, three system prompts: (i) concise and clear, (ii) detailed and informative, and (iii) balanced and well-structured. Figure 11 shows the rollout template, where high-quality segments sampled from treated as actions guiding iterative regeneration.
H.3 PROGRAMMATIC SYNTHESIS
H.3.1 PARAMETER SETTING
We adopt LLaMA-3.. 3.1 3.B-8B---..... SB-Instruct the backbone model. TMPC runs for iterations, with candidate programs sampled at each iteration. candidate is considered promising state if passes least unit test.
H.3.2
CONCRETE EXAMPLES OF SUBGOALS
Consider the task:
"Write Python function to check whether a number can be represented as the difference of two squares."
first rollout might produce:
def is ds_diffeed ofrerrence_oossuurrsstttee
o*"
Checks whether the given number can be represented as difference of two squares or not. и*"
if < 0: return False
# Initialize lower and upper bounds for the possible squares lower
upper = int (n ** 0.5)
while lower <= upper: mid (lower + upper) // square mid ** 2
if square ==
return True
elif square
lower mid + 1
else:
upper mid
return False
which passes of tests. TMPC stores the system prompt responsible for this rollout subgoal.
A later iteration might produce:
def is ofl.ffrrnnneoossuuursss/ss
Checks whether the given number can be represented as di difference of two squares or not. wnn
return False
# Ini Initialize lower and upper bounds for the possible squares
lower upper int(n (n ** 0.5)
26

---

Published as a conference paper at ICLR 2026
while lower <= upper: mid (lower + upper) // square mid ** 2
if square == n:
return True
elif square n:
lower mid
else: upper mid
Check the number can be represented as difference of two squares
for i in r1nge int(n ** 0.5) 1): if (n ** 2) >= and (n i ** 2) 2 == 0 and ((n i ** 2) // 2) ** == (n ** 2):
return True
return False
which passes 2 tests and replaces the previous subgoal.
Here, passing partial tests serves as the operational definition of a subgoal, capturing promising algorithmic structure even when the overall solution is incorrect.
H.3.3 PROMPT DESIGN
We maintain a buffer of promising programs discovered so far and keep track of an incumbent with the highest test score observed. During candidate generation, TMPC conditions each rollout program sampled from this buffer, and instructs the model to rewrite entire solution (i.e., produce single, self-contained program) rather than patching fragments. This stabilizes iterative improvement and avoids the brittleness of segment-level intervention. To encourage exploration and avoid local optima, sample exploration strategies from small prompt portfolio that induces diverse rollouts.
gystem_prrmpt = f"*"
You are a Python expert.
You have a program that passes [passed] out of [total} test cases. Analyze what could cause some tests to fail and produce an improved version. and conittt__oottrints You must only output a single, complete Python code block.
Do not include any explanations surrounding text. u
context_prompt f"n"
Problem: guroblem_descrpptonn
CURRENT BEST ((passed)/(total]] tests passing): cest_progrrm_innbuiieee Write improved version that passes all (total} test cases: wuw
input_messages
("role": "system", "content": system_prompt), ("role": "user", "content": context_prompt)
]
Figure 8: The prompt template in TMPC. We maintain buffer of promising programs and sample one as the conditioning subgoal for the next rollout. The model is prompted to fully rewrite the program, preserving useful structure while improving correctness under sampled exploration angle and format constraints.
27

---

Published as a conference paper at ICLR 2026
processed #H x
7
*
(a blonde-haized girl in white robe walked in, has eyes wide with surprise as she gazed at Rod, who was half Salf-aittin up.
20 T E.
AISR
[Foz scmee reason,
had this scene before, if were familier
ME
scrutint room, door auddenly swung open.
system_ Bactom__roppss-.
"You are a meticulous tranalater. Frovide a literal, word for- translation that preserses the structure and meaning each incividual word.", "You professicnal translator. Deliver clesr, formal, and precise translation that faithfully corveye the originel meaniag.", "You are a creative and expressive tranalator. Render the text in vivid and imaginative way, narrating captivating story."
context_prompt-
context prompt
""ReTow 1s apectalized, internediate translation task, The input text 1s a mix of Chinese and garLial [Ianguage] Lranslalions.
Lexi, Chinese senleaaes ace alzeady Tollowed by preliminary (language: trenslations enclosed in parentheses. These provided tcanslations are rougl fully alignad seferences they may incemplete, inconsistent, translation according to the fol lowing meaning. Your task is to produce an improved language) guidelines:
original
1. *Refinemenl;** For sections with exisLing language) trenslations (in pacenLheses), reline and polish them that they fluent, accurate, and ccherent, fully capturing the meaning of the corresponding Chinese text.
2. ""*Coppletion and naturally in the specified style.
sections
remoin untranalated, translate Chinese accurately
**Trenslation.
Structure Preservasions Maintain original
structure
the text. Every chinese sentence must appear in the same sequence as in the source text, with its sorresponding llanguage) tronslation (if available) insarted immediately after Do "earrange or reorder
part of the text.
1. "Coonsstencyy:** Ensure a uniform Lone and style acruss the entice translation, adhezing to the tsenslator zole specified.
5. **final Output:** Provide the final ousput as single, well-structured language) text. Do not include extraneous commentary, explanations, annctacions, headers output the tronslation in the correct order.
intermediate veratin may later merged with Lranslations. Tocus on clarily, coherence, and fidellly Lo the source Lext.
Note: translation
Here is the impet data for sorclaatioon/ppoceeseeee source \nhn Apply the above guidel ines producs an improved, ccherent translation that strictly fol the original order of the text." "
source
meputgeessages
("role": "ayatem", "content": prompt), ("sola"; "user", "scntent": context context_prompt)
Figure The prompt template used for translation task TMPC and an actual example.
28

---

Published as a conference paper at ICLR 2026
source sentence
# R
M -
mE
# #EAM
* 7
*
t E. 1 EE
Contrary to this, the arm that waa right in front him was slender weak, pale-skinned due to lack of sunlight for long time.
This was his own body in the real world, a fact he was well awore of. how
up injured?
And besides, this doesn't look like hoapital all!
Rod lifted his head to take in the reom, which resembled a cramped compartment, There were no lights, phones, There was only wooden table, two chairs, and cabiret fixed to the wall. familiar
alarm bells.
For some reason, Just Rod acrutinizing the the door suddenly open.
that
scene before, as
rewrite prompt
1"Below is initial tranalation of Chinese text inco (language) This tranalation include emissions, inaccuracies, or awkward phraaing. Your task is to produce a refined veraion that fluent, accurste, coherent, meaning of the original Chinese text. \nin"
faithrul preserving the
4+# Instructions:\a
1. Ensure that every detail the original Chinese text is accurately represented, Corzect any gramnatical errere, unaatural expressions, inconelatencies.
3. Tmprove the natural flow that the translation reads as if written by nstive speaker.\n
oz change any essential details 5. Cucput on.y the tinal retined translation without additional commentary, \n\n snnssourceseentteeee #*# Tritial [language] Translati on: (initial_ translation
add,
#t Original Chinese Text: /ssource sentence: \nin"
** Refined Cransletion:"
input_ messages
("scle": "system", "content" "You are a helpful translater and only output result.";,
("role"s "user", "content"; rewrite_prompt:
Figure 10: The prompt template used for generating final translation in TMPC and actual example.
29

---

Published as a conference paper at ICLR 2026
system_prompt =
"Rewzite the following partial responses single improved more concise and cleat.
"Rewzite the following partial zesponses as single improved ansver that is more detailed and intormative.
"Rowrite the following partial responses single improved that is balanced and well-stzuctured
best_prompt
Human: You'ze over-explaining your answer, but the idea of entertainment is interesting. But feels entertaimment is larger part modern day lives used that used to find entertainment in mere productive things that involved physical exercise or sociallzing.
Assistant:
context_promee
Partial responses:
That's valid point. You're right that the role of entertainment in our 1ives have expanded or changed over time, one ble reason for this is that many people have more leisure and essier access often had to work leng heurs to make ends meet, and lelaure time scarce. Historically, people had limited leisure time and ofter: engaged in physical activities like sports or socializing. However, with the rise of autonation and technological advancementa, many jobs hecome leas physically demanding and time for many peaple. The accessibility and convenience of media have also played significant role its wideepread consumption. The proliferation of streaming services, array
they
past. Historically, people
sedertary, leading
increase in laisure
social media, mobi devices made easier for people entertainment content from anywhere,
time. <Complete it>
Input messages
[rrole:: "ayatem", "content": system pasten_prompt, ("rcle": "user", "content": user prompt context_prompt
Figure 11: The prompt template used for response generation task in TMPC and an actual example.
30


Reviews:
[{"review_id": "dShQntiin8", "reviewer": "Reviewer_UYVg", "summary": "The paper considers the problem of test-time alignment of a fixed large language model (LLM) with a given reward function that encodes externally specified preferences. The paper proposes an iterative improvement method, called Textual Model Predictive Control (TMPC), that combines (i) identification of promising, high-reward subgoals (tokens, sections of text, etc.) within a response with (ii) aggregation of promising subgoals into subsequent prompts. The proposed method is loosely based on the standard model predictive control (MPC) formulation, with the MPC-based formulation of Section 4.1 serving as the point of departure; Principles 1 and 2 of Section 4.2 are the core components of the approach, however, and these are not clearly MPC-based. Experimental evaluation of TPMC against relevant baselines on machine translation, long-form response generation, and program synthesis problems is provided. The experimental evaluation indicates that TMPC outperforms baselines on two-thirds of the machine learning tasks, outperforms all baselines on the long-form response generation task, and achieves higher pass rates than all baselines on the program synthesis tasks.", "questions": "1. In what way is the formulation of the test-time alignment problem as a sequential decision-making problem (lines 97-98) novel?\n2. By what mechanism does the does the design of TMPC \"achieve a better balance between accurate credit assignment and the size of the search space\" (lines 240-241)?\n3. Is it true that Principles 1 and 2 on page 5 are not clearly related to MPC?\n4. Can you elaborate on the fact that GPT-4o outperforms or is highly competitive with TMPC in the experiments presented in Table 1?\n5. Can you comment on the fact that on lines 401-402 it is stated that \"TMPC consistently outperforms all test-time alignment baselines\" on the machine translation tasks, yet Table 1 shows that TMPC is outperformed by TPO on one of the three tasks?\n6. As mentioned in footnote 3 on page 8, on the long-form response task TPO was stopped after two iterations instead of four iterations; does this negatively impact the fairness of the comparison between TPO and TMPC in Figure 3?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The test-time alignment problem has seen immense interest from the community in recent years, so the topic of the paper is timely. The proposed TMPC approach is an intuitively appealing and natural approach to this problem, so the paper is likely of interest to the community. Drawing inspiration from MPC through the formulation given in Section 4.1 provides motivation and technical clarity for the (non-MPC) specifics of the proposed approach described in Section 4.2. The experiments consider three well-known benchmark problems and compare the proposed approach with a good variety of appropriately chosen, well-known baseline methods. The experimental results support the effectiveness of the proposed TMPC approach.", "weaknesses": "1. The connection to MPC feels somewhat overstated. Section 4.1 provides enough context to justify describing the approach as loosely MPC-inspired. However, since the main elements of the method are the non-MPC components outlined in Principles 1 and 2 (Section 4.2), framing as an MPC-based approach may be a bit strong.\n2. The experimental results, though promising, contain some drawbacks that are not adequately discussed. First, though it is claimed on lines 401-402 that \"TMPC consistently outperforms all test-time alignment baselines\" on the machine translation tasks, Table 1 shows that TMPC is in fact outperformed by TPO on the Chinese-to-Russian translation task. In addition, as mentioned in footnote 3 on page 8, on the long-form response task TPO was stopped after two iterations instead of the four iterations used elsewhere due to implementation and/or hardware issues, not problems inherent to TPO. For this reason, the comparison results presented in Figure 3 are incomplete with respect to TPO; it is especially important to have an accurate picture of TPO's performance given that TPO was a top competitor on the machine translation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P0UIYWremX", "reviewer": "Reviewer_1rTe", "summary": "The paper proposes Textual Model Predictive Control (TMPC), a novel framework for aligning large language models (LLMs) with human preferences at test time by casting the generation process as a sequential decision-making problem. TMPC draws inspiration from model predictive control (MPC), incorporating Hindsight Subgoal Identification to automatically detect useful intermediate subgoals and Subgoal-Conditioned Re-Generation to iteratively refine generations. The framework is evaluated on long-form machine translation, long-form response generation, and program synthesis, demonstrating improvements over several training-time and test-time baselines.", "questions": "1. Can the authors provide more formal analysis or empirical ablation regarding the impact and tuning sensitivity of the buffer threshold $\\alpha$ and the aggregation function $\\mathcal{G}$? For example, does increasing $\\alpha$ risk filtering out necessary diversity or causing premature convergence?\n\n2. What explicit strategies does TMPC employ to avoid redundancy or incoherence when assembling non-contiguous subgoals in the generation process? Are there failure cases where the compositionality breaks down?\n\n3. Can you include more thorough ablation on the contributions of Hindsight Subgoal Identification versus Subgoal-Conditioned Re-Generation? A demonstration on at least one benchmark of removing or varying each component in isolation would help clarify this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. TMPC elegantly adapts concepts from control theory (specifically model predictive control) to the LLM test-time alignment setting, innovating with subgoal identification and iterative planning, as detailed in the mathematical framework of Section 4 and illustrated in Figure 2.\n\n2. The methodology is general: TMPC is applied to tasks with both natural and abstract segment boundaries (e.g., machine translation sentences, code unit tests, long-form response chunks), supporting claims about its versatility.\n\n3. Empirical results cover multiple domains using both automatic and large-model-based human-proxy metrics. On WMT'24 translation (Table 1), TMPC outperforms both test-time and competitive training-time alignment baselines; for example, it achieves state-of-the-art SEGALE scores and minimal NA Ratios across several language directions, substantially exceeding methods like ARGS and GenARM.", "weaknesses": "1. Reward-model dependence / shared-judge bias. Long-form tasks use similar reward models for both alignment and evaluation, inviting bias and potential reward gaming; despite noise-robustness tests, evidence for agreement with human preferences and cross-evaluator consistency is limited.\n\n2. Underspecified subgoal & buffer aggregation. The threshold α, buffer 𝓑 update/size policy, and aggregation function 𝒢 (composition of non-contiguous subgoals, overlap limits, length control) are not concretely specified; scalability as buffer/search grow is untested.\n\n3. Unquantified compute/latency cost. Multi-round rollouts and rewrites likely incur higher cost than token-level guidance (e.g., ARGS, GenARM), but per-task wall-clock time, GPU-hours, and throughput/latency are not reported.\n\n4. **Missing comparisons to sequence-level rewriting baselines.** No direct comparison against sequence-level rewriters (e.g., **aligner**, **sentence aligner**), leaving TMPC’s advantage over these methods unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7bEfoVb4Db", "reviewer": "Reviewer_FkBi", "summary": "This paper formulates LLM-generated subgoals as a model predictive control problem (TMPC). They identify subgoals via hindsight replay and use the identified subgoals to regenerate the final answer. The have extensive results on against many baseline methods on various datasets.", "questions": "Minor Suggestion:\n-There is an extra comma on L28\n-Maybe move the legend of Figure 1 to the top or bottom and spanning across all three methods to make it more clear it applies to all three", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-Figure 1 clearly shows the high-level idea of the approach compared to traditional methods. The caption is also descriptive and clear to reinforce the ideas in the Figure and introduction\n\n-The paper is easy to follow and mythology is clear\n\n-The proposed approach outperforms baselines in various tasks. I specifically appreciate the failure cases in baselines such as in L406.", "weaknesses": "The related works discussion lacks a discussion on subgoal generation for LLM/VLM-based tasks, where prior work already exists [1, 2]. I do think there is novelty in formulating this as a test-time model predictive control problem.\n\n[1] Logeswaran, Lajanugen, et al. \"Few-shot Subgoal Planning with Language Models.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n[2] Wang, Jiawei, et al. \"Discovering intrinsic subgoals for vision-and-language navigation via hierarchical reinforcement learning.\" IEEE Transactions on Neural Networks and Learning Systems 36.4 (2024): 6516-6528.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GtGOnPWDNI", "reviewer": "Reviewer_SBgL", "summary": "This paper presents Textual Model Predictive Control (TMPC), a novel framework for test-time LLM alignment. It insightfully frames the problem as a trade-off between the 'curse of horizon' in token-level optimization and the 'curse of dimensionality' in response-level optimization. TMPC introduces a middle ground by first identifying meaningful \"subgoals\" from generated text and then optimizing generation conditioned on these discovered segments. The framework uses two core principles: Hindsight Subgoal Identification to find high-reward text segments, and Subgoal-Conditioned Re-Generation to build upon them, achieving strong results without updating model parameters.\n\nThis is a very good paper with a highly promising and original core idea. The conceptual framing is excellent, and the initial results are strong. However, its claims of novelty rest on an unsubstantiated comparison, and the paper writing to clarity of its core mechanisms and analysis needs improvement. With the addition of a crucial heuristic baseline comparison and a clearer analysis of its mechanics, this paper has the potential to be a top-tier.", "questions": "* To justify the core novelty of hindsight identification, is TMPC demonstrably better than using simple, human-curated subgoals? For instance, in the paragraph-level MT task, how does TMPC's performance compare against a version that uses pre-defined sentence boundaries as the subgoals? This single experiment is critical to proving the value of your proposed mechanism over cheaper, task-specific alternatives.\n* Figure 2 suggests subgoals from different rollouts are composed as conditioning for the next generation step. How does the model ensure narrative and logical coherence when concatenating these segments, which may originate from conflicting generation paths? Can you provide analysis showing this composition does not harm the fluency and continuity of the output?\n* The concept of a \"subgoal\" remains abstract. Could you provide concrete examples of the subgoals TMPC actually identifies across the three tasks? For program synthesis, what does an \"abstract\" subgoal that resolves a single test case look like in the generated text? This is essential for understanding what the model is learning to value as a planning step.\n* The subgoal buffer B is a finite resource. In very long generation tasks, what prevents the buffer from being filled with early, locally-optimal subgoals, thus hindering exploration in later stages of generation? Is there a mechanism to manage the buffer's contents beyond a simple capacity limit, such as diversity promotion or aging out older subgoals?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper's primary strength is its excellent framing of the alignment problem as a trade-off between token-level and response-level, providing a clear motivation for a new class of solutions. The central concept of planning via subgoals is elegant. Figure 1 is a great illustration.\n* The method is designed to be task-agnostic, discovering subgoals automatically rather than relying on pre-defined structures. This is a significant advantage over methods that require task-specific engineering.\n* Bringing ideas from control theory is interesting.", "weaknesses": "* The paper's central claim is that its hindsight identification of subgoals is a key innovation. However, given this is for test-time alignment, users can easily use a cheap, task-specific heuristic (e.g., sentence boundaries for translation) for their own task. So, I think task-specific handmade subgoal is a very strong and obvious baseline. Without showing that TMPC's discovered subgoals outperform these simple heuristics, the true value of the complex discovery mechanism is questionable.\n* The paper's idea is good but a better writing may improve the quality and make it easier to follow. The central concepts of \"subgoal\" and \"re-generation\" remain abstract due to a lack of concrete examples, and the results are not deeply explored maybe due to limited space. We strongly suggest  maybe a better use of appendix and a concise revision on the methods background and literature to using the space to provide the essential examples and analysis needed to fully validate the framework's contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)