PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models
Authors: Ruiqi Wang
Dezhong Zhao
Ziqin Yuan
Tianyu Shao
Guohua Chen
Dominic Kao
Sungeun Hong
Byung-Cheol Min
Venue: NeurIPS 2025 oral ()
Keywords: Preference-based Reinforcement Learning
Foundation Models for Robotics
Neuro-Symbolic Fusion
Multimodal Feedback
Causal Inference
Trajectory Synthesis
Robot Manipulation
Primary area: reinforcement_learning

Abstract: Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for teaching robots complex behaviors without reward engineering. However, its effectiveness is often limited by two critical challenges: the reliance on extensive human input and the inherent difficulties in resolving query ambiguity and credit assignment during reward learning. In this paper, we introduce PRIMT, a PbRL framework designed to overcome these challenges by leveraging foundation models (FMs) for multimodal synthetic feedback and trajectory synthesis. Unlike prior approaches that rely on single-modality FM evaluations, PRIMT employs a hierarchical neuro-symbolic fusion strategy, integrating the complementary strengths of vision-language models (VLMs) and large language models (LLMs) in evaluating robot behaviors for more reliable and comprehensive feedback. PRIMT also incorporates foresight trajectory generation to warm-start the trajectory buffer with bootstrapped samples, reducing early-stage query ambiguity, and hindsight trajectory augmentation for counterfactual reasoning with a causal auxiliary loss to improve credit assignment. We evaluate PRIMT on 2 locomotion and 6 manipulation tasks on various benchmarks, demonstrating superior performance over FM-based and scripted baselines. Website at https://primt25.github.io/.

PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models
Ruiqi Wang** Dezhong Zhao Ziqin Yuan
Tianyu Shao Dominic Kao Sungeun Hong* Byung-Cheol Min
Guohua Chen²
Purdue University, West Lafayette, IN, USA
Beijing University of Chemical Technology. Beijing, China
University of Illinois Urbana-Champaign, Champaign, IL. USA
"Sungkyunkwan University. Scoul, South Korea Indiana University Bloomington, Bloomington, IN, USA
Abstract
Preference-based reinforcement learning (PbRL) has emerged as promising paradigm for teaching robots complex behaviors without reward engineering. How- ever, its effectiveness is often limited by two critical challenges: the reliance extensive human input and the inherent difficulties in resolving query ambiguity and credit assignment during reward learning. In this paper, introduce PRIMT, PbRL framework designed to overcome these challenges by leveraging foundation models (FMs) for multimodal synthetic feedback and trajectory synthesis. Unlike prior approaches that rely on single-modails FM evaluations, PRIMT employs hi- erarchical neuro-symbolic fusion strategy, integrating the complementary strengths of large language models and vision-language models in evaluating robot behaviors for more reliable and comprehensive feedback. PRIMT also incorporates foresight trajectory generation, which reduces early-stage query ambiguity by warm-starting the trajectory buffer with bootstrapped samples, and hindsight trajectory augmenta- tion, which enables counterfactual reasoning with a causal auxiliary loss to improve credit assignment. We evaluate PRIMT 2 locomotion and 6 manipulation tasks on various benchmarks, demonstrating superior performance over FM-based and scripted baselines. Website at httttts://prrmnii gi thub. io/
Introduction
Reinforcement learning (RL) has shown great success in various robotics domains 144. remains reliant on carefully designed reward functions. In many practical scenarios. designing an informative reward function is highly challenging, as task objectives are often implicit and multi- faceted [5). Preference-baseee RL (PbRL) [ has emerged as promising alternative to address this challenge by learning reward models from human comparative feedback over robot trajectories, providing a more intuitive means of aligning robotic systems with human intent [2 Nevertheless, the extensive human input required for preference labeling restricts the scalability of PbRL [13].
yet
To mitigate this bottleneck, recent work has explored leveraging foundation models (FMs), e.g., large language models (L.MMs) and lasiunel models (VLMs), synthetic feedback sources, drawing on their broad world knowledge [14177]. Compared to using FMs to design dense reward functions [8) provide auxiliary contrastive signals 21 24). incorporating them as evaluators within PbRL offers potentially more efficient and robust paradigm.
EEqual contribution. Corresponding authors: wang5357dpurdue. edu; minb0iu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Off-Policy Foresight Trajectory Generation
Hierarchical Neuro-Symbolic Preference Fusion
Kedhone
Kvis
Crewd
Amdering,
Intra Fusien
Frear Cycar
Enstronment
Task Cantest
Boutstrap Trajecteries
VLMS
textler
texti
PCR: DE
Prstiction
003:
Reip: lo
Gesp:
lntra Fasion
Cuar 0.89
Yasei oft >
Policy
Trajectory Butfer
Saugling
Pot:
LLM-
Poaker Finwriio
PSL-bnsed InterFusien
Trajectory
Reward Learning with Causal Auxiliary Loss
Hindsight Trajectory Augmentation
Prediction
Action
Abduction Identiff Ey
(or), oo Y
Predict
Minimal
Tina' of > ab
Reward
Traisiy
Counterfactual Trajecturies
outcooe
under
Interventionn
prefernnce to key
the
LLM
Model
Loref
change
causa1 stape
SCM-bbased Reasoning
(e', Yrou)
Figure I: Overview of PRIMT, comprising two synergistic modules: 1) Hierarchical neuro- cuuro-yymoolic preference fusion improves the quality of synthetic feedback by leveraging the complementary strengths of VLMs and LLMs for multimodal evaluation robot behaviors; and 2) Bidirectional trajectory synthesis mitigates stage query ambiguity through foresight generation and enhances credit assignment reward learning via hindsight counterfactual augmentation with a causal auxiliary loss.
However, obtaining reliable and high-quality FM feedback remains challenging, primarily due to the dominant reliance on single-modalig evaluation. LLM-based approaches [15] 16] interpret
structured textual projections of trajectories, such as sequential arrays of state-action pairs, enabling sophisticated temporal reasoning over procedural logic and motion progression 225 these textual descriptions can be abstract or incomplete, making LLMs prone hallucinations of key events, especially when inferring fine-grained spatial interactions (27 28
26 However,
On the other hand, VLM-based methods (14) analyze spatial cues from visual renderings of robot trajectories, such as final-state images or intermediate frames, effectively capturing spatial goal completion [29]. Yet these methods often overlook subtle temporal dynamics within the trajectory [1] Consequently, relying on either modality alone risks incomplete or unreliable feedback (see Appendix A for more
analysis), highlighting the need for a more comprehensive multimodal evaluation framework.
Furthermore, even if feedback from FMs reaches suman--xpprt-levee quality, PbRL still faces two intrinsic challenges: Query ambiguity: trajectory pairs often exhibit uniformly low quality in early training stages. This happens when they are generated from random or weakly optimized policies and lack task-relevant variations, making it hard elicit meaningful preferences [7 32]; and ii) Credit assignment: even when reliable trajectory-level preferences are available, it often remains difficult to accurately attribute the observed preference differences to specific states actions 33). Without effective state-action conte-actioon-eeell credit assignment, the learned reward model may result misaligned behaviors subsequent RL training (33). Meanwhile, FMs have shown strong abilities in planning [7]. control [38] 39,, and causal reasoming [1) These advances lead us to the following question: Can FMs move beyond serving as passive preference providers to be actively
41)
leveraged to mitigate query ambiguity and improve credit assignment in PhRL?
In this paper, we propose PRIMT, foundation nodel-driven framework for PbRL designed to address key challenges in synthetic feedback quality, query ambiguity, and credit assignment. PRIMT, as illustrated in Fig. comprises two core components:
i) Multimodal feedback fusioo, which enhances synthetic feedback quality by combining the comple- mentary advantages of LLM- and VLM- evaluations. Rather than directly feeding multimodal trajectory representations into generic multimodal FMs, PRIMT adopts hierarchical neuro-symboit preference fusion strategy. It first performs intra-modal fusion to produce modality-specific labels and confidence estimations. Then, inter-modal fusion is conducted using probabilistic soft logic (PSL) 42,, which infers the final preference label via structured and interpretable reasoning over multimodal evaluation outputs and trajectory context, enabling robust aggregation of preference beliefs from heterogeneous sources.
iii Bidirectional trajectory synthesis, which leverages FMs to actively enhance reward learning in PbRL. In the foresgght phase, LLMs generate diverse task-aligned trajectories to initialize the

---

trajectory buffer. Unlike prior work [16 36 that assumes FM-generated trajectories are optimal, we treat them as semantically meaningful anchors for informative comparisons, reducing early-stage query ambiguity. In the hindsight phase, LLMs are prompted to generate counterfactual trajectories via causal reasoning based on the structural causal model (SCM) 43]]. When clear preference is detected, the model identifies causal steps and applies minimal edits to reverse the preference, producing counterfactuals that highlight critical distinctions. To better exploit these samples for credit assignment, we introduce a causal-aware auxiliary loss that enforces reward separation at edited steps while ensuring consistency elsewhere. This enables more precise preference attribution, thereby improving the efficiency and generalization of the learned reward model in downstream RL training.
Our key contributions are summarized as follows:
We present PRIMT, a general FM-driven framework for zero-shot PbRL., which leverages founda- tion models not only as synthetic teachers to eliminate human annotation but also as active agents to facilitate preference reward learning.
We introduce a hierarchical neuro-symbolic preference fusion strategy that combines the comple- mentary strengths of LLMs and VLMs for multimodal evaluation robot trajectories, improving the reliability and quality of synthetic feedback.
We propose foresight trajectory generation to bootstrap early-stage query informativeness, and hindsight trajectory augmentation via counterfactual reasoning, coupled with causal auxiliary loss, to improve credit assignment in reward learning.
We conduct extensive experiments across 2 locomotion and 6 manipulation tasks from the DMC [44), MetaWorld [45], and ManiSkill [46] benchmarks, demonstrating that PRIMT con- sistently outperforms state-of-the-art baselines. Ablation studies provide insight into component effectiveness, and we further validate PRIMT's real-world applicability on Kinova Jaco robot.
Related Works and Preliminaries
Foundation Models as Rewards for RI. Foundation models refer to large-scale pre-trained models with strong generalization and reasoning capabilities across tasks [47]. Recent work has explored leveraging FMs to address the reward engineering challenge in RL. One line of research uses coding LLMs to directly generate executable reward functions I Another employs VLMs as
contrastive reward signals 241 for example, RoboCLIP 24 rewards agents by aligning trajectory images with task descriptions or demonstrations. However, such explicit FM-based reward signals often noisy and high- variance [4] Recent approaches have adopted the PbRL paradigm, using FMs as synthetic evaluators to generate trajectory-level preference labels and train reward models: PrefCLM (15) and RL-SaLLLM-F [16] leverage L.MMs analyze numerical state-action sequences, while RL-VLM-F [4] uses VLMs to assess final-state images of robot trajectories. These approaches have shown improved performance over FM-generated scalar rewards. Our work builds on this PbRL- with-FM direction but introduces two key innovations. First, PRIMT adopts a multimodal evaluation scheme that combines VLM and LLM perspectives via hierarchical neuro-symbolic fusion, improving the robustness and quality of synthetic supervision. Second, rather than using FMs solely for passive evaluation, we actively incorporate them facilitate reward learning via trajectory synthesis.
Preference-based RL PbRL aims to learn a reward model re from human comparative feedback over pairs of robot trajectories 77. A trajectory a is defined sequence of states and actions {($1, ((81,a))... ... (sT, } with a length T' The annotator provides preference label T E 0, 1} for each pair (a RB), where Y indicates that の is preferred, 0 means o is preferred, and -1 denotes indecision. A preference predictor is constructed using the Bradley- estimate the preference probabilities. The likelihood that 0d is preferred over o is computed as:
model [48] to
ET (1 az)))
exp
pel Polor
exp EE To (sf (asss..)) (sf)
+ exp ET (e
The reward model is trained to align with human preferences by minimizing cross-entropy objective over a collected preference dataset D = {(oA, _ r)) as:
Lpref = -B T)~D [L(T-- = 1} log Py [ > o + = 0} log Polo > o
Training alternates between reward learning and RL-based policy optimization with learned reward.
3

---

Query Ambiguity PbRL relies on informative preference queries to train effective reward models. However, trajectory pairs often exhibit low task-relevant diversity, leading to query ambiguity [8] This ambiguity is especially pronounced in early training stages, when trajectories are uniformly low- quality and incoherent due to randomly initialized policies [49]. Prior works address this by selecting maximally distinguishable or uncertain samples 8
32 50 or initializing the reward model with
expert demonstrations [51 52]. More recent works employ LLMs to revise ambiguous trajectories during training, assuming that the edited versions are task-complete and preferred [16]. In contrast, we proactively initialize the replay buffer with LLM-generated trajectories that are diverse and task-aligned. Unlike prior work, we do not assume these optimal, but use them as preference anchors to support more informative and efficient early-stage evaluation.
Credit Assignment Another core challenge in PbRL is the granularity mismatch between trajectory- level preference supervision and the desired state-action-level reward signal 33 This mismatch introduces uncertainty in attributing credit to specific decisions, impairing both the alignment and gen- eralization of the learned reward model [53] Prior work mitigates this issue by training transformer-
based world models that estimate state importance [3 54 55]. or by collecting additional human annotations to highlight key moments [34]. In contrast, our approach requires neither extra supervi- sion nor architectural changes. Inspired by causal counterfactual reasoning [43, 56458), we prompt LLMs to generate hindsight-basee counterfactual trajectories by minimally editing key decision points in the preferred trajectory to reverse the preference. By asking, "What minimal change would make this trajectory less preferred?", we obtain contrastive examples that expose the underlying reasons for preference. To effectively leverage these counterfactuals in reward learning, we introduce a causal-aware auxiliary loss. It enforces reward separation at edited points while maintaining consistency in the unedited parts, leading more precise credit assignment.
3 Methodology
In this section, present PRIMT: PReference- based reInforcement learning with Multimodal feedback and Trajectory synthesis from foundation models. An overview of the PRIMT is illustrated in Fig. Detailed prompts with example outputs for each component are included in Appendix C
3.1 Multimodal Feedback Generation and Fusion
Trajectory Preprocessing Given a trajectory pair (aA, sampled from the trajectory buffer, we first obtain their textual projections (ox)( and (o for LLM-based evaluation, follow- ing [16). These projections organize each trajectory into dimension-speciffc sequences, capturing structured temporal patterns across state and action components in format that enhances semantic interpretability. For VLM-based evaluation, instead of using all frames or final-state images as in prior work [14), we propose a hybrid keyframe extraction method to capture both low-level motion cues and high-level behavior transitions while avoiding visual overload: i) mear-zero velocity detection identifies frames where the robot motion minimal, typically marking subgoal completions or transi- tional pauses [59] ii) smoothing residual peaks detect high-curvature or abrupt motion transitions by comparing the raw trajectory its smoothed version, capturing key motion shifts (6);; and iii) change point detection segments the trajectory into semantically coherent phases to identify structural high-level task changes [61]. We take the union of the selected frames from all three methods, together with the first and last steps of the trajectory, to form the final keyframe sets kvis(o and kvis(o B). Full details are provided Appendix B
Intra-modal Preference Fusion We then query LLM and VLM separately with corresponding textual projections and keyframe sequences, along with a brief task description, to elicit preference judgments. Each query follows a structured three-step thain-of-thome each trajectory in terms of effectiveness in achieving the task goal; ii) based on this analysis, output preference label; and iii) verify the decision and assign a confidence score from 0 to 1, reflecting the preference certainty. To mitigate variance and improve the reliability of intra-modal labels, we adopt crowd-check mechanism, querying LLM VLM multiple times with randomly permuted trajectory orderings. This produces K predictions from each feedback modality M E (LLM, VLM).
(CoT) prompt: i) analyze
each consisting of preference label T ((() e {-1, 0, 1} and confidence score (k) E [0, 1]. We then

---

aggregate these judgments into a final moddalit-sspecifii preference label T via major voting as:
K
TM
argmnax 1,
1 M =1)
(3)
To estimate and calibrate the confidence Ca associated with the final label, we compute a weighted combination of two complementary signals: i) the average confidence Cm among N judgments that agree with the final label, and ii) the label consistency ratio CM representing vote agreement:
K
CM
1 N
Σ
M
T(
M = M); CN
Σ
T( = aa))
(4)
K
k=1
k=1
The final confidence Cм is then computed as:
CM = CM + (1-a) a.. CM
(5)
where E (0, 1] is balancing hyperparameter (typically set 0.5). This formulation ensures that the final confidence reflects both internal certainty and stability under input perturbations, thereby improving the robustness of modality-specinf confidence estimation.
Inter-modal Preference Fusion The next step is to integrate modality-specifie preference labels into a unified decision. This process non-trivial, as it must consider multiple factors, including intra- modal uncertainty, cross-modal conflicts, and trajectory context that reflects the relative difficulty of visual versus textual evaluation. Intuitively, one might define heuristic rules for each factor, for example, favoring the VLM label when the visual difference between trajectories is high, trusting the label with higher confidence. Yet, such heuristics are brittle and hard to generalize: the conditions involved are often continuous rather than binary, and the interactions among rules be complex.
To efficiently model these latent dependency structures among inputs, heuristics, and decisions, we employ Probabilistic Soft Logic (PSL.) [42, probabilistic framework representing entities of interest as logical aloms interconnected by weighted first-order logic rules. Specifically, we define four rules to guide inter-modal preference fusion:
Agreement Rule: If the VLM and LLM agree on the same preference label Y and least one modality reports high confidence, the agreed label is used as the final decision:
(6) Here, IsAgree( T) is binary indicator set to 1 if both VIM and L.M predict the same label Y, and 0 otherwise; ConfHigh( M) is continuous atom representing the modality-speciie confidence score Cm E [0, 1}: and FinalLabel (MY) [0, 1] is the output atom to be inferred by PSL., representing the final soft confidence assigned label Y E (-1,0,1}
VT, M IsAgree T) A ConfHigh(M (M) - FinalLabel(T))
ii) Conflict Resolution Rules: When modality labels conflict, we resolve the disagreement by considering the associated confidence and trajectory context (detailed rationale can be found in Appendix A.2). Specifically, we prioritize the VLM prediction if the visual discriminability between trajectories and VLM confidence is high:
VT: : -IsAgree(T) A VLMLabel(T) A ConfHigh(VLMM A VDHigh/ + FinalLabel(T)) (7)
Likewise, if the LLM predicts a label Y with high confidence and the temporal discriminability of the trajectory pair is high, we prioritize the LLM prediction:
VT: -IsAgree(Y)) LLMLabel(M)) ( ConfHigh(LLLM) A TDHigh - FinalLabel (Y)
(8)
Here, VLMLabel(T) and LLMLabel (T) are indicators set to if the modality predicts label Y, and 0 otherwise. The atom VDHigh captures the visual discriminability between the two trajectories as:
VDHigh = p (W(((kriss(o*)) ((vvis(o)))
where f() denotes the CLIP encoder applied keyframe sets kvis(-), W denotes the Wasserstein distance, and p(-) is sigmoid function used for normalization.
Similarly, IDHigh captures temporal discriminabilitty based trajectory volatility differences:
TDHigh
o ([IrjVol(o>)) (DHighh=<([[IiVol[((
TriVol(o
(10)
5

---

where Tr)Vol(-) measures the state-action volatility of a trajectory, defined as the mean L2 norm of second-order finite differences:
1
TrjVol(o)
Σ
[[s+11, - az) + (51-1,
(11)
T
iii) Indecision Rule: When both modalities exhibit low confidence, we assign the indecision label:
-ConfHigh A (LLM) + FinalLabel(-1)
(12)
During PSL inference, each logical atom is instantiated with data and grounded into either an observed input variable X (e.g.. IsAgree, ConfHigh, TDHigh) Valid substitutions of these atoms within rule templates yield a set of ground rules. Each ground rule induces one or more hinge-loss potentials, relaxed from the logical clauses using Lukasiewicz continuous-valuee semantics. Formally, each potential takes the form:
output variable (e.g., FinalLabel).
o X) [max (0, (YY
(13)
where & is linear function in PSL representing the distance to satisfaction of the corresponding ground rule, and p 21, controls whether the penalty linear quadratic. Given observed variables X and target variables Y, PSL defines a hinge-loss Markov random field and performs inference by solving convex constrained optimization problem (more details of PSL inference provided in Appendix
Y arg min
Ws. (Y,
s.t.
Σ TE(-1.0.13
FinalLabel(T)- =
(14)
where m is the number of instantiated potentials, oe denotes the 1th potential function, and w; is the weight assigned to the corresponding rule template. Unlike standard PSI. formulations, we impose a one-hot constraint over the final label atoms to reflect the single-label nature in PbRL. By encoding structured dependencies among modality-speeific outputs and trajectory-level context, PSL facilitates robust and adaptive integration of complementary cues from multiple feedback modalities, effectively managing uncertainties and cross-modal conflicts.
3.2 Bidirectional Trajectory Synthesis
Foresight Trajectory Generation Prior to PbRL training, employ LLMs to generate boot- strapped trajectories that exhibit diverse, semantically meaningful, and task-aligned behaviors, providing warm-start initialization for the trajectory buffer. Inspired by structured code-generation paradigms (8). we adopt a three-step CoT strategy: i) generate a high-level, multi-step action plan from the task specification; ii) translate each step into executable code snippets that implement concrete motion primitives; and iii) execute these programs under varied initial conditions (e.g.. robot start positions) and strategy parameters (e.g., height to approach the target) synthesize diverse set of plausible trajectories. Compared to directly prompting LLMs to generate low-level trajectory arrays [16), our method improves physical feasibility and semantic coherence by grounding trajectory synthesis in program logic. The generated trajectories are considered as bootstrapped demonstrations rather than optimal ones, subsequently evaluated by our multimodal feedback module. Combined with strategic sampling schemes, such as uncertainty-baut based sampling [8) these trajectories serve as informative preference anchors when paired with exploration trajectories, reducing ambiguity in early-stage preference queries and accelerating reward learning.
Hindsight Trajectory Augmentation with Causal Auxiliary Loss During PbRL training, when- ever clear preference is identified by the multimodal feedback module, we prompt LLMs to perform hindsight reasoning to generate counterfactual variants of the preferred trajectory. This process follows three-step reasoning pattern based the structural causal model [43]:
i) Abduction:
Identify the causal rationale behind the observed preference by extracting a set of critical causal steps 7" in the preferred trajectory o* that contribute to the preference. To assist this process, we provide the step indices corresponding to keyframes in kvis(a") as reference candidates, though the selected causal steps are not necessarily limited to them.

---

ii) Action: Select a key step T for minimal intervention, generating a counterfactual trajectory oer that reverses the original preference. This involves modifying some critical state-action features the selected step, such as introducing small gripper delay or adding local end-effector position perturbation. The rest of the trajectory remains identical to the original while we allow the LLMs to apply light smoothing to the immediate neighbors (e.g., 2-3 steps before and after the intervention) to ensure physical continuity and avoid abrupt state transitions. Multiple counterfactual variants can be generated through repeated LLM sampling, providing diverse set of sub-prefee sub-preferre alternatives. Following the minimal edit principle [62] [63], we filter the generated counterfactuals based on the L1 distance between the edited state-action pairs and the original, ensuring a small deviation threshold.
tii) Prediction:
Pair each counterfactual variant with the originally preferred trajectory and feed them into the M..M-based intra-modal fusion module to verify whether the counterfactual is sub-preferred, ie., satisfying the preference condition (o" > o.y). Only counterfactuals that meet this criterion are stored and used for reward learning.
Through this hindsight reasoning process, we now have counterfactual trajectories of the preferred trajectory that share a common structure except at minimally edited steps, because of which their preference outcomes diverge. As such, we can assume that the edited steps are responsible for the observed preference signal, which the reward model should learn to correctly attribute. Given this, we introduce a causal auxiliary loss that encourages discriminability at the edited steps while maintaining consistency elsewhere to guide the model to focus on causal differences that drive preferences:
He log (1 (1-exp) exp r
H.) (10 ro (15)
i) causal contrast Inss
ii) reward consistency loss.
where H1 binary mask indicating the edited steps, i.e., He for edited time steps and Hf, otherwise. The first term encourages the model to assign higher rewards to the preferred trajectory at casual steps, while the second enforces consistent rewards on unchanged regions. This auxiliary loss is combined with the trajectory- preference loss as in Eq 2 forming the final loss for reward learning with the generated counterfactuals:
Linal = Gpeef + Aef Caux
(16)
where Aet is a weight used to scale the auxiliary loss to the same magnitude as the primary preference loss. This integrated loss formulation enables the reward model to capture trajectory-level preferences while providing more precise state-action-leve credit attributions.
4 Experiments
4.1 Setup
We evaluate PRIMT across a diverse set of tasks, including locomotion tasks: Hopper Stand and Walker Walk from DeepMind Control (DMC) suite [44, as well as 6 articulation or rigid body manipulation tasks: Button Press, Door Open, and Sweep Into from MetaWorld suite (55). and PickSingleYCB, StackCube, and PegInsertionSide from ManiSkill suite [46]. Detailed task descriptions are provided in Appendix E.
We compare PRIMT against the following baselines:
RL-VLM-F 14): This baseline utilizes VLM to analyze visual renderings of trajectories to provide preference labels, representing a state-of-the-art VLM-based method.
RL-SaLLM-F [16]
This model employs LLM provide preference labels based on textual descriptions of trajectories, and to modify ambiguous trajectories by generating self-improved alternatives, which are assumed be preferred when paired with the original ones. This denotes LLM-based method with trajectory augmentation for addressing query ambiguity.
This baseline leverages crowdsourced LLMs to provide evaluation feedback for improved feedback quality, presenting another state-of-the-ae LLM-basee method.
PrefCLM [5]:
PrefMul: We build this baseline by directly providing the multimodal trajectory inputs used in PRIMT to multimodal FM for evaluation, representing naive approach to multimodal feedback.
7

---

PRIMT
PreGT
SILLM
PROCLM
PiuMuu
RL-VEM-
Puop
Figure 2:
Learning curves of PRIMT and baseline methods across all tasks, averaged over runs with solid lines denoting the average and shaded regions representing the standard error. A moving average window of 5 steps for locomotion tasks and 10 steps for manipulation tasks is applicd improve readability.
This baseline uses expert-designed reward functions provided by the benchmarks in a scripted teacher manner [ provide preference labels. This should, in theory, serve as an upper-bound oracle of PbRL performance on each task.
PrefGT:
We also build several ablation models to assess the impact of each PRIMT component:
w/o.Inrra: without the crowd-check mechanism and intra-modal preference fusion module.
w/o.Inter: without the inter-modal preference fusion module, directly selecting the modality- specific label with highest confidence the final label.
Wo.Foreeen: without the foresight trajectory generation module.
Wo.IlindAug: without the hindsight trajectory augmentation module.
w/o.CauAu: without the causal auxiliary loss for counterfactuals reward learning.
To eliminate the impact of non-model factors, use the same trajectory inputs and CoT prompts as in PRIMT for preference label elicitation across all FM-based baselines and ablation models. The only exception PrefCLM, which relies heavily on direct access the environment code; for this baseline, we follow the original settings from the source paper. For all FM-based methods, we use gpt-4o as the LLM backbone. For the PbRL backbone, we use PEBBLE 7 with the uncertainty- based sampling schedule [ for all methods, along with consistent set of hyperparameters for the RL--based policy learning phase with SAC. This design ensures that the only difference between methods lies in the preference reward learning, allowing for a more controlled comparison.
We evaluate all baselines across all tasks and conduct ablation studies the Door Open and PickS- ingleYCD tasks, each with five random seed runs to ensure statistical robustness. For manipulation tasks, report the success rate, whereas for locomotion tasks, use the episodic return provided by the benchmarks. Further implementation details are provided in Appendix F
4.2 Does PRIMT Learn Effective Rewards and Policies that Prompt Task Performance?
We first investigate whether PRIMT learn effective reward models that lead to policies capable of solving complex tasks. Fig. [ shows the Iearn- ing curves of all methods across 8 tasks. We observe that PRIMT con- sistently outperforms all FM-based baselines that rely on modaliyy feedback, demonstrating superior fi- performance and faster conver- gence. However, the naive multi- modal feedback method, PrefMul, performs poorly. We attribute this to the fact that directly feeding multimodal trajectory inputs to multimodal FMs without appropriate fusion can overwhelm the
PRIMI
willeell
whiltr
whiltar
ww Hindig
Figure 3: Learning curves of PRIMT and ablation models on the Door Open and PickSingleYCB tasks.

---

No
PINTT
Tac Souar
BLAES
StucCCuee
Waler
Figure 4: Left: Distribution of preference labels, showing the proportion of correct, incorrect, and indecisive labels across different methods. Right: Reward alignment analysis, comparing the learned reward outputs of PRIMT, ablations, and baselines against ground-truth rewards. Plots more tasks are provided in Appendix G.
models, potentially even degrading performance. This highlights the need for carefully designed fusion strategies to fully leverage the complementary strengths of multimodal signals, as implemented in the hierarchical neuro-symbolic fusion strategy of PRIMT. Moreover, each component in this hierarchical design is crucial: as shown in Fig. removing either intra-modal or inter-modal fusion significantly degrades task performance. This drop is especially severe when inter-modal fusion is removed, underscoring the critical role of PSL-based reasoning in PRIMT. Unlike simple based fusion in Int.r PSI.based fusion more effectively captures intra-modal uncertainty, cross-modal conflicts, and the influence of trajectory context, enabling more robust preference integration from heterogeneous sources.
Notably, PRIMT is competitive with the oracle PrefGT, even surpassing it on the Sweep Into and Peg Insertion Side tasks, and generally achieving faster early-stage learning, while other FM- based methods fall behind. This suggests that while PrefGT benefits from fine-grained oracle preference labels, PRIMT narrows this gap by improving the quality of synthetic feedback and leveraging trajectory synthesis to address inherent challenges in PbRL. As shown in Fig. 3 foresight generation appears contribute more to the acceleration of early- learning, as the W/o.ForeGen variant reaches a similar final performance but learns more slowly in the initial stages. In contrast, hindsight augmentation plays critical role in achieving high final performance, evidenced by the significant performance drop when either w/o. W/n.Hin or Wau.CauAux is removed. Interestingly, while w/o. CauAux can still benefit from the counterfactual trajectory augmentation, its performance is significantly worse, indicating that the causal auxiliary loss we designed can more effectively leverage these counterfactuals in reward learning, capturing state-action-level preference causation.
4.3 Does PRIMT Improve the Quality of Synthetic Feedback and Mitigate Query Ambiguity?
We next examine the distribution of synthetic feedback generated by PRIMT to assess its impact on preference label quality and query informativeness. We calculate accuracy by comparing the synthetic preference labels with those in PrefGT and record the preference decisions. Fig. 4 shows the percentages of correct labeling, incorrect labeling. and preference indecision for PRIMT, Wor.on, w/o. w/o. and RL-VLM-F (left right). We observe that PRIMT consistently produces more accurate preference labels compared to the baseline and ablations, confirming the effectiveness of the hierarchical fusion design in improving label quality. Additionally, PRIMT significantly reduces indecision rates compared to both the baseline and ForeGnn indicating that foresight generation effectively mitigates query ambiguity.
is worth noting that we could not directly compare indecision rates with RL-SaLLM-F,, another method that uses LLMs to address query ambiguity, because it inherently eliminates indecision by always treating self-augmented trajectories as preferable. However, as shown in Fig. 2 this baseline struggles with etrly-stage learning on tasks from the ManiSkill, which involve nigh-dimensional state and action spaces. We attribute this to the difficulty of directly generating optimal trajectories at the low level in such tasks, making the assumption that generated trajectories are always preferable highly misleading. This highlights the advantages of our foresight generation approach, which addresses query ambiguity from the outset by initializing diverse, bootstrapped trajectories potential preference anchors, and our code-generation paradigm, which improves the trajectory sample quality.
4.4
Does PRIMT Enhance Credit Assignment in the Reward Model?
We further investigate how the learned reward models align with the task progress at state-action level. Fig. 4 shows the normalized reward outputs from the learned reward models of PRIMT, its two

---

ablations without full hindsight trajectory augmentation, and the baseline RL-VLM-F, along with the normalized ground-truth truth reward values on the same trajectories. We observe that PRIMT produces more aligned reward patterns that closely reflect task progress, while the baselines and ablations exhibit either noisy signals or high variance, indicating that their learned reward models struggle to accurately assign rewards at the state-action level, even if they capture trajectory-level preferences. This demonstrates that PRIMT, particularly its hindsight augmentation and causal auxiliary loss, enables more precise
credit assignment in reward learning. Extra quantitative results using the R2 coefficient are provided in Appendix G.8
4.5 Additional Experimental Results
We conducted additional analyses to provide further validation insights of the proposed framework. These include: qualitative evaluations of the trajectory synthesis module (Appendices G.3 and (.44). visualization of policy outcomes across different methods (Appendix (.55. comparison with the dense-reward RL baseline (Appendix G.7). and preliminary experiments on a more complex dual-arm manipulation task (Appendix G.6).
We also performed ablation studies on the influence of the foundation model backbone (Appendix G.2) The results show good potential for more accessible deployment with smaller or cheaper foundation models: with GPT-4o-mini, we achieved a 94% reduction in cost with 16-26% drop in performance, leading to 13x improvement in cost cost-performance efficiency.
4.6 Real-world Deployment
We further demonstrated the effectiveness of PRIMT on Kinova Jaco robot in block lifting and stacking tasks (Fig. 5). Detailed experimental settings and results are provided in Appendix G.9
Block Lifting
Block Stacking
Figure PRIMT successfully completes real-world block lifting and stacking tasks on Kinova Jaco robot.
5 Conclusion and Future Work
In this work, we presented PRIMT, a method that leverages foundation models for multimodal synthetic evaluation and trajectory synthesis reduce human effort and address query ambiguity and credit-assignment challenges in preference-based reinforcement learning (PbRL.). We demonstrated the advantages of PRIMT across wide range of locomotion and manipulation tasks, achieving superior task performance, higher-quality synthetic feedback, and more accurately aligned reward models.
limitation of this study involves the increased cost associated with foundation model usage due to the multimodal components of the framework. To provide clearer understanding of the resource requirements, we conducted a detailed analysis of computational usage and the corresponding cost cost-performance trade-offs, presented in Appendix H.Il We observe that PRIMT strikes good balance between performance and cost-effectiveness, providing a practical path toward scalable preference learning.
Looking forward, while PRIMT was evaluated in single-agent robotic domains, extending key components, such as multimodal evaluation and trajectory synthesis, to non-robotic and multi- agent settings represents an exciting direction for future work. Further discussion assumptions, limitations, and broader impacts is provided in Appendix H
10

---

Acknowledgments
The authors would like thank Jinjin Cai from Purdue University for insightful comments on the figures and visualizations. This work was partially supported by the Ministry of Science and ICT (MSIT), Korea, under the Global Research Support Program in the Digital Field RRS-2024-0042535 supervised by the Institute for Information & Communications Technology Planning & Evaluation (IITP). The support applies authors Hong and Min.
References
[1] Xu Wang, Sen Wang, Xingxing I. Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai, and Qiguang Miao. Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35 (4):5064-5078. 2022.
[2] Ruiqi Wang, Dezhong Zhao, Arjun Gupte, and Byung-Cheol Min. Initial task allocation in multi--umman multi-robot teams: An attention-enhant enhanced hierarchical reinforcement learning approach. IEEE Robotics and Automation Letters, 2024.
[3] Jens Kober, Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(11):1238 1274, 2013.
[4] Wonse Jo. Ruiqi Wang, Baijian Yang. Daniel Foti, Mo Rastgaar, and Byung-Cheol Min. Cognitive load-hased affective workload allocation for multihuman multirobot teams. IEEE Tronsactions Human- Machine Systems, 2024.
[5] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances Neural Information Processing Systems, 35:15221-15222 2022.
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pages 6152-6163. PMLR, 2021.
[8] K Lee, L Smith, A Dragan, and P Abbeel. B-pref: Benchmarking preference-base baseed reinforvement learning. Neural Information Processing Systems (NeurIPS), 2021.
[9] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588
2022.
[10] Ruiqi Wang. Dezhong Zhao, Dayoon Suh, Ziqia Yuan, Guohua Chen, and Byung- Crungg-Cheol Min. Person- alization in human-robot interaction through preference-based action representation learning. In IEEE International Conference on Robotics and Automation (ICRA), 2025.
[11] Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi. Pieter Abbeel, and Michael Laskin. Skill preferences: Learning extract and execute robotic skills from human feedback. In Conference on Robot Learning, pages 1259-1268. PMLR, 2022.
[12] Ruiqi Wang, Weizheng Wang. and Byung-Cheol socially aware robot navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11336-11343. IEEE, 2022.
Cheol Min. Fredback-efficican active preference learning for
[13] Jongjin Park, Younggyo Sco, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lec. Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference- reinforcement learning. arXiv preprint arXiv:2203. 00000.00000000 10050, 2022.
[14] Yufei Wang, Zhanyi Sun. Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI-vIm-f: reinforcement learning from vision language foundation model feedback. In Proceedings of the 41st International Conference on Machine Learning, pages 51484-51501, 2024.
[15] Ruiqi Wang. Dezhong Zhao, Ziqin Yuan, Ike Obi, and Byung- Cheol Min. Prefelm: Enhancing preference- based reinforcement learning with crowdsoureed large language models. IEEE Robotics and Automation Letters, 2025.

---

[16] Songjun Tu, Jingbo Sun, Qichao Zhang. Xiangyuan Lan, and Dongbin Zhao. Online prefeence--bsse reinforcement learning with self-augmented feedback from large language model. 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025). 2025.
[17] Harrison Lee, Samrat Phatale. Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
[18] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations.
[19] David Venuto, Mohammad Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Codc as reward: Empowering reinforcement learning with vlms. In Forty-first International Conference on Machine Learning.
[20] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations.
[21] Yecheng Jason Ma, Vikash Kumar, Amy Zhang. Osbert Bastani, and Dinesh Jayaraman. Liv: Language- image representations and rewards for robotic control. In International Conference Machine Learning, pages 23301- 233200. PMLR, 2023.
[22] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, pages 14743 14752, PMLR, 2022.
[23] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310. 2023.
[24] Sumedh Sontakke, Jesse Zhang. Séb Amold, Karl Pertsch, Erdem BIyIk, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36:55681- 55691, 2023.
[25] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-lim: Large language models are effective temporal leamerr. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[26] Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint ar arXiv:2401.0663 2024.
[27] Fangjun Li, David C Hogg, and Anthony G Cohn. Advancing spatial reasoning in large language models: An in-depth evaluation and enhancement using the stepgame benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18500-18507, 2024.
[28] Hongjic Zhang, Hourui Deng, Jie Ou, and Chaosheng Feng. Mitigating spatial hallucination in large language models for path planning via prompt engineering. Scientific Reports, 15(1))8881, 2025.
[29] Fiona Luo. Vision-language models for robot success detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23750-23752, 2024.
[30] Xi Ding and Lei Wang. Do language models understand time? arXiv preprint arXiv:2412.13844 2024.
[31] Tonko EW Bossen, Andreas Møgelmose, and Ross Greer. Can laaguanla models understand and interpret dynamic gestures from pedestrians? pilot datasets and exploration towards instructive nonverbal commands for cooperative autonomous vehicles. arXiv preprint 1xXix:554411077 2025.
[32] Xuening Feng, Zhaohui JIANG, Timo Kaufmann, Eyke Hüllermeier, Paul Weng, and Yifei Zhu. Comparing comparisons: Informative and easy human feedback with distinguishability queries. In ICML 2024 Workshop on Models of Human Feedback for Al Alignment, 2024.
[33] Mudit Verma and Katherine Metcalf. Hindsight priors for reward learning from human preferences. In The Twelfth International Conference on Learning Representations.
[34] Simon Holk, Daniel Marta, and Iolanda Leite. Polite: Preferences combined with highlights in rein- forcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages IEEE, 2024.
[35] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-ban based reinforcement learning. In International Conference on Learning Representations, 2021.
12

---

[36] Teyun Kwon, Norman Di Palo, and Edward Johns. Language models as zero-shot trajectory generators. IEEE Robotics and Automation Letters, 2024.
[37] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprini S.Giv:220.00699. 2022.
[38] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEF, 2023.
[39] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei. Voxposer: Composable
3d value maps for robotic manipulation with language models. In Conference Robot Learning, pages 540-562. PMLR, 2023.
[40] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. Transactions Machine Learning Research, 2023.
[41] Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards unified agent with foundation models. arXiv preprint arXiv:2307.0960 2023.
[42] Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. Journal of Machine Learning Research, 18(109); 1 67, 2017.
[43] Judea Pearl. Causal inference. Causality: objectives and assessment, pages 39-58, 2010.
[44] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801.0869000 2018.
[45] Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pages 1094-1100 PMLR, 2020.
[46] Jiayuan Gu, Fanbo Xiang. Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04633 2023.
[47] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
[48] Ralph A. Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324-355, 1952.
[49] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Open problems and fundamental limitations of reinforeement learning from human foedback. arXiv preprint arXiv:2307.1522 2023.
[50] Erdem Bryik, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: user-friendly approach active reward learning. arXiv preprint arXiv: 19Xx:0990000465. 2019.
[51] Erdem Brykk, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45 67, 2022.
[52] Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. arXiv preprint 3XX1v:99060092. 2019.
[53] Nan Rosemary Kc, Anirudh Goyal ALIAS PARTH GOYAL, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding, Advances neural information processing systems, 31, 2018
[54] Changycon Kim, Jongjin Park, et al. Preference transformer: Modeling human preferences using trans- formers for rl In The Eleventh International Conference on Learning Representations, 2022.
[55] Dezhong Zhao, Ruiqi Wang, Dayoon Suh, Taehyeon Kim, Ziqin Yuan, Byung-Cheol Min, and Guohua Chen. Prefmunt: Modeling human preferences in preference based reinforcement learning with multimodal transformers. arXiv preprint arXiv:2409.1366 2024.
13

---

[56] Thomas Mesnard. Theophane Weber, Fabio Viola, Shantanu Thakoor. Alaa Saade, Anna Harutyunyan. Will Dabney, Thomas S Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. International Conference on Machine Learning, pages 7654-7664. PMLR, 2021.
[57] Mengyue Yang, Quanyu Dai, Zhenhua Dong. Chen, Xiuqiang He, dation with counterfactual user preference simulation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2342-2351, 2021.
Wang. Top-n recommen-
[58] Sahil Verma, John Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review. arXiv preprint arXiv:2010. 100550000..0096 2(1):1, 2020.
[59] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[60] Baris Akgun, Maya Cakmak, Karl Jiang. and Andrea Thomaz. Keyframe- based learning from demon- stration: Method and evaluation. International Journal of Social Robotics, 4:343-355 2012.
[61] Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing, 167:107299, 2020.
[62] Yash Goyal, Ziyan Wu, Jan Erast, Dhruv Batra. Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pages 2374-2384, PMLR, 2019.
[63] Andi Peng, Aviv Netanyahu, Mark K Ho, Tianmin Shu, Andreea Bobu, Julie Shah, and Pulkit Agrawal. Diagnosis, feedback. adaptation: A human-in- in-the-loopp framework for rest-time policy adaptation. In International Conference Machine Learning, pages 27630-200 27641. PMLR, 2023.
[64] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martin Abhishek Joshi, Kevin Lin, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12299 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The scope of this work is preference- based reinforcement learning (PbRL) for robotics. The abstract and introduction clearly state our core contributions: (1) introducing PRIMT, framework for PbRL enhanced by foundation models; (2) proposing hicrarchical neuro-symbolic fusion strategy for multimodal feedback integration; and (3) incorporating hidirectional trajectory synthesis to address query ambiguity and credit assignment. These claims are directly supported by experimental results across diverse rohotic manipulation and locomotion tasks. Key assumptions are explicitly discussed in both the main paper and appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper explicitly discusses limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" scction in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicablle authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include formal theoretical results proofs.
15

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, including benchmark environments, backbone algorithms, and evaluation metrics. We disclose the architecture and training settings for both the reward model and policy (e.g.. SAC), and include tables listing hyperparameters for reward learning query sampling (Appendix F. The implementation details of each baseline and ablation component are also explained. While access to specific APIS (e.g.. GPT-4o) is required, we provide all prompts and sampling configurations to support reproduction.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should deseribe the steps taken make their results reproducible verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., a large language model), then there should either be
(d) We recognize that reproducibility may he tricky in some cases, which case authors welcome to describe the particular way they provide closed-source -ource models, it may be that access to the model is limited some way (e.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: The benchmarks we have used are all publicly available. We plan to publicly release the full codebase via GitHub upon acceptance of the paper.
Guidelines:
16

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ctttpp://niip.cc/pubbbb. for more details.
guides Staeeess/oodSSubnssssooooPoo...
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (c.g., for new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa://nipa cttccccppuuo cc/public/ guides/CodeSutee soippiiiiiiioooccii.....c for more details.
The authors should provide instructions on data access and preparation, including how access tee raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed descriptions of the training and evaluation setup in both the main paper and Appendix ( Additional tables summarize the settings used for SAC and reward learning across tasks. These details are sufficient to interpret and reproduce results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report mean and standard deviation across multiple independent runs for all learning
curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions madc should given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they have 96% CL. if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
17

---

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We stated that all experiments were conducted on workstation equipped with five NVIDIA RTX 4090 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the Code of Ethics and affirm that our work fully complies with its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special considcration due laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification: We have included detailed impact statement in Section IL.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
18

---

Answer: [NA]
Justification: Our paper does not involve the release any models or datasets that pose a high risk of misuse. We do not release any large pretrained generative models, scraped image datasets, or other resources with potential dual-use concerns. All models used (e.g., LLMs or VLMs) are accessed through existing APIs with their own safety mechanisms in place, and additional deployment distribution is carried out by authors.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets and pretrained models, such as the Meta MetaWorld World benchmark (MIT license) and pre-trained large language models (e.g.. GPT-4 via the OpenAT API) All benchmarks and models used properly cited in the main text, and their licenses and terms of use have been followed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
authors should state which version
asset is used and, possible, include URL.
The name of the license (c.g.. .. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. cww/com/aaoooooo det//attsstss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Are
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coole/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
19

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it an important, original. or non standaad component of the core methods in this research? Note that if the LLM used only for writing. editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [Yes]
Justification: As described the paper,
foundation models, specifically LLMs and VLMs, integral components our preference baseed reinforvement learning (PbRL) framework. LLMs employed to synthesize structured feedback and to generate counterfactual trajectories through causal reasoning, hoth of which directly impact the reward learning pipeline.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp::////uuii..peeeeee htt.r/ppp...coooeeeeeeeeee////2/ http/////g/t//////////////////// should or should not be described.
for what
20

---

APPENDIX
Table of Contents
Limitations Single-Modality Feedhack in Rohot Trajectory Evaluation
22
A.1 Failure Cases VLM- and LLM-Based Preference Feedback
A.2 Quantitative Analysis of Trajectory Context Influence on Each Modality
B Details of Keyframe Extraction
24
B. Ncar-Zero Velocity Detection
B.2 Smoothing Residual Peaks
B.3 Change Point Detection (CPD)
B.4 Combining Methods for Keyframe Extraction
C Prompts and Example Outputs
26
C.I Synthetic Preference Generation
C.2 Foresight Trajectory Generation
C.3 Hindsight Trajectory Augmentation
PSL Inference Details
27
D.1 Variables
D.2 HL-MRF Formulation
D.3 Eukasiewiez Relaxation
D.4 Template Rule Grounding
D.5 Example Rule Expansion
Details Task Environments
30
E.1 MetaWorld
E.2 ManiSkill
E.3 DeepMind Control Suite
F Additional Implementation Details
31
F.1 Baselines
F.2 Reward Learning
F.3 Policy Learning
Additional Experimental Results
G.1 More Visualizations of Label Distributions and Learned Reward Outputs
G.2 Ablation Study FM Backbone Selection
G.3 Qualitative Analysis
Foresight Trajectory Generation Module
G.4 Qualitative Analysis of the Hindsight Trajectory Augmentation Module
G.5 Policy Visualizations of Different Methods
G.6 Preliminary Experiments on a Bimanual Manipulation Task
G.7 Comparison with Dense- Dense-Reward RI.
G.8 Qualitative Reward Alignment Analysis
G.9 Real-world Experiments
Discussion on Limitations and Broader Impact
40
H.l Assumptions and Limitations
H.2 Impact Statement
21

---

A Limitations of Single-Modality Feedback in Robot Trajectory Evaluation
In this section, we provide a detailed analysis of how single-modality evaluation patterns can lead to incomplete unreliable feedback in robot trajectory judgment. We also present the rationale behind our hierarchical neuro-symbolic fusion module design, which addresses these limitations by integrating complementary signals from VLMs and LLMs.
A.1 Failure Cases of VLM- and LLM-Based Preference Feedback
To illustrate the potential pitfalls of relying solely on VLM or LLM evaluations, present series of failure cases from the Door Open task in the MetaWorld benchmark. These cases highlight situations where either VLMs or LLMs struggle to provide accurate preference judgments, Icading to suboptimal or inconsistent feedback. For each example, we use structured prompts and trajectory inputs as in PRIMT, with responses obtained by querying got-4o as the representative VLM//LMM
VLM Limitations and LLM Advantages Figs. 6a and 6b present two illustrative cases from the World Door Open task, highlighting the complementary strengths of LLMs and the inherent limitations of VL in trajectory evaluation.
Presi Sanderigg
Shecial Decrrptine
Testall Dacrlppter
WEF Onler oo M. 3 of7 ag
Wape 1477 5 20 2.00 is (i (.20)
Thjettyy
.921
Wodee Mnwer
Raccodey A
fastring Fa boot colattin
Feseed og the 19000 acfined
sight
TrjetheyB
-001
(a) LLM accurately infers task intent from TCP movement patterns, while VLM struggles to distinguish between visu- ally similar failure cases.
(b) LLM captures motion fluency and continuous state transitions, while VLM is misled by similar frame appearances.
Figure 6: Examples where L.L.-based evaluation outperforms VI.M-based evaluation, highlighting the advantages of temporal reasoning and task intent understanding.
In Fig. 6a both Trajectory A and Trajectory B ultimately fail to open the door, resulting in similar final visual states. The VLM, relying solely frame-based image sequences, cannot differentiate the two and returns an indecisive response. In contrast, the LLM, which processes structured state encodings (c.g.. TCP position, gripper state, object coordinates), explicitly favors Trajectory A. Its response indicates that Trajectory A exhibits a more consistent progression toward the door handle, reflecting clearer task intent, while Trajectory B lacks such directed movement. This distinction is critical for evaluating task success, as it captures the underlying purpose of each motion sequence rather than merely the final outcome.
Fig. 6b presents different challenge, where both trajectories successfully complete the task. Despite similar final visual outcomes, the I.I. identifies qualitative differences in the motion patterns, preferring Trajectory for its smooth, uninterrupted approach. In contrast, Trajectory B exhibits a brief hesitation and slight backward drift, disrupting the task's fluidity. These subtle behavioral cues, emphasized by the green circle in Trajectory A and the red circle in Trajectory B. detectable only through temporal reasoning and sequential context, which are inherently absent from frame-based VLM evaluations.
These examples underscore a fundamental limitation of VLMs: their reliance on discrete visual frames makes them highly effective at capturing spatial relationships but poorly suited for interpreting task intent, motion quality, or temporal consistency. Without structured state data, VLMs can be casily misled by visually similar, yet semantically different, trajectories, leading to indecision or incorrect preferences.
In contrast, LLMs offer a powerful complement by incorporating temporal context, semantic task cues, and structured state information into their evaluations. This allows them assess not just where robot ends up, but how arrived there, capturing fine- grained distinctions like hesitation free execution, consistent approach angles, and purposeful task progression. These strengths make LLMs an ideal complementary modality for evaluating complex robotic behaviors, particularly in tasks where motion fluency and goal-directed intent are critical but difficult capture through static visual frames alone.
22

---

LLM Limitations and VLM Advantages Figs.
7a Faladd 7b present two cases from the MetaWorld Door
Open task where VLMs outperform LLMs in preference evaluation. Fig. 7 highlights subtle but critical spatial
Yinml Benderrer
Temraal Deectpaane
Beaderaag
Thated Deverpton
Topeaay&
Pateedal
degienr A. for piiper coumriy
Fall lupedable
Roaee up. hagntle shoee
Anannday Tounkrr
par
anr al
dhe dust
Tejetayy
(a) VLM correctly identifies a grasp error in Trajectory B. where the gripper contacts the hinge side instead of the intended handle region, while the LLM fails to detect this misalignment and assigns equal preference.
(b) VLM accurately recognizes that only Trajectory A successfully opens the door, while the LLM incorrectly infers that both trajectories are equally successful.
Figure 7: Examples where VLM outperforms LLM in preference evaluation, highlighting the advantages of direct visual perception for detecting spatial errors, physical contact quality, and task success.
distinction. In Trajectory A, the gripper correctly makes contact with the handle at its intended grasp region, leading to a successful interaction. In contrast, Trajectory B misaligns the gripper with the hinge side of the door, resulting in a mechanically incorrect grasp. This error is visible in the red-circled region of Trajectory B. The VLM, grounded in direct visual perception, correctly identifies this spatial discrepancy and selects Trajectory A as the preferred option. However, the LLM, which relies on abstract state variables such as TCP position, gripper state, and object coordinates, fails to capture this fine-grained spatial error, instead outputting equal preference for both trajectories. This occurs because the LLM lacks direct perceptual grounding, making it blind to critical spatial misalignments that are immediately evident in visual frames.
Fig. 7b presents more pronounced failure casc. Here, only Trajectory A successfully opens the door, as indicated by the green circle, while Trajectory B fails produce any meaningful outcome, marked by the red circle. Despite this clear visual difference, the LLM mistakenly infers that both trajectories are equally successful, likely due to similar coordinate sequences that superficially resemble goal-directed hehavior. This is a classic hallucination error, where the LLM abstracts away from the actual physical outcome, ignoring critical perceptual cues. In contrast, the VLM, which directly observes the task's visual consequences, correctly identifies Trajectory as superior.
Together, these examples highlight the limitations of relying solely on language- based state representations for preference evaluation. Without direct visual input, L.MSs can miss critical spatial alignments, physical contacts, and fine-grained task completions, lcading to misleading or incorrect preferences. In contrast, VLMs are inherently suited to capture these spatial relationships, making them indispensable for evaluating precise robotic interactions. This complementarity underscores the need for multimodal preference model that integrates the perceptual grounding of VLMs with the temporal and semantic insights LLMs, enabling more robust and context-aware trajectory evaluation.
A.2 Quantitative Analysis of Trajectory Context Influence on Each Modality
From the qualitative examples above, it is clear that each feedback modality has its own potential limitations yet complementary strengths. We can observe that the reliability of these labels can vary significantly depending on the specific trajectory context, such whether the trajectories contain visually distinguishable features cxhibit meaningful temporal progression.
To fully leverage these complementary capabilities, it essential to quantitatively analyze how trajectory content impacts the labeling reliability of cach foedback modality. Given our observations that VLMs tend to excel at analyzing spatial cues from visual renderings, we hypothesize that VLM labels should be more accurate when the visual discriminability (i.e., spatial differences) between trajectories is higher. Similarly, we expect LLM labels be more accurate when the temporal discriminability (i.e., time-dependent behavioral differences) between trajectories is more pronounced.
To validate these hypotheses. we conducted a quantitative analysis to assess the impact of trajectory context on labeling accuracy for each modality. Specifically, we examined the correlation between the degree of context contrast, i.e., VDHiggh defined in Eq. 9 for VLMs and TDHigh defined in Eq. [o LLMs, and the corresponding labeling accuracy.
23

---

<LM LINL
0.
183
"53
SCrigh
10mign
(a) VLM label accuracy generally increases VDHigh increases.
(b) LLM label accuracy generally increases TDHigh
increases.
Figure 8: Impact of trajectory context on labeling accuracy of each feedback modality.
For this analysis, sampled 200 trajectory pairs from the Button- Button-press press task the MetaWorld benchmark clustered them into 20 groups separately based on their VDHigh and TDHigh values, respectively. We then collected human expert labels as ground truth and computed the labeling accuracy of both VLMs and LLMs under varying levels of context contrast. The results are presented in Fig. Sa and Fig. 8b
As shown in Fig. 8a VLM label accuracy generally increases with higher VDliigh, confirming that VLMs benefit from stronger visual cues when assigning preference labels. Similarly, Fig. 8b demonstrates that LLM label accuracy improves as TDHigh increases, supporting our hypothesis that LLMs are more effective capturing temporal dependencies in trajectory comparisons.
These results validate conflict resolution rules in our PSI.-hased inter-modal fusion, confirming that context- specific predicates (VDHigh gh and TDHigh) can enhance preference labeling reliability by aligning each modality with its respective strengths, in addition to considering modality-speciffe confidence. This structured approach only reduces noise but also improves the overall robustness and interpretability of the inter-modal preference aggregation process.
B Details of Keyframe Extraction
In this section, we provide details of our hybrid method in extracting keyframes from robot trajectories. Given a trajectory a, we propose three complementary methods extracting keyframes. The final keyframe set, denoted as kvis(a), ). formed by taking the union the selected frames from all three methods along with the first and last frames of the trajectory. Below, we detail each method:
B.1 Near-Zero Velocity Detection
Near-zero velocity detection aims identify moments where the agent's motion is minimal, which often corresponds to sub-goal completions (e.g., grasping, placing) or phase boundaries in Iocomotion tasks. This method leverages the intuition that significant actions or transitions in a task are often preceded or followed
by periods of reduced movement even complete stillness. In our setting, a robot trajectory is defined as sequence of state-action pairs:
{(st, a))),
(17)
where each time step consists of state vector s, E R2 and an action vector a, Rd The state vector s typically encapsulates the environment and robot configuration at time f, such as target position, joint angles, end-effector pose (position and orientation), and potentially other relevant sensor readings. The action vector represents the commands issued to the robot's actuators, like joint velocities or torques. The combined observation-actie vector then given by:
[st, as) E R Rot4,
(18)
By concatenating the state and action vectors, we create comprehensive representation of the robot's instan- taneous context, capturing both its current configuration and the commands being executed. The L2 velocity between consecutive frames is defined as:

---

da Σ
= |x441
x 2
(#++1,4 - 20,1)²
(19)
where x, captures both the state and action at each time step. A timestep + is selected as a keyframe if the velocity magnitude is below predefined threshold:
Kzero {t + }d0
(20)
where 6, is the velocity threshold that controls the sensitivity to small movements. A smaller value of 6, results subtle pauses in the motion. Conversely, larger 6, only
in the selection of more keyframes, capturing identifies keyframes where the robot comes more significant halt.
We set & to 0.005 for manipulation tasks in MetaWorld, 0.025 for high-dimensional manipulation tasks in ManiSkill2, and 0.065 for locomotion tasks DeepMind Control Suite (DMC). These values are chosen based on task characteristics and observation-action dimensionality across environments. MetaWorld tasks typically involve smooth, low-dimensional end-effector control, where even small pauses (e.g.. before grasping) are meaningful and should be captured. ManiSkill2 features dimensional state-action spaces. Therefore, a moderately larger threshold needed to account naturally higher baseline velocities. In contrast, DMC locomotion tasks involve rhythmic, continuous movement patterns and higher physical velocity variance, requiring larger threshold to isolate meaningful slowdowns (c.g., stance phase transitions contact events).
complex manipulation scenarios with higher-
B.2 Smoothing Residual Peaks
This method aims to identify keyframes that correspond to significant changes in the robot's motion, such as sharp turns, sudden accelerations or decelcrations, and other high-curvature segments of the trajectory. By comparing the original, raw trajectory with smoothed version, can highlight these abrupt transitions as deviations with large residual errors. The residual error is computed as:
de tuu
=
X x
Σ
(Te $2,)
(21)
where is the smoothed trajectory point, typically computed using moving average filter:
x: 2k +1
Σ
X
(22)
Here, the moving average filter acts as low-pass filter, effectively smoothing out high- figh-frequency components in the trajectory that correspond to rapid changes in motion. The smoothing window size is controlled by the parameter k. A larger ls produces smoother baseline trajectory x,, increasing sensitivity raw trajectory x₃- The value should he chosen based on the typical temporal scale of meaningful motion transitions in the task: short for fine-grained manipulation, longer for rhythmic locomotion.
local spikes in the
Frames with the top K largest residual errors are selected:
Ksmooth = Top- ({fe: e > 5c,)
(23)
where the maximum number keyframes to extract using this method, and 8, is residual error threshold that filters out minor fluctuations due to noise.
We set 2, 5. and 6, = 0.01 MetaWorld, reflecting short-horizon low cow-freuuency transitions typical of single-object manipulation. For ManiSkil12, which features higher-dimensional and more dynamic manipulation behaviors, we use slightly larger window & = 4, K 8, and relaxed threshold 8, = 0.02 to tolerate high-frequency noise. For DMC locomotion tasks, set A 6. K 10, and 6. 0.04, as abrupt transitions (e.g.. foot contact, turning) occur lower frequency with higher magnitude. These settings ensure that the residual- based method captures meaningful structure transitions across environments with diverse temporal dynamics.
B.3 Change Point Detection (CPD)
Change Point Detection (CPD) offers powerful approach to segment a robot's trajectory into distinct phases characterized by different motion dynamics. These phases often correspond to meaningful parts of the task, such as reaching, grasping, moving, and releasing an object. By identifying the boundaries between these phases, we extract keyframes that represent transitions between different stages of the robot's behavior.
25

---

We implement the Pruned Exact Linear Time (PELT) algorithm using the ruptures library which minimizes penalized cost function to efficiently detect change points in the trajectory data:
min ICTISHSTTO CT
C( + 8M
(24)
Here, To = and TM++1 represent the start and end the trajectory, and 71, ......... TMr are the detected change points that segment the trajectory into M homogeneous phases. The cost function C(-) quantifies intra- segment consistency, typically using the sum of squared L2 deviations. The penalty term & controls the trade-off between segmentation fidelity and the number of segments: larger / values lead coarser segmentations (fewer change points), while smaller values allow for finer-grained segmentations.
We set 20 for MetaWorld, which generally contains short-horizon manipulation behaviors with 3-5 semantically distinct phases. For high-dimensional manipulation in ManiSkil12, we use = 30, reflecting its more complex task structures with frequent mid-task corrections and multiple contact events. For locomotion tasks in DMC, we set 3 40, thythmic gaits tend to repeat smoothly over time, and fewer change points expected to capture major phase transitions (e.g., stance-to-swing, to-swing). These values are chosen reflect the temporal structure and motion complexity each task environment. The resulting change points TM are used directly as keyframes:
Kepd {1I, 72.,,,...., TM
(25)
These change points correspond to moments where the motion dynamics of the trajectory undergo a significant shift, making them valuable for summarizing high-level hehavioral transitions during task execution.
B.4 Combining Methods for Keyframe Extraction
To achieve comprehensive and robust selection of keyframes, recognize that each of the aforementioned methods captures different aspects of salient motion. Near-zero velocity detection identifies periods of stagnation, smoothing residual peaks highlights abrupt changes, and change point detection pinpoints transitions between distinct motion phases. By combining the keyframes identified by these complementary approaches. we aim to obtain more complete representation of the robot's 's task execution. The final set of visually significant keyframes, denoted as Kivis, is obtained by taking the union of the keyframe sets generated by each individual method (KCerro KCamoohh and Kepaa). Additionally, trajectory are always included in our set of keyframes, we explicitly add the first frame (index 1) and the last frame (index T) to the combined set:
ensure that beginning and end of the entire
Kviss Kezero U Kemooth U Kepd U (1,7)
(26)
Empirically, the parameter settings specified above yield about 10-20 keyframes per trajectory on Meta World, 15-20 on ManiSkil12, and DMC, providing balanced summary without overwhelming the downstream multimodal evaluator.
C Prompts and Example Outputs
C.1 Synthetic Preference Generation
this section, we provide prompt templates used to elicit the preference judgments from the VLM and LLM in We also provide the details of the trask_dcsrription, which at run-time is filled with the natural- -aauual-langugge goal statement for the current environment, in Table
Fig. |
C.2 Foresight Trajectory Generation
In this section, we provide the prompt template for foresight trajectory generation, Fig. 10a as well example inputs the prompt with Morlds DoorOpen task, as shown in Fig. [0b] The foundation model instructed first to sketch a high-level motion plan, then to translate that plan into exccutable Python code, and finally to diversify the result by changing initial conditions and motion parameters. An example of the resulting executable Python code for the Door Open task, generated using the prompt, is provided in Fig. ||
26

---

Table
Task deseription for each task used in our experiments.
Task Name
Task Description
Button Press
to press button on a surface
Door Open
to open door with a revolving joint. to sweep the object into the target area
Sweep Into
PickSingleYCB
to pick up random object sampled from the YCB dataset and move it to a random goal position
StackCube
to pick up red cube and stack it on top of green cube and let go of the cube without it falling
PegInsertionSide
pick up an orange-w white peg and insert the orange end into the box with hole in it
Hopper Stand
to stabilize a planar one-legged hopper initialized in a random pose, encouraging upright posture with minimal torso height loss control a planar bipedal walker to move forward with target velocity.
Walker Walk
Prompt Template for VLM Evaluation
Prompt Template for LLM Evaluation
Intiaa Systems
Initial System:
leyfiame reprevnting ting toall momennt tiaedtor..
cataa each time step.
Impute
Impute:
deampttio::
(baak, description)
images nwo trapcttries, each repreventdd by set leyftarme.
Two rolectories each cors sting of 9rd staps, farnotteda as follows:
Step E Trajectory Analysts
Step I: Trajectory Analysts
tngectary
trakctooy
Describee the robott befoviee aqpaaed enl in each lightam.. everts
Decorize the pehwetin..
onticul seatts
the rokk god or diccns Callue
Urpanior your responue cherryy by theeciogg
your dler valestory.
Preference Decision
Step Preference Label Decitlon
Boord the prev our yols, compare the two trojectoriss terme of actineement
Based on the prev ous stalynk, compare the two trajectories in terme of god achinvement
Repby
the goalis batter advenved trgectory!
of the poul bettor acheovdion in Irlectooy 1 the
oi the god better aewed in trjectory
traictery
is uncler. you uriurr iidecsiost
drdession
Reply Dormat Lakel: [6 7 -11
Supli Formal Labell
-1]
Step
Verification and Confidence Score
Raflect tud andyya carfer Uhe chove ldbel consntant with the clservee alton and outoores uncarsinty.
Reflect oe the Wirtl anchss and confim if the chosen tibel consisent with the corfidenee between where meanns confidence and complete untemaieyy
confitente means
Racly Toomat
paply rorrat
Final Tebblll 18 7
(9-11)
Reasoning:
your confidence))
Sesacning: [beked sxplanation of your label and contidence)
(a) Illustration of the VLM prompt.
(b) Illustration of the LLM prompt.
Figure 9: Prompt templates used for synthetic preference generation in PRIMT.
C.3 Hindsight Trajectory Augmentation
The prompt template for counterfactual trajectory augmentation is shown in Fig. 12 In this template, kvisti, kvist2 denote the keyframe indices extracted using the methods described in Sectior Examples of the generated counterfactuals are provided in Fig. 20
D PSL Inference Details
We provide additional details on the Probabilistic Soft Logic (PSL) inference procedure used in our inter-modal preference fusion module.
27

---

Foresight Trajectory Generation Prompt
Example Inputs and Requirements MetaWorid
Trijectory Fomat:
Fomett
You zre a istats plrningg agant tseldd with genersting diverse, sena nt cally mean night noost trajectorie sirabted menbo Jatton or locomotion caccs Your and creete righ- bosstrappig in frainin
1. [0:2]
End eftecter (RC2) ponation Cartenlan spece,
phydical
2. [3)
Grapper opentng level: closed,
3. [628]
Object ponition (doer handlel. 20, normaliced c (9,
31
Inpute
[T:30)
deject oclentation (quaternion, 42)
5. [31:13]
PCP dalta action (6x, ay, asl
descripte cescripton:
[34)
arippec tozque control (socmellzad)
2 Tralectory Fornet and Conctro nur
(Pregectocy Fermas) ioou pappoyttcyy.coom Constouints; Erwiorment Setup (Rave conment, Settpp
TrajectoryConstteee Constrainss:
Trajectory wist contain essotly 150 stope
Stap High Level Motion Plan Decompositio
Each teanaition mest be smcoth (no iarge discontimuttion batween consectiive abspai
Desumpane the 3 dhares ligh leol phows tur each oae cieely spedyys
Prate No me
Subaoal whnh conchtin spotial configuation defive success phase?
Strttay: ty phoyical
aeson reecec?
Antioon walone [11 IRcP deltas grigper torgue) want be changes
Consraim Handing
Kiwmaatc constratee
Tap by Format
Envionment.eetup:
("ptaxe"
"aubgoal"
"atcatagy"
Robot: Saryer are with top down grapper
"coratrainta"
Roskepaoe frame
x-axis: left right
Stap Code Tra rnaiattoon for Motion Primitives
y-axle:
Tronsatt the hightieet action plnn noo excorable Python cods.
each
uctured, reusable motion primttvet that adopt
Geagper logie:
diftrerr
-ey considendding
Cantral Logk: Imptement conteeazo costrdl tage smooth trars taen Fleaibity rputa
Close (3.0) during interection
oyen again during release
Initial values peonided:
ob start: Initial object positicn (30): [0.1, 0.95, 0.15)
Aoply Pounat
abi start: mittial object ociantation (401: 10, 0. 9)
(3B):
Retum
Step Trajectory Diveriffaatin Sampling
trajectory
stzategy anr_pacu
...
both:
Sererate a full trajuctosy by aequetiaily compoaing functimnsl moticn plasss.
Iniiaal Conditins Samnie from ditribution sturt states (ng difivent otgect locaioon
Strutegs Paramnnn: Aquat trpectory leval conndl (nus approach height speact
bxajeotary []
Phase se
exjedtory
[xa) platal plase appreach(...) exajentary en traj_ptatel
Seply Forrat
These mow txso poes_ phase (eleese( trajectory teeo_phere_n
rateategy contiig: 1oreh
"tzaSeotary" E...1
raturn trajectory
(a) Foresight trajectory generation prompt.
(b) Example inputs the Metaworld environment.
Figure 10: Prompt templates for foresight trajectory generation in PRIMT.
Let X denote the set of observed atoms, which include modality-specific predictions and trajectory- level context features, such VLMLabel(T), LLMLabel(T), ConfHigh (M). VDHigh, and TDHigh. Let Y denote set of target atoms, corresponding to the final preference decision to be inferred: FinalLabel(T)) for E 1).
Variables.
HL-MRF Formulation. PSL defines a hinge-loss Markov random field (HL-MRF) over the target atoms Y. representing a log-linear probabilistic model:
P(Y x)
1 Z exp
Σ
m
(27)
where w, is the weight assigned to the i-th rule, and (Y,
[max(0,, ((Y, the relaxed hinge-loss potential derived from the rule's linear distance to satisfaction e, The normalization constant Z integrates over the feasible soft assignment space:
Σ wi (xXx))) dY.
Z
exp
(28)
rE(o,1
Inference is performed via convex optimization over continuous variables Y. subject to any additional linear constraints.
Lukasiewicz Relaxation. Given two grounded atoms Az E [0, 1]. PSL uses the following relaxation rules for conjunction, disjunction, and negation:
28

---

Eromple Python Code for Door-Open Foresigtt Trijetory Generation
atrulage.ooo unify.
phoplets, slape, grip ptastet,, grip and on
one peraral
pripper emsoritty.
Centrate full 100 stop traectory a door ooen task 50
stspue B
Ing =
gro
for rangsb lepa:
g10.
to vetory is composde of mastor prosse
gne - aaappo grourain alpta pripuend
o2) stat6/o0)
Graug Lover gripper and dose.
10g tig.appondt-iid atephis, pr pou, obj orl, zeroud), telque-orrgee returis Dig
light pper.
trgectory tojuahass??
keluse Stop pulling
pel Transport
-nburrap
def thase.tarssporsstn
obuori
in Luttt oeent Inttal wilees redloiing
put ci handle widl lelling an],nl
segmentiron
prb, raeque, posl
briph Reer, inital or poer open lnval (D
any. olyes Thamo posities laby
trgushadd plose bettgorr
Jul phoc rew.noigooo sless slurk and, toroue
crrsegy.con cerigg oBier Mecion poramater
unre int www.eon (nl
anjort
above
Cayy place
pripper
yull dx pul iksianss along ants
relarn phae gropptoos saus, srip start, srip and, longue.
"Pt snall seriics Bn
torque torgua-unoeeyyc www. 02] oneri statd on on)
ggrts fance:
when prisping
del aegreenti tart, goal steps, gmap tonque, dluj po. onl
Hdkill Outional
constanta
trjectory -- toj_ilssed
interpolate TCP between
Renurta: rdaermyli
Phose Reteas
calculate deita at each stsp. and hull 15D vecter.
Dos-euilLtng Larget
Tray =
tngeddary- II
stepa-25,
for ranpahomett
11
Plase Approsh
(.Oolsl II
o oI alsta suat abyhe goal
ccars int, enni 1es).
ingasppeddts lid mephrop. gr oni,oos, shlont cets. retarm
P
Ingactary teg phosel
D, tompre-B.. i
www.erreeeeerry...oom sui--ds (100, 1.5)
del copporr grts, oh onijonn detta
approachssaan, goul, prip. tonpee, ob) pes, (hio
31 TCP ES ont, de 2hin pois, 11 niy
goal
ronque
trajedtary traj phase!
stmm segmento goal, dbj pm, cbj ant
tog np aerestt
Sinn - gro
(mpp7:331] - -bjjon
scgill 14)-
return step
Figure 11: An example of the resulting executable Python code for the Door Open task.
Hindsight Trajectory Augmentation Prompt
Hindsight Trajectory Augmentation Prompt
Inttial System:
Step
Counterfactual Variant
unebyrrencbe ganeading
cumnel in meserr bead un atrutand
Gener
-rafenoed ligellay
Imputs:
Carael Stepo Litadl JaEArTT
ttaak descripetion)
labss
inguts (1ewt,)))
Perfaned Trojectoy: (br), preffered))
Seeest bey stop from
fied Gissll steps nediying state-actoon feature
faym indiam (5) au)let(c)))
reanoning. ressonnngg
stop to revess the to ecar iss preference.
-Lght smoohhoo rgll-3 (3233-00 steps before and dooeds mainta continttt
Step I:
Ab Abduction-ii - Identify Causal Steps
stess the sraedtar must remain stent cheerinol
Aychee
reeerry and mine which steat are cousilyy rerpont
banky Fornat
preferred
2dt ced Stap: Lindex of the acdidied acep)
Foour meatingtu tatt-releart chonges in mone- behovvor that lieyy infueenedd
zdit Descesption: [besed natozad language desceiption of the
The pnefwrene decitio
may provided leyframe considites but you
them
Countactactual Tewjectocy: [tuet soonat trajectory description cocacatect with the ootginel. Ampus decnat)
Juply Turmat
Cester, Stage: [LAab of sdent iied step indioee in the peetecred
tmajectary)
Pessoniag: seplanation why abepa orinionl.
juigrent)
Figure 12: Prompt template used for counterfactual trajectory augmentation in PRIMT
A,AA, max(0, A₁ - 1}
(29)
A.V min(A1+ + A2, 1}
(30)
-A 1 A
(31)
These allow soft rules to be represented as continuous linear functions over truth values, enabling efficient convex inference.
Template Rule Grounding. PSL rules are first written templates over logical variables (e.g.. labels T modalities M), and then instantiated into multiple ground rules. For example, the template:
YY, M IsAgree(T) A ConfHlgghMM) FinalLabel(T')
29

---

is expanded into six rules by enumerating all Y (-1,0, and M é (VLM, LLM), such as:
IsAgree(1) A (LnfHigh(LLM) -> FinalLabel(1)
Example Rule Expansion.
Consider the rule:
VLMLabel(1) A ConfHigh(VLM) A VDHigg -> PinalLabel(1)
Its logical equivalent is:
VLMLabel(I) "ConfHigh V -VDHigh v FinalLabel(1)
Using Lukasiewicz semantics, the corresponding relaxed satisfaction distance is:
=
VLMLabel(1) + ConfHigh(VLM)) + VDHiigh FinalLabel(1) 3
and the resulting hinge-loss potential is:
[max(0, 0)}² [0, VLMLabel(1) ConfHigh(VVMM + VDHigh - FinalLabel(1 33]
E Details on Task Environments
this section, we provide details tasks used our experiments.
E.1
Meta MetaWorld
The Button Press task requires the robot to manipulate its end-effector to press a specific button located on surface. The goal is to make contact with and depress the button. The reward is typically sparse, given only when the button successfully pressed. This task tests the robot's ability to achieve precise movements and interact with small objects the environment, as visualized Fig. 13a
Button Press
(a) Button Press Task
(b) Door Open Task
(c) Sweep Into Task
Figure 13: Visualizations of MetaWorld Tasks
Door Open The Door Open task involves the robot grasping the handle of hinged door and opening it to a desired angle. The robot must first reach and grasp the handle, then apply the appropriate force and motion swing the door open. This task assesses the robot's 's ability to perform sequential manipulation actions and interact articulated objects, shown Fig. 13b
Sweep Into
Sweep Into task challenges robot to use end-effector (or an object held by it) sweep a target object into a designated goal region. This requires the robot to make contact with the object and apply a sweeping motion to push it into the target. This task evaluntes the robot's 's ability to perform planar manipulation and reason about pushing dynamics, illustrated in Fig. 13c
E.2 ManiSkill
PickSingleYCCB YCBB The PickSingle YCB involves robot picking up a single object from YCB object set. The robot must perceive the object, plan a grasp, execute the grasp, and lift the object to a desired height or location. This is a fundamental object manipulation task that tests grasping and lifting skills, as visualized in
Fig. 14a
30

---

(a) PickSingle YCB Task
(b) StackCube Task
(c) PegInsertion Task
Figure 14: Visualizations of ManiSkill Tasks
StackCube The StackCube task requires the robot to pick up one more cubes and stack them top of each other in stable configuration. This task builds upon basic picking and placing skills and introduces the challenge achieving a stable multi-object arrangement, as shown in Fig, 14b
PegInsertionSide
The PegInsertionSide task involves the robot inserting peg into hole the side of surface. This task requires precise alignment and control the robot's end-effector to successfully insert the peg without collision. tests fine manipulation and spatial reasoning, illustrated in Fig. 14c
E.3
DeepMind Control Suite
The Hopper Stand task from the DeepMind Control (DMC) suite involves single-legged hopping robot. The goal to control robot stand upright and maintain balance without falling. This task assesses the agent's 's ability to learn stable control policies for a dynamically challenging system, as visualized in Fig. 15a
Hopper Stand
(a) Hopper Stand Task
(b) Walker Walk Task
Figure 15: Visualizations of DeepMind Control Suite Tasks
Walker Walk The Walker Walk task from the DeepMind Control (DMC) suite features a bipedal walking robot. The objective is to control the robot to walk forward with a consistent velocity without falling. This task evaluates the agent's ability learn complex locomotion gaits and maintain stability articulated system, as shown Fig. 15b
F Additional Implementation Details
In this section, we provide further implementation details of our experiments.
Baselines
We implemented the baselines RL-VLM-F [4), RL-SaLLM-F [66). and PrefCLM (15) using the source code released by the authors. To eliminate non mone differences, we replaced the original prompts in RL-VLM-F and RL-SaLLM-FF F with the shree-step chain-of thought prompt used in PRIMT. This three-step prompt
31

---

be considered an enhanced version of the two-step reasoning used in their original papers, with additional self-verification step. The trajectory inputs also the same in PRIMT: kvis(() for RL-VLM-F and (en)(() for RL-SaLLLM-F.
The only exception is PrefCLM. This method assumes access the environment code, which uses to generate diverse evaluation functions by prompting multiple LLMs, and then fuses the resulting preference scores using Dempster- Sompster-Shafer theory. Due to its reliance on environment-speen sppeciffc code, we retain its original setup. For faimess,, we evaluate PrefCLM under its zero-shot variant, which does not involve few-shot expert selection of the generated functions and human-in-the-loop correction. The crowd size set 10, following the original configuration the paper.
We implemented the PrefMul baseline ourselves by feeding the multimodal trajectory representations, i.e., (o) and (o)) into multimodal LLM (e.g.. gpt-4o), using the same CoT prompt as PRIMT to elicit the preference label.
The PrefGT baseline follows the scripted teacher approach introduced in [8]. It serves as an upper- upper-bound bound oracle that has complete access to the benchmark's ground truth reward function. Although such access is infeasible woald robotic systems, provides useful reference to assess the best possible performance that any preforence-basee learning method could achieve.
For any pair trajectories o and o² the oracle computes their cumulative reward returns:
R(o))
r(s az)
where r(ss, ax) is the environment's ground-truth reward time step L. The oracle then assigns the preference
label as:
if R(a > R(o
r =
if R(o < RR -1 if R(o*) = R)
F.2 Reward Learning
We adopt PEBBLE 7 as the PbRL backbone for methods. PEBBLE performs unsupervised pre-training tr-inrniing to maximize state entropy and initialize the policy, followed by off-policy reinforcement learning using Soft Actor-Critic (SAC) for policy optimization. During training, all reward values in the replay buffer are relabeled whenever new reward model is learned.
For the reward model, we implement the image- reward network as described in their original paper for RI.-VI.M-F, For all other baselines using the standard PEBBLE 7 reward model, we adopt 3-layer ensemble architecture following their design.
selecting informative queries, we adopt an uncertainty-baat query selection strategy following B. Specifi- cally, we measure the uncertainty of preference predictions either by computing the variance across an ensemble of preference predictors. We then select the top-Nguery trajectory segment pairs with the highest uncertainty as query candidates to clicit preference labels.
For all experiments conducted in this work, the hyperparameter settings used for reward learning are summarized Table 2
Table 2: Hyperparameters used for reward learning.
Hyperparameter
Value
Trajectory segment length
100
Feedback frequency
5000
Maximum feedback samples
20000
Number of foresight trajectories
200
Max counterfactuals per trajectory
5
F.3 Policy Learning
For all methods evaluated in this work, we follow PEBBLE 7 and adopt Soft Actor-Critic (SAC) as the off-polie policy reinforcement learning algorithm. To ensure fair comparison, all methods share the same actor-critic architecture and hyperparameter settings during policy learning.
32

---

Throughout training, use the same network configurations and learning schedules as those in the original PEBBLE implementation. A summary of these hyperparameters is provided in Table
All experiments conducted workstation equipped with five NVIDIA RTX 4090 GPUs.
Table 3: Hyperparameters used for SAC.
Hyperparameter
Value
Hyperparameter
Value
Initial temperature Learning rate
0.1
Batch Size
1024
0,0003
Optimizer
Adam
Critic target update freq
2
Critic EMA
0.005
(31, 32)
(0,9, 0.999)
Discount
0.99
Hidden units per each layer
1024
G
Additional Experimental Results
this section, we provide additional experimental results further support our main findings. These include:
More visualizations of label distributions learned reward outputs across tasks.
An ablation study on the impact of foundation model (FM) backbone selection.
Policy visualizations comparing PRIMT and baseline behaviors.
Qualitative analysis of both the foresight and hindsight trajectory generation modules.
Real-world deployment on Kinova Jaco robot.
G.1 More Visualizations of Label Distributions and Learned Reward Outputs
We present additional visualizations of label distributions and learned reward outputs other tasks in Figs. 16 and 17 addition to those shown in Fig.
We observe that PRIMT consistently produces higher-quality synthetic feedback with fewer indecisive labels. Furthermore, the reward functions learned by PRIMT exhibit more precise saateacctoon-oe credit assignment, resulting in reward patterns that better align with ground-truth task progress.
Correct Labeling
Incorrect I Laheling
No Preference
Button Press
PegInsertionSide
Hopper Stand
Percentage (%5
Percentage (%))
(%)
30
60-
80
PRIMT
GireFore Intra
wit Inter
RL VLM-F
PRIMT
wio.ForeGien
w/o Intra
win Inter
RL- VLM-F
PRIMT
ForeFor
win Intra
Inter
RL- -LM-E
Sweep Into
PickSingleYCHB
Percentage (%))
Percentage (%)
8 8
se-
PRIMT
Goreeer
wio Intra
wio Inter
RL- -LM-
PRIMT
ForeFerteien
weo Intra
win Inter
RI- VLM-F
Figure 16: Extra visualizations the distribution of preference labels, showing the proportion of correct, incorrect, and indecisive labels across different methods.
33

---

PRIMT
Wio.CauAx
w/o. HindAug
GT Task Roward
RL-VLM-F
Swsgp I930
Makke: Wark
Patsingecc
CT
Figure 17: Extra visualizations the reward aligmment, comparing the learned reward outputs of PRIMT, ablations, and baselines against ground-truth reward.
G.2 Ablation Study on FM Backbone Selection
We further conduct an ablation study to investigate the influence of the FM hackbone two representative tasks: Door Open from the MetaWorld henchmark and PeglnsertionSide from the ManiSkill benehmark. We report the performance PRIMT and best-performing baseline on each task using gpt-4o, and compare it the same setup using gpt-4o-rp 4o-min,, a weaker model variant.
As shown in Fig. 18 both methods experience a performance drop when using gpt-4o mimini which is expected due the reduced reasoning and perception capabilities of the smaller model. However. PRIMT demonstrates greater robustness, maintaining performance more effectively compared to the best-performing baseline. This again highlights the benefit of hierarchical fusion and trajectory synthesis design improving generalizability under lower-capacity foundation models.
PRIMT W. GPT-4o
RL-SaLLM-F W. GPT-4o
PrefCLM W. GPT-4o
PRIMT W. GPT-4o-Mini
RI-SaLLM-F GPT-4o-Mini
PrefCLM GPT-4o-Mini
Door Open
PogInsertionSide
100
100
Success Rate
40
Steps
Figure 18: Ablation study on FM backbone selection.
G.3 Qualitative Analysis of the Foresight Trajectory Generation Module
To better understand the behaviors produced by our foresight trajectory generation module, we visualize several examples of LLM- LLM-ggeerated trajectories and compare them to early-stage random explorations. The visualizations shown in Fig. 19
34

---

(a) LLM- generated trajectories that successfully complete the task, showing diversity in robot starting positions (c.g., the first vs. the other two), and in strategies such varying gripper height to press the button (sccond and third rows).
(b) LLM- generated trajectories that are not task-successful but still semantically meaningful and task-aligned. Although these examples fail press the button, they exhibit structured behaviors such approaching the button and making plausible press attempts.
(c) Early-stage random exploration trajectories, which are typically noisy and unstructured. Compared to these. non-successful LLM LLMe-genratee trajectories can serve preference anchors due their goal-oriented structure.
Figure 19: Examples of LLM ALM--enerated bootstrap trajectories from foresight generation, compared with random trajectories from early-stage policy rollout.
We observe that the L.M--generted trajectories exhibit typical patterns. In cases, shown in Fig. 19a
they successfully complete task while showing meaningful diversity execution, as varying the robot's starting position or adjusting the gripper height when pressing the button. In other cases, as shown in Fig. 19b the generated trajectories fail to complete the task but still demonstrate semantically aligned and structured behaviors, including approaching the button and attempting press in plausible manner.
35

---

In contrast, as shown in Fig. 19c early-stage random exploration trajectories are generally unstructured and uniformly poor, lacking coherence goal-directed motion. Notably, the unsuccessful but structured LLM-generated samples serve useful preference anchors when paired these random trajectories. When used in combination with uncertainty- based query selection, such pairings are more likely to selected for labeling due to their high predictive disagreement across the preference ensemble. This leads to more informative preference queries in the early stages of reward model training.
G.4 Qualitative Analysis of the Hindsight Trajectory Augmentation Module
To further illustrate how the hindsight trajectory augmentation module operates, we visualize two counterfactual variants of preferred trajectory, as shown in Fig. 20 These examples demonstrate how the LLM generates counterfactual samples by performing minimal interventions key causal steps.
In Fig. 20a the LLM introduces delay by holding the object longer hefore releasing it, simulating temporally adjusted strategy while preserving the of the trajectory. Fig. 20b the I.I.M adds brief hesitation before grasping the object, representing minimal behavioral perturbation that weakens preference without altering the trajectory's overall structure. By applying such targeted, minimal edits to causally critical steps, the preference difference becomes more sharply attributed to specific actions or states, effectively isolating the cause of preference reversal. This sharpens the causal signal and provides more informative supervision for the reward model, thereby improving temporal credit assignment and helping the model Icarn which precise behaviors lead to better or worse preferences.
Preferred Trajectory
Counterfactual Trajectory
Hold the cube for a longer period of time before releasing it to the target area (a) LLM generates a counterfactual trajectory by adding delay: holding the cube for a longer period of time before relcasing it to the target area, while other steps remain the same as in the preferred trajectory. Preferred Trajectory
Counterfactual Trajectory
Add short hesitation before grasping the cube
(b) LLM generates counterfactual trajectory by inserting a brief hesitation before grasping the cube, while keeping the other steps identical to those in the preferred trajectory.
Figure 20: Examples LLMM-gnnrated counterfactual trajectories based on minimal intervention causally critical steps.
36

---

G.5 Policy Visualizations of Different Methods
To gain more insights into how different methods shape policy behaviors, visualize the robot behaviors learned PRIMT and the best-performing baseline several representative tasks. The visualizations shown in Fig. 2nand Fig. 22] We observe that PRIMT leads to more efficient and coherent manipulation and locomotion behaviors, demonstrating better task completion strategies and smoother trajectories compared to the baseline.
RL-SaLLM-F
PRIMT
(a) Policy Visualizations between PRIMT and RL-SaLLM-F on the Door Open task.
PrefCLM
PRIMT
(b) Policy Visualizations between PRIMT and PrefCLM the Button Press task.
PrefCLM
PRIMT
(c) Policy Visualizations between PRIMT and PrefCLM the Sweep Into task.
Figure 21: Examples of policy visualization between different methods MetaWorld.
G.6 Preliminary Experiments on a Bimanual Manipulation Task
We further conducted preliminary experiments on more difficult bimanual manipulation task, TwoArmPegIn- Hole from RoboSuite. As shown in Table PRIMT still significantly outperforms the baseline RL-VLM-F and approaches the PbRL oracle PrefGT. These results suggest that PRIMT potentially generalizes well to more complex, high-dimensional settings.
Table 4: Preliminary results the TwoArmPeglntHole task: success rates (%) during training.
Method
Max SR (%)
Mean SR (%)
Final SR (%)
PrefGT
78.40
67.17
78.03
RL-VLN BL-V
32.66
25.48
32,66
PRIMT
65.06
53.26
64.15
37

---

RL-SaLLM-F
PRIMT
(a) Policy Visualizations between PRIMT and RL-SaLLM-F on the Hopper Stand task.
RL-SaLLM-F
PRIMT
(b) Policy Visualizations between PRIMT and RL-SaLLM-F on the Walker Walk task.
Figure 22: Examples of policy visualization between different methods DeepMind Control.
G.7 Comparison with Dense-Reward RL
To further evaluate the effectiveness of our method, introduce new baseline GT, an RL policy trained using ground-truth dense rewards provided the benchmark environments and SAC. This baseline enables direct comparison between PRIMT, preference-based learning, and conventional dense-reward are reported in Figure 23) which summarizes the final success rates and episode returns of GT, PrefGT, the best-perforiing baseline, and PRIMT across all tasks.
Learning curves
Dog Dpa
Shop
Figure 23: Learning curves including the ground-truth dense-reward RL baseline.
The results indicate that GT (dense-reward RL) achieves the highest performance all tasks, expected access to dense, step-level supervision. Nonetheless, PRIMT attains between 1.9% 17.3% of the performance gap relative to GT, demonstrating that PRIMT effectively approximates the behavior of dense- reward RL without requiring explicit reward shaping. This highlights PRIMT's scalability and efficiency in scenarios where reward engineering extensive human feedback is infeasible.
38

---

G.8 Qualitative Reward Alignment Analysis
To qualitatively examine the relationship between learned rewards and the ground-trath task rewards shown in Figure performed a quantitative analysis using the Rº coefficient to measure reward alignment different methods. The baseline RL- LL-VLM-F was selected as reference comparison. The computed R² values for each task and method are summarized Table
Table 5: Rº Coefficient Analysis (Reward Alignment with Ground Truth).
Task
PRIMT
w/o CauAux
w/o HindAug
RL-VLM-F
PegInsertionSide
0.56
0.28
0.23
0.37
PickSingleYCt
0.84
0.01
0.34
-0.05
StackCube
0.78
-1.31
-2.28
-1.50
ButtonPress
0.87
0.68
0.53
-0.61
DoorOpen 0.64
-1.19
0.15
-4.72
SweepInto 0.88
0.83
0.73
-0.27
WalkerWalk
0.33
0.19
0.02
-2.29
The results show that PRIMT achieves consistently higher πt² coefficients compared to other variants and the bascline RI-VVI--F, indicating stronger alignment between its learned reward signals and ground-truth task rewards. Both the hindsight augmentation and causal auxiliary loss components contribute positively this improvement, suggesting their importance in enhancing credit assignment quality during reward learning.
G.9 Real-world Experiments
We further evaluate our method in the real world on two tasks from the Robosuite benchmark [64] Block Lifting and Block Stacking. In the Block Lifting task, the goal control robotic arm to lift cube certain height. the Block Stacking task, the objective is to place one cube on top of another. We adopt PrefCLM 15 as real-world baseline.
Policies are first trained in simulation using Robosuite [64], and then deployed on a physical Kinova Jaco2 robotic arm. We leverage RoboSuite's 's comprehensive sim2real capabilities, which include dynamics randomization and sensor modeling with realistic sampling rates, delays, and noise corruption.
During training. closely matched the robot model, camera configuration, and workspace setup with the real hardware. The control frequency and action space were also kept consistent. To ensure safety during deployment, we imposed soft constraint: if the turning angle or acceleration exceeded a threshold, the corresponding action was discarded. Performance results are shown in Figure 24] We evaluated each model over 10 randomized trials (e.g.. varying initial block positions). PRIMT achieved 7/TO successful lifts and 6/10 successful stacks, whereas the PrefCI.M M baseline achieved 4/10 and 2/10 successes, respectively. More videos of real-world experiments are on our project website.
Lifting
Stacking
PRIMT
PrefCLM
Figure 24: Demos of real-worl weal-world manipulation experiments on the Block Lifting and Block Stacking tasks.
H Discussion on Assumptions, Limitations, and Broader Impact
H.1 Assumptions and Limitations
In this work, we make the following assumptions, each grounded in prior work and empirical observations: 1) We assume that foundation models (FMs), including LLMs and VLMs, pretrained large-scale and diverse
39

---

corpora, enabling generalization across wide range of environments and tasks; 2) We assume that VLMs are capable of analyzing sequences of visual inputs extract spatial and semantic cues, while LLMs can interpret temporal structures, reason over structured inpuls, and generate coherent lask-related feedback; 3) We assume that LLMs, when guided by structured prompts and domain domain-aligned vocabulary, are capable of generating or editing robot trajectories in coherent and task-relevan manner. This assumption builds upon recent work demonstrating the ability of LLMs to plan actions 411, generate code-based controllers [8] and perform trajectory synthesis via natural language [36]; and 4) We assume that task progress can be inferred at least partially through either visual sequences (e.g., keyframe differences, spatial transitions) or textual descriptions (e.g., arrays of actions or states).
Our work is fundamentally built upon FMs, and therefore, like other FM-based methods, it inherits certain dependencies on the capabilitiess and limitations of these models. While this reliance is inherent to the design, we have taken steps mitigate it: for example, leveraging multimodal feedback to enhance the quality of preference signals and adopting structured code-generation paradigm to improve the quality of generated trajectories. As shown our ablation study on LLM backhone selection, PRIMT maintains reasonable performance even when using a weaker model cgpt-4o-mini. indicating robustness to FM capacity.
Another limitation lies in the cost associated with FM usage. Specifically, our multimodal feedback generation, fusion module, and bidirectional trajectory synthesis involve multiple FM calls. To provide clearer picture of the resource requirements, we present detailed resource usage comparison in Table/oland the corresponding cost-performance trade-off's Table
We observe that compared to the RL-VLM-F and Se-LLM-F baselines, cost and training time increased moderately (by 38 47% and 30-699 69%, respectively). While this increase is notable, performance gains were substantial (+19- 1 117%). resulting in efficiency improvements of 2.0x and 1,4x, respectively. This justifies the additional FM usage in PRIMT, given the performance benefits are notable. More importantly, compared to the performance achieved by collecting human feedback (represented by PrfSIt) where expert-sourced teachers provide ground truth preferences, PRIMT achieves comparable performance (within 1-3%) while reducing estimated human annotation costs by over 92%. (based on 0.05-0.1 per preference label for 20, 000 queries on platforms such as Prolific and MTuuk). This highlights PRIMT's scalability and practicality as paramount step to expensive human-in- -n-tann---tte-ooo loop methods. Therefore, believe PRIMT strikes a good balance between performance and cost-effectiveness, providing a practical path toward scalable preference learning.
Table 6: Resource usage comparison across different methods and environments
Average Usage Cost ($)
Average Training Time (h) MetaWorld / ManiSkill DMC
Method
MetaWorld / ManiSkill DMC
RL-VLM-F
84.14 / 57.42/ / 84.27
4.3 5.1
Sa-LLM-F
83.22 / 55.31 89.69
5.2 5.6
PRIMT
120.42 / 79.73 / 124.01
6.8 / 7.3 7.6 N/A
Human
1,000-2,000
FM cost estimated based on GPT-4o API pricing at the time of experiments
Human cost estimated from 20,000 preference queries for task at $0.05-$00 $0.10 per label on platforms like Prolific and MTurr
Table 7: Cost-performance trade-off comparison of PRIMT against baseline methods
Cost
Time
Performance
Efficiency'
Baseline
Meta World / ManiSkill DMC
vs RL-VLM-F
+43%/499%/447%
-58%%/443%/4699
+95% 1096/+1177// 17%
2,0x
vs Sa-LLM-F
+5%%
+32%/+109%/41999 +31 9//30%//446%
1.4x
Human (PrefGT)
-92%/-95%/-929
- -1%
47x
[t] Efficiency = Averuge performance gain/ / Average resource increase
Performance is measured using the final return from the learning curves presented in Figure 2 Values shown are relative changes across Meta Boetlorldd/ ManiSkill / DMC environments, respectively
H.2 Impact Statement
This work explores the integration of foundation models into preference-basee based reinforcement learning (PbRL), which aims to improve learning efficiency and robustness through multimodal feedback and trajectory synthesis. By leveraging VLMs and LLMs. our approach reduces the need for extensive human supervision, potentially broadening access PbRL techniques in domains with limited annotation resources.
40

---

However, the use of FMs raises concerns around data privacy, transparency, and decision- making fairness. As with other FM-based methods, system may inherit unintended biases from pretrained models, which could impact downstream behavior. We encourage future work to explore bias mitigation, model auditing, and safe deployment strategies, especially high-stakes or safety-critical applications.
Overall, our work offers promising step toward scalable and generalizable robot learning systems. Moreover, the underlying principles, multimodal feedback fusion, and trajectory-leve reasoning, can extend beyond robotics to broader sequential decision-making scenarios in fields such as education, healthcare, and interactive agents.
41


Reviews:
[{"review_id": "uvYajLh7nh", "reviewer": "Reviewer_r5ng", "summary": "The paper describes a preference-based RL technique where feedback is produced by VLMs and LLMs. LLMs can interpret trajectories, e.g. in the form of state-action pairs, but such descriptions can be incomplete and the LLM can be confused by some fine-grained spatial interactions. VLMs offer an alternative, but have their own limitations, e.g. in terms of recognising subtle temporal dynamics. Hence, the work's pursuit of a multimodal approach where feedback from both LLMs and VLMs (with help of a keyframe extraction method) is fused. Preference fusion is supported by the use of Probabilistic Soft Logic (PSL) helping to define rules for fusion (i.e. agreement, conflict resolution and indecision). The issues of low-quality trajectory pairs early in training and inaccurate state-action-level credit assignment (i.e. attributing preference differences to specific states / actions) are overcome with the help of foresight trajectory generation (using LLMs) and hindsight trajectory augmentation. \n\nResults are generated for two locomotion and six manipulation tasks. PRIMT is compared against a VLM-based method, LLM-based methods, a naive multimodal approach, and the use of expert-designed reward functions (representing the upper bound, \"prefGT\"). gpt-4o is used as the FM backbone (section G.2 also compares to the use of the weaker gpt-4o-mini model). An ablation study supports the inclusion of the major components of the technique. Results show that PRIMT is competitive with the prefGT baseline and normally significantly outperforms other baselines.", "questions": "1. For PrefCLM, how many LLM agents were in the crowd? \n2. G.6 can you quantify \"superior performance\" for this experiment?\n3. Could you comment more on the LLM query cost? (beyond H.1). What if you simply compared techniques based on time or compute costs? Are there obvious ways to reduce the compute costs? (beyond cheaper/local models)", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: Interesting work that has produced significant improvements in performance.  \nWeaknesses: The FM usage represents a very significant amount of compute", "was_revised": true, "final_justification": "The authors have provided useful answers to my questions. I remain positive and will retain my rating at 4.", "rebuttal": "# Response to Reviewer r5ng\n\n**Dear Reviewer r5ng,**\n\nThank you for your encouraging assessment of our work and for your thoughtful comments. We are grateful for the opportunity to respond to your questions as follows:\n\n---\n\n### **Q1. Crowd Size in PrefCLM**\n\nIn our experiments, we followed the original setting in the source paper and used a crowd size of 10 for PrefCLM. We will add this detail, along with other implementation details of the baseline, in Section F.1 of the revised manuscript.\n\n---\n\n### **Q2. Quantifying “Superior Performance” in G.6**\n\nIn the real-world Block Lifting and Block Stacking tasks, we evaluated each model over 10 randomized trials (e.g., varying initial block positions). PRIMT achieved 8/10 successful lifts and 6/10 successful stacks, whereas the PrefCLM baseline achieved 4/10 and 2/10 successes, respectively. This represents a doubling of the success rate compared to the baseline. We will include these success rates in Section G.6 to support the “superior performance” claim with concrete evidence.\n\n---\n\n### **Q3 & W1. FM Usage and Cost**\n\nWe acknowledge that our method involves greater usage of foundation models (FMs) than prior work, due to its multi-agent nature. To provide a clearer picture of the resource requirements, we present a detailed **resource usage comparison** in **Table 1**, and the corresponding **cost-performance trade-offs** in **Table 2**.\n\nNote that we exclude PrefCLM from this comparison for fairness. This baseline assumes access to environment code and prompts LLMs to generate evaluation functions directly in code, thereby avoiding LLM-based preference queries and incurring minimal FM usage cost.\n\nWe observe that compared to the **RL-VLM-F** and **Sa-LLM-F** baselines, **cost** and **training time** increased moderately (by **38–47%** and **30–69%**, respectively), while **performance gains** were substantial (**+19–117%**), resulting in **efficiency improvements of 2.0× and 1.4×**, respectively. This justifies the additional FM usage in PRIMT, given the performance benefits it enables.\n\nMore importantly, compared to the performance achieved by collecting human feedback—represented by PrefGT where expert scripted teachers provide ground-truth preferences—PRIMT achieves **comparable performance** (within **1–3%**) while **reducing estimated human annotation costs by over 92%** ( based on ~$0.05–$0.10 per preference label for 20,000 queries on platforms such as Prolific and MTurk). This highlights PRIMT’s scalability and practicality as an alternative to expensive human-in-the-loop methods.\n\n Overall, we believe PRIMT strikes a **good balance between performance and cost-effectiveness**, providing a practical path toward scalable preference learning.\n\n---\n\n### **Table 1: Resource Usage Comparison**\n\n| Method         | Usage Cost ($)     | Training Time (h)      |\n|----------------|-------------------------------|-------------------------|\n| **RL-VLM-F**   | 84.14 / 57.42 / 84.27         | 4.3 / 5.1 / 4.5         |\n| **Sa-LLM-F**   | 83.22 / 55.31 / 89.69         | 5.2 / 5.6 / 5.2         |\n| **PRIMT**      | 120.42 / 79.73 / 124.01       | 6.8 / 7.3 / 7.6         |\n| **Human**      | 1,000–2,000 (all envs)        | N/A                     |\n\n*Values across MetaWorld / ManiSkill / DMC, respectively*  \n\n*FM cost estimated based on gpt-4o API pricing at the time of experiments*\n\n*Human cost estimated from 20,000 preference queries at $0.05–$0.10 per label on platforms like Prolific and MTurk*\n\n---\n\n### **Table 2: Cost-Performance Trade-off**\n\n| Baseline | **Cost** | **Time** | **Performance** | **Efficiency†** |\n|----------|----------|----------|-----------------|-----------------|\n| **vs RL-VLM-F** | **+43%/+39%/+47%** | **+58%/+43%/+69%** | **+95%/+117%/+68%** | **2.0×** |\n| **vs Sa-LLM-F** | **+45%/+44%/+38%** | **+31%/+30%/+46%** | **+32%/+109%/+19%** | **1.4×** |\n| **vs Human (PrefGT)** | **−92%/−95%/−92%** | **—/—/—** | **−1%/−3%/−2%** | **47×** |\n\n*Values across MetaWorld / ManiSkill / DMC, respectively*  \n\n*Performance is measured using the final return from the learning curves presented in Figure 2 of the paper*  \n\n*† Efficiency = Average performance gain / Average resource increase*\n\n---\n\nAdditionally, our ablation study in Section G.2 suggests that our method shows reasonable **robustness to cheaper foundation models**: GPT-4o-mini achieves 94% cost savings with only 16-26% performance reduction, resulting in 13× better cost-performance efficiency and suggesting potential for more accessible deployment.\n\nThat being said, we fully acknowledge the importance of **reducing FM usage cost**. PRIMT already adopts several strategies to this end. For example, we extract **keyframes** from trajectories instead of passing full image sequences to the VLM. For foresight generation, we leverage **code-based** generation to synthesize multiple trajectories offline in a single batch, reducing LLM cost. We also implement **parallel querying** for VLM/LLM-based preference generation to reduce training time and wall-clock latency.\n\nIn the revision, we will include a more detailed cost analysis and expand the discussion of optimization strategies to make our approach even more cost-effective and scalable for broader adoption.\n\n---\n\nThank you again for your time in reviewing our paper and rebuttal!"}, {"review_id": "kr9azpdwlu", "reviewer": "Reviewer_Cuxg", "summary": "This paper introduces PRIMT, a foundation model-driven framework for preference-based reinforcement learning (PbRL) that addresses challenges in synthetic feedback quality, query ambiguity, and credit assignment. PRIMT employs a hierarchical neuro-symbolic fusion strategy to integrate multimodal feedback from large language models (LLMs) and vision-language models (VLMs). Experiments across 2 locomotion and 6 manipulation tasks validate the effectiveness of its components.", "questions": "1. How much did it cost to tune the gpt-4o api for the whole experiment?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to follow.\n2. The experiments are very adequate and comprehensively demonstrate the effectiveness of the algorithm.\n3. Real-world experiment on Kinova Jaco robot enhances the paper’s credibility, demonstrating practical applicability beyond simulation.\n\nWeaknesses\n1. The method is overly complex and difficult to re-implement, and too much hyperparameters make it hard to tune the parameters.\n2. Lack of hyperparameter analysis. Conducting more experiments on hyperparameter tuning or ablation studies could be beneficial.", "was_revised": true, "final_justification": "Resolved issues:\n1. Hard to tune the parameters.\n2. Lack of hyperparameter analysis.\n\nUnresolved issues:\nN/A", "rebuttal": "# Response to Reviewer Cuxg\n\n**Dear Reviewer Cuxg,**\n\nThank you for your positive assessment of our presentation, contributions, and experiments, and for your insightful comments. We are glad to respond to your concerns below.\n\n---\n\n### **W1 & W2: Complexity and Hyperparameters**\n\nWe acknowledge that PRIMT consists of multiple components. However, each component is **well-motivated by specific limitations** in existing PbRL approaches:\n\n- Multimodal feedback fusion addresses the complementary weaknesses of single-modality evaluation (as demonstrated in Appendix A)\n- Foresight generation tackles early-stage query ambiguity when random policies produce uniformly poor trajectories  \n- Hindsight augmentation improves credit assignment through targeted counterfactual reasoning\n\nAs shown in our ablation studies (Figure 3 and Section G.1), each component contributes meaningfully to performance improvements, validating our overall architecture. Our goal is to develop a general zero-shot PbRL framework by fully leveraging foundation models, and the multi-agent/component design reflects this objective.\n\n---\nRegarding hyperparameters, we clarify that they fall into two distinct categories:\n\n**1. Backbone PbRL training parameters**  \nPRIMT is designed to be plug-and-play with any PbRL backbone. In our experiments, we adopt the PEBBLE setup for reward model and policy learning (e.g., optimizer, learning rate, batch size, training schedule), as detailed in Appendix F.2 and F.3. These were **shared across all baselines and tasks without individual tuning**.\n\n**2. PRIMT-specific module parameters**  \nThe hyperparameters related to PRIMT modules were mostly **held constant across all tasks**:\n- The confidence balancing weight α in Eq. 5 was fixed at 0.5\n- PSL rule weights were uniformly set to [3, 1, 1, 1] for the agreement, conflict resolution, and indecision rules (Eqs. 6, 7, 8, 12)\n- The number of foresight and hindsight trajectories per round was fixed as described in Appendix F.2\n\nThese parameters were not extensively tuned, and the consistent performance across diverse tasks spanning multiple benchmark suites suggests that PRIMT **does not heavily depend on extensive hyperparameter tuning to be effective**.\n\nThat said, we fully agree that conducting more systematic hyperparameter analysis would be valuable for the community. While current resource constraints prevent extensive additional experiments, we acknowledge this as an exciting direction. In the Discussion section of the revised manuscript, we will highlight this as a promising direction for future research, such as sensitivity analysis of VLM/LLM agent quantities in crowd-checking and the impact of foresight/hindsight trajectory amount requirements on performance. \n\nFor reproducibility, we have provided comprehensive implementation details in the appendix and plan to release our codebase upon acceptance to facilitate such investigations by the research community.\n\nWe will revise the manuscript to better foreground these design motivations and hyperparameter grouping for clarity.\n\n---\n\n### **Q1: LLM Usage Cost**\n\nThe average cost per task ranges from 79-124 USD across different benchmark environments, making training on all 8 tasks cost approximately $850 in total. We provide detailed cost breakdowns and trade-off analysis in our response to Reviewer r5ng (Tables 1 and 2) for additional context.\n\nCompared to FM-based baselines, PRIMT increases costs moderately (38-47%) while delivering substantial performance gains (19-117%), achieving 1.4-2.0× efficiency improvements. PRIMT also achieves comparable performance to expert human teachers (within 1-3%) while reducing human annotation costs by 92-95%. \n\n---\n\n\nThank you again for your constructive feedback and positive evaluation!"}, {"review_id": "BvgopbVPr4", "reviewer": "Reviewer_87iK", "summary": "This paper introduces a framework for Preference-based Reinforcement Learning (PbRL), called PRIMT, from Foundation Model (FM) feedback. It addresses two core challenges: (i) the ambiguity of early-stage feedback due to low-quality trajectories at the beginning of training and (ii) the difficulty of credit assignment in the case of trajectory feedback. To improve feedback quality, PRIMT combines trajectory-level evaluations from LLMs and VLMs using a probabilistic soft logic approach. This \"fusion\" process incorporates confidence scores, cross-modal agreement, and trajectory context to resolve conflicts. To handle sparse and ambiguous training data, PRIMT also uses LLMs to generate synthetic trajectories in the form of \"foresight\" generation (populating a replay buffer) and \"hindsight\" generation (edits to create counterfactuals). The authors evaluate and compare PRIMT on various simulated robotics tasks and two real-world robot demos. They also conduct ablations to quantify the impact of each component.", "questions": "1. How often do \"hindsight\" edits yield invalid or infeasible actions, and how do you filter them? The paper mentions that LLMs produce \"plausible\" edits, but there’s no quantitative analysis of failure rates or their impact on learning stability. \n2. As far as I can tell, you prompt the FMs to return a \"confidence\" score via chat templates. How realiable is that really? Are there other alternatives, e.g., using log-probabilities from the LLM or maybe other uncertainty quantification methods?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths:\n\n1. The experimental evaluation in the paper is extensive and thoroughly performed. Various ablations are provided which anticipated many of the questions I had when reading the paper. The chosen baselines and the experimental setup make sense and strongly support the paper's claims. The evlauation also covers an exceptional breadth of test scenarios including DMC, MetaWorld, Maniskill and a real-world example. Additional metrics such as label accuracy w.r.t. a proxy oracle feedback model are also provided and ablated. \n2. The structured fusion of VLM and LLM feedback using probabilistic soft logic is interesting and shows significant empirical improvements over using either modality alone. Similarly, the problems of query ambiguity and credit assignment are clearly motivated and convincingly addressed using the proposed \"foresight\" trajectory generation and the \"hindsight\" trajectory edits. \n3. The presentation of the methodology, the related work, and the explanations of the proposed approach are detailed and clear.    \n4. I also appreciated the extensive section on the paper's assumptions and limitations in Appendix H. \n\n#### Weaknesses:\n\n1. My biggest concern is that much of the methodology feels like \"stacking heuristics\" and informed guesswork. This is probably expected (to some extent) when working with FM-based feedback and aiming for better empirical performance in complex robotics tasks. My concern is slightly alleviated by the extensive experimental evaluation and many ablations, which also lead to my overall positive assessment of the paper despite the above. \n2. The cost associated to the extensive use of FMs is a natural drawback of the proposed approach. Here, it would have been great to see the total number of FM queries used per experiment, the cost associated with them, and similar metrics. If these are available, I recommend to report them in the paper.", "was_revised": true, "final_justification": "In my opinion, this is a good paper. One of the biggest strengths and a key reason for my positive assessment is the extensive experimental evaluation. The experiments include an unusually large amount of different tasks and various ablations.", "rebuttal": "# Response to Reviewer 87iK\n\n**Dear Reviewer 87iK,**\n\nThank you for your thorough evaluation and positive assessment of our methodology and experiments. We sincerely appreciate your recognition of our contributions and are glad to address your insightful questions below.\n\n---\n\n### **W1: \"Stacking Heuristics\" Concern**\n\nThank you for raising this thoughtful point. We understand the concern regarding modular complexity due to the multi-component nature of our method, and we appreciate your acknowledgment that our extensive ablations help alleviate this issue.\n\nHere, we would like to further clarify our design rationale. PRIMT is not simply a set of stacked heuristics, but rather a **principled, problem-driven framework** where each module is designed to address a specific, empirically observed challenge in FM-driven preference learning. \n\nOur design process began with the observation (Appendix A) that **LLMs and VLMs exhibit distinct limitations yet complementary strengths** (Figure 8), motivating multimodal feedback fusion. We further identified **context-dependent performance patterns** across modalities, leading to the adoption of Probabilistic Soft Logic (PSL) for more interpretable and robust integration. Finally, we found that even high-quality synthetic feedback could not resolve **inherent challenges in PbRL**: query ambiguity and credit assignment, which we address through foresight and hindsight generation.\n\nThese components are not standalone heuristics but are mutually complementary, forming an integrated system where fusion improves label quality, foresight provides early-stage preference anchors, and hindsight enhances fine-grained credit assignment—together aiming to facilitate a generalizable and robust zero-shot PbRL framework.\n\nWe will revise the main paper to make this design motivation clearer and more explicit.\n\n---\n\n### **W2: Extensive FM Usage**\n\nWe appreciate the opportunity to provide more insights on the resource usage of PRIMT. Due to its multi-agent architecture, PRIMT incurs higher FM usage compared to prior FM-based baselines. To provide a more transparent analysis, we have compiled a detailed comparison of **compute costs and efficiency trade-offs**, presented in **Tables 1 & 2** in our response to Reviewer r5ng.\n\nCompared to FM-based baselines (RL-VLM-F and Sa-LLM-F), PRIMT increases FM usage cost and training time **moderately** (by 38–47% and 30–69%, respectively), but yields **substantial performance improvements** (+19–117%), resulting in **efficiency gains of up to 2.0×**. We believe this could justify the added cost.\n\nMore importantly, when compared to collecting **real human preference labels**—represented by PrefGT where expert scripted teachers provide ground-truth preferences—PRIMT achieves **comparable performance (within 1–3%) while reducing estimated annotation costs by over 92%** (based on $0.05–$0.10 per label for 20,000 queries on platforms like Prolific or MTurk). This suggests PRIMT offers a scalable and cost-effective alternative to traditional human-in-the-loop methods.\n\nFurthermore, our ablation in Section G.2 demonstrates that PRIMT remains effective even with lower-cost FMs: using GPT-4o-mini yields 94% cost savings with a 16–26% drop in performance, resulting in a **13× improvement in cost-performance ratio**. This highlights the framework’s **flexibility for resource-constrained scenarios**.\n\nWe will integrate this analysis into the revised version to better contextualize PRIMT’s scalability and cost-efficiency.\n\n---\n\n### **Q1: Quality of Hindsight Trajectory Samples**\n\nTo ensure the quality and informativeness of counterfactual trajectories generated during hindsight augmentation, we apply a two-step filtering process: \n\n- We compute the L1 distance between the edited state-action pairs and their original counterparts, retaining only those with minimal deviation to ensure the counterfactual reflects a plausible, fine-grained edit.\n\n- The filtered counterfactuals are then passed through the LLM-based intra-modal fusion module to verify that they are indeed less preferred than the original. \n\nThese ensure the final counterfactuals support the loss in Eq. 15 for meaningful credit assignment during reward model training.\n\nHowever, we did observe that not all counterfactuals strictly adhere to physical constraints—for example, some edits introduce abnormal accelerations or sharp turns to make a trajectory appear less preferred. \n\nWe initially considered adding a separate physical feasibility filter, such as VLM-based rendering evaluation or simulation-based rollouts (as in [1]). However, we ultimately decided against it for several principled reasons:\n\n- First, as established in recent work [2], **generated trajectories do not need to adhere to strict physical constraints**, as they serve solely for preference comparisons to train the reward model, rather than for extracting policies, rollong out in real-world or constructing a world model.  Given that our **reward model is Markovian**—with immediate rewards depending only on the current state—physically implausible transitions do not undermine reward learning, as long as the local preference signal remains valid.\n\n- We also conducted **pilot experiments** and found that our hindsight trajectory generation aligns well with this principle—despite occasional infeasible transitions, the generated edits consistently support effective reward learning. \n\n\nThat said, we also observed that our chain-of-thought trategy for generating counterfactuals encourages more coherent and feasible edits compared to directly prompting LLMs to generate full alternative trajectories (as used in the baseline RL-SaLLM-F). \n\nWe will clarify these design choices and provide additional insights and examples in the revised version.\n\n---\n\n### **Q2: Confidence Estimation**\n\nEstimating preference confidence plays a critical role in our multimodal feedback fusion pipeline. We explored several popular methods [3], including logit-based/entropy-based uncertainty, chain-of-thought (CoT) self-evaluation, repeated sampling, and external critic models. \n\nHowever, our design goal was to maintain general applicability across both open- and closed-source foundation models, as well as to reduce inference overhead. Therefore, we avoided logit-based or entropy-based metrics, which are often unavailable in API-based FM deployments. We also excluded external evaluators, which would significantly increase cost.\n\nFinally, we adopted a dual-component confidence estimation (as shown in Eqs. 4-5) that combines complementary signals:\n\n- Internal certainty: Average self-reported confidence from predictions that align with the final consensus decision\n- Behavioral consistency: Agreement ratio across multiple queries with randomized trajectory orderings\n\nThis formulation ensures that the final confidence captures both internal certainty and robustness under minor input perturbations, improving the reliability of modality-specific estimates. In preliminary experiments, we observed that this combined signal led to improved label accuracy compared to using either component in isolation.\n\nWe will clarify this motivation further in the revised version.\n\n---\n**References:**\n\n[1] Kwon, N., Di Palo, N., & Johns, E. (2024). *Language Models as Zero-Shot Trajectory Generators*. IEEE Robotics and Automation Letters.\n\n[2] Tu, S., Sun, J., Zhang, Q., Lan, X., & Zhao, D. (2025). *Online Preference-Based Reinforcement Learning with Self-Augmented Feedback from Large Language Model*. Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS).\n\n[3] Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., & Gurevych, I. (2023). *A Survey of Confidence Estimation and Calibration in Large Language Models*. arXiv preprint arXiv:2311.08298."}, {"review_id": "SUyhFK0dbq", "reviewer": "Reviewer_397q", "summary": "This submission introduces PRIMT, a preference-based RL framework. It uses foundation models (LLMs and VLMs) to annotate preference labels of trajectories and synthesize both foresight and hindsight counterfactual trajectories. All these efforts are for better learning reward models. RL trained with the learned reward models demonstrates faster convergence and better asymptotic performance over several baseline methods on three benchmarks, ranging from locomotion to robot arm manipulation. Real-world deployment is also provided.", "questions": "- What is the exact implementation of the reward model? How did you train that? \n- What is the RL algorithm used?\n- Fig 4b, what is the blue curve for?\n- Fig 4b, it would be more informative to quantify reward alignment. Could consider metrics such as R2 coefficient.\n- Sec. 4.5, real-world deployment, do you perform sim2real transfer? Is it state based or image based? Or do you simply replay trajectories collected in sim?", "limitations": "Limitations not discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The submission is well written and clearly presented.\n- The intuition of using multiple modalities for preference-based RL makes sense.\n- Experiment evaluation and ablation studies are comprehensive.\n- The proposed method performs quite good.\n\n## Weaknesses\n- While the proposed method is impressive, it is a system consisting of many existing techniques. It's unclear how the proposed method compared to others at a system level.\n- While authors claim PbRL alleviates reward engineering burden, demonstrated tasks are still largely simple. It would be beneficial to see if the proposed method can work on more complicated robotic tasks, such as mobile manipulation, or long-horizon tasks such as MineRL where reward is sparse.\n- Authors compare against several PbRL methods. It would be more informative to see how the proposed method compare against RL trained directly on ground-truth dense reward (if exist), or imitation learning agents trained with demonstrations.\n- Runtime complexity and required compute not discussed. I imagine the proposed method would take longer time and consume more compute than baseline methods.\n- Missing some details, see questions below.\n- No limitation discussed.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in authors' response. I'd like to keep my original positive rating.", "rebuttal": "# Response to Reviewer 397q\n\n**Dear Reviewer 397q,**\n\nThank you for your encouraging feedback on the clarity of our presentation, the motivation behind our methodology, and the rigor of our experiments. We are glad to have the opportuniy to repsond to your insightful comments as below.\n\n---\n### **W1: System Composition**\n\nWe appreciate this thoughtful comment and would like to clarify that while PRIMT builds upon emerging directions in FM-based learning (e.g., using foundation models for preference labeling and trajectory synthesis), each component is **specifically designed to address distinct, well-motivated challenges** in PbRL and introduces **novel architectural elements**.\n\nFor example, to our best knowledge, we are **the first to propose a multimodal evaluation setting** in PbRL, and introduce a **neuro-symbolic fusion mechanism** to overcome feedback inconsistency in existing single-modality approaches. We also propose **bidirectional trajectory synthesis** to tackle early-stage query ambiguity and poor credit assignment—introducing, for the first time in PbRL, **counterfactual reasoning coupled with a causal auxiliary loss** for fine-grained credit attribution.\n\nRegarding **system-level comparisons**, we conduct explicit evaluations between PRIMT and several **full-system baselines**, all of which support complete PbRL pipelines and represent distinct system configurations:\n\n- RL-VLM-F: a unimodal system using only VLMs for feedback.  \n- RL-SaLLM-F: a unimodal LLM-based system augmented with foresight trajectory generation.  \n- PrefCLM: a crowd-sourced unimodal fusion method.  \n- PrefMul: our constructed naïve multimodal system.  \n- PrefGT: a human-in-the-loop PbRL upper bound using scripted expert preferences.\n\nAs shown in Section 4 of the manuscript, PRIMT consistently outperforms FM-based baselines, resulting in **non-trivial gains at the system level**. \n\nAdditionally, following your suggestion, we include a **new RL baseline with dense rewards** for additional reference. Please see our response to your **W3** for details. This comparison further highlights that PRIMT can be even competitive with dense-reward performance in some tasks, despite relying only on **FM-based zero-shot preference supervision**. \n\n---\n\n\n### **W2: Task Complexity**\n\nOur task selection **follows established practices in the PbRL literature**, where MetaWorld and DMC are widely adopted as standard benchmarks for preference-based methods. To further expand the evaluation scope, we have already incorporated ManiSkill, **a newer and more challenging benchmark** featuring complex object interactions and high-dimensional control.\n\nPbRL itself is still an emerging field, and even human-in-the-loop PbRL algorithms currently face limitations when applied to highly complex or long-horizon tasks. That said, we conducted **preliminary experiments on a more difficult bimanual manipulation task**, *TwoArmPegInHole* from RoboSuite. As shown in Table 3, PRIMT still significantly outperforms the baseline RL-VLM-F and approaches the PbRL oracle PrefGT. These results suggest that PRIMT potentially generalizes well to more complex, high-dimensional settings.\n\nIn the revision, we will include this result and expand the discussion section to provide further insights on extending PRIMT to more complex and long-horizon robotic tasks.\n\n---\n\n**Table 3: Preliminary results on the TwoArmPegInHole Task in Success Rate during training**\n| Method      | Max SR | Mean SR | Final SR |\n|-------------|--------|---------|-----------|\n| PrefGT      | 78.40  | 67.17   | 78.03     |\n| RL-VLM-F    | 32.66  | 25.48   | 32.66     |\n| PRIMT       | 65.06  | 53.26   | 64.15     |\n\n\n---\n\n### **W3: Comparison with Dense-Reward RL**\n\nFollowing your suggestion, we added a new baseline **GT** (RL policy trained using ground-truth dense rewards provided by the benchmark environments and SAC) to provide additional comparison insights.\n\nUnfortunately, due to the new rebuttal policy, we cannot provide updated learning curves. As an alternative, we extract key performance data in Table 4, showing the final success rates/episode returns of GT, PrefGT, the best performing baseline, and PRIMT across all tasks.\n\n---\n\n**Table 4: Final Performance (Success Rate % / Episode Return) Across Tasks**\n\n| Task               | GT (Dense RL) | PrefGT (Oracle PbRL) | PRIMT (Ours) | Best Baseline |\n|--------------------|---------------|----------------------|--------------|---------------|\n| PegInsertionSide   | 82.38         | 77.65                |79.63    | 53.28         |\n| PickSingleYCB      | 88.14         | 78.89                | 72.85        | 49.00         |\n| StackCube          | 82.51         | 76.64                | 73.95        | 44.73         |\n| ButtonPress        | 97.60         | 95.67                | 93.21        | 80.19         |\n| SweepInto          | 84.15         | 68.38                | 74.15   | 66.87         |\n| DoorOpen           | 100.00        | 98.10               | 95.00        | 74.00         |\n| WalkerWalk         | 970.89        | 969.44               | 937.58       | 854.59        |\n| HopperStand        | 924.09        | 909.65               | 906.58       | 704.89        |\n\n---\n\nAs expected, the results reveal that GT (dense-reward RL) consistently outperforms all PbRL methods, even PrefGT (oracle PbRL), due to its access to dense, step-level supervision. However, we observe that **PRIMT reaches within 1.9%–17.3% of the performance of RL policies trained with ground-truth dense rewards** across a variety of tasks. This result further highlights the potential of PRIMT towards scalable and efficient PbRL without requiring hand-crafted dense rewards or human feedback, reinforcing its practicality in real-world settings where reward engineering is difficult or infeasible.\n\n---\n### **W4: Computational Cost**\n\nThank you for highlighting this important practical consideration. Due to its multimodal architecture, PRIMT does involve higher computational overhead compared to other FM-based baselines. To provide a more transparent analysis, we have compiled a detailed comparison of compute costs and efficiency trade-offs, presented in **Tables 1 & 2** in our response to Reviewer r5ng.\n\nCompared to existing FM-based methods (RL-VLM-F and Sa-LLM-F), PRIMT increases **FM Usage costs by 38–47%** and **training duration by 30–69%**. However, this moderate cost increase yields **substantial performance gains of 19–117%**, resulting in **overall efficiency improvements of 1.4–2.0×**. \n\nLastly, as discussed in Appendix G.2, PRIMT can also **run with lighter foundation models** (e.g., GPT-4o-mini), achieving 94% cost savings at a 16–26% drop in performance—still yielding strong cost-effectiveness and adaptability for lower-resource settings.\n\nWe will incorporate this analysis into the final version to more clearly situate PRIMT’s computational demands within the broader cost-performance landscape of PbRL methods.\n\n---\n\n### **W5 & Q1-Q2: Implementation Details**\n\nFirst, we want to apologize that due to page limitations in the current version, we could not include all necessary implementation details in the main paper. The comprehensive experimental details are provided in **Appendix F** of current manuscript.\n\nAs noted in Section F.2&3, we use **PEBBLE as the PbRL backbone**. Following the standard PEBBLE design, we adopt a 3-layer ensemble architecture for the reward model and SAC as the RL algorithm. \n\n---\n### **W6: Discussion of Limitation**\n\nWe would like to clarify that a detailed discussion of limitations is already included in **Appendix H.1** of the current submission. We apologize if this was not sufficiently visible, and will move a condensed version of the limitations discussion into the main paper in the revision for improved clarity.\n\n---\n\n### **Q3 & Q4: Figure 4b**\n\nThank you for pointing out the confusion. The **blue curve in Figure 4b represents the baseline RL-VLM-F**. We will update the legend of Figure 4b for clarity.\n\nThanks for the excellent suggestion of using **R² coefficient to quantify reward alignment**. Below we present the R² coefficients comparing learned rewards to ground-truth task rewards:\n\n---\n\n**Table 5: R² Coefficient Analysis (Reward Alignment with Ground Truth)**\n\n| Task | PRIMT | w/o.CauAux | w/o.HindAug | RL-VLM-F |\n|------|-------|------------|-------------|----------|\n| PegInsertionSide | 0.56 | 0.28 | 0.23 | 0.37 |\n| PickSingleYCB | 0.84| 0.01 | 0.34 | -0.05 |\n| StackCube | 0.78 | -1.31 | -2.28 | -1.50 |\n| ButtonPress | 0.87 | 0.68 | 0.53 | -0.61 |\n| DoorOpen | 0.64| -1.19 | 0.15 | -4.72 |\n| SweepInto |0.88 | 0.83 | 0.73 | -0.27 |\n| WalkerWalk |0.33 | 0.19 | 0.02 | -2.29 |\n\n---\n\nThese results further highlight that both hindsight augmentation and our causal auxiliary loss significantly **improve the quality of credit assignment during reward learning**. We will integrate this analysis into the paper to provide a more quantitative perspective on alignment quality.\n\n---\n### **Q5: Real-World Deployment**\n\nThank you for the question. Policies were trained in RoboSuite simulation using state-based observations and then deployed on the physical Kinova Jaco robot. We leverage RoboSuite's comprehensive sim2real capabilities, which include dynamics randomization and sensor modeling with realistic sampling rates, delays, and noise corruption. \n\nDuring training, we closely matched the robot model, camera configuration, and workspace setup with the real hardware. The control frequency and action space were also kept consistent. To ensure safety during deployment, we imposed a soft constraint: if the turning angle or acceleration exceeded a threshold, the corresponding action was discarded.\n\nWe will clarify the real-world deployment protocol in more detail in Appendix G.6 of the revised version.\n\n---\n\nThank you again for taking the time to review our paper and rebuttal. We hope our responses could address your concerns."}]

Author rebuttal:

Meta-review: This paper tackles the problem of Preference-based Reinforcement Learning in developing the proposed system, PRIMT. This approach how to handle multi-modal feedback through a fusing feedback from a large language model and a vision language model. A neurosymbolic approach is used for this fusion. Next, the approach explores how to populate the PbRL replay buffer without assuming that the demonstrations are optimal -- an approach called "foresight" in this paper. In hindsight, the approach generates counterfacutals for learning as well. The approach is evaluated empirically and with ablations and a real-world evaluation on a Jaco robot. Reviewers felt the authors delivered on what they claimed as contributions. The paper was clear, the topic relevant, and the approach well-evaluated. However, there were concerns about how complex the model is -- perhaps with insufficient design and implementation details. As one reviewer summarized, the paper appears to be basically "`stacking heuristics` and informed guesswork." The appraoch also seems quite computationally expensive. The paper could be improved by providng a better understanding of what components of necessary vs. sufficient and why. Overall, the paper is solid and the real-world demonstration is a plus.
Decision: Accept (oral)