MVR: Multi-view Video Reward Shaping for Reinforcement Learning
Authors: Lirui Luo
Guoxi Zhang
Hongming Xu
Yaodong Yang
Cong Fang
Qing Li
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Shaping; Reinforcement Learning; VLM Reward
Primary area: reinforcement learning
TL;DR: We propose state-dependent reward shaping using multi-view videos that automatically decays visual guidance as agents achieve    desired motion patterns in RL.

Abstract: Reward design is of great importance for solving complex tasks with reinforcement learning. Recent studies have explored using image-text similarity produced by vision-language models (VLMs) to augment rewards of a task with visual feedback. A common practice linearly adds VLM scores to task or success rewards without explicit shaping, potentially altering the optimal policy. Moreover, such approaches, often relying on single static images, struggle with tasks whose desired behavior involves complex, dynamic motions spanning multiple visually different states. Furthermore, single viewpoints can occlude critical aspects of an agent's behavior. To address these issues, this paper presents Multi-View Video Reward Shaping (MVR), a framework that models the relevance of states regarding the target task using videos captured from multiple viewpoints. MVR leverages video-text similarity from a frozen pre-trained VLM to learn a state relevance function that mitigates the bias towards specific static poses inherent in image-based methods. Additionally, we introduce a state-dependent reward shaping formulation that integrates task-specific rewards and VLM-based guidance, automatically reducing the influence of VLM guidance once the desired motion pattern is achieved. We confirm the efficacy of the proposed framework with extensive experiments on challenging humanoid locomotion tasks from HumanoidBench and manipulation tasks from MetaWorld, verifying the design choices through ablation studies.

Published as a conference paper at ICLR 2026
MVR: MULTI-VIEW VIDEO REWARD SHAPING FOR REINFORCEMENT LEARNING
Lirui Luo" Guoxi Zhang"; Hongming Xu², Yaodong Yang Cong Fang! Qing Lif
I State Key Lab of General AI, School of Intelligence Science and Technology, Peking University
2 State Key Laboratory of General Artificial Intellignnce, BIGAI
Institute for Artificial Intelligence, Peking University
fangcongêpku. edu. en, dylan. bylan.1iqian liqing@gmail. com Project page: https://mvr-rl.gihhbit
ABSTRACT
Reward design is of great importance for solving complex tasks with reinforcement learning. Recent studies have explored using image-text similarity produced by vision-language models (VLMs) to augment rewards of task with visual feedback. A common practice lincarly adds VLM scores to task or success rewards without explicit shaping, potentially altering the optimal policy. Moreover, such approaches, often relying on single static images, struggle with tasks whose desired behavior involves complex, dynamic motions spanning multiple visually different states. Furthermore, single viewpoints can occlude critical aspects of an agent's 's behavior. To address these issues, this paper presents Multi-View Video Reward Shaping (MVR), framework that models the relevance of states regarding the target task using videos captured from multiple viewpoints. MVR leverages video-text similarity from frozen pre-trained VI.M to learn state relevance function that mitigates the bias towards specific static poses inherent in image-based methods. Additionally, introduce a state-dependent reward shaping formulation that integrates task-specific rewards and VLM-bbseed guidance, automatically reducing the influence of VLM guidance once the desired motion pattern is achieved. We confirm the efficacy of the proposed framework with extensive experiments on challenging humanoid locomotion tasks from HumanoidBench and manipulation tasks from MetaWorld, verifying the design choices through ablation studies.
INTRODUCTION
When using reinforcement learning (RL), agents learn to optimize task rewards encoding quantitative objectives. For example, the rumning task shown in Fig. 1, the task rewards incentivize the tracking of designated forward speed. In contrast, when learning new skills, humans can benefit from visual guidance tailored the desired motion pattern, thus preventing suboptimal postures. Inspired by this observation, we study the incorporation of visual feedback into task rewards in this paper.
With the development of Vision Languuge Model (VLM)s, an emerging paradigm is to leverage image-text text similarity as rewards Chan et al. (2023); Rocamonde et al. (2024), which guides agents to visit states that best match the textual description of task. However, this paradigm faces three shortcomings. Firstly, as single-image similarity cannot characterize dynamic motions, the generated rewards are mis-specified they incentivize agents to visit the very states with maximal similarity score (Fu et al., 2024). As result, agents cannot learn rhythmic movements involving repeated transitions between different states. Moreover, single viewpoints suffer from occlusion between robot limbs, creating view-dependent bias that further destabilizes dense guidance. Selecting the viewpoint with minimal occlusion often requires extensive human effort and domain knowledge. In addition, most methods linearly add single-view VLM scores to task or success rewards without explicit shaping, which can alter the optimal policy.
Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Visual Guidance
VLM
MVR
Figure 1: The proposed MVR computes visual guidance using VLM and videos collected from multiple viewpoints. In this example, the task requires humanoid robot to run forward. Being captured from different viewpoints, the image sequences encode complementary information and enable comprehensive evaluation of the agent's behaviors. This example also illustrates the pitfall of using image-text similarity for dynamic tasks-running requires rhythmic alternation of legs, but optimizing image-text similarity leads to realizing the pose that best matches "running" repeatedly, The shaping term prioritizes states that establish alternating leg cadence: once a stable gait and the target forward speed are achieved, its influence automatically decreases and the task reward takes precedence (see Sec. 4).
This paper presents Multi-view Video Reward Shaping (MVR), an online RL framework that over- comes these issues. As illustrated in Fig. 1, MVR collects multi-view videos of agent behaviors and estimates state relevance from video-text similarity. This learned relevance constructs a state- dependent reward shaping term that provides dense guidance early on and automatically diminishes as behavior improves, avoiding persistent conflict with task objectives.
We evaluate the effectiveness of MVR on total of 19 tasks from HumanoidBench (Sferrazza et al., 2024) and WerlW (Yu al., 2019). MVR consistently outperforms VI.M-RM (Rocamonde et al., 2024), and RoboCLIP (Sontakke et al., 2024), which are existing VLM-based methods. Furthermore, we validate the design MVR in an ablation study and investigate the influence using multiple views. Lastly, showcase the effectiveness of visual feedback for preventing suboptimal behaviors. The contributions of this paper are as follows.
We propose MVR, framework for generating feedback from multi-view videos.
2. We introduce a state sete-dependent reward shaping formulation that integrates task rewards with VLMMbased guidance and decays its influence as behaviour aligns with the target.
3. We validate MVR's effectiveness through simulated experiments on humanoid locomotion tasks in HumanoidBench and manipulation tasks in Meta World, and present case study to showcase the importance of visual guidance.
2 RELATED WORK
-LM-based and Video-based Rewards Recent work leverages VLMs to generate reward signals through image-text similarity (Chan et al., 2023: Rocamonde al., 2024; Kim et al., 2023; Cui et al., 2024) or goal images (Gao et al., 2023). These methods typically treat the VLM frozen scoring function and then improve reward quality via VLM fine-tuning (Fu al., 2024), binarization (Huang et al., 2024a), ranking-basse based objectives (Wang al., 2024b).
Beyond single-frame similarity, a growing body of work learns rewards directly from videos of task executions. Rewrd (Yang et al., 2024) combines temporal ranking-based reward model with an adversarial (tAll-style) discriminator; PROGRESSOR (Ayalew et al., 2024) learns a progress estimator and applies eeward-wightee regression real robots; and GVL (Ma et al., 2025) uses a Gemini-based VLM to provide zero-shot progression rewards for downstream applications. Generative video models such Diffusion Reward (Huang al., 2024b) and GenFlowRI. (Yu
2

---

Published as a conference paper at ICLR 2026
et al., 2025) define rewards from conditional video diffusion or object-centric flow, while Video- Language Critic (VLC) (Alakuijala et al., 2025) fine-tunes CLIP4Clip-styLe backbone so that a video-language critic can provide transferable rewards for training mngguage-conditione RL policies. REDS (Kim et al., 2025) learns subtask-aware dense rewards from segmented demonstrations, and ReWiND (Zhang al., 2025) trains video-language transformer produce dense language- conditioned rewards and demonstrates real-world RL fine-tuning.
However, most of these approaches define rewards directly on raw image or video observations, without learning an explicit state-space relevance function, or assume access to curated demonstration datasets. Classical example-guie state-only imitation-emulla emulation frameworks such as CIMER (Han et al., 2024) also rely on expert trajectories rather than re-using unlabeled online rollouts. RoboCLIIP (Sontakke et al., 2024) is closer to our setting and uses video-text similarity but provides only sparse trajectory-level rewards. MVR differs by learning state relevance from multi-view videos through paired comparisons and introducing state-dependent shaping that automatically decays, providing dense guidance during online RL while remaining compatible with task rewards.
controllers such as DeepMimic (Peng et al., 2018) and recent
Foundation models have been extensively integrated into RL pipelines (Xu et al., 2024; Moroncelli et al., 2024; Kawaharazuka et al., 2024). VLMs enable planning (Pan et al., 2024; Patel et al., 2023), success detection (Du et al., 2023a), and representation learning (Chen et al., 2024), though most rely on single-frame inputs unsuitable for dynamic tasks. LLMs facilitate planning (brian ichter al., 2022; Shinn al., 2024), exploration (Du et al., 2023b), code-based policy generation (Liang et al., 2023), and reward design (Ma et al., 2024; Xie et al., 2024). World models provide alternative approaches for reward generation (Escontrela al., 2023) and model-based RL (Wu et al., 2024). Our work specifically addresses the unique challenges of using pre-trainee trained VLMs for continuous motion tasks, proposing practical solutions for video-based multi-view reward learning.
RL with Foundation Models
3 PROBLEM STATEMENT
Markov Decision Process The Markov Decision Process (MDP) Sutton (2018) is defined as (S, A, P.
r),, where S and A denote the state space and the action space. P: S A..A -+ A is the transition probability of states, and r x A - IR is the reward function. 7 E (0, is the discount factor. An MDP prescribes an interaction protocol in which an agent repeatedly receives states from an environment, samples actions from its policy m(aas)) S + A(A), and receives rewards. The
value function of (s) E tt= 7 " 'r:] characterizes the expected discounted return of a starting
from state s. The objective RL is to find the optimal policy x such that e. (s) 2 v (s) for any state
The Learning Problem
The video-text similarity computed by models such as VICLIP (Wang et al.. 2024a) is denoted by VL.M and characterizes how well video matches a text description. During policy learning, we periodically sample state sequences s, render them into videos o from different viewpoints, and compute the video-text similarity scores ei vVVLM E). where f is the task description. For notational convenience, we also use v VLLM o') ) for the similarity between two videos and o'. These steps result in dataset D {(s, o, vb VLM (o, E))] Throughout the paper, we use bold symbols (e.g., s) for sequences and plain symbols (e.g.. s) for single timesteps. Given 2, we aim learn state relevance model fMVR S R. with parameters @MVRR to model the relevance of states for the task of interest. With fMR we derive reward function y VL.M S - IR as visual feedback for the agent, complementing the task rewards task
4 MULTI-VIEW VIDEO REWARD SHAPING
The proposed MVR illustrated in Fig. 2. Section 4.1 explains how the relevance model MMV is learned, and Sec. 4.2 introduces the proposed reward function and the entire framework.
3

---

Published as a conference paper at ICLR 2026
4.1 LEARNING STATE RELEVANCE FROM MULTI-VIEW VIDEOS
Challenges Operating in state space S enables efficient reward computation without per-step video rendering, which is crucial for million-step training. However, this design raises two coupled challenges. First, there is semantic gap between states (proprioception, joint angles) and visual features (textures, colors), which prohibits directly regressing video-text similarity from states. Second, MVR renders videos from one of four viewpoints (0", 90" 180", and 270"), which introduces systematic viewpoint biases: for instance, frontal views in Fig. I often yield higher similarities due to better visibility, creating spurious correlations unrelated to behavioral quality. In the remainder of this subsection, we address the semantic gap with Matching Paired Comparisons and the viewpoint- induced bias with Regularizing State Representations.
Matching Paired Comparisons To tackle the semantic gap between states and videos, we propose to match comparisons in the two domains instead of regressing raw similarity scores. Being trained with contrastive objectives, the video-text similarity scores of two videos reflect the probability that one video better matches the task than the other. We claim that such orderings should be preserved between the corresponding state sequences. Specifically, the probability that a video o better matches the task prompt l than another video o' under the Bradley-Terry (BT) model (Bradley & Terry, 1952) is given by
Ivid (o, o') VI (o, E)
(o', f)),
where (x) = (1+ + exp(-x))- is the sigmoid function. Similarly, the probability that a state sequence is better than another state sequence s' is
hstate (s, s')
MVR
n(s') Σ (s)):
#E8
where n(s) is the length of s. Given two samples (s, (I. E)) and (s', o', VOV, to preserve the ordering between videos and state sequences, we propose to minimize:
Trmatciing - Vivaa o') (ngghsaneesss (s,s')) s))) - (1 hiaa o')) log(1 Issmees s
By minimizing this cross-entropy, fMR can capture the relevance of states for the task without being distracted by the visual details. is akin to the objective of preference-basee RL Christiano et al. (2017), but with the crucial difference that we fit MVR to probabilities rather than binary labels. Importantly, (o, o') can originate from different viewpoints because each video shares the same underlying state sequence; sampling pairs across views enlarges the comparison dataset without requiring additional rendering.
Regularizing State Representations The second challenge is the viewpoint-induced bias in &/V.M. different camera angles can change similarity scores for the same underlying behavior. straightforward solution is to randomize the choice of viewpoint, but relevance learning then becomes difficult as we use more viewpoints, since we will have fewer samples from any single informative view. To tackle this data scareity while reducing viewpoint bias, we enhance relevance learning by regularizing state representations with the similarity struciure between video embeddings. Specifically, we parameterize /MVR as (s) = (ggt) .y sytate
(s)), where g S - RRd is a state encoder and nd Rd is learnable vector. The scalar d is the dimension of the state embeddings. This dccomposition decouples representation learning from relevance scoring: Loeg shapes a shared state representation, while Lontching selects a single relevance direction gdd in that space, rather than entangling multi-view alignment and task-specific ranking in single monolithic scalar predictor. Then, we use the following regularization term for relevance learning:
Treg
100 VLM o,) (g stati (st), (st),,"tmn
where (s) = n(s))n1 Σ aes I (s) is the representation of a state sequence. This regularization leads to averaged state representations from which view-specific interference is reduced.
This regularization produces viewpoint-invariant state representations by aligning state and video similarity structures. While Ematching preserves orderings, Lug anchors the learned embeddings, together stabilizing multi-view learning.
Finally, the overall objective for relevance learning is:
Tapel Tmmatching Treg.
4

---

Published as a conference paper at ICLR 2026
Policy leamning
2. Mutt-Viaw Videos Generation
Every steps
Task Description
State
Environment
Polky
humanoid robot sitting a chair"
3. Dataset Construction
4. Update Reference Model
Top
Video Language Model
Relevance
Every Tieaee steps
Reference Dataset
Model
States
Similailies Video Embeddings
MVR Dataset
Figure 2: The entire framework of the proposed MVR. MVR periodically samples state sequences and renders them into videos from different viewpoints (step 2). It then queries VLM for the similarity scores and video embeddings of the videos and augments its dataset with state sequences, video embeddings, and similarity scores (step 3). Additionally, it keeps the state sequences with top-k similarity scores in reference set Dr With D, MVR updates the state relevance model (step 4). Lastly, using the latest state relevance model and the reference set, MVR computes visual feedback r VLM for the online RL agent (step 1), which is combined with task rewards pask through state- dete-deeendent reward shaping that automatically decays as the agent's s behavior aligns with the reference set.
4.2 POLICY LEARNING WITH y VLM
Desiderata of r VLM
Similar to the case image-text similarity, directly using state relevance as y YLM is prone to mis-specification. Instead, we opt for matching the expectation of state relevance-- VIM should lead to matching the learner's state distribution with videos the high- est video-text similarity, thus leading the desired behaviors. Meanwhile, gradually dominate y VLM as the learned behaviors resemble the desired one, which crucial for avoiding the expensive VLM fine-tuning, especially when the VLM is not aligned with the target task.
argue that pask should
The Proposed Reward Function We first extend the notion of state relevance policies.
Definition 4.1. Denote by ds)(s = (1 (1- (11-1 Et r² P(s==s T the state-occupancy distribution of policy æ. The policy relevance had the expectation of state relevance under d=
h"
I (x).
SES
Policy relevance characterizes the extent policy matches the task. Denote by o' the policy that best matches the task prompt f. Our idea is to learn policies that maximize lask while being indiscernible from n²:
max T + wog(oo(h"" -10
(6)
where log - h²²)) is the log-likelihooo of π being better than o' under the BT model. w E IR a weighting term. Using Jensen's 's inequality, this log-likelihood can be dower-bounded as:
log h=²) 2 E. soudt s' oad [log[o (MVR (s) (
Thus, we can maximize Eq. (6) by using the shaped reward MVR defined as
MVR (s) a (s) + wr (LM,
(8)
where y VLM given by:
.VLM(s) Eygny
M
5

---

Published as a conference paper at ICLR 2026
This additive form serves state-dependent reward shaping term rather than fixed-weight sum of unrelated signals. Since we do not have access to samples of  Eq. (9) is computed using samples from a reference set Dref that contains state sequences with top-k video-text similarity scores (aggregated across viewpoints) collected during policy learning. In literature, a similar problem is solved by learning separate policy solely from VI.M-based rewards (Fu al., 2024), yet such approach fails when r yLLM alone cannot solve the task. Moreover, sampling from Ded bears an interesting analogy human skill learning: trying to repeat our previous good trials an effective learning strategy for consolidating our understanding of a skill.
Automatic shaping. As the policy improves, MVR naturally relinquishes control to plask; when the state distribution induced by aligns with the reference set Die the expectation in Eq. (9) approaches zero, causing r VLM to vanish automatically. This provides strong early guidance that naturally decays as behaviors improve, avoiding persistent conflicts with task objectives.
The Entire Framework Fig. 2 illustrates MVR's operation. During policy learning. MVR renders the agent's 's latest trajectories into videos at a frequency Trender It then queries VLM to obtain video-text similarity scores and video embeddings, augmenting the dataset 27 (lines 10-13). D is updated at the same time. At frequency Tupdate MVR updates fMVR using Eq. (5) and samples in DD (line 16-18). Importantly, for off- off-policy algorithms such as TQC (Kuznetsov et al., 2020), the rewards the agent's 's replay buffer must be recalculated after MMR is updated. The full pseudocode of MVR is provided in Algorithm Al in the appendix.
EXPERIMENTS
This section presents the task performance of MVR, followed by an ablation study for its components and results for the influence of using multiple views. We also showcase the importance of combining visual guidance with task rewards using a case study.
5.1 SETUP
Tasks We evaluate MVR on total of 19 tasks from two robotics benchmarks: Humanoid- Bench (Sferrazza et al., 2024) and MetaWorld (Yu et al., 2019). Tasks from HumanoidBench are well-suited for exploring the use of VL.Ms skill learning, as they encompass static posture genera- tion (stand, pellnce_simplee sit_ si__imple, balance_ hard, and sit_ hard) and dynamic motion generation (run, walk, stair, and slide). Mcanwhile, tasks from MetaWorld (hammer, push-wall, faucet-close, push-back, stick-pull, handle-press-side, push, shelf-place, window-close, peg-unplug-side) are used to evaluate performance in fine-graine men-grained manipulation tasks across diverse visual contexts. See Appxs. A.3 and A.4 for detailed descriptions of the tasks considered in this paper. Among the 27 tasks available in HumanoidBench, we follow the benchmark authors and prior work by focusing on the nine locomotion and posture tasks listed above, excluding aecemmaniplltiion tasks and the most challenging locomotion task Hurdle, which do not yield meaningful learning progress within 10M-step budget. For Meta World, we adopt the CW 10 subset from Continual World (Wolezyk et al., 2021), which contains ten single-object manipulation tasks that are commonly evaluated under a IM-step budget.
Evaluation Metries For HumanoidBench tasks, agents are trained for ten million environment steps, and we report the mean and standard deviation of the episodic returns over three random seeds. A higher score indicates better performance. For MetaWorld tasks, agents are trained for one million environment steps, and we report the mean and standard deviation of the task success rates over five random seeds. In our ablation studies HumanoidBench, showcase the relative effect of MVR's s components, we normalize the performance of MVR's variants using the performance of MVR. In addition, report the average rank of methods assess the overall performance for domain. The lower the better.
Alternative Methods For methods only using plack select two recent RL algorithms: TQC (Kuznetsov et al., 2020) and DreamerV3 (Hafner et al., 2023). For existing methods that leverage VL.Ms, we report results for VLMMRM (Rocamonde al., 2024) and RoboCLIP (Sontakke

---

Published as a conference paper at ICLR 2026
Table MVR outperforms VLM-RM and RoboCLIP for HumanoidBench tasks. The first four tasks are dynamic tasks and the last five are static tasks. We compare MVR with TQC (Kuznetsov et al., 2020), DreamerV3 (Hafner al., 2023). VLM-RM (Rocamonde et al., 2024), and RoboCLIP (Son- takke et al., 2024). The DreamerV3 originate from the TD-MPC2 benchmark suite (Hansen et al., 2024). MVR performs the best for five tasks and has the best rank averaged over all tasks. VLM-RM
performs the best for two tasks. RoboCLIP cannot perform well for these tasks.
Task
MVR
TQC
VEM-RM
RoboCIIP
DreamerV3
Success Max Return
Walk
927.47 1.83 /
510.58 1. 299.11
533.35 1. 355.18
137.31 191.62
800.2 1: 156.7
700/ 1000
Run
749. 23 56.82 /
647.87 +: 186.96
14.93 + 1.11
501.15 + 179.71
633.9 +: 222.4
700/ 1000
Stair
208.00 166.22
282.95
120.54 54
44.96 4: 4.12
211.33 4. 56.25
151.1 à. 03.6
700/ 1000
Slide
735.03 85
514.91 106.36
163.13 41.22
494.20 26
436.5 200.1
700/
Stand
918.55 = 29.30 /
576.50 = 371.0
728.69 + 102.11
849.73 + 108 96
(22.7 + 404.8
800/ 1000
Sit Simple
861 (17 +: 19:35 /
822.07
101 /
253.72 65.46
663.90 +: 244.51
89144 4:
750 1000
Sit Hard
79
5111
322..16 45.70
600.38 192.00
365.9
750/
Balance Simple
299 57 1: 58,
256.79 = 54
654 42 1: 335 19
215.81 1: 18.19
19.8 7.0
800/ 1000
Balance Hard
95.78 11.03
92.35 à: 10.76
107 36 -: 24.93
97.03 + 18 13
45.9 +: 27.4
800/ 1000
Average Rank
1.67
3.11
3.78
2.89
3,56
indicates that the mean return exceeds the official HumanoidBBnnh success threshold.
et al., 2024). VLM-RM combines tssk with image-text similarity scores'. RoboCLIP uses video-text similarity but generates single reward value for each trajectory. Since VLM-RM and RoboCLIP use different RL algorithms in their original implementation, we implement both on top of TQC to ensure fair comparison with MVR. All VLM-based baselines, including RoboCLIP, shape their policies with both the task reward and their respective VLM-derived signals so that differences stem purely from the multimodal guidance design.
Table 2: MVR achieves higher success rate on Metaworld tasks. We use the results for DreamerV3 reported by TD-MPC2 benchmark suite (Hansen et al., 2024). The three methods that incorporate visual guidance, MVR, VLM-RM, and RoboCLIP, all outperform TQC and DreamerV3. Within them, MVR achieves the highest success rate, which is followed by RoboCLIP. These results support our claim for the importance of using visual feedback and derive the feedback from multi-view videos.
Task
MVR
TQC
VLM-RM
RoboCLIP
DreamerV3
hammer
0.20 0.24
0.20 1 0.40
0.40
0.49
0.00 1 0.00
0.10 1 0.17
push-wall
0.60 = 0.49
0.10
4: 0.20
0.40
0.37
0.60
+ 0.37
0.00 +: 0.00
close
1.00 0.00
1.00
0.00
1.00
0.00
1.00
0.00
0.13 0.15
push-hack
0.50 = 0.45
0.60
0.37
0.30
000==000 0.40
0.20 1 0.40
0.37 4: 0.38
stick pull
0.20 0.24
0.10
1 0.20
0.00
1 0.00
0.00 1
0.00 0.00
predeeeeess-side
1.00 = 0.00
1.00 1:00::00 0.00
1.00 0.00
1.00 + 0.00
0.77 +: 0.17
push
0.10 0.20
0.10 0.20
0.00 0.00
0.30 1 0.40
0.00 0.00
phacc
0.20 = 0.24
0.00 0.00
1.00=000 0.00
0.20
+ 0.25
0.00 1: 0.00
window close
1.00 0.00
1,00 0.00
1.00 1 0.00
1.00
0.00
0.61 0.16
peg-unplug-side
0.60 0.37
0.30 +: 0.40
0.60 = 0.49
0.80 + 0.40
0.77 + 0.19
Average Rank
1,50
2.20
2.40
2,00
3.90
We compare MVR with several variants to validate its components. The variant w/o reg removes Treg from Eq. (5). For w/o reference, we use the raw relevance (M) rather than Eq. (9) as visual guidance, ablating the state-dependent expectation matching (and its natural decay). The variant MVR-CLIP uses image-text similarity computed by CLIP model. For the variant direct, we train FVLM by directly fitting it to the video-text similarity scores. Moreover, we compare the results using one to four views analyze the influence of using multiple viewpoints. Lastly, we compare the results of using ViCLIP-L (428M parameters) and ViCLIP-B -B (150M parameters) to study the influence of VLM size. Additional ablations on reward components and alternative temporal pooling strategies are summarised in Tabs. A10 and A11.
Ablations
Implementation Details MVR renders one trajectory out of every nine into video (Trender 9). For each rendered episode we draw one viewpoint uniformly at random; across training this rotates
'hiile Rocamonde al. (2024) only uses --bsebased rewards for pose generation, it is difficult solve Incomotion tasks without lankk
"uuccess thresholds follow the official HumanoidBench specification (Sferrazza et al., 2024); the maximum returns correspond to the theoretical upper bounds defined in the benchmark.
7

---

Published as a conference paper at ICLR 2026
MVR
MVR-CUP direct
Stand
Run
1.4
1000
1000
Weo reference
800
800
600
600
400
400
Norma lzed Return
1.0
200
200
2
0.8
#Views
VViews
0.6
Sii de
Sit Hard
800
0.4
Retume
600
600
400
400
200
200
0.0
4
2
VViews
VViews
Stand
Run
Slide
Sit Hard
(a) The of videos, Eq. (3), Eq. (4), and Eq. are all useful. The variant w/o reg and wo reference stand for disabling Eq. (4) and Eq. (9). MVR-CLIP and direct means using images and fitting the f farv to the similarity scores rather than using Eq. (3). No variant consistently good performance as MVR does.
(b) Using more views is beneficial. These figures show the performance of MVR when using one to four views. For Stand, one view seems sufficient as the return is close to optimal. For other tasks, general trend is that performance improves when using more views, though fluctuation exists.
Figure 3: Method ablation and the influence of the number of views.
through all cameras without exceeding the rendering/encoding budget of single-view pipeline, For each video, MVR extracts segments of length 64 (I 64) and queries the ViCLIP model (Wang et al., 2024a) for video-text similarity scores. The relevance model is updated every 100,000 environment steps upudee = 100, 000) with early stopping. The reference set Deet keeps the top- k most relevant state sequences observed far with = 10 for all experiments, i.c., only the highest-quality trajectories remain in memory. The hyperparameter w in Eq. (6) is selected from (0.01, 0.1, 0.5} via grid-search. In contrast, VLM-RM queries VLMs at each policy learning step, which is not scalable for the tasks considered here. Therefore, we adopt the same rendering frequency MVR and fit reward model for policy learning. Both MVR and RoboCLIP parameters). VLM-RM uses CLIP-ViT-H-IE (986M parameters). More details can be found in Appx. A.7.
ViCLIP-L (428M
Table 3: MVR can aggregate information from multiple views. This table compares the task performance using four different viewpoints only and random viewpoints. While the best viewpoint is task desk-dependent, using random viewpoints (MVR) has the best performance for Slide and Sit Hard and second-bess performance for Run, indicating that MVR can aggregate multi-view information.
Stund
914.17
32.54
1229.12 10.58
912.20 1: 26.66
931.10
32.08
918.55 29.30
Run
611.43
29141
651.23 + 164.9
875.33 6.44
567.0 91
159.47
749.23 56.82
Slide
559.59
1 0.00
639.26
1: 131.48
647.18
154.73
657.64
1: 205.81
735.03 :: 142.85
Sit Hard
528.14
176.60
639.10 + 163.77
152.15
210.89
723.72 + 112.11
756.67
108.79
5.2 RESULTS
Proficiency for Soliciting Visual Guidance Tab. presents results for the task performance of HumanoidBench tasks. MVR has the best performance for five tasks. VI.M-MM outperforms other methods for two tasks, but RoboCLIP cannot outperform others for any tasks. Notably, MVR has the best average rank for these tasks, corroborating its proficiency for soliciting visual guidance. The comparison among MVR, VLM-RM, and TQC reveals an interesting insight. VLM-RM performs worse than TQC for three dynamic tasks but surpasses it for three static tasks. In contrast, MVR outperforms TQC for three dynamic tasks and all static tasks. Thus, generating visual guidance from videos not only applies to challenging dynamic tasks but is also more effective for static tasks.
Tab. shows the results for World tasks. MVR attains the highest average success rate and has the best average rank, followed by RoboCLIP. Despite RoboCI.IPs strong overall performance
8

---

Published as a conference paper at ICLR 2026
Task Rewards Only
MVR
Figure MVR identifies suboptimal states through reward shaping. We visualize states generated by TQC agent for the Sit_ Hard task and annotate them with plask (left) and the shaped reward MVR pas + VLM (right) computed by trained MVR agent. While task rewards are high when the agent is close to the chair, MVR's visual guidance component assigns low values to improper sitting poses (sitting on chair's leg, leaning, or sitting at the edge), effectively shaping the reward landscape to discourage these visually suboptimal but task-rewarding states.
(second in average success rate and rank), it scored zero on two specific tasks, indicating potential instability. VLM-RM has higher average success rate than TQC but worse average rank, indicating inferior performance on certain tasks.
The performance margins between multi-view methods smaller on Meta World than on Humanoid- Bench for two reasons. First, Meta episodes are short (500 steps) and focus on single tabletop objects, where sparse trajectory-Ievel feedback already provides useful exploration hints. Second, the long-horizon HunanoidBench locomotion suite demands dense state-conditioned guidance across multiple, occasionally occluded joints, which explains the larger performance gaps observed Tab. 1.
Meanwhile, readers may notice that, among the 19 tasks considered in this paper, MVR performs worse than TQC for four tasks, indicating that using MVR might compromise the optimization of p.ack A possible reason is that the underlying VI.M is trained on mostly human activities rather than robotics data, and we leave the few-shot extension of MVR for future work. Model Ablation We now discuss the results presented in Fig. 3a from three perspectives. Firstly, the inferior performance of direct and the unstable performance of w/o reg indicate that both Eq. (3) and Eq. (4) are essential for learning the relevance model. Moreover, the decreased performance of MVR-CLIP shows that assessing agent behaviors with videos is critical for deriving visual guidance, further confirming the insight discussed above. Lastly, the variant w/o reference is outperformed by MVR for three tasks, confirming the efficacy of our proposed reward function Eq. (9). Altogether, these results demonstrate the effectiveness of components in MVR.
Table 4: MVR with small and large VLMs.
As shown in Table 4, a comparison between the ViCLIP-B and ViCLIP-L models reveals that the larger ViCLIP-L model yields better performance in three of the four evaluated tasks, confirming the common hypothesis of the scaling law. In addition, we present the results for using different RL algo- rithms in MVR in Appx. B. and the results for using different values for the weight u of VLM in Appx. B.8.
Task
ViCLIP (428M)
VICLIP-B (150M)
Stand
918.55 + 29.30
917.97 + 30.71
Run
749.23 + 56.66
517.25 383
Slide
735.03 + 142.85
778.84 24.80
Hard
756.67 1 108.70 79
689.50 1. 234.37
The Influence of Views Fig. 3b presents the results of MVR when using different numbers of views. While using one view is enough for the Stand task, it is beneficial to increase the number of views for other tasks, which confirms our claim for using multi-view videos for computing r VLM Meanwhile, Tab. 3 shows the results of different views and compares them with MVR, which selects views uniformly at random. The best choice of view is task-dependent, which can be explained by the difference in the required motion patterns. For example, for Sit Hard, the two side views 90" and 270" more effective than the rest two because it is easier determine the robot-chair distance

---

Published as a conference paper at ICLR 2026
from side views. Notably, moving from view to four orthogonal views raises the mean returns Run, Slide, and Sit Hard by +98, +96, and +98 respectively, pushing them past the HumanoidBench success thresholds (700, 700, 750) and turning previously failing policies into successful ones. Since the HumanoidBench returns are capped at 1000, these gains amount to roughly 10% of the total attainable return. Interestingly, we obtain competitive even better results by randomly sampling viewpoints, which highlights MVR' s capability in aggregating information in multi-view videos.
Case Study Finally, we showcase the significance of visual guidance using the Hard. Fig. 4 is a t-SNE (van der Maaten & Hinton, 2008) visualization of 20 episodes produced by trained TQC agent. We annotate the states with plskk and r MVR computed by a trained MVR agent in Fig. 4. The rewards are normalized to the range [-1, 1] for visualization purpose.
general trend in Fig. 4 is that rlu and MVR align in many parts of the state space. For example, both reward functions assign high rewards to the bottom left part of the space and low rewards to the bottom right. However, upon closer inspection of the regions where the two reward functions diverge, we gain valuable insights. Specifically, the center region corresponds to undesirable sitting poses, such as sitting on the chair's leg (the top example), leaning on the chair (the middle example), and sitting at the chair's edge (the bottom example). Although these poses receive high values of y.lusk due to the minimal distance between the robot and the chair, they are inherently unstable and suboptimal. Notably, MVR is able to identify them, which explains its superior performance over TQC in this task. Without visual guidance, a much larger number of online samples will be required for recognizing such visually suboptimal behaviors by their long-term consequences. This emphasizes the importance of visual guidance in learning general and versatile skills,
Quantitative analyses. Appendix B.7 details the correlation between the learned shaping signals and binary success, and also documents the temporal decay of r VLM that drives the automatic handover to task
CONCLUSIONS
In this paper, we present novel framework named Multi-view Video Reward to generate visual guidance using VLMs for RL agents. In contrast to existing attempts that use VLMs to generate rewards, MVR leverages videos collected from multiple viewpoints to evaluate agent behaviors, which enables it to characterize dynamic motions and overcome occlusions. Moreover, we propose a state-dependent reward shaping formulation that integrates task rewards with VLM-based guidance and automatically reduces the influence of the auxiliary signal as the behavior aligns with the desired motion pattern. We evaluate MVR for nine humanoid tasks and ten manipulation tasks. MVR has the best average rank when compared to existing methods that use VLMs and methods that only use task rewards, confirming its superiority for soliciting visual guidance for RL agents. Furthermore, we confirm the design of MVR with an ablation study and investigate the influence of using multiple views. Lastly, we present a case study to showcase the importance of visual feedback for fostering the desired motion of task.
Ethics Statement. This work uses only simulated benchmarks, does not collect human data, and does not interact with sensitive or safety-critical systems. To the best of our knowledge, poses no foreseeable ethical, privacy, or security risks.
Reproducibility Statement. Section 4.1 and Alg. A1 specify the training objectives, optimization steps, and update schedule for MVR. Implementation details covering architectures, environment settings, rendering schedules, and all hyperparameters appear in Appendix A. while additional ablations and alternative backbones are summarized in Appendices I. and B.6. Appendix B compiles the evaluation tables cited in the main text. The submission bundles the Hydra configuration files, evaluation scripts, and fixed random seeds required reproduce every experiment.
ACKNOWLEDGMENTS
C. Fang was supported by the National Science and Technology Major Project (No. 2022/D01 14902).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Minttu Alakuijala, Reginald McLean, Isaac Woungang, Samuel Kaski, Pekka Marttinen, Kai Yuan, and Nariman Farsad. Video-language critic: Transferable reward functions for language- conditioned robotics. Transactions on Machine Learning Research, 2025.
Tewodros W. Ayalew Xiao Zhang, Kevin Yuanbo Wu, Tianchong Jiang, Michael Maire, and Matthew R. Walter. Progressor: A perceptually guided reward estimator with self-supervised online refinement. arXiv preprint arXiv:2411. 17764, 2024.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345. 1952. ISSN 00063444.
brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev. Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Hues Lee, Yuheng Kuang, Sally Jesmonth, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as can, not as say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning. 2022.
Harris Chan, Volodymyr Mnih, Feryal Behbahani, Michael Laskin, Luyu Wang, Fabio Pardo, Maxime Gazeau, Himanshu Sahni, Dan Horgan, Kate Baumli, Yannick Schroecker, Stephen Spencer, Richie Steigerwald, John Quan, Gheorghe Comanici, Sebastian Flennerhag, Alexander Neitz, Lei M Zhang. Tom Schaul, Satinder Singh, Clare Lyle, Tim Rocktäschel, Jack Parker-Holder, and Kristian Holsheimer. Vision-language models as source of rewards. 2023.
William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision- models provide promptable representations for reinforcement learning. arXiv preprint arXiv:2402.02651, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4302-4310. Curran Associates, Inc., 2017.
Jieming Cui, Tengyu Liu, Nian Liu, Yaodong Yang, Yixin Zhu, and Siyuan Huang. Anyskill: Learning open- vpea-locabylary physical skill for interactive agents. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision- langualan models as success detectors. Proceedings The 2nd Conference on Lifelong Learning Agents, 2023a.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, 2023b.
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. FuRL: Visual-language models as fuzzy rewards for reinforcement learning. In roceedings of the Forty-first International Conference on Machine Learning, 2024.
Jialu Gao, Kaizhe Hu, Guowei Xu, and Huazhe Xu. Can pre-traiinee mreitrinned text-to-image models generate visual goals for reinforcement learning? Advances in Neural Information Processing Systems, 2023.
Danijar Hafner. Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 104, 2023.

---

Published as a conference paper at ICLR 2026
Yunhai Han, Zhenyang Chen, Kyle A. Williams, and Harish Ravichandar. Learning prehensile dexterity by imitating and emulating state-only observations. IEEE Robotics and Automation Letters, 2024.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, 2024.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 2020.
Sukai Huang, Nir Lipovetzky, and Trevor Cohn. The dark side of rich rewards: Understanding and mitigating noise in vlm rewards. arXiv preprint arXiv:2409. 15922, 2024a.
Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. In European Conference on Computer Vision (ECCV), 2024b.
Kento Kawaharazuka, Tatsuya Matsushima, Andrew Gambardella, Jiaxian Guo, Chris Paxton, and Andy Zeng. Real-world robot applications of foundation models: review. Advanced Robotics, 2024.
Changyeon Kim, Younggyo Sco, Hao Liu, Lisa Lee, Jinwoo Shin, Honglak Lee, and Kimin Lee. Guide your agent with adaptive multimodal rewards. Advances Neural Information Processing Systems, 2023.
Changyeon Kim. Minho Heo, Doohyun Lee. Honglak Lee, Jinwoo Shin, Joseph Lim, and Kimin Lee. REDS: Subtask-aware visual reward learning from segmented demonstrations, In Interna- fional Conference on Learning Representations, 2025.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, 2020.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim. Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. SimBa: Simplicity bias for scaling up parameters in deep reinforcement learning. In International Conference on Learning Representations, 2025.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference Learning Representations, 2024.
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani. Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. Vision language models are in-context value learners. In International Conference on Learning Representations, 2025.
Antoine Miech, Dimitri Zhukov, Sean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto 100m: Learning text-video embedding by watching hundred million narrated vidco clips. In Proceedings of the IEEF/CVF international conference on computer vision, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Angelo Moroncelli, Vishal Soni. Asad Ali Shahid, Marco Maccarini, Marco Forgione, Dario Piga, Blerina Spahiu, and Loris Roveda. Integrating reinforcement learning with foundation models for autonomous robotics: Methods and perspectives. arXiv preprint arXiv:2410. 1641 2024.
12

---

Published as a conference paper at ICLR 2026
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning, 2022.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14760-14769, 2024.
Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis luzzolino, Unnat Jain, and Ruta Desai. Pretrained language models visual planners for human assistance. In Proceedings the IEEE/CVF International Conference on Computer Vision, 2023.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-base character skills. ACM Transactions on Graphics, 37(4):143, 2018.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Htallebaseelinss. Reliable reinforcement learning implementations. Journal of Machine Learning Rexearch, 2021.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on robot learning, 2022.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 2024.
Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bryik, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 2024.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang. Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. Proceedings of the Twelfth International Conference on Learning Representations, 2024a.
Yufei Wang, Zhanyi Sun, Jesse Zhang. Zhou Xian, Erdem Biyik, David Held, and Zackory Erick- son. RL-VIL--F; Reinforcement learning from vision language foundation model feedback. In International Conference Machine Learning, 2024b.
Maciej Wotezyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciński, and Piotr Mitos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye HAO, and Mingsheng Long. ivideoGPT: Interactive videoGPTs are scalable world models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), 2018
13

---

Published as a conference paper at ICLR 2026
Tianbao Xie, Stheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference Learning Representations, 2024.
Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang. A survey on robotics with foundation models: toward embodied ai. arXiv preprint urXiv:2402.023600 2024.
Daniel Yang, Davin Tjia, Jacob Berg, Dima Damen, Pulkit Agrawal, and Abhishek Gupta. Rank2reward: Learning shaped reward functions from passive video. In IEEE International Conference Robotics and Automation (ICRA), 2024.
Kelin Yu, Sheng Zhang, Harshit Soora, Furong Huang, Heng Huang. Pratap Tokekar, and Ruohan Gao. Genflowrl: Shaping rewards with generative object-centric flow in visual reinforcement learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2025.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Third Conference on Robot Learning, 2019.
Jiahui Zhang, Yusen Luo, Abrar Anwar, Sumedh A. Sontakke, Joseph J. Lim, Jesse Thomason, Erdem Biyik, and Jesse Zhang. ReWiND: Language- gugdeed rewards teach robot policies without new demonstrations. In Conference on Robot Learning, 2025.
AI

---

Published as a conference paper at ICLR 2026
Algorithm A1 The Pseudocode of MVR
Input:
textual goal description f, environment steps 7Tm,, video sampling frequency Treneer model update frequency Tuppue
Initialize the policy and relevance model f fAIVR
3: Initialize the replay buffer of MVR: D = II.
Initialize the reference set: Dr =
t= 0, .... T do
Observe state St. apply action d, =+(+&)) and observe Compute r? and form the shuped reward IVR nak ur) WrYLM. and p update
8e+1 reward plak,
Use
9:
i O then
10:
if graals = then
11:
Render the last episode into videos.
Augment with the latest sequences, vidcos, and their video-text similarity
D 2-=77 D ((s, VLM {))}. Switch to the next viewpoint.
13:
14:
Update the reference set D if necessary.
15:
end
16:
" then Use data in update fMV with Eq. (5).
17:
18:
end
19:
end if
20:
end for
A EXPERIMENTAL SETUP
A.1 IMPLEMENTATION DETAILS
In this study, adopted the JAX implementation of TQC from the SBX library (Raffin al., 2021) as our baseline RL algorithm. The parameters of this algorithm were subsequently fine-tuned to address the complexities inherent humanoid robot control tasks. Our proposed Multi-view Video Rewards (MVR) framework builds upon this TQC foundation. The experiments were conducted using the most recent version of the HumanoidBench simulation environment (Sferrazza et al., 2024).
All computational experiments were performed on NVIDIA RTX 4090 GPUs. Under this hardware configuration, each MetaWorke task required approximately minutes to complete, while each HumanoidBench task took roughly 4 hours. The specific versions of key software packages utilized throughout our experiments are enumerated below:
Python 3.11
Torch 2.3.1
Torch Vision 0.18.1
JAX 0.4.37
JAXLib 0.4.36
Flax 0.10.2
Gymnasium 0.29.1
SBX-RL 0.18.0
A.2 MVR ALGORITHM PSEUDOCODE
For completeness, we present the full pseudocode of the proposed MVR framework below; see Fig. in the main text for the high-level pipeline.
A.3 HUMANOIDBENCH
Humanoid robots offer significant promise for assisting humans in multitude of environments and tasks, primarily due to their human-like morphology, which grants them flexibility and adaptability. HumanoidBench (Sferrazza al., 2024) is significant contribution in this area, offering simulated benchmark with 27 distinct whole-body control tasks. These tasks are designed to present unique and substantial challenges, including those requiring intricate long-horizon control and sophisticated
A2

---

Published as a conference paper at ICLR 2026
coordination. As demonstrated by (Sferrazza al., 2024), state-of-the -fe--h--rtt-tttt- RL algorithms face difficulties with many of these tasks. For our study, we excluded tasks that were identified as poorly designed (e.g., the ppowerlitt' task lacked appropriate reward signals for lifting the barbell), resulting set of nine valid tasks for evaluation.
A brief description of each selected HumanoidBench task is provided below.
Stand The robot's 's objective is to maintain an upright standing posture for the duration of the trial. This requires dynamic adjustments to ensure stability, compensating for environmental changes or internal balance shifts. This task evaluates the robot's capacity for stable positioning, a fundamental skill for stationary operations and transitions between movements.
Walk This task involves the robot maintaining a consistent forward velocity of approximately 1 m/s along the global x-axis, without falling. challenges the robot's balance, coordination, and its ability to generate efficient forward locomotion, crucial for navigation in varied settings.
Run Here, the robot must achieve and maintain forward running speed of 5 m/s. This higher-speed task demands advanced coordination, strength, and precise control over leg movements. It involves not just maintaining velocity but also managing the complex dynamics transitioning between walking and running gaits.
The robot is tasked with traversing a repeating sequence of upward and downward stairs at a steady speed of m/s. This requires the robot adapt its gait and step height for each stair while preserving balance and forward momentum, simulating real-world obstacle navigation.
Stair
Slide In this task, the robot navigates alternating sequence of upward and downward inclines (slides) at a speed of 1 m/s. The primary challenge lies adapting to varying surface inclinations and maintaining stability on potentially slippery uneven terrain, testing real-time gait adjustment capabilities.
Sit Simple The robot must sit on chair positioned closely behind it. This task tests the robot's environmental perception, motion planning for the sitting sequence, and controlled execution of the sit-down action while maintaining stability.
This task increases the complexity of sitting by randomizing the chair's ' initial position. Furthermore, the chair can move due forces exerted by the robot, meaning its position and orientation can change dynamically. The robot must continuously adapt its approach, handling the uncertainty of the chair's movement while maintaining balance.
Sit Hard
The robot is required to maintain balance on an unstable board that pivots fixed spherical point. The core challenge is to remain stable on surface that can tilt in any direction, demanding continuous postural adjustments and center of mass control.
Salance_Simple
Balance Hard This task extends the balancing challenge by introducing a moving pivot point for the unstable board. This adds layer of complexity, the robot must simultaneously adjust for the board's 's instability and the dynamic movement of its pivot, necessitating a high degree coordination and real-time responsiveness.
Prompts derived from these task descriptions are summarized Tab. A1.
A.4 METAWORLD
Meta- World is an open- sourcs simulated benchmark for meta-RL and multi-task learning consisting of 50 distinct robotic manipulation tasks. We selected 10 Metaworld tasks in CW10 (Wotezyk et al., 2021) because their difficulty is suitable for solving in 1M steps. brief introduction to each selected task is summarized below.
A3

---

humanoid robot walking a humanoid robot running humanoid robot walking a humanoid robot walking humanoid robot sitting chair a humanoid robot sitting on chair a humanoid robot balancing on the board Published as a conference paper at ICLR 2026
Table Al: Text instructions for humanoid robot tasks.
Prompt
Description
Stand
a humanoid robot standing
Walk
Run
Stair
Slide
Sit_Simple
Sir_Hard
Balance Simple
Balance Hard
a humanoid robot balancing on the board
Hammer In this task, the robot gripper is required to pick up hammer and use to strike nail or a similar target on a surface. This task tests the robot's abilities in grasping, tool manipulation, force control, and aiming accuracy.
Push-Wai Wall This task requires the robot gripper to push an object, such as a puck or cylinder, towards target location. However, a wall obstructs direct path, so the robot must navigate the object around or over the wall. This task challenges path planning, obstacle avoidance, and pushing mechanics.
The robot gripper must turn faucet handle clockwise close it. This involves precise grasping of the handle and applying the correct rotational force. It tests fine motor control and the understanding of object affordances.
Faucet-Close
Push-Back In this task, the robot gripper is required to push wooden block backward to designated goal position. This primarily tests the robot's ability to apply controlled force a specific direction and manage the displacement of the object.
Stick-Pull The robot gripper needs to grasp a stick and pull towards itself or out a fixture. This task assesses grasping strength, pulling force control, and potentially the ability to dislodge object from its holder.
This task requires the robot gripper press handle downwards, approaching it from the side. This tests the robot's ability to apply force at specific angle and direction, often simulating actions like opening door or activating a mechanism.
Handle-Press-Siin
Push This is a fundamental pushing task where the robot gripper must push an object, typically cylinder or puck, target location on a flat surface. evaluates the robot's basic pushing capabilities and goal-oriented movement.
The robot gripper tasked with picking up an object, such as cube, and then accurately placing it onto designated spot shelf. This involves a sequence of grasping, lifting, 3D path planning, and precise placement.
Shelf-Place
Window-Clen In this task, the robot gripper needs to close window. This might involve pushing a sliding window or manipulating a latch or handle, depending on the window type. The task tests interaction with larger articulated objects and achieving specific state change in the environment.
The robot gripper must grasp a peg that inserted into hole and then unplug by pulling it out sideways. This requires precise grasping, applying force in specific lateral direction, and successfully disengaging the parts.
Peg-Unplug-Side
Prompts derived from these task descriptions are summarized in Table A2.
A4

---

Published as a conference paper at ICLR 2026
Table A2:
Text instructions for Meta World robot tasks.
Prompt
Description
hammer
a robot gripper hammering a screw on the wall robot gripper bypassing wall and pushing cylinder a robot gripper rotating the faucet clockwise
push-wall
faucet-close
push-back
robot gripper pushing the wooden block backward a robot gripper pulling a stick
stick-pull
handle-press-side
robot gripper pressing a handle down sideways a robot gripper pushing a cylinder robot gripper picking and placing a cube onto shelf robot gripper pushing and closing a window robot gripper unplugging a peg sideways
push
shelf-place
window-close
peg-unplug neg-unplug-side
A.5 ARCHITECTURE AND HYPERPARAMETER
RL Training Each humanoid task was trained for 10 million steps, and each metaworld task was trained for I million, following the setup outlined in (Sferrazza et al., 2024; Hansen al., 2024). Samples were collected from 8 parallel environments, with the learning rate decaying from 6 x 10-4 the start to 5 x 10-5 5 at the final step, ensuring training stability. The policy was updated by sampling from the replay buffer with a batch size of 256. Every 16 steps, MVR performs 16 gradient updates. Both actor and critic networks consist of three-layer neural networks, each with a width of 256 and quantile count of 50, while the number of top quantiles to drop per network is set to 5. SDE (Raffin et al., 2022) was enabled to enhance the algorithm's 's exploration. The key parameters summarized in Tab. A3.
Compute cost matches that of a single-view implementation. HumanoidBench run takes roughly four hours on one NVIDIA RTX 4090 using MuJoCo's EGL backend (about 20 GB GPU memory). When GPU rendering is unavailable we fall back to the OSMesa backend, which requires approximately 2.5 GB of CPU memory. MetaWorld tasks are shorter, each finishes in about 25 minutes and needs roughly 2 GB of GPU memory. These numbers are consistent with the settings reported in the official HumanoidBench repository.
For each rendered trajectory sample only one viewpoint, so the wall-clock cost
Reward Model The proposed reward model is composed of two components: multi-layer backbone network and parameterized predictor vector. As outlined in Tab. A3, the backbone adopts fully-connected architecture with two hidden layers of dimension 512, activated by ReLU nonlinearity. The predictor is implemented as a single learnable vector of dimension 512, initialized via Kaiming uniform distribution stabilize gradient dynamics during training. Formally, given an input state s E Rd, the backbone first projects it into a latent embedding space through sequential transformations:
h ReLU(W b,), W2h + b
(A1)
where W, 512xd W₂ 512x511 weight matrices. The final embedding e h2/ h2ll2 is
fz-normalized along the feature dimension to constrain the output scale. The predictor p E R512 operates direction-sensitive projection head. It is normalized p = p/llpl and broadcasted across the batch dimension. The value MMVR is computed via cosine similarity between the normalized embedding and the predictor:
512
MMVR (e. p)
Σ E P
(A2)
This design bounds reward outputs within [-1, 1}, promoting training stability. Optionally, the model can return the intermediate embedding e for auxiliary tasks or interpretability analysis.
Reward Model Learning MVR renders out of every nine trajectories into videos with resolution of 224 x 224. From each video, MVR extracts segments length 64 and queries the ViCLIP model (Wang al., 2024a) for video-text similarity scores, using a batch size of 32. These
A5

---

Published as a conference paper at ICLR 2026
Table A3: Hyperparameters of MVR
Category
Hyperparameter
Value
TQC Network Architecture
Policy Layers
[256, 256, 256] [256, 256, 256] 50
Critic Layers
Quantile Count
Reward Model
Backbone Predictor
[512,512] Parameter(5122
RL Training
Total Timesteps T humanoiddennch Total Timesteps Tay metaworrd
1 x 10?
1 x 10t
Number Environments
Learning Rate
lin_ 6 x 10-1 - 5 x 10-3
Batch Size
256
Buffer Size
1 x 106
Gradient Steps
16
Tau T
0.01
Gamma y
0.99
Sde Sample Frequency
Top Quantiles to Drop Per Net
5
Reward Model Learning
Model
ViCLIP
Image Resolution
224 224
Reward Scale
{0.01, 0.1, 0.5) best
Video Clip Length vidooo
64
Relabel Interval
1 x 105
Update Interval Tupdaaoo
1 x 105
Video Sampling Frequency Treader
9
Encoding Batch Size
Reward Dataset Size
2 x 104
scores are then added to the reward dataset, which has a size of 2 x 10 The relevance model is updated every 100,000 environment steps, with early stopping employed to prevent overfitting. The hyperparameter w in Eq. (6) is selected from {0.01, 0.1, 0.5} using grid search. MVR relabels the dataset every 1 x 10 steps. Key parameters are summarised Tab, A3.
A.6 TOP-/BOTTOM-RANEE RANKED FRAMES IN SIT HARD
To further clarify where MVR actively modifies the reward landscape, we visualize frames from the Sit_ Hard task ranked by the task reward lisk and by the full shaped reward ,.MVR = rlsk + wr VLM Figure Al shows the top- and bottom-ranked frames under both criteria. The top-ranked frames for takk and MVR consistently correspond to clearly successful behaviors in which the humanoid sits stably on the chair with an upright torso, indicating that the visual shaping term preserves the optimal behavior selected by the task reward. The bottom-ranked frames for both rewards are dominated by catastrophic failures, where the robot has fallen off the chair or never approaches seated configuration, confirming that both signals assign low value clearly unsuccessful states though the exact ordering among these failures is less critical for control. This qualitative alignment on clearly successful behaviors is consistent with our goal that VLM automatically decays once the policy reliably reaches the desired motion pattern.
In contrast, differences between lusk and MVR primarily emerge in the intermediate- regime. As illustrated by the analysis in Fig. 4, clusters of states that receive relatively high iank but correspond visually unstable sitting poses (e.g., sitting a chair leg, slipping off the edge, or leaning heavily)
A6

---

Published as a conference paper at ICLR 2026
Figure A1: Top-hottomm-ranked frames by task and MMR in the Sit Hard task. The first two rows show the top and bottom frames under the task reward piusk while the last two rows show the corresponding frames under the shaped reward MVR For both rewards, the highest-ranked frames coincide with visually unambiguous successful sitting poses, confirming that MVR preserves the optimal behavior emphasized by the task reward. The bottom-ranked frames are dominated by clearly failed attempts under both signals, serving mainly as a sanity check that obviously bad states receive low values. Together with the qualitative analysis in Fig. 4, this suggests that the main effect of MVR is to reshuffle intermediate-rewaad states, down-weighting visually unstable but task-rewarding poses while leaving the optimal behavior unchanged.
are systematically down-weights weinhweighted by MMR These mid-range corrections are precisely where MVR active: the shaped reward preserves the ordering of clearly successful states while sharpening the ranking among borderline cases, assigning lower values to visually unstable yet task-rewarding poses and relatively higher values to visually robust configurations.
A7

---

Published as a conference paper at ICLR 2026
A.7 VLM BASELINES
This section provides a detailed overview of the VLM baselines. Tab. A4 summarizes the models used.'
Table A4: Models Used in MVR, VLM-RM, and RoboCLIP
Algorithm
Model
MVR
ViCLIP
VLM-RM
RoboCLIP
ViCLIP
VLM-RM VLM-RM uses a pre-trained vision-language model (VLM) as a zero-shot reward model (RM) to represent tasks in natural language. The original implementation employs CLIP to train a MuJoCo humanoid robot for static tasks, such as kneeling, doing splits, and sitting cross-legged. In this paper. we use 1---ViTT------lain22--3333 We select TQC instead of SAC because it demonstrates superior performance and serves as the RL backbone for MVR, eliminating performance discrepancies related to algorithm selection. We directly use MSE loss to fit the CLIP output, avoiding ranking loss to ensure maximum consistency with the original VLM-RM implementation. VLMMRMM is defined as follows:
to match the model size with ViCLIP.
VLM-RM - Egga
(LLM RM (s)]
(A3)
where FVIM-RM represents the reference function. The remaining parameters of VLM-RM align with MVR (see Tab. A3).
RoboCLIP RoboCLIP generates rewards for online RL agents using pre-trained video-language models. It provides sparse, end-of-trajectory reward based the similarity between the agent's video and a target text description. The original version embedded inputs using S3D (Xie et al., 2018) (pre-trained on How HowTo000M (Miech al., 2019)) and used the unscaled scalar product of embeddings as the reward. In this paper, we adapt RoboCLIP by substituting ViCLIP for S3D and replacing the SAC algorithm with TQC to align with the MVR baseline for comparison. The algorithms and corresponding VLM models are summarized in Tab. A4.
B ADDITIONAL RESULTS
This section presents further experimental results provide more comprehensive evaluation of MVR and its components.
B.1 COMPATIBILITY WITH DIFFERENT RI. ALGORITHMS
To evaluate the general applicability of the MVR framework, we integrated it with different RL algorithm, SAC, addition to the TQC algorithm used the main experiments. Tab, A5 compares the performance of MVR-SAC against the baseline SAC algorithm, alongside the MVR-TQC and TQC results for reference. We further pair MVR with Simba (Lee et al., 2025), a large-scale off officy learner, to confirm that the shaping term scales to stronger architectures.
The results indicate that the performance benefit from VLM-based rewards provided by MVR is less pronounced when using SAC compared to TQC. By contrast, integrating MVR with Simba preserves the gains on the hardest locomotion tasks, demonstrating that multi-view shaping transfers to the latest high-capacity agents (Tab. A6).
For fairness, exclude methods such as RL-VLL--F (Wang et al., 2024h) that require repeated queries to large proprictary VI.Ms, which falls outside our automated large-scale evaluation setting.
A8

---

Published as a conference paper at ICLR 2026
Table A5: MVR yields greater performance improvements when integrated with the TQC algorithm compared to SAC on complex humanoid tasks. This table presents average returns for MVR-SAC and MVR-TQC against their SAC and TQC baselines across selected HumanoidBench tasks.
Task
MVR-SAC
SAC
MVR-TQC
TQC
Stand
704.24 + 329.17
581.77 #
918.55 + 29.30
576.59 371.
Run
280.24 103.91
299.39 + 314.65
749.23 + 56.82
647.87 = 186.98
Slide
219.79 + 176.13
94.43 + 52.89
735.03 + 142.85
514.91 + 106.36
Sir_Hard
172.46 + 124.35
263.87 + 97.12
756.67 + 108.79
511.85 + 155.45
Table A6: HumanoidBench returns for Simba with and without MVR shaping. Simba numbers are taken from the original publication; Simba+MVR averages three seeds under the same horizon.
Task
Simba
Simbe+MVR
Stand
906.94
Run
741.16
817.69
Slide
577.87
814.53
Sit Hard
783.95
785.2
B.2 VLM BACKBONE ABLATION
To assess the sensitivity of MVR to the choice of vision-language backbone, additionally train S3D-based variant of MVR on four HumanoidBench tasks and compare with the ViCLIP-based version and the TQC baseline. Tab. A7 shows that MVR-S3D consistently outperforms TQC and achieves performance comparable to VVC-VI (slightly better on Run/Stand and slightly worse on Slide/Sit Hard), suggesting that our gains do not rely on particular VLM.
Table A7: Effect of VLM backbone on MVR performance. Returns on four HumanoidBench tasks
for TQC, MVR with ViCLIP, and MVR with S3D.
Task
TQC
MVR-ViCLIP
MVR-S3D
Run
647.87 + 186.98
4
793.61 + 57.27
Slide
514.91 + 106.36
735.03 1 142.85
696.51 32.39
Stand
576.59 = 371.00
918.55 29.30
937 20 14.28
Sit Hard
511.85 = 155.45
756.67 108.79
678.53 = 106.32
B.3 OCCLUSION-HEAVY HUMANOIDBENCI TASK
To further assess MVR under severe occlusion, we evaluate the Pole task from HumanoidBench. where the humanoid must move forward through a dense forest of thin poles without colliding with them. In this scenario, different body parts are frequently occluded from any single viewpoint, making multi-view coverage especially important. As shown in Tab. A8, MVR substantially outperforms the task-reward baseline TQC on this occlusion-heavy locomotion task.
B.4 WRIST-MOUNTHR CAMERA VIEWS IN METAWORLD
To complement the exocentric setup used in the main MetaWorld experiments, we consider a variant that incorporates wrist-mounted hand camera when computing VLM scores. This variant, denoted WVR-HandCamerr. uses the same RL and reward-mode hyperparameters as MVR; the only change that videos can also be rendered from camera rigidly attached to the end-effector to better capture near-fiell contacts.
Tab. A9 compares success rates for TQC, exocentric MVR, and MVV-HandCamera the ten CW tasks.
achieves comparable average performance to MVR (0.51 = 0.15 0.54 + 0.22), with clear gains contact-rich tasks such push-wall and neg-unplug-sicde while
A9

---

Published as a conference paper at ICLR 2026
Table A8: HumanoidBench Pole task under heavy occlusion. Returns for TQC and MVR.
Task
TQC
MVR
Pole
601.62 192.34
956.43 13.02
slightly underperforming some tasks that mainly require coarse arm motion (e.g.. push-back). These results suggest that near-field field wrist views are particularly beneficial when local object-hand interactions determine success, whereas they may provide limited benefit mild distraction tasks dominated by global motion.
Table A9: Meta World success rates with a wrist-mounted camera. TQC uses only task rewards. MVR uses exocentric videos, while AVV--aanCamera additionally incorporates a wrist-mounted hand camera in the reward pipeline.
Task
TQC
MVR
MVR-HandCamerr
Hammer
0.20 = 0.40
0.20 = 0.24
0.20 + 0.24
Push-Was Wall
0.10 + 0.20
0.60 + 0.49
0.80 + 0.40
Faucet-Close
1.00 = 0.00
1.00 100
1.00 + 0.00
Push-Back
0.60 0.37
0.50
0.20 = 0.24
Stick-Pull
0
0.20
0.00 0.00
Handle-Press-Sice
1.00 0.00
0.00
1.00 + 0.00
Push
0.20
0.10 0.10
Shelf-Place
0.00 = 0.00
0.20 0.24
0.00 = 0.00
Window Close
1.00 + 0.00
1.00 0.00
1.00 0.00
Peg-U Reg-Unplug-Side
0.30 + 0.40
0.60 + 0.37
0.80 + 0.40
Average
0.44 + 0.18
0.54 + 0.22
0.51 + 0.15
B.5 METAWORLD TRAINING CURVES
To further examine how MVR shapes the learning dynamics, we report success rate as a function of training steps for Meta World tasks where the performance gap between MVR and TQC is particularly pronounced: push and neg-umplug-side In both cases, the MVR curves lie above the TQC baseline already in the early training phase, indicating that the shaped visual reward accelerates learning rather than only improving final asymptotic performance.
1.0
1.0
TQC
TQC
0.8
MVR
0.8
MVR
Success rate
Success rate
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
1e6
Timesteps
1e6
(a) push-wall.
(b) peg unplug-sidce
Figure A2: Success rate vs. training step on two MetaWorld tasks. Across both (a) push-wall and (b) peg- peg-umplan caa---uplugssde MVR consistently outperforms TQC throughout training, with success rates already higher in the early stages, indicating that multi-view visual shaping accelerates learning rather than only improving final performance.
A10

---

Published as a conference paper at ICLR 2026
Table A10: Ablating the reward components. Using only the VLM reward fails to solve the humanoid tasks, whereas combining it with plask recovers the full performance reported in Tab. 1.
Method
Stand
Run
Slide
Sit Hard
MVR (only y VLM)
7.73 + 0.68
1.51 + 0.05
0.83 + 0.02
2.60 + 0.19
MVR (full)
918.55 + 29.30
749.23 + 56.82
735.03 + 142.85
756.67 + 108.79
TQC (only plask)
576.59 + 371.00
647.87 186.98
514.91 + 106.36
511.85 + 155.45
B.6 ADDITIONAL ABLATIONS
The comparison shows that relying exclusively on VLM collapses learning: returns stay below ten across all four tasks. Adding the standard task reward lifts performance back to the levels reported in the main paper, highlighting that the VLM signal acts as a complement rather than a replacement for p task
Table A11: Temporal averaging versus
pooling, Attention can help static tasks but remains unstable on dynamic locomotion tasks, so we adopt simple averaging in the main paper.
Task
MVR (Temporal Avg.)
MVR (Attention)
Stand
918.55 = 29.30
940.80 + 11.11
Run
749.23 = 56.82
519.83 = 366.45
Slide
735.03 = 142.85
683.57 = 243.58
Sir_Hard
756.67 + 108.79
822.27 + 18.04
For the pooling study, attention improves static, single-stage behaviours such as Stand and _it_ Hard but degrades performance on locomotion tasks (Run, Slide) where the policy must react continuously. Because the weighting network observes only individual states, its outputs become noisy when the task depends on long-horizon momentum, reinforcing our choice of temporal averaging in the main ablations.
All-view VLM ablation. To ablate the rendering strategy, we evaluate an all-view variant, MVR- ALL, which renders four synchronized videos per rollout and concatenates them before querying the VLM so that every viewpoint is processed jointly. This eliminates viewpoint selection but multiplies rendering and encoding cost by four. Tab. A12 compares TQC, MVR-ALL, and our default MVR. The results are informative: MVR-ALL delivers a noticeable uplift on the static 'Stand' task yet offers only marginal no gains on the dynamic locomotion tasks. For high-speed motions, maintaining single coherent viewpoint preserves temporal consistency that the naive multi-view concatenation tends to disrupt, which explains why our lightweight single-vier sampling coupled with relevance aggregation remains more effective despite the reduced rendering cost.
Table A12: HumanoidBench returns for the all-view baseline. MVR-ALL renders four videos simultaneously and feeds them jointly the VLM. Means and standard deviations are computed over three seeds.
TQC
MVR-AL.I
MVR
Stand
576.59 + 371.00
879.06 + 54.27
918.55
29.30
Run
617.87 + 186.98
672.53
115.21
Slide
514.91
+ 106.36
596.27
1: 59.93
735.03 : 142.85
511.85
510.78
756.67
B.7 REWARD CORRELATION ANALYSIS
Quantitative alignment. We evaluated 100 Sit Hard rollouts and computed the selectory-averagee fVM fMVR, and y. ,VR As summarised in Tab. A13, their Pearson correlations with the binary success indicator reach 0.91, 0.96, and 0.98, respectively. Importantly, the relevance score pMVi keeps strong link to success while remaining only weakly correlated with the sparse environment
A11

---

Published as a conference paper at ICLR 2026
reward (0.22), indicating that the learned shaping emphasises task completion cues rather than copying pass Consequently the shaped reward MVR maintains the success alignment and, because it respects the ordering of successful trajectories, leaves the optimal policy approximately unchanged while reinforcing the visual priors encoded by the relevance model. These measurements explain why visual feedback is indispensable: it resolves ambiguities in plask while remaining tightly aligned with the actual success criterion.
Automatic shaping. Tracking the same 100 rollouts over training shows that the average magnitude of y VLM during the final 100 steps of successful episodes drops to about 23% of its early-training value. This confirms empirically that MMR hands control back to y.ask as soon as the desired behaviour emerges, preventing long-term gradient conflicts.
Tab. A13 summarises the correlations measured on Sit Hard (identical to those referenced in the main text). The statistics highlight that fMV remains strongly aligned with success yet only weakly correlated with plask while the shaped reward MMR maintains the ordering of successful rollouts, keeping the optimal policy approximately unchanged.
Table A13: Reward-signal correlations on Sit Hard.
Signal
Pearson vs success
Pearson vs sook
Raw VLM similarity PMM
0.91
0.21
Learned relevance f Mve
0.96
0.22
Shaped reward .VR
0.98
0.85
We further compare reward quality across VIM backbones. Tab. A14 reports the correlation between the raw similarity signal and plak together with the fraction of trajectories exhibiting negative correlation; the raw signal varies widely across ViCLIP-L and ViCLIP-B, whereas the learned relevance remains small but positive in both cases.
We also assess robustness with respect to success correlation. Tab. A15 shows that the learned relevance and shaped reward remain positive and nearly unchanged across the two backbones, so the MVR pipeline still produces a stable, task-aligned signal when the base VLM is reduced in capacity,
Table A14: Correlation with y fask and fraction of negatively correlated trajectories (Sit Hard).
Backbone
cor'f y (MM plack)
Negative traj.
corr( JMV prank)
VICLIP (428M)
0.21
50%
0.22
IPCLIP-R (150M)
0.51
10%
0.23
Table A15: Correlation with success across VLM backbones (Sit Hard).
Backhone
com( success)
corr( MMV success)
corr(r MVN success)
VICLIP-L (428M)
0.91
0.98 0.96 0.95
VICLIP- B (150M)
0.90
0.97
B.8 THE EFFECT WEIGHTING PARAMETER w
The MVR framework introduces hyperparameter w that scales the V'MM-deried component in the shaped reward ,MR within the policy objective Eq. (6). We investigated the impact of different values of w on MVR's performance, selecting from (0.01, 0.1, 0.5) via grid search. Fig. A3 shows the final performance for different w values across selected tasks.
The results suggest that the optimal balance between task rewards and VLM guidance is task-specific. Tasks involving rapid, dynamic motion (e.g., 'Run') may benefit from smaller w, prioritizing the optimization of the primary task objective (r-fask). Tasks requiring precise posture or stability (e.g.. 'Stand') might benefit more from larger w, leveraging the VLM's visual understanding encoded in y MVR to achieve the desired configuration. This highlights the need for careful tuning of w according to the specific demands of the task. (Hu et al., 2020) is a potential solution for automatic parameter selection.
A12

---

Published as a conference paper at ICLR 2026
Stand
Run
800
1200
700
1000
600
Return
800
500
600
400
300
400
200-
200
100-
0.01
0.1
0.5
0.01
0.1
0.5
Slide
Sit Hard
1000
800
800
Return
600
600
400
400
200
200
0.01
0.1
0.5
0.01
0.1
0.5
The optimal value of w in Eq. (6) is task-dependent. Smaller values (w = 0.01) appear beneficial for tasks requiring rapid motion like 'Run' allowing the agent prioritize the task reward. Conversely, tasks emphasizing stability like 'Stand' benefit from larger values (w 0.1 or giving more weight to the VLM guidance. "Sit_Hard' shows peak performance at w 0.5, while 'Slide' performs best at 0.1. This indicates that tuning w based on task requirements may be necessary for optimal results.
Figure A3:
0.5),
A13

---

Published as a conference paper at ICLR 2026
B.9 COMPARISON WITH FURL
We include additional comparative results against FuRL (Fu et al., 2024), method that also utilizes visual information for reward generation. FuRL based on image-text similarity, its core mecha- nism involves leveraging successful trajectories to perform online fine-tuning of the Vision-LLnguage Model's (VLM) representations. This online fine-tuning aims to address potential VL.M misalignment with specific task objectives. To facilitate comparison on HumanoidBench. under a VLM interaction and representation update schedule comparable to MVR's s (i.e., fitting a reward model based similar rendering frequency), ensuring computational tractability for these demanding tasks. The results for FuRL on MetaWorld tasks, as presented in Tab. A16, are taken directly from the original FuRL publication (Fu et al., 2024). For HumanoidBench, conducted new experiments with FuRL operating under this adjusted interaction scheme, and these results are summarized in Tab. A17.
operated FuRL
FuRL's performance on HumanoidBench tasks (Table A17), when operated under VLM interaction schedule adjusted for scalability, is notably lower. core potential reason is that FuRL's strategy of fine-tuning VLM representations using successful trajectories is severely hampered in Humanoid- Bench. The inherent difficulty of exploration in these complex environments (cf. Appendix B.1) likely means that high-quality successful trajectories, which are crucial for FuRL's VLM fine-tuning process, are rarely discovered. This scareity of effective training data would directly impede the optimization of VLM embeddings. thus limiting the quality of the visual guidance FuRl. can provide for these challenging humanoid control tasks.
Table A16: MVR demonstrates competitive or superior performance compared to FuRL on the selected MetaWorkd tasks for which FuRL results are available. Both methods combine task-specific rewards (r.k with VLM-based rewards. FuRL results are sourced from (Fu et al., 2024).
Task
MVR
FuRL
push
+
window-close
1.00 1 0.00
1.00 1: 0.00
Average
0.55 + 0.10
0.53 0.04
Table A17: MVR significantly outperforms FuRL on all tested dynamic and static Humanoid- Bench tasks. For this comparison, FuRL,, which combines task rewards (1-tak) with VLM-bbseed rewards, was assessed using a VLM guidance mechanism updated at a frequency similar to MVR's for scalability. These results further underscore the effectiveness of MVR's approach in leveraging multi-view video guidance from VLMs for complex humanoid control.
Task
MVR
FuRL
Walk
927. 47 = 1.83
1.64 + U.31
Run
749.23 56.56.88
13.32 12.79
Stair
208.60 = 166.22 22
1.41 # 2.32
Slide
735.03 142.85
19.74 11.83
Stand
918.55 :: 29.30
30.15 +: 8.31
Simple
861.07 19.35
196.
154.17
Sit_ Hard
756.67 = 108.79
21.56
= 2.69
Balance Simple
299.87
58.45 45
21.32 11.43
Balance Haad
95.78 11.03
15.67 1: 3.11
B.10 PERFORMANCE AGAINST ORIGINAL ROBOCLIP ON HUMANOIDBENCH
We further benchmark MVR against the original RoboCLIP codebase (PPO. S3D) on Humanoid- Bench. This specific evaluation is critical as it assesses MVR against RoboCLIP in its unaltered, publicly available form. The substantially poorer performance the original RoboCLIP, detailed in Table A18, likely stems from several factors inherent to its design. Firstly, its reliance the S3D vision model, while capable video processing, may provide less nuanced visual-so semantcc understanding for reward generation compared to the advanced VLMs leveraged by MVR.
A14

---

Published as a conference paper at ICLR 2026
Secondly, RoboCLIP's original reward mechanism might not effectively capture complex, tempo- rally extended behaviors or handle viewpoint limitations as MVR's multi-view video-based approach. This contrasts with other analyses in this paper where baseline configurations (e.g., PPO to TQC. S3D to ViCLIP) were adapted for fairer component-wise comparisons, isolating specific architectural contributions.
Table A18: MVR substantially outperforms the original RoboCLIP (PPO, S3D) across all Bench tasks. The original RoboCLIP combines task rewards (rlisk) with VLM--bseed rewards from its S3D model.
Category
Task
MVR
RoboCLIP
Walk
927.47 1.83
25.31 + 5.98
Dynamic
Run
581.33 210.35
8.19 4. 3.09
Stair
164.74 = 132.96
19.25 = 8.63
Slide
622.05
27.37
8.28
Stand
925.55 25.10
36.67 + 7.06
Sit, Simple
757.34 143.96
16.95 1 1.10
Static
Sir Hard
705.29 = 105.18
11.53 - 4.20
Balance Simple
284.54 46. 82
1.39
Balance Hard
95.78 11.03
36. + 1.83
B.11 PIXEL-BASED METAWORLD EXPERIMENTS
We additionally evaluate MVR on the MetaWorld suite when policies receive pixel observations. Each agent augments proprioceptive input with embeddings from the R3M-VT VIT encoder (Nair et al., 2022), while MVR reuses the multi-view reward pipeline without extra tuning. The shaped reward MMR still combines environment feedback with the learned visual guidance. Results in Tab. A19 rt
show that MVR preserves its success-rate advantage even with raw RGB inputs.
Table A19: Pixel-conditioned MetaWorld success rates. Both agents operate on proprioception concatenated with R3M-ViT features. Returns are averaged over five seeds.
TQC (vision)
(vision)
hammer
0.17 0.24
0.80 1: 0.28
push-wall
0.33 0.47
0.17 1 0.24
faucet-close
1.00 = 0.00
1.00 + 0.00
push-buck
stick pull
0.00 0.00
0.00 1 0.00
handle predleepress-siee
1.00
1.00 + 0.00
push
0.00-10.00
0.00 1 0.00
shelf-place
0.00 0.00
0.000
window-close
0.00
1.00 0.00
peg-unplug-side
0.00 =: 0.00
0.20 + 0.22
Average
0.35 0.07
0.42 1: 0.07
B.12 SPARSE-REWARD METAWORLD EXPERIMENTS
To further assess MVR in settings where dense task rewards are unavailable, we construct sparse- reward variant the MetaWorld benchmark. In this setting, we remove all dense environment rewards and keep only the binary success signal, while leaving the visual shaping signal , ,LM unchanged. We then compare three configurations: TQC with dense rewards only (TQC-Dense), MVR with dense rewards (MVR-Dense, identical to the main-paper setting), and MVR with sparse environment rewards (MVR-Sparse).
Overall, MVR-Sparee attains average success rate comparable to TQC-Dense while removing dense environment rewards, indicating that MVR can leverage visual shaping to compensate for the lack of hand-engineered dense task rewards.
A15

---

Published as a conference paper at ICLR 2026
Table A20: Sparse-reward MetaWorld experiments. MVR-Sparse uses only sparse success signal from the environment plus visual shaping, yet achieves competitive performance with the dense-reward baseline TQC-Dense. Results are averaged over five seeds.
Task
TQC-Dense
MVR-Dense
MVR-Sparse
hammer
0.20 + 0.40
0.20 + 0.24
0.90 + 0.20
push-wall
0.10 0.20
0.60 + 0.49
0.30 0.24
faucet-close
1.00 0.00
1.00 0.00
1.00 + 0.00
push-back
0.60 + 0.37
0.50 + 0.45
0.00 + 0.00
stick-pull
0.10
0.20 0.24
0.00 0.00
handle-press piids--press
1.00
1.00 0.00
1.00 1.00 0.00
push
0.10
0.10
0.00 +
shelf-place
0.00 0.00
0.20 0.24
0.00
window-close
1.00
1.00 0.00
1.00
peg-unplug-side
0.30 + 0.40
0.60 - 0.37
Average
0.44 + 0.18
0.54 + 0.22
0.51 + 0.064
C ADDITIONAL ILLUSTRATION
C.1 OBJECTIVE FUNCTIONS FOR STATE RELEVANCE LEARNING
As detailed in Sec. 4.1, learning the state relevance function fMV involves addressing challenges arising from the discrepancy between state-space representations and visual video data, as well as viewpoint-induced variability in similarity scores when the same trajectory is rendered from different cameras. MVR tackles these using two learning objectives, Lmatehing and Lygs illustrated in Fig. A4.
Video Embeddings
Video Embeddings
Log
g
v-t Similarity
Similarity
gral
h(s.5
Leatching
h(o.o' )
Figure A4: The two objectives for learning state relevance. The state relevance function MVR comprises state encoder staae and a relevance vector get. Lmaacchin aligns the relative ranking of state sequences (derived from (MVR) with the ranking induced by VL.M video-text similarity scores. Trey regularizes the state sequence representations by enforcing consistency with the similarity structure observed in the VLM's video embedding space. This dual approach allows MVR learn state relevance effectively while mitigating noise from visual details and viewpoint variations.
Lmmiching encourages the state relevance function to preserve the relative ordering of behaviors indicated by the VLM's assessment of corresponding videos. Leg aligns the distance metric in the leamedd state embedding space with the distances in the VLM's video embedding space, promoting representations robust to view-specific interference. Together, these objectives enable MVR to learn meaningful state relevance function from multi-view video feedback.
A16

---

Published as a conference paper at ICLR 2026
D LIMITATIONS AND FUTURE WORK
MVR currently relies on a single textual description per task, which can underspecify complex behaviours. Extending the framework with adaptive stage-specific prompts promising direction for capturing richer task semantics.
The prototype assumes access multiple calibrated cameras to gather multi-view footage. Deploy- ments with limited viewpoints may require coupling MVR with novel-view synthesis modules such as NeRFs (Mildenhall et al., 2020) or active camera placement to maintain coverage.
While VLM guidance helps expose dynamic motion cues, combining video-text similarity with other perceptual signals may broaden applicability tasks where text alignment alone is insufficient. Finally, automating the selection of the weighting parameter w in the policy objective would reduce hyperparameter tuning effort and improve robustness across domains.
E BROADER IMPACTS
Robot skill learning an important topic for robotics and embodied artificial intelligence. This paper presents an approach that solicits feedback from VLMs for skill learning, thereby leveraging the rich knowledge stored in such models. With such method, this paper aims to pave the road towards automatic robot skill learning, desideratum of learning systems that enables intelligent robots to acquire diverse real-world tasks.
As foundational methodological contribution, this work does not present direct applications with immediately foreseeable negative societal impacts. Nevertheless, continued research in this founda- tional area, conducted with awareness of broader ethical considerations, is crucial for building a deeper understanding and fostering the responsible development more advanced autonomous systems in the future.
LLM Usage. A general- general-purpose purpose large language model (ChatGPT) was used solely for polishing grammar and improving the clarity of draft paragraphs. All technical content, equations, experimental design, and analysis were created and verified by the authors.
A17


Reviews:
[{"review_id": "ELyuvrJIDh", "reviewer": "Reviewer_LfqC", "summary": "This paper proposes Multi-View Video Reward Shaping (MVR), a reinforcement learning framework that uses multi-view video-text similarity from pre-trained VLMs for reward shaping. Unlike image-based methods that misguide agents toward static poses, MVR captures dynamic motions and mitigates viewpoint bias by introducing a state-dependent reward shaping that provides visual guidance early in training and automatically decays as the agent masters the task. Experiments on HumanoidBench and MetaWorld show MVR outperforms prior VLM-based methods, confirming its effectiveness in learning both dynamic and static skills", "questions": "- For 64-frame videos, this may be too short for locomotion tasks but potentially too long for simple manipulation tasks such as those in MetaWorld, where meaningful subtasks are limited. Does the video length affect downstream policy performance?\n- In Figure 4, why is t-SNE used? Would it not be more intuitive to visualize visual observations corresponding to the top/bottom-N frames ranked by task reward or MVR-shaped reward instead?\n- The experiments use only exocentric views. Are there ways—or any experiments conducted—to incorporate egocentric views (for humanoids) or wrist-camera views (for robotic arms)?\n- The default reward function in MetaWorld is a manually engineered dense reward. Was training conducted instead using a sparse reward?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper improves prior methods that measure rewards solely through static image-text similarity by introducing video-text-based rewards.", "weaknesses": "- Since both HumanoidBench and MetaWorld rely on reward engineering for task performance, the proposed method does not fundamentally solve the problem of reward shaping.\n- In Section 3, it is confusing and inappropriate to use the same function notation fff for both the off-the-shelf VLM’s text-similarity function and the function defined within the MVR framework, as they are conceptually different.\n- The writing could be improved for clarity. For instance, in Section 4.1, two challenges—(1) the semantic gap between states and videos, and (2) differences in viewpoints—are mentioned. “Matching Paired Comparisons” appears to address the semantic gap, while “Regularizing State Representations” handles viewpoint differences. These two should be presented more in parallel for better readability.\n- The performance gains are marginal compared to the baseline, especially on MetaWorld. The improvement attributed to introducing multi-view inputs, which is claimed as the paper’s main contribution, is not significant.\n- The task selection criteria for each benchmark are unclear. It is necessary to specify how tasks were chosen. Moreover, in DreamerV3 + MetaWorld, the number of RL training environment steps required for convergence varies across tasks, yet the paper uniformly uses 1M training steps, making it difficult to fairly compare performance.\n- In Figure 3b, adding multiple viewpoints does not show statistically significant improvement.\n- Some related works are missing in the citations: [1], [2].\n- Using only three random seeds is insufficient to demonstrate statistical significance. More seeds are needed for reliability.\n\n**References**\\\n[1] Subtask-Aware Visual Reward Learning from Segmented Demonstrations, ICLR 2025.\\\n[2] ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations, CoRL 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cHFxUTah7O", "reviewer": "Reviewer_mwgo", "summary": "This paper introduces a novel dense reward shaping method for reinforcement learning via VLM and real-time multi-view videos. The proposed method utilizes VLM to select the top-K reference multi-view videos and trajectories for shaping state-based dense rewards via the learned reference model fMVR. With this learned reward function, this paper conduct comprehensive experiments accross both the humanoidbench and metaworld to showcase its effectiveness.", "questions": "No specific questions", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method utilize multi-view reference videos for dense reward shaping, which can be generalizable to various different tasks without a hugh amount of human efford for reward design.\n\n2. The multi-view reference videos can improve the spatial understanding, which enhance the training efficiency.\n\n3. This paper conducts extensive experiments accross 19 tasks accross two simulation benchmarks.", "weaknesses": "1. During the earily stage of the training process, the video quality might be pretty low, even choose the top-k trajectories. Those low-quality data might bring limited guidance to finish the task. Especially for some challenging tasks such as stick pull and hammer in MetaWorld.\n\n2. Some previous works rely on generative model with reference trajectories and videos for reward shaping. Such as using reference trajectory [1], generated robot videos [2, 3, 4], and generated object motions [5, 6] from cross-embodiment dataset. Although those methods required reference data from robot or human, it might be helpful to discuss those related work. Also, some reward shaping ideas from those papers like conditional entropy and ranking might be good baselines for MVR.\n\n3. I think the tables and writings have some space to be improved, such as the table 4...\n\nI am willing to support this paper if the author can address my concerns\n\n[1]. Peng et al., DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills, SIGGRAPH 2018\n\n[2]. Escontrela et al., Video Prediction Models as Rewards for Reinforcement Learning, NeurIPS 2023\n\n[3]. Huang et al., Diffusion Reward: Learning Rewards via Conditional Video Diffusion, ECCV 2024\n\n[4]. Yang et al., Rank2Reward: Learning Shaped Reward Functions from Passive Video, ICRA 2024\n\n[5]. Yu et al., GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning, ICCV 2025\n\n[6]. Han et al., Learning Prehensile Dexterity by Imitating and Emulating State-only Observations, RA-L", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVwCt5HPqi", "reviewer": "Reviewer_yc5u", "summary": "This paper introduces VLM-based reward shaping for dynamic, suboptimal-view scenarios by using multi-view video–text similarity. First, to lower the cost of video-based rewards, this method distills each video into a low-dimensional state that preserves the task-relevance score. Second, to mitigate VLM bias toward particular viewpoints, it regularizes the representation so that similar videos map to nearby embeddings. The learned state then guides policy learning by rewarding states that best match the task’s textual description.", "questions": "- The authors use ViCLIP for RoboCLIP to ensure fairness. How would results differ with the original S3D VLM? Given that VLM choice can significantly impact performance, if applicable, could you report a backbone comparison and indicate whether your method remains strong across different VLMs?\n\n- Additional suggestion: the benefits of multi-view are well shown in Humanoid and Meta-World. The method would shine even more in occlusion-heavy settings (e.g., humanoid locomotion on obstacle terrain, conveyor-belt picking). Adding experiments in such scenarios would make the paper more solid.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strong motivation and sound strategy\n- The paper clearly motivates its aim of guiding policies toward optimal motion patterns. The VLM guidance automatically decays during training, curbing early suboptimal actions and ensuring convergence to the true task reward (not the shaping signal), achieving the goal in a principled way.\n\nThorough empirical analysis\n- The paper provides comprehensive analyses, baselines, and ablations of their design choices.\n- The paper acknowledges cases where it trails a task-reward-only baseline (TQC), likely due to domain mismatch from human-trained VLMs; this limitation is reasonable, and proposed mitigations (e.g., domain adaptation) are clearly outlined.", "weaknesses": "Loss of temporal information?\n- Although the method trains state sequences to match task relevance, the final reward in Eq. (9) is computed from a single timestep. Does this discard temporal information, thereby weakening the benefits of a video-based approach? This concern may be due to my misunderstanding; please clarify how temporal cues are preserved.\n\nNotation & Clarity\n- L187: I believe the term “similarity fluctuation” isn’t defined and was confusing—especially in the opening problem paragraph. Authors can replace it with a clearer term or add a brief definition.\n- (minor) L250–L255: When describing the full framework, adding a pointer to Algorithm A1 would help. I was looking for it while reading.\n- (minor) It was hard to tell when state/observation denotes a sequence vs. a single timestep. A short notation note (e.g., bold = sequence, regular = single step) would help. If this is obvious to your audience, feel free to skip.\n- (minor) L85: I believe the paper does not include real-world tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dmXbHaOKeA", "reviewer": "Reviewer_nDAs", "summary": "This paper proposes to use multiple views to shape VLM-guided rewards for learning policies. They ground VLM alignment scores from multiple views with the environment state and then use a learned state-based reward function to guide the policy in addition to using original task rewards. They evaluate this method, called MVR, on HumanoidBench and MetaWorld across many tasks.", "questions": "How effective do the authors think MVR will be with only sparse environment reward? This is important in manipulation settings where good task reward functions can be hard to obtain in the real world.\n\nWhat’s the difference between the main paper Metaworld results and Appendix B.7 “pixel-based metaworld experiments”? Is it just the policy inputs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Experimental Thoroughness:** The authors properly perform a wide array of ablations and experiments in two separate environments, each with many tasks. This paper is overall thorough, with a lot of analysis also in the appendix. \n\n**Well-motivated:** The approach is well-motivated, demonstrating the importance of using multiple views for reward shaping, and the reward shaping term for $r^{\\text{VLM}}$ is mathematically sound.\n\n**Results:** Overall, results are good and demonstrate the benefits of the approach over a decent set of baselines.", "weaknesses": "**Quite a bit of recent (and not-so-recent) prior work missing:** the related works section is quite short, missing some important recent work that is very relevant to this paper, and imo not comprehensive enough for an ICLR publication on robot learning. In fact, there are no reward model learning papers cited past 2024. All of the below reward learning work have been arxiv’ed and/or published beyond the 30 day ICLR limit, and I’m sure that I am still missing some important work:\n\n- [GVL](https://generative-value-learning.github.io/) can generate *video-based progression* rewards, zero-shot, using the Gemini VLM for a variety of downstream applications\n- [VLC](https://arxiv.org/abs/2405.19988) fine-tunes a VLM (clip4clip) to produce video-based rewards for trianing RL policies\n- [ReWiND](https://rewind-reward.github.io/) trains a video-language input transformer to produce *dense* rewards and demonstrates real-world RL fine-tuning\n- [PROGRESSOR](https://arxiv.org/abs/2411.17764) also performs real-world reward-weighted-regression with learned reward estimator from videos\n- [Rank2Reward](https://rank2reward.github.io/) learns a reward function with an extra GAIL term from video\n- The citation for RL-VLM-F is wrong; there are two citations at L619 and L623, the L619 one is for a completely different paper but with the RL-VLM-F title.\n\n**Notation:** This is a minor issue, but re-using $h$ $h$$h(o, o’)$ and $h(s, s’)$ is a bit confusing given the function $h$ itself in either case is applying two slightly different operations, and $h(o, o’)$ actually depends on $l$ too. \n\n**Writing:** At some points, the writing jumps around a bit. Things could be made clearer:\n\n- 4.1 **matching paired comparisons**: introduces a problem in the first paragraph (preserving orderings), but writing a little bit more at L153/154 to introduce the intuition behind Eq. 3 and how it solves the problem in the first paragraph would help with clarity\n- L187 starts with “A straightforward solution to the challenge of similarity fluctuation is…” when was this term introduced? It directly starts with a solution to a challenge not mentioned in the prior paragraph, this interrupted the flow while reading. Either introduce the problem again here or hint at it in the previous paragraph.\n- L198: $n(s)$ isn’t defined. I can understand what it is but this can be clearer by either defining it or stating that this is the “*average* representation of a state sequence”\n- L192: please explain why this exact parameterization for $f^{\\text{MVR}}$ was done. It took me as a reader extra time to think about *why* (i interpreted as a fixed, learnable orthogonal projection—actually, whether $g^{rel}$ is learnable isn’t even specified here—that helps align state representations), but a well-written paper should clearly *explain.*\n\n**Baselines:** Related to the above point on prior work, I’d like to see a modern VLM-only based baseline to demonstrate the need for the state alignment. For example, using a Qwen-3-VL (or to match params, SmolVLM-500M) model to zero-shot process video sequence input, at the same frequency as MVR (so not sparse reward like with RoboCLIP), for RL-VLM-F/GVL would make sense here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)