Reinforcement Learning with Foundation Priors: Let Embodied Agent Efficiently Learn on Its Own
Authors: Weirui Ye
Yunsheng Zhang
Haoyang Weng
Xianfan Gu
Shengjie Wang
Tong Zhang
Mengchen Wang
Pieter Abbeel
Yang Gao
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Foundation Models
Robotics
VLMs

Abstract: Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks.
However, it is challenging to apply the RL algorithms directly in the real world.
For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. 
For another, it is necessary to make heavy engineering efforts to design reward functions manually. 
To address these issues, we leverage foundation models in this paper. 
We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models.
Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions.
The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. 
Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. 
We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks.

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own
Weirui Y2
Yunsheng Zhang Haoyang Weng'
Xianfan Gu² Shengjie Wang
Tong Zhang 123
Mengchen Wang's Pieter Abbeel Yang Gao 123
Tsinnhua University, "Shanghai Qi Zhi Institute
Shanghai Artificial Intelligence Laboratory, "UC Berkeley
Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks. However, it challenging apply the RI algorithms directly in the real world. For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenar- ios. For another, is necessary to make heavy engineering efforts to design reward functions manually. To address these issues, we leverage foundation models in this paper. We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, introduce the Foundation-gul guided Actor-Crite (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. The benefits of our framework are threefold: (1) sam- ple efficient; (2) minimal and effective reward engineering; (3) agnostic to foun- dation model forms and robust to noisy priors. Our method achieves remarkable performances various manipulation tasks on both real robots and simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86% after one hour of real-time learning. Across tasks in the simulated Meta- world, FAC achieves 100% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designes rewards in IM frames. We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks. Visual- izations and code are available at https: : //ywwr.ee.e////// gitthu..io////
Abstract:
Keywords: Reinforcement Learning, Foundation Models, Robotics
Introduction
Reinforcement Learning (RL) has achieved remarkable success in various domains, including video games [1, 2, 3]. simulated robotics [4], and embodied agents for decades, such as real robots [5, 6, 8,8, 9]. However, current RL algorithms encounter two primary challenges: sample efficiency and heavy reward engineering, which hinder deployment in the real world. For example, researchers [1, 10] require millions of data to master games solve simulated robotics tasks [11, 12, 13]. Such amounts of data are unaffordable in real. And they also depend manually designed rewards, which is overly burdensome. Therefore, it is essential to address these two issues.
Humans acquire skills through minimal interactions with the environment by leveraging the innate abilities and abundant commonsense accumulated in daily life. They start from reasonable behaviors with fewer aimless explorations, make adjustments and corrections, and reinforce successful behav- iors. We take the baby pressing bottom an example illustrate this learning paradigm, which is commonly used in psychology [14, 15]. As shown in Fig. a baby named Alice is presented with novel toy box with button. The baby observes an adult pressing the button, causing the box to light
Cooresponding author. Emails: www.200miils..tsngaaaa...... gaoyangiiis @tsinghua.cdu.cn.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Coat: Fresa the bottor
Excadedie
Whatshoddd
goal!
Isucceed!
Y
Palicy Frler Knswrliss
Yaloe Pitcr Keowledge
Susceaa Prior Knowhelge
Figure 1: An example of how human solves tasks under the policy, value, and success-rewarl prior knowl- edge. The proposed Reinforcement Learning from Foundation Priors framework utilizes the corresponding foundation models to acquire prior knowledge.
and make sounds. Alice knows the behavior to solve the task, e.g.. pressing the button to activate the toy. She realizes that she can make it easier by positioning the hand closer to the button. Once she sees the toy box light up and hears the sound, she will reinforce the successful trial and repeat it.
Inspired by such a learning paradigm, we explore various foundation models provide learning signals and enhance learning efficiency. Then two fundamental challenges arise: (1) what is the con- crete form to present prior knowledge for RL; (2) how to leverage the corresponding prior knowledge effectively for downstream tasks. Based on the example shown in Fig. 1, three kinds of prior knowl- edge answer the question: what should I do now? am I closer to the goal? did I succeed? These prior knowledge are aligned with the core concepts well the Markov Decision Process (MDP), namely the policy function, the value function, and the success-reward function. Fortunately, the great success of the foundation models in natural language processing [16, 17, 18, 19] and computer vision [20, 21, 22, 23] makes it possible to acquire considerable and informative prior knowledge.
Consequently, for systematically utilizing the abundant prior knowledge to facilitate the embodied agent efficiently learning on its own, we introduce the Reinforcement Learning from Foundation Priors (RLFP) to leverage policy, value, and success-reward prior knowledge. The policy prior gives a warm start behavior of the agent; the value prior informs to reach better states, and the success-reward prior gives the final success feedback.
To verify the efficacy of RLFP, we instantiate actor-critic algorithm, named Foundation- guidded Actor-Critic (FAC), inspired by some works about building foundation models [24, 25, 26]. We conduct experiments on real robots and in simulation, and extensive ablations are made simula- tion. RLFP demonstrates three key benefits: (I) Sample efficient learning. Across the 5 tasks on real robots, FAC can achieve 86% success rates after hour of real-time learning. Across the 8 tasks in the simulated Meta-world, FAC can achieve 100% success rates in 7/8 tasks under less than 100k frames (about hour of training). It surpasses baseline methods that rely on manually designed rewards over 1M frames. (2) Minimal and effective reward engineering. The reward function is de- rived from the value and success-reward prior knowledge. eliminating the need for human-specified dense rewards or teleoperated demonstrations. (3) Agnostic to prior foundation model forms and robust against noisy priors. FAC demonstrates resilience under quantization errors simulations. To ensure high efficiency and performance, we utilize several well-trained foundation models or fine-tuning foundation models. The contributions are:
We propose the Reinforcement Learning with Foundation Priors (RLFP) framework. It systematically introduces three priors that are essential to embodied agents, and suggests how to leverage the existing foundation models as the priors.
We propose the Foundation-guided Actor-Critic (FAC), an RL algorithm under the RLFP framework that utilizes the policy, value, and success success-reward prior knowledge.
Empirical results demonstrate the remarkable performances of FAC. The ablations under- score the importance of each priors and validate the robustness against prior qualities.
Related Work
Foundation Models for Policy Learning. The ability leverage generalized knowledge from large and varied datasets has been proved in the fields of CV and NLP. In embodied AI, researchers at- tempt to learn universal policies based on large language models (LLMs) or vision-language models

---

(VLMs). Approaches include training large transformers through imitation learning [27, 28, 29, 30], offline RI [31], or fine-tuning pre-trained VI.Ms for downstream tasks [8]. Others use LLMs or VLMs for reasoning and control based on language descriptions [32, 33, 34, 35, 36, 37, 38, 39]. These methods rely on human teleoperation for data collection, but scaling this difficult. Some works generate code policies [40, 41] or predict future videos for action generation [25, 42], though they struggle with robustness due limited interaction with environments.
Foundation Models for Representation Learning. In addition to learning policies from founda- tion models, some researchers focus on extracting universal representations for downstream tasks. Several works leverage pre-trained visual representations to initialize perception encoders or extract latent image states [43, 44, 45, 46]. while others use pre-trained LLMs or VLMs for linguistic in- struction encoding [47, 48, 49]. Researchers have also explored applying LLMs/VLMs for universal reward value representation in RL, such language-conditione reward models [50, 51, 52] and universal goal-conditioned value functions trained large-scale videos [24]. Some approaches ac- quire value and success signals from human demonstrations [53, 54], while others generate rewards based on value differences to avoid manual reward design [55, 8j. However, these methods face limitations, including inefficient exploration due to a lack of policy-side guidance and unstable opti- mization from continuous reward signals with high error. Our framework addresses these issues by foundation priors, orthogonal existing data-efficient RL algorithms [56, 11, 2. 57, 58, 59].
3 Method
This paper aims enable embodied agents efficiently learn various tasks autonomously. In Sec. 3.1, we formulate the proposed Reinforcement Learning with Foundation Priors (RLFP) framework. Next, in Sec. 3.2, we present the Foundation-guice Actor-Critic (FAC) algorithm based on RLFP framework. Finally, Sec. 3.3 explains how to obtain the foundation models used in FAC.
3.1 Reinforcement Learning with Foundation Priors
We model the tasks for embodied agents as the Goal-Conditioned Markov Decision Processes (GCMDP) G: g P,, R. 7)). E IR" denotes the state. AA the action space, which the continuous delta movement of the end effector in this work. P is the transition probability function. T is the task identifier. R denotes the rewards. is the discounting factor, equal to 0.99 in the work. To learn efficiently and automatically, we propose the Reinforcement Learning from Foundation Priors (RLFP) framework by leveraging the policy, value, and success-reward priors.
Here we demonstrate how we formulate the priors RLFP. Back to the case of Alice in Fig. the commonsense of behavior can be formulated as conliconditooned policy function, Ms 7) S T A. The prior knowledge that the state closer the button is closer success can be formulated as the value function My T) S x T R'. The ability to recognize the success state can be formulated as the 0-1 success-reward function MR (s, T) S x T + {0, 1}, which equals 1 only if the task succeeds, We assume the success-reward prior is relatively precise, given the simplicity of binary classification in determining success. The value and policy prior knowledge are noisier. The RLFP framework is to solve an expansion G. termed = MG) where M the foundation model set that represents various foundation prior knowledge. Here, M,, My, MR E M.
Compared to vanilla RL, all the signals for the RLFP come from the foundation models. The vanilla RL relies on uninformative trial and eтror explorations and manually designed reward functions. It is not only of poor sample efficiency but also requires much human reward engineering. Instead, in RLFP, prior knowledge from the foundation model set M provides guidance or feedback on policy, value, and success-reward, enabling more automatic and effective task resolution.
3.2 Foudation-guided Actor-Critic
Under the proposed RLFP framework, instantiate actor-critic algorithm named Foundation- guided Actor-Critic (FAC), demonstrating how to inject the three priors into RL algorithms.
3

---

Guided by Success- reward Signals. We consider the task as MDP 91 with 0-1 success rewards, where Rg, = MR(s, 7) E (0, 13. Inspired by how humans learn from successful trials, we propose a success buffer to store the "successful" trajectories identified by Mg. Each time the actor ae updates via policy gradient, also imitates samples from the success buffer Dasse (if available). The objective is Suce (6) (T) o²)), St, aq succ. where ô is the standard deviation.
Guided by Policy Regularization. To encourage efficient explorations, we regularize the actor T by the policy prior from M₂ (s, T). Assuming the prior follows Gaussian distributions, the regularization term is (0) = ( T), ))))),, which is commonly used in other algorithms [60, 61]. The bias introduced by the policy prior is bounded, shown in Theorem
Guided by Reward-shaping from Value Prior. Noisy policy prior can mislead agents to undesirable states, so we propose using the value model My to guide ex- ploration and avoid unpromising states. While initializing and fine- tuning with My(s, 7) is natu- ral approach, it suffers from catas- trophic forgetting. Instead, we em- ploy the reward-shaping technique [62] using the potential-based func- tion F(s,8', T) = 7My 7) Mv(s, T), where y the discount factor. Since My estimates state val- ues, F measures the value increase from to s'. This shaping reward is positive when s' is better than and shares the same opti- mal solution as the 0-1 success-reward MDP G1. Proof and details are in App. A.1.
Pollcy gradlent: (E(Qu(s.4)
Pollcy Prior
stz
Pollcy regularization: VVlrey (9)
critic Qa
Shaping reward: F= <My(9 My(s))
Value Prior
Actor Re
Reward function: AMg
Success
Success trajectory Imitation:
M
-VLquer())
The overview of Foundation-guided Actor-Criie Critic. In FAC, rewards are derived from foundation success rewards and value shaping. Besides policy gradient updates, the actor trained using prior policy regularization and success trajectory imitation.
Figure 2:
Foundation-guided Actor-Critic. In summary, we deal with a new MDP 92, where Rg₂ = XM + F. with X (set to 100) emphasizing success feedback. We train the agent using DrQ-v2 [11], a variant of Actor-Critie, and call the proposed method Foundation-guile Fig. 2, FAC leverages foundation policy guidance and an automatic reward function, enabling the agent to efficiently learn from abundant prior knowledge. The objectives of FAC are detailed in Eq. (1), where tradeoff parameters and 3 are both set to 1, is the n-step TD target, and Qa is the target network. We use clipped double Q-learning [63] to reduce overestimation.
Actor-Critic (FAC). As shown in
(x) - -B,, BaNN
min Qe (oe, (8), ai) + aLuuce + BBooogi a, (8()) k=
(1)
Leriti( ERNND [[Qo, (se, at)
y) iy
Σ 1=0
A'ota + + min Qū, (Sc+n,
k=1.2
3.3 Acquiring Foundation Prior in FAC.
We focus on leveraging Foundation Priors in RI, not building large-scale foundation models, though we consider it an exciting future direction. For strong performance, we recommend using well- trained or pre-trained models. In this work, we use existing models as proxy foundation models. GPT-4V serves as the success-reward model MR for real tasks, but due to its poor performance in simulations, we use ground-truth 0-1 success rewards sim and distill a noisy success reward function for ablations. For value prior, we use the VIP model [24], which predicts value based on current and goal image observations. The policy prior M,, is built using code generation [40, 41] or video diffusion models [25]. Details are in the next section and in App. A.3.1 and App. A.3.2.

---

Pick Place
Open Door
Watering Plants
Unscrew Bottle Cap
Play Galf
Figure 3: Five tasks real robots, demonstrating the efficiency and accuracy of FAC in real.
Experiments
In this section, we provide detailed evaluations of the Foundation guiiedd Actor-Critic (FAC) on robotics manipulation tasks in both real-world and simulated environments. We also examine the impact of foundation prior knowledge through ablations simulation, focusing on sample efficiency and robustness. Our experiments aim to answer the following questions: (a) How sample-efficient FAC (Sec. 4.1 and Sec. 4.2): (b) What is the significance of each foundation prior (Sec. 4.3); (c) How does the quality of the foundation model affect FAC's performance (Sec. 4.3). Additional ablations and running time analysis are in App. A.5. Demo videos are included in the supplementary.
4.1 Manipulation tasks on Real Robots
Experimental Setup. We set up a real-world tabletop environment using a Franka Emika Panda robot with a 7-DoF arm and a 1 1-DoF parallel jaw gripper. Observations are collected from RGB images captured by fixed external camera and a wrist-mounted camera. We designed five dexterous manipulation tasks evaluate FAC, shown in Fig. 3. The agent learns for hour per task, except for "Pick Place," which trains for 30 minutes. The number of real-world trajectories collected for Pick Place, Open Door, Watering Plants, Unscrew Bottle Cap, and Play Golf are 40, 75, 60, 50, and 115, respectively. These tasks highlight the need for leveraging prior knowledge for accurate manipulation, which can be challenging for standard RI. The Franka Arm's starting position is fixed for each task, but object positions vary within predefined range. We evaluate each task across 10 trajectories with slight environment variations. Additional training details are in App. A.2.
Acquring Foundation Prior. (1) Prior Success-Reward: We use GPT-4V to determine success- rewards, evaluating only the final observation of each task. In test of 20 trajectories per task, GPT-4V demonstrated high accuracy, with no false negatives or positives, except in the Watering Plants task, which had 25% false- calse-positive rate due to the challenge correctly orienting the spout. Overall, GPT-4V is an effective proxy for success-reward prior knowledge. (2) Prior Policy: We use GPT-4V generate code for initial states, building on previous work [40, 41, 34] by providing primitive low-level skills like move_to, grasp, release, and rotate_antics ant.ccoccciis............o...... The generated code, based the robot's state, outputs prior actions, allowing us apply KL loss between the policy and prior actions in non-initial states. (3) Prior Value: We use the VIP model [24], universal value model trained on large-scale datasets. By inputting goal and current observations, we infer the value of states. Detailed prompts, generated policies, and prior settings are App. 1.3.1.
Efficient and Safe Exploration on Real Robots. To achieve higher sample efficiency on real robots, we choose high update-to-data (UTD) ratios with layer nor- malization in all MLP layers [64]. To achieve safer explo- ration, we introduce two key modifications. Firstly, we warm up the learning by taking the prior action to gather the first 10 trajectories. Secondly, under the guidance of critics, we selectively choose the superior action a" a; from either the actor Te or the policy prior Mn. where i arg max |ming Qe (s, a:)] a1 TT, 02 My. As training progresses, the agent increasingly opts for the actor's actions over the prior, shown in Fig. 4. For the actor, we maintain a constant standard deviation of O.1.
How often the action from the sctor by chosen dur traiiing
Auk Hace
Open
Unooren Bells Cap
May Out
Trsting Time (iuuto
Figure During training, the agent progres- sively favors actions from the actor, reducing reliance on the prior policy.
Baselines and Performance Analysis in Real. We compared our method to two baselines on real robots: (1) Vanilla RL using DrQ-v2 [11] with manually designed rewards, and (2) Policy Prior using the GPT-4V-generated code policy. Results are shown in Tab. 1. Vanilla RL failed in all tasks
5

---

Table 1: Quantitative Results of FAC during One-Hour Learning on the Franka Arm. The reported results are based on 10 evaluation trials. FAC achieves 86% success rate on average after one hour of training. This performance notably surpasees the code policy prior, underscoring its efficiency.
Task (Succ.)
Pick Place
Open Door
Watering.
Unserew.
Play Golfs
Avg.
Vanilla RL
0.00
0.00
0.00
0.00
0.00
0.00
Code Policy
0.30
0.10
0.30
0.20
0.20
0.22
FAC
1.00
0.90
0.80
0.90
0.70
0.86
after one hour of training due to excessive exploration. The code policy prior achieved an average success rate of 22%, benefiting from VLM's commonsense knowledge. Notably, FAC, guided by foundation priors, achieved an impressive average success rate of 86% across all tasks.
The learned policy in FAC adapts and refines its approach based on the prior policy to successfully complete tasks, as shown in Fig. 5. While the prior pol- icy struggles with grasping the door han- dle, FAC persistently secures the han- dle before pulling, significantly improv- ing performance. FAC's impressive results demonstrate the efficiency of our frame- work and highlight the potential of lever- aging abundant prior knowledge for em- bodied agents using the RLFP framework.
Both fail grasp
Try to pull back directly, fails!
AEsT Poary
Prior Policy
FAC
Change positions to grasp, success!
Figure 5: Prior policy attempts open the door without successful grasp, whereas FAC persistently tries to secure the handle before pulling back the arm.
4.2 Manipulation tasks in Simulations
Environments. We conduct experiments in 8 tasks from simulated robotics environment Meta- World [65], widely recognized for their ability to test diverse manipulation skills [60]. We average success rates over 10 evaluation episodes across runs with different seeds.
Few foundation models can distinguish success behaviors in embodied AI, and GPT-4V often fails with simulation images. Therefore, use the ground-trutt 0-1 success reward in simulation and distill noisy success-reward model for ablation in Sec. 4.3. (2) Prior Policy: To show that the prior model can be agnostic to form, we use a diffusion-based policy prior, following the UniPi [25] pipeline, which generates videos using dif- fusion models and infers actions from an inverse dynamics model. For efficiency, we offline distill policy model from videos generated by the open-source Seer [26] model, which predicts videos conditioned images and language instructions. While in-domain fine-tuning is ideally unneces- sary, current models fail in the simulator. Thus, we fine-tuned Seer with 10 example videos per task, though these videos are much noisier than the 200k used by UniPi, as shown in the supplementary materials. (3) Prior Value: We use the VIP model [24] with the same setup as in the real robot experiments. Implementation details of the diffusion policy are in App. A.3.2.
Acquiring Foundation Prior. (1) Prior Success-
Baselines. We benchmark our method against the following baselines: (1) Vanilla DrQ-v2 [11], with manually designed rewards from the suite; (2) R3M [46], VIP [24), where integrate the DrQ-v2 with either the R3M or VIP visual representation backbones. These baselines also rely on manually designed rewards from the suite; (3) UniPi [25]; (4) The distilled policy from UniPi
Performance Analysis in World. We compared our method to baselines on 8 Meta- Meta-World tasks with 1M frames. FAC achieved 100% success rates in all tasks, with 7/8 requiring fewer than 100k frames (about) 1 hour of training), and the harder bin-picking task needing under 400k frames. In contrast, bascline methods fail to reach 100% success most tasks. As shown in Fig. 6, FAC significantly outperforms baselines in both sample efficiency and success rates. While DrQ-v2 can complete some tasks, it learns much slower than FAC. Additionally, we warm up the actor with 10 success demos from the iiffusion-based prior policy before training. Although the improvement in DrQ-v2 with warmup is noticeable (Fig. 7), falls behind FAC, which benefits from policy prior.
R3M and VIP backbones provide visual representation prior knowledge for RL, but perform worse than DrQ-v2, likely due to the loss of plasticity in pre-trained models [66]. As UniPi and the dis-

---

bn pikking
button preds kepdown
door open
doa dxa-sriodos22
hummers22
15
Dkower
Drower open
245
2000
www.-aawe-aaa 2012
Prataza
Tiunso
Plano
Figure 6: Success Rate Curves for the 8 Tasks in Meta-Worke Our method consistently achieves 100% % success rates across all lasks, under the constrained performance of the policy prior model. It significantly outperforms the basclines with manual-designes rewards.
Picking
Butoa Pros Tepdown
Door Open
Dooe Unlock
Dower Close
Dawer Opes
Hamner
Wirdow Closs
75880
FAC
DrQ-
DoQ.+2 (wnamaa))
Figure 7: Comparison to the DrQ-v2 (warmup), which warm the actor by 10 collected success demos from the prior policy model. FAC achieves better performance across the 8 tasks generally.
tilled foundation prior policy lack environment interaction during training, they are represented as horizontal lines Fig. While UniPi outperforms the distilled prior environments, since the latter is learned from UniPi, both still far inferior to FAC.
4.3 Ablation Study in Simulations
In this section, we answer the following questions: (I) What's the importance of the three proposed priors? (II) How does FAC perform with noisier priors?
Ablation of Each Foundation Prior. To assess the importance of each foundation prior, we con- ducted experiments by removing them individually and comparing the results to the full method, along with an ablation study the success buffer (Fig. (a)). We found the reward prior to be the most crucial. Without it, performance drops significantly or fails, as the reward function reduces to shaping rewards, making policies indistinguishable. Without the policy prior, the agent struggles with difficult tasks like bin-picking and door-opening, and convergence slows on most tasks. Re- moving the value prior or success buffer reduces sample efficiency, especially for bin-picking, as the value prior helps guide policies under noisy policy priors. Success trajectory imitation further improves sample efficiency. Overall, using all foundation priors yields the best results.
As previous ablations show, without value prior knowledge, FAC's sample efficiency decreases, suggesting that better value prior can further boost performance. We now explore FAC's robustness to the quality of policy and success reward priors. First, we create noisier policy priors by discretizing each action dimension into 1,, 0, +1, providing only rough directional information, which we call the discretized policy. We also add uniform noise to the discretized actions at 20% and 50% probabilities. As shown in Fig. (b1), the discretized policy (blue curve) performs similarly the original (green curve), except for harder tasks like
FAC with Various Quality of Foundation Priors.

---

Ria peckang
Buctoo peess
Rook spco
Duoc snkck
Dosieec dlace
Droest ogen
Bammer
Window diose
Prunmss
Prusress
Prusuis
Piumis
Trunss
Preszes
butter
391
Bis perkeng
Button prese
Bear open
Dete snleck
Drewee clos
Drawer open
Pammer
Window deew
Fromes
Franses
Franes
Froist
Frozzes
Franess
FY1.005
Frames
policy
policy
(82)
Topdous
Unioc
Close
Dosser
Winden Clost
0.55
4.55 038
625
025
K21
X76
435
K11
polccy wrong!
polacy wrorg)
Bia perkeng
Butioa press
Door opan
Dese unleck
Deuwee close
Brawet open
Hamamer
Wasdoww cloee
Frursss
Frumues
Proctss
DMC
didilled aucovesenrd
Figure 8: Results of Ablation Study (a) Three Foundation Priors and the Success Buffer. (b) The Quality of Policy Prior. (c) The Quality of the nuccess-reward Prior.
bin- bin-picking and door-unlocking, which take more frames to reach 100% success. Adding noise further reduces performance, but even with 50% noise, FAC still achieves 100% success in many environments. We also tested robustness with systematically wrong policy priors, where actions have a 20% or 50% chance of being inverted (e.g.. -1 to 1) (Fig. 8 (b2)). FAC performs well with 20% wrong direction but struggles at 50%, where mislcading information is abundant. This shows FAC's robustness to moderate systematic noise.
To assess the impact of success-reward prior quality, we distilled proxy model using 50k offline data from the replay buffer across all 8 tasks. The model, conditioned on task embeddings, has 1.7% false positive and 9.9% false negative eгror on the evaluation datasets. We replaced the oracle 0-1 success reward with the predicted reward. As shown in Fig. (c), FAC with the 50k-image proxy model experiences only limited performance drop compared to the oracle reward and outperforms FAC without success rewards. This demonstrates that FAC performs well even with noisy success- reward prior. In conclusion, our ablation studies show that FAC resilient variations in the quality of foundation priors. The higher the quality, the more sample-efficient FAC becomes.
5 Discussion
In this paper, introduce novel framework, termed Reinforcement Learning from Foundation Priors (RLFP), which leverages policy, value, and success-reward prior knowledge for RL. Addi- tionally, we also detail the implementation of this concept within actor-critic methods, introducing the Foundation- Actor-Criie Critic (FAC) approach. Extensive experiments on real robots and sim- ulated environments demonstrate the effectiveness of RLFP in efficient autonomous learning.
Limitation and Future Work This work has limitations, such as relying human engineering for designing low-level skills, prompts, and code policy demos. Additionally, we fine-tune diffusion models with in-domain data for better prior knowledge. However, as foundation models improve, future methods may only require simple task instructions without fine-tuning. Future exploration can focus on building more accurate and broadly applicable foundation priors and incorporating richer priors into the RLFP framework. For example, humans can predict future states, and integrating the predictive knowledge from dynamic foundation models could enhance policy learning.

---

Acknowledgments
This work is supported by the Ministry of Science and Technology of the People 's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), This work is also supported by the National Key R&D Program of China (202ZZD0161700).
References
[1] Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(588)) 609, 2020.
[2] W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25444 2021.
[3] K. Arulkumaran, A. Cully, and J. Togelius. Alphastar: An evolutionary computation perspec- tive. In Proceedings of the genetic and evolutionary computation conference companion, pages 314-315, 2019.
[4] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 2021.
[5] A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
[6] Mahadevan and Connell. Automatic programming of behavior- -based robots using rein- forcement learning. Artificial intelligence, 55(2-3):311-366 1992.
[7] P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226-2266622222 2240. PMLR, 2023.
[8] J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn. Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning. arXiv preprint arXiv:2310. /0.15455, 2023.
[9] S. Haldar, Pari, A. Rai, and Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint a:X1v:2303.01497, 2023.
[10] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. arXiv preprint 1r 2017.
[11] D. Yarats, Fergus, Lazaric, and L. Pinto. Mastering visual continuous control: Improved dutm-augee reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[12] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprini wxXiv:2301001104, 2023.
[13] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16825 2023.
[14] A. N. Meltzoff. Infant imitation after a I-week delay: long-term memory for novel acts and multiple stimuli. Developmental psychology. 24(4):470, 1988.
[15] A. N. Meltzoff. Understanding the intentions of others: re-enactment of intended acts by
18- -ld children. Developmental psychology, 31(5):838, 1995.

---

[16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[17] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini 1XX:::8800004005 2018.
[18] T. Brown, B. Mann, N. Ryder. M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1101, 2020.
[19] OpenAl. GPT-4 technical report. CoRR, abs/2333.087744 2023. doi: 06::10448550//xXx..333. 08774. URL https:///oot: wrw...r.g/... 0 0555//0..85500/xxx..22222 08774.
[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXi::2010.1192 2020.
[21] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMIR, 2021.
[22] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125, 2022.
[23] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.026. 2023.
[24] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via inlueiimplicit pre-training. arXiv preprint arXiv:2210.00030, 2022.
[25] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv preprint 200000000001 2023.
[26] X. Gu, C. Wen, Song, and Y. Gao. Seer: Language instructed video prediction with latent diffusion models. arXiv preprint arXiv:2303. 2023.
[27] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 2rXiv:22/2.068//1 2022.
[28] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: lasiinnlagguage-ace models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.18811 15818, 2023.
[29] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2005.06175, 2022.
[30] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. 15500, 2023.
[31] Y. Chebotar, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog, K. Pertsch, K. Gopalakrishnan, et al. Q-transformer: Scalable offline reinforcement learning via autore- gressive fffunctions. In 7th Annual Conference Robot Learning, 2023.
10

---

[32] N. Di Palo, A. Byravan, L. Hasenclever, M. Wulfmeier, N. Heess, and M. Riedmiller. To- wards a unified agent with foundation models. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023.
[33] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models, arXiv preprint arXiv:2207. 005608, 2022.
[34] M. Ahn, Brohan, N. Brown, Chebotar, Cortes, B. David, Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, al. Do can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 2022.
[35] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0333
2023.
[36] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658, 2023.
[37] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523- 11530. IEEE, 2023.
[38] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[39] Y. Hu, F. Lin, T. Zhang, L. Yi, and Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17844 2023.
[40] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3493-9500 IEEE, 2023.
[41] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and I.. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint 7rXiv:2307.05973. 2023.
[42] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[43] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. arXiv preprint arXiv:2302.12766, 2023.
[44] R. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning, arXiv preprint arXiv:2107.033000 033880 2021.
[45] A. Majumdar, K. Yadav, S. Arnaud, Y. Ma, C. Chen, S. Silwal, A. Jain, V.P. Berges, Abbeel, J. Malik, al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303. 18240, 2023.
[46] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203./2601, 2022.
[47] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.

---

[48] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language- coniiiii-nnd robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[49] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:22 10.03094, 2022.
[50] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with snternetsscale knowledge. arXiv preprini wrXiv2220.008833 2022.
[51] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning dunguage-conditiona robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315, PMLR, 2022.
[52] P. Mahmoudich, D. Pathak, and T. Damell. sero-shot reward specification via grounded natural language. In International Conference Machine Learning, pages 14743-14752. PMLR, 2022.
[53] C. Eteke, D. Kebüde, and B. Akgün Reward learning from very few demonstrations. 1EEE Transactions on Robotics, 37(3):893- 9004,099992000000 2020.
[54] B. Wu, F. Xu, He, A. Gupta, and K. Allen. Squirl: Robust and efficient learning from video demonstration of long-horizon robotic manipulation tasks. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9720-9727. IEEE, 2020.
[55] H. Xiong, R. Mendonca, K. Shaw, and D. Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint arXiv:2401.14403, 2024.
[56] I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, and M. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipu- lation, arXiv preprint orXiv:1704.0300 2017.
[57] W. Ye, Y. Zhang, P. Abbeel, and Y. Gao. Become proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations, 2022.
[58] S. Wang, S. Liu, W. Ye, You, and Y. Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprini arXiv:2403.005: 2403006644 2024.
[59] D. Hafner, Pasukonis, J. Ba, and Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301.0414 104104, 2023.
[60] Haldar, V. Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference Robot Learning, pages 32 43.43. PMLR, 2023.
[61] P. Lancaster, N. Hansen, A. Rajeswaran, and V. Kumar. Modem-v2: Visuo-motor world models for real-world robot manipulation. arXiv preprint arXiv:2309. 14236, 2023.
[62] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287. Citeseer, 1999.
[63] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[64] J. Ball. L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02944 2023.
12

---

[65] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[66] P. D'Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
[67] M. Dorigo and M. Colombetti. Robot shaping: an experiment in behavior engineering. MIT press, 1998.
[68] M. J. Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, pages 181-189. Elsevier, 1994.
[69] J. Randløv and P. Alstrøm. Learning to drive bicycle using reinforcement learning and shap- ing. In ICML, volume 98, pages 463-471, 1998.
[70] E. Cetin, P. J. Ball, S. Roberts, and O. Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. arXiv preprint arXiv:2207.00988, 2022.
[71] R. Zheng, Wang, Y. Sun, S. Ma, Zhao, H. Xu, H. Daumé III, and F. Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. arXiv preprint 1322299.06.13229, 2023.
[72] R. Cheng, A. Verma, G. Orosz, S. Chaudhuri, Y. Yue, and J. Burdick. Control regularization for reduced variance reinforcement learning. International Conference on Machine Learning. pages 1141-1150. PMLR, 2019.
13

---

A
Appendix
Appendix Table of Contents
Appendix A.1: Reward Shaping in FAC
Appendix A.2: Experimental Details of FAC
Appendix A.3: Details of Acquring Foundation Priors
Appendix A.4: Running Clock Time Analysis
Appendix A.5: More Ablations Results
Appendix A.6: Proof of the Optimality under Policy Regularization
A.1 Reward Shaping in FAC
In this work, we apply the value prior knowledge in Actor-Critic algorithms in the format of reward shaping. Reward shaping guides the RL process of an agent by supplying additional rewards for the MDP [67, 68, 69]. In practice, it is considered a promising method to speed up the learning process for complex problems. Ng et [62] introduce a formal framework for designing shaping rewards. Specifically, we define the MDP G (S, P, R), where A denotes the action space, and p Pr[5 (c, denotes the transition probabilities. Rather than handling the MDP G, the agent learns policies on some transformed MDP g' = (S)),, R' = R F, where F is the shaping reward function. When there exists a state-only function $ S - R such that F(s,a, (0''') ((s) (7 is the discounting factor), the F is called potential-based shaping functioo. Ng et al. [62] prove that the potential-basee shaping function F has optimal policy consistency under some conditions as follows:
Theorem [62] Suppose that F takes the form of F(s, 8') -b()) ((s0) - 0 if? = 1,
thenfor Vs E S, a E A, the potential- based F preserve optimal policies and we have:
(3)(s, a) = Q2 (s, a) 4sss
(2)
(z) VG (s) (s)
The theorem indicates that the potential basel shaping rewards treat every policy equally. Thus it does not prefer zτ. Moreover, under the guidance shaping rewards for the agents, a significant reduction in learning time can be achieved. In practical settings, the real-valued function & can be determined based on domain knowledge.
A.2 Experimental Details of FAC
Since FAC is built upon DrQ-v2, the Syper-parameee of training the actor-critic model the same as DrQ-v2 [11]. The n-step TD target value and the action in Eq. I are as follows, where 0g are the moving weights for Q target networks.
y Tres + 7 min Qs (St.n, Ut+n)
k=1.2
(3)
==0
4. (s()) - clip clip(
Sayeee-parameters in FAC Morever, the observation shape is 84 x 84 on real robots and in simula- tion. For better representations of the scene, we use a wrist camera on real robots. In simulation, we stack frames and repeat actions for 2 steps, while we do not stack frames or repeat actions on real robots. In Meta-World, we follow the experimental setup of [60]. Specifically, the horizon length which
set to 125 frames for all tasks except for bin-picking and button-press-topoonn 175. The total frames of the 8 tasks 100k, except for the task bin bin-picking, which is set to IM. Notably, set the same camera view of all the tasks for consistency. On real robots, we set the
to
14

---

actor Joss
critic lows
policy reg loss
success, br. loss
0.59
Success rate
-06
Success retee
Success
200
Surrass
6.75
c.03
-256
50-3.00 ngvv
-4.25
0.0
145
actoor loss
critic loss
policy reg loss
success bc loss
Success Fath
LOC
335
#xonoperv
Environment 104
Environment Steps
Environment
Environment Steps
Figure Training loss curves of bin-picking and door-open.
horizon the 5 tasks (Pick Place, Open Door, Water Plants, Unscrew Bottle Cap. Play Golf) 40, 40, 50, 60, and 25, respectively. We evaluate them every 10 minutes, and the positions of the objects vary in 5cm in Pick Place, Open Door, Watering Plants, and 3em in Unscrew Bottle Cap and Play Golf. We reset the objects manually because the current foundation model cannot deal with reset problems. The difference in the hyper-parameters between the real robots and the simulation is as follows:
Table 2: Main Difference the Hyper-parameters FAC on real robots and in simulation.
Parameter
On robots
In simulation
Frame Stack
Action Repeat
Seed Frames
10 Trajectory
4000
Std of Actor
0.1
1 + 0.1
Feature_ dim in DrQ-v2
512
50
UTD ratio
20
(N/A)
Choose hetter action under Q
No
Use wrist camera
Yes
No
The training loss curves of the task bin-picking and door-open for reference, as shown in Fig. 9.
More Experimental Setup Details on real robots
Here give the task description of the 5 tasks Franka Arm. For the task Unscrew Bottle Cap, will fix the bottle when the arm tries to unscrew the cap.
Pick Place: Pick up the purple eggplant and place it onto the blue plate.
Open Door: There is a white cabinet on the table and there is a cucumber in it. The door can only be opened from the outside. Please open the door by a large margin so that the cucumber can be seen.
Watering Plants: There are blue watering kettle and a potted plant. Please help me watering the plant and keep watering still.
Unscrew Bottle Cap: There are plastic bottle with green cap and pink plate the table. The bottle will be fixed on the table, so that you cannot lift the bottle. Please help
15

---

unserew the bottle cap and place it on the pink plate. You mush know how to unserew the bottle cap, anticlockwise or clockwise?
Play Golf: There is golf hole and a golf ball on the table. The robotic arm is tied by golf club. Please shoot the golf ball into the set.
The ground-truth success reward functions of each task are as follows, which is leveraged in evalu- ation.
Pick Place: dist(xy position of the eggplant, xy position of the blue plate center) < radius & dist(z of the eggplant, of the blue plate) < 0.01.
Open Door: dist(xy position of the handle, xy position of the cabinet center) E, e 0.1.
Watering Plants: dist(z of the sprout, 2 of the plants) > 61 & dist(xy of the sprout, xy of the
plants) < radius abs(horizonta orientation of the sprout) < e2 degree, e1 0.1, 62 = 15.
Unserew Bottle Cap: dist(xyz of the cap, xyz of the bottle center) > & (cap in the plate),
E = 0.15. ('cap in the plate" is the same as the function in pick place).
Play Golf: (golf in the golf hole). ( ('golf in the golf hole' is the same as the function in pick place).
Reward functions vanilla RL Since is hard to get the object state in real time, we assume the object state will be attached to the robot state after some actions. For example, we have the initial position of the eggplant, the position, and the size of the target plate. After the gripper reaches the pos of the eggplant and takes action "grasp", the pos of the eggplant will be set the same as the eef pos. The reward function is the normal pick-place reward function. For the door-open, we get the handle initial position, and once the arm is reaching in the bound of 5cm of the handle and closes the gripper, the position of the handle will be attached to the eef of the robot. Therefore, the reward functions are based on the distance between the gripper and some target position sequences, including the position distance and the orientation distance. We give the reward functions of the Door Open and Play Golf examples, which are harder tasks.
Open Door: Rapproach = max (0, 1 - tanh(10 x Pgripper Phandiell). encourages the robot to get closer to the handle by diminishing rewards as the distance decreases. Recient = 19ppe=-quail incentivizes aligning the robot's gripper orientation with that of the handle. Rpull
max (0, Paver ppeev_dour) increases with the door's opening angle. Ropen min faue 1.0), pri- oritizes achieving 90-degree opening. A small time penalty, Riime -0.01, is applied per time step to encourage faster task completion. The total reward is then formulated as: Ryotal 0.01* Rappoach + 0.05 Rorient 0.1 Rpall + 0.1 Ropen + Rime-
Play Golf: We use the golf club's length and the end-effector's (eef) position to determine the clubhead's state. The ball back position is defined as the back of the ball, which is the optimal point strike. Rapprouch max (0, tanh(10 10 Peluhheed Poall buck I])), encourages the robot to position the clubhead at the back of the ball. Rocient =
Sahhhaa--uure which incentivizes
2
proper alignment of the clubhead for an accurate strike. Additionally, the reward for the clubhead's velocity at the moment of impact, Rvelocity = max (0, - tanh (10 x ||V Vdesired| )), ensures that the clubhead reaches the optimal speed to strike the ball effectively. So the final reward is: R 0.1 Ryppoaac + 0.5 Ruiient + 0.5 Ryelocit + Riime-
16

---

The robotic arm in the image appears to nave positioned the spout the watering can directly over the which suggests that is the process watering the plant This setup typically indicative of sutometed or rubolic syslem designed lo plants without human interventon.
0 The spou: of the wataring can is not oriented horizontally
plant, rether, it lhted at angle suggests may not activlly wataring plant at this moment. appeee to be position that might be before alter the walering action. possibly paused state.
Figure 10: Example of success identification by GPT-4V in task Watering Plants. Given the question. Does the robotic arm water the plants? Attention, the spout orients horizontally over plant, you should output for yes. Otherwise, you should output 0 for no without any space first. Be sure of your answer and explain your reason afterward. The foundation model can give the correct success reward as well as the corresponding explanations.
A.3 Details of Acquring Foundation Priors
A.3.1 Acquring Foundation Priors on Franka
Examples of Success-reward Discrimination Prompts is significant feed the success-reward model by high-resolution images, which choose 512 512. Here give example of the success-reward prompts in FAC towards the task Watering Plants, by leveraging GPT-4V, as follows: Does the robotic arm water the plants? Attention, if the spout orients over horizontally over the plant, you should output for yes. Otherwise, you should output 0 for no without any space first. Be sure of your answer and explain your reason afterward. Then we can receive the corresponding success-reward as well as the explanations, shown in Fig. 10.
Examples of Code Policy Prompts. Before code generation, define some primitive skills. We implement the corresponding interface between primitive skills and control systems, so that they can be executed directly by the robot. The primitive low-level skills and the corresponding params are as follows:
move_to x y z: move the gripper to the position (x, y, 2).
grasp: grasp with the gripper.
release: release the gripper.
rotate clauk_eelocwiie: rotate gripper clockwise by 90 degree.
rotate_anticlockwise: rotate gripper anticlockwise by 90 degree.
orient half horizontally: change the orientation of the gripper half horizontally to make it grasp more casily.
orient_ verttcclet change the orientation of the gripper vertically make grasp some- thing vertically.
strike_from strike something to the front of the arm.
strick_backe strike something the back of the arm.
Firstly, we input prompts that include task description, the initial scene, and the expected format for the generated code. Notably, the task used in the example differs from the five tasks we evaluate. For
17

---

the downstream tasks, we rely on the GPT-4V to generate the code policy based on new images and the corresponding task descriptions, shown in Fig. 11. We also specify the range of object positions within the scene, enabling the VLM to estimate the position of the target object and generate code that includes skill plans and position parameters. Limited to the current ability of GPT-4V, it can build reasonable code policies under small changes in object positions.
Then, we feed in the current image and the task descriptions: There are a plastic bottle with green cap and a pink plate on the table. The bottle will fixed on the table, so that you can- not lift the bottle. Please help me unserew the bottle cap and place on the pink plate. You mush know how unscrew the bottle cap, ticlockwise or clockwise? Then we can receive the generated code policy as shown in the code text, which can make the correct rotation to un- screw the cap.
On real robots, the action space is 7-dim, con- sisting of the delta position (x, of the grip- per, the delta orientation (roll yaw pitch) of the gripper, and the grasp/release of the gripper. We use the OSC control mode. As for the policy output of the code policy, generates plans like "move to x1 yl z1". Then we use robotics con- trol toolkits to compute the delta position/ori- entation for the robot, which is the prior action. Once the plan "move_ta xl yl z1" is finished (close to epsilon in practice), the code policy will execute the next plan. In this way, we can acquire the corresponding prior action in each step. The KI. constraint between the prior action and the RL policy. Consequently, the GPT-4V generates code plans that consist of the primitive skills, and then the primitive skill can generate delta prior actions by robotics control codes.
Figure 11: Initial observation image of the task Un- screw Bottle Cap. including the fixed camera view and the wrist camera view.
18

---

Policy Description Prompts Unserew Bottle Cap
Generated Code Policy Unserew Bottle Cap
Query: pick the plastic bottle
def code_policy(1
8
Sone infa... (skip here)
8 Bottle cap pesition (estinated near height top table z-axis) the bottle cap_pos [0.5, 0., 9.26]
are skills
you choose:
nove the gripper to the position of [x, y, 2)
nove_to
grasp: grasp with the gripper 8 3. release: release the gripper
yl, cap_pos
rotate_clockwise:
the gripper clackwise by only 96 degree rotate_anticlockwiss rotate gripper anticlockwise by 90 degree reset: move back to the initial position
Plate position (estimated the provided pasition) plate_pos [0.75, 8., 8.06]
x2,
plate_pos
a Plans
the bottle and place it on the plate
Attention: For safety, you cannot rotate clockwise
plans
anticlockwise twice continuously. Think carefully about to finish the task.
Nove above the bottle cap
('move_to to (x1:,2f) (y1:.20) (21 8.05:.
Move to the bottle cap
def code_policy() bottle. Estimate position. object_pos [e.5, 0., 8.12]
f'move_to (x1:.2f) (y1:.2f) (z1:.2
Obtain the position of plastic
f)
Grasp bottle cap f' f'grasp', Rotate anticlockwise to unserew "esstte.nttcclockisse..
x1, yl, 21 object_pos
Pick
plans [
Lift the cap t'move_to (x1:.2f) (y1:.2f) 0.05:
above the bottle f toove_to (x1:,2f) (y1:.2f) (2)
8.05:.2f)'
plate
a nove to the bottle
"nvve_to (x2:.2f) (y2:.2f) (zl 2f7'
nove__t (x1:.2f) (y1:22f) (21:.2
f)',
Nove down to the plate F'nove_to (x2:.2f) (y2:.2f) (x2:.2
# grasp the bottle grasp
f)',
# lift the cucunber
Release the cap onto the plate f'release',
to (xl:.2f) (y1:22f) (2t 0.2:.2f)',
Nove away f'move to (x2:.2F) (y2:.2f) (22 8.2:.2f)*
return plans
] done
plans
A.3.2 Acuqring Foundation Priors on Meta-world
Training Inverse Dynamics Model We build the inverse dynamics model p(st, 86+1) as follows:
Takes inputs as St, S1+1, with the shape of 84 84.
A Downsample Model, which outputs the representation with the shape 128 x 2 x 2.
Flatten the planes into 312-dimension vectors.
Linear layer with ReLU, which outputs the 64-dimension vectors.
Linear layer with ReLU, which outputs the 64-dimension vectors.
Linear layer with ReLU, which outputs the action dimension vectors (equal to 4).
The Downsample model is designed as follows:
convolution with stride and 128 output planes, output resolution 42 x 42, (ReLU)
2 residual block with 128 planes.
Average pooling with stride 2 (kernel size is 3), output resolution 21 x 21. (ReLU)
2 residual block with 128 planes.
Average pooling with stride (kernel size is 5), output resolution (ReLU)
2 residual block with 128 planes.
Average pooling with stride 3 (kernel size is 4, no padding), output resolution 2. (Rel.U)
19

---

picking
of training FAC agents the same as button press- topdown
door-opor poor-open
door-unlock
drawer-close
drawer
hammer
window close
Figure The generated videos from the diffusion model Seer given the initial images as well as task descrip- tions.
We use the 1M replay buffer trained from vanilla DrQ-v2 for each task and collect them together as the dataset.
Distilling Diffusion Policy Foundation Models We use the fine-tuned VLM Seer to collect 100 videos for each task (1000 in bin-picking-v2), and use the trained inverse dynamics model p(s,, St41) to labal pseudo actions for the videos. The example of generated videos is illustrated in Fig. 12. Then, we do supervised learning to train the policy foundation prior model under the dataset, which conditioned on the task. For convenience, we encode the task embedding as a one- hot vector, which labels the corresponding task. Thus, the size the task embedding Here, the architecture of the distilled policy model follows, where the downsample model the as that in the inverse dynamics model.
Takes inputs 8,, Ct, with the shape of x 84 x 84 and 8.
A Downsample Model, which outputs the representation with the shape of 128 x 2 x 2.
Flatten the planes into
vectors.
Concat the 512 vector and the task embedding into 520-dimension vectors.
Linear layer with ReLU, which outputs the 64-dimension vectors.
Linear layer with ReLU, which outputs the 64-dimension vectors.
Linear layer with ReLU, which outputs the action dimension vectors (equal 4).
The training hyper-parameters of the inverse dynamics model p(st, St41) and the distilled policy model M2 T) are in Table 3. The DrQ-v2 [11].
20

---

Table Hyperr-prammelers for Building the Policy Foundation Models in Neeta-World.
Parameter
Training p
Training M,
Minibatch size
256
256
Optimizer
Adam
AdamW W
Optimizer: learning rate
lc-4
5c-4
Optimizer: weight decay
le-4
le-4
Learning rale schedule
Cosine
Cosine
Max gradient
Training Epochs
50
300
A.4 Running Clock Time Analysis
A.4.1 Analysis of Extra Running Clock Time in FAC
Since it brings great performance and efficiency increases for RL by leveraging the foundation prior knowledge, it is interesting and significant to investigate the clock time of running the foundation models in practice. Here we discuss the time consumed by each part from the foundation models in real and simulation. In practice, some foundation models are accessed only at the beginning or at the final step, which results in much smaller amortization time over the trajectory. For example, GPT- 4V detects success only in the final step. To make fair comparisons, time per trajectory and get the amortized time per step. We set the trajectory length to 50 steps for convenience. To make clear comparisons, we conclude the operations without access to foundation models into Normal Operation part, and those operations with access to foundation models into Foundation Model Operation part. All the results are evaluated a single 3090 GPU, and we calculate the average clock time during training among 50 trajectories on average from all the tasks.
record the running clock
Tab. 4 and 5 the results of real robots and simulations concerning clock time during training. Here, Action Move means the process of taking actions by the robotic arm. The Action Inference means the inference time of the learned policy model. Notably, in the simulation, we offline distill a prior policy from the diffusion model due to the heavy time-cost of video generation (in Sec. 3.3). Thus, the clock time the Diffusion Policy Prior for video generation is not included in the training time, although it much more consumingg than the other operations simulation.
Generally, in both environments, the time complexity of rumning foundation models is about 2 times more than the vanilla setting in amortization. And the most time-consuming part is the Action
Table 4: Running Time of Each Part in FAC on Real Robots. On real robots, the most time consuminn
part of the foundation model operation is generating policy code from GPT-4V. The total amortized time of foundation model operations little larger the normal operations.
Clock Time on Real Robots (s)
Per Trajectory
Amontized Step
Normal Operation
Action Move of the gripper in the real world Action Inference from policy क (als) Total
0.320
0.001
0.321
Foundation Model Operation
Code Policy Prior: policy code generation from GPT-4V
14.76
0.290
Code Policy Prior: action generation by code policy
0.001
Success Prior: from GPT-4V
4.48
0.089
Value Prior: value inference from VIP
0.013
Total
0.393
21

---

Table 5: Running Time of Each Part in FAC Meta- Meta-World. In the simulation, the most time-consuming part of foundation model operation is value inference from VIP The total amortized time of foundation model operations is smaller than the normal operations.
Clock Time in Simulation (s)
Per Trajectory
Amontized Step
Normal Operation
Action Move of the gripper in simulation Action Inference from policy π (a]s)
0.009
0.001
Total
0.010
Foundation Model Operation
Diffusion Policy Prior: video generation (done offline)
H1.62
0.230
Diffusion Policy Prior: action generation
0.001
Success Prior: from the success model
0.001
Value Prior: value inference from VIP
0.007
Total
0.008
pxstng
Netton preas
Dost open
sclrck
Doweer cless
Drower spes
Henner
Window
Praties
France
Frunse
Pramus
Freties
PAC
D=Q
ALD
1A00
Figure 13: Here er DrQ-v2, ALIX, and TACO means only the success reward is provided from environment, which different from the original settings in their works. FAC can work for all the tasks while the other baselines fail in half of them. It is significant and sample-efficient to utilize prior knowledge for reinforcement learning.
Move. In foundation model operations, generating the code policy from GPT-4V is the slowest on real robots, while the value inference is the slowest in simulation.
Moreover, we record the total training time spent by FAC and vanilla DrQ-v2 make comparisons. We average the seeds running for each task where we train about 100k steps on single 3090 GPU. Vanilla DrQ-v2 takes 32m 2s average; while FAC takes Ih 1 Immn 17s average, which is 2 times more than that without foundation priors. As shown in Fig. 6, even the same clock time, FAC is better than vanilla DrQ-v2 with ground-truth truth rewards. FAC can solve 7/8 tasks in 100k frames, while the vanilla DrQ-v2 can only solve 2/8 in 200k frames. Furthermore, FAC can solve more tasks than the baselines. Consequently, we conclude that our method significantly outperforms the baselines concerning training clock time.
A.5 More Ablations Results
Comparison to More Baselines with Success-reward Only Here, we also add some baselines under the setting, where only the success-reward foundation prior is provided. We choose the recent SOTA model-free RL. RI. algorithms on Meta- World ALIX [70] and TACO [71], as well as the baseline DrQ-v2 [11] with the success-reward only. Notably, ALIX and TACO are both built on DrQ-v2. The results are shown in Fig. 13, where ** means that only 0-1 success reward is given. Only FAC can achieve 100% success rates in all the environments. DrQ-v2*, ALIX*, and TACO* can not work on hard tasks such as bin bin-picking and door-open. FAC requires fewer environmental steps to reach 100% success rates, as shown in the Figure. The results on the new baselines can verify the significance and efficiency of utilizing the abundant prior knowledge for RI. in a way.
Comprasion to BC Policies from Policy Prior Another interesting baseline of leveraging founda- tion policy prior is to collect success demonstrations by the prior policy M, and train behavior cloning policy. Here collect 100 successful demonstrations on Meta-World for each task by the
22

---

Table 6: The performance comparison between the distilled policy prior and the learned be policy on Meta- World.
Prior Policy M
BC Policy
FAC
bin-picking-v2
0
1.00
button
0.45
0.15
1.00
door-open-v2
0
0
1.00
door-unlock-v2
1.00
drawer-close-v2
0.10
0.10
1.00
drawer-open-v2
0.05
0
1.00
hammer-v2
0.15
0.10
1.00
window-close-v2
0.30
0.05
1.00
prior policy Mr. Noticing that the policy prior achieves 0% success rate some tasks, we col- lect 100 trajectories then. The results are in Tab. Generally, we find that the learned BC policy cannot outperform the prior policy itself in the tasks. More significantly, in some tasks, such as windowclosev2, the BC policy achieves much worse results than the prior policy. This is because the prior policy works only in some certain scenarios, which introduces bias in the collected de- mos. Although the prior policy fails in some scenarios, can introduce some informative actions, which can be much better than random actions. Therefore, it is more reasonable to leverage the prior actions as guidance for RL.
A.6 Proof of the Optimality under Policy Regularization
Lemma The policy Tm I+3 fm... actor shown in Equation 1.
+ 143 Ma. the solution to the optimization problem of the
Proof First, fm.. the RL policy optimized by standard RL optimization problem in m-th itera- tion, illustrated in the following equation.
Rho arg max ET (Q(, as m
0c
(4)
Note that the following derivation omits the variance of Gaussian distribution for convenience. This because the variance is independent of the state in the deterministic Actor-Criti algorithms DrQ- v2 algorithm.
According to Equation 1, the policy Tm be represented as:
Tm arg min -Er= 33L(m, Mz)]
(5)
Adding Erntyy (ss, a) in Equation 5, we can rewrite it as:
Tm arg minnInnnns....... Q(s a) Er- (s,, 3KL(T, Mx)]
(6)
Considering E (ss, a) is not related to the optimization objective, the above equation holds. Intuitively, we can observe that there exist two parts in the objective. About the first part, we can use importance sampling to obtain:
Tr Q(s,a))
ErT+on Q(s,a) a) ET Q(8,a) = E
frmm
(7)
fr
Since #o... can be represented arg maxs, Ernag ((ss, a)) when approaching to infinity, the minimum of Er-atom Q(s, Ex,= Q(s, a) can be achieved when the minimum of the following equation exits.
arg min f xl arg min arg max E,wa, [Q(s, - allas - Ro
(8)
Let see the second part in Equation 6. and Mx Gaussian distributions and the variances of distributions constant in our framework. Thus, KL(T,, M,) $ || Mx || holds.
23

---

Hereafter, we can reformulate Equation 6 as follows: 7m arg min
[||
arg max Br Ar T| 311= M
(9)
Based on the Lemma 1 in [72], the solution to the above problem is derived as:
3
+ B mm +
M
(10)
1
To this end, the policy Tm is the solution to the proposed optimization problem in this paper.
Theorem 2 Let Sub Drv(Fopt: Mx) be the bias between the optimal policy and the prior policy, the policy bias DrV (T Topt) in m-th iteration can be bounded as follows:
Drv(Tms Tape) 2 Drub
1 Drmm Mn)
(11)
B
Drv(Tmy Tops) 1 Daub as m +
Proof 2 Note that the following derivation is most inspired by Theorem in [72]. According to Lemma 1, the policy nm can be represented as T4 + T4 M..
Then, let us define the policy bias as Drv(Tm, Topt), and Duuh DrV (Tppt; Mx). Since Drv is a metric that represents the total variational distance, we can use the triangle inequality to obtain:
Drv (Tm, Topt) 2 Dry (Mx, Topt) (M
(12)
According to the mixed policy definition in Equation 10, can further decompose the term Dry (My, m.):
1
B
Drv(MT; km)
sup (s.0)ESxA
M
1 Tpm
M
1 3
sup
M
(13)
+ 3
(s,a)ESxA
1
DrV D M₂)
1 3
This holds for all m E N from Equation 12 and Equation 13, and we can obtain the lower bound as follows:
Dry (Tmy Topt) Daub
1 3 Drv Ma)
(14)
The RL policy fe can achieve asymptotic convergence to the (locally) optimal policy Topt through the policy gradient algorithm. In this case, we can derive the bias between the mixed policy nm and the optimal policy Topt as follows:
DrV (Topt: Tm) = sup
Topt
1 B $ tm
1 B M
(s.A)eSxA
3
sup (s,a)ESxA
Topt
M
as OC
1 B
(15)
Drv (Topt, Mx) as m 0c
=
+ B
8
Daub
+ B
Therefore, we obtain the upper bound.
24


Reviews:
[{"review_id": "mT4mM7ErsQ", "reviewer": "Reviewer_M4P9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper utilizes \"foundation models\" to improve the efficiency of reinforcement learning of robotic manipulation tasks. The idea is to leverage these models to get policy priors (to help with exploration), value/reward \"priors\" (provide dense signals, mostly for \"reward shaping\") and success \"priors\" (the final and arguably the main reward signal which is sparse). This way, there is no need to provide human demonstrations and/or to engineer rewards. The effort is shifted to prompting the foundation models (authors imply but do not explicitly state that this is easier) and designing primitive robot actions. The authors test their approach on both real world and simulated manipulation tasks. There is no strong baseline for the robotic task but the results are positive. For simulated tasks, the presented method performs much better than the regular RL algorithms with engineered rewards for the allocated time budget. The authors also ablate the components of their approach.", "summary_of_recommendation": "Overall, the results are strong and limitations are sufficiently presented and discussed.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Take the review as the main source. This part emphasizes some of the more important aspects:\n\n- Concerns about the tite\n- Concerns about the specific contributions of the method\n- Addressing the assumptions about the foundation models. On a related note, a discussion on different types of noise for the acquired priors.\n- The RL formulation should be clearly stated. The simulation one can be left for the appendix.\n- How to acquire the priors are left for the appendix, is it possible to summarize them in the main text? \n- How are the losses combined? What are their respective weights? The loss curves can be given in the appendix to help visualize their contribution to learning in addition to the ablations.\n- Collecting the required human effort (e.g. prompt engineering, fine-tuning, providing goal states etc.) in a single place (e.g. discussion) would better serve the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PjestENsDC", "reviewer": "Reviewer_kNo7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper explores the best ways to incorporate foundation models into training a new RL policy. It explores three kinds of priors the foundation model can provide - policy, reward, and value. The paper performs a comprehensive study of these components in simulation and the real world. Utilizing all the components leads to an impressive sample efficiency, with the algorithm learning real-world policies in 1 hour of interaction with a new environment.", "summary_of_recommendation": "Overall, the paper's contribution and results are solid, and while still limited, they provide an important step in incorporating foundation models into RL training pipelines.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- In section 4.1, you mention, \"the positions of objects vary within a predefined range across different trajectories.\" Can you provide more information on that? In addition, have you tried to vary the position beyond the training range to check for generalization capabilities? In general, the paper lacks generalization experiments.\n- For section 4.1, while the wall clock is indeed the correct value for comparison, please also provide the number of real-world trajectories that were collected during this time. It is important to understand the sample efficiency of the algorithm.\n- One thing that is missing is the evaluation of the prior success reward. How accurate is it? Can you provide an analysis, either in an environment where you have access to the GT or using a human observer.\n- You mentioned that the GPT-4V prior policy generate a trajectory based only on the initial state. If so, how do you take the KL loss between the policy and prior policy actions on non-initial states? You don't have the prior policy actions for these states.\n- The idea of keeping a success buffer and adding a BC loss to it doesn't have anything to do with foundation models and can be applied with any reward. However, it is not a common practice in off-policy RL. Why do you believe that it should be used here? Especially when your ablation shows that it helps in only one of the tasks you consider (Bin Picking).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TaBSibcK3l", "reviewer": "Reviewer_bb5a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an algorithm using three foundation models to ease reinforcement learning for real-world tasks.", "summary_of_recommendation": "I think that this paper requires a bit more work regarding the experiments. Also I would change the storyline, just combining a few common techniques is not enough for calling it a general framework. I would just calling it an algorithm. Hence, I am voting for rejection, but I am willing to raise my score if my concerns are addressed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was initially evaluated with mixed but mostly positive reviews. After the rebuttal, the reviewers suggest strongly accepting the paper.

Here is a high level overview of the reviews. (before rebuttal)
### Strengths
- The paper performs a comprehensive study of policy, reward, and value priors in both simulation and real-world settings, demonstrating impressive sample efficiency and strong results. (Reviewer M4P9, Reviewer kNo7)
- The approach is sound and novel, combining the acquired "priors" effectively, and includes sufficient ablations to validate its effectiveness. (Reviewer M4P9)
- The paper shows good results and extensive experiments, evaluated on a range of real-world and simulated tasks. (Reviewer bb5a)

### Weaknesses
- The title is convoluted and not informative enough, and the introduction lacks proper historical context with some inaccuracies. (Reviewer M4P9)
- The priors, especially the policy, heavily rely on human engineering for designing low-level skills and prompts, and the method's claim of being "agnostic to foundation models" needs expansion. (Reviewer M4P9, Reviewer kNo7)
- The RL formulation is missing crucial details such as action space, state space, and discount factor, and the paper lacks generalization experiments beyond the predefined training range. (Reviewer M4P9, Reviewer kNo7)
- The storyline should be changed, as combining common techniques is not enough to call it a general framework, and additional ablations are needed for better validation. (Reviewer bb5a)

Most concerns were addressed in the rebuttal. 
The authors are asked to read the updated reviews and make sure to implement the promised and suggested changes.
As an additional note:
- the font sizes in the graphics are way too small, please update them. (e.g. by using a small fig_size in matplotlib)
Decision: Accept