Reinforcement Learning with Foundation Priors: Let Embodied Agent Efficiently Learn on Its Own

Authors: weirui  ye
yunsheng  zhang
haoyang  weng
xianfan  gu
shengjie  wang
tong  zhang
mengchen  wang
pieter  abbeel
yang  gao
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-02 02:50:41

Abstract: Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks.
However, it is challenging to apply the RL algorithms directly in the real world.
For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. 
For another, it is necessary to make heavy engineering efforts to design reward functions manually. 
To address these issues, we leverage foundation models in this paper. 
We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models.
Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions.
The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. 
Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. 
We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks.

Main paper: 
 introduction: 
 Reinforcement Learning (RL) has achieved remarkable success in various domains, including video games [1, 2, 3] , simulated robotics [4] , and embodied agents for decades, such as real robots [5, 6, 7, 8, 9] . However, current RL algorithms encounter two primary challenges: sample efficiency and heavy reward engineering, which hinder deployment in the real world. For example, researchers [1, 10] require millions of data to master games or solve simulated robotics tasks [11, 12, 13] . Such amounts of data are unaffordable in real. And they also depend on manually designed rewards, which is overly burdensome. Therefore, it is essential to address these two issues.Humans acquire skills through minimal interactions with the environment by leveraging the innate abilities and abundant commonsense accumulated in daily life. They start from reasonable behaviors with fewer aimless explorations, make adjustments and corrections, and reinforce successful behaviors. We take the baby pressing bottom as an example to illustrate this learning paradigm, which is commonly used in psychology [14, 15] . As shown in Fig. 1 , a baby named Alice is presented with a novel toy box with a button. The baby observes an adult pressing the button, causing the box to light igure 1: An example of how human solves tasks under the policy, value, and success-reward prior knowledge. The proposed Reinforcement Learning from Foundation Priors framework utilizes the corresponding foundation models to acquire prior knowledge. and make sounds. Alice knows the behavior to solve the task, e.g., pressing the button to activate the toy. She realizes that she can make it easier by positioning the hand closer to the button. Once she sees the toy box light up and hears the sound, she will reinforce the successful trial and repeat it.Inspired by such a learning paradigm, we explore various foundation models to provide learning signals and enhance learning efficiency. Then two fundamental challenges arise: (1) what is the concrete form to present prior knowledge for RL; (2) how to leverage the corresponding prior knowledge effectively for downstream tasks. Based on the example shown in Fig. 1 , three kinds of prior knowledge answer the question: what should I do now? am I closer to the goal? did I succeed? These prior knowledge are aligned with the core concepts well in the Markov Decision Process (MDP), namely the policy function, the value function, and the success-reward function. Fortunately, the great success of the foundation models in natural language processing [16, 17, 18, 19] and computer vision [20, 21, 22, 23] makes it possible to acquire considerable and informative prior knowledge.Consequently, for systematically utilizing the abundant prior knowledge to facilitate the embodied agent efficiently learning on its own, we introduce the Reinforcement Learning from Foundation Priors (RLFP) to leverage policy, value, and success-reward prior knowledge. The policy prior gives a warm start behavior of the agent; the value prior informs to reach better states, and the success-reward prior gives the final success feedback.To verify the efficacy of RLFP, we instantiate an actor-critic algorithm, named Foundation-guided Actor-Critic (FAC), inspired by some works about building foundation models [24, 25, 26] . We conduct experiments on real robots and in simulation, and extensive ablations are made in simulation. RLFP demonstrates three key benefits: (1) Sample efficient learning. Across the 5 tasks on real robots, FAC can achieve 86% success rates after 1 hour of real-time learning. Across the 8 tasks in the simulated Meta-world, FAC can achieve 100% success rates in 7/8 tasks under less than 100k frames (about 1 hour of training). It surpasses baseline methods that rely on manually designed rewards over 1M frames. (2) Minimal and effective reward engineering. The reward function is derived from the value and success-reward prior knowledge, eliminating the need for human-specified dense rewards or teleoperated demonstrations. (3) Agnostic to prior foundation model forms and robust against noisy priors. FAC demonstrates resilience under quantization errors in simulations. To ensure high efficiency and performance, we utilize several well-trained foundation models or fine-tuning foundation models. The contributions are:• We propose the Reinforcement Learning with Foundation Priors (RLFP) framework. It systematically introduces three priors that are essential to embodied agents, and suggests how to leverage the existing foundation models as the priors.• We propose the Foundation-guided Actor-Critic (FAC), an RL algorithm under the RLFP framework that utilizes the policy, value, and success-reward prior knowledge.• Empirical results demonstrate the remarkable performances of FAC. The ablations underscore the importance of each priors and validate the robustness against prior qualities. 
related work: 
 Foundation Models for Policy Learning. The ability to leverage generalized knowledge from large and varied datasets has been proved in the fields of CV and NLP. In embodied AI, researchers attempt to learn universal policies based on large language models (LLMs) or vision-language models (VLMs). Approaches include training large transformers through imitation learning [27, 28, 29, 30] , offline RL [31] , or fine-tuning pre-trained VLMs for downstream tasks [8] . Others use LLMs or VLMs for reasoning and control based on language descriptions [32, 33, 34, 35, 36, 37, 38, 39] . These methods rely on human teleoperation for data collection, but scaling this is difficult. Some works generate code policies [40, 41] or predict future videos for action generation [25, 42] , though they struggle with robustness due to limited interaction with environments.Foundation Models for Representation Learning. In addition to learning policies from foundation models, some researchers focus on extracting universal representations for downstream tasks. Several works leverage pre-trained visual representations to initialize perception encoders or extract latent image states [43, 44, 45, 46] , while others use pre-trained LLMs or VLMs for linguistic instruction encoding [47, 48, 49] . Researchers have also explored applying LLMs/VLMs for universal reward or value representation in RL, such as language-conditioned reward models [50, 51, 52] and universal goal-conditioned value functions trained on large-scale videos [24] . Some approaches acquire value and success signals from human demonstrations [53, 54] , while others generate rewards based on value differences to avoid manual reward design [55, 8] . However, these methods face limitations, including inefficient exploration due to a lack of policy-side guidance and unstable optimization from continuous reward signals with high error. Our framework addresses these issues by foundation priors, orthogonal to existing data-efficient RL algorithms [56, 11, 2, 57, 58, 59] . 
method: 
 This paper aims to enable embodied agents to efficiently learn various tasks autonomously. In Sec.3.1, we formulate the proposed Reinforcement Learning with Foundation Priors (RLFP) framework.Next, in Sec. 3.2, we present the Foundation-guided Actor-Critic (FAC) algorithm based on RLFP framework. Finally, Sec. 3.3 explains how to obtain the foundation models used in FAC.We model the tasks for embodied agents as the Goal-Conditioned Markov Decision Processes (GCMDP) G: G = (S, A, P, R, T , γ). S ∈ R m denotes the state. A is the action space, which is the continuous delta movement of the end effector in this work. P is the transition probability function. T is the task identifier. R denotes the rewards. γ is the discounting factor, equal to 0.99 in the work. To learn efficiently and automatically, we propose the Reinforcement Learning from Foundation Priors (RLFP) framework by leveraging the policy, value, and success-reward priors.Here we demonstrate how we formulate the priors in RLFP. Back to the case of Alice in Fig. 1 , the commonsense of behavior can be formulated as a goal-conditioned policy function, M π (s, T ) : S × T → A. The prior knowledge that the state closer to the button is closer to success can be formulated as the value function M V (s, T ) : S × T → R 1 . The ability to recognize the success state can be formulated as the 0-1 success-reward function M R (s, T ) : S × T → {0, 1}, which equals 1 only if the task succeeds. We assume the success-reward prior is relatively precise, given the simplicity of binary classification in determining success. The value and policy prior knowledge are noisier. The RLFP framework is to solve an expansion of G, termed G ′ = (G, M), where M is the foundation model set that represents various foundation prior knowledge. Here, M π , M V , M R ∈ M.Compared to vanilla RL, all the signals for the RLFP come from the foundation models. The vanilla RL relies on uninformative trial and error explorations and manually designed reward functions. It is not only of poor sample efficiency but also requires much human reward engineering. Instead, in RLFP, prior knowledge from the foundation model set M provides guidance or feedback on policy, value, and success-reward, enabling more automatic and effective task resolution.Under the proposed RLFP framework, we instantiate an actor-critic algorithm named Foundationguided Actor-Critic (FAC), demonstrating how to inject the three priors into RL algorithms.Guided by Success-reward Signals. We consider the task as MDP G 1 with 0-1 success rewards, where R G1 = M R (s, T ) ∈ {0, 1}. Inspired by how humans learn from successful trials, we propose a success buffer to store the "successful" trajectories identified by M R . Each time the actor π ϕ updates via policy gradient, it also imitates samples from the success buffer D succ (if available). The objective is L succ (ϕ) = KL(π ϕ (s t ), N (a t , σ2 )), s t , a t ∼ D succ , where σ is the standard deviation.Guided by Policy Regularization. To encourage efficient explorations, we regularize the actor π ϕ by the policy prior from M π (s, T ). Assuming the prior follows Gaussian distributions, the regularization term is L reg (ϕ) = KL(π ϕ , N (M π (s t , T ), σ2 )), which is commonly used in other algorithms [60, 61] . The bias introduced by the policy prior is bounded, shown in Theorem 2. Guided by Reward-shaping from Value Prior. Noisy policy prior can mislead agents to undesirable states, so we propose using the value model M V to guide exploration and avoid unpromising states. While initializing and finetuning with M V (s, T ) is a natural approach, it suffers from catastrophic forgetting. Instead, we employ the reward-shaping technique [62] using the potential-based functionF (s, s ′ , T ) = γM V (s ′ , T ) - M V (s, T ), where γ is the discount factor. Since M V estimates state values, F measures the value increase from s to s ′ . This shaping reward is positive when s ′ is better than s and shares the same optimal solution as the 0-1 success-reward MDP G 1 . Proof and details are in App. A.1.Foundation-guided Actor-Critic. In summary, we deal with a new MDP G 2 , where R G2 = λM R + F , with λ (set to 100) emphasizing success feedback. We train the agent using DrQ-v2 [11] , a variant of Actor-Critic, and call the proposed method Foundation-guided Actor-Critic (FAC). As shown in Fig. 2 , FAC leverages foundation policy guidance and an automatic reward function, enabling the agent to efficiently learn from abundant prior knowledge. The objectives of FAC are detailed in Eq.(1), where tradeoff parameters α and β are both set to 1, y is the n-step TD target, and Qθ is the target network. We use clipped double Q-learning [63] to reduce overestimation.EQUATIONWe focus on leveraging Foundation Priors in RL, not building large-scale foundation models, though we consider it an exciting future direction. For strong performance, we recommend using welltrained or pre-trained models. In this work, we use existing models as proxy foundation models. GPT-4V serves as the success-reward model M R for real tasks, but due to its poor performance in simulations, we use ground-truth 0-1 success rewards in sim and distill a noisy success reward function for ablations. For value prior, we use the VIP model [24] , which predicts value based on current and goal image observations. The policy prior M π is built using code generation [40, 41] or video diffusion models [25] . Details are in the next section and in App. A.3.1 and App. A.3.2.  
experiments: 
 In this section, we provide detailed evaluations of the Foundation-guided Actor-Critic (FAC) on robotics manipulation tasks in both real-world and simulated environments. We also examine the impact of foundation prior knowledge through ablations in simulation, focusing on sample efficiency and robustness. Our experiments aim to answer the following questions: Experimental Setup. We set up a real-world tabletop environment using a Franka Emika Panda robot with a 7-DoF arm and a 1-DoF parallel jaw gripper. Observations are collected from RGB images captured by a fixed external camera and a wrist-mounted camera. We designed five dexterous manipulation tasks to evaluate FAC, shown in Fig. Efficient and Safe Exploration on Real Robots. To achieve higher sample efficiency on real robots, we choose high update-to-data (UTD) ratios with layer normalization in all MLP layers [64] . To achieve safer exploration, we introduce two key modifications. Firstly, we warm up the learning by taking the prior action to gather the first 10 trajectories. Secondly, under the guidance of critics, we selectively choose the superior action a * = a i from either the actor π ϕ or the policy prior M π , wherei = arg max i∈1,2 [min k Q θ k (s, a i )] , a 1 ∼ π ϕ , a 2 ∼ M π .As training progresses, the agent increasingly opts for the actor's actions over the prior, shown in Fig. 4 . For the actor, we maintain a constant standard deviation of 0.1.Baselines and Performance Analysis in Real. We compared our method to two baselines on real robots: (1) Vanilla RL using DrQ-v2 [11] with manually designed rewards, and (2) Policy Prior using the GPT-4V-generated code policy. Results are shown in Tab. 1. Vanilla RL failed in all tasks The learned policy in FAC adapts and refines its approach based on the prior policy to successfully complete tasks, as shown in Fig. 5 . While the prior policy struggles with grasping the door handle, FAC persistently secures the handle before pulling, significantly improving performance. FAC's impressive results demonstrate the efficiency of our framework and highlight the potential of leveraging abundant prior knowledge for embodied agents using the RLFP framework.Environments. We conduct experiments in 8 tasks from simulated robotics environment Meta-World [65] , widely recognized for their ability to test diverse manipulation skills [60] . We average the success rates over 10 evaluation episodes across 3 runs with different seeds.Acquiring Foundation Prior. (1) Prior Success-Reward: Few foundation models can distinguish success behaviors in embodied AI, and GPT-4V often fails with simulation images. Therefore, we use the ground-truth 0-1 success reward in simulation and distill a noisy success-reward model for ablation in Sec. 4.3. (2) Prior Policy: To show that the prior model can be agnostic to form, we use a diffusion-based policy prior, following the UniPi [25] pipeline, which generates videos using diffusion models and infers actions from an inverse dynamics model. For efficiency, we offline distill a policy model from videos generated by the open-source Seer [26] model, which predicts videos conditioned on images and language instructions. While in-domain fine-tuning is ideally unnecessary, current models fail in the simulator. Thus, we fine-tuned Seer with 10 example videos per task, though these videos are much noisier than the 200k used by UniPi, as shown in the supplementary materials.(3) Prior Value: We use the VIP model [24] with the same setup as in the real robot experiments. Implementation details of the diffusion policy are in App. A.3.2.Baselines. We benchmark our method against the following baselines: (1) Vanilla DrQ-v2 [11] , with manually designed rewards from the suite; (2) R3M [46] , VIP [24] , where we integrate the DrQ-v2 with either the R3M or VIP visual representation backbones. These baselines also rely on manually designed rewards from the suite; (3) UniPi [25] ; (4) The distilled policy from UniPi.Performance Analysis in Meta-World. We compared our method to baselines on 8 Meta-World tasks with 1M frames. FAC achieved 100% success rates in all tasks, with 7/8 requiring fewer than 100k frames (about 1 hour of training), and the harder bin-picking task needing under 400k frames. In contrast, baseline methods fail to reach 100% success on most tasks. As shown in Fig. 6 , FAC significantly outperforms baselines in both sample efficiency and success rates. While DrQ-v2 can complete some tasks, it learns much slower than FAC. Additionally, we warm up the actor with 10 success demos from the diffusion-based prior policy before training. Although the improvement in DrQ-v2 with warmup is noticeable (Fig. 7 ), it falls behind FAC, which benefits from policy prior.R3M and VIP backbones provide visual representation prior knowledge for RL, but perform worse than DrQ-v2, likely due to the loss of plasticity in pre-trained models [66] . As UniPi and the dis- tilled foundation prior policy lack environment interaction during training, they are represented as horizontal lines in Fig. 6 . While UniPi outperforms the distilled prior in most environments, since the latter is learned from UniPi, both are still far inferior to FAC.In this section, we answer the following questions: (I) What's the importance of the three proposed priors? (II) How does FAC perform with noisier priors?Ablation of Each Foundation Prior. To assess the importance of each foundation prior, we conducted experiments by removing them individually and comparing the results to the full method, along with an ablation study on the success buffer (Fig. 8 (a) ). We found the reward prior to be the most crucial. Without it, performance drops significantly or fails, as the reward function reduces to shaping rewards, making policies indistinguishable. Without the policy prior, the agent struggles with difficult tasks like bin-picking and door-opening, and convergence slows on most tasks. Removing the value prior or success buffer reduces sample efficiency, especially for bin-picking, as the value prior helps guide policies under noisy policy priors. Success trajectory imitation further improves sample efficiency. Overall, using all foundation priors yields the best results.FAC with Various Quality of Foundation Priors. As previous ablations show, without value prior knowledge, FAC's sample efficiency decreases, suggesting that a better value prior can further boost performance. We now explore FAC's robustness to the quality of policy and success reward priors. First, we create noisier policy priors by discretizing each action dimension into -1, 0, +1, providing only rough directional information, which we call the discretized policy. We also add uniform noise to the discretized actions at 20% and 50% probabilities. As shown in Fig. 8 (b1), the discretized policy (blue curve) performs similarly to the original (green curve), except for harder tasks like bin-picking and door-unlocking, which take more frames to reach 100% success. Adding noise further reduces performance, but even with 50% noise, FAC still achieves 100% success in many environments. We also tested robustness with systematically wrong policy priors, where actions have a 20% or 50% chance of being inverted (e.g., -1 to 1) (Fig. 8 (b2)). FAC performs well with 20% wrong direction but struggles at 50%, where misleading information is abundant. This shows FAC's robustness to moderate systematic noise.To assess the impact of success-reward prior quality, we distilled a proxy model using 50k offline data from the replay buffer across all 8 tasks. The model, conditioned on task embeddings, has a 1.7% false positive and 9.9% false negative error on the evaluation datasets. We replaced the oracle 0-1 success reward with the predicted reward. As shown in Fig. 8 (c), FAC with the 50k-image proxy model experiences only a limited performance drop compared to the oracle reward and outperforms FAC without success rewards. This demonstrates that FAC performs well even with a noisy successreward prior. In conclusion, our ablation studies show that FAC is resilient to variations in the quality of foundation priors. The higher the quality, the more sample-efficient FAC becomes. 
discussion: 
 In this paper, we introduce a novel framework, termed Reinforcement Learning from Foundation Priors (RLFP), which leverages policy, value, and success-reward prior knowledge for RL. Additionally, we also detail the implementation of this concept within actor-critic methods, introducing the Foundation-guided Actor-Critic (FAC) approach. Extensive experiments on real robots and simulated environments demonstrate the effectiveness of RLFP in efficient autonomous learning.Limitation and Future Work This work has limitations, such as relying on human engineering for designing low-level skills, prompts, and code policy demos. Additionally, we fine-tune diffusion models with in-domain data for better prior knowledge. However, as foundation models improve, future methods may only require simple task instructions without fine-tuning. Future exploration can focus on building more accurate and broadly applicable foundation priors and incorporating richer priors into the RLFP framework. For example, humans can predict future states, and integrating the predictive knowledge from dynamic foundation models could enhance policy learning. horizon of the 5 tasks (Pick Place, Open Door, Water Plants, Unscrew Bottle Cap, Play Golf) to 40, 40, 50, 60, and 25, respectively. We evaluate them every 10 minutes, and the positions of the objects vary in 5cm in Pick Place, Open Door, Watering Plants, and 3cm in Unscrew Bottle Cap and Play Golf. We reset the objects manually because the current foundation model cannot deal with reset problems. The difference in the hyper-parameters between the real robots and the simulation is as follows: The training loss curves of the task bin-picking and door-open for reference, as shown in Fig. 9 .Here we give the task description of the 5 tasks on Franka Arm. For the task Unscrew Bottle Cap, we will fix the bottle when the arm tries to unscrew the cap.• Pick Place: Pick up the purple eggplant and place it onto the blue plate.• Open Door: There is a white cabinet on the table and there is a cucumber in it. The door can only be opened from the outside. Please open the door by a large margin so that the cucumber can be seen.• Watering Plants: There are a blue watering kettle and a potted plant. Please help me watering the plant and keep watering still.• Unscrew Bottle Cap: There are a plastic bottle with a green cap and a pink plate on the table. The bottle will be fixed on the table, so that you cannot lift the bottle. Please help me unscrew the bottle cap and place it on the pink plate. You mush know how to unscrew the bottle cap, anticlockwise or clockwise?• Play Golf: There is a golf hole and a golf ball on the table. The robotic arm is tied by a golf club. Please shoot the golf ball into the set.The ground-truth success reward functions of each task are as follows, which is leveraged in evaluation.• • Play Golf: (golf in the golf hole). ('golf in the golf hole' is the same as the function in pick place).Reward functions for vanilla RL Since it is hard to get the object state in real time, we assume the object state will be attached to the robot state after some actions. For example, we have the initial position of the eggplant, the position, and the size of the target plate. After the gripper reaches the pos of the eggplant and takes action "grasp", the pos of the eggplant will be set the same as the eef pos. The reward function is the normal pick-place reward function. For the door-open, we get the handle initial position, and once the arm is reaching in the bound of 5cm of the handle and closes the gripper, the position of the handle will be attached to the eef of the robot. Therefore, the reward functions are based on the distance between the gripper and some target position sequences, including the position distance and the orientation distance. We give the reward functions of the Door Open and Play Golf as examples, which are harder tasks.Open Door: R approach = max (0, 1 -tanh(10 × p gripper -p handle ∥), encourages the robot to get closer to the handle by diminishing rewards as the distance decreases. R orient = 1 -∥qgripper-qhandle∥ 2, incentivizes aligning the robot's gripper orientation with that of the handle. R pull = max (0, θ door -θ prev_door ), increases with the door's opening angle. R open = min θdoor 90.0 , 1.0 , prioritizes achieving a 90-degree opening. A small time penalty, R time = -0.01, is applied per time step to encourage faster task completion. The total reward is then formulated as: R total = 0.01 * R approach + 0.05* R orient + 0.1 * R pull + 0.1 * R open + R time .Play Golf: We use the golf club's length and the end-effector's (eef) position to determine the clubhead's state. The ball back position is defined as the back of the ball, which is the optimal point to strike. R approach = max (0, 1 -tanh(10 × ∥p clubhead -p ball_back ∥)), encourages the robot to position the clubhead at the back of the ball.EQUATION, which incentivizes proper alignment of the clubhead for an accurate strike. Additionally, the reward for the clubhead's velocity at the moment of impact, R velocity = max (0, 1 -tanh(10 × ∥v clubhead -v desired ∥)), ensures that the clubhead reaches the optimal speed to strike the ball effectively. So the final reward is: R total = 0.1 * R approach + 0.5 * R orient + 0.5 * R velocity + R time . Examples of Success-reward Discrimination Prompts It is significant to feed the success-reward model by high-resolution images, which we choose 512 × 512. Here we give an example of the success-reward prompts in FAC towards the task Watering Plants, by leveraging GPT-4V, as follows: Does the robotic arm water the plants? Attention, if the spout orients over horizontally over the plant, you should output 1 for yes. Otherwise, you should output 0 for no without any space first. Be sure of your answer and explain your reason afterward. Then we can receive the corresponding success-reward as well as the explanations, shown in Fig. 10 .Examples of Code Policy Prompts. Before code generation, we define some primitive skills. We implement the corresponding interface between primitive skills and control systems, so that they can be executed directly by the robot. The primitive low-level skills and the corresponding params are as follows:• move_to x y z: move the gripper to the position (x, y, z).• grasp: grasp with the gripper.• release: release the gripper.• rotate_clockwise: rotate gripper clockwise by 90 degree.• rotate_anticlockwise: rotate gripper anticlockwise by 90 degree.• orient_half_horizontally: change the orientation of the gripper half horizontally to make it grasp more easily.• orient_vertically: change the orientation of the gripper vertically to make ti grasp something vertically.• strike_front: strike something to the front of the arm.• strick_back: strike something to the back of the arm.Firstly, we input prompts that include a task description, the initial scene, and the expected format for the generated code. Notably, the task used in the example differs from the five tasks we evaluate. For the downstream tasks, we rely on the GPT-4V to generate the code policy based on new images and the corresponding task descriptions, shown in Fig. 11 . We also specify the range of object positions within the scene, enabling the VLM to estimate the position of the target object and generate code that includes skill plans and position parameters. Limited to the current ability of GPT-4V, it can build reasonable code policies under small changes in object positions. Then, we feed in the current image and the task descriptions: There are a plastic bottle with a green cap and a pink plate on the table. The bottle will be fixed on the table, so that you cannot lift the bottle. Please help me unscrew the bottle cap and place it on the pink plate. You mush know how to unscrew the bottle cap, anticlockwise or clockwise? Then we can receive the generated code policy as shown in the code text, which can make the correct rotation to unscrew the cap.On real robots, the action space is 7-dim, consisting of the delta position (x, y, z) of the gripper, the delta orientation (roll yaw pitch) of the gripper, and the grasp/release of the gripper. We use the OSC control mode. As for the policy output of the code policy, it generates plans like "move_to x1 y1 z1". Then we use robotics control toolkits to compute the delta position/orientation for the robot, which is the prior action.Once the plan "move_to x1 y1 z1" is finished (close to epsilon in practice), the code policy will execute the next plan. In this way, we can acquire the corresponding prior action in each step. The KL constraint is between the prior action and the RL policy. Consequently, the GPT-4V generates code plans that consist of the primitive skills, and then the primitive skill can generate delta prior actions by robotics control codes. • Takes inputs as s t , s t+1 , with the shape of 3 × 84 × 84.• A Downsample Model, which outputs the representation with the shape of 128 × 2 × 2.• Flatten the planes into 512-dimension vectors.• 1 Linear layer with ReLU, which outputs the 64-dimension vectors.• 1 Linear layer with ReLU, which outputs the 64-dimension vectors.• 1 Linear layer with ReLU, which outputs the action dimension vectors (equal to 4).The Downsample model is designed as follows:• 1 convolution with stride 2 and 128 output planes, output resolution 42 × 42. (ReLU)• 2 residual block with 128 planes.• Average pooling with stride 2 (kernel size is 3), output resolution 21 × 21. (ReLU)• 2 residual block with 128 planes.• Average pooling with stride 3 (kernel size is 5), output resolution 7 × 7. (ReLU)• 2 residual block with 128 planes.• Average pooling with stride 3 (kernel size is 4, no padding), output resolution 2×2. (ReLU) We use the 1M replay buffer trained from vanilla DrQ-v2 for each task and collect them together as the dataset.We use the fine-tuned VLM Seer to collect 100 videos for each task (1000 in bin-picking-v2), and use the trained inverse dynamics model ρ(s t , s t+1 ) to labal pseudo actions for the videos. The example of generated videos is illustrated in Fig. 12 . Then, we do supervised learning to train the policy foundation prior model under the dataset, which is conditioned on the task. For convenience, we encode the task embedding as a onehot vector, which labels the corresponding task. Thus, the size of the task embedding is 8. Here, the architecture of the distilled policy model is as follows, where the downsample model is the same as that in the inverse dynamics model.• Takes inputs as s t , e t , with the shape of 3 × 84 × 84 and 1 × 8.• A Downsample Model, which outputs the representation with the shape of 128 × 2 × 2.• Flatten the planes into 512-dimension vectors.• Concat the 512 vector and the task embedding into 520-dimension vectors.• 1 Linear layer with ReLU, which outputs the 64-dimension vectors.• 1 Linear layer with ReLU, which outputs the 64-dimension vectors.• 1 Linear layer with ReLU, which outputs the action dimension vectors (equal to 4).The training hyper-parameters of the inverse dynamics model ρ(s t , s t+1 ) and the distilled policy model M π (s t , T ) are in Table 3 . The hyper-parameters of training FAC agents are the same as DrQ-v2 [11] . Generally, in both environments, the time complexity of running foundation models is about 2 times more than the vanilla setting in amortization. And the most time-consuming part is the Action prior policy M π . Noticing that the policy prior achieves 0% success rate on some tasks, we collect 100 trajectories then. The results are in Tab. 6. Generally, we find that the learned BC policy cannot outperform the prior policy itself in all the tasks. More significantly, in some tasks, such as windowclosev2, the BC policy achieves much worse results than the prior policy. This is because the prior policy works only in some certain scenarios, which introduces bias in the collected demos. Although the prior policy fails in some scenarios, it can introduce some informative actions, which can be much better than random actions. Therefore, it is more reasonable to leverage the prior actions as guidance for RL. Note that the following derivation omits the variance of Gaussian distribution for convenience. This is because the variance is independent of the state in the deterministic Actor-Critic algorithms DrQ-v2 algorithm.According to Equation 1, the policy π m can be represented as:EQUATIONAdding E τ ∼π ϕm Q(s, a) in Equation 5, we can rewrite it as:EQUATIONConsidering E τ ∼π ϕm Q(s, a) is not related to the optimization objective, the above equation holds.Intuitively, we can observe that there exist two parts in the objective. About the first part, we can use importance sampling to obtain:EQUATIONSince πϕm can be represented as arg max πϕ E τ ∼π ϕ [Q(s, a)] when m approaching to infinity, the minimum of E τ ∼π ϕm Q(s, a) -E τ ∼π Q(s, a) can be achieved when the minimum of the following equation exits. Let us see the second part in Equation 6. π and M π are Gaussian distributions and the variances of distributions are constant in our framework. Thus, KL(π, M π ) ⇐⇒ ∥π -M π ∥ holds.Hereafter, we can reformulate Equation 6 as follows:EQUATIONBased on the Lemma 1 in [72] , the solution to the above problem is derived as:EQUATIONTo this end, the policy π m is the solution to the proposed optimization problem in this paper.Theorem 2 Let D sub = D TV (π opt , M π ) be the bias between the optimal policy and the prior policy, the policy bias D T V (π m , π opt ) in m-th iteration can be bounded as follows:EQUATIONProof 2 Note that the following derivation is most inspired by Theorem 1 in [72] . According to Lemma 1, the policy π m can be represented as 1 1+β πϕm + β 1+β M π . Then, let us define the policy bias as D T V (π m , π opt ), and D sub = D T V (π opt , M π ). Since D T V is a metric that represents the total variational distance, we can use the triangle inequality to obtain:EQUATIONAccording to the mixed policy definition in Equation 10, we can further decompose the term D T V (M π , π m ): D T V (M π , π m ) = supThis holds for all m ∈ N from Equation 12and Equation 13, and we can obtain the lower bound as follows:EQUATIONThe RL policy πϕm can achieve asymptotic convergence to the (locally) optimal policy π opt through the policy gradient algorithm. In this case, we can derive the bias between the mixed policy π m and the optimal policy π opt as follows: Therefore, we obtain the upper bound.D T V (π opt , π m ) = sup 
 
 Appendix:  


Decision: The paper was initially evaluated with mixed but mostly positive reviews. After the rebuttal, the reviewers suggest strongly accepting the paper.

Here is a high level overview of the reviews. (before rebuttal)
### Strengths
- The paper performs a comprehensive study of policy, reward, and value priors in both simulation and real-world settings, demonstrating impressive sample efficiency and strong results. (Reviewer M4P9, Reviewer kNo7)
- The approach is sound and novel, combining the acquired "priors" effectively, and includes sufficient ablations to validate its effectiveness. (Reviewer M4P9)
- The paper shows good results and extensive experiments, evaluated on a range of real-world and simulated tasks. (Reviewer bb5a)

### Weaknesses
- The title is convoluted and not informative enough, and the introduction lacks proper historical context with some inaccuracies. (Reviewer M4P9)
- The priors, especially the policy, heavily rely on human engineering for designing low-level skills and prompts, and the method's claim of being "agnostic to foundation models" needs expansion. (Reviewer M4P9, Reviewer kNo7)
- The RL formulation is missing crucial details such as action space, state space, and discount factor, and the paper lacks generalization experiments beyond the predefined training range. (Reviewer M4P9, Reviewer kNo7)
- The storyline should be changed, as combining common techniques is not enough to call it a general framework, and additional ablations are needed for better validation. (Reviewer bb5a)

Most concerns were addressed in the rebuttal. 
The authors are asked to read the updated reviews and make sure to implement the promised and suggested changes.
As an additional note:
- the font sizes in the graphics are way too small, please update them. (e.g. by using a small fig_size in matplotlib)

Reviews: review: title: title: The paper brings together interesting ideas and show clear results. However, there are missing details, some of them crucial. Moreover,  limitations are not addressed but written as future work.
paper_summary: paper_summary: The paper utilizes "foundation models" to improve the efficiency of reinforcement learning of robotic manipulation tasks. The idea is to leverage these models to get policy priors (to help with exploration), value/reward "priors" (provide dense signals, mostly for "reward shaping") and success "priors" (the final and arguably the main reward signal which is sparse). This way, there is no need to provide human demonstrations and/or to engineer rewards. The effort is shifted to prompting the foundation models (authors imply but do not explicitly state that this is easier) and designing primitive robot actions. The authors test their approach on both real world and simulated manipulation tasks. There is no strong baseline for the robotic task but the results are positive. For simulated tasks, the presented method performs much better than the regular RL algorithms with engineered rewards for the allocated time budget. The authors also ablate the components of their approach.
main_review: main_review: Strengths:
- The main idea and motivations, on a high level, is described clearly.
- The approach is sound and the combination of the acquired "priors" are novel. (with some caveats on the foundation model assumptions, see below)
- Authors perform both real and simulated evaluations, with strong results and sufficient ablations.

The rest will concentrate on the weaknesses and missing information, mostly in the order of reading:

Title: The title is convoluted. This is also my personal opinion but I do not like the part after the colon character (:). This style is overused. In addition, it is not informative; RL already implies learning "on its own". Maybe the authors wanted to imply "no human demonstrations". Lastly, I am not sure if using prior is the best word. The policy prior is an actual prior. However, the value and success priors are not really priors. The former is the value estimate used for reward shaping and the latter is the main success reward signal, and both of them do not change during learning. However, this is not a strong objection.

Introduction:
- There have been robotic applications of RL going back more than three decades. The first paragraph of introduction sort of implies otherwise. 
- Humans also leverage their innate abilities to acquire skills in addition to accumulated knowledge and common sense. 
- Even though the human example demonstrates the idea well, it is not really correct. An example with a baby and toys would have been better. There are interesting papers by Meltzoff et al, mainly involving imitation (there are learning aspects which align with the presented idea) that can be used.
- The claim of "agnostic to foundation models" needs some expansion. The method definitely requires well trained models. The authors find that some of the foundation models are not good enough and fine-tune them. Existence of strong/appropriate foundation models is the main assumption of the method and this needs to be expanded on. 
- The method combines some of the existing ideas but the paper is written as if all the ideas originate from the authors. This should be addressed.

Related Work:
- Most of the relevant papers about foundation models and RL are cited but they are grouped together. A few of the most relevant papers can be expanded upon.
- There are "human demonstration seeded RL" papers that utilize even less amount of training. The authors touch upon this issue by mentioning the use of teleoperation. However, there are also "inverse RL" papers that can be interpreted to acquire value and success "priors" from demonstrations. 
-- Eteke et al., Reward Learning From Very Few Demonstrations 
-- Wu et al., SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks
- Similarly there are "date-efficient" RL methods to learn manipulation tasks but admittedly they are still not as efficient as the presented paper:
-- Popov et al., Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

Method:
- RL formulation is missing. The action space, state space and discount factor are not stated.
- The assumptions and requirements on the foundation models can also be described here. 
- Similar to the above point, the assumption of the performance of the "success-reward priors" is a bit shaky, as the authors do not seem to train a binary classifier but rely on the last image of the task. Note that the ground-truth is used for the simulation. The learned classification model with 50k data points, uses data from a trained policy.
- On a related note, the fact that goal configurations are needed for the value prior (ViP) is not explicitly stated.
- If the authors want to address the "prior" semantics, this is a good section to do it
- The way to acquire such priors is a core part of the paper in my opinion and describing them in the appendix leaves the paper a bit on the air. There is not much space for this but maybe the way it is done can be summarized in this section.
- I am a bit curious about the policy regularization term. The prior policy is shown to be useful in reducing the exploration steps. However, it is not very successful and as such, one would expect using this term in the long run to hinder the performance which was not the case. Is there anything that was done specifically to avoid this (e.g. removing this term after some steps/based on selection frequency, setting the loss weight etc.)
- In equation 1 second line, "y" is not specified (even though it is apparent for people working on RL)

Experiments:
- It looks like there are some hyper-parameter tuning going on for specific tasks. This is not a deal breaker but it is also hidden in the appendix.
- The noisy prior tests mainly use "symmetric" noise. The only systematic noise is for the discretized policy. What would happen if the priors are systematically wrong (e.g. wrong directions, persistent wrong success signals for certain configurations)? Moreover, is there any such evaluations for the value prior (i.e. ViP)? Due to the time and space concerns, these can be left for discussion.

Discussion:
- Limitations are written more like future directions.
- The human and fine-tuning efforts are not addressed in the limitations. 

Other:
- There are grammar and spelling issues throughout the paper. This starts from the title! It should be either "Let Embodied Agents Learn on Their Own" or "Let The Embodied Agent Learn on Its Own". However, I recommend changing the title entirely.


After rebuttal:
- Most of my concerns are addressed and I have updated my scores accordingly. 
- About the title, I am not going to insist on any change but my suggestion would be similar to what the authors propose: "Leveraging Foundation Priors for Sample Efficient and Reward Engineering Free Reinforcement Learning". If authors want to keep their current title, using "the embodied agent" still sounds wrong to my ear since it implies a specific embodied agent. 
- There are some typos both in the original test and the new additions introduced so I strongly recommend the authors to fix them for the camera ready.
- I recommend that the authors add the prior policy loss discussion (weight decrease) that the authors presented in their rebuttal in the paper or the appendix.
questions: questions: Take the review as the main source. This part emphasizes some of the more important aspects:

- Concerns about the tite
- Concerns about the specific contributions of the method
- Addressing the assumptions about the foundation models. On a related note, a discussion on different types of noise for the acquired priors.
- The RL formulation should be clearly stated. The simulation one can be left for the appendix.
- How to acquire the priors are left for the appendix, is it possible to summarize them in the main text? 
- How are the losses combined? What are their respective weights? The loss curves can be given in the appendix to help visualize their contribution to learning in addition to the ablations.
- Collecting the required human effort (e.g. prompt engineering, fine-tuning, providing goal states etc.) in a single place (e.g. discussion) would better serve the paper.
limitations: limitations: 3
review_summary: review_summary: Overall, the results are strong and limitations are sufficiently presented and discussed.
title: title: A Comprehensive Evaluation of Policy, Reward, and Value Priors for RL training
paper_summary: paper_summary: The paper explores the best ways to incorporate foundation models into training a new RL policy. It explores three kinds of priors the foundation model can provide - policy, reward, and value. The paper performs a comprehensive study of these components in simulation and the real world. Utilizing all the components leads to an impressive sample efficiency, with the algorithm learning real-world policies in 1 hour of interaction with a new environment.
main_review: main_review: Recently, many papers have proposed different usages of foundation models for improving RL training, but most focus on a single aspect - reward function, prior policy, etc. This paper tries to incorporate all of them with an ablation study that shows how each component complements the others and leads to even better sample efficiency. The main real-world results are impressive, resulting in the training of a policy from scratch in under an hour. 

The main downside of the work is that the priors, especially the policy, still heavily rely on human engineering in designing the low-level skills and the prompts. From the examples in the appendix, the prompts are highly tailored to the task - "There is a plastic bottle with a green cap and a pink plate on the table," or "If the spout orients horizontally over the plant, you should output 1". In addition, writing these prompts usually requires a few iterations until they work well. Therefore, while it still shows impressive sample efficiency, the method doesn't remove the need for human engineering.
questions: questions: - In section 4.1, you mention, "the positions of objects vary within a predefined range across different trajectories." Can you provide more information on that? In addition, have you tried to vary the position beyond the training range to check for generalization capabilities? In general, the paper lacks generalization experiments.
- For section 4.1, while the wall clock is indeed the correct value for comparison, please also provide the number of real-world trajectories that were collected during this time. It is important to understand the sample efficiency of the algorithm.
- One thing that is missing is the evaluation of the prior success reward. How accurate is it? Can you provide an analysis, either in an environment where you have access to the GT or using a human observer.
- You mentioned that the GPT-4V prior policy generate a trajectory based only on the initial state. If so, how do you take the KL loss between the policy and prior policy actions on non-initial states? You don't have the prior policy actions for these states.
- The idea of keeping a success buffer and adding a BC loss to it doesn't have anything to do with foundation models and can be applied with any reward. However, it is not a common practice in off-policy RL. Why do you believe that it should be used here? Especially when your ablation shows that it helps in only one of the tasks you consider (Bin Picking).
limitations: limitations: 3
review_summary: review_summary: Overall, the paper's contribution and results are solid, and while still limited, they provide an important step in incorporating foundation models into RL training pipelines.
title: title: Review
paper_summary: paper_summary: This paper proposes an algorithm using three foundation models to ease reinforcement learning for real-world tasks.
main_review: main_review: The foundation models used to ease reinforcement learning comprise a policy prior, which uses GPT4 together with a code policy defining a set of useful movements, a value prior derived from VIP trained on general robotic tasks, and a success prior where GPT4 was prompted to judge completion. This approach is evaluated on a range of real-world and simulated tasks. 

### Strengths:
- Good results and extensive experiments. 
   
### Weaknesses:
- The proposed methods are too simple to be called a framework. This work is just reusing and combining prior work. 
- Experiments needs an additional ablation. (see below)
- requires code policy.
    - This makes this algorithm hard to extend to more complex tasks.

### Suggestions:
- Just present it as the FAC algorithm, rather than a framework. There is no theory nor novelty behind it to be a useful framework.
- Table 1: 
    - add DrQ-v2 results (even if they are 0) ; also what rewards have been used there?
    - I think another experiment is require in which DrQ is initialized not from a random policy, but one which is pretrained using BC (it is common to warm-start a policy). I think you can use the action generated from code-policy from GPT4.
    - maybe for the vanilla one, one can check whether success rewards  work better that the handcrafted ones if the policy has a warm start.
- Describe what code policy means in more detail in the main paper and make it clear that a detailed description can be found in the appendix. I found it by luck; people usually don’t spend much time reading the appendix!

### Post-Rebuttal
- my concerns were addressed, which is why I updated my score. 
- while I still not fully agree that the introduction of the framework is actually needed, I do understand their perspective.
- happy to see the additional ablations were done during the rebuttal.
questions: questions: See above.
limitations: limitations: 3
review_summary: review_summary: I think that this paper requires a bit more work regarding the experiments. Also I would change the storyline, just combining a few common techniques is not enough for calling it a general framework. I would just calling it an algorithm. Hence, I am voting for rejection, but I am willing to raise my score if my concerns are addressed.