Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation

Authors: vivek  myers
bill chunyuan zheng
oier  mees
sergey  levine
kuan  fang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:47:44

Abstract: Learned language-conditioned robot policies often struggle to effectively adapt to new real-world tasks even when pre-trained across a diverse set of instructions. We propose a novel approach for few-shot adaptation to unseen tasks that exploits the semantic understanding of task decomposition provided by vision-language models (VLMs). Our method, Policy Adaptation via Language Optimization (PALO), combines a handful of demonstrations of a task with proposed language decompositions sampled from a VLM to quickly enable rapid nonparametric adaptation, avoiding the need for a larger fine-tuning dataset. We evaluate PALO on extensive real-world experiments consisting of challenging unseen, long-horizon robot manipulation tasks. We find that PALO is able of consistently complete long-horizon, multi-tier tasks in the real world, outperforming state of the art pre-trained generalist policies, and methods that have access to the same demonstrations.

Main paper: 
 introduction: 
 Robot learning promises policies that can adapt and generalize to new behaviors. However, in practice, today's robotic policies often struggle to effectively finetune for truly new tasks [1, 2, 3, 4, 5] . For example, consider the task of making a salad: while a person could likely follow a new recipe with only a few examples by remembering the key steps, a robot learning approach may need many more demonstrations to achieve similar performance, and still recover a more brittle policy.A key difference that allows humans to learn tasks so quickly is their semantic understanding of the world. Human have a symbolic representation of the task, such as the names of the ingredients and the steps to prepare them, rather than a series of low-level actions. This representation enables them to understand the task at a higher level, mapping directly into low-level behaviors they are already familiar with [6, 7] . How can we enable robots to quickly learn new tasks through a semantic understanding of the world? EQUATIONEQUATIONmin L bc = kâ ak 2 (7)EQUATIONEQUATIONEQUATIONEQUATION⇡ ( 13) Language provides a potential bridge between these task semantics and low-level control [8] .c ⇤ = arg min ĉ1:K ⇠M min partitions ( • , • ,..., • ) ⇥ J ⇤ References 1 only 5 demos needed! optimize over parameters min L BC R ⇡ (⇡ PALO ; ⇢ target ) R ⇡ (⇡; ⇢ prior ) + 1 H H X i=1 D TV (p target (c i ) , p p D D target ⌧ 1 ⌧ 2 . . . ⌧ N D prior ⇡(â | s, c k ) L LCBC min L bc = kâ ak 2 L (1) bc L (2) bc L (K) bc J = L (1) bc + L (2) bc + . . . + L (K) bc J = L (1) bc + . . . + L (K) bc ⇡ c ⇤ = arg min ĉ1:K ⇠M min partitions ( • , • ,..., • ) ⇥ J ⇤ References 1 need >100 demosRecent advances in large language models (LLMs) and vision-language models (VLMs) have shown promise in understanding and grounding language from a few demonstrations [9, 10] . We propose Policy Adaptation via Language Optimization (PALO), a method for exploiting the semantic understanding of VLMs in combination with a pre-trained robot policy to enable adaptation to new tasks with only a few demonstrations (Fig. 1 ).Past approaches that fine-tune directly to new demonstrations are often overparameterized and sample-inefficient, due to the cost inherent in collecting teleoperated trajectories [11] . Instead, we use a few demonstrations as a calibration set to guide the decomposition of a new language task into a sequence of subtasks that can be used by a language-conditioned policy. Our approach samples possible decompositions of the task from a VLM and chooses the one that minimizes the validation error of the learned policy on the calibration set.The key is that in the few-shot setting, a few demonstrations provide a better signal for adapting to new tasks when used to select the right sequence of language subtasks with the help of a VLM, rather than directly fine-tuning the policy parameters (Fig. 2 ). Unlike prior work, our approach can learn unseen, long-horizon behaviors with fewer than 10 demonstrations across a variety of tabletop manipulation tasks. 
related work: 
 Our approach lies at the intersection of few-shot learning and approaches that leverage language and large pre-trained models for robotics.Few-shot learning. Broadly speaking, few-shot learning approaches utilize diverse data to enable rapid test-time adaptation to a new task from a few examples. These techniques have been applied in various domains, including vision [12, 13, 14] , natural language processing [9, 15] , and reinforcement learning [16, 17] . Frameworks for few-shot learning include optimization-based metalearning [18, 19, 20] , where a model is trained to quickly fine-tune to new tasks, nonparametric methods based on particular modeling assumptions such as metric approaches and Gaussian processes [21, 22, 23, 24] , and in-context learning [9, 25, 26, 27] , where a large model is conditioned on a context to adapt to a new task. Unlike past approaches to few-shot learning in robotics [28, 11] , we show that language can be used to enable nonparametric adaptation without fine-tuning.Language-conditioned robotic control. While early approaches to instruction-following in robotics relied on manually designed symbolic representations [29, 30, 31] , recent work has focused on applying deep learning techniques to understand natural language instructions [32, 33] . These approaches use learned behavioral cloning policies on top of language [2, 34, 35] , connect language representations to grounded representations of the environment [36, 37, 38, 39] , or use the compositional structure of language to decompose tasks and plan [40, 41, 42, 43, 44] . Our approach is the first to enable few-shot adaptation to new demonstrations in robotics by leveraging the structure of language.Foundation models and robotics. Large-scale internet pre-training has seen recent success in the domains of vision and natural language processing [9, 45, 46, 47, 9, 48] . Recent work has investigated if these models can be trained and/or fine-tuned for downstream robotics tasks [32, 49, 50, 11, 51, 52] . Other work has investigated if these models can be used to provide semantic knowledge for downstream robot learning pipelines [53, 54, 55, 56, 57, 58, 59] . Our approach falls into this latter category, but unlike the past works, we perform few-shot adaptation in languageconditioned robot control using the semantic knowledge in large pre-trained VLMs. 
policy adaptation via language optimization: 
 Our goal is to enable a learned language-conditioned robot policy to perform new tasks with only a few demonstrations. The key insight is that the structure of language can be exploited to enable fewshot adaptation to new demonstrations in robotics. Fundamentally, few-shot adaptation to new tasks depends on a policy's ability to generalize its existing knowledge to correctly fit to new demonstrations. One approach for adapting a learned policy is to directly fine-tune to new demonstrations, but in robotics settings where expert data collection is costly, this is often infeasible due to overfitting.We propose Policy Adaptation via Language Optimization (PALO), which instead uses demonstrations of a task that is outside the training distribution with the reasoning capabilities of a pre-trained vision-language model (VLM) to determine the correct sequence of decomposed subtasks that are in-distribution for the robot policy. Given a language instruction ℓ, we compute a task decomposition c 1:K that is both semantically consistent with the instruction (determined by the VLM) and feasible in the environment (measured by policy validation loss on expert demonstrations).Formally, we assume a contextual Markov Decision Process (MDP) structure. We have a state space S, continuous action space A = (0, 1) d A , initial state distribution p 0 , transition probabilities P , and free-form language instruction ℓ ∈ L chosen from the language instruction space L. We use the notation P(X) to denote the set of probability distributions over a space X.The robot selects the action a t ∈ A based on the observed state s t ∈ S at each time step t ∈ {1 . . . H} over a finite horizon H to achieve states in S ℓ . We denote a robot policy as a map π(â t | s t , ℓ), which maps the state s t and instruction ℓ to a distribution over actions ât . For convenience, we assume actions are selected under a fixed isotropic Gaussian noise model unless otherwise specified, and will denote the mode of the distribution π(â | s t , •) as π(s t , •). A robot policy then yields a distribution over trajectories {(s i , a i )} H i=1 , ℓ ∼ T ρ π given a task distribution ρ ∈ P(L).We want to solve out-of-distribution instruction-following tasks involving unseen objects and skills given only a few demonstrations. For (pre-)training the instruction-following policy we assume access to a dataset that has been generated using language tasks sampled from some distribution ρ prior ∈ P(L) with an expert policy π β (â | s, ℓ). For training an instruction-following policy π(s, ℓ), we assume a prior dataset D prior = (τ (i) , c(i) 1:K , ℓ (i) ) Nprior i=1 for τ (i) , ℓ ∼ T ρprior π βand additional hierarchically-decomposed subtask instructions c 1:K ∈ L K that are distributed according to p(c 1:K | s 0 , ℓ) for decomposition size K < H.A target task is sampled from a separate distribution ρ target ∈ P(L) which requires interacting with unseen objects in novel ways, so the policy trained on D prior performs poorly zero-shot. To solve this new task, we assume there exists an additional dataset D target = ({τ 1 . . . τ n }, ℓ) for τ i ∼ T δ ℓ π β | s 0 with s 0 ∼ p 0 and ℓ ∼ ρ target collected by human experts. While a large D target can enable directly training π(s, ℓ) to solve the target task, we are interested in challenging few-shot scenarios in which D target only contains a handful of demonstrations (e.g., 5). In this paper, we tackle this challenge by decomposing the novel target task into a sequence of subtasks that are solvable by the pretrained π(s, c) using a VLM M. Notably, we do not assume any ground truth labels for the task decomposition are given, and aim to generate the optimal language decomposition c 1:K based on the unlabeled demonstration dataset D target collected by human operators.Our approach makes two assumptions about the structure of the target task. Assumption 1. The target task subtask annotations c i locally match those of the prior dataset, i.e., are distributed identically for i ∼ Unif(1 . . . H)EQUATIONAssumption 1 states that even if the overall target tasks in ρ target are unseen, the low-level manipulation skills (e.g., "close the gripper," "move the arm right") will be represented in the policy training.Assumption 2. The VLM M can approximate the distribution of the subtask annotations c 1:K in the target task, i.e., p M (cEQUATIONAssumption 2 states that the VLM can propose candidate task decompositions that are consistent with the instruction ℓ in new scenes. Qualitatively, these assumptions are consistent with recent advances in robot manipulation training data [5, 51] and embodied reasoning with VLMs [60] and are empirically validated in our experiments in Section 4 using the BridgeDataV2 dataset [5] and GPT-4o [46] with prompting described in Appendix F.In Section 3.6 we show that under these assumptions, our PALO algorithm can achieve low regret on out-of-distribution tasks, and discuss how violating these assumptions affects performance.To guide the pre-trained policy π to solve the unseen target task, we decompose the high-level language instruction ℓ of the target task into a sequence of subtask instructions c 1:K = (c 1 , . . . , c K ) for the K subtasks as a set of language decomposition. Instead of commanding π with the original instruction ℓ, we use a combination of ℓ and the subtask instructions c k as the input in each subtask to produce the action as a t ← π(s t , c k ). In our methods, we used GPT-4o [46] as a backbone to generate instruction sets. We denote by M(s 0 , ℓ) the support of possible task decompositions sampled from this VLM (see details in Appendix F).Aside from the sequential order of the subtasks, the robot needs to decide when to switch to the next subtask. For this purpose, we introduce an additional variable u = (u 1 , . . . , u K ) ∼ Unif(U) where U is the space of ordered partitions of {0 . . . H}, so u k denotes the time steps on which the robot is executing the k-th subtask. Notably, we assume the optimal solution to the target task follows a fixed structure, i.e., the same subtask sequence c can be used to solve the task, regardless of the initial state s 0 . Meanwhile, u can be different in each episode, since the number of steps needed to complete each subtask depends on s 0 as well as stochasticity in the environment and the policy.We design a simple sampling-based inference algorithm to find the best c * for guiding the policy π to solve the target task. Since the resulting actions depend on both c and u, as discussed in Section 3.3, we jointly optimize c and u to minimize a cost function J over all trajectories in D target :EQUATION)"put the turnip in the drawer" To measure how well c and u enable the policy π to reproduce each τ , the cost function is defined with the mean squared error between the predicted action ât and the ground truth a t at each time step t. More specifically, we evaluate the policy π on the demonstration trajectory given c and u to compute ât ← π(s t , c min{k:t∈u k } ). Then the cost function is defined as:EQUATIONBy minimizing this cost across demonstrations, we compute a decomposition of the task c that would optimally perform the task by minimizing the loss between the action of the robot and the expert.We use language-conditioned behavior cloning [61] to learn a policy π(s t , ℓ) based on the expert trajectories of D prior . To enable conditioning on fine-grained hierarchical language instructions, we factorize π through c 1:K :EQUATIONfor the subtask index at time t: k t = min{k : t ∈ u k , u 1:K ∼ Unif(U)}. We learn parameters θ for πθ by minimizing the following behavioral cloning objective:EQUATION)The training dataset D prior is an augmented version of BridgeData [5] , a dataset containing a diverse set of manipulation tasks on common household objects. Details about how the subtask instructions are generated are discussed in Appendix D. Each c i is further partitioned into a high-level component c H i and a low-level component c L i . Our full implementation is described in Appendix D.Our theoretical results study the regret of this approach on out-of-distribution tasks in ρ target , showing that it trades off the performance of the pre-trained policy on ρ prior and the performance of the VLM M in accurately modeling the hierarchical language decomposition p(c 1:K ) in ρ target . We define regret with respect to the expert policy π β and a given task distribution in terms of the MSE: EQUATION)c (i) 1:K ∼ M(s 0 , ℓ) 3: for j = 1 to N do 4: u (i,j) 1:K ∼ Unif(U) 5: ĉ1:K ← arg min c 1:K ∈{c (i) } M i=1 min u∈{u (i,j) } N j=1 J (c 1:K , u, τ ) (eq. 4) 6: π PALO (â | s t , ℓ) ← π(â | s t , ĉkt ) 7: return π PALO .Theorem 3.7. The (out-of-distribution) regret of PALO on ρ target can be bounded as:R π β (π PALO ; ρ target ) ≤ R π β (π; ρ prior ) + E D TV p target (c kt ), p prior (c kt ) + 2D KL p(c 1:K ), p M 1/2 + √ M + √ n log(M n) n + 1/M + 1/K + N -2/K (8)where π PALO is from Algorithm 1, π(s t , ℓ) is trained on D prior (Section 3.5), and t ∼ Unif(1 . . . H).The proof is in Appendix A. Theorem 3.7 shows that in the limit as N, M → ∞, we can decompose the out-of-distribution regret of PALO into a sum of the in-distribution regret of the pre-trained policy, and the performance of the VLM in accurately decomposing language tasks:EQUATIONViewing the VLM accuracy and local marginal conformity terms as the extent to which Assumptions 1 and 2 are satisfied, we can see that under these conditions, Theorem 3.7 lets us directly relate the performance of the pre-trained policy π on the training data D prior to the performance of the PALO algorithm on out-of-distribution tasks.We use a ResNet-34 [62] to model the policy π(a | s, c), where c = (c H , c L ) is a concatenation of high-and low-level instructions. The instruction c = (c L , c H ) is passed through a frozen MUSE model [63] to obtain embeddings before being fused into the ResNet with FiLM layers [64] . Architecture details are presented in Appendix C, and the overall algorithm is shown in Algorithm 1. 
experiments: 
 In this section, we show that PALO can better adapt to long-horizon and out-of-distribution tasks from a few expert demonstrations than existing learned language-conditioned manipulation policies (both zero-shot and when finetuned to demonstrations), as well as a nonparametric few-shot adaptation method. Ablation studies also show all components of PALO are necessary.We evaluate on a variety of long-horizon and/or unseen tasks across four scenes from the Bridge tabletop manipulation setup [5] . These involve manipulating new combinations of objects and behaviors unseen in the training data to accomplish long-horizon tasks, such as making a salad or pouring into a bowl. For each task, we collect a set of five expert demonstrations D target for few-shot learning. Besides separating by scenes, we can also separate the tasks into 4 long-horizon tasks (put in, salad, sweep mints, sweep skittles) and 4 unseen-skills tasks (pry away, pour spoon, rotate marker, rotate spoon). Experimental details and example rollouts are presented in Appendix B.We compare against the following baselines trained on BridgeData: Octo [11] : A general transformer-based robot manipulation policy with diffusion action head. GRIF [36] : A language-conditioned robot control method that uses pre-trained CLIP [45] representations to connect language instructions to goals for the policy to reach. RT-2-X [32] : A language-conditioned robot control model with 55B parameters that transfers knowledge from internet-scale pre-training to manipulation zero-shot. LCBC [61] : Language imitation with a ResNet and pretrained MUSE [63] Across eight different tasks, our PALO method yielded a success rate of 71.3%, while the best zero-shot policies only resulted in a success rate of 26.3%. While most of the zero-shot methods degrade when the task became increasingly more out-of-distribution for the pretrained policy (for example, tasks in the "salad" scene achieved a 30% overall performance across the 4 baseline models while pouring from scoop only achieved 12% performance across the models), our method remained effective, with all 8 tasks performing at a success rate of 50% or better.The FT-Octo and FT-LCBC baselines allow us to compare the nonparametric adaptation of PALO to conventional parametric finetuning. While Octo trained only on BridgeData achieved moderate zero-shot success, finetuning on only five demonstrations overfit and worsened performance. The FT-LCBC baseline did benefit from finetuning, but still failed to ever exceed 30% success rate across all tasks. We observe that the small size of trajectories made these datasets an unfavorable candidate for finetuning, as any variance brought by the human controller may be amplified and cause unfavorable movements during evaluation. The nonparametric VINN baseline performed well on the rotation tasks (45% success rate), but failed to achieve greater than 5% success rate on the other tasks.We ablate the following components of our method in Fig. 6 While the sweeping and rotation scenes gave comparable performance with masked high level instructions (No c H ), the performance deteriorated in Drawer and Bowl, which involved more unfamiliar items for the pretrained policy. The remaining ablations (No c L , Fixed Times, Zero-Shot Decomposition, No VLM) decreased performance across all scenes. These approaches are discussed in more depth in Appendix E. Overall results in Fig. 6 show that all components of the PALO method are needed.We study the scaling of our nonparametric method and a parametric finetuning approach with > 5 demonstrations of the skittle sweeping task in Fig. 7 . We observe that while Policy Adaptation via Language Optimization achieves the best performance (80%) using any number of demonstrations, the Octo finetuning baseline needs at least 80 expert demonstrations to achieve comparable performance, while LCBC needs at least 120 demonstrations.We show successful task executions in Figs. 3 and 5 . While the full method is robust to logically unsound instructions generated by the VLM, failures in reasoning and execution occur when we ablate our methods. Fig. 12 and Fig. 13 are two examples in which reasoning break down in ablations. See Appendix G for details. 
discussion: 
 We introduced PALO, an approach for few-shot adaptation to unseen tasks that exploits the semantic understanding of task decomposition provided by vision-language models. In extensive real world experiments, we find that PALO is able to use language to adapt to unseen long-horizon robot manipulation tasks across a wide range of tabletop setups.Limitations and Future Work. We assume the dataset has a consistent format of high-level language labels and proprioception, making it more challenging to generalize our low-level heuristic generation on drastically different embodiments. The discrete optimization over subtask time steps may also scale poorly with the number of subtasks and time steps. Future work could explore more efficient optimization methods for this problem.Taking c PALO to be the output of the PALO algorithm, we can relate the true regret of PALO on the current task (left) to its empirical regret (right):EQUATION)Since regret is bounded by 1, we can convert to an expectation:E Dtarget R π β (π | s 0 , ℓ, c PALO ) ≤ E Dtarget R EMP (c PALO ) + log M -log ε 2n + ε.Lemma A.2. Suppose u, u ′ ∼ Unif(U) are i.i.d. samples from a uniform distribution over the ordered K-partitions U of {1 . . . H}. For any ε ∈ [0, 1/K], we haveEQUATION)Since Algorithm 1 (line 4) only samples N values for u instead of the full space for the min in Eq. ( 10), we must separately consider the degree of suboptimality in the decomposition c PALO relative to the optimal c * = arg min c∈M(s0,ℓ) R EMP (c) that results from our approach to determine the effect of N on the final bound. Applying Lemma A.2, we can say:EQUATION)So, we have related the true regret of PALO on the current task (left) to its empirical regret in the limit of infinite samples (right). All that remains is to compute the empirical regret, for which we make use of the following lemmas.Lemma A.4. Denote the true (unobserved) target decomposition as c 1:K . We can relate the empirical regret of the optimal PALO solution c * to the empirical regret of the true decomposition.E Dtarget R EMP (c * ) ≤ E Dtarget R EMP (c 1:K ) + D TV p(c 1:K ), p M (c 1:K ) + 1/M Lemma A.5.The empirical regret of π can be bounded for i ∼ Unif(1 . . . K) asE Dtarget R EMP (c 1:K ) ≤ R π β (π; ρ prior ) + E D TV p target (c kt ), p prior (c kt ) .Applying Lemma A.4 and Lemma A.5 to Eq. ( 17) yields a bound of the correct form.E Dtarget R EMP (c * ) ≤ E Dtarget R EMP (c 1:K ) + D TV p(c 1:K ), p M + 1/M ≤ E Dprior R π β (π; ρ prior ) + E D TV p target (c kt ), p prior (c kt ) + D TV p(c 1:K ), p M + 1/M.To make the D TV p(c 1:K ), p M term more interpretable as a VLM accuracy, we convert to a KL divergence with Pinsker's inequality [69] : 19) into Eq. ( 17) gives the desired result:EQUATIONEQUATION) Since E Dtarget R π β (π | s 0 , ℓ, c PALO ) = R π β (π PALO ; ρ target ) , plugging Eq. (EQUATION)Proof of Lemma A.2. Define {X i } H i=1 to be the unique index k such that i ∈ u k , andEQUATIONNow, we observeEQUATIONEq. ( 22) is concave in p Xi , and so is maximized when for any δp Xi and some λ,λδp Xi (k) = -2 K k=1 p Xi (k)δp Xi (k),i.e., when p Xi (k) = const. = 1/K for all k. Thus, we haveEQUATION)Taking the complement of Eq. ( 24) yields the desired result:EQUATION)Proof of Lemma A.3. The statement follows from the ansatzε = 1 K - log N N HK Plugging in, ε + e -2H( 1 K -ε) 2 = N -2/K + 1 K - log N HKN 1/2 ≤ 1/K + N -2/K .Proof of Lemma A.4. Recall the definition of the optimal PALO solutionEQUATIONNow, noting regrets are bounded by 1 from Eq. ( 7), we haveE Dtarget R EMP (c * ) = E Dtarget min c∈M(s0,ℓ) R EMP (c) = E Dtarget p(c) p M (c) min {c (i) } M i=1 ∼pc 1:K R EMP (c (i) ) = E Dtarget min {c (i) } M i=1 ∼pc 1:K R EMP (c (i) ) + E Dtarget p(c) p M (c) -1 min {c (i) } M i=1 ∼pc 1:K R EMP (c (i) ) ≤ E Dtarget min {c (i) } M i=1 ∼pc 1:K R EMP (c (i) ) + E Dtarget p(c) p M (c) -1 ≤ E Dtarget min {c (i) } M i=1 ∼pc 1:K R EMP (c (i) ) + D TV p(c 1:K ), p M (c 1:K ) = E Dtarget Pr R EMP (c 1:K ) < c (i) for {c (i) } M i=1 ∼ p c 1:K + R EMP (c 1:K ) + D TV p(c 1:K ), p M (c 1:K ) = E Dtarget R EMP (c 1:K ) + D TV p(c 1:K ), p M (c 1:K ) + 1/M.Proof of Lemma A.5. We consider the empirical regret of π using the true decomposition u 1:K , c 1:K ∼ p target , for t ∼ Unif(1 . . . H) and k t defined as in Eq. ( 5):E Dtarget R EMP (c 1:K ) = E Dtarget 1 H √ d A τ ∈Dtarget min u 1:K J (c 1:K , u 1:K , τ ) = E Dtarget 1 H √ d A τ ∈Dtarget min u 1:K K n=1 t∈un a t -π(s t , c n ) 2 ≤ E un,cn∼Dtarget 1 H √ d A τ ∈Dtarget K n=1 t∈un a t -π(s t , c n ) 2 ≤ 1 H √ d A E ptarget K n=1 t∈un a t -π(s t , c n ) 2 + D TV p target (c n ), p prior (c n ) = E un,cn∼pprior 1 H √ d A K n=1 t∈un a t -π(s t , c n ) 2 + E D TV p target (c kt ), p prior (c kt ) = R π β (π; ρ prior ) + E D TV p target (c kt ), p prior (c kt ) .We evaluate our method in a real-world tabletop manipulation setup. We use a 6DOF WidowX-250 robot interacting with various objects both inside and outside of our training distribution at 5 Hz. We use one 640×480 RGB camera mounted on top of the model as set up in BridgeData [5] . When computing observations we downsample images to 224 × 224."put the marker into the box while aligning it""put the spoon into the cleaner while aligning it" "put the beet toy/purple thing into the drawer" "pry out the pot in the drawer using the ladle" "make a salad bowl with corn and mushroom""pour the contents of the scoop into the bowl" "sweep the mints to the right after putting the mushroom in the bowl""sweep the skittles into the bin after putting the mushroom in the container" We evaluate our method in the following scenes, which include:Instruction t = 0 t = 0 t = 180 t = 180Sweep: This scene involves an object manipulation as well as sweeping task unseen in the Bridge-Data's initial training trajectories. mint: Placing the mushroom in the pot, then sweep the mints on the right using the towel. skittles: Instead of using mints and towel for sweeping, we use a swiffer and skittles instead. Drawer: This scene involves using a drawer and perform manipulation within the space of the drawer. put in: Open the drawer, and then put a purple object (beet/sweet potato) inside the drawer. pry away: A pot is stored inside the drawer space, and the robot must use a ladle to pry away the pot within drawer. Bowl: This scene involves object manipulation to a bowl and perform long-horizon or 6DOF manipulation. salad: This task requires sequential object manipulation by putting a corn cob and a mushroom in the bowl. pouring: This task requires the robot to grasp a scoop and pour almonds inside the scoop into the bowl. Rotation: This scene involves rotating a spoon and a marker to fit into a white container not aligned with the pen/marker, and naive pick-and-place will not correctly align the object into the container. spoon: Placing the spoon in the container placed on the left side of the table. marker: Replacing the spoon with the marker and randomize location of the container while being misaligned.We summarize the evaluation tasks in Table 1 , and show example rollouts in Fig. 8 .We train on an augmented version of the BridgeDataV2 dataset [5] , which features over 50k trajectories with 72k language annotations. We algorithmically augment the dataset with low-level instructions using heuristics designed over the proprioceptive states of the robot and incorporate language context by parsing the language instruction using a language model. We use the Adam optimizer [71] to minimize the loss function in Eq. (27) .Instead of naively looping through Algorithm 1, we batch our implementation with the exception of the outermost for loop, thus reducing time consumption during optimization by a significant margin via vectorization. We record an empirical time consumption of 470 seconds for our language optimization module on computations ran on a V4 TPU module, in which only 200 seconds are required for sampling 20000 different partitions to complete the optimization for all of the 15 sets of language instructions. We save our optimal plans for future use, thus reducing overhead even more.We encode both language instructions using a frozen MUSE model [63] before passing them into the main ResNet with FiLM layers [64] .where s 1 . . . s H ∈ S, a 1 . . . a H ∈ A, c L , c H ∈ L ∪ {0}, and θ are the parameters of the policy network, 0 is an additional point representing the absence of a high-or low-level instruction, which will be represented as an embedding vector of zero during training, and τ = (s 0 , a 0 , . . . , s H , a H ) is a trajectory sampled from the dataset.This objective encourages the policy to learn to follow instructions at both levels of abstraction, marginalizing over missing instructions. We chunk actions within training data into segments of length 4 and evaluate the low level instruction within these segments and append them into the training data.We algorithmically enhance our training data by using heuristics generated by the proprioception of the robot and language context, which generates the low-level instructions. The labeled language instruction is passed into a language model to obtain manipulation keywords, and we combine the keywords with the proprioceptive information within that time span including translation, rotation, and gripper movement into coherent language commands.Proprioception. We use standard deviation of each action against the metadata of BridgeData [5] and determine how to describe the proprioception of the label. We determine the largest direction in which the gripper is moving (up, down, left, right, forward, backward) and the orientation it is rotating (up, down, left, right, clockwise, counterclockwise), and determine whether the movement is unambiguous enough by checking the largest z-score in translation and rotation. We then combine the movement as well as the keywords extracted to form language primitive commands.Target Object. We identify the target object using a prompt heuristic to be fed into GPT3.5-Turbo [9] by taking advantage of the fact that BridgeData consists of mainly object manipulation data. We extract two keywords: the object to be manipulated and the destination of the object, based on the fact that much of BridgeData is focused on object manipulation. The precise prompt can be found at Appendix F Data Filtering. We filter low-level instruction on two occasions: when the movement itself is ambiguous and when the language model gives inconsistent results. We check the former by looking up the norm of the translation and the norm of rotation, and we check the latter by using regular expression to see if the result was against the desired format and manually filtering out some common keywords of inadmissible GPT query. On the former occasion, we use an empty string as the low level instruction, and on the second occasion, we use only proprioceptive information for low-level instruction.We additionally augment the high-level language annotations by generating context-free rephrasings with GPT-3.5 [9] . For each trajectory with crowdsourced language annotations in the BridgeData v2 dataset, we generate 5 such augmented language strings following the approach of Myers et al. [36] .We ablate our experiment in progressive manners, going from full implementation to using only the barebone hierarchical policy network.• PALO w/o high level instruction: while running PALO, we derive both high and low level instruction sets. However, during inference on robot, we mask out the high level instruction and feed in zero embeddings. • PALO w/o low level instruction: mask out the low level instruction and replace them with zero embeddings during inference.• Fixed Time During Optimization: for each trajectory that has corresponding length H 1 , H 2 , . . . , H i , we choose fixed u i = [ Hi k , 2Hi k , . . . , (k-1)Hi k ] during optimization. We implement no u sampling, which reduce PALO into an arg max operation.• Zero-Shot Plan Generation: instead of sampling 15 plans, we sample only one plan from VLM and examine the behavior of the robot using that specific plan. • No VLM Guidance: We use only ℓ as our high level instruction, and mask out low level instruction with zero embeddings during inference.We employ a keyword decomposition prompt in our augmentation method and a planning prompt to generate VLM outputs. They are listed below:User : " You are presented with a text for high level instruction for a robot , and you need to extract keywords in the task description text . In this instruction , the first keyword is the object being moved , and the second keyword , if applicable , what is the moving taking this to ( either another object or a location ) within the instruction . Only return the first and second keyword , and they should be separated by a comma . If the instruction is in another language , write your response in English . For example , if the text instruction says " Pick up the silver lid on the left to the middle of two burners " , return " silver lid , middle of two burners ". Or if the instruction says : " Move the object to the top middle side of the table ." , your response should be " object , top middle side of the table ". Or if the instruction says : " Move the red greenish thing on the towel to the right ." , return " red greendish thing on the towel , the right ". Try your best to find the two key phrases , but if you can ' t find the second keyword within the instruction sentence , write " N / A ". For example , if the instruction is " Move the pot lid ." , the response should be " pot lid , N / A ". There might be some other description regarding confidence at the end , you are safe to ignore it .\ n The specific task description for you to analyze is : \ n { instruction } \ n "User : Here is an image observed by the robot in a tabletop robot manipulation environment . The gripper situated at the top of the center of table and perpendicular to it . Now plan for the list of subtasks and skills the robot needs to perform in order to { instrs }.Each step in the plan can be selected from the available skills below :You may choose between using one of movement direction . rotation direction , or gripper movement . If you were to choose to use movement direction , you may use one or two directions and include a target object , and you should format it like this : " move the gripper x towards z " or " move the gripper x and y towards z " where x and y are the directions and z is the target object . You also must start your command with " move the gripper ".Therefore , instead of saying something like " down " or " up " , you should phrase it like " move the gripper down " and " move the gripper up ". Make sure to include at least one direction in your command since otherwise this command format won ' t make sense .If you were to choose to use gripper movement , you should format the command as " close the gripper to pick up x " or " open the gripper to release x " , where x is the target object . You may discard the target object if necessary . In that case use " close the gripper " or " open the gripper ". If you think the gripper is close to the target object , then you must choose to use gripper movement to grasp the target object to maintain efficiency .If you were to choose gripper rotation , you should format the command as " rotate the gripper x " , where x is the target rotation direction . You need to make sure that in pouring tasks , the opening of the container is aligned with the pot . For example , if the object is aligned vertically but you want it to align it horizontally , then you should call " rotate the gripper counterclockwise ". If you want to tilt the object in the gripper to pour it , you should call " rotate the gripper left "Pay close attention to these factors : * Which task are you doing . * Whether the gripper is closed . * Whether the gripper is holding the target object . * How far the two target objects are . If they are across the table , then duplicate the commands with a copy of it . * Where the gripper is . After the end of each subtask , it is reasonable to assume that the gripper will not be at where it originally was in the image , but somewhere close to the last target object .Especially pay attention to the actual direction between the gripper and the target object . Remember that the robot ' s angle is roughly the same as the camera ' s angle . To determine whether the gripper should move forward or backward , look into the edge of the table . If the target object is closer to the edge of the table that is near the top of the image , you should move forward , and if it is closer to the edge that is near the bottom of the image , you should move backward . At the end of each subtask , you need to use the skill " move the gripper back to neutral . This will move the gripper back to the original position of the image after completing the task .Start by looking at what objects are in the image , and then plan with the direction of the objects in mind . The tasks should be completed sequentially , therefore you need to consider the position of the gripper after each task before planning the next task . You should return a json dictionary with the following fields : -subtask : this should be the key of the dictionary . It should contain the only the verbal description of the subtask the robot needs to perform sequentially in order to finish the task , and they should be ordered in the same way the task is completed . -list of skills : this should be the value of the dictionary . It should be a list of skills the robot needs to perform in order to finish the corresponding subtask . Be concise , and do not return any other comments other than the dictionary mentioned above . Do not put " subtask : " or " lsit of skills : " in the key and value of the dictionary you generate . Remember only the description and list should be returned .In this section, we provide additional qualitative results for PALO.During inference, we chunk each low-level instruction into length 8 intervals, switching to the new set of low-level (and high-level, if applicable) after these 8 steps. We chose a fixed interval instead of a dynamically allocated one due to the policy choosing to mostly stay put after finishing the action prescribed by the low-level instruction.We show the full breakdowns of success cases here. Fig. 9 and Fig. 10 gives detailed description of the robot's action primitives generated by PALO during inference.While PALO is robust in generating language primitives that help achieve the task, it does not guarantee a successful execution of the policy as shown in Fig. 11 . PALO can fail when the underlying policy fails to execute a low-level motion, after which the robot may not be able to recover and complete the task.When we ablate the components of PALO, we begin to see more critical failures. Fig. 14 demonstrates a case of grounding failure when c H is masked out, i.e., when PALO loses half of the optimized task decomposition. 
: 
 (2)EQUATION⇡(â | s, c k ) (5) L LCBC (6) min L bc = kâ ak 2 (7)EQUATION(2) bc (2)EQUATION⇡(â | s, c k ) (5) L LCBC (6) min L bc = kâ ak 2L(2) bc  
 
 Appendix:  


Decision: Strengths:

- Interesting approach that uses GPT4o for decomposing tasks into subtasks and uses it to do few-shot adaptation 
- Well-written paper with lots of experiments, ablations and comparisons that highlight the strengths of the work

Weaknesses:
- Some details about the demonstration dataset and pre-training are missing as pointed out by the reviewers. 
- More insights and analysis of failures seen during experiments would make the paper stronger. Moreover, please provide multiple runs of experiments, along with some statistical analysis of the performance.

Post-rebuttal discussion:
In the rebuttal phase, authors provided additional details missing in the original submission, as well as baselines and ablations, including multiple real-world runs of the baselines.

Reviews: review: title: title: Review for Submission698
paper_summary: paper_summary: This paper presents PALO, a method for few-shot adaptation of language-conditioned robot policies to unseen tasks. The key idea is to leverage a VLM to generate candidate task decompositions from a sequence of frames, and then optimize over these decompositions using a small set of demonstrations. This allows the method to adapt a pre-trained policy to new tasks without fine-tuning, by selecting an appropriate sequence of subtasks that the policy can execute.
main_review: main_review: # 

### Clarity

The paper is well motivated, but some crucial details are missing or unclear:

- The exact procedure for optimizing over task decompositions is not fully explained.
- The distinction between high-level and low-level instructions in the method could be elaborated on.
- Section 4.4: The ablations performed and the information gained from each ablation is hard to follow. A clearer presentation of the ablation studies and their implications would strengthen the paper.

### Originality and Significance

The concept of using a sampling-based method for few-shot adaptation in robotics is interesting and potentially significant. While sampling methods have been shown to perform well in limited data settings in previous work, applying this approach to language-guided robotics tasks is novel.

### Strengths

1. Presents a simple, general recipe for few-shot adaptation in robotic tasks.
2. Demonstrates the potential of leveraging large language models for robot task planning.

### Weaknesses

1. Methodology: The details of the sampling method used are not clearly explained. More information on how the optimal task decomposition is selected would be beneficial.
2. Experimental details: Crucial information such as the number of demonstrations used in experiments and number of evaluations done per task is missing. This makes it difficult to fully assess the method's few-shot capabilities.
3. Baselines: The comparison is primarily against parametric methods. Including a comparison to other non-parametric methods for sampling would provide a more comprehensive evaluation, especially given that non-parametric methods have been shown to perform well in limited data settings.[1]
4. Analysis of failures: The paper would benefit from a more in-depth analysis of failure cases. For tasks such as "pry away" where the method struggles, it would be valuable to understand if these failures are due to limitations in the language embeddings generated, the image encoder used, or other factors.
5. Non parametric methods, while performant in regimes with limited data, may not scale to a large number of tasks. It would be useful to add this to the limitations section.

[1] Pari, J., Muhammad, N., Pandian, S. and Pinto, L., The Surprising Effectiveness of Representation Learning for Visual Imitation.
questions: questions: 1. How many demonstrations were typically used in your few-shot experiments?
2. How many evaluations were run per task?
3. Can you provide more details on the optimization procedure used to select the best task decomposition? (My current understanding is that it is based on randomly sampling combinations of subtasks and finding action sequences with the lowest MSE)
4. Have you considered comparing PALO to other non-parametric methods for few-shot learning in robotics? Including comparisons with non-parametric baselines to could better contextualize the method's performance in few-shot settings.
5. Do you have any insights into what causes failures in tasks such as "pry away"? Could these be attributed to limitations in the language embeddings generated or the image encoder used?
limitations: limitations: 1
review_summary: review_summary: This paper presents an interesting approach to few-shot adaptation in language conditioned robot manipulation. Making appropriate revisions to address concerns around methodological details and experimental details, could make strengthen this paper.
title: title: Interesting method, needs more analysis and experiments to validate the claims
paper_summary: paper_summary: Learning long-horizon tasks from few demonstrations by decomposing the task into subtasks using GPT4o and then learning a high and low-level conditioned policy
main_review: main_review: ### Strengths: 
1. Interesting method for learning long-horizon tasks from few demonstrations by decomposing the task into subtasks using GPT4o and then learning a high and low-level conditioned policy.
2. Pretraining objective is interesting, as is conditions on both low and high level instructions, while also adapting to missing instructions in pretraining data. 
3. Several baselines are included in the comparison.
4. Intuitive ablations are also performed, giving insight into the method.
5. Qualitative experiments are also shown along with failure cases, and limitation are discussed.
6. The supplementary video is good and improves understanding of the method.

### Weaknesses:
1. All experiments are missing std deviations, to ensure that the runs are not cherry-picked, several runs should be done for every expt.
2. Since this method is more of an algorithm rather than an architecture, it would much more general to show that it works with more architectures as well, like Octo, RT2X, R3M, MVP, etc.
questions: questions: 1. How exactly is ‘u’ sequence initialized? Is it annotated by the human (sec 3.2, 3.3)?
2. Is it useful to condition the policy on s0? Why or why not?
3. How do you get c_L for the D_prior?
4. Please also include how D_prior is different from D_target, via both qualitative examples and visualizing in encoding space with methods like t-SNE. It is unclear how D_target is OoD. Also, in what sense is it unseen - is it an unseen object, scene, or skill? A plot visualizing and summarizing this information should be added.
5. Pretraining data stats are missing.
limitations: limitations: 3
review_summary: review_summary: I think if the authors add required details pointed above and add the missing experiments, then the paper will be more complete.
title: title: PALO provides an intuitive method for language decomposition to aid few-shot imitation of language conditioned policies that has been explained well, thoroughly evaluated through several experiments and shows interesting results.
paper_summary: paper_summary: The paper proposes PALO, a method that performs few-shot imitation by leveraging a language conditioned policy pre-trained on BridgeData and task-specific demonstrations to find a sequence of high + low level commands that make the pre-trained policy succeed in a task for which a human provides a few demonstrations. To achieve this, given the task specific demonstrations and language instruction, PALO uses an LLM to decompose the language instruction into lower level commands. Then, by optimising over the different possible low level commands and the timing to switch between them, PALO selects the sequence of high+low level commands that minimises the validation loss of the pre-trained policy on the provided demonstrations. This sequence of language commands is then selected and the task is performed. The authors find that prompting the policy with a good sequence of low level commands outperforms other alternatives, such as fine-tuning on the demos and through several experiments and ablations the paper shows that PALO outperforms several baselines.
main_review: main_review: Strengths

- PALO is a well written paper that is easy to follow and understand. The provided appendix provides sufficient additional details to understand the method’s implementation details.
- It addresses an important problem: that is how can we find a good decomposition of language descriptions for a task such that a pretrained policy can succeed on a previously unseen setting which will be of interest to the robot learning community
- The method is simple and intuitive making it appealing. In particular the fact that with appropriate prompting GPT4-o can provide low level language commands that focus so well on instructing the policy about what low level actions to take appears to be very powerful; especially, given that doing so appears to outperform simply fine-tuning on new demonstrations.
- The experiments are well executed and thorough. A number of good baselines were selected to compare PALO against and the performed ablations  confirm the authors’ design choices for PALO.

Weaknesses 

- See questions for rebuttal
questions: questions: - Can you elaborate on how you created the training dataset? How did you select the right language decompositions of the tasks present such that the low level language commands are correct?
- Could you provide an experiment that studies the performance of PALO vs fine-tuning under different numbers of demonstrations ? Fine tuning OCTO for example on a very large number of demonstrations could result in better performance than PALO, it would be interesting to see when that happens in terms of task specific demonstrations.
- It would be interesting to see how would the pretrained policy perform if a random language decomposition is selected and used to fine-tune the network used for PALO on the few provided demonstrations, instead of fine-tuning Octo.
- If a failure occurs during task execution, would PALO be able to recover given that the low level instructions are fed to the policy sequentially without checking whether the low level action was successfully performed?
limitations: limitations: 3
review_summary: review_summary: My recommendation relies on the fact that overall the paper presents a novel method on how to decompose a task into lower level instructions to aid few shot policy learning that will be of interest to the community. The method explanation is simple and straightforward and experimental evaluations along with their ablations are convincing.