Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments (2024)

Abstract: In embodied instruction-following (EIF), the integration of pretrained language models (LMs) as task planners emerges as a significant branch, where tasks are planned at the skill level by prompting LMs with pretrained skills and user instructions. However, grounding these pretrained skills in different domains remains challenging due to their intricate entanglement with the domain-specific knowledge. To address this challenge, we present a semantic skill grounding (SemGro) framework that leverages the hierarchical nature of semantic skills. SemGro recognizes the broad spectrum of these skills, ranging from short-horizon low-semantic skills that are universally applicable across domains to long-horizon rich-semantic skills that are highly specialized and tailored for particular domains. The framework employs an iterative skill decomposition approach, starting from the higher levels of semantic skill hierarchy and then moving downwards, so as to ground each planned skill to an executable level within the target domain. To do so, we use the reasoning capabilities of LMs for composing and decomposing semantic skills, as well as their multimodal extension for assessing the skill feasibility in the target domain. Our experiments in the VirtualHome benchmark show the efficacy of SemGro in 300 cross-domain EIF scenarios.

heading: 1 Introduction
text: In the area of embodied instruction-following (EIF), the quest to develop systems capable of understanding user instructions and achieving tasks in a physical environment has been a pivotal aspiration for general artificial intelligence. Recent advancements in language models (LMs) have demonstrated their extensive and impressive application in EIF, where LMs are employed as task planners, drawing on their commonsense knowledge of the
*Equally contributed to this work †Honguk Woo is the corresponding author
physical world (Yang et al., 2023; Padmakumar et al., 2023; Pantazopoulos et al., 2023; Yun et al., 2023; Logeswaran et al., 2022).
The process of LM-based task planning in EIF unfolds via a two-stage pipeline: (i) Interpretable skills (i.e., semantic skills) are acquired, and (ii) given a user instruction, the LM is prompted with the instruction and the skill semantics to identify appropriate skills to perform. To ensure that such planned skill can be executed within the domain context the agent is placed in, several works have focused on incorporating environmental observations into task planning (Singh et al., 2023; Song et al., 2023; Xiang et al., 2023; Lin et al., 2023b,a; Shinn et al., 2023; Carta et al., 2023). Yet, EIF in cross-domain environments has not been fully explored. In cross-domain settings, the complexity goes beyond logical task planning by LMs; it necessitates the adaptation of pretrained skills to align with the specific characteristics and limitations of the target physical environment. This adaptation challenge stems from the inherent domain specificity of semantic skill acquisition, calling for a strategy that disentangles these learned behaviors from their original domain contexts and reapplies them in new domains.
In response to this challenge, we present a semantic skill grounding framework SemGro. We observe a trade-off between semantic depth and domain-agnostic executability within the skill hierarchy. High-level skills, such as “prepare a meal”, embody a richer semantic content and long-horizon behavior compared to low-level skills, such as “grab a cup”. However, these high-level skills tend to be effective only within specific domains, as their execution heavily depends on a multitude of environmental factors, unlike their low-level counterparts which can be universally applicable across domains. SemGro capitalizes on this insight by employing an iterative skill grounding scheme, designed to progressively decompose pretrained skills
3354
to fit in the cross-domain EIF environment as well as to fully utilize the reasoning strengths of LMs.
To be specific, the iterative mechanism of SemGro involves two procedures: (i) task planning with skills that align with a given instruction (in Section 3.2) and (ii) assessing the executability of planned skills (in Section 3.3). In SemGro, a task planner employs in-context learning of LMs to interpret a given instruction and generate a series of semantic skills for the instruction through iterative skill grounding and decomposition. Concurrently, a skill critic employs the multi-modal extension of LMs to evaluate the applicability and execution potential of these skills within target domains. When the critic deems a planned skill as unsuitable for execution, the planner responds by generating an alternative, detailed instruction. This instruction is intended to be carried out through a sequence of lower-level skills that align with the previously planned skill. Through the collaborative interaction between the task planner and the skill critic, SemGro navigates the semantic hierarchy of skills to establish the best trade-off between the logical relevance of skills to a given instruction and the skill feasibility in cross-domain environments.
Using the VirtualHome benchmark (Puig et al., 2018) to evaluate EIF agents (Singh et al., 2023; Hazra et al., 2023; Chen et al., 2023; Xiang et al., 2023), we demonstrate the advantages of SemGro for cross-domain EIF scenarios. For instance, SemGro outperforms the state-of-the-art task planning framework LLM-Planner (Song et al., 2023) by 23.97% in task completion performance.
Our contributions are three-fold: (i) We present a novel framework SemGro to address the issue of cross-domain skill grounding in EIF. (ii) We devise an iterative skill grounding algorithm, leveraging the reasoning capabilities of LMs and their multimodal extension. (iii) We conduct an intensive evaluation of SemGro in 300 cross-domain scenarios of the VirtualHome benchmark, complemented by ablation studies to explore the efficacy of SemGro.

heading: 2 Preliminaries

heading: 2.1 Language Models for Task Planning
text: For complex embodied tasks, prior works have leveraged the notion of semantic skills, which represent interpretable patterns of expert behavior. These skills have a wide spectrum, ranging from short-horizon basic low-level skills (e.g., “stir the drink”) to long-horizon high-level skills
Training Environment
(e.g., “prepare a meal”). These diverse skills Πl = {πl | l : skill semantic} can be efficiently established within a training environment, through deterministic algorithms, reinforcement learning (RL), or imitation learning. The integration of semantic skills and the reasoning capabilities of LMs enables the zero-shot adaptation of EIF agents, without additional environment interactions or data collection efforts (Song et al., 2023; Singh et al., 2023; Brohan et al., 2023c). In these approaches, an LM is prompted with the semantics of pretrained skills Lπ = {l : πl ∈ Π} along with a specific instruction i from the user. Subsequently, the LM produces a series of semantic skills. The execution of these skills facilitates the decomposition of the complex task associated with instruction i into smaller, achievable skill-level subtasks.

heading: 2.2 Cross-Domain Skill Grounding
text: While several works have explored the use of LMs as task planners, they primarily focus on singledomain agent deployment, wherein skills are both acquired and used within the same environment setting. In contrast, the cross-domain setting presents unique challenges for embodied task planning, as the feasibility of skill execution in the environment differs, necessitating the adaptation of pretrained skills to the specific domain contexts of the target environment. Consider the skill “Make a cup of coffee”, as illustrated in Figure 1, learned in a high-tech kitchen environment equipped with voice-activated appliances, a digital inventory system, and automated utensil handlers. In this environment, the skill entails complex interactions with
technology to efficiently prepare a cup of coffee. However, when a robot is deployed in a traditional kitchen devoid of such advanced features, it encounters a significantly different domain context. The skill “Make a cup of coffee”, initially tailored for the environment with high-tech gadgets, now faces significant execution challenges due to the unavailability of similar technological supports.
To address this gap, we introduce cross-domain skill grounding, a process aimed at adapting pretrained skills πl ∈ Π for diverse domains. This process involves not only reinterpreting the original skill but also deconstructing it into a series of achievable lower-level subtasks, such as “Boil water”, “Add coffee to cup”, “Pour water”, and “Stir the drink”. Through cross-domain skill grounding, we ensure that embodied agents can perform a vast set of skills within domains that differ significantly from those in which the skills were initially learned.

heading: 3 Approaches
text: We utilize the intrinsic semantic hierarchy of skills, a structure derived from their compositional nature; high-level semantic skills are established by chain-
ing short-horizon low-level skills. High-level skills, characterized by rich semantics, closely align with the abstract nature of user instructions. This alignment in abstraction levels facilitates the task planning process, providing the LM with skills that are directly beneficial for interpreting and accomplishing the instructions. However, the execution of these long-horizon high-level skills is contingent upon the precise alignment of various environmental and domain-specific conditions. This precludes their adaptability in cross-domain settings. In contrast, low-level skills exhibit a robust capacity for execution that is not as tightly bound to specific domain contexts, offering greater adaptability across diverse domains. Nonetheless, the sheer volume of low-level skills and their limited semantic depths pose additional challenges for task planning.
The core of SemGro lies in the hierarchical task planning scheme via iterative skill grounding. The framework initiates task planning with high-level rich-semantic skills, and then progressively decomposes each of these planned high-level skills into a series of low-level skills executable within the domain of the target environment. This can be seen
as finding an optimal intermediate semantic representation for skills pertinent to a given instruction, exploring a balance between their logical relevance and actual feasibility in the target environment. Such a balance ensures that the planned skills not only match the conceptual demands of the given instruction but are also effective within the limitations and resources of the target environment.

heading: 3.1 Overall Framework
text: As illustrated in Figure 2, SemGro is structured with (a) a task planner and (b) a multi-modal skill critic, which collaboratively ground skills for user instructions in cross-domains. At each skill-step, the task planner produces an appropriate semantic skill by prompting an LM with information retrieved from a skill database. The skill critic then evaluates each skill from the skill planner by prompting a multi-modal LM with visual observations from the environment. When evaluated as unsuitable, the task planner decomposes the skill into smaller, more general subtasks using another retrieval from the database. Through this iterative process, the framework systematically assembles a sequence of semantic skills, executable within the domain context of the target environment.

heading: 3.2 LM-based Task Planner
text: For LM-based task planners, the development of effective prompts is essential to fully harness the reasoning capabilities of LMs. We consider two aspects: (i) selecting skill candidates that facilitate the skill-level task decomposition and (ii) incorporating in-context examples applied to similar instructions. To capture these aspects, we construct a hierarchical semantic skill database, leveraging the reasoning prowess of LMs. Integrated with a kNN retriever, this database not only assists in identifying suitable high-level skill candidates for task planning but also facilitating the retrieval of pertinent in-context examples that can further aid the planning process. The task planner, enriched with this selection of skill candidates and in-context examples, ensures the application of suitable skill semantics in cross-domains.
Hierarchical semantic skill database. For the hierarchical semantic skill databaseD, we use a structured approach to semantic skill training, similar to (Zhang et al., 2023). The skill acquisition initiates with a set of short-horizon basic low-level skills Π1l = {π (1,1) l , · · · , π (1,N1) l } in the training
environment, which can be acquired via RL or imitation learning. Under the LM guidance, the agent produces skill chains from Π1l , achieving high-level long-horizon skills Π2l . This bootstrapping process iteratively expands to form a comprehensive set of skill levels Πl = {Π1l , · · · ,ΠMl }. This collection, encompassing skills from low-level to high-level, is used to establish the skill database D:    Πml = {π (m,1) l , · · · , π (m,Nm) l } D := {e(m,n) : 1 ≤ m ≤M, 1 ≤ n ≤ Nm} e(m,n) := (l(m,n), dn(e(m,n)), p(e(m,n)))
l(m,n) := Skill semantic of π(m,n)l p(e(m,n)) := (e(m−1,nj1 ), · · · ).
(1) The entry e(m,n) of D includes the skill semantic l(m,n) of π(m,n)l , detected object names dn(e (m,n)) collected during training of π(m,n)l , and a onestep low-level semantic skill planning p(e(m,n)) = (e(m−1,nj1 ), · · · ), in that the skill π(m,n)l is acquired by chaining of (π
(m−1,nj1 ) l , · · · ). To train the skill
critic (Section 3.3), we also establish a dataset:
DO = {o(TR)t , dn(o (TR) t ), ds(o (TR) t ) : t} (2)
of visual observations o(TR)t in the training environment, detected object names dn(o(TR)t ) and object physical states ds(o(TR)t ) at each timestep t during the establishment of Πl. To obtain such object names (e.g., “microwave”) and physical states (e.g., “microwave: closed”), we use an open-vocabulary detector tailored for the training environment.
Skill generator. For a given user instruction i and target environment observation ot, the LMbased skill generator ϕG integrates skill candidates c(i, ot), in-context examples x(i, ot), and the skill execution history h to generate the skill semantic l̄ which is most likely to be executed at the present:
ϕG : (i, h,x(i, ot), c(i, ot)) 7→ l̄ ∈ c(i, ot). (3)
To derive these in-context examples and skill candidates, we use a kNN retriever that selects entries e(i, ot) = {e(m1,n1), · · · , e(mk,nk)} inD. The retrieval process calculates similarity by combining the instruction i and the visual context of ot into a single query. It evaluates ⟨i, l(m,n)⟩ + ⟨dn(e(m,n)), ot⟩ to determine the k entries with the highest similarity scores, where ⟨·, ·⟩ denotes the cosine similarity. Here, l(m,n) is the skill semantic
and dn(e(m,n)) is the detected object names associated with entry e(m,n). These retrieved entries are then used to define one-step lower-level plans as incontext examples x(i, ot) and one-step lower-level skill semantics c(i, ot) as the planning candidates for the skill generator ϕG:    e(i, ot) = {(e(m1,n1), · · · , e(mk,nk))} x(i, ot) = {(l(m1,n1), pl(e(m1,n1)), · · · } c(i, ot) = {l(m1−1,nj1 ), · · · }. (4)
Here, pl(e(m,n)) is the semantics in p(e(m,n)).
Task retriever. If the planned skill semantic l̄ from the skill generator ϕG is identified as inexecutable by the skill critic, the LM-based task retriever ϕR steps in to generate a detailed, fine-grained language instruction i∗. This new instruction i∗ is generated in response to the feedback f regarding the rationale for this inexecutability and lower-level skill semantics lc(i, ot) that are constituent of the original skill semantic candidates c(i, ot) in Eq (4): { ϕR : (l̄, f, lc(i, ot)) 7→ i∗ lc(i, ot) = ⋃k j=1{pl(e′) : e′ ∈ p(e(mj ,nj))}. (5) This instruction i∗ guides the framework to perform smaller, more manageable subtasks by triggering the reiteration of skill generation in Eq (3).

heading: 3.3 Multi-Modal Skill Critic
text: For observation ot and skill semantic l̄ derived from the skill generator ϕG (in Eq (3)), the skill critic ψ assesses the execution feasibility of skill πl̄:
ψ : (ot, l̄) 7→ c ∈ {E, (NE, f)}. (6)
Here, E specifies that the skill is executable, while NE denotes inexecutability with language feedback f . The critic ψ includes the vision-language model (VLM)-part ψVLM and LM-part ψLM. Here, ψVLM identifies object names dn(ot) and states ds(ot) from ot, and ψLM utilizes this information to determine the executability of the skill πl̄:
   ψ = ψLM ◦ ψVLM : (ot, l̄) 7→ {E, (NE, f)} ψVLM : ot 7→ (dn(ot), ds(ot)) ψLM : (l̄, dn(ot), ds(ot)) 7→ {E, (NE, f)}.
(7) We use the dataset DO (in Eq (2)) to fine-tune the physically grounded VLM (Gao et al., 2023). Our ablation study (Section 4.3) specifies the benefits
Algorithm 1 Iterative Skill Grounding for EIF 1: User instruction i, skill database D 2: Task planner ϕ = (ϕG, ϕR) 3: Skill critic ψ = ψLM ◦ ψVLM 4: Instruction stack I ← [i] 5: Execution history H ← [] 6: t← 0, ot ← env.reset() 7: repeat 8: i← I .pop(), h← H .pop() 9: e(i, ot)← k entries via kNN retriever
10: x(i, ot)← in-context examples (Eq (4)) 11: c(i, ot)← skill candidate (Eq (4)) 12: l̄← ϕG(i, h,x(i, ot), c(i, ot)) (Eq (3)) 13: c← ψ(ot, l̄) (Eq (6)) 14: if c = E then 15: I ← I .append(i), H ← H .append(l̄) 16: t← t+ 1 17: ot ← env.step(l̄) 18: else if c = (NE, f) then 19: lc(i, ot)← lower skills (Eq (5)) 20: i∗ ← ϕR(l̄, f, lc(i, ot))(Eq (5)) 21: I ← I .append(i) 22: I ← I .append(i∗) 23: end if 24: until env.done()
of this LM’s multi-modal extension with VLMs in cross-domains.
The iterative skill grounding scheme of SemGro is illustrated in Algorithm 1.

heading: 4 Experiments
text: Environment. We use VirtualHome, a realistic household activity simulator and benchmark. For EIF agents, we define 3949 low-level skills (i.e., Π1 in Section 3.2) based on the combination of behaviors (e.g., “grasp”) and objects (e.g., “apple”). For cross-domain evaluations, we use 15 distinct environment settings, categorized by one of domain shift types: OL involves changes in the placements and locations of objects; PA presents changes in the physical attributes of objects; RS is characterized by different room structures and visual attributes. These cross-domain contexts necessitate the refinement of certain pretrained high-level skills into low-level skills for successful execution. Consider a high-level skill “cook salmon” intended to fulfill the user instruction “prepare delicious food for me”. If this skill was trained in a setup where salmon is always prepared on a cutting board, it might not
work in OL cross-domain scenarios where salmon is instead found inside a refrigerator. In this case, it becomes necessary to decompose the skill into a sequence of lower-level executable skills such as “go to the refrigerator”, “grasp the refrigerator handle”, among others, ensuring the skill be adapted to the new domain context.
Instructions. Given 5 evaluation tasks, we use 4 different instruction types per task: Abstract Nouns and Abstract Verbs focus on instruction queries involving abstract nouns and verbs, respectively; Structured language queries include precise verbs; Long-Horizon queries involve multiple reasoning steps. These instruction types are used to test various linguistic aspects of task interpretation and execution, similar to (Brohan et al., 2023c). By combining 15 environment settings, 5 tasks, and 4 types of instructions, we establish a comprehensive set of 300 distinct unseen cross-domain scenarios.
Evaluation metrics. We use several evaluation metrics, consistent with previous works (Singh et al., 2023; Song et al., 2023). SR measures the percentage of successfully completed tasks, where a task is considered to be completed if its all goal conditions are completed; CGC measures the percentage of completed goal conditions; Plan measures the percentage of the planned skill sequence that continuously aligns with the ground-truth sequence from the beginning.
Baselines. We compare our SemGro with three baselines, each leveraging an LM as a task planner but with distinct approaches to filter executable skill candidates, including SayCan (Brohan et al., 2023c) which employs an offline RL dataset collected from the training environment to learn an affordance function; LLM-Planner (Song
et al., 2023) which leverages templatized semantic skills; ProgPrompt (Singh et al., 2023) which uses human-crafted programmatic assertion syntax to verify the pre-conditions of skill execution and respond to the failed cases with recovery.

heading: 4.1 SemGro Implementation
text: For the kNN retriever, we configure k to be 10 and utilize the sentence-transformer (Reimers and Gurevych, 2019) to compute the similarity between the instruction and skill semantics in the database. The task planner ϕ and the LM-part ψLM of the skill critic are both implemented using GPT-3.5 API with the temperature hyperparameter set to 0. For fair comparisons, the same GPT-3.5 configuration is used across all baseline task planners. More details, including the baseline implementation, are in the Appendix.

heading: 4.2 Performance
text: Overall performance. Table 1 compares the crossdomain performance, achieved by our SemGro and the baselines (LLM-Planner, SayCan, ProgPrompt). SemGro outperforms the baselines in all cross-domain scenarios, achieving average performance gains of 25.02%, 26.83%, and 27.65% over the most competitive baseline SayCan in SR and CGC, and LLM-Planner in Plan, respectively. The exceptional performance of SemGro highlights its superiority in hierarchical task planning via iterative skill grounding across unseen domains.
Interestingly, for long-horizon instruction scenarios, there are several instances where SR and CGC metrics exceed Plan. In the PA cross-domain context, SemGro achieves Plan of 61.11%, while SR and CGC reach 62.96% and 74.07%, respectively. This phenomenon, reminiscent of solving math problems in an NLP context (Lightman et al.,
2023; Fu et al., 2023; Imani et al., 2023), suggests that tasks requiring multiple reasoning steps can be accomplished through diverse skill sequences.
Executable skill identification. We investigate the efficacy of SemGro in generating executable skills. To this end, we introduce the Exec metric, which quantifies the percentage of the planned semantic skills that are executable in a given domain, regardless of their impact on task completion. In Table 2, SemGro surpasses the most competitive baseline ProgPrompt by 17.99% in Exec at average. This specifies our framework’s proficiency in discerning the varying conditions of cross-domain environments for skill grounding. This ability is attributed to the skill critic which uses the domain-agnostic representation capabilities of the adopted VLM.
Iterations for cross-domain skill grounding. We investigate the number of iterations used for skill grounding in the target domain across varying degrees of domain shift. To quantify the degree of shift, we calculate the number of high-level skills that must be decomposed into different series of low-level skills for execution in the target domain, compared to the training environment. The degree of shift is categorized into 4 levels. ‘None’ indicates no domain shift, while ‘Small,’ ‘Medium,’ and ‘Large’ denote increasing degrees of shift. The detailed criterion for quantifying domain shift degrees can be found in Appendix B. To accurately measure the iterations induced by domain shift, distinctions are made between skills rendered inexecutable due to the immediate observational state (Obs. & Dom. in the row name) and those affected by domain disparities (Dom.). Table 3 reveals a positive correlation between the domain shift degrees and the number of iterations. This specifies the benefits of SemGro in handling cross-domain settings with varying degrees of domain shift, progressively searching the appropriate abstraction level of skills.

heading: 4.3 Ablation Studies
text: Skill hierarchy for planning. To verify the effectiveness of the iterative skill grounding scheme, we conduct task planning with the fixed abstraction
level of skills, bypassing the executability check of the skill critic. In Table 4, SG-L and SG-H employ the lowest-level skills Π1l and highest-level skills ΠMl for task planning, respectively, while SG-M employs middle-tier skills {Πml : 1 < m < M}. As shown, SG-H exhibits the most accurate task planning (high Plan), while its planned skills are rarely executable (low Exec) due to domain shift. Conversely, SG-L, equipped with potentially executable skills, shows unreliable task planning due to the abstraction level mismatch with the instruction. Although SG-M employs skills with moderate semantic content and executability by adhering to a fixed abstraction level, it falls short in performance. SemGro adaptively identifies skill semantics at the intermediate level, through iterative downward decomposition within the semantic hierarchy of skills. This strategy not only facilitates task planning but also achieves robust executions in new domains.
In-context example retrieval. In Table 5, we conduct an ablation study on the kNN retriever by varying the number k of retrieved entries in Eq (4). Furthermore, we test a random-selection retriever that selects 10 examples at random (Random in the column name) as well as another variant that calculates the similarity based solely on instructions without observation information (k = 10 \O in the column name). For the random-selection retriever, skill candidates are determined using the kNN retriever to precisely assess the impact of in-context examples. As observed, selecting an appropriate number of examples (k = 10) leads to performance gain in Plan by 30.89% and 7.87% compared to the random retriever and k = 10 \O, respectively. Furthermore, we observe an increase in planning
performance when k ranges from 5 to 10, but beyond this range, larger values do not yield further enhancements. This implies that including skill candidates unrelated to the task may deteriorate the task planning.
The dual-model structure for skill critic. As explained in Section 3.3, we structure the skill critic ψ with two models ψVLM and ψLM. To validate this dual-model structure, we conduct an ablation study using a single VLM for the skill critic. Specifically, we use InstructBLIP (Dai et al., 2023), a state-of-the-art general-purpose VLM, alongside its fine-tuned counterpart PG-InstructBLIP (Gao et al., 2023), which is optimized for object-centric physical understanding. We fine-tune these VLMs on the VirtualHome dataset. We provide the dataset collection process in the Appendix. In Table 6, the dual-model structure for ψ improves the performance in Exec by 9.62% over the singular VLM. This specifies that while VLMs are adept at assessing physical attributes, their capacity to discern the executability of semantic skills in diverse physical environments can be more effectively augmented by the inclusion of LMs.
LMs for task planner and skill critic. To implement the task planner ϕ = (ϕG, ϕR) and the LMpart ψLM of the skill critic, we test a range of LMs from open-source LLaMA-2-70B to proprietary capable LMs such as PaLM, GPT-3.5, and GPT-4. In Table 7, while LLaMa-2-70B, with significantly fewer parameters, does not achieve comparable performance, SemGro demonstrates robust performance consistently with capable LM backbones.

heading: 5 Related Works
text: Language models for embodied control. The use of pretrained LMs for embodied control tasks has been increasingly prevalent. A significant focus of this research is on bridging the gap between LMs’ extensive knowledge and their lack of inherent understanding of embodied experiences. Several works tackle the challenge by directly incorporating environmental contexts to LMs through textual prompts (Xiang et al., 2023; Lin et al., 2023b,a; Shinn et al., 2023; Carta et al., 2023), based on the premise that observations can be described in text form. For example, (Lin et al., 2023b) utilizes a text-based table to display objects’ location and their relation. Expanding beyond the mere textual environment, other works have pivoted towards visual tasks and EIF (Wang et al., 2023; Song et al., 2023; Driess et al., 2023; Wu et al., 2023; Singh et al., 2023). Particularly, (Brohan et al., 2023c) introduces a method to learn skill affordance from visual observations using an offline RL dataset collected from the specific environment. This approach integrates skill logits from an LM with identified skill affordance to select the most valuable skill for execution. Meanwhile, (Wu et al., 2023; Song et al., 2023) use detected object classes in the observation to refine the selection of actionable skills.
Our SemGro distinguishes itself in EIF by being the first to harness the capabilities of LMs for grounding pretrained skills in cross-domains.
Language conditioned policy. A number of works have concentrated on the generalization of language-conditioned policies or skills for unseen language instructions (Guhur et al., 2022; Garg et al., 2022), visual attributes (Brohan et al., 2023b,a; Stone et al., 2023; Shridhar et al., 2022, 2021), or environment dynamics (Shin et al., 2023), with some extending to multi-modal instructions (Li et al., 2023b; Shah et al., 2023; Shin et al., 2024; Jiang et al., 2022; Jang et al., 2022). For instance, (Brohan et al., 2023a) adapts a pretrained foundation model to robot trajectory datasets in an
end-to-end manner, by representing robot actions as text tokens. (Li et al., 2023b; Jiang et al., 2022) offer a unified framework capable of achieving interleaved vision-language instructions, employing a set of pretrained models to encode them.
While prior works emphasize the robust policy structure, SemGro leverages an LM-based iterative reasoning mechanism, aiming for balanced skill grounding across diverse domain shifts. Moreover, SemGro addresses realistic EIF tasks with the VirtualHome benchmark, unlike the prior works that often focus on straightforward manipulation tasks.

heading: 6 Conclusion
text: In this paper, we presented a semantic skill grounding framework SemGro, which supports the adaptation of a broad range of skills in cross-domain environments. SemGro harnesses the reasoning capabilities of LMs not only to compose semantic skills for establishing a hierarchical skill database but also to decompose semantic skills into executable low-level skills. This novel approach uniquely explores the trade-off between semantic richness and domain-agnostic executability of semantic skills. We hope our work provides valuable insights into the development and application of real-world EIF agents.

heading: 7 Limitations
text: Despite the strong performance of SemGro, we identify several limitations for improvement. (i) Single-step visual observation for skill feasibility: Currently, the skill feasibility is assessed based on a single observation. This approach can be improved by integrating a VLM into our framework that is capable of generating imaginary observations, at the expense of increased resource usage by the embodied agent. By employing Monte Carlo Tree Search, the feasibility of skills could then be computed across multiple planning paths. Addressing this enhancement remains for our future work.
(ii) Unidirectional decomposition strategy: The strategy for semantic skill decomposition is unidirectional, proceeding only in a downward direction. This may often restrict the framework as it cannot revert to initiating planning from a highlevel skill at an appropriate juncture. Introducing bidirectionality in task planning could improve the versatility of SemGro. In scenarios with substantial domain shifts, such bidirectionality would allow for completely novel planning that has not been
trained to achieve complex, unseen tasks. Enhancing our framework to incorporate this bidirectional planning presents a challenge but crucial direction for future development.

heading: 8 Ethics Statement
text: This work introduces a technology that can impact the application of robotics in household tasks. This technology signals a major shift in the human experience, as household tasks increasingly become automated by robotic agents. The deployment of robotic agents in household tasks presents both significant benefits and challenges. On the positive side, this technology can save time and effort for users, leading to increased convenience and efficiency in managing household chores. However, it also raises important ethical considerations related to labor displacement. As robotic agents take over tasks traditionally performed by humans, there is a potential impact on the livelihoods of current service providers who may need to seek alternative sources of income. This transition necessitates careful consideration and planning to mitigate adverse effects on workers and to ensure that the benefits of technological advancement are equitably distributed. By acknowledging and addressing these societal impacts, our research aims to contribute positively to the field of robotics while fostering an inclusive approach that considers the broader implications for society.

heading: 9 Acknowledgements
text: We would like to thank anonymous reviewers for their valuable comments and suggestions. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-01045, 2022-0-00043, 2019-0-00421, 2020-0-01821), and by the National Research Foundation of Korea (NRF) grant funded by MSIT (RS-2023-00213118).

heading: B Experiment Details
text: In this section, we describe the experiment settings in detail. We also specify the cross-domain scenarios and environment details used in the experiments, as well as the data collection process for our hierarchical skill database and the training procedure of our multi-modal critic. Cross-domain scenarios. We used 5 EIF tasks, 4 types of language instructions, and 15 environmental scenes to test across 300 cross-domain scenarios. The variation in instructions, akin to the approach used in SayCan, is crafted to assess the extent to which our framework can deconstruct abstract instructions and ground skills in the target domain. We utilize 4 instruction types such as Abstract Noun, Abstract Verb, Structured, and Long Horizon, comprising a total of 137 instructions.
The Abstract Noun and Abstract Verb categories are created to examine whether SemGro can execute the appropriate sequences when given abstractions are in nouns and verbs. Structured Language, paralleling Abstract Verbs, aims to evaluate SemGro’s performance with well-structured languages. Long Horizon, necessitating multi-step reasoning, is designed to evaluate SemGro’s capability to process and reason through temporally extended instructions. VirtualHome simulation. We use VirtualHome (Puig et al., 2018), an interactive simulation platform for complex household tasks. The skills provided in the environment include manipulation actions such as picking up objects and toggling appliances on or off. We impose restrictions on these actions within the environment to align with our settings; the skills performed are implemented to be feasible only with the objects currently visible. For example, “grab apple” is executable when the apple is closed enough and is visible to the agent. The action space of the VirtualHome experiments is provided in Table 8.
Table 8: VirtualHome Actions
Actions
find <obj>, grab <obj>, walk <obj>, sit <obj> put <obj> <obj>, open <obj>, close <obj> switchon <obj>, switchoff <obj>
We use a single training environment and 15 distinct environment settings for cross-domain evaluation, with the training environment being environment ID 0 in VirtualHome. We use three different domain-shift types, OL, PA, and RS as explained in the main manuscript. For each type, we use 5 distinct environment settings. The OL scenario involves alterations in the locations of objects and their relationships compared to the training environment. The PA scenario reflects changes in the physical attributes of objects from the training environment. RS scenario encompasses shifts in the visual context, object locations, and their physical attributes. Figure 3 presents visual examples of tasks, and Table 9 details the cross-domain scenario examples employed in our experiments. Dataset collection process. The skill database D comprises 4 distinct abstraction levels, each containing 3949, 183, 142, and 90 skills, from the first to the fourth level, respectively. We adopt a bottomup approach to construct the skill database by progressively chaining low-level skills to form complex high-level skills. Specifically, we employ the skill bootstrapping method in (Zhang et al., 2023), which leverages an LM to guide the development of high-level skills from pretrained low-level skills.
Upon initiation in the environment, the agent selects a skill from the available low-level skills. The LM is then queried to identify the most suitable next skill to execute, based on the provided prompt. Following this, the agent performs the selected skill. This cycle repeats until reaching a predetermined maximum number of steps or encountering a skill execution failure. Successful skill executions are summarized by the LM to derive a high-level abstract skill name, enabling the accumulation of a
comprehensive skill set across different abstraction levels. For both skill selection and summarization, the text-bison-001 model from PaLM (Chowdhery et al., 2023) is utilized.
Furthermore, we use an open-vocabulary detection model owl-ViT (Stone et al., 2023) to obtain the object names dn(o(TR)) (e.g., “microwave”) and InstructBLIP (Dai et al., 2023) to obtain the physical state of each object ds(o(TR)) (e.g., “microwave: closed”). VLM fine-tuning. We fine-tune the PGInstructBLIP to implement the VLM-part ψVLM of the skill critic, using the dataset DO in Eq (2). This process enhances the VLM’s understanding of physical objects within the VirtualHome environment by adopting a Visual Question Answering (VQA) training approach. An object detector is employed to isolate the portions of images that feature objects, and these cropped images are then used to train the VLM in recognizing the objects’ states. Specifically, we formulate questions about the objects’ conditions (e.g., “Is this object open or closed? Answer ’unknown’ if uncertain. Short answer:”) and train the VLM to accurately answer these queries.
We use the FlanT5-XL (Chung et al., 2022) variant of InstructBLIP (Dai et al., 2023), implemented in vision-language multi-modal opensource project LAVIS (Li et al., 2023a). The model is trained for a total of 12 GPU hours on a system equipped with an Intel(R) Core(TM) i9-10980XE processor and an NVIDIA RTX A6000 GPU. Quantification of domain disparities in Table 3. To quantify domain disparities, we calculate the number of high-level skills that must be decomposed into different series of low-level skills for execution in the target domain, compared to the training environment.
For instance, the high-level skill "cook salmon", initially learned in the training environment through a skill-chaining process involving lowlevel skills such as "grasp salmon" and "place salmon in the microwave", may require adaptation in the target domain. This adaptation might involve a different series of low-level skills "go to refrigerator", and "grasp refrigerator handle", among others, for successful execution. Each instance where a high-level skill’s execution strategy deviates from its training environment counterpart contributes to the domain shift quantification. The term ‘None’ refers to scenarios without domain shift, meaning the model is evaluated within the training environ-
ment itself. The categories of "Small," "Medium," and "Large" domain shifts correspond to the increasing percentages (20%, 30%, and 50%, respectively) of high-level skills that require adaptation through decomposition into low-level skills in the target domain.

heading: C Trajectory Examples and Prompts
text: From Table 10 to 12, we describe the trajectory collected by the VirtualHome agent. Specifically, across each skill step, the inputs and outputs of the task planner and skill critic (e.g., detected object names and their states, retrieved skill semantics and skill candidates, generated skills, inputs and outputs of the critic, and fine-grained instructions) are shown in the Table. Following these tables, we also document the prompts used for task planning and skill chaining.
Prompt examples utilized by the skill generator in SemGro.
Examples of common household tasks and their skill sequence to achieve the tasks using skills from Skill Library:
Skill Library: find kitchencabinet, put bananas kitchencabinet, open kitchencabinet, walk kitchencabinet, put bananas kitchentable, find peach, close kitchencabinet, walk bananas, walk cereal, put peach kitchentable, put bananas coffeetable, find coffeetable, find kitchentable, grab bananas Task: open kitchencabinet and put bananas in kitchencabinet skill steps: 1. walk bananas 2. grab bananas 3. walk kitchencabinet 4. find kitchencabinet 5. open
kitchencabinet 6. put bananas kitchencabinet 7. close kitchencabinet
Skill Library: find bananas, put bananas kitchentable, walk peach, open kitchencabinet, walk coffeetable, put apple kitchencabinet, walk kitchencabinet, close kitchencabinet, walk cereal, find kitchencabinet, put peach kitchentable, put peach kitchencabinet, grab peach Task: open kitchencabinet and put peach in kitchencabinet skill steps: 1. walk peach 2. grab peach 3. walk kitchencabinet 4. find kitchencabinet 5. open
kitchencabinet 6. put peach kitchencabinet 7. close kitchencabinet
Skill Library: walk peach, walk bananas, walk kitchentable, open kitchencabinet, put apple kitchencabinet, find apple, put peach kitchencabinet, grab peach, walk kitchencabinet, walk apple, find kitchencabinet, find peach, grab apple, grab cereal, close kitchencabinet Task: put peach in kitchencabinet skill steps: 1. walk peach 2. find peach 3. grab peach 4. walk kitchencabinet 5. find kitchencabinet
6. open kitchencabinet 7. put peach kitchencabinet 8. close kitchencabinet
Skill Library: put cereal kitchencabinet, find bananas, walk bananas, put bananas kitchentable, walk cereal, walk coffeetable, put bananas kitchencabinet, walk kitchencabinet, find kitchentable, grab peach, find apple, find coffeetable, walk kitchentable, grab bananas, put peach kitchencabinet Task: find bananas and put bananas on kitchentable skill steps: 1. walk bananas 2. find bananas 3. grab bananas 4. walk kitchentable 5. find
kitchentable 6. put bananas kitchentable
Skill Library: close kitchencabinet, find apple, grab apple, walk apple, walk kitchentable, open kitchencabinet, walk coffeetable, put cereal kitchencabinet, find kitchencabinet, put apple kitchencabinet, walk cereal, find coffeetable, walk bananas, walk kitchencabinet Task: put apple in kitchencabinet skill steps: 1. walk apple 2. find apple 3. grab apple 4. walk kitchencabinet 5. find kitchencabinet
6. open kitchencabinet 7. put apple kitchencabinet 8. close kitchencabinet
Skill Library: put apple kitchencabinet, walk cereal, put bananas kitchencabinet, find coffeetable, find kitchentable, walk bananas, grab bananas, open kitchencabinet, put peach kitchentable, put bananas coffeetable, grab peach, walk coffeetable, find peach, put bananas kitchentable, find bananas, walk kitchentable Task: find bananas and put bananas on coffeetable skill steps: 1. walk bananas 2. find bananas 3. grab bananas 4. walk coffeetable 5. find coffeetable
6. put bananas coffeetable
Skill Library: find bananas, walk coffeetable, grab cereal, walk bananas, find coffeetable, open kitchencabinet, grab bananas, find kitchencabinet, close kitchencabinet, put bananas coffeetable, walk kitchencabinet, put cereal kitchencabinet, put peach kitchentable, walk cereal Task: open kitchencabinet and put cereal in kitchencabinet skill steps: 1. walk cereal 2. grab cereal 3. walk kitchencabinet 4. find kitchencabinet 5. open
kitchencabinet 6. put cereal kitchencabinet 7. close kitchencabinet
Choose the next skill from Skill library to successfully accomplish the task. Skill library: walk bananas, grab bananas, walk kitchencabinet, find kitchencabinet, open
kitchencabinet, put bananas kitchencabinet, close kitchencabinet, walk bananas, find bananas, grab bananas, walk kitchencabinet, find kitchencabinet, open kitchencabinet, put bananas kitchencabinet, close kitchencabinet, walk peach, grab peach, walk kitchencabinet, find kitchencabinet, open kitchencabinet, put peach kitchencabinet, close kitchencabinet, walk apple, grab apple, walk kitchencabinet, find kitchencabinet, open kitchencabinet, put apple kitchencabinet,
skill steps: 1.
Prompt examples utilized for the skill chaining.
Examples of common household tasks and their descriptions: The Task are composed of skill steps chosen from Skill Library, Skill Library: wash the frying pan, put the candle on the living room shelf,throw away the lime, open
window, listen to radio, wash clothes, wash the plate, get my pants and shirt and place on the bed, call a friend, wash mug, wash the rug in washing machine,
cut apple, make toast, read book under table lamp, brush teeth, open the curtains, put clothes in closet, throw away apple, put all the cutlery in the sink, put papers in the folder, put my slippers near the bed, take out the trash, microwave chicken, heat cupcakte
Skill Steps: 1. wash the plate 2. wash mug Task: wash the plate and mug
Skill Steps: 1. brush teeth and wash clothes Task: brush teeth and wash clothes
Skill Steps: 1. brush teeth 2. read book under table lamp Task: read book under table lamp after brushing teeth
Skill Steps: 1. wash the frying pan 2. wash the plate 3. plate mug Task: wash the kitchenware
Skill Steps: 1. wash clothes 2. wash the rug in washing machine Task: wash clothes and rug using washing machine
Skill Steps: 1. throw away apple 2. throw away the lime Task: throw away apple and lime
Skill Steps: 1. cut apple 2. make toast Task: cut apple and make toast
Skill Steps: 1. make tea and bring it to the coffee table 2. set up coffee table for fruits Task: set up tea and fruits on coffeetable
Skill Steps 1. put apple in fridge and bring cellphone to sofa 2. put hairproduct in bathroom cabinet and bring paper to sofa Task: clean up and bring cellphone and paper to sofa
Skill Steps: 1. reheat food 2. prepare chicken for dinner Task: cook chichen for dinner
Skill Steps: 1. put book back and sit on the sofa 2. watch tv and turn off microwave Task: organize books and watch tv
Skill Steps: 1. put things back in their place 2. put away mug and lime Task: clean bedroom
Skill Steps: 1. reheat cupcake and put it on coffeetable 2. reheat chicken and put it on kitchentable Task: heat cupcake and chicken and put them on coffeetable
Skill Steps: 1. put away mug and lime 2. eat bread on the sofa and put book back in the bookshelf Task: clean livingroom
Skill Steps: 1. open the curtains 2. put my slippers near the bed Task: open the curtains and put my slippers near the bed
Skill Steps: 1. open window 2. call a friend Task: open window and call a friend
Skill Steps: 1. microwave chicken 2. heat some cupcake Task: microwave the chicken and heat some cupcake
Skill Steps: 1. microwave chicken 2. heat some cupcake Task: microwave the chicken and heat some cupcake
Predict the next skill for creating plausible task by correctly choosing from Skill Library:
Prompt examples utilized for the skill chaining.
Instructions: give a high-level description for the following steps describing common household tasks .
Skill Steps: 1. wash the plate 2. wash mug Summary: wash the plate and mug
Skill Steps: 1. brush teeth and wash clothes Summary: brush teeth and wash clothes
Skill Steps: 1. brush teeth 2. read book under table lamp Summary: read book under table lamp after brushing teeth
Skill Steps: 1. wash the frying pan 2. wash the plate 3. plate mug Summary: wash the kitchenware
Skill Steps: 1. wash clothes 2. wash the rug in washing machine Summary: wash clothes and rug using washing machine
Skill Steps: 1. make tea and bring it to the coffee table 2. set up coffee table for fruits Summary: set up tea and fruits on coffeetable
Skill Steps 1. put apple in fridge and bring cellphone to sofa 2. put hairproduct in bathroom cabinet and bring paper to sofa Summary: clean up and bring cellphone and paper to sofa
Skill Steps: 1. reheat food 2. prepare chicken for dinner Summary: cook chichen for dinner
Skill Steps: 1. put book back and sit on the sofa 2. watch tv and turn off microwave Summary: organize books and watch tv
Skill Steps: 1. put things back in their place 2. put away mug and lime Summary: clean bedroom
Skill Steps: 1. reheat cupcake and put it on coffeetable 2. reheat chicken and put it on kitchentable Summary: heat cupcake and chicken and put them on coffeetable
Skill Steps: 1. put away mug and lime 2. eat bread on the sofa and put book back in the bookshelf Summary: clean livingroom
Skill Steps: 1. throw away apple 2. throw away the lime Summary: throw away apple and lime
Skill Steps: 1. cut apple 2. make toast Summary: cut apple and make toast
Skill Steps: 1. open the curtains 2. put my slippers near the bed Summary: open the curtains and put my slippers near the bed
Skill Steps: 1. open window 2. call a friend Summary: open window and call a friend
Skill Steps: 1. microwave chicken 2. put chicken on coffeetable Summary: heat chicken and put it on coffeetable
Summarize the skill steps and give a plausible name for the household task executed.

heading: Future Work
text: Addressing this enhancement remains for our future work. (ii) Unidirectional decomposition strategy: The strategy for semantic skill decomposition is unidirectional, proceeding only in a downward direction. This may often restrict the framework as it cannot revert to initiating planning from a highlevel skill at an appropriate juncture. Introducing bidirectionality in task planning could improve the versatility of SemGro. In scenarios with substantial domain shifts, such bidirectionality would allow for completely novel planning that has not been
trained to achieve complex, unseen tasks. Enhancing our framework to incorporate this bidirectional planning presents a challenge but crucial direction for future development