Embodied Language Learning: Opportunities, Challenges, and Future Directions (2024)

Abstract: While large language and vision-language models showcase impressive capabilities, they face a notable limitation: the inability to connect language with the physical world. To bridge this gap, research has focused on embodied language learning, where the language learner is situated in the world, perceives it, and interacts with it. This article explores the current standing of research in embodied language learning, highlighting opportunities and discussing common challenges. Lastly, it identifies existing gaps from the perspective of language understanding research within the embodied world and suggests potential future directions.

heading: 1 Introduction
text: Besides observing their surroundings, humans actively contribute to their understanding of the world by interacting with it and communicating with others (Smith and Gasser, 2005; Barsalou, 2008). This interactive experience is integral to language acquisition and understanding (Bender and Koller, 2020). A corresponding notion of World Scopes (Bisk et al., 2020a), namely Corpus, Internet, Perception, Embodiment, and Social, has been proposed to measure progress in language understanding research. While today’s large language models (LLMs) have exhibited powerful capabilities (Bommasani et al., 2021), their textual training data constrain them to the Internet world scope. In turn, large visionlanguage models, trained on image-text corpora, fall within the world of Perception.
Significant research efforts (see §2) have been devoted to transitioning into the embodied realm. However, lying at the intersection of language and robotics research, embodied language learning has been primarily explored from a robotics perspective, with a focus on the general use of language in robotics (Tellex et al., 2020), embodied visionlanguage tasks (Francis et al., 2022; Duan et al., 2022; Deitke et al., 2022), or foundation models for
decision-making (Yang et al., 2023) or as agents (Xi et al., 2023). The main contribution of this article is providing an overview of embodied language learning from the perspective of language understanding research, summarizing relevant background (§2), opportunities (§3), challenges (§4), and research gaps (§5). Throughout the article, embodied language learning is taken to be the process of language acquisition by a language learner, referred to as an agent or a robot, while it is situated in a physical or a virtual world which it perceives and interacts with through action taking; hence, grounding language in its percepts and actions.

heading: 2 Embodied Language Learning
text: Current dominant approaches to language modeling, as represented by LLMs, involve training on a vast quantity of textual data. However, as symbol tokens cannot be grounded in other symbol tokens (Harnad, 1990) and meaning is perceived to be residing in the connection between language and extrinsic non-symbolic representations (Ervin-Tripp, 1973; Bisk et al., 2020a; Bender and Koller, 2020; Lake and Murphy, 2020), merely training language models on as many textual corpora as possible remains insufficient for capturing meaning (Harnad, 1990; Lucy and Gauthier, 2017; Bender and Koller, 2020). It is essential to ground textual corpora in extra-linguistic data (Harnad, 1990; Bisk et al., 2020a). Meaning can then be captured to the extent reflected in such data (Bender and Koller, 2020).
One source of grounding data is perception (Harnad, 1990), including visual, tactile, and auditory inputs (Smith and Gasser, 2005; Bisk et al., 2020a). Efforts have been directed towards vision-language models that attempt to ground language in the visual input, learning alignments between the modalities. Nevertheless, studies on language acquisition among infants (Snow et al., 1976; Kuhl, 2007) suggested that mere perception is inadequate, and that
15369
language cannot be learned from a television (Bisk et al., 2020a; Bender and Koller, 2020).
Meaning and language understanding have been investigated with regard to embodiment theories in cognitive science (Glenberg and Robertson, 2000), which highlight the significance of situated actions (Smith and Gasser, 2005; Barsalou, 2008). Evidence has supported that meaning representations are grounded in embodied experiences and sensorimotor interactions with the world (Jones et al., 1991; Glenberg and Kaschak, 2002; Barsalou and Wiemer-Hastings, 2005). This has motivated research in embodied language learning, where language is grounded in both perception and action (Heinrich et al., 2020).
Much of the research in embodied language learning has been focused on robot learning, following two approaches: embodied exploration and embodied instruction following (see Appendix A). In the first approach, a robot interacts with objects while receiving natural language descriptions of its actions and/or object attributes (Heinrich et al., 2020; Özdemir et al., 2021; Zhang et al., 2023; Tatiya et al., 2023). In the second approach, a robot is given a natural language instruction and learns to execute a corresponding short-horizon skill (Jang et al., 2022; Brohan et al., 2023; Zitkovich et al., 2023; Vuong et al., 2023; Jiang et al., 2023) or plan and carry out sequences of actions to reach a corresponding long-horizon goal (Suglia et al., 2021; Hong et al., 2021; Jin et al., 2023; Driess et al., 2023; Jiang et al., 2023). Recently, Liu et al. (2023) experimented with a robot assigned a nonlanguage task in an environment with language annotations that are useful but not required for task completion. In all approaches, the robot learns to ground language in its sensorimotor experience.

heading: 3 Opportunities & Prospects
text: Opportunities of grounding language manifest in the various dimensions of understanding it unlocks (Bisk et al., 2020a), which are discussed below.

heading: 3.1 Attributes
text: The meaning of some attributes cannot be fully grasped through mere perception (Gibson, 1988; Bisk et al., 2020a; Tatiya et al., 2023). Understanding attributes such as deformability, weight, and hardness requires interacting with objects and perceiving the resulting multi-sensory effects. For instance, lifting an opaque container lends meaning
to its emptiness (Zhang et al., 2023). Establishing such connections between language and actions allows an embodied language learner to reason about these properties (Zellers et al., 2021). For example, it can recognize that a greater force is needed to push a heavier object (Lake and Murphy, 2020).

heading: 3.2 World Dynamics & Affordances
text: Embodiment also allows agents to experiment with different actions (Smith and Gasser, 2005; Bisk et al., 2020a) and associate them with the change they induce in the world (Smith and Gasser, 2005; McClelland et al., 2019; Zellers et al., 2021). This establishes notions of cause and effect (Piaget et al., 1952; Engstrø, 2000), fostering an understanding of world dynamics, physical constraints, and affordances (Gibson, 1988; Jamone et al., 2018). Upon grounding language describing actions in its embodied experience, a language learner’s planning (Driess et al., 2023) and reasoning (Zellers et al., 2021) capabilities are enhanced. For instance, it should be able to judge that a paper plate makes a better frisbee than a ceramic one (Bisk et al., 2020a) or that, upon boiling butter, it should be poured into a jar and not a plate (Bisk et al., 2020b).

heading: 3.3 Metaphors & Abstract Concepts
text: Much of the language contained in corpora is figurative. Yet, the meaning behind metaphors is derived from experiencing the world (Engstrø, 2000; Bisk et al., 2020a). Thus, understanding figurative language is challenging to disembodied language models (Liu et al., 2022; Wicke, 2023). In addition, through metaphors, abstract concepts can be represented by concrete ones (Engstrø, 2000; Feldman and Narayanan, 2004). For instance, the metaphor
“similarity is proximity” connects the abstract similarity to the concrete proximity (Casasanto and Gijssels, 2015). Hence, embodiment can enhance the understanding of abstract concepts.
With these dimensions of understanding opened up, embodied language learning enables a more robust language understanding in the context of the physical world, which is crucial for applications such as language-supported robots (Taniguchi et al., 2019). It provides the opportunity for agents to be better equipped to follow human instructions (Shridhar et al., 2020; Zhang and Chai, 2021; Suglia et al., 2021; Nguyen et al., 2021; Padmakumar et al., 2022; Gao et al., 2022; Blukis et al., 2022), answer human inquiries (Das et al., 2018; Gordon
et al., 2018; Wijmans et al., 2019), or engage in robust human-robot interactions (Tellex et al., 2020).

heading: 4 Challenges & Corresponding Efforts

heading: 4.1 Data Scarcity
text: One major challenge in embodied language learning is data scarcity (Wang et al., 2019, 2020; Vuong et al., 2023). Unlike text or image datasets, embodied research calls for ego-centric data from the agent’s perspective within its environment (Mu et al., 2023), which is comparably limited (Duan et al., 2022; Driess et al., 2023) as it is expensive and time consuming to collect (Wang et al., 2020; Zhang et al., 2023). Embodied language learning additionally requires that data be annotated with natural language descriptions (Yang et al., 2023; Mu et al., 2023). Whether the robot is completing a task or exploring its environment, corresponding textual annotations are essential for learning to ground language (Lake and Murphy, 2020).
To address data scarcity, one adopted approach is multi-task learning (Wang et al., 2019, 2020; Reed et al., 2022; Brohan et al., 2023; Driess et al., 2023; Jiang et al., 2023). With the aim of capturing meaning that transcends specific tasks (Bender and Koller, 2020), this approach is beneficial as it enables knowledge transfer (Wang et al., 2019). Another commonly adopted approach is leveraging foundation models (Yang et al., 2023). Several works have employed pretrained language models (Majumdar et al., 2020; Suglia et al., 2021; Blukis et al., 2022; Jin et al., 2023; Jiang et al., 2023; Mu et al., 2023; Driess et al., 2023) and visionlanguage models (Majumdar et al., 2020; Khandelwal et al., 2022; Shridhar et al., 2022; Zitkovich et al., 2023). This approach leverages language and vision representations learned from large-scale data (Lake and Murphy, 2020; Deitke et al., 2022; Yang et al., 2023) which serve as priors to be further enhanced through fine-tuning on the limited ego-centric data available (Driess et al., 2023).

heading: 4.2 Generalizability
text: Learned language representations should be generalizable, detached from irrelevant features specific to training tasks or environments (Lake and Murphy, 2020; Francis et al., 2022). However, especially with data scarcity, models tend to overfit and perform poorly in unseen environments (Wang et al., 2020; Deitke et al., 2022). Embodied language learning also faces the challenge of gener-
alizing across robot embodiments (Zhang et al., 2023), which dictate the perceptual modalities and types of actions used for interacting with the environment. These variabilities reflect back on how robots can understand and ground language.
Efforts towards generalizability have been parallel to those addressing data scarcity. Incorporating foundation models allows the agent to benefit from the broad knowledge learned during pretraining (Driess et al., 2023) and leverage the generalization capability of these models (Shah et al., 2023). Multi-task learning and/or training in multiple environments (Wang et al., 2019, 2020; Reed et al., 2022; Brohan et al., 2023; Driess et al., 2023; Jiang et al., 2023) have also been adopted. To generalize across robot embodiments, training on data from multiple robots has been experimented with (Vuong et al., 2023; Brohan et al., 2023; Driess et al., 2023). However, further research is encouraged as positive transfer was reported when robots had similar sensory and action mechanisms (Vuong et al., 2023), or when low-level actuators were trained separately for each robot embodiment (Driess et al., 2023).

heading: 4.3 Simulator Realism
text: With the expense of real-world data collection and robot training, exploiting simulators that mimic world dynamics has been a cheaper alternative (Francis et al., 2022). However, several challenges arise with regard to the realism of simulators (Duan et al., 2022). Fewer simulators have photo-realistic scenes (Chang et al., 2017; Li et al., 2022a) compared to synthetic ones (Kolve et al., 2017; Puig et al., 2018; Wu et al., 2018; Gao et al., 2019; Kim et al., 2020; Gan et al., 2021; Puig et al., 2024). Simulated physics is also usually simplified to basic interactions (Duan et al., 2022), limiting the scope of language that agents can learn to ground. In addition, most simulators (Chang et al., 2017; Puig et al., 2018; Wu et al., 2018; Kim et al., 2020; Li et al., 2022a; Puig et al., 2024) do not include audio or tactile modalities, despite their significant role in language acquisition (Tatiya et al., 2023; Zhang et al., 2023). Despite efforts towards simulating more advanced physics (Seita et al., 2021; Gan et al., 2021; Li et al., 2022a; Fu et al., 2023) and non-visual modalities (Gan et al., 2021; Chen et al., 2022; Gao et al., 2023), only a few corresponding datasets exist (Mees et al., 2022; Gong et al., 2023). Several simulators (Chang et al., 2017; Puig et al., 2018; Wu et al., 2018; Kim et al., 2020) also discretize robot actions and restrict their gran-
ularity (Duan et al., 2022), presenting the risk of models overfitting to simplified dynamics (Francis et al., 2022), unrepresentative of the real world.

heading: 5 Gaps & Future Directions

heading: 5.1 Towards More Stringent Evaluation
text: Whether models have learned to ground language is implicitly assessed using task-related metrics (Li et al., 2022b; Gao et al., 2022; Zitkovich et al., 2023; Jin et al., 2023; Brohan et al., 2023). However, models can learn tasks by overfitting to spurious statistical patterns in their training data (Lake and Murphy, 2020; Zellers et al., 2021; Deitke et al., 2022). Hence, rigorous evaluation and model probing are needed to ascertain what the model has captured (Bender and Koller, 2020).
One avenue to explore is novel concept grounding. Zellers et al. (2021) pretrained their language model on data from which they removed all presence of certain words to assess if the final model learns to ground them. However, pretraining the language model from scratch renders this approach expensive. In Jiang et al. (2023), novel concepts were introduced using dummy labels, but most tested concept categories only required visual grounding. Similar experiments focused on grounding concepts in actions can be valuable.

heading: 5.2 Towards Enhanced Language Modeling
text: The focus of most embodied language learning works has not been on enhancing general language modeling capabilities. Research (Zhang and Chai, 2021; Jang et al., 2022; Brohan et al., 2023; Jiang et al., 2023; Liu et al., 2023) has focused on models that learn to ground language only to execute the respective tasks. In some works (Jin et al., 2023; Mu et al., 2023; Driess et al., 2023), the embodied system incorporates a generative language model that breaks down language instructions into subgoals that are then executed by lower-level control policies. In CogLoop (Jin et al., 2023) and EmbodiedGPT (Mu et al., 2023), however, the pretrained language model is frozen during the end-to-end system training. Hence, its language modeling capabilities do not benefit from the embodied training. In PaLM-E (Driess et al., 2023), while the control policies are used off-the-shelf, the language model is fine-tuned. Nevertheless, a catastrophic forgetting of its general language modeling capabilities was reported upon such embodied fine-tuning, especially for smaller size models (Driess et al.,
2023). There are recent research attempts towards an
end-to-end trained system that can output both robot actions and text, such as Gato (Reed et al., 2022) and RT2-PaLM-E (Zitkovich et al., 2023). However, Gato (Reed et al., 2022) was only qualitatively tested on language generation and reported to exhibit a poor performance. RT2-PaLME (Zitkovich et al., 2023) was end-to-end fine-tuned for chain-of-thought reasoning, but the full range of its language modeling capabilities upon this finetuning was not assessed.
We suggest that an embodied language model should not only retain but also enhance the language modeling capabilities of its disembodied versions. This can then be tested on datasets evaluating figurative language understanding (Liu et al., 2022), physical reasoning abilities (Bisk et al., 2020b; Aroca-Ouellette et al., 2021; Zellers et al., 2021; He et al., 2023; Lanchantin et al., 2023; Li et al., 2023), or general language benchmarks; hence, providing insights into the effect of embodiment.

heading: 5.3 Towards a Full Embodiment Experience
text: Although many attributes and action effects cannot be perceived through vision alone (Tatiya et al., 2023; Zhang et al., 2023), only a few works (Heinrich et al., 2020; Tatiya et al., 2023; Zhang et al., 2023) consider other sensory modalities. From another perspective, an embodied agent’s actions should not be restricted to a predefined set (Bisk et al., 2020a), as is the case in most works (Zellers et al., 2021; Pashevich et al., 2021; Zhang and Chai, 2021; Blukis et al., 2022; Zhang et al., 2023). Agents should freely interact with the environment and acquire new behaviors (Tatiya et al., 2023). However, it was reported that the pretraining and co-fine-tuning of RT-2 (Zitkovich et al., 2023) on vision-language and robotic datasets was still unable to elicit new motions from the agent. With the significance of fully exploiting the embodiment experience (Heinrich et al., 2020), further corresponding research efforts are encouraged.

heading: 6 Conclusion
text: Despite the associated challenges of data scarcity, generalizability, and simulator realism, learning language through embodiment is crucial for establishing the connection between language and the world. With the opportunities it holds for an enhanced understanding of attributes, world dy-
namics and affordances, as well as metaphors and abstract concepts, embodied language learning enables a robust language understanding that is essential for language-supported robots. By identifying current research gaps from a language understanding perspective, this article aims to motivate future efforts towards fully exploiting the embodiment experience and more rigorously evaluating embodied language models for their language modeling capabilities.

heading: 7 Limitations
text: This article presents an overview of embodied language learning from the point of view of language understanding research. Specific details of the robot learning techniques and task-specific evaluation metrics adopted for embodied exploration and embodied instruction following (referred to in §2) are out of this article’s scope. Interested readers are directed to pertinent surveys such as in Francis et al. (2022) and Duan et al. (2022).

heading: 8 Statement of Ethics & Risks
text: Authors do not foresee any ethical concerns or potential risks associated with this work.

heading: A Summary of Embodied Language Learning Works
text: Table 1 provides a brief summary of the approaches adopted and datasets used by the representative embodied language learning works discussed in Section §2.
Ta bl
e 1:
Su m
m ar
y of
A do
pt ed
A pp
ro ac
he s
an d
U se
d D
at as
et s
of R
ep re
se nt
at iv
e E
m bo
di ed
L an
gu ag
e L
ea rn
in g
W or
ks
W or
k A
pp ro
ac h
R ob
ot ic
sD at
as et
E m
bo di
ed E
xp lo
ra tio
n E
m bo
di ed
In st
ru ct
io n
Fo llo
w in g Sh or tH or iz on Sk ill L on gH or iz
on G
oa l
H ei
nr ic
h et
al .(
20 20
) ✓
E M
IL (H
ei nr
ic h
et al
., 20
18 )
Ö zd
em ir
et al
.( 20
21 )
✓ -
Z ha
ng et
al .(
20 23
) ✓
IS py
(T ho
m as
on et
al .,
20 16
), 10
0- O
bj ec
ts (S
in ap
ov et
al .,
20 14
a) ,S
in ap
ov et
al .(
20 14 b) Ta tiy a et al .( 20 23 ) ✓ 10 0- O bj ec ts (S in ap ov et al ., 20 14 a) Ja ng et al .( 20 22 ) ✓ B C -Z B ro ha n et al .( 20 23 ) ✓ R T1 Z itk ov ic h et al .( 20 23 ) ✓ A L FR E D (S hr id ha re ta l., 20 20 ) V uo ng et al .( 20 23 ) ✓ O pe n X -E m bo di m en t Ji an g et al .( 20 23 ) ✓ ✓ V IM A B en ch Su gl ia et al .( 20 21 ) ✓ A L FR E D (S hr id ha re ta l., 20 20 ) H on g et al .( 20 21 ) ✓ R 2R (A nd er so n et al ., 20 18 ), R E V E R IE (Q ie ta l., 20 20 ) Ji n et al .( 20 23 ) ✓ A lp ha B lo ck D ri es s et al .( 20 23 ) ✓ M ob ile M an ip ul at or ,L an gu ag e Ta bl e (L yn ch et al ., 20 23 ), TA M P N ot e. D at as et s w ith ou tr ef er en ce s ar e or ig in al to th ei rc or re sp on di ng w or ks .

heading: Future Work
text: By identifying current research gaps from a language understanding perspective, this article aims to motivate future efforts towards fully exploiting the embodiment experience and more rigorously evaluating embodied language models for their language modeling capabilities