Multitask Multimodal Prompted Training for Interactive Embodied Task Completion
Authors: Georgios Pantazopoulos
Malvina Nikandrou
Amit Parekh
Bhathiya Hemanthage
Arash Eshghi
Ioannis Konstas
Verena Rieser
Oliver Lemon
Alessandro Suglia
Venue: EMNLP 2023 Main (2023)
Keywords: Vision and Language
Embodied AI
Natural Language Interaction

Abstract: Interactive and embodied tasks pose at least two fundamental challenges to existing Vision \& Language (VL) models, including 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation. To tackle these challenges, we propose an Embodied MultiModal Agent (EMMA): a unified encoder-decoder model that reasons over images and trajectories, and casts action prediction as multimodal text generation. By unifying all tasks as text generation, EMMA learns a language of actions which facilitates transfer across tasks. Different to previous modular approaches with independently trained components, we use a single multitask model where each task contributes to goal completion.  EMMA  performs on par with similar models on several VL benchmarks and sets a new state-of-the-art performance (36.81\% success rate) on the Dialog-guided Task Completion (DTC), a benchmark to evaluate dialog-guided agents in the Alexa Arena.

Multitask Multimodal Prompted Training for Interactive Embodied Task Completion
Georgios Pantazopoulos Malvina Nikandrou' Amit Parekh' Bhathiya Hemanthage' Arash Eshghi'.2 Ioannis Konstas Verena Rieser1.2* Oliver Lemon
Alessandro Suglia
l Heriot-Wall University; "Alana Al
(gmp2000, mn2002, i konstas, v. vit.rieser, o. o.lemon, a. suglia} @hw. ac. uk
hsb2000, a. esshhi,
Abstract
observations, and 2) referential disambiguation, where agents must interact with users - often as clarifications - to successfully disambiguate refer- ents. In contrast to language-only planners (Huang et al., 2022a; Min et al., 2021), which cannot gener- ate feasible plans without prompts that encode all the visual nuances colors, attributes, relational information, inter alia as text; instead VLP mod- els can ground language directly to images. How- ever, enable grounded decision-makee makinng VLP models must generalize from static images to tra- jectories that express how a situation evolves over time (McClelland et al., 2020). Additionally, in language-guide collaborative tasks, referents for target objects may be ambiguous, depending on the complexity of the visual scene. Therefore, clar- ifications must be adopted to resolve uncertainty between referents and act upon the correct target object (Madureira and Schlangen, 2023).
Interactive and embodied tasks pose at least fundamental challenges to existing Vision & Language (VL) models, including 1) ground- ing language in trajectories of actions and ob- servations, and 2) referential disambiguation. To tackle these challenges, we propose an Em- bodied MultiModal Agent (EMMA): unified encoder-decoder model that reasons over im- ages and trajectories, and casts action predic- tion as multimodal text generation. By unify- ing all tasks as text generation, FEMMA learns a language of actions which facilitates trans- fer across tasks. Different previous mod- ular approaches with independently trained components, we use single multitask model where each task contributes to goal completion. EMMA performs on par with similar models on several VL benchmarks and sets new state- of-the-art performance (36.81 % success rate) on the Dialog-guides Task Completion (DTC), a benchmark to evaluate dialog-guided agents in the Alexa Arena (Gao et al., 2023).'
To address both of these challenges, we develop EMMA: a unified approach which casts several VL and embodied tasks, such as image captioning and action execution, as text generation inspired by Cho et al. (2021). By doing so, EMMA is able to learn a language of actions which facilitates trans- fer across tasks. Concretely, EMMA is multitask encoder-decoder model, which encodes language and pairs of object-centric representations with sen- tinel tokens. As a result, EMMA can encode both trajectories and clarifications to reference individ- ual frames and objects across an entire trajectory.
1 Introduction
Embodied Al aims to develop agents that interact with the environment, reason over natural language instructions and visual observations, and plan fu- ture actions. Recently, Vision & Language pretrain- ing (VLP) has established new standards across image-based tasks (Bao et al., 2022; Wang et al., 2022a,b; Yang et al., 2021; Zhang et al., 2021) by aligning visual and textual input to generate image captions, answer visual questions, and reason over images. As result, VLP promotes learning of important skills transferable to embodied tasks. Interactive and embodied tasks present signif- icant challenges for VLP models including 1) grounding language in environments where the interpretation of instructions depends not just on static images, but on trajectories of actions and
We focus on dialog-enabled task completion, which requires an agent to follow natural language instructions, perform navigation and object manip- ulation actions, and utilize dialog interactions to disambiguate visual observations. Similar to pre- vious modular approaches (Min et al., 2021), we break down the embodied task into separate pro- cesses for search and action execution. The search process combines exploration with visual ground- ing. Differently to prior work that only uses object
"Now at Google DeepMind "Code available on this link

---

labels from object detectors directly (Min al., 2021: Pashevich al., 2021), EMMA uses both language and visual representations to discriminate between objects of the same class (e.g., discerning between a 'red' and a 'green' desk). Separately, the action exccution process predicts the naviga- tion and object interaction actions as textual tokens. Notably, key distinction in our approach is that we perform all tasks using single VL. model.
et al., 2022; Min al., 2021; Singh al., 2021). State-of-the-ari works employ modular architec- ture that incrementally builds a semantic map of the enviromment used by symbolic planners. However, these methods often make strong domain-specific assumptions, or they are typically trained from scratch, underutilizing recent progress in VLP.
An exciting direction is leveraging large-scale pretrained models. Recent works (Ahn et al., 2022; Dricss et al., 2023; Huang et al., 2022a; Singh et al., 2022b; Zeng et al., 2022) use a large language model or a mixture of pretrained models for object manipulation. Additionally, Huang et al. (2022b); Shridhar al. (2021) use text describe the en- vironment state and employ language model for planning. Different to previous work, we use a VLP model to ground instructions on visual scenes. For example, given the instruction 'Gel the milk from the fridge', EMMA executes the correct ac- tion sequence by inferring whether the fridge door is open or not from its visual observations.
We evaluate our approach on the Stalog-guided Task Completion (DTC) benchmark (Gao et al., 2023). Our experiments show the benefits of the multitask VLP, and the importance of data augmen- tations for completing multi-step instructions. By exploiting clarifications, EMMA improves the suc- cess rate by 3.55%. Finally, when fine-tuned on DTC, EMMA can transfer the downstream action execution task back real-world images, showcas- ing how using text tokens to represent actions in the environment enables cross-domain generalization.
2 Related Work
Phalog-enablee interactive task completion in- troduces an additional layer of complexity as the agent often needs to incorporate clarification con- text before acting on the environment (Gao et al., 2023, 2022; Padmakumar et al., 2022). Both end- to-end and modular architectures underperform in these tasks, which Min et al. (2022) argues is due this is due imitation learning using few and sub- optimal demonstrations. EMMA is trained using expert demonstrations and interactions the form of question-answer pairs about the visual scene. These clarifications do not involve complex dia- log phenomena (e.g., repair), yet are sufficiently informative to disambiguate target objects.
Vision & Language Pretraining Early methods for developing VLP models rely multimodal en- coders with object-centric or patch representations (Chen et al., 2020; Kim et al., 2021; Li et al., 2021, 2020; Lu et al., 2019; Singh et al., 2022a: Tan and Bansal, 2019). While effective, these methods in- troduce task-specific heads for each downstream task. Generative and unified architectures repre- sent an alternative approach for tuckling VL tasks with single model. Previous work on unified models explores region proposals (Cho al., 2021; Gupta et al., 2022), or patch features (Wang et al., b022a,b; Yang et al., 2022). More recently, the paradigm of connecting vision-only and language- only experts allows leveraging unimodal pretrain- ing for generative and multimodal models (Alayrac et al.. 2022: Driess et al., 2023; Li et al., 2023; Liu et al., 2023; Tsimpoukelli et al., 2021: Zhu al., 2023). By casting every task as text gen- eration task, unified VLP models are transitioning from specialized to generalist models. Therefore, adopting a similar multitask pretraining approach enables EMMA to learn a language that describes actions conditioned on visual observations.
3 Task Description
The DTC benchmark evaluates dialog-enabled agents complete missions in the Alexa Arena, simulated environment with multi-room layouts (Gao et al., 2023). Each example corresponds to a mission completed by an expert planner and an- notated by three humans. Each instruction is op- tionally accompanied by a clarification question- answer pair. Navigation is performed with primi- tive actions, but also with the GoTo action, which al- lows moving to different rooms, or objects. Along with the GoTo action, the agent needs to predict the name of the room or an object mask within the current observation. Each room also contains a set of viewpoints that differ between layouts and can
Embodied AI Approaches Previous approaches on interactive task completion (Shridhar et al., 2020) involve both end-to-end (Pashevich et al., 2021; Suglia et al., 2021) and modular architec- tures (Blukis et al., 2022; Kim and Lee, 2023; Liu

---

<pepickup plate
<stogo:44s
Ridirectional Multimodal Encoder
Autoregressive Language Decoder
Embeddings
Object Embbeddings
Tokenizer
crrame_ Lokeee_
tokeeooken_p
Act according to the instruction: pick up plate with pear on it.
Rel
Object
Frame Token
Visual Token ID
Festures
Coordinates
Figure 1: Overview of the EMMA architecture. EMMA encodes vision and language inputs with modality-specife layers before providing the input to the bidirectional multimodal encoder. Scene embeddings act as global features combination region features, object coordinates, and their identities within the frame sequence. Language input is concatenated with task-specifie prompts. The shared autoregressive language decoder treats every task as text generation task.
for frame. Object embeddings correspond
be visited by the agent explore the room. To per- form an action on an object, the agent needs pro- vide the action type and the mask of the object. The set of supported actions for an object is determined by its affordances such as openable, toggleable, inter alia (see Appendix C for details).
sentinel token embeddings. Spatial embeddings code the 2D position of the region within the frame by its normalized bounding box coordinates, while temporal embeddings encode the temporal order within the sequence using frame sentinel token.
The choice between object-centric and patch rep- resentations for embodied tasks remains an open question. Although patch representations contain explicit information about the semantic segmenta- tion of image (Caron al.. 2021), their static grid structure has not been proven beneficial in pre- vious embodied tasks (Driess et al., 2023; Jiang et al., 2023). Furthermore, patch representations lead to an increase in both input length, as more visual tokens are required to represent frame, and output length, the model must generate coordi- nate tokens instead of single sentinel token reference object. For these reasons, opted for object-centrie representations.
4 EMMA
EMMA is an encoder-decoder model following the architecture of BART-base (Lewis et al., 2020). As shown in Figure 1, both vision and language inputs are embedded through modality-speeifii projec- tion layers, concatenated into sequence of em- beddings, and fed into a single-stream encoder. EMMA uses a shared decoder across all pretraining tasks with sentinel tokens in the vocabulary to al- low referencing specific image frames and regions.
Text Embedding For the language input, we ap- ply sub-word byte-pair encoding (Sennrich et al., 2016) with vocabulary of 10K tokens extracted from our pretraining data. Each token is repre- sented by the sum of its word and absolute posi- tional embedding (Lewis et al., 2020). Similar to Sanh al. (2022), we use natural language prompts as task prefixes to prompt the model for each task.
4.1 Pretraining Setup
We formulate seven VL tasks in text-to-text frame- work motivated by established pretraining tasks (Cho et al., 2021; Wang et al., 2022b). Pretraining details are provided in Appendix A. Below give brief description of each task: Masked Language Modeling: Given an image de- scription, we mask each word with probability 0.3. The model must learn to reconstruct the original input by predicting the masked words.
Scene and Object Embeddings EMMA is ca- pable of encoding sequences of images. For each frame of the visual input, we extract global scene features representing the entire frame and max- imum of 36 region features. To reference frame an object within it, the language vocabu- lary is extended with sentinel tokens in the form of <frame_ <frame_token token_ i> and wwsssulllkeee____ projecting the visual features to the encoder's di- mensionality, we add spatial, temporal, and visual
Image-Text Matching: Determine if pair of visual-tes inputs match by predicting true/false after we randomly combine the visual input with either the correct caption or a sampled caption. Visual Question Answering: Answer a question about the input image.
After

---

Subgoai
tndge
dhome -dd McLnnoen
'ind the rable
Imputt
from the retrig geestor does wefigeerm lee? Ae the sitigenaice gray CR: ract-ine, manihe fiige
-fiemne Inieen, vou tonon bi.
<faamee jicken, 24 2i
ofame, Jocken
loksn salopr.
CR: vastimione mazche
bool
AB pour bow colan,
Figure 2: Example trajectory where the objective is to pour cereal into the bowl. The user instructions and agent response are shown on the left and right of each frame respectively. At each timestep the agent is provided with the current view, a user instruction (Input), Routing task (CR) determines whether the agent is going to use the Action Execution task (AE) to interact with the environment, or the Visual Grounding task (VG) to search for an object in the scene.
optionally a clarification question (Q) and answer (A). The Contextual
(Dense) Captioning: Produce a textual description of the overall image specified region denoted by a visual sentinel token.
matches>. The input to the model consists of the current frame, the instruction, and the clarification pair. We use the <follower> and <commander> tokens as prefixes each dialog turn. The model predicts the action type, the name of the object, and, for manipulation actions, the frame and visual token id that correspond to the object (Figure I), Each action yields a new observation concatenated with the previous visual context to make new prediction. The period character (f.") delimits tions, and the <stop> token marks the end of the trajectory for that instruction.
Visual Grounding: Given a description of an im- age region, the model must predict the visual token for the region that matches this description.
Relationship Prediction: Describe the relation- ship between two regions of an image. The output follows the format: Subject Attributes, Subject, Re- lationship Predicate, Object Attributes, Object.
4.2 Interactive Task Completion
Our agent uses three distinct yet complementary tasks to complete the DTC missions: Contextual Routing (CR), Action Execution (AB), and Visual Grounding (VG). As shown by Figure we use a single model to tackle all three tasks. Since CR & AE are new prompts when fine-tuning.
Visual Grounding (VG) For <search> and <act><no match> predictions, the agent tries to find the object outputted by the CR task. We iter- ate viewpoints in room to collect panoramic views and the pretraining visual grounding pre- fix with the instruction input the model (Ap- pendix D). If the instruction matches with visual token, the model outputs that token; else, the model outputs 'no OBJECT'. For <act><no match>, once the object has been found, we invoke the AE task.
speccifc tasks, we introduce
Contextual Routing (CR) Given the current in- struction and observation, the CR task determines whether the agent should act search for an object, disentangling the decision of what do next from how achieve it. The output of the model follows a structured hierarchical scheme (Figure 3). The first token determines whether the instruction refers to an action or to search for an object. The sec- ond token indicates if there are no objects present, one object present, or multiple matches of the ob- ject mentioned in the instruction. Finally, for non- navigation actions, the model generates the name of the target object.
5 Data Augmentations
Since the trajectories have been generated by an expert planner, the predictions <search> and <act><no match> used by the CR task are under- represented in the training data. Training for VG is unsupported DTC instances only contain the ground-trutt object mask for each action but not for all other objects in scene. Furthermore, prelimi- nary experiments revealed that, when trained solely on instances from the DTC benchmark, our agent struggled to learn associations between frame and visual tokens. For example, when interacting with a fridge across multiple frames, the agent sometimes
Action Execution (AE) The agent acts on the environment the output of the CR task is <act><one match> or <act><multiple

---

+
orefts prompt
bum compute
vacte
<onc motche
montor
tretl promer
uAAT compular
sorw melele
orefis promps
tur the compuleer
EMMA
VacEe
compuler
tretu promer
Turn the marhing
Timn machinn
refix promor
gC bnwalrosam
adb
40h matchh
Figure 3: Example outputs for the Contextual Routing (CR) task. The first token determines whether the agent will search or act on the environment. The second token determines if the instruction matches an entity in the scene. Finally, for instructions that require manipulation of objects, the model generates the name of the object.
predicted visual tokens that matched the fridge in a previous rather than the current frame. Finally, there is a mismatch between how the trajectories were annotated and the data the agent is being ex- posed to. While annotators observed interpolated trajectories, the agent only perceives the state be- fore and after an action. This discrepancy adds sig- nificant language variability and complexity, pos- ing challenges to the agent's 's comprehension.
these trajectories as CDF augmentations follow- ing the challenge definition format of the Alexa Arena (Gao et al., 2023). Overall, we sampled 38k missions for training and 15k for validation. These missions include: 1) picking up and placing objects from receptacles containers; 2) picking up ob- jects under the presence of objects acting as distrac- tors (i.e., the same object but with different color, or the same object placed in a different position); 3) cleaning and filling objects in the sink; and 4) interacting with domain-specite objects (c.g., the color changer and the time machine).
To address these challenges, we use the vision dataset provided by Gao et al. (2023) to create syn- thetic instruction data. Additionally, we deploy an early version of our agent to collect mini-episodes of successful trajectories within the Alexa Arena. Supplementary material regarding the data genera- tion process is provided in Appendix E.
Experiments & Results
6.1 VLP Evaluation
We evaluate EMMA on four image-based down- stream tasks including image captioning, visual question answering, referring expression compre- hension, and natural language for visual reasoning. We treat all tasks language generation and fine- tune EMMA separately on each downstream task without introducing any task task-specific parameters.
Starting from the images within the vision dataset, we create an additional 180k for training and 88k instances for validation, where each instance is an (image, instruction, ac- tion) triplet. When generating target actions, used the ground truth positions of the agent and the object to ensure the agent was in proximity. For synthetic instructions, we used ChatGPT (OpenAI, 2022) as a starting point to create templates. To select diverse visual scenes, we used CLIP (Rad- ford et al., 2021) to embed the images. We then applied k-means clustering, where the number of clusters is determined by the maximum number of action-objes instances. Finally, we selected the image that is closest to each centroid.
Visual Augmentations
To evaluate the ability of our model generate image captions, we use the MS-COCO dataset (Lin et al., 2014) and report evaluation results on the Karpathy test split (Karpathy and Fei-Fei, 2015) for BLEU-4 (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), CIDEr (Vedantam et al., 2015), and SPICE (Anderson al., 2016). For visual question answering, we report the VQA accuracy (Antol et al., 2015) on the test-std set of the VQA-v2 dataset (Goyal et al., 2017).
CDF Augmentations While the visual augmen- tations are useful for single-step interactions, they cannot help the model on longer trajectories requir- ing consecutive interaction with the environment. To address this challenge, variants of EMMA collect more training trajec- tories. For each mission, we provide the agent with step-by-step instructions and only include success- ful trajectories our training data. We refer to
We also evaluate referring expressions compre- hension on RefCOCOg (Mao et al., 2016) a visual grounding task that requires selecting the region described by given phrase. We the visual grounding prompts from our pretraining and ask the model generate the visual token id that matches the given phrase in the image. A predicted region is correct if the intersection over union with
one of the early

---

# Pretrain
COCO Captioning
VQA-22
RefCOCOg Avcuracy @0..5
NLVR Accuracy
Sumples
Params
BLBU-4
METEOR
CIDEr
SPICE
Accuracy
VL-T5 (Cho et 2021)
7.6M
172M
34,5
28.7
116.5
21.9
70.3
71.3
73.6
VL-BART (Cho et 2021)
7.6M
172M
25.1
28.7
116.6
21.5
71.3
22.4
70.3
UniTAB (Yang al, 2022)
& 1M
211M
36.1
28.6
119.8
21.7
71.0
84.5
OFA-hase (Wang et al., 20226)
21.3M
182M
41.0
30.9
138.2
24.2
78.1
82.3
EMMA
10.2M
133M
36.5
29.7
122.3
22.5
73.2
80.3
70.3
Table I: Performance of the pretrained model on downstream image-based tasks. We report the number of pretraining samples as the number of image-text pairs. OFA uses additional vision-only and language-only data.
the ground truth region is larger than 0.5.
MSR
NRA
QA
In the NLVR dataset (Suhr al., 2019), given a caption and two images, the model must predict whether the caption matches both images. We pro- vide the two images directly to the model and prefix the caption with the image-text matching prompt. The model then predicts either true or false.
Leaderbourd:
GauchoAl
SEAGULL
Kingfisher
22.37
Baseline:
12.73
VL (Goo et al., 2023)
18.19
11.82
X
Table illustrates the results of our model. We compare EMMA against similar VI. models i.e., single-stream mncoder-decoder models like VL-T5 and VL-BART (Cho et al., 2021), UniTab (Yang et al., 2022), and OFA (Wang et al., 2022b). Among these models OFA- base achieves state-of- the-art performance, however, it has 1.37x more parameters than FMMA and is pretrained with nearly double the amount of data. On RefCOCOg, our model achieves competitive performance with UniTAB, even though it is almost half the size. Cho et al. (2021) hypothesize that the reason for the poor performance of VL-BART on RefCOCOg is that the of absolute positional embeddings leads memorization during training. However, EMMA achieves competitive performance despite using absolute positional embeddings. We believe this due to our sampling procedure that takes into account the dataset imbalance in pretraining (see Appendix A for details).
VI. (Gio et al., 2023)
34,20
8.82
x
EMMA modular
33.95
9.05
CR
EMMA-moodalar
35.16
8.92
V
EMMA-anifted
33,26
8.79
X
EMMA-miiied
33.59
8.89
CR
EMMA-amified
36.81
Table Mission Success Rate (MSR) and Num- ber of Robot Actions (NRA) of EMMA against top-3 leaderboard and baseline models. CR denotes QA based on the CR task <aat><multiple matches>).
We also compare the success rate under three conditions for the inclusion of clarifications: 1) no clarifications, 2) clarifications whenever they are available for an instruction, 3) clarifications whenever they are available, and the CR predicts an ambiguous referent, that is bact><multiple matches> is predicted. We observe that both ver- sions of EMMA improve when clarification ex- changes are added to the input. Performance gains are marginal in the CR case, however, the agents benefit substantially when clarifications are always provided. Note that the role the CR task not to decide when to clarify but to drive the agent to either act or search the environment. On top of that, during training, the agent learns to rely on all available clarification types.
6.2 Interactive Task Completion
We compare two variants EMMA: a modular agent with two experts that are each fine-tuned on the downstream-specific tasks (i.e., CR & AE), and a unified model fine-tuned with all the tasks. As shown in Table 2, both agent variants outperform the baseline models. Additionally, the modular agent performs par with the best-performing models on the leaderboard, while the unified agent slightly outperforms the state-of-the-art. We be- lieve that the synergy between tasks provides an advantage to the unified model.
Performance per Mission Type Table 3 shows the performance of the modular and the unified agents in every mission category along with the average number of ground- truth actions required

---

EMMA modular
EMMA unified
Average
EMMA modalar
EMMA unified
of ground
QA
QA
QA
QA
truth actions
QA X
QA
Gain
QA
QA
breskObject
Description
66.40
+7.35
+3.14
elean&deliver
13.79
18.39
Direction
76.56
43.13
coloe&deliver
0.00
0.00
0.00
0.00
14.00
Location
62.27
66.45
+4.18
64.27
66.66
+2.39
fill&deliver
29.17
29.17
27.08
27.08
12.87
Reference
59.05
64.09
+5.04
62.02
66.56
+4.54
dellvedelliver
12.50
16.67
20.83
20.83
15.75
Other
75.00
75.00
+0.00
79.41
79.41
40.00
hen&ddlivvrr
5.13
7.69
10.26
10.26
16.92
insertaDeviie
37.29
38.42
39.55
45.76
7.44
Table 4: Object localization accuracy per question type for the modular and unified model. QA X columns refer the performance of the model on the same instructions but without clarification.
pickup&deliver
pouronnaanne
18.95
47.86
41.03
46.15
8.33
repaär&deliver
11.11
20.37
12,96
22:22
17.05
scanObjoct
69.37
63.06
63.96
62.16
3.91
inggleDevice
67.62
68.57
62.86
65.71
3.94
Table 3: MSR of EMMA for each mission category.
and without clarification. Table illustrates the localization accuracy for the modular and the uni- fied model across question types. We use the same taxonomy Gao et al. (2023) with the addition of 'other', which refers to out-of-scope clarifica- tions (e.g, 'What should deliver', while the agent already holds the mug and instructed to place on the desk). Both models benefit the most from description clarifications (e.g.. 'What does the cartridge look like?', 'It is black with a white label'). The modular agent benefits more from reference and location clarifications, while the uni- fied agent benefits from reference and direction clarifications. We hypothesize that the transfer be- tween tasks enables the unified agent to perform well on instructions even without the location clar- ifications, thereby the clarifications provide only small improvements.
Performance curves ablating sugmentations
38
Vidtr 2i3mertzthie 2ettee
32
32
26-
26
Aug
Aug
20
20
0.0
0.25
0.5
D.75
1.0
Proportion of train Instances
Figure 4: MSR against data ablations. The DTC line the performance of the model after training exclusively on the DTC data, without any augmentations. Since the visual grounding task is not modeled by the DTC benchmark, we keep the grounding instances and only downsample the action execution instances.
Data Ablations We also provide ablations in the dataset that showcase the effectiveness of our aug- mentations by ablating the visual as well as the CDF augmentations. Figure 4 depicts the perfor- mance of the unified model for both ablations. We observe that the model benefits from both aug- mentations. Vision augmentations provide perfor- mance gains from very early stages since they en- able certain properties of the agent (e.g., <act><no match>). On the other hand, CDF augmentations provide a steady boost as they enable the agent to solve missions that require longer trajectories. Note that pretraining consists of image-caption objectives that enable the model to learn ground text in singular frames. Learning frame-visual ken associations obtained during fine-tuning only. Future work could explore agents pretrained on tasks that favor learning this ability as well.
by the expert planner solve the mission. Both agents perform well on small trajectories, whereas for longer trajectories the unified approach yields better performance. Clarifications play a key role across all mission types with the exception of scanObject missions, a behavior which is also shown in baseline models (Gao et al., 2023). Inter- estingly, both agents are completely unable to solve color&deliver missions. Based on the predictions of the model, the main challenge is finding the cor- rect receptacle at the end of the trajectory, which can be easily confused with other similar objects.
Impact of Clarification Type Next, we are in- terested in identifying which clarifications help the model disambiguate object referents. Similarly to Ehiyah-Garde et al. (2022), we compare the ob- ject localization performance of the model with

---

Instruction: Put some coffoe in the cup. Output: pour mug <trame tolenn 1-- "Viz. token. peen_17-- <stop..
Instruction: Lift the piata with the bread on Outputi picieup plate <trame token.. <vis.roken_.2 2+ rssop>.
Instruction: Can you the spoon into thes boed??
Instruction: open up the metallic coant Output: open jar <trame totttoooe.... Is "vis_roken, estops.
Output: place bowl strame -trammetoken.... token..I> "vis_token. La estopp.
Figure 5: Example transfer of the action prediction task to the visual domain of real images.
7 Performance Analysis
Objoct Detector
Arena Classes
Non-Arena Classes
Oveall
Fine tuned
42.65
32.13
33,88
DTC Error Analysis We inspected the trajecto- ries of our unified model for 120 unsuccessful tra- jectories (ten per mission type). Overall, the main source of errors stems from the output of the contex- tual routing component. This task plays a key role in our model since an incorrect <act><one match> prediction triggers the action execution task. In the follow-up timestep, the agent is positioned differ- ently from what is expected and it is forced to act on the environment, which likely leads to unre- coverable situation. For instance, the agent may be facing empty desk while being asked to pick up mug. Consequently, the agent acts unpredictable ways, resulting state from which the agent can- not recover. The incorrect predictions affect mostly <x>&del iver missions, where the agent completed the <x> subgoal, however, failed on the del iver subgoal as it was unable to disambiguate between multiple receptacles.
Base
67.85
58.6
60.35
Table 5: Action prediction accuracy for real images from Visual Genome (Krishna et al., 2017).
detector, the EMMA-unified model is able make reasonable predictions as shown in Figure To quantify the improvement of using the base object detector, we use scene graph annotations of GQA (Hudson and Manning, 2019) by creating synthetic instructions, as described in the Appendix H.
Table 5 shows the accuracy for single-step in- structions using real images where the target ob- ject can be from any class or the subset of classes that appear in Alexa Arena. Although EMMA- unified has been fine-tuned on the action predic- tion task with image inputs from the finc-tuned object detector, we see 26.48% absolute improve- ment when switching to the base object detector. We leave further exploration of the potential for Sim2Real transfer future work.
Furthermore, the output of the action execution task is also susceptible to errors. In long trajec- tories, we observed a few cases where the model seems lack temporal understanding. For exam- ple, when interacting with objects on consecutive timesteps the model used visual token to refer- ence the object that matched one the previous frames in the trajectory. We also noticed a few er- ror cases due to a lack commonsense reasoning. Some instructions describe properties of an object, like its color and shape without explicit reference to the object but our agent could not make the asso- ciations between object and its properties.
8 Conclusion & Future Work
We described EMMA, unified and multitask model for embodied task completion. EMMA performs par with VLP models of comparable size in VL benchmarks, and also achieves new state-of-the-ar performance on the DTC bench- mark (Gao al., 2023). We decouple the down- stream embodied task into three complementary tasks all solved by the same unified model. Finally, EMMA benefits from interaction with users in the form of clarifications that are crucial for the agent's 's success, particularly for referential disambiguation. In terms of future work, our experiments show that trajectory augmentations are required in order to learn frame and visual token associations. How- ever, acquiring additional data might prohibitive
Visual Domain Transfer
We inspect the model's ability transfer the action prediction task real images. We observe qualitatively that after fine- tuning the object detector struggles to detect objects from classes that are not present in the Alexa Arena. However, after reverting back to the base object

---

for target applications where an environment sim- ulator is not available. This requirement could be alleviated by introducing the concept of trajectories in the pretraining. Furthermore, given our visual domain transfer analysis, would be interesting to explore if visual encoders specialized to vari- ous target domains can be plugged into the same multimodal model to allow cross-domain transfer.
uses, and environmental harms (Bender al., 2021; Dinan et al.. 2022; Weidinger et al., 2022).
Furthermore, their multimodal nature creates ad- ditional privacy concerns, especially when devel- oping systems for assistive home settings, as is the case in EMMA. In order to address the need for large amounts of in-domain data, EMMA is de- veloped using simulated environments. However, transferring the learned behavior real-world en- vironments is still an open challenge. For example, VLP models can be prone to distribution shifts, c.g. through naturally occurring noise (Yu et al., 2023), or targeted attacks (Yu and Rieser, 2023). Consequently, embodied agents based on VLP may exhibit unintended behaviors when operating in complex real-world environments. These failures especially grave when interacting with vulnera- ble groups, is the case for care assistants.
Limitations
We focused on the development of dialog-enabled agent within an embodied environment. Our design aimed at creating an agent that as autonomous as possible without relying on external components to support it. We took one step away from modu- lar agents with individual components to a single model that facilitates all tasks. Nevertheless, the search routine is an external component that our agent depends upon. In theory, agent could learn low-level actions in order to search for an object. However, this is challenging problem that would require an excessive amount of demonstra- tions even for simple instructions. Consider the case where the model has to search within mul- tiple containers (e.g., cabinets) for the required object (e.g., a spoon). This would require the agent to learn to summarize the state from the context of history and recognize which cabinets it has al- ready visited before opening the next cabinet. Fi- nally, as shown by our results, our agent does not fully model dialog actions. We used the output of the contextual routing task to request clarifications, however, we did not emphasize on the type of clar- ifications when making request. An interesting follow-up work would be to explore how an agent further incorporate dialog actions similar to the ones supported in the environment and pose clarification requests when necessary.
It is therefore paramount that these models are re- leased in responsible way (Bergman er al., 2022), developed, and tested in ecologically valid setups in order to anticipate real-world impact (De Vries et al., 2020). Addressing these issues responsibly and proactively is essential to maximize the bene- fits while minimizing the potential negative impacts individuals and society whole.
Acknowledgements
We would like to thank the Alexa Prize team and Amazon Science for their crucial technical and ad- ministrative support, along with the resources that greatly contributed towards model development. This work also used the Cirrus UK National Tier-2 HPC Service EPCC funded by the University of Edinburgh and EPSRC (EP/P020267/1).
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do as i cam, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691
Ethics Statement
Embodied AI and physical robots have some well- known ethical trade-offs. On the one hand, they can generate a positive impact, e.g. as part of care settings or helping in disaster zones. On the other hand, they can also lead to negative socio- technical consequences, such job displacement dis- tisseempowrreent of individuals (Müller, 2021). Embodied agents based on large-scale VLP in- herit additional challenges - some of which they share with large language models, including hallu- cinations, discrimination and exclusion, malicious
Jean- Man-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Newral Information Processing Systems, 35:23716-22 -37366.....
Peter Anderson, Basura Femannd, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic proposi- tional image caption evaluation. In Computer Vision-

---

ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, 2020. Uniter: Universal image-text representation learning. In Computer VCCi 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX, pages 104- 120. Springer.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015, Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Francisco Javier Obiyah-Garcia Alessandro Suglia, José Lopes. Arash Eshghi, and Helen Hastie. 2022. Exploring multi-modal representations for ambiguity detection & coreference resolution in the simme 2.0 challenge. arXiv preprint 22Xi:2202..16455
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subho- jit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of- mrodality-xxperti Advances in Neural Information Processing Systems, 35:32897-322 32912.
Jaemin Cho. Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and- -an--aggg-- tasks via text genera- tion. In International Conference on Machine Learn- ing. pages 1931-194 PMLR.
Emily M. Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Mod- els Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountabilidy and Trans- parency, FAceT '21, pages 610-623. Association for Computing Machinery.
Harm De Vries, Dzmitry Bahdanau, and Christopher Manning. 2020. Towards ecologically valid re- search on language user interfaces. arXiv preprint arXiv:2007.1444
Emily Dinan, Gavin Abererombie, Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser, et al. 2022. Safetykit: First aid for measuring safety in open-domain conversational systems. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers). Association for Computational Lin- guistics.
A Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, Verena Rieser, et al. 2022. Guiding the release of safer e2c conversational ai through value sensitive design. In Proceedings of the 23rd Annual Meeting of the Spe- cial Interest Group on Discourse and Dialogue. As- sociation for Computational Linguistics.
Danny Driess, Fei Xia, Mehdi SM Sajaadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033:
Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A Knepper, and Yoav Artzi. 2018a. Following high- level navigation instructions on a simulated quad- copter with imitation learning. arXiv preprint arXiv:1806.00047.
Qiaozi Gao, Govind Thattai, Xiaofeng Gao, Suhaila Shakiah. Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjie Shi, Bofei Yang. Desheng Zheng, et al. 2023. Alexa arena: A user-centric inter- active platform for embodied ai. arXiv preprint arXiv:2303.01588
Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. 2018b. Mapping navigation instructions to continuous control actions with position-visitatinn prediction. In Conference Robot Learning, pages 505-518. PMLR.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre- rreitrained language models better few-shot learners. In Joint Conference of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACI-IJCNLP 2021, pages 38 16-3830. Association for Computational Linguistics (ACL).
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language instruc- tion execution. In Conference on Robot Learning, pages 706-717. PMI.R
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties self-sur supervved vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. 2022. Di- alfred: Dialogue-enabled agents for embodied in- struction following. JEEE Robotics and Automation Letters, 7(4):10049-1005
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. 2020. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Pro- cessing Systems, 33:4247-422
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the

---

IEEE Conference on Computer Vision and Pattern Recognition. pages 6904-666 6913.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceed- ings of the second workshop on statistical machine translation, pages 228-231.
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2022. Towards general pur- pose vision systems: An end-to-end task-agnostic vision-language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 16399-164099
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising -oreeucee---eeeence pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the A Asooiiaiion for Computational Linguistics, pages
Xiaotian Han, Jianwei Yang, Houdong Hu, Lei Zhang, Jianfeng Gao, and Pengchuan Zhang. 2021. Image scene graph generation (sgg) benchmark.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a. Language models zero- shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning. pages 9118-1998 11-91147. PMLR.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprini arXiv:2301.12591
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence. Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through plamning with language models. arXiv preprint XXix:220700560
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705.
Drew A Hudson and Christopher D Manning. 2019, Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2700-6709.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang. Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Objent-semantiss aligned pre-training for vision languaee tasks. In European Conference on Computer Vision (ECCV).
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, An- ima Anandkumar, Yuke Zhu, and Linxi Fan. 2023. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 740-755. Springer.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128- 3137.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint 2rXiv:3304.08485
Haoyu Liu, Yang Liu, Hongkai He, and Hangfang Yang. 2022. langulag expectation & binding pol- iey: A two-stream framework for embodied vision- and-language interaction task learning agents. arXiv preprint mmxiv:2203.0623
Dahyun Kim and Yong-Ju Lee. 2023. Improving bodied instruction following with deterministic meth- ods. In 2023 25th International Conference on Ad- vanced Communication Technology (ICACT), pages 424-428. IEEE.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- representt: tic representations for sadion-and-langueee tasks. In Advances in Neural Information Processing Systems. pages 13-23.
Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: lasion-and-languaal transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning, pages 5583-5594, PMLR.
Brielen Madureira and David Schlangen. 2023. Instruc- tion Clarification Requests in Multimodal Collabo- rative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 2303- 2319, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision, 123:32-73.

---

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 11 11-20.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555)
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero- shot task generalization. In The Tenth International Conference on Learning Representations.
James I. McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. 2020. Placing lan- guage in an integrated understanding system: Next steps toward human-level performance in neural lan- guage models. Proceedings of the National Academy of Sciences, 117(42):259662255 25974.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725.
So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021. Film: Following instructions in language with modular methods. In International Conference on Learning Representations.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2556-2565.
So Yeon Min, Hao Zhu Ruslan Salakhutdinov. and Yonatan Bisk. 2022. Don't copy the teacher: Data and model challenges in embodied dialogue. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9361 9368, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozheh Mottaghi, Luke Zeitlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10740-10749.
Vincent C. Müller. 2021. Ethics of Artificial Intelli- gence and Robotics In Edward N. 7alta, editor, The Stanford Encyclopedia of Philosophy, Summer 2021 edition. Metaphysics Research Lab, Stanford Univer- sity.
Mohit Shridhar, Xingdi Yuan, Mlee-Alex Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. Alfworld: Aligning text and em- bodied environments for interactive learning. In In- ternational Conference on Learning Representations.
OpenAI. 2022. Introducing ChatGPT.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan-Chen, Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek -ur 2022. Teach: Task-driven em- bodied agents that chat. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2017-2025.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami. Guillaume Couairon, Wojeiech Galuba, Mareus Rohrbach, and Douwe Kiela. 2022a. Flava: A foun- dational language and vision alignment model. In Proceedings of the IEFF/CVF Conference on Com- puter Vision and Pattern Recognition, pages 15638- 15650.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu, 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting on association for compu- lational linguisticx, pages 3-18-1 Association for Computational Linguistics.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox. Jesse Thomason, and Animesh Garg. 2022b. Prog- prompt: Generating situated robot task plans using large language models. In Workshop on Language and Robotics at CoRL 2022.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for vision-and- language navigation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15942- 15952.
-sioo-and-language
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. 2021. Factorizing perception and policy for interactive in- struction following. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1888-1897.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8768 PMLR.
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. 2021. Em- bodied bert: A transformer model for embodied,

---

language-guide visual task completion. preprint 1rXiv:2108.0492.
arXiv
outputs for grounded vision-language modeling. Computer Eccio-ECCC 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part XXXVI, pages 521-539. Springer.
Alane Subr. Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Annual Meet- ing of the Association for Compulationl Linguistics, pages 6418-6428, 6422, Florence, Italy, Association for Computational Linguistics.
Lu Yu, Malvina Nikandrou, Jiali Jin, and Verena Rieser. 2023. Quality- -agnosti image captioning to safely as- sist people with vision impairment. In International Joint Conference on Artificial Intelligence.
Lu Yu and Verena Rieser. 2023. Adversarial robustness of visual dialog. In Findings of the Association for Computational Linguistics (ACL).
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNIL)), pages 5103-5114
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, et al. 2022. Socratic models: Com- posing zero-shot multimodal reasoning with lan- guage. arXiv preprint arXiv:2204.0083
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi. SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Mul- timodal few-shot learning with frozen language mod- cls. Advances in Neural Information Processing Sys- lems, 34:200-212.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5579-5588.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4566-4575.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4; Enhancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022a. Git: A generative image-to-text transformer for vision and language. Transactions of Machine Learning Research.
Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. Ofa: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sequeee se-queeee learning framework. In Inter- national Conference on Machine Learning, pages 23318-23340 PMLR.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang. John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks posed by Language Mod- els. In 2022 ACM Conference on Fairness, Account- ability, and Transparency, pages 214-229. ACM.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021. Crossing the format boundary of text and boxes: Towards unified vasion-language modeling, arXiv preprint arXiv:211 12085.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Li- juan Wang. 2022. Unitab: Unifying text and box

---

A Pretraining Details
apply linear learning rate schedule with warm-up for 10K sleps and a maximum learning rate of lc-4. The pretrained model was trained on 8 NVIDIA Tesla V100 GPUs.
Vision Encoder For encoding images, we use
VinVI. (Zhang et al., 2021), as it has showed strong performance on established VL benchmarks, keep- ing it frozen for pretraining and only fine-tuning it on the Alexa Arena.
Vision Encoder For the Alexa Arena, we fine- tune the VLn pretrained checkpoint for 300K steps with batch size of 4. We set the base learn- ing rate to 10 and weight decay to 10 with SGD optimizer, decaying the learning rate by 0.1 after steps 55K and 75K steps. During training, we use the default preprocessing and image transfor- mations (Han et al., 2021). The model is trained on 4 RTX 2080 Ti GPUs.
Pretraining Dataset We pretrain our model on publicly available VI. datasets including COCO captions (Lin et al., 2014). Conceptual Captions 3M (Sharma et al., 2018), GQA (Hudson and Man- ning, 2019), VQA (Goyal et al., 2017), and Visual Genome (Krishna et al., 2017). Table 6 shows the statistics for our compiled pretraining corpus. Overall we used 10.4M examples 3.3M different images. We used custom task-specific prefixes for each task. Table 7 illustrates example onput-output pairs for each pretraining task, while Table illus- trates the prompts used for each task.
Fine-tuning EMMA VL downstream tasks For all tasks, we fine-tune the pretrained model using LM loss for up 20 epochs.
Fine-tuning EMMA on Alexa Arena A key fea- ture of EMMA is the ability to predict 'negative' output for the contextual routing and the visual grounding task. For example, the agent output <act><no match> apple ing to pick up or find an apple. Since the DTC data do not include these types of outputs, we use visual and CDF augmentations to simulate these instances. In particular, with a probability 50% we convert a 'positive' instance (an instance that the synthetic instruction matches with the visual scenc), into a 'negative' one by selecting an image from the train set where the target object is missing.
Model Pretraining Strategy During pretraining, we apply teacher forcing and compute the cross- entropy loss between the predicted and target token. As the typical length of the target prediction varies per task, we ensure the losses across tasks are com- parable in scale by averaging the loss by the target sequence length first and then by the number of samples in the batch.
apple when try-
We employ a 'mixed batches" pretraining scheme where each batch contains examples sam- pled from any task. Assuming that task é has n, examples, the probability of sampling an example from task from all j tasks is pi na/ E, n-j- However, as shown in Table 6, the pretraining tasks have a large variance in the number of avail- able examples, which can lead to poor perfor- mance on low-resource tasks. Therefore, simi- lar to Raffel et al. (2020), we re-adjust the prob- ability pi by limiting the maximum number of examples allowed per task. The limit is con- trolled by a ratio R. which defines how many more samples are included in training task i ver- sus the task with the smallest quantity of exam- ples memin Therefore, Vj tasks, the final probabil- ity of sampling an example from task i becomes P = min(n,, R x Timin)/ / E; min(my) (n., , R x Temin). In our experiments we set R = 3.
For the modular and unified models, we fine-tune our pretrained model for 10K steps, using cross- entropy loss and teacher-forcing. We use a batch size of 256 and the A optimizer with learn- ing rate 10-4, weight decay 0.01, a linear learning rate schedule with IK warmup steps, and 0.1 label smoothing. We shuffle the identities of the visual tokens for cach frame. The model is trained on single RTX 2080 Ti GPUs.
C DTC Benchmark
The DTC benchmark contains 2661 missions training and 383 in validation. Each mission is an- notated by three separate annotators. Each human annotation corresponds to single episode, mean- ing that there are 7983 training episodes and 1149 validation episodes.
B Implementation Details
Primitive navigation actions include MoveForward, MoveBackward, RotateLeft, and RotateRight. To collect panoramic views from its position, the agent can perform the LookAround
Pretraining Setup We pretrain our model for 100k steps using a batch size of 2048 and the AdamW optimizer with weight decay of 0.01. We

---

Dataset
Images
# Samples
Tasks
VQA-x2 (Geyal al,, 2017)
83K
VQA
GQA (Hadson and Manning, 2019)
86K
987K
VQA
COCO Cuptioning (Lin et al., 2014)
11SK
592K
MLM. ITM, Captioning
Conceprual Captions (Sharna et al. 2018)
3M
3M
MLM, ITM, Captioning
MLM. Dense Captioning. Visual
Visual Genome (Krishma et al., 2017)
108K
5.4M
Grounding, Relationship Detection
Total
3.3M
10.4M
Table Dataset statistics for pretraining.
Task
Example ingut
Example target outpat
MLM
Denoise: Fridge <MAS is open
Fridge door is open
ITM
Assess the statement: Fridge door is open
True
Captioning
Describe the image
Food inside refrigeratoe with its duor open
Donse Captioning
Doseribe objoct <vl sual_token_.>>
Silver fridge
Visual Grounding
Locate the milk
taken_3>
color cabinets?
White
Relationship Detection
Explain how crissull_toen,,
wvisuul__tkenn
inside fridge
Table 7: Example input and output formats used for the pretrained tasks.
action. The agent also performs manipulation actions on each object which include PickUp, Place, Open, Close, Toggle, Fill, Clean, Pour, Break, and Scan. The set of supported actions for each object are determined based on its affordances. In total, there are 14 affordances: pickupable, openable, breakable, receptacle, toggleable, powerable, dirtyable, heatable, eatable, chillable, fillable, cookable, decor, and infectable.
number of redundant steps. To mitigate this, the agent selects subset of viewpoints. We assume that the original agent position and each viewpoint can cover an area of fixed radius. This way, we create a graph where each candidate position a node and add edges between nodes whose areas overlap. As a result, we turn the viewpoint selec- tion into the Maximum Vertex Coverage problem and apply greedy algorithm. We empirically set the radius to four, which leads selecting up two viewpoints depending on the room size. After preparing the search plan, the agent starts executing step-by-step. We use the visual grounding task to localize a referenced object within each new frame. The agent continues to execute the search plan until the object is found or the plan is exhausted.
D Search Routine
We focused the development of a dialogue- enabled agent within embodied environment. Our design aimed at creating an agent that is as autonomous as possible without relying on exter- nal components to support it. Nevertheless, the search routine is an external component that our agent depends upon. Learning how to search for an object is challenging problem that would require excessive amount of demonstrations.
To minimize the number of search steps, we also keep track the objects observed at each position. If any of the object labels from the vision model is not present in the memory, we create a new entry that stores the object label, its bounding box area, and the closest viewpoint. Since we are not using depth estimation, approximation of distance. The memory is queried at the beginning of the search routine. If the target object memory for the current room, the agent will consider the retrieved viewpoint as the starting position for the search.
The search pipeline is triggered when the output of the contextual routing task requires the agent to search for an object or to interact with an object that is not in the agent's view. The agent searches the current room by iterating through selected view- points - including the agent's 's original position. At each viewpoint, we collect panoramic view by rotating left by 90 degrees three times.
used the
A limitation of our setup is that our routine does not keep track of the spatial positions of an object. Some objects may be inside containers like a fridge,
Since each room has a maximum of eight view- points, the search routine could amount to a large

---

Task
Prompts
"Denoisc: [caption)"
"Dennise the statement: [caption]"
"Densise the description: [suption]"
"Reconstruct: [caption)"
'Reconstruct the description: (caption)"
"Reconstract the statement: [caption]
ITM
statement: [statement]
"Assess the description: [statement]*
"Evaluate the statement [statement]"
"Evaluatic the description: (statement)"
Captioning
"Caption this"
"Caption the image" "Caption image" "Deseribe this"
"Describe the image"
"Deseribe this image"
Dense Captioning
"Caption [region]" )
"Caption abject (region)"
"Describe [region]"
"Deseribe object fregion]"
Grounding
"Find the objoct: [caption]"
"Locate the object: [caption]"
"Pick the object: [caption]"
"Select the object: [caption]"
VQA
"Anseer: [question]"
"Answer question: [question]"
answer for [question]"
answer question: [question]"
Relation Detection
'Explain the relationship between: (subject) and [abject]"
"Esphain bow [subject] relates to [ubject]"
"Deseribe the relationship hetween (subject] and (object)"
"Describe how (subject) relates to (object]"
Action Execurion
according instruction: (instmuction""
(instruction)"
"Follow the instruction: [instruction]"
Countextual Routing
"Predier the system act: instruction"
Table 8: Task prompts used for each pretraining and downstream task.
cabinet. the search routine is triggered after the object has been observed inside a container then the agent would need first to interact with the container before retrieving the object. more sophisticated semantic search policy (Blukis et al., 2018:a 30,8,,b, 2022; Chaplot et al., 2020; Min et al., 2021) would likely translate to better performance.
Train
Validation
Break
750
400
Clean
400
200
Close
750
400
Fill
750
400
Goto
750
400
Open
750
Pickup
750
400
E Data Augmentations
Place
750
400
Pour
750
400
Visual Augmentations We leveraged the images used to fine-tune the object detector to create syn- thetic instances. Figure 6 shows comparison be- tween the distribution of objects from the DTC and our synthetic dataset. Our synthetic dataset includes objects that benchmark. To prevent overpopulation of the syn- thetic dataset with objects that occur frequently (tables and desks), we set a maximum number of objects for each action. Table 9 shows the maxi-
Scan
400
200
Search
750
Toggle
750
400
Table 9: Maximum number per object for each action. Search instances used for multimodal grounding which is not supported by the DTC benchmark.
used within the DTC
mum number of each object used for every action used for train and validation.

---

mombitio
DTC
cond ha
Calie Bonn
compun
94
nower ae
pist
12
Canoo
Figure 6: Distribution of objects used in our visual augmentations and the DTC benchmark.
Each image contains ground truth information regarding the position of the agent, the position of the objects well their current states. We used the ground truth metadata ensure the credibility of the synthetic instances. First, we used the dis- tance between the agent and the object to be less than the minimum interaction distance defined by the Alexa Arena. For search instances, we omit this requirement since we want the agent to be able to find any object that is visible from the current view but arbitrarily far away from its position. Second, used the states of the objects determine if an action can be executed on an object. For example, the close action on a fridge cannot be performed if the fridge is already closed. The goto and scan actions are executable regardless of the state of the object. In these cases, we ignore the preconditions regarding the state of the object.
the layout, as well as the position of the agent at the start of the mission.
We collected successful mini-cpisodes and used them to train the model. Note that these mini- episodes were hard for the initial version of EMMA and were only feasible due to external guidance, such as the search routine.
Paraphrazable Instructions To create instruc- tions for both visual and CDF augmentations we used ChatGPT starting point. Examples of in- teractions with the ChatGPT are shown in Figure 7. To create the high-level instruction for each CDF trajectory, we merged the low-level instructions and asked ChatGPT to provide a high-level one. We did not use any sophisticated prompt engincering tech- niques. We only provided introductory sentences along with few examples of semantically similar instructions and asked the model to provide para- phrases. Some of the generated paraphrases do not match the target domain. For example, to clean the bowl the sink, the agent does not require soap. We manually filtered these out before compiling our set of instructions.
CDF Augmentations The objective of the CDF augmentations is to help the agent learn frame and visual token associations. The DTC benchmark includes long trajectories but from our preliminary experiments, we found that the model was not learn- ing these associations, particularly as the number of steps for single instruction increased.
F
Experiments
In order to create CDF augmentations we de- ployed one of the earliest versions of EMMA in the Alexa Arena. We created missions similar to those in the DTC benchmark and manually wrote low-level instructions so that the model could com- plete the missions with one or few-shot interactions without having to perform long trajectories. When creating mission we randomly sampled objects equipped with the affordances that we are inter- ested in for the specific mission. To ensure variety within the missions, we randomized the layout of the room, the position and the color of objects in
Object category
Baseline
Ours
Small (0-1.3k)
37.63
51.90
Medium (1.3k 9.2k)
60.41
89.60
Large (9.2k-90k)
64.72
91.90
All
46.03
56.70
Table 10: Object detection results for small, medium and large objects. The allowed area of object each category is shown in parentheses.

---

instrctting nickor
Imagiees ane resporsicle dle for pars paraphrae mennn censoms inttructione leer flwv
up with few instruction than are cresc simple, and errighttowart
new are froc mecommmplee
Iratructier pot with from the
trrewr plake the
was the piano the Sinv
Pomppee the per with sweeer
Witte Simore partphoose phopphosess of instruction fill with waner "rom the sink
White 5 instrucctios ond have some meaning The insoructios must te fierret from each
Sung hare a passphreasa of the iratructin the oor with water from the aint.
Duri, ne's are S salieee inatructions to clean prate araartic
lddo water to the pot from esr
36 the une thore ght the pank to move any food deoess Claan using
mater
and blo sing solution.
Take from
the
the mug the hamm ment
waner from cank imoorhe
lise cinim remove any dur food part reec place.
need come with creative instrctiion, where someeoddy muslifd hollow repar brckkn bowt.
udting mebeodri brewn hammer carizon
garma, You
with crean suco waye matract them.
o are a exomples
hore ure coupli of csommlos
unt time manhiie impaitin the briionn bowe
emooth m-gg
use ima machhiee
bowi
use the hammen break-chee kmosch mug using hammeeer
repair Broken
Unite 5 instructiona that have same mouning. my. The instructions must cifisrenthoo from
White instrcctirs for sher und. inttrcctins must oe de difleront from oech other
Sune hore a cdl teliinaall ucturctoon for reco brekon bowd using mochnn
Suro, ne are Selffren intructtonn to broak a with hammar corttonn game
Ruinn the noot bark no edigina sone with the time maciiee
Pn ihe mup lnn slocee with hammeee
Tremsport baof through mand
machine.
Cruan mag with hammmr
Setivate muchine
Fiwas the mug nitting hemmmer
Figure 7: Examples of using ChatGPT starting point for paraphrasing goals and subgoals within the Alexa Arena.
Task-specific Prefixes Previous approaches have shown that variations in the text prompt can affect the results on downstream performance (Cho et al., 2021; Gao et al., 2021; Radford et al., 2021). We initially experimented with two types prefixes: task-specific special tokens and natural language prompts. Task-specific special tokens are single- word descriptors (Cho et al., 2021), while natural language prompts are longer, varied descriptions similar the approach proposed by Sanh al. (2022). For example, the image captioning task is denoted either by the single token [Cap] or by prompts such as 'Describe this' or *Caption the image'. Before pretraining our EMMA- we explored this design choice using an EMMA-small variant following BART architecture (Lewis et al., 2020). In particular, loss of the model with tag and text prefixes after pretraining for 75K steps. The model using tex- tual prefixes was slightly outperforming the model using tags (0.818 vs 0.824) which aligns with pre- vious findings (Cho et al., 2021). For this reason,
pickup
place
5.8% 4.4%
7.2%
p pikppsaabiuuuus
Interst
Figure 8: Distribution of CDF augmentations during training.
Model
Accuracy
F1
observed the validation
EMMAA-mddlar
0.96
83.33
EMMA-unified
0.97
84.67
Table 11: Offline evaluation of the CR task.

---

we chose use natural language prompts but also they allow for more flexible interface with the model which is especially useful if the downstream task involves language variety and complexity.
of errors because as the first step in completing an instruction, an error will have cascading effects. We evaluate offline the model performance on the CR task using the validation set of DTC. Table shows the accuracy and macro-average FI score for the model predictions of the CR-specifie to- kens. Both the modular and unified models achieve high scores. However, assuming that action execu- tion and search are performed without errors, the probability of mission success for mission with 5 instructions is 0.03) 0.85.
Data Ablations When training solely on the DTC data, the baseline model outperforms EMMA. To verify that our approach scales better in terms of the data, we also experiment with applying our augmentations to the existing baseline. We train the baseline model using the same set of hyperpa- rameters as in (Gao al., 2023) on the DTC data and vision augmentations. Then, we compare against our model trained on the same data (see red curve in Figure 4). The baseline model achieves a 34.42% success rate, a small performance boost opposed to training with only the DTC data, while model benefits substantially from these augmentations achieving success rate of 34.72%.
G Examples of trajectories
In Section 7 we highlighted the main source of er- rors; the output of the contextual routing task and often the lack of temporal understanding when our model acts on the environment. We observed that when performing the contextual routing task the model can be confused when there multiple ob- ject candidates in the scene that could be suitable candidates for an instruction. When performing multiple actions over the same object, the model may use a token to reference the object in a pre- vious timestep as opposed to the current one. We show two examples of such errors in Figure 9.
Model Scaling We also experiment with small variant of our model (3 encoder layers, 3 decoder layers, 8 attention heads, 368 hidden size, 512 feed- forward size) with total of 20M parameters and the base model trained from scratch on the entire dataset. The performance of the small variant is 32.11%, while the base model trained from scratch performed poorly with success rate of 5.2%. Our results indicate that scaling the model's size up provides substantial performance increase and highlights the necessity of pretraining.
H Visual Domain Transfer
We create synthetic dataset based on scene graphs from the validation set of GQA (Hudson and Man- ning, 2019), which provides cleaned version of Visual Genome scene graphs (Krishna al., 2017). This toy dataset created estimate the model's ability to transfer the downstream task of action execution to the pretraining visual domain, and quantify the impact of using the base or fine-tune object decoder. We generate synthetic instructions by populating templates for the action types: go to, pick up, place, open, close. The templates are in the form of ((action verb synonym) the (optional attribute] (target object)', e.g. 'head towards the wooden table'. We generate go to instructions for all objects, and use the affordances from Alexa Arena to determine if the object class is compati- ble with the remaining action types. We want to avoid cluttered images and ambiguous instructions. Therefore, we keep images with up to 10 objects and skip objects whenever multiple objects of that class appear in the image. We create a total of 57740 instructions.
Object Detector Comparison We validate the performance of our vision encoder by comparing against the baseline (Gao et al., 2023). We evalu- ate our object detection model using the standard COCO evaluation metric, the Mean Average Preci- sion (mAP), calculated by averaging the precision at IoU thresholds ranging from 0.5 to 0.95 in steps of 0.05. Similar to Gao et al. (2023), we the maximum detection proposals 100 for evalua- tion. Table 10 compares our vision encoder with the baseline across different object sizes as defined in (Gao et al.. 2023), Our model achieves strong performance with approximately 40% relative im- provement for all object sizes. We note, however, that our vision encoder is not directly comparable the baseline since the baseline model is trained on fewer classes (86 vs 133) and also performs segmentation instead of object detection.
Offline Contextual Routing Performance Sec- tion concludes that CR is one of the main sources
We also investigate the ability of the fine-tuned model to perform the pretraining tasks in the down-

---

OO
aoproach the apoll
sporoach station apple yrartto
VADs ractioooe, mascho S0 sittpt.
VAG: caypuopen marchs VG.
rision nd puttme ame avnn
rederting monemndddennnn conderss groin with she supio, WAB: zastooome, mandhs offannu rilinn Sh
All yoto char changer talan, VGc.
coler changgee
sollat
cheme Jokenn aesiiinonnn
Vic-
vG:
H
O
BOO
insu
edltuton
Ieput pick thth spode
Inputiion
Ingunt numileh prouch dhedee VADK sacorioma matche off Jriwnn Io Uniienn Lth.
Ioput num af, snporoch dedk
+octmmor matche goto roffice vatoos.
Alt logole rod button
AB pickup spola stoometteen.ceeee to ViL
VADI sytinonnummach matthh <foome <fforee-gnann pwienn"> 2 infi
eatopo
VG.
FREEZE RAY
HEA
RAY
Input: caka
Input: breakreom dact--ione match gola salppe
Input: upper <soa/ch--one match
:search-snne metcre -hamm_cckeen
-act-ione matchy
ROMULDY
EDNO
Input: upper fridge.
Input:
upper fridken.
Lupper fridgge
upper fridgn
sectiiimeumaacc <uis <frame_cokan_.
freezer <fharnerokknn_
<frame_ttkaa
Figure 9; Example of two unsuccessful trajectories. The objective the top mission the apple and deliver it to the red desk. The instruction at the second to last timestep requires the agent to place the apple on red desk.
the color changer on
bottom mission, objective is to freeze the cake in the freezer and deliver to the desk. The agent the last time step tried open the vending machine (visual token 8) which the token reference the freezer in the previous timestep.

---

stream visual domain through qualitative examples. This requires the model not to forget the pretraining tasks and to demonstrate generalization to the vi- sual domain the downstream tasks. To improve model outputs, we prohibit the generation added special tokens (frame, visual, stop and CR tokens). The examples in Figure 10 show that the fine-tuned model retrains VQA. dense captioning and relation- ship detection capabilities and transfers them the simulated environment domain. Figure 11 shows some additional examples of the action prediction task on real images.

---

Input: What type of fruit on the counter? Output: pear
Input: How many computers are the office? Output:
Input: What color is the desk on the left?
Input: What color is the desk on the right?
Output:
Output: green
Input: Describe <vis token. 2> Output: bow!
Input: Caption object Output: a banana
Input:
Explain
Input: Describe the relationship between <vis, vis_token_113- and vosetokeen.402 Output: apple on table
relates to <vis, token Br. Output: banana to the left of cereal box
Figure 10: Examples of transfer of pretraining tasks to the downstream domain after finetuning the pretrained model.
Instruction: Collect the banana Outgut: pickup faranaa-nemm,, i vis Jo satops.
Instruction: Get the orange on your right. Output: pickup carre visemn, tokeen token, <atogy.
Instructions Leave the bowl in sink. Outpul: parce ffarre_okke__ tokan. vatopr.
Instruction: Turn on the microwave. Outpul: toggle micressss <tame.coken.B syls.tkken_d egoops
Instruction: Approach the wink
Instruction:
Instruction: turn wcrite computeer Instructions front refrigenater.
Output: gota fridge shaame,
Outputi gota cabinet <rrom tolenn 1- vy Novoen 2ove wrap
Outputi place bowl <tram toven 1> vuis Jrikannnnnnn. jo- vatope.
estops.
Outputt toggle computer do
Figure 11: Additional examples of the action prediction task the visual domain of real images.


Reviews:
[{"review_id": "JzURqt6IeE", "reviewer": "Reviewer_GKE9", "summary": "", "questions": "1. Can you further elaborate the reason to put Table-1 in the main paper? Comparing with very few previous VLMs is not a very useful experiment to show. Do you mean your model is a new and better VLM with its advanced pretraining/sampling method?\n2. More implementation details should be added. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer GKE9 for their thoughtful feedback. We are pleased that they find our approach interesting. We report below the answers to the comments and questions they have made.\n\nRe: few novel insights and contributions\n\nWhile a unified pretraining is not novel (e.g., Cho et al 2021), applying a unified approach to an embodied task offers new insights because it enables the agent to learn a language of actions that addresses two fundamental challenges within embodied AI: 1) grounding language in trajectories of actions and observations, and 2) referential disambiguation. \n\nMore importantly, our approach departs from current embodied AI approaches that use an ensemble of independently trained models, or use separate heads to independently predict an action and optionally an object class and mask (Blukis et al.,; 2022, Min et al., 2021; Pashevich et al., 2021; Singh et al., 2021). Our model, on the other hand, generates actions and references to objects in the environment as text tokens enabling better performance, higher flexibility, and increased ability to scale up the training.  \n\nFurthermore, pretraining this type of architecture is necessary; an EMMA model trained from scratch achieves 5.22% success rate while our pretraining setup enables our model to achieve 36.81%. This highlights the importance of multitask pretraining, especially for learning the special tokens used to reference objects. Finally, our work shows that, thanks to the learned language of actions, it is possible to achieve strong performance for task completion and the potential to transfer to real images. \n\nWe will add a section showing the benefits of our pretraining in the final version of the paper.\n\nRe: missing details\n\n[Language decoder] As specified in Section 4 (lines 188-189), the EMMA model it’s an encoder-decoder model based on the BART-base architecture. As shown in Figure 2, the language decoder is used to generate the outputs for all downstream tasks, including the V&L tasks and DTC. \n\n[Implementation details missing]  As the reviewer GWnB also pointed out, in the final version of the paper we will include additional implementation details, for example the prompts used during pretraining / finetuning. \n\nWe believe that our approach is reproducible because: 1) we will release our experimental framework based on PyTorch Lightning and Huggingface Transformers together with model checkpoints and inference scripts, 2) we will release all the data and training configurations required to reproduce our model, 3) Appendix A specifies the sampling strategy used to balance the pretraining tasks and 4) Appendix B reports the hyperparameters for pretraining and finetuning our model. \n\nGiven this information, we believe that our setting is reproducible, but we are happy to include any further details suggested by the reviewers.\n\nRe: incomplete comparisons\n\n[Comparison with other VL - Table 1] Our objective was to explore if advances in unified V&L models can successfully transfer to dialog-based task completion in an embodied environment. This introduces the following four constraints: \n\n1) a unified encoder-decoder architecture where different tasks are defined by text prompts\n2) the ability to perform visual grounding in order to complete the embodied task\n3) object-centric representations due to inductive bias for interacting with objects in the environment and limits to input / output sequence lengths\n4) a lightweight model architecture that can facilitate running the costly evaluation in the simulated 3D environment. \n\nWe consider Table 1 as essential because, before evaluating EMMA on embodied AI tasks, it allows us to verify the effectiveness of the pretraining compared to other comparable encoder-decoder models. By comparable we mean models that: 1) are architecturally similar to EMMA (i.e. encoder-decoder models), 2) have comparable size with our EMMA, 3) have visual grounding capabilities (i.e., ability to refer to objects in the scene), and 4) are open-source to ensure future reproducibility. The only available models that fully satisfies the above criteria were VL-BART / VL-T5, which are both outperformed by EMMA on the visual grounding task with the same number of visual grounding instances during pretraining. Additionally, we report UniTab and OFA-base as additional comparisons because they are encoder-decoder models that use patch embeddings. The former is outperformed by EMMA while the latter has a higher parameter count than our model. It is also important to underline that it’s essential to have a lightweight model for Embodied AI task considering that the model has to undergo an expensive inference process when interacting with the environment in real time.\n\nWe will incorporate this justification at the beginning of Section 6.1 to clarify this point raised by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The main focus of training a mutil-modal agent is an interesting topic.\n2. Their method achieves better performance on the Dialog-guided Task Completion benchmark.", "reasons_to_reject": "1. The model and pretraining introduces few insights or contributions comparing to previous vision-language models and existing work. \n2. Some implementation details may be missing and should be improved. e.g., what is the language decoder used for main results?\n3. Though the paper claims to focus on interactive embodied tasks, table-1 compares with an incomplete list of some previous vision-language models on some of the standard VL tasks such as image captioning and VQA. Results are not fully explained.", "questions_to_authors": "1. Can you further elaborate the reason to put Table-1 in the main paper? Comparing with very few previous VLMs is not a very useful experiment to show. Do you mean your model is a new and better VLM with its advanced pretraining/sampling method?\n2. More implementation details should be added. ", "paper_topic_and_main_contributions": "This paper studies interactive and embodied tasks in vision-language. They propose a single mutil-task model, that takes images and trajectories as input and casts action prediction as multimodal text generation, which performs on par with similar models on several VL benchmarks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dEZaBRyA7O", "reviewer": "Reviewer_GWnB", "summary": "", "questions": "Question A: What are the task-specific prompts used for the subtasks?\n\nQuestion B: I'm curious how the visual grounding task works when the object is not found immediately. When you invoke the search algorithm described in Appendix D do you simply select two out of four available viewpoints per room that maximize vertex coverage in the created viewpoint graph? Does that mean you simply iterate through the four directions for each of of the two viewpoints for a maximum of eight possible views until the object is found? In that case does the agent simply navigate through search and object memory without any of the primitive navigation actions being directly invoked by EMMA?\n\nQuestion C: You say that \"when interacting with a fridge across multiple frames, the agent sometimes predicted visual tokens that matched the fridge in a previous rather than the current frame.\" (lines 317-320). Do you find that this behavior, and therefore grounding to the current frame, has improved after data augmentation?\n\nQuestion D: Have you experimented with integrating differently sized pre-trained models (i.e. a smaller model or one that was trained on fewer samples) or larger existing pre-trained models such as OFA-base into your agent? It would be interesting to see how the agent's task performance scales with the performance of the pre-trained model on vision-language tasks.\n\nQuestion E: How does your model compare to the baseline VL model by Gao et al. in terms of model size? How much of the improvement over the baseline can be attributed to using a better VL model?\n\nQuestion F: In Figure 4, is this ablation with or without clarification questions? If it with clarifications, why is performance without augmentations so much lower than the VL baseline? I would expect your proposed augmentations to yield benefits for the VL baseline as well so I am curious whether you have done any experiments applying your augmentations to the existing baseline and how your model compares.\n\nQuestion G: This is more of a suggestion but I think the results from Figure 10 and the simulation to real-world transfer that you mention are quite interesting and deserve to be explored more.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer GWnB for their insightful feedback! We are very pleased that they find the topic of grounding language in vision and actions an important research problem. We are glad that they found our evaluation solid as well as the transferring of the model to real images appealing. We address the questions below:\n\nRe: clearer definition of model inputs and outputs \n\nWe agree that some details require checking the appendix. The task-specific prompts for CR (Contextual Routing) and AE (Action Execution) are not presented in the paper. An example of the VG (Visual Grounding) input-output format is shown in Table 7 for the pretraining though the exact same format used during the downstream embodied task. We provide some examples for the CR and the AE tasks in Question A and we will add the full prompts in the final version of the paper. Finally, Table 7 shows examples of input-output for each pretraining task. \n\nAdditionally, we would like to point to Section 4.2 that provides a more descriptive of the model inputs and outputs for each downstream task.\n\nCR: (line 268-280 & Figure 3) the model accepts the current instruction and observation and outputs a sequence where the first special token determines whether the instruction refers to an action or to search for an object, while the second special token indicates if the object can be matched in the current view. Finally, for manipulation actions the model also generates the name of the object.\n\nAE: (lines 284-295 & Figure 1) the model accepts the observations up to the current time step and the instruction prompt, which includes the instruction and if applicable the inventory object and clarification. The output of the model consists of the action type followed by the name of the object as well as the special frame and visual tokens used to reference the object in the trajectory. Each predicted action results in a new observation which we combine with the previous context to make a new prediction\n\nVG: (lines 298-305 & Figure 2) we invoke the search routine and for each view we use the pretraining VG prompt along with the current view and the instruction. If the object is present in the view the model outputs the special tokens that match it otherwise it outputs ‘no OBJECT NAME’.\n\nWe hope that this is sufficient to understand the behavior of our model and we are happy to restructure any specific sections that might seem confusing or include additional details that you believe are missing. The final version of the paper will be proofread to clarify any unclear details, which we also address in the questions below.\n\nRe: Novelty & benefits over VL baseline \n\nPlease see the response to the questions below.\n\n[Question A] We used the following prompts for each of the pretraining tasks. We will add a complete list of prompt templates in the appendix.\n\n* MLM: \"Reconstruct the description: {caption}\"\n* ITM: \"Evaluate the description: {statement}\"\n* Captioning: \"Caption this image\"\n* Dense Captioning: \"Describe object {region}\"\n* Relationship Detection: \"Explain the relationship between: {subject} and {object}\"\n* VQA: \"Answer the question: {question}\"\n* Visual Grounding: \"Locate the object: {caption}\"\n* Contextual Routing: \"Inventory: {inventory}. Predict the system act: <<commander>> {instruction}\"\n* Action Execution: \"Inventory: {inventory}. Act according to the instruction: <<commander>> {instruction} <<driver>> {question} <<commander>> {answer}\"\n\n[Question B] That is correct. Following the DTC API definition, the agent explores the rooms by going to different viewpoints and approaches objects by selecting their bounding box. This is in contrast with other benchmarks such as ALFRED (Shidhar et al., 2020) that do not have such option.\n\n[Question C] Yes, we find that the CDF augmentations are required for the model to learn frame-visual token associations. Note that our pretraining consists of image-caption pairs which allows the model to learn to ground referring expressions to objects in the image. The skill of learning frame-visual token associations is obtained during fine-tuning only. During the DTC task, the agent has to take into account the temporal order of objects, indicated by the frame tokens. We will consider for future work how to define pretraining tasks that favor learning this ability as well. \n\n[Question D] Yes, that is an exciting experiment that deserves investigation. \n\nDuring the rebuttal period, we also experimented with a small variant of our model (3 encoder layers, 3 decoder layers, 8 attention heads, 368 hidden size, 512 feedforward size) with a total of 20M parameters. The performance of the small variant is 32.11%. We also experiment with the base-model trained from scratch on the entire dataset (DTC + augmentations). The base model trained from scratch performed poorly with a success rate of 5.2%, meaning that pretraining such agents is essential. Unfortunately in the limited amount of time and with limited sources, we were unable to scale up the size of the model. We will add these results to the final version of the paper.\n\nAdding OFA-base could potentially yield higher performance. We note however, that OFA-base employs patch representations which may not be as beneficial as object-centric representations for embodied tasks (Driess et al., 2023; Jiang et al., 2023). We discuss this issue in lines 220-233. We believe that this is an important research question that deserves a more controlled comparison between patch-embeddings and object-centric representations in embodied tasks. However, we leave this additional experiment for future work considering that it would require additional computational resources because OFA-base is bigger than EMMA (69M more parameters) and requires longer input sequences (due to patch embeddings). Additionally, adapting OFA for embodied tasks requires substantial changes that we are not able to complete in the short time available for the rebuttal.\n\n[Question E] The baseline VL model is a specialized / task-specific architecture with 152M trainable parameters while EMMA has only 113M. Each frame and instruction are encoded with CLIP-ResNet-50 text and image encoders followed by a joint 3-layer transformer encoder. Then representations for each time step are passed to a two-layer LSTM and three separate classification heads for action prediction, object class prediction and end of sequence prediction. The object mask is predicted by additional deconvolutional layers. \n\nOur model follows a simpler unified architecture that jointly predicts actions and bounding boxes (specified by the frame and visual tokens). This approach is appealing because it’s easier to scale to bigger and more diverse datasets involving multiple tasks.\n\n[Question F] Interesting question! The ablation is with clarification questions. We were also wondering why the performance of our model without augmentations is worse than the baseline. We hypothesize that this is due to treating the problem as a classification as opposed to generation. We also note that, on top of predicting actions and object masks, we have the contextual routing (CR) as an additional task.\n\nWe also experiment with applying our augmentations to the existing baseline. Unlike the baseline, our approach does not model the “look around” actions which makes the CDF augmentations not useful for the baseline model. We therefore train the baseline model with only the vision augmentations and the DTC data and compare it with our model trained on the same data (see Figure 4, first bullet on the red curve). During training we used the same set of hyperparameters as in (Gao et al., 2023). This model achieves a 34.42% success rate, a small performance boost as opposed to training with only the DTC data (34.20%) and is comparable with EMMA’s success rate of 34.72%. Our model benefits substantially from the augmentations as opposed to the baseline model, which we believe is due to the improvement in the performance of the contextual routing (CR) as this is the only task that is not part of the baseline model.\n\n[Question G] We agree that this is a very promising aspect for future work. We construct a synthetic evaluation set to measure transfer of the downstream action execution tasks to real world images. In addition, we provide qualitative examples of transfer ability of the pretraining tasks (see Figure 5). Unfortunately, creating an evaluation set for this would require human annotations which we will consider in future work. We also conducted a preliminary qualitative evaluation of the VL baseline model and found that this model is unable to adapt to real images making our approach more appealing for sim-to-real adaptation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and deals with the important topic of grounding language in vision and actions through a proposed embodied agent model for the DTC benchmark.\n\nThe proposed method achieves promising results and in particular the data augmentation methods seem to yield significant improvements which make this sort of approach possible.\n\nThe evlauation is thorough and contains a separate performance evaluation for success rates by category and question type as well as augmentation ablations and qualitative error analysis.\n\nI also found the results when transfering the model to real images quite interesting.", "reasons_to_reject": "Although the paper is mostly well written sometimes details can be a bit unclear (see questions) or require re-reading or checking the appendix. For example a clearer definition of model inputs and outputs would be helpful.\n\nIn particular I would be interested in seeing the particular task-specific prompts used for the CR, AE and VG subtasks.\n\nAlthough evaluation results are promising the novelty is somewhat limited: benefits over the existing VL baseline (Gao et al., 2023) from the original DTC paper are small and it is unclear if the comparison between EMMA and this VL baseline are fair in terms of parameter counts and computational costs. It seems simply using a larger model trained on more data would likely yield better results so some more discussion of this aspect of the evaluation would be helpful.\n\nUpdate after rebuttal:\n\nThe authors have sufficiently addressed unclear details, answered all of my remaining questions and stated they will also add the full prompts to the final papers. Given that most of my concerns have been addressed, I have decided to raise the soundness score of my review.", "questions_to_authors": "Question A: What are the task-specific prompts used for the subtasks?\n\nQuestion B: I'm curious how the visual grounding task works when the object is not found immediately. When you invoke the search algorithm described in Appendix D do you simply select two out of four available viewpoints per room that maximize vertex coverage in the created viewpoint graph? Does that mean you simply iterate through the four directions for each of of the two viewpoints for a maximum of eight possible views until the object is found? In that case does the agent simply navigate through search and object memory without any of the primitive navigation actions being directly invoked by EMMA?\n\nQuestion C: You say that \"when interacting with a fridge across multiple frames, the agent sometimes predicted visual tokens that matched the fridge in a previous rather than the current frame.\" (lines 317-320). Do you find that this behavior, and therefore grounding to the current frame, has improved after data augmentation?\n\nQuestion D: Have you experimented with integrating differently sized pre-trained models (i.e. a smaller model or one that was trained on fewer samples) or larger existing pre-trained models such as OFA-base into your agent? It would be interesting to see how the agent's task performance scales with the performance of the pre-trained model on vision-language tasks.\n\nQuestion E: How does your model compare to the baseline VL model by Gao et al. in terms of model size? How much of the improvement over the baseline can be attributed to using a better VL model?\n\nQuestion F: In Figure 4, is this ablation with or without clarification questions? If it with clarifications, why is performance without augmentations so much lower than the VL baseline? I would expect your proposed augmentations to yield benefits for the VL baseline as well so I am curious whether you have done any experiments applying your augmentations to the existing baseline and how your model compares.\n\nQuestion G: This is more of a suggestion but I think the results from Figure 10 and the simulation to real-world transfer that you mention are quite interesting and deserve to be explored more.", "paper_topic_and_main_contributions": "The authors propose a novel embodied agent model for the Dialog-Guided Task Completion benchmark in the Alex Arena. The agent is based on a single vision-language backbone that is used to complete all subtasks such as visual grounding and action execution. To this end the authors pre-train their own vision-language model from scratch and benchmark it against existing VL models. To train their agent the authors further introduce two data augmentation methods which significantly boost performance and enable the model to perform the task successfully. Finally, they evaluate their approach extensively both quantitatively and qualitatively and compare it to baselines and the Alexa Arena leaderboard.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "While I understand the amount of content that can be fit into the main paper is limited some very important information (like the search routine which describes how the agent navigates and therefore seems to be very relevant) is only contained in the Appendix. It would be helpful if the paper was re-structured a little and with references to the Appendix where appropriate."}}, {"review_id": "CsGi7pT3r1", "reviewer": "Reviewer_eEZM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer eEZM and we are happy that they found the motivation for our approach inspiring and our unified model effective.\n\nPaLM-E is indeed concurrent work exploring similar themes that we have cited in our related work section. However, the model is not publicly available and additionally it poses significant computational cost due to its scale. We have opted for a more lightweight model whose training can be reproduced using publicly available data and using a modest computational budget (8 x V100 for pretraining, 1 GPU with 11GB for finetuning). In order to facilitate reproducibility, we will release our experimental framework together with model checkpoints, training data and training configurations. We will add these important considerations in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation and the identification of current challenges on leveraging VL models for interactive and embodied tasks are accurate and inspiring.\n- The proposed EMMA model has simple yet effective architecture with the key idea of unifying everything into text generation.\n- The paper is well-written and easy-to-follow.", "reasons_to_reject": "1. Although it’s probably a concurrent work, it would be great to have some comparison between EMMA and PaLM-E (https://palm-e.github.io/). The main idea seems to be very related.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main topic of the paper is the development of an Embodied MultiModal Agent (EMMA) to address the challenges of interactive and embodied tasks in Vision & Language (VL) models. The key contributions include:\n\n1. Proposing a single, unified encoder-decoder model that reasons over images and trajectories, and casts action prediction as multimodal text generation.\n2. Achieving state-of-the-art performance (36.81% success rate) on the Dialog-guided Task Completion (DTC) benchmark in the Alexa Arena.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers viewed this topic as important, and appreciated the strong results in this paper on the DTC benchmark in the Alexa Arena. While reviewers identified that the model and pretraining procedures largely follow past work, none viewed this as a crucial weakness and I agree with reviewer eEZM, and the authors' response to reviewer GKE9 -- that predicting actions and object references as text is an interesting and likely scalable approach. Some clarification concerns identified by one reviewer were also resolved sufficiently in the author response.
Decision: Accept-Main